I think that it’s more the benefit of offloading that responsibility to others such as Amazon, instead of attempting to roll your own database solution.
It’s the question of „I want to design good software“ or „I need a solution as cheap as possible“
Sorry but of course you can store data in megabytes of json columns and hope that the hardware will be always fast enough to search for it without any care of the database design, what mostly result in software where the user has 10 or more seconds to wait for a result, or I can create a good database design where an old c64 will be fast enough to deliver all needed data in less then 1ms.
But yes, the future is the first solution because it’s cheaper to create bad software from some Indian programmers by merging as many existing open source software matching as possible to nearly fulfill the needs, instead of creating a new good designed software which is fast, performant and does exact what is needed.
1 Like
“Cloud Delivery Network” – and yes i had to look it up. 
do you have a favorite? like firebase perhaps?
Not exactly. It is faster persist flat files of json to a CDN and fetch those files than a database. However, as you have mentioned this does not work great for search.
My professional recommendation for search is https://opensearch.org/. Open Search is the open source predecessor of elastic search offering several new features. Not to plug aws more than I already have but AWS has also replaced their search service with a production grade managed open search solution: https://docs.aws.amazon.com/opensearch-service/. Search is an area though where it is difficult to achieve zero cost. However, I think the pure power of open search out weighs the cost factor which isn’t terribly expensive.
However, if you are seeking a low cost alternative that works well with s3 AWS Athena: https://aws.amazon.com/athena. With Athena you can consume unstructured data in mass quantities and catalog it as structured data. Once cataloged SQL can be used to query the info. The very nice part of this service is it is serverless with a pay for use model instead of time. The amount of free queries and data transfer is probably more than enough for many apps.
These are some other high profile search solutions which I have also really liked.
- Typesense: https://typesense.org/
- Algolia: https://www.algolia.com/
Again though not incredibly cheap to run a production grade cluster. However, all of these are highly available, auto scaling cloud solutions unlike a single db instance.
Correct. One of the advantages of amazon is any services api can be used directly in the browser securely. This eliminates the need for both a database and server entirely for modern JavaScript browser native applications like React, Angular, Vue, etc. You can merely implement services as needed to take the place of a relational database, email, queue, etc. All with generous free tiers on aws.
Building an application to start with production grade performance, low latency, scalability, robustness, flexibility, and cost is well thought architecture. What is not great is building an application in isolation of these things and just tossing it up on a server. Distributing responsibilities of an application across services specific to use case is responsible well intended design. Slimming down traditional stacks by completely removing servers and databases is environmentally responsible software architecture.
I have dedicated myself to AWS which offers more than enough to achieve just about anything and is a highly coveted skill in the industry. However, I am intrigued by some things firebase offers. I think the aws free tier surpasses all vendors out there. You can do a lot for free in AWS. The free tier does end after a year but that is only for a the account. Nothing stops you from creating another separate account and migrating resources over there. This model effectively creates a lifetime of free tier on aws.
One thing I really like about aws is being able to securely call all the rest apis with fine grained access control directly in the browser. This completely eliminates the needs for servers. Instead applications can be built as mere JavaScript apps that run completely in the browser and communicate with the cloud directly. Eliminating servers is a key aspect in reducing ongoing infrastructure costs and reducing world wide energy consumption.
This mentality is one I’ve been researching and experimenting for a while. Running enterprise grade JavaScript apps without servers as inexpensive as possible without sacrificing latency, scalability, security, and overall performance.
There are a lot of options for this even relational databases using AWS App Sync https://aws.amazon.com/appsync/. In App Sync you can run secure fine grained access controlled graph queries directly from the browser to consume and persist data to relational databases. App Sync is pretty neat if you have an existing relational database but would like to spend less time redesigning data storage and more time focusing on delivering a modern web experience for users using a JavaScript app.
At work I have floated around the idea of using App Sync since we are somewhat married to a relational database. We currently run a grails app (MVC) Java rest API with a JavaScript browser app. The MVC rest api can be eliminated an replaced with App Sync graph queries directly in the browser instead. Completely eliminating an entire tier from our stack. This is the same for any modern JavaScript front-end app that relies on a proprietary rest api.
This demo puts some of these into action. In particular running a JavaScript app in the browser without any servers that communicates directly with aws s3 and open search rest api. Opening the network tab will reveal fetches to s3 and open search using secure signed http requests that enable fine grained access control of resources and operations in aws. This is a fairly new concept in the industry known as 0 trust and all AWS rest APIs and asset linking support it.
That entire page is actually configurable via an editor.
When saved the page is stored inside s3 as a json file. There is also a partial copy inside open search. The previous page matches routes using open search queries. All this is achieved directly in the browser. No servers except for those running the open search cluster which at the moment is only a single node.
2 Likes
This is a form that collects data and upon clicking save pushes it as a json file to aws s3. The bucket is private and only authenticated users can write to it. Therefore, the request will fail and it will be clearly visible in the network tab.
https://ng-druid.github.io/native_forms_rebuild_v1/89087abb-326d-4a93-888e-9c597ba81b8e
This is super fast in comparison to a relational database since all that is happening is a file is being written to storage. It doesn’t really matter the complexity of the object since its just json. In contrast to relational tables where as the complexity of objects increase the liklyhood is high that performance will become problematic. Furthermore, unless using an orm of some kind that can easily manage relationships nested objects and arrays that are associations increase all around complexity of the business logic. In this model the “entity” is treated as one single document and merely just pushed as a file of json. Super simple in comparison to persistence of complex domain entities backed by a relational database.
It sounds like though that its currently theoretical between the two, and that actual measurements haven’t been done yet.
2 Likes
While I have not conducted performance tests everything I have read and experienced indicates that writing json files to s3 out performs all rest apis backed by a traditional database that I have used. The write is nearly immediate less than 100 ms for a json file. I’m definitely open to alternatives for json file storage that can be run at no cost in the cloud at production grade scale and availability. That type of write speed on a consistent basis might be achievable using graph without a middle layer api built in php, java, .net, nodejs, etc but this is not something I have explored.
AWS S3 provides a great performance. It automatically scales to high request rates, with a very low latency of 100–200 milliseconds . Your application can achieve at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per prefix in a bucket
This being said you might have given me a reason to learn k8 to conduct some performance testing of my own.
It would be interesting to compare the performance of say hitting s3 with 3000 writes in parallel compared to a rest api backed by a database. However, I think it would be difficult to achieve a true picture of the performance benefits given all the variables of a traditional rest apis and databases. Though it seems common sense that removing the server and database from the equation and writing directly to object storage would by nature perform much better at scale.
I think a better measure is the average time it takes a post/put rest api built in all top languages to respond. I don’t typically see less than 1 - 2 seconds for your average rest api writing complex objects to a relational database. That number only becomes worse as more traffic hits the database at once.
Also I have found that Observability over time is the best measure of performance. Observing request durations on the production environment provides the best picture of performance. This type of performance monitoring can be combined with ai and machine learning to provide a true picture of how an application is performing in real time.
is MySQL still in existence? it seems to me that MariaDB has all but taken over MySQLs base.
We actually just migrated from MySQL to Mariadb including schema and data level encryption. This is not something offered by MySQL but is part of MariaDB. So if someone hacks our servers the data on them is worthless without the encryption key. We operate in aws gov cloud and have slowly been improving our security implementing schemes and data level encryption was a major step in the right direction not to fall victum easily to hackers. My opinion is to do away with MySQL entirely but that is a monumental effort for an existing application.
when i was first looking at MariaDB, they “advertised” to just do the following:
- install MariadDB
- stop MySQL
- start MariaDB
and that is all that was required. i was very skeptical, but i tried it late Saturday night, and really i could not tell any difference at all.
i dont know if MySQL has incorporated all the NoSQL options that MariadDB has. there is a lot they are doing that i dont have the time to even keep up with.
We have had several hurdles with a database that goes back about half a decade. Much of it was related to degrading performance. Thankfully I work more on the application side and only know about this type of stuff because it’s a small team with scrums each day. So I didn’t really need to work on that. Like I was saying previously managing databases takes a completely separate skill set that can be eliminated from app dev by simply using more modern technology. I prefer to spend as much time as possible on the front end and as little as time as possible moving an app from dev to production grade environment. I have found the cloud combined with mvvm to be a worthy tool in accomplishing that goal. I speak from a perspective of having used relational databases for over a decade. It is so much easier to use NoSQL solutions than it is to reliably implement a performant scalable production grade app with relational databases.
if i knew where you were, i would buy you an expensive dinner and ask you about referential integrity concerns. 
for a time there i was playing with CockroachDB and it also looked pretty good, but felt a bit too experimental for me.
And ACID. And the CAP theorem.
hey paul - you may be interested in this: https://blog.bitsrc.io/stop-using-function-parameters-now-d320cf0932df
i could not get the first example to work (found out the example is in typeScript), but it sounds like what you are talking about.
NoSQL databases often make tradeoffs by relaxing some of the ACID properties of relational databases for a more flexible data model that can scale horizontally. This makes NoSQL databases an excellent choice for high throughput, low-latency use cases that need to scale horizontally beyond the limitations of a single instance.
Each of his code examples gives both the typescript and the javascript version of the code. You can use only the javascript version of the code to work through his article.
ya, but look at the comments in the article. the author only included the jScript examples after i asked for it. and yet i did not receive any credit whatsoever. 
(sorry i could not resist that last quip)
1 Like
This topic was automatically closed 91 days after the last reply. New replies are no longer allowed.