This is a nice article, and I appreciate the examples. The next problem to solve is how to persist state on disk across two different accounts after a transfer has been done.
microvms as espoused by things like firecracker offer full machines but have tradeoffs like no gpu (which makes it boot faster)
hyperlight shaves way more off - (eg: no access to various devices that you'd find via qemu or firecracker) it does make use of virtualization but it doesn't try to have a full blown machine so it's better for things like embedding simple functions - I actually think it's an interesting concept but it is very different than what firecracker is doing
FoundationDB has been growing as my favorite database lately. Even though it is only key-value store.
Out of curiosity: what are the scale limits of FoundationDB? What kind of issues would it start to have? For example, being able to store all of Discord messages on it?
I see blog posts of Discord moving to Scylla and ElasticSearch, but I wonder if there would be any difficulties here.
There are a lot of strict limits so AFAIK everyone uses FoundationDB for fast, consistent, highly-available metadata while doing replication/storage of actual data elsewhere (such as in S3).
In their data modelling pages they mention you should break up rows into separate keys per column. Or separate keys per field in a document. This is indeed how many databases model rows on a distributed kv store. So this might be how they achieved 100TB.
However you still have the issue of any single key-value needing to be in their limit. (But it's not like people typically store enormous blobs in Postgres or MySQL either I think?)
The documentation is woefully out of date, sadly. Despite the code being in active development no one is touching the public docs. Though I don’t know for sure, that limitation was probably written something like 10 years ago.
Before you execute the query, you should be able to query any of the data, and after you execute the query, none of the data should be available. The mechanisms to make a transactional database is tricky.
Some databases, like CockroachDB, provides some built-in TTL capabilities.
But also- if you are having to delete huge ranges of data and do not care about consistency, you are probably looking at an analytical workload, and there would be better databases suited for that, like Clickhouse.
As written, that's not required. The data should not be retrieveable by query, but ACID only specifies what happens at the client-server boundary, not what happens at the server-storage boundary (Durability prescribes that the server must persist the data, but not how to persist it). A database that implements DELETEs by only tombstoning the row and doesn't discard the data until the next re-index or vacuum operation would still be ACID-compliant.
As usual, there is a spectrum of data safety vs. performance.
Redis is at the "very fast, but unsafe" side of the scale.
ScyllaDB for me is in the middle of being high performance key-value store, but not really supporting transactions. FoundationDB is another one that I would consider.
Depends on the kind of safety you’re looking for. Redis is entirely safe from concurrency issues because it’s single-threaded. It supports an append-only file for persistence to disk.
Not sure what the poster above was thinking of, but it seems kinda the same as every other language that’s adopted it - powerful, but footguns abound. I ran some async C# in a debugger - in Rider - the other month, and the debugger just goes off the deep end.
Does C# have the same issue Python does with accidentally calling blocking code? In async Python - which I mostly quite like actually - it’s terrifying to bring in a library because if you’re unlucky it does some blocking network call deep inside that stalls your whole app..
.NET uses threadpool with hill-climbing algorithm. Each worker thread has its own queue and can also perform work stealing. However, if that is not enough and the work queues fill up faster than the items in them get processed, the threadpool will spawn additional threads until the processing lag is counteracted (within reason).
This historically allowed it to take a lot of punishment but also led to many legacy codebases to be really shoddily written in regards to async/await, where it would take some time for threadpool to find the optimal amount of threads to run or devs would just set a really high number of minimum threads, increasing overhead.
In .NET 6, threadpool was rewritten in C# and gained the ability to proactively detect blocked threads outside of hill-climbing algorithms and inject new threads immediately. This made it much more resilient against degenerate patterns like for loop + Task.Run(() => Thread.Sleep(n)). Naturally it is still not ideal - operating system can only manage so many threads, but it is pretty darn foolproof if not the most foolproof amongst all threadpool implementations.
As of today, it is in a really good spot and with tuned hill-climbing and thread blocking detection you would see the .NET processes have thread counts that more or less reflect the nature of the work they are doing (if you don't do any "parallel" work while using async - it will kill most threads, sometimes leaving just two).
> Does C# have the same issue Python does with accidentally calling blocking code?
This can't really happen in C# except maybe if you are working on a GUI thread and you make the mistake of running blocking code on a GUI thread.
For APIs, console apps, and such, it's not a concern for actual parallel-concurrent code. Of course, if you write a standard non-parallel `for` loop that has blocking code, it's going to block in a console app as well if you don't run it on a thread.
But I think that once you do enough JS/TS or C#, `async/await` doesn't feel very onerous if you have the basic concept.
Python's async works on a single thread, C# uses a thread pool. Calling a blocking method is not ideal, but doesn't ruin everything, and it's easy to hand that work off to a separate thread by using Task.Run.
Agreed. The task system in C# is pretty clean imo. Same with Rust (sans the type system implementation of Futures) and Go's goroutines. Especially compared to CompletableFuture in Java.
I am happy using CockroachDB. The performance is not as good, since all your database writes require a 2 out of 3 quorum. But managing the database with the CockroachDB is pretty simple, since it can perform a rolling upgrade with no downtime.
Upgrades is handled with an operator, and happens by waiting all queries to finish, draining all connections, and restarting the pod with the newer version. The application can connect to any pod without any difference.
I perform upgrades twice a year, never really worried about it, and never had any availability problems with the database, even when GCP decides to restart the nodes to update the underlying k8s version.
