The .join() that should be a bug

# The .join() That Should Be a Bug Source: [https://kronotop.com/blog/the-join-that-should-be-a-bug/](https://kronotop.com/blog/the-join-that-should-be-a-bug/) ### Serving thousands of connections on a backend where every call blocks [Section titled “Serving thousands of connections on a backend where every call blocks”](https://kronotop.com/blog/the-join-that-should-be-a-bug/#serving-thousands-of-connections-on-a-backend-where-every-call-blocks) There are two well\-known ways to implement connection management in database systems\. Each way has a cost\.[Kronotop](https://github.com/kronotop/kronotop)takes a third approach\. Let’s start with the problem we have\. Kronotop stores its metadata in[FoundationDB](https://github.com/apple/foundationdb)\. It also stores document bodies on the local file system\. So almost every operation a client asks for turns into a network call to FDB and a read from the disk\. A network call takes milliseconds, not nanoseconds\. Reading a document, committing a transaction, looking up an index: all of it waits on I/O\. Our command path is not in\-memory work\. It is dominated by waiting\. ## Two classic models [Section titled “Two classic models”](https://kronotop.com/blog/the-join-that-should-be-a-bug/#two-classic-models) Redisone thread1 thread✓holds thousands cheaply✕nothing may blockvsPostgresprocess per connectionprocessprocessprocess✓any command may block✕one process per connection, heavyRedis\(before 6\.0\) runs a single thread\. That thread listens on every connection and processes commands one at a time\. This scales connections well, but it forbids blocking\. Nothing in the command path is allowed to wait\. If one command waited, every other client would wait with it\. That is fine when all your data is in memory\. It does not work when a command has to do more\. Our commands call a transactional store over the network, and they append data to a file on disk and call fsync\. Every one of those steps blocks\. Postgres goes the other way\. It gives each connection its own process\. Now blocking is not a problem\. Each connection runs on its own, so you can write plain, sequential code\. But there is a price\! A connection is now an operating system process, and that is heavy\. A few thousand connections become too much\. This is why Postgres deployments almost always sit behind a separate connection pooler\. One model scales connections but cannot wait\. The other can wait but does not scale large numbers of idle/open connections cheaply\. Kronotop needs both\. ## Splitting the connection from the work [Section titled “Splitting the connection from the work”](https://kronotop.com/blog/the-join-that-should-be-a-bug/#splitting-the-connection-from-the-work) The trick is to stop treating “the connection” and “the work” as the same thing\. The connection side works the way Redis does\. We build it on[Netty](https://netty.io/)\. A small number of event loop threads listen on many sockets\. They react to whatever socket is ready\. They parse the incoming command and write the reply back\. This part never blocks and never waits\. So a handful of threads can keep thousands of connections alive\. The work side is the part that does disk and network I/O\. It runs somewhere else, on a[virtual thread](https://docs.oracle.com/en/java/javase/21/core/virtual-threads.html)\. A virtual thread lets us write the I/O call as plain, blocking, top\-to\-bottom code\. It is the Postgres style, but without the Postgres cost\. The virtual thread blocks while it waits on FoundationDB or the disk\. When it blocks, the Java runtime unmounts it and frees the carrier thread underneath for other work\. Thousands of virtual threads can wait on I/O at the same time, while only a few real threads do anything\. When the I/O completes, the virtual thread picks up where it left off\. So the network threads stay free\. They can serve every other connection\. The slow part happens off to the side, where waiting is cheap\. The result is then handed back to the connection thread, which writes the reply\. We keep the write there by design\. That hand\-off is the one rule we keep strict\. CONNECTION SIDEWORK SIDEclients…Netty event loopA few threads serve thousands of sockets\. Parse and write\.Never blocks, never waits\.the reply is always written hereVirtual threadtr\.get\(…\)\.join\(\)blocks on FDB or diskThe runtime parks it\. The carrier thread is freed for other work\.1offloadsupplyAsync\(…, vtExecutor\)2resultthenAcceptAsync\(…, nettyExecutor\)In code, the whole offload comes down to two moving parts: ``` CompletableFuture .supplyAsync(supplier, context.getVirtualThreadPerTaskExecutor()) // run on a virtual thread .thenAcceptAsync(action, response.getCtx().executor()); // resume on the Netty thread ``` Two executors, two phases\. The`supplier`is the slow part\. It runs on the virtual thread executor, where blocking on FoundationDB is allowed\. The`action`is the reply\. It runs on`response\.getCtx\(\)\.executor\(\)`, the Netty event loop that owns this connection\. So the work that waits and the write that must not move stay on separate threads\. The second only starts once the first is done\. A read command is written against exactly that shape\. The first block fetches the value, the second sends it: ``` public void execute(Request request, Response response) { AsyncCommandExecutor.supplyAsync(context, response, () -> { // virtual thread: open the transaction and wait on FDB Session session = request.getSession(); Transaction tr = TransactionUtil.getOrCreateTransaction(context, session); DirectorySubspace subspace = openZMapSubspace(tr, session); return tr.get(subspace.pack(message.getKey())).join(); }, (value) -> { // Netty thread: write the reply response.writeFullBulkString(toMessage(value)); });} ``` The`\.join\(\)`in the middle is the interesting line\. On a normal thread it would be a waste\. The thread would sit there, blocked and useless\. On a virtual thread it is the whole point\. The runtime parks the call and frees the real thread\. It resumes the call when FDB answers\. The handler reads top to bottom, like plain blocking code, and the cost of waiting is gone\. Once the supplier returns, the executor moves to the reply phase\. If the command was an auto\-commit, it cleans up the transaction\. If something threw, it writes the error back to the client\. You can see the relevant implementation parts from the following links: - [AsyncCommandExecutor](https://github.com/kronotop/kronotop/blob/main/kronotop/src/main/java/com/kronotop/AsyncCommandExecutor.java) - [ZGetRangeHandler](https://github.com/kronotop/kronotop/blob/main/kronotop/src/main/java/com/kronotop/zmap/handlers/ZGetRangeHandler.java) - [BucketQueryHandler](https://github.com/kronotop/kronotop/blob/main/kronotop/src/main/java/com/kronotop/bucket/handlers/BucketQueryHandler.java) ## What lives on a connection [Section titled “What lives on a connection”](https://kronotop.com/blog/the-join-that-should-be-a-bug/#what-lives-on-a-connection) A connection carries a small amount of state while it is open\. It knows which client it is, which namespaces it has open, and whether it has authenticated\. It also holds the details of any transaction in flight\. That last part matters\. In Kronotop, a transaction belongs to the connection\. You begin it, run several commands inside it, and then commit or roll it back\. The whole time, that transaction is tied to your session\. If your connection drops in the middle, the open transaction is cancelled and cleaned up\. Nothing is left dangling\. You can also reset your session state without dropping the connection\. This clears cursors, watched keys, and any half\-finished transaction in one step\. That reset is also what makes a connection easy to reuse in a client\-side pool\. You hand it back, reset it, and hand it out again\. A connection is a sessionclientidentityopennamespacesauthstatetransactionin flightConnection dropsthe open transaction is cancelledand cleaned up, nothing danglesRESETclears cursors, watched keys,and any half\-finished transactionIf you read this post until the end, why not try Kronotop? Kronotop is a distributed, transactional document database built on FoundationDB\. See the[Quickstart](https://kronotop.com/docs/quickstart)guide to start a cluster in a few minutes\.