Redis is not a map you talk to over TCP

# Redis is not a map you talk to over TCP Source: [https://blog.verygoodsoftwarenotvirus.dev/posts/2026/09/22/redis-is-not-a-map-you-talk-to-over-tcp/index.html](https://blog.verygoodsoftwarenotvirus.dev/posts/2026/09/22/redis-is-not-a-map-you-talk-to-over-tcp/index.html) I work on a service that decides which courier gets offered your delivery at a gig economy delivery app\. When you open the app and ask for something to be brought to you, some code somewhere has to decide which of the couriers should be offered the job\. We’re the ones who make those offers\. I’m not at liberty to discuss the myriad parameters that go into the decision, but a pretty obvious one you can suss out externally without knowing much is distance\. To make good matches, we need to know*how far away is everybody, really*\. Straight\-line distance will cheerfully tell you that a courier on the far side of a river is close by, so the number we actually need is driving time\. For that we lean on a routing engine, which is far and away the biggest contributor to our own latency\. When it’s slow, we’re slow, and when we’re slow, the app is a worse product\. ## The shape of the problem A busy area needs a tremendous amount of route estimates to get through a single pass at assigning work, and it needs them again continuously, for as long as the business is running\. Volume isn’t a choice anyone gets to make\. Scale is the requirement, not the optimization\. \(More on this later\.\) The useful thing about those estimates is that they’re frequently repetitive\. It doesn’t take the guy three houses down from me any meaningful amount more or less time to get to the grocery store than it takes me, and neither of us could tell the difference between driving to that grocery store and driving to the gas station in its parking lot\. Restaurants, meanwhile, do not move at all\. Two couriers a block apart produce two nearly identical route requests, and thirty seconds later, from two new positions, they produce two more\. Without any form of caching layer between us and the routing engine, you’re invariably duplicating work, re\-calculating what are functionally the same distances on a regular cadence for sometimes dozens of minutes at a time\. A process\-local cache doesn’t get us there: the process that computes an estimate is rarely the one that needs it next, so the cache has to be shared\. I don’t like to reflexively reach for caching, anecdotally it’s been a “but this will make it faster\!” button for naive implementers\. This felt like it fit the bill in a different way\. ## The naive approach Caching on raw coordinates is a non\-starter\. You need to use[H3](https://h3geo.org/)to dedupe the coordinates\. H3 chops the earth into hexagons at different resolutions, and gives you a stable ID for a hex containing a given coordinate\. Snap the origin and destination to a hex, and now every coordinate pair inside those two hexes collapses onto one cache key\. The resolution is a precision dial\. Lower resolutions give you a great hit rate and a worse answer, higher resolutions give you an answer relevant to almost nobody\. The resolution is baked into the key itself rather than assumed, so two resolutions can coexist in the cache, and you can move between them without a flush\. So the key was`<origin hex\>:<dest hex\>:<resolution\>`, the value was the estimate, and the write path was an`MSET`\. Compute a batch, write the batch\. I dusted my hands, shipped it, and sat back ready to reap the rewards of forthcoming scale\. It did not scale even a little bit\. ## Not so fast, there Our Redis is a cluster, and clustered Redis splits the keyspace into 16,384 hash slots divided up among the primary nodes\. A multi\-key command like`MSET`is only legal if every key in it lands in the same slot\. I found this out via trace\. A single read was showing up as scores of**individual`MGET`spans for a singular key each**, and that was likely an undercount\. The root span reported the total key count, and the arithmetic didn’t come close to agreeing with the number of child spans that had survived, so the Otel collector was almost assuredly dropping them\. Our max read latency metric was way higher than our anticipated worst case\. This is how I learned slots exist\. ## Slot it to me Which slot a key lands in is neither configurable nor random\. It’s`CRC16\(key\) mod 16384`, so two keys that differ by one character land in slots that have nothing to do with each other, which is wonderful for even distribution and catastrophic for batching\. My keys had a different origin hex, destination hex in every one of them\. The odds of any two of them sharing a slot were, functionally, 1 in 16,384\. The client wasn’t being silly\. A multi\-key command can only address one slot, so a well\-built client takes your single`MGET`of a ton of keys, groups them by the slot each one belongs to, and puts every group on its own wire\. Given what I’d handed it, that was the correct thing to do\. There is no combination of settings under which one`MGET`per key is fast\. The fix isn’t in the client’s config, it’s in handing it keys that don’t live in so many different slots\. The write path had the same disease wearing a different hat: either an outright`CROSSSLOT`error, or one`SET`and one round trip per key\. ## \#hashtags Redis has an escape hatch exactly for this, called a hash tag\. If a key contains a chunk of text between curly braces, then Redis hashes*only*the text between the braces and ignores the rest of the key\. That means you get to decide, deliberately, which keys share a slot\. client process primary 1 primary 2 primary 3 \#01 \#02 \#03 \#04 \#05 \#06 \#07 \#08 \#09 \#10 \#11 \#12 1/4 · A batch of keys, still in the process**0**round trips `89283082a53ffff:892830828efffff:9` Twelve route keys, 3 primaries\. Hashed whole, every key lands in a slot of its own and one MGET turns into 12 round trips\. Wrap a shared tag in braces and only the braces get hashed, so the same twelve keys collapse onto 3 slots: 3 round trips, one per node, running at the same time\.The trap is what you choose to put in the braces\. The tempting move is something semantic: tag by origin hex, so all the routes out of one hex batch together\. That’s how you build a hot spot\. Downtown at dinnertime is one hex, one tag, one slot, one node, and that node is now on fire while the rest of the cluster snoozes\. What you actually want is arbitrary key nihilism\. You want the tag to carry no meaning at all, and to be chosen so the buckets land evenly across the cluster\. So the tag became an integer, found by brute force\. The tag is really a template, something like`\{routing:v1:<n\>\}`, and at startup we walk`n`upward from zero, hashing the whole tag each time to see which primary owns the slot it lands in\. If that node still has room, keep the integer\. If it doesn’t, throw it away and try the next one\. Stop when every node has as many as we asked for\. What falls out is a list of integers that are known\-good addresses: a configured number of them per node, deliberately spread across all of them\. Nothing about any particular number is meaningful; it’s just the first integer whose hash happened to land on a node that still had a vacancy\. Two nodes with two apiece might hand you`1, 2, 3, 5`\. Ask for three apiece and the next ones might be 8 and 11\. The whole answer is a function of the cluster’s shape, and it’s recomputed deterministically every time the process starts\. Once the tag is doing the work, a batch of thousands of keys can be split into a handful of piles, and every pile is a single legal multi\-key command aimed at exactly one node\. ## Watching the walk Reasoning about which integers land where is not something you can do in your head, so here’s the walk run for real\. Every integer from zero upward gets dropped into the tag, hashed with the same CRC16 Redis uses, and handed to whichever primary owns the slot it lands in\. If that primary still has room, the integer is kept; if not, it gets thrown away and the walk moves on to the next one\. primaries5tags per primary4 n = 0, 1, 2, … until every primary has 4 slot 016383 primary 10–3276 primary 23277–6553 primary 36554–9830 primary 49831–13107 primary 513108–16383 0/28 tried · 0 skipped**0**of 20 kept `anchors = \[\]` The tag template is`routing:v1:%d`\. Each integer gets hashed exactly the way Redis will hash it, and lands on whichever primary owns that slot\. If that primary still has a seat, the integer is kept; if not, it's thrown away and the walk moves on\. For 5 primaries at 4 apiece that takes 28 integers, 8 of them landing somewhere already full\. Nothing is stored: every process that runs the same walk against the same cluster gets the same list\.The integers it keeps look arbitrary because they’re just the numbers whose CRC16 happened to land where I needed it to\. Early on nearly everything gets kept, because every primary has room\. It’s only near the end, when most of the cluster is full and the walk is fishing for the last few seats, that it starts throwing numbers away\. Change the number of primaries and the whole answer changes, which is the honest version of what resharding does to a scheme like this\. ## Sorting keys before you fan out Collapsing that fan\-out fixed the span count, but it mattered for a second reason as well:*which*node a concurrent request lands on decides whether the concurrency does anything at all\. keys per chunk41. 3 commands 2. 3 commands 3. 3 commands 4. 2 commands 5. 2 commands 6. 3 commands primary 1 primary 2 primary 3 0123456 command times · 16 commands**0\.0**to finish 24 tagged keys, 3 slots, 3 primaries, every command costing the same round trip whatever it carries\. Cut the list into chunks and each chunk still holds keys for more than one slot, so the client splits it and every primary gets one command per chunk to run one after another\. Group by slot first and it's one command per primary, all of them in flight at once\. Chunk size is the whole dial, and it runs the wrong way: the finer you cut, the more commands you make, which is how fanning out over more goroutines buys you a slower read\.Naively, you take your key list, chunk it, and fan the chunks out across some goroutines\. This feels like parallelism and mostly isn’t\. If the chunks aren’t organized by destination, several of them will target the same node at the same time, and that node will do them one after another while other nodes sit idle\. Your throughput ceiling ends up being whatever your unluckiest node can do serially\. So before anything goes out on the wire, we compute the slot for every key locally, which is just a cheap CRC16, and group by it\. Then the fan\-out is over*nodes*rather than over arbitrary chunks, and every in\-flight request is doing useful work somewhere different\. ## MSET doesn’t do expiry Here’s the one that genuinely annoyed me\. Cached route estimates have to have an expiry\.`SET`takes an`EX`argument\.`MSET`does not take anything\. There is no`MSETEX`\. You can have your bulk write or you can have your TTLs, and Redis will not sell you both\. The usual workaround is to pipeline a`SET \.\.\. EX`per key, which gets you correctness at the cost of turning one command into thousands\. Or you can`MSET`and then`EXPIRE`in a second pass, which doubles your commands*and*leaves a window where a crash between the two passes strands keys in the cache forever\. The answer was to send Redis a script instead\.`EVAL`runs Lua on the node, atomically, and the script can do whatever`MSET`refuses to\. ## JSON is not free The last one is embarrassing in hindsight\. The cached value started as JSON, because of course it did\. It’s two numbers, a duration and a distance, wrapped in the most convenient serialization format in the world\. At our scale, “the most convenient” and “the cheapest” stop being the same thing\. Every read meant a full JSON decode, which consumes a meaningful amount of CPU time\. So the value became CSV:`412\.3,5120\.7`\. Smaller, no field names, no reflection, and parsing is a`strings\.Cut`and two`strconv\.ParseFloat`calls\. ## What I’d tell myself at the start None of this arrived in one piece\. It was the product of a lot of inching forward, and every single step was a thing I learned by watching the previous step fail in a way I hadn’t predicted\. Being a Go fanatic, I’m naturally reverent of Rob Pike, and in particular, his[5 rules of programming](https://web.archive.org/web/20260318104226/https://www.cs.unc.edu/~stotts/COMP590-059-f24/robsrules.html)\. Two of them describe exactly how I got here: > **Rule 2\.**Measure\. Don’t tune for speed until you’ve measured, and even then don’t unless one part of the code overwhelms the rest\. **Rule 3\.**Fancy algorithms are slow when*n*is small, and*n*is usually small\. Fancy algorithms have big constants\. Until you know that*n*is frequently going to be big, don’t get fancy\. I took Rule 3 as permission and skipped Rule 2 entirely\. I never benchmarked anything, I wrote the simple thing, it was fine on my machine, and I shipped it\. What I neglected to contemplate was that in our workloads*n*is not usually small, it’s reliably large\. The part I’d glossed over is that Rule 3 comes with an escape clause:*until you know that n is frequently going to be big*\. I did know\. What I’d actually done was take a rule of thumb about the general case and apply it to a specific case I had measurements for and hadn’t bothered to look at\. The huge read times didn’t teach me anything our existing dashboards couldn’t have told me before I wrote a line of it, and much more cheaply\. Rule 2 is the one that would have caught it, and I think it’s usually taught too narrowly\. Measuring isn’t only for deciding whether something is worth optimizing, it’s also for finding out which regime you’re in at all\. In our workloads*n*is reliably large, and every change in this post came from eventually looking at a number that was already on a dashboard before I started\. The honest reason I skipped it is that measuring used to be work\. I cut my teeth in this industry mostly at nascent startups where the default is ship now and fix later, and later shows up as a trace\. A benchmark meant a harness, and a harness meant inventing input that resembled production, and all of that meant explaining why a two\-point ticket was now a five\. I have consequently never developed a benchmarking reflex\. That excuse has a shelf life, and I think it has expired in the post\-Claude era of programming\. Asking a model for a benchmark is a minute of my time, and generating plausible input at a realistic size is precisely the sort of tedium it’s good at\. I don’t know whether the industry at large has moved on this \(I’d guess the ticket\-points argument is as alive as it ever was\), but mine has moved completely\. The cost of finding out which regime I’m in has dropped far enough that not knowing is a thing I’m choosing now, rather than a thing I’m stuck with\. But the thing I’d actually go back and tell myself has nothing to do with benchmarks\. It’s that I was carrying the wrong model of Redis around in my head\. I thought of it as a map I could talk to over TCP\. That’s a fine model when you’re fetching a user’s session by their ID on page load, which is most of what anyone ever uses Redis for\. It is a terrible model for what we were doing here, and every problem in this post is a place where the map quietly stopped being the truth\. The map model isn’t wrong so much as it’s scoped\. It accurately describes one key and one round trip, and it’s what every tutorial shows you, because that’s the case almost everybody has\. Push past it and the parts it was eliding start filing complaints\. Concurrency isn’t concurrency if every request lands on the same command thread\. A TTL turns out to be a property of a particular command rather than of the store\. None of this is obscure, all of it is in the documentation, and none of it ever comes up while you’re getting sessions by ID\. I suspect that’s the general shape of it rather than anything specific to Redis\. The reason an abstraction is worth having is that it hides the machine, and the reason it eventually bites is that it hid the machine\. The only reliable tell I’ve found is volume\. Once you’re doing something at the scale we operate in, the mental model you picked up doing it once per page load is almost certainly load\-bearing in ways it wasn’t built for, and it’s worth going to read what’s actually underneath before production explains it to you\.