Distributed systems for web developers

You never decided to build a distributed system.

You deployed to a platform that autoscales. You added a cache because a page was slow. You wired up a webhook so billing would stay in sync. You put a background job on a timer. Each of those was a normal Tuesday, and not one of them came with a warning that your application had stopped being a single program and become several programs that can disagree with each other.

This post is not an introduction to distributed systems theory. You do not need CAP, consensus, or vector clocks to run a web app, and being handed them is why most writing in this genre is useless to the people who need it. What you need is a list of what actually breaks, roughly in the order you will meet it, and what each failure looks like from inside your own logs.

There are four.

1. There is more than one of you

The first thing that changes is that in-memory state stops being shared. Every lock, cache, rate limiter, counter, and deduplication set in a module-level variable is now per instance. It works perfectly on your laptop, where there is one process. In production there are several, each with its own private copy.

What it looks like: a mutex or singleflight that demonstrably works and demonstrably does not prevent the thing it exists to prevent. A rate limiter that lets through roughly N times your limit. A cache that is stale for some users and not others.

Coordination has to live somewhere every instance can see: your database, or something built for it. An in-memory lock is not coordination when the thing you are coordinating with is another machine.

Here is the part that cost us more than the original bug. We hit exactly this, moved coordination into what we described as "the single-writer store," shipped it, and watched the problem stop. Then an alert fired from a different server, because there is one store per server. Every safeguard we had just built was scoped to a boundary the problem crossed.

A coordination mechanism that is correct within a scope is structurally incapable of coordinating across it, and from the outside the two failures look identical. So write down the scope your coordination actually covers, as a sentence, and compare it to the scope of the thing going wrong. Ours said "the store." The truthful version was "this box's store," and the gap between those two sentences was a second outage.

2. Everything happens more than once

Most delivery mechanisms you did not write yourself are at-least-once, and the ones advertising stronger guarantees usually mean it only within their own boundary. Webhook providers retry. Queues redeliver. Cron fires twice across a deploy boundary. Your own retry helper is part of this problem, not separate from it. Users double-clicking a button is a different source of the same duplicate.

What it looks like: duplicate rows. A confirmation email sent twice. A charge that happened twice, which is the version people notice.

The subtle case is the one worth internalizing, because it is not really about duplicates. Our end-to-end test harness creates a database over HTTP with a retry on failure. It started failing with 409 already exists on a name containing a fresh random UUID, one that only that run could possibly have generated. The first POST had succeeded, the response was lost in transit, and the retry arrived to find the thing it was trying to create already there.

The request succeeded and reported failure. Those are not opposites, and the caller cannot tell the difference from the outside.

What makes that instructive is where the fix could not go. Not in the generic retry helper, because it cannot know whether the network failed before or after the server acted. Only the caller knows what its operation means, so only the caller can decide that a 409 on a name it just generated is its own earlier success. The fix also has to be narrow, adopting the existing resource only when the name is the one this run generated and a lookup actually resolves it. Anything looser and a real failure gets swallowed as "probably mine."

So: give operations an idempotency key before you need one, make handlers safe to run twice, and distrust any blanket retry wrapper, because a retry is a correctness decision wearing an infrastructure costume.

One more thing moves under you here. Independent paginated reads are not a snapshot, since each page is its own query against a database that keeps changing, and sorting by a stable key does not fix it. A single transaction under snapshot isolation does give you a consistent view, and keyset pagination narrows the window. Where neither is available, count before and after: a change proves something moved, though matching counts prove less than they look like, because an update or a balanced insert and delete is invisible to a count.

3. Two sources of truth will diverge

The moment the same fact is written in two places, they will disagree. Not might. The only variable is how long it takes and whether you find out from a test or a customer.

You do this more often than it sounds. A cache. A denormalized column. A search index. A mirror in another service. A legacy table you kept dual-writing "for now."

What it looks like: a constraint violation from a table your code did not think it was writing.

Ours was a port allocator. It computed the next free port by reading one table, but the unique index enforcing uniqueness lived on a second, legacy table that was also being written. One drifted row was enough: the allocator believed a port was free, handed it out, and the insert died on the other table's constraint. Then it did the same thing for the next signup, deterministically, because nothing about the first failure changed the input to the calculation.

Two habits help. Have exactly one writer for any given fact and derive everything else from it. Where you cannot, make the divergence visible on purpose with a job that compares the two and reports drift, rather than waiting for a constraint to find it for you.

There is a corollary here that makes a lot of verification worthless. A check that reads from the same source as the code it is checking inherits that code's blind spots. We had an importer that wrongly skipped some tables, then verified itself by walking its own list of tables to copy. Every table on the list matched, and it said so. The excluded tables could not appear in the comparison, because the comparison came from the list that excluded them. The full version of that one is its own post.

If your verification consults the same manifest, schema introspection, or type map your code did, it is an echo, not a second opinion.

4. Your automatic recovery is the dangerous part

This is the one that gets skipped, and it is the one that produced our worst incident.

At some point you add self-healing. A handler that retries, resets, reconnects, re-provisions, clears a bad cache. It is good engineering and it will make your system more reliable most of the time. The question nobody asks is what it does when it fires on a false positive.

Ours was a 401 handler. On an authentication failure it dropped the cached API key and got a new one, which sounds obviously correct. Getting a new key meant rotating, and rotating revoked every key on the account. So one instance seeing a single spurious 401 revoked the credentials held by every other in-flight request, each of which then got a 401 of its own and reached for the same cure. Each instance's recovery was every other instance's failure.

The signal was ambiguous and we treated it as proof. A 401 means this request was not authorized. It does not mean the key is dead; it can mean the key is fine and the request went somewhere that could not see it, which is exactly what a routing bug was producing at the time.

Recovery should be the least destructive action that could possibly work, in order: re-read your cache in case someone else already fixed it, ask whether the thing is actually broken, then do the cheap fix, and only then, gated and rate-limited, the expensive one.

There is a second, quieter problem with recovery that works. A failure that heals itself generates no tickets. Ours ran for months: users reloaded, got a working key, and moved on. No threshold crossed, nothing to investigate. The failure mode hid its own symptoms.

What made it visible was adding a circuit breaker, which fixed nothing. It converted an invisible self-healing storm into a bounded, loud one: a lockout a user could report, throttle events in the audit log, alerts in real time. If a class of failure is always fixed by a refresh, you do not know how often it happens. Add the counter before you need it.

Getting ahead of it

Four habits, one per failure above. Write your coordination scope down as a sentence and check it against the state you are trying to keep consistent, because the gap between those two is where these bugs live and it survives your first fix. Assume every message arrives twice and every request may have succeeded before it reported failure. For each automatic recovery, write down what it touches when it fires wrongly, and gate anything both automatic and account-global. And verify from a different source than you act from.

None of this requires a distributed systems course. It requires noticing that you are already running one.

We learned all four of these running Layerbase Cloud, mostly the expensive way. If you are at the point where your app has more than one instance and you would rather not learn them the same way, start with the coordination-scope sentence. It is free and it catches more than it should.