The bug that only happened half the time

The worst bugs are not the ones that break everything. Those get found in minutes, because everything is broken. The worst ones work most of the time.

We had a product where uploading a file occasionally failed. Not always — perhaps half the time. Retry and it usually worked. Every ingredient of a bug that never gets properly investigated was present: it was intermittent, it recovered on retry, and it looked exactly like a flaky connection.

It was not a flaky connection. Roughly half of that product's API traffic had been served by the wrong environment for weeks.

How two environments ended up sharing a name

The setup was ordinary. A production environment and a staging environment, defined with the same compose file, each with a service called api and a service called web. Staging had no database of its own, because giving it one would have meant maintaining a second copy of the data. Instead, the staging api was attached to production's network so it could reach the production database.

That single decision is the whole incident.

Container orchestrators give every service a DNS name, and that name is registered on every network the service joins. Two containers were now called api on the same network. When production's web container looked up api, DNS returned two addresses and it used them both.

There was no error anywhere. Both containers were healthy. Both answered correctly. Both were running the same application. Health checks passed on both, because both were genuinely healthy.

Why it stayed hidden

The two environments shared a database, so any request that only touched data behaved identically no matter which container served it. Reads were right. Writes were right. Nothing in the data could reveal the split.

Files were a different matter. The two environments had separate file storage. So an upload handled by the staging container wrote a file into staging's storage and a row into the shared database. Production could see the row and could not find the file.

That is the intermittent 404. The row said the file existed. It did, just not anywhere production could reach.

What made it hard, stated plainly

A failure that happens every time is a bug report. A failure that happens half the time is an argument. Someone reproduces it, someone else does not, and the difference between them is a coin flip nobody knows is being tossed. It gets filed under network flakiness and stops being investigated.

The lesson we took is about how you verify a fix, not how you find the cause:

A single successful request proves nothing about a system that fails half the time. It proves you won one coin toss.

Now, when we verify anything in this class, we repeat the request a dozen times and require every attempt to succeed. That sounds obvious written down. It is not what anyone does by reflex — the reflex is to try it once, see a 200, and move on.

The fix, and the rule behind it

The repair was to stop the two environments sharing a namespace. The database got a second network carrying nothing but database traffic, and staging was moved onto that instead of onto production's general network. Staging still reaches the database. It is no longer resident on a network where its own service names can collide with production's.

The general rule we now apply:

• Environments may share a resource. They must never share a namespace.
• If a service must reach across an environment boundary, give it a dedicated path that carries only that one thing.
• When something is reachable by name, ask what else answers to that name.

The part that still bothers us

Every monitoring signal was green throughout. Containers were up. Health endpoints returned success. Deploys reported clean. The application logged no errors, because from each container's point of view nothing had gone wrong.

We only found it because someone looked at what the system actually contained rather than what its status endpoints reported. That is the durable lesson, and it is not a comfortable one: a health check tells you a process is running, not that it is the right process serving the right data.

Since then, checks on that product verify by content. Not "does the service respond" but "does this specific known file come back, with the right number of bytes, on every environment, every time".