Testing in production

Testing in production has a bad name because it is usually said sarcastically, about a team that skipped the other levels. That is not this.

It is a deliberate option, taken for properties that genuinely cannot be observed anywhere else, and it has entry conditions that most teams do not meet. Both halves of that sentence matter. Denying the first buys a staging environment that lies convincingly; ignoring the second is how “we test in production” becomes a euphemism for not testing.

What only production can tell you

Four things, and no pre-production environment reproduces any of them convincingly.

Real traffic shape. Not the volume — the shape. The correlations, the bursts, the long-tail endpoints nobody remembered, the client that retries aggressively at 3am. Synthetic load reproduces what you thought to model, which is the subset you already understood.

Real data. Production data is older, weirder and more skewed than any generated set: the account created in 2009 under a schema that no longer exists, the name with the character your form never anticipated, the customer with 40,000 orders. See test data for how far a copy gets you, and where it stops.

Real scale and real infrastructure. Cache hit rates, connection-pool saturation, cross-zone latency, noisy neighbours, and the behaviour of the actual third parties rather than their sandboxes. Halving the environment does not halve these; it changes their character.

Real users. Whether people can find the feature, and whether it changes what they do. No amount of testing answers that question — only a controlled experiment does.

The entry conditions

Do not adopt any technique below until all four hold. This list is the difference between a discipline and an excuse.

Observability. You can tell, from telemetry alone and within minutes, that something is wrong and roughly where. If you find out from users, you are not testing in production — production is testing you.

Rollback or disablement, fast. A feature flag that turns it off, or a deployment that reverts, inside minutes and without a meeting. Anything you cannot undo cheaply must not be exercised this way.

Bounded blast radius. A mechanism to expose a change to a small, chosen fraction — canary, percentage rollout, cohort — so that the impact of being wrong is proportional to the confidence, not to the deploy.

Consent and legality where it applies. Experiments on humans have ethical and legal constraints. Real user data crossing into a test path has regulatory ones. Neither is negotiable because the technique is convenient.

And, honestly: the other levels have to be in place. Testing in production covers what earlier levels cannot, not what they should have.

The techniques

Ordered by how much they demand of the four conditions above.

Synthetic monitoring. Scripted journeys running continuously against production, as a real user, from outside. The cheapest entry, and it doubles as the alert that means something — “checkout has been failing for four minutes” is worth waking someone for, in a way that “CPU is at 80%” is not.

Canary releases. The new version takes a small share of traffic while its error rate, latency and business metrics are compared against the old one. The decision rule is agreed in advance and evaluated automatically, because a human watching a dashboard at 2am will always find a reason the numbers are fine.

Progressive rollout. The same idea extended over hours or days, by percentage or cohort. Slower, and it catches what only shows up with volume or after a cache warms.

Dark launching. The new path runs on real traffic and its output is discarded, or compared against the old path, without affecting the response. This is how a rewrite gets validated against reality before anyone depends on it, and it is the technique most worth its setup cost.

Controlled experiments (A/B). The only way to answer whether the change was good rather than whether it worked. A different discipline with its own statistics, and the one place where a “test” in production is not about defects at all.

Chaos experiments. Deliberate fault injection against the live system, with a hypothesis stated first. The most demanding of the four conditions, and out of scope until resilience testing is routine in pre-production.

What it does not replace

Everything cheap and deterministic. A unit test costs milliseconds and fails before a merge; the same defect found by a canary costs a rollback, an incident channel and some fraction of real users’ afternoon.

The rule from the strategy — check each property at the cheapest level that can check it honestly — is unchanged. Production is simply the level for the properties where “honestly” turns out to mean “with real traffic on it”.

The mechanics — environments, rollout controls, and what runs where — are in the production topology.