In production

The principle — why this is an option and what its entry conditions are — is in testing in production. This page is the mechanics: what runs, where it runs, and what has to be in place around it.

Nothing here should be adopted before the four entry conditions hold: observability, fast rollback, bounded blast radius, and consent where it applies.

Synthetic journeys

Scripted user journeys running continuously against production from outside the network, as a real user would.

What they are. The same critical journeys as the end-to-end suite, often the same code, pointed at production and run every few minutes from several locations.

What they buy. An alert that means something. “Checkout has been failing for four minutes” is worth a page; “CPU is at 80%” is not. They also cover the periods with no organic traffic, when a broken deploy would otherwise sit undetected until morning.

The traps.

• Test accounts must be marked as such, and excluded from analytics, billing, fraud scoring and any model trained on behaviour. A synthetic journey contaminating a conversion metric is a self-inflicted data quality incident.
• Side effects must be reversible or inert. A synthetic checkout either uses a provider test mode or is cancelled immediately. Design the journey with the cleanup as part of it.
• Never seed real customer data. The synthetic user is a real record in the production database; treat it accordingly.

Canary releases

The new version takes a small share of traffic; its behaviour is compared against the current one.

Compare what matters, in this order. Error rate first, latency second, saturation third, and — the one that is usually missing — business metrics. A canary with identical error rates and 30% fewer completed checkouts is a failure that no technical metric reports.

Decide automatically, on a rule agreed in advance. This is the part that distinguishes a canary from a deploy someone is watching. Write the rule before the deploy: abort if the 5xx rate exceeds baseline by 0.5 percentage points, or p95 latency by 20%, over a 10-minute window with at least 1,000 requests. A human watching a dashboard at 2am will always find a reason the numbers are fine.

Give it enough traffic to be significant. 1% of a low-traffic service for five minutes is noise. Either raise the share or lengthen the window until the comparison can actually distinguish a difference.

Include the rollback in the rehearsal. An automated abort that has never fired is an automated abort nobody knows the behaviour of.

Progressive rollout

The same comparison, extended over hours or days by percentage or by cohort.

Catches what only appears with volume or over time: a cache that degrades as it fills, a memory leak, a query that slows as a table grows, a scheduled job that only runs at midnight. It also limits exposure — the difference between a defect affecting 2% of users for ten minutes and everybody for an hour.

Cohorts before percentages, where possible. Internal users, then a friendly segment, then a geography, then everyone. Each step has people who will report what a metric cannot.

Dark launching

The new code path runs on real production traffic and its result is discarded or compared against the old path, without affecting the response.

This is the most underused technique on the page and the one most worth its setup cost. It is how a rewrite is validated against reality before anything depends on it: run both implementations, return the old one’s answer, log where they differ. After a week of real traffic, the diff report is a list of every case the new implementation gets wrong — including all the cases nobody thought to test.

Watch the cost. Two implementations means twice the work per request, and any side effect in the shadow path must be suppressed. A shadow write is not a dark launch, it is a bug.

Controlled experiments

A/B tests answer whether the change was good, not whether it worked. They are a different discipline with their own statistics — sample size before starting, a metric agreed in advance, and no peeking until the run completes.

The engineering prerequisites are the same as everything above: a flag, a stable assignment, and clean event data. The ethical ones are not: experiments on people have consent and harm constraints that a feature flag does not remove.

Feature flags, and their debt

Every technique above needs flags, and flags are the mechanism by which “testing in production” is survivable — the ability to turn something off in seconds without a deploy.

They also accumulate. A codebase with 300 flags has an untestable combinatorial state space, and each stale flag is a branch nobody exercises.

Every flag gets an owner and an expiry when it is created. A release flag that has been fully on for a month is deleted, along with its dead branch. This is not tidiness — it is the difference between flags being a testing capability and flags being the reason nobody can reason about the system.

What has to exist around all of it

Telemetry that distinguishes versions and cohorts. Without a version label on every metric and trace, a canary comparison is not possible.

Correlated logs and traces, so a canary anomaly can be followed to a span rather than admired on a chart.

An abort that anyone on call can trigger, tested, documented, and faster than the conversation about whether to use it.

A record of what ran. Which experiment, which cohort, which window. Six weeks later, someone will ask why the numbers looked strange that Tuesday.