Determinism

A test that sometimes fails is worse than no test at all.

That is not rhetoric. A test with a 1% flake rate in a suite of 2,000 run twenty times a day produces four unexplained failures a day. The team learns to re-run. Once re-running is the reflex, the suite has stopped being a signal — the real failure gets re-run too, and is discovered later and more expensively than if the test had never existed.

The six sources

Nearly every flaky test comes from one of these.

Time. LocalDate.now() in production code makes behaviour depend on when the test runs — and something will be wrong on the 31st, at midnight, on a leap day, or during a DST transition. Inject a clock. Every language has one; use it everywhere, and the test that pins it to 29 February becomes possible rather than impossible.

Randomness. Unseeded random values mean the test exercises a different case each run: green a hundred times, then a failure nobody can reproduce. Inject the source and seed it. Property-based testing is the deliberate exception — it randomises on purpose and, done properly, prints the seed and shrinks the failing case so it can be reproduced.

Ordering. Tests that pass in the order they were written and fail in parallel, or under a different runner. Almost always shared mutable state: a static field, a singleton, a database row, a file. The cure is ownership — each test creates what it needs and cleans up. See test data.

The related trap: asserting on the order of an unordered collection. It is stable until a map implementation changes.

Concurrency. The test that passes on a fast laptop and fails on a loaded CI agent. Usually a sleep standing in for synchronisation. Never sleep — wait for a condition with a timeout. await().atMost(5.seconds).until { order.isConfirmed } is both faster in the normal case and correct in the slow one.

The network. Any test reaching a real external service will fail when that service is slow, rate-limited, deploying, or having a bad Tuesday. Nothing below end-to-end should touch a real external system; use a mock server driven by the contract.

The environment. Locale, timezone, character encoding, filesystem case sensitivity, line endings, available memory. The classic is a test that passes in Europe and fails in America because a date formats differently. Pin the locale and the timezone explicitly in the test setup — one line, and it removes a whole category.

Waiting properly

The single largest source of flakiness in browser and integration suites.

Never sleep(2000). It is too short on a loaded agent — a flake — and too long on every other run, which is a second added to every future execution forever.

Always wait for the condition that actually matters: the element is visible, the request completed, the message arrived, the state settled. Modern browser tooling auto-waits on interactions, and the remaining explicit waits should name what they are waiting for.

The related mistake is waiting for the wrong thing — waiting for a spinner to disappear when the spinner disappears before the data is rendered. Wait for the data.

The policy

Flakiness needs a rule, applied without discussion, because every individual case has a reason to be an exception.

Detect. Track pass rates per test over time. A test that fails and then passes with no code change is flaky, and this can be computed automatically — without the data, flakiness is a matter of opinion and the loudest opinion wins.

Quarantine within a day. A known-flaky test is removed from the blocking suite immediately, and it keeps running so its rate is still measured. Leaving it in the blocking suite trains the team to re-run.

Fix within a sprint, or delete. Quarantine is a holding pattern, not a destination. A quarantine list that only grows is a suite in slow decline, and deleting a test nobody will fix is more honest than pretending it is coverage.

Never re-run to green. Automatic retries are the mechanism by which a suite stops meaning anything. If retries are unavoidable during a migration, record every retry as a flake event and treat the count as a defect backlog.

The uncomfortable part

Flakiness is often not a test problem.

A test that fails once in fifty because of a race condition has found a race condition. The system has it too; the test is just the only thing looking. The reflex — add a wait, add a retry, quarantine it — hides a real defect that will reappear in production as an incident nobody can reproduce.

Before writing off a flake as “test infrastructure”, answer: could the production system exhibit this same interleaving? If yes, the test is the most valuable one in the suite.