Test data

Test data decides more about a suite than any tooling choice. It sets the ceiling on what can be tested, the floor on how fast it runs, and the probability that a failure is about the change rather than about the fixture.

It is also where the most expensive mistakes live, in both directions: data too clean to find real defects, and real data in places it must not be.

Four strategies

Strategy	What it is	Good for	Fails at
Synthesised	Built by the test, in the test	Unit, component, integration	Realistic scale and weirdness
Seeded	A known fixture set loaded into an environment	Demos, preview environments	Coupling — every test depends on it
Subsetted	A referentially-consistent slice of production	Staging, E2E	Extraction complexity, and it is still real data
Masked	Production, with identifying values replaced	Staging, performance, security	Masking that breaks meaning or does not actually anonymise

Synthesised: the default

Each test creates exactly the data it needs and nothing else.

This is the single most valuable habit in the whole section, and it is what makes tests independent, parallelisable and readable. A test that reads

given a customer with 3 previous orders and a gold membership

states its own preconditions. A test that reads given customer 4711 requires the reader to open a fixture file and hope nobody changed row 4711 this week.

Use builders with sensible defaults, so a test only names what it cares about:

aCustomer().withOrders(3).gold().build()

The alternative — a shared fixture set — creates the failure mode where adding a test changes an unrelated one, because both read the same row and one of them mutated it.

Where synthesised stops

Three places, and knowing them is what keeps the strategy honest.

Scale. A query that is fine over 5,000 generated rows can be catastrophic over 5 million real ones. Generating 5 million rows is possible; generating 5 million rows with a realistic distribution — the customer with 40,000 orders next to the 90% with one — is the part that gets skipped, and it is the part that matters. Performance work needs production-scale data. See performance.

Weirdness. Production data has history: records created under a schema that no longer exists, a nullable column that was mandatory for two years, an encoding decision from 2011, a name that breaks a form. No generator produces these, because nobody thought of them — which is exactly why they cause defects.

Referential depth. Some journeys need a graph of related records that is tedious to synthesise and trivial to subset.

Masking, carefully

Masking replaces identifying values in a production copy. Two requirements pull against each other and both are non-negotiable:

It must actually anonymise. Replacing names while keeping date of birth, postcode and purchase history re-identifies most people trivially. Consistent pseudonymisation, generalisation of quasi-identifiers, and someone who has thought about linkage attacks.

It must preserve meaning. Masking that scrambles a postcode into an invalid one breaks every validation-dependent path. Masking that replaces all salaries with the same value destroys the distribution the performance test needed. Format and distribution have to survive.

Rules that follow, and there are no exceptions to them:

• Never in a non-production environment unmasked. Not “temporarily to debug”.
• Masking runs where the data already is, not by copying it out first.
• Reversibility is a breach. If the mapping is kept, the data is not anonymised — it is encrypted, which is a different property with different obligations.
• The regulatory obligations (GDPR and equivalents) apply to test environments exactly as to production. A staging database with real customer data is a production system that nobody is treating as one.

Managing it over time

Data lives with the test. Created before, removed after — or the whole environment is thrown away, which is better. See environments.

Never depend on data you did not create. The one rule that prevents most cross-test flakiness.

Refresh masked copies on a schedule, and version the masking rules with the schema. A new column arrives; the masking has to know about it before the copy does.

Cleanup is part of the test, not a nightly job. A nightly cleanup means a failing test leaves state behind for the rest of the day.

The failure to recognise

Data too clean to be useful. Every generated customer has a two-part Latin name, a valid address, one order and no history. The suite is green, and every production defect involves a customer who does not look like that.

The countermeasure is cheap: put the awkward cases into the generators deliberately. A name with an apostrophe, a name in a non-Latin script, an address with no postcode, a customer with zero orders and one with thousands, a record that predates the current schema. They cost an hour to add and they find defects for years.