Reporting

A test run produces two audiences. The person who broke the build needs to know what broke and why, in seconds. The test manager needs to know whether the suite itself is getting better or worse, over months.

Most reporting serves neither, and produces a wall of green with a number at the bottom.

For the person who broke it

The goal is diagnosis without reproduction. If the standard response to a red build is “let me run it locally”, the reporting has failed and the cost is measured in half-hours.

A failure message that names the cause. The first and cheapest thing. See writing a test.

Evidence captured automatically at the moment of failure, because it cannot be captured afterwards:

Level	Capture
Unit, component	Stack trace, the actual and expected values
Integration	The above, plus the request/response or query, and container logs
End-to-end	Screenshot, DOM snapshot, console log, network log, and a trace or video
Performance	Response-time distribution, error breakdown, saturation metrics, correlated traces

A browser trace is the highest-value item on that list: it lets someone step through what happened without an environment. It is one configuration line in most tooling and it removes most re-runs.

Retained long enough to be useful. Artefacts deleted after 24 hours are useless for anything discovered on Monday.

Attributed. Which commit, which branch, which environment, which run. A failure nobody can attribute becomes everybody’s and therefore nobody’s.

For the test manager

One run says something about one build. Only the trend says anything about the suite, and these are the numbers that decide where the next improvement goes.

Pass rate per test over time. The flakiness detector. A test that fails and then passes with no code change is flaky, and this is computable rather than debatable. See determinism.

Duration per test and per suite. Suites do not become slow suddenly; they gain four seconds a week for a year. The trend catches it while it is still one test’s fault.

Failure attribution. How many failures were real defects, flakes, environment problems, or test defects. This ratio is the single most informative number about a suite’s health, and almost nobody collects it because it needs a human to classify — a dropdown at triage is enough.

Escaped defects by level. For each defect that reached production, where it should have been caught. The distribution says which part of the strategy is not working.

That is what the campaign results service on the campaigns page is for. It is not deployed yet, and the page says so.

Reporting a defect

A defect report is a message to someone who was not there. Six fields, and the first two are the ones usually missing.

1. What you expected, and where that expectation comes from — an acceptance criterion, a contract, a documented behaviour. Without this it is an opinion.
2. What happened, precisely. Not “it broke”.
3. How to reproduce it: the exact steps, the data, the account, the environment, the build.
4. Evidence: the recording, the request, the correlation id, the log excerpt.
5. Impact: who is affected, how often, and whether there is a workaround. This is what drives priority, and it is the tester’s judgement to offer.
6. When it started, if known. A build number turns a search into a diff.

A report missing 1 or 5 generates a round trip, and the round trip costs more than writing the fields did.

What not to build

A dashboard nobody opens. If the numbers are not in the conversation where decisions are made — the release go/no-go, the retrospective — the dashboard is a hobby.

A green wall. A report showing 4,000 passing tests communicates nothing. Report failures, new failures, flakes, and what changed since the last run.

A percentage as the headline. See coverage. The headline should be a decision: safe to release, with these three known failures and this uncovered risk.