Management and reporting

Two tools that sit either side of a campaign rather than inside it. Xray defines what a campaign is and what it covers; Allure reports what happened when it ran.

Neither executes anything. That is the property to hold on to: the tests stay plain code in the repository, run by the runners in the other tooling pages, and these two consume the result. A test management tool that also stores the test logic recreates exactly the artefact the tooling overview rules out — a suite nobody can review in a pull request.

Xray: defining campaigns in Jira

Xray is a test management application that lives inside Jira as native issue types. That is its whole argument: a campaign is a decision with a scope, an owner and an exit condition — see campaigns — and in most organisations decisions with owners already live in Jira, next to the requirements they relate to.

The issue types, and what each maps to

Xray type	What it is	In our vocabulary
Test	One test case — manual steps, a Gherkin scenario, or a pointer to an automated test	A single check
Precondition	Reusable setup shared by several tests	The arrange block, named
Test Set	A flat, reusable grouping of tests	A suite
Test Plan	The scope of a campaign, tracked over time	The campaign
Test Execution	One run of a scope in one environment	One execution of the campaign
Test Run	The result of one test inside one execution	One result

The mapping that matters: Test Plan is the campaign, Test Execution is a run of it. A release regression campaign is one Test Plan that accumulates several Test Executions — release candidate 1, the fix, the re-run — and the Plan’s progress is what the go/no-go conversation reads. Conflating the two is the same mistake as conflating a campaign with a suite.

What it does that a runner cannot

Requirement coverage and traceability. A Test is linked to the story or requirement it verifies, so coverage is computed against requirements rather than against lines. “Which acceptance criteria have no test” is a question no coverage tool can answer and this one can — and it is a far better question than the one coverage warns about.

Defect linking, both directions. A failed Test Run creates or links a defect; the defect shows which test found it; the requirement shows both. Six weeks later, “what broke when we changed this” is a query rather than an archaeology exercise.

Test environments. An execution records which environment it ran in, so the same scope can be tracked separately across staging, the performance environment and production smoke — see environments.

Gherkin storage with export and import. Scenarios can live as Jira issues and be exported to .feature files for the runner, or imported from the repository. Both directions exist, and choosing one is mandatory — see the traps below.

Result import from the runners. JUnit XML, TestNG, NUnit, xUnit, Cucumber JSON, Robot Framework, Behave, and Xray’s own JSON format. This is the join point: the pipeline runs the tests, publishes the report, and Xray turns it into Test Runs against the Plan.

APIs and CI integration. A REST API on both Server/Data Center and Cloud, a GraphQL API on Cloud, plus plugins for Jenkins, Azure DevOps and GitHub Actions. Capabilities and endpoints differ between the Cloud and Server/DC editions more than the marketing suggests — check against the edition in use before designing anything around them.

The traps

Do not let it become the source of truth for automated test logic. Tests authored in Jira and exported to the repository are tests that cannot be refactored with the code, reviewed in a pull request, or run offline. Pick one direction and enforce it: the repository is the source of truth, Xray receives results and holds the manual and exploratory work.

Do not create a Test issue per automated test. A suite of 4,000 unit tests does not belong in Jira as 4,000 issues; it belongs as a result summary. Xray earns its place at the level where a human decides something — the campaign scope, the manual and exploratory work, the requirement coverage — and becomes noise below it.

The licence and the administration are real costs. So is the coupling: a campaign defined only inside Jira cannot be reviewed as a diff, which is precisely the gap the campaign definition DSL research is about, and the reason the Xray CLI research sits next to it.

Allure: the run report

Allure Report is an open-source reporting layer. Adapters for the runner write JSON results during the run; a generator turns a results directory into a static HTML report:

allure generate allure-results --clean -o allure-report

Adapters exist for essentially every runner in the other tooling pages — JUnit 5, TestNG, pytest, Vitest and Jest, Cucumber, Playwright, NUnit, Behave and more — so one report format spans a polyglot estate. That is the main reason to adopt it: a team with a JVM service, a TypeScript front end and a Python job gets one report rather than three.

What it gives that a runner’s own output does not

Steps and attachments. Screenshots, DOM snapshots, request and response bodies, logs and videos attached to the step that produced them. This is the reporting requirement — diagnosis without reproduction — made concrete.

History and trends. Pass rate and duration per test across runs, which is the only view that says anything about the suite rather than about one build. It is what makes flakiness measurable rather than anecdotal — see determinism.

Categories. Rules that classify failures automatically — product defect, test defect, environment problem, known issue — by matching the message or stack trace. This is the failure-attribution ratio from reporting, and having it computed rather than tallied by hand is what makes anyone actually look at it.

Retries and flaky marking. A test that passed on retry is shown as such rather than as green, which keeps the re-run reflex visible instead of hidden.

Behaviour grouping. Epic, feature and story labels group results the way the product is described rather than the way the code is arranged — useful when the audience for the report is not the team that wrote it.

The traps

History needs to be carried forward, and it is the thing everyone misses. The trend graphs are built from a history directory inside the previous report. If the pipeline generates each report from a clean workspace, every run shows a single data point and the most valuable feature silently does nothing. Copy allure-report/history into allure-results/history before generating, or publish through something that does it for you.

It reports, it does not decide. The build’s pass or fail comes from the runner’s exit code. A green Allure report over a failed run is a configuration error waiting to be believed.

Attachments grow fast. Videos and traces on every test produce reports of hundreds of megabytes. Attach on failure, and on first retry — not on success.

Allure TestOps is a different, commercial product. Server-side history, long-term storage and its own management features that overlap with Xray. Worth knowing it exists; adopting both it and Xray means owning two answers to the same question.

How the two fit together

The pipeline runs the tests. Allure turns the results into a report a human can diagnose from. Xray turns the same results into Test Runs against a Test Plan, so the campaign’s progress and its requirement coverage are visible to people who will never open a build log.

runner → JUnit XML / Cucumber JSON ──┬──→ Allure  → report artefact, trends
                                     └──→ Xray    → Test Runs on the Test Plan

One execution, two consumers, no duplicated source of truth. The campaign results service the portal links to is where the long-lived side of that is meant to live; it is not deployed yet, and the page says so.