Campaign definition DSL

The observation: a campaign is a decision, and decisions with owners deserve review. Today that decision is made by clicking through Jira, and the result cannot be diffed, templated, or read alongside the code it gates.

The hypothesis: the campaign definition is small, structured and repetitive enough to be declared as a file in the repository and applied to Xray — the same relationship infrastructure-as-code has with a cloud console.

This is one of a pair. This page is the declaration; the Xray CLI is what would apply it.

The problem

From campaigns: every campaign has a scope, an owner, a cadence and an exit condition. Those four are the whole reason a campaign is not a suite.

Where do they live? In a Jira Test Plan, filled in by hand, once per release. Which means:

• No review. Nobody can look at a pull request and say “the release scope dropped the payment journey this time”. The change is invisible because there is no diff.
• No template. Every release recreates the same structure by hand, so it is recreated slightly differently each time, and the drift is discovered by noticing something missing after the fact.
• No versioning against the code. The campaign that gated release 4.2 cannot be recovered as it was, next to the code it gated.
• Exit conditions live in someone’s memory. “No open criticals” is agreed in a meeting and re-agreed, differently, next quarter.
• The risk decisions are not attached. The risk model says what is deliberately not tested; the campaign says what is run. They are two halves of one decision and they sit in different tools.

The failure this produces is quiet, which is what makes it worth attention: a release regression campaign whose scope has been shrinking for four releases looks identical, from the outside, to one that has not.

The shape

campaign: release-regression
description: Scope for a release candidate, gating go/no-go

owner: release-manager          # a role, resolved to a person per run
cadence: per release candidate

entry:
  - the candidate is deployed to staging
  - the continuous campaign is green on the candidate commit
  - the risk register for this release is signed

scope:
  # Selection by label, not enumeration. An enumerated list of 400 tests is a
  # list that rots; a label is a rule the suite maintains itself.
  - suite: e2e
    tags: [critical-journey]
  - suite: contract
    tags: [external-consumer]
  - manual: [exploratory-charter-basket, exploratory-charter-payment]

environments: [staging]

exit:
  - no open critical or major defect on the scope
  - every failing test triaged with an owner
  - uncovered risks listed in the release notes

# What is deliberately not run, and why. The half that is never written down.
excluded:
  - scope: admin-csv-export
    reason: low likelihood, low impact, internal workaround exists
    accepted_by: product-owner

Twenty-five lines that a release manager would read and correct — the same test the performance DSL sets for its model file.

The excluded block is the part that does not exist in any tool today, and it is arguably the most valuable: it puts the risk decision in the same file as the scope, where it is reviewed at the same moment, by the same people.

What applying it would mean

The file declares the intent. Applying it creates or updates the Test Plan in Xray, resolves the scope selectors against the actual tests, and opens a Test Execution per environment. A dry run prints the diff:

+ 3 tests added to scope (tag: critical-journey)
- 1 test removed from scope (deleted: basket_legacy_flow)
~ exit condition changed: "no open criticals" → "no open critical or major"

That diff, in a pull request, is the entire point of the exercise.

What would make this a bad idea

It may be a very small problem. Most teams run three or four campaign types, defined once and edited rarely. A language for a file that changes twice a year is a large answer to a small question.

Xray already has templates. A Test Plan can be cloned, and a saved filter can define scope dynamically. If cloning plus filters gets most of the way, the remaining gap is reviewability alone — real, but perhaps not worth a toolchain.

Two sources of truth. The moment someone edits the Test Plan in Jira, the file and the tool disagree. Every infrastructure-as-code tool has this problem and solves it with drift detection and a policy that the file wins — which is a policy about how people work, not about software, and it is the hard part.

It assumes Xray. A DSL coupled to one commercial tool’s data model inherits that tool’s lifespan. Keeping the file tool-neutral and the applier tool-specific is the obvious answer and it makes the mapping harder.

Status and next step

Investigation. Same three possible outcomes as the rest of this section: a thin applier over Xray, a convention with no tooling, or nothing.

The cheap first step is the same one as for the performance DSL: take the last four releases’ Test Plans and express each as a file by hand. If they are nearly identical, the value is templating and cloning already covers it. If they differ in ways nobody noticed at the time, that is the evidence this is worth building — and it is a finding regardless of what gets built.