CLI for the Xray API

The observation: every interaction with Xray from a pipeline today goes through either a CI-specific plugin or a hand-rolled curl invocation, and both age badly.

The hypothesis: the same handful of operations are being reimplemented in every repository, and they belong in one small, testable command-line tool.

This is the second half of a pair. The campaign definition DSL declares the campaign; this is what would apply it.

Why not the existing options

The CI plugins. Xray ships integrations for Jenkins, Azure DevOps and GitHub Actions. They work, and they bind the interaction to one CI product — so moving CI means rewriting the Xray integration, and running the same operation locally is impossible. A tool that only exists inside a pipeline cannot be dry-run before it is committed, which is the property that makes people brave enough to change it.

Raw curl. What most teams actually do. It works until it does not: the authentication dance is duplicated in six repositories, nobody handles a 429, the JSON is assembled with jq in a shell heredoc, and errors surface as an HTTP status in a log nobody reads. The failure mode is silent — a results import that returns 400 and a pipeline that carries on green.

The API differences are the real trap. Xray Cloud and Xray Server/Data Center have genuinely different APIs: different base paths, different authentication (an API key exchanged for a token on Cloud, Jira credentials or a personal access token on Server/DC), and a GraphQL API that exists on Cloud only. Code written against one does not run against the other, and organisations migrate. That divergence is exactly the kind of thing worth hiding behind one interface — and also the reason the tool is more work than it first appears.

The operations

Small surface. These five cover most of what a pipeline and a test manager actually do.

xray campaign apply campaign.yaml --dry-run
    Create or update the Test Plan from the declaration, resolve scope
    selectors, print the diff. See the campaign DSL research.

xray results import --format junit --path build/test-results \
                    --plan REL-4.2 --environment staging
    Publish a run. Accepts what the runners already emit — JUnit XML, Cucumber
    JSON, TestNG, Behave, Robot — and creates the Test Execution.

xray gherkin export --plan REL-4.2 --out features/
    Pull the scenarios that live in Jira into feature files for the runner.

xray coverage --requirement PROJ-1234
    Which acceptance criteria have tests, which have none, and what the last
    result was. The question no coverage tool can answer.

xray defects --execution EXEC-88 --format json
    What failed, with the linked defects, for a release report.

Two design decisions carry most of the value:

--dry-run on everything that writes. Printing what would change, and exiting non-zero if it would change anything, makes the tool usable as a check rather than only as an action.

Exit codes that mean something. A results import that partially failed must not exit 0. The current curl approach’s most common defect is a pipeline that reports success over an import that silently rejected half the file.

What it would need to get right

Idempotency. Applying the same campaign twice produces one Test Plan, not two. Re-importing the same results updates rather than duplicating. This is the hardest requirement and the one that decides whether the tool can be run safely from a pipeline that retries.

Authentication that suits a pipeline. Environment variables, no interactive prompt, no token written to disk, and a clear failure when a credential has expired rather than a 401 buried in a stack trace.

Rate limits and retries. Jira Cloud rate-limits, and a large results import is many calls. Backoff with jitter, and a bounded retry — the same discipline this hub asks of any client.

One interface over both editions. Detect or configure the edition, and keep the divergence in one adapter layer rather than in every command.

Useful errors. “400 Bad Request” is not a diagnosis. “Test PROJ-4711 is not in the scope of plan REL-4.2” is.

What would make this a bad idea

Someone may already have built it. There are community CLIs and API wrappers of varying maintenance. Checking that properly is the first task, and finding a maintained one is the cheapest possible outcome.

It is a client for a commercial API that changes. Every Xray release can move something. A wrapper is a maintenance commitment proportional to somebody else’s release cadence, taken on by a team whose job is not building CLI tools.

The surface may creep. Five commands is a tool. Twenty-five is a product with a documentation site, and the boundary between them is one reasonable feature request at a time.

It only matters if Xray does. Coupled to one vendor by definition. If the organisation moves test management, the tool is deleted rather than ported — which is an argument for keeping it thin, not for not building it.

Status and next step

Investigation, and the most tractable of the three in this section — it is a few hundred lines of a well-understood shape, not a language.

Two steps, in order. Survey what already exists, including the official plugins’ source, since they are doing these operations already. Then implement results import alone and use it for one release. If that single command removes the curl block from three repositories and catches one silently-failed import, the rest is justified; if it does not, the conclusion is that the plugins were fine and this page ends with a no.