Performance

The question: does it still hold at the load we promised?

The italics are on promised. A performance test without a stated objective produces a number, and a number without a threshold produces a discussion. The first artefact is not a script — it is the SLO.

Start from the objective

An objective has four parts, and dropping any one makes it unusable:

The search endpoint responds in under 300 ms at p95 under 200 concurrent users, with an error rate below 0.1%.

Operation, threshold with a percentile, load, and error budget. Compare with “the system should be fast”, which nobody can fail, or “the average response is 200 ms”, which is compatible with one user in twenty waiting nine seconds.

Use percentiles, and know which one you are quoting. The mean hides the tail, and the tail is where users are. p95 is the usual working threshold; p99 is where the pathological cases live, and on a page that makes twenty requests it is the one most users meet at least once.

The four shapes

Different questions, different runs. Conflating them is why “we did a load test” means so little.

Load — the expected traffic, sustained. Answers: does it meet the SLO under normal conditions? Run per release. This is the one that gates.

Stress — increase until something breaks. Answers: where is the limit, and what fails first — and does it fail gracefully, shedding load and returning 503, or catastrophically, timing out everything at once? The failure mode matters more than the number.

Soak — expected load, for hours. Answers: what degrades with time? Memory leaks, connection-pool exhaustion, unbounded caches, log volume, disk. Nothing else finds these, and a four-hour soak before a major release is cheap insurance.

Spike — sudden 10× for a short burst. Answers: does it survive the marketing email, the flash sale, the retry storm after an outage? Tests autoscaling reaction time and queue behaviour, which are usually configured optimistically.

Modelling the workload

The most common mistake is not the tooling — it is testing one endpoint at a time at a made-up rate.

Model journeys, not endpoints. Real users browse, search, view, add, check out, with think time in between. A flat hammer on /search produces a cache hit rate no real traffic ever achieves.

Use the real mix. Take it from production access logs. The proportions are never what anyone guesses.

Include think time. Without it, 200 virtual users generate the load of 20,000 real ones and you are testing a scenario that will never occur.

Use realistic data. A search for the same term 10,000 times measures the cache. See test data.

Include the cold start. Discard the warm-up explicitly, or report it explicitly. Do not average it in silently.

Where it runs

Production-like and isolated. Two hard requirements and they are in tension with cost.

Production-like: same instance classes, same replica count, same database size and shape, same network topology. A test on half the hardware does not produce half the numbers — cache hit rates, connection pooling and GC behaviour change character rather than scale.

Isolated: nothing else on the hardware, and no other traffic. Otherwise you are measuring the neighbours.

The pragmatic compromise most teams land on: a scaled-down environment for trend — is this release slower than the last? — plus a full-scale run before major releases for absolute numbers. Say which one you are quoting, always.

Reading the result

Compare against the previous release, not only against the threshold. A service at 280 ms against a 300 ms SLO is passing, and if it was 140 ms last month it is also the most important finding of the release.

Check the error rate first. A run with 4% errors has fast response times because failing is quick.

Look at saturation, not just latency. CPU, memory, connection pools, thread pools, queue depth. Latency tells you it broke; saturation tells you what to fix.

Correlate with traces. A number says the p99 is 4 seconds. A trace says which span it was spent in. Without the second, the result is a fact rather than an action.

The scripts rot

Performance scripts decay faster than any other test: they encode endpoints, payloads, auth flows and data assumptions, and they are usually written in a tool nobody uses daily, reviewed by nobody, and touched twice a year.

That is the problem the research section is about — whether a domain-specific language for workload description would make these artefacts readable, reviewable and durable rather than disposable. It is an open investigation, not a recommendation.

Tooling

See performance tooling for k6, Gatling and JMeter, and what each is actually good at.