Resilience

The question: what happens when a dependency misbehaves?

Every other typology assumes the things around the system work. This one assumes they do not, which is the assumption production actually holds. The distributed computing fallacies are not a list of mistakes people make once — they are a list of things every system quietly assumes until the day it is proven otherwise, at 3am, in the way that costs most.

The failure modes worth testing

Ordered by how often they are the cause of a real incident, which is not the order teams test them in.

Slow, not down. The dominant one. A dependency that returns in 30 seconds instead of 30 milliseconds is far more dangerous than one that refuses the connection: the connection pool fills, threads block, the queue grows, and the service becomes unavailable for reasons unrelated to its own health. Every call that leaves the process needs a timeout, and the test is that the timeout is shorter than the caller’s patience.

Down. Connection refused. Cheap to handle and cheap to test — and consequently the only one many teams test.

Intermittent. One request in twenty fails. This is where retry logic is proven or found to be a load amplifier: naive immediate retries turn a 5% failure into a 15% load increase exactly when the dependency is struggling. Exponential backoff with jitter, and a cap.

Wrong. Returns 200 with a malformed body, a field missing, a null where the schema promises a value. The case nobody writes, and the one that produces the stack trace nobody can read.

Partitioned. The network is up in one direction. The caller waits; the callee already committed. This is where idempotency is proven, and where at-least-once delivery stops being a footnote.

The patterns being verified

The test is not “does the failure happen” — it is does our handling of the failure behave as designed. Each pattern has a specific, checkable property.

Pattern	What the test asserts
Timeout	The call is abandoned within the budget, and the budget is smaller than the caller’s
Retry with backoff	Retries are bounded, spaced, jittered, and only on retryable errors
Circuit breaker	It opens under sustained failure, stops calling, and half-opens to probe recovery
Bulkhead	One slow dependency exhausts its own pool, not every thread in the process
Fallback	A degraded but useful answer — the cached price, the empty list, the “try later”
Idempotency	The same request applied twice has the effect of once

The most valuable of these to test is the fallback, because it is the one whose absence is invisible: a system without a fallback looks identical to one with an untested fallback, right up until the dependency fails.

Where it runs

Pre-production, faults injected deliberately. Two levels, and the cheap one is underused.

At integration level, against a mock server configured to be slow, to fail intermittently, to return garbage. This costs almost nothing — WireMock will delay a response or return a 503 on demand — and it catches most timeout and retry defects. It belongs in the same suite as the integration tests and should run on every merge.

At environment level, against a deployed system with a real dependency degraded — a service scaled to zero, latency injected by the mesh, a network rule dropping packets. Slower, more setup, and the only place where the emergent behaviour appears: the cascade, the thundering herd on recovery, the queue that never drains.

Chaos experiments in production are the same idea with the four entry conditions from testing in production. Out of scope until the two levels above are routine — injecting faults into a system whose failure handling has never been tested is not an experiment, it is an outage with a hypothesis attached.

Running one

Like an experiment, because it is one.

1. State the hypothesis. “If the pricing service takes 5 seconds, checkout still completes within 2 seconds using the cached price, and no error reaches the user.”
2. Define the blast radius and the abort condition before starting.
3. Inject one fault, and only one. Two faults produce a result nobody can attribute.
4. Observe against the hypothesis — including whether the alert fired, which is half of what is being tested.
5. Write down what actually happened. The interesting outcome is usually not the one predicted.

Step 4’s clause matters: a resilience test that passes while no alert fires has found a defect in the observability, which is a prerequisite for everything else.

Tooling

Fault injection at integration level with WireMock or Toxiproxy; at environment level with the service mesh’s fault injection, or a chaos tool if one is already run by the platform. See performance tooling for the load side, since a resilience test is usually run under load — a circuit breaker’s behaviour with one request in flight tells you very little.