ADR 0012: Testing Strategy — Automated, Contract, and Load

Ratification

Adopted before ADR 0018. There was no separate ratification process. Git history for this file on main is the record.

• Discussion Issue: not recorded (before ADR 0018)
• Merge PR: see git history for this file
• Accepted: as merged to main

Context

Why this matters: Different tests answer different questions. Fast API tests prove correctness of behavior and errors. Contract tests prove the published OpenAPI snapshot did not change by accident. Load tests prove behavior under concurrency and rate limits—something unit tests cannot do—and should not run on every commit if they are slow or environment-specific.

ADR 0002 mandates API tests; ADR 0007 adds OpenAPI checks. This ADR ties those layers together and places optional load testing beside observability (ADR 0009, ADR 0011).

Decision

Layers

1. Automated API / integration tests (required) — pytest against the FastAPI app (in-process TestClient or equivalent). Covers happy paths, validation, auth, and business errors. This is the default regression net before merge.
2. Contract / OpenAPI governance (required) — make openapi-check and make contract-test guard schema drift and breaking changes relative to the checked-in baseline.
3. Manual exploratory testing — allowed for UX and edge cases not worth automating; findings should become tests or ADRs when behavior must stay fixed.
4. Load and traffic-shape testing (optional, not a default CI gate) — used to validate latency/error mix, rate limits, and dashboards under synthetic traffic. Implemented via the repository tool python -m tools.load_testing.runner with Python scenarios under tools/load_testing/scenarios/. Run against local or staging instances, not as a substitute for unit/API tests.

Load testing and safety

• Simulating HTTP 5xx against the same process for metrics requires an explicit opt-in endpoint enabled only when LOADTEST_HTTP_500=true (see application configuration). Never enable in production.
• Load scripts must not embed secrets; use the same .env / key conventions as local development.
• Interpret results together with Prometheus/Grafana (SLO panels per ADR 0011), not raw request counts alone.

Implementation in this repository

• Tests: tests/, entrypoints make test, make verify / make verify-ci.
• Traffic runner: tools/load_testing/runner.py, scenarios tools/load_testing/scenarios/, tools/load_testing/README.html.
• Developer-facing walkthrough: API load testing (step-by-step).

Consequences

Positive

• Clear separation: CI proves correctness; load tools prove behavior under volume and error mix.
• Repeatable traffic recipes checked into git (YAML).

Trade-offs

• Load scenarios can go stale; owners must update YAML when routes or auth change.
• High request rates may hit in-memory rate limits — throttle via --delay-ms or test profiles.

Related

• ADR 0002 — mandatory API testing policy.
• Developer guide: API load testing

Page history