Runbook: API Security Failing

Trigger

• 401 responses for protected endpoints even with expected credentials.
• 429 spikes on normal traffic profile.
• 413 responses after payload changes.
• CORS errors in browser or missing expected security headers in responses.

Fast triage

1. Check current env values for security settings in .env.
2. Verify request contains auth header configured by API_AUTH_HEADER.
3. Reproduce with one endpoint: POST /api/v1/user.
4. Inspect application logs for security middleware warnings.

Most common causes

• Wrong API_MOCK_API_KEY or wrong header name.
• Unexpected aggressive rate-limit settings for local/test load.
• Payload grew beyond API_BODY_MAX_BYTES.
• Frontend origin missing in CORS_ALLOW_ORIGINS.
• Middleware order or behavior changed without tests or an ADR.

Recovery steps by category

Auth failures (401)

• Validate API_AUTH_STRATEGY is expected (mock_api_key or disabled).
• Confirm request sends API_AUTH_HEADER with API_MOCK_API_KEY value.
• Re-run tests: make test-one path=tests/api/v1/test_user_create.py.

Rate-limit overblocking (429)

• Check API_RATE_LIMIT_REQUESTS and API_RATE_LIMIT_WINDOW_SECONDS.
• Validate client retry/backoff behavior.
• Before raising limits, agree policy in an ADR and update docs.

Body limit errors (413)

• Measure actual payload size and compare with API_BODY_MAX_BYTES.
• Reduce request body or plan dedicated upload flow.
• If increasing limit, update env/example, docs, and test coverage.

CORS / security headers

• Ensure frontend origin is explicitly allowlisted.
• Verify response headers include CSP and frame/content-type protections.
• Confirm browser preflight (OPTIONS) behavior is not blocked by auth/rate-limit.

Final recovery sequence

1. make test
2. make verify
3. make docs-fix

Exit criteria

• Protected endpoints behave as expected for auth, rate limit, and body size.
• CORS and security headers are present and validated.
• CI passes and docs match the security settings you ship.

Follow-up

• Security policy changes belong in an ADR.
• Add new repeating patterns to this runbook.

Page history