ADR 0023: Structured Logging, Request Correlation, and Local Elasticsearch

Ratification

• Discussion: backlog item 18 (structured logging) and operational need for searchable logs.
• Accepted: as merged to main with this ADR and implementation.

Context

Why this matters: Plain text logs are fine for a single terminal but do not scale to centralized search, dashboards, or joining logs with future distributed traces. Large teams standardize on structured logs (JSON), request correlation (X-Request-Id), and optional trace/span ids once OpenTelemetry is adopted.

Elasticsearch and Kibana are widely used for log search and analytics. Elastic publishes Docker images that are free to run locally for development and learning; production licensing and support are separate commercial decisions. This ADR covers local optional Compose only—not a commitment to Elastic Cloud or a paid tier.

Decision

• Support LOG_FORMAT=text and LOG_FORMAT=json (NDJSON: one JSON object per line). When LOG_FORMAT is unset, the process default is json so request_id is a first-class field. Stable JSON fields include @timestamp, level, message, service (LOG_SERVICE_NAME), app_env, request_id, trace_id, span_id (latter two null until tracing is wired).
• Add HTTP middleware (outermost) that sets X-Request-Id on every response: reuse a valid incoming header or generate a UUID.
• CORS expose_headers (CORS_EXPOSE_HEADERS) lists X-Request-Id (and rate-limit headers) so browser fetch on another origin can read them; OpenAPI is patched so Swagger UI shows an optional Parameters → X-Request-Id field and response headers for each operation.
• Optional Docker Compose stack: Elasticsearch + Kibana + Filebeat (docker-compose.logging.yml), mounting ./logs read-only into Filebeat. Ship NDJSON only: use LOG_FORMAT=json (or rely on the default json when LOG_FORMAT is unset).
• Full distributed tracing in Elasticsearch (Elastic APM or OTLP → Elastic) remains out of scope here; see backlog item 17 (OpenTelemetry).

Implementation

• app/core/request_context.py — context variables and X-Request-Id validation.
• app/core/logging.py — text vs JSON formatters, RequestContextFilter.
• app/main.py — request_context_middleware registered last (runs first on the wire).
• ops/filebeat/filebeat.yml, docker-compose.logging.yml, make logging-up / logging-down / logging-smoke / logging-reset.
• scripts/check_logging_links.py — smoke checks for Elasticsearch and Kibana.
• scripts/check_es_request_id.py / make logging-es-query — optional CLI search in Elasticsearch (no Kibana).
• Uvicorn is started with --no-access-log; access-style lines without request_id are avoided—use structured app.main request_done entries. Default LOG_FORMAT is json when unset so request_id is a first-class field for Kibana.

Kibana index pattern (important)

Create the data view with index pattern *study-app-logs*, not only study-app-logs-*. Otherwise Discover may target no indices: Elasticsearch 8 often writes Filebeat output to data stream backing indices named .ds-study-app-logs-…, which do not match a prefix of study-app-logs-. A symptom is “No results” for valid request_id queries even though documents exist in logs/app.log. Filebeat is configured with setup.template.type: legacy so new daily indices align with study-app-logs-*; the broad pattern still matches old and new names.

JSON lines must follow ECS shapes where the index template expects objects: use service.name (nested under service), not a string at service. Otherwise bulk indexing can fail with mapper_parsing_exception and indices show docs.count: 0 while logs/app.log still grows on disk.

Consequences

Positive

• Logs can be indexed and searched in Kibana during local debugging.
• Correlation id is visible to API clients and appears in log lines.
• No new Python dependency for JSON logging (stdlib only).

Trade-offs

• Elasticsearch + Kibana are memory-heavy (~2 GiB or more for a comfortable local stack).
• Filebeat expects JSON lines; rotating text logs with stack traces can break NDJSON—ship with LOG_FORMAT=json (or the default when unset).
• Security is disabled on the local single-node Elasticsearch profile—never expose that profile to the public internet.

Related

• ADR 0009 — metrics and probes.
• Local development — run targets and ports.
• Backlog item 17 — OpenTelemetry / distributed tracing.

Page history