docs: record log and trace correlation

docs/evidence.md

@@ -57,6 +57,48 @@ probe-556d5b659b-3-pre   Failed       vector(1.0)
 After the failed analysis, the active Service selector remained on the healthy
 revision `746bbb94df`.
 
+## Log and trace correlation
+
+Captured on 2026-08-27 from the two-node Fedora cluster. Loki and Tempo run in
+the `observability` namespace, and a DaemonSet OTel Collector tails
+`/var/log/pods` on both nodes.
+
+The collector reads those files as non-root. k3s writes `/var/log/pods` as
+`0750 root:root` and each container log as `0640 root:root`, so the pod runs
+with `supplementalGroups: [0]` and keeps `runAsNonRoot: true`, `drop: ["ALL"]`,
+a read-only root filesystem, and a read-only host mount. Granting
+`DAC_READ_SEARCH` instead does not work: Kubernetes sets no ambient
+capabilities, so the capability is cleared on exec and the container runs with
+`CapEff: 0000000000000000`, matching no files and reporting no error.
+
+Loki received logs from every Nereus container:
+
+```text
+k8s_container_name   ["api", "loadgen", "postgres"]
+k8s_namespace_name   ["nereus"]
+```
+
+A single API request log line carries the trace identifier:
+
+```text
+{"time":"2026-08-27T19:21:24.107201Z","level":"INFO","msg":"request",
+ "method":"GET","path":"/api/v1/readings","status":200,"duration_ms":2,
+ "trace_id":"fd7812d8463298f50b4570374b03dfdb"}
+```
+
+Querying Tempo for that same identifier returned the matching trace, which is
+the hop the provisioned Grafana data sources make in both directions:
+
+```text
+nereus-api   http.request     SPAN_KIND_SERVER
+nereus-api   list_readings    SPAN_KIND_INTERNAL
+```
+
+Both data sources are provisioned by the labelled `nereus-grafana-datasources`
+ConfigMap, which the kube-prometheus-stack Grafana sidecar loads as
+`nereus.yaml`. The correlation requires the API at `LOG_LEVEL=info`, because the
+per-request log line is the only one that carries `trace_id`.
+
 ## Presentation commands
 
 ```bash

docs/roadmap.md

@@ -1,6 +1,6 @@
 # Nereus delivery roadmap
 
-Updated: 2026-08-26
+Updated: 2026-08-27
 
 `[x]` means locally verified. `[~]` means implemented but awaiting a real
 integration environment. `[ ]` means not implemented. `[H]` is human-owned and
@@ -115,14 +115,14 @@ task. Mark a task `[x]` only after its stated verification passes.
 
 - [x] Receive OTLP/gRPC traces in the Collector.
 - [x] Export traces from the Collector to Tempo.
-- [~] Ship structured Kubernetes pod logs to Loki with the cluster Collector.
+- [x] Ship structured Kubernetes pod logs to Loki with the cluster Collector.
 - [x] Start Loki locally and verify its Grafana data source readiness.
 - [x] Configure Prometheus to scrape `/metrics`.
 - [x] Add request-rate, error-rate, and latency panels.
 - [x] Add database-operation latency panels.
 - [x] Add readings-ingested and active-buoy panels.
 - [x] Add a deployed-version dashboard variable.
-- [~] Correlate logs and traces using `trace_id` after cluster log ingestion exists.
+- [x] Correlate logs and traces using `trace_id` after cluster log ingestion exists.
 - [x] Add API error-rate, latency, readiness, and scrape alerts.
 - [x] Load the dashboard and evaluate all six PromQL queries.
 - [x] Trigger chaos and distinguish the failing version in Grafana.