diff --git a/docs/evidence.md b/docs/evidence.md index 2ae05f2..d8f5f9d 100644 --- a/docs/evidence.md +++ b/docs/evidence.md @@ -57,6 +57,48 @@ probe-556d5b659b-3-pre Failed vector(1.0) After the failed analysis, the active Service selector remained on the healthy revision `746bbb94df`. +## Log and trace correlation + +Captured on 2026-08-27 from the two-node Fedora cluster. Loki and Tempo run in +the `observability` namespace, and a DaemonSet OTel Collector tails +`/var/log/pods` on both nodes. + +The collector reads those files as non-root. k3s writes `/var/log/pods` as +`0750 root:root` and each container log as `0640 root:root`, so the pod runs +with `supplementalGroups: [0]` and keeps `runAsNonRoot: true`, `drop: ["ALL"]`, +a read-only root filesystem, and a read-only host mount. Granting +`DAC_READ_SEARCH` instead does not work: Kubernetes sets no ambient +capabilities, so the capability is cleared on exec and the container runs with +`CapEff: 0000000000000000`, matching no files and reporting no error. + +Loki received logs from every Nereus container: + +```text +k8s_container_name ["api", "loadgen", "postgres"] +k8s_namespace_name ["nereus"] +``` + +A single API request log line carries the trace identifier: + +```text +{"time":"2026-08-27T19:21:24.107201Z","level":"INFO","msg":"request", + "method":"GET","path":"/api/v1/readings","status":200,"duration_ms":2, + "trace_id":"fd7812d8463298f50b4570374b03dfdb"} +``` + +Querying Tempo for that same identifier returned the matching trace, which is +the hop the provisioned Grafana data sources make in both directions: + +```text +nereus-api http.request SPAN_KIND_SERVER +nereus-api list_readings SPAN_KIND_INTERNAL +``` + +Both data sources are provisioned by the labelled `nereus-grafana-datasources` +ConfigMap, which the kube-prometheus-stack Grafana sidecar loads as +`nereus.yaml`. The correlation requires the API at `LOG_LEVEL=info`, because the +per-request log line is the only one that carries `trace_id`. + ## Presentation commands ```bash diff --git a/docs/roadmap.md b/docs/roadmap.md index 68f2ac9..6b3a941 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -1,6 +1,6 @@ # Nereus delivery roadmap -Updated: 2026-08-26 +Updated: 2026-08-27 `[x]` means locally verified. `[~]` means implemented but awaiting a real integration environment. `[ ]` means not implemented. `[H]` is human-owned and @@ -115,14 +115,14 @@ task. Mark a task `[x]` only after its stated verification passes. - [x] Receive OTLP/gRPC traces in the Collector. - [x] Export traces from the Collector to Tempo. -- [~] Ship structured Kubernetes pod logs to Loki with the cluster Collector. +- [x] Ship structured Kubernetes pod logs to Loki with the cluster Collector. - [x] Start Loki locally and verify its Grafana data source readiness. - [x] Configure Prometheus to scrape `/metrics`. - [x] Add request-rate, error-rate, and latency panels. - [x] Add database-operation latency panels. - [x] Add readings-ingested and active-buoy panels. - [x] Add a deployed-version dashboard variable. -- [~] Correlate logs and traces using `trace_id` after cluster log ingestion exists. +- [x] Correlate logs and traces using `trace_id` after cluster log ingestion exists. - [x] Add API error-rate, latency, readiness, and scrape alerts. - [x] Load the dashboard and evaluate all six PromQL queries. - [x] Trigger chaos and distinguish the failing version in Grafana.