fiwdev/Nereus
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: record log and trace correlation

Browse source
This commit is contained in:
Alex 2026-08-27 21:22:25 +02:00
parent 02a40a0d6a
commit 32d615a539
2 changed files with 45 additions and 3 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

42
docs/evidence.md
View file

@ -57,6 +57,48 @@ probe-556d5b659b-3-pre Failed vector(1.0)
After the failed analysis, the active Service selector remained on the healthy
revision `746bbb94df`.
## Log and trace correlation
Captured on 2026-08-27 from the two-node Fedora cluster. Loki and Tempo run in
the `observability` namespace, and a DaemonSet OTel Collector tails
`/var/log/pods` on both nodes.
The collector reads those files as non-root. k3s writes `/var/log/pods` as
`0750 root:root` and each container log as `0640 root:root`, so the pod runs
with `supplementalGroups: [0]` and keeps `runAsNonRoot: true`, `drop: ["ALL"]`,
a read-only root filesystem, and a read-only host mount. Granting
`DAC_READ_SEARCH` instead does not work: Kubernetes sets no ambient
capabilities, so the capability is cleared on exec and the container runs with
`CapEff: 0000000000000000`, matching no files and reporting no error.
Loki received logs from every Nereus container:
```text
k8s_container_name ["api", "loadgen", "postgres"]
k8s_namespace_name ["nereus"]
```
A single API request log line carries the trace identifier:
```text
{"time":"2026-08-27T19:21:24.107201Z","level":"INFO","msg":"request",
"method":"GET","path":"/api/v1/readings","status":200,"duration_ms":2,
"trace_id":"fd7812d8463298f50b4570374b03dfdb"}
```
Querying Tempo for that same identifier returned the matching trace, which is
the hop the provisioned Grafana data sources make in both directions:
```text
nereus-api http.request SPAN_KIND_SERVER
nereus-api list_readings SPAN_KIND_INTERNAL
```
Both data sources are provisioned by the labelled `nereus-grafana-datasources`
ConfigMap, which the kube-prometheus-stack Grafana sidecar loads as
`nereus.yaml`. The correlation requires the API at `LOG_LEVEL=info`, because the
per-request log line is the only one that carries `trace_id`.
## Presentation commands
```bash

6
docs/roadmap.md
View file

@ -1,6 +1,6 @@
# Nereus delivery roadmap
Updated: 2026-08-26
Updated: 2026-08-27
`[x]` means locally verified. `[~]` means implemented but awaiting a real
integration environment. `[ ]` means not implemented. `[H]` is human-owned and
@ -115,14 +115,14 @@ task. Mark a task `[x]` only after its stated verification passes.
- [x] Receive OTLP/gRPC traces in the Collector.
- [x] Export traces from the Collector to Tempo.
- [~] Ship structured Kubernetes pod logs to Loki with the cluster Collector.
- [x] Ship structured Kubernetes pod logs to Loki with the cluster Collector.
- [x] Start Loki locally and verify its Grafana data source readiness.
- [x] Configure Prometheus to scrape `/metrics`.
- [x] Add request-rate, error-rate, and latency panels.
- [x] Add database-operation latency panels.
- [x] Add readings-ingested and active-buoy panels.
- [x] Add a deployed-version dashboard variable.
- [~] Correlate logs and traces using `trace_id` after cluster log ingestion exists.
- [x] Correlate logs and traces using `trace_id` after cluster log ingestion exists.
- [x] Add API error-rate, latency, readiness, and scrape alerts.
- [x] Load the dashboard and evaluate all six PromQL queries.
- [x] Trigger chaos and distinguish the failing version in Grafana.