Final project verification evidence

Captured on 2026-08-24 from the disposable QEMU and k3d integration labs. The commands below expose no credentials and can be rerun before the presentation to refresh screenshots or terminal recordings.

Two-node Fedora cluster

scripts/provision/lab.sh check passed after provisioning, host reboot, node 2 disconnect/reconnect, and complete deletion/recreation of node 2. At the final check both Fedora 44 machines ran k3s v1.33.4+k3s1 and reported Ready:

nereus-node1   Ready   control-plane,master   192.168.122.10
nereus-node2   Ready   <none>                 192.168.122.11

The node-replacement test found and fixed two recovery requirements: remove the replacement address from the private SSH known-hosts file, and delete the old Kubernetes node identity before the new agent joins. The PostgreSQL local-path volume was correctly treated as lost with the deleted VM, recreated empty, and reseeded by the load generator.

Live application

The final namespace check showed three ready API pods, one ready load-generator pod, and one ready PostgreSQL pod. The same-origin Mini PC route returned:

GET /                 200
GET /healthz          200
GET /readyz           200
X-Nereus-Version      0.1.0
GET /api/v1/buoys     42 records

The static dashboard stayed available while the cluster API was unavailable during node recovery, then recovered without a page reload.

Container evidence

The locally inspected nereus-api:0.1.0 image is 7,815,732 bytes and declares numeric non-root user 65532. The container verification also passed with a read-only root filesystem.

Rollback evidence

The local k3d lab was destroyed and recreated from scratch. Re-running scripts/k3d/lab.sh up reconciled the running cluster successfully. The proof produced these AnalysisRun results:

probe-746bbb94df-2-pre   Successful   vector(0.0)
probe-556d5b659b-3-pre   Failed       vector(1.0)

After the failed analysis, the active Service selector remained on the healthy revision 746bbb94df.

Log and trace correlation

Captured on 2026-08-27 from the two-node Fedora cluster. Loki and Tempo run in the observability namespace, and a DaemonSet OTel Collector tails /var/log/pods on both nodes.

The collector reads those files as non-root. k3s writes /var/log/pods as 0750 root:root and each container log as 0640 root:root, so the pod runs with supplementalGroups: [0] and keeps runAsNonRoot: true, drop: ["ALL"], a read-only root filesystem, and a read-only host mount. Granting DAC_READ_SEARCH instead does not work: Kubernetes sets no ambient capabilities, so the capability is cleared on exec and the container runs with CapEff: 0000000000000000, matching no files and reporting no error.

Loki received logs from every Nereus container:

k8s_container_name   ["api", "loadgen", "postgres"]
k8s_namespace_name   ["nereus"]

A single API request log line carries the trace identifier:

{"time":"2026-08-27T19:21:24.107201Z","level":"INFO","msg":"request",
 "method":"GET","path":"/api/v1/readings","status":200,"duration_ms":2,
 "trace_id":"fd7812d8463298f50b4570374b03dfdb"}

Querying Tempo for that same identifier returned the matching trace, which is the hop the provisioned Grafana data sources make in both directions:

nereus-api   http.request     SPAN_KIND_SERVER
nereus-api   list_readings    SPAN_KIND_INTERNAL

Both data sources are provisioned by the labelled nereus-grafana-datasources ConfigMap, which the kube-prometheus-stack Grafana sidecar loads as nereus.yaml. The correlation requires the API at LOG_LEVEL=info, because the per-request log line is the only one that carries trace_id.

Presentation commands

scripts/provision/lab.sh check
curl -i https://nereus.fiwlabs.dev/healthz
curl -i https://nereus.fiwlabs.dev/readyz
curl -fsS https://nereus.fiwlabs.dev/api/v1/buoys | jq length
scripts/k3d/lab.sh prove

3.9 KiB Raw Blame History