Final project verification evidence

Captured on 2026-08-24 from the disposable QEMU and k3d integration labs. The commands below expose no credentials and can be rerun before the presentation to refresh screenshots or terminal recordings.

Two-node Fedora cluster

scripts/provision/lab.sh check passed after provisioning, host reboot, node 2 disconnect/reconnect, and complete deletion/recreation of node 2. At the final check both Fedora 44 machines ran k3s v1.33.4+k3s1 and reported Ready:

nereus-node1   Ready   control-plane,master   192.168.122.10
nereus-node2   Ready   <none>                 192.168.122.11

The node-replacement test found and fixed two recovery requirements: remove the replacement address from the private SSH known-hosts file, and delete the old Kubernetes node identity before the new agent joins. The PostgreSQL local-path volume was correctly treated as lost with the deleted VM, recreated empty, and reseeded by the load generator.

Live application

The final namespace check showed three ready API pods, one ready load-generator pod, and one ready PostgreSQL pod. The same-origin Mini PC route returned:

GET /                 200
GET /healthz          200
GET /readyz           200
X-Nereus-Version      0.1.0
GET /api/v1/buoys     42 records

The static dashboard stayed available while the cluster API was unavailable during node recovery, then recovered without a page reload.

Container evidence

The locally inspected nereus-api:0.1.0 image is 7,815,732 bytes and declares numeric non-root user 65532. The container verification also passed with a read-only root filesystem.

Rollback evidence

The local k3d lab was destroyed and recreated from scratch. Re-running scripts/k3d/lab.sh up reconciled the running cluster successfully. The proof produced these AnalysisRun results:

probe-746bbb94df-2-pre   Successful   vector(0.0)
probe-556d5b659b-3-pre   Failed       vector(1.0)

After the failed analysis, the active Service selector remained on the healthy revision 746bbb94df.

Log and trace correlation

Captured on 2026-08-27 from the two-node Fedora cluster. Loki and Tempo run in the observability namespace, and a DaemonSet OTel Collector tails /var/log/pods on both nodes.

The collector reads those files as non-root. k3s writes /var/log/pods as 0750 root:root and each container log as 0640 root:root, so the pod runs with supplementalGroups: [0] and keeps runAsNonRoot: true, drop: ["ALL"], a read-only root filesystem, and a read-only host mount. Granting DAC_READ_SEARCH instead does not work: Kubernetes sets no ambient capabilities, so the capability is cleared on exec and the container runs with CapEff: 0000000000000000, matching no files and reporting no error.

Loki received logs from every Nereus container:

k8s_container_name   ["api", "loadgen", "postgres"]
k8s_namespace_name   ["nereus"]

A single API request log line carries the trace identifier:

{"time":"2026-08-27T19:21:24.107201Z","level":"INFO","msg":"request",
 "method":"GET","path":"/api/v1/readings","status":200,"duration_ms":2,
 "trace_id":"fd7812d8463298f50b4570374b03dfdb"}

Querying Tempo for that same identifier returned the matching trace, which is the hop the provisioned Grafana data sources make in both directions:

nereus-api   http.request     SPAN_KIND_SERVER
nereus-api   list_readings    SPAN_KIND_INTERNAL

Both data sources are provisioned by the labelled nereus-grafana-datasources ConfigMap, which the kube-prometheus-stack Grafana sidecar loads as nereus.yaml. The correlation requires the API at LOG_LEVEL=info, because the per-request log line is the only one that carries trace_id.

Two collector defects were found and fixed on the way to this result, and both belong in the report as corrective actions. The first is the capability problem above. The second only appears after a restart: start_at: beginning makes the receiver replay whole log files, the batch processor had no size cap, and Loki refused the resulting push.

HTTP 503, ResourceExhausted: grpc: received message larger than max
(4378796 vs. 4194304)

The exporter retries that request unchanged, so the backlog never drains. Capping the batch with send_batch_size: 1000 and send_batch_max_size: 2000 keeps every push under Loki's 4 MiB gRPC limit. What remains after a restart is a Permanent rejection of replayed lines older than Loki's acceptance window, which drops those lines rather than blocking, and leaves live traffic unaffected.

Sealed database cutover

Verified on 2026-08-28 against the two-node Fedora cluster. PostgreSQL was stopped, its disposable PVC and manually managed Secret were removed, and the committed nereus-db SealedSecret created the replacement Secret. The controller reported Synced=True, the new 5 GiB PVC bound, PostgreSQL reached one ready replica, three restarted API pods became ready, and both load generators became healthy after reseeding the empty database. The first analysis attempts correctly rejected stale reset errors. After one full clean Prometheus window, AnalysisRun nereus-api-75d884b4dd-19-pre passed all five measurements and promoted revision 75d884b4dd to the active Service.

No plaintext credential was written to the repository, Terraform state, shell output, or evidence.

Presentation commands

scripts/provision/lab.sh check
curl -i https://nereus.fiwlabs.dev/healthz
curl -i https://nereus.fiwlabs.dev/readyz
curl -fsS https://nereus.fiwlabs.dev/api/v1/buoys | jq length
scripts/k3d/lab.sh prove

5.5 KiB Raw Blame History