# Final project verification evidence

Captured on 2026-08-24 from the disposable QEMU and k3d integration labs. The
commands below expose no credentials and can be rerun before the presentation
to refresh screenshots or terminal recordings.

## Two-node Fedora cluster

`scripts/provision/lab.sh check` passed after provisioning, host reboot, node 2
disconnect/reconnect, and complete deletion/recreation of node 2. At the final
check both Fedora 44 machines ran k3s `v1.33.4+k3s1` and reported `Ready`:

```text
nereus-node1   Ready   control-plane,master   192.168.122.10
nereus-node2   Ready   <none>                 192.168.122.11
```

The node-replacement test found and fixed two recovery requirements: remove the
replacement address from the private SSH known-hosts file, and delete the old
Kubernetes node identity before the new agent joins. The PostgreSQL local-path
volume was correctly treated as lost with the deleted VM, recreated empty, and
reseeded by the load generator.

## Live application

The final namespace check showed three ready API pods, one ready load-generator
pod, and one ready PostgreSQL pod. The same-origin Mini PC route returned:

```text
GET /                 200
GET /healthz          200
GET /readyz           200
X-Nereus-Version      0.1.0
GET /api/v1/buoys     42 records
```

The static dashboard stayed available while the cluster API was unavailable
during node recovery, then recovered without a page reload.

## Container evidence

The locally inspected `nereus-api:0.1.0` image is 7,815,732 bytes and declares
numeric non-root user `65532`. The container verification also passed with a
read-only root filesystem.

## Rollback evidence

The local k3d lab was destroyed and recreated from scratch. Re-running
`scripts/k3d/lab.sh up` reconciled the running cluster successfully. The proof
produced these AnalysisRun results:

```text
probe-746bbb94df-2-pre   Successful   vector(0.0)
probe-556d5b659b-3-pre   Failed       vector(1.0)
```

After the failed analysis, the active Service selector remained on the healthy
revision `746bbb94df`.

## Log and trace correlation

Captured on 2026-08-27 from the two-node Fedora cluster. Loki and Tempo run in
the `observability` namespace, and a DaemonSet OTel Collector tails
`/var/log/pods` on both nodes.

The collector reads those files as non-root. k3s writes `/var/log/pods` as
`0750 root:root` and each container log as `0640 root:root`, so the pod runs
with `supplementalGroups: [0]` and keeps `runAsNonRoot: true`, `drop: ["ALL"]`,
a read-only root filesystem, and a read-only host mount. Granting
`DAC_READ_SEARCH` instead does not work: Kubernetes sets no ambient
capabilities, so the capability is cleared on exec and the container runs with
`CapEff: 0000000000000000`, matching no files and reporting no error.

Loki received logs from every Nereus container:

```text
k8s_container_name   ["api", "loadgen", "postgres"]
k8s_namespace_name   ["nereus"]
```

A single API request log line carries the trace identifier:

```text
{"time":"2026-08-27T19:21:24.107201Z","level":"INFO","msg":"request",
 "method":"GET","path":"/api/v1/readings","status":200,"duration_ms":2,
 "trace_id":"fd7812d8463298f50b4570374b03dfdb"}
```

Querying Tempo for that same identifier returned the matching trace, which is
the hop the provisioned Grafana data sources make in both directions:

```text
nereus-api   http.request     SPAN_KIND_SERVER
nereus-api   list_readings    SPAN_KIND_INTERNAL
```

Both data sources are provisioned by the labelled `nereus-grafana-datasources`
ConfigMap, which the kube-prometheus-stack Grafana sidecar loads as
`nereus.yaml`. The correlation requires the API at `LOG_LEVEL=info`, because the
per-request log line is the only one that carries `trace_id`.

## Presentation commands

```bash
scripts/provision/lab.sh check
curl -i https://nereus.fiwlabs.dev/healthz
curl -i https://nereus.fiwlabs.dev/readyz
curl -fsS https://nereus.fiwlabs.dev/api/v1/buoys | jq length
scripts/k3d/lab.sh prove
```