# Final project verification evidence

Captured on 2026-08-24 from the disposable QEMU and k3d integration labs. The
commands below expose no credentials and can be rerun before the presentation
to refresh screenshots or terminal recordings.

## Two-node Fedora cluster

`scripts/provision/lab.sh check` passed after provisioning, host reboot, node 2
disconnect/reconnect, and complete deletion/recreation of node 2. At the final
check both Fedora 44 machines ran k3s `v1.33.4+k3s1` and reported `Ready`:

```text
nereus-node1   Ready   control-plane,master   192.168.122.10
nereus-node2   Ready   <none>                 192.168.122.11
```

The node-replacement test found and fixed two recovery requirements: remove the
replacement address from the private SSH known-hosts file, and delete the old
Kubernetes node identity before the new agent joins. The PostgreSQL local-path
volume was correctly treated as lost with the deleted VM, recreated empty, and
reseeded by the load generator.

## Live application

The final namespace check showed three ready API pods, one ready load-generator
pod, and one ready PostgreSQL pod. The same-origin Mini PC route returned:

```text
GET /                 200
GET /healthz          200
GET /readyz           200
X-Nereus-Version      0.1.0
GET /api/v1/buoys     42 records
```

The static dashboard stayed available while the cluster API was unavailable
during node recovery, then recovered without a page reload.

## Container evidence

The locally inspected `nereus-api:0.1.0` image is 7,815,732 bytes and declares
numeric non-root user `65532`. The container verification also passed with a
read-only root filesystem.

## Rollback evidence

The local k3d lab was destroyed and recreated from scratch. Re-running
`scripts/k3d/lab.sh up` reconciled the running cluster successfully. The proof
produced these AnalysisRun results:

```text
probe-746bbb94df-2-pre   Successful   vector(0.0)
probe-556d5b659b-3-pre   Failed       vector(1.0)
```

After the failed analysis, the active Service selector remained on the healthy
revision `746bbb94df`.

## Log and trace correlation

Captured on 2026-08-27 from the two-node Fedora cluster. Loki and Tempo run in
the `observability` namespace, and a DaemonSet OTel Collector tails
`/var/log/pods` on both nodes.

The collector reads those files as non-root. k3s writes `/var/log/pods` as
`0750 root:root` and each container log as `0640 root:root`, so the pod runs
with `supplementalGroups: [0]` and keeps `runAsNonRoot: true`, `drop: ["ALL"]`,
a read-only root filesystem, and a read-only host mount. Granting
`DAC_READ_SEARCH` instead does not work: Kubernetes sets no ambient
capabilities, so the capability is cleared on exec and the container runs with
`CapEff: 0000000000000000`, matching no files and reporting no error.

Loki received logs from every Nereus container:

```text
k8s_container_name   ["api", "loadgen", "postgres"]
k8s_namespace_name   ["nereus"]
```

A single API request log line carries the trace identifier:

```text
{"time":"2026-08-27T19:21:24.107201Z","level":"INFO","msg":"request",
 "method":"GET","path":"/api/v1/readings","status":200,"duration_ms":2,
 "trace_id":"fd7812d8463298f50b4570374b03dfdb"}
```

Querying Tempo for that same identifier returned the matching trace, which is
the hop the provisioned Grafana data sources make in both directions:

```text
nereus-api   http.request     SPAN_KIND_SERVER
nereus-api   list_readings    SPAN_KIND_INTERNAL
```

Both data sources are provisioned by the labelled `nereus-grafana-datasources`
ConfigMap, which the kube-prometheus-stack Grafana sidecar loads as
`nereus.yaml`. The correlation requires the API at `LOG_LEVEL=info`, because the
per-request log line is the only one that carries `trace_id`.

Two collector defects were found and fixed on the way to this result, and both
belong in the report as corrective actions. The first is the capability problem
above. The second only appears after a restart: `start_at: beginning` makes the
receiver replay whole log files, the batch processor had no size cap, and Loki
refused the resulting push.

```text
HTTP 503, ResourceExhausted: grpc: received message larger than max
(4378796 vs. 4194304)
```

The exporter retries that request unchanged, so the backlog never drains.
Capping the batch with `send_batch_size: 1000` and `send_batch_max_size: 2000`
keeps every push under Loki's 4 MiB gRPC limit. What remains after a restart is
a `Permanent` rejection of replayed lines older than Loki's acceptance window,
which drops those lines rather than blocking, and leaves live traffic
unaffected.

## Presentation commands

```bash
scripts/provision/lab.sh check
curl -i https://nereus.fiwlabs.dev/healthz
curl -i https://nereus.fiwlabs.dev/readyz
curl -fsS https://nereus.fiwlabs.dev/api/v1/buoys | jq length
scripts/k3d/lab.sh prove
```