128 lines
4.7 KiB
Markdown
128 lines
4.7 KiB
Markdown
# Final project verification evidence
|
|
|
|
Captured on 2026-08-24 from the disposable QEMU and k3d integration labs. The
|
|
commands below expose no credentials and can be rerun before the presentation
|
|
to refresh screenshots or terminal recordings.
|
|
|
|
## Two-node Fedora cluster
|
|
|
|
`scripts/provision/lab.sh check` passed after provisioning, host reboot, node 2
|
|
disconnect/reconnect, and complete deletion/recreation of node 2. At the final
|
|
check both Fedora 44 machines ran k3s `v1.33.4+k3s1` and reported `Ready`:
|
|
|
|
```text
|
|
nereus-node1 Ready control-plane,master 192.168.122.10
|
|
nereus-node2 Ready <none> 192.168.122.11
|
|
```
|
|
|
|
The node-replacement test found and fixed two recovery requirements: remove the
|
|
replacement address from the private SSH known-hosts file, and delete the old
|
|
Kubernetes node identity before the new agent joins. The PostgreSQL local-path
|
|
volume was correctly treated as lost with the deleted VM, recreated empty, and
|
|
reseeded by the load generator.
|
|
|
|
## Live application
|
|
|
|
The final namespace check showed three ready API pods, one ready load-generator
|
|
pod, and one ready PostgreSQL pod. The same-origin Mini PC route returned:
|
|
|
|
```text
|
|
GET / 200
|
|
GET /healthz 200
|
|
GET /readyz 200
|
|
X-Nereus-Version 0.1.0
|
|
GET /api/v1/buoys 42 records
|
|
```
|
|
|
|
The static dashboard stayed available while the cluster API was unavailable
|
|
during node recovery, then recovered without a page reload.
|
|
|
|
## Container evidence
|
|
|
|
The locally inspected `nereus-api:0.1.0` image is 7,815,732 bytes and declares
|
|
numeric non-root user `65532`. The container verification also passed with a
|
|
read-only root filesystem.
|
|
|
|
## Rollback evidence
|
|
|
|
The local k3d lab was destroyed and recreated from scratch. Re-running
|
|
`scripts/k3d/lab.sh up` reconciled the running cluster successfully. The proof
|
|
produced these AnalysisRun results:
|
|
|
|
```text
|
|
probe-746bbb94df-2-pre Successful vector(0.0)
|
|
probe-556d5b659b-3-pre Failed vector(1.0)
|
|
```
|
|
|
|
After the failed analysis, the active Service selector remained on the healthy
|
|
revision `746bbb94df`.
|
|
|
|
## Log and trace correlation
|
|
|
|
Captured on 2026-08-27 from the two-node Fedora cluster. Loki and Tempo run in
|
|
the `observability` namespace, and a DaemonSet OTel Collector tails
|
|
`/var/log/pods` on both nodes.
|
|
|
|
The collector reads those files as non-root. k3s writes `/var/log/pods` as
|
|
`0750 root:root` and each container log as `0640 root:root`, so the pod runs
|
|
with `supplementalGroups: [0]` and keeps `runAsNonRoot: true`, `drop: ["ALL"]`,
|
|
a read-only root filesystem, and a read-only host mount. Granting
|
|
`DAC_READ_SEARCH` instead does not work: Kubernetes sets no ambient
|
|
capabilities, so the capability is cleared on exec and the container runs with
|
|
`CapEff: 0000000000000000`, matching no files and reporting no error.
|
|
|
|
Loki received logs from every Nereus container:
|
|
|
|
```text
|
|
k8s_container_name ["api", "loadgen", "postgres"]
|
|
k8s_namespace_name ["nereus"]
|
|
```
|
|
|
|
A single API request log line carries the trace identifier:
|
|
|
|
```text
|
|
{"time":"2026-08-27T19:21:24.107201Z","level":"INFO","msg":"request",
|
|
"method":"GET","path":"/api/v1/readings","status":200,"duration_ms":2,
|
|
"trace_id":"fd7812d8463298f50b4570374b03dfdb"}
|
|
```
|
|
|
|
Querying Tempo for that same identifier returned the matching trace, which is
|
|
the hop the provisioned Grafana data sources make in both directions:
|
|
|
|
```text
|
|
nereus-api http.request SPAN_KIND_SERVER
|
|
nereus-api list_readings SPAN_KIND_INTERNAL
|
|
```
|
|
|
|
Both data sources are provisioned by the labelled `nereus-grafana-datasources`
|
|
ConfigMap, which the kube-prometheus-stack Grafana sidecar loads as
|
|
`nereus.yaml`. The correlation requires the API at `LOG_LEVEL=info`, because the
|
|
per-request log line is the only one that carries `trace_id`.
|
|
|
|
Two collector defects were found and fixed on the way to this result, and both
|
|
belong in the report as corrective actions. The first is the capability problem
|
|
above. The second only appears after a restart: `start_at: beginning` makes the
|
|
receiver replay whole log files, the batch processor had no size cap, and Loki
|
|
refused the resulting push.
|
|
|
|
```text
|
|
HTTP 503, ResourceExhausted: grpc: received message larger than max
|
|
(4378796 vs. 4194304)
|
|
```
|
|
|
|
The exporter retries that request unchanged, so the backlog never drains.
|
|
Capping the batch with `send_batch_size: 1000` and `send_batch_max_size: 2000`
|
|
keeps every push under Loki's 4 MiB gRPC limit. What remains after a restart is
|
|
a `Permanent` rejection of replayed lines older than Loki's acceptance window,
|
|
which drops those lines rather than blocking, and leaves live traffic
|
|
unaffected.
|
|
|
|
## Presentation commands
|
|
|
|
```bash
|
|
scripts/provision/lab.sh check
|
|
curl -i https://nereus.fiwlabs.dev/healthz
|
|
curl -i https://nereus.fiwlabs.dev/readyz
|
|
curl -fsS https://nereus.fiwlabs.dev/api/v1/buoys | jq length
|
|
scripts/k3d/lab.sh prove
|
|
```
|