docs: record deployment checks

docs/evidence.md

@@ -0,0 +1,68 @@
+# Final project verification evidence
+
+Captured on 2026-08-24 from the disposable QEMU and k3d integration labs. The
+commands below expose no credentials and can be rerun before the presentation
+to refresh screenshots or terminal recordings.
+
+## Two-node Fedora cluster
+
+`scripts/provision/lab.sh check` passed after provisioning, host reboot, node 2
+disconnect/reconnect, and complete deletion/recreation of node 2. At the final
+check both Fedora 44 machines ran k3s `v1.33.4+k3s1` and reported `Ready`:
+
+```text
+nereus-node1   Ready   control-plane,master   192.168.122.10
+nereus-node2   Ready   <none>                 192.168.122.11
+```
+
+The node-replacement test found and fixed two recovery requirements: remove the
+replacement address from the private SSH known-hosts file, and delete the old
+Kubernetes node identity before the new agent joins. The PostgreSQL local-path
+volume was correctly treated as lost with the deleted VM, recreated empty, and
+reseeded by the load generator.
+
+## Live application
+
+The final namespace check showed three ready API pods, one ready load-generator
+pod, and one ready PostgreSQL pod. The same-origin Mini PC route returned:
+
+```text
+GET /                 200
+GET /healthz          200
+GET /readyz           200
+X-Nereus-Version      0.1.0
+GET /api/v1/buoys     42 records
+```
+
+The static dashboard stayed available while the cluster API was unavailable
+during node recovery, then recovered without a page reload.
+
+## Container evidence
+
+The locally inspected `nereus-api:0.1.0` image is 7,815,732 bytes and declares
+numeric non-root user `65532`. The container verification also passed with a
+read-only root filesystem.
+
+## Rollback evidence
+
+The local k3d lab was destroyed and recreated from scratch. Re-running
+`scripts/k3d/lab.sh up` reconciled the running cluster successfully. The proof
+produced these AnalysisRun results:
+
+```text
+probe-746bbb94df-2-pre   Successful   vector(0.0)
+probe-556d5b659b-3-pre   Failed       vector(1.0)
+```
+
+After the failed analysis, the active Service selector remained on the healthy
+revision `746bbb94df`.
+
+## Presentation commands
+
+```bash
+scripts/provision/lab.sh check
+curl -i https://nereus.fiwlabs.dev/healthz
+curl -i https://nereus.fiwlabs.dev/readyz
+curl -fsS https://nereus.fiwlabs.dev/api/v1/buoys | jq length
+scripts/k3d/lab.sh prove
+```

docs/mini-pc.md

@@ -0,0 +1,31 @@
+# Mini PC web stack
+
+The Mini PC stack joins the host's existing Traefik `proxy` network and proxies
+the frozen API paths through nginx. It does not run its own Traefik instance,
+PostgreSQL, the API, or the load generator.
+
+Export the following values in the operator shell or service manager:
+
+```text
+WEB_HOST          public dashboard hostname; defaults to nereus.fiwlabs.dev
+CLUSTER_API_UPSTREAM cluster address and port, without a URL scheme
+WEB_ASSET_PATH    asset path relative to the build context; defaults to apps/web
+```
+
+Do not put these values in a repository `.env` file. Start the isolated stack
+from the repository root:
+
+```sh
+docker compose -f build/compose.mini.yaml up -d --build
+```
+
+The Mini PC keeps the static assets directly in `~/Server/web/nereus`. Copy the
+three build files into its `build/` directory, set `WEB_ASSET_PATH=.` when
+building there, and join the existing external `proxy` network. The existing
+Traefik instance supplies the `websecure` entrypoint and `porkbun` certificate
+resolver; do not start a second proxy on ports 80 and 443.
+
+When the cluster is unavailable, nginx continues serving the dashboard and
+returns a gateway failure only for `/api/v1/*`, `/healthz`, and `/readyz`. The
+browser keeps polling those relative paths and recovers when the cluster is
+reachable again.

docs/roadmap.md

@@ -0,0 +1,232 @@
+# Nereus delivery roadmap
+
+Updated: 2026-08-24
+
+`[x]` means locally verified. `[~]` means implemented but awaiting a real
+integration environment. `[ ]` means not implemented. `[H]` is human-owned and
+must not be edited by agents.
+
+Update this file in the same change that completes or materially changes a
+task. Mark a task `[x]` only after its stated verification passes.
+
+## API service
+
+- [x] Create the Go 1.26 module and dependency lock file.
+- [x] Create the idempotent `buoys` table migration.
+- [x] Create the idempotent `readings` table migration.
+- [x] Create `idx_readings_buoy_time` idempotently.
+- [x] Parse `PORT`, `CHAOS_ERROR_RATE`, and `LOG_LEVEL`.
+- [x] Preserve the build-injected `APP_VERSION` value.
+- [x] Require `DATABASE_URL` without logging it.
+- [x] Start HTTP without waiting for PostgreSQL.
+- [x] Retry migrations while PostgreSQL is unavailable.
+- [x] Keep `/healthz` independent of PostgreSQL.
+- [x] Require a reachable, migrated database for `/readyz`.
+- [x] Implement `GET /api/v1/buoys`.
+- [x] Implement `POST /api/v1/buoys`.
+- [x] Implement `GET /api/v1/buoys/{id}`.
+- [x] Implement `DELETE /api/v1/buoys/{id}`.
+- [x] Implement filtered, paginated `GET /api/v1/readings`.
+- [x] Implement `POST /api/v1/readings`.
+- [x] Implement the real grouped aggregate query.
+- [x] Bound JSON request bodies and return JSON errors.
+- [x] Restrict chaos injection to `/api/v1/*`.
+- [x] Expose all five required Prometheus metrics.
+- [x] Use chi route templates for the HTTP `path` label.
+- [x] Enable OTLP/gRPC tracing only when configured.
+- [x] Create HTTP spans and child database spans.
+- [x] Emit structured request logs containing `trace_id`.
+- [x] Add table-driven handler tests without a live database.
+- [x] Test chaos and unaffected health behavior.
+- [x] Add a `testing.Short()`-guarded PostgreSQL integration test.
+- [x] Pass `go test -short ./...`.
+- [x] Pass `go vet ./...`.
+- [x] Build with `CGO_ENABLED=0`.
+- [x] Run the integration test against PostgreSQL 17.
+- [x] Test database failure behavior on every data endpoint.
+- [x] Assert exact metric names and route-template labels in tests.
+- [x] Test malformed and oversized JSON request bodies.
+- [ ] Run `golangci-lint run` with the CI-selected version.
+
+## Load generator
+
+- [x] Create the standalone Go 1.26 module.
+- [x] Require `TARGET_URL` and default `RPS` to 5.
+- [x] Reuse existing buoys or seed 42 validated offshore buoys.
+- [x] Generate the 60/25/15 request mix.
+- [x] Generate plausible synthetic reading values.
+- [x] Count HTTP statuses and network errors.
+- [x] Log a JSON status summary every ten seconds.
+- [x] Continue traffic while requests and seeding fail.
+- [x] Retry seeding without blocking read traffic.
+- [x] Handle SIGTERM and SIGINT.
+- [x] Keep the implementation below 200 lines.
+- [x] Pass `go vet` and a static build.
+- [x] Test request weighting deterministically.
+- [x] Test seeding against an empty `httptest` API.
+- [x] Test continued traffic through repeated HTTP 500 responses.
+- [x] Run against the real API for at least five minutes.
+- [x] Confirm Prometheus receives continuous request samples.
+
+## Container build and local Compose
+
+- [x] Create the API builder stage from `golang:1.26-alpine`.
+- [x] Cache dependencies before copying API source.
+- [x] Build with `CGO_ENABLED=0` and `-ldflags "-s -w"`.
+- [x] Inject `APP_VERSION` with `-X`.
+- [x] Copy only the API binary into `gcr.io/distroless/static:nonroot`.
+- [x] Create the load-generator multi-stage image.
+- [x] Use `gcr.io/distroless/static:nonroot` for the load generator.
+- [x] Add the required `.dockerignore` exclusions.
+- [x] Confirm both images run non-root and read-only.
+- [x] Confirm the API image is below 25 MB (7,815,877 bytes locally).
+- [x] Add API, PostgreSQL 17, and OTEL Collector to local Compose.
+- [x] Add the PostgreSQL healthcheck.
+- [x] Make the API wait on healthy PostgreSQL in Compose.
+- [x] Verify `docker compose up` reaches API readiness.
+- [x] Keep Mini PC and local-development Compose files separate.
+
+## Static web dashboard
+
+- [x] Create the no-build static HTML entry point.
+- [x] Create the dark teal/navy glass visual system.
+- [x] Replace decorative background waves with an operational chart grid.
+- [x] Keep version, health, and error panels visible together.
+- [x] Show the build-injected API version from the existing health response header.
+- [x] Poll `/healthz` every two seconds.
+- [x] Poll `/readyz` every two seconds.
+- [x] Show each health indicator's last transition timestamp.
+- [x] Count non-2xx `/api/v1/*` browser responses.
+- [x] Calculate the rolling 60-second HTTP error rate.
+- [x] Distinguish network-offline events from HTTP errors.
+- [x] Preserve and grey the last known telemetry values offline.
+- [x] Recover automatically when the API returns.
+- [x] Add a fully local Natural Earth buoy map and pulsing markers.
+- [x] Add animated reading charts.
+- [x] Keep all API calls same-origin and relative.
+- [x] Remove all frontend runtime dependencies on external hosts.
+- [x] Verify all panels fit at 1280×720 without page scrolling.
+- [x] Create the unprivileged nginx image.
+- [x] Create the separate Mini PC Traefik Compose file.
+- [x] Test the finished offline/degraded rendering without API connectivity.
+- [x] Test the rendered dashboard while the API returns HTTP 500 responses.
+
+## Observability
+
+- [x] Receive OTLP/gRPC traces in the Collector.
+- [x] Export traces from the Collector to Tempo.
+- [~] Ship structured Kubernetes pod logs to Loki with the cluster Collector.
+- [x] Start Loki locally and verify its Grafana data source readiness.
+- [x] Configure Prometheus to scrape `/metrics`.
+- [x] Add request-rate, error-rate, and latency panels.
+- [x] Add database-operation latency panels.
+- [x] Add readings-ingested and active-buoy panels.
+- [x] Add a deployed-version dashboard variable.
+- [~] Correlate logs and traces using `trace_id` after cluster log ingestion exists.
+- [x] Add API error-rate, latency, readiness, and scrape alerts.
+- [x] Load the dashboard and evaluate all six PromQL queries.
+- [x] Trigger chaos and distinguish the failing version in Grafana.
+
+## Fedora 44 host provisioning
+
+- [x] Separate node 1 and node 2 inventory groups.
+- [x] Pin the project k3s version.
+- [x] Reject managed hosts that are not Fedora 44.
+- [x] Install common host dependencies.
+- [x] Enable firewalld.
+- [x] Trust the configured pod and service CIDRs.
+- [x] Open VXLAN port 8472/UDP between nodes.
+- [x] Open kubelet port 10250/TCP between nodes.
+- [x] Open API server port 6443/TCP on node 1.
+- [x] Install node 1 as the k3s server.
+- [x] Install node 2 as a k3s agent.
+- [x] Keep the cluster token out of repository files and output.
+- [x] Enable both k3s systemd services.
+- [x] Wait for every Kubernetes node to become Ready.
+- [x] Add the equivalent manual configuration checklist.
+- [x] Document Terraform-to-Ansible inventory handoff.
+- [x] Keep libvirt addresses, bridge rules, and VM network profiles out of the reusable Ansible roles.
+- [x] Run the playbook twice on two clean Fedora 44 QEMU machines.
+- [x] Confirm the second run reports no unintended changes (`changed=0`).
+- [x] Restrict firewall sources to actual node/operator networks.
+- [x] Confirm SELinux remains enforcing on both QEMU machines.
+- [x] Reboot both QEMU machines and confirm automatic recovery.
+- [x] Disconnect and reconnect node 2, then confirm it rejoins.
+- [x] Record tested minimum CPU, memory, and disk requirements.
+
+## One-command QEMU lab
+
+- [x] Check QEMU, libvirt, cloud-utils, Ansible, SSH, ping, and KVM.
+- [x] Install missing dependencies on a Fedora workstation.
+- [x] Enable libvirt and its default network.
+- [x] Discover and cache the Fedora 44 x86_64 cloud image.
+- [x] Keep SSH and VM state outside the repository.
+- [x] Generate cloud-init media for both nodes.
+- [x] Create reusable copy-on-write disks.
+- [x] Create each VM with two vCPUs and 3 GiB RAM.
+- [x] Assign deterministic addresses to both local VMs.
+- [x] Wait for ICMP ping and SSH on both nodes.
+- [x] Generate the Ansible inventory automatically.
+- [x] Configure both nodes automatically.
+- [x] Check Ansible reachability, k3s files, and binaries.
+- [x] Check firewalld, k3s services, ports 6443 and 10250.
+- [x] Check Kubernetes node readiness.
+- [x] Add `check`, `stop`, and explicit `destroy` actions.
+- [x] Execute `lab.sh up` with working `/dev/kvm` and pass the full verification.
+- [x] Reboot both local VMs and pass the bounded post-reboot service and node checks.
+- [x] Execute `lab.sh check` after a workstation reboot.
+- [x] Test recovery from an interrupted image download and validate the resumed QCOW2.
+- [x] Test recovery after node 2 is deleted manually and rejoin it with a fresh identity.
+- [ ] Test on a workstation with no dependencies installed.
+- [x] Decide whether an aarch64 image path is required; keep the lab x86_64-only.
+- [ ] After final sign-off, destroy the QEMU lab and remove packages installed only for it with Pacman's dependency-aware cleanup.
+
+## Local k3d rollback mechanism
+
+- [x] Define one local server and two local agent nodes.
+- [x] Map Traefik to host ports 8080 and 8443.
+- [x] Add lean kube-prometheus-stack values.
+- [x] Add the synthetic Argo Rollouts analysis harness.
+- [x] Add an idempotent `up/check/destroy` command and verify fresh creation and reconciliation.
+- [x] Install Argo Rollouts.
+- [x] Install kube-prometheus-stack with the local values.
+- [x] Apply the synthetic harness.
+- [x] Prove `vector(0.0)` permits promotion.
+- [x] Change the test value to `vector(1.0)`.
+- [x] Prove failed analysis aborts promotion with `vector(0.9)`.
+- [x] Tear down and recreate the test from scratch.
+
+## Human-owned deployment and infrastructure
+
+- [H] Define infrastructure modules under `terraform/`.
+- [H] Export provisioned addresses for Ansible inventory.
+- [H] Create Kubernetes base resources and overlays. Agent-authored 2026-08-21 at the owner's request; builds and server dry-runs pass, not applied.
+- [H] Create API and load-generator workloads. Agent-authored 2026-08-21; not applied.
+- [H] Create active and preview Services. Agent-authored 2026-08-21; not applied.
+- [H] Create the Prometheus ServiceMonitor. Agent-authored 2026-08-21; not applied, scrape not confirmed.
+- [H] Create the real error-rate AnalysisTemplate. Agent-authored 2026-08-21; PromQL never evaluated against real series.
+- [H] Configure blue-green promotion and automated rollback. Agent-authored 2026-08-21; promotion and abort paths untested with the real API.
+- [H] Maintain Forgejo Actions and GitHub mirror workflows.
+- [H] Create sealed secrets from off-repository plaintext inputs.
+
+## End-to-end acceptance
+
+- [ ] Provision two clean Fedora hosts.
+- [ ] Reboot both hosts before platform installation.
+- [x] Install the cluster platform components.
+- [x] Deploy PostgreSQL and wait for health.
+- [x] Verify API liveness before database readiness.
+- [x] Verify readiness after migrations complete.
+- [x] Deploy the load generator and observe continuous samples.
+- [x] Send one request to every API endpoint in the local Compose stack.
+- [x] Verify every required metric and exact label name locally.
+- [x] Verify an HTTP trace contains a child database span.
+- [x] Correlate one stdout request log and Tempo trace by `trace_id`.
+- [x] Promote a healthy preview through analysis.
+- [x] Deploy a preview with a nonzero chaos rate.
+- [x] Observe preview errors in Prometheus.
+- [x] Confirm analysis aborts the unhealthy promotion.
+- [x] Confirm the active Service remains on the healthy version.
+- [x] Confirm load generation continues throughout failure.
+- [x] Confirm the Mini PC page remains usable with the cluster off.
+- [x] Capture reproducible, non-secret evidence needed for the final project report.