diff --git a/docs/evidence.md b/docs/evidence.md new file mode 100644 index 0000000..2ae05f2 --- /dev/null +++ b/docs/evidence.md @@ -0,0 +1,68 @@ +# Final project verification evidence + +Captured on 2026-08-24 from the disposable QEMU and k3d integration labs. The +commands below expose no credentials and can be rerun before the presentation +to refresh screenshots or terminal recordings. + +## Two-node Fedora cluster + +`scripts/provision/lab.sh check` passed after provisioning, host reboot, node 2 +disconnect/reconnect, and complete deletion/recreation of node 2. At the final +check both Fedora 44 machines ran k3s `v1.33.4+k3s1` and reported `Ready`: + +```text +nereus-node1 Ready control-plane,master 192.168.122.10 +nereus-node2 Ready 192.168.122.11 +``` + +The node-replacement test found and fixed two recovery requirements: remove the +replacement address from the private SSH known-hosts file, and delete the old +Kubernetes node identity before the new agent joins. The PostgreSQL local-path +volume was correctly treated as lost with the deleted VM, recreated empty, and +reseeded by the load generator. + +## Live application + +The final namespace check showed three ready API pods, one ready load-generator +pod, and one ready PostgreSQL pod. The same-origin Mini PC route returned: + +```text +GET / 200 +GET /healthz 200 +GET /readyz 200 +X-Nereus-Version 0.1.0 +GET /api/v1/buoys 42 records +``` + +The static dashboard stayed available while the cluster API was unavailable +during node recovery, then recovered without a page reload. + +## Container evidence + +The locally inspected `nereus-api:0.1.0` image is 7,815,732 bytes and declares +numeric non-root user `65532`. The container verification also passed with a +read-only root filesystem. + +## Rollback evidence + +The local k3d lab was destroyed and recreated from scratch. Re-running +`scripts/k3d/lab.sh up` reconciled the running cluster successfully. The proof +produced these AnalysisRun results: + +```text +probe-746bbb94df-2-pre Successful vector(0.0) +probe-556d5b659b-3-pre Failed vector(1.0) +``` + +After the failed analysis, the active Service selector remained on the healthy +revision `746bbb94df`. + +## Presentation commands + +```bash +scripts/provision/lab.sh check +curl -i https://nereus.fiwlabs.dev/healthz +curl -i https://nereus.fiwlabs.dev/readyz +curl -fsS https://nereus.fiwlabs.dev/api/v1/buoys | jq length +scripts/k3d/lab.sh prove +``` diff --git a/docs/mini-pc.md b/docs/mini-pc.md new file mode 100644 index 0000000..f3d1099 --- /dev/null +++ b/docs/mini-pc.md @@ -0,0 +1,31 @@ +# Mini PC web stack + +The Mini PC stack joins the host's existing Traefik `proxy` network and proxies +the frozen API paths through nginx. It does not run its own Traefik instance, +PostgreSQL, the API, or the load generator. + +Export the following values in the operator shell or service manager: + +```text +WEB_HOST public dashboard hostname; defaults to nereus.fiwlabs.dev +CLUSTER_API_UPSTREAM cluster address and port, without a URL scheme +WEB_ASSET_PATH asset path relative to the build context; defaults to apps/web +``` + +Do not put these values in a repository `.env` file. Start the isolated stack +from the repository root: + +```sh +docker compose -f build/compose.mini.yaml up -d --build +``` + +The Mini PC keeps the static assets directly in `~/Server/web/nereus`. Copy the +three build files into its `build/` directory, set `WEB_ASSET_PATH=.` when +building there, and join the existing external `proxy` network. The existing +Traefik instance supplies the `websecure` entrypoint and `porkbun` certificate +resolver; do not start a second proxy on ports 80 and 443. + +When the cluster is unavailable, nginx continues serving the dashboard and +returns a gateway failure only for `/api/v1/*`, `/healthz`, and `/readyz`. The +browser keeps polling those relative paths and recovers when the cluster is +reachable again. diff --git a/docs/roadmap.md b/docs/roadmap.md new file mode 100644 index 0000000..033b865 --- /dev/null +++ b/docs/roadmap.md @@ -0,0 +1,232 @@ +# Nereus delivery roadmap + +Updated: 2026-08-24 + +`[x]` means locally verified. `[~]` means implemented but awaiting a real +integration environment. `[ ]` means not implemented. `[H]` is human-owned and +must not be edited by agents. + +Update this file in the same change that completes or materially changes a +task. Mark a task `[x]` only after its stated verification passes. + +## API service + +- [x] Create the Go 1.26 module and dependency lock file. +- [x] Create the idempotent `buoys` table migration. +- [x] Create the idempotent `readings` table migration. +- [x] Create `idx_readings_buoy_time` idempotently. +- [x] Parse `PORT`, `CHAOS_ERROR_RATE`, and `LOG_LEVEL`. +- [x] Preserve the build-injected `APP_VERSION` value. +- [x] Require `DATABASE_URL` without logging it. +- [x] Start HTTP without waiting for PostgreSQL. +- [x] Retry migrations while PostgreSQL is unavailable. +- [x] Keep `/healthz` independent of PostgreSQL. +- [x] Require a reachable, migrated database for `/readyz`. +- [x] Implement `GET /api/v1/buoys`. +- [x] Implement `POST /api/v1/buoys`. +- [x] Implement `GET /api/v1/buoys/{id}`. +- [x] Implement `DELETE /api/v1/buoys/{id}`. +- [x] Implement filtered, paginated `GET /api/v1/readings`. +- [x] Implement `POST /api/v1/readings`. +- [x] Implement the real grouped aggregate query. +- [x] Bound JSON request bodies and return JSON errors. +- [x] Restrict chaos injection to `/api/v1/*`. +- [x] Expose all five required Prometheus metrics. +- [x] Use chi route templates for the HTTP `path` label. +- [x] Enable OTLP/gRPC tracing only when configured. +- [x] Create HTTP spans and child database spans. +- [x] Emit structured request logs containing `trace_id`. +- [x] Add table-driven handler tests without a live database. +- [x] Test chaos and unaffected health behavior. +- [x] Add a `testing.Short()`-guarded PostgreSQL integration test. +- [x] Pass `go test -short ./...`. +- [x] Pass `go vet ./...`. +- [x] Build with `CGO_ENABLED=0`. +- [x] Run the integration test against PostgreSQL 17. +- [x] Test database failure behavior on every data endpoint. +- [x] Assert exact metric names and route-template labels in tests. +- [x] Test malformed and oversized JSON request bodies. +- [ ] Run `golangci-lint run` with the CI-selected version. + +## Load generator + +- [x] Create the standalone Go 1.26 module. +- [x] Require `TARGET_URL` and default `RPS` to 5. +- [x] Reuse existing buoys or seed 42 validated offshore buoys. +- [x] Generate the 60/25/15 request mix. +- [x] Generate plausible synthetic reading values. +- [x] Count HTTP statuses and network errors. +- [x] Log a JSON status summary every ten seconds. +- [x] Continue traffic while requests and seeding fail. +- [x] Retry seeding without blocking read traffic. +- [x] Handle SIGTERM and SIGINT. +- [x] Keep the implementation below 200 lines. +- [x] Pass `go vet` and a static build. +- [x] Test request weighting deterministically. +- [x] Test seeding against an empty `httptest` API. +- [x] Test continued traffic through repeated HTTP 500 responses. +- [x] Run against the real API for at least five minutes. +- [x] Confirm Prometheus receives continuous request samples. + +## Container build and local Compose + +- [x] Create the API builder stage from `golang:1.26-alpine`. +- [x] Cache dependencies before copying API source. +- [x] Build with `CGO_ENABLED=0` and `-ldflags "-s -w"`. +- [x] Inject `APP_VERSION` with `-X`. +- [x] Copy only the API binary into `gcr.io/distroless/static:nonroot`. +- [x] Create the load-generator multi-stage image. +- [x] Use `gcr.io/distroless/static:nonroot` for the load generator. +- [x] Add the required `.dockerignore` exclusions. +- [x] Confirm both images run non-root and read-only. +- [x] Confirm the API image is below 25 MB (7,815,877 bytes locally). +- [x] Add API, PostgreSQL 17, and OTEL Collector to local Compose. +- [x] Add the PostgreSQL healthcheck. +- [x] Make the API wait on healthy PostgreSQL in Compose. +- [x] Verify `docker compose up` reaches API readiness. +- [x] Keep Mini PC and local-development Compose files separate. + +## Static web dashboard + +- [x] Create the no-build static HTML entry point. +- [x] Create the dark teal/navy glass visual system. +- [x] Replace decorative background waves with an operational chart grid. +- [x] Keep version, health, and error panels visible together. +- [x] Show the build-injected API version from the existing health response header. +- [x] Poll `/healthz` every two seconds. +- [x] Poll `/readyz` every two seconds. +- [x] Show each health indicator's last transition timestamp. +- [x] Count non-2xx `/api/v1/*` browser responses. +- [x] Calculate the rolling 60-second HTTP error rate. +- [x] Distinguish network-offline events from HTTP errors. +- [x] Preserve and grey the last known telemetry values offline. +- [x] Recover automatically when the API returns. +- [x] Add a fully local Natural Earth buoy map and pulsing markers. +- [x] Add animated reading charts. +- [x] Keep all API calls same-origin and relative. +- [x] Remove all frontend runtime dependencies on external hosts. +- [x] Verify all panels fit at 1280×720 without page scrolling. +- [x] Create the unprivileged nginx image. +- [x] Create the separate Mini PC Traefik Compose file. +- [x] Test the finished offline/degraded rendering without API connectivity. +- [x] Test the rendered dashboard while the API returns HTTP 500 responses. + +## Observability + +- [x] Receive OTLP/gRPC traces in the Collector. +- [x] Export traces from the Collector to Tempo. +- [~] Ship structured Kubernetes pod logs to Loki with the cluster Collector. +- [x] Start Loki locally and verify its Grafana data source readiness. +- [x] Configure Prometheus to scrape `/metrics`. +- [x] Add request-rate, error-rate, and latency panels. +- [x] Add database-operation latency panels. +- [x] Add readings-ingested and active-buoy panels. +- [x] Add a deployed-version dashboard variable. +- [~] Correlate logs and traces using `trace_id` after cluster log ingestion exists. +- [x] Add API error-rate, latency, readiness, and scrape alerts. +- [x] Load the dashboard and evaluate all six PromQL queries. +- [x] Trigger chaos and distinguish the failing version in Grafana. + +## Fedora 44 host provisioning + +- [x] Separate node 1 and node 2 inventory groups. +- [x] Pin the project k3s version. +- [x] Reject managed hosts that are not Fedora 44. +- [x] Install common host dependencies. +- [x] Enable firewalld. +- [x] Trust the configured pod and service CIDRs. +- [x] Open VXLAN port 8472/UDP between nodes. +- [x] Open kubelet port 10250/TCP between nodes. +- [x] Open API server port 6443/TCP on node 1. +- [x] Install node 1 as the k3s server. +- [x] Install node 2 as a k3s agent. +- [x] Keep the cluster token out of repository files and output. +- [x] Enable both k3s systemd services. +- [x] Wait for every Kubernetes node to become Ready. +- [x] Add the equivalent manual configuration checklist. +- [x] Document Terraform-to-Ansible inventory handoff. +- [x] Keep libvirt addresses, bridge rules, and VM network profiles out of the reusable Ansible roles. +- [x] Run the playbook twice on two clean Fedora 44 QEMU machines. +- [x] Confirm the second run reports no unintended changes (`changed=0`). +- [x] Restrict firewall sources to actual node/operator networks. +- [x] Confirm SELinux remains enforcing on both QEMU machines. +- [x] Reboot both QEMU machines and confirm automatic recovery. +- [x] Disconnect and reconnect node 2, then confirm it rejoins. +- [x] Record tested minimum CPU, memory, and disk requirements. + +## One-command QEMU lab + +- [x] Check QEMU, libvirt, cloud-utils, Ansible, SSH, ping, and KVM. +- [x] Install missing dependencies on a Fedora workstation. +- [x] Enable libvirt and its default network. +- [x] Discover and cache the Fedora 44 x86_64 cloud image. +- [x] Keep SSH and VM state outside the repository. +- [x] Generate cloud-init media for both nodes. +- [x] Create reusable copy-on-write disks. +- [x] Create each VM with two vCPUs and 3 GiB RAM. +- [x] Assign deterministic addresses to both local VMs. +- [x] Wait for ICMP ping and SSH on both nodes. +- [x] Generate the Ansible inventory automatically. +- [x] Configure both nodes automatically. +- [x] Check Ansible reachability, k3s files, and binaries. +- [x] Check firewalld, k3s services, ports 6443 and 10250. +- [x] Check Kubernetes node readiness. +- [x] Add `check`, `stop`, and explicit `destroy` actions. +- [x] Execute `lab.sh up` with working `/dev/kvm` and pass the full verification. +- [x] Reboot both local VMs and pass the bounded post-reboot service and node checks. +- [x] Execute `lab.sh check` after a workstation reboot. +- [x] Test recovery from an interrupted image download and validate the resumed QCOW2. +- [x] Test recovery after node 2 is deleted manually and rejoin it with a fresh identity. +- [ ] Test on a workstation with no dependencies installed. +- [x] Decide whether an aarch64 image path is required; keep the lab x86_64-only. +- [ ] After final sign-off, destroy the QEMU lab and remove packages installed only for it with Pacman's dependency-aware cleanup. + +## Local k3d rollback mechanism + +- [x] Define one local server and two local agent nodes. +- [x] Map Traefik to host ports 8080 and 8443. +- [x] Add lean kube-prometheus-stack values. +- [x] Add the synthetic Argo Rollouts analysis harness. +- [x] Add an idempotent `up/check/destroy` command and verify fresh creation and reconciliation. +- [x] Install Argo Rollouts. +- [x] Install kube-prometheus-stack with the local values. +- [x] Apply the synthetic harness. +- [x] Prove `vector(0.0)` permits promotion. +- [x] Change the test value to `vector(1.0)`. +- [x] Prove failed analysis aborts promotion with `vector(0.9)`. +- [x] Tear down and recreate the test from scratch. + +## Human-owned deployment and infrastructure + +- [H] Define infrastructure modules under `terraform/`. +- [H] Export provisioned addresses for Ansible inventory. +- [H] Create Kubernetes base resources and overlays. Agent-authored 2026-08-21 at the owner's request; builds and server dry-runs pass, not applied. +- [H] Create API and load-generator workloads. Agent-authored 2026-08-21; not applied. +- [H] Create active and preview Services. Agent-authored 2026-08-21; not applied. +- [H] Create the Prometheus ServiceMonitor. Agent-authored 2026-08-21; not applied, scrape not confirmed. +- [H] Create the real error-rate AnalysisTemplate. Agent-authored 2026-08-21; PromQL never evaluated against real series. +- [H] Configure blue-green promotion and automated rollback. Agent-authored 2026-08-21; promotion and abort paths untested with the real API. +- [H] Maintain Forgejo Actions and GitHub mirror workflows. +- [H] Create sealed secrets from off-repository plaintext inputs. + +## End-to-end acceptance + +- [ ] Provision two clean Fedora hosts. +- [ ] Reboot both hosts before platform installation. +- [x] Install the cluster platform components. +- [x] Deploy PostgreSQL and wait for health. +- [x] Verify API liveness before database readiness. +- [x] Verify readiness after migrations complete. +- [x] Deploy the load generator and observe continuous samples. +- [x] Send one request to every API endpoint in the local Compose stack. +- [x] Verify every required metric and exact label name locally. +- [x] Verify an HTTP trace contains a child database span. +- [x] Correlate one stdout request log and Tempo trace by `trace_id`. +- [x] Promote a healthy preview through analysis. +- [x] Deploy a preview with a nonzero chaos rate. +- [x] Observe preview errors in Prometheus. +- [x] Confirm analysis aborts the unhealthy promotion. +- [x] Confirm the active Service remains on the healthy version. +- [x] Confirm load generation continues throughout failure. +- [x] Confirm the Mini PC page remains usable with the cluster off. +- [x] Capture reproducible, non-secret evidence needed for the final project report.