Nereus/docs/roadmap.md

# Nereus delivery roadmap

Updated: 2026-08-28

`[x]` means locally verified. `[~]` means implemented but awaiting a real
integration environment. `[ ]` means not implemented. `[H]` is human-owned and
must not be edited by agents.

Update this file in the same change that completes or materially changes a
task. Mark a task `[x]` only after its stated verification passes.

## API service

- [x] Create the Go 1.26 module and dependency lock file.
- [x] Create the idempotent `buoys` table migration.
- [x] Create the idempotent `readings` table migration.
- [x] Create `idx_readings_buoy_time` idempotently.
- [x] Parse `PORT`, `CHAOS_ERROR_RATE`, and `LOG_LEVEL`.
- [x] Preserve the build-injected `APP_VERSION` value.
- [x] Require `DATABASE_URL` without logging it.
- [x] Start HTTP without waiting for PostgreSQL.
- [x] Retry migrations while PostgreSQL is unavailable.
- [x] Keep `/healthz` independent of PostgreSQL.
- [x] Require a reachable, migrated database for `/readyz`.
- [x] Implement `GET /api/v1/buoys`.
- [x] Implement `POST /api/v1/buoys`.
- [x] Implement `GET /api/v1/buoys/{id}`.
- [x] Implement `DELETE /api/v1/buoys/{id}`.
- [x] Implement filtered, paginated `GET /api/v1/readings`.
- [x] Implement `POST /api/v1/readings`.
- [x] Implement the real grouped aggregate query.
- [x] Bound JSON request bodies and return JSON errors.
- [x] Restrict chaos injection to `/api/v1/*`.
- [x] Expose all five required Prometheus metrics.
- [x] Use chi route templates for the HTTP `path` label.
- [x] Enable OTLP/gRPC tracing only when configured.
- [x] Create HTTP spans and child database spans.
- [x] Emit structured request logs containing `trace_id`.
- [x] Add table-driven handler tests without a live database.
- [x] Test chaos and unaffected health behavior.
- [x] Add a `testing.Short()`-guarded PostgreSQL integration test.
- [x] Pass `go test -short ./...`.
- [x] Pass `go vet ./...`.
- [x] Build with `CGO_ENABLED=0`.
- [x] Run the integration test against PostgreSQL 17.
- [x] Test database failure behavior on every data endpoint.
- [x] Assert exact metric names and route-template labels in tests.
- [x] Test malformed and oversized JSON request bodies.
- [x] Run `golangci-lint run` with the CI-selected version.

## Load generator

- [x] Create the standalone Go 1.26 module.
- [x] Require `TARGET_URL` and default `RPS` to 5.
- [x] Reuse existing buoys or seed 42 validated offshore buoys.
- [x] Generate the 60/25/15 request mix.
- [x] Generate plausible synthetic reading values.
- [x] Count HTTP statuses and network errors.
- [x] Log a JSON status summary every ten seconds.
- [x] Continue traffic while requests and seeding fail.
- [x] Retry seeding without blocking read traffic.
- [x] Handle SIGTERM and SIGINT.
- [x] Keep the implementation below 200 lines.
- [x] Pass `go vet` and a static build.
- [x] Test request weighting deterministically.
- [x] Test seeding against an empty `httptest` API.
- [x] Test continued traffic through repeated HTTP 500 responses.
- [x] Run against the real API for at least five minutes.
- [x] Confirm Prometheus receives continuous request samples.

## Container build and local Compose

- [x] Create the API builder stage from `golang:1.26-alpine`.
- [x] Cache dependencies before copying API source.
- [x] Build with `CGO_ENABLED=0` and `-ldflags "-s -w"`.
- [x] Inject `APP_VERSION` with `-X`.
- [x] Copy only the API binary into `gcr.io/distroless/static:nonroot`.
- [x] Create the load-generator multi-stage image.
- [x] Use `gcr.io/distroless/static:nonroot` for the load generator.
- [x] Add the required `.dockerignore` exclusions.
- [x] Confirm both images run non-root and read-only.
- [x] Confirm the API image is below 25 MB (7,815,877 bytes locally).
- [x] Add API, PostgreSQL 17, and OTEL Collector to local Compose.
- [x] Add the PostgreSQL healthcheck.
- [x] Make the API wait on healthy PostgreSQL in Compose.
- [x] Verify `docker compose up` reaches API readiness.
- [x] Keep Mini PC and local-development Compose files separate.

## Static web dashboard

- [x] Create the no-build static HTML entry point.
- [x] Create the dark teal/navy glass visual system.
- [x] Replace decorative background waves with an operational chart grid.
- [x] Keep version, health, and error panels visible together.
- [x] Show the build-injected API version from the existing health response header.
- [x] Poll `/healthz` every two seconds.
- [x] Poll `/readyz` every two seconds.
- [x] Show each health indicator's last transition timestamp.
- [x] Count non-2xx `/api/v1/*` browser responses.
- [x] Calculate the rolling 60-second HTTP error rate.
- [x] Distinguish network-offline events from HTTP errors.
- [x] Preserve and grey the last known telemetry values offline.
- [x] Recover automatically when the API returns.
- [x] Add a fully local Natural Earth buoy map and pulsing markers.
- [x] Add animated reading charts.
- [x] Keep all API calls same-origin and relative.
- [x] Remove all frontend runtime dependencies on external hosts.
- [x] Verify all panels fit at 1280×720 without page scrolling.
- [x] Create the unprivileged nginx image.
- [x] Create the separate Mini PC Traefik Compose file.
- [x] Test the finished offline/degraded rendering without API connectivity.
- [x] Test the rendered dashboard while the API returns HTTP 500 responses.

## Observability

- [x] Receive OTLP/gRPC traces in the Collector.
- [x] Export traces from the Collector to Tempo.
- [x] Ship structured Kubernetes pod logs to Loki with the cluster Collector.
- [x] Start Loki locally and verify its Grafana data source readiness.
- [x] Configure Prometheus to scrape `/metrics`.
- [x] Add request-rate, error-rate, and latency panels.
- [x] Add database-operation latency panels.
- [x] Add readings-ingested and active-buoy panels.
- [x] Add a deployed-version dashboard variable.
- [x] Correlate logs and traces using `trace_id` after cluster log ingestion exists.
- [x] Add API error-rate, latency, readiness, and scrape alerts.
- [~] Route warning and critical alerts to Discord; configuration validates and awaits the owner-provided sealed webhook.
- [x] Load the dashboard and evaluate all six PromQL queries.
- [x] Trigger chaos and distinguish the failing version in Grafana.

## Fedora 44 host provisioning

- [x] Separate node 1 and node 2 inventory groups.
- [x] Pin the project k3s version.
- [x] Reject managed hosts that are not Fedora 44.
- [x] Install common host dependencies.
- [x] Enable firewalld.
- [x] Trust the configured pod and service CIDRs.
- [x] Open VXLAN port 8472/UDP between nodes.
- [x] Open kubelet port 10250/TCP between nodes.
- [x] Open API server port 6443/TCP on node 1.
- [x] Install node 1 as the k3s server.
- [x] Install node 2 as a k3s agent.
- [x] Keep the cluster token out of repository files and output.
- [x] Enable both k3s systemd services.
- [x] Wait for every Kubernetes node to become Ready.
- [x] Add the equivalent manual configuration checklist.
- [x] Document Terraform-to-Ansible inventory handoff.
- [x] Keep libvirt addresses, bridge rules, and VM network profiles out of the reusable Ansible roles.
- [x] Run the playbook twice on two clean Fedora 44 QEMU machines.
- [x] Confirm the second run reports no unintended changes (`changed=0`).
- [x] Restrict firewall sources to actual node/operator networks.
- [x] Confirm SELinux remains enforcing on both QEMU machines.
- [x] Reboot both QEMU machines and confirm automatic recovery.
- [x] Disconnect and reconnect node 2, then confirm it rejoins.
- [x] Record tested minimum CPU, memory, and disk requirements.

## One-command QEMU lab

- [x] Check QEMU, libvirt, cloud-utils, Ansible, SSH, ping, and KVM.
- [x] Install missing dependencies on a Fedora workstation.
- [x] Enable libvirt and its default network.
- [x] Discover and cache the Fedora 44 x86_64 cloud image.
- [x] Keep SSH and VM state outside the repository.
- [x] Generate cloud-init media for both nodes.
- [x] Create reusable copy-on-write disks.
- [x] Create each VM with two vCPUs and 3 GiB RAM.
- [x] Assign deterministic addresses to both local VMs.
- [x] Wait for ICMP ping and SSH on both nodes.
- [x] Generate the Ansible inventory automatically.
- [x] Configure both nodes automatically.
- [x] Check Ansible reachability, k3s files, and binaries.
- [x] Check firewalld, k3s services, ports 6443 and 10250.
- [x] Check Kubernetes node readiness.
- [x] Add `check`, `stop`, and explicit `destroy` actions.
- [x] Execute `lab.sh up` with working `/dev/kvm` and pass the full verification.
- [x] Reboot both local VMs and pass the bounded post-reboot service and node checks.
- [x] Execute `lab.sh check` after a workstation reboot.
- [x] Test recovery from an interrupted image download and validate the resumed QCOW2.
- [x] Test recovery after node 2 is deleted manually and rejoin it with a fresh identity.
- [~] Test on a workstation with no dependencies installed; the reproducible fresh-VM path is implemented and awaits a second host.
- [x] Decide whether an aarch64 image path is required; keep the lab x86_64-only.
- [ ] After final sign-off, destroy the QEMU lab and remove packages installed only for it with Pacman's dependency-aware cleanup.

## Local k3d rollback mechanism

- [x] Define one local server and two local agent nodes.
- [x] Map Traefik to host ports 8080 and 8443.
- [x] Add lean kube-prometheus-stack values.
- [x] Add the synthetic Argo Rollouts analysis harness.
- [x] Add an idempotent `up/check/destroy` command and verify fresh creation and reconciliation.
- [x] Install Argo Rollouts.
- [x] Install kube-prometheus-stack with the local values.
- [x] Apply the synthetic harness.
- [x] Prove `vector(0.0)` permits promotion.
- [x] Change the test value to `vector(1.0)`.
- [x] Prove failed analysis aborts promotion with `vector(0.9)`.
- [x] Tear down and recreate the test from scratch.

## Human-owned deployment and infrastructure

- [x] Define libvirt infrastructure and Helm platform stacks under `terraform/`.
- [x] Export stable node addresses to an Ansible inventory without provisioner hooks.
- [H] Create Kubernetes base resources and overlays. Agent-authored 2026-08-21 at the owner's request; builds and server dry-runs pass, not applied.
- [H] Create API and load-generator workloads. Agent-authored 2026-08-21; not applied.
- [H] Create active and preview Services. Agent-authored 2026-08-21; not applied.
- [H] Create the Prometheus ServiceMonitor. Agent-authored 2026-08-21; not applied, scrape not confirmed.
- [H] Create the real error-rate AnalysisTemplate. Agent-authored 2026-08-21; PromQL never evaluated against real series.
- [H] Configure blue-green promotion and automated rollback. Agent-authored 2026-08-21; promotion and abort paths untested with the real API.
- [H] Maintain Forgejo Actions and GitHub mirror workflows.
- [x] Install Sealed Secrets, commit the encrypted database resource, and verify the fresh-volume cutover.

## Forgejo CI/CD

- [x] Run API and load-generator race tests, vet, and pinned lint in containers.
- [x] Scan committed history and the working tree for secrets.
- [x] Scan release configuration and images for high or critical findings.
- [x] Build immutable commit-SHA API, load-generator, and web image tags.
- [x] Keep active and preview traffic flowing during rollout analysis.
- [x] Run the Forgejo workflow after a private `docker` runner and encrypted secrets are configured.
- [x] Publish images and execute the first automated production rollout.

## End-to-end acceptance

- [ ] Provision two clean Fedora hosts.
- [ ] Reboot both hosts before platform installation.
- [x] Install the cluster platform components.
- [x] Deploy PostgreSQL and wait for health.
- [x] Verify API liveness before database readiness.
- [x] Verify readiness after migrations complete.
- [x] Deploy the load generator and observe continuous samples.
- [x] Send one request to every API endpoint in the local Compose stack.
- [x] Verify every required metric and exact label name locally.
- [x] Verify an HTTP trace contains a child database span.
- [x] Correlate one stdout request log and Tempo trace by `trace_id`.
- [x] Promote a healthy preview through analysis.
- [x] Deploy a preview with a nonzero chaos rate.
- [x] Observe preview errors in Prometheus.
- [x] Confirm analysis aborts the unhealthy promotion.
- [x] Confirm the active Service remains on the healthy version.
- [x] Confirm load generation continues throughout failure.
- [x] Confirm the Mini PC page remains usable with the cluster off.
- [x] Capture reproducible, non-secret evidence needed for the final project report.