Nereus/docs/roadmap.md

Nereus delivery roadmap

Updated: 2026-08-26

[x] means locally verified. [~] means implemented but awaiting a real integration environment. [ ] means not implemented. [H] is human-owned and must not be edited by agents.

Update this file in the same change that completes or materially changes a task. Mark a task [x] only after its stated verification passes.

API service

• Create the Go 1.26 module and dependency lock file.
• Create the idempotent buoys table migration.
• Create the idempotent readings table migration.
• Create idx_readings_buoy_time idempotently.
• Parse PORT, CHAOS_ERROR_RATE, and LOG_LEVEL.
• Preserve the build-injected APP_VERSION value.
• Require DATABASE_URL without logging it.
• Start HTTP without waiting for PostgreSQL.
• Retry migrations while PostgreSQL is unavailable.
• Keep /healthz independent of PostgreSQL.
• Require a reachable, migrated database for /readyz.
• Implement GET /api/v1/buoys.
• Implement POST /api/v1/buoys.
• Implement GET /api/v1/buoys/{id}.
• Implement DELETE /api/v1/buoys/{id}.
• Implement filtered, paginated GET /api/v1/readings.
• Implement POST /api/v1/readings.
• Implement the real grouped aggregate query.
• Bound JSON request bodies and return JSON errors.
• Restrict chaos injection to /api/v1/*.
• Expose all five required Prometheus metrics.
• Use chi route templates for the HTTP path label.
• Enable OTLP/gRPC tracing only when configured.
• Create HTTP spans and child database spans.
• Emit structured request logs containing trace_id.
• Add table-driven handler tests without a live database.
• Test chaos and unaffected health behavior.
• Add a testing.Short()-guarded PostgreSQL integration test.
• Pass go test -short ./....
• Pass go vet ./....
• Build with CGO_ENABLED=0.
• Run the integration test against PostgreSQL 17.
• Test database failure behavior on every data endpoint.
• Assert exact metric names and route-template labels in tests.
• Test malformed and oversized JSON request bodies.
• Run golangci-lint run with the CI-selected version.

Load generator

• Create the standalone Go 1.26 module.
• Require TARGET_URL and default RPS to 5.
• Reuse existing buoys or seed 42 validated offshore buoys.
• Generate the 60/25/15 request mix.
• Generate plausible synthetic reading values.
• Count HTTP statuses and network errors.
• Log a JSON status summary every ten seconds.
• Continue traffic while requests and seeding fail.
• Retry seeding without blocking read traffic.
• Handle SIGTERM and SIGINT.
• Keep the implementation below 200 lines.
• Pass go vet and a static build.
• Test request weighting deterministically.
• Test seeding against an empty httptest API.
• Test continued traffic through repeated HTTP 500 responses.
• Run against the real API for at least five minutes.
• Confirm Prometheus receives continuous request samples.

Container build and local Compose

• Create the API builder stage from golang:1.26-alpine.
• Cache dependencies before copying API source.
• Build with CGO_ENABLED=0 and -ldflags "-s -w".
• Inject APP_VERSION with -X.
• Copy only the API binary into gcr.io/distroless/static:nonroot.
• Create the load-generator multi-stage image.
• Use gcr.io/distroless/static:nonroot for the load generator.
• Add the required .dockerignore exclusions.
• Confirm both images run non-root and read-only.
• Confirm the API image is below 25 MB (7,815,877 bytes locally).
• Add API, PostgreSQL 17, and OTEL Collector to local Compose.
• Add the PostgreSQL healthcheck.
• Make the API wait on healthy PostgreSQL in Compose.
• Verify docker compose up reaches API readiness.
• Keep Mini PC and local-development Compose files separate.

Static web dashboard

• Create the no-build static HTML entry point.
• Create the dark teal/navy glass visual system.
• Replace decorative background waves with an operational chart grid.
• Keep version, health, and error panels visible together.
• Show the build-injected API version from the existing health response header.
• Poll /healthz every two seconds.
• Poll /readyz every two seconds.
• Show each health indicator's last transition timestamp.
• Count non-2xx /api/v1/* browser responses.
• Calculate the rolling 60-second HTTP error rate.
• Distinguish network-offline events from HTTP errors.
• Preserve and grey the last known telemetry values offline.
• Recover automatically when the API returns.
• Add a fully local Natural Earth buoy map and pulsing markers.
• Add animated reading charts.
• Keep all API calls same-origin and relative.
• Remove all frontend runtime dependencies on external hosts.
• Verify all panels fit at 1280×720 without page scrolling.
• Create the unprivileged nginx image.
• Create the separate Mini PC Traefik Compose file.
• Test the finished offline/degraded rendering without API connectivity.
• Test the rendered dashboard while the API returns HTTP 500 responses.

Observability

• Receive OTLP/gRPC traces in the Collector.
• Export traces from the Collector to Tempo.
• [~] Ship structured Kubernetes pod logs to Loki with the cluster Collector.
• Start Loki locally and verify its Grafana data source readiness.
• Configure Prometheus to scrape /metrics.
• Add request-rate, error-rate, and latency panels.
• Add database-operation latency panels.
• Add readings-ingested and active-buoy panels.
• Add a deployed-version dashboard variable.
• [~] Correlate logs and traces using trace_id after cluster log ingestion exists.
• Add API error-rate, latency, readiness, and scrape alerts.
• Load the dashboard and evaluate all six PromQL queries.
• Trigger chaos and distinguish the failing version in Grafana.

Fedora 44 host provisioning

• Separate node 1 and node 2 inventory groups.
• Pin the project k3s version.
• Reject managed hosts that are not Fedora 44.
• Install common host dependencies.
• Enable firewalld.
• Trust the configured pod and service CIDRs.
• Open VXLAN port 8472/UDP between nodes.
• Open kubelet port 10250/TCP between nodes.
• Open API server port 6443/TCP on node 1.
• Install node 1 as the k3s server.
• Install node 2 as a k3s agent.
• Keep the cluster token out of repository files and output.
• Enable both k3s systemd services.
• Wait for every Kubernetes node to become Ready.
• Add the equivalent manual configuration checklist.
• Document Terraform-to-Ansible inventory handoff.
• Keep libvirt addresses, bridge rules, and VM network profiles out of the reusable Ansible roles.
• Run the playbook twice on two clean Fedora 44 QEMU machines.
• Confirm the second run reports no unintended changes (changed=0).
• Restrict firewall sources to actual node/operator networks.
• Confirm SELinux remains enforcing on both QEMU machines.
• Reboot both QEMU machines and confirm automatic recovery.
• Disconnect and reconnect node 2, then confirm it rejoins.
• Record tested minimum CPU, memory, and disk requirements.

One-command QEMU lab

• Check QEMU, libvirt, cloud-utils, Ansible, SSH, ping, and KVM.
• Install missing dependencies on a Fedora workstation.
• Enable libvirt and its default network.
• Discover and cache the Fedora 44 x86_64 cloud image.
• Keep SSH and VM state outside the repository.
• Generate cloud-init media for both nodes.
• Create reusable copy-on-write disks.
• Create each VM with two vCPUs and 3 GiB RAM.
• Assign deterministic addresses to both local VMs.
• Wait for ICMP ping and SSH on both nodes.
• Generate the Ansible inventory automatically.
• Configure both nodes automatically.
• Check Ansible reachability, k3s files, and binaries.
• Check firewalld, k3s services, ports 6443 and 10250.
• Check Kubernetes node readiness.
• Add check, stop, and explicit destroy actions.
• Execute lab.sh up with working /dev/kvm and pass the full verification.
• Reboot both local VMs and pass the bounded post-reboot service and node checks.
• Execute lab.sh check after a workstation reboot.
• Test recovery from an interrupted image download and validate the resumed QCOW2.
• Test recovery after node 2 is deleted manually and rejoin it with a fresh identity.
• Test on a workstation with no dependencies installed.
• Decide whether an aarch64 image path is required; keep the lab x86_64-only.
• After final sign-off, destroy the QEMU lab and remove packages installed only for it with Pacman's dependency-aware cleanup.

Local k3d rollback mechanism

• Define one local server and two local agent nodes.
• Map Traefik to host ports 8080 and 8443.
• Add lean kube-prometheus-stack values.
• Add the synthetic Argo Rollouts analysis harness.
• Add an idempotent up/check/destroy command and verify fresh creation and reconciliation.
• Install Argo Rollouts.
• Install kube-prometheus-stack with the local values.
• Apply the synthetic harness.
• Prove vector(0.0) permits promotion.
• Change the test value to vector(1.0).
• Prove failed analysis aborts promotion with vector(0.9).
• Tear down and recreate the test from scratch.

Human-owned deployment and infrastructure

• [H] Define infrastructure modules under terraform/.
• [H] Export provisioned addresses for Ansible inventory.
• [H] Create Kubernetes base resources and overlays. Agent-authored 2026-08-21 at the owner's request; builds and server dry-runs pass, not applied.
• [H] Create API and load-generator workloads. Agent-authored 2026-08-21; not applied.
• [H] Create active and preview Services. Agent-authored 2026-08-21; not applied.
• [H] Create the Prometheus ServiceMonitor. Agent-authored 2026-08-21; not applied, scrape not confirmed.
• [H] Create the real error-rate AnalysisTemplate. Agent-authored 2026-08-21; PromQL never evaluated against real series.
• [H] Configure blue-green promotion and automated rollback. Agent-authored 2026-08-21; promotion and abort paths untested with the real API.
• [H] Maintain Forgejo Actions and GitHub mirror workflows.
• [H] Create sealed secrets from off-repository plaintext inputs.

Forgejo CI/CD

• Run API and load-generator race tests, vet, and pinned lint in containers.
• Scan committed history and the working tree for secrets.
• Scan release configuration and images for high or critical findings.
• Build immutable commit-SHA API, load-generator, and web image tags.
• Keep active and preview traffic flowing during rollout analysis.
• Run the Forgejo workflow after a private docker runner and encrypted secrets are configured.
• Publish images and execute the first automated production rollout.

End-to-end acceptance

• Provision two clean Fedora hosts.
• Reboot both hosts before platform installation.
• Install the cluster platform components.
• Deploy PostgreSQL and wait for health.
• Verify API liveness before database readiness.
• Verify readiness after migrations complete.
• Deploy the load generator and observe continuous samples.
• Send one request to every API endpoint in the local Compose stack.
• Verify every required metric and exact label name locally.
• Verify an HTTP trace contains a child database span.
• Correlate one stdout request log and Tempo trace by trace_id.
• Promote a healthy preview through analysis.
• Deploy a preview with a nonzero chaos rate.
• Observe preview errors in Prometheus.
• Confirm analysis aborts the unhealthy promotion.
• Confirm the active Service remains on the healthy version.
• Confirm load generation continues throughout failure.
• Confirm the Mini PC page remains usable with the cluster off.
• Capture reproducible, non-secret evidence needed for the final project report.