fiwdev/Nereus
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
Nereus/docs/roadmap.md

12 KiB
Raw Blame History

Nereus delivery roadmap

Updated: 2026-08-24

[x] means locally verified. [~] means implemented but awaiting a real integration environment. [ ] means not implemented. [H] is human-owned and must not be edited by agents.

Update this file in the same change that completes or materially changes a task. Mark a task [x] only after its stated verification passes.

API service

  • Create the Go 1.26 module and dependency lock file.
  • Create the idempotent buoys table migration.
  • Create the idempotent readings table migration.
  • Create idx_readings_buoy_time idempotently.
  • Parse PORT, CHAOS_ERROR_RATE, and LOG_LEVEL.
  • Preserve the build-injected APP_VERSION value.
  • Require DATABASE_URL without logging it.
  • Start HTTP without waiting for PostgreSQL.
  • Retry migrations while PostgreSQL is unavailable.
  • Keep /healthz independent of PostgreSQL.
  • Require a reachable, migrated database for /readyz.
  • Implement GET /api/v1/buoys.
  • Implement POST /api/v1/buoys.
  • Implement GET /api/v1/buoys/{id}.
  • Implement DELETE /api/v1/buoys/{id}.
  • Implement filtered, paginated GET /api/v1/readings.
  • Implement POST /api/v1/readings.
  • Implement the real grouped aggregate query.
  • Bound JSON request bodies and return JSON errors.
  • Restrict chaos injection to /api/v1/*.
  • Expose all five required Prometheus metrics.
  • Use chi route templates for the HTTP path label.
  • Enable OTLP/gRPC tracing only when configured.
  • Create HTTP spans and child database spans.
  • Emit structured request logs containing trace_id.
  • Add table-driven handler tests without a live database.
  • Test chaos and unaffected health behavior.
  • Add a testing.Short()-guarded PostgreSQL integration test.
  • Pass go test -short ./....
  • Pass go vet ./....
  • Build with CGO_ENABLED=0.
  • Run the integration test against PostgreSQL 17.
  • Test database failure behavior on every data endpoint.
  • Assert exact metric names and route-template labels in tests.
  • Test malformed and oversized JSON request bodies.
  • Run golangci-lint run with the CI-selected version.

Load generator

  • Create the standalone Go 1.26 module.
  • Require TARGET_URL and default RPS to 5.
  • Reuse existing buoys or seed 42 validated offshore buoys.
  • Generate the 60/25/15 request mix.
  • Generate plausible synthetic reading values.
  • Count HTTP statuses and network errors.
  • Log a JSON status summary every ten seconds.
  • Continue traffic while requests and seeding fail.
  • Retry seeding without blocking read traffic.
  • Handle SIGTERM and SIGINT.
  • Keep the implementation below 200 lines.
  • Pass go vet and a static build.
  • Test request weighting deterministically.
  • Test seeding against an empty httptest API.
  • Test continued traffic through repeated HTTP 500 responses.
  • Run against the real API for at least five minutes.
  • Confirm Prometheus receives continuous request samples.

Container build and local Compose

  • Create the API builder stage from golang:1.26-alpine.
  • Cache dependencies before copying API source.
  • Build with CGO_ENABLED=0 and -ldflags "-s -w".
  • Inject APP_VERSION with -X.
  • Copy only the API binary into gcr.io/distroless/static:nonroot.
  • Create the load-generator multi-stage image.
  • Use gcr.io/distroless/static:nonroot for the load generator.
  • Add the required .dockerignore exclusions.
  • Confirm both images run non-root and read-only.
  • Confirm the API image is below 25 MB (7,815,877 bytes locally).
  • Add API, PostgreSQL 17, and OTEL Collector to local Compose.
  • Add the PostgreSQL healthcheck.
  • Make the API wait on healthy PostgreSQL in Compose.
  • Verify docker compose up reaches API readiness.
  • Keep Mini PC and local-development Compose files separate.

Static web dashboard

  • Create the no-build static HTML entry point.
  • Create the dark teal/navy glass visual system.
  • Replace decorative background waves with an operational chart grid.
  • Keep version, health, and error panels visible together.
  • Show the build-injected API version from the existing health response header.
  • Poll /healthz every two seconds.
  • Poll /readyz every two seconds.
  • Show each health indicator's last transition timestamp.
  • Count non-2xx /api/v1/* browser responses.
  • Calculate the rolling 60-second HTTP error rate.
  • Distinguish network-offline events from HTTP errors.
  • Preserve and grey the last known telemetry values offline.
  • Recover automatically when the API returns.
  • Add a fully local Natural Earth buoy map and pulsing markers.
  • Add animated reading charts.
  • Keep all API calls same-origin and relative.
  • Remove all frontend runtime dependencies on external hosts.
  • Verify all panels fit at 1280×720 without page scrolling.
  • Create the unprivileged nginx image.
  • Create the separate Mini PC Traefik Compose file.
  • Test the finished offline/degraded rendering without API connectivity.
  • Test the rendered dashboard while the API returns HTTP 500 responses.

Observability

  • Receive OTLP/gRPC traces in the Collector.
  • Export traces from the Collector to Tempo.
  • [~] Ship structured Kubernetes pod logs to Loki with the cluster Collector.
  • Start Loki locally and verify its Grafana data source readiness.
  • Configure Prometheus to scrape /metrics.
  • Add request-rate, error-rate, and latency panels.
  • Add database-operation latency panels.
  • Add readings-ingested and active-buoy panels.
  • Add a deployed-version dashboard variable.
  • [~] Correlate logs and traces using trace_id after cluster log ingestion exists.
  • Add API error-rate, latency, readiness, and scrape alerts.
  • Load the dashboard and evaluate all six PromQL queries.
  • Trigger chaos and distinguish the failing version in Grafana.

Fedora 44 host provisioning

  • Separate node 1 and node 2 inventory groups.
  • Pin the project k3s version.
  • Reject managed hosts that are not Fedora 44.
  • Install common host dependencies.
  • Enable firewalld.
  • Trust the configured pod and service CIDRs.
  • Open VXLAN port 8472/UDP between nodes.
  • Open kubelet port 10250/TCP between nodes.
  • Open API server port 6443/TCP on node 1.
  • Install node 1 as the k3s server.
  • Install node 2 as a k3s agent.
  • Keep the cluster token out of repository files and output.
  • Enable both k3s systemd services.
  • Wait for every Kubernetes node to become Ready.
  • Add the equivalent manual configuration checklist.
  • Document Terraform-to-Ansible inventory handoff.
  • Keep libvirt addresses, bridge rules, and VM network profiles out of the reusable Ansible roles.
  • Run the playbook twice on two clean Fedora 44 QEMU machines.
  • Confirm the second run reports no unintended changes (changed=0).
  • Restrict firewall sources to actual node/operator networks.
  • Confirm SELinux remains enforcing on both QEMU machines.
  • Reboot both QEMU machines and confirm automatic recovery.
  • Disconnect and reconnect node 2, then confirm it rejoins.
  • Record tested minimum CPU, memory, and disk requirements.

One-command QEMU lab

  • Check QEMU, libvirt, cloud-utils, Ansible, SSH, ping, and KVM.
  • Install missing dependencies on a Fedora workstation.
  • Enable libvirt and its default network.
  • Discover and cache the Fedora 44 x86_64 cloud image.
  • Keep SSH and VM state outside the repository.
  • Generate cloud-init media for both nodes.
  • Create reusable copy-on-write disks.
  • Create each VM with two vCPUs and 3 GiB RAM.
  • Assign deterministic addresses to both local VMs.
  • Wait for ICMP ping and SSH on both nodes.
  • Generate the Ansible inventory automatically.
  • Configure both nodes automatically.
  • Check Ansible reachability, k3s files, and binaries.
  • Check firewalld, k3s services, ports 6443 and 10250.
  • Check Kubernetes node readiness.
  • Add check, stop, and explicit destroy actions.
  • Execute lab.sh up with working /dev/kvm and pass the full verification.
  • Reboot both local VMs and pass the bounded post-reboot service and node checks.
  • Execute lab.sh check after a workstation reboot.
  • Test recovery from an interrupted image download and validate the resumed QCOW2.
  • Test recovery after node 2 is deleted manually and rejoin it with a fresh identity.
  • Test on a workstation with no dependencies installed.
  • Decide whether an aarch64 image path is required; keep the lab x86_64-only.
  • After final sign-off, destroy the QEMU lab and remove packages installed only for it with Pacman's dependency-aware cleanup.

Local k3d rollback mechanism

  • Define one local server and two local agent nodes.
  • Map Traefik to host ports 8080 and 8443.
  • Add lean kube-prometheus-stack values.
  • Add the synthetic Argo Rollouts analysis harness.
  • Add an idempotent up/check/destroy command and verify fresh creation and reconciliation.
  • Install Argo Rollouts.
  • Install kube-prometheus-stack with the local values.
  • Apply the synthetic harness.
  • Prove vector(0.0) permits promotion.
  • Change the test value to vector(1.0).
  • Prove failed analysis aborts promotion with vector(0.9).
  • Tear down and recreate the test from scratch.

Human-owned deployment and infrastructure

  • [H] Define infrastructure modules under terraform/.
  • [H] Export provisioned addresses for Ansible inventory.
  • [H] Create Kubernetes base resources and overlays. Agent-authored 2026-08-21 at the owner's request; builds and server dry-runs pass, not applied.
  • [H] Create API and load-generator workloads. Agent-authored 2026-08-21; not applied.
  • [H] Create active and preview Services. Agent-authored 2026-08-21; not applied.
  • [H] Create the Prometheus ServiceMonitor. Agent-authored 2026-08-21; not applied, scrape not confirmed.
  • [H] Create the real error-rate AnalysisTemplate. Agent-authored 2026-08-21; PromQL never evaluated against real series.
  • [H] Configure blue-green promotion and automated rollback. Agent-authored 2026-08-21; promotion and abort paths untested with the real API.
  • [H] Maintain Forgejo Actions and GitHub mirror workflows.
  • [H] Create sealed secrets from off-repository plaintext inputs.

Forgejo CI/CD

  • Run API and load-generator race tests, vet, and pinned lint in containers.
  • Scan committed history and the working tree for secrets.
  • Scan release configuration and images for high or critical findings.
  • Build immutable commit-SHA API, load-generator, and web image tags.
  • Keep active and preview traffic flowing during rollout analysis.
  • [~] Run the Forgejo workflow after a private docker runner and encrypted secrets are configured.
  • [~] Publish images and execute the first automated production rollout.

End-to-end acceptance

  • Provision two clean Fedora hosts.
  • Reboot both hosts before platform installation.
  • Install the cluster platform components.
  • Deploy PostgreSQL and wait for health.
  • Verify API liveness before database readiness.
  • Verify readiness after migrations complete.
  • Deploy the load generator and observe continuous samples.
  • Send one request to every API endpoint in the local Compose stack.
  • Verify every required metric and exact label name locally.
  • Verify an HTTP trace contains a child database span.
  • Correlate one stdout request log and Tempo trace by trace_id.
  • Promote a healthy preview through analysis.
  • Deploy a preview with a nonzero chaos rate.
  • Observe preview errors in Prometheus.
  • Confirm analysis aborts the unhealthy promotion.
  • Confirm the active Service remains on the healthy version.
  • Confirm load generation continues throughout failure.
  • Confirm the Mini PC page remains usable with the cluster off.
  • Capture reproducible, non-secret evidence needed for the final project report.