# Nereus Nereus is an ocean buoy telemetry API that exists so there is something real to deploy. It stores buoys and their readings, exposes Prometheus metrics and OpenTelemetry traces, and can be told to start failing on demand. The application is deliberately small and boring. The project is everything wrapped around it: custom images, a two-node k3s cluster, CI/CD on a self-hosted runner, blue-green deploys that abort themselves when the new version is bad, and logs, metrics and traces that actually join up. This is my final DevOps project for Tokio School. The fictional company in the brief is TechWave Solutions, which fits the water theme, so I ran with it. **If you only run one thing, run the k3d lab.** It puts a real multi-node k3s inside Docker and proves the automated rollback end to end on any OS with Docker and 8 GB of RAM: ```bash scripts/k3d/lab.sh up scripts/k3d/lab.sh prove ``` ## What the API does Go 1.26, `chi` for routing, `pgx/v5` against PostgreSQL 17, no ORM. Migrations are plain SQL files applied on startup and written to be idempotent, so a pod restart or a second replica does not break anything. ``` GET /healthz liveness, 200 whenever the process is up GET /readyz readiness, 200 only once the database is reachable GET /metrics Prometheus exposition GET /api/v1/buoys list POST /api/v1/buoys create GET /api/v1/buoys/{id} fetch one DELETE /api/v1/buoys/{id} delete GET /api/v1/readings?buoy_id=&from=&to= list, paginated POST /api/v1/readings create GET /api/v1/readings/aggregate?window=1h deliberately expensive GROUP BY ``` The split between `/healthz` and `/readyz` is the part that matters for Kubernetes. The service starts and serves `/healthz` even with the database down, so a PostgreSQL outage makes pods unready instead of killing them in a restart loop. `/api/v1/readings/aggregate` runs a real grouped query over a time window and is not optimised on purpose, because the dashboards need something that produces visible latency. `CHAOS_ERROR_RATE` is a float between 0 and 1. Above zero, that fraction of `/api/v1/*` requests return HTTP 500 with a JSON body. Health endpoints are never affected. This is the entire mechanism for simulating a bad release, and it is controlled by an environment variable rather than an admin endpoint so that turning it on is a deployment, not a runtime toggle. `apps/loadgen/` is a small Go binary that hits the API continuously with a weighted mix of reads, writes and aggregates. It is not decoration. The rollback analysis computes an error rate from Prometheus, and with no traffic there are no samples and the analysis has nothing to decide on. ## Running it There are three levels, and they exist because nobody grading this is going to install KVM and download a Fedora image. ### Level 1: Docker Compose, about three minutes ```bash docker compose up --build ``` API, PostgreSQL and an OTel Collector, with the API on `127.0.0.1:18080` and the load generator already producing traffic. Add the observability overlay for Grafana on `127.0.0.1:13000` and Prometheus on `127.0.0.1:19090`: ```bash docker compose -f compose.yaml -f observability/compose.yaml up -d ``` ### Level 2: k3d, real Kubernetes, any OS with Docker ```bash scripts/k3d/lab.sh up # one server, two agents, Argo Rollouts, kube-prometheus-stack scripts/k3d/lab.sh prove # promote a healthy version, then abort a bad one scripts/k3d/lab.sh destroy ``` `prove` is the interesting one. It drives a blue-green Rollout through a real AnalysisRun that queries a real Prometheus over the network, first with a query that returns a healthy value and then with one that returns a failing value, and it fails loudly if the bad revision ever reaches the active Service. Be clear about what is faked here: the harness in `scripts/k3d/analysis-harness/` uses a throwaway nginx Rollout and a hardcoded metric value. Argo Rollouts, the AnalysisRun, the Prometheus query and the abort are all real. Only the number is synthetic. The real gate lives in `deploy/rollouts/analysistemplate.yaml` and ran against the real API on the Fedora cluster. ### Level 3: two Fedora 44 machines This is my actual cluster. `scripts/provision/` holds an Ansible playbook that turns two already-installed Fedora 44 hosts into a k3s server and agent: ```bash cp scripts/provision/inventory.example.yml scripts/provision/inventory.yml # edit addresses, CIDRs and SSH user K3S_TOKEN="$(openssl rand -hex 32)" scripts/provision/bootstrap.sh scripts/provision/inventory.yml ``` The playbook is idempotent and reruns clean. The token is passed through the process environment and never written to the inventory or the repository. `scripts/provision/lab.sh` builds the same two hosts as local QEMU VMs from the Fedora 44 cloud image, with 2 vCPUs, 3 GiB of RAM and a 30 GiB thin disk each. Those numbers are the lowest I actually tested, not a recommendation. VM state and its SSH key live under `~/.local/state/nereus-lab`, outside the repo. ## How a bad deploy gets killed This is the part I would defend first, so it is worth spelling out. The API is an Argo Rollouts `Rollout`, not a Deployment. New pods come up alongside the old ones and take no production traffic. `nereus-api-active` points at the stable version and `nereus-api-preview` at the candidate. A second load generator drives the preview Service, because an analysis that measures nothing would treat "no samples" as success. Before promotion, a `prePromotionAnalysis` runs this query against Prometheus: ```promql ( sum(rate(nereus_http_requests_total{service="nereus-api-preview",status=~"5.."}[1m])) or vector(0) ) / sum(rate(nereus_http_requests_total{service="nereus-api-preview"}[1m])) ``` Five samples, 20 seconds apart, after a 30 second delay to let the new pods take traffic. The success condition is `len(result) == 0 || result[0] < 0.05`, and `failureLimit: 0` means a single bad sample aborts. The `or vector(0)` keeps the division defined when there are no errors at all, which is otherwise an empty result rather than a zero. When it aborts, the active Service selector never moves. `scaleDownDelaySeconds: 30` keeps the old ReplicaSet warm so the fallback is instant rather than a cold start. From `docs/evidence.md`, the two AnalysisRuns from the mechanism proof: ```text probe-746bbb94df-2-pre Successful vector(0.0) probe-556d5b659b-3-pre Failed vector(1.0) ``` To trigger it for real, deploy a version with `CHAOS_ERROR_RATE` above 0.05 and watch it refuse to promote. ## Architecture ``` Workstation (CachyOS) └── libvirt/KVM ├── nereus-node1 Fedora 44, k3s server └── nereus-node2 Fedora 44, k3s agent Mini PC (always on) ├── Forgejo at git.fiwlabs.dev, behind Traefik with automatic TLS ├── Forgejo Runner, docker mode, same LAN as the cluster ├── Container registry, part of Forgejo └── Static dashboard at nereus.fiwlabs.dev, nginx behind Traefik Mirror to GitHub, which is the link I hand in. ``` Inside the cluster: ``` Traefik (ships with k3s) └── Ingress → nereus-api-active ├── stable ReplicaSet (real traffic) └── preview ReplicaSet (candidate, no traffic) PostgreSQL StatefulSet on a local-path PVC loadgen one Deployment against active, one against preview observability/ kube-prometheus-stack Prometheus, Grafana, node-exporter, kube-state-metrics OTel Collector Deployment, receives OTLP traces, exports to Tempo OTel Collector DaemonSet, tails pod logs, exports to Loki Tempo traces Loki logs ``` The web dashboard runs on the Mini PC, not in the cluster, and that is deliberate. The cluster is off most of the time. If the page lived inside it, the domain would be dead 95% of the time and there would be nothing to show. Traefik on the Mini PC serves the static files at `/` and proxies `/api/v1/*`, `/healthz` and `/readyz` through to the cluster, so everything is same-origin with no CORS and no hardcoded hostnames. With the cluster off, those paths return 502 and the page renders the last known values greyed out and keeps polling. An unreachable API is a normal state for that page, not an error. ## Why I built it this way **distroless, not alpine.** The final image is `gcr.io/distroless/static:nonroot` with a statically linked binary copied in and nothing else. No shell, no package manager, no busybox. If someone gets code execution there is nothing to pivot with. It also happens to be tiny: the API image is under 8 MB, with the exact figure recorded in `docs/evidence.md`. **k3s, not full Kubernetes.** Same API, one binary, Traefik and a local-path provisioner included, and it runs on two 3 GiB VMs. Nothing in this project uses an API that k3s does not have. **On-premise, not EKS or AKS.** The brief asks for a managed cloud cluster. I did not do that, and the reason is cost and control rather than difficulty: I have hardware sitting here and no interest in paying for a load balancer to prove a point about Terraform. The manifests are plain Kubernetes and would apply to a managed cluster unchanged. This is the deviation from the brief I expect to be asked about, so it is stated here rather than buried. **Forgejo Actions, not GitHub Actions.** The brief allows the tool of your choice. The runner lives on the same LAN as the cluster, which means the deploy job can reach the Kubernetes API without exposing it to the internet or punching a hole through a firewall for a hosted runner. The registry is part of the same Forgejo instance, so images never leave the network either. The GitHub mirror still runs its own workflow so the repo a reviewer opens shows green checks, and that workflow consumes no secrets by design. **The cluster is ephemeral, on purpose.** I bring it up, record, and shut it down. That is not a gap in the project, it is the evidence that the infrastructure as code works. If the whole thing rebuilds from the repository with a command, then the repository really is the source of truth. **Everything runs non-root with a read-only root filesystem**, including the pieces where that is inconvenient. The log collector was the awkward one. k3s writes `/var/log/pods` as `0750 root:root` and each container log as `0640 root:root`, so a collector running as UID 10001 cannot even traverse the directory. Adding the `DAC_READ_SEARCH` capability looks like the fix and does nothing, because Kubernetes sets no ambient capabilities and the capability is cleared on exec for a non-root user. The container ended up with `CapEff: 0000000000000000`, matched no files, and reported no error at all. The fix is `supplementalGroups: [0]`, since group root already has read on those files. Non-root, no capabilities, read-only mount. ## Secrets and security No secret is committed, and none is read by any tooling that logs. Gitleaks scans both the working tree and the committed history on every push and a hit fails the build. Trivy scans the release configuration and the built images for HIGH and CRITICAL findings. The `nereus-db` Secret holds the PostgreSQL password and DSN. Production gets it from the encrypted `deploy/secrets/nereus-db-sealed.yaml`; the Sealed Secrets controller is the only component that can decrypt it. The current Fedora cluster was reset onto a fresh PVC and verified with that committed resource. Development can still create an ephemeral Secret directly. See `deploy/secrets/README.md` for both workflows. The CI pipeline never uses a cluster-admin kubeconfig. It authenticates as a `deployer` ServiceAccount with RBAC scoped to the `nereus` namespace, supplied as an encrypted `KUBECONFIG_B64` repository secret, written to the workspace with mode `0600` and deleted in an `always()` step. The registry pull secret is created through a pipe so the generated manifest is never logged or written to disk. Containers drop all capabilities, disallow privilege escalation, run with a read-only root filesystem and a `RuntimeDefault` seccomp profile, and mount `tmpfs` where a writable path is genuinely needed. SELinux stays enforcing on both Fedora nodes. ## Observability Metrics, logs and traces come from the same request and can be walked between. The API exports five metrics with fixed names, since the dashboards and the rollback query depend on them literally: ``` nereus_http_requests_total{method, path, status, version} counter nereus_http_request_duration_seconds{method, path, version} histogram nereus_db_query_duration_seconds{operation} histogram nereus_readings_ingested_total counter nereus_buoys_active gauge ``` `path` is always the chi route template, `/api/v1/buoys/{id}` rather than the resolved path, because resolved paths give unbounded cardinality and will eventually take Prometheus down. Traces go out over OTLP/gRPC to the collector Deployment and on to Tempo. Every handler is a span and every database query is a child span. If `OTEL_EXPORTER_OTLP_ENDPOINT` is unset, tracing is a no-op, so a missing collector can never stop the service from starting. Logs are structured JSON on stdout via `log/slog`. A DaemonSet collector tails `/var/log/pods` on each node and ships to Loki. Every request logs one line carrying its `trace_id`, and Grafana's Loki data source has a derived field that turns that value into a link into Tempo, with the reverse link configured on the Tempo side. A verified example, log line and the trace it resolves to, is in `docs/evidence.md`. Alert rules for error rate, latency, readiness failures and scrape failure are in `observability/alerts/nereus.yaml`. cAdvisor and node-exporter metrics arrive with kube-prometheus-stack rather than being installed separately. ## CI/CD `.forgejo/workflows/pipeline.yml` runs on pushes and pull requests to `main`. Verification runs the Go tests with the race detector, `go vet`, a pinned `golangci-lint`, gitleaks over history and tree, a kustomize build of every overlay, and a Trivy config scan. Everything runs as a plain container image rather than a marketplace action, because Forgejo resolves actions from `code.forgejo.org` and most third-party ones simply are not mirrored there. A push to `main` then builds the three images, tags them with the immutable commit SHA, scans them, pushes them to the Forgejo registry, and deploys the production overlay. The deploy job waits on the Rollout, which means a failed pre-promotion analysis fails the pipeline. ## Where each requirement from the brief lives | Requirement | Where | |---|---| | Custom, optimised Docker images | `build/*.Dockerfile`, multi-stage into distroless, API under 8 MB | | Docker Compose | `compose.yaml`, plus `observability/compose.yaml` and `build/compose.mini.yaml` | | Kubernetes orchestration | k3s, two Fedora 44 nodes; k3d for a portable equivalent | | Deployments, Services, Ingress, ConfigMaps, Secrets | `deploy/base/`, composed with kustomize overlays | | Infrastructure as code | `terraform/infra/` for libvirt machines, `terraform/platform/` for cluster controllers, and `scripts/provision/` for host configuration | | CI/CD pipeline | `.forgejo/workflows/pipeline.yml`, self-hosted runner | | Container registry | Forgejo registry at `git.fiwlabs.dev`, TLS from Traefik | | Secure credential handling | Gitleaks, Trivy, namespace-scoped `deployer` kubeconfig, and a committed encrypted `nereus-db` SealedSecret | | Blue-green deployment | Argo Rollouts, `deploy/rollouts/rollout.yaml` | | Automated rollback | `deploy/rollouts/analysistemplate.yaml`, Prometheus-driven abort | | OpenTelemetry Collector | `deploy/observability/otel-collector.yaml` for traces, `otel-log-collector.yaml` for logs | | Prometheus | kube-prometheus-stack, scraped through a ServiceMonitor | | Grafana dashboards | `observability/dashboards/`, data sources in `deploy/observability/grafana-datasources.yaml` | | Loki | `deploy/observability/loki.yaml` | | cAdvisor and node-exporter | included with kube-prometheus-stack | | Alerting | `observability/alerts/nereus.yaml` | ## What is not done Stated plainly so it does not have to be discovered. Discord routing is configured but disabled until an owner-provided webhook is sealed. The Sealed Secrets controller key also needs an off-repository backup; without it, a rebuilt cluster cannot decrypt the committed database resource. The remaining acceptance gap is environmental: the reproducible fresh-VM path has not been rerun on a second workstation with no dependencies installed. After final sign-off, the local QEMU lab and packages installed only for it can be removed. ## Fedora traps worth knowing Three things broke the cluster in ways that took longer to find than to fix, all of them handled by the playbook now. | Symptom | Cause | Fix | |---|---|---| | Pods on different nodes cannot reach each other, DNS fails strangely | firewalld blocks VXLAN | open `8472/udp` between nodes | | kubelet fails with permission errors | SELinux enforcing without the policy | `dnf install k3s-selinux` | | k3s complains at startup | zram swap enabled | `systemctl disable --now zram-generator` | ## Repo layout ``` apps/api/ Go service apps/loadgen/ traffic generator apps/web/ static dashboard, runs on the Mini PC build/ Dockerfiles and the Mini PC compose file deploy/base/ kustomize base deploy/overlays/ dev and prod deploy/rollouts/ Rollout and AnalysisTemplate deploy/observability/ Loki, Tempo, collectors, Grafana data sources observability/ dashboards, alert rules, collector configs, local compose scripts/k3d/ portable rollback lab scripts/provision/ Ansible roles, bootstrap, QEMU lab terraform/ libvirt machines, Ansible inventory and cluster-wide Helm releases docs/ roadmap, evidence, CI/CD and Mini PC notes ``` `AGENTS.md` holds the rules the AI agents working in this repo had to follow. `PLAN.md` is my own working plan, in Spanish, and it is a worklog rather than documentation.