From 1289f40532c80aea2217e840e2b0cdd504bbdaf2 Mon Sep 17 00:00:00 2001 From: Fi3w0 Date: Thu, 27 Aug 2026 21:47:49 +0200 Subject: [PATCH] docs: record the collector batch size defect --- docs/evidence.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/evidence.md b/docs/evidence.md index d8f5f9d..1e69298 100644 --- a/docs/evidence.md +++ b/docs/evidence.md @@ -99,6 +99,24 @@ ConfigMap, which the kube-prometheus-stack Grafana sidecar loads as `nereus.yaml`. The correlation requires the API at `LOG_LEVEL=info`, because the per-request log line is the only one that carries `trace_id`. +Two collector defects were found and fixed on the way to this result, and both +belong in the report as corrective actions. The first is the capability problem +above. The second only appears after a restart: `start_at: beginning` makes the +receiver replay whole log files, the batch processor had no size cap, and Loki +refused the resulting push. + +```text +HTTP 503, ResourceExhausted: grpc: received message larger than max +(4378796 vs. 4194304) +``` + +The exporter retries that request unchanged, so the backlog never drains. +Capping the batch with `send_batch_size: 1000` and `send_batch_max_size: 2000` +keeps every push under Loki's 4 MiB gRPC limit. What remains after a restart is +a `Permanent` rejection of replayed lines older than Loki's acceptance window, +which drops those lines rather than blocking, and leaves live traffic +unaffected. + ## Presentation commands ```bash