docs: record the collector batch size defect
This commit is contained in:
parent
dc3115a82d
commit
1289f40532
1 changed files with 18 additions and 0 deletions
|
|
@ -99,6 +99,24 @@ ConfigMap, which the kube-prometheus-stack Grafana sidecar loads as
|
||||||
`nereus.yaml`. The correlation requires the API at `LOG_LEVEL=info`, because the
|
`nereus.yaml`. The correlation requires the API at `LOG_LEVEL=info`, because the
|
||||||
per-request log line is the only one that carries `trace_id`.
|
per-request log line is the only one that carries `trace_id`.
|
||||||
|
|
||||||
|
Two collector defects were found and fixed on the way to this result, and both
|
||||||
|
belong in the report as corrective actions. The first is the capability problem
|
||||||
|
above. The second only appears after a restart: `start_at: beginning` makes the
|
||||||
|
receiver replay whole log files, the batch processor had no size cap, and Loki
|
||||||
|
refused the resulting push.
|
||||||
|
|
||||||
|
```text
|
||||||
|
HTTP 503, ResourceExhausted: grpc: received message larger than max
|
||||||
|
(4378796 vs. 4194304)
|
||||||
|
```
|
||||||
|
|
||||||
|
The exporter retries that request unchanged, so the backlog never drains.
|
||||||
|
Capping the batch with `send_batch_size: 1000` and `send_batch_max_size: 2000`
|
||||||
|
keeps every push under Loki's 4 MiB gRPC limit. What remains after a restart is
|
||||||
|
a `Permanent` rejection of replayed lines older than Loki's acceptance window,
|
||||||
|
which drops those lines rather than blocking, and leaves live traffic
|
||||||
|
unaffected.
|
||||||
|
|
||||||
## Presentation commands
|
## Presentation commands
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue