Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill

This branch had already made the same corrections independently, and in
richer form, so the resolution keeps this branch's version nearly throughout:

- 09-data-collection-reference.md: this branch already documents the
  state-accounting gauges as cumulative **microseconds** with an explanatory
  callout, and already names `jobq_job_count` with its `jobq` group. Kept.
- telemetry-runbook.md: already carries `jobq_job_count` in both tables. Kept,
  along with this branch's larger additions.
- OpenTelemetryPlan.md: kept this branch's rewritten section 9 blurb, which
  describes the inventory without hardcoding counts and so cannot drift.
- consensus-health.json: kept this branch's rewrite. It deliberately removed
  the four TraceQL close-time detail panels and renamed the agreement panel;
  the incoming side would have resurrected them. Panel count unchanged at 26.
- integration-test.sh: this branch's unprefixed native metric names were kept,
  but it still asserted `job_count`, so the `jobq_job_count` correction was
  carried over. That check would otherwise always fail.
This commit is contained in:
Pratik Mankawde
2026-08-17 12:07:56 +01:00
7 changed files with 95 additions and 30 deletions

View File

@@ -2648,6 +2648,8 @@ Log files are ingested by the OTel Collector's `filelog` receiver, which tails `
The receiver tails `/var/log/xrpld/*/debug.log` inside the collector container. docker-compose bind-mounts the host log root there; the source defaults to the repo-relative `docker/telemetry/data/logs`, which the telemetry configs write to (`data/logs/<network>/debug.log`) and which needs no root. To tail logs from elsewhere, set `XRPLD_LOG_DIR` before `docker compose up` (the integration test does this to point at its own workdir). The single trailing `*` matches one per-network or per-node subdirectory.
Each file is read from the beginning, because the receiver's own default (`end`) would skip anything a node wrote before the collector's first poll and would never read a log that has stopped being written to. Read offsets are held in memory by default, so a restarted collector re-reads the files it already ingested. The developer stack avoids that by layering `otel-collector-filestorage.yaml` as a second `--config`, which adds a `file_storage` extension that keeps the offsets on a named volume; a one-shot init service prepares that volume, because the collector runs as a non-root user and a fresh Docker volume is owned by root. Ephemeral stacks such as the workload validation harness create a fresh log directory per run, so they have nothing to resume from and deliberately omit the overlay.
### LogQL Query Examples
The OTel Collector emits logs to Loki with `service_name="xrpld"` (not `job="xrpld"`).