rippled/tasks/fix-validation-checks.md

Fix Telemetry Validation Checks

Context

The CI pipeline infrastructure is fully operational (build + deploy + run). However, the validate_telemetry.py validation suite fails 35 checks due to mismatches between what the validation expects and what the telemetry stack actually produces. These fall into 4 categories.

CI run: https://github.com/XRPLF/rippled/actions/runs/23026466191

---

Category 1: StatsD Metrics — 0 Series (25 failures)

Symptoms:

[FAIL] metric.statsd_gauges.xrpld_LedgerMaster_Validated_Ledger_Age: 0 series
[FAIL] metric.statsd_counters.xrpld_rpc_requests: 0 series
[FAIL] metric.statsd_histograms.xrpld_rpc_time: 0 series
[FAIL] metric.overlay_traffic.xrpld_total_Bytes_In: 0 series
[FAIL] metric.phase9_nodestore.xrpld_nodestore_reads_total: 0 series
... (25 total)

Root Cause: Two issues compounding:

1. StatsD receiver is commented out in otel-collector-config.yaml (lines 39-54). The collector config was updated to expect native OTLP metrics from beast::insight (comment: "StatsD UDP port removed — beast::insight now uses native OTLP"), but the validation harness configures xrpld nodes with server=statsd.

2. Metric name mismatch: The expected_metrics.json expects StatsD-style metric names (e.g., xrpld_LedgerMaster_Validated_Ledger_Age). When using server=otel, beast::insight emits OTLP metrics which may have different names/structure.

Fix Options (pick one):

• Option A (recommended): Change the node config in run-full-validation.sh from server=statsd to server=otel (line 255), remove the address=127.0.0.1:8125 line, then update expected_metrics.json with the actual OTLP metric names. This aligns with the collector config's OTLP-first design and avoids re-enabling the StatsD receiver.

• Option B: Uncomment the StatsD receiver in otel-collector-config.yaml, add statsd to the metrics pipeline receivers list, and keep node config as server=statsd. Simpler but goes against the migration to native OTLP.

Investigation needed for Option A:

• Run xrpld locally with server=otel, query Prometheus, and capture the actual OTLP metric names to update expected_metrics.json.

Files to modify:

• docker/telemetry/workload/run-full-validation.sh — change [insight] section
• docker/telemetry/workload/expected_metrics.json — update metric names for OTLP
• docker/telemetry/workload/validate_telemetry.py — may need metric query adjustments

---

Category 2: Missing Spans — tx.process, tx.receive (2 failures)

Symptoms:

[FAIL] span.tx.process: tx.process: 0 traces (expected > 0)
[FAIL] span.tx.receive: tx.receive: 0 traces (expected > 0)

Root Cause: The span names exist in the code:

• src/xrpld/app/misc/NetworkOPs.cpp:1228 — XRPL_TRACE_TX("tx.process")
• src/xrpld/overlay/detail/PeerImp.cpp:1273 — XRPL_TRACE_TX("tx.receive")

Likely causes (investigate in order):

1. Batch delay: The 2-second batch delay (batch_delay_ms=2000) plus 30s propagation wait may not be enough if these spans are created late in the workload.
2. Code path not triggered: tx.process fires in NetworkOPs::processTransaction(). The tx_submitter submits via RPC submit command which calls this path. But if the transactions fail validation before reaching processTransaction(), no span is emitted.
3. Span naming mismatch: The validation queries Tempo for exact operation name tx.process. Verify Tempo stores the span with this exact name.

Investigation:

• Check the tx_submitter output in CI logs — are transactions actually succeeding?
• Query Tempo API locally for all span names to see what's actually emitted.

Files to modify:

• Possibly docker/telemetry/workload/validate_telemetry.py — adjust timing/queries
• Possibly docker/telemetry/workload/run-full-validation.sh — increase propagation wait

---

Category 3: Span Hierarchy — rpc.request -> rpc.process (1 failure)

Symptoms:

[FAIL] span.hierarchy.rpc.request->rpc.process: rpc.process not found in rpc.request traces

Root Cause: The validator fetches traces containing rpc.request from Tempo and checks if any child span is named rpc.process. Both spans are emitted (they pass individual checks), but the parent-child relationship isn't established.

Investigation:

• Check src/xrpld/rpc/detail/ServerHandler.cpp — rpc.request (line 271) and rpc.process (line 573) are in the same file. Verify that rpc.process is created as a child of rpc.request (i.e., its parent context is set).
• The issue may be that rpc.process creates a new root span instead of linking to the rpc.request span context.

Files to modify:

• Possibly src/xrpld/rpc/detail/ServerHandler.cpp — fix span parenting
• OR docker/telemetry/workload/validate_telemetry.py — if hierarchy check logic is wrong

---

Category 4: Dashboard 404s (5 failures)

Symptoms:

[FAIL] dashboard.xrpld-statsd-node-health: HTTP 404
[FAIL] dashboard.xrpld-statsd-network: HTTP 404
[FAIL] dashboard.xrpld-statsd-rpc: HTTP 404
[FAIL] dashboard.xrpld-statsd-overlay-detail: HTTP 404
[FAIL] dashboard.xrpld-statsd-ledger-sync: HTTP 404

Root Cause: Dashboard UIDs were renamed from xrpld-statsd-* to xrpld-system-* but expected_metrics.json still references the old names.

Actual UIDs in docker/telemetry/grafana/dashboards/:

Expected (in expected_metrics.json)	Actual (in dashboard JSON)
xrpld-statsd-node-health	xrpld-system-node-health
xrpld-statsd-network	xrpld-system-network
xrpld-statsd-rpc	xrpld-system-rpc
xrpld-statsd-overlay-detail	xrpld-system-overlay-detail
xrpld-statsd-ledger-sync	xrpld-system-ledger-sync

Fix: Update the 5 UIDs in expected_metrics.json → grafana_dashboards.uids[].

Files to modify:

• docker/telemetry/workload/expected_metrics.json — update dashboard UIDs

---

Execution Order

1. Category 4 (Dashboard UIDs) — trivial rename, no investigation needed
2. Category 1 (StatsD/OTLP metrics) — requires investigation to choose Option A vs B and capture actual metric names
3. Category 2 (Missing tx spans) — requires investigation into transaction code paths
4. Category 3 (Span hierarchy) — requires investigation into span context propagation

Branch

All changes go on: pratik/otel-phase10-workload-validation Worktree: /tmp/otel-phase10-iter