From 11108000426b8a18c94efd6b6e0174396ff4bff0 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Fri, 14 Aug 2026 14:25:39 +0100 Subject: [PATCH] chore: stop tracking the phase-10 validation scratch notes tasks/fix-validation-checks.md was committed with the phase-10 harness and shows up in the PR diff. tasks/ is a scratch area -- roughly forty analysis and plan documents live there locally and none of the others are tracked, on this branch or on develop. The file stays on disk; only the tracking is removed. --- tasks/fix-validation-checks.md | 169 --------------------------------- 1 file changed, 169 deletions(-) delete mode 100644 tasks/fix-validation-checks.md diff --git a/tasks/fix-validation-checks.md b/tasks/fix-validation-checks.md deleted file mode 100644 index ea33fd9223..0000000000 --- a/tasks/fix-validation-checks.md +++ /dev/null @@ -1,169 +0,0 @@ -# Fix Telemetry Validation Checks - -## Context - -The CI pipeline infrastructure is fully operational (build + deploy + run). However, -the `validate_telemetry.py` validation suite fails 35 checks due to mismatches between -what the validation expects and what the telemetry stack actually produces. These fall -into 4 categories. - -CI run: https://github.com/XRPLF/rippled/actions/runs/23026466191 - ---- - -## Category 1: StatsD Metrics — 0 Series (25 failures) - -**Symptoms:** - -``` -[FAIL] metric.statsd_gauges.xrpld_LedgerMaster_Validated_Ledger_Age: 0 series -[FAIL] metric.statsd_counters.xrpld_rpc_requests: 0 series -[FAIL] metric.statsd_histograms.xrpld_rpc_time: 0 series -[FAIL] metric.overlay_traffic.xrpld_total_Bytes_In: 0 series -[FAIL] metric.phase9_nodestore.xrpld_nodestore_reads_total: 0 series -... (25 total) -``` - -**Root Cause:** Two issues compounding: - -1. **StatsD receiver is commented out** in `otel-collector-config.yaml` (lines 39-54). - The collector config was updated to expect native OTLP metrics from beast::insight - (comment: "StatsD UDP port removed — beast::insight now uses native OTLP"), but - the validation harness configures xrpld nodes with `server=statsd`. - -2. **Metric name mismatch:** The `expected_metrics.json` expects StatsD-style metric - names (e.g., `xrpld_LedgerMaster_Validated_Ledger_Age`). When using `server=otel`, - beast::insight emits OTLP metrics which may have different names/structure. - -**Fix Options (pick one):** - -- **Option A (recommended):** Change the node config in `run-full-validation.sh` from - `server=statsd` to `server=otel` (line 255), remove the `address=127.0.0.1:8125` line, - then update `expected_metrics.json` with the actual OTLP metric names. This aligns with - the collector config's OTLP-first design and avoids re-enabling the StatsD receiver. - -- **Option B:** Uncomment the StatsD receiver in `otel-collector-config.yaml`, add - `statsd` to the metrics pipeline receivers list, and keep node config as `server=statsd`. - Simpler but goes against the migration to native OTLP. - -**Investigation needed for Option A:** - -- Run xrpld locally with `server=otel`, query Prometheus, and capture the actual OTLP - metric names to update `expected_metrics.json`. - -**Files to modify:** - -- `docker/telemetry/workload/run-full-validation.sh` — change `[insight]` section -- `docker/telemetry/workload/expected_metrics.json` — update metric names for OTLP -- `docker/telemetry/workload/validate_telemetry.py` — may need metric query adjustments - ---- - -## Category 2: Missing Spans — tx.process, tx.receive (2 failures) - -**Symptoms:** - -``` -[FAIL] span.tx.process: tx.process: 0 traces (expected > 0) -[FAIL] span.tx.receive: tx.receive: 0 traces (expected > 0) -``` - -**Root Cause:** The span names exist in the code: - -- `src/xrpld/app/misc/NetworkOPs.cpp:1228` — `XRPL_TRACE_TX("tx.process")` -- `src/xrpld/overlay/detail/PeerImp.cpp:1273` — `XRPL_TRACE_TX("tx.receive")` - -Likely causes (investigate in order): - -1. **Batch delay:** The 2-second batch delay (`batch_delay_ms=2000`) plus 30s propagation - wait may not be enough if these spans are created late in the workload. -2. **Code path not triggered:** `tx.process` fires in `NetworkOPs::processTransaction()`. - The tx_submitter submits via RPC `submit` command which calls this path. But if the - transactions fail validation before reaching `processTransaction()`, no span is emitted. -3. **Span naming mismatch:** The validation queries Tempo for exact operation name - `tx.process`. Verify Tempo stores the span with this exact name. - -**Investigation:** - -- Check the tx_submitter output in CI logs — are transactions actually succeeding? -- Query Tempo API locally for all span names to see what's actually emitted. - -**Files to modify:** - -- Possibly `docker/telemetry/workload/validate_telemetry.py` — adjust timing/queries -- Possibly `docker/telemetry/workload/run-full-validation.sh` — increase propagation wait - ---- - -## Category 3: Span Hierarchy — rpc.request -> rpc.process (1 failure) - -**Symptoms:** - -``` -[FAIL] span.hierarchy.rpc.request->rpc.process: rpc.process not found in rpc.request traces -``` - -**Root Cause:** The validator fetches traces containing `rpc.request` from Tempo and -checks if any child span is named `rpc.process`. Both spans are emitted (they pass -individual checks), but the parent-child relationship isn't established. - -**Investigation:** - -- Check `src/xrpld/rpc/detail/ServerHandler.cpp` — `rpc.request` (line 271) and - `rpc.process` (line 573) are in the same file. Verify that `rpc.process` is created - as a child of `rpc.request` (i.e., its parent context is set). -- The issue may be that `rpc.process` creates a new root span instead of linking to the - `rpc.request` span context. - -**Files to modify:** - -- Possibly `src/xrpld/rpc/detail/ServerHandler.cpp` — fix span parenting -- OR `docker/telemetry/workload/validate_telemetry.py` — if hierarchy check logic is wrong - ---- - -## Category 4: Dashboard 404s (5 failures) - -**Symptoms:** - -``` -[FAIL] dashboard.xrpld-statsd-node-health: HTTP 404 -[FAIL] dashboard.xrpld-statsd-network: HTTP 404 -[FAIL] dashboard.xrpld-statsd-rpc: HTTP 404 -[FAIL] dashboard.xrpld-statsd-overlay-detail: HTTP 404 -[FAIL] dashboard.xrpld-statsd-ledger-sync: HTTP 404 -``` - -**Root Cause:** Dashboard UIDs were renamed from `xrpld-statsd-*` to `xrpld-system-*` -but `expected_metrics.json` still references the old names. - -**Actual UIDs in `docker/telemetry/grafana/dashboards/`:** - -| Expected (in expected_metrics.json) | Actual (in dashboard JSON) | -| ----------------------------------- | ----------------------------- | -| `xrpld-statsd-node-health` | `xrpld-system-node-health` | -| `xrpld-statsd-network` | `xrpld-system-network` | -| `xrpld-statsd-rpc` | `xrpld-system-rpc` | -| `xrpld-statsd-overlay-detail` | `xrpld-system-overlay-detail` | -| `xrpld-statsd-ledger-sync` | `xrpld-system-ledger-sync` | - -**Fix:** Update the 5 UIDs in `expected_metrics.json` → `grafana_dashboards.uids[]`. - -**Files to modify:** - -- `docker/telemetry/workload/expected_metrics.json` — update dashboard UIDs - ---- - -## Execution Order - -1. **Category 4 (Dashboard UIDs)** — trivial rename, no investigation needed -2. **Category 1 (StatsD/OTLP metrics)** — requires investigation to choose Option A vs B - and capture actual metric names -3. **Category 2 (Missing tx spans)** — requires investigation into transaction code paths -4. **Category 3 (Span hierarchy)** — requires investigation into span context propagation - -## Branch - -All changes go on: `pratik/otel-phase10-workload-validation` -Worktree: `/tmp/otel-phase10-iter`