Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation

Three doc conflicts, all inside Phase 10's own sections. Resolved by keeping
phase-9's corrections and re-applying phase-10's ownership of those sections on
top, rather than taking either side wholesale:

06 §6.8.3 — Status now says Phase 10 is implemented on THIS branch (phase-9's
copy said "not merged into this branch", true there, false here). The stale
"71 checks" enumeration and "Current Status" list are replaced by phase-9's
dynamic-count description plus a Known Gaps list that states the real reason
the rpc.process hierarchy is unassertable: rpc.process is HTTP-only, so a
WebSocket-only workload never produces it. Phase-9's CI Deliverable subsection
is carried through; exit criteria keep phase-10's tick state with phase-9's
corrected wording.

06 §6.8.3 Architecture — the automatic merge had resurrected phase-10's stale
2-node cluster prose and diagram over phase-9's 5-node correction. Reconciled
to phase-10's topology (native xrpld processes vs the containerised backend,
which is accurate) with phase-9's node count: 5 validators, and the collector
labelled OTLP + filelog rather than StatsD, which the config has never had.
Dropped the "all 26 metrics required" label in favour of the manifest.

06 §6.8.3 Key Implementation Details — two claims corrected against the code.
The StatsD m_dirty gauge fix describes a member that exists nowhere in the
repo, and the harness sets [insight] server=otel anyway, so gauges export
through an observable-gauge callback. The tx.receive attribute keys are bare
suppressed and tx_status, not dotted, and tx_status is set only on the
reject/known-bad/dropped paths, so it is absent on a successful receive.

09 §5c — kept phase-10's four-column table shape, with phase-9's corrected
counts: 40 of 41 emitted spans, 67 required attributes, 14 of 15 dashboards.

Phase10_taskList — dynamic inventory totals, the real RPC span trees, and exit
criteria ticked where the code on this branch closes them. Per-RPC timings are
recorded as not gated: regression-metrics.json defines only spans and job_queue.

Verified every hunk of the phase-10 diff falls inside a Phase-10-owned section
(06 §6.8.3, 09 §5c, Phase10_taskList, and the runbook's appended Phase 10
sections); no phase-9-owned text is modified from this branch.

OpenTelemetryPlan/Phase10_taskList.md

@@ -22,10 +22,13 @@
 
 Before Phases 1-9 can be considered production-ready, we need proof that:
 
-1. All required spans fire with correct attributes under real transaction workloads
+1. Every emitted span fires with its required attributes under real transaction
+   workloads — the harness derives the span and attribute totals from
+   `expected_spans.json`, so no fixed "16 spans / 22 attributes" figure applies
 2. All 255+ StatsD metrics + ~50 Phase 9 metrics appear in Prometheus with non-zero values
 3. Log-trace correlation (Phase 8) produces clickable trace_id links in Loki
-4. All 10 Grafana dashboards render meaningful data (no empty panels)
+4. The 14 harness-asserted Grafana dashboards render meaningful data (no empty
+   panels); 15 are on disk
 5. Performance overhead stays within bounds (< 3% CPU, < 5MB memory)
 6. The telemetry stack survives sustained load without data loss or queue backpressure
 
@@ -37,25 +40,42 @@ Before Phases 1-9 can be considered production-ready, we need proof that:
 
 **What to do**:
 
-- Create `docker/telemetry/docker-compose.workload.yaml`:
-  - 5 xrpld validator nodes with UNL configured for each other
-  - All telemetry enabled: `[telemetry] enabled=1`, `[insight] server=otel`
-  - Full OTel stack: Collector, Tempo, Prometheus, Loki, Grafana
-  - Shared network with service discovery
+- Create `docker/telemetry/docker-compose.workload.yaml` — **as shipped this file
+  holds only the observability backend**: `otel-collector`, `tempo`,
+  `prometheus`, `loki`, `grafana`. It contains **no xrpld services**.
+  - Shared network (`workload-net`) with service discovery
 
-- Each node should:
-  - Generate validator keys at startup
-  - Configure all 5 nodes in its UNL
-  - Enable all trace categories including `trace_peer=1`
-  - Write logs to a file tailed by the OTel Collector filelog receiver
+- The 5 validators are **native `xrpld` processes**, not containers.
+  `docker/telemetry/workload/run-full-validation.sh` (`NUM_NODES=5`) generates
+  keys, writes a per-node `xrpld.cfg`, and launches each node on
+  `127.0.0.1` with sequential RPC / WS / peer ports. Each node:
+  - Gets its validator key from `generate-validator-keys.sh`
+  - Lists the other 4 nodes in `ips_fixed`
+  - Has all telemetry enabled: `[telemetry] enabled=1`, `[insight] server=otel`
+  - Enables all trace categories including `trace_peer=1`
+  - Writes logs to a file tailed by the OTel Collector filelog receiver
 
-- Include a `Makefile` target: `make telemetry-workload-up` / `make telemetry-workload-down`
+- ❌ **`make telemetry-workload-up` / `make telemetry-workload-down` were never
+  implemented.** There is no `Makefile` anywhere in the repository. The entry
+  point is `run-full-validation.sh` (with `--profile`, `--nodes`,
+  `--skip-loki`, `--skip-regression`, `--with-benchmark`). The node-count flag is
+  spelled `--nodes`, **not** `--num-nodes` — `run-full-validation.sh:80` (usage)
+  and `:100` (the `case` arm). `NUM_NODES` is the internal shell variable it
+  assigns to.
 
 **Key files**:
 
-- New: `docker/telemetry/docker-compose.workload.yaml`
+- New: `docker/telemetry/docker-compose.workload.yaml` (backend only)
 - New: `docker/telemetry/workload/generate-validator-keys.sh`
-- New: `docker/telemetry/workload/xrpld-validator.cfg.template`
+- New: `docker/telemetry/workload/run-full-validation.sh` — writes each node's
+  cfg **inline** via a heredoc at `run-full-validation.sh:242`
+  (`cat >"$NODE_DIR/xrpld.cfg" <<EOCFG`)
+- New: `docker/telemetry/workload/xrpld-validator.cfg.template` (96 lines) — it
+  **was** created and is tracked on the Phase 10 branch, but it is **unused**:
+  nothing reads it, and its `{{NODE_INDEX}}` / `{{RPC_PORT}}` / `{{OTEL_ENDPOINT}}`
+  placeholders are never substituted, because the inline heredoc above supersedes
+  it. Either wire the script to the template or delete the template — keeping both
+  guarantees they drift.
 
 ---
 
@@ -122,18 +142,32 @@ Before Phases 1-9 can be considered production-ready, we need proof that:
 - Create `docker/telemetry/workload/validate_telemetry.py`:
 
   **Span validation** (queries Tempo API):
-  - Assert all required span names appear in traces (conditional spans — `grpc.*`,
-    `ledger.acquire`, `txq.*`, `consensus.mode_change` — are marked `optional` and
-    skipped when not exercised by the workload)
-  - Assert each span has its required attributes (bare/underscore keys per the
-    2026-05-13 span-attr naming redesign; dotted `xrpl.*` reserved for resource attrs)
-  - Assert parent-child relationships are correct (`rpc.ws_message` → `rpc.process` → `rpc.command.*`)
+  - Assert every required span name in `expected_spans.json` appears in traces.
+    Conditional spans — `grpc.*`, `ledger.acquire`, `txq.*`,
+    `consensus.mode_change`, `rpc.process` — are marked `optional` and skipped
+    when the workload does not exercise them.
+  - Assert each span has its required attributes (bare/underscore keys; dotted
+    `xrpl.*` is reserved for resource attributes)
+  - Assert parent-child relationships are correct. The two live RPC trees are:
+    - HTTP: `rpc.http_request` -> `rpc.process` -> `rpc.command.*`
+    - WebSocket: `rpc.ws_message` -> `rpc.command.*` — **there is no
+      `rpc.process` on the WS path**. `rpc.process` is created only in
+      `ServerHandler::processRequest()` (`ServerHandler.cpp:705`), reached from
+      `processSession(Session, coro)`, i.e. HTTP only. Under WS-only load
+      `rpc.process` never appears, and `rpc.command.*` parents directly to
+      `rpc.ws_message`.
   - Assert span durations are reasonable (> 0, < 60s)
 
   **Metric validation** (queries Prometheus API):
-  - Assert all SpanMetrics-derived metrics are non-zero: `traces_span_metrics_calls_total`, `traces_span_metrics_duration_milliseconds_bucket`
-  - Assert all StatsD metrics are non-zero: `xrpld_LedgerMaster_Validated_Ledger_Age`, `xrpld_Peer_Finder_Active_*`, etc.
-  - Assert all Phase 9 metrics are non-zero: `xrpld_nodestore_*`, `xrpld_cache_*`, `xrpld_txq_*`, `xrpld_rpc_method_*`, `xrpld_object_count`, `xrpld_load_factor*`
+  - Assert all SpanMetrics-derived metrics are non-zero: `span_calls_total`,
+    `span_duration_milliseconds_bucket` (the connector's `namespace` is `span`,
+    not `traces_span_metrics` — `otel-collector-config.yaml:113-114`)
+  - Assert the insight-sourced metrics are non-zero: `ledgermaster_validated_ledger_age`,
+    `peer_finder_active_{inbound,outbound}_peers`, etc. — all lowercase, no
+    `xrpld_` prefix (`77f35c03db` removed the prefix and lowercased names)
+  - Assert all Phase 9 metrics are non-zero: `nodestore_state`, `cache_metrics`,
+    `txq_metrics`, `rpc_method_{started,finished,errored}_total`, `object_count`,
+    `load_factor_metrics`
   - Assert metric label cardinality is within bounds
 
   **Log-trace correlation validation** (queries Loki API):
@@ -142,7 +176,9 @@ Before Phases 1-9 can be considered production-ready, we need proof that:
   - Assert Grafana derived field links are functional
 
   **Dashboard validation**:
-  - For each of the 10 Grafana dashboards, query the dashboard API and assert no panels show "No data"
+  - For each dashboard, query the dashboard API and assert no panels show "No
+    data". There are **15 dashboards on disk**; the harness asserts **14** —
+    `log-derived-insights` is provisioned but unasserted.
 
 - Output: JSON report with pass/fail per check, suitable for CI.
 
@@ -234,17 +270,28 @@ Before Phases 1-9 can be considered production-ready, we need proof that:
 
 ## Exit Criteria — Delivered in PR #6519
 
-- [x] Multi-node validator cluster starts and reaches consensus
+- [x] 5-node validator cluster starts and reaches consensus — as native `xrpld`
+      processes driven by `run-full-validation.sh` (`NUM_NODES=5`), not from
+      docker-compose
 - [x] RPC load generator fires all traced RPC commands at configurable rates
 - [x] Transaction submitter generates 6+ transaction types at configurable TPS
-- [x] Validation suite confirms all required spans, attributes, and metrics
-- [x] Log-trace correlation validated end-to-end (Loki ↔ Tempo)
-- [x] Grafana dashboards render data (no empty panels)
-- [x] Overhead benchmark (`benchmark.sh`) measures telemetry-off vs telemetry-on deltas
+- [x] Validation suite confirms the full span / attribute / metric inventory
+      (totals computed dynamically from `expected_spans.json` /
+      `expected_metrics.json`)
+- [x] Log-trace correlation validated end-to-end (Loki <-> Tempo) — implemented
+      and passing locally, but CI runs with `--skip-loki`, so it is not gated
+- [x] All 14 harness-asserted Grafana dashboards render data (no empty panels);
+      15 on disk, `log-derived-insights` unasserted
+- [x] Overhead benchmark (`benchmark.sh`) measures telemetry-off vs telemetry-on
+      deltas
+- [ ] Benchmark shows < 3% CPU overhead, < 5MB memory overhead — needs a
+      measured run
 - [x] CI workflow runs validation on telemetry branch changes
 - [x] Validation report output is CI-parseable (JSON with exit codes)
-- [x] OTel-driven regression gate captures per-span/per-RPC/per-job timings from
-      Prometheus and compares against a committed baseline
+- [x] OTel-driven regression gate captures per-span and per-job timings from
+      Prometheus and compares against a committed baseline. Per-RPC timings are
+      **not** gated: `regression-metrics.json` defines only `spans` and
+      `job_queue` groups (FU-4).
 
 ## Follow-up Work (tracked in separate PRs)
 
@@ -253,6 +300,9 @@ Before Phases 1-9 can be considered production-ready, we need proof that:
       requires a manual baseline-refresh PR.
 - [ ] FU-4: Replace the proxy measurements in `benchmark.sh` (wall-clock curl
       p99, ledger-cadence-as-TPS, ledger-cadence-as-consensus-p95) with
-      PromQL quantile queries from the same pipeline the regression gate uses.
+      PromQL quantile queries from the same pipeline the regression gate uses,
+      and add an `rpc_methods` group to `regression-metrics.json` plus a
+      `defaults.rpc_method` block to `regression-thresholds.json` (without both,
+      any `rpc.*` metric resolves to "no threshold configured" and never gates).
 - [ ] FU-6: Grafana dashboard plotting historical baseline values keyed by
       commit SHA, for triaging noisy regressions.