rippled/OpenTelemetryPlan/Phase9_taskList.md

# Phase 9: Internal Metric Instrumentation Gap Fill — Task List

> **Status**: Future Enhancement
>
> **Goal**: Instrument xrpld to emit ~50+ metrics that exist in `get_counts`/`server_info`/TxQ/PerfLog but currently lack time-series export via the OTel or beast::insight pipelines.
>
> **Scope**: Hybrid approach — extend `beast::insight` for metrics near existing registrations, use OTel Metrics SDK `ObservableGauge` callbacks for new categories (TxQ, PerfLog, CountedObjects).
>
> **Branch**: `pratik/otel-phase9-metric-gap-fill` (from `pratik/otel-phase8-log-correlation`)
>
> **Depends on**: Phase 7 (native OTel metrics pipeline) and Phase 8 (log-trace correlation)

### Related Plan Documents

| Document                                                             | Relevance                                                      |
| -------------------------------------------------------------------- | -------------------------------------------------------------- |
| [06-implementation-phases.md](./06-implementation-phases.md)         | Phase 9 plan: motivation, architecture, exit criteria (§6.8.2) |
| [09-data-collection-reference.md](./09-data-collection-reference.md) | Current metric inventory + future metrics section              |
| [Phase7_taskList.md](./Phase7_taskList.md)                           | Prerequisite — OTel Metrics SDK and `OTelCollector` class      |
| [Phase8_taskList.md](./Phase8_taskList.md)                           | Prerequisite — log-trace correlation                           |

### Third-Party Consumer Context

These metrics serve multiple external consumer categories identified during research:

| Consumer Category         | Key Metrics They Need                                           |
| ------------------------- | --------------------------------------------------------------- |
| **Exchanges**             | Fee escalation levels, TxQ depth, settlement latency            |
| **Payment Processors**    | Load factors, io_latency, transaction throughput                |
| **Analytics Providers**   | NodeStore I/O, cache hit rates, counted objects                 |
| **Validators/Operators**  | Per-job execution times, PerfLog RPC counters, consensus timing |
| **Academic Researchers**  | Consensus performance time-series, fee market dynamics          |
| **Institutional Custody** | Server health scores, reserve calculations, node availability   |

---

## Task 9.1: NodeStore I/O Metrics

**Objective**: Export node store read/write performance as time-series metrics.

**What to do**:

- In `src/libxrpl/nodestore/Database.cpp`, extend existing `beast::insight` registrations to add:
  - Gauge: `node_reads_total` (cumulative read operations)
  - Gauge: `node_reads_hit` (fetches that found an object — not a cache hit; `fetchHitCount_` increments whatever served the fetch)
  - Gauge: `node_writes` (cumulative write operations)
  - Gauge: `node_written_bytes` (cumulative bytes written)
  - Gauge: `node_read_bytes` (cumulative bytes read)
  - Gauge: `node_reads_duration_us` (cumulative read time in microseconds)
  - Gauge: `write_load` (current write load score)
  - Gauge: `read_queue` (items in read queue)

- These values are already computed in `Database::getCountsJson()` (line ~236). Wire the same counters to `beast::insight` hooks.

**Key modified files**:

- `src/libxrpl/nodestore/Database.cpp`
- `src/libxrpl/nodestore/Database.h` (add insight members)

**Derived Prometheus metrics**: `xrpld_nodestore_reads_total`, `xrpld_nodestore_reads_hit`, `xrpld_nodestore_write_load`, etc.

**Grafana dashboard**: Add "NodeStore I/O" panel group to _Node Health_ dashboard.

---

## Task 9.2: Cache Hit Rate Metrics

**Objective**: Export SHAMap and ledger cache performance as time-series gauges.

**What to do**:

- Register OTel `ObservableGauge` callbacks (via Phase 7's `OTelCollector`) for:
  - `SLE_hit_rate` — SLE cache hit rate (0.0–1.0)
  - `ledger_hit_rate` — Ledger object cache hit rate
  - `AL_hit_rate` — AcceptedLedger cache hit rate
  - `treenode_cache_size` — SHAMap TreeNode cache size (entries)
  - `treenode_track_size` — Tracked tree nodes
  - `fullbelow_size` — FullBelow cache size

- The callback should read from the same sources as `GetCounts.cpp` handler (line ~43).

- Create a centralized `MetricsRegistry` class that holds all OTel async gauge registrations, polled at 10-second intervals by the `PeriodicMetricReader`.

**Key modified files**:

- New: `src/xrpld/telemetry/MetricsRegistry.h` / `.cpp`
- `src/xrpld/rpc/handlers/GetCounts.cpp` (extract shared access methods)
- `src/xrpld/app/main/Application.cpp` (register MetricsRegistry at startup)

**Derived Prometheus metrics**: `xrpld_cache_SLE_hit_rate`, `xrpld_cache_ledger_hit_rate`, `xrpld_cache_treenode_size`, etc.

---

## Task 9.3: Transaction Queue (TxQ) Metrics

**Objective**: Export TxQ depth, capacity, and fee escalation levels as time-series.

**What to do**:

- Register OTel `ObservableGauge` callbacks for TxQ state (from `TxQ.h` line ~143):
  - `txq_count` — Current transactions in queue
  - `txq_max_size` — Maximum queue capacity
  - `txq_in_ledger` — Transactions in current open ledger
  - `txq_per_ledger` — Expected transactions per ledger
  - `txq_reference_fee_level` — Reference fee level
  - `txq_min_processing_fee_level` — Minimum fee to get processed
  - `txq_med_fee_level` — Median fee level in queue
  - `txq_open_ledger_fee_level` — Open ledger fee escalation level

- Add to the `MetricsRegistry` (Task 9.2).

**Key modified files**:

- `src/xrpld/telemetry/MetricsRegistry.cpp` (add TxQ callbacks)
- `src/xrpld/app/tx/detail/TxQ.h` (expose metrics accessor if needed)

**Derived Prometheus metrics**: `xrpld_txq_count`, `xrpld_txq_max_size`, `xrpld_txq_open_ledger_fee_level`, etc.

**Grafana dashboard**: New _Fee Market & TxQ_ dashboard (`fee-market`).

---

## Task 9.4: PerfLog Per-RPC Method Metrics

**Objective**: Export per-RPC-method call counts and latency as OTel metrics.

**What to do**:

- Register OTel instruments for PerfLog RPC counters (from `PerfLogImp.cpp` line ~63):
  - Counter: `xrpld_rpc_method_started_total{method="<name>"}` — calls started
  - Counter: `xrpld_rpc_method_finished_total{method="<name>"}` — calls completed
  - Counter: `xrpld_rpc_method_errored_total{method="<name>"}` — calls errored
  - Histogram: `xrpld_rpc_method_duration_us{method="<name>"}` — execution time distribution

- Use OTel `Counter<int64_t>` and `Histogram<double>` instruments with `method` attribute label.

- Hook into the existing PerfLog callback mechanism rather than adding new instrumentation points.

**Key modified files**:

- `src/xrpld/perflog/detail/PerfLogImp.cpp` (add OTel instrument updates alongside existing JSON counters)
- `src/xrpld/telemetry/MetricsRegistry.cpp` (register instruments)

**Derived Prometheus metrics**: `xrpld_rpc_method_started_total{method="server_info"}`, `xrpld_rpc_method_duration_us_bucket{method="ledger"}`, etc.

**Grafana dashboard**: Add "Per-Method RPC Breakdown" panel group to _RPC Performance_ dashboard.

---

## Task 9.5: PerfLog Per-Job-Type Metrics

**Objective**: Export per-job-type queue and execution metrics.

**What to do**:

- Register OTel instruments for PerfLog job counters:
  - Counter: `xrpld_job_queued_total{job_type="<name>"}` — jobs queued
  - Counter: `xrpld_job_started_total{job_type="<name>"}` — jobs started
  - Counter: `xrpld_job_finished_total{job_type="<name>"}` — jobs completed
  - Histogram: `xrpld_job_queued_duration_us{job_type="<name>"}` — time spent waiting in queue
  - Histogram: `xrpld_job_running_duration_us{job_type="<name>"}` — execution time distribution

- Hook into PerfLog's existing job tracking alongside Task 9.4.

**Key modified files**:

- `src/xrpld/perflog/detail/PerfLogImp.cpp`
- `src/xrpld/telemetry/MetricsRegistry.cpp`

**Derived Prometheus metrics**: `xrpld_job_queued_total{job_type="ledgerData"}`, `xrpld_job_running_duration_us_bucket{job_type="transaction"}`, etc.

**Grafana dashboard**: New _Job Queue Analysis_ dashboard (`job-queue`).

---

## Task 9.6: Counted Object Instance Metrics

**Objective**: Export live instance counts for key internal object types.

**What to do**:

- Register OTel `ObservableGauge` callbacks for `CountedObject<T>` instance counts:
  - `xrpld_object_count{type="Transaction"}` — live Transaction objects
  - `xrpld_object_count{type="Ledger"}` — live Ledger objects
  - `xrpld_object_count{type="NodeObject"}` — live NodeObject instances
  - `xrpld_object_count{type="STTx"}` — serialized transaction objects
  - `xrpld_object_count{type="STLedgerEntry"}` — serialized ledger entries
  - `xrpld_object_count{type="InboundLedger"}` — ledgers being fetched
  - `xrpld_object_count{type="Pathfinder"}` — active pathfinding computations
  - `xrpld_object_count{type="PathRequest"}` — active path requests
  - `xrpld_object_count{type="HashRouterEntry"}` — hash router entries

- The `CountedObject` template already tracks these via atomic counters. The callback just reads the current counts.

**Key modified files**:

- `src/xrpld/telemetry/MetricsRegistry.cpp` (add counted object callbacks)
- `include/xrpl/basics/CountedObject.h` (may need static accessor for iteration)

**Derived Prometheus metrics**: `xrpld_object_count{type="Transaction"}`, `xrpld_object_count{type="NodeObject"}`, etc.

**Grafana dashboard**: Add "Object Instance Counts" panel to _Node Health_ dashboard.

---

## Task 9.7: Fee Escalation & Load Factor Metrics

**Objective**: Export the full load factor breakdown as time-series.

**What to do**:

- Register OTel `ObservableGauge` callbacks for load factors (from `NetworkOPs.cpp` line ~2694):
  - `load_factor` — combined transaction cost multiplier
  - `load_factor_server` — server + cluster + network contribution
  - `load_factor_local` — local server load only
  - `load_factor_net` — network-wide load estimate
  - `load_factor_cluster` — cluster peer load
  - `load_factor_fee_escalation` — open ledger fee escalation
  - `load_factor_fee_queue` — queue entry fee level

- These overlap with some existing StatsD metrics but provide finer granularity (individual factor breakdown vs. combined value).

**Key modified files**:

- `src/xrpld/telemetry/MetricsRegistry.cpp`
- `src/xrpld/app/misc/NetworkOPs.cpp` (expose load factor accessors if needed)

**Derived Prometheus metrics**: `xrpld_load_factor`, `xrpld_load_factor_fee_escalation`, etc.

**Grafana dashboard**: Add "Load Factor Breakdown" panel to _Fee Market & TxQ_ dashboard.

---

## Task 9.7a: push_metrics.py Parity — Missing Observable Gauges

**Objective**: Fill the remaining metric gaps between the external `push_metrics.py` script (in `ripplex-ansible`) and the internal OTel `MetricsRegistry` observable gauges. After this task, all metrics collected by `push_metrics.py` that CAN be collected internally are covered.

**What was done**:

- Extended existing `cacheHitRateGauge_` callback with `AL_size` (AcceptedLedger cache size)
- Extended existing `nodeStoreGauge_` callback with 4 new metrics from `getCountsJson()`:
  - `node_reads_duration_us` (JSON string — uses `std::stoll(asString())`)
  - `read_request_bundle` (native JSON int)
  - `read_threads_running` (native JSON int)
  - `read_threads_total` (native JSON int)
- Added new `xrpld_server_info` Int64ObservableGauge with 8 metrics:
  - `server_state` — operating mode as int (0=DISCONNECTED .. 4=FULL)
  - `uptime` — seconds since server start
  - `peers` — total peer count
  - `validated_ledger_seq` — validated ledger sequence (atomic read)
  - `ledger_current_index` — current open ledger sequence
  - `peer_disconnects_resources` — cumulative resource-related disconnects
  - `last_close_proposers` — from `getConsensusInfo()["previous_proposers"]`
  - `last_close_converge_time_ms` — from `getConsensusInfo()["previous_mseconds"]`
- Added new `xrpld_build_info` Int64ObservableGauge (info-style, value=1 with `version` label)
- Added new `xrpld_complete_ledgers` Int64ObservableGauge parsing comma-separated ranges into `{bound, index}` pairs
- Added new `xrpld_db_metrics` Int64ObservableGauge with 4 metrics:
  - `db_kb_total`, `db_kb_ledger`, `db_kb_transaction` (SQLite stat queries)
  - `historical_perminute` (historical ledger fetch rate)

**Key modified files**:

- `src/xrpld/telemetry/MetricsRegistry.h` (4 new gauge members, updated ASCII diagram)
- `src/xrpld/telemetry/MetricsRegistry.cpp` (4 new callback registrations, 2 callback extensions)

**Not implementable inside xrpld**:

- `connection_count_51233/51234` — OS-level port connection counts from external shell script (`get_connection.sh`)

**Derived Prometheus metrics**: `xrpld_server_info{metric="server_state"}`, `xrpld_build_info{version="2.4.0"}`, `xrpld_complete_ledgers{bound="start",index="0"}`, `xrpld_db_metrics{metric="db_kb_total"}`, etc.

**Grafana dashboard**: New panels added to _Node Health_ dashboard (`node-health.json`).

---

## Task 9.8: New Grafana Dashboards

**Objective**: Create Grafana dashboards for the new metric categories.

**What to do**:

- Create 2 new dashboards:
  1. **Fee Market & TxQ** (`fee-market`) — TxQ depth/capacity, fee levels, load factor breakdown, fee escalation timeline
  2. **Job Queue Analysis** (`job-queue`) — Per-job-type rates, queue wait times, execution times, job queue depth

- Update 2 existing dashboards:
  1. **Node Health** (`xrpld-statsd-node-health`) — Add NodeStore I/O panels, cache hit rate panels, object instance counts
  2. **RPC Performance** (`rpc-performance`) — Add per-method RPC breakdown panels

**Key modified files**:

- New: `docker/telemetry/grafana/dashboards/rippled-fee-market.json`
- New: `docker/telemetry/grafana/dashboards/rippled-job-queue.json`
- `docker/telemetry/grafana/dashboards/rippled-statsd-node-health.json`
- `docker/telemetry/grafana/dashboards/rippled-rpc-perf.json`

---

## Task 9.9: Update Documentation

**Objective**: Update telemetry reference docs with all new metrics.

**What to do**:

- Update `OpenTelemetryPlan/09-data-collection-reference.md`:
  - Add new section for OTel SDK-exported metrics (NodeStore, cache, TxQ, PerfLog, CountedObjects, load factors)
  - Update Grafana dashboard reference table (add 2 new dashboards)
  - Add Prometheus query examples for new metrics

- Update `docs/telemetry-runbook.md`:
  - Add an Alerting section covering the provisioned rules and how to wire a receiver
  - Add troubleshooting entries for new metric categories

- Provision Grafana alert rules (`docker/telemetry/grafana/provisioning/alerting/`):
  - 6 rules in 3 groups — consensus/ledger (`LedgerHistoryMismatch`, `LedgerCloseStalled`), validator (`ValidationsMissed`, `ValidationsNotChecked`), job queue (`JobQueueTxOverflow`, `JobQueueLatencyHigh`)
  - `xrpld-default` webhook contact point + flat notification policy; auto-loaded via the existing `provisioning/` mount (no docker-compose change)
  - Alerting operator docs (per-alert meaning, tuning, receiver wiring) now live in the Alerting section of `docs/telemetry-runbook.md`

**Key modified files**:

- `OpenTelemetryPlan/09-data-collection-reference.md`
- `docs/telemetry-runbook.md`
- `docker/telemetry/grafana/provisioning/alerting/{rules,contactpoints,policies}.yaml` (new)
- `docs/telemetry-runbook.md` (Alerting section added)

---

## Task 9.10: Integration Tests

**Objective**: Verify all new metrics appear in Prometheus after a test workload.

**What to do**:

- Extend the existing telemetry integration test:
  - Start xrpld with `[telemetry] enabled=1` and `[insight] server=otel`
  - Submit a batch of RPC calls and transactions
  - Query Prometheus for each new metric family
  - Assert non-zero values for: NodeStore reads, cache hit rates, TxQ count, PerfLog RPC counters, object counts, load factors

- Add unit tests for the `MetricsRegistry` class:
  - Verify callback registration and deregistration
  - Verify metric values match `get_counts` JSON output
  - Verify graceful behavior when telemetry is disabled

**Key modified files**:

- `src/test/telemetry/MetricsRegistry_test.cpp` (new)
- Existing integration test script (extend assertions)

---

## Task 9.11: Validator Health Dashboard (External Dashboard Parity)

> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity) — dashboards for Phase 7 metrics inspired by the community [xrpl-validator-dashboard](https://github.com/realgrapedrop/xrpl-validator-dashboard).
>
> **Upstream**: Phase 7 Tasks 7.9-7.16 (metrics must be emitting).
> **Downstream**: Phase 10 (dashboard load checks), Phase 11 (alert rules reference these panels).

**Objective**: Create a Grafana dashboard for validation agreement, amendment/UNL health, and state tracking.

**Dashboard**: `validator-health.json`

| Panel                      | Type       | PromQL                                                         |
| -------------------------- | ---------- | -------------------------------------------------------------- |
| Agreement % (1h)           | stat       | `xrpld_validation_agreement{metric="agreement_pct_1h"}`        |
| Agreement % (24h)          | stat       | `xrpld_validation_agreement{metric="agreement_pct_24h"}`       |
| Agreements vs Missed (1h)  | bargauge   | `agreements_1h` and `missed_1h` side by side                   |
| Agreements vs Missed (24h) | bargauge   | `agreements_24h` and `missed_24h` side by side                 |
| Validation Rate            | stat       | `rate(xrpld_validations_sent_total[5m]) * 60`                  |
| Validations Checked Rate   | stat       | `rate(xrpld_validations_checked_total[5m]) * 60`               |
| Amendment Blocked          | stat       | `xrpld_validator_health{metric="amendment_blocked"}`           |
| UNL Expiry (days)          | stat       | `xrpld_validator_health{metric="unl_expiry_days"}`             |
| Validation Quorum          | stat       | `xrpld_validator_health{metric="validation_quorum"}`           |
| State Value Timeline       | timeseries | `xrpld_state_tracking{metric="state_value"}`                   |
| Time in Current State      | stat       | `xrpld_state_tracking{metric="time_in_current_state_seconds"}` |
| State Changes Rate         | stat       | `rate(xrpld_state_changes_total[1h])`                          |
| Ledgers Closed Rate        | stat       | `rate(xrpld_ledgers_closed_total[5m]) * 60`                    |

**Dashboard conventions**: `$node` template variable for `service_instance_id` filtering, dark theme, matching existing panel sizes and color schemes.

**Key new files**: `docker/telemetry/grafana/dashboards/rippled-validator-health.json`

**Exit Criteria**:

- [ ] All 13 panels render with non-zero data during normal operation
- [ ] `$node` filter works correctly for multi-node deployments
- [ ] Amendment blocked and UNL expiry panels use color thresholds (red=blocked/expiring)

---

## Task 9.12: Peer Quality Dashboard (External Dashboard Parity)

> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity)

**Objective**: Create a Grafana dashboard for peer health aggregates.

**Dashboard**: `peer-quality.json`

| Panel                  | Type       | PromQL                                                         |
| ---------------------- | ---------- | -------------------------------------------------------------- |
| P90 Peer Latency       | timeseries | `xrpld_peer_quality{metric="peer_latency_p90_ms"}`             |
| Insane/Diverged Peers  | stat       | `xrpld_peer_quality{metric="peers_insane_count"}`              |
| Higher Version Peers % | stat       | `xrpld_peer_quality{metric="peers_higher_version_pct"}`        |
| Upgrade Recommended    | stat       | `xrpld_peer_quality{metric="upgrade_recommended"}`             |
| Resource Disconnects   | timeseries | `xrpld_Overlay_Peer_Disconnects_Charges`                       |
| Inbound vs Outbound    | bargauge   | `xrpld_Peer_Finder_Active_Inbound_Peers`, `..._Outbound_Peers` |

**Key new files**: `docker/telemetry/grafana/dashboards/rippled-peer-quality.json`

**Exit Criteria**:

- [ ] All 6 panels render correctly
- [ ] P90 latency panel shows trend over time
- [ ] Upgrade recommended panel uses color threshold (red=1, green=0)

---

## Task 9.13: Ledger Economy Dashboard Panels (External Dashboard Parity)

> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity)

**Objective**: Add "Ledger Economy" row to the existing `node-health.json` dashboard.

| Panel                | Type       | PromQL                                              |
| -------------------- | ---------- | --------------------------------------------------- |
| Base Fee (drops)     | stat       | `xrpld_ledger_economy{metric="base_fee_xrp"}`       |
| Reserve Base (drops) | stat       | `xrpld_ledger_economy{metric="reserve_base_xrp"}`   |
| Reserve Inc (drops)  | stat       | `xrpld_ledger_economy{metric="reserve_inc_xrp"}`    |
| Ledger Age           | stat       | `xrpld_ledger_economy{metric="ledger_age_seconds"}` |
| Transaction Rate     | timeseries | `xrpld_ledger_economy{metric="transaction_rate"}`   |

**Key modified files**: `docker/telemetry/grafana/dashboards/node-health.json`

**Exit Criteria**:

- [ ] 5 new panels render correctly in existing dashboard
- [ ] Fee values match `server_info` RPC output
- [ ] Transaction rate shows smooth trend (not spiky)

---

## Task 9.14: Overlay Traffic Accounting Defects (Documentation Only)

> **Status**: DOCUMENTED, NOT FIXED. Reference: [09 §6.0-§6.2](./09-data-collection-reference.md#6-known-issues)

**Objective**: Record four pre-existing overlay traffic-accounting defects so
dashboard readers are not misled. All four originate in `develop`-owned overlay
files, so **no code fix lands on this branch**.

| #   | Defect                                 | Effect                                                                                    | Fix location (NOT this branch)                   |
| --- | -------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------ |
| 1   | `mtCLUSTER` missing from `kTypeLookup` | `overhead_cluster_*` always zero; 8 panels flatline; cluster traffic counted as `unknown` | `TrafficCount.cpp:11-27`                         |
| 2   | Stale `Total` header comment           | Claims uncategorized traffic is excluded; it is included                                  | `TrafficCount.h:28-31`                           |
| 3   | `SquelchIgnored` reported with size 0  | `squelch_ignored_bytes_*` always zero, inconsistent with `SquelchSuppressed`              | `OverlayImpl.cpp:1460,1489` (+ signature change) |
| 4   | In/out byte-basis asymmetry            | `_bytes_in` vs `_bytes_out` not comparable under compression                              | `PeerImp.cpp:1079` vs `:313`                     |

**Why deferred**: Defect 3 requires widening the two
`OverlayImpl::updateSlotAndSquelch` overloads — a public signature change on
shared overlay code. Defects 1, 2 and 4 sit in `TrafficCount.{h,cpp}`, likewise
not telemetry-owned. Routing them through the telemetry chain would hide overlay
changes from overlay reviewers and couple them to a 12-PR merge timeline.

**Key modified files**: `OpenTelemetryPlan/09-data-collection-reference.md` only.

**Exit Criteria**:

- [x] Each defect documented with file:line evidence in `09` §6
- [x] `overhead_cluster_*` documented as "no data", not "no cluster traffic"
- [ ] Follow-up overlay-owned branch raised for the four code fixes
- [ ] Re-baseline any threshold keyed on `unknown_bytes_in` when defect 1 lands

---

## Task 9.15: Peer Keepalive and Discovery Instrumentation

> **Status**: NOT IMPLEMENTED — awaiting a decision on whether `XRPL_METRIC_*`
> call sites may be added to `src/xrpld/overlay/detail/PeerImp.cpp` from this
> branch. Reference: [09 §6.3](./09-data-collection-reference.md#63-peer-keepalive-and-discovery-traffic-gaps-not-implemented)

**Objective**: Make peer keepalive and peer-discovery health observable. Today
`mtPING`, `mtSTATUS_CHANGE` and `mtENDPOINTS` are byte counters only.

| Proposed metric                 | Type      | Labels                           | Record site                                         |
| ------------------------------- | --------- | -------------------------------- | --------------------------------------------------- |
| `peer_ping_rtt_ms`              | Histogram | none (see note)                  | `PeerImp.cpp:1150-1163`, where the EWMA is computed |
| `peer_ping_timeouts_total`      | Counter   | `reason="timeout"\|"bad_cookie"` | `PeerImp.cpp:762` and `:1146`                       |
| `peer_endpoints_received_total` | Counter   | `result="accepted"\|"malformed"` | `PeerImp.cpp:1265-1270`                             |

**Design notes / open questions**:

- A histogram needs an explicit bucket view: the SDK default tops out at 10000,
  and these are milliseconds. Follow the µs-ladder precedent in
  `MetricsRegistry.cpp` (see [09 § GetObject Request Path](./09-data-collection-reference.md#getobject-request-path-synchronous-countershistograms)).
- `peer_id` as a label is unbounded cardinality — rejected. A bounded
  `peer_role`-style label is the alternative if per-peer attribution is needed.
- Splitting `mtPING` out of `Category::Base` is a `TrafficCount.cpp` change and
  therefore blocked with Task 9.14.
- Per the runbook's "Adding a New Metric" contract, `_total` is reserved for
  monotonic counters; a histogram takes no suffix.

**Key files (if approved)**: `src/xrpld/overlay/detail/PeerImp.cpp`,
`09-data-collection-reference.md`, `docs/telemetry-runbook.md` § Metric Reference,
`docker/telemetry/grafana/dashboards/peer-quality.json`, and
`docker/telemetry/workload/expected_metrics.json` (**Phase 10 branch**).

**Exit Criteria**:

- [ ] Decision recorded on editing `PeerImp.cpp` from the telemetry chain
- [ ] Three instruments emitting, with an explicit histogram bucket view
- [ ] Rows added to `09` §5b, runbook § Metric Reference, and `expected_metrics.json`
- [ ] Peer Quality dashboard panels follow the Task 9.12 conventions (`$node`, Title Case, legend dimensions)
- [ ] `check_otel_naming.py` passes (Rules D and E cover the new labels)

---

## Task 9.16: PeerFinder Slot and Cache Metrics

> **Status**: NOT IMPLEMENTED. Reference: [09 §6.5](./09-data-collection-reference.md#65-peerfinder-slot-and-cache-metrics-not-implemented)

**Objective**: Export the PeerFinder slot counts and discovery-cache sizes.
Only 2 of ~17 available readings are exported today.

**What to do**: Extend the existing `Stats` struct in
`src/libxrpl/peerfinder/PeerfinderManager.cpp:227-236` with gauges for the
`Counts` accessors listed in [09 §6.5](./09-data-collection-reference.md#65-peerfinder-slot-and-cache-metrics-not-implemented)
(slot caps and frees, attempt counts, handshake pipeline depth, fixed-peer state,
network reachability), plus `Livecache::size()` and `Bootcache::size()`.

**Pipeline constraint**: `PeerfinderManager.cpp` is in `libxrpl`, which **cannot**
use the `XRPL_METRIC_*` macros. These must go through `beast::insight` —
arrow **B**, not **C**. Naming follows `GroupImp::makeName()` +
`OTelCollectorImp::formatName()`, so the `"Peer_Finder"` group yields
`peer_finder_<name>` lowercased.

**Known obstacle**: `Livecache` and `Bootcache` hold no collector reference, so
their sizes must either be read through the existing `Manager` hook or have a
collector plumbed in.

**Exit Criteria**:

- [ ] Slot caps exported so utilization (`active / max`) is computable
- [ ] Both cache sizes exported
- [ ] "Inbound vs Outbound" panel on `peer-quality` extended to show utilization %
- [ ] Rows added to `09` §2.1, runbook § Metric Reference, `expected_metrics.json` (Phase 10)

---

## Task 9.17: Peer Span Coverage (Deferred to Phase 11)

> **Status**: NOT IMPLEMENTED — design only, pending approval. Reference:
> [09 §6.4](./09-data-collection-reference.md#64-peer-span-coverage-gap-not-implemented)
> and [02 §2.3.2](./02-design-decisions.md#232-complete-span-catalog)

**Objective**: Close the gap between the `02` §2.3.2 span catalog and what
actually emits. `peer.connect`, `peer.disconnect`, `peer.message.send` and
`peer.message.receive` were catalogued from the start and never built; 11 of 13
protocol message families have no spans.

**Scope warning**: This is larger than Tasks 9.14-9.16 combined and changes the
"~37 spans" figure asserted in `09` §1.1 and in
`docker/telemetry/workload/expected_spans.json`. `trace_peer` is also **on by
default** and already flagged as high-volume, so adding per-message spans has a
volume cost that needs measuring before commitment.

**Exit Criteria**:

- [x] `02` §2.3.2 marked Live / Not built / Renamed against the real inventory
- [ ] User approval to proceed with span implementation
- [ ] Volume impact measured under `trace_peer=1` before any span is added

---

## Exit Criteria

- [ ] All ~50 new metrics visible in Prometheus via OTLP pipeline
- [ ] `MetricsRegistry` class registers/deregisters cleanly with OTel SDK
- [ ] Async gauge callbacks execute at 10s intervals without performance impact
- [ ] 2 new Grafana dashboards operational (Fee Market, Job Queue)
- [ ] 2 existing dashboards updated with new panel groups
- [ ] Integration test validates all new metric families are non-zero
- [ ] No performance regression (< 0.5% CPU overhead from new callbacks)
- [ ] Documentation updated with full new metric inventory
- [ ] Validator Health dashboard renders all 13 panels
- [ ] Peer Quality dashboard renders all 6 panels
- [ ] Ledger Economy panels added to node-health dashboard

---

## Appendix: Alerting Design

> Design for the provisioned Grafana alert rules (Task 9.9a). Previously a standalone spec; merged here so the phase plan is self-contained.

**Date:** 2026-07-06
**Branch:** `pratik/otel-phase9-metric-gap-fill` (PR #6513, Jira RIPD-5187)
**Status:** Approved

### Purpose

Phase 9 exports ~68 internal xrpld metrics and ships Grafana dashboards for
them. This adds the missing operator-facing piece: **provisioned Grafana alert
rules** that fire on the health-critical metrics phase 9 introduces. The
phase-9 task list (line 311) and Jira story RIPD-5187 both already list
"alerting rules" as a phase-9 deliverable, so this closes that gap.

Scope is deliberately narrow — the three subsystems whose failure is
node-fatal: **consensus/ledger health, validator health, job queue**. RPC/API
health is explicitly out of scope.

### Why phase 9 (not phase 11)

Every metric these alerts fire on is _born_ in phase 9
(`xrpld_ledger_history_mismatch_total`, `xrpld_ledgers_closed_total`,
`xrpld_validation_missed_total`, `xrpld_validations_checked_total`,
`xrpld_jq_trans_overflow_total`, `xrpld_job_queued_duration_us_bucket`). Alerts
belong with the metrics they watch, and this is where the dependency lives.

### Delivery

Provisioned YAML, version-controlled — matching the existing datasource /
dashboard provisioning pattern. No docker-compose change: the Grafana service
already mounts `./grafana/provisioning:/etc/grafana/provisioning:ro`, and
Grafana auto-loads `provisioning/alerting/*.yaml`.

New files under `docker/telemetry/grafana/provisioning/alerting/`:

| File                 | Purpose                                                                                                             |
| -------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `contactpoints.yaml` | One contact point `xrpld-default` (webhook to a documented placeholder; comments show how to swap for Slack/email). |
| `policies.yaml`      | Default notification policy: route all alerts → `xrpld-default`, grouped by `alertname` + `service_instance_id`.    |
| `rules.yaml`         | 6 alert rules across 3 groups (below).                                                                              |

Plus the Alerting section of `docs/telemetry-runbook.md` — operator runbook:
what each alert means, likely causes, and how to point the contact point at a
real receiver.

### Alert rules

All rules target Prometheus datasource `uid: prometheus`. Each rule uses the
Grafana rule shape: query (A) → reduce (B, last value) → threshold (C). All
`rate()`/`histogram_quantile()` expressions aggregate with
`sum by (service_instance_id)` (or `+ le`) so **each node alerts independently**.
Alert rules run headless, so they cannot use the dashboards' `$node` template
variables — they match all series and group by `service_instance_id` instead.

| Group     | Alert                 | Expression (5m window)                                                                                      | Fires                 | `for` | severity |
| --------- | --------------------- | ----------------------------------------------------------------------------------------------------------- | --------------------- | ----- | -------- |
| Consensus | LedgerHistoryMismatch | `sum by (service_instance_id)(rate(xrpld_ledger_history_mismatch_total[5m]))`                               | `> 0`                 | 5m    | critical |
| Consensus | LedgerCloseStalled    | `sum by (service_instance_id)(rate(xrpld_ledgers_closed_total[5m]))`                                        | `< 0.001` (≈0)        | 3m    | critical |
| Validator | ValidationsMissed     | `sum by (service_instance_id)(rate(xrpld_validation_missed_total[5m]))`                                     | `> 0`                 | 5m    | warning  |
| Validator | ValidationsNotChecked | `sum by (service_instance_id)(rate(xrpld_validations_checked_total[5m]))`                                   | `< 0.001` (≈0)        | 5m    | warning  |
| Job queue | JobQueueTxOverflow    | `sum by (service_instance_id)(rate(xrpld_jq_trans_overflow_total[5m]))`                                     | `> 0`                 | 5m    | warning  |
| Job queue | JobQueueLatencyHigh   | `histogram_quantile(0.99, sum by (le, service_instance_id)(rate(xrpld_job_queued_duration_us_bucket[5m])))` | `> 1000000` (µs = 1s) | 5m    | warning  |

Each rule carries labels `severity` and `category` (consensus/validator/jobqueue)
and annotations `summary` + `description` (with `{{ $labels.service_instance_id }}`
and `{{ $values.B.Value }}` interpolation).

#### Threshold rationale

- **LedgerCloseStalled `< 0.001` for 3m**: healthy nodes close a ledger every
  ~3-5s; a 5m rate decaying to ~0 means the node is stuck. The epsilon (not
  exact `0`) avoids float rate-noise suppressing the alert.
- **JobQueueLatencyHigh 1s p99**: a default starting point, easy to tune — jobs
  queued >1s at p99 indicate the node is saturated.
- Others are `> 0` on error/miss counters: any sustained nonzero rate is
  actionable.

### Non-goals / YAGNI

- No per-alert silencing schedules, no mute timings.
- No RPC/API, overlay, or fee-market alerts (dashboards cover those visually).
- Single contact point — multi-receiver routing is left to the operator.

### Verification

1. `yamllint` (or `python -c yaml.safe_load`) on all three YAML files.
2. `docker compose -f docker/telemetry/docker-compose.yml config -q` still parses.
3. Optional live check: start stack, `GET /api/v1/provisioning/alert-rules`
   returns the 6 rules; Grafana logs show no provisioning errors.
4. Code-review pass (subagent) against phase conventions before commit.