rippled/OpenTelemetryPlan/Phase9_taskList.md

Phase 9: Internal Metric Instrumentation Gap Fill — Task List

Status: Future Enhancement

Goal: Instrument xrpld to emit ~50+ metrics that exist in get_counts/server_info/TxQ/PerfLog but currently lack time-series export via the OTel or beast::insight pipelines.

Scope: Hybrid approach — extend beast::insight for metrics near existing registrations, use OTel Metrics SDK ObservableGauge callbacks for new categories (TxQ, PerfLog, CountedObjects).

Branch: pratik/otel-phase9-metric-gap-fill (from pratik/otel-phase8-log-correlation)

Depends on: Phase 7 (native OTel metrics pipeline) and Phase 8 (log-trace correlation)

Related Plan Documents

Document	Relevance
06-implementation-phases.md	Phase 9 plan: motivation, architecture, exit criteria (§6.8.2)
09-data-collection-reference.md	Current metric inventory + future metrics section
Phase7_taskList.md	Prerequisite — OTel Metrics SDK and OTelCollector class
Phase8_taskList.md	Prerequisite — log-trace correlation

Third-Party Consumer Context

These metrics serve multiple external consumer categories identified during research:

Consumer Category	Key Metrics They Need
Exchanges	Fee escalation levels, TxQ depth, settlement latency
Payment Processors	Load factors, io_latency, transaction throughput
Analytics Providers	NodeStore I/O, cache hit rates, counted objects
Validators/Operators	Per-job execution times, PerfLog RPC counters, consensus timing
Academic Researchers	Consensus performance time-series, fee market dynamics
Institutional Custody	Server health scores, reserve calculations, node availability

---

Task 9.1: NodeStore I/O Metrics

Objective: Export node store read/write performance as time-series metrics.

What to do:

• In src/libxrpl/nodestore/Database.cpp, extend existing beast::insight registrations to add:

  • Gauge: node_reads_total (cumulative read operations)
  • Gauge: node_reads_hit (fetches that found an object — not a cache hit; fetchHitCount_ increments whatever served the fetch)
  • Gauge: node_writes (cumulative write operations)
  • Gauge: node_written_bytes (cumulative bytes written)
  • Gauge: node_read_bytes (cumulative bytes read)
  • Gauge: node_reads_duration_us (cumulative read time in microseconds)
  • Gauge: write_load (current write load score)
  • Gauge: read_queue (items in read queue)

• These values are already computed in Database::getCountsJson() (line ~236). Wire the same counters to beast::insight hooks.

Key modified files:

• src/libxrpl/nodestore/Database.cpp
• src/libxrpl/nodestore/Database.h (add insight members)

Derived Prometheus metrics: xrpld_nodestore_reads_total, xrpld_nodestore_reads_hit, xrpld_nodestore_write_load, etc.

Grafana dashboard: Add "NodeStore I/O" panel group to Node Health dashboard.

---

Task 9.2: Cache Hit Rate Metrics

Objective: Export SHAMap and ledger cache performance as time-series gauges.

What to do:

• Register OTel ObservableGauge callbacks (via Phase 7's OTelCollector) for:

  • SLE_hit_rate — SLE cache hit rate (0.0–1.0)
  • ledger_hit_rate — Ledger object cache hit rate
  • AL_hit_rate — AcceptedLedger cache hit rate
  • treenode_cache_size — SHAMap TreeNode cache size (entries)
  • treenode_track_size — Tracked tree nodes
  • fullbelow_size — FullBelow cache size

• The callback should read from the same sources as GetCounts.cpp handler (line ~43).

• Create a centralized MetricsRegistry class that holds all OTel async gauge registrations, polled at 10-second intervals by the PeriodicMetricReader.

Key modified files:

• New: src/xrpld/telemetry/MetricsRegistry.h / .cpp
• src/xrpld/rpc/handlers/GetCounts.cpp (extract shared access methods)
• src/xrpld/app/main/Application.cpp (register MetricsRegistry at startup)

Derived Prometheus metrics: xrpld_cache_SLE_hit_rate, xrpld_cache_ledger_hit_rate, xrpld_cache_treenode_size, etc.

---

Task 9.3: Transaction Queue (TxQ) Metrics

Objective: Export TxQ depth, capacity, and fee escalation levels as time-series.

What to do:

• Register OTel ObservableGauge callbacks for TxQ state (from TxQ.h line ~143):

  • txq_count — Current transactions in queue
  • txq_max_size — Maximum queue capacity
  • txq_in_ledger — Transactions in current open ledger
  • txq_per_ledger — Expected transactions per ledger
  • txq_reference_fee_level — Reference fee level
  • txq_min_processing_fee_level — Minimum fee to get processed
  • txq_med_fee_level — Median fee level in queue
  • txq_open_ledger_fee_level — Open ledger fee escalation level

• Add to the MetricsRegistry (Task 9.2).

Key modified files:

• src/xrpld/telemetry/MetricsRegistry.cpp (add TxQ callbacks)
• src/xrpld/app/tx/detail/TxQ.h (expose metrics accessor if needed)

Derived Prometheus metrics: xrpld_txq_count, xrpld_txq_max_size, xrpld_txq_open_ledger_fee_level, etc.

Grafana dashboard: New Fee Market & TxQ dashboard (fee-market).

---

Task 9.4: PerfLog Per-RPC Method Metrics

Objective: Export per-RPC-method call counts and latency as OTel metrics.

What to do:

• Register OTel instruments for PerfLog RPC counters (from PerfLogImp.cpp line ~63):

  • Counter: xrpld_rpc_method_started_total{method="<name>"} — calls started
  • Counter: xrpld_rpc_method_finished_total{method="<name>"} — calls completed
  • Counter: xrpld_rpc_method_errored_total{method="<name>"} — calls errored
  • Histogram: xrpld_rpc_method_duration_us{method="<name>"} — execution time distribution

• Use OTel Counter<int64_t> and Histogram<double> instruments with method attribute label.

• Hook into the existing PerfLog callback mechanism rather than adding new instrumentation points.

Key modified files:

• src/xrpld/perflog/detail/PerfLogImp.cpp (add OTel instrument updates alongside existing JSON counters)
• src/xrpld/telemetry/MetricsRegistry.cpp (register instruments)

Derived Prometheus metrics: xrpld_rpc_method_started_total{method="server_info"}, xrpld_rpc_method_duration_us_bucket{method="ledger"}, etc.

Grafana dashboard: Add "Per-Method RPC Breakdown" panel group to RPC Performance dashboard.

---

Task 9.5: PerfLog Per-Job-Type Metrics

Objective: Export per-job-type queue and execution metrics.

What to do:

• Register OTel instruments for PerfLog job counters:

  • Counter: xrpld_job_queued_total{job_type="<name>"} — jobs queued
  • Counter: xrpld_job_started_total{job_type="<name>"} — jobs started
  • Counter: xrpld_job_finished_total{job_type="<name>"} — jobs completed
  • Histogram: xrpld_job_queued_duration_us{job_type="<name>"} — time spent waiting in queue
  • Histogram: xrpld_job_running_duration_us{job_type="<name>"} — execution time distribution

• Hook into PerfLog's existing job tracking alongside Task 9.4.

Key modified files:

• src/xrpld/perflog/detail/PerfLogImp.cpp
• src/xrpld/telemetry/MetricsRegistry.cpp

Derived Prometheus metrics: xrpld_job_queued_total{job_type="ledgerData"}, xrpld_job_running_duration_us_bucket{job_type="transaction"}, etc.

Grafana dashboard: New Job Queue Analysis dashboard (job-queue).

---

Task 9.6: Counted Object Instance Metrics

Objective: Export live instance counts for key internal object types.

What to do:

• Register OTel ObservableGauge callbacks for CountedObject<T> instance counts:

  • xrpld_object_count{type="Transaction"} — live Transaction objects
  • xrpld_object_count{type="Ledger"} — live Ledger objects
  • xrpld_object_count{type="NodeObject"} — live NodeObject instances
  • xrpld_object_count{type="STTx"} — serialized transaction objects
  • xrpld_object_count{type="STLedgerEntry"} — serialized ledger entries
  • xrpld_object_count{type="InboundLedger"} — ledgers being fetched
  • xrpld_object_count{type="Pathfinder"} — active pathfinding computations
  • xrpld_object_count{type="PathRequest"} — active path requests
  • xrpld_object_count{type="HashRouterEntry"} — hash router entries

• The CountedObject template already tracks these via atomic counters. The callback just reads the current counts.

Key modified files:

• src/xrpld/telemetry/MetricsRegistry.cpp (add counted object callbacks)
• include/xrpl/basics/CountedObject.h (may need static accessor for iteration)

Derived Prometheus metrics: xrpld_object_count{type="Transaction"}, xrpld_object_count{type="NodeObject"}, etc.

Grafana dashboard: Add "Object Instance Counts" panel to Node Health dashboard.

---

Task 9.7: Fee Escalation & Load Factor Metrics

Objective: Export the full load factor breakdown as time-series.

What to do:

• Register OTel ObservableGauge callbacks for load factors (from NetworkOPs.cpp line ~2694):

  • load_factor — combined transaction cost multiplier
  • load_factor_server — server + cluster + network contribution
  • load_factor_local — local server load only
  • load_factor_net — network-wide load estimate
  • load_factor_cluster — cluster peer load
  • load_factor_fee_escalation — open ledger fee escalation
  • load_factor_fee_queue — queue entry fee level

• These overlap with some existing StatsD metrics but provide finer granularity (individual factor breakdown vs. combined value).

Key modified files:

• src/xrpld/telemetry/MetricsRegistry.cpp
• src/xrpld/app/misc/NetworkOPs.cpp (expose load factor accessors if needed)

Derived Prometheus metrics: xrpld_load_factor, xrpld_load_factor_fee_escalation, etc.

Grafana dashboard: Add "Load Factor Breakdown" panel to Fee Market & TxQ dashboard.

---

Task 9.7a: push_metrics.py Parity — Missing Observable Gauges

Objective: Fill the remaining metric gaps between the external push_metrics.py script (in ripplex-ansible) and the internal OTel MetricsRegistry observable gauges. After this task, all metrics collected by push_metrics.py that CAN be collected internally are covered.

What was done:

• Extended existing cacheHitRateGauge_ callback with AL_size (AcceptedLedger cache size)
• Extended existing nodeStoreGauge_ callback with 4 new metrics from getCountsJson():
  • node_reads_duration_us (JSON string — uses std::stoll(asString()))
  • read_request_bundle (native JSON int)
  • read_threads_running (native JSON int)
  • read_threads_total (native JSON int)
• Added new xrpld_server_info Int64ObservableGauge with 8 metrics:
  • server_state — operating mode as int (0=DISCONNECTED .. 4=FULL)
  • uptime — seconds since server start
  • peers — total peer count
  • validated_ledger_seq — validated ledger sequence (atomic read)
  • ledger_current_index — current open ledger sequence
  • peer_disconnects_resources — cumulative resource-related disconnects
  • last_close_proposers — from getConsensusInfo()["previous_proposers"]
  • last_close_converge_time_ms — from getConsensusInfo()["previous_mseconds"]
• Added new xrpld_build_info Int64ObservableGauge (info-style, value=1 with version label)
• Added new xrpld_complete_ledgers Int64ObservableGauge parsing comma-separated ranges into {bound, index} pairs
• Added new xrpld_db_metrics Int64ObservableGauge with 4 metrics:
  • db_kb_total, db_kb_ledger, db_kb_transaction (SQLite stat queries)
  • historical_perminute (historical ledger fetch rate)

Key modified files:

• src/xrpld/telemetry/MetricsRegistry.h (4 new gauge members, updated ASCII diagram)
• src/xrpld/telemetry/MetricsRegistry.cpp (4 new callback registrations, 2 callback extensions)

Not implementable inside xrpld:

• connection_count_51233/51234 — OS-level port connection counts from external shell script (get_connection.sh)

Derived Prometheus metrics: xrpld_server_info{metric="server_state"}, xrpld_build_info{version="2.4.0"}, xrpld_complete_ledgers{bound="start",index="0"}, xrpld_db_metrics{metric="db_kb_total"}, etc.

Grafana dashboard: New panels added to Node Health dashboard (node-health.json).

---

Task 9.8: New Grafana Dashboards

Objective: Create Grafana dashboards for the new metric categories.

What to do:

• Create 2 new dashboards:

  1. Fee Market & TxQ (fee-market) — TxQ depth/capacity, fee levels, load factor breakdown, fee escalation timeline
  2. Job Queue Analysis (job-queue) — Per-job-type rates, queue wait times, execution times, job queue depth

• Update 2 existing dashboards:

  1. Node Health (xrpld-statsd-node-health) — Add NodeStore I/O panels, cache hit rate panels, object instance counts
  2. RPC Performance (rpc-performance) — Add per-method RPC breakdown panels

Key modified files:

• New: docker/telemetry/grafana/dashboards/rippled-fee-market.json
• New: docker/telemetry/grafana/dashboards/rippled-job-queue.json
• docker/telemetry/grafana/dashboards/rippled-statsd-node-health.json
• docker/telemetry/grafana/dashboards/rippled-rpc-perf.json

---

Task 9.9: Update Documentation

Objective: Update telemetry reference docs with all new metrics.

What to do:

• Update OpenTelemetryPlan/09-data-collection-reference.md:

  • Add new section for OTel SDK-exported metrics (NodeStore, cache, TxQ, PerfLog, CountedObjects, load factors)
  • Update Grafana dashboard reference table (add 2 new dashboards)
  • Add Prometheus query examples for new metrics

• Update docs/telemetry-runbook.md:

  • Add an Alerting section covering the provisioned rules and how to wire a receiver
  • Add troubleshooting entries for new metric categories

• Provision Grafana alert rules (docker/telemetry/grafana/provisioning/alerting/):

  • 6 rules in 3 groups — consensus/ledger (LedgerHistoryMismatch, LedgerCloseStalled), validator (ValidationsMissed, ValidationsNotChecked), job queue (JobQueueTxOverflow, JobQueueLatencyHigh)
  • xrpld-default webhook contact point + flat notification policy; auto-loaded via the existing provisioning/ mount (no docker-compose change)
  • Alerting operator docs (per-alert meaning, tuning, receiver wiring) now live in the Alerting section of docs/telemetry-runbook.md

Key modified files:

• OpenTelemetryPlan/09-data-collection-reference.md
• docs/telemetry-runbook.md
• docker/telemetry/grafana/provisioning/alerting/{rules,contactpoints,policies}.yaml (new)
• docs/telemetry-runbook.md (Alerting section added)

---

Task 9.10: Integration Tests

Objective: Verify all new metrics appear in Prometheus after a test workload.

What to do:

• Extend the existing telemetry integration test:

  • Start xrpld with [telemetry] enabled=1 and [insight] server=otel
  • Submit a batch of RPC calls and transactions
  • Query Prometheus for each new metric family
  • Assert non-zero values for: NodeStore reads, cache hit rates, TxQ count, PerfLog RPC counters, object counts, load factors

• Add unit tests for the MetricsRegistry class:

  • Verify callback registration and deregistration
  • Verify metric values match get_counts JSON output
  • Verify graceful behavior when telemetry is disabled

Key modified files:

• src/test/telemetry/MetricsRegistry_test.cpp (new)
• Existing integration test script (extend assertions)

---

Task 9.11: Validator Health Dashboard (External Dashboard Parity)

Source: External Dashboard Parity — dashboards for Phase 7 metrics inspired by the community xrpl-validator-dashboard.

Upstream: Phase 7 Tasks 7.9-7.16 (metrics must be emitting). Downstream: Phase 10 (dashboard load checks), Phase 11 (alert rules reference these panels).

Objective: Create a Grafana dashboard for validation agreement, amendment/UNL health, and state tracking.

Dashboard: validator-health.json

Panel	Type	PromQL
Agreement % (1h)	stat	xrpld_validation_agreement{metric="agreement_pct_1h"}
Agreement % (24h)	stat	xrpld_validation_agreement{metric="agreement_pct_24h"}
Agreements vs Missed (1h)	bargauge	agreements_1h and missed_1h side by side
Agreements vs Missed (24h)	bargauge	agreements_24h and missed_24h side by side
Validation Rate	stat	rate(xrpld_validations_sent_total[5m]) * 60
Validations Checked Rate	stat	rate(xrpld_validations_checked_total[5m]) * 60
Amendment Blocked	stat	xrpld_validator_health{metric="amendment_blocked"}
UNL Expiry (days)	stat	xrpld_validator_health{metric="unl_expiry_days"}
Validation Quorum	stat	xrpld_validator_health{metric="validation_quorum"}
State Value Timeline	timeseries	xrpld_state_tracking{metric="state_value"}
Time in Current State	stat	xrpld_state_tracking{metric="time_in_current_state_seconds"}
State Changes Rate	stat	rate(xrpld_state_changes_total[1h])
Ledgers Closed Rate	stat	rate(xrpld_ledgers_closed_total[5m]) * 60

Dashboard conventions: $node template variable for service_instance_id filtering, dark theme, matching existing panel sizes and color schemes.

Key new files: docker/telemetry/grafana/dashboards/rippled-validator-health.json

Exit Criteria:

• All 13 panels render with non-zero data during normal operation
• $node filter works correctly for multi-node deployments
• Amendment blocked and UNL expiry panels use color thresholds (red=blocked/expiring)

---

Task 9.12: Peer Quality Dashboard (External Dashboard Parity)

Source: External Dashboard Parity

Objective: Create a Grafana dashboard for peer health aggregates.

Dashboard: peer-quality.json

Panel	Type	PromQL
P90 Peer Latency	timeseries	xrpld_peer_quality{metric="peer_latency_p90_ms"}
Insane/Diverged Peers	stat	xrpld_peer_quality{metric="peers_insane_count"}
Higher Version Peers %	stat	xrpld_peer_quality{metric="peers_higher_version_pct"}
Upgrade Recommended	stat	xrpld_peer_quality{metric="upgrade_recommended"}
Resource Disconnects	timeseries	xrpld_Overlay_Peer_Disconnects_Charges
Inbound vs Outbound	bargauge	xrpld_Peer_Finder_Active_Inbound_Peers, ..._Outbound_Peers

Key new files: docker/telemetry/grafana/dashboards/rippled-peer-quality.json

Exit Criteria:

• All 6 panels render correctly
• P90 latency panel shows trend over time
• Upgrade recommended panel uses color threshold (red=1, green=0)

---

Task 9.13: Ledger Economy Dashboard Panels (External Dashboard Parity)

Source: External Dashboard Parity

Objective: Add "Ledger Economy" row to the existing node-health.json dashboard.

Panel	Type	PromQL
Base Fee (drops)	stat	xrpld_ledger_economy{metric="base_fee_xrp"}
Reserve Base (drops)	stat	xrpld_ledger_economy{metric="reserve_base_xrp"}
Reserve Inc (drops)	stat	xrpld_ledger_economy{metric="reserve_inc_xrp"}
Ledger Age	stat	xrpld_ledger_economy{metric="ledger_age_seconds"}
Transaction Rate	timeseries	xrpld_ledger_economy{metric="transaction_rate"}

Key modified files: docker/telemetry/grafana/dashboards/node-health.json

Exit Criteria:

• 5 new panels render correctly in existing dashboard
• Fee values match server_info RPC output
• Transaction rate shows smooth trend (not spiky)

---

Task 9.14: Overlay Traffic Accounting Defects (Documentation Only)

Status: DOCUMENTED, NOT FIXED. Reference: 09 §6.0-§6.2

Objective: Record four pre-existing overlay traffic-accounting defects so dashboard readers are not misled. All four originate in develop-owned overlay files, so no code fix lands on this branch.

#	Defect	Effect	Fix location (NOT this branch)
1	mtCLUSTER missing from kTypeLookup	overhead_cluster_* always zero; 8 panels flatline; cluster traffic counted as unknown	TrafficCount.cpp:11-27
2	Stale Total header comment	Claims uncategorized traffic is excluded; it is included	TrafficCount.h:28-31
3	SquelchIgnored reported with size 0	squelch_ignored_bytes_* always zero, inconsistent with SquelchSuppressed	OverlayImpl.cpp:1460,1489 (+ signature change)
4	In/out byte-basis asymmetry	_bytes_in vs _bytes_out not comparable under compression	PeerImp.cpp:1079 vs :313

Why deferred: Defect 3 requires widening the two OverlayImpl::updateSlotAndSquelch overloads — a public signature change on shared overlay code. Defects 1, 2 and 4 sit in TrafficCount.{h,cpp}, likewise not telemetry-owned. Routing them through the telemetry chain would hide overlay changes from overlay reviewers and couple them to a 12-PR merge timeline.

Key modified files: OpenTelemetryPlan/09-data-collection-reference.md only.

Exit Criteria:

• Each defect documented with file:line evidence in 09 §6
• overhead_cluster_* documented as "no data", not "no cluster traffic"
• Follow-up overlay-owned branch raised for the four code fixes
• Re-baseline any threshold keyed on unknown_bytes_in when defect 1 lands

---

Task 9.15: Peer Keepalive and Discovery Instrumentation

Status: NOT IMPLEMENTED — awaiting a decision on whether XRPL_METRIC_* call sites may be added to src/xrpld/overlay/detail/PeerImp.cpp from this branch. Reference: 09 §6.3

Objective: Make peer keepalive and peer-discovery health observable. Today mtPING, mtSTATUS_CHANGE and mtENDPOINTS are byte counters only.

Proposed metric	Type	Labels	Record site
peer_ping_rtt_ms	Histogram	none (see note)	PeerImp.cpp:1150-1163, where the EWMA is computed
peer_ping_timeouts_total	Counter	reason="timeout"|"bad_cookie"	PeerImp.cpp:762 and :1146
peer_endpoints_received_total	Counter	result="accepted"|"malformed"	PeerImp.cpp:1265-1270

Design notes / open questions:

• A histogram needs an explicit bucket view: the SDK default tops out at 10000, and these are milliseconds. Follow the µs-ladder precedent in MetricsRegistry.cpp (see 09 § GetObject Request Path).
• peer_id as a label is unbounded cardinality — rejected. A bounded peer_role-style label is the alternative if per-peer attribution is needed.
• Splitting mtPING out of Category::Base is a TrafficCount.cpp change and therefore blocked with Task 9.14.
• Per the runbook's "Adding a New Metric" contract, _total is reserved for monotonic counters; a histogram takes no suffix.

Key files (if approved): src/xrpld/overlay/detail/PeerImp.cpp, 09-data-collection-reference.md, docs/telemetry-runbook.md § Metric Reference, docker/telemetry/grafana/dashboards/peer-quality.json, and docker/telemetry/workload/expected_metrics.json (Phase 10 branch).

Exit Criteria:

• Decision recorded on editing PeerImp.cpp from the telemetry chain
• Three instruments emitting, with an explicit histogram bucket view
• Rows added to 09 §5b, runbook § Metric Reference, and expected_metrics.json
• Peer Quality dashboard panels follow the Task 9.12 conventions ($node, Title Case, legend dimensions)
• check_otel_naming.py passes (Rules D and E cover the new labels)

---

Task 9.16: PeerFinder Slot and Cache Metrics

Status: NOT IMPLEMENTED. Reference: 09 §6.5

Objective: Export the PeerFinder slot counts and discovery-cache sizes. Only 2 of ~17 available readings are exported today.

What to do: Extend the existing Stats struct in src/libxrpl/peerfinder/PeerfinderManager.cpp:227-236 with gauges for the Counts accessors listed in 09 §6.5 (slot caps and frees, attempt counts, handshake pipeline depth, fixed-peer state, network reachability), plus Livecache::size() and Bootcache::size().

Pipeline constraint: PeerfinderManager.cpp is in libxrpl, which cannot use the XRPL_METRIC_* macros. These must go through beast::insight — arrow B, not C. Naming follows GroupImp::makeName() + OTelCollectorImp::formatName(), so the "Peer_Finder" group yields peer_finder_<name> lowercased.

Known obstacle: Livecache and Bootcache hold no collector reference, so their sizes must either be read through the existing Manager hook or have a collector plumbed in.

Exit Criteria:

• Slot caps exported so utilization (active / max) is computable
• Both cache sizes exported
• "Inbound vs Outbound" panel on peer-quality extended to show utilization %
• Rows added to 09 §2.1, runbook § Metric Reference, expected_metrics.json (Phase 10)

---

Task 9.17: Peer Span Coverage (Deferred to Phase 11)

Status: NOT IMPLEMENTED — design only, pending approval. Reference: 09 §6.4 and 02 §2.3.2

Objective: Close the gap between the 02 §2.3.2 span catalog and what actually emits. peer.connect, peer.disconnect, peer.message.send and peer.message.receive were catalogued from the start and never built; 11 of 13 protocol message families have no spans.

Scope warning: This is larger than Tasks 9.14-9.16 combined and changes the "~37 spans" figure asserted in 09 §1.1 and in docker/telemetry/workload/expected_spans.json. trace_peer is also on by default and already flagged as high-volume, so adding per-message spans has a volume cost that needs measuring before commitment.

Exit Criteria:

• 02 §2.3.2 marked Live / Not built / Renamed against the real inventory
• User approval to proceed with span implementation
• Volume impact measured under trace_peer=1 before any span is added

---

Exit Criteria

• All ~50 new metrics visible in Prometheus via OTLP pipeline
• MetricsRegistry class registers/deregisters cleanly with OTel SDK
• Async gauge callbacks execute at 10s intervals without performance impact
• 2 new Grafana dashboards operational (Fee Market, Job Queue)
• 2 existing dashboards updated with new panel groups
• Integration test validates all new metric families are non-zero
• No performance regression (< 0.5% CPU overhead from new callbacks)
• Documentation updated with full new metric inventory
• Validator Health dashboard renders all 13 panels
• Peer Quality dashboard renders all 6 panels
• Ledger Economy panels added to node-health dashboard

---

Appendix: Alerting Design

Design for the provisioned Grafana alert rules (Task 9.9a). Previously a standalone spec; merged here so the phase plan is self-contained.

Date: 2026-07-06 Branch: pratik/otel-phase9-metric-gap-fill (PR #6513, Jira RIPD-5187) Status: Approved

Purpose

Phase 9 exports ~68 internal xrpld metrics and ships Grafana dashboards for them. This adds the missing operator-facing piece: provisioned Grafana alert rules that fire on the health-critical metrics phase 9 introduces. The phase-9 task list (line 311) and Jira story RIPD-5187 both already list "alerting rules" as a phase-9 deliverable, so this closes that gap.

Scope is deliberately narrow — the three subsystems whose failure is node-fatal: consensus/ledger health, validator health, job queue. RPC/API health is explicitly out of scope.

Why phase 9 (not phase 11)

Every metric these alerts fire on is born in phase 9 (xrpld_ledger_history_mismatch_total, xrpld_ledgers_closed_total, xrpld_validation_missed_total, xrpld_validations_checked_total, xrpld_jq_trans_overflow_total, xrpld_job_queued_duration_us_bucket). Alerts belong with the metrics they watch, and this is where the dependency lives.

Delivery

Provisioned YAML, version-controlled — matching the existing datasource / dashboard provisioning pattern. No docker-compose change: the Grafana service already mounts ./grafana/provisioning:/etc/grafana/provisioning:ro, and Grafana auto-loads provisioning/alerting/*.yaml.

New files under docker/telemetry/grafana/provisioning/alerting/:

File	Purpose
contactpoints.yaml	One contact point xrpld-default (webhook to a documented placeholder; comments show how to swap for Slack/email).
policies.yaml	Default notification policy: route all alerts → xrpld-default, grouped by alertname + service_instance_id.
rules.yaml	6 alert rules across 3 groups (below).

Plus the Alerting section of docs/telemetry-runbook.md — operator runbook: what each alert means, likely causes, and how to point the contact point at a real receiver.

Alert rules

All rules target Prometheus datasource uid: prometheus. Each rule uses the Grafana rule shape: query (A) → reduce (B, last value) → threshold (C). All rate()/histogram_quantile() expressions aggregate with sum by (service_instance_id) (or + le) so each node alerts independently. Alert rules run headless, so they cannot use the dashboards' $node template variables — they match all series and group by service_instance_id instead.

Group	Alert	Expression (5m window)	Fires	for	severity
Consensus	LedgerHistoryMismatch	sum by (service_instance_id)(rate(xrpld_ledger_history_mismatch_total[5m]))	> 0	5m	critical
Consensus	LedgerCloseStalled	sum by (service_instance_id)(rate(xrpld_ledgers_closed_total[5m]))	< 0.001 (≈0)	3m	critical
Validator	ValidationsMissed	sum by (service_instance_id)(rate(xrpld_validation_missed_total[5m]))	> 0	5m	warning
Validator	ValidationsNotChecked	sum by (service_instance_id)(rate(xrpld_validations_checked_total[5m]))	< 0.001 (≈0)	5m	warning
Job queue	JobQueueTxOverflow	sum by (service_instance_id)(rate(xrpld_jq_trans_overflow_total[5m]))	> 0	5m	warning
Job queue	JobQueueLatencyHigh	histogram_quantile(0.99, sum by (le, service_instance_id)(rate(xrpld_job_queued_duration_us_bucket[5m])))	> 1000000 (µs = 1s)	5m	warning

Each rule carries labels severity and category (consensus/validator/jobqueue) and annotations summary + description (with {{ $labels.service_instance_id }} and {{ $values.B.Value }} interpolation).

Threshold rationale

• LedgerCloseStalled < 0.001 for 3m: healthy nodes close a ledger every ~3-5s; a 5m rate decaying to ~0 means the node is stuck. The epsilon (not exact 0) avoids float rate-noise suppressing the alert.
• JobQueueLatencyHigh 1s p99: a default starting point, easy to tune — jobs queued >1s at p99 indicate the node is saturated.
• Others are > 0 on error/miss counters: any sustained nonzero rate is actionable.

Non-goals / YAGNI

• No per-alert silencing schedules, no mute timings.
• No RPC/API, overlay, or fee-market alerts (dashboards cover those visually).
• Single contact point — multi-receiver routing is left to the operator.

Verification

1. yamllint (or python -c yaml.safe_load) on all three YAML files.
2. docker compose -f docker/telemetry/docker-compose.yml config -q still parses.
3. Optional live check: start stack, GET /api/v1/provisioning/alert-rules returns the 6 rules; Grafana logs show no provisioning errors.
4. Code-review pass (subagent) against phase conventions before commit.