ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-21 22:30:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(telemetry): wire A5-A7 and B1 signals through the pipeline

Browse Source 
Registers the new gauges, renders them, asserts them and documents them, so
each signal reaches an operator rather than stopping at the emit site:

- MetricsRegistry: gauge registration for ledger_quorum_publish,
  nodestore_latency, peer_ledger_supply, peerfinder_slot_census and
  amendment_block, each guarded by the detached-callbacks check and
  tolerant of services that are not ready yet.
- Ledger Sync Health dashboard: panels for the new signals, filtered by
  the node template variable like every other board.
- Workload validation: the new series are asserted, so a signal that
  regresses to absent fails CI. Signals the local cluster structurally
  cannot produce, such as a replay fallback or an amendment block, are
  noted rather than asserted, which would fail red on a healthy run.
- Reference, runbook and glossary entries, including the diagnosis order
  for a node that has peers and validators but never validates.
- Regenerated levelization baseline: three new one-way edges from the
  telemetry and test modules, no new cycles.

Also drops an unused cstddef include from the macro tests, which the
include checker rejects.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pratik Mankawde
2026-07-25 16:15:26 +01:00
parent c0f223ee11
commit 827525c86b
10 changed files with 3660 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.github/scripts/levelization/results/ordering.txt vendored
View File

@@ -200,6 +200,8 @@ tests.libxrpl > xrpl.basics
tests.libxrpl > xrpl.config
tests.libxrpl > xrpl.core
tests.libxrpl > xrpld.app
tests.libxrpl > xrpld.overlay
tests.libxrpl > xrpld.peerfinder
tests.libxrpl > xrpld.telemetry
tests.libxrpl > xrpl.json
tests.libxrpl > xrpl.ledger
@@ -336,6 +338,7 @@ xrpld.telemetry > xrpl.core
xrpld.telemetry > xrpld.consensus
xrpld.telemetry > xrpld.core
xrpld.telemetry > xrpl.json
xrpld.telemetry > xrpl.ledger
xrpld.telemetry > xrpl.nodestore
xrpld.telemetry > xrpl.protocol
xrpld.telemetry > xrpl.rdb

67
OpenTelemetryPlan/09-data-collection-reference.md
View File

@@ -1401,28 +1401,45 @@ signal as it lands. `Type` is the instrument kind (counter / gauge / histogram /
span / span attr), `Emit site` the owning source file, and `Panel` the dashboard
panel that renders it.
| Signal | Type | Emit site | Panel | Meaning |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------- | --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `dns_resolve_total` (`outcome` = `resolved` \| `empty`) | counter | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Outcome Rate | Peer hostname resolutions. `empty` means a configured bootstrap or `[ips_fixed]` name returned no address, so that peer is never dialled. |
| `dns_resolve_latency_ms` | histogram | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Latency (p95) | Time to resolve a configured peer hostname. Seconds-scale values mean the resolver is timing out ahead of every dial. |
| `overlay_connect_total` (`outcome` = `connected` \| `tcp_fail` \| `tls_fail` \| `upgrade_fail` \| `timeout`) | counter | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Outcome Rate | Outbound peer connection attempts by terminal outcome. The outcome names the stage that broke: TCP, TLS, HTTP upgrade, or no terminal state in time. |
| `overlay_dial_latency_ms` | histogram | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Latency (p95) | Time from starting an outbound dial to its terminal outcome, successes and failures together. A p95 near the dial timeout means peers accept TCP but never finish the handshake. |
| `handshake_negotiation_fail_total` (`reason`, 14 values incl. `wrong_network`, `invalid_network_id`, `clock_skew`, `self_connection`, `session_verify_failed`) | counter | `Handshake.cpp` — `throwNegotiationFailure` (from `verifyHandshake`) | Handshake Negotiation Failures by Reason | Peer handshakes rejected after TLS while checking network id, clock, keys and addresses. `reason` names the failing check. |
| `unl_fetch_total` (`site` = configured UNL URI; `outcome` = the 9 `ListDisposition` strings `accepted` \| `expired` \| `same_sequence` \| `pending` \| `known_sequence` \| `unsupported_version` \| `untrusted` \| `stale` \| `invalid`, plus `fetch_error` \| `bad_status` \| `parse_error`) | counter | `ValidatorSite.cpp` — `ValidatorSite::reportFetchOutcome` | UNL Fetch Rate by Site & Outcome | Validator-list fetches per site. `accepted` is the only success; `same_sequence` and `known_sequence` are normal no-op refreshes; the three literals are transport or content faults. |
| `unl_quorum` (`metric` = `trusted_keys` \| `quorum`) | observable gauge | `MetricsRegistry.cpp` — `registerUnlQuorumGauge` | UNL Trusted Keys vs Quorum; UNL Quorum Headroom | Trusted UNL key count against the validations a ledger needs. `trusted_keys` at or below `quorum` means the node can never declare a ledger validated. |
| `clock_close_offset_seconds` (`metric` = `offset`) | observable gauge | `MetricsRegistry.cpp` — `registerClockSkewGauge` | Clock Close Offset | Network close time offset from the local clock. Negative means the local clock runs ahead. `server_info` only surfaces `close_time_offset` at 60 s or more, so this gauge sees skew far earlier. |
| `state_changes_total` (`from`, `to` = `disconnected` \| `connected` \| `syncing` \| `tracking` \| `full`) | counter | `NetworkOPs.cpp` — `NetworkOPsImp::setMode` | Mode Transitions by Edge | Operating-mode transitions keyed on the (`from`, `to`) edge. The edge is what separates a clean `disconnected`→`connected`→`syncing`→`tracking`→`full` climb from `full`→`connected` flapping; an unlabelled total cannot tell them apart. |
| `sync_state` (`metric` = `initial_full_duration_us`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Time to First FULL | Microseconds from process start to the first `full` transition, sourced from `NetworkOPs::getInitialSyncDurationUs()`. Stays 0 until `full` is reached, so a flat 0 is itself the "never synced" signal; once set it never changes. |
| `sync_state` (`metric` = `network_ledger_gate`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Network Ledger Gate | 1 while the node is still waiting to see a full network ledger (`NetworkOPs::isNeedNetworkLedger()`), else 0. A persistent 1 blocks transaction submission and `full`, whatever the rest of the pipeline shows. |
| `sync_state` (`metric` = `server_stall_seconds`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Server Stall | Current main-loop stall duration from `LoadManager::getCurrentStallSeconds()`, 0 when healthy. Same duration the load monitor logs as "Server stalled for N seconds", which previously existed only in that log line. |
| `sync_state` (`metric` = `ledgers_behind`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Ledgers Behind Network | Peer-reported network tip minus our validated sequence, floored at 0 (`NetworkOPs::getLedgersBehindNetwork()`). Reads each peer's already cached ledger range, so no new network round trip. |
| `server_stall_events_total` | observable counter | `MetricsRegistry.cpp` — `registerStallEventsCounter` | Server Stall Event Rate | Distinct stall episodes since process start, counted once per episode rather than per stalled second. A rising rate is repeated fresh stalls; a flat rate with a large `server_stall_seconds` is one long stall. |
| `sync_acquire` (`metric` = `missing_state_nodes_max` \| `missing_tx_nodes_max`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Missing SHAMap Nodes per Acquire (state/tx) | Largest outstanding SHAMap node count across in-flight acquires, split by tree, from the count `getMissingNodes()` already produces during its sweep (`InboundLedger.cpp` — `InboundLedger::trigger`). **The headline stuck-sync signal:** flat and non-zero across ticks means the acquire will never finish; shrinking means slow but alive. Aggregated as a max rather than labelled per ledger, because a `ledger_seq` label would mint one series per ledger acquired — per-ledger identity stays on the `ledger.acquire` span. |
| `sync_acquire` (`metric` = `received_data_depth`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Peer packets stashed across all in-flight acquires waiting to be applied, summed because it measures one shared processing backlog. A growing depth means arriving node data outpaces processing, so the limit is the job queue or disk rather than peer supply. |
| `sync_acquire` (`metric` = `in_flight`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Number of ledger acquires currently running. Exported so the three values above can be read in context: all zero with `in_flight` zero is an idle node, not a healthy one. |
| `shamap_cache_hit_rate` (`metric` = `treenode`) | observable gauge | `MetricsRegistry.cpp` — `registerCacheHitRateDetailGauge` | SHAMap TreeNode Cache Hit Rate | Share of SHAMap tree-node lookups served from memory, from the previously-uncalled `TaggedCache::getHitRate()`, normalized from 0-100 to 0.0-1.0. Distinct from `nodestore_state`-derived NuDB Cache Hit Ratio on the Ledger Data Sync dashboard: this is the in-memory layer **above** the node store, so a miss here is what causes a read there. The full-below cache is not reported — it is a `KeyCache` whose only lookup path increments `stats_.hits`/`stats_.misses` while `getHitRate()` reads the separate `hits_`/`misses_` members, so its rate is hard-wired to 0 until that accounting is fixed. |
| `sync_acquire_no_progress_total` | counter | `InboundLedger.cpp` — `InboundLedger::onTimer` | Acquire Stall Rate (no progress) | Acquire timeouts where not one new node arrived since the previous timeout, from the `progress_` flag that was previously log-only. Fires on the 3 s acquire timer, never per node. A sustained rate together with a flat missing-node count is the definitive "stuck, not slow" signature. |
| `sync_addnode_total` (`outcome` = `good` \| `duplicate` \| `invalid`) | counter | `InboundLedger.cpp` — `InboundLedger::recordBatchOutcome` | Add-Node Outcomes | SHAMap nodes received during acquire, split by result. Emitted once per received packet from the aggregated batch tally the trace log already printed — never inside the per-node `receiveNode()` loop. Separates real progress (`good`) from wasted bandwidth (`duplicate`) and a misbehaving peer (`invalid`), all three of which look like healthy throughput in traffic metrics. |
| `sync_acquire_source_total` (`source` = `local` \| `network`) | counter | `InboundLedger.cpp` — `InboundLedger::init` | Acquire Source (local vs network) | Whether an acquire was satisfied entirely from the local node store or needed peers, emitted once per new acquire after the first local lookup. Sustained `network` on a node that should already hold the range means sync is disk-bound rather than peer-bound. |
| `jobq_backlog` (`metric` = `waiting` \| `running` \| `deferred`; `job_type` = the `JobTypes::name()` string) | observable gauge | `MetricsRegistry.cpp` — `registerJobQueueBacklogGauge` | Deferred Jobs by Type (starvation); Job Queue Occupancy by Type (waiting/running) | Instantaneous per-job-type queue occupancy, from `JobQueue::getJobTypeCounts()` (one mutex acquire per ~10 s tick). **`deferred` is the signal this adds:** jobs the queue accepted but withheld because the type is already at its concurrency limit, which is counted in neither `waiting` nor `running` and had no exposure anywhere before. The sync-critical types are capped at 3 (`JtLedgerReq`, `JtLedgerData` in `JobTypes.h`), so they starve first. Distinct from the existing `job_queued_total` / `job_started_total` / `job_finished_total` counters and `job_queued_us` / `job_running_us` histograms, which are event-driven from PerfLogImp and describe jobs that already moved, and from the StatsD `jobq_job_count`, which is queue-wide with no per-type split. Cardinality is bounded by the JobType enum (~46 values); every type is observed every tick, so an idle type reports 0 rather than dropping its series. |
| `jobq_saturation` (`metric` = `running_tasks` \| `worker_threads` \| `total_waiting`) | observable gauge | `MetricsRegistry.cpp` — `registerJobQueueSaturationGauge` | Worker Pool Saturation; Worker Pool Capacity & Total Backlog | Global worker-pool saturation from `JobQueue::getWorkerSaturation()`: tasks in flight, threads the pool is configured to run, and jobs queued across all types, all from one reading so the ratio and the backlog describe the same instant. `worker_threads` is exported rather than hardcoded in the dashboard because it is derived at startup from `[workers]`, node size and hardware concurrency. Exists separately from `jobq_backlog` because a pool-wide slowdown otherwise appears as an independent fault in every subsystem queued behind it; a `running_tasks / worker_threads` ratio at 1.0 **with** a non-zero `total_waiting` attributes it to pool exhaustion once. |
<!-- cspell:ignore txset -->
<!-- "txset" is a label value emitted verbatim by serve_refused_total; it is
the code literal, not prose, so it cannot be respelled here. -->
| Signal | Type | Emit site | Panel | Meaning |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `dns_resolve_total` (`outcome` = `resolved` \| `empty`) | counter | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Outcome Rate | Peer hostname resolutions. `empty` means a configured bootstrap or `[ips_fixed]` name returned no address, so that peer is never dialled. |
| `dns_resolve_latency_ms` | histogram | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Latency (p95) | Time to resolve a configured peer hostname. Seconds-scale values mean the resolver is timing out ahead of every dial. |
| `overlay_connect_total` (`outcome` = `connected` \| `tcp_fail` \| `tls_fail` \| `upgrade_fail` \| `timeout`) | counter | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Outcome Rate | Outbound peer connection attempts by terminal outcome. The outcome names the stage that broke: TCP, TLS, HTTP upgrade, or no terminal state in time. |
| `overlay_dial_latency_ms` | histogram | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Latency (p95) | Time from starting an outbound dial to its terminal outcome, successes and failures together. A p95 near the dial timeout means peers accept TCP but never finish the handshake. |
| `handshake_negotiation_fail_total` (`reason`, 14 values incl. `wrong_network`, `invalid_network_id`, `clock_skew`, `self_connection`, `session_verify_failed`) | counter | `Handshake.cpp` — `throwNegotiationFailure` (from `verifyHandshake`) | Handshake Negotiation Failures by Reason | Peer handshakes rejected after TLS while checking network id, clock, keys and addresses. `reason` names the failing check. |
| `unl_fetch_total` (`site` = configured UNL URI; `outcome` = the 9 `ListDisposition` strings `accepted` \| `expired` \| `same_sequence` \| `pending` \| `known_sequence` \| `unsupported_version` \| `untrusted` \| `stale` \| `invalid`, plus `fetch_error` \| `bad_status` \| `parse_error`) | counter | `ValidatorSite.cpp` — `ValidatorSite::reportFetchOutcome` | UNL Fetch Rate by Site & Outcome | Validator-list fetches per site. `accepted` is the only success; `same_sequence` and `known_sequence` are normal no-op refreshes; the three literals are transport or content faults. |
| `unl_quorum` (`metric` = `trusted_keys` \| `quorum`) | observable gauge | `MetricsRegistry.cpp` — `registerUnlQuorumGauge` | UNL Trusted Keys vs Quorum; UNL Quorum Headroom | Trusted UNL key count against the validations a ledger needs. `trusted_keys` at or below `quorum` means the node can never declare a ledger validated. |
| `clock_close_offset_seconds` (`metric` = `offset`) | observable gauge | `MetricsRegistry.cpp` — `registerClockSkewGauge` | Clock Close Offset | Network close time offset from the local clock. Negative means the local clock runs ahead. `server_info` only surfaces `close_time_offset` at 60 s or more, so this gauge sees skew far earlier. |
| `state_changes_total` (`from`, `to` = `disconnected` \| `connected` \| `syncing` \| `tracking` \| `full`) | counter | `NetworkOPs.cpp` — `NetworkOPsImp::setMode` | Mode Transitions by Edge | Operating-mode transitions keyed on the (`from`, `to`) edge. The edge is what separates a clean `disconnected`→`connected`→`syncing`→`tracking`→`full` climb from `full`→`connected` flapping; an unlabelled total cannot tell them apart. |
| `sync_state` (`metric` = `initial_full_duration_us`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Time to First FULL | Microseconds from process start to the first `full` transition, sourced from `NetworkOPs::getInitialSyncDurationUs()`. Stays 0 until `full` is reached, so a flat 0 is itself the "never synced" signal; once set it never changes. |
| `sync_state` (`metric` = `network_ledger_gate`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Network Ledger Gate | 1 while the node is still waiting to see a full network ledger (`NetworkOPs::isNeedNetworkLedger()`), else 0. A persistent 1 blocks transaction submission and `full`, whatever the rest of the pipeline shows. |
| `sync_state` (`metric` = `server_stall_seconds`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Server Stall | Current main-loop stall duration from `LoadManager::getCurrentStallSeconds()`, 0 when healthy. Same duration the load monitor logs as "Server stalled for N seconds", which previously existed only in that log line. |
| `sync_state` (`metric` = `ledgers_behind`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Ledgers Behind Network | Peer-reported network tip minus our validated sequence, floored at 0 (`NetworkOPs::getLedgersBehindNetwork()`). Reads each peer's already cached ledger range, so no new network round trip. |
| `server_stall_events_total` | observable counter | `MetricsRegistry.cpp` — `registerStallEventsCounter` | Server Stall Event Rate | Distinct stall episodes since process start, counted once per episode rather than per stalled second. A rising rate is repeated fresh stalls; a flat rate with a large `server_stall_seconds` is one long stall. |
| `sync_acquire` (`metric` = `missing_state_nodes_max` \| `missing_tx_nodes_max`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Missing SHAMap Nodes per Acquire (state/tx) | Largest outstanding SHAMap node count across in-flight acquires, split by tree, from the count `getMissingNodes()` already produces during its sweep (`InboundLedger.cpp` — `InboundLedger::trigger`). **The headline stuck-sync signal:** flat and non-zero across ticks means the acquire will never finish; shrinking means slow but alive. Aggregated as a max rather than labelled per ledger, because a `ledger_seq` label would mint one series per ledger acquired — per-ledger identity stays on the `ledger.acquire` span. |
| `sync_acquire` (`metric` = `received_data_depth`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Peer packets stashed across all in-flight acquires waiting to be applied, summed because it measures one shared processing backlog. A growing depth means arriving node data outpaces processing, so the limit is the job queue or disk rather than peer supply. |
| `sync_acquire` (`metric` = `in_flight`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Number of ledger acquires currently running. Exported so the three values above can be read in context: all zero with `in_flight` zero is an idle node, not a healthy one. |
| `shamap_cache_hit_rate` (`metric` = `treenode`) | observable gauge | `MetricsRegistry.cpp` — `registerCacheHitRateDetailGauge` | SHAMap TreeNode Cache Hit Rate | Share of SHAMap tree-node lookups served from memory, from the previously-uncalled `TaggedCache::getHitRate()`, normalized from 0-100 to 0.0-1.0. Distinct from `nodestore_state`-derived NuDB Cache Hit Ratio on the Ledger Data Sync dashboard: this is the in-memory layer **above** the node store, so a miss here is what causes a read there. The full-below cache is not reported — it is a `KeyCache` whose only lookup path increments `stats_.hits`/`stats_.misses` while `getHitRate()` reads the separate `hits_`/`misses_` members, so its rate is hard-wired to 0 until that accounting is fixed. |
| `sync_acquire_no_progress_total` | counter | `InboundLedger.cpp` — `InboundLedger::onTimer` | Acquire Stall Rate (no progress) | Acquire timeouts where not one new node arrived since the previous timeout, from the `progress_` flag that was previously log-only. Fires on the 3 s acquire timer, never per node. A sustained rate together with a flat missing-node count is the definitive "stuck, not slow" signature. |
| `sync_addnode_total` (`outcome` = `good` \| `duplicate` \| `invalid`) | counter | `InboundLedger.cpp` — `InboundLedger::recordBatchOutcome` | Add-Node Outcomes | SHAMap nodes received during acquire, split by result. Emitted once per received packet from the aggregated batch tally the trace log already printed — never inside the per-node `receiveNode()` loop. Separates real progress (`good`) from wasted bandwidth (`duplicate`) and a misbehaving peer (`invalid`), all three of which look like healthy throughput in traffic metrics. |
| `sync_acquire_source_total` (`source` = `local` \| `network`) | counter | `InboundLedger.cpp` — `InboundLedger::init` | Acquire Source (local vs network) | Whether an acquire was satisfied entirely from the local node store or needed peers, emitted once per new acquire after the first local lookup. Sustained `network` on a node that should already hold the range means sync is disk-bound rather than peer-bound. |
| `jobq_backlog` (`metric` = `waiting` \| `running` \| `deferred`; `job_type` = the `JobTypes::name()` string) | observable gauge | `MetricsRegistry.cpp` — `registerJobQueueBacklogGauge` | Deferred Jobs by Type (starvation); Job Queue Occupancy by Type (waiting/running) | Instantaneous per-job-type queue occupancy, from `JobQueue::getJobTypeCounts()` (one mutex acquire per ~10 s tick). **`deferred` is the signal this adds:** jobs the queue accepted but withheld because the type is already at its concurrency limit, which is counted in neither `waiting` nor `running` and had no exposure anywhere before. The sync-critical types are capped at 3 (`JtLedgerReq`, `JtLedgerData` in `JobTypes.h`), so they starve first. Distinct from the existing `job_queued_total` / `job_started_total` / `job_finished_total` counters and `job_queued_us` / `job_running_us` histograms, which are event-driven from PerfLogImp and describe jobs that already moved, and from the StatsD `jobq_job_count`, which is queue-wide with no per-type split. Cardinality is bounded by the JobType enum (~46 values); every type is observed every tick, so an idle type reports 0 rather than dropping its series. |
| `jobq_saturation` (`metric` = `running_tasks` \| `worker_threads` \| `total_waiting`) | observable gauge | `MetricsRegistry.cpp` — `registerJobQueueSaturationGauge` | Worker Pool Saturation; Worker Pool Capacity & Total Backlog | Global worker-pool saturation from `JobQueue::getWorkerSaturation()`: tasks in flight, threads the pool is configured to run, and jobs queued across all types, all from one reading so the ratio and the backlog describe the same instant. `worker_threads` is exported rather than hardcoded in the dashboard because it is derived at startup from `[workers]`, node size and hardware concurrency. Exists separately from `jobq_backlog` because a pool-wide slowdown otherwise appears as an independent fault in every subsystem queued behind it; a `running_tasks / worker_threads` ratio at 1.0 **with** a non-zero `total_waiting` attributes it to pool exhaustion once. |
| `peer_ledger_supply` (`metric` = `peers_reporting` \| `peers_serving_validated` \| `peers_serving_next` \| `supply_min_seq` \| `supply_max_seq`) | observable gauge | `MetricsRegistry.cpp` — `registerPeerLedgerSupplyGauge` (aggregating `OverlayImpl::getPeerLedgerSupply`) | Peers Able to Serve Needed Sequence; Peer Ledger Supply Window | How much of the sequence range this node needs its connected peer set can actually serve, from one pass over the active peers reading the range each already advertised in `mtSTATUS_CHANGE`. **`peers_serving_next` is the signal this exists for:** zero there with a non-zero `peers_reporting` means no connected peer holds validated + 1, so the peer set must change and waiting cannot finish the sync. `peers_reporting` is the denominator that makes the rest readable — peers advertising `[0, 0]` have not reported yet and are excluded from every field, so they cannot make a healthy peer set appear to serve from genesis; when nothing has reported, both window fields read 0 meaning **unknown**, not genesis. `supply_min_seq` / `supply_max_seq` separate "asking for history nobody kept" from "asking for a tip nobody reached". Distinct from `server_info{metric="peers"}`, a bare connection count with no notion of what those peers hold; from `sync_state{metric="ledgers_behind"}`, which uses the same per-peer maxima but collapses them to a single distance-to-tip number that cannot say how many peers can serve that distance or whether the range has a hole; and from `peer_quality{metric="peers_insane_count"}`, which counts peers on a different chain and is therefore a correctness signal, not an availability one. |
| `peer_disconnect_total` (`reason` = `graceful` \| `shutdown` \| `stopping` \| `read_error` \| `write_error` \| `timer_error` \| `ping_timeout` \| `not_useful` \| `large_sendq` \| `charge_resources` \| `malformed_handshake` \| `shared_value` \| `unknown`; `direction` = `inbound` \| `outbound`) | counter | `PeerImp.cpp` — `PeerImp::close` | Peer Disconnects by Reason | Peer teardowns split by cause and by which side opened the connection. Emitted once per teardown at `close()`, the single funnel every disconnect path passes through, and `close()` already self-guards on the socket being open, so a repeated close cannot double-count and the total matches the existing unlabelled tally. `reason` is set by whichever site decided to disconnect, first writer wins, so a later generic reason never masks the real one; the value is always one of a fixed set of literals in `PeerImp.cpp`, never peer-supplied data, so cardinality is bounded by the code. The split is the whole point: it separates our-fault backpressure (`large_sendq`, `charge_resources`) from topology and network faults (`not_useful`, `ping_timeout`, `read_error`), and normal churn (`graceful`) from either. Distinct from the existing `server_info{metric="peer_disconnects_resources"}`, which counts only the resource-charge subset and carries no labels, and from the StatsD `overlay_peer_disconnects`, which is the unlabelled grand total in which every reason above collapses into one number. |
| `peer_accept_total` (`outcome` = `accepted` \| `local_endpoint_fail` \| `resource_limit` \| `no_slot` \| `not_peer_request` \| `protocol_mismatch` \| `bad_cookie` \| `slot_refused` \| `handshake_error`) | counter | `OverlayImpl.cpp` — `OverlayImpl::onHandoff` via `reportAcceptOutcome` | Inbound Peer Accept Outcomes | Terminal outcome of every inbound connection this node is offered, one emit per handoff. `accepted` is reported only after `run()`, so anything that threw on the way lands on `handshake_error` instead; the two early returns that are not peer attempts at all (a handled HTTP request, and a request that never asked to upgrade) are deliberately not counted. The `outcome` names the stage that refused: no local endpoint, the resource manager, PeerFinder having no slot or seeing a duplicate, a non-peer upgrade request, protocol version disagreement, a bad security cookie, or activation being refused. This is the **inbound twin** of the existing `overlay_connect_total{outcome}`, which covers outbound dials only; without it a node refusing every inbound connection is indistinguishable from one nobody dials, and reading the two together gives the full in/out split. |
| `peerfinder_slot_census` (`metric` = `out_active` \| `out_max` \| `in_active` \| `in_max` \| `connecting` \| `fixed_configured` \| `fixed_active` \| `bootcache` \| `livecache`) | observable gauge | `MetricsRegistry.cpp` — `registerSlotCensusGauge` (from `Logic::getSlotCensus`) | PeerFinder Slot Census; PeerFinder Address Caches & Fixed Peers | Slot occupancy against capacity, outbound dials in flight, configured-versus-connected fixed peers, and the depth of both address caches. All nine come from a single acquire of the PeerFinder lock, so they are mutually consistent, share one label set and can be compared against each other. That is what makes the three most common bootstrap failures visible: `connecting` non-zero while `out_active` stays below `out_max` (dials starting and never completing), `bootcache` and `livecache` both at 0 (nothing to dial at all), and `fixed_active` below `fixed_configured` (a peer named in the configuration is unreachable). `fixed_configured` is the count of peers named in the config, so the pair an operator reads is "how many did I ask for" against "how many do I have" — the same comparison `autoconnect()` makes. All nine values already existed inside PeerFinder; only two of them were exported, as the legacy beast::insight gauges `peer_finder_active_inbound_peers` and `peer_finder_active_outbound_peers`. Those two carry no capacity, attempt or cache term, are read at unrelated instants, and so cannot be joined with each other let alone with a capacity term — leaving all three failures above indistinguishable from a node that is simply not dialling. |
| `serve_refused_total` (`request` = `ledger` \| `txset` \| `object` \| `fetchpack`; `reason` = `sendq_full` \| `load_shed` \| `not_found` \| `no_map` \| `bad_type` \| `empty_reply`) | counter | `PeerImp.cpp` — `processLedgerRequest`, `onMessage(TMGetObjectByHash)`, `doFetchPack` | Ledger/Object Serve Refusals | Peer data requests this node declined to answer, split by what was asked for and why. This is the **supply side** of the sync exchange — what this node refuses to serve OTHERS — and nothing equivalent existed before, so a node shedding every ledger request looked identical to one being asked for nothing. `sendq_full` and `load_shed` are self-inflicted backpressure (the send queue at `Tuning::kDropSendQueue`, or the local fee track loaded, or too many pack jobs queued), while `not_found` is a genuine history gap and `no_map` / `bad_type` / `empty_reply` mean the request was answerable in principle but produced nothing to send. `fetchpack` is counted apart from `ledger` because a fetch pack is how a syncing peer catches up in bulk and its shed threshold is a different one. Emitted at most once per request — `empty_reply` is reported after the node loop, never inside it — and both labels are code literals, so cardinality is bounded at compile time. |
| `amendment_block` (`metric` = `warned` \| `seconds_to_block`) | observable gauge | `MetricsRegistry.cpp` — `registerAmendmentBlockGauge` | Amendment Block Countdown; Amendment Warned | `warned` is 1 once an unsupported amendment has reached majority (`NetworkOPs::isAmendmentWarned()`, previously only an admin-only `server_info` warning). **`seconds_to_block` is the leading indicator:** seconds until that amendment activates, from `AmendmentTable::firstUnsupportedExpected()` against the network close time. It reads `-1` when nothing is pending — a distinct healthy value rather than a missing series, matching the sentinel `validator_health{metric="unl_expiry_days"}` already uses — and is clamped at 0 rather than going negative, because past-due means the block is imminent, not overdue by some amount worth charting; the subtraction is done in `std::int64_t` so a past-due activation cannot wrap. Amendment-blocked is a terminal sync blocker: the node stops validating and never resumes without a software upgrade. The existing `validator_health{metric="amendment_blocked"}` reports that state after the fact, when nothing can be done about it; this gauge is the window before it, which is the only actionable part. The blocking amendment's identity is deliberately **not** a label — the network can vote on an arbitrary 256-bit amendment id, not drawn from this build's known features, so an id label would be unbounded cardinality and would mint a permanent new series per amendment. The id is available in logs from `AmendmentTableImpl::doValidatedLedger` ("Unsupported amendment \<hash\> reached majority at ..."), correlated to this series by node and time. |
| `ledger_jump_total` | counter | `NetworkOPs.cpp` — `NetworkOPsImp::switchLastClosedLedger` | Byzantine Ledger Jumps | Forced jumps of the last closed ledger onto a divergent chain: the node was told the network's LCL is not the one it built on and discarded its own chain tip to follow. Nothing equivalent existed — this was log-only ("JUMP last closed ledger to ..."), so a node repeatedly thrashing between chains left no time series to correlate against the rest of the sync pipeline. Any non-zero rate is abnormal by construction; repeated jumps are wrong-chain thrash, which points at the peer set and the configured network id rather than anywhere in the acquire pipeline. Deliberately unlabelled: the ledger hash and sequence would both be unbounded as label values, and the log line beside the emit already carries them. |
| `nodestore_latency` (`metric` = `write_mean_us` \| `read_mean_us` \| `write_count` \| `read_count`) | observable gauge | `MetricsRegistry.cpp` — `registerNodeStoreLatencyGauge` | NodeStore Write vs Read Latency (us/op); NodeStore Operation Rate | Mean microseconds per node-store store and per fetch, with both operation counts so a panel can divide the two rates and read _interval_ latency instead of the since-boot average. **The write side is the new signal.** `storeDurationUs_` was declared in `Database.h` and never written, and no accessor existed, so no write-path latency was observable anywhere; the read total was already exposed as `nodestore_state{metric="node_reads_duration_us"}`. This is the fingerprint of the "a node with a large existing DB syncs slower than a fresh one" symptom, which is write-bound and therefore invisible in every read-side metric. Chosen as a gauge over a histogram deliberately: a histogram gives true percentiles but costs one `Record()` per node object on the store/fetch path, and a single ledger write walks thousands of SHAMap nodes — this gauge instead reads four existing atomics once per ~10 s tick and adds nothing to the hot path. Consequence: **p99 is not obtainable from this signal**, and a histogram added later would also need an explicit-bucket View (`addMicrosecondHistogramView`) because the SDK default buckets top out at 10,000. Distinct from the Ledger Data Sync dashboard's NuDB Read Latency panel, which divides two `nodestore_state` fields in PromQL: that panel has no write-duration input to divide, because the quantity did not exist. **Known gap:** `write_mean_us` is emitted only when the store-duration total is non-zero, and that total is fed by `Database::recordStoreDuration`, today called only from `Database::importInternal` (the `[import_db]` admin path). `Database::store()` is pure virtual and neither `DatabaseNodeImp::store` nor `DatabaseRotatingImp::store` times itself yet, so an ordinary node reports `write_count` with no `write_mean_us`. The mean is omitted rather than reported as 0 so the gap stays visible instead of reading as "writes are instantaneous". |
| `ledger_replay_fallback_total` (`stage` = `skiplist` \| `delta`) | counter | `SkipListAcquire.cpp` / `LedgerDeltaAcquire.cpp` — `trigger` | Replay Fallback to Full Acquire (by stage) | A ledger-replay sub-task abandoning its shortcut and acquiring the whole ledger through `InboundLedger` instead, because too few connected peers support the `LedgerReplay` protocol feature. Both branches were debug-log-only, so a silently defeated replay optimisation left no metric at all — back-fill simply ran on the slower path with nothing to show why. Emitted once, on the transition into fallback, not at the acquire call, which re-runs on every later trigger. The `stage` label separates the skip-list acquire (which fetches the list of historical ledger hashes) from the per-ledger delta acquire, because they fail independently. |
| `ledger_replay_outcome_total` (`outcome` = `success` \| `timeout` \| `build_failed` \| `parameter_failed`) | counter | `LedgerReplayTask.cpp` — `LedgerReplayTask::recordOutcome` | Replay Outcomes (by terminal state) | Terminal state of every ledger-replay task, one emit per task. Every terminal path previously only set an internal `complete_`/`failed_` flag and wrote a log line, so a replay that never succeeded was indistinguishable from one that was never attempted. The outcome names the layer at fault: `timeout` means the deltas never arrived (a peer-supply problem), `build_failed` means a delta would not apply to its parent, and `parameter_failed` means a peer served a skip list inconsistent with what the task asked for — the latter two are data faults, not slowness. Read with `ledger_replay_fallback_total`: fallbacks rising while successes stay flat is replay-based catch-up degrading to full-ledger acquisition. |
| `nodestore_latency` (`metric` = `write_mean_us` \| `read_mean_us` \| `write_count` \| `read_count`) | observable gauge | `MetricsRegistry.cpp` — `registerNodeStoreLatencyGauge` | NodeStore Write vs Read Latency (us/op); NodeStore Operation Rate | Mean microseconds per node-store store and per fetch, with both operation counts so a panel can divide the two rates and read _interval_ latency instead of the since-boot average. **The write side is the new signal.** `storeDurationUs_` was declared in `Database.h` and never written, and no accessor existed, so no write-path latency was observable anywhere; the read total was already exposed as `nodestore_state{metric="node_reads_duration_us"}`. This is the fingerprint of the "a node with a large existing DB syncs slower than a fresh one" symptom, which is write-bound and therefore invisible in every read-side metric. Chosen as a gauge over a histogram deliberately: a histogram gives true percentiles but costs one `Record()` per node object on the store/fetch path, and a single ledger write walks thousands of SHAMap nodes — this gauge instead reads four existing atomics once per ~10 s tick and adds nothing to the hot path. Consequence: **p99 is not obtainable from this signal**, and a histogram added later would also need an explicit-bucket View (`addMicrosecondHistogramView`) because the SDK default buckets top out at 10,000. Distinct from the Ledger Data Sync dashboard's NuDB Read Latency panel, which divides two `nodestore_state` fields in PromQL: that panel has no write-duration input to divide, because the quantity did not exist. **Known gap:** `write_mean_us` is emitted only when the store-duration total is non-zero, and that total is fed by `Database::recordStoreDuration`, today called only from `Database::importInternal` (the `[import_db]` admin path). `Database::store()` is pure virtual and neither `DatabaseNodeImp::store` nor `DatabaseRotatingImp::store` times itself yet, so an ordinary node reports `write_count` with no `write_mean_us`. The mean is omitted rather than reported as 0 so the gap stays visible instead of reading as "writes are instantaneous". |
| `ledger_replay_fallback_total` (`stage` = `skiplist` \| `delta`) | counter | `SkipListAcquire.cpp` / `LedgerDeltaAcquire.cpp` — `trigger` | Replay Fallback to Full Acquire (by stage) | A ledger-replay sub-task abandoning its shortcut and acquiring the whole ledger through `InboundLedger` instead, because too few connected peers support the `LedgerReplay` protocol feature. Both branches were debug-log-only, so a silently defeated replay optimisation left no metric at all — back-fill simply ran on the slower path with nothing to show why. Emitted once, on the transition into fallback, not at the acquire call, which re-runs on every later trigger. The `stage` label separates the skip-list acquire (which fetches the list of historical ledger hashes) from the per-ledger delta acquire, because they fail independently. |
| `ledger_replay_outcome_total` (`outcome` = `success` \| `timeout` \| `build_failed` \| `parameter_failed`) | counter | `LedgerReplayTask.cpp` — `LedgerReplayTask::recordOutcome` | Replay Outcomes (by terminal state) | Terminal state of every ledger-replay task, one emit per task. Every terminal path previously only set an internal `complete_`/`failed_` flag and wrote a log line, so a replay that never succeeded was indistinguishable from one that was never attempted. The outcome names the layer at fault: `timeout` means the deltas never arrived (a peer-supply problem), `build_failed` means a delta would not apply to its parent, and `parameter_failed` means a peer served a skip list inconsistent with what the task asked for — the latter two are data faults, not slowness. Read with `ledger_replay_fallback_total`: fallbacks rising while successes stay flat is replay-based catch-up degrading to full-ledger acquisition. |

1522
docker/telemetry/grafana/dashboards/ledger-sync-health.json
View File

File diff suppressed because it is too large Load Diff

26
docker/telemetry/workload/expected_metrics.json
View File

@@ -156,12 +156,34 @@
"jobq_backlog{metric=\"deferred\",job_type=\"ledgerRequest\"}",
"jobq_saturation{metric=\"running_tasks\"}",
"jobq_saturation{metric=\"worker_threads\"}",
"jobq_saturation{metric=\"total_waiting\"}"
"jobq_saturation{metric=\"total_waiting\"}",
"peer_ledger_supply{metric=\"peers_reporting\"}",
"peer_ledger_supply{metric=\"peers_serving_validated\"}",
"peer_ledger_supply{metric=\"peers_serving_next\"}",
"peer_ledger_supply{metric=\"supply_min_seq\"}",
"peer_ledger_supply{metric=\"supply_max_seq\"}",
"peerfinder_slot_census{metric=\"out_active\"}",
"peerfinder_slot_census{metric=\"out_max\"}",
"peerfinder_slot_census{metric=\"in_active\"}",
"peerfinder_slot_census{metric=\"in_max\"}",
"peerfinder_slot_census{metric=\"connecting\"}",
"peerfinder_slot_census{metric=\"fixed_configured\"}",
"peerfinder_slot_census{metric=\"fixed_active\"}",
"peerfinder_slot_census{metric=\"bootcache\"}",
"peerfinder_slot_census{metric=\"livecache\"}",
"amendment_block{metric=\"warned\"}",
"amendment_block{metric=\"seconds_to_block\"}",
"peer_accept_total",
"nodestore_latency{metric=\"write_count\"}",
"nodestore_latency{metric=\"read_count\"}",
"nodestore_latency{metric=\"read_mean_us\"}"
],
"_acquire_note": "The four sync_acquire sub-series and shamap_cache_hit_rate are unconditional: both are observable gauges whose callbacks observe every series on each collection tick, so each is present even when the value is 0 (an idle node reports in_flight=0 and missing_state_nodes_max=0, and a cold cache reports a 0.0 hit rate). Absence, not a zero, is the regression. The three WP-A3 counters (sync_acquire_source_total, sync_addnode_total, sync_acquire_no_progress_total) are deliberately NOT asserted here: all three are emitted only from InboundLedger, which runs only when a node must fetch a ledger it lacks. expected_spans.json already marks the ledger.acquire span optional for exactly this reason (\"A healthy local cluster rarely back-fills history\"), and the metric validator has no per-metric optional flag, so listing them would fail the harness red on a healthy run. They are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp and by the ledger-sync-health panels; add them here only alongside a harness step that forces a real acquire (e.g. starting a node against an existing ledger history).",
"_jobq_note": "The jobq_backlog and jobq_saturation series are unconditional: both are observable gauges whose callbacks iterate EVERY registered JobType (jobData_ is populated from JobTypes at JobQueue construction) and observe all three fields on each collection tick, so a series exists even when the value is 0. That is why an idle-but-registered type like ledgerData is safe to assert by name here — a fresh harness node that never defers a single job still reports jobq_backlog{metric=\"deferred\",job_type=\"ledgerData\"} = 0, and absence, not the zero, is the regression. Two job_type values are asserted (ledgerData and ledgerRequest) because they are the sync-critical types capped at concurrency 3 in JobTypes.h, so they are the ones whose deferred series must never silently vanish. Only deferred is asserted for ledgerRequest to keep the list short: the three-field fan-out is already proven by ledgerData. worker_threads is asserted because it is the denominator of the dashboard saturation ratio, and it is always at least 1 (the JobQueue ctor gives standalone mode exactly one worker), so a zero or missing reading there means the accessor regressed rather than the node being idle.",
"_conditional_note": "handshake_negotiation_fail_total and unl_fetch_total are conditional under the local harness: the first only exists once a handshake is rejected, and the second needs a [validator_list_sites] entry (run-full-validation.sh generates a static [validators] file instead). The validator has no per-metric optional flag, so if either reports 0 series in a harness run, move it out of this group rather than weakening the check.",
"_sync_state_note": "The four sync_state sub-series are unconditional: the gauge observes all four on every collection tick, so each is present as a series even when its value is 0 (a node that never reached FULL reports initial_full_duration_us=0, and a healthy node reports server_stall_seconds=0). The check asserts series presence, not a non-zero value, which is exactly right here — a zero is a meaningful reading for these signals, and absence is the regression. server_stall_events_total is likewise always present because the observable counter reports the tally (0 or more) every tick. state_changes_total is asserted here with a from!=\"\",to!=\"\" selector rather than bare (parity_counters already asserts the bare name): the selector is what proves the WP-A2 {from,to} label dimension actually reached Prometheus, so a regression to the old unlabelled counter fails this check instead of silently passing on the bare name. It needs at least one real mode transition, which any node reaching connected/syncing produces during startup."
"_sync_state_note": "The four sync_state sub-series are unconditional: the gauge observes all four on every collection tick, so each is present as a series even when its value is 0 (a node that never reached FULL reports initial_full_duration_us=0, and a healthy node reports server_stall_seconds=0). The check asserts series presence, not a non-zero value, which is exactly right here — a zero is a meaningful reading for these signals, and absence is the regression. server_stall_events_total is likewise always present because the observable counter reports the tally (0 or more) every tick. state_changes_total is asserted here with a from!=\"\",to!=\"\" selector rather than bare (parity_counters already asserts the bare name): the selector is what proves the WP-A2 {from,to} label dimension actually reached Prometheus, so a regression to the old unlabelled counter fails this check instead of silently passing on the bare name. It needs at least one real mode transition, which any node reaching connected/syncing produces during startup.",
"_a7_note": "WP-A7 adds three observable gauges and four counters. The 16 gauge sub-series (peer_ledger_supply, peerfinder_slot_census, amendment_block) are unconditional and asserted individually: each callback in MetricsRegistry.cpp calls observe() for every field on every collection tick with no early return between them, so the series exists whatever the value. That includes the two sentinel readings — a node whose peers have advertised nothing reports peer_ledger_supply{metric=\"supply_min_seq\"} = 0 meaning unknown, and a node with no pending amendment reports amendment_block{metric=\"seconds_to_block\"} = -1 meaning healthy. Absence, not the sentinel, is the regression. Of the four counters only peer_accept_total is asserted: run-full-validation.sh gives every node a [port_peer] on 0.0.0.0 and lists the other four nodes in [ips], so all 5 nodes dial each other and each one is also dialled, which means OverlayImpl::onHandoff runs and reports outcome=accepted (or slot_refused/no_slot on the duplicate half of each mutual dial) on every node. It is asserted bare rather than with an outcome= selector because which outcome a given node records depends on dial ordering, which the harness does not control. The other three counters are deliberately NOT asserted. peer_disconnect_total is emitted only from PeerImp::close, and a healthy 5-node localhost cluster holds its 4 fixed peers for the whole run: the timer-driven reasons need maxUnknownTime (600 s) or maxDivergedTime (300 s) to elapse (Config.h) while the full-validation profile totals well under that, and the shutdown reasons only fire during teardown, which happens in run-full-validation.sh after Step 5 has already scraped. serve_refused_total needs a peer to ask this node for a ledger, tx set or object it cannot serve — on a cluster where every node has the same complete history from genesis, getLedger()/getTxSet() succeed and the send queues never approach Tuning::kDropSendQueue. ledger_jump_total needs NetworkOPsImp::switchLastClosedLedger, reached only when consensus reports an LCL this node did not build on; a healthy 5-node cluster agrees every round, so it never jumps. The metric validator has no per-metric optional flag, so listing any of the three would fail the harness red on a healthy run — the same reasoning _acquire_note applies to the WP-A3 InboundLedger counters. All four counters are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp and rendered by the ledger-sync-health panels Peer Disconnects by Reason, Ledger/Object Serve Refusals and Byzantine Ledger Jumps. To make them assertable the harness would need a fault-injection step: kill one node mid-run and re-scrape before teardown (peer_disconnect_total, reason=read_error/graceful), request a ledger sequence outside the cluster's history or drive a node past its send-queue limit (serve_refused_total), and start a node on a divergent chain tip or partition the cluster and heal it (ledger_jump_total).",
"_a6_note": "WP-A6 adds one observable gauge (nodestore_latency) and two counters (ledger_replay_fallback_total, ledger_replay_outcome_total). Only three of the four gauge sub-series are asserted. write_count and read_count are unconditional: the callback observes both on every collection tick with no early return before them, so a series exists whatever the value, and a node that has written nothing reports write_count=0 rather than dropping the series. read_mean_us is safe because any node that has opened a ledger has already fetched objects, so the fetch duration total is non-zero. write_mean_us is deliberately NOT asserted: the mean is emitted only when the store-duration total is non-zero, and that total is fed by Database::recordStoreDuration(), which today is called only from Database::importInternal -- the [import_db] admin path. Database::store() is pure virtual and the two concrete runtime overrides (DatabaseNodeImp::store, DatabaseRotatingImp::store) do not time themselves yet, so an ordinary harness node produces write_count but no write_mean_us. Asserting it would ship a permanently red CI check for a known, documented gap; the omission is the honest encoding of that gap. The two replay counters are likewise NOT asserted, for the same reason _acquire_note gives for the WP-A3 InboundLedger counters: both are emitted only from the ledger-replay path, which requires the [ledger_replay] config stanza AND a real historical back-fill against peers that support the LedgerReplay protocol feature. run-full-validation.sh starts a fresh local cluster with no history to back-fill, so no replay task is ever created and neither counter can produce a series. All three unasserted signals are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp and rendered by the ledger-sync-health panels NodeStore Write vs Read Latency, Replay Fallback to Full Acquire and Replay Outcomes. To make them assertable the harness would need to enable [ledger_replay] and start a node against an existing ledger history so it back-fills through the replay path, and to time the two concrete store overrides."
},
"grafana_dashboards": {
"description": "All Grafana dashboards that must render data (UIDs as provisioned on disk under docker/telemetry/grafana/dashboards/).",

110
docs/telemetry-glossary.md
View File

@@ -491,6 +491,16 @@ When a node is missing ledgers (at startup, after an outage, or to extend histor
**Scope:** per node — measured on and specific to this individual server.
<a id="byzantine-ledger-jump"></a>
### Byzantine ledger jump
Being told that the network's last closed ledger is not the one this node built on, and discarding its own chain tip to follow the network instead. It is an abnormal event by construction: the node had already closed a ledger, and it is now throwing that work away because the peers it listens to agree on a different one. A single jump while a fresh node is still settling onto the network's chain can be benign. Repeated jumps are wrong-chain thrash — the node keeps switching between chains and never settles — and the cause is upstream of the sync pipeline, in which peers it is listening to or which network it thinks it is on, so nothing in ledger acquisition can fix it.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Fork](#fork) · [Ledger history mismatch](#ledger-history-mismatch) · [Insane / diverged peers](#insane-diverged-peers)
<a id="clock-close-offset"></a>
### Clock close offset
@@ -605,6 +615,16 @@ Acquiring a ledger means requesting it and its contents from peers when the node
**Scope:** per node — measured on and specific to this individual server.
<a id="ledger-replay"></a>
### Ledger replay
An optional faster way to rebuild a run of historical ledgers: instead of downloading each ledger whole, the node fetches one starting ledger plus the list of ledger hashes that links the range, then fetches only what changed in each subsequent ledger and applies those changes on top of its predecessor. It is only available when enough connected peers support the protocol feature that serves those pieces, so whether it is used at all depends on the peer set rather than on local configuration alone.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Replay fallback](#replay-fallback) · [Ledger acquire (inbound fetch)](#ledger-acquire-inbound-fetch)
<a id="ledgers-behind-network"></a>
### Ledgers behind network
@@ -635,6 +655,36 @@ The startup guard that holds a node back until it has seen a complete ledger fro
**See also:** [Operating mode / server state](#operating-mode-server-state) · [UNL quorum headroom](#unl-quorum-headroom) · [Operating mode / server state on xrpl.org](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states)
<a id="node-store-read-latency"></a>
### Node-store read latency
How long the node store takes to return one stored object. Every ledger traversal that is not already answered from an in-memory cache pays this cost, so it is the floor under ledger acquisition and under most queries. It is reported as an average over an interval rather than as a distribution, which means a slow minority of reads shows up as a raised average rather than as a separate tail figure.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Node-store write latency](#node-store-write-latency) · [SHAMap cache hit rate](#shamap-cache-hit-rate)
<a id="node-store-operation-rate"></a>
### Node-store operation rate
How many objects per second the node store is storing and retrieving. It is the companion an average latency needs in order to be read correctly: latency measured over an interval with almost no operations in it is a stale number rather than a good one, and a node writing nothing at all while still behind the network is stalled somewhere upstream of storage rather than slowed by it.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Node-store write latency](#node-store-write-latency)
<a id="node-store-write-latency"></a>
### Node-store write latency
How long the node store takes to persist one object. This is the cost that governs how fast a node can absorb ledger history, because filling in history is dominated by writing rather than by reading. It is the measurement that distinguishes the two ways a sync can be slow: starved of data from peers, or unable to write down the data it already has. A node with a large existing database can be slower to start and catch up than an empty one for exactly this reason, and no read-side measurement reveals it. Like the read figure it is an interval average, not a distribution.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Node-store read latency](#node-store-read-latency) · [Node-store operation rate](#node-store-operation-rate)
<a id="operating-mode-server-state"></a>
### Operating mode / server state
@@ -655,6 +705,16 @@ The elapsed time of one outbound peer connection attempt, from starting the TCP
**See also:** [DNS resolve](#dns-resolve) · [Handshake negotiation failure](#handshake-negotiation-failure) · [Peer protocol on xrpl.org](https://xrpl.org/docs/concepts/networks-and-servers/peer-protocol)
<a id="peer-ledger-supply"></a>
### Peer ledger supply
The idea that a connected peer set collectively offers a window of ledger sequences, rather than being simply present or absent. Each peer advertises the oldest and newest ledger it holds, so the set as a whole can serve some range and nothing outside it. This turns "how many peers do I have" into the question that actually matters during a sync: does any connected peer hold the next ledger this node needs. Being unable to advance because nobody holds that sequence is a fundamentally different fault from being slow — it is a supply gap fixed only by changing the peer set, whereas slowness with the data available is a throughput problem fixed locally, and the two are indistinguishable from inside the acquire itself. The shape of a gap matters too: needing a sequence below the window means asking for history nobody kept, while needing one above it means asking for a tip nobody has reached. Peers that have not advertised a range yet are excluded from the counts entirely, so a zero window means unknown rather than empty, and the count of peers that have reported anything is what makes the rest readable.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Ledgers behind network](#ledgers-behind-network) · [Slot census](#slot-census) · [Acquire stall](#acquire-stall) · [Complete ledger ranges](#complete-ledger-ranges)
<a id="received-data-stash"></a>
### Received-data stash
@@ -665,6 +725,16 @@ Peer packets held for later processing because a ledger acquire cannot apply the
**See also:** [Add-node outcome](#add-node-outcome) · [Acquire stall](#acquire-stall)
<a id="replay-fallback"></a>
### Replay fallback
A replay sub-task giving up on the delta shortcut and acquiring the entire ledger instead, which happens when too few connected peers support the feature that serves the pieces replay needs. Nothing fails when this occurs and no error is raised — the node still completes its back-fill, just on the slower path — which is why it is easy to miss: the optimisation is simply absent. It is counted separately for each of the two sub-tasks, because the one that fetches the list of historical ledger hashes and the one that fetches a single ledger's changes can fail independently of each other.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Ledger replay](#ledger-replay)
<a id="shamap-cache-hit-rate"></a>
### SHAMap cache hit rate
@@ -739,6 +809,16 @@ A cluster is a set of servers run by the same operator that trust each other, ex
**See also:** [Cluster on xrpl.org](https://xrpl.org/docs/concepts/networks-and-servers/clustering)
<a id="disconnect-reason"></a>
### Disconnect reason
The cause recorded when a peer connection is torn down, kept alongside the direction the connection was originally opened in. A single disconnect count cannot separate the two situations that matter, because they produce the same number: a node shedding load, which drops peers deliberately because it could not keep up with what it owed them or because a peer exceeded its resource allowance, and a network or topology fault, where the peer became unreachable, stopped answering keepalives, or turned out to be following a different chain. The first is a local capacity problem and the peer list is not the fix; the second is the opposite. A third group is neither — clean teardown at shutdown and peers closing their own side are ordinary churn, and a count dominated by those is healthy. The direction matters separately, since churn among the peers a node dials points somewhere different from churn among the peers that dial it.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Resource disconnect](#resource-disconnect) · [Slot census](#slot-census) · [Insane / diverged peers](#insane-diverged-peers)
<a id="fetch-pack"></a>
### Fetch-pack
@@ -837,6 +917,26 @@ Set-get (fetch) and set-share messages exchange transaction-set data between pee
**Scope:** per node — measured on and specific to this individual server.
<a id="serve-refusal"></a>
### Serve refusal
A peer data request that this node declined to answer — the supply side of the sync exchange, as opposed to everything a node measures about its own fetching. It matters because a node that refuses everything it is asked for looks, from the outside, exactly like a node nobody asks: both serve nothing. From the asking peer's point of view a refusal is indistinguishable from a peer that does not hold the data, so refusals directly slow the sync of every peer that depends on this node. The reason divides them into two kinds. Self-inflicted refusals mean the node was too loaded to answer — its outgoing queue to that peer had grown past its limit, or the local fee track showed it under load, or too much bulk-transfer work was already queued — and these are the serving-side symptom of the same overload that shows up as stalls and job-queue backlog locally. A refusal because the data was simply not held is different: that is a genuine history gap, a question of what this node retains rather than how busy it is.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Fetch-pack](#fetch-pack) · [GetObject / object fetch](#getobject-object-fetch) · [Complete ledger ranges](#complete-ledger-ranges) · [Peer ledger supply](#peer-ledger-supply)
<a id="slot-census"></a>
### Slot census
A single consistent reading of everything PeerFinder knows about this node's peering position: how many outbound and inbound slots are occupied against how many exist, how many outbound attempts are in flight, how many configured fixed peers are connected against how many were configured, and the depth of the two address stores. Taken together at one instant, so the numbers can be compared against each other. The three terms worth defining plainly: an occupied outbound slot is a peer this node dialled and is now connected to; the bootstrap address store is a persisted list of addresses kept across restarts purely so a starting node has somewhere to dial; and the live address store holds addresses learned from peers during this session and exists only in memory. Occupancy alone cannot explain a peering failure, which is the reason the census exists. A node with no outbound peers might be dialling continuously and never completing, or not dialling at all because it has no addresses to try, or dialling only configured peers that are unreachable — three different faults with three different fixes, and the occupancy count is identical in all of them. It is the attempt count, the address-store depths, and the configured-versus-connected comparison that tell them apart.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Overlay](#overlay) · [Disconnect reason](#disconnect-reason) · [Peer ledger supply](#peer-ledger-supply) · [DNS resolve](#dns-resolve) · [Outbound dial latency](#outbound-dial-latency)
<a id="squelch"></a>
### Squelch
@@ -905,6 +1005,16 @@ The NodeStore serves reads through a pool of read threads (optionally bundling r
## Validator Health
<a id="amendment-block-countdown"></a>
### Amendment block countdown
The window between an amendment this build does not understand reaching majority among validators and that amendment actually activating. It exists because amendment activation is not instantaneous: once an unsupported amendment has majority support it becomes expected to activate at a known future time, and until then the node still works normally. That window is the only actionable part of an otherwise terminal condition — after activation the node stops validating and cannot resume without a software upgrade, so there is no operational fix left, only a rebuild and restart. Read as a countdown it therefore outranks every other sync signal in urgency: a node counting down is going to stop validating at a knowable moment, and anything else that looks wrong is secondary. The healthy state is reported as an explicit sentinel value rather than as absent data, so a node with nothing pending is distinguishable from a node whose reporting has broken, and the countdown is held at zero rather than going negative once activation is due. The identity of the blocking amendment is deliberately not carried on the metric — the network can vote on any amendment identifier, including ones this build has never heard of, which would make it an unbounded label — so the hash comes from the log line that records the amendment reaching majority.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Amendment blocked](#amendment-blocked) · [UNL blocked](#unl-blocked) · [Amendments on xrpl.org](https://xrpl.org/docs/concepts/networks-and-servers/amendments)
<a id="amendment-blocked"></a>
### Amendment blocked

190
docs/telemetry-runbook.md
View File

@@ -2310,6 +2310,196 @@ each step gates the next: stop at the first one that is wrong.
waited, so it reports the past; these gauges report what is sitting in
the queue right now, including the part being actively withheld.
11. **Can the network even serve the ledgers this node needs?**
Steps 6 to 10 all assume some peer holds what the node is asking for. This
step tests that assumption, and it is the one that separates "slow" from
"impossible". Panel _Peers Able to Serve Needed Sequence_
(`peer_ledger_supply`, `metric=peers_reporting`,
`peers_serving_validated` and `peers_serving_next`). Read the two counts
together — `peers_reporting` is the denominator that makes the rest
meaningful:
- **`peers_serving_next` at zero with `peers_reporting` above zero** —
the decisive reading. Peers are connected and have advertised their
ranges, and **none of them holds the next ledger this node must
acquire.** No amount of waiting finishes the sync; the peer set itself
has to change. Add peers that hold the range, or point the node at a
full-history server. Everything in steps 6 to 10 will look starved as a
consequence, so do not chase them.
- **`peers_serving_next` above zero but the sync is still slow** — supply
is fine and the fault is downstream. Go back to steps 6 to 10: the data
is available, so the limit is acquire progress, local processing or
worker threads.
- **`peers_reporting` at zero** — nothing has advertised a range yet.
This is not a supply gap; it means the node has no peers, or its peers
have not sent a status change. Peers advertising an empty range are
excluded from every field, so the two window fields read 0 meaning
**unknown**, not genesis — do not read a zero window here as "peers
serve from the start of history". Go back to the Bootstrap row, and to
step 12 for why there are no peers.
Then read _Peer Ledger Supply Window_ (`peer_ledger_supply`,
`metric=supply_min_seq` and `supply_max_seq`) against the sequence the
node wants. This is what tells the two shapes of a supply gap apart: a
needed sequence **below** `supply_min_seq` means the node is asking for
history nobody kept, so it needs a full-history peer; a needed sequence
**above** `supply_max_seq` means it is asking for a tip nobody has
reached, which is a peer set lagging the real network rather than a
history problem.
12. **Is this node failing to get or keep peers, and why?**
Step 11 says whether the peer set can serve; this step says why the peer
set is what it is. Panel _PeerFinder Slot Census_
(`peerfinder_slot_census`) with _PeerFinder Address Caches & Fixed Peers_
beside it. All nine fields come from one lock acquire, so occupancy and
capacity can be compared directly — which is what separates three faults
that otherwise look identical:
- **`connecting` non-zero while `out_active` stays below `out_max`** —
the node is dialling and the dials never complete. Without the attempt
count this looks exactly like a node that is not dialling at all. Pair
it with _Outbound Dial Outcome Rate_ in the Bootstrap row for the stage
that breaks.
- **`bootcache` and `livecache` both at 0** — there is nothing to dial.
No seed addresses at all, so check `[ips]` and DNS in Bootstrap step 1.
- **`fixed_active` below `fixed_configured`** — a peer named in the
configuration is unreachable. `fixed_configured` is what was asked for
and `fixed_active` is what was obtained, so any shortfall names a
specific configured peer to check.
Then split the traffic by direction. _Inbound Peer Accept Outcomes_
(`peer_accept_total`, by `outcome`) covers connections offered **to**
this node; the already-documented `overlay_connect_total{outcome}` in
Bootstrap step 2 covers dials **from** it. Reading both is the only way
to get the full in/out picture: a node refusing every inbound
connection looks the same as one nobody dials until these are separated.
On the inbound side `resource_limit`, `no_slot` and `slot_refused` are
this node declining (load, capacity, or a duplicate), while
`protocol_mismatch`, `bad_cookie` and `handshake_error` point at the
peer or at a network-id mismatch.
Then read _Peer Disconnects by Reason_ (`peer_disconnect_total`, by
`reason` and `direction`). One disconnect count cannot separate the two
causes; the label can:
- `large_sendq`, `charge_resources` — **our fault.** This node could not
keep up with what it owed the peer, or charged it past the resource
limit, so it shed the connection as backpressure. The fix is local
capacity, not the peer list, and it sends you back to steps 9 and 10.
- `not_useful`, `ping_timeout`, `read_error` — topology or network
faults. The peer is on a different chain or unreachable, so the fix is
the peer set.
- `graceful`, `shutdown`, `stopping` — normal churn and clean teardown,
not faults. A run dominated by these is healthy.
Use `direction` to tell churn in the peers this node dials from churn
in the peers that dial it.
Finally, the mirror-image question: _Ledger/Object Serve Refusals_
(`serve_refused_total`, by `request` and `reason`) is what **this node
refuses to serve OTHERS**. It does not explain this node's own sync, but
it explains its peers' — and a node that refuses everything is why some
other operator is reading step 11 on their side. `sendq_full` and
`load_shed` are self-inflicted: this node is too loaded or too far
behind on its send queue to answer, so treat them as the serving-side
symptom of the same overload steps 3, 9 and 10 cover. `not_found` is
different — it is a genuine history gap, meaning the data was asked for
and this node simply does not hold it, which is a configuration and
retention question rather than a load one.
13. **Is the node about to stop validating for good?**
Panel _Amendment Block Countdown_ (`amendment_block`,
`metric=seconds_to_block`) with _Amendment Warned_ (`metric=warned`)
beside it. **This step outranks every other step in urgency**, so check it
whenever a sync looks wrong, not only after the ten above are clean:
- `seconds_to_block` at **-1** — healthy. The -1 is an explicit sentinel
meaning nothing is pending, chosen so the healthy state is a distinct
value rather than a missing series. Do not read it as a negative
duration or as absent data.
- `seconds_to_block` at **any non-negative value** — a countdown to a
**terminal** state. When it expires the node becomes amendment-blocked:
it stops validating and will never validate again without a software
upgrade. The value is clamped at 0 rather than going negative, so a 0
means the activation is due or past due, not that it just started.
Nothing else on this dashboard matters if this is counting down — plan
the upgrade inside the window, because after it there is no
operational fix.
`warned` reaching 1 is the same condition seen as a flag: an
unsupported amendment has reached majority. The existing
`validator_health{metric="amendment_blocked"}` on the Validator Health
dashboard is the after-the-fact companion — it reports the block once it
has happened, when nothing can be done, whereas this countdown is the
only actionable part.
The blocking amendment's hash is **not** a metric label, deliberately:
the network can vote on an arbitrary 256-bit amendment id, not drawn
from this build's known features, so an id label would be unbounded
cardinality and would mint a permanent new series per amendment.
Get the hash from the log line in `AmendmentTableImpl::doValidatedLedger`
("Unsupported amendment ... reached majority at ...") via Loki,
correlated to this series by node and time.
Finally, read _Byzantine Ledger Jumps_ (`ledger_jump_total`) in the
same pass. Any non-zero rate means the node was fed a last-closed
ledger it had not built on and **discarded its own chain tip** to
follow. A single jump during a fresh sync can be benign as the node
settles onto the network's chain. Repeated jumps are wrong-chain
thrash: the node keeps switching between chains and never settles, so
check the peer set from step 12 and the configured network id from
Bootstrap step 3 — those are what put a node on the wrong chain in the
first place. Nothing in the acquire pipeline can fix it.
14. **Is the node store itself the bottleneck — and is it the write side?**
This is the step for the specific symptom **"a node with a large existing
database starts and syncs slower than a fresh one"**. Back-fill is
write-bound, so no read-side panel can show it; check this step whenever a
node with existing history is the slow one.
Panel _NodeStore Write vs Read Latency (us/op)_ (`nodestore_latency`,
`metric=write_mean_us` and `read_mean_us`) with _NodeStore Operation Rate_
(`metric=write_count` / `read_count`) beside it:
- **Write line rising during history back-fill** — the backend cannot
absorb writes fast enough. Sync will stay slow however many peers are
available, so adding peers will not help. Check storage IOPS, the
`[node_db]` backend and its tuning, and whether the online-delete /
rotation cycle is competing with the back-fill writes. Correlate with
`nodestore_state{metric="write_load"}` on the Ledger Data Sync dashboard.
- **Read line far above the write line** — the read path, not the write
path, is the cost. Read it together with _SHAMap TreeNode Cache Hit Rate_
(step 7): a cold in-memory cache sends every tree walk to disk, and that
shows up here as read latency rather than as a node-store fault.
- **Write rate at zero while the node is still behind the network** —
nothing is being persisted at all, so the stall is upstream of the node
store. Go back to peer supply (step 12) and the acquire panels (steps
6-8); storage is not the problem.
- Both panels use the **rate of the mean divided by the rate of the
count**, which is why the count series exist. Read as an interval
latency, not a since-boot average — on a long-running node the raw
cumulative mean moves so slowly that a current stall is invisible in it.
- Two limits to keep in mind. First, this is a **mean, not a percentile**:
a tail that matters will move it, but there is no p99 here. That is a
deliberate cost trade — a histogram would need one `Record()` per node
object, and a single ledger write walks thousands of SHAMap nodes.
Second, `write_mean_us` is currently emitted only for store paths that
record their duration, which today is the `[import_db]` admin import.
On an ordinary node you will see `write_count` climbing with **no**
`write_mean_us` line: that is a known instrumentation gap, not a healthy
zero, and the mean is deliberately omitted rather than drawn as 0 so it
cannot be misread as "writes are instantaneous".
15. **Is replay-based back-fill silently falling back to the slow path?**
Only relevant when `[ledger_replay]` is enabled. Panels _Replay Fallback to
Full Acquire (by stage)_ (`ledger_replay_fallback_total`) and _Replay
Outcomes (by terminal state)_ (`ledger_replay_outcome_total`):
- **Any sustained fallback rate** — too few connected peers support the
`LedgerReplay` protocol feature, so every historical ledger is fetched
whole instead of as a delta. Back-fill still completes, just far slower,
which is why this is easy to miss: nothing fails, the optimisation is
simply gone. The `stage` label says which sub-task gave up — `skiplist`
(fetching the list of historical ledger hashes) or `delta` (a single
ledger's changes). Fix by peering with nodes that support the feature.
- **Failure outcomes climbing while `success` stays flat** — replay runs
but never completes. The outcome names the layer at fault: `timeout`
means the deltas never arrived, so treat it as a peer-supply problem and
read it with step 12; `build_failed` means a delta would not apply to its
parent, and `parameter_failed` means a peer served a skip list
inconsistent with the request — those two are **data** faults from the
serving peers, not slowness, so the peer set is suspect rather than the
network.
- **All series absent** — expected when `[ledger_replay]` is not
configured, or on a node with no history to back-fill. Absence here is
not a regression; it means no replay task was ever created. For the same
reason neither counter is asserted by the local validation harness.
## Performance Tuning
| Scenario | Recommendation |

1173
src/tests/libxrpl/telemetry/MetricMacros.cpp
View File

File diff suppressed because it is too large Load Diff

50
src/tests/libxrpl/telemetry/MetricsRegistry.cpp
View File

@@ -9,7 +9,8 @@
* - Destructor handles cleanup without crash.
* - Compile-time-disabled proof for the sync-diagnostics gauges: the whole
* async-gauge registration surface is compiled away, and a full disabled
* lifecycle never touches any ServiceRegistry service.
* lifecycle never touches any ServiceRegistry service -- including the
* Overlay and AmendmentTable the WP-A7 gauges would read.
*
* NOTE: These tests only exercise the no-op path (telemetry disabled).
* When XRPL_ENABLE_TELEMETRY is defined, MetricsRegistry.cpp pulls in
@@ -19,7 +20,8 @@
* CONSEQUENCE for the sync-diagnostics gauges (`unl_quorum`,
* `clock_close_offset_seconds`, `sync_state`,
* `server_stall_events_total`, `sync_acquire`, `shamap_cache_hit_rate`,
* `jobq_backlog`, `jobq_saturation`):
* `jobq_backlog`, `jobq_saturation`, `peer_ledger_supply`,
* `peerfinder_slot_census`, `amendment_block`, `nodestore_latency`):
* this file CANNOT assert an observed gauge
* value, because on this build the gauges do not exist -- their registration
* methods and the OTel instrument members are inside
@@ -27,8 +29,10 @@
* is provable here, and what the tests below assert, is the complementary
* half: that nothing is registered and no service is consulted. The exact
* observed values (trusted_keys=5, quorum=4, offset=-3, the sync_state /
* stall-episode values, the acquire-progress / cache-hit-rate values, and the
* per-type backlog / pool-saturation values) are
* stall-episode values, the acquire-progress / cache-hit-rate values, the
* per-type backlog / pool-saturation values, the peer-supply /
* slot-census / amendment-countdown values, and the nodestore
* read/write mean-latency values) are
* asserted in MetricMacros.cpp, which is the file compiled when telemetry IS
* enabled.
*/
@@ -385,10 +389,14 @@ TEST_F(MetricsRegistryTest, destructor_calls_stop)
// `server_stall_events_total` read NetworkOPs and LoadManager; `sync_acquire`
// reads InboundLedgers::acquireProgress() and `shamap_cache_hit_rate` reads the
// node Family's tree-node cache; `jobq_backlog` and `jobq_saturation` read
// JobQueue::getJobTypeCounts() / getWorkerSaturation(). All are
// JobQueue::getJobTypeCounts() / getWorkerSaturation(); `peer_ledger_supply` and
// `peerfinder_slot_census` read Overlay::getPeerLedgerSupply() /
// getSlotCensus() and `amendment_block` reads
// AmendmentTable::firstUnsupportedExpected(). All are
// reached through the ServiceRegistry, and MockServiceRegistry::getValidators()
// / getTimeKeeper() / getOPs() / getLoadManager() / getInboundLedgers() /
// getNodeFamily() / getJobQueue() THROW std::logic_error. So "no
// getNodeFamily() / getJobQueue() / getOverlay() / getAmendmentTable() THROW
// std::logic_error. So "no
// gauge callback ran" is directly observable here: had registerAsyncGauges() run
// and had a callback fired, one of those accessors would have thrown.
//
@@ -429,7 +437,9 @@ TEST_F(MetricsRegistryTest, disabled_lifecycle_never_consults_gauge_services)
// registerUnlQuorumGauge() / registerClockSkewGauge() /
// registerSyncStateGauge() / registerStallEventsCounter() /
// registerSyncAcquireGauge() / registerCacheHitRateDetailGauge() /
// registerJobQueueBacklogGauge() / registerJobQueueSaturationGauge() --
// registerJobQueueBacklogGauge() / registerJobQueueSaturationGauge() /
// registerPeerLedgerSupplyGauge() / registerSlotCensusGauge() /
// registerAmendmentBlockGauge() / registerNodeStoreLatencyGauge() --
// would run.
EXPECT_NO_THROW(registry.start("http://localhost:4318/v1/metrics"));
@@ -464,6 +474,25 @@ TEST_F(MetricsRegistryTest, disabled_lifecycle_never_consults_gauge_services)
// both on the JobQueue. Neither was consulted above, so neither gauge
// took the JobQueue mutex on a telemetry-off build.
EXPECT_THROW(mockApp_.getJobQueue(), std::logic_error);
// The service both WP-A7 peer gauges read: peer_ledger_supply polls
// getPeerLedgerSupply(), which walks the active-peer list, and
// peerfinder_slot_census polls getSlotCensus(), which takes the PeerFinder
// lock. Both go through the Overlay, so a single throw here proves neither
// gauge walked the peer list nor took the PeerFinder lock on a
// telemetry-off build.
EXPECT_THROW(mockApp_.getOverlay(), std::logic_error);
// The service the WP-A7 amendment countdown reads: amendment_block polls
// firstUnsupportedExpected() on the AmendmentTable, which takes that
// table's mutex. Not consulted above, so the countdown never ran. (Its
// `warned` half reads NetworkOPs, already covered by the getOPs() check.)
EXPECT_THROW(mockApp_.getAmendmentTable(), std::logic_error);
// The service the WP-A6 nodestore latency gauge reads: nodestore_latency
// polls getStoreDurationUs()/getStoreCount() and
// getFetchDurationUs()/getFetchTotalCount() on the node-store Database.
// Not consulted above, so the latency gauge never read those atomics on a
// telemetry-off build. (The existing nodestore_state gauge reads the same
// service, so this single throw covers both.)
EXPECT_THROW(mockApp_.getNodeStore(), std::logic_error);
}
// Even asking for enabled=true registers no sync-diagnostics gauge on a
@@ -484,9 +513,12 @@ TEST_F(MetricsRegistryTest, enabled_flag_alone_registers_no_gauges_when_compiled
// registered registerUnlQuorumGauge()/registerClockSkewGauge()/
// registerSyncStateGauge()/registerStallEventsCounter()/
// registerSyncAcquireGauge()/registerCacheHitRateDetailGauge()/
// registerJobQueueBacklogGauge()/registerJobQueueSaturationGauge(), a
// registerJobQueueBacklogGauge()/registerJobQueueSaturationGauge()/
// registerPeerLedgerSupplyGauge()/registerSlotCensusGauge()/
// registerAmendmentBlockGauge()/registerNodeStoreLatencyGauge(), a
// callback would reach getValidators()/getTimeKeeper()/getOPs()/
// getLoadManager()/getInboundLedgers()/getNodeFamily()/getJobQueue() and
// getLoadManager()/getInboundLedgers()/getNodeFamily()/getJobQueue()/
// getOverlay()/getAmendmentTable()/getNodeStore() and
// throw std::logic_error.
EXPECT_NO_THROW(enabledRequest.start("http://localhost:4318/v1/metrics"));
EXPECT_NO_THROW(enabledRequest.detachCallbacks());

296
src/xrpld/telemetry/MetricsRegistry.cpp
View File

@@ -41,6 +41,7 @@
#include <xrpl/core/JobTypes.h>
#include <xrpl/core/ServiceRegistry.h>
#include <xrpl/json/json_value.h>
#include <xrpl/ledger/AmendmentTable.h>
#include <xrpl/nodestore/Database.h>
#include <xrpl/protocol/BuildInfo.h>
#include <xrpl/protocol/jss.h>
@@ -494,6 +495,11 @@ MetricsRegistry::registerAsyncGauges()
registerCacheHitRateDetailGauge();
registerJobQueueBacklogGauge();
registerJobQueueSaturationGauge();
registerPeerLedgerSupplyGauge();
registerSlotCensusGauge();
registerAmendmentBlockGauge();
registerNodeStoreLatencyGauge();
registerLedgerQuorumPublishGauge();
}
void
@@ -1818,6 +1824,296 @@ MetricsRegistry::registerJobQueueSaturationGauge()
this);
}
void
MetricsRegistry::registerPeerLedgerSupplyGauge()
{
// --- Sync diagnostics: can the network even serve what I need? ---
// Each peer advertises its ledger range and the connection caches it, but
// nothing ever compared those ranges, so "no peer holds the sequence I
// want" looked exactly like "my peers are slow" -- two faults with
// completely different fixes.
peerLedgerSupplyGauge_ = meter_->CreateInt64ObservableGauge(
"peer_ledger_supply", "Peer coverage of the ledger sequence this node needs");
peerLedgerSupplyGauge_->AddCallback(
[](opentelemetry::metrics::ObserverResult result, void* state) {
auto* self = static_cast<MetricsRegistry*>(state);
if (self->callbacksDetached_.load(std::memory_order_acquire))
return;
auto& app = self->app_;
try
{
auto observe = [&](char const* field, int64_t value) {
opentelemetry::nostd::get<opentelemetry::nostd::shared_ptr<
opentelemetry::metrics::ObserverResultT<int64_t>>>(result)
->Observe(value, {{"metric", field}});
};
// One pass over the peers, so all five series describe the
// same peer set at the same instant.
auto const supply = app.getOverlay().getPeerLedgerSupply(
app.getLedgerMaster().getValidLedgerIndex());
// The denominator. Zero serving out of zero reporting is
// silence; zero out of many is a real supply gap.
observe("peers_reporting", supply.peersReporting);
observe("peers_serving_validated", supply.peersServingValidated);
// The verdict: zero here while peers_reporting is non-zero
// means waiting cannot finish the sync.
observe("peers_serving_next", supply.peersServingNext);
// The window the peer set covers, so an operator can tell a
// request for discarded history from one for an unreached tip.
observe("supply_min_seq", supply.supplyMinSeq);
observe("supply_max_seq", supply.supplyMaxSeq);
}
catch (...) // NOLINT(bugprone-empty-catch)
{
// Silently skip if services are not yet ready.
}
},
this);
}
void
MetricsRegistry::registerSlotCensusGauge()
{
// --- Sync diagnostics: why can this node not get peers? ---
// All nine numbers already exist inside PeerFinder; only the two active
// counts are exported today, which cannot distinguish "not dialling",
// "dialling and failing" and "nothing to dial".
slotCensusGauge_ = meter_->CreateInt64ObservableGauge(
"peerfinder_slot_census", "PeerFinder slots, connection attempts and address caches");
slotCensusGauge_->AddCallback(
[](opentelemetry::metrics::ObserverResult result, void* state) {
auto* self = static_cast<MetricsRegistry*>(state);
if (self->callbacksDetached_.load(std::memory_order_acquire))
return;
auto& app = self->app_;
try
{
auto observe = [&](char const* field, int64_t value) {
opentelemetry::nostd::get<opentelemetry::nostd::shared_ptr<
opentelemetry::metrics::ObserverResultT<int64_t>>>(result)
->Observe(value, {{"metric", field}});
};
// One snapshot under one PeerFinder lock acquire, so occupancy
// and capacity can be compared against each other.
auto const census = app.getOverlay().getSlotCensus();
observe("out_active", census.outActive);
observe("out_max", census.outMax);
observe("in_active", census.inActive);
observe("in_max", census.inMax);
// Dials in flight. Non-zero while out_active stays under
// out_max is the "starting and never completing" case.
observe("connecting", census.connecting);
// fixed_active below fixed_configured names a configured peer
// that cannot be reached.
observe("fixed_configured", census.fixedConfigured);
observe("fixed_active", census.fixedActive);
// Both at zero on a fresh node means there is nothing to dial.
observe("bootcache", census.bootcache);
observe("livecache", census.livecache);
}
catch (...) // NOLINT(bugprone-empty-catch)
{
// Silently skip if services are not yet ready.
}
},
this);
}
void
MetricsRegistry::registerAmendmentBlockGauge()
{
// --- Sync diagnostics: how long until this node stops validating? ---
// The existing validator_health{metric="amendment_blocked"} reports the
// terminal state, when nothing can be done. This is the window before it.
amendmentBlockGauge_ = meter_->CreateInt64ObservableGauge(
"amendment_block", "Amendment-block warning and seconds until the node stops validating");
amendmentBlockGauge_->AddCallback(
[](opentelemetry::metrics::ObserverResult result, void* state) {
auto* self = static_cast<MetricsRegistry*>(state);
if (self->callbacksDetached_.load(std::memory_order_acquire))
return;
auto& app = self->app_;
try
{
auto observe = [&](char const* field, int64_t value) {
opentelemetry::nostd::get<opentelemetry::nostd::shared_ptr<
opentelemetry::metrics::ObserverResultT<int64_t>>>(result)
->Observe(value, {{"metric", field}});
};
// An unsupported amendment has reached majority. Until now this
// only surfaced as an admin-only server_info warning.
observe("warned", app.getOPs().isAmendmentWarned() ? 1 : 0);
// Seconds until that amendment activates. -1 means nothing is
// pending: a distinct healthy value rather than an absent
// series, matching validator_health{metric="unl_expiry_days"}.
std::int64_t secondsToBlock = -1;
if (auto const expected = app.getAmendmentTable().firstUnsupportedExpected())
{
// NetClock's representation is unsigned, so the difference
// is taken in int64_t: subtracting the time_points directly
// would wrap once the activation time has passed.
auto const expectedSecs =
static_cast<std::int64_t>(expected->time_since_epoch().count());
auto const nowSecs = static_cast<std::int64_t>(
app.getTimeKeeper().closeTime().time_since_epoch().count());
// Clamped at 0: past due means the block is imminent, not
// overdue by an amount worth charting.
secondsToBlock = std::max<std::int64_t>(expectedSecs - nowSecs, 0);
}
observe("seconds_to_block", secondsToBlock);
}
catch (...) // NOLINT(bugprone-empty-catch)
{
// Silently skip if services are not yet ready.
}
},
this);
}
void
MetricsRegistry::registerNodeStoreLatencyGauge()
{
// --- Sync diagnostics: is the node store slow, and on which side? ---
// The write mean is the new signal. storeDurationUs_ was declared and
// never written, so no write latency existed anywhere; only the read side
// had a duration total. A node with a large existing DB back-fills slower
// than a fresh one, and back-fill is write-bound, so the read-side
// metrics cannot show it. Exporting both means from one reading also makes
// the two sides directly comparable.
//
// Gauge rather than histogram: a histogram would cost one Record() per
// node object on the store/fetch path, which runs thousands of times per
// ledger write. This reads four atomics per ~10 s tick instead. The
// trade-off is that percentiles are unavailable -- see the header comment.
nodeStoreLatencyGauge_ = meter_->CreateInt64ObservableGauge(
"nodestore_latency", "NodeStore mean store/fetch latency in microseconds, with counts");
nodeStoreLatencyGauge_->AddCallback(
[](opentelemetry::metrics::ObserverResult result, void* state) {
auto* self = static_cast<MetricsRegistry*>(state);
if (self->callbacksDetached_.load(std::memory_order_acquire))
return;
auto& app = self->app_;
try
{
auto observe = [&](char const* field, int64_t value) {
opentelemetry::nostd::get<opentelemetry::nostd::shared_ptr<
opentelemetry::metrics::ObserverResultT<int64_t>>>(result)
->Observe(value, {{"metric", field}});
};
auto& db = app.getNodeStore();
// One reading of each pair, so a mean and its own denominator
// describe the same instant.
auto const storeCount = db.getStoreCount();
auto const storeDurationUs = db.getStoreDurationUs();
auto const fetchCount = db.getFetchTotalCount();
auto const fetchDurationUs = db.getFetchDurationUs();
// Counts are always observed, including zero: that is what
// separates "nothing written yet" from "writes are instant".
observe("write_count", static_cast<int64_t>(storeCount));
observe("read_count", static_cast<int64_t>(fetchCount));
// A mean needs a non-zero denominator, and it needs a
// numerator that was actually measured. Both are required, and
// the series is omitted rather than observed as 0 when either
// is missing: a reported 0 us would claim writes are
// instantaneous, which is worse than no reading at all.
//
// The numerator guard is load-bearing, not defensive.
// Database::store() is pure virtual, so only the store paths
// that call recordStoreDuration() contribute. Today that is
// Database::importInternal (the [import_db] path). The two
// concrete runtime databases -- DatabaseNodeImp::store and
// DatabaseRotatingImp::store -- do not call it yet, so on an
// ordinary node write_count climbs while the duration total
// stays 0. Omitting the mean makes that a visible data gap
// instead of a false "writes take 0 us" line on the panel.
if (storeCount > 0 && storeDurationUs > 0)
observe("write_mean_us", static_cast<int64_t>(storeDurationUs / storeCount));
if (fetchCount > 0 && fetchDurationUs > 0)
observe("read_mean_us", static_cast<int64_t>(fetchDurationUs / fetchCount));
}
catch (...) // NOLINT(bugprone-empty-catch)
{
// Silently skip if services are not yet ready.
}
},
this);
}
void
MetricsRegistry::registerLedgerQuorumPublishGauge()
{
// --- Sync diagnostics: the quorum gate and the publish pipeline ---
// The last two stages of a fresh sync, and the two whose failures were
// invisible: a node can hold every ledger it needs and still never declare
// one validated (quorum short), or validate correctly and never publish
// (pipeline behind). Both used to be trace-log-only or not derivable at all.
ledgerQuorumPublishGauge_ = meter_->CreateInt64ObservableGauge(
"ledger_quorum_publish",
"Pre-accept quorum gate and publish lag (tally vs quorum, first-validated, lag)");
ledgerQuorumPublishGauge_->AddCallback(
[](opentelemetry::metrics::ObserverResult result, void* state) {
auto* self = static_cast<MetricsRegistry*>(state);
if (self->callbacksDetached_.load(std::memory_order_acquire))
return;
auto& app = self->app_;
try
{
auto observe = [&](char const* field, int64_t value) {
opentelemetry::nostd::get<opentelemetry::nostd::shared_ptr<
opentelemetry::metrics::ObserverResultT<int64_t>>>(result)
->Observe(value, {{"metric", field}});
};
auto const& ledgerMaster = app.getLedgerMaster();
// The pair that separates "slow" from "stuck". A tally climbing
// toward the target will get there; a tally flat below it never
// will, and no acquire or peer panel says which is happening.
observe("trusted_validation_tally", ledgerMaster.getTrustedValidationTally());
// What the last gate evaluation actually required, as opposed to
// unl_quorum{quorum} which is what the trusted list configures.
// Already clamped against the SIZE_MAX "quorum disabled"
// sentinel by LedgerMaster, so this never wraps negative.
observe("quorum_target", ledgerMaster.getQuorumTarget());
// One-shot: a value is the time the first ledger took to pass
// the gate, and 0 means it never has. Not a trend.
observe("time_to_first_validated_us", ledgerMaster.getTimeToFirstValidatedUs());
// Validated but not yet published. pubLedgerSeq_ was never
// exported, so this gap was not derivable from any other series.
observe("publish_lag", ledgerMaster.getPublishLag());
}
catch (...) // NOLINT(bugprone-empty-catch)
{
// Silently skip if services are not yet ready.
}
},
this);
}
#endif // XRPL_ENABLE_TELEMETRY
// -----------------------------------------------------------------

277
src/xrpld/telemetry/MetricsRegistry.h
View File

@@ -66,6 +66,12 @@
* | server stall seconds, ledgers behind network)
* +-- JobQueue backlog (waiting/running/deferred per job type)
* +-- JobQueue saturation (running tasks vs worker threads vs backlog)
* +-- Peer ledger supply (how many peers can serve the needed sequence)
* +-- PeerFinder slot census (slots, attempts, fixed peers, address caches)
* +-- Amendment block (warned flag + seconds until the node stops validating)
* +-- NodeStore latency (mean us per store and per fetch, with counts)
* +-- Ledger quorum + publish (validation tally vs quorum target,
* | time to first validated, publish lag)
* +-- jq_trans_overflow_total (observed from Overlay)
* +-- server_stall_events_total (observed from LoadManager)
*
@@ -596,6 +602,38 @@ private:
*/
opentelemetry::nostd::shared_ptr<opentelemetry::metrics::ObservableInstrument>
jobQueueSaturationGauge_;
/**
* Observable gauge for how much of the needed ledger range the connected
* peer set can actually serve.
*/
opentelemetry::nostd::shared_ptr<opentelemetry::metrics::ObservableInstrument>
peerLedgerSupplyGauge_;
/**
* Observable gauge for PeerFinder slot occupancy, connection attempts,
* fixed peers and address-cache depth.
*/
opentelemetry::nostd::shared_ptr<opentelemetry::metrics::ObservableInstrument> slotCensusGauge_;
/**
* Observable gauge for the amendment-block warning flag and the countdown
* to the amendment activating.
*/
opentelemetry::nostd::shared_ptr<opentelemetry::metrics::ObservableInstrument>
amendmentBlockGauge_;
/**
* Observable gauge for node-store read and write latency, as mean
* microseconds per operation derived from the cumulative duration and
* operation-count totals the node store already keeps.
*/
opentelemetry::nostd::shared_ptr<opentelemetry::metrics::ObservableInstrument>
nodeStoreLatencyGauge_;
/**
* Observable gauge for the pre-accept quorum gate and the publish lag:
* the trusted-validation tally against the quorum it must reach, the
* time to the first fully-validated ledger, and how far publishing
* trails validation.
*/
opentelemetry::nostd::shared_ptr<opentelemetry::metrics::ObservableInstrument>
ledgerQuorumPublishGauge_;
/**
* Observable gauge for build version info (label-based, value=1).
*/
@@ -978,7 +1016,244 @@ private:
*/
void
registerJobQueueSaturationGauge(); // sync diagnostics: pool saturation
#endif // XRPL_ENABLE_TELEMETRY
/**
* Register the `peer_ledger_supply` gauge.
*
* Five series under the `metric` attribute, from one
* Overlay::getPeerLedgerSupply() pass over the active peers:
*
* `peers_reporting` — peers that have advertised a ledger range at all.
* The denominator that makes the rest readable.
* `peers_serving_validated` — peers whose range covers this node's
* validated sequence.
* `peers_serving_next` — **the signal this gauge exists for.** Peers
* whose range covers validated + 1, the next ledger this node must
* acquire. Zero here with a non-zero `peers_reporting` means no
* connected peer holds what this node needs, so no amount of waiting
* will finish the sync; the peer set has to change.
* `supply_min_seq` / `supply_max_seq` — the sequence window the peer set
* covers, so an operator can see whether the node is asking for
* history nobody kept or for a tip nobody has reached.
*
* Each peer already caches the range it advertises in mtSTATUS_CHANGE
* (`PeerImp::minLedger_` / `maxLedger_`, read via `Peer::ledgerRange()`),
* but those ranges were never compared against each other, so "no peer has
* what I need" was indistinguishable from "my peers are slow" — the two
* faults with completely different fixes.
*
* Distinct from what already exists. `server_info{metric="peers"}` is a
* bare connection count with no notion of what those peers hold.
* `sync_state{metric="ledgers_behind"}` uses the same per-peer maxima but
* collapses them to a single distance-to-tip number, which cannot say how
* many peers can serve that distance or whether the range has a hole.
* `peer_quality{metric="peers_insane_count"}` counts peers on a different
* chain, which is a correctness signal, not an availability one.
*
* Peers advertising [0, 0] have not reported yet and are excluded from
* every field, so they cannot make a healthy peer set appear to serve from
* genesis. When nothing has reported, both window fields read 0, which is
* why `peers_reporting` must be read alongside them.
*
* @note Pulled on the OTel reader thread (~10 s tick), never on a message
* path. O(peers): `getActivePeers()` copies the peer list under the overlay
* lock and releases it, then each peer's cached range is read under that
* peer's own short-lived lock.
*/
void
registerPeerLedgerSupplyGauge(); // sync diagnostics: peer range coverage
/**
* Register the `peerfinder_slot_census` gauge.
*
* Nine series under the `metric` attribute, from one
* Overlay::getSlotCensus() snapshot: `out_active`, `out_max`, `in_active`,
* `in_max`, `connecting`, `fixed_configured`, `fixed_active`, `bootcache`
* and `livecache`.
*
* All nine are already computed inside PeerFinder (`Counts`, `Bootcache`,
* `Livecache`, the fixed-peer map) and only two of them are exported
* today, as the legacy beast::insight gauges
* `peer_finder_active_inbound_peers` and
* `peer_finder_active_outbound_peers`. Those two carry no capacity,
* attempt or cache term, which leaves the three most common bootstrap
* failures invisible:
*
* - `connecting` non-zero while `out_active` stays below `out_max` —
* dials are being started and never completing. Without the attempt
* count this looks the same as a node that is not dialling at all.
* - `bootcache` at 0 — no seed addresses to dial in the first place.
* - `fixed_active` below `fixed_configured` — a peer named in the
* configuration is unreachable.
*
* The nine fields come from a single acquire of the PeerFinder lock, so
* they are mutually consistent and share one label set. The two legacy
* gauges are read at unrelated instants and cannot be joined with each
* other, let alone with a capacity term.
*
* @note Pulled on the OTel reader thread (~10 s tick). One lock acquire,
* then integer and container-size reads.
*/
void
registerSlotCensusGauge(); // sync diagnostics: peerfinder slot census
/**
* Register the `amendment_block` gauge.
*
* Two series under the `metric` attribute:
*
* `warned` — 1 once an unsupported amendment has reached majority, from
* NetworkOPs::isAmendmentWarned().
* `seconds_to_block` — **the leading indicator.** Seconds until that
* amendment activates, derived from
* `AmendmentTable::firstUnsupportedExpected()` against the network
* close time. `-1` when nothing is pending, matching the sentinel
* `validator_health{metric="unl_expiry_days"}` already uses, so the
* healthy state is a distinct value rather than a missing series.
* Clamped at 0 rather than going negative, because past-due means the
* block is imminent, not overdue by some amount.
*
* Amendment-blocked is a terminal sync blocker: the node stops validating
* and never resumes without a software upgrade. The existing
* `validator_health{metric="amendment_blocked"}` reports that state after
* it has happened, when nothing can be done about it. This gauge is the
* window before it, which is the only actionable part.
*
* The blocking amendment's identity is deliberately NOT a label. The
* network can vote on an arbitrary 256-bit amendment id — the set is not
* drawn from this build's known features — so an id label would be
* unbounded cardinality and would mint a permanent new series per
* amendment. The id is already logged by
* `AmendmentTableImpl::doValidatedLedger` ("Unsupported amendment <hash>
* reached majority at ..."), so it is available through logs, correlated
* to this series by node and time.
*
* @note Pulled on the OTel reader thread (~10 s tick). One mutex acquire
* inside the amendment table plus one clock read.
* @note The subtraction is done in `std::int64_t`, not in NetClock's
* unsigned representation, so a past-due activation cannot wrap to a huge
* positive count.
*/
void
registerAmendmentBlockGauge(); // sync diagnostics: amendment countdown
/**
* Register the `nodestore_latency` gauge.
*
* Four series under the `metric` attribute, from the node store's own
* cumulative totals:
*
* `write_mean_us` — **the signal this gauge exists for.** Mean
* microseconds per store, `getStoreDurationUs() / getStoreCount()`.
* No write-side latency existed anywhere before this:
* `storeDurationUs_` was declared in Database.h and never written, and
* there was no accessor for it. This is the fingerprint of the
* "a node with a large existing DB syncs slower than a fresh one"
* symptom, which is write-bound and therefore invisible in every
* read-side metric.
* `read_mean_us` — mean microseconds per fetch,
* `getFetchDurationUs() / getFetchTotalCount()`, so the write mean has
* a same-instant, same-derivation counterpart to be compared against.
* `write_count` / `read_count` — the denominators, exported so a
* dashboard can recover *interval* latency as
* `rate(duration) / rate(count)`. Without them the means above are
* since-boot averages, which on a long-running node move so slowly
* that a current stall is invisible.
*
* Gauge, not a histogram — deliberate. A histogram would give true
* percentiles, which a mean cannot, but it costs one `Record()` per
* operation on a path that runs per node object: a single ledger write
* walks thousands of SHAMap nodes, and fetches are more frequent still.
* That is a per-object synchronous instrument call plus bucket search on
* the hot store/fetch path. This gauge instead reads four already-existing
* atomics once per ~10 s collection tick, adding nothing whatsoever to the
* hot path — the store side pays only the one clock-sample pair per store
* that the read side has always paid per fetch. For the question this work
* package answers ("is the write path slow, and slower than the read
* path?") a rate-derived mean is sufficient, and a tail latency that
* matters will move the mean. Consequence, stated plainly: p99 is NOT
* obtainable from this signal. Adding a histogram later would also require
* an explicit-bucket View registered in initExporterAndProvider() via
* addMicrosecondHistogramView(), because the SDK's default buckets top out
* at 10,000 and every microsecond duration above 10 ms would saturate.
*
* Distinct from `nodestore_state`, which already carries the raw
* cumulative `node_reads_duration_us`, `node_reads_total` and
* `node_writes` fields, and from the Ledger Data Sync dashboard's "NuDB
* Read Latency" panel that divides the first two in PromQL. Neither has
* any write-duration input to divide — that quantity did not exist. This
* gauge adds the missing write numerator and publishes both means from one
* reading so the two sides are directly comparable.
*
* @note Pulled on the OTel reader thread (~10 s tick). Four relaxed atomic
* loads and two integer divisions; no lock, no allocation, no hot-path
* cost.
* @note A mean is observed only when both its count and its duration total
* are non-zero; otherwise the series is omitted rather than reported as 0,
* because a 0 would claim the operation is instantaneous. The counts are
* always observed, so `write_count` still distinguishes "nothing written
* yet" from "writes are instant".
* @warning `write_mean_us` is currently produced only by store paths that
* call `Database::recordStoreDuration()`, which today is
* `Database::importInternal` (the `[import_db]` admin path). `store()` is
* pure virtual, and neither `DatabaseNodeImp::store` nor
* `DatabaseRotatingImp::store` calls it yet, so on an ordinary node
* `write_count` climbs while `write_mean_us` is absent. That is a
* deliberate, visible gap: closing it means adding one clock-sample pair to
* those two concrete store overrides, which live outside this work
* package's file scope.
* @note Both totals are monotonic and never reset. A panel wanting current
* rather than since-boot latency must divide the two rates, which is why
* the counts are exported alongside the means.
*/
void
registerNodeStoreLatencyGauge(); // sync diagnostics: store/fetch latency
/**
* Register the `ledger_quorum_publish` gauge.
*
* Four series under the `metric` attribute, read from LedgerMaster:
*
* `trusted_validation_tally` — trusted validations counted at the last
* pre-accept gate in `LedgerMaster::checkAccept`.
* `quorum_target` — validations that gate required. **The pair is the
* signal.** The tally alone cannot separate a node accumulating
* validations toward quorum (slow, will finish) from one whose tally
* plateaus below the target (stuck, never will); with the target
* beside it, the two shapes are unmistakable.
* `time_to_first_validated_us` — how long the node took to get its
* first ledger through that gate. One-shot, like the
* `sync_state{initial_full_duration_us}` milestone: a value means it
* happened and this is how long it took, 0 means it never has.
* `publish_lag` — validated sequence minus published sequence. Non-zero
* and growing means validation is fine and the publish pipeline is
* behind, which no other signal distinguishes.
*
* All four are grouped under one instrument because they answer one
* question in sequence — did enough validations arrive, did the gate pass,
* how long did that take, and did the result reach clients — so an
* operator reads them from a single consistent poll.
*
* Distinct from what already exists. `unl_quorum{quorum}` is the quorum
* the validator list *configures*, a static property of the trusted set;
* `quorum_target` is what an actual gate evaluation *required*, and the
* tally beside it is the live count that must reach it — neither existed
* anywhere before. `server_info{validated_ledger_seq}` publishes the
* validated sequence but nothing published the pubLedgerSeq_ counterpart,
* so the lag between them was not derivable at all.
*
* @note `quorum_target` reports int64 max when the validator list has
* switched quorum off (`ValidatorList::quorum()` returns SIZE_MAX). The
* clamp lives in `LedgerMaster::checkAccept`, so the wrap to -1 that would
* invert a tally-versus-target panel cannot happen here.
* @note Pulled on the OTel reader thread (~10 s tick). Five relaxed atomic
* loads through lock-free LedgerMaster accessors: no lock is taken, which
* is what keeps an OTel callback from ever contending with, or inverting
* lock order against, the LedgerMaster mutex held by the emit path.
*/
void
registerLedgerQuorumPublishGauge(); // sync diagnostics: quorum + publish
#endif // XRPL_ENABLE_TELEMETRY
};
} // namespace telemetry