From 827525c86bd76059dd398e6524ee067f136861a0 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Sat, 25 Jul 2026 16:15:26 +0100 Subject: [PATCH] feat(telemetry): wire A5-A7 and B1 signals through the pipeline Registers the new gauges, renders them, asserts them and documents them, so each signal reaches an operator rather than stopping at the emit site: - MetricsRegistry: gauge registration for ledger_quorum_publish, nodestore_latency, peer_ledger_supply, peerfinder_slot_census and amendment_block, each guarded by the detached-callbacks check and tolerant of services that are not ready yet. - Ledger Sync Health dashboard: panels for the new signals, filtered by the node template variable like every other board. - Workload validation: the new series are asserted, so a signal that regresses to absent fails CI. Signals the local cluster structurally cannot produce, such as a replay fallback or an amendment block, are noted rather than asserted, which would fail red on a healthy run. - Reference, runbook and glossary entries, including the diagnosis order for a node that has peers and validators but never validates. - Regenerated levelization baseline: three new one-way edges from the telemetry and test modules, no new cycles. Also drops an unused cstddef include from the macro tests, which the include checker rejects. Co-Authored-By: Claude Opus 5 (1M context) --- .../scripts/levelization/results/ordering.txt | 3 + .../09-data-collection-reference.md | 67 +- .../dashboards/ledger-sync-health.json | 1522 +++++++++++++++++ .../telemetry/workload/expected_metrics.json | 26 +- docs/telemetry-glossary.md | 110 ++ docs/telemetry-runbook.md | 190 ++ src/tests/libxrpl/telemetry/MetricMacros.cpp | 1173 ++++++++++++- .../libxrpl/telemetry/MetricsRegistry.cpp | 50 +- src/xrpld/telemetry/MetricsRegistry.cpp | 296 ++++ src/xrpld/telemetry/MetricsRegistry.h | 277 ++- 10 files changed, 3660 insertions(+), 54 deletions(-) diff --git a/.github/scripts/levelization/results/ordering.txt b/.github/scripts/levelization/results/ordering.txt index f0a7741368..92f14ed1ed 100644 --- a/.github/scripts/levelization/results/ordering.txt +++ b/.github/scripts/levelization/results/ordering.txt @@ -200,6 +200,8 @@ tests.libxrpl > xrpl.basics tests.libxrpl > xrpl.config tests.libxrpl > xrpl.core tests.libxrpl > xrpld.app +tests.libxrpl > xrpld.overlay +tests.libxrpl > xrpld.peerfinder tests.libxrpl > xrpld.telemetry tests.libxrpl > xrpl.json tests.libxrpl > xrpl.ledger @@ -336,6 +338,7 @@ xrpld.telemetry > xrpl.core xrpld.telemetry > xrpld.consensus xrpld.telemetry > xrpld.core xrpld.telemetry > xrpl.json +xrpld.telemetry > xrpl.ledger xrpld.telemetry > xrpl.nodestore xrpld.telemetry > xrpl.protocol xrpld.telemetry > xrpl.rdb diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md index 813965ced7..53e3ec00db 100644 --- a/OpenTelemetryPlan/09-data-collection-reference.md +++ b/OpenTelemetryPlan/09-data-collection-reference.md @@ -1401,28 +1401,45 @@ signal as it lands. `Type` is the instrument kind (counter / gauge / histogram / span / span attr), `Emit site` the owning source file, and `Panel` the dashboard panel that renders it. -| Signal | Type | Emit site | Panel | Meaning | -| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------- | --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `dns_resolve_total` (`outcome` = `resolved` \| `empty`) | counter | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Outcome Rate | Peer hostname resolutions. `empty` means a configured bootstrap or `[ips_fixed]` name returned no address, so that peer is never dialled. | -| `dns_resolve_latency_ms` | histogram | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Latency (p95) | Time to resolve a configured peer hostname. Seconds-scale values mean the resolver is timing out ahead of every dial. | -| `overlay_connect_total` (`outcome` = `connected` \| `tcp_fail` \| `tls_fail` \| `upgrade_fail` \| `timeout`) | counter | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Outcome Rate | Outbound peer connection attempts by terminal outcome. The outcome names the stage that broke: TCP, TLS, HTTP upgrade, or no terminal state in time. | -| `overlay_dial_latency_ms` | histogram | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Latency (p95) | Time from starting an outbound dial to its terminal outcome, successes and failures together. A p95 near the dial timeout means peers accept TCP but never finish the handshake. | -| `handshake_negotiation_fail_total` (`reason`, 14 values incl. `wrong_network`, `invalid_network_id`, `clock_skew`, `self_connection`, `session_verify_failed`) | counter | `Handshake.cpp` — `throwNegotiationFailure` (from `verifyHandshake`) | Handshake Negotiation Failures by Reason | Peer handshakes rejected after TLS while checking network id, clock, keys and addresses. `reason` names the failing check. | -| `unl_fetch_total` (`site` = configured UNL URI; `outcome` = the 9 `ListDisposition` strings `accepted` \| `expired` \| `same_sequence` \| `pending` \| `known_sequence` \| `unsupported_version` \| `untrusted` \| `stale` \| `invalid`, plus `fetch_error` \| `bad_status` \| `parse_error`) | counter | `ValidatorSite.cpp` — `ValidatorSite::reportFetchOutcome` | UNL Fetch Rate by Site & Outcome | Validator-list fetches per site. `accepted` is the only success; `same_sequence` and `known_sequence` are normal no-op refreshes; the three literals are transport or content faults. | -| `unl_quorum` (`metric` = `trusted_keys` \| `quorum`) | observable gauge | `MetricsRegistry.cpp` — `registerUnlQuorumGauge` | UNL Trusted Keys vs Quorum; UNL Quorum Headroom | Trusted UNL key count against the validations a ledger needs. `trusted_keys` at or below `quorum` means the node can never declare a ledger validated. | -| `clock_close_offset_seconds` (`metric` = `offset`) | observable gauge | `MetricsRegistry.cpp` — `registerClockSkewGauge` | Clock Close Offset | Network close time offset from the local clock. Negative means the local clock runs ahead. `server_info` only surfaces `close_time_offset` at 60 s or more, so this gauge sees skew far earlier. | -| `state_changes_total` (`from`, `to` = `disconnected` \| `connected` \| `syncing` \| `tracking` \| `full`) | counter | `NetworkOPs.cpp` — `NetworkOPsImp::setMode` | Mode Transitions by Edge | Operating-mode transitions keyed on the (`from`, `to`) edge. The edge is what separates a clean `disconnected`→`connected`→`syncing`→`tracking`→`full` climb from `full`→`connected` flapping; an unlabelled total cannot tell them apart. | -| `sync_state` (`metric` = `initial_full_duration_us`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Time to First FULL | Microseconds from process start to the first `full` transition, sourced from `NetworkOPs::getInitialSyncDurationUs()`. Stays 0 until `full` is reached, so a flat 0 is itself the "never synced" signal; once set it never changes. | -| `sync_state` (`metric` = `network_ledger_gate`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Network Ledger Gate | 1 while the node is still waiting to see a full network ledger (`NetworkOPs::isNeedNetworkLedger()`), else 0. A persistent 1 blocks transaction submission and `full`, whatever the rest of the pipeline shows. | -| `sync_state` (`metric` = `server_stall_seconds`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Server Stall | Current main-loop stall duration from `LoadManager::getCurrentStallSeconds()`, 0 when healthy. Same duration the load monitor logs as "Server stalled for N seconds", which previously existed only in that log line. | -| `sync_state` (`metric` = `ledgers_behind`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Ledgers Behind Network | Peer-reported network tip minus our validated sequence, floored at 0 (`NetworkOPs::getLedgersBehindNetwork()`). Reads each peer's already cached ledger range, so no new network round trip. | -| `server_stall_events_total` | observable counter | `MetricsRegistry.cpp` — `registerStallEventsCounter` | Server Stall Event Rate | Distinct stall episodes since process start, counted once per episode rather than per stalled second. A rising rate is repeated fresh stalls; a flat rate with a large `server_stall_seconds` is one long stall. | -| `sync_acquire` (`metric` = `missing_state_nodes_max` \| `missing_tx_nodes_max`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Missing SHAMap Nodes per Acquire (state/tx) | Largest outstanding SHAMap node count across in-flight acquires, split by tree, from the count `getMissingNodes()` already produces during its sweep (`InboundLedger.cpp` — `InboundLedger::trigger`). **The headline stuck-sync signal:** flat and non-zero across ticks means the acquire will never finish; shrinking means slow but alive. Aggregated as a max rather than labelled per ledger, because a `ledger_seq` label would mint one series per ledger acquired — per-ledger identity stays on the `ledger.acquire` span. | -| `sync_acquire` (`metric` = `received_data_depth`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Peer packets stashed across all in-flight acquires waiting to be applied, summed because it measures one shared processing backlog. A growing depth means arriving node data outpaces processing, so the limit is the job queue or disk rather than peer supply. | -| `sync_acquire` (`metric` = `in_flight`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Number of ledger acquires currently running. Exported so the three values above can be read in context: all zero with `in_flight` zero is an idle node, not a healthy one. | -| `shamap_cache_hit_rate` (`metric` = `treenode`) | observable gauge | `MetricsRegistry.cpp` — `registerCacheHitRateDetailGauge` | SHAMap TreeNode Cache Hit Rate | Share of SHAMap tree-node lookups served from memory, from the previously-uncalled `TaggedCache::getHitRate()`, normalized from 0-100 to 0.0-1.0. Distinct from `nodestore_state`-derived NuDB Cache Hit Ratio on the Ledger Data Sync dashboard: this is the in-memory layer **above** the node store, so a miss here is what causes a read there. The full-below cache is not reported — it is a `KeyCache` whose only lookup path increments `stats_.hits`/`stats_.misses` while `getHitRate()` reads the separate `hits_`/`misses_` members, so its rate is hard-wired to 0 until that accounting is fixed. | -| `sync_acquire_no_progress_total` | counter | `InboundLedger.cpp` — `InboundLedger::onTimer` | Acquire Stall Rate (no progress) | Acquire timeouts where not one new node arrived since the previous timeout, from the `progress_` flag that was previously log-only. Fires on the 3 s acquire timer, never per node. A sustained rate together with a flat missing-node count is the definitive "stuck, not slow" signature. | -| `sync_addnode_total` (`outcome` = `good` \| `duplicate` \| `invalid`) | counter | `InboundLedger.cpp` — `InboundLedger::recordBatchOutcome` | Add-Node Outcomes | SHAMap nodes received during acquire, split by result. Emitted once per received packet from the aggregated batch tally the trace log already printed — never inside the per-node `receiveNode()` loop. Separates real progress (`good`) from wasted bandwidth (`duplicate`) and a misbehaving peer (`invalid`), all three of which look like healthy throughput in traffic metrics. | -| `sync_acquire_source_total` (`source` = `local` \| `network`) | counter | `InboundLedger.cpp` — `InboundLedger::init` | Acquire Source (local vs network) | Whether an acquire was satisfied entirely from the local node store or needed peers, emitted once per new acquire after the first local lookup. Sustained `network` on a node that should already hold the range means sync is disk-bound rather than peer-bound. | -| `jobq_backlog` (`metric` = `waiting` \| `running` \| `deferred`; `job_type` = the `JobTypes::name()` string) | observable gauge | `MetricsRegistry.cpp` — `registerJobQueueBacklogGauge` | Deferred Jobs by Type (starvation); Job Queue Occupancy by Type (waiting/running) | Instantaneous per-job-type queue occupancy, from `JobQueue::getJobTypeCounts()` (one mutex acquire per ~10 s tick). **`deferred` is the signal this adds:** jobs the queue accepted but withheld because the type is already at its concurrency limit, which is counted in neither `waiting` nor `running` and had no exposure anywhere before. The sync-critical types are capped at 3 (`JtLedgerReq`, `JtLedgerData` in `JobTypes.h`), so they starve first. Distinct from the existing `job_queued_total` / `job_started_total` / `job_finished_total` counters and `job_queued_us` / `job_running_us` histograms, which are event-driven from PerfLogImp and describe jobs that already moved, and from the StatsD `jobq_job_count`, which is queue-wide with no per-type split. Cardinality is bounded by the JobType enum (~46 values); every type is observed every tick, so an idle type reports 0 rather than dropping its series. | -| `jobq_saturation` (`metric` = `running_tasks` \| `worker_threads` \| `total_waiting`) | observable gauge | `MetricsRegistry.cpp` — `registerJobQueueSaturationGauge` | Worker Pool Saturation; Worker Pool Capacity & Total Backlog | Global worker-pool saturation from `JobQueue::getWorkerSaturation()`: tasks in flight, threads the pool is configured to run, and jobs queued across all types, all from one reading so the ratio and the backlog describe the same instant. `worker_threads` is exported rather than hardcoded in the dashboard because it is derived at startup from `[workers]`, node size and hardware concurrency. Exists separately from `jobq_backlog` because a pool-wide slowdown otherwise appears as an independent fault in every subsystem queued behind it; a `running_tasks / worker_threads` ratio at 1.0 **with** a non-zero `total_waiting` attributes it to pool exhaustion once. | + + + +| Signal | Type | Emit site | Panel | Meaning | +| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `dns_resolve_total` (`outcome` = `resolved` \| `empty`) | counter | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Outcome Rate | Peer hostname resolutions. `empty` means a configured bootstrap or `[ips_fixed]` name returned no address, so that peer is never dialled. | +| `dns_resolve_latency_ms` | histogram | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Latency (p95) | Time to resolve a configured peer hostname. Seconds-scale values mean the resolver is timing out ahead of every dial. | +| `overlay_connect_total` (`outcome` = `connected` \| `tcp_fail` \| `tls_fail` \| `upgrade_fail` \| `timeout`) | counter | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Outcome Rate | Outbound peer connection attempts by terminal outcome. The outcome names the stage that broke: TCP, TLS, HTTP upgrade, or no terminal state in time. | +| `overlay_dial_latency_ms` | histogram | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Latency (p95) | Time from starting an outbound dial to its terminal outcome, successes and failures together. A p95 near the dial timeout means peers accept TCP but never finish the handshake. | +| `handshake_negotiation_fail_total` (`reason`, 14 values incl. `wrong_network`, `invalid_network_id`, `clock_skew`, `self_connection`, `session_verify_failed`) | counter | `Handshake.cpp` — `throwNegotiationFailure` (from `verifyHandshake`) | Handshake Negotiation Failures by Reason | Peer handshakes rejected after TLS while checking network id, clock, keys and addresses. `reason` names the failing check. | +| `unl_fetch_total` (`site` = configured UNL URI; `outcome` = the 9 `ListDisposition` strings `accepted` \| `expired` \| `same_sequence` \| `pending` \| `known_sequence` \| `unsupported_version` \| `untrusted` \| `stale` \| `invalid`, plus `fetch_error` \| `bad_status` \| `parse_error`) | counter | `ValidatorSite.cpp` — `ValidatorSite::reportFetchOutcome` | UNL Fetch Rate by Site & Outcome | Validator-list fetches per site. `accepted` is the only success; `same_sequence` and `known_sequence` are normal no-op refreshes; the three literals are transport or content faults. | +| `unl_quorum` (`metric` = `trusted_keys` \| `quorum`) | observable gauge | `MetricsRegistry.cpp` — `registerUnlQuorumGauge` | UNL Trusted Keys vs Quorum; UNL Quorum Headroom | Trusted UNL key count against the validations a ledger needs. `trusted_keys` at or below `quorum` means the node can never declare a ledger validated. | +| `clock_close_offset_seconds` (`metric` = `offset`) | observable gauge | `MetricsRegistry.cpp` — `registerClockSkewGauge` | Clock Close Offset | Network close time offset from the local clock. Negative means the local clock runs ahead. `server_info` only surfaces `close_time_offset` at 60 s or more, so this gauge sees skew far earlier. | +| `state_changes_total` (`from`, `to` = `disconnected` \| `connected` \| `syncing` \| `tracking` \| `full`) | counter | `NetworkOPs.cpp` — `NetworkOPsImp::setMode` | Mode Transitions by Edge | Operating-mode transitions keyed on the (`from`, `to`) edge. The edge is what separates a clean `disconnected`→`connected`→`syncing`→`tracking`→`full` climb from `full`→`connected` flapping; an unlabelled total cannot tell them apart. | +| `sync_state` (`metric` = `initial_full_duration_us`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Time to First FULL | Microseconds from process start to the first `full` transition, sourced from `NetworkOPs::getInitialSyncDurationUs()`. Stays 0 until `full` is reached, so a flat 0 is itself the "never synced" signal; once set it never changes. | +| `sync_state` (`metric` = `network_ledger_gate`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Network Ledger Gate | 1 while the node is still waiting to see a full network ledger (`NetworkOPs::isNeedNetworkLedger()`), else 0. A persistent 1 blocks transaction submission and `full`, whatever the rest of the pipeline shows. | +| `sync_state` (`metric` = `server_stall_seconds`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Server Stall | Current main-loop stall duration from `LoadManager::getCurrentStallSeconds()`, 0 when healthy. Same duration the load monitor logs as "Server stalled for N seconds", which previously existed only in that log line. | +| `sync_state` (`metric` = `ledgers_behind`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Ledgers Behind Network | Peer-reported network tip minus our validated sequence, floored at 0 (`NetworkOPs::getLedgersBehindNetwork()`). Reads each peer's already cached ledger range, so no new network round trip. | +| `server_stall_events_total` | observable counter | `MetricsRegistry.cpp` — `registerStallEventsCounter` | Server Stall Event Rate | Distinct stall episodes since process start, counted once per episode rather than per stalled second. A rising rate is repeated fresh stalls; a flat rate with a large `server_stall_seconds` is one long stall. | +| `sync_acquire` (`metric` = `missing_state_nodes_max` \| `missing_tx_nodes_max`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Missing SHAMap Nodes per Acquire (state/tx) | Largest outstanding SHAMap node count across in-flight acquires, split by tree, from the count `getMissingNodes()` already produces during its sweep (`InboundLedger.cpp` — `InboundLedger::trigger`). **The headline stuck-sync signal:** flat and non-zero across ticks means the acquire will never finish; shrinking means slow but alive. Aggregated as a max rather than labelled per ledger, because a `ledger_seq` label would mint one series per ledger acquired — per-ledger identity stays on the `ledger.acquire` span. | +| `sync_acquire` (`metric` = `received_data_depth`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Peer packets stashed across all in-flight acquires waiting to be applied, summed because it measures one shared processing backlog. A growing depth means arriving node data outpaces processing, so the limit is the job queue or disk rather than peer supply. | +| `sync_acquire` (`metric` = `in_flight`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Number of ledger acquires currently running. Exported so the three values above can be read in context: all zero with `in_flight` zero is an idle node, not a healthy one. | +| `shamap_cache_hit_rate` (`metric` = `treenode`) | observable gauge | `MetricsRegistry.cpp` — `registerCacheHitRateDetailGauge` | SHAMap TreeNode Cache Hit Rate | Share of SHAMap tree-node lookups served from memory, from the previously-uncalled `TaggedCache::getHitRate()`, normalized from 0-100 to 0.0-1.0. Distinct from `nodestore_state`-derived NuDB Cache Hit Ratio on the Ledger Data Sync dashboard: this is the in-memory layer **above** the node store, so a miss here is what causes a read there. The full-below cache is not reported — it is a `KeyCache` whose only lookup path increments `stats_.hits`/`stats_.misses` while `getHitRate()` reads the separate `hits_`/`misses_` members, so its rate is hard-wired to 0 until that accounting is fixed. | +| `sync_acquire_no_progress_total` | counter | `InboundLedger.cpp` — `InboundLedger::onTimer` | Acquire Stall Rate (no progress) | Acquire timeouts where not one new node arrived since the previous timeout, from the `progress_` flag that was previously log-only. Fires on the 3 s acquire timer, never per node. A sustained rate together with a flat missing-node count is the definitive "stuck, not slow" signature. | +| `sync_addnode_total` (`outcome` = `good` \| `duplicate` \| `invalid`) | counter | `InboundLedger.cpp` — `InboundLedger::recordBatchOutcome` | Add-Node Outcomes | SHAMap nodes received during acquire, split by result. Emitted once per received packet from the aggregated batch tally the trace log already printed — never inside the per-node `receiveNode()` loop. Separates real progress (`good`) from wasted bandwidth (`duplicate`) and a misbehaving peer (`invalid`), all three of which look like healthy throughput in traffic metrics. | +| `sync_acquire_source_total` (`source` = `local` \| `network`) | counter | `InboundLedger.cpp` — `InboundLedger::init` | Acquire Source (local vs network) | Whether an acquire was satisfied entirely from the local node store or needed peers, emitted once per new acquire after the first local lookup. Sustained `network` on a node that should already hold the range means sync is disk-bound rather than peer-bound. | +| `jobq_backlog` (`metric` = `waiting` \| `running` \| `deferred`; `job_type` = the `JobTypes::name()` string) | observable gauge | `MetricsRegistry.cpp` — `registerJobQueueBacklogGauge` | Deferred Jobs by Type (starvation); Job Queue Occupancy by Type (waiting/running) | Instantaneous per-job-type queue occupancy, from `JobQueue::getJobTypeCounts()` (one mutex acquire per ~10 s tick). **`deferred` is the signal this adds:** jobs the queue accepted but withheld because the type is already at its concurrency limit, which is counted in neither `waiting` nor `running` and had no exposure anywhere before. The sync-critical types are capped at 3 (`JtLedgerReq`, `JtLedgerData` in `JobTypes.h`), so they starve first. Distinct from the existing `job_queued_total` / `job_started_total` / `job_finished_total` counters and `job_queued_us` / `job_running_us` histograms, which are event-driven from PerfLogImp and describe jobs that already moved, and from the StatsD `jobq_job_count`, which is queue-wide with no per-type split. Cardinality is bounded by the JobType enum (~46 values); every type is observed every tick, so an idle type reports 0 rather than dropping its series. | +| `jobq_saturation` (`metric` = `running_tasks` \| `worker_threads` \| `total_waiting`) | observable gauge | `MetricsRegistry.cpp` — `registerJobQueueSaturationGauge` | Worker Pool Saturation; Worker Pool Capacity & Total Backlog | Global worker-pool saturation from `JobQueue::getWorkerSaturation()`: tasks in flight, threads the pool is configured to run, and jobs queued across all types, all from one reading so the ratio and the backlog describe the same instant. `worker_threads` is exported rather than hardcoded in the dashboard because it is derived at startup from `[workers]`, node size and hardware concurrency. Exists separately from `jobq_backlog` because a pool-wide slowdown otherwise appears as an independent fault in every subsystem queued behind it; a `running_tasks / worker_threads` ratio at 1.0 **with** a non-zero `total_waiting` attributes it to pool exhaustion once. | +| `peer_ledger_supply` (`metric` = `peers_reporting` \| `peers_serving_validated` \| `peers_serving_next` \| `supply_min_seq` \| `supply_max_seq`) | observable gauge | `MetricsRegistry.cpp` — `registerPeerLedgerSupplyGauge` (aggregating `OverlayImpl::getPeerLedgerSupply`) | Peers Able to Serve Needed Sequence; Peer Ledger Supply Window | How much of the sequence range this node needs its connected peer set can actually serve, from one pass over the active peers reading the range each already advertised in `mtSTATUS_CHANGE`. **`peers_serving_next` is the signal this exists for:** zero there with a non-zero `peers_reporting` means no connected peer holds validated + 1, so the peer set must change and waiting cannot finish the sync. `peers_reporting` is the denominator that makes the rest readable — peers advertising `[0, 0]` have not reported yet and are excluded from every field, so they cannot make a healthy peer set appear to serve from genesis; when nothing has reported, both window fields read 0 meaning **unknown**, not genesis. `supply_min_seq` / `supply_max_seq` separate "asking for history nobody kept" from "asking for a tip nobody reached". Distinct from `server_info{metric="peers"}`, a bare connection count with no notion of what those peers hold; from `sync_state{metric="ledgers_behind"}`, which uses the same per-peer maxima but collapses them to a single distance-to-tip number that cannot say how many peers can serve that distance or whether the range has a hole; and from `peer_quality{metric="peers_insane_count"}`, which counts peers on a different chain and is therefore a correctness signal, not an availability one. | +| `peer_disconnect_total` (`reason` = `graceful` \| `shutdown` \| `stopping` \| `read_error` \| `write_error` \| `timer_error` \| `ping_timeout` \| `not_useful` \| `large_sendq` \| `charge_resources` \| `malformed_handshake` \| `shared_value` \| `unknown`; `direction` = `inbound` \| `outbound`) | counter | `PeerImp.cpp` — `PeerImp::close` | Peer Disconnects by Reason | Peer teardowns split by cause and by which side opened the connection. Emitted once per teardown at `close()`, the single funnel every disconnect path passes through, and `close()` already self-guards on the socket being open, so a repeated close cannot double-count and the total matches the existing unlabelled tally. `reason` is set by whichever site decided to disconnect, first writer wins, so a later generic reason never masks the real one; the value is always one of a fixed set of literals in `PeerImp.cpp`, never peer-supplied data, so cardinality is bounded by the code. The split is the whole point: it separates our-fault backpressure (`large_sendq`, `charge_resources`) from topology and network faults (`not_useful`, `ping_timeout`, `read_error`), and normal churn (`graceful`) from either. Distinct from the existing `server_info{metric="peer_disconnects_resources"}`, which counts only the resource-charge subset and carries no labels, and from the StatsD `overlay_peer_disconnects`, which is the unlabelled grand total in which every reason above collapses into one number. | +| `peer_accept_total` (`outcome` = `accepted` \| `local_endpoint_fail` \| `resource_limit` \| `no_slot` \| `not_peer_request` \| `protocol_mismatch` \| `bad_cookie` \| `slot_refused` \| `handshake_error`) | counter | `OverlayImpl.cpp` — `OverlayImpl::onHandoff` via `reportAcceptOutcome` | Inbound Peer Accept Outcomes | Terminal outcome of every inbound connection this node is offered, one emit per handoff. `accepted` is reported only after `run()`, so anything that threw on the way lands on `handshake_error` instead; the two early returns that are not peer attempts at all (a handled HTTP request, and a request that never asked to upgrade) are deliberately not counted. The `outcome` names the stage that refused: no local endpoint, the resource manager, PeerFinder having no slot or seeing a duplicate, a non-peer upgrade request, protocol version disagreement, a bad security cookie, or activation being refused. This is the **inbound twin** of the existing `overlay_connect_total{outcome}`, which covers outbound dials only; without it a node refusing every inbound connection is indistinguishable from one nobody dials, and reading the two together gives the full in/out split. | +| `peerfinder_slot_census` (`metric` = `out_active` \| `out_max` \| `in_active` \| `in_max` \| `connecting` \| `fixed_configured` \| `fixed_active` \| `bootcache` \| `livecache`) | observable gauge | `MetricsRegistry.cpp` — `registerSlotCensusGauge` (from `Logic::getSlotCensus`) | PeerFinder Slot Census; PeerFinder Address Caches & Fixed Peers | Slot occupancy against capacity, outbound dials in flight, configured-versus-connected fixed peers, and the depth of both address caches. All nine come from a single acquire of the PeerFinder lock, so they are mutually consistent, share one label set and can be compared against each other. That is what makes the three most common bootstrap failures visible: `connecting` non-zero while `out_active` stays below `out_max` (dials starting and never completing), `bootcache` and `livecache` both at 0 (nothing to dial at all), and `fixed_active` below `fixed_configured` (a peer named in the configuration is unreachable). `fixed_configured` is the count of peers named in the config, so the pair an operator reads is "how many did I ask for" against "how many do I have" — the same comparison `autoconnect()` makes. All nine values already existed inside PeerFinder; only two of them were exported, as the legacy beast::insight gauges `peer_finder_active_inbound_peers` and `peer_finder_active_outbound_peers`. Those two carry no capacity, attempt or cache term, are read at unrelated instants, and so cannot be joined with each other let alone with a capacity term — leaving all three failures above indistinguishable from a node that is simply not dialling. | +| `serve_refused_total` (`request` = `ledger` \| `txset` \| `object` \| `fetchpack`; `reason` = `sendq_full` \| `load_shed` \| `not_found` \| `no_map` \| `bad_type` \| `empty_reply`) | counter | `PeerImp.cpp` — `processLedgerRequest`, `onMessage(TMGetObjectByHash)`, `doFetchPack` | Ledger/Object Serve Refusals | Peer data requests this node declined to answer, split by what was asked for and why. This is the **supply side** of the sync exchange — what this node refuses to serve OTHERS — and nothing equivalent existed before, so a node shedding every ledger request looked identical to one being asked for nothing. `sendq_full` and `load_shed` are self-inflicted backpressure (the send queue at `Tuning::kDropSendQueue`, or the local fee track loaded, or too many pack jobs queued), while `not_found` is a genuine history gap and `no_map` / `bad_type` / `empty_reply` mean the request was answerable in principle but produced nothing to send. `fetchpack` is counted apart from `ledger` because a fetch pack is how a syncing peer catches up in bulk and its shed threshold is a different one. Emitted at most once per request — `empty_reply` is reported after the node loop, never inside it — and both labels are code literals, so cardinality is bounded at compile time. | +| `amendment_block` (`metric` = `warned` \| `seconds_to_block`) | observable gauge | `MetricsRegistry.cpp` — `registerAmendmentBlockGauge` | Amendment Block Countdown; Amendment Warned | `warned` is 1 once an unsupported amendment has reached majority (`NetworkOPs::isAmendmentWarned()`, previously only an admin-only `server_info` warning). **`seconds_to_block` is the leading indicator:** seconds until that amendment activates, from `AmendmentTable::firstUnsupportedExpected()` against the network close time. It reads `-1` when nothing is pending — a distinct healthy value rather than a missing series, matching the sentinel `validator_health{metric="unl_expiry_days"}` already uses — and is clamped at 0 rather than going negative, because past-due means the block is imminent, not overdue by some amount worth charting; the subtraction is done in `std::int64_t` so a past-due activation cannot wrap. Amendment-blocked is a terminal sync blocker: the node stops validating and never resumes without a software upgrade. The existing `validator_health{metric="amendment_blocked"}` reports that state after the fact, when nothing can be done about it; this gauge is the window before it, which is the only actionable part. The blocking amendment's identity is deliberately **not** a label — the network can vote on an arbitrary 256-bit amendment id, not drawn from this build's known features, so an id label would be unbounded cardinality and would mint a permanent new series per amendment. The id is available in logs from `AmendmentTableImpl::doValidatedLedger` ("Unsupported amendment \ reached majority at ..."), correlated to this series by node and time. | +| `ledger_jump_total` | counter | `NetworkOPs.cpp` — `NetworkOPsImp::switchLastClosedLedger` | Byzantine Ledger Jumps | Forced jumps of the last closed ledger onto a divergent chain: the node was told the network's LCL is not the one it built on and discarded its own chain tip to follow. Nothing equivalent existed — this was log-only ("JUMP last closed ledger to ..."), so a node repeatedly thrashing between chains left no time series to correlate against the rest of the sync pipeline. Any non-zero rate is abnormal by construction; repeated jumps are wrong-chain thrash, which points at the peer set and the configured network id rather than anywhere in the acquire pipeline. Deliberately unlabelled: the ledger hash and sequence would both be unbounded as label values, and the log line beside the emit already carries them. | +| `nodestore_latency` (`metric` = `write_mean_us` \| `read_mean_us` \| `write_count` \| `read_count`) | observable gauge | `MetricsRegistry.cpp` — `registerNodeStoreLatencyGauge` | NodeStore Write vs Read Latency (us/op); NodeStore Operation Rate | Mean microseconds per node-store store and per fetch, with both operation counts so a panel can divide the two rates and read _interval_ latency instead of the since-boot average. **The write side is the new signal.** `storeDurationUs_` was declared in `Database.h` and never written, and no accessor existed, so no write-path latency was observable anywhere; the read total was already exposed as `nodestore_state{metric="node_reads_duration_us"}`. This is the fingerprint of the "a node with a large existing DB syncs slower than a fresh one" symptom, which is write-bound and therefore invisible in every read-side metric. Chosen as a gauge over a histogram deliberately: a histogram gives true percentiles but costs one `Record()` per node object on the store/fetch path, and a single ledger write walks thousands of SHAMap nodes — this gauge instead reads four existing atomics once per ~10 s tick and adds nothing to the hot path. Consequence: **p99 is not obtainable from this signal**, and a histogram added later would also need an explicit-bucket View (`addMicrosecondHistogramView`) because the SDK default buckets top out at 10,000. Distinct from the Ledger Data Sync dashboard's NuDB Read Latency panel, which divides two `nodestore_state` fields in PromQL: that panel has no write-duration input to divide, because the quantity did not exist. **Known gap:** `write_mean_us` is emitted only when the store-duration total is non-zero, and that total is fed by `Database::recordStoreDuration`, today called only from `Database::importInternal` (the `[import_db]` admin path). `Database::store()` is pure virtual and neither `DatabaseNodeImp::store` nor `DatabaseRotatingImp::store` times itself yet, so an ordinary node reports `write_count` with no `write_mean_us`. The mean is omitted rather than reported as 0 so the gap stays visible instead of reading as "writes are instantaneous". | +| `ledger_replay_fallback_total` (`stage` = `skiplist` \| `delta`) | counter | `SkipListAcquire.cpp` / `LedgerDeltaAcquire.cpp` — `trigger` | Replay Fallback to Full Acquire (by stage) | A ledger-replay sub-task abandoning its shortcut and acquiring the whole ledger through `InboundLedger` instead, because too few connected peers support the `LedgerReplay` protocol feature. Both branches were debug-log-only, so a silently defeated replay optimisation left no metric at all — back-fill simply ran on the slower path with nothing to show why. Emitted once, on the transition into fallback, not at the acquire call, which re-runs on every later trigger. The `stage` label separates the skip-list acquire (which fetches the list of historical ledger hashes) from the per-ledger delta acquire, because they fail independently. | +| `ledger_replay_outcome_total` (`outcome` = `success` \| `timeout` \| `build_failed` \| `parameter_failed`) | counter | `LedgerReplayTask.cpp` — `LedgerReplayTask::recordOutcome` | Replay Outcomes (by terminal state) | Terminal state of every ledger-replay task, one emit per task. Every terminal path previously only set an internal `complete_`/`failed_` flag and wrote a log line, so a replay that never succeeded was indistinguishable from one that was never attempted. The outcome names the layer at fault: `timeout` means the deltas never arrived (a peer-supply problem), `build_failed` means a delta would not apply to its parent, and `parameter_failed` means a peer served a skip list inconsistent with what the task asked for — the latter two are data faults, not slowness. Read with `ledger_replay_fallback_total`: fallbacks rising while successes stay flat is replay-based catch-up degrading to full-ledger acquisition. | +| `nodestore_latency` (`metric` = `write_mean_us` \| `read_mean_us` \| `write_count` \| `read_count`) | observable gauge | `MetricsRegistry.cpp` — `registerNodeStoreLatencyGauge` | NodeStore Write vs Read Latency (us/op); NodeStore Operation Rate | Mean microseconds per node-store store and per fetch, with both operation counts so a panel can divide the two rates and read _interval_ latency instead of the since-boot average. **The write side is the new signal.** `storeDurationUs_` was declared in `Database.h` and never written, and no accessor existed, so no write-path latency was observable anywhere; the read total was already exposed as `nodestore_state{metric="node_reads_duration_us"}`. This is the fingerprint of the "a node with a large existing DB syncs slower than a fresh one" symptom, which is write-bound and therefore invisible in every read-side metric. Chosen as a gauge over a histogram deliberately: a histogram gives true percentiles but costs one `Record()` per node object on the store/fetch path, and a single ledger write walks thousands of SHAMap nodes — this gauge instead reads four existing atomics once per ~10 s tick and adds nothing to the hot path. Consequence: **p99 is not obtainable from this signal**, and a histogram added later would also need an explicit-bucket View (`addMicrosecondHistogramView`) because the SDK default buckets top out at 10,000. Distinct from the Ledger Data Sync dashboard's NuDB Read Latency panel, which divides two `nodestore_state` fields in PromQL: that panel has no write-duration input to divide, because the quantity did not exist. **Known gap:** `write_mean_us` is emitted only when the store-duration total is non-zero, and that total is fed by `Database::recordStoreDuration`, today called only from `Database::importInternal` (the `[import_db]` admin path). `Database::store()` is pure virtual and neither `DatabaseNodeImp::store` nor `DatabaseRotatingImp::store` times itself yet, so an ordinary node reports `write_count` with no `write_mean_us`. The mean is omitted rather than reported as 0 so the gap stays visible instead of reading as "writes are instantaneous". | +| `ledger_replay_fallback_total` (`stage` = `skiplist` \| `delta`) | counter | `SkipListAcquire.cpp` / `LedgerDeltaAcquire.cpp` — `trigger` | Replay Fallback to Full Acquire (by stage) | A ledger-replay sub-task abandoning its shortcut and acquiring the whole ledger through `InboundLedger` instead, because too few connected peers support the `LedgerReplay` protocol feature. Both branches were debug-log-only, so a silently defeated replay optimisation left no metric at all — back-fill simply ran on the slower path with nothing to show why. Emitted once, on the transition into fallback, not at the acquire call, which re-runs on every later trigger. The `stage` label separates the skip-list acquire (which fetches the list of historical ledger hashes) from the per-ledger delta acquire, because they fail independently. | +| `ledger_replay_outcome_total` (`outcome` = `success` \| `timeout` \| `build_failed` \| `parameter_failed`) | counter | `LedgerReplayTask.cpp` — `LedgerReplayTask::recordOutcome` | Replay Outcomes (by terminal state) | Terminal state of every ledger-replay task, one emit per task. Every terminal path previously only set an internal `complete_`/`failed_` flag and wrote a log line, so a replay that never succeeded was indistinguishable from one that was never attempted. The outcome names the layer at fault: `timeout` means the deltas never arrived (a peer-supply problem), `build_failed` means a delta would not apply to its parent, and `parameter_failed` means a peer served a skip list inconsistent with what the task asked for — the latter two are data faults, not slowness. Read with `ledger_replay_fallback_total`: fallbacks rising while successes stay flat is replay-based catch-up degrading to full-ledger acquisition. | diff --git a/docker/telemetry/grafana/dashboards/ledger-sync-health.json b/docker/telemetry/grafana/dashboards/ledger-sync-health.json index 16d023db5e..b5c28881b4 100644 --- a/docker/telemetry/grafana/dashboards/ledger-sync-health.json +++ b/docker/telemetry/grafana/dashboards/ledger-sync-health.json @@ -2427,6 +2427,1388 @@ ], "title": "Worker Pool Capacity & Total Backlog", "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*How many connected peers can actually serve the ledger this node needs next, next to how many advertise a ledger range at all.*\n\n###### How it's computed:\n*peer_ledger_supply series peers_reporting, peers_serving_validated and peers_serving_next, counted from the ledger ranges connected peers advertise and compared against this node's own validated sequence.*\n\n###### Reading it:\n*peers_serving_next is the line that matters: it counts peers holding validated+1, the one ledger this node must acquire before it can advance. peers_serving_validated counts peers that can re-serve the ledger already held, and peers_reporting is the denominator — peers that advertised any range at all.*\n\n###### Healthy range:\n*peers_serving_next at or near peers_reporting.*\n\n###### Watch for:\n*peers_serving_next at 0 while peers_reporting is above 0. No peer this node is connected to holds the ledger it needs, so this is a supply problem, not a slow-peer problem — retrying harder or waiting longer cannot fix it, the node needs different peers. Distinct from the acquire panels above, which measure requests already sent to peers that did hold the data.*\n\n###### Keywords:\n- **Peer ledger supply** *(per node)* — the count of connected peers whose advertised ledger range covers a sequence this node needs.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerPeerLedgerSupplyGauge`\n\n###### References:\n[Complete ledger ranges](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#complete-ledger-ranges) · [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#fresh-node-sync-diagnostics)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Peers", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 158 + }, + "id": 28, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(peer_ledger_supply{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=~\"peers_reporting|peers_serving_validated|peers_serving_next\"}, \"series\", \"$1\", \"metric\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Peers Able to Serve Needed Sequence", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*The lowest and highest ledger sequence any connected peer offers — the sequence window the peer set can actually serve from.*\n\n###### How it's computed:\n*peer_ledger_supply series supply_min_seq and supply_max_seq: the minimum and maximum over the ledger ranges advertised by all connected peers.*\n\n###### Reading it:\n*supply_max_seq is the network tip as this node's peers report it. supply_min_seq is the oldest history reachable from the current peer set; anything below it cannot be back-filled from these peers at all.*\n\n###### Healthy range:\n*supply_max_seq advancing with the network, supply_min_seq well below this node's own validated sequence.*\n\n###### Watch for:\n*supply_min_seq sitting above this node's validated sequence — every connected peer has already pruned the history this node still needs, so the gap can never be closed from this peer set. Pair with Peers Able to Serve Needed Sequence: that panel says how many peers can serve the next ledger, this one says which sequences are on offer at all.*\n\n###### Keywords:\n- **Peer ledger supply window** *(per node)* — the lowest and highest ledger sequence offered by any connected peer; the span the peer set can serve.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerPeerLedgerSupplyGauge`\n\n###### References:\n[Complete ledger ranges](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#complete-ledger-ranges) · [Back-fill / catch-up](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#back-fill-catch-up)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Ledger Sequence", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 158 + }, + "id": 29, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(peer_ledger_supply{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=~\"supply_min_seq|supply_max_seq\"}, \"series\", \"$1\", \"metric\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Peer Ledger Supply Window", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*PeerFinder's slot accounting in one snapshot: outbound and inbound slots in use against their configured maxima, plus outbound attempts still in flight.*\n\n###### How it's computed:\n*peerfinder_slot_census series out_active, out_max, in_active, in_max and connecting, all read from ONE PeerFinder snapshot taken under a single lock so the five values describe the same instant.*\n\n###### Reading it:\n*out_active should climb to out_max and stay there. connecting counts outbound attempts started but not yet resolved either way, so it is the in-flight term the active counts cannot show.*\n\n###### Healthy range:\n*out_active at out_max; connecting low and transient.*\n\n###### Watch for:\n*out_active pinned below out_max while connecting stays non-zero — dials are being started and never completing, so the node is trying and failing rather than sitting idle. Note out_active and in_active are also exported separately as the legacy beast::insight gauges peer_finder_active_outbound_peers and peer_finder_active_inbound_peers. Those two are unrelated single series read at different instants with no capacity, attempt or cache terms, so no reading of them can distinguish this case. The census exists to report all nine fields from one snapshot under one lock, so they are mutually consistent and joinable on one labelset.*\n\n###### Keywords:\n- **PeerFinder slot census** *(per node)* — one consistent snapshot of PeerFinder's outbound and inbound slot use, capacities, in-flight attempts, fixed peers and address caches.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerSlotCensusGauge`\n\n###### References:\n[Overlay](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#overlay) · [Outbound dial latency](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#outbound-dial-latency)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Slots", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 170 + }, + "id": 30, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(peerfinder_slot_census{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=~\"out_active|out_max|in_active|in_max|connecting\"}, \"series\", \"$1\", \"metric\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "PeerFinder Slot Census", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*The address supply PeerFinder dials from — its boot cache and live cache — next to configured versus currently connected fixed peers.*\n\n###### How it's computed:\n*peerfinder_slot_census series bootcache, livecache, fixed_configured and fixed_active, read from the same single snapshot as the slot census.*\n\n###### Reading it:\n*bootcache holds seed addresses persisted across restarts; livecache holds addresses learned from peers while running. fixed_active is how many of the fixed peers named in the config are connected right now.*\n\n###### Healthy range:\n*bootcache and livecache non-zero; fixed_active equal to fixed_configured.*\n\n###### Watch for:\n*bootcache at 0 on a fresh node means there are no seed addresses to dial at all, so no outbound connection is ever attempted and every downstream sync signal on this dashboard stays empty for a reason that has nothing to do with sync. fixed_active below fixed_configured means a configured fixed peer is unreachable.*\n\n###### Keywords:\n- **PeerFinder address cache** *(per node)* — the boot cache (seed addresses persisted across restarts) and live cache (addresses learned from peers) that supply outbound dial candidates.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerSlotCensusGauge`\n\n###### References:\n[Overlay](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#overlay) · [DNS resolve](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#dns-resolve)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Count", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 170 + }, + "id": 31, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(peerfinder_slot_census{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=~\"bootcache|livecache|fixed_configured|fixed_active\"}, \"series\", \"$1\", \"metric\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "PeerFinder Address Caches & Fixed Peers", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Rate of inbound peer connection handoffs, split by terminal outcome.*\n\n###### How it's computed:\n*Per-second rate of finished inbound handoffs grouped by outcome, per node. Filter the outcome set with the Accept Outcome variable.*\n\n###### Reading it:\n*accepted should dominate. The failure outcomes name what rejected the connection: resource_limit and no_slot are this node's own capacity, protocol_mismatch and bad_cookie are the peer or the network identity, handshake_error and local_endpoint_fail are the transport.*\n\n###### Healthy range:\n*accepted dominant, failures near zero.*\n\n###### Watch for:\n*Read this together with Outbound Dial Outcome Rate above, which is the outbound twin on overlay_connect_total{outcome}. The two share the same one-outcome-per-attempt shape, so together they give the full in/out split: a node that accepts nothing but dials successfully has a very different fault from one that can neither dial nor accept. A steady no_slot or resource_limit is this node refusing peers it has no room for, which is capacity rather than a fault.*\n\n###### Keywords:\n- **Inbound peer accept outcome** *(per node)* — the terminal result of an inbound peer connection handoff; each handoff ends in exactly one outcome.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl::onHandoff`\n\n###### References:\n[Overlay](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#overlay) · [Outbound dial latency](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#outbound-dial-latency)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Handoffs / Sec", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 182 + }, + "id": 32, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sum by (outcome, service_instance_id, xrpl_branch, xrpl_work_item) (rate(peer_accept_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", outcome=~\"$accept_outcome\"}[$__rate_interval])), \"series\", \"$1\", \"outcome\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Inbound Peer Accept Outcomes", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Rate at which peer connections are closed, split by why they closed and by which side opened them.*\n\n###### How it's computed:\n*Per-second rate of peer closes grouped by reason and direction (inbound or outbound), per node. Filter with the Disconnect Reason and Disconnect Direction variables.*\n\n###### Reading it:\n*The split separates our-fault backpressure from topology and network faults. large_sendq and charge_resources are this node shedding a peer it cannot keep up with. not_useful and ping_timeout are topology and liveness. read_error and write_error are the transport. graceful, stopping and shutdown are ordinary lifecycle, not faults.*\n\n###### Healthy range:\n*Mostly graceful; the fault reasons near zero.*\n\n###### Watch for:\n*A sustained large_sendq or charge_resources means this node is the bottleneck and is dropping peers — which removes the very peers it needs to sync from, so a slow node makes itself slower. A rising not_useful or ping_timeout on outbound points at the peer set instead. Distinct from the single unlabelled peer-disconnect total on the Node Health dashboard, which cannot say which of these is happening.*\n\n###### Keywords:\n- **Peer disconnect reason** *(per node)* — the cause recorded when a peer connection is closed, paired with whether that connection was inbound or outbound.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[PeerImp.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/PeerImp.cpp)\n\n###### Function:\n`PeerImp::close`\n\n###### References:\n[Resource disconnect](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#resource-disconnect) · [Insane / diverged peers](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#insane-diverged-peers)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Disconnects / Sec", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 182 + }, + "id": 33, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_join(sum by (reason, direction, service_instance_id, xrpl_branch, xrpl_work_item) (rate(peer_disconnect_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", reason=~\"$disconnect_reason\", direction=~\"$disconnect_direction\"}[$__rate_interval])), \"series\", \" \", \"reason\", \"direction\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Peer Disconnects by Reason", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Peer requests that this node declined to serve, split by why it declined and by what was asked for.*\n\n###### How it's computed:\n*Per-second rate of refused serve attempts grouped by reason and request type (ledger, tx-set, object or fetchpack), per node. Filter with the Serve Reason and Serve Request variables, which list the emitted label values verbatim (the tx-set one is a single unhyphenated word).*\n\n###### Reading it:\n*This is the other side of the sync exchange: every other panel here measures what this node fetches, this one measures what it refuses to give back. sendq_full and load_shed are self-inflicted backpressure — the node holds the data but will not send it. not_found and no_map mean the requester asked for history this node does not hold.*\n\n###### Healthy range:\n*Near zero.*\n\n###### Watch for:\n*sendq_full or load_shed climbing. This node is starving its peers, and on a network of similarly loaded nodes that is exactly the condition that makes everybody's sync slow, so a refusal here can be the cause of another operator's stall. A high not_found is usually benign on a node configured with short history, but it tells peers to look elsewhere.*\n\n###### Keywords:\n- **Serve refusal** *(per node)* — a peer request for ledger, transaction-set, object or fetch-pack data that this node declined, together with the reason it declined.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[PeerImp.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/PeerImp.cpp)\n\n###### Function:\n`PeerImp::processLedgerRequest` · `PeerImp::onMessage(TMGetObjectByHash)` · `PeerImp::doFetchPack`\n\n###### References:\n[Fetch-pack](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#fetch-pack) · [GetObject / object fetch](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#getobject-object-fetch)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Refusals / Sec", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 194 + }, + "id": 34, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_join(sum by (reason, request, service_instance_id, xrpl_branch, xrpl_work_item) (rate(serve_refused_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", reason=~\"$serve_reason\", request=~\"$serve_request\"}[$__rate_interval])), \"series\", \" \", \"reason\", \"request\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Ledger/Object Serve Refusals", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Rate at which this node abandoned the chain tip it had built and switched to a different last-closed ledger reported by the network.*\n\n###### How it's computed:\n*Per-second rate of last-closed-ledger switches where the ledger the network reported was not the one this node built on, per node. The counter carries no labels, so the sum is over one series per node.*\n\n###### Reading it:\n*Any non-zero value means this node was told the network's last-closed ledger is not the one it built on, and it discarded its own chain tip in response. A single jump around startup or a restart is ordinary recovery.*\n\n###### Healthy range:\n*0.*\n\n###### Watch for:\n*Repeated jumps — that is wrong-chain thrash, not one-off recovery. Check the peer set and the configured network id: a node peered to the wrong network, or into a minority partition, keeps being overruled and keeps throwing away work. Pair with the Bootstrap row and with Ledgers Behind Network.*\n\n###### Keywords:\n- **Byzantine ledger jump** *(per node)* — the node replaced its own last-closed ledger with a different one reported by the network, discarding the chain tip it had built.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[NetworkOPs.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/misc/NetworkOPs.cpp)\n\n###### Function:\n`NetworkOPsImp::switchLastClosedLedger`\n\n###### References:\n[Fork](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#fork) · [Ledger history mismatch](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-history-mismatch)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Jumps / Sec", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "line" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 0.001 + } + ] + }, + "unit": "ops" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 194 + }, + "id": 35, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(ledger_jump_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])), \"series\", \"Ledger Jumps\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Byzantine Ledger Jumps", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Seconds until an unsupported amendment activates and this node stops validating for good. This is the LEADING indicator — the countdown before the block, while there is still time to upgrade.*\n\n###### How it's computed:\n*amendment_block series seconds_to_block. A value of -1 is the sentinel for \"no unsupported amendment is pending\", the same convention validator_health{metric=\"unl_expiry_days\"} already uses.*\n\n###### Reading it:\n*Because -1 is the healthy sentinel and a small positive number is the emergency, the colour scale is not monotonic — read the value, not only the colour. -1 is green and means nothing is pending. A large positive value is green above 7 days and yellow under 7 days: an unsupported amendment holds majority but there is still time to upgrade. A small positive value under 1 day is red: at 0 this node stops validating and does not resume without a software upgrade.*\n\n###### Healthy range:\n*-1.*\n\n###### Watch for:\n*Any value at or above 0. This is distinct from the Amendment Blocked stat on the Validator Health dashboard (validator_health{metric=\"amendment_blocked\"}), which reports the TERMINAL state — already blocked, too late to act. This panel is the window before that happens, so the two are read together: countdown first, terminal state as confirmation. The identity of the blocking amendment is deliberately NOT a metric label, because the network can vote on an arbitrary 256-bit amendment id and a label would be unbounded cardinality. The hash is logged instead by AmendmentTableImpl::doValidatedLedger (\"Unsupported amendment reached majority at ...\") and is available via Loki.*\n\n###### Keywords:\n- **Amendment block countdown** *(per node)* — seconds until an unsupported amendment that has reached majority activates and blocks this node; -1 when none is pending.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerAmendmentBlockGauge`\n\n###### References:\n[Amendments on xrpl.org](https://xrpl.org/docs/concepts/networks-and-servers/amendments) · [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#amendment-blocked)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 0 + }, + { + "color": "yellow", + "value": 86400 + }, + { + "color": "green", + "value": 604800 + } + ] + }, + "unit": "s" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 206 + }, + "id": 36, + "options": { + "colorMode": "value", + "graphMode": "none", + "justifyMode": "center", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "value_and_name", + "wideLayout": true + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(amendment_block{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"seconds_to_block\"}, \"series\", \"Seconds to Block\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Amendment Block Countdown", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Whether an amendment this build does not support has reached majority on the network.*\n\n###### How it's computed:\n*amendment_block series warned: 1 while an unsupported amendment holds majority, 0 otherwise.*\n\n###### Reading it:\n*0 is healthy. A 1 is the first warning that an upgrade is required, and it is raised before the amendment activates rather than after.*\n\n###### Healthy range:\n*0.*\n\n###### Watch for:\n*The transition from 0 to 1 — that is the moment the upgrade clock starts. Read Amendment Block Countdown next for how long is left, and the Amendment Blocked stat on the Validator Health dashboard for whether the block has already happened; that one is the terminal state, this one is the warning. Which amendment is blocking is not a label (an arbitrary 256-bit amendment id would be unbounded cardinality); the hash is logged by AmendmentTableImpl::doValidatedLedger and is available via Loki.*\n\n###### Keywords:\n- **Amendment warning** *(per node)* — an unsupported amendment has reached majority; the node still validates, but will stop when that amendment activates.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerAmendmentBlockGauge`\n\n###### References:\n[Amendments on xrpl.org](https://xrpl.org/docs/concepts/networks-and-servers/amendments) · [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#amendment-blocked)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Warned", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "line" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1 + } + ] + }, + "unit": "short" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 206 + }, + "id": 37, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(amendment_block{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"warned\"}, \"series\", \"Warned\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Amendment Warned", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Node-store write latency next to read latency, in microseconds per operation. The write side is the signal: a node with a large existing database back-fills slower than a fresh one, and back-fill is write-bound, so no read-side metric can show it.*\n\n###### How it's computed:\n*nodestore_latency series write_mean_us and read_mean_us, each divided by its own count series so the reading is the latency during the selected interval rather than the average since boot. The write numerator comes from a store-duration total that was declared but never written before this signal existed.*\n\n###### Reading it:\n*Compare the two lines. Reads far above writes points at the read path or a cold cache; writes far above reads points at backend write pressure, which is the large-existing-database case.*\n\n###### Healthy range:\n*Both well under a few hundred microseconds on healthy local storage.*\n\n###### Watch for:\n*A rising write line during history back-fill: the backend cannot absorb writes fast enough and sync will stay slow no matter how many peers are available. Read with Fetch-Pack Peer Starvation to tell a data-supply problem from a disk problem. This is a mean, not a percentile \u2014 a tail that matters will move it, but p99 is not available from this signal.*\n\n###### Keywords:\n- **Node-store write latency** *(per node)* \u2014 how long the node store takes to persist one object.\n- **Node-store read latency** *(per node)* \u2014 how long the node store takes to retrieve one object.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreLatencyGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#node-store-write-latency)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Microseconds / Op", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "\u00b5s" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 218 + }, + "id": 38, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"write_mean_us\"}[$__rate_interval])) / clamp_min(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"write_count\"}[$__rate_interval])), 1), \"series\", \"Write us/op (interval)\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"read_mean_us\"}[$__rate_interval])) / clamp_min(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"read_count\"}[$__rate_interval])), 1), \"series\", \"Read us/op (interval)\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "B" + } + ], + "title": "NodeStore Write vs Read Latency (us/op)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Node-store write and read operation rates \u2014 the denominators behind the latency panel.*\n\n###### How it's computed:\n*Rate of the nodestore_latency write_count and read_count series.*\n\n###### Reading it:\n*Writes climb while a node is back-filling history and fall to near the ledger-close rate once it is caught up.*\n\n###### Healthy range:\n*Non-zero writes whenever the node is ingesting ledgers.*\n\n###### Watch for:\n*Write rate at zero while the node is still behind the network: nothing is being persisted, so the stall is upstream of the node store \u2014 check peer supply and the acquire panels rather than storage. A flat latency with a collapsing operation rate also means the latency figure above has gone stale rather than good.*\n\n###### Keywords:\n- **Node-store operation rate** *(per node)* \u2014 stores and fetches per second against the node store.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreLatencyGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#node-store-operation-rate)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Ops / Sec", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 218 + }, + "id": 39, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"write_count\"}[$__rate_interval])), \"series\", \"Writes/s\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"read_count\"}[$__rate_interval])), \"series\", \"Reads/s\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "B" + } + ], + "title": "NodeStore Operation Rate (writes vs reads)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Rate at which ledger-replay sub-tasks give up and fall back to acquiring a whole ledger, split by which sub-task gave up.*\n\n###### How it's computed:\n*Rate of ledger_replay_fallback_total by stage. The skip-list stage and the delta stage each emit once, on the transition into fallback, from the branch that was debug-log-only before.*\n\n###### Reading it:\n*Zero when replay-capable peers are available. Any sustained rate means the replay optimisation is being defeated and back-fill has reverted to the slower full-acquire path.*\n\n###### Healthy range:\n*Zero, or brief spikes while the peer set changes.*\n\n###### Watch for:\n*A persistent rate on either stage: too few connected peers support the ledger-replay feature, so every historical ledger is fetched in full instead of as a delta. Read with Replay Outcomes.*\n\n###### Keywords:\n- **Replay fallback** *(per node)* \u2014 a replay sub-task abandoning the delta shortcut and acquiring the entire ledger instead.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[SkipListAcquire.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/SkipListAcquire.cpp)\n\n###### Function:\n`SkipListAcquire::trigger` / `LedgerDeltaAcquire::trigger`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#replay-fallback)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Fallbacks / Sec", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 230 + }, + "id": 40, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sum by (stage, service_instance_id, xrpl_branch, xrpl_work_item) (rate(ledger_replay_fallback_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", stage=~\"$replay_stage\"}[$__rate_interval])), \"series\", \"$1\", \"stage\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Replay Fallback to Full Acquire (by stage)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Rate of ledger-replay tasks reaching each terminal state: success, timeout, build failure or parameter failure.*\n\n###### How it's computed:\n*Rate of ledger_replay_outcome_total by outcome. Every terminal path in the replay task emits exactly once; before this, all four only set an internal flag and wrote a log line.*\n\n###### Reading it:\n*Successes only is healthy. Timeouts mean deltas never arrived; build failures mean a delta would not apply to its parent; parameter failures mean a peer served an inconsistent skip list.*\n\n###### Healthy range:\n*Successes non-zero while replaying, all failure outcomes at zero.*\n\n###### Watch for:\n*Any failure outcome climbing while successes stay flat: replay is running but never completing, so history back-fill is silently falling back to the slower path. The outcome value tells you which layer to look at \u2014 timeouts point at peers, build and parameter failures point at the data those peers served.*\n\n###### Keywords:\n- **Ledger replay** *(per node)* \u2014 rebuilding a range of historical ledgers from a start ledger plus per-ledger deltas instead of downloading each one whole.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[LedgerReplayTask.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/LedgerReplayTask.cpp)\n\n###### Function:\n`LedgerReplayTask::recordOutcome`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-replay)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Tasks / Sec", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 230 + }, + "id": 41, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sum by (outcome, service_instance_id, xrpl_branch, xrpl_work_item) (rate(ledger_replay_outcome_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", outcome=~\"$replay_outcome\"}[$__rate_interval])), \"series\", \"$1\", \"outcome\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Replay Outcomes (by terminal state)", + "type": "timeseries" } ], "schemaVersion": 39, @@ -2805,6 +4187,146 @@ "multi": true, "refresh": 2, "sort": 1 + }, + { + "name": "accept_outcome", + "label": "Accept Outcome", + "description": "Filter inbound peer handoffs by terminal outcome [accepted / no_slot / resource_limit / ...]", + "type": "query", + "query": "label_values(peer_accept_total, outcome)", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "includeAll": true, + "allValue": ".*", + "current": { + "text": "All", + "value": "$__all" + }, + "multi": true, + "refresh": 2, + "sort": 1 + }, + { + "name": "disconnect_reason", + "label": "Disconnect Reason", + "description": "Filter peer disconnects by cause [graceful / large_sendq / not_useful / ping_timeout / ...]", + "type": "query", + "query": "label_values(peer_disconnect_total, reason)", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "includeAll": true, + "allValue": ".*", + "current": { + "text": "All", + "value": "$__all" + }, + "multi": true, + "refresh": 2, + "sort": 1 + }, + { + "name": "disconnect_direction", + "label": "Disconnect Direction", + "description": "Filter peer disconnects by which side opened the connection [inbound / outbound]", + "type": "query", + "query": "label_values(peer_disconnect_total, direction)", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "includeAll": true, + "allValue": ".*", + "current": { + "text": "All", + "value": "$__all" + }, + "multi": true, + "refresh": 2, + "sort": 1 + }, + { + "name": "serve_reason", + "label": "Serve Reason", + "description": "Filter serve refusals by cause [sendq_full / load_shed / not_found / no_map / ...]", + "type": "query", + "query": "label_values(serve_refused_total, reason)", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "includeAll": true, + "allValue": ".*", + "current": { + "text": "All", + "value": "$__all" + }, + "multi": true, + "refresh": 2, + "sort": 1 + }, + { + "name": "serve_request", + "label": "Serve Request", + "description": "Filter serve refusals by what the peer asked for [ledger / tx-set / object / fetchpack]", + "type": "query", + "query": "label_values(serve_refused_total, request)", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "includeAll": true, + "allValue": ".*", + "current": { + "text": "All", + "value": "$__all" + }, + "multi": true, + "refresh": 2, + "sort": 1 + }, + { + "name": "replay_stage", + "label": "Replay Stage", + "description": "Filter replay fallbacks by sub-task stage [skiplist / delta]", + "type": "query", + "query": "label_values(ledger_replay_fallback_total, stage)", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "includeAll": true, + "allValue": ".*", + "current": { + "text": "All", + "value": "$__all" + }, + "multi": true, + "refresh": 2, + "sort": 1 + }, + { + "name": "replay_outcome", + "label": "Replay Outcome", + "description": "Filter replay tasks by terminal outcome [success / timeout / build_failed / parameter_failed]", + "type": "query", + "query": "label_values(ledger_replay_outcome_total, outcome)", + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "includeAll": true, + "allValue": ".*", + "current": { + "text": "All", + "value": "$__all" + }, + "multi": true, + "refresh": 2, + "sort": 1 } ] }, diff --git a/docker/telemetry/workload/expected_metrics.json b/docker/telemetry/workload/expected_metrics.json index 476d6c1400..f231a45434 100644 --- a/docker/telemetry/workload/expected_metrics.json +++ b/docker/telemetry/workload/expected_metrics.json @@ -156,12 +156,34 @@ "jobq_backlog{metric=\"deferred\",job_type=\"ledgerRequest\"}", "jobq_saturation{metric=\"running_tasks\"}", "jobq_saturation{metric=\"worker_threads\"}", - "jobq_saturation{metric=\"total_waiting\"}" + "jobq_saturation{metric=\"total_waiting\"}", + "peer_ledger_supply{metric=\"peers_reporting\"}", + "peer_ledger_supply{metric=\"peers_serving_validated\"}", + "peer_ledger_supply{metric=\"peers_serving_next\"}", + "peer_ledger_supply{metric=\"supply_min_seq\"}", + "peer_ledger_supply{metric=\"supply_max_seq\"}", + "peerfinder_slot_census{metric=\"out_active\"}", + "peerfinder_slot_census{metric=\"out_max\"}", + "peerfinder_slot_census{metric=\"in_active\"}", + "peerfinder_slot_census{metric=\"in_max\"}", + "peerfinder_slot_census{metric=\"connecting\"}", + "peerfinder_slot_census{metric=\"fixed_configured\"}", + "peerfinder_slot_census{metric=\"fixed_active\"}", + "peerfinder_slot_census{metric=\"bootcache\"}", + "peerfinder_slot_census{metric=\"livecache\"}", + "amendment_block{metric=\"warned\"}", + "amendment_block{metric=\"seconds_to_block\"}", + "peer_accept_total", + "nodestore_latency{metric=\"write_count\"}", + "nodestore_latency{metric=\"read_count\"}", + "nodestore_latency{metric=\"read_mean_us\"}" ], "_acquire_note": "The four sync_acquire sub-series and shamap_cache_hit_rate are unconditional: both are observable gauges whose callbacks observe every series on each collection tick, so each is present even when the value is 0 (an idle node reports in_flight=0 and missing_state_nodes_max=0, and a cold cache reports a 0.0 hit rate). Absence, not a zero, is the regression. The three WP-A3 counters (sync_acquire_source_total, sync_addnode_total, sync_acquire_no_progress_total) are deliberately NOT asserted here: all three are emitted only from InboundLedger, which runs only when a node must fetch a ledger it lacks. expected_spans.json already marks the ledger.acquire span optional for exactly this reason (\"A healthy local cluster rarely back-fills history\"), and the metric validator has no per-metric optional flag, so listing them would fail the harness red on a healthy run. They are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp and by the ledger-sync-health panels; add them here only alongside a harness step that forces a real acquire (e.g. starting a node against an existing ledger history).", "_jobq_note": "The jobq_backlog and jobq_saturation series are unconditional: both are observable gauges whose callbacks iterate EVERY registered JobType (jobData_ is populated from JobTypes at JobQueue construction) and observe all three fields on each collection tick, so a series exists even when the value is 0. That is why an idle-but-registered type like ledgerData is safe to assert by name here — a fresh harness node that never defers a single job still reports jobq_backlog{metric=\"deferred\",job_type=\"ledgerData\"} = 0, and absence, not the zero, is the regression. Two job_type values are asserted (ledgerData and ledgerRequest) because they are the sync-critical types capped at concurrency 3 in JobTypes.h, so they are the ones whose deferred series must never silently vanish. Only deferred is asserted for ledgerRequest to keep the list short: the three-field fan-out is already proven by ledgerData. worker_threads is asserted because it is the denominator of the dashboard saturation ratio, and it is always at least 1 (the JobQueue ctor gives standalone mode exactly one worker), so a zero or missing reading there means the accessor regressed rather than the node being idle.", "_conditional_note": "handshake_negotiation_fail_total and unl_fetch_total are conditional under the local harness: the first only exists once a handshake is rejected, and the second needs a [validator_list_sites] entry (run-full-validation.sh generates a static [validators] file instead). The validator has no per-metric optional flag, so if either reports 0 series in a harness run, move it out of this group rather than weakening the check.", - "_sync_state_note": "The four sync_state sub-series are unconditional: the gauge observes all four on every collection tick, so each is present as a series even when its value is 0 (a node that never reached FULL reports initial_full_duration_us=0, and a healthy node reports server_stall_seconds=0). The check asserts series presence, not a non-zero value, which is exactly right here — a zero is a meaningful reading for these signals, and absence is the regression. server_stall_events_total is likewise always present because the observable counter reports the tally (0 or more) every tick. state_changes_total is asserted here with a from!=\"\",to!=\"\" selector rather than bare (parity_counters already asserts the bare name): the selector is what proves the WP-A2 {from,to} label dimension actually reached Prometheus, so a regression to the old unlabelled counter fails this check instead of silently passing on the bare name. It needs at least one real mode transition, which any node reaching connected/syncing produces during startup." + "_sync_state_note": "The four sync_state sub-series are unconditional: the gauge observes all four on every collection tick, so each is present as a series even when its value is 0 (a node that never reached FULL reports initial_full_duration_us=0, and a healthy node reports server_stall_seconds=0). The check asserts series presence, not a non-zero value, which is exactly right here — a zero is a meaningful reading for these signals, and absence is the regression. server_stall_events_total is likewise always present because the observable counter reports the tally (0 or more) every tick. state_changes_total is asserted here with a from!=\"\",to!=\"\" selector rather than bare (parity_counters already asserts the bare name): the selector is what proves the WP-A2 {from,to} label dimension actually reached Prometheus, so a regression to the old unlabelled counter fails this check instead of silently passing on the bare name. It needs at least one real mode transition, which any node reaching connected/syncing produces during startup.", + "_a7_note": "WP-A7 adds three observable gauges and four counters. The 16 gauge sub-series (peer_ledger_supply, peerfinder_slot_census, amendment_block) are unconditional and asserted individually: each callback in MetricsRegistry.cpp calls observe() for every field on every collection tick with no early return between them, so the series exists whatever the value. That includes the two sentinel readings — a node whose peers have advertised nothing reports peer_ledger_supply{metric=\"supply_min_seq\"} = 0 meaning unknown, and a node with no pending amendment reports amendment_block{metric=\"seconds_to_block\"} = -1 meaning healthy. Absence, not the sentinel, is the regression. Of the four counters only peer_accept_total is asserted: run-full-validation.sh gives every node a [port_peer] on 0.0.0.0 and lists the other four nodes in [ips], so all 5 nodes dial each other and each one is also dialled, which means OverlayImpl::onHandoff runs and reports outcome=accepted (or slot_refused/no_slot on the duplicate half of each mutual dial) on every node. It is asserted bare rather than with an outcome= selector because which outcome a given node records depends on dial ordering, which the harness does not control. The other three counters are deliberately NOT asserted. peer_disconnect_total is emitted only from PeerImp::close, and a healthy 5-node localhost cluster holds its 4 fixed peers for the whole run: the timer-driven reasons need maxUnknownTime (600 s) or maxDivergedTime (300 s) to elapse (Config.h) while the full-validation profile totals well under that, and the shutdown reasons only fire during teardown, which happens in run-full-validation.sh after Step 5 has already scraped. serve_refused_total needs a peer to ask this node for a ledger, tx set or object it cannot serve — on a cluster where every node has the same complete history from genesis, getLedger()/getTxSet() succeed and the send queues never approach Tuning::kDropSendQueue. ledger_jump_total needs NetworkOPsImp::switchLastClosedLedger, reached only when consensus reports an LCL this node did not build on; a healthy 5-node cluster agrees every round, so it never jumps. The metric validator has no per-metric optional flag, so listing any of the three would fail the harness red on a healthy run — the same reasoning _acquire_note applies to the WP-A3 InboundLedger counters. All four counters are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp and rendered by the ledger-sync-health panels Peer Disconnects by Reason, Ledger/Object Serve Refusals and Byzantine Ledger Jumps. To make them assertable the harness would need a fault-injection step: kill one node mid-run and re-scrape before teardown (peer_disconnect_total, reason=read_error/graceful), request a ledger sequence outside the cluster's history or drive a node past its send-queue limit (serve_refused_total), and start a node on a divergent chain tip or partition the cluster and heal it (ledger_jump_total).", + "_a6_note": "WP-A6 adds one observable gauge (nodestore_latency) and two counters (ledger_replay_fallback_total, ledger_replay_outcome_total). Only three of the four gauge sub-series are asserted. write_count and read_count are unconditional: the callback observes both on every collection tick with no early return before them, so a series exists whatever the value, and a node that has written nothing reports write_count=0 rather than dropping the series. read_mean_us is safe because any node that has opened a ledger has already fetched objects, so the fetch duration total is non-zero. write_mean_us is deliberately NOT asserted: the mean is emitted only when the store-duration total is non-zero, and that total is fed by Database::recordStoreDuration(), which today is called only from Database::importInternal -- the [import_db] admin path. Database::store() is pure virtual and the two concrete runtime overrides (DatabaseNodeImp::store, DatabaseRotatingImp::store) do not time themselves yet, so an ordinary harness node produces write_count but no write_mean_us. Asserting it would ship a permanently red CI check for a known, documented gap; the omission is the honest encoding of that gap. The two replay counters are likewise NOT asserted, for the same reason _acquire_note gives for the WP-A3 InboundLedger counters: both are emitted only from the ledger-replay path, which requires the [ledger_replay] config stanza AND a real historical back-fill against peers that support the LedgerReplay protocol feature. run-full-validation.sh starts a fresh local cluster with no history to back-fill, so no replay task is ever created and neither counter can produce a series. All three unasserted signals are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp and rendered by the ledger-sync-health panels NodeStore Write vs Read Latency, Replay Fallback to Full Acquire and Replay Outcomes. To make them assertable the harness would need to enable [ledger_replay] and start a node against an existing ledger history so it back-fills through the replay path, and to time the two concrete store overrides." }, "grafana_dashboards": { "description": "All Grafana dashboards that must render data (UIDs as provisioned on disk under docker/telemetry/grafana/dashboards/).", diff --git a/docs/telemetry-glossary.md b/docs/telemetry-glossary.md index 4693723756..d25b5027c3 100644 --- a/docs/telemetry-glossary.md +++ b/docs/telemetry-glossary.md @@ -491,6 +491,16 @@ When a node is missing ledgers (at startup, after an outage, or to extend histor **Scope:** per node — measured on and specific to this individual server. + + +### Byzantine ledger jump + +Being told that the network's last closed ledger is not the one this node built on, and discarding its own chain tip to follow the network instead. It is an abnormal event by construction: the node had already closed a ledger, and it is now throwing that work away because the peers it listens to agree on a different one. A single jump while a fresh node is still settling onto the network's chain can be benign. Repeated jumps are wrong-chain thrash — the node keeps switching between chains and never settles — and the cause is upstream of the sync pipeline, in which peers it is listening to or which network it thinks it is on, so nothing in ledger acquisition can fix it. + +**Scope:** per node — measured on and specific to this individual server. + +**See also:** [Fork](#fork) · [Ledger history mismatch](#ledger-history-mismatch) · [Insane / diverged peers](#insane-diverged-peers) + ### Clock close offset @@ -605,6 +615,16 @@ Acquiring a ledger means requesting it and its contents from peers when the node **Scope:** per node — measured on and specific to this individual server. + + +### Ledger replay + +An optional faster way to rebuild a run of historical ledgers: instead of downloading each ledger whole, the node fetches one starting ledger plus the list of ledger hashes that links the range, then fetches only what changed in each subsequent ledger and applies those changes on top of its predecessor. It is only available when enough connected peers support the protocol feature that serves those pieces, so whether it is used at all depends on the peer set rather than on local configuration alone. + +**Scope:** per node — measured on and specific to this individual server. + +**See also:** [Replay fallback](#replay-fallback) · [Ledger acquire (inbound fetch)](#ledger-acquire-inbound-fetch) + ### Ledgers behind network @@ -635,6 +655,36 @@ The startup guard that holds a node back until it has seen a complete ledger fro **See also:** [Operating mode / server state](#operating-mode-server-state) · [UNL quorum headroom](#unl-quorum-headroom) · [Operating mode / server state on xrpl.org](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) + + +### Node-store read latency + +How long the node store takes to return one stored object. Every ledger traversal that is not already answered from an in-memory cache pays this cost, so it is the floor under ledger acquisition and under most queries. It is reported as an average over an interval rather than as a distribution, which means a slow minority of reads shows up as a raised average rather than as a separate tail figure. + +**Scope:** per node — measured on and specific to this individual server. + +**See also:** [Node-store write latency](#node-store-write-latency) · [SHAMap cache hit rate](#shamap-cache-hit-rate) + + + +### Node-store operation rate + +How many objects per second the node store is storing and retrieving. It is the companion an average latency needs in order to be read correctly: latency measured over an interval with almost no operations in it is a stale number rather than a good one, and a node writing nothing at all while still behind the network is stalled somewhere upstream of storage rather than slowed by it. + +**Scope:** per node — measured on and specific to this individual server. + +**See also:** [Node-store write latency](#node-store-write-latency) + + + +### Node-store write latency + +How long the node store takes to persist one object. This is the cost that governs how fast a node can absorb ledger history, because filling in history is dominated by writing rather than by reading. It is the measurement that distinguishes the two ways a sync can be slow: starved of data from peers, or unable to write down the data it already has. A node with a large existing database can be slower to start and catch up than an empty one for exactly this reason, and no read-side measurement reveals it. Like the read figure it is an interval average, not a distribution. + +**Scope:** per node — measured on and specific to this individual server. + +**See also:** [Node-store read latency](#node-store-read-latency) · [Node-store operation rate](#node-store-operation-rate) + ### Operating mode / server state @@ -655,6 +705,16 @@ The elapsed time of one outbound peer connection attempt, from starting the TCP **See also:** [DNS resolve](#dns-resolve) · [Handshake negotiation failure](#handshake-negotiation-failure) · [Peer protocol on xrpl.org](https://xrpl.org/docs/concepts/networks-and-servers/peer-protocol) + + +### Peer ledger supply + +The idea that a connected peer set collectively offers a window of ledger sequences, rather than being simply present or absent. Each peer advertises the oldest and newest ledger it holds, so the set as a whole can serve some range and nothing outside it. This turns "how many peers do I have" into the question that actually matters during a sync: does any connected peer hold the next ledger this node needs. Being unable to advance because nobody holds that sequence is a fundamentally different fault from being slow — it is a supply gap fixed only by changing the peer set, whereas slowness with the data available is a throughput problem fixed locally, and the two are indistinguishable from inside the acquire itself. The shape of a gap matters too: needing a sequence below the window means asking for history nobody kept, while needing one above it means asking for a tip nobody has reached. Peers that have not advertised a range yet are excluded from the counts entirely, so a zero window means unknown rather than empty, and the count of peers that have reported anything is what makes the rest readable. + +**Scope:** per node — measured on and specific to this individual server. + +**See also:** [Ledgers behind network](#ledgers-behind-network) · [Slot census](#slot-census) · [Acquire stall](#acquire-stall) · [Complete ledger ranges](#complete-ledger-ranges) + ### Received-data stash @@ -665,6 +725,16 @@ Peer packets held for later processing because a ledger acquire cannot apply the **See also:** [Add-node outcome](#add-node-outcome) · [Acquire stall](#acquire-stall) + + +### Replay fallback + +A replay sub-task giving up on the delta shortcut and acquiring the entire ledger instead, which happens when too few connected peers support the feature that serves the pieces replay needs. Nothing fails when this occurs and no error is raised — the node still completes its back-fill, just on the slower path — which is why it is easy to miss: the optimisation is simply absent. It is counted separately for each of the two sub-tasks, because the one that fetches the list of historical ledger hashes and the one that fetches a single ledger's changes can fail independently of each other. + +**Scope:** per node — measured on and specific to this individual server. + +**See also:** [Ledger replay](#ledger-replay) + ### SHAMap cache hit rate @@ -739,6 +809,16 @@ A cluster is a set of servers run by the same operator that trust each other, ex **See also:** [Cluster on xrpl.org](https://xrpl.org/docs/concepts/networks-and-servers/clustering) + + +### Disconnect reason + +The cause recorded when a peer connection is torn down, kept alongside the direction the connection was originally opened in. A single disconnect count cannot separate the two situations that matter, because they produce the same number: a node shedding load, which drops peers deliberately because it could not keep up with what it owed them or because a peer exceeded its resource allowance, and a network or topology fault, where the peer became unreachable, stopped answering keepalives, or turned out to be following a different chain. The first is a local capacity problem and the peer list is not the fix; the second is the opposite. A third group is neither — clean teardown at shutdown and peers closing their own side are ordinary churn, and a count dominated by those is healthy. The direction matters separately, since churn among the peers a node dials points somewhere different from churn among the peers that dial it. + +**Scope:** per node — measured on and specific to this individual server. + +**See also:** [Resource disconnect](#resource-disconnect) · [Slot census](#slot-census) · [Insane / diverged peers](#insane-diverged-peers) + ### Fetch-pack @@ -837,6 +917,26 @@ Set-get (fetch) and set-share messages exchange transaction-set data between pee **Scope:** per node — measured on and specific to this individual server. + + +### Serve refusal + +A peer data request that this node declined to answer — the supply side of the sync exchange, as opposed to everything a node measures about its own fetching. It matters because a node that refuses everything it is asked for looks, from the outside, exactly like a node nobody asks: both serve nothing. From the asking peer's point of view a refusal is indistinguishable from a peer that does not hold the data, so refusals directly slow the sync of every peer that depends on this node. The reason divides them into two kinds. Self-inflicted refusals mean the node was too loaded to answer — its outgoing queue to that peer had grown past its limit, or the local fee track showed it under load, or too much bulk-transfer work was already queued — and these are the serving-side symptom of the same overload that shows up as stalls and job-queue backlog locally. A refusal because the data was simply not held is different: that is a genuine history gap, a question of what this node retains rather than how busy it is. + +**Scope:** per node — measured on and specific to this individual server. + +**See also:** [Fetch-pack](#fetch-pack) · [GetObject / object fetch](#getobject-object-fetch) · [Complete ledger ranges](#complete-ledger-ranges) · [Peer ledger supply](#peer-ledger-supply) + + + +### Slot census + +A single consistent reading of everything PeerFinder knows about this node's peering position: how many outbound and inbound slots are occupied against how many exist, how many outbound attempts are in flight, how many configured fixed peers are connected against how many were configured, and the depth of the two address stores. Taken together at one instant, so the numbers can be compared against each other. The three terms worth defining plainly: an occupied outbound slot is a peer this node dialled and is now connected to; the bootstrap address store is a persisted list of addresses kept across restarts purely so a starting node has somewhere to dial; and the live address store holds addresses learned from peers during this session and exists only in memory. Occupancy alone cannot explain a peering failure, which is the reason the census exists. A node with no outbound peers might be dialling continuously and never completing, or not dialling at all because it has no addresses to try, or dialling only configured peers that are unreachable — three different faults with three different fixes, and the occupancy count is identical in all of them. It is the attempt count, the address-store depths, and the configured-versus-connected comparison that tell them apart. + +**Scope:** per node — measured on and specific to this individual server. + +**See also:** [Overlay](#overlay) · [Disconnect reason](#disconnect-reason) · [Peer ledger supply](#peer-ledger-supply) · [DNS resolve](#dns-resolve) · [Outbound dial latency](#outbound-dial-latency) + ### Squelch @@ -905,6 +1005,16 @@ The NodeStore serves reads through a pool of read threads (optionally bundling r ## Validator Health + + +### Amendment block countdown + +The window between an amendment this build does not understand reaching majority among validators and that amendment actually activating. It exists because amendment activation is not instantaneous: once an unsupported amendment has majority support it becomes expected to activate at a known future time, and until then the node still works normally. That window is the only actionable part of an otherwise terminal condition — after activation the node stops validating and cannot resume without a software upgrade, so there is no operational fix left, only a rebuild and restart. Read as a countdown it therefore outranks every other sync signal in urgency: a node counting down is going to stop validating at a knowable moment, and anything else that looks wrong is secondary. The healthy state is reported as an explicit sentinel value rather than as absent data, so a node with nothing pending is distinguishable from a node whose reporting has broken, and the countdown is held at zero rather than going negative once activation is due. The identity of the blocking amendment is deliberately not carried on the metric — the network can vote on any amendment identifier, including ones this build has never heard of, which would make it an unbounded label — so the hash comes from the log line that records the amendment reaching majority. + +**Scope:** per node — measured on and specific to this individual server. + +**See also:** [Amendment blocked](#amendment-blocked) · [UNL blocked](#unl-blocked) · [Amendments on xrpl.org](https://xrpl.org/docs/concepts/networks-and-servers/amendments) + ### Amendment blocked diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index 1132ef03ee..903b468670 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -2310,6 +2310,196 @@ each step gates the next: stop at the first one that is wrong. waited, so it reports the past; these gauges report what is sitting in the queue right now, including the part being actively withheld. +11. **Can the network even serve the ledgers this node needs?** + Steps 6 to 10 all assume some peer holds what the node is asking for. This + step tests that assumption, and it is the one that separates "slow" from + "impossible". Panel _Peers Able to Serve Needed Sequence_ + (`peer_ledger_supply`, `metric=peers_reporting`, + `peers_serving_validated` and `peers_serving_next`). Read the two counts + together — `peers_reporting` is the denominator that makes the rest + meaningful: + - **`peers_serving_next` at zero with `peers_reporting` above zero** — + the decisive reading. Peers are connected and have advertised their + ranges, and **none of them holds the next ledger this node must + acquire.** No amount of waiting finishes the sync; the peer set itself + has to change. Add peers that hold the range, or point the node at a + full-history server. Everything in steps 6 to 10 will look starved as a + consequence, so do not chase them. + - **`peers_serving_next` above zero but the sync is still slow** — supply + is fine and the fault is downstream. Go back to steps 6 to 10: the data + is available, so the limit is acquire progress, local processing or + worker threads. + - **`peers_reporting` at zero** — nothing has advertised a range yet. + This is not a supply gap; it means the node has no peers, or its peers + have not sent a status change. Peers advertising an empty range are + excluded from every field, so the two window fields read 0 meaning + **unknown**, not genesis — do not read a zero window here as "peers + serve from the start of history". Go back to the Bootstrap row, and to + step 12 for why there are no peers. + Then read _Peer Ledger Supply Window_ (`peer_ledger_supply`, + `metric=supply_min_seq` and `supply_max_seq`) against the sequence the + node wants. This is what tells the two shapes of a supply gap apart: a + needed sequence **below** `supply_min_seq` means the node is asking for + history nobody kept, so it needs a full-history peer; a needed sequence + **above** `supply_max_seq` means it is asking for a tip nobody has + reached, which is a peer set lagging the real network rather than a + history problem. + +12. **Is this node failing to get or keep peers, and why?** + Step 11 says whether the peer set can serve; this step says why the peer + set is what it is. Panel _PeerFinder Slot Census_ + (`peerfinder_slot_census`) with _PeerFinder Address Caches & Fixed Peers_ + beside it. All nine fields come from one lock acquire, so occupancy and + capacity can be compared directly — which is what separates three faults + that otherwise look identical: + - **`connecting` non-zero while `out_active` stays below `out_max`** — + the node is dialling and the dials never complete. Without the attempt + count this looks exactly like a node that is not dialling at all. Pair + it with _Outbound Dial Outcome Rate_ in the Bootstrap row for the stage + that breaks. + - **`bootcache` and `livecache` both at 0** — there is nothing to dial. + No seed addresses at all, so check `[ips]` and DNS in Bootstrap step 1. + - **`fixed_active` below `fixed_configured`** — a peer named in the + configuration is unreachable. `fixed_configured` is what was asked for + and `fixed_active` is what was obtained, so any shortfall names a + specific configured peer to check. + Then split the traffic by direction. _Inbound Peer Accept Outcomes_ + (`peer_accept_total`, by `outcome`) covers connections offered **to** + this node; the already-documented `overlay_connect_total{outcome}` in + Bootstrap step 2 covers dials **from** it. Reading both is the only way + to get the full in/out picture: a node refusing every inbound + connection looks the same as one nobody dials until these are separated. + On the inbound side `resource_limit`, `no_slot` and `slot_refused` are + this node declining (load, capacity, or a duplicate), while + `protocol_mismatch`, `bad_cookie` and `handshake_error` point at the + peer or at a network-id mismatch. + Then read _Peer Disconnects by Reason_ (`peer_disconnect_total`, by + `reason` and `direction`). One disconnect count cannot separate the two + causes; the label can: + - `large_sendq`, `charge_resources` — **our fault.** This node could not + keep up with what it owed the peer, or charged it past the resource + limit, so it shed the connection as backpressure. The fix is local + capacity, not the peer list, and it sends you back to steps 9 and 10. + - `not_useful`, `ping_timeout`, `read_error` — topology or network + faults. The peer is on a different chain or unreachable, so the fix is + the peer set. + - `graceful`, `shutdown`, `stopping` — normal churn and clean teardown, + not faults. A run dominated by these is healthy. + Use `direction` to tell churn in the peers this node dials from churn + in the peers that dial it. + Finally, the mirror-image question: _Ledger/Object Serve Refusals_ + (`serve_refused_total`, by `request` and `reason`) is what **this node + refuses to serve OTHERS**. It does not explain this node's own sync, but + it explains its peers' — and a node that refuses everything is why some + other operator is reading step 11 on their side. `sendq_full` and + `load_shed` are self-inflicted: this node is too loaded or too far + behind on its send queue to answer, so treat them as the serving-side + symptom of the same overload steps 3, 9 and 10 cover. `not_found` is + different — it is a genuine history gap, meaning the data was asked for + and this node simply does not hold it, which is a configuration and + retention question rather than a load one. + +13. **Is the node about to stop validating for good?** + Panel _Amendment Block Countdown_ (`amendment_block`, + `metric=seconds_to_block`) with _Amendment Warned_ (`metric=warned`) + beside it. **This step outranks every other step in urgency**, so check it + whenever a sync looks wrong, not only after the ten above are clean: + - `seconds_to_block` at **-1** — healthy. The -1 is an explicit sentinel + meaning nothing is pending, chosen so the healthy state is a distinct + value rather than a missing series. Do not read it as a negative + duration or as absent data. + - `seconds_to_block` at **any non-negative value** — a countdown to a + **terminal** state. When it expires the node becomes amendment-blocked: + it stops validating and will never validate again without a software + upgrade. The value is clamped at 0 rather than going negative, so a 0 + means the activation is due or past due, not that it just started. + Nothing else on this dashboard matters if this is counting down — plan + the upgrade inside the window, because after it there is no + operational fix. + `warned` reaching 1 is the same condition seen as a flag: an + unsupported amendment has reached majority. The existing + `validator_health{metric="amendment_blocked"}` on the Validator Health + dashboard is the after-the-fact companion — it reports the block once it + has happened, when nothing can be done, whereas this countdown is the + only actionable part. + The blocking amendment's hash is **not** a metric label, deliberately: + the network can vote on an arbitrary 256-bit amendment id, not drawn + from this build's known features, so an id label would be unbounded + cardinality and would mint a permanent new series per amendment. + Get the hash from the log line in `AmendmentTableImpl::doValidatedLedger` + ("Unsupported amendment ... reached majority at ...") via Loki, + correlated to this series by node and time. + Finally, read _Byzantine Ledger Jumps_ (`ledger_jump_total`) in the + same pass. Any non-zero rate means the node was fed a last-closed + ledger it had not built on and **discarded its own chain tip** to + follow. A single jump during a fresh sync can be benign as the node + settles onto the network's chain. Repeated jumps are wrong-chain + thrash: the node keeps switching between chains and never settles, so + check the peer set from step 12 and the configured network id from + Bootstrap step 3 — those are what put a node on the wrong chain in the + first place. Nothing in the acquire pipeline can fix it. + +14. **Is the node store itself the bottleneck — and is it the write side?** + This is the step for the specific symptom **"a node with a large existing + database starts and syncs slower than a fresh one"**. Back-fill is + write-bound, so no read-side panel can show it; check this step whenever a + node with existing history is the slow one. + Panel _NodeStore Write vs Read Latency (us/op)_ (`nodestore_latency`, + `metric=write_mean_us` and `read_mean_us`) with _NodeStore Operation Rate_ + (`metric=write_count` / `read_count`) beside it: + - **Write line rising during history back-fill** — the backend cannot + absorb writes fast enough. Sync will stay slow however many peers are + available, so adding peers will not help. Check storage IOPS, the + `[node_db]` backend and its tuning, and whether the online-delete / + rotation cycle is competing with the back-fill writes. Correlate with + `nodestore_state{metric="write_load"}` on the Ledger Data Sync dashboard. + - **Read line far above the write line** — the read path, not the write + path, is the cost. Read it together with _SHAMap TreeNode Cache Hit Rate_ + (step 7): a cold in-memory cache sends every tree walk to disk, and that + shows up here as read latency rather than as a node-store fault. + - **Write rate at zero while the node is still behind the network** — + nothing is being persisted at all, so the stall is upstream of the node + store. Go back to peer supply (step 12) and the acquire panels (steps + 6-8); storage is not the problem. + - Both panels use the **rate of the mean divided by the rate of the + count**, which is why the count series exist. Read as an interval + latency, not a since-boot average — on a long-running node the raw + cumulative mean moves so slowly that a current stall is invisible in it. + - Two limits to keep in mind. First, this is a **mean, not a percentile**: + a tail that matters will move it, but there is no p99 here. That is a + deliberate cost trade — a histogram would need one `Record()` per node + object, and a single ledger write walks thousands of SHAMap nodes. + Second, `write_mean_us` is currently emitted only for store paths that + record their duration, which today is the `[import_db]` admin import. + On an ordinary node you will see `write_count` climbing with **no** + `write_mean_us` line: that is a known instrumentation gap, not a healthy + zero, and the mean is deliberately omitted rather than drawn as 0 so it + cannot be misread as "writes are instantaneous". + +15. **Is replay-based back-fill silently falling back to the slow path?** + Only relevant when `[ledger_replay]` is enabled. Panels _Replay Fallback to + Full Acquire (by stage)_ (`ledger_replay_fallback_total`) and _Replay + Outcomes (by terminal state)_ (`ledger_replay_outcome_total`): + - **Any sustained fallback rate** — too few connected peers support the + `LedgerReplay` protocol feature, so every historical ledger is fetched + whole instead of as a delta. Back-fill still completes, just far slower, + which is why this is easy to miss: nothing fails, the optimisation is + simply gone. The `stage` label says which sub-task gave up — `skiplist` + (fetching the list of historical ledger hashes) or `delta` (a single + ledger's changes). Fix by peering with nodes that support the feature. + - **Failure outcomes climbing while `success` stays flat** — replay runs + but never completes. The outcome names the layer at fault: `timeout` + means the deltas never arrived, so treat it as a peer-supply problem and + read it with step 12; `build_failed` means a delta would not apply to its + parent, and `parameter_failed` means a peer served a skip list + inconsistent with the request — those two are **data** faults from the + serving peers, not slowness, so the peer set is suspect rather than the + network. + - **All series absent** — expected when `[ledger_replay]` is not + configured, or on a node with no history to back-fill. Absence here is + not a regression; it means no replay task was ever created. For the same + reason neither counter is asserted by the local validation harness. + ## Performance Tuning | Scenario | Recommendation | diff --git a/src/tests/libxrpl/telemetry/MetricMacros.cpp b/src/tests/libxrpl/telemetry/MetricMacros.cpp index 7904b4d632..eb37fce529 100644 --- a/src/tests/libxrpl/telemetry/MetricMacros.cpp +++ b/src/tests/libxrpl/telemetry/MetricMacros.cpp @@ -25,6 +25,9 @@ #include +#include +#include + #include #include #include @@ -44,12 +47,13 @@ #include #include +#include #include #include -#include #include #include #include +#include #include #include #include @@ -268,8 +272,10 @@ public: for (auto const& metric : scope.metric_data_) { for (auto const& point : metric.point_data_attr_) + { out[metric.instrument_descriptor.name_][point.attributes] = point.point_data; + } } } return true; @@ -886,10 +892,10 @@ TEST(MetricMacros, unl_quorum_gauge_observes_exact_trusted_keys_and_quorum) // registry reads them live from ValidatorList on each collection tick. struct Observed { - std::int64_t trustedKeys; - std::int64_t quorum; + std::int64_t trustedKeys{0}; + std::int64_t quorum{0}; }; - Observed observed{5, 4}; + Observed observed{.trustedKeys = 5, .quorum = 4}; // Keep the instrument alive for the whole test: destroying the handle // deregisters the callback (ObservableInstrument's destructor calls @@ -1121,14 +1127,18 @@ TEST(MetricMacros, sync_state_gauge_observes_exact_stuck_node_values) // real registry reads them from NetworkOPs/LoadManager on each tick. struct Observed { - std::int64_t initialFullDurationUs; - std::int64_t networkLedgerGate; - std::int64_t serverStallSeconds; - std::int64_t ledgersBehind; + std::int64_t initialFullDurationUs{0}; + std::int64_t networkLedgerGate{0}; + std::int64_t serverStallSeconds{0}; + std::int64_t ledgersBehind{0}; }; // A node that never synced: no FULL yet, gate closed, 42 s stalled, 150 // ledgers behind (network tip 250 vs our validated 100). - Observed observed{0, 1, 42, 150}; + Observed observed{ + .initialFullDurationUs = 0, + .networkLedgerGate = 1, + .serverStallSeconds = 42, + .ledgersBehind = 150}; // Keep the instrument alive for the whole test: destroying the handle // deregisters the callback, which is why the real registry holds a member. @@ -1177,7 +1187,11 @@ TEST(MetricMacros, sync_state_gauge_observes_exact_stuck_node_values) // A healthy node reports the complementary values through the same // callback: synced in 12.5 s, gate open, no stall, at the tip. - observed = Observed{12'500'000, 0, 0, 0}; + observed = Observed{ + .initialFullDurationUs = 12'500'000, + .networkLedgerGate = 0, + .serverStallSeconds = 0, + .ledgersBehind = 0}; auto const healthy = provider.collect(); EXPECT_EQ( gaugeValue(healthy, "sync_state", attrs("metric", "initial_full_duration_us")), 12'500'000); @@ -1433,14 +1447,15 @@ TEST(MetricMacros, sync_acquire_gauge_observes_exact_stuck_acquire_values) // registry reads it from InboundLedgers on each collection tick. struct Observed { - std::int64_t maxMissingStateNodes; - std::int64_t maxMissingTxNodes; - std::int64_t receivedDataDepth; - std::int64_t inFlight; + std::int64_t maxMissingStateNodes{0}; + std::int64_t maxMissingTxNodes{0}; + std::int64_t receivedDataDepth{0}; + std::int64_t inFlight{0}; }; // A stuck acquire: 256 state nodes still outstanding (the sweep cap), the tx // tree already done, 4 packets stashed, 2 acquires running. - Observed observed{256, 0, 4, 2}; + Observed observed{ + .maxMissingStateNodes = 256, .maxMissingTxNodes = 0, .receivedDataDepth = 4, .inFlight = 2}; // Keep the instrument alive for the whole test: destroying the handle // deregisters the callback, which is why the real registry holds a member. @@ -1487,12 +1502,13 @@ TEST(MetricMacros, sync_acquire_gauge_observes_exact_stuck_acquire_values) // A shrinking count is the "slow but alive" reading, and an idle node // reports all zeros with in_flight=0 -- distinguishable from a stuck node // only because in_flight is exported alongside. - observed = Observed{128, 0, 1, 2}; + observed = Observed{ + .maxMissingStateNodes = 128, .maxMissingTxNodes = 0, .receivedDataDepth = 1, .inFlight = 2}; EXPECT_EQ( gaugeValue(provider.collect(), "sync_acquire", attrs("metric", "missing_state_nodes_max")), 128); - observed = Observed{0, 0, 0, 0}; + observed = Observed{}; auto const idle = provider.collect(); EXPECT_EQ(gaugeValue(idle, "sync_acquire", attrs("metric", "missing_state_nodes_max")), 0); EXPECT_EQ(gaugeValue(idle, "sync_acquire", attrs("metric", "in_flight")), 0); @@ -1783,4 +1799,1127 @@ TEST(MetricMacros, jobq_saturation_gauge_observes_exact_pool_exhaustion_values) EXPECT_EQ(gaugeValue(standalone, "jobq_saturation", attrs("metric", "total_waiting")), 3); } +// ----------------------------------------------------------------- +// Peer-supply, slot-census and amendment-countdown diagnostics (WP-A7). +// +// Asserts the EXACT values and label shapes of the three gauges and the four +// counters: +// peer_ledger_supply{metric} MetricsRegistry::registerPeerLedgerSupplyGauge +// peerfinder_slot_census{metric} MetricsRegistry::registerSlotCensusGauge +// amendment_block{metric} MetricsRegistry::registerAmendmentBlockGauge +// peer_disconnect_total{reason,direction} PeerImp::close +// peer_accept_total{outcome} OverlayImpl::reportAcceptOutcome +// serve_refused_total{request,reason} PeerImp::reportServeRefusal +// ledger_jump_total (unlabelled) NetworkOPsImp::switchLastClosedLedger +// +// The three gauges are observable instruments registered directly on the SDK +// meter, mirroring the production callback shape, because the real +// MetricsRegistry's enabled path cannot be linked into this standalone binary +// (see the file header). The snapshot types are the REAL xrpl::PeerLedgerSupply +// and xrpl::PeerFinder::SlotCensus aggregates, so a field rename or a reorder on +// either side breaks these tests instead of silently drifting from production. +// Both are plain header-only aggregates with no out-of-line members, so using +// them here adds no xrpld link dependency. +// +// The four counters are driven through XRPL_METRIC_COUNTER_INC_LABELED / +// XRPL_METRIC_COUNTER_INC, which is exactly what the production call sites use. +// ----------------------------------------------------------------- + +// peer_ledger_supply must keep the two "who can serve me" counts on separate +// series from the "who is even talking" denominator. The values chosen are the +// headline supply gap: three peers connected and advertising a range, all three +// covering the validated sequence, and NOT ONE covering the next one needed. +TEST(MetricMacros, peer_ledger_supply_gauge_names_a_gap_no_peer_can_fill) +{ + CollectingProvider const provider; + + // The real aggregate the production callback reports, filled here as + // OverlayImpl::getPeerLedgerSupply() would fill it. Peer set holds + // [1000, 4000]; this node's validated sequence is 4000, so the next needed + // is 4001 -- past every peer's tip. + PeerLedgerSupply observed{ + .peersReporting = 3, + .peersServingValidated = 3, + .peersServingNext = 0, + .supplyMinSeq = 1000, + .supplyMaxSeq = 4000}; + + // Keep the instrument alive for the whole test: destroying the handle + // deregisters the callback, which is why the real registry holds a member. + auto gauge = provider.meter()->CreateInt64ObservableGauge( + "peer_ledger_supply", "Peer coverage of the ledger sequence this node needs"); + gauge->AddCallback( + [](opentelemetry::metrics::ObserverResult result, void* state) { + auto const* self = static_cast(state); + // Same single-label Observe() form the production callback uses. + auto observe = [&](char const* field, std::int64_t value) { + opentelemetry::nostd::get>>(result) + ->Observe(value, {{"metric", field}}); + }; + observe("peers_reporting", self->peersReporting); + observe("peers_serving_validated", self->peersServingValidated); + observe("peers_serving_next", self->peersServingNext); + observe("supply_min_seq", self->supplyMinSeq); + observe("supply_max_seq", self->supplyMaxSeq); + }, + &observed); + + auto const gap = provider.collect(); + + // Exactly five series, one per `metric` value: no field collapses into + // another, so the denominator and the verdict stay separately readable. + ASSERT_EQ(gap.at("peer_ledger_supply").size(), 5u); + + // THE verdict this signal exists for: zero peers can serve the next needed + // ledger while three are connected and reporting. That pair is the whole + // point -- "the network cannot supply what I need" is otherwise + // indistinguishable from "my peers are slow", and the two faults have + // completely different fixes (change the peer set vs. wait). + EXPECT_EQ(gaugeValue(gap, "peer_ledger_supply", attrs("metric", "peers_serving_next")), 0); + EXPECT_EQ(gaugeValue(gap, "peer_ledger_supply", attrs("metric", "peers_reporting")), 3); + + // Serving the validated sequence is NOT the same question, and reads 3 here: + // the peers can serve where this node already is, just not where it must go + // next. Without both counts the gap would look like a total peer failure. + EXPECT_EQ(gaugeValue(gap, "peer_ledger_supply", attrs("metric", "peers_serving_validated")), 3); + + // The window, so an operator can see whether the wanted sequence is below + // the peer set's floor (discarded history) or above its tip (unreached). + EXPECT_EQ(gaugeValue(gap, "peer_ledger_supply", attrs("metric", "supply_min_seq")), 1000); + EXPECT_EQ(gaugeValue(gap, "peer_ledger_supply", attrs("metric", "supply_max_seq")), 4000); + + // Exactly one label key, and it is "metric". This is the cardinality guard: + // a peer_id label here would mint a new series per connection. + for (auto const& [labels, point] : gap.at("peer_ledger_supply")) + { + ASSERT_EQ(labels.size(), 1u); + EXPECT_EQ(labels.begin()->first, "metric"); + } + + // NEGATIVE: a `metric` value outside the production set of five has no + // series, so the readings above are not an artifact of a catch-all series. + EXPECT_EQ(gap.at("peer_ledger_supply").count(attrs("metric", "peers_serving")), 0u); +} + +// The complementary reading to the gap above: a healthy peer set where every +// reporting peer covers the next needed sequence, so waiting WILL finish the +// sync. Split from the gap test to keep each under the length limit. +TEST(MetricMacros, peer_ledger_supply_gauge_reads_zero_window_as_unknown) +{ + CollectingProvider const provider; + + // Healthy: four peers, all covering both the validated sequence and the + // next one, window [1000, 5000]. + PeerLedgerSupply observed{ + .peersReporting = 4, + .peersServingValidated = 4, + .peersServingNext = 4, + .supplyMinSeq = 1000, + .supplyMaxSeq = 5000}; + + auto gauge = provider.meter()->CreateInt64ObservableGauge( + "peer_ledger_supply", "Peer coverage of the ledger sequence this node needs"); + gauge->AddCallback( + [](opentelemetry::metrics::ObserverResult result, void* state) { + auto const* self = static_cast(state); + auto observe = [&](char const* field, std::int64_t value) { + opentelemetry::nostd::get>>(result) + ->Observe(value, {{"metric", field}}); + }; + observe("peers_reporting", self->peersReporting); + observe("peers_serving_validated", self->peersServingValidated); + observe("peers_serving_next", self->peersServingNext); + observe("supply_min_seq", self->supplyMinSeq); + observe("supply_max_seq", self->supplyMaxSeq); + }, + &observed); + + auto const healthy = provider.collect(); + EXPECT_EQ(gaugeValue(healthy, "peer_ledger_supply", attrs("metric", "peers_serving_next")), 4); + EXPECT_EQ(gaugeValue(healthy, "peer_ledger_supply", attrs("metric", "peers_reporting")), 4); + + // NEGATIVE/edge: nothing has advertised a range yet. Peers that have not + // sent mtSTATUS_CHANGE report [0, 0] and are excluded from every field, so + // all five read 0. + observed = PeerLedgerSupply{}; + auto const silent = provider.collect(); + + // A 0 window means "unknown", NOT "the peer set serves from genesis". The + // only thing that separates the two is peers_reporting, which is why it must + // be read alongside: 0 out of 0 reporting is silence, 0 out of many would be + // a real supply gap. Asserting the pair together pins that contract. + EXPECT_EQ(gaugeValue(silent, "peer_ledger_supply", attrs("metric", "supply_min_seq")), 0); + EXPECT_EQ(gaugeValue(silent, "peer_ledger_supply", attrs("metric", "peers_reporting")), 0); + EXPECT_EQ(gaugeValue(silent, "peer_ledger_supply", attrs("metric", "supply_max_seq")), 0); + + // Every field is still a present series at 0, never absent: a dropped + // series would be indistinguishable from a dead exporter. + ASSERT_EQ(silent.at("peer_ledger_supply").size(), 5u); + EXPECT_EQ(gaugeValue(silent, "peer_ledger_supply", attrs("metric", "peers_serving_next")), 0); + EXPECT_EQ( + gaugeValue(silent, "peer_ledger_supply", attrs("metric", "peers_serving_validated")), 0); + + // A single reporting peer at the network tip: min and max collapse to the + // same sequence, which is a legitimate reading, not a defect. + observed = PeerLedgerSupply{ + .peersReporting = 1, + .peersServingValidated = 1, + .peersServingNext = 0, + .supplyMinSeq = 5000, + .supplyMaxSeq = 5000}; + auto const single = provider.collect(); + EXPECT_EQ(gaugeValue(single, "peer_ledger_supply", attrs("metric", "supply_min_seq")), 5000); + EXPECT_EQ(gaugeValue(single, "peer_ledger_supply", attrs("metric", "supply_max_seq")), 5000); +} + +// peerfinder_slot_census must export all nine numbers, because each of the three +// common bootstrap failures is named by a DIFFERENT pair of them and today only +// the two active counts exist. The values chosen are the "dialling but never +// completing" case, which the two legacy gauges cannot express at all. +TEST(MetricMacros, slot_census_gauge_names_each_bootstrap_fault_exactly) +{ + CollectingProvider const provider; + + // The real snapshot type the production callback consumes. Outbound is 2 of + // 10 with 6 dials in flight; one configured fixed peer is missing. + PeerFinder::SlotCensus observed{ + .outActive = 2, + .outMax = 10, + .inActive = 0, + .inMax = 0, + .connecting = 6, + .fixedConfigured = 2, + .fixedActive = 1, + .bootcache = 40, + .livecache = 12}; + + auto gauge = provider.meter()->CreateInt64ObservableGauge( + "peerfinder_slot_census", "PeerFinder slots, connection attempts and address caches"); + gauge->AddCallback( + [](opentelemetry::metrics::ObserverResult result, void* state) { + auto const* self = static_cast(state); + // Same single-label Observe() form the production callback uses. + auto observe = [&](char const* field, std::int64_t value) { + opentelemetry::nostd::get>>(result) + ->Observe(value, {{"metric", field}}); + }; + observe("out_active", self->outActive); + observe("out_max", self->outMax); + observe("in_active", self->inActive); + observe("in_max", self->inMax); + observe("connecting", self->connecting); + observe("fixed_configured", self->fixedConfigured); + observe("fixed_active", self->fixedActive); + observe("bootcache", self->bootcache); + observe("livecache", self->livecache); + }, + &observed); + + auto const dialling = provider.collect(); + + // Exactly nine series, one per `metric` value: all nine fields reach the + // exporter, not just the two the legacy insight gauges carried. + ASSERT_EQ(dialling.at("peerfinder_slot_census").size(), 9u); + // FAULT (a) "dialling but never completing": out_active below out_max WITH + // connecting non-zero. Without the attempt count this is indistinguishable + // from a node that is not dialling at all -- the capacity term alone says + // only "under-connected", never why. + EXPECT_EQ(gaugeValue(dialling, "peerfinder_slot_census", attrs("metric", "out_active")), 2); + EXPECT_EQ(gaugeValue(dialling, "peerfinder_slot_census", attrs("metric", "out_max")), 10); + EXPECT_EQ(gaugeValue(dialling, "peerfinder_slot_census", attrs("metric", "connecting")), 6); + + // FAULT (c) "configured fixed peer unreachable": fixed_active strictly below + // fixed_configured. 1 of 2 asked-for peers is connected. + EXPECT_EQ( + gaugeValue(dialling, "peerfinder_slot_census", attrs("metric", "fixed_configured")), 2); + EXPECT_EQ(gaugeValue(dialling, "peerfinder_slot_census", attrs("metric", "fixed_active")), 1); + + // Inbound disabled reads in_max=0, which makes in_active=0 a configuration + // fact rather than a fault -- the pair is what separates them. + EXPECT_EQ(gaugeValue(dialling, "peerfinder_slot_census", attrs("metric", "in_max")), 0); + EXPECT_EQ(gaugeValue(dialling, "peerfinder_slot_census", attrs("metric", "in_active")), 0); + + // Addresses ARE available here, so fault (b) is ruled out on this reading: + // the node has somewhere to dial and is still not completing. + EXPECT_EQ(gaugeValue(dialling, "peerfinder_slot_census", attrs("metric", "bootcache")), 40); + EXPECT_EQ(gaugeValue(dialling, "peerfinder_slot_census", attrs("metric", "livecache")), 12); + + // Exactly one label key on every series, and it is "metric": the nine + // fields form one fixed-cardinality group, not nine label dimensions. + for (auto const& [labels, point] : dialling.at("peerfinder_slot_census")) + { + ASSERT_EQ(labels.size(), 1u); + EXPECT_EQ(labels.begin()->first, "metric"); + } + // NEGATIVE: a `metric` value outside the production set of nine has no + // series, so the readings above are not an artifact of a catch-all series. + EXPECT_EQ(dialling.at("peerfinder_slot_census").count(attrs("metric", "out_count")), 0u); +} + +// FAULT (b) "nothing to dial", plus the idle-but-healthy reading. Separated from +// the fault-(a)/(c) test above to keep each function under the length limit. +TEST(MetricMacros, slot_census_gauge_reports_every_field_even_when_idle) +{ + CollectingProvider const provider; + + // A fresh node with no seed addresses at all: nothing dialled because there + // is nothing to dial. Distinct from fault (a), where dials are attempted. + PeerFinder::SlotCensus observed{ + .outActive = 0, + .outMax = 10, + .inActive = 0, + .inMax = 20, + .connecting = 0, + .fixedConfigured = 0, + .fixedActive = 0, + .bootcache = 0, + .livecache = 0}; + + auto gauge = provider.meter()->CreateInt64ObservableGauge( + "peerfinder_slot_census", "PeerFinder slots, connection attempts and address caches"); + gauge->AddCallback( + [](opentelemetry::metrics::ObserverResult result, void* state) { + auto const* self = static_cast(state); + auto observe = [&](char const* field, std::int64_t value) { + opentelemetry::nostd::get>>(result) + ->Observe(value, {{"metric", field}}); + }; + observe("out_active", self->outActive); + observe("out_max", self->outMax); + observe("in_active", self->inActive); + observe("in_max", self->inMax); + observe("connecting", self->connecting); + observe("fixed_configured", self->fixedConfigured); + observe("fixed_active", self->fixedActive); + observe("bootcache", self->bootcache); + observe("livecache", self->livecache); + }, + &observed); + + auto const nothingToDial = provider.collect(); + + // FAULT (b) "nothing to dial": both caches at exactly 0 while outbound + // capacity is available. The fix is [ips] / DNS, not the network. + EXPECT_EQ(gaugeValue(nothingToDial, "peerfinder_slot_census", attrs("metric", "bootcache")), 0); + EXPECT_EQ(gaugeValue(nothingToDial, "peerfinder_slot_census", attrs("metric", "livecache")), 0); + // connecting=0 here is what separates fault (b) from fault (a): no dial was + // even attempted, because there was no address to attempt. + EXPECT_EQ( + gaugeValue(nothingToDial, "peerfinder_slot_census", attrs("metric", "connecting")), 0); + EXPECT_EQ(gaugeValue(nothingToDial, "peerfinder_slot_census", attrs("metric", "out_max")), 10); + + // NEGATIVE: an idle-but-healthy node still reports EVERY field. A zero-valued + // field must be a present series, never an absent one -- absence would be + // indistinguishable from a dead exporter or a crashed callback. + observed = PeerFinder::SlotCensus{ + .outActive = 10, + .outMax = 10, + .inActive = 5, + .inMax = 20, + .connecting = 0, + .fixedConfigured = 0, + .fixedActive = 0, + .bootcache = 55, + .livecache = 30}; + auto const idle = provider.collect(); + + ASSERT_EQ(idle.at("peerfinder_slot_census").size(), 9u); + // The four fields that are legitimately 0 on a healthy node are each PRESENT + // with value 0, asserted one by one so a dropped series fails the test. + for (char const* field : {"connecting", "fixed_configured", "fixed_active"}) + { + ASSERT_EQ(idle.at("peerfinder_slot_census").count(attrs("metric", field)), 1u); + EXPECT_EQ(gaugeValue(idle, "peerfinder_slot_census", attrs("metric", field)), 0); + } + // Slots full: out_active has reached out_max, so nothing is dialling because + // nothing needs to be. Same connecting=0 as fault (b), opposite meaning -- + // only the capacity pair tells them apart. + EXPECT_EQ(gaugeValue(idle, "peerfinder_slot_census", attrs("metric", "out_active")), 10); + EXPECT_EQ(gaugeValue(idle, "peerfinder_slot_census", attrs("metric", "out_max")), 10); +} + +// amendment_block's whole value is the countdown, so the arithmetic is what this +// pins: the production callback computes max(expected - now, 0) in std::int64_t +// and observes -1 when firstUnsupportedExpected() is nullopt. All four states are +// asserted to EXACT values, including the two that a naive implementation gets +// wrong (the healthy sentinel, and the unsigned-subtraction wrap). +TEST(MetricMacros, amendment_block_gauge_observes_exact_countdown_and_sentinel) +{ + CollectingProvider const provider; + + // Two explicit epoch-second constants, so the expected difference is exact + // rather than approximate -- a clock read would make 7200 unassertable. + constexpr std::int64_t kNowEpochSeconds = 800'000'000; + constexpr std::int64_t kTwoHours = 7200; + + // Mirrors what the production callback reads: the warned flag from + // NetworkOPs::isAmendmentWarned(), and the optional activation time from + // AmendmentTable::firstUnsupportedExpected(). + struct Observed + { + bool warned{false}; + std::optional expectedEpochSeconds; + std::int64_t nowEpochSeconds{0}; + }; + // State (a): nothing pending at all -- the healthy case. + Observed observed{ + .warned = false, .expectedEpochSeconds = {}, .nowEpochSeconds = kNowEpochSeconds}; + + auto gauge = provider.meter()->CreateInt64ObservableGauge( + "amendment_block", "Amendment-block warning and seconds until the node stops validating"); + gauge->AddCallback( + [](opentelemetry::metrics::ObserverResult result, void* state) { + auto const* self = static_cast(state); + auto observe = [&](char const* field, std::int64_t value) { + opentelemetry::nostd::get>>(result) + ->Observe(value, {{"metric", field}}); + }; + observe("warned", self->warned ? 1 : 0); + + // The production arithmetic, reproduced exactly: -1 sentinel when + // nothing is pending, otherwise the difference taken in int64_t and + // clamped at 0. + std::int64_t secondsToBlock = -1; + if (self->expectedEpochSeconds) + { + secondsToBlock = + std::max(*self->expectedEpochSeconds - self->nowEpochSeconds, 0); + } + observe("seconds_to_block", secondsToBlock); + }, + &observed); + + auto const healthy = provider.collect(); + + // Exactly two series, one per `metric` value. + ASSERT_EQ(healthy.at("amendment_block").size(), 2u); + + // STATE (a) nothing pending: warned is exactly 0, and the countdown is + // exactly -1. Not 0 (which would read as "blocking right now", the most + // alarming possible value) and not absent (which a dashboard cannot tell + // from a stopped exporter). A distinct in-band sentinel is the only encoding + // that makes "healthy" assertable, matching unl_expiry_days' -1. + EXPECT_EQ(gaugeValue(healthy, "amendment_block", attrs("metric", "warned")), 0); + EXPECT_EQ(gaugeValue(healthy, "amendment_block", attrs("metric", "seconds_to_block")), -1); + ASSERT_EQ(healthy.at("amendment_block").count(attrs("metric", "seconds_to_block")), 1u); + + // STATE (b) pending two hours out: warned flips to exactly 1 and the + // countdown is exactly 7200, computed from the two constants above. + observed = Observed{ + .warned = true, + .expectedEpochSeconds = kNowEpochSeconds + kTwoHours, + .nowEpochSeconds = kNowEpochSeconds}; + auto const pending = provider.collect(); + EXPECT_EQ(gaugeValue(pending, "amendment_block", attrs("metric", "warned")), 1); + EXPECT_EQ(gaugeValue(pending, "amendment_block", attrs("metric", "seconds_to_block")), 7200); + + // Both series are present in both states, so the countdown never drops out + // when the warning flips. + ASSERT_EQ(pending.at("amendment_block").size(), 2u); +} + +// The clamp at the bottom of the countdown, and the deliberate absence of an +// amendment-id label. Split out of the test above to keep each function inside +// the length limit; it re-establishes the same callback shape. +TEST(MetricMacros, amendment_block_gauge_clamps_past_due_and_carries_no_amendment_id) +{ + CollectingProvider const provider; + + constexpr std::int64_t kNowEpochSeconds = 800'000'000; + constexpr std::int64_t kOneHour = 3600; + + struct Observed + { + bool warned{false}; + std::optional expectedEpochSeconds; + std::int64_t nowEpochSeconds{0}; + }; + // State (c): the activation time passed an hour ago. + Observed observed{ + .warned = true, + .expectedEpochSeconds = kNowEpochSeconds - kOneHour, + .nowEpochSeconds = kNowEpochSeconds}; + + auto gauge = provider.meter()->CreateInt64ObservableGauge( + "amendment_block", "Amendment-block warning and seconds until the node stops validating"); + gauge->AddCallback( + [](opentelemetry::metrics::ObserverResult result, void* state) { + auto const* self = static_cast(state); + auto observe = [&](char const* field, std::int64_t value) { + opentelemetry::nostd::get>>(result) + ->Observe(value, {{"metric", field}}); + }; + observe("warned", self->warned ? 1 : 0); + std::int64_t secondsToBlock = -1; + if (self->expectedEpochSeconds) + { + secondsToBlock = + std::max(*self->expectedEpochSeconds - self->nowEpochSeconds, 0); + } + observe("seconds_to_block", secondsToBlock); + }, + &observed); + + // STATE (c) past due by an hour: clamped to exactly 0, never negative. This + // is the assertion that proves the difference is taken in std::int64_t and + // NOT in NetClock's unsigned representation -- subtracting the time_points + // directly would wrap to roughly 4.29 billion and plot as 136 years away, + // turning the most urgent reading into the least. + auto const pastDue = provider.collect(); + EXPECT_EQ(gaugeValue(pastDue, "amendment_block", attrs("metric", "seconds_to_block")), 0); + EXPECT_EQ(gaugeValue(pastDue, "amendment_block", attrs("metric", "warned")), 1); + + // STATE (d) exactly at the boundary (expected == now): exactly 0. Pins the + // clamp's inclusive edge, so the transition into state (c) cannot skip a + // value or briefly emit -1. + observed = Observed{ + .warned = true, + .expectedEpochSeconds = kNowEpochSeconds, + .nowEpochSeconds = kNowEpochSeconds}; + auto const boundary = provider.collect(); + EXPECT_EQ(gaugeValue(boundary, "amendment_block", attrs("metric", "seconds_to_block")), 0); + + // NEGATIVE: there is NO label carrying the blocking amendment's id or hash. + // Exactly two series, and "metric" is the only label key. The identity is + // deliberately excluded: the network can vote on an arbitrary 256-bit + // amendment id, not just this build's known features, so an id label would + // be unbounded cardinality and would mint a permanent new series per + // amendment. The id is already in the log line, correlated by node and time. + ASSERT_EQ(boundary.at("amendment_block").size(), 2u); + for (auto const& [labels, point] : boundary.at("amendment_block")) + { + ASSERT_EQ(labels.size(), 1u); + EXPECT_EQ(labels.begin()->first, "metric"); + } + // An id-shaped label value has no series, so the two above are the whole set. + EXPECT_EQ(boundary.at("amendment_block").count(attrs("metric", "amendment_id")), 0u); +} + +// peer_disconnect_total is the signal that splits today's single unlabelled +// disconnect tally by cause AND direction. The series identity is the (reason, +// direction) PAIR: if it were not, a wave of our-fault backpressure on outbound +// links would be masked by ordinary inbound peer churn. +TEST(MetricMacros, peer_disconnect_total_keys_series_on_reason_and_direction_pair) +{ + CollectingProvider const provider; + FakeApp app; + wire(app, /*enabled=*/true, provider.meter()); + + // The exact call PeerImp::close() makes, once per teardown. + auto const bump = [&app](char const* reason, char const* direction) { + XRPL_METRIC_COUNTER_INC_LABELED( + app, + "peer_disconnect_total", + "Peer disconnects, by cause and connection direction", + {{"reason", std::string(reason)}, {"direction", std::string(direction)}}); + }; + + // Two OUR-FAULT reasons: this node could not keep up with what it owed the + // peer, or charged it off under its own resource pressure. + bump("large_sendq", "outbound"); + bump("charge_resources", "inbound"); + // Three NETWORK/TOPOLOGY reasons: the peer is on another chain, stopped + // answering, or the socket failed. + bump("not_useful", "outbound"); + bump("ping_timeout", "outbound"); + bump("read_error", "inbound"); + + auto const data = provider.collect(); + + // Five distinct (reason, direction) pairs -> exactly five series. Today all + // five collapse into one number, which is the defect this fixes. + ASSERT_EQ(data.at("peer_disconnect_total").size(), 5u); + + // Exact value 1 per distinct labelset. These two are also the proof that the + // labelsets do NOT collapse: each holds exactly 1 rather than one of them + // holding 2, so an our-fault outbound teardown and a network-fault inbound + // one stay two separate stories with two separate fixes. (counterValue() + // looks the key up with std::map::at, so a merged series fails here.) + EXPECT_EQ( + counterValue( + data, "peer_disconnect_total", attrs("reason", "large_sendq", "direction", "outbound")), + 1); + EXPECT_EQ( + counterValue( + data, "peer_disconnect_total", attrs("reason", "read_error", "direction", "inbound")), + 1); + + // Our-fault reasons are individually addressable, so "the node is shedding + // its own peers" is readable without reading logs. + EXPECT_EQ( + counterValue( + data, + "peer_disconnect_total", + attrs("reason", "charge_resources", "direction", "inbound")), + 1); + // Network/topology reasons stay distinct from each other too: a chain split + // ("not_useful") is not a dead link ("ping_timeout"). + EXPECT_EQ( + counterValue( + data, "peer_disconnect_total", attrs("reason", "not_useful", "direction", "outbound")), + 1); + EXPECT_EQ( + counterValue( + data, + "peer_disconnect_total", + attrs("reason", "ping_timeout", "direction", "outbound")), + 1); + + // Exactly two label keys on every series, and exactly these two: a third + // (a peer address, say) would be unbounded cardinality. + for (auto const& [labels, point] : data.at("peer_disconnect_total")) + { + ASSERT_EQ(labels.size(), 2u); + EXPECT_EQ(labels.count("reason"), 1u); + EXPECT_EQ(labels.count("direction"), 1u); + } +} + +// The `direction` label must genuinely participate in the series key, not just +// ride along: the SAME reason seen on both directions has to produce two series +// of 1 rather than one series of 2. Split out of the test above to keep each +// function inside the length limit. +TEST(MetricMacros, peer_disconnect_total_does_not_merge_directions_for_one_reason) +{ + CollectingProvider const provider; + FakeApp app; + wire(app, /*enabled=*/true, provider.meter()); + + auto const bump = [&app](char const* reason, char const* direction) { + XRPL_METRIC_COUNTER_INC_LABELED( + app, + "peer_disconnect_total", + "Peer disconnects, by cause and connection direction", + {{"reason", std::string(reason)}, {"direction", std::string(direction)}}); + }; + + // One read_error each way. If direction did not key the series, this would + // collapse to a single series holding 2. + bump("read_error", "inbound"); + bump("read_error", "outbound"); + + auto const data = provider.collect(); + + // Two series, not one: the same cause on opposite directions stays split. + ASSERT_EQ(data.at("peer_disconnect_total").size(), 2u); + EXPECT_EQ( + counterValue( + data, "peer_disconnect_total", attrs("reason", "read_error", "direction", "inbound")), + 1); + EXPECT_EQ( + counterValue( + data, "peer_disconnect_total", attrs("reason", "read_error", "direction", "outbound")), + 1); + + // Bumping ONE direction advances only that series. This is the assertion that + // would fail if the labels were merged: inbound must stay at exactly 1. + bump("read_error", "outbound"); + auto const second = provider.collect(); + EXPECT_EQ( + counterValue( + second, + "peer_disconnect_total", + attrs("reason", "read_error", "direction", "outbound")), + 2); + EXPECT_EQ( + counterValue( + second, "peer_disconnect_total", attrs("reason", "read_error", "direction", "inbound")), + 1); + EXPECT_EQ(second.at("peer_disconnect_total").size(), 2u); + + // NEGATIVE: a reason outside the fixed literal set in PeerImp.cpp has no + // series, so the counts above are not an artifact of a catch-all series. + EXPECT_EQ( + second.at("peer_disconnect_total") + .count(attrs("reason", "disconnected", "direction", "inbound")), + 0u); + // NEGATIVE: a direction value outside {inbound, outbound} has no series. + EXPECT_EQ( + second.at("peer_disconnect_total") + .count(attrs("reason", "read_error", "direction", "unknown")), + 0u); +} + +// peer_accept_total is the inbound twin of the existing +// overlay_connect_total{outcome}: that one counts OUTBOUND dials this node +// makes, this one counts INBOUND attempts it receives. Together they give the +// full in/out split, which neither provides alone -- a node whose outbound dials +// all succeed while every inbound attempt is refused looks perfectly healthy on +// overlay_connect_total by itself. +TEST(MetricMacros, peer_accept_total_keys_series_on_outcome) +{ + CollectingProvider const provider; + FakeApp app; + wire(app, /*enabled=*/true, provider.meter()); + + // The exact call OverlayImpl::reportAcceptOutcome() makes, once per + // terminal outcome of one inbound attempt. + auto const bump = [&app](char const* outcome) { + XRPL_METRIC_COUNTER_INC_LABELED( + app, + "peer_accept_total", + "Inbound peer connection attempts, by terminal outcome", + {{"outcome", std::string(outcome)}}); + }; + + // accepted x2, no_slot x3, handshake_error x1 -- three distinct + // multiplicities so no two series can be confused with each other. + bump("accepted"); + bump("accepted"); + bump("no_slot"); + bump("no_slot"); + bump("no_slot"); + bump("handshake_error"); + + auto const data = provider.collect(); + + // Three distinct outcomes stay three distinct series with exact values. + ASSERT_EQ(data.at("peer_accept_total").size(), 3u); + EXPECT_EQ(counterValue(data, "peer_accept_total", attrs("outcome", "accepted")), 2); + // "no_slot" is capacity, "handshake_error" is a protocol or crypto failure. + // Collapsed into one number they would be indistinguishable, yet the first + // is fixed by configuration and the second by investigation. + EXPECT_EQ(counterValue(data, "peer_accept_total", attrs("outcome", "no_slot")), 3); + EXPECT_EQ(counterValue(data, "peer_accept_total", attrs("outcome", "handshake_error")), 1); + + // Exactly one label key, and it is "outcome". + auto const& firstKey = data.at("peer_accept_total").begin()->first; + ASSERT_EQ(firstKey.size(), 1u); + EXPECT_EQ(firstKey.begin()->first, "outcome"); + + // NEGATIVE: an outcome from the production set that was not emitted here has + // no series, proving outcomes are not being merged into a catch-all. + EXPECT_EQ(data.at("peer_accept_total").count(attrs("outcome", "resource_limit")), 0u); + + // Accumulates rather than replaces: the reader is cumulative, so a second + // acceptance advances the existing series to exactly 3. + bump("accepted"); + EXPECT_EQ( + counterValue(provider.collect(), "peer_accept_total", attrs("outcome", "accepted")), 3); +} + +// serve_refused_total measures the SUPPLY side: what this node refuses to serve +// its peers. Nothing measured it before, so a node shedding every ledger request +// looked identical to one being asked for nothing. The series identity is the +// (request, reason) PAIR, because the same reason means different things on +// different request kinds. +TEST(MetricMacros, serve_refused_total_keys_series_on_request_and_reason_pair) +{ + CollectingProvider const provider; + FakeApp app; + wire(app, /*enabled=*/true, provider.meter()); + + // The exact call PeerImp::reportServeRefusal() makes, once per refused + // request. + auto const bump = [&app](char const* request, char const* reason) { + XRPL_METRIC_COUNTER_INC_LABELED( + app, + "serve_refused_total", + "Peer data requests this node declined to serve, by request kind and cause", + {{"request", std::string(request)}, {"reason", std::string(reason)}}); + }; + + // Same request kind, two different reasons -> two series. + bump("ledger", "sendq_full"); + bump("ledger", "sendq_full"); + bump("ledger", "not_found"); + // Different request kinds -> their own series, even sharing a reason. + bump("fetchpack", "load_shed"); + bump("fetchpack", "load_shed"); + bump("fetchpack", "load_shed"); + bump("txset", "not_found"); + + auto const data = provider.collect(); + + // Four distinct (request, reason) pairs -> exactly four series. + ASSERT_EQ(data.at("serve_refused_total").size(), 4u); + + // SELF-INFLICTED BACKPRESSURE: "sendq_full" and "load_shed" both mean this + // node chose not to answer because it was already behind. The fix is local + // (capacity, tuning), and the refusal directly slows the asking peer's sync. + EXPECT_EQ( + counterValue( + data, "serve_refused_total", attrs("request", "ledger", "reason", "sendq_full")), + 2); + EXPECT_EQ( + counterValue( + data, "serve_refused_total", attrs("request", "fetchpack", "reason", "load_shed")), + 3); + + // HISTORY GAP: "not_found" is not backpressure at all -- the node simply + // does not hold what was asked for. Same counter, completely different + // meaning and a completely different fix (history configuration), which is + // why it must not share a series with the two above. + EXPECT_EQ( + counterValue( + data, "serve_refused_total", attrs("request", "ledger", "reason", "not_found")), + 1); + EXPECT_EQ( + counterValue(data, "serve_refused_total", attrs("request", "txset", "reason", "not_found")), + 1); + + // The same reason on two different request kinds stays two series -- both + // present, each holding its own 1: a txset miss (consensus proposal data) + // and a ledger miss (history) are unrelated faults despite the shared slug. + EXPECT_EQ( + data.at("serve_refused_total").count(attrs("request", "ledger", "reason", "not_found")), + 1u); + EXPECT_EQ( + data.at("serve_refused_total").count(attrs("request", "txset", "reason", "not_found")), 1u); + + // Exactly two label keys on every series, and exactly these two. + for (auto const& [labels, point] : data.at("serve_refused_total")) + { + ASSERT_EQ(labels.size(), 2u); + EXPECT_EQ(labels.count("request"), 1u); + EXPECT_EQ(labels.count("reason"), 1u); + } + + // NEGATIVE: a pair that was never emitted has no series, so the four counts + // above are not an artifact of a catch-all series. + EXPECT_EQ( + data.at("serve_refused_total").count(attrs("request", "object", "reason", "sendq_full")), + 0u); +} + +// ledger_jump_total counts a node discarding its own chain tip to follow the +// network. It is deliberately UNLABELLED, so the single series' key is the empty +// attribute set -- and repeated jumps must accumulate on it, because a node +// thrashing between chains is the pattern worth alerting on and a single jump is +// not. +TEST(MetricMacros, ledger_jump_total_accumulates_on_one_unlabelled_series) +{ + CollectingProvider const provider; + FakeApp app; + wire(app, /*enabled=*/true, provider.meter()); + + // The exact call NetworkOPsImp::switchLastClosedLedger() makes. Three jumps + // from the same call site, as a thrashing node would produce. + for (int i = 0; i < 3; ++i) + { + XRPL_METRIC_COUNTER_INC( + app, + "ledger_jump_total", + "Forced jumps of the last closed ledger to a divergent chain"); + } + + auto const data = provider.collect(); + + // Exactly ONE series, and its key is the empty attribute set: the metric + // carries no labels at all. + ASSERT_EQ(data.at("ledger_jump_total").size(), 1u); + ASSERT_EQ(data.at("ledger_jump_total").count(otel_sdk::PointAttributes{}), 1u); + + // Three jumps accumulate to exactly 3 on that one series -- the reader is + // cumulative, so this is a running total, not a per-interval delta. + EXPECT_EQ(counterValue(data, "ledger_jump_total", otel_sdk::PointAttributes{}), 3); + + // ZERO labels, asserted on the series key itself. Both candidate labels were + // deliberately rejected as unbounded: the divergent ledger's hash is a + // 256-bit value and its sequence grows without limit, so either would mint a + // permanent new series per jump. The JLOG error line immediately above the + // emit already carries both, correlated to this series by node and time. + EXPECT_TRUE(data.at("ledger_jump_total").begin()->first.empty()); + EXPECT_EQ(data.at("ledger_jump_total").begin()->first.size(), 0u); + + // A fourth jump advances the SAME series to exactly 4 rather than creating a + // second one, which is what "no labels" has to mean over time. + XRPL_METRIC_COUNTER_INC( + app, "ledger_jump_total", "Forced jumps of the last closed ledger to a divergent chain"); + auto const fourth = provider.collect(); + ASSERT_EQ(fourth.at("ledger_jump_total").size(), 1u); + EXPECT_EQ(counterValue(fourth, "ledger_jump_total", otel_sdk::PointAttributes{}), 4); +} + +// RUNTIME-DISABLED no-op proof for all four WP-A7 counters: with the registry +// disabled, every one emits NOTHING -- no series at all, and meter() is never +// consulted, so not even an instrument was created. Proves the isEnabled() gate +// short-circuits BEFORE any SDK work, which is what makes these emits free on a +// node with telemetry turned off. +TEST(MetricMacros, sync_supply_counters_emit_nothing_when_registry_disabled) +{ + CollectingProvider const provider; + FakeApp app; + wire(app, /*enabled=*/false, provider.meter()); + + XRPL_METRIC_COUNTER_INC_LABELED( + app, + "peer_disconnect_total", + "Peer disconnects, by cause and connection direction", + {{"reason", std::string("large_sendq")}, {"direction", std::string("outbound")}}); + XRPL_METRIC_COUNTER_INC_LABELED( + app, + "peer_accept_total", + "Inbound peer connection attempts, by terminal outcome", + {{"outcome", std::string("accepted")}}); + XRPL_METRIC_COUNTER_INC_LABELED( + app, + "serve_refused_total", + "Peer data requests this node declined to serve, by request kind and cause", + {{"request", std::string("ledger")}, {"reason", std::string("sendq_full")}}); + XRPL_METRIC_COUNTER_INC( + app, "ledger_jump_total", "Forced jumps of the last closed ledger to a divergent chain"); + + auto const data = provider.collect(); + + // Total absence, not zero-valued series: the instruments never existed. + EXPECT_EQ(data.count("peer_disconnect_total"), 0u); + EXPECT_EQ(data.count("peer_accept_total"), 0u); + EXPECT_EQ(data.count("serve_refused_total"), 0u); + EXPECT_EQ(data.count("ledger_jump_total"), 0u); + EXPECT_EQ(data.size(), 0u); + + // Cause, not just state: the isEnabled() gate short-circuited before the + // macros asked for a meter, so no instrument was created either. + EXPECT_EQ(app.registry().meterCalls(), 0); +} + +// ----------------------------------------------------------------- +// WP-A6: back-fill and persistence sync diagnostics +// ----------------------------------------------------------------- + +// Replay falling back to a full ledger acquire silently defeats the replay +// optimisation. The `stage` label must keep the skip-list and delta stages +// apart, because they fail for different reasons. +TEST(MetricMacros, ledger_replay_fallback_counter_separates_stages_by_exact_count) +{ + CollectingProvider const provider; + FakeApp app; + wire(app, /*enabled=*/true, provider.meter()); + + // The skip-list stage falls back twice, the delta stage once: distinct + // counts so a collapsed label set cannot coincidentally look correct. + for (int i = 0; i < 2; ++i) + { + XRPL_METRIC_COUNTER_INC_LABELED( + app, + "ledger_replay_fallback_total", + "Replay sub-acquires that fell back to a full ledger acquire", + {{"stage", std::string("skiplist")}}); + } + XRPL_METRIC_COUNTER_INC_LABELED( + app, + "ledger_replay_fallback_total", + "Replay sub-acquires that fell back to a full ledger acquire", + {{"stage", std::string("delta")}}); + + auto const data = provider.collect(); + + // Exactly two series, and each carries its own exact total. + ASSERT_EQ(data.at("ledger_replay_fallback_total").size(), 2u); + EXPECT_EQ(counterValue(data, "ledger_replay_fallback_total", attrs("stage", "skiplist")), 2); + EXPECT_EQ(counterValue(data, "ledger_replay_fallback_total", attrs("stage", "delta")), 1); + + // NEGATIVE: a stage that never fell back has no series at all. An absent + // series, not a zero, is what a healthy replay path looks like. + EXPECT_EQ(data.at("ledger_replay_fallback_total").count(attrs("stage", "txset")), 0u); + + // The label key is exactly `stage` and is the only label, so this stays a + // bounded two-value group. + auto const& firstKey = data.at("ledger_replay_fallback_total").begin()->first; + ASSERT_EQ(firstKey.size(), 1u); + EXPECT_EQ(firstKey.begin()->first, "stage"); +} + +// The replay task's four terminal states must be four distinct series: a +// timeout and a failed build are diagnosed differently, and a success total +// with no failures is the only healthy reading. +TEST(MetricMacros, ledger_replay_outcome_counter_records_each_terminal_state_exactly) +{ + CollectingProvider const provider; + FakeApp app; + wire(app, /*enabled=*/true, provider.meter()); + + // Distinct counts per outcome, matching the four terminal sites in + // LedgerReplayTask: success, timeout, build_failed, parameter_failed. + auto const record = [&app](char const* outcome, int times) { + for (int i = 0; i < times; ++i) + { + XRPL_METRIC_COUNTER_INC_LABELED( + app, + "ledger_replay_outcome_total", + "Ledger replay tasks by terminal outcome", + {{"outcome", std::string(outcome)}}); + } + }; + record("success", 3); + record("timeout", 2); + record("build_failed", 1); + record("parameter_failed", 4); + + auto const data = provider.collect(); + + ASSERT_EQ(data.at("ledger_replay_outcome_total").size(), 4u); + EXPECT_EQ(counterValue(data, "ledger_replay_outcome_total", attrs("outcome", "success")), 3); + EXPECT_EQ(counterValue(data, "ledger_replay_outcome_total", attrs("outcome", "timeout")), 2); + EXPECT_EQ( + counterValue(data, "ledger_replay_outcome_total", attrs("outcome", "build_failed")), 1); + EXPECT_EQ( + counterValue(data, "ledger_replay_outcome_total", attrs("outcome", "parameter_failed")), 4); + + // NEGATIVE: an outcome value the code never emits has no series, proving + // the four above are real label values and not a catch-all. + EXPECT_EQ(data.at("ledger_replay_outcome_total").count(attrs("outcome", "cancelled")), 0u); + + // The two WP-A6 replay counters are separate instruments, so a fallback + // never inflates an outcome total. + EXPECT_EQ(data.count("ledger_replay_fallback_total"), 0u); +} + +// Telemetry disabled at runtime: both replay counters must be complete no-ops. +// Absence of the instrument, not a zero-valued series. +TEST(MetricMacros, ledger_replay_counters_emit_nothing_when_disabled) +{ + CollectingProvider const provider; + FakeApp app; + wire(app, /*enabled=*/false, provider.meter()); + + XRPL_METRIC_COUNTER_INC_LABELED( + app, + "ledger_replay_fallback_total", + "Replay sub-acquires that fell back to a full ledger acquire", + {{"stage", std::string("skiplist")}}); + XRPL_METRIC_COUNTER_INC_LABELED( + app, + "ledger_replay_outcome_total", + "Ledger replay tasks by terminal outcome", + {{"outcome", std::string("timeout")}}); + + auto const data = provider.collect(); + + EXPECT_EQ(data.count("ledger_replay_fallback_total"), 0u); + EXPECT_EQ(data.count("ledger_replay_outcome_total"), 0u); + EXPECT_EQ(data.size(), 0u); + + // Cause, not just state: the isEnabled() gate short-circuited before the + // macros ever asked for a meter, so no instrument was created. + EXPECT_EQ(app.registry().meterCalls(), 0); +} + +// The nodestore_latency gauge derives a mean from two cumulative totals the +// node store already keeps. This mirrors the production callback in +// MetricsRegistry::registerNodeStoreLatencyGauge, whose enabled path cannot be +// linked into this binary, so the derivation is asserted here against the same +// four inputs. +TEST(MetricMacros, nodestore_latency_gauge_observes_exact_derived_means) +{ + CollectingProvider const provider; + + // The four totals the production callback reads, chosen so each mean + // divides exactly and the two means differ: writes are 4x slower per + // operation than reads, which is the "existing DB back-fills slowly" + // shape this signal exists to show. + struct NodeStoreTotals + { + std::uint64_t storeCount; + std::uint64_t storeDurationUs; + std::uint64_t fetchCount; + std::uint64_t fetchDurationUs; + }; + NodeStoreTotals totals{ + .storeCount = 500, + .storeDurationUs = 2'000'000, // 2 s over 500 stores -> 4000 us + .fetchCount = 1000, + .fetchDurationUs = 1'000'000}; // 1 s over 1000 fetches -> 1000 us + + auto gauge = provider.meter()->CreateInt64ObservableGauge( + "nodestore_latency", "NodeStore mean store/fetch latency in microseconds, with counts"); + gauge->AddCallback( + [](opentelemetry::metrics::ObserverResult result, void* state) { + auto const* self = static_cast(state); + auto observe = [&](char const* field, std::int64_t value) { + opentelemetry::nostd::get>>(result) + ->Observe(value, {{"metric", field}}); + }; + observe("write_count", static_cast(self->storeCount)); + observe("read_count", static_cast(self->fetchCount)); + if (self->storeCount > 0 && self->storeDurationUs > 0) + { + observe( + "write_mean_us", + static_cast(self->storeDurationUs / self->storeCount)); + } + if (self->fetchCount > 0 && self->fetchDurationUs > 0) + { + observe( + "read_mean_us", + static_cast(self->fetchDurationUs / self->fetchCount)); + } + }, + &totals); + + auto const busy = provider.collect(); + + // Exactly four series: two means and the two denominators that let a + // dashboard recover interval latency from these cumulative totals. + ASSERT_EQ(busy.at("nodestore_latency").size(), 4u); + EXPECT_EQ(gaugeValue(busy, "nodestore_latency", attrs("metric", "write_mean_us")), 4000); + EXPECT_EQ(gaugeValue(busy, "nodestore_latency", attrs("metric", "read_mean_us")), 1000); + EXPECT_EQ(gaugeValue(busy, "nodestore_latency", attrs("metric", "write_count")), 500); + EXPECT_EQ(gaugeValue(busy, "nodestore_latency", attrs("metric", "read_count")), 1000); + + // The write mean is the new signal, and it must be legible next to the + // read mean rather than merely present. + EXPECT_GT( + gaugeValue(busy, "nodestore_latency", attrs("metric", "write_mean_us")), + gaugeValue(busy, "nodestore_latency", attrs("metric", "read_mean_us"))); + + // Single fixed-cardinality label group, keyed exactly `metric`. + auto const& firstKey = busy.at("nodestore_latency").begin()->first; + ASSERT_EQ(firstKey.size(), 1u); + EXPECT_EQ(firstKey.begin()->first, "metric"); + + // EDGE CASE: a node that has never written. The zero denominator must skip + // the mean rather than divide by zero, while the count is still reported -- + // that is what distinguishes "nothing written yet" from "writes are + // instant". The read side is unaffected and still reports both. + totals = NodeStoreTotals{ + .storeCount = 0, .storeDurationUs = 0, .fetchCount = 4, .fetchDurationUs = 800}; + auto const idle = provider.collect(); + + EXPECT_EQ(idle.at("nodestore_latency").count(attrs("metric", "write_mean_us")), 0u); + EXPECT_EQ(gaugeValue(idle, "nodestore_latency", attrs("metric", "write_count")), 0); + EXPECT_EQ(gaugeValue(idle, "nodestore_latency", attrs("metric", "read_mean_us")), 200); + EXPECT_EQ(gaugeValue(idle, "nodestore_latency", attrs("metric", "read_count")), 4); + + // EDGE CASE: integer division truncates rather than rounding. 7 stores + // over 100 us is 14.28 us, reported as 14 -- asserted so a future change + // to floating point is a deliberate, visible decision. + totals = NodeStoreTotals{ + .storeCount = 7, .storeDurationUs = 100, .fetchCount = 0, .fetchDurationUs = 0}; + auto const truncating = provider.collect(); + + EXPECT_EQ(gaugeValue(truncating, "nodestore_latency", attrs("metric", "write_mean_us")), 14); + // The read side now has the zero denominator, so its mean drops out too. + EXPECT_EQ(truncating.at("nodestore_latency").count(attrs("metric", "read_mean_us")), 0u); + EXPECT_EQ(gaugeValue(truncating, "nodestore_latency", attrs("metric", "read_count")), 0); + + // EDGE CASE, and the one that matters most on a real node: stores were + // counted but never TIMED. Database::store() is pure virtual and only the + // paths calling recordStoreDuration() contribute a numerator, so a node + // whose concrete store override does not time itself has a non-zero count + // with a zero duration. The mean must be OMITTED, not reported as 0 -- + // a 0 would read as "writes are instantaneous", which is worse than a + // visible gap. This assertion is the guard on that choice. + totals = NodeStoreTotals{ + .storeCount = 9000, .storeDurationUs = 0, .fetchCount = 10, .fetchDurationUs = 50}; + auto const untimed = provider.collect(); + + EXPECT_EQ(untimed.at("nodestore_latency").count(attrs("metric", "write_mean_us")), 0u); + // The count is still published, so the gap is visible rather than silent: + // a panel shows real write throughput with no latency line beside it. + EXPECT_EQ(gaugeValue(untimed, "nodestore_latency", attrs("metric", "write_count")), 9000); + // The read side is independent and unaffected by the write-side gap. + EXPECT_EQ(gaugeValue(untimed, "nodestore_latency", attrs("metric", "read_mean_us")), 5); + // Exactly three series: both counts plus the one mean that is derivable. + EXPECT_EQ(untimed.at("nodestore_latency").size(), 3u); +} + #endif // XRPL_ENABLE_TELEMETRY diff --git a/src/tests/libxrpl/telemetry/MetricsRegistry.cpp b/src/tests/libxrpl/telemetry/MetricsRegistry.cpp index 862d46faef..8453e191e4 100644 --- a/src/tests/libxrpl/telemetry/MetricsRegistry.cpp +++ b/src/tests/libxrpl/telemetry/MetricsRegistry.cpp @@ -9,7 +9,8 @@ * - Destructor handles cleanup without crash. * - Compile-time-disabled proof for the sync-diagnostics gauges: the whole * async-gauge registration surface is compiled away, and a full disabled - * lifecycle never touches any ServiceRegistry service. + * lifecycle never touches any ServiceRegistry service -- including the + * Overlay and AmendmentTable the WP-A7 gauges would read. * * NOTE: These tests only exercise the no-op path (telemetry disabled). * When XRPL_ENABLE_TELEMETRY is defined, MetricsRegistry.cpp pulls in @@ -19,7 +20,8 @@ * CONSEQUENCE for the sync-diagnostics gauges (`unl_quorum`, * `clock_close_offset_seconds`, `sync_state`, * `server_stall_events_total`, `sync_acquire`, `shamap_cache_hit_rate`, - * `jobq_backlog`, `jobq_saturation`): + * `jobq_backlog`, `jobq_saturation`, `peer_ledger_supply`, + * `peerfinder_slot_census`, `amendment_block`, `nodestore_latency`): * this file CANNOT assert an observed gauge * value, because on this build the gauges do not exist -- their registration * methods and the OTel instrument members are inside @@ -27,8 +29,10 @@ * is provable here, and what the tests below assert, is the complementary * half: that nothing is registered and no service is consulted. The exact * observed values (trusted_keys=5, quorum=4, offset=-3, the sync_state / - * stall-episode values, the acquire-progress / cache-hit-rate values, and the - * per-type backlog / pool-saturation values) are + * stall-episode values, the acquire-progress / cache-hit-rate values, the + * per-type backlog / pool-saturation values, the peer-supply / + * slot-census / amendment-countdown values, and the nodestore + * read/write mean-latency values) are * asserted in MetricMacros.cpp, which is the file compiled when telemetry IS * enabled. */ @@ -385,10 +389,14 @@ TEST_F(MetricsRegistryTest, destructor_calls_stop) // `server_stall_events_total` read NetworkOPs and LoadManager; `sync_acquire` // reads InboundLedgers::acquireProgress() and `shamap_cache_hit_rate` reads the // node Family's tree-node cache; `jobq_backlog` and `jobq_saturation` read -// JobQueue::getJobTypeCounts() / getWorkerSaturation(). All are +// JobQueue::getJobTypeCounts() / getWorkerSaturation(); `peer_ledger_supply` and +// `peerfinder_slot_census` read Overlay::getPeerLedgerSupply() / +// getSlotCensus() and `amendment_block` reads +// AmendmentTable::firstUnsupportedExpected(). All are // reached through the ServiceRegistry, and MockServiceRegistry::getValidators() // / getTimeKeeper() / getOPs() / getLoadManager() / getInboundLedgers() / -// getNodeFamily() / getJobQueue() THROW std::logic_error. So "no +// getNodeFamily() / getJobQueue() / getOverlay() / getAmendmentTable() THROW +// std::logic_error. So "no // gauge callback ran" is directly observable here: had registerAsyncGauges() run // and had a callback fired, one of those accessors would have thrown. // @@ -429,7 +437,9 @@ TEST_F(MetricsRegistryTest, disabled_lifecycle_never_consults_gauge_services) // registerUnlQuorumGauge() / registerClockSkewGauge() / // registerSyncStateGauge() / registerStallEventsCounter() / // registerSyncAcquireGauge() / registerCacheHitRateDetailGauge() / - // registerJobQueueBacklogGauge() / registerJobQueueSaturationGauge() -- + // registerJobQueueBacklogGauge() / registerJobQueueSaturationGauge() / + // registerPeerLedgerSupplyGauge() / registerSlotCensusGauge() / + // registerAmendmentBlockGauge() / registerNodeStoreLatencyGauge() -- // would run. EXPECT_NO_THROW(registry.start("http://localhost:4318/v1/metrics")); @@ -464,6 +474,25 @@ TEST_F(MetricsRegistryTest, disabled_lifecycle_never_consults_gauge_services) // both on the JobQueue. Neither was consulted above, so neither gauge // took the JobQueue mutex on a telemetry-off build. EXPECT_THROW(mockApp_.getJobQueue(), std::logic_error); + // The service both WP-A7 peer gauges read: peer_ledger_supply polls + // getPeerLedgerSupply(), which walks the active-peer list, and + // peerfinder_slot_census polls getSlotCensus(), which takes the PeerFinder + // lock. Both go through the Overlay, so a single throw here proves neither + // gauge walked the peer list nor took the PeerFinder lock on a + // telemetry-off build. + EXPECT_THROW(mockApp_.getOverlay(), std::logic_error); + // The service the WP-A7 amendment countdown reads: amendment_block polls + // firstUnsupportedExpected() on the AmendmentTable, which takes that + // table's mutex. Not consulted above, so the countdown never ran. (Its + // `warned` half reads NetworkOPs, already covered by the getOPs() check.) + EXPECT_THROW(mockApp_.getAmendmentTable(), std::logic_error); + // The service the WP-A6 nodestore latency gauge reads: nodestore_latency + // polls getStoreDurationUs()/getStoreCount() and + // getFetchDurationUs()/getFetchTotalCount() on the node-store Database. + // Not consulted above, so the latency gauge never read those atomics on a + // telemetry-off build. (The existing nodestore_state gauge reads the same + // service, so this single throw covers both.) + EXPECT_THROW(mockApp_.getNodeStore(), std::logic_error); } // Even asking for enabled=true registers no sync-diagnostics gauge on a @@ -484,9 +513,12 @@ TEST_F(MetricsRegistryTest, enabled_flag_alone_registers_no_gauges_when_compiled // registered registerUnlQuorumGauge()/registerClockSkewGauge()/ // registerSyncStateGauge()/registerStallEventsCounter()/ // registerSyncAcquireGauge()/registerCacheHitRateDetailGauge()/ - // registerJobQueueBacklogGauge()/registerJobQueueSaturationGauge(), a + // registerJobQueueBacklogGauge()/registerJobQueueSaturationGauge()/ + // registerPeerLedgerSupplyGauge()/registerSlotCensusGauge()/ + // registerAmendmentBlockGauge()/registerNodeStoreLatencyGauge(), a // callback would reach getValidators()/getTimeKeeper()/getOPs()/ - // getLoadManager()/getInboundLedgers()/getNodeFamily()/getJobQueue() and + // getLoadManager()/getInboundLedgers()/getNodeFamily()/getJobQueue()/ + // getOverlay()/getAmendmentTable()/getNodeStore() and // throw std::logic_error. EXPECT_NO_THROW(enabledRequest.start("http://localhost:4318/v1/metrics")); EXPECT_NO_THROW(enabledRequest.detachCallbacks()); diff --git a/src/xrpld/telemetry/MetricsRegistry.cpp b/src/xrpld/telemetry/MetricsRegistry.cpp index f6c684628c..3efec9dcb5 100644 --- a/src/xrpld/telemetry/MetricsRegistry.cpp +++ b/src/xrpld/telemetry/MetricsRegistry.cpp @@ -41,6 +41,7 @@ #include #include #include +#include #include #include #include @@ -494,6 +495,11 @@ MetricsRegistry::registerAsyncGauges() registerCacheHitRateDetailGauge(); registerJobQueueBacklogGauge(); registerJobQueueSaturationGauge(); + registerPeerLedgerSupplyGauge(); + registerSlotCensusGauge(); + registerAmendmentBlockGauge(); + registerNodeStoreLatencyGauge(); + registerLedgerQuorumPublishGauge(); } void @@ -1818,6 +1824,296 @@ MetricsRegistry::registerJobQueueSaturationGauge() this); } +void +MetricsRegistry::registerPeerLedgerSupplyGauge() +{ + // --- Sync diagnostics: can the network even serve what I need? --- + // Each peer advertises its ledger range and the connection caches it, but + // nothing ever compared those ranges, so "no peer holds the sequence I + // want" looked exactly like "my peers are slow" -- two faults with + // completely different fixes. + peerLedgerSupplyGauge_ = meter_->CreateInt64ObservableGauge( + "peer_ledger_supply", "Peer coverage of the ledger sequence this node needs"); + peerLedgerSupplyGauge_->AddCallback( + [](opentelemetry::metrics::ObserverResult result, void* state) { + auto* self = static_cast(state); + if (self->callbacksDetached_.load(std::memory_order_acquire)) + return; + auto& app = self->app_; + + try + { + auto observe = [&](char const* field, int64_t value) { + opentelemetry::nostd::get>>(result) + ->Observe(value, {{"metric", field}}); + }; + + // One pass over the peers, so all five series describe the + // same peer set at the same instant. + auto const supply = app.getOverlay().getPeerLedgerSupply( + app.getLedgerMaster().getValidLedgerIndex()); + + // The denominator. Zero serving out of zero reporting is + // silence; zero out of many is a real supply gap. + observe("peers_reporting", supply.peersReporting); + observe("peers_serving_validated", supply.peersServingValidated); + + // The verdict: zero here while peers_reporting is non-zero + // means waiting cannot finish the sync. + observe("peers_serving_next", supply.peersServingNext); + + // The window the peer set covers, so an operator can tell a + // request for discarded history from one for an unreached tip. + observe("supply_min_seq", supply.supplyMinSeq); + observe("supply_max_seq", supply.supplyMaxSeq); + } + catch (...) // NOLINT(bugprone-empty-catch) + { + // Silently skip if services are not yet ready. + } + }, + this); +} + +void +MetricsRegistry::registerSlotCensusGauge() +{ + // --- Sync diagnostics: why can this node not get peers? --- + // All nine numbers already exist inside PeerFinder; only the two active + // counts are exported today, which cannot distinguish "not dialling", + // "dialling and failing" and "nothing to dial". + slotCensusGauge_ = meter_->CreateInt64ObservableGauge( + "peerfinder_slot_census", "PeerFinder slots, connection attempts and address caches"); + slotCensusGauge_->AddCallback( + [](opentelemetry::metrics::ObserverResult result, void* state) { + auto* self = static_cast(state); + if (self->callbacksDetached_.load(std::memory_order_acquire)) + return; + auto& app = self->app_; + + try + { + auto observe = [&](char const* field, int64_t value) { + opentelemetry::nostd::get>>(result) + ->Observe(value, {{"metric", field}}); + }; + + // One snapshot under one PeerFinder lock acquire, so occupancy + // and capacity can be compared against each other. + auto const census = app.getOverlay().getSlotCensus(); + + observe("out_active", census.outActive); + observe("out_max", census.outMax); + observe("in_active", census.inActive); + observe("in_max", census.inMax); + + // Dials in flight. Non-zero while out_active stays under + // out_max is the "starting and never completing" case. + observe("connecting", census.connecting); + + // fixed_active below fixed_configured names a configured peer + // that cannot be reached. + observe("fixed_configured", census.fixedConfigured); + observe("fixed_active", census.fixedActive); + + // Both at zero on a fresh node means there is nothing to dial. + observe("bootcache", census.bootcache); + observe("livecache", census.livecache); + } + catch (...) // NOLINT(bugprone-empty-catch) + { + // Silently skip if services are not yet ready. + } + }, + this); +} + +void +MetricsRegistry::registerAmendmentBlockGauge() +{ + // --- Sync diagnostics: how long until this node stops validating? --- + // The existing validator_health{metric="amendment_blocked"} reports the + // terminal state, when nothing can be done. This is the window before it. + amendmentBlockGauge_ = meter_->CreateInt64ObservableGauge( + "amendment_block", "Amendment-block warning and seconds until the node stops validating"); + amendmentBlockGauge_->AddCallback( + [](opentelemetry::metrics::ObserverResult result, void* state) { + auto* self = static_cast(state); + if (self->callbacksDetached_.load(std::memory_order_acquire)) + return; + auto& app = self->app_; + + try + { + auto observe = [&](char const* field, int64_t value) { + opentelemetry::nostd::get>>(result) + ->Observe(value, {{"metric", field}}); + }; + + // An unsupported amendment has reached majority. Until now this + // only surfaced as an admin-only server_info warning. + observe("warned", app.getOPs().isAmendmentWarned() ? 1 : 0); + + // Seconds until that amendment activates. -1 means nothing is + // pending: a distinct healthy value rather than an absent + // series, matching validator_health{metric="unl_expiry_days"}. + std::int64_t secondsToBlock = -1; + if (auto const expected = app.getAmendmentTable().firstUnsupportedExpected()) + { + // NetClock's representation is unsigned, so the difference + // is taken in int64_t: subtracting the time_points directly + // would wrap once the activation time has passed. + auto const expectedSecs = + static_cast(expected->time_since_epoch().count()); + auto const nowSecs = static_cast( + app.getTimeKeeper().closeTime().time_since_epoch().count()); + + // Clamped at 0: past due means the block is imminent, not + // overdue by an amount worth charting. + secondsToBlock = std::max(expectedSecs - nowSecs, 0); + } + observe("seconds_to_block", secondsToBlock); + } + catch (...) // NOLINT(bugprone-empty-catch) + { + // Silently skip if services are not yet ready. + } + }, + this); +} + +void +MetricsRegistry::registerNodeStoreLatencyGauge() +{ + // --- Sync diagnostics: is the node store slow, and on which side? --- + // The write mean is the new signal. storeDurationUs_ was declared and + // never written, so no write latency existed anywhere; only the read side + // had a duration total. A node with a large existing DB back-fills slower + // than a fresh one, and back-fill is write-bound, so the read-side + // metrics cannot show it. Exporting both means from one reading also makes + // the two sides directly comparable. + // + // Gauge rather than histogram: a histogram would cost one Record() per + // node object on the store/fetch path, which runs thousands of times per + // ledger write. This reads four atomics per ~10 s tick instead. The + // trade-off is that percentiles are unavailable -- see the header comment. + nodeStoreLatencyGauge_ = meter_->CreateInt64ObservableGauge( + "nodestore_latency", "NodeStore mean store/fetch latency in microseconds, with counts"); + nodeStoreLatencyGauge_->AddCallback( + [](opentelemetry::metrics::ObserverResult result, void* state) { + auto* self = static_cast(state); + if (self->callbacksDetached_.load(std::memory_order_acquire)) + return; + auto& app = self->app_; + + try + { + auto observe = [&](char const* field, int64_t value) { + opentelemetry::nostd::get>>(result) + ->Observe(value, {{"metric", field}}); + }; + + auto& db = app.getNodeStore(); + + // One reading of each pair, so a mean and its own denominator + // describe the same instant. + auto const storeCount = db.getStoreCount(); + auto const storeDurationUs = db.getStoreDurationUs(); + auto const fetchCount = db.getFetchTotalCount(); + auto const fetchDurationUs = db.getFetchDurationUs(); + + // Counts are always observed, including zero: that is what + // separates "nothing written yet" from "writes are instant". + observe("write_count", static_cast(storeCount)); + observe("read_count", static_cast(fetchCount)); + + // A mean needs a non-zero denominator, and it needs a + // numerator that was actually measured. Both are required, and + // the series is omitted rather than observed as 0 when either + // is missing: a reported 0 us would claim writes are + // instantaneous, which is worse than no reading at all. + // + // The numerator guard is load-bearing, not defensive. + // Database::store() is pure virtual, so only the store paths + // that call recordStoreDuration() contribute. Today that is + // Database::importInternal (the [import_db] path). The two + // concrete runtime databases -- DatabaseNodeImp::store and + // DatabaseRotatingImp::store -- do not call it yet, so on an + // ordinary node write_count climbs while the duration total + // stays 0. Omitting the mean makes that a visible data gap + // instead of a false "writes take 0 us" line on the panel. + if (storeCount > 0 && storeDurationUs > 0) + observe("write_mean_us", static_cast(storeDurationUs / storeCount)); + if (fetchCount > 0 && fetchDurationUs > 0) + observe("read_mean_us", static_cast(fetchDurationUs / fetchCount)); + } + catch (...) // NOLINT(bugprone-empty-catch) + { + // Silently skip if services are not yet ready. + } + }, + this); +} + +void +MetricsRegistry::registerLedgerQuorumPublishGauge() +{ + // --- Sync diagnostics: the quorum gate and the publish pipeline --- + // The last two stages of a fresh sync, and the two whose failures were + // invisible: a node can hold every ledger it needs and still never declare + // one validated (quorum short), or validate correctly and never publish + // (pipeline behind). Both used to be trace-log-only or not derivable at all. + ledgerQuorumPublishGauge_ = meter_->CreateInt64ObservableGauge( + "ledger_quorum_publish", + "Pre-accept quorum gate and publish lag (tally vs quorum, first-validated, lag)"); + ledgerQuorumPublishGauge_->AddCallback( + [](opentelemetry::metrics::ObserverResult result, void* state) { + auto* self = static_cast(state); + if (self->callbacksDetached_.load(std::memory_order_acquire)) + return; + auto& app = self->app_; + + try + { + auto observe = [&](char const* field, int64_t value) { + opentelemetry::nostd::get>>(result) + ->Observe(value, {{"metric", field}}); + }; + + auto const& ledgerMaster = app.getLedgerMaster(); + + // The pair that separates "slow" from "stuck". A tally climbing + // toward the target will get there; a tally flat below it never + // will, and no acquire or peer panel says which is happening. + observe("trusted_validation_tally", ledgerMaster.getTrustedValidationTally()); + + // What the last gate evaluation actually required, as opposed to + // unl_quorum{quorum} which is what the trusted list configures. + // Already clamped against the SIZE_MAX "quorum disabled" + // sentinel by LedgerMaster, so this never wraps negative. + observe("quorum_target", ledgerMaster.getQuorumTarget()); + + // One-shot: a value is the time the first ledger took to pass + // the gate, and 0 means it never has. Not a trend. + observe("time_to_first_validated_us", ledgerMaster.getTimeToFirstValidatedUs()); + + // Validated but not yet published. pubLedgerSeq_ was never + // exported, so this gap was not derivable from any other series. + observe("publish_lag", ledgerMaster.getPublishLag()); + } + catch (...) // NOLINT(bugprone-empty-catch) + { + // Silently skip if services are not yet ready. + } + }, + this); +} + #endif // XRPL_ENABLE_TELEMETRY // ----------------------------------------------------------------- diff --git a/src/xrpld/telemetry/MetricsRegistry.h b/src/xrpld/telemetry/MetricsRegistry.h index 2a10e4a3d4..fb3830fc0e 100644 --- a/src/xrpld/telemetry/MetricsRegistry.h +++ b/src/xrpld/telemetry/MetricsRegistry.h @@ -66,6 +66,12 @@ * | server stall seconds, ledgers behind network) * +-- JobQueue backlog (waiting/running/deferred per job type) * +-- JobQueue saturation (running tasks vs worker threads vs backlog) + * +-- Peer ledger supply (how many peers can serve the needed sequence) + * +-- PeerFinder slot census (slots, attempts, fixed peers, address caches) + * +-- Amendment block (warned flag + seconds until the node stops validating) + * +-- NodeStore latency (mean us per store and per fetch, with counts) + * +-- Ledger quorum + publish (validation tally vs quorum target, + * | time to first validated, publish lag) * +-- jq_trans_overflow_total (observed from Overlay) * +-- server_stall_events_total (observed from LoadManager) * @@ -596,6 +602,38 @@ private: */ opentelemetry::nostd::shared_ptr jobQueueSaturationGauge_; + /** + * Observable gauge for how much of the needed ledger range the connected + * peer set can actually serve. + */ + opentelemetry::nostd::shared_ptr + peerLedgerSupplyGauge_; + /** + * Observable gauge for PeerFinder slot occupancy, connection attempts, + * fixed peers and address-cache depth. + */ + opentelemetry::nostd::shared_ptr slotCensusGauge_; + /** + * Observable gauge for the amendment-block warning flag and the countdown + * to the amendment activating. + */ + opentelemetry::nostd::shared_ptr + amendmentBlockGauge_; + /** + * Observable gauge for node-store read and write latency, as mean + * microseconds per operation derived from the cumulative duration and + * operation-count totals the node store already keeps. + */ + opentelemetry::nostd::shared_ptr + nodeStoreLatencyGauge_; + /** + * Observable gauge for the pre-accept quorum gate and the publish lag: + * the trusted-validation tally against the quorum it must reach, the + * time to the first fully-validated ledger, and how far publishing + * trails validation. + */ + opentelemetry::nostd::shared_ptr + ledgerQuorumPublishGauge_; /** * Observable gauge for build version info (label-based, value=1). */ @@ -978,7 +1016,244 @@ private: */ void registerJobQueueSaturationGauge(); // sync diagnostics: pool saturation -#endif // XRPL_ENABLE_TELEMETRY + + /** + * Register the `peer_ledger_supply` gauge. + * + * Five series under the `metric` attribute, from one + * Overlay::getPeerLedgerSupply() pass over the active peers: + * + * `peers_reporting` — peers that have advertised a ledger range at all. + * The denominator that makes the rest readable. + * `peers_serving_validated` — peers whose range covers this node's + * validated sequence. + * `peers_serving_next` — **the signal this gauge exists for.** Peers + * whose range covers validated + 1, the next ledger this node must + * acquire. Zero here with a non-zero `peers_reporting` means no + * connected peer holds what this node needs, so no amount of waiting + * will finish the sync; the peer set has to change. + * `supply_min_seq` / `supply_max_seq` — the sequence window the peer set + * covers, so an operator can see whether the node is asking for + * history nobody kept or for a tip nobody has reached. + * + * Each peer already caches the range it advertises in mtSTATUS_CHANGE + * (`PeerImp::minLedger_` / `maxLedger_`, read via `Peer::ledgerRange()`), + * but those ranges were never compared against each other, so "no peer has + * what I need" was indistinguishable from "my peers are slow" — the two + * faults with completely different fixes. + * + * Distinct from what already exists. `server_info{metric="peers"}` is a + * bare connection count with no notion of what those peers hold. + * `sync_state{metric="ledgers_behind"}` uses the same per-peer maxima but + * collapses them to a single distance-to-tip number, which cannot say how + * many peers can serve that distance or whether the range has a hole. + * `peer_quality{metric="peers_insane_count"}` counts peers on a different + * chain, which is a correctness signal, not an availability one. + * + * Peers advertising [0, 0] have not reported yet and are excluded from + * every field, so they cannot make a healthy peer set appear to serve from + * genesis. When nothing has reported, both window fields read 0, which is + * why `peers_reporting` must be read alongside them. + * + * @note Pulled on the OTel reader thread (~10 s tick), never on a message + * path. O(peers): `getActivePeers()` copies the peer list under the overlay + * lock and releases it, then each peer's cached range is read under that + * peer's own short-lived lock. + */ + void + registerPeerLedgerSupplyGauge(); // sync diagnostics: peer range coverage + + /** + * Register the `peerfinder_slot_census` gauge. + * + * Nine series under the `metric` attribute, from one + * Overlay::getSlotCensus() snapshot: `out_active`, `out_max`, `in_active`, + * `in_max`, `connecting`, `fixed_configured`, `fixed_active`, `bootcache` + * and `livecache`. + * + * All nine are already computed inside PeerFinder (`Counts`, `Bootcache`, + * `Livecache`, the fixed-peer map) and only two of them are exported + * today, as the legacy beast::insight gauges + * `peer_finder_active_inbound_peers` and + * `peer_finder_active_outbound_peers`. Those two carry no capacity, + * attempt or cache term, which leaves the three most common bootstrap + * failures invisible: + * + * - `connecting` non-zero while `out_active` stays below `out_max` — + * dials are being started and never completing. Without the attempt + * count this looks the same as a node that is not dialling at all. + * - `bootcache` at 0 — no seed addresses to dial in the first place. + * - `fixed_active` below `fixed_configured` — a peer named in the + * configuration is unreachable. + * + * The nine fields come from a single acquire of the PeerFinder lock, so + * they are mutually consistent and share one label set. The two legacy + * gauges are read at unrelated instants and cannot be joined with each + * other, let alone with a capacity term. + * + * @note Pulled on the OTel reader thread (~10 s tick). One lock acquire, + * then integer and container-size reads. + */ + void + registerSlotCensusGauge(); // sync diagnostics: peerfinder slot census + + /** + * Register the `amendment_block` gauge. + * + * Two series under the `metric` attribute: + * + * `warned` — 1 once an unsupported amendment has reached majority, from + * NetworkOPs::isAmendmentWarned(). + * `seconds_to_block` — **the leading indicator.** Seconds until that + * amendment activates, derived from + * `AmendmentTable::firstUnsupportedExpected()` against the network + * close time. `-1` when nothing is pending, matching the sentinel + * `validator_health{metric="unl_expiry_days"}` already uses, so the + * healthy state is a distinct value rather than a missing series. + * Clamped at 0 rather than going negative, because past-due means the + * block is imminent, not overdue by some amount. + * + * Amendment-blocked is a terminal sync blocker: the node stops validating + * and never resumes without a software upgrade. The existing + * `validator_health{metric="amendment_blocked"}` reports that state after + * it has happened, when nothing can be done about it. This gauge is the + * window before it, which is the only actionable part. + * + * The blocking amendment's identity is deliberately NOT a label. The + * network can vote on an arbitrary 256-bit amendment id — the set is not + * drawn from this build's known features — so an id label would be + * unbounded cardinality and would mint a permanent new series per + * amendment. The id is already logged by + * `AmendmentTableImpl::doValidatedLedger` ("Unsupported amendment + * reached majority at ..."), so it is available through logs, correlated + * to this series by node and time. + * + * @note Pulled on the OTel reader thread (~10 s tick). One mutex acquire + * inside the amendment table plus one clock read. + * @note The subtraction is done in `std::int64_t`, not in NetClock's + * unsigned representation, so a past-due activation cannot wrap to a huge + * positive count. + */ + void + registerAmendmentBlockGauge(); // sync diagnostics: amendment countdown + + /** + * Register the `nodestore_latency` gauge. + * + * Four series under the `metric` attribute, from the node store's own + * cumulative totals: + * + * `write_mean_us` — **the signal this gauge exists for.** Mean + * microseconds per store, `getStoreDurationUs() / getStoreCount()`. + * No write-side latency existed anywhere before this: + * `storeDurationUs_` was declared in Database.h and never written, and + * there was no accessor for it. This is the fingerprint of the + * "a node with a large existing DB syncs slower than a fresh one" + * symptom, which is write-bound and therefore invisible in every + * read-side metric. + * `read_mean_us` — mean microseconds per fetch, + * `getFetchDurationUs() / getFetchTotalCount()`, so the write mean has + * a same-instant, same-derivation counterpart to be compared against. + * `write_count` / `read_count` — the denominators, exported so a + * dashboard can recover *interval* latency as + * `rate(duration) / rate(count)`. Without them the means above are + * since-boot averages, which on a long-running node move so slowly + * that a current stall is invisible. + * + * Gauge, not a histogram — deliberate. A histogram would give true + * percentiles, which a mean cannot, but it costs one `Record()` per + * operation on a path that runs per node object: a single ledger write + * walks thousands of SHAMap nodes, and fetches are more frequent still. + * That is a per-object synchronous instrument call plus bucket search on + * the hot store/fetch path. This gauge instead reads four already-existing + * atomics once per ~10 s collection tick, adding nothing whatsoever to the + * hot path — the store side pays only the one clock-sample pair per store + * that the read side has always paid per fetch. For the question this work + * package answers ("is the write path slow, and slower than the read + * path?") a rate-derived mean is sufficient, and a tail latency that + * matters will move the mean. Consequence, stated plainly: p99 is NOT + * obtainable from this signal. Adding a histogram later would also require + * an explicit-bucket View registered in initExporterAndProvider() via + * addMicrosecondHistogramView(), because the SDK's default buckets top out + * at 10,000 and every microsecond duration above 10 ms would saturate. + * + * Distinct from `nodestore_state`, which already carries the raw + * cumulative `node_reads_duration_us`, `node_reads_total` and + * `node_writes` fields, and from the Ledger Data Sync dashboard's "NuDB + * Read Latency" panel that divides the first two in PromQL. Neither has + * any write-duration input to divide — that quantity did not exist. This + * gauge adds the missing write numerator and publishes both means from one + * reading so the two sides are directly comparable. + * + * @note Pulled on the OTel reader thread (~10 s tick). Four relaxed atomic + * loads and two integer divisions; no lock, no allocation, no hot-path + * cost. + * @note A mean is observed only when both its count and its duration total + * are non-zero; otherwise the series is omitted rather than reported as 0, + * because a 0 would claim the operation is instantaneous. The counts are + * always observed, so `write_count` still distinguishes "nothing written + * yet" from "writes are instant". + * @warning `write_mean_us` is currently produced only by store paths that + * call `Database::recordStoreDuration()`, which today is + * `Database::importInternal` (the `[import_db]` admin path). `store()` is + * pure virtual, and neither `DatabaseNodeImp::store` nor + * `DatabaseRotatingImp::store` calls it yet, so on an ordinary node + * `write_count` climbs while `write_mean_us` is absent. That is a + * deliberate, visible gap: closing it means adding one clock-sample pair to + * those two concrete store overrides, which live outside this work + * package's file scope. + * @note Both totals are monotonic and never reset. A panel wanting current + * rather than since-boot latency must divide the two rates, which is why + * the counts are exported alongside the means. + */ + void + registerNodeStoreLatencyGauge(); // sync diagnostics: store/fetch latency + + /** + * Register the `ledger_quorum_publish` gauge. + * + * Four series under the `metric` attribute, read from LedgerMaster: + * + * `trusted_validation_tally` — trusted validations counted at the last + * pre-accept gate in `LedgerMaster::checkAccept`. + * `quorum_target` — validations that gate required. **The pair is the + * signal.** The tally alone cannot separate a node accumulating + * validations toward quorum (slow, will finish) from one whose tally + * plateaus below the target (stuck, never will); with the target + * beside it, the two shapes are unmistakable. + * `time_to_first_validated_us` — how long the node took to get its + * first ledger through that gate. One-shot, like the + * `sync_state{initial_full_duration_us}` milestone: a value means it + * happened and this is how long it took, 0 means it never has. + * `publish_lag` — validated sequence minus published sequence. Non-zero + * and growing means validation is fine and the publish pipeline is + * behind, which no other signal distinguishes. + * + * All four are grouped under one instrument because they answer one + * question in sequence — did enough validations arrive, did the gate pass, + * how long did that take, and did the result reach clients — so an + * operator reads them from a single consistent poll. + * + * Distinct from what already exists. `unl_quorum{quorum}` is the quorum + * the validator list *configures*, a static property of the trusted set; + * `quorum_target` is what an actual gate evaluation *required*, and the + * tally beside it is the live count that must reach it — neither existed + * anywhere before. `server_info{validated_ledger_seq}` publishes the + * validated sequence but nothing published the pubLedgerSeq_ counterpart, + * so the lag between them was not derivable at all. + * + * @note `quorum_target` reports int64 max when the validator list has + * switched quorum off (`ValidatorList::quorum()` returns SIZE_MAX). The + * clamp lives in `LedgerMaster::checkAccept`, so the wrap to -1 that would + * invert a tally-versus-target panel cannot happen here. + * @note Pulled on the OTel reader thread (~10 s tick). Five relaxed atomic + * loads through lock-free LedgerMaster accessors: no lock is taken, which + * is what keeps an OTel callback from ever contending with, or inverting + * lock order against, the LedgerMaster mutex held by the emit path. + */ + void + registerLedgerQuorumPublishGauge(); // sync diagnostics: quorum + publish +#endif // XRPL_ENABLE_TELEMETRY }; } // namespace telemetry