From 22fd5e8601813b320188130721cfc276fe35cc8e Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Mon, 27 Jul 2026 11:12:26 +0100 Subject: [PATCH] feat(telemetry): make the sync board readable, and time real node writes (WP-B4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The board and runbook had grown by append across eight work packages, so they read in the order the work was done rather than the order a node progresses. This is the coherence pass; it adds no new instrumentation. - Dashboard: 52 panels regrouped from two rows into nine that follow the fresh-start sequence — bootstrap, peer supply, sync state, acquire and SHAMap fetch, job queue, quorum and publish, terminal blockers, then back-fill and spans collapsed since they answer conditional questions. Layout only: no title, query or description changed. - Runbook: the flat step list becomes a decision tree branching on the observed symptom, with the amendment-block check first because it is terminal. Each branch names the panels, what healthy and unhealthy look like, and what to conclude. The existing steps are kept as the detail bodies. - Reference table: every signal name re-checked against the code and every named panel against the board; four stale panel references fixed. - Validation: every signal is now either asserted or covered by a note explaining why a five-node local cluster cannot produce it. Also fixes the write-latency signal, which was inert on a real node: the store duration was only recorded on the database-import path, while the two production store implementations did not time themselves, so an ordinary node reported a write count with no latency. Both now time the backend write, which is the disk work this signal exists to expose. Without it the "existing database syncs slower than a fresh one" diagnosis had no primary signal. Co-Authored-By: Claude Opus 5 (1M context) --- .../09-data-collection-reference.md | 142 +- .../dashboards/ledger-sync-health.json | 4875 +++++++++-------- docker/telemetry/workload/README.md | 2 +- docker/telemetry/workload/expected_spans.json | 1 + docs/telemetry-glossary.md | 22 +- docs/telemetry-runbook.md | 232 +- src/libxrpl/nodestore/DatabaseNodeImp.cpp | 6 + src/libxrpl/nodestore/DatabaseRotatingImp.cpp | 6 + 8 files changed, 2817 insertions(+), 2469 deletions(-) diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md index 9dcca4d298..2c7c0e00b7 100644 --- a/OpenTelemetryPlan/09-data-collection-reference.md +++ b/OpenTelemetryPlan/09-data-collection-reference.md @@ -33,7 +33,7 @@ graph LR end subgraph viz["Visualization"] - F["Grafana :3000
13 dashboards"] + F["Grafana :3000
15 dashboards"] end A -->|"OTLP/HTTP :4318
(traces + attributes)"| R1 @@ -635,6 +635,23 @@ For each of the 45+ overlay traffic categories (defined in `TrafficCount.h`), fo > **See also**: [05-configuration-reference.md](./05-configuration-reference.md) §5.8 for Grafana data source provisioning (Tempo, Prometheus) and TraceQL query examples. +Fifteen dashboards are provisioned in total. §3.1 and §3.2 below cover the +original ten; the remaining five were added by later phases and are catalogued +where they were introduced, so this section is not the full inventory: + +| Dashboard | UID | Catalogued in | +| ------------------ | -------------------- | ---------------------------------------------------- | +| Fee Market & TxQ | `fee-market` | §5b "New Grafana Dashboards (Phase 9)" | +| Job Queue Analysis | `job-queue` | §5b "New Grafana Dashboards (Phase 9)" | +| Validator Health | `validator-health` | §5d "New Grafana Dashboards (Phase 9)" | +| Peer Quality | `peer-quality` | §5d "New Grafana Dashboards (Phase 9)" | +| Ledger Sync Health | `ledger-sync-health` | "Fresh-node sync diagnostics" (end of this document) | + +The authoritative count is whatever +`docker/telemetry/grafana/dashboards/*.json` holds; +`validate_dashboards.py` prints it and the workload harness asserts every +board renders. + ### 3.1 Span-Derived Dashboards (5) | Dashboard | UID | Data Source | Key Panels | @@ -683,7 +700,7 @@ for how the tier attributes are set and reach metrics. 1. Open Grafana at **http://localhost:3000** 2. Navigate to **Dashboards → xrpld** folder -3. All 10 dashboards are auto-provisioned from `docker/telemetry/grafana/dashboards/` +3. All 15 dashboards are auto-provisioned from `docker/telemetry/grafana/dashboards/` --- @@ -1389,8 +1406,24 @@ reaches, a validated ledger (`server_state=full`). Two groups: pre-quorum bootstrap (DNS, dial, handshake, UNL/quorum, clock skew) and the post-peering ledger/tx-set acquire pipeline. -Rendered by the **Ledger Sync Health** dashboard -(uid `ledger-sync-health`, rows `Bootstrap (Domain 0)` and `Sync pipeline`). +Rendered by the **Ledger Sync Health** dashboard (uid `ledger-sync-health`), +whose nine rows follow the order a fresh node progresses — so reading the board +top-to-bottom walks the same path a sync does: + +1. `Bootstrap (Domain 0)` — can it reach peers and form a quorum at all? +2. `Peer supply` — does any peer hold what this node needs? +3. `Sync state` — is the node advancing through the mode machine? +4. `Ledger acquire & SHAMap fetch` — is ledger data arriving and being applied? +5. `Job queue` — does arrived work ever get a worker thread? +6. `Quorum & publish` — does a held ledger ever validate, and reach clients? +7. `Terminal blockers & serving` — will the node stop validating for good? +8. `Back-fill & persistence` (collapsed) — is an existing database the bottleneck? +9. `Spans & traces` (collapsed) — _which_ fetch, peer or object, not how many? + +Rows 8 and 9 are collapsed by default because they answer conditional questions: +row 8 only applies to a node with existing history, and row 9 is span-derived, so +it inherits trace sampling and the `trace_ledger` / `trace_peer` flags. + Operator flow: [telemetry-runbook.md](../docs/telemetry-runbook.md) "Diagnosing slow/stuck fresh sync". Terms: [telemetry-glossary.md](../docs/telemetry-glossary.md) @@ -1399,57 +1432,60 @@ Operator flow: [telemetry-runbook.md](../docs/telemetry-runbook.md) The table below is the single index for these signals; one row is added per signal as it lands. `Type` is the instrument kind (counter / gauge / histogram / span / span attr), `Emit site` the owning source file, and `Panel` the dashboard -panel that renders it. +panel that renders it. Panel names are verbatim `ledger-sync-health` panel +titles unless another board is named explicitly, and `n/a` means the signal has +no panel (it is read in Tempo instead). -| Signal | Type | Emit site | Panel | Meaning | -| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `dns_resolve_total` (`outcome` = `resolved` \| `empty`) | counter | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Outcome Rate | Peer hostname resolutions. `empty` means a configured bootstrap or `[ips_fixed]` name returned no address, so that peer is never dialled. | -| `dns_resolve_latency_ms` | histogram | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Latency (p95) | Time to resolve a configured peer hostname. Seconds-scale values mean the resolver is timing out ahead of every dial. | -| `overlay_connect_total` (`outcome` = `connected` \| `tcp_fail` \| `tls_fail` \| `upgrade_fail` \| `timeout`) | counter | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Outcome Rate | Outbound peer connection attempts by terminal outcome. The outcome names the stage that broke: TCP, TLS, HTTP upgrade, or no terminal state in time. | -| `overlay_dial_latency_ms` | histogram | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Latency (p95) | Time from starting an outbound dial to its terminal outcome, successes and failures together. A p95 near the dial timeout means peers accept TCP but never finish the handshake. | -| `handshake_negotiation_fail_total` (`reason`, 14 values incl. `wrong_network`, `invalid_network_id`, `clock_skew`, `self_connection`, `session_verify_failed`) | counter | `Handshake.cpp` — `throwNegotiationFailure` (from `verifyHandshake`) | Handshake Negotiation Failures by Reason | Peer handshakes rejected after TLS while checking network id, clock, keys and addresses. `reason` names the failing check. | -| `unl_fetch_total` (`site` = configured UNL URI; `outcome` = the 9 `ListDisposition` strings `accepted` \| `expired` \| `same_sequence` \| `pending` \| `known_sequence` \| `unsupported_version` \| `untrusted` \| `stale` \| `invalid`, plus `fetch_error` \| `bad_status` \| `parse_error`) | counter | `ValidatorSite.cpp` — `ValidatorSite::reportFetchOutcome` | UNL Fetch Rate by Site & Outcome | Validator-list fetches per site. `accepted` is the only success; `same_sequence` and `known_sequence` are normal no-op refreshes; the three literals are transport or content faults. | -| `unl_quorum` (`metric` = `trusted_keys` \| `quorum`) | observable gauge | `MetricsRegistry.cpp` — `registerUnlQuorumGauge` | UNL Trusted Keys vs Quorum; UNL Quorum Headroom | Trusted UNL key count against the validations a ledger needs. `trusted_keys` at or below `quorum` means the node can never declare a ledger validated. | -| `clock_close_offset_seconds` (`metric` = `offset`) | observable gauge | `MetricsRegistry.cpp` — `registerClockSkewGauge` | Clock Close Offset | Network close time offset from the local clock. Negative means the local clock runs ahead. `server_info` only surfaces `close_time_offset` at 60 s or more, so this gauge sees skew far earlier. | -| `state_changes_total` (`from`, `to` = `disconnected` \| `connected` \| `syncing` \| `tracking` \| `full`) | counter | `NetworkOPs.cpp` — `NetworkOPsImp::setMode` | Mode Transitions by Edge | Operating-mode transitions keyed on the (`from`, `to`) edge. The edge is what separates a clean `disconnected`→`connected`→`syncing`→`tracking`→`full` climb from `full`→`connected` flapping; an unlabelled total cannot tell them apart. | -| `sync_state` (`metric` = `initial_full_duration_us`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Time to First FULL | Microseconds from process start to the first `full` transition, sourced from `NetworkOPs::getInitialSyncDurationUs()`. Stays 0 until `full` is reached, so a flat 0 is itself the "never synced" signal; once set it never changes. | -| `sync_state` (`metric` = `network_ledger_gate`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Network Ledger Gate | 1 while the node is still waiting to see a full network ledger (`NetworkOPs::isNeedNetworkLedger()`), else 0. A persistent 1 blocks transaction submission and `full`, whatever the rest of the pipeline shows. | -| `sync_state` (`metric` = `server_stall_seconds`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Server Stall | Current main-loop stall duration from `LoadManager::getCurrentStallSeconds()`, 0 when healthy. Same duration the load monitor logs as "Server stalled for N seconds", which previously existed only in that log line. | -| `sync_state` (`metric` = `ledgers_behind`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Ledgers Behind Network | Peer-reported network tip minus our validated sequence, floored at 0 (`NetworkOPs::getLedgersBehindNetwork()`). Reads each peer's already cached ledger range, so no new network round trip. | -| `server_stall_events_total` | observable counter | `MetricsRegistry.cpp` — `registerStallEventsCounter` | Server Stall Event Rate | Distinct stall episodes since process start, counted once per episode rather than per stalled second. A rising rate is repeated fresh stalls; a flat rate with a large `server_stall_seconds` is one long stall. | -| `sync_acquire` (`metric` = `missing_state_nodes_max` \| `missing_tx_nodes_max`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Missing SHAMap Nodes per Acquire (state/tx) | Largest outstanding SHAMap node count across in-flight acquires, split by tree, from the count `getMissingNodes()` already produces during its sweep (`InboundLedger.cpp` — `InboundLedger::trigger`). **The headline stuck-sync signal:** flat and non-zero across ticks means the acquire will never finish; shrinking means slow but alive. Aggregated as a max rather than labelled per ledger, because a `ledger_seq` label would mint one series per ledger acquired — per-ledger identity stays on the `ledger.acquire` span. | -| `sync_acquire` (`metric` = `received_data_depth`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Peer packets stashed across all in-flight acquires waiting to be applied, summed because it measures one shared processing backlog. A growing depth means arriving node data outpaces processing, so the limit is the job queue or disk rather than peer supply. | -| `sync_acquire` (`metric` = `in_flight`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Number of ledger acquires currently running. Exported so the three values above can be read in context: all zero with `in_flight` zero is an idle node, not a healthy one. | -| `shamap_cache_hit_rate` (`metric` = `treenode`) | observable gauge | `MetricsRegistry.cpp` — `registerCacheHitRateDetailGauge` | SHAMap TreeNode Cache Hit Rate | Share of SHAMap tree-node lookups served from memory, from the previously-uncalled `TaggedCache::getHitRate()`, normalized from 0-100 to 0.0-1.0. Distinct from `nodestore_state`-derived NuDB Cache Hit Ratio on the Ledger Data Sync dashboard: this is the in-memory layer **above** the node store, so a miss here is what causes a read there. The full-below cache is not reported — it is a `KeyCache` whose only lookup path increments `stats_.hits`/`stats_.misses` while `getHitRate()` reads the separate `hits_`/`misses_` members, so its rate is hard-wired to 0 until that accounting is fixed. | -| `sync_acquire_no_progress_total` | counter | `InboundLedger.cpp` — `InboundLedger::onTimer` | Acquire Stall Rate (no progress) | Acquire timeouts where not one new node arrived since the previous timeout, from the `progress_` flag that was previously log-only. Fires on the 3 s acquire timer, never per node. A sustained rate together with a flat missing-node count is the definitive "stuck, not slow" signature. | -| `sync_addnode_total` (`outcome` = `good` \| `duplicate` \| `invalid`) | counter | `InboundLedger.cpp` — `InboundLedger::recordBatchOutcome` | Add-Node Outcomes | SHAMap nodes received during acquire, split by result. Emitted once per received packet from the aggregated batch tally the trace log already printed — never inside the per-node `receiveNode()` loop. Separates real progress (`good`) from wasted bandwidth (`duplicate`) and a misbehaving peer (`invalid`), all three of which look like healthy throughput in traffic metrics. | -| `sync_acquire_source_total` (`source` = `local` \| `network`) | counter | `InboundLedger.cpp` — `InboundLedger::init` | Acquire Source (local vs network) | Whether an acquire was satisfied entirely from the local node store or needed peers, emitted once per new acquire after the first local lookup. Sustained `network` on a node that should already hold the range means sync is disk-bound rather than peer-bound. | -| `jobq_backlog` (`metric` = `waiting` \| `running` \| `deferred`; `job_type` = the `JobTypes::name()` string) | observable gauge | `MetricsRegistry.cpp` — `registerJobQueueBacklogGauge` | Deferred Jobs by Type (starvation); Job Queue Occupancy by Type (waiting/running) | Instantaneous per-job-type queue occupancy, from `JobQueue::getJobTypeCounts()` (one mutex acquire per ~10 s tick). **`deferred` is the signal this adds:** jobs the queue accepted but withheld because the type is already at its concurrency limit, which is counted in neither `waiting` nor `running` and had no exposure anywhere before. The sync-critical types are capped at 3 (`JtLedgerReq`, `JtLedgerData` in `JobTypes.h`), so they starve first. Distinct from the existing `job_queued_total` / `job_started_total` / `job_finished_total` counters and `job_queued_us` / `job_running_us` histograms, which are event-driven from PerfLogImp and describe jobs that already moved, and from the StatsD `jobq_job_count`, which is queue-wide with no per-type split. Cardinality is bounded by the JobType enum (~46 values); every type is observed every tick, so an idle type reports 0 rather than dropping its series. | -| `jobq_saturation` (`metric` = `running_tasks` \| `worker_threads` \| `total_waiting`) | observable gauge | `MetricsRegistry.cpp` — `registerJobQueueSaturationGauge` | Worker Pool Saturation; Worker Pool Capacity & Total Backlog | Global worker-pool saturation from `JobQueue::getWorkerSaturation()`: tasks in flight, threads the pool is configured to run, and jobs queued across all types, all from one reading so the ratio and the backlog describe the same instant. `worker_threads` is exported rather than hardcoded in the dashboard because it is derived at startup from `[workers]`, node size and hardware concurrency. Exists separately from `jobq_backlog` because a pool-wide slowdown otherwise appears as an independent fault in every subsystem queued behind it; a `running_tasks / worker_threads` ratio at 1.0 **with** a non-zero `total_waiting` attributes it to pool exhaustion once. | -| `peer_ledger_supply` (`metric` = `peers_reporting` \| `peers_serving_validated` \| `peers_serving_next` \| `supply_min_seq` \| `supply_max_seq`) | observable gauge | `MetricsRegistry.cpp` — `registerPeerLedgerSupplyGauge` (aggregating `OverlayImpl::getPeerLedgerSupply`) | Peers Able to Serve Needed Sequence; Peer Ledger Supply Window | How much of the sequence range this node needs its connected peer set can actually serve, from one pass over the active peers reading the range each already advertised in `mtSTATUS_CHANGE`. **`peers_serving_next` is the signal this exists for:** zero there with a non-zero `peers_reporting` means no connected peer holds validated + 1, so the peer set must change and waiting cannot finish the sync. `peers_reporting` is the denominator that makes the rest readable — peers advertising `[0, 0]` have not reported yet and are excluded from every field, so they cannot make a healthy peer set appear to serve from genesis; when nothing has reported, both window fields read 0 meaning **unknown**, not genesis. `supply_min_seq` / `supply_max_seq` separate "asking for history nobody kept" from "asking for a tip nobody reached". Distinct from `server_info{metric="peers"}`, a bare connection count with no notion of what those peers hold; from `sync_state{metric="ledgers_behind"}`, which uses the same per-peer maxima but collapses them to a single distance-to-tip number that cannot say how many peers can serve that distance or whether the range has a hole; and from `peer_quality{metric="peers_insane_count"}`, which counts peers on a different chain and is therefore a correctness signal, not an availability one. | -| `peer_disconnect_total` (`reason` = `graceful` \| `shutdown` \| `stopping` \| `read_error` \| `write_error` \| `timer_error` \| `ping_timeout` \| `not_useful` \| `large_sendq` \| `charge_resources` \| `malformed_handshake` \| `shared_value` \| `unknown`; `direction` = `inbound` \| `outbound`) | counter | `PeerImp.cpp` — `PeerImp::close` | Peer Disconnects by Reason | Peer teardowns split by cause and by which side opened the connection. Emitted once per teardown at `close()`, the single funnel every disconnect path passes through, and `close()` already self-guards on the socket being open, so a repeated close cannot double-count and the total matches the existing unlabelled tally. `reason` is set by whichever site decided to disconnect, first writer wins, so a later generic reason never masks the real one; the value is always one of a fixed set of literals in `PeerImp.cpp`, never peer-supplied data, so cardinality is bounded by the code. The split is the whole point: it separates our-fault backpressure (`large_sendq`, `charge_resources`) from topology and network faults (`not_useful`, `ping_timeout`, `read_error`), and normal churn (`graceful`) from either. Distinct from the existing `server_info{metric="peer_disconnects_resources"}`, which counts only the resource-charge subset and carries no labels, and from the StatsD `overlay_peer_disconnects`, which is the unlabelled grand total in which every reason above collapses into one number. | -| `peer_accept_total` (`outcome` = `accepted` \| `local_endpoint_fail` \| `resource_limit` \| `no_slot` \| `not_peer_request` \| `protocol_mismatch` \| `bad_cookie` \| `slot_refused` \| `handshake_error`) | counter | `OverlayImpl.cpp` — `OverlayImpl::onHandoff` via `reportAcceptOutcome` | Inbound Peer Accept Outcomes | Terminal outcome of every inbound connection this node is offered, one emit per handoff. `accepted` is reported only after `run()`, so anything that threw on the way lands on `handshake_error` instead; the two early returns that are not peer attempts at all (a handled HTTP request, and a request that never asked to upgrade) are deliberately not counted. The `outcome` names the stage that refused: no local endpoint, the resource manager, PeerFinder having no slot or seeing a duplicate, a non-peer upgrade request, protocol version disagreement, a bad security cookie, or activation being refused. This is the **inbound twin** of the existing `overlay_connect_total{outcome}`, which covers outbound dials only; without it a node refusing every inbound connection is indistinguishable from one nobody dials, and reading the two together gives the full in/out split. | -| `peerfinder_slot_census` (`metric` = `out_active` \| `out_max` \| `in_active` \| `in_max` \| `connecting` \| `fixed_configured` \| `fixed_active` \| `bootcache` \| `livecache`) | observable gauge | `MetricsRegistry.cpp` — `registerSlotCensusGauge` (from `Logic::getSlotCensus`) | PeerFinder Slot Census; PeerFinder Address Caches & Fixed Peers | Slot occupancy against capacity, outbound dials in flight, configured-versus-connected fixed peers, and the depth of both address caches. All nine come from a single acquire of the PeerFinder lock, so they are mutually consistent, share one label set and can be compared against each other. That is what makes the three most common bootstrap failures visible: `connecting` non-zero while `out_active` stays below `out_max` (dials starting and never completing), `bootcache` and `livecache` both at 0 (nothing to dial at all), and `fixed_active` below `fixed_configured` (a peer named in the configuration is unreachable). `fixed_configured` is the count of peers named in the config, so the pair an operator reads is "how many did I ask for" against "how many do I have" — the same comparison `autoconnect()` makes. All nine values already existed inside PeerFinder; only two of them were exported, as the legacy beast::insight gauges `peer_finder_active_inbound_peers` and `peer_finder_active_outbound_peers`. Those two carry no capacity, attempt or cache term, are read at unrelated instants, and so cannot be joined with each other let alone with a capacity term — leaving all three failures above indistinguishable from a node that is simply not dialling. | -| `serve_refused_total` (`request` = `ledger` \| `txset` \| `object` \| `fetchpack`; `reason` = `sendq_full` \| `load_shed` \| `not_found` \| `no_map` \| `bad_type` \| `empty_reply`) | counter | `PeerImp.cpp` — `processLedgerRequest`, `onMessage(TMGetObjectByHash)`, `doFetchPack` | Ledger/Object Serve Refusals | Peer data requests this node declined to answer, split by what was asked for and why. This is the **supply side** of the sync exchange — what this node refuses to serve OTHERS — and nothing equivalent existed before, so a node shedding every ledger request looked identical to one being asked for nothing. `sendq_full` and `load_shed` are self-inflicted backpressure (the send queue at `Tuning::kDropSendQueue`, or the local fee track loaded, or too many pack jobs queued), while `not_found` is a genuine history gap and `no_map` / `bad_type` / `empty_reply` mean the request was answerable in principle but produced nothing to send. `fetchpack` is counted apart from `ledger` because a fetch pack is how a syncing peer catches up in bulk and its shed threshold is a different one. Emitted at most once per request — `empty_reply` is reported after the node loop, never inside it — and both labels are code literals, so cardinality is bounded at compile time. | -| `amendment_block` (`metric` = `warned` \| `seconds_to_block`) | observable gauge | `MetricsRegistry.cpp` — `registerAmendmentBlockGauge` | Amendment Block Countdown; Amendment Warned | `warned` is 1 once an unsupported amendment has reached majority (`NetworkOPs::isAmendmentWarned()`, previously only an admin-only `server_info` warning). **`seconds_to_block` is the leading indicator:** seconds until that amendment activates, from `AmendmentTable::firstUnsupportedExpected()` against the network close time. It reads `-1` when nothing is pending — a distinct healthy value rather than a missing series, matching the sentinel `validator_health{metric="unl_expiry_days"}` already uses — and is clamped at 0 rather than going negative, because past-due means the block is imminent, not overdue by some amount worth charting; the subtraction is done in `std::int64_t` so a past-due activation cannot wrap. Amendment-blocked is a terminal sync blocker: the node stops validating and never resumes without a software upgrade. The existing `validator_health{metric="amendment_blocked"}` reports that state after the fact, when nothing can be done about it; this gauge is the window before it, which is the only actionable part. The blocking amendment's identity is deliberately **not** a label — the network can vote on an arbitrary 256-bit amendment id, not drawn from this build's known features, so an id label would be unbounded cardinality and would mint a permanent new series per amendment. The id is available in logs from `AmendmentTableImpl::doValidatedLedger` ("Unsupported amendment \ reached majority at ..."), correlated to this series by node and time. | -| `ledger_jump_total` | counter | `NetworkOPs.cpp` — `NetworkOPsImp::switchLastClosedLedger` | Byzantine Ledger Jumps | Forced jumps of the last closed ledger onto a divergent chain: the node was told the network's LCL is not the one it built on and discarded its own chain tip to follow. Nothing equivalent existed — this was log-only ("JUMP last closed ledger to ..."), so a node repeatedly thrashing between chains left no time series to correlate against the rest of the sync pipeline. Any non-zero rate is abnormal by construction; repeated jumps are wrong-chain thrash, which points at the peer set and the configured network id rather than anywhere in the acquire pipeline. Deliberately unlabelled: the ledger hash and sequence would both be unbounded as label values, and the log line beside the emit already carries them. | -| `nodestore_latency` (`metric` = `write_mean_us` \| `read_mean_us` \| `write_count` \| `read_count`) | observable gauge | `MetricsRegistry.cpp` — `registerNodeStoreLatencyGauge` | NodeStore Write vs Read Latency (us/op); NodeStore Operation Rate | Mean microseconds per node-store store and per fetch, with both operation counts so a panel can divide the two rates and read _interval_ latency instead of the since-boot average. **The write side is the new signal.** `storeDurationUs_` was declared in `Database.h` and never written, and no accessor existed, so no write-path latency was observable anywhere; the read total was already exposed as `nodestore_state{metric="node_reads_duration_us"}`. This is the fingerprint of the "a node with a large existing DB syncs slower than a fresh one" symptom, which is write-bound and therefore invisible in every read-side metric. Chosen as a gauge over a histogram deliberately: a histogram gives true percentiles but costs one `Record()` per node object on the store/fetch path, and a single ledger write walks thousands of SHAMap nodes — this gauge instead reads four existing atomics once per ~10 s tick and adds nothing to the hot path. Consequence: **p99 is not obtainable from this signal**, and a histogram added later would also need an explicit-bucket View (`addMicrosecondHistogramView`) because the SDK default buckets top out at 10,000. Distinct from the Ledger Data Sync dashboard's NuDB Read Latency panel, which divides two `nodestore_state` fields in PromQL: that panel has no write-duration input to divide, because the quantity did not exist. **Known gap:** `write_mean_us` is emitted only when the store-duration total is non-zero, and that total is fed by `Database::recordStoreDuration`, today called only from `Database::importInternal` (the `[import_db]` admin path). `Database::store()` is pure virtual and neither `DatabaseNodeImp::store` nor `DatabaseRotatingImp::store` times itself yet, so an ordinary node reports `write_count` with no `write_mean_us`. The mean is omitted rather than reported as 0 so the gap stays visible instead of reading as "writes are instantaneous". | -| `ledger_replay_fallback_total` (`stage` = `skiplist` \| `delta`) | counter | `SkipListAcquire.cpp` / `LedgerDeltaAcquire.cpp` — `trigger` | Replay Fallback to Full Acquire (by stage) | A ledger-replay sub-task abandoning its shortcut and acquiring the whole ledger through `InboundLedger` instead, because too few connected peers support the `LedgerReplay` protocol feature. Both branches were debug-log-only, so a silently defeated replay optimisation left no metric at all — back-fill simply ran on the slower path with nothing to show why. Emitted once, on the transition into fallback, not at the acquire call, which re-runs on every later trigger. The `stage` label separates the skip-list acquire (which fetches the list of historical ledger hashes) from the per-ledger delta acquire, because they fail independently. | -| `ledger_replay_outcome_total` (`outcome` = `success` \| `timeout` \| `build_failed` \| `parameter_failed`) | counter | `LedgerReplayTask.cpp` — `LedgerReplayTask::recordOutcome` | Replay Outcomes (by terminal state) | Terminal state of every ledger-replay task, one emit per task. Every terminal path previously only set an internal `complete_`/`failed_` flag and wrote a log line, so a replay that never succeeded was indistinguishable from one that was never attempted. The outcome names the layer at fault: `timeout` means the deltas never arrived (a peer-supply problem), `build_failed` means a delta would not apply to its parent, and `parameter_failed` means a peer served a skip list inconsistent with what the task asked for — the latter two are data faults, not slowness. Read with `ledger_replay_fallback_total`: fallbacks rising while successes stay flat is replay-based catch-up degrading to full-ledger acquisition. | -| `ledger_quorum_publish` (`metric` = `trusted_validation_tally` \| `quorum_target`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Trusted Validations vs Quorum Target | Trusted validations counted at the most recent pre-accept gate, beside the number that gate required. Snapshotted in `LedgerMaster::checkAccept` before the shortfall check, so a node that keeps failing the gate still reports both numbers — which is the whole point: the tally alone cannot say whether validations are accumulating toward quorum (slow, will finish) or plateaued below it (stuck). Read the sustained floor of the tally, not a single sample: each series is a snapshot of the last evaluation, and the first evaluation of each round runs before peer validations arrive, so a healthy node sawtooths. `quorum_target` is what the gate actually demanded, as opposed to `unl_quorum{metric="quorum"}` which is what the trusted list configures. When the trusted list disables quorum entirely (`getNeededValidations` returns `SIZE_MAX`) the target is reported as int64 max rather than wrapping to -1, so it reads far above any tally instead of inverting the comparison — the same sentinel handling as the `unl_quorum` gauge. | -| `ledger_quorum_publish` (`metric` = `publish_lag`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Publish Lag (validated minus published) | Ledgers fully validated but not yet published to clients and subscribers: the validated sequence minus the published sequence, floored at zero. `pubLedgerSeq_` was never exported, so this gap was not derivable from any other series. Publishing trails validation by design and a small lag drains each round; a lag that stays positive or grows means validation is healthy and the publish pipeline is not, which is a distinct fault from anything the quorum or acquire signals can show. The two sequences are read as independent relaxed loads, so a sample taken mid-update may be off by one ledger for one poll — immaterial for a lag trend, and the price of not taking the LedgerMaster mutex on the metrics poll thread. | -| `ledger_quorum_publish` (`metric` = `time_to_first_validated_us`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Time to First Validated Ledger | Microseconds from process start until the first ledger passed the pre-accept quorum gate. A one-shot measurement like `sync_state{metric="initial_full_duration_us"}`: written once under `mutex_` and never changed, so it has no trend. Exactly two readings are meaningful — a duration, meaning the node reached its first fully-validated ledger and this is how long that took, or 0, meaning it never has. Clamped to a minimum of 1 so a genuine sub-microsecond reading can never be confused with the never-reached zero. A value here alongside a zero on time-to-first-FULL, or the reverse, separates "reached the full server state" from "fully validated a ledger". | -| `ledger_quorum_shortfall_total` (`stage` = `pre_accept`) | counter | `LedgerMaster.cpp` — `LedgerMaster::checkAccept` | Pre-Accept Quorum Shortfall Rate | One increment per pre-accept gate evaluation rejected because the trusted validation tally was below quorum. Previously trace-log-only, which made a node that peers and receives validations yet never validates indistinguishable from an idle one. A non-zero rate is **not** by itself a fault: `doAccept` issues this node's own validation and calls `consensusBuilt` → `checkAccept` immediately, before peer validations for that ledger arrive, so the first evaluation of every round tallies short and is retried as validations come in — a healthy cluster emits this counter every round. The fault signature is the rate climbing well above the ledger-close rate while the tally stays flat below its target and `time_to_first_validated_us` stays at 0. Emitted while `mutex_` is held, which is safe against the metrics poll because every accessor the sync gauges read is a lock-free atomic load, so no OTel callback ever acquires `mutex_`. | -| `consensus_round_duration_ms` | histogram | `RCLConsensus.cpp` — `RCLConsensus::Adaptor::makeAcceptSpan` | Consensus Round Duration Distribution; Consensus Round Duration (p50/p95) | Wall-clock duration of a completed consensus round, in milliseconds. Promotes the long-standing `round_time_ms` span attribute into a native instrument: the attribute answers "how long did THIS round take" inside a trace, next to the proposers and disputes that explain it, while the histogram gives the distribution over time, which is what an alert or SLO panel needs and what a raw trace query cannot cheaply produce fleet-wide. Being native it is also **never sampled**, so it stays complete when tracing is head-sampled down. Recorded at exactly one site — `makeAcceptSpan` is the single function both the synchronous (`onForceAccept`) and asynchronous (`onAccept`) accept paths call once per round — so it can neither double-count nor be skipped, and it adds no per-peer, per-proposal or per-transaction work. **Explicit buckets** are registered for it in `MetricsRegistry::initExporterAndProvider` (`addRoundDurationHistogramView`, boundaries 500 ms → 120 s): the SDK default tops out at 10,000 ms, which would collapse every slow round into one saturated bucket and read every quantile as 10 s, and the consensus parameters themselves allow a round up to `ledgerAbandonConsensus` = 120 s. Needs **no collector change** — a native metric rides the existing OTLP → Prometheus path. | -| `consensus.validation.accept` (`validation_status`, `accept_gated`, `ledger_hash`, `ledger_seq`, `full_validation`) | span + span attr | `RCLValidations.cpp` — `handleNewValidation` | Trusted Validation Accept Rate by Status | One span per **trusted** validation as it reaches the ledger-acceptance gate, so its rate is bounded by the UNL size per ledger close (untrusted validations cannot move acceptance and get no span). Its trace id is derived from the **validated ledger's** hash, so it joins that ledger's trace rather than the round trace — see the per-ledger trace join below. `validation_status` is one of the six `ValStatus` values and only `current` continues to the gate, which is the difference between a node whose arriving validations are counting and one whose validations are all rejected; from the outside both look like a node that receives validations and never validates. `accept_gated` is true when another thread was already accepting the same ledger, which is why a trace can show a validation with no `ledger.validate` after it. Both are spanmetrics dimensions in **both** collector configs (6 and 2 values, bounded); `ledger_hash` / `ledger_seq` stay span-only and Tempo-indexed, since a per-ledger metric dimension would mint one series per ledger. | -| Per-ledger trace join (`ledger_hash` as trace-id seed) | trace scheme | `LedgerMaster.h/.cpp` — `LedgerMaster::makeLedgerTraceSpan` | n/a (read in Tempo: `{span.ledger_hash="<hash>"}`) | Makes one slow ledger readable as **one connected trace** instead of a set of orphan spans on different threads. `ledger.validate` (`LedgerMaster::checkAccept`), `ledger.store` (`LedgerMaster::storeLedger`) and `consensus.validation.accept` (`handleNewValidation`) each derive their trace id from the **same 32-byte ledger hash** via `SpanGuard::hashSpan`, which seeds the trace id from `hash[0:16]`. Nothing is propagated between the threads: every one of those sites already holds the ledger hash, which is the whole reason the key was chosen — the same pattern the apply pipeline uses to join `tx.preflight` / `tx.preclaim` / `tx.transactor` on the transaction id (`libxrpl/tx/applySteps.cpp`). Each span is a **true root** (deterministic trace id, empty parent), so the ledger's spans are siblings in one trace rather than a parent/child chain, which is the honest shape: none causes another directly and their order varies with the sync path (`checkAccept` is entered from a peer thread via `handleNewValidation`, from the acquire-completion job, and from the consensus thread via `switchLCL`). The full hash is also recorded as the `ledger_hash` attribute — it is what an operator searches by, and it is how a reader confirms two spans are genuinely the same ledger rather than a trace-id coincidence, since the trace id is only the leading 16 bytes. Asserted end-to-end by the `trace_join_groups` block in `expected_spans.json` (`assert_trace_join_groups` in `validate_telemetry.py`), which fails CI if the members stop sharing a trace. | -| `ledger.acquire.header` span (`outcome`, `timed_out`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::syncPhaseSpans` / `endPhaseSpan` | Ledger Acquire Phase Duration (p95 by phase); Ledger Acquire Phase Outcomes | Child of `ledger.acquire` covering the wait for the ledger header, which gates both tree phases — until it arrives the account-state and transaction root hashes are unknown, so nothing else can even be requested. The parent span is flat and its duration is dominated by the state tree, so a node stuck waiting to be TOLD what to fetch was indistinguishable from one stuck fetching it. No `missing_nodes`: a header is a single object, not a tree. Opened and closed by one idempotent state sync over the `have*_` flags rather than by open/close calls scattered through the fetch code, so the span boundary cannot drift out of step with the real phase boundary. | -| `ledger.acquire.astree` span (`outcome`, `timed_out`, `missing_nodes`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::syncPhaseSpans` / `endPhaseSpan` | Ledger Acquire Phase Duration (p95 by phase); Ledger Acquire Phase Outcomes | Child of `ledger.acquire` covering the account-state SHAMap fetch — **nearly all of the work in a real fresh sync**, and the reason the phase split exists: the flat parent span could not separate it from the small transaction tree. `missing_nodes` is read from the count `getMissingNodes()` already produced during its sweep, never recomputed, so no second tree walk is added. `outcome=timeout` together with a non-zero `missing_nodes` is the "peers are not serving this tree" signature; `timed_out` is a separate dimension from `outcome` because a phase can time out and still be retried by its parent acquire. | -| `ledger.acquire.txtree` span (`outcome`, `timed_out`, `missing_nodes`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::syncPhaseSpans` / `endPhaseSpan` | Ledger Acquire Phase Duration (p95 by phase); Ledger Acquire Phase Outcomes | Child of `ledger.acquire` covering the transaction SHAMap fetch. Usually completes long before the account-state phase, and that asymmetry is the point of separating them: the parent span's duration is the state tree's, not this one's, so a transaction tree that is genuinely slow is invisible inside it. Closed the moment its own tree completes (from `receiveNode`, `trigger` or `takeHeader`), so its duration is the real fetch time rather than stretching to the next trigger. | -| `txset.acquire` span (`outcome`, `txset_hash`, `duration_ms`, `timeouts`, `peer_count`) | span | `TransactionAcquire.cpp` — `TransactionAcquire::finalizeAcquireSpan` | Tx-Set Acquire Outcomes; Tx-Set Acquire Duration (p95) | One attempt to fetch the transaction set a consensus proposal referenced but this node did not hold. `TransactionAcquire` had **zero** telemetry of any kind before this, so a consensus round stalled waiting on a set was indistinguishable from an idle one. The sibling of `ledger.acquire`: same `TimeoutCounter` base, same trigger/onTimer/takeNodes shape, and the same `trace_ledger` flag so the two halves of a stuck sync cannot be enabled apart. `outcome` is `complete` \| `failed` \| `timeout` \| `abandoned`, stamped on both exits (`done()`, and the destructor when the round sweep in `InboundTransactions::newRound` drops a set that never arrived) by one idempotent finalizer. `timeout` is distinct from `failed` because the exhausted-budget path sets the terminal `failed_` flag too — that flag is how the timer loop stops — so the outcome rule checks the timeout first or every timeout would read as a data fault. `txset_hash` identifies WHICH set stalled and stays span-only: one metric series per consensus round would be unbounded. | -| `ledger.serve` span (`object_type`, `outcome`, `served_nodes`, `peer_id`, `ledger_seq`) | span | `PeerImp.cpp` — `PeerImp::processLedgerRequest` (the `JtLedgerReq` worker) | Ledger Serve Rate by Object Type | This node answering a peer's `TMGetLedger` request — the **supply side** of the sync exchange, and the trace-level companion to the existing `serve_refused_total` counter. The whole serve path had no span, so how long this node takes to answer, and whether it answered at all, was unobservable. A fresh trace root, because the request arrives from the wire on a shared worker whose ambient span is unrelated. `object_type` (`header` \| `tx` \| `as` \| `txset`) and `outcome` (`complete` \| `partial` \| `refused`) are both derived by shared rules in `LedgerSpanNames.h` rather than named per branch, which is what stops the eight exits of `processLedgerRequest` disagreeing about one request. `outcome` is derived from the reply itself — `served_nodes` is the reply's own node count and is 0 on all seven refusal paths — so nothing is accumulated and no work is added to the per-node assembly loop. `partial` means the reply hit `Tuning::kSoftMaxReplyNodes`, so the peer must make another round trip. | -| `peer.dial` span (`outcome`, `remote_endpoint`, `duration_ms`) | span | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Outcomes (span-derived, per attempt) | One outbound connect attempt, as a per-attempt timeline rather than a rate. The trace-level companion to `overlay_connect_total` / `overlay_dial_latency_ms`: it carries the same five `outcome` values, set from the same `reportOutcome` funnel, so span and counter cannot disagree, and the funnel's existing first-call-wins guard makes the span exactly-once for free. What it adds is `remote_endpoint` — WHICH peer — which the counter deliberately cannot carry, because one series per peer address would be unbounded cardinality; it is a dedicated Tempo span column instead. A fresh trace root: a dial is the first thing a starting node does, so there is nothing to parent it to. An attempt torn down mid-dial by shutdown ends its span in the destructor with no `outcome`, which is the honest record of "never concluded" rather than a dropped span. | +| Signal | Type | Emit site | Panel | Meaning | +| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `dns_resolve_total` (`outcome` = `resolved` \| `empty`) | counter | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Outcome Rate | Peer hostname resolutions. `empty` means a configured bootstrap or `[ips_fixed]` name returned no address, so that peer is never dialled. | +| `dns_resolve_latency_ms` | histogram | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Latency (p95) | Time to resolve a configured peer hostname. Seconds-scale values mean the resolver is timing out ahead of every dial. | +| `overlay_connect_total` (`outcome` = `connected` \| `tcp_fail` \| `tls_fail` \| `upgrade_fail` \| `timeout`) | counter | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Outcome Rate | Outbound peer connection attempts by terminal outcome. The outcome names the stage that broke: TCP, TLS, HTTP upgrade, or no terminal state in time. | +| `overlay_dial_latency_ms` | histogram | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Latency (p95) | Time from starting an outbound dial to its terminal outcome, successes and failures together. A p95 near the dial timeout means peers accept TCP but never finish the handshake. | +| `handshake_negotiation_fail_total` (`reason`, 14 values incl. `wrong_network`, `invalid_network_id`, `clock_skew`, `self_connection`, `session_verify_failed`) | counter | `Handshake.cpp` — `throwNegotiationFailure` (from `verifyHandshake`) | Handshake Negotiation Failures by Reason | Peer handshakes rejected after TLS while checking network id, clock, keys and addresses. `reason` names the failing check. | +| `unl_fetch_total` (`site` = configured UNL URI; `outcome` = the 9 `ListDisposition` strings `accepted` \| `expired` \| `same_sequence` \| `pending` \| `known_sequence` \| `unsupported_version` \| `untrusted` \| `stale` \| `invalid`, plus `fetch_error` \| `bad_status` \| `parse_error`) | counter | `ValidatorSite.cpp` — `ValidatorSite::reportFetchOutcome` | UNL Fetch Rate by Site & Outcome | Validator-list fetches per site. `accepted` is the only success; `same_sequence` and `known_sequence` are normal no-op refreshes; the three literals are transport or content faults. | +| `unl_quorum` (`metric` = `trusted_keys` \| `quorum`) | observable gauge | `MetricsRegistry.cpp` — `registerUnlQuorumGauge` | UNL Trusted Keys vs Quorum; UNL Quorum Headroom | Trusted UNL key count against the validations a ledger needs. `trusted_keys` at or below `quorum` means the node can never declare a ledger validated. | +| `clock_close_offset_seconds` (`metric` = `offset`) | observable gauge | `MetricsRegistry.cpp` — `registerClockSkewGauge` | Clock Close Offset | Network close time offset from the local clock. Negative means the local clock runs ahead. `server_info` only surfaces `close_time_offset` at 60 s or more, so this gauge sees skew far earlier. | +| `state_changes_total` (`from`, `to` = `disconnected` \| `connected` \| `syncing` \| `tracking` \| `full`) | counter | `NetworkOPs.cpp` — `NetworkOPsImp::setMode` | Mode Transitions by Edge | Operating-mode transitions keyed on the (`from`, `to`) edge. The edge is what separates a clean `disconnected`→`connected`→`syncing`→`tracking`→`full` climb from `full`→`connected` flapping; an unlabelled total cannot tell them apart. | +| `sync_state` (`metric` = `initial_full_duration_us`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Time to First FULL | Microseconds from process start to the first `full` transition, sourced from `NetworkOPs::getInitialSyncDurationUs()`. Stays 0 until `full` is reached, so a flat 0 is itself the "never synced" signal; once set it never changes. | +| `sync_state` (`metric` = `network_ledger_gate`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Network Ledger Gate | 1 while the node is still waiting to see a full network ledger (`NetworkOPs::isNeedNetworkLedger()`), else 0. A persistent 1 blocks transaction submission and `full`, whatever the rest of the pipeline shows. | +| `sync_state` (`metric` = `server_stall_seconds`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Server Stall | Current main-loop stall duration from `LoadManager::getCurrentStallSeconds()`, 0 when healthy. Same duration the load monitor logs as "Server stalled for N seconds", which previously existed only in that log line. | +| `sync_state` (`metric` = `ledgers_behind`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Ledgers Behind Network | Peer-reported network tip minus our validated sequence, floored at 0 (`NetworkOPs::getLedgersBehindNetwork()`). Reads each peer's already cached ledger range, so no new network round trip. | +| `server_stall_events_total` | observable counter | `MetricsRegistry.cpp` — `registerStallEventsCounter` | Server Stall Event Rate | Distinct stall episodes since process start, counted once per episode rather than per stalled second. A rising rate is repeated fresh stalls; a flat rate with a large `server_stall_seconds` is one long stall. | +| `sync_acquire` (`metric` = `missing_state_nodes_max` \| `missing_tx_nodes_max`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Missing SHAMap Nodes per Acquire (state/tx) | Largest outstanding SHAMap node count across in-flight acquires, split by tree, from the count `getMissingNodes()` already produces during its sweep (`InboundLedger.cpp` — `InboundLedger::trigger`). **The headline stuck-sync signal:** flat and non-zero across ticks means the acquire will never finish; shrinking means slow but alive. Aggregated as a max rather than labelled per ledger, because a `ledger_seq` label would mint one series per ledger acquired — per-ledger identity stays on the `ledger.acquire` span. | +| `sync_acquire` (`metric` = `received_data_depth`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Peer packets stashed across all in-flight acquires waiting to be applied, summed because it measures one shared processing backlog. A growing depth means arriving node data outpaces processing, so the limit is the job queue or disk rather than peer supply. | +| `sync_acquire` (`metric` = `in_flight`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Number of ledger acquires currently running. Exported so the three values above can be read in context: all zero with `in_flight` zero is an idle node, not a healthy one. | +| `shamap_cache_hit_rate` (`metric` = `treenode`) | observable gauge | `MetricsRegistry.cpp` — `registerCacheHitRateDetailGauge` | SHAMap TreeNode Cache Hit Rate | Share of SHAMap tree-node lookups served from memory, from the previously-uncalled `TaggedCache::getHitRate()`, normalized from 0-100 to 0.0-1.0. Distinct from `nodestore_state`-derived NuDB Cache Hit Ratio on the Ledger Data Sync dashboard: this is the in-memory layer **above** the node store, so a miss here is what causes a read there. The full-below cache is not reported — it is a `KeyCache` whose only lookup path increments `stats_.hits`/`stats_.misses` while `getHitRate()` reads the separate `hits_`/`misses_` members, so its rate is hard-wired to 0 until that accounting is fixed. | +| `sync_acquire_no_progress_total` | counter | `InboundLedger.cpp` — `InboundLedger::onTimer` | Acquire Stall Rate (no progress) | Acquire timeouts where not one new node arrived since the previous timeout, from the `progress_` flag that was previously log-only. Fires on the 3 s acquire timer, never per node. A sustained rate together with a flat missing-node count is the definitive "stuck, not slow" signature. | +| `sync_addnode_total` (`outcome` = `good` \| `duplicate` \| `invalid`) | counter | `InboundLedger.cpp` — `InboundLedger::recordBatchOutcome` | Add-Node Outcomes | SHAMap nodes received during acquire, split by result. Emitted once per received packet from the aggregated batch tally the trace log already printed — never inside the per-node `receiveNode()` loop. Separates real progress (`good`) from wasted bandwidth (`duplicate`) and a misbehaving peer (`invalid`), all three of which look like healthy throughput in traffic metrics. | +| `sync_acquire_source_total` (`source` = `local` \| `network`) | counter | `InboundLedger.cpp` — `InboundLedger::init` | Acquire Source (local vs network) | Whether an acquire was satisfied entirely from the local node store or needed peers, emitted once per new acquire after the first local lookup. Sustained `network` on a node that should already hold the range means sync is disk-bound rather than peer-bound. | +| `jobq_backlog` (`metric` = `waiting` \| `running` \| `deferred`; `job_type` = the `JobTypes::name()` string) | observable gauge | `MetricsRegistry.cpp` — `registerJobQueueBacklogGauge` | Deferred Jobs by Type (starvation); Job Queue Occupancy by Type (waiting/running) | Instantaneous per-job-type queue occupancy, from `JobQueue::getJobTypeCounts()` (one mutex acquire per ~10 s tick). **`deferred` is the signal this adds:** jobs the queue accepted but withheld because the type is already at its concurrency limit, which is counted in neither `waiting` nor `running` and had no exposure anywhere before. The sync-critical types are capped at 3 (`JtLedgerReq`, `JtLedgerData` in `JobTypes.h`), so they starve first. Distinct from the existing `job_queued_total` / `job_started_total` / `job_finished_total` counters and `job_queued_us` / `job_running_us` histograms, which are event-driven from PerfLogImp and describe jobs that already moved, and from the StatsD `jobq_job_count`, which is queue-wide with no per-type split. Cardinality is bounded by the JobType enum (~46 values); every type is observed every tick, so an idle type reports 0 rather than dropping its series. | +| `jobq_saturation` (`metric` = `running_tasks` \| `worker_threads` \| `total_waiting`) | observable gauge | `MetricsRegistry.cpp` — `registerJobQueueSaturationGauge` | Worker Pool Saturation; Worker Pool Capacity & Total Backlog | Global worker-pool saturation from `JobQueue::getWorkerSaturation()`: tasks in flight, threads the pool is configured to run, and jobs queued across all types, all from one reading so the ratio and the backlog describe the same instant. `worker_threads` is exported rather than hardcoded in the dashboard because it is derived at startup from `[workers]`, node size and hardware concurrency. Exists separately from `jobq_backlog` because a pool-wide slowdown otherwise appears as an independent fault in every subsystem queued behind it; a `running_tasks / worker_threads` ratio at 1.0 **with** a non-zero `total_waiting` attributes it to pool exhaustion once. | +| `peer_ledger_supply` (`metric` = `peers_reporting` \| `peers_serving_validated` \| `peers_serving_next` \| `supply_min_seq` \| `supply_max_seq`) | observable gauge | `MetricsRegistry.cpp` — `registerPeerLedgerSupplyGauge` (aggregating `OverlayImpl::getPeerLedgerSupply`) | Peers Able to Serve Needed Sequence; Peer Ledger Supply Window | How much of the sequence range this node needs its connected peer set can actually serve, from one pass over the active peers reading the range each already advertised in `mtSTATUS_CHANGE`. **`peers_serving_next` is the signal this exists for:** zero there with a non-zero `peers_reporting` means no connected peer holds validated + 1, so the peer set must change and waiting cannot finish the sync. `peers_reporting` is the denominator that makes the rest readable — peers advertising `[0, 0]` have not reported yet and are excluded from every field, so they cannot make a healthy peer set appear to serve from genesis; when nothing has reported, both window fields read 0 meaning **unknown**, not genesis. `supply_min_seq` / `supply_max_seq` separate "asking for history nobody kept" from "asking for a tip nobody reached". Distinct from `server_info{metric="peers"}`, a bare connection count with no notion of what those peers hold; from `sync_state{metric="ledgers_behind"}`, which uses the same per-peer maxima but collapses them to a single distance-to-tip number that cannot say how many peers can serve that distance or whether the range has a hole; and from `peer_quality{metric="peers_insane_count"}`, which counts peers on a different chain and is therefore a correctness signal, not an availability one. | +| `peer_disconnect_total` (`reason` = `graceful` \| `shutdown` \| `stopping` \| `read_error` \| `write_error` \| `timer_error` \| `ping_timeout` \| `not_useful` \| `large_sendq` \| `charge_resources` \| `malformed_handshake` \| `shared_value` \| `unknown`; `direction` = `inbound` \| `outbound`) | counter | `PeerImp.cpp` — `PeerImp::close` | Peer Disconnects by Reason | Peer teardowns split by cause and by which side opened the connection. Emitted once per teardown at `close()`, the single funnel every disconnect path passes through, and `close()` already self-guards on the socket being open, so a repeated close cannot double-count and the total matches the existing unlabelled tally. `reason` is set by whichever site decided to disconnect, first writer wins, so a later generic reason never masks the real one; the value is always one of a fixed set of literals in `PeerImp.cpp`, never peer-supplied data, so cardinality is bounded by the code. The split is the whole point: it separates our-fault backpressure (`large_sendq`, `charge_resources`) from topology and network faults (`not_useful`, `ping_timeout`, `read_error`), and normal churn (`graceful`) from either. Distinct from the existing `server_info{metric="peer_disconnects_resources"}`, which counts only the resource-charge subset and carries no labels, and from the StatsD `overlay_peer_disconnects`, which is the unlabelled grand total in which every reason above collapses into one number. | +| `peer_accept_total` (`outcome` = `accepted` \| `local_endpoint_fail` \| `resource_limit` \| `no_slot` \| `not_peer_request` \| `protocol_mismatch` \| `bad_cookie` \| `slot_refused` \| `handshake_error`) | counter | `OverlayImpl.cpp` — `OverlayImpl::onHandoff` via `reportAcceptOutcome` | Inbound Peer Accept Outcomes | Terminal outcome of every inbound connection this node is offered, one emit per handoff. `accepted` is reported only after `run()`, so anything that threw on the way lands on `handshake_error` instead; the two early returns that are not peer attempts at all (a handled HTTP request, and a request that never asked to upgrade) are deliberately not counted. The `outcome` names the stage that refused: no local endpoint, the resource manager, PeerFinder having no slot or seeing a duplicate, a non-peer upgrade request, protocol version disagreement, a bad security cookie, or activation being refused. This is the **inbound twin** of the existing `overlay_connect_total{outcome}`, which covers outbound dials only; without it a node refusing every inbound connection is indistinguishable from one nobody dials, and reading the two together gives the full in/out split. | +| `peerfinder_slot_census` (`metric` = `out_active` \| `out_max` \| `in_active` \| `in_max` \| `connecting` \| `fixed_configured` \| `fixed_active` \| `bootcache` \| `livecache`) | observable gauge | `MetricsRegistry.cpp` — `registerSlotCensusGauge` (from `Logic::getSlotCensus`) | PeerFinder Slot Census; PeerFinder Address Caches & Fixed Peers | Slot occupancy against capacity, outbound dials in flight, configured-versus-connected fixed peers, and the depth of both address caches. All nine come from a single acquire of the PeerFinder lock, so they are mutually consistent, share one label set and can be compared against each other. That is what makes the three most common bootstrap failures visible: `connecting` non-zero while `out_active` stays below `out_max` (dials starting and never completing), `bootcache` and `livecache` both at 0 (nothing to dial at all), and `fixed_active` below `fixed_configured` (a peer named in the configuration is unreachable). `fixed_configured` is the count of peers named in the config, so the pair an operator reads is "how many did I ask for" against "how many do I have" — the same comparison `autoconnect()` makes. All nine values already existed inside PeerFinder; only two of them were exported, as the legacy beast::insight gauges `peer_finder_active_inbound_peers` and `peer_finder_active_outbound_peers`. Those two carry no capacity, attempt or cache term, are read at unrelated instants, and so cannot be joined with each other let alone with a capacity term — leaving all three failures above indistinguishable from a node that is simply not dialling. | +| `serve_refused_total` (`request` = `ledger` \| `txset` \| `object` \| `fetchpack`; `reason` = `sendq_full` \| `load_shed` \| `not_found` \| `no_map` \| `bad_type` \| `empty_reply`) | counter | `PeerImp.cpp` — `processLedgerRequest`, `onMessage(TMGetObjectByHash)`, `doFetchPack` | Ledger/Object Serve Refusals | Peer data requests this node declined to answer, split by what was asked for and why. This is the **supply side** of the sync exchange — what this node refuses to serve OTHERS — and nothing equivalent existed before, so a node shedding every ledger request looked identical to one being asked for nothing. `sendq_full` and `load_shed` are self-inflicted backpressure (the send queue at `Tuning::kDropSendQueue`, or the local fee track loaded, or too many pack jobs queued), while `not_found` is a genuine history gap and `no_map` / `bad_type` / `empty_reply` mean the request was answerable in principle but produced nothing to send. `fetchpack` is counted apart from `ledger` because a fetch pack is how a syncing peer catches up in bulk and its shed threshold is a different one. Emitted at most once per request — `empty_reply` is reported after the node loop, never inside it — and both labels are code literals, so cardinality is bounded at compile time. | +| `amendment_block` (`metric` = `warned` \| `seconds_to_block`) | observable gauge | `MetricsRegistry.cpp` — `registerAmendmentBlockGauge` | Amendment Block Countdown; Amendment Warned | `warned` is 1 once an unsupported amendment has reached majority (`NetworkOPs::isAmendmentWarned()`, previously only an admin-only `server_info` warning). **`seconds_to_block` is the leading indicator:** seconds until that amendment activates, from `AmendmentTable::firstUnsupportedExpected()` against the network close time. It reads `-1` when nothing is pending — a distinct healthy value rather than a missing series, matching the sentinel `validator_health{metric="unl_expiry_days"}` already uses — and is clamped at 0 rather than going negative, because past-due means the block is imminent, not overdue by some amount worth charting; the subtraction is done in `std::int64_t` so a past-due activation cannot wrap. Amendment-blocked is a terminal sync blocker: the node stops validating and never resumes without a software upgrade. The existing `validator_health{metric="amendment_blocked"}` reports that state after the fact, when nothing can be done about it; this gauge is the window before it, which is the only actionable part. The blocking amendment's identity is deliberately **not** a label — the network can vote on an arbitrary 256-bit amendment id, not drawn from this build's known features, so an id label would be unbounded cardinality and would mint a permanent new series per amendment. The id is available in logs from `AmendmentTableImpl::doValidatedLedger` ("Unsupported amendment \ reached majority at ..."), correlated to this series by node and time. | +| `ledger_jump_total` | counter | `NetworkOPs.cpp` — `NetworkOPsImp::switchLastClosedLedger` | Byzantine Ledger Jumps | Forced jumps of the last closed ledger onto a divergent chain: the node was told the network's LCL is not the one it built on and discarded its own chain tip to follow. Nothing equivalent existed — this was log-only ("JUMP last closed ledger to ..."), so a node repeatedly thrashing between chains left no time series to correlate against the rest of the sync pipeline. Any non-zero rate is abnormal by construction; repeated jumps are wrong-chain thrash, which points at the peer set and the configured network id rather than anywhere in the acquire pipeline. Deliberately unlabelled: the ledger hash and sequence would both be unbounded as label values, and the log line beside the emit already carries them. | +| `nodestore_latency` (`metric` = `write_mean_us` \| `read_mean_us` \| `write_count` \| `read_count`) | observable gauge | `MetricsRegistry.cpp` — `registerNodeStoreLatencyGauge` | NodeStore Write vs Read Latency (us/op); NodeStore Operation Rate (writes vs reads) | Mean microseconds per node-store store and per fetch, with both operation counts so a panel can divide the two rates and read _interval_ latency instead of the since-boot average. **The write side is the new signal.** `storeDurationUs_` was declared in `Database.h` and never written, and no accessor existed, so no write-path latency was observable anywhere; the read total was already exposed as `nodestore_state{metric="node_reads_duration_us"}`. This is the fingerprint of the "a node with a large existing DB syncs slower than a fresh one" symptom, which is write-bound and therefore invisible in every read-side metric. Chosen as a gauge over a histogram deliberately: a histogram gives true percentiles but costs one `Record()` per node object on the store/fetch path, and a single ledger write walks thousands of SHAMap nodes — this gauge instead reads four existing atomics once per ~10 s tick and adds nothing to the hot path. Consequence: **p99 is not obtainable from this signal**, and a histogram added later would also need an explicit-bucket View (`addMicrosecondHistogramView`) because the SDK default buckets top out at 10,000. Distinct from the Ledger Data Sync dashboard's NuDB Read Latency panel, which divides two `nodestore_state` fields in PromQL: that panel has no write-duration input to divide, because the quantity did not exist. **Known gap:** `write_mean_us` is emitted only when the store-duration total is non-zero, and that total is fed by `Database::recordStoreDuration`, today called only from `Database::importInternal` (the `[import_db]` admin path). `Database::store()` is pure virtual and neither `DatabaseNodeImp::store` nor `DatabaseRotatingImp::store` times itself yet, so an ordinary node reports `write_count` with no `write_mean_us`. The mean is omitted rather than reported as 0 so the gap stays visible instead of reading as "writes are instantaneous". | +| `ledger_replay_fallback_total` (`stage` = `skiplist` \| `delta`) | counter | `SkipListAcquire.cpp` / `LedgerDeltaAcquire.cpp` — `trigger` | Replay Fallback to Full Acquire (by stage) | A ledger-replay sub-task abandoning its shortcut and acquiring the whole ledger through `InboundLedger` instead, because too few connected peers support the `LedgerReplay` protocol feature. Both branches were debug-log-only, so a silently defeated replay optimisation left no metric at all — back-fill simply ran on the slower path with nothing to show why. Emitted once, on the transition into fallback, not at the acquire call, which re-runs on every later trigger. The `stage` label separates the skip-list acquire (which fetches the list of historical ledger hashes) from the per-ledger delta acquire, because they fail independently. | +| `ledger_replay_outcome_total` (`outcome` = `success` \| `timeout` \| `build_failed` \| `parameter_failed`) | counter | `LedgerReplayTask.cpp` — `LedgerReplayTask::recordOutcome` | Replay Outcomes (by terminal state) | Terminal state of every ledger-replay task, one emit per task. Every terminal path previously only set an internal `complete_`/`failed_` flag and wrote a log line, so a replay that never succeeded was indistinguishable from one that was never attempted. The outcome names the layer at fault: `timeout` means the deltas never arrived (a peer-supply problem), `build_failed` means a delta would not apply to its parent, and `parameter_failed` means a peer served a skip list inconsistent with what the task asked for — the latter two are data faults, not slowness. Read with `ledger_replay_fallback_total`: fallbacks rising while successes stay flat is replay-based catch-up degrading to full-ledger acquisition. | +| `ledger_quorum_publish` (`metric` = `trusted_validation_tally` \| `quorum_target`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Trusted Validations vs Quorum Target | Trusted validations counted at the most recent pre-accept gate, beside the number that gate required. Snapshotted in `LedgerMaster::checkAccept` before the shortfall check, so a node that keeps failing the gate still reports both numbers — which is the whole point: the tally alone cannot say whether validations are accumulating toward quorum (slow, will finish) or plateaued below it (stuck). Read the sustained floor of the tally, not a single sample: each series is a snapshot of the last evaluation, and the first evaluation of each round runs before peer validations arrive, so a healthy node sawtooths. `quorum_target` is what the gate actually demanded, as opposed to `unl_quorum{metric="quorum"}` which is what the trusted list configures. When the trusted list disables quorum entirely (`getNeededValidations` returns `SIZE_MAX`) the target is reported as int64 max rather than wrapping to -1, so it reads far above any tally instead of inverting the comparison — the same sentinel handling as the `unl_quorum` gauge. | +| `ledger_quorum_publish` (`metric` = `publish_lag`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Publish Lag (validated minus published) | Ledgers fully validated but not yet published to clients and subscribers: the validated sequence minus the published sequence, floored at zero. `pubLedgerSeq_` was never exported, so this gap was not derivable from any other series. Publishing trails validation by design and a small lag drains each round; a lag that stays positive or grows means validation is healthy and the publish pipeline is not, which is a distinct fault from anything the quorum or acquire signals can show. The two sequences are read as independent relaxed loads, so a sample taken mid-update may be off by one ledger for one poll — immaterial for a lag trend, and the price of not taking the LedgerMaster mutex on the metrics poll thread. | +| `ledger_quorum_publish` (`metric` = `time_to_first_validated_us`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Time to First Validated Ledger | Microseconds from process start until the first ledger passed the pre-accept quorum gate. A one-shot measurement like `sync_state{metric="initial_full_duration_us"}`: written once under `mutex_` and never changed, so it has no trend. Exactly two readings are meaningful — a duration, meaning the node reached its first fully-validated ledger and this is how long that took, or 0, meaning it never has. Clamped to a minimum of 1 so a genuine sub-microsecond reading can never be confused with the never-reached zero. A value here alongside a zero on time-to-first-FULL, or the reverse, separates "reached the full server state" from "fully validated a ledger". | +| `ledger_quorum_shortfall_total` (`stage` = `pre_accept`) | counter | `LedgerMaster.cpp` — `LedgerMaster::checkAccept` | Pre-Accept Quorum Shortfall Rate | One increment per pre-accept gate evaluation rejected because the trusted validation tally was below quorum. Previously trace-log-only, which made a node that peers and receives validations yet never validates indistinguishable from an idle one. A non-zero rate is **not** by itself a fault: `doAccept` issues this node's own validation and calls `consensusBuilt` → `checkAccept` immediately, before peer validations for that ledger arrive, so the first evaluation of every round tallies short and is retried as validations come in — a healthy cluster emits this counter every round. The fault signature is the rate climbing well above the ledger-close rate while the tally stays flat below its target and `time_to_first_validated_us` stays at 0. Emitted while `mutex_` is held, which is safe against the metrics poll because every accessor the sync gauges read is a lock-free atomic load, so no OTel callback ever acquires `mutex_`. | +| `consensus_round_duration_ms` | histogram | `RCLConsensus.cpp` — `RCLConsensus::Adaptor::makeAcceptSpan` | Consensus Round Duration Distribution; Consensus Round Duration (p50/p95) | Wall-clock duration of a completed consensus round, in milliseconds. Promotes the long-standing `round_time_ms` span attribute into a native instrument: the attribute answers "how long did THIS round take" inside a trace, next to the proposers and disputes that explain it, while the histogram gives the distribution over time, which is what an alert or SLO panel needs and what a raw trace query cannot cheaply produce fleet-wide. Being native it is also **never sampled**, so it stays complete when tracing is head-sampled down. Recorded at exactly one site — `makeAcceptSpan` is the single function both the synchronous (`onForceAccept`) and asynchronous (`onAccept`) accept paths call once per round — so it can neither double-count nor be skipped, and it adds no per-peer, per-proposal or per-transaction work. **Explicit buckets** are registered for it in `MetricsRegistry::initExporterAndProvider` (`addRoundDurationHistogramView`, boundaries 500 ms → 120 s): the SDK default tops out at 10,000 ms, which would collapse every slow round into one saturated bucket and read every quantile as 10 s, and the consensus parameters themselves allow a round up to `ledgerAbandonConsensus` = 120 s. Needs **no collector change** — a native metric rides the existing OTLP → Prometheus path. | +| `consensus.validation.accept` (`validation_status`, `accept_gated`, `ledger_hash`, `ledger_seq`, `full_validation`) | span + span attr | `RCLValidations.cpp` — `handleNewValidation` | Trusted Validation Accept Rate by Status | One span per **trusted** validation as it reaches the ledger-acceptance gate, so its rate is bounded by the UNL size per ledger close (untrusted validations cannot move acceptance and get no span). Its trace id is derived from the **validated ledger's** hash, so it joins that ledger's trace rather than the round trace — see the per-ledger trace join below. `validation_status` is one of the six `ValStatus` values and only `current` continues to the gate, which is the difference between a node whose arriving validations are counting and one whose validations are all rejected; from the outside both look like a node that receives validations and never validates. `accept_gated` is true when another thread was already accepting the same ledger, which is why a trace can show a validation with no `ledger.validate` after it. Both are spanmetrics dimensions in **both** collector configs (6 and 2 values, bounded); `ledger_hash` / `ledger_seq` stay span-only and Tempo-indexed, since a per-ledger metric dimension would mint one series per ledger. | +| Per-ledger trace join (`ledger_hash` as trace-id seed) | trace scheme | `LedgerMaster.h/.cpp` — `LedgerMaster::makeLedgerTraceSpan` | n/a — read in Tempo, `{span.ledger_hash="LEDGER_HASH"}` | Makes one slow ledger readable as **one connected trace** instead of a set of orphan spans on different threads. `ledger.validate` (`LedgerMaster::checkAccept`), `ledger.store` (`LedgerMaster::storeLedger`) and `consensus.validation.accept` (`handleNewValidation`) each derive their trace id from the **same 32-byte ledger hash** via `SpanGuard::hashSpan`, which seeds the trace id from `hash[0:16]`. Nothing is propagated between the threads: every one of those sites already holds the ledger hash, which is the whole reason the key was chosen — the same pattern the apply pipeline uses to join `tx.preflight` / `tx.preclaim` / `tx.transactor` on the transaction id (`libxrpl/tx/applySteps.cpp`). Each span is a **true root** (deterministic trace id, empty parent), so the ledger's spans are siblings in one trace rather than a parent/child chain, which is the honest shape: none causes another directly and their order varies with the sync path (`checkAccept` is entered from a peer thread via `handleNewValidation`, from the acquire-completion job, and from the consensus thread via `switchLCL`). The full hash is also recorded as the `ledger_hash` attribute — it is what an operator searches by, and it is how a reader confirms two spans are genuinely the same ledger rather than a trace-id coincidence, since the trace id is only the leading 16 bytes. Asserted end-to-end by the `trace_join_groups` block in `expected_spans.json` (`assert_trace_join_groups` in `validate_telemetry.py`), which fails CI if the members stop sharing a trace. | +| `ledger.acquire` span (`outcome` = `complete` \| `failed` \| `abandoned`; `acquire_reason`, `timeouts`, `peer_count`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::init` / `InboundLedger::finalizeAcquireSpan` | Ledger Acquire Phase Duration (p95 by phase) (its three phase children); Ledger Acquire Duration (Inbound Fetch) and Ledger Acquire Rate by Outcome, both on the **node-health** board | Parent of the three phase spans: one whole fetch of one missing ledger, from the first request to the terminal state. Pre-existing since Phase 6, extended here with `ledger_hash` (set at `init()`, so a fetch that never finishes is still findable in a trace search, and it is the trace-id seed that joins this span to the `ledger.validate`, `ledger.store` and `consensus.validation.accept` spans for the same ledger) and with the fourth `outcome` value `abandoned`, recorded when the acquire is destroyed by a sweep or shutdown before reaching a result. Without `abandoned` a stuck-then-swept fetch left the span with no `outcome` at all, so it vanished from every outcome rate — the exact failure a stalled fresh sync produces. `ledger_seq` is re-stamped at the end because a by-hash acquire starts with `seq_ == 0` and learns the sequence only when the header arrives. | +| `ledger.acquire.header` span (`outcome`, `timed_out`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::syncPhaseSpans` / `endPhaseSpan` | Ledger Acquire Phase Duration (p95 by phase); Ledger Acquire Phase Outcomes (by phase & timeout) | Child of `ledger.acquire` covering the wait for the ledger header, which gates both tree phases — until it arrives the account-state and transaction root hashes are unknown, so nothing else can even be requested. The parent span is flat and its duration is dominated by the state tree, so a node stuck waiting to be TOLD what to fetch was indistinguishable from one stuck fetching it. No `missing_nodes`: a header is a single object, not a tree. Opened and closed by one idempotent state sync over the `have*_` flags rather than by open/close calls scattered through the fetch code, so the span boundary cannot drift out of step with the real phase boundary. | +| `ledger.acquire.astree` span (`outcome`, `timed_out`, `missing_nodes`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::syncPhaseSpans` / `endPhaseSpan` | Ledger Acquire Phase Duration (p95 by phase); Ledger Acquire Phase Outcomes (by phase & timeout) | Child of `ledger.acquire` covering the account-state SHAMap fetch — **nearly all of the work in a real fresh sync**, and the reason the phase split exists: the flat parent span could not separate it from the small transaction tree. `missing_nodes` is read from the count `getMissingNodes()` already produced during its sweep, never recomputed, so no second tree walk is added. `outcome=timeout` together with a non-zero `missing_nodes` is the "peers are not serving this tree" signature; `timed_out` is a separate dimension from `outcome` because a phase can time out and still be retried by its parent acquire. | +| `ledger.acquire.txtree` span (`outcome`, `timed_out`, `missing_nodes`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::syncPhaseSpans` / `endPhaseSpan` | Ledger Acquire Phase Duration (p95 by phase); Ledger Acquire Phase Outcomes (by phase & timeout) | Child of `ledger.acquire` covering the transaction SHAMap fetch. Usually completes long before the account-state phase, and that asymmetry is the point of separating them: the parent span's duration is the state tree's, not this one's, so a transaction tree that is genuinely slow is invisible inside it. Closed the moment its own tree completes (from `receiveNode`, `trigger` or `takeHeader`), so its duration is the real fetch time rather than stretching to the next trigger. | +| `txset.acquire` span (`outcome`, `txset_hash`, `duration_ms`, `timeouts`, `peer_count`) | span | `TransactionAcquire.cpp` — `TransactionAcquire::finalizeAcquireSpan` | Tx-Set Acquire Outcomes; Tx-Set Acquire Duration (p95) | One attempt to fetch the transaction set a consensus proposal referenced but this node did not hold. `TransactionAcquire` had **zero** telemetry of any kind before this, so a consensus round stalled waiting on a set was indistinguishable from an idle one. The sibling of `ledger.acquire`: same `TimeoutCounter` base, same trigger/onTimer/takeNodes shape, and the same `trace_ledger` flag so the two halves of a stuck sync cannot be enabled apart. `outcome` is `complete` \| `failed` \| `timeout` \| `abandoned`, stamped on both exits (`done()`, and the destructor when the round sweep in `InboundTransactions::newRound` drops a set that never arrived) by one idempotent finalizer. `timeout` is distinct from `failed` because the exhausted-budget path sets the terminal `failed_` flag too — that flag is how the timer loop stops — so the outcome rule checks the timeout first or every timeout would read as a data fault. `txset_hash` identifies WHICH set stalled and stays span-only: one metric series per consensus round would be unbounded. | +| `ledger.serve` span (`object_type`, `outcome`, `served_nodes`, `peer_id`, `ledger_seq`) | span | `PeerImp.cpp` — `PeerImp::processLedgerRequest` (the `JtLedgerReq` worker) | Ledger Serve Rate by Object Type | This node answering a peer's `TMGetLedger` request — the **supply side** of the sync exchange, and the trace-level companion to the existing `serve_refused_total` counter. The whole serve path had no span, so how long this node takes to answer, and whether it answered at all, was unobservable. A fresh trace root, because the request arrives from the wire on a shared worker whose ambient span is unrelated. `object_type` (`header` \| `tx` \| `as` \| `txset`) and `outcome` (`complete` \| `partial` \| `refused`) are both derived by shared rules in `LedgerSpanNames.h` rather than named per branch, which is what stops the eight exits of `processLedgerRequest` disagreeing about one request. `outcome` is derived from the reply itself — `served_nodes` is the reply's own node count and is 0 on all seven refusal paths — so nothing is accumulated and no work is added to the per-node assembly loop. `partial` means the reply hit `Tuning::kSoftMaxReplyNodes`, so the peer must make another round trip. | +| `peer.dial` span (`outcome`, `remote_endpoint`, `duration_ms`) | span | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Outcomes (span-derived, per attempt) | One outbound connect attempt, as a per-attempt timeline rather than a rate. The trace-level companion to `overlay_connect_total` / `overlay_dial_latency_ms`: it carries the same five `outcome` values, set from the same `reportOutcome` funnel, so span and counter cannot disagree, and the funnel's existing first-call-wins guard makes the span exactly-once for free. What it adds is `remote_endpoint` — WHICH peer — which the counter deliberately cannot carry, because one series per peer address would be unbounded cardinality; it is a dedicated Tempo span column instead. A fresh trace root: a dial is the first thing a starting node does, so there is nothing to parent it to. An attempt torn down mid-dial by shutdown ends its span in the destructor with no `outcome`, which is the honest record of "never concluded" rather than a dropped span. | diff --git a/docker/telemetry/grafana/dashboards/ledger-sync-health.json b/docker/telemetry/grafana/dashboards/ledger-sync-health.json index a9946e5f26..aaea1f051e 100644 --- a/docker/telemetry/grafana/dashboards/ledger-sync-health.json +++ b/docker/telemetry/grafana/dashboards/ledger-sync-health.json @@ -40,9 +40,6 @@ "links": [], "panels": [ { - "id": 1, - "type": "row", - "title": "Bootstrap (Domain 0)", "collapsed": false, "gridPos": { "h": 1, @@ -50,7 +47,10 @@ "x": 0, "y": 0 }, - "panels": [] + "id": 55, + "panels": [], + "title": "Bootstrap (Domain 0)", + "type": "row" }, { "datasource": { @@ -964,9 +964,6 @@ "type": "timeseries" }, { - "id": 2, - "type": "row", - "title": "Sync pipeline", "collapsed": false, "gridPos": { "h": 1, @@ -974,29 +971,63 @@ "x": 0, "y": 61 }, - "panels": [] + "id": 56, + "panels": [], + "title": "Peer supply", + "type": "row" }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Whether the node is still waiting to see a full network ledger before it will participate.*\n\n###### How it's computed:\n*sync_state series network_ledger_gate: 1 while the gate is closed, 0 once it opens.*\n\n###### Reading it:\n*0 (green) is healthy. A persistent 1 (red) means the node has never seen a complete network ledger, so it refuses transactions and can never reach full no matter how healthy the rest of the pipeline looks.*\n\n###### Healthy range:\n*0 within the first few minutes of startup.*\n\n###### Watch for:\n*A 1 that never clears. Pair it with the Bootstrap row \u2014 no peers or no quorum is the usual cause.*\n\n###### Keywords:\n- **Network ledger gate** *(per node)* \u2014 the startup guard that holds a node back until it has seen a full ledger from the network.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerSyncStateGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#network-ledger-gate)", + "description": "###### What this is:\n*How many connected peers can actually serve the ledger this node needs next, next to how many advertise a ledger range at all.*\n\n###### How it's computed:\n*peer_ledger_supply series peers_reporting, peers_serving_validated and peers_serving_next, counted from the ledger ranges connected peers advertise and compared against this node's own validated sequence.*\n\n###### Reading it:\n*peers_serving_next is the line that matters: it counts peers holding validated+1, the one ledger this node must acquire before it can advance. peers_serving_validated counts peers that can re-serve the ledger already held, and peers_reporting is the denominator \u2014 peers that advertised any range at all.*\n\n###### Healthy range:\n*peers_serving_next at or near peers_reporting.*\n\n###### Watch for:\n*peers_serving_next at 0 while peers_reporting is above 0. No peer this node is connected to holds the ledger it needs, so this is a supply problem, not a slow-peer problem \u2014 retrying harder or waiting longer cannot fix it, the node needs different peers. Distinct from the acquire panels above, which measure requests already sent to peers that did hold the data.*\n\n###### Keywords:\n- **Peer ledger supply** *(per node)* \u2014 the count of connected peers whose advertised ledger range covers a sequence this node needs.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerPeerLedgerSupplyGauge`\n\n###### References:\n[Complete ledger ranges](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#complete-ledger-ranges) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#fresh-node-sync-diagnostics)", "fieldConfig": { "defaults": { "color": { - "mode": "thresholds" + "mode": "palette-classic" }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Peers", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null - }, - { - "color": "red", - "value": 1 } ] }, @@ -1005,25 +1036,30 @@ }, "gridPos": { "h": 12, - "w": 8, + "w": 12, "x": 0, "y": 62 }, - "id": 12, + "id": 28, "options": { - "colorMode": "value", - "graphMode": "none", - "justifyMode": "center", - "orientation": "auto", - "percentChangeColorMode": "standard", - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false + "annotations": { + "clustering": -1, + "multiLane": false }, - "showPercentChange": false, - "textMode": "value_and_name", - "wideLayout": true + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } }, "pluginVersion": "13.2.0-28926505616", "targets": [ @@ -1032,12 +1068,520 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sync_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"network_ledger_gate\"}, \"series\", \"Gate Closed\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(peer_ledger_supply{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=~\"peers_reporting|peers_serving_validated|peers_serving_next\"}, \"series\", \"$1\", \"metric\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "A" } ], - "title": "Network Ledger Gate", - "type": "stat" + "title": "Peers Able to Serve Needed Sequence", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*The lowest and highest ledger sequence any connected peer offers \u2014 the sequence window the peer set can actually serve from.*\n\n###### How it's computed:\n*peer_ledger_supply series supply_min_seq and supply_max_seq: the minimum and maximum over the ledger ranges advertised by all connected peers.*\n\n###### Reading it:\n*supply_max_seq is the network tip as this node's peers report it. supply_min_seq is the oldest history reachable from the current peer set; anything below it cannot be back-filled from these peers at all.*\n\n###### Healthy range:\n*supply_max_seq advancing with the network, supply_min_seq well below this node's own validated sequence.*\n\n###### Watch for:\n*supply_min_seq sitting above this node's validated sequence \u2014 every connected peer has already pruned the history this node still needs, so the gap can never be closed from this peer set. Pair with Peers Able to Serve Needed Sequence: that panel says how many peers can serve the next ledger, this one says which sequences are on offer at all.*\n\n###### Keywords:\n- **Peer ledger supply window** *(per node)* \u2014 the lowest and highest ledger sequence offered by any connected peer; the span the peer set can serve.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerPeerLedgerSupplyGauge`\n\n###### References:\n[Complete ledger ranges](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#complete-ledger-ranges) \u00b7 [Back-fill / catch-up](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#back-fill-catch-up)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Ledger Sequence", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "none" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 62 + }, + "id": 29, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(peer_ledger_supply{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=~\"supply_min_seq|supply_max_seq\"}, \"series\", \"$1\", \"metric\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Peer Ledger Supply Window", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*PeerFinder's slot accounting in one snapshot: outbound and inbound slots in use against their configured maxima, plus outbound attempts still in flight.*\n\n###### How it's computed:\n*peerfinder_slot_census series out_active, out_max, in_active, in_max and connecting, all read from ONE PeerFinder snapshot taken under a single lock so the five values describe the same instant.*\n\n###### Reading it:\n*out_active should climb to out_max and stay there. connecting counts outbound attempts started but not yet resolved either way, so it is the in-flight term the active counts cannot show.*\n\n###### Healthy range:\n*out_active at out_max; connecting low and transient.*\n\n###### Watch for:\n*out_active pinned below out_max while connecting stays non-zero \u2014 dials are being started and never completing, so the node is trying and failing rather than sitting idle. Note out_active and in_active are also exported separately as the legacy beast::insight gauges peer_finder_active_outbound_peers and peer_finder_active_inbound_peers. Those two are unrelated single series read at different instants with no capacity, attempt or cache terms, so no reading of them can distinguish this case. The census exists to report all nine fields from one snapshot under one lock, so they are mutually consistent and joinable on one labelset.*\n\n###### Keywords:\n- **PeerFinder slot census** *(per node)* \u2014 one consistent snapshot of PeerFinder's outbound and inbound slot use, capacities, in-flight attempts, fixed peers and address caches.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerSlotCensusGauge`\n\n###### References:\n[Overlay](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#overlay) \u00b7 [Outbound dial latency](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#outbound-dial-latency)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Slots", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 74 + }, + "id": 30, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(peerfinder_slot_census{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=~\"out_active|out_max|in_active|in_max|connecting\"}, \"series\", \"$1\", \"metric\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "PeerFinder Slot Census", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*The address supply PeerFinder dials from \u2014 its boot cache and live cache \u2014 next to configured versus currently connected fixed peers.*\n\n###### How it's computed:\n*peerfinder_slot_census series bootcache, livecache, fixed_configured and fixed_active, read from the same single snapshot as the slot census.*\n\n###### Reading it:\n*bootcache holds seed addresses persisted across restarts; livecache holds addresses learned from peers while running. fixed_active is how many of the fixed peers named in the config are connected right now.*\n\n###### Healthy range:\n*bootcache and livecache non-zero; fixed_active equal to fixed_configured.*\n\n###### Watch for:\n*bootcache at 0 on a fresh node means there are no seed addresses to dial at all, so no outbound connection is ever attempted and every downstream sync signal on this dashboard stays empty for a reason that has nothing to do with sync. fixed_active below fixed_configured means a configured fixed peer is unreachable.*\n\n###### Keywords:\n- **PeerFinder address cache** *(per node)* \u2014 the boot cache (seed addresses persisted across restarts) and live cache (addresses learned from peers) that supply outbound dial candidates.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerSlotCensusGauge`\n\n###### References:\n[Overlay](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#overlay) \u00b7 [DNS resolve](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#dns-resolve)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Count", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "short" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 74 + }, + "id": 31, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(peerfinder_slot_census{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=~\"bootcache|livecache|fixed_configured|fixed_active\"}, \"series\", \"$1\", \"metric\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "PeerFinder Address Caches & Fixed Peers", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Rate of inbound peer connection handoffs, split by terminal outcome.*\n\n###### How it's computed:\n*Per-second rate of finished inbound handoffs grouped by outcome, per node. Filter the outcome set with the Accept Outcome variable.*\n\n###### Reading it:\n*accepted should dominate. The failure outcomes name what rejected the connection: resource_limit and no_slot are this node's own capacity, protocol_mismatch and bad_cookie are the peer or the network identity, handshake_error and local_endpoint_fail are the transport.*\n\n###### Healthy range:\n*accepted dominant, failures near zero.*\n\n###### Watch for:\n*Read this together with Outbound Dial Outcome Rate above, which is the outbound twin on overlay_connect_total{outcome}. The two share the same one-outcome-per-attempt shape, so together they give the full in/out split: a node that accepts nothing but dials successfully has a very different fault from one that can neither dial nor accept. A steady no_slot or resource_limit is this node refusing peers it has no room for, which is capacity rather than a fault.*\n\n###### Keywords:\n- **Inbound peer accept outcome** *(per node)* \u2014 the terminal result of an inbound peer connection handoff; each handoff ends in exactly one outcome.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl::onHandoff`\n\n###### References:\n[Overlay](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#overlay) \u00b7 [Outbound dial latency](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#outbound-dial-latency)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Handoffs / Sec", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 86 + }, + "id": 32, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sum by (outcome, service_instance_id, xrpl_branch, xrpl_work_item) (rate(peer_accept_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", outcome=~\"$accept_outcome\"}[$__rate_interval])), \"series\", \"$1\", \"outcome\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Inbound Peer Accept Outcomes", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Rate at which peer connections are closed, split by why they closed and by which side opened them.*\n\n###### How it's computed:\n*Per-second rate of peer closes grouped by reason and direction (inbound or outbound), per node. Filter with the Disconnect Reason and Disconnect Direction variables.*\n\n###### Reading it:\n*The split separates our-fault backpressure from topology and network faults. large_sendq and charge_resources are this node shedding a peer it cannot keep up with. not_useful and ping_timeout are topology and liveness. read_error and write_error are the transport. graceful, stopping and shutdown are ordinary lifecycle, not faults.*\n\n###### Healthy range:\n*Mostly graceful; the fault reasons near zero.*\n\n###### Watch for:\n*A sustained large_sendq or charge_resources means this node is the bottleneck and is dropping peers \u2014 which removes the very peers it needs to sync from, so a slow node makes itself slower. A rising not_useful or ping_timeout on outbound points at the peer set instead. Distinct from the single unlabelled peer-disconnect total on the Node Health dashboard, which cannot say which of these is happening.*\n\n###### Keywords:\n- **Peer disconnect reason** *(per node)* \u2014 the cause recorded when a peer connection is closed, paired with whether that connection was inbound or outbound.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[PeerImp.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/PeerImp.cpp)\n\n###### Function:\n`PeerImp::close`\n\n###### References:\n[Resource disconnect](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#resource-disconnect) \u00b7 [Insane / diverged peers](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#insane-diverged-peers)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Disconnects / Sec", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 86 + }, + "id": 33, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_join(sum by (reason, direction, service_instance_id, xrpl_branch, xrpl_work_item) (rate(peer_disconnect_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", reason=~\"$disconnect_reason\", direction=~\"$disconnect_direction\"}[$__rate_interval])), \"series\", \" \", \"reason\", \"direction\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Peer Disconnects by Reason", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 98 + }, + "id": 57, + "panels": [], + "title": "Sync state", + "type": "row" }, { "datasource": { @@ -1069,8 +1613,8 @@ "gridPos": { "h": 12, "w": 8, - "x": 8, - "y": 62 + "x": 0, + "y": 99 }, "id": 13, "options": { @@ -1102,6 +1646,69 @@ "title": "Time to First FULL", "type": "stat" }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Whether the node is still waiting to see a full network ledger before it will participate.*\n\n###### How it's computed:\n*sync_state series network_ledger_gate: 1 while the gate is closed, 0 once it opens.*\n\n###### Reading it:\n*0 (green) is healthy. A persistent 1 (red) means the node has never seen a complete network ledger, so it refuses transactions and can never reach full no matter how healthy the rest of the pipeline looks.*\n\n###### Healthy range:\n*0 within the first few minutes of startup.*\n\n###### Watch for:\n*A 1 that never clears. Pair it with the Bootstrap row \u2014 no peers or no quorum is the usual cause.*\n\n###### Keywords:\n- **Network ledger gate** *(per node)* \u2014 the startup guard that holds a node back until it has seen a full ledger from the network.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerSyncStateGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#network-ledger-gate)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1 + } + ] + }, + "unit": "short" + } + }, + "gridPos": { + "h": 12, + "w": 8, + "x": 8, + "y": 99 + }, + "id": 12, + "options": { + "colorMode": "value", + "graphMode": "none", + "justifyMode": "center", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "value_and_name", + "wideLayout": true + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sync_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"network_ledger_gate\"}, \"series\", \"Gate Closed\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Network Ledger Gate", + "type": "stat" + }, { "datasource": { "type": "prometheus", @@ -1133,7 +1740,7 @@ "h": 12, "w": 8, "x": 16, - "y": 62 + "y": 99 }, "id": 14, "options": { @@ -1235,7 +1842,7 @@ "h": 12, "w": 12, "x": 0, - "y": 74 + "y": 111 }, "id": 15, "options": { @@ -1338,7 +1945,7 @@ "h": 12, "w": 12, "x": 12, - "y": 74 + "y": 111 }, "id": 16, "options": { @@ -1403,7 +2010,7 @@ "h": 12, "w": 24, "x": 0, - "y": 86 + "y": 123 }, "id": 17, "options": { @@ -1442,6 +2049,19 @@ "title": "Mode Transitions by Edge", "type": "bargauge" }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 135 + }, + "id": 58, + "panels": [], + "title": "Ledger acquire & SHAMap fetch", + "type": "row" + }, { "datasource": { "type": "prometheus", @@ -1512,7 +2132,7 @@ "h": 12, "w": 12, "x": 0, - "y": 98 + "y": 136 }, "id": 18, "options": { @@ -1615,7 +2235,7 @@ "h": 12, "w": 12, "x": 12, - "y": 98 + "y": 136 }, "id": 19, "options": { @@ -1714,7 +2334,7 @@ "h": 12, "w": 12, "x": 0, - "y": 110 + "y": 148 }, "id": 20, "options": { @@ -1751,6 +2371,109 @@ "title": "Add-Node Outcomes", "type": "timeseries" }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Peer packets stashed across all in-flight acquires waiting to be applied, alongside how many acquires are running.*\n\n###### How it's computed:\n*sync_acquire series received_data_depth (summed across acquires) and in_flight (the acquire count). The depth mirrors the receive stash size; in_flight gives the context that makes an all-zero reading legible.*\n\n###### Reading it:\n*A depth near zero means node data is applied as fast as it arrives. in_flight at zero means the node is idle, which is why an all-zero Missing Nodes panel is not by itself a healthy reading.*\n\n###### Healthy range:\n*Depth 0 to a few; in_flight low single digits during sync.*\n\n###### Watch for:\n*A growing depth means arriving data outpaces processing, which is a job-queue or disk problem rather than a peer-supply one. Check the job-queue backlog next.*\n\n###### Keywords:\n- **Received-data stash** *(per node)* \u2014 peer packets held for later processing because the acquire cannot apply them as fast as they arrive.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerSyncAcquireGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#received-data-stash)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Count", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "line" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 16 + } + ] + }, + "unit": "short" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 148 + }, + "id": 23, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sync_acquire{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=~\"$acquire_metric\", metric=~\"received_data_depth|in_flight\"}, \"series\", \"$1\", \"metric\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Received-Data Stash Depth & In-Flight Acquires", + "type": "timeseries" + }, { "datasource": { "type": "prometheus", @@ -1812,8 +2535,8 @@ "gridPos": { "h": 12, "w": 12, - "x": 12, - "y": 110 + "x": 0, + "y": 160 }, "id": 21, "options": { @@ -1919,8 +2642,8 @@ "gridPos": { "h": 12, "w": 12, - "x": 0, - "y": 122 + "x": 12, + "y": 160 }, "id": 22, "options": { @@ -1957,12 +2680,92 @@ "title": "SHAMap TreeNode Cache Hit Rate", "type": "timeseries" }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 172 + }, + "id": 59, + "panels": [], + "title": "Job queue", + "type": "row" + }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Peer packets stashed across all in-flight acquires waiting to be applied, alongside how many acquires are running.*\n\n###### How it's computed:\n*sync_acquire series received_data_depth (summed across acquires) and in_flight (the acquire count). The depth mirrors the receive stash size; in_flight gives the context that makes an all-zero reading legible.*\n\n###### Reading it:\n*A depth near zero means node data is applied as fast as it arrives. in_flight at zero means the node is idle, which is why an all-zero Missing Nodes panel is not by itself a healthy reading.*\n\n###### Healthy range:\n*Depth 0 to a few; in_flight low single digits during sync.*\n\n###### Watch for:\n*A growing depth means arriving data outpaces processing, which is a job-queue or disk problem rather than a peer-supply one. Check the job-queue backlog next.*\n\n###### Keywords:\n- **Received-data stash** *(per node)* \u2014 peer packets held for later processing because the acquire cannot apply them as fast as they arrive.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerSyncAcquireGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#received-data-stash)", + "description": "###### What this is:\n*Share of the worker-thread pool currently executing a job. This is the pool-wide view: when the pool itself is exhausted, every subsystem queued behind it looks independently slow, and this panel attributes the whole slowdown once.*\n\n###### How it's computed:\n*jobq_saturation running_tasks divided by worker_threads (the denominator is clamped to at least 1). The thread count is exported rather than hardcoded because it is derived at startup from [workers], node size and hardware concurrency.*\n\n###### Reading it:\n*Below 80% (green) means the pool has spare capacity, so a slow stage is that stage's own fault. At 100% every worker is busy \u2014 read Total Jobs Queued next: 100% with a queue is an exhausted pool, 100% with an empty queue is merely busy.*\n\n###### Healthy range:\n*< 80%.*\n\n###### Watch for:\n*A sustained 100% together with a non-zero backlog. Every job type is then starved by the pool, so fix pool capacity or the long-running jobs holding it, not the individual victim subsystems.*\n\n###### Keywords:\n- **Worker-pool saturation** *(per node)* \u2014 worker threads executing a job as a share of the threads the pool is configured to run.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerJobQueueSaturationGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#worker-pool-saturation)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 0.8 + }, + { + "color": "red", + "value": 1 + } + ] + }, + "unit": "percentunit" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 173 + }, + "id": 26, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "center", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "value_and_name", + "wideLayout": true + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(jobq_saturation{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"running_tasks\"} / ignoring(metric) clamp_min(jobq_saturation{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"worker_threads\"}, 1), \"series\", \"Worker saturation\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Worker Pool Saturation", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*The three raw worker-pool numbers behind the saturation ratio: tasks in flight, threads configured, and total jobs queued across every job type.*\n\n###### How it's computed:\n*jobq_saturation series running_tasks, worker_threads and total_waiting, all read from one JobQueue sample so the ratio and the backlog describe the same instant.*\n\n###### Reading it:\n*running_tasks tracking worker_threads means the pool is fully committed. total_waiting is what makes that legible: queued work behind a fully committed pool is exhaustion, no queued work is just a busy moment.*\n\n###### Healthy range:\n*running_tasks below worker_threads; total_waiting near 0.*\n\n###### Watch for:\n*total_waiting climbing while running_tasks is pinned at worker_threads. Distinct from the jobq_job_count depth panel on the Ledger Data Sync dashboard, which has no capacity term at all, so a depth reading there cannot say whether the pool is the cause.*\n\n###### Keywords:\n- **Worker thread pool** *(per node)* \u2014 the fixed set of threads that execute all job-queue work, sized at startup.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerJobQueueSaturationGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#worker-pool-saturation)", "fieldConfig": { "defaults": { "color": { @@ -1999,7 +2802,7 @@ "mode": "none" }, "thresholdsStyle": { - "mode": "line" + "mode": "off" } }, "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", @@ -2009,10 +2812,6 @@ { "color": "green", "value": null - }, - { - "color": "yellow", - "value": 16 } ] }, @@ -2023,9 +2822,9 @@ "h": 12, "w": 12, "x": 12, - "y": 122 + "y": 173 }, - "id": 23, + "id": 27, "options": { "annotations": { "clustering": -1, @@ -2053,11 +2852,11 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sync_acquire{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=~\"$acquire_metric\", metric=~\"received_data_depth|in_flight\"}, \"series\", \"$1\", \"metric\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(jobq_saturation{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=~\"$saturation_metric\"}, \"series\", \"$1\", \"metric\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "A" } ], - "title": "Received-Data Stash Depth & In-Flight Acquires", + "title": "Worker Pool Capacity & Total Backlog", "type": "timeseries" }, { @@ -2126,7 +2925,7 @@ "h": 12, "w": 12, "x": 0, - "y": 134 + "y": 185 }, "id": 24, "options": { @@ -2225,7 +3024,7 @@ "h": 12, "w": 12, "x": 12, - "y": 134 + "y": 185 }, "id": 25, "options": { @@ -2263,1552 +3062,17 @@ "type": "timeseries" }, { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*Share of the worker-thread pool currently executing a job. This is the pool-wide view: when the pool itself is exhausted, every subsystem queued behind it looks independently slow, and this panel attributes the whole slowdown once.*\n\n###### How it's computed:\n*jobq_saturation running_tasks divided by worker_threads (the denominator is clamped to at least 1). The thread count is exported rather than hardcoded because it is derived at startup from [workers], node size and hardware concurrency.*\n\n###### Reading it:\n*Below 80% (green) means the pool has spare capacity, so a slow stage is that stage's own fault. At 100% every worker is busy \u2014 read Total Jobs Queued next: 100% with a queue is an exhausted pool, 100% with an empty queue is merely busy.*\n\n###### Healthy range:\n*< 80%.*\n\n###### Watch for:\n*A sustained 100% together with a non-zero backlog. Every job type is then starved by the pool, so fix pool capacity or the long-running jobs holding it, not the individual victim subsystems.*\n\n###### Keywords:\n- **Worker-pool saturation** *(per node)* \u2014 worker threads executing a job as a share of the threads the pool is configured to run.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerJobQueueSaturationGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#worker-pool-saturation)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "yellow", - "value": 0.8 - }, - { - "color": "red", - "value": 1 - } - ] - }, - "unit": "percentunit" - } - }, + "collapsed": false, "gridPos": { - "h": 12, - "w": 12, + "h": 1, + "w": 24, "x": 0, - "y": 146 + "y": 197 }, - "id": 26, - "options": { - "colorMode": "value", - "graphMode": "area", - "justifyMode": "center", - "orientation": "auto", - "percentChangeColorMode": "standard", - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false - }, - "showPercentChange": false, - "textMode": "value_and_name", - "wideLayout": true - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(jobq_saturation{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"running_tasks\"} / ignoring(metric) clamp_min(jobq_saturation{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"worker_threads\"}, 1), \"series\", \"Worker saturation\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "Worker Pool Saturation", - "type": "stat" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*The three raw worker-pool numbers behind the saturation ratio: tasks in flight, threads configured, and total jobs queued across every job type.*\n\n###### How it's computed:\n*jobq_saturation series running_tasks, worker_threads and total_waiting, all read from one JobQueue sample so the ratio and the backlog describe the same instant.*\n\n###### Reading it:\n*running_tasks tracking worker_threads means the pool is fully committed. total_waiting is what makes that legible: queued work behind a fully committed pool is exhaustion, no queued work is just a busy moment.*\n\n###### Healthy range:\n*running_tasks below worker_threads; total_waiting near 0.*\n\n###### Watch for:\n*total_waiting climbing while running_tasks is pinned at worker_threads. Distinct from the jobq_job_count depth panel on the Ledger Data Sync dashboard, which has no capacity term at all, so a depth reading there cannot say whether the pool is the cause.*\n\n###### Keywords:\n- **Worker thread pool** *(per node)* \u2014 the fixed set of threads that execute all job-queue work, sized at startup.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerJobQueueSaturationGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#worker-pool-saturation)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Count", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "short" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 12, - "y": 146 - }, - "id": 27, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(jobq_saturation{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=~\"$saturation_metric\"}, \"series\", \"$1\", \"metric\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "Worker Pool Capacity & Total Backlog", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*How many connected peers can actually serve the ledger this node needs next, next to how many advertise a ledger range at all.*\n\n###### How it's computed:\n*peer_ledger_supply series peers_reporting, peers_serving_validated and peers_serving_next, counted from the ledger ranges connected peers advertise and compared against this node's own validated sequence.*\n\n###### Reading it:\n*peers_serving_next is the line that matters: it counts peers holding validated+1, the one ledger this node must acquire before it can advance. peers_serving_validated counts peers that can re-serve the ledger already held, and peers_reporting is the denominator \u2014 peers that advertised any range at all.*\n\n###### Healthy range:\n*peers_serving_next at or near peers_reporting.*\n\n###### Watch for:\n*peers_serving_next at 0 while peers_reporting is above 0. No peer this node is connected to holds the ledger it needs, so this is a supply problem, not a slow-peer problem \u2014 retrying harder or waiting longer cannot fix it, the node needs different peers. Distinct from the acquire panels above, which measure requests already sent to peers that did hold the data.*\n\n###### Keywords:\n- **Peer ledger supply** *(per node)* \u2014 the count of connected peers whose advertised ledger range covers a sequence this node needs.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerPeerLedgerSupplyGauge`\n\n###### References:\n[Complete ledger ranges](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#complete-ledger-ranges) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#fresh-node-sync-diagnostics)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Peers", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "short" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 0, - "y": 158 - }, - "id": 28, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(peer_ledger_supply{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=~\"peers_reporting|peers_serving_validated|peers_serving_next\"}, \"series\", \"$1\", \"metric\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "Peers Able to Serve Needed Sequence", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*The lowest and highest ledger sequence any connected peer offers \u2014 the sequence window the peer set can actually serve from.*\n\n###### How it's computed:\n*peer_ledger_supply series supply_min_seq and supply_max_seq: the minimum and maximum over the ledger ranges advertised by all connected peers.*\n\n###### Reading it:\n*supply_max_seq is the network tip as this node's peers report it. supply_min_seq is the oldest history reachable from the current peer set; anything below it cannot be back-filled from these peers at all.*\n\n###### Healthy range:\n*supply_max_seq advancing with the network, supply_min_seq well below this node's own validated sequence.*\n\n###### Watch for:\n*supply_min_seq sitting above this node's validated sequence \u2014 every connected peer has already pruned the history this node still needs, so the gap can never be closed from this peer set. Pair with Peers Able to Serve Needed Sequence: that panel says how many peers can serve the next ledger, this one says which sequences are on offer at all.*\n\n###### Keywords:\n- **Peer ledger supply window** *(per node)* \u2014 the lowest and highest ledger sequence offered by any connected peer; the span the peer set can serve.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerPeerLedgerSupplyGauge`\n\n###### References:\n[Complete ledger ranges](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#complete-ledger-ranges) \u00b7 [Back-fill / catch-up](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#back-fill-catch-up)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Ledger Sequence", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "none" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 12, - "y": 158 - }, - "id": 29, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(peer_ledger_supply{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=~\"supply_min_seq|supply_max_seq\"}, \"series\", \"$1\", \"metric\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "Peer Ledger Supply Window", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*PeerFinder's slot accounting in one snapshot: outbound and inbound slots in use against their configured maxima, plus outbound attempts still in flight.*\n\n###### How it's computed:\n*peerfinder_slot_census series out_active, out_max, in_active, in_max and connecting, all read from ONE PeerFinder snapshot taken under a single lock so the five values describe the same instant.*\n\n###### Reading it:\n*out_active should climb to out_max and stay there. connecting counts outbound attempts started but not yet resolved either way, so it is the in-flight term the active counts cannot show.*\n\n###### Healthy range:\n*out_active at out_max; connecting low and transient.*\n\n###### Watch for:\n*out_active pinned below out_max while connecting stays non-zero \u2014 dials are being started and never completing, so the node is trying and failing rather than sitting idle. Note out_active and in_active are also exported separately as the legacy beast::insight gauges peer_finder_active_outbound_peers and peer_finder_active_inbound_peers. Those two are unrelated single series read at different instants with no capacity, attempt or cache terms, so no reading of them can distinguish this case. The census exists to report all nine fields from one snapshot under one lock, so they are mutually consistent and joinable on one labelset.*\n\n###### Keywords:\n- **PeerFinder slot census** *(per node)* \u2014 one consistent snapshot of PeerFinder's outbound and inbound slot use, capacities, in-flight attempts, fixed peers and address caches.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerSlotCensusGauge`\n\n###### References:\n[Overlay](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#overlay) \u00b7 [Outbound dial latency](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#outbound-dial-latency)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Slots", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "short" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 0, - "y": 170 - }, - "id": 30, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(peerfinder_slot_census{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=~\"out_active|out_max|in_active|in_max|connecting\"}, \"series\", \"$1\", \"metric\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "PeerFinder Slot Census", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*The address supply PeerFinder dials from \u2014 its boot cache and live cache \u2014 next to configured versus currently connected fixed peers.*\n\n###### How it's computed:\n*peerfinder_slot_census series bootcache, livecache, fixed_configured and fixed_active, read from the same single snapshot as the slot census.*\n\n###### Reading it:\n*bootcache holds seed addresses persisted across restarts; livecache holds addresses learned from peers while running. fixed_active is how many of the fixed peers named in the config are connected right now.*\n\n###### Healthy range:\n*bootcache and livecache non-zero; fixed_active equal to fixed_configured.*\n\n###### Watch for:\n*bootcache at 0 on a fresh node means there are no seed addresses to dial at all, so no outbound connection is ever attempted and every downstream sync signal on this dashboard stays empty for a reason that has nothing to do with sync. fixed_active below fixed_configured means a configured fixed peer is unreachable.*\n\n###### Keywords:\n- **PeerFinder address cache** *(per node)* \u2014 the boot cache (seed addresses persisted across restarts) and live cache (addresses learned from peers) that supply outbound dial candidates.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerSlotCensusGauge`\n\n###### References:\n[Overlay](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#overlay) \u00b7 [DNS resolve](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#dns-resolve)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Count", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "short" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 12, - "y": 170 - }, - "id": 31, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(peerfinder_slot_census{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=~\"bootcache|livecache|fixed_configured|fixed_active\"}, \"series\", \"$1\", \"metric\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "PeerFinder Address Caches & Fixed Peers", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*Rate of inbound peer connection handoffs, split by terminal outcome.*\n\n###### How it's computed:\n*Per-second rate of finished inbound handoffs grouped by outcome, per node. Filter the outcome set with the Accept Outcome variable.*\n\n###### Reading it:\n*accepted should dominate. The failure outcomes name what rejected the connection: resource_limit and no_slot are this node's own capacity, protocol_mismatch and bad_cookie are the peer or the network identity, handshake_error and local_endpoint_fail are the transport.*\n\n###### Healthy range:\n*accepted dominant, failures near zero.*\n\n###### Watch for:\n*Read this together with Outbound Dial Outcome Rate above, which is the outbound twin on overlay_connect_total{outcome}. The two share the same one-outcome-per-attempt shape, so together they give the full in/out split: a node that accepts nothing but dials successfully has a very different fault from one that can neither dial nor accept. A steady no_slot or resource_limit is this node refusing peers it has no room for, which is capacity rather than a fault.*\n\n###### Keywords:\n- **Inbound peer accept outcome** *(per node)* \u2014 the terminal result of an inbound peer connection handoff; each handoff ends in exactly one outcome.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl::onHandoff`\n\n###### References:\n[Overlay](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#overlay) \u00b7 [Outbound dial latency](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#outbound-dial-latency)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Handoffs / Sec", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 0, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 0, - "y": 182 - }, - "id": 32, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(sum by (outcome, service_instance_id, xrpl_branch, xrpl_work_item) (rate(peer_accept_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", outcome=~\"$accept_outcome\"}[$__rate_interval])), \"series\", \"$1\", \"outcome\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "Inbound Peer Accept Outcomes", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*Rate at which peer connections are closed, split by why they closed and by which side opened them.*\n\n###### How it's computed:\n*Per-second rate of peer closes grouped by reason and direction (inbound or outbound), per node. Filter with the Disconnect Reason and Disconnect Direction variables.*\n\n###### Reading it:\n*The split separates our-fault backpressure from topology and network faults. large_sendq and charge_resources are this node shedding a peer it cannot keep up with. not_useful and ping_timeout are topology and liveness. read_error and write_error are the transport. graceful, stopping and shutdown are ordinary lifecycle, not faults.*\n\n###### Healthy range:\n*Mostly graceful; the fault reasons near zero.*\n\n###### Watch for:\n*A sustained large_sendq or charge_resources means this node is the bottleneck and is dropping peers \u2014 which removes the very peers it needs to sync from, so a slow node makes itself slower. A rising not_useful or ping_timeout on outbound points at the peer set instead. Distinct from the single unlabelled peer-disconnect total on the Node Health dashboard, which cannot say which of these is happening.*\n\n###### Keywords:\n- **Peer disconnect reason** *(per node)* \u2014 the cause recorded when a peer connection is closed, paired with whether that connection was inbound or outbound.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[PeerImp.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/PeerImp.cpp)\n\n###### Function:\n`PeerImp::close`\n\n###### References:\n[Resource disconnect](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#resource-disconnect) \u00b7 [Insane / diverged peers](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#insane-diverged-peers)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Disconnects / Sec", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 0, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 12, - "y": 182 - }, - "id": 33, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_join(sum by (reason, direction, service_instance_id, xrpl_branch, xrpl_work_item) (rate(peer_disconnect_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", reason=~\"$disconnect_reason\", direction=~\"$disconnect_direction\"}[$__rate_interval])), \"series\", \" \", \"reason\", \"direction\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "Peer Disconnects by Reason", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*Peer requests that this node declined to serve, split by why it declined and by what was asked for.*\n\n###### How it's computed:\n*Per-second rate of refused serve attempts grouped by reason and request type (ledger, tx-set, object or fetchpack), per node. Filter with the Serve Reason and Serve Request variables, which list the emitted label values verbatim (the tx-set one is a single unhyphenated word).*\n\n###### Reading it:\n*This is the other side of the sync exchange: every other panel here measures what this node fetches, this one measures what it refuses to give back. sendq_full and load_shed are self-inflicted backpressure \u2014 the node holds the data but will not send it. not_found and no_map mean the requester asked for history this node does not hold.*\n\n###### Healthy range:\n*Near zero.*\n\n###### Watch for:\n*sendq_full or load_shed climbing. This node is starving its peers, and on a network of similarly loaded nodes that is exactly the condition that makes everybody's sync slow, so a refusal here can be the cause of another operator's stall. A high not_found is usually benign on a node configured with short history, but it tells peers to look elsewhere.*\n\n###### Keywords:\n- **Serve refusal** *(per node)* \u2014 a peer request for ledger, transaction-set, object or fetch-pack data that this node declined, together with the reason it declined.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[PeerImp.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/PeerImp.cpp)\n\n###### Function:\n`PeerImp::processLedgerRequest` \u00b7 `PeerImp::onMessage(TMGetObjectByHash)` \u00b7 `PeerImp::doFetchPack`\n\n###### References:\n[Fetch-pack](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#fetch-pack) \u00b7 [GetObject / object fetch](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#getobject-object-fetch)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Refusals / Sec", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 0, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 1, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 0, - "y": 194 - }, - "id": 34, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_join(sum by (reason, request, service_instance_id, xrpl_branch, xrpl_work_item) (rate(serve_refused_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", reason=~\"$serve_reason\", request=~\"$serve_request\"}[$__rate_interval])), \"series\", \" \", \"reason\", \"request\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "Ledger/Object Serve Refusals", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*Rate at which this node abandoned the chain tip it had built and switched to a different last-closed ledger reported by the network.*\n\n###### How it's computed:\n*Per-second rate of last-closed-ledger switches where the ledger the network reported was not the one this node built on, per node. The counter carries no labels, so the sum is over one series per node.*\n\n###### Reading it:\n*Any non-zero value means this node was told the network's last-closed ledger is not the one it built on, and it discarded its own chain tip in response. A single jump around startup or a restart is ordinary recovery.*\n\n###### Healthy range:\n*0.*\n\n###### Watch for:\n*Repeated jumps \u2014 that is wrong-chain thrash, not one-off recovery. Check the peer set and the configured network id: a node peered to the wrong network, or into a minority partition, keeps being overruled and keeps throwing away work. Pair with the Bootstrap row and with Ledgers Behind Network.*\n\n###### Keywords:\n- **Byzantine ledger jump** *(per node)* \u2014 the node replaced its own last-closed ledger with a different one reported by the network, discarding the chain tip it had built.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[NetworkOPs.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/misc/NetworkOPs.cpp)\n\n###### Function:\n`NetworkOPsImp::switchLastClosedLedger`\n\n###### References:\n[Fork](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#fork) \u00b7 [Ledger history mismatch](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-history-mismatch)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Jumps / Sec", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "line" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 0.001 - } - ] - }, - "unit": "ops" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 12, - "y": 194 - }, - "id": 35, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(ledger_jump_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])), \"series\", \"Ledger Jumps\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "Byzantine Ledger Jumps", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*Seconds until an unsupported amendment activates and this node stops validating for good. This is the LEADING indicator \u2014 the countdown before the block, while there is still time to upgrade.*\n\n###### How it's computed:\n*amendment_block series seconds_to_block. A value of -1 is the sentinel for \"no unsupported amendment is pending\", the same convention validator_health{metric=\"unl_expiry_days\"} already uses.*\n\n###### Reading it:\n*Because -1 is the healthy sentinel and a small positive number is the emergency, the colour scale is not monotonic \u2014 read the value, not only the colour. -1 is green and means nothing is pending. A large positive value is green above 7 days and yellow under 7 days: an unsupported amendment holds majority but there is still time to upgrade. A small positive value under 1 day is red: at 0 this node stops validating and does not resume without a software upgrade.*\n\n###### Healthy range:\n*-1.*\n\n###### Watch for:\n*Any value at or above 0. This is distinct from the Amendment Blocked stat on the Validator Health dashboard (validator_health{metric=\"amendment_blocked\"}), which reports the TERMINAL state \u2014 already blocked, too late to act. This panel is the window before that happens, so the two are read together: countdown first, terminal state as confirmation. The identity of the blocking amendment is deliberately NOT a metric label, because the network can vote on an arbitrary 256-bit amendment id and a label would be unbounded cardinality. The hash is logged instead by AmendmentTableImpl::doValidatedLedger (\"Unsupported amendment reached majority at ...\") and is available via Loki.*\n\n###### Keywords:\n- **Amendment block countdown** *(per node)* \u2014 seconds until an unsupported amendment that has reached majority activates and blocks this node; -1 when none is pending.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerAmendmentBlockGauge`\n\n###### References:\n[Amendments on xrpl.org](https://xrpl.org/docs/concepts/networks-and-servers/amendments) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#amendment-blocked)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 0 - }, - { - "color": "yellow", - "value": 86400 - }, - { - "color": "green", - "value": 604800 - } - ] - }, - "unit": "s" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 0, - "y": 206 - }, - "id": 36, - "options": { - "colorMode": "value", - "graphMode": "none", - "justifyMode": "center", - "orientation": "auto", - "percentChangeColorMode": "standard", - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false - }, - "showPercentChange": false, - "textMode": "value_and_name", - "wideLayout": true - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(amendment_block{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"seconds_to_block\"}, \"series\", \"Seconds to Block\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "Amendment Block Countdown", - "type": "stat" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*Whether an amendment this build does not support has reached majority on the network.*\n\n###### How it's computed:\n*amendment_block series warned: 1 while an unsupported amendment holds majority, 0 otherwise.*\n\n###### Reading it:\n*0 is healthy. A 1 is the first warning that an upgrade is required, and it is raised before the amendment activates rather than after.*\n\n###### Healthy range:\n*0.*\n\n###### Watch for:\n*The transition from 0 to 1 \u2014 that is the moment the upgrade clock starts. Read Amendment Block Countdown next for how long is left, and the Amendment Blocked stat on the Validator Health dashboard for whether the block has already happened; that one is the terminal state, this one is the warning. Which amendment is blocking is not a label (an arbitrary 256-bit amendment id would be unbounded cardinality); the hash is logged by AmendmentTableImpl::doValidatedLedger and is available via Loki.*\n\n###### Keywords:\n- **Amendment warning** *(per node)* \u2014 an unsupported amendment has reached majority; the node still validates, but will stop when that amendment activates.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerAmendmentBlockGauge`\n\n###### References:\n[Amendments on xrpl.org](https://xrpl.org/docs/concepts/networks-and-servers/amendments) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#amendment-blocked)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Warned", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "line" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 1 - } - ] - }, - "unit": "short" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 12, - "y": 206 - }, - "id": 37, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(amendment_block{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"warned\"}, \"series\", \"Warned\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "Amendment Warned", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*Node-store write latency next to read latency, in microseconds per operation. The write side is the signal: a node with a large existing database back-fills slower than a fresh one, and back-fill is write-bound, so no read-side metric can show it.*\n\n###### How it's computed:\n*nodestore_latency series write_mean_us and read_mean_us, each divided by its own count series so the reading is the latency during the selected interval rather than the average since boot. The write numerator comes from a store-duration total that was declared but never written before this signal existed.*\n\n###### Reading it:\n*Compare the two lines. Reads far above writes points at the read path or a cold cache; writes far above reads points at backend write pressure, which is the large-existing-database case.*\n\n###### Healthy range:\n*Both well under a few hundred microseconds on healthy local storage.*\n\n###### Watch for:\n*A rising write line during history back-fill: the backend cannot absorb writes fast enough and sync will stay slow no matter how many peers are available. Read with Fetch-Pack Peer Starvation to tell a data-supply problem from a disk problem. This is a mean, not a percentile \u2014 a tail that matters will move it, but p99 is not available from this signal.*\n\n###### Keywords:\n- **Node-store write latency** *(per node)* \u2014 how long the node store takes to persist one object.\n- **Node-store read latency** *(per node)* \u2014 how long the node store takes to retrieve one object.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreLatencyGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#node-store-write-latency)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Microseconds / Op", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "\u00b5s" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 0, - "y": 218 - }, - "id": 38, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"write_mean_us\"}[$__rate_interval])) / clamp_min(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"write_count\"}[$__rate_interval])), 1), \"series\", \"Write us/op (interval)\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"read_mean_us\"}[$__rate_interval])) / clamp_min(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"read_count\"}[$__rate_interval])), 1), \"series\", \"Read us/op (interval)\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "B" - } - ], - "title": "NodeStore Write vs Read Latency (us/op)", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*Node-store write and read operation rates \u2014 the denominators behind the latency panel.*\n\n###### How it's computed:\n*Rate of the nodestore_latency write_count and read_count series.*\n\n###### Reading it:\n*Writes climb while a node is back-filling history and fall to near the ledger-close rate once it is caught up.*\n\n###### Healthy range:\n*Non-zero writes whenever the node is ingesting ledgers.*\n\n###### Watch for:\n*Write rate at zero while the node is still behind the network: nothing is being persisted, so the stall is upstream of the node store \u2014 check peer supply and the acquire panels rather than storage. A flat latency with a collapsing operation rate also means the latency figure above has gone stale rather than good.*\n\n###### Keywords:\n- **Node-store operation rate** *(per node)* \u2014 stores and fetches per second against the node store.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreLatencyGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#node-store-operation-rate)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Ops / Sec", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 12, - "y": 218 - }, - "id": 39, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"write_count\"}[$__rate_interval])), \"series\", \"Writes/s\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"read_count\"}[$__rate_interval])), \"series\", \"Reads/s\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "B" - } - ], - "title": "NodeStore Operation Rate (writes vs reads)", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*Rate at which ledger-replay sub-tasks give up and fall back to acquiring a whole ledger, split by which sub-task gave up.*\n\n###### How it's computed:\n*Rate of ledger_replay_fallback_total by stage. The skip-list stage and the delta stage each emit once, on the transition into fallback, from the branch that was debug-log-only before.*\n\n###### Reading it:\n*Zero when replay-capable peers are available. Any sustained rate means the replay optimisation is being defeated and back-fill has reverted to the slower full-acquire path.*\n\n###### Healthy range:\n*Zero, or brief spikes while the peer set changes.*\n\n###### Watch for:\n*A persistent rate on either stage: too few connected peers support the ledger-replay feature, so every historical ledger is fetched in full instead of as a delta. Read with Replay Outcomes.*\n\n###### Keywords:\n- **Replay fallback** *(per node)* \u2014 a replay sub-task abandoning the delta shortcut and acquiring the entire ledger instead.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[SkipListAcquire.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/SkipListAcquire.cpp)\n\n###### Function:\n`SkipListAcquire::trigger` / `LedgerDeltaAcquire::trigger`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#replay-fallback)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Fallbacks / Sec", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 0, - "y": 230 - }, - "id": 40, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(sum by (stage, service_instance_id, xrpl_branch, xrpl_work_item) (rate(ledger_replay_fallback_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", stage=~\"$replay_stage\"}[$__rate_interval])), \"series\", \"$1\", \"stage\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "Replay Fallback to Full Acquire (by stage)", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*Rate of ledger-replay tasks reaching each terminal state: success, timeout, build failure or parameter failure.*\n\n###### How it's computed:\n*Rate of ledger_replay_outcome_total by outcome. Every terminal path in the replay task emits exactly once; before this, all four only set an internal flag and wrote a log line.*\n\n###### Reading it:\n*Successes only is healthy. Timeouts mean deltas never arrived; build failures mean a delta would not apply to its parent; parameter failures mean a peer served an inconsistent skip list.*\n\n###### Healthy range:\n*Successes non-zero while replaying, all failure outcomes at zero.*\n\n###### Watch for:\n*Any failure outcome climbing while successes stay flat: replay is running but never completing, so history back-fill is silently falling back to the slower path. The outcome value tells you which layer to look at \u2014 timeouts point at peers, build and parameter failures point at the data those peers served.*\n\n###### Keywords:\n- **Ledger replay** *(per node)* \u2014 rebuilding a range of historical ledgers from a start ledger plus per-ledger deltas instead of downloading each one whole.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[LedgerReplayTask.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/LedgerReplayTask.cpp)\n\n###### Function:\n`LedgerReplayTask::recordOutcome`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-replay)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Tasks / Sec", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 12, - "y": 230 - }, - "id": 41, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(sum by (outcome, service_instance_id, xrpl_branch, xrpl_work_item) (rate(ledger_replay_outcome_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", outcome=~\"$replay_outcome\"}[$__rate_interval])), \"series\", \"$1\", \"outcome\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "Replay Outcomes (by terminal state)", - "type": "timeseries" + "id": 60, + "panels": [], + "title": "Quorum & publish", + "type": "row" }, { "datasource": { @@ -3872,7 +3136,7 @@ "h": 12, "w": 12, "x": 0, - "y": 242 + "y": 198 }, "id": 42, "options": { @@ -3917,6 +3181,172 @@ "title": "Trusted Validations vs Quorum Target", "type": "timeseries" }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Rate at which the pre-accept gate refused to declare a candidate ledger validated because its trusted validations were below quorum.*\n\n###### How it's computed:\n*Rate of ledger_quorum_shortfall_total by stage. One increment per rejected gate evaluation, emitted from the early return that was trace-log-only before, so a node that peers and receives validations yet never validates is no longer indistinguishable from an idle one. The gate is re-entered on every fresh trusted validation for the candidate ledger, so one ledger that eventually validates can contribute several increments on its way there.*\n\n###### Reading it:\n*A steady low rate is NORMAL and is not a fault. The gate is evaluated the instant this node finishes building a ledger, before its peers' validations for that ledger have arrived, so the first evaluation of each round routinely tallies short and is retried as validations come in. Read this against the rate of ledger closes and against Trusted Validations vs Quorum Target \u2014 it is the ratio and the accompanying tally that carry the signal, never the bare presence of a rate.*\n\n###### Healthy range:\n*A low steady rate on the order of one per ledger close or less, on a node that is validating.*\n\n###### Watch for:\n*A rate that climbs well above the ledger-close rate while Publish Lag grows and Time to First Validated Ledger stays at zero \u2014 that combination is the retry loop never converging, so the tally is not merely early, it never reaches the target. Confirm on Trusted Validations vs Quorum Target: a climbing tally is slow and will finish, a flat tally below the target is stuck (too few trusted validators reachable, or a UNL / negative-UNL misconfiguration). Check UNL Quorum Headroom in the Bootstrap row before anything else in this row, because a trusted list that cannot satisfy quorum makes every panel below it look starved.*\n\n###### Keywords:\n- **Quorum shortfall** *(per node)* \u2014 trusted validations for a candidate ledger falling short of the quorum needed to declare it validated, so the node holds the ledger and still cannot call it validated.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (call-site metric macro, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[LedgerMaster.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/LedgerMaster.cpp)\n\n###### Function:\n`LedgerMaster::checkAccept`\n\n###### References:\n[Negative UNL and validation quorum on xrpl.org](https://xrpl.org/docs/concepts/consensus-protocol/negative-unl) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#quorum-shortfall)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Shortfalls / Sec", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "line" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 0.1 + } + ] + }, + "unit": "ops" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 198 + }, + "id": 44, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sum by (stage, service_instance_id, xrpl_branch, xrpl_work_item) (rate(ledger_quorum_shortfall_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", stage=~\"$shortfall_stage\"}[$__rate_interval])), \"series\", \"$1\", \"stage\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Pre-Accept Quorum Shortfall Rate", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*How long the node took, from process start, to pass the pre-accept quorum gate for the first time.*\n\n###### How it's computed:\n*ledger_quorum_publish series time_to_first_validated_us, converted from microseconds to seconds.*\n\n###### Reading it:\n*A one-shot measurement: it fills in the moment the node first fully validates a ledger and never changes again, so it has no trend to read. Exactly two readings matter \u2014 a duration, meaning the node got there and this is how long it took, or zero (red), meaning it never has.*\n\n###### Healthy range:\n*Seconds to a few minutes; longer on a fresh node that must acquire history first.*\n\n###### Watch for:\n*A flat zero while Time to First FULL shows a value: the node reached the full server state but has still never fully validated a ledger, which points at the quorum gate rather than at acquire. The measurement is clamped to a minimum of 1 microsecond so a genuine reading can never be confused with the never-reached zero.*\n\n###### Keywords:\n- **Time to first validated ledger** *(per node)* \u2014 elapsed time from process start until the node first declared a ledger fully validated.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerLedgerQuorumPublishGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#time-to-first-validated-ledger)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "red", + "value": null + }, + { + "color": "green", + "value": 1 + } + ] + }, + "unit": "s" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 210 + }, + "id": 45, + "options": { + "colorMode": "value", + "graphMode": "none", + "justifyMode": "center", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "value_and_name", + "wideLayout": true + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(ledger_quorum_publish{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"time_to_first_validated_us\"} / 1e6, \"series\", \"Time to First Validated\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Time to First Validated Ledger", + "type": "stat" + }, { "datasource": { "type": "prometheus", @@ -3987,7 +3417,7 @@ "h": 12, "w": 12, "x": 12, - "y": 242 + "y": 210 }, "id": 43, "options": { @@ -4024,782 +3454,6 @@ "title": "Publish Lag (validated minus published)", "type": "timeseries" }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*Rate at which the pre-accept gate refused to declare a candidate ledger validated because its trusted validations were below quorum.*\n\n###### How it's computed:\n*Rate of ledger_quorum_shortfall_total by stage. One increment per rejected gate evaluation, emitted from the early return that was trace-log-only before, so a node that peers and receives validations yet never validates is no longer indistinguishable from an idle one. The gate is re-entered on every fresh trusted validation for the candidate ledger, so one ledger that eventually validates can contribute several increments on its way there.*\n\n###### Reading it:\n*A steady low rate is NORMAL and is not a fault. The gate is evaluated the instant this node finishes building a ledger, before its peers' validations for that ledger have arrived, so the first evaluation of each round routinely tallies short and is retried as validations come in. Read this against the rate of ledger closes and against Trusted Validations vs Quorum Target \u2014 it is the ratio and the accompanying tally that carry the signal, never the bare presence of a rate.*\n\n###### Healthy range:\n*A low steady rate on the order of one per ledger close or less, on a node that is validating.*\n\n###### Watch for:\n*A rate that climbs well above the ledger-close rate while Publish Lag grows and Time to First Validated Ledger stays at zero \u2014 that combination is the retry loop never converging, so the tally is not merely early, it never reaches the target. Confirm on Trusted Validations vs Quorum Target: a climbing tally is slow and will finish, a flat tally below the target is stuck (too few trusted validators reachable, or a UNL / negative-UNL misconfiguration). Check UNL Quorum Headroom in the Bootstrap row before anything else in this row, because a trusted list that cannot satisfy quorum makes every panel below it look starved.*\n\n###### Keywords:\n- **Quorum shortfall** *(per node)* \u2014 trusted validations for a candidate ledger falling short of the quorum needed to declare it validated, so the node holds the ledger and still cannot call it validated.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (call-site metric macro, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[LedgerMaster.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/LedgerMaster.cpp)\n\n###### Function:\n`LedgerMaster::checkAccept`\n\n###### References:\n[Negative UNL and validation quorum on xrpl.org](https://xrpl.org/docs/concepts/consensus-protocol/negative-unl) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#quorum-shortfall)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Shortfalls / Sec", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "line" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - }, - { - "color": "red", - "value": 0.1 - } - ] - }, - "unit": "ops" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 0, - "y": 254 - }, - "id": 44, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(sum by (stage, service_instance_id, xrpl_branch, xrpl_work_item) (rate(ledger_quorum_shortfall_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", stage=~\"$shortfall_stage\"}[$__rate_interval])), \"series\", \"$1\", \"stage\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "Pre-Accept Quorum Shortfall Rate", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*How long the node took, from process start, to pass the pre-accept quorum gate for the first time.*\n\n###### How it's computed:\n*ledger_quorum_publish series time_to_first_validated_us, converted from microseconds to seconds.*\n\n###### Reading it:\n*A one-shot measurement: it fills in the moment the node first fully validates a ledger and never changes again, so it has no trend to read. Exactly two readings matter \u2014 a duration, meaning the node got there and this is how long it took, or zero (red), meaning it never has.*\n\n###### Healthy range:\n*Seconds to a few minutes; longer on a fresh node that must acquire history first.*\n\n###### Watch for:\n*A flat zero while Time to First FULL shows a value: the node reached the full server state but has still never fully validated a ledger, which points at the quorum gate rather than at acquire. The measurement is clamped to a minimum of 1 microsecond so a genuine reading can never be confused with the never-reached zero.*\n\n###### Keywords:\n- **Time to first validated ledger** *(per node)* \u2014 elapsed time from process start until the node first declared a ledger fully validated.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerLedgerQuorumPublishGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#time-to-first-validated-ledger)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "thresholds" - }, - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "red", - "value": null - }, - { - "color": "green", - "value": 1 - } - ] - }, - "unit": "s" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 12, - "y": 254 - }, - "id": 45, - "options": { - "colorMode": "value", - "graphMode": "none", - "justifyMode": "center", - "orientation": "auto", - "percentChangeColorMode": "standard", - "reduceOptions": { - "calcs": ["lastNotNull"], - "fields": "", - "values": false - }, - "showPercentChange": false, - "textMode": "value_and_name", - "wideLayout": true - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(ledger_quorum_publish{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"time_to_first_validated_us\"} / 1e6, \"series\", \"Time to First Validated\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "Time to First Validated Ledger", - "type": "stat" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*Rate at which attempts to fetch a transaction set referenced by a consensus proposal reach each terminal state.*\n\n###### How it's computed:\n*Rate of span-derived call counts for the txset.acquire span, split by outcome. The span is new: transaction-set acquisition had no telemetry at all before, so a consensus round stalled waiting on a set looked identical to an idle one. Every exit stamps an outcome, including the sweep that drops a set which never arrived.*\n\n###### Reading it:\n*Only complete is healthy. Timeout means peers never supplied the set. Abandoned means the round moved on and the set was dropped mid-fetch. Failed means a peer served data that would not build.*\n\n###### Healthy range:\n*A low rate of complete during consensus, everything else at zero. Zero everywhere is also normal on a node that already holds every proposed set locally.*\n\n###### Watch for:\n*Timeout or abandoned climbing: proposed sets are not arriving, so rounds are waiting on data rather than on agreement. Read with the Tx-Set Acquire Duration panel beside it \u2014 a rising p95 with outcomes still complete is slow-but-working, while the same p95 with timeouts is the set never arriving at all.*\n\n###### Keywords:\n- **Tx-set acquire** *(per node)* \u2014 one attempt to fetch the transaction set a consensus proposal referenced but this node did not hold.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Derived by the OTel Collector's spanmetrics connector from spans emitted by xrpld code; the collector counts the spans and their durations, and the Grafana query selects and aggregates the resulting series.*\n\n###### Source:\n[TransactionAcquire.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/TransactionAcquire.cpp)\n\n###### Function:\n`TransactionAcquire::finalizeAcquireSpan`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#tx-set-acquire)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Acquisitions / Sec", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 0, - "y": 266 - }, - "id": 46, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(sum by (outcome, service_instance_id, xrpl_branch, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", outcome=~\"$span_outcome\", span_name=\"txset.acquire\"}[$__rate_interval])), \"series\", \"$1\", \"outcome\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "Tx-Set Acquire Outcomes", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*95th-percentile wall time of one transaction-set acquisition, split by how it ended.*\n\n###### How it's computed:\n*95th percentile of the span-derived duration histogram for the txset.acquire span, grouped by outcome.*\n\n###### Reading it:\n*Splitting by outcome is what makes this readable: the complete series is how long a successful fetch takes, while the timeout series is pinned near the retry budget by construction and carries no information about speed.*\n\n###### Healthy range:\n*A complete-series p95 well under a consensus round interval.*\n\n###### Watch for:\n*A complete-series p95 approaching the round interval: sets are arriving, but so late that they delay the round they belong to. This is the case that a pure outcome rate cannot show, because those acquisitions do succeed.*\n\n###### Keywords:\n- **Tx-set acquire** *(per node)* \u2014 one attempt to fetch the transaction set a consensus proposal referenced but this node did not hold.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Derived by the OTel Collector's spanmetrics connector from spans emitted by xrpld code; the collector counts the spans and their durations, and the Grafana query selects and aggregates the resulting series.*\n\n###### Source:\n[TransactionAcquire.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/TransactionAcquire.cpp)\n\n###### Function:\n`TransactionAcquire::init`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#tx-set-acquire)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Duration", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ms" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 12, - "y": 266 - }, - "id": 47, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, outcome, service_instance_id, xrpl_branch, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", outcome=~\"$span_outcome\", span_name=\"txset.acquire\"}[$__rate_interval]))), \"series\", \"P95 $1\", \"outcome\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "Tx-Set Acquire Duration (p95)", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*95th-percentile duration of each of the three phases of one ledger acquisition: the header wait, the account-state tree fetch, and the transaction tree fetch.*\n\n###### How it's computed:\n*95th percentile of the span-derived duration histogram for the three ledger.acquire.header / .astree / .txtree child spans, one series per phase. The parent ledger.acquire span is flat and cannot separate them, which matters because the account-state tree is nearly all of the work in a real fresh sync.*\n\n###### Reading it:\n*The astree series dominating is expected and healthy. The value of the split is the comparison: a header series that is large means the node is waiting to be told what to fetch, which is a peer-supply problem upstream of either tree.*\n\n###### Healthy range:\n*The astree series largest, the txtree series small, the header series near zero.*\n\n###### Watch for:\n*A hot astree band that keeps growing means account-state nodes are not being served \u2014 check Ledger Acquire Phase Outcomes beside it for timed_out, and Outbound Dial Outcome Rate in the Bootstrap row for refused or timed-out dials. A large header series instead means no peer is answering the header request at all.*\n\n###### Keywords:\n- **Acquire phase** *(per node)* \u2014 one of the three sequential fetches a ledger acquisition is made of; the header gates both trees, because it is what names their root hashes.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Derived by the OTel Collector's spanmetrics connector from spans emitted by xrpld code; the collector counts the spans and their durations, and the Grafana query selects and aggregates the resulting series.*\n\n###### Source:\n[InboundLedger.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/InboundLedger.cpp)\n\n###### Function:\n`InboundLedger::syncPhaseSpans`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#acquire-phase)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Duration", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ms" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 0, - "y": 278 - }, - "id": 48, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, span_name, service_instance_id, xrpl_branch, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"ledger.acquire.header\"}[$__rate_interval]))), \"series\", \"P95 header\", \"span_name\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, span_name, service_instance_id, xrpl_branch, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"ledger.acquire.astree\"}[$__rate_interval]))), \"series\", \"P95 account-state tree\", \"span_name\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "B" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, span_name, service_instance_id, xrpl_branch, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"ledger.acquire.txtree\"}[$__rate_interval]))), \"series\", \"P95 transaction tree\", \"span_name\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "C" - } - ], - "title": "Ledger Acquire Phase Duration (p95 by phase)", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*Rate at which each ledger-acquire phase ends, split by which phase it was and whether it ran out of retry budget.*\n\n###### How it's computed:\n*Rate of span-derived call counts for the three ledger.acquire.* child spans, split by span name and by the timed_out flag. The flag is a separate dimension from the outcome because a phase can time out and still be retried by its parent acquire.*\n\n###### Reading it:\n*Series with timed_out=false are phases that finished on their own. Any series with timed_out=true is a phase whose retry budget expired, which names the exact stage of the fetch that peers are not serving.*\n\n###### Healthy range:\n*All series timed_out=false; zero everywhere on a node with no back-fill to do.*\n\n###### Watch for:\n*Any sustained timed_out=true rate. Pair it with the missing-node count on Missing SHAMap Nodes per Acquire: a timed-out phase together with a flat, non-zero missing-node count is the definitive stuck-sync signature, and the phase name says which tree to chase.*\n\n###### Keywords:\n- **Acquire phase timeout** *(per node)* \u2014 an acquire phase that ended because its retry budget expired rather than because its data arrived.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Derived by the OTel Collector's spanmetrics connector from spans emitted by xrpld code; the collector counts the spans and their durations, and the Grafana query selects and aggregates the resulting series.*\n\n###### Source:\n[InboundLedger.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/InboundLedger.cpp)\n\n###### Function:\n`InboundLedger::endPhaseSpan`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#acquire-phase)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Phases / Sec", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 12, - "y": 278 - }, - "id": 49, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_join(label_replace(sum by (span_name, timed_out, service_instance_id, xrpl_branch, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", timed_out=~\"$timed_out\", span_name=~\"ledger.acquire..*\"}[$__rate_interval])), \"phase\", \"$1\", \"span_name\", \"ledger\\.acquire\\.(.*)\"), \"series\", \" timed_out=\", \"phase\", \"timed_out\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "Ledger Acquire Phase Outcomes (by phase & timeout)", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*Rate of outbound peer dials reaching each terminal outcome, derived from the per-attempt peer.dial span.*\n\n###### How it's computed:\n*Rate of span-derived call counts for the peer.dial span, split by outcome. The same five outcome values the overlay_connect_total counter carries, set from the same funnel in the dial state machine so the two cannot disagree.*\n\n###### Reading it:\n*Read alongside Outbound Dial Outcome Rate in the Bootstrap row, which is the native counter for the same events. This panel exists for what the counter cannot do: each point here is backed by traces, so clicking through gives the individual attempt and the peer address it was dialling, which is never a metric label because one series per peer address would be unbounded.*\n\n###### Healthy range:\n*The connected series non-zero, failure series at or near zero.*\n\n###### Watch for:\n*A failure series dominating while connected stays at zero means the node has no outbound peers and cannot sync at all. Use the trace drill-down to find which endpoint keeps failing \u2014 the aggregate rate cannot tell one bad peer from a broken local network.*\n\n###### Keywords:\n- **Outbound dial** *(per node)* \u2014 one attempt by this node to open a peer connection, spanning the TCP connect, the TLS handshake and the protocol upgrade.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Derived by the OTel Collector's spanmetrics connector from spans emitted by xrpld code; the collector counts the spans and their durations, and the Grafana query selects and aggregates the resulting series.*\n\n###### Source:\n[ConnectAttempt.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/ConnectAttempt.cpp)\n\n###### Function:\n`ConnectAttempt::reportOutcome`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#outbound-dial-latency)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Dials / Sec", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 0, - "y": 290 - }, - "id": 50, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_replace(sum by (outcome, service_instance_id, xrpl_branch, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", outcome=~\"$span_outcome\", span_name=\"peer.dial\"}[$__rate_interval])), \"series\", \"$1\", \"outcome\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "Outbound Dial Outcomes (span-derived, per attempt)", - "type": "timeseries" - }, - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "description": "###### What this is:\n*Rate at which this node answers peers' ledger-data requests, split by what was asked for and how the reply ended.*\n\n###### How it's computed:\n*Rate of span-derived call counts for the ledger.serve span, split by object_type (header, transaction tree, account-state tree, or a proposed transaction set) and by outcome. The outcome is derived from the reply itself, so it is refused whenever nothing was sent.*\n\n###### Reading it:\n*This is the supply side \u2014 what this node does for its peers, not what it receives. The account-state series is the one that matters to a syncing peer, since that tree is the bulk of a fresh sync.*\n\n###### Healthy range:\n*Non-zero complete series on the types peers ask for, refused near zero.*\n\n###### Watch for:\n*A refused series climbing means this node is declining to serve; the paired serve_refused_total counter on Ledger/Object Serve Refusals gives the specific cause. A partial series means replies keep hitting the size cap, so peers must make repeated round trips for one tree. Note this panel does not explain this node's own sync \u2014 it explains its peers'.*\n\n###### Keywords:\n- **Ledger serve** *(per node)* \u2014 this node answering a peer's request for ledger data, as opposed to requesting data for itself.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Derived by the OTel Collector's spanmetrics connector from spans emitted by xrpld code; the collector counts the spans and their durations, and the Grafana query selects and aggregates the resulting series.*\n\n###### Source:\n[PeerImp.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/PeerImp.cpp)\n\n###### Function:\n`PeerImp::processLedgerRequest`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-serve)", - "fieldConfig": { - "defaults": { - "color": { - "mode": "palette-classic" - }, - "custom": { - "axisBorderShow": false, - "axisCenteredZero": false, - "axisColorMode": "text", - "axisLabel": "Requests / Sec", - "axisPlacement": "auto", - "barAlignment": 0, - "barWidthFactor": 0.6, - "drawStyle": "line", - "fillOpacity": 10, - "gradientMode": "none", - "hideFrom": { - "legend": false, - "tooltip": false, - "viz": false - }, - "insertNulls": false, - "lineInterpolation": "linear", - "lineWidth": 2, - "pointSize": 3, - "scaleDistribution": { - "type": "linear" - }, - "showPoints": "auto", - "showValues": false, - "spanNulls": 1800000, - "stacking": { - "group": "A", - "mode": "none" - }, - "thresholdsStyle": { - "mode": "off" - } - }, - "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "thresholds": { - "mode": "absolute", - "steps": [ - { - "color": "green", - "value": null - } - ] - }, - "unit": "ops" - } - }, - "gridPos": { - "h": 12, - "w": 12, - "x": 12, - "y": 290 - }, - "id": 51, - "options": { - "annotations": { - "clustering": -1, - "multiLane": false - }, - "legend": { - "calcs": [], - "displayMode": "list", - "enableFacetedFilter": false, - "overflow": "ellipsis", - "placement": "bottom", - "showLegend": true - }, - "tooltip": { - "hideZeros": false, - "maxHeight": 600, - "mode": "multi", - "sort": "desc" - } - }, - "pluginVersion": "13.2.0-28926505616", - "targets": [ - { - "datasource": { - "type": "prometheus", - "uid": "${DS_PROMETHEUS}" - }, - "expr": "label_replace(label_join(label_join(sum by (object_type, outcome, service_instance_id, xrpl_branch, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", object_type=~\"$object_type\", outcome=~\"$span_outcome\", span_name=\"ledger.serve\"}[$__rate_interval])), \"series\", \" \", \"object_type\", \"outcome\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", - "refId": "A" - } - ], - "title": "Ledger Serve Rate by Object Type", - "type": "timeseries" - }, { "datasource": { "type": "prometheus", @@ -4823,7 +3477,7 @@ "h": 12, "w": 12, "x": 0, - "y": 302 + "y": 222 }, "id": 52, "options": { @@ -4926,7 +3580,7 @@ "h": 12, "w": 12, "x": 12, - "y": 302 + "y": 222 }, "id": 53, "options": { @@ -4971,12 +3625,96 @@ "title": "Consensus Round Duration (p50/p95)", "type": "timeseries" }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 234 + }, + "id": 61, + "panels": [], + "title": "Terminal blockers & serving", + "type": "row" + }, { "datasource": { "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Rate of trusted validations reaching the ledger-acceptance gate, split by what the validation store did with each one and by whether it actually reached the gate.*\n\n###### How it's computed:\n*Rate of span-derived calls for the consensus.validation.accept span, by validation_status and accept_gated. The span is emitted once per trusted validation and carries the trace id of the ledger it validates, so any point here can be opened as the full trace for that ledger \u2014 the validation, the acceptance decision it drove, and the acquire and store spans for the same ledger.*\n\n###### Reading it:\n*Nearly all of the rate should be validation_status=current, which is the only status that continues to the gate. accept_gated=true means another thread was already accepting that ledger, so no acceptance followed this validation; a modest share is normal when validations for one ledger arrive together.*\n\n###### Healthy range:\n*Dominated by current, at roughly the trusted-validator count per ledger close.*\n\n###### Watch for:\n*Rate concentrated in stale, bad_seq, multiple or conflicting: validations are arriving and being counted for nothing, which is the difference between a node that is slow to validate and one that never will \u2014 from the outside the two look identical. Confirm on Trusted Validations vs Quorum Target and Pre-Accept Quorum Shortfall Rate, and check UNL Quorum Headroom in the Bootstrap row first, since a trusted list that cannot satisfy quorum starves everything below it. A flat zero on a peered node means no trusted validations are arriving at all.*\n\n###### Keywords:\n- **Validation status** *(per node)* \u2014 what this node's validation store did with an arriving validation; only current counts toward accepting a ledger.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Derived by the OTel Collector's spanmetrics connector from spans emitted by xrpld code; the collector counts the spans, and the Grafana query selects and aggregates the resulting series.*\n\n###### Source:\n[RCLValidations.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/consensus/RCLValidations.cpp)\n\n###### Function:\n`handleNewValidation`\n\n###### References:\n[Negative UNL and validation quorum on xrpl.org](https://xrpl.org/docs/concepts/consensus-protocol/negative-unl) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-status)", + "description": "###### What this is:\n*Seconds until an unsupported amendment activates and this node stops validating for good. This is the LEADING indicator \u2014 the countdown before the block, while there is still time to upgrade.*\n\n###### How it's computed:\n*amendment_block series seconds_to_block. A value of -1 is the sentinel for \"no unsupported amendment is pending\", the same convention validator_health{metric=\"unl_expiry_days\"} already uses.*\n\n###### Reading it:\n*Because -1 is the healthy sentinel and a small positive number is the emergency, the colour scale is not monotonic \u2014 read the value, not only the colour. -1 is green and means nothing is pending. A large positive value is green above 7 days and yellow under 7 days: an unsupported amendment holds majority but there is still time to upgrade. A small positive value under 1 day is red: at 0 this node stops validating and does not resume without a software upgrade.*\n\n###### Healthy range:\n*-1.*\n\n###### Watch for:\n*Any value at or above 0. This is distinct from the Amendment Blocked stat on the Validator Health dashboard (validator_health{metric=\"amendment_blocked\"}), which reports the TERMINAL state \u2014 already blocked, too late to act. This panel is the window before that happens, so the two are read together: countdown first, terminal state as confirmation. The identity of the blocking amendment is deliberately NOT a metric label, because the network can vote on an arbitrary 256-bit amendment id and a label would be unbounded cardinality. The hash is logged instead by AmendmentTableImpl::doValidatedLedger (\"Unsupported amendment reached majority at ...\") and is available via Loki.*\n\n###### Keywords:\n- **Amendment block countdown** *(per node)* \u2014 seconds until an unsupported amendment that has reached majority activates and blocks this node; -1 when none is pending.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerAmendmentBlockGauge`\n\n###### References:\n[Amendments on xrpl.org](https://xrpl.org/docs/concepts/networks-and-servers/amendments) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#amendment-blocked)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 0 + }, + { + "color": "yellow", + "value": 86400 + }, + { + "color": "green", + "value": 604800 + } + ] + }, + "unit": "s" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 235 + }, + "id": 36, + "options": { + "colorMode": "value", + "graphMode": "none", + "justifyMode": "center", + "orientation": "auto", + "percentChangeColorMode": "standard", + "reduceOptions": { + "calcs": ["lastNotNull"], + "fields": "", + "values": false + }, + "showPercentChange": false, + "textMode": "value_and_name", + "wideLayout": true + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(amendment_block{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"seconds_to_block\"}, \"series\", \"Seconds to Block\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Amendment Block Countdown", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Whether an amendment this build does not support has reached majority on the network.*\n\n###### How it's computed:\n*amendment_block series warned: 1 while an unsupported amendment holds majority, 0 otherwise.*\n\n###### Reading it:\n*0 is healthy. A 1 is the first warning that an upgrade is required, and it is raised before the amendment activates rather than after.*\n\n###### Healthy range:\n*0.*\n\n###### Watch for:\n*The transition from 0 to 1 \u2014 that is the moment the upgrade clock starts. Read Amendment Block Countdown next for how long is left, and the Amendment Blocked stat on the Validator Health dashboard for whether the block has already happened; that one is the terminal state, this one is the warning. Which amendment is blocking is not a label (an arbitrary 256-bit amendment id would be unbounded cardinality); the hash is logged by AmendmentTableImpl::doValidatedLedger and is available via Loki.*\n\n###### Keywords:\n- **Amendment warning** *(per node)* \u2014 an unsupported amendment has reached majority; the node still validates, but will stop when that amendment activates.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerAmendmentBlockGauge`\n\n###### References:\n[Amendments on xrpl.org](https://xrpl.org/docs/concepts/networks-and-servers/amendments) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#amendment-blocked)", "fieldConfig": { "defaults": { "color": { @@ -4986,7 +3724,7 @@ "axisBorderShow": false, "axisCenteredZero": false, "axisColorMode": "text", - "axisLabel": "Validations / Sec", + "axisLabel": "Warned", "axisPlacement": "auto", "barAlignment": 0, "barWidthFactor": 0.6, @@ -5013,7 +3751,7 @@ "mode": "none" }, "thresholdsStyle": { - "mode": "off" + "mode": "line" } }, "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", @@ -5023,20 +3761,23 @@ { "color": "green", "value": null + }, + { + "color": "red", + "value": 1 } ] }, - "unit": "ops" - }, - "overrides": [] + "unit": "short" + } }, "gridPos": { "h": 12, - "w": 24, - "x": 0, - "y": 314 + "w": 12, + "x": 12, + "y": 235 }, - "id": 54, + "id": 37, "options": { "annotations": { "clustering": -1, @@ -5064,12 +3805,1364 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_join(sum by (validation_status, accept_gated, service_instance_id, xrpl_branch, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.validation.accept\", validation_status=~\"$validation_status\"}[$__rate_interval])), \"series\", \" gated=\", \"validation_status\", \"accept_gated\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(amendment_block{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"warned\"}, \"series\", \"Warned\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "A" } ], - "title": "Trusted Validation Accept Rate by Status", + "title": "Amendment Warned", "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Peer requests that this node declined to serve, split by why it declined and by what was asked for.*\n\n###### How it's computed:\n*Per-second rate of refused serve attempts grouped by reason and request type (ledger, tx-set, object or fetchpack), per node. Filter with the Serve Reason and Serve Request variables, which list the emitted label values verbatim (the tx-set one is a single unhyphenated word).*\n\n###### Reading it:\n*This is the other side of the sync exchange: every other panel here measures what this node fetches, this one measures what it refuses to give back. sendq_full and load_shed are self-inflicted backpressure \u2014 the node holds the data but will not send it. not_found and no_map mean the requester asked for history this node does not hold.*\n\n###### Healthy range:\n*Near zero.*\n\n###### Watch for:\n*sendq_full or load_shed climbing. This node is starving its peers, and on a network of similarly loaded nodes that is exactly the condition that makes everybody's sync slow, so a refusal here can be the cause of another operator's stall. A high not_found is usually benign on a node configured with short history, but it tells peers to look elsewhere.*\n\n###### Keywords:\n- **Serve refusal** *(per node)* \u2014 a peer request for ledger, transaction-set, object or fetch-pack data that this node declined, together with the reason it declined.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[PeerImp.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/PeerImp.cpp)\n\n###### Function:\n`PeerImp::processLedgerRequest` \u00b7 `PeerImp::onMessage(TMGetObjectByHash)` \u00b7 `PeerImp::doFetchPack`\n\n###### References:\n[Fetch-pack](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#fetch-pack) \u00b7 [GetObject / object fetch](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#getobject-object-fetch)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Refusals / Sec", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 0, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 1, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 247 + }, + "id": 34, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_join(sum by (reason, request, service_instance_id, xrpl_branch, xrpl_work_item) (rate(serve_refused_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", reason=~\"$serve_reason\", request=~\"$serve_request\"}[$__rate_interval])), \"series\", \" \", \"reason\", \"request\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Ledger/Object Serve Refusals", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Rate at which this node abandoned the chain tip it had built and switched to a different last-closed ledger reported by the network.*\n\n###### How it's computed:\n*Per-second rate of last-closed-ledger switches where the ledger the network reported was not the one this node built on, per node. The counter carries no labels, so the sum is over one series per node.*\n\n###### Reading it:\n*Any non-zero value means this node was told the network's last-closed ledger is not the one it built on, and it discarded its own chain tip in response. A single jump around startup or a restart is ordinary recovery.*\n\n###### Healthy range:\n*0.*\n\n###### Watch for:\n*Repeated jumps \u2014 that is wrong-chain thrash, not one-off recovery. Check the peer set and the configured network id: a node peered to the wrong network, or into a minority partition, keeps being overruled and keeps throwing away work. Pair with the Bootstrap row and with Ledgers Behind Network.*\n\n###### Keywords:\n- **Byzantine ledger jump** *(per node)* \u2014 the node replaced its own last-closed ledger with a different one reported by the network, discarding the chain tip it had built.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[NetworkOPs.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/misc/NetworkOPs.cpp)\n\n###### Function:\n`NetworkOPsImp::switchLastClosedLedger`\n\n###### References:\n[Fork](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#fork) \u00b7 [Ledger history mismatch](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-history-mismatch)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Jumps / Sec", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "line" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 0.001 + } + ] + }, + "unit": "ops" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 247 + }, + "id": 35, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(ledger_jump_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval])), \"series\", \"Ledger Jumps\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Byzantine Ledger Jumps", + "type": "timeseries" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 259 + }, + "id": 62, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Node-store write latency next to read latency, in microseconds per operation. The write side is the signal: a node with a large existing database back-fills slower than a fresh one, and back-fill is write-bound, so no read-side metric can show it.*\n\n###### How it's computed:\n*nodestore_latency series write_mean_us and read_mean_us, each divided by its own count series so the reading is the latency during the selected interval rather than the average since boot. The write numerator comes from a store-duration total that was declared but never written before this signal existed.*\n\n###### Reading it:\n*Compare the two lines. Reads far above writes points at the read path or a cold cache; writes far above reads points at backend write pressure, which is the large-existing-database case.*\n\n###### Healthy range:\n*Both well under a few hundred microseconds on healthy local storage.*\n\n###### Watch for:\n*A rising write line during history back-fill: the backend cannot absorb writes fast enough and sync will stay slow no matter how many peers are available. Read with Fetch-Pack Peer Starvation to tell a data-supply problem from a disk problem. This is a mean, not a percentile \u2014 a tail that matters will move it, but p99 is not available from this signal.*\n\n###### Keywords:\n- **Node-store write latency** *(per node)* \u2014 how long the node store takes to persist one object.\n- **Node-store read latency** *(per node)* \u2014 how long the node store takes to retrieve one object.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreLatencyGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#node-store-write-latency)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Microseconds / Op", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "\u00b5s" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 260 + }, + "id": 38, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"write_mean_us\"}[$__rate_interval])) / clamp_min(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"write_count\"}[$__rate_interval])), 1), \"series\", \"Write us/op (interval)\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"read_mean_us\"}[$__rate_interval])) / clamp_min(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"read_count\"}[$__rate_interval])), 1), \"series\", \"Read us/op (interval)\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "B" + } + ], + "title": "NodeStore Write vs Read Latency (us/op)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Node-store write and read operation rates \u2014 the denominators behind the latency panel.*\n\n###### How it's computed:\n*Rate of the nodestore_latency write_count and read_count series.*\n\n###### Reading it:\n*Writes climb while a node is back-filling history and fall to near the ledger-close rate once it is caught up.*\n\n###### Healthy range:\n*Non-zero writes whenever the node is ingesting ledgers.*\n\n###### Watch for:\n*Write rate at zero while the node is still behind the network: nothing is being persisted, so the stall is upstream of the node store \u2014 check peer supply and the acquire panels rather than storage. A flat latency with a collapsing operation rate also means the latency figure above has gone stale rather than good.*\n\n###### Keywords:\n- **Node-store operation rate** *(per node)* \u2014 stores and fetches per second against the node store.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreLatencyGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#node-store-operation-rate)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Ops / Sec", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 260 + }, + "id": 39, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"write_count\"}[$__rate_interval])), \"series\", \"Writes/s\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"read_count\"}[$__rate_interval])), \"series\", \"Reads/s\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "B" + } + ], + "title": "NodeStore Operation Rate (writes vs reads)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Rate at which ledger-replay sub-tasks give up and fall back to acquiring a whole ledger, split by which sub-task gave up.*\n\n###### How it's computed:\n*Rate of ledger_replay_fallback_total by stage. The skip-list stage and the delta stage each emit once, on the transition into fallback, from the branch that was debug-log-only before.*\n\n###### Reading it:\n*Zero when replay-capable peers are available. Any sustained rate means the replay optimisation is being defeated and back-fill has reverted to the slower full-acquire path.*\n\n###### Healthy range:\n*Zero, or brief spikes while the peer set changes.*\n\n###### Watch for:\n*A persistent rate on either stage: too few connected peers support the ledger-replay feature, so every historical ledger is fetched in full instead of as a delta. Read with Replay Outcomes.*\n\n###### Keywords:\n- **Replay fallback** *(per node)* \u2014 a replay sub-task abandoning the delta shortcut and acquiring the entire ledger instead.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[SkipListAcquire.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/SkipListAcquire.cpp)\n\n###### Function:\n`SkipListAcquire::trigger` / `LedgerDeltaAcquire::trigger`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#replay-fallback)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Fallbacks / Sec", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 272 + }, + "id": 40, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sum by (stage, service_instance_id, xrpl_branch, xrpl_work_item) (rate(ledger_replay_fallback_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", stage=~\"$replay_stage\"}[$__rate_interval])), \"series\", \"$1\", \"stage\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Replay Fallback to Full Acquire (by stage)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Rate of ledger-replay tasks reaching each terminal state: success, timeout, build failure or parameter failure.*\n\n###### How it's computed:\n*Rate of ledger_replay_outcome_total by outcome. Every terminal path in the replay task emits exactly once; before this, all four only set an internal flag and wrote a log line.*\n\n###### Reading it:\n*Successes only is healthy. Timeouts mean deltas never arrived; build failures mean a delta would not apply to its parent; parameter failures mean a peer served an inconsistent skip list.*\n\n###### Healthy range:\n*Successes non-zero while replaying, all failure outcomes at zero.*\n\n###### Watch for:\n*Any failure outcome climbing while successes stay flat: replay is running but never completing, so history back-fill is silently falling back to the slower path. The outcome value tells you which layer to look at \u2014 timeouts point at peers, build and parameter failures point at the data those peers served.*\n\n###### Keywords:\n- **Ledger replay** *(per node)* \u2014 rebuilding a range of historical ledgers from a start ledger plus per-ledger deltas instead of downloading each one whole.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[LedgerReplayTask.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/LedgerReplayTask.cpp)\n\n###### Function:\n`LedgerReplayTask::recordOutcome`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-replay)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Tasks / Sec", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 272 + }, + "id": 41, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sum by (outcome, service_instance_id, xrpl_branch, xrpl_work_item) (rate(ledger_replay_outcome_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", outcome=~\"$replay_outcome\"}[$__rate_interval])), \"series\", \"$1\", \"outcome\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Replay Outcomes (by terminal state)", + "type": "timeseries" + } + ], + "title": "Back-fill & persistence", + "type": "row" + }, + { + "collapsed": true, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 260 + }, + "id": 63, + "panels": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*95th-percentile duration of each of the three phases of one ledger acquisition: the header wait, the account-state tree fetch, and the transaction tree fetch.*\n\n###### How it's computed:\n*95th percentile of the span-derived duration histogram for the three ledger.acquire.header / .astree / .txtree child spans, one series per phase. The parent ledger.acquire span is flat and cannot separate them, which matters because the account-state tree is nearly all of the work in a real fresh sync.*\n\n###### Reading it:\n*The astree series dominating is expected and healthy. The value of the split is the comparison: a header series that is large means the node is waiting to be told what to fetch, which is a peer-supply problem upstream of either tree.*\n\n###### Healthy range:\n*The astree series largest, the txtree series small, the header series near zero.*\n\n###### Watch for:\n*A hot astree band that keeps growing means account-state nodes are not being served \u2014 check Ledger Acquire Phase Outcomes beside it for timed_out, and Outbound Dial Outcome Rate in the Bootstrap row for refused or timed-out dials. A large header series instead means no peer is answering the header request at all.*\n\n###### Keywords:\n- **Acquire phase** *(per node)* \u2014 one of the three sequential fetches a ledger acquisition is made of; the header gates both trees, because it is what names their root hashes.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Derived by the OTel Collector's spanmetrics connector from spans emitted by xrpld code; the collector counts the spans and their durations, and the Grafana query selects and aggregates the resulting series.*\n\n###### Source:\n[InboundLedger.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/InboundLedger.cpp)\n\n###### Function:\n`InboundLedger::syncPhaseSpans`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#acquire-phase)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Duration", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ms" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 261 + }, + "id": 48, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, span_name, service_instance_id, xrpl_branch, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"ledger.acquire.header\"}[$__rate_interval]))), \"series\", \"P95 header\", \"span_name\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, span_name, service_instance_id, xrpl_branch, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"ledger.acquire.astree\"}[$__rate_interval]))), \"series\", \"P95 account-state tree\", \"span_name\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "B" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, span_name, service_instance_id, xrpl_branch, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"ledger.acquire.txtree\"}[$__rate_interval]))), \"series\", \"P95 transaction tree\", \"span_name\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "C" + } + ], + "title": "Ledger Acquire Phase Duration (p95 by phase)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Rate at which each ledger-acquire phase ends, split by which phase it was and whether it ran out of retry budget.*\n\n###### How it's computed:\n*Rate of span-derived call counts for the three ledger.acquire.* child spans, split by span name and by the timed_out flag. The flag is a separate dimension from the outcome because a phase can time out and still be retried by its parent acquire.*\n\n###### Reading it:\n*Series with timed_out=false are phases that finished on their own. Any series with timed_out=true is a phase whose retry budget expired, which names the exact stage of the fetch that peers are not serving.*\n\n###### Healthy range:\n*All series timed_out=false; zero everywhere on a node with no back-fill to do.*\n\n###### Watch for:\n*Any sustained timed_out=true rate. Pair it with the missing-node count on Missing SHAMap Nodes per Acquire: a timed-out phase together with a flat, non-zero missing-node count is the definitive stuck-sync signature, and the phase name says which tree to chase.*\n\n###### Keywords:\n- **Acquire phase timeout** *(per node)* \u2014 an acquire phase that ended because its retry budget expired rather than because its data arrived.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Derived by the OTel Collector's spanmetrics connector from spans emitted by xrpld code; the collector counts the spans and their durations, and the Grafana query selects and aggregates the resulting series.*\n\n###### Source:\n[InboundLedger.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/InboundLedger.cpp)\n\n###### Function:\n`InboundLedger::endPhaseSpan`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#acquire-phase)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Phases / Sec", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 261 + }, + "id": 49, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_join(label_replace(sum by (span_name, timed_out, service_instance_id, xrpl_branch, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", timed_out=~\"$timed_out\", span_name=~\"ledger.acquire..*\"}[$__rate_interval])), \"phase\", \"$1\", \"span_name\", \"ledger\\.acquire\\.(.*)\"), \"series\", \" timed_out=\", \"phase\", \"timed_out\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Ledger Acquire Phase Outcomes (by phase & timeout)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Rate at which attempts to fetch a transaction set referenced by a consensus proposal reach each terminal state.*\n\n###### How it's computed:\n*Rate of span-derived call counts for the txset.acquire span, split by outcome. The span is new: transaction-set acquisition had no telemetry at all before, so a consensus round stalled waiting on a set looked identical to an idle one. Every exit stamps an outcome, including the sweep that drops a set which never arrived.*\n\n###### Reading it:\n*Only complete is healthy. Timeout means peers never supplied the set. Abandoned means the round moved on and the set was dropped mid-fetch. Failed means a peer served data that would not build.*\n\n###### Healthy range:\n*A low rate of complete during consensus, everything else at zero. Zero everywhere is also normal on a node that already holds every proposed set locally.*\n\n###### Watch for:\n*Timeout or abandoned climbing: proposed sets are not arriving, so rounds are waiting on data rather than on agreement. Read with the Tx-Set Acquire Duration panel beside it \u2014 a rising p95 with outcomes still complete is slow-but-working, while the same p95 with timeouts is the set never arriving at all.*\n\n###### Keywords:\n- **Tx-set acquire** *(per node)* \u2014 one attempt to fetch the transaction set a consensus proposal referenced but this node did not hold.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Derived by the OTel Collector's spanmetrics connector from spans emitted by xrpld code; the collector counts the spans and their durations, and the Grafana query selects and aggregates the resulting series.*\n\n###### Source:\n[TransactionAcquire.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/TransactionAcquire.cpp)\n\n###### Function:\n`TransactionAcquire::finalizeAcquireSpan`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#tx-set-acquire)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Acquisitions / Sec", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 273 + }, + "id": 46, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sum by (outcome, service_instance_id, xrpl_branch, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", outcome=~\"$span_outcome\", span_name=\"txset.acquire\"}[$__rate_interval])), \"series\", \"$1\", \"outcome\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Tx-Set Acquire Outcomes", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*95th-percentile wall time of one transaction-set acquisition, split by how it ended.*\n\n###### How it's computed:\n*95th percentile of the span-derived duration histogram for the txset.acquire span, grouped by outcome.*\n\n###### Reading it:\n*Splitting by outcome is what makes this readable: the complete series is how long a successful fetch takes, while the timeout series is pinned near the retry budget by construction and carries no information about speed.*\n\n###### Healthy range:\n*A complete-series p95 well under a consensus round interval.*\n\n###### Watch for:\n*A complete-series p95 approaching the round interval: sets are arriving, but so late that they delay the round they belong to. This is the case that a pure outcome rate cannot show, because those acquisitions do succeed.*\n\n###### Keywords:\n- **Tx-set acquire** *(per node)* \u2014 one attempt to fetch the transaction set a consensus proposal referenced but this node did not hold.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Derived by the OTel Collector's spanmetrics connector from spans emitted by xrpld code; the collector counts the spans and their durations, and the Grafana query selects and aggregates the resulting series.*\n\n###### Source:\n[TransactionAcquire.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/TransactionAcquire.cpp)\n\n###### Function:\n`TransactionAcquire::init`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#tx-set-acquire)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Duration", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ms" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 273 + }, + "id": 47, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, outcome, service_instance_id, xrpl_branch, xrpl_work_item) (rate(span_duration_milliseconds_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", outcome=~\"$span_outcome\", span_name=\"txset.acquire\"}[$__rate_interval]))), \"series\", \"P95 $1\", \"outcome\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Tx-Set Acquire Duration (p95)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Rate of outbound peer dials reaching each terminal outcome, derived from the per-attempt peer.dial span.*\n\n###### How it's computed:\n*Rate of span-derived call counts for the peer.dial span, split by outcome. The same five outcome values the overlay_connect_total counter carries, set from the same funnel in the dial state machine so the two cannot disagree.*\n\n###### Reading it:\n*Read alongside Outbound Dial Outcome Rate in the Bootstrap row, which is the native counter for the same events. This panel exists for what the counter cannot do: each point here is backed by traces, so clicking through gives the individual attempt and the peer address it was dialling, which is never a metric label because one series per peer address would be unbounded.*\n\n###### Healthy range:\n*The connected series non-zero, failure series at or near zero.*\n\n###### Watch for:\n*A failure series dominating while connected stays at zero means the node has no outbound peers and cannot sync at all. Use the trace drill-down to find which endpoint keeps failing \u2014 the aggregate rate cannot tell one bad peer from a broken local network.*\n\n###### Keywords:\n- **Outbound dial** *(per node)* \u2014 one attempt by this node to open a peer connection, spanning the TCP connect, the TLS handshake and the protocol upgrade.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Derived by the OTel Collector's spanmetrics connector from spans emitted by xrpld code; the collector counts the spans and their durations, and the Grafana query selects and aggregates the resulting series.*\n\n###### Source:\n[ConnectAttempt.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/ConnectAttempt.cpp)\n\n###### Function:\n`ConnectAttempt::reportOutcome`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#outbound-dial-latency)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Dials / Sec", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 0, + "y": 285 + }, + "id": 50, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace(sum by (outcome, service_instance_id, xrpl_branch, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", outcome=~\"$span_outcome\", span_name=\"peer.dial\"}[$__rate_interval])), \"series\", \"$1\", \"outcome\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Outbound Dial Outcomes (span-derived, per attempt)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Rate at which this node answers peers' ledger-data requests, split by what was asked for and how the reply ended.*\n\n###### How it's computed:\n*Rate of span-derived call counts for the ledger.serve span, split by object_type (header, transaction tree, account-state tree, or a proposed transaction set) and by outcome. The outcome is derived from the reply itself, so it is refused whenever nothing was sent.*\n\n###### Reading it:\n*This is the supply side \u2014 what this node does for its peers, not what it receives. The account-state series is the one that matters to a syncing peer, since that tree is the bulk of a fresh sync.*\n\n###### Healthy range:\n*Non-zero complete series on the types peers ask for, refused near zero.*\n\n###### Watch for:\n*A refused series climbing means this node is declining to serve; the paired serve_refused_total counter on Ledger/Object Serve Refusals gives the specific cause. A partial series means replies keep hitting the size cap, so peers must make repeated round trips for one tree. Note this panel does not explain this node's own sync \u2014 it explains its peers'.*\n\n###### Keywords:\n- **Ledger serve** *(per node)* \u2014 this node answering a peer's request for ledger data, as opposed to requesting data for itself.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Derived by the OTel Collector's spanmetrics connector from spans emitted by xrpld code; the collector counts the spans and their durations, and the Grafana query selects and aggregates the resulting series.*\n\n###### Source:\n[PeerImp.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/PeerImp.cpp)\n\n###### Function:\n`PeerImp::processLedgerRequest`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-serve)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Requests / Sec", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops" + } + }, + "gridPos": { + "h": 12, + "w": 12, + "x": 12, + "y": 285 + }, + "id": 51, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_join(sum by (object_type, outcome, service_instance_id, xrpl_branch, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", object_type=~\"$object_type\", outcome=~\"$span_outcome\", span_name=\"ledger.serve\"}[$__rate_interval])), \"series\", \" \", \"object_type\", \"outcome\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Ledger Serve Rate by Object Type", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "description": "###### What this is:\n*Rate of trusted validations reaching the ledger-acceptance gate, split by what the validation store did with each one and by whether it actually reached the gate.*\n\n###### How it's computed:\n*Rate of span-derived calls for the consensus.validation.accept span, by validation_status and accept_gated. The span is emitted once per trusted validation and carries the trace id of the ledger it validates, so any point here can be opened as the full trace for that ledger \u2014 the validation, the acceptance decision it drove, and the acquire and store spans for the same ledger.*\n\n###### Reading it:\n*Nearly all of the rate should be validation_status=current, which is the only status that continues to the gate. accept_gated=true means another thread was already accepting that ledger, so no acceptance followed this validation; a modest share is normal when validations for one ledger arrive together.*\n\n###### Healthy range:\n*Dominated by current, at roughly the trusted-validator count per ledger close.*\n\n###### Watch for:\n*Rate concentrated in stale, bad_seq, multiple or conflicting: validations are arriving and being counted for nothing, which is the difference between a node that is slow to validate and one that never will \u2014 from the outside the two look identical. Confirm on Trusted Validations vs Quorum Target and Pre-Accept Quorum Shortfall Rate, and check UNL Quorum Headroom in the Bootstrap row first, since a trusted list that cannot satisfy quorum starves everything below it. A flat zero on a peered node means no trusted validations are arriving at all.*\n\n###### Keywords:\n- **Validation status** *(per node)* \u2014 what this node's validation store did with an arriving validation; only current counts toward accepting a ledger.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Derived by the OTel Collector's spanmetrics connector from spans emitted by xrpld code; the collector counts the spans, and the Grafana query selects and aggregates the resulting series.*\n\n###### Source:\n[RCLValidations.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/consensus/RCLValidations.cpp)\n\n###### Function:\n`handleNewValidation`\n\n###### References:\n[Negative UNL and validation quorum on xrpl.org](https://xrpl.org/docs/concepts/consensus-protocol/negative-unl) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-status)", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "axisBorderShow": false, + "axisCenteredZero": false, + "axisColorMode": "text", + "axisLabel": "Validations / Sec", + "axisPlacement": "auto", + "barAlignment": 0, + "barWidthFactor": 0.6, + "drawStyle": "line", + "fillOpacity": 10, + "gradientMode": "none", + "hideFrom": { + "legend": false, + "tooltip": false, + "viz": false + }, + "insertNulls": false, + "lineInterpolation": "linear", + "lineWidth": 2, + "pointSize": 3, + "scaleDistribution": { + "type": "linear" + }, + "showPoints": "auto", + "showValues": false, + "spanNulls": 1800000, + "stacking": { + "group": "A", + "mode": "none" + }, + "thresholdsStyle": { + "mode": "off" + } + }, + "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + } + ] + }, + "unit": "ops" + }, + "overrides": [] + }, + "gridPos": { + "h": 12, + "w": 24, + "x": 0, + "y": 297 + }, + "id": 54, + "options": { + "annotations": { + "clustering": -1, + "multiLane": false + }, + "legend": { + "calcs": [], + "displayMode": "list", + "enableFacetedFilter": false, + "overflow": "ellipsis", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "hideZeros": false, + "maxHeight": 600, + "mode": "multi", + "sort": "desc" + } + }, + "pluginVersion": "13.2.0-28926505616", + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_join(sum by (validation_status, accept_gated, service_instance_id, xrpl_branch, xrpl_work_item) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", span_name=\"consensus.validation.accept\", validation_status=~\"$validation_status\"}[$__rate_interval])), \"series\", \" gated=\", \"validation_status\", \"accept_gated\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "A" + } + ], + "title": "Trusted Validation Accept Rate by Status", + "type": "timeseries" + } + ], + "title": "Spans & traces", + "type": "row" } ], "schemaVersion": 39, diff --git a/docker/telemetry/workload/README.md b/docker/telemetry/workload/README.md index eb926af6b9..147ac79be0 100644 --- a/docker/telemetry/workload/README.md +++ b/docker/telemetry/workload/README.md @@ -203,7 +203,7 @@ Automated validation that all expected telemetry data exists. Every metric and s - **Span validation**: All span types from `expected_spans.json` with required attributes and parent-child hierarchies - **Metric validation**: All metrics from `expected_metrics.json` — SpanMetrics, StatsD gauges/counters/histograms, Phase 9 OTLP metrics. Every listed metric must have > 0 series. Uses the Prometheus `/api/v1/series` endpoint (not instant queries) to avoid false negatives from stale gauges. - **Log-trace correlation**: trace_id/span_id in Loki logs (requires Loki) -- **Dashboard validation**: All 10 Grafana dashboards load with panels +- **Dashboard validation**: All 15 Grafana dashboards load with panels ```bash # Run all validations diff --git a/docker/telemetry/workload/expected_spans.json b/docker/telemetry/workload/expected_spans.json index a10f17f7ea..fc8c5c43e2 100644 --- a/docker/telemetry/workload/expected_spans.json +++ b/docker/telemetry/workload/expected_spans.json @@ -521,6 +521,7 @@ } ] }, + "_conditional_attributes_note": "Five attributes documented in the 'Fresh-node sync diagnostics' table of OpenTelemetryPlan/09-data-collection-reference.md are deliberately absent from required_attributes above, because each is emitted only when its value is known and _validate_span_attributes_otlp() has no per-attribute optional flag -- listing one would fail CI red on a healthy run. ledger.acquire/peer_count is set only when finalizeAcquireSpan() is passed a peer count (InboundLedger.cpp), which the sweep and shutdown paths cannot supply. ledger_seq on the three ledger.acquire.header/.astree/.txtree phase spans is set only when seq_ != 0 (InboundLedger.cpp startPhaseSpan), and a by-hash acquire starts with seq_ == 0 and learns the sequence only when the header arrives -- so a phase that opens before the header legitimately carries no sequence. ledger_seq on ledger.serve is set only when the reply carries one (PeerImp.cpp), which an object-by-hash request does not. All five ARE indexed in the 09-reference table and rendered by the Ledger Sync Health board; the honest encoding is to document them here rather than assert a conditional attribute as required.", "total_span_types": 47, "total_unique_attributes": 76 } diff --git a/docs/telemetry-glossary.md b/docs/telemetry-glossary.md index 6e7d44dba4..0886f93d4f 100644 --- a/docs/telemetry-glossary.md +++ b/docs/telemetry-glossary.md @@ -723,7 +723,27 @@ The server state describes how fully the node is participating, in ascending ord **Scope:** per node — measured on and specific to this individual server. -**See also:** [Time to first FULL](#time-to-first-full) · [Network ledger gate](#network-ledger-gate) · [Operating mode / server state on xrpl.org](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) +**See also:** [Time to first FULL](#time-to-first-full) · [Network ledger gate](#network-ledger-gate) · [Mode flapping](#mode-flapping) · [Operating mode / server state on xrpl.org](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) + + + +### Mode flapping + +Mode flapping is a node repeatedly reaching the full server state and losing it again, rather than climbing to it once and staying. It is visible only because state transitions are recorded as a from-to edge: a clean fresh sync traverses each climb edge roughly once, whereas flapping shows repeated counts on a reverse edge paired with its forward partner. A bare transition count cannot distinguish the two, which is why the edge labels exist. Flapping alongside healthy ledger acquisition points away from the acquire pipeline and at whatever drops a node out of full once it has arrived — a stalling main loop, a clock disagreeing with the network, or a trusted list that keeps failing quorum. + +**Scope:** per node — measured on and specific to this individual server. + +**See also:** [Operating mode / server state](#operating-mode-server-state) · [Server stall](#server-stall) · [Clock close offset](#clock-close-offset) · [Validation quorum](#validation-quorum) + + + +### Sentinel reading + +A sentinel reading is a deliberately out-of-range value a gauge reports to mean "this condition does not apply", chosen so the healthy or unknown state is a distinct value rather than a missing series. The distinction matters because these are observable gauges: they report on every collection tick whatever the value, so absence is a regression while an unusual number may be the intended answer. Three appear in the sync diagnostics: an amendment-block countdown of -1 means nothing is pending, and is not a negative duration; a quorum target at the signed 64-bit maximum means quorum has been switched off entirely because too many list publishers are unavailable, reported as that maximum rather than allowed to wrap negative so it cannot be misread as a target already exceeded; and a peer supply window of zero means no peer has advertised a range yet, meaning unknown rather than the start of history. A one-shot duration reading of zero is the related case — it means the milestone was never reached, not that it took no time. + +**Scope:** per node — measured on and specific to this individual server. + +**See also:** [Amendment block countdown](#amendment-block-countdown) · [Validation quorum](#validation-quorum) · [Peer ledger supply](#peer-ledger-supply) · [Time to first FULL](#time-to-first-full) · [Time to first validated ledger](#time-to-first-validated-ledger) diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index 0872643ccc..378083dddd 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -2047,17 +2047,192 @@ The `getKBUsed*()` methods require SQLite databases to exist. If running with A fresh node that is slow to reach `server_state=full`, or never reaches it, is diagnosed from the **Ledger Sync Health** dashboard (uid `ledger-sync-health`). +Its nine rows are ordered the way a fresh node progresses, so reading the board +top-to-bottom walks the same path a sync does: -Walk the dashboard rows top-down — the first row that looks wrong is the stage -that is blocking: +| # | Row | Question it answers | +| --- | ----------------------------------- | ---------------------------------------------------- | +| 1 | Bootstrap (Domain 0) | Can the node reach peers and form a quorum at all? | +| 2 | Peer supply | Does any peer hold what this node needs? | +| 3 | Sync state | Is the node advancing through the mode machine? | +| 4 | Ledger acquire & SHAMap fetch | Is ledger data arriving and being applied? | +| 5 | Job queue | Does arrived work ever get a worker thread? | +| 6 | Quorum & publish | Does a held ledger ever validate, and reach clients? | +| 7 | Terminal blockers & serving | Will the node stop validating for good? | +| 8 | Back-fill & persistence (collapsed) | Is an existing database the bottleneck? | +| 9 | Spans & traces (collapsed) | _Which_ fetch, peer or object — not how many? | -1. **Bootstrap (Domain 0)** — can the node reach peers and form a quorum at all? - Covers DNS resolution, outbound dial, protocol/network-ID negotiation, UNL - fetch with trusted-key-vs-quorum, and clock skew. A node that never gets past - this row can never validate, no matter how healthy the pipeline looks. -2. **Sync pipeline** — is ledger data actually arriving and being applied? - Covers sync-state transitions, ledger/tx-set acquire progress, job-queue - backlog, quorum and publish lag, back-fill, and persistence latency. +**Start from the symptom, not from row 1.** Match what you observe to a branch +below; each branch names the panels that discriminate it, what healthy and +unhealthy look like, and what to conclude. The branch then points at the +numbered ordered-diagnosis steps further down, which are the detail. + +```mermaid +flowchart TD + S["Node is not reaching
server_state = full"] --> Q1{"Amendment Block Countdown
at -1?"} + Q1 -->|"No: counting down"| T["Branch F
Terminal blocker"] + Q1 -->|"Yes: healthy"| Q2{"Which sync mode
is it stuck in?"} + + Q2 -->|"disconnected"| A["Branch A
DNS / dial / negotiation"] + Q2 -->|"connected or syncing,
no validated ledger"| B["Branch B
UNL, quorum, clock"] + Q2 -->|"acquiring,
never finishing"| C["Branch C
SHAMap fetch and job queue"] + Q2 -->|"reaches full,
slowly or flapping"| D["Branch D
Publish, back-fill, rounds"] + Q2 -->|"was fine when the
DB was empty"| E["Branch E
Node-store and cache"] + + classDef start fill:#1f3b57,stroke:#8ab4d8,stroke-width:2px,color:#ffffff + classDef gate fill:#4a3a10,stroke:#d8b45a,stroke-width:2px,color:#ffffff + classDef leaf fill:#f2f6fa,stroke:#4a6f8a,stroke-width:1px,color:#12232e + classDef stop fill:#5c1f1f,stroke:#e08a8a,stroke-width:2px,color:#ffffff + class S start + class Q1,Q2 gate + class A,B,C,D,E leaf + class T stop +``` + +**Check branch F first, always.** It is the only branch whose window closes: +once an unsupported amendment activates there is no operational fix, so it +outranks every other symptom regardless of what the mode machine says. + +#### Branch A — stuck at `disconnected` + +Peer count flat at zero; _Mode Transitions by Edge_ shows the node never leaving +`disconnected`, or churning straight back to it. + +| Look at | Healthy | Unhealthy | Conclude | +| ------------------------------------------ | -------------------------------------- | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| _DNS Resolve Outcome Rate_ | all rate on `outcome=resolved` | any rate on `empty`, or both flat at zero | a name in `[ips]`/`[ips_fixed]` returns no address, or the list is empty — fix the hostname or use an IP | +| _DNS Resolve Latency (p95)_ | milliseconds | seconds-scale | the resolver is timing out and delaying every dial behind it | +| _Outbound Dial Outcome Rate_ | `connected` non-zero | all attempts on one failure outcome | `tcp_fail` = route/firewall/closed port · `tls_fail` = TLS · `upgrade_fail` = negotiation, go to the next row · `timeout` = never terminal | +| _Outbound Dial Latency (p95)_ | well under the dial timeout | pinned near it | peers accept TCP but never finish the handshake | +| _Handshake Negotiation Failures by Reason_ | flat, or a low background rate | any sustained `reason` | `wrong_network`/`invalid_network_id` is the most common fresh-node fault — the node is on a different network and can never reach quorum; `clock_skew` sends you to branch B | +| _PeerFinder Slot Census_ | `out_active` climbing toward `out_max` | `connecting` non-zero with `out_active` low; or `bootcache` and `livecache` both 0 | dials never complete; or there is nothing to dial at all | + +**Conclusion:** the node has no usable overlay. Nothing downstream can be +diagnosed until `connected` on _Outbound Dial Outcome Rate_ is non-zero. Detail: +[Bootstrap ordered diagnosis](#bootstrap-domain-0--ordered-diagnosis) steps 1-3 +and [Sync-pipeline](#sync-pipeline--ordered-diagnosis) step 12. + +#### Branch B — stuck at `connected`/`syncing`, no validated ledger + +The node has peers, `server_state` reaches `connected` or `syncing`, and +_Time to First Validated Ledger_ stays flat at zero. + +| Look at | Healthy | Unhealthy | Conclude | +| ---------------------------------------------------- | ------------------------------------------------------------------------- | ------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| _UNL Fetch Rate by Site & Outcome_ | `accepted` once per site, then `same_sequence`/`known_sequence` refreshes | `fetch_error`/`bad_status`/`parse_error` | the publisher site is unreachable, so no keys load from it | +| | | `expired` | the list applied but is past its validity window — refresh the blob and check the local clock, do not replace `validators.txt` | +| _UNL Trusted Keys vs Quorum_ + _UNL Quorum Headroom_ | `trusted_keys` above `quorum`, headroom positive | headroom zero or negative (red) | **the node will never validate**: the trusted list is too small to satisfy quorum. It can track ledgers forever and never declare one validated | +| | | `quorum` about 9.2e18 | the **quorum-disabled** sentinel — too many publishers unavailable. Fix publisher reachability; the key count is irrelevant until quorum is re-enabled | +| _Clock Close Offset_ | magnitude decaying under 1 s | above 1 s and not decaying | local NTP fault delaying consensus participation, and the cause behind a `clock_skew` handshake reason | +| _Network Ledger Gate_ | clears within minutes of startup | persistent 1 (red) | the node has never seen a complete network ledger — go back to branch A, not deeper | +| _Trusted Validations vs Quorum Target_ | tally climbing toward the target | tally flat below the target | **stuck at the quorum gate**: validations arrive and never reach quorum. Judge by the sustained floor over minutes, never one sample — the first evaluation of each round runs before peers' validations arrive, so a healthy node sawtooths | +| | | both flat at 0 | the gate has never been evaluated; nothing has been offered — upstream problem, back to branch A | +| _Trusted Validation Accept Rate by Status_ (row 9) | nearly all `current` | concentrated in `stale`, `bad_seq`, `multiple`, `conflicting` | validations arrive and are counted for nothing — this is what separates "slow to validate" from "never will", and it is invisible in the tally. Bulk `stale` = clock/timing; bulk `multiple`/`conflicting` = a misbehaving trusted validator or two chains | + +**Conclusion:** the fault is in trust and time, not in data supply. Detail: +[Bootstrap](#bootstrap-domain-0--ordered-diagnosis) steps 4-5 and +[Sync pipeline](#sync-pipeline--ordered-diagnosis) steps 2 and 16. + +#### Branch C — acquiring but never finishing + +Ledger acquires are in flight, _Ledgers Behind Network_ is flat or rising, and +`full` never arrives. + +| Look at | Healthy | Unhealthy | Conclude | +| ------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| _Missing SHAMap Nodes per Acquire (state/tx)_ | falling toward zero (read the trend over minutes) | **flat and non-zero** | **no peer is serving that tree** — the node will sit here forever. Pinned at 256 is the per-sweep cap, meaningful only with the trend | +| _Acquire Stall Rate (no progress)_ | flat | sustained rate **together with** a flat missing-node count | the definitive stuck-sync signature: requesting and nobody answering | +| _Peers Able to Serve Needed Sequence_ | `peers_serving_next` above zero | `peers_serving_next` = 0 while `peers_reporting` > 0 | **decisive**: peers are connected and none holds the next needed ledger. Waiting cannot finish it — the peer set must change. Everything else in branch C will look starved as a consequence, so do not chase it | +| _Peer Ledger Supply Window_ | needed sequence inside `[supply_min_seq, supply_max_seq]` | needed below `supply_min_seq` | asking for history nobody kept — needs a full-history peer | +| | | needed above `supply_max_seq` | the peer set lags the real network; not a history problem | +| _Ledger Acquire Phase Outcomes (by phase & timeout)_ + _Ledger Acquire Phase Duration (p95 by phase)_ (row 9) | `header` short, `astree` the bulk | `astree` hot with `timed_out=true` and non-zero `missing_nodes` | the common stuck shape — peers are not supplying account-state nodes | +| | | `header` hot | the node is waiting to be **told what to fetch**; invisible in the missing-node counts, which are both still zero | +| _Add-Node Outcomes_ | `good` dominates | `duplicate` swamps `good` | bandwidth busy, acquire standing still — peers re-sending known data | +| | | `invalid` rising | a specific misbehaving peer, not a local fault | +| _Received-Data Stash Depth & In-Flight Acquires_ | stash drains | stash growing | data arrives faster than it is applied — a job-queue or disk problem, the **opposite** conclusion from a stall rate, and only this panel separates them | +| _Deferred Jobs by Type (starvation)_ | flat at 0 | sustained non-zero on `ledgerData`/`ledgerRequest` | a job the queue accepted then **withheld** at its concurrency limit of 3 — it appears in neither `waiting` nor `running`, so no other signal can show it. Starved `ledgerData` is exactly why the stash grows while missing nodes stay flat | +| _Worker Pool Saturation_ + _Worker Pool Capacity & Total Backlog_ | under 80% | 100% with `total_waiting` climbing | the pool is **exhausted** — every stage looks slow at once. Stop here; no per-subsystem fix helps while no thread is free | +| Acquire outcome `abandoned` in Tempo (`{name="ledger.acquire" && span.outcome="abandoned"}`) | absent | present | the acquire was swept or shut down before reaching a result — without this value a stuck-then-swept fetch had no `outcome` at all and vanished from every outcome rate | + +**Conclusion:** distinguish "nobody is serving it" (peer supply) from "it arrives +and we cannot process it" (job queue / disk). The two look identical in a log and +are separated only by the stash-depth and deferred-jobs panels. Detail: +[Sync pipeline](#sync-pipeline--ordered-diagnosis) steps 6-12 and 17. + +#### Branch D — reaching `full` but slowly, or falling back out of it + +_Time to First FULL_ has a value, or the node flaps between `full` and +`connected`. + +| Look at | Healthy | Unhealthy | Conclude | +| ------------------------------------------------------------------------------ | ------------------------------- | ---------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| _Publish Lag (validated minus published)_ | flat at 0 or 1 | positive and growing | validation is healthy and the **publish pipeline is not** — the node is current while its clients see stale data. Local processing fault: go to the job-queue and stall panels | +| | | flat at 0 on a node that never validated | not healthy, merely empty — read _Trusted Validations vs Quorum Target_ first | +| _Server Stall_ + _Server Stall Event Rate_ | both zero | large seconds, **flat** event rate | one long unresolved stall; past 600 s the server deliberately fails | +| | | small seconds, **rising** event rate | repeated short stalls — periodic work (sweeps, large writes), not one stuck operation | +| _Mode Transitions by Edge_ | each climb edge roughly once | repeated `full`→`connected` paired with `connected`→`full` | flapping: reaching `full` and losing it. Sends you to the stall panels or to branch B's clock and quorum panels — those are what drop a node out of `full` | +| _Consensus Round Duration (p50/p95)_ + _Consensus Round Duration Distribution_ | band steady | band drifting up, or a second high band | rounds are taking longer; read against _Tx-Set Acquire Duration (p95)_ (rounds waiting on data) and _Trusted Validations vs Quorum Target_ (validations arriving too late) | +| | | p95 climbing, p50 flat | a minority of rounds stall — the early form of a second band | +| | | both rising together | the network is slowing, not this node. Compare another node via `$node` first | +| _Tx-Set Acquire Outcomes_ + _Tx-Set Acquire Duration (p95)_ (row 9) | flat at zero, or all `complete` | `timeout`/`abandoned` climbing | proposed sets never complete — rounds wait on data, not on agreement. This is the consensus path, not history back-fill | +| | | `complete` p95 approaching the round interval | sets arrive but so late they delay their own round; the outcome rate cannot show this because they succeed | +| _Replay Fallback to Full Acquire (by stage)_ (row 8) | flat | any sustained rate | too few peers support the `LedgerReplay` feature, so every historical ledger is fetched whole. Nothing fails — the optimisation is simply gone, which is why it is easy to miss. `stage` names the sub-task: `skiplist` or `delta` | +| _Replay Outcomes (by terminal state)_ (row 8) | `success` climbing | `timeout` climbing | deltas never arrived — treat as peer supply, read with branch C | +| | | `build_failed`/`parameter_failed` | **data** faults from the serving peers, not slowness — the peer set is suspect | + +**Conclusion:** the pipeline works; something behind it is not keeping up. +Publish lag and stalls are local, round duration is often network-wide, and +replay fallback is a silent loss of an optimisation rather than a failure. +Detail: [Sync pipeline](#sync-pipeline--ordered-diagnosis) steps 3, 5, 15, 16 +and 18. + +#### Branch E — an existing database syncs slower than a fresh one + +The specific symptom: a node with history starts and is slower than the same node +was when empty. Back-fill is **write**-bound, so no read-side panel shows it. +Expand the collapsed **Back-fill & persistence** row. + +| Look at | Healthy | Unhealthy | Conclude | +| ----------------------------------------------------------- | -------------------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| _NodeStore Write vs Read Latency (us/op)_ | write line flat and low | write line rising during back-fill | the backend cannot absorb writes. Adding peers will not help — check storage IOPS, the `[node_db]` backend and whether online-delete/rotation competes with the back-fill | +| | | read line far above write | the read path is the cost; read with the cache-hit panel below | +| _NodeStore Operation Rate (writes vs reads)_ | write rate non-zero while behind | write rate zero while still behind the network | nothing is being persisted — the stall is **upstream** of the node store. Go to branch C; storage is not the problem | +| _SHAMap TreeNode Cache Hit Rate_ | rising as the cache warms | persistently low | the working set does not fit the cache, or re-acquisition churns it, so every tree walk pays disk latency | +| _Acquire Source (local vs network)_ | `local` dominant on a warm node | sustained `network` on a range the node should hold | the local store is not retaining data | +| paired with _NuDB Cache Hit Ratio_ (Ledger Data Sync board) | both healthy | low on both | disk-bound sync | +| | | low here, NuDB healthy | cache pressure alone — this is the pairing that explains the whole symptom | + +**Conclusion:** the tree-node cache sits one layer **above** the node store, so a +miss here is what produces a node-store read there; reading the two together is +what tells cache pressure from a disk bottleneck. Two limits: these are **means, +not percentiles**, and `write_mean_us` is currently emitted only for the +`[import_db]` admin import path — on an ordinary node `write_count` climbs with +no `write_mean_us` line, which is a known instrumentation gap, not a healthy +zero. Detail: [Sync pipeline](#sync-pipeline--ordered-diagnosis) steps 9 and 14. + +#### Branch F — terminal: the node will stop validating for good + +Check this branch on **every** slow sync, before the mode machine, because it is +the only one with a deadline. + +| Look at | Healthy | Unhealthy | Conclude | +| ------------------------------ | ------------------------------------------------------------------------------------------------------ | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| _Amendment Block Countdown_ | **-1** — an explicit sentinel meaning nothing is pending. Not a negative duration and not missing data | any non-negative value | a countdown to a **terminal** state: at expiry the node becomes amendment-blocked and will never validate again without a software upgrade. Clamped at 0, so 0 means due or past due. Nothing else on this dashboard matters — plan the upgrade inside the window | +| _Amendment Warned_ | 0 | 1 | the same condition as a flag: an unsupported amendment reached majority | +| _Byzantine Ledger Jumps_ | flat at zero | a single jump during a fresh sync | benign — the node is settling onto the network's chain | +| | | repeated jumps | wrong-chain thrash: the node keeps switching chains and never settles. Check the peer set (branch A) and the configured network id. Nothing in the acquire pipeline can fix it | +| _Ledger/Object Serve Refusals_ | near zero | `sendq_full`/`load_shed` climbing | self-inflicted: this node is too loaded to answer. It does not explain **this** node's sync — it explains its peers', and is the serving-side symptom of the same overload branches C-E cover | +| | | `not_found` climbing | a genuine history gap — a retention and configuration question, not a load one | + +**Conclusion:** the countdown is the only actionable amendment signal. The +existing `validator_health{metric="amendment_blocked"}` on the Validator Health +dashboard reports the block after it has happened, when nothing can be done. The +blocking amendment's hash is deliberately not a label (unbounded cardinality) — +get it from the `AmendmentTableImpl::doValidatedLedger` log line via Loki, +correlated by node and time. Detail: +[Sync pipeline](#sync-pipeline--ordered-diagnosis) step 13. + +--- Signal definitions: [telemetry-glossary.md](./telemetry-glossary.md) "Fresh-node sync diagnostics". @@ -2065,8 +2240,14 @@ Signal index (instrument, emit site, panel): [09-data-collection-reference.md](../OpenTelemetryPlan/09-data-collection-reference.md) "Fresh-node sync diagnostics". -Note: the sync signals are native metrics, which are never sampled — unlike the -span-derived (spanmetrics) series, they are always complete. +Note: rows 1-8 are native metrics, which are never sampled — unlike the +span-derived (spanmetrics) series in row 9, they are always complete. An absent +series in row 9 can mean "tracing not enabled" (`trace_ledger` / `trace_peer`) +rather than "not happening", so use row 9 to **localise** a fault the native +rows have already established, not to detect one. + +The two sections below are the ordered detail the branches point into. They walk +every panel in sequence; the branches above are the fast path to the right step. #### Bootstrap (Domain 0) — ordered diagnosis @@ -2155,12 +2336,14 @@ first one that is wrong and fix it before reading further panels. step 3. If all five steps are clean the bootstrap stage is healthy, and the problem is -in the **Sync pipeline** row instead. +in the sync pipeline — rows 2 to 9 — instead. #### Sync pipeline — ordered diagnosis -Once bootstrap is clean, work the Sync pipeline row in this order. As above, -each step gates the next: stop at the first one that is wrong. +Once bootstrap is clean, work rows 2 to 9 in this order. As above, each step +gates the next: stop at the first one that is wrong. The step order is the +diagnosis order, which crosses rows deliberately — a step names the row and +panel it reads. 1. **Did the node ever sync at all?** Panel _Time to First FULL_ (`sync_state`, `metric=initial_full_duration_us`, @@ -2227,7 +2410,8 @@ each step gates the next: stop at the first one that is wrong. the trend: pinned-and-falling is a large but progressing tree. - **Zero on one tree, non-zero on the other** — that tree is already complete; concentrate on the one still reporting nodes. - One caveat: zero on both is only healthy if _In-Flight Acquires_ (step 8) + One caveat: zero on both is only healthy if the `in_flight` series on + _Received-Data Stash Depth & In-Flight Acquires_ (step 8) is non-zero. Zero everywhere with zero acquires in flight is an idle node, which says nothing about acquire health. @@ -2443,10 +2627,11 @@ each step gates the next: stop at the first one that is wrong. This is the step for the specific symptom **"a node with a large existing database starts and syncs slower than a fresh one"**. Back-fill is write-bound, so no read-side panel can show it; check this step whenever a - node with existing history is the slow one. + node with existing history is the slow one. Both panels live in the + collapsed **Back-fill & persistence** row — expand it. Panel _NodeStore Write vs Read Latency (us/op)_ (`nodestore_latency`, - `metric=write_mean_us` and `read_mean_us`) with _NodeStore Operation Rate_ - (`metric=write_count` / `read_count`) beside it: + `metric=write_mean_us` and `read_mean_us`) with _NodeStore Operation Rate + (writes vs reads)_ (`metric=write_count` / `read_count`) beside it: - **Write line rising during history back-fill** — the backend cannot absorb writes fast enough. Sync will stay slow however many peers are available, so adding peers will not help. Check storage IOPS, the @@ -2479,7 +2664,8 @@ each step gates the next: stop at the first one that is wrong. 15. **Is replay-based back-fill silently falling back to the slow path?** Only relevant when `[ledger_replay]` is enabled. Panels _Replay Fallback to Full Acquire (by stage)_ (`ledger_replay_fallback_total`) and _Replay - Outcomes (by terminal state)_ (`ledger_replay_outcome_total`): + Outcomes (by terminal state)_ (`ledger_replay_outcome_total`), also in the + collapsed **Back-fill & persistence** row: - **Any sustained fallback rate** — too few connected peers support the `LedgerReplay` protocol feature, so every historical ledger is fetched whole instead of as a delta. Back-fill still completes, just far slower, @@ -2610,7 +2796,7 @@ each step gates the next: stop at the first one that is wrong. on?** Every step above reads a native metric, which is an aggregate: it says how much and how often, never _which one_. This step reads the **span-derived** - panels at the bottom of the Sync pipeline row, which answer the "which" + panels in the collapsed **Spans & traces** row, which answer the "which" questions the aggregates structurally cannot — and each point on them is backed by a trace, so it can be clicked through to the individual fetch, request or dial. @@ -2627,9 +2813,9 @@ each step gates the next: stop at the first one that is wrong. duration is the account-state tree's and hides the other two: - **`astree` band hot, with `timed_out=true` and a non-zero `missing_nodes`** — peers are not supplying account-state nodes. This - is the common stuck-fresh-sync shape. Check _Outbound Dial Outcomes_ - below for `refused` / `timeout` spikes, and step 11 for whether any - peer holds the range at all. + is the common stuck-fresh-sync shape. Check _Outbound Dial Outcomes + (span-derived, per attempt)_ below for `tcp_fail` / `timeout` spikes, + and step 11 for whether any peer holds the range at all. - **`header` band hot** — the node is waiting to be **told what to fetch**. The header names both trees' root hashes, so until it arrives nothing else can even be requested. This is a peer-supply fault diff --git a/src/libxrpl/nodestore/DatabaseNodeImp.cpp b/src/libxrpl/nodestore/DatabaseNodeImp.cpp index 9323d69131..6dff8ce8d3 100644 --- a/src/libxrpl/nodestore/DatabaseNodeImp.cpp +++ b/src/libxrpl/nodestore/DatabaseNodeImp.cpp @@ -9,6 +9,7 @@ #include #include +#include #include #include #include @@ -23,7 +24,12 @@ DatabaseNodeImp::store(NodeObjectType type, Blob&& data, uint256 const& hash, st storeStats(1, data.size()); auto obj = NodeObject::createObject(type, std::move(data), hash); + // Time only the backend write, which is the disk work. One clock pair per + // stored object, accumulated into an atomic that the metrics gauge reads on + // its own schedule, so nothing is added to the read path or per tree node. + auto const begin = std::chrono::steady_clock::now(); backend_->store(obj); + recordStoreDuration(std::chrono::steady_clock::now() - begin); if (cache_) { // After the store, replace a negative cache entry if there is one diff --git a/src/libxrpl/nodestore/DatabaseRotatingImp.cpp b/src/libxrpl/nodestore/DatabaseRotatingImp.cpp index 23a48a3bf3..d5a839eb56 100644 --- a/src/libxrpl/nodestore/DatabaseRotatingImp.cpp +++ b/src/libxrpl/nodestore/DatabaseRotatingImp.cpp @@ -14,6 +14,7 @@ #include #include +#include #include #include #include @@ -128,7 +129,12 @@ DatabaseRotatingImp::store(NodeObjectType type, Blob&& data, uint256 const& hash return writableBackend_; }(); + // Time only the backend write, which is the disk work. One clock pair per + // stored object, accumulated into an atomic that the metrics gauge reads on + // its own schedule, so nothing is added to the read path or per tree node. + auto const begin = std::chrono::steady_clock::now(); backend->store(nObj); + recordStoreDuration(std::chrono::steady_clock::now() - begin); storeStats(1, nObj->getData().size()); }