ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-07-29 10:00:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation

Browse Source 
# Conflicts:
#	docker/telemetry/grafana/dashboards/rpc-performance.json
#	docker/telemetry/grafana/dashboards/xrpld-rpc-perf-otel.json
This commit is contained in:
Pratik Mankawde
2026-07-03 22:31:15 +01:00
parent 6f47a7ab77 94791ef924
commit 628726b51b
16 changed files with 719 additions and 797 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
OpenTelemetryPlan/09-data-collection-reference.md
View File

@@ -527,7 +527,7 @@ prefix=xrpld
| `xrpld_Overlay_Peer_Disconnects_Charges` | OverlayImpl.cpp | Disconnects due to resource limit charges | Low growth (subset of above) |
| `xrpld_jobq_job_count` | JobQueue.cpp | Current job queue depth (group `jobq`) | 0–100 (healthy) |
**Grafana dashboard**: _Node Health (System Metrics)_ (`xrpld-system-node-health`)
**Grafana dashboard**: _Node Health_ (`xrpld-system-node-health`)
### 2.2 Counters
@@ -541,7 +541,7 @@ prefix=xrpld
**Note**: With `server=otel`, `xrpld_warn` and `xrpld_drop` are properly exported as OTel Counter instruments. The previous StatsD `|m` type limitation no longer applies.
**Grafana dashboard**: _RPC & Pathfinding (System Metrics)_ (`xrpld-system-rpc`)
**Grafana dashboard**: _RPC & Pathfinding_ (`xrpld-system-rpc`)
### 2.3 Histograms (Event timers)
@@ -1211,13 +1211,13 @@ Lifetime validation agreement/miss tallies are exported as monotonic **Observabl
### New Grafana Dashboards (Phase 9)
| Dashboard | UID | Data Source | Key Panels |
| ---------------------- | ------------------------ | ----------- | --------------------------------------------------------- |
| Fee Market & TxQ | `xrpld-fee-market` | Prometheus | TxQ depth/capacity, fee levels, load factor breakdown |
| Job Queue Analysis | `xrpld-job-queue` | Prometheus | Per-job rates, queue wait times, execution times |
| RPC Performance (OTel) | `xrpld-rpc-perf` | Prometheus | Per-method call rates, error rates, latency distributions |
| Validator Health | `xrpld-validator-health` | Prometheus | Agreement %, validation rate, amendment/UNL, state |
| Peer Quality | `xrpld-peer-quality` | Prometheus | P90 latency, insane peers, version awareness, disconnects |
| Dashboard | UID | Data Source | Key Panels |
| ------------------------------------ | ------------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------ |
| Fee Market & TxQ | `xrpld-fee-market` | Prometheus | TxQ depth/capacity, fee levels, load factor breakdown |
| Job Queue Analysis | `xrpld-job-queue` | Prometheus | Per-job rates, queue wait times, execution times |
| RPC Performance (per-method section) | `xrpld-rpc-perf` | Prometheus | Per-method call rates, error rates, latency distributions (added as a section to the existing RPC Performance dashboard) |
| Validator Health | `xrpld-validator-health` | Prometheus | Agreement %, validation rate, amendment/UNL, state |
| Peer Quality | `xrpld-peer-quality` | Prometheus | P90 latency, insane peers, version awareness, disconnects |
### Updated Grafana Dashboards (Phase 9)

49
docker/telemetry/grafana/dashboards/consensus-health.json
View File

@@ -10,7 +10,7 @@
"panels": [
{
"title": "Validation Send Rate",
"description": "Rate at which this node sends ledger validations to the network. Sourced from the consensus.validation.send span (RCLConsensus.cpp). Each validation confirms the node has fully validated a ledger. The span carries ledger_seq and proposing. Should closely track the ledger close rate when the node is healthy.",
"description": "**What:** How often this node issues a ledger validation to the network. Each validation asserts the node fully validated one ledger.\n**How it's computed:** Per-second rate of validation events sent, summed per node over a 5-minute window.\n**Reading it:** A flat line at the network's ledger cadence is normal; higher is not better and lower means the node is validating less often.\n**Healthy range:** Roughly one validation per closed ledger (about 0.2-0.3/s on a healthy chain).\n**Watch for:** A drop to zero on a validator means it stopped validating; a value well below the close rate means validation is lagging.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::createValidationSpan",
"type": "stat",
"gridPos": {
"h": 8,
@@ -43,7 +43,7 @@
},
{
"title": "Consensus Mode Over Time",
"description": "Breakdown of consensus ledger close events by the node's consensus mode (Proposing, Observing, Wrong Ledger, Switched Ledger). Grouped by the consensus_mode span attribute from consensus.ledger_close. A healthy validator should be predominantly in Proposing mode. Frequent Wrong Ledger or Switched Ledger indicates sync issues.",
"description": "**What:** Ledger-close activity split by the node's consensus mode (Proposing, Observing, Wrong Ledger, Switched Ledger).\n**How it's computed:** Per-second rate of ledger-close events grouped by consensus mode, per node, over a 5-minute window.\n**Reading it:** One dominant band is normal; a validator should sit almost entirely in Proposing, a stock node in Observing.\n**Healthy range:** Nearly all activity in a single expected mode.\n**Watch for:** Sustained time in Wrong Ledger or Switched Ledger indicates the node is out of sync or flapping between chains.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::onClose",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -83,7 +83,7 @@
},
{
"title": "Consensus Proposals Sent Rate",
"description": "Rate at which this node sends consensus proposals to the network. Sourced from the consensus.proposal.send span (RCLConsensus.cpp) which fires each time the node proposes a transaction set. The span carries xrpl.consensus.round identifying the consensus round number. A healthy proposing node should show steady proposal output.",
"description": "**What:** How often this node proposes its candidate transaction set to peers during consensus.\n**How it's computed:** Per-second rate of proposal-send events, summed per node over a 5-minute window.\n**Reading it:** A steady line tracking the ledger cadence is healthy; only proposing (validator) nodes show activity.\n**Healthy range:** Steady output in step with the close rate for a proposing node; zero for a non-proposing node.\n**Watch for:** A proposing validator dropping to zero, or erratic spikes suggesting repeated re-proposals within rounds.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::propose",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -123,7 +123,7 @@
},
{
"title": "Accept vs Close Rate",
"description": "Compares the rate of consensus.accept (ledger accepted after consensus) vs consensus.ledger_close (ledger close initiated). These should track closely in a healthy network. A divergence means some close events are not completing the accept phase, potentially indicating consensus failures or timeouts.",
"description": "**What:** Two rates side by side: ledgers accepted after consensus versus ledger closes initiated.\n**How it's computed:** Per-second rate of accept events and of close events, each summed per node over a 5-minute window.\n**Reading it:** The two lines should overlap; read any gap between them as closes that did not finish accepting.\n**Healthy range:** Both lines equal and tracking the network close cadence.\n**Watch for:** A persistent gap where closes exceed accepts points to consensus rounds failing or timing out before acceptance.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::makeAcceptSpan / RCLConsensus::Adaptor::onClose",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -170,7 +170,7 @@
},
{
"title": "Validation vs Close Rate",
"description": "Compares the rate of consensus.validation.send vs consensus.ledger_close. Each validated ledger should produce one validation message. If validations lag behind closes, the node may be falling behind on validation or experiencing issues with the validation pipeline.",
"description": "**What:** Two rates side by side: validations sent versus ledger closes, so you can see whether every closed ledger gets validated.\n**How it's computed:** Per-second rate of validation-send events and of close events, each summed per node over a 5-minute window.\n**Reading it:** The lines should overlap; a validation line below the close line means validation is falling behind.\n**Healthy range:** Both lines equal at the network close cadence.\n**Watch for:** Validations persistently trailing closes, which means the node validates fewer ledgers than it closes.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::createValidationSpan / RCLConsensus::Adaptor::onClose",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -217,7 +217,7 @@
},
{
"title": "Time to Reach Consensus (p50/p95)",
"description": "Wall-clock time for the network to agree a ledger, from the round_time_ms attribute on consensus.accept. This is the real consensus latency, not span overhead.",
"description": "**What:** Wall-clock time for the network to agree a ledger, at the median and 95th percentile.\n**How it's computed:** The recorded per-round agreement time is aggregated to its 50th and 95th percentile over the window.\n**Reading it:** Lower is better; the p95 line shows the worst typical rounds while p50 shows the common case.\n**Healthy range:** Roughly 2-4 seconds on mainnet; workload- and network-dependent.\n**Watch for:** Rising percentiles, or a p95 that pulls far above p50, signal slow or contentious rounds under load or poor peer connectivity.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::makeAcceptSpan",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -270,7 +270,7 @@
},
{
"title": "Average Time to Reach Consensus",
"description": "Mean time to agree a ledger (round_time_ms on consensus.accept), averaged over the window. Tracks the typical consensus latency trend.",
"description": "**What:** Mean wall-clock time for the network to agree a ledger.\n**How it's computed:** The recorded per-round agreement time averaged over the window.\n**Reading it:** Lower is better; watch the trend rather than any single point.\n**Healthy range:** Roughly 2-4 seconds on mainnet; workload-dependent.\n**Watch for:** A steady upward drift indicates the network is taking longer to converge, often from load or connectivity problems.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::makeAcceptSpan",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -313,7 +313,7 @@
},
{
"title": "Consensus Rounds per Ledger (Establish Count)",
"description": "Distribution of ledgers by the number of establish-phase rounds (establish_count) needed to converge. Each series is an integer round count: 1 = agreed on the first proposal, 2/3 = needed more rounds. Watch for a shift toward higher counts under disagreement.",
"description": "**What:** How many establish-phase iterations each ledger needed before validators converged.\n**How it's computed:** Ledgers are counted over the window and grouped by their iteration count; each series is one count value.\n**Reading it:** Most weight on the '1' series is ideal; weight shifting to 2, 3 or more shows harder convergence.\n**Healthy range:** The large majority of ledgers converging in one iteration.\n**Watch for:** A growing share of ledgers needing several iterations, indicating disagreement or network stress.\n**Source:** src/xrpld/consensus/Consensus.h:Consensus::startEstablishTracing",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -363,7 +363,7 @@
},
{
"title": "Previous Round Time per Ledger",
"description": "Duration of the previous consensus round (previous_round_time_ms on consensus.round). Complements time-to-consensus with the prior round's cost.",
"description": "**What:** Duration of the immediately preceding consensus round, at the 95th percentile.\n**How it's computed:** The prior round's recorded duration is aggregated to its 95th percentile over the window.\n**Reading it:** Lower is better; complements the time-to-consensus panels with the previous round's cost.\n**Healthy range:** Comparable to the current time-to-consensus, roughly a few seconds.\n**Watch for:** A rising p95 means recent rounds have been expensive, often preceding a slowdown in ledger cadence.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::startRoundTracing",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -406,7 +406,7 @@
},
{
"title": "Position Update Duration",
"description": "p95 duration of the consensus.update_positions span, which tallies disputes and updates this node's position each round. Long durations indicate heavy dispute resolution or slow convergence on close time.",
"description": "**What:** Time this node spends each round tallying disputes and updating its consensus position, at the 95th percentile.\n**How it's computed:** Per-round position-update durations are aggregated to their 95th percentile over a 5-minute window.\n**Reading it:** Lower is better; short, flat durations are normal.\n**Healthy range:** A few milliseconds under normal load.\n**Watch for:** Sustained high durations point to heavy dispute resolution or slow convergence on close time.\n**Source:** src/xrpld/consensus/Consensus.h:Consensus::updateOurPositions",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -446,7 +446,7 @@
},
{
"title": "Ledger Close Duration",
"description": "p95 duration of the ledger close event. The consensus.ledger_close span (RCLConsensus.cpp) measures the time from when consensus triggers a ledger close to completion. Carries ledger_seq and consensus_mode attributes. Compare with Consensus Round Duration to understand how close timing relates to overall round time.",
"description": "**What:** Time from when consensus triggers a ledger close to when the close completes, at the 95th percentile.\n**How it's computed:** Per-close durations are aggregated to their 95th percentile per node over a 5-minute window.\n**Reading it:** Lower is better; compare against the round time to see how much of it the close accounts for.\n**Healthy range:** A few to tens of milliseconds; workload-dependent.\n**Watch for:** A rising p95 indicates the close step is becoming a bottleneck, often under heavy transaction volume.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::onClose",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -486,7 +486,7 @@
},
{
"title": "Ledger Apply Duration (doAccept)",
"description": "Time spent applying the consensus result to build a new ledger. Measured by the consensus.accept.apply span in doAccept().",
"description": "**What:** Time spent applying the agreed transaction set to build the new ledger, at the 95th percentile.\n**How it's computed:** Per-apply durations are aggregated to their 95th percentile per node over a 5-minute window.\n**Reading it:** Lower is better; this is the cost of executing the ledger's transactions.\n**Healthy range:** A few to tens of milliseconds, scaling with the number of transactions in the ledger.\n**Watch for:** A rising p95 alongside high transaction counts signals apply-stage load, whether organic or from a flood of transactions.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::doAccept",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -520,7 +520,7 @@
},
{
"title": "Consensus Accept Duration Heatmap",
"description": "Heatmap showing the distribution of consensus.accept span durations across histogram buckets over time. Each cell represents how many accept events fell into that duration bucket in a 5m window. Useful for detecting outlier consensus rounds that take abnormally long.",
"description": "**What:** Distribution of how long the accept step takes across rounds, over time.\n**How it's computed:** Accept durations are bucketed by duration band and counted per 5-minute window, drawn as a heatmap.\n**Reading it:** A tight low band is healthy; brighter cells higher up mean more slow rounds.\n**Healthy range:** Most rounds concentrated in the lowest duration bands.\n**Watch for:** A widening or upward-drifting hot band flags outlier rounds that take abnormally long.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::makeAcceptSpan",
"type": "heatmap",
"gridPos": {
"h": 8,
@@ -557,7 +557,7 @@
},
{
"title": "Close-Time Agreement Rate (Agreed vs Disagreed)",
"description": "Rate of consensus rounds where validators agreed on the close time (close_time_correct=true) versus agreed to disagree (false). 'Disagreed' is normal when validators' clocks differ slightly; a sustained rise indicates clock drift or network latency across the validator set.",
"description": "**What:** How often validators agreed on the ledger close time versus agreed to disagree.\n**How it's computed:** Per-second rate of rounds split into Agreed and Disagreed, per node, over the rate interval.\n**Reading it:** Mostly Agreed is healthy; occasional Disagreed is normal when validator clocks differ slightly.\n**Healthy range:** Overwhelmingly Agreed.\n**Watch for:** A sustained rise in Disagreed points to clock drift or latency spread across the validator set.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::doAccept",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -607,7 +607,7 @@
},
{
"title": "Close Time Vote Bins & Resolution",
"description": "Number of distinct close time vote bins (close_time_vote_bins) and the bin size / resolution in ms (close_resolution_ms). More bins = more clock disagreement. Resolution adapts: finer (10s) when validators agree, coarser (120s) when they disagree.",
"description": "**What:** Two related values: how many distinct close-time votes were seen, and the current close-time rounding granularity in ms.\n**How it's computed:** Both values are averaged over the window and plotted on dual axes.\n**Reading it:** Fewer vote bins and a fine resolution mean tight agreement; more bins and a coarse resolution mean disagreement.\n**Healthy range:** Few bins with a fine (about 10s) resolution.\n**Watch for:** Rising bins with the resolution widening (toward 120s) shows validators struggling to agree on close time.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::doAccept",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -698,7 +698,7 @@
},
{
"title": "Close-Time Resolution Change (per Round)",
"description": "How the close-time resolution (the rounding granularity used to bin close-time votes) changed versus the previous round: Coarser = increased (more disagreement, wider bins), Finer = decreased (better agreement), Steady = unchanged. Mostly Steady is healthy; frequent Coarser shifts indicate the network is struggling to agree on close time.",
"description": "**What:** Whether the close-time rounding granularity moved Coarser, Finer, or stayed Steady versus the previous round.\n**How it's computed:** Rounds are counted over the window and grouped by direction of change.\n**Reading it:** Mostly Steady is healthy; Coarser means widening bins (more disagreement), Finer means tightening.\n**Healthy range:** Predominantly Steady.\n**Watch for:** Frequent Coarser shifts indicate the network is repeatedly failing to agree on close time.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::doAccept",
"type": "bargauge",
"gridPos": {
"h": 8,
@@ -769,7 +769,7 @@
},
{
"title": "Close-Time Proposal Spread (Distinct Positions per Round)",
"description": "How spread out the validators' proposed close times were. close_time_vote_bins is the number of DISTINCT close-time values proposed by peers in a round (rawCloseTimes.peers.size()): 1 = everyone proposed the same time (tight agreement), 2-3 = proposals split across several times (clock drift or latency). Each bar series is a distinct-position count; the value is how many rounds in the window had that many distinct positions. Lower is better; a shift toward 2+ signals growing disagreement on close time.",
"description": "**What:** How spread out validators' proposed close times were, as the number of distinct positions per round.\n**How it's computed:** Rounds are counted over the window and grouped by their distinct-position count; each bar is one count value.\n**Reading it:** Weight on '1 distinct position' means everyone agreed; weight on 2 or more means the proposals split.\n**Healthy range:** Most rounds at a single distinct position.\n**Watch for:** A shift toward two or more distinct positions signals growing clock drift or latency across validators.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::doAccept",
"type": "bargauge",
"gridPos": {
"h": 8,
@@ -817,7 +817,7 @@
},
{
"title": "Consensus Outcome Distribution (per Round)",
"description": "How consensus rounds concluded, from consensus_state on consensus.accept. Agreed = normal success (all validators agreed); Moved On = closed the ledger without full agreement (a minority position was abandoned); Expired = the round timed out; No = consensus not reached. A healthy network is almost entirely Agreed; a rising Moved On / Expired share signals network stress or disagreement.",
"description": "**What:** How consensus rounds concluded: Agreed, Moved On (partial), Expired (timeout), or No Consensus.\n**How it's computed:** Rounds over the window are counted and grouped by outcome, shown as shares of a pie.\n**Reading it:** An almost fully Agreed pie is healthy; other slices represent degraded outcomes.\n**Healthy range:** Nearly 100 percent Agreed.\n**Watch for:** A growing Moved On or Expired share signals network stress, disagreement, or connectivity loss.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::makeAcceptSpan",
"type": "piechart",
"gridPos": {
"h": 8,
@@ -884,7 +884,7 @@
},
{
"title": "Consensus Failures Over Time",
"description": "Rate of non-normal consensus outcomes (moved_on + expired). Spikes indicate consensus instability.",
"description": "**What:** Rate of non-normal consensus outcomes (moved-on plus expired) over time.\n**How it's computed:** Per-second rate of the moved-on and expired outcomes, summed per node over a 5-minute window.\n**Reading it:** Flat at zero is healthy; any sustained line is a concern.\n**Healthy range:** Zero under normal operation.\n**Watch for:** Spikes or a persistent nonzero rate indicate consensus instability that can precede ledger stalls or forks.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::makeAcceptSpan",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -931,7 +931,7 @@
},
{
"title": "Consensus Stall Rate",
"description": "Rate of consensus.check spans reporting consensus_stalled=true, broken down by stall flag. A non-zero stalled rate surfaces stall conditions before they manifest as validated-ledger-age alarms. Requires the consensus_stalled spanmetrics dimension.",
"description": "**What:** Rate at which consensus health checks report a stalled condition versus not stalled.\n**How it's computed:** Per-second rate of consensus checks split by stalled flag, summed per node over a 5-minute window.\n**Reading it:** The Not Stalled line should carry all the weight; any Stalled line is an early warning.\n**Healthy range:** Stalled at zero.\n**Watch for:** A nonzero stalled rate surfaces stalls before they show up as validated-ledger-age alarms.\n**Source:** src/xrpld/consensus/Consensus.h:Consensus::haveConsensus",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -978,7 +978,7 @@
},
{
"title": "Consensus Mode-Change Rate by Target Mode",
"description": "Rate of consensus.mode_change spans broken down by the mode the node switched INTO (mode_new). Frequent switches into Wrong Ledger or Switched Ledger indicate an unstable node at fork risk. Requires the mode_new spanmetrics dimension.",
"description": "**What:** How often the node switches consensus mode, broken down by the mode it switched into.\n**How it's computed:** Per-second rate of mode-change events grouped by target mode, per node, over a 5-minute window.\n**Reading it:** Near-zero is healthy; a stable node rarely changes mode.\n**Healthy range:** Essentially zero mode changes in steady state.\n**Watch for:** Frequent switches into Wrong Ledger or Switched Ledger mark an unstable node at risk of forking.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::onModeChange",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -1018,7 +1018,7 @@
},
{
"title": "Ledger History Mismatch Rate by Reason",
"description": "Rate of built-vs-validated ledger mismatches broken down by reason (prior_ledger, close_time, consensus_txset, same_txset_diff_result, different_txset, unknown). Answers WHY the node forked \u2014 Byzantine close-time disagreement vs sync drift vs tx-processing difference.",
"description": "**What:** Rate of built-versus-validated ledger mismatches, broken down by why they diverged.\n**How it's computed:** Per-second rate of mismatch events grouped by reason, per node, over a 5-minute window.\n**Reading it:** Flat at zero is healthy; the reason label tells you the nature of any divergence.\n**Healthy range:** Zero under normal operation.\n**Watch for:** Any sustained rate marks a fork; the reason distinguishes close-time disagreement, sync drift, and transaction-processing differences.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:incrementLedgerHistoryMismatch",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -1150,7 +1150,7 @@
{
"name": "node",
"label": "Node",
"description": "Filter by rippled node (service.instance.id \u2014 e.g. Node-1)",
"description": "Filter by rippled node (service.instance.id — e.g. Node-1)",
"type": "query",
"query": "label_values(traces_span_metrics_calls_total, exported_instance)",
"datasource": {
@@ -1259,5 +1259,6 @@
"to": "now"
},
"title": "Consensus Health",
"uid": "xrpld-consensus"
"uid": "xrpld-consensus",
"description": "Consensus health for XRPL nodes: how reliably and quickly the network agrees each ledger, and where agreement breaks down."
}

21
docker/telemetry/grafana/dashboards/ledger-operations.json
View File

@@ -10,7 +10,7 @@
"panels": [
{
"title": "Ledger Build Rate",
"description": "Rate at which new ledgers are being built. The ledger.build span (BuildLedger.cpp) wraps the entire buildLedgerImpl() function which creates a new ledger from a parent, applies transactions, flushes SHAMap nodes, and sets the accepted state. Should match the consensus close rate (~0.25/sec on mainnet with ~4s rounds).",
"description": "**What:** How many new ledgers this node finishes building per second.\n**How it's computed:** Per-second rate of completed ledger-build operations, averaged over 5 minutes and split by node.\n**Reading it:** A steady flat line; the value should track the network close cadence.\n**Healthy range:** About 0.2-0.3 ledgers/sec on mainnet (roughly one every 3-5s); workload-dependent on test networks.\n**Watch for:** A drop toward zero (node fell out of sync or stalled) or a value well above the network rate (rebuilding history).\n**Source:** src/xrpld/app/ledger/detail/BuildLedger.cpp:buildLedgerImpl",
"type": "stat",
"gridPos": {
"h": 8,
@@ -42,7 +42,7 @@
},
{
"title": "Ledger Build Duration",
"description": "p95 duration of ledger builds. Measures the full buildLedgerImpl() call including transaction application, SHAMap flushing, and ledger acceptance. The span records xrpl.ledger.seq as an attribute. Long build times indicate expensive transaction sets or I/O pressure from SHAMap flushes.",
"description": "**What:** The time taken to build a single ledger, at the 95th percentile.\n**How it's computed:** 95th-percentile of ledger-build durations over a 5-minute window, per node.\n**Reading it:** Lower is better; the line should stay well under the ledger interval.\n**Healthy range:** Typically tens to low hundreds of milliseconds; workload-dependent.\n**Watch for:** Sustained rises approaching the close interval, which indicate heavy transaction sets or disk/I/O pressure.\n**Source:** src/xrpld/app/ledger/detail/BuildLedger.cpp:buildLedgerImpl",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -81,7 +81,7 @@
},
{
"title": "Ledger Validation Rate",
"description": "Rate at which ledgers pass the validation threshold and are accepted as fully validated. The ledger.validate span (LedgerMaster.cpp) fires in checkAccept() only after the ledger receives sufficient trusted validations (>= quorum). Records xrpl.ledger.seq and validations (the number of validations received).",
"description": "**What:** How often ledgers reach full validation (accepted by the trusted validator quorum) per second.\n**How it's computed:** Per-second rate of ledger-validation events over 5 minutes, per node.\n**Reading it:** Should closely match the build rate under normal, in-sync operation.\n**Healthy range:** About 0.2-0.3/sec on mainnet; workload-dependent elsewhere.\n**Watch for:** A validation rate that lags the build rate, signalling the node is building ahead of the network consensus it trusts.\n**Source:** src/xrpld/app/ledger/detail/LedgerMaster.cpp:LedgerMaster::checkAccept",
"type": "stat",
"gridPos": {
"h": 8,
@@ -113,7 +113,7 @@
},
{
"title": "Ledger Build Duration Heatmap",
"description": "Heatmap showing the distribution of ledger.build durations across histogram buckets over time. Each cell represents the count of ledger builds that fell into that duration bucket in a 5m window. Useful for spotting occasional slow ledger builds that may not appear in percentile charts.",
"description": "**What:** The full distribution of ledger-build times over the window, not just a single percentile.\n**How it's computed:** Counts of ledger builds falling in each duration band per 5-minute window, shown as color density.\n**Reading it:** A tight low band is healthy; scattered high cells mean occasional slow builds.\n**Healthy range:** Most mass concentrated in the low-millisecond bands; workload-dependent.\n**Watch for:** A second cluster of hot cells at high durations (bimodal build times) hidden by percentile charts.\n**Source:** src/xrpld/app/ledger/detail/BuildLedger.cpp:buildLedgerImpl",
"type": "heatmap",
"gridPos": {
"h": 8,
@@ -149,7 +149,7 @@
},
{
"title": "Transaction Apply Duration",
"description": "p95 duration of applying the consensus transaction set during ledger building. The tx.apply span (BuildLedger.cpp) wraps applyTransactions() which iterates through the CanonicalTXSet with multiple retry passes. Records tx_count (successful) and tx_failed (failed) as attributes.",
"description": "**What:** The time spent applying the agreed transaction set into the new ledger, at the 95th percentile.\n**How it's computed:** 95th-percentile of transaction-apply durations over 5 minutes, per node.\n**Reading it:** Lower is better; this is a large share of total build time.\n**Healthy range:** A few to tens of milliseconds; scales with transaction volume per ledger.\n**Watch for:** Spikes during large or expensive transaction sets, which push out overall ledger build time.\n**Source:** src/xrpld/app/ledger/detail/BuildLedger.cpp:applyTransactions",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -188,7 +188,7 @@
},
{
"title": "Transaction Apply Rate",
"description": "Rate of tx.apply span invocations, reflecting how frequently the transaction application phase runs during ledger building. Each ledger build triggers one tx.apply call. Should closely match the ledger build rate.",
"description": "**What:** How often the transaction-apply phase runs per second (once per ledger build).\n**How it's computed:** Per-second rate of transaction-apply operations over 5 minutes, per node.\n**Reading it:** Should track the ledger build rate almost exactly.\n**Healthy range:** About 0.2-0.3/sec on mainnet; workload-dependent.\n**Watch for:** Divergence from the build rate, which would indicate a metric or pipeline anomaly.\n**Source:** src/xrpld/app/ledger/detail/BuildLedger.cpp:applyTransactions",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -227,7 +227,7 @@
},
{
"title": "Ledger Store Rate",
"description": "Rate at which ledgers are stored into the ledger history. The ledger.store span (LedgerMaster.cpp) wraps storeLedger() which inserts the ledger into the LedgerHistory cache. Records xrpl.ledger.seq. Should match the ledger build rate under normal operation.",
"description": "**What:** How often completed ledgers are written into ledger history per second.\n**How it's computed:** Per-second rate of ledger-store operations over 5 minutes, per node.\n**Reading it:** Should match the build rate during normal operation.\n**Healthy range:** About 0.2-0.3/sec on mainnet; can burst higher while backfilling history.\n**Watch for:** A store rate below the build rate (storage falling behind) or a stall at zero.\n**Source:** src/xrpld/app/ledger/detail/LedgerMaster.cpp:LedgerMaster::storeLedger",
"type": "stat",
"gridPos": {
"h": 8,
@@ -259,7 +259,7 @@
},
{
"title": "Build vs Close Duration",
"description": "Compares p95 durations of ledger.build (the actual ledger construction in BuildLedger.cpp) vs consensus.ledger_close (the consensus close event in RCLConsensus.cpp). Build time is a subset of close time. A large gap between them indicates overhead in the consensus pipeline outside of ledger construction itself.",
"description": "**What:** Ledger build time compared with the total consensus ledger-close time, both at the 95th percentile.\n**How it's computed:** Two 95th-percentile duration series over 5 minutes: ledger construction and the full consensus close, per node.\n**Reading it:** Build should sit below close; the gap is consensus overhead outside construction.\n**Healthy range:** Close a bit above build; both workload-dependent and under the round interval.\n**Watch for:** A widening gap (consensus-pipeline overhead growing) or build time approaching close time.\n**Source:** src/xrpld/app/ledger/detail/BuildLedger.cpp:buildLedgerImpl; src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::onClose",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -384,7 +384,7 @@
{
"name": "node",
"label": "Node",
"description": "Filter by rippled node (service.instance.id \u2014 e.g. Node-1)",
"description": "Filter by rippled node (service.instance.id — e.g. Node-1)",
"type": "query",
"query": "label_values(traces_span_metrics_calls_total, exported_instance)",
"datasource": {
@@ -409,5 +409,6 @@
},
"title": "Ledger Operations",
"uid": "xrpld-ledger-ops",
"refresh": "5s"
"refresh": "5s",
"description": "Ledger construction, validation, and storage activity and timing for this node."
}

18
docker/telemetry/grafana/dashboards/peer-network.json
View File

@@ -2,7 +2,7 @@
"annotations": {
"list": []
},
"description": "Requires trace_peer=1 in the [telemetry] config section.",
"description": "Consensus proposals and validations received from peers, and transaction relay efficiency for this node.",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
@@ -11,7 +11,7 @@
"panels": [
{
"title": "Peer Proposal Receive Rate",
"description": "Rate of consensus proposals received from network peers. The peer.proposal.receive span (PeerImp.cpp) fires in onMessage(TMProposeSet) for each incoming proposal. Records xrpl.peer.id (sending peer) and proposal_trusted (whether the proposer is in our UNL). Requires trace_peer=1 in the telemetry config.",
"description": "**What:** How many consensus proposals this node receives from peers per second.\n**How it's computed:** Per-second rate of received proposals over 5 minutes, per node.\n**Reading it:** A steady rate roughly proportional to the number of proposing validators.\n**Healthy range:** Workload-dependent; scales with validator count and connectivity.\n**Watch for:** A drop toward zero (isolation from the network) or a sudden flood far above baseline (proposal spam).\n**Source:** src/xrpld/overlay/detail/PeerImp.cpp:PeerImp::onMessage(TMProposeSet)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -50,7 +50,7 @@
},
{
"title": "Peer Validation Receive Rate",
"description": "Rate of ledger validations received from network peers. The peer.validation.receive span (PeerImp.cpp) fires in onMessage(TMValidation) for each incoming validation message. Records xrpl.peer.id (sending peer) and validation_trusted (whether the validator is trusted). Requires trace_peer=1 in the telemetry config.",
"description": "**What:** How many ledger validations this node receives from peers per second.\n**How it's computed:** Per-second rate of received validations over 5 minutes, per node.\n**Reading it:** Steady and proportional to the number of validators the node hears from.\n**Healthy range:** Workload-dependent; roughly one burst per validator per closed ledger.\n**Watch for:** A fall toward zero (loss of validator connectivity) or an abnormal surge from untrusted sources.\n**Source:** src/xrpld/overlay/detail/PeerImp.cpp:PeerImp::onMessage(TMValidation)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -89,7 +89,7 @@
},
{
"title": "Proposals Trusted vs Untrusted",
"description": "Pie chart showing the ratio of proposals received from trusted validators (in our UNL) vs untrusted validators. Grouped by the proposal_trusted span attribute (true/false). A healthy node connected to a well-configured UNL should see a significant portion of trusted proposals. Note: proposals that fail early validation may not have the trusted attribute set.",
"description": "**What:** The share of received proposals that come from trusted (UNL) validators versus everyone else.\n**How it's computed:** Received-proposal rate split by trust status over 5 minutes, shown as proportions.\n**Reading it:** A healthy node with a good UNL shows a solid trusted slice.\n**Healthy range:** Workload-dependent; a meaningful trusted fraction is expected.\n**Watch for:** A collapsing trusted share or a large untrusted volume, consistent with proposal flooding from non-UNL peers.\n**Source:** src/xrpld/overlay/detail/PeerImp.cpp:PeerImp::onMessage(TMProposeSet)",
"type": "piechart",
"gridPos": {
"h": 8,
@@ -121,7 +121,7 @@
},
{
"title": "Validations Trusted vs Untrusted",
"description": "Pie chart showing the ratio of validations received from trusted validators (in our UNL) vs untrusted validators. Grouped by the validation_trusted span attribute (true/false). Monitoring this helps detect if the node is receiving validations from the expected set of trusted validators. Note: validations that fail early checks may not have the trusted attribute set.",
"description": "**What:** The share of received validations from trusted (UNL) validators versus untrusted sources.\n**How it's computed:** Received-validation rate split by trust status over 5 minutes, shown as proportions.\n**Reading it:** The trusted slice should dominate for a well-configured node.\n**Healthy range:** Workload-dependent; trusted validations expected to be the majority.\n**Watch for:** A shrinking trusted share or a spike of untrusted validations, which can indicate misconfiguration or abuse.\n**Source:** src/xrpld/overlay/detail/PeerImp.cpp:PeerImp::onMessage(TMValidation)",
"type": "piechart",
"gridPos": {
"h": 8,
@@ -153,7 +153,7 @@
},
{
"title": "Reduce-Relay Peer Selection",
"description": "Transaction reduce-relay efficiency: peers selected as relay sources vs suppressed, plus peers with the feature disabled. A high suppressed:selected ratio proves reduce-relay is saving bandwidth; a high not_enabled count means stale peers force full relay.",
"description": "**What:** How transaction relay picks peers: chosen relay sources, suppressed peers, and peers with the feature off.\n**How it's computed:** Current peer counts in each category (selected, suppressed, not-enabled), per node.\n**Reading it:** A high suppressed-to-selected ratio means relay is saving bandwidth as intended.\n**Healthy range:** Workload-dependent; suppressed should exceed selected in a well-connected mesh.\n**Watch for:** A large not-enabled count (older peers forcing full relay) or selected climbing while suppressed falls.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerReduceRelayGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -206,7 +206,7 @@
},
{
"title": "Reduce-Relay Missing-Tx Frequency",
"description": "Frequency of on-demand transaction fetches triggered when a peer is missing a relayed transaction. A rising value means the suppression is too aggressive and the on-demand fetch path is growing.",
"description": "**What:** How often a peer has to fetch a transaction it missed because relay suppressed it.\n**How it's computed:** The reported frequency of on-demand missing-transaction fetches, per node.\n**Reading it:** Lower is better; near-flat means suppression is well tuned.\n**Healthy range:** Workload-dependent; a low, stable value is expected.\n**Watch for:** A rising trend, meaning suppression is too aggressive and the on-demand fetch path is growing.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerReduceRelayGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -245,7 +245,7 @@
}
],
"schemaVersion": 39,
"tags": ["peer"],
"tags": ["network", "peer"],
"templating": {
"list": [
{
@@ -324,7 +324,7 @@
{
"name": "node",
"label": "Node",
"description": "Filter by rippled node (service.instance.id \u2014 e.g. Node-1)",
"description": "Filter by rippled node (service.instance.id — e.g. Node-1)",
"type": "query",
"query": "label_values(traces_span_metrics_calls_total, exported_instance)",
"datasource": {

513
docker/telemetry/grafana/dashboards/rpc-performance.json
View File

@@ -8,15 +8,28 @@
"id": null,
"links": [],
"panels": [
{
"id": 11,
"type": "row",
"title": "Aggregate RPC (all commands)",
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 0
},
"panels": []
},
{
"title": "RPC Request Rate by Command",
"description": "Per-second rate of RPC command executions, broken down by command name (e.g. server_info, submit). Calculated as rate(traces_span_metrics_calls_total{span_name=~\"rpc.command.*\"}) over a 5m window, grouped by the command span attribute.",
"description": "**What:** How many times each RPC command runs per second.\n**How it's computed:** Per-second rate of command executions over 5 minutes, grouped by command name and node.\n**Reading it:** Compare bars/lines across commands to see which endpoints drive load.\n**Healthy range:** Workload-dependent; dominated by whichever commands clients call most.\n**Watch for:** A single command spiking far above its norm, consistent with a client hammering one endpoint.\n**Source:** src/xrpld/rpc/detail/RPCHandler.cpp:callMethod",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 0
"y": 1
},
"options": {
"tooltip": {
@@ -45,17 +58,18 @@
}
},
"overrides": []
}
},
"id": 1
},
{
"title": "RPC Latency P95 by Command",
"description": "95th percentile response time for each RPC command. Computed from the spanmetrics duration histogram using histogram_quantile(0.95) over rpc.command.* spans, grouped by command. High values indicate slow commands that may need optimization.",
"description": "**What:** The 95th-percentile response time for each RPC command.\n**How it's computed:** 95th-percentile of per-command execution durations over 5 minutes, grouped by command and node.\n**Reading it:** Lower is better; watch the slowest commands.\n**Healthy range:** Fast lookups a few ms; heavy queries tens to hundreds of ms. Workload-dependent.\n**Watch for:** A command whose latency climbs steadily, pointing to an expensive or degrading query path.\n**Source:** src/xrpld/rpc/detail/RPCHandler.cpp:callMethod",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 0
"y": 1
},
"options": {
"tooltip": {
@@ -84,17 +98,18 @@
}
},
"overrides": []
}
},
"id": 2
},
{
"title": "RPC Error Rate",
"description": "Percentage of RPC commands that completed with an error status, per command. Calculated as (error calls / total calls) * 100, where errors have status_code=STATUS_CODE_ERROR. Thresholds: green < 1%, yellow 1-5%, red > 5%.",
"description": "**What:** The percentage of each command's calls that finished with an error.\n**How it's computed:** Error calls divided by total calls per command over 5 minutes, expressed as a percent.\n**Reading it:** Green under 1%, yellow 1-5%, red above 5%.\n**Healthy range:** Below 1% per command in normal operation.\n**Watch for:** A command sitting red, indicating malformed input, overload, or a broken handler.\n**Source:** src/xrpld/rpc/detail/RPCHandler.cpp:callMethod",
"type": "bargauge",
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 8
"y": 9
},
"options": {
"tooltip": {
@@ -132,17 +147,18 @@
}
},
"overrides": []
}
},
"id": 3
},
{
"title": "RPC Latency Heatmap",
"description": "Distribution of RPC command response times across histogram buckets. Shows the density of requests at each latency level over time. Each cell represents the count of requests that fell into that duration bucket in a 5m window. Useful for spotting bimodal latency patterns.",
"description": "**What:** The full distribution of RPC response times over the window.\n**How it's computed:** Counts of requests in each latency band per 5-minute window, shown as color density.\n**Reading it:** A tight low band is healthy; multiple bands mean mixed fast/slow requests.\n**Healthy range:** Most mass in the low-latency bands; workload-dependent.\n**Watch for:** A distinct high-latency cluster (bimodal behavior) that percentiles alone would hide.\n**Source:** src/xrpld/rpc/detail/RPCHandler.cpp:callMethod",
"type": "heatmap",
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 8
"y": 9
},
"options": {
"tooltip": {
@@ -168,17 +184,18 @@
"defaults": {
"unit": "ms"
}
}
},
"id": 4
},
{
"title": "Overall RPC Throughput",
"description": "Aggregate RPC throughput showing two layers of the request pipeline. rpc.http_request is the outer HTTP handler (ServerHandler.cpp) that accepts incoming connections. rpc.process is the inner processing layer (ServerHandler.cpp) that parses and dispatches. A gap between the two indicates requests being queued or rejected before processing.",
"description": "**What:** Two layers of the request pipeline: connections accepted by the HTTP handler versus requests parsed and dispatched.\n**How it's computed:** Per-second rates of the outer request-accept layer and the inner processing layer over 5 minutes, per node.\n**Reading it:** The two lines should track each other closely.\n**Healthy range:** Workload-dependent; both lines roughly equal.\n**Watch for:** A gap between accept and process, meaning requests are queued or rejected before dispatch.\n**Source:** src/xrpld/rpc/detail/ServerHandler.cpp:ServerHandler::processSession; src/xrpld/rpc/detail/ServerHandler.cpp:ServerHandler::processRequest",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 16
"y": 17
},
"options": {
"tooltip": {
@@ -214,17 +231,18 @@
}
},
"overrides": []
}
},
"id": 5
},
{
"title": "RPC Success vs Error",
"description": "Aggregate rate of successful vs failed RPC commands across all command types. Success = status_code UNSET (OpenTelemetry default for OK spans). Error = status_code STATUS_CODE_ERROR. A sustained error rate warrants investigation via per-command breakdown above.",
"description": "**What:** Aggregate rate of successful versus failed RPC commands across all command types.\n**How it's computed:** Per-second rates of ok and errored command completions over 5 minutes, per node.\n**Reading it:** Success should dominate; the error line should stay near the floor.\n**Healthy range:** Error line near zero relative to success; workload-dependent.\n**Watch for:** A sustained error line, which warrants drilling into the per-command breakdown.\n**Source:** src/xrpld/rpc/detail/RPCHandler.cpp:callMethod",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 16
"y": 17
},
"options": {
"tooltip": {
@@ -260,17 +278,18 @@
}
},
"overrides": []
}
},
"id": 6
},
{
"title": "Top Commands by Volume",
"description": "Top 10 most frequently called RPC commands by total invocation count over the last 5 minutes. Uses topk(10, increase(calls_total)) to rank commands. Helps identify the hottest API endpoints driving load on the node.",
"description": "**What:** The ten most-called RPC commands over the recent window.\n**How it's computed:** Total invocation counts per command over the last 5 minutes, ranked and capped at ten.\n**Reading it:** The top bars are the hottest endpoints driving node load.\n**Healthy range:** Workload-dependent; typically led by lightweight status queries.\n**Watch for:** An unexpected command dominating the ranking, consistent with automated abuse.\n**Source:** src/xrpld/rpc/detail/RPCHandler.cpp:callMethod",
"type": "bargauge",
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 24
"y": 25
},
"options": {
"tooltip": {
@@ -292,17 +311,18 @@
"unit": "none"
},
"overrides": []
}
},
"id": 7
},
{
"title": "WebSocket Message Rate",
"description": "Rate of incoming WebSocket RPC messages processed by the server. Sourced from the rpc.ws_message span (ServerHandler.cpp). Only active when clients connect via WebSocket instead of HTTP. Zero is normal if only HTTP RPC is in use.",
"description": "**What:** How many RPC messages arrive over WebSocket connections per second.\n**How it's computed:** Per-second rate of processed WebSocket RPC messages over 5 minutes, per node.\n**Reading it:** Non-zero only when clients use WebSocket; zero is normal for HTTP-only nodes.\n**Healthy range:** Workload-dependent; zero if no WebSocket clients.\n**Watch for:** A sudden surge from a single source, consistent with a chatty or abusive subscription client.\n**Source:** src/xrpld/rpc/detail/ServerHandler.cpp:ServerHandler::processSession",
"type": "stat",
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 24
"y": 25
},
"options": {
"tooltip": {
@@ -324,17 +344,18 @@
"unit": "ops"
},
"overrides": []
}
},
"id": 8
},
{
"title": "RPC Resource Cost by Command",
"description": "RPC commands grouped by load_type (resource cost category). High-cost categories like exception_rpc or malformed_rpc indicate problematic clients.",
"description": "**What:** RPC traffic grouped by resource-cost category rather than by command name.\n**How it's computed:** Per-second rate of commands over 5 minutes, grouped by load/cost category.\n**Reading it:** Cheap categories should dominate; costly ones should be a small slice.\n**Healthy range:** Workload-dependent; low-cost categories carry most traffic.\n**Watch for:** Growth in high-cost categories such as exception or malformed requests, which point to problematic clients.\n**Source:** src/xrpld/rpc/detail/RPCHandler.cpp:callMethod",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 32
"y": 33
},
"options": {
"tooltip": {
@@ -368,17 +389,18 @@
}
},
"overrides": []
}
},
"id": 9
},
{
"title": "Batch vs Single RPC Requests",
"description": "Rate of batch RPC requests vs single requests. High batch rate may indicate bulk automation clients.",
"description": "**What:** The rate of batched RPC requests versus single-command requests.\n**How it's computed:** Per-second rates of batch and single requests over 5 minutes, per node.\n**Reading it:** Single requests usually dominate; batch is used by bulk clients.\n**Healthy range:** Workload-dependent; batch typically a minority.\n**Watch for:** A batch rate climbing sharply, consistent with bulk automation or amplification attempts.\n**Source:** src/xrpld/rpc/detail/ServerHandler.cpp:ServerHandler::processRequest",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 32
"x": 0,
"y": 41
},
"options": {
"tooltip": {
@@ -414,7 +436,411 @@
}
},
"overrides": []
}
},
"id": 10
},
{
"id": 12,
"type": "row",
"title": "Per-Method Detail",
"collapsed": false,
"gridPos": {
"h": 1,
"w": 24,
"x": 0,
"y": 50
},
"panels": []
},
{
"title": "RPC Call Rate (All Methods)",
"description": "**What:** Overall rate of RPC method calls that started, finished, and errored, across all methods.\n**How it's computed:** Per-second rate of each counter over a 5-minute window, summed per node.\n**Reading it:** Started should closely track finished; errored should be a small fraction.\n**Healthy range:** Workload-dependent; started ≈ finished, errored near zero.\n**Watch for:** A growing gap between started and finished (calls hanging), or an errored line that rises with load.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp recordRpcStarted / recordRpcFinished / recordRpcErrored",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 51
},
"options": {
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": {
"type": "prometheus"
},
"expr": "sum by (exported_instance) (rate(xrpld_rpc_method_started_total{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m]))",
"legendFormat": "Started/s [{{exported_instance}}]"
},
{
"datasource": {
"type": "prometheus"
},
"expr": "sum by (exported_instance) (rate(xrpld_rpc_method_finished_total{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m]))",
"legendFormat": "Finished/s [{{exported_instance}}]"
},
{
"datasource": {
"type": "prometheus"
},
"expr": "sum by (exported_instance) (rate(xrpld_rpc_method_errored_total{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m]))",
"legendFormat": "Errored/s [{{exported_instance}}]"
}
],
"fieldConfig": {
"defaults": {
"unit": "ops",
"custom": {
"drawStyle": "line",
"lineWidth": 2,
"fillOpacity": 10,
"axisLabel": "Operations / Sec",
"spanNulls": true,
"insertNulls": false,
"showPoints": "auto",
"pointSize": 3
},
"color": {
"mode": "palette-classic"
}
},
"overrides": []
},
"id": 13
},
{
"title": "Per-Method Call Rate (Top 10)",
"description": "**What:** The ten busiest RPC methods by call rate.\n**How it's computed:** Per-second start rate over 5 minutes, per method, showing the top ten.\n**Reading it:** Identifies which methods dominate load; the mix shifts with client behaviour.\n**Healthy range:** Workload-dependent.\n**Watch for:** A single method suddenly dominating, which can signal a runaway client or abusive query pattern.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp recordRpcStarted",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 59
},
"options": {
"tooltip": {
"mode": "multi",
"sort": "desc"
},
"legend": {
"displayMode": "table",
"placement": "right",
"calcs": ["mean", "max"]
}
},
"targets": [
{
"datasource": {
"type": "prometheus"
},
"expr": "topk(10, rate(xrpld_rpc_method_started_total{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m]))",
"legendFormat": "{{method}} [{{exported_instance}}]"
}
],
"fieldConfig": {
"defaults": {
"unit": "ops",
"custom": {
"drawStyle": "line",
"lineWidth": 1,
"fillOpacity": 5,
"axisLabel": "Operations / Sec",
"spanNulls": true,
"insertNulls": false,
"showPoints": "auto",
"pointSize": 3
},
"color": {
"mode": "palette-classic"
}
},
"overrides": []
},
"id": 14
},
{
"title": "Per-Method Error Rate (Top 10)",
"description": "**What:** The ten RPC methods producing the most errors.\n**How it's computed:** Per-second error rate over 5 minutes, per method, top ten.\n**Reading it:** Normally near zero; persistent errors point to a specific failing method.\n**Healthy range:** Near zero for well-behaved traffic.\n**Watch for:** Sustained errors concentrated on one method — a broken client, a bad input, or probing.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp recordRpcErrored",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 67
},
"options": {
"tooltip": {
"mode": "multi",
"sort": "desc"
},
"legend": {
"displayMode": "table",
"placement": "right",
"calcs": ["mean", "max"]
}
},
"targets": [
{
"datasource": {
"type": "prometheus"
},
"expr": "topk(10, rate(xrpld_rpc_method_errored_total{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m]))",
"legendFormat": "{{method}} [{{exported_instance}}]"
}
],
"fieldConfig": {
"defaults": {
"unit": "ops",
"custom": {
"drawStyle": "line",
"lineWidth": 1,
"fillOpacity": 5,
"axisLabel": "Operations / Sec",
"spanNulls": true,
"insertNulls": false,
"showPoints": "auto",
"pointSize": 3
},
"color": {
"mode": "palette-classic"
}
},
"overrides": []
},
"id": 15
},
{
"title": "RPC Latency - All Methods",
"description": "**What:** Aggregate RPC handler latency across all methods (p75 and p99).\n**How it's computed:** Percentiles of the method-duration histogram over a 5-minute window.\n**Reading it:** p75 reflects typical responsiveness; p99 captures the slow tail.\n**Healthy range:** Sub-millisecond to low-millisecond for light commands; heavier commands run longer.\n**Watch for:** A rising p99 while p75 stays flat — a subset of calls degrading, often from expensive queries.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp recordRpcFinished",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 75
},
"options": {
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": {
"type": "prometheus"
},
"expr": "histogram_quantile(0.75, sum by (le, exported_instance) (rate(xrpld_rpc_method_duration_us_bucket{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m])))",
"legendFormat": "p75 [{{exported_instance}}]"
},
{
"datasource": {
"type": "prometheus"
},
"expr": "histogram_quantile(0.99, sum by (le, exported_instance) (rate(xrpld_rpc_method_duration_us_bucket{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m])))",
"legendFormat": "p99 [{{exported_instance}}]"
}
],
"fieldConfig": {
"defaults": {
"unit": "µs",
"custom": {
"drawStyle": "line",
"lineWidth": 2,
"fillOpacity": 5,
"axisLabel": "Duration (μs)",
"spanNulls": true,
"insertNulls": false,
"showPoints": "auto",
"pointSize": 3
},
"color": {
"mode": "palette-classic"
}
},
"overrides": []
},
"id": 16
},
{
"title": "Per-Method Latency (p99, Top 10 Slowest)",
"description": "**What:** The ten slowest RPC methods by tail latency.\n**How it's computed:** p99 of each method's duration histogram over 5 minutes, top ten.\n**Reading it:** Surfaces which specific methods are expensive.\n**Healthy range:** Method-dependent; ledger/account queries are heavier than status calls.\n**Watch for:** A method whose p99 climbs over time, or an unexpectedly cheap method appearing here.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp recordRpcFinished",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 83
},
"options": {
"tooltip": {
"mode": "multi",
"sort": "desc"
},
"legend": {
"displayMode": "table",
"placement": "right",
"calcs": ["mean", "max"]
}
},
"targets": [
{
"datasource": {
"type": "prometheus"
},
"expr": "topk(10, histogram_quantile(0.99, sum by (le, method, exported_instance) (rate(xrpld_rpc_method_duration_us_bucket{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m]))))",
"legendFormat": "{{method}} [{{exported_instance}}]"
}
],
"fieldConfig": {
"defaults": {
"unit": "µs",
"custom": {
"drawStyle": "line",
"lineWidth": 1,
"fillOpacity": 5,
"axisLabel": "Duration (μs)",
"spanNulls": true,
"insertNulls": false,
"showPoints": "auto",
"pointSize": 3
},
"color": {
"mode": "palette-classic"
}
},
"overrides": []
},
"id": 17
},
{
"title": "RPC Error Ratio by Method",
"description": "**What:** The methods with the highest error rates, for spotting failure hotspots.\n**How it's computed:** Per-second error rate over 5 minutes, per method, top ten.\n**Reading it:** Highlights where failures concentrate.\n**Healthy range:** Near zero.\n**Watch for:** One method with a persistently high error rate — malformed requests or targeted probing.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp recordRpcErrored",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 91
},
"options": {
"tooltip": {
"mode": "multi",
"sort": "desc"
},
"legend": {
"displayMode": "table",
"placement": "right",
"calcs": ["mean", "max"]
}
},
"targets": [
{
"datasource": {
"type": "prometheus"
},
"expr": "topk(10, rate(xrpld_rpc_method_errored_total{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m]) / (rate(xrpld_rpc_method_started_total{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m]) > 0))",
"legendFormat": "{{method}} [{{exported_instance}}]"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0,
"max": 1,
"custom": {
"drawStyle": "line",
"lineWidth": 1,
"fillOpacity": 5,
"axisLabel": "Error Ratio",
"spanNulls": true,
"insertNulls": false,
"showPoints": "auto",
"pointSize": 3
},
"color": {
"mode": "palette-classic"
},
"thresholds": {
"steps": [
{
"color": "green",
"value": null
},
{
"color": "yellow",
"value": 0.05
},
{
"color": "red",
"value": 0.25
}
]
}
},
"overrides": []
},
"id": 18
},
{
"title": "Current RPC Latency (p99 Gauge)",
"description": "**What:** Current tail latency (p99) of RPC handling across all methods, as a live gauge.\n**How it's computed:** p99 of the method-duration histogram over the recent window.\n**Reading it:** A single at-a-glance number for current RPC responsiveness.\n**Healthy range:** Low-millisecond under normal load.\n**Watch for:** Sustained elevation, indicating the node is under query pressure.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp recordRpcFinished",
"type": "gauge",
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 99
},
"options": {
"reduceOptions": {
"calcs": ["lastNotNull"],
"fields": "",
"values": false
},
"orientation": "auto",
"showThresholdLabels": false,
"showThresholdMarkers": true
},
"targets": [
{
"datasource": {
"type": "prometheus"
},
"expr": "histogram_quantile(0.99, sum by (le) (rate(xrpld_rpc_method_duration_us_bucket{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m])))",
"legendFormat": "p99 Latency"
}
],
"fieldConfig": {
"defaults": {
"unit": "µs",
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "yellow",
"value": 100000
},
{
"color": "red",
"value": 1000000
}
]
}
},
"overrides": []
},
"id": 19
}
],
"schemaVersion": 39,
@@ -497,7 +923,7 @@
{
"name": "node",
"label": "Node",
"description": "Filter by rippled node (service.instance.id \u2014 e.g. Node-1)",
"description": "Filter by rippled node (service.instance.id — e.g. Node-1)",
"type": "query",
"query": "label_values(traces_span_metrics_calls_total, exported_instance)",
"datasource": {
@@ -533,6 +959,26 @@
"multi": true,
"refresh": 2,
"sort": 1
},
{
"name": "method",
"label": "RPC Method",
"description": "Filter by RPC method",
"type": "query",
"query": "label_values(xrpld_rpc_method_started_total, method)",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"includeAll": true,
"allValue": ".*",
"current": {
"text": "All",
"value": "$__all"
},
"multi": true,
"refresh": 2,
"sort": 1
}
]
},
@@ -542,5 +988,6 @@
},
"title": "RPC Performance",
"uid": "xrpld-rpc-perf",
"refresh": "5s"
"refresh": "5s",
"description": "Per-command and per-method RPC performance: call rates, error rates, and latency distributions."
}

21
docker/telemetry/grafana/dashboards/system-ledger-data-sync.json
View File

@@ -2,7 +2,7 @@
"annotations": {
"list": []
},
"description": "Ledger data exchange and object fetch traffic from beast::insight System Metrics. Covers ledger sync, node data retrieval, and transaction set exchange. Requires [insight] server=otel in rippled config.",
"description": "Ledger data exchange and object-fetch traffic between this node and its peers: ledger sync, tree-node retrieval, and transaction-set exchange. Use it to see how much ledger data the node is pulling or serving and to spot catch-up activity.",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
@@ -11,7 +11,7 @@
"panels": [
{
"title": "Ledger Data Exchange (Bytes In)",
"description": "Inbound bytes for ledger data sub-categories. 'ledger_data' = aggregated ledger data, sub-types include Transaction_Set_candidate (proposed tx sets), Transaction_Node (tx tree nodes), and Account_State_Node (state tree nodes). High Account_State_Node traffic indicates state sync; high Transaction_Set_candidate indicates consensus catch-up. Sourced from TrafficCount.h ledger_data_* categories.",
"description": "**What:** Inbound bytes for ledger-data message categories, split into aggregate get/share plus the transaction-set, transaction-node, and account-state-node sub-types the node receives from peers.\n**How it's computed:** Per-category inbound byte counters, shown at their latest cumulative value per node.\n**Reading it:** Normally low and flat once synced. Account-state-node traffic dominates during state sync; transaction-set-candidate traffic dominates during consensus catch-up.\n**Healthy range:** workload-dependent; low and steady on a synced node.\n**Watch for:** Sustained high account-state or tx-node inbound bytes on a node that should be caught up (repeated re-sync, missing history), or a single peer driving all traffic.\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -99,7 +99,7 @@
},
{
"title": "Ledger Share/Get Traffic (Bytes)",
"description": "Legacy ledger share and get traffic by sub-type. These are the older ledger fetch protocol categories (as opposed to ledger_data_* which is the newer protocol). Sub-types: Transaction_Set_candidate, Transaction_node, Account_State_node, plus aggregate ledger_share and ledger_get. Sourced from TrafficCount.h ledger_* categories.",
"description": "**What:** Inbound bytes for the older ledger share/get message categories and their tx-set, tx-node, and account-state sub-types.\n**How it's computed:** Per-category inbound byte counters at their latest value per node.\n**Reading it:** Usually small; these legacy categories carry ledger-fetch traffic for peers using the older protocol.\n**Healthy range:** workload-dependent; low on a synced node.\n**Watch for:** Large sustained volumes indicating heavy fetch load or a peer repeatedly requesting the same data.\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -187,7 +187,7 @@
},
{
"title": "GetObject Traffic by Type (Bytes In)",
"description": "Object fetch traffic by object type. GetObject is the protocol for fetching specific SHAMap nodes. Types: Ledger (full ledger headers), Transaction (individual txs), Transaction_node (tx tree nodes), Account_State_node (state tree nodes), CAS (Content Addressable Storage objects), Fetch_Pack (batch fetch during catch-up), Transactions (bulk tx fetch). High Fetch_Pack traffic indicates a node is catching up. Sourced from TrafficCount.h getobject_* categories.",
"description": "**What:** Inbound bytes for object-fetch traffic broken down by object type: ledger headers, individual transactions, transaction-tree nodes, and state-tree nodes.\n**How it's computed:** Per-type inbound byte counters at their latest value per node.\n**Reading it:** Small during steady state; grows when the node fetches missing tree nodes.\n**Healthy range:** workload-dependent; low when synced.\n**Watch for:** A large share on state/tx nodes for long periods (persistent gap-filling), meaning the node keeps catching up.\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -275,7 +275,7 @@
},
{
"title": "GetObject Aggregate & Special Types (Bytes In)",
"description": "Aggregate getobject traffic plus special categories: CAS (Content Addressable Storage) for SHAMap node fetch, Fetch_Pack for bulk batch downloads during catch-up, Transactions for bulk tx fetch, and the aggregate getobject_get/getobject_share totals. Sourced from TrafficCount.h getobject_* categories.",
"description": "**What:** Aggregate object-fetch inbound bytes plus special buckets: content-addressed storage fetches, bulk fetch-pack downloads used during catch-up, and bulk transaction fetches.\n**How it's computed:** Per-category inbound byte counters at their latest value per node.\n**Reading it:** Fetch-pack rises sharply while catching up a range of ledgers; near zero when fully synced.\n**Healthy range:** workload-dependent; low when synced.\n**Watch for:** Continuous fetch-pack traffic (node never fully catches up) or unexpectedly high content-store volume.\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -356,7 +356,7 @@
},
{
"title": "GetObject Messages by Type",
"description": "Message counts for object fetch operations. Shows how many individual fetch requests and responses are exchanged per type. High message counts with low byte counts indicate small object fetches; the inverse indicates large batch transfers. Sourced from TrafficCount.h getobject_* categories.",
"description": "**What:** Count of individual object-fetch request/response messages per object type.\n**How it's computed:** Per-type inbound message counters at their latest value per node.\n**Reading it:** Many messages with few bytes means small piecemeal fetches; few messages with many bytes means large batch transfers.\n**Healthy range:** workload-dependent.\n**Watch for:** High message counts with tiny payloads sustained over time (inefficient per-node fetching).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -437,7 +437,7 @@
},
{
"title": "Overlay Traffic Heatmap (All Categories, Bytes In)",
"description": "Bar gauge showing all overlay traffic categories ranked by inbound bytes. Provides a complete at-a-glance view of which protocol message types consume the most bandwidth across all 57+ traffic categories. Sourced from all TrafficCount.h categories via wildcard match.",
"description": "**What:** All overlay traffic categories ranked by inbound bytes, giving an at-a-glance view of which message types consume the most receive bandwidth.\n**How it's computed:** Top categories by latest inbound byte value across all traffic categories.\n**Reading it:** The longest bars are the biggest bandwidth consumers; on a synced node transactions, proposals, and validations usually lead.\n**Healthy range:** workload-dependent.\n**Watch for:** A single ledger-data or fetch category dominating (ongoing sync) or an unexpected category topping the list.\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "bargauge",
"gridPos": {
"h": 8,
@@ -493,7 +493,7 @@
}
],
"schemaVersion": 39,
"tags": ["statsd", "ledger", "sync"],
"tags": ["ledger", "sync"],
"templating": {
"list": [
{
@@ -595,7 +595,6 @@
"from": "now-1h",
"to": "now"
},
"title": "Ledger Data & Sync (System Metrics)",
"uid": "xrpld-system-ledger-sync",
"refresh": "5s"
"title": "Ledger Data & Sync",
"uid": "xrpld-system-ledger-sync"
}

29
docker/telemetry/grafana/dashboards/system-network-traffic.json
View File

@@ -2,7 +2,7 @@
"annotations": {
"list": []
},
"description": "Network traffic and peer metrics from beast::insight System Metrics. Requires [insight] server=otel in rippled config.",
"description": "Peer connectivity and overlay bandwidth for this node: peer counts, disconnects, total bytes and messages exchanged, and the transaction/proposal/validation traffic that flows across the peer-to-peer network.",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
@@ -11,7 +11,7 @@
"panels": [
{
"title": "Active Peers",
"description": "Number of active inbound and outbound peer connections. Sourced from Peer_Finder.Active_Inbound_Peers and Peer_Finder.Active_Outbound_Peers gauges (PeerfinderManager.cpp). A healthy mainnet node typically has 10-21 outbound and 0-85 inbound peers depending on configuration.",
"description": "**What:** Number of active inbound and outbound peer connections the node currently holds.\n**How it's computed:** Current value of the inbound and outbound active-peer counts per node.\n**Reading it:** Outbound is what the node dials out; inbound is what others open to it. Both should be stable.\n**Healthy range:** roughly 10-21 outbound and 0-85 inbound on mainnet, depending on config.\n**Watch for:** Outbound dropping toward zero (isolation) or inbound pinned at the limit with churn (connection pressure).\n**Source:** src/xrpld/peerfinder/detail/PeerfinderManager.cpp Logic Stats ctor",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -57,7 +57,7 @@
},
{
"title": "Peer Disconnects",
"description": "Cumulative count of peer disconnections. Sourced from the Overlay.Peer_Disconnects gauge (OverlayImpl.h). A rising trend indicates network instability, aggressive peer management, or resource exhaustion causing connection drops.",
"description": "**What:** Cumulative count of peer connections that have dropped.\n**How it's computed:** Running total of disconnect events per node.\n**Reading it:** A flat or slowly rising line is normal; the slope matters more than the absolute value.\n**Healthy range:** workload-dependent; slow, steady growth.\n**Watch for:** Sharp step-ups in the slope (network instability, resource exhaustion, or many peers dropping the node at once).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.h OverlayImpl::Stats ctor",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -96,7 +96,7 @@
},
{
"title": "Total Network Bytes",
"description": "Total bytes sent and received across all peer connections. Sourced from the total.Bytes_In and total.Bytes_Out traffic category gauges (OverlayImpl.h). Provides a high-level view of network bandwidth consumption.",
"description": "**What:** Total bytes received and sent across all peer connections.\n**How it's computed:** Current cumulative in/out byte totals per node.\n**Reading it:** Overall bandwidth footprint; in and out usually track network activity together.\n**Healthy range:** workload-dependent.\n**Watch for:** Sudden sustained jumps not matched by ledger or transaction activity (relay storms or a noisy peer).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -142,7 +142,7 @@
},
{
"title": "Total Network Messages",
"description": "Total messages sent and received across all peer connections. Sourced from the total.Messages_In and total.Messages_Out traffic category gauges (OverlayImpl.h). Shows the overall message throughput of the overlay network.",
"description": "**What:** Total messages received and sent across all peer connections.\n**How it's computed:** Current cumulative in/out message totals per node.\n**Reading it:** Overall message throughput of the overlay; complements the byte totals.\n**Healthy range:** workload-dependent.\n**Watch for:** Message count climbing far faster than bytes (many tiny messages, possible flooding).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -188,7 +188,7 @@
},
{
"title": "Transaction Traffic",
"description": "Bytes and messages for transaction-related overlay traffic. Includes the transactions traffic category (OverlayImpl/TrafficCount.h). Spikes indicate high transaction volume on the network or transaction flooding.",
"description": "**What:** Transaction relay messages in and out, plus duplicate transaction messages received.\n**How it's computed:** Current message counts for the transaction and transaction-duplicate categories.\n**Reading it:** In/out rise with network transaction volume; duplicates are transactions the node already had.\n**Healthy range:** workload-dependent; duplicates a modest fraction of inbound.\n**Watch for:** Duplicate inbound approaching or exceeding unique inbound (redundant relay), or a sharp spike suggesting transaction flooding.\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -241,7 +241,7 @@
},
{
"title": "Proposal Traffic",
"description": "Messages for consensus proposal overlay traffic. Includes proposals, proposals_untrusted, and proposals_duplicate categories (TrafficCount.h). High untrusted or duplicate counts may indicate UNL misconfiguration or network spam.",
"description": "**What:** Consensus proposal messages in/out, plus untrusted and duplicate proposal messages received.\n**How it's computed:** Current message counts for the proposal, proposal-untrusted, and proposal-duplicate categories.\n**Reading it:** Trusted in/out track consensus rounds; untrusted come from validators not on this node's trusted list.\n**Healthy range:** workload-dependent; untrusted and duplicates low relative to trusted.\n**Watch for:** High untrusted (trusted-list misconfiguration) or high duplicates (inefficient relay or proposal spam).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -301,7 +301,7 @@
},
{
"title": "Validation Traffic",
"description": "Messages for validation overlay traffic. Includes validations, validations_untrusted, and validations_duplicate categories (TrafficCount.h). Monitoring trusted vs untrusted validation traffic helps detect UNL health issues.",
"description": "**What:** Validation messages in/out, plus untrusted and duplicate validation messages received.\n**How it's computed:** Current message counts for the validation, validation-untrusted, and validation-duplicate categories.\n**Reading it:** Trusted validations should arrive steadily each ledger; untrusted come from non-trusted validators.\n**Healthy range:** workload-dependent; untrusted and duplicates low relative to trusted.\n**Watch for:** Rising untrusted or duplicate validations (trusted-list health issues or validation spam).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -361,7 +361,7 @@
},
{
"title": "Overlay Traffic by Category (Bytes In)",
"description": "Top traffic categories by inbound bytes. Includes all 57 overlay traffic categories from TrafficCount.h. Shows which protocol message types consume the most bandwidth. Categories include transactions, proposals, validations, ledger data, getobject, and overlay overhead.",
"description": "**What:** Top overlay traffic categories ranked by inbound bytes, excluding the all-traffic total.\n**How it's computed:** Top categories by latest inbound byte value per node.\n**Reading it:** Shows which message types dominate receive bandwidth right now.\n**Healthy range:** workload-dependent; transactions, proposals, and validations typically lead on a synced node.\n**Watch for:** A fetch or ledger-data category topping the list (sync activity) or an unexpected category dominating.\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "bargauge",
"gridPos": {
"h": 8,
@@ -658,7 +658,7 @@
},
{
"title": "Duplicate Traffic (Wasted Bandwidth)",
"description": "Rate of duplicate overlay traffic across transaction, proposal, and validation categories. Duplicate messages are messages the node has already seen and discards. High duplicate rates indicate inefficient message routing or network topology issues causing redundant relays.",
"description": "**What:** Throughput of duplicate transaction, proposal, and validation traffic: messages the node had already seen and discarded.\n**How it's computed:** Per-second rate of the duplicate byte counters for each category, in and out.\n**Reading it:** Lower is better; this is bandwidth spent on redundant relays.\n**Healthy range:** workload-dependent; a small fraction of total traffic.\n**Watch for:** Duplicate rate climbing toward the same order as useful traffic (poor relay topology or redundant flooding).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -732,7 +732,7 @@
},
{
"title": "All Traffic Categories (Detail)",
"description": "Top 15 traffic categories by inbound byte rate, excluding the total aggregate. Provides a detailed timeseries view of which overlay message types are consuming the most bandwidth over time. Complements the bar gauge snapshot view in the Overlay Traffic panel.",
"description": "**What:** The busiest overlay categories by inbound byte rate over time, excluding the all-traffic total.\n**How it's computed:** Per-second inbound byte rate for the top categories, ranked.\n**Reading it:** Time-series companion to the category bar view; shows how the traffic mix shifts over the window.\n**Healthy range:** workload-dependent.\n**Watch for:** A category ramping up and staying high, or the mix suddenly changing (sync, spam, or a misbehaving peer).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -771,7 +771,7 @@
}
],
"schemaVersion": 39,
"tags": ["statsd", "network"],
"tags": ["network"],
"templating": {
"list": [
{
@@ -873,7 +873,6 @@
"from": "now-1h",
"to": "now"
},
"title": "Network Traffic (System Metrics)",
"uid": "xrpld-system-network",
"refresh": "5s"
"title": "Network Traffic",
"uid": "xrpld-system-network"
}

110
docker/telemetry/grafana/dashboards/system-node-health.json
View File

@@ -2,7 +2,7 @@
"annotations": {
"list": []
},
"description": "Node health metrics from beast::insight System Metrics. Requires [insight] server=otel in rippled config.",
"description": "Operational health of an XRPL node: sync state, ledger progress, caches, storage, job queue, and network economy.",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
@@ -11,7 +11,7 @@
"panels": [
{
"title": "Validated Ledger Age",
"description": "Age of the most recently validated ledger in seconds. Sourced from the LedgerMaster.Validated_Ledger_Age gauge (LedgerMaster.h) which is updated every collection interval via the insight hook. Values above 20s indicate the node is falling behind the network.",
"description": "**What:** How old the most recently validated ledger is, in seconds.\n**How it's computed:** Current value of the validated-ledger-age gauge, refreshed each collection interval.\n**Reading it:** Lower is better; it should stay near the network close interval.\n**Healthy range:** Under about 7 seconds on a healthy node.\n**Watch for:** Values above 20 seconds mean the node is falling behind the network.\n**Source:** src/xrpld/app/ledger/LedgerMaster.h:LedgerMaster::Stats::collectMetrics",
"type": "stat",
"gridPos": {
"h": 8,
@@ -60,7 +60,7 @@
},
{
"title": "Published Ledger Age",
"description": "Age of the most recently published ledger in seconds. Sourced from the LedgerMaster.Published_Ledger_Age gauge (LedgerMaster.h). Published ledger age should track close to validated ledger age. A growing gap indicates publish pipeline backlog.",
"description": "**What:** How old the most recently published ledger is, in seconds.\n**How it's computed:** Current value of the published-ledger-age gauge, refreshed each collection interval.\n**Reading it:** Lower is better; should track close to the validated ledger age.\n**Healthy range:** Under about 7 seconds on a healthy node.\n**Watch for:** A growing gap above the validated age indicates the publish pipeline is backing up.\n**Source:** src/xrpld/app/ledger/LedgerMaster.h:LedgerMaster::Stats::collectMetrics",
"type": "stat",
"gridPos": {
"h": 8,
@@ -109,7 +109,7 @@
},
{
"title": "Operating Mode Duration",
"description": "Cumulative time spent in each operating mode (Disconnected, Connected, Syncing, Tracking, Full). Sourced from State_Accounting.*_duration gauges (NetworkOPs.cpp) which report microseconds. A healthy node should spend the vast majority of time in Full mode.",
"description": "**What:** Cumulative time the node has spent in each operating mode (Disconnected, Connected, Syncing, Tracking, Full).\n**How it's computed:** Current value of each per-mode duration counter, plotted as lines.\n**Reading it:** The Full line should climb steadily and dominate; other lines should be flat.\n**Healthy range:** Almost all accumulated time in Full mode.\n**Watch for:** Time accumulating in Syncing, Connected, or Disconnected means the node is not staying fully synced.\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp:NetworkOPsImp::Stats",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -162,7 +162,7 @@
],
"fieldConfig": {
"defaults": {
"unit": "\u00b5s",
"unit": "µs",
"custom": {
"axisLabel": "Duration",
"spanNulls": true,
@@ -176,7 +176,7 @@
},
{
"title": "Operating Mode Transitions",
"description": "Count of transitions into each operating mode. Sourced from State_Accounting.*_transitions gauges (NetworkOPs.cpp). Frequent transitions out of Full mode indicate instability. Transitions to Disconnected or Syncing warrant investigation.",
"description": "**What:** Cumulative count of transitions into each operating mode.\n**How it's computed:** Current value of each per-mode transition counter, plotted as lines.\n**Reading it:** Flat lines are healthy; steps up mean the node changed mode.\n**Healthy range:** Few transitions once the node is stable in Full mode.\n**Watch for:** Frequent transitions out of Full, or into Disconnected or Syncing, indicate instability.\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp:NetworkOPsImp::Stats",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -243,7 +243,7 @@
},
{
"title": "I/O Latency",
"description": "P95 of the I/O service loop latency in milliseconds. Sourced from the ios_latency event (Application.cpp) which measures how long it takes for the io_context to process a timer callback. Values above 10ms are logged; above 500ms trigger warnings. High values indicate thread pool saturation or blocking operations.",
"description": "**What:** Latency of the I/O service loop, at the 95th percentile in milliseconds.\n**How it's computed:** I/O loop latency samples aggregated to their 95th percentile per node over a 5-minute window.\n**Reading it:** Lower is better; it reflects how promptly queued I/O work runs.\n**Healthy range:** A few milliseconds; brief spikes are tolerable.\n**Watch for:** Sustained values above a few hundred ms indicate thread-pool saturation or blocking operations.\n**Source:** src/xrpld/app/main/Application.cpp:ApplicationImp",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -282,7 +282,7 @@
},
{
"title": "Job Queue Depth",
"description": "Current number of jobs waiting in the job queue. Sourced from the job_count gauge (JobQueue.cpp). A sustained high value indicates the node cannot process work fast enough \u2014 common during ledger replay or heavy RPC load.",
"description": "**What:** Number of jobs currently waiting in the internal work queue.\n**How it's computed:** Current value of the job-count gauge, plotted over time.\n**Reading it:** Lower is better; a near-flat low line means the node keeps up with its workload.\n**Healthy range:** Low single digits at idle, brief bumps under load.\n**Watch for:** A sustained high depth means the node cannot process work fast enough, common during replay, heavy RPC, or a request flood.\n**Source:** src/libxrpl/core/detail/JobQueue.cpp:JobQueue",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -321,7 +321,7 @@
},
{
"title": "Ledger Fetch Rate",
"description": "Rate of ledger fetch requests initiated by the node. Sourced from the ledger_fetches counter (InboundLedgers.cpp) which increments each time the node requests a ledger from a peer. High rates indicate the node is catching up or missing ledgers.",
"description": "**What:** How often the node requests a ledger from its peers.\n**How it's computed:** Per-second rate of ledger-fetch requests over a 5-minute window.\n**Reading it:** Near zero in steady state; elevated while catching up.\n**Healthy range:** Close to zero once fully synced.\n**Watch for:** A sustained high rate means the node is repeatedly missing ledgers and back-filling from peers.\n**Source:** src/xrpld/app/ledger/detail/InboundLedgers.cpp:InboundLedgersImp",
"type": "stat",
"gridPos": {
"h": 8,
@@ -354,7 +354,7 @@
},
{
"title": "Ledger History Mismatches",
"description": "Rate of ledger history hash mismatches. Sourced from the ledger.history.mismatch counter (LedgerHistory.cpp) which increments when a built ledger hash does not match the expected validated hash. Non-zero values indicate consensus divergence or database corruption.",
"description": "**What:** Rate at which a locally built ledger hash fails to match the network-validated hash.\n**How it's computed:** Per-second rate of history-mismatch events over a 5-minute window.\n**Reading it:** Flat at zero is the only healthy reading.\n**Healthy range:** Zero.\n**Watch for:** Any nonzero value indicates consensus divergence or database corruption and warrants immediate investigation.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:incrementLedgerHistoryMismatch",
"type": "stat",
"gridPos": {
"h": 8,
@@ -398,7 +398,7 @@
}
},
{
"title": "OTel: NodeStore I/O",
"title": "NodeStore I/O",
"type": "row",
"gridPos": {
"h": 1,
@@ -411,7 +411,7 @@
},
{
"title": "NodeStore Read/Write Totals",
"description": "Cumulative NodeStore read and write operation counts. Sourced from MetricsRegistry nodestore_state observable gauge with metric=node_reads_total, node_writes, node_reads_hit.",
"description": "**What:** Cumulative object-store read, cache-hit, and write operation counts.\n**How it's computed:** Current values of the read, cache-hit, and write counters, plotted as lines.\n**Reading it:** Steadily rising lines are normal; the slope reflects store activity.\n**Healthy range:** Reads and writes rising smoothly, with cache-hits tracking a good fraction of reads.\n**Watch for:** A sudden surge in reads or writes signals heavy back-end I/O, from sync, replay, or query load.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerNodeStoreGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -466,7 +466,7 @@
},
{
"title": "NodeStore Write Load & Read Queue",
"description": "Instantaneous write load score and read queue depth. High write load indicates backend pressure. High read queue indicates prefetch thread saturation.",
"description": "**What:** Instantaneous write-load score and read-queue depth of the object store.\n**How it's computed:** Current values of the write-load and read-queue gauges, plotted over time.\n**Reading it:** Lower is better for both; short, flat lines are healthy.\n**Healthy range:** Write load near zero and read queue in low double digits or less.\n**Watch for:** High write load means back-end pressure; a high read queue means the prefetch threads are saturated.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerNodeStoreGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -530,7 +530,7 @@
},
{
"title": "NodeStore Bytes Read/Written",
"description": "Cumulative bytes read and written by the NodeStore backend. Sourced from MetricsRegistry nodestore_state observable gauge with metric=node_written_bytes, node_read_bytes.",
"description": "**What:** Cumulative bytes read from and written to the object-store back end.\n**How it's computed:** Current values of the bytes-read and bytes-written counters, plotted as lines.\n**Reading it:** Steadily rising lines are normal; the slope shows throughput.\n**Healthy range:** Smooth growth consistent with ledger and query activity.\n**Watch for:** A sharp acceleration indicates a heavy I/O phase such as sync, replay, or large queries.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerNodeStoreGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -578,7 +578,7 @@
},
{
"title": "NodeStore Read Threads & Duration",
"description": "Read thread utilization and cumulative read duration. read_threads_running/read_threads_total shows thread saturation. node_reads_duration_us tracks cumulative time spent in read I/O. read_request_bundle tracks bundled read operations.",
"description": "**What:** Read-thread utilization, bundled read count, and cumulative read time for the object store.\n**How it's computed:** Current values of the running/total read-thread gauges, read-bundle gauge, and cumulative read-duration counter, plotted as lines.\n**Reading it:** Running threads well below total means spare capacity; a rising duration line reflects time spent in read I/O.\n**Healthy range:** Running threads below the total count most of the time.\n**Watch for:** Running threads pinned at the total for long periods means read I/O is saturated.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerNodeStoreGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -619,14 +619,14 @@
"type": "prometheus"
},
"expr": "xrpld_nodestore_state{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", metric=\"node_reads_duration_us\"}",
"legendFormat": "Read Duration (\u00b5s) [{{exported_instance}}]"
"legendFormat": "Read Duration (µs) [{{exported_instance}}]"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": {
"axisLabel": "Count / \u00b5s",
"axisLabel": "Count / µs",
"drawStyle": "line",
"lineWidth": 2,
"fillOpacity": 10
@@ -639,7 +639,7 @@
}
},
{
"title": "OTel: Cache Hit Rates",
"title": "Cache Hit Rates",
"type": "row",
"gridPos": {
"h": 1,
@@ -652,7 +652,7 @@
},
{
"title": "Cache Hit Rates",
"description": "Hit rates for SLE cache, Ledger cache, and AcceptedLedger cache. Values from 0.0 to 1.0. Low values indicate cache thrashing. Sourced from MetricsRegistry cache_metrics observable gauge.",
"description": "**What:** Hit rates for the SLE, Ledger, and AcceptedLedger caches, from 0 to 1.\n**How it's computed:** Current values of the per-cache hit-rate gauges, plotted as lines.\n**Reading it:** Higher is better; each line is the fraction of lookups served from cache.\n**Healthy range:** Above roughly 0.8 in steady state.\n**Watch for:** Low or falling hit rates indicate cache thrashing and extra back-end reads.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerCacheHitRateGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -713,7 +713,7 @@
},
{
"title": "Cache Sizes",
"description": "TreeNode cache size, TreeNode track size, and FullBelow cache size. Sourced from MetricsRegistry cache_metrics observable gauge. Also includes AcceptedLedger (AL) cache size.",
"description": "**What:** Entry counts for the TreeNode cache and track set, the FullBelow cache, and the AcceptedLedger cache.\n**How it's computed:** Current values of the per-cache size gauges, plotted as lines.\n**Reading it:** Stable lines are normal; sizes grow with working set and shrink after sweeps.\n**Healthy range:** Stable within configured limits.\n**Watch for:** Unbounded growth suggests memory pressure or a cache not being swept.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerCacheHitRateGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -774,7 +774,7 @@
}
},
{
"title": "OTel: Object Instance Counts",
"title": "Object Instance Counts",
"type": "row",
"gridPos": {
"h": 1,
@@ -787,7 +787,7 @@
},
{
"title": "Object Instance Counts",
"description": "Live instance counts for key internal object types tracked by CountedObject<T>. Sourced from MetricsRegistry object_count observable gauge. High counts may indicate memory pressure or object leaks.",
"description": "**What:** Live instance counts for the busiest internal object types.\n**How it's computed:** Current per-type instance counts, showing the top 15 types over time.\n**Reading it:** Stable lines are healthy; each line is one object type's live count.\n**Healthy range:** Steady counts that rise and fall with load.\n**Watch for:** A single type climbing without bound suggests memory pressure or a leak.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerObjectCountGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -832,7 +832,7 @@
}
},
{
"title": "OTel: Server Info",
"title": "Server Info",
"type": "row",
"gridPos": {
"h": 1,
@@ -845,7 +845,7 @@
},
{
"title": "Server State",
"description": "Current operating mode: 0=DISCONNECTED, 1=CONNECTED, 2=SYNCING, 3=TRACKING, 4=FULL. Sourced from MetricsRegistry server_info observable gauge.",
"description": "**What:** The node's current operating mode, shown as a labeled state.\n**How it's computed:** Current value of the server-state gauge mapped to a mode name.\n**Reading it:** FULL (green) is the goal; lower states mean the node is not fully participating.\n**Healthy range:** FULL.\n**Watch for:** Sitting in SYNCING, CONNECTED, or DISCONNECTED means the node is not caught up to the network.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerServerInfoGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -925,7 +925,7 @@
},
{
"title": "Uptime",
"description": "Time since server started, in seconds. Sourced from MetricsRegistry server_info observable gauge via UptimeClock.",
"description": "**What:** How long the server process has been running, in seconds.\n**How it's computed:** Current value of the uptime gauge.\n**Reading it:** Higher is generally better; a reset to a small value means the process restarted.\n**Healthy range:** Continuously increasing.\n**Watch for:** An unexpected drop to near zero indicates a restart or crash.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerServerInfoGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -958,7 +958,7 @@
},
{
"title": "Peer Count",
"description": "Total connected peers (inbound + outbound). Sourced from MetricsRegistry server_info observable gauge via overlay().size().",
"description": "**What:** Total connected peers, inbound plus outbound.\n**How it's computed:** Current value of the peer-count gauge.\n**Reading it:** A stable count in the healthy range is good; too few limits connectivity.\n**Healthy range:** Workload- and config-dependent, typically 10 or more.\n**Watch for:** A sudden drop points to network or connectivity problems; an unusually high inbound count can indicate connection-flood pressure.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerServerInfoGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -991,7 +991,7 @@
},
{
"title": "Validated Ledger Seq",
"description": "Sequence number of the most recently validated ledger. Returns 0 before first validation. Sourced from MetricsRegistry server_info observable gauge.",
"description": "**What:** Sequence number of the most recently validated ledger.\n**How it's computed:** Current value of the validated-ledger-sequence gauge.\n**Reading it:** Should climb steadily; reads 0 before the first validation.\n**Healthy range:** Continuously increasing at the network close cadence.\n**Watch for:** A flat sequence means the node has stopped validating new ledgers.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerServerInfoGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -1024,7 +1024,7 @@
},
{
"title": "Last Close Info",
"description": "Proposers and convergence time from the last closed consensus round. Sourced from MetricsRegistry server_info observable gauge via getConsensusInfo().",
"description": "**What:** Proposer count and convergence time from the last closed consensus round.\n**How it's computed:** Current values of the last-close proposer-count and convergence-time gauges, plotted as lines.\n**Reading it:** A healthy proposer count with a low convergence time is good.\n**Healthy range:** Convergence time of a few seconds with the expected number of proposers.\n**Watch for:** A falling proposer count or rising convergence time signals degrading consensus conditions.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerServerInfoGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -1071,7 +1071,7 @@
},
{
"title": "Build Version",
"description": "Build version info metric. Value is always 1; version string is in the 'version' label. Sourced from MetricsRegistry build_info observable gauge.",
"description": "**What:** The running server's build version string.\n**How it's computed:** Read from the version label of the build-info metric (its value is always 1).\n**Reading it:** Confirms which version each node is running.\n**Healthy range:** The expected release version across all nodes.\n**Watch for:** A node on an unexpected or mismatched version in a fleet.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerBuildInfoGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -1105,7 +1105,7 @@
},
{
"title": "Current Ledger Index",
"description": "Current open ledger sequence number. The gap between this and validated_ledger_seq represents ledgers in flight.",
"description": "**What:** Sequence number of the current open ledger.\n**How it's computed:** Current value of the open-ledger index gauge.\n**Reading it:** Should climb steadily; the gap above the validated sequence is the ledgers in flight.\n**Healthy range:** One or two ahead of the validated sequence.\n**Watch for:** A large or growing gap above the validated sequence means validation is lagging behind ledger creation.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerServerInfoGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -1138,7 +1138,7 @@
},
{
"title": "NuDB Storage Size",
"description": "NuDB backend file size in bytes. Sourced from MetricsRegistry storage_detail observable gauge. Tracks database growth over time.",
"description": "**What:** On-disk size of the NuDB object-store back end, in bytes.\n**How it's computed:** Current value of the NuDB storage-size gauge, plotted over time.\n**Reading it:** A smoothly growing line is normal; the slope is the data growth rate.\n**Healthy range:** Gradual growth consistent with retained ledger history.\n**Watch for:** A sudden jump in growth rate can indicate runaway storage or an unexpected back-fill.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerStorageDetailGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -1178,7 +1178,7 @@
}
},
{
"title": "OTel: Complete Ledgers & DB",
"title": "Complete Ledgers & DB",
"type": "row",
"gridPos": {
"h": 1,
@@ -1191,7 +1191,7 @@
},
{
"title": "Complete Ledger Ranges",
"description": "Start and end of each contiguous complete ledger range. Parsed from getLedgerMaster().getCompleteLedgers() string. Sourced from MetricsRegistry complete_ledgers observable gauge.",
"description": "**What:** The contiguous ranges of ledgers the node holds locally.\n**How it's computed:** Current start and end bounds of each complete range, listed as table rows.\n**Reading it:** Fewer ranges is better; one continuous range means an unbroken history.\n**Healthy range:** A single range covering the configured retention window.\n**Watch for:** Many fragmented ranges indicate gaps in stored history from missed or failed fetches.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerCompleteLedgersGauge",
"type": "table",
"gridPos": {
"h": 8,
@@ -1223,7 +1223,7 @@
},
{
"title": "Database Sizes",
"description": "SQLite database sizes in KB (total, ledger, transaction). Sourced from MetricsRegistry db_metrics observable gauge via getRelationalDatabase().getKBUsed*().",
"description": "**What:** Sizes of the relational databases in KB (total, ledger, transaction).\n**How it's computed:** Current values of the per-database size gauges, plotted as lines.\n**Reading it:** Smoothly growing lines are normal; the split shows where storage is used.\n**Healthy range:** Gradual growth consistent with retained history.\n**Watch for:** An abrupt change in growth rate can indicate storage pressure or a pruning issue.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerDbMetricsGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -1278,7 +1278,7 @@
},
{
"title": "Historical Fetch Rate",
"description": "Historical ledger fetches per minute. Sourced from MetricsRegistry db_metrics observable gauge via getInboundLedgers().fetchRate().",
"description": "**What:** How many historical ledgers the node is back-filling per minute.\n**How it's computed:** Current value of the historical-fetch-per-minute gauge.\n**Reading it:** Near zero once history is complete; elevated while back-filling.\n**Healthy range:** Close to zero in steady state.\n**Watch for:** A sustained high rate means the node is still filling gaps in its stored history.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerDbMetricsGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -1311,7 +1311,7 @@
},
{
"title": "Peer Disconnects (Resources)",
"description": "Cumulative count of peer disconnections due to resource limits. Sourced from MetricsRegistry server_info observable gauge via overlay().getPeerDisconnectCharges().",
"description": "**What:** Cumulative count of peers disconnected for exceeding resource limits.\n**How it's computed:** Current value of the resource-disconnect gauge, plotted over time.\n**Reading it:** A flat line is healthy; steps up mean peers were dropped for overuse.\n**Healthy range:** Flat or very slowly rising.\n**Watch for:** A rising line indicates peers are being throttled off, consistent with abusive or misbehaving peers.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerServerInfoGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -1350,7 +1350,7 @@
}
},
{
"title": "OTel: Ledger Economy",
"title": "Ledger Economy",
"type": "row",
"gridPos": {
"h": 1,
@@ -1363,7 +1363,7 @@
},
{
"title": "Base Fee (drops)",
"description": "Current network base transaction fee in drops. Sourced from MetricsRegistry ledger_economy observable gauge. 1 XRP = 1,000,000 drops.",
"description": "**What:** The current network base transaction fee, in drops (1 XRP = 1,000,000 drops).\n**How it's computed:** Current value of the base-fee economy gauge.\n**Reading it:** Stable at the network minimum is normal; rises reflect fee escalation under load.\n**Healthy range:** The network base fee (10 drops on mainnet) in normal conditions.\n**Watch for:** A climbing base fee indicates the network or node is under transaction load pressure.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerLedgerEconomyGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -1396,7 +1396,7 @@
},
{
"title": "Reserve Base (drops)",
"description": "Current account reserve base in drops. The minimum XRP balance required to maintain an account on the ledger.",
"description": "**What:** The minimum XRP balance required to keep an account on the ledger, in drops.\n**How it's computed:** Current value of the reserve-base economy gauge.\n**Reading it:** A stable value is expected; it changes only via network amendment or vote.\n**Healthy range:** The configured network reserve base.\n**Watch for:** An unexpected change outside a known amendment or fee vote.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerLedgerEconomyGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -1429,7 +1429,7 @@
},
{
"title": "Reserve Inc (drops)",
"description": "Current owner reserve increment in drops. Additional XRP required per owned object (trust lines, offers, etc.).",
"description": "**What:** The additional XRP reserve required per owned ledger object, in drops.\n**How it's computed:** Current value of the reserve-increment economy gauge.\n**Reading it:** A stable value is expected; it changes only via network amendment or vote.\n**Healthy range:** The configured network owner reserve increment.\n**Watch for:** An unexpected change outside a known amendment or fee vote.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerLedgerEconomyGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -1462,7 +1462,7 @@
},
{
"title": "Ledger Age",
"description": "Age of the current open ledger in seconds. Values growing beyond the expected ~4s close interval indicate the node is not closing ledgers on schedule.",
"description": "**What:** Age of the current open ledger, in seconds.\n**How it's computed:** Current value of the ledger-age economy gauge.\n**Reading it:** Lower is better; it should stay near the close interval.\n**Healthy range:** Under about 8 seconds.\n**Watch for:** Growth beyond the expected close interval means the node is not closing ledgers on schedule.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerLedgerEconomyGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -1511,7 +1511,7 @@
},
{
"title": "Transaction Rate",
"description": "Current transaction throughput rate from ledger economy metrics. Shows the volume of transactions being processed per interval.",
"description": "**What:** The network transaction throughput reported by the ledger economy metrics.\n**How it's computed:** Current value of the transaction-rate economy gauge, plotted over time.\n**Reading it:** Reflects how many transactions are being processed; higher means busier.\n**Healthy range:** Workload-dependent.\n**Watch for:** A sudden sustained surge can indicate a transaction flood; a drop to zero can indicate the node stopped processing.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerLedgerEconomyGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -1555,7 +1555,7 @@
}
},
{
"title": "Extended Metrics (Recovered from Phase 6)",
"title": "Extended Metrics",
"type": "row",
"gridPos": {
"h": 1,
@@ -1568,7 +1568,7 @@
},
{
"title": "Key Jobs Execution Time",
"description": "Execution time for critical job types at the selected quantile. Sourced from per-job-type events in JobTypeData (JobTypeData.h). Shows how long key consensus, transaction, and maintenance jobs take to execute. Spikes indicate processing bottlenecks.",
"description": "**What:** Execution time for the most critical job types at the selected quantile.\n**How it's computed:** Per-job-type execution durations aggregated to the chosen quantile per node over a 5-minute window.\n**Reading it:** Lower is better; each line is one job type's run time.\n**Healthy range:** A few to tens of milliseconds for most jobs; workload-dependent.\n**Watch for:** Spikes on key jobs (accept, transaction, write) indicate processing bottlenecks.\n**Source:** include/xrpl/core/JobTypeData.h:JobTypeData",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -1677,7 +1677,7 @@
},
{
"title": "Key Jobs Dequeue Wait Time",
"description": "Time spent waiting in the job queue before execution for critical job types. Sourced from per-job-type dequeue events (JobTypeData.h). High dequeue times indicate the job queue is backlogged and jobs are waiting too long to be scheduled.",
"description": "**What:** Time critical jobs wait in the queue before running, at the selected quantile.\n**How it's computed:** Per-job-type queue-wait durations aggregated to the chosen quantile per node over a 5-minute window.\n**Reading it:** Lower is better; each line is one job type's wait time.\n**Healthy range:** Near zero to a few milliseconds when the queue is keeping up.\n**Watch for:** High waits mean the queue is backlogged and jobs are scheduled late.\n**Source:** include/xrpl/core/JobTypeData.h:JobTypeData",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -1786,7 +1786,7 @@
},
{
"title": "FullBelowCache Size",
"description": "Number of entries in the FullBelowCache. Sourced from the TaggedCache size gauge (TaggedCache.h) for the Node family full below cache (NodeFamily.cpp). This cache tracks which SHAMap nodes have all children present locally, avoiding redundant fetches during ledger acquisition.",
"description": "**What:** Number of entries in the FullBelowCache, which tracks subtrees known to be fully present locally.\n**How it's computed:** Current value of the cache size gauge, plotted over time.\n**Reading it:** A stable size is normal; it grows during acquisition and is trimmed by sweeps.\n**Healthy range:** Stable within its configured bound.\n**Watch for:** Unbounded growth suggests the cache is not being swept.\n**Source:** include/xrpl/basics/TaggedCache.h:TaggedCache::Stats",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -1825,7 +1825,7 @@
},
{
"title": "FullBelowCache Hit Rate",
"description": "Hit rate percentage for the FullBelowCache. Sourced from the TaggedCache hit_rate gauge (TaggedCache.h). A high hit rate means the node is efficiently reusing cached knowledge about complete SHAMap subtrees. Low hit rates during steady state warrant investigation.",
"description": "**What:** Hit-rate percentage for the FullBelowCache.\n**How it's computed:** Current value of the cache hit-rate gauge.\n**Reading it:** Higher is better; it shows how often cached subtree knowledge is reused.\n**Healthy range:** Above roughly 50 percent in steady state.\n**Watch for:** A low hit rate during steady state means redundant subtree work and warrants investigation.\n**Source:** include/xrpl/basics/TaggedCache.h:TaggedCache::Stats",
"type": "gauge",
"gridPos": {
"h": 8,
@@ -1875,7 +1875,7 @@
},
{
"title": "Ledger Publish Gap",
"description": "Difference between published and validated ledger ages. Computed as Published_Ledger_Age minus Validated_Ledger_Age. A value near zero means the publish pipeline keeps up with validation. A growing gap indicates the publish pipeline is falling behind, potentially causing stale data for subscribers.",
"description": "**What:** The lag between published and validated ledger ages, in seconds.\n**How it's computed:** Published ledger age minus validated ledger age, as a single derived value.\n**Reading it:** Near zero is healthy; a positive value is how far publishing trails validation.\n**Healthy range:** Close to zero.\n**Watch for:** A growing gap means the publish pipeline is falling behind and subscribers may see stale data.\n**Source:** src/xrpld/app/ledger/LedgerMaster.h:LedgerMaster::Stats::collectMetrics",
"type": "stat",
"gridPos": {
"h": 8,
@@ -1923,7 +1923,7 @@
},
{
"title": "State Duration Rate (Full vs Tracking)",
"description": "Rate of change of time spent in Full and Tracking operating modes, normalized to seconds. Sourced from State_Accounting duration gauges (NetworkOPs.cpp). In steady state the Full duration rate should be close to 1.0 (gaining one second of Full-mode time per wall-clock second). A drop below 1.0 means the node is spending time in other modes.",
"description": "**What:** Rate of change of time spent in Full and Tracking modes, normalized to seconds per second.\n**How it's computed:** Per-second rate of the Full and Tracking duration counters, scaled to seconds.\n**Reading it:** The Full line near 1.0 means the node gains a second of Full-mode time each wall-clock second.\n**Healthy range:** Full rate close to 1.0, Tracking near 0.\n**Watch for:** A Full rate below 1.0 means the node is spending time in other modes.\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp:NetworkOPsImp::Stats",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -1969,7 +1969,7 @@
},
{
"title": "All Jobs Execution Time (Detail)",
"description": "Execution time for ALL non-special job types at the selected quantile. Shows the complete picture of job execution performance. Use the Key Jobs panel for a focused view of the most critical jobs.",
"description": "**What:** Execution time for every non-special job type at the selected quantile.\n**How it's computed:** Per-job-type execution durations aggregated to the chosen quantile per node over a 5-minute window.\n**Reading it:** Lower is better; this is the full picture behind the Key Jobs view.\n**Healthy range:** A few to tens of milliseconds for most jobs.\n**Watch for:** Broad elevation across many job types indicates systemic processing load.\n**Source:** include/xrpl/core/JobTypeData.h:JobTypeData",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -2008,7 +2008,7 @@
},
{
"title": "All Jobs Dequeue Wait (Detail)",
"description": "Dequeue wait time for ALL non-special job types at the selected quantile. Shows the complete picture of job queue waiting times. High wait times across many job types indicate systemic job queue congestion.",
"description": "**What:** Queue wait time for every non-special job type at the selected quantile.\n**How it's computed:** Per-job-type queue-wait durations aggregated to the chosen quantile per node over a 5-minute window.\n**Reading it:** Lower is better; this is the full picture behind the Key Jobs wait view.\n**Healthy range:** Near zero to a few milliseconds when the queue keeps up.\n**Watch for:** High waits across many job types indicate systemic queue congestion.\n**Source:** include/xrpl/core/JobTypeData.h:JobTypeData",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -2047,7 +2047,7 @@
},
{
"title": "Ledger Acquire Duration (Inbound Fetch)",
"description": "p95 duration of the ledger.acquire span (InboundLedger): how long it takes to fetch a missing ledger from peers. A spike signals the node is falling behind or recovering from a fork. Populated under back-fill / sync activity.",
"description": "**What:** Time to fetch a missing ledger from peers, at the 95th percentile.\n**How it's computed:** Per-acquire durations aggregated to their 95th percentile per node over a 5-minute window.\n**Reading it:** Lower is better; populated mainly during sync or back-fill.\n**Healthy range:** Low when synced; higher and more active while catching up.\n**Watch for:** A spike signals the node is falling behind or recovering from a fork.\n**Source:** src/xrpld/app/ledger/detail/InboundLedger.cpp:InboundLedger::init",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -2086,7 +2086,7 @@
},
{
"title": "Ledger Acquire Rate by Outcome",
"description": "Rate of completed ledger.acquire spans broken down by outcome (complete / failed). A rising failed rate indicates the node cannot fetch needed ledgers from its peers. Requires the outcome spanmetrics dimension.",
"description": "**What:** Rate of completed ledger fetches split by outcome (complete or failed).\n**How it's computed:** Per-second rate of finished acquisitions grouped by outcome, per node, over a 5-minute window.\n**Reading it:** Complete should dominate; the failed line should stay near zero.\n**Healthy range:** Complete tracking fetch demand, failed near zero.\n**Watch for:** A rising failed rate means the node cannot fetch needed ledgers from its peers.\n**Source:** src/xrpld/app/ledger/detail/InboundLedger.cpp:InboundLedger::init",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -2125,7 +2125,7 @@
}
],
"schemaVersion": 39,
"tags": ["statsd", "node-health"],
"tags": ["node", "health"],
"templating": {
"list": [
{
@@ -2281,7 +2281,7 @@
"from": "now-1h",
"to": "now"
},
"title": "Node Health (System Metrics)",
"title": "Node Health",
"uid": "xrpld-system-node-health",
"refresh": "5s"
}

25
docker/telemetry/grafana/dashboards/system-overlay-traffic-detail.json
View File

@@ -2,7 +2,7 @@
"annotations": {
"list": []
},
"description": "Detailed overlay traffic breakdown for categories not covered by the main Network Traffic dashboard. Includes squelch, overhead, validator lists, object fetch, ledger sync, and protocol negotiation traffic. Requires [insight] server=otel in rippled config.",
"description": "Fine-grained breakdown of peer-to-peer overlay traffic beyond the main network view: squelch relay control, protocol overhead, validator-list distribution, transaction-set exchange, transaction availability, ledger-proof and replay traffic, and unclassified messages.",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
@@ -11,7 +11,7 @@
"panels": [
{
"title": "Squelch Traffic (Messages)",
"description": "Squelch-related overlay messages. Squelch is the peer traffic management protocol that suppresses redundant message forwarding. 'squelch' = squelch control messages, 'squelch_suppressed' = messages suppressed by squelch, 'squelch_ignored' = squelch directives that were ignored. High suppressed counts indicate effective bandwidth savings; high ignored counts may indicate misconfigured peers. Sourced from TrafficCount.h squelch categories.",
"description": "**What:** Squelch relay-control messages in/out, plus messages suppressed by squelch and squelch directives that were ignored. Squelch reduces redundant message forwarding between peers.\n**How it's computed:** Current message counts for the squelch, squelch-suppressed, and squelch-ignored categories, in and out.\n**Reading it:** High suppressed counts mean squelch is saving bandwidth; ignored should stay low.\n**Healthy range:** workload-dependent; suppressed far above ignored.\n**Watch for:** High ignored counts (peers not honoring squelch) or squelch traffic itself dominating.\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -85,7 +85,7 @@
},
{
"title": "Overhead Traffic Breakdown (Bytes)",
"description": "Overlay protocol overhead by sub-category. 'overhead' = base protocol overhead (ping, status, etc.), 'overhead_cluster' = intra-cluster communication overhead, 'overhead_manifest' = validator manifest distribution overhead. High cluster overhead may indicate frequent cluster state syncs; high manifest overhead occurs during UNL changes. Sourced from TrafficCount.h overhead categories.",
"description": "**What:** Overlay protocol overhead bytes split into base overhead, intra-cluster overhead, and validator-manifest distribution overhead.\n**How it's computed:** Current in/out byte counts for the overhead, overhead-cluster, and overhead-manifest categories.\n**Reading it:** Base overhead is routine; cluster and manifest rise around cluster syncs and manifest changes.\n**Healthy range:** workload-dependent; low and stable.\n**Watch for:** Sustained high cluster or manifest overhead (frequent cluster state churn or manifest reissue).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -159,7 +159,7 @@
},
{
"title": "Validator List Traffic",
"description": "Validator list (UNL) distribution traffic. Validator lists are exchanged when peers share their trusted validator configurations. Spikes occur during UNL updates or when new peers connect. Sourced from TrafficCount.h validator_lists category.",
"description": "**What:** Bytes and messages exchanged distributing validator lists (trusted-list configuration) between peers.\n**How it's computed:** Current in/out byte and message counts for the validator-lists category.\n**Reading it:** Bursts when lists update or new peers connect; quiet otherwise.\n**Healthy range:** workload-dependent; occasional bursts.\n**Watch for:** Continuous high volume (repeated list re-fetching or churn).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -236,7 +236,7 @@
},
{
"title": "Set Get/Share Traffic (Bytes)",
"description": "Transaction set get and share traffic. 'set_get' = requests to fetch transaction sets (sent during ledger close), 'set_share' = responses sharing transaction sets. High set_get traffic indicates peers frequently requesting missing transaction sets, which may signal sync delays. Sourced from TrafficCount.h set_get/set_share categories.",
"description": "**What:** Transaction-set fetch (get) and share bytes exchanged during ledger close.\n**How it's computed:** Current in/out byte counts for the set-get and set-share categories.\n**Reading it:** Some exchange each ledger is normal as peers reconcile transaction sets.\n**Healthy range:** workload-dependent.\n**Watch for:** High set-get (peers frequently missing transaction sets: possible sync delays).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -296,7 +296,7 @@
},
{
"title": "Have/Requested Transactions (Messages)",
"description": "Transaction availability protocol messages. 'have_transactions' = advertisements that a peer has specific transactions available, 'requested_transactions' = explicit requests for transaction data. A high ratio of requested to have may indicate peers are behind on transaction propagation. Sourced from TrafficCount.h have_transactions/requested_transactions categories.",
"description": "**What:** Transaction-availability messages: advertisements that a peer has certain transactions, and explicit requests for transaction data.\n**How it's computed:** Current in/out message counts for the have-transactions and requested-transactions categories.\n**Reading it:** Compare requested versus have to gauge how well transactions are propagating.\n**Healthy range:** workload-dependent.\n**Watch for:** Requested far exceeding have (peers behind on transaction propagation).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -356,7 +356,7 @@
},
{
"title": "Unknown / Unclassified Traffic",
"description": "Traffic that does not match any known overlay message category. Non-zero values may indicate protocol version mismatches, corrupted messages, or new message types not yet classified. Sourced from TrafficCount.h unknown category.",
"description": "**What:** Overlay traffic that matches no known message category, in bytes and messages.\n**How it's computed:** Current in/out byte and message counts for the unknown category.\n**Reading it:** Should be at or near zero.\n**Healthy range:** zero.\n**Watch for:** Any sustained non-zero value (protocol version mismatch, corrupted messages, or an unclassified new message type).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -433,7 +433,7 @@
},
{
"title": "Proof Path Traffic",
"description": "Proof path request/response traffic for ledger state proof exchange. Used by peers to verify specific ledger entries without downloading the full ledger. High request volume may indicate peers validating state during catch-up. Sourced from TrafficCount.h proof_path_request/proof_path_response categories.",
"description": "**What:** Proof-path request/response bytes used to verify individual ledger entries without downloading the whole ledger.\n**How it's computed:** Current in/out byte counts for the proof-path request and response categories.\n**Reading it:** Rises when peers verify specific state, often during catch-up.\n**Healthy range:** workload-dependent.\n**Watch for:** High sustained request volume (heavy state-verification load).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -493,7 +493,7 @@
},
{
"title": "Replay Delta Traffic",
"description": "Replay delta request/response traffic for ledger replay protocol. Used during catch-up to efficiently replay ledger state changes. Sourced from TrafficCount.h replay_delta_request/replay_delta_response categories.",
"description": "**What:** Replay-delta request/response bytes used to efficiently replay ledger state changes during catch-up.\n**How it's computed:** Current in/out byte counts for the replay-delta request and response categories.\n**Reading it:** Active during catch-up and replay; quiet when synced.\n**Healthy range:** workload-dependent; low when synced.\n**Watch for:** Continuous replay traffic (node repeatedly replaying rather than staying current).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -553,7 +553,7 @@
}
],
"schemaVersion": 39,
"tags": ["statsd", "overlay", "network"],
"tags": ["network", "peer"],
"templating": {
"list": [
{
@@ -655,7 +655,6 @@
"from": "now-1h",
"to": "now"
},
"title": "Overlay Traffic Detail (System Metrics)",
"uid": "xrpld-system-overlay-detail",
"refresh": "5s"
"title": "Overlay Traffic Detail",
"uid": "xrpld-system-overlay-detail"
}

36
docker/telemetry/grafana/dashboards/system-rpc-pathfinding.json
View File

@@ -2,7 +2,7 @@
"annotations": {
"list": []
},
"description": "RPC and pathfinding metrics from beast::insight System Metrics. Requires [insight] server=otel in rippled config.",
"description": "RPC request throughput and response timing, pathfinding cost, gRPC read traffic, and resource-manager enforcement for this node.",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
@@ -10,8 +10,8 @@
"links": [],
"panels": [
{
"title": "RPC Request Rate (System Metrics)",
"description": "Rate of RPC requests as counted by the beast::insight counter. Sourced from rpc.requests (ServerHandler.cpp) which increments on every HTTP and WebSocket RPC request. Compare with the span-based rpc.request rate in the RPC Performance dashboard for cross-validation.",
"title": "RPC Request Rate",
"description": "**What:** How many RPC requests the server counts per second.\n**How it's computed:** Per-second rate of the RPC request counter over 5 minutes, per node.\n**Reading it:** A steady line proportional to client demand; cross-checks the trace-based RPC dashboard.\n**Healthy range:** Workload-dependent; tracks client activity.\n**Watch for:** A sudden surge above baseline, consistent with a client flooding the RPC endpoint.\n**Source:** src/xrpld/rpc/detail/ServerHandler.cpp:ServerHandler ctor",
"type": "stat",
"gridPos": {
"h": 8,
@@ -42,8 +42,8 @@
}
},
{
"title": "RPC Response Time (System Metrics)",
"description": "P95 of RPC response time from the beast::insight timer. Sourced from the rpc.time event (ServerHandler.cpp) which records elapsed milliseconds for each RPC response. This measures the full HTTP handler time, not just command execution. Compare with span-based rpc.request duration.",
"title": "RPC Response Time",
"description": "**What:** The 95th-percentile end-to-end RPC response time, including HTTP handling.\n**How it's computed:** 95th-percentile of measured response times over 5 minutes, per node.\n**Reading it:** Lower is better; this is broader than command-only latency.\n**Healthy range:** A few to tens of milliseconds; workload-dependent.\n**Watch for:** A sustained climb, indicating handler overload or slow downstream work.\n**Source:** src/xrpld/rpc/detail/ServerHandler.cpp:ServerHandler ctor",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -82,7 +82,7 @@
},
{
"title": "RPC Response Size",
"description": "P95 of RPC response payload size in bytes. Sourced from the rpc.size event (ServerHandler.cpp) which records the byte length of each RPC JSON response. Large responses may indicate expensive queries (e.g. account_tx with many results) or API misuse.",
"description": "**What:** The 95th-percentile size of RPC response payloads in bytes.\n**How it's computed:** 95th-percentile of response payload sizes over 5 minutes, per node.\n**Reading it:** Smaller is cheaper; large responses cost bandwidth and memory.\n**Healthy range:** Workload-dependent; small for status queries, large for bulk data queries.\n**Watch for:** Growth in large responses, consistent with expensive queries or API misuse.\n**Source:** src/xrpld/rpc/detail/ServerHandler.cpp:ServerHandler ctor",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -121,7 +121,7 @@
},
{
"title": "RPC Response Time Distribution",
"description": "Distribution of RPC response times from the beast::insight timer showing P90, P95, and P99 quantiles. Sourced from the rpc.time event (ServerHandler.cpp). Useful for detecting bimodal latency or long-tail requests.",
"description": "**What:** RPC response time at the 90th, 95th, and 99th percentiles together.\n**How it's computed:** Three response-time quantiles over 5 minutes, per node.\n**Reading it:** Lower is better; the gap between P90 and P99 shows the long tail.\n**Healthy range:** Percentiles clustered low; workload-dependent.\n**Watch for:** A widening P99, revealing long-tail or bimodal slow requests.\n**Source:** src/xrpld/rpc/detail/ServerHandler.cpp:ServerHandler ctor",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -174,7 +174,7 @@
},
{
"title": "Pathfinding Fast Duration",
"description": "P95 of fast pathfinding execution time. Sourced from the pathfind_fast event (PathRequests.h) which records the duration of the fast pathfinding algorithm. Fast pathfinding uses a simplified search that trades accuracy for speed.",
"description": "**What:** The 95th-percentile time of the fast pathfinding search.\n**How it's computed:** 95th-percentile of fast pathfinding durations over 5 minutes, per node.\n**Reading it:** Lower is better; fast mode trades accuracy for speed.\n**Healthy range:** Typically a few to tens of milliseconds; workload-dependent.\n**Watch for:** A rising trend, indicating heavy pathfinding demand or an expensive order book.\n**Source:** src/xrpld/rpc/detail/PathRequestManager.h:ctor",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -213,7 +213,7 @@
},
{
"title": "Pathfinding Full Duration",
"description": "P95 of full pathfinding execution time. Sourced from the pathfind_full event (PathRequests.h) which records the duration of the exhaustive pathfinding search. Full pathfinding is more expensive and can take significantly longer than fast mode.",
"description": "**What:** The 95th-percentile time of the full, exhaustive pathfinding search.\n**How it's computed:** 95th-percentile of full pathfinding durations over 5 minutes, per node.\n**Reading it:** Lower is better; full mode is much more expensive than fast mode.\n**Healthy range:** Tens to hundreds of milliseconds; workload-dependent.\n**Watch for:** Sustained high durations, consistent with pathfinding-heavy clients straining the node.\n**Source:** src/xrpld/rpc/detail/PathRequestManager.h:ctor",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -252,7 +252,7 @@
},
{
"title": "Resource Warnings Rate",
"description": "Rate of resource warning events from the Resource Manager. Sourced from the warn meter (Logic.h) which increments when a consumer (peer or RPC client) exceeds the warning threshold for resource usage. A rising rate indicates aggressive clients that may need throttling. NOTE: This panel will show no data until the |m -> |c fix is applied in System MetricsCollector.cpp (Phase 6 Task 6.1).",
"description": "**What:** How often the resource manager warns a peer or client for excessive usage, per second.\n**How it's computed:** Per-second rate of resource-warning events over 5 minutes, per node.\n**Reading it:** Green near zero, yellow above 0.1/sec, red above 1/sec.\n**Healthy range:** Near zero in normal operation.\n**Watch for:** A rising rate, consistent with aggressive clients that may need throttling.\n**Source:** include/xrpl/resource/detail/Logic.h:Logic::Stats ctor",
"type": "stat",
"gridPos": {
"h": 8,
@@ -300,7 +300,7 @@
},
{
"title": "Resource Drops Rate",
"description": "Rate of resource drop events from the Resource Manager. Sourced from the drop meter (Logic.h) which increments when a consumer is disconnected or blocked due to excessive resource usage. Non-zero values mean the node is actively rejecting abusive connections. NOTE: This panel will show no data until the |m -> |c fix is applied in System MetricsCollector.cpp (Phase 6 Task 6.1).",
"description": "**What:** How often the resource manager drops or blocks a peer or client for abuse, per second.\n**How it's computed:** Per-second rate of resource-drop events over 5 minutes, per node.\n**Reading it:** Green near zero, yellow above 0.01/sec, red above 0.1/sec.\n**Healthy range:** Zero when no abusive consumers are present.\n**Watch for:** Non-zero values, meaning the node is actively rejecting abusive connections.\n**Source:** include/xrpl/resource/detail/Logic.h:Logic::Stats ctor",
"type": "stat",
"gridPos": {
"h": 8,
@@ -348,7 +348,7 @@
},
{
"title": "gRPC Request Rate by Method (Spans)",
"description": "Per-method gRPC call rate derived from the grpc.{Method} spans (GRPCServer.cpp). Covers the gRPC API used by reporting/Clio. Populated only when the node serves gRPC traffic.",
"description": "**What:** How many gRPC calls of each method the node serves per second.\n**How it's computed:** Per-second rate of gRPC calls over 5 minutes, grouped by method and node.\n**Reading it:** Non-zero only when reporting/Clio-style clients use the gRPC API.\n**Healthy range:** Workload-dependent; zero without gRPC traffic.\n**Watch for:** A single method spiking, consistent with a heavy or misbehaving gRPC consumer.\n**Source:** src/xrpld/app/main/GRPCServer.cpp:GRPCServerImpl::CallData::process",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -387,7 +387,7 @@
},
{
"title": "gRPC Latency P95 by Method (Spans)",
"description": "p95 latency per gRPC method from grpc.{Method} span durations. Identifies slow gRPC read paths.",
"description": "**What:** The 95th-percentile latency of each gRPC method.\n**How it's computed:** 95th-percentile of gRPC call durations over 5 minutes, grouped by method and node.\n**Reading it:** Lower is better; identifies slow gRPC read paths.\n**Healthy range:** Workload-dependent; scales with ledger data volume served.\n**Watch for:** A method whose latency climbs, indicating an expensive read path.\n**Source:** src/xrpld/app/main/GRPCServer.cpp:GRPCServerImpl::CallData::process",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -426,7 +426,7 @@
},
{
"title": "gRPC Error Rate by Status (Spans)",
"description": "Rate of gRPC spans broken down by grpc_status (success/error/resource_exhausted/failed_precondition). A rising error or resource_exhausted rate indicates gRPC clients hitting limits.",
"description": "**What:** gRPC call rate broken down by outcome status (success, error, resource-exhausted, failed-precondition).\n**How it's computed:** Per-second rate of gRPC calls over 5 minutes, grouped by status and node.\n**Reading it:** Success should dominate; error and resource-exhausted should stay low.\n**Healthy range:** Workload-dependent; errors near the floor.\n**Watch for:** A rising error or resource-exhausted rate, indicating gRPC clients hitting limits.\n**Source:** src/xrpld/app/main/GRPCServer.cpp:GRPCServerImpl::CallData::process",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -465,7 +465,7 @@
},
{
"title": "Pathfinding Compute Duration (Spans)",
"description": "p95 of the pathfind.compute span, the per-request path computation. Complements the StatsD pathfind_fast/full timers with span-level visibility. Populated under pathfinding (book/path) RPC load.",
"description": "**What:** The 95th-percentile time to compute paths for a single request.\n**How it's computed:** 95th-percentile of path-computation durations over 5 minutes, per node.\n**Reading it:** Lower is better; complements the fast/full pathfinding timers with span-level detail.\n**Healthy range:** Workload-dependent; grows with order-book depth and request complexity.\n**Watch for:** A rising trend under pathfinding load, indicating expensive path computation.\n**Source:** src/xrpld/rpc/detail/PathRequest.cpp:PathRequest::doUpdate",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -504,7 +504,7 @@
},
{
"title": "Pathfinding Request & Discovery Rate (Spans)",
"description": "Rate of pathfind.request (client path requests) and pathfind.discover (path-discovery passes) spans. Shows pathfinding demand and the discovery cost driver for subscription-heavy nodes.",
"description": "**What:** The rate of client path requests versus path-discovery passes.\n**How it's computed:** Per-second rates of path requests and discovery passes over 5 minutes, per node.\n**Reading it:** Discovery cost tracks request demand; useful for subscription-heavy nodes.\n**Healthy range:** Workload-dependent; both zero without pathfinding clients.\n**Watch for:** Discovery rate climbing out of proportion to requests, a cost driver for subscription-heavy nodes.\n**Source:** src/xrpld/rpc/handlers/orderbook/PathFind.cpp:doPathFind; src/xrpld/rpc/detail/PathRequest.cpp:PathRequest::findPaths",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -550,7 +550,7 @@
}
],
"schemaVersion": 39,
"tags": ["statsd", "rpc", "pathfinding"],
"tags": ["rpc", "pathfinding"],
"templating": {
"list": [
{
@@ -672,7 +672,7 @@
"from": "now-1h",
"to": "now"
},
"title": "RPC & Pathfinding (System Metrics)",
"title": "RPC & Pathfinding",
"uid": "xrpld-system-rpc",
"refresh": "5s"
}

45
docker/telemetry/grafana/dashboards/transaction-overview.json
View File

@@ -10,7 +10,7 @@
"panels": [
{
"title": "Transaction Apply Failed Rate",
"description": "Rate of tx.apply spans completing with error status, indicating transaction application failures during ledger building. The span records tx_failed as an attribute. Thresholds: green < 0.1/sec, yellow 0.1-1/sec, red > 1/sec. Some failures are normal (e.g. conflicting offers) but sustained high rates may indicate issues.",
"description": "**What:** How many transactions fail during application into a ledger, per second.\n**How it's computed:** Per-second rate of errored transaction-apply operations over 5 minutes, per node.\n**Reading it:** Green under 0.1/sec, yellow 0.1-1/sec, red above 1/sec.\n**Healthy range:** Some failures are normal (e.g. conflicting offers); a low steady rate is fine.\n**Watch for:** A sustained red rate, which can indicate a problem in transaction processing or a spam wave.\n**Source:** src/xrpld/app/ledger/detail/BuildLedger.cpp:applyTransactions",
"type": "stat",
"gridPos": {
"h": 4,
@@ -60,7 +60,7 @@
},
{
"title": "Transaction Processing Latency by Type",
"description": "Per-transaction-type processing latency (p95). Filter with $tx_type variable above.",
"description": "**What:** The 95th-percentile time to process a transaction, broken down by transaction type.\n**How it's computed:** 95th-percentile of processing durations over 5 minutes, grouped by transaction type and node.\n**Reading it:** Lower is better; compare types to find the expensive ones.\n**Healthy range:** A few milliseconds for simple payments; workload- and type-dependent.\n**Watch for:** A type whose latency climbs, pointing to expensive processing or resource pressure.\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp:NetworkOPsImp::processTransaction",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -106,7 +106,7 @@
},
{
"title": "Transaction Rate by Type",
"description": "Transaction processing rate broken down by tx_type (Payment, OfferCreate, AMMDeposit, etc.). Requires tx_type dimension in spanmetrics.",
"description": "**What:** How many transactions of each type enter processing per second.\n**How it's computed:** Per-second processing rate over 5 minutes, grouped by transaction type and node.\n**Reading it:** Shows the transaction mix; payments usually lead.\n**Healthy range:** Workload-dependent; mix reflects network activity.\n**Watch for:** A single type spiking far above baseline, consistent with a spam campaign of that type.\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp:NetworkOPsImp::processTransaction",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -152,7 +152,7 @@
},
{
"title": "Transaction Results by Type",
"description": "Transaction result codes (ter_result) broken down by tx_type. Shows which transaction types fail most often.",
"description": "**What:** The non-success result codes transactions return, by transaction type.\n**How it's computed:** Per-second rate of failing transactions over 5 minutes, grouped by type and result code.\n**Reading it:** Fewer failures is better; use it to see which types fail and why.\n**Healthy range:** Workload-dependent; a modest background of expected failures is normal.\n**Watch for:** A surge of one failure code for one type, indicating a systemic issue or abusive submissions.\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp:NetworkOPsImp::processTransaction",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -198,7 +198,7 @@
},
{
"title": "Transaction Receive vs Suppressed",
"description": "Total rate of raw transaction messages received from peers (tx.receive span from PeerImp.cpp). This fires before deduplication via the HashRouter, so the difference between tx.receive and tx.process reflects suppressed duplicate transactions.",
"description": "**What:** Raw transactions received from peers, split by whether they were duplicates suppressed before processing.\n**How it's computed:** Per-second rate of received transactions over the window, split by suppressed flag and node.\n**Reading it:** A large suppressed share is normal, since the same transaction arrives from many peers.\n**Healthy range:** Workload-dependent; suppressed typically exceeds newly processed.\n**Watch for:** A collapse in suppression (duplicate filtering failing) or an overall receive flood.\n**Source:** src/xrpld/overlay/detail/PeerImp.cpp:PeerImp::handleTransaction",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -239,7 +239,7 @@
},
{
"title": "Transaction Processing Rate",
"description": "Rate of transactions entering the processing pipeline. tx.process (NetworkOPs.cpp) fires when a transaction is submitted locally or received from a peer and enters processTransaction(). tx.receive (PeerImp.cpp) fires when a raw transaction message arrives from a peer before deduplication.",
"description": "**What:** Transactions entering the processing pipeline versus raw transactions arriving from peers.\n**How it's computed:** Per-second rates of processed transactions and received peer transactions over 5 minutes, per node.\n**Reading it:** Received sits above processed; the difference is deduplicated relay traffic.\n**Healthy range:** Workload-dependent; both scale with network volume.\n**Watch for:** A receive rate spiking well above processing, consistent with relay flooding.\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp:NetworkOPsImp::processTransaction; src/xrpld/overlay/detail/PeerImp.cpp:PeerImp::handleTransaction",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -287,7 +287,7 @@
},
{
"title": "Transaction Path Distribution",
"description": "Breakdown of transactions by origin path. The local attribute indicates whether the transaction was submitted locally (true) or received from a peer (false). Helps understand the ratio of locally-originated vs relayed transactions.",
"description": "**What:** The split of transactions by origin: submitted locally versus relayed from peers.\n**How it's computed:** Processed-transaction rate over 5 minutes, split by local-origin flag and shown as proportions.\n**Reading it:** Most traffic on a network node is peer-relayed; local dominates on a submission node.\n**Healthy range:** Workload-dependent on the node's role.\n**Watch for:** An unexpected surge in local submissions, consistent with a local client flooding the node.\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp:NetworkOPsImp::processTransaction",
"type": "piechart",
"gridPos": {
"h": 8,
@@ -315,7 +315,7 @@
},
{
"title": "Transaction Processing Duration Heatmap",
"description": "Heatmap showing the distribution of tx.process span durations across histogram buckets over time. Each cell represents the count of transactions that completed within that latency bucket in a 5m window. Reveals whether processing times are consistent or exhibit multi-modal patterns.",
"description": "**What:** The full distribution of transaction processing times over the window.\n**How it's computed:** Counts of transactions in each latency band per 5-minute window, shown as color density.\n**Reading it:** A tight low band is healthy; spread bands mean uneven processing time.\n**Healthy range:** Most mass in the low-latency bands; workload-dependent.\n**Watch for:** A distinct slow cluster (multi-modal timing) hidden by percentile charts.\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp:NetworkOPsImp::processTransaction",
"type": "heatmap",
"gridPos": {
"h": 8,
@@ -353,7 +353,7 @@
},
{
"title": "Peer Transaction Receive Rate",
"description": "Rate of transaction messages received from network peers. Sourced from the tx.receive span (PeerImp.cpp) which fires in the onMessage(TMTransaction) handler. High rates may indicate network-wide transaction volume spikes or peer flooding.",
"description": "**What:** How many transaction messages arrive from network peers per second.\n**How it's computed:** Per-second rate of received peer transactions over 5 minutes, per node.\n**Reading it:** Reflects network-wide transaction volume reaching this node.\n**Healthy range:** Workload-dependent; tracks overall network activity.\n**Watch for:** A sharp spike above baseline, consistent with network-wide volume surges or peer flooding.\n**Source:** src/xrpld/overlay/detail/PeerImp.cpp:PeerImp::handleTransaction",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -394,7 +394,7 @@
},
{
"title": "Transactor Duration by Type (p95)",
"description": "Per-transactor execution time (tx.transactor span). Shows which transaction types are most expensive to execute.",
"description": "**What:** The 95th-percentile execution time of the core transactor step, by transaction type.\n**How it's computed:** 95th-percentile of transactor durations over 5 minutes, grouped by type and node.\n**Reading it:** Lower is better; reveals which types are most expensive to execute.\n**Healthy range:** Sub-millisecond to a few ms for most types; workload-dependent.\n**Watch for:** A type whose execution time grows, indicating expensive logic or ledger contention.\n**Source:** src/libxrpl/tx/Transactor.cpp:Transactor::operator()",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -440,7 +440,7 @@
},
{
"title": "Tx Apply Pipeline Rate by Stage",
"description": "Span rate for each apply-pipeline stage (preflight, preclaim, apply). A drop between stages shows where transactions are filtered out. Requires the stage dimension in spanmetrics.",
"description": "**What:** How many transactions pass through each apply-pipeline stage (preflight, preclaim, apply) per second.\n**How it's computed:** Per-second rate over 5 minutes, grouped by pipeline stage and node.\n**Reading it:** Stages should step down slightly as invalid transactions are filtered out.\n**Healthy range:** Workload-dependent; preflight highest, apply lowest.\n**Watch for:** A large drop between stages, showing where most transactions are being rejected.\n**Source:** src/libxrpl/tx/applySteps.cpp:makeStageSpan; src/libxrpl/tx/Transactor.cpp:Transactor::operator()",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -486,7 +486,7 @@
},
{
"title": "Tx Apply Pipeline Latency by Stage (p95)",
"description": "95th-percentile duration of each apply-pipeline stage. Isolates which stage (preflight, preclaim, apply) dominates transaction processing time.",
"description": "**What:** The 95th-percentile duration of each apply-pipeline stage.\n**How it's computed:** 95th-percentile of per-stage durations over 5 minutes, grouped by stage and node.\n**Reading it:** Lower is better; isolates which stage dominates processing time.\n**Healthy range:** Each stage typically sub-millisecond to a few ms; workload-dependent.\n**Watch for:** One stage's latency rising, pinpointing the bottleneck in transaction handling.\n**Source:** src/libxrpl/tx/applySteps.cpp:makeStageSpan; src/libxrpl/tx/Transactor.cpp:Transactor::operator()",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -532,7 +532,7 @@
},
{
"title": "Tx Apply Pipeline Failure Rate by Stage",
"description": "Rate of apply-pipeline spans whose ter_result is not tesSUCCESS, split by stage. Shows whether failures concentrate in preflight, preclaim, or apply. Filters on ter_result rather than span status because a failing ter code completes the span normally; only thrown exceptions set an error status.",
"description": "**What:** How many transactions fail at each apply-pipeline stage per second.\n**How it's computed:** Per-second rate of non-success outcomes over 5 minutes, grouped by stage and node.\n**Reading it:** Shows whether rejections concentrate in preflight, preclaim, or apply.\n**Healthy range:** Workload-dependent; a modest background of expected rejections is normal.\n**Watch for:** A failure spike concentrated in one stage, consistent with malformed or spam submissions.\n**Source:** src/libxrpl/tx/applySteps.cpp:makeStageSpan; src/libxrpl/tx/Transactor.cpp:Transactor::operator()",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -578,7 +578,7 @@
},
{
"title": "Tx Apply Pipeline Latency by Type and Stage (p95)",
"description": "95th-percentile duration broken down by both transaction type and apply-pipeline stage. Shows, for each transaction type, which stage (preflight, preclaim, apply) dominates its latency. Higher cardinality than the by-stage view; filter with the $tx_type and $stage variables.",
"description": "**What:** The 95th-percentile stage duration broken down by both transaction type and pipeline stage.\n**How it's computed:** 95th-percentile durations over 5 minutes, grouped by type and stage; higher cardinality than the by-stage view.\n**Reading it:** Lower is better; shows which stage dominates each type's latency.\n**Healthy range:** Workload-dependent; most type/stage pairs sub-millisecond to a few ms.\n**Watch for:** A specific type/stage combination rising sharply, pinpointing a costly path.\n**Source:** src/libxrpl/tx/applySteps.cpp:makeStageSpan; src/libxrpl/tx/Transactor.cpp:Transactor::operator()",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -624,7 +624,7 @@
},
{
"title": "Transaction Apply Duration per Ledger",
"description": "p95 latency of applying the consensus transaction set to a new ledger. The tx.apply span (BuildLedger.cpp) wraps the applyTransactions() function that iterates through the CanonicalTXSet and applies each transaction to the OpenView. Long durations indicate heavy transaction sets or expensive transaction processing.",
"description": "**What:** The 95th-percentile time to apply the agreed transaction set into each new ledger.\n**How it's computed:** 95th-percentile of transaction-apply durations over 5 minutes, per node.\n**Reading it:** Lower is better; a major component of ledger build time.\n**Healthy range:** A few to tens of milliseconds; scales with transactions per ledger.\n**Watch for:** Rising durations during heavy or expensive transaction sets.\n**Source:** src/xrpld/app/ledger/detail/BuildLedger.cpp:applyTransactions",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -665,7 +665,7 @@
},
{
"title": "TxQ Enqueue Rate by Transaction Type",
"description": "Rate of txq.enqueue spans broken down by transaction type (tx_type). Shows what share of inbound demand is Payment vs OfferCreate vs other transactors, and how the mix shifts as the queue fills. A spam burst of one type is a leading indicator of fee escalation.",
"description": "**What:** How many transactions are queued (rather than applied immediately) per second, by type.\n**How it's computed:** Per-second rate of queue-enqueue operations over 5 minutes, grouped by transaction type and node.\n**Reading it:** Shows the demand mix hitting the queue and how it shifts as the queue fills.\n**Healthy range:** Workload-dependent; low or zero when the network is not congested.\n**Watch for:** A spam burst of one type, an early indicator of fee escalation.\n**Source:** src/xrpld/app/misc/detail/TxQ.cpp:TxQ::apply",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -706,7 +706,7 @@
},
{
"title": "TxQ Accept Status",
"description": "TxQ accept outcomes: applied (included in ledger), failed (removed), retried (kept for next round).",
"description": "**What:** The outcome of queued transactions when the queue drains into a ledger: applied, failed, or retried.\n**How it's computed:** Counts of queue-accept outcomes over the window, split by status and shown as proportions.\n**Reading it:** A high applied share is healthy; retries indicate the transaction stayed queued.\n**Healthy range:** Workload-dependent; applied should dominate when not congested.\n**Watch for:** A rising failed or retried share, indicating queue pressure or under-bidding.\n**Source:** src/xrpld/app/misc/detail/TxQ.cpp:TxQ::accept",
"type": "piechart",
"gridPos": {
"h": 8,
@@ -744,7 +744,7 @@
},
{
"title": "Queue Bypass Ratio (Direct Apply vs Enqueue)",
"description": "Ratio of transactions that applied directly to the open ledger (txq.apply_direct) versus those that had to be queued (txq.enqueue). A falling bypass ratio is the cleanest single signal the network has entered sustained fee escalation.",
"description": "**What:** The fraction of transactions that applied straight to the open ledger instead of being queued.\n**How it's computed:** Direct-apply rate divided by direct-apply plus enqueue rate over 5 minutes, per node.\n**Reading it:** Higher is better; near 1.0 means the network is not congested.\n**Healthy range:** Close to 1.0 when not congested; falls as demand exceeds capacity.\n**Watch for:** A falling ratio, the cleanest single signal the network has entered sustained fee escalation.\n**Source:** src/xrpld/app/misc/detail/TxQ.cpp:TxQ::tryDirectApply; src/xrpld/app/misc/detail/TxQ.cpp:TxQ::apply",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -785,7 +785,7 @@
},
{
"title": "Queue Accept (Drain) Duration per Ledger",
"description": "p95 duration of the txq.accept span, which drains queued transactions into a newly closed ledger. Rising drain time signals queue pressure at ledger close.",
"description": "**What:** The 95th-percentile time to drain queued transactions into a newly closed ledger.\n**How it's computed:** 95th-percentile of queue-drain durations over 5 minutes, per node.\n**Reading it:** Lower is better; rising time signals queue pressure at ledger close.\n**Healthy range:** A few milliseconds when the queue is light; workload-dependent.\n**Watch for:** A sustained climb, indicating a large or contended queue slowing ledger close.\n**Source:** src/xrpld/app/misc/detail/TxQ.cpp:TxQ::accept",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -826,7 +826,7 @@
},
{
"title": "Queue Cleanup Rate (Expired Entries)",
"description": "Rate of txq.cleanup spans, which remove expired transactions from the queue each ledger. A rising rate means submitters under-bid the escalating fee and abandoned their transactions \u2014 a demand-frustration signal distinct from acceptance throughput.",
"description": "**What:** How often expired transactions are removed from the queue each ledger, per second.\n**How it's computed:** Per-second rate of queue-cleanup operations over 5 minutes, per node.\n**Reading it:** Low is healthy; a rising rate means submitters are abandoning transactions.\n**Healthy range:** Near zero when not congested; workload-dependent.\n**Watch for:** A rising rate, a demand-frustration signal where submitters under-bid the escalating fee and gave up.\n**Source:** src/xrpld/app/misc/detail/TxQ.cpp:TxQ::processClosedLedger",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -946,7 +946,7 @@
{
"name": "node",
"label": "Node",
"description": "Filter by rippled node (service.instance.id \u2014 e.g. Node-1)",
"description": "Filter by rippled node (service.instance.id — e.g. Node-1)",
"type": "query",
"query": "label_values(traces_span_metrics_calls_total, exported_instance)",
"datasource": {
@@ -1063,5 +1063,6 @@
},
"title": "Transaction Overview",
"uid": "xrpld-transactions",
"refresh": "5s"
"refresh": "5s",
"description": "Transaction flow through this node: receipt, processing, results, per-stage timing, and queue behavior."
}

20
docker/telemetry/grafana/dashboards/xrpld-fee-market.json
View File

@@ -2,7 +2,7 @@
"annotations": {
"list": []
},
"description": "Fee market dynamics: TxQ depth/capacity, fee escalation levels, and load factor breakdown. Sourced from OTel MetricsRegistry observable gauges (Phase 9).",
"description": "Transaction queue depth and capacity, fee-escalation levels, and load-factor breakdown that together set the minimum fee to enter the open ledger.",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
@@ -11,7 +11,7 @@
"panels": [
{
"title": "Transaction Queue Depth",
"description": "Current number of transactions waiting in the queue vs. maximum capacity. Sourced from MetricsRegistry txq_metrics observable gauge with metric=txq_count and metric=txq_max_size.",
"description": "**What:** Transactions currently waiting in the transaction queue versus the queue's maximum capacity.\n**How it's computed:** Instantaneous gauge readings of current queue count and configured max size.\n**Reading it:** Queue depth well below capacity is normal; depth approaching capacity means the node is saturating.\n**Healthy range:** Depth near 0 in quiet periods; workload-dependent under load.\n**Watch for:** Depth pinned at capacity for sustained periods, which signals demand exceeding throughput or a fee-spam burst.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerTxqGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -59,7 +59,7 @@
},
{
"title": "Transactions Per Ledger",
"description": "Transactions in the current open ledger vs. expected per-ledger count. Sourced from txq_metrics with metric=txq_in_ledger and metric=txq_per_ledger.",
"description": "**What:** Transactions already placed in the current open ledger versus the expected per-ledger target.\n**How it's computed:** Instantaneous gauge readings of in-ledger count and the target count that governs fee escalation.\n**Reading it:** Staying at or below the expected target is normal; exceeding it triggers open-ledger fee escalation.\n**Healthy range:** At or under the expected per-ledger target.\n**Watch for:** In-ledger count persistently above target, indicating sustained congestion pushing fees up.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerTxqGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -107,7 +107,7 @@
},
{
"title": "Fee Escalation Levels",
"description": "Fee levels that control transaction queue admission. Reference fee level is the baseline; open ledger fee level triggers escalation. Sourced from txq_metrics observable gauge.",
"description": "**What:** The fee levels that govern queue admission: reference (baseline), minimum processing, median, and open-ledger levels.\n**How it's computed:** Instantaneous gauge readings of each fee level, shown on a log scale.\n**Reading it:** Open-ledger level near the reference level means cheap entry; a large gap above reference means escalation is active.\n**Healthy range:** Open-ledger level at or near reference during normal traffic.\n**Watch for:** Open-ledger level spiking far above reference, the hallmark of congestion or a fee-bidding war.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerTxqGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -173,7 +173,7 @@
},
{
"title": "Load Factor Breakdown",
"description": "Decomposed load factor components: server (max of local, net, cluster), fee escalation, fee queue, and combined. Values are unitless multipliers where 1.0 = no load. Sourced from load_factor_metrics observable gauge.",
"description": "**What:** The combined load factor and its server, fee-escalation, and fee-queue contributors as unitless fee multipliers (1.0 = no load).\n**How it's computed:** Instantaneous gauge readings of each load-factor component.\n**Reading it:** Values at 1.0 mean base fees; higher values raise the fee to transact.\n**Healthy range:** Around 1.0 under normal conditions.\n**Watch for:** Combined factor climbing well above 1.0, showing the node is charging premium fees due to congestion or overload.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerLoadFactorGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -251,7 +251,7 @@
},
{
"title": "Load Factor Components",
"description": "Individual load factor contributors: local server load, network load, and cluster load. Only differ from 1.0 under load conditions. Sourced from load_factor_metrics observable gauge.",
"description": "**What:** The individual load-factor inputs, local server load, network load, and cluster load, as unitless multipliers.\n**How it's computed:** Instantaneous gauge readings of each component.\n**Reading it:** All at 1.0 means no load pressure from any source; a raised component identifies where load originates.\n**Healthy range:** Around 1.0 for each component.\n**Watch for:** A single component rising sharply, which pinpoints whether the pressure is local, network-wide, or cluster-driven.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerLoadFactorGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -306,7 +306,7 @@
},
{
"title": "Load Factor Attribution (Stacked Components)",
"description": "Stacked contribution of each load-factor component (fee escalation, queue, local, net, cluster) to the effective transaction cost. Shows WHICH component is driving the fee at any moment, which the aggregate load_factor hides.",
"description": "**What:** The stacked contribution of each load-factor component (fee escalation, queue, local, net, cluster) to the effective transaction cost.\n**How it's computed:** Instantaneous gauge readings of each component stacked to show relative share.\n**Reading it:** The tallest band identifies which component is driving the fee, detail the aggregate load factor hides.\n**Healthy range:** Thin bands near 1.0 total under normal traffic.\n**Watch for:** One band dominating during a fee spike, revealing the specific driver of elevated cost.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerLoadFactorGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -378,7 +378,7 @@
},
{
"title": "Queue Abandonment Rate (Expired)",
"description": "Rate of transactions expired out of the queue (LastLedgerSequence passed). Rising expiry means submitters under-bid the escalating fee and gave up \u2014 a demand-frustration signal.",
"description": "**What:** Transactions dropped from the queue because their last-ledger deadline passed before they could be included.\n**How it's computed:** Per-second rate of the cumulative expired-transaction counter over a 5-minute window.\n**Reading it:** Near zero is healthy; a rising rate means submitters under-bid the escalating fee and their transactions timed out.\n**Healthy range:** Near 0 expirations per second.\n**Watch for:** Sustained expiry rate, a demand-frustration signal often coinciding with fee spikes or spam that crowds out honest traffic.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:incrementTxqExpired (caller src/xrpld/app/misc/detail/TxQ.cpp)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -417,7 +417,7 @@
},
{
"title": "Queue Admission Rejections (Dropped)",
"description": "Rate of transactions refused admission to the queue, by reason. queue_full means the queue is at capacity \u2014 admission-control backpressure distinct from expiry and from job-queue overflow.",
"description": "**What:** Transactions refused entry to the queue, broken down by reason such as queue_full.\n**How it's computed:** Per-second rate of the cumulative dropped-transaction counter over a 5-minute window, split by reason.\n**Reading it:** Near zero is healthy; queue_full rejections mean the queue is at capacity and applying backpressure.\n**Healthy range:** Near 0 rejections per second.\n**Watch for:** A burst of queue_full drops, distinct from expiry, indicating the node is being flooded faster than it can drain.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:incrementTxqDropped (caller src/xrpld/app/misc/detail/TxQ.cpp)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -456,7 +456,7 @@
}
],
"schemaVersion": 39,
"tags": ["fee-market"],
"tags": ["transactions", "fees"],
"templating": {
"list": [
{

34
docker/telemetry/grafana/dashboards/xrpld-job-queue.json
View File

@@ -2,7 +2,7 @@
"annotations": {
"list": []
},
"description": "Job queue analysis: per-job-type throughput rates, queue wait times, and execution times. Sourced from OTel MetricsRegistry synchronous counters and histograms (Phase 9).",
"description": "Per-job-type throughput, queue wait times, and execution times for the node's internal worker job queue.",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
@@ -11,7 +11,7 @@
"panels": [
{
"title": "Current Job Latency (p99 Gauge)",
"description": "At-a-glance p99 job queue wait and execution time over the last 5 minutes. Green < 100ms, yellow 100ms-1s, red > 1s.",
"description": "**What:** At-a-glance p99 of how long jobs wait in the queue and how long they run once started.\n**How it's computed:** 99th percentile derived from the job wait-time and run-time histograms over the last 5 minutes.\n**Reading it:** Lower is better; green under 100ms, yellow to 1s, red beyond 1s.\n**Healthy range:** Wait and exec p99 under 100ms.\n**Watch for:** p99 wait climbing into the red, meaning worker threads are saturated and jobs are backing up.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:recordJobStarted / recordJobFinished",
"type": "gauge",
"gridPos": {
"h": 8,
@@ -47,7 +47,7 @@
],
"fieldConfig": {
"defaults": {
"unit": "\u00b5s",
"unit": "µs",
"min": 0,
"thresholds": {
"mode": "absolute",
@@ -73,7 +73,7 @@
},
{
"title": "Job Throughput Rate (Per Second)",
"description": "Rate of jobs queued, started, and finished across all job types. Computed as rate() over the OTel counter values. High queue rates with low finish rates indicate backlog.",
"description": "**What:** Rate of jobs queued, started, and finished across all job types.\n**How it's computed:** Per-second rate of each cumulative job counter over a 5-minute window.\n**Reading it:** Queued, started, and finished tracking together means the queue keeps up.\n**Healthy range:** Workload-dependent; the three rates should stay roughly equal.\n**Watch for:** Queued rate persistently above finished rate, which indicates a growing backlog.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:recordJobQueued / recordJobStarted / recordJobFinished",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -133,7 +133,7 @@
},
{
"title": "Per-Job-Type Queued Rate",
"description": "Rate of jobs queued broken down by job_type label. Identifies which job types contribute most to queue activity.",
"description": "**What:** Rate of jobs entering the queue, broken down by job type (top 10).\n**How it's computed:** Per-second rate of the queued-job counter per job_type over a 5-minute window.\n**Reading it:** Identifies which job types generate the most queue activity.\n**Healthy range:** Workload-dependent.\n**Watch for:** A single job type surging unexpectedly, which can point to a flood of a particular request or peer message.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:recordJobQueued",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -184,7 +184,7 @@
},
{
"title": "Per-Job-Type Finish Rate",
"description": "Rate of jobs completing broken down by job_type. Compare with queued rate to identify backlog per type.",
"description": "**What:** Rate of jobs completing, broken down by job type (top 10).\n**How it's computed:** Per-second rate of the finished-job counter per job_type over a 5-minute window.\n**Reading it:** Compare against the queued rate per type to spot which types are falling behind.\n**Healthy range:** Workload-dependent; should match the per-type queued rate.\n**Watch for:** A type whose finish rate lags its queued rate, revealing where the backlog concentrates.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:recordJobFinished",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -235,7 +235,7 @@
},
{
"title": "Job Queue Wait Time",
"description": "Job queue wait time distribution (p75 typical, p99 tail). How long jobs sit in the queue before a worker picks them up.",
"description": "**What:** Distribution of how long jobs sit in the queue before a worker picks them up (p75 typical, p99 tail).\n**How it's computed:** Percentiles derived from the job wait-time histogram over a 5-minute window.\n**Reading it:** Lower is better; a widening gap between p75 and p99 signals occasional stalls.\n**Healthy range:** Sub-millisecond to low-millisecond waits on an unloaded node.\n**Watch for:** Rising p99 wait, an early sign of worker-thread saturation.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:recordJobStarted",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -267,12 +267,12 @@
],
"fieldConfig": {
"defaults": {
"unit": "\u00b5s",
"unit": "µs",
"custom": {
"drawStyle": "line",
"lineWidth": 2,
"fillOpacity": 5,
"axisLabel": "Duration (\u03bcs)",
"axisLabel": "Duration (μs)",
"spanNulls": true,
"insertNulls": false,
"showPoints": "auto",
@@ -288,7 +288,7 @@
},
{
"title": "Job Execution Time",
"description": "Job execution time distribution (p75 typical, p99 tail). How long jobs run once started.",
"description": "**What:** Distribution of how long jobs run once started (p75 typical, p99 tail).\n**How it's computed:** Percentiles derived from the job run-time histogram over a 5-minute window.\n**Reading it:** Lower is better; stable p75 with a controlled p99 is healthy.\n**Healthy range:** Workload-dependent, but stable over time.\n**Watch for:** Growing execution times, which point to CPU pressure or expensive individual jobs.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:recordJobFinished",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -320,12 +320,12 @@
],
"fieldConfig": {
"defaults": {
"unit": "\u00b5s",
"unit": "µs",
"custom": {
"drawStyle": "line",
"lineWidth": 2,
"fillOpacity": 5,
"axisLabel": "Duration (\u03bcs)",
"axisLabel": "Duration (μs)",
"spanNulls": true,
"insertNulls": false,
"showPoints": "auto",
@@ -341,7 +341,7 @@
},
{
"title": "Per-Job-Type Execution Time (p99)",
"description": "Top 10 slowest job types by p99 execution time.",
"description": "**What:** The 10 slowest job types ranked by p99 execution time.\n**How it's computed:** p99 derived from the run-time histogram per job_type over a 5-minute window, top 10 selected.\n**Reading it:** Highlights which job types cost the most CPU time at the tail.\n**Healthy range:** Workload-dependent.\n**Watch for:** A job type whose p99 grows over time, indicating a slow or degrading operation.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:recordJobFinished",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -371,12 +371,12 @@
],
"fieldConfig": {
"defaults": {
"unit": "\u00b5s",
"unit": "µs",
"custom": {
"drawStyle": "line",
"lineWidth": 1,
"fillOpacity": 5,
"axisLabel": "Duration (\u03bcs)",
"axisLabel": "Duration (μs)",
"spanNulls": true,
"insertNulls": false,
"showPoints": "auto",
@@ -392,7 +392,7 @@
},
{
"title": "Transaction Overflow Rate",
"description": "Rate of job queue transaction overflows per minute. Overflows occur when the job queue's transaction limit is exceeded, causing transactions to be dropped. Non-zero values indicate the node is under heavy transaction load.",
"description": "**What:** Rate at which transaction jobs are shed because the queue's transaction limit is exceeded.\n**How it's computed:** Per-second rate of the overflow counter over a 5-minute window, scaled to per minute.\n**Reading it:** Zero is healthy; any non-zero value means transaction jobs are being dropped under load.\n**Healthy range:** 0 overflows per minute under normal operation.\n**Watch for:** Non-zero overflows, which mark heavy transaction load or a flood; note this counter is typically absent when the node is healthy.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp (job-queue transaction overflow counter)",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -454,7 +454,7 @@
}
],
"schemaVersion": 39,
"tags": ["job-queue"],
"tags": ["node", "jobqueue"],
"templating": {
"list": [
{

16
docker/telemetry/grafana/dashboards/xrpld-peer-quality.json
View File

@@ -2,7 +2,7 @@
"annotations": {
"list": []
},
"description": "Peer network quality metrics: latency, divergence, version distribution, upgrade recommendations, disconnects, and connection direction balance. Requires push_metrics.py or equivalent OTel metric source.",
"description": "Peer connection quality: latency, divergence, version distribution and upgrade guidance, resource disconnects, and inbound/outbound balance.",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
@@ -11,7 +11,7 @@
"panels": [
{
"title": "P90 Peer Latency",
"description": "90th percentile peer-to-peer latency in milliseconds over time. High latency indicates network congestion or geographically distant peers.",
"description": "**What:** 90th-percentile round-trip latency to connected peers, in milliseconds.\n**How it's computed:** Instantaneous gauge reading of the p90 peer latency.\n**Reading it:** Lower is better; green under 200ms, yellow to 500ms, red above.\n**Healthy range:** Under 200ms.\n**Watch for:** Rising latency, which points to network congestion or geographically distant, poorly performing peers.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerPeerQualityGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -72,7 +72,7 @@
},
{
"title": "Insane/Diverged Peers",
"description": "Count of peers whose ledger state is considered insane or diverged from the network. Non-zero values suggest those peers are misbehaving or on a fork.",
"description": "**What:** Count of connected peers whose ledger state is insane or diverged from the network.\n**How it's computed:** Instantaneous gauge reading of the diverged-peer count.\n**Reading it:** Zero is healthy; any count means those peers disagree on ledger state.\n**Healthy range:** 0 diverged peers.\n**Watch for:** A persistent non-zero count, which can indicate peers on a fork or misbehaving/malicious peers.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerPeerQualityGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -121,7 +121,7 @@
},
{
"title": "Higher Version Peers %",
"description": "Percentage of connected peers running a higher rippled version. A high percentage suggests this node should be upgraded.",
"description": "**What:** Percentage of connected peers running a newer rippled version than this node.\n**How it's computed:** Instantaneous gauge reading of the higher-version peer percentage.\n**Reading it:** A high percentage suggests this node is behind and should be upgraded.\n**Healthy range:** Under 30%.\n**Watch for:** A majority of peers on a newer version, a strong upgrade signal.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerPeerQualityGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -172,7 +172,7 @@
},
{
"title": "Upgrade Recommended",
"description": "Whether an upgrade is recommended based on peer version analysis (1=recommended, 0=not needed). Triggered when a majority of peers run a newer version.",
"description": "**What:** A flag indicating whether an upgrade is advised based on peer version analysis (Yes/No).\n**How it's computed:** Instantaneous gauge reading of the upgrade-recommended flag.\n**Reading it:** No is healthy; Yes means most peers run a newer version.\n**Healthy range:** No.\n**Watch for:** A Yes state, indicating the node risks falling out of step with the network.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerPeerQualityGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -237,7 +237,7 @@
},
{
"title": "Resource Disconnects",
"description": "Cumulative peer disconnections due to resource limit violations over time. Rising values indicate aggressive or misbehaving peers being dropped.",
"description": "**What:** Cumulative count of peers dropped for exceeding resource (load) limits.\n**How it's computed:** Instantaneous gauge reading of the resource-disconnect total over time.\n**Reading it:** A flat line is healthy; a rising line means peers are being dropped for overuse.\n**Healthy range:** Flat / near constant.\n**Watch for:** A steep climb, which flags aggressive or misbehaving peers being shed as backpressure.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerServerInfoGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -282,7 +282,7 @@
},
{
"title": "Inbound vs Outbound Peers",
"description": "Comparison of active inbound and outbound peer connections. A healthy node should have a balanced mix. All-inbound may indicate NAT/firewall issues preventing outbound connections.",
"description": "**What:** Active inbound versus outbound peer connection counts.\n**How it's computed:** Instantaneous gauge readings of active inbound and outbound peer counts.\n**Reading it:** A balanced mix is healthy for connectivity and resilience.\n**Healthy range:** A mix of both; some outbound connections present.\n**Watch for:** All-inbound with no outbound, which usually means NAT or firewall issues block outbound peering.\n**Source:** src/xrpld/peerfinder/detail/PeerfinderManager.cpp:Logic Stats ctor",
"type": "bargauge",
"gridPos": {
"h": 8,
@@ -363,7 +363,7 @@
}
],
"schemaVersion": 39,
"tags": ["peer", "network"],
"tags": ["network", "peer"],
"templating": {
"list": [
{

525
docker/telemetry/grafana/dashboards/xrpld-rpc-perf-otel.json
View File

@@ -1,525 +0,0 @@
{
"annotations": {
"list": []
},
"description": "Per-RPC-method performance: call rates, error rates, and latency distributions. Sourced from OTel MetricsRegistry synchronous counters and histograms (Phase 9).",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
"id": null,
"links": [],
"panels": [
{
"title": "RPC Call Rate (All Methods)",
"description": "Aggregate rate of RPC calls started, finished, and errored across all methods. Computed as rate() over OTel counters.",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 0
},
"options": {
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": {
"type": "prometheus"
},
"expr": "sum by (exported_instance) (rate(xrpld_rpc_method_started_total{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m]))",
"legendFormat": "Started/s [{{exported_instance}}]"
},
{
"datasource": {
"type": "prometheus"
},
"expr": "sum by (exported_instance) (rate(xrpld_rpc_method_finished_total{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m]))",
"legendFormat": "Finished/s [{{exported_instance}}]"
},
{
"datasource": {
"type": "prometheus"
},
"expr": "sum by (exported_instance) (rate(xrpld_rpc_method_errored_total{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m]))",
"legendFormat": "Errored/s [{{exported_instance}}]"
}
],
"fieldConfig": {
"defaults": {
"unit": "ops",
"custom": {
"drawStyle": "line",
"lineWidth": 2,
"fillOpacity": 10,
"axisLabel": "Operations / Sec",
"spanNulls": true,
"insertNulls": false,
"showPoints": "auto",
"pointSize": 3
},
"color": {
"mode": "palette-classic"
}
},
"overrides": []
}
},
{
"title": "Per-Method Call Rate (Top 10)",
"description": "Per-method RPC call rate, showing the 10 most active methods. Useful for identifying hot paths.",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 8
},
"options": {
"tooltip": {
"mode": "multi",
"sort": "desc"
},
"legend": {
"displayMode": "table",
"placement": "right",
"calcs": ["mean", "max"]
}
},
"targets": [
{
"datasource": {
"type": "prometheus"
},
"expr": "topk(10, rate(xrpld_rpc_method_started_total{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m]))",
"legendFormat": "{{method}} [{{exported_instance}}]"
}
],
"fieldConfig": {
"defaults": {
"unit": "ops",
"custom": {
"drawStyle": "line",
"lineWidth": 1,
"fillOpacity": 5,
"axisLabel": "Operations / Sec",
"spanNulls": true,
"insertNulls": false,
"showPoints": "auto",
"pointSize": 3
},
"color": {
"mode": "palette-classic"
}
},
"overrides": []
}
},
{
"title": "Per-Method Error Rate (Top 10)",
"description": "Per-method RPC error rate. Non-zero values warrant investigation. Common culprits: invalid parameters, resource exhaustion.",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 8
},
"options": {
"tooltip": {
"mode": "multi",
"sort": "desc"
},
"legend": {
"displayMode": "table",
"placement": "right",
"calcs": ["mean", "max"]
}
},
"targets": [
{
"datasource": {
"type": "prometheus"
},
"expr": "topk(10, rate(xrpld_rpc_method_errored_total{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m]))",
"legendFormat": "{{method}} [{{exported_instance}}]"
}
],
"fieldConfig": {
"defaults": {
"unit": "ops",
"custom": {
"drawStyle": "line",
"lineWidth": 1,
"fillOpacity": 5,
"axisLabel": "Operations / Sec",
"spanNulls": true,
"insertNulls": false,
"showPoints": "auto",
"pointSize": 3
},
"color": {
"mode": "palette-classic"
}
},
"overrides": []
}
},
{
"title": "RPC Latency - All Methods",
"description": "RPC method latency distribution (p75 typical, p99 tail) across all methods.",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 16
},
"options": {
"tooltip": {
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": {
"type": "prometheus"
},
"expr": "histogram_quantile(0.75, sum by (le, exported_instance) (rate(xrpld_rpc_method_duration_us_bucket{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m])))",
"legendFormat": "p75 [{{exported_instance}}]"
},
{
"datasource": {
"type": "prometheus"
},
"expr": "histogram_quantile(0.99, sum by (le, exported_instance) (rate(xrpld_rpc_method_duration_us_bucket{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m])))",
"legendFormat": "p99 [{{exported_instance}}]"
}
],
"fieldConfig": {
"defaults": {
"unit": "\u00b5s",
"custom": {
"drawStyle": "line",
"lineWidth": 2,
"fillOpacity": 5,
"axisLabel": "Duration (\u03bcs)",
"spanNulls": true,
"insertNulls": false,
"showPoints": "auto",
"pointSize": 3
},
"color": {
"mode": "palette-classic"
}
},
"overrides": []
}
},
{
"title": "Per-Method Latency (p99, Top 10 Slowest)",
"description": "Top 10 slowest RPC methods by p99 latency.",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 16
},
"options": {
"tooltip": {
"mode": "multi",
"sort": "desc"
},
"legend": {
"displayMode": "table",
"placement": "right",
"calcs": ["mean", "max"]
}
},
"targets": [
{
"datasource": {
"type": "prometheus"
},
"expr": "topk(10, histogram_quantile(0.99, sum by (le, method, exported_instance) (rate(xrpld_rpc_method_duration_us_bucket{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m]))))",
"legendFormat": "{{method}} [{{exported_instance}}]"
}
],
"fieldConfig": {
"defaults": {
"unit": "\u00b5s",
"custom": {
"drawStyle": "line",
"lineWidth": 1,
"fillOpacity": 5,
"axisLabel": "Duration (\u03bcs)",
"spanNulls": true,
"insertNulls": false,
"showPoints": "auto",
"pointSize": 3
},
"color": {
"mode": "palette-classic"
}
},
"overrides": []
}
},
{
"title": "RPC Error Ratio by Method",
"description": "Error ratio (errors / total started) per method. Values above 0.05 (5%) warrant investigation.",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 24
},
"options": {
"tooltip": {
"mode": "multi",
"sort": "desc"
},
"legend": {
"displayMode": "table",
"placement": "right",
"calcs": ["mean", "max"]
}
},
"targets": [
{
"datasource": {
"type": "prometheus"
},
"expr": "topk(10, rate(xrpld_rpc_method_errored_total{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m]) / (rate(xrpld_rpc_method_started_total{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m]) > 0))",
"legendFormat": "{{method}} [{{exported_instance}}]"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0,
"max": 1,
"custom": {
"drawStyle": "line",
"lineWidth": 1,
"fillOpacity": 5,
"axisLabel": "Error Ratio",
"spanNulls": true,
"insertNulls": false,
"showPoints": "auto",
"pointSize": 3
},
"color": {
"mode": "palette-classic"
},
"thresholds": {
"steps": [
{
"color": "green",
"value": null
},
{
"color": "yellow",
"value": 0.05
},
{
"color": "red",
"value": 0.25
}
]
}
},
"overrides": []
}
},
{
"title": "Current RPC Latency (p99 Gauge)",
"description": "At-a-glance p99 RPC method latency over the last 5 minutes. Green < 100ms, yellow 100ms-1s, red > 1s.",
"type": "gauge",
"gridPos": {
"h": 8,
"w": 12,
"x": 0,
"y": 32
},
"options": {
"reduceOptions": {
"calcs": ["lastNotNull"],
"fields": "",
"values": false
},
"orientation": "auto",
"showThresholdLabels": false,
"showThresholdMarkers": true
},
"targets": [
{
"datasource": {
"type": "prometheus"
},
"expr": "histogram_quantile(0.99, sum by (le) (rate(xrpld_rpc_method_duration_us_bucket{exported_instance=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", method=~\"$method\"}[5m])))",
"legendFormat": "p99 Latency"
}
],
"fieldConfig": {
"defaults": {
"unit": "\u00b5s",
"min": 0,
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "yellow",
"value": 100000
},
{
"color": "red",
"value": 1000000
}
]
}
},
"overrides": []
}
}
],
"schemaVersion": 39,
"tags": ["rpc"],
"templating": {
"list": [
{
"name": "DS_PROMETHEUS",
"type": "datasource",
"label": "Prometheus",
"query": "prometheus",
"regex": "",
"current": {},
"hide": 2,
"refresh": 1,
"includeAll": false,
"multi": false,
"options": []
},
{
"name": "service_name",
"label": "Service",
"description": "Filter by service.name (e.g. xrpld, pratik-xrpld)",
"type": "query",
"query": "label_values(service_name)",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"includeAll": true,
"allValue": ".*",
"current": {
"text": "All",
"value": "$__all"
},
"multi": true,
"refresh": 2,
"sort": 1
},
{
"name": "deployment_environment",
"label": "Environment",
"description": "Filter by deployment tier (local/ci/test/prod)",
"type": "query",
"query": "label_values(deployment_environment)",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"includeAll": true,
"allValue": ".*",
"current": {
"text": "All",
"value": "$__all"
},
"multi": true,
"refresh": 2,
"sort": 1
},
{
"name": "xrpl_network_type",
"label": "Network",
"description": "Filter by XRPL network (mainnet/testnet/devnet)",
"type": "query",
"query": "label_values(xrpl_network_type)",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"includeAll": true,
"allValue": ".*",
"current": {
"text": "All",
"value": "$__all"
},
"multi": true,
"refresh": 2,
"sort": 1
},
{
"name": "node",
"label": "Node",
"description": "Filter by rippled node (service.instance.id)",
"type": "query",
"query": "label_values(exported_instance)",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"includeAll": true,
"allValue": ".*",
"current": {
"text": "All",
"value": "$__all"
},
"multi": true,
"refresh": 2,
"sort": 1
},
{
"name": "method",
"label": "RPC Method",
"description": "Filter by RPC method",
"type": "query",
"query": "label_values(xrpld_rpc_method_started_total, method)",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"includeAll": true,
"allValue": ".*",
"current": {
"text": "All",
"value": "$__all"
},
"multi": true,
"refresh": 2,
"sort": 1
}
]
},
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {},
"timezone": "browser",
"title": "RPC Performance (OTel)",
"uid": "xrpld-rpc-perf-otel",
"version": 1,
"refresh": "5s"
}

36
docker/telemetry/grafana/dashboards/xrpld-validator-health.json
View File

@@ -2,7 +2,7 @@
"annotations": {
"list": []
},
"description": "Validator health metrics: agreement rates, validation counts, amendment status, UNL expiry, server state tracking, and ledger close rates. Requires push_metrics.py or equivalent OTel metric source.",
"description": "Validator agreement rates, validation counts, amendment and UNL status, server state tracking, and ledger close rate.",
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 1,
@@ -23,7 +23,7 @@
},
{
"title": "Agreement % (1h)",
"description": "Validation agreement percentage over the last 1 hour. Values below 80% indicate the validator is frequently disagreeing with the network consensus.",
"description": "**What:** Share of ledgers over the last hour where this validator agreed with the network consensus.\n**How it's computed:** Instantaneous gauge reading of the 1-hour agreement percentage.\n**Reading it:** Higher is better; green at 95%+, yellow from 80%, red below.\n**Healthy range:** 95-100%.\n**Watch for:** Values below 80%, meaning the validator frequently disagrees with consensus.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerValidationAgreementGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -74,7 +74,7 @@
},
{
"title": "Agreement % (24h)",
"description": "Validation agreement percentage over the last 24 hours. A sustained value below 90% may indicate configuration drift or network partition.",
"description": "**What:** Share of ledgers over the last 24 hours where this validator agreed with consensus.\n**How it's computed:** Instantaneous gauge reading of the 24-hour agreement percentage.\n**Reading it:** Higher is better; a smoother, longer-term view than the 1h stat.\n**Healthy range:** 95-100%.\n**Watch for:** A sustained dip below 90%, which can indicate configuration drift or a network partition.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerValidationAgreementGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -125,7 +125,7 @@
},
{
"title": "Agreements vs Missed (1h)",
"description": "Comparison of successful agreements and missed validations over 1 hour. High missed count indicates the validator is not participating in consensus rounds.",
"description": "**What:** Counts of agreed versus missed validations over the last hour.\n**How it's computed:** Instantaneous gauge readings of the 1-hour agreed and missed counts.\n**Reading it:** Agreements should dominate; missed should be small.\n**Healthy range:** Missed near 0.\n**Watch for:** A high missed count, meaning the validator is skipping consensus rounds.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerValidationAgreementGauge",
"type": "bargauge",
"gridPos": {
"h": 8,
@@ -191,7 +191,7 @@
},
{
"title": "Agreements vs Missed (24h)",
"description": "Comparison of successful agreements and missed validations over 24 hours. Provides a longer-term view of validator reliability.",
"description": "**What:** Counts of agreed versus missed validations over the last 24 hours.\n**How it's computed:** Instantaneous gauge readings of the 24-hour agreed and missed counts.\n**Reading it:** Agreements should dominate over the full day.\n**Healthy range:** Missed a small fraction of agreements.\n**Watch for:** A growing missed share, signalling longer-term reliability problems.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerValidationAgreementGauge",
"type": "bargauge",
"gridPos": {
"h": 8,
@@ -269,7 +269,7 @@
},
{
"title": "Validation Rate",
"description": "Rate of validations sent per minute (5m average). Indicates how actively this node is producing validations. Expected ~10-12/min on mainnet (one per ledger close).",
"description": "**What:** Validations this node sends per minute.\n**How it's computed:** Per-second rate of the sent-validations counter over 5 minutes, scaled to per minute.\n**Reading it:** Should track the ledger close cadence; roughly one validation per closed ledger.\n**Healthy range:** About 10-12 per minute on mainnet.\n**Watch for:** A drop toward zero, meaning the validator has stopped participating.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:incrementValidationsSent (caller src/xrpld/app/consensus/RCLConsensus.cpp)",
"type": "stat",
"gridPos": {
"h": 8,
@@ -318,7 +318,7 @@
},
{
"title": "Validations Checked Rate",
"description": "Rate of validations checked (received from peers) per minute (5m average). Indicates how many validations from the network are being processed.",
"description": "**What:** Validations received from peers and checked per minute.\n**How it's computed:** Per-second rate of the checked-validations counter over 5 minutes, scaled to per minute.\n**Reading it:** Reflects how much validation traffic the network is delivering to this node.\n**Healthy range:** Workload-dependent; scales with trusted validator count.\n**Watch for:** A sudden collapse, which suggests peer connectivity loss or network isolation.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:incrementValidationsChecked (caller src/xrpld/app/misc/NetworkOPs.cpp)",
"type": "stat",
"gridPos": {
"h": 8,
@@ -351,7 +351,7 @@
},
{
"title": "Amendment Blocked",
"description": "Whether the node is amendment-blocked (1=blocked, 0=normal). An amendment-blocked node cannot validate and needs a software upgrade.",
"description": "**What:** Whether the node is amendment-blocked (OK/BLOCKED).\n**How it's computed:** Instantaneous gauge reading of the amendment-blocked flag.\n**Reading it:** OK is healthy; BLOCKED means an enabled amendment is unsupported by this build.\n**Healthy range:** OK.\n**Watch for:** A BLOCKED state, which halts validation and requires a software upgrade.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerValidatorHealthGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -416,7 +416,7 @@
},
{
"title": "UNL Expiry (days)",
"description": "Days until the UNL (Unique Node List) expires. A value below 7 requires attention to renew the UNL before it expires and the validator stops trusting peers.",
"description": "**What:** Days remaining until the current UNL (trusted validator list) expires.\n**How it's computed:** Instantaneous gauge reading of days-to-expiry.\n**Reading it:** Higher is safer; green at 30+, yellow under 7, red at expiry.\n**Healthy range:** 30+ days.\n**Watch for:** Fewer than 7 days, after which the node loses its trusted validator set if not renewed.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerValidatorHealthGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -465,7 +465,7 @@
},
{
"title": "UNL Blocked",
"description": "Whether the node's UNL (Unique Node List) is blocked (1=blocked, 0=normal). A UNL-blocked node cannot determine validator trust and may stop participating in consensus.",
"description": "**What:** Whether the node's UNL is blocked (OK/BLOCKED).\n**How it's computed:** Instantaneous gauge reading of the UNL-blocked flag.\n**Reading it:** OK is healthy; BLOCKED means validator trust cannot be established.\n**Healthy range:** OK.\n**Watch for:** A BLOCKED state, which can stop the node participating in consensus.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerValidatorHealthGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -530,7 +530,7 @@
},
{
"title": "Agreement/Missed Counters (Rate)",
"description": "Rate of cumulative validation agreements and misses per minute. These monotonic counters complement the rolling window percentages above.",
"description": "**What:** Rate of cumulative agreement and missed-validation counters per minute.\n**How it's computed:** Per-second rate of each monotonic counter over 5 minutes, scaled to per minute.\n**Reading it:** Agreements should dominate; the missed line should stay low.\n**Healthy range:** Missed rate near 0.\n**Watch for:** A rising missed rate, complementing the windowed agreement percentages above.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerValidationTotalsCounters",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -610,7 +610,7 @@
},
{
"title": "Validation Quorum",
"description": "Minimum number of trusted validations needed to declare a ledger validated. Tracks the current quorum requirement from the validator list.",
"description": "**What:** Minimum number of trusted validations required to declare a ledger fully validated.\n**How it's computed:** Instantaneous gauge reading of the current quorum requirement.\n**Reading it:** Tracks the quorum derived from the active validator list; changes when the list changes.\n**Healthy range:** Stable at the network-appropriate value.\n**Watch for:** An unexpected drop, which can weaken consensus safety guarantees.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerValidatorHealthGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -643,7 +643,7 @@
},
{
"title": "State Value Timeline",
"description": "Numeric encoding of the server operating state over time. Useful for correlating state changes with other metrics.",
"description": "**What:** Numeric encoding of the server operating state (disconnected, connected, syncing, tracking, full) over time.\n**How it's computed:** Instantaneous gauge reading of the encoded state value.\n**Reading it:** A flat line at the full-operation state is healthy; steps show transitions.\n**Healthy range:** Steady at the highest (full) state.\n**Watch for:** Frequent transitions, useful for correlating state flapping with other metrics.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerStateTrackingGauge",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -688,7 +688,7 @@
},
{
"title": "Time in Current State",
"description": "How long the server has been in its current operating state, in seconds. Short durations with frequent changes indicate instability.",
"description": "**What:** How long the server has held its current operating state, in seconds.\n**How it's computed:** Instantaneous gauge reading of the time-in-state value.\n**Reading it:** Long, growing durations indicate stability.\n**Healthy range:** Long and increasing while in the full-operation state.\n**Watch for:** Short durations resetting often, a sign of state instability; this value may read 0 where not yet populated.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerStateTrackingGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -721,7 +721,7 @@
},
{
"title": "State Changes Rate",
"description": "Rate of server state changes per hour. A healthy node should have near-zero state changes. Frequent transitions indicate network or configuration issues.",
"description": "**What:** Rate of server operating-state changes per hour.\n**How it's computed:** Per-second rate of the state-change counter over a 1-hour window.\n**Reading it:** Near zero is healthy; each increment is one state transition.\n**Healthy range:** Near 0 changes per hour.\n**Watch for:** Frequent transitions, which point to network instability or configuration problems.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:incrementStateChanges (caller src/xrpld/app/misc/NetworkOPs.cpp)",
"type": "stat",
"gridPos": {
"h": 8,
@@ -770,7 +770,7 @@
},
{
"title": "Ledgers Closed Rate",
"description": "Rate of ledgers closed per minute (5m average). On mainnet, expect roughly 10-12/min. Deviations indicate consensus timing issues.",
"description": "**What:** Ledgers closed per minute by this node.\n**How it's computed:** Per-second rate of the ledgers-closed counter over 5 minutes, scaled to per minute.\n**Reading it:** Should match the network's steady close cadence.\n**Healthy range:** About 10-12 per minute on mainnet.\n**Watch for:** Deviation from the expected cadence, which indicates consensus timing trouble or the node falling behind.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:incrementLedgersClosed (caller src/xrpld/app/consensus/RCLConsensus.cpp)",
"type": "stat",
"gridPos": {
"h": 8,
@@ -819,7 +819,7 @@
},
{
"title": "Agreement % (7d)",
"description": "Validation agreement percentage over the trailing 7 days \u2014 the long-term reliability window used by external validator dashboards. Complements the 1h/24h stats.",
"description": "**What:** Share of ledgers over the trailing 7 days where this validator agreed with consensus.\n**How it's computed:** Instantaneous gauge reading of the 7-day agreement percentage.\n**Reading it:** The long-term reliability window; higher is better.\n**Healthy range:** 95-100%.\n**Watch for:** A gradual decline, which reflects chronic rather than transient disagreement.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerValidationAgreementGauge",
"type": "stat",
"gridPos": {
"h": 8,
@@ -870,7 +870,7 @@
},
{
"title": "Agreements vs Missed (7d)",
"description": "Agreed vs missed validation counts over the trailing 7 days. A rising missed trend signals sustained validator unreliability.",
"description": "**What:** Agreed versus missed validation counts over the trailing 7 days.\n**How it's computed:** Instantaneous gauge readings of the 7-day agreed and missed counts.\n**Reading it:** Agreements should dominate across the week.\n**Healthy range:** Missed a small fraction of agreements.\n**Watch for:** A rising missed trend, signalling sustained validator unreliability.\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp:registerValidationAgreementGauge",
"type": "timeseries",
"gridPos": {
"h": 8,