diff --git a/docker/telemetry/grafana/dashboards/consensus-health.json b/docker/telemetry/grafana/dashboards/consensus-health.json index 8bda530684..484e40f691 100644 --- a/docker/telemetry/grafana/dashboards/consensus-health.json +++ b/docker/telemetry/grafana/dashboards/consensus-health.json @@ -26,7 +26,7 @@ "description": "**What:** Time to process an accepted ledger, from the moment consensus is reached through applying the transaction set and finalising state.\n**How it's computed:** 95th and 50th percentile of per-round durations over a 5-minute window, per node.\n**Reading it:** Flat p50 with a modest p95 gap is normal; both track network transaction load.\n**Healthy range:** Roughly 3-6 seconds on mainnet.\n**Watch for:** A rising p95 that pulls away from p50 means occasional slow rounds; sustained growth precedes ledger-age alarms.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::makeAcceptSpan", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, "y": 1 @@ -81,7 +81,7 @@ "h": 1, "w": 24, "x": 0, - "y": 9 + "y": 11 }, "collapsed": false, "panels": [], @@ -92,10 +92,10 @@ "description": "**What:** How often this node issues a validation confirming it has fully validated a ledger.\n**How it's computed:** Per-second rate over a 5-minute window, per node.\n**Reading it:** Should closely track the ledger close rate on a healthy validator.\n**Healthy range:** About one per ledger (~0.25/s on mainnet).\n**Watch for:** Validations lagging closes suggests the node is falling behind on validation.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::createValidationSpan", "type": "stat", "gridPos": { - "h": 4, + "h": 10, "w": 12, "x": 0, - "y": 10 + "y": 12 }, "options": { "tooltip": { @@ -129,10 +129,10 @@ "description": "**What:** How often this node broadcasts its own consensus proposal (its candidate transaction set) to the network.\n**How it's computed:** Per-second rate over a 5-minute window, per node.\n**Reading it:** A proposing validator emits a steady stream, roughly one per consensus round; observing-only nodes emit none.\n**Healthy range:** About one proposal per round (~0.2-0.3/s on mainnet) for a proposing node.\n**Watch for:** A validator that drops to zero has stopped proposing and may be unhealthy or misconfigured.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::propose", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 0, - "y": 14 + "x": 12, + "y": 12 }, "options": { "tooltip": { @@ -174,10 +174,10 @@ "description": "**What:** Time spent closing a ledger once consensus triggers the close, up to completion.\n**How it's computed:** 95th percentile of per-close durations over a 5-minute window, per node.\n**Reading it:** Should sit at or just below the full round duration; a subset of overall round time.\n**Healthy range:** A few seconds, tracking the ~4s mainnet ledger interval.\n**Watch for:** A widening gap versus round duration points to overhead outside ledger construction.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::onClose", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 12, - "y": 14 + "x": 0, + "y": 22 }, "options": { "tooltip": { @@ -219,9 +219,9 @@ "description": "**What:** Time spent applying the agreed consensus result to build the new ledger.\n**How it's computed:** 95th and 50th percentile of per-round apply durations over a 5-minute window, per node.\n**Reading it:** Scales with the size of the transaction set being applied.\n**Healthy range:** Tens to a few hundred milliseconds under normal load.\n**Watch for:** Spikes indicate heavy transaction sets or I/O pressure while writing ledger state.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::doAccept", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 0, + "x": 12, "y": 22 }, "targets": [ @@ -272,10 +272,10 @@ "description": "**What:** Whether validators agreed on the ledger close time (agreed) or agreed to disagree and used a fallback (disagreed), per round.\n**How it's computed:** Per-second rate of rounds split by agreement outcome over a 5-minute window, per node.\n**Reading it:** Nearly all rounds should show agreement.\n**Healthy range:** Overwhelmingly agreement; occasional disagreement is tolerable.\n**Watch for:** A rising disagreement share signals clock drift or network latency across validators.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::doAccept", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 12, - "y": 22 + "x": 0, + "y": 32 }, "targets": [ { @@ -317,10 +317,10 @@ "description": "**What:** Ledger close events split by the node's consensus mode: Proposing, Observing, Wrong Ledger, or Switched Ledger.\n**How it's computed:** Per-second rate per mode over a 5-minute window, per node.\n**Reading it:** A healthy validator stays predominantly in Proposing; a stock node stays in Observing.\n**Healthy range:** Mostly one dominant mode with negligible Wrong/Switched Ledger.\n**Watch for:** Frequent Wrong Ledger or Switched Ledger indicates the node is out of sync and at fork risk.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::onClose", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 0, - "y": 30 + "x": 12, + "y": 32 }, "options": { "tooltip": { @@ -362,10 +362,10 @@ "description": "**What:** Rate of ledgers accepted after consensus versus ledger-close events initiated.\n**How it's computed:** Per-second rate of each over a 5-minute window, per node.\n**Reading it:** The two lines should overlap on a healthy network.\n**Healthy range:** Both near the ledger cadence (~0.25/s on mainnet) and tracking each other.\n**Watch for:** Divergence means some closes never complete the accept phase, hinting at consensus failures or timeouts.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::makeAcceptSpan, onClose", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 12, - "y": 30 + "x": 0, + "y": 42 }, "options": { "tooltip": { @@ -415,10 +415,10 @@ "description": "**What:** Rate of validations sent versus ledger-close events.\n**How it's computed:** Per-second rate of each over a 5-minute window, per node.\n**Reading it:** Each validated ledger should yield one validation, so the lines should overlap.\n**Healthy range:** Both near the ledger cadence and tracking each other.\n**Watch for:** Validations trailing closes points to a stalled or slow validation pipeline.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::createValidationSpan, onClose", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 0, - "y": 38 + "x": 12, + "y": 42 }, "options": { "tooltip": { @@ -468,10 +468,10 @@ "description": "**What:** Distribution of per-round accept durations over time, showing how round times are spread rather than just a percentile.\n**How it's computed:** Counts of rounds by duration band in each 5-minute window, per node.\n**Reading it:** A tight band around the typical round time is healthy; brighter cells mark the common duration.\n**Healthy range:** Concentrated near the 3-6 second mainnet round time.\n**Watch for:** A second band at high durations reveals recurring slow rounds hidden by averages.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::makeAcceptSpan", "type": "heatmap", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 12, - "y": 38 + "x": 0, + "y": 52 }, "options": { "tooltip": { @@ -512,7 +512,7 @@ "h": 1, "w": 24, "x": 0, - "y": 46 + "y": 62 }, "collapsed": false, "panels": [], @@ -523,10 +523,10 @@ "description": "**What:** Each node's raw, unrounded proposed close time at the instant it closed its ledger.\n**How it's computed:** Latest raw close-time value per node, plotted per round.\n**Reading it:** Compare nodes at the same round; values should cluster tightly.\n**Healthy range:** All nodes within a few seconds of each other.\n**Watch for:** A node consistently offset from the pack indicates local clock drift.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::doAccept", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 47 + "y": 63 }, "fieldConfig": { "defaults": { @@ -572,10 +572,10 @@ "description": "**What:** The consensus-agreed close time after rounding to the active resolution bin, i.e. the value written to the ledger header.\n**How it's computed:** Latest effective close-time value per node, plotted per round.\n**Reading it:** All in-agreement nodes should report the identical value each round.\n**Healthy range:** Identical across agreeing nodes.\n**Watch for:** Nodes reporting different effective values are not in close-time agreement for that round.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::doAccept", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 47 + "y": 63 }, "fieldConfig": { "defaults": { @@ -621,10 +621,10 @@ "description": "**What:** Number of distinct close-time vote buckets and the current close-time resolution, per round.\n**How it's computed:** Latest bin count and resolution value plotted per round, per node.\n**Reading it:** Fewer bins and finer resolution mean tighter agreement; resolution coarsens automatically when validators disagree.\n**Healthy range:** Few bins with fine (about 10s) resolution during healthy operation.\n**Watch for:** Many bins or resolution jumping to its coarse ceiling signals widespread clock disagreement.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::doAccept", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 55 + "y": 73 }, "fieldConfig": { "defaults": { @@ -711,10 +711,10 @@ "description": "**What:** Whether close-time resolution coarsened, sharpened, or held steady versus the previous ledger.\n**How it's computed:** Latest direction outcome plotted per round, per node.\n**Reading it:** Mostly unchanged with occasional sharpening is healthy.\n**Healthy range:** Predominantly unchanged.\n**Watch for:** Repeated coarsening indicates worsening agreement round over round.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::doAccept", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 55 + "y": 73 }, "fieldConfig": { "defaults": { @@ -759,10 +759,10 @@ "description": "**What:** How raw proposed close times spread across the quantized buckets each round.\n**How it's computed:** Count of proposals landing in each bucket, stacked per round.\n**Reading it:** A single dominant bar means strong clock agreement.\n**Healthy range:** One dominant bucket per round.\n**Watch for:** Spread across many buckets points to clock drift or network latency between validators.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::doAccept", "type": "barchart", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 63 + "y": 83 }, "fieldConfig": { "defaults": { @@ -808,7 +808,7 @@ "h": 1, "w": 24, "x": 0, - "y": 71 + "y": 93 }, "collapsed": false, "panels": [], @@ -819,10 +819,10 @@ "description": "**What:** Share of consensus outcomes: normal agreement, moved on without full agreement, or expired (timed out).\n**How it's computed:** Total counts per outcome over a 5-minute window, shown as proportions.\n**Reading it:** The normal outcome should dominate overwhelmingly.\n**Healthy range:** Nearly all normal.\n**Watch for:** A growing moved-on or expired slice indicates network stress or instability.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::makeAcceptSpan", "type": "piechart", "gridPos": { - "h": 8, + "h": 10, "w": 24, "x": 0, - "y": 72 + "y": 94 }, "options": { "legend": { @@ -857,10 +857,10 @@ "description": "**What:** Rate of non-normal consensus outcomes (moved on plus expired) over time.\n**How it's computed:** Per-second rate of each failure outcome over a 5-minute window, per node.\n**Reading it:** Should sit at or near zero.\n**Healthy range:** Effectively zero on a stable network.\n**Watch for:** Spikes indicate consensus instability; sustained failures warrant investigation.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::makeAcceptSpan", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 80 + "y": 104 }, "options": { "tooltip": { @@ -910,10 +910,10 @@ "description": "**What:** End-to-end duration of a complete consensus round, filterable by consensus mode.\n**How it's computed:** 95th and 50th percentile of full-round durations over a 5-minute window, per node.\n**Reading it:** The single most important consensus-health signal; watch the trend.\n**Healthy range:** Roughly 3-6 seconds on mainnet.\n**Watch for:** Rising round time precedes validated-ledger-age alarms and points to convergence trouble.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::startRoundTracing", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 80 + "y": 104 }, "options": { "tooltip": { @@ -963,10 +963,10 @@ "description": "**What:** Duration of the open phase (collecting transactions) versus the establish phase (converging on a proposal), which together make up a round.\n**How it's computed:** 95th percentile of each phase's duration over a 5-minute window, per node.\n**Reading it:** Compare the two to see whether slowness is in gathering transactions or reaching agreement.\n**Healthy range:** Open phase near the close interval; establish phase typically shorter.\n**Watch for:** A ballooning establish phase means slow convergence; a long open phase means delayed close.\n**Source:** src/xrpld/consensus/Consensus.h:Consensus::phaseOpen, Consensus::phaseEstablish", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 88 + "y": 114 }, "options": { "tooltip": { @@ -1018,7 +1018,7 @@ "h": 1, "w": 24, "x": 0, - "y": 96 + "y": 124 }, "collapsed": false, "panels": [], @@ -1029,10 +1029,10 @@ "description": "**What:** Time spent each round tallying disputes and updating this node's consensus position.\n**How it's computed:** 95th and 50th percentile of per-round durations over a 5-minute window, per node.\n**Reading it:** Grows with the number of disputed transactions being reconciled.\n**Healthy range:** A few milliseconds under normal conditions.\n**Watch for:** Long durations indicate heavy dispute resolution or slow convergence on close time.\n**Source:** src/xrpld/consensus/Consensus.h:Consensus::updateOurPositions", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 97 + "y": 125 }, "options": { "tooltip": { @@ -1082,10 +1082,10 @@ "description": "**What:** Rate of consensus health checks reporting a stalled condition versus progressing normally.\n**How it's computed:** Per-second rate split by stalled flag over a 5-minute window, per node.\n**Reading it:** The stalled line should stay at zero.\n**Healthy range:** Zero stalled checks.\n**Watch for:** Any sustained stalled rate surfaces a stall before it becomes a validated-ledger-age alarm.\n**Source:** src/xrpld/consensus/Consensus.h:Consensus::haveConsensus", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 97 + "y": 125 }, "options": { "tooltip": { @@ -1135,10 +1135,10 @@ "description": "**What:** Rate of consensus mode transitions, split by the mode the node switched into.\n**How it's computed:** Per-second rate per target mode over a 5-minute window, per node.\n**Reading it:** A stable node changes mode rarely; occasional switches around startup are normal.\n**Healthy range:** Near zero once synced.\n**Watch for:** Frequent switches into Wrong Ledger or Switched Ledger mark an unstable node at fork risk.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::onModeChange", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 105 + "y": 135 }, "options": { "tooltip": { diff --git a/docker/telemetry/grafana/dashboards/ledger-data-sync.json b/docker/telemetry/grafana/dashboards/ledger-data-sync.json index 7de2692b78..3460d6d4f7 100644 --- a/docker/telemetry/grafana/dashboards/ledger-data-sync.json +++ b/docker/telemetry/grafana/dashboards/ledger-data-sync.json @@ -14,7 +14,7 @@ "description": "###### What this is:\n*All overlay traffic categories ranked by inbound bytes, giving an at-a-glance view of which message types consume the most receive bandwidth.*\n\n###### How it's computed:\n*Top categories by latest inbound byte value across all traffic categories. Each bar is labelled with its traffic category followed by the node identity; the shared `_bytes_in` suffix is dropped from the category name because the panel already reports inbound bytes.*\n\n###### Reading it:\n*The longest bars are the biggest bandwidth consumers; on a synced node transactions, proposals, and validations usually lead.*\n\n###### Healthy range:\n*workload-dependent.*\n\n###### Watch for:\n*A single ledger-data or fetch category dominating (ongoing sync) or an unexpected category topping the list.*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "bargauge", "gridPos": { - "h": 4, + "h": 10, "w": 12, "x": 0, "y": 0 @@ -75,10 +75,10 @@ "description": "###### What this is:\n*Inbound bytes for ledger-data message categories, split into aggregate get/share plus the transaction-set, transaction-node, and account-state-node sub-types the node receives from peers.*\n\n###### How it's computed:\n*Per-category inbound byte rate per node.*\n\n###### Reading it:\n*Normally low and flat once synced. Account-state-node traffic dominates during state sync; transaction-set-candidate traffic dominates during consensus catch-up.*\n\n###### Healthy range:\n*workload-dependent; low and steady on a synced node.*\n\n###### Watch for:\n*Sustained high account-state or tx-node inbound bytes on a node that should be caught up (repeated re-sync, missing history), or a single peer driving all traffic.*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 0, - "y": 4 + "x": 12, + "y": 0 }, "options": { "tooltip": { @@ -125,10 +125,10 @@ "description": "###### What this is:\n*Inbound bytes for ledger-data message categories, split into aggregate get/share plus the transaction-set, transaction-node, and account-state-node sub-types the node receives from peers.*\n\n###### How it's computed:\n*Per-category inbound byte rate per node.*\n\n###### Reading it:\n*Normally low and flat once synced. Account-state-node traffic dominates during state sync; transaction-set-candidate traffic dominates during consensus catch-up.*\n\n###### Healthy range:\n*workload-dependent; low and steady on a synced node.*\n\n###### Watch for:\n*Sustained high account-state or tx-node inbound bytes on a node that should be caught up (repeated re-sync, missing history), or a single peer driving all traffic.*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 12, - "y": 4 + "x": 0, + "y": 10 }, "options": { "tooltip": { @@ -187,10 +187,10 @@ "description": "###### What this is:\n*Inbound bytes for ledger-data message categories, split into aggregate get/share plus the transaction-set, transaction-node, and account-state-node sub-types the node receives from peers.*\n\n###### How it's computed:\n*Per-category inbound byte rate per node.*\n\n###### Reading it:\n*Normally low and flat once synced. Account-state-node traffic dominates during state sync; transaction-set-candidate traffic dominates during consensus catch-up.*\n\n###### Healthy range:\n*workload-dependent; low and steady on a synced node.*\n\n###### Watch for:\n*Sustained high account-state or tx-node inbound bytes on a node that should be caught up (repeated re-sync, missing history), or a single peer driving all traffic.*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 0, - "y": 12 + "x": 12, + "y": 10 }, "options": { "tooltip": { @@ -237,10 +237,10 @@ "description": "###### What this is:\n*Inbound bytes for the older ledger share/get message categories and their tx-set, tx-node, and account-state sub-types.*\n\n###### How it's computed:\n*Per-category inbound byte rate per node.*\n\n###### Reading it:\n*Usually small; these legacy categories carry ledger-fetch traffic for peers using the older protocol.*\n\n###### Healthy range:\n*workload-dependent; low on a synced node.*\n\n###### Watch for:\n*Large sustained volumes indicating heavy fetch load or a peer repeatedly requesting the same data.*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 12, - "y": 12 + "x": 0, + "y": 20 }, "options": { "tooltip": { @@ -287,9 +287,9 @@ "description": "###### What this is:\n*Inbound bytes for the older ledger share/get message categories and their tx-set, tx-node, and account-state sub-types.*\n\n###### How it's computed:\n*Per-category inbound byte rate per node.*\n\n###### Reading it:\n*Usually small; these legacy categories carry ledger-fetch traffic for peers using the older protocol.*\n\n###### Healthy range:\n*workload-dependent; low on a synced node.*\n\n###### Watch for:\n*Large sustained volumes indicating heavy fetch load or a peer repeatedly requesting the same data.*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 0, + "x": 12, "y": 20 }, "options": { @@ -349,10 +349,10 @@ "description": "###### What this is:\n*Inbound bytes for the older ledger share/get message categories and their tx-set, tx-node, and account-state sub-types.*\n\n###### How it's computed:\n*Per-category inbound byte rate per node.*\n\n###### Reading it:\n*Usually small; these legacy categories carry ledger-fetch traffic for peers using the older protocol.*\n\n###### Healthy range:\n*workload-dependent; low on a synced node.*\n\n###### Watch for:\n*Large sustained volumes indicating heavy fetch load or a peer repeatedly requesting the same data.*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 12, - "y": 20 + "x": 0, + "y": 30 }, "options": { "tooltip": { @@ -399,10 +399,10 @@ "description": "###### What this is:\n*Inbound bytes for object-fetch traffic broken down by object type: ledger headers, individual transactions, transaction-tree nodes, and state-tree nodes.*\n\n###### How it's computed:\n*Per-type inbound byte rate per node.*\n\n###### Reading it:\n*Small during steady state; grows when the node fetches missing tree nodes.*\n\n###### Healthy range:\n*workload-dependent; low when synced.*\n\n###### Watch for:\n*A large share on state/tx nodes for long periods (persistent gap-filling), meaning the node keeps catching up.*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 0, - "y": 28 + "x": 12, + "y": 30 }, "options": { "tooltip": { @@ -449,10 +449,10 @@ "description": "###### What this is:\n*Inbound bytes for object-fetch traffic broken down by object type: ledger headers, individual transactions, transaction-tree nodes, and state-tree nodes.*\n\n###### How it's computed:\n*Per-type inbound byte rate per node.*\n\n###### Reading it:\n*Small during steady state; grows when the node fetches missing tree nodes.*\n\n###### Healthy range:\n*workload-dependent; low when synced.*\n\n###### Watch for:\n*A large share on state/tx nodes for long periods (persistent gap-filling), meaning the node keeps catching up.*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 12, - "y": 28 + "x": 0, + "y": 40 }, "options": { "tooltip": { @@ -511,10 +511,10 @@ "description": "###### What this is:\n*Inbound bytes for object-fetch traffic broken down by object type: ledger headers, individual transactions, transaction-tree nodes, and state-tree nodes.*\n\n###### How it's computed:\n*Per-type inbound byte rate per node.*\n\n###### Reading it:\n*Small during steady state; grows when the node fetches missing tree nodes.*\n\n###### Healthy range:\n*workload-dependent; low when synced.*\n\n###### Watch for:\n*A large share on state/tx nodes for long periods (persistent gap-filling), meaning the node keeps catching up.*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 0, - "y": 36 + "x": 12, + "y": 40 }, "options": { "tooltip": { @@ -561,10 +561,10 @@ "description": "###### What this is:\n*Count of individual object-fetch request/response messages per object type.*\n\n###### How it's computed:\n*Per-type inbound message rate per node.*\n\n###### Reading it:\n*Many messages with few bytes means small piecemeal fetches; few messages with many bytes means large batch transfers.*\n\n###### Healthy range:\n*workload-dependent.*\n\n###### Watch for:\n*High message counts with tiny payloads sustained over time (inefficient per-node fetching).*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 12, - "y": 36 + "x": 0, + "y": 50 }, "options": { "tooltip": { @@ -605,10 +605,10 @@ "description": "###### What this is:\n*Count of individual object-fetch request/response messages per object type.*\n\n###### How it's computed:\n*Per-type inbound message rate per node.*\n\n###### Reading it:\n*Many messages with few bytes means small piecemeal fetches; few messages with many bytes means large batch transfers.*\n\n###### Healthy range:\n*workload-dependent.*\n\n###### Watch for:\n*High message counts with tiny payloads sustained over time (inefficient per-node fetching).*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 0, - "y": 44 + "x": 12, + "y": 50 }, "options": { "tooltip": { @@ -655,10 +655,10 @@ "description": "###### What this is:\n*Count of individual object-fetch request/response messages per object type.*\n\n###### How it's computed:\n*Per-type inbound message rate per node.*\n\n###### Reading it:\n*Many messages with few bytes means small piecemeal fetches; few messages with many bytes means large batch transfers.*\n\n###### Healthy range:\n*workload-dependent.*\n\n###### Watch for:\n*High message counts with tiny payloads sustained over time (inefficient per-node fetching).*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 12, - "y": 44 + "x": 0, + "y": 60 }, "options": { "tooltip": { @@ -699,10 +699,10 @@ "description": "###### What this is:\n*Count of individual object-fetch request/response messages per object type.*\n\n###### How it's computed:\n*Per-type inbound message rate per node.*\n\n###### Reading it:\n*Many messages with few bytes means small piecemeal fetches; few messages with many bytes means large batch transfers.*\n\n###### Healthy range:\n*workload-dependent.*\n\n###### Watch for:\n*High message counts with tiny payloads sustained over time (inefficient per-node fetching).*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 0, - "y": 52 + "x": 12, + "y": 60 }, "options": { "tooltip": { @@ -755,10 +755,10 @@ "description": "###### What this is:\n*Aggregate object-fetch inbound bytes plus special buckets: content-addressed storage fetches, bulk fetch-pack downloads used during catch-up, and bulk transaction fetches.*\n\n###### How it's computed:\n*Per-category inbound byte rate per node.*\n\n###### Reading it:\n*Fetch-pack rises sharply while catching up a range of ledgers; near zero when fully synced.*\n\n###### Healthy range:\n*workload-dependent; low when synced.*\n\n###### Watch for:\n*Continuous fetch-pack traffic (node never fully catches up) or unexpectedly high content-store volume.*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 12, - "y": 52 + "x": 0, + "y": 70 }, "options": { "tooltip": { @@ -838,7 +838,7 @@ "h": 1, "w": 24, "x": 0, - "y": 60 + "y": 80 }, "id": 16, "panels": [] @@ -848,10 +848,10 @@ "description": "###### What this is:\n*Current server operating state as a numeric code: 0 disconnected, 1 connected, 2 syncing, 3 tracking, 4 full, 5 validating, 6 proposing. A healthy validator sits at 6, a healthy non-validating node at 4.*\n\n###### How it's computed:\n*state_tracking{metric=\"state_value\"} (gauge). Companion time_in_current_state_seconds shows how long it has been stuck there.*\n\n###### Reading it:\n*Flat at 4-6 = full/healthy. Dropping to 1-2 and staying = the node fell out of sync and is re-acquiring (the primary red flag this row explains).*\n\n###### Healthy range:\n*4-6 steady, briefly 2-3 right after restart.*\n\n###### Watch for:\n*A node stuck below 4 for more than a few minutes, or oscillating - read the lower panels for the cause.*\n\n###### Source:\n[app/misc/NetworkOPs.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/misc/NetworkOPs.cpp)\n\n###### Function:\n`NetworkOPsImp::setMode`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 61 + "y": 81 }, "options": { "tooltip": { @@ -973,10 +973,10 @@ "description": "###### What this is:\n*Seconds since this node last had a freshly validated ledger. The single clearest 'am I keeping up with the network' signal.*\n\n###### How it's computed:\n*ledgermaster_validated_ledger_age gauge (seconds), per node.*\n\n###### Reading it:\n*Should hover at the network close interval (~3-5s). A rising sawtooth or a high plateau means the node is falling behind or not validating.*\n\n###### Healthy range:\n*<= ~6s on a synced node.*\n\n###### Watch for:\n*Sustained climb above ~15s, or a monotonic ramp = the node is not keeping up; correlate with job-queue wait and NuDB read latency below.*\n\n###### Source:\n[app/ledger/LedgerMaster.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/LedgerMaster.cpp)\n\n###### Function:\n`LedgerMaster::getValidatedLedgerAge`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 61 + "y": 81 }, "options": { "tooltip": { @@ -1021,14 +1021,14 @@ "id": 18 }, { - "title": "Time Spent Per State [$xrpl_network_type]", + "title": "Time Spent Per State", "description": "###### What this is:\n*Operating mode as a colour-coded timeline. Each band's width is the time spent in that state, so short-lived states show as thin slivers instead of vanishing.*\n\n###### How it's computed:\n*server_info{metric=\"server_state\"} (gauge), the raw OperatingMode 0-4. Uses server_state rather than state_value because state_value folds 5 and 6 onto FULL, which would split one Full band into three colours.*\n\n###### Reading it:\n*One green band across the window = healthy. Red/orange/yellow bands show when and for how long the node was degraded.*\n\n###### Healthy range:\n*Continuously green (Full), brief orange/yellow/blue only after a restart.*\n\n###### Watch for:\n*Repeated thin bands = the node is oscillating. This is sampled every 10s, so a state shorter than one sample can still be missed.*\n\n###### Source:\n[app/misc/NetworkOPs.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/misc/NetworkOPs.cpp)\n\n###### Function:\n`NetworkOPsImp::setMode`", "type": "state-timeline", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 69 + "y": 91 }, "options": { "mergeValues": true, @@ -1131,20 +1131,17 @@ }, "overrides": [] }, - "id": 19, - "repeat": "xrpl_network_type", - "repeatDirection": "h", - "maxPerRow": 2 + "id": 19 }, { "title": "Ledger Close Rate", "description": "###### What this is:\n*Rate at which the node closes ledgers, versus the ~0.25/s network cadence. A throughput deficit means it is not advancing in step with the network.*\n\n###### How it's computed:\n*rate(ledgers_closed_total[$__rate_interval]) per node.*\n\n###### Reading it:\n*Should track ~0.22-0.25 ledgers/s (one every ~4s). Near-zero while behind = stalled; a burst above network rate = catching up.*\n\n###### Healthy range:\n*~0.25/s steady on a synced node.*\n\n###### Watch for:\n*Near-zero close rate while Validated Ledger Age climbs = hard stall (e.g. genesis-flapping or disk-bound acquisition).*\n\n###### Source:\n[app/ledger/LedgerMaster.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/LedgerMaster.cpp)\n\n###### Function:\n`LedgerMaster::closeLedger`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 69 + "y": 91 }, "options": { "tooltip": { @@ -1210,10 +1207,10 @@ "description": "###### What this is:\n*95th-percentile time a job waits in the queue before a worker thread picks it up, for the sync-critical job types. This is the metric form of the 'ProcessLData wait: NNNNms' warnings in the debug log.*\n\n###### How it's computed:\n*histogram_quantile(0.95, rate(jobq__q_milliseconds_bucket[$__rate_interval])) for ledgerdata, acceptledger, fetchtxndata, transaction, advanceledger, ledgerrequest.*\n\n###### Reading it:\n*Queue wait should be single-digit to low-tens of ms. High ledgerdata/fetchtxndata wait = the node cannot process inbound ledger data fast enough.*\n\n###### Healthy range:\n*< ~50ms p95 per type on a healthy node.*\n\n###### Watch for:\n*ledgerdata or fetchtxndata q-wait spiking to seconds = worker threads are blocked (usually on NuDB reads - see the cause tier).*\n\n###### Source:\n[core/JobQueue.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/core/detail/JobQueue.cpp)\n\n###### Function:\n`JobQueue::getJson (per-type queue timing)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 77 + "y": 101 }, "options": { "tooltip": { @@ -1290,10 +1287,10 @@ "description": "###### What this is:\n*Average nodestore (NuDB) read latency: how long each on-disk object read takes. The direct disk-layer cost that backs up the job queue.*\n\n###### How it's computed:\n*rate(nodestore_state{metric=\"node_reads_duration_us\"}[$__rate_interval]) / rate(nodestore_state{metric=\"node_reads_total\"}[$__rate_interval]), in us.*\n\n###### Reading it:\n*Low single-digit us when the OS page cache is warm; tens-to-hundreds of us when reads hit the disk. Rises sharply during cold-cache catch-up.*\n\n###### Healthy range:\n*< ~10us/read warm; higher is expected briefly after a wipe/restart.*\n\n###### Watch for:\n*Sustained high us/read is the disk-bound signal on its own; the found ratio below stays near 100% even then, so do not wait for it to drop. Check EBS IOPS / io scheduler latency.*\n\n###### Source:\n[nodestore/backend/NuDBFactory.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/nodestore/backend/NuDBFactory.cpp)\n\n###### Function:\n`NuDB backend read path`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 77 + "y": 101 }, "options": { "tooltip": { @@ -1335,10 +1332,10 @@ "description": "###### What this is:\n*95th-percentile latency of the node's internal I/O service queue - the async task scheduler that dispatches network and disk callbacks.*\n\n###### How it's computed:\n*histogram_quantile(0.95, rate(ios_latency_milliseconds_bucket[$__rate_interval])).*\n\n###### Reading it:\n*Low and flat when the event loop is responsive. Rising p95 means callbacks are queuing behind long-running work (often blocking disk reads).*\n\n###### Healthy range:\n*< ~10ms p95.*\n\n###### Watch for:\n*p95 climbing into hundreds of ms = the I/O service is saturated; the node cannot service network/disk events promptly, stalling sync.*\n\n###### Source:\n[core/impl/Workers.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/core/detail/Workers.cpp)\n\n###### Function:\n`io_service latency histogram`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 85 + "y": 111 }, "options": { "tooltip": { @@ -1380,10 +1377,10 @@ "description": "###### What this is:\n*Fraction of nodestore fetches that found the object they asked for. This is not a cache hit ratio: `node_reads_hit` counts every fetch that returned an object, whatever served it, so a fetch that went all the way to disk still counts here.*\n\n###### How it's computed:\n*rate(nodestore_state{metric=\"node_reads_hit\"}[$__rate_interval]) / rate(nodestore_state{metric=\"node_reads_total\"}[$__rate_interval]).*\n\n###### Reading it:\n*Normally sits near 1.0 (100%) on any node that has the data, warm or cold, because a synced node almost always finds what it asks for. It does not fall when the page cache goes cold.*\n\n###### Healthy range:\n*Near 1.0. A ratio well below 1.0 means fetches are missing, which points at a gap in local history rather than at cache pressure.*\n\n###### Watch for:\n*Never read this panel on its own. A ~100% found ratio at over 100 microseconds per read is the cold-read signature, not a healthy cache: the data is found every time and paid for every time. Always pair it with NuDB Read Latency.*\n\n###### Source:\n[nodestore/Database.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/nodestore/Database.cpp)\n\n###### Function:\n`Database::fetchNodeObject`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 85 + "y": 111 }, "options": { "tooltip": { @@ -1425,10 +1422,10 @@ "description": "###### What this is:\n*Nodestore read throughput and backlog: reads per second, the pending read queue depth, and how many read threads are active. Shows disk saturation depth.*\n\n###### How it's computed:\n*rate(node_reads_total), and gauges read_queue / read_threads_running from nodestore_state.*\n\n###### Reading it:\n*reads/s spikes during catch-up. A growing read_queue with all read threads busy = disk cannot keep up with demand.*\n\n###### Healthy range:\n*read_queue ~0 and reads/s low on a warm synced node.*\n\n###### Watch for:\n*read_queue climbing while read_threads_running is pinned at read_threads_total = disk-bound; the IOPS ceiling is the limiter.*\n\n###### Source:\n[nodestore/backend/NuDBFactory.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/nodestore/backend/NuDBFactory.cpp)\n\n###### Function:\n`NuDB read scheduler`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 93 + "y": 121 }, "options": { "tooltip": { @@ -1517,10 +1514,10 @@ "description": "###### What this is:\n*Total number of jobs currently queued across the JobQueue. A backlog means work is arriving faster than worker threads can drain it.*\n\n###### How it's computed:\n*jobq_job_count gauge (total queued jobs), per node.*\n\n###### Reading it:\n*Near 0 when keeping up. Sustained positive depth = the node is overloaded or blocked on a downstream resource (usually disk reads).*\n\n###### Healthy range:\n*~0 on a healthy node.*\n\n###### Watch for:\n*Depth climbing in step with Validated Ledger Age = the queue backlog is why the node is falling behind.*\n\n###### Source:\n[core/JobQueue.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/core/detail/JobQueue.cpp)\n\n###### Function:\n`JobQueue::getJobCountTotal`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 93 + "y": 121 }, "options": { "tooltip": { @@ -1563,10 +1560,10 @@ "description": "###### What this is:\n*Server load factor (fee/throttle multiplier; 1 = unloaded) alongside active inbound/outbound peer counts. Rules out overload and insufficient fetch sources as causes.*\n\n###### How it's computed:\n*load_factor_metrics{metric=\"load_factor\"}, and peer_finder_active_inbound_peers / _outbound_peers gauges.*\n\n###### Reading it:\n*load_factor at 1 = no local overload. Outbound peers should be healthy (~10+); too few peers limits how fast the node can fetch ledger data.*\n\n###### Healthy range:\n*load_factor = 1; outbound peers >= ~8.*\n\n###### Watch for:\n*load_factor > 1 = local overload throttling; very low peer count = not enough sources to acquire history from (a distinct sync bottleneck).*\n\n###### Source:\n[overlay/detail/OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl load / PeerFinder counts`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 101 + "y": 131 }, "options": { "tooltip": { diff --git a/docker/telemetry/grafana/dashboards/ledger-operations.json b/docker/telemetry/grafana/dashboards/ledger-operations.json index e4b8f02b3d..4516e8455e 100644 --- a/docker/telemetry/grafana/dashboards/ledger-operations.json +++ b/docker/telemetry/grafana/dashboards/ledger-operations.json @@ -26,7 +26,7 @@ "description": "**What:** How often the node constructs a new ledger from its parent, applying transactions and writing state.\n**How it's computed:** Per-second rate over a 5-minute window, per node.\n**Reading it:** Should match the consensus close cadence.\n**Healthy range:** About 0.25/s on mainnet (~4s rounds).\n**Watch for:** A drop below the close rate means the node is not keeping up with ledger construction.\n**Source:** src/xrpld/app/ledger/detail/BuildLedger.cpp:buildLedgerImpl", "type": "stat", "gridPos": { - "h": 4, + "h": 10, "w": 12, "x": 0, "y": 1 @@ -63,7 +63,7 @@ "description": "**What:** How often ledgers reach the trusted-validation quorum and become fully validated.\n**How it's computed:** Per-second rate over a 5-minute window, per node.\n**Reading it:** Should match the build and close rates on a healthy node.\n**Healthy range:** About 0.25/s on mainnet.\n**Watch for:** A shortfall means ledgers are not accumulating enough trusted validations to be accepted.\n**Source:** src/xrpld/app/ledger/detail/LedgerMaster.cpp:LedgerMaster::checkAccept", "type": "stat", "gridPos": { - "h": 4, + "h": 10, "w": 12, "x": 12, "y": 1 @@ -100,10 +100,10 @@ "description": "**What:** Time to build a full ledger, including applying transactions, writing state, and finalising acceptance.\n**How it's computed:** 95th and 50th percentile of per-build durations over a 5-minute window, per node.\n**Reading it:** Scales with transaction volume and disk I/O.\n**Healthy range:** Tens to a few hundred milliseconds under typical load.\n**Watch for:** Long build times indicate expensive transaction sets or I/O pressure while flushing state.\n**Source:** src/xrpld/app/ledger/detail/BuildLedger.cpp:buildLedgerImpl", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 5 + "y": 11 }, "options": { "tooltip": { @@ -153,10 +153,10 @@ "description": "**What:** Distribution of ledger-build durations over time, revealing spread beyond a single percentile.\n**How it's computed:** Counts of builds by duration band in each 5-minute window, per node.\n**Reading it:** A tight band at low durations is healthy; brighter cells mark the common build time.\n**Healthy range:** Concentrated in the low tens of milliseconds.\n**Watch for:** An occasional high-duration band flags slow builds that percentile charts may miss.\n**Source:** src/xrpld/app/ledger/detail/BuildLedger.cpp:buildLedgerImpl", "type": "heatmap", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 5 + "y": 11 }, "options": { "tooltip": { @@ -197,7 +197,7 @@ "h": 1, "w": 24, "x": 0, - "y": 13 + "y": 21 }, "collapsed": false, "panels": [], @@ -208,10 +208,10 @@ "description": "**What:** Time to apply the agreed transaction set into the new ledger, including retry passes.\n**How it's computed:** 95th and 50th percentile of per-ledger apply durations over a 5-minute window, per node.\n**Reading it:** Grows with the number and complexity of transactions in the set.\n**Healthy range:** Tens to a few hundred milliseconds under normal load.\n**Watch for:** Sustained high durations indicate heavy or expensive transaction sets.\n**Source:** src/xrpld/app/ledger/detail/BuildLedger.cpp:applyTransactions", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 14 + "y": 22 }, "options": { "tooltip": { @@ -261,10 +261,10 @@ "description": "**What:** How often the transaction-application phase runs during ledger building.\n**How it's computed:** Per-second rate over a 5-minute window, per node.\n**Reading it:** One apply pass per ledger build, so it should track the build rate.\n**Healthy range:** About 0.25/s on mainnet.\n**Watch for:** Divergence from the build rate signals an accounting or pipeline anomaly.\n**Source:** src/xrpld/app/ledger/detail/BuildLedger.cpp:applyTransactions", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 14 + "y": 22 }, "options": { "tooltip": { @@ -308,7 +308,7 @@ "h": 1, "w": 24, "x": 0, - "y": 22 + "y": 32 }, "collapsed": false, "panels": [], @@ -319,10 +319,10 @@ "description": "**What:** How often completed ledgers are written into the local ledger history.\n**How it's computed:** Per-second rate over a 5-minute window, per node.\n**Reading it:** Should match the build rate under normal operation.\n**Healthy range:** About 0.25/s on mainnet.\n**Watch for:** A store rate below the build rate means ledgers are being built but not persisted.\n**Source:** src/xrpld/app/ledger/detail/LedgerMaster.cpp:LedgerMaster::storeLedger", "type": "stat", "gridPos": { - "h": 4, + "h": 10, "w": 12, "x": 0, - "y": 23 + "y": 33 }, "options": { "tooltip": { @@ -356,10 +356,10 @@ "description": "**What:** Ledger construction time versus total ledger-close time; construction is a subset of close.\n**How it's computed:** 95th percentile of each duration over a 5-minute window, per node.\n**Reading it:** Build should sit at or just below close.\n**Healthy range:** Both a few hundred milliseconds or less, tracking each other.\n**Watch for:** A large gap means significant overhead in the consensus pipeline outside ledger construction.\n**Source:** src/xrpld/app/ledger/detail/BuildLedger.cpp:buildLedgerImpl, src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::onClose", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 0, - "y": 27 + "x": 12, + "y": 33 }, "options": { "tooltip": { diff --git a/docker/telemetry/grafana/dashboards/network-traffic.json b/docker/telemetry/grafana/dashboards/network-traffic.json index 5b2e0d809f..15f4b80950 100644 --- a/docker/telemetry/grafana/dashboards/network-traffic.json +++ b/docker/telemetry/grafana/dashboards/network-traffic.json @@ -27,7 +27,7 @@ "description": "###### What this is:\n*Number of active inbound and outbound peer connections the node currently holds.*\n\n###### How it's computed:\n*Current value of the inbound and outbound active-peer counts per node.*\n\n###### Reading it:\n*Outbound is what the node dials out; inbound is what others open to it. Both should be stable.*\n\n###### Healthy range:\n*roughly 10-21 outbound and 0-85 inbound on mainnet, depending on config.*\n\n###### Watch for:\n*Outbound dropping toward zero (isolation) or inbound pinned at the limit with churn (connection pressure).*\n\n###### Source:\n[PeerfinderManager.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/peerfinder/detail/PeerfinderManager.cpp)\n\n###### Function:\n`Logic Stats ctor`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, "y": 1 @@ -78,7 +78,7 @@ "description": "###### What this is:\n*Peer connections that dropped, as a per-interval increase.*\n\n###### How it's computed:\n*Disconnect events per rate-interval, per node.*\n\n###### Reading it:\n*A flat or slowly rising line is normal; the slope matters more than the absolute value.*\n\n###### Healthy range:\n*workload-dependent; slow, steady growth.*\n\n###### Watch for:\n*Sharp step-ups in the slope (network instability, resource exhaustion, or many peers dropping the node at once).*\n\n###### Source:\n[OverlayImpl.h](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.h)\n\n###### Function:\n`OverlayImpl::Stats ctor`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, "y": 1 @@ -123,10 +123,10 @@ "description": "###### What this is:\n*Total bytes received and sent across all peer connections.*\n\n###### How it's computed:\n*Per-second in/out byte rate per node.*\n\n###### Reading it:\n*Overall bandwidth footprint; in and out usually track network activity together.*\n\n###### Healthy range:\n*workload-dependent.*\n\n###### Watch for:\n*Sudden sustained jumps not matched by ledger or transaction activity (relay storms or a noisy peer).*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 9 + "y": 11 }, "options": { "tooltip": { @@ -173,10 +173,10 @@ "description": "###### What this is:\n*Total messages received and sent across all peer connections.*\n\n###### How it's computed:\n*Per-second in/out message rate per node.*\n\n###### Reading it:\n*Overall message throughput of the overlay; complements the byte totals.*\n\n###### Healthy range:\n*workload-dependent.*\n\n###### Watch for:\n*Message count climbing far faster than bytes (many tiny messages, possible flooding).*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 9 + "y": 11 }, "options": { "tooltip": { @@ -223,10 +223,10 @@ "description": "###### What this is:\n*Transaction relay messages in and out, plus duplicate transaction messages received.*\n\n###### How it's computed:\n*Per-second message rate for the transaction and transaction-duplicate categories.*\n\n###### Reading it:\n*In/out rise with network transaction volume; duplicates are transactions the node already had.*\n\n###### Healthy range:\n*workload-dependent; duplicates a modest fraction of inbound.*\n\n###### Watch for:\n*Duplicate inbound approaching or exceeding unique inbound (redundant relay), or a sharp spike suggesting transaction flooding.*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 17 + "y": 21 }, "options": { "tooltip": { @@ -279,10 +279,10 @@ "description": "###### What this is:\n*Consensus proposal messages in/out, plus untrusted and duplicate proposal messages received.*\n\n###### How it's computed:\n*Per-second message rate for the proposal, proposal-untrusted, and proposal-duplicate categories.*\n\n###### Reading it:\n*Trusted in/out track consensus rounds; untrusted come from validators not on this node's trusted list.*\n\n###### Healthy range:\n*workload-dependent; untrusted and duplicates low relative to trusted.*\n\n###### Watch for:\n*High untrusted (trusted-list misconfiguration) or high duplicates (inefficient relay or proposal spam).*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 17 + "y": 21 }, "options": { "tooltip": { @@ -341,10 +341,10 @@ "description": "###### What this is:\n*Validation messages in/out, plus untrusted and duplicate validation messages received.*\n\n###### How it's computed:\n*Per-second message rate for the validation, validation-untrusted, and validation-duplicate categories.*\n\n###### Reading it:\n*Trusted validations should arrive steadily each ledger; untrusted come from non-trusted validators.*\n\n###### Healthy range:\n*workload-dependent; untrusted and duplicates low relative to trusted.*\n\n###### Watch for:\n*Rising untrusted or duplicate validations (trusted-list health issues or validation spam).*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 25 + "y": 31 }, "options": { "tooltip": { @@ -405,7 +405,7 @@ "h": 1, "w": 24, "x": 0, - "y": 33 + "y": 41 }, "collapsed": false, "panels": [], @@ -416,10 +416,10 @@ "description": "###### What this is:\n*Top overlay traffic categories ranked by inbound bytes, excluding the all-traffic total.*\n\n###### How it's computed:\n*Top categories by inbound byte rate per node.*\n\n###### Reading it:\n*Shows which message types dominate receive bandwidth right now.*\n\n###### Healthy range:\n*workload-dependent; transactions, proposals, and validations typically lead on a synced node.*\n\n###### Watch for:\n*A fetch or ledger-data category topping the list (sync activity) or an unexpected category dominating.*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "bargauge", "gridPos": { - "h": 4, + "h": 10, "w": 12, "x": 0, - "y": 34 + "y": 42 }, "options": { "tooltip": { @@ -717,10 +717,10 @@ "description": "###### What this is:\n*Throughput of duplicate transaction, proposal, and validation traffic: messages the node had already seen and discarded.*\n\n###### How it's computed:\n*Per-second rate of the duplicate byte counters for each category, in and out.*\n\n###### Reading it:\n*Lower is better; this is bandwidth spent on redundant relays.*\n\n###### Healthy range:\n*workload-dependent; a small fraction of total traffic.*\n\n###### Watch for:\n*Duplicate rate climbing toward the same order as useful traffic (poor relay topology or redundant flooding).*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 0, - "y": 38 + "x": 12, + "y": 42 }, "options": { "tooltip": { @@ -793,7 +793,7 @@ "h": 1, "w": 24, "x": 0, - "y": 46 + "y": 52 }, "collapsed": false, "panels": [], @@ -804,10 +804,10 @@ "description": "###### What this is:\n*The busiest overlay categories by inbound byte rate over time, excluding the all-traffic total.*\n\n###### How it's computed:\n*Per-second inbound byte rate for the busiest categories, ranked, excluding the all-traffic total.*\n\n###### Reading it:\n*Time-series companion to the category bar view; shows how the traffic mix shifts over the window.*\n\n###### Healthy range:\n*workload-dependent.*\n\n###### Watch for:\n*A category ramping up and staying high, or the mix suddenly changing (sync, spam, or a misbehaving peer).*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 47 + "y": 53 }, "options": { "tooltip": { diff --git a/docker/telemetry/grafana/dashboards/node-health.json b/docker/telemetry/grafana/dashboards/node-health.json index 0e11e71d9d..99bc8276e9 100644 --- a/docker/telemetry/grafana/dashboards/node-health.json +++ b/docker/telemetry/grafana/dashboards/node-health.json @@ -14,7 +14,7 @@ "description": "**What:** Seconds since the most recently validated ledger, i.e. how far behind the network the node is.\n**How it's computed:** Current value of the validated-ledger-age gauge per node.\n**Reading it:** Lower is better; a healthy node stays within a few ledger-close intervals.\n**Healthy range:** under ~10s (the network closes a ledger every 3-5s).\n**Watch for:** Above 20s or climbing steadily (the node is falling behind or has lost sync).\n**Source:** src/xrpld/app/ledger/LedgerMaster.h LedgerMaster::Stats ctor", "type": "stat", "gridPos": { - "h": 4, + "h": 10, "w": 12, "x": 0, "y": 0 @@ -66,7 +66,7 @@ "description": "**What:** Seconds since the most recently published ledger (the ledger exposed to clients and subscribers).\n**How it's computed:** Current value of the published-ledger-age gauge per node.\n**Reading it:** Should track validated-ledger age closely.\n**Healthy range:** under ~10s.\n**Watch for:** Published age growing while validated age stays low (publish-pipeline backlog).\n**Source:** src/xrpld/app/ledger/LedgerMaster.h LedgerMaster::Stats ctor", "type": "stat", "gridPos": { - "h": 4, + "h": 10, "w": 12, "x": 12, "y": 0 @@ -118,10 +118,10 @@ "description": "**What:** Rate at which the node requests ledgers from peers.\n**How it's computed:** Per-second rate of the ledger-fetches counter over 5 minutes.\n**Reading it:** Near zero when synced; rises when catching up or backfilling history.\n**Healthy range:** workload-dependent; low on a synced node.\n**Watch for:** Sustained high fetch rate on a node that should be current (repeatedly missing ledgers).\n**Source:** src/xrpld/app/ledger/detail/InboundLedgers.cpp InboundLedgersImp ctor", "type": "stat", "gridPos": { - "h": 4, + "h": 10, "w": 12, "x": 0, - "y": 4 + "y": 10 }, "options": { "tooltip": { @@ -154,10 +154,10 @@ "description": "**What:** Rate of ledger-history hash mismatches: built ledgers whose hash disagrees with the validated hash.\n**How it's computed:** Per-second rate of the history-mismatch counter over 5 minutes.\n**Reading it:** Should be flat at zero.\n**Healthy range:** zero.\n**Watch for:** Any non-zero value (consensus divergence, corrupted history, or database issues).\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp incrementLedgerHistoryMismatch (caller LedgerHistory.cpp)", "type": "stat", "gridPos": { - "h": 4, + "h": 10, "w": 12, "x": 12, - "y": 4 + "y": 10 }, "options": { "tooltip": { @@ -202,10 +202,10 @@ "description": "**What:** Fraction of time in each operating mode (per-interval): Disconnected, Connected, Syncing, Tracking, Full.\n**How it's computed:** Per-mode duration rate divided by the sum of all five mode rates over the interval; the five shares stack to 100%.\n**Reading it:** Full should fill almost the entire band on a healthy node.\n**Healthy range:** Full near 100%; other modes near 0.\n**Watch for:** A rising share of Syncing, Connected, or Disconnected (instability or repeated resync).\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp NetworkOPsImp::Stats ctor", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 8 + "y": 20 }, "options": { "tooltip": { @@ -277,10 +277,10 @@ "description": "**What:** Count of transitions into each operating mode.\n**How it's computed:** Per-mode transition counters at their latest value per node.\n**Reading it:** Few transitions is good; a stable node rarely leaves Full.\n**Healthy range:** workload-dependent; low and infrequent transitions.\n**Watch for:** Frequent transitions out of Full, or any into Disconnected/Syncing (flapping).\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp NetworkOPsImp::Stats ctor", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 8 + "y": 20 }, "options": { "tooltip": { @@ -349,10 +349,10 @@ "description": "**What:** P95 and P50 latency of the I/O service loop, i.e. how long a queued timer callback waits to run.\n**How it's computed:** 95th and 50th percentiles over a 5-minute window.\n**Reading it:** Low and flat is good; this reflects event-loop responsiveness.\n**Healthy range:** single-digit milliseconds; over ~10ms is notable, over ~500ms is bad.\n**Watch for:** Rising P95 (thread-pool saturation or blocking work on the I/O thread).\n**Source:** src/xrpld/app/main/Application.cpp ApplicationImp ctor", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 16 + "y": 30 }, "options": { "tooltip": { @@ -400,10 +400,10 @@ "description": "**What:** Number of jobs currently waiting in the job queue.\n**How it's computed:** Current value of the job-count gauge per node.\n**Reading it:** Near zero when the node keeps up; brief spikes during heavy work are normal.\n**Healthy range:** low, returning to baseline quickly.\n**Watch for:** Sustained high depth (node cannot process work fast enough: replay, heavy RPC, or overload).\n**Source:** src/libxrpl/core/detail/JobQueue.cpp JobQueue ctor", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 16 + "y": 30 }, "options": { "tooltip": { @@ -447,7 +447,7 @@ "h": 1, "w": 24, "x": 0, - "y": 24 + "y": 40 }, "collapsed": false, "panels": [], @@ -458,10 +458,10 @@ "description": "**What:** Hit-rate percentage of the FullBelowCache.\n**How it's computed:** Current value of the cache hit-rate gauge (0-100%).\n**Reading it:** Higher is better; means the node reuses knowledge of complete subtrees.\n**Healthy range:** above ~50% in steady state.\n**Watch for:** Low hit rate during steady state (cache too small or constant re-acquisition).\n**Source:** include/xrpl/basics/TaggedCache.h TaggedCache::Stats ctor", "type": "gauge", "gridPos": { - "h": 4, + "h": 10, "w": 12, "x": 0, - "y": 25 + "y": 41 }, "options": { "tooltip": { @@ -512,10 +512,10 @@ "description": "**What:** Difference between published and validated ledger ages: how far the publish pipeline trails validation.\n**How it's computed:** Published-ledger age minus validated-ledger age, in seconds.\n**Reading it:** Near zero means publishing keeps up with validation.\n**Healthy range:** within a few seconds of zero.\n**Watch for:** A growing gap (publish backlog, stale data for subscribers).\n**Source:** src/xrpld/app/ledger/LedgerMaster.h LedgerMaster::Stats ctor", "type": "stat", "gridPos": { - "h": 4, + "h": 10, "w": 12, "x": 12, - "y": 25 + "y": 41 }, "options": { "tooltip": { @@ -564,10 +564,10 @@ "description": "**What:** Execution time of the most important job types (accept/advance ledger, transaction, write objects, heartbeat, sweep, trusted validation/proposal, publish, client RPC, ledger data) at the selected quantile.\n**How it's computed:** Selected quantile of each job's execution-time histogram over 5 minutes.\n**Reading it:** Lower and stable is better; shows where consensus, transaction, and maintenance time goes.\n**Healthy range:** workload-dependent; most jobs in low tens of milliseconds.\n**Watch for:** One job type spiking (a specific bottleneck) or broad increases (overload).\n**Source:** include/xrpl/core/JobTypeData.h JobTypeData ctor", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 29 + "y": 51 }, "options": { "tooltip": { @@ -678,10 +678,10 @@ "description": "**What:** Time key jobs wait in the queue before they start running, at the selected quantile.\n**How it's computed:** Selected quantile of each job's queue-wait histogram over 5 minutes.\n**Reading it:** Low wait means the scheduler keeps up; high wait means backlog.\n**Healthy range:** workload-dependent; short waits.\n**Watch for:** Rising wait across job types (queue congestion delaying consensus and transaction work).\n**Source:** include/xrpl/core/JobTypeData.h JobTypeData ctor", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 29 + "y": 51 }, "options": { "tooltip": { @@ -792,10 +792,10 @@ "description": "**What:** Number of entries in the FullBelowCache, which tracks tree nodes known to have all children present locally.\n**How it's computed:** Current value of the cache-size gauge per node.\n**Reading it:** Grows as the node learns complete subtrees; helps avoid redundant fetches.\n**Healthy range:** workload-dependent; stable once synced.\n**Watch for:** Collapsing to near zero repeatedly (cache thrash) alongside rising fetch traffic.\n**Source:** include/xrpl/basics/TaggedCache.h TaggedCache::Stats ctor", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 37 + "y": 61 }, "options": { "tooltip": { @@ -836,10 +836,10 @@ "description": "**What:** How fast the node accrues time in Full versus Tracking mode, normalized to seconds per second.\n**How it's computed:** Per-second rate of the Full and Tracking accumulated-time gauges, scaled to seconds.\n**Reading it:** Full rate near 1.0 means the node is in Full essentially all the time.\n**Healthy range:** Full rate ~1.0, Tracking near 0.\n**Watch for:** Full rate dropping below 1.0 with Tracking rising (time being lost to non-Full modes).\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp NetworkOPsImp::Stats ctor", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 37 + "y": 61 }, "options": { "tooltip": { @@ -887,10 +887,10 @@ "description": "**What:** Execution time for all non-special job types at the selected quantile.\n**How it's computed:** Selected quantile of each job's execution-time histogram over 5 minutes.\n**Reading it:** Full breakdown of job performance; use the Key Jobs panel for the focused view.\n**Healthy range:** workload-dependent.\n**Watch for:** Any job type with a persistent upward trend, or many rising together (systemic slowdown).\n**Source:** include/xrpl/core/JobTypeData.h JobTypeData ctor", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 45 + "y": 71 }, "options": { "tooltip": { @@ -931,10 +931,10 @@ "description": "**What:** Queue wait time before execution for all non-special job types at the selected quantile.\n**How it's computed:** Selected quantile of each job's queue-wait histogram over 5 minutes.\n**Reading it:** Complete picture of scheduling delay across job types.\n**Healthy range:** workload-dependent; short waits.\n**Watch for:** High waits across many job types (systemic job-queue congestion).\n**Source:** include/xrpl/core/JobTypeData.h JobTypeData ctor", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 45 + "y": 71 }, "options": { "tooltip": { diff --git a/docker/telemetry/grafana/dashboards/overlay-traffic-detail.json b/docker/telemetry/grafana/dashboards/overlay-traffic-detail.json index 458cc5e087..71dc72a74d 100644 --- a/docker/telemetry/grafana/dashboards/overlay-traffic-detail.json +++ b/docker/telemetry/grafana/dashboards/overlay-traffic-detail.json @@ -14,7 +14,7 @@ "description": "###### What this is:\n*Squelch relay-control messages in/out, plus messages suppressed by squelch and squelch directives that were ignored. Squelch reduces redundant message forwarding between peers.*\n\n###### How it's computed:\n*Per-second message rate for the squelch, squelch-suppressed, and squelch-ignored categories, in and out.*\n\n###### Reading it:\n*High suppressed counts mean squelch is saving bandwidth; ignored should stay low.*\n\n###### Healthy range:\n*workload-dependent; suppressed far above ignored.*\n\n###### Watch for:\n*High ignored counts (peers not honoring squelch) or squelch traffic itself dominating.*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, "y": 0 @@ -88,7 +88,7 @@ "description": "###### What this is:\n*Overlay protocol overhead bytes split into base overhead, intra-cluster overhead, and validator-manifest distribution overhead.*\n\n###### How it's computed:\n*Per-second in/out byte rate for the overhead, overhead-cluster, and overhead-manifest categories.*\n\n###### Reading it:\n*Base overhead is routine; cluster and manifest rise around cluster syncs and manifest changes.*\n\n###### Healthy range:\n*workload-dependent; low and stable.*\n\n###### Watch for:\n*Sustained high cluster or manifest overhead (frequent cluster state churn or manifest reissue).*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, "y": 0 @@ -162,10 +162,10 @@ "description": "###### What this is:\n*Bytes and messages exchanged distributing validator lists (trusted-list configuration) between peers.*\n\n###### How it's computed:\n*Per-second in/out byte and message rate for the validator-lists category.*\n\n###### Reading it:\n*Bursts when lists update or new peers connect; quiet otherwise.*\n\n###### Healthy range:\n*workload-dependent; occasional bursts.*\n\n###### Watch for:\n*Continuous high volume (repeated list re-fetching or churn).*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 8 + "y": 10 }, "options": { "tooltip": { @@ -241,10 +241,10 @@ "description": "###### What this is:\n*Transaction-set fetch (get) and share bytes exchanged during ledger close.*\n\n###### How it's computed:\n*Per-second in/out byte rate for the set-get and set-share categories.*\n\n###### Reading it:\n*Some exchange each ledger is normal as peers reconcile transaction sets.*\n\n###### Healthy range:\n*workload-dependent.*\n\n###### Watch for:\n*High set-get (peers frequently missing transaction sets: possible sync delays).*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 8 + "y": 10 }, "options": { "tooltip": { @@ -303,10 +303,10 @@ "description": "###### What this is:\n*Transaction-availability messages: advertisements that a peer has certain transactions, and explicit requests for transaction data.*\n\n###### How it's computed:\n*Per-second in/out message rate for the have-transactions and requested-transactions categories.*\n\n###### Reading it:\n*Compare requested versus have to gauge how well transactions are propagating.*\n\n###### Healthy range:\n*workload-dependent.*\n\n###### Watch for:\n*Requested far exceeding have (peers behind on transaction propagation).*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 16 + "y": 20 }, "options": { "tooltip": { @@ -365,10 +365,10 @@ "description": "###### What this is:\n*Overlay traffic that matches no known message category, in bytes and messages.*\n\n###### How it's computed:\n*Current in/out byte and message counts for the unknown category.*\n\n###### Reading it:\n*Should be at or near zero.*\n\n###### Healthy range:\n*zero.*\n\n###### Watch for:\n*Any sustained non-zero value (protocol version mismatch, corrupted messages, or an unclassified new message type).*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 16 + "y": 20 }, "options": { "tooltip": { @@ -444,10 +444,10 @@ "description": "###### What this is:\n*Proof-path request/response bytes used to verify individual ledger entries without downloading the whole ledger.*\n\n###### How it's computed:\n*Per-second in/out byte rate for the proof-path request and response categories.*\n\n###### Reading it:\n*Rises when peers verify specific state, often during catch-up.*\n\n###### Healthy range:\n*workload-dependent.*\n\n###### Watch for:\n*High sustained request volume (heavy state-verification load).*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 24 + "y": 30 }, "options": { "tooltip": { @@ -506,10 +506,10 @@ "description": "###### What this is:\n*Replay-delta request/response bytes used to efficiently replay ledger state changes during catch-up.*\n\n###### How it's computed:\n*Per-second in/out byte rate for the replay-delta request and response categories.*\n\n###### Reading it:\n*Active during catch-up and replay; quiet when synced.*\n\n###### Healthy range:\n*workload-dependent; low when synced.*\n\n###### Watch for:\n*Continuous replay traffic (node repeatedly replaying rather than staying current).*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 24 + "y": 30 }, "options": { "tooltip": { diff --git a/docker/telemetry/grafana/dashboards/peer-network.json b/docker/telemetry/grafana/dashboards/peer-network.json index 279ab600e2..ea80916e31 100644 --- a/docker/telemetry/grafana/dashboards/peer-network.json +++ b/docker/telemetry/grafana/dashboards/peer-network.json @@ -27,7 +27,7 @@ "description": "**What:** Rate of consensus proposals arriving from connected network peers.\n**How it's computed:** Per-second rate over a 5-minute window, per node.\n**Reading it:** Reflects how many validators' proposals reach this node each round.\n**Healthy range:** Workload-dependent; scales with peer count and network activity.\n**Watch for:** A sudden drop means loss of proposal flow; a sharp sustained spike from a single peer can indicate flooding.\n**Source:** src/xrpld/overlay/detail/PeerImp.cpp:PeerImp::onMessage(TMProposeSet)", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, "y": 1 @@ -72,7 +72,7 @@ "description": "**What:** Rate of ledger validations arriving from connected network peers.\n**How it's computed:** Per-second rate over a 5-minute window, per node.\n**Reading it:** Reflects the volume of validation traffic reaching this node.\n**Healthy range:** Workload-dependent; scales with the number of validators and peers.\n**Watch for:** A drop starves the validation quorum; an abnormal spike from one peer can indicate abuse.\n**Source:** src/xrpld/overlay/detail/PeerImp.cpp:PeerImp::onMessage(TMValidation)", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, "y": 1 @@ -119,7 +119,7 @@ "h": 1, "w": 24, "x": 0, - "y": 9 + "y": 11 }, "collapsed": false, "panels": [], @@ -130,10 +130,10 @@ "description": "**What:** Share of received proposals from trusted validators (in this node's UNL) versus untrusted sources.\n**How it's computed:** Total counts split by trust status over a 5-minute window, shown as proportions.\n**Reading it:** A well-connected node with a good UNL sees a healthy trusted share.\n**Healthy range:** A substantial trusted proportion; exact mix is workload-dependent.\n**Watch for:** A collapse of the trusted share means poor connectivity to UNL validators; a flood of untrusted proposals can indicate abuse.\n**Source:** src/xrpld/overlay/detail/PeerImp.cpp:PeerImp::onMessage(TMProposeSet)", "type": "piechart", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 10 + "y": 12 }, "options": { "tooltip": { @@ -164,10 +164,10 @@ "description": "**What:** Share of received validations from trusted validators (in this node's UNL) versus untrusted sources.\n**How it's computed:** Total counts split by trust status over a 5-minute window, shown as proportions.\n**Reading it:** Confirms the node is hearing from its expected trusted validator set.\n**Healthy range:** A substantial trusted proportion; exact mix is workload-dependent.\n**Watch for:** A shrinking trusted share risks the validation quorum; a surge of untrusted validations can indicate abuse.\n**Source:** src/xrpld/overlay/detail/PeerImp.cpp:PeerImp::onMessage(TMValidation)", "type": "piechart", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 10 + "y": 12 }, "options": { "tooltip": { diff --git a/docker/telemetry/grafana/dashboards/rpc-pathfinding.json b/docker/telemetry/grafana/dashboards/rpc-pathfinding.json index 32c7868e93..e436f3d888 100644 --- a/docker/telemetry/grafana/dashboards/rpc-pathfinding.json +++ b/docker/telemetry/grafana/dashboards/rpc-pathfinding.json @@ -14,7 +14,7 @@ "description": "**What:** Rate of RPC requests handled, counting every HTTP and WebSocket call.\n**How it's computed:** Per-second rate of the RPC-requests counter over 5 minutes.\n**Reading it:** Tracks client demand on the node.\n**Healthy range:** workload-dependent.\n**Watch for:** Sudden sustained spikes (client surge or abusive polling) or a drop to zero (endpoint unavailable).\n**Source:** src/xrpld/rpc/detail/ServerHandler.cpp ServerHandler ctor", "type": "stat", "gridPos": { - "h": 4, + "h": 10, "w": 12, "x": 0, "y": 0 @@ -53,7 +53,7 @@ "h": 1, "w": 24, "x": 0, - "y": 4 + "y": 10 }, "collapsed": false, "panels": [], @@ -64,10 +64,10 @@ "description": "**What:** P95 and P50 of end-to-end RPC handler time (full HTTP handling, not just command execution).\n**How it's computed:** 95th and 50th percentiles over a 5-minute window.\n**Reading it:** Lower is better; P95 shows the slow tail.\n**Healthy range:** workload-dependent; most methods well under a second.\n**Watch for:** Rising P95 while request rate is flat (expensive queries or contention).\n**Source:** src/xrpld/rpc/detail/ServerHandler.cpp ServerHandler ctor", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 5 + "y": 11 }, "options": { "tooltip": { @@ -117,10 +117,10 @@ "description": "**⚠ Instrument mismatch — values unreliable.** Response size is recorded through the millisecond-scaled event histogram (rpc_size_milliseconds_bucket), so byte values saturate at the top time bucket (5000) and the percentiles are not true byte sizes. A dedicated byte-unit histogram is needed to fix this; tracked separately. Treat this panel as indicative only until then.\n\n**What:** P95 and P50 of RPC response payload size in bytes.\n**How it's computed:** 95th and 50th percentiles over the dashboard rate interval.\n**Reading it:** Larger responses cost more bandwidth and CPU to build.\n**Healthy range:** workload-dependent.\n**Watch for:** Large P95 (result-heavy queries such as broad account_tx, or API misuse).\n**Source:** src/xrpld/rpc/detail/ServerHandler.cpp ServerHandler ctor", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 5 + "y": 11 }, "options": { "tooltip": { @@ -170,10 +170,10 @@ "description": "**What:** RPC response-time spread shown as P50, P90, P95, and P99.\n**How it's computed:** Four quantiles of response time over a 5-minute window.\n**Reading it:** A wide gap between P50 and P99 signals a long latency tail.\n**Healthy range:** workload-dependent; quantiles clustered together.\n**Watch for:** P99 pulling far above P50 (bimodal latency or occasional very slow requests).\n**Source:** src/xrpld/rpc/detail/ServerHandler.cpp ServerHandler ctor", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 13 + "y": 21 }, "options": { "tooltip": { @@ -241,7 +241,7 @@ "h": 1, "w": 24, "x": 0, - "y": 21 + "y": 31 }, "collapsed": false, "panels": [], @@ -252,10 +252,10 @@ "description": "**What:** Rate of resource-limit warnings raised when a peer or client exceeds its usage warning threshold.\n**How it's computed:** Per-second rate of the warn counter over 5 minutes.\n**Reading it:** Occasional warnings are normal under load; a rising rate flags aggressive clients.\n**Healthy range:** workload-dependent; low.\n**Watch for:** A climbing warning rate (clients approaching limits, a precursor to drops).\n**Source:** include/xrpl/resource/detail/Logic.h Logic::Stats ctor", "type": "stat", "gridPos": { - "h": 4, + "h": 10, "w": 12, "x": 0, - "y": 22 + "y": 32 }, "options": { "tooltip": { @@ -305,10 +305,10 @@ "description": "**What:** Rate of resource drops: consumers disconnected or blocked for excessive usage.\n**How it's computed:** Per-second rate of the drop counter over 5 minutes.\n**Reading it:** Non-zero means the node is actively rejecting abusive connections.\n**Healthy range:** at or near zero.\n**Watch for:** Sustained non-zero drops (ongoing abuse or a misbehaving client/peer being throttled).\n**Source:** include/xrpl/resource/detail/Logic.h Logic::Stats ctor", "type": "stat", "gridPos": { - "h": 4, + "h": 10, "w": 12, "x": 12, - "y": 22 + "y": 32 }, "options": { "tooltip": { @@ -358,10 +358,10 @@ "description": "**What:** P95 and P50 execution time of the fast pathfinding search (simplified, favoring speed over completeness).\n**How it's computed:** 95th and 50th percentiles over a 5-minute window.\n**Reading it:** Lower is better; fast pathfinding should stay quick.\n**Healthy range:** workload-dependent; typically well below full pathfinding.\n**Watch for:** Rising fast-path latency (pathfinding load or complex order books).\n**Source:** src/xrpld/rpc/detail/PathRequestManager.h ctor", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 26 + "y": 42 }, "options": { "tooltip": { @@ -411,10 +411,10 @@ "description": "**What:** P95 and P50 execution time of the full, exhaustive pathfinding search.\n**How it's computed:** 95th and 50th percentiles over a 5-minute window.\n**Reading it:** Full pathfinding is heavier and slower than fast mode.\n**Healthy range:** workload-dependent.\n**Watch for:** High or rising full-path latency (expensive path computation under subscription load).\n**Source:** src/xrpld/rpc/detail/PathRequestManager.h ctor", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 26 + "y": 42 }, "options": { "tooltip": { @@ -466,7 +466,7 @@ "h": 1, "w": 24, "x": 0, - "y": 34 + "y": 52 }, "collapsed": false, "panels": [], @@ -477,10 +477,10 @@ "description": "**What:** Per-method gRPC call rate for the gRPC read API used by reporting and Clio.\n**How it's computed:** Per-second count of gRPC calls grouped by method.\n**Reading it:** Shows which gRPC methods are called and how often; only populated when the node serves gRPC.\n**Healthy range:** workload-dependent.\n**Watch for:** A single method spiking (heavy reporting load) or unexpected methods appearing.\n**Source:** src/xrpld/rpc/GRPCServer.cpp (gRPC method handlers)", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 35 + "y": 53 }, "options": { "tooltip": { @@ -522,10 +522,10 @@ "description": "**What:** P95 latency per gRPC method.\n**How it's computed:** 95th percentile of gRPC call duration per method over 5 minutes.\n**Reading it:** Identifies slow gRPC read paths.\n**Healthy range:** workload-dependent.\n**Watch for:** One method's P95 rising (expensive ledger reads or backend pressure).\n**Source:** src/xrpld/rpc/GRPCServer.cpp (gRPC method handlers)", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 35 + "y": 53 }, "options": { "tooltip": { @@ -567,10 +567,10 @@ "description": "**What:** Rate of gRPC calls broken down by result status: success, error, resource-exhausted, failed-precondition.\n**How it's computed:** Per-second count of gRPC calls grouped by status.\n**Reading it:** Errors and resource-exhausted should be a small share.\n**Healthy range:** workload-dependent; mostly success.\n**Watch for:** Rising error or resource-exhausted rate (clients hitting limits or failing reads).\n**Source:** src/xrpld/rpc/GRPCServer.cpp (gRPC method handlers)", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 43 + "y": 63 }, "options": { "tooltip": { @@ -612,10 +612,10 @@ "description": "**What:** P95 and P50 of per-request path computation time.\n**How it's computed:** 95th and 50th percentiles of path-computation duration over 5 minutes.\n**Reading it:** Complements the fast/full timers with per-request visibility; populated under book/path RPC load.\n**Healthy range:** workload-dependent.\n**Watch for:** Rising compute time (complex paths or heavy pathfinding demand).\n**Source:** src/xrpld/rpc/detail/PathRequest.cpp PathRequest::doUpdate", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 43 + "y": 63 }, "options": { "tooltip": { @@ -665,10 +665,10 @@ "description": "**What:** Rate of client path requests and of path-discovery passes.\n**How it's computed:** Per-second count of path-request and path-discovery operations.\n**Reading it:** Shows pathfinding demand and the discovery cost driver for subscription-heavy nodes.\n**Healthy range:** workload-dependent.\n**Watch for:** Discovery rate climbing (subscription load driving repeated path discovery).\n**Source:** src/xrpld/rpc/handlers/orderbook/PathFind.cpp doPathFind; src/xrpld/rpc/detail/PathRequest.cpp PathRequest::findPaths", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 51 + "y": 73 }, "options": { "tooltip": { diff --git a/docker/telemetry/grafana/dashboards/rpc-performance.json b/docker/telemetry/grafana/dashboards/rpc-performance.json index aa8bac5c83..c532fa8d38 100644 --- a/docker/telemetry/grafana/dashboards/rpc-performance.json +++ b/docker/telemetry/grafana/dashboards/rpc-performance.json @@ -13,7 +13,7 @@ "description": "**What:** Percentage of each RPC command's calls that ended in an error.\n**How it's computed:** Error calls divided by total calls per command over a 5-minute window, per node.\n**Reading it:** Green below 1%, yellow 1-5%, red above 5%.\n**Healthy range:** Below 1% for healthy commands.\n**Watch for:** A sustained error spike on one command is consistent with a client probing or misusing that endpoint.\n**Source:** src/xrpld/rpc/detail/RPCHandler.cpp:callMethod", "type": "bargauge", "gridPos": { - "h": 4, + "h": 10, "w": 12, "x": 0, "y": 0 @@ -66,7 +66,7 @@ "description": "**What:** The ten most-called RPC commands by total invocations recently.\n**How it's computed:** Ranked total counts per command over a 5-minute window, per node.\n**Reading it:** Identifies the hottest API endpoints driving node load.\n**Healthy range:** Workload-dependent.\n**Watch for:** An unexpected command dominating the ranking can indicate scripted abuse or a misbehaving client.\n**Source:** src/xrpld/rpc/detail/RPCHandler.cpp:callMethod", "type": "bargauge", "gridPos": { - "h": 4, + "h": 10, "w": 12, "x": 12, "y": 0 @@ -103,10 +103,10 @@ "description": "**What:** Rate of incoming WebSocket RPC messages the server processes.\n**How it's computed:** Per-second rate over a 5-minute window, per node.\n**Reading it:** Non-zero only when clients connect over WebSocket rather than HTTP.\n**Healthy range:** Zero is normal for HTTP-only deployments; otherwise workload-dependent.\n**Watch for:** An unexpected surge points to a chatty or abusive WebSocket client.\n**Source:** src/xrpld/rpc/detail/ServerHandler.cpp:ServerHandler::processSession", "type": "stat", "gridPos": { - "h": 4, + "h": 10, "w": 12, "x": 0, - "y": 4 + "y": 10 }, "options": { "tooltip": { @@ -140,10 +140,10 @@ "description": "**What:** Per-command throughput of RPC calls the node serves (e.g. server_info, submit, account_info).\n**How it's computed:** Per-second rate split by command name over a 5-minute window, per node.\n**Reading it:** Shows which API endpoints drive load and how demand shifts over time.\n**Healthy range:** Workload-dependent; varies with client mix.\n**Watch for:** A sudden spike on one command can indicate a runaway client or scripted abuse.\n**Source:** src/xrpld/rpc/detail/RPCHandler.cpp:callMethod", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 0, - "y": 8 + "x": 12, + "y": 10 }, "options": { "tooltip": { @@ -185,10 +185,10 @@ "description": "**What:** 95th-percentile response time for each RPC command.\n**How it's computed:** 95th percentile of per-command durations over a 5-minute window, per node.\n**Reading it:** Read-only lookups should be fast; heavy commands like path finding are naturally slower.\n**Healthy range:** Sub-second for most commands; command-dependent.\n**Watch for:** Rising p95 on a normally fast command signals contention or an expensive query pattern.\n**Source:** src/xrpld/rpc/detail/RPCHandler.cpp:callMethod", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 12, - "y": 8 + "x": 0, + "y": 20 }, "options": { "tooltip": { @@ -230,10 +230,10 @@ "description": "**What:** Distribution of RPC response times over time across all commands.\n**How it's computed:** Counts of requests by duration band in each 5-minute window, per node.\n**Reading it:** A single tight band is healthy; brighter cells mark the common latency.\n**Healthy range:** Concentrated at low latency; command-dependent.\n**Watch for:** Two separate bands (bimodal latency) reveal a slow path affecting some requests.\n**Source:** src/xrpld/rpc/detail/RPCHandler.cpp:callMethod", "type": "heatmap", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 0, - "y": 16 + "x": 12, + "y": 20 }, "options": { "tooltip": { @@ -272,10 +272,10 @@ "description": "**What:** Aggregate request flow through two pipeline layers: the outer HTTP handler that accepts connections and the inner layer that parses and dispatches.\n**How it's computed:** Per-second rate of each layer over a 5-minute window, per node.\n**Reading it:** The two lines should overlap when requests flow cleanly.\n**Healthy range:** Workload-dependent; both layers tracking each other.\n**Watch for:** A gap means requests are being queued or rejected before dispatch.\n**Source:** src/xrpld/rpc/detail/ServerHandler.cpp:ServerHandler::processSession, ServerHandler::processRequest", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 12, - "y": 16 + "x": 0, + "y": 30 }, "options": { "tooltip": { @@ -325,10 +325,10 @@ "description": "**What:** Aggregate rate of successful versus failed RPC commands across all types.\n**How it's computed:** Per-second rate of each outcome over a 5-minute window, per node.\n**Reading it:** Success should dominate; the error line should stay low.\n**Healthy range:** Error rate near zero; success rate is workload-dependent.\n**Watch for:** A sustained rise in the error line warrants drilling into the per-command breakdown.\n**Source:** src/xrpld/rpc/detail/RPCHandler.cpp:callMethod", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 0, - "y": 24 + "x": 12, + "y": 30 }, "options": { "tooltip": { @@ -378,10 +378,10 @@ "description": "**What:** RPC traffic grouped by its resource-cost category, distinguishing cheap lookups from expensive or malformed requests.\n**How it's computed:** Per-second rate per cost category over a 5-minute window.\n**Reading it:** Low-cost categories should dominate.\n**Healthy range:** Mostly low-cost traffic; workload-dependent.\n**Watch for:** A rising share of high-cost or malformed categories points to problematic or abusive clients.\n**Source:** src/xrpld/rpc/detail/RPCHandler.cpp:callMethod", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 24, "x": 0, - "y": 32 + "y": 40 }, "options": { "tooltip": { @@ -428,10 +428,10 @@ "description": "**What:** Rate of batched RPC requests versus single requests.\n**How it's computed:** Per-second rate of each over a 5-minute window, per node.\n**Reading it:** Most traffic is typically single requests.\n**Healthy range:** Workload-dependent; batch share usually small.\n**Watch for:** A high batch rate may indicate bulk-automation clients or an attempt to amplify load per connection.\n**Source:** src/xrpld/rpc/detail/ServerHandler.cpp:ServerHandler::processRequest", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 40 + "y": 50 }, "options": { "tooltip": { diff --git a/docker/telemetry/grafana/dashboards/transaction-overview.json b/docker/telemetry/grafana/dashboards/transaction-overview.json index 565edda59a..d142ae23d5 100644 --- a/docker/telemetry/grafana/dashboards/transaction-overview.json +++ b/docker/telemetry/grafana/dashboards/transaction-overview.json @@ -13,7 +13,7 @@ "description": "**What:** Rate of transactor-stage applies that returned a non-success result while building a ledger.\n**How it's computed:** Per-second rate of tx.transactor spans at stage=apply whose ter_result is not tesSUCCESS, per node.\n**Reading it:** Green below 1/s, yellow 1-5/s, red above 5/s.\n**Healthy range:** A steady low non-zero rate is normal (e.g. tefPAST_SEQ, tecUNFUNDED and other benign conflicts).\n**Watch for:** A sustained jump above the normal baseline can indicate malformed transaction floods or a systemic issue.\n**Source:** src/libxrpl/tx/Transactor.cpp:Transactor::operator()", "type": "stat", "gridPos": { - "h": 4, + "h": 10, "w": 12, "x": 0, "y": 0 @@ -67,10 +67,10 @@ "description": "**What:** Rate of transactions entering the processing pipeline (submitted locally or relayed by peers) versus raw transaction messages arriving from peers before deduplication.\n**How it's computed:** Per-second rate of each over a 5-minute window, per node.\n**Reading it:** The received line sits above the processed line by the volume of duplicates filtered out.\n**Healthy range:** Workload-dependent; scales with network transaction volume.\n**Watch for:** A large and growing gap means heavy duplicate traffic; a processed-rate collapse means submissions are not being handled.\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp:NetworkOPsImp::processTransaction, src/xrpld/overlay/detail/PeerImp.cpp:PeerImp::handleTransaction", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, - "x": 0, - "y": 4 + "x": 12, + "y": 0 }, "options": { "tooltip": { @@ -120,10 +120,10 @@ "description": "**What:** Per-transaction-type processing latency through the pipeline.\n**How it's computed:** 95th and 50th percentile of per-type durations over a 5-minute window, per node.\n**Reading it:** Simple payments are fast; complex types like offers or AMM operations run longer.\n**Healthy range:** Sub-millisecond to a few milliseconds; type-dependent.\n**Watch for:** A latency climb for one type signals contention or expensive processing specific to it.\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp:NetworkOPsImp::processTransaction", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 24, "x": 0, - "y": 12 + "y": 10 }, "options": { "tooltip": { @@ -178,7 +178,7 @@ "description": "**What:** Split of processed transactions by origin: submitted locally versus relayed from peers.\n**How it's computed:** Total counts split by origin over a 5-minute window, shown as proportions.\n**Reading it:** Most nodes see mostly relayed traffic; a submission endpoint sees more local.\n**Healthy range:** Workload-dependent.\n**Watch for:** A surge in local submissions on a node not meant to accept them can indicate misuse.\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp:NetworkOPsImp::processTransaction", "type": "piechart", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, "y": 20 @@ -212,7 +212,7 @@ "description": "**What:** Raw transaction messages received from peers, split by whether they were suppressed as duplicates before processing.\n**How it's computed:** Per-second rate split by suppressed flag over a 5-minute window, per node.\n**Reading it:** A healthy relay network produces a steady suppressed share as duplicates are filtered.\n**Healthy range:** Workload-dependent; suppression is expected and normal.\n**Watch for:** A sharp rise in the suppressed line can reflect gossip amplification or a peer replaying transactions.\n**Source:** src/xrpld/overlay/detail/PeerImp.cpp:PeerImp::handleTransaction", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, "y": 20 @@ -257,10 +257,10 @@ "description": "**What:** Distribution of transaction processing times over time.\n**How it's computed:** Counts of transactions by duration band in each 5-minute window, per node.\n**Reading it:** A single tight band is healthy; brighter cells mark the common processing time.\n**Healthy range:** Concentrated at low latency.\n**Watch for:** A second high-latency band reveals a subset of transactions that are consistently slow.\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp:NetworkOPsImp::processTransaction", "type": "heatmap", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 28 + "y": 30 }, "options": { "tooltip": { @@ -299,10 +299,10 @@ "description": "**What:** Time to apply the agreed transaction set into a new ledger.\n**How it's computed:** 95th and 50th percentile of per-ledger apply durations over a 5-minute window, per node.\n**Reading it:** Scales with the number and complexity of transactions per ledger.\n**Healthy range:** Tens to a few hundred milliseconds under normal load.\n**Watch for:** Sustained high durations indicate heavy transaction sets or expensive processing.\n**Source:** src/xrpld/app/ledger/detail/BuildLedger.cpp:applyTransactions", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 28 + "y": 30 }, "options": { "tooltip": { @@ -354,7 +354,7 @@ "h": 1, "w": 24, "x": 0, - "y": 36 + "y": 40 }, "collapsed": false, "panels": [], @@ -365,10 +365,10 @@ "description": "**What:** Processing rate broken down by transaction type (Payment, OfferCreate, AMM operations, etc.).\n**How it's computed:** Per-second rate per type over a 5-minute window.\n**Reading it:** Shows the transaction mix and how it shifts over time.\n**Healthy range:** Workload-dependent; usually Payment-heavy.\n**Watch for:** A sudden burst of one type is a leading indicator of spam or fee escalation.\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp:NetworkOPsImp::processTransaction", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 24, "x": 0, - "y": 37 + "y": 41 }, "options": { "tooltip": { @@ -415,10 +415,10 @@ "description": "**What:** Non-success result codes broken down by transaction type, showing which types fail and how.\n**How it's computed:** Per-second rate per type and result code over a 5-minute window (successes excluded).\n**Reading it:** Occasional failures are normal; watch which types and codes dominate.\n**Healthy range:** Low and workload-dependent.\n**Watch for:** A concentration of one failure code on one type can indicate a broken client or targeted spam.\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp:NetworkOPsImp::processTransaction", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 24, "x": 0, - "y": 45 + "y": 51 }, "options": { "tooltip": { @@ -461,14 +461,14 @@ "id": 10 }, { - "title": "TxQ Accept: Applied Ratio per Node (State Timeline) [$xrpl_network_type]", + "title": "TxQ Accept: Applied Ratio per Node (State Timeline)", "description": "**What:** Applied fraction of TxQ accepts per node over time: the share of queued transactions that were included in a ledger versus removed on failure.\n**How it's computed:** Per node, applied accepts divided by applied-plus-failed accepts over a 5-minute window.\n**Reading it:** Green (>=90% applied) is a healthy drain; yellow is degraded; red means accepts are mostly failing.\n**Healthy range:** At or near 100% applied when the queue is draining healthily; workload-dependent.\n**Watch for:** A node dropping into yellow or red, which signals queue pressure, under-bidding, or fee escalation on that node.\n**Source:** src/xrpld/app/misc/detail/TxQ.cpp:TxQ::accept", "type": "state-timeline", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 53 + "y": 61 }, "options": { "mergeValues": true, @@ -520,9 +520,6 @@ }, "overrides": [] }, - "repeat": "xrpl_network_type", - "repeatDirection": "h", - "maxPerRow": 2, "id": 11 }, { @@ -530,10 +527,10 @@ "description": "**What:** Per-type execution time of the transactor that actually applies each transaction's effects.\n**How it's computed:** 95th percentile of per-type durations over a 5-minute window, per node.\n**Reading it:** Reveals which transaction types are most expensive to execute.\n**Healthy range:** Sub-millisecond to a few milliseconds; type-dependent.\n**Watch for:** A climbing figure for one type points to expensive processing or contention for that type.\n**Source:** src/libxrpl/tx/Transactor.cpp:Transactor::operator()", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 24, "x": 0, - "y": 61 + "y": 71 }, "options": { "tooltip": { @@ -582,7 +579,7 @@ "h": 1, "w": 24, "x": 0, - "y": 69 + "y": 81 }, "collapsed": false, "panels": [], @@ -593,10 +590,10 @@ "description": "**What:** Rate at which transactions are placed into the queue, split by transaction type.\n**How it's computed:** Per-second rate per type over a 5-minute window, per node.\n**Reading it:** Shows which types make up inbound demand and how the mix shifts as the queue fills.\n**Healthy range:** Workload-dependent.\n**Watch for:** A spam burst of one type here is a leading indicator of fee escalation.\n**Source:** src/xrpld/app/misc/detail/TxQ.cpp:TxQ::apply", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 70 + "y": 82 }, "options": { "tooltip": { @@ -638,10 +635,10 @@ "description": "**What:** Time spent draining queued transactions into a newly closed ledger.\n**How it's computed:** 95th and 50th percentile of per-ledger drain durations over a 5-minute window, per node.\n**Reading it:** Rises as the queue holds more transactions to process at close.\n**Healthy range:** A few milliseconds when the queue is light.\n**Watch for:** Rising drain time signals queue pressure at ledger close.\n**Source:** src/xrpld/app/misc/detail/TxQ.cpp:TxQ::accept", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 12, - "y": 70 + "y": 82 }, "options": { "tooltip": { @@ -691,10 +688,10 @@ "description": "**What:** Rate at which expired transactions are removed from the queue each ledger.\n**How it's computed:** Per-second rate over a 5-minute window, per node.\n**Reading it:** Normally low; entries expire when submitters underbid the current fee.\n**Healthy range:** Near zero in calm conditions.\n**Watch for:** A rising rate means submitters abandoned under-fee transactions, a demand-frustration signal distinct from throughput.\n**Source:** src/xrpld/app/misc/detail/TxQ.cpp:TxQ::processClosedLedger", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 12, "x": 0, - "y": 78 + "y": 92 }, "options": { "tooltip": { @@ -736,10 +733,10 @@ "description": "**What:** Throughput of each apply-pipeline stage (preflight, preclaim, apply), showing where transactions drop out.\n**How it's computed:** Per-second rate per stage over a 5-minute window, per node.\n**Reading it:** A decline from earlier to later stages shows where transactions are filtered.\n**Healthy range:** Workload-dependent; later stages sit at or below earlier ones.\n**Watch for:** A large early-stage drop means many transactions fail basic checks, consistent with malformed floods.\n**Source:** src/libxrpl/tx/Transactor.cpp:Transactor::operator()", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 24, "x": 0, - "y": 86 + "y": 102 }, "options": { "tooltip": { @@ -788,7 +785,7 @@ "h": 1, "w": 24, "x": 0, - "y": 94 + "y": 112 }, "collapsed": false, "panels": [], @@ -799,10 +796,10 @@ "description": "**What:** 95th-percentile duration of each apply-pipeline stage (preflight, preclaim, apply).\n**How it's computed:** 95th percentile of per-stage durations over a 5-minute window, per node.\n**Reading it:** Isolates which stage dominates transaction processing time.\n**Healthy range:** Sub-millisecond to a few milliseconds per stage.\n**Watch for:** A stage that grows disproportionately points to a bottleneck in that step.\n**Source:** src/libxrpl/tx/Transactor.cpp:Transactor::operator()", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 24, "x": 0, - "y": 95 + "y": 113 }, "options": { "tooltip": { @@ -849,10 +846,10 @@ "description": "**What:** Rate of pipeline spans ending in a non-success result, split by stage (preflight, preclaim, apply).\n**How it's computed:** Per-second rate of non-success outcomes per stage over a 5-minute window, per node.\n**Reading it:** Shows whether failures concentrate early (basic checks) or late (application).\n**Healthy range:** Low and workload-dependent.\n**Watch for:** A surge of early-stage failures is consistent with malformed or spam transaction floods.\n**Source:** src/libxrpl/tx/Transactor.cpp:Transactor::operator()", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 24, "x": 0, - "y": 103 + "y": 123 }, "options": { "tooltip": { @@ -899,10 +896,10 @@ "description": "**What:** 95th-percentile stage duration broken down by both transaction type and pipeline stage.\n**How it's computed:** 95th percentile of durations per type and stage over a 5-minute window, per node.\n**Reading it:** For each transaction type, shows which stage dominates its latency.\n**Healthy range:** Sub-millisecond to a few milliseconds; type- and stage-dependent.\n**Watch for:** A specific type-and-stage combination climbing points to a targeted expensive or abusive pattern.\n**Source:** src/libxrpl/tx/Transactor.cpp:Transactor::operator()", "type": "timeseries", "gridPos": { - "h": 8, + "h": 10, "w": 24, "x": 0, - "y": 111 + "y": 133 }, "options": { "tooltip": {