mirror of
https://github.com/XRPLF/rippled.git
synced 2026-08-21 14:20:56 +00:00
docs(telemetry): drop (System Metrics) suffix, retag, rewrite descriptions to describe data not pipeline
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -2,7 +2,7 @@
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"description": "Ledger data exchange and object fetch traffic from beast::insight System Metrics. Covers ledger sync, node data retrieval, and transaction set exchange. Requires [insight] server=otel in rippled config.",
|
||||
"description": "Ledger data exchange and object-fetch traffic between this node and its peers: ledger sync, tree-node retrieval, and transaction-set exchange. Use it to see how much ledger data the node is pulling or serving and to spot catch-up activity.",
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
@@ -11,7 +11,7 @@
|
||||
"panels": [
|
||||
{
|
||||
"title": "Ledger Data Exchange (Bytes In)",
|
||||
"description": "Inbound bytes for ledger data sub-categories. 'ledger_data' = aggregated ledger data, sub-types include Transaction_Set_candidate (proposed tx sets), Transaction_Node (tx tree nodes), and Account_State_Node (state tree nodes). High Account_State_Node traffic indicates state sync; high Transaction_Set_candidate indicates consensus catch-up. Sourced from TrafficCount.h ledger_data_* categories.",
|
||||
"description": "**What:** Inbound bytes for ledger-data message categories, split into aggregate get/share plus the transaction-set, transaction-node, and account-state-node sub-types the node receives from peers.\n**How it's computed:** Per-category inbound byte counters, shown at their latest cumulative value per node.\n**Reading it:** Normally low and flat once synced. Account-state-node traffic dominates during state sync; transaction-set-candidate traffic dominates during consensus catch-up.\n**Healthy range:** workload-dependent; low and steady on a synced node.\n**Watch for:** Sustained high account-state or tx-node inbound bytes on a node that should be caught up (repeated re-sync, missing history), or a single peer driving all traffic.\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -99,7 +99,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Ledger Share/Get Traffic (Bytes)",
|
||||
"description": "Legacy ledger share and get traffic by sub-type. These are the older ledger fetch protocol categories (as opposed to ledger_data_* which is the newer protocol). Sub-types: Transaction_Set_candidate, Transaction_node, Account_State_node, plus aggregate ledger_share and ledger_get. Sourced from TrafficCount.h ledger_* categories.",
|
||||
"description": "**What:** Inbound bytes for the older ledger share/get message categories and their tx-set, tx-node, and account-state sub-types.\n**How it's computed:** Per-category inbound byte counters at their latest value per node.\n**Reading it:** Usually small; these legacy categories carry ledger-fetch traffic for peers using the older protocol.\n**Healthy range:** workload-dependent; low on a synced node.\n**Watch for:** Large sustained volumes indicating heavy fetch load or a peer repeatedly requesting the same data.\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -187,7 +187,7 @@
|
||||
},
|
||||
{
|
||||
"title": "GetObject Traffic by Type (Bytes In)",
|
||||
"description": "Object fetch traffic by object type. GetObject is the protocol for fetching specific SHAMap nodes. Types: Ledger (full ledger headers), Transaction (individual txs), Transaction_node (tx tree nodes), Account_State_node (state tree nodes), CAS (Content Addressable Storage objects), Fetch_Pack (batch fetch during catch-up), Transactions (bulk tx fetch). High Fetch_Pack traffic indicates a node is catching up. Sourced from TrafficCount.h getobject_* categories.",
|
||||
"description": "**What:** Inbound bytes for object-fetch traffic broken down by object type: ledger headers, individual transactions, transaction-tree nodes, and state-tree nodes.\n**How it's computed:** Per-type inbound byte counters at their latest value per node.\n**Reading it:** Small during steady state; grows when the node fetches missing tree nodes.\n**Healthy range:** workload-dependent; low when synced.\n**Watch for:** A large share on state/tx nodes for long periods (persistent gap-filling), meaning the node keeps catching up.\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -275,7 +275,7 @@
|
||||
},
|
||||
{
|
||||
"title": "GetObject Aggregate & Special Types (Bytes In)",
|
||||
"description": "Aggregate getobject traffic plus special categories: CAS (Content Addressable Storage) for SHAMap node fetch, Fetch_Pack for bulk batch downloads during catch-up, Transactions for bulk tx fetch, and the aggregate getobject_get/getobject_share totals. Sourced from TrafficCount.h getobject_* categories.",
|
||||
"description": "**What:** Aggregate object-fetch inbound bytes plus special buckets: content-addressed storage fetches, bulk fetch-pack downloads used during catch-up, and bulk transaction fetches.\n**How it's computed:** Per-category inbound byte counters at their latest value per node.\n**Reading it:** Fetch-pack rises sharply while catching up a range of ledgers; near zero when fully synced.\n**Healthy range:** workload-dependent; low when synced.\n**Watch for:** Continuous fetch-pack traffic (node never fully catches up) or unexpectedly high content-store volume.\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -356,7 +356,7 @@
|
||||
},
|
||||
{
|
||||
"title": "GetObject Messages by Type",
|
||||
"description": "Message counts for object fetch operations. Shows how many individual fetch requests and responses are exchanged per type. High message counts with low byte counts indicate small object fetches; the inverse indicates large batch transfers. Sourced from TrafficCount.h getobject_* categories.",
|
||||
"description": "**What:** Count of individual object-fetch request/response messages per object type.\n**How it's computed:** Per-type inbound message counters at their latest value per node.\n**Reading it:** Many messages with few bytes means small piecemeal fetches; few messages with many bytes means large batch transfers.\n**Healthy range:** workload-dependent.\n**Watch for:** High message counts with tiny payloads sustained over time (inefficient per-node fetching).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -437,7 +437,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Overlay Traffic Heatmap (All Categories, Bytes In)",
|
||||
"description": "Bar gauge showing all overlay traffic categories ranked by inbound bytes. Provides a complete at-a-glance view of which protocol message types consume the most bandwidth across all 57+ traffic categories. Sourced from all TrafficCount.h categories via wildcard match.",
|
||||
"description": "**What:** All overlay traffic categories ranked by inbound bytes, giving an at-a-glance view of which message types consume the most receive bandwidth.\n**How it's computed:** Top categories by latest inbound byte value across all traffic categories.\n**Reading it:** The longest bars are the biggest bandwidth consumers; on a synced node transactions, proposals, and validations usually lead.\n**Healthy range:** workload-dependent.\n**Watch for:** A single ledger-data or fetch category dominating (ongoing sync) or an unexpected category topping the list.\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "bargauge",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -493,7 +493,7 @@
|
||||
}
|
||||
],
|
||||
"schemaVersion": 39,
|
||||
"tags": ["statsd", "ledger", "sync", "telemetry"],
|
||||
"tags": ["ledger", "sync"],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
@@ -582,6 +582,6 @@
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"title": "Ledger Data & Sync (System Metrics)",
|
||||
"title": "Ledger Data & Sync",
|
||||
"uid": "xrpld-system-ledger-sync"
|
||||
}
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"description": "Network traffic and peer metrics from beast::insight System Metrics. Requires [insight] server=otel in rippled config.",
|
||||
"description": "Peer connectivity and overlay bandwidth for this node: peer counts, disconnects, total bytes and messages exchanged, and the transaction/proposal/validation traffic that flows across the peer-to-peer network.",
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
@@ -11,7 +11,7 @@
|
||||
"panels": [
|
||||
{
|
||||
"title": "Active Peers",
|
||||
"description": "Number of active inbound and outbound peer connections. Sourced from Peer_Finder.Active_Inbound_Peers and Peer_Finder.Active_Outbound_Peers gauges (PeerfinderManager.cpp). A healthy mainnet node typically has 10-21 outbound and 0-85 inbound peers depending on configuration.",
|
||||
"description": "**What:** Number of active inbound and outbound peer connections the node currently holds.\n**How it's computed:** Current value of the inbound and outbound active-peer counts per node.\n**Reading it:** Outbound is what the node dials out; inbound is what others open to it. Both should be stable.\n**Healthy range:** roughly 10-21 outbound and 0-85 inbound on mainnet, depending on config.\n**Watch for:** Outbound dropping toward zero (isolation) or inbound pinned at the limit with churn (connection pressure).\n**Source:** src/xrpld/peerfinder/detail/PeerfinderManager.cpp Logic Stats ctor",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -57,7 +57,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Peer Disconnects",
|
||||
"description": "Cumulative count of peer disconnections. Sourced from the Overlay.Peer_Disconnects gauge (OverlayImpl.h). A rising trend indicates network instability, aggressive peer management, or resource exhaustion causing connection drops.",
|
||||
"description": "**What:** Cumulative count of peer connections that have dropped.\n**How it's computed:** Running total of disconnect events per node.\n**Reading it:** A flat or slowly rising line is normal; the slope matters more than the absolute value.\n**Healthy range:** workload-dependent; slow, steady growth.\n**Watch for:** Sharp step-ups in the slope (network instability, resource exhaustion, or many peers dropping the node at once).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.h OverlayImpl::Stats ctor",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -96,7 +96,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Total Network Bytes",
|
||||
"description": "Total bytes sent and received across all peer connections. Sourced from the total.Bytes_In and total.Bytes_Out traffic category gauges (OverlayImpl.h). Provides a high-level view of network bandwidth consumption.",
|
||||
"description": "**What:** Total bytes received and sent across all peer connections.\n**How it's computed:** Current cumulative in/out byte totals per node.\n**Reading it:** Overall bandwidth footprint; in and out usually track network activity together.\n**Healthy range:** workload-dependent.\n**Watch for:** Sudden sustained jumps not matched by ledger or transaction activity (relay storms or a noisy peer).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -142,7 +142,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Total Network Messages",
|
||||
"description": "Total messages sent and received across all peer connections. Sourced from the total.Messages_In and total.Messages_Out traffic category gauges (OverlayImpl.h). Shows the overall message throughput of the overlay network.",
|
||||
"description": "**What:** Total messages received and sent across all peer connections.\n**How it's computed:** Current cumulative in/out message totals per node.\n**Reading it:** Overall message throughput of the overlay; complements the byte totals.\n**Healthy range:** workload-dependent.\n**Watch for:** Message count climbing far faster than bytes (many tiny messages, possible flooding).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -188,7 +188,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Transaction Traffic",
|
||||
"description": "Bytes and messages for transaction-related overlay traffic. Includes the transactions traffic category (OverlayImpl/TrafficCount.h). Spikes indicate high transaction volume on the network or transaction flooding.",
|
||||
"description": "**What:** Transaction relay messages in and out, plus duplicate transaction messages received.\n**How it's computed:** Current message counts for the transaction and transaction-duplicate categories.\n**Reading it:** In/out rise with network transaction volume; duplicates are transactions the node already had.\n**Healthy range:** workload-dependent; duplicates a modest fraction of inbound.\n**Watch for:** Duplicate inbound approaching or exceeding unique inbound (redundant relay), or a sharp spike suggesting transaction flooding.\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -241,7 +241,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Proposal Traffic",
|
||||
"description": "Messages for consensus proposal overlay traffic. Includes proposals, proposals_untrusted, and proposals_duplicate categories (TrafficCount.h). High untrusted or duplicate counts may indicate UNL misconfiguration or network spam.",
|
||||
"description": "**What:** Consensus proposal messages in/out, plus untrusted and duplicate proposal messages received.\n**How it's computed:** Current message counts for the proposal, proposal-untrusted, and proposal-duplicate categories.\n**Reading it:** Trusted in/out track consensus rounds; untrusted come from validators not on this node's trusted list.\n**Healthy range:** workload-dependent; untrusted and duplicates low relative to trusted.\n**Watch for:** High untrusted (trusted-list misconfiguration) or high duplicates (inefficient relay or proposal spam).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -301,7 +301,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Validation Traffic",
|
||||
"description": "Messages for validation overlay traffic. Includes validations, validations_untrusted, and validations_duplicate categories (TrafficCount.h). Monitoring trusted vs untrusted validation traffic helps detect UNL health issues.",
|
||||
"description": "**What:** Validation messages in/out, plus untrusted and duplicate validation messages received.\n**How it's computed:** Current message counts for the validation, validation-untrusted, and validation-duplicate categories.\n**Reading it:** Trusted validations should arrive steadily each ledger; untrusted come from non-trusted validators.\n**Healthy range:** workload-dependent; untrusted and duplicates low relative to trusted.\n**Watch for:** Rising untrusted or duplicate validations (trusted-list health issues or validation spam).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -361,7 +361,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Overlay Traffic by Category (Bytes In)",
|
||||
"description": "Top traffic categories by inbound bytes. Includes all 57 overlay traffic categories from TrafficCount.h. Shows which protocol message types consume the most bandwidth. Categories include transactions, proposals, validations, ledger data, getobject, and overlay overhead.",
|
||||
"description": "**What:** Top overlay traffic categories ranked by inbound bytes, excluding the all-traffic total.\n**How it's computed:** Top categories by latest inbound byte value per node.\n**Reading it:** Shows which message types dominate receive bandwidth right now.\n**Healthy range:** workload-dependent; transactions, proposals, and validations typically lead on a synced node.\n**Watch for:** A fetch or ledger-data category topping the list (sync activity) or an unexpected category dominating.\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "bargauge",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -658,7 +658,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Duplicate Traffic (Wasted Bandwidth)",
|
||||
"description": "Rate of duplicate overlay traffic across transaction, proposal, and validation categories. Duplicate messages are messages the node has already seen and discards. High duplicate rates indicate inefficient message routing or network topology issues causing redundant relays.",
|
||||
"description": "**What:** Throughput of duplicate transaction, proposal, and validation traffic: messages the node had already seen and discarded.\n**How it's computed:** Per-second rate of the duplicate byte counters for each category, in and out.\n**Reading it:** Lower is better; this is bandwidth spent on redundant relays.\n**Healthy range:** workload-dependent; a small fraction of total traffic.\n**Watch for:** Duplicate rate climbing toward the same order as useful traffic (poor relay topology or redundant flooding).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -732,7 +732,7 @@
|
||||
},
|
||||
{
|
||||
"title": "All Traffic Categories (Detail)",
|
||||
"description": "Top 15 traffic categories by inbound byte rate, excluding the total aggregate. Provides a detailed timeseries view of which overlay message types are consuming the most bandwidth over time. Complements the bar gauge snapshot view in the Overlay Traffic panel.",
|
||||
"description": "**What:** The busiest overlay categories by inbound byte rate over time, excluding the all-traffic total.\n**How it's computed:** Per-second inbound byte rate for the top categories, ranked.\n**Reading it:** Time-series companion to the category bar view; shows how the traffic mix shifts over the window.\n**Healthy range:** workload-dependent.\n**Watch for:** A category ramping up and staying high, or the mix suddenly changing (sync, spam, or a misbehaving peer).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -771,7 +771,7 @@
|
||||
}
|
||||
],
|
||||
"schemaVersion": 39,
|
||||
"tags": ["statsd", "network", "telemetry"],
|
||||
"tags": ["network"],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
@@ -860,6 +860,6 @@
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"title": "Network Traffic (System Metrics)",
|
||||
"title": "Network Traffic",
|
||||
"uid": "xrpld-system-network"
|
||||
}
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"description": "Node health metrics from beast::insight System Metrics. Requires [insight] server=otel in rippled config.",
|
||||
"description": "Overall health of this node: how current its ledgers are, how much time it spends fully synced, I/O and job-queue responsiveness, ledger-fetch and history-integrity signals, and job execution and wait timings.",
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
@@ -11,7 +11,7 @@
|
||||
"panels": [
|
||||
{
|
||||
"title": "Validated Ledger Age",
|
||||
"description": "Age of the most recently validated ledger in seconds. Sourced from the LedgerMaster.Validated_Ledger_Age gauge (LedgerMaster.h) which is updated every collection interval via the insight hook. Values above 20s indicate the node is falling behind the network.",
|
||||
"description": "**What:** Seconds since the most recently validated ledger, i.e. how far behind the network the node is.\n**How it's computed:** Current value of the validated-ledger-age gauge per node.\n**Reading it:** Lower is better; a healthy node stays within a few ledger-close intervals.\n**Healthy range:** under ~10s (the network closes a ledger every 3-5s).\n**Watch for:** Above 20s or climbing steadily (the node is falling behind or has lost sync).\n**Source:** src/xrpld/app/ledger/LedgerMaster.h LedgerMaster::Stats ctor",
|
||||
"type": "stat",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -59,7 +59,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Published Ledger Age",
|
||||
"description": "Age of the most recently published ledger in seconds. Sourced from the LedgerMaster.Published_Ledger_Age gauge (LedgerMaster.h). Published ledger age should track close to validated ledger age. A growing gap indicates publish pipeline backlog.",
|
||||
"description": "**What:** Seconds since the most recently published ledger (the ledger exposed to clients and subscribers).\n**How it's computed:** Current value of the published-ledger-age gauge per node.\n**Reading it:** Should track validated-ledger age closely.\n**Healthy range:** under ~10s.\n**Watch for:** Published age growing while validated age stays low (publish-pipeline backlog).\n**Source:** src/xrpld/app/ledger/LedgerMaster.h LedgerMaster::Stats ctor",
|
||||
"type": "stat",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -107,7 +107,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Operating Mode Duration",
|
||||
"description": "Cumulative time spent in each operating mode (Disconnected, Connected, Syncing, Tracking, Full). Sourced from State_Accounting.*_duration gauges (NetworkOPs.cpp) which report microseconds. A healthy node should spend the vast majority of time in Full mode.",
|
||||
"description": "**What:** Cumulative time the node has spent in each operating mode: Disconnected, Connected, Syncing, Tracking, Full.\n**How it's computed:** Per-mode accumulated time (microseconds) at its latest value per node.\n**Reading it:** Full should dominate and keep growing; other modes should be nearly flat.\n**Healthy range:** almost all time in Full on a healthy node.\n**Watch for:** Growth in Syncing, Connected, or Disconnected (instability or repeated resync).\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp NetworkOPsImp::Stats ctor",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -160,7 +160,7 @@
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "\u00b5s",
|
||||
"unit": "µs",
|
||||
"custom": {
|
||||
"axisLabel": "Duration",
|
||||
"spanNulls": true,
|
||||
@@ -174,7 +174,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Operating Mode Transitions",
|
||||
"description": "Count of transitions into each operating mode. Sourced from State_Accounting.*_transitions gauges (NetworkOPs.cpp). Frequent transitions out of Full mode indicate instability. Transitions to Disconnected or Syncing warrant investigation.",
|
||||
"description": "**What:** Count of transitions into each operating mode.\n**How it's computed:** Per-mode transition counters at their latest value per node.\n**Reading it:** Few transitions is good; a stable node rarely leaves Full.\n**Healthy range:** workload-dependent; low and infrequent transitions.\n**Watch for:** Frequent transitions out of Full, or any into Disconnected/Syncing (flapping).\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp NetworkOPsImp::Stats ctor",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -241,7 +241,7 @@
|
||||
},
|
||||
{
|
||||
"title": "I/O Latency",
|
||||
"description": "P95 and P50 of the I/O service loop latency in milliseconds. Sourced from the ios_latency event (Application.cpp) which measures how long it takes for the io_context to process a timer callback. Values above 10ms are logged; above 500ms trigger warnings. High values indicate thread pool saturation or blocking operations.",
|
||||
"description": "**What:** P95 and P50 latency of the I/O service loop, i.e. how long a queued timer callback waits to run.\n**How it's computed:** 95th and 50th percentiles over a 5-minute window.\n**Reading it:** Low and flat is good; this reflects event-loop responsiveness.\n**Healthy range:** single-digit milliseconds; over ~10ms is notable, over ~500ms is bad.\n**Watch for:** Rising P95 (thread-pool saturation or blocking work on the I/O thread).\n**Source:** src/xrpld/app/main/Application.cpp ApplicationImp ctor",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -287,7 +287,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Job Queue Depth",
|
||||
"description": "Current number of jobs waiting in the job queue. Sourced from the job_count gauge (JobQueue.cpp). A sustained high value indicates the node cannot process work fast enough \u2014 common during ledger replay or heavy RPC load.",
|
||||
"description": "**What:** Number of jobs currently waiting in the job queue.\n**How it's computed:** Current value of the job-count gauge per node.\n**Reading it:** Near zero when the node keeps up; brief spikes during heavy work are normal.\n**Healthy range:** low, returning to baseline quickly.\n**Watch for:** Sustained high depth (node cannot process work fast enough: replay, heavy RPC, or overload).\n**Source:** src/libxrpl/core/detail/JobQueue.cpp JobQueue ctor",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -326,7 +326,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Ledger Fetch Rate",
|
||||
"description": "Rate of ledger fetch requests initiated by the node. Sourced from the ledger_fetches counter (InboundLedgers.cpp) which increments each time the node requests a ledger from a peer. High rates indicate the node is catching up or missing ledgers.",
|
||||
"description": "**What:** Rate at which the node requests ledgers from peers.\n**How it's computed:** Per-second rate of the ledger-fetches counter over 5 minutes.\n**Reading it:** Near zero when synced; rises when catching up or backfilling history.\n**Healthy range:** workload-dependent; low on a synced node.\n**Watch for:** Sustained high fetch rate on a node that should be current (repeatedly missing ledgers).\n**Source:** src/xrpld/app/ledger/detail/InboundLedgers.cpp InboundLedgersImp ctor",
|
||||
"type": "stat",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -358,7 +358,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Ledger History Mismatches",
|
||||
"description": "Rate of ledger history hash mismatches. Sourced from the ledger.history.mismatch counter (LedgerHistory.cpp) which increments when a built ledger hash does not match the expected validated hash. Non-zero values indicate consensus divergence or database corruption.",
|
||||
"description": "**What:** Rate of ledger-history hash mismatches: built ledgers whose hash disagrees with the validated hash.\n**How it's computed:** Per-second rate of the history-mismatch counter over 5 minutes.\n**Reading it:** Should be flat at zero.\n**Healthy range:** zero.\n**Watch for:** Any non-zero value (consensus divergence, corrupted history, or database issues).\n**Source:** src/xrpld/telemetry/MetricsRegistry.cpp incrementLedgerHistoryMismatch (caller LedgerHistory.cpp)",
|
||||
"type": "stat",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -414,7 +414,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Key Jobs Execution Time",
|
||||
"description": "Execution time for critical job types at the selected quantile. Sourced from per-job-type events in JobTypeData (JobTypeData.h). Shows how long key consensus, transaction, and maintenance jobs take to execute. Spikes indicate processing bottlenecks.",
|
||||
"description": "**What:** Execution time of the most important job types (accept/advance ledger, transaction, write objects, heartbeat, sweep, trusted validation/proposal, publish, client RPC, ledger data) at the selected quantile.\n**How it's computed:** Selected quantile of each job's execution-time histogram over 5 minutes.\n**Reading it:** Lower and stable is better; shows where consensus, transaction, and maintenance time goes.\n**Healthy range:** workload-dependent; most jobs in low tens of milliseconds.\n**Watch for:** One job type spiking (a specific bottleneck) or broad increases (overload).\n**Source:** include/xrpl/core/JobTypeData.h JobTypeData ctor",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -523,7 +523,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Key Jobs Dequeue Wait Time",
|
||||
"description": "Time spent waiting in the job queue before execution for critical job types. Sourced from per-job-type dequeue events (JobTypeData.h). High dequeue times indicate the job queue is backlogged and jobs are waiting too long to be scheduled.",
|
||||
"description": "**What:** Time key jobs wait in the queue before they start running, at the selected quantile.\n**How it's computed:** Selected quantile of each job's queue-wait histogram over 5 minutes.\n**Reading it:** Low wait means the scheduler keeps up; high wait means backlog.\n**Healthy range:** workload-dependent; short waits.\n**Watch for:** Rising wait across job types (queue congestion delaying consensus and transaction work).\n**Source:** include/xrpl/core/JobTypeData.h JobTypeData ctor",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -632,7 +632,7 @@
|
||||
},
|
||||
{
|
||||
"title": "FullBelowCache Size",
|
||||
"description": "Number of entries in the FullBelowCache. Sourced from the TaggedCache size gauge (TaggedCache.h) for the Node family full below cache (NodeFamily.cpp). This cache tracks which SHAMap nodes have all children present locally, avoiding redundant fetches during ledger acquisition.",
|
||||
"description": "**What:** Number of entries in the FullBelowCache, which tracks tree nodes known to have all children present locally.\n**How it's computed:** Current value of the cache-size gauge per node.\n**Reading it:** Grows as the node learns complete subtrees; helps avoid redundant fetches.\n**Healthy range:** workload-dependent; stable once synced.\n**Watch for:** Collapsing to near zero repeatedly (cache thrash) alongside rising fetch traffic.\n**Source:** include/xrpl/basics/TaggedCache.h TaggedCache::Stats ctor",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -671,7 +671,7 @@
|
||||
},
|
||||
{
|
||||
"title": "FullBelowCache Hit Rate",
|
||||
"description": "Hit rate percentage for the FullBelowCache. Sourced from the TaggedCache hit_rate gauge (TaggedCache.h). A high hit rate means the node is efficiently reusing cached knowledge about complete SHAMap subtrees. Low hit rates during steady state warrant investigation.",
|
||||
"description": "**What:** Hit-rate percentage of the FullBelowCache.\n**How it's computed:** Current value of the cache hit-rate gauge (0-100%).\n**Reading it:** Higher is better; means the node reuses knowledge of complete subtrees.\n**Healthy range:** above ~50% in steady state.\n**Watch for:** Low hit rate during steady state (cache too small or constant re-acquisition).\n**Source:** include/xrpl/basics/TaggedCache.h TaggedCache::Stats ctor",
|
||||
"type": "gauge",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -721,7 +721,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Ledger Publish Gap",
|
||||
"description": "Difference between published and validated ledger ages. Computed as Published_Ledger_Age minus Validated_Ledger_Age. A value near zero means the publish pipeline keeps up with validation. A growing gap indicates the publish pipeline is falling behind, potentially causing stale data for subscribers.",
|
||||
"description": "**What:** Difference between published and validated ledger ages: how far the publish pipeline trails validation.\n**How it's computed:** Published-ledger age minus validated-ledger age, in seconds.\n**Reading it:** Near zero means publishing keeps up with validation.\n**Healthy range:** within a few seconds of zero.\n**Watch for:** A growing gap (publish backlog, stale data for subscribers).\n**Source:** src/xrpld/app/ledger/LedgerMaster.h LedgerMaster::Stats ctor",
|
||||
"type": "stat",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -769,7 +769,7 @@
|
||||
},
|
||||
{
|
||||
"title": "State Duration Rate (Full vs Tracking)",
|
||||
"description": "Rate of change of time spent in Full and Tracking operating modes, normalized to seconds. Sourced from State_Accounting duration gauges (NetworkOPs.cpp). In steady state the Full duration rate should be close to 1.0 (gaining one second of Full-mode time per wall-clock second). A drop below 1.0 means the node is spending time in other modes.",
|
||||
"description": "**What:** How fast the node accrues time in Full versus Tracking mode, normalized to seconds per second.\n**How it's computed:** Per-second rate of the Full and Tracking accumulated-time gauges, scaled to seconds.\n**Reading it:** Full rate near 1.0 means the node is in Full essentially all the time.\n**Healthy range:** Full rate ~1.0, Tracking near 0.\n**Watch for:** Full rate dropping below 1.0 with Tracking rising (time being lost to non-Full modes).\n**Source:** src/xrpld/app/misc/NetworkOPs.cpp NetworkOPsImp::Stats ctor",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -815,7 +815,7 @@
|
||||
},
|
||||
{
|
||||
"title": "All Jobs Execution Time (Detail)",
|
||||
"description": "Execution time for ALL non-special job types at the selected quantile. Shows the complete picture of job execution performance. Use the Key Jobs panel for a focused view of the most critical jobs.",
|
||||
"description": "**What:** Execution time for all non-special job types at the selected quantile.\n**How it's computed:** Selected quantile of each job's execution-time histogram over 5 minutes.\n**Reading it:** Full breakdown of job performance; use the Key Jobs panel for the focused view.\n**Healthy range:** workload-dependent.\n**Watch for:** Any job type with a persistent upward trend, or many rising together (systemic slowdown).\n**Source:** include/xrpl/core/JobTypeData.h JobTypeData ctor",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -854,7 +854,7 @@
|
||||
},
|
||||
{
|
||||
"title": "All Jobs Dequeue Wait (Detail)",
|
||||
"description": "Dequeue wait time for ALL non-special job types at the selected quantile. Shows the complete picture of job queue waiting times. High wait times across many job types indicate systemic job queue congestion.",
|
||||
"description": "**What:** Queue wait time before execution for all non-special job types at the selected quantile.\n**How it's computed:** Selected quantile of each job's queue-wait histogram over 5 minutes.\n**Reading it:** Complete picture of scheduling delay across job types.\n**Healthy range:** workload-dependent; short waits.\n**Watch for:** High waits across many job types (systemic job-queue congestion).\n**Source:** include/xrpl/core/JobTypeData.h JobTypeData ctor",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -893,7 +893,7 @@
|
||||
}
|
||||
],
|
||||
"schemaVersion": 39,
|
||||
"tags": ["statsd", "node-health", "telemetry"],
|
||||
"tags": ["node", "health"],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
@@ -1016,6 +1016,6 @@
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"title": "Node Health (System Metrics)",
|
||||
"title": "Node Health",
|
||||
"uid": "xrpld-system-node-health"
|
||||
}
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"description": "Detailed overlay traffic breakdown for categories not covered by the main Network Traffic dashboard. Includes squelch, overhead, validator lists, object fetch, ledger sync, and protocol negotiation traffic. Requires [insight] server=otel in rippled config.",
|
||||
"description": "Fine-grained breakdown of peer-to-peer overlay traffic beyond the main network view: squelch relay control, protocol overhead, validator-list distribution, transaction-set exchange, transaction availability, ledger-proof and replay traffic, and unclassified messages.",
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
@@ -11,7 +11,7 @@
|
||||
"panels": [
|
||||
{
|
||||
"title": "Squelch Traffic (Messages)",
|
||||
"description": "Squelch-related overlay messages. Squelch is the peer traffic management protocol that suppresses redundant message forwarding. 'squelch' = squelch control messages, 'squelch_suppressed' = messages suppressed by squelch, 'squelch_ignored' = squelch directives that were ignored. High suppressed counts indicate effective bandwidth savings; high ignored counts may indicate misconfigured peers. Sourced from TrafficCount.h squelch categories.",
|
||||
"description": "**What:** Squelch relay-control messages in/out, plus messages suppressed by squelch and squelch directives that were ignored. Squelch reduces redundant message forwarding between peers.\n**How it's computed:** Current message counts for the squelch, squelch-suppressed, and squelch-ignored categories, in and out.\n**Reading it:** High suppressed counts mean squelch is saving bandwidth; ignored should stay low.\n**Healthy range:** workload-dependent; suppressed far above ignored.\n**Watch for:** High ignored counts (peers not honoring squelch) or squelch traffic itself dominating.\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -85,7 +85,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Overhead Traffic Breakdown (Bytes)",
|
||||
"description": "Overlay protocol overhead by sub-category. 'overhead' = base protocol overhead (ping, status, etc.), 'overhead_cluster' = intra-cluster communication overhead, 'overhead_manifest' = validator manifest distribution overhead. High cluster overhead may indicate frequent cluster state syncs; high manifest overhead occurs during UNL changes. Sourced from TrafficCount.h overhead categories.",
|
||||
"description": "**What:** Overlay protocol overhead bytes split into base overhead, intra-cluster overhead, and validator-manifest distribution overhead.\n**How it's computed:** Current in/out byte counts for the overhead, overhead-cluster, and overhead-manifest categories.\n**Reading it:** Base overhead is routine; cluster and manifest rise around cluster syncs and manifest changes.\n**Healthy range:** workload-dependent; low and stable.\n**Watch for:** Sustained high cluster or manifest overhead (frequent cluster state churn or manifest reissue).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -159,7 +159,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Validator List Traffic",
|
||||
"description": "Validator list (UNL) distribution traffic. Validator lists are exchanged when peers share their trusted validator configurations. Spikes occur during UNL updates or when new peers connect. Sourced from TrafficCount.h validator_lists category.",
|
||||
"description": "**What:** Bytes and messages exchanged distributing validator lists (trusted-list configuration) between peers.\n**How it's computed:** Current in/out byte and message counts for the validator-lists category.\n**Reading it:** Bursts when lists update or new peers connect; quiet otherwise.\n**Healthy range:** workload-dependent; occasional bursts.\n**Watch for:** Continuous high volume (repeated list re-fetching or churn).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -236,7 +236,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Set Get/Share Traffic (Bytes)",
|
||||
"description": "Transaction set get and share traffic. 'set_get' = requests to fetch transaction sets (sent during ledger close), 'set_share' = responses sharing transaction sets. High set_get traffic indicates peers frequently requesting missing transaction sets, which may signal sync delays. Sourced from TrafficCount.h set_get/set_share categories.",
|
||||
"description": "**What:** Transaction-set fetch (get) and share bytes exchanged during ledger close.\n**How it's computed:** Current in/out byte counts for the set-get and set-share categories.\n**Reading it:** Some exchange each ledger is normal as peers reconcile transaction sets.\n**Healthy range:** workload-dependent.\n**Watch for:** High set-get (peers frequently missing transaction sets: possible sync delays).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -296,7 +296,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Have/Requested Transactions (Messages)",
|
||||
"description": "Transaction availability protocol messages. 'have_transactions' = advertisements that a peer has specific transactions available, 'requested_transactions' = explicit requests for transaction data. A high ratio of requested to have may indicate peers are behind on transaction propagation. Sourced from TrafficCount.h have_transactions/requested_transactions categories.",
|
||||
"description": "**What:** Transaction-availability messages: advertisements that a peer has certain transactions, and explicit requests for transaction data.\n**How it's computed:** Current in/out message counts for the have-transactions and requested-transactions categories.\n**Reading it:** Compare requested versus have to gauge how well transactions are propagating.\n**Healthy range:** workload-dependent.\n**Watch for:** Requested far exceeding have (peers behind on transaction propagation).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -356,7 +356,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Unknown / Unclassified Traffic",
|
||||
"description": "Traffic that does not match any known overlay message category. Non-zero values may indicate protocol version mismatches, corrupted messages, or new message types not yet classified. Sourced from TrafficCount.h unknown category.",
|
||||
"description": "**What:** Overlay traffic that matches no known message category, in bytes and messages.\n**How it's computed:** Current in/out byte and message counts for the unknown category.\n**Reading it:** Should be at or near zero.\n**Healthy range:** zero.\n**Watch for:** Any sustained non-zero value (protocol version mismatch, corrupted messages, or an unclassified new message type).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -433,7 +433,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Proof Path Traffic",
|
||||
"description": "Proof path request/response traffic for ledger state proof exchange. Used by peers to verify specific ledger entries without downloading the full ledger. High request volume may indicate peers validating state during catch-up. Sourced from TrafficCount.h proof_path_request/proof_path_response categories.",
|
||||
"description": "**What:** Proof-path request/response bytes used to verify individual ledger entries without downloading the whole ledger.\n**How it's computed:** Current in/out byte counts for the proof-path request and response categories.\n**Reading it:** Rises when peers verify specific state, often during catch-up.\n**Healthy range:** workload-dependent.\n**Watch for:** High sustained request volume (heavy state-verification load).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -493,7 +493,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Replay Delta Traffic",
|
||||
"description": "Replay delta request/response traffic for ledger replay protocol. Used during catch-up to efficiently replay ledger state changes. Sourced from TrafficCount.h replay_delta_request/replay_delta_response categories.",
|
||||
"description": "**What:** Replay-delta request/response bytes used to efficiently replay ledger state changes during catch-up.\n**How it's computed:** Current in/out byte counts for the replay-delta request and response categories.\n**Reading it:** Active during catch-up and replay; quiet when synced.\n**Healthy range:** workload-dependent; low when synced.\n**Watch for:** Continuous replay traffic (node repeatedly replaying rather than staying current).\n**Source:** src/xrpld/overlay/detail/OverlayImpl.cpp OverlayImpl ctor (TrafficGauges)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -553,7 +553,7 @@
|
||||
}
|
||||
],
|
||||
"schemaVersion": 39,
|
||||
"tags": ["statsd", "overlay", "network", "telemetry"],
|
||||
"tags": ["network", "peer"],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
@@ -642,6 +642,6 @@
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"title": "Overlay Traffic Detail (System Metrics)",
|
||||
"title": "Overlay Traffic Detail",
|
||||
"uid": "xrpld-system-overlay-detail"
|
||||
}
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"description": "RPC and pathfinding metrics from beast::insight System Metrics. Requires [insight] server=otel in rippled config.",
|
||||
"description": "Client-facing RPC behavior and payment pathfinding cost on this node: request rates, response times and sizes, resource-limit warnings and drops, gRPC read-path load, and pathfinding request, compute, and discovery activity.",
|
||||
"editable": true,
|
||||
"fiscalYearStartMonth": 0,
|
||||
"graphTooltip": 1,
|
||||
@@ -10,8 +10,8 @@
|
||||
"links": [],
|
||||
"panels": [
|
||||
{
|
||||
"title": "RPC Request Rate (System Metrics)",
|
||||
"description": "Rate of RPC requests as counted by the beast::insight counter. Sourced from rpc.requests (ServerHandler.cpp) which increments on every HTTP and WebSocket RPC request. Compare with the span-based rpc.request rate in the RPC Performance dashboard for cross-validation.",
|
||||
"title": "RPC Request Rate",
|
||||
"description": "**What:** Rate of RPC requests handled, counting every HTTP and WebSocket call.\n**How it's computed:** Per-second rate of the RPC-requests counter over 5 minutes.\n**Reading it:** Tracks client demand on the node.\n**Healthy range:** workload-dependent.\n**Watch for:** Sudden sustained spikes (client surge or abusive polling) or a drop to zero (endpoint unavailable).\n**Source:** src/xrpld/rpc/detail/ServerHandler.cpp ServerHandler ctor",
|
||||
"type": "stat",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -42,8 +42,8 @@
|
||||
}
|
||||
},
|
||||
{
|
||||
"title": "RPC Response Time (System Metrics)",
|
||||
"description": "P95 and P50 of RPC response time from the beast::insight timer. Sourced from the rpc.time event (ServerHandler.cpp) which records elapsed milliseconds for each RPC response. This measures the full HTTP handler time, not just command execution. Compare with span-based rpc.request duration.",
|
||||
"title": "RPC Response Time",
|
||||
"description": "**What:** P95 and P50 of end-to-end RPC handler time (full HTTP handling, not just command execution).\n**How it's computed:** 95th and 50th percentiles over a 5-minute window.\n**Reading it:** Lower is better; P95 shows the slow tail.\n**Healthy range:** workload-dependent; most methods well under a second.\n**Watch for:** Rising P95 while request rate is flat (expensive queries or contention).\n**Source:** src/xrpld/rpc/detail/ServerHandler.cpp ServerHandler ctor",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -89,7 +89,7 @@
|
||||
},
|
||||
{
|
||||
"title": "RPC Response Size",
|
||||
"description": "P95 and P50 of RPC response payload size in bytes. Sourced from the rpc.size event (ServerHandler.cpp) which records the byte length of each RPC JSON response. Large responses may indicate expensive queries (e.g. account_tx with many results) or API misuse.",
|
||||
"description": "**What:** P95 and P50 of RPC response payload size in bytes.\n**How it's computed:** 95th and 50th percentiles over a 5-minute window.\n**Reading it:** Larger responses cost more bandwidth and CPU to build.\n**Healthy range:** workload-dependent.\n**Watch for:** Large P95 (result-heavy queries such as broad account_tx, or API misuse).\n**Source:** src/xrpld/rpc/detail/ServerHandler.cpp ServerHandler ctor",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -135,7 +135,7 @@
|
||||
},
|
||||
{
|
||||
"title": "RPC Response Time Distribution",
|
||||
"description": "Distribution of RPC response times from the beast::insight timer showing P50, P90, P95, and P99 quantiles. Sourced from the rpc.time event (ServerHandler.cpp). Useful for detecting bimodal latency or long-tail requests.",
|
||||
"description": "**What:** RPC response-time spread shown as P50, P90, P95, and P99.\n**How it's computed:** Four quantiles of response time over a 5-minute window.\n**Reading it:** A wide gap between P50 and P99 signals a long latency tail.\n**Healthy range:** workload-dependent; quantiles clustered together.\n**Watch for:** P99 pulling far above P50 (bimodal latency or occasional very slow requests).\n**Source:** src/xrpld/rpc/detail/ServerHandler.cpp ServerHandler ctor",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -195,7 +195,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Pathfinding Fast Duration",
|
||||
"description": "P95 and P50 of fast pathfinding execution time. Sourced from the pathfind_fast event (PathRequests.h) which records the duration of the fast pathfinding algorithm. Fast pathfinding uses a simplified search that trades accuracy for speed.",
|
||||
"description": "**What:** P95 and P50 execution time of the fast pathfinding search (simplified, favoring speed over completeness).\n**How it's computed:** 95th and 50th percentiles over a 5-minute window.\n**Reading it:** Lower is better; fast pathfinding should stay quick.\n**Healthy range:** workload-dependent; typically well below full pathfinding.\n**Watch for:** Rising fast-path latency (pathfinding load or complex order books).\n**Source:** src/xrpld/rpc/detail/PathRequestManager.h ctor",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -241,7 +241,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Pathfinding Full Duration",
|
||||
"description": "P95 and P50 of full pathfinding execution time. Sourced from the pathfind_full event (PathRequests.h) which records the duration of the exhaustive pathfinding search. Full pathfinding is more expensive and can take significantly longer than fast mode.",
|
||||
"description": "**What:** P95 and P50 execution time of the full, exhaustive pathfinding search.\n**How it's computed:** 95th and 50th percentiles over a 5-minute window.\n**Reading it:** Full pathfinding is heavier and slower than fast mode.\n**Healthy range:** workload-dependent.\n**Watch for:** High or rising full-path latency (expensive path computation under subscription load).\n**Source:** src/xrpld/rpc/detail/PathRequestManager.h ctor",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -287,7 +287,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Resource Warnings Rate",
|
||||
"description": "Rate of resource warning events from the Resource Manager. Sourced from the warn meter (Logic.h) which increments when a consumer (peer or RPC client) exceeds the warning threshold for resource usage. A rising rate indicates aggressive clients that may need throttling. NOTE: This panel will show no data until the |m -> |c fix is applied in System MetricsCollector.cpp (Phase 6 Task 6.1).",
|
||||
"description": "**What:** Rate of resource-limit warnings raised when a peer or client exceeds its usage warning threshold.\n**How it's computed:** Per-second rate of the warn counter over 5 minutes.\n**Reading it:** Occasional warnings are normal under load; a rising rate flags aggressive clients.\n**Healthy range:** workload-dependent; low.\n**Watch for:** A climbing warning rate (clients approaching limits, a precursor to drops).\n**Source:** include/xrpl/resource/detail/Logic.h Logic::Stats ctor",
|
||||
"type": "stat",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -335,7 +335,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Resource Drops Rate",
|
||||
"description": "Rate of resource drop events from the Resource Manager. Sourced from the drop meter (Logic.h) which increments when a consumer is disconnected or blocked due to excessive resource usage. Non-zero values mean the node is actively rejecting abusive connections. NOTE: This panel will show no data until the |m -> |c fix is applied in System MetricsCollector.cpp (Phase 6 Task 6.1).",
|
||||
"description": "**What:** Rate of resource drops: consumers disconnected or blocked for excessive usage.\n**How it's computed:** Per-second rate of the drop counter over 5 minutes.\n**Reading it:** Non-zero means the node is actively rejecting abusive connections.\n**Healthy range:** at or near zero.\n**Watch for:** Sustained non-zero drops (ongoing abuse or a misbehaving client/peer being throttled).\n**Source:** include/xrpl/resource/detail/Logic.h Logic::Stats ctor",
|
||||
"type": "stat",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -383,7 +383,7 @@
|
||||
},
|
||||
{
|
||||
"title": "gRPC Request Rate by Method (Spans)",
|
||||
"description": "Per-method gRPC call rate derived from the grpc.{Method} spans (GRPCServer.cpp). Covers the gRPC API used by reporting/Clio. Populated only when the node serves gRPC traffic.",
|
||||
"description": "**What:** Per-method gRPC call rate for the gRPC read API used by reporting and Clio.\n**How it's computed:** Per-second count of gRPC calls grouped by method.\n**Reading it:** Shows which gRPC methods are called and how often; only populated when the node serves gRPC.\n**Healthy range:** workload-dependent.\n**Watch for:** A single method spiking (heavy reporting load) or unexpected methods appearing.\n**Source:** src/xrpld/rpc/GRPCServer.cpp (gRPC method handlers)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -422,7 +422,7 @@
|
||||
},
|
||||
{
|
||||
"title": "gRPC Latency P95 by Method (Spans)",
|
||||
"description": "p95 latency per gRPC method from grpc.{Method} span durations. Identifies slow gRPC read paths.",
|
||||
"description": "**What:** P95 latency per gRPC method.\n**How it's computed:** 95th percentile of gRPC call duration per method over 5 minutes.\n**Reading it:** Identifies slow gRPC read paths.\n**Healthy range:** workload-dependent.\n**Watch for:** One method's P95 rising (expensive ledger reads or backend pressure).\n**Source:** src/xrpld/rpc/GRPCServer.cpp (gRPC method handlers)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -461,7 +461,7 @@
|
||||
},
|
||||
{
|
||||
"title": "gRPC Error Rate by Status (Spans)",
|
||||
"description": "Rate of gRPC spans broken down by grpc_status (success/error/resource_exhausted/failed_precondition). A rising error or resource_exhausted rate indicates gRPC clients hitting limits.",
|
||||
"description": "**What:** Rate of gRPC calls broken down by result status: success, error, resource-exhausted, failed-precondition.\n**How it's computed:** Per-second count of gRPC calls grouped by status.\n**Reading it:** Errors and resource-exhausted should be a small share.\n**Healthy range:** workload-dependent; mostly success.\n**Watch for:** Rising error or resource-exhausted rate (clients hitting limits or failing reads).\n**Source:** src/xrpld/rpc/GRPCServer.cpp (gRPC method handlers)",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -500,7 +500,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Pathfinding Compute Duration (Spans)",
|
||||
"description": "p95/p50 of the pathfind.compute span, the per-request path computation. Complements the StatsD pathfind_fast/full timers with span-level visibility. Populated under pathfinding (book/path) RPC load.",
|
||||
"description": "**What:** P95 and P50 of per-request path computation time.\n**How it's computed:** 95th and 50th percentiles of path-computation duration over 5 minutes.\n**Reading it:** Complements the fast/full timers with per-request visibility; populated under book/path RPC load.\n**Healthy range:** workload-dependent.\n**Watch for:** Rising compute time (complex paths or heavy pathfinding demand).\n**Source:** src/xrpld/rpc/detail/PathRequest.cpp PathRequest::doUpdate",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -546,7 +546,7 @@
|
||||
},
|
||||
{
|
||||
"title": "Pathfinding Request & Discovery Rate (Spans)",
|
||||
"description": "Rate of pathfind.request (client path requests) and pathfind.discover (path-discovery passes) spans. Shows pathfinding demand and the discovery cost driver for subscription-heavy nodes.",
|
||||
"description": "**What:** Rate of client path requests and of path-discovery passes.\n**How it's computed:** Per-second count of path-request and path-discovery operations.\n**Reading it:** Shows pathfinding demand and the discovery cost driver for subscription-heavy nodes.\n**Healthy range:** workload-dependent.\n**Watch for:** Discovery rate climbing (subscription load driving repeated path discovery).\n**Source:** src/xrpld/rpc/handlers/orderbook/PathFind.cpp doPathFind; src/xrpld/rpc/detail/PathRequest.cpp PathRequest::findPaths",
|
||||
"type": "timeseries",
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
@@ -592,7 +592,7 @@
|
||||
}
|
||||
],
|
||||
"schemaVersion": 39,
|
||||
"tags": ["statsd", "rpc", "pathfinding", "telemetry"],
|
||||
"tags": ["rpc", "pathfinding"],
|
||||
"templating": {
|
||||
"list": [
|
||||
{
|
||||
@@ -701,6 +701,6 @@
|
||||
"from": "now-1h",
|
||||
"to": "now"
|
||||
},
|
||||
"title": "RPC & Pathfinding (System Metrics)",
|
||||
"title": "RPC & Pathfinding",
|
||||
"uid": "xrpld-system-rpc"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user