ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-21 22:30:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(telemetry): add ledger-acquire and SHAMap fetch diagnostics (WP-A3)

Browse Source 
Signals that separate a sync that is merely slow from one that will never
finish:

- sync_acquire{missing_state_nodes_max, missing_tx_nodes_max, in_flight,
  received_data_depth}: how many SHAMap nodes each in-flight acquire is
  still waiting for. getMissingNodes already computed this and the callers
  discarded it after a trace log. A count that stays flat means the
  acquire is wedged; a shrinking count means it is progressing. Recorded
  once per sweep, never inside the per-node walk, and reset when a tree
  completes so a finished acquire does not read as stuck forever.
- shamap_cache_hit_rate{treenode}: hit rate of the in-memory tree-node
  cache, which sits above the node store, so it is distinct from the
  existing NuDB ratio. A cold cache on a fresh node sends every traversal
  step to disk.
- sync_acquire_no_progress_total: timer ticks where an acquire made no
  progress, previously only logged.
- sync_addnode_total{good,duplicate,invalid}: whether arriving nodes are
  useful, duplicated or rejected, so wasted fetch work is visible.
- sync_acquire_source_total{local,network}: whether a ledger was served
  from the local store or had to be fetched.

Adds getBad()/getDuplicate() to SHAMapAddNode and an acquireProgress()
accessor on InboundLedgers so the xrpld gauge can read these without
libxrpl depending on telemetry.

ledger_seq is deliberately not a metric label: it is unbounded. Per-ledger
identity stays on the ledger.acquire span; the metrics expose bounded
aggregates instead.

The full-below cache hit rate is not exported: KeyCache updates different
counters than getHitRate() reads, so it would always report zero. That
libxrpl bug is documented rather than papered over.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pratik Mankawde
2026-07-25 10:18:08 +01:00
parent 7c7509d01f
commit 3e2a1ea958
16 changed files with 1676 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files

39
OpenTelemetryPlan/09-data-collection-reference.md
View File

@@ -1401,19 +1401,26 @@ signal as it lands. `Type` is the instrument kind (counter / gauge / histogram /
span / span attr), `Emit site` the owning source file, and `Panel` the dashboard
panel that renders it.
| Signal | Type | Emit site | Panel | Meaning |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------- | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `dns_resolve_total` (`outcome` = `resolved` \| `empty`) | counter | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Outcome Rate | Peer hostname resolutions. `empty` means a configured bootstrap or `[ips_fixed]` name returned no address, so that peer is never dialled. |
| `dns_resolve_latency_ms` | histogram | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Latency (p95) | Time to resolve a configured peer hostname. Seconds-scale values mean the resolver is timing out ahead of every dial. |
| `overlay_connect_total` (`outcome` = `connected` \| `tcp_fail` \| `tls_fail` \| `upgrade_fail` \| `timeout`) | counter | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Outcome Rate | Outbound peer connection attempts by terminal outcome. The outcome names the stage that broke: TCP, TLS, HTTP upgrade, or no terminal state in time. |
| `overlay_dial_latency_ms` | histogram | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Latency (p95) | Time from starting an outbound dial to its terminal outcome, successes and failures together. A p95 near the dial timeout means peers accept TCP but never finish the handshake. |
| `handshake_negotiation_fail_total` (`reason`, 14 values incl. `wrong_network`, `invalid_network_id`, `clock_skew`, `self_connection`, `session_verify_failed`) | counter | `Handshake.cpp` — `throwNegotiationFailure` (from `verifyHandshake`) | Handshake Negotiation Failures by Reason | Peer handshakes rejected after TLS while checking network id, clock, keys and addresses. `reason` names the failing check. |
| `unl_fetch_total` (`site` = configured UNL URI; `outcome` = the 9 `ListDisposition` strings `accepted` \| `expired` \| `same_sequence` \| `pending` \| `known_sequence` \| `unsupported_version` \| `untrusted` \| `stale` \| `invalid`, plus `fetch_error` \| `bad_status` \| `parse_error`) | counter | `ValidatorSite.cpp` — `ValidatorSite::reportFetchOutcome` | UNL Fetch Rate by Site & Outcome | Validator-list fetches per site. `accepted` is the only success; `same_sequence` and `known_sequence` are normal no-op refreshes; the three literals are transport or content faults. |
| `unl_quorum` (`metric` = `trusted_keys` \| `quorum`) | observable gauge | `MetricsRegistry.cpp` — `registerUnlQuorumGauge` | UNL Trusted Keys vs Quorum; UNL Quorum Headroom | Trusted UNL key count against the validations a ledger needs. `trusted_keys` at or below `quorum` means the node can never declare a ledger validated. |
| `clock_close_offset_seconds` (`metric` = `offset`) | observable gauge | `MetricsRegistry.cpp` — `registerClockSkewGauge` | Clock Close Offset | Network close time offset from the local clock. Negative means the local clock runs ahead. `server_info` only surfaces `close_time_offset` at 60 s or more, so this gauge sees skew far earlier. |
| `state_changes_total` (`from`, `to` = `disconnected` \| `connected` \| `syncing` \| `tracking` \| `full`) | counter | `NetworkOPs.cpp` — `NetworkOPsImp::setMode` | Mode Transitions by Edge | Operating-mode transitions keyed on the (`from`, `to`) edge. The edge is what separates a clean `disconnected`→`connected`→`syncing`→`tracking`→`full` climb from `full`→`connected` flapping; an unlabelled total cannot tell them apart. |
| `sync_state` (`metric` = `initial_full_duration_us`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Time to First FULL | Microseconds from process start to the first `full` transition, sourced from `NetworkOPs::getInitialSyncDurationUs()`. Stays 0 until `full` is reached, so a flat 0 is itself the "never synced" signal; once set it never changes. |
| `sync_state` (`metric` = `network_ledger_gate`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Network Ledger Gate | 1 while the node is still waiting to see a full network ledger (`NetworkOPs::isNeedNetworkLedger()`), else 0. A persistent 1 blocks transaction submission and `full`, whatever the rest of the pipeline shows. |
| `sync_state` (`metric` = `server_stall_seconds`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Server Stall | Current main-loop stall duration from `LoadManager::getCurrentStallSeconds()`, 0 when healthy. Same duration the load monitor logs as "Server stalled for N seconds", which previously existed only in that log line. |
| `sync_state` (`metric` = `ledgers_behind`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Ledgers Behind Network | Peer-reported network tip minus our validated sequence, floored at 0 (`NetworkOPs::getLedgersBehindNetwork()`). Reads each peer's already cached ledger range, so no new network round trip. |
| `server_stall_events_total` | observable counter | `MetricsRegistry.cpp` — `registerStallEventsCounter` | Server Stall Event Rate | Distinct stall episodes since process start, counted once per episode rather than per stalled second. A rising rate is repeated fresh stalls; a flat rate with a large `server_stall_seconds` is one long stall. |
| Signal | Type | Emit site | Panel | Meaning |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------- | ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `dns_resolve_total` (`outcome` = `resolved` \| `empty`) | counter | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Outcome Rate | Peer hostname resolutions. `empty` means a configured bootstrap or `[ips_fixed]` name returned no address, so that peer is never dialled. |
| `dns_resolve_latency_ms` | histogram | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Latency (p95) | Time to resolve a configured peer hostname. Seconds-scale values mean the resolver is timing out ahead of every dial. |
| `overlay_connect_total` (`outcome` = `connected` \| `tcp_fail` \| `tls_fail` \| `upgrade_fail` \| `timeout`) | counter | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Outcome Rate | Outbound peer connection attempts by terminal outcome. The outcome names the stage that broke: TCP, TLS, HTTP upgrade, or no terminal state in time. |
| `overlay_dial_latency_ms` | histogram | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Latency (p95) | Time from starting an outbound dial to its terminal outcome, successes and failures together. A p95 near the dial timeout means peers accept TCP but never finish the handshake. |
| `handshake_negotiation_fail_total` (`reason`, 14 values incl. `wrong_network`, `invalid_network_id`, `clock_skew`, `self_connection`, `session_verify_failed`) | counter | `Handshake.cpp` — `throwNegotiationFailure` (from `verifyHandshake`) | Handshake Negotiation Failures by Reason | Peer handshakes rejected after TLS while checking network id, clock, keys and addresses. `reason` names the failing check. |
| `unl_fetch_total` (`site` = configured UNL URI; `outcome` = the 9 `ListDisposition` strings `accepted` \| `expired` \| `same_sequence` \| `pending` \| `known_sequence` \| `unsupported_version` \| `untrusted` \| `stale` \| `invalid`, plus `fetch_error` \| `bad_status` \| `parse_error`) | counter | `ValidatorSite.cpp` — `ValidatorSite::reportFetchOutcome` | UNL Fetch Rate by Site & Outcome | Validator-list fetches per site. `accepted` is the only success; `same_sequence` and `known_sequence` are normal no-op refreshes; the three literals are transport or content faults. |
| `unl_quorum` (`metric` = `trusted_keys` \| `quorum`) | observable gauge | `MetricsRegistry.cpp` — `registerUnlQuorumGauge` | UNL Trusted Keys vs Quorum; UNL Quorum Headroom | Trusted UNL key count against the validations a ledger needs. `trusted_keys` at or below `quorum` means the node can never declare a ledger validated. |
| `clock_close_offset_seconds` (`metric` = `offset`) | observable gauge | `MetricsRegistry.cpp` — `registerClockSkewGauge` | Clock Close Offset | Network close time offset from the local clock. Negative means the local clock runs ahead. `server_info` only surfaces `close_time_offset` at 60 s or more, so this gauge sees skew far earlier. |
| `state_changes_total` (`from`, `to` = `disconnected` \| `connected` \| `syncing` \| `tracking` \| `full`) | counter | `NetworkOPs.cpp` — `NetworkOPsImp::setMode` | Mode Transitions by Edge | Operating-mode transitions keyed on the (`from`, `to`) edge. The edge is what separates a clean `disconnected`→`connected`→`syncing`→`tracking`→`full` climb from `full`→`connected` flapping; an unlabelled total cannot tell them apart. |
| `sync_state` (`metric` = `initial_full_duration_us`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Time to First FULL | Microseconds from process start to the first `full` transition, sourced from `NetworkOPs::getInitialSyncDurationUs()`. Stays 0 until `full` is reached, so a flat 0 is itself the "never synced" signal; once set it never changes. |
| `sync_state` (`metric` = `network_ledger_gate`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Network Ledger Gate | 1 while the node is still waiting to see a full network ledger (`NetworkOPs::isNeedNetworkLedger()`), else 0. A persistent 1 blocks transaction submission and `full`, whatever the rest of the pipeline shows. |
| `sync_state` (`metric` = `server_stall_seconds`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Server Stall | Current main-loop stall duration from `LoadManager::getCurrentStallSeconds()`, 0 when healthy. Same duration the load monitor logs as "Server stalled for N seconds", which previously existed only in that log line. |
| `sync_state` (`metric` = `ledgers_behind`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Ledgers Behind Network | Peer-reported network tip minus our validated sequence, floored at 0 (`NetworkOPs::getLedgersBehindNetwork()`). Reads each peer's already cached ledger range, so no new network round trip. |
| `server_stall_events_total` | observable counter | `MetricsRegistry.cpp` — `registerStallEventsCounter` | Server Stall Event Rate | Distinct stall episodes since process start, counted once per episode rather than per stalled second. A rising rate is repeated fresh stalls; a flat rate with a large `server_stall_seconds` is one long stall. |
| `sync_acquire` (`metric` = `missing_state_nodes_max` \| `missing_tx_nodes_max`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Missing SHAMap Nodes per Acquire (state/tx) | Largest outstanding SHAMap node count across in-flight acquires, split by tree, from the count `getMissingNodes()` already produces during its sweep (`InboundLedger.cpp` — `InboundLedger::trigger`). **The headline stuck-sync signal:** flat and non-zero across ticks means the acquire will never finish; shrinking means slow but alive. Aggregated as a max rather than labelled per ledger, because a `ledger_seq` label would mint one series per ledger acquired — per-ledger identity stays on the `ledger.acquire` span. |
| `sync_acquire` (`metric` = `received_data_depth`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Peer packets stashed across all in-flight acquires waiting to be applied, summed because it measures one shared processing backlog. A growing depth means arriving node data outpaces processing, so the limit is the job queue or disk rather than peer supply. |
| `sync_acquire` (`metric` = `in_flight`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Number of ledger acquires currently running. Exported so the three values above can be read in context: all zero with `in_flight` zero is an idle node, not a healthy one. |
| `shamap_cache_hit_rate` (`metric` = `treenode`) | observable gauge | `MetricsRegistry.cpp` — `registerCacheHitRateDetailGauge` | SHAMap TreeNode Cache Hit Rate | Share of SHAMap tree-node lookups served from memory, from the previously-uncalled `TaggedCache::getHitRate()`, normalized from 0-100 to 0.0-1.0. Distinct from `nodestore_state`-derived NuDB Cache Hit Ratio on the Ledger Data Sync dashboard: this is the in-memory layer **above** the node store, so a miss here is what causes a read there. The full-below cache is not reported — it is a `KeyCache` whose only lookup path increments `stats_.hits`/`stats_.misses` while `getHitRate()` reads the separate `hits_`/`misses_` members, so its rate is hard-wired to 0 until that accounting is fixed. |
| `sync_acquire_no_progress_total` | counter | `InboundLedger.cpp` — `InboundLedger::onTimer` | Acquire Stall Rate (no progress) | Acquire timeouts where not one new node arrived since the previous timeout, from the `progress_` flag that was previously log-only. Fires on the 3 s acquire timer, never per node. A sustained rate together with a flat missing-node count is the definitive "stuck, not slow" signature. |
| `sync_addnode_total` (`outcome` = `good` \| `duplicate` \| `invalid`) | counter | `InboundLedger.cpp` — `InboundLedger::recordBatchOutcome` | Add-Node Outcomes | SHAMap nodes received during acquire, split by result. Emitted once per received packet from the aggregated batch tally the trace log already printed — never inside the per-node `receiveNode()` loop. Separates real progress (`good`) from wasted bandwidth (`duplicate`) and a misbehaving peer (`invalid`), all three of which look like healthy throughput in traffic metrics. |
| `sync_acquire_source_total` (`source` = `local` \| `network`) | counter | `InboundLedger.cpp` — `InboundLedger::init` | Acquire Source (local vs network) | Whether an acquire was satisfied entirely from the local node store or needed peers, emitted once per new acquire after the first local lookup. Sustained `network` on a node that should already hold the range means sync is disk-bound rather than peer-bound. |

678
docker/telemetry/grafana/dashboards/ledger-sync-health.json
View File

@@ -1441,6 +1441,624 @@
],
"title": "Mode Transitions by Edge",
"type": "bargauge"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "###### What this is:\n*Outstanding SHAMap nodes the busiest in-flight ledger acquire still needs, split by tree. This is the signal that separates a sync that is merely slow from one that will never finish.*\n\n###### How it's computed:\n*sync_acquire series missing_state_nodes_max and missing_tx_nodes_max: the largest outstanding node count across all in-flight acquires, refreshed after each getMissingNodes sweep. The maximum, not the sum, so one stuck acquire stays visible instead of being averaged away.*\n\n###### Reading it:\n*Falling toward zero means the acquire is progressing. A value pinned at 256 is the sweep cap, meaning there are at least that many nodes outstanding. Zero on one tree with a value on the other means that tree is already complete.*\n\n###### Healthy range:\n*Falling to 0 within seconds per ledger.*\n\n###### Watch for:\n*A flat, non-zero value across several minutes: no peer is serving that tree, so this acquire will never complete. Pair with Acquire Stall Rate \u2014 both flat and climbing together is the definitive stuck-sync signature.*\n\n###### Keywords:\n- **Missing SHAMap node** *(per node)* \u2014 a tree node this node needs to complete a ledger but does not yet hold.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerSyncAcquireGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#missing-shamap-node)",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "Nodes",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 2,
"pointSize": 3,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"showValues": false,
"spanNulls": 1800000,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "line"
}
},
"displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "yellow",
"value": 1
},
{
"color": "red",
"value": 256
}
]
},
"unit": "short"
}
},
"gridPos": {
"h": 12,
"w": 12,
"x": 0,
"y": 98
},
"id": 18,
"options": {
"annotations": {
"clustering": -1,
"multiLane": false
},
"legend": {
"calcs": [],
"displayMode": "list",
"enableFacetedFilter": false,
"overflow": "ellipsis",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"maxHeight": 600,
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "13.2.0-28926505616",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "label_replace(label_join(label_replace(sync_acquire{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=~\"$acquire_metric\", metric=~\"missing_(state|tx)_nodes_max\"}, \"series\", \"$1 tree\", \"metric\", \"missing_(.*)_nodes_max\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")",
"refId": "A"
}
],
"title": "Missing SHAMap Nodes per Acquire (state/tx)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "###### What this is:\n*Rate of ledger-acquire timeouts where not a single new node arrived since the previous timeout.*\n\n###### How it's computed:\n*rate of sync_acquire_no_progress_total, incremented on each acquire timeout whose progress flag was false. The acquire timer fires every 3 seconds at most.*\n\n###### Reading it:\n*Zero means every timeout window saw at least some new data. Any sustained rate means acquires are repeatedly timing out with nothing received.*\n\n###### Healthy range:\n*0 on a synced node; brief non-zero bursts during initial sync are normal.*\n\n###### Watch for:\n*A sustained rate together with a flat Missing SHAMap Nodes panel: the node is asking and no peer is answering. Check peer count and whether any peer holds the ledger range being requested.*\n\n###### Keywords:\n- **Acquire stall** *(per node)* \u2014 an acquire timeout in which no new SHAMap node was received, so the acquire made no progress at all.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (call-site metric macro, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[InboundLedger.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/InboundLedger.cpp)\n\n###### Function:\n`InboundLedger::onTimer`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#acquire-stall)",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "Stalls / Sec",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 2,
"pointSize": 3,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"showValues": false,
"spanNulls": 1800000,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "line"
}
},
"displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 0.1
}
]
},
"unit": "ops"
}
},
"gridPos": {
"h": 12,
"w": 12,
"x": 12,
"y": 98
},
"id": 19,
"options": {
"annotations": {
"clustering": -1,
"multiLane": false
},
"legend": {
"calcs": [],
"displayMode": "list",
"enableFacetedFilter": false,
"overflow": "ellipsis",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"maxHeight": 600,
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "13.2.0-28926505616",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "label_replace(label_join(label_replace(rate(sync_acquire_no_progress_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]), \"series\", \"Stalled Timeouts\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")",
"refId": "A"
}
],
"title": "Acquire Stall Rate (no progress)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "###### What this is:\n*Rate of SHAMap nodes received during ledger acquire, split by whether each node was useful, already held, or rejected as invalid.*\n\n###### How it's computed:\n*rate of sync_addnode_total by outcome (good / duplicate / invalid), emitted once per received packet from the batch tally that the acquire trace log already computed.*\n\n###### Reading it:\n*good is real progress. duplicate is bandwidth spent on nodes already held. invalid is a peer sending data that failed validation. Traffic-level metrics show all three as healthy throughput, which is why the split matters.*\n\n###### Healthy range:\n*good dominant during sync; a small duplicate share is normal.*\n\n###### Watch for:\n*A rising invalid share points at a specific misbehaving peer. A duplicate share that swamps good means peers keep re-sending known data, so the acquire burns bandwidth without progressing.*\n\n###### Keywords:\n- **Add-node outcome** *(per node)* \u2014 the result of applying one received SHAMap node: good (new and valid), duplicate (already held), or invalid (rejected).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (call-site metric macro, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[InboundLedger.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/InboundLedger.cpp)\n\n###### Function:\n`InboundLedger::recordBatchOutcome`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#add-node-outcome)",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "Nodes / Sec",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 30,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 2,
"pointSize": 3,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"showValues": false,
"spanNulls": 1800000,
"stacking": {
"group": "A",
"mode": "normal"
},
"thresholdsStyle": {
"mode": "off"
}
},
"displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "ops"
}
},
"gridPos": {
"h": 12,
"w": 12,
"x": 0,
"y": 110
},
"id": 20,
"options": {
"annotations": {
"clustering": -1,
"multiLane": false
},
"legend": {
"calcs": [],
"displayMode": "list",
"enableFacetedFilter": false,
"overflow": "ellipsis",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"maxHeight": 600,
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "13.2.0-28926505616",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "label_replace(label_join(label_replace(sum by (outcome, service_instance_id, xrpl_branch, xrpl_work_item) (rate(sync_addnode_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", outcome=~\"$addnode_outcome\"}[$__rate_interval])), \"series\", \"$1\", \"outcome\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")",
"refId": "A"
}
],
"title": "Add-Node Outcomes",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "###### What this is:\n*Rate of ledger acquires split by whether the local node store already held the whole ledger or the data had to come from peers.*\n\n###### How it's computed:\n*rate of sync_acquire_source_total by source, emitted once per new acquire right after the first local-store lookup.*\n\n###### Reading it:\n*network dominant during initial sync is expected \u2014 nothing is local yet. local dominant on a warm node means the store is serving requests without peer traffic.*\n\n###### Healthy range:\n*Mostly local on a warm node with complete history.*\n\n###### Watch for:\n*Sustained network on a node that should already hold the range: the local store is not retaining data, so sync is disk-bound rather than peer-bound. Read with the SHAMap cache hit-rate panel.*\n\n###### Keywords:\n- **Acquire source** *(per node)* \u2014 whether a ledger acquire was satisfied entirely from the local node store or required fetching from peers.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (call-site metric macro, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[InboundLedger.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/InboundLedger.cpp)\n\n###### Function:\n`InboundLedger::init`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#acquire-source)",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "Acquires / Sec",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 30,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 2,
"pointSize": 3,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"showValues": false,
"spanNulls": 1800000,
"stacking": {
"group": "A",
"mode": "normal"
},
"thresholdsStyle": {
"mode": "off"
}
},
"displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
}
]
},
"unit": "ops"
}
},
"gridPos": {
"h": 12,
"w": 12,
"x": 12,
"y": 110
},
"id": 21,
"options": {
"annotations": {
"clustering": -1,
"multiLane": false
},
"legend": {
"calcs": [],
"displayMode": "list",
"enableFacetedFilter": false,
"overflow": "ellipsis",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"maxHeight": 600,
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "13.2.0-28926505616",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "label_replace(label_join(label_replace(sum by (source, service_instance_id, xrpl_branch, xrpl_work_item) (rate(sync_acquire_source_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", source=~\"$acquire_source\"}[$__rate_interval])), \"series\", \"$1\", \"source\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")",
"refId": "A"
}
],
"title": "Acquire Source (local vs network)",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "###### What this is:\n*Fraction of SHAMap tree-node lookups served from memory instead of the node store. The layer above the NuDB cache: a miss here is what causes a node-store read.*\n\n###### How it's computed:\n*shamap_cache_hit_rate series treenode, from TaggedCache::getHitRate() on the node family's tree-node cache, normalized from 0-100 to 0.0-1.0.*\n\n###### Reading it:\n*Near 1.0 on a warm node. Low during a fresh sync while the cache fills. Distinct from the NuDB Cache Hit Ratio panel on the Ledger Data Sync dashboard, which measures the node-store layer beneath this one.*\n\n###### Healthy range:\n*> 0.9 on a warm node.*\n\n###### Watch for:\n*A persistently low rate on a node that should be warm: the working set does not fit the cache, or continuous re-acquisition is churning it, so every tree walk pays disk latency. Read with Acquire Source.*\n\n###### Keywords:\n- **SHAMap cache hit rate** *(per node)* \u2014 the share of SHAMap tree-node lookups answered from the in-memory cache rather than the node store.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerCacheHitRateDetailGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#shamap-cache-hit-rate)",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "Hit Rate",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 2,
"pointSize": 3,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"showValues": false,
"spanNulls": 1800000,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "line"
}
},
"displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "red",
"value": null
},
{
"color": "yellow",
"value": 0.5
},
{
"color": "green",
"value": 0.9
}
]
},
"unit": "percentunit"
}
},
"gridPos": {
"h": 12,
"w": 12,
"x": 0,
"y": 122
},
"id": 22,
"options": {
"annotations": {
"clustering": -1,
"multiLane": false
},
"legend": {
"calcs": [],
"displayMode": "list",
"enableFacetedFilter": false,
"overflow": "ellipsis",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"maxHeight": 600,
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "13.2.0-28926505616",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "label_replace(label_join(label_replace(shamap_cache_hit_rate{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}, \"series\", \"TreeNode Cache\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")",
"refId": "A"
}
],
"title": "SHAMap TreeNode Cache Hit Rate",
"type": "timeseries"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "###### What this is:\n*Peer packets stashed across all in-flight acquires waiting to be applied, alongside how many acquires are running.*\n\n###### How it's computed:\n*sync_acquire series received_data_depth (summed across acquires) and in_flight (the acquire count). The depth mirrors the receive stash size; in_flight gives the context that makes an all-zero reading legible.*\n\n###### Reading it:\n*A depth near zero means node data is applied as fast as it arrives. in_flight at zero means the node is idle, which is why an all-zero Missing Nodes panel is not by itself a healthy reading.*\n\n###### Healthy range:\n*Depth 0 to a few; in_flight low single digits during sync.*\n\n###### Watch for:\n*A growing depth means arriving data outpaces processing, which is a job-queue or disk problem rather than a peer-supply one. Check the job-queue backlog next.*\n\n###### Keywords:\n- **Received-data stash** *(per node)* \u2014 peer packets held for later processing because the acquire cannot apply them as fast as they arrive.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerSyncAcquireGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#received-data-stash)",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisBorderShow": false,
"axisCenteredZero": false,
"axisColorMode": "text",
"axisLabel": "Count",
"axisPlacement": "auto",
"barAlignment": 0,
"barWidthFactor": 0.6,
"drawStyle": "line",
"fillOpacity": 10,
"gradientMode": "none",
"hideFrom": {
"legend": false,
"tooltip": false,
"viz": false
},
"insertNulls": false,
"lineInterpolation": "linear",
"lineWidth": 2,
"pointSize": 3,
"scaleDistribution": {
"type": "linear"
},
"showPoints": "auto",
"showValues": false,
"spanNulls": 1800000,
"stacking": {
"group": "A",
"mode": "none"
},
"thresholdsStyle": {
"mode": "line"
}
},
"displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}",
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "yellow",
"value": 16
}
]
},
"unit": "short"
}
},
"gridPos": {
"h": 12,
"w": 12,
"x": 12,
"y": 122
},
"id": 23,
"options": {
"annotations": {
"clustering": -1,
"multiLane": false
},
"legend": {
"calcs": [],
"displayMode": "list",
"enableFacetedFilter": false,
"overflow": "ellipsis",
"placement": "bottom",
"showLegend": true
},
"tooltip": {
"hideZeros": false,
"maxHeight": 600,
"mode": "multi",
"sort": "desc"
}
},
"pluginVersion": "13.2.0-28926505616",
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "label_replace(label_join(label_replace(sync_acquire{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=~\"$acquire_metric\", metric=~\"received_data_depth|in_flight\"}, \"series\", \"$1\", \"metric\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")",
"refId": "A"
}
],
"title": "Received-Data Stash Depth & In-Flight Acquires",
"type": "timeseries"
}
],
"schemaVersion": 39,
@@ -1719,6 +2337,66 @@
"multi": true,
"refresh": 2,
"sort": 1
},
{
"name": "acquire_metric",
"label": "Acquire Metric",
"description": "Filter the acquire-progress gauge sub-series [missing_state_nodes_max / missing_tx_nodes_max / received_data_depth / in_flight]",
"type": "query",
"query": "label_values(sync_acquire, metric)",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"includeAll": true,
"allValue": ".*",
"current": {
"text": "All",
"value": "$__all"
},
"multi": true,
"refresh": 2,
"sort": 1
},
{
"name": "addnode_outcome",
"label": "Add-Node Outcome",
"description": "Filter received SHAMap nodes by outcome [good / duplicate / invalid]",
"type": "query",
"query": "label_values(sync_addnode_total, outcome)",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"includeAll": true,
"allValue": ".*",
"current": {
"text": "All",
"value": "$__all"
},
"multi": true,
"refresh": 2,
"sort": 1
},
{
"name": "acquire_source",
"label": "Acquire Source",
"description": "Filter ledger acquires by where the data came from [local / network]",
"type": "query",
"query": "label_values(sync_acquire_source_total, source)",
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"includeAll": true,
"allValue": ".*",
"current": {
"text": "All",
"value": "$__all"
},
"multi": true,
"refresh": 2,
"sort": 1
}
]
},

8
docker/telemetry/workload/expected_metrics.json
View File

@@ -144,8 +144,14 @@
"sync_state{metric=\"server_stall_seconds\"}",
"sync_state{metric=\"ledgers_behind\"}",
"server_stall_events_total",
"state_changes_total{from!=\"\",to!=\"\"}"
"state_changes_total{from!=\"\",to!=\"\"}",
"sync_acquire{metric=\"missing_state_nodes_max\"}",
"sync_acquire{metric=\"missing_tx_nodes_max\"}",
"sync_acquire{metric=\"received_data_depth\"}",
"sync_acquire{metric=\"in_flight\"}",
"shamap_cache_hit_rate{metric=\"treenode\"}"
],
"_acquire_note": "The four sync_acquire sub-series and shamap_cache_hit_rate are unconditional: both are observable gauges whose callbacks observe every series on each collection tick, so each is present even when the value is 0 (an idle node reports in_flight=0 and missing_state_nodes_max=0, and a cold cache reports a 0.0 hit rate). Absence, not a zero, is the regression. The three WP-A3 counters (sync_acquire_source_total, sync_addnode_total, sync_acquire_no_progress_total) are deliberately NOT asserted here: all three are emitted only from InboundLedger, which runs only when a node must fetch a ledger it lacks. expected_spans.json already marks the ledger.acquire span optional for exactly this reason (\"A healthy local cluster rarely back-fills history\"), and the metric validator has no per-metric optional flag, so listing them would fail the harness red on a healthy run. They are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp and by the ledger-sync-health panels; add them here only alongside a harness step that forces a real acquire (e.g. starting a node against an existing ledger history).",
"_conditional_note": "handshake_negotiation_fail_total and unl_fetch_total are conditional under the local harness: the first only exists once a handshake is rejected, and the second needs a [validator_list_sites] entry (run-full-validation.sh generates a static [validators] file instead). The validator has no per-metric optional flag, so if either reports 0 series in a harness run, move it out of this group rather than weakening the check.",
"_sync_state_note": "The four sync_state sub-series are unconditional: the gauge observes all four on every collection tick, so each is present as a series even when its value is 0 (a node that never reached FULL reports initial_full_duration_us=0, and a healthy node reports server_stall_seconds=0). The check asserts series presence, not a non-zero value, which is exactly right here — a zero is a meaningful reading for these signals, and absence is the regression. server_stall_events_total is likewise always present because the observable counter reports the tally (0 or more) every tick. state_changes_total is asserted here with a from!=\"\",to!=\"\" selector rather than bare (parity_counters already asserts the bare name): the selector is what proves the WP-A2 {from,to} label dimension actually reached Prometheus, so a regression to the old unlabelled counter fails this check instead of silently passing on the bare name. It needs at least one real mode transition, which any node reaching connected/syncing produces during startup."
},

10
docker/telemetry/workload/validate_telemetry.py
View File

@@ -598,6 +598,16 @@ async def validate_metrics(
"sync_state",
"state_changes_total",
"server_stall_events",
# Acquire + SHAMap signals. sync_acquire carries the
# missing-node, stash-depth and in-flight sub-series and
# shamap_cache_hit_rate the tree-node cache rate; both
# are asserted. The sync_acquire_* / sync_addnode_total
# counters are listed here for diagnosis only -- they
# need a real ledger acquire, so they are not asserted
# (see _acquire_note in expected_metrics.json).
"sync_acquire",
"sync_addnode_total",
"shamap_cache_hit_rate",
)
)
]

60
docs/telemetry-glossary.md
View File

@@ -519,6 +519,36 @@ Turning each configured peer hostname into IP addresses, which happens before an
**See also:** [Outbound dial latency](#outbound-dial-latency) · [Peer protocol on xrpl.org](https://xrpl.org/docs/concepts/networks-and-servers/peer-protocol)
<a id="acquire-source"></a>
### Acquire source
Whether a ledger acquire was satisfied entirely from the local node store or required fetching the data from peers. During a genuine fresh sync almost every acquire is network-sourced, because nothing is local yet. The signal becomes diagnostic on a node that should already hold the range: acquires that still go to the network mean the local store is not retaining data, so the slowness is disk-bound rather than peer-bound. This is the pairing that explains why a node with a large existing database can start slower than a fresh one.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [SHAMap cache hit rate](#shamap-cache-hit-rate) · [Ledger acquire (inbound fetch)](#ledger-acquire-inbound-fetch)
<a id="acquire-stall"></a>
### Acquire stall
A ledger-acquire timeout in which not a single new tree node arrived since the previous timeout, so the acquire made no progress at all. Distinct from a slow acquire, which still receives data between timeouts. A sustained stall rate alongside a missing-node count that never falls is the definitive "this sync will never complete" signature: the node keeps asking and no peer answers.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Missing SHAMap node](#missing-shamap-node) · [Ledger acquire (inbound fetch)](#ledger-acquire-inbound-fetch)
<a id="add-node-outcome"></a>
### Add-node outcome
The result of applying one SHAMap node received from a peer during a ledger acquire: good (new and valid), duplicate (already held), or invalid (failed validation). The split matters because traffic-level metrics count all three as healthy throughput. Only good represents progress; a duplicate share that swamps it means peers keep re-sending data the node already has, and a rising invalid share points at one misbehaving peer rather than a local fault.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Received-data stash](#received-data-stash) · [Missing SHAMap node](#missing-shamap-node)
<a id="fresh-node-sync-diagnostics"></a>
### Fresh-node sync diagnostics
@@ -565,6 +595,16 @@ How many ledgers this node's validated sequence trails the network's. The networ
**See also:** [Time to first FULL](#time-to-first-full) · [Ledger acquire (inbound fetch)](#ledger-acquire-inbound-fetch) · [Validated ledger on xrpl.org](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers)
<a id="missing-shamap-node"></a>
### Missing SHAMap node
A node of a ledger's account-state or transaction tree that this server needs in order to complete the ledger but does not yet hold. The count of outstanding missing nodes is the clearest available answer to "is this acquire progressing?": a count falling toward zero is progress, while a count that stays flat and non-zero means no peer is serving that tree and the acquire will never finish. Reported per tree, as the maximum across in-flight acquires, and capped per sweep — so a value sitting at the cap means the real backlog is at least that large, and only the trend distinguishes a large-but-progressing tree from a stuck one.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Acquire stall](#acquire-stall) · [Received-data stash](#received-data-stash) · [Ledger acquire (inbound fetch)](#ledger-acquire-inbound-fetch)
<a id="network-ledger-gate"></a>
### Network ledger gate
@@ -595,6 +635,26 @@ The elapsed time of one outbound peer connection attempt, from starting the TCP
**See also:** [DNS resolve](#dns-resolve) · [Handshake negotiation failure](#handshake-negotiation-failure) · [Peer protocol on xrpl.org](https://xrpl.org/docs/concepts/networks-and-servers/peer-protocol)
<a id="received-data-stash"></a>
### Received-data stash
Peer packets held for later processing because a ledger acquire cannot apply them as fast as they arrive. A shallow stash means node data is applied as it lands. A growing stash means the bottleneck is local processing — job-queue depth or disk latency — rather than peer supply, which is the opposite conclusion from an acquire that receives nothing at all. Read alongside the in-flight acquire count, since an empty stash on an idle node says nothing about acquire health.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Add-node outcome](#add-node-outcome) · [Acquire stall](#acquire-stall)
<a id="shamap-cache-hit-rate"></a>
### SHAMap cache hit rate
The share of SHAMap tree-node lookups answered from the in-memory tree-node cache rather than the node store. This is the layer above the node store's own hit ratio: a miss here is what causes a node-store read there. A low rate is expected during a fresh sync while the cache fills. A persistently low rate on a node that should be warm means the working set does not fit the cache, or continuous re-acquisition is churning it, so every tree walk pays disk latency.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Acquire source](#acquire-source) · [Missing SHAMap node](#missing-shamap-node)
<a id="server-stall"></a>
### Server stall

60
docs/telemetry-runbook.md
View File

@@ -2214,6 +2214,66 @@ each step gates the next: stop at the first one that is wrong.
of `full` once it has arrived. Use the _Mode From_ and _Mode To_ template
variables to isolate one edge.
6. **Is ledger acquisition progressing, or permanently stuck?**
Panel _Missing SHAMap Nodes per Acquire (state/tx)_ (`sync_acquire`,
`metric=missing_state_nodes_max` and `missing_tx_nodes_max`). **This is the
panel that separates "sync is slow" from "sync will never finish."** Read the
shape over several minutes, not the instantaneous value:
- **Falling toward zero** — the acquire is progressing. Slow is not stuck.
- **Flat and non-zero** — no peer is serving that tree. The node will sit
here forever; no amount of waiting fixes it. Go to step 7.
- **Pinned at 256** — that is the per-sweep cap (`kMissingNodesFind`), so the
real backlog is at least that large. Meaningful only in combination with
the trend: pinned-and-falling is a large but progressing tree.
- **Zero on one tree, non-zero on the other** — that tree is already
complete; concentrate on the one still reporting nodes.
One caveat: zero on both is only healthy if _In-Flight Acquires_ (step 8)
is non-zero. Zero everywhere with zero acquires in flight is an idle node,
which says nothing about acquire health.
7. **Is the node asking and getting nothing back?**
Panel _Acquire Stall Rate (no progress)_ (`sync_acquire_no_progress_total`).
This counts acquire timeouts in which not a single node arrived. A sustained
rate here **together with** a flat missing-node count from step 6 is the
definitive stuck-sync signature: the node is requesting and no peer is
answering. Check the peer count and whether any connected peer actually holds
the ledger range being requested — a peer set that cannot serve the range
looks identical to no peers at all from inside the acquire.
8. **Is the data arriving useful, or wasted?**
Panel _Add-Node Outcomes_ (`sync_addnode_total`, stacked by `outcome`).
Traffic-level metrics count all three outcomes as healthy throughput, which
is why this split matters:
- `good` — new, valid nodes. This is the only line that represents progress.
- `duplicate` — nodes already held. A share that swamps `good` means peers
keep re-sending known data, so bandwidth is busy while the acquire stands
still.
- `invalid` — nodes that failed validation. A rising share points at a
specific misbehaving peer rather than a local fault.
Then read _Received-Data Stash Depth & In-Flight Acquires_
(`sync_acquire`, `metric=received_data_depth` and `in_flight`). A growing
stash means node data is arriving faster than it can be applied, which is a
job-queue or disk problem, not a peer-supply one — the opposite conclusion
from step 7, and the two are distinguished only by this panel.
9. **Is it disk-bound rather than peer-bound?**
Panels _Acquire Source (local vs network)_ (`sync_acquire_source_total`) and
_SHAMap TreeNode Cache Hit Rate_ (`shamap_cache_hit_rate`). During a genuine
fresh sync `network` dominates and the cache hit rate is low — both expected,
because nothing is local yet. The diagnostic case is a node that should
already be warm:
- Sustained `network` on a range the node should already hold means the local
store is not retaining data.
- A persistently low tree-node hit rate means the working set does not fit
the cache, or continuous re-acquisition is churning it, so every tree walk
pays disk latency.
Note this is the in-memory tree-node cache, one layer **above** the
_NuDB Cache Hit Ratio_ panel on the Ledger Data Sync dashboard: a miss here
is what produces a node-store read there. A low rate on both is disk-bound
sync; a low rate here with a healthy NuDB ratio is cache pressure alone.
This is also the pairing that explains a large existing database syncing
slower than a fresh one.
## Performance Tuning
| Scenario | Recommendation |

34
include/xrpl/shamap/SHAMapAddNode.h
View File

@@ -24,6 +24,28 @@ public:
reset();
[[nodiscard]] int
getGood() const;
/**
* Nodes rejected as invalid in this tally.
*
* Complements getGood(): isInvalid() only answers "was there at least one",
* which cannot distinguish one bad node from a peer sending nothing but bad
* data. Exposed for the acquire telemetry counters, which need the count.
*
* @return Number of invalid nodes; 0 if none.
*/
[[nodiscard]] int
getBad() const;
/**
* Nodes already held, so re-receiving them was wasted work.
*
* A high duplicate share against a low good share means peers are re-sending
* data the node already has, which looks like healthy traffic but makes no
* acquire progress.
*
* @return Number of duplicate nodes; 0 if none.
*/
[[nodiscard]] int
getDuplicate() const;
[[nodiscard]] bool
isGood() const;
[[nodiscard]] bool
@@ -86,6 +108,18 @@ SHAMapAddNode::getGood() const
return good_;
}
inline int
SHAMapAddNode::getBad() const
{
return bad_;
}
inline int
SHAMapAddNode::getDuplicate() const
{
return duplicate_;
}
inline bool
SHAMapAddNode::isInvalid() const
{

8
src/test/app/LedgerReplay_test.cpp
View File

@@ -252,6 +252,14 @@ public:
return 0;
}
// This mock holds no InboundLedger objects, so there is no acquire progress
// to report; the all-zero snapshot is the honest answer.
AcquireProgress
acquireProgress() override
{
return {};
}
LedgerMaster& ledgerSource;
LedgerMaster& ledgerSink;
InboundLedgersBehavior bhvr;

338
src/tests/libxrpl/telemetry/MetricMacros.cpp
View File

@@ -1246,4 +1246,342 @@ TEST(MetricMacros, state_changes_total_emits_nothing_when_registry_disabled)
EXPECT_EQ(app.registry().meterCalls(), 0);
}
// -----------------------------------------------------------------
// Acquire + SHAMap sync diagnostics (WP-A3).
//
// Asserts the EXACT values and label shapes of the five acquire signals:
// sync_acquire_source_total{source} InboundLedger::init
// sync_acquire_no_progress_total InboundLedger::onTimer
// sync_addnode_total{outcome} InboundLedger::recordBatchOutcome
// sync_acquire{metric} MetricsRegistry::registerSyncAcquireGauge
// missing_state_nodes_max
// missing_tx_nodes_max
// received_data_depth
// in_flight
// shamap_cache_hit_rate{metric} MetricsRegistry::registerCacheHitRateDetailGauge
//
// The counters go through the same macros production uses. The two observable
// instruments are registered directly on the SDK meter, mirroring the production
// callback shape, because the real MetricsRegistry's enabled path cannot be
// linked into this standalone binary (see the file header).
// -----------------------------------------------------------------
// sync_acquire_source_total splits acquires by whether the local node store
// already held the ledger. This is the disk-bound vs peer-bound distinction, so
// the two sources must never collapse into one series.
TEST(MetricMacros, acquire_source_splits_local_and_network)
{
CollectingProvider const provider;
FakeApp app;
wire(app, /*enabled=*/true, provider.meter());
// Mirrors the production call site in InboundLedger::init(): one macro
// invocation whose label is derived from complete_ after the first tryDB().
auto const acquire = [&app](bool localComplete) {
XRPL_METRIC_COUNTER_INC_LABELED(
app,
"sync_acquire_source_total",
"Ledger acquires by where the data came from",
{{"source", std::string(localComplete ? "local" : "network")}});
};
// One satisfied locally, two needing the network.
acquire(true);
acquire(false);
acquire(false);
auto const data = provider.collect();
ASSERT_EQ(data.at("sync_acquire_source_total").size(), 2u);
EXPECT_EQ(counterValue(data, "sync_acquire_source_total", attrs("source", "local")), 1);
EXPECT_EQ(counterValue(data, "sync_acquire_source_total", attrs("source", "network")), 2);
// The label key is exactly "source" and nothing rides along with it.
auto const& firstKey = data.at("sync_acquire_source_total").begin()->first;
ASSERT_EQ(firstKey.size(), 1u);
EXPECT_EQ(firstKey.begin()->first, "source");
// NEGATIVE: a source value that was never emitted has no series, so the
// counts above are not an artifact of a catch-all series.
EXPECT_EQ(data.at("sync_acquire_source_total").count(attrs("source", "fetch_pack")), 0u);
}
// sync_acquire_no_progress_total counts ONLY timeouts where no node arrived.
// InboundLedger::onTimer reaches the macro exclusively on its !wasProgress
// branch, so a tick that made progress must leave the total unchanged -- that
// is the whole difference between "slow" and "stuck".
TEST(MetricMacros, acquire_no_progress_counts_only_stalled_timeouts)
{
CollectingProvider const provider;
FakeApp app;
wire(app, /*enabled=*/true, provider.meter());
// Stands in for onTimer(): the guard lives at the call site in production,
// so the harness reproduces the guard rather than the macro alone.
auto const onTimer = [&app](bool wasProgress) {
if (!wasProgress)
{
XRPL_METRIC_COUNTER_INC(
app,
"sync_acquire_no_progress_total",
"Ledger-acquire timeouts where no new node arrived");
}
};
onTimer(/*wasProgress=*/false);
onTimer(/*wasProgress=*/false);
// Exactly two stalled timeouts so far, on one unlabelled series.
auto const stalled = provider.collect();
ASSERT_EQ(stalled.at("sync_acquire_no_progress_total").size(), 1u);
EXPECT_TRUE(stalled.at("sync_acquire_no_progress_total").begin()->first.empty());
EXPECT_EQ(
counterValue(stalled, "sync_acquire_no_progress_total", otel_sdk::PointAttributes{}), 2);
// A tick that DID make progress must not advance the counter: still 2.
onTimer(/*wasProgress=*/true);
EXPECT_EQ(
counterValue(
provider.collect(), "sync_acquire_no_progress_total", otel_sdk::PointAttributes{}),
2);
// A further stalled tick does advance it, proving the counter is live and
// the unchanged reading above was the guard working, not a dead instrument.
onTimer(/*wasProgress=*/false);
EXPECT_EQ(
counterValue(
provider.collect(), "sync_acquire_no_progress_total", otel_sdk::PointAttributes{}),
3);
}
// sync_addnode_total separates useful progress from wasted work. All three
// outcomes come from ONE aggregated batch tally, added after the per-node loop
// has finished, so each outcome must land on its own series with its exact count.
TEST(MetricMacros, addnode_outcomes_record_exact_batch_tallies)
{
CollectingProvider const provider;
FakeApp app;
wire(app, /*enabled=*/true, provider.meter());
// Mirrors InboundLedger::recordBatchOutcome(): three _ADD calls per batch,
// each skipped when its tally is zero (a zero Add would create a series that
// says "we saw invalid nodes", which would be false).
auto const emitBatch = [&app](int good, int duplicate, int invalid) {
auto const emit = [&app](char const* outcome, int count) {
if (count <= 0)
return;
XRPL_METRIC_COUNTER_ADD_LABELED(
app,
"sync_addnode_total",
"SHAMap nodes received during ledger acquire, by outcome",
static_cast<std::uint64_t>(count),
{{"outcome", std::string(outcome)}});
};
emit("good", good);
emit("duplicate", duplicate);
emit("invalid", invalid);
};
// One batch: 5 good, 2 duplicate, 1 invalid.
emitBatch(5, 2, 1);
auto const oneBatch = provider.collect();
ASSERT_EQ(oneBatch.at("sync_addnode_total").size(), 3u);
EXPECT_EQ(counterValue(oneBatch, "sync_addnode_total", attrs("outcome", "good")), 5);
EXPECT_EQ(counterValue(oneBatch, "sync_addnode_total", attrs("outcome", "duplicate")), 2);
EXPECT_EQ(counterValue(oneBatch, "sync_addnode_total", attrs("outcome", "invalid")), 1);
// Every series key carries exactly the one expected label name.
for (auto const& [labels, point] : oneBatch.at("sync_addnode_total"))
{
ASSERT_EQ(labels.size(), 1u);
EXPECT_EQ(labels.count("outcome"), 1u);
}
// A second batch accumulates per outcome rather than replacing: 3 more good
// and 4 more duplicates, no invalid this time.
emitBatch(3, 4, 0);
auto const twoBatches = provider.collect();
EXPECT_EQ(counterValue(twoBatches, "sync_addnode_total", attrs("outcome", "good")), 8);
EXPECT_EQ(counterValue(twoBatches, "sync_addnode_total", attrs("outcome", "duplicate")), 6);
// The zero-tally outcome did NOT advance: still exactly 1 from the first
// batch, so an all-good batch cannot inflate the invalid series.
EXPECT_EQ(counterValue(twoBatches, "sync_addnode_total", attrs("outcome", "invalid")), 1);
// Still exactly three series after two batches: the zero-tally guard means a
// batch never invents a series for an outcome it did not observe.
EXPECT_EQ(twoBatches.at("sync_addnode_total").size(), 3u);
// NEGATIVE: an outcome value outside the production set has no series, so
// the three counts above are not an artifact of a catch-all series.
EXPECT_EQ(twoBatches.at("sync_addnode_total").count(attrs("outcome", "stale")), 0u);
}
// sync_acquire fans four values out of ONE aggregated snapshot, mirroring
// MetricsRegistry::registerSyncAcquireGauge(). The values chosen are the
// headline stuck-sync reading: two acquires in flight, the state tree still
// missing nodes, a backed-up stash.
TEST(MetricMacros, sync_acquire_gauge_observes_exact_stuck_acquire_values)
{
CollectingProvider const provider;
// The snapshot the callback reports, owned by the test exactly as the real
// registry reads it from InboundLedgers on each collection tick.
struct Observed
{
std::int64_t maxMissingStateNodes;
std::int64_t maxMissingTxNodes;
std::int64_t receivedDataDepth;
std::int64_t inFlight;
};
// A stuck acquire: 256 state nodes still outstanding (the sweep cap), the tx
// tree already done, 4 packets stashed, 2 acquires running.
Observed observed{256, 0, 4, 2};
// Keep the instrument alive for the whole test: destroying the handle
// deregisters the callback, which is why the real registry holds a member.
auto gauge = provider.meter()->CreateInt64ObservableGauge(
"sync_acquire", "Aggregate ledger-acquire progress across in-flight acquires");
gauge->AddCallback(
[](opentelemetry::metrics::ObserverResult result, void* state) {
auto const* self = static_cast<Observed const*>(state);
// Same Observe() form the production callback uses.
auto observe = [&](char const* name, std::int64_t value) {
opentelemetry::nostd::get<opentelemetry::nostd::shared_ptr<
opentelemetry::metrics::ObserverResultT<std::int64_t>>>(result)
->Observe(value, {{"metric", name}});
};
observe("missing_state_nodes_max", self->maxMissingStateNodes);
observe("missing_tx_nodes_max", self->maxMissingTxNodes);
observe("received_data_depth", self->receivedDataDepth);
observe("in_flight", self->inFlight);
},
&observed);
auto const stuck = provider.collect();
// Exactly four series, one per `metric` value.
ASSERT_EQ(stuck.at("sync_acquire").size(), 4u);
EXPECT_EQ(gaugeValue(stuck, "sync_acquire", attrs("metric", "missing_state_nodes_max")), 256);
EXPECT_EQ(gaugeValue(stuck, "sync_acquire", attrs("metric", "received_data_depth")), 4);
EXPECT_EQ(gaugeValue(stuck, "sync_acquire", attrs("metric", "in_flight")), 2);
// Zero is a REAL reading here, not a missing series: it says the tx tree
// needs nothing while the state tree is still stuck, which is exactly the
// per-map split this signal exists to provide.
ASSERT_EQ(stuck.at("sync_acquire").count(attrs("metric", "missing_tx_nodes_max")), 1u);
EXPECT_EQ(gaugeValue(stuck, "sync_acquire", attrs("metric", "missing_tx_nodes_max")), 0);
// The label key is exactly "metric" and it is the only label present. This
// is the cardinality guard: a ledger_seq label here would mint a new series
// per ledger acquired.
auto const& firstKey = stuck.at("sync_acquire").begin()->first;
ASSERT_EQ(firstKey.size(), 1u);
EXPECT_EQ(firstKey.begin()->first, "metric");
EXPECT_EQ(stuck.at("sync_acquire").count(attrs("metric", "ledger_seq")), 0u);
// A shrinking count is the "slow but alive" reading, and an idle node
// reports all zeros with in_flight=0 -- distinguishable from a stuck node
// only because in_flight is exported alongside.
observed = Observed{128, 0, 1, 2};
EXPECT_EQ(
gaugeValue(provider.collect(), "sync_acquire", attrs("metric", "missing_state_nodes_max")),
128);
observed = Observed{0, 0, 0, 0};
auto const idle = provider.collect();
EXPECT_EQ(gaugeValue(idle, "sync_acquire", attrs("metric", "missing_state_nodes_max")), 0);
EXPECT_EQ(gaugeValue(idle, "sync_acquire", attrs("metric", "in_flight")), 0);
}
// shamap_cache_hit_rate reports the tree-node cache rate normalized to 0.0-1.0,
// mirroring MetricsRegistry::registerCacheHitRateDetailGauge(). The scaling is
// the part worth pinning: TaggedCache::getHitRate() returns 0-100, and the
// dashboard panel uses percentunit, so an unnormalized value would render as
// 9000% instead of 90%.
TEST(MetricMacros, shamap_cache_hit_rate_gauge_normalizes_to_unit_fraction)
{
CollectingProvider const provider;
// What TaggedCache::getHitRate() would return: 90 means 90%.
float rawHitRatePercent = 90.0F;
auto gauge = provider.meter()->CreateDoubleObservableGauge(
"shamap_cache_hit_rate", "SHAMap tree-node cache hit rate (0.0-1.0), by cache");
gauge->AddCallback(
[](opentelemetry::metrics::ObserverResult result, void* state) {
auto const* raw = static_cast<float const*>(state);
// Same normalization the production callback performs.
opentelemetry::nostd::get<
opentelemetry::nostd::shared_ptr<opentelemetry::metrics::ObserverResultT<double>>>(
result)
->Observe(static_cast<double>(*raw / 100.0F), {{"metric", "treenode"}});
},
&rawHitRatePercent);
auto const warm = provider.collect();
// Exactly one series: the full-below cache is deliberately not reported
// (its TaggedCache hit accounting writes members getHitRate() never reads,
// so it would be a hard-wired zero).
ASSERT_EQ(warm.at("shamap_cache_hit_rate").size(), 1u);
EXPECT_EQ(warm.at("shamap_cache_hit_rate").count(attrs("metric", "full_below")), 0u);
// 90 percent arrives as exactly 0.9, not 90 and not 9000.
auto const& warmPoint = warm.at("shamap_cache_hit_rate").at(attrs("metric", "treenode"));
auto const& warmLast = opentelemetry::nostd::get<otel_sdk::LastValuePointData>(warmPoint);
EXPECT_DOUBLE_EQ(opentelemetry::nostd::get<double>(warmLast.value_), 0.9);
// A cold cache reads exactly 0.0 -- the fresh-sync case, where every lookup
// goes to the node store.
rawHitRatePercent = 0.0F;
auto const cold = provider.collect();
auto const& coldPoint = cold.at("shamap_cache_hit_rate").at(attrs("metric", "treenode"));
auto const& coldLast = opentelemetry::nostd::get<otel_sdk::LastValuePointData>(coldPoint);
EXPECT_DOUBLE_EQ(opentelemetry::nostd::get<double>(coldLast.value_), 0.0);
// A fully warm cache reads exactly 1.0, pinning the upper bound of the
// normalized range.
rawHitRatePercent = 100.0F;
auto const full = provider.collect();
auto const& fullPoint = full.at("shamap_cache_hit_rate").at(attrs("metric", "treenode"));
auto const& fullLast = opentelemetry::nostd::get<otel_sdk::LastValuePointData>(fullPoint);
EXPECT_DOUBLE_EQ(opentelemetry::nostd::get<double>(fullLast.value_), 1.0);
}
// RUNTIME-DISABLED no-op proof for the counter half of WP-A3: with the registry
// disabled, all three acquire counters emit NOTHING -- no series at all, and
// meter() is never consulted, so not even an instrument was created.
TEST(MetricMacros, acquire_counters_emit_nothing_when_registry_disabled)
{
CollectingProvider const provider;
FakeApp app;
wire(app, /*enabled=*/false, provider.meter());
XRPL_METRIC_COUNTER_INC_LABELED(
app,
"sync_acquire_source_total",
"Ledger acquires by where the data came from",
{{"source", std::string("network")}});
XRPL_METRIC_COUNTER_INC(
app, "sync_acquire_no_progress_total", "Ledger-acquire timeouts where no new node arrived");
XRPL_METRIC_COUNTER_ADD_LABELED(
app,
"sync_addnode_total",
"SHAMap nodes received during ledger acquire, by outcome",
static_cast<std::uint64_t>(5),
{{"outcome", std::string("good")}});
auto const data = provider.collect();
// Total absence, not zero-valued series: the instruments never existed.
EXPECT_EQ(data.count("sync_acquire_source_total"), 0u);
EXPECT_EQ(data.count("sync_acquire_no_progress_total"), 0u);
EXPECT_EQ(data.count("sync_addnode_total"), 0u);
EXPECT_EQ(data.size(), 0u);
// Cause, not just state: the isEnabled() gate short-circuited before the
// macros asked for a meter.
EXPECT_EQ(app.registry().meterCalls(), 0);
}
#endif // XRPL_ENABLE_TELEMETRY

32
src/tests/libxrpl/telemetry/MetricsRegistry.cpp
View File

@@ -18,15 +18,17 @@
*
* CONSEQUENCE for the sync-diagnostics gauges (`unl_quorum`,
* `clock_close_offset_seconds`, `sync_state`,
* `server_stall_events_total`): this file CANNOT assert an observed gauge
* `server_stall_events_total`, `sync_acquire`, `shamap_cache_hit_rate`):
* this file CANNOT assert an observed gauge
* value, because on this build the gauges do not exist -- their registration
* methods and the OTel instrument members are inside
* `#ifdef XRPL_ENABLE_TELEMETRY`, and there is no MeterProvider at all. What
* is provable here, and what the tests below assert, is the complementary
* half: that nothing is registered and no service is consulted. The exact
* observed values (trusted_keys=5, quorum=4, offset=-3, and the sync_state /
* stall-episode values) are asserted in MetricMacros.cpp, which is the file
* compiled when telemetry IS enabled.
* observed values (trusted_keys=5, quorum=4, offset=-3, the sync_state /
* stall-episode values, and the acquire-progress / cache-hit-rate values) are
* asserted in MetricMacros.cpp, which is the file compiled when telemetry IS
* enabled.
*/
// When telemetry is globally enabled, MetricsRegistry.cpp requires xrpld
@@ -378,9 +380,12 @@ TEST_F(MetricsRegistryTest, destructor_calls_stop)
//
// `unl_quorum` reads ValidatorList::trustedKeyCount() and quorum();
// `clock_close_offset_seconds` reads TimeKeeper::closeOffset(); `sync_state` and
// `server_stall_events_total` read NetworkOPs and LoadManager. All are
// `server_stall_events_total` read NetworkOPs and LoadManager; `sync_acquire`
// reads InboundLedgers::acquireProgress() and `shamap_cache_hit_rate` reads the
// node Family's tree-node cache. All are
// reached through the ServiceRegistry, and MockServiceRegistry::getValidators()
// / getTimeKeeper() / getOPs() / getLoadManager() THROW std::logic_error. So "no
// / getTimeKeeper() / getOPs() / getLoadManager() / getInboundLedgers() /
// getNodeFamily() THROW std::logic_error. So "no
// gauge callback ran" is directly observable here: had registerAsyncGauges() run
// and had a callback fired, one of those accessors would have thrown.
//
@@ -419,7 +424,8 @@ TEST_F(MetricsRegistryTest, disabled_lifecycle_never_consults_gauge_services)
// start() is where registerAsyncGauges() -- and with it
// registerUnlQuorumGauge() / registerClockSkewGauge() /
// registerSyncStateGauge() / registerStallEventsCounter() -- would run.
// registerSyncStateGauge() / registerStallEventsCounter() /
// registerSyncAcquireGauge() / registerCacheHitRateDetailGauge() -- would run.
EXPECT_NO_THROW(registry.start("http://localhost:4318/v1/metrics"));
// detachCallbacks() is the shutdown hook the real gauges honour. It must be
@@ -443,6 +449,11 @@ TEST_F(MetricsRegistryTest, disabled_lifecycle_never_consults_gauge_services)
// firing would have thrown above.
EXPECT_THROW(mockApp_.getOPs(), std::logic_error);
EXPECT_THROW(mockApp_.getLoadManager(), std::logic_error);
// The two services the WP-A3 acquire signals read: sync_acquire polls the
// in-flight acquire collection, shamap_cache_hit_rate polls the node
// Family's tree-node cache. Neither was consulted above.
EXPECT_THROW(mockApp_.getInboundLedgers(), std::logic_error);
EXPECT_THROW(mockApp_.getNodeFamily(), std::logic_error);
}
// Even asking for enabled=true registers no sync-diagnostics gauge on a
@@ -461,9 +472,10 @@ TEST_F(MetricsRegistryTest, enabled_flag_alone_registers_no_gauges_when_compiled
// Yet the whole lifecycle stays inert. If registerAsyncGauges() had run and
// registered registerUnlQuorumGauge()/registerClockSkewGauge()/
// registerSyncStateGauge()/registerStallEventsCounter(), a callback would
// reach getValidators()/getTimeKeeper()/getOPs()/getLoadManager() and throw
// std::logic_error.
// registerSyncStateGauge()/registerStallEventsCounter()/
// registerSyncAcquireGauge()/registerCacheHitRateDetailGauge(), a callback
// would reach getValidators()/getTimeKeeper()/getOPs()/getLoadManager()/
// getInboundLedgers()/getNodeFamily() and throw std::logic_error.
EXPECT_NO_THROW(enabledRequest.start("http://localhost:4318/v1/metrics"));
EXPECT_NO_THROW(enabledRequest.detachCallbacks());
EXPECT_NO_THROW(enabledRequest.stop());

91
src/xrpld/app/ledger/InboundLedger.h
View File

@@ -14,11 +14,13 @@
#include <xrpl/nodestore/Database.h>
#include <xrpl/shamap/SHAMap.h>
#include <xrpl/shamap/SHAMapAddNode.h>
#include <xrpl/shamap/SHAMapMissingNode.h>
#include <xrpl/shamap/SHAMapNodeID.h>
#include <xrpl/telemetry/SpanGuard.h>
#include <xrpl.pb.h>
#include <atomic>
#include <chrono>
#include <cstddef>
#include <cstdint>
@@ -122,6 +124,43 @@ public:
return lastAction_;
}
/**
* Outstanding missing SHAMap nodes in one of this acquire's two trees.
*
* Refreshed by trigger() after each getMissingNodes() sweep, which already
* computes the count as a byproduct of its walk — this accessor adds no
* traversal of its own and does not take the acquire lock.
*
* Read by the telemetry observable-gauge callback (~10 s cadence). A count
* that stays flat and non-zero across ticks means the acquire will never
* finish; a shrinking count means it is slow but alive.
*
* @param type Which tree to report: SHAMapType::TRANSACTION selects the
* transaction tree, every other value selects the account-state tree.
* @return Node count from the most recent sweep of that tree; 0 before the
* first sweep and after the tree completes.
*
* @note Thread-safe and lock-free: a relaxed atomic load. The reader
* tolerates a value one sweep out of date.
*/
[[nodiscard]] int
getMissingNodeCount(SHAMapType type) const noexcept;
/**
* Number of peer packets stashed in receivedData_ awaiting processing.
*
* A deep stash means node data is arriving faster than runData() can apply
* it, which is a processing bottleneck rather than a peer-supply one.
*
* @return Current stash depth; 0 when nothing is pending.
*
* @note Thread-safe and lock-free: a relaxed atomic load of a counter
* mirrored on every push/drain, so it never blocks the receive path
* nor waits on receivedDataLock_.
*/
[[nodiscard]] std::size_t
getReceivedDataDepth() const noexcept;
private:
enum class TriggerReason { Added, Reply, Timeout };
@@ -173,6 +212,37 @@ private:
std::vector<uint256>
neededStateHashes(int max, SHAMapSyncFilter const* filter) const;
/**
* Re-publish the missing-node counts from the completion flags.
*
* Called under mtx_ wherever a tree flips to complete. A tree that needs no
* more nodes must publish 0: otherwise the last sweep's count lingers and a
* finished acquire keeps reporting a flat non-zero, which is exactly the
* "permanently stuck" reading the gauge exists to detect. Idempotent, so it
* is safe to call from every flip site.
*/
void
refreshMissingNodeCounts() noexcept;
/**
* Fold one processed batch into the acquire totals and emit its telemetry.
*
* Both processData() branches (header batch and node batch) finished with
* the same three steps, so they share this one helper: mark progress, add to
* stats_, and emit the per-outcome add-node counters.
*
* @param san Outcome tally for the batch just processed.
* @return Number of good nodes in the batch, which is processData()'s
* "useful data from this peer" return value.
*
* @note Called once per received packet, after the per-node loop inside
* receiveNode() has completed. The tallies are already aggregated, so
* the counters are emitted once per batch and never per node.
* @note Call with mtx_ held, as both call sites already do.
*/
int
recordBatchOutcome(SHAMapAddNode const& san);
clock_type& clock_;
clock_type::time_point lastAction_;
@@ -196,6 +266,27 @@ private:
bool receiveDispatched_{false};
std::unique_ptr<PeerSet> peerSet_;
/**
* Outstanding missing nodes in the account-state tree, as counted by the
* last getMissingNodes() sweep in trigger(). Relaxed atomic: written by the
* acquiring thread, read by the telemetry gauge callback, and a value one
* sweep stale is acceptable for a ~10 s gauge.
*/
std::atomic<int> missingStateNodes_{0};
/**
* Outstanding missing nodes in the transaction tree. Same ownership and
* staleness contract as missingStateNodes_.
*/
std::atomic<int> missingTxNodes_{0};
/**
* Mirror of receivedData_.size(), maintained under receivedDataLock_ on
* every push and drain. Exists so the telemetry gauge callback can read the
* depth without contending for that lock on the node-receive path.
*/
std::atomic<std::size_t> receivedDataDepth_{0};
/**
* Spans the acquire lifecycle: started in init(), finalized in done()
* with the outcome (complete/failed), timeout count, and peer count.

50
src/xrpld/app/ledger/InboundLedgers.h
View File

@@ -91,6 +91,56 @@ public:
virtual std::size_t
cacheSize() = 0;
/**
* Aggregate acquire-progress snapshot across every in-flight acquire.
*
* Bounded, pre-aggregated telemetry: one value per field regardless of how
* many acquires are in flight, so the derived metric cannot grow a series
* per ledger. Per-ledger detail stays available on the `ledger.acquire`
* span, which is where unbounded identity belongs.
*/
struct AcquireProgress
{
/**
* Largest outstanding account-state node count of any in-flight
* acquire. The max, not the sum, so one stuck acquire stays visible
* instead of being averaged away by healthy ones.
*/
int maxMissingStateNodes{0};
/**
* Largest outstanding transaction-tree node count of any in-flight
* acquire.
*/
int maxMissingTxNodes{0};
/**
* Total unprocessed peer packets stashed across all in-flight acquires.
* Summed because it measures one shared processing backlog.
*/
std::size_t receivedDataDepth{0};
/**
* Number of acquires currently in flight, so the three values above can
* be read in context (all zero with zero acquires is idle, not healthy).
*/
std::size_t inFlight{0};
};
/**
* Collect the aggregate acquire-progress snapshot.
*
* Intended for the telemetry observable gauge, polled about every 10 s.
* Takes the collection lock only long enough to copy the handles, then reads
* each acquire's relaxed atomics without holding it, so it never blocks the
* node-receive path.
*
* @return Aggregated progress; an all-zero value with `inFlight == 0` when
* nothing is being acquired.
*/
[[nodiscard]] virtual AcquireProgress
acquireProgress() = 0;
};
std::unique_ptr<InboundLedgers>

117
src/xrpld/app/ledger/detail/InboundLedger.cpp
View File

@@ -10,6 +10,7 @@
#include <xrpld/overlay/Message.h>
#include <xrpld/overlay/Overlay.h>
#include <xrpld/overlay/PeerSet.h>
#include <xrpld/telemetry/MetricMacros.h>
#include <xrpl/basics/Blob.h>
#include <xrpl/basics/Log.h>
@@ -29,6 +30,7 @@
#include <xrpl/protocol/SystemParameters.h> // IWYU pragma: keep
#include <xrpl/protocol/jss.h>
#include <xrpl/resource/Fees.h>
#include <xrpl/shamap/SHAMapMissingNode.h>
#include <xrpl/shamap/SHAMapNodeID.h>
#include <xrpl/shamap/SHAMapSyncFilter.h>
#include <xrpl/telemetry/SpanGuard.h>
@@ -39,6 +41,7 @@
#include <xrpl.pb.h>
#include <algorithm>
#include <atomic>
#include <chrono>
#include <cstddef>
#include <cstdint>
@@ -133,6 +136,17 @@ InboundLedger::init(ScopedLockType& collectionLock)
if (failed_)
return;
// Whether the local node store already held the whole ledger. Emitted once
// per new acquire (init() runs exactly once), never per node, so the cost is
// a single labelled counter Add. This is what separates disk-bound sync
// ("everything was local, we are just slow to read it") from peer-bound sync
// ("nothing was local, every node must come over the wire").
XRPL_METRIC_COUNTER_INC_LABELED(
app_,
"sync_acquire_source_total",
"Ledger acquires by where the data came from",
{{"source", std::string(complete_ ? "local" : "network")}});
if (!complete_)
{
addPeers();
@@ -157,6 +171,28 @@ InboundLedger::init(ScopedLockType& collectionLock)
app_.getLedgerMaster().checkAccept(ledger_);
}
int
InboundLedger::getMissingNodeCount(SHAMapType type) const noexcept
{
return (type == SHAMapType::TRANSACTION ? missingTxNodes_ : missingStateNodes_)
.load(std::memory_order_relaxed);
}
std::size_t
InboundLedger::getReceivedDataDepth() const noexcept
{
return receivedDataDepth_.load(std::memory_order_relaxed);
}
void
InboundLedger::refreshMissingNodeCounts() noexcept
{
if (haveState_)
missingStateNodes_.store(0, std::memory_order_relaxed);
if (haveTransactions_)
missingTxNodes_.store(0, std::memory_order_relaxed);
}
std::size_t
InboundLedger::getPeerCount() const
{
@@ -359,6 +395,10 @@ InboundLedger::tryDB(NodeStore::Database& srcDB)
}
}
// A tree satisfied from the local store never ran a sweep, so publish its
// zero here rather than leaving the gauge on a stale count.
refreshMissingNodeCounts();
if (haveTransactions_ && haveState_)
{
JLOG(journal_.debug()) << "Had everything locally";
@@ -408,6 +448,16 @@ InboundLedger::onTimer(bool wasProgress, ScopedLockType&)
std::size_t const pc = getPeerCount();
JLOG(journal_.debug()) << "No progress(" << pc << ") for ledger " << hash_;
// A timeout with no node received since the previous one means this
// acquire is stalled. Fires on the acquire timer (once every 3 s at
// most), not on any per-node path, so one counter Add here is free.
// A climbing rate here alongside a flat missing-node count is the
// signature of a sync that will never complete.
XRPL_METRIC_COUNTER_INC(
app_,
"sync_acquire_no_progress_total",
"Ledger-acquire timeouts where no new node arrived");
// addPeers triggers if the reason is not HISTORY
// So if the reason IS HISTORY, need to trigger after we add
// otherwise, we need to trigger before we add
@@ -681,6 +731,12 @@ InboundLedger::trigger(std::shared_ptr<Peer> const& peer, TriggerReason reason)
auto nodes = ledger_->stateMap().getMissingNodes(kMissingNodesFind, &filter);
sl.lock();
// Publish the outstanding count for the telemetry gauge. The sweep
// above already produced it, so this is one relaxed atomic store per
// sweep and never per tree node -- getMissingNodes() walks thousands
// of nodes internally and must stay free of metric work.
missingStateNodes_.store(static_cast<int>(nodes.size()), std::memory_order_relaxed);
// Make sure nothing happened while we released the lock
if (!failed_ && !complete_ && !haveState_)
{
@@ -749,6 +805,10 @@ InboundLedger::trigger(std::shared_ptr<Peer> const& peer, TriggerReason reason)
auto nodes = ledger_->txMap().getMissingNodes(kMissingNodesFind, &filter);
// Same contract as the state-tree store above: one atomic store per
// sweep, outside the per-node walk.
missingTxNodes_.store(static_cast<int>(nodes.size()), std::memory_order_relaxed);
if (nodes.empty())
{
if (!ledger_->txMap().isValid())
@@ -874,6 +934,10 @@ InboundLedger::takeHeader(std::string const& data)
if (ledger_->header().accountHash.isZero())
haveState_ = true;
// An empty tree is complete on arrival of the header, with no sweep to
// publish its count.
refreshMissingNodeCounts();
ledger_->txMap().setSynching();
ledger_->stateMap().setSynching();
@@ -969,6 +1033,11 @@ InboundLedger::receiveNode(protocol::TMLedgerData const& packet, SHAMapAddNode&
haveState_ = true;
}
// The tree finished on this batch, so the last sweep's count is now
// stale. Publishing 0 here is what stops a completed acquire from
// reading as a permanently stuck one. Outside the per-node loop above.
refreshMissingNodeCounts();
if (haveTransactions_ && haveState_)
{
complete_ = true;
@@ -1077,6 +1146,8 @@ InboundLedger::gotData(
return false;
receivedData_.emplace_back(peer, data);
// Mirror the depth for the telemetry gauge, which must not take this lock.
receivedDataDepth_.store(receivedData_.size(), std::memory_order_relaxed);
if (receiveDispatched_)
return false;
@@ -1144,11 +1215,7 @@ InboundLedger::processData(std::shared_ptr<Peer> peer, protocol::TMLedgerData co
return -1;
}
if (san.isUseful())
progress_ = true;
stats_ += san;
return san.getGood();
return recordBatchOutcome(san);
}
if ((packet.type() == protocol::liTX_NODE) || (packet.type() == protocol::liAS_NODE))
@@ -1180,16 +1247,43 @@ InboundLedger::processData(std::shared_ptr<Peer> peer, protocol::TMLedgerData co
<< ((packet.type() == protocol::liTX_NODE) ? "TX" : "AS")
<< " node stats: " << san.get();
if (san.isUseful())
progress_ = true;
stats_ += san;
return san.getGood();
return recordBatchOutcome(san);
}
return -1;
}
int
InboundLedger::recordBatchOutcome(SHAMapAddNode const& san)
{
if (san.isUseful())
progress_ = true;
stats_ += san;
// Emit the tallies the trace log above already printed. receiveNode() walks
// every node in the packet, so these MUST stay out here: the loop has
// finished and the tallies are aggregated, giving at most three counter Adds
// per received packet rather than per node. The split is what separates real
// progress (good) from wasted bandwidth (duplicate) and a misbehaving peer
// (invalid) -- traffic-level metrics show all three as healthy throughput.
auto const emit = [this](char const* outcome, int count) {
if (count <= 0)
return;
XRPL_METRIC_COUNTER_ADD_LABELED(
app_,
"sync_addnode_total",
"SHAMap nodes received during ledger acquire, by outcome",
static_cast<std::uint64_t>(count),
{{"outcome", std::string(outcome)}});
};
emit("good", san.getGood());
emit("duplicate", san.getDuplicate());
emit("invalid", san.getBad());
return san.getGood();
}
namespace detail {
// Track the amount of useful data that each peer returns
struct PeerDataCounts
@@ -1290,10 +1384,13 @@ InboundLedger::runData()
if (receivedData_.empty())
{
receiveDispatched_ = false;
receivedDataDepth_.store(0, std::memory_order_relaxed);
break;
}
data.swap(receivedData_);
// The stash was just drained into `data`; keep the mirror in step.
receivedDataDepth_.store(receivedData_.size(), std::memory_order_relaxed);
}
for (auto& entry : data)

33
src/xrpld/app/ledger/detail/InboundLedgers.cpp
View File

@@ -24,10 +24,12 @@
#include <xrpl/protocol/Serializer.h>
#include <xrpl/protocol/jss.h>
#include <xrpl/server/NetworkOPs.h>
#include <xrpl/shamap/SHAMapMissingNode.h>
#include <xrpl/shamap/SHAMapTreeNode.h>
#include <xrpl.pb.h>
#include <algorithm>
#include <chrono>
#include <cstddef>
#include <cstdint>
@@ -435,6 +437,37 @@ public:
return ledgers_.size();
}
AcquireProgress
acquireProgress() override
{
// Copy the handles under the lock, then read each acquire's atomics
// without it -- same pattern gotFetchPack() uses, so the ~10 s telemetry
// poll never contends with the node-receive path.
std::vector<std::shared_ptr<InboundLedger>> acquires;
{
ScopedLockType const sl(lock_);
acquires.reserve(ledgers_.size());
for (auto const& it : ledgers_)
{
XRPL_ASSERT(
it.second, "xrpl::InboundLedgersImp::acquireProgress : non-null ledger");
acquires.push_back(it.second);
}
}
AcquireProgress out;
out.inFlight = acquires.size();
for (auto const& acquire : acquires)
{
out.maxMissingStateNodes =
std::max(out.maxMissingStateNodes, acquire->getMissingNodeCount(SHAMapType::STATE));
out.maxMissingTxNodes = std::max(
out.maxMissingTxNodes, acquire->getMissingNodeCount(SHAMapType::TRANSACTION));
out.receivedDataDepth += acquire->getReceivedDataDepth();
}
return out;
}
private:
clock_type& clock_;

84
src/xrpld/telemetry/MetricsRegistry.cpp
View File

@@ -488,6 +488,8 @@ MetricsRegistry::registerAsyncGauges()
registerClockSkewGauge();
registerSyncStateGauge();
registerStallEventsCounter();
registerSyncAcquireGauge();
registerCacheHitRateDetailGauge();
}
void
@@ -1637,6 +1639,88 @@ MetricsRegistry::registerStallEventsCounter()
this);
}
void
MetricsRegistry::registerSyncAcquireGauge()
{
// --- Sync diagnostics: is ledger acquisition actually progressing? ---
// Aggregated on purpose: a per-ledger label would add one series per ledger
// acquired, which is unbounded. The per-ledger view lives on the
// ledger.acquire span instead.
syncAcquireGauge_ = meter_->CreateInt64ObservableGauge(
"sync_acquire", "Aggregate ledger-acquire progress across in-flight acquires");
syncAcquireGauge_->AddCallback(
[](opentelemetry::metrics::ObserverResult result, void* state) {
auto* self = static_cast<MetricsRegistry*>(state);
if (self->callbacksDetached_.load(std::memory_order_acquire))
return;
auto& app = self->app_;
try
{
auto observe = [&](char const* name, int64_t value) {
opentelemetry::nostd::get<opentelemetry::nostd::shared_ptr<
opentelemetry::metrics::ObserverResultT<int64_t>>>(result)
->Observe(value, {{"metric", name}});
};
// One snapshot feeds all four series, so they are mutually
// consistent rather than read at four different instants.
auto const progress = app.getInboundLedgers().acquireProgress();
// Flat and non-zero across ticks = this acquire will never
// finish. Shrinking = slow but alive.
observe(
"missing_state_nodes_max", static_cast<int64_t>(progress.maxMissingStateNodes));
observe("missing_tx_nodes_max", static_cast<int64_t>(progress.maxMissingTxNodes));
// Deep stash = arriving data outpaces processing.
observe("received_data_depth", static_cast<int64_t>(progress.receivedDataDepth));
// Context for the three above: zero everywhere with zero
// in-flight acquires is idle, not healthy.
observe("in_flight", static_cast<int64_t>(progress.inFlight));
}
catch (...) // NOLINT(bugprone-empty-catch)
{
// Silently skip if services are not yet ready.
}
},
this);
}
void
MetricsRegistry::registerCacheHitRateDetailGauge()
{
// --- Sync diagnostics: SHAMap tree-node cache hit rate ---
// The memory layer above the node store: a miss here is what causes a
// node-store read, which the NuDB hit-ratio panel then measures.
shamapCacheHitRateGauge_ = meter_->CreateDoubleObservableGauge(
"shamap_cache_hit_rate", "SHAMap tree-node cache hit rate (0.0-1.0), by cache");
shamapCacheHitRateGauge_->AddCallback(
[](opentelemetry::metrics::ObserverResult result, void* state) {
auto* self = static_cast<MetricsRegistry*>(state);
if (self->callbacksDetached_.load(std::memory_order_acquire))
return;
auto& app = self->app_;
try
{
// TaggedCache::getHitRate() returns 0-100; normalize to 0.0-1.0
// so the panel can use Grafana's "percentunit" unit, matching
// how cache_metrics already reports its rates.
auto const rate = app.getNodeFamily().getTreeNodeCache()->getHitRate() / 100.0F;
opentelemetry::nostd::get<opentelemetry::nostd::shared_ptr<
opentelemetry::metrics::ObserverResultT<double>>>(result)
->Observe(static_cast<double>(rate), {{"metric", "treenode"}});
}
catch (...) // NOLINT(bugprone-empty-catch)
{
// Silently skip if services are not yet ready.
}
},
this);
}
#endif // XRPL_ENABLE_TELEMETRY
// -----------------------------------------------------------------

72
src/xrpld/telemetry/MetricsRegistry.h
View File

@@ -570,6 +570,18 @@ private:
*/
opentelemetry::nostd::shared_ptr<opentelemetry::metrics::ObservableInstrument>
stallEventsObservable_;
/**
* Observable gauge for aggregate ledger-acquire progress (max missing state
* and tx nodes, received-data stash depth, in-flight acquire count).
*/
opentelemetry::nostd::shared_ptr<opentelemetry::metrics::ObservableInstrument>
syncAcquireGauge_;
/**
* Observable gauge for the SHAMap tree-node cache hit rate, which is the
* memory layer above the node store's own hit ratio.
*/
opentelemetry::nostd::shared_ptr<opentelemetry::metrics::ObservableInstrument>
shamapCacheHitRateGauge_;
/**
* Observable gauge for build version info (label-based, value=1).
*/
@@ -832,7 +844,65 @@ private:
*/
void
registerStallEventsCounter(); // sync diagnostics: stall episode count
#endif // XRPL_ENABLE_TELEMETRY
/**
* Register the `sync_acquire` gauge.
*
* One instrument fanning out four series under the `metric` attribute, all
* from a single InboundLedgers::acquireProgress() snapshot:
*
* `missing_state_nodes_max` — largest outstanding account-state node count
* of any in-flight acquire. THE headline stuck-sync signal: flat and
* non-zero across ticks means the acquire will never finish, shrinking
* means it is slow but alive.
* `missing_tx_nodes_max` — the same for the transaction tree.
* `received_data_depth` — peer packets stashed across all acquires waiting
* to be applied. Deep means processing, not peer supply, is the limit.
* `in_flight` — how many acquires are running, so the three values above
* can be read in context: all zero with `in_flight` zero is idle, not
* healthy.
*
* Deliberately aggregated rather than per-ledger. A `ledger_seq` label would
* mint a new time series for every ledger the node ever acquires, which is
* unbounded cardinality; the max/sum keeps the "is it stuck?" answer while
* the per-ledger identity stays on the `ledger.acquire` span, where
* high-cardinality identity belongs.
*
* @note Pulled on the OTel reader thread (~10 s tick), never on a hot path.
* The snapshot takes the acquire-collection lock only to copy shared_ptrs,
* then reads relaxed atomics; the emit sites that feed those atomics all sit
* outside the per-tree-node loops.
*/
void
registerSyncAcquireGauge(); // sync diagnostics: acquire progress
/**
* Register the `shamap_cache_hit_rate` gauge.
*
* Observes one series, `treenode`, from TreeNodeCache::getHitRate(): the
* percentage of SHAMap tree-node lookups served from memory instead of the
* node store. During a fresh sync a low rate means the node re-reads the
* same subtrees from disk, so sync is disk-bound rather than peer-bound.
*
* Distinct from the `NuDB Cache Hit Ratio` panel on the ledger-data-sync
* dashboard: that one is derived from `nodestore_state` and measures the
* node-store layer (`node_reads_hit / node_reads_total`). This gauge
* measures the in-memory tree-node cache that sits ABOVE it, so a request
* missing here is what produces a node-store read there.
*
* The full-below cache is deliberately NOT reported. It is a KeyCache, whose
* only lookup path is TaggedCache::touchIfExists(), and that method
* increments `stats_.hits`/`stats_.misses` while `getHitRate()` reads the
* separate `hits_`/`misses_` members. Its hit rate is therefore hard-wired
* to 0 regardless of behaviour, so exporting it would ship a permanently
* empty panel; fixing that accounting belongs in a libxrpl change of its own.
*
* @note Pulled on the OTel reader thread (~10 s tick). Takes the cache's
* mutex for two integer reads and a divide; no hot-path cost.
*/
void
registerCacheHitRateDetailGauge(); // sync diagnostics: treenode cache
#endif // XRPL_ENABLE_TELEMETRY
};
} // namespace telemetry