From c7cacfb0e48acbae41f283f28fcfb271459086f6 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Mon, 27 Jul 2026 18:18:53 +0100 Subject: [PATCH] fix(telemetry): plot peer supply window as margins, not absolute sequences The Peer Ledger Supply Window panel drew supply_min_seq, supply_max_seq and nothing else on one linear axis. Measured on a mainnet node, those sit around 105,890,000 and roughly 300,000 apart, so the 588-ledger tip movement that shows whether sync is progressing was 0.0006% of the axis and read as a flat line. unit "none" also printed the sequences unabbreviated and clipped the legend. The panel's own "Watch for" text asked the reader to compare supply_min_seq against this node's validated sequence, but that line was not on the panel at all, so the comparison meant switching dashboards. Plot the two distances instead, which is what the panel was always asking about: History Headroom = validated_ledger_seq - supply_min_seq Tip Gap = supply_max_seq - validated_ledger_seq Zero is now the boundary in both directions: negative headroom is exactly the "every peer pruned what I still need" case the description warns about, and it becomes a zero crossing rather than a line-order comparison. Tip Gap gets the right-hand axis because the two ranges differ by orders of magnitude (measured: 299999..300001 against -1..1). Both operands are gated `> 0`. Ungated, differencing the documented "unknown window" sentinel of 0 yields the whole sequence space: measured -105854935 for headroom and 105890295 for tip gap during the first ticks, which destroys the axis for the rest of the window. Gated, the panel stays blank until the node has a validated ledger and a peer has advertised a range, which is the honest reading for that state. Both queries verified against a live mainnet node through the full template substitution: refId A = 300001 legend "History Headroom [xrpld-mainnet]", refId B = -1 legend "Tip Gap [xrpld-mainnet]". Runbook branch-C table, step 11 walkthrough and the 09 reference row follow the rename and the new reading. Co-Authored-By: Claude Opus 5 (1M context) --- .../09-data-collection-reference.md | 2 +- .../dashboards/ledger-sync-health.json | 48 +++++++++++++---- docs/telemetry-runbook.md | 51 ++++++++++--------- 3 files changed, 68 insertions(+), 33 deletions(-) diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md index 6ac7a598ae..f9e53c9a08 100644 --- a/OpenTelemetryPlan/09-data-collection-reference.md +++ b/OpenTelemetryPlan/09-data-collection-reference.md @@ -1698,7 +1698,7 @@ no panel (it is read in Tempo instead). | `sync_addnode_total` (`outcome` = `good` \| `duplicate` \| `invalid`) | counter | `InboundLedger.cpp` — `InboundLedger::recordBatchOutcome` | Add-Node Outcomes | SHAMap nodes received during acquire, split by result. Emitted once per received packet from the aggregated batch tally the trace log already printed — never inside the per-node `receiveNode()` loop. Separates real progress (`good`) from wasted bandwidth (`duplicate`) and a misbehaving peer (`invalid`), all three of which look like healthy throughput in traffic metrics. | | `sync_acquire_source_total` (`source` = `local` \| `network`) | counter | `InboundLedger.cpp` — `InboundLedger::init` | Acquire Source (local vs network) | Whether an acquire was satisfied entirely from the local node store or needed peers, emitted once per new acquire after the first local lookup. Sustained `network` on a node that should already hold the range means sync is disk-bound rather than peer-bound. | | `jobq_saturation` (`metric` = `running_tasks` \| `worker_threads` \| `total_waiting`) | observable gauge | `MetricsRegistry.cpp` — `registerJobQueueSaturationGauge` | Worker Pool Saturation; Worker Pool Capacity & Total Backlog | Global worker-pool saturation from `JobQueue::getWorkerSaturation()`: tasks in flight, threads the pool is configured to run, and jobs queued across all types, all from one reading so the ratio and the backlog describe the same instant. `worker_threads` is exported rather than hardcoded in the dashboard because it is derived at startup from `[workers]`, node size and hardware concurrency. Exists separately from the per-job-type gauges `JobQueue::collect()` publishes (`jobq__waiting` / `_running` / `_deferred`) because a pool-wide slowdown otherwise appears as an independent fault in every subsystem queued behind it; a `running_tasks / worker_threads` ratio at 1.0 **with** a non-zero `total_waiting` attributes it to pool exhaustion once. | -| `peer_ledger_supply` (`metric` = `peers_reporting` \| `peers_serving_validated` \| `peers_serving_next` \| `supply_min_seq` \| `supply_max_seq`) | observable gauge | `MetricsRegistry.cpp` — `registerPeerLedgerSupplyGauge` (aggregating `OverlayImpl::getPeerLedgerSupply`) | Peers Able to Serve Needed Sequence; Peer Ledger Supply Window | How much of the sequence range this node needs its connected peer set can actually serve, from one pass over the active peers reading the range each already advertised in `mtSTATUS_CHANGE`. **`peers_serving_next` is the signal this exists for:** zero there with a non-zero `peers_reporting` means no connected peer holds validated + 1, so the peer set must change and waiting cannot finish the sync. `peers_reporting` is the denominator that makes the rest readable — peers advertising `[0, 0]` have not reported yet and are excluded from every field, so they cannot make a healthy peer set appear to serve from genesis; when nothing has reported, both window fields read 0 meaning **unknown**, not genesis. `supply_min_seq` / `supply_max_seq` separate "asking for history nobody kept" from "asking for a tip nobody reached". Distinct from `server_info{metric="peers"}`, a bare connection count with no notion of what those peers hold; from `sync_state{metric="ledgers_behind"}`, which uses the same per-peer maxima but collapses them to a single distance-to-tip number that cannot say how many peers can serve that distance or whether the range has a hole; and from `peer_quality{metric="peers_insane_count"}`, which counts peers on a different chain and is therefore a correctness signal, not an availability one. | +| `peer_ledger_supply` (`metric` = `peers_reporting` \| `peers_serving_validated` \| `peers_serving_next` \| `supply_min_seq` \| `supply_max_seq`) | observable gauge | `MetricsRegistry.cpp` — `registerPeerLedgerSupplyGauge` (aggregating `OverlayImpl::getPeerLedgerSupply`) | Peers Able to Serve Needed Sequence; Peer Supply Window Margin (history headroom vs tip gap) | How much of the sequence range this node needs its connected peer set can actually serve, from one pass over the active peers reading the range each already advertised in `mtSTATUS_CHANGE`. **`peers_serving_next` is the signal this exists for:** zero there with a non-zero `peers_reporting` means no connected peer holds validated + 1, so the peer set must change and waiting cannot finish the sync. `peers_reporting` is the denominator that makes the rest readable — peers advertising `[0, 0]` have not reported yet and are excluded from every field, so they cannot make a healthy peer set appear to serve from genesis; when nothing has reported, both window fields read 0 meaning **unknown**, not genesis. `supply_min_seq` / `supply_max_seq` separate "asking for history nobody kept" from "asking for a tip nobody reached". The _Peer Supply Window Margin_ panel renders both as distances from `server_info{metric="validated_ledger_seq"}` rather than as absolute sequences, because the raw values sit around 1.05e8 and roughly 3e5 apart, so one linear axis flattens the tip movement that shows whether sync is progressing; the subtraction also makes zero the boundary in both directions. Both operands are gated `> 0` in PromQL so the unknown-window sentinel cannot turn into a whole-sequence-space spike when subtracted. Distinct from `server_info{metric="peers"}`, a bare connection count with no notion of what those peers hold; from `sync_state{metric="ledgers_behind"}`, which uses the same per-peer maxima but collapses them to a single distance-to-tip number that cannot say how many peers can serve that distance or whether the range has a hole; and from `peer_quality{metric="peers_insane_count"}`, which counts peers on a different chain and is therefore a correctness signal, not an availability one. | | `peer_disconnect_total` (`reason` = `graceful` \| `shutdown` \| `stopping` \| `read_error` \| `write_error` \| `timer_error` \| `ping_timeout` \| `not_useful` \| `large_sendq` \| `charge_resources` \| `malformed_handshake` \| `shared_value` \| `unknown`; `direction` = `inbound` \| `outbound`) | counter | `PeerImp.cpp` — `PeerImp::close` | Peer Disconnects by Reason | Peer teardowns split by cause and by which side opened the connection. Emitted once per teardown at `close()`, the single funnel every disconnect path passes through, and `close()` already self-guards on the socket being open, so a repeated close cannot double-count and the total matches the existing unlabelled tally. `reason` is set by whichever site decided to disconnect, first writer wins, so a later generic reason never masks the real one; the value is always one of a fixed set of literals in `PeerImp.cpp`, never peer-supplied data, so cardinality is bounded by the code. The split is the whole point: it separates our-fault backpressure (`large_sendq`, `charge_resources`) from topology and network faults (`not_useful`, `ping_timeout`, `read_error`), and normal churn (`graceful`) from either. Distinct from the existing `server_info{metric="peer_disconnects_resources"}`, which counts only the resource-charge subset and carries no labels, and from the StatsD `overlay_peer_disconnects`, which is the unlabelled grand total in which every reason above collapses into one number. | | `peer_accept_total` (`outcome` = `accepted` \| `local_endpoint_fail` \| `resource_limit` \| `no_slot` \| `not_peer_request` \| `protocol_mismatch` \| `bad_cookie` \| `slot_refused` \| `handshake_error`) | counter | `OverlayImpl.cpp` — `OverlayImpl::onHandoff` via `reportAcceptOutcome` | Inbound Peer Accept Outcomes | Terminal outcome of every inbound connection this node is offered, one emit per handoff. `accepted` is reported only after `run()`, so anything that threw on the way lands on `handshake_error` instead; the two early returns that are not peer attempts at all (a handled HTTP request, and a request that never asked to upgrade) are deliberately not counted. The `outcome` names the stage that refused: no local endpoint, the resource manager, PeerFinder having no slot or seeing a duplicate, a non-peer upgrade request, protocol version disagreement, a bad security cookie, or activation being refused. This is the **inbound twin** of the existing `overlay_connect_total{outcome}`, which covers outbound dials only; without it a node refusing every inbound connection is indistinguishable from one nobody dials, and reading the two together gives the full in/out split. | | `peerfinder_slot_census` (`metric` = `out_active` \| `out_max` \| `in_active` \| `in_max` \| `connecting` \| `fixed_configured` \| `fixed_active` \| `bootcache` \| `livecache`) | observable gauge | `MetricsRegistry.cpp` — `registerSlotCensusGauge` (from `Logic::getSlotCensus`) | PeerFinder Slot Census; PeerFinder Address Caches & Fixed Peers | Slot occupancy against capacity, outbound dials in flight, configured-versus-connected fixed peers, and the depth of both address caches. All nine come from a single acquire of the PeerFinder lock, so they are mutually consistent, share one label set and can be compared against each other. That is what makes the three most common bootstrap failures visible: `connecting` non-zero while `out_active` stays below `out_max` (dials starting and never completing), `bootcache` and `livecache` both at 0 (nothing to dial at all), and `fixed_active` below `fixed_configured` (a peer named in the configuration is unreachable). `fixed_configured` is the count of peers named in the config, so the pair an operator reads is "how many did I ask for" against "how many do I have" — the same comparison `autoconnect()` makes. All nine values already existed inside PeerFinder; only two of them were exported, as the legacy beast::insight gauges `peer_finder_active_inbound_peers` and `peer_finder_active_outbound_peers`. Those two carry no capacity, attempt or cache term, are read at unrelated instants, and so cannot be joined with each other let alone with a capacity term — leaving all three failures above indistinguishable from a node that is simply not dialling. | diff --git a/docker/telemetry/grafana/dashboards/ledger-sync-health.json b/docker/telemetry/grafana/dashboards/ledger-sync-health.json index 56c6cc587f..1db7c67921 100644 --- a/docker/telemetry/grafana/dashboards/ledger-sync-health.json +++ b/docker/telemetry/grafana/dashboards/ledger-sync-health.json @@ -1080,7 +1080,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*The lowest and highest ledger sequence any connected peer offers \u2014 the sequence window the peer set can actually serve from.*\n\n###### How it's computed:\n*peer_ledger_supply series supply_min_seq and supply_max_seq: the minimum and maximum over the ledger ranges advertised by all connected peers.*\n\n###### Reading it:\n*supply_max_seq is the network tip as this node's peers report it. supply_min_seq is the oldest history reachable from the current peer set; anything below it cannot be back-filled from these peers at all.*\n\n###### Healthy range:\n*supply_max_seq advancing with the network, supply_min_seq well below this node's own validated sequence.*\n\n###### Watch for:\n*supply_min_seq sitting above this node's validated sequence \u2014 every connected peer has already pruned the history this node still needs, so the gap can never be closed from this peer set. Pair with Peers Able to Serve Needed Sequence: that panel says how many peers can serve the next ledger, this one says which sequences are on offer at all.*\n\n###### Keywords:\n- **Peer ledger supply window** *(per node)* \u2014 the lowest and highest ledger sequence offered by any connected peer; the span the peer set can serve.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerPeerLedgerSupplyGauge`\n\n###### References:\n[Complete ledger ranges](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#complete-ledger-ranges) \u00b7 [Back-fill / catch-up](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#back-fill-catch-up)", + "description": "###### What this is:\n*Two distances that say whether the sequence this node needs is inside the window its peers can serve: how much history is still reachable behind it, and how far the network tip is ahead of it.*\n\n###### How it's computed:\n*History Headroom = this node's validated_ledger_seq minus peer_ledger_supply supply_min_seq. Tip Gap = supply_max_seq minus validated_ledger_seq. Both operands are gated on being above zero, so neither line is drawn until the node has a validated ledger and at least one peer has advertised a range.*\n\n###### Reading it:\n*Plotted as distances rather than absolute sequences on purpose. The raw sequences sit around 105,890,000 and roughly 300,000 apart, so on one linear axis the tip movement that shows whether sync is progressing is a fraction of a percent of the scale and reads as a flat line. Differencing puts the meaning on the axis: zero is the boundary in both cases. History Headroom uses the left axis, Tip Gap the right, because the two differ by orders of magnitude.*\n\n###### Healthy range:\n*History Headroom comfortably positive and roughly steady. Tip Gap at or near zero.*\n\n###### Watch for:\n*History Headroom crossing below zero — every connected peer has pruned the history this node still needs, so the gap can never be closed from this peer set and the node needs different peers. Tip Gap growing steadily means the network is closing ledgers faster than this node validates them. Pair with Peers Able to Serve Needed Sequence: that panel counts how many peers can serve the next ledger, this one says how far outside the served window the node has drifted.*\n\n###### Keywords:\n- **History headroom** *(per node)* — validated sequence minus the lowest sequence any connected peer offers; how much of the history behind this node is still reachable.\n- **Tip gap** *(per node)* — the highest sequence any connected peer offers minus this node's validated sequence; how far behind the peer-reported tip this node is.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query subtracts the two series and aggregates them.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerPeerLedgerSupplyGauge`\n\n###### References:\n[Complete ledger ranges](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#complete-ledger-ranges) · [Back-fill / catch-up](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#back-fill-catch-up)", "fieldConfig": { "defaults": { "color": { @@ -1088,9 +1088,9 @@ }, "custom": { "axisBorderShow": false, - "axisCenteredZero": false, + "axisCenteredZero": true, "axisColorMode": "text", - "axisLabel": "Ledger Sequence", + "axisLabel": "Ledgers", "axisPlacement": "auto", "barAlignment": 0, "barWidthFactor": 0.6, @@ -1117,7 +1117,7 @@ "mode": "none" }, "thresholdsStyle": { - "mode": "off" + "mode": "line" } }, "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", @@ -1125,13 +1125,35 @@ "mode": "absolute", "steps": [ { - "color": "green", + "color": "red", "value": null + }, + { + "color": "green", + "value": 0 } ] }, - "unit": "none" - } + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "Tip Gap.*" + }, + "properties": [ + { + "id": "custom.axisPlacement", + "value": "right" + }, + { + "id": "custom.axisLabel", + "value": "Ledgers behind tip" + } + ] + } + ] }, "gridPos": { "h": 12, @@ -1167,11 +1189,19 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(peer_ledger_supply{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=~\"supply_min_seq|supply_max_seq\"}, \"series\", \"$1\", \"metric\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace((server_info{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"validated_ledger_seq\"} > 0) - ignoring(metric) (peer_ledger_supply{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"supply_min_seq\"} > 0), \"series\", \"History Headroom\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "${DS_PROMETHEUS}" + }, + "expr": "label_replace(label_join(label_replace((peer_ledger_supply{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"supply_max_seq\"} > 0) - ignoring(metric) (server_info{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"validated_ledger_seq\"} > 0), \"series\", \"Tip Gap\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "refId": "B" } ], - "title": "Peer Ledger Supply Window", + "title": "Peer Supply Window Margin (history headroom vs tip gap)", "type": "timeseries" }, { diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index c7ad89bb3d..a68f8e8081 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -2453,21 +2453,21 @@ _Time to First Validated Ledger_ stays flat at zero. Ledger acquires are in flight, _Ledgers Behind Network_ is flat or rising, and `full` never arrives. -| Look at | Healthy | Unhealthy | Conclude | -| ------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| _Missing SHAMap Nodes per Acquire (state/tx)_ | falling toward zero (read the trend over minutes) | **flat and non-zero** | **no peer is serving that tree** — the node will sit here forever. Pinned at 256 is the per-sweep cap, meaningful only with the trend | -| _Acquire Stall Rate (no progress)_ | flat | sustained rate **together with** a flat missing-node count | the definitive stuck-sync signature: requesting and nobody answering | -| _Peers Able to Serve Needed Sequence_ | `peers_serving_next` above zero | `peers_serving_next` = 0 while `peers_reporting` > 0 | **decisive**: peers are connected and none holds the next needed ledger. Waiting cannot finish it — the peer set must change. Everything else in branch C will look starved as a consequence, so do not chase it | -| _Peer Ledger Supply Window_ | needed sequence inside `[supply_min_seq, supply_max_seq]` | needed below `supply_min_seq` | asking for history nobody kept — needs a full-history peer | -| | | needed above `supply_max_seq` | the peer set lags the real network; not a history problem | -| _Ledger Acquire Phase Outcomes (by phase & timeout)_ + _Ledger Acquire Phase Duration (p95 by phase)_ (row 9) | `header` short, `astree` the bulk | `astree` hot with `timed_out=true` and non-zero `missing_nodes` | the common stuck shape — peers are not supplying account-state nodes | -| | | `header` hot | the node is waiting to be **told what to fetch**; invisible in the missing-node counts, which are both still zero | -| _Add-Node Outcomes_ | `good` dominates | `duplicate` swamps `good` | bandwidth busy, acquire standing still — peers re-sending known data | -| | | `invalid` rising | a specific misbehaving peer, not a local fault | -| _Received-Data Stash Depth & In-Flight Acquires_ | stash drains | stash growing | data arrives faster than it is applied — a job-queue or disk problem, the **opposite** conclusion from a stall rate, and only this panel separates them | -| `jobq__deferred` | flat at 0 | sustained non-zero on `ledgerdata`/`ledgerrequest` | a job the queue accepted then **withheld** at its concurrency limit of 3 — it appears in neither `waiting` nor `running`, so no other signal can show it. Starved `ledgerdata` is exactly why the stash grows while missing nodes stay flat | -| _Worker Pool Saturation_ + _Worker Pool Capacity & Total Backlog_ | under 80% | 100% with `total_waiting` climbing | the pool is **exhausted** — every stage looks slow at once. Stop here; no per-subsystem fix helps while no thread is free | -| Acquire outcome `abandoned` in Tempo (`{name="ledger.acquire" && span.outcome="abandoned"}`) | absent | present | the acquire was swept or shut down before reaching a result — without this value a stuck-then-swept fetch had no `outcome` at all and vanished from every outcome rate | +| Look at | Healthy | Unhealthy | Conclude | +| ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| _Missing SHAMap Nodes per Acquire (state/tx)_ | falling toward zero (read the trend over minutes) | **flat and non-zero** | **no peer is serving that tree** — the node will sit here forever. Pinned at 256 is the per-sweep cap, meaningful only with the trend | +| _Acquire Stall Rate (no progress)_ | flat | sustained rate **together with** a flat missing-node count | the definitive stuck-sync signature: requesting and nobody answering | +| _Peers Able to Serve Needed Sequence_ | `peers_serving_next` above zero | `peers_serving_next` = 0 while `peers_reporting` > 0 | **decisive**: peers are connected and none holds the next needed ledger. Waiting cannot finish it — the peer set must change. Everything else in branch C will look starved as a consequence, so do not chase it | +| _Peer Supply Window Margin (history headroom vs tip gap)_ | _History Headroom_ positive, _Tip Gap_ near zero | _History Headroom_ **below zero** | asking for history nobody kept — needs a full-history peer | +| | | _Tip Gap_ growing steadily | the peer set lags the real network; not a history problem | +| _Ledger Acquire Phase Outcomes (by phase & timeout)_ + _Ledger Acquire Phase Duration (p95 by phase)_ (row 9) | `header` short, `astree` the bulk | `astree` hot with `timed_out=true` and non-zero `missing_nodes` | the common stuck shape — peers are not supplying account-state nodes | +| | | `header` hot | the node is waiting to be **told what to fetch**; invisible in the missing-node counts, which are both still zero | +| _Add-Node Outcomes_ | `good` dominates | `duplicate` swamps `good` | bandwidth busy, acquire standing still — peers re-sending known data | +| | | `invalid` rising | a specific misbehaving peer, not a local fault | +| _Received-Data Stash Depth & In-Flight Acquires_ | stash drains | stash growing | data arrives faster than it is applied — a job-queue or disk problem, the **opposite** conclusion from a stall rate, and only this panel separates them | +| `jobq__deferred` | flat at 0 | sustained non-zero on `ledgerdata`/`ledgerrequest` | a job the queue accepted then **withheld** at its concurrency limit of 3 — it appears in neither `waiting` nor `running`, so no other signal can show it. Starved `ledgerdata` is exactly why the stash grows while missing nodes stay flat | +| _Worker Pool Saturation_ + _Worker Pool Capacity & Total Backlog_ | under 80% | 100% with `total_waiting` climbing | the pool is **exhausted** — every stage looks slow at once. Stop here; no per-subsystem fix helps while no thread is free | +| Acquire outcome `abandoned` in Tempo (`{name="ledger.acquire" && span.outcome="abandoned"}`) | absent | present | the acquire was swept or shut down before reaching a result — without this value a stuck-then-swept fetch had no `outcome` at all and vanished from every outcome rate | **Conclusion:** distinguish "nobody is serving it" (peer supply) from "it arrives and we cannot process it" (job queue / disk). The two look identical in a log and @@ -2860,14 +2860,19 @@ panel it reads. **unknown**, not genesis — do not read a zero window here as "peers serve from the start of history". Go back to the Bootstrap row, and to step 12 for why there are no peers. - Then read _Peer Ledger Supply Window_ (`peer_ledger_supply`, - `metric=supply_min_seq` and `supply_max_seq`) against the sequence the - node wants. This is what tells the two shapes of a supply gap apart: a - needed sequence **below** `supply_min_seq` means the node is asking for - history nobody kept, so it needs a full-history peer; a needed sequence - **above** `supply_max_seq` means it is asking for a tip nobody has - reached, which is a peer set lagging the real network rather than a - history problem. + Then read _Peer Supply Window Margin (history headroom vs tip gap)_, + which plots the two window fields as distances from this node's own + validated sequence rather than as absolute sequences: _History Headroom_ + is `validated_ledger_seq − supply_min_seq` and _Tip Gap_ is + `supply_max_seq − validated_ledger_seq`. This is what tells the two + shapes of a supply gap apart, and zero is the boundary in both cases: + _History Headroom_ **below zero** means the node is asking for history + nobody kept, so it needs a full-history peer; a _Tip Gap_ that grows + steadily means it is chasing a tip its peers have not reached, which is a + peer set lagging the real network rather than a history problem. Both + lines stay blank until the node has a validated ledger and a peer has + advertised a range, so an empty panel here is the `peers_reporting` = 0 + case above, not a healthy reading. 12. **Is this node failing to get or keep peers, and why?** Step 11 says whether the peer set can serve; this step says why the peer