ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-22 14:50:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(telemetry): plot peer supply window as margins, not absolute sequences

Browse Source 
The Peer Ledger Supply Window panel drew supply_min_seq, supply_max_seq and
nothing else on one linear axis. Measured on a mainnet node, those sit around
105,890,000 and roughly 300,000 apart, so the 588-ledger tip movement that
shows whether sync is progressing was 0.0006% of the axis and read as a flat
line. unit "none" also printed the sequences unabbreviated and clipped the
legend.

The panel's own "Watch for" text asked the reader to compare supply_min_seq
against this node's validated sequence, but that line was not on the panel at
all, so the comparison meant switching dashboards.

Plot the two distances instead, which is what the panel was always asking
about:

  History Headroom = validated_ledger_seq - supply_min_seq
  Tip Gap          = supply_max_seq - validated_ledger_seq

Zero is now the boundary in both directions: negative headroom is exactly the
"every peer pruned what I still need" case the description warns about, and it
becomes a zero crossing rather than a line-order comparison. Tip Gap gets the
right-hand axis because the two ranges differ by orders of magnitude
(measured: 299999..300001 against -1..1).

Both operands are gated `> 0`. Ungated, differencing the documented
"unknown window" sentinel of 0 yields the whole sequence space: measured
-105854935 for headroom and 105890295 for tip gap during the first ticks,
which destroys the axis for the rest of the window. Gated, the panel stays
blank until the node has a validated ledger and a peer has advertised a
range, which is the honest reading for that state.

Both queries verified against a live mainnet node through the full template
substitution: refId A = 300001 legend "History Headroom [xrpld-mainnet]",
refId B = -1 legend "Tip Gap [xrpld-mainnet]".

Runbook branch-C table, step 11 walkthrough and the 09 reference row follow
the rename and the new reading.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pratik Mankawde
2026-07-27 18:18:53 +01:00
parent 8fe6645825
commit c7cacfb0e4
3 changed files with 68 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files

51
docs/telemetry-runbook.md
View File

@@ -2453,21 +2453,21 @@ _Time to First Validated Ledger_ stays flat at zero.
Ledger acquires are in flight, _Ledgers Behind Network_ is flat or rising, and
`full` never arrives.
| Look at | Healthy | Unhealthy | Conclude |
| ------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| _Missing SHAMap Nodes per Acquire (state/tx)_ | falling toward zero (read the trend over minutes) | **flat and non-zero** | **no peer is serving that tree** — the node will sit here forever. Pinned at 256 is the per-sweep cap, meaningful only with the trend |
| _Acquire Stall Rate (no progress)_ | flat | sustained rate **together with** a flat missing-node count | the definitive stuck-sync signature: requesting and nobody answering |
| _Peers Able to Serve Needed Sequence_ | `peers_serving_next` above zero | `peers_serving_next` = 0 while `peers_reporting` > 0 | **decisive**: peers are connected and none holds the next needed ledger. Waiting cannot finish it — the peer set must change. Everything else in branch C will look starved as a consequence, so do not chase it |
| _Peer Ledger Supply Window_ | needed sequence inside `[supply_min_seq, supply_max_seq]` | needed below `supply_min_seq` | asking for history nobody kept — needs a full-history peer |
| | | needed above `supply_max_seq` | the peer set lags the real network; not a history problem |
| _Ledger Acquire Phase Outcomes (by phase & timeout)_ + _Ledger Acquire Phase Duration (p95 by phase)_ (row 9) | `header` short, `astree` the bulk | `astree` hot with `timed_out=true` and non-zero `missing_nodes` | the common stuck shape — peers are not supplying account-state nodes |
| | | `header` hot | the node is waiting to be **told what to fetch**; invisible in the missing-node counts, which are both still zero |
| _Add-Node Outcomes_ | `good` dominates | `duplicate` swamps `good` | bandwidth busy, acquire standing still — peers re-sending known data |
| | | `invalid` rising | a specific misbehaving peer, not a local fault |
| _Received-Data Stash Depth & In-Flight Acquires_ | stash drains | stash growing | data arrives faster than it is applied — a job-queue or disk problem, the **opposite** conclusion from a stall rate, and only this panel separates them |
| `jobq_<jobtype>_deferred` | flat at 0 | sustained non-zero on `ledgerdata`/`ledgerrequest` | a job the queue accepted then **withheld** at its concurrency limit of 3 — it appears in neither `waiting` nor `running`, so no other signal can show it. Starved `ledgerdata` is exactly why the stash grows while missing nodes stay flat |
| _Worker Pool Saturation_ + _Worker Pool Capacity & Total Backlog_ | under 80% | 100% with `total_waiting` climbing | the pool is **exhausted** — every stage looks slow at once. Stop here; no per-subsystem fix helps while no thread is free |
| Acquire outcome `abandoned` in Tempo (`{name="ledger.acquire" && span.outcome="abandoned"}`) | absent | present | the acquire was swept or shut down before reaching a result — without this value a stuck-then-swept fetch had no `outcome` at all and vanished from every outcome rate |
| Look at | Healthy | Unhealthy | Conclude |
| ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| _Missing SHAMap Nodes per Acquire (state/tx)_ | falling toward zero (read the trend over minutes) | **flat and non-zero** | **no peer is serving that tree** — the node will sit here forever. Pinned at 256 is the per-sweep cap, meaningful only with the trend |
| _Acquire Stall Rate (no progress)_ | flat | sustained rate **together with** a flat missing-node count | the definitive stuck-sync signature: requesting and nobody answering |
| _Peers Able to Serve Needed Sequence_ | `peers_serving_next` above zero | `peers_serving_next` = 0 while `peers_reporting` > 0 | **decisive**: peers are connected and none holds the next needed ledger. Waiting cannot finish it — the peer set must change. Everything else in branch C will look starved as a consequence, so do not chase it |
| _Peer Supply Window Margin (history headroom vs tip gap)_ | _History Headroom_ positive, _Tip Gap_ near zero | _History Headroom_ **below zero** | asking for history nobody kept — needs a full-history peer |
| | | _Tip Gap_ growing steadily | the peer set lags the real network; not a history problem |
| _Ledger Acquire Phase Outcomes (by phase & timeout)_ + _Ledger Acquire Phase Duration (p95 by phase)_ (row 9) | `header` short, `astree` the bulk | `astree` hot with `timed_out=true` and non-zero `missing_nodes` | the common stuck shape — peers are not supplying account-state nodes |
| | | `header` hot | the node is waiting to be **told what to fetch**; invisible in the missing-node counts, which are both still zero |
| _Add-Node Outcomes_ | `good` dominates | `duplicate` swamps `good` | bandwidth busy, acquire standing still — peers re-sending known data |
| | | `invalid` rising | a specific misbehaving peer, not a local fault |
| _Received-Data Stash Depth & In-Flight Acquires_ | stash drains | stash growing | data arrives faster than it is applied — a job-queue or disk problem, the **opposite** conclusion from a stall rate, and only this panel separates them |
| `jobq_<jobtype>_deferred` | flat at 0 | sustained non-zero on `ledgerdata`/`ledgerrequest` | a job the queue accepted then **withheld** at its concurrency limit of 3 — it appears in neither `waiting` nor `running`, so no other signal can show it. Starved `ledgerdata` is exactly why the stash grows while missing nodes stay flat |
| _Worker Pool Saturation_ + _Worker Pool Capacity & Total Backlog_ | under 80% | 100% with `total_waiting` climbing | the pool is **exhausted** — every stage looks slow at once. Stop here; no per-subsystem fix helps while no thread is free |
| Acquire outcome `abandoned` in Tempo (`{name="ledger.acquire" && span.outcome="abandoned"}`) | absent | present | the acquire was swept or shut down before reaching a result — without this value a stuck-then-swept fetch had no `outcome` at all and vanished from every outcome rate |
**Conclusion:** distinguish "nobody is serving it" (peer supply) from "it arrives
and we cannot process it" (job queue / disk). The two look identical in a log and
@@ -2860,14 +2860,19 @@ panel it reads.
**unknown**, not genesis — do not read a zero window here as "peers
serve from the start of history". Go back to the Bootstrap row, and to
step 12 for why there are no peers.
Then read _Peer Ledger Supply Window_ (`peer_ledger_supply`,
`metric=supply_min_seq` and `supply_max_seq`) against the sequence the
node wants. This is what tells the two shapes of a supply gap apart: a
needed sequence **below** `supply_min_seq` means the node is asking for
history nobody kept, so it needs a full-history peer; a needed sequence
**above** `supply_max_seq` means it is asking for a tip nobody has
reached, which is a peer set lagging the real network rather than a
history problem.
Then read _Peer Supply Window Margin (history headroom vs tip gap)_,
which plots the two window fields as distances from this node's own
validated sequence rather than as absolute sequences: _History Headroom_
is `validated_ledger_seq − supply_min_seq` and _Tip Gap_ is
`supply_max_seq − validated_ledger_seq`. This is what tells the two
shapes of a supply gap apart, and zero is the boundary in both cases:
_History Headroom_ **below zero** means the node is asking for history
nobody kept, so it needs a full-history peer; a _Tip Gap_ that grows
steadily means it is chasing a tip its peers have not reached, which is a
peer set lagging the real network rather than a history problem. Both
lines stay blank until the node has a validated ledger and a peer has
advertised a range, so an empty panel here is the `peers_reporting` = 0
case above, not a healthy reading.
12. **Is this node failing to get or keep peers, and why?**
Step 11 says whether the peer set can serve; this step says why the peer