ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-21 22:30:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(telemetry): expose the sweep-trim and rotation costs (WP-B5)

Browse Source 
Two suspects from the 3.3.0 slowdown investigation had no signal. Both were
already computing the numbers and throwing them away, so this exposes them
rather than adding measurement.

Per-sweep heap trim. The trim runs after every cache sweep, and its cost
scales with resident heap, so it is the leading explanation for a node with
a populated database syncing slower than a fresh one. The report already
carried duration, fault deltas and reclaimed pages, but the whole
measurement sat behind a debug-journal check, so an ordinary node measured
nothing, and the call site discarded the result. The measurement now always
runs and only the log line stays gated. Records trim duration, minor faults
and reclaimed kilobytes. Measured cost of the always-on path is about six
microseconds per sweep against a trim costing milliseconds, at a cadence of
ten to a hundred and twenty seconds.

Honest limit, stated in the runbook: the fault delta spans only the trim
call, so it shows the trim itself faulting but not the faults that follow as
caches refill. The duration is the signal to correlate against sweep-job
queueing.

Rotation writes. Rotation copies archive-served reads forward and re-stores
nodes missing from both backends, both of which compete with sync I/O and
only happen on a populated online_delete database. The copy-forward count
existed but was reset by the rotation's own log line, so a metric reading it
would drop to zero on every swap; a never-reset total sits beside it now.
The re-store count was not measured at all. Rotation duration is
deliberately not recorded: the health throttle sleeps at eight points inside
the sequence and dominates exactly when the node is unhealthy, so the number
would conflate work with waiting.

Nothing added for the other two suspects. Get-object serving is already
covered by the handler label, the lookup histogram and the deferred and
saturation gauges; peer churn by the disconnect-reason counter.

Also replaces nine per-file cspell ignores with one ignoreRegExpList entry
for the telemetry macro names, and picks up the levelization baseline for the
consensus span-name test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pratik Mankawde
2026-07-27 16:39:09 +01:00
parent 371f10934e
commit 8633df7a3e
25 changed files with 1402 additions and 111 deletions
Download Patch File Download Diff File Expand all files Collapse all files

70
docs/telemetry-glossary.md
View File

@@ -744,6 +744,76 @@ How long the node store takes to persist one object. This is the cost that gover
**See also:** [Node-store read latency](#node-store-read-latency) · [Node-store operation rate](#node-store-operation-rate)
<a id="heap-trim"></a>
### Heap trim
Asking the memory allocator to return free pages from its own pools back to the operating system. The node does this at the end of every periodic cache sweep, because a sweep is exactly when a large amount of memory has just been released. The cost is not fixed: the allocator has to walk its pools to find what is returnable, so the work grows with how much memory the process is holding — which is why a node with a large existing database pays more for it, every sweep, than an empty one does. The pages handed back are not gone for good; the next access to that memory has to take them again, which is the reason a trim is a trade rather than a pure saving.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Minor page fault](#minor-page-fault) · [Reclaimed resident memory](#reclaimed-resident-memory) · [Sweep interval](#sweep-interval)
<a id="minor-page-fault"></a>
### Minor page fault
A memory access the operating system satisfies without touching a disk, by attaching a page it already had available. Cheap next to a disk read but not free, and taken in volume it becomes a real cost. Faults counted during a heap trim show the trim doing its own work of releasing memory. They deliberately say nothing about the faults paid afterwards, when caches refill and touch the memory that was given back — that later cost is the reason a trim can slow other work down, and counting the faults inside the trim does not capture it. Reading the figure as the total price of trimming overstates what was measured.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Heap trim](#heap-trim) · [Reclaimed resident memory](#reclaimed-resident-memory)
<a id="reclaimed-resident-memory"></a>
### Reclaimed resident memory
How much memory a heap trim actually handed back to the operating system, as opposed to how long it spent looking. This is what makes the trim's cost judgeable: time spent with memory returned is a trade, and time spent with nothing returned is pure loss. Only memory the allocator holds in its own pools can be returned at all, so a trim can legitimately reclaim nothing — and a reading of zero is a real answer rather than a missing one. Memory can also grow across a trim, when other threads allocate faster than it releases; that is reported as no reclaim rather than as a negative amount, since a running total cannot go backwards.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Heap trim](#heap-trim) · [Sweep interval](#sweep-interval)
<a id="sweep-interval"></a>
### Sweep interval
How often the node runs its periodic pass over the in-memory caches, expiring what is stale. The period is chosen from the configured node size, so a larger node sweeps less often. It sets the cadence of everything the sweep does, including the heap trim at the end of it, and it is therefore the number against which any per-sweep cost has to be judged: the same expense is negligible at one interval and significant at another. The sweep runs as a queued job, so its cost is paid on a worker thread and competes with other work rather than happening in the background.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Heap trim](#heap-trim) · [Job queue occupancy](#job-queue-occupancy)
<a id="rotation-window"></a>
### Rotation window
The interval during which the node is swapping the pair of storage backends that make online deletion of old history possible. New data goes to one backend while the older one is kept for reading; on a swap the older one is discarded and a fresh one takes over. The window matters because it is the only time certain extra writes happen, so a cost seen inside it and a cost seen outside it have different explanations. A node that is not configured for online deletion has no window at all, which is a different situation from a window that costs nothing.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Copy-forward write](#copy-forward-write) · [Node re-store](#node-re-store) · [Node-store write latency](#node-store-write-latency)
<a id="copy-forward-write"></a>
### Copy-forward write
Rewriting a stored object out of the backend that is about to be discarded and into the one replacing it. An ordinary read would not write anything; this one must, because the copy it just read is about to be deleted and would otherwise survive only in memory. The volume scales with how much of the outgoing backend gets read during the swap, so it is a cost only a node that already holds history can incur — the reason this competes with catching up on a populated database and never appears on a fresh one.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Rotation window](#rotation-window) · [Node re-store](#node-re-store)
<a id="node-re-store"></a>
### Node re-store
Writing a tree node back to storage from memory because it could not be found in either storage backend. It signals more than cost. The node is still reachable from the current validated state, yet its only stored copy was in a backend an earlier swap discarded, and it was never rewritten because nothing had modified it. Rescuing it is an extra write, and skipping the rescue would leave the node unresolvable later. A sustained rate therefore reports two things at once: added write pressure now, and history quietly dropped by an earlier swap.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Rotation window](#rotation-window) · [Copy-forward write](#copy-forward-write) · [Missing SHAMap node](#missing-shamap-node)
<a id="operating-mode-server-state"></a>
### Operating mode / server state

45
docs/telemetry-runbook.md
View File

@@ -2507,23 +2507,44 @@ The specific symptom: a node with history starts and is slower than the same nod
was when empty. Back-fill is **write**-bound, so no read-side panel shows it.
Expand the collapsed **Back-fill & persistence** row.
| Look at | Healthy | Unhealthy | Conclude |
| ----------------------------------------------------------- | -------------------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| _NodeStore Write vs Read Latency (us/op)_ | write line flat and low | write line rising during back-fill | the backend cannot absorb writes. Adding peers will not help — check storage IOPS, the `[node_db]` backend and whether online-delete/rotation competes with the back-fill |
| | | read line far above write | the read path is the cost; read with the cache-hit panel below |
| _NodeStore Operation Rate (writes vs reads)_ | write rate non-zero while behind | write rate zero while still behind the network | nothing is being persisted — the stall is **upstream** of the node store. Go to branch C; storage is not the problem |
| _SHAMap TreeNode Cache Hit Rate_ | rising as the cache warms | persistently low | the working set does not fit the cache, or re-acquisition churns it, so every tree walk pays disk latency |
| _Acquire Source (local vs network)_ | `local` dominant on a warm node | sustained `network` on a range the node should hold | the local store is not retaining data |
| paired with _NuDB Cache Hit Ratio_ (Ledger Data Sync board) | both healthy | low on both | disk-bound sync |
| | | low here, NuDB healthy | cache pressure alone — this is the pairing that explains the whole symptom |
| Look at | Healthy | Unhealthy | Conclude |
| ----------------------------------------------------------- | ------------------------------------------------------------------------- | --------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| _NodeStore Write vs Read Latency (us/op)_ | write line flat and low | write line rising during back-fill | the backend cannot absorb writes. Adding peers will not help — check storage IOPS, the `[node_db]` backend and whether online-delete/rotation competes with the back-fill |
| | | read line far above write | the read path is the cost; read with the cache-hit panel below |
| _NodeStore Operation Rate (writes vs reads)_ | write rate non-zero while behind | write rate zero while still behind the network | nothing is being persisted — the stall is **upstream** of the node store. Go to branch C; storage is not the problem |
| _SHAMap TreeNode Cache Hit Rate_ | rising as the cache warms | persistently low | the working set does not fit the cache, or re-acquisition churns it, so every tree walk pays disk latency |
| _Acquire Source (local vs network)_ | `local` dominant on a warm node | sustained `network` on a range the node should hold | the local store is not retaining data |
| paired with _NuDB Cache Hit Ratio_ (Ledger Data Sync board) | both healthy | low on both | disk-bound sync |
| | | low here, NuDB healthy | cache pressure alone — this is the pairing that explains the whole symptom |
| _Sweep Heap-Trim Duration (p50/p95)_ | sub-millisecond | tens of milliseconds and rising with database size | the per-sweep heap trim is walking a large resident heap. It runs **on the sweep job**, so the cost lands on the job queue, not in the background — read it next to the sweep job's queue wait |
| | | flat and low while the symptom persists | the trim is not the cause; the remaining rows in this branch are |
| _Sweep Heap-Trim Faults & Reclaim Rate_ | reclaim rate tracking cache turnover, faults near zero | reclaim near zero while the duration panel shows real time | the trim is walking the heap and freeing nothing — pure cost, and the clearest case for tuning the sweep interval up |
| | | fault rate moving with the reclaim rate | pages are being handed back and immediately taken again. **Do not over-read this:** the fault delta covers the trim call only, so it shows the trim faulting — it does NOT prove the trim caused the later faults as caches refill. That is the mechanism, but it is not what this counter measures |
| _Online-Delete Rotation Window & Copy-Forward Writes_ | flag briefly 1 once per delete interval, writes only inside those windows | copy-forward rate large enough to move node-store write latency | rotation is competing with sync I/O — the extra writes exist only on a populated, already-rotated database, which is why the symptom is specific to an existing one |
| | | no series at all on either query | `online_delete` is not configured on this node, which is **not** the same as rotation costing nothing — rule the whole rotation hypothesis out and move on |
| | | copy-forward writes while the flag reads 0 | the window flag leaked; treat the rate as unattributed rather than concluding rotation is cheap |
| _Rotation Node Re-Store Rate_ | flat at zero | any sustained rate | an earlier rotation removed the only on-disk copy of clean nodes the current state map still reaches. Two consequences: each rescue is an extra write competing with sync, and without it the node would later hit an unresolvable missing-node error. Get the hashes from the `copyNode` warning in Loki — they are deliberately not labels |
**Conclusion:** the tree-node cache sits one layer **above** the node store, so a
miss here is what produces a node-store read there; reading the two together is
what tells cache pressure from a disk bottleneck. Two limits: these are **means,
not percentiles**, and `write_mean_us` is currently emitted only for the
what tells cache pressure from a disk bottleneck. The last four rows add the two
costs that are _specific_ to a node that already has data — the per-sweep heap
trim, whose price scales with the resident heap, and online-delete rotation,
whose extra writes need an archive to read from. Both are absent by construction
on a fresh node, which is what makes them candidate explanations for this branch's
symptom rather than general slowness.
Three limits to respect here. The node-store numbers are **means, not
percentiles**, and `write_mean_us` is currently emitted only for the
`[import_db]` admin import path — on an ordinary node `write_count` climbs with
no `write_mean_us` line, which is a known instrumentation gap, not a healthy
zero. Detail: [Sync pipeline](#sync-pipeline--ordered-diagnosis) steps 9 and 14.
zero. And the trim's fault counter is scoped to the **trim call only**: it shows
that the trim itself faults, and it cannot show the faults paid later as the
caches refill and touch the pages the trim returned. That later re-fault cost is
the actual mechanism by which a trim would slow a sync, and no metric here
measures it — so correlate the trim **duration** against the sweep job's queue
wait, and do not present the fault rate as proof the trim caused a slow sync.
Detail: [Sync pipeline](#sync-pipeline--ordered-diagnosis) steps 9 and 14.
#### Branch F — terminal: the node will stop validating for good