ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-19 21:30:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(telemetry): sync alerting runbook with corrected thresholds

Browse Source 
Update the catalogue table and prose for the two thresholds corrected by the
7-day backtest, and document why ValidatedLedgerStale must exclude the
1209600s sentinel that getValidatedLedgerAge() returns when no validated
ledger exists — that clause looks redundant and would otherwise be removed
as a simplification.

Also record the tuning lesson: justify thresholds from a 7-day sample, not
24 hours.
This commit is contained in:
Pratik Mankawde
2026-08-04 15:14:54 +01:00
parent ebe54d9db5
commit feabcc5b89
1 changed files with 21 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
docs/telemetry-runbook.md
View File

@@ -1224,7 +1224,7 @@ Alerts fire only after the condition holds for the `for` dwell time.
| `NodeStateFlapping` | warning | > 3 re-entries into FULL per hour | 15m |
| `NodeNotFull` | warning | `server_state` < 4 (FULL) | 15m |
| `ManifestJobQueueConvoy` | warning | `jobq_manifest_waiting` > 3 | 10m |
| `ManifestFloodInbound` | warning | `rate(overhead_manifest_bytes_in)` > 50 kB/s | 10m |
| `ManifestFloodInbound` | warning | `rate(overhead_manifest_bytes_in)` > 512 kB/s | 10m |
| `PeerResourceDisconnects` | warning | > 5 resource-driven peer disconnects per 30m | 5m |
Two expression idioms recur and are load-bearing — do not "simplify" them away:
@@ -1253,8 +1253,17 @@ node is likely down. Check peer count and process health first.
**ValidatedLedgerStale** — The validated ledger has fallen more than 60s behind.
This is the clearest single "is this node healthy" signal on XRPL: it is the
symptom nearly every consensus or sync failure eventually produces, so it is
often the first thing to check and the last thing to clear. Measured p95 is ~4s
on a healthy node.
often the first thing to check and the last thing to clear. Measured over 7 days:
p50 2s, p95 4s, p99 5s on every node.
> **The `< 1209600` clause in this rule's expression is required — do not remove
> it.** When a node holds no validated ledger at all,
> `LedgerMaster::getValidatedLedgerAge()` returns `weeks{2}` (1 209 600 s) as a
> **sentinel**, not a measurement. Without the clause the rule reads that as "14
> days stale" and fires on every node during startup — measured, it produced
> sustained firing on all nine nodes over a six-day window, healthy ones included.
> A node genuinely stuck without a validated ledger is caught by
> `LedgerCloseStalled` and `NodeNotFull` instead.
#### Validator health
@@ -1335,11 +1344,16 @@ This is the most reliable manifest-flood signal because `jobq_manifest_waiting`
is `0` at the 99.9th percentile on every node over 24h — any sustained backlog is
a genuine outlier rather than normal variance.
**ManifestFloodInbound** — Inbound manifest byte-rate exceeds 50 kB/s. Catches the
**ManifestFloodInbound** — Inbound manifest byte-rate exceeds 512 kB/s. Catches the
wire-level cause (a peer shipping oversized dumps) even when the job pool absorbs
it without a visible backlog. Measured steady state is 0.7-2.3 kB/s against a p99
of 267-420 kB/s during the startup flood, so the threshold sits ~20x above normal
and well below a real flood.
it without a visible backlog. Measured over 7 days: healthy p95 0.2-0.5 kB/s and
p99 1.0-1.8 kB/s, against peaks up to 2.7 MB/s during real storms — so the
threshold sits ~280x above healthy p99 and ~5x below the peaks.
> An earlier revision used 50 kB/s, justified from a 24-hour window. Over a full
> week that produced ~41 sustained 5-minute firings across six **healthy** nodes,
> i.e. routine paging. Prefer a 7-day sample when tuning any threshold here; 24
> hours is too short to expose weekly variation.
> **Both manifest rules deliberately suppress startup.** The manifest storm at
> boot is _measured normal behaviour_, so `ManifestFloodInbound` carries an