From feabcc5b895534e6348351fa6fe3fa43e3dbacd7 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Tue, 4 Aug 2026 15:14:54 +0100 Subject: [PATCH] docs(telemetry): sync alerting runbook with corrected thresholds MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update the catalogue table and prose for the two thresholds corrected by the 7-day backtest, and document why ValidatedLedgerStale must exclude the 1209600s sentinel that getValidatedLedgerAge() returns when no validated ledger exists — that clause looks redundant and would otherwise be removed as a simplification. Also record the tuning lesson: justify thresholds from a 7-day sample, not 24 hours. --- docs/telemetry-runbook.md | 28 +++++++++++++++++++++------- 1 file changed, 21 insertions(+), 7 deletions(-) diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index 813c7bf9b7..8582a8dd6a 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -1224,7 +1224,7 @@ Alerts fire only after the condition holds for the `for` dwell time. | `NodeStateFlapping` | warning | > 3 re-entries into FULL per hour | 15m | | `NodeNotFull` | warning | `server_state` < 4 (FULL) | 15m | | `ManifestJobQueueConvoy` | warning | `jobq_manifest_waiting` > 3 | 10m | -| `ManifestFloodInbound` | warning | `rate(overhead_manifest_bytes_in)` > 50 kB/s | 10m | +| `ManifestFloodInbound` | warning | `rate(overhead_manifest_bytes_in)` > 512 kB/s | 10m | | `PeerResourceDisconnects` | warning | > 5 resource-driven peer disconnects per 30m | 5m | Two expression idioms recur and are load-bearing — do not "simplify" them away: @@ -1253,8 +1253,17 @@ node is likely down. Check peer count and process health first. **ValidatedLedgerStale** — The validated ledger has fallen more than 60s behind. This is the clearest single "is this node healthy" signal on XRPL: it is the symptom nearly every consensus or sync failure eventually produces, so it is -often the first thing to check and the last thing to clear. Measured p95 is ~4s -on a healthy node. +often the first thing to check and the last thing to clear. Measured over 7 days: +p50 2s, p95 4s, p99 5s on every node. + +> **The `< 1209600` clause in this rule's expression is required — do not remove +> it.** When a node holds no validated ledger at all, +> `LedgerMaster::getValidatedLedgerAge()` returns `weeks{2}` (1 209 600 s) as a +> **sentinel**, not a measurement. Without the clause the rule reads that as "14 +> days stale" and fires on every node during startup — measured, it produced +> sustained firing on all nine nodes over a six-day window, healthy ones included. +> A node genuinely stuck without a validated ledger is caught by +> `LedgerCloseStalled` and `NodeNotFull` instead. #### Validator health @@ -1335,11 +1344,16 @@ This is the most reliable manifest-flood signal because `jobq_manifest_waiting` is `0` at the 99.9th percentile on every node over 24h — any sustained backlog is a genuine outlier rather than normal variance. -**ManifestFloodInbound** — Inbound manifest byte-rate exceeds 50 kB/s. Catches the +**ManifestFloodInbound** — Inbound manifest byte-rate exceeds 512 kB/s. Catches the wire-level cause (a peer shipping oversized dumps) even when the job pool absorbs -it without a visible backlog. Measured steady state is 0.7-2.3 kB/s against a p99 -of 267-420 kB/s during the startup flood, so the threshold sits ~20x above normal -and well below a real flood. +it without a visible backlog. Measured over 7 days: healthy p95 0.2-0.5 kB/s and +p99 1.0-1.8 kB/s, against peaks up to 2.7 MB/s during real storms — so the +threshold sits ~280x above healthy p99 and ~5x below the peaks. + +> An earlier revision used 50 kB/s, justified from a 24-hour window. Over a full +> week that produced ~41 sustained 5-minute firings across six **healthy** nodes, +> i.e. routine paging. Prefer a 7-day sample when tuning any threshold here; 24 +> hours is too short to expose weekly variation. > **Both manifest rules deliberately suppress startup.** The manifest storm at > boot is _measured normal behaviour_, so `ManifestFloodInbound` carries an