From d68ffbe9d6acff9713b4825d09d9315b843908b4 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Tue, 4 Aug 2026 13:43:18 +0100 Subject: [PATCH] alerts updatedpha Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> --- .cspell.config.yaml | 2 + docker/telemetry/.env.alerting.example | 33 +- .../provisioning/alerting/contactpoints.yaml | 76 +- .../grafana/provisioning/alerting/rules.yaml | 664 ++++++++++++++++-- docs/telemetry-runbook.md | 231 +++++- 5 files changed, 891 insertions(+), 115 deletions(-) diff --git a/.cspell.config.yaml b/.cspell.config.yaml index 4674295e40..bc5a74b5ac 100644 --- a/.cspell.config.yaml +++ b/.cspell.config.yaml @@ -334,6 +334,7 @@ words: - summands - superpeer - superpeers + - synthesise - takergets - takerpays - ters @@ -380,6 +381,7 @@ words: - unvetoed - upvotes - USDB + - utilisation - variadics - venv - vfalco diff --git a/docker/telemetry/.env.alerting.example b/docker/telemetry/.env.alerting.example index 9822c635f1..594c55c064 100644 --- a/docker/telemetry/.env.alerting.example +++ b/docker/telemetry/.env.alerting.example @@ -1,24 +1,31 @@ # rippled OTel alerting delivery — copy to `.env.alerting` and fill in. # # `.env.alerting` is gitignored; never commit a real Slack webhook or address. -# Grafana reads these when the stack starts and expands the ${VARS} referenced -# in grafana/provisioning/alerting/contactpoints.yaml. See the Alerting -# section of docs/telemetry-runbook.md. +# See the Alerting section of docs/telemetry-runbook.md. # -# Any var left blank simply disables that delivery path — the stack still runs. +# IMPORTANT — these variables do NOT feed contactpoints.yaml. +# That file carries literal placeholder values which you edit in place, because +# Grafana expands ${VAR} but does NOT support ${VAR:-default}: an unset variable +# expands to empty, fails provisioning validation, and Grafana EXITS 1 — taking +# the whole telemetry stack down, not just alerting. So a var left blank here +# does not "disable a delivery path"; referencing a blank one breaks startup. +# +# What these are actually for: +# GF_SMTP_* consumed by the Grafana container (compose `env_file`) to +# turn on mail delivery. Without these, an email contact point +# provisions fine and then silently sends nothing. +# ALERT_EMAIL_TO read by upload_alerts_to_grafana.py to build the Grafana +# CLOUD email contact point over the REST API (Cloud has no +# provisioning filesystem). Not used by the local stack. -# --- Slack --- +# --- Slack (local stack: paste the webhook into contactpoints.yaml instead) --- # Incoming-webhook URL from the Slack app (Incoming Webhooks feature). -# Used by both the warning (xrpld-default) and critical (xrpld-critical) tiers. +# Kept here as a convenient place to record it, NOT as a ${VAR} source. SLACK_WEBHOOK_URL= -# Channel label. With an incoming webhook the target channel is fixed by the -# webhook itself; this only satisfies Grafana's Slack validator. Defaults to -# #xrpld-alerts if unset. -SLACK_CHANNEL=#xrpld-alerts - -# --- Email (critical tier only) --- -# Comma-separated recipient list for critical alerts. +# --- Email --- +# Recipient for Grafana Cloud alerts (comma- or semicolon-separated). +# Consumed by upload_alerts_to_grafana.py. ALERT_EMAIL_TO= # SMTP relay Grafana sends through. Email only delivers when SMTP is enabled diff --git a/docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml b/docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml index 922caed081..03fa2344b4 100644 --- a/docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml +++ b/docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml @@ -7,23 +7,44 @@ # xrpld-critical — Slack + email; receives critical-severity alerts. # The severity split is wired in policies.yaml. # -# Secrets and personal addresses are NOT hard-coded. To enable delivery, -# replace the disabled placeholder values below with a real Slack webhook -# and alert email — e.g. by copying docker/telemetry/.env.alerting.example -# to .env.alerting (gitignored) and swapping the placeholder lines to -# ${SLACK_WEBHOOK_URL} / ${ALERT_EMAIL_TO}. See the Alerting section of -# docs/telemetry-runbook.md for setup. -# -# The defaults are deliberately non-empty, invalid-but-valid-shaped +# --------------------------------------------------------------------------- +# Supplying a real destination +# --------------------------------------------------------------------------- +# The values below are deliberately non-empty, invalid-but-valid-shaped # placeholders (an unroutable webhook host and a .invalid email). Grafana's -# alerting provisioning validator REQUIRES a non-empty Slack url and email -# addresses, and it does NOT support ${VAR:-default} expansion — an unset -# ${VAR} expands to empty and crashes Grafana on startup. The placeholders -# keep the whole stack booting with zero configuration; alerts simply route -# nowhere until a real destination is supplied. +# provisioning validator REQUIRES a non-empty Slack url and a non-empty email +# `addresses`, so the placeholders keep the whole stack booting with zero +# configuration; alerts simply route nowhere until a destination is supplied. # -# Email delivery additionally requires SMTP configured on the Grafana -# service (GF_SMTP_* in docker-compose.yml). +# To enable delivery, edit the two placeholder values IN PLACE with a real +# webhook / address. Do NOT commit the result — and do not substitute +# ${SLACK_WEBHOOK_URL} / ${ALERT_EMAIL_TO} here expecting a fallback: +# Grafana expands ${VAR} but does NOT support ${VAR:-default}, so an unset +# variable expands to empty, fails validation, and Grafana exits 1 — taking +# the whole telemetry stack down, not just alerting. +# +# Two further traps when editing this file: +# +# * Never leave a `receivers:` list empty to "disable" a tier. A contact +# point with no receivers ceases to exist, the policy tree then references +# a missing receiver, and Grafana refuses to boot. +# +# * File provisioning is upsert-only. Deleting a receiver here does NOT +# remove it from an instance that already booted with it — the old +# receiver keeps delivering. Removal requires an explicit +# `deleteContactPoints:` block (see the commented example at the bottom). +# +# Email delivery additionally requires SMTP configured on the Grafana service. +# The GF_SMTP_* variables live in docker/telemetry/.env.alerting (gitignored, +# see .env.alerting.example) and reach Grafana via the compose `env_file`; +# they are NOT set in docker-compose.yml. Email stays off until +# GF_SMTP_ENABLED=true and the relay settings point somewhere real. +# +# Grafana Cloud does NOT use this file — Cloud has no provisioning filesystem. +# Cloud delivery is created over the REST API by upload_alerts_to_grafana.py, +# which builds a single email-only contact point and attaches it to each rule +# via per-rule notification_settings. See that script's header for why the +# notification policy tree must not be touched on a shared Cloud instance. apiVersion: 1 @@ -37,10 +58,14 @@ contactPoints: settings: # Disabled placeholder: an unroutable webhook host. A non-empty url # selects Slack webhook mode (no recipient/token required) and keeps - # provisioning valid. Replace with a real ${SLACK_WEBHOOK_URL} to - # enable delivery. + # provisioning valid. Replace with a real webhook to enable delivery. url: https://hooks.slack.invalid/disabled - title: "{{ .CommonLabels.alertname }} on {{ .CommonLabels.service_instance_id }}" + # `rulename` is used rather than CommonLabels.service_instance_id + # because on NoData/Error evaluations Grafana replaces the query's + # label set with only {datasource_uid, ref_id}, so the node label is + # absent and the title would render blank — and several rules are + # deliberately configured to fire that way. + title: "{{ .CommonLabels.rulename }}" disableResolveMessage: false # --- Critical tier: Slack + email --- @@ -52,14 +77,25 @@ contactPoints: settings: # Disabled placeholder — see xrpld-slack-default above. url: https://hooks.slack.invalid/disabled - title: "[CRITICAL] {{ .CommonLabels.alertname }} on {{ .CommonLabels.service_instance_id }}" + title: "[CRITICAL] {{ .CommonLabels.rulename }}" disableResolveMessage: false - uid: xrpld-email-critical type: email settings: # Disabled placeholder: a .invalid address keeps the required # `addresses` field non-empty so provisioning validates. Replace - # with a real ${ALERT_EMAIL_TO} (and enable SMTP) to deliver. + # with a real address (and enable SMTP) to deliver. Multiple + # recipients are semicolon- or comma-separated. addresses: alerts-disabled@xrpld.invalid + # One message listing all recipients, rather than one message each. singleEmail: true disableResolveMessage: false + +# To retire a receiver that a running Grafana has already stored, uncomment +# and list its uid here — deleting the block above is not sufficient: +# +# deleteContactPoints: +# - orgId: 1 +# uid: xrpld-slack-default +# - orgId: 1 +# uid: xrpld-slack-critical diff --git a/docker/telemetry/grafana/provisioning/alerting/rules.yaml b/docker/telemetry/grafana/provisioning/alerting/rules.yaml index cf396da0c5..d8a2803c9f 100644 --- a/docker/telemetry/grafana/provisioning/alerting/rules.yaml +++ b/docker/telemetry/grafana/provisioning/alerting/rules.yaml @@ -2,20 +2,42 @@ # # Phase 9: Internal metric gap fill — alerting on health-critical metrics. # -# Six rules across three node-fatal subsystems: consensus/ledger health, -# validator health, and the job queue. Every metric referenced here is -# introduced by phase 9's MetricsRegistry. +# Twelve rules across five subsystems: consensus/ledger health, validator +# health, the job queue, node operating state, and the overlay (manifests). # # Rule shape (Grafana server-side evaluation): -# A Prometheus query — a 5-minute rate / histogram_quantile, aggregated +# A Prometheus query — a rate / increase / histogram_quantile, aggregated # `by (service_instance_id)` so each node evaluates # independently. Alert rules run headless and cannot # use the dashboards' `$node` template variable. # B reduce (last) — collapse A's series to its most recent value. # C threshold — the firing condition; `condition: C`. # +# Conventions that are load-bearing — do not "simplify" these away: +# +# * Every expr selects {service_name="xrpld"}. The same Prometheus also +# hosts a legacy statsd fleet under job="integrations/unix" which exports +# some of these names (state_accounting_* in particular) with no xrpld +# resource attributes. Without the selector those series get summed in. +# +# * `isPaused: true` on every rule. The key is camelCase; `is_paused` is +# SILENTLY IGNORED by the provisioning loader (no error, no warning) and +# leaves the rule live. Note the inconsistency: the sibling field +# `notification_settings` IS snake_case. +# +# * Sparse counters use increase(...[15m]) with a short `for`, not +# rate(...[5m]) with for: 5m. A single increment keeps rate[5m] nonzero +# for only ~4 minutes of dwell, so `for: 5m` can never be satisfied and +# the rule silently never fires for one-off events. +# +# * Rules whose intent is "this node stopped doing X" synthesise an explicit +# zero via `or (0 * max_over_time(...))`, because `sum by()` returns rows +# only for still-reporting nodes: a single dead node's row just disappears +# and noDataState never triggers. +# # Thresholds are documented in docs/telemetry-runbook.md (Alerting section) -# and are intended to be tuned. +# and are derived from measured values across a 7-node dev/devnet population. +# Production nodes (higher peer counts, real traffic) need a re-tune. apiVersion: 1 @@ -29,12 +51,14 @@ groups: interval: 1m rules: # A closed ledger that later fails validation against the network — - # any sustained nonzero rate means this node built history the rest - # of the network rejects. + # any mismatch means this node built history the rest of the network + # rejects. A healthy node never mismatches, so a single event matters: + # hence increase() over a wide window rather than a decaying rate(). - uid: xrpld-ledger-history-mismatch title: LedgerHistoryMismatch condition: C - for: 5m + for: 2m + isPaused: true noDataState: NoData execErrState: Error labels: @@ -43,25 +67,25 @@ groups: annotations: summary: "Ledger history mismatch on {{ $labels.service_instance_id }}" description: >- - Node {{ $labels.service_instance_id }} is recording ledger history - mismatches ({{ $values.B.Value }}/s over 5m). The node's built ledger - diverges from the validated network chain. + Node {{ $labels.service_instance_id }} recorded + {{ $values.B.Value }} ledger history mismatch(es) in the last 15m. + The node's built ledger diverges from the validated network chain. data: - refId: A relativeTimeRange: - from: 600 + from: 1200 to: 0 datasourceUid: prometheus model: refId: A - expr: sum by (service_instance_id) (rate(ledger_history_mismatch_total[5m])) + expr: sum by (service_instance_id) (increase(ledger_history_mismatch_total{service_name="xrpld"}[15m])) instant: true range: false intervalMs: 1000 maxDataPoints: 43200 - refId: B relativeTimeRange: - from: 600 + from: 1200 to: 0 datasourceUid: __expr__ model: @@ -74,7 +98,7 @@ groups: uid: __expr__ - refId: C relativeTimeRange: - from: 600 + from: 1200 to: 0 datasourceUid: __expr__ model: @@ -90,12 +114,15 @@ groups: uid: __expr__ # Healthy nodes close a ledger every ~3-5s. Zero closes for 3 minutes - # means consensus/ledger advancement is stuck. NoData (metric absent) - # also fires — a vanished series here means the node is down. + # means consensus/ledger advancement is stuck. The `or 0 * + # max_over_time` term synthesises a zero row for a node that was + # reporting within the last hour but has now gone silent, so a single + # dead node trips the threshold instead of vanishing from the result. - uid: xrpld-ledger-close-stalled title: LedgerCloseStalled condition: C for: 3m + isPaused: true noDataState: Alerting execErrState: Error labels: @@ -106,7 +133,7 @@ groups: description: >- Node {{ $labels.service_instance_id }} has closed no ledgers for several minutes (5m rate has decayed to zero). Consensus or ledger - advancement is stuck. + advancement is stuck, or the process is gone. data: - refId: A relativeTimeRange: @@ -115,7 +142,9 @@ groups: datasourceUid: prometheus model: refId: A - expr: sum by (service_instance_id) (rate(ledgers_closed_total[5m])) + expr: |- + sum by (service_instance_id) (rate(ledgers_closed_total{service_name="xrpld"}[5m])) + or (0 * max by (service_instance_id) (max_over_time(ledgers_closed_total{service_name="xrpld"}[1h]))) instant: true range: false intervalMs: 1000 @@ -150,32 +179,26 @@ groups: type: __expr__ uid: __expr__ - # ------------------------------------------------------------------ # - # Validator health # - # ------------------------------------------------------------------ # - - orgId: 1 - name: xrpld-validator - folder: xrpld - interval: 1m - rules: - # This validator's own validations are not reaching / agreeing with - # the network. A sustained nonzero miss rate risks the validator being - # dropped from UNLs. - - uid: xrpld-validations-missed - title: ValidationsMissed + # The validated ledger falling behind wall-clock is the single clearest + # "this node is unhealthy" signal on XRPL: it is the symptom every other + # consensus/sync failure eventually produces. Measured p95 is 4s on every + # node over 24h, so 60s carries ~15x headroom. + - uid: xrpld-validated-ledger-stale + title: ValidatedLedgerStale condition: C for: 5m + isPaused: true noDataState: NoData execErrState: Error labels: - severity: warning - category: validator + severity: critical + category: consensus annotations: - summary: "Validations missed on {{ $labels.service_instance_id }}" + summary: "Validated ledger stale on {{ $labels.service_instance_id }}" description: >- - Node {{ $labels.service_instance_id }} is missing validations - ({{ $values.B.Value }}/s over 5m). Its validations are not agreeing with - the validated ledger. + Node {{ $labels.service_instance_id }} has a validated ledger age of + {{ $values.B.Value }}s (>60s). The node is not keeping up with the + validated network chain. data: - refId: A relativeTimeRange: @@ -184,7 +207,7 @@ groups: datasourceUid: prometheus model: refId: A - expr: sum by (service_instance_id) (rate(validation_missed_total[5m])) + expr: max by (service_instance_id) (ledgermaster_validated_ledger_age{service_name="xrpld"}) instant: true range: false intervalMs: 1000 @@ -214,18 +237,109 @@ groups: conditions: - evaluator: type: gt - params: [0] + params: [60] + datasource: + type: __expr__ + uid: __expr__ + + # ------------------------------------------------------------------ # + # Validator health # + # ------------------------------------------------------------------ # + - orgId: 1 + name: xrpld-validator + folder: xrpld + interval: 1m + rules: + # This validator's own validations are not agreeing with the network. + # + # IMPORTANT — why this is a ratio gated on validations_sent_total, and + # not `rate(validation_missed_total) > 0`: + # ValidationTracker classifies a ledger as a miss whenever + # (weValidated && networkValidated) is not both true. A node that does + # not validate never sets weValidated, so EVERY reconciled ledger counts + # as a miss and the raw rate is permanently nonzero — measured ratio is + # exactly 1.0 on non-validating nodes. No threshold can separate "not a + # validator" from "validator disagreeing"; the `and on(...)` gate + # excludes non-validators entirely, and the ratio then measures real + # disagreement among nodes that do validate. + - uid: xrpld-validations-missed + title: ValidationsMissed + condition: C + for: 15m + isPaused: true + noDataState: NoData + execErrState: Error + labels: + severity: warning + category: validator + annotations: + summary: "Validations missed on {{ $labels.service_instance_id }}" + description: >- + Validator {{ $labels.service_instance_id }} is missing + {{ $values.B.Value }} (fraction) of its validations over 15m. Its + validations are not agreeing with the validated ledger, which risks + removal from UNLs. + data: + - refId: A + relativeTimeRange: + from: 1200 + to: 0 + datasourceUid: prometheus + model: + refId: A + expr: |- + ( + sum by (service_instance_id) (rate(validation_missed_total{service_name="xrpld"}[15m])) + / clamp_min( + sum by (service_instance_id) (rate(validation_missed_total{service_name="xrpld"}[15m])) + + sum by (service_instance_id) (rate(validation_agreements_total{service_name="xrpld"}[15m])), + 1e-9) + ) + and on (service_instance_id) + (sum by (service_instance_id) (rate(validations_sent_total{service_name="xrpld"}[15m])) > 0) + instant: true + range: false + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: + from: 1200 + to: 0 + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: + type: __expr__ + uid: __expr__ + - refId: C + relativeTimeRange: + from: 1200 + to: 0 + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + conditions: + - evaluator: + type: gt + params: [0.1] datasource: type: __expr__ uid: __expr__ # The node has stopped checking incoming validations. Zero checked # validations means it is no longer processing the validation stream - # from peers. + # from peers. Synthesises a zero for a silent-but-recently-seen node + # (see the LedgerCloseStalled comment). - uid: xrpld-validations-not-checked title: ValidationsNotChecked condition: C for: 5m + isPaused: true noDataState: Alerting execErrState: Error labels: @@ -245,7 +359,9 @@ groups: datasourceUid: prometheus model: refId: A - expr: sum by (service_instance_id) (rate(validations_checked_total[5m])) + expr: |- + sum by (service_instance_id) (rate(validations_checked_total{service_name="xrpld"}[5m])) + or (0 * max by (service_instance_id) (max_over_time(validations_checked_total{service_name="xrpld"}[1h]))) instant: true range: false intervalMs: 1000 @@ -289,11 +405,14 @@ groups: interval: 1m rules: # Transactions are being dropped because the job queue is full — the - # node is shedding load it cannot process. + # node is shedding load it cannot process. Overflow arrives in bursts, + # so this uses increase() over a wide window (see the header note on + # sparse counters). - uid: xrpld-jobqueue-tx-overflow title: JobQueueTxOverflow condition: C - for: 5m + for: 2m + isPaused: true noDataState: NoData execErrState: Error labels: @@ -302,25 +421,25 @@ groups: annotations: summary: "Job queue transaction overflow on {{ $labels.service_instance_id }}" description: >- - Node {{ $labels.service_instance_id }} is overflowing its transaction - job queue ({{ $values.B.Value }}/s over 5m). Transactions are being - dropped under load. + Node {{ $labels.service_instance_id }} overflowed its transaction + job queue {{ $values.B.Value }} time(s) in the last 15m. + Transactions are being dropped under load. data: - refId: A relativeTimeRange: - from: 600 + from: 1200 to: 0 datasourceUid: prometheus model: refId: A - expr: sum by (service_instance_id) (rate(jq_trans_overflow_total[5m])) + expr: sum by (service_instance_id) (increase(jq_trans_overflow_total{service_name="xrpld"}[15m])) instant: true range: false intervalMs: 1000 maxDataPoints: 43200 - refId: B relativeTimeRange: - from: 600 + from: 1200 to: 0 datasourceUid: __expr__ model: @@ -333,7 +452,7 @@ groups: uid: __expr__ - refId: C relativeTimeRange: - from: 600 + from: 1200 to: 0 datasourceUid: __expr__ model: @@ -349,12 +468,13 @@ groups: uid: __expr__ # p99 time a job waits in the queue before running. A sustained p99 - # above 1s means the node is saturated and work is backing up. Tune the - # threshold (in microseconds) to the deployment. + # above 1s means the node is saturated and work is backing up. `le` must + # stay inside the inner sum or histogram_quantile cannot interpolate. - uid: xrpld-jobqueue-latency-high title: JobQueueLatencyHigh condition: C for: 5m + isPaused: true noDataState: NoData execErrState: Error labels: @@ -374,7 +494,7 @@ groups: datasourceUid: prometheus model: refId: A - expr: histogram_quantile(0.99, sum by (le, service_instance_id) (rate(job_queued_us_bucket[5m]))) + expr: histogram_quantile(0.99, sum by (le, service_instance_id) (rate(job_queued_us_bucket{service_name="xrpld"}[5m]))) instant: true range: false intervalMs: 1000 @@ -408,3 +528,439 @@ groups: datasource: type: __expr__ uid: __expr__ + + # Node-store read/write latency. Sustained high IO latency is the usual + # upstream cause of state flapping and sync stalls, so this often fires + # first and explains the others. Measured p99-of-p95 is 37-49ms on + # healthy nodes and 488-566ms on nodes that are actively flapping, so + # 1000ms flags genuine degradation rather than the current baseline. + - uid: xrpld-nodestore-io-latency-high + title: NodeStoreIOLatencyHigh + condition: C + for: 10m + isPaused: true + noDataState: NoData + execErrState: Error + labels: + severity: warning + category: jobqueue + annotations: + summary: "Node store IO latency high on {{ $labels.service_instance_id }}" + description: >- + Node {{ $labels.service_instance_id }} has a p95 node-store IO + latency of {{ $values.B.Value }}ms (>1s) over 10m. Check disk + utilisation and whether the store is on a slow volume. + data: + - refId: A + relativeTimeRange: + from: 900 + to: 0 + datasourceUid: prometheus + model: + refId: A + expr: histogram_quantile(0.95, sum by (le, service_instance_id) (rate(ios_latency_milliseconds_bucket{service_name="xrpld"}[10m]))) + instant: true + range: false + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: + from: 900 + to: 0 + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: + type: __expr__ + uid: __expr__ + - refId: C + relativeTimeRange: + from: 900 + to: 0 + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + conditions: + - evaluator: + type: gt + params: [1000] + datasource: + type: __expr__ + uid: __expr__ + + # ------------------------------------------------------------------ # + # Node operating state # + # ------------------------------------------------------------------ # + - orgId: 1 + name: xrpld-node-state + folder: xrpld + interval: 1m + rules: + # Node state flapping: full -> syncing/tracking -> full, repeatedly. + # + # state_accounting_full_transitions counts transitions INTO full + # (NetworkOPs.cpp StateAccounting::mode) and is exported as a cumulative + # gauge, so increase() is correct — and its counter-reset correction + # turns a process restart into a small positive delta rather than a + # false spike. + # + # state_changes_total cannot be used here: it carries no from/to labels, + # so it cannot distinguish a flap from a normal startup walk. + # + # The uptime gate is load-bearing. Every node walks + # disconnected -> connected -> syncing -> tracking -> full once at boot; + # without the gate every restart pages. Measured: flapping nodes re-enter + # full 4-6 times per hour sustained, healthy nodes 0-1, so >3 separates + # the populations with a 3x margin. + - uid: xrpld-node-state-flapping + title: NodeStateFlapping + condition: C + for: 15m + isPaused: true + noDataState: NoData + execErrState: Error + labels: + severity: warning + category: node_state + annotations: + summary: "Node state flapping on {{ $labels.service_instance_id }}" + description: >- + Node {{ $labels.service_instance_id }} re-entered the FULL state + {{ $values.B.Value }} times in the last hour (>3). It is oscillating + between full and syncing/connected rather than holding sync. Check + node-store IO latency, peer connectivity, and clock sync. + data: + - refId: A + relativeTimeRange: + from: 3900 + to: 0 + datasourceUid: prometheus + model: + refId: A + expr: |- + sum by (service_instance_id) (increase(state_accounting_full_transitions{service_name="xrpld"}[1h])) + and on (service_instance_id) + (sum by (service_instance_id) (server_info{service_name="xrpld", metric="uptime"}) > 3600) + instant: true + range: false + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: + from: 3900 + to: 0 + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: + type: __expr__ + uid: __expr__ + - refId: C + relativeTimeRange: + from: 3900 + to: 0 + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + conditions: + - evaluator: + type: gt + params: [3] + datasource: + type: __expr__ + uid: __expr__ + + # A node stuck OUT of full. Distinct from flapping: a node that drops to + # syncing and stays there produces no further full-transitions, so the + # flapping rule by definition cannot catch it. + # server_state enum (NetworkOPs.h): DISCONNECTED=0, CONNECTED=1, + # SYNCING=2, TRACKING=3, FULL=4. + - uid: xrpld-node-not-full + title: NodeNotFull + condition: C + for: 15m + isPaused: true + noDataState: NoData + execErrState: Error + labels: + severity: warning + category: node_state + annotations: + summary: "Node not in FULL state on {{ $labels.service_instance_id }}" + description: >- + Node {{ $labels.service_instance_id }} has been below FULL + (state={{ $values.B.Value }}; 0=disconnected 1=connected 2=syncing + 3=tracking 4=full) for 15m. It is not fully synced with the network. + data: + - refId: A + relativeTimeRange: + from: 1200 + to: 0 + datasourceUid: prometheus + model: + refId: A + expr: |- + max by (service_instance_id) (server_info{service_name="xrpld", metric="server_state"}) + and on (service_instance_id) + (sum by (service_instance_id) (server_info{service_name="xrpld", metric="uptime"}) > 3600) + instant: true + range: false + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: + from: 1200 + to: 0 + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: + type: __expr__ + uid: __expr__ + - refId: C + relativeTimeRange: + from: 1200 + to: 0 + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + conditions: + - evaluator: + type: lt + params: [4] + datasource: + type: __expr__ + uid: __expr__ + + # ------------------------------------------------------------------ # + # Overlay / manifests # + # ------------------------------------------------------------------ # + - orgId: 1 + name: xrpld-overlay + folder: xrpld + interval: 1m + rules: + # Manifest job convoy — the primary manifest-flooding signal. + # + # Peers send TMManifests dumps up to ~57MB (just under + # kMaximumMessageSize, overlay/Message.h). JtManifest is registered with + # maxLimit (core/JobTypes.h), so every peer's dump runs concurrently and + # they convoy on ManifestCache::mutex_; OverlayImpl::onManifests also + # re-verifies the blob a second time on Accept. Measured effect: each + # RcvManifests job took 16-18s and the whole 8-worker pool was occupied. + # + # jobq_manifest_waiting is 0 at the 99.9th percentile on every node over + # 24h, so any sustained backlog is a genuine outlier rather than normal + # variance. Threshold >3 with a 10m dwell keeps the measured startup + # burst (peaks of 5 and 11, lasting well under 10m) from paging. + - uid: xrpld-manifest-job-convoy + title: ManifestJobQueueConvoy + condition: C + for: 10m + isPaused: true + noDataState: NoData + execErrState: Error + labels: + severity: warning + category: overlay + annotations: + summary: "Manifest job convoy on {{ $labels.service_instance_id }}" + description: >- + Node {{ $labels.service_instance_id }} has {{ $values.B.Value }} + manifest jobs waiting (>3) for 10m. Peer manifest dumps are + saturating the job pool and convoying on the manifest cache lock. + data: + - refId: A + relativeTimeRange: + from: 900 + to: 0 + datasourceUid: prometheus + model: + refId: A + expr: sum by (service_instance_id) (jobq_manifest_waiting{service_name="xrpld"}) + instant: true + range: false + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: + from: 900 + to: 0 + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: + type: __expr__ + uid: __expr__ + - refId: C + relativeTimeRange: + from: 900 + to: 0 + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + conditions: + - evaluator: + type: gt + params: [3] + datasource: + type: __expr__ + uid: __expr__ + + # Inbound manifest byte-rate flood. Complements the convoy rule: this + # catches the wire-level cause (a peer shipping huge dumps) even when the + # job pool absorbs it without a visible backlog. + # + # Measured: steady state 0.7-2.3 kB/s; p99 during the startup flood + # 267-420 kB/s. 50 kB/s sits ~20x above steady state and well below the + # flood. The uptime gate suppresses the measured startup storm, which is + # normal behaviour — the trade-off is that a flood confined to the first + # 30 minutes after boot is deliberately not alerted. + - uid: xrpld-manifest-flood-inbound + title: ManifestFloodInbound + condition: C + for: 10m + isPaused: true + noDataState: NoData + execErrState: Error + labels: + severity: warning + category: overlay + annotations: + summary: "Inbound manifest flood on {{ $labels.service_instance_id }}" + description: >- + Node {{ $labels.service_instance_id }} is receiving + {{ $values.B.Value }} B/s of manifest traffic (>50 kB/s) over 10m. + A peer is flooding oversized TMManifests dumps. + data: + - refId: A + relativeTimeRange: + from: 900 + to: 0 + datasourceUid: prometheus + model: + refId: A + expr: |- + sum by (service_instance_id) (rate(overhead_manifest_bytes_in{service_name="xrpld"}[10m])) + and on (service_instance_id) + (sum by (service_instance_id) (server_info{service_name="xrpld", metric="uptime"}) > 1800) + instant: true + range: false + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: + from: 900 + to: 0 + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: + type: __expr__ + uid: __expr__ + - refId: C + relativeTimeRange: + from: 900 + to: 0 + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + conditions: + - evaluator: + type: gt + params: [51200] + datasource: + type: __expr__ + uid: __expr__ + + # Resource-driven peer disconnects. The node is dropping peers for + # exceeding resource budgets, which precedes peer starvation and sync + # loss. Sparse and bursty, so increase() over a wide window. + # + # Measured p95 of the 30m increase: 0 on every healthy node, 4.0 and 11.7 + # on the two nodes that are independently known to be degraded (the same + # two that flap). So >5 sits above the healthy baseline entirely and only + # trips on a node already in trouble. + - uid: xrpld-peer-resource-disconnects + title: PeerResourceDisconnects + condition: C + for: 5m + isPaused: true + noDataState: NoData + execErrState: Error + labels: + severity: warning + category: overlay + annotations: + summary: "Resource-driven peer disconnects on {{ $labels.service_instance_id }}" + description: >- + Node {{ $labels.service_instance_id }} disconnected + {{ $values.B.Value }} peer(s) for resource-budget violations in the + last 30m. Sustained disconnects can starve the node of peers. + data: + - refId: A + relativeTimeRange: + from: 2100 + to: 0 + datasourceUid: prometheus + model: + refId: A + expr: sum by (service_instance_id) (increase(server_info{service_name="xrpld", metric="peer_disconnects_resources"}[30m])) + instant: true + range: false + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: + from: 2100 + to: 0 + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: + type: __expr__ + uid: __expr__ + - refId: C + relativeTimeRange: + from: 2100 + to: 0 + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + conditions: + - evaluator: + type: gt + params: [5] + datasource: + type: __expr__ + uid: __expr__ diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index 5a47345507..743b5a301c 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -2027,26 +2027,54 @@ Requires `trace_peer=1` in the `[telemetry]` config section. ## Alerting -xrpld provisions six Grafana alert rules on the health-critical metrics, so a -stock stack alerts out of the box with no UI setup. Rules are provisioned from +xrpld provisions thirteen Grafana alert rules on the health-critical metrics, so +a stock stack alerts out of the box with no UI setup. Rules are provisioned from `docker/telemetry/grafana/provisioning/alerting/` and load automatically when the Grafana container starts. They appear under **Alerting → Alert rules**, folder **xrpld**. +> **All rules ship `isPaused: true`.** Thresholds are tuned against a small +> dev/devnet population, so every rule is deactivated on arrival — compare it +> against your own baseline, then unpause. The key is camelCase: `is_paused` is +> **silently ignored** by the provisioning loader (no error, no warning) and +> leaves the rule live. Note the sibling field `notification_settings` _is_ +> snake_case. + ### Alert catalogue -All rules evaluate every minute against the Prometheus datasource, over a -5-minute window, and group by `service_instance_id` so each node alerts on its -own. Alerts fire only after the condition holds for the `for` dwell time. +All rules evaluate every minute against the Prometheus datasource and aggregate +`by (service_instance_id)` so each node alerts on its own. Every expr selects +`{service_name="xrpld"}` — the same Prometheus may also host a legacy statsd +fleet exporting some of these names (`state_accounting_*` in particular) with no +xrpld resource attributes, and without the selector those series get summed in. +Alerts fire only after the condition holds for the `for` dwell time. -| Alert | Severity | Fires when | For | -| ----------------------- | -------- | ----------------------------------------- | --- | -| `LedgerHistoryMismatch` | critical | `rate(ledger_history_mismatch_total)` > 0 | 5m | -| `LedgerCloseStalled` | critical | `rate(ledgers_closed_total)` ≈ 0 | 3m | -| `ValidationsMissed` | warning | `rate(validation_missed_total)` > 0 | 5m | -| `ValidationsNotChecked` | warning | `rate(validations_checked_total)` ≈ 0 | 5m | -| `JobQueueTxOverflow` | warning | `rate(jq_trans_overflow_total)` > 0 | 5m | -| `JobQueueLatencyHigh` | warning | p99 `job_queued_us` > 1s | 5m | +| Alert | Severity | Fires when | For | +| ------------------------- | -------- | -------------------------------------------------- | --- | +| `LedgerHistoryMismatch` | critical | `increase(ledger_history_mismatch_total[15m])` > 0 | 2m | +| `LedgerCloseStalled` | critical | `rate(ledgers_closed_total)` ≈ 0 | 3m | +| `ValidatedLedgerStale` | critical | `ledgermaster_validated_ledger_age` > 60s | 5m | +| `ValidationsMissed` | warning | validator miss _ratio_ > 0.1 | 15m | +| `ValidationsNotChecked` | warning | `rate(validations_checked_total)` ≈ 0 | 5m | +| `JobQueueTxOverflow` | warning | `increase(jq_trans_overflow_total[15m])` > 0 | 2m | +| `JobQueueLatencyHigh` | warning | p99 `job_queued_us` > 1s | 5m | +| `NodeStoreIOLatencyHigh` | warning | p95 `ios_latency_milliseconds` > 1s | 10m | +| `NodeStateFlapping` | warning | > 3 re-entries into FULL per hour | 15m | +| `NodeNotFull` | warning | `server_state` < 4 (FULL) | 15m | +| `ManifestJobQueueConvoy` | warning | `jobq_manifest_waiting` > 3 | 10m | +| `ManifestFloodInbound` | warning | `rate(overhead_manifest_bytes_in)` > 50 kB/s | 10m | +| `PeerResourceDisconnects` | warning | > 5 resource-driven peer disconnects per 30m | 5m | + +Two expression idioms recur and are load-bearing — do not "simplify" them away: + +- **Sparse counters use `increase(...[15m])` with a short `for`,** not + `rate(...[5m])` with `for: 5m`. A single increment keeps `rate[5m]` nonzero for + only ~4 minutes of dwell, so a 5-minute `for` can never be satisfied and the + rule silently never fires for the one-off events it exists to catch. +- **"Node stopped doing X" rules synthesise an explicit zero** via + `or (0 * max_over_time(...[1h]))`, because `sum by()` returns rows only for + still-reporting nodes: one dead node's row simply disappears from the result, + so `noDataState` never triggers unless _every_ node vanishes at once. #### Consensus / ledger health @@ -2060,12 +2088,28 @@ one every ~3-5s. Likely causes: lost peer connectivity, consensus stall, or the process is hung. This rule also fires on _NoData_ — if the series disappears the node is likely down. Check peer count and process health first. +**ValidatedLedgerStale** — The validated ledger has fallen more than 60s behind. +This is the clearest single "is this node healthy" signal on XRPL: it is the +symptom nearly every consensus or sync failure eventually produces, so it is +often the first thing to check and the last thing to clear. Measured p95 is ~4s +on a healthy node. + #### Validator health **ValidationsMissed** — This validator's validations are not agreeing with the validated ledger. Sustained misses risk removal from UNLs. Check clock sync, peer connectivity, and whether the node is keeping up with ledger close. +> **Why this is a ratio gated on `validations_sent_total`, not +> `rate(validation_missed_total) > 0`:** `ValidationTracker` classifies a ledger +> as a miss whenever `weValidated && networkValidated` is not _both_ true. A node +> that does not validate never sets `weValidated`, so **every** reconciled ledger +> counts as a miss and the raw rate is permanently nonzero — the measured miss +> ratio is exactly `1.0` on non-validating nodes. No threshold can separate "not +> a validator" from "validator disagreeing", so the rule gates on +> `validations_sent_total > 0` to exclude non-validators entirely, and then +> measures the ratio among nodes that genuinely do validate. + **ValidationsNotChecked** — The node has stopped checking incoming validations from peers. Likely causes: overlay/peer disconnection or a stalled validation pipeline. Fires on NoData as well. @@ -2080,6 +2124,71 @@ being dropped. The node is shedding load it cannot process. Check CPU, the before running. The node is saturated. Correlate with CPU and the Job Queue dashboard. +**NodeStoreIOLatencyHigh** — p95 node-store IO latency exceeds 1s. Sustained +store latency is the usual _upstream cause_ of state flapping and sync stalls, so +this often fires alongside `NodeStateFlapping` and explains it. Check disk +utilisation and whether the node store sits on a slow volume — moving it to a +local NVMe has previously cut time-to-`full` by more than 3x. Measured p99-of-p95 +is 37-49ms on healthy nodes and 488-566ms on nodes that are actively flapping. + +#### Node operating state + +**NodeStateFlapping** — The node is oscillating `full → syncing/connected → full` +instead of holding sync. Measured: a flapping node re-enters `full` 4-6 times per +hour sustained, while a healthy node manages 0-1, so the `> 3` threshold sits +between the two populations with roughly a 3x margin. + +The rule counts `state_accounting_full_transitions`, which counts transitions +_into_ `full` and is exported as a cumulative gauge — `increase()` is therefore +correct, and its counter-reset correction turns a process restart into a small +positive delta rather than a false spike. `state_changes_total` cannot be used +here: it carries no from/to labels, so it cannot tell a flap from a normal +startup walk. + +**The `uptime > 3600` gate is load-bearing.** Every node walks +`disconnected → connected → syncing → tracking → full` once at boot; without the +gate, every restart pages. The trade-off is deliberate: flapping confined to the +first hour after boot is not alerted. + +Investigate in this order: `NodeStoreIOLatencyHigh` (most common cause), peer +connectivity, then clock sync. + +**NodeNotFull** — The node has been below `FULL` for 15m +(`0`=disconnected, `1`=connected, `2`=syncing, `3`=tracking, `4`=full). This is +deliberately a _separate_ rule from `NodeStateFlapping`: a node that drops to +syncing and stays there produces no further full-transitions, so the flapping +counter by definition cannot catch it. + +#### Overlay / manifests + +**ManifestJobQueueConvoy** — Manifest jobs are backing up in the job queue. Peers +send `TMManifests` dumps up to ~57MB (just under `kMaximumMessageSize`, see +`overlay/Message.h`), and `JtManifest` is registered with `maxLimit` +(`core/JobTypes.h`), so every peer's dump runs concurrently and they convoy on +`ManifestCache::mutex_`; `OverlayImpl::onManifests` also re-verifies the blob a +second time on Accept. Measured effect: each `RcvManifests` job took 16-18s and +the entire 8-worker pool was occupied. + +This is the most reliable manifest-flood signal because `jobq_manifest_waiting` +is `0` at the 99.9th percentile on every node over 24h — any sustained backlog is +a genuine outlier rather than normal variance. + +**ManifestFloodInbound** — Inbound manifest byte-rate exceeds 50 kB/s. Catches the +wire-level cause (a peer shipping oversized dumps) even when the job pool absorbs +it without a visible backlog. Measured steady state is 0.7-2.3 kB/s against a p99 +of 267-420 kB/s during the startup flood, so the threshold sits ~20x above normal +and well below a real flood. + +> **Both manifest rules deliberately suppress startup.** The manifest storm at +> boot is _measured normal behaviour_, so `ManifestFloodInbound` carries an +> `uptime > 1800` gate and `ManifestJobQueueConvoy` relies on a 10m dwell that the +> startup burst does not outlast. A flood confined to the first 30 minutes after +> boot will therefore not alert. + +**PeerResourceDisconnects** — The node dropped more than 5 peers in 30m for +exceeding resource budgets. Sustained disconnects starve the node of peers and +precede sync loss. + ### Tuning thresholds Thresholds live in @@ -2117,45 +2226,111 @@ to Slack **and** email; a warning goes to Slack only. Both group by #### Configure delivery (no secrets in git) -The Slack webhook and email address are **not** hard-coded — the YAML -references `${SLACK_WEBHOOK_URL}` and `${ALERT_EMAIL_TO}`, which Grafana -expands from the environment at startup. Supply them through a gitignored -env file: +The Slack webhook and email address are **not** hard-coded. `contactpoints.yaml` +ships deliberately unroutable placeholders — an `https://hooks.slack.invalid/…` +host and an `…@xrpld.invalid` address — which keep provisioning valid so the +stack boots with zero configuration while alerts route nowhere. + +To enable delivery, edit those two values **in place** with a real webhook and +address, and do not commit the result. ```bash cp docker/telemetry/.env.alerting.example docker/telemetry/.env.alerting -# edit .env.alerting — this file is gitignored, never commit the webhook/address +# edit .env.alerting — gitignored; holds the SMTP relay settings +$EDITOR docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml docker compose -f docker/telemetry/docker-compose.yml up -d grafana ``` -- **Slack** — set `SLACK_WEBHOOK_URL` to an incoming-webhook URL. Drives both - tiers. -- **Email** — set `ALERT_EMAIL_TO` (comma-separated) **and** point the - `GF_SMTP_*` vars at a real relay with `GF_SMTP_ENABLED=true`. Grafana can - only send mail once SMTP is configured. +- **Slack** — replace the placeholder `url:` with an incoming-webhook URL. Drives + both tiers. +- **Email** — replace the placeholder `addresses:` (comma- or semicolon-separated) + **and** point the `GF_SMTP_*` vars in `.env.alerting` at a real relay with + `GF_SMTP_ENABLED=true`. Grafana can only send mail once SMTP is configured. -Any variable left blank disables that path; the stack still runs. To add a -third destination (PagerDuty, Opsgenie, a custom webhook), add a receiver to -the relevant contact point. +Three traps worth knowing before you edit this file: + +- **Do not substitute `${SLACK_WEBHOOK_URL}` / `${ALERT_EMAIL_TO}` here.** Grafana + expands `${VAR}` but does **not** support `${VAR:-default}`, so an unset variable + expands to empty, fails validation, and Grafana **exits 1** — taking the whole + telemetry stack down, not just alerting. A blank variable does not "disable that + path"; it breaks startup. +- **Never empty a `receivers:` list to disable a tier.** A contact point with no + receivers ceases to exist, the policy tree then references a missing receiver, + and Grafana refuses to boot. Point the route at a contact point that still + exists instead. +- **File provisioning is upsert-only.** Deleting a receiver from the YAML does not + remove it from an instance that already booted with it — the old receiver keeps + delivering. Removal needs an explicit `deleteContactPoints:` block listing the + uid (a commented example sits at the bottom of `contactpoints.yaml`). + +To add a third destination (PagerDuty, Opsgenie, a custom webhook), add a receiver +to the relevant contact point. + +#### Deploying alerts to Grafana Cloud + +Grafana Cloud has **no provisioning filesystem**, so these `apiVersion: 1` files +cannot be loaded there. Cloud deployment goes through the REST API via +`docker/telemetry/upload_alerts_to_grafana.py`, which reads the same tracked +`rules.yaml` as the single source of truth (so local and Cloud cannot drift) and +applies the Cloud-specific transforms: the local `prometheus` datasource uid is +swapped for the Cloud one, the `folder:` _name_ becomes an existing `folderUID`, +and `interval` becomes integer seconds. + +```bash +cd docker/telemetry +python3 upload_alerts_to_grafana.py --dry-run # always dry-run first +python3 upload_alerts_to_grafana.py # create rules, paused +python3 upload_alerts_to_grafana.py --verify # read back what is deployed +``` + +Credentials come from `.env.grafanaserviceapi` (gitignored, a service-account +token with `alert.rules:write`); the recipient address comes from `ALERT_EMAIL_TO` +in `.env.alerting`. Neither is ever written to a tracked file. Use +`--no-delivery` to land the rules before a recipient is chosen, and `--activate` +only once the thresholds have been checked against the target fleet's baseline. + +> **The Cloud notification policy tree must not be pushed.** There is exactly one +> policy tree per org and the PUT endpoint **replaces it wholesale**. On a shared +> stack the root receiver and its sibling routes belong to other teams, so pushing +> an xrpld-shaped tree would silently re-route their alerts. The uploader +> therefore never touches the tree; instead each rule carries +> `notification_settings.receiver`, which routes that rule directly to the xrpld +> contact point and bypasses the tree entirely. Verify with a before/after hash of +> `GET /api/v1/provisioning/policies`. ### Verifying alert provisioning loaded After the stack is up: ```bash -# All six rules present? -curl -s http://localhost:3000/api/v1/provisioning/alert-rules | jq '.[].title' +# All thirteen rules present, and is each one paused? +curl -s http://localhost:3000/api/v1/provisioning/alert-rules | + jq -r '.[] | "\(.title)\tpaused=\(.isPaused)"' # Contact points present? curl -s http://localhost:3000/api/v1/provisioning/contact-points | jq '.[].name' ``` +Check `paused=true` explicitly rather than assuming it: a mis-spelled +`is_paused` is dropped without any error and the rule provisions **live**. + Grafana logs a provisioning error and skips the file if the YAML is malformed: ```bash docker compose -f docker/telemetry/docker-compose.yml logs grafana | grep -i alerting ``` +A malformed _expression_ fails differently and more quietly — the rule loads but +every evaluation errors. After a threshold or expr change, confirm each rule's +query still returns data: + +```bash +# Should print a numeric value per node, and no empty results +curl -sG http://localhost:9090/api/v1/query \ + --data-urlencode 'query=sum by (service_instance_id) (rate(ledgers_closed_total{service_name="xrpld"}[5m]))' | + jq '.data.result | length' +``` + ## Log-Trace Correlation When xrpld is built with `telemetry=ON`, log lines emitted within an active OpenTelemetry span automatically include `trace_id` and `span_id` fields: