ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-21 14:20:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

alerts updatedpha

Browse Source 
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
This commit is contained in:
Pratik Mankawde
2026-08-04 13:43:18 +01:00
parent 7d3e83611e
commit d68ffbe9d6
5 changed files with 891 additions and 115 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.cspell.config.yaml
View File

@@ -334,6 +334,7 @@ words:
- summands
- superpeer
- superpeers
- synthesise
- takergets
- takerpays
- ters
@@ -380,6 +381,7 @@ words:
- unvetoed
- upvotes
- USDB
- utilisation
- variadics
- venv
- vfalco

33
docker/telemetry/.env.alerting.example
View File

@@ -1,24 +1,31 @@
# rippled OTel alerting delivery — copy to `.env.alerting` and fill in.
#
# `.env.alerting` is gitignored; never commit a real Slack webhook or address.
# Grafana reads these when the stack starts and expands the ${VARS} referenced
# in grafana/provisioning/alerting/contactpoints.yaml. See the Alerting
# section of docs/telemetry-runbook.md.
# See the Alerting section of docs/telemetry-runbook.md.
#
# Any var left blank simply disables that delivery path — the stack still runs.
# IMPORTANT — these variables do NOT feed contactpoints.yaml.
# That file carries literal placeholder values which you edit in place, because
# Grafana expands ${VAR} but does NOT support ${VAR:-default}: an unset variable
# expands to empty, fails provisioning validation, and Grafana EXITS 1 — taking
# the whole telemetry stack down, not just alerting. So a var left blank here
# does not "disable a delivery path"; referencing a blank one breaks startup.
#
# What these are actually for:
# GF_SMTP_* consumed by the Grafana container (compose `env_file`) to
# turn on mail delivery. Without these, an email contact point
# provisions fine and then silently sends nothing.
# ALERT_EMAIL_TO read by upload_alerts_to_grafana.py to build the Grafana
# CLOUD email contact point over the REST API (Cloud has no
# provisioning filesystem). Not used by the local stack.
# --- Slack ---
# --- Slack (local stack: paste the webhook into contactpoints.yaml instead) ---
# Incoming-webhook URL from the Slack app (Incoming Webhooks feature).
# Used by both the warning (xrpld-default) and critical (xrpld-critical) tiers.
# Kept here as a convenient place to record it, NOT as a ${VAR} source.
SLACK_WEBHOOK_URL=
# Channel label. With an incoming webhook the target channel is fixed by the
# webhook itself; this only satisfies Grafana's Slack validator. Defaults to
# #xrpld-alerts if unset.
SLACK_CHANNEL=#xrpld-alerts
# --- Email (critical tier only) ---
# Comma-separated recipient list for critical alerts.
# --- Email ---
# Recipient for Grafana Cloud alerts (comma- or semicolon-separated).
# Consumed by upload_alerts_to_grafana.py.
ALERT_EMAIL_TO=
# SMTP relay Grafana sends through. Email only delivers when SMTP is enabled

76
docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml
View File

@@ -7,23 +7,44 @@
# xrpld-critical — Slack + email; receives critical-severity alerts.
# The severity split is wired in policies.yaml.
#
# Secrets and personal addresses are NOT hard-coded. To enable delivery,
# replace the disabled placeholder values below with a real Slack webhook
# and alert email — e.g. by copying docker/telemetry/.env.alerting.example
# to .env.alerting (gitignored) and swapping the placeholder lines to
# ${SLACK_WEBHOOK_URL} / ${ALERT_EMAIL_TO}. See the Alerting section of
# docs/telemetry-runbook.md for setup.
#
# The defaults are deliberately non-empty, invalid-but-valid-shaped
# ---------------------------------------------------------------------------
# Supplying a real destination
# ---------------------------------------------------------------------------
# The values below are deliberately non-empty, invalid-but-valid-shaped
# placeholders (an unroutable webhook host and a .invalid email). Grafana's
# alerting provisioning validator REQUIRES a non-empty Slack url and email
# addresses, and it does NOT support ${VAR:-default} expansion — an unset
# ${VAR} expands to empty and crashes Grafana on startup. The placeholders
# keep the whole stack booting with zero configuration; alerts simply route
# nowhere until a real destination is supplied.
# provisioning validator REQUIRES a non-empty Slack url and a non-empty email
# `addresses`, so the placeholders keep the whole stack booting with zero
# configuration; alerts simply route nowhere until a destination is supplied.
#
# Email delivery additionally requires SMTP configured on the Grafana
# service (GF_SMTP_* in docker-compose.yml).
# To enable delivery, edit the two placeholder values IN PLACE with a real
# webhook / address. Do NOT commit the result — and do not substitute
# ${SLACK_WEBHOOK_URL} / ${ALERT_EMAIL_TO} here expecting a fallback:
# Grafana expands ${VAR} but does NOT support ${VAR:-default}, so an unset
# variable expands to empty, fails validation, and Grafana exits 1 — taking
# the whole telemetry stack down, not just alerting.
#
# Two further traps when editing this file:
#
# * Never leave a `receivers:` list empty to "disable" a tier. A contact
# point with no receivers ceases to exist, the policy tree then references
# a missing receiver, and Grafana refuses to boot.
#
# * File provisioning is upsert-only. Deleting a receiver here does NOT
# remove it from an instance that already booted with it — the old
# receiver keeps delivering. Removal requires an explicit
# `deleteContactPoints:` block (see the commented example at the bottom).
#
# Email delivery additionally requires SMTP configured on the Grafana service.
# The GF_SMTP_* variables live in docker/telemetry/.env.alerting (gitignored,
# see .env.alerting.example) and reach Grafana via the compose `env_file`;
# they are NOT set in docker-compose.yml. Email stays off until
# GF_SMTP_ENABLED=true and the relay settings point somewhere real.
#
# Grafana Cloud does NOT use this file — Cloud has no provisioning filesystem.
# Cloud delivery is created over the REST API by upload_alerts_to_grafana.py,
# which builds a single email-only contact point and attaches it to each rule
# via per-rule notification_settings. See that script's header for why the
# notification policy tree must not be touched on a shared Cloud instance.
apiVersion: 1
@@ -37,10 +58,14 @@ contactPoints:
settings:
# Disabled placeholder: an unroutable webhook host. A non-empty url
# selects Slack webhook mode (no recipient/token required) and keeps
# provisioning valid. Replace with a real ${SLACK_WEBHOOK_URL} to
# enable delivery.
# provisioning valid. Replace with a real webhook to enable delivery.
url: https://hooks.slack.invalid/disabled
title: "{{ .CommonLabels.alertname }} on {{ .CommonLabels.service_instance_id }}"
# `rulename` is used rather than CommonLabels.service_instance_id
# because on NoData/Error evaluations Grafana replaces the query's
# label set with only {datasource_uid, ref_id}, so the node label is
# absent and the title would render blank — and several rules are
# deliberately configured to fire that way.
title: "{{ .CommonLabels.rulename }}"
disableResolveMessage: false
# --- Critical tier: Slack + email ---
@@ -52,14 +77,25 @@ contactPoints:
settings:
# Disabled placeholder — see xrpld-slack-default above.
url: https://hooks.slack.invalid/disabled
title: "[CRITICAL] {{ .CommonLabels.alertname }} on {{ .CommonLabels.service_instance_id }}"
title: "[CRITICAL] {{ .CommonLabels.rulename }}"
disableResolveMessage: false
- uid: xrpld-email-critical
type: email
settings:
# Disabled placeholder: a .invalid address keeps the required
# `addresses` field non-empty so provisioning validates. Replace
# with a real ${ALERT_EMAIL_TO} (and enable SMTP) to deliver.
# with a real address (and enable SMTP) to deliver. Multiple
# recipients are semicolon- or comma-separated.
addresses: alerts-disabled@xrpld.invalid
# One message listing all recipients, rather than one message each.
singleEmail: true
disableResolveMessage: false
# To retire a receiver that a running Grafana has already stored, uncomment
# and list its uid here — deleting the block above is not sufficient:
#
# deleteContactPoints:
# - orgId: 1
# uid: xrpld-slack-default
# - orgId: 1
# uid: xrpld-slack-critical

664
docker/telemetry/grafana/provisioning/alerting/rules.yaml
View File

@@ -2,20 +2,42 @@
#
# Phase 9: Internal metric gap fill — alerting on health-critical metrics.
#
# Six rules across three node-fatal subsystems: consensus/ledger health,
# validator health, and the job queue. Every metric referenced here is
# introduced by phase 9's MetricsRegistry.
# Twelve rules across five subsystems: consensus/ledger health, validator
# health, the job queue, node operating state, and the overlay (manifests).
#
# Rule shape (Grafana server-side evaluation):
# A Prometheus query — a 5-minute rate / histogram_quantile, aggregated
# A Prometheus query — a rate / increase / histogram_quantile, aggregated
# `by (service_instance_id)` so each node evaluates
# independently. Alert rules run headless and cannot
# use the dashboards' `$node` template variable.
# B reduce (last) — collapse A's series to its most recent value.
# C threshold — the firing condition; `condition: C`.
#
# Conventions that are load-bearing — do not "simplify" these away:
#
# * Every expr selects {service_name="xrpld"}. The same Prometheus also
# hosts a legacy statsd fleet under job="integrations/unix" which exports
# some of these names (state_accounting_* in particular) with no xrpld
# resource attributes. Without the selector those series get summed in.
#
# * `isPaused: true` on every rule. The key is camelCase; `is_paused` is
# SILENTLY IGNORED by the provisioning loader (no error, no warning) and
# leaves the rule live. Note the inconsistency: the sibling field
# `notification_settings` IS snake_case.
#
# * Sparse counters use increase(...[15m]) with a short `for`, not
# rate(...[5m]) with for: 5m. A single increment keeps rate[5m] nonzero
# for only ~4 minutes of dwell, so `for: 5m` can never be satisfied and
# the rule silently never fires for one-off events.
#
# * Rules whose intent is "this node stopped doing X" synthesise an explicit
# zero via `or (0 * max_over_time(...))`, because `sum by()` returns rows
# only for still-reporting nodes: a single dead node's row just disappears
# and noDataState never triggers.
#
# Thresholds are documented in docs/telemetry-runbook.md (Alerting section)
# and are intended to be tuned.
# and are derived from measured values across a 7-node dev/devnet population.
# Production nodes (higher peer counts, real traffic) need a re-tune.
apiVersion: 1
@@ -29,12 +51,14 @@ groups:
interval: 1m
rules:
# A closed ledger that later fails validation against the network —
# any sustained nonzero rate means this node built history the rest
# of the network rejects.
# any mismatch means this node built history the rest of the network
# rejects. A healthy node never mismatches, so a single event matters:
# hence increase() over a wide window rather than a decaying rate().
- uid: xrpld-ledger-history-mismatch
title: LedgerHistoryMismatch
condition: C
for: 5m
for: 2m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
@@ -43,25 +67,25 @@ groups:
annotations:
summary: "Ledger history mismatch on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} is recording ledger history
mismatches ({{ $values.B.Value }}/s over 5m). The node's built ledger
diverges from the validated network chain.
Node {{ $labels.service_instance_id }} recorded
{{ $values.B.Value }} ledger history mismatch(es) in the last 15m.
The node's built ledger diverges from the validated network chain.
data:
- refId: A
relativeTimeRange:
from: 600
from: 1200
to: 0
datasourceUid: prometheus
model:
refId: A
expr: sum by (service_instance_id) (rate(ledger_history_mismatch_total[5m]))
expr: sum by (service_instance_id) (increase(ledger_history_mismatch_total{service_name="xrpld"}[15m]))
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 600
from: 1200
to: 0
datasourceUid: __expr__
model:
@@ -74,7 +98,7 @@ groups:
uid: __expr__
- refId: C
relativeTimeRange:
from: 600
from: 1200
to: 0
datasourceUid: __expr__
model:
@@ -90,12 +114,15 @@ groups:
uid: __expr__
# Healthy nodes close a ledger every ~3-5s. Zero closes for 3 minutes
# means consensus/ledger advancement is stuck. NoData (metric absent)
# also fires — a vanished series here means the node is down.
# means consensus/ledger advancement is stuck. The `or 0 *
# max_over_time` term synthesises a zero row for a node that was
# reporting within the last hour but has now gone silent, so a single
# dead node trips the threshold instead of vanishing from the result.
- uid: xrpld-ledger-close-stalled
title: LedgerCloseStalled
condition: C
for: 3m
isPaused: true
noDataState: Alerting
execErrState: Error
labels:
@@ -106,7 +133,7 @@ groups:
description: >-
Node {{ $labels.service_instance_id }} has closed no ledgers for
several minutes (5m rate has decayed to zero). Consensus or ledger
advancement is stuck.
advancement is stuck, or the process is gone.
data:
- refId: A
relativeTimeRange:
@@ -115,7 +142,9 @@ groups:
datasourceUid: prometheus
model:
refId: A
expr: sum by (service_instance_id) (rate(ledgers_closed_total[5m]))
expr: |-
sum by (service_instance_id) (rate(ledgers_closed_total{service_name="xrpld"}[5m]))
or (0 * max by (service_instance_id) (max_over_time(ledgers_closed_total{service_name="xrpld"}[1h])))
instant: true
range: false
intervalMs: 1000
@@ -150,32 +179,26 @@ groups:
type: __expr__
uid: __expr__
# ------------------------------------------------------------------ #
# Validator health #
# ------------------------------------------------------------------ #
- orgId: 1
name: xrpld-validator
folder: xrpld
interval: 1m
rules:
# This validator's own validations are not reaching / agreeing with
# the network. A sustained nonzero miss rate risks the validator being
# dropped from UNLs.
- uid: xrpld-validations-missed
title: ValidationsMissed
# The validated ledger falling behind wall-clock is the single clearest
# "this node is unhealthy" signal on XRPL: it is the symptom every other
# consensus/sync failure eventually produces. Measured p95 is 4s on every
# node over 24h, so 60s carries ~15x headroom.
- uid: xrpld-validated-ledger-stale
title: ValidatedLedgerStale
condition: C
for: 5m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
severity: warning
category: validator
severity: critical
category: consensus
annotations:
summary: "Validations missed on {{ $labels.service_instance_id }}"
summary: "Validated ledger stale on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} is missing validations
({{ $values.B.Value }}/s over 5m). Its validations are not agreeing with
the validated ledger.
Node {{ $labels.service_instance_id }} has a validated ledger age of
{{ $values.B.Value }}s (>60s). The node is not keeping up with the
validated network chain.
data:
- refId: A
relativeTimeRange:
@@ -184,7 +207,7 @@ groups:
datasourceUid: prometheus
model:
refId: A
expr: sum by (service_instance_id) (rate(validation_missed_total[5m]))
expr: max by (service_instance_id) (ledgermaster_validated_ledger_age{service_name="xrpld"})
instant: true
range: false
intervalMs: 1000
@@ -214,18 +237,109 @@ groups:
conditions:
- evaluator:
type: gt
params: [0]
params: [60]
datasource:
type: __expr__
uid: __expr__
# ------------------------------------------------------------------ #
# Validator health #
# ------------------------------------------------------------------ #
- orgId: 1
name: xrpld-validator
folder: xrpld
interval: 1m
rules:
# This validator's own validations are not agreeing with the network.
#
# IMPORTANT — why this is a ratio gated on validations_sent_total, and
# not `rate(validation_missed_total) > 0`:
# ValidationTracker classifies a ledger as a miss whenever
# (weValidated && networkValidated) is not both true. A node that does
# not validate never sets weValidated, so EVERY reconciled ledger counts
# as a miss and the raw rate is permanently nonzero — measured ratio is
# exactly 1.0 on non-validating nodes. No threshold can separate "not a
# validator" from "validator disagreeing"; the `and on(...)` gate
# excludes non-validators entirely, and the ratio then measures real
# disagreement among nodes that do validate.
- uid: xrpld-validations-missed
title: ValidationsMissed
condition: C
for: 15m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
severity: warning
category: validator
annotations:
summary: "Validations missed on {{ $labels.service_instance_id }}"
description: >-
Validator {{ $labels.service_instance_id }} is missing
{{ $values.B.Value }} (fraction) of its validations over 15m. Its
validations are not agreeing with the validated ledger, which risks
removal from UNLs.
data:
- refId: A
relativeTimeRange:
from: 1200
to: 0
datasourceUid: prometheus
model:
refId: A
expr: |-
(
sum by (service_instance_id) (rate(validation_missed_total{service_name="xrpld"}[15m]))
/ clamp_min(
sum by (service_instance_id) (rate(validation_missed_total{service_name="xrpld"}[15m]))
+ sum by (service_instance_id) (rate(validation_agreements_total{service_name="xrpld"}[15m])),
1e-9)
)
and on (service_instance_id)
(sum by (service_instance_id) (rate(validations_sent_total{service_name="xrpld"}[15m])) > 0)
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 1200
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 1200
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [0.1]
datasource:
type: __expr__
uid: __expr__
# The node has stopped checking incoming validations. Zero checked
# validations means it is no longer processing the validation stream
# from peers.
# from peers. Synthesises a zero for a silent-but-recently-seen node
# (see the LedgerCloseStalled comment).
- uid: xrpld-validations-not-checked
title: ValidationsNotChecked
condition: C
for: 5m
isPaused: true
noDataState: Alerting
execErrState: Error
labels:
@@ -245,7 +359,9 @@ groups:
datasourceUid: prometheus
model:
refId: A
expr: sum by (service_instance_id) (rate(validations_checked_total[5m]))
expr: |-
sum by (service_instance_id) (rate(validations_checked_total{service_name="xrpld"}[5m]))
or (0 * max by (service_instance_id) (max_over_time(validations_checked_total{service_name="xrpld"}[1h])))
instant: true
range: false
intervalMs: 1000
@@ -289,11 +405,14 @@ groups:
interval: 1m
rules:
# Transactions are being dropped because the job queue is full — the
# node is shedding load it cannot process.
# node is shedding load it cannot process. Overflow arrives in bursts,
# so this uses increase() over a wide window (see the header note on
# sparse counters).
- uid: xrpld-jobqueue-tx-overflow
title: JobQueueTxOverflow
condition: C
for: 5m
for: 2m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
@@ -302,25 +421,25 @@ groups:
annotations:
summary: "Job queue transaction overflow on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} is overflowing its transaction
job queue ({{ $values.B.Value }}/s over 5m). Transactions are being
dropped under load.
Node {{ $labels.service_instance_id }} overflowed its transaction
job queue {{ $values.B.Value }} time(s) in the last 15m.
Transactions are being dropped under load.
data:
- refId: A
relativeTimeRange:
from: 600
from: 1200
to: 0
datasourceUid: prometheus
model:
refId: A
expr: sum by (service_instance_id) (rate(jq_trans_overflow_total[5m]))
expr: sum by (service_instance_id) (increase(jq_trans_overflow_total{service_name="xrpld"}[15m]))
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 600
from: 1200
to: 0
datasourceUid: __expr__
model:
@@ -333,7 +452,7 @@ groups:
uid: __expr__
- refId: C
relativeTimeRange:
from: 600
from: 1200
to: 0
datasourceUid: __expr__
model:
@@ -349,12 +468,13 @@ groups:
uid: __expr__
# p99 time a job waits in the queue before running. A sustained p99
# above 1s means the node is saturated and work is backing up. Tune the
# threshold (in microseconds) to the deployment.
# above 1s means the node is saturated and work is backing up. `le` must
# stay inside the inner sum or histogram_quantile cannot interpolate.
- uid: xrpld-jobqueue-latency-high
title: JobQueueLatencyHigh
condition: C
for: 5m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
@@ -374,7 +494,7 @@ groups:
datasourceUid: prometheus
model:
refId: A
expr: histogram_quantile(0.99, sum by (le, service_instance_id) (rate(job_queued_us_bucket[5m])))
expr: histogram_quantile(0.99, sum by (le, service_instance_id) (rate(job_queued_us_bucket{service_name="xrpld"}[5m])))
instant: true
range: false
intervalMs: 1000
@@ -408,3 +528,439 @@ groups:
datasource:
type: __expr__
uid: __expr__
# Node-store read/write latency. Sustained high IO latency is the usual
# upstream cause of state flapping and sync stalls, so this often fires
# first and explains the others. Measured p99-of-p95 is 37-49ms on
# healthy nodes and 488-566ms on nodes that are actively flapping, so
# 1000ms flags genuine degradation rather than the current baseline.
- uid: xrpld-nodestore-io-latency-high
title: NodeStoreIOLatencyHigh
condition: C
for: 10m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
severity: warning
category: jobqueue
annotations:
summary: "Node store IO latency high on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} has a p95 node-store IO
latency of {{ $values.B.Value }}ms (>1s) over 10m. Check disk
utilisation and whether the store is on a slow volume.
data:
- refId: A
relativeTimeRange:
from: 900
to: 0
datasourceUid: prometheus
model:
refId: A
expr: histogram_quantile(0.95, sum by (le, service_instance_id) (rate(ios_latency_milliseconds_bucket{service_name="xrpld"}[10m])))
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 900
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 900
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [1000]
datasource:
type: __expr__
uid: __expr__
# ------------------------------------------------------------------ #
# Node operating state #
# ------------------------------------------------------------------ #
- orgId: 1
name: xrpld-node-state
folder: xrpld
interval: 1m
rules:
# Node state flapping: full -> syncing/tracking -> full, repeatedly.
#
# state_accounting_full_transitions counts transitions INTO full
# (NetworkOPs.cpp StateAccounting::mode) and is exported as a cumulative
# gauge, so increase() is correct — and its counter-reset correction
# turns a process restart into a small positive delta rather than a
# false spike.
#
# state_changes_total cannot be used here: it carries no from/to labels,
# so it cannot distinguish a flap from a normal startup walk.
#
# The uptime gate is load-bearing. Every node walks
# disconnected -> connected -> syncing -> tracking -> full once at boot;
# without the gate every restart pages. Measured: flapping nodes re-enter
# full 4-6 times per hour sustained, healthy nodes 0-1, so >3 separates
# the populations with a 3x margin.
- uid: xrpld-node-state-flapping
title: NodeStateFlapping
condition: C
for: 15m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
severity: warning
category: node_state
annotations:
summary: "Node state flapping on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} re-entered the FULL state
{{ $values.B.Value }} times in the last hour (>3). It is oscillating
between full and syncing/connected rather than holding sync. Check
node-store IO latency, peer connectivity, and clock sync.
data:
- refId: A
relativeTimeRange:
from: 3900
to: 0
datasourceUid: prometheus
model:
refId: A
expr: |-
sum by (service_instance_id) (increase(state_accounting_full_transitions{service_name="xrpld"}[1h]))
and on (service_instance_id)
(sum by (service_instance_id) (server_info{service_name="xrpld", metric="uptime"}) > 3600)
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 3900
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 3900
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [3]
datasource:
type: __expr__
uid: __expr__
# A node stuck OUT of full. Distinct from flapping: a node that drops to
# syncing and stays there produces no further full-transitions, so the
# flapping rule by definition cannot catch it.
# server_state enum (NetworkOPs.h): DISCONNECTED=0, CONNECTED=1,
# SYNCING=2, TRACKING=3, FULL=4.
- uid: xrpld-node-not-full
title: NodeNotFull
condition: C
for: 15m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
severity: warning
category: node_state
annotations:
summary: "Node not in FULL state on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} has been below FULL
(state={{ $values.B.Value }}; 0=disconnected 1=connected 2=syncing
3=tracking 4=full) for 15m. It is not fully synced with the network.
data:
- refId: A
relativeTimeRange:
from: 1200
to: 0
datasourceUid: prometheus
model:
refId: A
expr: |-
max by (service_instance_id) (server_info{service_name="xrpld", metric="server_state"})
and on (service_instance_id)
(sum by (service_instance_id) (server_info{service_name="xrpld", metric="uptime"}) > 3600)
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 1200
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 1200
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: lt
params: [4]
datasource:
type: __expr__
uid: __expr__
# ------------------------------------------------------------------ #
# Overlay / manifests #
# ------------------------------------------------------------------ #
- orgId: 1
name: xrpld-overlay
folder: xrpld
interval: 1m
rules:
# Manifest job convoy — the primary manifest-flooding signal.
#
# Peers send TMManifests dumps up to ~57MB (just under
# kMaximumMessageSize, overlay/Message.h). JtManifest is registered with
# maxLimit (core/JobTypes.h), so every peer's dump runs concurrently and
# they convoy on ManifestCache::mutex_; OverlayImpl::onManifests also
# re-verifies the blob a second time on Accept. Measured effect: each
# RcvManifests job took 16-18s and the whole 8-worker pool was occupied.
#
# jobq_manifest_waiting is 0 at the 99.9th percentile on every node over
# 24h, so any sustained backlog is a genuine outlier rather than normal
# variance. Threshold >3 with a 10m dwell keeps the measured startup
# burst (peaks of 5 and 11, lasting well under 10m) from paging.
- uid: xrpld-manifest-job-convoy
title: ManifestJobQueueConvoy
condition: C
for: 10m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
severity: warning
category: overlay
annotations:
summary: "Manifest job convoy on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} has {{ $values.B.Value }}
manifest jobs waiting (>3) for 10m. Peer manifest dumps are
saturating the job pool and convoying on the manifest cache lock.
data:
- refId: A
relativeTimeRange:
from: 900
to: 0
datasourceUid: prometheus
model:
refId: A
expr: sum by (service_instance_id) (jobq_manifest_waiting{service_name="xrpld"})
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 900
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 900
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [3]
datasource:
type: __expr__
uid: __expr__
# Inbound manifest byte-rate flood. Complements the convoy rule: this
# catches the wire-level cause (a peer shipping huge dumps) even when the
# job pool absorbs it without a visible backlog.
#
# Measured: steady state 0.7-2.3 kB/s; p99 during the startup flood
# 267-420 kB/s. 50 kB/s sits ~20x above steady state and well below the
# flood. The uptime gate suppresses the measured startup storm, which is
# normal behaviour — the trade-off is that a flood confined to the first
# 30 minutes after boot is deliberately not alerted.
- uid: xrpld-manifest-flood-inbound
title: ManifestFloodInbound
condition: C
for: 10m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
severity: warning
category: overlay
annotations:
summary: "Inbound manifest flood on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} is receiving
{{ $values.B.Value }} B/s of manifest traffic (>50 kB/s) over 10m.
A peer is flooding oversized TMManifests dumps.
data:
- refId: A
relativeTimeRange:
from: 900
to: 0
datasourceUid: prometheus
model:
refId: A
expr: |-
sum by (service_instance_id) (rate(overhead_manifest_bytes_in{service_name="xrpld"}[10m]))
and on (service_instance_id)
(sum by (service_instance_id) (server_info{service_name="xrpld", metric="uptime"}) > 1800)
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 900
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 900
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [51200]
datasource:
type: __expr__
uid: __expr__
# Resource-driven peer disconnects. The node is dropping peers for
# exceeding resource budgets, which precedes peer starvation and sync
# loss. Sparse and bursty, so increase() over a wide window.
#
# Measured p95 of the 30m increase: 0 on every healthy node, 4.0 and 11.7
# on the two nodes that are independently known to be degraded (the same
# two that flap). So >5 sits above the healthy baseline entirely and only
# trips on a node already in trouble.
- uid: xrpld-peer-resource-disconnects
title: PeerResourceDisconnects
condition: C
for: 5m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
severity: warning
category: overlay
annotations:
summary: "Resource-driven peer disconnects on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} disconnected
{{ $values.B.Value }} peer(s) for resource-budget violations in the
last 30m. Sustained disconnects can starve the node of peers.
data:
- refId: A
relativeTimeRange:
from: 2100
to: 0
datasourceUid: prometheus
model:
refId: A
expr: sum by (service_instance_id) (increase(server_info{service_name="xrpld", metric="peer_disconnects_resources"}[30m]))
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 2100
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 2100
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [5]
datasource:
type: __expr__
uid: __expr__

231
docs/telemetry-runbook.md
View File

@@ -2027,26 +2027,54 @@ Requires `trace_peer=1` in the `[telemetry]` config section.
## Alerting
xrpld provisions six Grafana alert rules on the health-critical metrics, so a
stock stack alerts out of the box with no UI setup. Rules are provisioned from
xrpld provisions thirteen Grafana alert rules on the health-critical metrics, so
a stock stack alerts out of the box with no UI setup. Rules are provisioned from
`docker/telemetry/grafana/provisioning/alerting/` and load automatically when
the Grafana container starts. They appear under **Alerting → Alert rules**,
folder **xrpld**.
> **All rules ship `isPaused: true`.** Thresholds are tuned against a small
> dev/devnet population, so every rule is deactivated on arrival — compare it
> against your own baseline, then unpause. The key is camelCase: `is_paused` is
> **silently ignored** by the provisioning loader (no error, no warning) and
> leaves the rule live. Note the sibling field `notification_settings` _is_
> snake_case.
### Alert catalogue
All rules evaluate every minute against the Prometheus datasource, over a
5-minute window, and group by `service_instance_id` so each node alerts on its
own. Alerts fire only after the condition holds for the `for` dwell time.
All rules evaluate every minute against the Prometheus datasource and aggregate
`by (service_instance_id)` so each node alerts on its own. Every expr selects
`{service_name="xrpld"}` — the same Prometheus may also host a legacy statsd
fleet exporting some of these names (`state_accounting_*` in particular) with no
xrpld resource attributes, and without the selector those series get summed in.
Alerts fire only after the condition holds for the `for` dwell time.
| Alert | Severity | Fires when | For |
| ----------------------- | -------- | ----------------------------------------- | --- |
| `LedgerHistoryMismatch` | critical | `rate(ledger_history_mismatch_total)` > 0 | 5m |
| `LedgerCloseStalled` | critical | `rate(ledgers_closed_total)` ≈ 0 | 3m |
| `ValidationsMissed` | warning | `rate(validation_missed_total)` > 0 | 5m |
| `ValidationsNotChecked` | warning | `rate(validations_checked_total)` ≈ 0 | 5m |
| `JobQueueTxOverflow` | warning | `rate(jq_trans_overflow_total)` > 0 | 5m |
| `JobQueueLatencyHigh` | warning | p99 `job_queued_us` > 1s | 5m |
| Alert | Severity | Fires when | For |
| ------------------------- | -------- | -------------------------------------------------- | --- |
| `LedgerHistoryMismatch` | critical | `increase(ledger_history_mismatch_total[15m])` > 0 | 2m |
| `LedgerCloseStalled` | critical | `rate(ledgers_closed_total)` ≈ 0 | 3m |
| `ValidatedLedgerStale` | critical | `ledgermaster_validated_ledger_age` > 60s | 5m |
| `ValidationsMissed` | warning | validator miss _ratio_ > 0.1 | 15m |
| `ValidationsNotChecked` | warning | `rate(validations_checked_total)` ≈ 0 | 5m |
| `JobQueueTxOverflow` | warning | `increase(jq_trans_overflow_total[15m])` > 0 | 2m |
| `JobQueueLatencyHigh` | warning | p99 `job_queued_us` > 1s | 5m |
| `NodeStoreIOLatencyHigh` | warning | p95 `ios_latency_milliseconds` > 1s | 10m |
| `NodeStateFlapping` | warning | > 3 re-entries into FULL per hour | 15m |
| `NodeNotFull` | warning | `server_state` < 4 (FULL) | 15m |
| `ManifestJobQueueConvoy` | warning | `jobq_manifest_waiting` > 3 | 10m |
| `ManifestFloodInbound` | warning | `rate(overhead_manifest_bytes_in)` > 50 kB/s | 10m |
| `PeerResourceDisconnects` | warning | > 5 resource-driven peer disconnects per 30m | 5m |
Two expression idioms recur and are load-bearing — do not "simplify" them away:
- **Sparse counters use `increase(...[15m])` with a short `for`,** not
`rate(...[5m])` with `for: 5m`. A single increment keeps `rate[5m]` nonzero for
only ~4 minutes of dwell, so a 5-minute `for` can never be satisfied and the
rule silently never fires for the one-off events it exists to catch.
- **"Node stopped doing X" rules synthesise an explicit zero** via
`or (0 * max_over_time(...[1h]))`, because `sum by()` returns rows only for
still-reporting nodes: one dead node's row simply disappears from the result,
so `noDataState` never triggers unless _every_ node vanishes at once.
#### Consensus / ledger health
@@ -2060,12 +2088,28 @@ one every ~3-5s. Likely causes: lost peer connectivity, consensus stall, or the
process is hung. This rule also fires on _NoData_ — if the series disappears the
node is likely down. Check peer count and process health first.
**ValidatedLedgerStale** — The validated ledger has fallen more than 60s behind.
This is the clearest single "is this node healthy" signal on XRPL: it is the
symptom nearly every consensus or sync failure eventually produces, so it is
often the first thing to check and the last thing to clear. Measured p95 is ~4s
on a healthy node.
#### Validator health
**ValidationsMissed** — This validator's validations are not agreeing with the
validated ledger. Sustained misses risk removal from UNLs. Check clock sync,
peer connectivity, and whether the node is keeping up with ledger close.
> **Why this is a ratio gated on `validations_sent_total`, not
> `rate(validation_missed_total) > 0`:** `ValidationTracker` classifies a ledger
> as a miss whenever `weValidated && networkValidated` is not _both_ true. A node
> that does not validate never sets `weValidated`, so **every** reconciled ledger
> counts as a miss and the raw rate is permanently nonzero — the measured miss
> ratio is exactly `1.0` on non-validating nodes. No threshold can separate "not
> a validator" from "validator disagreeing", so the rule gates on
> `validations_sent_total > 0` to exclude non-validators entirely, and then
> measures the ratio among nodes that genuinely do validate.
**ValidationsNotChecked** — The node has stopped checking incoming validations
from peers. Likely causes: overlay/peer disconnection or a stalled validation
pipeline. Fires on NoData as well.
@@ -2080,6 +2124,71 @@ being dropped. The node is shedding load it cannot process. Check CPU, the
before running. The node is saturated. Correlate with CPU and the Job Queue
dashboard.
**NodeStoreIOLatencyHigh** — p95 node-store IO latency exceeds 1s. Sustained
store latency is the usual _upstream cause_ of state flapping and sync stalls, so
this often fires alongside `NodeStateFlapping` and explains it. Check disk
utilisation and whether the node store sits on a slow volume — moving it to a
local NVMe has previously cut time-to-`full` by more than 3x. Measured p99-of-p95
is 37-49ms on healthy nodes and 488-566ms on nodes that are actively flapping.
#### Node operating state
**NodeStateFlapping** — The node is oscillating `full → syncing/connected → full`
instead of holding sync. Measured: a flapping node re-enters `full` 4-6 times per
hour sustained, while a healthy node manages 0-1, so the `> 3` threshold sits
between the two populations with roughly a 3x margin.
The rule counts `state_accounting_full_transitions`, which counts transitions
_into_ `full` and is exported as a cumulative gauge — `increase()` is therefore
correct, and its counter-reset correction turns a process restart into a small
positive delta rather than a false spike. `state_changes_total` cannot be used
here: it carries no from/to labels, so it cannot tell a flap from a normal
startup walk.
**The `uptime > 3600` gate is load-bearing.** Every node walks
`disconnected → connected → syncing → tracking → full` once at boot; without the
gate, every restart pages. The trade-off is deliberate: flapping confined to the
first hour after boot is not alerted.
Investigate in this order: `NodeStoreIOLatencyHigh` (most common cause), peer
connectivity, then clock sync.
**NodeNotFull** — The node has been below `FULL` for 15m
(`0`=disconnected, `1`=connected, `2`=syncing, `3`=tracking, `4`=full). This is
deliberately a _separate_ rule from `NodeStateFlapping`: a node that drops to
syncing and stays there produces no further full-transitions, so the flapping
counter by definition cannot catch it.
#### Overlay / manifests
**ManifestJobQueueConvoy** — Manifest jobs are backing up in the job queue. Peers
send `TMManifests` dumps up to ~57MB (just under `kMaximumMessageSize`, see
`overlay/Message.h`), and `JtManifest` is registered with `maxLimit`
(`core/JobTypes.h`), so every peer's dump runs concurrently and they convoy on
`ManifestCache::mutex_`; `OverlayImpl::onManifests` also re-verifies the blob a
second time on Accept. Measured effect: each `RcvManifests` job took 16-18s and
the entire 8-worker pool was occupied.
This is the most reliable manifest-flood signal because `jobq_manifest_waiting`
is `0` at the 99.9th percentile on every node over 24h — any sustained backlog is
a genuine outlier rather than normal variance.
**ManifestFloodInbound** — Inbound manifest byte-rate exceeds 50 kB/s. Catches the
wire-level cause (a peer shipping oversized dumps) even when the job pool absorbs
it without a visible backlog. Measured steady state is 0.7-2.3 kB/s against a p99
of 267-420 kB/s during the startup flood, so the threshold sits ~20x above normal
and well below a real flood.
> **Both manifest rules deliberately suppress startup.** The manifest storm at
> boot is _measured normal behaviour_, so `ManifestFloodInbound` carries an
> `uptime > 1800` gate and `ManifestJobQueueConvoy` relies on a 10m dwell that the
> startup burst does not outlast. A flood confined to the first 30 minutes after
> boot will therefore not alert.
**PeerResourceDisconnects** — The node dropped more than 5 peers in 30m for
exceeding resource budgets. Sustained disconnects starve the node of peers and
precede sync loss.
### Tuning thresholds
Thresholds live in
@@ -2117,45 +2226,111 @@ to Slack **and** email; a warning goes to Slack only. Both group by
#### Configure delivery (no secrets in git)
The Slack webhook and email address are **not** hard-coded — the YAML
references `${SLACK_WEBHOOK_URL}` and `${ALERT_EMAIL_TO}`, which Grafana
expands from the environment at startup. Supply them through a gitignored
env file:
The Slack webhook and email address are **not** hard-coded. `contactpoints.yaml`
ships deliberately unroutable placeholders — an `https://hooks.slack.invalid/…`
host and an `…@xrpld.invalid` address — which keep provisioning valid so the
stack boots with zero configuration while alerts route nowhere.
To enable delivery, edit those two values **in place** with a real webhook and
address, and do not commit the result.
```bash
cp docker/telemetry/.env.alerting.example docker/telemetry/.env.alerting
# edit .env.alerting — this file is gitignored, never commit the webhook/address
# edit .env.alerting — gitignored; holds the SMTP relay settings
$EDITOR docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml
docker compose -f docker/telemetry/docker-compose.yml up -d grafana
```
- **Slack** — set `SLACK_WEBHOOK_URL` to an incoming-webhook URL. Drives both
tiers.
- **Email** — set `ALERT_EMAIL_TO` (comma-separated) **and** point the
`GF_SMTP_*` vars at a real relay with `GF_SMTP_ENABLED=true`. Grafana can
only send mail once SMTP is configured.
- **Slack** — replace the placeholder `url:` with an incoming-webhook URL. Drives
both tiers.
- **Email** — replace the placeholder `addresses:` (comma- or semicolon-separated)
**and** point the `GF_SMTP_*` vars in `.env.alerting` at a real relay with
`GF_SMTP_ENABLED=true`. Grafana can only send mail once SMTP is configured.
Any variable left blank disables that path; the stack still runs. To add a
third destination (PagerDuty, Opsgenie, a custom webhook), add a receiver to
the relevant contact point.
Three traps worth knowing before you edit this file:
- **Do not substitute `${SLACK_WEBHOOK_URL}` / `${ALERT_EMAIL_TO}` here.** Grafana
expands `${VAR}` but does **not** support `${VAR:-default}`, so an unset variable
expands to empty, fails validation, and Grafana **exits 1** — taking the whole
telemetry stack down, not just alerting. A blank variable does not "disable that
path"; it breaks startup.
- **Never empty a `receivers:` list to disable a tier.** A contact point with no
receivers ceases to exist, the policy tree then references a missing receiver,
and Grafana refuses to boot. Point the route at a contact point that still
exists instead.
- **File provisioning is upsert-only.** Deleting a receiver from the YAML does not
remove it from an instance that already booted with it — the old receiver keeps
delivering. Removal needs an explicit `deleteContactPoints:` block listing the
uid (a commented example sits at the bottom of `contactpoints.yaml`).
To add a third destination (PagerDuty, Opsgenie, a custom webhook), add a receiver
to the relevant contact point.
#### Deploying alerts to Grafana Cloud
Grafana Cloud has **no provisioning filesystem**, so these `apiVersion: 1` files
cannot be loaded there. Cloud deployment goes through the REST API via
`docker/telemetry/upload_alerts_to_grafana.py`, which reads the same tracked
`rules.yaml` as the single source of truth (so local and Cloud cannot drift) and
applies the Cloud-specific transforms: the local `prometheus` datasource uid is
swapped for the Cloud one, the `folder:` _name_ becomes an existing `folderUID`,
and `interval` becomes integer seconds.
```bash
cd docker/telemetry
python3 upload_alerts_to_grafana.py --dry-run # always dry-run first
python3 upload_alerts_to_grafana.py # create rules, paused
python3 upload_alerts_to_grafana.py --verify # read back what is deployed
```
Credentials come from `.env.grafanaserviceapi` (gitignored, a service-account
token with `alert.rules:write`); the recipient address comes from `ALERT_EMAIL_TO`
in `.env.alerting`. Neither is ever written to a tracked file. Use
`--no-delivery` to land the rules before a recipient is chosen, and `--activate`
only once the thresholds have been checked against the target fleet's baseline.
> **The Cloud notification policy tree must not be pushed.** There is exactly one
> policy tree per org and the PUT endpoint **replaces it wholesale**. On a shared
> stack the root receiver and its sibling routes belong to other teams, so pushing
> an xrpld-shaped tree would silently re-route their alerts. The uploader
> therefore never touches the tree; instead each rule carries
> `notification_settings.receiver`, which routes that rule directly to the xrpld
> contact point and bypasses the tree entirely. Verify with a before/after hash of
> `GET /api/v1/provisioning/policies`.
### Verifying alert provisioning loaded
After the stack is up:
```bash
# All six rules present?
curl -s http://localhost:3000/api/v1/provisioning/alert-rules | jq '.[].title'
# All thirteen rules present, and is each one paused?
curl -s http://localhost:3000/api/v1/provisioning/alert-rules |
jq -r '.[] | "\(.title)\tpaused=\(.isPaused)"'
# Contact points present?
curl -s http://localhost:3000/api/v1/provisioning/contact-points | jq '.[].name'
```
Check `paused=true` explicitly rather than assuming it: a mis-spelled
`is_paused` is dropped without any error and the rule provisions **live**.
Grafana logs a provisioning error and skips the file if the YAML is malformed:
```bash
docker compose -f docker/telemetry/docker-compose.yml logs grafana | grep -i alerting
```
A malformed _expression_ fails differently and more quietly — the rule loads but
every evaluation errors. After a threshold or expr change, confirm each rule's
query still returns data:
```bash
# Should print a numeric value per node, and no empty results
curl -sG http://localhost:9090/api/v1/query \
--data-urlencode 'query=sum by (service_instance_id) (rate(ledgers_closed_total{service_name="xrpld"}[5m]))' |
jq '.data.result | length'
```
## Log-Trace Correlation
When xrpld is built with `telemetry=ON`, log lines emitted within an active OpenTelemetry span automatically include `trace_id` and `span_id` fields: