ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-07-24 15:40:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(telemetry): route alerts to Slack + email with severity tiers

Browse Source 
Replace the placeholder webhook contact point with two env-driven tiers:
- xrpld-default (Slack) receives warning-severity alerts
- xrpld-critical (Slack + email) receives critical-severity alerts, via a
  severity=critical child route in the notification policy

Secrets stay out of git: contactpoints.yaml references ${SLACK_WEBHOOK_URL},
${SLACK_CHANNEL}, and ${ALERT_EMAIL_TO}, which Grafana expands from a
gitignored .env.alerting (loaded via an optional env_file on the grafana
service). .env.alerting.example documents the variables; email additionally
needs GF_SMTP_* configured.

Fold the ALERTING.md runbook content into the Alerting section of
docs/telemetry-runbook.md (operator docs live in one place) and delete
ALERTING.md. Move the .env.* ignore rule into docker/telemetry/.gitignore.

Verified live: Grafana 11.5.2 provisioned both contact points, the
severity-routed policy, and all 6 rules with no errors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Pratik Mankawde
2026-07-07 12:57:43 +01:00
parent dce39bc71d
commit 4ef8a02389
9 changed files with 228 additions and 165 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.gitignore vendored
View File

@@ -86,6 +86,3 @@ __pycache__
# clangd cache
/.cache
# Env. file carrying environmental setup data for local or cloud runs.
.env.*

32
docker/telemetry/.env.alerting.example Normal file
View File

@@ -0,0 +1,32 @@
# rippled OTel alerting delivery — copy to `.env.alerting` and fill in.
#
# `.env.alerting` is gitignored; never commit a real Slack webhook or address.
# Grafana reads these when the stack starts and expands the ${VARS} referenced
# in grafana/provisioning/alerting/contactpoints.yaml. See the Alerting
# section of docs/telemetry-runbook.md.
#
# Any var left blank simply disables that delivery path — the stack still runs.
# --- Slack ---
# Incoming-webhook URL from the Slack app (Incoming Webhooks feature).
# Used by both the warning (xrpld-default) and critical (xrpld-critical) tiers.
SLACK_WEBHOOK_URL=
# Channel label. With an incoming webhook the target channel is fixed by the
# webhook itself; this only satisfies Grafana's Slack validator. Defaults to
# #xrpld-alerts if unset.
SLACK_CHANNEL=#xrpld-alerts
# --- Email (critical tier only) ---
# Comma-separated recipient list for critical alerts.
ALERT_EMAIL_TO=
# SMTP relay Grafana sends through. Email only delivers when SMTP is enabled
# and these point at a real relay (e.g. an internal smart host or an
# authenticated provider).
GF_SMTP_ENABLED=false
GF_SMTP_HOST=smtp.example.com:587
GF_SMTP_USER=
GF_SMTP_PASSWORD=
GF_SMTP_FROM_ADDRESS=alerts@xrpld.local
GF_SMTP_FROM_NAME=xrpld Alerts

6
docker/telemetry/.gitignore vendored
View File

@@ -1,2 +1,8 @@
# Runtime data generated by xrpld and telemetry stack
data/
# Env. file carrying environmental setup data for local or cloud runs.
.env.*
# Keep examples
!.env.alerting.example

133
docker/telemetry/ALERTING.md
View File

@@ -1,133 +0,0 @@
# rippled OpenTelemetry Alerting Runbook
rippled exports its internal metrics and provisions Grafana alert rules on the
health-critical ones. This document explains each alert, its likely cause, and
how to point alerts at a real receiver.
The rules are provisioned from
`grafana/provisioning/alerting/` and load automatically when the Grafana
container in [docker-compose.yml](docker-compose.yml) starts — no manual setup
in the Grafana UI. Rules appear in Grafana under **Alerting → Alert rules**,
folder **xrpld**.
---
## Alert catalogue
All rules evaluate every minute against the Prometheus datasource, over a
5-minute window, and group by `exported_instance` so each node alerts on its
own. Alerts fire only after the condition holds for the `for` dwell time.
| Alert | Severity | Fires when | For |
| ----------------------- | -------- | ----------------------------------------------- | --- |
| `LedgerHistoryMismatch` | critical | `rate(xrpld_ledger_history_mismatch_total)` > 0 | 5m |
| `LedgerCloseStalled` | critical | `rate(xrpld_ledgers_closed_total)` ≈ 0 | 3m |
| `ValidationsMissed` | warning | `rate(xrpld_validation_missed_total)` > 0 | 5m |
| `ValidationsNotChecked` | warning | `rate(xrpld_validations_checked_total)` ≈ 0 | 5m |
| `JobQueueTxOverflow` | warning | `rate(xrpld_jq_trans_overflow_total)` > 0 | 5m |
| `JobQueueLatencyHigh` | warning | p99 `xrpld_job_queued_duration_us` > 1s | 5m |
### Consensus / ledger health
**LedgerHistoryMismatch** — The node closed a ledger whose history diverges
from the validated network chain. Likely causes: corrupted local state, a bug,
or a node that fell out of sync and rebuilt incorrectly. Investigate the node's
ledger acquisition logs; a healthy node never mismatches.
**LedgerCloseStalled** — No ledgers closed for 3 minutes. A healthy node closes
one every ~3-5s. Likely causes: lost peer connectivity, consensus stall, or the
process is hung. This rule also fires on _NoData_ — if the series disappears the
node is likely down. Check peer count and process health first.
### Validator health
**ValidationsMissed** — This validator's validations are not agreeing with the
validated ledger. Sustained misses risk removal from UNLs. Check clock sync,
peer connectivity, and whether the node is keeping up with ledger close.
**ValidationsNotChecked** — The node has stopped checking incoming validations
from peers. Likely causes: overlay/peer disconnection or a stalled validation
pipeline. Fires on NoData as well.
### Job queue / resource health
**JobQueueTxOverflow** — The transaction job queue is full and transactions are
being dropped. The node is shedding load it cannot process. Check CPU, the
`JobQueueLatencyHigh` alert, and offered load.
**JobQueueLatencyHigh** — p99 queue wait exceeds 1 second, i.e. jobs back up
before running. The node is saturated. Correlate with CPU and the Job Queue
dashboard.
---
## Tuning thresholds
Thresholds live in
[grafana/provisioning/alerting/rules.yaml](grafana/provisioning/alerting/rules.yaml)
as the `params` array of each rule's `C` (threshold) node. Common tunables:
- **`JobQueueLatencyHigh`** — `params: [1000000]` is 1 000 000 µs (1s). Lower
it for latency-sensitive deployments.
- **`LedgerCloseStalled` / `ValidationsNotChecked`** — use `lt` with a tiny
epsilon (`0.001`) rather than `0`, so floating-point rate noise near zero
does not suppress the alert.
Edit the file and restart the Grafana container to reload:
```bash
docker compose -f docker/telemetry/docker-compose.yml restart grafana
```
---
## Sending alerts somewhere real
The provisioned contact point `xrpld-default`
([contactpoints.yaml](grafana/provisioning/alerting/contactpoints.yaml)) ships
as a **local-dev webhook** to a placeholder URL — alerts fire but go nowhere
until you change it.
**Option A — repoint the webhook.** Replace the `url` under the `webhook`
receiver with a real endpoint (PagerDuty Events API, Opsgenie, a custom sink).
**Option B — add a Slack receiver.** Add a second receiver to the same contact
point:
```yaml
- uid: xrpld-slack
type: slack
settings:
url: https://hooks.slack.com/services/XXX/YYY/ZZZ
title: "{{ .CommonLabels.alertname }} on {{ .CommonLabels.exported_instance }}"
```
**Option C — email.** Requires Grafana SMTP configured via `GF_SMTP_*`
environment variables on the Grafana service in `docker-compose.yml`, then an
`email` receiver with `addresses`.
Routing is a single flat policy in
[policies.yaml](grafana/provisioning/alerting/policies.yaml): all alerts →
`xrpld-default`, grouped by `alertname` + `exported_instance`. To route
critical alerts to a different receiver, add child routes matching
`severity = critical`.
---
## Verifying provisioning loaded
After the stack is up:
```bash
# All six rules present?
curl -s http://localhost:3000/api/v1/provisioning/alert-rules | jq '.[].title'
# Contact point present?
curl -s http://localhost:3000/api/v1/provisioning/contact-points | jq '.[].name'
```
Grafana logs a provisioning error and skips the file if the YAML is malformed:
```bash
docker compose -f docker/telemetry/docker-compose.yml logs grafana | grep -i alerting
```

13
docker/telemetry/docker-compose.yml
View File

@@ -94,6 +94,13 @@ services:
# Anonymous admin access enabled for local development convenience.
grafana:
image: grafana/grafana:11.5.2
# Alerting secrets/addresses (Slack webhook, alert email) come from this
# gitignored file; Grafana expands the ${VARS} referenced in the alerting
# provisioning YAML. Absent file = unset vars = receivers have no live
# destination, which is fine for a local stack. See .env.alerting.example.
env_file:
- path: .env.alerting
required: false
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true # No login required for local dev
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin # Full access without auth
@@ -102,6 +109,12 @@ services:
# the callback URL is how the renderer fetches the panel from grafana.
- GF_RENDERING_SERVER_URL=http://renderer:8081/render
- GF_RENDERING_CALLBACK_URL=http://grafana:3000/
# SMTP for the critical-tier email receiver and the Slack webhook / email
# address for the contact points all come from the env_file above, which
# injects them straight into the container environment for Grafana to
# expand. They are deliberately NOT duplicated here: a compose-level
# `environment:` entry is interpolated from the host shell and would
# override (and blank out) the env_file values.
ports:
- "3000:3000" # Grafana web UI
volumes:

54
docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml
View File

@@ -2,27 +2,53 @@
#
# Phase 9: Internal metric gap fill — alerting on health-critical metrics.
#
# A contact point is where a firing alert is delivered. This stack ships a
# single webhook receiver as a working default for local development. To route
# alerts to a real destination, replace the webhook `url` below, or add a
# `slack` / `email` receiver to the same contact point (see ALERTING.md).
# A contact point is where a firing alert is delivered. Two are defined:
# xrpld-default — Slack only; receives warning-severity alerts.
# xrpld-critical — Slack + email; receives critical-severity alerts.
# The severity split is wired in policies.yaml.
#
# The default anonymous-admin Grafana in docker-compose.yml picks this file up
# from /etc/grafana/provisioning/alerting/ on startup.
# Secrets and personal addresses are NOT hard-coded. Grafana expands
# ${ENV_VAR} references in provisioning files at load time, so the real
# Slack webhook and alert-email addresses come from environment variables
# supplied via docker/telemetry/.env.alerting (gitignored). See
# .env.alerting.example for the variable list and the Alerting section of
# docs/telemetry-runbook.md for setup.
# If a variable is unset, that receiver has no live destination.
#
# Email delivery additionally requires SMTP configured on the Grafana
# service (GF_SMTP_* in docker-compose.yml).
apiVersion: 1
contactPoints:
# --- Warning tier: Slack only ---
- orgId: 1
name: xrpld-default
receivers:
# Local-dev webhook sink. Points at a placeholder host; alerts are
# emitted but silently dropped unless a receiver listens there.
# Swap the url for a real endpoint (PagerDuty/Opsgenie/custom) or
# replace the whole receiver block with a slack/email type.
- uid: xrpld-default-webhook
type: webhook
- uid: xrpld-slack-default
type: slack
settings:
url: http://localhost:9999/xrpld-alerts
httpMethod: POST
# Incoming-webhook delivery: the channel is fixed by the webhook, so
# `recipient` is only here to satisfy Grafana's Slack validator.
url: ${SLACK_WEBHOOK_URL}
recipient: ${SLACK_CHANNEL}
title: "{{ .CommonLabels.alertname }} on {{ .CommonLabels.exported_instance }}"
disableResolveMessage: false
# --- Critical tier: Slack + email ---
- orgId: 1
name: xrpld-critical
receivers:
- uid: xrpld-slack-critical
type: slack
settings:
url: ${SLACK_WEBHOOK_URL}
recipient: ${SLACK_CHANNEL}
title: "[CRITICAL] {{ .CommonLabels.alertname }} on {{ .CommonLabels.exported_instance }}"
disableResolveMessage: false
- uid: xrpld-email-critical
type: email
settings:
addresses: ${ALERT_EMAIL_TO}
singleEmail: true
disableResolveMessage: false

20
docker/telemetry/grafana/provisioning/alerting/policies.yaml
View File

@@ -3,8 +3,11 @@
# Phase 9: Internal metric gap fill — alerting on health-critical metrics.
#
# The notification policy tree decides which contact point receives a firing
# alert and how alerts are batched. This stack uses a single flat policy: every
# xrpld alert routes to the `xrpld-default` contact point.
# alert and how alerts are batched. Routing is split by severity:
# - default (root) → xrpld-default (Slack) — catches warnings.
# - severity = critical → xrpld-critical (Slack + email) — child route.
# A critical alert matches the child route and is delivered to Slack AND email;
# everything else falls through to the root receiver.
#
# Grouping by alertname + exported_instance means one notification per
# (alert, node) pair, so a fleet-wide problem does not collapse into a single
@@ -26,3 +29,16 @@ policies:
# Minimum time before re-sending a notification for an alert that is still
# firing.
repeat_interval: 4h
routes:
# Critical alerts page harder: Slack + email. matchers uses the
# PromQL-style label match "severity = critical" set on the rules.
- receiver: xrpld-critical
matchers:
- severity = critical
group_by:
- alertname
- exported_instance
group_wait: 30s
group_interval: 5m
# Re-page critical alerts more often than the 4h default while still firing.
repeat_interval: 1h

3
docker/telemetry/grafana/provisioning/alerting/rules.yaml
View File

@@ -14,7 +14,8 @@
# B reduce (last) — collapse A's series to its most recent value.
# C threshold — the firing condition; `condition: C`.
#
# Thresholds are documented in ALERTING.md and are intended to be tuned.
# Thresholds are documented in docs/telemetry-runbook.md (Alerting section)
# and are intended to be tuned.
apiVersion: 1

129
docs/telemetry-runbook.md
View File

@@ -888,14 +888,17 @@ Requires `trace_peer=1` in the `[telemetry]` config section.
## Alerting
Grafana ships six provisioned alert rules on the health-critical metrics, so a
stock stack alerts out of the box with no UI setup. Rules are loaded from
`docker/telemetry/grafana/provisioning/alerting/` on startup and appear under
**Alerting → Alert rules**, folder **xrpld**.
rippled provisions six Grafana alert rules on the health-critical metrics, so a
stock stack alerts out of the box with no UI setup. Rules are provisioned from
`docker/telemetry/grafana/provisioning/alerting/` and load automatically when
the Grafana container starts. They appear under **Alerting → Alert rules**,
folder **xrpld**.
Each rule runs every minute against the Prometheus datasource over a 5-minute
window and groups by `exported_instance`, so every node alerts independently.
An alert fires only after its condition holds for the `for` dwell time.
### Alert catalogue
All rules evaluate every minute against the Prometheus datasource, over a
5-minute window, and group by `exported_instance` so each node alerts on its
own. Alerts fire only after the condition holds for the `for` dwell time.
| Alert | Severity | Fires when | For |
| ----------------------- | -------- | ----------------------------------------------- | --- |
@@ -906,11 +909,113 @@ An alert fires only after its condition holds for the `for` dwell time.
| `JobQueueTxOverflow` | warning | `rate(xrpld_jq_trans_overflow_total)` > 0 | 5m |
| `JobQueueLatencyHigh` | warning | p99 `xrpld_job_queued_duration_us` > 1s | 5m |
Alerts route through the `xrpld-default` contact point, which ships as a
local-dev webhook to a placeholder URL — alerts fire but go nowhere until it is
pointed at a real receiver. See
[ALERTING.md](../docker/telemetry/ALERTING.md) for per-alert causes, threshold
tuning, and how to wire alerts to Slack/email/PagerDuty.
#### Consensus / ledger health
**LedgerHistoryMismatch** — The node closed a ledger whose history diverges
from the validated network chain. Likely causes: corrupted local state, a bug,
or a node that fell out of sync and rebuilt incorrectly. Investigate the node's
ledger acquisition logs; a healthy node never mismatches.
**LedgerCloseStalled** — No ledgers closed for 3 minutes. A healthy node closes
one every ~3-5s. Likely causes: lost peer connectivity, consensus stall, or the
process is hung. This rule also fires on _NoData_ — if the series disappears the
node is likely down. Check peer count and process health first.
#### Validator health
**ValidationsMissed** — This validator's validations are not agreeing with the
validated ledger. Sustained misses risk removal from UNLs. Check clock sync,
peer connectivity, and whether the node is keeping up with ledger close.
**ValidationsNotChecked** — The node has stopped checking incoming validations
from peers. Likely causes: overlay/peer disconnection or a stalled validation
pipeline. Fires on NoData as well.
#### Job queue / resource health
**JobQueueTxOverflow** — The transaction job queue is full and transactions are
being dropped. The node is shedding load it cannot process. Check CPU, the
`JobQueueLatencyHigh` alert, and offered load.
**JobQueueLatencyHigh** — p99 queue wait exceeds 1 second, i.e. jobs back up
before running. The node is saturated. Correlate with CPU and the Job Queue
dashboard.
### Tuning thresholds
Thresholds live in
`docker/telemetry/grafana/provisioning/alerting/rules.yaml` as the `params`
array of each rule's `C` (threshold) node. Common tunables:
- **`JobQueueLatencyHigh`** — `params: [1000000]` is 1 000 000 µs (1s). Lower
it for latency-sensitive deployments.
- **`LedgerCloseStalled` / `ValidationsNotChecked`** — use `lt` with a tiny
epsilon (`0.001`) rather than `0`, so floating-point rate noise near zero
does not suppress the alert.
Edit the file and restart the Grafana container to reload:
```bash
docker compose -f docker/telemetry/docker-compose.yml restart grafana
```
### Sending alerts somewhere real
Two contact points are provisioned in
`docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml`:
| Contact point | Receivers | Gets |
| ---------------- | ------------- | ------------------------ |
| `xrpld-default` | Slack | warning-severity alerts |
| `xrpld-critical` | Slack + email | critical-severity alerts |
The severity split lives in
`docker/telemetry/grafana/provisioning/alerting/policies.yaml`: the root route
sends everything to `xrpld-default`, and a child route matching
`severity = critical` overrides to `xrpld-critical`. So a critical alert goes
to Slack **and** email; a warning goes to Slack only. Both group by
`alertname` + `exported_instance`; critical alerts re-page hourly vs the 4h default.
#### Configure delivery (no secrets in git)
The Slack webhook and email address are **not** hard-coded — the YAML
references `${SLACK_WEBHOOK_URL}` and `${ALERT_EMAIL_TO}`, which Grafana
expands from the environment at startup. Supply them through a gitignored
env file:
```bash
cp docker/telemetry/.env.alerting.example docker/telemetry/.env.alerting
# edit .env.alerting — this file is gitignored, never commit the webhook/address
docker compose -f docker/telemetry/docker-compose.yml up -d grafana
```
- **Slack** — set `SLACK_WEBHOOK_URL` to an incoming-webhook URL. Drives both
tiers.
- **Email** — set `ALERT_EMAIL_TO` (comma-separated) **and** point the
`GF_SMTP_*` vars at a real relay with `GF_SMTP_ENABLED=true`. Grafana can
only send mail once SMTP is configured.
Any variable left blank disables that path; the stack still runs. To add a
third destination (PagerDuty, Opsgenie, a custom webhook), add a receiver to
the relevant contact point.
### Verifying alert provisioning loaded
After the stack is up:
```bash
# All six rules present?
curl -s http://localhost:3000/api/v1/provisioning/alert-rules | jq '.[].title'
# Contact points present?
curl -s http://localhost:3000/api/v1/provisioning/contact-points | jq '.[].name'
```
Grafana logs a provisioning error and skips the file if the YAML is malformed:
```bash
docker compose -f docker/telemetry/docker-compose.yml logs grafana | grep -i alerting
```
## Log-Trace Correlation