diff --git a/.gitignore b/.gitignore index de59dd3712..6bd34ece04 100644 --- a/.gitignore +++ b/.gitignore @@ -86,6 +86,3 @@ __pycache__ # clangd cache /.cache - -# Env. file carrying environmental setup data for local or cloud runs. -.env.* diff --git a/docker/telemetry/.env.alerting.example b/docker/telemetry/.env.alerting.example new file mode 100644 index 0000000000..9822c635f1 --- /dev/null +++ b/docker/telemetry/.env.alerting.example @@ -0,0 +1,32 @@ +# rippled OTel alerting delivery — copy to `.env.alerting` and fill in. +# +# `.env.alerting` is gitignored; never commit a real Slack webhook or address. +# Grafana reads these when the stack starts and expands the ${VARS} referenced +# in grafana/provisioning/alerting/contactpoints.yaml. See the Alerting +# section of docs/telemetry-runbook.md. +# +# Any var left blank simply disables that delivery path — the stack still runs. + +# --- Slack --- +# Incoming-webhook URL from the Slack app (Incoming Webhooks feature). +# Used by both the warning (xrpld-default) and critical (xrpld-critical) tiers. +SLACK_WEBHOOK_URL= + +# Channel label. With an incoming webhook the target channel is fixed by the +# webhook itself; this only satisfies Grafana's Slack validator. Defaults to +# #xrpld-alerts if unset. +SLACK_CHANNEL=#xrpld-alerts + +# --- Email (critical tier only) --- +# Comma-separated recipient list for critical alerts. +ALERT_EMAIL_TO= + +# SMTP relay Grafana sends through. Email only delivers when SMTP is enabled +# and these point at a real relay (e.g. an internal smart host or an +# authenticated provider). +GF_SMTP_ENABLED=false +GF_SMTP_HOST=smtp.example.com:587 +GF_SMTP_USER= +GF_SMTP_PASSWORD= +GF_SMTP_FROM_ADDRESS=alerts@xrpld.local +GF_SMTP_FROM_NAME=xrpld Alerts diff --git a/docker/telemetry/.gitignore b/docker/telemetry/.gitignore index bba4819d66..047abc6702 100644 --- a/docker/telemetry/.gitignore +++ b/docker/telemetry/.gitignore @@ -1,2 +1,8 @@ # Runtime data generated by xrpld and telemetry stack data/ + +# Env. file carrying environmental setup data for local or cloud runs. +.env.* + +# Keep examples +!.env.alerting.example diff --git a/docker/telemetry/ALERTING.md b/docker/telemetry/ALERTING.md deleted file mode 100644 index ceaeaadaf5..0000000000 --- a/docker/telemetry/ALERTING.md +++ /dev/null @@ -1,133 +0,0 @@ -# rippled OpenTelemetry Alerting Runbook - -rippled exports its internal metrics and provisions Grafana alert rules on the -health-critical ones. This document explains each alert, its likely cause, and -how to point alerts at a real receiver. - -The rules are provisioned from -`grafana/provisioning/alerting/` and load automatically when the Grafana -container in [docker-compose.yml](docker-compose.yml) starts — no manual setup -in the Grafana UI. Rules appear in Grafana under **Alerting → Alert rules**, -folder **xrpld**. - ---- - -## Alert catalogue - -All rules evaluate every minute against the Prometheus datasource, over a -5-minute window, and group by `exported_instance` so each node alerts on its -own. Alerts fire only after the condition holds for the `for` dwell time. - -| Alert | Severity | Fires when | For | -| ----------------------- | -------- | ----------------------------------------------- | --- | -| `LedgerHistoryMismatch` | critical | `rate(xrpld_ledger_history_mismatch_total)` > 0 | 5m | -| `LedgerCloseStalled` | critical | `rate(xrpld_ledgers_closed_total)` ≈ 0 | 3m | -| `ValidationsMissed` | warning | `rate(xrpld_validation_missed_total)` > 0 | 5m | -| `ValidationsNotChecked` | warning | `rate(xrpld_validations_checked_total)` ≈ 0 | 5m | -| `JobQueueTxOverflow` | warning | `rate(xrpld_jq_trans_overflow_total)` > 0 | 5m | -| `JobQueueLatencyHigh` | warning | p99 `xrpld_job_queued_duration_us` > 1s | 5m | - -### Consensus / ledger health - -**LedgerHistoryMismatch** — The node closed a ledger whose history diverges -from the validated network chain. Likely causes: corrupted local state, a bug, -or a node that fell out of sync and rebuilt incorrectly. Investigate the node's -ledger acquisition logs; a healthy node never mismatches. - -**LedgerCloseStalled** — No ledgers closed for 3 minutes. A healthy node closes -one every ~3-5s. Likely causes: lost peer connectivity, consensus stall, or the -process is hung. This rule also fires on _NoData_ — if the series disappears the -node is likely down. Check peer count and process health first. - -### Validator health - -**ValidationsMissed** — This validator's validations are not agreeing with the -validated ledger. Sustained misses risk removal from UNLs. Check clock sync, -peer connectivity, and whether the node is keeping up with ledger close. - -**ValidationsNotChecked** — The node has stopped checking incoming validations -from peers. Likely causes: overlay/peer disconnection or a stalled validation -pipeline. Fires on NoData as well. - -### Job queue / resource health - -**JobQueueTxOverflow** — The transaction job queue is full and transactions are -being dropped. The node is shedding load it cannot process. Check CPU, the -`JobQueueLatencyHigh` alert, and offered load. - -**JobQueueLatencyHigh** — p99 queue wait exceeds 1 second, i.e. jobs back up -before running. The node is saturated. Correlate with CPU and the Job Queue -dashboard. - ---- - -## Tuning thresholds - -Thresholds live in -[grafana/provisioning/alerting/rules.yaml](grafana/provisioning/alerting/rules.yaml) -as the `params` array of each rule's `C` (threshold) node. Common tunables: - -- **`JobQueueLatencyHigh`** — `params: [1000000]` is 1 000 000 µs (1s). Lower - it for latency-sensitive deployments. -- **`LedgerCloseStalled` / `ValidationsNotChecked`** — use `lt` with a tiny - epsilon (`0.001`) rather than `0`, so floating-point rate noise near zero - does not suppress the alert. - -Edit the file and restart the Grafana container to reload: - -```bash -docker compose -f docker/telemetry/docker-compose.yml restart grafana -``` - ---- - -## Sending alerts somewhere real - -The provisioned contact point `xrpld-default` -([contactpoints.yaml](grafana/provisioning/alerting/contactpoints.yaml)) ships -as a **local-dev webhook** to a placeholder URL — alerts fire but go nowhere -until you change it. - -**Option A — repoint the webhook.** Replace the `url` under the `webhook` -receiver with a real endpoint (PagerDuty Events API, Opsgenie, a custom sink). - -**Option B — add a Slack receiver.** Add a second receiver to the same contact -point: - -```yaml -- uid: xrpld-slack - type: slack - settings: - url: https://hooks.slack.com/services/XXX/YYY/ZZZ - title: "{{ .CommonLabels.alertname }} on {{ .CommonLabels.exported_instance }}" -``` - -**Option C — email.** Requires Grafana SMTP configured via `GF_SMTP_*` -environment variables on the Grafana service in `docker-compose.yml`, then an -`email` receiver with `addresses`. - -Routing is a single flat policy in -[policies.yaml](grafana/provisioning/alerting/policies.yaml): all alerts → -`xrpld-default`, grouped by `alertname` + `exported_instance`. To route -critical alerts to a different receiver, add child routes matching -`severity = critical`. - ---- - -## Verifying provisioning loaded - -After the stack is up: - -```bash -# All six rules present? -curl -s http://localhost:3000/api/v1/provisioning/alert-rules | jq '.[].title' - -# Contact point present? -curl -s http://localhost:3000/api/v1/provisioning/contact-points | jq '.[].name' -``` - -Grafana logs a provisioning error and skips the file if the YAML is malformed: - -```bash -docker compose -f docker/telemetry/docker-compose.yml logs grafana | grep -i alerting -``` diff --git a/docker/telemetry/docker-compose.yml b/docker/telemetry/docker-compose.yml index 28ee64336c..1ee95734c1 100644 --- a/docker/telemetry/docker-compose.yml +++ b/docker/telemetry/docker-compose.yml @@ -94,6 +94,13 @@ services: # Anonymous admin access enabled for local development convenience. grafana: image: grafana/grafana:11.5.2 + # Alerting secrets/addresses (Slack webhook, alert email) come from this + # gitignored file; Grafana expands the ${VARS} referenced in the alerting + # provisioning YAML. Absent file = unset vars = receivers have no live + # destination, which is fine for a local stack. See .env.alerting.example. + env_file: + - path: .env.alerting + required: false environment: - GF_AUTH_ANONYMOUS_ENABLED=true # No login required for local dev - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin # Full access without auth @@ -102,6 +109,12 @@ services: # the callback URL is how the renderer fetches the panel from grafana. - GF_RENDERING_SERVER_URL=http://renderer:8081/render - GF_RENDERING_CALLBACK_URL=http://grafana:3000/ + # SMTP for the critical-tier email receiver and the Slack webhook / email + # address for the contact points all come from the env_file above, which + # injects them straight into the container environment for Grafana to + # expand. They are deliberately NOT duplicated here: a compose-level + # `environment:` entry is interpolated from the host shell and would + # override (and blank out) the env_file values. ports: - "3000:3000" # Grafana web UI volumes: diff --git a/docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml b/docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml index 2cd7f47fd1..a0c1694172 100644 --- a/docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml +++ b/docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml @@ -2,27 +2,53 @@ # # Phase 9: Internal metric gap fill — alerting on health-critical metrics. # -# A contact point is where a firing alert is delivered. This stack ships a -# single webhook receiver as a working default for local development. To route -# alerts to a real destination, replace the webhook `url` below, or add a -# `slack` / `email` receiver to the same contact point (see ALERTING.md). +# A contact point is where a firing alert is delivered. Two are defined: +# xrpld-default — Slack only; receives warning-severity alerts. +# xrpld-critical — Slack + email; receives critical-severity alerts. +# The severity split is wired in policies.yaml. # -# The default anonymous-admin Grafana in docker-compose.yml picks this file up -# from /etc/grafana/provisioning/alerting/ on startup. +# Secrets and personal addresses are NOT hard-coded. Grafana expands +# ${ENV_VAR} references in provisioning files at load time, so the real +# Slack webhook and alert-email addresses come from environment variables +# supplied via docker/telemetry/.env.alerting (gitignored). See +# .env.alerting.example for the variable list and the Alerting section of +# docs/telemetry-runbook.md for setup. +# If a variable is unset, that receiver has no live destination. +# +# Email delivery additionally requires SMTP configured on the Grafana +# service (GF_SMTP_* in docker-compose.yml). apiVersion: 1 contactPoints: + # --- Warning tier: Slack only --- - orgId: 1 name: xrpld-default receivers: - # Local-dev webhook sink. Points at a placeholder host; alerts are - # emitted but silently dropped unless a receiver listens there. - # Swap the url for a real endpoint (PagerDuty/Opsgenie/custom) or - # replace the whole receiver block with a slack/email type. - - uid: xrpld-default-webhook - type: webhook + - uid: xrpld-slack-default + type: slack settings: - url: http://localhost:9999/xrpld-alerts - httpMethod: POST + # Incoming-webhook delivery: the channel is fixed by the webhook, so + # `recipient` is only here to satisfy Grafana's Slack validator. + url: ${SLACK_WEBHOOK_URL} + recipient: ${SLACK_CHANNEL} + title: "{{ .CommonLabels.alertname }} on {{ .CommonLabels.exported_instance }}" + disableResolveMessage: false + + # --- Critical tier: Slack + email --- + - orgId: 1 + name: xrpld-critical + receivers: + - uid: xrpld-slack-critical + type: slack + settings: + url: ${SLACK_WEBHOOK_URL} + recipient: ${SLACK_CHANNEL} + title: "[CRITICAL] {{ .CommonLabels.alertname }} on {{ .CommonLabels.exported_instance }}" + disableResolveMessage: false + - uid: xrpld-email-critical + type: email + settings: + addresses: ${ALERT_EMAIL_TO} + singleEmail: true disableResolveMessage: false diff --git a/docker/telemetry/grafana/provisioning/alerting/policies.yaml b/docker/telemetry/grafana/provisioning/alerting/policies.yaml index 5ef96a5a9a..af2932f0b8 100644 --- a/docker/telemetry/grafana/provisioning/alerting/policies.yaml +++ b/docker/telemetry/grafana/provisioning/alerting/policies.yaml @@ -3,8 +3,11 @@ # Phase 9: Internal metric gap fill — alerting on health-critical metrics. # # The notification policy tree decides which contact point receives a firing -# alert and how alerts are batched. This stack uses a single flat policy: every -# xrpld alert routes to the `xrpld-default` contact point. +# alert and how alerts are batched. Routing is split by severity: +# - default (root) → xrpld-default (Slack) — catches warnings. +# - severity = critical → xrpld-critical (Slack + email) — child route. +# A critical alert matches the child route and is delivered to Slack AND email; +# everything else falls through to the root receiver. # # Grouping by alertname + exported_instance means one notification per # (alert, node) pair, so a fleet-wide problem does not collapse into a single @@ -26,3 +29,16 @@ policies: # Minimum time before re-sending a notification for an alert that is still # firing. repeat_interval: 4h + routes: + # Critical alerts page harder: Slack + email. matchers uses the + # PromQL-style label match "severity = critical" set on the rules. + - receiver: xrpld-critical + matchers: + - severity = critical + group_by: + - alertname + - exported_instance + group_wait: 30s + group_interval: 5m + # Re-page critical alerts more often than the 4h default while still firing. + repeat_interval: 1h diff --git a/docker/telemetry/grafana/provisioning/alerting/rules.yaml b/docker/telemetry/grafana/provisioning/alerting/rules.yaml index 6b69059720..6617f52788 100644 --- a/docker/telemetry/grafana/provisioning/alerting/rules.yaml +++ b/docker/telemetry/grafana/provisioning/alerting/rules.yaml @@ -14,7 +14,8 @@ # B reduce (last) — collapse A's series to its most recent value. # C threshold — the firing condition; `condition: C`. # -# Thresholds are documented in ALERTING.md and are intended to be tuned. +# Thresholds are documented in docs/telemetry-runbook.md (Alerting section) +# and are intended to be tuned. apiVersion: 1 diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index a9cb25cae1..4344c5c012 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -888,14 +888,17 @@ Requires `trace_peer=1` in the `[telemetry]` config section. ## Alerting -Grafana ships six provisioned alert rules on the health-critical metrics, so a -stock stack alerts out of the box with no UI setup. Rules are loaded from -`docker/telemetry/grafana/provisioning/alerting/` on startup and appear under -**Alerting → Alert rules**, folder **xrpld**. +rippled provisions six Grafana alert rules on the health-critical metrics, so a +stock stack alerts out of the box with no UI setup. Rules are provisioned from +`docker/telemetry/grafana/provisioning/alerting/` and load automatically when +the Grafana container starts. They appear under **Alerting → Alert rules**, +folder **xrpld**. -Each rule runs every minute against the Prometheus datasource over a 5-minute -window and groups by `exported_instance`, so every node alerts independently. -An alert fires only after its condition holds for the `for` dwell time. +### Alert catalogue + +All rules evaluate every minute against the Prometheus datasource, over a +5-minute window, and group by `exported_instance` so each node alerts on its +own. Alerts fire only after the condition holds for the `for` dwell time. | Alert | Severity | Fires when | For | | ----------------------- | -------- | ----------------------------------------------- | --- | @@ -906,11 +909,113 @@ An alert fires only after its condition holds for the `for` dwell time. | `JobQueueTxOverflow` | warning | `rate(xrpld_jq_trans_overflow_total)` > 0 | 5m | | `JobQueueLatencyHigh` | warning | p99 `xrpld_job_queued_duration_us` > 1s | 5m | -Alerts route through the `xrpld-default` contact point, which ships as a -local-dev webhook to a placeholder URL — alerts fire but go nowhere until it is -pointed at a real receiver. See -[ALERTING.md](../docker/telemetry/ALERTING.md) for per-alert causes, threshold -tuning, and how to wire alerts to Slack/email/PagerDuty. +#### Consensus / ledger health + +**LedgerHistoryMismatch** — The node closed a ledger whose history diverges +from the validated network chain. Likely causes: corrupted local state, a bug, +or a node that fell out of sync and rebuilt incorrectly. Investigate the node's +ledger acquisition logs; a healthy node never mismatches. + +**LedgerCloseStalled** — No ledgers closed for 3 minutes. A healthy node closes +one every ~3-5s. Likely causes: lost peer connectivity, consensus stall, or the +process is hung. This rule also fires on _NoData_ — if the series disappears the +node is likely down. Check peer count and process health first. + +#### Validator health + +**ValidationsMissed** — This validator's validations are not agreeing with the +validated ledger. Sustained misses risk removal from UNLs. Check clock sync, +peer connectivity, and whether the node is keeping up with ledger close. + +**ValidationsNotChecked** — The node has stopped checking incoming validations +from peers. Likely causes: overlay/peer disconnection or a stalled validation +pipeline. Fires on NoData as well. + +#### Job queue / resource health + +**JobQueueTxOverflow** — The transaction job queue is full and transactions are +being dropped. The node is shedding load it cannot process. Check CPU, the +`JobQueueLatencyHigh` alert, and offered load. + +**JobQueueLatencyHigh** — p99 queue wait exceeds 1 second, i.e. jobs back up +before running. The node is saturated. Correlate with CPU and the Job Queue +dashboard. + +### Tuning thresholds + +Thresholds live in +`docker/telemetry/grafana/provisioning/alerting/rules.yaml` as the `params` +array of each rule's `C` (threshold) node. Common tunables: + +- **`JobQueueLatencyHigh`** — `params: [1000000]` is 1 000 000 µs (1s). Lower + it for latency-sensitive deployments. +- **`LedgerCloseStalled` / `ValidationsNotChecked`** — use `lt` with a tiny + epsilon (`0.001`) rather than `0`, so floating-point rate noise near zero + does not suppress the alert. + +Edit the file and restart the Grafana container to reload: + +```bash +docker compose -f docker/telemetry/docker-compose.yml restart grafana +``` + +### Sending alerts somewhere real + +Two contact points are provisioned in +`docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml`: + +| Contact point | Receivers | Gets | +| ---------------- | ------------- | ------------------------ | +| `xrpld-default` | Slack | warning-severity alerts | +| `xrpld-critical` | Slack + email | critical-severity alerts | + +The severity split lives in +`docker/telemetry/grafana/provisioning/alerting/policies.yaml`: the root route +sends everything to `xrpld-default`, and a child route matching +`severity = critical` overrides to `xrpld-critical`. So a critical alert goes +to Slack **and** email; a warning goes to Slack only. Both group by +`alertname` + `exported_instance`; critical alerts re-page hourly vs the 4h default. + +#### Configure delivery (no secrets in git) + +The Slack webhook and email address are **not** hard-coded — the YAML +references `${SLACK_WEBHOOK_URL}` and `${ALERT_EMAIL_TO}`, which Grafana +expands from the environment at startup. Supply them through a gitignored +env file: + +```bash +cp docker/telemetry/.env.alerting.example docker/telemetry/.env.alerting +# edit .env.alerting — this file is gitignored, never commit the webhook/address +docker compose -f docker/telemetry/docker-compose.yml up -d grafana +``` + +- **Slack** — set `SLACK_WEBHOOK_URL` to an incoming-webhook URL. Drives both + tiers. +- **Email** — set `ALERT_EMAIL_TO` (comma-separated) **and** point the + `GF_SMTP_*` vars at a real relay with `GF_SMTP_ENABLED=true`. Grafana can + only send mail once SMTP is configured. + +Any variable left blank disables that path; the stack still runs. To add a +third destination (PagerDuty, Opsgenie, a custom webhook), add a receiver to +the relevant contact point. + +### Verifying alert provisioning loaded + +After the stack is up: + +```bash +# All six rules present? +curl -s http://localhost:3000/api/v1/provisioning/alert-rules | jq '.[].title' + +# Contact points present? +curl -s http://localhost:3000/api/v1/provisioning/contact-points | jq '.[].name' +``` + +Grafana logs a provisioning error and skips the file if the YAML is malformed: + +```bash +docker compose -f docker/telemetry/docker-compose.yml logs grafana | grep -i alerting +``` ## Log-Trace Correlation