mirror of
https://github.com/XRPLF/rippled.git
synced 2026-07-27 00:50:45 +00:00
Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-phase11-telemetry-off
This commit is contained in:
133
docker/telemetry/ALERTING.md
Normal file
133
docker/telemetry/ALERTING.md
Normal file
@@ -0,0 +1,133 @@
|
||||
# rippled OpenTelemetry Alerting Runbook
|
||||
|
||||
Phase 9 exports rippled's internal metrics and provisions Grafana alert rules
|
||||
on the health-critical ones. This document explains each alert, its likely
|
||||
cause, and how to point alerts at a real receiver.
|
||||
|
||||
The rules are provisioned from
|
||||
`grafana/provisioning/alerting/` and load automatically when the Grafana
|
||||
container in [docker-compose.yml](docker-compose.yml) starts — no manual setup
|
||||
in the Grafana UI. Rules appear in Grafana under **Alerting → Alert rules**,
|
||||
folder **xrpld**.
|
||||
|
||||
---
|
||||
|
||||
## Alert catalogue
|
||||
|
||||
All rules evaluate every minute against the Prometheus datasource, over a
|
||||
5-minute window, and group by `exported_instance` so each node alerts on its
|
||||
own. Alerts fire only after the condition holds for the `for` dwell time.
|
||||
|
||||
| Alert | Severity | Fires when | For |
|
||||
| ----------------------- | -------- | ----------------------------------------------- | --- |
|
||||
| `LedgerHistoryMismatch` | critical | `rate(xrpld_ledger_history_mismatch_total)` > 0 | 5m |
|
||||
| `LedgerCloseStalled` | critical | `rate(xrpld_ledgers_closed_total)` ≈ 0 | 3m |
|
||||
| `ValidationsMissed` | warning | `rate(xrpld_validation_missed_total)` > 0 | 5m |
|
||||
| `ValidationsNotChecked` | warning | `rate(xrpld_validations_checked_total)` ≈ 0 | 5m |
|
||||
| `JobQueueTxOverflow` | warning | `rate(xrpld_jq_trans_overflow_total)` > 0 | 5m |
|
||||
| `JobQueueLatencyHigh` | warning | p99 `xrpld_job_queued_duration_us` > 1s | 5m |
|
||||
|
||||
### Consensus / ledger health
|
||||
|
||||
**LedgerHistoryMismatch** — The node closed a ledger whose history diverges
|
||||
from the validated network chain. Likely causes: corrupted local state, a bug,
|
||||
or a node that fell out of sync and rebuilt incorrectly. Investigate the node's
|
||||
ledger acquisition logs; a healthy node never mismatches.
|
||||
|
||||
**LedgerCloseStalled** — No ledgers closed for 3 minutes. A healthy node closes
|
||||
one every ~3-5s. Likely causes: lost peer connectivity, consensus stall, or the
|
||||
process is hung. This rule also fires on _NoData_ — if the series disappears the
|
||||
node is likely down. Check peer count and process health first.
|
||||
|
||||
### Validator health
|
||||
|
||||
**ValidationsMissed** — This validator's validations are not agreeing with the
|
||||
validated ledger. Sustained misses risk removal from UNLs. Check clock sync,
|
||||
peer connectivity, and whether the node is keeping up with ledger close.
|
||||
|
||||
**ValidationsNotChecked** — The node has stopped checking incoming validations
|
||||
from peers. Likely causes: overlay/peer disconnection or a stalled validation
|
||||
pipeline. Fires on NoData as well.
|
||||
|
||||
### Job queue / resource health
|
||||
|
||||
**JobQueueTxOverflow** — The transaction job queue is full and transactions are
|
||||
being dropped. The node is shedding load it cannot process. Check CPU, the
|
||||
`JobQueueLatencyHigh` alert, and offered load.
|
||||
|
||||
**JobQueueLatencyHigh** — p99 queue wait exceeds 1 second, i.e. jobs back up
|
||||
before running. The node is saturated. Correlate with CPU and the Job Queue
|
||||
dashboard.
|
||||
|
||||
---
|
||||
|
||||
## Tuning thresholds
|
||||
|
||||
Thresholds live in
|
||||
[grafana/provisioning/alerting/rules.yaml](grafana/provisioning/alerting/rules.yaml)
|
||||
as the `params` array of each rule's `C` (threshold) node. Common tunables:
|
||||
|
||||
- **`JobQueueLatencyHigh`** — `params: [1000000]` is 1 000 000 µs (1s). Lower
|
||||
it for latency-sensitive deployments.
|
||||
- **`LedgerCloseStalled` / `ValidationsNotChecked`** — use `lt` with a tiny
|
||||
epsilon (`0.001`) rather than `0`, so floating-point rate noise near zero
|
||||
does not suppress the alert.
|
||||
|
||||
Edit the file and restart the Grafana container to reload:
|
||||
|
||||
```bash
|
||||
docker compose -f docker/telemetry/docker-compose.yml restart grafana
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Sending alerts somewhere real
|
||||
|
||||
The provisioned contact point `xrpld-default`
|
||||
([contactpoints.yaml](grafana/provisioning/alerting/contactpoints.yaml)) ships
|
||||
as a **local-dev webhook** to a placeholder URL — alerts fire but go nowhere
|
||||
until you change it.
|
||||
|
||||
**Option A — repoint the webhook.** Replace the `url` under the `webhook`
|
||||
receiver with a real endpoint (PagerDuty Events API, Opsgenie, a custom sink).
|
||||
|
||||
**Option B — add a Slack receiver.** Add a second receiver to the same contact
|
||||
point:
|
||||
|
||||
```yaml
|
||||
- uid: xrpld-slack
|
||||
type: slack
|
||||
settings:
|
||||
url: https://hooks.slack.com/services/XXX/YYY/ZZZ
|
||||
title: "{{ .CommonLabels.alertname }} on {{ .CommonLabels.exported_instance }}"
|
||||
```
|
||||
|
||||
**Option C — email.** Requires Grafana SMTP configured via `GF_SMTP_*`
|
||||
environment variables on the Grafana service in `docker-compose.yml`, then an
|
||||
`email` receiver with `addresses`.
|
||||
|
||||
Routing is a single flat policy in
|
||||
[policies.yaml](grafana/provisioning/alerting/policies.yaml): all alerts →
|
||||
`xrpld-default`, grouped by `alertname` + `exported_instance`. To route
|
||||
critical alerts to a different receiver, add child routes matching
|
||||
`severity = critical`.
|
||||
|
||||
---
|
||||
|
||||
## Verifying provisioning loaded
|
||||
|
||||
After the stack is up:
|
||||
|
||||
```bash
|
||||
# All six rules present?
|
||||
curl -s http://localhost:3000/api/v1/provisioning/alert-rules | jq '.[].title'
|
||||
|
||||
# Contact point present?
|
||||
curl -s http://localhost:3000/api/v1/provisioning/contact-points | jq '.[].name'
|
||||
```
|
||||
|
||||
Grafana logs a provisioning error and skips the file if the YAML is malformed:
|
||||
|
||||
```bash
|
||||
docker compose -f docker/telemetry/docker-compose.yml logs grafana | grep -i alerting
|
||||
```
|
||||
@@ -0,0 +1,28 @@
|
||||
# Grafana contact-point provisioning for rippled OTel alerts.
|
||||
#
|
||||
# Phase 9: Internal metric gap fill — alerting on health-critical metrics.
|
||||
#
|
||||
# A contact point is where a firing alert is delivered. This stack ships a
|
||||
# single webhook receiver as a working default for local development. To route
|
||||
# alerts to a real destination, replace the webhook `url` below, or add a
|
||||
# `slack` / `email` receiver to the same contact point (see ALERTING.md).
|
||||
#
|
||||
# The default anonymous-admin Grafana in docker-compose.yml picks this file up
|
||||
# from /etc/grafana/provisioning/alerting/ on startup.
|
||||
|
||||
apiVersion: 1
|
||||
|
||||
contactPoints:
|
||||
- orgId: 1
|
||||
name: xrpld-default
|
||||
receivers:
|
||||
# Local-dev webhook sink. Points at a placeholder host; alerts are
|
||||
# emitted but silently dropped unless a receiver listens there.
|
||||
# Swap the url for a real endpoint (PagerDuty/Opsgenie/custom) or
|
||||
# replace the whole receiver block with a slack/email type.
|
||||
- uid: xrpld-default-webhook
|
||||
type: webhook
|
||||
settings:
|
||||
url: http://localhost:9999/xrpld-alerts
|
||||
httpMethod: POST
|
||||
disableResolveMessage: false
|
||||
28
docker/telemetry/grafana/provisioning/alerting/policies.yaml
Normal file
28
docker/telemetry/grafana/provisioning/alerting/policies.yaml
Normal file
@@ -0,0 +1,28 @@
|
||||
# Grafana notification-policy provisioning for rippled OTel alerts.
|
||||
#
|
||||
# Phase 9: Internal metric gap fill — alerting on health-critical metrics.
|
||||
#
|
||||
# The notification policy tree decides which contact point receives a firing
|
||||
# alert and how alerts are batched. This stack uses a single flat policy: every
|
||||
# xrpld alert routes to the `xrpld-default` contact point.
|
||||
#
|
||||
# Grouping by alertname + exported_instance means one notification per
|
||||
# (alert, node) pair, so a fleet-wide problem does not collapse into a single
|
||||
# ambiguous message.
|
||||
|
||||
apiVersion: 1
|
||||
|
||||
policies:
|
||||
- orgId: 1
|
||||
receiver: xrpld-default
|
||||
group_by:
|
||||
- alertname
|
||||
- exported_instance
|
||||
# Wait before sending the first notification for a new group, so a burst of
|
||||
# related series is batched together.
|
||||
group_wait: 30s
|
||||
# Wait before adding new alerts to an already-notified group.
|
||||
group_interval: 5m
|
||||
# Minimum time before re-sending a notification for an alert that is still
|
||||
# firing.
|
||||
repeat_interval: 4h
|
||||
409
docker/telemetry/grafana/provisioning/alerting/rules.yaml
Normal file
409
docker/telemetry/grafana/provisioning/alerting/rules.yaml
Normal file
@@ -0,0 +1,409 @@
|
||||
# Grafana alert-rule provisioning for rippled OTel metrics.
|
||||
#
|
||||
# Phase 9: Internal metric gap fill — alerting on health-critical metrics.
|
||||
#
|
||||
# Six rules across three node-fatal subsystems: consensus/ledger health,
|
||||
# validator health, and the job queue. Every metric referenced here is
|
||||
# introduced by phase 9's MetricsRegistry.
|
||||
#
|
||||
# Rule shape (Grafana server-side evaluation):
|
||||
# A Prometheus query — a 5-minute rate / histogram_quantile, aggregated
|
||||
# `by (exported_instance)` so each node evaluates
|
||||
# independently. Alert rules run headless and cannot
|
||||
# use the dashboards' `$node` template variable.
|
||||
# B reduce (last) — collapse A's series to its most recent value.
|
||||
# C threshold — the firing condition; `condition: C`.
|
||||
#
|
||||
# Thresholds are documented in ALERTING.md and are intended to be tuned.
|
||||
|
||||
apiVersion: 1
|
||||
|
||||
groups:
|
||||
# ------------------------------------------------------------------ #
|
||||
# Consensus / ledger health #
|
||||
# ------------------------------------------------------------------ #
|
||||
- orgId: 1
|
||||
name: xrpld-consensus
|
||||
folder: xrpld
|
||||
interval: 1m
|
||||
rules:
|
||||
# A closed ledger that later fails validation against the network —
|
||||
# any sustained nonzero rate means this node built history the rest
|
||||
# of the network rejects.
|
||||
- uid: xrpld-ledger-history-mismatch
|
||||
title: LedgerHistoryMismatch
|
||||
condition: C
|
||||
for: 5m
|
||||
noDataState: NoData
|
||||
execErrState: Error
|
||||
labels:
|
||||
severity: critical
|
||||
category: consensus
|
||||
annotations:
|
||||
summary: "Ledger history mismatch on {{ $labels.exported_instance }}"
|
||||
description: >-
|
||||
Node {{ $labels.exported_instance }} is recording ledger history
|
||||
mismatches ({{ $values.B.Value }}/s over 5m). The node's built ledger
|
||||
diverges from the validated network chain.
|
||||
data:
|
||||
- refId: A
|
||||
relativeTimeRange:
|
||||
from: 600
|
||||
to: 0
|
||||
datasourceUid: prometheus
|
||||
model:
|
||||
refId: A
|
||||
expr: sum by (exported_instance) (rate(xrpld_ledger_history_mismatch_total[5m]))
|
||||
instant: true
|
||||
range: false
|
||||
intervalMs: 1000
|
||||
maxDataPoints: 43200
|
||||
- refId: B
|
||||
relativeTimeRange:
|
||||
from: 600
|
||||
to: 0
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
refId: B
|
||||
type: reduce
|
||||
reducer: last
|
||||
expression: A
|
||||
datasource:
|
||||
type: __expr__
|
||||
uid: __expr__
|
||||
- refId: C
|
||||
relativeTimeRange:
|
||||
from: 600
|
||||
to: 0
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
refId: C
|
||||
type: threshold
|
||||
expression: B
|
||||
conditions:
|
||||
- evaluator:
|
||||
type: gt
|
||||
params: [0]
|
||||
datasource:
|
||||
type: __expr__
|
||||
uid: __expr__
|
||||
|
||||
# Healthy nodes close a ledger every ~3-5s. Zero closes for 3 minutes
|
||||
# means consensus/ledger advancement is stuck. NoData (metric absent)
|
||||
# also fires — a vanished series here means the node is down.
|
||||
- uid: xrpld-ledger-close-stalled
|
||||
title: LedgerCloseStalled
|
||||
condition: C
|
||||
for: 3m
|
||||
noDataState: Alerting
|
||||
execErrState: Error
|
||||
labels:
|
||||
severity: critical
|
||||
category: consensus
|
||||
annotations:
|
||||
summary: "Ledger closing stalled on {{ $labels.exported_instance }}"
|
||||
description: >-
|
||||
Node {{ $labels.exported_instance }} has closed no ledgers for
|
||||
several minutes (5m rate has decayed to zero). Consensus or ledger
|
||||
advancement is stuck.
|
||||
data:
|
||||
- refId: A
|
||||
relativeTimeRange:
|
||||
from: 600
|
||||
to: 0
|
||||
datasourceUid: prometheus
|
||||
model:
|
||||
refId: A
|
||||
expr: sum by (exported_instance) (rate(xrpld_ledgers_closed_total[5m]))
|
||||
instant: true
|
||||
range: false
|
||||
intervalMs: 1000
|
||||
maxDataPoints: 43200
|
||||
- refId: B
|
||||
relativeTimeRange:
|
||||
from: 600
|
||||
to: 0
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
refId: B
|
||||
type: reduce
|
||||
reducer: last
|
||||
expression: A
|
||||
datasource:
|
||||
type: __expr__
|
||||
uid: __expr__
|
||||
- refId: C
|
||||
relativeTimeRange:
|
||||
from: 600
|
||||
to: 0
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
refId: C
|
||||
type: threshold
|
||||
expression: B
|
||||
conditions:
|
||||
- evaluator:
|
||||
type: lt
|
||||
params: [0.001]
|
||||
datasource:
|
||||
type: __expr__
|
||||
uid: __expr__
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# Validator health #
|
||||
# ------------------------------------------------------------------ #
|
||||
- orgId: 1
|
||||
name: xrpld-validator
|
||||
folder: xrpld
|
||||
interval: 1m
|
||||
rules:
|
||||
# This validator's own validations are not reaching / agreeing with
|
||||
# the network. A sustained nonzero miss rate risks the validator being
|
||||
# dropped from UNLs.
|
||||
- uid: xrpld-validations-missed
|
||||
title: ValidationsMissed
|
||||
condition: C
|
||||
for: 5m
|
||||
noDataState: NoData
|
||||
execErrState: Error
|
||||
labels:
|
||||
severity: warning
|
||||
category: validator
|
||||
annotations:
|
||||
summary: "Validations missed on {{ $labels.exported_instance }}"
|
||||
description: >-
|
||||
Node {{ $labels.exported_instance }} is missing validations
|
||||
({{ $values.B.Value }}/s over 5m). Its validations are not agreeing with
|
||||
the validated ledger.
|
||||
data:
|
||||
- refId: A
|
||||
relativeTimeRange:
|
||||
from: 600
|
||||
to: 0
|
||||
datasourceUid: prometheus
|
||||
model:
|
||||
refId: A
|
||||
expr: sum by (exported_instance) (rate(xrpld_validation_missed_total[5m]))
|
||||
instant: true
|
||||
range: false
|
||||
intervalMs: 1000
|
||||
maxDataPoints: 43200
|
||||
- refId: B
|
||||
relativeTimeRange:
|
||||
from: 600
|
||||
to: 0
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
refId: B
|
||||
type: reduce
|
||||
reducer: last
|
||||
expression: A
|
||||
datasource:
|
||||
type: __expr__
|
||||
uid: __expr__
|
||||
- refId: C
|
||||
relativeTimeRange:
|
||||
from: 600
|
||||
to: 0
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
refId: C
|
||||
type: threshold
|
||||
expression: B
|
||||
conditions:
|
||||
- evaluator:
|
||||
type: gt
|
||||
params: [0]
|
||||
datasource:
|
||||
type: __expr__
|
||||
uid: __expr__
|
||||
|
||||
# The node has stopped checking incoming validations. Zero checked
|
||||
# validations means it is no longer processing the validation stream
|
||||
# from peers.
|
||||
- uid: xrpld-validations-not-checked
|
||||
title: ValidationsNotChecked
|
||||
condition: C
|
||||
for: 5m
|
||||
noDataState: Alerting
|
||||
execErrState: Error
|
||||
labels:
|
||||
severity: warning
|
||||
category: validator
|
||||
annotations:
|
||||
summary: "No validations checked on {{ $labels.exported_instance }}"
|
||||
description: >-
|
||||
Node {{ $labels.exported_instance }} has checked no incoming
|
||||
validations for several minutes (5m rate has decayed to zero). The
|
||||
validation stream from peers may have stopped.
|
||||
data:
|
||||
- refId: A
|
||||
relativeTimeRange:
|
||||
from: 600
|
||||
to: 0
|
||||
datasourceUid: prometheus
|
||||
model:
|
||||
refId: A
|
||||
expr: sum by (exported_instance) (rate(xrpld_validations_checked_total[5m]))
|
||||
instant: true
|
||||
range: false
|
||||
intervalMs: 1000
|
||||
maxDataPoints: 43200
|
||||
- refId: B
|
||||
relativeTimeRange:
|
||||
from: 600
|
||||
to: 0
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
refId: B
|
||||
type: reduce
|
||||
reducer: last
|
||||
expression: A
|
||||
datasource:
|
||||
type: __expr__
|
||||
uid: __expr__
|
||||
- refId: C
|
||||
relativeTimeRange:
|
||||
from: 600
|
||||
to: 0
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
refId: C
|
||||
type: threshold
|
||||
expression: B
|
||||
conditions:
|
||||
- evaluator:
|
||||
type: lt
|
||||
params: [0.001]
|
||||
datasource:
|
||||
type: __expr__
|
||||
uid: __expr__
|
||||
|
||||
# ------------------------------------------------------------------ #
|
||||
# Job queue / resource health #
|
||||
# ------------------------------------------------------------------ #
|
||||
- orgId: 1
|
||||
name: xrpld-jobqueue
|
||||
folder: xrpld
|
||||
interval: 1m
|
||||
rules:
|
||||
# Transactions are being dropped because the job queue is full — the
|
||||
# node is shedding load it cannot process.
|
||||
- uid: xrpld-jobqueue-tx-overflow
|
||||
title: JobQueueTxOverflow
|
||||
condition: C
|
||||
for: 5m
|
||||
noDataState: NoData
|
||||
execErrState: Error
|
||||
labels:
|
||||
severity: warning
|
||||
category: jobqueue
|
||||
annotations:
|
||||
summary: "Job queue transaction overflow on {{ $labels.exported_instance }}"
|
||||
description: >-
|
||||
Node {{ $labels.exported_instance }} is overflowing its transaction
|
||||
job queue ({{ $values.B.Value }}/s over 5m). Transactions are being
|
||||
dropped under load.
|
||||
data:
|
||||
- refId: A
|
||||
relativeTimeRange:
|
||||
from: 600
|
||||
to: 0
|
||||
datasourceUid: prometheus
|
||||
model:
|
||||
refId: A
|
||||
expr: sum by (exported_instance) (rate(xrpld_jq_trans_overflow_total[5m]))
|
||||
instant: true
|
||||
range: false
|
||||
intervalMs: 1000
|
||||
maxDataPoints: 43200
|
||||
- refId: B
|
||||
relativeTimeRange:
|
||||
from: 600
|
||||
to: 0
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
refId: B
|
||||
type: reduce
|
||||
reducer: last
|
||||
expression: A
|
||||
datasource:
|
||||
type: __expr__
|
||||
uid: __expr__
|
||||
- refId: C
|
||||
relativeTimeRange:
|
||||
from: 600
|
||||
to: 0
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
refId: C
|
||||
type: threshold
|
||||
expression: B
|
||||
conditions:
|
||||
- evaluator:
|
||||
type: gt
|
||||
params: [0]
|
||||
datasource:
|
||||
type: __expr__
|
||||
uid: __expr__
|
||||
|
||||
# p99 time a job waits in the queue before running. A sustained p99
|
||||
# above 1s means the node is saturated and work is backing up. Tune the
|
||||
# threshold (in microseconds) to the deployment.
|
||||
- uid: xrpld-jobqueue-latency-high
|
||||
title: JobQueueLatencyHigh
|
||||
condition: C
|
||||
for: 5m
|
||||
noDataState: NoData
|
||||
execErrState: Error
|
||||
labels:
|
||||
severity: warning
|
||||
category: jobqueue
|
||||
annotations:
|
||||
summary: "Job queue latency high on {{ $labels.exported_instance }}"
|
||||
description: >-
|
||||
Node {{ $labels.exported_instance }} has a p99 job-queue wait of
|
||||
{{ $values.B.Value }}µs (>1s) over 5m. The node is saturated and jobs are
|
||||
backing up.
|
||||
data:
|
||||
- refId: A
|
||||
relativeTimeRange:
|
||||
from: 600
|
||||
to: 0
|
||||
datasourceUid: prometheus
|
||||
model:
|
||||
refId: A
|
||||
expr: histogram_quantile(0.99, sum by (le, exported_instance) (rate(xrpld_job_queued_duration_us_bucket[5m])))
|
||||
instant: true
|
||||
range: false
|
||||
intervalMs: 1000
|
||||
maxDataPoints: 43200
|
||||
- refId: B
|
||||
relativeTimeRange:
|
||||
from: 600
|
||||
to: 0
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
refId: B
|
||||
type: reduce
|
||||
reducer: last
|
||||
expression: A
|
||||
datasource:
|
||||
type: __expr__
|
||||
uid: __expr__
|
||||
- refId: C
|
||||
relativeTimeRange:
|
||||
from: 600
|
||||
to: 0
|
||||
datasourceUid: __expr__
|
||||
model:
|
||||
refId: C
|
||||
type: threshold
|
||||
expression: B
|
||||
conditions:
|
||||
- evaluator:
|
||||
type: gt
|
||||
params: [1000000]
|
||||
datasource:
|
||||
type: __expr__
|
||||
uid: __expr__
|
||||
99
docs/superpowers/specs/2026-07-06-phase9-alerting-design.md
Normal file
99
docs/superpowers/specs/2026-07-06-phase9-alerting-design.md
Normal file
@@ -0,0 +1,99 @@
|
||||
# Phase-9 Alerting — Design
|
||||
|
||||
**Date:** 2026-07-06
|
||||
**Branch:** `pratik/otel-phase9-metric-gap-fill` (PR #6513, Jira RIPD-5187)
|
||||
**Status:** Approved
|
||||
|
||||
## Purpose
|
||||
|
||||
Phase 9 exports ~68 internal rippled metrics and ships Grafana dashboards for
|
||||
them. This adds the missing operator-facing piece: **provisioned Grafana alert
|
||||
rules** that fire on the health-critical metrics phase 9 introduces. The
|
||||
phase-9 task list (line 311) and Jira story RIPD-5187 both already list
|
||||
"alerting rules" as a phase-9 deliverable, so this closes that gap.
|
||||
|
||||
Scope is deliberately narrow — the three subsystems whose failure is
|
||||
node-fatal: **consensus/ledger health, validator health, job queue**. RPC/API
|
||||
health is explicitly out of scope.
|
||||
|
||||
## Why phase 9 (not phase 11)
|
||||
|
||||
Every metric these alerts fire on is _born_ in phase 9
|
||||
(`xrpld_ledger_history_mismatch_total`, `xrpld_ledgers_closed_total`,
|
||||
`xrpld_validation_missed_total`, `xrpld_validations_checked_total`,
|
||||
`xrpld_jq_trans_overflow_total`, `xrpld_job_queued_duration_us_bucket`). Alerts
|
||||
belong with the metrics they watch, and this is where the dependency lives.
|
||||
|
||||
## Delivery
|
||||
|
||||
Provisioned YAML, version-controlled — matching the existing datasource /
|
||||
dashboard provisioning pattern. No docker-compose change: the Grafana service
|
||||
already mounts `./grafana/provisioning:/etc/grafana/provisioning:ro`, and
|
||||
Grafana auto-loads `provisioning/alerting/*.yaml`.
|
||||
|
||||
New files under `docker/telemetry/grafana/provisioning/alerting/`:
|
||||
|
||||
| File | Purpose |
|
||||
| -------------------- | ------------------------------------------------------------------------------------------------------------------- |
|
||||
| `contactpoints.yaml` | One contact point `xrpld-default` (webhook to a documented placeholder; comments show how to swap for Slack/email). |
|
||||
| `policies.yaml` | Default notification policy: route all alerts → `xrpld-default`, grouped by `alertname` + `exported_instance`. |
|
||||
| `rules.yaml` | 6 alert rules across 3 groups (below). |
|
||||
|
||||
Plus `docker/telemetry/ALERTING.md` — operator runbook: what each alert means,
|
||||
likely causes, and how to point the contact point at a real receiver.
|
||||
|
||||
## Alert rules
|
||||
|
||||
All rules target Prometheus datasource `uid: prometheus`. Each rule uses the
|
||||
Grafana rule shape: query (A) → reduce (B, last value) → threshold (C). All
|
||||
`rate()`/`histogram_quantile()` expressions aggregate with
|
||||
`sum by (exported_instance)` (or `+ le`) so **each node alerts independently**.
|
||||
Alert rules run headless, so they cannot use the dashboards' `$node` template
|
||||
variables — they match all series and group by `exported_instance` instead.
|
||||
|
||||
| Group | Alert | Expression (5m window) | Fires | `for` | severity |
|
||||
| --------- | --------------------- | --------------------------------------------------------------------------------------------------------- | --------------------- | ----- | -------- |
|
||||
| Consensus | LedgerHistoryMismatch | `sum by (exported_instance)(rate(xrpld_ledger_history_mismatch_total[5m]))` | `> 0` | 5m | critical |
|
||||
| Consensus | LedgerCloseStalled | `sum by (exported_instance)(rate(xrpld_ledgers_closed_total[5m]))` | `< 0.001` (≈0) | 3m | critical |
|
||||
| Validator | ValidationsMissed | `sum by (exported_instance)(rate(xrpld_validation_missed_total[5m]))` | `> 0` | 5m | warning |
|
||||
| Validator | ValidationsNotChecked | `sum by (exported_instance)(rate(xrpld_validations_checked_total[5m]))` | `< 0.001` (≈0) | 5m | warning |
|
||||
| Job queue | JobQueueTxOverflow | `sum by (exported_instance)(rate(xrpld_jq_trans_overflow_total[5m]))` | `> 0` | 5m | warning |
|
||||
| Job queue | JobQueueLatencyHigh | `histogram_quantile(0.99, sum by (le, exported_instance)(rate(xrpld_job_queued_duration_us_bucket[5m])))` | `> 1000000` (µs = 1s) | 5m | warning |
|
||||
|
||||
Each rule carries labels `severity` and `category` (consensus/validator/jobqueue)
|
||||
and annotations `summary` + `description` (with `{{ $labels.exported_instance }}`
|
||||
and `{{ $values.B.Value }}` interpolation).
|
||||
|
||||
### Threshold rationale
|
||||
|
||||
- **LedgerCloseStalled `< 0.001` for 3m**: healthy nodes close a ledger every
|
||||
~3-5s; a 5m rate decaying to ~0 means the node is stuck. The epsilon (not
|
||||
exact `0`) avoids float rate-noise suppressing the alert.
|
||||
- **JobQueueLatencyHigh 1s p99**: a default starting point, easy to tune — jobs
|
||||
queued >1s at p99 indicate the node is saturated.
|
||||
- Others are `> 0` on error/miss counters: any sustained nonzero rate is
|
||||
actionable.
|
||||
|
||||
## Non-goals / YAGNI
|
||||
|
||||
- No per-alert silencing schedules, no mute timings.
|
||||
- No RPC/API, overlay, or fee-market alerts (dashboards cover those visually).
|
||||
- Single contact point — multi-receiver routing is left to the operator.
|
||||
|
||||
## Verification
|
||||
|
||||
1. `yamllint` (or `python -c yaml.safe_load`) on all three YAML files.
|
||||
2. `docker compose -f docker/telemetry/docker-compose.yml config -q` still parses.
|
||||
3. Optional live check: start stack, `GET /api/v1/provisioning/alert-rules`
|
||||
returns the 6 rules; Grafana logs show no provisioning errors.
|
||||
4. Code-review pass (subagent) against phase conventions before commit.
|
||||
|
||||
## Branch / PR restructure (independent, same session)
|
||||
|
||||
- Rename `pratik/otel-phase11-benchmark` → `pratik/perf-test-otel-on`
|
||||
(0 unique commits — equals phase-10 tip; the telemetry-IN baseline).
|
||||
- Rename `pratik/otel-phase11-telemetry-off` → `pratik/perf-test-otel-off`
|
||||
(carries the "compile telemetry OUT" build change + benchmark runbook).
|
||||
- Close PR #7433, delete stale `origin/pratik/otel-phase11-telemetry-off`.
|
||||
- Push both; open one perf PR `perf-test-otel-off → perf-test-otel-on`
|
||||
(diff = the ON-vs-OFF delta; `-on` needs no PR of its own).
|
||||
@@ -371,6 +371,12 @@ LedgerHistory::handleMismatch(
|
||||
return;
|
||||
}
|
||||
|
||||
// Tracks whether the mismatch reason has already been recorded. The
|
||||
// consensus tx-set hash disagreement is the root cause, so it is counted
|
||||
// once here; the tx-level comparison below still logs its diagnostics but
|
||||
// must not record a second reason for the same mismatch event.
|
||||
bool reasonRecorded = false;
|
||||
|
||||
if (builtConsensusHash && validatedConsensusHash)
|
||||
{
|
||||
if (builtConsensusHash != validatedConsensusHash)
|
||||
@@ -378,11 +384,8 @@ LedgerHistory::handleMismatch(
|
||||
JLOG(j_.error()) << "MISMATCH on consensus transaction set "
|
||||
<< " built: " << to_string(*builtConsensusHash)
|
||||
<< " validated: " << to_string(*validatedConsensusHash);
|
||||
// The consensus tx-set hashes disagree — this is the root cause,
|
||||
// so record it as the single reason and stop. The tx-level
|
||||
// comparison below would otherwise double-count the same mismatch.
|
||||
recordReason("consensus_txset");
|
||||
return;
|
||||
reasonRecorded = true;
|
||||
}
|
||||
else
|
||||
{
|
||||
@@ -398,13 +401,15 @@ LedgerHistory::handleMismatch(
|
||||
if (builtTx == validTx)
|
||||
{
|
||||
JLOG(j_.error()) << "MISMATCH with same " << builtTx.size() << " transactions";
|
||||
recordReason("same_txset_diff_result");
|
||||
if (!reasonRecorded)
|
||||
recordReason("same_txset_diff_result");
|
||||
}
|
||||
else
|
||||
{
|
||||
JLOG(j_.error()) << "MISMATCH with " << builtTx.size() << " built and " << validTx.size()
|
||||
<< " valid transactions.";
|
||||
recordReason("different_txset");
|
||||
if (!reasonRecorded)
|
||||
recordReason("different_txset");
|
||||
}
|
||||
|
||||
JLOG(j_.error()) << "built\n" << getJson({*builtLedger, {}});
|
||||
|
||||
@@ -22,11 +22,11 @@
|
||||
#include <xrpl/nodestore/Database.h>
|
||||
#include <xrpl/nodestore/NodeObject.h>
|
||||
#include <xrpl/protocol/HashPrefix.h>
|
||||
#include <xrpl/protocol/Indexes.h>
|
||||
#include <xrpl/protocol/Indexes.h> // IWYU pragma: keep
|
||||
#include <xrpl/protocol/LedgerHeader.h>
|
||||
#include <xrpl/protocol/Rules.h>
|
||||
#include <xrpl/protocol/Serializer.h>
|
||||
#include <xrpl/protocol/SystemParameters.h>
|
||||
#include <xrpl/protocol/SystemParameters.h> // IWYU pragma: keep
|
||||
#include <xrpl/protocol/jss.h>
|
||||
#include <xrpl/resource/Fees.h>
|
||||
#include <xrpl/shamap/SHAMapNodeID.h>
|
||||
|
||||
@@ -79,10 +79,12 @@
|
||||
#include <xrpl/protocol/ApiVersion.h>
|
||||
#include <xrpl/protocol/BuildInfo.h>
|
||||
#include <xrpl/protocol/Feature.h>
|
||||
#include <xrpl/protocol/Indexes.h> // IWYU pragma: keep
|
||||
#include <xrpl/protocol/Protocol.h>
|
||||
#include <xrpl/protocol/PublicKey.h>
|
||||
#include <xrpl/protocol/STParsedJSON.h>
|
||||
#include <xrpl/protocol/Serializer.h>
|
||||
#include <xrpl/protocol/SystemParameters.h> // IWYU pragma: keep
|
||||
#include <xrpl/protocol/jss.h>
|
||||
#include <xrpl/protocol/tokens.h>
|
||||
#include <xrpl/rdb/DatabaseCon.h>
|
||||
|
||||
Reference in New Issue
Block a user