From b5d7e3984f4bb81c7b7207164d4e3f9942378216 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Mon, 6 Jul 2026 16:46:14 +0100 Subject: [PATCH] docs(telemetry): add Alerting section, drop phase label, expand alerting task - Add an operator-facing Alerting section to the runbook covering the six provisioned Grafana rules and how to point them at a real receiver. - Strip the "(Phase 9)" suffix from the MetricsRegistry Gauges heading. - ALERTING.md: drop the internal dev-phase reference from the intro. - Phase9_taskList.md: expand Task 9.9 to describe the provisioned alerting (rules, contact point, policy, ALERTING.md runbook). Co-Authored-By: Claude Opus 4.8 --- OpenTelemetryPlan/Phase9_taskList.md | 9 ++++++++- docker/telemetry/ALERTING.md | 6 +++--- docs/telemetry-runbook.md | 28 +++++++++++++++++++++++++++- 3 files changed, 38 insertions(+), 5 deletions(-) diff --git a/OpenTelemetryPlan/Phase9_taskList.md b/OpenTelemetryPlan/Phase9_taskList.md index febd58d6b8..f3b79d7af5 100644 --- a/OpenTelemetryPlan/Phase9_taskList.md +++ b/OpenTelemetryPlan/Phase9_taskList.md @@ -308,13 +308,20 @@ These metrics serve multiple external consumer categories identified during rese - Add Prometheus query examples for new metrics - Update `docs/telemetry-runbook.md`: - - Add alerting rules for new metrics (NodeStore write_load, TxQ capacity, cache hit rate degradation) + - Add an Alerting section covering the provisioned rules and how to wire a receiver - Add troubleshooting entries for new metric categories +- Provision Grafana alert rules (`docker/telemetry/grafana/provisioning/alerting/`): + - 6 rules in 3 groups — consensus/ledger (`LedgerHistoryMismatch`, `LedgerCloseStalled`), validator (`ValidationsMissed`, `ValidationsNotChecked`), job queue (`JobQueueTxOverflow`, `JobQueueLatencyHigh`) + - `xrpld-default` webhook contact point + flat notification policy; auto-loaded via the existing `provisioning/` mount (no docker-compose change) + - `docker/telemetry/ALERTING.md` operator runbook (per-alert meaning, tuning, receiver wiring) + **Key modified files**: - `OpenTelemetryPlan/09-data-collection-reference.md` - `docs/telemetry-runbook.md` +- `docker/telemetry/grafana/provisioning/alerting/{rules,contactpoints,policies}.yaml` (new) +- `docker/telemetry/ALERTING.md` (new) --- diff --git a/docker/telemetry/ALERTING.md b/docker/telemetry/ALERTING.md index 2bb7285dc2..ceaeaadaf5 100644 --- a/docker/telemetry/ALERTING.md +++ b/docker/telemetry/ALERTING.md @@ -1,8 +1,8 @@ # rippled OpenTelemetry Alerting Runbook -Phase 9 exports rippled's internal metrics and provisions Grafana alert rules -on the health-critical ones. This document explains each alert, its likely -cause, and how to point alerts at a real receiver. +rippled exports its internal metrics and provisions Grafana alert rules on the +health-critical ones. This document explains each alert, its likely cause, and +how to point alerts at a real receiver. The rules are provisioned from `grafana/provisioning/alerting/` and load automatically when the Grafana diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index c65812a2de..88c1a80f1c 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -579,7 +579,7 @@ The `OTelCollector` implementation exports metrics via OTLP/HTTP to the same OTe | `xrpld_{category}_Bytes_In/Out` | OverlayImpl.h:535 | Overlay traffic bytes per category (57 categories) | | `xrpld_{category}_Messages_In/Out` | OverlayImpl.h:535 | Overlay traffic messages per category | -#### OTel MetricsRegistry Gauges (Phase 9) +#### OTel MetricsRegistry Gauges These gauges are exported via the OTel Metrics SDK `PeriodicMetricReader` (10s interval), NOT through beast::insight. @@ -872,6 +872,32 @@ Requires `trace_peer=1` in the `[telemetry]` config section. | `peer.proposal.receive` | `{span_name="peer.proposal.receive"}` | Peer Network (Rate, Trusted/Untrusted) | | `peer.validation.receive` | `{span_name="peer.validation.receive"}` | Peer Network (Rate, Trusted/Untrusted) | +## Alerting + +Grafana ships six provisioned alert rules on the health-critical metrics, so a +stock stack alerts out of the box with no UI setup. Rules are loaded from +`docker/telemetry/grafana/provisioning/alerting/` on startup and appear under +**Alerting → Alert rules**, folder **xrpld**. + +Each rule runs every minute against the Prometheus datasource over a 5-minute +window and groups by `exported_instance`, so every node alerts independently. +An alert fires only after its condition holds for the `for` dwell time. + +| Alert | Severity | Fires when | For | +| ----------------------- | -------- | ----------------------------------------------- | --- | +| `LedgerHistoryMismatch` | critical | `rate(xrpld_ledger_history_mismatch_total)` > 0 | 5m | +| `LedgerCloseStalled` | critical | `rate(xrpld_ledgers_closed_total)` ≈ 0 | 3m | +| `ValidationsMissed` | warning | `rate(xrpld_validation_missed_total)` > 0 | 5m | +| `ValidationsNotChecked` | warning | `rate(xrpld_validations_checked_total)` ≈ 0 | 5m | +| `JobQueueTxOverflow` | warning | `rate(xrpld_jq_trans_overflow_total)` > 0 | 5m | +| `JobQueueLatencyHigh` | warning | p99 `xrpld_job_queued_duration_us` > 1s | 5m | + +Alerts route through the `xrpld-default` contact point, which ships as a +local-dev webhook to a placeholder URL — alerts fire but go nowhere until it is +pointed at a real receiver. See +[ALERTING.md](../docker/telemetry/ALERTING.md) for per-alert causes, threshold +tuning, and how to wire alerts to Slack/email/PagerDuty. + ## Log-Trace Correlation When xrpld is built with `telemetry=ON`, log lines emitted within an active OpenTelemetry span automatically include `trace_id` and `span_id` fields: