From 22940969ff2f81d79e5faea05eb40bed7f2c435a Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Mon, 6 Jul 2026 16:43:15 +0100 Subject: [PATCH 1/6] docs(telemetry): drop dev-phase labels and personal name from runbook The operator runbook is for people running OpenTelemetry against a node, not a record of the internal development phases. Strip the "(Phase N)" suffixes from the span-reference headings and replace the personal-name example service.name value with a generic one. Co-Authored-By: Claude Opus 4.8 --- docs/telemetry-runbook.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index b73dfcce0c..c614795433 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -66,7 +66,7 @@ cmake --build --preset default All spans instrumented in xrpld, grouped by subsystem: -### RPC Spans (Phase 2) +### RPC Spans | Span Name | Source File | Attributes | Description | | -------------------- | ----------------- | ----------------------------------------------------------- | ----------------------------------------------------- | @@ -76,14 +76,14 @@ All spans instrumented in xrpld, grouped by subsystem: | `rpc.process` | ServerHandler.cpp | `is_batch`, `batch_size` | RPC processing (child of rpc.http_request/ws_message) | | `rpc.command.` | RPCHandler.cpp | `command`, `version`, `rpc_role`, `rpc_status`, `load_type` | Per-command span (e.g., `rpc.command.server_info`) | -### Transaction Spans (Phase 3) +### Transaction Spans | Span Name | Source File | Attributes | Description | | ------------ | -------------- | --------------------------------------------------------------------------------- | ------------------------------------- | | `tx.process` | NetworkOPs.cpp | `tx_hash`, `local`, `path`, `tx_type`, `fee`, `sequence`, `ter_result`, `applied` | Transaction submission and processing | | `tx.receive` | PeerImp.cpp | `peer_id`, `tx_hash`, `tx_type`, `peer_version`, `suppressed`, `tx_status` | Transaction received from peer relay | -### Transaction Queue Spans (Phase 3) +### Transaction Queue Spans | Span Name | Source File | Attributes | Description | | ------------------ | ----------- | -------------------------------------------------------- | -------------------------------------------------- | @@ -94,7 +94,7 @@ All spans instrumented in xrpld, grouped by subsystem: | `txq.accept_tx` | TxQ.cpp | `tx_hash`, `retries_remaining`, `ter_code`, `txq_status` | Per-transaction apply during accept | | `txq.cleanup` | TxQ.cpp | `ledger_seq` | Post-close cleanup of expired queue entries | -### Consensus Spans (Phase 4) +### Consensus Spans | Span Name | Source File | Attributes | Description | | ------------------------------ | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | @@ -400,7 +400,7 @@ one dashboard set serves every deployment: | Dimension | Attribute | Set by | Example values | | ----------- | ------------------------ | ---------- | ------------------------------ | | Node | `service.instance.id` | xrpld cfg | `alice-laptop`, `ci-runner-7` | -| Service | `service.name` | xrpld cfg | `xrpld`, `pratik-xrpld` | +| Service | `service.name` | xrpld cfg | `xrpld`, `xrpld-validator` | | Network | `xrpl.network.type` | xrpld node | `mainnet`, `testnet`, `devnet` | | Environment | `deployment.environment` | collector | `local`, `test`, `ci`, `prod` | From d84fdbd29f42fdf84a6decb454abd1152a95e8b6 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Mon, 6 Jul 2026 16:43:40 +0100 Subject: [PATCH 2/6] docs(telemetry): drop dev-phase labels from Ledger/Peer span headings Same operator-facing cleanup as the RPC/transaction/consensus headings: the runbook documents the running system, not the development phase that added each span. Strip the "(Phase 6)" suffixes from the Ledger and Peer span reference headings. Co-Authored-By: Claude Opus 4.8 --- docs/telemetry-runbook.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index 15ad5de78c..d0afe693e1 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -151,7 +151,7 @@ RED metrics on the _Transaction Overview_ dashboard. {name="consensus.update_positions"} >> {event:name="dispute.resolve"} ``` -### Ledger Spans (Phase 6) +### Ledger Spans | Span Name | Source File | Attributes | Description | | ----------------- | -------------------- | ------------------------------------- | ----------------------------- | @@ -159,7 +159,7 @@ RED metrics on the _Transaction Overview_ dashboard. | `ledger.validate` | LedgerMaster.cpp:915 | `ledger_seq`, `validations` | Ledger promoted to validated | | `ledger.store` | LedgerMaster.cpp:409 | `ledger_seq` | Ledger stored in history | -### Peer Spans (Phase 6) +### Peer Spans | Span Name | Source File | Attributes | Description | | ------------------------- | ---------------- | ------------------------------- | ----------------------------- | From 2b419338fbfa4c2c5940616c85b9d6a26d315d2f Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Mon, 6 Jul 2026 16:44:19 +0100 Subject: [PATCH 3/6] docs(telemetry): drop dev-phase label from Log-Trace Correlation heading Operator-facing cleanup: strip the "(Phase 8)" suffix from the Log-Trace Correlation section heading in the runbook. Co-Authored-By: Claude Opus 4.8 --- docs/telemetry-runbook.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index eda34de959..f628f15dc5 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -754,7 +754,7 @@ Requires `trace_peer=1` in the `[telemetry]` config section. | `peer.proposal.receive` | `{span_name="peer.proposal.receive"}` | Peer Network (Rate, Trusted/Untrusted) | | `peer.validation.receive` | `{span_name="peer.validation.receive"}` | Peer Network (Rate, Trusted/Untrusted) | -## Log-Trace Correlation (Phase 8) +## Log-Trace Correlation When xrpld is built with `telemetry=ON`, log lines emitted within an active OpenTelemetry span automatically include `trace_id` and `span_id` fields: From b5d7e3984f4bb81c7b7207164d4e3f9942378216 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Mon, 6 Jul 2026 16:46:14 +0100 Subject: [PATCH 4/6] docs(telemetry): add Alerting section, drop phase label, expand alerting task - Add an operator-facing Alerting section to the runbook covering the six provisioned Grafana rules and how to point them at a real receiver. - Strip the "(Phase 9)" suffix from the MetricsRegistry Gauges heading. - ALERTING.md: drop the internal dev-phase reference from the intro. - Phase9_taskList.md: expand Task 9.9 to describe the provisioned alerting (rules, contact point, policy, ALERTING.md runbook). Co-Authored-By: Claude Opus 4.8 --- OpenTelemetryPlan/Phase9_taskList.md | 9 ++++++++- docker/telemetry/ALERTING.md | 6 +++--- docs/telemetry-runbook.md | 28 +++++++++++++++++++++++++++- 3 files changed, 38 insertions(+), 5 deletions(-) diff --git a/OpenTelemetryPlan/Phase9_taskList.md b/OpenTelemetryPlan/Phase9_taskList.md index febd58d6b8..f3b79d7af5 100644 --- a/OpenTelemetryPlan/Phase9_taskList.md +++ b/OpenTelemetryPlan/Phase9_taskList.md @@ -308,13 +308,20 @@ These metrics serve multiple external consumer categories identified during rese - Add Prometheus query examples for new metrics - Update `docs/telemetry-runbook.md`: - - Add alerting rules for new metrics (NodeStore write_load, TxQ capacity, cache hit rate degradation) + - Add an Alerting section covering the provisioned rules and how to wire a receiver - Add troubleshooting entries for new metric categories +- Provision Grafana alert rules (`docker/telemetry/grafana/provisioning/alerting/`): + - 6 rules in 3 groups — consensus/ledger (`LedgerHistoryMismatch`, `LedgerCloseStalled`), validator (`ValidationsMissed`, `ValidationsNotChecked`), job queue (`JobQueueTxOverflow`, `JobQueueLatencyHigh`) + - `xrpld-default` webhook contact point + flat notification policy; auto-loaded via the existing `provisioning/` mount (no docker-compose change) + - `docker/telemetry/ALERTING.md` operator runbook (per-alert meaning, tuning, receiver wiring) + **Key modified files**: - `OpenTelemetryPlan/09-data-collection-reference.md` - `docs/telemetry-runbook.md` +- `docker/telemetry/grafana/provisioning/alerting/{rules,contactpoints,policies}.yaml` (new) +- `docker/telemetry/ALERTING.md` (new) --- diff --git a/docker/telemetry/ALERTING.md b/docker/telemetry/ALERTING.md index 2bb7285dc2..ceaeaadaf5 100644 --- a/docker/telemetry/ALERTING.md +++ b/docker/telemetry/ALERTING.md @@ -1,8 +1,8 @@ # rippled OpenTelemetry Alerting Runbook -Phase 9 exports rippled's internal metrics and provisions Grafana alert rules -on the health-critical ones. This document explains each alert, its likely -cause, and how to point alerts at a real receiver. +rippled exports its internal metrics and provisions Grafana alert rules on the +health-critical ones. This document explains each alert, its likely cause, and +how to point alerts at a real receiver. The rules are provisioned from `grafana/provisioning/alerting/` and load automatically when the Grafana diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index c65812a2de..88c1a80f1c 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -579,7 +579,7 @@ The `OTelCollector` implementation exports metrics via OTLP/HTTP to the same OTe | `xrpld_{category}_Bytes_In/Out` | OverlayImpl.h:535 | Overlay traffic bytes per category (57 categories) | | `xrpld_{category}_Messages_In/Out` | OverlayImpl.h:535 | Overlay traffic messages per category | -#### OTel MetricsRegistry Gauges (Phase 9) +#### OTel MetricsRegistry Gauges These gauges are exported via the OTel Metrics SDK `PeriodicMetricReader` (10s interval), NOT through beast::insight. @@ -872,6 +872,32 @@ Requires `trace_peer=1` in the `[telemetry]` config section. | `peer.proposal.receive` | `{span_name="peer.proposal.receive"}` | Peer Network (Rate, Trusted/Untrusted) | | `peer.validation.receive` | `{span_name="peer.validation.receive"}` | Peer Network (Rate, Trusted/Untrusted) | +## Alerting + +Grafana ships six provisioned alert rules on the health-critical metrics, so a +stock stack alerts out of the box with no UI setup. Rules are loaded from +`docker/telemetry/grafana/provisioning/alerting/` on startup and appear under +**Alerting → Alert rules**, folder **xrpld**. + +Each rule runs every minute against the Prometheus datasource over a 5-minute +window and groups by `exported_instance`, so every node alerts independently. +An alert fires only after its condition holds for the `for` dwell time. + +| Alert | Severity | Fires when | For | +| ----------------------- | -------- | ----------------------------------------------- | --- | +| `LedgerHistoryMismatch` | critical | `rate(xrpld_ledger_history_mismatch_total)` > 0 | 5m | +| `LedgerCloseStalled` | critical | `rate(xrpld_ledgers_closed_total)` ≈ 0 | 3m | +| `ValidationsMissed` | warning | `rate(xrpld_validation_missed_total)` > 0 | 5m | +| `ValidationsNotChecked` | warning | `rate(xrpld_validations_checked_total)` ≈ 0 | 5m | +| `JobQueueTxOverflow` | warning | `rate(xrpld_jq_trans_overflow_total)` > 0 | 5m | +| `JobQueueLatencyHigh` | warning | p99 `xrpld_job_queued_duration_us` > 1s | 5m | + +Alerts route through the `xrpld-default` contact point, which ships as a +local-dev webhook to a placeholder URL — alerts fire but go nowhere until it is +pointed at a real receiver. See +[ALERTING.md](../docker/telemetry/ALERTING.md) for per-alert causes, threshold +tuning, and how to wire alerts to Slack/email/PagerDuty. + ## Log-Trace Correlation When xrpld is built with `telemetry=ON`, log lines emitted within an active OpenTelemetry span automatically include `trace_id` and `span_id` fields: From f64f4b35e7029ec412f521e4f2a0ba22689bdc70 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Mon, 6 Jul 2026 16:50:21 +0100 Subject: [PATCH 5/6] config(telemetry): use generic /var/log/xrpld path, drop personal home dir Replace the hardcoded /home/pratik/xrpld-logs path with the conventional /var/log/xrpld in the mainnet config's [debug_logfile], the docker-compose log volume mount, and the collector-config comment. Removes a personal home directory from shipped config. Co-Authored-By: Claude Opus 4.8 --- docker/telemetry/docker-compose.yml | 4 ++-- docker/telemetry/otel-collector-config.yaml | 2 +- docker/telemetry/xrpld-telemetry-mainnet.cfg | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docker/telemetry/docker-compose.yml b/docker/telemetry/docker-compose.yml index becbbb811b..e93338515e 100644 --- a/docker/telemetry/docker-compose.yml +++ b/docker/telemetry/docker-compose.yml @@ -37,10 +37,10 @@ services: # Mount collector pipeline config (receivers → processors → exporters) - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:ro # Phase 8: Mount rippled log directories for filelog receiver. - # User-run xrpld writes to /home/pratik/xrpld-logs//debug.log. + # User-run xrpld writes to /var/log/xrpld//debug.log. # Integration test still writes to /tmp/xrpld-integration/. # Both are mounted read-only so the collector can tail debug.log files. - - /home/pratik/xrpld-logs:/var/log/rippled:ro + - /var/log/xrpld:/var/log/rippled:ro - /tmp/xrpld-integration:/var/log/rippled-integration:ro depends_on: - tempo diff --git a/docker/telemetry/otel-collector-config.yaml b/docker/telemetry/otel-collector-config.yaml index 3c2cc3a030..aa57680d78 100644 --- a/docker/telemetry/otel-collector-config.yaml +++ b/docker/telemetry/otel-collector-config.yaml @@ -36,7 +36,7 @@ receivers: # trace_id, span_id, message) via regex. The trace_id and span_id are # optional — only present when the log was emitted within an active span. filelog: - # Tails both user-run xrpld logs (/home/pratik/xrpld-logs/...) + # Tails both user-run xrpld logs (/var/log/xrpld/...) # and integration-test logs (/tmp/xrpld-integration/...). include: - /var/log/rippled/*/debug.log diff --git a/docker/telemetry/xrpld-telemetry-mainnet.cfg b/docker/telemetry/xrpld-telemetry-mainnet.cfg index 816ada7275..8d95bb14f7 100644 --- a/docker/telemetry/xrpld-telemetry-mainnet.cfg +++ b/docker/telemetry/xrpld-telemetry-mainnet.cfg @@ -95,7 +95,7 @@ docker/telemetry/data # --- Logging ---------------------------------------------------------------- [debug_logfile] -/home/pratik/xrpld-logs/mainnet/debug.log +/var/log/xrpld/mainnet/debug.log [rpc_startup] { "command": "log_level", "severity": "debug" } From c6b9f0260646bcaf9374b82ce5afe1d9f13e6b41 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Mon, 6 Jul 2026 16:51:04 +0100 Subject: [PATCH 6/6] docs(telemetry): match runbook to generic /var/log/xrpld log path Follow the config change that replaced the personal home directory: the runbook prose now points at /var/log/xrpld/mainnet/debug.log, matching the mainnet config and the collector's filelog receiver. Co-Authored-By: Claude Opus 4.8 --- docs/telemetry-runbook.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index dc4be8a794..110b5471de 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -59,7 +59,7 @@ public network with all tracing and native metrics enabled: Both set `[insight] server=otel` (native metrics → collector → Prometheus, which drives the dashboards) and `service_instance_id`, exposed by Prometheus as the `exported_instance` label that the `$node` dashboard variable filters on. The -mainnet config logs to `/home/pratik/xrpld-logs/mainnet/debug.log` — the path +mainnet config logs to `/var/log/xrpld/mainnet/debug.log` — the path the collector's filelog receiver tails for log-trace correlation. Metrics begin flowing as soon as the node connects to peers (`server_state`