From a2a58a5d85d6b2291069ebefc6a18c8f9f712a73 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Wed, 8 Jul 2026 14:57:29 +0100 Subject: [PATCH 1/2] docs(telemetry): update plan docs to use service_instance_id Co-Authored-By: Claude Opus 4.6 --- OpenTelemetryPlan/06-implementation-phases.md | 2 +- OpenTelemetryPlan/Phase7_taskList.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/OpenTelemetryPlan/06-implementation-phases.md b/OpenTelemetryPlan/06-implementation-phases.md index cecc2e49fd..29b29fff18 100644 --- a/OpenTelemetryPlan/06-implementation-phases.md +++ b/OpenTelemetryPlan/06-implementation-phases.md @@ -1378,7 +1378,7 @@ New Grafana dashboard: `validator-health.json` | State Changes Rate | stat | `rate(xrpld_state_changes_total[1h])` | | Ledgers Closed Rate | stat | `rate(xrpld_ledgers_closed_total[5m]) * 60` | -**Dashboard conventions**: `$node` template variable for `exported_instance` filtering, dark theme, matching existing panel sizes and color schemes. +**Dashboard conventions**: `$node` template variable for `service_instance_id` filtering, dark theme, matching existing panel sizes and color schemes. --- diff --git a/OpenTelemetryPlan/Phase7_taskList.md b/OpenTelemetryPlan/Phase7_taskList.md index c61e1b2d8d..971372fdb0 100644 --- a/OpenTelemetryPlan/Phase7_taskList.md +++ b/OpenTelemetryPlan/Phase7_taskList.md @@ -358,7 +358,7 @@ validatorParticipationGauge_ = meter_->CreateInt64ObservableGauge( - **Label dimensions**: - `validator` — base58-encoded validator master public key - - `exported_instance` — this node's identity (standard) + - `service_instance_id` — this node's identity (standard) - **Emission**: every flag ledger (256 ledgers, ~15 minutes) or on a 10-second async gauge callback with cached results (recompute only From d24534a3403606b555004b9dbc09cde95b354865 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Wed, 8 Jul 2026 14:57:46 +0100 Subject: [PATCH 2/2] docs(telemetry): update Phase9 task list to use service_instance_id Co-Authored-By: Claude Opus 4.6 --- OpenTelemetryPlan/Phase9_taskList.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/OpenTelemetryPlan/Phase9_taskList.md b/OpenTelemetryPlan/Phase9_taskList.md index ad3c08b902..a29cfb8c79 100644 --- a/OpenTelemetryPlan/Phase9_taskList.md +++ b/OpenTelemetryPlan/Phase9_taskList.md @@ -376,7 +376,7 @@ These metrics serve multiple external consumer categories identified during rese | State Changes Rate | stat | `rate(xrpld_state_changes_total[1h])` | | Ledgers Closed Rate | stat | `rate(xrpld_ledgers_closed_total[5m]) * 60` | -**Dashboard conventions**: `$node` template variable for `exported_instance` filtering, dark theme, matching existing panel sizes and color schemes. +**Dashboard conventions**: `$node` template variable for `service_instance_id` filtering, dark theme, matching existing panel sizes and color schemes. **Key new files**: `docker/telemetry/grafana/dashboards/rippled-validator-health.json` @@ -495,7 +495,7 @@ New files under `docker/telemetry/grafana/provisioning/alerting/`: | File | Purpose | | -------------------- | ------------------------------------------------------------------------------------------------------------------- | | `contactpoints.yaml` | One contact point `xrpld-default` (webhook to a documented placeholder; comments show how to swap for Slack/email). | -| `policies.yaml` | Default notification policy: route all alerts → `xrpld-default`, grouped by `alertname` + `exported_instance`. | +| `policies.yaml` | Default notification policy: route all alerts → `xrpld-default`, grouped by `alertname` + `service_instance_id`. | | `rules.yaml` | 6 alert rules across 3 groups (below). | Plus the Alerting section of `docs/telemetry-runbook.md` — operator runbook: @@ -507,21 +507,21 @@ real receiver. All rules target Prometheus datasource `uid: prometheus`. Each rule uses the Grafana rule shape: query (A) → reduce (B, last value) → threshold (C). All `rate()`/`histogram_quantile()` expressions aggregate with -`sum by (exported_instance)` (or `+ le`) so **each node alerts independently**. +`sum by (service_instance_id)` (or `+ le`) so **each node alerts independently**. Alert rules run headless, so they cannot use the dashboards' `$node` template -variables — they match all series and group by `exported_instance` instead. +variables — they match all series and group by `service_instance_id` instead. -| Group | Alert | Expression (5m window) | Fires | `for` | severity | -| --------- | --------------------- | --------------------------------------------------------------------------------------------------------- | --------------------- | ----- | -------- | -| Consensus | LedgerHistoryMismatch | `sum by (exported_instance)(rate(xrpld_ledger_history_mismatch_total[5m]))` | `> 0` | 5m | critical | -| Consensus | LedgerCloseStalled | `sum by (exported_instance)(rate(xrpld_ledgers_closed_total[5m]))` | `< 0.001` (≈0) | 3m | critical | -| Validator | ValidationsMissed | `sum by (exported_instance)(rate(xrpld_validation_missed_total[5m]))` | `> 0` | 5m | warning | -| Validator | ValidationsNotChecked | `sum by (exported_instance)(rate(xrpld_validations_checked_total[5m]))` | `< 0.001` (≈0) | 5m | warning | -| Job queue | JobQueueTxOverflow | `sum by (exported_instance)(rate(xrpld_jq_trans_overflow_total[5m]))` | `> 0` | 5m | warning | -| Job queue | JobQueueLatencyHigh | `histogram_quantile(0.99, sum by (le, exported_instance)(rate(xrpld_job_queued_duration_us_bucket[5m])))` | `> 1000000` (µs = 1s) | 5m | warning | +| Group | Alert | Expression (5m window) | Fires | `for` | severity | +| --------- | --------------------- | ----------------------------------------------------------------------------------------------------------- | --------------------- | ----- | -------- | +| Consensus | LedgerHistoryMismatch | `sum by (service_instance_id)(rate(xrpld_ledger_history_mismatch_total[5m]))` | `> 0` | 5m | critical | +| Consensus | LedgerCloseStalled | `sum by (service_instance_id)(rate(xrpld_ledgers_closed_total[5m]))` | `< 0.001` (≈0) | 3m | critical | +| Validator | ValidationsMissed | `sum by (service_instance_id)(rate(xrpld_validation_missed_total[5m]))` | `> 0` | 5m | warning | +| Validator | ValidationsNotChecked | `sum by (service_instance_id)(rate(xrpld_validations_checked_total[5m]))` | `< 0.001` (≈0) | 5m | warning | +| Job queue | JobQueueTxOverflow | `sum by (service_instance_id)(rate(xrpld_jq_trans_overflow_total[5m]))` | `> 0` | 5m | warning | +| Job queue | JobQueueLatencyHigh | `histogram_quantile(0.99, sum by (le, service_instance_id)(rate(xrpld_job_queued_duration_us_bucket[5m])))` | `> 1000000` (µs = 1s) | 5m | warning | Each rule carries labels `severity` and `category` (consensus/validator/jobqueue) -and annotations `summary` + `description` (with `{{ $labels.exported_instance }}` +and annotations `summary` + `description` (with `{{ $labels.service_instance_id }}` and `{{ $values.B.Value }}` interpolation). #### Threshold rationale