ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-07-29 10:00:30 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation

Browse Source
This commit is contained in:
Pratik Mankawde
2026-07-08 14:57:54 +01:00
parent e64afb53d0 d24534a340
commit 45d000cd27
3 changed files with 15 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
OpenTelemetryPlan/06-implementation-phases.md
View File

@@ -2001,7 +2001,7 @@ New Grafana dashboard: `validator-health.json`
| State Changes Rate | stat | `rate(xrpld_state_changes_total[1h])` |
| Ledgers Closed Rate | stat | `rate(xrpld_ledgers_closed_total[5m]) * 60` |
**Dashboard conventions**: `$node` template variable for `exported_instance` filtering, dark theme, matching existing panel sizes and color schemes.
**Dashboard conventions**: `$node` template variable for `service_instance_id` filtering, dark theme, matching existing panel sizes and color schemes.
---

2
OpenTelemetryPlan/Phase7_taskList.md
View File

@@ -358,7 +358,7 @@ validatorParticipationGauge_ = meter_->CreateInt64ObservableGauge(
- **Label dimensions**:
- `validator` — base58-encoded validator master public key
- `exported_instance` — this node's identity (standard)
- `service_instance_id` — this node's identity (standard)
- **Emission**: every flag ledger (256 ledgers, ~15 minutes) or on a
10-second async gauge callback with cached results (recompute only

26
OpenTelemetryPlan/Phase9_taskList.md
View File

@@ -376,7 +376,7 @@ These metrics serve multiple external consumer categories identified during rese
| State Changes Rate | stat | `rate(xrpld_state_changes_total[1h])` |
| Ledgers Closed Rate | stat | `rate(xrpld_ledgers_closed_total[5m]) * 60` |
**Dashboard conventions**: `$node` template variable for `exported_instance` filtering, dark theme, matching existing panel sizes and color schemes.
**Dashboard conventions**: `$node` template variable for `service_instance_id` filtering, dark theme, matching existing panel sizes and color schemes.
**Key new files**: `docker/telemetry/grafana/dashboards/rippled-validator-health.json`
@@ -495,7 +495,7 @@ New files under `docker/telemetry/grafana/provisioning/alerting/`:
| File | Purpose |
| -------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `contactpoints.yaml` | One contact point `xrpld-default` (webhook to a documented placeholder; comments show how to swap for Slack/email). |
| `policies.yaml` | Default notification policy: route all alerts → `xrpld-default`, grouped by `alertname` + `exported_instance`. |
| `policies.yaml` | Default notification policy: route all alerts → `xrpld-default`, grouped by `alertname` + `service_instance_id`. |
| `rules.yaml` | 6 alert rules across 3 groups (below). |
Plus the Alerting section of `docs/telemetry-runbook.md` — operator runbook:
@@ -507,21 +507,21 @@ real receiver.
All rules target Prometheus datasource `uid: prometheus`. Each rule uses the
Grafana rule shape: query (A) → reduce (B, last value) → threshold (C). All
`rate()`/`histogram_quantile()` expressions aggregate with
`sum by (exported_instance)` (or `+ le`) so **each node alerts independently**.
`sum by (service_instance_id)` (or `+ le`) so **each node alerts independently**.
Alert rules run headless, so they cannot use the dashboards' `$node` template
variables — they match all series and group by `exported_instance` instead.
variables — they match all series and group by `service_instance_id` instead.
| Group | Alert | Expression (5m window) | Fires | `for` | severity |
| --------- | --------------------- | --------------------------------------------------------------------------------------------------------- | --------------------- | ----- | -------- |
| Consensus | LedgerHistoryMismatch | `sum by (exported_instance)(rate(xrpld_ledger_history_mismatch_total[5m]))` | `> 0` | 5m | critical |
| Consensus | LedgerCloseStalled | `sum by (exported_instance)(rate(xrpld_ledgers_closed_total[5m]))` | `< 0.001` (≈0) | 3m | critical |
| Validator | ValidationsMissed | `sum by (exported_instance)(rate(xrpld_validation_missed_total[5m]))` | `> 0` | 5m | warning |
| Validator | ValidationsNotChecked | `sum by (exported_instance)(rate(xrpld_validations_checked_total[5m]))` | `< 0.001` (≈0) | 5m | warning |
| Job queue | JobQueueTxOverflow | `sum by (exported_instance)(rate(xrpld_jq_trans_overflow_total[5m]))` | `> 0` | 5m | warning |
| Job queue | JobQueueLatencyHigh | `histogram_quantile(0.99, sum by (le, exported_instance)(rate(xrpld_job_queued_duration_us_bucket[5m])))` | `> 1000000` (µs = 1s) | 5m | warning |
| Group | Alert | Expression (5m window) | Fires | `for` | severity |
| --------- | --------------------- | ----------------------------------------------------------------------------------------------------------- | --------------------- | ----- | -------- |
| Consensus | LedgerHistoryMismatch | `sum by (service_instance_id)(rate(xrpld_ledger_history_mismatch_total[5m]))` | `> 0` | 5m | critical |
| Consensus | LedgerCloseStalled | `sum by (service_instance_id)(rate(xrpld_ledgers_closed_total[5m]))` | `< 0.001` (≈0) | 3m | critical |
| Validator | ValidationsMissed | `sum by (service_instance_id)(rate(xrpld_validation_missed_total[5m]))` | `> 0` | 5m | warning |
| Validator | ValidationsNotChecked | `sum by (service_instance_id)(rate(xrpld_validations_checked_total[5m]))` | `< 0.001` (≈0) | 5m | warning |
| Job queue | JobQueueTxOverflow | `sum by (service_instance_id)(rate(xrpld_jq_trans_overflow_total[5m]))` | `> 0` | 5m | warning |
| Job queue | JobQueueLatencyHigh | `histogram_quantile(0.99, sum by (le, service_instance_id)(rate(xrpld_job_queued_duration_us_bucket[5m])))` | `> 1000000` (µs = 1s) | 5m | warning |
Each rule carries labels `severity` and `category` (consensus/validator/jobqueue)
and annotations `summary` + `description` (with `{{ $labels.exported_instance }}`
and annotations `summary` + `description` (with `{{ $labels.service_instance_id }}`
and `{{ $values.B.Value }}` interpolation).
#### Threshold rationale