mirror of
https://github.com/XRPLF/rippled.git
synced 2026-08-21 22:30:57 +00:00
refactor(telemetry): route dashboards, runbook and collector work to phase-9
These changes were developed on the phase-10 branch but belong to content this branch and its upstreams introduced. Carrying them on phase-10 made its PR diff report churn in files phase-10 does not own, and left each PR claiming a scope that did not match its contents. Moved here from phase-10 (identical content, no functional change): - Dashboards: all 14 existing boards plus the new log-derived-insights board. - Docs: telemetry-runbook.md (minus the workload/benchmark sections, which describe phase-10 tooling) and the new telemetry-glossary.md. - Grafana Cloud + Alloy export path: collector config, compose override, the two .env examples and alloy/config.alloy. - Local stack: otel-collector-config.yaml gains sub-millisecond and second-scale spanmetrics buckets, pins unit=ms, and promotes close_time_correct; integration-test.sh and TESTING.md follow. - Node configs: exported_instance -> service_instance_id in comments; the mainnet sample now logs at warning to bound log volume. - Metrics code: Telemetry.cpp builds the metrics pipeline in the constructor via initMetrics() so the global MeterProvider is published before any subsystem creates a beast::insight instrument, and the histogram view keeps each instrument's own name instead of collapsing them under one series. MetricsRegistry gains a last_close_time gauge and skips negative job-queue durations. OTelCollector drops an unused accessor. - Naming CI: xrpl_work_item joins EXTERNAL_INFRA_LABELS and Rule E accepts the dotted perf-iac resource-attribute form. This must travel with the dashboards and runbook that reference those labels, or the rules fail. - Doxygen input glob no longer recurses dot-directories. Sections describing phase-10 tooling stay on phase-10 and keep their "Future Enhancement" / "Planned, not yet implemented" markers here; phase-10 removes those markers when it lands the tooling.
This commit is contained in:
Pratik Mankawde
@@ -266,7 +266,7 @@ In the Prometheus datasource, set `exemplarTraceIdDestinations` to map the `trac
|
||||
|
||||
**Step 4: Dashboard panel with exemplars**
|
||||
|
||||
Add a timeseries panel over Prometheus (e.g. `histogram_quantile(0.99, rate(xrpld_rpc_duration_seconds_bucket[5m]))`) with `exemplar: true` enabled.
|
||||
Add a timeseries panel over Prometheus (e.g. `histogram_quantile(0.99, rate(rpc_duration_seconds_bucket[5m]))`) with `exemplar: true` enabled.
|
||||
|
||||
This allows clicking on metric data points to jump directly to the related trace.
|
||||
|
||||
|
||||
@@ -336,21 +336,21 @@ xrpld has a mature metrics framework (`beast::insight`) that emits StatsD-format
|
||||
|
||||
### Metric Inventory
|
||||
|
||||
| Category | Group | Type | Count | Key Metrics |
|
||||
| --------------- | ------------------ | ------------- | ---------- | ------------------------------------------------------ |
|
||||
| Node State | `State_Accounting` | Gauge | 10 | `*_duration`, `*_transitions` per operating mode |
|
||||
| Ledger | `LedgerMaster` | Gauge | 2 | `Validated_Ledger_Age`, `Published_Ledger_Age` |
|
||||
| Ledger Fetch | — | Counter | 1 | `ledger_fetches` |
|
||||
| Ledger History | `ledger.history` | Counter | 1 | `mismatch` |
|
||||
| RPC | `rpc` | Counter+Event | 3 | `requests`, `time` (histogram), `size` (histogram) |
|
||||
| Job Queue | — | Gauge+Event | 1 + 2×N | `job_count`, per-job `{name}` and `{name}_q` |
|
||||
| Peer Finder | `Peer_Finder` | Gauge | 2 | `Active_Inbound_Peers`, `Active_Outbound_Peers` |
|
||||
| Overlay | `Overlay` | Gauge | 1 | `Peer_Disconnects` |
|
||||
| Overlay Traffic | per-category | Gauge | 4×57 = 228 | `Bytes_In/Out`, `Messages_In/Out` per traffic category |
|
||||
| Pathfinding | — | Event | 2 | `pathfind_fast`, `pathfind_full` (histograms) |
|
||||
| I/O | — | Event | 1 | `ios_latency` (histogram) |
|
||||
| Resource Mgr | — | Meter | 2 | `warn`, `drop` (rate counters) |
|
||||
| Caches | per-cache | Gauge | 2×N | `{cache}.size`, `{cache}.hit_rate` |
|
||||
| Category | Group | Type | Count | Key Metrics |
|
||||
| --------------- | ------------------ | ------------- | ---------- | ----------------------------------------------------------------------------------------------------------- |
|
||||
| Node State | `State_Accounting` | Gauge | 10 | `*_duration`, `*_transitions` per operating mode |
|
||||
| Ledger | `LedgerMaster` | Gauge | 2 | `Validated_Ledger_Age`, `Published_Ledger_Age` |
|
||||
| Ledger Fetch | — | Counter | 1 | `ledger_fetches` |
|
||||
| Ledger History | `ledger.history` | Counter | 1 | `mismatch` |
|
||||
| RPC | `rpc` | Counter+Event | 3 | `requests`, `time` (histogram), `size` (histogram) |
|
||||
| Job Queue | `jobq` | Gauge+Event | 1 + 2×N | `job_count`, per-job `{name}` and `{name}_q` (emitted with the `jobq_` group prefix, e.g. `jobq_job_count`) |
|
||||
| Peer Finder | `Peer_Finder` | Gauge | 2 | `Active_Inbound_Peers`, `Active_Outbound_Peers` |
|
||||
| Overlay | `Overlay` | Gauge | 1 | `Peer_Disconnects` |
|
||||
| Overlay Traffic | per-category | Gauge | 4×57 = 228 | `Bytes_In/Out`, `Messages_In/Out` per traffic category |
|
||||
| Pathfinding | — | Event | 2 | `pathfind_fast`, `pathfind_full` (histograms) |
|
||||
| I/O | — | Event | 1 | `ios_latency` (histogram) |
|
||||
| Resource Mgr | — | Meter | 2 | `warn`, `drop` (rate counters) |
|
||||
| Caches | per-cache | Gauge | 2×N | `{cache}.size`, `{cache}.hit_rate` |
|
||||
|
||||
**Total**: ~255+ unique metrics (plus dynamic job-type and cache metrics)
|
||||
|
||||
@@ -390,10 +390,10 @@ The `StatsDMeterImpl` in `StatsDCollector.cpp` sends metrics with `|m` suffix, w
|
||||
|
||||
### Exit Criteria
|
||||
|
||||
- [ ] StatsD metrics visible in Prometheus (`curl localhost:9090/api/v1/query?query=xrpld_LedgerMaster_Validated_Ledger_Age`)
|
||||
- [ ] StatsD metrics visible in Prometheus (`curl localhost:9090/api/v1/query?query=ledgermaster_validated_ledger_age`)
|
||||
- [ ] All 3 new Grafana dashboards load without errors
|
||||
- [ ] Integration test verifies at least core StatsD metrics (ledger age, peer counts, RPC requests)
|
||||
- [ ] ~~Meter metrics (`warn`, `drop`) flow correctly after `|m` → `|c` fix~~ — DEFERRED (breaking change, tracked separately)
|
||||
- [ ] ~~Meter metrics (`warn`, `drop`) flow correctly after `|m` → `|c` fix~~ — DEFERRED (breaking change, tracked separately; resolved by Phase 7's OTel Counter mapping)
|
||||
|
||||
---
|
||||
|
||||
@@ -500,7 +500,7 @@ graph LR
|
||||
|
||||
BP -->|"OTLP/gRPC"| D
|
||||
SM -->|"RED metrics"| E
|
||||
R1 -->|"xrpld_* metrics<br/>(native OTLP)"| E
|
||||
R1 -->|"system metrics<br/>(native OTLP)"| E
|
||||
|
||||
E --> F
|
||||
D --> F
|
||||
@@ -581,11 +581,11 @@ See [Phase7_taskList.md](./Phase7_taskList.md) for detailed per-task breakdown.
|
||||
|
||||
### Motivation
|
||||
|
||||
xrpld's `beast::Journal` logs and OpenTelemetry traces are currently two disjoint observability signals. When investigating an issue, operators must manually correlate timestamps between log files and Jaeger/Tempo traces. Phase 8 bridges this gap by injecting trace context (`trace_id`, `span_id`) into every log line emitted within an active span, and ingesting those logs into Grafana Loki via the OTel Collector's filelog receiver.
|
||||
xrpld's `beast::Journal` logs and OpenTelemetry traces are currently two disjoint observability signals. When investigating an issue, operators must manually correlate timestamps between log files and Tempo traces. Phase 8 bridges this gap by injecting trace context (`trace_id`, `span_id`) into every log line emitted within an active span, and ingesting those logs into Grafana Loki via the OTel Collector's filelog receiver.
|
||||
|
||||
#### Gains
|
||||
|
||||
1. **One-click trace-to-log navigation** — Click a trace in Tempo/Jaeger and immediately see the corresponding log lines in Loki, filtered by `trace_id`.
|
||||
1. **One-click trace-to-log navigation** — Click a trace in Tempo and immediately see the corresponding log lines in Loki, filtered by `trace_id`.
|
||||
2. **Reverse lookup (log-to-trace)** — Loki derived fields make `trace_id` values clickable links back to Tempo.
|
||||
3. **Unified observability** — All three pillars (traces, metrics, logs) flow through the same OTel Collector pipeline and are visible in a single Grafana instance.
|
||||
4. **Zero new dependencies in xrpld** — Uses existing OTel SDK headers (`GetSpan`, `GetContext`) already linked in Phase 1.
|
||||
@@ -1215,19 +1215,19 @@ Clear, measurable criteria for each phase.
|
||||
|
||||
### 6.12.6 Success Metrics Summary
|
||||
|
||||
| Phase | Primary Metric | Secondary Metric | Deadline | Status |
|
||||
| -------- | -------------------------------- | --------------------------- | -------------- | ------------------ |
|
||||
| Phase 1 | SDK compiles and runs | Zero overhead when disabled | End of Week 2 | Active |
|
||||
| Phase 2 | 100% RPC coverage | <1ms latency overhead | End of Week 4 | Active |
|
||||
| Phase 3 | Cross-node traces work | <5% throughput impact | End of Week 6 | Active |
|
||||
| Phase 4 | Consensus fully traced | No consensus timing impact | End of Week 8 | Active |
|
||||
| Phase 5 | Production deployment | Operators trained | End of Week 9 | Active |
|
||||
| Phase 6 | StatsD metrics in Prometheus | 3 dashboards operational | End of Week 10 | Active |
|
||||
| Phase 7 | All metrics via OTLP | No StatsD dependency | End of Week 12 | Active |
|
||||
| Phase 8 | trace_id in logs + Loki | Tempo↔Loki correlation | End of Week 13 | Active |
|
||||
| Phase 9 | 68+ new internal metrics in Prom | 2 new dashboards | End of Week 15 | Future Enhancement |
|
||||
| Phase 10 | Full telemetry stack validated | < 3% CPU overhead proven | End of Week 17 | Future Enhancement |
|
||||
| Phase 11 | Third-party metrics via receiver | 4 new dashboards + alerting | End of Week 20 | Future Enhancement |
|
||||
| Phase | Primary Metric | Secondary Metric | Deadline | Status |
|
||||
| -------- | ------------------------------------------------------------------ | --------------------------- | -------------- | ------------------ |
|
||||
| Phase 1 | SDK compiles and runs | Zero overhead when disabled | End of Week 2 | Active |
|
||||
| Phase 2 | 100% RPC coverage | <1ms latency overhead | End of Week 4 | Active |
|
||||
| Phase 3 | Cross-node traces work | <5% throughput impact | End of Week 6 | Active |
|
||||
| Phase 4 | Consensus fully traced | No consensus timing impact | End of Week 8 | Active |
|
||||
| Phase 5 | Production deployment | Operators trained | End of Week 9 | Active |
|
||||
| Phase 6 | StatsD metrics in Prometheus | 3 dashboards operational | End of Week 10 | Active |
|
||||
| Phase 7 | All metrics via OTLP | No StatsD dependency | End of Week 12 | Active |
|
||||
| Phase 8 | trace_id in logs + Loki | Tempo↔Loki correlation | End of Week 13 | Active |
|
||||
| Phase 9 | 68+ new internal metrics in Prom | 2 new dashboards | End of Week 15 | Future Enhancement |
|
||||
| Phase 10 | Full telemetry stack validated; OTel-sourced regression gate in CI | < 3% CPU overhead proven | End of Week 17 | Future Enhancement |
|
||||
| Phase 11 | Third-party metrics via receiver | 4 new dashboards + alerting | End of Week 20 | Future Enhancement |
|
||||
|
||||
---
|
||||
|
||||
@@ -1469,12 +1469,12 @@ The overlay already tracks resource-limit disconnects via `OverlayImpl::Stats::p
|
||||
|
||||
**What to do**:
|
||||
|
||||
- Ensure `xrpld_Overlay_Peer_Disconnects_Charges` appears in the StatsD-to-Prometheus metric name mapping
|
||||
- Ensure `overlay_peer_disconnects_charges` appears in the StatsD-to-Prometheus metric name mapping
|
||||
- Verify the metric appears in Prometheus after StatsD bridge is active
|
||||
|
||||
**File**: `src/xrpld/overlay/detail/OverlayImpl.cpp`
|
||||
|
||||
**Prometheus name**: `xrpld_Overlay_Peer_Disconnects_Charges`
|
||||
**Prometheus name**: `overlay_peer_disconnects_charges`
|
||||
|
||||
---
|
||||
|
||||
@@ -1581,12 +1581,12 @@ class ValidationTracker
|
||||
|
||||
New MetricsRegistry observable gauge for amendment, UNL, and quorum health.
|
||||
|
||||
| Gauge Name | Label `metric=` | Type | Source |
|
||||
| ------------------------ | ------------------- | ------ | ------------------------------------------------- |
|
||||
| `xrpld_validator_health` | `amendment_blocked` | int64 | `app_.getOPs().isAmendmentBlocked()` → 0/1 |
|
||||
| | `unl_blocked` | int64 | `app_.getOPs().isUNLBlocked()` → 0/1 |
|
||||
| | `unl_expiry_days` | double | `app_.validators().expires()` → days until expiry |
|
||||
| | `validation_quorum` | int64 | `app_.validators().quorum()` |
|
||||
| Gauge Name | Label `metric=` | Type | Source |
|
||||
| ------------------ | ------------------- | ------ | ------------------------------------------------- |
|
||||
| `validator_health` | `amendment_blocked` | int64 | `app_.getOPs().isAmendmentBlocked()` → 0/1 |
|
||||
| | `unl_blocked` | int64 | `app_.getOPs().isUNLBlocked()` → 0/1 |
|
||||
| | `unl_expiry_days` | double | `app_.validators().expires()` → days until expiry |
|
||||
| | `validation_quorum` | int64 | `app_.validators().quorum()` |
|
||||
|
||||
**File**: `src/xrpld/telemetry/MetricsRegistry.cpp` (new gauge callback in `registerAsyncGauges()`)
|
||||
|
||||
@@ -1602,12 +1602,12 @@ New MetricsRegistry observable gauge for amendment, UNL, and quorum health.
|
||||
|
||||
New MetricsRegistry observable gauge for peer health aggregates.
|
||||
|
||||
| Gauge Name | Label `metric=` | Type | Source |
|
||||
| -------------------- | -------------------------- | ------ | ------------------------------------------ |
|
||||
| `xrpld_peer_quality` | `peer_latency_p90_ms` | double | Iterate peers, compute P90 from `latency_` |
|
||||
| | `peers_insane_count` | int64 | Count peers with `tracking_ == diverged` |
|
||||
| | `peers_higher_version_pct` | double | Compare `getVersion()` to own version |
|
||||
| | `upgrade_recommended` | int64 | 1 if `peers_higher_version_pct > 60%` |
|
||||
| Gauge Name | Label `metric=` | Type | Source |
|
||||
| -------------- | -------------------------- | ------ | ------------------------------------------ |
|
||||
| `peer_quality` | `peer_latency_p90_ms` | double | Iterate peers, compute P90 from `latency_` |
|
||||
| | `peers_insane_count` | int64 | Count peers with `tracking_ == diverged` |
|
||||
| | `peers_higher_version_pct` | double | Compare `getVersion()` to own version |
|
||||
| | `upgrade_recommended` | int64 | 1 if `peers_higher_version_pct > 60%` |
|
||||
|
||||
**Implementation note**: The callback iterates `app_.overlay().foreach(...)` to collect per-peer latency and version data. This runs every 10s on the metrics reader thread — acceptable overhead for ~50-200 peers.
|
||||
|
||||
@@ -1626,13 +1626,13 @@ New MetricsRegistry observable gauge for peer health aggregates.
|
||||
|
||||
New MetricsRegistry observable gauge for fee and ledger metrics.
|
||||
|
||||
| Gauge Name | Label `metric=` | Type | Source |
|
||||
| ---------------------- | -------------------- | ------ | ----------------------------------------- |
|
||||
| `xrpld_ledger_economy` | `base_fee_xrp` | double | `app_.getFeeTrack().getBaseFee()` → drops |
|
||||
| | `reserve_base_xrp` | double | From validated ledger fee settings |
|
||||
| | `reserve_inc_xrp` | double | From validated ledger fee settings |
|
||||
| | `ledger_age_seconds` | double | `now - lastValidatedCloseTime` |
|
||||
| | `transaction_rate` | double | Derived: tx count delta / time delta |
|
||||
| Gauge Name | Label `metric=` | Type | Source |
|
||||
| ---------------- | -------------------- | ------ | ----------------------------------------- |
|
||||
| `ledger_economy` | `base_fee_xrp` | double | `app_.getFeeTrack().getBaseFee()` → drops |
|
||||
| | `reserve_base_xrp` | double | From validated ledger fee settings |
|
||||
| | `reserve_inc_xrp` | double | From validated ledger fee settings |
|
||||
| | `ledger_age_seconds` | double | `now - lastValidatedCloseTime` |
|
||||
| | `transaction_rate` | double | Derived: tx count delta / time delta |
|
||||
|
||||
**File**: `src/xrpld/telemetry/MetricsRegistry.cpp`
|
||||
|
||||
@@ -1648,10 +1648,10 @@ New MetricsRegistry observable gauge for fee and ledger metrics.
|
||||
|
||||
New MetricsRegistry observable gauge for node state duration.
|
||||
|
||||
| Gauge Name | Label `metric=` | Type | Source |
|
||||
| ---------------------- | ------------------------------- | ------ | ------------------------------------------------ |
|
||||
| `xrpld_state_tracking` | `state_value` | int64 | 0-7 numeric encoding matching external dashboard |
|
||||
| | `time_in_current_state_seconds` | double | `now - lastModeChangeTime` |
|
||||
| Gauge Name | Label `metric=` | Type | Source |
|
||||
| ---------------- | ------------------------------- | ------ | ------------------------------------------------ |
|
||||
| `state_tracking` | `state_value` | int64 | 0-7 numeric encoding matching external dashboard |
|
||||
| | `time_in_current_state_seconds` | double | `now - lastModeChangeTime` |
|
||||
|
||||
**State value encoding**:
|
||||
|
||||
@@ -1680,9 +1680,9 @@ xrpld's `OperatingMode` enum maps 0-4 (DISCONNECTED through FULL). The external
|
||||
|
||||
**Task 7.13: Storage Detail Observable Gauge**
|
||||
|
||||
| Gauge Name | Label `metric=` | Type | Source |
|
||||
| ---------------------- | --------------------- | ----- | ---------------------------------------------------- |
|
||||
| `xrpld_storage_detail` | `stored_object_bytes` | int64 | `Database::getStoreSize()` — cumulative object bytes |
|
||||
| Gauge Name | Label `metric=` | Type | Source |
|
||||
| ---------------- | --------------------- | ----- | ---------------------------------------------------- |
|
||||
| `storage_detail` | `stored_object_bytes` | int64 | `Database::getStoreSize()` — cumulative object bytes |
|
||||
|
||||
This is not a filesystem measurement. `getStoreSize()` sums the object payloads this
|
||||
process has written, so it excludes NuDB's keys, bucket padding and log, and it
|
||||
@@ -1709,15 +1709,15 @@ misdescribed it and the old name implied an on-disk size it never reported.
|
||||
|
||||
New counters incremented at event sites. Declared in MetricsRegistry, recording sites added in consensus/overlay/network code.
|
||||
|
||||
| Counter Name | Increment Site | Source File |
|
||||
| ----------------------------------- | -------------------------------- | --------------------- |
|
||||
| `xrpld_ledgers_closed_total` | `onAccept()` in consensus | RCLConsensus.cpp |
|
||||
| `xrpld_validations_sent_total` | `validate()` in consensus | RCLConsensus.cpp |
|
||||
| `xrpld_validations_checked_total` | Network validation received | LedgerMaster.cpp |
|
||||
| `xrpld_validation_agreements_total` | ValidationTracker reconciliation | ValidationTracker.cpp |
|
||||
| `xrpld_validation_missed_total` | ValidationTracker reconciliation | ValidationTracker.cpp |
|
||||
| `xrpld_state_changes_total` | `setMode()` in NetworkOPs | NetworkOPs.cpp |
|
||||
| `xrpld_jq_trans_overflow_total` | Job queue overflow path | JobQueue.cpp |
|
||||
| Counter Name | Increment Site | Source File |
|
||||
| ----------------------------- | -------------------------------- | --------------------- |
|
||||
| `ledgers_closed_total` | `onAccept()` in consensus | RCLConsensus.cpp |
|
||||
| `validations_sent_total` | `validate()` in consensus | RCLConsensus.cpp |
|
||||
| `validations_checked_total` | Network validation received | LedgerMaster.cpp |
|
||||
| `validation_agreements_total` | ValidationTracker reconciliation | ValidationTracker.cpp |
|
||||
| `validation_missed_total` | ValidationTracker reconciliation | ValidationTracker.cpp |
|
||||
| `state_changes_total` | `setMode()` in NetworkOPs | NetworkOPs.cpp |
|
||||
| `jq_trans_overflow_total` | Job queue overflow path | JobQueue.cpp |
|
||||
|
||||
**Key modified files**:
|
||||
|
||||
@@ -1738,14 +1738,14 @@ New counters incremented at event sites. Declared in MetricsRegistry, recording
|
||||
|
||||
Reads from the `ValidationTracker` (Task 7.8) to export rolling window stats.
|
||||
|
||||
| Gauge Name | Label `metric=` | Type | Source |
|
||||
| ---------------------------- | ------------------- | ------ | --------------------------- |
|
||||
| `xrpld_validation_agreement` | `agreement_pct_1h` | double | `tracker.agreementPct1h()` |
|
||||
| | `agreements_1h` | int64 | `tracker.agreements1h()` |
|
||||
| | `missed_1h` | int64 | `tracker.missed1h()` |
|
||||
| | `agreement_pct_24h` | double | `tracker.agreementPct24h()` |
|
||||
| | `agreements_24h` | int64 | `tracker.agreements24h()` |
|
||||
| | `missed_24h` | int64 | `tracker.missed24h()` |
|
||||
| Gauge Name | Label `metric=` | Type | Source |
|
||||
| ---------------------- | ------------------- | ------ | --------------------------- |
|
||||
| `validation_agreement` | `agreement_pct_1h` | double | `tracker.agreementPct1h()` |
|
||||
| | `agreements_1h` | int64 | `tracker.agreements1h()` |
|
||||
| | `missed_1h` | int64 | `tracker.missed1h()` |
|
||||
| | `agreement_pct_24h` | double | `tracker.agreementPct24h()` |
|
||||
| | `agreements_24h` | int64 | `tracker.agreements24h()` |
|
||||
| | `missed_24h` | int64 | `tracker.missed24h()` |
|
||||
|
||||
**File**: `src/xrpld/telemetry/MetricsRegistry.cpp`
|
||||
|
||||
@@ -1765,21 +1765,21 @@ Reads from the `ValidationTracker` (Task 7.8) to export rolling window stats.
|
||||
|
||||
New Grafana dashboard: `validator-health.json`
|
||||
|
||||
| Panel | Type | PromQL |
|
||||
| -------------------------- | ---------- | -------------------------------------------------------------- |
|
||||
| Agreement % (1h) | stat | `xrpld_validation_agreement{metric="agreement_pct_1h"}` |
|
||||
| Agreement % (24h) | stat | `xrpld_validation_agreement{metric="agreement_pct_24h"}` |
|
||||
| Agreements vs Missed (1h) | bargauge | `agreements_1h` and `missed_1h` side by side |
|
||||
| Agreements vs Missed (24h) | bargauge | `agreements_24h` and `missed_24h` side by side |
|
||||
| Validation Rate | stat | `rate(xrpld_validations_sent_total[5m]) * 60` |
|
||||
| Validations Checked Rate | stat | `rate(xrpld_validations_checked_total[5m]) * 60` |
|
||||
| Amendment Blocked | stat | `xrpld_validator_health{metric="amendment_blocked"}` |
|
||||
| UNL Expiry (days) | stat | `xrpld_validator_health{metric="unl_expiry_days"}` |
|
||||
| Validation Quorum | stat | `xrpld_validator_health{metric="validation_quorum"}` |
|
||||
| State Value Timeline | timeseries | `xrpld_state_tracking{metric="state_value"}` |
|
||||
| Time in Current State | stat | `xrpld_state_tracking{metric="time_in_current_state_seconds"}` |
|
||||
| State Changes Rate | stat | `rate(xrpld_state_changes_total[1h])` |
|
||||
| Ledgers Closed Rate | stat | `rate(xrpld_ledgers_closed_total[5m]) * 60` |
|
||||
| Panel | Type | PromQL |
|
||||
| -------------------------- | ---------- | -------------------------------------------------------- |
|
||||
| Agreement % (1h) | stat | `validation_agreement{metric="agreement_pct_1h"}` |
|
||||
| Agreement % (24h) | stat | `validation_agreement{metric="agreement_pct_24h"}` |
|
||||
| Agreements vs Missed (1h) | bargauge | `agreements_1h` and `missed_1h` side by side |
|
||||
| Agreements vs Missed (24h) | bargauge | `agreements_24h` and `missed_24h` side by side |
|
||||
| Validation Rate | stat | `rate(validations_sent_total[5m]) * 60` |
|
||||
| Validations Checked Rate | stat | `rate(validations_checked_total[5m]) * 60` |
|
||||
| Amendment Blocked | stat | `validator_health{metric="amendment_blocked"}` |
|
||||
| UNL Expiry (days) | stat | `validator_health{metric="unl_expiry_days"}` |
|
||||
| Validation Quorum | stat | `validator_health{metric="validation_quorum"}` |
|
||||
| State Value Timeline | timeseries | `state_tracking{metric="state_value"}` |
|
||||
| Time in Current State | stat | `state_tracking{metric="time_in_current_state_seconds"}` |
|
||||
| State Changes Rate | stat | `rate(state_changes_total[1h])` |
|
||||
| Ledgers Closed Rate | stat | `rate(ledgers_closed_total[5m]) * 60` |
|
||||
|
||||
**Dashboard conventions**: `$node` template variable for `service_instance_id` filtering, dark theme, matching existing panel sizes and color schemes.
|
||||
|
||||
@@ -1789,14 +1789,14 @@ New Grafana dashboard: `validator-health.json`
|
||||
|
||||
New Grafana dashboard: `peer-quality.json`
|
||||
|
||||
| Panel | Type | PromQL |
|
||||
| ---------------------- | ---------- | -------------------------------------------------------------- |
|
||||
| P90 Peer Latency | timeseries | `xrpld_peer_quality{metric="peer_latency_p90_ms"}` |
|
||||
| Insane/Diverged Peers | stat | `xrpld_peer_quality{metric="peers_insane_count"}` |
|
||||
| Higher Version Peers % | stat | `xrpld_peer_quality{metric="peers_higher_version_pct"}` |
|
||||
| Upgrade Recommended | stat | `xrpld_peer_quality{metric="upgrade_recommended"}` |
|
||||
| Resource Disconnects | timeseries | `xrpld_Overlay_Peer_Disconnects_Charges` |
|
||||
| Inbound vs Outbound | bargauge | `xrpld_Peer_Finder_Active_Inbound_Peers`, `..._Outbound_Peers` |
|
||||
| Panel | Type | PromQL |
|
||||
| ---------------------- | ---------- | -------------------------------------------------------- |
|
||||
| P90 Peer Latency | timeseries | `peer_quality{metric="peer_latency_p90_ms"}` |
|
||||
| Insane/Diverged Peers | stat | `peer_quality{metric="peers_insane_count"}` |
|
||||
| Higher Version Peers % | stat | `peer_quality{metric="peers_higher_version_pct"}` |
|
||||
| Upgrade Recommended | stat | `peer_quality{metric="upgrade_recommended"}` |
|
||||
| Resource Disconnects | timeseries | `overlay_peer_disconnects_charges` |
|
||||
| Inbound vs Outbound | bargauge | `peer_finder_active_inbound_peers`, `..._outbound_peers` |
|
||||
|
||||
---
|
||||
|
||||
@@ -1804,13 +1804,13 @@ New Grafana dashboard: `peer-quality.json`
|
||||
|
||||
Add a "Ledger Economy" row to the existing `node-health.json` dashboard:
|
||||
|
||||
| Panel | Type | PromQL |
|
||||
| -------------------- | ---------- | --------------------------------------------------- |
|
||||
| Base Fee (drops) | stat | `xrpld_ledger_economy{metric="base_fee_xrp"}` |
|
||||
| Reserve Base (drops) | stat | `xrpld_ledger_economy{metric="reserve_base_xrp"}` |
|
||||
| Reserve Inc (drops) | stat | `xrpld_ledger_economy{metric="reserve_inc_xrp"}` |
|
||||
| Ledger Age | stat | `xrpld_ledger_economy{metric="ledger_age_seconds"}` |
|
||||
| Transaction Rate | timeseries | `xrpld_ledger_economy{metric="transaction_rate"}` |
|
||||
| Panel | Type | PromQL |
|
||||
| -------------------- | ---------- | --------------------------------------------- |
|
||||
| Base Fee (drops) | stat | `ledger_economy{metric="base_fee_xrp"}` |
|
||||
| Reserve Base (drops) | stat | `ledger_economy{metric="reserve_base_xrp"}` |
|
||||
| Reserve Inc (drops) | stat | `ledger_economy{metric="reserve_inc_xrp"}` |
|
||||
| Ledger Age | stat | `ledger_economy{metric="ledger_age_seconds"}` |
|
||||
| Transaction Rate | timeseries | `ledger_economy{metric="transaction_rate"}` |
|
||||
|
||||
---
|
||||
|
||||
@@ -1837,21 +1837,21 @@ Add checks to `validate_telemetry.py` for all new span attributes and metrics.
|
||||
|
||||
**New metric existence checks (~13)**:
|
||||
|
||||
| Metric Name |
|
||||
| -------------------------------------------------------- |
|
||||
| `xrpld_validation_agreement{metric="agreement_pct_1h"}` |
|
||||
| `xrpld_validation_agreement{metric="agreement_pct_24h"}` |
|
||||
| `xrpld_validator_health{metric="amendment_blocked"}` |
|
||||
| `xrpld_validator_health{metric="unl_expiry_days"}` |
|
||||
| `xrpld_peer_quality{metric="peer_latency_p90_ms"}` |
|
||||
| `xrpld_peer_quality{metric="peers_insane_count"}` |
|
||||
| `xrpld_ledger_economy{metric="base_fee_xrp"}` |
|
||||
| `xrpld_ledger_economy{metric="transaction_rate"}` |
|
||||
| `xrpld_state_tracking{metric="state_value"}` |
|
||||
| `xrpld_ledgers_closed_total` |
|
||||
| `xrpld_validations_sent_total` |
|
||||
| `xrpld_state_changes_total` |
|
||||
| `xrpld_storage_detail{metric="stored_object_bytes"}` |
|
||||
| Metric Name |
|
||||
| -------------------------------------------------- |
|
||||
| `validation_agreement{metric="agreement_pct_1h"}` |
|
||||
| `validation_agreement{metric="agreement_pct_24h"}` |
|
||||
| `validator_health{metric="amendment_blocked"}` |
|
||||
| `validator_health{metric="unl_expiry_days"}` |
|
||||
| `peer_quality{metric="peer_latency_p90_ms"}` |
|
||||
| `peer_quality{metric="peers_insane_count"}` |
|
||||
| `ledger_economy{metric="base_fee_xrp"}` |
|
||||
| `ledger_economy{metric="transaction_rate"}` |
|
||||
| `state_tracking{metric="state_value"}` |
|
||||
| `ledgers_closed_total` |
|
||||
| `validations_sent_total` |
|
||||
| `state_changes_total` |
|
||||
| `storage_detail{metric="stored_object_bytes"}` |
|
||||
|
||||
**New dashboard load checks (~3)**:
|
||||
|
||||
@@ -1884,36 +1884,36 @@ Port 18 alert rules from the external `xrpl-validator-dashboard` to Grafana aler
|
||||
|
||||
**Critical Group** (8 rules, eval interval 10s):
|
||||
|
||||
| Rule | Condition | For |
|
||||
| ------------------- | ------------------------------------------------------------- | --- |
|
||||
| Agreement Below 90% | `xrpld_validation_agreement{metric="agreement_pct_24h"} < 90` | 30s |
|
||||
| Not Proposing | `xrpld_state_tracking{metric="state_value"} < 6` | 10s |
|
||||
| Unhealthy State | `xrpld_state_tracking{metric="state_value"} < 4` | 10s |
|
||||
| Amendment Blocked | `xrpld_validator_health{metric="amendment_blocked"} == 1` | 1m |
|
||||
| UNL Expiring | `xrpld_validator_health{metric="unl_expiry_days"} < 14` | 1h |
|
||||
| High IO Latency | `histogram_quantile(0.95, xrpld_ios_latency_bucket) > 50` | 1m |
|
||||
| High Load Factor | `xrpld_load_factor_metrics{metric="load_factor"} > 1000` | 1m |
|
||||
| Peer Count Critical | `xrpld_server_info{metric="peers"} < 5` | 1m |
|
||||
| Rule | Condition | For |
|
||||
| ------------------- | ------------------------------------------------------- | --- |
|
||||
| Agreement Below 90% | `validation_agreement{metric="agreement_pct_24h"} < 90` | 30s |
|
||||
| Not Proposing | `state_tracking{metric="state_value"} < 6` | 10s |
|
||||
| Unhealthy State | `state_tracking{metric="state_value"} < 4` | 10s |
|
||||
| Amendment Blocked | `validator_health{metric="amendment_blocked"} == 1` | 1m |
|
||||
| UNL Expiring | `validator_health{metric="unl_expiry_days"} < 14` | 1h |
|
||||
| High IO Latency | `histogram_quantile(0.95, ios_latency_bucket) > 50` | 1m |
|
||||
| High Load Factor | `load_factor_metrics{metric="load_factor"} > 1000` | 1m |
|
||||
| Peer Count Critical | `server_info{metric="peers"} < 5` | 1m |
|
||||
|
||||
**Network Group** (3 rules, eval interval 10s):
|
||||
|
||||
| Rule | Condition | For |
|
||||
| ------------------------- | ----------------------------------------------------------------- | --- |
|
||||
| Peer Drop >10% | `delta(xrpld_server_info{metric="peers"}[30s]) / ... * 100 < -10` | 30s |
|
||||
| Peer Drop >30% | Same formula, threshold -30 | 30s |
|
||||
| P90 Latency + Disconnects | `peer_latency_p90_ms > 500 AND rate(disconnects) > 0` | 2m |
|
||||
| Rule | Condition | For |
|
||||
| ------------------------- | ----------------------------------------------------------- | --- |
|
||||
| Peer Drop >10% | `delta(server_info{metric="peers"}[30s]) / ... * 100 < -10` | 30s |
|
||||
| Peer Drop >30% | Same formula, threshold -30 | 30s |
|
||||
| P90 Latency + Disconnects | `peer_latency_p90_ms > 500 AND rate(disconnects) > 0` | 2m |
|
||||
|
||||
**Performance Group** (7 rules, eval interval 10s):
|
||||
|
||||
| Rule | Condition | For |
|
||||
| ------------------- | ------------------------------------------------------------ | --- |
|
||||
| CPU High | Per-core CPU > 80% | 2m |
|
||||
| Memory Critical | Memory usage > 90% | 1m |
|
||||
| Disk Warning | Disk usage > 85% | 2m |
|
||||
| Job Queue Overflow | `rate(xrpld_jq_trans_overflow_total[5m]) > 0` | 1m |
|
||||
| Upgrade Recommended | `xrpld_peer_quality{metric="peers_higher_version_pct"} > 60` | 1m |
|
||||
| TX Rate Drop | Transaction rate dropped > 50% in 5m window | 5m |
|
||||
| Stale Ledger | `xrpld_ledger_economy{metric="ledger_age_seconds"} > 30` | 1m |
|
||||
| Rule | Condition | For |
|
||||
| ------------------- | ------------------------------------------------------ | --- |
|
||||
| CPU High | Per-core CPU > 80% | 2m |
|
||||
| Memory Critical | Memory usage > 90% | 1m |
|
||||
| Disk Warning | Disk usage > 85% | 2m |
|
||||
| Job Queue Overflow | `rate(jq_trans_overflow_total[5m]) > 0` | 1m |
|
||||
| Upgrade Recommended | `peer_quality{metric="peers_higher_version_pct"} > 60` | 1m |
|
||||
| TX Rate Drop | Transaction rate dropped > 50% in 5m window | 5m |
|
||||
| Stale Ledger | `ledger_economy{metric="ledger_age_seconds"} > 30` | 1m |
|
||||
|
||||
**Notification channels**: Template configs for Email/SMTP, Discord, Slack, PagerDuty.
|
||||
|
||||
|
||||
@@ -59,85 +59,19 @@
|
||||
|
||||
## 8.2 Span Hierarchy Visualization
|
||||
|
||||
> **TxQ** = Transaction Queue
|
||||
The authoritative span-flow diagrams — a master overview plus per-stage
|
||||
flowcharts (ingress, the shared apply pipeline, the consensus round, ledger
|
||||
finalize, and the pathfinding / ledger-acquire side flows) — live in the operator
|
||||
runbook. They map every span onto the **real xrpld control flow and XRPL protocol
|
||||
order** (verified against code and `docs/consensus.md`, with file:line evidence),
|
||||
label every node and branch with the span that represents that state or
|
||||
transition, and call out where the OpenTelemetry span parent links diverge from
|
||||
that flow.
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph trace["Trace: Transaction Lifecycle"]
|
||||
rpc["rpc.request<br/>(entry point)"]
|
||||
validate["tx.validate"]
|
||||
relay["tx.relay<br/>(parent span)"]
|
||||
> **See**: [docs/telemetry-runbook.md § Protocol Span Flow](../docs/telemetry-runbook.md#protocol-span-flow).
|
||||
|
||||
subgraph peers["Peer Spans"]
|
||||
p1["peer.send<br/>Peer A"]
|
||||
p2["peer.send<br/>Peer B"]
|
||||
p3["peer.send<br/>Peer C"]
|
||||
end
|
||||
|
||||
subgraph pathfinding["PathFinding Spans"]
|
||||
pathfind["pathfind.request"]
|
||||
pathcomp["pathfind.compute"]
|
||||
end
|
||||
|
||||
consensus["consensus.round"]
|
||||
apply["tx.apply"]
|
||||
|
||||
subgraph txqueue["TxQ Spans"]
|
||||
txq["txq.enqueue"]
|
||||
txqApply["txq.apply"]
|
||||
end
|
||||
|
||||
feeCalc["fee.escalate"]
|
||||
end
|
||||
|
||||
subgraph validators["Validator Spans"]
|
||||
valFetch["validator.list.fetch"]
|
||||
valManifest["validator.manifest"]
|
||||
end
|
||||
|
||||
rpc --> validate
|
||||
rpc --> pathfind
|
||||
pathfind --> pathcomp
|
||||
validate --> relay
|
||||
relay --> p1
|
||||
relay --> p2
|
||||
relay --> p3
|
||||
p1 -.->|"context propagation"| consensus
|
||||
consensus --> apply
|
||||
apply --> txq
|
||||
txq --> txqApply
|
||||
txq --> feeCalc
|
||||
|
||||
style trace fill:#0f172a,stroke:#020617,color:#fff
|
||||
style peers fill:#1e3a8a,stroke:#172554,color:#fff
|
||||
style pathfinding fill:#134e4a,stroke:#0f766e,color:#fff
|
||||
style txqueue fill:#064e3b,stroke:#047857,color:#fff
|
||||
style validators fill:#4c1d95,stroke:#6d28d9,color:#fff
|
||||
style rpc fill:#1d4ed8,stroke:#1e40af,color:#fff
|
||||
style validate fill:#047857,stroke:#064e3b,color:#fff
|
||||
style relay fill:#047857,stroke:#064e3b,color:#fff
|
||||
style p1 fill:#0e7490,stroke:#155e75,color:#fff
|
||||
style p2 fill:#0e7490,stroke:#155e75,color:#fff
|
||||
style p3 fill:#0e7490,stroke:#155e75,color:#fff
|
||||
style consensus fill:#fef3c7,stroke:#fde68a,color:#1e293b
|
||||
style apply fill:#047857,stroke:#064e3b,color:#fff
|
||||
style pathfind fill:#0e7490,stroke:#155e75,color:#fff
|
||||
style pathcomp fill:#0e7490,stroke:#155e75,color:#fff
|
||||
style txq fill:#047857,stroke:#064e3b,color:#fff
|
||||
style txqApply fill:#047857,stroke:#064e3b,color:#fff
|
||||
style feeCalc fill:#047857,stroke:#064e3b,color:#fff
|
||||
style valFetch fill:#6d28d9,stroke:#4c1d95,color:#fff
|
||||
style valManifest fill:#6d28d9,stroke:#4c1d95,color:#fff
|
||||
```
|
||||
|
||||
**Reading the diagram:**
|
||||
|
||||
- **rpc.request (blue, top)**: The entry point — every traced transaction starts as an RPC call; this root span is the parent of all downstream work.
|
||||
- **tx.validate and pathfind.request (green/teal, first fork)**: The RPC request fans out into transaction validation and, for cross-currency payments, a PathFinding branch (`pathfind.request` -> `pathfind.compute`).
|
||||
- **tx.relay -> Peer Spans (teal, middle)**: After validation, the transaction is relayed to peers A, B, and C in parallel; each `peer.send` is a sibling child span showing fan-out across the network.
|
||||
- **context propagation (dashed arrow)**: The dotted line from `peer.send Peer A` to `consensus.round` represents the trace context crossing a node boundary — the receiving validator picks up the same `trace_id` and continues the trace.
|
||||
- **consensus.round -> tx.apply -> TxQ Spans (green, lower)**: Once consensus accepts the transaction, it is applied to the ledger; the TxQ spans (`txq.enqueue`, `txq.apply`, `fee.escalate`) capture queue depth and fee escalation behavior.
|
||||
- **Validator Spans (purple, detached)**: `validator.list.fetch` and `validator.manifest` are independent workflows for UNL management — they run on their own traces and are linked to consensus via Span Links, not parent-child relationships.
|
||||
The full span inventory (names, attributes, parents as instrumented) is in
|
||||
[09-data-collection-reference.md §1](./09-data-collection-reference.md#1-opentelemetry-spans).
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -43,7 +43,7 @@ graph LR
|
||||
BP -->|"OTLP/gRPC :4317"| D
|
||||
|
||||
SM -->|"span_calls_total<br/>span_duration_ms<br/>(6 dimension labels)"| E
|
||||
R1 -->|"* gauges<br/>* counters<br/>* histograms"| E
|
||||
R1 -->|"gauges, counters,<br/>histograms (OTLP)"| E
|
||||
|
||||
E -->|"Prometheus<br/>data source"| F
|
||||
D -->|"Tempo<br/>data source"| F
|
||||
@@ -732,15 +732,24 @@ sampled at the same instant.
|
||||
|
||||
### 3.3 Deployment-Tier Template Variables
|
||||
|
||||
Every dashboard carries four filtering template variables (each variable name
|
||||
matches its Prometheus label), letting one Grafana stack be sliced by tier:
|
||||
Every dashboard carries seven filtering template variables (each variable name
|
||||
matches its Prometheus label), letting one Grafana stack be sliced by tier and
|
||||
by perf-comparison run:
|
||||
|
||||
| Variable | Source label | Description |
|
||||
| ------------------------- | ------------------------ | ------------------------------------------------------------ |
|
||||
| `$node` | `exported_instance` | Filter by xrpld node instance |
|
||||
| `$service_name` | `service_name` | Filter by service (`service.name`, e.g. `xrpld`) |
|
||||
| `$deployment_environment` | `deployment_environment` | Filter by deployment tier (`local` / `test` / `ci` / `prod`) |
|
||||
| `$xrpl_network_type` | `xrpl_network_type` | Filter by network (`mainnet` / `testnet` / `devnet`) |
|
||||
| Variable | Source label | Description |
|
||||
| ------------------------- | ------------------------ | ---------------------------------------------------------------- |
|
||||
| `$node` | `service_instance_id` | Filter by xrpld node instance |
|
||||
| `$service_name` | `service_name` | Filter by service (`service.name`, e.g. `xrpld`) |
|
||||
| `$deployment_environment` | `deployment_environment` | Filter by deployment tier (`local` / `test` / `ci` / `prod`) |
|
||||
| `$xrpl_network_type` | `xrpl_network_type` | Filter by network (`mainnet` / `testnet` / `devnet` / `perf`) |
|
||||
| `$xrpl_work_item` | `xrpl_work_item` | Filter by perf-iac work item / ticket (e.g. `RIPD-7455`) |
|
||||
| `$xrpl_branch` | `xrpl_branch` | Filter by comparison side (`baseline:<ref>:<commit>` / `test:…`) |
|
||||
| `$xrpl_node_role` | `xrpl_node_role` | Filter by node role (`validator` / `peer`) |
|
||||
|
||||
The last three are populated only during perf-iac comparison runs (stamped as
|
||||
resource attributes by perf-iac's own alloy pipeline, not the repo collector).
|
||||
Outside those runs the labels are absent; the filters default to **All**, which
|
||||
matches series lacking the label so every dashboard still renders.
|
||||
|
||||
See [telemetry-runbook.md](../docs/telemetry-runbook.md) "Deployment Tiers"
|
||||
for how the tier attributes are set and reach metrics.
|
||||
@@ -869,7 +878,7 @@ Example:
|
||||
2024-Jan-15 10:30:45.123456 UTC LedgerMaster:NFO trace_id=abc123def456789012345678abcdef01 span_id=0123456789abcdef Validated ledger 42
|
||||
```
|
||||
|
||||
- **`trace_id=<hex32>`** — 32-character lowercase hex trace identifier. Links to the distributed trace in Tempo/Jaeger.
|
||||
- **`trace_id=<hex32>`** — 32-character lowercase hex trace identifier. Links to the distributed trace in Tempo.
|
||||
- **`span_id=<hex16>`** — 16-character lowercase hex span identifier. Identifies the specific span within the trace.
|
||||
- **Only present** when the log is emitted within an active OTel span. Log lines outside of traced code paths have no trace context fields.
|
||||
|
||||
@@ -942,16 +951,17 @@ async callbacks for new categories.
|
||||
|
||||
#### Server Info (via OTel MetricsRegistry)
|
||||
|
||||
| Prometheus Metric | Type | Labels | Description |
|
||||
| --------------------------------------------------- | ----- | -------- | -------------------------------------------- |
|
||||
| `server_info{metric="server_state"}` | Gauge | `metric` | Operating mode (0=DISCONNECTED .. 4=FULL) |
|
||||
| `server_info{metric="uptime"}` | Gauge | `metric` | Seconds since server start |
|
||||
| `server_info{metric="peers"}` | Gauge | `metric` | Total connected peers |
|
||||
| `server_info{metric="validated_ledger_seq"}` | Gauge | `metric` | Validated ledger sequence number |
|
||||
| `server_info{metric="ledger_current_index"}` | Gauge | `metric` | Current open ledger sequence |
|
||||
| `server_info{metric="peer_disconnects_resources"}` | Gauge | `metric` | Cumulative resource-related peer disconnects |
|
||||
| `server_info{metric="last_close_proposers"}` | Gauge | `metric` | Proposers in last closed round |
|
||||
| `server_info{metric="last_close_converge_time_ms"}` | Gauge | `metric` | Last close convergence time (milliseconds) |
|
||||
| Prometheus Metric | Type | Labels | Description |
|
||||
| --------------------------------------------------- | ----- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `server_info{metric="server_state"}` | Gauge | `metric` | Operating mode (0=DISCONNECTED .. 4=FULL) |
|
||||
| `server_info{metric="uptime"}` | Gauge | `metric` | Seconds since server start |
|
||||
| `server_info{metric="peers"}` | Gauge | `metric` | Total connected peers |
|
||||
| `server_info{metric="validated_ledger_seq"}` | Gauge | `metric` | Validated ledger sequence number |
|
||||
| `server_info{metric="ledger_current_index"}` | Gauge | `metric` | Current open ledger sequence |
|
||||
| `server_info{metric="peer_disconnects_resources"}` | Gauge | `metric` | Cumulative resource-related peer disconnects |
|
||||
| `server_info{metric="last_close_proposers"}` | Gauge | `metric` | Proposers in last closed round |
|
||||
| `server_info{metric="last_close_converge_time_ms"}` | Gauge | `metric` | Last close convergence time (milliseconds) |
|
||||
| `server_info{metric="last_close_time"}` | Gauge | `metric` | Network close time of last closed ledger (NetClock secs since XRPL epoch). Query `time() - (value + 946684800)` for last-close age (staleness). Use `1/rate(ledgers_closed_total)` — not a gauge delta — for the close interval |
|
||||
|
||||
#### Build Info (via OTel MetricsRegistry)
|
||||
|
||||
|
||||
@@ -209,7 +209,7 @@ The appendix contains a glossary of OpenTelemetry and xrpld-specific terms, refe
|
||||
|
||||
## 9. Data Collection Reference
|
||||
|
||||
A single-source-of-truth reference documenting every piece of telemetry data collected by xrpld. Covers all 16 OpenTelemetry spans with their 22 attributes, all StatsD metrics (gauges, counters, histograms, overlay traffic), SpanMetrics-derived Prometheus metrics, and all 10 Grafana dashboards. Includes Jaeger search guides and Prometheus query examples.
|
||||
A single-source-of-truth reference documenting every piece of telemetry data collected by xrpld. Covers all 16 OpenTelemetry spans with their 22 attributes, all StatsD metrics (gauges, counters, histograms, overlay traffic), SpanMetrics-derived Prometheus metrics, and all 10 Grafana dashboards. Includes Tempo search guides and Prometheus query examples.
|
||||
|
||||
➡️ **[View Data Collection Reference](./09-data-collection-reference.md)**
|
||||
|
||||
|
||||
@@ -439,6 +439,95 @@ This phase addresses the cross-cutting gap identified during research: **xrpld h
|
||||
|
||||
---
|
||||
|
||||
## Task 11.12: Alert Rules for External Dashboard Parity Metrics
|
||||
|
||||
> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity) — 18 alert rules ported from the community [xrpl-validator-dashboard](https://github.com/realgrapedrop/xrpl-validator-dashboard).
|
||||
>
|
||||
> **Upstream**: Phase 7 Tasks 7.9-7.16 (metrics), Phase 9 Tasks 9.11-9.13 (dashboards).
|
||||
> **Downstream**: None — terminal task in the parity chain.
|
||||
|
||||
**Objective**: Add Grafana alerting rules for the Phase 7+ parity metrics (validation agreement, validator health, peer quality, state tracking, ledger economy). These complement Task 11.8's `xrpl_*` alerts by covering the `xrpld_*` internal metrics.
|
||||
|
||||
**Critical Group** (8 rules, eval interval 10s):
|
||||
|
||||
| Rule | Condition | For |
|
||||
| ------------------- | ------------------------------------------------------------- | --- |
|
||||
| Agreement Below 90% | `xrpld_validation_agreement{metric="agreement_pct_24h"} < 90` | 30s |
|
||||
| Not Proposing | `xrpld_state_tracking{metric="state_value"} < 6` | 10s |
|
||||
| Unhealthy State | `xrpld_state_tracking{metric="state_value"} < 4` | 10s |
|
||||
| Amendment Blocked | `xrpld_validator_health{metric="amendment_blocked"} == 1` | 1m |
|
||||
| UNL Expiring | `xrpld_validator_health{metric="unl_expiry_days"} < 14` | 1h |
|
||||
| High IO Latency | `histogram_quantile(0.95, xrpld_ios_latency_bucket) > 50` | 1m |
|
||||
| High Load Factor | `xrpld_load_factor_metrics{metric="load_factor"} > 1000` | 1m |
|
||||
| Peer Count Critical | `xrpld_server_info{metric="peers"} < 5` | 1m |
|
||||
|
||||
**Network Group** (3 rules, eval interval 10s):
|
||||
|
||||
| Rule | Condition | For |
|
||||
| ------------------------- | ----------------------------------------------------------------- | --- |
|
||||
| Peer Drop >10% | `delta(xrpld_server_info{metric="peers"}[30s]) / ... * 100 < -10` | 30s |
|
||||
| Peer Drop >30% | Same formula, threshold -30 | 30s |
|
||||
| P90 Latency + Disconnects | `peer_latency_p90_ms > 500 AND rate(disconnects) > 0` | 2m |
|
||||
|
||||
**Performance Group** (7 rules, eval interval 10s):
|
||||
|
||||
| Rule | Condition | For |
|
||||
| ------------------- | ------------------------------------------------------------ | --- |
|
||||
| CPU High | Per-core CPU > 80% (requires node_exporter) | 2m |
|
||||
| Memory Critical | Memory usage > 90% (requires node_exporter) | 1m |
|
||||
| Disk Warning | Disk usage > 85% (requires node_exporter) | 2m |
|
||||
| Job Queue Overflow | `rate(xrpld_jq_trans_overflow_total[5m]) > 0` | 1m |
|
||||
| Upgrade Recommended | `xrpld_peer_quality{metric="peers_higher_version_pct"} > 60` | 1m |
|
||||
| TX Rate Drop | Transaction rate dropped > 50% in 5m window | 5m |
|
||||
| Stale Ledger | `xrpld_ledger_economy{metric="ledger_age_seconds"} > 30` | 1m |
|
||||
|
||||
**Notification channel templates**: Email/SMTP, Discord, Slack, PagerDuty.
|
||||
|
||||
**Key files**:
|
||||
|
||||
- New/extend: `docker/telemetry/grafana/alerting/alert-rules-parity.yaml`
|
||||
- New: `docker/telemetry/grafana/alerting/contact-points.yaml` (template configs)
|
||||
- New: `docker/telemetry/grafana/alerting/notification-policies.yaml`
|
||||
|
||||
**Exit Criteria**:
|
||||
|
||||
- [ ] All 18 rules evaluate without errors in Grafana alerting UI
|
||||
- [ ] Critical rules fire within expected timeframe when conditions are met
|
||||
- [ ] Notification channel templates are documented (not hard-coded to any service)
|
||||
|
||||
---
|
||||
|
||||
## Task 11.13: Dual-Datasource Architecture Documentation
|
||||
|
||||
> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity)
|
||||
|
||||
**Objective**: Document the external dashboard's "fast path" pattern as a future optimization for real-time panels.
|
||||
|
||||
**Pattern**: A lightweight Prometheus scrape endpoint (separate from OTLP pipeline) that polls critical metrics every 2-5s, bypassing the 10s OTLP metric reader interval and Prometheus scrape interval.
|
||||
|
||||
**Use case**: Real-time state panels (server state, ledger age, peer count) where 10-15s latency is too slow for operational dashboards.
|
||||
|
||||
**Decision**: Document as a future option, not implement now. The current 10s interval is acceptable for v1. The external dashboard achieves 2-5s freshness by polling RPC directly, which is what the Phase 11 receiver already does. Adding a separate scrape endpoint to xrpld would only be needed if sub-second metric freshness is required from the internal metrics pipeline.
|
||||
|
||||
**What to document**:
|
||||
|
||||
- Architecture comparison: OTLP pipeline (10-15s) vs. direct scrape (2-5s) vs. push gateway
|
||||
- When to consider: operator feedback indicating 10s is insufficient for alerting SLOs
|
||||
- How to implement if needed: add `/metrics` HTTP endpoint to xrpld with Prometheus client library
|
||||
- Trade-offs: additional port, additional dependency, duplication with OTLP metrics
|
||||
|
||||
**Key files**:
|
||||
|
||||
- Update: `OpenTelemetryPlan/09-data-collection-reference.md` (add "Future: Dual-Datasource Architecture" section)
|
||||
- Update: `docs/telemetry-runbook.md` (add brief note in performance tuning section)
|
||||
|
||||
**Exit Criteria**:
|
||||
|
||||
- [ ] Architecture comparison documented with clear trade-offs
|
||||
- [ ] Decision rationale recorded (why deferred, when to revisit)
|
||||
|
||||
---
|
||||
|
||||
## Exit Criteria
|
||||
|
||||
- [ ] Custom OTel Collector receiver builds and starts without errors
|
||||
@@ -451,3 +540,5 @@ This phase addresses the cross-cutting gap identified during research: **xrpld h
|
||||
- [ ] Receiver handles xrpld restart/unavailability gracefully (no crash, logs warning, retries)
|
||||
- [ ] Documentation complete: receiver README, metric reference, alerting playbook
|
||||
- [ ] Go receiver has unit tests with >80% coverage
|
||||
- [ ] 18 Grafana alert rules for Phase 7+ parity metrics evaluate correctly (Task 11.12)
|
||||
- [ ] Dual-datasource architecture documented with trade-offs (Task 11.13)
|
||||
|
||||
@@ -289,7 +289,7 @@ design.
|
||||
|
||||
- [ ] `tx.receive` spans carry `peer_version` attribute with a non-empty version string
|
||||
- [ ] Attribute is omitted (not set to empty string) when `getVersion()` returns empty
|
||||
- [ ] Attribute visible in Jaeger span detail view
|
||||
- [ ] Attribute visible in Tempo trace detail view
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -379,7 +379,7 @@ Two strategies for cross-node trace correlation, switchable via config:
|
||||
Derive `trace_id = SHA256(previousLedger.id())[0:16]` so all nodes in the same
|
||||
consensus round share the same trace_id without P2P context propagation.
|
||||
|
||||
- **Pros**: All nodes appear in the same trace in Tempo/Jaeger automatically.
|
||||
- **Pros**: All nodes appear in the same trace in Tempo automatically.
|
||||
No collector-side post-processing needed.
|
||||
- **Cons**: Overrides OTel's random trace_id generation; requires custom
|
||||
`IdGenerator` or manual span context construction.
|
||||
@@ -926,7 +926,7 @@ Received messages use **span links** (follows-from), NOT parent-child:
|
||||
|
||||
- The receiver's processing span links to the sender's context
|
||||
- This preserves each node's independent trace tree
|
||||
- Cross-node correlation visible via linked traces in Tempo/Jaeger
|
||||
- Cross-node correlation visible via linked traces in Tempo
|
||||
|
||||
## Interaction with Deterministic Trace ID (Strategy A)
|
||||
|
||||
|
||||
@@ -127,10 +127,10 @@ These metrics serve multiple external consumer categories identified during rese
|
||||
**What to do**:
|
||||
|
||||
- Register OTel instruments for PerfLog RPC counters (from `PerfLogImp.cpp` line ~63):
|
||||
- Counter: `rpc_method_started_total{method="<name>"}` — calls started
|
||||
- Counter: `rpc_method_finished_total{method="<name>"}` — calls completed
|
||||
- Counter: `rpc_method_errored_total{method="<name>"}` — calls errored
|
||||
- Histogram: `rpc_method_duration_us{method="<name>"}` — execution time distribution
|
||||
- Counter: `xrpld_rpc_method_started_total{method="<name>"}` — calls started
|
||||
- Counter: `xrpld_rpc_method_finished_total{method="<name>"}` — calls completed
|
||||
- Counter: `xrpld_rpc_method_errored_total{method="<name>"}` — calls errored
|
||||
- Histogram: `xrpld_rpc_method_duration_us{method="<name>"}` — execution time distribution
|
||||
|
||||
- Use OTel `Counter<int64_t>` and `Histogram<double>` instruments with `method` attribute label.
|
||||
|
||||
@@ -154,11 +154,11 @@ These metrics serve multiple external consumer categories identified during rese
|
||||
**What to do**:
|
||||
|
||||
- Register OTel instruments for PerfLog job counters:
|
||||
- Counter: `job_queued_total{job_type="<name>"}` — jobs queued
|
||||
- Counter: `job_started_total{job_type="<name>"}` — jobs started
|
||||
- Counter: `job_finished_total{job_type="<name>"}` — jobs completed
|
||||
- Histogram: `job_queued_duration_us{job_type="<name>"}` — time spent waiting in queue
|
||||
- Histogram: `job_running_duration_us{job_type="<name>"}` — execution time distribution
|
||||
- Counter: `xrpld_job_queued_total{job_type="<name>"}` — jobs queued
|
||||
- Counter: `xrpld_job_started_total{job_type="<name>"}` — jobs started
|
||||
- Counter: `xrpld_job_finished_total{job_type="<name>"}` — jobs completed
|
||||
- Histogram: `xrpld_job_queued_duration_us{job_type="<name>"}` — time spent waiting in queue
|
||||
- Histogram: `xrpld_job_running_duration_us{job_type="<name>"}` — execution time distribution
|
||||
|
||||
- Hook into PerfLog's existing job tracking alongside Task 9.4.
|
||||
|
||||
@@ -180,15 +180,15 @@ These metrics serve multiple external consumer categories identified during rese
|
||||
**What to do**:
|
||||
|
||||
- Register OTel `ObservableGauge` callbacks for `CountedObject<T>` instance counts:
|
||||
- `object_count{type="Transaction"}` — live Transaction objects
|
||||
- `object_count{type="Ledger"}` — live Ledger objects
|
||||
- `object_count{type="NodeObject"}` — live NodeObject instances
|
||||
- `object_count{type="STTx"}` — serialized transaction objects
|
||||
- `object_count{type="STLedgerEntry"}` — serialized ledger entries
|
||||
- `object_count{type="InboundLedger"}` — ledgers being fetched
|
||||
- `object_count{type="Pathfinder"}` — active pathfinding computations
|
||||
- `object_count{type="PathRequest"}` — active path requests
|
||||
- `object_count{type="HashRouterEntry"}` — hash router entries
|
||||
- `xrpld_object_count{type="Transaction"}` — live Transaction objects
|
||||
- `xrpld_object_count{type="Ledger"}` — live Ledger objects
|
||||
- `xrpld_object_count{type="NodeObject"}` — live NodeObject instances
|
||||
- `xrpld_object_count{type="STTx"}` — serialized transaction objects
|
||||
- `xrpld_object_count{type="STLedgerEntry"}` — serialized ledger entries
|
||||
- `xrpld_object_count{type="InboundLedger"}` — ledgers being fetched
|
||||
- `xrpld_object_count{type="Pathfinder"}` — active pathfinding computations
|
||||
- `xrpld_object_count{type="PathRequest"}` — active path requests
|
||||
- `xrpld_object_count{type="HashRouterEntry"}` — hash router entries
|
||||
|
||||
- The `CountedObject` template already tracks these via atomic counters. The callback just reads the current counts.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user