Merge branch 'pratik/otel-phase10-workload-validation' into pratik/perf-test-otel-off

2026-07-24 15:40:26 +00:00 · 2026-07-07 13:11:40 +01:00
parent d1d0e4c688 f8ac3e38cc
commit df96e48a01
16 changed files with 994 additions and 933 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -88,6 +88,3 @@ __pycache__
 /.cache
 docker/telemetry/workload/__pycache__/
 .claude/
-
-# Env. file carrying environmental setup data for local or cloud runs.
-.env.*
--- a/OpenTelemetryPlan/06-implementation-phases.md
+++ b/OpenTelemetryPlan/06-implementation-phases.md
@@ -1545,4 +1545,664 @@ flowchart TB

 ---

+---
+
+## Appendix: External Dashboard Parity
+
+> Cross-phase plan for reaching parity with the community [xrpl-validator-dashboard](https://github.com/realgrapedrop/xrpl-validator-dashboard). Previously a standalone design spec; merged here so the phase plan is self-contained.
+
+> **Date**: 2026-03-30
+> **Status**: Draft
+> **Source**: [realgrapedrop/xrpl-validator-dashboard](https://github.com/realgrapedrop/xrpl-validator-dashboard)
+> **Jira Epic**: RIPD-5060
+
+### Summary
+
+Integrate 29 missing metrics, 18 alert rules, and enriched span attributes from the community `xrpl-validator-dashboard` into xrpld's native OpenTelemetry instrumentation. Changes are distributed across phases 2, 3, 4, 6, 7, 9, 10, and 11 of the OTel PR chain.
+
+### Gap Analysis
+
+#### Coverage Breakdown (86 external metrics)
+
+| Status            | Count | Notes                                                          |
+| ----------------- | ----- | -------------------------------------------------------------- |
+| Already covered   | 30    | peer_count, load_factor, io_latency, uptime, overlay traffic   |
+| Partially covered | 3     | state_value encoding, NuDB granularity, validation_quorum      |
+| Missing           | 29    | Validation agreement, ledger economy, peer quality, UNL health |
+| N/A (external)    | 24    | Monitor health, realtime duplicates, system metrics            |
+
+#### Missing Metrics by Category
+
+| Category             | Metrics                                                                                                                                                                                                                            | Count |
+| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
+| Validation Agreement | `validations_sent_total`, `validations_checked_total`, `validation_agreements_total`, `validation_missed_total`, `validation_agreement_pct_1h/24h`, `validation_agreements_1h/24h`, `validation_missed_1h/24h`, `validation_event` | 11    |
+| Ledger Economy       | `ledgers_closed_total`, `ledger_age_seconds`, `base_fee_xrp`, `reserve_base_xrp`, `reserve_inc_xrp`, `transaction_rate`                                                                                                            | 6     |
+| State Tracking       | `time_in_current_state_seconds`, `state_changes_total`, `validator_state_info`                                                                                                                                                     | 3     |
+| Peer Quality         | `peers_insane`, `peer_latency_p90_ms`                                                                                                                                                                                              | 2     |
+| Validator Health     | `amendment_blocked`, `unl_expiry_days`                                                                                                                                                                                             | 2     |
+| Upgrade Awareness    | `peers_higher_version_pct`, `upgrade_recommended`                                                                                                                                                                                  | 2     |
+| Storage / Other      | `ledger_nudb_bytes`, `jq_trans_overflow_total`, `initial_sync_duration_seconds`                                                                                                                                                    | 3     |
+
+#### Alert Rules (18 total, from external dashboard)
+
+| Group       | Count | Rules                                                                                                                   |
+| ----------- | ----- | ----------------------------------------------------------------------------------------------------------------------- |
+| Critical    | 8     | Agreement <90%, not proposing, unhealthy state, amendment blocked, UNL expiring, IO latency, load factor, peer count <5 |
+| Network     | 3     | Peer drop >10%/30%, P90 latency + disconnect correlation                                                                |
+| Performance | 7     | CPU >80%, memory >90%, disk >85%, job queue overflow, upgrade recommended, tx rate drop, stale ledger                   |
+
+---
+
+### Branch-to-Change Mapping
+
+#### Phase 2 — `pratik/otel-phase2-rpc-tracing`
+
+> **Ref**: Adds to existing Phase 2 task list. Consumed by Phase 7 (MetricsRegistry) and Phase 10 (validation checks).
+
+**Task 2.8: RPC Span Attribute Enrichment**
+
+Add node-level health context to every `rpc.command.*` span so operators can correlate RPC behavior with node state.
+
+New span attributes on `rpc.command.*`:
+
+| Attribute                     | Type   | Source                               | Value Example         |
+| ----------------------------- | ------ | ------------------------------------ | --------------------- |
+| `xrpl.node.amendment_blocked` | bool   | `app_.getOPs().isAmendmentBlocked()` | `true`                |
+| `xrpl.node.server_state`      | string | `app_.getOPs().strOperatingMode()`   | `"full"`, `"syncing"` |
+
+**File**: `src/xrpld/rpc/detail/RPCHandler.cpp` (in the `rpc.command.*` span creation block, after existing setAttribute calls)
+
+**Rationale**: RPC is the operator's primary interaction point. When a node is amendment-blocked or degraded, every RPC response is suspect. Tagging spans with this state enables Jaeger queries like `{name=~"rpc.command.*"} | xrpl.node.amendment_blocked = true` to find all RPCs served during a blocked period.
+
+**Exit Criteria**:
+
+- [ ] `rpc.command.server_info` spans carry `xrpl.node.amendment_blocked` and `xrpl.node.server_state` attributes
+- [ ] No measurable latency impact (attribute values are cached atomics, not computed per-call)
+
+---
+
+#### Phase 3 — `pratik/otel-phase3-tx-tracing`
+
+> **Ref**: Adds to existing Phase 3 task list. Consumed by Phase 10 (validation checks).
+
+**Task 3.7: Transaction Span Peer Version Attribute**
+
+Add the relaying peer's xrpld version to transaction receive spans to enable version-mismatch correlation.
+
+New span attribute on `tx.receive`:
+
+| Attribute           | Type   | Source               | Value Example   |
+| ------------------- | ------ | -------------------- | --------------- |
+| `xrpl.peer.version` | string | `peer->getVersion()` | `"xrpld-2.4.0"` |
+
+**File**: `src/xrpld/overlay/detail/PeerImp.cpp` (in the `tx.receive` span block, after existing `xrpl.peer.id` setAttribute)
+
+**Rationale**: Transaction relay is where version mismatches cause subtle serialization or validation bugs. Tracing "this tx came from a v2.3.0 peer" helps diagnose compatibility issues during network upgrades.
+
+**Exit Criteria**:
+
+- [ ] `tx.receive` spans carry `xrpl.peer.version` attribute with a non-empty version string
+- [ ] Attribute is omitted (not empty-string) when `getVersion()` returns empty
+
+---
+
+#### Phase 4 — `pratik/otel-phase4-consensus-tracing`
+
+> **Ref**: Adds to existing Phase 4 task list. Provides the span-level foundation that Phase 7 (ValidationTracker) builds upon. Consumed by Phase 10 (validation checks).
+
+**Task 4.8: Consensus Validation Span Enrichment**
+
+Add ledger hash and validation type to validation spans on both send and receive paths. This enables trace-level agreement analysis — filter by ledger hash to see which validators agreed.
+
+New span attributes on `consensus.validation.send`:
+
+| Attribute                     | Type   | Source                                  | Value Example               |
+| ----------------------------- | ------ | --------------------------------------- | --------------------------- |
+| `xrpl.validation.ledger_hash` | string | Ledger hash from `validate()` call args | `"A1B2C3..."` (64-char hex) |
+| `xrpl.validation.full`        | bool   | Whether this is a full validation       | `true`                      |
+
+New span attributes on `peer.validation.receive`:
+
+| Attribute                          | Type   | Source                                | Value Example               |
+| ---------------------------------- | ------ | ------------------------------------- | --------------------------- |
+| `xrpl.peer.validation.ledger_hash` | string | From deserialized STValidation object | `"A1B2C3..."` (64-char hex) |
+| `xrpl.peer.validation.full`        | bool   | From STValidation flags               | `true`                      |
+
+New span attributes on `consensus.accept`:
+
+| Attribute                            | Type  | Source                                   | Value Example |
+| ------------------------------------ | ----- | ---------------------------------------- | ------------- |
+| `xrpl.consensus.validation_quorum`   | int64 | `app_.validators().quorum()`             | `28`          |
+| `xrpl.consensus.proposers_validated` | int64 | `result.proposers` from consensus result | `35`          |
+
+**Files**:
+
+- `src/xrpld/app/consensus/RCLConsensus.cpp` (validation.send and accept spans)
+- `src/xrpld/overlay/detail/PeerImp.cpp` (peer.validation.receive span)
+
+**Rationale**: The external dashboard's most valuable feature is validation agreement tracking. By recording the ledger hash on both outgoing and incoming validation spans, we create the raw data for agreement analysis at the trace level. Phase 7's ValidationTracker builds the metric-level aggregation on top of this.
+
+**Exit Criteria**:
+
+- [ ] `consensus.validation.send` spans carry `xrpl.validation.ledger_hash` and `xrpl.validation.full`
+- [ ] `peer.validation.receive` spans carry `xrpl.peer.validation.ledger_hash` and `xrpl.peer.validation.full`
+- [ ] `consensus.accept` spans carry `xrpl.consensus.validation_quorum` and `xrpl.consensus.proposers_validated`
+- [ ] Ledger hash attributes match between send and receive for the same ledger
+
+---
+
+#### Phase 6 — `pratik/otel-phase6-statsd`
+
+> **Ref**: Adds to existing Phase 6 scope. No separate task list file exists for Phase 6 per project convention.
+
+**Addition: Bridge `peerDisconnectsCharges_` metric**
+
+The overlay already tracks resource-limit disconnects via `OverlayImpl::Stats::peerDisconnectsCharges_` (a `beast::insight::Gauge`). This metric is registered but not included in the StatsD bridge mapping.
+
+**What to do**:
+
+- Ensure `xrpld_Overlay_Peer_Disconnects_Charges` appears in the StatsD-to-Prometheus metric name mapping
+- Verify the metric appears in Prometheus after StatsD bridge is active
+
+**File**: `src/xrpld/overlay/detail/OverlayImpl.cpp`
+
+**Prometheus name**: `xrpld_Overlay_Peer_Disconnects_Charges`
+
+---
+
+#### Phase 7 — `pratik/otel-phase7-native-metrics`
+
+> **Ref**: Adds to existing Phase 7 task list. This is the largest addition. Depends on Phase 4 span attributes for validation tracking context. Consumed by Phase 9 (dashboards), Phase 10 (validation), Phase 11 (alerts).
+
+**Task 7.8: ValidationTracker — Validation Agreement Computation**
+
+The most valuable missing component. A stateful class that tracks whether our validator's validations agree with network consensus, maintaining rolling 1h and 24h windows.
+
+**Architecture**:
+
+```
+
+  consensus.validation.send ─────> ValidationTracker ──────> MetricsRegistry
+  (records our validation          (reconciles after         (exports agreement
+   for ledger X)                    8s grace period)          gauges every 10s)
+
+  ledger.validate ───────────────> ValidationTracker
+  (records which ledger            (marks ledger X as
+   network validated)               agreed or missed)
+```
+
+**Design**:
+
+```cpp
+/// Tracks validation agreement between this node and network consensus.
+///
+///  ValidationTracker
+///  ├── recordOurValidation(ledgerHash, ledgerSeq)  // called when we send
+///  ├── recordNetworkValidation(ledgerHash, seq)     // called on ledger validate
+///  ├── reconcile()                                  // called periodically (timer)
+///  ├── agreementPct1h() -> double                   // 0.0-100.0
+///  ├── agreementPct24h() -> double
+///  ├── agreements1h() -> uint64_t
+///  ├── missed1h() -> uint64_t
+///  ├── agreements24h() -> uint64_t
+///  ├── missed24h() -> uint64_t
+///  ├── totalAgreements() -> uint64_t
+///  ├── totalMissed() -> uint64_t
+///  ├── totalValidationsSent() -> uint64_t
+///  └── totalValidationsChecked() -> uint64_t        // all network validations seen
+class ValidationTracker
+{
+    // Ring buffer of pending ledger events (max 1000)
+    struct LedgerEvent {
+        uint256 ledgerHash;
+        LedgerIndex seq;
+        TimePoint closeTime;
+        bool weValidated = false;     // did we send a validation for this ledger?
+        bool networkValidated = false; // did network validate this ledger?
+        bool reconciled = false;       // has 8s grace period elapsed?
+        bool agreed = false;           // after reconciliation: did we agree?
+    };
+
+    // Sliding window deques for pre-computed window stats
+    struct WindowEvent {
+        TimePoint time;
+        bool agreed;
+    };
+    std::deque<WindowEvent> window1h_;  // events in last 1 hour
+    std::deque<WindowEvent> window24h_; // events in last 24 hours
+
+    // Reconciliation: 8s grace period after ledger close.
+    // If our validation hasn't arrived by then, mark as missed.
+    // 5-minute late repair: if a late validation arrives, correct the miss.
+    static constexpr auto kGracePeriod = std::chrono::seconds(8);
+    static constexpr auto kLateRepairWindow = std::chrono::minutes(5);
+};
+```
+
+**Recording sites** (modifications to consensus code from Phase 7 branch):
+
+| Hook Point                   | File                | What to Record                                                          |
+| ---------------------------- | ------------------- | ----------------------------------------------------------------------- |
+| `validate()` in `doAccept()` | RCLConsensus.cpp    | `tracker.recordOurValidation(ledgerHash, seq)`                          |
+| `onValidation()` callback    | RCLValidations path | `tracker.recordNetworkValidation(...)` — increment `validationsChecked` |
+| LedgerMaster fully-validated | LedgerMaster.cpp    | `tracker.recordNetworkValidation(validatedHash, seq)`                   |
+
+**Key new files**:
+
+- `src/xrpld/telemetry/ValidationTracker.h`
+- `src/xrpld/telemetry/detail/ValidationTracker.cpp`
+
+**Key modified files**:
+
+- `src/xrpld/telemetry/MetricsRegistry.h` (add ValidationTracker member)
+- `src/xrpld/telemetry/MetricsRegistry.cpp` (add gauge callback reading from tracker)
+- `src/xrpld/app/consensus/RCLConsensus.cpp` (add recording hooks)
+- `src/xrpld/app/ledger/detail/LedgerMaster.cpp` (add recording hook)
+
+**Exit Criteria**:
+
+- [ ] `ValidationTracker` correctly tracks agreement with 8s grace period
+- [ ] 5-minute late repair corrects false-positive misses
+- [ ] Thread-safe (atomics + mutex for window deques)
+- [ ] Rolling windows correctly evict stale entries
+- [ ] Unit tests for: normal agreement, missed validation, late repair, window eviction
+
+---
+
+**Task 7.9: Validator Health Observable Gauges**
+
+New MetricsRegistry observable gauge for amendment, UNL, and quorum health.
+
+| Gauge Name               | Label `metric=`     | Type   | Source                                            |
+| ------------------------ | ------------------- | ------ | ------------------------------------------------- |
+| `xrpld_validator_health` | `amendment_blocked` | int64  | `app_.getOPs().isAmendmentBlocked()` → 0/1        |
+|                          | `unl_blocked`       | int64  | `app_.getOPs().isUNLBlocked()` → 0/1              |
+|                          | `unl_expiry_days`   | double | `app_.validators().expires()` → days until expiry |
+|                          | `validation_quorum` | int64  | `app_.validators().quorum()`                      |
+
+**File**: `src/xrpld/telemetry/MetricsRegistry.cpp` (new gauge callback in `registerAsyncGauges()`)
+
+**Exit Criteria**:
+
+- [ ] All 4 label values emitted every 10s
+- [ ] `unl_expiry_days` is negative when expired, positive when active
+- [ ] Values visible in Prometheus
+
+---
+
+**Task 7.10: Peer Quality Observable Gauges**
+
+New MetricsRegistry observable gauge for peer health aggregates.
+
+| Gauge Name           | Label `metric=`            | Type   | Source                                     |
+| -------------------- | -------------------------- | ------ | ------------------------------------------ |
+| `xrpld_peer_quality` | `peer_latency_p90_ms`      | double | Iterate peers, compute P90 from `latency_` |
+|                      | `peers_insane_count`       | int64  | Count peers with `tracking_ == diverged`   |
+|                      | `peers_higher_version_pct` | double | Compare `getVersion()` to own version      |
+|                      | `upgrade_recommended`      | int64  | 1 if `peers_higher_version_pct > 60%`      |
+
+**Implementation note**: The callback iterates `app_.overlay().foreach(...)` to collect per-peer latency and version data. This runs every 10s on the metrics reader thread — acceptable overhead for ~50-200 peers.
+
+**File**: `src/xrpld/telemetry/MetricsRegistry.cpp`
+
+**Exit Criteria**:
+
+- [ ] P90 latency computed correctly (sort peer latencies, pick 90th percentile)
+- [ ] Insane count matches `peers` RPC output
+- [ ] Version comparison handles format variations (e.g., "xrpld-2.4.0-rc1")
+- [ ] Values visible in Prometheus
+
+---
+
+**Task 7.11: Ledger Economy Observable Gauges**
+
+New MetricsRegistry observable gauge for fee and ledger metrics.
+
+| Gauge Name             | Label `metric=`      | Type   | Source                                    |
+| ---------------------- | -------------------- | ------ | ----------------------------------------- |
+| `xrpld_ledger_economy` | `base_fee_xrp`       | double | `app_.getFeeTrack().getBaseFee()` → drops |
+|                        | `reserve_base_xrp`   | double | From validated ledger fee settings        |
+|                        | `reserve_inc_xrp`    | double | From validated ledger fee settings        |
+|                        | `ledger_age_seconds` | double | `now - lastValidatedCloseTime`            |
+|                        | `transaction_rate`   | double | Derived: tx count delta / time delta      |
+
+**File**: `src/xrpld/telemetry/MetricsRegistry.cpp`
+
+**Exit Criteria**:
+
+- [ ] Fee values match `server_info` RPC output
+- [ ] `ledger_age_seconds` increases monotonically between ledger closes, resets on close
+- [ ] `transaction_rate` is smoothed (rolling average, not instantaneous)
+
+---
+
+**Task 7.12: State Tracking Observable Gauges**
+
+New MetricsRegistry observable gauge for node state duration.
+
+| Gauge Name             | Label `metric=`                 | Type   | Source                                           |
+| ---------------------- | ------------------------------- | ------ | ------------------------------------------------ |
+| `xrpld_state_tracking` | `state_value`                   | int64  | 0-7 numeric encoding matching external dashboard |
+|                        | `time_in_current_state_seconds` | double | `now - lastModeChangeTime`                       |
+
+**State value encoding**:
+
+xrpld's `OperatingMode` enum maps 0-4 (DISCONNECTED through FULL). The external dashboard extends this to 0-6 by combining operating mode with consensus participation:
+
+| Value | State        | Source                                                      |
+| ----- | ------------ | ----------------------------------------------------------- |
+| 0     | disconnected | `OperatingMode::DISCONNECTED`                               |
+| 1     | connected    | `OperatingMode::CONNECTED`                                  |
+| 2     | syncing      | `OperatingMode::SYNCING`                                    |
+| 3     | tracking     | `OperatingMode::TRACKING`                                   |
+| 4     | full         | `OperatingMode::FULL` and not validating                    |
+| 5     | validating   | `OperatingMode::FULL` and `mConsensus.validating()` is true |
+| 6     | proposing    | `OperatingMode::FULL` and consensus mode is `proposing`     |
+
+**Note**: Values 5-6 require checking both `OperatingMode` and `ConsensusMode`. The callback should derive these from `app_.getOPs().getOperatingMode()` combined with `mConsensus.mode()`. If operating mode is FULL and consensus is proposing → 6; if FULL and validating → 5; otherwise use the raw OperatingMode enum value.
+
+**File**: `src/xrpld/telemetry/MetricsRegistry.cpp`
+
+**Exit Criteria**:
+
+- [ ] `state_value` matches external dashboard encoding
+- [ ] `time_in_current_state_seconds` resets on mode change
+
+---
+
+**Task 7.13: Storage Detail Observable Gauge**
+
+| Gauge Name             | Label `metric=` | Type  | Source                                   |
+| ---------------------- | --------------- | ----- | ---------------------------------------- |
+| `xrpld_storage_detail` | `nudb_bytes`    | int64 | NuDB backend file size (filesystem stat) |
+
+**File**: `src/xrpld/telemetry/MetricsRegistry.cpp`
+
+**Exit Criteria**:
+
+- [ ] NuDB file size reported in bytes
+- [ ] Gracefully returns 0 if NuDB not configured
+
+---
+
+**Task 7.14: New Synchronous Counters**
+
+New counters incremented at event sites. Declared in MetricsRegistry, recording sites added in consensus/overlay/network code.
+
+| Counter Name                        | Increment Site                   | Source File           |
+| ----------------------------------- | -------------------------------- | --------------------- |
+| `xrpld_ledgers_closed_total`        | `onAccept()` in consensus        | RCLConsensus.cpp      |
+| `xrpld_validations_sent_total`      | `validate()` in consensus        | RCLConsensus.cpp      |
+| `xrpld_validations_checked_total`   | Network validation received      | LedgerMaster.cpp      |
+| `xrpld_validation_agreements_total` | ValidationTracker reconciliation | ValidationTracker.cpp |
+| `xrpld_validation_missed_total`     | ValidationTracker reconciliation | ValidationTracker.cpp |
+| `xrpld_state_changes_total`         | `setMode()` in NetworkOPs        | NetworkOPs.cpp        |
+| `xrpld_jq_trans_overflow_total`     | Job queue overflow path          | JobQueue.cpp          |
+
+**Key modified files**:
+
+- `src/xrpld/telemetry/MetricsRegistry.h/.cpp` (counter declarations)
+- `src/xrpld/app/consensus/RCLConsensus.cpp` (recording: ledgers_closed, validations_sent)
+- `src/xrpld/app/ledger/detail/LedgerMaster.cpp` (recording: validations_checked)
+- `src/xrpld/app/misc/NetworkOPs.cpp` (recording: state_changes)
+
+**Exit Criteria**:
+
+- [ ] All 7 counters monotonically increase during normal operation
+- [ ] Counter values match expected rates (e.g., ledgers_closed ≈ 1 per 3-5s)
+- [ ] Values visible in Prometheus
+
+---
+
+**Task 7.15: Validation Agreement Observable Gauge**
+
+Reads from the `ValidationTracker` (Task 7.8) to export rolling window stats.
+
+| Gauge Name                   | Label `metric=`     | Type   | Source                      |
+| ---------------------------- | ------------------- | ------ | --------------------------- |
+| `xrpld_validation_agreement` | `agreement_pct_1h`  | double | `tracker.agreementPct1h()`  |
+|                              | `agreements_1h`     | int64  | `tracker.agreements1h()`    |
+|                              | `missed_1h`         | int64  | `tracker.missed1h()`        |
+|                              | `agreement_pct_24h` | double | `tracker.agreementPct24h()` |
+|                              | `agreements_24h`    | int64  | `tracker.agreements24h()`   |
+|                              | `missed_24h`        | int64  | `tracker.missed24h()`       |
+
+**File**: `src/xrpld/telemetry/MetricsRegistry.cpp`
+
+**Exit Criteria**:
+
+- [ ] Agreement percentages in range [0.0, 100.0]
+- [ ] Window stats match manual count from validation counters
+- [ ] Percentages stabilize after 1h/24h of operation
+
+---
+
+#### Phase 9 — `pratik/otel-phase9-metric-gap-fill`
+
+> **Ref**: Adds to existing Phase 9 task list. Depends on Phase 7 gauges/counters. Consumed by Phase 10 (dashboard load checks).
+
+**Task 9.11: Validator Health Dashboard**
+
+New Grafana dashboard: `validator-health.json`
+
+| Panel                      | Type       | PromQL                                                         |
+| -------------------------- | ---------- | -------------------------------------------------------------- |
+| Agreement % (1h)           | stat       | `xrpld_validation_agreement{metric="agreement_pct_1h"}`        |
+| Agreement % (24h)          | stat       | `xrpld_validation_agreement{metric="agreement_pct_24h"}`       |
+| Agreements vs Missed (1h)  | bargauge   | `agreements_1h` and `missed_1h` side by side                   |
+| Agreements vs Missed (24h) | bargauge   | `agreements_24h` and `missed_24h` side by side                 |
+| Validation Rate            | stat       | `rate(xrpld_validations_sent_total[5m]) * 60`                  |
+| Validations Checked Rate   | stat       | `rate(xrpld_validations_checked_total[5m]) * 60`               |
+| Amendment Blocked          | stat       | `xrpld_validator_health{metric="amendment_blocked"}`           |
+| UNL Expiry (days)          | stat       | `xrpld_validator_health{metric="unl_expiry_days"}`             |
+| Validation Quorum          | stat       | `xrpld_validator_health{metric="validation_quorum"}`           |
+| State Value Timeline       | timeseries | `xrpld_state_tracking{metric="state_value"}`                   |
+| Time in Current State      | stat       | `xrpld_state_tracking{metric="time_in_current_state_seconds"}` |
+| State Changes Rate         | stat       | `rate(xrpld_state_changes_total[1h])`                          |
+| Ledgers Closed Rate        | stat       | `rate(xrpld_ledgers_closed_total[5m]) * 60`                    |
+
+**Dashboard conventions**: `$node` template variable for `exported_instance` filtering, dark theme, matching existing panel sizes and color schemes.
+
+---
+
+**Task 9.12: Peer Quality Dashboard**
+
+New Grafana dashboard: `peer-quality.json`
+
+| Panel                  | Type       | PromQL                                                         |
+| ---------------------- | ---------- | -------------------------------------------------------------- |
+| P90 Peer Latency       | timeseries | `xrpld_peer_quality{metric="peer_latency_p90_ms"}`             |
+| Insane/Diverged Peers  | stat       | `xrpld_peer_quality{metric="peers_insane_count"}`              |
+| Higher Version Peers % | stat       | `xrpld_peer_quality{metric="peers_higher_version_pct"}`        |
+| Upgrade Recommended    | stat       | `xrpld_peer_quality{metric="upgrade_recommended"}`             |
+| Resource Disconnects   | timeseries | `xrpld_Overlay_Peer_Disconnects_Charges`                       |
+| Inbound vs Outbound    | bargauge   | `xrpld_Peer_Finder_Active_Inbound_Peers`, `..._Outbound_Peers` |
+
+---
+
+**Task 9.13: Ledger Economy Dashboard Panels**
+
+Add a "Ledger Economy" row to the existing `node-health.json` dashboard:
+
+| Panel                | Type       | PromQL                                              |
+| -------------------- | ---------- | --------------------------------------------------- |
+| Base Fee (drops)     | stat       | `xrpld_ledger_economy{metric="base_fee_xrp"}`       |
+| Reserve Base (drops) | stat       | `xrpld_ledger_economy{metric="reserve_base_xrp"}`   |
+| Reserve Inc (drops)  | stat       | `xrpld_ledger_economy{metric="reserve_inc_xrp"}`    |
+| Ledger Age           | stat       | `xrpld_ledger_economy{metric="ledger_age_seconds"}` |
+| Transaction Rate     | timeseries | `xrpld_ledger_economy{metric="transaction_rate"}`   |
+
+---
+
+#### Phase 10 — `pratik/otel-phase10-workload-validation`
+
+> **Ref**: Adds to existing Phase 10 task list. Validates all additions from Phases 2-9.
+
+**Task 10.6: External Dashboard Parity Validation Checks**
+
+Add checks to `validate_telemetry.py` for all new span attributes and metrics.
+
+**New span attribute checks (~8)**:
+
+| Span Name                   | New Attribute                        |
+| --------------------------- | ------------------------------------ |
+| `rpc.command.server_info`   | `xrpl.node.amendment_blocked`        |
+| `rpc.command.server_info`   | `xrpl.node.server_state`             |
+| `tx.receive`                | `xrpl.peer.version`                  |
+| `consensus.validation.send` | `xrpl.validation.ledger_hash`        |
+| `consensus.validation.send` | `xrpl.validation.full`               |
+| `peer.validation.receive`   | `xrpl.peer.validation.ledger_hash`   |
+| `consensus.accept`          | `xrpl.consensus.validation_quorum`   |
+| `consensus.accept`          | `xrpl.consensus.proposers_validated` |
+
+**New metric existence checks (~13)**:
+
+| Metric Name                                              |
+| -------------------------------------------------------- |
+| `xrpld_validation_agreement{metric="agreement_pct_1h"}`  |
+| `xrpld_validation_agreement{metric="agreement_pct_24h"}` |
+| `xrpld_validator_health{metric="amendment_blocked"}`     |
+| `xrpld_validator_health{metric="unl_expiry_days"}`       |
+| `xrpld_peer_quality{metric="peer_latency_p90_ms"}`       |
+| `xrpld_peer_quality{metric="peers_insane_count"}`        |
+| `xrpld_ledger_economy{metric="base_fee_xrp"}`            |
+| `xrpld_ledger_economy{metric="transaction_rate"}`        |
+| `xrpld_state_tracking{metric="state_value"}`             |
+| `xrpld_ledgers_closed_total`                             |
+| `xrpld_validations_sent_total`                           |
+| `xrpld_state_changes_total`                              |
+| `xrpld_storage_detail{metric="nudb_bytes"}`              |
+
+**New dashboard load checks (~3)**:
+
+| Dashboard               |
+| ----------------------- |
+| `validator-health`      |
+| `peer-quality`          |
+| `node-health` (updated) |
+
+**New metric value sanity checks (~4)**:
+
+| Check                         | Condition         |
+| ----------------------------- | ----------------- |
+| `validation_agreement_pct_1h` | in [0, 100]       |
+| `unl_expiry_days`             | > 0 (not expired) |
+| `peer_latency_p90_ms`         | > 0 (peers exist) |
+| `state_value`                 | in [0, 7]         |
+
+**Total new checks: ~28** (bringing total from 73 to ~101)
+
+---
+
+#### Phase 11 — (future branch)
+
+> **Ref**: Adds to existing Phase 11 task list. Depends on Phase 7 metrics and Phase 9 dashboards.
+
+**Task 11.9: Alert Rules from External Dashboard**
+
+Port 18 alert rules from the external `xrpl-validator-dashboard` to Grafana alerting provisioning.
+
+**Critical Group** (8 rules, eval interval 10s):
+
+| Rule                | Condition                                                     | For |
+| ------------------- | ------------------------------------------------------------- | --- |
+| Agreement Below 90% | `xrpld_validation_agreement{metric="agreement_pct_24h"} < 90` | 30s |
+| Not Proposing       | `xrpld_state_tracking{metric="state_value"} < 6`              | 10s |
+| Unhealthy State     | `xrpld_state_tracking{metric="state_value"} < 4`              | 10s |
+| Amendment Blocked   | `xrpld_validator_health{metric="amendment_blocked"} == 1`     | 1m  |
+| UNL Expiring        | `xrpld_validator_health{metric="unl_expiry_days"} < 14`       | 1h  |
+| High IO Latency     | `histogram_quantile(0.95, xrpld_ios_latency_bucket) > 50`     | 1m  |
+| High Load Factor    | `xrpld_load_factor_metrics{metric="load_factor"} > 1000`      | 1m  |
+| Peer Count Critical | `xrpld_server_info{metric="peers"} < 5`                       | 1m  |
+
+**Network Group** (3 rules, eval interval 10s):
+
+| Rule                      | Condition                                                         | For |
+| ------------------------- | ----------------------------------------------------------------- | --- |
+| Peer Drop >10%            | `delta(xrpld_server_info{metric="peers"}[30s]) / ... * 100 < -10` | 30s |
+| Peer Drop >30%            | Same formula, threshold -30                                       | 30s |
+| P90 Latency + Disconnects | `peer_latency_p90_ms > 500 AND rate(disconnects) > 0`             | 2m  |
+
+**Performance Group** (7 rules, eval interval 10s):
+
+| Rule                | Condition                                                    | For |
+| ------------------- | ------------------------------------------------------------ | --- |
+| CPU High            | Per-core CPU > 80%                                           | 2m  |
+| Memory Critical     | Memory usage > 90%                                           | 1m  |
+| Disk Warning        | Disk usage > 85%                                             | 2m  |
+| Job Queue Overflow  | `rate(xrpld_jq_trans_overflow_total[5m]) > 0`                | 1m  |
+| Upgrade Recommended | `xrpld_peer_quality{metric="peers_higher_version_pct"} > 60` | 1m  |
+| TX Rate Drop        | Transaction rate dropped > 50% in 5m window                  | 5m  |
+| Stale Ledger        | `xrpld_ledger_economy{metric="ledger_age_seconds"} > 30`     | 1m  |
+
+**Notification channels**: Template configs for Email/SMTP, Discord, Slack, PagerDuty.
+
+**Files**:
+
+- `docker/telemetry/grafana/alerting/alert-rules.yaml` (new or extend existing)
+- `docker/telemetry/grafana/alerting/contact-points.yaml`
+- `docker/telemetry/grafana/alerting/notification-policies.yaml`
+
+---
+
+**Task 11.10: Dual-Datasource Architecture Documentation**
+
+Document the external dashboard's "fast path" pattern as a future optimization for real-time panels:
+
+- **Pattern**: A lightweight Prometheus scrape endpoint (separate from OTLP pipeline) that polls critical metrics every 2-5s, bypassing the 10s OTLP metric reader interval and Prometheus scrape interval.
+- **Use case**: Real-time state panels (server state, ledger age, peer count) where 10-15s latency is too slow.
+- **Decision**: Document as a future option, not implement now. Current 10s interval is acceptable for v1.
+
+**File**: `OpenTelemetryPlan/Phase11_taskList.md` (documentation task, no code)
+
+---
+
+### Documentation Updates
+
+#### `docs/telemetry-runbook.md` (on Phase 9 branch)
+
+Add new sections after "Phase 9: OTel Metrics Alerting Rules":
+
+1. **Validator Health Monitoring** — explains agreement tracking, amendment blocked, UNL expiry, with example PromQL queries
+2. **Peer Quality Monitoring** — explains P90 latency, insane peers, version awareness
+3. **Ledger Economy Monitoring** — explains fee/reserve gauges, transaction rate, ledger age
+4. **Validation Agreement Explained** — operator-facing explanation of the reconciliation algorithm (8s grace, 5m late repair), what "missed" means, and when to worry
+
+#### `OpenTelemetryPlan/09-data-collection-reference.md` (on Phase 9 branch)
+
+Add new metric tables in a "Phase 7+: External Dashboard Parity" section covering all 29 new metrics with their gauge names, label values, types, and sources.
+
+---
+
+### Cross-Phase Dependency Chain
+
+```
+Phase 2 (span attrs: amendment_blocked, server_state)
+Phase 3 (span attrs: peer.version)
+Phase 4 (span attrs: validation.ledger_hash, validation.full, quorum)
+Phase 6 (StatsD bridge: peerDisconnectsCharges)
+    │
+    ├── all above rebase into ──>
+    │
+Phase 7 (ValidationTracker + 7 gauges + 7 counters + agreement gauge)
+    │
+Phase 9 (3 dashboards + ledger economy panels + runbook + data-collection-ref)
+    │
+Phase 10 (28 new validation checks in validate_telemetry.py)
+    │
+Phase 11 (18 alert rules + dual-datasource docs)
+```
+
+### Rebase Strategy
+
+After committing changes to each branch (starting from Phase 2):
+
+1. Commit on `pratik/otel-phase2-rpc-tracing`
+2. Rebase `phase3` onto `phase2`, resolve conflicts (task list files only — low risk)
+3. Commit on `phase3`, rebase `phase4` onto `phase3`
+4. Continue through chain: 4 → 5 → 5b → 6 → 7 → 8 → 9 → 10
+5. Force-push-with-lease all affected branches
+
+Since these are documentation-only changes (task list .md files), merge conflicts should be minimal — each file is unique to its branch.
+
 _Previous: [Configuration Reference](./05-configuration-reference.md)_ | _Next: [Observability Backends](./07-observability-backends.md)_ | _Back to: [Overview](./OpenTelemetryPlan.md)_
--- a/OpenTelemetryPlan/Phase3_taskList.md
+++ b/OpenTelemetryPlan/Phase3_taskList.md
@@ -252,7 +252,7 @@ design.

 ## Task 3.8: Transaction Span Peer Version Attribute

-> **Source**: [External Dashboard Parity](../docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md) — adds peer version context inspired by the community [xrpl-validator-dashboard](https://github.com/realgrapedrop/xrpl-validator-dashboard).
+> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity) — adds peer version context inspired by the community [xrpl-validator-dashboard](https://github.com/realgrapedrop/xrpl-validator-dashboard).
 >
 > **Upstream**: Phase 2 (RPC span infrastructure must exist).
 > **Downstream**: Phase 10 (validation checks for this attribute).
--- a/OpenTelemetryPlan/Phase4_taskList.md
+++ b/OpenTelemetryPlan/Phase4_taskList.md
@@ -218,7 +218,7 @@

 ## Task 4.8: Consensus Validation Span Enrichment — NOT DONE

-> **Source**: [External Dashboard Parity](../docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md) — adds validation agreement context inspired by the community [xrpl-validator-dashboard](https://github.com/realgrapedrop/xrpl-validator-dashboard).
+> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity) — adds validation agreement context inspired by the community [xrpl-validator-dashboard](https://github.com/realgrapedrop/xrpl-validator-dashboard).
 >
 > **Upstream**: Phase 4 tasks 4.1-4.4 (span creation must exist).
 > **Downstream**: Phase 7 (ValidationTracker reads these attributes), Phase 10 (validation checks).
--- a/OpenTelemetryPlan/Phase7_taskList.md
+++ b/OpenTelemetryPlan/Phase7_taskList.md
@@ -230,7 +230,7 @@

 ## Task 7.9: ValidationTracker — Validation Agreement Computation

-> **Source**: [External Dashboard Parity](../docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md) — the most valuable metric from the community [xrpl-validator-dashboard](https://github.com/realgrapedrop/xrpl-validator-dashboard).
+> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity) — the most valuable metric from the community [xrpl-validator-dashboard](https://github.com/realgrapedrop/xrpl-validator-dashboard).
 >
 > **Upstream**: Phase 4 Task 4.8 (validation span attributes provide ledger hash context).
 > **Downstream**: Phase 9 (Validator Health dashboard), Phase 10 (validation checks), Phase 11 (agreement alert rules).
@@ -311,7 +311,7 @@ struct WindowEvent {

 ## Task 7.10: Validator Health Observable Gauges

-> **Source**: [External Dashboard Parity](../docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md)
+> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity)

 **Objective**: Export amendment blocked, UNL health, and quorum data as a native OTel observable gauge.

@@ -396,7 +396,7 @@ validatorParticipationGauge_ = meter_->CreateInt64ObservableGauge(

 ## Task 7.11: Peer Quality Observable Gauges

-> **Source**: [External Dashboard Parity](../docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md)
+> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity)

 **Objective**: Export peer health aggregates (latency P90, insane peers, version awareness) as a native OTel observable gauge.

@@ -430,7 +430,7 @@ validatorParticipationGauge_ = meter_->CreateInt64ObservableGauge(

 ## Task 7.12: Ledger Economy Observable Gauges

-> **Source**: [External Dashboard Parity](../docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md)
+> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity)

 **Objective**: Export fee, reserve, ledger age, and transaction rate as a native OTel observable gauge.

@@ -456,7 +456,7 @@ validatorParticipationGauge_ = meter_->CreateInt64ObservableGauge(

 ## Task 7.13: State Tracking Observable Gauges

-> **Source**: [External Dashboard Parity](../docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md)
+> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity)

 **Objective**: Export extended state value (0-6 encoding combining OperatingMode + ConsensusMode) and time-in-current-state.

@@ -480,7 +480,7 @@ validatorParticipationGauge_ = meter_->CreateInt64ObservableGauge(

 ## Task 7.14: Storage Detail and Sync Info Gauges

-> **Source**: [External Dashboard Parity](../docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md)
+> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity)

 **Objective**: Export NuDB-specific storage size and initial sync duration.

@@ -502,7 +502,7 @@ validatorParticipationGauge_ = meter_->CreateInt64ObservableGauge(

 ## Task 7.15: New Synchronous Counters

-> **Source**: [External Dashboard Parity](../docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md)
+> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity)

 **Objective**: Add 7 new event counters incremented at their respective instrumentation sites.

@@ -527,7 +527,7 @@ validatorParticipationGauge_ = meter_->CreateInt64ObservableGauge(

 ## Task 7.16: Validation Agreement Observable Gauge

-> **Source**: [External Dashboard Parity](../docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md)
+> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity)

 **Objective**: Export rolling window agreement stats from `ValidationTracker` (Task 7.9).

--- a/OpenTelemetryPlan/Phase9_taskList.md
+++ b/OpenTelemetryPlan/Phase9_taskList.md
@@ -314,14 +314,14 @@ These metrics serve multiple external consumer categories identified during rese
 - Provision Grafana alert rules (`docker/telemetry/grafana/provisioning/alerting/`):
  - 6 rules in 3 groups — consensus/ledger (`LedgerHistoryMismatch`, `LedgerCloseStalled`), validator (`ValidationsMissed`, `ValidationsNotChecked`), job queue (`JobQueueTxOverflow`, `JobQueueLatencyHigh`)
  - `xrpld-default` webhook contact point + flat notification policy; auto-loaded via the existing `provisioning/` mount (no docker-compose change)
-  - `docker/telemetry/ALERTING.md` operator runbook (per-alert meaning, tuning, receiver wiring)
+  - Alerting operator docs (per-alert meaning, tuning, receiver wiring) now live in the Alerting section of `docs/telemetry-runbook.md`

 **Key modified files**:

 - `OpenTelemetryPlan/09-data-collection-reference.md`
 - `docs/telemetry-runbook.md`
 - `docker/telemetry/grafana/provisioning/alerting/{rules,contactpoints,policies}.yaml` (new)
- `docker/telemetry/ALERTING.md` (new)
+- `docs/telemetry-runbook.md` (Alerting section added)

 ---

@@ -452,3 +452,98 @@ These metrics serve multiple external consumer categories identified during rese
 - [ ] Validator Health dashboard renders all 13 panels
 - [ ] Peer Quality dashboard renders all 6 panels
 - [ ] Ledger Economy panels added to node-health dashboard
+
+---
+
+## Appendix: Alerting Design
+
+> Design for the provisioned Grafana alert rules (Task 9.9a). Previously a standalone spec; merged here so the phase plan is self-contained.
+
+**Date:** 2026-07-06
+**Branch:** `pratik/otel-phase9-metric-gap-fill` (PR #6513, Jira RIPD-5187)
+**Status:** Approved
+
+### Purpose
+
+Phase 9 exports ~68 internal rippled metrics and ships Grafana dashboards for
+them. This adds the missing operator-facing piece: **provisioned Grafana alert
+rules** that fire on the health-critical metrics phase 9 introduces. The
+phase-9 task list (line 311) and Jira story RIPD-5187 both already list
+"alerting rules" as a phase-9 deliverable, so this closes that gap.
+
+Scope is deliberately narrow — the three subsystems whose failure is
+node-fatal: **consensus/ledger health, validator health, job queue**. RPC/API
+health is explicitly out of scope.
+
+### Why phase 9 (not phase 11)
+
+Every metric these alerts fire on is _born_ in phase 9
+(`xrpld_ledger_history_mismatch_total`, `xrpld_ledgers_closed_total`,
+`xrpld_validation_missed_total`, `xrpld_validations_checked_total`,
+`xrpld_jq_trans_overflow_total`, `xrpld_job_queued_duration_us_bucket`). Alerts
+belong with the metrics they watch, and this is where the dependency lives.
+
+### Delivery
+
+Provisioned YAML, version-controlled — matching the existing datasource /
+dashboard provisioning pattern. No docker-compose change: the Grafana service
+already mounts `./grafana/provisioning:/etc/grafana/provisioning:ro`, and
+Grafana auto-loads `provisioning/alerting/*.yaml`.
+
+New files under `docker/telemetry/grafana/provisioning/alerting/`:
+
+| File                 | Purpose                                                                                                             |
+| -------------------- | ------------------------------------------------------------------------------------------------------------------- |
+| `contactpoints.yaml` | One contact point `xrpld-default` (webhook to a documented placeholder; comments show how to swap for Slack/email). |
+| `policies.yaml`      | Default notification policy: route all alerts → `xrpld-default`, grouped by `alertname` + `exported_instance`.      |
+| `rules.yaml`         | 6 alert rules across 3 groups (below).                                                                              |
+
+Plus the Alerting section of `docs/telemetry-runbook.md` — operator runbook:
+what each alert means, likely causes, and how to point the contact point at a
+real receiver.
+
+### Alert rules
+
+All rules target Prometheus datasource `uid: prometheus`. Each rule uses the
+Grafana rule shape: query (A) → reduce (B, last value) → threshold (C). All
+`rate()`/`histogram_quantile()` expressions aggregate with
+`sum by (exported_instance)` (or `+ le`) so **each node alerts independently**.
+Alert rules run headless, so they cannot use the dashboards' `$node` template
+variables — they match all series and group by `exported_instance` instead.
+
+| Group     | Alert                 | Expression (5m window)                                                                                    | Fires                 | `for` | severity |
+| --------- | --------------------- | --------------------------------------------------------------------------------------------------------- | --------------------- | ----- | -------- |
+| Consensus | LedgerHistoryMismatch | `sum by (exported_instance)(rate(xrpld_ledger_history_mismatch_total[5m]))`                               | `> 0`                 | 5m    | critical |
+| Consensus | LedgerCloseStalled    | `sum by (exported_instance)(rate(xrpld_ledgers_closed_total[5m]))`                                        | `< 0.001` (≈0)        | 3m    | critical |
+| Validator | ValidationsMissed     | `sum by (exported_instance)(rate(xrpld_validation_missed_total[5m]))`                                     | `> 0`                 | 5m    | warning  |
+| Validator | ValidationsNotChecked | `sum by (exported_instance)(rate(xrpld_validations_checked_total[5m]))`                                   | `< 0.001` (≈0)        | 5m    | warning  |
+| Job queue | JobQueueTxOverflow    | `sum by (exported_instance)(rate(xrpld_jq_trans_overflow_total[5m]))`                                     | `> 0`                 | 5m    | warning  |
+| Job queue | JobQueueLatencyHigh   | `histogram_quantile(0.99, sum by (le, exported_instance)(rate(xrpld_job_queued_duration_us_bucket[5m])))` | `> 1000000` (µs = 1s) | 5m    | warning  |
+
+Each rule carries labels `severity` and `category` (consensus/validator/jobqueue)
+and annotations `summary` + `description` (with `{{ $labels.exported_instance }}`
+and `{{ $values.B.Value }}` interpolation).
+
+#### Threshold rationale
+
+- **LedgerCloseStalled `< 0.001` for 3m**: healthy nodes close a ledger every
+  ~3-5s; a 5m rate decaying to ~0 means the node is stuck. The epsilon (not
+  exact `0`) avoids float rate-noise suppressing the alert.
+- **JobQueueLatencyHigh 1s p99**: a default starting point, easy to tune — jobs
+  queued >1s at p99 indicate the node is saturated.
+- Others are `> 0` on error/miss counters: any sustained nonzero rate is
+  actionable.
+
+### Non-goals / YAGNI
+
+- No per-alert silencing schedules, no mute timings.
+- No RPC/API, overlay, or fee-market alerts (dashboards cover those visually).
+- Single contact point — multi-receiver routing is left to the operator.
+
+### Verification
+
+1. `yamllint` (or `python -c yaml.safe_load`) on all three YAML files.
+2. `docker compose -f docker/telemetry/docker-compose.yml config -q` still parses.
+3. Optional live check: start stack, `GET /api/v1/provisioning/alert-rules`
+   returns the 6 rules; Grafana logs show no provisioning errors.
+4. Code-review pass (subagent) against phase conventions before commit.
--- a/docker/telemetry/.env.alerting.example
+++ b/docker/telemetry/.env.alerting.example
@@ -0,0 +1,32 @@
+# rippled OTel alerting delivery — copy to `.env.alerting` and fill in.
+#
+# `.env.alerting` is gitignored; never commit a real Slack webhook or address.
+# Grafana reads these when the stack starts and expands the ${VARS} referenced
+# in grafana/provisioning/alerting/contactpoints.yaml. See the Alerting
+# section of docs/telemetry-runbook.md.
+#
+# Any var left blank simply disables that delivery path — the stack still runs.
+
+# --- Slack ---
+# Incoming-webhook URL from the Slack app (Incoming Webhooks feature).
+# Used by both the warning (xrpld-default) and critical (xrpld-critical) tiers.
+SLACK_WEBHOOK_URL=
+
+# Channel label. With an incoming webhook the target channel is fixed by the
+# webhook itself; this only satisfies Grafana's Slack validator. Defaults to
+# #xrpld-alerts if unset.
+SLACK_CHANNEL=#xrpld-alerts
+
+# --- Email (critical tier only) ---
+# Comma-separated recipient list for critical alerts.
+ALERT_EMAIL_TO=
+
+# SMTP relay Grafana sends through. Email only delivers when SMTP is enabled
+# and these point at a real relay (e.g. an internal smart host or an
+# authenticated provider).
+GF_SMTP_ENABLED=false
+GF_SMTP_HOST=smtp.example.com:587
+GF_SMTP_USER=
+GF_SMTP_PASSWORD=
+GF_SMTP_FROM_ADDRESS=alerts@xrpld.local
+GF_SMTP_FROM_NAME=xrpld Alerts
--- a/docker/telemetry/.gitignore
+++ b/docker/telemetry/.gitignore
@@ -1,6 +1,9 @@
 # Runtime data generated by xrpld and telemetry stack
 data/

-# Grafana Cloud OTLP secrets (never commit). The .example template IS tracked.
-.env.grafanacloud
+# Env. file carrying environmental setup data for local or cloud runs.
+.env.*
+
+# Keep examples
+!.env.alerting.example
 !.env.grafanacloud.example
--- a/docker/telemetry/ALERTING.md
+++ b/docker/telemetry/ALERTING.md
@@ -1,133 +0,0 @@
-# rippled OpenTelemetry Alerting Runbook
-
-rippled exports its internal metrics and provisions Grafana alert rules on the
-health-critical ones. This document explains each alert, its likely cause, and
-how to point alerts at a real receiver.
-
-The rules are provisioned from
-`grafana/provisioning/alerting/` and load automatically when the Grafana
-container in [docker-compose.yml](docker-compose.yml) starts — no manual setup
-in the Grafana UI. Rules appear in Grafana under **Alerting → Alert rules**,
-folder **xrpld**.
-
---
-
-## Alert catalogue
-
-All rules evaluate every minute against the Prometheus datasource, over a
-5-minute window, and group by `exported_instance` so each node alerts on its
-own. Alerts fire only after the condition holds for the `for` dwell time.
-
-| Alert                   | Severity | Fires when                                      | For |
-| ----------------------- | -------- | ----------------------------------------------- | --- |
-| `LedgerHistoryMismatch` | critical | `rate(xrpld_ledger_history_mismatch_total)` > 0 | 5m  |
-| `LedgerCloseStalled`    | critical | `rate(xrpld_ledgers_closed_total)` ≈ 0          | 3m  |
-| `ValidationsMissed`     | warning  | `rate(xrpld_validation_missed_total)` > 0       | 5m  |
-| `ValidationsNotChecked` | warning  | `rate(xrpld_validations_checked_total)` ≈ 0     | 5m  |
-| `JobQueueTxOverflow`    | warning  | `rate(xrpld_jq_trans_overflow_total)` > 0       | 5m  |
-| `JobQueueLatencyHigh`   | warning  | p99 `xrpld_job_queued_duration_us` > 1s         | 5m  |
-
-### Consensus / ledger health
-
-**LedgerHistoryMismatch** — The node closed a ledger whose history diverges
-from the validated network chain. Likely causes: corrupted local state, a bug,
-or a node that fell out of sync and rebuilt incorrectly. Investigate the node's
-ledger acquisition logs; a healthy node never mismatches.
-
-**LedgerCloseStalled** — No ledgers closed for 3 minutes. A healthy node closes
-one every ~3-5s. Likely causes: lost peer connectivity, consensus stall, or the
-process is hung. This rule also fires on _NoData_ — if the series disappears the
-node is likely down. Check peer count and process health first.
-
-### Validator health
-
-**ValidationsMissed** — This validator's validations are not agreeing with the
-validated ledger. Sustained misses risk removal from UNLs. Check clock sync,
-peer connectivity, and whether the node is keeping up with ledger close.
-
-**ValidationsNotChecked** — The node has stopped checking incoming validations
-from peers. Likely causes: overlay/peer disconnection or a stalled validation
-pipeline. Fires on NoData as well.
-
-### Job queue / resource health
-
-**JobQueueTxOverflow** — The transaction job queue is full and transactions are
-being dropped. The node is shedding load it cannot process. Check CPU, the
-`JobQueueLatencyHigh` alert, and offered load.
-
-**JobQueueLatencyHigh** — p99 queue wait exceeds 1 second, i.e. jobs back up
-before running. The node is saturated. Correlate with CPU and the Job Queue
-dashboard.
-
---
-
-## Tuning thresholds
-
-Thresholds live in
-[grafana/provisioning/alerting/rules.yaml](grafana/provisioning/alerting/rules.yaml)
-as the `params` array of each rule's `C` (threshold) node. Common tunables:
-
- **`JobQueueLatencyHigh`** — `params: [1000000]` is 1 000 000 µs (1s). Lower
-  it for latency-sensitive deployments.
- **`LedgerCloseStalled` / `ValidationsNotChecked`** — use `lt` with a tiny
-  epsilon (`0.001`) rather than `0`, so floating-point rate noise near zero
-  does not suppress the alert.
-
-Edit the file and restart the Grafana container to reload:
-
-```bash
-docker compose -f docker/telemetry/docker-compose.yml restart grafana
-```
-
---
-
-## Sending alerts somewhere real
-
-The provisioned contact point `xrpld-default`
-([contactpoints.yaml](grafana/provisioning/alerting/contactpoints.yaml)) ships
-as a **local-dev webhook** to a placeholder URL — alerts fire but go nowhere
-until you change it.
-
-**Option A — repoint the webhook.** Replace the `url` under the `webhook`
-receiver with a real endpoint (PagerDuty Events API, Opsgenie, a custom sink).
-
-**Option B — add a Slack receiver.** Add a second receiver to the same contact
-point:
-
-```yaml
- uid: xrpld-slack
-  type: slack
-  settings:
-    url: https://hooks.slack.com/services/XXX/YYY/ZZZ
-    title: "{{ .CommonLabels.alertname }} on {{ .CommonLabels.exported_instance }}"
-```
-
-**Option C — email.** Requires Grafana SMTP configured via `GF_SMTP_*`
-environment variables on the Grafana service in `docker-compose.yml`, then an
-`email` receiver with `addresses`.
-
-Routing is a single flat policy in
-[policies.yaml](grafana/provisioning/alerting/policies.yaml): all alerts →
-`xrpld-default`, grouped by `alertname` + `exported_instance`. To route
-critical alerts to a different receiver, add child routes matching
-`severity = critical`.
-
---
-
-## Verifying provisioning loaded
-
-After the stack is up:
-
-```bash
-# All six rules present?
-curl -s http://localhost:3000/api/v1/provisioning/alert-rules | jq '.[].title'
-
-# Contact point present?
-curl -s http://localhost:3000/api/v1/provisioning/contact-points | jq '.[].name'
-```
-
-Grafana logs a provisioning error and skips the file if the YAML is malformed:
-
-```bash
-docker compose -f docker/telemetry/docker-compose.yml logs grafana | grep -i alerting
-```
--- a/docker/telemetry/docker-compose.yml
+++ b/docker/telemetry/docker-compose.yml
@@ -94,6 +94,13 @@ services:
  # Anonymous admin access enabled for local development convenience.
  grafana:
    image: grafana/grafana:11.5.2
+    # Alerting secrets/addresses (Slack webhook, alert email) come from this
+    # gitignored file; Grafana expands the ${VARS} referenced in the alerting
+    # provisioning YAML. Absent file = unset vars = receivers have no live
+    # destination, which is fine for a local stack. See .env.alerting.example.
+    env_file:
+      - path: .env.alerting
+        required: false
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true # No login required for local dev
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin # Full access without auth
@@ -102,6 +109,12 @@ services:
      # the callback URL is how the renderer fetches the panel from grafana.
      - GF_RENDERING_SERVER_URL=http://renderer:8081/render
      - GF_RENDERING_CALLBACK_URL=http://grafana:3000/
+    # SMTP for the critical-tier email receiver and the Slack webhook / email
+    # address for the contact points all come from the env_file above, which
+    # injects them straight into the container environment for Grafana to
+    # expand. They are deliberately NOT duplicated here: a compose-level
+    # `environment:` entry is interpolated from the host shell and would
+    # override (and blank out) the env_file values.
    ports:
      - "3000:3000" # Grafana web UI
    volumes:
--- a/docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml
+++ b/docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml
@@ -2,27 +2,53 @@
 #
 # Phase 9: Internal metric gap fill — alerting on health-critical metrics.
 #
-# A contact point is where a firing alert is delivered. This stack ships a
-# single webhook receiver as a working default for local development. To route
-# alerts to a real destination, replace the webhook `url` below, or add a
-# `slack` / `email` receiver to the same contact point (see ALERTING.md).
+# A contact point is where a firing alert is delivered. Two are defined:
+#   xrpld-default   — Slack only; receives warning-severity alerts.
+#   xrpld-critical  — Slack + email; receives critical-severity alerts.
+# The severity split is wired in policies.yaml.
 #
-# The default anonymous-admin Grafana in docker-compose.yml picks this file up
-# from /etc/grafana/provisioning/alerting/ on startup.
+# Secrets and personal addresses are NOT hard-coded. Grafana expands
+# ${ENV_VAR} references in provisioning files at load time, so the real
+# Slack webhook and alert-email addresses come from environment variables
+# supplied via docker/telemetry/.env.alerting (gitignored). See
+# .env.alerting.example for the variable list and the Alerting section of
+# docs/telemetry-runbook.md for setup.
+# If a variable is unset, that receiver has no live destination.
+#
+# Email delivery additionally requires SMTP configured on the Grafana
+# service (GF_SMTP_* in docker-compose.yml).

 apiVersion: 1

 contactPoints:
+  # --- Warning tier: Slack only ---
  - orgId: 1
    name: xrpld-default
    receivers:
-      # Local-dev webhook sink. Points at a placeholder host; alerts are
-      # emitted but silently dropped unless a receiver listens there.
-      # Swap the url for a real endpoint (PagerDuty/Opsgenie/custom) or
-      # replace the whole receiver block with a slack/email type.
-      - uid: xrpld-default-webhook
-        type: webhook
+      - uid: xrpld-slack-default
+        type: slack
        settings:
-          url: http://localhost:9999/xrpld-alerts
-          httpMethod: POST
+          # Incoming-webhook delivery: the channel is fixed by the webhook, so
+          # `recipient` is only here to satisfy Grafana's Slack validator.
+          url: ${SLACK_WEBHOOK_URL}
+          recipient: ${SLACK_CHANNEL}
+          title: "{{ .CommonLabels.alertname }} on {{ .CommonLabels.exported_instance }}"
+        disableResolveMessage: false
+
+  # --- Critical tier: Slack + email ---
+  - orgId: 1
+    name: xrpld-critical
+    receivers:
+      - uid: xrpld-slack-critical
+        type: slack
+        settings:
+          url: ${SLACK_WEBHOOK_URL}
+          recipient: ${SLACK_CHANNEL}
+          title: "[CRITICAL] {{ .CommonLabels.alertname }} on {{ .CommonLabels.exported_instance }}"
+        disableResolveMessage: false
+      - uid: xrpld-email-critical
+        type: email
+        settings:
+          addresses: ${ALERT_EMAIL_TO}
+          singleEmail: true
        disableResolveMessage: false
--- a/docker/telemetry/grafana/provisioning/alerting/policies.yaml
+++ b/docker/telemetry/grafana/provisioning/alerting/policies.yaml
@@ -3,8 +3,11 @@
 # Phase 9: Internal metric gap fill — alerting on health-critical metrics.
 #
 # The notification policy tree decides which contact point receives a firing
-# alert and how alerts are batched. This stack uses a single flat policy: every
-# xrpld alert routes to the `xrpld-default` contact point.
+# alert and how alerts are batched. Routing is split by severity:
+#   - default (root)        → xrpld-default  (Slack) — catches warnings.
+#   - severity = critical   → xrpld-critical (Slack + email) — child route.
+# A critical alert matches the child route and is delivered to Slack AND email;
+# everything else falls through to the root receiver.
 #
 # Grouping by alertname + exported_instance means one notification per
 # (alert, node) pair, so a fleet-wide problem does not collapse into a single
@@ -26,3 +29,16 @@ policies:
    # Minimum time before re-sending a notification for an alert that is still
    # firing.
    repeat_interval: 4h
+    routes:
+      # Critical alerts page harder: Slack + email. matchers uses the
+      # PromQL-style label match "severity = critical" set on the rules.
+      - receiver: xrpld-critical
+        matchers:
+          - severity = critical
+        group_by:
+          - alertname
+          - exported_instance
+        group_wait: 30s
+        group_interval: 5m
+        # Re-page critical alerts more often than the 4h default while still firing.
+        repeat_interval: 1h
--- a/docker/telemetry/grafana/provisioning/alerting/rules.yaml
+++ b/docker/telemetry/grafana/provisioning/alerting/rules.yaml
@@ -14,7 +14,8 @@
 #   B  reduce (last)      — collapse A's series to its most recent value.
 #   C  threshold          — the firing condition; `condition: C`.
 #
-# Thresholds are documented in ALERTING.md and are intended to be tuned.
+# Thresholds are documented in docs/telemetry-runbook.md (Alerting section)
+# and are intended to be tuned.

 apiVersion: 1

--- a/docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md
+++ b/docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md
@@ -1,655 +0,0 @@
-# External Dashboard Parity — Design Spec
-
-> **Date**: 2026-03-30
-> **Status**: Draft
-> **Source**: [realgrapedrop/xrpl-validator-dashboard](https://github.com/realgrapedrop/xrpl-validator-dashboard)
-> **Jira Epic**: RIPD-5060
-
-## Summary
-
-Integrate 29 missing metrics, 18 alert rules, and enriched span attributes from the community `xrpl-validator-dashboard` into xrpld's native OpenTelemetry instrumentation. Changes are distributed across phases 2, 3, 4, 6, 7, 9, 10, and 11 of the OTel PR chain.
-
-## Gap Analysis
-
-### Coverage Breakdown (86 external metrics)
-
-| Status            | Count | Notes                                                          |
-| ----------------- | ----- | -------------------------------------------------------------- |
-| Already covered   | 30    | peer_count, load_factor, io_latency, uptime, overlay traffic   |
-| Partially covered | 3     | state_value encoding, NuDB granularity, validation_quorum      |
-| Missing           | 29    | Validation agreement, ledger economy, peer quality, UNL health |
-| N/A (external)    | 24    | Monitor health, realtime duplicates, system metrics            |
-
-### Missing Metrics by Category
-
-| Category             | Metrics                                                                                                                                                                                                                            | Count |
-| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----- |
-| Validation Agreement | `validations_sent_total`, `validations_checked_total`, `validation_agreements_total`, `validation_missed_total`, `validation_agreement_pct_1h/24h`, `validation_agreements_1h/24h`, `validation_missed_1h/24h`, `validation_event` | 11    |
-| Ledger Economy       | `ledgers_closed_total`, `ledger_age_seconds`, `base_fee_xrp`, `reserve_base_xrp`, `reserve_inc_xrp`, `transaction_rate`                                                                                                            | 6     |
-| State Tracking       | `time_in_current_state_seconds`, `state_changes_total`, `validator_state_info`                                                                                                                                                     | 3     |
-| Peer Quality         | `peers_insane`, `peer_latency_p90_ms`                                                                                                                                                                                              | 2     |
-| Validator Health     | `amendment_blocked`, `unl_expiry_days`                                                                                                                                                                                             | 2     |
-| Upgrade Awareness    | `peers_higher_version_pct`, `upgrade_recommended`                                                                                                                                                                                  | 2     |
-| Storage / Other      | `ledger_nudb_bytes`, `jq_trans_overflow_total`, `initial_sync_duration_seconds`                                                                                                                                                    | 3     |
-
-### Alert Rules (18 total, from external dashboard)
-
-| Group       | Count | Rules                                                                                                                   |
-| ----------- | ----- | ----------------------------------------------------------------------------------------------------------------------- |
-| Critical    | 8     | Agreement <90%, not proposing, unhealthy state, amendment blocked, UNL expiring, IO latency, load factor, peer count <5 |
-| Network     | 3     | Peer drop >10%/30%, P90 latency + disconnect correlation                                                                |
-| Performance | 7     | CPU >80%, memory >90%, disk >85%, job queue overflow, upgrade recommended, tx rate drop, stale ledger                   |
-
---
-
-## Branch-to-Change Mapping
-
-### Phase 2 — `pratik/otel-phase2-rpc-tracing`
-
-> **Ref**: Adds to existing Phase 2 task list. Consumed by Phase 7 (MetricsRegistry) and Phase 10 (validation checks).
-
-**Task 2.8: RPC Span Attribute Enrichment**
-
-Add node-level health context to every `rpc.command.*` span so operators can correlate RPC behavior with node state.
-
-New span attributes on `rpc.command.*`:
-
-| Attribute                     | Type   | Source                               | Value Example         |
-| ----------------------------- | ------ | ------------------------------------ | --------------------- |
-| `xrpl.node.amendment_blocked` | bool   | `app_.getOPs().isAmendmentBlocked()` | `true`                |
-| `xrpl.node.server_state`      | string | `app_.getOPs().strOperatingMode()`   | `"full"`, `"syncing"` |
-
-**File**: `src/xrpld/rpc/detail/RPCHandler.cpp` (in the `rpc.command.*` span creation block, after existing setAttribute calls)
-
-**Rationale**: RPC is the operator's primary interaction point. When a node is amendment-blocked or degraded, every RPC response is suspect. Tagging spans with this state enables Tempo TraceQL queries like `{name=~"rpc.command.*" && span.xrpl.node.amendment_blocked = true}` to find all RPCs served during a blocked period.
-
-**Exit Criteria**:
-
- [ ] `rpc.command.server_info` spans carry `xrpl.node.amendment_blocked` and `xrpl.node.server_state` attributes
- [ ] No measurable latency impact (attribute values are cached atomics, not computed per-call)
-
---
-
-### Phase 3 — `pratik/otel-phase3-tx-tracing`
-
-> **Ref**: Adds to existing Phase 3 task list. Consumed by Phase 10 (validation checks).
-
-**Task 3.7: Transaction Span Peer Version Attribute**
-
-Add the relaying peer's xrpld version to transaction receive spans to enable version-mismatch correlation.
-
-New span attribute on `tx.receive`:
-
-| Attribute           | Type   | Source               | Value Example   |
-| ------------------- | ------ | -------------------- | --------------- |
-| `xrpl.peer.version` | string | `peer->getVersion()` | `"xrpld-2.4.0"` |
-
-**File**: `src/xrpld/overlay/detail/PeerImp.cpp` (in the `tx.receive` span block, after existing `xrpl.peer.id` setAttribute)
-
-**Rationale**: Transaction relay is where version mismatches cause subtle serialization or validation bugs. Tracing "this tx came from a v2.3.0 peer" helps diagnose compatibility issues during network upgrades.
-
-**Exit Criteria**:
-
- [ ] `tx.receive` spans carry `xrpl.peer.version` attribute with a non-empty version string
- [ ] Attribute is omitted (not empty-string) when `getVersion()` returns empty
-
---
-
-### Phase 4 — `pratik/otel-phase4-consensus-tracing`
-
-> **Ref**: Adds to existing Phase 4 task list. Provides the span-level foundation that Phase 7 (ValidationTracker) builds upon. Consumed by Phase 10 (validation checks).
-
-**Task 4.8: Consensus Validation Span Enrichment**
-
-Add ledger hash and validation type to validation spans on both send and receive paths. This enables trace-level agreement analysis — filter by ledger hash to see which validators agreed.
-
-New span attributes on `consensus.validation.send`:
-
-| Attribute                     | Type   | Source                                  | Value Example               |
-| ----------------------------- | ------ | --------------------------------------- | --------------------------- |
-| `xrpl.validation.ledger_hash` | string | Ledger hash from `validate()` call args | `"A1B2C3..."` (64-char hex) |
-| `xrpl.validation.full`        | bool   | Whether this is a full validation       | `true`                      |
-
-New span attributes on `peer.validation.receive`:
-
-| Attribute                          | Type   | Source                                | Value Example               |
-| ---------------------------------- | ------ | ------------------------------------- | --------------------------- |
-| `xrpl.peer.validation.ledger_hash` | string | From deserialized STValidation object | `"A1B2C3..."` (64-char hex) |
-| `xrpl.peer.validation.full`        | bool   | From STValidation flags               | `true`                      |
-
-New span attributes on `consensus.accept`:
-
-| Attribute                            | Type  | Source                                   | Value Example |
-| ------------------------------------ | ----- | ---------------------------------------- | ------------- |
-| `xrpl.consensus.validation_quorum`   | int64 | `app_.validators().quorum()`             | `28`          |
-| `xrpl.consensus.proposers_validated` | int64 | `result.proposers` from consensus result | `35`          |
-
-**Files**:
-
- `src/xrpld/app/consensus/RCLConsensus.cpp` (validation.send and accept spans)
- `src/xrpld/overlay/detail/PeerImp.cpp` (peer.validation.receive span)
-
-**Rationale**: The external dashboard's most valuable feature is validation agreement tracking. By recording the ledger hash on both outgoing and incoming validation spans, we create the raw data for agreement analysis at the trace level. Phase 7's ValidationTracker builds the metric-level aggregation on top of this.
-
-**Exit Criteria**:
-
- [ ] `consensus.validation.send` spans carry `xrpl.validation.ledger_hash` and `xrpl.validation.full`
- [ ] `peer.validation.receive` spans carry `xrpl.peer.validation.ledger_hash` and `xrpl.peer.validation.full`
- [ ] `consensus.accept` spans carry `xrpl.consensus.validation_quorum` and `xrpl.consensus.proposers_validated`
- [ ] Ledger hash attributes match between send and receive for the same ledger
-
---
-
-### Phase 6 — `pratik/otel-phase6-statsd`
-
-> **Ref**: Adds to existing Phase 6 scope. No separate task list file exists for Phase 6 per project convention.
-
-**Addition: Bridge `peerDisconnectsCharges_` metric**
-
-The overlay already tracks resource-limit disconnects via `OverlayImpl::Stats::peerDisconnectsCharges_` (a `beast::insight::Gauge`). This metric is registered but not included in the StatsD bridge mapping.
-
-**What to do**:
-
- Ensure `xrpld_Overlay_Peer_Disconnects_Charges` appears in the StatsD-to-Prometheus metric name mapping
- Verify the metric appears in Prometheus after StatsD bridge is active
-
-**File**: `src/xrpld/overlay/detail/OverlayImpl.cpp`
-
-**Prometheus name**: `xrpld_Overlay_Peer_Disconnects_Charges`
-
---
-
-### Phase 7 — `pratik/otel-phase7-native-metrics`
-
-> **Ref**: Adds to existing Phase 7 task list. This is the largest addition. Depends on Phase 4 span attributes for validation tracking context. Consumed by Phase 9 (dashboards), Phase 10 (validation), Phase 11 (alerts).
-
-**Task 7.8: ValidationTracker — Validation Agreement Computation**
-
-The most valuable missing component. A stateful class that tracks whether our validator's validations agree with network consensus, maintaining rolling 1h and 24h windows.
-
-**Architecture**:
-
-```
-
-  consensus.validation.send ─────> ValidationTracker ──────> MetricsRegistry
-  (records our validation          (reconciles after         (exports agreement
-   for ledger X)                    8s grace period)          gauges every 10s)
-
-  ledger.validate ───────────────> ValidationTracker
-  (records which ledger            (marks ledger X as
-   network validated)               agreed or missed)
-```
-
-**Design**:
-
-```cpp
-/// Tracks validation agreement between this node and network consensus.
-///
-///  ValidationTracker
-///  ├── recordOurValidation(ledgerHash, ledgerSeq)  // called when we send
-///  ├── recordNetworkValidation(ledgerHash, seq)     // called on ledger validate
-///  ├── reconcile()                                  // called periodically (timer)
-///  ├── agreementPct1h() -> double                   // 0.0-100.0
-///  ├── agreementPct24h() -> double
-///  ├── agreements1h() -> uint64_t
-///  ├── missed1h() -> uint64_t
-///  ├── agreements24h() -> uint64_t
-///  ├── missed24h() -> uint64_t
-///  ├── totalAgreements() -> uint64_t
-///  ├── totalMissed() -> uint64_t
-///  ├── totalValidationsSent() -> uint64_t
-///  └── totalValidationsChecked() -> uint64_t        // all network validations seen
-class ValidationTracker
-{
-    // Ring buffer of pending ledger events (max 1000)
-    struct LedgerEvent {
-        uint256 ledgerHash;
-        LedgerIndex seq;
-        TimePoint closeTime;
-        bool weValidated = false;     // did we send a validation for this ledger?
-        bool networkValidated = false; // did network validate this ledger?
-        bool reconciled = false;       // has 8s grace period elapsed?
-        bool agreed = false;           // after reconciliation: did we agree?
-    };
-
-    // Sliding window deques for pre-computed window stats
-    struct WindowEvent {
-        TimePoint time;
-        bool agreed;
-    };
-    std::deque<WindowEvent> window1h_;  // events in last 1 hour
-    std::deque<WindowEvent> window24h_; // events in last 24 hours
-
-    // Reconciliation: 8s grace period after ledger close.
-    // If our validation hasn't arrived by then, mark as missed.
-    // 5-minute late repair: if a late validation arrives, correct the miss.
-    static constexpr auto kGracePeriod = std::chrono::seconds(8);
-    static constexpr auto kLateRepairWindow = std::chrono::minutes(5);
-};
-```
-
-**Recording sites** (modifications to consensus code from Phase 7 branch):
-
-| Hook Point                   | File                | What to Record                                                          |
-| ---------------------------- | ------------------- | ----------------------------------------------------------------------- |
-| `validate()` in `doAccept()` | RCLConsensus.cpp    | `tracker.recordOurValidation(ledgerHash, seq)`                          |
-| `onValidation()` callback    | RCLValidations path | `tracker.recordNetworkValidation(...)` — increment `validationsChecked` |
-| LedgerMaster fully-validated | LedgerMaster.cpp    | `tracker.recordNetworkValidation(validatedHash, seq)`                   |
-
-**Key new files**:
-
- `src/xrpld/telemetry/ValidationTracker.h`
- `src/xrpld/telemetry/detail/ValidationTracker.cpp`
-
-**Key modified files**:
-
- `src/xrpld/telemetry/MetricsRegistry.h` (add ValidationTracker member)
- `src/xrpld/telemetry/MetricsRegistry.cpp` (add gauge callback reading from tracker)
- `src/xrpld/app/consensus/RCLConsensus.cpp` (add recording hooks)
- `src/xrpld/app/ledger/detail/LedgerMaster.cpp` (add recording hook)
-
-**Exit Criteria**:
-
- [ ] `ValidationTracker` correctly tracks agreement with 8s grace period
- [ ] 5-minute late repair corrects false-positive misses
- [ ] Thread-safe (atomics + mutex for window deques)
- [ ] Rolling windows correctly evict stale entries
- [ ] Unit tests for: normal agreement, missed validation, late repair, window eviction
-
---
-
-**Task 7.9: Validator Health Observable Gauges**
-
-New MetricsRegistry observable gauge for amendment, UNL, and quorum health.
-
-| Gauge Name               | Label `metric=`     | Type   | Source                                            |
-| ------------------------ | ------------------- | ------ | ------------------------------------------------- |
-| `xrpld_validator_health` | `amendment_blocked` | int64  | `app_.getOPs().isAmendmentBlocked()` → 0/1        |
-|                          | `unl_blocked`       | int64  | `app_.getOPs().isUNLBlocked()` → 0/1              |
-|                          | `unl_expiry_days`   | double | `app_.validators().expires()` → days until expiry |
-|                          | `validation_quorum` | int64  | `app_.validators().quorum()`                      |
-
-**File**: `src/xrpld/telemetry/MetricsRegistry.cpp` (new gauge callback in `registerAsyncGauges()`)
-
-**Exit Criteria**:
-
- [ ] All 4 label values emitted every 10s
- [ ] `unl_expiry_days` is negative when expired, positive when active
- [ ] Values visible in Prometheus
-
---
-
-**Task 7.10: Peer Quality Observable Gauges**
-
-New MetricsRegistry observable gauge for peer health aggregates.
-
-| Gauge Name           | Label `metric=`            | Type   | Source                                     |
-| -------------------- | -------------------------- | ------ | ------------------------------------------ |
-| `xrpld_peer_quality` | `peer_latency_p90_ms`      | double | Iterate peers, compute P90 from `latency_` |
-|                      | `peers_insane_count`       | int64  | Count peers with `tracking_ == diverged`   |
-|                      | `peers_higher_version_pct` | double | Compare `getVersion()` to own version      |
-|                      | `upgrade_recommended`      | int64  | 1 if `peers_higher_version_pct > 60%`      |
-
-**Implementation note**: The callback iterates `app_.overlay().foreach(...)` to collect per-peer latency and version data. This runs every 10s on the metrics reader thread — acceptable overhead for ~50-200 peers.
-
-**File**: `src/xrpld/telemetry/MetricsRegistry.cpp`
-
-**Exit Criteria**:
-
- [ ] P90 latency computed correctly (sort peer latencies, pick 90th percentile)
- [ ] Insane count matches `peers` RPC output
- [ ] Version comparison handles format variations (e.g., "xrpld-2.4.0-rc1")
- [ ] Values visible in Prometheus
-
---
-
-**Task 7.11: Ledger Economy Observable Gauges**
-
-New MetricsRegistry observable gauge for fee and ledger metrics.
-
-| Gauge Name             | Label `metric=`      | Type   | Source                                    |
-| ---------------------- | -------------------- | ------ | ----------------------------------------- |
-| `xrpld_ledger_economy` | `base_fee_xrp`       | double | `app_.getFeeTrack().getBaseFee()` → drops |
-|                        | `reserve_base_xrp`   | double | From validated ledger fee settings        |
-|                        | `reserve_inc_xrp`    | double | From validated ledger fee settings        |
-|                        | `ledger_age_seconds` | double | `now - lastValidatedCloseTime`            |
-|                        | `transaction_rate`   | double | Derived: tx count delta / time delta      |
-
-**File**: `src/xrpld/telemetry/MetricsRegistry.cpp`
-
-**Exit Criteria**:
-
- [ ] Fee values match `server_info` RPC output
- [ ] `ledger_age_seconds` increases monotonically between ledger closes, resets on close
- [ ] `transaction_rate` is smoothed (rolling average, not instantaneous)
-
---
-
-**Task 7.12: State Tracking Observable Gauges**
-
-New MetricsRegistry observable gauge for node state duration.
-
-| Gauge Name             | Label `metric=`                 | Type   | Source                                           |
-| ---------------------- | ------------------------------- | ------ | ------------------------------------------------ |
-| `xrpld_state_tracking` | `state_value`                   | int64  | 0-7 numeric encoding matching external dashboard |
-|                        | `time_in_current_state_seconds` | double | `now - lastModeChangeTime`                       |
-
-**State value encoding**:
-
-xrpld's `OperatingMode` enum maps 0-4 (DISCONNECTED through FULL). The external dashboard extends this to 0-6 by combining operating mode with consensus participation:
-
-| Value | State        | Source                                                      |
-| ----- | ------------ | ----------------------------------------------------------- |
-| 0     | disconnected | `OperatingMode::DISCONNECTED`                               |
-| 1     | connected    | `OperatingMode::CONNECTED`                                  |
-| 2     | syncing      | `OperatingMode::SYNCING`                                    |
-| 3     | tracking     | `OperatingMode::TRACKING`                                   |
-| 4     | full         | `OperatingMode::FULL` and not validating                    |
-| 5     | validating   | `OperatingMode::FULL` and `mConsensus.validating()` is true |
-| 6     | proposing    | `OperatingMode::FULL` and consensus mode is `proposing`     |
-
-**Note**: Values 5-6 require checking both `OperatingMode` and `ConsensusMode`. The callback should derive these from `app_.getOPs().getOperatingMode()` combined with `mConsensus.mode()`. If operating mode is FULL and consensus is proposing → 6; if FULL and validating → 5; otherwise use the raw OperatingMode enum value.
-
-**File**: `src/xrpld/telemetry/MetricsRegistry.cpp`
-
-**Exit Criteria**:
-
- [ ] `state_value` matches external dashboard encoding
- [ ] `time_in_current_state_seconds` resets on mode change
-
---
-
-**Task 7.13: Storage Detail Observable Gauge**
-
-| Gauge Name             | Label `metric=` | Type  | Source                                   |
-| ---------------------- | --------------- | ----- | ---------------------------------------- |
-| `xrpld_storage_detail` | `nudb_bytes`    | int64 | NuDB backend file size (filesystem stat) |
-
-**File**: `src/xrpld/telemetry/MetricsRegistry.cpp`
-
-**Exit Criteria**:
-
- [ ] NuDB file size reported in bytes
- [ ] Gracefully returns 0 if NuDB not configured
-
---
-
-**Task 7.14: New Synchronous Counters**
-
-New counters incremented at event sites. Declared in MetricsRegistry, recording sites added in consensus/overlay/network code.
-
-| Counter Name                        | Increment Site                   | Source File           |
-| ----------------------------------- | -------------------------------- | --------------------- |
-| `xrpld_ledgers_closed_total`        | `onAccept()` in consensus        | RCLConsensus.cpp      |
-| `xrpld_validations_sent_total`      | `validate()` in consensus        | RCLConsensus.cpp      |
-| `xrpld_validations_checked_total`   | Network validation received      | LedgerMaster.cpp      |
-| `xrpld_validation_agreements_total` | ValidationTracker reconciliation | ValidationTracker.cpp |
-| `xrpld_validation_missed_total`     | ValidationTracker reconciliation | ValidationTracker.cpp |
-| `xrpld_state_changes_total`         | `setMode()` in NetworkOPs        | NetworkOPs.cpp        |
-| `xrpld_jq_trans_overflow_total`     | Job queue overflow path          | JobQueue.cpp          |
-
-**Key modified files**:
-
- `src/xrpld/telemetry/MetricsRegistry.h/.cpp` (counter declarations)
- `src/xrpld/app/consensus/RCLConsensus.cpp` (recording: ledgers_closed, validations_sent)
- `src/xrpld/app/ledger/detail/LedgerMaster.cpp` (recording: validations_checked)
- `src/xrpld/app/misc/NetworkOPs.cpp` (recording: state_changes)
-
-**Exit Criteria**:
-
- [ ] All 7 counters monotonically increase during normal operation
- [ ] Counter values match expected rates (e.g., ledgers_closed ≈ 1 per 3-5s)
- [ ] Values visible in Prometheus
-
---
-
-**Task 7.15: Validation Agreement Observable Gauge**
-
-Reads from the `ValidationTracker` (Task 7.8) to export rolling window stats.
-
-| Gauge Name                   | Label `metric=`     | Type   | Source                      |
-| ---------------------------- | ------------------- | ------ | --------------------------- |
-| `xrpld_validation_agreement` | `agreement_pct_1h`  | double | `tracker.agreementPct1h()`  |
-|                              | `agreements_1h`     | int64  | `tracker.agreements1h()`    |
-|                              | `missed_1h`         | int64  | `tracker.missed1h()`        |
-|                              | `agreement_pct_24h` | double | `tracker.agreementPct24h()` |
-|                              | `agreements_24h`    | int64  | `tracker.agreements24h()`   |
-|                              | `missed_24h`        | int64  | `tracker.missed24h()`       |
-
-**File**: `src/xrpld/telemetry/MetricsRegistry.cpp`
-
-**Exit Criteria**:
-
- [ ] Agreement percentages in range [0.0, 100.0]
- [ ] Window stats match manual count from validation counters
- [ ] Percentages stabilize after 1h/24h of operation
-
---
-
-### Phase 9 — `pratik/otel-phase9-metric-gap-fill`
-
-> **Ref**: Adds to existing Phase 9 task list. Depends on Phase 7 gauges/counters. Consumed by Phase 10 (dashboard load checks).
-
-**Task 9.11: Validator Health Dashboard**
-
-New Grafana dashboard: `validator-health.json`
-
-| Panel                      | Type       | PromQL                                                         |
-| -------------------------- | ---------- | -------------------------------------------------------------- |
-| Agreement % (1h)           | stat       | `xrpld_validation_agreement{metric="agreement_pct_1h"}`        |
-| Agreement % (24h)          | stat       | `xrpld_validation_agreement{metric="agreement_pct_24h"}`       |
-| Agreements vs Missed (1h)  | bargauge   | `agreements_1h` and `missed_1h` side by side                   |
-| Agreements vs Missed (24h) | bargauge   | `agreements_24h` and `missed_24h` side by side                 |
-| Validation Rate            | stat       | `rate(xrpld_validations_sent_total[5m]) * 60`                  |
-| Validations Checked Rate   | stat       | `rate(xrpld_validations_checked_total[5m]) * 60`               |
-| Amendment Blocked          | stat       | `xrpld_validator_health{metric="amendment_blocked"}`           |
-| UNL Expiry (days)          | stat       | `xrpld_validator_health{metric="unl_expiry_days"}`             |
-| Validation Quorum          | stat       | `xrpld_validator_health{metric="validation_quorum"}`           |
-| State Value Timeline       | timeseries | `xrpld_state_tracking{metric="state_value"}`                   |
-| Time in Current State      | stat       | `xrpld_state_tracking{metric="time_in_current_state_seconds"}` |
-| State Changes Rate         | stat       | `rate(xrpld_state_changes_total[1h])`                          |
-| Ledgers Closed Rate        | stat       | `rate(xrpld_ledgers_closed_total[5m]) * 60`                    |
-
-**Dashboard conventions**: `$node` template variable for `exported_instance` filtering, dark theme, matching existing panel sizes and color schemes.
-
---
-
-**Task 9.12: Peer Quality Dashboard**
-
-New Grafana dashboard: `peer-quality.json`
-
-| Panel                  | Type       | PromQL                                                         |
-| ---------------------- | ---------- | -------------------------------------------------------------- |
-| P90 Peer Latency       | timeseries | `xrpld_peer_quality{metric="peer_latency_p90_ms"}`             |
-| Insane/Diverged Peers  | stat       | `xrpld_peer_quality{metric="peers_insane_count"}`              |
-| Higher Version Peers % | stat       | `xrpld_peer_quality{metric="peers_higher_version_pct"}`        |
-| Upgrade Recommended    | stat       | `xrpld_peer_quality{metric="upgrade_recommended"}`             |
-| Resource Disconnects   | timeseries | `xrpld_Overlay_Peer_Disconnects_Charges`                       |
-| Inbound vs Outbound    | bargauge   | `xrpld_Peer_Finder_Active_Inbound_Peers`, `..._Outbound_Peers` |
-
---
-
-**Task 9.13: Ledger Economy Dashboard Panels**
-
-Add a "Ledger Economy" row to the existing `node-health.json` dashboard:
-
-| Panel                | Type       | PromQL                                              |
-| -------------------- | ---------- | --------------------------------------------------- |
-| Base Fee (drops)     | stat       | `xrpld_ledger_economy{metric="base_fee_xrp"}`       |
-| Reserve Base (drops) | stat       | `xrpld_ledger_economy{metric="reserve_base_xrp"}`   |
-| Reserve Inc (drops)  | stat       | `xrpld_ledger_economy{metric="reserve_inc_xrp"}`    |
-| Ledger Age           | stat       | `xrpld_ledger_economy{metric="ledger_age_seconds"}` |
-| Transaction Rate     | timeseries | `xrpld_ledger_economy{metric="transaction_rate"}`   |
-
---
-
-### Phase 10 — `pratik/otel-phase10-workload-validation`
-
-> **Ref**: Adds to existing Phase 10 task list. Validates all additions from Phases 2-9.
-
-**Task 10.6: External Dashboard Parity Validation Checks**
-
-Add checks to `validate_telemetry.py` for all new span attributes and metrics.
-
-**New span attribute checks (~8)**:
-
-| Span Name                   | New Attribute                        |
-| --------------------------- | ------------------------------------ |
-| `rpc.command.server_info`   | `xrpl.node.amendment_blocked`        |
-| `rpc.command.server_info`   | `xrpl.node.server_state`             |
-| `tx.receive`                | `xrpl.peer.version`                  |
-| `consensus.validation.send` | `xrpl.validation.ledger_hash`        |
-| `consensus.validation.send` | `xrpl.validation.full`               |
-| `peer.validation.receive`   | `xrpl.peer.validation.ledger_hash`   |
-| `consensus.accept`          | `xrpl.consensus.validation_quorum`   |
-| `consensus.accept`          | `xrpl.consensus.proposers_validated` |
-
-**New metric existence checks (~13)**:
-
-| Metric Name                                              |
-| -------------------------------------------------------- |
-| `xrpld_validation_agreement{metric="agreement_pct_1h"}`  |
-| `xrpld_validation_agreement{metric="agreement_pct_24h"}` |
-| `xrpld_validator_health{metric="amendment_blocked"}`     |
-| `xrpld_validator_health{metric="unl_expiry_days"}`       |
-| `xrpld_peer_quality{metric="peer_latency_p90_ms"}`       |
-| `xrpld_peer_quality{metric="peers_insane_count"}`        |
-| `xrpld_ledger_economy{metric="base_fee_xrp"}`            |
-| `xrpld_ledger_economy{metric="transaction_rate"}`        |
-| `xrpld_state_tracking{metric="state_value"}`             |
-| `xrpld_ledgers_closed_total`                             |
-| `xrpld_validations_sent_total`                           |
-| `xrpld_state_changes_total`                              |
-| `xrpld_storage_detail{metric="nudb_bytes"}`              |
-
-**New dashboard load checks (~3)**:
-
-| Dashboard               |
-| ----------------------- |
-| `validator-health`      |
-| `peer-quality`          |
-| `node-health` (updated) |
-
-**New metric value sanity checks (~4)**:
-
-| Check                         | Condition         |
-| ----------------------------- | ----------------- |
-| `validation_agreement_pct_1h` | in [0, 100]       |
-| `unl_expiry_days`             | > 0 (not expired) |
-| `peer_latency_p90_ms`         | > 0 (peers exist) |
-| `state_value`                 | in [0, 7]         |
-
-**Total new checks: ~28** (bringing total from 73 to ~101)
-
---
-
-### Phase 11 — (future branch)
-
-> **Ref**: Adds to existing Phase 11 task list. Depends on Phase 7 metrics and Phase 9 dashboards.
-
-**Task 11.9: Alert Rules from External Dashboard**
-
-Port 18 alert rules from the external `xrpl-validator-dashboard` to Grafana alerting provisioning.
-
-**Critical Group** (8 rules, eval interval 10s):
-
-| Rule                | Condition                                                     | For |
-| ------------------- | ------------------------------------------------------------- | --- |
-| Agreement Below 90% | `xrpld_validation_agreement{metric="agreement_pct_24h"} < 90` | 30s |
-| Not Proposing       | `xrpld_state_tracking{metric="state_value"} < 6`              | 10s |
-| Unhealthy State     | `xrpld_state_tracking{metric="state_value"} < 4`              | 10s |
-| Amendment Blocked   | `xrpld_validator_health{metric="amendment_blocked"} == 1`     | 1m  |
-| UNL Expiring        | `xrpld_validator_health{metric="unl_expiry_days"} < 14`       | 1h  |
-| High IO Latency     | `histogram_quantile(0.95, xrpld_ios_latency_bucket) > 50`     | 1m  |
-| High Load Factor    | `xrpld_load_factor_metrics{metric="load_factor"} > 1000`      | 1m  |
-| Peer Count Critical | `xrpld_server_info{metric="peers"} < 5`                       | 1m  |
-
-**Network Group** (3 rules, eval interval 10s):
-
-| Rule                      | Condition                                                         | For |
-| ------------------------- | ----------------------------------------------------------------- | --- |
-| Peer Drop >10%            | `delta(xrpld_server_info{metric="peers"}[30s]) / ... * 100 < -10` | 30s |
-| Peer Drop >30%            | Same formula, threshold -30                                       | 30s |
-| P90 Latency + Disconnects | `peer_latency_p90_ms > 500 AND rate(disconnects) > 0`             | 2m  |
-
-**Performance Group** (7 rules, eval interval 10s):
-
-| Rule                | Condition                                                    | For |
-| ------------------- | ------------------------------------------------------------ | --- |
-| CPU High            | Per-core CPU > 80%                                           | 2m  |
-| Memory Critical     | Memory usage > 90%                                           | 1m  |
-| Disk Warning        | Disk usage > 85%                                             | 2m  |
-| Job Queue Overflow  | `rate(xrpld_jq_trans_overflow_total[5m]) > 0`                | 1m  |
-| Upgrade Recommended | `xrpld_peer_quality{metric="peers_higher_version_pct"} > 60` | 1m  |
-| TX Rate Drop        | Transaction rate dropped > 50% in 5m window                  | 5m  |
-| Stale Ledger        | `xrpld_ledger_economy{metric="ledger_age_seconds"} > 30`     | 1m  |
-
-**Notification channels**: Template configs for Email/SMTP, Discord, Slack, PagerDuty.
-
-**Files**:
-
- `docker/telemetry/grafana/alerting/alert-rules.yaml` (new or extend existing)
- `docker/telemetry/grafana/alerting/contact-points.yaml`
- `docker/telemetry/grafana/alerting/notification-policies.yaml`
-
---
-
-**Task 11.10: Dual-Datasource Architecture Documentation**
-
-Document the external dashboard's "fast path" pattern as a future optimization for real-time panels:
-
- **Pattern**: A lightweight Prometheus scrape endpoint (separate from OTLP pipeline) that polls critical metrics every 2-5s, bypassing the 10s OTLP metric reader interval and Prometheus scrape interval.
- **Use case**: Real-time state panels (server state, ledger age, peer count) where 10-15s latency is too slow.
- **Decision**: Document as a future option, not implement now. Current 10s interval is acceptable for v1.
-
-**File**: `OpenTelemetryPlan/Phase11_taskList.md` (documentation task, no code)
-
---
-
-## Documentation Updates
-
-### `docs/telemetry-runbook.md` (on Phase 9 branch)
-
-Add new sections after "Phase 9: OTel Metrics Alerting Rules":
-
-1. **Validator Health Monitoring** — explains agreement tracking, amendment blocked, UNL expiry, with example PromQL queries
-2. **Peer Quality Monitoring** — explains P90 latency, insane peers, version awareness
-3. **Ledger Economy Monitoring** — explains fee/reserve gauges, transaction rate, ledger age
-4. **Validation Agreement Explained** — operator-facing explanation of the reconciliation algorithm (8s grace, 5m late repair), what "missed" means, and when to worry
-
-### `OpenTelemetryPlan/09-data-collection-reference.md` (on Phase 9 branch)
-
-Add new metric tables in a "Phase 7+: External Dashboard Parity" section covering all 29 new metrics with their gauge names, label values, types, and sources.
-
---
-
-## Cross-Phase Dependency Chain
-
-```
-Phase 2 (span attrs: amendment_blocked, server_state)
-Phase 3 (span attrs: peer.version)
-Phase 4 (span attrs: validation.ledger_hash, validation.full, quorum)
-Phase 6 (StatsD bridge: peerDisconnectsCharges)
-    │
-    ├── all above rebase into ──>
-    │
-Phase 7 (ValidationTracker + 7 gauges + 7 counters + agreement gauge)
-    │
-Phase 9 (3 dashboards + ledger economy panels + runbook + data-collection-ref)
-    │
-Phase 10 (28 new validation checks in validate_telemetry.py)
-    │
-Phase 11 (18 alert rules + dual-datasource docs)
-```
-
-## Rebase Strategy
-
-After committing changes to each branch (starting from Phase 2):
-
-1. Commit on `pratik/otel-phase2-rpc-tracing`
-2. Rebase `phase3` onto `phase2`, resolve conflicts (task list files only — low risk)
-3. Commit on `phase3`, rebase `phase4` onto `phase3`
-4. Continue through chain: 4 → 5 → 5b → 6 → 7 → 8 → 9 → 10
-5. Force-push-with-lease all affected branches
-
-Since these are documentation-only changes (task list .md files), merge conflicts should be minimal — each file is unique to its branch.
--- a/docs/superpowers/specs/2026-07-06-phase9-alerting-design.md
+++ b/docs/superpowers/specs/2026-07-06-phase9-alerting-design.md
@@ -1,99 +0,0 @@
-# Phase-9 Alerting — Design
-
-**Date:** 2026-07-06
-**Branch:** `pratik/otel-phase9-metric-gap-fill` (PR #6513, Jira RIPD-5187)
-**Status:** Approved
-
-## Purpose
-
-Phase 9 exports ~68 internal rippled metrics and ships Grafana dashboards for
-them. This adds the missing operator-facing piece: **provisioned Grafana alert
-rules** that fire on the health-critical metrics phase 9 introduces. The
-phase-9 task list (line 311) and Jira story RIPD-5187 both already list
-"alerting rules" as a phase-9 deliverable, so this closes that gap.
-
-Scope is deliberately narrow — the three subsystems whose failure is
-node-fatal: **consensus/ledger health, validator health, job queue**. RPC/API
-health is explicitly out of scope.
-
-## Why phase 9 (not phase 11)
-
-Every metric these alerts fire on is _born_ in phase 9
-(`xrpld_ledger_history_mismatch_total`, `xrpld_ledgers_closed_total`,
-`xrpld_validation_missed_total`, `xrpld_validations_checked_total`,
-`xrpld_jq_trans_overflow_total`, `xrpld_job_queued_duration_us_bucket`). Alerts
-belong with the metrics they watch, and this is where the dependency lives.
-
-## Delivery
-
-Provisioned YAML, version-controlled — matching the existing datasource /
-dashboard provisioning pattern. No docker-compose change: the Grafana service
-already mounts `./grafana/provisioning:/etc/grafana/provisioning:ro`, and
-Grafana auto-loads `provisioning/alerting/*.yaml`.
-
-New files under `docker/telemetry/grafana/provisioning/alerting/`:
-
-| File                 | Purpose                                                                                                             |
-| -------------------- | ------------------------------------------------------------------------------------------------------------------- |
-| `contactpoints.yaml` | One contact point `xrpld-default` (webhook to a documented placeholder; comments show how to swap for Slack/email). |
-| `policies.yaml`      | Default notification policy: route all alerts → `xrpld-default`, grouped by `alertname` + `exported_instance`.      |
-| `rules.yaml`         | 6 alert rules across 3 groups (below).                                                                              |
-
-Plus `docker/telemetry/ALERTING.md` — operator runbook: what each alert means,
-likely causes, and how to point the contact point at a real receiver.
-
-## Alert rules
-
-All rules target Prometheus datasource `uid: prometheus`. Each rule uses the
-Grafana rule shape: query (A) → reduce (B, last value) → threshold (C). All
-`rate()`/`histogram_quantile()` expressions aggregate with
-`sum by (exported_instance)` (or `+ le`) so **each node alerts independently**.
-Alert rules run headless, so they cannot use the dashboards' `$node` template
-variables — they match all series and group by `exported_instance` instead.
-
-| Group     | Alert                 | Expression (5m window)                                                                                    | Fires                 | `for` | severity |
-| --------- | --------------------- | --------------------------------------------------------------------------------------------------------- | --------------------- | ----- | -------- |
-| Consensus | LedgerHistoryMismatch | `sum by (exported_instance)(rate(xrpld_ledger_history_mismatch_total[5m]))`                               | `> 0`                 | 5m    | critical |
-| Consensus | LedgerCloseStalled    | `sum by (exported_instance)(rate(xrpld_ledgers_closed_total[5m]))`                                        | `< 0.001` (≈0)        | 3m    | critical |
-| Validator | ValidationsMissed     | `sum by (exported_instance)(rate(xrpld_validation_missed_total[5m]))`                                     | `> 0`                 | 5m    | warning  |
-| Validator | ValidationsNotChecked | `sum by (exported_instance)(rate(xrpld_validations_checked_total[5m]))`                                   | `< 0.001` (≈0)        | 5m    | warning  |
-| Job queue | JobQueueTxOverflow    | `sum by (exported_instance)(rate(xrpld_jq_trans_overflow_total[5m]))`                                     | `> 0`                 | 5m    | warning  |
-| Job queue | JobQueueLatencyHigh   | `histogram_quantile(0.99, sum by (le, exported_instance)(rate(xrpld_job_queued_duration_us_bucket[5m])))` | `> 1000000` (µs = 1s) | 5m    | warning  |
-
-Each rule carries labels `severity` and `category` (consensus/validator/jobqueue)
-and annotations `summary` + `description` (with `{{ $labels.exported_instance }}`
-and `{{ $values.B.Value }}` interpolation).
-
-### Threshold rationale
-
- **LedgerCloseStalled `< 0.001` for 3m**: healthy nodes close a ledger every
-  ~3-5s; a 5m rate decaying to ~0 means the node is stuck. The epsilon (not
-  exact `0`) avoids float rate-noise suppressing the alert.
- **JobQueueLatencyHigh 1s p99**: a default starting point, easy to tune — jobs
-  queued >1s at p99 indicate the node is saturated.
- Others are `> 0` on error/miss counters: any sustained nonzero rate is
-  actionable.
-
-## Non-goals / YAGNI
-
- No per-alert silencing schedules, no mute timings.
- No RPC/API, overlay, or fee-market alerts (dashboards cover those visually).
- Single contact point — multi-receiver routing is left to the operator.
-
-## Verification
-
-1. `yamllint` (or `python -c yaml.safe_load`) on all three YAML files.
-2. `docker compose -f docker/telemetry/docker-compose.yml config -q` still parses.
-3. Optional live check: start stack, `GET /api/v1/provisioning/alert-rules`
-   returns the 6 rules; Grafana logs show no provisioning errors.
-4. Code-review pass (subagent) against phase conventions before commit.
-
-## Branch / PR restructure (independent, same session)
-
- Rename `pratik/otel-phase11-benchmark` → `pratik/perf-test-otel-on`
-  (0 unique commits — equals phase-10 tip; the telemetry-IN baseline).
- Rename `pratik/otel-phase11-telemetry-off` → `pratik/perf-test-otel-off`
-  (carries the "compile telemetry OUT" build change + benchmark runbook).
- Close PR #7433, delete stale `origin/pratik/otel-phase11-telemetry-off`.
- Push both; open one perf PR `perf-test-otel-off → perf-test-otel-on`
-  (diff = the ON-vs-OFF delta; `-on` needs no PR of its own).
--- a/docs/telemetry-runbook.md
+++ b/docs/telemetry-runbook.md
@@ -1011,14 +1011,17 @@ Requires `trace_peer=1` in the `[telemetry]` config section.

 ## Alerting

-Grafana ships six provisioned alert rules on the health-critical metrics, so a
-stock stack alerts out of the box with no UI setup. Rules are loaded from
-`docker/telemetry/grafana/provisioning/alerting/` on startup and appear under
-**Alerting → Alert rules**, folder **xrpld**.
+rippled provisions six Grafana alert rules on the health-critical metrics, so a
+stock stack alerts out of the box with no UI setup. Rules are provisioned from
+`docker/telemetry/grafana/provisioning/alerting/` and load automatically when
+the Grafana container starts. They appear under **Alerting → Alert rules**,
+folder **xrpld**.

-Each rule runs every minute against the Prometheus datasource over a 5-minute
-window and groups by `exported_instance`, so every node alerts independently.
-An alert fires only after its condition holds for the `for` dwell time.
+### Alert catalogue
+
+All rules evaluate every minute against the Prometheus datasource, over a
+5-minute window, and group by `exported_instance` so each node alerts on its
+own. Alerts fire only after the condition holds for the `for` dwell time.

 | Alert                   | Severity | Fires when                                      | For |
 | ----------------------- | -------- | ----------------------------------------------- | --- |
@@ -1029,11 +1032,113 @@ An alert fires only after its condition holds for the `for` dwell time.
 | `JobQueueTxOverflow`    | warning  | `rate(xrpld_jq_trans_overflow_total)` > 0       | 5m  |
 | `JobQueueLatencyHigh`   | warning  | p99 `xrpld_job_queued_duration_us` > 1s         | 5m  |

-Alerts route through the `xrpld-default` contact point, which ships as a
-local-dev webhook to a placeholder URL — alerts fire but go nowhere until it is
-pointed at a real receiver. See
-[ALERTING.md](../docker/telemetry/ALERTING.md) for per-alert causes, threshold
-tuning, and how to wire alerts to Slack/email/PagerDuty.
+#### Consensus / ledger health
+
+**LedgerHistoryMismatch** — The node closed a ledger whose history diverges
+from the validated network chain. Likely causes: corrupted local state, a bug,
+or a node that fell out of sync and rebuilt incorrectly. Investigate the node's
+ledger acquisition logs; a healthy node never mismatches.
+
+**LedgerCloseStalled** — No ledgers closed for 3 minutes. A healthy node closes
+one every ~3-5s. Likely causes: lost peer connectivity, consensus stall, or the
+process is hung. This rule also fires on _NoData_ — if the series disappears the
+node is likely down. Check peer count and process health first.
+
+#### Validator health
+
+**ValidationsMissed** — This validator's validations are not agreeing with the
+validated ledger. Sustained misses risk removal from UNLs. Check clock sync,
+peer connectivity, and whether the node is keeping up with ledger close.
+
+**ValidationsNotChecked** — The node has stopped checking incoming validations
+from peers. Likely causes: overlay/peer disconnection or a stalled validation
+pipeline. Fires on NoData as well.
+
+#### Job queue / resource health
+
+**JobQueueTxOverflow** — The transaction job queue is full and transactions are
+being dropped. The node is shedding load it cannot process. Check CPU, the
+`JobQueueLatencyHigh` alert, and offered load.
+
+**JobQueueLatencyHigh** — p99 queue wait exceeds 1 second, i.e. jobs back up
+before running. The node is saturated. Correlate with CPU and the Job Queue
+dashboard.
+
+### Tuning thresholds
+
+Thresholds live in
+`docker/telemetry/grafana/provisioning/alerting/rules.yaml` as the `params`
+array of each rule's `C` (threshold) node. Common tunables:
+
+- **`JobQueueLatencyHigh`** — `params: [1000000]` is 1 000 000 µs (1s). Lower
+  it for latency-sensitive deployments.
+- **`LedgerCloseStalled` / `ValidationsNotChecked`** — use `lt` with a tiny
+  epsilon (`0.001`) rather than `0`, so floating-point rate noise near zero
+  does not suppress the alert.
+
+Edit the file and restart the Grafana container to reload:
+
+```bash
+docker compose -f docker/telemetry/docker-compose.yml restart grafana
+```
+
+### Sending alerts somewhere real
+
+Two contact points are provisioned in
+`docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml`:
+
+| Contact point    | Receivers     | Gets                     |
+| ---------------- | ------------- | ------------------------ |
+| `xrpld-default`  | Slack         | warning-severity alerts  |
+| `xrpld-critical` | Slack + email | critical-severity alerts |
+
+The severity split lives in
+`docker/telemetry/grafana/provisioning/alerting/policies.yaml`: the root route
+sends everything to `xrpld-default`, and a child route matching
+`severity = critical` overrides to `xrpld-critical`. So a critical alert goes
+to Slack **and** email; a warning goes to Slack only. Both group by
+`alertname` + `exported_instance`; critical alerts re-page hourly vs the 4h default.
+
+#### Configure delivery (no secrets in git)
+
+The Slack webhook and email address are **not** hard-coded — the YAML
+references `${SLACK_WEBHOOK_URL}` and `${ALERT_EMAIL_TO}`, which Grafana
+expands from the environment at startup. Supply them through a gitignored
+env file:
+
+```bash
+cp docker/telemetry/.env.alerting.example docker/telemetry/.env.alerting
+# edit .env.alerting — this file is gitignored, never commit the webhook/address
+docker compose -f docker/telemetry/docker-compose.yml up -d grafana
+```
+
+- **Slack** — set `SLACK_WEBHOOK_URL` to an incoming-webhook URL. Drives both
+  tiers.
+- **Email** — set `ALERT_EMAIL_TO` (comma-separated) **and** point the
+  `GF_SMTP_*` vars at a real relay with `GF_SMTP_ENABLED=true`. Grafana can
+  only send mail once SMTP is configured.
+
+Any variable left blank disables that path; the stack still runs. To add a
+third destination (PagerDuty, Opsgenie, a custom webhook), add a receiver to
+the relevant contact point.
+
+### Verifying alert provisioning loaded
+
+After the stack is up:
+
+```bash
+# All six rules present?
+curl -s http://localhost:3000/api/v1/provisioning/alert-rules | jq '.[].title'
+
+# Contact points present?
+curl -s http://localhost:3000/api/v1/provisioning/contact-points | jq '.[].name'
+```
+
+Grafana logs a provisioning error and skips the file if the YAML is malformed:
+
+```bash
+docker compose -f docker/telemetry/docker-compose.yml logs grafana | grep -i alerting
+```

 ## Log-Trace Correlation