ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-07-28 09:30:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation

Browse Source
This commit is contained in:
Pratik Mankawde
2026-07-04 02:49:47 +01:00
parent 3c4e67b33f 83f8fb7063
commit 21bcedfd4a
20 changed files with 76 additions and 76 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
OpenTelemetryPlan/08-appendix.md
View File

@@ -229,7 +229,7 @@ This guide maps Phase 9–11 content to its location across the documentation.
| Task list (10 tasks) | [Phase9_taskList.md](./Phase9_taskList.md) |
| Future metric definitions (~50) | [09-data-collection-reference.md §5b](./09-data-collection-reference.md) |
| New class: `MetricsRegistry` | `src/xrpld/telemetry/MetricsRegistry.h/.cpp` (planned) |
| New dashboards | `xrpld-fee-market`, `xrpld-job-queue` (planned) |
| New dashboards | `fee-market`, `job-queue` (planned) |
**Metric categories**: NodeStore I/O, Cache Hit Rates, TxQ, PerfLog Per-RPC, PerfLog Per-Job, Counted Objects, Fee Escalation & Load Factors.

74
OpenTelemetryPlan/09-data-collection-reference.md
View File

@@ -102,7 +102,7 @@ Controlled by `trace_rpc=1` in `[telemetry]` config.
**Where to find**: Tempo → TraceQL: `{resource.service.name="xrpld" && name=~"rpc.http_request|rpc.command.*"}`
**Grafana dashboard**: _RPC Performance_ (`xrpld-rpc-perf`)
**Grafana dashboard**: _RPC Performance_ (`rpc-performance`)
#### gRPC Spans
@@ -118,7 +118,7 @@ rates without TraceQL attribute filters.
**Where to find**: Tempo → TraceQL: `{resource.service.name="xrpld" && name=~"grpc.*"}`
**Grafana dashboard**: _RPC Performance_ (`xrpld-rpc-perf`)
**Grafana dashboard**: _RPC Performance_ (`rpc-performance`)
#### Transaction Spans
@@ -142,7 +142,7 @@ later spans — the `stage` attribute identifies where it stopped.
**Where to find**: Tempo → TraceQL: `{resource.service.name="xrpld" && name=~"tx.process|tx.receive"}`
or, for the apply pipeline: `{resource.service.name="xrpld" && name=~"tx.preflight|tx.preclaim|tx.transactor"}`
**Grafana dashboard**: _Transaction Overview_ (`xrpld-transactions`)
**Grafana dashboard**: _Transaction Overview_ (`transaction-overview`)
#### Transaction Queue (TxQ) Spans
@@ -159,7 +159,7 @@ Controlled by `trace_transactions=1` in `[telemetry]` config.
**Where to find**: Tempo → TraceQL: `{resource.service.name="xrpld" && name=~"txq.*"}`
**Grafana dashboard**: _Transaction Overview_ (`xrpld-transactions`)
**Grafana dashboard**: _Transaction Overview_ (`transaction-overview`)
#### Consensus Spans
@@ -187,7 +187,7 @@ round trace via context propagation rather than direct parenting. The
**Where to find**: Tempo → TraceQL: `{resource.service.name="xrpld" && name=~"consensus.*"}`
**Grafana dashboard**: _Consensus Health_ (`xrpld-consensus`)
**Grafana dashboard**: _Consensus Health_ (`consensus-health`)
#### Ledger Spans
@@ -202,7 +202,7 @@ Controlled by `trace_ledger=1` in `[telemetry]` config.
**Where to find**: Tempo → TraceQL: `{resource.service.name="xrpld" && name=~"ledger.*"}`
**Grafana dashboard**: _Ledger Operations_ (`xrpld-ledger-ops`)
**Grafana dashboard**: _Ledger Operations_ (`ledger-operations`)
#### Peer Spans
@@ -215,7 +215,7 @@ Controlled by `trace_peer` in `[telemetry]` config. **Enabled by default** (high
**Where to find**: Tempo → TraceQL: `{resource.service.name="xrpld" && name=~"peer.*"}`
**Grafana dashboard**: _Peer Network_ (`xrpld-peer-net`)
**Grafana dashboard**: _Peer Network_ (`peer-network`)
#### PathFind Spans
@@ -527,7 +527,7 @@ prefix=xrpld
| `xrpld_Overlay_Peer_Disconnects_Charges` | OverlayImpl.cpp | Disconnects due to resource limit charges | Low growth (subset of above) |
| `xrpld_jobq_job_count` | JobQueue.cpp | Current job queue depth (group `jobq`) | 0–100 (healthy) |
**Grafana dashboard**: _Node Health_ (`xrpld-system-node-health`)
**Grafana dashboard**: _Node Health_ (`node-health`)
### 2.2 Counters
@@ -541,7 +541,7 @@ prefix=xrpld
**Note**: With `server=otel`, `xrpld_warn` and `xrpld_drop` are properly exported as OTel Counter instruments. The previous StatsD `|m` type limitation no longer applies.
**Grafana dashboard**: _RPC & Pathfinding_ (`xrpld-system-rpc`)
**Grafana dashboard**: _RPC & Pathfinding_ (`rpc-pathfinding`)
### 2.3 Histograms (Event timers)
@@ -583,7 +583,7 @@ For each of the 45+ overlay traffic categories (defined in `TrafficCount.h`), fo
| `ping` / `status` | Keepalive and status |
| `set_get` | Set requests |
**Grafana dashboards**: _Network Traffic_ (`xrpld-system-network`), _Overlay Traffic Detail_ (`xrpld-system-overlay-detail`), _Ledger Data & Sync_ (`xrpld-system-ledger-sync`)
**Grafana dashboards**: _Network Traffic_ (`network-traffic`), _Overlay Traffic Detail_ (`overlay-traffic-detail`), _Ledger Data & Sync_ (`ledger-data-sync`)
---
@@ -593,23 +593,23 @@ For each of the 45+ overlay traffic categories (defined in `TrafficCount.h`), fo
### 3.1 Span-Derived Dashboards (5)
| Dashboard | UID | Data Source | Key Panels |
| -------------------- | -------------------- | ------------------------ | ---------------------------------------------------------------------------------- |
| RPC Performance | `xrpld-rpc-perf` | Prometheus (SpanMetrics) | Request rate by command, p95 latency by command, error rate, heatmap, top commands |
| Transaction Overview | `xrpld-transactions` | Prometheus (SpanMetrics) | Processing rate, latency p95/p50, local vs relay split, apply duration, heatmap |
| Consensus Health | `xrpld-consensus` | Prometheus (SpanMetrics) | Round duration p95/p50, proposals rate, close duration, mode timeline, heatmap |
| Ledger Operations | `xrpld-ledger-ops` | Prometheus (SpanMetrics) | Build rate, build duration, validation rate, store rate, build vs close comparison |
| Peer Network | `xrpld-peer-net` | Prometheus (SpanMetrics) | Proposal receive rate, validation receive rate, trusted vs untrusted breakdown |
| Dashboard | UID | Data Source | Key Panels |
| -------------------- | ---------------------- | ------------------------ | ---------------------------------------------------------------------------------- |
| RPC Performance | `rpc-performance` | Prometheus (SpanMetrics) | Request rate by command, p95 latency by command, error rate, heatmap, top commands |
| Transaction Overview | `transaction-overview` | Prometheus (SpanMetrics) | Processing rate, latency p95/p50, local vs relay split, apply duration, heatmap |
| Consensus Health | `consensus-health` | Prometheus (SpanMetrics) | Round duration p95/p50, proposals rate, close duration, mode timeline, heatmap |
| Ledger Operations | `ledger-operations` | Prometheus (SpanMetrics) | Build rate, build duration, validation rate, store rate, build vs close comparison |
| Peer Network | `peer-network` | Prometheus (SpanMetrics) | Proposal receive rate, validation receive rate, trusted vs untrusted breakdown |
### 3.2 System Metrics Dashboards (5)
| Dashboard | UID | Data Source | Key Panels |
| ---------------------- | ----------------------------- | ----------------- | --------------------------------------------------------------------------------- |
| Node Health | `xrpld-system-node-health` | Prometheus (OTLP) | Ledger age, operating mode, I/O latency, job queue, fetch rate |
| Network Traffic | `xrpld-system-network` | Prometheus (OTLP) | Active peers, disconnects, bytes in/out, messages in/out, traffic by category |
| RPC & Pathfinding | `xrpld-system-rpc` | Prometheus (OTLP) | RPC rate, response time/size, pathfinding duration, resource warnings/drops |
| Overlay Traffic Detail | `xrpld-system-overlay-detail` | Prometheus (OTLP) | Squelch, overhead, validator lists, set get/share, have/requested tx, proof paths |
| Ledger Data & Sync | `xrpld-system-ledger-sync` | Prometheus (OTLP) | Ledger data exchange, legacy ledger share/get, getobject by type, traffic heatmap |
| Dashboard | UID | Data Source | Key Panels |
| ---------------------- | ------------------------ | ----------------- | --------------------------------------------------------------------------------- |
| Node Health | `node-health` | Prometheus (OTLP) | Ledger age, operating mode, I/O latency, job queue, fetch rate |
| Network Traffic | `network-traffic` | Prometheus (OTLP) | Active peers, disconnects, bytes in/out, messages in/out, traffic by category |
| RPC & Pathfinding | `rpc-pathfinding` | Prometheus (OTLP) | RPC rate, response time/size, pathfinding duration, resource warnings/drops |
| Overlay Traffic Detail | `overlay-traffic-detail` | Prometheus (OTLP) | Squelch, overhead, validator lists, set get/share, have/requested tx, proof paths |
| Ledger Data & Sync | `ledger-data-sync` | Prometheus (OTLP) | Ledger data exchange, legacy ledger share/get, getobject by type, traffic heatmap |
### 3.3 Deployment-Tier Template Variables
@@ -873,10 +873,10 @@ async callbacks for new categories.
### New Grafana Dashboards (Phase 9)
| Dashboard | UID | Data Source | Key Panels |
| ------------------ | ------------------ | ----------- | ----------------------------------------------------------------- |
| Fee Market & TxQ | `xrpld-fee-market` | Prometheus | TxQ depth/capacity, fee levels, load factor breakdown, escalation |
| Job Queue Analysis | `xrpld-job-queue` | Prometheus | Per-job rates, queue wait times, execution times, queue depth |
| Dashboard | UID | Data Source | Key Panels |
| ------------------ | ------------ | ----------- | ----------------------------------------------------------------- |
| Fee Market & TxQ | `fee-market` | Prometheus | TxQ depth/capacity, fee levels, load factor breakdown, escalation |
| Job Queue Analysis | `job-queue` | Prometheus | Per-job rates, queue wait times, execution times, queue depth |
---
@@ -1211,26 +1211,26 @@ Lifetime validation agreement/miss tallies are exported as monotonic **Observabl
### New Grafana Dashboards (Phase 9)
| Dashboard | UID | Data Source | Key Panels |
| ------------------------------------ | ------------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------ |
| Fee Market & TxQ | `xrpld-fee-market` | Prometheus | TxQ depth/capacity, fee levels, load factor breakdown |
| Job Queue Analysis | `xrpld-job-queue` | Prometheus | Per-job rates, queue wait times, execution times |
| RPC Performance (per-method section) | `xrpld-rpc-perf` | Prometheus | Per-method call rates, error rates, latency distributions (added as a section to the existing RPC Performance dashboard) |
| Validator Health | `xrpld-validator-health` | Prometheus | Agreement %, validation rate, amendment/UNL, state |
| Peer Quality | `xrpld-peer-quality` | Prometheus | P90 latency, insane peers, version awareness, disconnects |
| Dashboard | UID | Data Source | Key Panels |
| ------------------------------------ | ------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------ |
| Fee Market & TxQ | `fee-market` | Prometheus | TxQ depth/capacity, fee levels, load factor breakdown |
| Job Queue Analysis | `job-queue` | Prometheus | Per-job rates, queue wait times, execution times |
| RPC Performance (per-method section) | `rpc-performance` | Prometheus | Per-method call rates, error rates, latency distributions (added as a section to the existing RPC Performance dashboard) |
| Validator Health | `validator-health` | Prometheus | Agreement %, validation rate, amendment/UNL, state |
| Peer Quality | `peer-quality` | Prometheus | P90 latency, insane peers, version awareness, disconnects |
### Updated Grafana Dashboards (Phase 9)
| Dashboard | UID | New Panels Added |
| -------------------- | -------------------------- | -------------------------------------------------------------------- |
| Node Health (StatsD) | `xrpld-statsd-node-health` | NodeStore I/O, cache hit rates, object instance counts |
| System Node Health | `xrpld-system-node-health` | Ledger economy row: base fee, reserves, ledger age, transaction rate |
| System Node Health | `node-health` | Ledger economy row: base fee, reserves, ledger age, transaction rate |
### New Grafana Dashboards (Phase 11)
| Dashboard | UID | Data Source | Key Panels |
| ------------------ | --------------------------- | ----------- | ---------------------------------------------------------------------- |
| Validator Health | `xrpld-validator-health` | Prometheus | Server state timeline, proposer count, converge time, amendment voting |
| Validator Health | `validator-health` | Prometheus | Server state timeline, proposer count, converge time, amendment voting |
| Network Topology | `xrpld-network-topology` | Prometheus | Peer count, version distribution, latency distribution, diverged peers |
| Fee Market (Ext) | `xrpld-fee-market-external` | Prometheus | Fee levels, queue depth, load factor breakdown, escalation timeline |
| DEX & AMM Overview | `xrpld-dex-amm` | Prometheus | AMM TVL, order book depth, spread trends, trading fee revenue |

2
OpenTelemetryPlan/Phase11_taskList.md
View File

@@ -362,7 +362,7 @@ This phase addresses the cross-cutting gap identified during research: **xrpld h
**What to do**:
- **Validator Health** (`xrpld-validator-health`):
- **Validator Health** (`validator-health`):
- Server state timeline, state duration breakdown
- Proposer count trend, converge time trend, validation quorum
- Validator list expiration countdown

20
OpenTelemetryPlan/Phase9_taskList.md
View File

@@ -116,7 +116,7 @@ These metrics serve multiple external consumer categories identified during rese
**Derived Prometheus metrics**: `xrpld_txq_count`, `xrpld_txq_max_size`, `xrpld_txq_open_ledger_fee_level`, etc.
**Grafana dashboard**: New _Fee Market & TxQ_ dashboard (`xrpld-fee-market`).
**Grafana dashboard**: New _Fee Market & TxQ_ dashboard (`fee-market`).
---
@@ -169,7 +169,7 @@ These metrics serve multiple external consumer categories identified during rese
**Derived Prometheus metrics**: `xrpld_job_queued_total{job_type="ledgerData"}`, `xrpld_job_running_duration_us_bucket{job_type="transaction"}`, etc.
**Grafana dashboard**: New _Job Queue Analysis_ dashboard (`xrpld-job-queue`).
**Grafana dashboard**: New _Job Queue Analysis_ dashboard (`job-queue`).
---
@@ -269,7 +269,7 @@ These metrics serve multiple external consumer categories identified during rese
**Derived Prometheus metrics**: `xrpld_server_info{metric="server_state"}`, `xrpld_build_info{version="2.4.0"}`, `xrpld_complete_ledgers{bound="start",index="0"}`, `xrpld_db_metrics{metric="db_kb_total"}`, etc.
**Grafana dashboard**: New panels added to _Node Health_ dashboard (`system-node-health.json`).
**Grafana dashboard**: New panels added to _Node Health_ dashboard (`node-health.json`).
---
@@ -280,12 +280,12 @@ These metrics serve multiple external consumer categories identified during rese
**What to do**:
- Create 2 new dashboards:
1. **Fee Market & TxQ** (`xrpld-fee-market`) — TxQ depth/capacity, fee levels, load factor breakdown, fee escalation timeline
2. **Job Queue Analysis** (`xrpld-job-queue`) — Per-job-type rates, queue wait times, execution times, job queue depth
1. **Fee Market & TxQ** (`fee-market`) — TxQ depth/capacity, fee levels, load factor breakdown, fee escalation timeline
2. **Job Queue Analysis** (`job-queue`) — Per-job-type rates, queue wait times, execution times, job queue depth
- Update 2 existing dashboards:
1. **Node Health** (`xrpld-statsd-node-health`) — Add NodeStore I/O panels, cache hit rate panels, object instance counts
2. **RPC Performance** (`xrpld-rpc-perf`) — Add per-method RPC breakdown panels
2. **RPC Performance** (`rpc-performance`) — Add per-method RPC breakdown panels
**Key modified files**:
@@ -351,7 +351,7 @@ These metrics serve multiple external consumer categories identified during rese
**Objective**: Create a Grafana dashboard for validation agreement, amendment/UNL health, and state tracking.
**Dashboard**: `xrpld-validator-health.json`
**Dashboard**: `validator-health.json`
| Panel | Type | PromQL |
| -------------------------- | ---------- | -------------------------------------------------------------- |
@@ -387,7 +387,7 @@ These metrics serve multiple external consumer categories identified during rese
**Objective**: Create a Grafana dashboard for peer health aggregates.
**Dashboard**: `xrpld-peer-quality.json`
**Dashboard**: `peer-quality.json`
| Panel | Type | PromQL |
| ---------------------- | ---------- | -------------------------------------------------------------- |
@@ -412,7 +412,7 @@ These metrics serve multiple external consumer categories identified during rese
> **Source**: [External Dashboard Parity](../docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md)
**Objective**: Add "Ledger Economy" row to the existing `system-node-health.json` dashboard.
**Objective**: Add "Ledger Economy" row to the existing `node-health.json` dashboard.
| Panel | Type | PromQL |
| -------------------- | ---------- | --------------------------------------------------- |
@@ -422,7 +422,7 @@ These metrics serve multiple external consumer categories identified during rese
| Ledger Age | stat | `xrpld_ledger_economy{metric="ledger_age_seconds"}` |
| Transaction Rate | timeseries | `xrpld_ledger_economy{metric="transaction_rate"}` |
**Key modified files**: `docker/telemetry/grafana/dashboards/system-node-health.json`
**Key modified files**: `docker/telemetry/grafana/dashboards/node-health.json`
**Exit Criteria**: