ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-07-28 09:30:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation

Browse Source
This commit is contained in:
Pratik Mankawde
2026-07-08 17:49:56 +01:00
parent e76eed5296 b3a25456b1
commit 6a50635f10
20 changed files with 465 additions and 475 deletions
Download Patch File Download Diff File Expand all files Collapse all files

208
docs/telemetry-runbook.md
View File

@@ -738,64 +738,64 @@ The `OTelCollector` implementation exports metrics via OTLP/HTTP to the same OTe
#### Gauges
| Prometheus Metric | Source | Description |
| ------------------------------------------- | ------------------------- | -------------------------------------------------------------------------- |
| `xrpld_LedgerMaster_Validated_Ledger_Age` | LedgerMaster.h:373 | Age of validated ledger (seconds) |
| `xrpld_LedgerMaster_Published_Ledger_Age` | LedgerMaster.h:374 | Age of published ledger (seconds) |
| `xrpld_State_Accounting_{Mode}_duration` | NetworkOPs.cpp:774 | Time in each operating mode (Disconnected/Connected/Syncing/Tracking/Full) |
| `xrpld_State_Accounting_{Mode}_transitions` | NetworkOPs.cpp:780 | Transition count per mode |
| `xrpld_Peer_Finder_Active_Inbound_Peers` | PeerfinderManager.cpp:214 | Active inbound peer connections |
| `xrpld_Peer_Finder_Active_Outbound_Peers` | PeerfinderManager.cpp:215 | Active outbound peer connections |
| `xrpld_Overlay_Peer_Disconnects` | OverlayImpl.h:557 | Peer disconnect count |
| `xrpld_job_count` | JobQueue.cpp:26 | Current job queue depth |
| `xrpld_{category}_Bytes_In/Out` | OverlayImpl.h:535 | Overlay traffic bytes per category (57 categories) |
| `xrpld_{category}_Messages_In/Out` | OverlayImpl.h:535 | Overlay traffic messages per category |
| Prometheus Metric | Source | Description |
| ------------------------------------- | ------------------------- | -------------------------------------------------------------------------- |
| `ledgermaster_validated_ledger_age` | LedgerMaster.h:373 | Age of validated ledger (seconds) |
| `ledgermaster_published_ledger_age` | LedgerMaster.h:374 | Age of published ledger (seconds) |
| `state_accounting_{mode}_duration` | NetworkOPs.cpp:774 | Time in each operating mode (Disconnected/Connected/Syncing/Tracking/Full) |
| `state_accounting_{mode}_transitions` | NetworkOPs.cpp:780 | Transition count per mode |
| `peer_finder_active_inbound_peers` | PeerfinderManager.cpp:214 | Active inbound peer connections |
| `peer_finder_active_outbound_peers` | PeerfinderManager.cpp:215 | Active outbound peer connections |
| `overlay_peer_disconnects` | OverlayImpl.h:557 | Peer disconnect count |
| `job_count` | JobQueue.cpp:26 | Current job queue depth |
| `{category}_bytes_in/Out` | OverlayImpl.h:535 | Overlay traffic bytes per category (57 categories) |
| `{category}_messages_in/Out` | OverlayImpl.h:535 | Overlay traffic messages per category |
#### OTel MetricsRegistry Gauges
These gauges are exported via the OTel Metrics SDK `PeriodicMetricReader` (10s interval), NOT through beast::insight.
| Prometheus Metric | Source | Description |
| --------------------------------------------------------- | ------------------- | -------------------------------------------- |
| `xrpld_server_info{metric="server_state"}` | MetricsRegistry.cpp | Operating mode (0=DISCONNECTED .. 4=FULL) |
| `xrpld_server_info{metric="uptime"}` | MetricsRegistry.cpp | Seconds since server start |
| `xrpld_server_info{metric="peers"}` | MetricsRegistry.cpp | Total connected peers |
| `xrpld_server_info{metric="validated_ledger_seq"}` | MetricsRegistry.cpp | Validated ledger sequence number |
| `xrpld_server_info{metric="ledger_current_index"}` | MetricsRegistry.cpp | Current open ledger sequence |
| `xrpld_server_info{metric="peer_disconnects_resources"}` | MetricsRegistry.cpp | Cumulative resource-related peer disconnects |
| `xrpld_server_info{metric="last_close_proposers"}` | MetricsRegistry.cpp | Proposers in last closed round |
| `xrpld_server_info{metric="last_close_converge_time_ms"}` | MetricsRegistry.cpp | Last close convergence time (ms) |
| `xrpld_build_info{version="<ver>"}` | MetricsRegistry.cpp | Info-style metric (always 1) |
| `xrpld_complete_ledgers{bound="start\|end",index="<N>"}` | MetricsRegistry.cpp | Complete ledger range start/end pairs |
| `xrpld_db_metrics{metric="db_kb_total"}` | MetricsRegistry.cpp | Total database size (KB) |
| `xrpld_db_metrics{metric="db_kb_ledger"}` | MetricsRegistry.cpp | Ledger database size (KB) |
| `xrpld_db_metrics{metric="db_kb_transaction"}` | MetricsRegistry.cpp | Transaction database size (KB) |
| `xrpld_db_metrics{metric="historical_perminute"}` | MetricsRegistry.cpp | Historical ledger fetches per minute |
| `xrpld_cache_metrics{metric="AL_size"}` | MetricsRegistry.cpp | AcceptedLedger cache size |
| `xrpld_nodestore_state{metric="node_reads_duration_us"}` | MetricsRegistry.cpp | Cumulative read time (microseconds) |
| `xrpld_nodestore_state{metric="read_request_bundle"}` | MetricsRegistry.cpp | Read request bundle count |
| `xrpld_nodestore_state{metric="read_threads_running"}` | MetricsRegistry.cpp | Active read threads |
| `xrpld_nodestore_state{metric="read_threads_total"}` | MetricsRegistry.cpp | Total read threads configured |
| Prometheus Metric | Source | Description |
| --------------------------------------------------- | ------------------- | -------------------------------------------- |
| `server_info{metric="server_state"}` | MetricsRegistry.cpp | Operating mode (0=DISCONNECTED .. 4=FULL) |
| `server_info{metric="uptime"}` | MetricsRegistry.cpp | Seconds since server start |
| `server_info{metric="peers"}` | MetricsRegistry.cpp | Total connected peers |
| `server_info{metric="validated_ledger_seq"}` | MetricsRegistry.cpp | Validated ledger sequence number |
| `server_info{metric="ledger_current_index"}` | MetricsRegistry.cpp | Current open ledger sequence |
| `server_info{metric="peer_disconnects_resources"}` | MetricsRegistry.cpp | Cumulative resource-related peer disconnects |
| `server_info{metric="last_close_proposers"}` | MetricsRegistry.cpp | Proposers in last closed round |
| `server_info{metric="last_close_converge_time_ms"}` | MetricsRegistry.cpp | Last close convergence time (ms) |
| `build_info{version="<ver>"}` | MetricsRegistry.cpp | Info-style metric (always 1) |
| `complete_ledgers{bound="start\|end",index="<N>"}` | MetricsRegistry.cpp | Complete ledger range start/end pairs |
| `db_metrics{metric="db_kb_total"}` | MetricsRegistry.cpp | Total database size (KB) |
| `db_metrics{metric="db_kb_ledger"}` | MetricsRegistry.cpp | Ledger database size (KB) |
| `db_metrics{metric="db_kb_transaction"}` | MetricsRegistry.cpp | Transaction database size (KB) |
| `db_metrics{metric="historical_perminute"}` | MetricsRegistry.cpp | Historical ledger fetches per minute |
| `cache_metrics{metric="AL_size"}` | MetricsRegistry.cpp | AcceptedLedger cache size |
| `nodestore_state{metric="node_reads_duration_us"}` | MetricsRegistry.cpp | Cumulative read time (microseconds) |
| `nodestore_state{metric="read_request_bundle"}` | MetricsRegistry.cpp | Read request bundle count |
| `nodestore_state{metric="read_threads_running"}` | MetricsRegistry.cpp | Active read threads |
| `nodestore_state{metric="read_threads_total"}` | MetricsRegistry.cpp | Total read threads configured |
#### Counters
| Prometheus Metric | Source | Description |
| ------------------------------- | --------------------- | ------------------------------ |
| `xrpld_rpc_requests` | ServerHandler.cpp:108 | Total RPC request count |
| `xrpld_ledger_fetches` | InboundLedgers.cpp:44 | Ledger fetch request count |
| `xrpld_ledger_history_mismatch` | LedgerHistory.cpp:16 | Ledger hash mismatch count |
| `xrpld_warn` | Logic.h:33 | Resource manager warning count |
| `xrpld_drop` | Logic.h:34 | Resource manager drop count |
| Prometheus Metric | Source | Description |
| ------------------------- | --------------------- | ------------------------------ |
| `rpc_requests` | ServerHandler.cpp:108 | Total RPC request count |
| `ledger_fetches` | InboundLedgers.cpp:44 | Ledger fetch request count |
| `ledger_history_mismatch` | LedgerHistory.cpp:16 | Ledger hash mismatch count |
| `warn` | Logic.h:33 | Resource manager warning count |
| `drop` | Logic.h:34 | Resource manager drop count |
#### Histograms
| Prometheus Metric | Source | Description |
| --------------------- | --------------------- | ------------------------------ |
| `xrpld_rpc_time` | ServerHandler.cpp:110 | RPC response time (ms) |
| `xrpld_rpc_size` | ServerHandler.cpp:109 | RPC response size (bytes) |
| `xrpld_ios_latency` | Application.cpp:438 | I/O service loop latency (ms) |
| `xrpld_pathfind_fast` | PathRequests.h:23 | Fast pathfinding duration (ms) |
| `xrpld_pathfind_full` | PathRequests.h:24 | Full pathfinding duration (ms) |
| Prometheus Metric | Source | Description |
| ----------------- | --------------------- | ------------------------------ |
| `rpc_time` | ServerHandler.cpp:110 | RPC response time (ms) |
| `rpc_size` | ServerHandler.cpp:109 | RPC response size (bytes) |
| `ios_latency` | Application.cpp:438 | I/O service loop latency (ms) |
| `pathfind_fast` | PathRequests.h:23 | Fast pathfinding duration (ms) |
| `pathfind_full` | PathRequests.h:24 | Full pathfinding duration (ms) |
## Deployment Tiers
@@ -952,60 +952,60 @@ Requires `trace_peer=1` in the `[telemetry]` config section.
### Node Health -- System Metrics (`node-health`)
| Panel | Type | PromQL | Labels Used |
| -------------------------------------- | ---------- | --------------------------------------------------------------- | ---------------- |
| Validated Ledger Age | stat | `xrpld_LedgerMaster_Validated_Ledger_Age` | — |
| Published Ledger Age | stat | `xrpld_LedgerMaster_Published_Ledger_Age` | — |
| Operating Mode Duration | timeseries | `xrpld_State_Accounting_*_duration` | — |
| Operating Mode Transitions | timeseries | `xrpld_State_Accounting_*_transitions` | — |
| I/O Latency | timeseries | `histogram_quantile(0.95, xrpld_ios_latency_bucket)` | — |
| Job Queue Depth | timeseries | `xrpld_job_count` | — |
| Ledger Fetch Rate | stat | `rate(xrpld_ledger_fetches[5m])` | — |
| Ledger History Mismatches | stat | `rate(xrpld_ledger_history_mismatch[5m])` | — |
| Key Jobs Execution Time | timeseries | `xrpld_acceptLedger{quantile="$quantile"}` (+ 10 more key jobs) | `quantile` |
| Key Jobs Dequeue Wait Time | timeseries | `xrpld_acceptLedger_q{quantile="$quantile"}` (+ 10 more) | `quantile` |
| FullBelowCache Size | timeseries | `xrpld_Node_family_full_below_cache_size` | — |
| FullBelowCache Hit Rate | gauge | `xrpld_Node_family_full_below_cache_hit_rate` | — |
| Ledger Publish Gap | stat | `Published_Ledger_Age - Validated_Ledger_Age` | — |
| State Duration Rate (Full vs Tracking) | timeseries | `rate(xrpld_State_Accounting_Full_duration[5m]) / 1000000` | — |
| All Jobs Execution Time (Detail) | timeseries | `{__name__=~"xrpld_<all_jobs>", quantile="$quantile"}` | `quantile` |
| All Jobs Dequeue Wait (Detail) | timeseries | `{__name__=~"xrpld_<all_jobs>_q", quantile="$quantile"}` | `quantile` |
| Server State | stat | `xrpld_server_info{metric="server_state"}` | `metric` |
| Uptime | stat | `xrpld_server_info{metric="uptime"}` | `metric` |
| Peer Count | stat | `xrpld_server_info{metric="peers"}` | `metric` |
| Validated Ledger Seq | stat | `xrpld_server_info{metric="validated_ledger_seq"}` | `metric` |
| Build Version | stat | `xrpld_build_info` | `version` |
| Complete Ledger Ranges | table | `xrpld_complete_ledgers` | `bound`, `index` |
| Database Sizes | timeseries | `xrpld_db_metrics{metric=~"db_kb_.*"}` | `metric` |
| Historical Fetch Rate | stat | `xrpld_db_metrics{metric="historical_perminute"}` | `metric` |
| Panel | Type | PromQL | Labels Used |
| -------------------------------------- | ---------- | --------------------------------------------------------- | ---------------- |
| Validated Ledger Age | stat | `ledgermaster_validated_ledger_age` | — |
| Published Ledger Age | stat | `ledgermaster_published_ledger_age` | — |
| Operating Mode Duration | timeseries | `state_accounting_*_duration` | — |
| Operating Mode Transitions | timeseries | `state_accounting_*_transitions` | — |
| I/O Latency | timeseries | `histogram_quantile(0.95, ios_latency_bucket)` | — |
| Job Queue Depth | timeseries | `job_count` | — |
| Ledger Fetch Rate | stat | `rate(ledger_fetches[5m])` | — |
| Ledger History Mismatches | stat | `rate(ledger_history_mismatch[5m])` | — |
| Key Jobs Execution Time | timeseries | `acceptledger{quantile="$quantile"}` (+ 10 more key jobs) | `quantile` |
| Key Jobs Dequeue Wait Time | timeseries | `acceptledger_q{quantile="$quantile"}` (+ 10 more) | `quantile` |
| FullBelowCache Size | timeseries | `node_family_full_below_cache_size` | — |
| FullBelowCache Hit Rate | gauge | `node_family_full_below_cache_hit_rate` | — |
| Ledger Publish Gap | stat | `Published_Ledger_Age - Validated_Ledger_Age` | — |
| State Duration Rate (Full vs Tracking) | timeseries | `rate(state_accounting_full_duration[5m]) / 1000000` | — |
| All Jobs Execution Time (Detail) | timeseries | `{__name__=~"<all_jobs>", quantile="$quantile"}` | `quantile` |
| All Jobs Dequeue Wait (Detail) | timeseries | `{__name__=~"<all_jobs>_q", quantile="$quantile"}` | `quantile` |
| Server State | stat | `server_info{metric="server_state"}` | `metric` |
| Uptime | stat | `server_info{metric="uptime"}` | `metric` |
| Peer Count | stat | `server_info{metric="peers"}` | `metric` |
| Validated Ledger Seq | stat | `server_info{metric="validated_ledger_seq"}` | `metric` |
| Build Version | stat | `build_info` | `version` |
| Complete Ledger Ranges | table | `complete_ledgers` | `bound`, `index` |
| Database Sizes | timeseries | `db_metrics{metric=~"db_kb_.*"}` | `metric` |
| Historical Fetch Rate | stat | `db_metrics{metric="historical_perminute"}` | `metric` |
### Network Traffic -- System Metrics (`network-traffic`)
| Panel | Type | PromQL | Labels Used |
| ------------------------------------ | ---------- | ------------------------------------------ | ----------- |
| Active Peers | timeseries | `xrpld_Peer_Finder_Active_*_Peers` | — |
| Peer Disconnects | timeseries | `xrpld_Overlay_Peer_Disconnects` | — |
| Total Network Bytes | timeseries | `rate(xrpld_total_Bytes_In/Out[5m])` | — |
| Total Network Messages | timeseries | `xrpld_total_Messages_In/Out` | — |
| Transaction Traffic | timeseries | `xrpld_transactions_Messages_In/Out` | — |
| Proposal Traffic | timeseries | `xrpld_proposals_Messages_In/Out` | — |
| Validation Traffic | timeseries | `xrpld_validations_Messages_In/Out` | — |
| Traffic by Category | bargauge | `topk(10, xrpld_*_Bytes_In)` | — |
| Duplicate Traffic (Wasted Bandwidth) | timeseries | `rate(xrpld_*_duplicate_Bytes_In/Out[5m])` | — |
| All Traffic Categories (Detail) | timeseries | `topk(15, rate(xrpld_*_Bytes_In[5m]))` | — |
| Panel | Type | PromQL | Labels Used |
| ------------------------------------ | ---------- | ------------------------------------ | ----------- |
| Active Peers | timeseries | `peer_finder_active_*_Peers` | — |
| Peer Disconnects | timeseries | `overlay_peer_disconnects` | — |
| Total Network Bytes | timeseries | `rate(total_bytes_in/Out[5m])` | — |
| Total Network Messages | timeseries | `total_messages_in/Out` | — |
| Transaction Traffic | timeseries | `transactions_messages_in/Out` | — |
| Proposal Traffic | timeseries | `proposals_messages_in/Out` | — |
| Validation Traffic | timeseries | `validations_messages_in/Out` | — |
| Traffic by Category | bargauge | `topk(10, *_bytes_in)` | — |
| Duplicate Traffic (Wasted Bandwidth) | timeseries | `rate(*_duplicate_bytes_in/out[5m])` | — |
| All Traffic Categories (Detail) | timeseries | `topk(15, rate(*_bytes_in[5m]))` | — |
### RPC & Pathfinding -- System Metrics (`rpc-pathfinding`)
| Panel | Type | PromQL | Labels Used |
| ------------------------- | ---------- | ------------------------------------------------------ | ----------- |
| RPC Request Rate | stat | `rate(xrpld_rpc_requests[5m])` | — |
| RPC Response Time | timeseries | `histogram_quantile(0.95, xrpld_rpc_time_bucket)` | — |
| RPC Response Size | timeseries | `histogram_quantile(0.95, xrpld_rpc_size_bucket)` | — |
| RPC Response Time Heatmap | heatmap | `xrpld_rpc_time_bucket` | — |
| Pathfinding Fast Duration | timeseries | `histogram_quantile(0.95, xrpld_pathfind_fast_bucket)` | — |
| Pathfinding Full Duration | timeseries | `histogram_quantile(0.95, xrpld_pathfind_full_bucket)` | — |
| Resource Warnings Rate | stat | `rate(xrpld_warn[5m])` | — |
| Resource Drops Rate | stat | `rate(xrpld_drop[5m])` | — |
| Panel | Type | PromQL | Labels Used |
| ------------------------- | ---------- | ------------------------------------------------ | ----------- |
| RPC Request Rate | stat | `rate(rpc_requests[5m])` | — |
| RPC Response Time | timeseries | `histogram_quantile(0.95, rpc_time_bucket)` | — |
| RPC Response Size | timeseries | `histogram_quantile(0.95, rpc_size_bucket)` | — |
| RPC Response Time Heatmap | heatmap | `rpc_time_bucket` | — |
| Pathfinding Fast Duration | timeseries | `histogram_quantile(0.95, pathfind_fast_bucket)` | — |
| Pathfinding Full Duration | timeseries | `histogram_quantile(0.95, pathfind_full_bucket)` | — |
| Resource Warnings Rate | stat | `rate(warn[5m])` | — |
| Resource Drops Rate | stat | `rate(drop[5m])` | — |
### Span → Metric → Dashboard Summary
@@ -1058,14 +1058,14 @@ All rules evaluate every minute against the Prometheus datasource, over a
5-minute window, and group by `service_instance_id` so each node alerts on its
own. Alerts fire only after the condition holds for the `for` dwell time.
| Alert | Severity | Fires when | For |
| ----------------------- | -------- | ----------------------------------------------- | --- |
| `LedgerHistoryMismatch` | critical | `rate(xrpld_ledger_history_mismatch_total)` > 0 | 5m |
| `LedgerCloseStalled` | critical | `rate(xrpld_ledgers_closed_total)` ≈ 0 | 3m |
| `ValidationsMissed` | warning | `rate(xrpld_validation_missed_total)` > 0 | 5m |
| `ValidationsNotChecked` | warning | `rate(xrpld_validations_checked_total)` ≈ 0 | 5m |
| `JobQueueTxOverflow` | warning | `rate(xrpld_jq_trans_overflow_total)` > 0 | 5m |
| `JobQueueLatencyHigh` | warning | p99 `xrpld_job_queued_duration_us` > 1s | 5m |
| Alert | Severity | Fires when | For |
| ----------------------- | -------- | ----------------------------------------- | --- |
| `LedgerHistoryMismatch` | critical | `rate(ledger_history_mismatch_total)` > 0 | 5m |
| `LedgerCloseStalled` | critical | `rate(ledgers_closed_total)` ≈ 0 | 3m |
| `ValidationsMissed` | warning | `rate(validation_missed_total)` > 0 | 5m |
| `ValidationsNotChecked` | warning | `rate(validations_checked_total)` ≈ 0 | 5m |
| `JobQueueTxOverflow` | warning | `rate(jq_trans_overflow_total)` > 0 | 5m |
| `JobQueueLatencyHigh` | warning | p99 `job_queued_us` > 1s | 5m |
#### Consensus / ledger health
@@ -1243,7 +1243,7 @@ count_over_time({service_name="xrpld"} |= "trace_id=" [5m])
2. Verify `server=otel` in the `[insight]` config section
3. Verify the endpoint in `[insight]` points to the OTLP/HTTP port (default: `http://localhost:4318/v1/metrics`)
4. Check that the `otlp` receiver is in the metrics pipeline receivers in `otel-collector-config.yaml`
5. Query Prometheus directly: `curl 'http://localhost:9090/api/v1/query?query=xrpld_job_count'`
5. Query Prometheus directly: `curl 'http://localhost:9090/api/v1/query?query=job_count'`
### Server info gauge shows server_state=0