From 7231d450a4e0f739fc2a4a2889697e03bac02012 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Sat, 11 Jul 2026 19:08:12 +0100 Subject: [PATCH] fix(telemetry): correct job_count/network-traffic queries, 10s refresh, dashboard layouts Dashboard audit against the live datasource surfaced two broken panel queries and applied UX polish across all 14 dashboards. - node-health "Job Queue Depth": job_count -> jobq_job_count. The JobQueue collector is wrapped in group("jobq") (Application.cpp), so the registered job_count gauge is emitted with the jobq_ prefix; the panel queried the unprefixed name and returned nothing. - network-traffic "Overlay Traffic by Category" + "All Traffic Categories": topk(N, rate({__name__=~".*_bytes_in"}[...])) errors on Mimir ("vector cannot contain metrics with the same labelset") because rate() drops __name__ and the many counters collapse. Replaced with an enumerated label_replace form that re-attaches __name__ per metric, preserving the {{__name__}} legend and per-series display-name overrides. - All 14 dashboards: refresh set to 10s. - peer-quality: each panel full screen width. - validator-health: at most two panels per row (row headers preserved). - docs: telemetry-runbook and 06-implementation-phases updated for the jobq_ prefix and the network-traffic query pattern. Verified end-to-end against a local mainnet xrpld node feeding the local stack: jobq_job_count returns data (old job_count empty), both network-traffic exprs execute (old form reproduces the labelset error), and panels render through the Grafana proxy. All 14 pass validate_dashboards.py. Co-Authored-By: Claude Opus 4.8 (1M context) --- OpenTelemetryPlan/06-implementation-phases.md | 30 +++--- .../grafana/dashboards/consensus-health.json | 3 +- .../grafana/dashboards/fee-market.json | 2 +- .../grafana/dashboards/job-queue.json | 2 +- .../grafana/dashboards/ledger-data-sync.json | 3 +- .../grafana/dashboards/ledger-operations.json | 2 +- .../grafana/dashboards/network-traffic.json | 7 +- .../grafana/dashboards/node-health.json | 4 +- .../dashboards/overlay-traffic-detail.json | 3 +- .../grafana/dashboards/peer-network.json | 2 +- .../grafana/dashboards/peer-quality.json | 32 +++---- .../grafana/dashboards/rpc-pathfinding.json | 2 +- .../grafana/dashboards/rpc-performance.json | 2 +- .../dashboards/transaction-overview.json | 2 +- .../grafana/dashboards/validator-health.json | 92 +++++++++---------- docs/telemetry-runbook.md | 62 +++++++------ 16 files changed, 132 insertions(+), 118 deletions(-) diff --git a/OpenTelemetryPlan/06-implementation-phases.md b/OpenTelemetryPlan/06-implementation-phases.md index dbdb204627..504b9d7363 100644 --- a/OpenTelemetryPlan/06-implementation-phases.md +++ b/OpenTelemetryPlan/06-implementation-phases.md @@ -336,21 +336,21 @@ xrpld has a mature metrics framework (`beast::insight`) that emits StatsD-format ### Metric Inventory -| Category | Group | Type | Count | Key Metrics | -| --------------- | ------------------ | ------------- | ---------- | ------------------------------------------------------ | -| Node State | `State_Accounting` | Gauge | 10 | `*_duration`, `*_transitions` per operating mode | -| Ledger | `LedgerMaster` | Gauge | 2 | `Validated_Ledger_Age`, `Published_Ledger_Age` | -| Ledger Fetch | — | Counter | 1 | `ledger_fetches` | -| Ledger History | `ledger.history` | Counter | 1 | `mismatch` | -| RPC | `rpc` | Counter+Event | 3 | `requests`, `time` (histogram), `size` (histogram) | -| Job Queue | — | Gauge+Event | 1 + 2×N | `job_count`, per-job `{name}` and `{name}_q` | -| Peer Finder | `Peer_Finder` | Gauge | 2 | `Active_Inbound_Peers`, `Active_Outbound_Peers` | -| Overlay | `Overlay` | Gauge | 1 | `Peer_Disconnects` | -| Overlay Traffic | per-category | Gauge | 4×57 = 228 | `Bytes_In/Out`, `Messages_In/Out` per traffic category | -| Pathfinding | — | Event | 2 | `pathfind_fast`, `pathfind_full` (histograms) | -| I/O | — | Event | 1 | `ios_latency` (histogram) | -| Resource Mgr | — | Meter | 2 | `warn`, `drop` (rate counters) | -| Caches | per-cache | Gauge | 2×N | `{cache}.size`, `{cache}.hit_rate` | +| Category | Group | Type | Count | Key Metrics | +| --------------- | ------------------ | ------------- | ---------- | ----------------------------------------------------------------------------------------------------------- | +| Node State | `State_Accounting` | Gauge | 10 | `*_duration`, `*_transitions` per operating mode | +| Ledger | `LedgerMaster` | Gauge | 2 | `Validated_Ledger_Age`, `Published_Ledger_Age` | +| Ledger Fetch | — | Counter | 1 | `ledger_fetches` | +| Ledger History | `ledger.history` | Counter | 1 | `mismatch` | +| RPC | `rpc` | Counter+Event | 3 | `requests`, `time` (histogram), `size` (histogram) | +| Job Queue | `jobq` | Gauge+Event | 1 + 2×N | `job_count`, per-job `{name}` and `{name}_q` (emitted with the `jobq_` group prefix, e.g. `jobq_job_count`) | +| Peer Finder | `Peer_Finder` | Gauge | 2 | `Active_Inbound_Peers`, `Active_Outbound_Peers` | +| Overlay | `Overlay` | Gauge | 1 | `Peer_Disconnects` | +| Overlay Traffic | per-category | Gauge | 4×57 = 228 | `Bytes_In/Out`, `Messages_In/Out` per traffic category | +| Pathfinding | — | Event | 2 | `pathfind_fast`, `pathfind_full` (histograms) | +| I/O | — | Event | 1 | `ios_latency` (histogram) | +| Resource Mgr | — | Meter | 2 | `warn`, `drop` (rate counters) | +| Caches | per-cache | Gauge | 2×N | `{cache}.size`, `{cache}.hit_rate` | **Total**: ~255+ unique metrics (plus dynamic job-type and cache metrics) diff --git a/docker/telemetry/grafana/dashboards/consensus-health.json b/docker/telemetry/grafana/dashboards/consensus-health.json index 5898f7de63..b6bbf10f3e 100644 --- a/docker/telemetry/grafana/dashboards/consensus-health.json +++ b/docker/telemetry/grafana/dashboards/consensus-health.json @@ -1279,5 +1279,6 @@ }, "title": "Consensus Health", "uid": "consensus-health", - "description": "What this shows: Consensus health for XRPL nodes: how reliably and quickly the network agrees each ledger, and where agreement breaks down.\nUse it to: Spot stalled or slow consensus rounds and pinpoint the phase where agreement is failing." + "description": "What this shows: Consensus health for XRPL nodes: how reliably and quickly the network agrees each ledger, and where agreement breaks down.\nUse it to: Spot stalled or slow consensus rounds and pinpoint the phase where agreement is failing.", + "refresh": "10s" } diff --git a/docker/telemetry/grafana/dashboards/fee-market.json b/docker/telemetry/grafana/dashboards/fee-market.json index d57805f8d5..0741888594 100644 --- a/docker/telemetry/grafana/dashboards/fee-market.json +++ b/docker/telemetry/grafana/dashboards/fee-market.json @@ -508,5 +508,5 @@ "title": "Fee Market & TxQ", "uid": "fee-market", "version": 1, - "refresh": "5s" + "refresh": "10s" } diff --git a/docker/telemetry/grafana/dashboards/job-queue.json b/docker/telemetry/grafana/dashboards/job-queue.json index 72adfefa06..3b7a3ec18d 100644 --- a/docker/telemetry/grafana/dashboards/job-queue.json +++ b/docker/telemetry/grafana/dashboards/job-queue.json @@ -594,5 +594,5 @@ "title": "Job Queue Analysis", "uid": "job-queue", "version": 1, - "refresh": "5s" + "refresh": "10s" } diff --git a/docker/telemetry/grafana/dashboards/ledger-data-sync.json b/docker/telemetry/grafana/dashboards/ledger-data-sync.json index 3c62bc81f1..612f60ea7d 100644 --- a/docker/telemetry/grafana/dashboards/ledger-data-sync.json +++ b/docker/telemetry/grafana/dashboards/ledger-data-sync.json @@ -635,5 +635,6 @@ "to": "now" }, "title": "Ledger Data & Sync", - "uid": "ledger-data-sync" + "uid": "ledger-data-sync", + "refresh": "10s" } diff --git a/docker/telemetry/grafana/dashboards/ledger-operations.json b/docker/telemetry/grafana/dashboards/ledger-operations.json index 7e96149c86..03d96f746f 100644 --- a/docker/telemetry/grafana/dashboards/ledger-operations.json +++ b/docker/telemetry/grafana/dashboards/ledger-operations.json @@ -418,6 +418,6 @@ }, "title": "Ledger Operations", "uid": "ledger-operations", - "refresh": "5s", + "refresh": "10s", "description": "What this shows: Ledger construction, validation, and storage activity and timing for this node.\nUse it to: Confirm ledgers are being built, validated, and stored on schedule and find the slow stage when they are not." } diff --git a/docker/telemetry/grafana/dashboards/network-traffic.json b/docker/telemetry/grafana/dashboards/network-traffic.json index 4eb21b72f8..78d319ec8e 100644 --- a/docker/telemetry/grafana/dashboards/network-traffic.json +++ b/docker/telemetry/grafana/dashboards/network-traffic.json @@ -399,7 +399,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "topk(10, rate({service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", __name__=~\".*_bytes_in\", __name__!~\"total_.*\"}[$__rate_interval]))", + "expr": "topk(10, label_replace(sum by (service_instance_id)(rate(transactions_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"transactions_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(proposals_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"proposals_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(validations_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"validations_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(overhead_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"overhead_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(overhead_overlay_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"overhead_overlay_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ping_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ping_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(status_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"status_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(getobject_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"getobject_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(havetxset_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"havetxset_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledgerdata_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledgerdata_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_data_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_data_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_data_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_data_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_data_account_state_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_data_account_state_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_data_account_state_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_data_account_state_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_data_transaction_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_data_transaction_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_data_transaction_set_candidate_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_set_candidate_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_account_state_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_account_state_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_transaction_set_candidate_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_transaction_set_candidate_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_transaction_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_transaction_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(set_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"set_get_bytes_in\",\"\",\"\"))", "legendFormat": "{{__name__}} [{{service_instance_id}}]" } ], @@ -777,7 +777,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "topk(15, rate({__name__=~\".*_bytes_in\", __name__!~\"total_.*\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval]))", + "expr": "topk(15, label_replace(sum by (service_instance_id)(rate(getobject_account_state_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"getobject_account_state_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(getobject_account_state_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"getobject_account_state_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(getobject_cas_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"getobject_cas_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(getobject_cas_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"getobject_cas_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(getobject_fetch_pack_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"getobject_fetch_pack_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(getobject_fetch_pack_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"getobject_fetch_pack_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(getobject_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"getobject_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(getobject_ledger_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"getobject_ledger_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(getobject_ledger_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"getobject_ledger_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(getobject_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"getobject_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(getobject_transaction_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"getobject_transaction_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(getobject_transaction_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"getobject_transaction_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(getobject_transaction_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"getobject_transaction_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(getobject_transaction_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"getobject_transaction_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(getobject_transactions_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"getobject_transactions_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(have_transactions_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"have_transactions_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_account_state_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_account_state_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_account_state_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_account_state_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_data_account_state_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_data_account_state_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_data_account_state_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_data_account_state_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_data_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_data_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_data_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_data_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_data_transaction_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_data_transaction_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_data_transaction_set_candidate_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_set_candidate_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_data_transaction_set_candidate_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_data_transaction_set_candidate_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_transaction_node_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_transaction_node_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_transaction_node_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_transaction_node_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_transaction_set_candidate_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_transaction_set_candidate_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(ledger_transaction_set_candidate_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"ledger_transaction_set_candidate_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(overhead_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"overhead_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(overhead_cluster_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"overhead_cluster_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(overhead_manifest_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"overhead_manifest_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(overhead_overlay_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"overhead_overlay_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(proof_path_request_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"proof_path_request_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(proof_path_response_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"proof_path_response_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(proposals_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"proposals_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(proposals_duplicate_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"proposals_duplicate_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(proposals_untrusted_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"proposals_untrusted_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(replay_delta_request_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"replay_delta_request_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(replay_delta_response_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"replay_delta_response_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(requested_transactions_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"requested_transactions_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(set_get_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"set_get_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(set_share_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"set_share_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(squelch_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"squelch_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(squelch_ignored_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"squelch_ignored_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(squelch_suppressed_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"squelch_suppressed_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(transactions_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"transactions_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(transactions_duplicate_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"transactions_duplicate_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(unknown_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"unknown_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(validations_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"validations_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(validations_duplicate_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"validations_duplicate_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(validations_untrusted_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"validations_untrusted_bytes_in\",\"\",\"\") or label_replace(sum by (service_instance_id)(rate(validator_lists_bytes_in{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval])),\"__name__\",\"validator_lists_bytes_in\",\"\",\"\"))", "legendFormat": "{{__name__}}" } ], @@ -900,5 +900,6 @@ "to": "now" }, "title": "Network Traffic", - "uid": "network-traffic" + "uid": "network-traffic", + "refresh": "10s" } diff --git a/docker/telemetry/grafana/dashboards/node-health.json b/docker/telemetry/grafana/dashboards/node-health.json index de05ddf815..abafdf188f 100644 --- a/docker/telemetry/grafana/dashboards/node-health.json +++ b/docker/telemetry/grafana/dashboards/node-health.json @@ -317,7 +317,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "job_count{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}", + "expr": "jobq_job_count{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}", "legendFormat": "Job Queue Depth [{{service_instance_id}}]" } ], @@ -2592,5 +2592,5 @@ }, "title": "Node Health", "uid": "node-health", - "refresh": "5s" + "refresh": "10s" } diff --git a/docker/telemetry/grafana/dashboards/overlay-traffic-detail.json b/docker/telemetry/grafana/dashboards/overlay-traffic-detail.json index ededa5324e..26c140c82a 100644 --- a/docker/telemetry/grafana/dashboards/overlay-traffic-detail.json +++ b/docker/telemetry/grafana/dashboards/overlay-traffic-detail.json @@ -692,5 +692,6 @@ "to": "now" }, "title": "Overlay Traffic Detail", - "uid": "overlay-traffic-detail" + "uid": "overlay-traffic-detail", + "refresh": "10s" } diff --git a/docker/telemetry/grafana/dashboards/peer-network.json b/docker/telemetry/grafana/dashboards/peer-network.json index 2f323501d5..8bef20ddc3 100644 --- a/docker/telemetry/grafana/dashboards/peer-network.json +++ b/docker/telemetry/grafana/dashboards/peer-network.json @@ -397,5 +397,5 @@ }, "title": "Peer Network", "uid": "peer-network", - "refresh": "5s" + "refresh": "10s" } diff --git a/docker/telemetry/grafana/dashboards/peer-quality.json b/docker/telemetry/grafana/dashboards/peer-quality.json index 9c41880109..0367e98868 100644 --- a/docker/telemetry/grafana/dashboards/peer-quality.json +++ b/docker/telemetry/grafana/dashboards/peer-quality.json @@ -15,7 +15,7 @@ "type": "timeseries", "gridPos": { "h": 8, - "w": 12, + "w": 24, "x": 0, "y": 0 }, @@ -77,9 +77,9 @@ "type": "stat", "gridPos": { "h": 8, - "w": 6, - "x": 12, - "y": 0 + "w": 24, + "x": 0, + "y": 8 }, "options": { "tooltip": { @@ -127,9 +127,9 @@ "type": "stat", "gridPos": { "h": 8, - "w": 6, - "x": 18, - "y": 0 + "w": 24, + "x": 0, + "y": 16 }, "options": { "tooltip": { @@ -179,9 +179,9 @@ "type": "stat", "gridPos": { "h": 8, - "w": 6, + "w": 24, "x": 0, - "y": 8 + "y": 24 }, "options": { "tooltip": { @@ -245,9 +245,9 @@ "type": "timeseries", "gridPos": { "h": 8, - "w": 10, - "x": 6, - "y": 8 + "w": 24, + "x": 0, + "y": 32 }, "options": { "tooltip": { @@ -291,9 +291,9 @@ "type": "bargauge", "gridPos": { "h": 8, - "w": 8, - "x": 16, - "y": 8 + "w": 24, + "x": 0, + "y": 40 }, "options": { "orientation": "horizontal", @@ -474,5 +474,5 @@ }, "title": "Peer Quality", "uid": "peer-quality", - "refresh": "5s" + "refresh": "10s" } diff --git a/docker/telemetry/grafana/dashboards/rpc-pathfinding.json b/docker/telemetry/grafana/dashboards/rpc-pathfinding.json index 1746d82d7c..3599dd545f 100644 --- a/docker/telemetry/grafana/dashboards/rpc-pathfinding.json +++ b/docker/telemetry/grafana/dashboards/rpc-pathfinding.json @@ -690,5 +690,5 @@ }, "title": "RPC & Pathfinding", "uid": "rpc-pathfinding", - "refresh": "5s" + "refresh": "10s" } diff --git a/docker/telemetry/grafana/dashboards/rpc-performance.json b/docker/telemetry/grafana/dashboards/rpc-performance.json index 400953d831..da7ff67b10 100644 --- a/docker/telemetry/grafana/dashboards/rpc-performance.json +++ b/docker/telemetry/grafana/dashboards/rpc-performance.json @@ -1011,6 +1011,6 @@ }, "title": "RPC Performance", "uid": "rpc-performance", - "refresh": "5s", + "refresh": "10s", "description": "What this shows: Per-command and per-method RPC performance: call rates, error rates, and latency distributions.\nUse it to: Identify slow or failing RPC commands and track client-facing request latency." } diff --git a/docker/telemetry/grafana/dashboards/transaction-overview.json b/docker/telemetry/grafana/dashboards/transaction-overview.json index 36e0804c1c..826f8bf568 100644 --- a/docker/telemetry/grafana/dashboards/transaction-overview.json +++ b/docker/telemetry/grafana/dashboards/transaction-overview.json @@ -1006,6 +1006,6 @@ }, "title": "Transaction Overview", "uid": "transaction-overview", - "refresh": "5s", + "refresh": "10s", "description": "What this shows: Transaction flow through this node: receipt, processing, results, per-stage timing, and queue behavior.\nUse it to: Trace transactions from arrival to ledger, and locate stalls in processing or the queue." } diff --git a/docker/telemetry/grafana/dashboards/validator-health.json b/docker/telemetry/grafana/dashboards/validator-health.json index d04d2ef794..81393fd425 100644 --- a/docker/telemetry/grafana/dashboards/validator-health.json +++ b/docker/telemetry/grafana/dashboards/validator-health.json @@ -27,7 +27,7 @@ "type": "stat", "gridPos": { "h": 8, - "w": 6, + "w": 12, "x": 0, "y": 1 }, @@ -79,8 +79,8 @@ "type": "stat", "gridPos": { "h": 8, - "w": 6, - "x": 6, + "w": 12, + "x": 12, "y": 1 }, "options": { @@ -131,9 +131,9 @@ "type": "bargauge", "gridPos": { "h": 8, - "w": 6, - "x": 12, - "y": 1 + "w": 12, + "x": 0, + "y": 9 }, "options": { "orientation": "horizontal", @@ -199,9 +199,9 @@ "type": "bargauge", "gridPos": { "h": 8, - "w": 6, - "x": 18, - "y": 1 + "w": 12, + "x": 12, + "y": 9 }, "options": { "orientation": "horizontal", @@ -268,7 +268,7 @@ "h": 1, "w": 24, "x": 0, - "y": 9 + "y": 17 }, "collapsed": false, "panels": [] @@ -279,9 +279,9 @@ "type": "stat", "gridPos": { "h": 8, - "w": 6, + "w": 12, "x": 0, - "y": 10 + "y": 18 }, "options": { "tooltip": { @@ -329,9 +329,9 @@ "type": "stat", "gridPos": { "h": 8, - "w": 6, - "x": 6, - "y": 10 + "w": 12, + "x": 12, + "y": 18 }, "options": { "tooltip": { @@ -363,9 +363,9 @@ "type": "stat", "gridPos": { "h": 8, - "w": 6, - "x": 12, - "y": 10 + "w": 12, + "x": 0, + "y": 26 }, "options": { "tooltip": { @@ -429,9 +429,9 @@ "type": "stat", "gridPos": { "h": 8, - "w": 6, - "x": 18, - "y": 10 + "w": 12, + "x": 12, + "y": 26 }, "options": { "tooltip": { @@ -479,9 +479,9 @@ "type": "stat", "gridPos": { "h": 8, - "w": 6, + "w": 12, "x": 0, - "y": 18 + "y": 34 }, "options": { "tooltip": { @@ -545,9 +545,9 @@ "type": "timeseries", "gridPos": { "h": 8, - "w": 18, - "x": 6, - "y": 18 + "w": 12, + "x": 12, + "y": 34 }, "options": { "tooltip": { @@ -616,7 +616,7 @@ "h": 1, "w": 24, "x": 0, - "y": 26 + "y": 42 }, "collapsed": false, "panels": [] @@ -627,9 +627,9 @@ "type": "stat", "gridPos": { "h": 8, - "w": 6, + "w": 12, "x": 0, - "y": 27 + "y": 43 }, "options": { "tooltip": { @@ -661,9 +661,9 @@ "type": "timeseries", "gridPos": { "h": 8, - "w": 18, - "x": 6, - "y": 27 + "w": 12, + "x": 12, + "y": 43 }, "options": { "tooltip": { @@ -707,9 +707,9 @@ "type": "stat", "gridPos": { "h": 8, - "w": 8, + "w": 12, "x": 0, - "y": 35 + "y": 51 }, "options": { "tooltip": { @@ -741,9 +741,9 @@ "type": "stat", "gridPos": { "h": 8, - "w": 8, - "x": 8, - "y": 35 + "w": 12, + "x": 12, + "y": 51 }, "options": { "tooltip": { @@ -791,9 +791,9 @@ "type": "stat", "gridPos": { "h": 8, - "w": 8, - "x": 16, - "y": 35 + "w": 12, + "x": 0, + "y": 59 }, "options": { "tooltip": { @@ -842,8 +842,8 @@ "gridPos": { "h": 8, "w": 12, - "x": 0, - "y": 43 + "x": 12, + "y": 59 }, "options": { "tooltip": { @@ -893,9 +893,9 @@ "type": "timeseries", "gridPos": { "h": 8, - "w": 12, - "x": 12, - "y": 43 + "w": 24, + "x": 0, + "y": 67 }, "options": { "tooltip": { @@ -1041,5 +1041,5 @@ }, "title": "Validator Health", "uid": "validator-health", - "refresh": "5s" + "refresh": "10s" } diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index a6f4b0f722..cfb57407bb 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -738,18 +738,18 @@ The `OTelCollector` implementation exports metrics via OTLP/HTTP to the same OTe #### Gauges -| Prometheus Metric | Source | Description | -| ------------------------------------- | ------------------------- | -------------------------------------------------------------------------- | -| `ledgermaster_validated_ledger_age` | LedgerMaster.h:373 | Age of validated ledger (seconds) | -| `ledgermaster_published_ledger_age` | LedgerMaster.h:374 | Age of published ledger (seconds) | -| `state_accounting_{mode}_duration` | NetworkOPs.cpp:774 | Time in each operating mode (Disconnected/Connected/Syncing/Tracking/Full) | -| `state_accounting_{mode}_transitions` | NetworkOPs.cpp:780 | Transition count per mode | -| `peer_finder_active_inbound_peers` | PeerfinderManager.cpp:214 | Active inbound peer connections | -| `peer_finder_active_outbound_peers` | PeerfinderManager.cpp:215 | Active outbound peer connections | -| `overlay_peer_disconnects` | OverlayImpl.h:557 | Peer disconnect count | -| `job_count` | JobQueue.cpp:26 | Current job queue depth | -| `{category}_bytes_in/out` | OverlayImpl.h:535 | Overlay traffic bytes per category (57 categories) | -| `{category}_messages_in/out` | OverlayImpl.h:535 | Overlay traffic messages per category | +| Prometheus Metric | Source | Description | +| ------------------------------------- | ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `ledgermaster_validated_ledger_age` | LedgerMaster.h:373 | Age of validated ledger (seconds) | +| `ledgermaster_published_ledger_age` | LedgerMaster.h:374 | Age of published ledger (seconds) | +| `state_accounting_{mode}_duration` | NetworkOPs.cpp:774 | Time in each operating mode (Disconnected/Connected/Syncing/Tracking/Full) | +| `state_accounting_{mode}_transitions` | NetworkOPs.cpp:780 | Transition count per mode | +| `peer_finder_active_inbound_peers` | PeerfinderManager.cpp:214 | Active inbound peer connections | +| `peer_finder_active_outbound_peers` | PeerfinderManager.cpp:215 | Active outbound peer connections | +| `overlay_peer_disconnects` | OverlayImpl.h:557 | Peer disconnect count | +| `jobq_job_count` | JobQueue.cpp:26 | Current job queue depth (emitted as `jobq_job_count`: the JobQueue collector is wrapped in `group("jobq")`, so the registered `job_count` gauge gains the `jobq_` prefix) | +| `{category}_bytes_in/out` | OverlayImpl.h:535 | Overlay traffic bytes per category (57 categories) | +| `{category}_messages_in/out` | OverlayImpl.h:535 | Overlay traffic messages per category | #### OTel MetricsRegistry Gauges @@ -960,7 +960,7 @@ Requires `trace_peer=1` in the `[telemetry]` config section. | Operating Mode (Time Share) | timeseries | `rate(state_accounting_X_duration) / sum(rate(all modes))` | — | | Operating Mode Transitions | timeseries | `state_accounting_*_transitions` | — | | I/O Latency | timeseries | `histogram_quantile(0.95, ios_latency_bucket)` | — | -| Job Queue Depth | timeseries | `job_count` | — | +| Job Queue Depth | timeseries | `jobq_job_count` | — | | Ledger Fetch Rate | stat | `rate(ledger_fetches[5m])` | — | | Ledger History Mismatches | stat | `rate(ledger_history_mismatch[5m])` | — | | Key Jobs Execution Time | timeseries | `acceptledger{quantile="$quantile"}` (+ 10 more key jobs) | `quantile` | @@ -982,18 +982,28 @@ Requires `trace_peer=1` in the `[telemetry]` config section. ### Network Traffic -- System Metrics (`network-traffic`) -| Panel | Type | PromQL | Labels Used | -| ------------------------------------ | ---------- | ------------------------------------------------------ | ----------- | -| Active Peers | timeseries | `peer_finder_active_*_peers` | — | -| Peer Disconnects | timeseries | `increase(overlay_peer_disconnects[$__rate_interval])` | — | -| Total Network Bytes | timeseries | `rate(total_bytes_in/out[$__rate_interval])` | — | -| Total Network Messages | timeseries | `rate(total_messages_in/out[$__rate_interval])` | — | -| Transaction Traffic | timeseries | `rate(transactions_messages_in/out[$__rate_interval])` | — | -| Proposal Traffic | timeseries | `rate(proposals_messages_in/out[$__rate_interval])` | — | -| Validation Traffic | timeseries | `rate(validations_messages_in/out[$__rate_interval])` | — | -| Traffic by Category | bargauge | `topk(10, rate(*_bytes_in[$__rate_interval]))` | — | -| Duplicate Traffic (Wasted Bandwidth) | timeseries | `rate(*_duplicate_bytes_in/out[$__rate_interval])` | — | -| All Traffic Categories (Detail) | timeseries | `topk(15, rate(*_bytes_in[$__rate_interval]))` | — | +| Panel | Type | PromQL | Labels Used | +| ------------------------------------ | ---------- | -------------------------------------------------------------------------------------------------------------------------- | ----------- | +| Active Peers | timeseries | `peer_finder_active_*_peers` | — | +| Peer Disconnects | timeseries | `increase(overlay_peer_disconnects[$__rate_interval])` | — | +| Total Network Bytes | timeseries | `rate(total_bytes_in/out[$__rate_interval])` | — | +| Total Network Messages | timeseries | `rate(total_messages_in/out[$__rate_interval])` | — | +| Transaction Traffic | timeseries | `rate(transactions_messages_in/out[$__rate_interval])` | — | +| Proposal Traffic | timeseries | `rate(proposals_messages_in/out[$__rate_interval])` | — | +| Validation Traffic | timeseries | `rate(validations_messages_in/out[$__rate_interval])` | — | +| Traffic by Category | bargauge | `topk(10, label_replace(sum by (service_instance_id)(rate([$__rate_interval])),"__name__","","","") or …)` | — | +| Duplicate Traffic (Wasted Bandwidth) | timeseries | `rate(*_duplicate_bytes_in/out[$__rate_interval])` | — | +| All Traffic Categories (Detail) | timeseries | `topk(15, label_replace(sum by (service_instance_id)(rate([$__rate_interval])),"__name__","","","") or …)` | — | + +> **Why the per-category panels enumerate each metric.** A bare +> `rate({__name__=~".*_bytes_in"}[…])` fails on Mimir/Cloud with _"vector +> cannot contain metrics with the same labelset"_: `rate()` drops the +> `__name__` label, so the many matched counters collapse to identical +> labelsets. Wrapping in `sum by (__name__, …)` does **not** help (the inner +> vector is rejected before the outer `sum`). The working form enumerates each +> `*_bytes_in` metric and re-attaches its name with `label_replace(..., +"__name__", "", "", "")`, so the existing `{{__name__}}` legend and +> the per-series display-name overrides keep working. ### RPC & Pathfinding -- System Metrics (`rpc-pathfinding`) @@ -1244,7 +1254,7 @@ count_over_time({service_name="xrpld"} |= "trace_id=" [5m]) 2. Verify `server=otel` in the `[insight]` config section 3. Verify the endpoint in `[insight]` points to the OTLP/HTTP port (default: `http://localhost:4318/v1/metrics`) 4. Check that the `otlp` receiver is in the metrics pipeline receivers in `otel-collector-config.yaml` -5. Query Prometheus directly: `curl 'http://localhost:9090/api/v1/query?query=job_count'` +5. Query Prometheus directly: `curl 'http://localhost:9090/api/v1/query?query=jobq_job_count'` ### Server info gauge shows server_state=0