ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-21 14:20:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(telemetry): surface stalled consensus rounds with a p99 series

Browse Source 
The "Consensus Round Duration" panel plotted only p50 and p95. At the
devnet rate of ~19 rounds/min, a single stalled round is one sample in
several hundred, so p95 over a rate interval stays near the normal 2s
close time and the stall is invisible.

Measured on devnet-otel-usw2-02 over a 40 min window: 761 rounds, of
which exactly one fell in the 7500-10000ms bucket while the mean never
exceeded 2270ms. The histogram already records the outlier; only the
panel could not show it.

Add a p99 series alongside p50/p95 and retitle the panel accordingly.
No metric or collector change: the existing
consensus_round_duration_ms_bucket histogram already has 20 boundaries
up to 120000ms.
This commit is contained in:
Pratik Mankawde
2026-08-12 16:52:08 +01:00
parent 6bbb4cfcf5
commit 277f7c9655
1 changed files with 9 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
docker/telemetry/grafana/dashboards/ledger-sync-health.json
View File

@@ -3234,9 +3234,17 @@
},
"expr": "label_replace(label_join(label_replace(histogram_quantile(0.95, sum by (le, service_instance_id, xrpl_branch, xrpl_work_item) (rate(consensus_round_duration_ms_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"P95 round duration\", \"__name__\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")",
"refId": "B"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "label_replace(label_join(label_replace(histogram_quantile(0.99, sum by (le, service_instance_id, xrpl_branch, xrpl_work_item) (rate(consensus_round_duration_ms_bucket{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"series\", \"P99 round duration\", \"__name__\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")",
"refId": "C"
}
],
"title": "Consensus Round Duration (p50/p95)",
"title": "Consensus Round Duration (p50/p95/p99)",
"type": "timeseries"
},
{