fix(telemetry): show discrete sync events as counts, not per-second rates

Browse Source

Seventeen panels on ledger-sync-health drew low-frequency discrete counters as
timeseries with an ops/s unit. The rate is arithmetically right but unreadable:
"0.1394 ops/s" is 278 abandoned tree phases over 33 minutes, and no reader can
recover the 278. rate() also extrapolates, so whole events rendered as
fractions -- peer disconnects showed 8.008 for 8 actual disconnects.

Measured every candidate against live Prometheus over a 33-minute mainnet sync
before converting; none exceeded 0.48 events/s, so all are count-shaped:

  serve_refused_total            955 events   0.4785/s
  ledger_quorum_shortfall_total  670          0.3354/s
  sync_acquire_source_total      511          0.2559/s
  overlay_connect_total          130          0.0652/s
  sync_acquire_no_progress_total 109          0.0545/s
  unl_fetch_total                 12          0.0061/s
  peer_disconnect_total            8          0.0040/s

Each becomes a bargauge over round(increase(...[$__range])) with unit short and
decimals 0, matching the existing Mode Transitions panel. Legends follow
instructions.md OTel rule 7 -- "MetricName [labels]" -- and keep node identity
via xrpl_ident, so a nine-node view no longer collapses to one bar. Panel 49
also gained node identity in its aggregation, which it was missing entirely.

Label values stay as emitted (upgrade_fail, not "Upgrade Fail"). They are wire
identifiers: rewriting them in the legend would hide what the metric reports and
break silently when a new value appears. The Title Case sits in the metric name.

The five zero-valued panels were checked rather than assumed dead --
ledger_replay_*, sweep_malloc_trim_* and rotation_copy_node_restore_total all
have real emit sites and are feature-gated off in this configuration.

Verified against Grafana Cloud Prometheus: 0 parse errors across all panel
queries, whole-number results throughout.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

...

This commit is contained in:

Pratik Mankawde

2026-07-27 21:22:33 +01:00

parent b0b02a5b10

commit d13f2fe6b4

1 changed files with 1668 additions and 2145 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

3813

docker/telemetry/grafana/dashboards/ledger-sync-health.json

File diff suppressed because it is too large Load Diff