fix(telemetry): apply the panel-audit findings on ledger-sync-health

Browse Source

A 54-panel audit across 11 dimensions, each finding adversarially re-verified
against live data, returned 38 confirmed defects. This fixes them. Most were
introduced by the recent rate-to-count conversion itself.

1. increase() was the wrong function for these counters. A counter that only
   moves at startup is born at its final value inside the window and never
   rises, so increase() reports 0. Measured: dns_resolve_total reads 4 but
   round(increase(...[$__range])) returned 0 -- the panel lost the signal
   entirely. increase() also drops whatever accrued before the window opened,
   which under-reported the rest (overlay_connect_total 110 against a true 130,
   unl_fetch_total 10 against 14, peer_disconnect_total 7 against 8).
   All 18 count panels now use last_over_time(...[$__range]), which on a
   fresh-node board is the cumulative total since the process started -- exactly
   what "count" means here.

   EXCEPT panel 49. span_calls_total comes from the collector's spanmetrics
   connector, which is collector-side state and does NOT reset when xrpld
   restarts, so last_over_time would report the collector's lifetime across every
   run: it read 2624 header completions where run C actually had 297. That panel
   keeps round(increase(...)) and now reports 297/278/212/7, matching the
   analysis. The distinction is process-level counter vs collector-side counter,
   and it decides which function is correct.

2. Descriptions still described rates after the conversion, over three passes of
   wording (Reading it / Healthy range / Watch for blocks, "Rate of", "per
   second", "/s", "a rising rate"). 11 panels corrected; the two surviving uses
   of "rate" are legitimate (a cache hit-rate reference, and panel 49 explaining
   why a count reads better than a rate).

3. Ten descriptions pointed at panel titles that no longer exist, because the
   conversion renamed the panels they cross-referenced. Two others named panels
   that never existed on this board at all: "Total Jobs Queued" (now Worker Pool
   Capacity & Total Backlog, panel 27) and "Fetch-Pack Peer Starvation" (now
   Peers Able to Serve Needed Sequence, panel 28).

4. Panels 18 and 23 applied $acquire_metric on top of a hard-coded metric
   selector, so the two ANDed: any selection outside the panel's own values gave
   an empty graph and All was the only usable state. The redundant template
   selector is gone; the panel's metric pair is its identity.

5. Panel 23 drew two series with different ranges (received_data_depth 0-20,
   in_flight 13-49) under one yellow threshold at 16, so in_flight was
   permanently yellow. The threshold is now scoped to received_data_depth.

6. The "Spans & traces" row sat at y=248, the same y as panels 38/39, so Grafana
   folded the Back-fill panels into the wrong row. Moved to y=296, below the last
   back-fill panel. Rows are now strictly ascending with no collision.

Verified against Grafana Cloud Prometheus: all 66 panel queries parse, 0 errors,
57 returning data (up from 56 -- panel 3 was one of the ones increase() had
silenced). validate_dashboards and check_otel_naming both pass.

Not verified: no PNG renders this round; the local Grafana and Prometheus are
down, so every check ran against Cloud data via the datasource proxy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

...

This commit is contained in:

Pratik Mankawde

2026-07-28 13:46:52 +01:00

parent 36e8cf1fe4

commit 7551d57db9

1 changed files with 629 additions and 601 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

1230

docker/telemetry/grafana/dashboards/ledger-sync-health.json

File diff suppressed because it is too large Load Diff