Pratik Mankawde 8b14176609 feat(telemetry): add sync bottleneck discrimination dashboard row ...

A stalled ledger sync has two distinct causes that look identical from the
job queue: in both, the ledgerData lane sits at its concurrency limit of 3
with work waiting. Lane occupancy therefore discriminates nothing, and the
dashboard had no panel that separated the two.

Adds a Sync Bottleneck Discrimination row to Ledger Data & Sync with seven
panels built on the nodestore_state write-path and acquire counters:

- NodeStore Read Latency, lifetime and windowed, on a log axis with
  threshold lines at 10 and 100 us. This is the discriminator: fast reads
  mean the cost is on the write side, slow reads with a high hit ratio
  mean every access is paying disk latency.
- NuDB Writer Queue Depth, the confirming half. NuDB takes one global
  mutex per insert, so depth above 1 is queueing at that mutex.
- NuDB Insert Time, mean against true max, which quantifies how much of
  each insert is wait rather than work.
- Acquire Deferrals vs Timeouts on one panel, because the livelock
  fingerprint is deferrals rising while timeouts stay flat, and neither
  counter shows that alone.
- Acquisition Progress and Discarded Acquire Work, the outcome side: no
  completions while the lane is busy means acquisitions resolve neither
  way.
- NodeStore Read Latency Distribution split by fetch_type and found, from
  the nodestore_read_us histogram, since a mean cannot tell a uniformly
  warm store from a mostly-warm one with a cold tail.

Every expression carries the existing service_instance_id and tier filters.
The histogram introduces two new label dimensions, so fetch_type and found
template variables are added to match.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-07-28 12:20:47 +01:00

..

telemetry

feat(telemetry): add sync bottleneck discrimination dashboard row

2026-07-28 12:20:47 +01:00