fix(telemetry): resolve microsecond latencies below 100us

The microsecond ladder's first edge was 100us, which sat ABOVE the mass of
every instrument using it. Measured on devnet: 99.3% of job_queued_us
samples, 92.5% of job_running_us and 90.4% of getobject_lookup_us fell in
that first bucket. histogram_quantile then interpolated inside bucket 0 and
returned `quantile / fraction_in_bucket_0 x first_edge` -- p75/p95/p99 of
job_queued_us read 75.52/95.66/99.69us against a prediction of
75.53/95.67/99.70. Three-decimal agreement: those panels were reporting
arithmetic on the bucket edge, not latency.

The fix was already half-written. kSubMillisecondBoundaries had been parked
in MetricsRegistry.cpp as [[maybe_unused]] with a comment noting exactly this
problem for nodestore reads. Its edges are now folded into kMicrosecondBuckets
rather than deleted, so the parked intent is carried forward: 1..1000us
resolution where the mass is, upper edges unchanged so multi-second stalls
stay measurable.

Also moves the GetObject count and charge ladders into HistogramBuckets.h, so
all five ladders have one owner and one set of invariant tests (29 now).

Adds check_bucket_parity.py, wired into the existing OTel naming workflow.
The C++ millisecond ladder and the collector's spanmetrics ladder are
specified to agree over their shared range; they were identical when shipped,
then the collector side alone was extended and nothing noticed for eleven
phases. The check asserts containment rather than equality, because jobs
outlive spans -- jobq_updatepaths averages ~60s, which no span approaches, so
demanding equality would force a ceiling that censors it. Verified it rejects
a missing collector edge, a bogus in-range edge, and a return to the 5s
ceiling.

ledger-data-sync's "Job Queue Wait p95 By Type" moves off the beast
jobq_*_q_milliseconds pair onto job_queued_us filtered by job_type. Those
beast metrics are ms-quantised at the source (Event rounds up to a whole
millisecond), so 94-100% of their samples sat in the first bucket and no
ladder change could fix them. Note the label values are camelCase
(job_type="ledgerData"), not the lowercase metric-name fragments.

Both histogram-fed alert thresholds re-validated and left unchanged, with the
measured basis recorded so neither gets tuned against the old artefact: only
0.0022% of job_queued_us samples exceed the 1s threshold, and every edge
bracketing the 1000ms ios_latency threshold survived the ladder change.

Docs: the rpc_size "known issue -- tracked separately" notes in the runbook
and 09-data-collection-reference are now resolved notes, the stale 10-edge
span_duration bucket list is corrected to the collector's real 20, and the
runbook gains a "Reading A Histogram Percentile" section covering both
saturation traps and the expected discontinuity after a ladder change.
This commit is contained in:
Pratik Mankawde
2026-08-21 12:46:56 +01:00
parent 7735d725fb
commit 6e2b2da772
10 changed files with 354 additions and 97 deletions

View File

@@ -2262,22 +2262,68 @@ Requires `trace_peer=1` in the `[telemetry]` config section.
| ------------------------- | ---------- | ------------------------------------------------------------- | ----------- |
| RPC Request Rate | stat | `rate(rpc_requests[5m])` | — |
| RPC Response Time | timeseries | `histogram_quantile(0.95, rpc_time_milliseconds_bucket)` | — |
| RPC Response Size | timeseries | `histogram_quantile(0.95, rpc_size_milliseconds_bucket)` | — |
| RPC Response Size | timeseries | `histogram_quantile(0.95, rpc_size_bytes_bucket)` | — |
| RPC Response Time Heatmap | heatmap | `rpc_time_milliseconds_bucket` | — |
| Pathfinding Fast Duration | timeseries | `histogram_quantile(0.95, pathfind_fast_milliseconds_bucket)` | — |
| Pathfinding Full Duration | timeseries | `histogram_quantile(0.95, pathfind_full_milliseconds_bucket)` | — |
| Resource Warnings Rate | stat | `rate(warn_total[$__rate_interval])` | — |
| Resource Drops Rate | stat | `rate(drop_total[$__rate_interval])` | — |
> **The `_milliseconds` suffix comes from the exporter, not from xrpld.** These
> histograms are created with unit `"ms"`
> ([OTelCollector.cpp:615](../src/libxrpl/beast/insight/OTelCollector.cpp#L615)),
> so the Prometheus exporter appends the unit to the family name — `rpc_time`
> becomes `rpc_time_milliseconds_bucket`. Querying the bare `rpc_time_bucket`,
> **The unit suffix comes from the exporter, not from xrpld.** Each histogram
> declares a unit, and the Prometheus exporter appends the unit's name to the
> family name — a `ms` instrument like `rpc_time` becomes
> `rpc_time_milliseconds_bucket`. Querying the bare `rpc_time_bucket`,
> `ios_latency_bucket` or `pathfind_fast_bucket` returns no data and no error.
> **Known issue**: `rpc_size` counts bytes but shares the same `"ms"` histogram
> constructor, so it is exported as `rpc_size_milliseconds_bucket` — the suffix
> is wrong, the name is nonetheless the one to query.
>
> The unit an `Event` declares also selects its bucket ladder, because the
> histogram views match on unit. `rpc_size` measures bytes, so it declares
> `Unit::Bytes` and exports as **`rpc_size_bytes_bucket`** on the byte ladder.
> It used to share the `ms` constructor and export as
> `rpc_size_milliseconds_bucket` on a latency ladder — if you find that name in
> an old query or bookmark, it no longer exists.
#### Reading A Histogram Percentile
Two failure modes make a percentile panel lie, and neither looks like an error —
both produce a believable number. Check for them before trusting any p95/p99.
**Saturated at the top.** If the quantile falls in the `+Inf` bucket, Prometheus
returns the **second-highest** bucket edge, not `+Inf`. A panel pinned to a round
number that happens to equal the ladder's top edge is the signature. Confirm by
comparing the top finite bucket against the total:
```promql
1 - (
sum(last_over_time(<metric>_bucket{le="<top edge>"}[15m]))
/ sum(last_over_time(<metric>_bucket{le="+Inf"}[15m]))
)
```
A non-trivial result means samples are being censored and the percentile is a
lower bound, not a measurement.
**Saturated at the bottom.** If nearly every sample lands in the first bucket,
`histogram_quantile` interpolates _inside_ it and returns
`quantile / fraction_in_bucket_0 × first_edge`. The signature is a p75/p95/p99
that sit in near-constant proportion to each other and to the first edge — for
example 75.5 / 95.7 / 99.7 against a 100 µs floor. Confirm with:
```promql
sum(last_over_time(<metric>_bucket{le="<first edge>"}[15m]))
/ sum(last_over_time(<metric>_bucket{le="+Inf"}[15m]))
```
Anything close to 1 means the panel is reporting arithmetic on the bucket edge.
**After a ladder change, expect a discontinuity.** Existing series keep their old
`le` values, so a percentile panel shows a step at the restart that introduced
new edges. That break is the ladder changing, not an incident.
Bucket edges for the native instruments live in one place —
[`include/xrpl/telemetry/HistogramBuckets.h`](../include/xrpl/telemetry/HistogramBuckets.h).
The millisecond ladder is required to contain every representable edge of the
collector's spanmetrics ladder; `.github/scripts/telemetry/check_bucket_parity.py`
enforces that in CI, because the two silently drifted once already.
### Span → Metric → Dashboard Summary