mirror of
https://github.com/XRPLF/rippled.git
synced 2026-08-21 22:30:57 +00:00
fix(telemetry): resolve microsecond latencies below 100us
The microsecond ladder's first edge was 100us, which sat ABOVE the mass of every instrument using it. Measured on devnet: 99.3% of job_queued_us samples, 92.5% of job_running_us and 90.4% of getobject_lookup_us fell in that first bucket. histogram_quantile then interpolated inside bucket 0 and returned `quantile / fraction_in_bucket_0 x first_edge` -- p75/p95/p99 of job_queued_us read 75.52/95.66/99.69us against a prediction of 75.53/95.67/99.70. Three-decimal agreement: those panels were reporting arithmetic on the bucket edge, not latency. The fix was already half-written. kSubMillisecondBoundaries had been parked in MetricsRegistry.cpp as [[maybe_unused]] with a comment noting exactly this problem for nodestore reads. Its edges are now folded into kMicrosecondBuckets rather than deleted, so the parked intent is carried forward: 1..1000us resolution where the mass is, upper edges unchanged so multi-second stalls stay measurable. Also moves the GetObject count and charge ladders into HistogramBuckets.h, so all five ladders have one owner and one set of invariant tests (29 now). Adds check_bucket_parity.py, wired into the existing OTel naming workflow. The C++ millisecond ladder and the collector's spanmetrics ladder are specified to agree over their shared range; they were identical when shipped, then the collector side alone was extended and nothing noticed for eleven phases. The check asserts containment rather than equality, because jobs outlive spans -- jobq_updatepaths averages ~60s, which no span approaches, so demanding equality would force a ceiling that censors it. Verified it rejects a missing collector edge, a bogus in-range edge, and a return to the 5s ceiling. ledger-data-sync's "Job Queue Wait p95 By Type" moves off the beast jobq_*_q_milliseconds pair onto job_queued_us filtered by job_type. Those beast metrics are ms-quantised at the source (Event rounds up to a whole millisecond), so 94-100% of their samples sat in the first bucket and no ladder change could fix them. Note the label values are camelCase (job_type="ledgerData"), not the lowercase metric-name fragments. Both histogram-fed alert thresholds re-validated and left unchanged, with the measured basis recorded so neither gets tuned against the old artefact: only 0.0022% of job_queued_us samples exceed the 1s threshold, and every edge bracketing the 1000ms ios_latency threshold survived the ladder change. Docs: the rpc_size "known issue -- tracked separately" notes in the runbook and 09-data-collection-reference are now resolved notes, the stale 10-edge span_duration bucket list is corrected to the collector's real 20, and the runbook gains a "Reading A Histogram Percentile" section covering both saturation traps and the expected discontinuity after a ladder change.
This commit is contained in:
@@ -531,12 +531,12 @@ a destructor must not depend on still existing. A query that only groups by
|
||||
|
||||
The OTel Collector's SpanMetrics connector automatically generates RED (Rate, Errors, Duration) metrics from every span. No custom metrics code in xrpld is needed.
|
||||
|
||||
| Prometheus Metric | Type | Description |
|
||||
| ----------------------------------- | --------- | ------------------------------------------------------------------------------ |
|
||||
| `span_calls_total` | Counter | Total span invocations |
|
||||
| `span_duration_milliseconds_bucket` | Histogram | Latency distribution (buckets: 1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000 ms) |
|
||||
| `span_duration_milliseconds_count` | Histogram | Observation count |
|
||||
| `span_duration_milliseconds_sum` | Histogram | Cumulative latency |
|
||||
| Prometheus Metric | Type | Description |
|
||||
| ----------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `span_calls_total` | Counter | Total span invocations |
|
||||
| `span_duration_milliseconds_bucket` | Histogram | Latency distribution. Buckets come from the collector's spanmetrics config: 0.01, 0.05, 0.1, 0.25, 0.5, 1, 5, 10, 25, 50, 100, 250, 500 ms then 1, 2, 3, 4, 5, 10, 30 s. The sub-millisecond edges exist because most xrpld spans are far below 1 ms; without them every p95/p99 pinned to a constant 0.95 ms |
|
||||
| `span_duration_milliseconds_count` | Histogram | Observation count |
|
||||
| `span_duration_milliseconds_sum` | Histogram | Cumulative latency |
|
||||
|
||||
**Standard labels on every metric**: `span_name`, `status_code`, `service_name`, `span_kind`
|
||||
|
||||
@@ -684,13 +684,14 @@ prefix=xrpld
|
||||
|
||||
Quantiles collected: 0th, 50th, 90th, 95th, 99th, 100th percentile.
|
||||
|
||||
\* **`rpc_size` instrument mismatch (known issue):** response size in bytes is
|
||||
recorded through the millisecond-scaled event histogram (`makeEvent`), so it is
|
||||
exported as `rpc_size_milliseconds_bucket` with time-scaled boundaries that top
|
||||
out at 5000. Byte values above ~5 KB saturate in the last bucket, so the
|
||||
percentiles are not true byte sizes. The _RPC & Pathfinding_ panel is flagged
|
||||
accordingly. A dedicated byte-unit histogram is needed to fix this; tracked
|
||||
separately.
|
||||
\* **`rpc_size` now records bytes as bytes (fixed).** It used to go through the
|
||||
millisecond-scaled event histogram and export as `rpc_size_milliseconds_bucket`
|
||||
on a ladder topping out at 5000, so the 24.9% of responses larger than 5 kB all
|
||||
landed in the last bucket and every percentile read back as a flat 5000 — a
|
||||
plausible-looking constant rather than a byte size. `beast::insight::Event` now
|
||||
declares a `Unit`, so this instrument is created with unit `By` and exports as
|
||||
**`rpc_size_bytes_bucket`** on `kByteBuckets` (512 B to 1 MiB, placed from the
|
||||
measured distribution). Queries and panels must use the new name.
|
||||
|
||||
**Grafana dashboards**: _Node Health_ (`ios_latency`), _RPC & Pathfinding_ (`rpc_time`, `rpc_size`, `pathfind_*`)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user