fix(telemetry): resolve microsecond latencies below 100us

The microsecond ladder's first edge was 100us, which sat ABOVE the mass of
every instrument using it. Measured on devnet: 99.3% of job_queued_us
samples, 92.5% of job_running_us and 90.4% of getobject_lookup_us fell in
that first bucket. histogram_quantile then interpolated inside bucket 0 and
returned `quantile / fraction_in_bucket_0 x first_edge` -- p75/p95/p99 of
job_queued_us read 75.52/95.66/99.69us against a prediction of
75.53/95.67/99.70. Three-decimal agreement: those panels were reporting
arithmetic on the bucket edge, not latency.

The fix was already half-written. kSubMillisecondBoundaries had been parked
in MetricsRegistry.cpp as [[maybe_unused]] with a comment noting exactly this
problem for nodestore reads. Its edges are now folded into kMicrosecondBuckets
rather than deleted, so the parked intent is carried forward: 1..1000us
resolution where the mass is, upper edges unchanged so multi-second stalls
stay measurable.

Also moves the GetObject count and charge ladders into HistogramBuckets.h, so
all five ladders have one owner and one set of invariant tests (29 now).

Adds check_bucket_parity.py, wired into the existing OTel naming workflow.
The C++ millisecond ladder and the collector's spanmetrics ladder are
specified to agree over their shared range; they were identical when shipped,
then the collector side alone was extended and nothing noticed for eleven
phases. The check asserts containment rather than equality, because jobs
outlive spans -- jobq_updatepaths averages ~60s, which no span approaches, so
demanding equality would force a ceiling that censors it. Verified it rejects
a missing collector edge, a bogus in-range edge, and a return to the 5s
ceiling.

ledger-data-sync's "Job Queue Wait p95 By Type" moves off the beast
jobq_*_q_milliseconds pair onto job_queued_us filtered by job_type. Those
beast metrics are ms-quantised at the source (Event rounds up to a whole
millisecond), so 94-100% of their samples sat in the first bucket and no
ladder change could fix them. Note the label values are camelCase
(job_type="ledgerData"), not the lowercase metric-name fragments.

Both histogram-fed alert thresholds re-validated and left unchanged, with the
measured basis recorded so neither gets tuned against the old artefact: only
0.0022% of job_queued_us samples exceed the 1s threshold, and every edge
bracketing the 1000ms ios_latency threshold survived the ladder change.

Docs: the rpc_size "known issue -- tracked separately" notes in the runbook
and 09-data-collection-reference are now resolved notes, the stale 10-edge
span_duration bucket list is corrected to the collector's real 20, and the
runbook gains a "Reading A Histogram Percentile" section covering both
saturation traps and the expected discontinuity after a ladder change.
This commit is contained in:
Pratik Mankawde
2026-08-21 12:46:56 +01:00
parent 7735d725fb
commit 6e2b2da772
10 changed files with 354 additions and 97 deletions

View File

@@ -60,7 +60,52 @@ INSTANTIATE_TEST_SUITE_P(
HistogramBucketsTest,
::testing::Values(
std::span<double const>{kMillisecondBuckets},
std::span<double const>{kByteBuckets}));
std::span<double const>{kByteBuckets},
std::span<double const>{kMicrosecondBuckets},
std::span<double const>{kObjectCountBuckets},
std::span<double const>{kChargeBuckets}));
TEST(HistogramBucketsRange, microsecondFloorLandsBelowTheMeasuredMass)
{
// Measured: 99.3% of job_queued_us samples sat below the old 100 us floor,
// so p75/p95/p99 all interpolated inside bucket 0 and returned
// 75.5/95.7/99.7 us -- the boundary scaled by the requested quantile,
// not a latency. Warm nodestore reads are ~1.5 us, so the floor has to
// reach single microseconds and several edges must precede 100 us.
EXPECT_LE(kMicrosecondBuckets.front(), 1.0);
auto const belowHundred =
std::ranges::count_if(kMicrosecondBuckets, [](double edge) { return edge < 100.0; });
EXPECT_GE(belowHundred, 5) << "too little resolution below 100 us";
}
TEST(HistogramBucketsRange, microsecondCeilingStillReachesOneMinute)
{
// Job waits and RPC latencies routinely exceed the SDK default ceiling of
// 10,000; multi-second stalls must stay measurable rather than censored.
EXPECT_EQ(kMicrosecondBuckets.back(), 60'000'000.0);
}
TEST(HistogramBucketsRange, objectCountLadderCannotSaturate)
{
// GetObject counts run 1..kHardMaxReplyNodes, so the top edge IS the hard
// cap and censoring is impossible by construction.
EXPECT_EQ(kObjectCountBuckets.front(), 1.0);
EXPECT_EQ(kObjectCountBuckets.back(), 12'288.0);
}
TEST(HistogramBucketsRange, chargeLadderBracketsTheResourceThresholds)
{
// The two edges that decide a peer's fate must be present so a dashboard
// can show how close charges run to each: warning at 5000, drop at 25000.
// A leading 0 separates the free tier from everything else.
EXPECT_EQ(kChargeBuckets.front(), 0.0);
for (double const threshold : {5'000.0, 25'000.0})
{
EXPECT_NE(std::ranges::find(kChargeBuckets, threshold), kChargeBuckets.end())
<< threshold << " is a resource threshold and must be an edge";
}
}
// The validator must also REJECT. A predicate that only ever returns true
// would let every ladder above pass while proving nothing.

View File

@@ -63,6 +63,7 @@
#include <xrpl/server/LoadFeeTrack.h>
#include <xrpl/server/NetworkOPs.h>
#include <xrpl/telemetry/GetObjectMetricNames.h>
#include <xrpl/telemetry/HistogramBuckets.h>
#include <xrpl/telemetry/SpanNames.h>
#include <opentelemetry/context/context.h>
@@ -84,7 +85,6 @@
#include <opentelemetry/semconv/incubating/service_attributes.h>
#include <algorithm>
#include <array>
#include <atomic>
#include <chrono>
#include <cstddef>
@@ -119,66 +119,16 @@ constexpr char kRpcMethodDurationUs[] = "rpc_method_us";
constexpr char kJobTypeLabel[] = "job_type";
constexpr char kHandlerLabel[] = "handler";
/**
* Bucket boundaries for microsecond-valued duration instruments.
*
* 100 µs, 500 µs, 1 ms, 5 ms, 10 ms, 25 ms, 50 ms, 100 ms, 250 ms, 500 ms,
* 1 s, 2.5 s, 5 s, 10 s, 30 s, 60 s. Covers sub-millisecond jobs through
* multi-second stalls without saturating.
*/
constexpr std::array kMicrosecondBoundaries{
100.0,
500.0,
1'000.0,
5'000.0,
10'000.0,
25'000.0,
50'000.0,
100'000.0,
250'000.0,
500'000.0,
1'000'000.0,
2'500'000.0,
5'000'000.0,
10'000'000.0,
30'000'000.0,
60'000'000.0};
/**
* Bucket boundaries for latencies that are normally sub-millisecond.
*
* 1 µs, 2 µs, 5 µs, 10 µs, 25 µs, 50 µs, 100 µs, 250 µs, 500 µs, 1 ms, 5 ms,
* 25 ms.
*
* kMicrosecondBoundaries starts at 100 µs, which is above the entire range a
* healthy nodestore read occupies, so every warm read falls in its first
* bucket and the distribution reads as flat. These edges resolve the warm
* range instead, while still reaching far enough to show a cold tail against
* it.
*
* Currently unused: no sub-millisecond histogram instrument exists yet. The
* edges live here so the instrument that records nodestore read latency gets
* a ladder that fits it, rather than silently inheriting the wrong one.
*/
[[maybe_unused]] constexpr std::array kSubMillisecondBoundaries{
1.0,
2.0,
5.0,
10.0,
25.0,
50.0,
100.0,
250.0,
500.0,
1'000.0,
5'000.0,
25'000.0};
/**
* Register an explicit-bucket histogram view.
*
* The SDK's default boundaries top out at 10,000, so any instrument whose
* values exceed that saturates and every quantile reads as the ceiling.
* values exceed that saturates and every quantile reads as the ceiling. The
* floor matters just as much and is easier to miss: a ladder whose first edge
* sits above the mass of the distribution makes every low quantile an
* interpolation inside bucket 0 -- a number derived from the bucket edge
* rather than from any sample. Both ends are chosen from measured
* distributions in HistogramBuckets.h.
*
* @param views The registry to add the view to.
* @param name Instrument name to match (e.g. "job_running_us").
@@ -206,7 +156,7 @@ addHistogramView(
* Register the microsecond-ladder view for a duration instrument.
*
* Job wait/run times and RPC latencies routinely exceed the SDK default
* ceiling, so they all share `kMicrosecondBoundaries`.
* ceiling, so they all share `buckets::kMicrosecondBuckets`.
*
* @param views The registry to add the view to.
* @param name Instrument name to match.
@@ -214,7 +164,10 @@ addHistogramView(
void
addMicrosecondHistogramView(metric_sdk::ViewRegistry& views, std::string const& name)
{
addHistogramView(views, name, {kMicrosecondBoundaries.begin(), kMicrosecondBoundaries.end()});
addHistogramView(
views,
name,
xrpl::telemetry::buckets::toVector(xrpl::telemetry::buckets::kMicrosecondBuckets));
}
} // namespace
@@ -352,18 +305,13 @@ MetricsRegistry::initExporterAndProvider(
// asks for at most 8, so the low buckets are fine-grained and the upper
// ones follow the charge size bands (64, 1024) up to the hard cap.
addHistogramView(
*views,
kGetObjectRequestObjects,
{1.0, 2.0, 4.0, 8.0, 16.0, 64.0, 256.0, 1'024.0, 4'096.0, 12'288.0});
*views, kGetObjectRequestObjects, buckets::toVector(buckets::kObjectCountBuckets));
// Charge values span 0 (free tier) to ~99k for a full-size all-miss
// request. Boundaries bracket the resource thresholds that decide a
// peer's fate -- kWarningThreshold (5000) and kDropThreshold (25000) --
// so a dashboard can show how close charges run to each.
addHistogramView(
*views,
kGetObjectCharge,
{0.0, 100.0, 500.0, 1'000.0, 5'000.0, 10'000.0, 25'000.0, 50'000.0, 100'000.0});
addHistogramView(*views, kGetObjectCharge, buckets::toVector(buckets::kChargeBuckets));
// Create MeterProvider with resource, then attach the metric reader.
provider_ = metric_sdk::MeterProviderFactory::Create(std::move(views), resourceAttrs);