mirror of
https://github.com/XRPLF/rippled.git
synced 2026-09-27 15:28:03 +00:00
a734da8b33
Refreshes baselines/baseline-timings.json from run 32964262700 at8418d474a7, byte-identical to the CI artifact. The previous baseline was captured at6a82fc6f37, before the path-finding load was removed from the workload, so it described a load shape the harness no longer runs. Every absolute bound is re-derived, because the rule is hi_next minus baseline and the baselines moved. Three more keys stop being gated: span.tx.apply.p50, span.ledger.build.p50 and span.consensus.ledger_close.p50. This is the rule the previous commit recorded being applied, not a new exception -- a key is gateable only when its run-to-run spread fits inside its bound. The evidence is span.tx.apply.p50, which read 0.7917 ms in the old baseline and 0.00597 ms in this one. That is a 132x move between two runs of the SAME workload. The old value happened to land mid-distribution, so hi_next minus baseline gave a 4.21 ms bound that absorbed the spread; the new value lands in the ladder's first bucket, so the same rule gives 0.0440 ms and cannot survive one. Whether the gate functioned was decided by where in the distribution the captured run happened to fall, which is not a threshold in need of tuning. Measured spreads across four runs agree: 364x, 25.3x and 5.9x respectively. All five excluded keys share one shape -- a baseline landing in the ladder's low buckets, where the derived bound is tiny, together with large run-to-run spread. Single-run baselines cannot support them; a multi-run baseline, or a spread measurement captured alongside the baseline, is what would let them be gated again. Not attempted here. Both runs that would have reddened CI now replay clean, and an injected 10x regression is still caught on 19 of the 20 remaining keys, 20 of 20 at 20x. The exception is job.acceptLedger.running.p95, whose baseline fell while its hi_next did not, moving its floor to 16.28x. It stays gated with that floor recorded beside the other weak keys. Also makes the bounds checker report a zero or negative baseline as a named rule failure instead of dividing by it and raising.
148 lines
18 KiB
JSON
148 lines
18 KiB
JSON
{
|
|
"_description": "Per-metric regression thresholds. A metric regresses when current - baseline exceeds BOTH the percentage and absolute bounds (AND, not OR \u2014 this tolerates small-value noise). Defaults apply unless a per-metric override exists.",
|
|
"_bucket_note": "SpanMetrics latency histograms use explicit buckets [0.01,0.05,0.1,0.25,0.5,1,5,10,25,50,100,250,500]ms then [1,2,3,4,5,10,30]s (20 edges; docker/telemetry/otel-collector-config.yaml is the authoritative list). Second-scale consensus spans have 2s/3s/4s boundaries, so their quantiles quantize to ~1s widths there \u2014 the ladder is NOT uniformly 2x-or-coarser, which matters for _percentage_bound_note. The native job_queue histograms are microsecond-valued on the ladder [1,2,5,10,25,50,100,250,500,1000,5000,25000,100000,500000]us then [1,5,10,30,60]s (19 edges; include/xrpl/telemetry/HistogramBuckets.h is authoritative). NOTE: BOTH ladders were re-cut, and a baseline captured before its own ladder changed is an interpolation artefact, not a latency. The job_queue floor moved 100us \u2192 1us. The span ladder was re-cut on 2026-08-04 in 3860c93db2, moving the floor 1ms \u2192 0.01ms; so any sub-millisecond span quantile captured before that date is equally void \u2014 a p95 reading 0.95ms is 0.95 \u00d7 the old 1ms first edge, not a measurement. An earlier note asserted that the surviving span baselines were unaffected by the ladder work; that is wrong for every span quantile below 1ms. Only the band from 1ms to 1s is safe: those edges are byte-identical across the two ladders. The re-cut also ADDED edges above 1s (2s/3s/4s/10s/30s), so a span whose quantiles land in the second-scale range \u2014 consensus.round ~3.9s, consensus.establish ~1.9s, the ledger.acquire tail \u2014 is distorted just as much, and any pre-2026-08-04 baseline for it is equally void. Do not read this note as licensing a stale second-scale baseline.",
|
|
"_absolute_bound_derivation": "HOW EVERY max_abs_increase_* NUMBER BELOW WAS OBTAINED. Rule: locate the baseline value in the half-open bucket (lo, hi] of its own ladder, take hi_next = the next edge above hi, and set the bound to (hi_next - baseline). The trip point is therefore exactly hi_next: the gate fires only when the reported value EXCEEDS the top of the bucket above the baseline's own bucket. WHY THAT AND NOT A MULTIPLE OF THE BUCKET WIDTH: histogram_quantile returns a value interpolated inside whichever bucket the true quantile falls in, so a reading taken while the true quantile sits anywhere in the baseline's bucket OR anywhere in the one immediately above is at most hi_next and cannot fire. Firing requires the true quantile to have moved at least two buckets up. A multiple of the ENCLOSING width cannot deliver that, because once the quantile crosses hi the interpolation happens across the NEXT bucket, which on this ladder is up to 8x wider \u2014 (0.5,1] has width 0.5 and (1,5] has width 4 \u2014 so the reading's excursion is not bounded by any multiple of the enclosing width. Worked example: span.tx.process.p99 has baseline 2.7588ms in bucket (1, 5], hi_next = 10, so its bound is 7.2412ms and the gate fires only above 10ms. Bounds are stored as exact doubles rather than rounded figures so that rounding cannot break the guarantee and so check_regression_bounds.py can assert each one against the ladder to within a 1e-12 relative tolerance -- tight enough that a bound rounded for readability, such as 7.2412 for 7.241212121212123, is rejected; _derivation_table below shows the arithmetic for each one. Measured over the committed baseline this rule yields a detection floor of 2.21x to 16.28x of baseline, per key. WHAT THIS RULE DOES NOT COVER, AND THE ONE CHECK TO RUN BEFORE GATING ANY KEY: hi_next - baseline is derived from the LADDER, so it budgets for QUANTIZATION noise -- one bucket of interpolation headroom -- and for nothing else. It knows nothing about how much the metric itself moves between runs on identical code. Where run-to-run workload variance is the larger term the bound is simply the wrong size, and the gate reddens on a healthy run. So before adding a key here, capture it over several runs and check its OBSERVED MAXIMUM against its trip point (baseline + bound); gate it only if the observed maximum stays below that trip point with margin. Spread on its own proves nothing -- it is spread RELATIVE TO THE TRIP POINT that decides, and a baseline that lands at the LOW end of a metric's own range shrinks that trip point even though nothing about the metric changed. THREE KEYS FAILED THIS TEST ON THE 2026-08-26 BASELINE AND ARE NOW EXCLUDED, all of them p50: span.tx.apply.p50 (bound 0.0440ms, trips at 0.05ms, observed max 2.3378ms = 46.76x its trip point), span.ledger.build.p50 (bound 0.3849ms, trips at 0.5ms, observed max 2.3826ms = 4.77x) and span.consensus.ledger_close.p50 (bound 0.0613ms, trips at 0.1ms, observed max 0.2377ms = 2.38x). Their spreads across three runs are 391.8x, 20.7x and 6.1x. This is the general rule above being APPLIED, not a new exception: a key is gateable only when its run-to-run spread fits inside its bound, and these three do not. The evidence that settles it is span.tx.apply.p50's own history -- it read 0.7917ms in the previous baseline and 0.00597ms in this one, a 132x difference between two runs of the SAME workload. At the old value the identical rule produced a 4.21ms bound whose 5ms trip point absorbed the whole range; at the new one it produces 0.0440ms and cannot. Whether the gate functioned was therefore decided by where in its distribution the captured run happened to land, which is not a threshold needing tuning but a key that cannot be gated from a single-run baseline at all. Before the exclusion, replaying the two preceding CI runs 32862589645 and 32867433073 against this baseline reported exactly those three and nothing else on BOTH runs, and 32867433073 carries the same post-path-finding-removal workload as the baseline itself -- so the movement was metric variance, not a workload difference. After it, both runs replay clean. The remaining 20 keys sit at or below 0.58 of their trip points, the worst being span.consensus.accept.p50. See _excluded_shape in regression-metrics.json for what all five excluded keys have in common and for the multi-run-baseline work that would let them be gated again. A key that fails this test is not fixed by widening its bound: see excluded_keys in regression-metrics.json. WHAT THIS REPLACED, IN TWO GENERATIONS: (1) a single flat pair of bounds (10ms for span p50/p95, 15ms for span p99, 20000us for job_queue p95) justified as 'roughly two bucket widths in the 5-25ms band where most span quantiles actually sit'. The 2026-08-24 capture falsifies that premise \u2014 18 of the 28 quantiles gated at that time sat below 1ms \u2014 so the absolute bound sat 1.15x to 2000x above the metric it guarded and, because the rule is an AND, the percentage bound could never carry a regression on its own; a 10x regression injected into each key in turn was caught on only 5 of 28, and a 100x regression injected into span.ledger.store.p95 produced 0 regressions and exit 0. (2) a first correction to 2 \u00d7 the ENCLOSING bucket width, which caught 10x on 28 of 28 but placed the trip point INSIDE the adjacent bucket -- and so left a single-crossing false positive reachable -- on 21 of the 25 keys gated at the time, 4 of them tripping on a tail-mass shift under 1.5% of samples. That is the assumption this rule removes. RE-DERIVE THESE NUMBERS whenever baseline-timings.json is refreshed or either ladder changes: a refreshed baseline can land in a different bucket, which changes hi_next. .github/scripts/telemetry/check_regression_bounds.py enforces the rule in CI so a stale bound cannot survive a baseline refresh. LIMITATION \u2014 WHICH KEYS ARE ONLY WEAKLY GUARDED: the guarantee costs sensitivity wherever the ladder is coarse, and the detection floor is hi_next/baseline, so a baseline sitting just above an edge is guarded loosely. job.acceptLedger.running.p95 (baseline 6142.86us, fires at 100000us, 16.28x) is NOT meaningfully guarded, and it is now the one key a 10x regression does NOT catch: measured, 10x reaches 61429us and passes, and the gate first fires at 16.28x. It sits just above the 5000us edge while hi_next is 100000us, two steps up. Its floor moved there in this refresh, from 5.74x, because its baseline fell 17428.57us to 6142.86us while hi_next stayed at 100000us -- it does NOT fire on any observed run, so it stays gated, but the weak floor is recorded here so it is visible rather than surprising. span.consensus.accept.p50 (9.46x), job.transaction.running.p95 (8.33x), span.tx.process.p95 (8.20x), span.rpc.ws_message.p95 (7.17x), span.consensus.ledger_close.p95 (6.39x) and span.rpc.ws_message.p99 (5.12x) are also weak. Four of the seven are limited by the 1ms\u21925ms step; the rest by 1000us\u21925000us (job.transaction.running.p95) and 25000us\u2192100000us (job.acceptLedger.running.p95). The fix is a 2ms edge (and ideally 3ms) in the collector's spanmetrics ladder plus the matching edges in kMillisecondBuckets, and 2000us plus 50000us edges in kMicrosecondBuckets \u2014 that work belongs to the branch that owns the ladders, not here. Until then do not read these keys as guarded. span.ledger.store is absent from the overrides below because it was removed from the gated surface entirely: its quantiles were the ladder floor times the quantile, so no bound could gate it. See _excluded_ledger_store in regression-metrics.json.",
|
|
"_percentage_bound_note": "For every key gated today the absolute bound is the binding half of the AND and the percentage bound never decides the outcome: measured, (bound / baseline) ranges from 121% (span.ledger.build.p95) to 1528% (job.acceptLedger.running.p95), all above the 50% and 5% percentage bounds configured here, and the minimum trip multiple of all 20 keys is set by the absolute bound. THIS IS NOT A GENERAL GUARANTEE, and an earlier version of this note wrongly claimed it was, on the false premise that 'every step of both ladders is at least a factor of 2'. The span ladder breaks that three times at the top: 2s->3s is 1.5x, 3s->4s is 1.33x, 4s->5s is 1.25x, so second-scale consensus quantiles quantize to ~1s widths there. Because the bound is (hi_next - baseline), a baseline between about 2667ms and 3000ms, or between about 3334ms and 4000ms, gets an absolute bound worth less than 50% of itself and the PERCENTAGE bound becomes the operative one -- at which point the metric fires on a 50% move that is smaller than one bucket width, and the single-crossing guarantee in _absolute_bound_derivation is lost. That band is not hypothetical: the collector config names consensus.round (~3.9s) as a reason those edges exist, and 3900ms sits in the second sub-band with an absolute bound of 5000 - 3900 = 1100, only 28.2% of baseline. Whoever gates a key whose baseline lands in either sub-band MUST lower its max_pct_increase below (bound / baseline) for that key, or state explicitly that the metric is percentage-gated and the bucket guarantee does not hold for it. check_regression_bounds.py enforces this as rule D so the trap cannot be walked into silently. The percentage entries are required and still meaningful regardless: compare_to_baseline.py treats a missing max_pct_increase as 'no threshold configured' and would stop gating the metric entirely; they record the intended relative tolerance (consensus spans 5%, everything else 50%); and they are the operative bound on the defaults path (see _defaults_note).",
|
|
"_defaults_note": "A MISSING OVERRIDE IS DETECTED BY CI, NOT BY THESE DEFAULTS. .github/scripts/telemetry/check_regression_bounds.py fails the build at lint time, naming the key and the exact value its bound should have, before the workload ever runs. That is the mechanism; the defaults below are only a runtime backstop for the case where that check is bypassed. The defaults carry the FLOOR of each ladder as their absolute bound \u2014 0.01ms for spans, 1us for job_queue \u2014 deliberately too small to bind for any real metric, which leaves max_pct_increase (50%) as the operative bound on this path. Measured: a metric with no override and a baseline of 3900ms passes at +49% and fires at +51%; a job metric with a baseline of 5000us behaves the same. The backstop is honestly imperfect and the earlier version of this note oversold it. At 50% relative it CAN false-fire: a metric whose baseline is 1.06ms inside the 4ms-wide (1,5] bucket fires on a single-bucket-width move (measured: 1.06 \u2192 5.06ms, +377%, regressed). An earlier note called that 'the intended signal that the override is missing', which was wrong \u2014 CI prints REGRESSION and a reader cannot tell it from a real one, and rejecting a tighter alternative for exactly that cries-wolf risk while shipping it here would be inconsistent. The check is what makes the signal legible. The backstop is kept only because a metric silently not gated at all is the worse of the two failures.",
|
|
"_derivation_table": {
|
|
"_format": "override key: <quantile> <baseline> in <bucket> -> hi_next - baseline = <bound>",
|
|
"job.acceptLedger.queued": "p95 166.13636363636323 in (100,250] -> hi_next 500 - baseline = 333.8636363636368",
|
|
"job.acceptLedger.running": "p95 6142.857142857149 in (5000,25000] -> hi_next 100000 - baseline = 93857.14285714286",
|
|
"job.transaction.queued": "p95 426.19047619047586 in (250,500] -> hi_next 1000 - baseline = 573.8095238095241",
|
|
"job.transaction.running": "p95 599.9999999999986 in (500,1000] -> hi_next 5000 - baseline = 4400.000000000002",
|
|
"span.consensus.accept": "p50 0.5287356321839081 in (0.5,1] -> hi_next 5 - baseline = 4.471264367816092 | p95 8.969696969696969 in (5,10] -> hi_next 25 - baseline = 16.03030303030303 | p99 20.800000000000026 in (10,25] -> hi_next 50 - baseline = 29.199999999999974",
|
|
"span.consensus.ledger_close": "p95 0.7829999999999997 in (0.5,1] -> hi_next 5 - baseline = 4.2170000000000005 | p99 2.0299999999999896 in (1,5] -> hi_next 10 - baseline = 7.97000000000001 (p50 is NOT gated -- see excluded_keys in regression-metrics.json)",
|
|
"span.ledger.build": "p95 4.53333333333333 in (1,5] -> hi_next 10 - baseline = 5.46666666666667 | p99 9.109090909090913 in (5,10] -> hi_next 25 - baseline = 15.890909090909087 (p50 is NOT gated -- see excluded_keys in regression-metrics.json)",
|
|
"span.ledger.validate": "p50 0.06471894002114831 in (0.05,0.1] -> hi_next 0.25 - baseline = 0.1852810599788517 (p95 and p99 are NOT gated -- see excluded_keys in regression-metrics.json: their run-to-run spread, 5.9x and 66.8x over four CI runs, exceeds any bound this rule can derive)",
|
|
"span.rpc.ws_message": "p50 0.16003451676528602 in (0.1,0.25] -> hi_next 0.5 - baseline = 0.339965483234714 | p95 0.6977397260273974 in (0.5,1] -> hi_next 5 - baseline = 4.302260273972602 | p99 0.9757123287671235 in (0.5,1] -> hi_next 5 - baseline = 4.024287671232877",
|
|
"span.tx.apply": "p95 3.524999999999999 in (1,5] -> hi_next 10 - baseline = 6.475000000000001 | p99 5.066666666666704 in (5,10] -> hi_next 25 - baseline = 19.933333333333294 (p50 is NOT gated -- see excluded_keys in regression-metrics.json)",
|
|
"span.tx.process": "p50 0.20062219789579117 in (0.1,0.25] -> hi_next 0.5 - baseline = 0.29937780210420883 | p95 0.6100467289719625 in (0.5,1] -> hi_next 5 - baseline = 4.389953271028038 | p99 2.758787878787877 in (1,5] -> hi_next 10 - baseline = 7.241212121212123"
|
|
},
|
|
"defaults": {
|
|
"span": {
|
|
"p50": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 0.01
|
|
},
|
|
"p95": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 0.01
|
|
},
|
|
"p99": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 0.01
|
|
}
|
|
},
|
|
"job_queue": {
|
|
"p95": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_us": 1.0
|
|
}
|
|
}
|
|
},
|
|
"overrides": {
|
|
"job.acceptLedger.queued": {
|
|
"p95": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_us": 333.8636363636368
|
|
}
|
|
},
|
|
"job.acceptLedger.running": {
|
|
"p95": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_us": 93857.14285714286
|
|
}
|
|
},
|
|
"job.transaction.queued": {
|
|
"p95": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_us": 573.8095238095241
|
|
}
|
|
},
|
|
"job.transaction.running": {
|
|
"p95": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_us": 4400.000000000002
|
|
}
|
|
},
|
|
"span.consensus.accept": {
|
|
"p50": {
|
|
"max_pct_increase": 5.0,
|
|
"max_abs_increase_ms": 4.471264367816092
|
|
},
|
|
"p95": {
|
|
"max_pct_increase": 5.0,
|
|
"max_abs_increase_ms": 16.03030303030303
|
|
},
|
|
"p99": {
|
|
"max_pct_increase": 5.0,
|
|
"max_abs_increase_ms": 29.199999999999974
|
|
}
|
|
},
|
|
"span.consensus.ledger_close": {
|
|
"p95": {
|
|
"max_pct_increase": 5.0,
|
|
"max_abs_increase_ms": 4.2170000000000005
|
|
},
|
|
"p99": {
|
|
"max_pct_increase": 5.0,
|
|
"max_abs_increase_ms": 7.97000000000001
|
|
}
|
|
},
|
|
"span.ledger.build": {
|
|
"p95": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 5.46666666666667
|
|
},
|
|
"p99": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 15.890909090909087
|
|
}
|
|
},
|
|
"span.ledger.validate": {
|
|
"p50": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 0.1852810599788517
|
|
}
|
|
},
|
|
"span.rpc.ws_message": {
|
|
"p50": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 0.339965483234714
|
|
},
|
|
"p95": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 4.302260273972602
|
|
},
|
|
"p99": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 4.024287671232877
|
|
}
|
|
},
|
|
"span.tx.apply": {
|
|
"p95": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 6.475000000000001
|
|
},
|
|
"p99": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 19.933333333333294
|
|
}
|
|
},
|
|
"span.tx.process": {
|
|
"p50": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 0.29937780210420883
|
|
},
|
|
"p95": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 4.389953271028038
|
|
},
|
|
"p99": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 7.241212121212123
|
|
}
|
|
}
|
|
}
|
|
}
|