mirror of
https://github.com/XRPLF/rippled.git
synced 2026-10-04 10:48:15 +00:00
3836078a78
The regression gate has been red on runs with no code change. Only two of the 25 gated keys ever tripped, both on the same span and never together: run 32862589645 failed p99 at 25.8750 ms against a 1.0600 ms baseline (+2341%), run 32867433073 failed p95 at 0.7500 ms against 0.2404 ms (+212%), and in each run the other quantile sat well inside its own bound. A real slowdown would move both. This is variance, not a defect. Measured across four CI runs: span.ledger.validate.p50 0.0484 to 0.0778 ms 1.6x spread kept span.ledger.validate.p95 0.1281 to 0.7500 ms 5.9x spread excluded span.ledger.validate.p99 0.3875 to 25.8750 ms 66.8x spread excluded Both excluded quantiles reach past their trip point on a healthy run. The mechanism is arrival timing, not slow code: the span opens only once a quorum-completing validation arrives (LedgerMaster.cpp:987, inside checkAccept, past the early return) and wraps the promotion work that follows, so one slow consensus round dominates the tail of a 3m rate window and which round that is differs every run. Widening is not available and must not be attempted later: tolerating 25.8750 ms against a 1.0600 ms baseline needs a bound of about 24.8 ms, which gates nothing. A bound admitting every healthy run's worst case admits every regression too. p50 stays gated; it is stable. THE GENERAL RULE, recorded so this does not recur: an absolute bound derived as hi_next minus baseline comes from the histogram ladder, so it budgets for quantization noise and for nothing else. It knows nothing about how far a metric moves between runs on identical code. Before gating any key, check its observed maximum across several runs against its trip point and gate it only with margin. Spread alone proves nothing: tx.apply.p50 swings 364x and never fires, because its 5 ms trip point absorbs the range. Of the 23 keys still gated the worst reaches 0.67 of its trip point. Mechanism: spans.names lists span names while _quantiles is shared, so dropping two quantiles of one span cannot be expressed by deleting a name. regression-metrics.json gains an excluded_keys map from a flat key to the reason it is not gated, subtracted by both prom_queries.py (so the key is never queried) and check_regression_bounds.py rule A. A per-name quantile override was rejected: a typo there leaves the key gating, whereas a typo in an exclusion subtracts nothing and new rule F rejects it, along with an empty reason, a leftover threshold override and a leftover baseline value. Derived figures recomputed from the committed baseline: 25 gated keys to 23, detection floor 2.02x-9.43x to 2.02x-9.42x, weakly guarded keys ten to nine, bound over baseline 102%-843% to 102%-842%. The baseline edit is a deletion of two entries only, with no value rewritten. Verified: both previously failing runs replay to zero regressions and exit 0; a tenfold increase injected into each of the 23 remaining keys in turn is still caught in all 23 cases; rule F was confirmed load-bearing by stubbing it out, which lets a stale exclusion pass.
160 lines
16 KiB
JSON
160 lines
16 KiB
JSON
{
|
|
"_description": "Per-metric regression thresholds. A metric regresses when current - baseline exceeds BOTH the percentage and absolute bounds (AND, not OR \u2014 this tolerates small-value noise). Defaults apply unless a per-metric override exists.",
|
|
"_bucket_note": "SpanMetrics latency histograms use explicit buckets [0.01,0.05,0.1,0.25,0.5,1,5,10,25,50,100,250,500]ms then [1,2,3,4,5,10,30]s (20 edges; docker/telemetry/otel-collector-config.yaml is the authoritative list). Second-scale consensus spans have 2s/3s/4s boundaries, so their quantiles quantize to ~1s widths there \u2014 the ladder is NOT uniformly 2x-or-coarser, which matters for _percentage_bound_note. The native job_queue histograms are microsecond-valued on the ladder [1,2,5,10,25,50,100,250,500,1000,5000,25000,100000,500000]us then [1,5,10,30,60]s (19 edges; include/xrpl/telemetry/HistogramBuckets.h is authoritative). NOTE: BOTH ladders were re-cut, and a baseline captured before its own ladder changed is an interpolation artefact, not a latency. The job_queue floor moved 100us \u2192 1us. The span ladder was re-cut on 2026-08-04 in 3860c93db2, moving the floor 1ms \u2192 0.01ms; so any sub-millisecond span quantile captured before that date is equally void \u2014 a p95 reading 0.95ms is 0.95 \u00d7 the old 1ms first edge, not a measurement. An earlier note asserted that the surviving span baselines were unaffected by the ladder work; that is wrong for every span quantile below 1ms. Only the band from 1ms to 1s is safe: those edges are byte-identical across the two ladders. The re-cut also ADDED edges above 1s (2s/3s/4s/10s/30s), so a span whose quantiles land in the second-scale range \u2014 consensus.round ~3.9s, consensus.establish ~1.9s, the ledger.acquire tail \u2014 is distorted just as much, and any pre-2026-08-04 baseline for it is equally void. Do not read this note as licensing a stale second-scale baseline.",
|
|
"_absolute_bound_derivation": "HOW EVERY max_abs_increase_* NUMBER BELOW WAS OBTAINED. Rule: locate the baseline value in the half-open bucket (lo, hi] of its own ladder, take hi_next = the next edge above hi, and set the bound to (hi_next - baseline). The trip point is therefore exactly hi_next: the gate fires only when the reported value EXCEEDS the top of the bucket above the baseline's own bucket. WHY THAT AND NOT A MULTIPLE OF THE BUCKET WIDTH: histogram_quantile returns a value interpolated inside whichever bucket the true quantile falls in, so a reading taken while the true quantile sits anywhere in the baseline's bucket OR anywhere in the one immediately above is at most hi_next and cannot fire. Firing requires the true quantile to have moved at least two buckets up. A multiple of the ENCLOSING width cannot deliver that, because once the quantile crosses hi the interpolation happens across the NEXT bucket, which on this ladder is up to 8x wider \u2014 (0.5,1] has width 0.5 and (1,5] has width 4 \u2014 so the reading's excursion is not bounded by any multiple of the enclosing width. Worked example: span.tx.process.p99 has baseline 0.9945ms in bucket (0.5, 1], hi_next = 5, so its bound is 4.0055ms and the gate fires only above 5ms. Bounds are stored as exact doubles rather than rounded figures so that rounding cannot break the guarantee and so check_regression_bounds.py can assert each one against the ladder to within a 1e-12 relative tolerance -- tight enough that a bound rounded for readability, such as 4.0055 for 4.005485184848892, is rejected; _derivation_table below shows the arithmetic for each one. Measured over the committed baseline this rule yields a detection floor of 2.02x to 9.42x of baseline, per key. WHAT THIS RULE DOES NOT COVER, AND THE ONE CHECK TO RUN BEFORE GATING ANY KEY: hi_next - baseline is derived from the LADDER, so it budgets for QUANTIZATION noise -- one bucket of interpolation headroom -- and for nothing else. It knows nothing about how much the metric itself moves between runs on identical code. Where run-to-run workload variance is the larger term the bound is simply the wrong size, and the gate reddens on a healthy run. So before adding a key here, capture it over several runs and check its OBSERVED MAXIMUM against its trip point (baseline + bound); gate it only if the observed maximum stays below that trip point with margin. Spread on its own proves nothing -- span.tx.apply.p50 swings 364x across runs and never fires, because its 5ms trip point absorbs the whole range -- it is spread RELATIVE TO THE TRIP POINT that decides. Measured across the runs behind this baseline, the worst surviving key reaches 0.67 of its trip point (span.consensus.ledger_close.p95), and the only two that exceeded it were span.ledger.validate p95 and p99, now excluded. A key that fails this test is not fixed by widening its bound: see excluded_keys in regression-metrics.json. WHAT THIS REPLACED, IN TWO GENERATIONS: (1) a single flat pair of bounds (10ms for span p50/p95, 15ms for span p99, 20000us for job_queue p95) justified as 'roughly two bucket widths in the 5-25ms band where most span quantiles actually sit'. The 2026-08-24 capture falsifies that premise \u2014 18 of the 28 quantiles gated at that time sat below 1ms \u2014 so the absolute bound sat 1.15x to 2000x above the metric it guarded and, because the rule is an AND, the percentage bound could never carry a regression on its own; a 10x regression injected into each key in turn was caught on only 5 of 28, and a 100x regression injected into span.ledger.store.p95 produced 0 regressions and exit 0. (2) a first correction to 2 \u00d7 the ENCLOSING bucket width, which caught 10x on 28 of 28 but placed the trip point INSIDE the adjacent bucket -- and so left a single-crossing false positive reachable -- on 21 of the 25 keys gated at the time, 4 of them tripping on a tail-mass shift under 1.5% of samples. That is the assumption this rule removes. RE-DERIVE THESE NUMBERS whenever baseline-timings.json is refreshed or either ladder changes: a refreshed baseline can land in a different bucket, which changes hi_next. .github/scripts/telemetry/check_regression_bounds.py enforces the rule in CI so a stale bound cannot survive a baseline refresh. LIMITATION \u2014 WHICH KEYS ARE ONLY WEAKLY GUARDED: the guarantee costs sensitivity wherever the ladder is coarse, and the detection floor is hi_next/baseline, so a baseline sitting just above an edge is guarded loosely. span.ledger.build.p50 (baseline 1.0612ms, fires at 10ms, 9.42x) is NOT meaningfully guarded: it fires only at 10ms, so it could get up to 9.4x slower -- 1.06ms to just under 10ms -- and still pass. span.tx.process.p95 (6.91x), span.tx.apply.p50 (6.32x), span.rpc.ws_message.p95 (5.92x), job.acceptLedger.running.p95 (5.74x), span.consensus.accept.p50 (5.74x), span.consensus.ledger_close.p99 (5.37x), span.rpc.ws_message.p99 (5.06x) and span.tx.process.p99 (5.03x) are weak. All nine are limited by two 5x-wide ladder steps, 1ms\u21925ms and 5000us\u219225000us. The fix is a 2ms edge (and ideally 3ms) in the collector's spanmetrics ladder plus the matching edges in kMillisecondBuckets, and a 10000us edge in kMicrosecondBuckets \u2014 that work belongs to the branch that owns the ladders, not here. Until then do not read these keys as guarded. span.ledger.store is absent from the overrides below because it was removed from the gated surface entirely: its quantiles were the ladder floor times the quantile, so no bound could gate it. See _excluded_ledger_store in regression-metrics.json.",
|
|
"_percentage_bound_note": "For every key gated today the absolute bound is the binding half of the AND and the percentage bound never decides the outcome: measured, (bound / baseline) ranges from 102% (span.tx.apply.p99) to 842% (span.ledger.build.p50), all above the 50% and 5% percentage bounds configured here, and the minimum trip multiple of all 23 keys is set by the absolute bound. THIS IS NOT A GENERAL GUARANTEE, and an earlier version of this note wrongly claimed it was, on the false premise that 'every step of both ladders is at least a factor of 2'. The span ladder breaks that three times at the top: 2s->3s is 1.5x, 3s->4s is 1.33x, 4s->5s is 1.25x, so second-scale consensus quantiles quantize to ~1s widths there. Because the bound is (hi_next - baseline), a baseline between about 2667ms and 3000ms, or between about 3334ms and 4000ms, gets an absolute bound worth less than 50% of itself and the PERCENTAGE bound becomes the operative one -- at which point the metric fires on a 50% move that is smaller than one bucket width, and the single-crossing guarantee in _absolute_bound_derivation is lost. That band is not hypothetical: the collector config names consensus.round (~3.9s) as a reason those edges exist, and 3900ms sits in the second sub-band with an absolute bound of 5000 - 3900 = 1100, only 28.2% of baseline. Whoever gates a key whose baseline lands in either sub-band MUST lower its max_pct_increase below (bound / baseline) for that key, or state explicitly that the metric is percentage-gated and the bucket guarantee does not hold for it. check_regression_bounds.py enforces this as rule D so the trap cannot be walked into silently. The percentage entries are required and still meaningful regardless: compare_to_baseline.py treats a missing max_pct_increase as 'no threshold configured' and would stop gating the metric entirely; they record the intended relative tolerance (consensus spans 5%, everything else 50%); and they are the operative bound on the defaults path (see _defaults_note).",
|
|
"_defaults_note": "A MISSING OVERRIDE IS DETECTED BY CI, NOT BY THESE DEFAULTS. .github/scripts/telemetry/check_regression_bounds.py fails the build at lint time, naming the key and the exact value its bound should have, before the workload ever runs. That is the mechanism; the defaults below are only a runtime backstop for the case where that check is bypassed. The defaults carry the FLOOR of each ladder as their absolute bound \u2014 0.01ms for spans, 1us for job_queue \u2014 deliberately too small to bind for any real metric, which leaves max_pct_increase (50%) as the operative bound on this path. Measured: a metric with no override and a baseline of 3900ms passes at +49% and fires at +51%; a job metric with a baseline of 5000us behaves the same. The backstop is honestly imperfect and the earlier version of this note oversold it. At 50% relative it CAN false-fire: a metric whose baseline is 1.06ms inside the 4ms-wide (1,5] bucket fires on a single-bucket-width move (measured: 1.06 \u2192 5.06ms, +377%, regressed). An earlier note called that 'the intended signal that the override is missing', which was wrong \u2014 CI prints REGRESSION and a reader cannot tell it from a real one, and rejecting a tighter alternative for exactly that cries-wolf risk while shipping it here would be inconsistent. The check is what makes the signal legible. The backstop is kept only because a metric silently not gated at all is the worse of the two failures.",
|
|
"_derivation_table": {
|
|
"_format": "override key: <quantile> <baseline> in <bucket> -> hi_next - baseline = <bound>",
|
|
"job.acceptLedger.queued": "p95 91.10576923076925 in (50,100] -> hi_next 250 - baseline = 158.89423076923075",
|
|
"job.acceptLedger.running": "p95 17428.571428571428 in (5000,25000] -> hi_next 100000 - baseline = 82571.42857142858",
|
|
"job.transaction.queued": "p95 476.6129032258061 in (250,500] -> hi_next 1000 - baseline = 523.3870967741939",
|
|
"job.transaction.running": "p95 427.1084337349388 in (250,500] -> hi_next 1000 - baseline = 572.8915662650612",
|
|
"span.consensus.accept": "p50 1.7435897435897438 in (1,5] -> hi_next 10 - baseline = 8.256410256410255 | p95 8.9296875 in (5,10] -> hi_next 25 - baseline = 16.0703125 | p99 15.150000000000082 in (10,25] -> hi_next 50 - baseline = 34.849999999999916",
|
|
"span.consensus.ledger_close": "p50 0.15142857142857144 in (0.1,0.25] -> hi_next 0.5 - baseline = 0.34857142857142853 | p95 0.49328358208955236 in (0.25,0.5] -> hi_next 1 - baseline = 0.5067164179104476 | p99 0.9314285714285726 in (0.5,1] -> hi_next 5 - baseline = 4.068571428571428",
|
|
"span.ledger.build": "p50 1.0612244897959187 in (1,5] -> hi_next 10 - baseline = 8.938775510204081 | p95 4.679591836734694 in (1,5] -> hi_next 10 - baseline = 5.320408163265306 | p99 5.075000000000041 in (5,10] -> hi_next 25 - baseline = 19.924999999999958",
|
|
"span.ledger.validate": "p50 0.07787769784172663 in (0.05,0.1] -> hi_next 0.25 - baseline = 0.17212230215827337 (p95 and p99 are NOT gated -- see excluded_keys in regression-metrics.json: their run-to-run spread, 5.9x and 66.8x over four CI runs, exceeds any bound this rule can derive)",
|
|
"span.rpc.ws_message": "p50 0.15990942879094971 in (0.1,0.25] -> hi_next 0.5 - baseline = 0.3400905712090503 | p95 0.8443074003795064 in (0.5,1] -> hi_next 5 - baseline = 4.155692599620494 | p99 0.9878368121442125 in (0.5,1] -> hi_next 5 - baseline = 4.012163187855787",
|
|
"span.tx.apply": "p50 0.791666666666667 in (0.5,1] -> hi_next 5 - baseline = 4.208333333333333 | p95 4.627368421052632 in (1,5] -> hi_next 10 - baseline = 5.372631578947368 | p99 4.959157894736842 in (1,5] -> hi_next 10 - baseline = 5.040842105263158",
|
|
"span.tx.process": "p50 0.34281067382135394 in (0.25,0.5] -> hi_next 1 - baseline = 0.6571893261786461 | p95 0.7239865112994349 in (0.5,1] -> hi_next 5 - baseline = 4.276013488700565 | p99 0.9945148151511076 in (0.5,1] -> hi_next 5 - baseline = 4.005485184848892"
|
|
},
|
|
"defaults": {
|
|
"span": {
|
|
"p50": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 0.01
|
|
},
|
|
"p95": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 0.01
|
|
},
|
|
"p99": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 0.01
|
|
}
|
|
},
|
|
"job_queue": {
|
|
"p95": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_us": 1.0
|
|
}
|
|
}
|
|
},
|
|
"overrides": {
|
|
"job.acceptLedger.queued": {
|
|
"p95": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_us": 158.89423076923075
|
|
}
|
|
},
|
|
"job.acceptLedger.running": {
|
|
"p95": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_us": 82571.42857142858
|
|
}
|
|
},
|
|
"job.transaction.queued": {
|
|
"p95": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_us": 523.3870967741939
|
|
}
|
|
},
|
|
"job.transaction.running": {
|
|
"p95": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_us": 572.8915662650612
|
|
}
|
|
},
|
|
"span.consensus.accept": {
|
|
"p50": {
|
|
"max_pct_increase": 5.0,
|
|
"max_abs_increase_ms": 8.256410256410255
|
|
},
|
|
"p95": {
|
|
"max_pct_increase": 5.0,
|
|
"max_abs_increase_ms": 16.0703125
|
|
},
|
|
"p99": {
|
|
"max_pct_increase": 5.0,
|
|
"max_abs_increase_ms": 34.849999999999916
|
|
}
|
|
},
|
|
"span.consensus.ledger_close": {
|
|
"p50": {
|
|
"max_pct_increase": 5.0,
|
|
"max_abs_increase_ms": 0.34857142857142853
|
|
},
|
|
"p95": {
|
|
"max_pct_increase": 5.0,
|
|
"max_abs_increase_ms": 0.5067164179104476
|
|
},
|
|
"p99": {
|
|
"max_pct_increase": 5.0,
|
|
"max_abs_increase_ms": 4.068571428571428
|
|
}
|
|
},
|
|
"span.ledger.build": {
|
|
"p50": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 8.938775510204081
|
|
},
|
|
"p95": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 5.320408163265306
|
|
},
|
|
"p99": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 19.924999999999958
|
|
}
|
|
},
|
|
"span.ledger.validate": {
|
|
"p50": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 0.17212230215827337
|
|
}
|
|
},
|
|
"span.rpc.ws_message": {
|
|
"p50": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 0.3400905712090503
|
|
},
|
|
"p95": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 4.155692599620494
|
|
},
|
|
"p99": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 4.012163187855787
|
|
}
|
|
},
|
|
"span.tx.apply": {
|
|
"p50": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 4.208333333333333
|
|
},
|
|
"p95": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 5.372631578947368
|
|
},
|
|
"p99": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 5.040842105263158
|
|
}
|
|
},
|
|
"span.tx.process": {
|
|
"p50": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 0.6571893261786461
|
|
},
|
|
"p95": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 4.276013488700565
|
|
},
|
|
"p99": {
|
|
"max_pct_increase": 50.0,
|
|
"max_abs_increase_ms": 4.005485184848892
|
|
}
|
|
}
|
|
}
|
|
}
|