mirror of
https://github.com/XRPLF/rippled.git
synced 2026-08-21 22:30:57 +00:00
1282645289
The workload harness gates regressions on histogram_quantile over job_queued_us / job_running_us, so re-cutting the microsecond ladder changes what those queries return and the stored baselines no longer describe the same measurement. baseline-timings.json's job.acceptLedger.queued.p95 was 96.79us, which is 0.95 / 0.9926 x 100 -- the old 100us bucket edge scaled by the quantile, with 99.3% of samples beneath it. It was never a latency. Keeping it would make the gate LESS sensitive rather than more: a genuine regression from a real 40us to 90us would still sit under 96.79us + 50% and pass. Removes the four job.* entries and records why, including their values. The comparer reports a metric absent from the baseline as "new metric (not in baseline)" and skips it, so the span baselines stay live and gating continues for everything unaffected. is_placeholder() still returns False, so this does not disable the gate wholesale. Recapture the job.* numbers on a node running the re-cut ladder. Also corrects _bucket_note in regression-thresholds.json. It described the spanmetrics ladder as 15 edges starting at 1ms; the collector config has 20, including five sub-millisecond edges. The note's own reasoning was void too -- it justified the 10ms absolute span bound as "~2 low-end bucket widths", but the low-end bucket width is 0.01ms, not 5ms. The bound is kept and justified on the band where span quantiles actually sit, rather than on a derivation from a ladder that no longer exists.
27 lines
2.2 KiB
JSON
27 lines
2.2 KiB
JSON
{
|
|
"_description": "Per-metric regression thresholds. A metric regresses when current - baseline exceeds BOTH the percentage and absolute bounds (AND, not OR — this tolerates small-value noise). Defaults apply unless a per-metric override exists.",
|
|
"_bucket_note": "SpanMetrics latency histograms use explicit buckets [0.01,0.05,0.1,0.25,0.5,1,5,10,25,50,100,250,500]ms then [1,2,3,4,5,10,30]s (20 edges; docker/telemetry/otel-collector-config.yaml is the authoritative list). An earlier version of this note claimed 15 edges starting at 1ms and justified the 10ms absolute span bound as \"~2 low-end bucket widths\" — that derivation is void, because the sub-millisecond edges make the low-end bucket width 0.01ms, not 5ms. The 10ms bound is retained on its own merit: it is roughly two bucket widths in the 5-25ms band where most span quantiles actually sit, so it still absorbs single-bucket quantization jitter while catching multi-bucket regressions. Second-scale consensus spans have 2s/3s/4s boundaries, so their quantiles quantize to ~1s widths there. The job_queue running bound is widened similarly — per-ledger apply work scales with TxQ burst load. NOTE: the native job_queue histograms are microsecond-valued and their ladder was re-cut (floor 100us → 1us), so any job_queue baseline captured before that change is an interpolation artefact, not a latency.",
|
|
"defaults": {
|
|
"span": {
|
|
"p50": { "max_pct_increase": 50.0, "max_abs_increase_ms": 10.0 },
|
|
"p95": { "max_pct_increase": 50.0, "max_abs_increase_ms": 10.0 },
|
|
"p99": { "max_pct_increase": 50.0, "max_abs_increase_ms": 15.0 }
|
|
},
|
|
"job_queue": {
|
|
"p95": { "max_pct_increase": 50.0, "max_abs_increase_us": 20000.0 }
|
|
}
|
|
},
|
|
"overrides": {
|
|
"span.consensus.ledger_close": {
|
|
"p50": { "max_pct_increase": 5.0, "max_abs_increase_ms": 200.0 },
|
|
"p95": { "max_pct_increase": 5.0, "max_abs_increase_ms": 500.0 },
|
|
"p99": { "max_pct_increase": 5.0, "max_abs_increase_ms": 1000.0 }
|
|
},
|
|
"span.consensus.accept": {
|
|
"p50": { "max_pct_increase": 5.0, "max_abs_increase_ms": 200.0 },
|
|
"p95": { "max_pct_increase": 5.0, "max_abs_increase_ms": 500.0 },
|
|
"p99": { "max_pct_increase": 5.0, "max_abs_increase_ms": 1000.0 }
|
|
}
|
|
}
|
|
}
|