mirror of
https://github.com/XRPLF/rippled.git
synced 2026-09-27 23:38:08 +00:00
e4926f55be
The gate could not catch a regression on any sub-millisecond span. compare_to_baseline.py requires both the percentage and the absolute bound to breach, and every span shared one flat absolute bound of 10 ms (15 ms for p99) calibrated for a 5-25 ms band the spans do not occupy. Against the baseline captured on 2026-08-24, where 18 of the 28 quantiles gated at the time sat below 1 ms, that bound sat 1.15x to 2000x above the metric it guarded, so the AND never fired: a 100x regression injected into span.ledger.store.p95 reported 0 regressions and exit 0. Injecting a 10x regression into each key in turn was caught on only 5 of 28. Give every gated key its own absolute bound, equal to the distance from its baseline to hi_next, the edge above the top of the bucket the baseline sits in. The trip point is then exactly hi_next, so the gate fires only once the reading clears the bucket above the baseline's own. That is the property a multiple of the enclosing bucket width cannot provide: after the quantile crosses hi, the interpolation happens across the next bucket, which on this ladder is up to eight times wider, so no multiple of the enclosing width bounds the excursion. Measured with a model-free reachability test, a single bucket crossing can produce a false regression on 2 of 25 keys under the old flat bound and 0 of 25 under this rule. The smallest catchable regression is 2.02x to 9.43x per key. The job queue bound had the same shape of problem on three of its four keys (42x, 47x, 220x before). Defaults now sit at each ladder floor, leaving the percentage bound operative for a metric that somehow reaches them. Drop span.ledger.store from the gated surface. Its captured quantiles were 0.005, 0.0095 and 0.0099 ms, which is the ladder's 0.01 ms floor times the quantile: every sample lands under 10 us, so the reported value does not move even if each store slows from 2 us to 9 us. No bound can gate it. Presence is still asserted by expected_spans.json and the integration test, and the rate is still on the ledger-operations dashboard. Add check_regression_bounds.py, wired into the same workflow step as the bucket parity check. It fails when a bound is not the one its own baseline implies, when a gated key has no override, when the baseline and metric surface disagree, when the percentage bound would become operative, and when a baseline carries the ladder floor signature. This gate has now broken three times through the same drift between ladder, baseline and bounds, so documentation alone is not enough. compare_to_baseline.py is unchanged: its existing per-metric override mechanism already expresses all of this. A missing, unreadable or malformed input makes that check exit 1 naming the input, rather than reporting success without having checked anything; only a placeholder baseline, the documented bootstrap state, still exits 0. Its own tests cover both halves of that contract plus one case per rule, and run in the workflow before the check so a broken rule reads as a broken rule.
29 lines
3.0 KiB
JSON
29 lines
3.0 KiB
JSON
{
|
|
"_description": "Metric surface for the OTel-driven regression gate. Each entry names a metric, the quantiles to capture, and how to query Prometheus. The comparator compares current run against baseline-timings.json under these exact keys.",
|
|
"_key_format": "{category}.{name}.p{quantile} (e.g. span.tx.process.p99, job.transaction.queued.p95). Only the categories defined below are captured; there is no rpc_methods group, so no rpc.* key is produced or gated (FU-4).",
|
|
"_excluded_spans": "rpc.process is deliberately absent from spans.names. It is created only in ServerHandler::processRequest() on the HTTP/JSON-RPC path, which the workload load generators, being WebSocket-only, never reach, so its quantiles were captured as null every run and could never gate. (The harness shell scripts do issue a few HTTP JSON-RPC health polls, far too few to produce a meaningful quantile.) See baselines/README.md.",
|
|
"_excluded_ledger_store": "ledger.store is deliberately absent from spans.names too, for a different reason: it is below the ladder's resolution. The 2026-08-24 capture returned p50/p95/p99 of exactly 0.005/0.0095/0.0099 ms, which is 0.5/0.95/0.99 x the ladder's first edge of 0.01 ms — the signature of every sample landing in the first bucket, so the numbers are interpolation arithmetic on the bucket floor rather than latencies. That is physically plausible: LedgerMaster.cpp:463 wraps an in-memory ledgerHistory_.insert, which completes in single-digit microseconds. While all mass stays under 10 us the reported quantile cannot move materially, so NO absolute bound can gate it — every ledger.store slowing from 2 us to 9 us, 4.5x, leaves the reported value unchanged. Three keys that read as covered but cannot fire are worse than no keys (the same argument that excluded rpc.process), so they were removed rather than left in with a bound that looks derived. Restoring the key needs sub-10us edges on the collector's spanmetrics ladder (for example 0.001ms and 0.005ms) plus the matching entries in HistogramBuckets.h — that is the ladder's branch, not this file. ledger.store presence is still asserted by expected_spans.json and docker/telemetry/integration-test.sh, and its rate is still on the ledger-operations dashboard; only the latency gate drops it.",
|
|
"spans": {
|
|
"_query_template": "histogram_quantile({quantile}, sum by (le) (rate(span_duration_milliseconds_bucket{span_name=\"{name}\"}[{window}])))",
|
|
"_unit": "ms",
|
|
"_quantiles": [0.5, 0.95, 0.99],
|
|
"names": [
|
|
"rpc.ws_message",
|
|
"tx.process",
|
|
"tx.apply",
|
|
"ledger.build",
|
|
"ledger.validate",
|
|
"consensus.ledger_close",
|
|
"consensus.accept"
|
|
]
|
|
},
|
|
"job_queue": {
|
|
"_queued_template": "histogram_quantile({quantile}, sum by (le) (rate(job_queued_us_bucket{job_type=\"{name}\"}[{window}])))",
|
|
"_running_template": "histogram_quantile({quantile}, sum by (le) (rate(job_running_us_bucket{job_type=\"{name}\"}[{window}])))",
|
|
"_unit": "us",
|
|
"_quantiles": [0.95],
|
|
"_phases": ["queued", "running"],
|
|
"names": ["transaction", "acceptLedger"]
|
|
}
|
|
}
|