mirror of
https://github.com/XRPLF/rippled.git
synced 2026-07-23 15:10:34 +00:00
8dd64d4dcd
P95 of second-scale spans was a meaningless interpolation. The spanmetrics histogram topped out at [.. 1s, 5s], so consensus.round (~3.9s) and consensus.establish (~1.9s) all fell into one 1s-5s bucket and histogram_quantile interpolated linearly across that 4s-wide gap — the "Build vs Close" / "Ledger Close Duration" panels' P95 read ~4800ms purely as an artifact (verified: sum/count avg = 3824ms). ledger.acquire was worse: ~17% of samples exceeded the 5s ceiling, so its p95/p99 were unmeasurable. Add 2s, 3s, 4s (resolve the 1-5s pile-up) and 10s, 30s (give the ledger.acquire catch-up tail a measurable home). All ten existing boundaries are preserved and the list stays strictly ascending (the connector binary-searches buckets and silently misbuckets otherwise). Pin unit=ms so a future collector default-unit flip can't rename the metric to _seconds. Buckets chosen from the live mainnet distribution, not guessed. Native beast::insight histograms (ms-scale RPC/IO timers in Telemetry.cpp) are 100% under 5s, so they keep the original buckets — this is collector-only. Applies on collector restart (cumulative series reset once, handled by rate()). Runbook and regression-threshold bucket notes updated to match. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
27 lines
1.8 KiB
JSON
27 lines
1.8 KiB
JSON
{
|
|
"_description": "Per-metric regression thresholds. A metric regresses when current - baseline exceeds BOTH the percentage and absolute bounds (AND, not OR — this tolerates small-value noise). Defaults apply unless a per-metric override exists.",
|
|
"_bucket_note": "SpanMetrics latency histograms use explicit buckets [1,5,10,25,50,100,250,500,1000,2000,3000,4000,5000,10000,30000]ms. A quantile sitting near a low-end boundary can jump a full bucket (e.g. 1ms->5ms) between runs with no real change, so absolute span bounds are set to ~2 low-end bucket widths (10ms) to tolerate that quantization noise while still catching genuine multi-bucket regressions. Second-scale consensus spans now have 2s/3s/4s boundaries (previously all fell in one 1s-5s bucket); their quantiles quantize to ~1s widths there. The job_queue running bound is widened similarly — per-ledger apply work scales with TxQ burst load.",
|
|
"defaults": {
|
|
"span": {
|
|
"p50": { "max_pct_increase": 50.0, "max_abs_increase_ms": 10.0 },
|
|
"p95": { "max_pct_increase": 50.0, "max_abs_increase_ms": 10.0 },
|
|
"p99": { "max_pct_increase": 50.0, "max_abs_increase_ms": 15.0 }
|
|
},
|
|
"job_queue": {
|
|
"p95": { "max_pct_increase": 50.0, "max_abs_increase_us": 20000.0 }
|
|
}
|
|
},
|
|
"overrides": {
|
|
"span.consensus.ledger_close": {
|
|
"p50": { "max_pct_increase": 5.0, "max_abs_increase_ms": 200.0 },
|
|
"p95": { "max_pct_increase": 5.0, "max_abs_increase_ms": 500.0 },
|
|
"p99": { "max_pct_increase": 5.0, "max_abs_increase_ms": 1000.0 }
|
|
},
|
|
"span.consensus.accept": {
|
|
"p50": { "max_pct_increase": 5.0, "max_abs_increase_ms": 200.0 },
|
|
"p95": { "max_pct_increase": 5.0, "max_abs_increase_ms": 500.0 },
|
|
"p99": { "max_pct_increase": 5.0, "max_abs_increase_ms": 1000.0 }
|
|
}
|
|
}
|
|
}
|