mirror of
https://github.com/XRPLF/rippled.git
synced 2026-09-27 15:28:03 +00:00
ffc4ee40fe
The harness killed and probed node directories named `node<N>`, but the
directories it creates are `validator-<N>` in run-full-validation.sh and
`bench-node-<N>` in benchmark.sh. Verified with pgrep against processes whose
command lines mimic the real ones: the pattern matched nothing either script
produces. Three consequences, all live:
- `--cleanup` deleted the workdir and left the xrpld processes running. They
are host processes, so the compose teardown does not reach them.
- The pre-run cleanup could not free the previous run's RPC, WS and peer
ports, which surfaces much later as a cluster that never reaches consensus.
- The startup crash fast-fail read a pid path that never exists, so its
`stopped > 0` branch was unreachable and a dead node waited out the full
120-attempt window.
Rather than patch four literals, derive every node path, kill pattern and log
glob from one NODE_PREFIX per script. The directory name is also the node's
identity: the collector's file_log receiver lifts that segment into
service.instance.id, so the directory and the [telemetry] service_instance_id
must agree. Deriving both from one value is what stops them drifting again.
Also in the same files, each confirmed by test rather than inspection:
- The collector readiness probe could never fail. curl -w '%{http_code}'
prints 000 on a refused connection and then exits non-zero, so the
`|| echo 000` inside the substitution appended a second 000 and the
"not ready" comparison never matched. Move the fallback outside.
- The generated config wrote [ips], the starter-list section. A loopback mesh
that must reach quorum is the [ips_fixed] case, which is what the variable,
the comment and the sibling cfg template already said.
- benchmark.sh returned exit 1 for a row it could not measure, though the
exit-code table reserves 1 for "every metric was measured and one breached".
Report 2 there instead.
- Five bc computations fell back to 0, which clears every threshold. The
guards beside them already fall back to the inconclusive token; these now
do too.
- A comment claimed a `|| guard` after a heredoc lands in the heredoc, and
that claim had removed a real guard from the config write. It does not: the
guard runs, and fires when cat fails.
- The EXIT trap was installed 88 lines before stop_workload was defined. If it
fired in that window, errexit aborted the handler on "command not found" and
the cluster reap never ran. Install it below both handlers.
- jq exits 5 on malformed JSON, outside this script's documented codes, so
read_metric now routes that through cannot_measure.
- --nodes and --duration were unvalidated, and --nodes 0 made the pid-count
guard compare 0 with 0 and pass, handing the sampler no pids at all.
- --cleanup now passes -v so the named tempo-data volume goes with it.
Otherwise the next run's Tempo still serves the previous run's traces and a
span assertion can be satisfied by them.
- Five messages reported an attempt count as seconds, though each attempt is
a sleep plus every node's probe.
The baselines README and the two regression JSON files had gone stale when the
baseline was refreshed to a three-run median: they described 20 gated keys and
five exclusions, against an actual 19 and six, and cited the superseded run,
date and commit. Re-derive every affected figure from the committed files. The
detection floors are recomputed (2.00x to 7.41x, so a 10x regression is now
caught on all 19 keys), the newly excluded span.ledger.build.p99 is documented,
and figures that no committed artifact can verify are either replaced with
derivable ones or labelled with their numerator.
No baseline value, threshold bound or derivation entry changes.
144 lines
19 KiB
JSON
144 lines
19 KiB
JSON
{
|
|
"_absolute_bound_derivation": "HOW EVERY max_abs_increase_* NUMBER BELOW WAS OBTAINED. Rule: locate the baseline value in the half-open bucket (lo, hi] of its own ladder, take hi_next = the next edge above hi, and set the bound to (hi_next - baseline). The trip point is therefore exactly hi_next: the gate fires only when the reported value EXCEEDS the top of the bucket above the baseline's own bucket. WHY THAT AND NOT A MULTIPLE OF THE BUCKET WIDTH: histogram_quantile returns a value interpolated inside whichever bucket the true quantile falls in, so a reading taken while the true quantile sits anywhere in the baseline's bucket OR anywhere in the one immediately above is at most hi_next and cannot fire. Firing requires the true quantile to have moved at least two buckets up. A multiple of the ENCLOSING width cannot deliver that, because once the quantile crosses hi the interpolation happens across the NEXT bucket, which on this ladder is up to 8x wider \u2014 (0.5,1] has width 0.5 and (1,5] has width 4 \u2014 so the reading's excursion is not bounded by any multiple of the enclosing width. Worked example: span.tx.process.p99 has baseline 0.9940ms in bucket (0.5, 1], hi_next = 5, so its bound is 4.0060ms and the gate fires only above 5ms. Bounds are stored as exact doubles rather than rounded figures so that rounding cannot break the guarantee and so check_regression_bounds.py can assert each one against the ladder to within a 1e-12 relative tolerance -- tight enough that a bound rounded for readability, such as 4.0060 for 4.006030232040599, is rejected; _derivation_table below shows the arithmetic for each one. Measured over the committed 2026-09-10 baseline this rule yields a detection floor of 2.00x to 7.41x of baseline, per key. WHAT THIS RULE DOES NOT COVER, AND THE ONE CHECK TO RUN BEFORE GATING ANY KEY: hi_next - baseline is derived from the LADDER, so it budgets for QUANTIZATION noise -- one bucket of interpolation headroom -- and for nothing else. It knows nothing about how much the metric itself moves between runs on identical code. Where run-to-run workload variance is the larger term the bound is simply the wrong size, and the gate reddens on a healthy run. So before adding a key here, capture it over several runs and check its OBSERVED MAXIMUM against its trip point (baseline + bound); gate it only if the observed maximum stays below that trip point with margin. Spread on its own proves nothing -- it is spread RELATIVE TO THE TRIP POINT that decides, and a baseline that lands at the LOW end of a metric's own range shrinks that trip point even though nothing about the metric changed. THREE KEYS FAILED THIS TEST ON THE 2026-08-26 BASELINE AND ARE EXCLUDED, all of them p50 (six keys are excluded in total; the other three are span.ledger.validate.p95, span.ledger.validate.p99 and span.ledger.build.p99): span.tx.apply.p50 (bound 0.0440ms, trips at 0.05ms, observed max 2.3378ms = 46.76x its trip point), span.ledger.build.p50 (bound 0.3849ms, trips at 0.5ms, observed max 2.3826ms = 4.77x) and span.consensus.ledger_close.p50 (bound 0.0613ms, trips at 0.1ms, observed max 0.2377ms = 2.38x). Their spreads across three runs are 391.8x, 20.7x and 6.1x. This is the general rule above being APPLIED, not a new exception: a key is gateable only when its run-to-run spread fits inside its bound, and these three do not. The evidence that settles it is span.tx.apply.p50's own history -- it read 0.7917ms in the 2026-08-24 baseline and 0.00597ms in the 2026-08-26 one, a 132x difference between two runs of the SAME workload. At the old value the identical rule produced a 4.21ms bound whose 5ms trip point absorbed the whole range; at the new one it produces 0.0440ms and cannot. Whether the gate functioned was therefore decided by where in its distribution the captured run happened to land, which is not a threshold needing tuning but a key that cannot be gated from a single-run baseline at all. Before the exclusion, replaying the two preceding CI runs 32862589645 and 32867433073 against the 2026-08-26 baseline reported exactly those three and nothing else on BOTH runs, and 32867433073 carries the same post-path-finding-removal workload as the baseline itself -- so the movement was metric variance, not a workload difference. After it, both runs replay clean. On the current 2026-09-10 baseline the 19 gated keys sit between 0.14 and 0.50 of baseline over trip point, the tightest being span.tx.process.p95 at 0.50. That ratio is derivable from this file and baseline-timings.json. A headroom figure against each key's OBSERVED MAXIMUM is not derivable here, because no per-run timings.json is committed -- so do not restate one without citing the run it came from. See _excluded_shape in regression-metrics.json for what the excluded keys have in common and for the multi-run-baseline work that would let them be gated again. A key that fails this test is not fixed by widening its bound: see excluded_keys in regression-metrics.json. WHAT THIS REPLACED, IN TWO GENERATIONS: (1) a single flat pair of bounds (10ms for span p50/p95, 15ms for span p99, 20000us for job_queue p95) justified as 'roughly two bucket widths in the 5-25ms band where most span quantiles actually sit'. The 2026-08-24 capture falsifies that premise \u2014 18 of the 28 quantiles gated at that time sat below 1ms \u2014 so the absolute bound sat 1.15x to 2000x above the metric it guarded and, because the rule is an AND, the percentage bound could never carry a regression on its own; a 10x regression injected into each key in turn was caught on only 5 of 28, and a 100x regression injected into span.ledger.store.p95 produced 0 regressions and exit 0. (2) a first correction to 2 \u00d7 the ENCLOSING bucket width, which caught 10x on 28 of 28 but placed the trip point INSIDE the adjacent bucket -- and so left a single-crossing false positive reachable -- on 21 of the 25 keys gated at the time, 4 of them tripping on a tail-mass shift under 1.5% of samples. That is the assumption this rule removes. RE-DERIVE THESE NUMBERS whenever baseline-timings.json is refreshed or either ladder changes: a refreshed baseline can land in a different bucket, which changes hi_next. .github/scripts/telemetry/check_regression_bounds.py enforces the rule in CI so a stale bound cannot survive a baseline refresh. LIMITATION \u2014 WHICH KEYS ARE ONLY WEAKLY GUARDED: the guarantee costs sensitivity wherever the ladder is coarse, and the detection floor is hi_next/baseline, so a baseline sitting just above an edge is guarded loosely. On the current 2026-09-10 baseline the six weakest keys are span.consensus.ledger_close.p95 (baseline 0.6750ms, fires at 5ms, 7.41x), span.consensus.accept.p50 (1.4364ms, 10ms, 6.96x), job.acceptLedger.running.p95 (15967.74us, 100000us, 6.26x), span.rpc.ws_message.p95 (0.8122ms, 5ms, 6.16x), span.rpc.ws_message.p99 (0.9873ms, 5ms, 5.06x) and span.tx.process.p99 (0.9940ms, 5ms, 5.03x). Four of the six are limited by the 1ms\u21925ms step; the other two by 5ms\u219210ms (span.consensus.accept.p50) and 25000us\u2192100000us (job.acceptLedger.running.p95). None of the six fires on any observed run, so all six stay gated, but the weak floors are recorded here so they are visible rather than surprising. Because every floor is now under 10x, a 10x regression is caught on all 19 gated keys -- that is derived from the floors, not sampled. On the 2026-08-26 baseline job.acceptLedger.running.p95 had a 16.28x floor and was the one key 10x missed; its baseline rose 6142.86us to 15967.74us while hi_next stayed at 100000us, which pulled its floor to 6.26x. Sensitivity therefore moves with each refresh even when no code changes, so re-derive these floors whenever the baseline is refreshed. The fix is a 2ms edge (and ideally 3ms) in the collector's spanmetrics ladder plus the matching edges in kMillisecondBuckets, and 2000us plus 50000us edges in kMicrosecondBuckets \u2014 that work belongs to the branch that owns the ladders, not here. Until then do not read these keys as guarded. span.ledger.store is absent from the overrides below because it is excluded from the gated surface entirely: its quantiles are the ladder floor times the quantile, so no bound can gate it. See _excluded_ledger_store in regression-metrics.json. REFRESHED 2026-09-10 from the median of CI runs 34495527952, 34505215266 and 34507425933, the first three runs with the account-funding race fixed. The 2026-08-26 baseline predated that fix, so the phases that lost their traffic captured artificially low ledger and transaction timings. Applying the observed-maximum test to the refreshed numbers leaves 19 of 20 keys between 0.17 and 0.76 of their trip points, and disqualifies span.ledger.build.p99 -- see excluded_keys in regression-metrics.json. span.tx.process.p95 is the tightest survivor at 0.76 and is the key to re-measure first if the gate reddens again.",
|
|
"_bucket_note": "SpanMetrics latency histograms use explicit buckets [0.01,0.05,0.1,0.25,0.5,1,5,10,25,50,100,250,500]ms then [1,2,3,4,5,10,30]s (20 edges; docker/telemetry/otel-collector-config.yaml is the authoritative list). Second-scale consensus spans have 2s/3s/4s boundaries, so their quantiles quantize to ~1s widths there \u2014 the ladder is NOT uniformly 2x-or-coarser, which matters for _percentage_bound_note. The native job_queue histograms are microsecond-valued on the ladder [1,2,5,10,25,50,100,250,500,1000,5000,25000,100000,500000]us then [1,5,10,30,60]s (19 edges; include/xrpl/telemetry/HistogramBuckets.h is authoritative). NOTE: BOTH ladders were re-cut, and a baseline captured before its own ladder changed is an interpolation artefact, not a latency. The job_queue floor moved 100us \u2192 1us. The span floor is 0.01ms; a span baseline captured against a 1ms floor is void below 1ms \u2014 a p95 reading 0.95ms there is 0.95 \u00d7 that 1ms first edge, not a measurement. Do not assume a surviving span baseline is unaffected by ladder work: every span quantile below 1ms is affected. Only the band from 1ms to 1s is safe: those edges are byte-identical across the two ladders. The re-cut also ADDED edges above 1s (2s/3s/4s/10s/30s), so a span whose quantiles land in the second-scale range \u2014 consensus.round ~3.9s, consensus.establish ~1.9s, the ledger.acquire tail \u2014 is distorted just as much, and any pre-2026-08-04 baseline for it is equally void. Do not read this note as licensing a stale second-scale baseline.",
|
|
"_defaults_note": "A MISSING OVERRIDE IS DETECTED BY CI, NOT BY THESE DEFAULTS. .github/scripts/telemetry/check_regression_bounds.py fails the build at lint time, naming the key and the exact value its bound should have, before the workload ever runs. That is the mechanism; the defaults below are only a runtime backstop for the case where that check is bypassed. The defaults carry the FLOOR of each ladder as their absolute bound \u2014 0.01ms for spans, 1us for job_queue \u2014 deliberately too small to bind for any real metric, which leaves max_pct_increase (50%) as the operative bound on this path. Measured: a metric with no override and a baseline of 3900ms passes at +49% and fires at +51%; a job metric with a baseline of 5000us behaves the same. The backstop is honestly imperfect and should not be oversold. At 50% relative it CAN false-fire: a metric whose baseline is 1.06ms inside the 4ms-wide (1,5] bucket fires on a single-bucket-width move (measured: 1.06 \u2192 5.06ms, +377%, regressed). That false fire is NOT to be read as 'the intended signal that the override is missing' \u2014 CI prints REGRESSION and a reader cannot tell it from a real one, and rejecting a tighter alternative for exactly that cries-wolf risk while shipping it here would be inconsistent. The check is what makes the signal legible. The backstop is kept only because a metric silently not gated at all is the worse of the two failures.",
|
|
"_derivation_table": {
|
|
"_format": "override key: <quantile> <baseline> in <bucket> -> hi_next - baseline = <bound>",
|
|
"job.acceptLedger.queued": "p95 95.58333333333331 in (50,100] -> hi_next 250 - baseline = 154.41666666666669",
|
|
"job.acceptLedger.running": "p95 15967.741935483866 in (5000,25000] -> hi_next 100000 - baseline = 84032.25806451614",
|
|
"job.transaction.queued": "p95 403.74177631578937 in (250,500] -> hi_next 1000 - baseline = 596.2582236842106",
|
|
"job.transaction.running": "p95 376.7512077294688 in (250,500] -> hi_next 1000 - baseline = 623.2487922705312",
|
|
"span.consensus.accept": "p50 1.4363636363636365 in (1,5] -> hi_next 10 - baseline = 8.563636363636363 | p95 8.811111111111114 in (5,10] -> hi_next 25 - baseline = 16.188888888888886 | p99 20.928571428571445 in (10,25] -> hi_next 50 - baseline = 29.071428571428555",
|
|
"span.consensus.ledger_close": "p95 0.6749999999999998 in (0.5,1] -> hi_next 5 - baseline = 4.325 | p99 3.049999999999991 in (1,5] -> hi_next 10 - baseline = 6.950000000000009",
|
|
"span.ledger.build": "p95 4.7714285714285705 in (1,5] -> hi_next 10 - baseline = 5.2285714285714295",
|
|
"span.ledger.validate": "p50 0.059761904761904766 in (0.05,0.1] -> hi_next 0.25 - baseline = 0.19023809523809523",
|
|
"span.rpc.ws_message": "p50 0.16282758747645082 in (0.1,0.25] -> hi_next 0.5 - baseline = 0.3371724125235492 | p95 0.8122454466253443 in (0.5,1] -> hi_next 5 - baseline = 4.187754553374655 | p99 0.9872660098522166 in (0.5,1] -> hi_next 5 - baseline = 4.012733990147783",
|
|
"span.tx.apply": "p95 4.639716312056738 in (1,5] -> hi_next 10 - baseline = 5.360283687943262 | p99 4.9733333333333345 in (1,5] -> hi_next 10 - baseline = 5.0266666666666655",
|
|
"span.tx.process": "p50 0.21390674968918655 in (0.1,0.25] -> hi_next 0.5 - baseline = 0.28609325031081345 | p95 0.49949970576841685 in (0.25,0.5] -> hi_next 1 - baseline = 0.5005002942315832 | p99 0.9939697679594013 in (0.5,1] -> hi_next 5 - baseline = 4.006030232040599"
|
|
},
|
|
"_description": "Per-metric regression thresholds. A metric regresses when current - baseline exceeds BOTH the percentage and absolute bounds (AND, not OR \u2014 this tolerates small-value noise). Defaults apply unless a per-metric override exists.",
|
|
"_percentage_bound_note": "For every key gated today the absolute bound is the binding half of the AND and the percentage bound never decides the outcome: measured on the 2026-09-10 baseline, (bound / baseline) ranges from 100.2% (span.tx.process.p95) to 640.7% (span.consensus.ledger_close.p95), all above the 50% and 5% percentage bounds configured here, and the minimum trip multiple of all 19 gated keys is set by the absolute bound. Re-measure these two extremes on every baseline refresh: both changed identity in the 2026-09-10 refresh. THIS IS NOT A GENERAL GUARANTEE. Do not reason from 'every step of both ladders is at least a factor of 2' -- that premise is false. The span ladder breaks it three times at the top: 2s->3s is 1.5x, 3s->4s is 1.33x, 4s->5s is 1.25x, so second-scale consensus quantiles quantize to ~1s widths there. Because the bound is (hi_next - baseline), a baseline between about 2667ms and 3000ms, or between about 3334ms and 4000ms, gets an absolute bound worth less than 50% of itself and the PERCENTAGE bound becomes the operative one -- at which point the metric fires on a 50% move that is smaller than one bucket width, and the single-crossing guarantee in _absolute_bound_derivation is lost. That band is not hypothetical: the collector config names consensus.round (~3.9s) as a reason those edges exist, and 3900ms sits in the second sub-band with an absolute bound of 5000 - 3900 = 1100, only 28.2% of baseline. Whoever gates a key whose baseline lands in either sub-band MUST lower its max_pct_increase below (bound / baseline) for that key, or state explicitly that the metric is percentage-gated and the bucket guarantee does not hold for it. check_regression_bounds.py enforces this as rule D so the trap cannot be walked into silently. The percentage entries are required and still meaningful regardless: compare_to_baseline.py treats a missing max_pct_increase as 'no threshold configured' and would stop gating the metric entirely; they record the intended relative tolerance (consensus spans 5%, everything else 50%); and they are the operative bound on the defaults path (see _defaults_note).",
|
|
"defaults": {
|
|
"job_queue": {
|
|
"p95": {
|
|
"max_abs_increase_us": 1.0,
|
|
"max_pct_increase": 50.0
|
|
}
|
|
},
|
|
"span": {
|
|
"p50": {
|
|
"max_abs_increase_ms": 0.01,
|
|
"max_pct_increase": 50.0
|
|
},
|
|
"p95": {
|
|
"max_abs_increase_ms": 0.01,
|
|
"max_pct_increase": 50.0
|
|
},
|
|
"p99": {
|
|
"max_abs_increase_ms": 0.01,
|
|
"max_pct_increase": 50.0
|
|
}
|
|
}
|
|
},
|
|
"overrides": {
|
|
"job.acceptLedger.queued": {
|
|
"p95": {
|
|
"max_abs_increase_us": 154.41666666666669,
|
|
"max_pct_increase": 50.0
|
|
}
|
|
},
|
|
"job.acceptLedger.running": {
|
|
"p95": {
|
|
"max_abs_increase_us": 84032.25806451614,
|
|
"max_pct_increase": 50.0
|
|
}
|
|
},
|
|
"job.transaction.queued": {
|
|
"p95": {
|
|
"max_abs_increase_us": 596.2582236842106,
|
|
"max_pct_increase": 50.0
|
|
}
|
|
},
|
|
"job.transaction.running": {
|
|
"p95": {
|
|
"max_abs_increase_us": 623.2487922705312,
|
|
"max_pct_increase": 50.0
|
|
}
|
|
},
|
|
"span.consensus.accept": {
|
|
"p50": {
|
|
"max_abs_increase_ms": 8.563636363636363,
|
|
"max_pct_increase": 5.0
|
|
},
|
|
"p95": {
|
|
"max_abs_increase_ms": 16.188888888888886,
|
|
"max_pct_increase": 5.0
|
|
},
|
|
"p99": {
|
|
"max_abs_increase_ms": 29.071428571428555,
|
|
"max_pct_increase": 5.0
|
|
}
|
|
},
|
|
"span.consensus.ledger_close": {
|
|
"p95": {
|
|
"max_abs_increase_ms": 4.325,
|
|
"max_pct_increase": 5.0
|
|
},
|
|
"p99": {
|
|
"max_abs_increase_ms": 6.950000000000009,
|
|
"max_pct_increase": 5.0
|
|
}
|
|
},
|
|
"span.ledger.build": {
|
|
"p95": {
|
|
"max_abs_increase_ms": 5.2285714285714295,
|
|
"max_pct_increase": 50.0
|
|
}
|
|
},
|
|
"span.ledger.validate": {
|
|
"p50": {
|
|
"max_abs_increase_ms": 0.19023809523809523,
|
|
"max_pct_increase": 50.0
|
|
}
|
|
},
|
|
"span.rpc.ws_message": {
|
|
"p50": {
|
|
"max_abs_increase_ms": 0.3371724125235492,
|
|
"max_pct_increase": 50.0
|
|
},
|
|
"p95": {
|
|
"max_abs_increase_ms": 4.187754553374655,
|
|
"max_pct_increase": 50.0
|
|
},
|
|
"p99": {
|
|
"max_abs_increase_ms": 4.012733990147783,
|
|
"max_pct_increase": 50.0
|
|
}
|
|
},
|
|
"span.tx.apply": {
|
|
"p95": {
|
|
"max_abs_increase_ms": 5.360283687943262,
|
|
"max_pct_increase": 50.0
|
|
},
|
|
"p99": {
|
|
"max_abs_increase_ms": 5.0266666666666655,
|
|
"max_pct_increase": 50.0
|
|
}
|
|
},
|
|
"span.tx.process": {
|
|
"p50": {
|
|
"max_abs_increase_ms": 0.28609325031081345,
|
|
"max_pct_increase": 50.0
|
|
},
|
|
"p95": {
|
|
"max_abs_increase_ms": 0.5005002942315832,
|
|
"max_pct_increase": 50.0
|
|
},
|
|
"p99": {
|
|
"max_abs_increase_ms": 4.006030232040599,
|
|
"max_pct_increase": 50.0
|
|
}
|
|
}
|
|
}
|
|
}
|