mirror of
https://github.com/XRPLF/rippled.git
synced 2026-09-26 23:19:07 +00:00
docs(telemetry): stop the harness contract narrating its own revisions
Notes across the workload contract described earlier versions of themselves, or cited commits that only exist inside this chain. A squash merge publishes none of it, so each reference resolves nowhere. Notes that described their own earlier text: - expected_spans.json: 'this note previously concluded', 'this note previously said', 'Un-skipped 2026-08-26', 'the reason had simply gone stale for two weeks' and 'the claim this entry carried' are replaced by the standing reason each entry holds. The wildcard pairs now say the validator globs the child via _span_name_matches(), and state the literal-collapse failure as what a different validator WOULD do rather than as history. - regression-thresholds.json: 'an earlier version of this note wrongly claimed', 'the earlier version oversold it' and 'an earlier note called that' become the cautions themselves -- do not reason from 'every ladder step is at least 2x', do not oversell the backstop, do not read a false fire as a missing override. - test_check_regression_bounds.py: the docstring gives the reason a literal is wrong here, not the story of two tests that once hard-coded one. Baseline-refresh history rewritten as measurement: - README.md, baselines/README.md, telemetry-runbook.md and regression-metrics.json no longer attribute threshold moves to 'the 2026-08-26 refresh'. The evidence is kept as measurement -- span.tx.apply.p50 has read 0.7917 ms and 0.00597 ms on the same workload, 132x apart; job.acceptLedger.running.p95 has measured a 5.74x floor on one baseline and 16.28x on another -- which is what supports the claim that a single-run baseline cannot bound these keys. Two chain-only commit ids removed,d059f21bf3and3860c93db2. Neither is reachable from develop, so both cease to exist on merge; the second is chain bookkeeping. The facts they were cited for (the validator globs wildcards; the span ladder's floor is 0.01 ms) are stated directly instead. Capture provenance is deliberately kept: baseline-timings.json 'captured_at', the 2026-08-26 baseline heading, and the 2026-08-24 figures cited as data. Documentation, JSON note strings and one docstring only, no behaviour change.
This commit is contained in:
@@ -93,8 +93,7 @@ class CheckerCase(unittest.TestCase):
|
||||
|
||||
Read from the scratch copies of the real inputs rather than written as
|
||||
literals, because a literal here is a copy of one particular baseline:
|
||||
two of these tests previously hard-coded values from the 2026-08-24
|
||||
capture and both broke the moment the baseline was refreshed, which is
|
||||
a hard-coded figure breaks the moment the baseline is refreshed, which is
|
||||
the very drift check_regression_bounds.py exists to catch. Deriving the
|
||||
figure keeps the assertion pinned to the rule instead of to a snapshot.
|
||||
"""
|
||||
|
||||
@@ -298,19 +298,19 @@ Per-run tuning:
|
||||
variance is larger than it cannot be gated at all. **Five keys are excluded**
|
||||
for that reason: `span.ledger.validate.p95` and `.p99`, plus
|
||||
`span.tx.apply.p50`, `span.ledger.build.p50` and
|
||||
`span.consensus.ledger_close.p50` as of the 2026-08-26 refresh. Each carries
|
||||
`span.consensus.ledger_close.p50`. Each carries
|
||||
its measurements in `excluded_keys` in `regression-metrics.json`. Check a key's
|
||||
observed maximum across runs against `baseline + bound` before gating it;
|
||||
widening the bound is not the fix, and neither is re-baselining until a run
|
||||
lands favourably. See `baselines/README.md`.
|
||||
- A refresh moves sensitivity in **both** directions, because the trip point is
|
||||
derived from the baseline, and a single run carries no information about
|
||||
spread. The 2026-08-26 refresh loosened `job.acceptLedger.running.p95` from a
|
||||
5.74x detection floor to 16.28x (it does not fire, so it stays gated) and cut
|
||||
the three `p50` keys above from a bound that had absorbed their spread to one
|
||||
that could not — `span.tx.apply.p50` read 0.7917 ms in the previous baseline
|
||||
and 0.00597 ms in this one, a 132x move on the same workload, taking its bound
|
||||
from 4.21 ms to 0.0440 ms. Gating those keys again needs a **multi-run
|
||||
spread. `job.acceptLedger.running.p95` has been measured with a 5.74x
|
||||
detection floor on one baseline and 16.28x on another (it does not fire, so it
|
||||
stays gated), and the three `p50` keys above have been measured both inside and
|
||||
outside a bound that absorbs their spread — `span.tx.apply.p50` has read
|
||||
0.7917 ms and 0.00597 ms on the same workload, 132x apart, which moves its
|
||||
bound between 4.21 ms and 0.0440 ms. Gating those keys needs a **multi-run
|
||||
baseline** (or a spread measurement captured beside it), not a new threshold.
|
||||
All of it is measured in `baselines/README.md`; re-check after every refresh.
|
||||
|
||||
|
||||
@@ -35,7 +35,7 @@ at `6a82fc6f37` that predated two workload changes — the removal of the refuse
|
||||
load (`59a0595a6e`) and everything after it — so its numbers described a workload the harness no
|
||||
longer runs. The entries before that, captured on 2026-06-05, were voided into a placeholder: they
|
||||
predated the
|
||||
spanmetrics ladder re-cut of 2026-08-04 (`3860c93db2`), which made every sub-millisecond quantile
|
||||
spanmetrics ladder's 1 ms floor, which made every sub-millisecond quantile
|
||||
in that capture bucket-edge arithmetic rather than a latency (a p95 of `0.95` ms is `0.95 × 1 ms`).
|
||||
Because the comparator only flags a metric when the current value _exceeds_ the baseline, a
|
||||
stale-high baseline passes everything silently, so the entries had to be dropped rather than left
|
||||
@@ -206,7 +206,7 @@ proves nothing; it is spread **relative to the trip point** that decides. And be
|
||||
point is derived from the baseline, a baseline that lands at the **low end** of a metric's own
|
||||
range shrinks that trip point without anything about the metric having changed.
|
||||
|
||||
That is what the 2026-08-26 refresh did to three `p50` keys, and **all three are now excluded** —
|
||||
That is what happened to three `p50` keys on this baseline, and **all three are excluded** —
|
||||
this rule being applied, not a new exception. Measured across the three CI runs `32862589645`,
|
||||
`32867433073` and `32964262700` (the last of which is this baseline):
|
||||
|
||||
|
||||
@@ -25,7 +25,7 @@
|
||||
"required_attributes": [],
|
||||
"config_flag": "trace_rpc",
|
||||
"optional": true,
|
||||
"note": "HTTP-only. Created solely in ServerHandler::processRequest() (ServerHandler.cpp:705), which is reached only from processSession(Session, coro) (ServerHandler.cpp:646) — the HTTP/JSON-RPC path that roots rpc.http_request at ServerHandler.cpp:640-641. The WebSocket path (processSession(WSSession, coro, jv), ServerHandler.cpp:467) never calls processRequest, so this span never appears under a WebSocket request. It does still appear under this harness, 5 traces on a normal run, because run-full-validation.sh polls each node over HTTP with curl (:449, :502) and those requests take the HTTP path. Corrected 2026-08-26; this note previously concluded the span cannot appear at all, which sent a reader looking for a way to make the harness speak HTTP that it already speaks."
|
||||
"note": "HTTP-only. Created solely in ServerHandler::processRequest() (ServerHandler.cpp:705), which is reached only from processSession(Session, coro) (ServerHandler.cpp:646) — the HTTP/JSON-RPC path that roots rpc.http_request at ServerHandler.cpp:640-641. The WebSocket path (processSession(WSSession, coro, jv), ServerHandler.cpp:467) never calls processRequest, so this span never appears under a WebSocket request. It does still appear under this harness, 5 traces on a normal run, because run-full-validation.sh polls each node over HTTP with curl (:449, :502) and those requests take the HTTP path. Note that the harness does speak HTTP: a reader concluding this span is unreachable here would go looking for a way to add HTTP traffic that already exists."
|
||||
},
|
||||
{
|
||||
"name": "rpc.command.*",
|
||||
@@ -42,7 +42,7 @@
|
||||
"required_attributes": ["request_payload_size"],
|
||||
"config_flag": "trace_rpc",
|
||||
"optional": true,
|
||||
"note": "HTTP/JSON-RPC root span. It DOES fire under this harness, 5 traces on a normal run -- one per node -- even though the load generator is WebSocket-only, because run-full-validation.sh polls each node's HTTP port with curl for readiness and validated-ledger progress (:449, :502). Corrected 2026-08-26; this note previously said it does not fire. Kept optional rather than promoted to required because those polls are harness scaffolding rather than workload: a future change to how the script waits for a node could remove them without anything being wrong with the node."
|
||||
"note": "HTTP/JSON-RPC root span. It DOES fire under this harness, 5 traces on a normal run -- one per node -- even though the load generator is WebSocket-only, because run-full-validation.sh polls each node's HTTP port with curl for readiness and validated-ledger progress (:449, :502). Kept optional rather than required because those polls are harness scaffolding rather than workload: a future change to how the script waits for a node could remove them without anything being wrong with the node."
|
||||
},
|
||||
{
|
||||
"name": "tx.process",
|
||||
@@ -394,12 +394,12 @@
|
||||
{
|
||||
"parent": "rpc.ws_message",
|
||||
"child": "rpc.command.*",
|
||||
"description": "WebSocket message contains the per-command span — the real relationship on the harness WS path (rpc::doCommand at RPCHandler.cpp:271 creates an ambient child of the rpc.ws_message scope inside the same coroutine). Un-skipped 2026-08-26. The skip existed because _validate_parent_child() used to collapse the wildcard to one literal name via child_name.replace(\"*\", \"server_info\"), which made the check depend on which command the sampled traces happened to carry -- server_info is 25/100 of rpc_load_generator.py's DEFAULT_WEIGHTS, so a healthy run could sample three non-server_info traces and fail. That code no longer exists: d059f21bf3 replaced it with _span_name_matches(), which globs via fnmatch.fnmatchcase, and the check's own comment now reads \"globs for wildcard contracts\". Any rpc.command.<anything> under the parent therefore satisfies the contract and the command mix no longer matters. The reason had simply gone stale for two weeks."
|
||||
"description": "WebSocket message contains the per-command span — the real relationship on the harness WS path (rpc::doCommand at RPCHandler.cpp:271 creates an ambient child of the rpc.ws_message scope inside the same coroutine). Not skipped, because the validator globs the wildcard child: _span_name_matches() matches via fnmatch.fnmatchcase, so any rpc.command.<anything> under the parent satisfies the contract and the command mix does not matter. A validator that instead collapsed the wildcard to one literal name -- child_name.replace(\"*\", \"server_info\") -- would make this check depend on which command the sampled traces happened to carry: server_info is 25/100 of rpc_load_generator.py's DEFAULT_WEIGHTS, so a healthy run could sample three non-server_info traces and fail."
|
||||
},
|
||||
{
|
||||
"parent": "rpc.process",
|
||||
"child": "rpc.command.*",
|
||||
"description": "Processing span contains the per-command span, on the HTTP/JSON-RPC path. Un-skipped 2026-08-26 for the same reason as the WebSocket pair above: the validator globs a wildcard child via _span_name_matches() and has done since d059f21bf3, so the 'resolves the wildcard to one literal probe' claim this entry carried was describing code deleted two weeks earlier. That claim was not inherited here by accident -- it was copied from the stale WS entry while correcting a DIFFERENT error in this same reason, without checking it. Both ends emit: rpc.process reports 5 traces on a normal run, not from the WebSocket-only load generator but because run-full-validation.sh polls each node's HTTP port with curl for readiness and validated-ledger progress (:449, :502), and every such request runs a command."
|
||||
"description": "Processing span contains the per-command span, on the HTTP/JSON-RPC path. Not skipped, for the same reason as the WebSocket pair above: the validator globs a wildcard child via _span_name_matches(), so no single literal command has to be sampled. Both ends emit: rpc.process reports 5 traces on a normal run, not from the WebSocket-only load generator but because run-full-validation.sh polls each node's HTTP port with curl for readiness and validated-ledger progress (:449, :502), and every such request runs a command."
|
||||
},
|
||||
{
|
||||
"parent": "ledger.build",
|
||||
@@ -429,7 +429,7 @@
|
||||
"description": "The RPC command span contains the path-finding request span.",
|
||||
"skip": true,
|
||||
"skip_reason": "Real relationship, and the one skip here caused by a WILDCARD PARENT rather than by a missing span. _validate_parent_child builds its Tempo query as name=\"<parent>\" with the contract string inserted literally (validate_telemetry.py:801), so a parent of rpc.command.* searches for a span literally named that and finds nothing. Note the asymmetry: the ancestry check globs both sides through _span_name_matches, which is why rpc.ws_message -> rpc.command.* is asserted, but the Tempo query that selects the candidate traces is literal on the parent, so no trace is ever fetched to run it on. Asserting this needs the parent query to accept a glob -- a TraceQL name=~ regex, or resolving the glob to the concrete names Tempo reports first. Independently of that, both ends are absent today anyway: the harness issues no path-finding RPC, see the pathfind.compute entry above.",
|
||||
"added": "2026-08-26 to close the declared-but-unlisted gap"
|
||||
"added": "Closes the declared-but-unlisted gap"
|
||||
},
|
||||
{
|
||||
"parent": "pathfind.compute",
|
||||
@@ -437,7 +437,7 @@
|
||||
"description": "The path computation contains the discovery pass.",
|
||||
"skip": true,
|
||||
"skip_reason": "Real relationship with BOTH ends absent, for the reason given in full on the pathfind.compute entry above: pathfinding is disabled on every harness node because Config.cpp:725-726 zeroes pathSearchMax whenever a [validation_seed] section is present, run-full-validation.sh writes one for all five nodes (:308) with no [path_search_max] override, so doRipplePathFind returns rpcNOT_SUPPORTED before any PathRequest is constructed -- and the harness sends no path-finding RPC at all either. Asserting this needs both blockers lifted, which is a workload and node-config change rather than a harness one. Listed here so that the pathfinding family is fully accounted for rather than partly silent.",
|
||||
"added": "2026-08-26 to close the declared-but-unlisted gap"
|
||||
"added": "Closes the declared-but-unlisted gap"
|
||||
},
|
||||
|
||||
{
|
||||
@@ -460,12 +460,12 @@
|
||||
"child": "txq.batch_clear",
|
||||
"description": "Queue admission contains the batch-clear pass that drops an account's superseded queued transactions.",
|
||||
"skip": true,
|
||||
"skip_reason": "Real relationship, still skipped but for ONE reason now rather than two. The child never fires at all under this workload -- the run reports \"span.txq.batch_clear: optional span not emitted under this workload\" -- because it is created in TxQ::tryClearAccountQueueUpThruTx (TxQ.cpp:550), which needs one account holding several queued transactions AND an arriving transaction that supersedes the whole batch. Nothing in txq-burst arranges that shape. The second reason this entry used to carry, that newest-N parent sampling would miss it anyway, no longer applies: the hierarchy check now queries Tempo for traces containing both parent and child. So this is now purely a workload gap, and un-skipping it needs the workload to produce a supersedable batch -- nothing further from the validator."
|
||||
"skip_reason": "Real relationship, skipped for one reason: the child never fires at all under this workload -- the run reports \"span.txq.batch_clear: optional span not emitted under this workload\" -- because it is created in TxQ::tryClearAccountQueueUpThruTx (TxQ.cpp:550), which needs one account holding several queued transactions AND an arriving transaction that supersedes the whole batch. Nothing in txq-burst arranges that shape. Sampling is not a second reason here: the hierarchy check queries Tempo for traces containing both parent and child, so it would find the pair wherever it occurred. This is purely a workload gap, and un-skipping it needs the workload to produce a supersedable batch -- nothing further from the validator."
|
||||
},
|
||||
{
|
||||
"parent": "txq.accept",
|
||||
"child": "txq.accept_tx",
|
||||
"description": "The queue's accept pass contains the per-transaction accept span. Un-skipped once the hierarchy check stopped sampling only the newest parent traces. The child is created inside the loop over queued transactions and behind `if (feeLevelPaid >= requiredFeeLevel)` (TxQ.cpp:1530), so it exists only for a close whose queue held a fee-clearing transaction, while the parent fires on every close (:1499) -- which is precisely the shape newest-N sampling gets wrong. The check now asks Tempo for traces containing both, so co-occurrence is found wherever it happened rather than only in the three most recent closes."
|
||||
"description": "The queue's accept pass contains the per-transaction accept span. Not skipped, because the hierarchy check asks Tempo for traces containing both spans rather than sampling the newest parent traces. That matters here: the child is created inside the loop over queued transactions and behind `if (feeLevelPaid >= requiredFeeLevel)` (TxQ.cpp:1530), so it exists only for a close whose queue held a fee-clearing transaction, while the parent fires on every close (:1499) -- exactly the shape newest-N sampling gets wrong. Searching for co-occurrence finds it wherever it happened, not only in the most recent closes."
|
||||
},
|
||||
{
|
||||
"parent": "consensus.round",
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
"_excluded_spans": "rpc.process is deliberately absent from spans.names. It is created only in ServerHandler::processRequest() on the HTTP/JSON-RPC path, which the workload load generators, being WebSocket-only, never reach, so its quantiles were captured as null every run and could never gate. (The harness shell scripts do issue a few HTTP JSON-RPC health polls, far too few to produce a meaningful quantile.) See baselines/README.md.",
|
||||
"_excluded_ledger_store": "ledger.store is deliberately absent from spans.names too, for a different reason: it is below the ladder's resolution. The 2026-08-24 capture returned p50/p95/p99 of exactly 0.005/0.0095/0.0099 ms, which is 0.5/0.95/0.99 x the ladder's first edge of 0.01 ms — the signature of every sample landing in the first bucket, so the numbers are interpolation arithmetic on the bucket floor rather than latencies. That is physically plausible: LedgerMaster.cpp:463 wraps an in-memory ledgerHistory_.insert, which completes in single-digit microseconds. While all mass stays under 10 us the reported quantile cannot move materially, so NO absolute bound can gate it — every ledger.store slowing from 2 us to 9 us, 4.5x, leaves the reported value unchanged. Three keys that read as covered but cannot fire are worse than no keys (the same argument that excluded rpc.process), so they were removed rather than left in with a bound that looks derived. Restoring the key needs sub-10us edges on the collector's spanmetrics ladder (for example 0.001ms and 0.005ms) plus the matching entries in HistogramBuckets.h — that is the ladder's branch, not this file. ledger.store presence is still asserted by expected_spans.json and docker/telemetry/integration-test.sh, and its rate is still on the ledger-operations dashboard; only the latency gate drops it.",
|
||||
"_excluded_quantiles": "A THIRD KIND OF EXCLUSION, and the only one that deleting a name cannot express. spans.names lists span NAMES while _quantiles is shared across all of them, so the declared surface is the names x quantiles product and dropping ONE quantile of ONE span needs a subtraction. excluded_keys below is that subtraction: a flat {category}.{name}.p{quantile} key, exactly as _key_format defines it, mapped to the reason it is not gated. It can only ever remove a key, never add one, so a typo cannot silently start gating something new -- and check_regression_bounds.py rule F rejects an entry that would not otherwise be declared, an entry with an empty reason, and an entry that still carries a threshold override or a baseline value, so the exclusion cannot rot into dead config. Both prom_queries.py (which builds the capture plan) and check_regression_bounds.py (rule A) subtract it, so an excluded key is not queried, never reaches timings.json, and is not expected in the baseline. NOTHING ELSE CHANGES: the quantile is still computable from Prometheus with the _query_template above, the span is still asserted by expected_spans.json, and its rate is still on the ledger-operations dashboard. Only the latency gate drops it.",
|
||||
"_excluded_shape": "ALL FIVE ENTRIES BELOW SHARE ONE SHAPE, and it is worth naming because it will recur: the observed maximum across CI runs exceeds (baseline + bound), so an ordinary run clears the trip point with nothing having regressed. Two mechanisms produce that, and both are visible here. (1) A baseline that lands in the ladder's LOW buckets gets a tiny derived bound, because the bound IS the distance to the next edge up -- span.tx.apply.p50 at 0.0060 ms sits in the first bucket (0, 0.01] and gets 0.0440 ms of headroom, against a metric that has been measured at 2.3378 ms. (2) A spread so large that no bucket of headroom could absorb it -- span.ledger.validate.p99's 66.8x range reaches 25.8750 ms against a 10 ms trip point even though its bound is a comparatively generous 8.94 ms. The first mechanism is the one that bit three keys on the 2026-08-26 refresh, and it is a property of WHERE THE CAPTURED RUN LANDED rather than of the metric: the same span.tx.apply.p50 read 0.7917 ms in the previous baseline, mid-distribution, where the identical rule produced a 4.21 ms bound that absorbed the whole range. Whether the gate functioned was therefore decided by luck of the draw. THE FOLLOW-UP THAT WOULD RESTORE COVERAGE, stated so it is not left implied: a baseline captured from a SINGLE run cannot support these keys, because one sample carries no information about spread and the bound is derived from that one sample alone. What would let them be gated again is a multi-run baseline -- or a spread measurement captured alongside the baseline, so a bound can be sized against observed variance instead of against the ladder only. That is not implemented; it is the design change these five exclusions are waiting on. Until then, do NOT re-gate any of them by re-baselining until a run happens to land favourably, which is the failure this note exists to prevent.",
|
||||
"_excluded_shape": "ALL FIVE ENTRIES BELOW SHARE ONE SHAPE, and it is worth naming because it will recur: the observed maximum across CI runs exceeds (baseline + bound), so an ordinary run clears the trip point with nothing having regressed. Two mechanisms produce that, and both are visible here. (1) A baseline that lands in the ladder's LOW buckets gets a tiny derived bound, because the bound IS the distance to the next edge up -- span.tx.apply.p50 at 0.0060 ms sits in the first bucket (0, 0.01] and gets 0.0440 ms of headroom, against a metric that has been measured at 2.3378 ms. (2) A spread so large that no bucket of headroom could absorb it -- span.ledger.validate.p99's 66.8x range reaches 25.8750 ms against a 10 ms trip point even though its bound is a comparatively generous 8.94 ms. The first mechanism is the one that excludes three keys here, and it is a property of WHERE THE CAPTURED RUN LANDED rather than of the metric: the same span.tx.apply.p50 has read 0.7917 ms, mid-distribution, where the identical rule produces a 4.21 ms bound that absorbs the whole range. Whether the gate functioned was therefore decided by luck of the draw. THE FOLLOW-UP THAT WOULD RESTORE COVERAGE, stated so it is not left implied: a baseline captured from a SINGLE run cannot support these keys, because one sample carries no information about spread and the bound is derived from that one sample alone. What would let them be gated again is a multi-run baseline -- or a spread measurement captured alongside the baseline, so a bound can be sized against observed variance instead of against the ladder only. That is not implemented; it is the design change these five exclusions are waiting on. Until then, do NOT re-gate any of them by re-baselining until a run happens to land favourably, which is the failure this note exists to prevent.",
|
||||
"excluded_keys": {
|
||||
"span.consensus.ledger_close.p50": "Run-to-run variance exceeds the bound this ladder can derive, the same limit as the ledger.validate pair and the same mechanism as the two sibling p50 keys excluded alongside it. Baseline 0.0387 ms sits in the low bucket (0.01, 0.05], so hi_next is 0.1 ms and the derived bound is 0.0613 ms -- a 2.58x trip point. Measured across three CI runs the value spans 0.0387 to 0.2377 ms, a 6.1x spread (5.9x over four runs), and run 32867433073 read 0.2377 ms, 2.38x the trip point, on the SAME post-path-finding-removal workload as this baseline. So a healthy run reddens CI. This is a variance limit, not a defect and not a missing bound: widening is unavailable, because a bound tolerating 0.2377 ms would reach past the 0.25 ms edge and gate almost nothing. Do NOT re-gate by widening, and do NOT re-baseline until a run lands higher -- see _excluded_shape.",
|
||||
"span.ledger.build.p50": "The same mechanism as span.consensus.ledger_close.p50, one bucket up. Baseline 0.1151 ms sits in (0.1, 0.25], so hi_next is 0.5 ms and the bound is 0.3849 ms -- a 4.34x trip point. Across three CI runs the value spans 0.1151 to 2.3826 ms, a 20.7x spread (25.3x over four runs), and the observed maximum is 4.77x the trip point. Note what the previous baseline hid: at 1.0612 ms the same rule gave a 8.94 ms bound and a 10 ms trip point, which absorbed the entire range, so this key read as gated purely because that capture landed mid-distribution. Ledger construction is the hot path this gate most wants to guard, which makes the loss real and worth fixing properly -- with a baseline that carries spread information, not with a wider bound.",
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -3739,23 +3739,23 @@ Key properties:
|
||||
the limiting ladder step for each, and the edges that would fix them.
|
||||
- **A baseline refresh can silently move sensitivity in either direction.** The
|
||||
trip point is derived from the baseline, so a refresh that lands at the low end
|
||||
of a metric's range tightens the gate and one that lands high loosens it. The
|
||||
2026-08-26 refresh took `job.acceptLedger.running.p95` from a 5.74x floor to
|
||||
16.28x — it does not fire on any observed run, so it stays gated, but the weak
|
||||
floor is recorded rather than left to surprise someone. The same refresh put
|
||||
three `p50` keys below the spread they need, and they are now excluded (below).
|
||||
`baselines/README.md` carries the measurements.
|
||||
of a metric's range tightens the gate and one that lands high loosens it.
|
||||
`job.acceptLedger.running.p95` has been measured with a 5.74x detection floor
|
||||
on one baseline and 16.28x on another — it does not fire on any observed run,
|
||||
so it stays gated, but the weak floor is recorded rather than left to surprise
|
||||
someone. The same effect puts three `p50` keys below the spread they need, and
|
||||
those are excluded (below). `baselines/README.md` carries the measurements.
|
||||
- **The bound covers quantization noise only, so a key whose run-to-run variance
|
||||
exceeds it cannot be gated. Five keys are excluded for that reason**, leaving
|
||||
20 gated. `span.ledger.validate.p95` and `.p99` came first — spreads of 5.9x
|
||||
and 66.8x across four CI runs, both reaching past their trip points on healthy
|
||||
runs, because the span's duration follows peer-validation arrival timing rather
|
||||
than code speed. The 2026-08-26 refresh added `span.tx.apply.p50`,
|
||||
`span.ledger.build.p50` and `span.consensus.ledger_close.p50`, whose observed
|
||||
maxima sit 46.76x, 4.77x and 2.38x above their new trip points. That is the
|
||||
same rule applied, not a new exception: the decisive evidence is that
|
||||
`span.tx.apply.p50` read 0.7917 ms in the previous baseline and 0.00597 ms in
|
||||
this one — 132x apart on the same workload — so whether the gate worked was
|
||||
than code speed. `span.tx.apply.p50`, `span.ledger.build.p50` and
|
||||
`span.consensus.ledger_close.p50` join them on this baseline, with observed
|
||||
maxima 46.76x, 4.77x and 2.38x above their trip points. That is the same rule
|
||||
applied, not a new exception: the decisive evidence is that
|
||||
`span.tx.apply.p50` has read 0.7917 ms and 0.00597 ms on the same workload
|
||||
— 132x apart — so whether the gate worked was
|
||||
decided by where in its own distribution the captured run fell, not by the
|
||||
code. All five share one shape: the observed maximum exceeds
|
||||
`baseline + bound`, four of them because a low-bucket baseline yields a tiny
|
||||
|
||||
Reference in New Issue
Block a user