Two conflicts, both additions at the same spot:
NetworkOPs.cpp include block kept develop's rpc/detail/SyntheticFields.h
alongside this branch's telemetry/MetricsRegistry.h; .cspell.config.yaml
kept both new words.
NetworkOPs.cpp include block: develop replaced DeliveredAmount.h,
MPTokenIssuanceID.h and NFTSyntheticSerializer.h with the single
rpc/detail/SyntheticFields.h. Kept that and this branch's two telemetry
includes.
Notes across the workload contract described earlier versions of themselves, or
cited commits that only exist inside this chain. A squash merge publishes none of
it, so each reference resolves nowhere.
Notes that described their own earlier text:
- expected_spans.json: 'this note previously concluded', 'this note previously
said', 'Un-skipped 2026-08-26', 'the reason had simply gone stale for two
weeks' and 'the claim this entry carried' are replaced by the standing reason
each entry holds. The wildcard pairs now say the validator globs the child via
_span_name_matches(), and state the literal-collapse failure as what a
different validator WOULD do rather than as history.
- regression-thresholds.json: 'an earlier version of this note wrongly claimed',
'the earlier version oversold it' and 'an earlier note called that' become the
cautions themselves -- do not reason from 'every ladder step is at least 2x',
do not oversell the backstop, do not read a false fire as a missing override.
- test_check_regression_bounds.py: the docstring gives the reason a literal is
wrong here, not the story of two tests that once hard-coded one.
Baseline-refresh history rewritten as measurement:
- README.md, baselines/README.md, telemetry-runbook.md and regression-metrics.json
no longer attribute threshold moves to 'the 2026-08-26 refresh'. The evidence
is kept as measurement -- span.tx.apply.p50 has read 0.7917 ms and 0.00597 ms
on the same workload, 132x apart; job.acceptLedger.running.p95 has measured a
5.74x floor on one baseline and 16.28x on another -- which is what supports the
claim that a single-run baseline cannot bound these keys.
Two chain-only commit ids removed, d059f21bf3 and 3860c93db2. Neither is
reachable from develop, so both cease to exist on merge; the second is chain
bookkeeping. The facts they were cited for (the validator globs wildcards; the
span ladder's floor is 0.01 ms) are stated directly instead.
Capture provenance is deliberately kept: baseline-timings.json 'captured_at',
the 2026-08-26 baseline heading, and the 2026-08-24 figures cited as data.
Documentation, JSON note strings and one docstring only, no behaviour change.
These comments dated themselves against this branch or against an earlier
revision of the same change, neither of which survives a squash merge.
- 'as of this branch' in the pricing-case doc and the pass-through static_assert
becomes a statement about what the constants currently produce.
- 'Extracted from processGetObjectByHash()' and 'Split from start()' describe
edits internal to this change; both now say why the method stands alone.
- Recording.h: the mock-abstract mismatch is a standing consequence of a member
set that differs between builds, not something that 'has previously' happened.
- MetricsRegistry.cpp: describe the loops.txt entry as recording two cycles
rather than as what ordering.txt 'previously had'.
- InboundLedger.h: the acquire span is the only signal for back-fill cost; it
did not 'previously emit' nothing.
- check_bucket_parity.py: replace the eleven-phase drift story with the reason
the check exists, and point the failure message at HistogramBuckets.h and the
collector config instead of OpenTelemetryPlan/, which does not reach develop.
Comments and one error message only, no behaviour change.
capture_timings.py decides whether a captured timings file may become a
regression baseline. Every way of getting that wrong is silently green: a
capture that asked Prometheus for nothing still writes valid JSON, and once
accepted it is pasted in as a baseline, still reads as a placeholder, and the
regression gate stays off while the workflow reports it as activated.
Covered: an empty surface is not complete (0 of 0 is 100% by arithmetic), the
minimum ratio is inclusive, null values count as declared but not captured, the
threshold is recorded so a rejected capture can be judged later, and the exit
code follows the flag rather than recomputing the ratio. The empty case has its
own error path because the percentage message divides by the declared count.
Neither this file nor test_validate_telemetry.py ran anywhere before: not in
CI, not in run-full-validation.sh, not in pre-commit. They now run in the
naming job, which is fast and fires on nearly every PR, so a broken harness
surfaces in seconds rather than after an xrpld build.
They run as plain scripts. unittest discover would collect nothing from them,
since they hold bare functions rather than TestCase subclasses, and would exit
0 -- which is why each file fails when it collects no tests. The dependency
install is a separate step, placed after every stdlib-only check so those stay
reachable if PyPI is unavailable.
Every existing rule in this checker runs one way: take a consumer -- a collector
dimension, a Tempo tag, a dashboard label, a doc, an asserted metric name -- and
require it to resolve to the *SpanNames.h constants. Rule H looks closest to the
reverse but is still consumer-side: a constant USED at a call site that no header
defines. Nothing looked the other way.
So deleting a setAttribute from a .cpp and leaving its constant in the header
passed every rule and every compiler, while the telemetry it described stopped
being emitted. The workload validation job would eventually notice, but only when
it happens to run, and its path filter deliberately does not watch daemon .cpp
files -- widening it to 1827 C++ files to catch this would fire a twenty-minute
Docker job on nearly every commit.
Rule M closes that: an L1 constant no code under src/ or include/ references. It
searches all references, not just telemetry call sites, because constants are
passed to helpers, stored in locals and used as attribute VALUES -- a
call-site-only scan would report false positives. Constants referenced only by
test code are reported separately, since a constant exercised by a test but by no
production path is still dead in production.
A WARNING, not a failure, for two reasons that are both real here. Six constants
in this tree are already dead, so failing would redden the branch immediately. And
in a stacked chain a constant legitimately lands one commit before its call site,
so a failing rule would break intermediate branches for a condition that resolves
downstream.
What it reports today, all verified unreferenced across the whole repository and
not just src/include: ConsensusSpanNames.h val::increased, val::decreased and
val::unchanged; SpanNames.h seg::link, attr_val::success and attr_val::error.
Placed here rather than upstream on phase-1c, where the checker was introduced,
because the gap it closes is a workload-harness concern -- the contract asserting
a name nothing emits -- and the harness is this branch's. Putting it on 1c would
also mean union-resolving a 1900-line file across ten merge hops, each an
opportunity to silently drop the metric rules that live downstream.
Ported from a patch written against the sync-diagnostics copy, which carries the
metric-side rules I/J/K/L that this branch does not. The rule itself is purely
span-side; the only shared dependency it needed was iter_sources, which arrived
with those metric rules, so that helper is added here on its own. The rule-L
docstring entry and README row that came with the patch context were dropped --
this branch has no rule L.
Verification: rule M reports 262 constants checked and 6 unreferenced, exit code
still 0 because warnings do not change it; 169 unittest cases pass; the checker
compiles; no metric-rule content leaked in from the patch context (0 occurrences
of METRIC_MACRO_CALL or run_rule_i_metric_literals). Proven non-vacuous by
blanking all three call sites of attr::rpcStatus -- the count went 6 to 7 and
named that constant -- then restoring them and watching it return to 6.
The comments I added with the hierarchy sampling fix and the trigger change ran
to sixteen and twelve lines. The guideline is short and plain English. Rationale,
CI run numbers and the list of which relationships were affected belong in the
commit message, which is where they already are; inline they push the code apart
and go stale as soon as the reasons change.
Trimmed the sampling comment from sixteen lines to four, the re-check comment
from eight to four, _traceql_name_predicate's docstring from fourteen lines of
explanation to three, and the push-trigger comment from twelve to seven. Each
keeps what a reader needs at that line -- what the code does and the one
non-obvious reason -- and drops the history.
Comment-only: 13 insertions against 35 deletions, no statement changed.
Left alone deliberately: this file has ten pre-existing comment blocks longer
than six lines, including one added recently by another party. Rewriting someone
else's comments is not mine to do here, and the guideline is being applied to what
I wrote.
Verification: 7/7 validator tests pass; validate_telemetry.py compiles; the
workflow YAML parses, still carries no branches filter, and still lists 12 paths;
otel-naming exits 0.
An empty metric surface counted as a complete capture. build_query_plan
returns an empty plan without complaining for any config that yields no
gated keys, so pointing --metrics at the wrong file exits 0 and hands the
paste-me path a metrics:{} artifact to offer as the next baseline. Nothing
about such a run is evidence the pipeline works, so declared == 0 is now a
failure rather than vacuously complete.
The bounds checker also raised AttributeError on a baseline entry that is
not an object, instead of naming the key. A validator whose job is to catch
a malformed contract should report it, not crash on it.
Test cleanup is bound to its own temp tree, so a loop no longer leaves five
of six directories behind.
The regression baseline is bootstrapped by copying a CI artifact. The workflow
tested only that timings.json existed, then printed it verbatim under a heading
inviting the reader to paste it in as the new baseline.
capture_timings.py writes that file and only then enforces --min-capture-ratio,
so an incomplete capture leaves a file that exists but covers fewer keys than
the contract declares. The verdict lived in CAPTURE_EXIT, a shell variable local
to run-full-validation.sh that no other program could read. So on a placeholder
baseline plus a thin capture, CI offered an incomplete artifact as the next
baseline, and pasting it narrowed the gate with nothing reporting that it had.
That is the failure shape this harness keeps producing: a degraded result that
looks exactly like a good one.
The artifact now carries its own completeness, next to metrics:
"capture": { "declared": 20, "captured": 20, "min_ratio": 0.5, "complete": true }
complete is the same condition the producer exits 0 on, computed once with the
exit code read off it, so the flag and the status cannot drift apart. Any
consumer can now tell a complete capture from a thin one, not just CI.
Both paste-me paths refuse rather than warn: the workflow prints the counts and
an error annotation with no JSON, and the comparator explains on stderr while
leaving stdout empty, so a redirect cannot produce a plausible-looking file. A
warning above a copyable block is still a copyable block, and a reader who has
just hit a red gate is already predisposed to re-baseline. A missing capture
block fails closed.
Refusal is scoped to bootstrapping a baseline, not to comparing against one, so
artifacts captured before this change still replay: verified against the run the
current baseline came from, which carries no capture block and still reports 0
regressions. An injected regression is still caught, and the gated surface is
unchanged at 20 keys with 5 excluded.
Two problems, both about coverage this workflow claims to have and does not.
The push trigger gated on branch NAME as well as path, and GitHub ANDs the two.
Branch names are not something this repository controls, so a push to any branch
outside "pratik/otel-phase*", "feature/otel-*" or "feature/telemetry-*" was never
dispatched -- not queued, not skipped, no run to look at. That is not a
theoretical gap: two rounds of harness fixes on pratik/otel-sync-diagnostics
produced no signal at all before anyone noticed the workflow had never started.
The branches filter is removed; the paths already express the real question.
The path list was also incomplete in a way that matters more than it looks. The
span-name and metric-name headers are the wire contract this harness asserts
against by literal string, and the convention colocates each one with the class
it serves -- so eight of the ten *SpanNames.h headers live under consensus/,
overlay/, app/ledger/, app/main/, app/misc/, rpc/ and tx/, none of which was
matched. Renaming a span constant therefore compiled clean, emptied the
assertions and triggered nothing. Matched now by filename, "**/*SpanNames.h" and
"**/*MetricNames.h", so future headers are covered wherever they land. Added for
the same reason: include/xrpl/beast/insight (the interface headers decide what
the collector can publish, so they move the metric surface as surely as the
implementation), src/tests/libxrpl/telemetry (the GTests pinning those
constants), and the two checker directories that gate this surface in CI.
Second, the span hierarchy. Each span entry documents its parent, and a separate
list holds the pairs the validator actually checks in Tempo. Those had drifted
apart: 18 parentings were documented, 7 were checked. A span that stops nesting
under its parent -- which is what a detached guard does -- leaves every span and
every attribute intact, so no other check in this harness notices; the trace
simply stops being readable as one operation. Eleven pairs are added, each one
where both ends emitted on a real run: rpc.http_request -> rpc.process, the three
txq parentings, and seven consensus ones under consensus.round and
consensus.establish. Fourteen of eighteen are now asserted; the four still
skipped are the wildcard rpc.command.* families and pathfind.compute.
Three notes were also factually wrong, all repeating one mistake. They said
rpc.process and rpc.http_request cannot appear because that path is HTTP-only
while the load generator is WebSocket-only. The premise is right, the conclusion
is not: both appear on every run, five traces each, because
run-full-validation.sh polls each node's HTTP port with curl for readiness and
validated-ledger progress (:449, :502). Those polls take the HTTP path. A reader
acting on the old text would have gone looking for a way to make the harness
speak HTTP that it already speaks. The rpc.process -> rpc.command.* skip reason
inherited the same error and additionally claimed the WebSocket equivalent is
"asserted above instead", which it is not -- that one is skipped for the same
wildcard limitation. All three now state the real blocker, which is that
_validate_parent_child resolves a wildcard child to a single literal probe.
Both HTTP spans stay optional rather than being promoted: the curl polls are
harness scaffolding, not workload, and a future change to how the script waits
for a node could legitimately remove them.
Verification: JSON parses; 18 relationships, no duplicates, every non-wildcard
endpoint resolves to a declared span entry; counters still 41 span types and 62
unique attributes; workflow YAML parses, has no branches key, keeps
workflow_dispatch, and every new glob was checked against the tracked file list
with a matcher that reproduces GitHub's ** semantics; otel-naming exits 0;
pre-commit clean on both files. The eleven new assertions are proven only to the
extent that both ends emitted on run 32969481032 -- that a child is findable
INSIDE the parent's fetched trace is what CI will now decide.
Refreshes baselines/baseline-timings.json from run 32964262700 at 8418d474a7,
byte-identical to the CI artifact. The previous baseline was captured at
6a82fc6f37, before the path-finding load was removed from the workload, so it
described a load shape the harness no longer runs.
Every absolute bound is re-derived, because the rule is hi_next minus baseline
and the baselines moved.
Three more keys stop being gated: span.tx.apply.p50, span.ledger.build.p50 and
span.consensus.ledger_close.p50. This is the rule the previous commit recorded
being applied, not a new exception -- a key is gateable only when its
run-to-run spread fits inside its bound.
The evidence is span.tx.apply.p50, which read 0.7917 ms in the old baseline and
0.00597 ms in this one. That is a 132x move between two runs of the SAME
workload. The old value happened to land mid-distribution, so hi_next minus
baseline gave a 4.21 ms bound that absorbed the spread; the new value lands in
the ladder's first bucket, so the same rule gives 0.0440 ms and cannot survive
one. Whether the gate functioned was decided by where in the distribution the
captured run happened to fall, which is not a threshold in need of tuning.
Measured spreads across four runs agree: 364x, 25.3x and 5.9x respectively.
All five excluded keys share one shape -- a baseline landing in the ladder's
low buckets, where the derived bound is tiny, together with large run-to-run
spread. Single-run baselines cannot support them; a multi-run baseline, or a
spread measurement captured alongside the baseline, is what would let them be
gated again. Not attempted here.
Both runs that would have reddened CI now replay clean, and an injected 10x
regression is still caught on 19 of the 20 remaining keys, 20 of 20 at 20x.
The exception is job.acceptLedger.running.p95, whose baseline fell while its
hi_next did not, moving its floor to 16.28x. It stays gated with that floor
recorded beside the other weak keys.
Also makes the bounds checker report a zero or negative baseline as a named
rule failure instead of dividing by it and raising.
The regression gate has been red on runs with no code change. Only two of
the 25 gated keys ever tripped, both on the same span and never together:
run 32862589645 failed p99 at 25.8750 ms against a 1.0600 ms baseline
(+2341%), run 32867433073 failed p95 at 0.7500 ms against 0.2404 ms
(+212%), and in each run the other quantile sat well inside its own bound.
A real slowdown would move both. This is variance, not a defect.
Measured across four CI runs:
span.ledger.validate.p50 0.0484 to 0.0778 ms 1.6x spread kept
span.ledger.validate.p95 0.1281 to 0.7500 ms 5.9x spread excluded
span.ledger.validate.p99 0.3875 to 25.8750 ms 66.8x spread excluded
Both excluded quantiles reach past their trip point on a healthy run. The
mechanism is arrival timing, not slow code: the span opens only once a
quorum-completing validation arrives (LedgerMaster.cpp:987, inside
checkAccept, past the early return) and wraps the promotion work that
follows, so one slow consensus round dominates the tail of a 3m rate
window and which round that is differs every run.
Widening is not available and must not be attempted later: tolerating
25.8750 ms against a 1.0600 ms baseline needs a bound of about 24.8 ms,
which gates nothing. A bound admitting every healthy run's worst case
admits every regression too. p50 stays gated; it is stable.
THE GENERAL RULE, recorded so this does not recur: an absolute bound
derived as hi_next minus baseline comes from the histogram ladder, so it
budgets for quantization noise and for nothing else. It knows nothing about
how far a metric moves between runs on identical code. Before gating any
key, check its observed maximum across several runs against its trip point
and gate it only with margin. Spread alone proves nothing: tx.apply.p50
swings 364x and never fires, because its 5 ms trip point absorbs the range.
Of the 23 keys still gated the worst reaches 0.67 of its trip point.
Mechanism: spans.names lists span names while _quantiles is shared, so
dropping two quantiles of one span cannot be expressed by deleting a name.
regression-metrics.json gains an excluded_keys map from a flat key to the
reason it is not gated, subtracted by both prom_queries.py (so the key is
never queried) and check_regression_bounds.py rule A. A per-name quantile
override was rejected: a typo there leaves the key gating, whereas a typo
in an exclusion subtracts nothing and new rule F rejects it, along with an
empty reason, a leftover threshold override and a leftover baseline value.
Derived figures recomputed from the committed baseline: 25 gated keys to
23, detection floor 2.02x-9.43x to 2.02x-9.42x, weakly guarded keys ten to
nine, bound over baseline 102%-843% to 102%-842%. The baseline edit is a
deletion of two entries only, with no value rewritten.
Verified: both previously failing runs replay to zero regressions and exit
0; a tenfold increase injected into each of the 23 remaining keys in turn
is still caught in all 23 cases; rule F was confirmed load-bearing by
stubbing it out, which lets a stale exclusion pass.
The two log-correlation checks have never executed in CI: the workflow
hardcoded --skip-loki, so validate_telemetry.py never constructed
log.trace_id_present or log.trace_id_cross_reference. A green Telemetry
Validation therefore carried no evidence that a log line reaches Loki with
trace context. Drop the flag so both checks run and can fail the job.
Correlation spans four independent legs and a failed check names none of
them, so run-full-validation.sh now prints a per-leg diagnostic after the
suite whenever the checks are enabled:
node per-node debug.log line count, the count matching the injected
trace_id/span_id shape, one sample line, and the severity mix,
so "no log at all", "log level too high" and "no active sampled
span" are distinguishable
mount the container-side listing of /var/log/xrpld, taken with the
collector's own mounts and uid. That image is built from
scratch and carries no shell, so the listing runs in a
throwaway container with --volumes-from, not via docker exec
collector the receiver's watched files, logs-pipeline warnings, and the
internal log-record counters, read from inside the container's
network namespace because that endpoint binds to the
container's own localhost and its port is not published
loki the exact query used, the label inventory, and entry counts for
the stream selector with and without the line filter, so "Loki
has nothing" and "Loki has lines but none carry a trace id" are
distinguishable
The diagnostics are non-fatal by construction: every leg runs in its own
subshell with errexit off, each docker and curl call is guarded, and the
coordinator always returns success. Verified with no containers and no Loki
reachable, with an emptied PATH, and with a leg forced to exit non-zero.
validate_telemetry.py gains a matching diagnostic beside the checks,
following _log_prometheus_metric_names: warnings only, never a check
result. Its stream selector and line filter move into module constants
that the shell diagnostic reads back, so the two cannot drift into
describing different queries.
No check was widened or auto-passed, and LOG_QUERY_WINDOW_SECONDS stays at
four hours; a wider window would let a check pass on a previous run's logs.
The gate could not catch a regression on any sub-millisecond span.
compare_to_baseline.py requires both the percentage and the absolute bound to
breach, and every span shared one flat absolute bound of 10 ms (15 ms for p99)
calibrated for a 5-25 ms band the spans do not occupy. Against the baseline
captured on 2026-08-24, where 18 of the 28 quantiles gated at the time sat
below 1 ms, that bound sat 1.15x to 2000x above the metric it guarded, so the
AND never fired: a 100x regression injected into span.ledger.store.p95 reported
0 regressions and exit 0. Injecting a 10x regression into each key in turn was
caught on only 5 of 28.
Give every gated key its own absolute bound, equal to the distance from its
baseline to hi_next, the edge above the top of the bucket the baseline sits in.
The trip point is then exactly hi_next, so the gate fires only once the reading
clears the bucket above the baseline's own. That is the property a multiple of
the enclosing bucket width cannot provide: after the quantile crosses hi, the
interpolation happens across the next bucket, which on this ladder is up to
eight times wider, so no multiple of the enclosing width bounds the excursion.
Measured with a model-free reachability test, a single bucket crossing can
produce a false regression on 2 of 25 keys under the old flat bound and 0 of 25
under this rule. The smallest catchable regression is 2.02x to 9.43x per key.
The job queue bound had the same shape of problem on three of its four keys
(42x, 47x, 220x before). Defaults now sit at each ladder floor, leaving the
percentage bound operative for a metric that somehow reaches them.
Drop span.ledger.store from the gated surface. Its captured quantiles were
0.005, 0.0095 and 0.0099 ms, which is the ladder's 0.01 ms floor times the
quantile: every sample lands under 10 us, so the reported value does not move
even if each store slows from 2 us to 9 us. No bound can gate it. Presence is
still asserted by expected_spans.json and the integration test, and the rate is
still on the ledger-operations dashboard.
Add check_regression_bounds.py, wired into the same workflow step as the bucket
parity check. It fails when a bound is not the one its own baseline implies,
when a gated key has no override, when the baseline and metric surface disagree,
when the percentage bound would become operative, and when a baseline carries
the ladder floor signature. This gate has now broken three times through the
same drift between ladder, baseline and bounds, so documentation alone is not
enough.
compare_to_baseline.py is unchanged: its existing per-metric override mechanism
already expresses all of this.
A missing, unreadable or malformed input makes that check exit 1 naming the
input, rather than reporting success without having checked anything; only a
placeholder baseline, the documented bootstrap state, still exits 0. Its own
tests cover both halves of that contract plus one case per rule, and run in the
workflow before the check so a broken rule reads as a broken rule.
The microsecond ladder's first edge was 100us, which sat ABOVE the mass of
every instrument using it. Measured on devnet: 99.3% of job_queued_us
samples, 92.5% of job_running_us and 90.4% of getobject_lookup_us fell in
that first bucket. histogram_quantile then interpolated inside bucket 0 and
returned `quantile / fraction_in_bucket_0 x first_edge` -- p75/p95/p99 of
job_queued_us read 75.52/95.66/99.69us against a prediction of
75.53/95.67/99.70. Three-decimal agreement: those panels were reporting
arithmetic on the bucket edge, not latency.
The fix was already half-written. kSubMillisecondBoundaries had been parked
in MetricsRegistry.cpp as [[maybe_unused]] with a comment noting exactly this
problem for nodestore reads. Its edges are now folded into kMicrosecondBuckets
rather than deleted, so the parked intent is carried forward: 1..1000us
resolution where the mass is, upper edges unchanged so multi-second stalls
stay measurable.
Also moves the GetObject count and charge ladders into HistogramBuckets.h, so
all five ladders have one owner and one set of invariant tests (29 now).
Adds check_bucket_parity.py, wired into the existing OTel naming workflow.
The C++ millisecond ladder and the collector's spanmetrics ladder are
specified to agree over their shared range; they were identical when shipped,
then the collector side alone was extended and nothing noticed for eleven
phases. The check asserts containment rather than equality, because jobs
outlive spans -- jobq_updatepaths averages ~60s, which no span approaches, so
demanding equality would force a ceiling that censors it. Verified it rejects
a missing collector edge, a bogus in-range edge, and a return to the 5s
ceiling.
ledger-data-sync's "Job Queue Wait p95 By Type" moves off the beast
jobq_*_q_milliseconds pair onto job_queued_us filtered by job_type. Those
beast metrics are ms-quantised at the source (Event rounds up to a whole
millisecond), so 94-100% of their samples sat in the first bucket and no
ladder change could fix them. Note the label values are camelCase
(job_type="ledgerData"), not the lowercase metric-name fragments.
Both histogram-fed alert thresholds re-validated and left unchanged, with the
measured basis recorded so neither gets tuned against the old artefact: only
0.0022% of job_queued_us samples exceed the 1s threshold, and every edge
bracketing the 1000ms ios_latency threshold survived the ladder change.
Docs: the rpc_size "known issue -- tracked separately" notes in the runbook
and 09-data-collection-reference are now resolved notes, the stale 10-edge
span_duration bucket list is corrected to the collector's real 20, and the
runbook gains a "Reading A Histogram Percentile" section covering both
saturation traps and the expected discontinuity after a ladder change.
