Brings phase-10 up to 29673de531, three commits: a Loki-diagnostic count fix with
Tempo errors made visible, a baseline recapture that stops gating keys whose
run-to-run variance dominates their bound, and WebSocket reply correlation with
silent placeholder data removed.
Merged clean -- git reported no conflicts, so no resolution decisions were made
here. The 14 incoming files are all harness and docs: six .py, three .md, three
.json, two .sh. No runtime C++ and no include-line changes.
The baseline recapture resolves the regression gate that reddened the previous
run on this branch. That run reported span.consensus.ledger_close p95 1.62ms and
p99 7.14ms against a baseline of 0.49 and 0.93; the recaptured baseline puts p95
at 0.783 with a 5ms trip point and p99 at 2.03 with a 10ms trip point, so both
readings now sit inside the gate. That matches the diagnosis recorded at the
time: everything changed between the last green gate and that failure was a .py,
a .md and a GTest file, with zero runtime C++, and p50 had improved while only
the tail moved -- variance, not a slowdown. The incoming commit reaches the same
conclusion from the other side, excluding three p50 keys after measuring spreads
of 391.8x, 20.7x and 6.1x across three runs.
Eight defects found in review of PR 6519. Twenty-four review threads reported
them; ten were duplicates of one another and four were wrong about the code.
The two that corrupt data. Both WebSocket clients reuse the socket after a recv
timeout, and the library queues the late reply, so the NEXT request reads the
previous response. In rpc_load_generator that misattributes latency, and the
skew is permanent rather than one-off. In tx_submitter it is worse: a submit
that reads an account_info reply freezes that account's sequence number and
every later transaction for it fails. Both now correlate replies by request id
under a single overall deadline, with a counter rather than a wall clock, since
time.time() is not monotonic and collides within a tick.
workload_orchestrator never cleared its fixed report paths, so a run that
produced no report silently adopted the previous run's totals -- breaking the
invariant evaluate_exit_gate documents. Reproduced by planting a stale total
and watching it appear in a later summary.
collect_system_metrics reported placeholders as if they were measurements. The
consensus mean used bc with a || echo 0 fallback that neither warned nor
cleared METRICS_COMPLETE, unlike every sibling path; it now uses awk, already a
hard dependency here, which removes the failure mode instead of reporting it.
Note this moves the mean from truncation to rounding, at most 1 ms on a value
of about 45 s. Unmeasurable TPS now warns and clears the flag too. All four
curl probes gained a timeout, not just the one the review named -- an
unresponsive node could hang any of them.
The orchestrator's help text claimed 18-dashboard coverage; there are 15 on
disk, 15 uids in the contract, and the profile already said 15. The
tx_submitter docstring listed twelve transaction types where ten exist, and
claimed issued-currency payments that build_payment never sends.
Two suggested patches were deliberately not taken. A recursive delete of the report
parent sits in a per-task function and would delete earlier phases' reports
mid-run, and recursively remove a caller-supplied --report-dir.
The stale microsecond axis label on ledger-data-sync is real but belongs to
phase 9, which carries a byte-identical copy of that dashboard, so fixing it
here would leave that PR wrong and guarantee a conflict.
Refreshes baselines/baseline-timings.json from run 32964262700 at 8418d474a7,
byte-identical to the CI artifact. The previous baseline was captured at
6a82fc6f37, before the path-finding load was removed from the workload, so it
described a load shape the harness no longer runs.
Every absolute bound is re-derived, because the rule is hi_next minus baseline
and the baselines moved.
Three more keys stop being gated: span.tx.apply.p50, span.ledger.build.p50 and
span.consensus.ledger_close.p50. This is the rule the previous commit recorded
being applied, not a new exception -- a key is gateable only when its
run-to-run spread fits inside its bound.
The evidence is span.tx.apply.p50, which read 0.7917 ms in the old baseline and
0.00597 ms in this one. That is a 132x move between two runs of the SAME
workload. The old value happened to land mid-distribution, so hi_next minus
baseline gave a 4.21 ms bound that absorbed the spread; the new value lands in
the ladder's first bucket, so the same rule gives 0.0440 ms and cannot survive
one. Whether the gate functioned was decided by where in the distribution the
captured run happened to fall, which is not a threshold in need of tuning.
Measured spreads across four runs agree: 364x, 25.3x and 5.9x respectively.
All five excluded keys share one shape -- a baseline landing in the ladder's
low buckets, where the derived bound is tiny, together with large run-to-run
spread. Single-run baselines cannot support them; a multi-run baseline, or a
spread measurement captured alongside the baseline, is what would let them be
gated again. Not attempted here.
Both runs that would have reddened CI now replay clean, and an injected 10x
regression is still caught on 19 of the 20 remaining keys, 20 of 20 at 20x.
The exception is job.acceptLedger.running.p95, whose baseline fell while its
hi_next did not, moving its floor to 16.28x. It stays gated with that floor
recorded beside the other weak keys.
Also makes the bounds checker report a zero or negative baseline as a named
rule failure instead of dividing by it and raising.
Three defects in the harness's own instrumentation, all of the same shape: a
failure that reads as an absence.
The Loki diagnostic reported "unavailable entries" rather than a count. It
issued an unaggregated count_over_time, and because the filelog regex_parser
leaves message and timestamp as log-record attributes, Loki's OTLP path turns
those into structured metadata, which joins a metric query's label set. The
query therefore produced one series per log line and Loki answered HTTP 400,
maximum number of series reached. A second bug hid the first: the JSON helper
never checked resp.status, so Loki's own explanation arrived as a mimetype
complaint instead. Both fixed, in the Python and the shell twin, and verified
against a real loki 3.7.6 including a genuine-zero control so that zero stays
distinguishable from unavailable.
_tempo_search and _tempo_get_trace called resp.json() with no status check, so
any non-2xx became "0 traces" or "0 spans" -- the same class of bug as the
span.name tag returning 200 with an empty list. A 404 on /api/traces/<id>
legitimately means "not indexed yet", so that stays an absence and every other
non-200 now raises.
log.trace_id_cross_reference queried Tempo once, with no retry, while the
metric checks share a poll deadline for exactly this race. It now polls on the
existing METRIC_POLL_TIMEOUT_SEC/INTERVAL, so a trace that has not yet been
indexed is retried rather than reported missing. The window stays at 4 hours
and the assertion is unchanged.
Brings phase-10 up to 8418d474a7, one commit: the span reverse-coverage check
was querying the tag `span.name`, but a span's name is a TraceQL intrinsic
rather than a span-scoped attribute, so Tempo answered 200 with an empty
tagValues list and the check silently never evaluated. It now queries the bare
`name` intrinsic.
That is the same inertness the previous CI round on this branch observed from the
other end -- the run logged "Tempo span names (0 total)" while per-span TraceQL
searches each found traces and a logged trace id resolved to 100 spans. So the
incoming fix converts a check that could only ever pass into one that can
actually fail.
validate_telemetry.py auto-merged: the incoming change and this branch's are in
different regions. Verified afterwards that both survived -- the bare `name`
endpoint is in and the old `span.name` one is gone, alongside this branch's
assert_sync_diagnostics_metrics, its gather-based fan-out, and the
SKIPPED_METRIC_GROUPS exclusion inside _metric_check_targets that keeps the
sync-diagnostics group from being walked twice.
One conflict, in the runbook's validation-coverage table, and it needed both
sides rather than either. This branch's copy carries the counts recomputed from
the real contract during the previous merge (48 span types as 28 required and 20
optional, 145 metric checks across 26 categories, 16 dashboards); phase-10's copy
still carries the pre-merge figures. But phase-10's copy also corrected the
Reverse coverage row's description from the `span.name` attribute to the `name`
intrinsic, which is precisely the bug its commit fixes -- this branch's row still
described the broken query. Resolved as this branch's rows with phase-10's
Reverse coverage row substituted in. Every other cell was byte-identical between
the two sides apart from separator padding.
Verification: no conflict markers repo-wide; two parents; validate_telemetry.py
compiles; both sides' contributions asserted present by name rather than assumed;
otel-naming exits 0 including Rule E over the edited doc; levelization baseline
clean and the incoming diff changes no include lines; pre-commit clean on both
files, prettier having re-padded only the eight table rows, with the recomputed
counts and the corrected intrinsic wording confirmed present afterwards. No C++
changed, so no compile is implicated by this merge.
1e341d5413 fixed half of this. The slice that assembled the union dropped
`using namespace xrpl::telemetry;`, which supplied TWO names: `consensus` and
`seg`. Aliasing only the first left ConsensusSpanNames.cpp:156 --
`seg::consensus` -- unresolved, and clang, gcc and MSVC all failed there. It was
invisible in the previous round only because clang stops after 20 errors and the
50 `consensus` sites filled that budget, so the same defect had been present since
the merge rather than being introduced by the partial fix.
Why it was missed: the prefix scan behind the first fix sampled a line range that
did not contain line 156. This time every leading namespace qualifier in the file
was enumerated from comment- and string-stripped source and checked against the
names the file makes available, with the line each becomes available: attr, val,
op, part and span from the using-directive, consensus and seg from the aliases,
AvalancheState from a function-local using-declaration inside the test that uses
it, and std/xrpl needing nothing. Every one is declared before its first use, and
nothing else is qualified anywhere in the file.
The predicted hazard did not materialise. No `reference to 'attr' is ambiguous`
error appeared in any of the three compilers, so keeping xrpl::telemetry out of
scope and naming the two members explicitly was the right shape.
This also clears the misc-include-cleaner error that came with it. clang-tidy
reported SpanNames.h as not used directly and its exported fix deleted the
include; that fix was wrong. `seg` is declared at SpanNames.h:103, so the alias
makes the include genuinely used and the diagnostic goes away rather than needing
the include removed.
Verification: compiled. `c++ -fsyntax-only` with this file's real flags from its
compile_commands.json entry exits 0. Proven non-vacuous by removing the alias
again and reproducing CI's exact message -- "'seg' was not declared in this
scope; did you mean 'xrpl::telemetry::seg'?" -- then restoring it and returning to
exit 0. Braces balance 27/27 on stripped source, 17 TEST cases intact, pre-commit
clean including clang-format and clang-tidy. The tests still have not RUN: this is
a syntax-only check of one translation unit, so nothing linked and no assertion
executed.
The span reverse-coverage check has never evaluated. It reported "no span
names were reported (backend unreachable or empty)" on a run where Tempo
demonstrably held data -- the same run resolved a logged trace id to 32
spans.
Root cause: the tag-values query asked for `span.name`. A span's name is a
TraceQL intrinsic, not a span-scoped attribute, so `span.name` resolves to
an attribute nothing sets. Tempo answers 200 with an empty tagValues list,
which is indistinguishable from an empty backend and never raises, so the
surrounding try/except stayed silent.
Verified against tempo 2.9.4 holding exactly one span named
probe.reverse.coverage, with the collector in front of it:
/api/v2/search/tag/span.name/values -> {"tagValues":[]}
/api/v2/search/tag/name/values -> that span's name
/api/v2/search/tag/resource.service.name/values -> xrpld
The third line is the control: the span was in Tempo, so the first line's
emptiness was the wrong tag rather than no data. Cross-checked against a
populated Tempo elsewhere, whose span scope lists real attributes
(command, ledger_seq, tx_hash) and no name tag at all, while the bare
intrinsic returns the whole span inventory.
This is pre-existing, not a regression in the reverse check: the same URL
fed the operations diagnostic before that check existed, and the last
green run before it also logged "Tempo operations (0 total)". The check
faithfully reported an empty input; the input was broken.
The neighbouring resource.service.name query is correctly scoped and is
left alone.
The add/add resolution in 493475a9d4 did not compile. Two defects, one cause.
Both branches wrote this file independently and the union was assembled by
slicing each side's tests out of its own copy. That slice was asymmetric: it
started at the first TEST( line, which skipped the pre-merge side's
`using namespace xrpl::telemetry;` and its `namespace {` opener, but ended at the
`#endif`, which kept that side's `} // namespace` closer. So the file lost the
scope its own tests depend on and gained a brace with nothing to close: 27
openers against 28 closers.
Clang reported 50 "use of undeclared identifier 'consensus'" sites from line 144
and gave up at 20; GCC got far enough to also report "expected declaration
before '}' token". Neither parent has this combination -- it exists only in the
merge.
The nine validation-accept tests spell their constants out from `consensus`,
which the surviving directive does not make visible: `using namespace
xrpl::telemetry::consensus::span` imports the MEMBERS of consensus::span, not the
enclosing namespace name. Fixed with a namespace alias rather than by restoring
the blanket `using namespace xrpl::telemetry;`, because that would pull
xrpl::telemetry::attr (SpanNames.h:117) into scope alongside
consensus::span::attr and make all eleven bare `attr::` references in the
phase-span tests ambiguous -- trading a hard error for a subtler one.
The orphan closer is removed rather than matched with a new opener. The nine
tests it used to wrap declare no helpers, so the anonymous namespace bought no
internal linkage, and the eight tests from the other side were never inside one.
Verification: braces balance 27/27 counted on comment- and string-stripped
source; 17 TEST cases intact; the alias at line 59 precedes the first bare
`consensus::` in CODE at 155 and the directive at 48 precedes the first bare
`attr::` at 64, both measured after stripping comments, because an earlier check
matched its own explanatory prose and reported a false ordering violation; no
blanket using-directive in code; pre-commit clean including clang-format and
clang-tidy. NOT compiled -- there is no approval to build here, so this fix is
structurally verified only, and CI is the first compiler to see it.
Brings phase-10 up to 3836078a78 (74 commits). Seven conflicts, resolved
per-hunk; no side was taken wholesale.
validate_telemetry.py, four hunks. The module docstring keeps both category
lists, renumbered. _log_prometheus_metric_names takes phase-10's version: it
returns the family list that the new reverse-coverage check consumes, and
_log_name_list prints every family sorted one per line, which supersedes the
hand-maintained prefix filter this branch had been extending -- that filter
existed only to keep the log readable and listed strictly less. validate_metrics
takes phase-10's _metric_check_targets call. assert_sync_diagnostics_metrics and
phase-10's _check_metric_label both landed at the same place; both are kept.
That third hunk carried the hazard this branch had flagged in advance.
_metric_check_targets selects every group satisfying isinstance(dict) and had no
equivalent of SKIPPED_METRIC_GROUPS, because on phase-10 there was no group that
needed excluding. Merged as-is it would have walked sync_diagnostics while
assert_sync_diagnostics_metrics also walks it, polling and reporting all 61
metrics twice. The exclusion is reinstated inside that function, and its
docstring claim that the isinstance test "selects exactly the same groups the
previous name-based exclusion list did" is corrected -- true on phase-10, false
here, and the reason is ownership, which no structural test can express.
expected_metrics.json: both sides appended to metrics_excluded, so both sets are
kept, 29 entries. expected_spans.json: phase-10's fuller pathfind.compute
skip_reason replaces this branch's, and this branch's ledger.acquire ->
ledger.acquire.astree relationship is kept.
Both docs carried stale counts, and the two sides disagreed with each other --
15 dashboards against 16, and both claiming 41 span types when the contract holds
48. Rather than pick a stale side, every figure is recomputed from the resolved
contract: 48 span types as 28 required and 20 optional, 145 metric checks across
26 asserting categories as 140 names plus 5 required_labels, of which 61 are the
sync_diagnostics names, and 16 dashboards, which matches both the uid list and
the files on disk. The runbook keeps phase-10's table, which adds the reverse-
coverage row.
ConsensusSpanNames.h: both sides added different constants to namespace val;
both kept. Confirmed no identifier is redefined -- the merged file's duplicate
set is identical to this branch's, and those duplicates are distinct namespaces
(op::round against the enclosing span::round), not redefinitions.
ConsensusSpanNames.cpp was an add/add: both branches wrote this file
independently, 9 tests here and 8 on phase-10, with no name in common. All 17
are kept. The guard is dropped rather than applied to the union: SpanNames.h
documents that its constants are deliberately NOT guarded by
XRPL_ENABLE_TELEMETRY, ConsensusSpanNames.h has no guard, and the tests this
branch contributed reference ValStatus only in comments while calling
validationStatusValue with plain ints. So they compile without telemetry, and
unguarding them gains coverage in a -Dtelemetry=OFF build rather than losing it.
One defect belongs to the merge itself, appearing on neither parent. phase-10
added a job_queue_per_type_gauges group holding six jobq_<type>_running/_waiting
names; this branch declared jobq_saturation in MetricNames.h. Rule K checks a
name only when its family is owned, so declaring that constant made jobq_ owned
and turned phase-10's six entries into violations. They are beast::insight gauges
created per job type by JobTypeData's constructor, so no constant can exist for
them -- one triple per job type, minted at runtime. The group joins
NON_OTEL_METRIC_GROUPS alongside statsd_gauges for the same reason.
Verification: no conflict markers repo-wide and no unmerged index entries; both
JSON contracts parse; validate_telemetry.py compiles; every count written into
the docs re-derived from the resolved files and matching; span counters still 48
and 74; check_otel_naming.py exits 0, and Rule K proven still able to fail by
injecting a bogus name in an owned family; levelization baseline clean after
regeneration, with 17 incoming include changes; doxygen style clean across all
tracked C++ at CI scope; pre-commit --all-files clean except cargo-fmt, which
reports "Executable `cargo` not found" and touches none of the 0 Rust files here.
NOT compiled -- no approval to build, so the incoming C++ is unverified by a
compiler on this branch.
Review of 42a72863bb found that several of the reasons it recorded were wrong,
and one of its own changes was half-applied. A wrong rationale in a contract
file is worse than none, because the next reader treats it as evidence.
The histogram parity claim falsified itself. That commit's group description
says histograms are listed by all three Prometheus series, but dns_resolve and
overlay_dial were given only _bucket and _sum. Both gain _count, so the four new
histograms now match the claim and the rpc_method_us convention. Two notes still
said the histogram is asserted "by its _bucket and _count series"; both now say
all three.
The reason given for excluding serve_refused_total was wrong. It said every node
holds the same history so getLedger()/getTxSet() succeed. PeerSetImpl::addPeers
uses hasItem(peer) only as a sort SCORE and then adds peers in score order up to
its limit, so peers that lack the item are asked anyway and would be refused.
The real reason nothing is refused is that nothing asks: an inbound TMGetLedger
originates only from the ledger.acquire and txset.acquire paths, both optional.
That ties this counter to those spans, which the note now records.
The reason for excluding peer_disconnect_total rested on the run window being
270 s, shorter than maxDivergedTime. The window is over 300 s once the 60 s
propagation wait, node startup and up to 45 s of metric polling are counted, so
that argument does not hold. The counter also sits after the socket-already-
closed early return, which only de-duplicates repeat closes, and covers 13 reason
values rather than the three cited. It stays excluded on the ground that a
healthy cluster produces no disconnect cause and a mutual-dial race can produce
one non-deterministically, which is flaky either way.
ledger_jump_total was described as unable to fire. Its counter is unconditional
and checkLastClosedLedger runs every round on every node, so a node that falls
behind can follow a chain tip it did not build on -- and the burst phases exist
to create exactly that load. Reworded as not deliberately provoked, with the
decisive test named: grep the nodes' debug.log for "JUMP last closed ledger".
The claim that the group had never asserted anything was false, and the true
history is worth keeping. The five-argument call was CORRECT when it landed on
2026-07-24: the function then took report as its fifth parameter and recorded the
result itself. phase-10 added the deadline and semaphore on 2026-08-14, updating
its own caller, and never saw this one. The merge that first contained both,
7c70e142e9, was CLEAN because the two edits sit in different regions, and it
produced a caller that no longer matches its callee. So the group asserted for
about four weeks and then broke silently at a clean merge -- a semantic conflict
git cannot see and, in Python, no compile step rejects. Recorded as
_gate_history_note, because the standing rule it implies is that a clean merge is
not evidence that a cross-branch call site still matches.
_b5_rotation_note had three defects of its own. Deleting "Two independent
reasons." left "First, ... Second, ..." dangling; it counted four signals where
there are three; and its second half still described the dynamic_cast as making
registerRotationStateGauge return early, which contradicts the correction in its
first half and would tell a reader no instrument exists. The cast and early
return are inside the observe callback; the instrument is created eagerly.
Two further corrections. The comment explaining the fix said four later phases
were lost; CI passes --skip-loki, so three are. And the claim about phase-10's
unaccounted-metric pass overstated it: it is warning-only, cannot fail CI, and
also accepts an accounted_patterns regex list.
Recorded a merge hazard where the resolver will see it. phase-10 replaces the
group flatten with _metric_check_targets, which selects every dict group and has
no equivalent of SKIPPED_METRIC_GROUPS, so resolving that merge in phase-10's
favour makes validate_metrics walk sync_diagnostics while its own validator still
does -- every metric polled and reported twice.
The runbook contradicted the CI requirement this work introduces. It carried a
"Gap: no ledger.* span carries a ledger hash" block stating the attribute is
never set by any call site and that filtering on span.ledger_hash returns
nothing, while makeLedgerTraceSpan sets it unconditionally on ledger.validate and
ledger.store and InboundLedger sets it on ledger.acquire and its three phase
children. ledger.build is the one real exception. The block, the ledger-span
attribute table and the sentence listing what init() sets are all corrected.
Verification: both JSON files parse; the four histograms each carry _bucket,
_count and _sum; 61 asserted names with no duplicates and no overlap with the 22
excluded; span counters still 48 and 74; the crash repro records 61 checks with
no TypeError; check_otel_naming.py exits 0; pre-commit passes on all three files
(prettier re-padded only the runbook table rows touched here). NOT compiled -- no
C++ changed.
The sync_diagnostics group asserted nothing. assert_sync_diagnostics_metrics
called _check_prometheus_metric with five positional arguments against a
six-parameter signature: `report` landed in `deadline` and `sem` was omitted
entirely, so the call raised TypeError before a single metric was queried.
Neither run_validation nor main catches anything, so the traceback propagated,
run-full-validation.sh recorded the non-zero exit as a validation failure, and
the four phases ordered after it -- dashboards, both parity checks and log-trace
correlation -- never ran at all. Reproduced directly: TypeError, zero checks
recorded. Even with the arity corrected the group would still have passed
silently, because _check_prometheus_metric RETURNS its CheckResult rather than
recording it and the value was discarded. Both halves are fixed by adopting the
fan-out validate_metrics already uses: one shared deadline, a concurrency
semaphore, gather, then report.add per result. The same call now records 55
checks where it previously recorded none.
With the gate live, the inventory it guards had to be made honest.
Two metrics could never have passed it. unl_fetch_total is emitted only from
ValidatorSite::reportFetchOutcome, which indexes sites_[siteIdx]; sites_ comes
from [validator_list_sites], and the harness writes a static [validators] file
with no list site anywhere, so no fetch outcome is ever reported.
handshake_negotiation_fail_total needs a rejected handshake, and no reject path
was found to be reachable between identical localhost nodes. Both move to
not_asserted.metrics_excluded, which is where the file's own description says
workload-gated names belong. Eleven further conditional metrics -- the acquire,
replay, disconnect, serve, jump and sweep counters -- were documented only
inside free-text notes; they move to the same map. That matters beyond tidiness:
_accounted_metric_names harvests metrics_excluded keys, so a name recorded only
in prose is reported as unaccounted, and a prose note cannot be linted at all.
Two metrics were wrongly excluded. rotation_state's callback gates only on
dynamic_cast<DatabaseRotating*>, and online_delete=256 is set by both the cfg
template and run-full-validation.sh, so SHAMapStoreImp builds a
DatabaseRotatingImp, the cast succeeds, and both sub-series are observed on
every collection tick. The note claiming the harness could not produce them
conflated "no rotation runs" with "no series published"; the first is true and
bounds the values, the second is false. Both are now asserted at value 0, where
absence rather than the zero is the regression, and the note is corrected.
The four new histograms listed only _bucket, or _bucket and _count. Each now
lists _sum as well, matching the rpc_method_us and job_queued_us convention, so
an exporter regression that drops one series cannot pass.
On the span side, ledger.validate and ledger.store are the two ends of the
per_ledger trace-join group, and the join is computed by hashing ledger_hash --
yet neither required it. Both spans take it unconditionally from
makeLedgerTraceSpan, so requiring it is free, and without it a lost join key
surfaces only as "spans landed in separate traces", naming the consequence
instead of the cause.
Deliberately unchanged: ledger.serve stays required and peer.dial keeps its
current required attributes, though both look unsafe -- ledger.serve can only
fire if an optional span fires first, and peer.dial's destructor exit sets
neither outcome nor duration_ms. Those weaken assertions rather than add
coverage, so they are reported rather than changed here.
Verification: TypeError reproduced before the fix and absent after, with 55
checks recorded; both JSON files parse; no name is both asserted and excluded
and none is duplicated; the declared span counters remain consistent at 48 and
74, proven by injecting an extra attribute and watching the check fail;
check_otel_naming.py exits 0, and Rule K was proven to read these entries by
injecting a bogus name in an owned family and observing exit 1; pre-commit
passes on all three files; the levelization baseline is unchanged. NOT compiled
-- no C++ changed.
The regression gate has been red on runs with no code change. Only two of
the 25 gated keys ever tripped, both on the same span and never together:
run 32862589645 failed p99 at 25.8750 ms against a 1.0600 ms baseline
(+2341%), run 32867433073 failed p95 at 0.7500 ms against 0.2404 ms
(+212%), and in each run the other quantile sat well inside its own bound.
A real slowdown would move both. This is variance, not a defect.
Measured across four CI runs:
span.ledger.validate.p50 0.0484 to 0.0778 ms 1.6x spread kept
span.ledger.validate.p95 0.1281 to 0.7500 ms 5.9x spread excluded
span.ledger.validate.p99 0.3875 to 25.8750 ms 66.8x spread excluded
Both excluded quantiles reach past their trip point on a healthy run. The
mechanism is arrival timing, not slow code: the span opens only once a
quorum-completing validation arrives (LedgerMaster.cpp:987, inside
checkAccept, past the early return) and wraps the promotion work that
follows, so one slow consensus round dominates the tail of a 3m rate
window and which round that is differs every run.
Widening is not available and must not be attempted later: tolerating
25.8750 ms against a 1.0600 ms baseline needs a bound of about 24.8 ms,
which gates nothing. A bound admitting every healthy run's worst case
admits every regression too. p50 stays gated; it is stable.
THE GENERAL RULE, recorded so this does not recur: an absolute bound
derived as hi_next minus baseline comes from the histogram ladder, so it
budgets for quantization noise and for nothing else. It knows nothing about
how far a metric moves between runs on identical code. Before gating any
key, check its observed maximum across several runs against its trip point
and gate it only with margin. Spread alone proves nothing: tx.apply.p50
swings 364x and never fires, because its 5 ms trip point absorbs the range.
Of the 23 keys still gated the worst reaches 0.67 of its trip point.
Mechanism: spans.names lists span names while _quantiles is shared, so
dropping two quantiles of one span cannot be expressed by deleting a name.
regression-metrics.json gains an excluded_keys map from a flat key to the
reason it is not gated, subtracted by both prom_queries.py (so the key is
never queried) and check_regression_bounds.py rule A. A per-name quantile
override was rejected: a typo there leaves the key gating, whereas a typo
in an exclusion subtracts nothing and new rule F rejects it, along with an
empty reason, a leftover threshold override and a leftover baseline value.
Derived figures recomputed from the committed baseline: 25 gated keys to
23, detection floor 2.02x-9.43x to 2.02x-9.42x, weakly guarded keys ten to
nine, bound over baseline 102%-843% to 102%-842%. The baseline edit is a
deletion of two entries only, with no value rewritten.
Verified: both previously failing runs replay to zero regressions and exit
0; a tenfold increase injected into each of the 23 remaining keys in turn
is still caught in all 23 cases; rule F was confirmed load-bearing by
stubbing it out, which lets a stale exclusion pass.
The two log-correlation checks have never executed in CI: the workflow
hardcoded --skip-loki, so validate_telemetry.py never constructed
log.trace_id_present or log.trace_id_cross_reference. A green Telemetry
Validation therefore carried no evidence that a log line reaches Loki with
trace context. Drop the flag so both checks run and can fail the job.
Correlation spans four independent legs and a failed check names none of
them, so run-full-validation.sh now prints a per-leg diagnostic after the
suite whenever the checks are enabled:
node per-node debug.log line count, the count matching the injected
trace_id/span_id shape, one sample line, and the severity mix,
so "no log at all", "log level too high" and "no active sampled
span" are distinguishable
mount the container-side listing of /var/log/xrpld, taken with the
collector's own mounts and uid. That image is built from
scratch and carries no shell, so the listing runs in a
throwaway container with --volumes-from, not via docker exec
collector the receiver's watched files, logs-pipeline warnings, and the
internal log-record counters, read from inside the container's
network namespace because that endpoint binds to the
container's own localhost and its port is not published
loki the exact query used, the label inventory, and entry counts for
the stream selector with and without the line filter, so "Loki
has nothing" and "Loki has lines but none carry a trace id" are
distinguishable
The diagnostics are non-fatal by construction: every leg runs in its own
subshell with errexit off, each docker and curl call is guarded, and the
coordinator always returns success. Verified with no containers and no Loki
reachable, with an emptied PATH, and with a leg forced to exit non-zero.
validate_telemetry.py gains a matching diagnostic beside the checks,
following _log_prometheus_metric_names: warnings only, never a check
result. Its stream selector and line filter move into module constants
that the shell diagnostic reads back, so the two cannot drift into
describing different queries.
No check was widened or auto-passed, and LOG_QUERY_WINDOW_SECONDS stays at
four hours; a wider window would let a check pass on a previous run's logs.
Every node the harness starts is a validator, and validators disable
pathfinding: Config.cpp:725-726 zeroes pathSearchMax whenever a
[validation_seed] or [validator_token] section is present, and
run-full-validation.sh writes [validation_seed] into every generated node
cfg (:308) with no [path_search] section to put the default back. So
doRipplePathFind refused every call at RipplePathFind.cpp:48-49 and the
3% ripple_path_find weight bought no coverage at all.
It was not free either. The pathfind.request guard is constructed at
RipplePathFind.cpp:35, above that refusal, so each refused call still
exported a span, and the enclosing rpc.command.ripple_path_find span
carried rpc_status=error. That put a steady 3% error floor into
span_calls_total for STATUS_CODE_ERROR: any error-rate threshold derived
from harness data before this change was measuring the harness rather
than xrpld, and needs re-deriving.
Removing the load makes pathfind.request unreachable, so it moves from
required to optional in expected_spans.json; without that the span check
would fail on every run. Three notes in that file and three in
expected_metrics.json made claims that are now false, two of them citing
line numbers this commit deletes; all six are corrected. The runbook
required/optional count moves 26/15 to 25/16.
Two facts a future reader needs.
First, the weights previously summed to 103, not 100, so every percentage
the docstring stated was wrong: health checks were really 38.8%, not 40%.
Dropping the 3 makes the sum exactly 100 and every stated percentage
correct for the first time. expected_spans.json also carried live
arithmetic off the old total, "25/103 ... roughly 43%", now 25/100 and
42%.
Second, baselines/baseline-timings.json was captured WITH this load. Only
span.rpc.ws_message p50/p95/p99 of the 25 gated keys sees the RPC mix,
and their trip points sit 3.1x to 5.9x above baseline, so the gate will
not fire. But a timing baseline is workload-specific and its profile
field still reads full-validation, so nothing will flag the drift:
refresh it from the next CI run's timings artifact.
Pathfinding now has no coverage in this harness at all. The workload
README section "Pathfinding is not exercised" records that cost, the
manual verification route, and a four-step restore recipe in which steps
1 and 2 alone only reinstate the error floor.
Resolved .codecov.yml: kept the incoming telemetry ignore block, which now
originates on phase-1b. It is a superset of the block this branch carried
(adds *SpanLabels.h) and states the correct reason telemetry files record no
coverage.
The block was added on the StatsD branch, but telemetry sources start here.
Codecov config only flows child-ward, so every branch between this one and
that one kept reporting telemetry files as uncovered patch lines.
Also corrects the rationale. The old comment said telemetry is "not enabled
in coverage builds"; it is enabled — conanfile.py and CMakeLists.txt both
default it ON, and the coverage matrix leg does not turn it off. The reason
these files record no coverage is that the unit-test suite never starts an
exporter.
Adds *SpanLabels.h alongside *SpanNames.h: same compile-time-constant
category, and the existing glob did not match it.
This narrows the patch gap but does not close it — instrumentation added to
consensus and overlay files is still counted, so codecov/patch stays red on
the early phases.
validate_metrics and validate_spans only ever run one direction: read the
contract, ask the backend whether each listed name exists. Nothing looked the
other way, so a metric family or span name the contract omitted was invisible
by construction. Both emitted inventories were already being fetched for the
CI log and neither was compared back, which is how a 345 family metric gap and
7 unknown span names went unnoticed.
Add two reverse checks, metric.reverse_coverage and span.reverse_coverage.
Each names every emitted family the contract never mentions, sorted, one per
line, with counts in the report details.
Warn only, by design. passed is hardcoded True in a single shared builder, so
an unaccounted name cannot turn CI red: downstream branches legitimately add
telemetry an upstream contract has not seen yet, and a hard failure would
redden all of them for doing the right thing.
Bulk families are accounted for declaratively. A new top level
accounted_patterns list in expected_metrics.json holds anchored regexes with a
written reason each, covering the 105 per job type queue gauges, the 70 per job
type histogram families, the 228 overlay per category traffic families, and the
Prometheus scrape plumbing that is not xrpld telemetry. Job type shapes are
reduced structurally because every job type name lowercases to letters only;
traffic categories are enumerated instead, because they contain underscores and
a structural pattern there would swallow unrelated names. Anything outside
these shapes still surfaces.
Exporter shapes are folded before matching, so a histogram triple is accounted
for by an entry written for its base family and is never reported as three
separate gaps. Spans need no pattern list: the reverse check reuses the same
matcher the forward check uses, so a glob such as rpc.command.* covers every
command it expands to, and an optional entry still counts as known.
Also fix the diagnostic these checks feed on: both emitted lists were logged as
a single Python list repr, about 15 kB on one line for 422 families, unreadable
and impossible to compare between runs. Both now print one name per line.
_metric_check_targets now selects groups by testing that the value is an
object, rather than by excluding two key names, so a non group top level key
cannot break it. Output is byte identical: 79 metric plus 5 label checks, same
names in the same order.
The assert / do-not-assert decisions in expected_metrics.json and
expected_spans.json were all correct, but several recorded reasons were not.
Pathfinding is disabled outright on every harness node: Config.cpp:725-726
zeroes pathSearchMax whenever a [validation_seed] or [validator_token] section
is present, run-full-validation.sh writes [validation_seed] for every node and
has no [path_search] override, and both handlers return rpcNOT_SUPPORTED
before constructing a PathRequest.
- pathfind_full_milliseconds no longer claims a probabilistic path, nor
prescribes an explicit ledger index, which cannot help: the config gate
fires before the ledger parameter is read.
- pathfind_fast_milliseconds keeps its hasCompletion argument but now leads
with the config gate, which is the operative blocker.
- The pathfind.compute and pathfind.discover notes and the
pathfind.request to pathfind.compute skip reason no longer blame missing
liquidity. pathfind.update_all now records why its request list stays empty.
- statsd_gauges states the arming precondition: a beast gauge is only as safe
as an observable gauge when its object exists before Application.cpp:1570,
where onCollectionReady arms the registered gauges exactly once.
- Alert wiring claims softened: every rule in rules.yaml is paused.
- The per job type gauge group loses its bogus poll bandwidth reason, and its
regex claim is corrected: there is no running state regex, so 30 of those
gauges have no consumer at all.
- overlay_peer_disconnects has one query consumer, not two.
- Cloud dashboard copies dropped from consumer counts: that tree is ignored by
git and has no tracked files.
- rpc_method_errored_total explains that a refused RPC is a normal return, not
a throw, so the refusals above do not make it fire.
- 09-data-collection-reference.md no longer claims a Prometheus name query in
expected_metrics.json.
No behaviour change: the flattened check name list is byte identical.
_log_prometheus_metric_names exists to make name mismatches between
expected_metrics.json and actual emissions visible in CI logs, but it kept
only names matching 19 hard-coded prefixes. On the last CI run that showed
147 of 422 families, and none of the prefixes covered state_accounting_*,
node_family_*, overlay_peer_disconnects or the pathfind_* histograms, so
the coverage gap the preceding commit closes could not be seen through it
at all.
An allow-list can only ever surface names someone already thought to look
for, which is the opposite of what a discovery aid has to do, so the
filter is removed rather than extended. The whole list is a few kilobytes
of CI log. Sorted, so two runs' output can be diffed directly; the
Prometheus API promises no order.
Dashboards and alert rules reference 186 metrics; the harness asserted 57.
Excluding the 107 per-category overlay-traffic expansions, the meaningful
gap was 20 names. This closes it under the contract file's own doctrine:
assert only what the workload guarantees, and record the rest with a
precise reason.
Asserted 18, taking the metric checks from 61 to 79 and the whole metric
phase from 66 to 84. No pre-existing check name or position changes.
statsd_gauges gains the nine state_accounting_* siblings of the one member
already asserted, plus the two NodeFamily full-below-cache gauges and
overlay_peer_disconnects. All twelve rest on one mechanism the group
description now spells out: on the OTel path a beast gauge is an
Int64ObservableGauge, every instance self-registers in its constructor,
onCollectionReady arms all of them unconditionally, and the armed callback
Observes on every export cycle whether or not set was ever called, so the
series exist at 0. The state_accounting family is set in one unconditional
block in NetworkOPsImp::collectMetrics, and full_transitions is the input
to the NodeStateFlapping alert rule, so the alert's own signal had been
going unverified.
A new job_queue_per_type_gauges group asserts the six per-job-type gauges
that a panel or a rule names literally, jobq_manifest_waiting among them
as the ManifestJobQueueConvoy rule's input. The description records why
those six and not all 105: the guarantee is identical for every
non-special job type, so the discriminator is consumer coverage, and the
remaining names are only reached through topk queries over the family that
do not depend on any single type being present.
Recorded four more in not_asserted rather than asserting them.
pathfind_fast_milliseconds is unreachable for this workload, not merely
rare: reportFast fires only from the doCreate fast pass, which is guarded
by !hasCompletion(), and both ripple_path_find entry points construct the
request with a completion function. Only the path_find subscription
reaches it, and the generator does not use it.
pathfind_full_milliseconds is reachable but only one ledger close after
the request, through PathRequestManager::updateAll, and nothing in the
harness arranges or checks that, so the guarantee is probabilistic.
warn_total and drop_total are resource-manager meters gated on a consumer
crossing the warn or drop threshold; their rpc-pathfinding panels are
correct and render empty only because the condition has not occurred,
which is worth stating because both were briefly mis-read as phantoms.
Runbook check counts updated to match.
Splitting the consensus span labels into their own header moved both
constexpr std::string_view helpers out of ConsensusSpanNames.h, but left
the include they needed behind. The header names string_view nowhere now,
and SpanNames.h already provides the type for the conversion operator.
clang-tidy misc-include-cleaner reports this as an error under
-warnings-as-errors, which fails the clang-tidy job for the whole chain.
The gate could not catch a regression on any sub-millisecond span.
compare_to_baseline.py requires both the percentage and the absolute bound to
breach, and every span shared one flat absolute bound of 10 ms (15 ms for p99)
calibrated for a 5-25 ms band the spans do not occupy. Against the baseline
captured on 2026-08-24, where 18 of the 28 quantiles gated at the time sat
below 1 ms, that bound sat 1.15x to 2000x above the metric it guarded, so the
AND never fired: a 100x regression injected into span.ledger.store.p95 reported
0 regressions and exit 0. Injecting a 10x regression into each key in turn was
caught on only 5 of 28.
Give every gated key its own absolute bound, equal to the distance from its
baseline to hi_next, the edge above the top of the bucket the baseline sits in.
The trip point is then exactly hi_next, so the gate fires only once the reading
clears the bucket above the baseline's own. That is the property a multiple of
the enclosing bucket width cannot provide: after the quantile crosses hi, the
interpolation happens across the next bucket, which on this ladder is up to
eight times wider, so no multiple of the enclosing width bounds the excursion.
Measured with a model-free reachability test, a single bucket crossing can
produce a false regression on 2 of 25 keys under the old flat bound and 0 of 25
under this rule. The smallest catchable regression is 2.02x to 9.43x per key.
The job queue bound had the same shape of problem on three of its four keys
(42x, 47x, 220x before). Defaults now sit at each ladder floor, leaving the
percentage bound operative for a metric that somehow reaches them.
Drop span.ledger.store from the gated surface. Its captured quantiles were
0.005, 0.0095 and 0.0099 ms, which is the ladder's 0.01 ms floor times the
quantile: every sample lands under 10 us, so the reported value does not move
even if each store slows from 2 us to 9 us. No bound can gate it. Presence is
still asserted by expected_spans.json and the integration test, and the rate is
still on the ledger-operations dashboard.
Add check_regression_bounds.py, wired into the same workflow step as the bucket
parity check. It fails when a bound is not the one its own baseline implies,
when a gated key has no override, when the baseline and metric surface disagree,
when the percentage bound would become operative, and when a baseline carries
the ladder floor signature. This gate has now broken three times through the
same drift between ladder, baseline and bounds, so documentation alone is not
enough.
compare_to_baseline.py is unchanged: its existing per-metric override mechanism
already expresses all of this.
A missing, unreadable or malformed input makes that check exit 1 naming the
input, rather than reporting success without having checked anything; only a
placeholder baseline, the documented bootstrap state, still exits 0. Its own
tests cover both halves of that contract plus one case per rule, and run in the
workflow before the check so a broken rule reads as a broken rule.
Copied verbatim from the timings.json produced by the telemetry-validation
run at 6a82fc6f37 (166/166 checks passed), which is the hand-off the
workflow prints for a placeholder baseline.
The numbers confirm why the previous baseline had to be voided. It was
captured 2026-06-05, before the collector's spanmetrics ladder gained
sub-millisecond edges, and its sub-1ms entries were arithmetic on the old
1ms first edge rather than latencies:
span.ledger.store p50/p95/p99 0.5 / 0.95 / 0.99 -> 0.005 / 0.0095 / 0.0099
Exactly 100x, because the old values were quantile x 1ms and the real ones
are quantile x 0.01ms. Since the gate only trips on increases, every
sub-millisecond span was unguarded against a 100x regression.
The job.* pair is back too, recaptured on the re-cut microsecond ladder
(floor 1us): job.acceptLedger.queued.p95 now reads 91.1us as a measurement,
where the voided value of 96.79us was 0.95/0.9926 x 100.
The runbook said a correlated log line was "guaranteed" at info severity.
Info is necessary but not sufficient. Replaced the flat claim with the four
real preconditions, each with the code that enforces it and the failure mode
it produces: telemetry enabled, trace_consensus=1, a valid roundSpanContext_
(SpanGuard::childSpan returns a null guard on an invalid parent), and a valid
plus sampled span context (Log.cpp gates injection on IsValid and IsSampled).
Also noted which harness cfgs satisfy them -- run-full-validation.sh and
integration-test.sh set all three config keys; benchmark.sh deliberately
stays at warning and runs no correlation check.
Second, the two checks this work exists to make pass are not exercised by
CI. The workflow hardcodes --skip-loki, and validate_telemetry.py builds
log.trace_id_present and log.trace_id_cross_reference only inside an
"if not skip_loki" branch, so they are never constructed rather than merely
skipped, and never appear in the report. No workflow runs integration-test.sh
either, so its own check_log_correlation() never runs in CI. Recorded that in
the runbook's CI workflow section and in the workload README, with the local
command that does cover it: run-full-validation.sh without --skip-loki.
The workflow itself is unchanged on purpose. Dropping the flag would make CI
exercise Loki ingestion and filelog mounting for the first time on the same
run that must produce a clean regression baseline, so a red result would not
be attributable.
RPC Response Size on rpc-pathfinding said its p95 was computed "over the
dashboard rate interval", but the query hardcodes [5m]. Every other panel on
that dashboard with a hardcoded [5m] -- RPC Response Time, RPC Response Time
Distribution, both Pathfinding duration panels, the gRPC latency panel and
Pathfinding Compute Duration -- says "over 5 minutes". Matched the clause to
the query. The same edit was applied to the local grafanacloud copy so the
two stay identical; that tree is gitignored, so it is not in this commit.
The io_latency group in expected_metrics.json claimed "all 6 panels that
query it". That 6 was a raw string-occurrence count over the dashboards and
included two panel descriptions. Verified truth: two distinct panels query
the metric, ledger-data-sync "I/O Scheduler Latency p95" and node-health
"I/O Latency", each mirrored in a grafanacloud copy, plus one alert rule in
grafana/provisioning/alerting/rules.yaml. Stated that instead of a count.
ff8629bb11 dropped prefix=xrpld as inert and misleading, but four OTel-path
sites still set it, so the branch contradicted itself.
OTelCollector routes every instrument name through a static formatName()
that only lowercases and maps '.'/space to '_'; the sole read of prefix_ is
the startup log line at OTelCollector.cpp:810. All four instrument factories
funnel through formatName(), so no prefix can reach an exported name.
StatsDCollector does prepend it (StatsDCollector.cpp:551/592/640/715), so the
StatsD example legitimately keeps it.
Removed from the 09 reference's OTel config block and from both
quick-reference setups, and from the cfg integration-test.sh generates. The
StatsD example is unchanged and now states why it keeps the key.
Also corrected run-full-validation.sh: [insight] endpoint was described as
"already matches the built-in default", implying it would matter if it
differed. CollectorManager reads it and hands it to OTelCollector, which also
only logs it; the exporter URL is built in Telemetry::initMetrics() from
[telemetry] endpoint. It is as inert as prefix was.
The panel described itself as showing which categories consume the most
receive bandwidth, and told operators to watch for a category dominating
during ongoing sync, but it ranked raw cumulative counters. That ranks by
lifetime total, so a sync burst days earlier still dominates the bars and
the stated guidance cannot work.
Wrapping the existing selector in rate() does not work: rate() drops
__name__, so the many *_bytes_in series collapse to one labelset and the
query fails with "vector cannot contain metrics with the same labelset".
Grouping with sum by (__name__) does not help either, because Prometheus
strips __name__ from aggregation output regardless of the grouping. Both
were tried against a local Prometheus with synthetic multi-category
counters and both error.
Instead recover the category into a normal `series` label before rating,
and use a subquery so the renamed series can be rated. Verified against
the same synthetic data: rates match a plain single-metric rate control
to within subquery resolution, ranking order is correct, and both the
category and node labels survive.
Also set unit to Bps, matching the ten sibling rate panels in this
dashboard (this was the only byte panel still on decbytes and the only
one plotting raw), and drop "Heatmap" from the title since the panel is
a bargauge.
Note the subquery evaluates the selector at several steps, so it costs
more than the previous instant query across the traffic categories.
Event::notify takes std::uint64_t but the header never included <cstdint>,
relying on it arriving through another include. clang-tidy's include-cleaner
flags it, which fails CI on any branch where this file lands in the
changed-file set.
Fixed here, on the branch that introduced the std::uint64_t parameter, rather
than only downstream where it happened to surface.
CI's clang-tidy job failed on this branch. clang-tidy itself succeeded; the
step that failed was the gate that fails the job when findings exist, and the
uploaded diff named exactly two missing includes.
Event.h calls notify(std::uint64_t) but never included <cstdint>, relying on
it arriving transitively. This is pre-existing on the branch rather than new,
surfaced now because clang-tidy only inspects changed files and this one has
not been touched since.
InboundTransactions.cpp uses uint256 throughout and had no direct include for
it either. The round-request work added further direct uses, which is what
brought the file into clang-tidy's changed-file set.
Both fixes are the ones clang-tidy generated itself, applied with the repo's
angle-bracket include style; the include-style and clang-format hooks accept
the placement unchanged.
Note on why this was not caught before pushing: TIDY=1 clang-tidy was run
locally and passed, but against the compile database under
~/sourceCode/.clangdcache/<worktree>/, which was stale -- generated before
these edits. A stale database makes a local clang-tidy pass meaningless, as
project-map.md warns. CI holds the only current one.
