capture_timings.py decides whether a captured timings file may become a
regression baseline. Every way of getting that wrong is silently green: a
capture that asked Prometheus for nothing still writes valid JSON, and once
accepted it is pasted in as a baseline, still reads as a placeholder, and the
regression gate stays off while the workflow reports it as activated.
Covered: an empty surface is not complete (0 of 0 is 100% by arithmetic), the
minimum ratio is inclusive, null values count as declared but not captured, the
threshold is recorded so a rejected capture can be judged later, and the exit
code follows the flag rather than recomputing the ratio. The empty case has its
own error path because the percentage message divides by the declared count.
Neither this file nor test_validate_telemetry.py ran anywhere before: not in
CI, not in run-full-validation.sh, not in pre-commit. They now run in the
naming job, which is fast and fires on nearly every PR, so a broken harness
surfaces in seconds rather than after an xrpld build.
They run as plain scripts. unittest discover would collect nothing from them,
since they hold bare functions rather than TestCase subclasses, and would exit
0 -- which is why each file fails when it collects no tests. The dependency
install is a separate step, placed after every stdlib-only check so those stay
reachable if PyPI is unavailable.
Every existing rule in this checker runs one way: take a consumer -- a collector
dimension, a Tempo tag, a dashboard label, a doc, an asserted metric name -- and
require it to resolve to the *SpanNames.h constants. Rule H looks closest to the
reverse but is still consumer-side: a constant USED at a call site that no header
defines. Nothing looked the other way.
So deleting a setAttribute from a .cpp and leaving its constant in the header
passed every rule and every compiler, while the telemetry it described stopped
being emitted. The workload validation job would eventually notice, but only when
it happens to run, and its path filter deliberately does not watch daemon .cpp
files -- widening it to 1827 C++ files to catch this would fire a twenty-minute
Docker job on nearly every commit.
Rule M closes that: an L1 constant no code under src/ or include/ references. It
searches all references, not just telemetry call sites, because constants are
passed to helpers, stored in locals and used as attribute VALUES -- a
call-site-only scan would report false positives. Constants referenced only by
test code are reported separately, since a constant exercised by a test but by no
production path is still dead in production.
A WARNING, not a failure, for two reasons that are both real here. Six constants
in this tree are already dead, so failing would redden the branch immediately. And
in a stacked chain a constant legitimately lands one commit before its call site,
so a failing rule would break intermediate branches for a condition that resolves
downstream.
What it reports today, all verified unreferenced across the whole repository and
not just src/include: ConsensusSpanNames.h val::increased, val::decreased and
val::unchanged; SpanNames.h seg::link, attr_val::success and attr_val::error.
Placed here rather than upstream on phase-1c, where the checker was introduced,
because the gap it closes is a workload-harness concern -- the contract asserting
a name nothing emits -- and the harness is this branch's. Putting it on 1c would
also mean union-resolving a 1900-line file across ten merge hops, each an
opportunity to silently drop the metric rules that live downstream.
Ported from a patch written against the sync-diagnostics copy, which carries the
metric-side rules I/J/K/L that this branch does not. The rule itself is purely
span-side; the only shared dependency it needed was iter_sources, which arrived
with those metric rules, so that helper is added here on its own. The rule-L
docstring entry and README row that came with the patch context were dropped --
this branch has no rule L.
Verification: rule M reports 262 constants checked and 6 unreferenced, exit code
still 0 because warnings do not change it; 169 unittest cases pass; the checker
compiles; no metric-rule content leaked in from the patch context (0 occurrences
of METRIC_MACRO_CALL or run_rule_i_metric_literals). Proven non-vacuous by
blanking all three call sites of attr::rpcStatus -- the count went 6 to 7 and
named that constant -- then restoring them and watching it return to 6.
The comments I added with the hierarchy sampling fix and the trigger change ran
to sixteen and twelve lines. The guideline is short and plain English. Rationale,
CI run numbers and the list of which relationships were affected belong in the
commit message, which is where they already are; inline they push the code apart
and go stale as soon as the reasons change.
Trimmed the sampling comment from sixteen lines to four, the re-check comment
from eight to four, _traceql_name_predicate's docstring from fourteen lines of
explanation to three, and the push-trigger comment from twelve to seven. Each
keeps what a reader needs at that line -- what the code does and the one
non-obvious reason -- and drops the history.
Comment-only: 13 insertions against 35 deletions, no statement changed.
Left alone deliberately: this file has ten pre-existing comment blocks longer
than six lines, including one added recently by another party. Rewriting someone
else's comments is not mine to do here, and the guideline is being applied to what
I wrote.
Verification: 7/7 validator tests pass; validate_telemetry.py compiles; the
workflow YAML parses, still carries no branches filter, and still lists 12 paths;
otel-naming exits 0.
An empty metric surface counted as a complete capture. build_query_plan
returns an empty plan without complaining for any config that yields no
gated keys, so pointing --metrics at the wrong file exits 0 and hands the
paste-me path a metrics:{} artifact to offer as the next baseline. Nothing
about such a run is evidence the pipeline works, so declared == 0 is now a
failure rather than vacuously complete.
The bounds checker also raised AttributeError on a baseline entry that is
not an object, instead of naming the key. A validator whose job is to catch
a malformed contract should report it, not crash on it.
Test cleanup is bound to its own temp tree, so a loop no longer leaves five
of six directories behind.
The regression baseline is bootstrapped by copying a CI artifact. The workflow
tested only that timings.json existed, then printed it verbatim under a heading
inviting the reader to paste it in as the new baseline.
capture_timings.py writes that file and only then enforces --min-capture-ratio,
so an incomplete capture leaves a file that exists but covers fewer keys than
the contract declares. The verdict lived in CAPTURE_EXIT, a shell variable local
to run-full-validation.sh that no other program could read. So on a placeholder
baseline plus a thin capture, CI offered an incomplete artifact as the next
baseline, and pasting it narrowed the gate with nothing reporting that it had.
That is the failure shape this harness keeps producing: a degraded result that
looks exactly like a good one.
The artifact now carries its own completeness, next to metrics:
"capture": { "declared": 20, "captured": 20, "min_ratio": 0.5, "complete": true }
complete is the same condition the producer exits 0 on, computed once with the
exit code read off it, so the flag and the status cannot drift apart. Any
consumer can now tell a complete capture from a thin one, not just CI.
Both paste-me paths refuse rather than warn: the workflow prints the counts and
an error annotation with no JSON, and the comparator explains on stderr while
leaving stdout empty, so a redirect cannot produce a plausible-looking file. A
warning above a copyable block is still a copyable block, and a reader who has
just hit a red gate is already predisposed to re-baseline. A missing capture
block fails closed.
Refusal is scoped to bootstrapping a baseline, not to comparing against one, so
artifacts captured before this change still replay: verified against the run the
current baseline came from, which carries no capture block and still reports 0
regressions. An injected regression is still caught, and the gated surface is
unchanged at 20 keys with 5 excluded.
Two problems, both about coverage this workflow claims to have and does not.
The push trigger gated on branch NAME as well as path, and GitHub ANDs the two.
Branch names are not something this repository controls, so a push to any branch
outside "pratik/otel-phase*", "feature/otel-*" or "feature/telemetry-*" was never
dispatched -- not queued, not skipped, no run to look at. That is not a
theoretical gap: two rounds of harness fixes on pratik/otel-sync-diagnostics
produced no signal at all before anyone noticed the workflow had never started.
The branches filter is removed; the paths already express the real question.
The path list was also incomplete in a way that matters more than it looks. The
span-name and metric-name headers are the wire contract this harness asserts
against by literal string, and the convention colocates each one with the class
it serves -- so eight of the ten *SpanNames.h headers live under consensus/,
overlay/, app/ledger/, app/main/, app/misc/, rpc/ and tx/, none of which was
matched. Renaming a span constant therefore compiled clean, emptied the
assertions and triggered nothing. Matched now by filename, "**/*SpanNames.h" and
"**/*MetricNames.h", so future headers are covered wherever they land. Added for
the same reason: include/xrpl/beast/insight (the interface headers decide what
the collector can publish, so they move the metric surface as surely as the
implementation), src/tests/libxrpl/telemetry (the GTests pinning those
constants), and the two checker directories that gate this surface in CI.
Second, the span hierarchy. Each span entry documents its parent, and a separate
list holds the pairs the validator actually checks in Tempo. Those had drifted
apart: 18 parentings were documented, 7 were checked. A span that stops nesting
under its parent -- which is what a detached guard does -- leaves every span and
every attribute intact, so no other check in this harness notices; the trace
simply stops being readable as one operation. Eleven pairs are added, each one
where both ends emitted on a real run: rpc.http_request -> rpc.process, the three
txq parentings, and seven consensus ones under consensus.round and
consensus.establish. Fourteen of eighteen are now asserted; the four still
skipped are the wildcard rpc.command.* families and pathfind.compute.
Three notes were also factually wrong, all repeating one mistake. They said
rpc.process and rpc.http_request cannot appear because that path is HTTP-only
while the load generator is WebSocket-only. The premise is right, the conclusion
is not: both appear on every run, five traces each, because
run-full-validation.sh polls each node's HTTP port with curl for readiness and
validated-ledger progress (:449, :502). Those polls take the HTTP path. A reader
acting on the old text would have gone looking for a way to make the harness
speak HTTP that it already speaks. The rpc.process -> rpc.command.* skip reason
inherited the same error and additionally claimed the WebSocket equivalent is
"asserted above instead", which it is not -- that one is skipped for the same
wildcard limitation. All three now state the real blocker, which is that
_validate_parent_child resolves a wildcard child to a single literal probe.
Both HTTP spans stay optional rather than being promoted: the curl polls are
harness scaffolding, not workload, and a future change to how the script waits
for a node could legitimately remove them.
Verification: JSON parses; 18 relationships, no duplicates, every non-wildcard
endpoint resolves to a declared span entry; counters still 41 span types and 62
unique attributes; workflow YAML parses, has no branches key, keeps
workflow_dispatch, and every new glob was checked against the tracked file list
with a matcher that reproduces GitHub's ** semantics; otel-naming exits 0;
pre-commit clean on both files. The eleven new assertions are proven only to the
extent that both ends emitted on run 32969481032 -- that a child is findable
INSIDE the parent's fetched trace is what CI will now decide.
Refreshes baselines/baseline-timings.json from run 32964262700 at 8418d474a7,
byte-identical to the CI artifact. The previous baseline was captured at
6a82fc6f37, before the path-finding load was removed from the workload, so it
described a load shape the harness no longer runs.
Every absolute bound is re-derived, because the rule is hi_next minus baseline
and the baselines moved.
Three more keys stop being gated: span.tx.apply.p50, span.ledger.build.p50 and
span.consensus.ledger_close.p50. This is the rule the previous commit recorded
being applied, not a new exception -- a key is gateable only when its
run-to-run spread fits inside its bound.
The evidence is span.tx.apply.p50, which read 0.7917 ms in the old baseline and
0.00597 ms in this one. That is a 132x move between two runs of the SAME
workload. The old value happened to land mid-distribution, so hi_next minus
baseline gave a 4.21 ms bound that absorbed the spread; the new value lands in
the ladder's first bucket, so the same rule gives 0.0440 ms and cannot survive
one. Whether the gate functioned was decided by where in the distribution the
captured run happened to fall, which is not a threshold in need of tuning.
Measured spreads across four runs agree: 364x, 25.3x and 5.9x respectively.
All five excluded keys share one shape -- a baseline landing in the ladder's
low buckets, where the derived bound is tiny, together with large run-to-run
spread. Single-run baselines cannot support them; a multi-run baseline, or a
spread measurement captured alongside the baseline, is what would let them be
gated again. Not attempted here.
Both runs that would have reddened CI now replay clean, and an injected 10x
regression is still caught on 19 of the 20 remaining keys, 20 of 20 at 20x.
The exception is job.acceptLedger.running.p95, whose baseline fell while its
hi_next did not, moving its floor to 16.28x. It stays gated with that floor
recorded beside the other weak keys.
Also makes the bounds checker report a zero or negative baseline as a named
rule failure instead of dividing by it and raising.
The regression gate has been red on runs with no code change. Only two of
the 25 gated keys ever tripped, both on the same span and never together:
run 32862589645 failed p99 at 25.8750 ms against a 1.0600 ms baseline
(+2341%), run 32867433073 failed p95 at 0.7500 ms against 0.2404 ms
(+212%), and in each run the other quantile sat well inside its own bound.
A real slowdown would move both. This is variance, not a defect.
Measured across four CI runs:
span.ledger.validate.p50 0.0484 to 0.0778 ms 1.6x spread kept
span.ledger.validate.p95 0.1281 to 0.7500 ms 5.9x spread excluded
span.ledger.validate.p99 0.3875 to 25.8750 ms 66.8x spread excluded
Both excluded quantiles reach past their trip point on a healthy run. The
mechanism is arrival timing, not slow code: the span opens only once a
quorum-completing validation arrives (LedgerMaster.cpp:987, inside
checkAccept, past the early return) and wraps the promotion work that
follows, so one slow consensus round dominates the tail of a 3m rate
window and which round that is differs every run.
Widening is not available and must not be attempted later: tolerating
25.8750 ms against a 1.0600 ms baseline needs a bound of about 24.8 ms,
which gates nothing. A bound admitting every healthy run's worst case
admits every regression too. p50 stays gated; it is stable.
THE GENERAL RULE, recorded so this does not recur: an absolute bound
derived as hi_next minus baseline comes from the histogram ladder, so it
budgets for quantization noise and for nothing else. It knows nothing about
how far a metric moves between runs on identical code. Before gating any
key, check its observed maximum across several runs against its trip point
and gate it only with margin. Spread alone proves nothing: tx.apply.p50
swings 364x and never fires, because its 5 ms trip point absorbs the range.
Of the 23 keys still gated the worst reaches 0.67 of its trip point.
Mechanism: spans.names lists span names while _quantiles is shared, so
dropping two quantiles of one span cannot be expressed by deleting a name.
regression-metrics.json gains an excluded_keys map from a flat key to the
reason it is not gated, subtracted by both prom_queries.py (so the key is
never queried) and check_regression_bounds.py rule A. A per-name quantile
override was rejected: a typo there leaves the key gating, whereas a typo
in an exclusion subtracts nothing and new rule F rejects it, along with an
empty reason, a leftover threshold override and a leftover baseline value.
Derived figures recomputed from the committed baseline: 25 gated keys to
23, detection floor 2.02x-9.43x to 2.02x-9.42x, weakly guarded keys ten to
nine, bound over baseline 102%-843% to 102%-842%. The baseline edit is a
deletion of two entries only, with no value rewritten.
Verified: both previously failing runs replay to zero regressions and exit
0; a tenfold increase injected into each of the 23 remaining keys in turn
is still caught in all 23 cases; rule F was confirmed load-bearing by
stubbing it out, which lets a stale exclusion pass.
The two log-correlation checks have never executed in CI: the workflow
hardcoded --skip-loki, so validate_telemetry.py never constructed
log.trace_id_present or log.trace_id_cross_reference. A green Telemetry
Validation therefore carried no evidence that a log line reaches Loki with
trace context. Drop the flag so both checks run and can fail the job.
Correlation spans four independent legs and a failed check names none of
them, so run-full-validation.sh now prints a per-leg diagnostic after the
suite whenever the checks are enabled:
node per-node debug.log line count, the count matching the injected
trace_id/span_id shape, one sample line, and the severity mix,
so "no log at all", "log level too high" and "no active sampled
span" are distinguishable
mount the container-side listing of /var/log/xrpld, taken with the
collector's own mounts and uid. That image is built from
scratch and carries no shell, so the listing runs in a
throwaway container with --volumes-from, not via docker exec
collector the receiver's watched files, logs-pipeline warnings, and the
internal log-record counters, read from inside the container's
network namespace because that endpoint binds to the
container's own localhost and its port is not published
loki the exact query used, the label inventory, and entry counts for
the stream selector with and without the line filter, so "Loki
has nothing" and "Loki has lines but none carry a trace id" are
distinguishable
The diagnostics are non-fatal by construction: every leg runs in its own
subshell with errexit off, each docker and curl call is guarded, and the
coordinator always returns success. Verified with no containers and no Loki
reachable, with an emptied PATH, and with a leg forced to exit non-zero.
validate_telemetry.py gains a matching diagnostic beside the checks,
following _log_prometheus_metric_names: warnings only, never a check
result. Its stream selector and line filter move into module constants
that the shell diagnostic reads back, so the two cannot drift into
describing different queries.
No check was widened or auto-passed, and LOG_QUERY_WINDOW_SECONDS stays at
four hours; a wider window would let a check pass on a previous run's logs.
The gate could not catch a regression on any sub-millisecond span.
compare_to_baseline.py requires both the percentage and the absolute bound to
breach, and every span shared one flat absolute bound of 10 ms (15 ms for p99)
calibrated for a 5-25 ms band the spans do not occupy. Against the baseline
captured on 2026-08-24, where 18 of the 28 quantiles gated at the time sat
below 1 ms, that bound sat 1.15x to 2000x above the metric it guarded, so the
AND never fired: a 100x regression injected into span.ledger.store.p95 reported
0 regressions and exit 0. Injecting a 10x regression into each key in turn was
caught on only 5 of 28.
Give every gated key its own absolute bound, equal to the distance from its
baseline to hi_next, the edge above the top of the bucket the baseline sits in.
The trip point is then exactly hi_next, so the gate fires only once the reading
clears the bucket above the baseline's own. That is the property a multiple of
the enclosing bucket width cannot provide: after the quantile crosses hi, the
interpolation happens across the next bucket, which on this ladder is up to
eight times wider, so no multiple of the enclosing width bounds the excursion.
Measured with a model-free reachability test, a single bucket crossing can
produce a false regression on 2 of 25 keys under the old flat bound and 0 of 25
under this rule. The smallest catchable regression is 2.02x to 9.43x per key.
The job queue bound had the same shape of problem on three of its four keys
(42x, 47x, 220x before). Defaults now sit at each ladder floor, leaving the
percentage bound operative for a metric that somehow reaches them.
Drop span.ledger.store from the gated surface. Its captured quantiles were
0.005, 0.0095 and 0.0099 ms, which is the ladder's 0.01 ms floor times the
quantile: every sample lands under 10 us, so the reported value does not move
even if each store slows from 2 us to 9 us. No bound can gate it. Presence is
still asserted by expected_spans.json and the integration test, and the rate is
still on the ledger-operations dashboard.
Add check_regression_bounds.py, wired into the same workflow step as the bucket
parity check. It fails when a bound is not the one its own baseline implies,
when a gated key has no override, when the baseline and metric surface disagree,
when the percentage bound would become operative, and when a baseline carries
the ladder floor signature. This gate has now broken three times through the
same drift between ladder, baseline and bounds, so documentation alone is not
enough.
compare_to_baseline.py is unchanged: its existing per-metric override mechanism
already expresses all of this.
A missing, unreadable or malformed input makes that check exit 1 naming the
input, rather than reporting success without having checked anything; only a
placeholder baseline, the documented bootstrap state, still exits 0. Its own
tests cover both halves of that contract plus one case per rule, and run in the
workflow before the check so a broken rule reads as a broken rule.
The microsecond ladder's first edge was 100us, which sat ABOVE the mass of
every instrument using it. Measured on devnet: 99.3% of job_queued_us
samples, 92.5% of job_running_us and 90.4% of getobject_lookup_us fell in
that first bucket. histogram_quantile then interpolated inside bucket 0 and
returned `quantile / fraction_in_bucket_0 x first_edge` -- p75/p95/p99 of
job_queued_us read 75.52/95.66/99.69us against a prediction of
75.53/95.67/99.70. Three-decimal agreement: those panels were reporting
arithmetic on the bucket edge, not latency.
The fix was already half-written. kSubMillisecondBoundaries had been parked
in MetricsRegistry.cpp as [[maybe_unused]] with a comment noting exactly this
problem for nodestore reads. Its edges are now folded into kMicrosecondBuckets
rather than deleted, so the parked intent is carried forward: 1..1000us
resolution where the mass is, upper edges unchanged so multi-second stalls
stay measurable.
Also moves the GetObject count and charge ladders into HistogramBuckets.h, so
all five ladders have one owner and one set of invariant tests (29 now).
Adds check_bucket_parity.py, wired into the existing OTel naming workflow.
The C++ millisecond ladder and the collector's spanmetrics ladder are
specified to agree over their shared range; they were identical when shipped,
then the collector side alone was extended and nothing noticed for eleven
phases. The check asserts containment rather than equality, because jobs
outlive spans -- jobq_updatepaths averages ~60s, which no span approaches, so
demanding equality would force a ceiling that censors it. Verified it rejects
a missing collector edge, a bogus in-range edge, and a return to the 5s
ceiling.
ledger-data-sync's "Job Queue Wait p95 By Type" moves off the beast
jobq_*_q_milliseconds pair onto job_queued_us filtered by job_type. Those
beast metrics are ms-quantised at the source (Event rounds up to a whole
millisecond), so 94-100% of their samples sat in the first bucket and no
ladder change could fix them. Note the label values are camelCase
(job_type="ledgerData"), not the lowercase metric-name fragments.
Both histogram-fed alert thresholds re-validated and left unchanged, with the
measured basis recorded so neither gets tuned against the old artefact: only
0.0022% of job_queued_us samples exceed the 1s threshold, and every edge
bracketing the 1000ms ios_latency threshold survived the ladder change.
Docs: the rpc_size "known issue -- tracked separately" notes in the runbook
and 09-data-collection-reference are now resolved notes, the stale 10-edge
span_duration bucket list is corrected to the collector's real 20, and the
runbook gains a "Reading A Histogram Percentile" section covering both
saturation traps and the expected discontinuity after a ladder change.
check-otel-naming fails Rule D on this branch: ledger-data-sync.json
aggregates by xrpl_work_item, which nothing in-tree emits because
perf-iac's alloy pipeline stamps it, so it cannot be derived from
*SpanNames.h. The sibling perf-iac identities xrpl_branch and
xrpl_node_role are already allowlisted; this one was missed when the
by() clauses were reintroduced.
Add it to EXTERNAL_INFRA_LABELS alongside them. Verified by removing
the entry again, which restores the failure.
Both the CMake option and the Conan recipe default telemetry on, so the
build worked, but this workflow exists to exercise telemetry and should not
depend on a default it does not control. If that default ever flipped, the
binary would build cleanly and then every span and metric assertion would
fail for a reason no log names.
Stated explicitly, the failure mode inverts: CMakeLists.txt does
find_package(opentelemetry-cpp CONFIG REQUIRED) under this option, so a
dependency graph without it fails at configure time with a clear message.
The shared build-deps action is left alone deliberately — forcing the Conan
option there would change every other workflow that uses it.
A validation run timed out at Step 3 with only 4 of 5 nodes proposing, and
the reason was unrecoverable afterwards. Two gaps caused that.
The node-log artifact collected `node*/debug.log` but not `node*/stdout.log`.
A node that dies before its log sink opens never writes a debug.log at all,
so stdout is the only place its reason survives — and that file is written by
the harness and read by nothing, so it went to the runner and was discarded.
The failing node's log was simply absent from the artifact.
The readiness loop also fetched each node's `server_state` and threw it away,
reporting only a count. "4/5 nodes proposing" says a node is missing but not
which one, so there is nothing to grep for even once the logs are kept. The
timeout now names each node that is not proposing along with the state it
last reported, distinguishing a node that answered with a non-proposing
state from one whose RPC port did not answer at all.
Neither change affects a healthy run: the accumulator resets each attempt and
stays empty while every node is proposing.
The integration-test sections, alert provisioning headers and a naming-check
test comment were indexed by rollout phase and task numbers defined only in
planning documents outside the shipped tree. Each now names what it covers.
The harness manifests asserted things the code cannot produce and missed most
of what it does. Two assertions were failing every run, and the metric set
covered 16 of the ~41 emitted names.
expected_spans.json: rpc.process was required with rpc.ws_message as its
parent, but it is created only in ServerHandler::processRequest() on the HTTP
path, so a WebSocket-only workload never produces it -- it is now optional and
parented to rpc.http_request, and the rpc.process -> rpc.command.* edge is
skipped with the real reason instead of a coroutine-context-loss diagnosis that
was never the cause. Adds the missing rpc.ws_upgrade span, corrects four
parents (consensus.mode_change, pathfind.request, and update_positions/check,
which are children of consensus.establish rather than consensus.round), and
demotes conditionally-set attributes out of required_attributes so a healthy
run stops failing. Counts recomputed from the file: 41 span types, 62 unique
required attributes.
expected_metrics.json: 16 -> 52 asserted entries across the job-queue, RPC
method, reduce-relay, overflow and validation families, plus the fifteenth
dashboard uid. Metrics the harness workload cannot exercise -- erroring RPC,
ledger-mismatch, TxQ overflow, and the lazily-created getobject_* instruments
-- are listed in a not_asserted group the validator skips, rather than as
assertions that would fail on a healthy node.
The workflow's push trigger listed two globs matching nothing
(include/xrpl/basics/Telemetry*.h, src/xrpld/app/misc/Telemetry*), so no C++
telemetry change ever triggered validation. Replaced with the paths the code
actually lives in, including src/libxrpl/beast/insight/** for the insight
export path the harness depends on. The four inert workflow_dispatch inputs are
now labelled UNUSED rather than looking like working knobs.
Docs: the workload README described a StatsD dirty-flag mechanism under a
member name that does not exist, on a code path the harness never uses -- it
sets [insight] server=otel, so gauges export through an observable-gauge
callback every cycle. Adds the missing txq-burst phase, reconciles three
different dashboard counts, and drops "posts summary to PR", which the workflow
has no permission to do. The runbook's phase-10 section loses the last
sampling_ratio reference (not a config key), gains a Regression Gate and CI
subsection covering the gate that can fail CI, and its compose-logs command now
names the workload compose file. cmake --preset default is left for a separate
change: no CMakePresets.json is tracked, so it is wrong everywhere it appears.
Also drops the dead exporter=otlp_http key the harness wrote into every node
config, and stops capture_timings.py defaulting --profile to a profile that
does not exist.
Three doc conflicts, all inside Phase 10's own sections. Resolved by keeping
phase-9's corrections and re-applying phase-10's ownership of those sections on
top, rather than taking either side wholesale:
06 §6.8.3 — Status now says Phase 10 is implemented on THIS branch (phase-9's
copy said "not merged into this branch", true there, false here). The stale
"71 checks" enumeration and "Current Status" list are replaced by phase-9's
dynamic-count description plus a Known Gaps list that states the real reason
the rpc.process hierarchy is unassertable: rpc.process is HTTP-only, so a
WebSocket-only workload never produces it. Phase-9's CI Deliverable subsection
is carried through; exit criteria keep phase-10's tick state with phase-9's
corrected wording.
06 §6.8.3 Architecture — the automatic merge had resurrected phase-10's stale
2-node cluster prose and diagram over phase-9's 5-node correction. Reconciled
to phase-10's topology (native xrpld processes vs the containerised backend,
which is accurate) with phase-9's node count: 5 validators, and the collector
labelled OTLP + filelog rather than StatsD, which the config has never had.
Dropped the "all 26 metrics required" label in favour of the manifest.
06 §6.8.3 Key Implementation Details — two claims corrected against the code.
The StatsD m_dirty gauge fix describes a member that exists nowhere in the
repo, and the harness sets [insight] server=otel anyway, so gauges export
through an observable-gauge callback. The tx.receive attribute keys are bare
suppressed and tx_status, not dotted, and tx_status is set only on the
reject/known-bad/dropped paths, so it is absent on a successful receive.
09 §5c — kept phase-10's four-column table shape, with phase-9's corrected
counts: 40 of 41 emitted spans, 67 required attributes, 14 of 15 dashboards.
Phase10_taskList — dynamic inventory totals, the real RPC span trees, and exit
criteria ticked where the code on this branch closes them. Per-RPC timings are
recorded as not gated: regression-metrics.json defines only spans and job_queue.
Verified every hunk of the phase-10 diff falls inside a Phase-10-owned section
(06 §6.8.3, 09 §5c, Phase10_taskList, and the runbook's appended Phase 10
sections); no phase-9-owned text is modified from this branch.
