The runbook's nodestore_state table kept AppMetricGauges.cpp as the source
column, which is where those gauges are actually registered on every branch
in the chain; the incoming side named MetricsRegistry.cpp, which registers
none of them. Took the incoming side's clearer wording for the sweep row.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The sanity bound required the value to be strictly positive, which rejected
three of the four readings the gauge can legitimately produce: a negative
count once the validator list has expired, the -1 sentinel for "no published
list fetched", and +inf for a config-listed list that never expires.
The negative case is the one that matters. It is the signal that expiry has
already passed, so the gauge deliberately does not clamp at zero, and a gate
that rejects it would fail exactly when an operator most needs the reading.
The -1 sentinel already violated the bound and had simply never been hit,
because the validation cluster always fetches a published list.
The floor is now a century, which still catches a broken clock. Detecting the
unsigned wrap this bound used to hide moves to the MetricsRegistry::daysUntil
unit tests, which are deterministic and do not need a running cluster.
The collector reads the per-node directory off the log file path and stamps it
as the Loki label service_instance_id, so the directory name has to equal the
node's own [telemetry] service_instance_id or log lines carry a node name that
no trace or metric shares and nothing joins.
Both harness scripts disagreed with themselves: run-full-validation.sh wrote to
node$i while setting validator-${i}, and benchmark.sh wrote to node$i while
setting bench-node-${i}. Rename the directories to match the ids rather than
the reverse, so no existing trace or metric label value moves and no harness
expectation has to be re-checked. Only path references are renamed; the
human-readable "node$i" in log and error messages is left as prose.
The config template is not rendered by any script, so its DATA_DIR
documentation gains a note about the same constraint instead.
Also rename the deprecated otlphttp/filelog collector component names in the
harness scripts and docs.
The four external-parity bounds checks each ran a single Prometheus instant query
and failed on an empty result. The metric checks that run earlier poll
/api/v1/series, which returns a series regardless of staleness, but a bounds
check needs the sample value and so cannot use that endpoint. This file's own
docstring records the consequence: a beast::insight gauge that stops changing can
fall out of an instant query while /api/v1/series still returns it, so one
attempt is not enough to call the series absent.
Poll to the same deadline the metric checks use. A Prometheus error is raised
rather than retried, because a rejected query never becomes valid and retrying it
only burns the full timeout.
The check reported span.hierarchy.<parent>-><child> and a message reading
"Found <child> as child of <parent>" on the strength of both names appearing
somewhere in the same trace. A span parented by something unrelated passed, so
the one property the check exists to prove was never tested.
It now walks the child's parentSpanId chain looking for a span matching the
parent name. Ancestry rather than a direct edge, because all 21 declared
relationships are worded as the parent containing the child, so a scope
appearing in between is a refactor and not a broken relationship. Span ids are
compared as opaque strings: both fields come from the same Tempo response and
share its encoding, so nothing here depends on whether that is hex or base64.
Co-occurrence is still the search filter, which is what lets a conditional
child be found in an older trace instead of only the newest ones.
Verdicts are separated because they send the reader to different places: a
child that is present but not under the parent is a hierarchy bug, a chain
running into a span the trace lacks is one that never reached Tempo, and an
unusable parent span is neither. A definite negative outranks an indefinite
one, and one trace proving ancestry settles the relationship.
Tests cover each verdict plus the cross-trace and cyclic-chain cases, and each
one was checked against the specific defect it names. The runner now fails when
it collects no tests and reports SystemExit, both of which otherwise produce a
silent pass.
The pathfind.request skip_reason said only the child side handles globs. Both
sides do now; the blocker is the literal parent name in the Tempo query, so the
skip itself stands.
Two conflicts.
RCLConsensus.cpp: upstream restructured makeAcceptSpan so the accept span's
attributes sit behind if (*span). This branch's own contribution there is the
consensus round-duration histogram, which is kept -- placed inside the
telemetry guard but OUTSIDE the span-liveness test, because a metric must
still record when the trace category is disabled or the span was not created.
The duplicated attribute lines on this side are dropped; the guarded block
upstream added supersedes them.
MetricsRegistry.cpp: kept this branch's JobQueue.h include, which it uses.
Its Journal.h include was dropped as a duplicate -- the file already includes
that header higher up, with a comment explaining why it is unguarded, and
readability-duplicate-include is fatal under WarningsAsErrors.
The comments I added with the hierarchy sampling fix and the trigger change ran
to sixteen and twelve lines. The guideline is short and plain English. Rationale,
CI run numbers and the list of which relationships were affected belong in the
commit message, which is where they already are; inline they push the code apart
and go stale as soon as the reasons change.
Trimmed the sampling comment from sixteen lines to four, the re-check comment
from eight to four, _traceql_name_predicate's docstring from fourteen lines of
explanation to three, and the push-trigger comment from twelve to seven. Each
keeps what a reader needs at that line -- what the code does and the one
non-obvious reason -- and drops the history.
Comment-only: 13 insertions against 35 deletions, no statement changed.
Left alone deliberately: this file has ten pre-existing comment blocks longer
than six lines, including one added recently by another party. Rewriting someone
else's comments is not mine to do here, and the guideline is being applied to what
I wrote.
Verification: 7/7 validator tests pass; validate_telemetry.py compiles; the
workflow YAML parses, still carries no branches filter, and still lists 12 paths;
otel-naming exits 0.
The conjunction query works. Run 33062418036 proved it on real Tempo: both
hierarchies that newest-N sampling made unassertable now PASS --
txq.accept -> txq.accept_tx and ledger.acquire -> ledger.acquire.txtree -- along
with every other literal-child pair. Only the two wildcard children failed, and
not because of the sampling change.
They failed with HTTP 400, "invalid TraceQL query: parse error at line 1, col 68:
invalid char escape". _traceql_name_predicate built the pattern with re.escape,
giving name=~"rpc\.command\..*", and TraceQL's string lexer refuses a backslash
escape it does not recognise -- the query never reached the regex engine at all. A
literal dot is now written as the character class [.], which carries no backslash
for the lexer to refuse while still meaning a literal dot to the engine behind it.
Leaving the dots bare would have parsed, but would match any character in those
positions, which is the looseness _span_name_matches exists to avoid.
The builder now also rejects a span name containing anything outside
lower_snake_case, dots and the glob star, rather than passing it through
unescaped. Every name in the contract is of that shape, so this changes nothing
today; it exists because the failure mode it guards against is exactly the one
above -- a character that means something to one layer and something else to the
next, discovered only from a 400 in CI.
Worth recording why the tests did not catch this. The stub evaluated the pattern
with Python's re, which accepts \. happily, so it modelled the regex engine and
not the query lexer sitting in front of it. A stub is only as good as the layer it
imitates, and the layer that rejected this was one the stub did not represent. The
new test therefore asserts the property the lexer enforces -- that no backslash
appears in the predicate at all -- rather than any particular spelling, plus that
the pattern still accepts rpc.command.fee and still rejects a near-miss whose
separators are not dots.
Verification: 7/7 tests pass, and the new one was watched failing first with the
exact string Tempo rejected, name=~"rpc\.command\..*"; the full query the check
now builds was printed and confirmed backslash-free; validate_telemetry.py
compiles. Three unrelated files in this worktree are another party's live work and
were left unstaged.
Brings in the hierarchy-check sampling fix: the check now asks Tempo for traces
containing both parent and child rather than inspecting the three newest parent
traces, so a child conditional on a state the workload rarely reaches is found
wherever it occurred. Merged clean, no conflicts, no resolution decisions.
This unblocks ledger.acquire -> ledger.acquire.txtree on this branch, which was
skipped for exactly that sampling problem and is un-skipped in the next commit.
The hierarchy check searched the parent span and inspected the three newest
traces it returned. That is wrong whenever the child is conditional on a state
the workload only sometimes reaches: the parent fires constantly, so its newest
traces are the ones LEAST likely to carry a rare child. Three relationships had
been skipped as unassertable for exactly this, and in none of them was the child
missing -- each emitted traces of its own and simply was not in the three most
recent parent traces.
The check now issues a second query, a TraceQL trace-level conjunction of the
parent and child name predicates, and inspects those traces. Tempo searches its
whole retention for co-occurrence instead of leaving the answer to which traces
happen to be newest. The parent-only query is kept and still runs first, so "the
parent stopped being emitted" stays a distinct failure from "the parent is there
but the child never co-occurs" -- they mean different things to whoever reads the
report, and collapsing them would lose that.
The returned traces are still verified with _span_name_matches rather than the
query result being trusted on its own. Tempo has already guaranteed
co-occurrence, so this is redundant on the happy path; it is kept because it
keeps the glob semantics in one place and means a wrongly built query cannot
silently pass.
_traceql_name_predicate handles the wildcard contracts. TraceQL has no glob
operator, so `rpc.command.*` is sent as name=~"rpc\.command\..*" with the dots
escaped -- unescaped they would match any character in those positions, which is
the looseness _span_name_matches exists to avoid.
Two entries follow from the fix. txq.accept -> txq.accept_tx is asserted again:
its child is created inside the queued-transaction loop behind
`if (feeLevelPaid >= requiredFeeLevel)` (TxQ.cpp:1530) while the parent fires on
every close (:1499), which was the whole reason it failed. txq.enqueue ->
txq.batch_clear stays skipped but for ONE reason now instead of two -- its child
never fires at all under this workload, needing an account with a supersedable
batch, so it is purely a workload gap and needs nothing further from the
validator. The third, ledger.acquire -> ledger.acquire.txtree, lives on the
sync-diagnostics branch and is un-skipped there once this merges forward.
Written test-first, and the first test this module has had. The failing test
reproduces the exact CI message, "txq.accept_tx not found in txq.accept traces",
against a stubbed Tempo whose corpus holds the child only in a trace outside the
newest three. Three sibling tests guard the ways this could be "fixed" wrongly: an
absent child must still fail, a missing parent must still name the parent rather
than the child, and a wildcard child must be satisfied by any family member. The
stub records the queries issued, so the conjunction is asserted rather than
assumed. A stub rather than a live Tempo because the behaviour under test is which
traces the check ASKS FOR -- a passing query against real data proves the data
co-operated, not that the query was right.
The first run of those tests failed for the wrong reason: my stub's name-predicate
regex also matched the resource.service.name="xrpld" term every query carries and
so demanded a span literally named "xrpld". Fixed in the stub, with the lookbehind
commented as load-bearing, before touching production code.
Verification: 4/4 tests pass, and the failing one was watched failing first with
the production message; the issued queries were printed and confirmed to contain
the conjunction; validate_telemetry.py compiles; expected_spans.json parses;
21 relationships, 16 asserted and 5 skipped; counters still 41 span types;
otel-naming exits 0. Three unrelated files in this worktree are another party's
live work and were deliberately left unstaged.
Brings phase-10 up to 29673de531, three commits: a Loki-diagnostic count fix with
Tempo errors made visible, a baseline recapture that stops gating keys whose
run-to-run variance dominates their bound, and WebSocket reply correlation with
silent placeholder data removed.
Merged clean -- git reported no conflicts, so no resolution decisions were made
here. The 14 incoming files are all harness and docs: six .py, three .md, three
.json, two .sh. No runtime C++ and no include-line changes.
The baseline recapture resolves the regression gate that reddened the previous
run on this branch. That run reported span.consensus.ledger_close p95 1.62ms and
p99 7.14ms against a baseline of 0.49 and 0.93; the recaptured baseline puts p95
at 0.783 with a 5ms trip point and p99 at 2.03 with a 10ms trip point, so both
readings now sit inside the gate. That matches the diagnosis recorded at the
time: everything changed between the last green gate and that failure was a .py,
a .md and a GTest file, with zero runtime C++, and p50 had improved while only
the tail moved -- variance, not a slowdown. The incoming commit reaches the same
conclusion from the other side, excluding three p50 keys after measuring spreads
of 391.8x, 20.7x and 6.1x across three runs.
Three defects in the harness's own instrumentation, all of the same shape: a
failure that reads as an absence.
The Loki diagnostic reported "unavailable entries" rather than a count. It
issued an unaggregated count_over_time, and because the filelog regex_parser
leaves message and timestamp as log-record attributes, Loki's OTLP path turns
those into structured metadata, which joins a metric query's label set. The
query therefore produced one series per log line and Loki answered HTTP 400,
maximum number of series reached. A second bug hid the first: the JSON helper
never checked resp.status, so Loki's own explanation arrived as a mimetype
complaint instead. Both fixed, in the Python and the shell twin, and verified
against a real loki 3.7.6 including a genuine-zero control so that zero stays
distinguishable from unavailable.
_tempo_search and _tempo_get_trace called resp.json() with no status check, so
any non-2xx became "0 traces" or "0 spans" -- the same class of bug as the
span.name tag returning 200 with an empty list. A 404 on /api/traces/<id>
legitimately means "not indexed yet", so that stays an absence and every other
non-200 now raises.
log.trace_id_cross_reference queried Tempo once, with no retry, while the
metric checks share a poll deadline for exactly this race. It now polls on the
existing METRIC_POLL_TIMEOUT_SEC/INTERVAL, so a trace that has not yet been
indexed is retried rather than reported missing. The window stays at 4 hours
and the assertion is unchanged.
Brings phase-10 up to 8418d474a7, one commit: the span reverse-coverage check
was querying the tag `span.name`, but a span's name is a TraceQL intrinsic
rather than a span-scoped attribute, so Tempo answered 200 with an empty
tagValues list and the check silently never evaluated. It now queries the bare
`name` intrinsic.
That is the same inertness the previous CI round on this branch observed from the
other end -- the run logged "Tempo span names (0 total)" while per-span TraceQL
searches each found traces and a logged trace id resolved to 100 spans. So the
incoming fix converts a check that could only ever pass into one that can
actually fail.
validate_telemetry.py auto-merged: the incoming change and this branch's are in
different regions. Verified afterwards that both survived -- the bare `name`
endpoint is in and the old `span.name` one is gone, alongside this branch's
assert_sync_diagnostics_metrics, its gather-based fan-out, and the
SKIPPED_METRIC_GROUPS exclusion inside _metric_check_targets that keeps the
sync-diagnostics group from being walked twice.
One conflict, in the runbook's validation-coverage table, and it needed both
sides rather than either. This branch's copy carries the counts recomputed from
the real contract during the previous merge (48 span types as 28 required and 20
optional, 145 metric checks across 26 categories, 16 dashboards); phase-10's copy
still carries the pre-merge figures. But phase-10's copy also corrected the
Reverse coverage row's description from the `span.name` attribute to the `name`
intrinsic, which is precisely the bug its commit fixes -- this branch's row still
described the broken query. Resolved as this branch's rows with phase-10's
Reverse coverage row substituted in. Every other cell was byte-identical between
the two sides apart from separator padding.
Verification: no conflict markers repo-wide; two parents; validate_telemetry.py
compiles; both sides' contributions asserted present by name rather than assumed;
otel-naming exits 0 including Rule E over the edited doc; levelization baseline
clean and the incoming diff changes no include lines; pre-commit clean on both
files, prettier having re-padded only the eight table rows, with the recomputed
counts and the corrected intrinsic wording confirmed present afterwards. No C++
changed, so no compile is implicated by this merge.
The span reverse-coverage check has never evaluated. It reported "no span
names were reported (backend unreachable or empty)" on a run where Tempo
demonstrably held data -- the same run resolved a logged trace id to 32
spans.
Root cause: the tag-values query asked for `span.name`. A span's name is a
TraceQL intrinsic, not a span-scoped attribute, so `span.name` resolves to
an attribute nothing sets. Tempo answers 200 with an empty tagValues list,
which is indistinguishable from an empty backend and never raises, so the
surrounding try/except stayed silent.
Verified against tempo 2.9.4 holding exactly one span named
probe.reverse.coverage, with the collector in front of it:
/api/v2/search/tag/span.name/values -> {"tagValues":[]}
/api/v2/search/tag/name/values -> that span's name
/api/v2/search/tag/resource.service.name/values -> xrpld
The third line is the control: the span was in Tempo, so the first line's
emptiness was the wrong tag rather than no data. Cross-checked against a
populated Tempo elsewhere, whose span scope lists real attributes
(command, ledger_seq, tx_hash) and no name tag at all, while the bare
intrinsic returns the whole span inventory.
This is pre-existing, not a regression in the reverse check: the same URL
fed the operations diagnostic before that check existed, and the last
green run before it also logged "Tempo operations (0 total)". The check
faithfully reported an empty input; the input was broken.
The neighbouring resource.service.name query is correctly scoped and is
left alone.
Brings phase-10 up to 3836078a78 (74 commits). Seven conflicts, resolved
per-hunk; no side was taken wholesale.
validate_telemetry.py, four hunks. The module docstring keeps both category
lists, renumbered. _log_prometheus_metric_names takes phase-10's version: it
returns the family list that the new reverse-coverage check consumes, and
_log_name_list prints every family sorted one per line, which supersedes the
hand-maintained prefix filter this branch had been extending -- that filter
existed only to keep the log readable and listed strictly less. validate_metrics
takes phase-10's _metric_check_targets call. assert_sync_diagnostics_metrics and
phase-10's _check_metric_label both landed at the same place; both are kept.
That third hunk carried the hazard this branch had flagged in advance.
_metric_check_targets selects every group satisfying isinstance(dict) and had no
equivalent of SKIPPED_METRIC_GROUPS, because on phase-10 there was no group that
needed excluding. Merged as-is it would have walked sync_diagnostics while
assert_sync_diagnostics_metrics also walks it, polling and reporting all 61
metrics twice. The exclusion is reinstated inside that function, and its
docstring claim that the isinstance test "selects exactly the same groups the
previous name-based exclusion list did" is corrected -- true on phase-10, false
here, and the reason is ownership, which no structural test can express.
expected_metrics.json: both sides appended to metrics_excluded, so both sets are
kept, 29 entries. expected_spans.json: phase-10's fuller pathfind.compute
skip_reason replaces this branch's, and this branch's ledger.acquire ->
ledger.acquire.astree relationship is kept.
Both docs carried stale counts, and the two sides disagreed with each other --
15 dashboards against 16, and both claiming 41 span types when the contract holds
48. Rather than pick a stale side, every figure is recomputed from the resolved
contract: 48 span types as 28 required and 20 optional, 145 metric checks across
26 asserting categories as 140 names plus 5 required_labels, of which 61 are the
sync_diagnostics names, and 16 dashboards, which matches both the uid list and
the files on disk. The runbook keeps phase-10's table, which adds the reverse-
coverage row.
ConsensusSpanNames.h: both sides added different constants to namespace val;
both kept. Confirmed no identifier is redefined -- the merged file's duplicate
set is identical to this branch's, and those duplicates are distinct namespaces
(op::round against the enclosing span::round), not redefinitions.
ConsensusSpanNames.cpp was an add/add: both branches wrote this file
independently, 9 tests here and 8 on phase-10, with no name in common. All 17
are kept. The guard is dropped rather than applied to the union: SpanNames.h
documents that its constants are deliberately NOT guarded by
XRPL_ENABLE_TELEMETRY, ConsensusSpanNames.h has no guard, and the tests this
branch contributed reference ValStatus only in comments while calling
validationStatusValue with plain ints. So they compile without telemetry, and
unguarding them gains coverage in a -Dtelemetry=OFF build rather than losing it.
One defect belongs to the merge itself, appearing on neither parent. phase-10
added a job_queue_per_type_gauges group holding six jobq_<type>_running/_waiting
names; this branch declared jobq_saturation in MetricNames.h. Rule K checks a
name only when its family is owned, so declaring that constant made jobq_ owned
and turned phase-10's six entries into violations. They are beast::insight gauges
created per job type by JobTypeData's constructor, so no constant can exist for
them -- one triple per job type, minted at runtime. The group joins
NON_OTEL_METRIC_GROUPS alongside statsd_gauges for the same reason.
Verification: no conflict markers repo-wide and no unmerged index entries; both
JSON contracts parse; validate_telemetry.py compiles; every count written into
the docs re-derived from the resolved files and matching; span counters still 48
and 74; check_otel_naming.py exits 0, and Rule K proven still able to fail by
injecting a bogus name in an owned family; levelization baseline clean after
regeneration, with 17 incoming include changes; doxygen style clean across all
tracked C++ at CI scope; pre-commit --all-files clean except cargo-fmt, which
reports "Executable `cargo` not found" and touches none of the 0 Rust files here.
NOT compiled -- no approval to build, so the incoming C++ is unverified by a
compiler on this branch.
Review of 42a72863bb found that several of the reasons it recorded were wrong,
and one of its own changes was half-applied. A wrong rationale in a contract
file is worse than none, because the next reader treats it as evidence.
The histogram parity claim falsified itself. That commit's group description
says histograms are listed by all three Prometheus series, but dns_resolve and
overlay_dial were given only _bucket and _sum. Both gain _count, so the four new
histograms now match the claim and the rpc_method_us convention. Two notes still
said the histogram is asserted "by its _bucket and _count series"; both now say
all three.
The reason given for excluding serve_refused_total was wrong. It said every node
holds the same history so getLedger()/getTxSet() succeed. PeerSetImpl::addPeers
uses hasItem(peer) only as a sort SCORE and then adds peers in score order up to
its limit, so peers that lack the item are asked anyway and would be refused.
The real reason nothing is refused is that nothing asks: an inbound TMGetLedger
originates only from the ledger.acquire and txset.acquire paths, both optional.
That ties this counter to those spans, which the note now records.
The reason for excluding peer_disconnect_total rested on the run window being
270 s, shorter than maxDivergedTime. The window is over 300 s once the 60 s
propagation wait, node startup and up to 45 s of metric polling are counted, so
that argument does not hold. The counter also sits after the socket-already-
closed early return, which only de-duplicates repeat closes, and covers 13 reason
values rather than the three cited. It stays excluded on the ground that a
healthy cluster produces no disconnect cause and a mutual-dial race can produce
one non-deterministically, which is flaky either way.
ledger_jump_total was described as unable to fire. Its counter is unconditional
and checkLastClosedLedger runs every round on every node, so a node that falls
behind can follow a chain tip it did not build on -- and the burst phases exist
to create exactly that load. Reworded as not deliberately provoked, with the
decisive test named: grep the nodes' debug.log for "JUMP last closed ledger".
The claim that the group had never asserted anything was false, and the true
history is worth keeping. The five-argument call was CORRECT when it landed on
2026-07-24: the function then took report as its fifth parameter and recorded the
result itself. phase-10 added the deadline and semaphore on 2026-08-14, updating
its own caller, and never saw this one. The merge that first contained both,
7c70e142e9, was CLEAN because the two edits sit in different regions, and it
produced a caller that no longer matches its callee. So the group asserted for
about four weeks and then broke silently at a clean merge -- a semantic conflict
git cannot see and, in Python, no compile step rejects. Recorded as
_gate_history_note, because the standing rule it implies is that a clean merge is
not evidence that a cross-branch call site still matches.
_b5_rotation_note had three defects of its own. Deleting "Two independent
reasons." left "First, ... Second, ..." dangling; it counted four signals where
there are three; and its second half still described the dynamic_cast as making
registerRotationStateGauge return early, which contradicts the correction in its
first half and would tell a reader no instrument exists. The cast and early
return are inside the observe callback; the instrument is created eagerly.
Two further corrections. The comment explaining the fix said four later phases
were lost; CI passes --skip-loki, so three are. And the claim about phase-10's
unaccounted-metric pass overstated it: it is warning-only, cannot fail CI, and
also accepts an accounted_patterns regex list.
Recorded a merge hazard where the resolver will see it. phase-10 replaces the
group flatten with _metric_check_targets, which selects every dict group and has
no equivalent of SKIPPED_METRIC_GROUPS, so resolving that merge in phase-10's
favour makes validate_metrics walk sync_diagnostics while its own validator still
does -- every metric polled and reported twice.
The runbook contradicted the CI requirement this work introduces. It carried a
"Gap: no ledger.* span carries a ledger hash" block stating the attribute is
never set by any call site and that filtering on span.ledger_hash returns
nothing, while makeLedgerTraceSpan sets it unconditionally on ledger.validate and
ledger.store and InboundLedger sets it on ledger.acquire and its three phase
children. ledger.build is the one real exception. The block, the ledger-span
attribute table and the sentence listing what init() sets are all corrected.
Verification: both JSON files parse; the four histograms each carry _bucket,
_count and _sum; 61 asserted names with no duplicates and no overlap with the 22
excluded; span counters still 48 and 74; the crash repro records 61 checks with
no TypeError; check_otel_naming.py exits 0; pre-commit passes on all three files
(prettier re-padded only the runbook table rows touched here). NOT compiled -- no
C++ changed.
The sync_diagnostics group asserted nothing. assert_sync_diagnostics_metrics
called _check_prometheus_metric with five positional arguments against a
six-parameter signature: `report` landed in `deadline` and `sem` was omitted
entirely, so the call raised TypeError before a single metric was queried.
Neither run_validation nor main catches anything, so the traceback propagated,
run-full-validation.sh recorded the non-zero exit as a validation failure, and
the four phases ordered after it -- dashboards, both parity checks and log-trace
correlation -- never ran at all. Reproduced directly: TypeError, zero checks
recorded. Even with the arity corrected the group would still have passed
silently, because _check_prometheus_metric RETURNS its CheckResult rather than
recording it and the value was discarded. Both halves are fixed by adopting the
fan-out validate_metrics already uses: one shared deadline, a concurrency
semaphore, gather, then report.add per result. The same call now records 55
checks where it previously recorded none.
With the gate live, the inventory it guards had to be made honest.
Two metrics could never have passed it. unl_fetch_total is emitted only from
ValidatorSite::reportFetchOutcome, which indexes sites_[siteIdx]; sites_ comes
from [validator_list_sites], and the harness writes a static [validators] file
with no list site anywhere, so no fetch outcome is ever reported.
handshake_negotiation_fail_total needs a rejected handshake, and no reject path
was found to be reachable between identical localhost nodes. Both move to
not_asserted.metrics_excluded, which is where the file's own description says
workload-gated names belong. Eleven further conditional metrics -- the acquire,
replay, disconnect, serve, jump and sweep counters -- were documented only
inside free-text notes; they move to the same map. That matters beyond tidiness:
_accounted_metric_names harvests metrics_excluded keys, so a name recorded only
in prose is reported as unaccounted, and a prose note cannot be linted at all.
Two metrics were wrongly excluded. rotation_state's callback gates only on
dynamic_cast<DatabaseRotating*>, and online_delete=256 is set by both the cfg
template and run-full-validation.sh, so SHAMapStoreImp builds a
DatabaseRotatingImp, the cast succeeds, and both sub-series are observed on
every collection tick. The note claiming the harness could not produce them
conflated "no rotation runs" with "no series published"; the first is true and
bounds the values, the second is false. Both are now asserted at value 0, where
absence rather than the zero is the regression, and the note is corrected.
The four new histograms listed only _bucket, or _bucket and _count. Each now
lists _sum as well, matching the rpc_method_us and job_queued_us convention, so
an exporter regression that drops one series cannot pass.
On the span side, ledger.validate and ledger.store are the two ends of the
per_ledger trace-join group, and the join is computed by hashing ledger_hash --
yet neither required it. Both spans take it unconditionally from
makeLedgerTraceSpan, so requiring it is free, and without it a lost join key
surfaces only as "spans landed in separate traces", naming the consequence
instead of the cause.
Deliberately unchanged: ledger.serve stays required and peer.dial keeps its
current required attributes, though both look unsafe -- ledger.serve can only
fire if an optional span fires first, and peer.dial's destructor exit sets
neither outcome nor duration_ms. Those weaken assertions rather than add
coverage, so they are reported rather than changed here.
Verification: TypeError reproduced before the fix and absent after, with 55
checks recorded; both JSON files parse; no name is both asserted and excluded
and none is duplicated; the declared span counters remain consistent at 48 and
74, proven by injecting an extra attribute and watching the check fail;
check_otel_naming.py exits 0, and Rule K was proven to read these entries by
injecting a bogus name in an owned family and observing exit 1; pre-commit
passes on all three files; the levelization baseline is unchanged. NOT compiled
-- no C++ changed.
The two log-correlation checks have never executed in CI: the workflow
hardcoded --skip-loki, so validate_telemetry.py never constructed
log.trace_id_present or log.trace_id_cross_reference. A green Telemetry
Validation therefore carried no evidence that a log line reaches Loki with
trace context. Drop the flag so both checks run and can fail the job.
Correlation spans four independent legs and a failed check names none of
them, so run-full-validation.sh now prints a per-leg diagnostic after the
suite whenever the checks are enabled:
node per-node debug.log line count, the count matching the injected
trace_id/span_id shape, one sample line, and the severity mix,
so "no log at all", "log level too high" and "no active sampled
span" are distinguishable
mount the container-side listing of /var/log/xrpld, taken with the
collector's own mounts and uid. That image is built from
scratch and carries no shell, so the listing runs in a
throwaway container with --volumes-from, not via docker exec
collector the receiver's watched files, logs-pipeline warnings, and the
internal log-record counters, read from inside the container's
network namespace because that endpoint binds to the
container's own localhost and its port is not published
loki the exact query used, the label inventory, and entry counts for
the stream selector with and without the line filter, so "Loki
has nothing" and "Loki has lines but none carry a trace id" are
distinguishable
The diagnostics are non-fatal by construction: every leg runs in its own
subshell with errexit off, each docker and curl call is guarded, and the
coordinator always returns success. Verified with no containers and no Loki
reachable, with an emptied PATH, and with a leg forced to exit non-zero.
validate_telemetry.py gains a matching diagnostic beside the checks,
following _log_prometheus_metric_names: warnings only, never a check
result. Its stream selector and line filter move into module constants
that the shell diagnostic reads back, so the two cannot drift into
describing different queries.
No check was widened or auto-passed, and LOG_QUERY_WINDOW_SECONDS stays at
four hours; a wider window would let a check pass on a previous run's logs.
validate_metrics and validate_spans only ever run one direction: read the
contract, ask the backend whether each listed name exists. Nothing looked the
other way, so a metric family or span name the contract omitted was invisible
by construction. Both emitted inventories were already being fetched for the
CI log and neither was compared back, which is how a 345 family metric gap and
7 unknown span names went unnoticed.
Add two reverse checks, metric.reverse_coverage and span.reverse_coverage.
Each names every emitted family the contract never mentions, sorted, one per
line, with counts in the report details.
Warn only, by design. passed is hardcoded True in a single shared builder, so
an unaccounted name cannot turn CI red: downstream branches legitimately add
telemetry an upstream contract has not seen yet, and a hard failure would
redden all of them for doing the right thing.
Bulk families are accounted for declaratively. A new top level
accounted_patterns list in expected_metrics.json holds anchored regexes with a
written reason each, covering the 105 per job type queue gauges, the 70 per job
type histogram families, the 228 overlay per category traffic families, and the
Prometheus scrape plumbing that is not xrpld telemetry. Job type shapes are
reduced structurally because every job type name lowercases to letters only;
traffic categories are enumerated instead, because they contain underscores and
a structural pattern there would swallow unrelated names. Anything outside
these shapes still surfaces.
Exporter shapes are folded before matching, so a histogram triple is accounted
for by an entry written for its base family and is never reported as three
separate gaps. Spans need no pattern list: the reverse check reuses the same
matcher the forward check uses, so a glob such as rpc.command.* covers every
command it expands to, and an optional entry still counts as known.
Also fix the diagnostic these checks feed on: both emitted lists were logged as
a single Python list repr, about 15 kB on one line for 422 families, unreadable
and impossible to compare between runs. Both now print one name per line.
_metric_check_targets now selects groups by testing that the value is an
object, rather than by excluding two key names, so a non group top level key
cannot break it. Output is byte identical: 79 metric plus 5 label checks, same
names in the same order.
_log_prometheus_metric_names exists to make name mismatches between
expected_metrics.json and actual emissions visible in CI logs, but it kept
only names matching 19 hard-coded prefixes. On the last CI run that showed
147 of 422 families, and none of the prefixes covered state_accounting_*,
node_family_*, overlay_peer_disconnects or the pathfind_* histograms, so
the coverage gap the preceding commit closes could not be seen through it
at all.
An allow-list can only ever surface names someone already thought to look
for, which is the opposite of what a discovery aid has to do, so the
filter is removed rather than extended. The whole list is a few kilobytes
of CI log. Sorted, so two runs' output can be diffed directly; the
Prometheus API promises no order.
Both checks selected on {job="xrpld"}. Loki's OTLP ingestion promotes
service.name to the label `service_name` and keeps a `job` attribute as
structured metadata, which a stream selector cannot match, so the selector
returned zero streams whatever had been ingested. The collector config and
TESTING.md already say to select on `service_name`.
Invert the cross-reference. Picking an arbitrary trace from Tempo and
expecting it in Loki fails even when correlation works, because a log line
carries a trace_id only when emitted inside a sampled span and most spans
log nothing at `warning` level. Start from a logged trace_id instead and
resolve it in Tempo, which is the invariant worth asserting, and try every
id found so one unexported trace does not fail the check.
Bound the log queries in time. Nothing here set start/end, so every query
relied on Loki's one-hour default and returned nothing when re-run later to
investigate a result.
Nine conflicts, resolved as follows.
src/xrpld/app/ledger/detail/InboundLedger.cpp -- kept this branch's version.
phase10 sets the span's outcome/timeouts/peer_count attributes inline at each
exit; this branch replaced that with the idempotent finalizeAcquireSpan(), called
on all four exits (init, done, give-up, destructor). Taking phase10's blocks
would have set the outcome twice against a helper documented as not overwriting
what the real exit recorded. phase10's comment explains why peer_count must not
be read in a destructor; the helper solves that structurally by taking
std::optional<std::size_t> and being passed std::nullopt from there.
src/xrpld/telemetry/MetricsRegistry.cpp -- kept metric::ledgerEconomy over
phase10's "ledger_economy" literal. This branch added the naming check that
requires constants for converted families, so the literal would regress it. Took
phase10's comment cleanup.
src/xrpld/telemetry/MetricsRegistry.h -- kept registerRotationStateGauge(), which
only exists here, and took phase10's removal of the stale task-number comment.
validate_telemetry.py -- combined both. phase10 replaced serial metric polling
with a concurrent fan-out on one shared deadline, because 58 metrics x 45 s of
additive timeout overran the CI budget; that is kept. Its target list filters on
SKIPPED_METRIC_GROUPS rather than the two literals it hardcoded, so the
sync_diagnostics group stays owned by assert_sync_diagnostics_metrics() instead
of being polled and reported twice. Both SYNC_DIAGNOSTICS_GROUP and
METRIC_POLL_CONCURRENCY are needed and both are kept.
check_otel_naming.py -- both sides extend the rule docstring. Took phase10's
fuller Rule E text (doc discovery, allow-dotted markers) and re-appended rules
I/J/K/L, which exist only here.
expected_metrics.json -- the two sides add disjoint sibling groups, so both are
kept: sync_diagnostics alongside node_health_gauges, overlay_reduce_relay,
overlay_overflow, validation_lifetime_counters and not_asserted. Both dashboard
uids are kept, giving 16 asserted uids against 16 dashboards on disk.
expected_spans.json -- kept this branch's span set, a superset that adds the
acquire phase spans, ledger.serve, txset.acquire and peer.dial, and expands
ledger.acquire's required attributes. Took phase10's description, which documents
what the totals mean, and its note on how the RPC wildcard span is created.
total_span_types and total_unique_attributes are recomputed for the union: 48 and
74, since each side's figure counted only its own spans.
Docs: took phase10's more accurate wording on what the dashboard check actually
covers, and corrected the dashboard count from 15 to 16 where the merge made it
stale.
Verified: no conflict markers remain, both JSON contracts parse, both Python
files compile, asserted dashboard uids match the dashboards on disk exactly, and
the OTel naming check reports all layers consistent.
Two defects reported against the harness, both confirmed.
The TPS field was computed with `bc` at scale=2, and bc omits the leading
zero: it prints ".25", not "0.25". A bare ".25" is not valid JSON, and this
was the normal case rather than an edge case — ledgers close every few
seconds, so ledger-advance over elapsed-seconds is well under 1 for any
realistic window. It survived earlier checks because those piped the file
through jq, which accepts the malformed form; Python's json rejects the whole
file. awk's %.2f always pads, so the field is now produced with awk. Audited
the other numeric fields at the same time: CPU average and memory peak
already used awk, and the p99, sample count and consensus mean are integers,
so TPS was the only one affected.
Separately, a failing attribute fetch was reported under the span's own check
name, which had already recorded the trace as found. That produced two
entries for one name, one passing and one failing, inflating the check total
and blaming the trace-existence check for a failure in a later network call.
The fetch now carries its own error handling and reports under
`span.attrs.<span>`, matching where its successful counterpart reports. It
moved into a helper rather than growing `validate_spans`, which was already
well over the line limit.
Two defects reported against the validation harness. Both premises were
correct, but neither suggested fix was, so the remedies differ.
Dashboard panel count: `len(dashboard["panels"])` treated Grafana row
objects as panels and skipped the panels nested inside collapsed rows, so
every dashboard was over-reported by between 1 and 10 (`log-derived-insights`
read 41 against a true 31). The check also passed unconditionally on HTTP
200, so a dashboard that renders nothing would still pass. `_leaf_panel_count`
now walks row children and the result gates the verdict. Gating on the old
top-level length, as suggested, would not have caught the case it was aimed
at: a dashboard made only of collapsed rows counts its rows and reports a
positive number while rendering nothing.
RPC latency percentiles: `LoadStats.record` appended a latency for every
outcome, including requests that never got a reply, where the value is a
time-to-failure rather than a round trip. A timeout contributed the full
receive timeout, and at the error rate a real run shows this reported p95 and
p99 of 10000 ms where the true figure was 5 ms. `record` now takes an
optional latency and the timeout path passes none. The suggestion to append
only on success was not adopted: a reply carrying `status: error` is a
completed, timely round trip whose latency is a genuine measurement, and
discarding it would throw away real data. `per_command` is now keyed off the
request counts rather than the latency map, so a command whose every request
timed out still appears in the report instead of vanishing from it, and each
entry carries a `latency_samples` count.
Fixes the review findings on this PR that belong to files it owns, plus
several defects found while verifying those fixes. Findings in files owned
by upstream branches are routed there and left untouched here.
Correctness:
- tx_submitter: advance the account sequence only on results that actually
consume one (tes*, tec*, terQUEUED). tem*/tef*/tel* never reach the
ledger, so advancing left a permanent gap that every later submit from
that account inherited. Add a re-fetch hatch so a repeated non-consuming
failure cannot livelock on the same sequence, and gate the account check
on funded-ness rather than list length.
- validate_telemetry: filter spans by name before collecting attributes, so
a per-span attribute contract can no longer be satisfied by a sibling
span; require exact name equality for non-wildcard children and glob
matching for wildcards; bounds-check every returned series instead of
only the first.
- collect_system_metrics: select xrpld by argv[0] rather than a substring
match on the whole command line, which averaged in unrelated processes
and reported their RSS as xrpld's. Count genuine 0.0 CPU readings, use a
clamped nearest-rank p99 index, and record RPC latency only on success.
- benchmark: return each verdict through a named variable instead of a
command substitution, so the pass/fail counters survive and the exit gate
can fire. Scale before dividing in the percentage math, which truncated a
1.26% impact to 1.00% and cleared a 1% threshold.
- compare_to_baseline: fall back to the absolute bound when the baseline is
not positive, so a 0 -> 500 ms jump is no longer "within bounds".
- rpc_load_generator: bound each connection to one in-flight recv(), drain
in-flight requests before closing, use a nearest-rank percentile, and
report delivery shortfall so an under-delivered run cannot pass with a 0%
error rate.
Fail loudly instead of silently:
- run-full-validation: treat a consensus timeout and a missing validated
ledger as fatal infrastructure errors, and fold the orchestrator and
benchmark exit codes into the final status. A degraded cluster previously
ran a full validation pass and reported misleading downstream failures.
- collect_system_metrics: warn per empty measurement source, emit
metrics_complete, and exit non-zero instead of substituting zeros that
pass every threshold. Require GNU date with %N rather than falling back
to a per-sample python3 fork that costs more than the threshold it is
measured against.
- benchmark: distinguish "could not measure" from "exceeded thresholds",
install a cleanup trap so a failure cannot leak nodes and ports, and
report an unusable baseline as inconclusive.
- workload_orchestrator: bound subprocess communicate() and fail the exit
gate on per-phase errors.
Also pins the workload compose images to the versions the sibling stack
already uses, hash-pins the Python dependencies, restricts the validator
config template to loopback, corrects the dashboard and metric counts in
the reference docs, drops a span from the regression gate that cannot fire
under a WebSocket-only workload, and narrows the teardown pkill pattern so
it no longer matches processes that merely mention the work directory.
Verified with a full harness run against a local five-node cluster:
158 of 158 checks passed with no regressions detected.
Phase-10 independently instrumented the peer object-fetch path while this
branch instrumented fresh-node sync, so the two overlapped in three places.
Resolved by keeping each side's stronger implementation rather than shipping
both.
Per-job-type waiting/running/deferred existed twice. Phase-10's version
survives: it publishes per-type gauges from JobQueue::collect(), which
snapshots under the queue lock and publishes after releasing it, a
deliberate lock-order fix against the collector's own lock. This branch's
jobq_backlog gauge and the JobQueue::getJobTypeCounts() accessor that fed it
are removed, along with their panels, assertions and reference rows.
jobq_saturation stays: it reports the whole worker pool, which phase-10 has
no equivalent for.
The histogram view helper also existed twice with identical bodies under two
names; one survives, and the microsecond ladder is now the named array
rather than boundaries repeated inline. The job_type label was declared
twice, once as a file-local constant invisible to the naming check; both it
and handler now come from the constants header.
Two things phase-10 adds are complementary, not duplicates, and are kept as
they are: the handler label, which separates the two request kinds that both
report as the same job type, and getobject_rejected_total, which counts
malformed requests where this branch's serve_refused_total counts requests
this node declined to serve.
Also fixes two naming-check failures that pre-date this merge on phase-10.
The check derived label keys only from namespaced constants, so it could not
see the per-subsystem headers' flat k-prefixed style and rejected dashboards
querying labels the code really emits. It now reads both styles, with the
enforcement rules unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A slow fresh-sync ledger produced spans scattered across threads with no
way to relate them. They now share a trace id derived from the ledger's own
hash, the one value every participating site already holds, so nothing new
is plumbed across threads. This is the pattern the transaction pipeline
already uses for its tx id.
Joined: ledger.validate, ledger.store, and a new
consensus.validation.accept recorded when a trusted validation arrives. In
Tempo, searching one ledger hash returns them together, so an operator can
tell whether the ledger was slow to arrive, slow to be accepted, or slow to
be stored. They are siblings rather than a chain because the accept gate is
entered from three different threads, so no fixed parent order exists.
consensus.validation.accept also records why an arriving validation did or
did not advance the gate, which makes "validations arrive but are all
rejected" visible for the first time.
consensus_round_duration_ms turns the existing round-time span attribute
into a histogram, so a fleet trend needs a metric query rather than raw
trace inspection. An explicit bucket view is required, not optional: the
SDK default tops out at ten seconds while consensus abandons a round at two
minutes, so slow rounds would all fall in one bucket and every quantile
would read exactly ten seconds. Cost is one record per round.
Record layer: the histogram is native and needs no collector change. The
two new bounded attributes are added as span-metric dimensions to both
collector configs. The ledger hash stays out of them, since a per-ledger
dimension mints a series per ledger; it is indexed in Tempo as the join key.
The ledger.acquire span is not joined yet, because that file was being
changed concurrently. It is registered as an optional member of the join
group so nothing fails, and switching it is a one-line follow-up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sync-critical job types run at very low concurrency limits (ledgerRequest
and ledgerData allow 3 each), so a node can stall simply because those
jobs are held back behind other work. Nothing exposed that until now:
the existing job metrics are rates and quantiles of jobs that already
moved, or a single queue-wide depth.
- jobq_backlog{metric,job_type}: instantaneous waiting, running and
deferred counts per job type. Deferred is the starvation signal and had
no exposure anywhere; it is set when a type is at its concurrency limit.
- jobq_saturation{metric}: running tasks, worker-thread count and total
waiting, so a slowdown spanning several subsystems can be attributed to
worker-pool exhaustion instead of being diagnosed once per victim.
Both read through two new const accessors on JobQueue that take the
existing mutex once and copy integers, so a single reading is internally
consistent and no per-job cost is added. The job_type label reuses the
same JobTypes name helper the existing job counters use, so the two label
sets join.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signals that separate a sync that is merely slow from one that will never
finish:
- sync_acquire{missing_state_nodes_max, missing_tx_nodes_max, in_flight,
received_data_depth}: how many SHAMap nodes each in-flight acquire is
still waiting for. getMissingNodes already computed this and the callers
discarded it after a trace log. A count that stays flat means the
acquire is wedged; a shrinking count means it is progressing. Recorded
once per sweep, never inside the per-node walk, and reset when a tree
completes so a finished acquire does not read as stuck forever.
- shamap_cache_hit_rate{treenode}: hit rate of the in-memory tree-node
cache, which sits above the node store, so it is distinct from the
existing NuDB ratio. A cold cache on a fresh node sends every traversal
step to disk.
- sync_acquire_no_progress_total: timer ticks where an acquire made no
progress, previously only logged.
- sync_addnode_total{good,duplicate,invalid}: whether arriving nodes are
useful, duplicated or rejected, so wasted fetch work is visible.
- sync_acquire_source_total{local,network}: whether a ledger was served
from the local store or had to be fetched.
Adds getBad()/getDuplicate() to SHAMapAddNode and an acquireProgress()
accessor on InboundLedgers so the xrpld gauge can read these without
libxrpl depending on telemetry.
ledger_seq is deliberately not a metric label: it is unbounded. Per-ledger
identity stays on the ledger.acquire span; the metrics expose bounded
aggregates instead.
The full-below cache hit rate is not exported: KeyCache updates different
counters than getHitRate() reads, so it would always report zero. That
libxrpl bug is documented rather than papered over.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five signals that explain why a node is not advancing toward full, none of
which were observable before:
- state_changes_total now carries {from,to} mode labels, emitted at
setMode using the existing strOperatingMode helper. A bare count could
not distinguish a healthy climb from a node flapping between tracking
and connected. Removes the now-unused incrementStateChanges wrapper.
- sync_state{initial_full_duration_us}: time to first reach full, which
StateAccounting already computed but exposed only in server_info.
- sync_state{network_ledger_gate}: whether the node is still refusing to
build ledgers because it has no network ledger.
- sync_state{server_stall_seconds} and server_stall_events_total: how
long the main thread has been unresponsive. LoadManager computed this
and only logged it, so a stall was invisible until the fatal threshold.
The episode rule is a pure function so it can be tested without adding
a test-only mutator to LoadManager.
- sync_state{ledgers_behind}: how far our validated sequence trails the
best sequence any peer advertises, read from already-cached peer ranges
so no extra network traffic is added.
Also fixes the naming checker: it derived only the first label of a
multi-label instrument, so a dashboard querying the second label was
wrongly rejected.
Note: the clang-tidy hook cannot run in this worktree (no build
directory); the remaining pre-commit hooks, the naming check, dashboard
schema and harness syntax all pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A freshly started node most often stalls before it ever peers or reaches
quorum, and that whole chain had no telemetry. Adds the six signals that
make it observable:
- dns_resolve_total / dns_resolve_latency_ms: configured-peer hostname
resolution, emitted from OverlayImpl so libxrpl stays independent.
- overlay_connect_total / overlay_dial_latency_ms: outbound dial outcome
by terminal reason, plus dial duration.
- handshake_negotiation_fail_total: protocol and network-id negotiation
rejections, labelled by reason, so a misconfigured network is no longer
indistinguishable from unreachable peers.
- unl_fetch_total and the unl_quorum gauge: validator-list fetch outcome
per site and trusted key count against the required quorum. Without
these a bad validators.txt leaves the node syncing forever with no
signal.
- clock_close_offset_seconds: network close-time offset, which server_info
hides below 60s but which stalls consensus participation.
Panels land in the Bootstrap row of the Ledger Sync Health dashboard, the
metrics are asserted by the workload validator, and both the reference and
the runbook flow describe them.
Levelization baseline regenerated: overlay now includes MetricMacros.h, so
the overlay/telemetry pair is reported one-way instead of bidirectional.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the anchors the sync-diagnostics signals attach to, with no signals
emitted yet:
- New "Ledger Sync Health" dashboard (uid ledger-sync-health) with the
standard template-variable block copied from an existing board, plus
empty "Bootstrap (Domain 0)" and "Sync pipeline" rows.
- Signal index section in the data-collection reference, an operator-flow
stub in the telemetry runbook, and a glossary anchor.
- A sync_diagnostics group in expected_metrics.json and a matching
assertion helper in validate_telemetry.py so CI fails when a signal
regresses to absent.
Also registers the new dashboard uid with the harness so the board is
covered by validation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Strip xrpld_ prefix, lowercase beast::insight names, and replace
traces_span_metrics_ with span_ in all remaining tracked files:
alert rules, integration tests, workload validation, TESTING.md,
OpenTelemetryPlan docs, code comments, and config templates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The workload validator queried each expected metric once, immediately after
a fixed post-workload propagation wait. Several beast::insight metrics
(ledger-age and peer-finder gauges, overlay-traffic and rpc-request counters)
only populate after the node validates ledgers and sustains peer traffic,
then travel a 1s OTLP export + 15s Prometheus scrape before they are
queryable. On a slower CI runner that pipeline can settle after the wait
ends, so the single query raced and reported "0 series", failing 12 checks
that pass locally with the same config and binary.
Poll each metric on the /api/v1/series endpoint until it appears or a 45s
window (two scrape cycles) elapses. Present metrics still return on the first
query with no added delay; a genuinely-absent metric still fails after the
timeout. Makes the check robust to runner speed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- peer.validation.receive now asserts the shared bare ledger_hash /
full_validation keys (was the dotted xrpl.ledger.hash and validation_full);
PARITY_SPAN_ATTRS checks both on the peer span too.
- Fix a span-name drift: the per-transaction accept span is txq.accept_tx
(op::acceptTx = "accept_tx"), not txq.accept.tx — the old assertion never
matched and was silently skipped as optional.
- Drop the "intentionally dotted" notes; there is no dotted span attribute.
The Phase 10 validation harness had drifted from the code's recording surface
and the telemetry-validation CI job was failing before it could build.
CI fix (telemetry-validation.yml):
- Replace nonexistent local action ./.github/actions/print-env with the remote
XRPLF/actions/print-build-env (the build-xrpld job failed in 56s on this).
- Sync prepare-runner and upload-artifact action SHAs to the canonical workflow.
Recording-surface reconciliation (docker/telemetry/workload/):
- Migrate span attributes from dotted xrpl.<domain>.<field> to the bare/underscore
form introduced by the 2026-05-13 span-attr naming redesign (tx_hash, peer_id,
ledger_seq, consensus_mode, consensus_round, full_validation, quorum, ...).
Dotted xrpl.ledger.hash is retained only on peer.validation.receive (shared
constant), while consensus.validation.send uses bare ledger_hash.
- Fix attribute placement: tx.apply carries tx_count/tx_failed (not ledger_seq);
ledger.build carries ledger_seq/close_* (not tx_count/tx_failed).
- Replace the phantom rpc.request span with the real WS root rpc.ws_message; drop
the never-emitted duration_ms; rebuild the parent-child map accordingly.
- Add the new spans the code emits: apply-pipeline stage spans
(tx.preflight/preclaim/transactor with stage/tx_type/ter_result), txq.*,
consensus sub-spans (round/establish/update_positions/check/phase.open),
ledger.acquire, grpc.*, pathfind.*. Conditional spans are marked optional so
they are skipped (not failed) when the workload does not exercise them.
- validate_telemetry.py: service.name and Loki job label rippled -> xrpld; fix
PARITY_SPAN_ATTRS (rename the 4 real attrs, drop the 3 that are metrics not span
attrs); add optional-span handling that skips missing optional spans while still
validating attributes when present.
- expected_metrics.json: rippled_ -> xrpld_ on all beast::insight/overlay metrics,
xrpld_job_count, the 15 on-disk xrpld-* dashboard UIDs, and the real bare
spanmetrics dimension labels.
- regression-metrics.json + baseline-timings.json: rpc.request -> rpc.ws_message.
Metrics pipeline fix:
- Switch node [insight] config from server=statsd/prefix=rippled to server=otel +
/v1/metrics endpoint + prefix=xrpld across run-full-validation.sh,
xrpld-validator.cfg.template, benchmark.sh and the workload compose. The
collector has no StatsD receiver, so system metrics only reach Prometheus over
OTLP.
Synthetic load for new spans:
- Add ripple_path_find to the RPC load generator (drives pathfind.* spans).
- Add a high-TPS txq-burst workload phase to force fee escalation (drives txq.*).
All facts verified against the *SpanNames.h headers and a live xrpld node +
collector (Tempo service.name=xrpld, tx.preflight attrs [stage,ter_result,tx_type],
279 xrpld_ Prometheus metrics and zero rippled_).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 10's workload validation configs (expected_metrics.json,
regression-metrics.json, validate_telemetry.py) queried the
MetricsRegistry metrics under the rippled_ prefix, but MetricsRegistry
emits them as xrpld_ (see MetricsRegistry.cpp). On a live run the
workload validator reported every MetricsRegistry metric as missing,
masking genuine regressions.
Rename the following to xrpld_ across the workload validator,
expected-metrics manifest, and regression-metrics template:
- nodestore_state, cache_metrics, txq_metrics, load_factor_metrics,
object_count
- rpc_method_started_total / _finished_total / _errored_total /
_duration_us
- job_queued_total / _started_total / _finished_total /
_queued_duration_us_bucket / _running_duration_us_bucket
- peer_quality, server_info, validator_health, ledger_economy,
db_metrics, complete_ledgers, build_info, state_tracking,
storage_detail
- ledgers_closed_total, validations_sent_total,
validations_checked_total, state_changes_total
- validation_agreement, validation_agreements_total,
validation_missed_total
Mirrors the phase-9 fix in commit 5601615952.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Migrate validate_telemetry.py to Tempo TraceQL search API, remove
Jaeger service from workload docker-compose, update readiness checks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
