The list mirrors the collector's spanmetrics dimensions, and that dimension is
gone. The reduce-relay and squelch counters keep their own names, which are a
different thing.
The online-delete rotation's cache freshen called TaggedCache::getKeys(),
which held the cache mutex while copying every key. On the dev box's 26
million entry tree-node cache that hold lasted 5-6 s, froze every job
that touches the cache, and dropped the RocksDB node out of sync once per
rotation: each "getKeys held the lock" warning was followed within 1-5 s
by "View of consensus changed" (5 of 5 rotations on 2026-09-15).
Copy the keys one map partition at a time instead. TaggedCache gains
forEachKeyPartition(), which holds the mutex only while one partition's
keys are copied and runs the callback with the mutex released, so the
longest hold shrinks by the partition count (8 on the dev box). The
freshen.keys rotation phase no longer exists as one step, so its span,
stage value, harness entries and docs are removed; the per-partition hold
still shows on the cache lock-hold peak gauge.
Measure what the freshen achieves, which no existing signal did.
DatabaseRotating gains duplicateCopyForwardTotal(), counting archive
copies made on duplicate fetches (the rotation's own copy walk and
freshen); copyForwardTotal() deliberately excludes those. The freshen
phase records rotation_freshen_keys_total{cache,outcome} and stamps
key_count, cache and keys_copied on its span; the copy phase stamps
nodes_copied. A warn log line per freshen reports the same numbers, and
the ledger-sync-health dashboard gets a Rotation Freshen Yield panel.
Log the "STATE->" operating-mode change at warn instead of info. It is
the only record of a mode change with an exact timestamp; the
state_changes_total counter is scraped once a minute and cannot order a
flap against a multi-second event.
Tests: five GTests for forEachKeyPartition (every key once, empty cache,
mutex free during the callback, concurrent insert, lock-hold peak), three
for duplicateCopyForwardTotal over two memory backends, one for the new
counter's series, and the new name literals.
CI's clang-tidy leg flagged eight include-cleaner errors and three
misc-const-correctness / readability-convert-member-functions-to-static /
modernize-use-designated-initializers issues, all inside WP-B6's own code.
Fixed as follows:
- `MetricsRegistry.h`: `#include <opentelemetry/metrics/observer_result.h>`
for ObserverResult; `observeCacheLockHoldPeaks` is now `static` because it
touches neither instance state nor telemetry members.
- `SHAMapStoreImp.h`: adds direct includes for `<cstddef>`, `<string_view>`
and `<xrpl/telemetry/SpanNames.h>` (the StaticStr provider). `seconds` in
`RotationPhase::~RotationPhase` is `[[maybe_unused]]` so a
`-DXRPL_ENABLE_TELEMETRY=0` build under `-Werror` keeps compiling.
- `SHAMapStoreImp.cpp`: direct includes for `SHAMapStoreSpanNames.h`,
`SpanGuard.h`, `SpanNames.h`; `RotationPhase` locals that never call
`setAttribute` are declared `const`; `RotationOutcome` uses designated
initialisers.
Final-review findings (WP-B6-rotation-stall-tracing.md, "What to check
when reviewing"):
- Panels 74 and 75 on `ledger-sync-health.json` still carried panel 41's
description, axisLabel, Source and Keywords copy; rewritten to describe
rotation phase duration and cache lock hold respectively.
- `consensus_view_change_total` and the `view.change` round-span event
were emitted but not registered with the harness. Added the counter to
`not_asserted.metrics_excluded` (workload-gated) and annotated the
`consensus.round` span note with the event and its two attribute keys.
Not fixed (parked, see progress ledger):
- The reviewer's second Important finding — a plan/code contradiction on
the consensus counter — was based on a misread of the plan; the plan's
"Rejected alternatives" table lists a new `TraceCategory::Nodestore` and
the getKeys() fix, not the consensus counter. No action.
- The Minor note about `sweep()`'s peak including lock-acquire time and
`getKeys()`'s not: `sweep()` acquires and releases the lock via a
`scoped_lock`, so `noteLockHold` still runs after the release and the
numbers are comparable. No action.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `nodestore.rotate` and its eight phase children to expected_spans.json,
all `optional: true` because the 5-node localhost harness cluster never reaches
`online_delete`. Their parent-child relationships are asserted but skip-marked
so a run without a rotation stays green.
Adds `cache_metrics{metric="treenode_lock_hold_peak_us"|"fullbelow_lock_hold_peak_us"}`
to the asserted sync_diagnostics group -- both are observable and always emit,
even at zero. Puts `rotation_phase_duration_seconds` and `jobq_stall_total` in
`not_asserted.metrics_excluded`; both are workload-gated.
On the Cloud collector, adds an `ottl_condition` policy that keeps any trace
carrying a span whose name matches `^nodestore\.rotate`, so the 0.5% probabilistic
tail sampler cannot drop a rotation trace. Sampler is OR'd across policies.
The workload README contradicted itself on --skip-loki: one bullet said CI always
passes it and so the two log-correlation checks are never exercised, another said
the workflow no longer passes it. The workflow mentions the flag nowhere, so the
first was the stale half.
Other claims checked against the tree and corrected:
- both the README and the plan doc described the push trigger as filtered on
branch names. The workflow has no branches filter, deliberately, because
GitHub ANDs branches with paths
- the plan doc printed 6 of the workflow's 12 paths globs, and claimed the
workflow was 367 lines against an actual 451. The glob block is now generated
from the workflow, and the line count dropped rather than restated
- rpcNOT_SUPPORTED does not exist anywhere in the tree. The symbol is
RpcNotSupported, and the refusal sites are RipplePathFind.cpp:59-60 and
PathFind.cpp:50-51, not :48-49 and :39
- RCLConsensus.cpp:666 and :663 are not log or event lines; the tx.included event
is at :720 and the per-transaction debug log at :715
- LedgerMaster.cpp:463 is fixIndex, not the ledger.store span, which is at :470
- ServerHandler.cpp:705 is inside makeJsonError; processRequest is at :718
- file counts: docker/telemetry/workload/ is 25 files, include/xrpl/telemetry/ 13
- the optional-span bullet named five causes covering 10 of 16 entries, omitting
the txq.* family and the WebSocket handshake
- the /api/v1/series choice was attributed to stale StatsD gauges; this harness
runs no StatsD
A line number in run-full-validation.sh was cited in five places and drifts on
every edit to that file, so those now name the file only. The keygen helper's
header records what production does instead -- validator-keys-tool create_keys
then create_token, keeping the master key off the node -- and why a disposable
cluster does not.
Three conflicts, all between this branch's own sync-diagnostics work and
phase-10's older versions. Resolved to this branch in each case, since it
owns the newer content:
- InboundLedger.h keeps the missing-node and receive-depth gauges and the
fuller acquire-span contract.
- MetricsRegistry.cpp keeps the namespaced label:: constants.
- LedgerMaster.cpp keeps makeLedgerTraceSpan(), which joins the store and
validate spans into one per-ledger trace by hash.
LedgerMaster.cpp needed a second pass. The automatic merge had kept both
sides outside the conflict markers, nesting phase-10's older promotion
block inside this branch's `if (!pubLedger_)` — so setValidated,
setFull and setValidLedger would each have run twice. Taking this
branch's file wholesale removes the duplicate; brace balance and a single
"Advancing accepted ledger" confirm it.
That resolution drops two things phase-10 was carrying into this file:
the storeSpan/validateSpan guard names, and the explicit scope that keeps
the one-in-256 flag-ledger check outside the ledger.validate measurement.
Both are re-applied on this branch in the next commit; the scope needs a
variable-lifetime check that does not belong in a merge.
The harness contract described how path-finding load came to be absent rather
than why it is absent. The load exists on no branch before this one, so a squash
merge publishes no revision that ever issued it: the 2026-08-25 date resolves
nowhere, and 'removing it', 'used to satisfy' and 'has now cleared' compare
against a state a reader cannot reach.
- README: state that DEFAULT_WEIGHTS carries no ripple_path_find entry, and give
the error floor as what WOULD happen if it did, rather than what removing it
fixed. 'Putting it back' becomes 'Enabling it'.
- expected_spans.json: the pathfind.request and pathfind.compute notes, and both
hierarchy skip_reasons, now put the span's presence in the conditional -- the
parent would appear if the RPC were issued, because the ScopedSpanGuard at
RipplePathFind.cpp:35 sits above the rpcNOT_SUPPORTED guard at :48-49.
- expected_metrics.json: the rpc_method_errored_total, pathfind_fast and
pathfind_full notes drop the date and keep both independent reasons the
metrics stay absent.
- regression-thresholds.json: span.ledger.store 'is excluded from' the gated
surface rather than 'was removed from' it.
The reasoning is unchanged: pathfinding is off because Config.cpp:725-726 zeroes
pathSearchMax when [validation_seed] is present, a refused call still exports an
error span, and at a 3% weight that is a ~3% STATUS_CODE_ERROR floor.
Documentation and JSON note strings only, no behaviour change.
Comments across the sync-diagnostic work described earlier revisions of the same
change, or cited identifiers a reader of the merged tree cannot resolve.
Prior-state comparisons rewritten in the present tense:
- MetricsRegistry.cpp carried two adjacent paragraphs prescribing opposite
behaviour for a disabled quorum, one publishing int64 max and one omitting
the series. The code omits it; the superseded paragraph is gone and the
surviving reason SIZE_MAX must not be cast is kept.
- MallocTrim, LedgerMaster, LedgerReplayTask, TransactionAcquire, Application:
say what the signal is the only record of, rather than what was 'previously
trace-only', 'not logged at all here' or 'used to sit inside if (debug())'.
- LedgerMaster.h and SpanGuardScope: without an explicit join each ledger's
spans WOULD be separate traces -- not that they were 'before this'.
- Handshake: the message is forwarded byte for byte, not 'byte-identical to the
previous behaviour', and the helper throws rather than 'throws as before'.
- MetricNames: quorum_disabled is a separate boolean rather than a sentinel,
stated without what the state 'used to be encoded by'.
- LedgerMaster.cpp no longer claims to mirror the unl_quorum gauge; it does not.
That gauge omits the series while this stores int64 max.
- 'Split out of' / 'Split from' become 'Kept separate from' in five places.
Plan-internal identifiers removed:
- All 24 WP-Ax / WP-Bx work-package labels across the telemetry tests, the
collector configs, tempo.yaml and the expected_* inventories. They are defined
in no file in the repo, so they resolve nowhere once merged.
- The two references to OpenTelemetryPlan/, which does not reach develop, now
point at docs/telemetry-glossary.md 'Fresh-node sync diagnostics'.
Comments and JSON note strings only, no behaviour change.
overlay_connect_total and the three overlay_dial_latency_ms series come back
with 4 series per run while sibling families such as dns_resolve_* come back
with 5, one per node. That was unexplained, so anyone reading the group had to
choose between suspecting the exporter and re-deriving the cause. It is a
topology artefact and nothing is wrong.
OverlayImpl::connect asks peerFinder().newOutboundSlot for a slot and returns
early when it gets a null one (OverlayImpl.cpp:464-470), before it constructs
the ConnectAttempt that emits both signals (:472). Every node is seeded to dial
every other node -- run-full-validation.sh:324-331 builds IPS_FIXED from all
NUM_NODES-1 peers and :373-374 writes it into [ips] -- so all five nodes do try.
In a full mesh each pair is dialled from both ends, and the node whose peer got
there first is refused an outbound slot for an address it already holds inbound:
no ConnectAttempt, so neither the counter nor the histogram. dns_resolve_*
reports 5 because reportDnsResolve fires inside the resolver handler
(OverlayImpl.cpp:603), which runs before any slot allocation.
The note also records why the four entries assert series presence rather than a
count: hard-coding 4 would bake today's mesh into the contract and break on any
cluster-size change, while gaining nothing -- and it says that fewer than 4
would be worth investigating, since that means a node did not dial at all.
Verified in code: the early return and its position relative to the
ConnectAttempt, the [ips] construction, and the resolver call site. Not verified
against a run: which node is missing on any given run, because dial ordering is
not controlled and the identity is not expected to be stable. No assertion
changed -- this commit adds documentation only.
Brings phase-10 up to 3836078a78 (74 commits). Seven conflicts, resolved
per-hunk; no side was taken wholesale.
validate_telemetry.py, four hunks. The module docstring keeps both category
lists, renumbered. _log_prometheus_metric_names takes phase-10's version: it
returns the family list that the new reverse-coverage check consumes, and
_log_name_list prints every family sorted one per line, which supersedes the
hand-maintained prefix filter this branch had been extending -- that filter
existed only to keep the log readable and listed strictly less. validate_metrics
takes phase-10's _metric_check_targets call. assert_sync_diagnostics_metrics and
phase-10's _check_metric_label both landed at the same place; both are kept.
That third hunk carried the hazard this branch had flagged in advance.
_metric_check_targets selects every group satisfying isinstance(dict) and had no
equivalent of SKIPPED_METRIC_GROUPS, because on phase-10 there was no group that
needed excluding. Merged as-is it would have walked sync_diagnostics while
assert_sync_diagnostics_metrics also walks it, polling and reporting all 61
metrics twice. The exclusion is reinstated inside that function, and its
docstring claim that the isinstance test "selects exactly the same groups the
previous name-based exclusion list did" is corrected -- true on phase-10, false
here, and the reason is ownership, which no structural test can express.
expected_metrics.json: both sides appended to metrics_excluded, so both sets are
kept, 29 entries. expected_spans.json: phase-10's fuller pathfind.compute
skip_reason replaces this branch's, and this branch's ledger.acquire ->
ledger.acquire.astree relationship is kept.
Both docs carried stale counts, and the two sides disagreed with each other --
15 dashboards against 16, and both claiming 41 span types when the contract holds
48. Rather than pick a stale side, every figure is recomputed from the resolved
contract: 48 span types as 28 required and 20 optional, 145 metric checks across
26 asserting categories as 140 names plus 5 required_labels, of which 61 are the
sync_diagnostics names, and 16 dashboards, which matches both the uid list and
the files on disk. The runbook keeps phase-10's table, which adds the reverse-
coverage row.
ConsensusSpanNames.h: both sides added different constants to namespace val;
both kept. Confirmed no identifier is redefined -- the merged file's duplicate
set is identical to this branch's, and those duplicates are distinct namespaces
(op::round against the enclosing span::round), not redefinitions.
ConsensusSpanNames.cpp was an add/add: both branches wrote this file
independently, 9 tests here and 8 on phase-10, with no name in common. All 17
are kept. The guard is dropped rather than applied to the union: SpanNames.h
documents that its constants are deliberately NOT guarded by
XRPL_ENABLE_TELEMETRY, ConsensusSpanNames.h has no guard, and the tests this
branch contributed reference ValStatus only in comments while calling
validationStatusValue with plain ints. So they compile without telemetry, and
unguarding them gains coverage in a -Dtelemetry=OFF build rather than losing it.
One defect belongs to the merge itself, appearing on neither parent. phase-10
added a job_queue_per_type_gauges group holding six jobq_<type>_running/_waiting
names; this branch declared jobq_saturation in MetricNames.h. Rule K checks a
name only when its family is owned, so declaring that constant made jobq_ owned
and turned phase-10's six entries into violations. They are beast::insight gauges
created per job type by JobTypeData's constructor, so no constant can exist for
them -- one triple per job type, minted at runtime. The group joins
NON_OTEL_METRIC_GROUPS alongside statsd_gauges for the same reason.
Verification: no conflict markers repo-wide and no unmerged index entries; both
JSON contracts parse; validate_telemetry.py compiles; every count written into
the docs re-derived from the resolved files and matching; span counters still 48
and 74; check_otel_naming.py exits 0, and Rule K proven still able to fail by
injecting a bogus name in an owned family; levelization baseline clean after
regeneration, with 17 incoming include changes; doxygen style clean across all
tracked C++ at CI scope; pre-commit --all-files clean except cargo-fmt, which
reports "Executable `cargo` not found" and touches none of the 0 Rust files here.
NOT compiled -- no approval to build, so the incoming C++ is unverified by a
compiler on this branch.
Review of 42a72863bb found that several of the reasons it recorded were wrong,
and one of its own changes was half-applied. A wrong rationale in a contract
file is worse than none, because the next reader treats it as evidence.
The histogram parity claim falsified itself. That commit's group description
says histograms are listed by all three Prometheus series, but dns_resolve and
overlay_dial were given only _bucket and _sum. Both gain _count, so the four new
histograms now match the claim and the rpc_method_us convention. Two notes still
said the histogram is asserted "by its _bucket and _count series"; both now say
all three.
The reason given for excluding serve_refused_total was wrong. It said every node
holds the same history so getLedger()/getTxSet() succeed. PeerSetImpl::addPeers
uses hasItem(peer) only as a sort SCORE and then adds peers in score order up to
its limit, so peers that lack the item are asked anyway and would be refused.
The real reason nothing is refused is that nothing asks: an inbound TMGetLedger
originates only from the ledger.acquire and txset.acquire paths, both optional.
That ties this counter to those spans, which the note now records.
The reason for excluding peer_disconnect_total rested on the run window being
270 s, shorter than maxDivergedTime. The window is over 300 s once the 60 s
propagation wait, node startup and up to 45 s of metric polling are counted, so
that argument does not hold. The counter also sits after the socket-already-
closed early return, which only de-duplicates repeat closes, and covers 13 reason
values rather than the three cited. It stays excluded on the ground that a
healthy cluster produces no disconnect cause and a mutual-dial race can produce
one non-deterministically, which is flaky either way.
ledger_jump_total was described as unable to fire. Its counter is unconditional
and checkLastClosedLedger runs every round on every node, so a node that falls
behind can follow a chain tip it did not build on -- and the burst phases exist
to create exactly that load. Reworded as not deliberately provoked, with the
decisive test named: grep the nodes' debug.log for "JUMP last closed ledger".
The claim that the group had never asserted anything was false, and the true
history is worth keeping. The five-argument call was CORRECT when it landed on
2026-07-24: the function then took report as its fifth parameter and recorded the
result itself. phase-10 added the deadline and semaphore on 2026-08-14, updating
its own caller, and never saw this one. The merge that first contained both,
7c70e142e9, was CLEAN because the two edits sit in different regions, and it
produced a caller that no longer matches its callee. So the group asserted for
about four weeks and then broke silently at a clean merge -- a semantic conflict
git cannot see and, in Python, no compile step rejects. Recorded as
_gate_history_note, because the standing rule it implies is that a clean merge is
not evidence that a cross-branch call site still matches.
_b5_rotation_note had three defects of its own. Deleting "Two independent
reasons." left "First, ... Second, ..." dangling; it counted four signals where
there are three; and its second half still described the dynamic_cast as making
registerRotationStateGauge return early, which contradicts the correction in its
first half and would tell a reader no instrument exists. The cast and early
return are inside the observe callback; the instrument is created eagerly.
Two further corrections. The comment explaining the fix said four later phases
were lost; CI passes --skip-loki, so three are. And the claim about phase-10's
unaccounted-metric pass overstated it: it is warning-only, cannot fail CI, and
also accepts an accounted_patterns regex list.
Recorded a merge hazard where the resolver will see it. phase-10 replaces the
group flatten with _metric_check_targets, which selects every dict group and has
no equivalent of SKIPPED_METRIC_GROUPS, so resolving that merge in phase-10's
favour makes validate_metrics walk sync_diagnostics while its own validator still
does -- every metric polled and reported twice.
The runbook contradicted the CI requirement this work introduces. It carried a
"Gap: no ledger.* span carries a ledger hash" block stating the attribute is
never set by any call site and that filtering on span.ledger_hash returns
nothing, while makeLedgerTraceSpan sets it unconditionally on ledger.validate and
ledger.store and InboundLedger sets it on ledger.acquire and its three phase
children. ledger.build is the one real exception. The block, the ledger-span
attribute table and the sentence listing what init() sets are all corrected.
Verification: both JSON files parse; the four histograms each carry _bucket,
_count and _sum; 61 asserted names with no duplicates and no overlap with the 22
excluded; span counters still 48 and 74; the crash repro records 61 checks with
no TypeError; check_otel_naming.py exits 0; pre-commit passes on all three files
(prettier re-padded only the runbook table rows touched here). NOT compiled -- no
C++ changed.
The sync_diagnostics group asserted nothing. assert_sync_diagnostics_metrics
called _check_prometheus_metric with five positional arguments against a
six-parameter signature: `report` landed in `deadline` and `sem` was omitted
entirely, so the call raised TypeError before a single metric was queried.
Neither run_validation nor main catches anything, so the traceback propagated,
run-full-validation.sh recorded the non-zero exit as a validation failure, and
the four phases ordered after it -- dashboards, both parity checks and log-trace
correlation -- never ran at all. Reproduced directly: TypeError, zero checks
recorded. Even with the arity corrected the group would still have passed
silently, because _check_prometheus_metric RETURNS its CheckResult rather than
recording it and the value was discarded. Both halves are fixed by adopting the
fan-out validate_metrics already uses: one shared deadline, a concurrency
semaphore, gather, then report.add per result. The same call now records 55
checks where it previously recorded none.
With the gate live, the inventory it guards had to be made honest.
Two metrics could never have passed it. unl_fetch_total is emitted only from
ValidatorSite::reportFetchOutcome, which indexes sites_[siteIdx]; sites_ comes
from [validator_list_sites], and the harness writes a static [validators] file
with no list site anywhere, so no fetch outcome is ever reported.
handshake_negotiation_fail_total needs a rejected handshake, and no reject path
was found to be reachable between identical localhost nodes. Both move to
not_asserted.metrics_excluded, which is where the file's own description says
workload-gated names belong. Eleven further conditional metrics -- the acquire,
replay, disconnect, serve, jump and sweep counters -- were documented only
inside free-text notes; they move to the same map. That matters beyond tidiness:
_accounted_metric_names harvests metrics_excluded keys, so a name recorded only
in prose is reported as unaccounted, and a prose note cannot be linted at all.
Two metrics were wrongly excluded. rotation_state's callback gates only on
dynamic_cast<DatabaseRotating*>, and online_delete=256 is set by both the cfg
template and run-full-validation.sh, so SHAMapStoreImp builds a
DatabaseRotatingImp, the cast succeeds, and both sub-series are observed on
every collection tick. The note claiming the harness could not produce them
conflated "no rotation runs" with "no series published"; the first is true and
bounds the values, the second is false. Both are now asserted at value 0, where
absence rather than the zero is the regression, and the note is corrected.
The four new histograms listed only _bucket, or _bucket and _count. Each now
lists _sum as well, matching the rpc_method_us and job_queued_us convention, so
an exporter regression that drops one series cannot pass.
On the span side, ledger.validate and ledger.store are the two ends of the
per_ledger trace-join group, and the join is computed by hashing ledger_hash --
yet neither required it. Both spans take it unconditionally from
makeLedgerTraceSpan, so requiring it is free, and without it a lost join key
surfaces only as "spans landed in separate traces", naming the consequence
instead of the cause.
Deliberately unchanged: ledger.serve stays required and peer.dial keeps its
current required attributes, though both look unsafe -- ledger.serve can only
fire if an optional span fires first, and peer.dial's destructor exit sets
neither outcome nor duration_ms. Those weaken assertions rather than add
coverage, so they are reported rather than changed here.
Verification: TypeError reproduced before the fix and absent after, with 55
checks recorded; both JSON files parse; no name is both asserted and excluded
and none is duplicated; the declared span counters remain consistent at 48 and
74, proven by injecting an extra attribute and watching the check fail;
check_otel_naming.py exits 0, and Rule K was proven to read these entries by
injecting a bogus name in an owned family and observing exit 1; pre-commit
passes on all three files; the levelization baseline is unchanged. NOT compiled
-- no C++ changed.
Every node the harness starts is a validator, and validators disable
pathfinding: Config.cpp:725-726 zeroes pathSearchMax whenever a
[validation_seed] or [validator_token] section is present, and
run-full-validation.sh writes [validation_seed] into every generated node
cfg (:308) with no [path_search] section to put the default back. So
doRipplePathFind refused every call at RipplePathFind.cpp:48-49 and the
3% ripple_path_find weight bought no coverage at all.
It was not free either. The pathfind.request guard is constructed at
RipplePathFind.cpp:35, above that refusal, so each refused call still
exported a span, and the enclosing rpc.command.ripple_path_find span
carried rpc_status=error. That put a steady 3% error floor into
span_calls_total for STATUS_CODE_ERROR: any error-rate threshold derived
from harness data before this change was measuring the harness rather
than xrpld, and needs re-deriving.
Removing the load makes pathfind.request unreachable, so it moves from
required to optional in expected_spans.json; without that the span check
would fail on every run. Three notes in that file and three in
expected_metrics.json made claims that are now false, two of them citing
line numbers this commit deletes; all six are corrected. The runbook
required/optional count moves 26/15 to 25/16.
Two facts a future reader needs.
First, the weights previously summed to 103, not 100, so every percentage
the docstring stated was wrong: health checks were really 38.8%, not 40%.
Dropping the 3 makes the sum exactly 100 and every stated percentage
correct for the first time. expected_spans.json also carried live
arithmetic off the old total, "25/103 ... roughly 43%", now 25/100 and
42%.
Second, baselines/baseline-timings.json was captured WITH this load. Only
span.rpc.ws_message p50/p95/p99 of the 25 gated keys sees the RPC mix,
and their trip points sit 3.1x to 5.9x above baseline, so the gate will
not fire. But a timing baseline is workload-specific and its profile
field still reads full-validation, so nothing will flag the drift:
refresh it from the next CI run's timings artifact.
Pathfinding now has no coverage in this harness at all. The workload
README section "Pathfinding is not exercised" records that cost, the
manual verification route, and a four-step restore recipe in which steps
1 and 2 alone only reinstate the error floor.
validate_metrics and validate_spans only ever run one direction: read the
contract, ask the backend whether each listed name exists. Nothing looked the
other way, so a metric family or span name the contract omitted was invisible
by construction. Both emitted inventories were already being fetched for the
CI log and neither was compared back, which is how a 345 family metric gap and
7 unknown span names went unnoticed.
Add two reverse checks, metric.reverse_coverage and span.reverse_coverage.
Each names every emitted family the contract never mentions, sorted, one per
line, with counts in the report details.
Warn only, by design. passed is hardcoded True in a single shared builder, so
an unaccounted name cannot turn CI red: downstream branches legitimately add
telemetry an upstream contract has not seen yet, and a hard failure would
redden all of them for doing the right thing.
Bulk families are accounted for declaratively. A new top level
accounted_patterns list in expected_metrics.json holds anchored regexes with a
written reason each, covering the 105 per job type queue gauges, the 70 per job
type histogram families, the 228 overlay per category traffic families, and the
Prometheus scrape plumbing that is not xrpld telemetry. Job type shapes are
reduced structurally because every job type name lowercases to letters only;
traffic categories are enumerated instead, because they contain underscores and
a structural pattern there would swallow unrelated names. Anything outside
these shapes still surfaces.
Exporter shapes are folded before matching, so a histogram triple is accounted
for by an entry written for its base family and is never reported as three
separate gaps. Spans need no pattern list: the reverse check reuses the same
matcher the forward check uses, so a glob such as rpc.command.* covers every
command it expands to, and an optional entry still counts as known.
Also fix the diagnostic these checks feed on: both emitted lists were logged as
a single Python list repr, about 15 kB on one line for 422 families, unreadable
and impossible to compare between runs. Both now print one name per line.
_metric_check_targets now selects groups by testing that the value is an
object, rather than by excluding two key names, so a non group top level key
cannot break it. Output is byte identical: 79 metric plus 5 label checks, same
names in the same order.
The assert / do-not-assert decisions in expected_metrics.json and
expected_spans.json were all correct, but several recorded reasons were not.
Pathfinding is disabled outright on every harness node: Config.cpp:725-726
zeroes pathSearchMax whenever a [validation_seed] or [validator_token] section
is present, run-full-validation.sh writes [validation_seed] for every node and
has no [path_search] override, and both handlers return rpcNOT_SUPPORTED
before constructing a PathRequest.
- pathfind_full_milliseconds no longer claims a probabilistic path, nor
prescribes an explicit ledger index, which cannot help: the config gate
fires before the ledger parameter is read.
- pathfind_fast_milliseconds keeps its hasCompletion argument but now leads
with the config gate, which is the operative blocker.
- The pathfind.compute and pathfind.discover notes and the
pathfind.request to pathfind.compute skip reason no longer blame missing
liquidity. pathfind.update_all now records why its request list stays empty.
- statsd_gauges states the arming precondition: a beast gauge is only as safe
as an observable gauge when its object exists before Application.cpp:1570,
where onCollectionReady arms the registered gauges exactly once.
- Alert wiring claims softened: every rule in rules.yaml is paused.
- The per job type gauge group loses its bogus poll bandwidth reason, and its
regex claim is corrected: there is no running state regex, so 30 of those
gauges have no consumer at all.
- overlay_peer_disconnects has one query consumer, not two.
- Cloud dashboard copies dropped from consumer counts: that tree is ignored by
git and has no tracked files.
- rpc_method_errored_total explains that a refused RPC is a normal return, not
a throw, so the refusals above do not make it fire.
- 09-data-collection-reference.md no longer claims a Prometheus name query in
expected_metrics.json.
No behaviour change: the flattened check name list is byte identical.
Dashboards and alert rules reference 186 metrics; the harness asserted 57.
Excluding the 107 per-category overlay-traffic expansions, the meaningful
gap was 20 names. This closes it under the contract file's own doctrine:
assert only what the workload guarantees, and record the rest with a
precise reason.
Asserted 18, taking the metric checks from 61 to 79 and the whole metric
phase from 66 to 84. No pre-existing check name or position changes.
statsd_gauges gains the nine state_accounting_* siblings of the one member
already asserted, plus the two NodeFamily full-below-cache gauges and
overlay_peer_disconnects. All twelve rest on one mechanism the group
description now spells out: on the OTel path a beast gauge is an
Int64ObservableGauge, every instance self-registers in its constructor,
onCollectionReady arms all of them unconditionally, and the armed callback
Observes on every export cycle whether or not set was ever called, so the
series exist at 0. The state_accounting family is set in one unconditional
block in NetworkOPsImp::collectMetrics, and full_transitions is the input
to the NodeStateFlapping alert rule, so the alert's own signal had been
going unverified.
A new job_queue_per_type_gauges group asserts the six per-job-type gauges
that a panel or a rule names literally, jobq_manifest_waiting among them
as the ManifestJobQueueConvoy rule's input. The description records why
those six and not all 105: the guarantee is identical for every
non-special job type, so the discriminator is consumer coverage, and the
remaining names are only reached through topk queries over the family that
do not depend on any single type being present.
Recorded four more in not_asserted rather than asserting them.
pathfind_fast_milliseconds is unreachable for this workload, not merely
rare: reportFast fires only from the doCreate fast pass, which is guarded
by !hasCompletion(), and both ripple_path_find entry points construct the
request with a completion function. Only the path_find subscription
reaches it, and the generator does not use it.
pathfind_full_milliseconds is reachable but only one ledger close after
the request, through PathRequestManager::updateAll, and nothing in the
harness arranges or checks that, so the guarantee is probabilistic.
warn_total and drop_total are resource-manager meters gated on a consumer
crossing the warn or drop threshold; their rpc-pathfinding panels are
correct and render empty only because the condition has not occurred,
which is worth stating because both were briefly mis-read as phantoms.
Runbook check counts updated to match.
RPC Response Size on rpc-pathfinding said its p95 was computed "over the
dashboard rate interval", but the query hardcodes [5m]. Every other panel on
that dashboard with a hardcoded [5m] -- RPC Response Time, RPC Response Time
Distribution, both Pathfinding duration panels, the gRPC latency panel and
Pathfinding Compute Duration -- says "over 5 minutes". Matched the clause to
the query. The same edit was applied to the local grafanacloud copy so the
two stay identical; that tree is gitignored, so it is not in this commit.
The io_latency group in expected_metrics.json claimed "all 6 panels that
query it". That 6 was a raw string-occurrence count over the dashboards and
included two panel descriptions. Verified truth: two distinct panels query
the metric, ledger-data-sync "I/O Scheduler Latency p95" and node-health
"I/O Latency", each mirrored in a grafanacloud copy, plus one alert rule in
grafana/provisioning/alerting/rules.yaml. Stated that instead of a count.
Nine conflicts, resolved as follows.
src/xrpld/app/ledger/detail/InboundLedger.cpp -- kept this branch's version.
phase10 sets the span's outcome/timeouts/peer_count attributes inline at each
exit; this branch replaced that with the idempotent finalizeAcquireSpan(), called
on all four exits (init, done, give-up, destructor). Taking phase10's blocks
would have set the outcome twice against a helper documented as not overwriting
what the real exit recorded. phase10's comment explains why peer_count must not
be read in a destructor; the helper solves that structurally by taking
std::optional<std::size_t> and being passed std::nullopt from there.
src/xrpld/telemetry/MetricsRegistry.cpp -- kept metric::ledgerEconomy over
phase10's "ledger_economy" literal. This branch added the naming check that
requires constants for converted families, so the literal would regress it. Took
phase10's comment cleanup.
src/xrpld/telemetry/MetricsRegistry.h -- kept registerRotationStateGauge(), which
only exists here, and took phase10's removal of the stale task-number comment.
validate_telemetry.py -- combined both. phase10 replaced serial metric polling
with a concurrent fan-out on one shared deadline, because 58 metrics x 45 s of
additive timeout overran the CI budget; that is kept. Its target list filters on
SKIPPED_METRIC_GROUPS rather than the two literals it hardcoded, so the
sync_diagnostics group stays owned by assert_sync_diagnostics_metrics() instead
of being polled and reported twice. Both SYNC_DIAGNOSTICS_GROUP and
METRIC_POLL_CONCURRENCY are needed and both are kept.
check_otel_naming.py -- both sides extend the rule docstring. Took phase10's
fuller Rule E text (doc discovery, allow-dotted markers) and re-appended rules
I/J/K/L, which exist only here.
expected_metrics.json -- the two sides add disjoint sibling groups, so both are
kept: sync_diagnostics alongside node_health_gauges, overlay_reduce_relay,
overlay_overflow, validation_lifetime_counters and not_asserted. Both dashboard
uids are kept, giving 16 asserted uids against 16 dashboards on disk.
expected_spans.json -- kept this branch's span set, a superset that adds the
acquire phase spans, ledger.serve, txset.acquire and peer.dial, and expands
ledger.acquire's required attributes. Took phase10's description, which documents
what the totals mean, and its note on how the RPC wildcard span is created.
total_span_types and total_unique_attributes are recomputed for the union: 48 and
74, since each side's figure counted only its own spans.
Docs: took phase10's more accurate wording on what the dashboard check actually
covers, and corrected the dashboard count from 15 to 16 where the merge made it
stale.
Verified: no conflict markers remain, both JSON contracts parse, both Python
files compile, asserted dashboard uids match the dashboards on disk exactly, and
the OTel naming check reports all layers consistent.
Fixes the review findings on this PR that belong to files it owns, plus
several defects found while verifying those fixes. Findings in files owned
by upstream branches are routed there and left untouched here.
Correctness:
- tx_submitter: advance the account sequence only on results that actually
consume one (tes*, tec*, terQUEUED). tem*/tef*/tel* never reach the
ledger, so advancing left a permanent gap that every later submit from
that account inherited. Add a re-fetch hatch so a repeated non-consuming
failure cannot livelock on the same sequence, and gate the account check
on funded-ness rather than list length.
- validate_telemetry: filter spans by name before collecting attributes, so
a per-span attribute contract can no longer be satisfied by a sibling
span; require exact name equality for non-wildcard children and glob
matching for wildcards; bounds-check every returned series instead of
only the first.
- collect_system_metrics: select xrpld by argv[0] rather than a substring
match on the whole command line, which averaged in unrelated processes
and reported their RSS as xrpld's. Count genuine 0.0 CPU readings, use a
clamped nearest-rank p99 index, and record RPC latency only on success.
- benchmark: return each verdict through a named variable instead of a
command substitution, so the pass/fail counters survive and the exit gate
can fire. Scale before dividing in the percentage math, which truncated a
1.26% impact to 1.00% and cleared a 1% threshold.
- compare_to_baseline: fall back to the absolute bound when the baseline is
not positive, so a 0 -> 500 ms jump is no longer "within bounds".
- rpc_load_generator: bound each connection to one in-flight recv(), drain
in-flight requests before closing, use a nearest-rank percentile, and
report delivery shortfall so an under-delivered run cannot pass with a 0%
error rate.
Fail loudly instead of silently:
- run-full-validation: treat a consensus timeout and a missing validated
ledger as fatal infrastructure errors, and fold the orchestrator and
benchmark exit codes into the final status. A degraded cluster previously
ran a full validation pass and reported misleading downstream failures.
- collect_system_metrics: warn per empty measurement source, emit
metrics_complete, and exit non-zero instead of substituting zeros that
pass every threshold. Require GNU date with %N rather than falling back
to a per-sample python3 fork that costs more than the threshold it is
measured against.
- benchmark: distinguish "could not measure" from "exceeded thresholds",
install a cleanup trap so a failure cannot leak nodes and ports, and
report an unusable baseline as inconclusive.
- workload_orchestrator: bound subprocess communicate() and fail the exit
gate on per-phase errors.
Also pins the workload compose images to the versions the sibling stack
already uses, hash-pins the Python dependencies, restricts the validator
config template to loopback, corrects the dashboard and metric counts in
the reference docs, drops a span from the regression gate that cannot fire
under a WebSocket-only workload, and narrows the teardown pkill pattern so
it no longer matches processes that merely mention the work directory.
Verified with a full harness run against a local five-node cluster:
158 of 158 checks passed with no regressions detected.
The Telemetry Validation workflow failed with three "0 series" checks:
rpc_method_us, job_queued_us and job_running_us. All three are Histograms,
and the Prometheus exporter emits a histogram only as the
_bucket/_count/_sum triple -- the bare instrument name is never a series,
so validate_metrics() could never match it.
Evidence from the failing run (31804450127): its own metric-name dump
lists rpc_method_us_bucket/_count/_sum and no bare rpc_method_us, while
the sibling counters recorded in the same function bodies passed with 100
and 67 series. capture_timings.py, which queries job_queued_us_bucket and
job_running_us_bucket, returned real values for the acceptLedger job type
in that same run. Every one of the 10 histograms present exposes the full
triple, so all three suffixes are safe to assert.
Name them the way the exporter does, matching what the spanmetrics group
above already does for span_duration_milliseconds and what
regression-metrics.json and the job-queue dashboard already query. The
metrics stay in their asserted groups because they are genuinely
unconditional, so `not_asserted` would be wrong.
The harness manifests asserted things the code cannot produce and missed most
of what it does. Two assertions were failing every run, and the metric set
covered 16 of the ~41 emitted names.
expected_spans.json: rpc.process was required with rpc.ws_message as its
parent, but it is created only in ServerHandler::processRequest() on the HTTP
path, so a WebSocket-only workload never produces it -- it is now optional and
parented to rpc.http_request, and the rpc.process -> rpc.command.* edge is
skipped with the real reason instead of a coroutine-context-loss diagnosis that
was never the cause. Adds the missing rpc.ws_upgrade span, corrects four
parents (consensus.mode_change, pathfind.request, and update_positions/check,
which are children of consensus.establish rather than consensus.round), and
demotes conditionally-set attributes out of required_attributes so a healthy
run stops failing. Counts recomputed from the file: 41 span types, 62 unique
required attributes.
expected_metrics.json: 16 -> 52 asserted entries across the job-queue, RPC
method, reduce-relay, overflow and validation families, plus the fifteenth
dashboard uid. Metrics the harness workload cannot exercise -- erroring RPC,
ledger-mismatch, TxQ overflow, and the lazily-created getobject_* instruments
-- are listed in a not_asserted group the validator skips, rather than as
assertions that would fail on a healthy node.
The workflow's push trigger listed two globs matching nothing
(include/xrpl/basics/Telemetry*.h, src/xrpld/app/misc/Telemetry*), so no C++
telemetry change ever triggered validation. Replaced with the paths the code
actually lives in, including src/libxrpl/beast/insight/** for the insight
export path the harness depends on. The four inert workflow_dispatch inputs are
now labelled UNUSED rather than looking like working knobs.
Docs: the workload README described a StatsD dirty-flag mechanism under a
member name that does not exist, on a code path the harness never uses -- it
sets [insight] server=otel, so gauges export through an observable-gauge
callback every cycle. Adds the missing txq-burst phase, reconciles three
different dashboard counts, and drops "posts summary to PR", which the workflow
has no permission to do. The runbook's phase-10 section loses the last
sampling_ratio reference (not a config key), gains a Regression Gate and CI
subsection covering the gate that can fail CI, and its compose-logs command now
names the workload compose file. cmake --preset default is left for a separate
change: no CMakePresets.json is tracked, so it is wrong everywhere it appears.
Also drops the dead exporter=otlp_http key the harness wrote into every node
config, and stops capture_timings.py defaulting --profile to a profile that
does not exist.
Add two Peer Quality panels reading peer_disconnect_total: Peer Disconnect
Rate, the per-second teardown rate per node, and Peer Disconnects By Reason
& Direction, the per-interval increase split by cause and by which side
opened the connection. Both sit in the existing Disconnects & Connection Mix
row beside Resource Disconnects, which counts only the resource-charge
subset and carries no reason label.
The Ledger Sync Health board already shows the same split as a window
total, so it says how much of each reason but not when. These give the
time-shaped view, letting a reason spike be lined up against a stall.
Add disconnect_reason and disconnect_direction template variables for the
two new label dimensions and wire both queries to them, so the panels
filter on every dimension their series carry.
Update the 09 reference panel column and the _a7_note panel list to name
the panels that now render this counter.
Brings in the phase-10 revert of the nodestore read-latency histogram plus the
nudb_bytes -> stored_object_bytes rename.
Conflicts in MetricsRegistry.{h,cpp} resolved keeping both intents:
- MetricsRegistry.cpp: dropped everything that existed only to serve the
reverted nodestore_read_us histogram -- the addSubMillisecondHistogramView()
helper, its call site, the kSubMillisecondBoundaries array and the
NodeStoreMetricNames.h include. Kept every view this branch registers
(consensus round duration, sweep_malloc_trim_us, dns_resolve_latency_ms,
overlay_dial_latency_ms) and the shared addHistogramView() base helper.
Took the rename at the storage_detail observe() call site.
- MetricsRegistry.h: took phase-10's move of the four nodestore_state observe
helpers and their ObserveFn sink from private to public, while keeping this
branch's enriched Doxygen on observeNodeStoreTotals().
Also corrected the registered-view count in the 09 reference doc: neither side's
arithmetic survives the merge, since this branch adds four views phase-10 never
saw and the revert removes one. Ten views are registered now, not six or seven.
nodestore_latency published six values that nodestore_state already
publishes from the same Database accessors, so the two gauges were
duplicate readings of the same atomics:
write_count -> node_writes getStoreCount()
read_count -> node_reads_total getFetchTotalCount()
write_duration_us -> node_writes_duration_us getStoreDurationUs()
read_duration_us -> node_reads_duration_us getFetchDurationUs()
write_mean_us -> write_mean_us store duration / count
read_mean_us -> read_mean_us fetch duration / count
nodestore_state is kept because its means go through scaledMean(), which
saturates at INT64_MAX instead of wrapping and omits a mean when the
denominator is zero rather than reporting a misleading 0 us.
Removes registerNodeStoreLatencyGauge, its instrument member, the
metric::nodestoreLatency constant and the lval::nodestore_latency label
namespace. The gauge-over-histogram rationale and the "p99 is not
obtainable" consequence are folded into observeNodeStoreTotals' docs.
Retargets the gauge-contract test onto nodestore_state rather than
deleting it: the scaledMean arithmetic is covered by the static_asserts
in tests/libxrpl/telemetry/MetricsRegistry.cpp, but nothing else asserts
that these named series multiplex onto one instrument keyed by `metric`.
The test now calls the production scaledMean instead of a copy of the
division, and its sub-microsecond case asserts scaledMean's actual
behaviour (a genuine mean of 0 on a zero numerator with a non-zero
count), which differs from the retired gauge's extra numerator guard.
Rewrites both ledger-sync-health copies' panel 38/39 queries and drops
the obsolete claim that the write numerator was never written: all three
concrete store paths call recordStoreDuration, so write_mean_us is live
on an ordinary node. The same stale [import_db] caveat is removed from
the runbook, the 09 reference row and the workload validator's note.
Two suspects from the 3.3.0 slowdown investigation had no signal. Both were
already computing the numbers and throwing them away, so this exposes them
rather than adding measurement.
Per-sweep heap trim. The trim runs after every cache sweep, and its cost
scales with resident heap, so it is the leading explanation for a node with
a populated database syncing slower than a fresh one. The report already
carried duration, fault deltas and reclaimed pages, but the whole
measurement sat behind a debug-journal check, so an ordinary node measured
nothing, and the call site discarded the result. The measurement now always
runs and only the log line stays gated. Records trim duration, minor faults
and reclaimed kilobytes. Measured cost of the always-on path is about six
microseconds per sweep against a trim costing milliseconds, at a cadence of
ten to a hundred and twenty seconds.
Honest limit, stated in the runbook: the fault delta spans only the trim
call, so it shows the trim itself faulting but not the faults that follow as
caches refill. The duration is the signal to correlate against sweep-job
queueing.
Rotation writes. Rotation copies archive-served reads forward and re-stores
nodes missing from both backends, both of which compete with sync I/O and
only happen on a populated online_delete database. The copy-forward count
existed but was reset by the rotation's own log line, so a metric reading it
would drop to zero on every swap; a never-reset total sits beside it now.
The re-store count was not measured at all. Rotation duration is
deliberately not recorded: the health throttle sleeps at eight points inside
the sequence and dominates exactly when the node is unhealthy, so the number
would conflate work with waiting.
Nothing added for the other two suspects. Get-object serving is already
covered by the handler label, the lookup histogram and the deferred and
saturation gauges; peer churn by the disconnect-reason counter.
Also replaces nine per-file cspell ignores with one ignoreRegExpList entry
for the telemetry macro names, and picks up the levelization baseline for the
consensus span-name test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by reviewing what each metric actually measures, with attention to the
derived and bucketed ones. All five could report healthy while the node was
not, or the reverse.
- The nodestore latency panel took rate() of a mean. The gauge already
divides duration by count in code, so rating it produced a figure with no
unit, and Prometheus discards a gauge's decreases, so a heavy back-fill
read as roughly zero microseconds per operation. The cumulative duration
totals are now exported alongside the means, and the panel divides the
rate of the total by the rate of the count, which is the latency over the
panel's own window rather than a since-boot average that flattens with
uptime.
- The DNS-resolve and outbound-dial histograms had no explicit buckets, so
they inherited a ladder that stops at ten seconds while the dial timer is
fifteen. Every timed-out dial fell in the overflow bucket and p95 read
exactly ten seconds however bad it got. Both now have a ladder reaching
thirty seconds with fifteen on its own boundary, so a timeout is
distinguishable from merely slow.
- The missing-node counts only cleared when a tree completed, so a
timed-out or failed acquire left its last count latched. Since the gauge
reports the maximum across everything still in the collection, and
eviction waits on a grace period plus the sweep interval, a finished node
reported as stuck for minutes. That inverts the one signal that separates
stuck from slow. Cleared unconditionally on the terminal path instead.
- A disabled quorum published a sentinel so large that, on a timeseries
axis shared with the trusted-key count, it flattened the key line to the
baseline and hid the outage it was meant to mark. The series is now
omitted and a quorum_disabled flag carries the state.
- Two panel descriptions claimed a one-second export cycle. The reader is
configured for ten.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase-10 independently instrumented the peer object-fetch path while this
branch instrumented fresh-node sync, so the two overlapped in three places.
Resolved by keeping each side's stronger implementation rather than shipping
both.
Per-job-type waiting/running/deferred existed twice. Phase-10's version
survives: it publishes per-type gauges from JobQueue::collect(), which
snapshots under the queue lock and publishes after releasing it, a
deliberate lock-order fix against the collector's own lock. This branch's
jobq_backlog gauge and the JobQueue::getJobTypeCounts() accessor that fed it
are removed, along with their panels, assertions and reference rows.
jobq_saturation stays: it reports the whole worker pool, which phase-10 has
no equivalent for.
The histogram view helper also existed twice with identical bodies under two
names; one survives, and the microsecond ladder is now the named array
rather than boundaries repeated inline. The job_type label was declared
twice, once as a file-local constant invisible to the naming check; both it
and handler now come from the constants header.
Two things phase-10 adds are complementary, not duplicates, and are kept as
they are: the handler label, which separates the two request kinds that both
report as the same job type, and getobject_rejected_total, which counts
malformed requests where this branch's serve_refused_total counts requests
this node declined to serve.
Also fixes two naming-check failures that pre-date this merge on phase-10.
The check derived label keys only from namespaced constants, so it could not
see the per-subsystem headers' flat k-prefixed style and rejected dashboards
querying labels the code really emits. It now reads both styles, with the
enforcement rules unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A slow fresh-sync ledger produced spans scattered across threads with no
way to relate them. They now share a trace id derived from the ledger's own
hash, the one value every participating site already holds, so nothing new
is plumbed across threads. This is the pattern the transaction pipeline
already uses for its tx id.
Joined: ledger.validate, ledger.store, and a new
consensus.validation.accept recorded when a trusted validation arrives. In
Tempo, searching one ledger hash returns them together, so an operator can
tell whether the ledger was slow to arrive, slow to be accepted, or slow to
be stored. They are siblings rather than a chain because the accept gate is
entered from three different threads, so no fixed parent order exists.
consensus.validation.accept also records why an arriving validation did or
did not advance the gate, which makes "validations arrive but are all
rejected" visible for the first time.
consensus_round_duration_ms turns the existing round-time span attribute
into a histogram, so a fleet trend needs a metric query rather than raw
trace inspection. An explicit bucket view is required, not optional: the
SDK default tops out at ten seconds while consensus abandons a round at two
minutes, so slow rounds would all fall in one bucket and every quantile
would read exactly ten seconds. Cost is one record per round.
Record layer: the histogram is native and needs no collector change. The
two new bounded attributes are added as span-metric dimensions to both
collector configs. The ledger hash stays out of them, since a per-ledger
dimension mints a series per ledger; it is indexed in Tempo as the join key.
The ledger.acquire span is not joined yet, because that file was being
changed concurrently. It is registered as an optional member of the join
group so nothing fails, and switching it is a one-line follow-up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The quorum and publish gauges were emitted but never surfaced: no panel, no
harness assertion, no reference entry. Completes those layers.
- Four panels: trusted validations against the quorum target on one axis so
a tally climbing toward quorum is visually distinct from one flat below
it; publish lag; pre-accept shortfall rate; and time to first validated
ledger.
- Both signals are asserted by the workload validator. The shortfall
counter does fire on a healthy cluster, because this node validates and
then immediately re-enters the accept gate before its peers' validations
arrive, so the first evaluation of every round tallies short. The panel
and note say so, and give the fault signature instead: the shortfall rate
outpacing the ledger-close rate while the tally stays flat and nothing
ever reaches first-validated.
- The quorum target is deliberately drawn as its own line rather than as a
headroom stat, so the disabled-quorum sentinel reads as an unreachable
target instead of an unreadable negative number.
Also removes three reference rows that were appended twice when two agents
each documented the same back-fill signals.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Registers the new gauges, renders them, asserts them and documents them, so
each signal reaches an operator rather than stopping at the emit site:
- MetricsRegistry: gauge registration for ledger_quorum_publish,
nodestore_latency, peer_ledger_supply, peerfinder_slot_census and
amendment_block, each guarded by the detached-callbacks check and
tolerant of services that are not ready yet.
- Ledger Sync Health dashboard: panels for the new signals, filtered by
the node template variable like every other board.
- Workload validation: the new series are asserted, so a signal that
regresses to absent fails CI. Signals the local cluster structurally
cannot produce, such as a replay fallback or an amendment block, are
noted rather than asserted, which would fail red on a healthy run.
- Reference, runbook and glossary entries, including the diagnosis order
for a node that has peers and validators but never validates.
- Regenerated levelization baseline: three new one-way edges from the
telemetry and test modules, no new cycles.
Also drops an unused cstddef include from the macro tests, which the
include checker rejects.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sync-critical job types run at very low concurrency limits (ledgerRequest
and ledgerData allow 3 each), so a node can stall simply because those
jobs are held back behind other work. Nothing exposed that until now:
the existing job metrics are rates and quantiles of jobs that already
moved, or a single queue-wide depth.
- jobq_backlog{metric,job_type}: instantaneous waiting, running and
deferred counts per job type. Deferred is the starvation signal and had
no exposure anywhere; it is set when a type is at its concurrency limit.
- jobq_saturation{metric}: running tasks, worker-thread count and total
waiting, so a slowdown spanning several subsystems can be attributed to
worker-pool exhaustion instead of being diagnosed once per victim.
Both read through two new const accessors on JobQueue that take the
existing mutex once and copy integers, so a single reading is internally
consistent and no per-job cost is added. The job_type label reuses the
same JobTypes name helper the existing job counters use, so the two label
sets join.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signals that separate a sync that is merely slow from one that will never
finish:
- sync_acquire{missing_state_nodes_max, missing_tx_nodes_max, in_flight,
received_data_depth}: how many SHAMap nodes each in-flight acquire is
still waiting for. getMissingNodes already computed this and the callers
discarded it after a trace log. A count that stays flat means the
acquire is wedged; a shrinking count means it is progressing. Recorded
once per sweep, never inside the per-node walk, and reset when a tree
completes so a finished acquire does not read as stuck forever.
- shamap_cache_hit_rate{treenode}: hit rate of the in-memory tree-node
cache, which sits above the node store, so it is distinct from the
existing NuDB ratio. A cold cache on a fresh node sends every traversal
step to disk.
- sync_acquire_no_progress_total: timer ticks where an acquire made no
progress, previously only logged.
- sync_addnode_total{good,duplicate,invalid}: whether arriving nodes are
useful, duplicated or rejected, so wasted fetch work is visible.
- sync_acquire_source_total{local,network}: whether a ledger was served
from the local store or had to be fetched.
Adds getBad()/getDuplicate() to SHAMapAddNode and an acquireProgress()
accessor on InboundLedgers so the xrpld gauge can read these without
libxrpl depending on telemetry.
ledger_seq is deliberately not a metric label: it is unbounded. Per-ledger
identity stays on the ledger.acquire span; the metrics expose bounded
aggregates instead.
The full-below cache hit rate is not exported: KeyCache updates different
counters than getHitRate() reads, so it would always report zero. That
libxrpl bug is documented rather than papered over.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five signals that explain why a node is not advancing toward full, none of
which were observable before:
- state_changes_total now carries {from,to} mode labels, emitted at
setMode using the existing strOperatingMode helper. A bare count could
not distinguish a healthy climb from a node flapping between tracking
and connected. Removes the now-unused incrementStateChanges wrapper.
- sync_state{initial_full_duration_us}: time to first reach full, which
StateAccounting already computed but exposed only in server_info.
- sync_state{network_ledger_gate}: whether the node is still refusing to
build ledgers because it has no network ledger.
- sync_state{server_stall_seconds} and server_stall_events_total: how
long the main thread has been unresponsive. LoadManager computed this
and only logged it, so a stall was invisible until the fatal threshold.
The episode rule is a pure function so it can be tested without adding
a test-only mutator to LoadManager.
- sync_state{ledgers_behind}: how far our validated sequence trails the
best sequence any peer advertises, read from already-cached peer ranges
so no extra network traffic is added.
Also fixes the naming checker: it derived only the first label of a
multi-label instrument, so a dashboard querying the second label was
wrongly rejected.
Note: the clang-tidy hook cannot run in this worktree (no build
directory); the remaining pre-commit hooks, the naming check, dashboard
schema and harness syntax all pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A freshly started node most often stalls before it ever peers or reaches
quorum, and that whole chain had no telemetry. Adds the six signals that
make it observable:
- dns_resolve_total / dns_resolve_latency_ms: configured-peer hostname
resolution, emitted from OverlayImpl so libxrpl stays independent.
- overlay_connect_total / overlay_dial_latency_ms: outbound dial outcome
by terminal reason, plus dial duration.
- handshake_negotiation_fail_total: protocol and network-id negotiation
rejections, labelled by reason, so a misconfigured network is no longer
indistinguishable from unreachable peers.
- unl_fetch_total and the unl_quorum gauge: validator-list fetch outcome
per site and trusted key count against the required quorum. Without
these a bad validators.txt leaves the node syncing forever with no
signal.
- clock_close_offset_seconds: network close-time offset, which server_info
hides below 60s but which stalls consensus participation.
Panels land in the Bootstrap row of the Ledger Sync Health dashboard, the
metrics are asserted by the workload validator, and both the reference and
the runbook flow describe them.
Levelization baseline regenerated: overlay now includes MetricMacros.h, so
the overlay/telemetry pair is reported one-way instead of bidirectional.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the anchors the sync-diagnostics signals attach to, with no signals
emitted yet:
- New "Ledger Sync Health" dashboard (uid ledger-sync-health) with the
standard template-variable block copied from an existing board, plus
empty "Bootstrap (Domain 0)" and "Sync pipeline" rows.
- Signal index section in the data-collection reference, an operator-flow
stub in the telemetry runbook, and a glossary anchor.
- A sync_diagnostics group in expected_metrics.json and a matching
assertion helper in validate_telemetry.py so CI fails when a signal
regresses to absent.
Also registers the new dashboard uid with the harness so the board is
covered by validation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Strip xrpld_ prefix, lowercase beast::insight names, and replace
traces_span_metrics_ with span_ in all remaining tracked files:
alert rules, integration tests, workload validation, TESTING.md,
OpenTelemetryPlan docs, code comments, and config templates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Point the workload validation uid list and implementation-phases doc at
the renamed bare dashboard uids.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The separate xrpld-rpc-perf-otel dashboard was merged into xrpld-rpc-perf,
so the validation harness must no longer expect it to exist.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The print-env CI fix let the Telemetry Stack Validation job build and run the
workload harness end-to-end for the first time. It reported 129/136 checks
passing; this commit fixes the 7 real failures plus a latent regression-gate bug.
Validation-suite fixes (verified against the CI run's actual emission + live node):
- expected_metrics.json: the beast::insight job-depth gauge is `xrpld_jobq_job_count`,
not `xrpld_job_count` (the latter is a Phase 9 OTel counter). Reverted the prior
rename. Removed the statsd_histograms block (`xrpld_rpc_time`/`xrpld_rpc_size`):
these RPC timers do not emit under the WS workload (0 series in CI).
- expected_spans.json: `tx_status` is only set on suppressed/known-bad receives, so
it is no longer a required attribute of every `tx.receive`. Marked `pathfind.compute`
and `pathfind.discover` optional and the `pathfind.request -> pathfind.compute`
hierarchy as skip — the self-to-self XRP probe returns before computing paths in a
fresh cluster with no liquidity, so only `pathfind.request` fires.
Regression-gate bug (telemetry-validation.yml "Print regression summary"):
- `jq -e` exits non-zero when its filter result is boolean false — the normal case
for a populated (non-placeholder) baseline — which was misreported as
"Failed to parse baseline JSON" and failed the job. Dropped `-e` (kept `-r`) so a
non-zero exit genuinely means malformed JSON.
The optional-span handling and regression comparison both worked correctly in the
CI run (txq.* / pathfind.update_all skipped-when-absent, 0 regressions detected).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Phase 10 validation harness had drifted from the code's recording surface
and the telemetry-validation CI job was failing before it could build.
CI fix (telemetry-validation.yml):
- Replace nonexistent local action ./.github/actions/print-env with the remote
XRPLF/actions/print-build-env (the build-xrpld job failed in 56s on this).
- Sync prepare-runner and upload-artifact action SHAs to the canonical workflow.
Recording-surface reconciliation (docker/telemetry/workload/):
- Migrate span attributes from dotted xrpl.<domain>.<field> to the bare/underscore
form introduced by the 2026-05-13 span-attr naming redesign (tx_hash, peer_id,
ledger_seq, consensus_mode, consensus_round, full_validation, quorum, ...).
Dotted xrpl.ledger.hash is retained only on peer.validation.receive (shared
constant), while consensus.validation.send uses bare ledger_hash.
- Fix attribute placement: tx.apply carries tx_count/tx_failed (not ledger_seq);
ledger.build carries ledger_seq/close_* (not tx_count/tx_failed).
- Replace the phantom rpc.request span with the real WS root rpc.ws_message; drop
the never-emitted duration_ms; rebuild the parent-child map accordingly.
- Add the new spans the code emits: apply-pipeline stage spans
(tx.preflight/preclaim/transactor with stage/tx_type/ter_result), txq.*,
consensus sub-spans (round/establish/update_positions/check/phase.open),
ledger.acquire, grpc.*, pathfind.*. Conditional spans are marked optional so
they are skipped (not failed) when the workload does not exercise them.
- validate_telemetry.py: service.name and Loki job label rippled -> xrpld; fix
PARITY_SPAN_ATTRS (rename the 4 real attrs, drop the 3 that are metrics not span
attrs); add optional-span handling that skips missing optional spans while still
validating attributes when present.
- expected_metrics.json: rippled_ -> xrpld_ on all beast::insight/overlay metrics,
xrpld_job_count, the 15 on-disk xrpld-* dashboard UIDs, and the real bare
spanmetrics dimension labels.
- regression-metrics.json + baseline-timings.json: rpc.request -> rpc.ws_message.
Metrics pipeline fix:
- Switch node [insight] config from server=statsd/prefix=rippled to server=otel +
/v1/metrics endpoint + prefix=xrpld across run-full-validation.sh,
xrpld-validator.cfg.template, benchmark.sh and the workload compose. The
collector has no StatsD receiver, so system metrics only reach Prometheus over
OTLP.
Synthetic load for new spans:
- Add ripple_path_find to the RPC load generator (drives pathfind.* spans).
- Add a high-TPS txq-burst workload phase to force fee escalation (drives txq.*).
All facts verified against the *SpanNames.h headers and a live xrpld node +
collector (Tempo service.name=xrpld, tx.preflight attrs [stage,ter_result,tx_type],
279 xrpld_ Prometheus metrics and zero rippled_).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 10's workload validation configs (expected_metrics.json,
regression-metrics.json, validate_telemetry.py) queried the
MetricsRegistry metrics under the rippled_ prefix, but MetricsRegistry
emits them as xrpld_ (see MetricsRegistry.cpp). On a live run the
workload validator reported every MetricsRegistry metric as missing,
masking genuine regressions.
Rename the following to xrpld_ across the workload validator,
expected-metrics manifest, and regression-metrics template:
- nodestore_state, cache_metrics, txq_metrics, load_factor_metrics,
object_count
- rpc_method_started_total / _finished_total / _errored_total /
_duration_us
- job_queued_total / _started_total / _finished_total /
_queued_duration_us_bucket / _running_duration_us_bucket
- peer_quality, server_info, validator_health, ledger_economy,
db_metrics, complete_ledgers, build_info, state_tracking,
storage_detail
- ledgers_closed_total, validations_sent_total,
validations_checked_total, state_changes_total
- validation_agreement, validation_agreements_total,
validation_missed_total
Mirrors the phase-9 fix in commit 5601615952.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
