Nine conflicts, resolved as follows.
src/xrpld/app/ledger/detail/InboundLedger.cpp -- kept this branch's version.
phase10 sets the span's outcome/timeouts/peer_count attributes inline at each
exit; this branch replaced that with the idempotent finalizeAcquireSpan(), called
on all four exits (init, done, give-up, destructor). Taking phase10's blocks
would have set the outcome twice against a helper documented as not overwriting
what the real exit recorded. phase10's comment explains why peer_count must not
be read in a destructor; the helper solves that structurally by taking
std::optional<std::size_t> and being passed std::nullopt from there.
src/xrpld/telemetry/MetricsRegistry.cpp -- kept metric::ledgerEconomy over
phase10's "ledger_economy" literal. This branch added the naming check that
requires constants for converted families, so the literal would regress it. Took
phase10's comment cleanup.
src/xrpld/telemetry/MetricsRegistry.h -- kept registerRotationStateGauge(), which
only exists here, and took phase10's removal of the stale task-number comment.
validate_telemetry.py -- combined both. phase10 replaced serial metric polling
with a concurrent fan-out on one shared deadline, because 58 metrics x 45 s of
additive timeout overran the CI budget; that is kept. Its target list filters on
SKIPPED_METRIC_GROUPS rather than the two literals it hardcoded, so the
sync_diagnostics group stays owned by assert_sync_diagnostics_metrics() instead
of being polled and reported twice. Both SYNC_DIAGNOSTICS_GROUP and
METRIC_POLL_CONCURRENCY are needed and both are kept.
check_otel_naming.py -- both sides extend the rule docstring. Took phase10's
fuller Rule E text (doc discovery, allow-dotted markers) and re-appended rules
I/J/K/L, which exist only here.
expected_metrics.json -- the two sides add disjoint sibling groups, so both are
kept: sync_diagnostics alongside node_health_gauges, overlay_reduce_relay,
overlay_overflow, validation_lifetime_counters and not_asserted. Both dashboard
uids are kept, giving 16 asserted uids against 16 dashboards on disk.
expected_spans.json -- kept this branch's span set, a superset that adds the
acquire phase spans, ledger.serve, txset.acquire and peer.dial, and expands
ledger.acquire's required attributes. Took phase10's description, which documents
what the totals mean, and its note on how the RPC wildcard span is created.
total_span_types and total_unique_attributes are recomputed for the union: 48 and
74, since each side's figure counted only its own spans.
Docs: took phase10's more accurate wording on what the dashboard check actually
covers, and corrected the dashboard count from 15 to 16 where the merge made it
stale.
Verified: no conflict markers remain, both JSON contracts parse, both Python
files compile, asserted dashboard uids match the dashboards on disk exactly, and
the OTel naming check reports all layers consistent.
The harness manifests asserted things the code cannot produce and missed most
of what it does. Two assertions were failing every run, and the metric set
covered 16 of the ~41 emitted names.
expected_spans.json: rpc.process was required with rpc.ws_message as its
parent, but it is created only in ServerHandler::processRequest() on the HTTP
path, so a WebSocket-only workload never produces it -- it is now optional and
parented to rpc.http_request, and the rpc.process -> rpc.command.* edge is
skipped with the real reason instead of a coroutine-context-loss diagnosis that
was never the cause. Adds the missing rpc.ws_upgrade span, corrects four
parents (consensus.mode_change, pathfind.request, and update_positions/check,
which are children of consensus.establish rather than consensus.round), and
demotes conditionally-set attributes out of required_attributes so a healthy
run stops failing. Counts recomputed from the file: 41 span types, 62 unique
required attributes.
expected_metrics.json: 16 -> 52 asserted entries across the job-queue, RPC
method, reduce-relay, overflow and validation families, plus the fifteenth
dashboard uid. Metrics the harness workload cannot exercise -- erroring RPC,
ledger-mismatch, TxQ overflow, and the lazily-created getobject_* instruments
-- are listed in a not_asserted group the validator skips, rather than as
assertions that would fail on a healthy node.
The workflow's push trigger listed two globs matching nothing
(include/xrpl/basics/Telemetry*.h, src/xrpld/app/misc/Telemetry*), so no C++
telemetry change ever triggered validation. Replaced with the paths the code
actually lives in, including src/libxrpl/beast/insight/** for the insight
export path the harness depends on. The four inert workflow_dispatch inputs are
now labelled UNUSED rather than looking like working knobs.
Docs: the workload README described a StatsD dirty-flag mechanism under a
member name that does not exist, on a code path the harness never uses -- it
sets [insight] server=otel, so gauges export through an observable-gauge
callback every cycle. Adds the missing txq-burst phase, reconciles three
different dashboard counts, and drops "posts summary to PR", which the workflow
has no permission to do. The runbook's phase-10 section loses the last
sampling_ratio reference (not a config key), gains a Regression Gate and CI
subsection covering the gate that can fail CI, and its compose-logs command now
names the workload compose file. cmake --preset default is left for a separate
change: no CMakePresets.json is tracked, so it is wrong everywhere it appears.
Also drops the dead exporter=otlp_http key the harness wrote into every node
config, and stops capture_timings.py defaulting --profile to a profile that
does not exist.
Adversarial validation of the previous commit found one of its two code fixes
was diagnosed wrongly and the other incomplete. Both are corrected here, along
with the layers the first pass missed.
1. The new dial outcome was named for the wrong condition. It was added as
`duplicate` on the belief that PeerFinder had already granted a slot for the
address. It has not: `Logic::onConnected` contains exactly ONE false-returning
path and it is the self-connect check, which logs "Logic dropping as self
connect" (include/xrpl/peerfinder/detail/Logic.h). The duplicate check lives
in `newOutboundSlot`, evaluated before a ConnectAttempt exists, so a real
duplicate can never reach this branch.
That mattered beyond the name: the previous commit told operators the outcome
was benign churn to ignore, when it actually reports a local misconfiguration
-- this node has its own address in [ips_fixed] or behind its advertised
endpoint, and every dial to it is wasted. Renamed to `self_connection`,
reusing the slug `handshake_negotiation_fail_total` already publishes for the
same fault so it reads identically on both signals, and every description
corrected to say so. The fail() string now reads "Self connection" too.
The first pass also missed three enforcement and contract sites: the
ConnectAttempt.h Doxygen state machine (which still mapped the slot branch
onto tls_fail), the LedgerSpanNames unit test (which pinned exactly five
values over a std::array<..., 5> and so left the new member untested), and the
span-derived twin panel plus two reference docs that still published the old
five-value domain.
2. The credential-free site label was incomplete twice over.
- It appended the port, and `Resource::Resource` DEFAULTS that to 443/https
and 80/http when the config omits one. The label would have become
`https://vl.ripple.com:443/` where Grafana Cloud currently holds
`https://vl.ripple.com`, silently renaming the series for every deployment
already scraping this metric. Verified against live label values before and
after; the port is now omitted.
- parseUrl's path group is `(/.*)?`, greedy to end of string, so a query or
fragment lands inside `path`. A list URL authenticated by `?token=...` would
have leaked exactly as userinfo did. The path is now truncated at the first
'?' or '#'.
Also updated the MetricNames.h usage example, which still taught the raw-URI
pattern to the next author, and the 09-doc row that described the label as the
configured URI.
3. Rule J hardening from the same review: `classify_instrument_kind` returns an
`other` sentinel for a non-factory macro, and storing it in the kind set could
render a future conflict as "created as counter and other". The sentinel is
now skipped, keeping it doing what it already did -- matching no shape rule.
Added a second regression test whose input the pre-fix code reported as CLEAN
(gauge-then-histogram on a `_us` name), so the guard is proven by a 0-vs-1
difference and not only by a changed message. Both new tests were run against
a reconstructed last-wins implementation and both fail against it.
Documented the conflict class in the Rule J rows of the checker README and
CONTRIBUTING, which previously described only the suffix conventions.
Verified: naming checker exits 0 with Rule J passing all 40 real names; 140
checker tests pass; 15 dashboards validate; both workload JSON files parse;
clang-tidy over the full compile database reports no finding on any changed line
of ConnectAttempt.cpp or ValidatorSite.cpp; pre-commit passes.
Not verified: not compiled. The label change adds string truncation and the
outcome rename touches a constexpr used across three translation units, so CI's
build remains the first real check on both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four defects from the automated review on PR #7875, each verified against the
current tree before fixing (one further comment, the row-63 dashboard overlap,
was already fixed by an earlier commit and needed nothing).
1. Rule J could not detect an instrument-kind mismatch. instrument_kinds() wrote
`kinds[wire] = ...`, so a wire name created through two different factories
kept only the kind visited last and whichever emit site the file walk reached
last silently decided the verdict. It now collects a set per name and reports
the conflict itself -- one name exporting two instruments is the defect, and
no suffix can be correct for both. Added a regression test that builds a name
as both a counter and an observable gauge and asserts the message names both.
2. A duplicate connection was reported as `tls_fail`. The TLS handshake had in
fact succeeded; PeerFinder simply already held a slot for that address, which
is ordinary churn on a healthy node. Conflating the two made a rising
`tls_fail` unreadable -- it could mean unreachable peers or merely a busy
PeerFinder, and those need opposite responses. Added a distinct `duplicate`
outcome and carried the widened vocabulary through every place that
enumerates it: the panel description, both filter descriptions, the runbook
branch table, the runbook outcome list and the expected_spans note. The
`dial_outcome` template variable is a label_values() query, so it picks the
new value up on its own.
3. ConnectAttempt::onShutdown had no `operation_aborted` guard, unlike the five
other handlers in the same file. A clean teardown was therefore counted as
`upgrade_fail`, inflating that outcome on any node shutting down with dials in
flight.
4. ValidatorSite used the raw configured URI as a Prometheus label.
[validator_list_sites] accepts credentials in the URI and ParsedUrl keeps them
in username/password, so a configured `https://user:pass@host` would have
copied the secret into a metric label and on into the collector, Prometheus
and every dashboard. The label is now rebuilt from scheme, host, port and
path -- everything needed to tell one site apart, and nothing more.
Verified: naming checker exits 0 with Rule J still passing all 40 real
instrument names; its unit tests now number 139 and all pass; 15 dashboards
validate; both workload JSON files parse; clang-tidy over the full compile
database reports no finding on either changed .cpp; pre-commit passes.
Not verified: not compiled. Item 4 introduces string concatenation and item 2 a
new constexpr, so CI's build is the first real check on both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The board and runbook had grown by append across eight work packages, so
they read in the order the work was done rather than the order a node
progresses. This is the coherence pass; it adds no new instrumentation.
- Dashboard: 52 panels regrouped from two rows into nine that follow the
fresh-start sequence — bootstrap, peer supply, sync state, acquire and
SHAMap fetch, job queue, quorum and publish, terminal blockers, then
back-fill and spans collapsed since they answer conditional questions.
Layout only: no title, query or description changed.
- Runbook: the flat step list becomes a decision tree branching on the
observed symptom, with the amendment-block check first because it is
terminal. Each branch names the panels, what healthy and unhealthy look
like, and what to conclude. The existing steps are kept as the detail
bodies.
- Reference table: every signal name re-checked against the code and every
named panel against the board; four stale panel references fixed.
- Validation: every signal is now either asserted or covered by a note
explaining why a five-node local cluster cannot produce it.
Also fixes the write-latency signal, which was inert on a real node: the
store duration was only recorded on the database-import path, while the two
production store implementations did not time themselves, so an ordinary
node reported a write count with no latency. Both now time the backend
write, which is the disk work this signal exists to expose. Without it the
"existing database syncs slower than a fresh one" diagnosis had no primary
signal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A slow fresh-sync ledger produced spans scattered across threads with no
way to relate them. They now share a trace id derived from the ledger's own
hash, the one value every participating site already holds, so nothing new
is plumbed across threads. This is the pattern the transaction pipeline
already uses for its tx id.
Joined: ledger.validate, ledger.store, and a new
consensus.validation.accept recorded when a trusted validation arrives. In
Tempo, searching one ledger hash returns them together, so an operator can
tell whether the ledger was slow to arrive, slow to be accepted, or slow to
be stored. They are siblings rather than a chain because the accept gate is
entered from three different threads, so no fixed parent order exists.
consensus.validation.accept also records why an arriving validation did or
did not advance the gate, which makes "validations arrive but are all
rejected" visible for the first time.
consensus_round_duration_ms turns the existing round-time span attribute
into a histogram, so a fleet trend needs a metric query rather than raw
trace inspection. An explicit bucket view is required, not optional: the
SDK default tops out at ten seconds while consensus abandons a round at two
minutes, so slow rounds would all fall in one bucket and every quantile
would read exactly ten seconds. Cost is one record per round.
Record layer: the histogram is native and needs no collector change. The
two new bounded attributes are added as span-metric dimensions to both
collector configs. The ledger hash stays out of them, since a per-ledger
dimension mints a series per ledger; it is indexed in Tempo as the join key.
The ledger.acquire span is not joined yet, because that file was being
changed concurrently. It is registered as an optional member of the join
group so nothing fails, and switching it is a one-line follow-up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The span only recorded an outcome on the normal completion path. An
acquire that stalled and was later swept ended with no outcome at all, and
its duration stretched to the sweep interval rather than the real fetch
time. So the one case these signals exist to catch, a fetch that never
finishes, was the one case that could not be traced, and aggregate outcome
and timeout rates read low exactly when nodes are stuck.
- Adds an abandoned outcome value for the swept-while-fetching case.
- Routes every exit through one idempotent finalizer, so a span is
finalized exactly once whether it completes, fails, short-circuits on
local data, or is destroyed mid-fetch. The destructor path cannot throw.
- Adds the ledger hash to the span and backfills the sequence once known,
since by-hash acquires start without one and could not otherwise be tied
to a specific ledger.
- Record layer: outcome stays a span-metrics dimension in both collector
configs, which drift apart if only one is edited. The ledger hash is
indexed in Tempo for trace search instead, because a per-ledger value as
a metric dimension would mint a new series every ledger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- peer.validation.receive now asserts the shared bare ledger_hash /
full_validation keys (was the dotted xrpl.ledger.hash and validation_full);
PARITY_SPAN_ATTRS checks both on the peer span too.
- Fix a span-name drift: the per-transaction accept span is txq.accept_tx
(op::acceptTx = "accept_tx"), not txq.accept.tx — the old assertion never
matched and was silently skipped as optional.
- Drop the "intentionally dotted" notes; there is no dotted span attribute.
The txq.enqueue span sets tx_hash, tx_type, and txq_status on every
code path (TxQ.cpp:746/748/751), but fee_level_paid and
required_fee_level are set only on the fee-evaluated path
(TxQ.cpp:895-898), which is reached after the rejected and
applied_direct early exits. They are therefore not guaranteed on every
txq.enqueue span, so requiring them caused the validation harness to
fail whenever a txq.enqueue span took an early-exit path.
Remove the two conditional attributes from required_attributes and
document why in the span note.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The print-env CI fix let the Telemetry Stack Validation job build and run the
workload harness end-to-end for the first time. It reported 129/136 checks
passing; this commit fixes the 7 real failures plus a latent regression-gate bug.
Validation-suite fixes (verified against the CI run's actual emission + live node):
- expected_metrics.json: the beast::insight job-depth gauge is `xrpld_jobq_job_count`,
not `xrpld_job_count` (the latter is a Phase 9 OTel counter). Reverted the prior
rename. Removed the statsd_histograms block (`xrpld_rpc_time`/`xrpld_rpc_size`):
these RPC timers do not emit under the WS workload (0 series in CI).
- expected_spans.json: `tx_status` is only set on suppressed/known-bad receives, so
it is no longer a required attribute of every `tx.receive`. Marked `pathfind.compute`
and `pathfind.discover` optional and the `pathfind.request -> pathfind.compute`
hierarchy as skip — the self-to-self XRP probe returns before computing paths in a
fresh cluster with no liquidity, so only `pathfind.request` fires.
Regression-gate bug (telemetry-validation.yml "Print regression summary"):
- `jq -e` exits non-zero when its filter result is boolean false — the normal case
for a populated (non-placeholder) baseline — which was misreported as
"Failed to parse baseline JSON" and failed the job. Dropped `-e` (kept `-r`) so a
non-zero exit genuinely means malformed JSON.
The optional-span handling and regression comparison both worked correctly in the
CI run (txq.* / pathfind.update_all skipped-when-absent, 0 regressions detected).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Phase 10 validation harness had drifted from the code's recording surface
and the telemetry-validation CI job was failing before it could build.
CI fix (telemetry-validation.yml):
- Replace nonexistent local action ./.github/actions/print-env with the remote
XRPLF/actions/print-build-env (the build-xrpld job failed in 56s on this).
- Sync prepare-runner and upload-artifact action SHAs to the canonical workflow.
Recording-surface reconciliation (docker/telemetry/workload/):
- Migrate span attributes from dotted xrpl.<domain>.<field> to the bare/underscore
form introduced by the 2026-05-13 span-attr naming redesign (tx_hash, peer_id,
ledger_seq, consensus_mode, consensus_round, full_validation, quorum, ...).
Dotted xrpl.ledger.hash is retained only on peer.validation.receive (shared
constant), while consensus.validation.send uses bare ledger_hash.
- Fix attribute placement: tx.apply carries tx_count/tx_failed (not ledger_seq);
ledger.build carries ledger_seq/close_* (not tx_count/tx_failed).
- Replace the phantom rpc.request span with the real WS root rpc.ws_message; drop
the never-emitted duration_ms; rebuild the parent-child map accordingly.
- Add the new spans the code emits: apply-pipeline stage spans
(tx.preflight/preclaim/transactor with stage/tx_type/ter_result), txq.*,
consensus sub-spans (round/establish/update_positions/check/phase.open),
ledger.acquire, grpc.*, pathfind.*. Conditional spans are marked optional so
they are skipped (not failed) when the workload does not exercise them.
- validate_telemetry.py: service.name and Loki job label rippled -> xrpld; fix
PARITY_SPAN_ATTRS (rename the 4 real attrs, drop the 3 that are metrics not span
attrs); add optional-span handling that skips missing optional spans while still
validating attributes when present.
- expected_metrics.json: rippled_ -> xrpld_ on all beast::insight/overlay metrics,
xrpld_job_count, the 15 on-disk xrpld-* dashboard UIDs, and the real bare
spanmetrics dimension labels.
- regression-metrics.json + baseline-timings.json: rpc.request -> rpc.ws_message.
Metrics pipeline fix:
- Switch node [insight] config from server=statsd/prefix=rippled to server=otel +
/v1/metrics endpoint + prefix=xrpld across run-full-validation.sh,
xrpld-validator.cfg.template, benchmark.sh and the workload compose. The
collector has no StatsD receiver, so system metrics only reach Prometheus over
OTLP.
Synthetic load for new spans:
- Add ripple_path_find to the RPC load generator (drives pathfind.* spans).
- Add a high-TPS txq-burst workload phase to force fee escalation (drives txq.*).
All facts verified against the *SpanNames.h headers and a live xrpld node +
collector (Tempo service.name=xrpld, tx.preflight attrs [stage,ter_result,tx_type],
279 xrpld_ Prometheus metrics and zero rippled_).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Remove duplicate 'system-node-health' UID from expected_metrics.json
(already covered by 'rippled-system-node-health')
- Add parity span attributes to expected_spans.json: node health on
rpc.command.*, validation hash/full on consensus.validation.send,
quorum/proposers on consensus.accept, validation hash/full on
peer.validation.receive
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
