Both workload scripts decoded JSON and went straight to .get() or .items().
An array or a scalar decodes fine and then raises AttributeError, which is not a
ValueError, so the handler around the weights parsing could not catch it and the
operator saw a traceback naming neither the command nor what arrived.
Four sites across the two files, all four guarded, and the weights handler now
also catches TypeError for a non-numeric weight value.
Negative weights are left alone: they do not raise, the type is dropped from the
mix silently, and changing that changes accepted input.
Three ways a run could produce no traffic and still report success:
- tx_submitter logged a funding shortfall and returned an empty stats object;
main() then printed the summary and exited 0, so the failure only surfaced
later as "spans missing", which points nowhere. It now records setup_failed in
the summary and exits 1 after the report is written.
- --weights was checked for valid JSON but not for a positive sum. An all-zero
mapping reached random.choices, which raises ValueError from inside the
dispatch loop where only CancelledError is caught. Rejected at parse time now,
in both generators.
- a profile phase declaring neither rpc nor tx logged a warning and returned no
error. Both error rates short-circuit to 0.0 when nothing was sent, so a
mistyped key produced zero traffic and still passed the exit gate. That phase
is now an error.
Eight defects found in review of PR 6519. Twenty-four review threads reported
them; ten were duplicates of one another and four were wrong about the code.
The two that corrupt data. Both WebSocket clients reuse the socket after a recv
timeout, and the library queues the late reply, so the NEXT request reads the
previous response. In rpc_load_generator that misattributes latency, and the
skew is permanent rather than one-off. In tx_submitter it is worse: a submit
that reads an account_info reply freezes that account's sequence number and
every later transaction for it fails. Both now correlate replies by request id
under a single overall deadline, with a counter rather than a wall clock, since
time.time() is not monotonic and collides within a tick.
workload_orchestrator never cleared its fixed report paths, so a run that
produced no report silently adopted the previous run's totals -- breaking the
invariant evaluate_exit_gate documents. Reproduced by planting a stale total
and watching it appear in a later summary.
collect_system_metrics reported placeholders as if they were measurements. The
consensus mean used bc with a || echo 0 fallback that neither warned nor
cleared METRICS_COMPLETE, unlike every sibling path; it now uses awk, already a
hard dependency here, which removes the failure mode instead of reporting it.
Note this moves the mean from truncation to rounding, at most 1 ms on a value
of about 45 s. Unmeasurable TPS now warns and clears the flag too. All four
curl probes gained a timeout, not just the one the review named -- an
unresponsive node could hang any of them.
The orchestrator's help text claimed 18-dashboard coverage; there are 15 on
disk, 15 uids in the contract, and the profile already said 15. The
tx_submitter docstring listed twelve transaction types where ten exist, and
claimed issued-currency payments that build_payment never sends.
Two suggested patches were deliberately not taken. A recursive delete of the report
parent sits in a per-task function and would delete earlier phases' reports
mid-run, and recursively remove a caller-supplied --report-dir.
The stale microsecond axis label on ledger-data-sync is real but belongs to
phase 9, which carries a byte-identical copy of that dashboard, so fixing it
here would leave that PR wrong and guarantee a conflict.
Every node the harness starts is a validator, and validators disable
pathfinding: Config.cpp:725-726 zeroes pathSearchMax whenever a
[validation_seed] or [validator_token] section is present, and
run-full-validation.sh writes [validation_seed] into every generated node
cfg (:308) with no [path_search] section to put the default back. So
doRipplePathFind refused every call at RipplePathFind.cpp:48-49 and the
3% ripple_path_find weight bought no coverage at all.
It was not free either. The pathfind.request guard is constructed at
RipplePathFind.cpp:35, above that refusal, so each refused call still
exported a span, and the enclosing rpc.command.ripple_path_find span
carried rpc_status=error. That put a steady 3% error floor into
span_calls_total for STATUS_CODE_ERROR: any error-rate threshold derived
from harness data before this change was measuring the harness rather
than xrpld, and needs re-deriving.
Removing the load makes pathfind.request unreachable, so it moves from
required to optional in expected_spans.json; without that the span check
would fail on every run. Three notes in that file and three in
expected_metrics.json made claims that are now false, two of them citing
line numbers this commit deletes; all six are corrected. The runbook
required/optional count moves 26/15 to 25/16.
Two facts a future reader needs.
First, the weights previously summed to 103, not 100, so every percentage
the docstring stated was wrong: health checks were really 38.8%, not 40%.
Dropping the 3 makes the sum exactly 100 and every stated percentage
correct for the first time. expected_spans.json also carried live
arithmetic off the old total, "25/103 ... roughly 43%", now 25/100 and
42%.
Second, baselines/baseline-timings.json was captured WITH this load. Only
span.rpc.ws_message p50/p95/p99 of the 25 gated keys sees the RPC mix,
and their trip points sit 3.1x to 5.9x above baseline, so the gate will
not fire. But a timing baseline is workload-specific and its profile
field still reads full-validation, so nothing will flag the drift:
refresh it from the next CI run's timings artifact.
Pathfinding now has no coverage in this harness at all. The workload
README section "Pathfinding is not exercised" records that cost, the
manual verification route, and a four-step restore recipe in which steps
1 and 2 alone only reinstate the error floor.
Two defects reported against the validation harness. Both premises were
correct, but neither suggested fix was, so the remedies differ.
Dashboard panel count: `len(dashboard["panels"])` treated Grafana row
objects as panels and skipped the panels nested inside collapsed rows, so
every dashboard was over-reported by between 1 and 10 (`log-derived-insights`
read 41 against a true 31). The check also passed unconditionally on HTTP
200, so a dashboard that renders nothing would still pass. `_leaf_panel_count`
now walks row children and the result gates the verdict. Gating on the old
top-level length, as suggested, would not have caught the case it was aimed
at: a dashboard made only of collapsed rows counts its rows and reports a
positive number while rendering nothing.
RPC latency percentiles: `LoadStats.record` appended a latency for every
outcome, including requests that never got a reply, where the value is a
time-to-failure rather than a round trip. A timeout contributed the full
receive timeout, and at the error rate a real run shows this reported p95 and
p99 of 10000 ms where the true figure was 5 ms. `record` now takes an
optional latency and the timeout path passes none. The suggestion to append
only on success was not adopted: a reply carrying `status: error` is a
completed, timely round trip whose latency is a genuine measurement, and
discarding it would throw away real data. `per_command` is now keyed off the
request counts rather than the latency map, so a command whose every request
timed out still appears in the report instead of vanishing from it, and each
entry carries a `latency_samples` count.
Fixes the review findings on this PR that belong to files it owns, plus
several defects found while verifying those fixes. Findings in files owned
by upstream branches are routed there and left untouched here.
Correctness:
- tx_submitter: advance the account sequence only on results that actually
consume one (tes*, tec*, terQUEUED). tem*/tef*/tel* never reach the
ledger, so advancing left a permanent gap that every later submit from
that account inherited. Add a re-fetch hatch so a repeated non-consuming
failure cannot livelock on the same sequence, and gate the account check
on funded-ness rather than list length.
- validate_telemetry: filter spans by name before collecting attributes, so
a per-span attribute contract can no longer be satisfied by a sibling
span; require exact name equality for non-wildcard children and glob
matching for wildcards; bounds-check every returned series instead of
only the first.
- collect_system_metrics: select xrpld by argv[0] rather than a substring
match on the whole command line, which averaged in unrelated processes
and reported their RSS as xrpld's. Count genuine 0.0 CPU readings, use a
clamped nearest-rank p99 index, and record RPC latency only on success.
- benchmark: return each verdict through a named variable instead of a
command substitution, so the pass/fail counters survive and the exit gate
can fire. Scale before dividing in the percentage math, which truncated a
1.26% impact to 1.00% and cleared a 1% threshold.
- compare_to_baseline: fall back to the absolute bound when the baseline is
not positive, so a 0 -> 500 ms jump is no longer "within bounds".
- rpc_load_generator: bound each connection to one in-flight recv(), drain
in-flight requests before closing, use a nearest-rank percentile, and
report delivery shortfall so an under-delivered run cannot pass with a 0%
error rate.
Fail loudly instead of silently:
- run-full-validation: treat a consensus timeout and a missing validated
ledger as fatal infrastructure errors, and fold the orchestrator and
benchmark exit codes into the final status. A degraded cluster previously
ran a full validation pass and reported misleading downstream failures.
- collect_system_metrics: warn per empty measurement source, emit
metrics_complete, and exit non-zero instead of substituting zeros that
pass every threshold. Require GNU date with %N rather than falling back
to a per-sample python3 fork that costs more than the threshold it is
measured against.
- benchmark: distinguish "could not measure" from "exceeded thresholds",
install a cleanup trap so a failure cannot leak nodes and ports, and
report an unusable baseline as inconclusive.
- workload_orchestrator: bound subprocess communicate() and fail the exit
gate on per-phase errors.
Also pins the workload compose images to the versions the sibling stack
already uses, hash-pins the Python dependencies, restricts the validator
config template to loopback, corrects the dashboard and metric counts in
the reference docs, drops a span from the regression gate that cannot fire
under a WebSocket-only workload, and narrows the teardown pkill pattern so
it no longer matches processes that merely mention the work directory.
Verified with a full harness run against a local five-node cluster:
158 of 158 checks passed with no regressions detected.
The Phase 10 validation harness had drifted from the code's recording surface
and the telemetry-validation CI job was failing before it could build.
CI fix (telemetry-validation.yml):
- Replace nonexistent local action ./.github/actions/print-env with the remote
XRPLF/actions/print-build-env (the build-xrpld job failed in 56s on this).
- Sync prepare-runner and upload-artifact action SHAs to the canonical workflow.
Recording-surface reconciliation (docker/telemetry/workload/):
- Migrate span attributes from dotted xrpl.<domain>.<field> to the bare/underscore
form introduced by the 2026-05-13 span-attr naming redesign (tx_hash, peer_id,
ledger_seq, consensus_mode, consensus_round, full_validation, quorum, ...).
Dotted xrpl.ledger.hash is retained only on peer.validation.receive (shared
constant), while consensus.validation.send uses bare ledger_hash.
- Fix attribute placement: tx.apply carries tx_count/tx_failed (not ledger_seq);
ledger.build carries ledger_seq/close_* (not tx_count/tx_failed).
- Replace the phantom rpc.request span with the real WS root rpc.ws_message; drop
the never-emitted duration_ms; rebuild the parent-child map accordingly.
- Add the new spans the code emits: apply-pipeline stage spans
(tx.preflight/preclaim/transactor with stage/tx_type/ter_result), txq.*,
consensus sub-spans (round/establish/update_positions/check/phase.open),
ledger.acquire, grpc.*, pathfind.*. Conditional spans are marked optional so
they are skipped (not failed) when the workload does not exercise them.
- validate_telemetry.py: service.name and Loki job label rippled -> xrpld; fix
PARITY_SPAN_ATTRS (rename the 4 real attrs, drop the 3 that are metrics not span
attrs); add optional-span handling that skips missing optional spans while still
validating attributes when present.
- expected_metrics.json: rippled_ -> xrpld_ on all beast::insight/overlay metrics,
xrpld_job_count, the 15 on-disk xrpld-* dashboard UIDs, and the real bare
spanmetrics dimension labels.
- regression-metrics.json + baseline-timings.json: rpc.request -> rpc.ws_message.
Metrics pipeline fix:
- Switch node [insight] config from server=statsd/prefix=rippled to server=otel +
/v1/metrics endpoint + prefix=xrpld across run-full-validation.sh,
xrpld-validator.cfg.template, benchmark.sh and the workload compose. The
collector has no StatsD receiver, so system metrics only reach Prometheus over
OTLP.
Synthetic load for new spans:
- Add ripple_path_find to the RPC load generator (drives pathfind.* spans).
- Add a high-TPS txq-burst workload phase to force fee escalation (drives txq.*).
All facts verified against the *SpanNames.h headers and a live xrpld node +
collector (Tempo service.name=xrpld, tx.preflight attrs [stage,ter_result,tx_type],
279 xrpld_ Prometheus metrics and zero rippled_).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
