Account setup submitted the funding Payments, slept a flat 10 seconds, then
read each sequence once. The txq-burst and mixed-peak phases escalate the
open-ledger fee on purpose, so the funding transactions were queued, every
account read Sequence 0, and the phase aborted with "only 0 of 8 created
accounts were funded". The run then reddened on a workload gate rather than on
anything telemetry had done.
Poll the ledger until each account has a sequence, with a deadline, so a late
confirmation is still seen and a healthy cluster pays no waiting cost. Pay a
multiple of the current open-ledger fee, so funding is not queued behind the
load a phase creates deliberately. terQUEUED no longer marks an account funded:
only a ledger read does.
Retry the accounts that never confirmed, once, after re-reading the genesis
sequence from the ledger. consumes_sequence advances the local counter on
terQUEUED, so a dropped funding transaction leaves it ahead of the ledger and
every resubmit would otherwise land on a future sequence.
The funding wait can run twice, so raise the orchestrator's grace above twice
the timeout. A test pins that relationship, since the two constants live in
different files.
Also save each generator's full stdout and stderr beside its JSON report. Only
the last 200 characters of stderr reached the phase error and stdout was
dropped, so none of the per-account funding results appeared in CI.
Three ways a run could produce no traffic and still report success:
- tx_submitter logged a funding shortfall and returned an empty stats object;
main() then printed the summary and exited 0, so the failure only surfaced
later as "spans missing", which points nowhere. It now records setup_failed in
the summary and exits 1 after the report is written.
- --weights was checked for valid JSON but not for a positive sum. An all-zero
mapping reached random.choices, which raises ValueError from inside the
dispatch loop where only CancelledError is caught. Rejected at parse time now,
in both generators.
- a profile phase declaring neither rpc nor tx logged a warning and returned no
error. Both error rates short-circuit to 0.0 when nothing was sent, so a
mistyped key produced zero traffic and still passed the exit gate. That phase
is now an error.
Three harness comments described the behaviour this change replaced, which the
squash merge does not publish.
- run-full-validation.sh: the capture flag means CAPTURE_EXIT is not the only
record of capture health; and a gated capture failure is an infrastructure
error, stated without 'exactly as before'.
- tx_submitter.py: give the reason the first occurrence logs at WARNING (DEBUG
is off in CI) rather than what a failed run 'previously produced'.
- workload_orchestrator.py: a wedged process cannot stall the profile, rather
than 'can no longer'.
Comments only, no behaviour change.
Eight defects found in review of PR 6519. Twenty-four review threads reported
them; ten were duplicates of one another and four were wrong about the code.
The two that corrupt data. Both WebSocket clients reuse the socket after a recv
timeout, and the library queues the late reply, so the NEXT request reads the
previous response. In rpc_load_generator that misattributes latency, and the
skew is permanent rather than one-off. In tx_submitter it is worse: a submit
that reads an account_info reply freezes that account's sequence number and
every later transaction for it fails. Both now correlate replies by request id
under a single overall deadline, with a counter rather than a wall clock, since
time.time() is not monotonic and collides within a tick.
workload_orchestrator never cleared its fixed report paths, so a run that
produced no report silently adopted the previous run's totals -- breaking the
invariant evaluate_exit_gate documents. Reproduced by planting a stale total
and watching it appear in a later summary.
collect_system_metrics reported placeholders as if they were measurements. The
consensus mean used bc with a || echo 0 fallback that neither warned nor
cleared METRICS_COMPLETE, unlike every sibling path; it now uses awk, already a
hard dependency here, which removes the failure mode instead of reporting it.
Note this moves the mean from truncation to rounding, at most 1 ms on a value
of about 45 s. Unmeasurable TPS now warns and clears the flag too. All four
curl probes gained a timeout, not just the one the review named -- an
unresponsive node could hang any of them.
The orchestrator's help text claimed 18-dashboard coverage; there are 15 on
disk, 15 uids in the contract, and the profile already said 15. The
tx_submitter docstring listed twelve transaction types where ten exist, and
claimed issued-currency payments that build_payment never sends.
Two suggested patches were deliberately not taken. A recursive delete of the report
parent sits in a per-task function and would delete earlier phases' reports
mid-run, and recursively remove a caller-supplied --report-dir.
The stale microsecond axis label on ledger-data-sync is real but belongs to
phase 9, which carries a byte-identical copy of that dashboard, so fixing it
here would leave that PR wrong and guarantee a conflict.
Fixes the review findings on this PR that belong to files it owns, plus
several defects found while verifying those fixes. Findings in files owned
by upstream branches are routed there and left untouched here.
Correctness:
- tx_submitter: advance the account sequence only on results that actually
consume one (tes*, tec*, terQUEUED). tem*/tef*/tel* never reach the
ledger, so advancing left a permanent gap that every later submit from
that account inherited. Add a re-fetch hatch so a repeated non-consuming
failure cannot livelock on the same sequence, and gate the account check
on funded-ness rather than list length.
- validate_telemetry: filter spans by name before collecting attributes, so
a per-span attribute contract can no longer be satisfied by a sibling
span; require exact name equality for non-wildcard children and glob
matching for wildcards; bounds-check every returned series instead of
only the first.
- collect_system_metrics: select xrpld by argv[0] rather than a substring
match on the whole command line, which averaged in unrelated processes
and reported their RSS as xrpld's. Count genuine 0.0 CPU readings, use a
clamped nearest-rank p99 index, and record RPC latency only on success.
- benchmark: return each verdict through a named variable instead of a
command substitution, so the pass/fail counters survive and the exit gate
can fire. Scale before dividing in the percentage math, which truncated a
1.26% impact to 1.00% and cleared a 1% threshold.
- compare_to_baseline: fall back to the absolute bound when the baseline is
not positive, so a 0 -> 500 ms jump is no longer "within bounds".
- rpc_load_generator: bound each connection to one in-flight recv(), drain
in-flight requests before closing, use a nearest-rank percentile, and
report delivery shortfall so an under-delivered run cannot pass with a 0%
error rate.
Fail loudly instead of silently:
- run-full-validation: treat a consensus timeout and a missing validated
ledger as fatal infrastructure errors, and fold the orchestrator and
benchmark exit codes into the final status. A degraded cluster previously
ran a full validation pass and reported misleading downstream failures.
- collect_system_metrics: warn per empty measurement source, emit
metrics_complete, and exit non-zero instead of substituting zeros that
pass every threshold. Require GNU date with %N rather than falling back
to a per-sample python3 fork that costs more than the threshold it is
measured against.
- benchmark: distinguish "could not measure" from "exceeded thresholds",
install a cleanup trap so a failure cannot leak nodes and ports, and
report an unusable baseline as inconclusive.
- workload_orchestrator: bound subprocess communicate() and fail the exit
gate on per-phase errors.
Also pins the workload compose images to the versions the sibling stack
already uses, hash-pins the Python dependencies, restricts the validator
config template to loopback, corrects the dashboard and metric counts in
the reference docs, drops a span from the regression gate that cannot fire
under a WebSocket-only workload, and narrows the teardown pkill pattern so
it no longer matches processes that merely mention the work directory.
Verified with a full harness run against a local five-node cluster:
158 of 158 checks passed with no regressions detected.
Raise TX error rate threshold from 50% to 95%. Short-lived CI test
environments lack pre-funded accounts for complex transactions
(AMMCreate, EscrowFinish, NFTokenCreateOffer, etc.), causing expected
failures that do not indicate an instrumentation problem.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
