Fixes the review findings on this PR that belong to files it owns, plus
several defects found while verifying those fixes. Findings in files owned
by upstream branches are routed there and left untouched here.
Correctness:
- tx_submitter: advance the account sequence only on results that actually
consume one (tes*, tec*, terQUEUED). tem*/tef*/tel* never reach the
ledger, so advancing left a permanent gap that every later submit from
that account inherited. Add a re-fetch hatch so a repeated non-consuming
failure cannot livelock on the same sequence, and gate the account check
on funded-ness rather than list length.
- validate_telemetry: filter spans by name before collecting attributes, so
a per-span attribute contract can no longer be satisfied by a sibling
span; require exact name equality for non-wildcard children and glob
matching for wildcards; bounds-check every returned series instead of
only the first.
- collect_system_metrics: select xrpld by argv[0] rather than a substring
match on the whole command line, which averaged in unrelated processes
and reported their RSS as xrpld's. Count genuine 0.0 CPU readings, use a
clamped nearest-rank p99 index, and record RPC latency only on success.
- benchmark: return each verdict through a named variable instead of a
command substitution, so the pass/fail counters survive and the exit gate
can fire. Scale before dividing in the percentage math, which truncated a
1.26% impact to 1.00% and cleared a 1% threshold.
- compare_to_baseline: fall back to the absolute bound when the baseline is
not positive, so a 0 -> 500 ms jump is no longer "within bounds".
- rpc_load_generator: bound each connection to one in-flight recv(), drain
in-flight requests before closing, use a nearest-rank percentile, and
report delivery shortfall so an under-delivered run cannot pass with a 0%
error rate.
Fail loudly instead of silently:
- run-full-validation: treat a consensus timeout and a missing validated
ledger as fatal infrastructure errors, and fold the orchestrator and
benchmark exit codes into the final status. A degraded cluster previously
ran a full validation pass and reported misleading downstream failures.
- collect_system_metrics: warn per empty measurement source, emit
metrics_complete, and exit non-zero instead of substituting zeros that
pass every threshold. Require GNU date with %N rather than falling back
to a per-sample python3 fork that costs more than the threshold it is
measured against.
- benchmark: distinguish "could not measure" from "exceeded thresholds",
install a cleanup trap so a failure cannot leak nodes and ports, and
report an unusable baseline as inconclusive.
- workload_orchestrator: bound subprocess communicate() and fail the exit
gate on per-phase errors.
Also pins the workload compose images to the versions the sibling stack
already uses, hash-pins the Python dependencies, restricts the validator
config template to loopback, corrects the dashboard and metric counts in
the reference docs, drops a span from the regression gate that cannot fire
under a WebSocket-only workload, and narrows the teardown pkill pattern so
it no longer matches processes that merely mention the work directory.
Verified with a full harness run against a local five-node cluster:
158 of 158 checks passed with no regressions detected.
Two gaps meant the last failure produced no evidence of its cause.
The node-log upload was gated on `if: failure()`, but the validation step
sets continue-on-error, so the job is not failing at that point and the
condition never fired. Every failed run silently skipped the one artifact
that records why a node did not reach consensus. It now keys on the
validation step's own outcome, and also collects the harness logs.
Transaction failures were logged at DEBUG, which CI does not enable, so a
run where all 3052 submissions failed on a refused connection reported
nothing about it. The first occurrence of each distinct failure kind is now
a warning and repeats stay at DEBUG, so one refused connection says so once
instead of 3052 times.
Two CI failures traced to root cause:
1. rippled_jobq_job_count: 0 series — StatsDGaugeImpl declared
m_dirty{false} despite the constructor comment saying "start dirty".
Gauges whose value starts and stays at 0 never emitted, so Prometheus
never scraped them. Fix: m_dirty{true} on the member initializer.
2. TX error rate 82.8% — the submitter tracked account sequences
locally, but in a multi-node consensus network other nodes' txns
advance sequences independently. After a few ledger closes the
locally-tracked sequence fell behind the ledger, producing
tefPAST_SEQ for every subsequent submission. Fix: refresh account
sequences from account_info every 10 s during the submission loop.
