Files
rippled/docker/telemetry/workload/generate-validator-keys.sh
Pratik Mankawde d059f21bf3 fix(telemetry): address review findings in the workload validation harness
Fixes the review findings on this PR that belong to files it owns, plus
several defects found while verifying those fixes. Findings in files owned
by upstream branches are routed there and left untouched here.

Correctness:
- tx_submitter: advance the account sequence only on results that actually
  consume one (tes*, tec*, terQUEUED). tem*/tef*/tel* never reach the
  ledger, so advancing left a permanent gap that every later submit from
  that account inherited. Add a re-fetch hatch so a repeated non-consuming
  failure cannot livelock on the same sequence, and gate the account check
  on funded-ness rather than list length.
- validate_telemetry: filter spans by name before collecting attributes, so
  a per-span attribute contract can no longer be satisfied by a sibling
  span; require exact name equality for non-wildcard children and glob
  matching for wildcards; bounds-check every returned series instead of
  only the first.
- collect_system_metrics: select xrpld by argv[0] rather than a substring
  match on the whole command line, which averaged in unrelated processes
  and reported their RSS as xrpld's. Count genuine 0.0 CPU readings, use a
  clamped nearest-rank p99 index, and record RPC latency only on success.
- benchmark: return each verdict through a named variable instead of a
  command substitution, so the pass/fail counters survive and the exit gate
  can fire. Scale before dividing in the percentage math, which truncated a
  1.26% impact to 1.00% and cleared a 1% threshold.
- compare_to_baseline: fall back to the absolute bound when the baseline is
  not positive, so a 0 -> 500 ms jump is no longer "within bounds".
- rpc_load_generator: bound each connection to one in-flight recv(), drain
  in-flight requests before closing, use a nearest-rank percentile, and
  report delivery shortfall so an under-delivered run cannot pass with a 0%
  error rate.

Fail loudly instead of silently:
- run-full-validation: treat a consensus timeout and a missing validated
  ledger as fatal infrastructure errors, and fold the orchestrator and
  benchmark exit codes into the final status. A degraded cluster previously
  ran a full validation pass and reported misleading downstream failures.
- collect_system_metrics: warn per empty measurement source, emit
  metrics_complete, and exit non-zero instead of substituting zeros that
  pass every threshold. Require GNU date with %N rather than falling back
  to a per-sample python3 fork that costs more than the threshold it is
  measured against.
- benchmark: distinguish "could not measure" from "exceeded thresholds",
  install a cleanup trap so a failure cannot leak nodes and ports, and
  report an unusable baseline as inconclusive.
- workload_orchestrator: bound subprocess communicate() and fail the exit
  gate on per-phase errors.

Also pins the workload compose images to the versions the sibling stack
already uses, hash-pins the Python dependencies, restricts the validator
config template to loopback, corrects the dashboard and metric counts in
the reference docs, drops a span from the regression gate that cannot fire
under a WebSocket-only workload, and narrows the teardown pkill pattern so
it no longer matches processes that merely mention the work directory.

Verified with a full harness run against a local five-node cluster:
158 of 158 checks passed with no regressions detected.
2026-08-14 19:59:19 +01:00

160 lines
4.8 KiB
Bash
Executable File

#!/usr/bin/env bash
# generate-validator-keys.sh — Generate validator key pairs for the workload harness.
#
# Uses a temporary standalone xrpld instance to call `validation_create` RPC
# for each node. Outputs a JSON file mapping node index to seed + public key.
#
# Usage:
# ./generate-validator-keys.sh <xrpld_binary> <num_nodes> <output_dir>
#
# Output:
# <output_dir>/validator-keys.json — JSON array of {index, seed, public_key}
# <output_dir>/validators.txt — [validators] section for xrpld.cfg
set -euo pipefail
# ---------------------------------------------------------------------------
# Colored output helpers
# ---------------------------------------------------------------------------
log() { printf "\033[1;34m[KEYGEN]\033[0m %s\n" "$*"; }
ok() { printf "\033[1;32m[KEYGEN]\033[0m %s\n" "$*"; }
die() {
printf "\033[1;31m[KEYGEN]\033[0m %s\n" "$*" >&2
exit 1
}
# ---------------------------------------------------------------------------
# Argument parsing
# ---------------------------------------------------------------------------
usage() {
echo "Usage: $0 <xrpld_binary> <num_nodes> <output_dir>"
echo ""
echo "Arguments:"
echo " xrpld_binary Path to xrpld binary (built with telemetry=ON)"
echo " num_nodes Number of validator key pairs to generate (1-20)"
echo " output_dir Directory to write validator-keys.json and validators.txt"
exit 1
}
if [ $# -lt 3 ]; then
usage
fi
XRPLD="$1"
NUM_NODES="$2"
OUTPUT_DIR="$3"
# Validate arguments
[ -x "$XRPLD" ] || die "xrpld binary not found or not executable: $XRPLD"
[[ "$NUM_NODES" =~ ^[0-9]+$ ]] || die "num_nodes must be a positive integer"
[ "$NUM_NODES" -ge 1 ] && [ "$NUM_NODES" -le 20 ] || die "num_nodes must be between 1 and 20"
mkdir -p "$OUTPUT_DIR"
# ---------------------------------------------------------------------------
# Start a temporary standalone xrpld for key generation
# ---------------------------------------------------------------------------
TEMP_DIR="$(mktemp -d)"
TEMP_PORT=5099
TEMP_CFG="$TEMP_DIR/xrpld.cfg"
log "Starting temporary xrpld for key generation (port $TEMP_PORT)..."
cat >"$TEMP_CFG" <<EOCFG
[server]
port_rpc_keygen
[port_rpc_keygen]
port = $TEMP_PORT
ip = 127.0.0.1
admin = 127.0.0.1
protocol = http
[node_db]
type=NuDB
path=$TEMP_DIR/nudb
online_delete=256
[database_path]
$TEMP_DIR/db
[debug_logfile]
$TEMP_DIR/debug.log
[ssl_verify]
0
EOCFG
"$XRPLD" --conf "$TEMP_CFG" -a --start >"$TEMP_DIR/stdout.log" 2>&1 &
TEMP_PID=$!
# Ensure cleanup on exit
cleanup_temp() {
kill "$TEMP_PID" 2>/dev/null || true
wait "$TEMP_PID" 2>/dev/null || true
rm -rf "$TEMP_DIR"
}
trap cleanup_temp EXIT
# Wait for RPC to become available
for attempt in $(seq 1 30); do
if curl -sf "http://localhost:$TEMP_PORT" \
-d '{"method":"server_info"}' >/dev/null 2>&1; then
log "Temporary xrpld RPC ready (attempt $attempt)."
break
fi
if [ "$attempt" -eq 30 ]; then
die "Temporary xrpld RPC not ready after 30s"
fi
sleep 1
done
# ---------------------------------------------------------------------------
# Generate key pairs
# ---------------------------------------------------------------------------
log "Generating $NUM_NODES validator key pairs..."
KEYS_JSON="["
VALIDATORS_TXT="[validators]"
for i in $(seq 1 "$NUM_NODES"); do
result=$(curl -sf "http://localhost:$TEMP_PORT" \
-d '{"method":"validation_create"}')
seed=$(echo "$result" | jq -r '.result.validation_seed')
pubkey=$(echo "$result" | jq -r '.result.validation_public_key')
# Both fields must be present. jq -r prints the literal string "null" for
# a missing field, so an unvalidated pubkey would be written verbatim into
# validators.txt and xrpld would reject the file at startup.
if [ -z "$seed" ] || [ "$seed" = "null" ]; then
die "Failed to generate key pair for node $i: no validation_seed in response"
fi
if [ -z "$pubkey" ] || [ "$pubkey" = "null" ]; then
die "Failed to generate key pair for node $i: no validation_public_key in response"
fi
log " Node $i: ${pubkey:0:20}..."
# Build JSON entry
entry="{\"index\": $i, \"seed\": \"$seed\", \"public_key\": \"$pubkey\"}"
if [ "$i" -gt 1 ]; then
KEYS_JSON="$KEYS_JSON,"
fi
KEYS_JSON="$KEYS_JSON$entry"
VALIDATORS_TXT="$VALIDATORS_TXT
$pubkey"
done
KEYS_JSON="$KEYS_JSON]"
# ---------------------------------------------------------------------------
# Write output files
# ---------------------------------------------------------------------------
echo "$KEYS_JSON" | jq '.' >"$OUTPUT_DIR/validator-keys.json"
echo "$VALIDATORS_TXT" >"$OUTPUT_DIR/validators.txt"
ok "Generated $NUM_NODES key pairs:"
ok " Keys: $OUTPUT_DIR/validator-keys.json"
ok " Validators: $OUTPUT_DIR/validators.txt"