mirror of
https://github.com/XRPLF/rippled.git
synced 2026-08-21 14:20:56 +00:00
fix(telemetry): address review findings in the workload validation harness
Fixes the review findings on this PR that belong to files it owns, plus several defects found while verifying those fixes. Findings in files owned by upstream branches are routed there and left untouched here. Correctness: - tx_submitter: advance the account sequence only on results that actually consume one (tes*, tec*, terQUEUED). tem*/tef*/tel* never reach the ledger, so advancing left a permanent gap that every later submit from that account inherited. Add a re-fetch hatch so a repeated non-consuming failure cannot livelock on the same sequence, and gate the account check on funded-ness rather than list length. - validate_telemetry: filter spans by name before collecting attributes, so a per-span attribute contract can no longer be satisfied by a sibling span; require exact name equality for non-wildcard children and glob matching for wildcards; bounds-check every returned series instead of only the first. - collect_system_metrics: select xrpld by argv[0] rather than a substring match on the whole command line, which averaged in unrelated processes and reported their RSS as xrpld's. Count genuine 0.0 CPU readings, use a clamped nearest-rank p99 index, and record RPC latency only on success. - benchmark: return each verdict through a named variable instead of a command substitution, so the pass/fail counters survive and the exit gate can fire. Scale before dividing in the percentage math, which truncated a 1.26% impact to 1.00% and cleared a 1% threshold. - compare_to_baseline: fall back to the absolute bound when the baseline is not positive, so a 0 -> 500 ms jump is no longer "within bounds". - rpc_load_generator: bound each connection to one in-flight recv(), drain in-flight requests before closing, use a nearest-rank percentile, and report delivery shortfall so an under-delivered run cannot pass with a 0% error rate. Fail loudly instead of silently: - run-full-validation: treat a consensus timeout and a missing validated ledger as fatal infrastructure errors, and fold the orchestrator and benchmark exit codes into the final status. A degraded cluster previously ran a full validation pass and reported misleading downstream failures. - collect_system_metrics: warn per empty measurement source, emit metrics_complete, and exit non-zero instead of substituting zeros that pass every threshold. Require GNU date with %N rather than falling back to a per-sample python3 fork that costs more than the threshold it is measured against. - benchmark: distinguish "could not measure" from "exceeded thresholds", install a cleanup trap so a failure cannot leak nodes and ports, and report an unusable baseline as inconclusive. - workload_orchestrator: bound subprocess communicate() and fail the exit gate on per-phase errors. Also pins the workload compose images to the versions the sibling stack already uses, hash-pins the Python dependencies, restricts the validator config template to loopback, corrects the dashboard and metric counts in the reference docs, drops a span from the regression gate that cannot fire under a WebSocket-only workload, and narrows the teardown pkill pattern so it no longer matches processes that merely mention the work directory. Verified with a full harness run against a local five-node cluster: 158 of 158 checks passed with no regressions detected.
This commit is contained in:
Pratik Mankawde
@@ -1,24 +1,32 @@
|
||||
# Docker Compose workload harness for Phase 10 telemetry validation.
|
||||
# Docker Compose workload harness for telemetry validation.
|
||||
#
|
||||
# Runs a 5-node validator cluster with full OTel telemetry stack:
|
||||
# - 5 rippled validator nodes (consensus network)
|
||||
# Runs the OTel telemetry backend only. There are no validator services here:
|
||||
# - OTel Collector (traces + native OTLP metrics)
|
||||
# - Tempo (trace backend + search API)
|
||||
# - Prometheus (metrics)
|
||||
# - Loki (log aggregation for log-trace correlation)
|
||||
# - Grafana (dashboards + trace/log exploration)
|
||||
#
|
||||
# The validator cluster runs as host processes, not containers.
|
||||
# run-full-validation.sh starts NUM_NODES (default 5) xrpld instances on
|
||||
# 127.0.0.1, each with a cfg it generates inline, peered to each other via
|
||||
# [ips_fixed]. They reach the collector through the published ports below and
|
||||
# write their logs into the bind-mounted workdir for the filelog receiver.
|
||||
#
|
||||
# Usage:
|
||||
# # Start the harness (requires pre-built xrpld image or mount binary):
|
||||
# # Start the telemetry backend on its own:
|
||||
# docker compose -f docker/telemetry/docker-compose.workload.yaml up -d
|
||||
#
|
||||
# # Or use the orchestrator:
|
||||
# # Or let the orchestrator start this stack and the node cluster together:
|
||||
# docker/telemetry/workload/run-full-validation.sh
|
||||
#
|
||||
# Prerequisites:
|
||||
# Prerequisites (for the orchestrator, not for this stack):
|
||||
# - xrpld binary built with -DXRPL_ENABLE_TELEMETRY=ON
|
||||
# - Validator keys generated via generate-validator-keys.sh
|
||||
# - Node configs generated by run-full-validation.sh
|
||||
#
|
||||
# Image tags are pinned to the same versions as docker-compose.yml, which
|
||||
# mounts these same collector, Tempo and Prometheus config files. Floating
|
||||
# tags would let an upstream release change the harness result.
|
||||
#
|
||||
# Note: No Docker healthchecks are defined here. The orchestrator script
|
||||
# (run-full-validation.sh) polls each service endpoint directly from the
|
||||
@@ -30,7 +38,7 @@ services:
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
otel-collector:
|
||||
image: otel/opentelemetry-collector-contrib:latest
|
||||
image: otel/opentelemetry-collector-contrib:0.158.0
|
||||
command: ["--config=/etc/otel-collector-config.yaml"]
|
||||
ports:
|
||||
- "4317:4317" # OTLP gRPC
|
||||
@@ -49,7 +57,7 @@ services:
|
||||
- workload-net
|
||||
|
||||
tempo:
|
||||
image: grafana/tempo:2.7.2
|
||||
image: grafana/tempo:2.9.4
|
||||
command: ["-config.file=/etc/tempo.yaml"]
|
||||
ports:
|
||||
- "3200:3200" # Tempo HTTP API
|
||||
@@ -60,7 +68,7 @@ services:
|
||||
- workload-net
|
||||
|
||||
prometheus:
|
||||
image: prom/prometheus:latest
|
||||
image: prom/prometheus:v3.13.2
|
||||
ports:
|
||||
- "9090:9090"
|
||||
volumes:
|
||||
@@ -71,7 +79,7 @@ services:
|
||||
- workload-net
|
||||
|
||||
loki:
|
||||
image: grafana/loki:3.4.2
|
||||
image: grafana/loki:3.7.6
|
||||
ports:
|
||||
- "3100:3100" # Loki HTTP API
|
||||
command: ["-config.file=/etc/loki/local-config.yaml"]
|
||||
@@ -79,7 +87,7 @@ services:
|
||||
- workload-net
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:latest
|
||||
image: grafana/grafana:13.1.2
|
||||
environment:
|
||||
- GF_AUTH_ANONYMOUS_ENABLED=true
|
||||
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
|
||||
|
||||
@@ -56,6 +56,7 @@ run-full-validation.sh (shell orchestrator)
|
||||
| -> validation-report.json
|
||||
|
|
||||
|-- benchmark.sh (baseline vs telemetry comparison)
|
||||
|-- collect_system_metrics.sh (per-leg CPU/RSS/latency/TPS sampling)
|
||||
-> benchmark-report-*.md
|
||||
```
|
||||
|
||||
@@ -132,7 +133,7 @@ Orchestrates the complete validation pipeline. Starts the telemetry stack, start
|
||||
# Stress test with benchmarks
|
||||
./run-full-validation.sh --xrpld /path/to/xrpld --profile stress --with-benchmark
|
||||
|
||||
# Skip Loki checks (if Phase 8 not deployed)
|
||||
# Skip Loki checks (if log export is not deployed)
|
||||
./run-full-validation.sh --xrpld /path/to/xrpld --skip-loki
|
||||
```
|
||||
|
||||
@@ -213,7 +214,7 @@ python3 tx_submitter.py --endpoint ws://localhost:6006 \
|
||||
Automated validation that all expected telemetry data exists. Every metric in `expected_metrics.json` is required — if it doesn't fire, the validation fails. Spans are required unless the entry carries `"optional": true`.
|
||||
|
||||
- **Span validation**: All span types from `expected_spans.json` with required attributes and parent-child hierarchies. Entries marked `"optional": true` only fire under traffic the harness may not produce (HTTP/JSON-RPC client, gRPC client, missing-ledger fetch, mode transitions); their absence is recorded as a passing skip, not a failure.
|
||||
- **Metric validation**: All metrics from `expected_metrics.json` — SpanMetrics, `beast::insight` gauges/counters/histograms, Phase 9 OTLP metrics. Every listed metric must have > 0 series. Uses the Prometheus `/api/v1/series` endpoint (not instant queries), polled until the metric appears or the poll window elapses, so a late-populating or quiet series is not a false negative.
|
||||
- **Metric validation**: All metrics from `expected_metrics.json` — SpanMetrics, `beast::insight` gauges/counters/histograms, `MetricsRegistry` OTLP metrics. Every listed metric must have > 0 series. Uses the Prometheus `/api/v1/series` endpoint (not instant queries), polled until the metric appears or the poll window elapses, so a late-populating or quiet series is not a false negative.
|
||||
- **Log-trace correlation**: trace_id/span_id in Loki logs (requires Loki)
|
||||
- **Dashboard validation**: Every dashboard uid listed under `grafana_dashboards.uids` in `expected_metrics.json` loads with panels. That list currently covers **all 15** dashboards provisioned in `docker/telemetry/grafana/dashboards/`. Note the scope of this check: it asks the Grafana API whether the dashboard exists and returns a panel count — it does **not** run the panels' queries, so a dashboard can pass here while individual panels render empty.
|
||||
|
||||
@@ -286,6 +287,65 @@ Thresholds (configurable via environment):
|
||||
| Throughput impact | < 5% | BENCH_TPS_IMPACT_PCT |
|
||||
| Consensus impact | < 1% | BENCH_CONSENSUS_IMPACT_PCT |
|
||||
|
||||
Each report row is `PASS`, `FAIL`, or `INCONCLUSIVE`. The throughput and
|
||||
consensus rows are ratios of the baseline, so they have nothing to report when
|
||||
the baseline run measured zero — that row becomes `INCONCLUSIVE` and **counts
|
||||
as a failure**, because an undefined result must never read as a pass.
|
||||
|
||||
Exit codes:
|
||||
|
||||
| Code | Meaning |
|
||||
| ---- | --------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 0 | Every metric was measured and is within its threshold |
|
||||
| 1 | Every metric was measured and at least one exceeded its threshold |
|
||||
| 2 | The overhead could not be measured — missing prerequisite, cluster never reached consensus, or incomplete metric collection |
|
||||
|
||||
`run-full-validation.sh` keeps the last two apart: 1 folds into its own
|
||||
"checks failed" exit, 2 into its "infrastructure error" exit. A run that
|
||||
measured nothing is therefore never reported as a performance regression.
|
||||
|
||||
### collect_system_metrics.sh
|
||||
|
||||
Samples CPU, peak RSS, RPC p99 latency, TPS and the mean inter-ledger interval
|
||||
from the running nodes, and writes them as JSON. `benchmark.sh` calls it once
|
||||
per leg; it is rarely run by hand.
|
||||
|
||||
```bash
|
||||
./collect_system_metrics.sh 5020,5021,5022 300 /tmp/metrics.json
|
||||
```
|
||||
|
||||
Processes are selected by matching `argv[0]`'s basename against `xrpld` or
|
||||
`rippled`. A wrapper that merely names the binary in its arguments, and
|
||||
unrelated tools whose command line happens to contain the string, are not
|
||||
sampled — including them diluted the CPU average and attributed a foreign
|
||||
process's RSS to the node. `ps -C xrpld` is not usable for this: xrpld renames
|
||||
itself, so its `comm` is `xrpld-main`.
|
||||
|
||||
Selection covers the whole host, so a second xrpld from another checkout is
|
||||
sampled as well. Benchmark on a machine running one cluster only.
|
||||
|
||||
The output carries a `metrics_complete` flag. It is `false` when any
|
||||
measurement source came back empty — no matching process, no successful RPC
|
||||
probe, or a ledger sequence that never advanced — and the affected metrics are
|
||||
then `0` placeholders. Since `0` clears every threshold, a `false` flag must be
|
||||
read as inconclusive, never as a pass.
|
||||
|
||||
Exit codes:
|
||||
|
||||
| Code | Meaning |
|
||||
| ---- | --------------------------------------------------------------------------------------------------------- |
|
||||
| 0 | Every metric was measured; `metrics_complete` is `true` |
|
||||
| 1 | Cannot run: bad arguments, no GNU `date` with `%N`, or a failed process sample. No output file is written |
|
||||
| 3 | The output file was written, but `metrics_complete` is `false` |
|
||||
|
||||
`benchmark.sh` treats either non-zero code — and an explicit
|
||||
`"metrics_complete": false` in an otherwise successful run — as fatal, and
|
||||
exits 2 rather than comparing an incomplete run.
|
||||
|
||||
A nanosecond clock is required. RPC latency is graded against a 2 ms
|
||||
threshold, and GNU `date +%s%N` is the only source cheap enough that the clock
|
||||
does not dominate what it measures, so the script refuses to start without it.
|
||||
|
||||
## Reading Validation Reports
|
||||
|
||||
The validation report (`validation-report.json`) is structured as follows. The
|
||||
|
||||
@@ -15,7 +15,10 @@ declared in [`../regression-metrics.json`](../regression-metrics.json) and write
|
||||
exits 0 without gating. This is how we bootstrap the baseline.
|
||||
- **Populated baseline**: the comparator diffs per-metric, enforces the thresholds
|
||||
(regression = current exceeds baseline on BOTH the percentage AND absolute bound),
|
||||
and exits non-zero on any regression.
|
||||
and exits non-zero on any regression. The single exception is a baseline that is
|
||||
not positive: the percentage change is undefined there, so the absolute bound
|
||||
decides alone. Without that fallback the AND gate would be unreachable and a
|
||||
0 ms → 500 ms jump would be reported as "within bounds".
|
||||
|
||||
The regression gate runs against whatever workload profile `run-full-validation.sh`
|
||||
was invoked with. Capture and comparison are profile-agnostic — they only read
|
||||
@@ -102,3 +105,24 @@ captured. Two independent blockers:
|
||||
Closing this needs **both** an `rpc_methods` group in `regression-metrics.json`
|
||||
and a `defaults.rpc_method` block in `regression-thresholds.json`. Adding only
|
||||
the first produces metrics that look gated in the report but are not.
|
||||
|
||||
## Known exclusion: `rpc.process` is not captured
|
||||
|
||||
`rpc.process` is deliberately absent from the `spans.names` list in
|
||||
`regression-metrics.json`, so no `span.rpc.process.*` key appears in this
|
||||
baseline. The span is created only in `ServerHandler::processRequest()`
|
||||
(`src/xrpld/rpc/detail/ServerHandler.cpp:705`), which is reached only from the
|
||||
HTTP/JSON-RPC session path. The harness load generator is WebSocket-only and
|
||||
that path never calls `processRequest`, so the span is never emitted under any
|
||||
workload profile here — `expected_spans.json` marks it `"optional": true` for
|
||||
the same reason.
|
||||
|
||||
While it was listed, the three quantiles were captured as `null` on every run
|
||||
and the comparator short-circuited them as `"new metric (not in baseline)"` —
|
||||
so a 9999 ms value would still have reported `regressed: false`. Three keys
|
||||
that can never gate are worse than no keys: they inflate `summary.total` and
|
||||
read as covered.
|
||||
|
||||
If per-request HTTP timings are wanted, the fix is to give the harness an
|
||||
HTTP/JSON-RPC load path first, then re-add `rpc.process` and bootstrap a real
|
||||
baseline for it.
|
||||
|
||||
@@ -78,18 +78,6 @@
|
||||
"unit": "ms",
|
||||
"value": 6.699999999999978
|
||||
},
|
||||
"span.rpc.process.p50": {
|
||||
"unit": "ms",
|
||||
"value": null
|
||||
},
|
||||
"span.rpc.process.p95": {
|
||||
"unit": "ms",
|
||||
"value": null
|
||||
},
|
||||
"span.rpc.process.p99": {
|
||||
"unit": "ms",
|
||||
"value": null
|
||||
},
|
||||
"span.rpc.ws_message.p50": {
|
||||
"unit": "ms",
|
||||
"value": 0.5026522773001647
|
||||
|
||||
@@ -17,6 +17,20 @@
|
||||
# BENCH_RPC_LATENCY_IMPACT_MS=2 RPC p99 latency impact < 2ms
|
||||
# BENCH_TPS_IMPACT_PCT=5 Throughput impact < 5%
|
||||
# BENCH_CONSENSUS_IMPACT_PCT=1 Consensus round time impact < 1%
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 Every overhead metric was measured and is within its threshold.
|
||||
# 1 Every overhead metric was measured and at least one exceeded its
|
||||
# threshold. This is the only "telemetry is too expensive" signal.
|
||||
# Also returned for a command-line usage error, which the pipeline
|
||||
# cannot trigger (run-full-validation.sh passes a fixed flag list).
|
||||
# 2 The overhead could not be measured at all — a missing prerequisite, a
|
||||
# cluster that never reached consensus, or an incomplete metric
|
||||
# collection. Nothing was compared, so nothing was breached.
|
||||
#
|
||||
# run-full-validation.sh depends on that split: it folds 1 into its own
|
||||
# "checks failed" exit and 2 into its "infrastructure error" exit. Reporting a
|
||||
# run that measured nothing as a threshold breach would be a false regression.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
@@ -27,11 +41,21 @@ log() { printf "\033[1;34m[BENCH]\033[0m %s\n" "$*"; }
|
||||
ok() { printf "\033[1;32m[BENCH]\033[0m %s\n" "$*"; }
|
||||
warn() { printf "\033[1;33m[BENCH]\033[0m %s\n" "$*"; }
|
||||
fail() { printf "\033[1;31m[BENCH]\033[0m %s\n" "$*"; }
|
||||
|
||||
# Usage error. Exit 1 by shell convention; see the exit-code block above.
|
||||
die() {
|
||||
printf "\033[1;31m[BENCH]\033[0m %s\n" "$*" >&2
|
||||
fail "$*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Fatal, and no overhead figure was produced. Exit 2 keeps exit 1 exclusively
|
||||
# for a measured threshold breach, so the caller never grades a run that
|
||||
# measured nothing as a performance regression.
|
||||
cannot_measure() {
|
||||
fail "$*" >&2
|
||||
exit 2
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Defaults and thresholds
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -91,11 +115,12 @@ while [ $# -gt 0 ]; do
|
||||
esac
|
||||
done
|
||||
|
||||
# Validate prerequisites.
|
||||
[ -x "$XRPLD" ] || die "xrpld not found at $XRPLD"
|
||||
command -v jq >/dev/null 2>&1 || die "jq not found"
|
||||
command -v bc >/dev/null 2>&1 || die "bc not found"
|
||||
command -v curl >/dev/null 2>&1 || die "curl not found"
|
||||
# Validate prerequisites. A missing binary or tool means no measurement can be
|
||||
# taken, which is "cannot measure", not "too slow".
|
||||
[ -x "$XRPLD" ] || cannot_measure "xrpld not found at $XRPLD"
|
||||
command -v jq >/dev/null 2>&1 || cannot_measure "jq not found"
|
||||
command -v bc >/dev/null 2>&1 || cannot_measure "bc not found"
|
||||
command -v curl >/dev/null 2>&1 || cannot_measure "curl not found"
|
||||
|
||||
mkdir -p "$RESULTS_DIR"
|
||||
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||
@@ -103,6 +128,11 @@ TIMESTAMP=$(date +%Y%m%d_%H%M%S)
|
||||
# ---------------------------------------------------------------------------
|
||||
# Node cluster management
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# True while xrpld children spawned by start_cluster may still be alive.
|
||||
# Read by stop_cluster so it can be called any number of times.
|
||||
CLUSTER_RUNNING=false
|
||||
|
||||
start_cluster() {
|
||||
local telemetry_enabled="$1"
|
||||
local label="$2"
|
||||
@@ -115,6 +145,10 @@ start_cluster() {
|
||||
# Generate keys using first node.
|
||||
bash "$SCRIPT_DIR/generate-validator-keys.sh" "$XRPLD" "$NUM_NODES" "$WORKDIR"
|
||||
|
||||
# Set before the spawn loop so a failure part-way through it still gets
|
||||
# cleaned up by the EXIT trap.
|
||||
CLUSTER_RUNNING=true
|
||||
|
||||
# Build per-node configs.
|
||||
for i in $(seq 1 "$NUM_NODES"); do
|
||||
local node_dir="$WORKDIR/node$i"
|
||||
@@ -214,9 +248,12 @@ EOCFG
|
||||
echo $! >"$node_dir/xrpld.pid"
|
||||
done
|
||||
|
||||
# Wait for consensus.
|
||||
log "Waiting for consensus..."
|
||||
for attempt in $(seq 1 120); do
|
||||
# Wait for consensus. Reaching the limit is fatal: numbers taken from a
|
||||
# cluster that never got to "proposing" would silently corrupt the
|
||||
# baseline-vs-telemetry comparison.
|
||||
local max_wait=120
|
||||
log "Waiting for consensus (up to ${max_wait}s)..."
|
||||
for attempt in $(seq 1 "$max_wait"); do
|
||||
local ready=0
|
||||
for i in $(seq 1 "$NUM_NODES"); do
|
||||
local port
|
||||
@@ -233,8 +270,8 @@ EOCFG
|
||||
ok "All $NUM_NODES nodes proposing (attempt $attempt)"
|
||||
break
|
||||
fi
|
||||
if [ "$attempt" -eq 120 ]; then
|
||||
warn "Consensus timeout — $ready/$NUM_NODES nodes ready"
|
||||
if [ "$attempt" -eq "$max_wait" ]; then
|
||||
cannot_measure "Consensus timeout — only $ready/$NUM_NODES nodes proposing after ${max_wait}s"
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
@@ -244,6 +281,11 @@ EOCFG
|
||||
}
|
||||
|
||||
stop_cluster() {
|
||||
# Idempotent. The happy path calls this directly and the EXIT trap calls
|
||||
# it again, so a second call must not re-kill or log a misleading message.
|
||||
[ "$CLUSTER_RUNNING" = true ] || return 0
|
||||
CLUSTER_RUNNING=false
|
||||
|
||||
log "Stopping cluster..."
|
||||
for i in $(seq 1 "$NUM_NODES"); do
|
||||
local pidfile="$WORKDIR/node$i/xrpld.pid"
|
||||
@@ -251,10 +293,30 @@ stop_cluster() {
|
||||
kill "$(cat "$pidfile")" 2>/dev/null || true
|
||||
fi
|
||||
done
|
||||
pkill -f "$WORKDIR" 2>/dev/null || true
|
||||
sleep 3
|
||||
# Belt and braces for a node whose pidfile is missing or stale. Matched on
|
||||
# the per-node config path — the shape start_cluster launches nodes with
|
||||
# (`--conf $WORKDIR/nodeN/xrpld.cfg`) — rather than on the workdir alone.
|
||||
# The loose form killed anything whose command line merely mentioned the
|
||||
# workdir, including a developer's `tail -f $WORKDIR/node1/debug.log`, and
|
||||
# the EXIT trap now makes this run on every exit path.
|
||||
pkill -f "$WORKDIR/node[0-9]+/xrpld\.cfg" 2>/dev/null || true
|
||||
|
||||
# Guarded on purpose. This runs as the EXIT trap, where any unguarded
|
||||
# failure makes `set -e` exit with that command's status and discard the
|
||||
# status the script meant to report — a threshold breach would surface as
|
||||
# a plain 1 and "cannot measure" would lose its 2. With every command here
|
||||
# guarded, the explicit `return 0` below is reachable and authoritative.
|
||||
sleep 3 || true
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
# Reap the cluster on every exit path. Installed here rather than straight
|
||||
# after argument parsing so the handler name always resolves. Without it, any
|
||||
# failure between start_cluster and stop_cluster leaks the xrpld children
|
||||
# along with their RPC ports (5020+) and peer ports (51250+).
|
||||
trap stop_cluster EXIT
|
||||
|
||||
# Build RPC ports CSV string.
|
||||
rpc_ports_csv() {
|
||||
local ports=""
|
||||
@@ -265,6 +327,30 @@ rpc_ports_csv() {
|
||||
echo "$ports"
|
||||
}
|
||||
|
||||
# Collects one leg of the benchmark.
|
||||
#
|
||||
# The collector exits non-zero when it cannot run (1) or when a measurement
|
||||
# source came back empty (3). An all-zero or partial sample set clears every
|
||||
# threshold, so an incomplete leg aborts with "cannot measure" instead of being
|
||||
# compared and passed.
|
||||
collect_metrics() {
|
||||
local label="$1"
|
||||
local out_file="$2"
|
||||
|
||||
local status=0
|
||||
bash "$SCRIPT_DIR/collect_system_metrics.sh" \
|
||||
"$(rpc_ports_csv)" "$DURATION" "$out_file" || status=$?
|
||||
[ "$status" -eq 0 ] ||
|
||||
cannot_measure "$label metric collection failed (exit $status) — refusing to compare an incomplete run"
|
||||
|
||||
# Only an explicit false counts. The flag is absent from older artifacts,
|
||||
# and jq's "//" operator would turn a real false into the default.
|
||||
local complete
|
||||
complete=$(jq -r '.metrics_complete' "$out_file" 2>/dev/null || echo "null")
|
||||
[ "$complete" != "false" ] ||
|
||||
cannot_measure "$label metrics are flagged incomplete — refusing to compare an incomplete run"
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Run benchmark
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -276,13 +362,13 @@ log "="
|
||||
# --- Baseline run ---
|
||||
BASELINE_FILE="$RESULTS_DIR/baseline-${TIMESTAMP}.json"
|
||||
start_cluster "0" "baseline"
|
||||
bash "$SCRIPT_DIR/collect_system_metrics.sh" "$(rpc_ports_csv)" "$DURATION" "$BASELINE_FILE"
|
||||
collect_metrics "baseline" "$BASELINE_FILE"
|
||||
stop_cluster
|
||||
|
||||
# --- Telemetry run ---
|
||||
TELEMETRY_FILE="$RESULTS_DIR/telemetry-${TIMESTAMP}.json"
|
||||
start_cluster "1" "telemetry"
|
||||
bash "$SCRIPT_DIR/collect_system_metrics.sh" "$(rpc_ports_csv)" "$DURATION" "$TELEMETRY_FILE"
|
||||
collect_metrics "telemetry" "$TELEMETRY_FILE"
|
||||
stop_cluster
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -290,6 +376,10 @@ stop_cluster
|
||||
# ---------------------------------------------------------------------------
|
||||
log "Comparing results..."
|
||||
|
||||
# Written into an impact variable when its baseline could not be used.
|
||||
# check_threshold turns this into an INCONCLUSIVE verdict.
|
||||
INCONCLUSIVE="n/a"
|
||||
|
||||
read_metric() {
|
||||
local file="$1"
|
||||
local key="$2"
|
||||
@@ -308,20 +398,30 @@ BASE_RPC=$(read_metric "$BASELINE_FILE" "rpc_p99_ms")
|
||||
TELE_RPC=$(read_metric "$TELEMETRY_FILE" "rpc_p99_ms")
|
||||
RPC_DELTA=$(echo "scale=2; $TELE_RPC - $BASE_RPC" | bc 2>/dev/null || echo "0")
|
||||
|
||||
# Both impacts below are ratios of the baseline, so a non-positive baseline
|
||||
# leaves them undefined. The collector writes tps=0 whenever no ledger
|
||||
# advanced and read_metric defaults a missing key to 0, so this is a routine
|
||||
# outcome rather than an edge case. Reporting it as "0% impact" would clear
|
||||
# the threshold and hide a failed baseline run.
|
||||
#
|
||||
# Both expressions scale by 100 before dividing. bc truncates at "scale" after
|
||||
# every operation, so dividing first would floor the ratio to 2 decimals and
|
||||
# then multiply the lost precision by 100 — a real 1.25% consensus impact came
|
||||
# out as exactly 1.00 and passed the 1% threshold.
|
||||
BASE_TPS=$(read_metric "$BASELINE_FILE" "tps")
|
||||
TELE_TPS=$(read_metric "$TELEMETRY_FILE" "tps")
|
||||
if [[ "$(echo "$BASE_TPS > 0" | bc 2>/dev/null)" = "1" ]]; then
|
||||
TPS_IMPACT=$(echo "scale=2; ($BASE_TPS - $TELE_TPS) / $BASE_TPS * 100" | bc 2>/dev/null || echo "0")
|
||||
TPS_IMPACT=$(echo "scale=2; ($BASE_TPS - $TELE_TPS) * 100 / $BASE_TPS" | bc 2>/dev/null || echo "0")
|
||||
else
|
||||
TPS_IMPACT="0"
|
||||
TPS_IMPACT="$INCONCLUSIVE"
|
||||
fi
|
||||
|
||||
BASE_CONS=$(read_metric "$BASELINE_FILE" "consensus_round_p95_ms")
|
||||
TELE_CONS=$(read_metric "$TELEMETRY_FILE" "consensus_round_p95_ms")
|
||||
BASE_CONS=$(read_metric "$BASELINE_FILE" "consensus_round_mean_ms")
|
||||
TELE_CONS=$(read_metric "$TELEMETRY_FILE" "consensus_round_mean_ms")
|
||||
if [[ "$(echo "$BASE_CONS > 0" | bc 2>/dev/null)" = "1" ]]; then
|
||||
CONS_IMPACT=$(echo "scale=2; ($TELE_CONS - $BASE_CONS) / $BASE_CONS * 100" | bc 2>/dev/null || echo "0")
|
||||
CONS_IMPACT=$(echo "scale=2; ($TELE_CONS - $BASE_CONS) * 100 / $BASE_CONS" | bc 2>/dev/null || echo "0")
|
||||
else
|
||||
CONS_IMPACT="0"
|
||||
CONS_IMPACT="$INCONCLUSIVE"
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -329,30 +429,63 @@ fi
|
||||
# ---------------------------------------------------------------------------
|
||||
PASS_COUNT=0
|
||||
FAIL_COUNT=0
|
||||
INCONCLUSIVE_COUNT=0
|
||||
|
||||
# Records the verdict for one row of the report.
|
||||
#
|
||||
# Arguments: metric name, measured value, threshold, unit, and the name of the
|
||||
# variable to write the bare verdict into.
|
||||
#
|
||||
# The verdict travels through that named variable and every diagnostic goes to
|
||||
# stderr. Calling this through a command substitution would run it in a
|
||||
# subshell, which drops the counter updates and captures the colored log line
|
||||
# into the caller's variable.
|
||||
check_threshold() {
|
||||
local name="$1"
|
||||
local actual="$2"
|
||||
local threshold="$3"
|
||||
local unit="$4"
|
||||
local result_var="$5"
|
||||
|
||||
# Unusable measurement. Counted as a failure so the exit gate fires: an
|
||||
# undefined result must never read as a pass.
|
||||
if [ "$actual" = "$INCONCLUSIVE" ]; then
|
||||
fail "$name: INCONCLUSIVE — baseline was zero or missing" >&2
|
||||
FAIL_COUNT=$((FAIL_COUNT + 1))
|
||||
INCONCLUSIVE_COUNT=$((INCONCLUSIVE_COUNT + 1))
|
||||
printf -v "$result_var" 'INCONCLUSIVE'
|
||||
return
|
||||
fi
|
||||
|
||||
# Compare: actual <= threshold
|
||||
if [[ "$(echo "$actual <= $threshold" | bc 2>/dev/null)" = "1" ]]; then
|
||||
ok "$name: ${actual}${unit} <= ${threshold}${unit} PASS"
|
||||
ok "$name: ${actual}${unit} <= ${threshold}${unit} PASS" >&2
|
||||
PASS_COUNT=$((PASS_COUNT + 1))
|
||||
echo "PASS"
|
||||
printf -v "$result_var" 'PASS'
|
||||
else
|
||||
fail "$name: ${actual}${unit} > ${threshold}${unit} FAIL"
|
||||
fail "$name: ${actual}${unit} > ${threshold}${unit} FAIL" >&2
|
||||
FAIL_COUNT=$((FAIL_COUNT + 1))
|
||||
echo "FAIL"
|
||||
printf -v "$result_var" 'FAIL'
|
||||
fi
|
||||
}
|
||||
|
||||
CPU_RESULT=$(check_threshold "CPU overhead" "$CPU_DELTA" "$CPU_THRESHOLD" "%")
|
||||
MEM_RESULT=$(check_threshold "Memory overhead" "$MEM_DELTA" "$MEM_THRESHOLD" "MB")
|
||||
RPC_RESULT=$(check_threshold "RPC p99 impact" "$RPC_DELTA" "$RPC_THRESHOLD" "ms")
|
||||
TPS_RESULT=$(check_threshold "TPS impact" "$TPS_IMPACT" "$TPS_THRESHOLD" "%")
|
||||
CONS_RESULT=$(check_threshold "Consensus impact" "$CONS_IMPACT" "$CONSENSUS_THRESHOLD" "%")
|
||||
# Formats a delta for the report table: appends the unit to a real number and
|
||||
# leaves the INCONCLUSIVE placeholder bare.
|
||||
fmt_delta() {
|
||||
local value="$1"
|
||||
local unit="$2"
|
||||
if [ "$value" = "$INCONCLUSIVE" ]; then
|
||||
printf '%s' "$value"
|
||||
else
|
||||
printf '%s%s' "$value" "$unit"
|
||||
fi
|
||||
}
|
||||
|
||||
check_threshold "CPU overhead" "$CPU_DELTA" "$CPU_THRESHOLD" "%" CPU_RESULT
|
||||
check_threshold "Memory overhead" "$MEM_DELTA" "$MEM_THRESHOLD" "MB" MEM_RESULT
|
||||
check_threshold "RPC p99 impact" "$RPC_DELTA" "$RPC_THRESHOLD" "ms" RPC_RESULT
|
||||
check_threshold "TPS impact" "$TPS_IMPACT" "$TPS_THRESHOLD" "%" TPS_RESULT
|
||||
check_threshold "Consensus impact" "$CONS_IMPACT" "$CONSENSUS_THRESHOLD" "%" CONS_RESULT
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Output Markdown table
|
||||
@@ -373,13 +506,20 @@ cat >"$REPORT_FILE" <<EOMD
|
||||
| CPU (avg %) | ${BASE_CPU}% | ${TELE_CPU}% | ${CPU_DELTA}% | < ${CPU_THRESHOLD}% | ${CPU_RESULT} |
|
||||
| Memory RSS (peak MB) | ${BASE_MEM} MB | ${TELE_MEM} MB | ${MEM_DELTA} MB | < ${MEM_THRESHOLD} MB | ${MEM_RESULT} |
|
||||
| RPC p99 Latency (ms) | ${BASE_RPC} ms | ${TELE_RPC} ms | ${RPC_DELTA} ms | < ${RPC_THRESHOLD} ms | ${RPC_RESULT} |
|
||||
| Throughput (TPS) | ${BASE_TPS} | ${TELE_TPS} | ${TPS_IMPACT}% | < ${TPS_THRESHOLD}% | ${TPS_RESULT} |
|
||||
| Consensus Round p95 (ms) | ${BASE_CONS} ms | ${TELE_CONS} ms | ${CONS_IMPACT}% | < ${CONSENSUS_THRESHOLD}% | ${CONS_RESULT} |
|
||||
| Throughput (TPS) | ${BASE_TPS} | ${TELE_TPS} | $(fmt_delta "$TPS_IMPACT" "%") | < ${TPS_THRESHOLD}% | ${TPS_RESULT} |
|
||||
| Consensus Round Mean (ms) | ${BASE_CONS} ms | ${TELE_CONS} ms | $(fmt_delta "$CONS_IMPACT" "%") | < ${CONSENSUS_THRESHOLD}% | ${CONS_RESULT} |
|
||||
|
||||
\`INCONCLUSIVE\` means the baseline for that row was zero or missing, so the
|
||||
impact could not be computed. Such rows count as failures.
|
||||
|
||||
\`Consensus Round Mean\` is the mean inter-ledger interval derived from the
|
||||
collector's 5 s ledger-sequence samples, not a percentile.
|
||||
|
||||
## Summary
|
||||
|
||||
- **Passed**: $PASS_COUNT / $((PASS_COUNT + FAIL_COUNT))
|
||||
- **Failed**: $FAIL_COUNT / $((PASS_COUNT + FAIL_COUNT))
|
||||
- **Inconclusive**: $INCONCLUSIVE_COUNT (included in Failed)
|
||||
|
||||
## Raw Data
|
||||
|
||||
|
||||
@@ -17,9 +17,18 @@
|
||||
# "memory_rss_mb_peak": 450.2,
|
||||
# "rpc_p99_ms": 15.3,
|
||||
# "tps": 4.8,
|
||||
# "consensus_round_p95_ms": 3200,
|
||||
# "consensus_round_mean_ms": 3200,
|
||||
# "metrics_complete": true,
|
||||
# "samples": 60
|
||||
# }
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 Every metric was measured; "metrics_complete" is true.
|
||||
# 1 Cannot run at all: bad arguments, no GNU date with %N, or a failed
|
||||
# process sample. No output file is written.
|
||||
# 3 Output file was written, but at least one measurement source was empty,
|
||||
# so the affected metrics are 0 placeholders and "metrics_complete" is
|
||||
# false. Callers must treat this as inconclusive, never as a pass.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
@@ -28,6 +37,8 @@ set -euo pipefail
|
||||
# ---------------------------------------------------------------------------
|
||||
log() { printf "\033[1;34m[METRICS]\033[0m %s\n" "$*"; }
|
||||
ok() { printf "\033[1;32m[METRICS]\033[0m %s\n" "$*"; }
|
||||
# Warnings go to stderr so they never mix into the JSON echoed on stdout.
|
||||
warn() { printf "\033[1;33m[METRICS]\033[0m %s\n" "$*" >&2; }
|
||||
die() {
|
||||
printf "\033[1;31m[METRICS]\033[0m %s\n" "$*" >&2
|
||||
exit 1
|
||||
@@ -56,10 +67,53 @@ OUTPUT_FILE="$3"
|
||||
|
||||
IFS=',' read -ra RPC_PORTS <<<"$RPC_PORTS_CSV"
|
||||
SAMPLE_INTERVAL=5
|
||||
SAMPLES=$((DURATION / SAMPLE_INTERVAL))
|
||||
|
||||
# Reject anything the sample arithmetic cannot use, instead of silently
|
||||
# treating it as 0.
|
||||
case "$DURATION" in
|
||||
'' | *[!0-9]*) die "duration_seconds must be a positive integer, got '$DURATION'" ;;
|
||||
esac
|
||||
|
||||
# Normalise to base 10 once, right after the digits check. Bash arithmetic
|
||||
# reads a leading zero as octal, so "08" aborted with "value too great for
|
||||
# base" and "0100" was silently taken as 64. Doing it here also keeps the
|
||||
# value a valid JSON number in the output below, where "08" is not.
|
||||
DURATION=$((10#$DURATION))
|
||||
|
||||
# Round up, so a duration shorter than one interval still takes one sample
|
||||
# rather than truncating to zero and emitting an all-zero JSON.
|
||||
SAMPLES=$(((DURATION + SAMPLE_INTERVAL - 1) / SAMPLE_INTERVAL))
|
||||
if [ "$SAMPLES" -lt 1 ]; then
|
||||
die "duration_seconds=$DURATION yields $SAMPLES samples; need at least 1"
|
||||
fi
|
||||
|
||||
log "Collecting metrics for ${DURATION}s (${SAMPLES} samples, ${#RPC_PORTS[@]} nodes)..."
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Nanosecond clock
|
||||
#
|
||||
# GNU date supports "+%s%N". BSD/macOS date has no %N and echoes it back
|
||||
# literally, which used to abort the sampling loop under `set -e` and still
|
||||
# exit 0 with an all-zero JSON — a silent false pass.
|
||||
#
|
||||
# The clock has to be cheap as well as precise, because the latency it
|
||||
# measures is compared against a 2 ms threshold. Measured on a dev box: `date
|
||||
# +%s%N` costs ~1.2 ms per call, forking python3 for the same value ~13 ms.
|
||||
# Two calls bracket every request, so a python3 fallback would add ~26 ms of
|
||||
# its own overhead to a 2 ms budget and make the number meaningless. There is
|
||||
# no cheap alternative worth having, so probe once and refuse to run without
|
||||
# GNU date rather than report a figure that is quietly an order of magnitude
|
||||
# wrong.
|
||||
# ---------------------------------------------------------------------------
|
||||
if [[ ! "$(date +%s%N 2>/dev/null)" =~ ^[0-9]+$ ]]; then
|
||||
die "GNU coreutils date with %N is required for RPC latency timing; this date does not support it"
|
||||
fi
|
||||
|
||||
# Echo the current time in nanoseconds since the epoch.
|
||||
now_ns() {
|
||||
date +%s%N
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Temporary files for aggregation
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -95,40 +149,48 @@ log "Initial validated ledger seq: $INITIAL_SEQ"
|
||||
# Sampling loop
|
||||
# ---------------------------------------------------------------------------
|
||||
for sample in $(seq 1 "$SAMPLES"); do
|
||||
# Collect CPU usage for xrpld processes.
|
||||
# Uses ps to find all xrpld processes and average their CPU%.
|
||||
cpu_sum=0
|
||||
cpu_count=0
|
||||
while IFS= read -r line; do
|
||||
cpu_val=$(echo "$line" | awk '{print $1}')
|
||||
if [ -n "$cpu_val" ] && [ "$cpu_val" != "0.0" ]; then
|
||||
cpu_sum=$(echo "$cpu_sum + $cpu_val" | bc 2>/dev/null || echo "$cpu_sum")
|
||||
cpu_count=$((cpu_count + 1))
|
||||
fi
|
||||
done < <(ps aux 2>/dev/null | grep '[x]rpld' | awk '{print $3}')
|
||||
# Sample CPU% and RSS for the xrpld processes. One ps pass feeds both
|
||||
# files, so the two numbers always come from the same instant.
|
||||
#
|
||||
# Selection is on argv[0]'s basename, NOT on "the command line mentions
|
||||
# xrpld". The loose form matched every bystander whose command line
|
||||
# happened to contain the string: this harness's own launcher (invoked as
|
||||
# `--xrpld .build/xrpld`, whose argv[0] is bash), an editor's clangd, and
|
||||
# any shell sitting in a directory with xrpld in its path. Measured
|
||||
# against a live 5-node cluster, that pulled in 8-15 processes instead of
|
||||
# 5, halved the CPU average with idle bystanders, and reported clangd's
|
||||
# 1.1 GB RSS as xrpld's peak against a 5 MB threshold. `ps -C xrpld` is
|
||||
# not an alternative: xrpld renames itself, so its comm is "xrpld-main"
|
||||
# and -C matches nothing. "rippled" is accepted alongside "xrpld" so a
|
||||
# rename of the binary cannot silently zero the collector.
|
||||
#
|
||||
# Scope is the whole host, as it always was: a second xrpld from another
|
||||
# checkout is sampled too. Only run a benchmark on a box with one cluster.
|
||||
#
|
||||
# A %cpu of exactly 0.0 is a real reading and is counted — dropping idle
|
||||
# samples would inflate the average — while non-numeric output is
|
||||
# rejected by the pattern. An RSS of 0 is not a live process, so it
|
||||
# contributes no memory sample; counting it would leave the file non-empty
|
||||
# and mark a dead cluster's 0 MB peak as a complete measurement.
|
||||
ps -eo %cpu=,rss=,args= |
|
||||
awk -v cpu_file="$CPU_FILE" -v mem_file="$MEM_FILE" '
|
||||
$3 !~ /(^|\/)(xrpld|rippled)$/ { next }
|
||||
$1 ~ /^[0-9]+(\.[0-9]+)?$/ { cpu_sum += $1; cpu_n++ }
|
||||
$2 ~ /^[0-9]+$/ && $2 + 0 > 0 { printf("%.2f\n", $2 / 1024) >> mem_file }
|
||||
END { if (cpu_n > 0) printf("%.2f\n", cpu_sum / cpu_n) >> cpu_file }
|
||||
' || die "process sampling failed on sample $sample/$SAMPLES"
|
||||
|
||||
if [ "$cpu_count" -gt 0 ]; then
|
||||
cpu_avg=$(echo "scale=2; $cpu_sum / $cpu_count" | bc 2>/dev/null || echo "0")
|
||||
echo "$cpu_avg" >>"$CPU_FILE"
|
||||
fi
|
||||
|
||||
# Collect memory RSS for xrpld processes.
|
||||
while IFS= read -r line; do
|
||||
rss_kb=$(echo "$line" | awk '{print $1}')
|
||||
if [ -n "$rss_kb" ] && [ "$rss_kb" != "0" ]; then
|
||||
rss_mb=$(echo "scale=2; $rss_kb / 1024" | bc 2>/dev/null || echo "0")
|
||||
echo "$rss_mb" >>"$MEM_FILE"
|
||||
fi
|
||||
done < <(ps aux 2>/dev/null | grep '[x]rpld' | awk '{print $6}')
|
||||
|
||||
# Collect RPC latency from each node.
|
||||
# Collect RPC latency from each node. Only a successful call is a latency
|
||||
# measurement: a refused connection returns in well under a millisecond,
|
||||
# and recording that as ~0 ms would pull the reported p99 down.
|
||||
for port in "${RPC_PORTS[@]}"; do
|
||||
start_ms=$(date +%s%N)
|
||||
curl -sf "http://localhost:$port" \
|
||||
-d '{"method":"server_info"}' >/dev/null 2>&1 || true
|
||||
end_ms=$(date +%s%N)
|
||||
latency_ms=$(((end_ms - start_ms) / 1000000))
|
||||
echo "$latency_ms" >>"$RPC_FILE"
|
||||
start_ns=$(now_ns)
|
||||
if curl -sf "http://localhost:$port" \
|
||||
-d '{"method":"server_info"}' >/dev/null 2>&1; then
|
||||
end_ns=$(now_ns)
|
||||
latency_ms=$(((end_ns - start_ns) / 1000000))
|
||||
echo "$latency_ms" >>"$RPC_FILE"
|
||||
fi
|
||||
done
|
||||
|
||||
# Record current validated ledger seq.
|
||||
@@ -153,28 +215,55 @@ done
|
||||
# ---------------------------------------------------------------------------
|
||||
log "Computing aggregated metrics..."
|
||||
|
||||
# Cleared by any empty measurement source. A 0 metric is otherwise
|
||||
# indistinguishable from a real reading, so the flag is exported in the JSON
|
||||
# and drives the exit-3 contract documented in the header.
|
||||
METRICS_COMPLETE=true
|
||||
|
||||
# CPU average.
|
||||
if [ -s "$CPU_FILE" ]; then
|
||||
CPU_AVG=$(awk '{ sum += $1; n++ } END { if (n>0) printf "%.2f", sum/n; else print "0" }' "$CPU_FILE")
|
||||
else
|
||||
# Now that the selector cannot match this harness's own processes, an
|
||||
# empty file means no xrpld process was running for any sample.
|
||||
warn "No CPU samples collected (no xrpld process matched); cpu_pct_avg is a 0 placeholder"
|
||||
CPU_AVG="0"
|
||||
METRICS_COMPLETE=false
|
||||
fi
|
||||
|
||||
# Memory peak RSS (MB).
|
||||
if [ -s "$MEM_FILE" ]; then
|
||||
MEM_PEAK=$(sort -n "$MEM_FILE" | tail -1)
|
||||
else
|
||||
warn "No memory samples collected (no xrpld process matched); memory_rss_mb_peak is a 0 placeholder"
|
||||
MEM_PEAK="0"
|
||||
METRICS_COMPLETE=false
|
||||
fi
|
||||
|
||||
# RPC latency p99 (ms).
|
||||
if [ -s "$RPC_FILE" ]; then
|
||||
RPC_COUNT=$(wc -l <"$RPC_FILE")
|
||||
P99_INDEX=$(echo "scale=0; $RPC_COUNT * 99 / 100" | bc)
|
||||
# Nearest-rank p99: ceil(count * 99 / 100), clamped into [1, count].
|
||||
# Integer arithmetic avoids both the floor bias of the old bc expression
|
||||
# and the bc dependency, and the lower clamp keeps sed off line address 0
|
||||
# (a file with no trailing newline makes wc -l report 0).
|
||||
P99_INDEX=$(((RPC_COUNT * 99 + 99) / 100))
|
||||
if [ "$P99_INDEX" -lt 1 ]; then
|
||||
P99_INDEX=1
|
||||
fi
|
||||
if [ "$P99_INDEX" -gt "$RPC_COUNT" ]; then
|
||||
P99_INDEX="$RPC_COUNT"
|
||||
fi
|
||||
RPC_P99=$(sort -n "$RPC_FILE" | sed -n "${P99_INDEX}p")
|
||||
[ -z "$RPC_P99" ] && RPC_P99="0"
|
||||
if [ -z "$RPC_P99" ]; then
|
||||
warn "RPC latency file has no line $P99_INDEX; rpc_p99_ms is a 0 placeholder"
|
||||
RPC_P99="0"
|
||||
METRICS_COMPLETE=false
|
||||
fi
|
||||
else
|
||||
warn "No successful RPC probes; rpc_p99_ms is a 0 placeholder"
|
||||
RPC_P99="0"
|
||||
METRICS_COMPLETE=false
|
||||
fi
|
||||
|
||||
# TPS calculation from ledger sequence advancement.
|
||||
@@ -198,20 +287,26 @@ else
|
||||
TPS="0"
|
||||
fi
|
||||
|
||||
# Consensus round time p95 (from ledger close interval).
|
||||
# Approximate by looking at ledger sequence progression intervals.
|
||||
# Mean inter-ledger interval in ms: DURATION / (distinct ledgers - 1) * 1000.
|
||||
#
|
||||
# This is a MEAN, not a percentile — the JSON key says so. It is also aliased
|
||||
# by the sample loop: LEDGER_FILE gets one sequence per sample, so at a
|
||||
# SAMPLE_INTERVAL of 5 s the series cannot resolve a close interval faster
|
||||
# than that (a ~4 s close is invisible). Read it as a coarse trend only.
|
||||
if [ -s "$LEDGER_FILE" ]; then
|
||||
# Calculate intervals between consecutive ledger sequences.
|
||||
LEDGER_COUNT=$(wc -l <"$LEDGER_FILE")
|
||||
# Rough estimate: DURATION / number_of_distinct_ledgers * 1000 ms
|
||||
UNIQUE_LEDGERS=$(sort -u "$LEDGER_FILE" | wc -l)
|
||||
# The > 1 test also keeps the divisor below at 1 or more.
|
||||
if [ "$UNIQUE_LEDGERS" -gt 1 ]; then
|
||||
CONSENSUS_P95=$(echo "scale=0; $DURATION * 1000 / ($UNIQUE_LEDGERS - 1)" | bc 2>/dev/null || echo "0")
|
||||
CONSENSUS_MEAN=$(echo "scale=0; $DURATION * 1000 / ($UNIQUE_LEDGERS - 1)" | bc 2>/dev/null || echo "0")
|
||||
else
|
||||
CONSENSUS_P95="0"
|
||||
warn "Ledger seq never advanced ($UNIQUE_LEDGERS distinct); consensus_round_mean_ms is a 0 placeholder"
|
||||
CONSENSUS_MEAN="0"
|
||||
METRICS_COMPLETE=false
|
||||
fi
|
||||
else
|
||||
CONSENSUS_P95="0"
|
||||
warn "No ledger samples collected; consensus_round_mean_ms is a 0 placeholder"
|
||||
CONSENSUS_MEAN="0"
|
||||
METRICS_COMPLETE=false
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -223,7 +318,8 @@ cat >"$OUTPUT_FILE" <<EOF_JSON
|
||||
"memory_rss_mb_peak": $MEM_PEAK,
|
||||
"rpc_p99_ms": $RPC_P99,
|
||||
"tps": $TPS,
|
||||
"consensus_round_p95_ms": $CONSENSUS_P95,
|
||||
"consensus_round_mean_ms": $CONSENSUS_MEAN,
|
||||
"metrics_complete": $METRICS_COMPLETE,
|
||||
"samples": $SAMPLES,
|
||||
"duration_seconds": $DURATION,
|
||||
"node_count": ${#RPC_PORTS[@]},
|
||||
@@ -234,3 +330,9 @@ EOF_JSON
|
||||
|
||||
ok "Metrics written to $OUTPUT_FILE"
|
||||
cat "$OUTPUT_FILE"
|
||||
|
||||
# The file is always written first so CI still has an artifact to publish.
|
||||
if [ "$METRICS_COMPLETE" != "true" ]; then
|
||||
warn "metrics_complete=false — treat this run as inconclusive (exit 3)"
|
||||
exit 3
|
||||
fi
|
||||
|
||||
@@ -12,6 +12,8 @@ Operating modes (chosen automatically based on the baseline file contents):
|
||||
2. **Populated baseline** — per-metric percentage AND absolute deltas are
|
||||
computed against thresholds from ``regression-thresholds.json``. A
|
||||
regression occurs when BOTH bounds are breached for the same quantile.
|
||||
The one exception is a non-positive baseline, where the percentage is
|
||||
undefined: there the absolute bound decides alone.
|
||||
Prints a human-readable table and writes a full JSON report.
|
||||
Exits 1 if any regression was detected, else 0.
|
||||
|
||||
@@ -53,8 +55,10 @@ class MetricDelta:
|
||||
unit: Unit from baseline (preserved as-is).
|
||||
threshold_pct: Resolved per-metric pct threshold.
|
||||
threshold_abs: Resolved per-metric absolute threshold.
|
||||
regressed: True iff both bounds breached.
|
||||
note: Human-readable classification when not regressed.
|
||||
regressed: True iff both bounds breached, or — when the
|
||||
baseline is not positive and pct_change is
|
||||
therefore None — iff the absolute bound breached.
|
||||
note: Human-readable classification of the outcome.
|
||||
"""
|
||||
|
||||
key: str
|
||||
@@ -175,6 +179,28 @@ def _skip_delta(
|
||||
)
|
||||
|
||||
|
||||
def _delta_note(regressed: bool, delta: float, pct_change: float | None) -> str:
|
||||
"""Classify one comparison outcome for the report and the table."""
|
||||
if regressed:
|
||||
note = "REGRESSION"
|
||||
elif delta < 0:
|
||||
note = "improved"
|
||||
else:
|
||||
note = "within bounds"
|
||||
if pct_change is None:
|
||||
note += " (absolute bound only; baseline not positive)"
|
||||
return note
|
||||
|
||||
|
||||
# The regression rule, applied by compute_delta below.
|
||||
#
|
||||
# A regression normally requires BOTH bounds to be breached simultaneously.
|
||||
# That tolerates small-value noise: a 100% increase on a 0.5 ms metric (to
|
||||
# 1.0 ms) is not a regression under a 5 ms absolute bound.
|
||||
#
|
||||
# A non-positive baseline has no defined percentage change, so there the
|
||||
# absolute bound decides alone. Requiring both bounds in that case would make
|
||||
# the gate unreachable and let a 0 -> 500 ms jump pass as "within bounds".
|
||||
def compute_delta(
|
||||
key: str,
|
||||
baseline_entry: dict | None,
|
||||
@@ -183,9 +209,8 @@ def compute_delta(
|
||||
) -> MetricDelta:
|
||||
"""Compute a MetricDelta for one metric key.
|
||||
|
||||
A regression requires BOTH bounds to be breached simultaneously. This
|
||||
tolerates small-value noise: a 100% increase on a 0.5 ms metric
|
||||
(to 1.0 ms) is not a regression under a 5 ms absolute bound.
|
||||
Follows the regression rule set out in the comment above, including the
|
||||
non-positive-baseline exception.
|
||||
"""
|
||||
baseline = baseline_entry.get("value") if baseline_entry else None
|
||||
current = current_entry.get("value") if current_entry else None
|
||||
@@ -224,16 +249,13 @@ def compute_delta(
|
||||
note="no threshold configured",
|
||||
)
|
||||
|
||||
pct_breach = pct_change is not None and pct_change > pct_threshold
|
||||
abs_breach = delta > abs_threshold
|
||||
regressed = pct_breach and abs_breach
|
||||
|
||||
if regressed:
|
||||
note = "REGRESSION"
|
||||
elif delta < 0:
|
||||
note = "improved"
|
||||
if pct_change is None:
|
||||
# Baseline is not positive, so there is no percentage to compare.
|
||||
# The absolute bound is the only usable signal here.
|
||||
regressed = abs_breach
|
||||
else:
|
||||
note = "within bounds"
|
||||
regressed = pct_change > pct_threshold and abs_breach
|
||||
|
||||
return MetricDelta(
|
||||
key=key,
|
||||
@@ -245,7 +267,7 @@ def compute_delta(
|
||||
threshold_pct=pct_threshold,
|
||||
threshold_abs=abs_threshold,
|
||||
regressed=regressed,
|
||||
note=note,
|
||||
note=_delta_note(regressed, delta, pct_change),
|
||||
)
|
||||
|
||||
|
||||
@@ -265,7 +287,10 @@ def print_summary(deltas: list[MetricDelta]) -> None:
|
||||
print("=" * 72)
|
||||
|
||||
if regressions:
|
||||
print("\nRegressions (breached BOTH pct AND absolute bounds):")
|
||||
print(
|
||||
"\nRegressions (breached BOTH pct AND absolute bounds, or the "
|
||||
"absolute bound alone where the baseline is not positive):"
|
||||
)
|
||||
_print_table(regressions)
|
||||
|
||||
if improvements:
|
||||
|
||||
@@ -55,20 +55,20 @@
|
||||
"total_messages_out"
|
||||
]
|
||||
},
|
||||
"phase9_nodestore": {
|
||||
"description": "Phase 9 NodeStore I/O observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label distinguishing sub-metrics.",
|
||||
"nodestore_io": {
|
||||
"description": "NodeStore I/O observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label distinguishing sub-metrics.",
|
||||
"metrics": ["nodestore_state"]
|
||||
},
|
||||
"phase9_cache": {
|
||||
"description": "Phase 9 cache hit rate observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label.",
|
||||
"cache_hit_rates": {
|
||||
"description": "Cache hit rate observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label.",
|
||||
"metrics": ["cache_metrics"]
|
||||
},
|
||||
"phase9_txq": {
|
||||
"description": "Phase 9 transaction queue observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label.",
|
||||
"transaction_queue": {
|
||||
"description": "Transaction queue observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label.",
|
||||
"metrics": ["txq_metrics"]
|
||||
},
|
||||
"phase9_rpc_method": {
|
||||
"description": "Phase 9 per-RPC-method counters and duration histogram (MetricsRegistry.cpp:351-357). rpc_method_errored_total is deliberately absent — see not_asserted below. rpc_method_us is a Histogram, so the Prometheus exporter emits only the _bucket/_count/_sum triple and there is no bare rpc_method_us series to match — same convention as span_duration_milliseconds in the spanmetrics group above.",
|
||||
"rpc_method_detail": {
|
||||
"description": "Per-RPC-method counters and duration histogram (MetricsRegistry.cpp:351-357). rpc_method_errored_total is deliberately absent — see not_asserted below. rpc_method_us is a Histogram, so the Prometheus exporter emits only the _bucket/_count/_sum triple and there is no bare rpc_method_us series to match — same convention as span_duration_milliseconds in the spanmetrics group above.",
|
||||
"metrics": [
|
||||
"rpc_method_started_total",
|
||||
"rpc_method_finished_total",
|
||||
@@ -77,8 +77,8 @@
|
||||
"rpc_method_us_sum"
|
||||
]
|
||||
},
|
||||
"phase9_job_queue": {
|
||||
"description": "Phase 9 job-queue counters and latency histograms (MetricsRegistry.cpp:360-366). Every xrpld job passes through these, so they populate under any workload. Both histograms are recorded in the same function bodies as job_started_total / job_finished_total, under the same guard and with the same labels, so their presence is equally guaranteed. They are named with the _bucket/_count/_sum suffixes the Prometheus exporter emits: regression-metrics.json and the job-queue dashboard both query job_queued_us_bucket / job_running_us_bucket, and no bare series exists.",
|
||||
"job_queue": {
|
||||
"description": "Job-queue counters and latency histograms (MetricsRegistry.cpp:360-366). Every xrpld job passes through these, so they populate under any workload. Both histograms are recorded in the same function bodies as job_started_total / job_finished_total, under the same guard and with the same labels, so their presence is equally guaranteed. They are named with the _bucket/_count/_sum suffixes the Prometheus exporter emits: regression-metrics.json and the job-queue dashboard both query job_queued_us_bucket / job_running_us_bucket, and no bare series exists.",
|
||||
"metrics": [
|
||||
"job_queued_total",
|
||||
"job_started_total",
|
||||
@@ -95,12 +95,12 @@
|
||||
"description": "In-flight RPC gauge via the XRPL_METRIC_UPDOWN_ADD call-site macro (PerfLogImp.cpp, +1 rpcStart / -1 rpcEnd). UpDownCounter: no _total suffix.",
|
||||
"metrics": ["rpc_in_flight_requests"]
|
||||
},
|
||||
"phase9_objects": {
|
||||
"description": "Phase 9 counted object instances observable gauge (MetricsRegistry via OTLP).",
|
||||
"object_counts": {
|
||||
"description": "Counted object instances observable gauge (MetricsRegistry via OTLP).",
|
||||
"metrics": ["object_count"]
|
||||
},
|
||||
"phase9_load": {
|
||||
"description": "Phase 9 fee escalation and load factor observable gauge (MetricsRegistry via OTLP).",
|
||||
"load_factors": {
|
||||
"description": "Fee escalation and load factor observable gauge (MetricsRegistry via OTLP).",
|
||||
"metrics": ["load_factor_metrics"]
|
||||
},
|
||||
"parity_validation_agreement": {
|
||||
@@ -125,7 +125,7 @@
|
||||
]
|
||||
},
|
||||
"parity_ledger_economy": {
|
||||
"description": "External dashboard parity: ledger economy metrics (MetricsRegistry).",
|
||||
"description": "External dashboard parity: ledger economy metrics (MetricsRegistry.cpp:1401). transaction_rate is observed on every export, in both branches of the ledger-age test (MetricsRegistry.cpp:1444-1451). base_fee_xrp is observed only inside the 'if (ledger)' guard on getValidatedLedger() (MetricsRegistry.cpp:1418-1423), and that returns validLedger_ (LedgerMaster.cpp:1569-1572), which stays null until a ledger validates — the same precondition complete_ledgers has. Both are asserted because run-full-validation.sh waits for a validated ledger before running the workload. base_fee_xrp absent while transaction_rate is present is the signature of a cluster that never validated, not of a missing metric.",
|
||||
"metrics": [
|
||||
"ledger_economy{metric=\"base_fee_xrp\"}",
|
||||
"ledger_economy{metric=\"transaction_rate\"}"
|
||||
@@ -149,7 +149,7 @@
|
||||
"metrics": ["storage_detail{metric=\"stored_object_bytes\"}"]
|
||||
},
|
||||
"node_health_gauges": {
|
||||
"description": "Node-health observable gauges (MetricsRegistry.cpp:997, :1081, :1102, :1161). All four are registered with callbacks that fire on every periodic export and Observe unconditionally (build_info observes a literal 1; server_info and db_metrics read live services; complete_ledgers observes the parsed ledger range, which is non-empty once the cluster has closed a ledger), so their series exist regardless of workload shape.",
|
||||
"description": "Node-health observable gauges (MetricsRegistry.cpp:997, :1081, :1102, :1161). server_info, build_info and db_metrics Observe unconditionally on every periodic export (build_info observes a literal 1; server_info and db_metrics read live services), so their series exist regardless of workload shape. complete_ledgers is the exception and is asserted on a narrower guarantee: its callback returns without observing when the range is empty (MetricsRegistry.cpp:1113-1114) and skips any segment that carries no '-' (:1122-1127), and a one-sequence range renders with no '-' (RangeSet.h:70-71), so it needs a complete range spanning at least two sequences. completeLedgers_ is filled by setFullLedger (LedgerMaster.cpp:862-863), which on a peered node is reached only from the publish path in doAdvance (LedgerMaster.cpp:1972) — closing a ledger is not enough, it has to validate. run-full-validation.sh waits for that before the workload starts, so on a healthy cluster the series always exists — a 5-node run yields 10 series, one start and one end per node. If this check ever fails, read the Step 3 output first: a run that logged 'No validated ledger' cannot produce this series and the cluster, not the exporter, is what broke.",
|
||||
"metrics": ["server_info", "build_info", "complete_ledgers", "db_metrics"]
|
||||
},
|
||||
"overlay_reduce_relay": {
|
||||
@@ -169,7 +169,7 @@
|
||||
"metrics_excluded": {
|
||||
"rpc_method_errored_total": "MetricsRegistry.cpp:354, push counter — needs an RPC that returns an error. rpc_load_generator.py issues only well-formed server_info / fee / ledger / ripple_path_find calls, so no series may ever be created.",
|
||||
"ledger_history_mismatch_total": "MetricsRegistry.cpp:377, incremented only from LedgerHistory.cpp:332 on a built-vs-validated ledger mismatch. On a healthy run it never fires — asserting it would mean asserting a defect.",
|
||||
"txq_expired_total": "MetricsRegistry.cpp:379, incremented only at TxQ.cpp:1428 when a queued tx expires past its LastLedgerSequence. Requires sustained fee escalation plus expiry; the CI job drives rpc_load_generator/tx_submitter directly with --rpc-rate/--tx-tps and never runs the workload-profiles.json txq-burst phase, so this is not reachable in CI.",
|
||||
"txq_expired_total": "MetricsRegistry.cpp:379, incremented only at TxQ.cpp:1428 when a queued tx expires past its LastLedgerSequence. CI does run a txq-burst phase (workload-profiles.json:41, 30 s of single-type Payment at 60 TPS), but that does not guarantee sustained fee escalation followed by expiry: a run in which every other check passed still exposed only txq_metrics and no txq_expired_total.",
|
||||
"txq_dropped_total": "MetricsRegistry.cpp:381, incremented only at TxQ.cpp:1302 / :1347 on queue-full admission refusal. Same reason as txq_expired_total.",
|
||||
"getobject_rejected_total": "GetObjectMetricNames.h:81, emitted from PeerImp.cpp:2725/:2743 only for a TMGetObjectByHash message refused as oversize or malformed_ledgerhash. A cooperating cluster never sends one.",
|
||||
"getobject_request_objects": "GetObjectMetricNames.h:86, emitted from PeerImp.cpp:2926 only while serving an inbound TMGetObjectByHash. The XRPL_METRIC_* macros create their instrument lazily on first use (MetricMacros.h:174-285), so no series exists until a peer actually requests objects by hash — which a 5-node cluster started at genesis and already in sync may never do.",
|
||||
|
||||
@@ -123,8 +123,14 @@ for i in $(seq 1 "$NUM_NODES"); do
|
||||
seed=$(echo "$result" | jq -r '.result.validation_seed')
|
||||
pubkey=$(echo "$result" | jq -r '.result.validation_public_key')
|
||||
|
||||
# Both fields must be present. jq -r prints the literal string "null" for
|
||||
# a missing field, so an unvalidated pubkey would be written verbatim into
|
||||
# validators.txt and xrpld would reject the file at startup.
|
||||
if [ -z "$seed" ] || [ "$seed" = "null" ]; then
|
||||
die "Failed to generate key pair for node $i"
|
||||
die "Failed to generate key pair for node $i: no validation_seed in response"
|
||||
fi
|
||||
if [ -z "$pubkey" ] || [ "$pubkey" = "null" ]; then
|
||||
die "Failed to generate key pair for node $i: no validation_public_key in response"
|
||||
fi
|
||||
|
||||
log " Node $i: ${pubkey:0:20}..."
|
||||
|
||||
@@ -28,6 +28,7 @@ Usage::
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
@@ -38,6 +39,13 @@ import aiohttp
|
||||
|
||||
logger = logging.getLogger("prom_queries")
|
||||
|
||||
# Instant queries run in parallel, but not all at once: a single-node
|
||||
# Prometheus is easily saturated by a burst of the whole plan. With this cap
|
||||
# the worst case (every query hitting the 30 s timeout) is
|
||||
# ceil(len(plan) / 8) * 30 s rather than len(plan) * 30 s, which for a
|
||||
# ~30-entry plan is ~2 min instead of ~15 min of a 30 min CI job.
|
||||
MAX_CONCURRENT_QUERIES = 8
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class QueryEntry:
|
||||
@@ -147,6 +155,17 @@ async def run_query_plan(
|
||||
as "not yet observed" rather than as a regression. This keeps the
|
||||
baseline schema stable across runs with different load levels.
|
||||
|
||||
Queries run concurrently, at most MAX_CONCURRENT_QUERIES at a time. Keys
|
||||
are emitted in plan order regardless of which query answers first.
|
||||
|
||||
``return_exceptions=True`` is what keeps the fan-out self-contained.
|
||||
Without it the first escaping exception returns from ``gather`` while its
|
||||
siblings keep running against a session the caller is about to close, so
|
||||
the capture ends with dozens of orphaned queries and warnings from a
|
||||
closed session. With it, every query is awaited before this returns, and
|
||||
an unexpected failure is logged and recorded as no data -- the same
|
||||
outcome _instant_query already produces for a query that fails.
|
||||
|
||||
Args:
|
||||
session: Shared aiohttp session.
|
||||
prom_url: Base URL of Prometheus (e.g. ``http://localhost:9090``).
|
||||
@@ -155,11 +174,32 @@ async def run_query_plan(
|
||||
Returns:
|
||||
Mapping from metric key to ``{"value": float|None, "unit": str}``.
|
||||
"""
|
||||
results: dict[str, dict[str, Any]] = {}
|
||||
for entry in plan:
|
||||
value = await _instant_query(session, prom_url, entry.promql)
|
||||
results[entry.key] = {"value": value, "unit": entry.unit}
|
||||
return results
|
||||
gate = asyncio.Semaphore(MAX_CONCURRENT_QUERIES)
|
||||
|
||||
async def fetch(entry: QueryEntry) -> float | None:
|
||||
"""Run one plan entry once a query slot is free."""
|
||||
async with gate:
|
||||
return await _instant_query(session, prom_url, entry.promql)
|
||||
|
||||
results = await asyncio.gather(
|
||||
*(fetch(entry) for entry in plan), return_exceptions=True
|
||||
)
|
||||
|
||||
captured: dict[str, dict[str, Any]] = {}
|
||||
for entry, result in zip(plan, results, strict=True):
|
||||
value: float | None
|
||||
if isinstance(result, BaseException):
|
||||
logger.error(
|
||||
"query for %s raised %s: %s",
|
||||
entry.key,
|
||||
type(result).__name__,
|
||||
result,
|
||||
)
|
||||
value = None
|
||||
else:
|
||||
value = result
|
||||
captured[entry.key] = {"value": value, "unit": entry.unit}
|
||||
return captured
|
||||
|
||||
|
||||
async def _instant_query(
|
||||
@@ -181,7 +221,10 @@ async def _instant_query(
|
||||
logger.warning("query HTTP %d: %s", resp.status, promql)
|
||||
return None
|
||||
body = await resp.json()
|
||||
except (aiohttp.ClientError, TimeoutError) as exc:
|
||||
# JSONDecodeError covers a 200 response whose body is not JSON: without it
|
||||
# one malformed reply aborts the whole capture, since no caller of
|
||||
# run_query_plan wraps it.
|
||||
except (aiohttp.ClientError, TimeoutError, json.JSONDecodeError) as exc:
|
||||
logger.warning("query failed: %s — %s", promql, exc)
|
||||
return None
|
||||
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
{
|
||||
"_description": "Metric surface for the OTel-driven regression gate. Each entry names a metric, the quantiles to capture, and how to query Prometheus. The comparator compares current run against baseline-timings.json under these exact keys.",
|
||||
"_key_format": "{category}.{name}.p{quantile} (e.g. span.tx.process.p99, job.transaction.queued.p95). Only the categories defined below are captured; there is no rpc_methods group, so no rpc.* key is produced or gated (FU-4).",
|
||||
"_excluded_spans": "rpc.process is deliberately absent from spans.names. It is created only in ServerHandler::processRequest() on the HTTP/JSON-RPC path, which the workload load generators, being WebSocket-only, never reach, so its quantiles were captured as null every run and could never gate. (The harness shell scripts do issue a few HTTP JSON-RPC health polls, far too few to produce a meaningful quantile.) See baselines/README.md.",
|
||||
"spans": {
|
||||
"_query_template": "histogram_quantile({quantile}, sum by (le) (rate(span_duration_milliseconds_bucket{span_name=\"{name}\"}[{window}])))",
|
||||
"_unit": "ms",
|
||||
"_quantiles": [0.5, 0.95, 0.99],
|
||||
"names": [
|
||||
"rpc.ws_message",
|
||||
"rpc.process",
|
||||
"tx.process",
|
||||
"tx.apply",
|
||||
"ledger.build",
|
||||
|
||||
@@ -1,6 +1,786 @@
|
||||
# Python dependencies for Phase 10 workload tools.
|
||||
# Python dependencies for the telemetry workload tools.
|
||||
#
|
||||
# cspell:ignore aiohappyeyeballs
|
||||
#
|
||||
# Install: pip install -r requirements.txt
|
||||
|
||||
websockets>=12.0
|
||||
aiohttp>=3.9.0
|
||||
#
|
||||
# Pinned with hashes so every install resolves to exactly these releases.
|
||||
# Direct dependencies are websockets and aiohttp; the rest is aiohttp's
|
||||
# transitive closure, which pip requires once any hash is present.
|
||||
#
|
||||
# Floors to keep when bumping: aiohttp >= 3.10.11 (earliest release carrying
|
||||
# every 2024 aiohttp fix), websockets >= 14 (rejects a concurrent recv() with
|
||||
# ConcurrencyError instead of a bare RuntimeError).
|
||||
#
|
||||
# Regenerate (python >= 3.11, the repo minimum, matching the universal floor):
|
||||
# printf 'websockets>=17.0\naiohttp>=3.14.3\n' > requirements.in
|
||||
# uv pip compile requirements.in --generate-hashes --universal \
|
||||
# --python-version 3.11 --output-file requirements.txt
|
||||
aiohappyeyeballs==2.7.1 \
|
||||
--hash=sha256:065665c041c42a5938ed220bdcd7230f22527fbec085e1853d2402c8a3615d9d \
|
||||
--hash=sha256:9243213661e29250eb41368e5daa826fc017156c3b8a11440826b2e3ed376472
|
||||
# via aiohttp
|
||||
aiohttp==3.14.3 \
|
||||
--hash=sha256:03cd2bde3d7f085b64e549c985f4bb928cad7e8ecf5323bfca320db548d81b39 \
|
||||
--hash=sha256:041badb8f84396357c4d3ad26de6afd7a32b112f43d3c63045c0c8278cfd2043 \
|
||||
--hash=sha256:0a5ff2dfbb9ce645fa5b8ef3e02c6c0b9cc3f6030ff863d0c51fffc50cb5541b \
|
||||
--hash=sha256:0fdea2281997af69da84c77ffa6f5938a0285f21fb3887c249d67419ca865b3d \
|
||||
--hash=sha256:11fb37ef075669eee52ab1928fbf6e1741fada40409fa309ebde9607a962aebf \
|
||||
--hash=sha256:134ac5ddcf61c6fad984b9a5727d83492ada43d63471db20fb73042c13fca62f \
|
||||
--hash=sha256:152516815ef926786a0b6ae2b8f1fd2e0c71582dee0b435636865316fd4891b7 \
|
||||
--hash=sha256:1576145bdceeb92382d899751e12743a3a5b8e460a841e3e50543859e54864dc \
|
||||
--hash=sha256:16100ad3ab8d649fdfbee87602d9d2dcdca9df0b9eda8a1b5fdc0d41f96da559 \
|
||||
--hash=sha256:16ea7e24c309fb7c0bbd505d149abe4fe4dccfb8db911db7dbec0921bc889a6f \
|
||||
--hash=sha256:18c441d0a8fca6de8d1f546849b9f0ab20d435993e2c5b59562b2fae6be2f929 \
|
||||
--hash=sha256:18cb43369747b2ae007bd2655fb8e63a099c2ff1d207962943636dac989b3147 \
|
||||
--hash=sha256:1b59533861b70a2185c8f4f350f791f39d64358ef6944ce71c5240c9ec0982c9 \
|
||||
--hash=sha256:1c5281acc88b92396f88c7e1e2748f8466689df22b80170e4f51efa712fb47a8 \
|
||||
--hash=sha256:1c5ec8fb1bcc31a8466f74aaf26c345d5c386fa4bd08a3f0eb9c7a4a3fe8b5bf \
|
||||
--hash=sha256:1caa7b0d05f3e3a36f87788c59e970a7ee1cefcfcbb924a9f138c4a6551c9cb7 \
|
||||
--hash=sha256:21c016079415ed3fd676963e9793700a566d85dbbd6bfc564b9b2d209147dcc8 \
|
||||
--hash=sha256:2498f0fe69ead802f9675beca44a7c21c62fdaa4ec5145ea1c3ad6edbee29f85 \
|
||||
--hash=sha256:25bd2708db6bdf6a6630dd37bdcdfcb47c4434d22ac69c64665b802910140b30 \
|
||||
--hash=sha256:270d3dace9ca2f10f0da5d8ebe519b7a310fc6112ed916e32df5866df0888553 \
|
||||
--hash=sha256:2e1161602f45a54de2ce0905243a95f58cb42dcd378402f3697f5e0b21e9d2e7 \
|
||||
--hash=sha256:2e9878ae68e4a5f1c0abe4dd497dbc3d51946f5837b56759e2a02e78fa90ef86 \
|
||||
--hash=sha256:30402d03a7c0ff52bce290b57e564e9079fd9d0cb545c8aba73f86a103162d2e \
|
||||
--hash=sha256:33a2d7c28d33797a2e99923dffa63f83d908a19b6bf26cfe80fa790aa5e1a75a \
|
||||
--hash=sha256:362a3fd481769cac1a824514bcd86fda51c65e8fe6e051099e008fddde6db17c \
|
||||
--hash=sha256:38901a84da3ce22249f6e860bf8f90d141bcab7da090cc398f8bb58c0e44b7da \
|
||||
--hash=sha256:39aded8c7f3b935b54aab1d8d73c70ec0ee2d3ec3b943e0e86611bc150ba47f5 \
|
||||
--hash=sha256:3a26434dafe408229ff3403458ca58de24fb51936504decac49ce6755f77e59d \
|
||||
--hash=sha256:3ae5b3a59436d089b5395d910121a390feed4d00578eb95a0fd1a329fe963100 \
|
||||
--hash=sha256:3d4f72af88ac2474bb5bca640030320e3d38a0163a1d7533500e87be458eef71 \
|
||||
--hash=sha256:3f42e9b78301f11c8f861746175d8b9c1ccef713fcad9eab396e2f6db8ed4a22 \
|
||||
--hash=sha256:42a67efc36300d052fb4508a53e8b6901b9284b599ae63945c377569c5fcc1e1 \
|
||||
--hash=sha256:48d67b87db6279c044760787eb01f6413032c2e6f3ba1cafaa492b1c8e578479 \
|
||||
--hash=sha256:498c6c623134f8e09a3c4e60bcd607a0b4590dd7dbf08dd40851b27cbb520ccb \
|
||||
--hash=sha256:49f7325beb0f85ef4aef5f48f490269575f83e6e2acad00a1d80b807eb027062 \
|
||||
--hash=sha256:4e3ac92d90e92773b2362d506068e9a948192bd553e743c5b2429e28527c8661 \
|
||||
--hash=sha256:530125ee1163c4219af35dc3aa1206e541e7b31b6efc1a3f93b70a136f65d427 \
|
||||
--hash=sha256:5373dc80ad1aa2fb9ad95c83f24eef418bbda3a61375f128e5b0192e4f3f9b32 \
|
||||
--hash=sha256:53e5179d8abb5710f8e83ba207c41c8d1261fcffd4616500e15ca2b7a33be10a \
|
||||
--hash=sha256:53e7b4ce82b54a8bcc71b3b67a5cbd177ca1d7f592cbc92cd38b7349f73482db \
|
||||
--hash=sha256:543906c127fb1d929b95076db19b83fa2d46751006ff1e23b093aa5ac4d8db42 \
|
||||
--hash=sha256:54cfcdee2770dac994417cbb0ee1f3eb0e7cb6b30c79bf44f2c02ff79ec5124a \
|
||||
--hash=sha256:55bdcc472aafe2de4a253045cc128007a64f1e0264fb675791e132ea5edaa3bd \
|
||||
--hash=sha256:56f355e79f71aef2a85c80305cc915f894b170dba76de5fe84f6351939b83c06 \
|
||||
--hash=sha256:5895ef58c4620afe02fa16044f023dc4dafec08158f9d08874a46a7dbc0341b8 \
|
||||
--hash=sha256:5bcb6ff3fdab1258a192679ff1a05d44f59626430aa05cd1a9d2447423599228 \
|
||||
--hash=sha256:5f08ec777f35ee70720233b8b9811d3bb5d728137f30ac91b7457709c3261ac0 \
|
||||
--hash=sha256:614c61d478b83953e261d02bb2df750f17227cd33ef8002945bf5aebbde21919 \
|
||||
--hash=sha256:617105e2c3018ee38d0c8ce5ee3c84f621a6d8b9f723202aacaff28449ca91ee \
|
||||
--hash=sha256:6debfa7312ff9d4c124dc71d72e9a0a4b9e0879e48ba6fcb42bef5c3300289e2 \
|
||||
--hash=sha256:7041d52c3a7fa20c9e8c182b534704abb19502c8bdcbde7ab23bfda6f642394f \
|
||||
--hash=sha256:70c987b27534f9ae1a723f47ae921571d616da21d3208282bf4c52af5164ac43 \
|
||||
--hash=sha256:74ab5b6a9fb13e873e5a90946588baecaf488745e1db1a4a5c433f971f035098 \
|
||||
--hash=sha256:78253b573e6ffab5028924fc98bc281aae05445969982a10864bc360dea2016c \
|
||||
--hash=sha256:7a75aa63cbf9b21cfaf60dc2657e19df2c2867d91707d653fee171ffeedd1371 \
|
||||
--hash=sha256:8800c996b01c2772a783e3e46f3e1abd5823029adca0df54231960de9bfefa5b \
|
||||
--hash=sha256:89176250f686cb9853c0fb7ead90e639e915b84a6f43eedc2a4e7ec21f1037f0 \
|
||||
--hash=sha256:8a5fd34f7f7410d1730d5c2ba873cacb2eed3fede366feb268a70ba22581ed8f \
|
||||
--hash=sha256:8b3b60de05f3dcb6f6a00f818bb2ec781cee4de0645f59ccaf99b1d1823b6100 \
|
||||
--hash=sha256:8f2f1c4c032c7cedd7d8da6f54c97b70266c6570c3108d3fdffee7188bb70529 \
|
||||
--hash=sha256:9491196535a88924a60afd5b5f434b5b203b6cc616250878dbdb223a8f7844bc \
|
||||
--hash=sha256:9aa6e61fdf20105c4144e755bd586008ff450791d67b1c8146fdc15959c4d51c \
|
||||
--hash=sha256:9d9edccfe496b476db5f398d97b865e9a6752bcf8aec4eef8390ce20fb64bb41 \
|
||||
--hash=sha256:9fc7b5bfec6573f3ae844f457fdde5adeb713f8b8e4a81ad64fc207b49383716 \
|
||||
--hash=sha256:a0dc483c00da8b673abbb367eb6f8d8f4bcec30eb58529ea13cb42e7fd2dfa33 \
|
||||
--hash=sha256:a3a8296e7ab5c295f53f1041487cb088e1480775aafbf7fe545d93b770a0f96f \
|
||||
--hash=sha256:a3e22975f905b89a55a488c2a08f2fdb2186175349e917d48985cc468a3d4c6e \
|
||||
--hash=sha256:a4af35c443e0b1a1bd6a8af3f3485d7fda15c142751a00f3ff8090f0b93346fa \
|
||||
--hash=sha256:a94dbaae5ae27bd849c93570669bff91e0510f33a80805738e3de72a7be0447b \
|
||||
--hash=sha256:ac74facc01463f138b0da5580329cfcc82818dea5656e83ddcd11268fc12ff80 \
|
||||
--hash=sha256:ad4c8b7488d745d2ca4838ebd8ae5ba9b56341d30b1da43640e4ce87f9f49646 \
|
||||
--hash=sha256:b014a6ed7cf912e787149fdc529166d3ceabac23f26efeea3158c9aba2354e7e \
|
||||
--hash=sha256:b20032766aedf6261c7a566585a40867d092ac03a0d81592d5370ef9b054f99b \
|
||||
--hash=sha256:b2466434105a4e03113c36ec775cc2ebe6676b62eae326fa670bb607ef788c1c \
|
||||
--hash=sha256:b304db572b4368edd8dda8a2274f73156fe15558fca4a917cb8a09fc47af5963 \
|
||||
--hash=sha256:ba59d59aba08ac02fc03b0c8983ccd5ee39a199d0552ce9e6d2b4845b34d59ae \
|
||||
--hash=sha256:bd52f811e65f6fb634b1047159657c98f52b407f8efec907bcfc09da9a4c0a25 \
|
||||
--hash=sha256:bdd0e2834dce1a26c1bbe26464861e16bbe217042cbff619247c11594472518c \
|
||||
--hash=sha256:c23ec8ee9d5ab2f5421f9c7fffce208435607af27fd46d4a44e031954352838f \
|
||||
--hash=sha256:c39846c3aad97a8530c89d7a3869a8f8e9e3762c6ac0504481e5c80948f7e807 \
|
||||
--hash=sha256:c3c200cf9757edd785051dc699c7ecbec22110dbfcb3fefc7a9f9695eda8ea7a \
|
||||
--hash=sha256:c7d3a97c678d34fc5b59da671ee9cd630096ddc643e7b5a30d54a2a6f3574d3f \
|
||||
--hash=sha256:c8653fd547c93a61aadc612007790f5555cdd18946fa48cf45e26d8ea4ea473d \
|
||||
--hash=sha256:cc7cb243a68167172f48c1fd43cee91ec4b1d40cefd190edd43369d1a6bc9c82 \
|
||||
--hash=sha256:ccd4893707b3e2a13e39c90d43cf80edf2e4d0457935bcc103bf2346214c3f15 \
|
||||
--hash=sha256:cd817772b2fcf2b8c0905795318485f9ec16eae60b29feb7f4c77085311637f0 \
|
||||
--hash=sha256:cda5fd5c95ad7a125a2e8464acc78b98b94c475a3780d6aa0aa157c93f470f4d \
|
||||
--hash=sha256:cef89a58e628c4efcac3275c2d68083f82426dcdc89c1492a6f654f9f7ea6ab9 \
|
||||
--hash=sha256:d1558173930a5a8d3069cee5c92fc91c87c4dbcb099debbb3622053717145a19 \
|
||||
--hash=sha256:d6088ec9894113802bddb3c09e974929aed2c7b3a8c456219b8aab4481f1a239 \
|
||||
--hash=sha256:d6218d92e450824e9b4881f44e8c09f1853b490f9a64130801024a4793b1b3b0 \
|
||||
--hash=sha256:d77640cc618c1d99fc4f8589c0f24a730adfa54eb1e57ef7bf0c8dfb78da898c \
|
||||
--hash=sha256:d7d2deec16eeedf55f2c7cf75b521ea3856a5177e123844f8fd0f114ce252cb5 \
|
||||
--hash=sha256:db332af25642007330fca8be5c4d194caf2bea7a7fc84415aff3497af5dfee6b \
|
||||
--hash=sha256:dd54d0e8717de95939766febac482ac0474d8ac3b048115f9f2b1d23a16e7db4 \
|
||||
--hash=sha256:ddcac3c6b382e81f1dd0499199d4136b877beb4cb5ef770bbbfba56c4b8f55d2 \
|
||||
--hash=sha256:df82f3787c940c94986b34222d59c9e38843fba85139f36e85255a82ad5355a9 \
|
||||
--hash=sha256:dfa68deb2a443bdaa3ea5297b0699c1464f08aef3812b486d1348eee61b07dc0 \
|
||||
--hash=sha256:dff9461ec275f22135650d5ba4b4931a11f3958df7dfbb8db630000d4dee0883 \
|
||||
--hash=sha256:e1e74298bab6ee0d6e749ed4fd1901c7e604bdda32c03d787a2cc71c46d0433d \
|
||||
--hash=sha256:e2667f0bbe7eb6c74eae5e9691441ad186e5845ca3cff63230fc09c4e7514f5d \
|
||||
--hash=sha256:e3be98a7c30b8c25d573dafba7171d66dfb05ee6a9070fc46535464ff97700a6 \
|
||||
--hash=sha256:e568e14940c09955aa51f4e645b6daa18a581c5dcfcd73744dcc86a856e3ced3 \
|
||||
--hash=sha256:e72ee89e28d907a18f46959b4eb0bb06701cc7f8cf4366e00029e2ccfaaf5924 \
|
||||
--hash=sha256:e92eb8acc45eb6a9f4935071a77edf5b85cc6f8dfad5cd99e97653c26593cdde \
|
||||
--hash=sha256:ea05e1f97ceea523942d9b2a7d7c0359d781d683d6b043f5943a602b14da4787 \
|
||||
--hash=sha256:eac645b09bcfdf73df7536331f0678c1086ea250981118ddb5199e17ccef72bb \
|
||||
--hash=sha256:eb0495d778817619273c108784292be161a924b9f5ae5cbbc70a2caa6838250b \
|
||||
--hash=sha256:ebe8e504f058fe91223351cecd2d9d6946c9d241bb0250d898ffbdf584cc72b0 \
|
||||
--hash=sha256:ed099d105449c4f9e84f24af203cd131349d4761d8813fa7e02c32e7128cd910 \
|
||||
--hash=sha256:f0f177d1b195b9e06376cfd7d308d8a1b920909a609d03ac82a8c73bbb16d3b9 \
|
||||
--hash=sha256:f3d2669fe7dec7fc359ecdb5984b29b50d85d5d00f8c1cb61de4f4a24ee42627 \
|
||||
--hash=sha256:f4e05329faa0ea1a404b37de4f034fd2c2defcca06a68dc6745e4e56c88e8a48 \
|
||||
--hash=sha256:f53bcd52f585e1ac3e590d61434eb61f9a88c38df041b4ea126d97144344a77b \
|
||||
--hash=sha256:f55119f7bf25f49ed210f6096090715da24f2943c62102448915fde3c62877ce \
|
||||
--hash=sha256:f631fe87a6f30df5fbe6d79640b25e4cffb38c31c7fb6f10871517b84b0f8c1a \
|
||||
--hash=sha256:f8fb78a83c9e5f741ca3a68cfb455c1f5bb83b4e7249a3848b3cd78d0a8563b0 \
|
||||
--hash=sha256:fa9467a8113aa69d3d7c55a70ef0b7c636010a40993f3df9d9d0d73b3eb7ef24 \
|
||||
--hash=sha256:fd51ebf9d3a00c074df4ede271023f4d2dba289bcc740b88191872716014e3c5
|
||||
# via -r requirements.in
|
||||
aiosignal==1.4.0 \
|
||||
--hash=sha256:053243f8b92b990551949e63930a839ff0cf0b0ebbe0597b0f3fb19e1a0fe82e \
|
||||
--hash=sha256:f47eecd9468083c2029cc99945502cb7708b082c232f9aca65da147157b251c7
|
||||
# via aiohttp
|
||||
attrs==26.1.0 \
|
||||
--hash=sha256:c647aa4a12dfbad9333ca4e71fe62ddc36f4e63b2d260a37a8b83d2f043ac309 \
|
||||
--hash=sha256:d03ceb89cb322a8fd706d4fb91940737b6642aa36998fe130a9bc96c985eff32
|
||||
# via aiohttp
|
||||
frozenlist==1.8.0 \
|
||||
--hash=sha256:0325024fe97f94c41c08872db482cf8ac4800d80e79222c6b0b7b162d5b13686 \
|
||||
--hash=sha256:032efa2674356903cd0261c4317a561a6850f3ac864a63fc1583147fb05a79b0 \
|
||||
--hash=sha256:03ae967b4e297f58f8c774c7eabcce57fe3c2434817d4385c50661845a058121 \
|
||||
--hash=sha256:06be8f67f39c8b1dc671f5d83aaefd3358ae5cdcf8314552c57e7ed3e6475bdd \
|
||||
--hash=sha256:073f8bf8becba60aa931eb3bc420b217bb7d5b8f4750e6f8b3be7f3da85d38b7 \
|
||||
--hash=sha256:07cdca25a91a4386d2e76ad992916a85038a9b97561bf7a3fd12d5d9ce31870c \
|
||||
--hash=sha256:09474e9831bc2b2199fad6da3c14c7b0fbdd377cce9d3d77131be28906cb7d84 \
|
||||
--hash=sha256:0c18a16eab41e82c295618a77502e17b195883241c563b00f0aa5106fc4eaa0d \
|
||||
--hash=sha256:0f96534f8bfebc1a394209427d0f8a63d343c9779cda6fc25e8e121b5fd8555b \
|
||||
--hash=sha256:102e6314ca4da683dca92e3b1355490fed5f313b768500084fbe6371fddfdb79 \
|
||||
--hash=sha256:11847b53d722050808926e785df837353bd4d75f1d494377e59b23594d834967 \
|
||||
--hash=sha256:119fb2a1bd47307e899c2fac7f28e85b9a543864df47aa7ec9d3c1b4545f096f \
|
||||
--hash=sha256:13d23a45c4cebade99340c4165bd90eeb4a56c6d8a9d8aa49568cac19a6d0dc4 \
|
||||
--hash=sha256:154e55ec0655291b5dd1b8731c637ecdb50975a2ae70c606d100750a540082f7 \
|
||||
--hash=sha256:168c0969a329b416119507ba30b9ea13688fafffac1b7822802537569a1cb0ef \
|
||||
--hash=sha256:17c883ab0ab67200b5f964d2b9ed6b00971917d5d8a92df149dc2c9779208ee9 \
|
||||
--hash=sha256:1a7607e17ad33361677adcd1443edf6f5da0ce5e5377b798fba20fae194825f3 \
|
||||
--hash=sha256:1a7fa382a4a223773ed64242dbe1c9c326ec09457e6b8428efb4118c685c3dfd \
|
||||
--hash=sha256:1aa77cb5697069af47472e39612976ed05343ff2e84a3dcf15437b232cbfd087 \
|
||||
--hash=sha256:1b9290cf81e95e93fdf90548ce9d3c1211cf574b8e3f4b3b7cb0537cf2227068 \
|
||||
--hash=sha256:20e63c9493d33ee48536600d1a5c95eefc870cd71e7ab037763d1fbb89cc51e7 \
|
||||
--hash=sha256:21900c48ae04d13d416f0e1e0c4d81f7931f73a9dfa0b7a8746fb2fe7dd970ed \
|
||||
--hash=sha256:229bf37d2e4acdaf808fd3f06e854a4a7a3661e871b10dc1f8f1896a3b05f18b \
|
||||
--hash=sha256:2552f44204b744fba866e573be4c1f9048d6a324dfe14475103fd51613eb1d1f \
|
||||
--hash=sha256:27c6e8077956cf73eadd514be8fb04d77fc946a7fe9f7fe167648b0b9085cc25 \
|
||||
--hash=sha256:28bd570e8e189d7f7b001966435f9dac6718324b5be2990ac496cf1ea9ddb7fe \
|
||||
--hash=sha256:294e487f9ec720bd8ffcebc99d575f7eff3568a08a253d1ee1a0378754b74143 \
|
||||
--hash=sha256:29548f9b5b5e3460ce7378144c3010363d8035cea44bc0bf02d57f5a685e084e \
|
||||
--hash=sha256:2c5dcbbc55383e5883246d11fd179782a9d07a986c40f49abe89ddf865913930 \
|
||||
--hash=sha256:2dc43a022e555de94c3b68a4ef0b11c4f747d12c024a520c7101709a2144fb37 \
|
||||
--hash=sha256:2f05983daecab868a31e1da44462873306d3cbfd76d1f0b5b69c473d21dbb128 \
|
||||
--hash=sha256:33139dc858c580ea50e7e60a1b0ea003efa1fd42e6ec7fdbad78fff65fad2fd2 \
|
||||
--hash=sha256:332db6b2563333c5671fecacd085141b5800cb866be16d5e3eb15a2086476675 \
|
||||
--hash=sha256:33f48f51a446114bc5d251fb2954ab0164d5be02ad3382abcbfe07e2531d650f \
|
||||
--hash=sha256:34187385b08f866104f0c0617404c8eb08165ab1272e884abc89c112e9c00746 \
|
||||
--hash=sha256:342c97bf697ac5480c0a7ec73cd700ecfa5a8a40ac923bd035484616efecc2df \
|
||||
--hash=sha256:3462dd9475af2025c31cc61be6652dfa25cbfb56cbbf52f4ccfe029f38decaf8 \
|
||||
--hash=sha256:39ecbc32f1390387d2aa4f5a995e465e9e2f79ba3adcac92d68e3e0afae6657c \
|
||||
--hash=sha256:3e0761f4d1a44f1d1a47996511752cf3dcec5bbdd9cc2b4fe595caf97754b7a0 \
|
||||
--hash=sha256:3ede829ed8d842f6cd48fc7081d7a41001a56f1f38603f9d49bf3020d59a31ad \
|
||||
--hash=sha256:3ef2d026f16a2b1866e1d86fc4e1291e1ed8a387b2c333809419a2f8b3a77b82 \
|
||||
--hash=sha256:405e8fe955c2280ce66428b3ca55e12b3c4e9c336fb2103a4937e891c69a4a29 \
|
||||
--hash=sha256:42145cd2748ca39f32801dad54aeea10039da6f86e303659db90db1c4b614c8c \
|
||||
--hash=sha256:4314debad13beb564b708b4a496020e5306c7333fa9a3ab90374169a20ffab30 \
|
||||
--hash=sha256:433403ae80709741ce34038da08511d4a77062aa924baf411ef73d1146e74faf \
|
||||
--hash=sha256:44389d135b3ff43ba8cc89ff7f51f5a0bb6b63d829c8300f79a2fe4fe61bcc62 \
|
||||
--hash=sha256:48e6d3f4ec5c7273dfe83ff27c91083c6c9065af655dc2684d2c200c94308bb5 \
|
||||
--hash=sha256:494a5952b1c597ba44e0e78113a7266e656b9794eec897b19ead706bd7074383 \
|
||||
--hash=sha256:4970ece02dbc8c3a92fcc5228e36a3e933a01a999f7094ff7c23fbd2beeaa67c \
|
||||
--hash=sha256:4e0c11f2cc6717e0a741f84a527c52616140741cd812a50422f83dc31749fb52 \
|
||||
--hash=sha256:50066c3997d0091c411a66e710f4e11752251e6d2d73d70d8d5d4c76442a199d \
|
||||
--hash=sha256:517279f58009d0b1f2e7c1b130b377a349405da3f7621ed6bfae50b10adf20c1 \
|
||||
--hash=sha256:54b2077180eb7f83dd52c40b2750d0a9f175e06a42e3213ce047219de902717a \
|
||||
--hash=sha256:5500ef82073f599ac84d888e3a8c1f77ac831183244bfd7f11eaa0289fb30714 \
|
||||
--hash=sha256:581ef5194c48035a7de2aefc72ac6539823bb71508189e5de01d60c9dcd5fa65 \
|
||||
--hash=sha256:59a6a5876ca59d1b63af8cd5e7ffffb024c3dc1e9cf9301b21a2e76286505c95 \
|
||||
--hash=sha256:5a3a935c3a4e89c733303a2d5a7c257ea44af3a56c8202df486b7f5de40f37e1 \
|
||||
--hash=sha256:5c1c8e78426e59b3f8005e9b19f6ff46e5845895adbde20ece9218319eca6506 \
|
||||
--hash=sha256:5d63a068f978fc69421fb0e6eb91a9603187527c86b7cd3f534a5b77a592b888 \
|
||||
--hash=sha256:667c3777ca571e5dbeb76f331562ff98b957431df140b54c85fd4d52eea8d8f6 \
|
||||
--hash=sha256:6da155091429aeba16851ecb10a9104a108bcd32f6c1642867eadaee401c1c41 \
|
||||
--hash=sha256:6dc4126390929823e2d2d9dc79ab4046ed74680360fc5f38b585c12c66cdf459 \
|
||||
--hash=sha256:7398c222d1d405e796970320036b1b563892b65809d9e5261487bb2c7f7b5c6a \
|
||||
--hash=sha256:74c51543498289c0c43656701be6b077f4b265868fa7f8a8859c197006efb608 \
|
||||
--hash=sha256:776f352e8329135506a1d6bf16ac3f87bc25b28e765949282dcc627af36123aa \
|
||||
--hash=sha256:778a11b15673f6f1df23d9586f83c4846c471a8af693a22e066508b77d201ec8 \
|
||||
--hash=sha256:78f7b9e5d6f2fdb88cdde9440dc147259b62b9d3b019924def9f6478be254ac1 \
|
||||
--hash=sha256:799345ab092bee59f01a915620b5d014698547afd011e691a208637312db9186 \
|
||||
--hash=sha256:7bf6cdf8e07c8151fba6fe85735441240ec7f619f935a5205953d58009aef8c6 \
|
||||
--hash=sha256:8009897cdef112072f93a0efdce29cd819e717fd2f649ee3016efd3cd885a7ed \
|
||||
--hash=sha256:80f85f0a7cc86e7a54c46d99c9e1318ff01f4687c172ede30fd52d19d1da1c8e \
|
||||
--hash=sha256:8585e3bb2cdea02fc88ffa245069c36555557ad3609e83be0ec71f54fd4abb52 \
|
||||
--hash=sha256:878be833caa6a3821caf85eb39c5ba92d28e85df26d57afb06b35b2efd937231 \
|
||||
--hash=sha256:8a76ea0f0b9dfa06f254ee06053d93a600865b3274358ca48a352ce4f0798450 \
|
||||
--hash=sha256:8b7b94a067d1c504ee0b16def57ad5738701e4ba10cec90529f13fa03c833496 \
|
||||
--hash=sha256:8d92f1a84bb12d9e56f818b3a746f3efba93c1b63c8387a73dde655e1e42282a \
|
||||
--hash=sha256:908bd3f6439f2fef9e85031b59fd4f1297af54415fb60e4254a95f75b3cab3f3 \
|
||||
--hash=sha256:92db2bf818d5cc8d9c1f1fc56b897662e24ea5adb36ad1f1d82875bd64e03c24 \
|
||||
--hash=sha256:940d4a017dbfed9daf46a3b086e1d2167e7012ee297fef9e1c545c4d022f5178 \
|
||||
--hash=sha256:957e7c38f250991e48a9a73e6423db1bb9dd14e722a10f6b8bb8e16a0f55f695 \
|
||||
--hash=sha256:96153e77a591c8adc2ee805756c61f59fef4cf4073a9275ee86fe8cba41241f7 \
|
||||
--hash=sha256:96f423a119f4777a4a056b66ce11527366a8bb92f54e541ade21f2374433f6d4 \
|
||||
--hash=sha256:97260ff46b207a82a7567b581ab4190bd4dfa09f4db8a8b49d1a958f6aa4940e \
|
||||
--hash=sha256:974b28cf63cc99dfb2188d8d222bc6843656188164848c4f679e63dae4b0708e \
|
||||
--hash=sha256:9ff15928d62a0b80bb875655c39bf517938c7d589554cbd2669be42d97c2cb61 \
|
||||
--hash=sha256:a6483e309ca809f1efd154b4d37dc6d9f61037d6c6a81c2dc7a15cb22c8c5dca \
|
||||
--hash=sha256:a88f062f072d1589b7b46e951698950e7da00442fc1cacbe17e19e025dc327ad \
|
||||
--hash=sha256:ac913f8403b36a2c8610bbfd25b8013488533e71e62b4b4adce9c86c8cea905b \
|
||||
--hash=sha256:adbeebaebae3526afc3c96fad434367cafbfd1b25d72369a9e5858453b1bb71a \
|
||||
--hash=sha256:b2a095d45c5d46e5e79ba1e5b9cb787f541a8dee0433836cea4b96a2c439dcd8 \
|
||||
--hash=sha256:b3210649ee28062ea6099cfda39e147fa1bc039583c8ee4481cb7811e2448c51 \
|
||||
--hash=sha256:b37f6d31b3dcea7deb5e9696e529a6aa4a898adc33db82da12e4c60a7c4d2011 \
|
||||
--hash=sha256:b4dec9482a65c54a5044486847b8a66bf10c9cb4926d42927ec4e8fd5db7fed8 \
|
||||
--hash=sha256:b4f3b365f31c6cd4af24545ca0a244a53688cad8834e32f56831c4923b50a103 \
|
||||
--hash=sha256:b6db2185db9be0a04fecf2f241c70b63b1a242e2805be291855078f2b404dd6b \
|
||||
--hash=sha256:b9be22a69a014bc47e78072d0ecae716f5eb56c15238acca0f43d6eb8e4a5bda \
|
||||
--hash=sha256:bac9c42ba2ac65ddc115d930c78d24ab8d4f465fd3fc473cdedfccadb9429806 \
|
||||
--hash=sha256:bf0a7e10b077bf5fb9380ad3ae8ce20ef919a6ad93b4552896419ac7e1d8e042 \
|
||||
--hash=sha256:c23c3ff005322a6e16f71bf8692fcf4d5a304aaafe1e262c98c6d4adc7be863e \
|
||||
--hash=sha256:c4c800524c9cd9bac5166cd6f55285957fcfc907db323e193f2afcd4d9abd69b \
|
||||
--hash=sha256:c7366fe1418a6133d5aa824ee53d406550110984de7637d65a178010f759c6ef \
|
||||
--hash=sha256:c8d1634419f39ea6f5c427ea2f90ca85126b54b50837f31497f3bf38266e853d \
|
||||
--hash=sha256:c9a63152fe95756b85f31186bddf42e4c02c6321207fd6601a1c89ebac4fe567 \
|
||||
--hash=sha256:cb89a7f2de3602cfed448095bab3f178399646ab7c61454315089787df07733a \
|
||||
--hash=sha256:cba69cb73723c3f329622e34bdbf5ce1f80c21c290ff04256cff1cd3c2036ed2 \
|
||||
--hash=sha256:cee686f1f4cadeb2136007ddedd0aaf928ab95216e7691c63e50a8ec066336d0 \
|
||||
--hash=sha256:cf253e0e1c3ceb4aaff6df637ce033ff6535fb8c70a764a8f46aafd3d6ab798e \
|
||||
--hash=sha256:d1eaff1d00c7751b7c6662e9c5ba6eb2c17a2306ba5e2a37f24ddf3cc953402b \
|
||||
--hash=sha256:d3bb933317c52d7ea5004a1c442eef86f426886fba134ef8cf4226ea6ee1821d \
|
||||
--hash=sha256:d4d3214a0f8394edfa3e303136d0575eece0745ff2b47bd2cb2e66dd92d4351a \
|
||||
--hash=sha256:d6a5df73acd3399d893dafc71663ad22534b5aa4f94e8a2fabfe856c3c1b6a52 \
|
||||
--hash=sha256:d8b7138e5cd0647e4523d6685b0eac5d4be9a184ae9634492f25c6eb38c12a47 \
|
||||
--hash=sha256:db1e72ede2d0d7ccb213f218df6a078a9c09a7de257c2fe8fcef16d5925230b1 \
|
||||
--hash=sha256:e25ac20a2ef37e91c1b39938b591457666a0fa835c7783c3a8f33ea42870db94 \
|
||||
--hash=sha256:e2de870d16a7a53901e41b64ffdf26f2fbb8917b3e6ebf398098d72c5b20bd7f \
|
||||
--hash=sha256:e4a3408834f65da56c83528fb52ce7911484f0d1eaf7b761fc66001db1646eff \
|
||||
--hash=sha256:eaa352d7047a31d87dafcacbabe89df0aa506abb5b1b85a2fb91bc3faa02d822 \
|
||||
--hash=sha256:eab8145831a0d56ec9c4139b6c3e594c7a83c2c8be25d5bcf2d86136a532287a \
|
||||
--hash=sha256:ec3cc8c5d4084591b4237c0a272cc4f50a5b03396a47d9caaf76f5d7b38a4f11 \
|
||||
--hash=sha256:edee74874ce20a373d62dc28b0b18b93f645633c2943fd90ee9d898550770581 \
|
||||
--hash=sha256:eefdba20de0d938cec6a89bd4d70f346a03108a19b9df4248d3cf0d88f1b0f51 \
|
||||
--hash=sha256:ef2b7b394f208233e471abc541cc6991f907ffd47dc72584acee3147899d6565 \
|
||||
--hash=sha256:f21f00a91358803399890ab167098c131ec2ddd5f8f5fd5fe9c9f2c6fcd91e40 \
|
||||
--hash=sha256:f4be2e3d8bc8aabd566f8d5b8ba7ecc09249d74ba3c9ed52e54dc23a293f0b92 \
|
||||
--hash=sha256:f57fb59d9f385710aa7060e89410aeb5058b99e62f4d16b08b91986b9a2140c2 \
|
||||
--hash=sha256:f6292f1de555ffcc675941d65fffffb0a5bcd992905015f85d0592201793e0e5 \
|
||||
--hash=sha256:f833670942247a14eafbb675458b4e61c82e002a148f49e68257b79296e865c4 \
|
||||
--hash=sha256:fa47e444b8ba08fffd1c18e8cdb9a75db1b6a27f17507522834ad13ed5922b93 \
|
||||
--hash=sha256:fb30f9626572a76dfe4293c7194a09fb1fe93ba94c7d4f720dfae3b646b45027 \
|
||||
--hash=sha256:fe3c58d2f5db5fbd18c2987cba06d51b0529f52bc3a6cdc33d3f4eab725104bd
|
||||
# via
|
||||
# aiohttp
|
||||
# aiosignal
|
||||
idna==3.18 \
|
||||
--hash=sha256:7f952cbe720b688055e3f87de14f5c3e5fdaa8bc3928985c4077ca689de849a2 \
|
||||
--hash=sha256:ffb385a7e039654cef1ab9ef32c6fafe283c0c0467bba1d9029738ce4a14a848
|
||||
# via yarl
|
||||
multidict==6.7.1 \
|
||||
--hash=sha256:026d264228bcd637d4e060844e39cdc60f86c479e463d49075dedc21b18fbbe0 \
|
||||
--hash=sha256:03ede2a6ffbe8ef936b92cb4529f27f42be7f56afcdab5ab739cd5f27fb1cbf9 \
|
||||
--hash=sha256:0458c978acd8e6ea53c81eefaddbbee9c6c5e591f41b3f5e8e194780fe026581 \
|
||||
--hash=sha256:067343c68cd6612d375710f895337b3a98a033c94f14b9a99eff902f205424e2 \
|
||||
--hash=sha256:08ccb2a6dc72009093ebe7f3f073e5ec5964cba9a706fa94b1a1484039b87941 \
|
||||
--hash=sha256:0b38ebffd9be37c1170d33bc0f36f4f262e0a09bc1aac1c34c7aa51a7293f0b3 \
|
||||
--hash=sha256:0b4c48648d7649c9335cf1927a8b87fa692de3dcb15faa676c6a6f1f1aabda43 \
|
||||
--hash=sha256:0d17522c37d03e85c8098ec8431636309b2682cf12e58f4dbc76121fb50e4962 \
|
||||
--hash=sha256:0e161ddf326db5577c3a4cc2d8648f81456e8a20d40415541587a71620d7a7d1 \
|
||||
--hash=sha256:0e697826df7eb63418ee190fd06ce9f1803593bb4b9517d08c60d9b9a7f69d8f \
|
||||
--hash=sha256:10ae39c9cfe6adedcdb764f5e8411d4a92b055e35573a2eaa88d3323289ef93c \
|
||||
--hash=sha256:121a34e5bfa410cdf2c8c49716de160de3b1dbcd86b49656f5681e4543bcd1a8 \
|
||||
--hash=sha256:128441d052254f42989ef98b7b6a6ecb1e6f708aa962c7984235316db59f50fa \
|
||||
--hash=sha256:12fad252f8b267cc75b66e8fc51b3079604e8d43a75428ffe193cd9e2195dfd6 \
|
||||
--hash=sha256:14525a5f61d7d0c94b368a42cff4c9a4e7ba2d52e2672a7b23d84dc86fb02b0c \
|
||||
--hash=sha256:17207077e29342fdc2c9a82e4b306f1127bf1ea91f8b71e02d4798a70bb99991 \
|
||||
--hash=sha256:17307b22c217b4cf05033dabefe68255a534d637c6c9b0cc8382718f87be4262 \
|
||||
--hash=sha256:1b99af4d9eec0b49927b4402bcbb58dea89d3e0db8806a4086117019939ad3dd \
|
||||
--hash=sha256:1d540e51b7e8e170174555edecddbd5538105443754539193e3e1061864d444d \
|
||||
--hash=sha256:1e3a8bb24342a8201d178c3b4984c26ba81a577c80d4d525727427460a50c22d \
|
||||
--hash=sha256:1fa6609d0364f4f6f58351b4659a1f3e0e898ba2a8c5cac04cb2c7bc556b0bc5 \
|
||||
--hash=sha256:21f830fe223215dffd51f538e78c172ed7c7f60c9b96a2bf05c4848ad49921c3 \
|
||||
--hash=sha256:233b398c29d3f1b9676b4b6f75c518a06fcb2ea0b925119fb2c1bc35c05e1601 \
|
||||
--hash=sha256:24c0cf81544ca5e17cfcb6e482e7a82cd475925242b308b890c9452a074d4505 \
|
||||
--hash=sha256:25167cc263257660290fba06b9318d2026e3c910be240a146e1f66dd114af2b0 \
|
||||
--hash=sha256:253282d70d67885a15c8a7716f3a73edf2d635793ceda8173b9ecc21f2fb8292 \
|
||||
--hash=sha256:273d23f4b40f3dce4d6c8a821c741a86dec62cded82e1175ba3d99be128147ed \
|
||||
--hash=sha256:283ddac99f7ac25a4acadbf004cb5ae34480bbeb063520f70ce397b281859362 \
|
||||
--hash=sha256:28ca5ce2fd9716631133d0e9a9b9a745ad7f60bac2bccafb56aa380fc0b6c511 \
|
||||
--hash=sha256:2b41f5fed0ed563624f1c17630cb9941cf2309d4df00e494b551b5f3e3d67a23 \
|
||||
--hash=sha256:2bbd113e0d4af5db41d5ebfe9ccaff89de2120578164f86a5d17d5a576d1e5b2 \
|
||||
--hash=sha256:2e1425e2f99ec5bd36c15a01b690a1a2456209c5deed58f95469ffb46039ccbb \
|
||||
--hash=sha256:2e2d2ed645ea29f31c4c7ea1552fcfd7cb7ba656e1eafd4134a6620c9f5fdd9e \
|
||||
--hash=sha256:3758692429e4e32f1ba0df23219cd0b4fc0a52f476726fff9337d1a57676a582 \
|
||||
--hash=sha256:38fb49540705369bab8484db0689d86c0a33a0a9f2c1b197f506b71b4b6c19b0 \
|
||||
--hash=sha256:3943debf0fbb57bdde5901695c11094a9a36723e5c03875f87718ee15ca2f4d2 \
|
||||
--hash=sha256:398c1478926eca669f2fd6a5856b6de9c0acf23a2cb59a14c0ba5844fa38077e \
|
||||
--hash=sha256:3ab8b9d8b75aef9df299595d5388b14530839f6422333357af1339443cff777d \
|
||||
--hash=sha256:3bd231490fa7217cc832528e1cd8752a96f0125ddd2b5749390f7c3ec8721b65 \
|
||||
--hash=sha256:3d51ff4785d58d3f6c91bdbffcb5e1f7ddfda557727043aa20d20ec4f65e324a \
|
||||
--hash=sha256:3fccb473e87eaa1382689053e4a4618e7ba7b9b9b8d6adf2027ee474597128cd \
|
||||
--hash=sha256:401c5a650f3add2472d1d288c26deebc540f99e2fb83e9525007a74cd2116f1d \
|
||||
--hash=sha256:41f2952231456154ee479651491e94118229844dd7226541788be783be2b5108 \
|
||||
--hash=sha256:432feb25a1cb67fe82a9680b4d65fb542e4635cb3166cd9c01560651ad60f177 \
|
||||
--hash=sha256:439cbebd499f92e9aa6793016a8acaa161dfa749ae86d20960189f5398a19144 \
|
||||
--hash=sha256:4885cb0e817aef5d00a2e8451d4665c1808378dc27c2705f1bf4ef8505c0d2e5 \
|
||||
--hash=sha256:497394b3239fc6f0e13a78a3e1b61296e72bf1c5f94b4c4eb80b265c37a131cd \
|
||||
--hash=sha256:497bde6223c212ba11d462853cfa4f0ae6ef97465033e7dc9940cdb3ab5b48e5 \
|
||||
--hash=sha256:4cfb48c6ea66c83bcaaf7e4dfa7ec1b6bbcf751b7db85a328902796dfde4c060 \
|
||||
--hash=sha256:538cec1e18c067d0e6103aa9a74f9e832904c957adc260e61cd9d8cf0c3b3d37 \
|
||||
--hash=sha256:55d97cc6dae627efa6a6e548885712d4864b81110ac76fa4e534c03819fa4a56 \
|
||||
--hash=sha256:563fe25c678aaba333d5399408f5ec3c383ca5b663e7f774dd179a520b8144df \
|
||||
--hash=sha256:57b46b24b5d5ebcc978da4ec23a819a9402b4228b8a90d9c656422b4bdd8a963 \
|
||||
--hash=sha256:5884a04f4ff56c6120f6ccf703bdeb8b5079d808ba604d4d53aec0d55dc33568 \
|
||||
--hash=sha256:59bc83d3f66b41dac1e7460aac1d196edc70c9ba3094965c467715a70ecb46db \
|
||||
--hash=sha256:5a37ca18e360377cfda1d62f5f382ff41f2b8c4ccb329ed974cc2e1643440118 \
|
||||
--hash=sha256:5c4b9bfc148f5a91be9244d6264c53035c8a0dcd2f51f1c3c6e30e30ebaa1c84 \
|
||||
--hash=sha256:5e01429a929600e7dab7b166062d9bb54a5eed752384c7384c968c2afab8f50f \
|
||||
--hash=sha256:5fa6a95dfee63893d80a34758cd0e0c118a30b8dcb46372bf75106c591b77889 \
|
||||
--hash=sha256:619e5a1ac57986dbfec9f0b301d865dddf763696435e2962f6d9cf2fdff2bb71 \
|
||||
--hash=sha256:65573858d27cdeaca41893185677dc82395159aa28875a8867af66532d413a8f \
|
||||
--hash=sha256:6704fa2b7453b2fb121740555fa1ee20cd98c4d011120caf4d2b8d4e7c76eec0 \
|
||||
--hash=sha256:6aac4f16b472d5b7dc6f66a0d49dd57b0e0902090be16594dc9ebfd3d17c47e7 \
|
||||
--hash=sha256:6b10359683bd8806a200fd2909e7c8ca3a7b24ec1d8132e483d58e791d881048 \
|
||||
--hash=sha256:6b83cabdc375ffaaa15edd97eb7c0c672ad788e2687004990074d7d6c9b140c8 \
|
||||
--hash=sha256:6d3bc717b6fe763b8be3f2bee2701d3c8eb1b2a8ae9f60910f1b2860c82b6c49 \
|
||||
--hash=sha256:6f77ce314a29263e67adadc7e7c1bc699fcb3a305059ab973d038f87caa42ed0 \
|
||||
--hash=sha256:749aa54f578f2e5f439538706a475aa844bfa8ef75854b1401e6e528e4937cf9 \
|
||||
--hash=sha256:7a7e590ff876a3eaf1c02a4dfe0724b6e69a9e9de6d8f556816f29c496046e59 \
|
||||
--hash=sha256:7dfb78d966b2c906ae1d28ccf6e6712a3cd04407ee5088cd276fe8cb42186190 \
|
||||
--hash=sha256:7eee46ccb30ff48a1e35bb818cc90846c6be2b68240e42a78599166722cea709 \
|
||||
--hash=sha256:7ff981b266af91d7b4b3793ca3382e53229088d193a85dfad6f5f4c27fc73e5d \
|
||||
--hash=sha256:841189848ba629c3552035a6a7f5bf3b02eb304e9fea7492ca220a8eda6b0e5c \
|
||||
--hash=sha256:844c5bca0b5444adb44a623fb0a1310c2f4cd41f402126bb269cd44c9b3f3e1e \
|
||||
--hash=sha256:84e61e3af5463c19b67ced91f6c634effb89ef8bfc5ca0267f954451ed4bb6a2 \
|
||||
--hash=sha256:8affcf1c98b82bc901702eb73b6947a1bfa170823c153fe8a47b5f5f02e48e40 \
|
||||
--hash=sha256:8be1802715a8e892c784c0197c2ace276ea52702a0ede98b6310c8f255a5afb3 \
|
||||
--hash=sha256:8f333ec9c5eb1b7105e3b84b53141e66ca05a19a605368c55450b6ba208cb9ee \
|
||||
--hash=sha256:9004d8386d133b7e6135679424c91b0b854d2d164af6ea3f289f8f2761064609 \
|
||||
--hash=sha256:90efbcf47dbe33dcf643a1e400d67d59abeac5db07dc3f27d6bdeae497a2198c \
|
||||
--hash=sha256:935434b9853c7c112eee7ac891bc4cb86455aa631269ae35442cb316790c1445 \
|
||||
--hash=sha256:93b1818e4a6e0930454f0f2af7dfce69307ca03cdcfb3739bf4d91241967b6c1 \
|
||||
--hash=sha256:95922cee9a778659e91db6497596435777bd25ed116701a4c034f8e46544955a \
|
||||
--hash=sha256:960c83bf01a95b12b08fd54324a4eb1d5b52c88932b5cba5d6e712bb3ed12eb5 \
|
||||
--hash=sha256:97231140a50f5d447d3164f994b86a0bed7cd016e2682f8650d6a9158e14fd31 \
|
||||
--hash=sha256:974e72a2474600827abaeda71af0c53d9ebbc3c2eb7da37b37d7829ae31232d8 \
|
||||
--hash=sha256:97891f3b1b3ffbded884e2916cacf3c6fc87b66bb0dde46f7357404750559f33 \
|
||||
--hash=sha256:98655c737850c064a65e006a3df7c997cd3b220be4ec8fe26215760b9697d4d7 \
|
||||
--hash=sha256:98bc624954ec4d2c7cb074b8eefc2b5d0ce7d482e410df446414355d158fe4ca \
|
||||
--hash=sha256:98c5787b0a0d9a41d9311eae44c3b76e6753def8d8870ab501320efe75a6a5f8 \
|
||||
--hash=sha256:9b0d9b91d1aa44db9c1f1ecd0d9d2ae610b2f4f856448664e01a3b35899f3f92 \
|
||||
--hash=sha256:9c90fed18bffc0189ba814749fdcc102b536e83a9f738a9003e569acd540a733 \
|
||||
--hash=sha256:9d624335fd4fa1c08a53f8b4be7676ebde19cd092b3895c421045ca87895b429 \
|
||||
--hash=sha256:9f9af11306994335398293f9958071019e3ab95e9a707dc1383a35613f6abcb9 \
|
||||
--hash=sha256:a0543217a6a017692aa6ae5cc39adb75e587af0f3a82288b1492eb73dd6cc2a4 \
|
||||
--hash=sha256:a088b62bd733e2ad12c50dad01b7d0166c30287c166e137433d3b410add807a6 \
|
||||
--hash=sha256:a407f13c188f804c759fc6a9f88286a565c242a76b27626594c133b82883b5c2 \
|
||||
--hash=sha256:a90f75c956e32891a4eda3639ce6dd86e87105271f43d43442a3aedf3cddf172 \
|
||||
--hash=sha256:a9fc4caa29e2e6ae408d1c450ac8bf19892c5fca83ee634ecd88a53332c59981 \
|
||||
--hash=sha256:aa23b001d968faef416ff70dc0f1ab045517b9b42a90edd3e9bcdb06479e31d5 \
|
||||
--hash=sha256:ac1c665bad8b5d762f5f85ebe4d94130c26965f11de70c708c75671297c776de \
|
||||
--hash=sha256:af959b9beeb66c822380f222f0e0a1889331597e81f1ded7f374f3ecb0fd6c52 \
|
||||
--hash=sha256:b0fa96985700739c4c7853a43c0b3e169360d6855780021bfc6d0f1ce7c123e7 \
|
||||
--hash=sha256:b26684587228afed0d50cf804cc71062cc9c1cdf55051c4c6345d372947b268c \
|
||||
--hash=sha256:b4938326284c4f1224178a560987b6cf8b4d38458b113d9b8c1db1a836e640a2 \
|
||||
--hash=sha256:b8c990b037d2fff2f4e33d3f21b9b531c5745b33a49a7d6dbe7a177266af44f6 \
|
||||
--hash=sha256:ba0a9fb644d0c1a2194cf7ffb043bd852cea63a57f66fbd33959f7dae18517bf \
|
||||
--hash=sha256:bb08271280173720e9fea9ede98e5231defcbad90f1624bea26f32ec8a956e2f \
|
||||
--hash=sha256:bdbf9f3b332abd0cdb306e7c2113818ab1e922dc84b8f8fd06ec89ed2a19ab8b \
|
||||
--hash=sha256:bfde23ef6ed9db7eaee6c37dcec08524cb43903c60b285b172b6c094711b3961 \
|
||||
--hash=sha256:c0abd12629b0af3cf590982c0b413b1e7395cd4ec026f30986818ab95bfaa94a \
|
||||
--hash=sha256:c102791b1c4f3ab36ce4101154549105a53dc828f016356b3e3bcae2e3a039d3 \
|
||||
--hash=sha256:c3a32d23520ee37bf327d1e1a656fec76a2edd5c038bf43eddfa0572ec49c60b \
|
||||
--hash=sha256:c524c6fb8fc342793708ab111c4dbc90ff9abd568de220432500e47e990c0358 \
|
||||
--hash=sha256:c5f0c21549ab432b57dcc82130f388d84ad8179824cc3f223d5e7cfbfd4143f6 \
|
||||
--hash=sha256:c6b3228e1d80af737b72925ce5fb4daf5a335e49cd7ab77ed7b9fdfbf58c526e \
|
||||
--hash=sha256:c76c4bec1538375dad9d452d246ca5368ad6e1c9039dadcf007ae59c70619ea1 \
|
||||
--hash=sha256:c9035dde0f916702850ef66460bc4239d89d08df4d02023a5926e7446724212c \
|
||||
--hash=sha256:c93c3db7ea657dd4637d57e74ab73de31bccefe144d3d4ce370052035bc85fb5 \
|
||||
--hash=sha256:cb2a55f408c3043e42b40cc8eecd575afa27b7e0b956dfb190de0f8499a57a53 \
|
||||
--hash=sha256:cdea2e7b2456cfb6694fb113066fd0ec7ea4d67e3a35e1f4cbeea0b448bf5872 \
|
||||
--hash=sha256:ce1bbd7d780bb5a0da032e095c951f7014d6b0a205f8318308140f1a6aba159e \
|
||||
--hash=sha256:cf37cbe5ced48d417ba045aca1b21bafca67489452debcde94778a576666a1df \
|
||||
--hash=sha256:d4f49cb5661344764e4c7c7973e92a47a59b8fc19b6523649ec9dc4960e58a03 \
|
||||
--hash=sha256:d54ecf9f301853f2c5e802da559604b3e95bb7a3b01a9c295c6ee591b9882de8 \
|
||||
--hash=sha256:d62b7f64ffde3b99d06b707a280db04fb3855b55f5a06df387236051d0668f4a \
|
||||
--hash=sha256:d82dd730a95e6643802f4454b8fdecdf08667881a9c5670db85bc5a56693f122 \
|
||||
--hash=sha256:da62917e6076f512daccfbbde27f46fed1c98fee202f0559adec8ee0de67f71a \
|
||||
--hash=sha256:dd96c01a9dcd4889dcfcf9eb5544ca0c77603f239e3ffab0524ec17aea9a93ee \
|
||||
--hash=sha256:df9f19c28adcb40b6aae30bbaa1478c389efd50c28d541d76760199fc1037c32 \
|
||||
--hash=sha256:e1c5988359516095535c4301af38d8a8838534158f649c05dd1050222321bcb3 \
|
||||
--hash=sha256:e628ef0e6859ffd8273c69412a2465c4be4a9517d07261b33334b5ec6f3c7489 \
|
||||
--hash=sha256:e82d14e3c948952a1a85503817e038cba5905a3352de76b9a465075d072fba23 \
|
||||
--hash=sha256:e954b24433c768ce78ab7929e84ccf3422e46deb45a4dc9f93438f8217fa2d34 \
|
||||
--hash=sha256:eb0ce7b2a32d09892b3dd6cc44877a0d02a33241fafca5f25c8b6b62374f8b75 \
|
||||
--hash=sha256:eb304767bca2bb92fb9c5bd33cedc95baee5bb5f6c88e63706533a1c06ad08c8 \
|
||||
--hash=sha256:eb351f72c26dc9abe338ca7294661aa22969ad8ffe7ef7d5541d19f368dc854a \
|
||||
--hash=sha256:ec6652a1bee61c53a3e5776b6049172c53b6aaba34f18c9ad04f82712bac623d \
|
||||
--hash=sha256:f2a0a924d4c2e9afcd7ec64f9de35fcd96915149b2216e1cb2c10a56df483855 \
|
||||
--hash=sha256:f33dc2a3abe9249ea5d8360f969ec7f4142e7ac45ee7014d8f8d5acddf178b7b \
|
||||
--hash=sha256:f537b55778cd3cbee430abe3131255d3a78202e0f9ea7ffc6ada893a4bcaeea4 \
|
||||
--hash=sha256:f5dd81c45b05518b9aa4da4aa74e1c93d715efa234fd3e8a179df611cc85e5f4 \
|
||||
--hash=sha256:f99fe611c312b3c1c0ace793f92464d8cd263cc3b26b5721950d977b006b6c4d \
|
||||
--hash=sha256:fa263a02f4f2dd2d11a7b1bb4362aa7cb1049f84a9235d31adf63f30143469a0 \
|
||||
--hash=sha256:fc5907494fccf3e7d3f94f95c91d6336b092b5fc83811720fae5e2765890dfba \
|
||||
--hash=sha256:fcee94dfbd638784645b066074b338bc9cc155d4b4bffa4adce1615c5a426c19
|
||||
# via
|
||||
# aiohttp
|
||||
# yarl
|
||||
propcache==0.5.2 \
|
||||
--hash=sha256:01c4fc7480cd0598bb4b57022df55b9ca296da7fc5a8760bd8451a7e63a7d427 \
|
||||
--hash=sha256:04dc2390d9edbbaef7461f33322555976ffddf0b650a038649d026358714e6c5 \
|
||||
--hash=sha256:06187263ddad280d05b4d8a8b3bb7d164cbebd469236544a42e6d9b28ac6a4fa \
|
||||
--hash=sha256:0958834041a0166d343b8d2cedcd8bcbaeb4fdbe0cf08320c5379f143c3be6e7 \
|
||||
--hash=sha256:099aaf4b4d1a02265b92a977edf00b5c4f63b3b17ac6de39b0d637c9cac0188a \
|
||||
--hash=sha256:0d2c9bf8528f135dbb805ce027567e09164f7efa51a2be07458a2c0420f292d0 \
|
||||
--hash=sha256:0fd59b5af35f74da48d905dcbad55449ba13be91823cb05a9bd590bbf5b61660 \
|
||||
--hash=sha256:10734b5484ea113152ee25a91dccedf81631791805d2c9ccb054958e51842c94 \
|
||||
--hash=sha256:13fef48778b5a2a756523fdb781326b028ca75e32858b04f2cdd19f394564917 \
|
||||
--hash=sha256:178b4a2cdaac1818e2bf1c5a99b94383fa73ea5382e032a48dec07dc5668dc42 \
|
||||
--hash=sha256:196913dea116aeb5a2ba95af4ddcb7ea85559ae07d8eee8751688310d09168c3 \
|
||||
--hash=sha256:1b31822f4474c4036bae62de9402710051d431a606d6a0f907fec79935a071aa \
|
||||
--hash=sha256:1ca071adabaab6e9219924bbe00af821f1ee7de113a9eca1cdc292de3d120f4d \
|
||||
--hash=sha256:1d1ad32d9d4355e2be65574fd0bfd3677e7066b009cd5b9b2dee8aa6a6393b33 \
|
||||
--hash=sha256:1dbcf7675229b35d31abb6547d8ebc8c27a830ac3f9a794edff6254873ec7c0a \
|
||||
--hash=sha256:2293949b855ce597f2826452d17c2d545fb5622379c4ea6fdf525e9b8e8a2511 \
|
||||
--hash=sha256:26a4dca084132874e639895c3135dfad5eb20bae209f62d1aeb31b03e601c3c0 \
|
||||
--hash=sha256:2800a4a8ead6b28cccd1ec54b59346f0def7922ee1c7598e8499c733cfbb7c84 \
|
||||
--hash=sha256:29cbaac5ea0212663e6845e04b5e188d5a6ae6dd919810ac835bf1d3b42c3f4c \
|
||||
--hash=sha256:29f9309a2e42b0d273be006fdb4be2d6c39a47f6f57d8fb1cf9f81481df81b66 \
|
||||
--hash=sha256:2d7aa89ebca5acc98cba9d1472d976e394782f587bad6661003602a619fd1821 \
|
||||
--hash=sha256:2f22cbbac9e26a8e864c0985ff1268d5d939d53d9d9411a9824279097e03a2cb \
|
||||
--hash=sha256:2f8ea531c794b9d6274acd4e8d2c2ebcac590a4361d27482edd3010b79f1325e \
|
||||
--hash=sha256:3115559b8effafd63b142ea5ed53d63a16ea6469cbc63dce4ee194b42db5d853 \
|
||||
--hash=sha256:32775082acd2d807ee3db715c7770d38767b817870acfa08c29e057f3c4d5b56 \
|
||||
--hash=sha256:3430bb2bfe1331885c427745a751e774ee679fd4344f80b97bf879815fe8fa55 \
|
||||
--hash=sha256:3b199b9b2b3d6a7edf3183ba8a9a137a22b97f7df525feb5ae1eccf026d2a9c6 \
|
||||
--hash=sha256:40314bca9ac559716fe374094fc81c11dcc34b64fd6c585360f5775690505704 \
|
||||
--hash=sha256:44e488ef40dbb452700b2b1f8188934121f6648f52c295055662d2191959ff82 \
|
||||
--hash=sha256:452b5065457eb9991ec5eb38ff41d6cd4c991c9ac7c531c4d5849ae473a9a13f \
|
||||
--hash=sha256:45f11346f884bc47444f6e6647131055844134c3175b629f84952e2b5cd62b64 \
|
||||
--hash=sha256:46088abff4cba581dea21ae0467a480526cb25aa5f3c269e909f800328bc3999 \
|
||||
--hash=sha256:4621064bbf28fa77ff64dd5d94367c04684c67d3a5bf1dff25f0cd0d98a38f3b \
|
||||
--hash=sha256:4bc8ff1feffc6a61c7002ffe84634c41b822e104990ae009f44a0834430070bb \
|
||||
--hash=sha256:4db0ba63d693afd40d249bd93f842b5f144f8fcbb83de05660373bcf30517b1d \
|
||||
--hash=sha256:51f96d685ab16e88cab128cd37a52c5da540809c8b879fa047731bfcb4ad35a4 \
|
||||
--hash=sha256:54adaa85a22078d1e306304a40984dc5be99d599bf3dc0a24dc98f7daeab89ab \
|
||||
--hash=sha256:552ffadf6ad409844bc5919c42a0a83d88314cedddaea0e41e80a8b8fffe881f \
|
||||
--hash=sha256:5538d2c13d93e4698af7e092b57bc7298fd35d1d58e656ae18f23ee0d0378e03 \
|
||||
--hash=sha256:5570dbcc97571c15f68068e529c92715a12f8d54030e272d264b377e22bd17a5 \
|
||||
--hash=sha256:5671d09a36b06d0fd4a3da0fccbcae360e9b1570924171a15e9e0997f0249fba \
|
||||
--hash=sha256:583c19759d9eec1e5b69e2fbef36a7d9c326041be9746cb822d335c8cedc2979 \
|
||||
--hash=sha256:5aaa2b923c1944ac8febd6609cb373540a5563e7cbcb0fd770f75dace2eb817b \
|
||||
--hash=sha256:5dbc581d2814337da56222fab8dc5f161cd798a434e49bac27930aaef798e144 \
|
||||
--hash=sha256:5fcb98e7598b1ee0addab320d90f65b530297a867dbfe9de52ea838077e16e3d \
|
||||
--hash=sha256:6041d31504dc1779d700e1edcfb08eea334b357620b06681a4eabb57a74e574e \
|
||||
--hash=sha256:66ea454f095ddf5b6b14f56c064c0941c4788be11e18d2464cf643bf7203ff67 \
|
||||
--hash=sha256:68ce1c44c7a813a7f71ea04315a8c7b330b63db99d059a797a4651bb6f69f117 \
|
||||
--hash=sha256:6a997d0489e9668a384fcfd5061b857aa5361de73191cac204d04b889cfbbafa \
|
||||
--hash=sha256:6bf3be92233808fcd338eba0fb4d0b59ec5772af4f4ecfcec450d1bfc0f8b5eb \
|
||||
--hash=sha256:6de8bd93ddde9b992cf2b2e0d796d501a19026b5b9fd87356d7d0779531a8d96 \
|
||||
--hash=sha256:6e7b8719005dd1175be4ab1cd25e9b98659a5e0347331506ec6760d2773a7fb5 \
|
||||
--hash=sha256:6f328175a2cde1f0ff2c4ed8ce968b9dcfb55f3a7153f39e2957ed994da13476 \
|
||||
--hash=sha256:72d61e16dd78228b58c5d47be830ff3da7e5f139abdf0aef9d86cde1c5cf2191 \
|
||||
--hash=sha256:74b70780220e2dd89175ca24b81b68b67c83db499ae611e7f2313cb329801c78 \
|
||||
--hash=sha256:79aa3ff0a9b566633b642fa9caf7e21ed1c13d6feca718187873f199e1514078 \
|
||||
--hash=sha256:7afa37062e6650640e932e4cc9297d81f9f42d9944029cc386b8247dea4da837 \
|
||||
--hash=sha256:80168e2ebe4d3ec6599d10ad8f520304ae1cad9b6c5a95372aef1b66b7bfb53a \
|
||||
--hash=sha256:806719138ecd720339a12410fb9614ac9b2b2d3a5fdf8235d56981c36f4039ba \
|
||||
--hash=sha256:8114f28879e0904748e831c3a7774261bd9e75f49be089f389a76f959dcd13fe \
|
||||
--hash=sha256:81e3a30b0bb60caa22033dd0f8a3618d1d67356212514f62c57db75cb0ef410c \
|
||||
--hash=sha256:823581fd5cb08b12a48bfa11fe962a7916766b6170c17b028fbdf762b85eb9bf \
|
||||
--hash=sha256:85341b12b9d55bad0bded24cac341bb34289469e03a11f3f583ea1cc1db0326c \
|
||||
--hash=sha256:857187f381f88c8e2fa2fe56ab94879d011b883d5a2ee5a1b60a8cd2a06846d9 \
|
||||
--hash=sha256:8a90efd5777e996e42d568db9ac740b944d691e565cbfd31b2f7832f9184b2b8 \
|
||||
--hash=sha256:8b73ab70f1a3351fbc71f663b3e645af6dd0329100c353081cf69c37433fc6fe \
|
||||
--hash=sha256:8c7972d8f193740d9175f0998ab38717e6cd322d5935c5b0fef8c0d323fd9031 \
|
||||
--hash=sha256:8e778ebd44ef4f66ed60a0416b06b489687db264a9c0b3620362f26489492913 \
|
||||
--hash=sha256:9282fb1a3bccd038da9f768b927b24a0c753e466c086b7c4f3c6982851eefb2d \
|
||||
--hash=sha256:949c91d1a990cf3b2e8188dfcfb25005e0b834a06c63fa4ef9f360878ce21ecf \
|
||||
--hash=sha256:95f1e3f4760d404b13c9976c0229b2b49a3c8e2c62a9ce92efdd2b11ada75e3f \
|
||||
--hash=sha256:97797ebb098e670a2f92dd66f32897e30d7615b14e7f59711de23e30a9072539 \
|
||||
--hash=sha256:a0e399a2eccb91ed18721f86aa85757727400b6865c89e88934781deb9c8498b \
|
||||
--hash=sha256:a473b3440261e0c60706e732b2ed2f517857344fc21bf48fdfe211e2d98eb285 \
|
||||
--hash=sha256:a4840ab0ae0216d952f4b53dc6d0b992bfc2bedbfe360bdd9b548bc184c08959 \
|
||||
--hash=sha256:a592f5f3da71c8691c788c13cb6734b6d17663d2e1cb8caddf0673d01ef8847d \
|
||||
--hash=sha256:a6ae2198be502c10f09b2516e7b5d019816924bc3183a43ce792a7bd6625e6f4 \
|
||||
--hash=sha256:a6ddc6ac9e25de626c1f129c1b467d7ecd33ce2237d3fd0c4e429feef0a7ee1f \
|
||||
--hash=sha256:acd2c8edba48e31e58a363b8cf4e5c7db3b04b3f9e371f601df30d9b0d244836 \
|
||||
--hash=sha256:b05d643f944a8c3c4bd86d65ffd87bf3264b617f87791940302bc474d2ff5274 \
|
||||
--hash=sha256:b96db7141a592cbc968daf1feea83a118e6ab378af4abbc72b248c895414c22d \
|
||||
--hash=sha256:ba338430e87ceb9c8f0cf754de38a9860560261e56c00376debd628698a7364f \
|
||||
--hash=sha256:ba57fffe4ac99c5d30076161b5866336d97600769bad35cc68f7774b15298a4e \
|
||||
--hash=sha256:be1ddfcbb376e3de5d2e2db1d58d6d67463e6b4f9f040c000de8e300295465fe \
|
||||
--hash=sha256:c0cb9ed24c8964e172768d455a38254c2dd8a552905729ce006cad3d3dda59b1 \
|
||||
--hash=sha256:c60462af8e6dc30c35407c7237ea908d777b22862bbee27bc4699c0d8bcdc45a \
|
||||
--hash=sha256:c66afea89b1e43725731d2004732a046fe6fe955d51f952c3e95a7314a284a39 \
|
||||
--hash=sha256:c6844ba6364fb12f403928a82cfd295ab103a2b315c77c747b2dbe4a41894ea7 \
|
||||
--hash=sha256:c80f4ba3e8f00189165999a742ee526ebeccedf6c3f7beb0c7df821e9772435a \
|
||||
--hash=sha256:cafca7e56c12bb02ae16d283742bef25a61122e9dab2b5b3f2ccbe589ce32164 \
|
||||
--hash=sha256:cc1177027eda740fdb152706bd215a3f124e3eea15afc39f2cb9fe351b50619e \
|
||||
--hash=sha256:cc49723e2f60d6b32a0f0b08a3fd6d13203c07f1cd9566cfce0f12a917c967a2 \
|
||||
--hash=sha256:cc6fc3cc62e8501d3ed62894425040d2728ecddb1ed072737a5c70bd537aa9f0 \
|
||||
--hash=sha256:cd416c1de191973c52ff1a12a57446bfc7642797b282d7caf2162d7d1b8aa9a0 \
|
||||
--hash=sha256:cd645f03898405cabe694fb8bc35241e3a9c332ec85627584fe3de201452b335 \
|
||||
--hash=sha256:cef6cea3922890dd6c9654971001fa797b526c16ab5e1e46c05fd6f877be7568 \
|
||||
--hash=sha256:cfa21e036ce1e1db2be04ba3b85d2df1bb1702fa01932d984c5464c665228ff4 \
|
||||
--hash=sha256:d0326e2e5e1f3163fa306c834e48e8d490e5fae607a097a40c0648109b47ba80 \
|
||||
--hash=sha256:d310c013aad2c72f1c3f2f8dd3279d460a858c551f97aeb8c63e4693cca7b4d2 \
|
||||
--hash=sha256:d447bb0b3054be5818458fbb171208b1d9ff11eba14e18ca18b90cbb45767370 \
|
||||
--hash=sha256:d4dc37dec6c6cdad0b57881a5658fd14fbf53e333b1a86cf86559f190e1d9ec4 \
|
||||
--hash=sha256:d5a81be28596d6559f6131ef33e10200de6e17643b3c74ce03f9eb103be6ae8b \
|
||||
--hash=sha256:d9ee8826a7d47863a08ac44e1a5f611a462eefc3a194b492da242128bec75b42 \
|
||||
--hash=sha256:db2b80ea58eab4f86b2beec3cc8b39e8ff9276ac20e96b7cce43c8ae84cd6b5a \
|
||||
--hash=sha256:decfca4c79dd53ebab484b00cc4b6717d8c369f86e74aa4ca395a64ac651495e \
|
||||
--hash=sha256:dfed59d0a5aeb01e242e66ff0300bc4a265a7c05f612d30016f0b60b1017d757 \
|
||||
--hash=sha256:e00820e192c8dbebcafb383ebbf99030895f09905e7a0eb2e0340a0bcc2bc825 \
|
||||
--hash=sha256:e4294d04a94dcab1b3bccd8b66d962dcad411a1d19414b2a41d1445f1de32ad0 \
|
||||
--hash=sha256:e59bc9e66329185b93dab73f210f1a37f81cb40f321501db8017c9aea15dba27 \
|
||||
--hash=sha256:e5cbfac9f61484f7e9f3597775500cd3ebe8274e9b050c38f9525c77c97520bf \
|
||||
--hash=sha256:f064f8d2b59177878b7615df1735cd8fe3462ed6be8c7b217d17a276489c2b7f \
|
||||
--hash=sha256:f156a3529f38063b6dbaf356e15602a7f95f8055b1295a438433a6386f10463d \
|
||||
--hash=sha256:f19bb891234d72535764d703bfed1153cc34f4214d5bd7150aee1eec9e8f4366 \
|
||||
--hash=sha256:f7467da8a9822bf1a55336f877340c5bcbd3c482afc43a99771169f74a26dedc \
|
||||
--hash=sha256:f78abfa8dfc32376fd1aacf597b2f2fbbe0ea751419aee718af5d4f82537ef8c \
|
||||
--hash=sha256:f7eabc04151c78a9f4d5bbb5f1faf571e4defeb4b585e0fe95b60ff2dbe4d3d7 \
|
||||
--hash=sha256:f814362777a9f841adddb200ecdf8f5cb1e5a3c4b7a86378edbd6ccb26edd702 \
|
||||
--hash=sha256:fc299c129490f55f254cd90be0deca4764e36e9a7c08b4aa588479a3bbed3098 \
|
||||
--hash=sha256:fc76378c62a0f04d0cd82fbb1a2cd2d7e28fcb40d5873f28a6c44e388aaa2751 \
|
||||
--hash=sha256:fc88b26f08d634f7bc819a7852e5214f5802641ab8d9fd5326892292eee1993e \
|
||||
--hash=sha256:fe67a3d11cd9b4efabfa45c3d00ffba2b26811442a73a581a94b67c2b5faccf6
|
||||
# via
|
||||
# aiohttp
|
||||
# yarl
|
||||
typing-extensions==4.16.0 ; python_full_version < '3.13' \
|
||||
--hash=sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8 \
|
||||
--hash=sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5
|
||||
# via
|
||||
# aiohttp
|
||||
# aiosignal
|
||||
websockets==17.0.1 \
|
||||
--hash=sha256:02ed63bf26dda9fa27df730a41f6664586c4ee05972c8fb667ce1725b3fd13d3 \
|
||||
--hash=sha256:02f0b037a737d0cb0c33866c97bcd1a0b73170dfbf42d69d8fb86f51002fd5ae \
|
||||
--hash=sha256:038cfad5d5417f8bb09295abe986029a26d22f34bda622ccc79b670efd4dab56 \
|
||||
--hash=sha256:07abc3bd196a48af476a82fd47f3f79a6a3f70937a9f930cef703cfa0c9d83b6 \
|
||||
--hash=sha256:07d78a509c3333f5908c83d7f78144ea68a6c9ec28110f5c54d81d8fcdc262c4 \
|
||||
--hash=sha256:0b52c76b8a870b141b7ca0705289452183ce7a523101954ccfe29a25986a673f \
|
||||
--hash=sha256:10b1587c599fa0f2c89154587c80e0fda98ade6c9fa8c0260a2823fb1800b685 \
|
||||
--hash=sha256:10f461191125c63902ea7394ae9e752b1b5785641850c1d365bb30b0f88bc53f \
|
||||
--hash=sha256:15920057a6b723f84734f0641403bca163a4b176e5af809ee4f0c4a1e75e9fed \
|
||||
--hash=sha256:17ac37716c0244e82c9e384c41653c090b1864c6610224ca3857e7f7b58fce10 \
|
||||
--hash=sha256:18ded646ce98cdd3c0235825b3252f1df55765ba49b616bb10282f758667b4d0 \
|
||||
--hash=sha256:1b363bfd72a52c0658a3154a4cff219f15a474b35a235057d38853bf151acce7 \
|
||||
--hash=sha256:1bdd8c4be420905dd732e00dcd669852d8128cc723efa585a0c0e51adb00a28a \
|
||||
--hash=sha256:1d4cf7e8e5b8b1fa40758ac7524843a00237b124ab217e227542cafcfeb7a946 \
|
||||
--hash=sha256:1df81d174c1561292de9e40b141cafc04f69077272f6c352afe1d743e20810df \
|
||||
--hash=sha256:20a92f78ac8250984ed459faa9ca48c285adbfc0038ddc3fdac6046990a9c9ed \
|
||||
--hash=sha256:22bd00f8bae2bccdb5dbe41e20f58ba44ca9fff0b4b561aaf39099c35da762ed \
|
||||
--hash=sha256:2437d4ca208cc0f246d3a2297ae7474b4ba18261aaf5b9c79c84c031ecf348e1 \
|
||||
--hash=sha256:246927ae9ae06ca0d42a483a4bdb80d4862e1ee5b4cab37c354a5e1ad8356448 \
|
||||
--hash=sha256:2503c7e2a5049a12d5dac917a46d5d52591283a766165b8176bb167560421b38 \
|
||||
--hash=sha256:2604de7228506b13a44a256a9d223943340c0e725af5d367dc068e192b027761 \
|
||||
--hash=sha256:28012a54510fe8301bb893ef143cec30a2780a2d3bc20b7bbdf4379d7a63945d \
|
||||
--hash=sha256:2a855b6dfe21c4d3420be265ae031829ba8ba0be0ea350d9f7c3ef30ae63ebe2 \
|
||||
--hash=sha256:2abb1ba0a5133b7d2ef3c1c9f4b0c1e8a101012dce0b594ab2b2888d9a64820e \
|
||||
--hash=sha256:2b3f3020171202b135ca078e20434977c6b2b02af647130d6980c9e39b9462e3 \
|
||||
--hash=sha256:2bc14b481e05e331811108daa1aeb41a5e237a5564ef2f02ec5a356a0f102f78 \
|
||||
--hash=sha256:2fa2cb465a131c347ba6717a78c887746e73edb1c131d01c982d6ef0d68b82e0 \
|
||||
--hash=sha256:409d93efcaa14f7a99592c5baaef5ec6ca94fba0f5aec1a86f693977c69c9c1c \
|
||||
--hash=sha256:41d6aa06b5ab832aee72fedf47a149535b121ac900b6bb4d3fe14712afac9a79 \
|
||||
--hash=sha256:49266e4488309b38783257293a38298942b9a03aa106fcb45195377a77c0c1e2 \
|
||||
--hash=sha256:4d1d99db29b5444e3982f1ce2ba8a833508ad44b2f1fbd0bd99e81d825c0b461 \
|
||||
--hash=sha256:4d41c0a1d47a478bc432b3b9068097bee1ce0c5b19327ea6f75c2ab34ab1f2fb \
|
||||
--hash=sha256:5033ffe6804dd53afafa7d08e8c3eef2d2431f34d58ca30507a8442dd04a033a \
|
||||
--hash=sha256:53b90c00bc6201ab6695c7ff51a04d0e425514c37515e9eeecd2c1b978ac6c0e \
|
||||
--hash=sha256:54cdcaa56f5d3eafd57058f0fa4a3de93a310b43a3c4699f06efc4c0bd054a5a \
|
||||
--hash=sha256:5508f38c98ac29def9e747b87543b008a58b075df6da70b2cf2e0b47073d33bb \
|
||||
--hash=sha256:55383d8177b3c99fd873ee5db0e0193f4c1dd4a3feaccf1a4a03c1b7cf539cac \
|
||||
--hash=sha256:55b12e47dcee83673a40d07686cfb6f9d6dfc285976ade9463f61d2bef3fad22 \
|
||||
--hash=sha256:5661f868ef191d33dfc6a0cc7c5b3d495f0cc8bb3f8b30d87bda8755c61c95f5 \
|
||||
--hash=sha256:57d2ee9b24b404ce75f3814f92073c0ed88106c950148d2427fe8d25ca254d1f \
|
||||
--hash=sha256:599b03beb77633bffc095334338fad79cafc2b01fbd58953838130a9ae967d7b \
|
||||
--hash=sha256:5baa9bc0dfbae8c507e51c8cf1b6d4628086f7a87bbd3a9952bd5f035451f1cc \
|
||||
--hash=sha256:5f33a649bfcb8312524173cc4bbafa7dbb236e18eee9aa31a1d324ca0ddda28c \
|
||||
--hash=sha256:6740be6d1bab69f08ab52cb15b08f76c143b6fe61c580ba62bd929f3ab7a1d42 \
|
||||
--hash=sha256:6a434e59962a4fb9016bea327e1d14d6cd67670ecfb8942b4f4a0c24036634ce \
|
||||
--hash=sha256:6db9e5bf3649ab506c6ae8a3ac85a00fb1ae3816d75962771b2df8adbc5d40d2 \
|
||||
--hash=sha256:6fd88365da261c53d3e943fb37e0d0721b9cde119f6b2e3fc84369b6ab234d63 \
|
||||
--hash=sha256:7002d5f9e1c3ddd991cdfdbfee18cc8c8b196b2445022892badacd6cb338bbbc \
|
||||
--hash=sha256:70d438268e49f1a4bd096b6b6f7010f3ab48b5db2574dbf7d8c864c46ce7a06a \
|
||||
--hash=sha256:72d7f2a5aeb4e82daa4ee18f125b4277f427033359be5c745ad709608446cc2c \
|
||||
--hash=sha256:733e3cc7171fa1b899edbe725ef9382d0e960657dc1fd933f3281ae910c01dab \
|
||||
--hash=sha256:734d20364dc2cfe03674883cafcf580b6e431c5ce42b476312b9285310230cf9 \
|
||||
--hash=sha256:759adeb5b0c5775b563254ec63b5b79089fc0045b479143a0b1b8c0ebaae1253 \
|
||||
--hash=sha256:769ce7e2acfd9a89f2bed3a9c0da229459516bbc00bd4c9e2ca492c613ae4861 \
|
||||
--hash=sha256:810cb3fb5fa6e447216f4e82d9a85cb8aed0929ae3538153ddfe8a6e3121a58d \
|
||||
--hash=sha256:81ce19c6046ace11da7001781be7317bb1dc389f399af4b2ed962190f76f9add \
|
||||
--hash=sha256:846a4a8b0833e3cad57523d9e3bd50ec8ea05ab9d06c582f82a1340ba096af5f \
|
||||
--hash=sha256:872273e629ca7e3d35f16a2dc6ede84e1d5c831e616b8277de6e4f83114e7c58 \
|
||||
--hash=sha256:8848c207049ad49d318e5f64a3d4d7bb189f8328d0d98e65647788f2a085785c \
|
||||
--hash=sha256:884af729b8ab50486acd94d9768c2b60914bf39b579ebba0a5cb73bfdfd61fd2 \
|
||||
--hash=sha256:8c07f145d0b9e90cbd96035f31fb79199aef4da1872854e36ebeb258e3d57594 \
|
||||
--hash=sha256:8cd3369e42c0246afaf9d669cfc19797e3a49e8c0a639544459c57597108b966 \
|
||||
--hash=sha256:8e387adb0c692c6b5571bdeafc8ac9d1901ea30f10309134780b16ecd35e6605 \
|
||||
--hash=sha256:90246fa9e6cb192a778ce6ce024057ec54317a894db7899c922dcdc1f4cbf6a5 \
|
||||
--hash=sha256:90973a3a00f23afdfd1c9b06fb84289bf0220f247ef8a62501a1967c7af54f7b \
|
||||
--hash=sha256:9493314a99e599163c854fb5900ad7f7ea38c5cb9d9103aa30b3c6b8181c01fa \
|
||||
--hash=sha256:9f7747d3daa41a11f25f7cca5dc988fc51da97b311bed4c9d843860f79779283 \
|
||||
--hash=sha256:a39ce3a7b0e6059be093213d637963101380157bcbad355916738fafb490698d \
|
||||
--hash=sha256:a60fa1a25cca1bcc2bf87b8d6be37a741f0a3239fb5e9cfb7a37173b68ffcf87 \
|
||||
--hash=sha256:a68e604c6d1b0338e46652e2688cbce8096ad9c03548b075fda9e2ea19a9b7dd \
|
||||
--hash=sha256:a8af570fc29cd998a921c7131c8ac81d9434466d6d25300cb12a690fb56a8a08 \
|
||||
--hash=sha256:aadc298969ad229d8e3029fc5cc751fdad286696230f9cf014e90ff9cd8e6ea0 \
|
||||
--hash=sha256:ab56439c9f74c52770690c7b2f616b3bf775cb3920453ee355ac765c032d8bbf \
|
||||
--hash=sha256:ab9f962a5b64a5c3c845d556b7dc4e6fb683f7b67179f8205e814bb2e0213ffe \
|
||||
--hash=sha256:afbce6e3f0fac32dc87c2a0d84869d1a706460d64f39f3889386413e6e4d3d26 \
|
||||
--hash=sha256:b3ff0ad440ad52dda64138f16895f66403f40192365e39b1010e889f289746b0 \
|
||||
--hash=sha256:b580794e926cab7ff42ee4371ef14e0b22cb2bb722a607f77769136468f49a3f \
|
||||
--hash=sha256:b85b960a4507b0714c0a1246d031be9118d908ee974dc085257297a955205f1d \
|
||||
--hash=sha256:b98860aefbd3d9bc8e3c7f0eefb83b11142b16110739c68cd33d3b4d6e84e536 \
|
||||
--hash=sha256:bb31f42ea095ea826463c770829aa188a86c9a5c976b1467cbbf583c811de833 \
|
||||
--hash=sha256:bc0bca48ba24c6c866847fd20478a51dd547fa0ad258dab9615c414ec534bbc0 \
|
||||
--hash=sha256:bd1470d2c53fe53269bf5619da7725d30dd9b9693f1689f7a85eab8dea734442 \
|
||||
--hash=sha256:c09e097d0e46e3c289bedab9a475ae344b70c30ff5646e46af22b4e6fdc97b21 \
|
||||
--hash=sha256:c1bec5d6a19f5fbe87e4940739cfc65e7bb53d8b353e1029b8037a1653b321bc \
|
||||
--hash=sha256:c1c118a6b0e25bfc9a6802075d748fa6321714ffbdf3c88d29d9a0e3c7386c75 \
|
||||
--hash=sha256:c23e532c8a2325a1e7486de8763a60dc43e83f01bcaeca07e3ba79652c156db1 \
|
||||
--hash=sha256:c356dbddab0a529ed7574f78f559d75a223735c321c28f6f587fbf02b11ed301 \
|
||||
--hash=sha256:c38515cb54902f7e97d0239e81ef46c4444f9475f4807fb9bbdb789b4089abcf \
|
||||
--hash=sha256:c395bda8e7d8f51a02e80261fb57127979e5c472675d9a96b2860619ad47da48 \
|
||||
--hash=sha256:c6be9cba65c65cc76dfa3d4619e359ff02a4476c74e179b215236c11a0b32345 \
|
||||
--hash=sha256:cd526c8228e759c1006c4b7c9ac71dc4e925ced1a6a6a5a8e94643709738f63e \
|
||||
--hash=sha256:cddc675ec31bca65473321f9a9794e488b43b3b8de5d02c8ef4810c5d5792163 \
|
||||
--hash=sha256:cffc84ddec6da7f447677266fee2a3c40ecc78172f00752aa1150b8a8d65df1d \
|
||||
--hash=sha256:d41e9845514754a42d1d83b2fca9d27fee2ca7b3b0bee6843ba5a9bb2b6e25ac \
|
||||
--hash=sha256:d69fd559f9f0e8a52d2fce6f04ee143f86e70df0a189cd95164eddac599e810f \
|
||||
--hash=sha256:d7d72843691f50b91127c50688df10cb72ec6f4c4b1d7e2c11ab33b16acf8e51 \
|
||||
--hash=sha256:d9aac6081513f02eac3f8caace800dbfc5c608b69e4a7bef69e414eabfc95aa1 \
|
||||
--hash=sha256:dbfae8e75b342e31fc6fd1a8bbb393b7cbb91d6cfd581650300a94381e7b7e2b \
|
||||
--hash=sha256:e8208f2729cba030ff872a92064c97584eeb9502f53d32a05a0f05d5a17ca6c6 \
|
||||
--hash=sha256:e95e321d0d763f2b6633512605f6112ebd70d5746f3ce05c941909d4a25233f2 \
|
||||
--hash=sha256:e98ec9ec61cce5bc4b8b218322ad090b0994eb060bb04da704c62ef0a3d864e6 \
|
||||
--hash=sha256:eab6de8a98b9a7772cf686d00b4de439fc7efb8ab05ae106ef227291d06f87c5 \
|
||||
--hash=sha256:efe0ae052a8d023b87198921e8a7ce1dc7768816bcd2fbc20df171ac73a04891 \
|
||||
--hash=sha256:f11a398d8170b7ac5000baf7f258dcda579ef3ea744e0cc6a165e0dfbc0d3198 \
|
||||
--hash=sha256:f3fd9a1f87f8f0f3f8e9f9bd0195f7516562d13f5b178db8c5784d1f60b60bed \
|
||||
--hash=sha256:f47b0815af3948ec6a440b3afa02f05b18cc0939549e91b5c677b5d9c2c8472a \
|
||||
--hash=sha256:f991247276797d0c61ab7770bc9791eadc16f683b4d83517f624932adc1a8bab \
|
||||
--hash=sha256:ffad64ce7ad3703d652a3fd9af26238377d24ce52c6ad8ff35d26d82f61f493f
|
||||
# via -r requirements.in
|
||||
yarl==1.24.5 \
|
||||
--hash=sha256:0055afc45e864b92729ac7600e2d102c17bef060647e74bca75fa84d66b9ff36 \
|
||||
--hash=sha256:0465ec8cedc2349b97a6b595ace64084a50c6e839eca40aa0626f38b8350e331 \
|
||||
--hash=sha256:0ebfaffe1a16cb72141c8e09f18cc76856dbe58639f393a4f2b26e474b96b871 \
|
||||
--hash=sha256:16a2f5010280020e90f5330257e6944bc33e73593b136cc5a241e6c1dc292498 \
|
||||
--hash=sha256:17f57620f5475b3c69109376cc87e42a7af5db13c9398e4292772a706ff10780 \
|
||||
--hash=sha256:2120b96872df4a117cde97d270bac96aea7cc52205d305cf4611df694a487027 \
|
||||
--hash=sha256:240cbec09667c1fed4c6cd0060b9ec57332427d7441289a2ed8875dc9fb2b224 \
|
||||
--hash=sha256:24e861e9630e0daddcb9191fb187f60f034e17a4426f8101279f0c475cd74144 \
|
||||
--hash=sha256:2729fcfc4f6a596fb0c50f32090400aa9367774ac296a00387e65098c0befa76 \
|
||||
--hash=sha256:2c1fe720934a16ea8e7146175cba2126f87f54912c8c5435e7f7c7a51ef808d3 \
|
||||
--hash=sha256:2cabe6546e41dabe439999a23fcb5246e0c3b595b4315b96ef755252be90caeb \
|
||||
--hash=sha256:2dbe06fc16bc91502bca713704022182e5729861ae00277c3a23354b40929740 \
|
||||
--hash=sha256:3363fcc96e665878946ad7a106b9a13eac0541766a690ef287c0232ac768b6ec \
|
||||
--hash=sha256:377fe3732edbaf78ee74efdf2c9f49f6e99f20e7f9d2649fda3eb4badd77d76e \
|
||||
--hash=sha256:3ac6aff147deb9c09461b2d4bbdf6256831198f5d8a23f5d37138213090b6d8a \
|
||||
--hash=sha256:3f45789ce415a7ec0820dc4f82925f9b5f7732070be1dec1f5f23ec381435a24 \
|
||||
--hash=sha256:4103b77b8a8225e413107d2349b65eb3c1c52627b5cc5c3c4c1c6a798b218950 \
|
||||
--hash=sha256:4377407001ca3c057773f44d8ddd6358fa5f691407c1ba92210bd3cf8d9e4c95 \
|
||||
--hash=sha256:46c2f213e23a04b93a392942d782eb9e413e6ef6bf7c8c53884e599a5c174dcb \
|
||||
--hash=sha256:47e98aab9d8d82ff682e7b0b5dded33bf138a32b817fcf7fa3b27b2d7c412928 \
|
||||
--hash=sha256:4a36f9becdd4c5c52a20c3e9484128b070b1dcfc8944c006f3a528295a359a9c \
|
||||
--hash=sha256:4af7b7e1be0a69bee8210735fe6dcfc38879adfac6d62e789d53ba432d1ffa41 \
|
||||
--hash=sha256:4d97a951a81039050e45f04e96689b58b8243fa5e62aa14fe67cb6075300885e \
|
||||
--hash=sha256:4db9aecb141cb7a5447171b57aa1ed3a8fee06af40b992ffc31206c0b0121550 \
|
||||
--hash=sha256:53e549287ef628fecba270045c9701b0c564563a9b0577d24a4ec75b8ab8040f \
|
||||
--hash=sha256:56b149b22de33b23b0c6077ab9518c6dcb538ad462e1830e68d06591ccf6e38b \
|
||||
--hash=sha256:570fec8fbd22b032733625f03f10b7ff023bc399213db15e72a7acaef28c2f4e \
|
||||
--hash=sha256:5b8ee53be440a0cffc991a27be3057e0530122548dbe7c0892df08822fce5ede \
|
||||
--hash=sha256:5ba4f78df2bcc19f764a4b26a8a4f5049c110090ad5825993aacb052bf8003ad \
|
||||
--hash=sha256:5c55256dee8f4b27bfbf636c8363383c7c8db7890c7cba5217d7bd5f5f21dab6 \
|
||||
--hash=sha256:5c88e5815a49d289e599f3513aa7fde0bc2092ff188f99c940f007f90f53d104 \
|
||||
--hash=sha256:5fede79c6f73ff2c3ef822864cb1ada23196e62756df53bc6231d351a49516a2 \
|
||||
--hash=sha256:65be18ec59496c13908f02a2472751d9ef840b4f3fb5726f129306bf6a2a7bba \
|
||||
--hash=sha256:66410eb6345d467151934b49bfa70fb32f5b35a6140baa40ad97d6436abea2e9 \
|
||||
--hash=sha256:665b0a2c463cc9423dd647e0bfd9f4ccc9b50f768c55304d5e9f80b177c1de12 \
|
||||
--hash=sha256:6b8536851f9f65e7f00c7a1d49ba7f2be0ffe2c11555367fc9f50d9f842410a1 \
|
||||
--hash=sha256:6c95b17fe34ed802f17e205112e6e10db92275c34fee290aa9bdc55a9c724027 \
|
||||
--hash=sha256:6e73e7fe93f17a7b191f52ec9da9dd8c06a8fe735a1ecbd13b97d1c723bff385 \
|
||||
--hash=sha256:6efbccc3d7f75d5b03105172a8dc86d82ba4da86817952529dd93185f4a88be2 \
|
||||
--hash=sha256:709f1efed56c4a145793c046cd4939f9959bcd818979a787b77d8e09c57a0840 \
|
||||
--hash=sha256:79af890482fc94648e8cde4c68620378f7fef60932710fa17a66abc039244da2 \
|
||||
--hash=sha256:7bcbe0fcf850eae67b6b01749815a4f7161c560a844c769ad7b48fcd99f791c4 \
|
||||
--hash=sha256:7c0494a31a1ac5461a226e7947a9c9b78c44e1dc7185164fa7e9651557a5d9bc \
|
||||
--hash=sha256:7ce27823052e2013b597e0c738b13e7e36b8ccb9400df8959417b052ab0fd92c \
|
||||
--hash=sha256:7f72c74aa99359e27a2ee8d6613fefa28b5f76a983c083074dfc2aaa4ab46213 \
|
||||
--hash=sha256:7fa5e51397466ea7e98de493fa2ff1b8193cfef8a7b0f9b4842f92d342df0dba \
|
||||
--hash=sha256:82632daed195dcc8ea664e8556dc9bdbd671960fb3776bd92806ce05792c2448 \
|
||||
--hash=sha256:82f75e05912e84b7a0fe57075d9c59de3cb352b928330f2eb69b2e1f54c3e1f0 \
|
||||
--hash=sha256:841f0852f48fefea3b12c9dfec00704dfa3aef5215d0e3ce564bb3d7cd8d57c6 \
|
||||
--hash=sha256:874019bd513008b009f58657134e5d0c5e030b3559bd0553976837adf52fe966 \
|
||||
--hash=sha256:88f50c94e21a0a7f14042c015b0eba1881af78562e7bf007e0033e624da59750 \
|
||||
--hash=sha256:89a1bbb58e0e3f7a283653d854b1e95d65e5cfd4af224dac5f02629ec1a3e621 \
|
||||
--hash=sha256:8a6987eaad834cb32dd57d9d582225f0054a5d1af706ccfbbdba735af4927e13 \
|
||||
--hash=sha256:8ac73abdc7ab75610f95a8fd994c6457e87752b02a63987e188f937a1fc180f0 \
|
||||
--hash=sha256:8ccf9aca873b767977c73df497a85dbedee4ee086ae9ae49dc461333b9b79f58 \
|
||||
--hash=sha256:90333fd89b43c0d08ac85f3f1447593fc2c66de18c3d6378d7125ea118dc7a54 \
|
||||
--hash=sha256:92ab3e11448f2ff7bf53c5a26eff0edc086898ec8b21fb154b85839ce1d88075 \
|
||||
--hash=sha256:9335a099ad87287c37fe5d1a982ff392fa5efe5d14b40a730b1ec1d6a41382b4 \
|
||||
--hash=sha256:96d30286dd02679e32a39aa8f0b7498fc847fcda46cfc09df5513e82ce252440 \
|
||||
--hash=sha256:9baafc71b04f8f4bb0703b21d6fc9f0c30b346c636a532ff16ec8491a5ea4b1f \
|
||||
--hash=sha256:9d1216a7f6f77836617dba35687c5b78a4170afc3c3f18fc788f785ba26565c4 \
|
||||
--hash=sha256:9d399bdcfb4a0f659b9b3788bbc89babe63d9a6a65aacdf4d4e7065ff2e6316c \
|
||||
--hash=sha256:9e4e16c73d717c5cf27626c524d0a2e261ad20e46932b2670f64ad5dde23e26f \
|
||||
--hash=sha256:9f4d8cf085a4c6a40fb97ea0f46938a8df43c85d31f9d45e2a8867ea9293790d \
|
||||
--hash=sha256:a33700d13d9b7d84fd10947b09ff69fb9a792e519c8cb9764a3ca70baa6c23a7 \
|
||||
--hash=sha256:a3732e66413163e72508da9eff9ce9d2846fde51fae45d3605393d3e6cd303e9 \
|
||||
--hash=sha256:a4582acf7ef76482f6f511ebaf1946dae7f2e85ec4728b81a678c01df63bd723 \
|
||||
--hash=sha256:a61834fb15d81322d872eaafd333838ae7c9cea84067f232656f75965933d047 \
|
||||
--hash=sha256:a7cff474ab7cd149765bb784cf6d78b32e18e20473fb7bda860bce98ab58e9da \
|
||||
--hash=sha256:a8fe66b8f300da93798025a785a5b90b42f3810dc2b72283ff84a41aaaebc293 \
|
||||
--hash=sha256:a929d878fec099030c292803b31e5d5540a7b6a31e6a3cc76cb4685fc2a2f51b \
|
||||
--hash=sha256:ad5d8201d310b031e6cd839d9bac2d4e5a01533ce5d3d5b50b7de1ef3af1de61 \
|
||||
--hash=sha256:af3aefa655adb5869491fa907e652290386800ae99cc50095cba71e2c6aefdca \
|
||||
--hash=sha256:c0ebc836c47a6477e182169c6a476fc691d12b518894bf7dd2572f0d59f1c7ed \
|
||||
--hash=sha256:c687ed078e145f5fd53a14854beff320e1d2ab76df03e2009c98f39a0f68f39a \
|
||||
--hash=sha256:cbb833ccacdb5519eff9b8b71ee618cc2801c878e77e288775d77c3a2ced858a \
|
||||
--hash=sha256:cf139c02f5f23ef6532040a30ff662c00a318c952334f211046b8e60b7f17688 \
|
||||
--hash=sha256:d46b86567dd4e248c6c159fcbcdcce01e0a5c8a7cd2334a0fff759d0fa075b16 \
|
||||
--hash=sha256:d693396e5aea78db03decd60aec9ece16c9b40ba00a587f089615ff4e718a81d \
|
||||
--hash=sha256:d897129df1a22b12aeed2c2c98df0785a2e8e6e0bde87b389491d0025c187077 \
|
||||
--hash=sha256:daba5e594f06114e37db186efd2dd916609071e59daca901a0a2e71f02b142ce \
|
||||
--hash=sha256:dd625535328fd9882374356269227670189adfcc6a2d90284f323c05862eecbd \
|
||||
--hash=sha256:e006d3a974c4ee19512e5f058abedb6eef36a5e553c14812bdeba1758d812e6d \
|
||||
--hash=sha256:e1ae548a9d901adca07899a4147a7c826bbcc06239d3ce9a59f57886a28a4c88 \
|
||||
--hash=sha256:e2935f8c39e3b03e83519292d78f075189978f3f4adc15a78144c7c8e2a1cba5 \
|
||||
--hash=sha256:e42d75862735da90e7fc5a7b23db0c976f737113a54b3c9777a9b665e9cbff75 \
|
||||
--hash=sha256:e7d42c531243450ef0d4d9c172e7ed6ef052640f195629065041b5add4e058d1 \
|
||||
--hash=sha256:e81b83143bee16329c23db3c1b2d82b29892fcbcb849186d2f6e98a5abe9a57f \
|
||||
--hash=sha256:e8ffa78582120024f476a611d7befc123cee59e47e8309d470cf667d806e613b \
|
||||
--hash=sha256:ebb0ec7f17803063d5aeb982f3b1bd2b2f4e4fae6751226cbd6ba1fcfe9e63ff \
|
||||
--hash=sha256:f08c7513ecef5aad65687bfdf6bc601ae9fccd04a42904501f8f7141abad9eb9 \
|
||||
--hash=sha256:f0a658a6d3fafee5c6f63c58f3e785c8c43c93fbc02bf9f2b6663f8185e0971f \
|
||||
--hash=sha256:f0e466ed7511fe9d459a819edbc6c2585c0b6eabde9fa8a8947552468a7a6ef0 \
|
||||
--hash=sha256:f141474e85b7e54998ec5180530a7cda99ab29e282fa50e0756d89981a9b43c5 \
|
||||
--hash=sha256:f4239bbec5a3577ddb49e4b50aeb32d8e5792098262ae2f63723f916a29b1a25 \
|
||||
--hash=sha256:f540c013589084679a6c7fac07096b10159737918174f5dfc5e11bf5bca4dfe6 \
|
||||
--hash=sha256:f9f3e9c8a9ecffa57bef8fb4fa19e5fa4d2d8307cf6bac5b1fca5e5860f4ba00 \
|
||||
--hash=sha256:fa139875ff98ab97da323cfadfaff08900d1ad42f1b5087b0b812a55c5a06373 \
|
||||
--hash=sha256:fcd3b77e2f17bbe4ca56ec7bcb07992647d19d0b9c05d84886dcd6f9eb810afd \
|
||||
--hash=sha256:fd8c81f346b58f45818d09ea11db69a8d5fd34a224b79871f6d44f12cd7977b1 \
|
||||
--hash=sha256:fe7b7bb170daccbba19ad33012d2b15f1e7942296fd4d45fc1b79013da8cc0f2 \
|
||||
--hash=sha256:ff330d3c30db4eb6b01d79e29d2d0b407a7ecad39cfd9ec993ece57396a2ec0d \
|
||||
--hash=sha256:ff405d91509d88e8d44129cd87b18d70acd1f0c1aeabd7bc3c46792b1fe2acba \
|
||||
--hash=sha256:ffcd54362564dc1a30fb74d8b8a6e5a6b11ebd5e27266adc3b7427a21a6c9104
|
||||
# via aiohttp
|
||||
|
||||
@@ -30,6 +30,7 @@ import argparse
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
import random
|
||||
import sys
|
||||
import time
|
||||
@@ -39,6 +40,11 @@ from typing import Any
|
||||
|
||||
import websockets
|
||||
|
||||
# websockets loads its submodules lazily, so websockets.exceptions is not
|
||||
# reachable through the package until something imports it. REQUEST_FAILURES
|
||||
# is built at import time and needs it now.
|
||||
import websockets.exceptions
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Configuration
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -71,9 +77,81 @@ DEFAULT_WEIGHTS: dict[str, int] = {
|
||||
# Well-known genesis account for queries that require an account parameter.
|
||||
GENESIS_ACCOUNT = "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh"
|
||||
|
||||
# How long a single request waits for its reply.
|
||||
RECV_TIMEOUT_S = 10.0
|
||||
|
||||
# Teardown budget for requests still in flight. Only the at-most-one request
|
||||
# per connection that already holds the gate can still finish, and its worst
|
||||
# case is the full receive timeout, so that plus a small grace is the whole
|
||||
# useful wait. Requests still queued behind the gate would need
|
||||
# queue_depth x round-trip, which is unbounded; they are cancelled instead.
|
||||
DRAIN_TIMEOUT_S = RECV_TIMEOUT_S + 2.0
|
||||
|
||||
# Requests allowed to own one connection's recv() at a time. websockets
|
||||
# rejects a second concurrent recv() on the same socket, so this must stay 1
|
||||
# unless request/response correlation by id is added. Concurrency comes from
|
||||
# spreading requests round-robin over the endpoints instead.
|
||||
MAX_INFLIGHT_PER_CONNECTION = 1
|
||||
|
||||
# Fraction of dispatched requests that must reach the server for a run to
|
||||
# count as a measurement. The gate is one connection deep, so the ceiling is
|
||||
# len(connections) / round-trip requests per second; asking for more than
|
||||
# that silently drops the excess instead of erroring, which would report a
|
||||
# perfect score on a run that generated a fraction of its intended load.
|
||||
# Fifty percent mirrors the error-rate limit below, on the same reasoning: a
|
||||
# run in which most requests did not happen is not a valid baseline.
|
||||
MIN_DELIVERY_PCT = 50.0
|
||||
|
||||
# Error rate above which the run is treated as a failure.
|
||||
MAX_ERROR_RATE_PCT = 50.0
|
||||
|
||||
# Failures that mean "this request failed", not "the generator is broken":
|
||||
# asyncio.TimeoutError - no reply within RECV_TIMEOUT_S. Same class as the
|
||||
# builtin TimeoutError on Python 3.11+.
|
||||
# WebSocketException - transport failure, including the connection being
|
||||
# closed while a receive was outstanding, and the
|
||||
# ConcurrencyError raised for a rejected concurrent
|
||||
# recv() (it subclasses WebSocketException).
|
||||
# json.JSONDecodeError - reply body was not valid JSON.
|
||||
# AttributeError - reply parsed to something with no .get(), e.g. a
|
||||
# JSON array.
|
||||
REQUEST_FAILURES: tuple[type[BaseException], ...] = (
|
||||
asyncio.TimeoutError,
|
||||
websockets.exceptions.WebSocketException,
|
||||
json.JSONDecodeError,
|
||||
AttributeError,
|
||||
)
|
||||
|
||||
logger = logging.getLogger("rpc_load_generator")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Latency helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _percentile(sorted_values: list[float], quantile: float) -> float:
|
||||
"""Return the nearest-rank percentile of an ascending list of values.
|
||||
|
||||
The nearest-rank index is ``ceil(n * q) - 1``, clamped to the last
|
||||
element. Plain ``int(n * q)`` truncation selects one rank too high
|
||||
whenever ``n * q`` is a whole number — at n=100 it picks index 99, the
|
||||
maximum, so the reported p99 was really p100 (n=20 for p95).
|
||||
|
||||
Args:
|
||||
sorted_values: Values sorted ascending.
|
||||
quantile: Quantile to select, in (0, 1] — e.g. 0.99.
|
||||
|
||||
Returns:
|
||||
The selected value, or 0.0 when the list is empty.
|
||||
"""
|
||||
n = len(sorted_values)
|
||||
if n == 0:
|
||||
return 0.0
|
||||
idx = min(math.ceil(n * quantile) - 1, n - 1)
|
||||
return sorted_values[max(idx, 0)]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Data classes
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -83,17 +161,29 @@ logger = logging.getLogger("rpc_load_generator")
|
||||
class LoadStats:
|
||||
"""Tracks request counts and latencies during a load run.
|
||||
|
||||
``total_dispatched`` counts intent and ``total_sent`` counts outcome, so
|
||||
the difference is the load that never happened. They diverge whenever the
|
||||
requested rate exceeds what the connections can carry: the dispatch loop
|
||||
keeps pace, the requests queue behind the per-connection gate, and
|
||||
teardown cancels whatever never got its turn. Without the two extra
|
||||
counters that shortfall shows up as nothing at all -- no error, no
|
||||
warning, and an error_rate_pct of 0 over a fraction of the traffic.
|
||||
|
||||
Attributes:
|
||||
total_sent: Total RPC requests dispatched.
|
||||
total_success: Requests that returned a valid result.
|
||||
total_errors: Requests that returned an error or timed out.
|
||||
latencies: Per-command list of round-trip times in seconds.
|
||||
command_counts: Per-command request count.
|
||||
total_dispatched: Requests the dispatch loop created a task for.
|
||||
total_sent: Requests that completed and were recorded.
|
||||
total_success: Requests that returned a valid result.
|
||||
total_errors: Requests that returned an error or timed out.
|
||||
total_cancelled: Requests cancelled at teardown, never recorded.
|
||||
latencies: Per-command list of round-trip times in seconds.
|
||||
command_counts: Per-command request count.
|
||||
"""
|
||||
|
||||
total_dispatched: int = 0
|
||||
total_sent: int = 0
|
||||
total_success: int = 0
|
||||
total_errors: int = 0
|
||||
total_cancelled: int = 0
|
||||
latencies: dict[str, list[float]] = field(default_factory=dict)
|
||||
command_counts: dict[str, int] = field(default_factory=dict)
|
||||
|
||||
@@ -108,30 +198,71 @@ class LoadStats:
|
||||
self.command_counts[command] = self.command_counts.get(command, 0) + 1
|
||||
|
||||
def summary(self) -> dict[str, Any]:
|
||||
"""Return a summary dict suitable for JSON serialization."""
|
||||
"""Return a summary dict suitable for JSON serialization.
|
||||
|
||||
``total_sent``, ``total_success``, ``total_errors``,
|
||||
``error_rate_pct`` and ``per_command`` keep their names and meanings;
|
||||
workload_orchestrator.py reads the first and third of those. The three
|
||||
delivery keys are additions. ``delivery_pct`` is 0.0 when nothing was
|
||||
dispatched at all -- a run that opened no connection delivered none of
|
||||
its load, and reporting 100% for it would be the same blind spot the
|
||||
key exists to close.
|
||||
"""
|
||||
per_command: dict[str, Any] = {}
|
||||
for cmd, lats in self.latencies.items():
|
||||
sorted_lats = sorted(lats)
|
||||
n = len(sorted_lats)
|
||||
per_command[cmd] = {
|
||||
"count": self.command_counts.get(cmd, 0),
|
||||
"p50_ms": round(sorted_lats[n // 2] * 1000, 2) if n else 0,
|
||||
"p95_ms": (round(sorted_lats[int(n * 0.95)] * 1000, 2) if n else 0),
|
||||
"p99_ms": (round(sorted_lats[int(n * 0.99)] * 1000, 2) if n else 0),
|
||||
"p50_ms": round(_percentile(sorted_lats, 0.50) * 1000, 2),
|
||||
"p95_ms": round(_percentile(sorted_lats, 0.95) * 1000, 2),
|
||||
"p99_ms": round(_percentile(sorted_lats, 0.99) * 1000, 2),
|
||||
}
|
||||
return {
|
||||
"total_dispatched": self.total_dispatched,
|
||||
"total_sent": self.total_sent,
|
||||
"total_success": self.total_success,
|
||||
"total_errors": self.total_errors,
|
||||
"total_cancelled": self.total_cancelled,
|
||||
"error_rate_pct": (
|
||||
round(self.total_errors / self.total_sent * 100, 2)
|
||||
if self.total_sent
|
||||
else 0
|
||||
),
|
||||
"delivery_pct": (
|
||||
round(self.total_sent / self.total_dispatched * 100, 2)
|
||||
if self.total_dispatched
|
||||
else 0.0
|
||||
),
|
||||
"per_command": per_command,
|
||||
}
|
||||
|
||||
|
||||
@dataclass
|
||||
class Connection:
|
||||
"""One open WebSocket endpoint together with its request gate.
|
||||
|
||||
websockets raises rather than mis-delivering when two coroutines call
|
||||
``recv()`` on the same socket, so every request holds ``gate`` across its
|
||||
send and its matching receive. Parallelism comes from the round-robin
|
||||
spread over endpoints: N endpoints allow N requests in flight.
|
||||
|
||||
run_load ──round-robin──> Connection[0] ─gate─> send_rpc (1 at a time)
|
||||
└─> Connection[1] ─gate─> send_rpc (1 at a time)
|
||||
|
||||
Attributes:
|
||||
url: WebSocket URL this connection was opened against, for logging.
|
||||
ws: The open connection.
|
||||
gate: Limits concurrent send/receive pairs on ``ws`` to
|
||||
MAX_INFLIGHT_PER_CONNECTION.
|
||||
"""
|
||||
|
||||
url: str
|
||||
ws: websockets.ClientConnection
|
||||
gate: asyncio.Semaphore = field(
|
||||
default_factory=lambda: asyncio.Semaphore(MAX_INFLIGHT_PER_CONNECTION)
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# RPC command builders
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -167,7 +298,9 @@ def build_rpc_request(command: str) -> dict[str, Any]:
|
||||
req["limit"] = 5
|
||||
elif command == "tx":
|
||||
# Use a dummy hash — returns "txnNotFound" error but still exercises
|
||||
# the full RPC span pipeline (rpc.ws_message -> rpc.process -> rpc.command.tx).
|
||||
# the full RPC span pipeline for this transport (rpc.ws_message ->
|
||||
# rpc.command.tx). rpc.process is not in that chain: it is created
|
||||
# only on the HTTP/JSON-RPC path, which this client never uses.
|
||||
req["transaction"] = "0" * 64
|
||||
req["binary"] = False
|
||||
elif command == "account_tx":
|
||||
@@ -220,15 +353,23 @@ def choose_command(weights: dict[str, int]) -> str:
|
||||
|
||||
|
||||
async def send_rpc(
|
||||
ws: websockets.WebSocketClientProtocol,
|
||||
conn: Connection,
|
||||
command: str,
|
||||
stats: LoadStats,
|
||||
inject_traceparent: bool = True,
|
||||
) -> None:
|
||||
"""Send a single RPC request over WebSocket and record the result.
|
||||
|
||||
Holds ``conn.gate`` across the send and the matching receive so only one
|
||||
request at a time owns the connection's ``recv()``. The latency clock
|
||||
starts after the gate is acquired, so time spent queued behind a busy
|
||||
connection is not charged to the server.
|
||||
|
||||
Every outcome is recorded, including failures, so error_rate_pct in the
|
||||
summary reflects every request that was actually sent.
|
||||
|
||||
Args:
|
||||
ws: Open WebSocket connection.
|
||||
conn: Target connection and its request gate.
|
||||
command: RPC command name.
|
||||
stats: LoadStats instance to record results.
|
||||
inject_traceparent: If True, add a W3C traceparent header field
|
||||
@@ -238,26 +379,173 @@ async def send_rpc(
|
||||
|
||||
# Inject W3C traceparent for context propagation testing.
|
||||
# The rippled WebSocket handler extracts this from the JSON body
|
||||
# when present (Phase 2 context propagation).
|
||||
# when present.
|
||||
if inject_traceparent:
|
||||
trace_id = uuid.uuid4().hex
|
||||
span_id = uuid.uuid4().hex[:16]
|
||||
request["traceparent"] = f"00-{trace_id}-{span_id}-01"
|
||||
|
||||
t0 = time.monotonic()
|
||||
try:
|
||||
await ws.send(json.dumps(request))
|
||||
raw = await asyncio.wait_for(ws.recv(), timeout=10.0)
|
||||
latency = time.monotonic() - t0
|
||||
response = json.loads(raw)
|
||||
# Native WS responses have {"status": "success", "result": {...}}
|
||||
# or {"status": "error", "error": "...", "error_message": "..."}.
|
||||
success = response.get("status") == "success"
|
||||
async with conn.gate:
|
||||
t0 = time.monotonic()
|
||||
# The try covers the I/O and the parse only. Recording sits outside it
|
||||
# so a bug in record() surfaces as the task failure it is, instead of
|
||||
# being counted as one more failed request.
|
||||
try:
|
||||
await conn.ws.send(json.dumps(request))
|
||||
raw = await asyncio.wait_for(conn.ws.recv(), timeout=RECV_TIMEOUT_S)
|
||||
latency = time.monotonic() - t0
|
||||
# Native WS responses have {"status": "success", "result": {...}}
|
||||
# or {"status": "error", "error": "...", "error_message": "..."}.
|
||||
success = json.loads(raw).get("status") == "success"
|
||||
except REQUEST_FAILURES as exc:
|
||||
logger.debug("RPC %s failed: %s", command, exc)
|
||||
stats.record(command, time.monotonic() - t0, False)
|
||||
return
|
||||
stats.record(command, latency, success)
|
||||
except (asyncio.TimeoutError, websockets.exceptions.WebSocketException) as exc:
|
||||
latency = time.monotonic() - t0
|
||||
stats.record(command, latency, False)
|
||||
logger.debug("RPC %s failed: %s", command, exc)
|
||||
|
||||
|
||||
async def open_connections(endpoints: list[str]) -> list[Connection]:
|
||||
"""Open one persistent WebSocket connection per endpoint.
|
||||
|
||||
Endpoints that refuse the connection are logged and skipped, so a partly
|
||||
reachable cluster still produces load.
|
||||
|
||||
Args:
|
||||
endpoints: List of WebSocket URLs (ws://host:port).
|
||||
|
||||
Returns:
|
||||
The connections that were established, possibly empty.
|
||||
"""
|
||||
connections: list[Connection] = []
|
||||
for ep in endpoints:
|
||||
try:
|
||||
ws = await websockets.connect(ep, ping_interval=20, ping_timeout=10)
|
||||
connections.append(Connection(url=ep, ws=ws))
|
||||
logger.info("Connected to %s", ep)
|
||||
except Exception as exc:
|
||||
logger.error("Failed to connect to %s: %s", ep, exc)
|
||||
return connections
|
||||
|
||||
|
||||
async def drain_requests(inflight: set[asyncio.Task[None]]) -> int:
|
||||
"""Let in-flight requests finish, then cancel whatever is still stuck.
|
||||
|
||||
A request may wait up to RECV_TIMEOUT_S for its reply, so closing the
|
||||
connections straight away would turn late replies into errors. Anything
|
||||
still unfinished after DRAIN_TIMEOUT_S is cancelled, and the count is
|
||||
returned so the caller can put the shortfall in the summary instead of
|
||||
losing it to a log line.
|
||||
|
||||
Args:
|
||||
inflight: Tasks still tracked as unfinished. Finished tasks remove
|
||||
themselves, so this is the outstanding set.
|
||||
|
||||
Returns:
|
||||
Number of requests cancelled without being recorded.
|
||||
"""
|
||||
pending = {task for task in inflight if not task.done()}
|
||||
if not pending:
|
||||
return 0
|
||||
|
||||
logger.info(
|
||||
"Draining %d in-flight request(s), up to %.0fs...",
|
||||
len(pending),
|
||||
DRAIN_TIMEOUT_S,
|
||||
)
|
||||
_, stuck = await asyncio.wait(pending, timeout=DRAIN_TIMEOUT_S)
|
||||
if not stuck:
|
||||
logger.info("All in-flight requests completed.")
|
||||
return 0
|
||||
|
||||
for task in stuck:
|
||||
task.cancel()
|
||||
await asyncio.gather(*stuck, return_exceptions=True)
|
||||
# Most of these never left the client: they were still waiting for the
|
||||
# per-connection gate. Up to one per connection had already been sent and
|
||||
# was waiting on recv() when it was cancelled, so the server may have
|
||||
# handled it. Either way CancelledError is not a REQUEST_FAILURES member,
|
||||
# so none of them reached stats.record and none are in total_sent.
|
||||
logger.warning(
|
||||
"Cancelled %d request(s) unfinished after %.0fs — not counted in "
|
||||
"total_sent; see total_cancelled and delivery_pct",
|
||||
len(stuck),
|
||||
DRAIN_TIMEOUT_S,
|
||||
)
|
||||
return len(stuck)
|
||||
|
||||
|
||||
def log_progress(stats: LoadStats, elapsed: float) -> None:
|
||||
"""Log throughput every 100 recorded requests.
|
||||
|
||||
Args:
|
||||
stats: Live counters.
|
||||
elapsed: Seconds since the run started.
|
||||
"""
|
||||
if stats.total_sent % 100 != 0 or stats.total_sent == 0:
|
||||
return
|
||||
logger.info(
|
||||
"Progress: %d sent, %d errors, %.1f RPS (%.0fs elapsed)",
|
||||
stats.total_sent,
|
||||
stats.total_errors,
|
||||
stats.total_sent / elapsed if elapsed > 0 else 0,
|
||||
elapsed,
|
||||
)
|
||||
|
||||
|
||||
async def dispatch_requests(
|
||||
connections: list[Connection],
|
||||
rate: float,
|
||||
duration: float,
|
||||
weights: dict[str, int],
|
||||
stats: LoadStats,
|
||||
inject_traceparent: bool,
|
||||
) -> None:
|
||||
"""Fire requests round-robin at the target rate, then drain them.
|
||||
|
||||
Each request runs as its own task so the dispatch loop keeps its pace
|
||||
regardless of reply latency. Tasks are tracked, not forgotten, so
|
||||
teardown can drain them and so an exception can never escape unseen.
|
||||
|
||||
Every task created counts towards ``stats.total_dispatched`` and every one
|
||||
cancelled at teardown towards ``stats.total_cancelled``, which is what
|
||||
makes an under-delivering run visible in the summary.
|
||||
|
||||
Args:
|
||||
connections: Open connections to spread requests over.
|
||||
rate: Target requests per second.
|
||||
duration: Total run time in seconds.
|
||||
weights: Command distribution weights.
|
||||
stats: LoadStats instance to record results in.
|
||||
inject_traceparent: Whether to inject W3C traceparent headers.
|
||||
"""
|
||||
interval = 1.0 / rate if rate > 0 else 0.1
|
||||
start = time.monotonic()
|
||||
conn_idx = 0
|
||||
inflight: set[asyncio.Task[None]] = set()
|
||||
|
||||
def reap(task: asyncio.Task[None]) -> None:
|
||||
"""Untrack a finished request and report anything that escaped it."""
|
||||
inflight.discard(task)
|
||||
if not task.cancelled() and task.exception() is not None:
|
||||
logger.error("RPC task failed unexpectedly: %s", task.exception())
|
||||
|
||||
try:
|
||||
while (time.monotonic() - start) < duration:
|
||||
conn = connections[conn_idx % len(connections)]
|
||||
conn_idx += 1
|
||||
task = asyncio.create_task(
|
||||
send_rpc(conn, choose_command(weights), stats, inject_traceparent)
|
||||
)
|
||||
inflight.add(task)
|
||||
task.add_done_callback(reap)
|
||||
stats.total_dispatched += 1
|
||||
|
||||
await asyncio.sleep(interval)
|
||||
log_progress(stats, time.monotonic() - start)
|
||||
except asyncio.CancelledError:
|
||||
logger.info("Load generation cancelled.")
|
||||
finally:
|
||||
stats.total_cancelled += await drain_requests(inflight)
|
||||
|
||||
|
||||
async def run_load(
|
||||
@@ -270,7 +558,9 @@ async def run_load(
|
||||
"""Run the RPC load generator against the given endpoints.
|
||||
|
||||
Distributes requests round-robin across endpoints at the specified
|
||||
rate (requests per second) for the given duration.
|
||||
rate (requests per second) for the given duration. Each connection
|
||||
serves one request at a time, so the ceiling per connection is
|
||||
1 / round-trip-latency requests per second; add endpoints to raise it.
|
||||
|
||||
Args:
|
||||
endpoints: List of WebSocket URLs (ws://host:port).
|
||||
@@ -283,18 +573,8 @@ async def run_load(
|
||||
LoadStats with aggregated results.
|
||||
"""
|
||||
stats = LoadStats()
|
||||
interval = 1.0 / rate if rate > 0 else 0.1
|
||||
|
||||
# Open persistent connections to all endpoints.
|
||||
connections: list[websockets.WebSocketClientProtocol] = []
|
||||
for ep in endpoints:
|
||||
try:
|
||||
ws = await websockets.connect(ep, ping_interval=20, ping_timeout=10)
|
||||
connections.append(ws)
|
||||
logger.info("Connected to %s", ep)
|
||||
except Exception as exc:
|
||||
logger.error("Failed to connect to %s: %s", ep, exc)
|
||||
|
||||
connections = await open_connections(endpoints)
|
||||
if not connections:
|
||||
logger.error("No connections established. Aborting.")
|
||||
return stats
|
||||
@@ -307,43 +587,26 @@ async def run_load(
|
||||
)
|
||||
|
||||
start = time.monotonic()
|
||||
conn_idx = 0
|
||||
|
||||
try:
|
||||
while (time.monotonic() - start) < duration:
|
||||
command = choose_command(weights)
|
||||
ws = connections[conn_idx % len(connections)]
|
||||
conn_idx += 1
|
||||
|
||||
# Fire-and-forget style with bounded concurrency via sleep.
|
||||
asyncio.create_task(send_rpc(ws, command, stats, inject_traceparent))
|
||||
await asyncio.sleep(interval)
|
||||
|
||||
# Periodic progress log.
|
||||
elapsed = time.monotonic() - start
|
||||
if stats.total_sent % 100 == 0 and stats.total_sent > 0:
|
||||
actual_rps = stats.total_sent / elapsed if elapsed > 0 else 0
|
||||
logger.info(
|
||||
"Progress: %d sent, %d errors, %.1f RPS (%.0fs elapsed)",
|
||||
stats.total_sent,
|
||||
stats.total_errors,
|
||||
actual_rps,
|
||||
elapsed,
|
||||
)
|
||||
except asyncio.CancelledError:
|
||||
logger.info("Load generation cancelled.")
|
||||
await dispatch_requests(
|
||||
connections, rate, duration, weights, stats, inject_traceparent
|
||||
)
|
||||
finally:
|
||||
# Allow in-flight requests to complete.
|
||||
await asyncio.sleep(2)
|
||||
for ws in connections:
|
||||
await ws.close()
|
||||
# Only reached once dispatch_requests has drained: closing a
|
||||
# connection under an outstanding receive raises ConnectionClosed,
|
||||
# which would be recorded as a failed request.
|
||||
for conn in connections:
|
||||
await conn.ws.close()
|
||||
|
||||
elapsed = time.monotonic() - start
|
||||
logger.info(
|
||||
"Load complete: %d sent, %d success, %d errors in %.1fs (%.1f RPS)",
|
||||
"Load complete: %d of %d dispatched sent, %d success, %d errors, "
|
||||
"%d cancelled in %.1fs (%.1f RPS)",
|
||||
stats.total_sent,
|
||||
stats.total_dispatched,
|
||||
stats.total_success,
|
||||
stats.total_errors,
|
||||
stats.total_cancelled,
|
||||
elapsed,
|
||||
stats.total_sent / elapsed if elapsed > 0 else 0,
|
||||
)
|
||||
@@ -455,9 +718,38 @@ def main() -> None:
|
||||
json.dump(summary, f, indent=2)
|
||||
logger.info("Summary written to %s", args.output)
|
||||
|
||||
# Exit with error if error rate exceeds 50%.
|
||||
if summary["error_rate_pct"] > 50:
|
||||
logger.error("High error rate: %.1f%%", summary["error_rate_pct"])
|
||||
# Both gates are evaluated and reported before exiting, and the summary is
|
||||
# already on disk, so the caller sees every reason plus the numbers behind
|
||||
# it. A run that under-delivers has to fail as loudly as one that errors:
|
||||
# every downstream span and metric assertion would otherwise be checked
|
||||
# against a fraction of the intended traffic and still look healthy.
|
||||
failures: list[str] = []
|
||||
if summary["error_rate_pct"] > MAX_ERROR_RATE_PCT:
|
||||
failures.append(
|
||||
"error rate %.2f%% exceeds %.0f%% (%d of %d requests failed)"
|
||||
% (
|
||||
summary["error_rate_pct"],
|
||||
MAX_ERROR_RATE_PCT,
|
||||
summary["total_errors"],
|
||||
summary["total_sent"],
|
||||
)
|
||||
)
|
||||
if summary["delivery_pct"] < MIN_DELIVERY_PCT:
|
||||
failures.append(
|
||||
"delivered %.2f%% of dispatched requests, below %.0f%% "
|
||||
"(%d sent, %d cancelled, of %d dispatched) — lower --rate or add "
|
||||
"endpoints"
|
||||
% (
|
||||
summary["delivery_pct"],
|
||||
MIN_DELIVERY_PCT,
|
||||
summary["total_sent"],
|
||||
summary["total_cancelled"],
|
||||
summary["total_dispatched"],
|
||||
)
|
||||
)
|
||||
for reason in failures:
|
||||
logger.error("%s", reason)
|
||||
if failures:
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
|
||||
@@ -19,7 +19,14 @@
|
||||
# Exit codes:
|
||||
# 0 — All validation checks and the regression gate passed
|
||||
# 1 — Validation checks failed OR the regression gate detected a regression
|
||||
# 2 — Infrastructure error (cluster/stack failed to start, timing capture failed)
|
||||
# OR the benchmark exceeded its overhead thresholds
|
||||
# 2 — Infrastructure error (cluster/stack failed to start, workload
|
||||
# orchestration failed, timing capture failed, overhead could not be
|
||||
# measured)
|
||||
#
|
||||
# Every step below records its status and folds it into FINAL_EXIT; the first
|
||||
# non-zero status in pipeline order is the one returned, so the earliest
|
||||
# failure — the one that explains the later ones — is what the caller sees.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
@@ -35,6 +42,17 @@ die() {
|
||||
exit 2
|
||||
}
|
||||
|
||||
# Overall run status, folded step by step (see the exit-code table above).
|
||||
FINAL_EXIT=0
|
||||
|
||||
# fold_exit STATUS — record a step's status in FINAL_EXIT.
|
||||
# First non-zero wins, so FINAL_EXIT names the earliest failing step.
|
||||
fold_exit() {
|
||||
if [ "$1" -ne 0 ] && [ "$FINAL_EXIT" -eq 0 ]; then
|
||||
FINAL_EXIT="$1"
|
||||
fi
|
||||
}
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Configuration
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -49,6 +67,9 @@ NUM_NODES=5
|
||||
RPC_PORT_BASE=5005
|
||||
WS_PORT_BASE=6006
|
||||
PEER_PORT_BASE=51235
|
||||
# Inert: parsed from --rpc-rate/--rpc-duration/--tx-tps/--tx-duration and never
|
||||
# read again. Load shape comes from the workload profile instead. Kept because
|
||||
# the CI workflow still passes the four flags.
|
||||
RPC_RATE=50
|
||||
RPC_DURATION=120
|
||||
TX_TPS=5
|
||||
@@ -78,16 +99,22 @@ usage() {
|
||||
echo "Options:"
|
||||
echo " --xrpld PATH Path to xrpld binary"
|
||||
echo " --nodes NUM Number of validator nodes (default: 5)"
|
||||
echo " --rpc-rate RPS RPC load rate (default: 50)"
|
||||
echo " --rpc-duration SECS RPC load duration (default: 120)"
|
||||
echo " --tx-tps TPS Transaction submit rate (default: 5)"
|
||||
echo " --tx-duration SECS Transaction submit duration (default: 120)"
|
||||
echo " --profile NAME Workload profile (default: full-validation)"
|
||||
echo " --with-benchmark Also run performance overhead benchmark (telemetry off vs on)"
|
||||
echo " --skip-loki Skip Loki log-trace correlation checks"
|
||||
echo " --skip-regression Skip the OTel-baseline regression gate"
|
||||
echo " --cleanup Tear down everything and exit"
|
||||
echo " -h, --help Show this help"
|
||||
echo ""
|
||||
echo "Accepted but INERT (parsed for compatibility, then ignored):"
|
||||
echo " --rpc-rate RPS no effect"
|
||||
echo " --rpc-duration SECS no effect"
|
||||
echo " --tx-tps TPS no effect"
|
||||
echo " --tx-duration SECS no effect"
|
||||
echo ""
|
||||
echo " Load shape comes from the workload profile (--profile), which sets"
|
||||
echo " the rate and duration of every phase in workload-profiles.json."
|
||||
echo " These four flags stay accepted because the CI workflow passes them."
|
||||
exit 0
|
||||
}
|
||||
|
||||
@@ -101,6 +128,7 @@ while [ $# -gt 0 ]; do
|
||||
NUM_NODES="$2"
|
||||
shift 2
|
||||
;;
|
||||
# The next four are inert — see the RPC_RATE default above.
|
||||
--rpc-rate)
|
||||
RPC_RATE="$2"
|
||||
shift 2
|
||||
@@ -135,7 +163,10 @@ while [ $# -gt 0 ]; do
|
||||
;;
|
||||
--cleanup) # Cleanup mode
|
||||
log "Cleaning up..."
|
||||
pkill -f "$WORKDIR" 2>/dev/null || true
|
||||
# Match the node config path, not the bare workdir: a plain
|
||||
# "$WORKDIR" pattern also matches any shell, editor or log tail
|
||||
# whose command line merely mentions that path.
|
||||
pkill -f "$WORKDIR/node[0-9]+/xrpld\.cfg" 2>/dev/null || true
|
||||
docker compose -f "$COMPOSE_FILE" down 2>/dev/null || true
|
||||
rm -rf "$WORKDIR"
|
||||
ok "Cleanup complete."
|
||||
@@ -170,7 +201,8 @@ ok "Prerequisites verified."
|
||||
# Cleanup previous run
|
||||
# ---------------------------------------------------------------------------
|
||||
log "Cleaning up previous run..."
|
||||
pkill -f "$WORKDIR" 2>/dev/null || true
|
||||
# Narrowed for the same reason as the --cleanup branch above.
|
||||
pkill -f "$WORKDIR/node[0-9]+/xrpld\.cfg" 2>/dev/null || true
|
||||
sleep 2
|
||||
rm -rf "$WORKDIR"
|
||||
mkdir -p "$WORKDIR" "$REPORT_DIR"
|
||||
@@ -339,7 +371,13 @@ for attempt in $(seq 1 120); do
|
||||
break
|
||||
fi
|
||||
if [ "$attempt" -eq 120 ]; then
|
||||
warn "Consensus timeout — $ready/$NUM_NODES nodes ready"
|
||||
# Fatal, not a warning. A partial cluster still answers queries, so the
|
||||
# run would complete and report unrelated span/metric failures: series
|
||||
# counts scale with the number of live nodes, and spans that need a
|
||||
# quorum are simply never emitted. One infrastructure error here is
|
||||
# worth more than a pile of misleading assertion failures later.
|
||||
echo ""
|
||||
die "Consensus timeout — only $ready/$NUM_NODES nodes proposing after ${attempt}s. Check $WORKDIR/node*/debug.log, then '$0 --cleanup'."
|
||||
fi
|
||||
printf "\r %d/%d nodes proposing..." "$ready" "$NUM_NODES"
|
||||
sleep 1
|
||||
@@ -356,7 +394,13 @@ for attempt in $(seq 1 60); do
|
||||
ok "Validated ledger: seq $val_seq"
|
||||
break
|
||||
fi
|
||||
[ "$attempt" -eq 60 ] && warn "No validated ledger after 60s"
|
||||
# Fatal for the same reason as the consensus timeout above, and because
|
||||
# several assertions are gated on a validated ledger existing at all:
|
||||
# ledger_economy{metric="base_fee_xrp"} is only observed from a validated
|
||||
# ledger, and complete_ledgers stays absent while the range is empty.
|
||||
if [ "$attempt" -eq 60 ]; then
|
||||
die "No validated ledger after ${attempt}s (last seq: $val_seq). Check $WORKDIR/node*/debug.log, then '$0 --cleanup'."
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
@@ -370,14 +414,21 @@ for i in $(seq 1 "$NUM_NODES"); do
|
||||
WS_ENDPOINTS="$WS_ENDPOINTS ws://localhost:$((WS_PORT_BASE + i - 1))"
|
||||
done
|
||||
|
||||
ORCHESTRATOR_EXIT=0
|
||||
python3 "$SCRIPT_DIR/workload_orchestrator.py" \
|
||||
--profile "$WORKLOAD_PROFILE" \
|
||||
--endpoints $WS_ENDPOINTS \
|
||||
--report "$REPORT_DIR/workload-report.json" \
|
||||
--report-dir "$REPORT_DIR" ||
|
||||
warn "Workload orchestrator returned non-zero exit"
|
||||
--report-dir "$REPORT_DIR" || ORCHESTRATOR_EXIT=$?
|
||||
|
||||
ok "Workload orchestration complete."
|
||||
if [ "$ORCHESTRATOR_EXIT" -eq 0 ]; then
|
||||
ok "Workload orchestration complete."
|
||||
else
|
||||
# Treated as an infrastructure error: the span and metric assertions below
|
||||
# would be graded against traffic that was never generated.
|
||||
fail "Workload orchestrator failed (exit $ORCHESTRATOR_EXIT) — the checks below run against incomplete traffic"
|
||||
fold_exit 2
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Step 5: Run telemetry validation suite
|
||||
@@ -397,6 +448,7 @@ if [ "$VALIDATION_EXIT" -eq 0 ]; then
|
||||
else
|
||||
fail "Some telemetry validation checks failed (exit $VALIDATION_EXIT)"
|
||||
fi
|
||||
fold_exit "$VALIDATION_EXIT"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Step 6: Capture OTel timings and run the regression comparison
|
||||
@@ -443,18 +495,33 @@ if [ "$SKIP_REGRESSION" != true ]; then
|
||||
else
|
||||
warn "Regression gate skipped."
|
||||
fi
|
||||
fold_exit "$REGRESSION_EXIT"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Step 7: (Optional) Run overhead benchmark
|
||||
# ---------------------------------------------------------------------------
|
||||
BENCHMARK_EXIT=0
|
||||
if [ "$WITH_BENCHMARK" = true ]; then
|
||||
log "Step 7: Running performance benchmark..."
|
||||
bash "$SCRIPT_DIR/benchmark.sh" \
|
||||
--xrpld "$XRPLD" \
|
||||
--duration 120 \
|
||||
--nodes 3 \
|
||||
--output "$REPORT_DIR" ||
|
||||
warn "Benchmark returned non-zero exit"
|
||||
--output "$REPORT_DIR" || BENCHMARK_EXIT=$?
|
||||
|
||||
if [ "$BENCHMARK_EXIT" -eq 0 ]; then
|
||||
ok "Benchmark within overhead thresholds."
|
||||
elif [ "$BENCHMARK_EXIT" -eq 1 ]; then
|
||||
# A measured threshold breach — same class as a failed check.
|
||||
fail "Benchmark exceeded overhead thresholds (exit 1)"
|
||||
fold_exit 1
|
||||
else
|
||||
# benchmark.sh could not produce a usable measurement (e.g. incomplete
|
||||
# system metrics). Reported as an infrastructure error, not a perf
|
||||
# regression: nothing was measured, so nothing was breached.
|
||||
fail "Benchmark could not measure overhead (exit $BENCHMARK_EXIT) — treated as an infrastructure error"
|
||||
fold_exit 2
|
||||
fi
|
||||
fi
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -485,15 +552,13 @@ echo ""
|
||||
echo " To tear down:"
|
||||
echo " $0 --cleanup"
|
||||
echo ""
|
||||
echo " Step statuses (0 = ok):"
|
||||
echo " Workload orchestration: $ORCHESTRATOR_EXIT"
|
||||
echo " Telemetry validation: $VALIDATION_EXIT"
|
||||
echo " Regression gate: $REGRESSION_EXIT"
|
||||
echo " Overhead benchmark: $BENCHMARK_EXIT"
|
||||
echo ""
|
||||
echo "==========================================================="
|
||||
|
||||
# Fail the run if EITHER validation or the regression gate failed. The
|
||||
# `[ "$VAR" -gt N ]` comparison works here because exit codes are numeric.
|
||||
FINAL_EXIT=0
|
||||
if [ "$VALIDATION_EXIT" -ne 0 ]; then
|
||||
FINAL_EXIT="$VALIDATION_EXIT"
|
||||
fi
|
||||
if [ "$REGRESSION_EXIT" -ne 0 ] && [ "$FINAL_EXIT" -eq 0 ]; then
|
||||
FINAL_EXIT="$REGRESSION_EXIT"
|
||||
fi
|
||||
# FINAL_EXIT already holds the first non-zero step status (see fold_exit).
|
||||
exit "$FINAL_EXIT"
|
||||
|
||||
@@ -90,6 +90,57 @@ DEFAULT_TX_WEIGHTS: dict[str, int] = {
|
||||
# Number of test accounts to create.
|
||||
NUM_TEST_ACCOUNTS = 8
|
||||
|
||||
# Minimum number of funded accounts the transaction builders need: TX_BUILDERS
|
||||
# indexes positions 0..5 of the account list.
|
||||
MIN_FUNDED_ACCOUNTS = 6
|
||||
|
||||
# Engine results that tie up the submitted sequence number, other than the
|
||||
# tec* family which is matched by prefix. See consumes_sequence().
|
||||
SEQ_CONSUMING_RESULTS = frozenset({"tesSUCCESS", "terQUEUED"})
|
||||
|
||||
# Consecutive non-consuming submit results from one account before its
|
||||
# sequence is re-read from the ledger. The periodic refresh only ever raises
|
||||
# the counter, so a counter that leads the ledger -- a queued transaction that
|
||||
# was later dropped, say -- would otherwise make every further submit from
|
||||
# that account fail forever. Five is above the two or three rejections a
|
||||
# single bad transaction type can produce in a row, and at the default 5 TPS
|
||||
# it triggers within a few seconds, well before the periodic refresh below.
|
||||
SEQ_REFETCH_AFTER_FAILURES = 5
|
||||
|
||||
# How often the submission loop re-reads every account's sequence from the
|
||||
# ledger, to stay close to sequences other submitters have advanced.
|
||||
SEQ_REFRESH_INTERVAL_S = 10.0
|
||||
|
||||
|
||||
def consumes_sequence(engine_result: str | None) -> bool:
|
||||
"""Report whether an engine result tied up the submitted sequence number.
|
||||
|
||||
Two outcomes do. ``tesSUCCESS`` is applied (TER.h:243) and every ``tec*``
|
||||
claims the fee "to use the sequence number" (TER.h:265-266), so both
|
||||
advance the account root. ``terQUEUED`` is the one ``ter*`` code rippled
|
||||
forwards (TER.h:205); the transaction waits in the queue still holding
|
||||
that sequence, so the next submit needs the following one.
|
||||
|
||||
Everything else leaves the sequence free. ``tem*``, ``tef*`` and ``tel*``
|
||||
are neither applied nor forwarded, and neither are the remaining ``ter*``
|
||||
codes (TER.h:202-206) -- including ``terPRE_SEQ``, which reports that the
|
||||
submitted sequence is already past the account root (TER.h:217).
|
||||
Advancing the local counter on those widens the gap TER.h:209 calls a
|
||||
"hole in sequence which jams transactions", and it is what
|
||||
SEQ_REFETCH_AFTER_FAILURES exists to recover from.
|
||||
|
||||
Args:
|
||||
engine_result: The ``engine_result`` field of a submit response. A
|
||||
missing or JSON-null field reads as non-consuming
|
||||
rather than raising, so the caller records the
|
||||
transaction exactly once.
|
||||
|
||||
Returns:
|
||||
True if the submitted sequence number is spoken for.
|
||||
"""
|
||||
result = str(engine_result or "")
|
||||
return result in SEQ_CONSUMING_RESULTS or result.startswith("tec")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Data classes
|
||||
@@ -98,19 +149,28 @@ NUM_TEST_ACCOUNTS = 8
|
||||
|
||||
@dataclass
|
||||
class Account:
|
||||
"""Represents a funded XRPL test account.
|
||||
"""Represents an XRPL test account, funded or not.
|
||||
|
||||
Attributes:
|
||||
name: Human-readable name (e.g., "alice").
|
||||
account: Classic address (rXXX...).
|
||||
seed: Secret seed for signing.
|
||||
sequence: Next available sequence number.
|
||||
funded: True once the account root exists on the ledger. False
|
||||
accounts cannot submit, so they are excluded from the
|
||||
submission loop rather than silently counted.
|
||||
stalled: Consecutive submit results from this account that consumed
|
||||
no sequence. Reset by any consuming result; at
|
||||
SEQ_REFETCH_AFTER_FAILURES the sequence is re-read from
|
||||
the ledger.
|
||||
"""
|
||||
|
||||
name: str
|
||||
account: str
|
||||
seed: str
|
||||
sequence: int = 0
|
||||
funded: bool = False
|
||||
stalled: int = 0
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -163,7 +223,7 @@ class TxStats:
|
||||
|
||||
|
||||
async def ws_request(
|
||||
ws: websockets.WebSocketClientProtocol,
|
||||
ws: websockets.ClientConnection,
|
||||
command: str,
|
||||
params: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
@@ -204,7 +264,7 @@ async def ws_request(
|
||||
return resp.get("result", resp)
|
||||
|
||||
|
||||
async def create_account(ws: websockets.WebSocketClientProtocol, name: str) -> Account:
|
||||
async def create_account(ws: websockets.ClientConnection, name: str) -> Account:
|
||||
"""Create a new account via wallet_propose RPC.
|
||||
|
||||
Args:
|
||||
@@ -227,7 +287,7 @@ async def create_account(ws: websockets.WebSocketClientProtocol, name: str) -> A
|
||||
|
||||
|
||||
async def fund_account(
|
||||
ws: websockets.WebSocketClientProtocol,
|
||||
ws: websockets.ClientConnection,
|
||||
dest: Account,
|
||||
genesis_seq: int,
|
||||
) -> tuple[bool, int]:
|
||||
@@ -239,7 +299,8 @@ async def fund_account(
|
||||
genesis_seq: Current genesis account sequence number.
|
||||
|
||||
Returns:
|
||||
Tuple of (success: bool, next_sequence: int).
|
||||
Tuple of (funded: bool, next_genesis_sequence: int). The sequence is
|
||||
unchanged when the ledger did not consume it.
|
||||
"""
|
||||
resp = await ws_request(
|
||||
ws,
|
||||
@@ -265,12 +326,16 @@ async def fund_account(
|
||||
engine_result,
|
||||
json.dumps(resp, indent=None)[:500],
|
||||
)
|
||||
return success, genesis_seq + 1
|
||||
# Advance the genesis sequence only when the ledger consumed it. The
|
||||
# caller reads it once and threads it through every funding submit, so
|
||||
# advancing past a tem*/tef*/tel* rejection would put every remaining
|
||||
# submit on a future sequence and fund nothing.
|
||||
if consumes_sequence(engine_result):
|
||||
genesis_seq += 1
|
||||
return success, genesis_seq
|
||||
|
||||
|
||||
async def get_account_sequence(
|
||||
ws: websockets.WebSocketClientProtocol, account: str
|
||||
) -> int:
|
||||
async def get_account_sequence(ws: websockets.ClientConnection, account: str) -> int:
|
||||
"""Get the current sequence number for an account.
|
||||
|
||||
Args:
|
||||
@@ -564,7 +629,7 @@ TX_BUILDERS: dict[str, Any] = {
|
||||
|
||||
|
||||
async def setup_accounts(
|
||||
ws: websockets.WebSocketClientProtocol,
|
||||
ws: websockets.ClientConnection,
|
||||
) -> list[Account]:
|
||||
"""Create and fund test accounts from genesis.
|
||||
|
||||
@@ -575,7 +640,8 @@ async def setup_accounts(
|
||||
ws: Open WebSocket connection to a rippled node.
|
||||
|
||||
Returns:
|
||||
List of funded Account instances.
|
||||
Every created Account. ``funded`` marks the ones the ledger accepted,
|
||||
so the caller must filter on it rather than on the list length.
|
||||
"""
|
||||
account_names = ["alice", "bob", "carol", "dave", "eve", "frank", "grace", "heidi"]
|
||||
|
||||
@@ -593,8 +659,8 @@ async def setup_accounts(
|
||||
# Fund all accounts.
|
||||
logger.info("Funding test accounts...")
|
||||
for acct in accounts:
|
||||
success, genesis_seq = await fund_account(ws, acct, genesis_seq)
|
||||
if success:
|
||||
acct.funded, genesis_seq = await fund_account(ws, acct, genesis_seq)
|
||||
if acct.funded:
|
||||
logger.info(" Funded %s", acct.name)
|
||||
else:
|
||||
logger.warning(" Failed to fund %s", acct.name)
|
||||
@@ -603,19 +669,34 @@ async def setup_accounts(
|
||||
logger.info("Waiting 10s for funding transactions to validate...")
|
||||
await asyncio.sleep(10)
|
||||
|
||||
# Refresh sequence numbers for all accounts.
|
||||
# Refresh sequence numbers, and confirm funding against the ledger rather
|
||||
# than trusting the submit result. get_account_sequence returns 0 when
|
||||
# account_info reports no account_data, which means the account root was
|
||||
# never created; a sequence we cannot read also makes the account
|
||||
# unusable, so either way it must not be submitted from.
|
||||
for acct in accounts:
|
||||
try:
|
||||
acct.sequence = await get_account_sequence(ws, acct.account)
|
||||
logger.info(" %s sequence: %d", acct.name, acct.sequence)
|
||||
except Exception as exc:
|
||||
logger.warning(" Failed to get sequence for %s: %s", acct.name, exc)
|
||||
if acct.sequence > 0:
|
||||
logger.info(" %s sequence: %d", acct.name, acct.sequence)
|
||||
else:
|
||||
acct.funded = False
|
||||
logger.warning(
|
||||
" %s has no ledger sequence — treating as unfunded", acct.name
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Funded %d of %d created accounts",
|
||||
sum(1 for a in accounts if a.funded),
|
||||
len(accounts),
|
||||
)
|
||||
return accounts
|
||||
|
||||
|
||||
async def submit_transaction(
|
||||
ws: websockets.WebSocketClientProtocol,
|
||||
ws: websockets.ClientConnection,
|
||||
tx_type: str,
|
||||
accounts: list[Account],
|
||||
stats: TxStats,
|
||||
@@ -652,8 +733,13 @@ async def submit_transaction(
|
||||
)
|
||||
stats.record(tx_type, success)
|
||||
|
||||
# The sequence gate is deliberately not `success`: that tuple is
|
||||
# narrower (every tec* consumes a sequence, only two are listed) and
|
||||
# wider (a tem*/tef*/tel*/ter* rejection consumes none) than the set
|
||||
# of results that actually consume one. _track_sequence also owns the
|
||||
# recovery path for a counter that has drifted ahead of the ledger.
|
||||
if sender:
|
||||
sender.sequence += 1
|
||||
await _track_sequence(ws, sender, engine_result)
|
||||
|
||||
if not success:
|
||||
# First occurrence of each distinct result at WARNING, the rest at
|
||||
@@ -675,15 +761,77 @@ async def submit_transaction(
|
||||
_log_first_failure("exc:%s" % type(exc).__name__, "%s error: %s", tx_type, exc)
|
||||
|
||||
|
||||
async def _track_sequence(
|
||||
ws: websockets.ClientConnection,
|
||||
sender: Account,
|
||||
engine_result: str | None,
|
||||
) -> None:
|
||||
"""Advance, or re-read from the ledger, one sender's sequence number.
|
||||
|
||||
A consuming result moves the counter on by one and clears the stall
|
||||
streak. A non-consuming one leaves the counter where it is, so the same
|
||||
sequence is offered again -- correct when the rejection was about the
|
||||
transaction, but a livelock when the counter itself is the problem, since
|
||||
_refresh_sequences never lowers it. After SEQ_REFETCH_AFTER_FAILURES
|
||||
consecutive non-consuming results the ledger's own value is taken
|
||||
instead, in either direction.
|
||||
|
||||
A zero from get_account_sequence means account_info returned no
|
||||
account_data, so it is ignored rather than written over a usable counter.
|
||||
|
||||
Args:
|
||||
ws: Open WebSocket connection.
|
||||
sender: The account the transaction was submitted from.
|
||||
engine_result: The ``engine_result`` of that submit.
|
||||
"""
|
||||
if consumes_sequence(engine_result):
|
||||
sender.sequence += 1
|
||||
sender.stalled = 0
|
||||
return
|
||||
|
||||
sender.stalled += 1
|
||||
if sender.stalled < SEQ_REFETCH_AFTER_FAILURES:
|
||||
return
|
||||
|
||||
sender.stalled = 0
|
||||
try:
|
||||
seq = await get_account_sequence(ws, sender.account)
|
||||
except Exception as exc:
|
||||
logger.warning("Sequence re-fetch for %s failed: %s", sender.name, exc)
|
||||
return
|
||||
if seq > 0 and seq != sender.sequence:
|
||||
logger.warning(
|
||||
"Re-syncing %s sequence %d -> %d after %d non-consuming results",
|
||||
sender.name,
|
||||
sender.sequence,
|
||||
seq,
|
||||
SEQ_REFETCH_AFTER_FAILURES,
|
||||
)
|
||||
sender.sequence = seq
|
||||
|
||||
|
||||
async def _refresh_sequences(
|
||||
ws: websockets.WebSocketClientProtocol,
|
||||
ws: websockets.ClientConnection,
|
||||
accounts: list[Account],
|
||||
) -> None:
|
||||
"""Re-sync account sequences from the validated ledger.
|
||||
|
||||
In a consensus network, other nodes' transactions advance sequences
|
||||
beyond the submitter's local tracking. Refreshing every ~10 s keeps
|
||||
the local counter close to the ledger and prevents tefPAST_SEQ storms.
|
||||
beyond the submitter's local tracking. Refreshing every
|
||||
SEQ_REFRESH_INTERVAL_S keeps the local counter close to the ledger and
|
||||
prevents tefPAST_SEQ storms.
|
||||
|
||||
The counter is only ever raised here, never lowered, because it may
|
||||
legitimately lead the ledger: a queued transaction holds its sequence
|
||||
without having applied yet, and lowering the counter would reuse it.
|
||||
Raising it fixes the opposite case, where the ledger has moved on -- a
|
||||
submit response that was lost while the transaction applied, or another
|
||||
submitter using the same account.
|
||||
|
||||
That leaves one case this cannot fix: a counter that leads the ledger and
|
||||
never catches up, because the transaction it was advanced for was dropped
|
||||
rather than applied. _track_sequence handles that one by re-reading the
|
||||
ledger value in either direction.
|
||||
"""
|
||||
for acct in accounts:
|
||||
try:
|
||||
@@ -694,6 +842,59 @@ async def _refresh_sequences(
|
||||
pass
|
||||
|
||||
|
||||
async def _submission_loop(
|
||||
ws: websockets.ClientConnection,
|
||||
accounts: list[Account],
|
||||
weights: dict[str, int],
|
||||
duration: float,
|
||||
interval: float,
|
||||
stats: TxStats,
|
||||
) -> float:
|
||||
"""Submit a weighted transaction mix until ``duration`` elapses.
|
||||
|
||||
Args:
|
||||
ws: Open WebSocket connection.
|
||||
accounts: Funded accounts to submit from.
|
||||
weights: Transaction type distribution weights.
|
||||
duration: Run time in seconds.
|
||||
interval: Delay between submissions, i.e. 1 / target TPS.
|
||||
stats: TxStats instance to record results in.
|
||||
|
||||
Returns:
|
||||
Seconds actually spent in the loop.
|
||||
"""
|
||||
tx_types = list(weights.keys())
|
||||
tx_weights = [weights[t] for t in tx_types]
|
||||
|
||||
start = time.monotonic()
|
||||
last_seq_refresh = start
|
||||
while (time.monotonic() - start) < duration:
|
||||
# Periodically re-sync account sequences from the ledger so
|
||||
# locally-tracked sequences don't drift behind consensus.
|
||||
if (time.monotonic() - last_seq_refresh) >= SEQ_REFRESH_INTERVAL_S:
|
||||
await _refresh_sequences(ws, accounts)
|
||||
last_seq_refresh = time.monotonic()
|
||||
|
||||
tx_type = random.choices(tx_types, weights=tx_weights, k=1)[0]
|
||||
await submit_transaction(ws, tx_type, accounts, stats)
|
||||
await asyncio.sleep(interval)
|
||||
|
||||
# Progress logging every 50 transactions.
|
||||
if stats.total_submitted % 50 == 0 and stats.total_submitted > 0:
|
||||
elapsed = time.monotonic() - start
|
||||
logger.info(
|
||||
"Progress: %d submitted, %d success, %d errors, "
|
||||
"%.1f TPS (%.0fs elapsed)",
|
||||
stats.total_submitted,
|
||||
stats.total_success,
|
||||
stats.total_errors,
|
||||
stats.total_submitted / elapsed if elapsed > 0 else 0,
|
||||
elapsed,
|
||||
)
|
||||
|
||||
return time.monotonic() - start
|
||||
|
||||
|
||||
async def run_submitter(
|
||||
endpoint: str,
|
||||
tps: float,
|
||||
@@ -713,60 +914,40 @@ async def run_submitter(
|
||||
"""
|
||||
stats = TxStats()
|
||||
interval = 1.0 / tps if tps > 0 else 0.5
|
||||
elapsed = 0.0
|
||||
|
||||
ws = await websockets.connect(endpoint, ping_interval=20, ping_timeout=10)
|
||||
logger.info("Connected to %s", endpoint)
|
||||
|
||||
try:
|
||||
# Setup test accounts.
|
||||
accounts = await setup_accounts(ws)
|
||||
if len(accounts) < 6:
|
||||
logger.error("Need at least 6 funded accounts, got %d", len(accounts))
|
||||
# Setup test accounts. Every created account is returned whether or
|
||||
# not funding worked, so submit only from the funded ones — the
|
||||
# builders address accounts by position and an unfunded account there
|
||||
# would fail every transaction it is picked for.
|
||||
created = await setup_accounts(ws)
|
||||
accounts = [acct for acct in created if acct.funded]
|
||||
if len(accounts) < MIN_FUNDED_ACCOUNTS:
|
||||
logger.error(
|
||||
"Need at least %d funded accounts, only %d of %d created "
|
||||
"accounts were funded",
|
||||
MIN_FUNDED_ACCOUNTS,
|
||||
len(accounts),
|
||||
len(created),
|
||||
)
|
||||
return stats
|
||||
|
||||
# Build weighted command list.
|
||||
tx_types = list(weights.keys())
|
||||
tx_weights = [weights[t] for t in tx_types]
|
||||
|
||||
logger.info(
|
||||
"Starting TX submission: tps=%s, duration=%ss, types=%d",
|
||||
tps,
|
||||
duration,
|
||||
len(tx_types),
|
||||
len(weights),
|
||||
)
|
||||
elapsed = await _submission_loop(
|
||||
ws, accounts, weights, duration, interval, stats
|
||||
)
|
||||
|
||||
start = time.monotonic()
|
||||
last_seq_refresh = start
|
||||
seq_refresh_interval = 10.0
|
||||
while (time.monotonic() - start) < duration:
|
||||
# Periodically re-sync account sequences from the ledger so
|
||||
# locally-tracked sequences don't drift behind consensus.
|
||||
if (time.monotonic() - last_seq_refresh) >= seq_refresh_interval:
|
||||
await _refresh_sequences(ws, accounts)
|
||||
last_seq_refresh = time.monotonic()
|
||||
|
||||
tx_type = random.choices(tx_types, weights=tx_weights, k=1)[0]
|
||||
await submit_transaction(ws, tx_type, accounts, stats)
|
||||
await asyncio.sleep(interval)
|
||||
|
||||
# Progress logging every 50 transactions.
|
||||
if stats.total_submitted % 50 == 0 and stats.total_submitted > 0:
|
||||
elapsed = time.monotonic() - start
|
||||
actual_tps = stats.total_submitted / elapsed if elapsed > 0 else 0
|
||||
logger.info(
|
||||
"Progress: %d submitted, %d success, %d errors, "
|
||||
"%.1f TPS (%.0fs elapsed)",
|
||||
stats.total_submitted,
|
||||
stats.total_success,
|
||||
stats.total_errors,
|
||||
actual_tps,
|
||||
elapsed,
|
||||
)
|
||||
|
||||
finally:
|
||||
await ws.close()
|
||||
|
||||
elapsed = time.monotonic() - start
|
||||
logger.info(
|
||||
"Submission complete: %d submitted, %d success, %d errors "
|
||||
"in %.1fs (%.1f TPS)",
|
||||
|
||||
@@ -6,10 +6,13 @@ a workload run. Queries Tempo (spans), Prometheus (metrics), Loki (logs),
|
||||
and Grafana (dashboards) APIs to produce a pass/fail report.
|
||||
|
||||
Validation categories:
|
||||
1. Span validation — All 16+ span types present with required attributes
|
||||
2. Metric validation — SpanMetrics, StatsD, and Phase 9 metrics are non-zero
|
||||
1. Span validation — Every required span type in expected_spans.json, each
|
||||
carrying its required attributes
|
||||
2. Metric validation — SpanMetrics, StatsD, and MetricsRegistry OTLP metrics
|
||||
are non-zero
|
||||
3. Log-trace correlation — Loki logs contain trace_id/span_id fields
|
||||
4. Dashboard validation — All 14 Grafana dashboards render data
|
||||
4. Dashboard validation — Every dashboard uid in expected_metrics.json
|
||||
provisions and loads (panel count only, not panel data)
|
||||
5. External parity — Span attrs, metric existence, and value sanity for
|
||||
external dashboard parity (validator-health,
|
||||
peer-quality, node-health)
|
||||
@@ -27,6 +30,7 @@ Usage:
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import fnmatch
|
||||
import json
|
||||
import logging
|
||||
import sys
|
||||
@@ -62,6 +66,12 @@ EXPECTED_METRICS_FILE = SCRIPT_DIR / "expected_metrics.json"
|
||||
METRIC_POLL_TIMEOUT_SEC = 45.0
|
||||
METRIC_POLL_INTERVAL_SEC = 5.0
|
||||
|
||||
# All metrics are polled concurrently against ONE shared deadline, so the
|
||||
# metric phase costs a single poll window instead of one per metric. This caps
|
||||
# how many /api/v1/series requests are in flight at a time, so the fan-out does
|
||||
# not hammer the single-container Prometheus the harness runs.
|
||||
METRIC_POLL_CONCURRENCY = 8
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Data classes
|
||||
@@ -218,6 +228,26 @@ def _otlp_span_attr_keys(span: dict[str, Any]) -> set[str]:
|
||||
return {a["key"] for a in span.get("attributes", []) if "key" in a}
|
||||
|
||||
|
||||
def _span_name_matches(emitted_name: str, expected_name: str) -> bool:
|
||||
"""Test an emitted span name against a name from expected_spans.json.
|
||||
|
||||
Contract names are either literals or globs containing "*" (for example
|
||||
"rpc.command.*"). Literals are compared for exact equality so a longer
|
||||
emitted name cannot satisfy a shorter contract: "consensus.accept.apply"
|
||||
must not stand in for "consensus.accept".
|
||||
|
||||
Args:
|
||||
emitted_name: Span name as reported by Tempo.
|
||||
expected_name: Span name or glob pattern from expected_spans.json.
|
||||
|
||||
Returns:
|
||||
True when the emitted name satisfies the expected name.
|
||||
"""
|
||||
if "*" in expected_name:
|
||||
return fnmatch.fnmatchcase(emitted_name, expected_name)
|
||||
return emitted_name == expected_name
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Span Validation (Tempo API)
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -404,10 +434,20 @@ async def _validate_span_attributes_otlp(
|
||||
span_def: dict[str, Any],
|
||||
report: ValidationReport,
|
||||
) -> None:
|
||||
"""Check that OTLP spans contain expected attributes.
|
||||
"""Check that the contract's own span carries its required attributes.
|
||||
|
||||
Only spans whose name matches ``span_def["name"]`` are inspected.
|
||||
Attributes are never borrowed from siblings: many span types share keys
|
||||
such as ledger_seq or tx_hash, so a trace-wide scan would satisfy every
|
||||
one of those contracts from a single carrier span and make the per-span
|
||||
contract unenforceable.
|
||||
|
||||
A span type passes when at least one instance of it carries every required
|
||||
attribute. When none does, the closest instance's missing keys are
|
||||
reported.
|
||||
|
||||
Args:
|
||||
spans: List of OTLP span dicts from Tempo.
|
||||
spans: Every OTLP span dict in the fetched trace.
|
||||
span_def: Span definition from expected_spans.json.
|
||||
report: ValidationReport to accumulate results.
|
||||
"""
|
||||
@@ -416,26 +456,53 @@ async def _validate_span_attributes_otlp(
|
||||
return
|
||||
|
||||
span_name = span_def["name"]
|
||||
# Collect all attribute keys from all spans.
|
||||
found_attrs: set[str] = set()
|
||||
for span in spans:
|
||||
found_attrs.update(_otlp_span_attr_keys(span))
|
||||
check_name = f"span.attrs.{span_name}"
|
||||
matching = [s for s in spans if _span_name_matches(s.get("name", ""), span_name)]
|
||||
|
||||
if not matching:
|
||||
report.add(
|
||||
CheckResult(
|
||||
name=check_name,
|
||||
category="span",
|
||||
passed=False,
|
||||
message=(
|
||||
f"{span_name}: no span named '{span_name}' in the fetched "
|
||||
"trace, cannot verify its attributes"
|
||||
),
|
||||
details={"required": required_attrs, "instances": 0},
|
||||
)
|
||||
)
|
||||
return
|
||||
|
||||
# Keep the instance that is missing the fewest required attributes, so the
|
||||
# failure message names the closest witness rather than an arbitrary one.
|
||||
best_found: set[str] = set()
|
||||
best_missing: list[str] = list(required_attrs)
|
||||
for span in matching:
|
||||
found = _otlp_span_attr_keys(span)
|
||||
missing = [a for a in required_attrs if a not in found]
|
||||
if len(missing) < len(best_missing):
|
||||
best_found, best_missing = found, missing
|
||||
if not best_missing:
|
||||
break
|
||||
|
||||
missing = [a for a in required_attrs if a not in found_attrs]
|
||||
report.add(
|
||||
CheckResult(
|
||||
name=f"span.attrs.{span_name}",
|
||||
name=check_name,
|
||||
category="span",
|
||||
passed=len(missing) == 0,
|
||||
passed=not best_missing,
|
||||
message=(
|
||||
f"{span_name}: all {len(required_attrs)} attributes present"
|
||||
if not missing
|
||||
else f"{span_name}: missing attributes: {missing}"
|
||||
if not best_missing
|
||||
else f"{span_name}: no '{span_name}' span carried all "
|
||||
f"{len(required_attrs)} required attributes; closest of "
|
||||
f"{len(matching)} instance(s) missing {best_missing}"
|
||||
),
|
||||
details={
|
||||
"required": required_attrs,
|
||||
"found": list(found_attrs),
|
||||
"missing": missing,
|
||||
"found": sorted(best_found),
|
||||
"missing": best_missing,
|
||||
"instances": len(matching),
|
||||
},
|
||||
)
|
||||
)
|
||||
@@ -474,21 +541,21 @@ async def _validate_parent_child(
|
||||
)
|
||||
return
|
||||
|
||||
# Check if child spans exist within parent traces.
|
||||
# Use the concrete child name for wildcard patterns.
|
||||
concrete_child = child_name.replace("*", "server_info")
|
||||
# Check if child spans exist within parent traces. Names are matched
|
||||
# exactly (globs for wildcard contracts) — a substring test let a
|
||||
# longer emitted name satisfy a shorter contract, so
|
||||
# consensus.round -> consensus.accept passed on a
|
||||
# consensus.accept.apply span alone.
|
||||
found_child = False
|
||||
for trace_summary in traces:
|
||||
trace_id = trace_summary.get("traceID", "")
|
||||
if not trace_id:
|
||||
continue
|
||||
spans = await _tempo_get_trace(session, tempo_url, trace_id)
|
||||
for span in spans:
|
||||
op = span.get("name", "")
|
||||
if concrete_child in op or ("*" not in child_name and op == child_name):
|
||||
found_child = True
|
||||
break
|
||||
if found_child:
|
||||
if any(
|
||||
_span_name_matches(span.get("name", ""), child_name) for span in spans
|
||||
):
|
||||
found_child = True
|
||||
break
|
||||
|
||||
report.add(
|
||||
@@ -519,29 +586,25 @@ async def _validate_parent_child(
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def validate_metrics(
|
||||
session: aiohttp.ClientSession,
|
||||
prometheus_url: str,
|
||||
report: ValidationReport,
|
||||
async def _log_prometheus_metric_names(
|
||||
session: aiohttp.ClientSession, prometheus_url: str
|
||||
) -> None:
|
||||
"""Validate that expected metrics appear in Prometheus with non-zero values.
|
||||
"""Log the harness-relevant metric names Prometheus currently knows.
|
||||
|
||||
Diagnostic only — this output appears in CI logs and helps debug name
|
||||
mismatches between expected_metrics.json and actual emissions. Failures
|
||||
are warnings, never check failures.
|
||||
|
||||
Args:
|
||||
session: aiohttp client session.
|
||||
prometheus_url: Base URL for Prometheus API (e.g., http://localhost:9090).
|
||||
report: ValidationReport to accumulate results.
|
||||
prometheus_url: Prometheus base URL.
|
||||
"""
|
||||
logger.info("--- Metric Validation (Prometheus) ---")
|
||||
|
||||
# Diagnostic: list all metric names in Prometheus. Helps debug name
|
||||
# mismatches between expected_metrics.json and actual emissions.
|
||||
try:
|
||||
async with session.get(
|
||||
f"{prometheus_url}/api/v1/label/__name__/values"
|
||||
) as resp:
|
||||
label_data = await resp.json()
|
||||
all_metrics = label_data.get("data", [])
|
||||
# Log relevant metrics for debugging.
|
||||
relevant = [
|
||||
m
|
||||
for m in all_metrics
|
||||
@@ -577,19 +640,99 @@ async def validate_metrics(
|
||||
except Exception as exc:
|
||||
logger.warning("Failed to fetch Prometheus metric names: %s", exc)
|
||||
|
||||
|
||||
async def validate_metrics(
|
||||
session: aiohttp.ClientSession,
|
||||
prometheus_url: str,
|
||||
report: ValidationReport,
|
||||
) -> None:
|
||||
"""Validate that expected metrics appear in Prometheus with non-zero values.
|
||||
|
||||
Args:
|
||||
session: aiohttp client session.
|
||||
prometheus_url: Base URL for Prometheus API (e.g., http://localhost:9090).
|
||||
report: ValidationReport to accumulate results.
|
||||
"""
|
||||
logger.info("--- Metric Validation (Prometheus) ---")
|
||||
|
||||
await _log_prometheus_metric_names(session, prometheus_url)
|
||||
|
||||
with open(EXPECTED_METRICS_FILE) as f:
|
||||
expected = json.load(f)
|
||||
|
||||
# Check each metric category.
|
||||
for category_key, category_data in expected.items():
|
||||
if category_key in ("description", "grafana_dashboards"):
|
||||
continue
|
||||
# Flatten every (category, metric) pair the contract asserts, then poll
|
||||
# them concurrently against ONE shared deadline. Polling them serially made
|
||||
# each metric own its own timeout, so the waits were additive: 58 metrics x
|
||||
# 45 s = 43.5 min, which overran the CI job budget and lost the
|
||||
# artifact-upload and summary diagnostics. Sharing the deadline bounds the
|
||||
# whole phase to a single poll window.
|
||||
targets = [
|
||||
(category_key, metric_name)
|
||||
for category_key, category_data in expected.items()
|
||||
if category_key not in ("description", "grafana_dashboards")
|
||||
for metric_name in category_data.get("metrics", [])
|
||||
]
|
||||
|
||||
metrics = category_data.get("metrics", [])
|
||||
for metric_name in metrics:
|
||||
await _check_prometheus_metric(
|
||||
session, prometheus_url, metric_name, category_key, report
|
||||
deadline = time.monotonic() + METRIC_POLL_TIMEOUT_SEC
|
||||
sem = asyncio.Semaphore(METRIC_POLL_CONCURRENCY)
|
||||
checks = await asyncio.gather(
|
||||
*(
|
||||
_check_prometheus_metric(
|
||||
session, prometheus_url, metric_name, category, deadline, sem
|
||||
)
|
||||
for category, metric_name in targets
|
||||
)
|
||||
)
|
||||
|
||||
# Add in contract order, not completion order, so the report and its log
|
||||
# lines stay deterministic across runs.
|
||||
for check in checks:
|
||||
report.add(check)
|
||||
|
||||
|
||||
async def _poll_series_count(
|
||||
session: aiohttp.ClientSession,
|
||||
prometheus_url: str,
|
||||
metric_name: str,
|
||||
deadline: float,
|
||||
sem: asyncio.Semaphore,
|
||||
) -> int:
|
||||
"""Poll Prometheus until a metric has series or the deadline passes.
|
||||
|
||||
Uses the /api/v1/series endpoint instead of an instant query.
|
||||
Beast::insight StatsD gauges only mark dirty on value *changes*, so a gauge
|
||||
that stabilizes (e.g. peer count stays at 1) may go stale in Prometheus and
|
||||
disappear from instant queries. The series endpoint returns any metric
|
||||
that existed in the window, regardless of staleness.
|
||||
|
||||
Polls rather than querying once: late-populating gauges/counters may not
|
||||
have completed the export+scrape pipeline when this runs, so a single query
|
||||
races. A metric that never appears still fails once the deadline passes.
|
||||
|
||||
Args:
|
||||
session: aiohttp client session.
|
||||
prometheus_url: Prometheus base URL.
|
||||
metric_name: Prometheus metric name.
|
||||
deadline: Monotonic deadline shared by every metric in the run.
|
||||
sem: Bounds how many requests reach Prometheus at once. It
|
||||
is held only across the request, never across the
|
||||
sleep, so one absent metric cannot starve the others.
|
||||
|
||||
Returns:
|
||||
Number of series found, or 0 if the metric never appeared.
|
||||
"""
|
||||
params: dict[str, str] = {"match[]": metric_name}
|
||||
while True:
|
||||
async with sem:
|
||||
async with session.get(
|
||||
f"{prometheus_url}/api/v1/series", params=params
|
||||
) as resp:
|
||||
data = await resp.json()
|
||||
series_count = len(data.get("data", []))
|
||||
if series_count > 0 or time.monotonic() >= deadline:
|
||||
return series_count
|
||||
# Never sleep past the shared deadline.
|
||||
await asyncio.sleep(min(METRIC_POLL_INTERVAL_SEC, deadline - time.monotonic()))
|
||||
|
||||
|
||||
async def _check_prometheus_metric(
|
||||
@@ -597,8 +740,9 @@ async def _check_prometheus_metric(
|
||||
prometheus_url: str,
|
||||
metric_name: str,
|
||||
category: str,
|
||||
report: ValidationReport,
|
||||
) -> None:
|
||||
deadline: float,
|
||||
sem: asyncio.Semaphore,
|
||||
) -> CheckResult:
|
||||
"""Query Prometheus for a specific metric and check it exists.
|
||||
|
||||
Args:
|
||||
@@ -606,54 +750,34 @@ async def _check_prometheus_metric(
|
||||
prometheus_url: Prometheus base URL.
|
||||
metric_name: Prometheus metric name.
|
||||
category: Metric category for the report.
|
||||
report: ValidationReport to accumulate results.
|
||||
deadline: Monotonic deadline shared by every metric in the run.
|
||||
sem: Bounds how many requests reach Prometheus at once.
|
||||
|
||||
Returns:
|
||||
The CheckResult for this metric. The caller adds it to the report so
|
||||
report order follows the contract file rather than completion order.
|
||||
"""
|
||||
try:
|
||||
# Use the /api/v1/series endpoint instead of an instant query.
|
||||
# Beast::insight StatsD gauges only mark dirty on value *changes*,
|
||||
# so a gauge that stabilizes (e.g. peer count stays at 1) may go
|
||||
# stale in Prometheus and disappear from instant queries. The
|
||||
# series endpoint returns any metric that existed in the window,
|
||||
# regardless of staleness.
|
||||
#
|
||||
# Poll rather than query once: late-populating gauges/counters may
|
||||
# not have completed the export+scrape pipeline when this runs, so a
|
||||
# single query races. Re-query until the metric appears or the poll
|
||||
# window elapses; a metric that never appears still fails after the
|
||||
# timeout.
|
||||
params: dict[str, str] = {"match[]": metric_name}
|
||||
series_count = 0
|
||||
deadline = time.monotonic() + METRIC_POLL_TIMEOUT_SEC
|
||||
while True:
|
||||
async with session.get(
|
||||
f"{prometheus_url}/api/v1/series", params=params
|
||||
) as resp:
|
||||
data = await resp.json()
|
||||
series_count = len(data.get("data", []))
|
||||
if series_count > 0 or time.monotonic() >= deadline:
|
||||
break
|
||||
await asyncio.sleep(METRIC_POLL_INTERVAL_SEC)
|
||||
report.add(
|
||||
CheckResult(
|
||||
name=f"metric.{category}.{metric_name}",
|
||||
category="metric",
|
||||
passed=series_count > 0,
|
||||
message=(
|
||||
f"{metric_name}: {series_count} series"
|
||||
if series_count > 0
|
||||
else f"{metric_name}: 0 series (expected > 0)"
|
||||
),
|
||||
details={"series_count": series_count},
|
||||
)
|
||||
series_count = await _poll_series_count(
|
||||
session, prometheus_url, metric_name, deadline, sem
|
||||
)
|
||||
return CheckResult(
|
||||
name=f"metric.{category}.{metric_name}",
|
||||
category="metric",
|
||||
passed=series_count > 0,
|
||||
message=(
|
||||
f"{metric_name}: {series_count} series"
|
||||
if series_count > 0
|
||||
else f"{metric_name}: 0 series (expected > 0)"
|
||||
),
|
||||
details={"series_count": series_count},
|
||||
)
|
||||
except Exception as exc:
|
||||
report.add(
|
||||
CheckResult(
|
||||
name=f"metric.{category}.{metric_name}",
|
||||
category="metric",
|
||||
passed=False,
|
||||
message=f"{metric_name}: query failed ({exc})",
|
||||
)
|
||||
return CheckResult(
|
||||
name=f"metric.{category}.{metric_name}",
|
||||
category="metric",
|
||||
passed=False,
|
||||
message=f"{metric_name}: query failed ({exc})",
|
||||
)
|
||||
|
||||
|
||||
@@ -1069,6 +1193,131 @@ async def validate_parity_span_attrs(
|
||||
)
|
||||
|
||||
|
||||
def _series_label(series: dict[str, Any]) -> str:
|
||||
"""Name a Prometheus series for use in a failure message.
|
||||
|
||||
Args:
|
||||
series: One entry from a Prometheus query result.
|
||||
|
||||
Returns:
|
||||
The series' service_instance_id when it carries one (the label that
|
||||
tells harness cluster nodes apart), else its full label set.
|
||||
"""
|
||||
metric = series.get("metric", {})
|
||||
instance = metric.get("service_instance_id")
|
||||
if instance:
|
||||
return f"service_instance_id={instance}"
|
||||
return str(metric) if metric else "<unlabelled series>"
|
||||
|
||||
|
||||
def _value_in_bounds(
|
||||
value: float, lo: float, hi: float | None, exclusive_lo: bool
|
||||
) -> bool:
|
||||
"""Test one sample against a sanity range.
|
||||
|
||||
Args:
|
||||
value: Sample value.
|
||||
lo: Lower bound.
|
||||
hi: Upper bound, or None when unbounded above.
|
||||
exclusive_lo: True when the lower bound is exclusive.
|
||||
|
||||
Returns:
|
||||
True when the value is inside the range.
|
||||
"""
|
||||
lo_ok = value > lo if exclusive_lo else value >= lo
|
||||
return lo_ok and (hi is None or value <= hi)
|
||||
|
||||
|
||||
def _bounds_description(lo: float, hi: float | None, exclusive_lo: bool) -> str:
|
||||
"""Build the human-readable bound text used in check messages.
|
||||
|
||||
Args:
|
||||
lo: Lower bound.
|
||||
hi: Upper bound, or None when unbounded above.
|
||||
exclusive_lo: True when the lower bound is exclusive.
|
||||
|
||||
Returns:
|
||||
A phrase such as "> 0 and <= 100".
|
||||
"""
|
||||
desc = f"{'>' if exclusive_lo else '>='} {lo}"
|
||||
if hi is not None:
|
||||
desc += f" and <= {hi}"
|
||||
return desc
|
||||
|
||||
|
||||
async def _check_parity_value(
|
||||
session: aiohttp.ClientSession,
|
||||
prometheus_url: str,
|
||||
entry: dict[str, Any],
|
||||
) -> CheckResult:
|
||||
"""Bounds-check every series returned by one parity sanity query.
|
||||
|
||||
Args:
|
||||
session: aiohttp client session.
|
||||
prometheus_url: Prometheus API base URL.
|
||||
entry: One PARITY_VALUE_SANITY entry.
|
||||
|
||||
Returns:
|
||||
A CheckResult that fails if any series is out of bounds, naming each
|
||||
offending series.
|
||||
"""
|
||||
name = entry["name"]
|
||||
lo = entry["lo"]
|
||||
hi = entry["hi"]
|
||||
exclusive_lo = entry.get("exclusive_lo", False)
|
||||
check_name = f"parity.value_sanity.{name}"
|
||||
|
||||
try:
|
||||
async with session.get(
|
||||
f"{prometheus_url}/api/v1/query", params={"query": entry["query"]}
|
||||
) as resp:
|
||||
data = await resp.json()
|
||||
results = data.get("data", {}).get("result", [])
|
||||
|
||||
if not results:
|
||||
return CheckResult(
|
||||
name=check_name,
|
||||
category="parity",
|
||||
passed=False,
|
||||
message=f"{name}: no data returned from Prometheus",
|
||||
)
|
||||
|
||||
values: list[float] = []
|
||||
offenders: list[str] = []
|
||||
for series in results:
|
||||
value = float(series["value"][1])
|
||||
values.append(value)
|
||||
if not _value_in_bounds(value, lo, hi, exclusive_lo):
|
||||
offenders.append(f"{_series_label(series)} value {value}")
|
||||
|
||||
bound_desc = _bounds_description(lo, hi, exclusive_lo)
|
||||
return CheckResult(
|
||||
name=check_name,
|
||||
category="parity",
|
||||
passed=not offenders,
|
||||
message=(
|
||||
f"{name}: all {len(values)} series within bounds ({bound_desc})"
|
||||
if not offenders
|
||||
else f"{name}: {len(offenders)} of {len(values)} series out of "
|
||||
f"bounds (expected {bound_desc}): " + "; ".join(offenders)
|
||||
),
|
||||
details={
|
||||
"values": values,
|
||||
"series_count": len(values),
|
||||
"out_of_bounds": offenders,
|
||||
"lo": lo,
|
||||
"hi": hi,
|
||||
},
|
||||
)
|
||||
except Exception as exc:
|
||||
return CheckResult(
|
||||
name=check_name,
|
||||
category="parity",
|
||||
passed=False,
|
||||
message=f"{name}: sanity check failed ({exc})",
|
||||
)
|
||||
|
||||
|
||||
async def validate_parity_value_sanity(
|
||||
session: aiohttp.ClientSession,
|
||||
prometheus_url: str,
|
||||
@@ -1076,8 +1325,11 @@ async def validate_parity_value_sanity(
|
||||
) -> None:
|
||||
"""Validate that external-parity metric values fall within sane bounds.
|
||||
|
||||
For each entry in PARITY_VALUE_SANITY, queries the current value from
|
||||
Prometheus and checks it against the specified [lo, hi] range.
|
||||
For each entry in PARITY_VALUE_SANITY, queries Prometheus and checks
|
||||
*every* returned series against the specified [lo, hi] range. These
|
||||
queries are bare selectors with no aggregation, so a multi-node harness
|
||||
cluster returns one series per service_instance_id; checking only the
|
||||
first would let an out-of-range node pass silently.
|
||||
|
||||
Args:
|
||||
session: aiohttp client session.
|
||||
@@ -1087,73 +1339,7 @@ async def validate_parity_value_sanity(
|
||||
logger.info("--- External Parity: Value Sanity Checks ---")
|
||||
|
||||
for entry in PARITY_VALUE_SANITY:
|
||||
name = entry["name"]
|
||||
query = entry["query"]
|
||||
lo = entry["lo"]
|
||||
hi = entry["hi"]
|
||||
exclusive_lo = entry.get("exclusive_lo", False)
|
||||
check_name = f"parity.value_sanity.{name}"
|
||||
|
||||
try:
|
||||
params = {"query": query}
|
||||
async with session.get(
|
||||
f"{prometheus_url}/api/v1/query", params=params
|
||||
) as resp:
|
||||
data = await resp.json()
|
||||
results = data.get("data", {}).get("result", [])
|
||||
|
||||
if not results:
|
||||
report.add(
|
||||
CheckResult(
|
||||
name=check_name,
|
||||
category="parity",
|
||||
passed=False,
|
||||
message=f"{name}: no data returned from Prometheus",
|
||||
)
|
||||
)
|
||||
continue
|
||||
|
||||
# Use the first result's value.
|
||||
value = float(results[0]["value"][1])
|
||||
|
||||
# Check bounds.
|
||||
in_range = True
|
||||
if exclusive_lo:
|
||||
in_range = in_range and (value > lo)
|
||||
else:
|
||||
in_range = in_range and (value >= lo)
|
||||
if hi is not None:
|
||||
in_range = in_range and (value <= hi)
|
||||
|
||||
# Build human-readable bound description.
|
||||
lo_op = ">" if exclusive_lo else ">="
|
||||
bound_desc = f"{lo_op} {lo}"
|
||||
if hi is not None:
|
||||
bound_desc += f" and <= {hi}"
|
||||
|
||||
report.add(
|
||||
CheckResult(
|
||||
name=check_name,
|
||||
category="parity",
|
||||
passed=in_range,
|
||||
message=(
|
||||
f"{name}: value {value} is within bounds ({bound_desc})"
|
||||
if in_range
|
||||
else f"{name}: value {value} out of bounds "
|
||||
f"(expected {bound_desc})"
|
||||
),
|
||||
details={"value": value, "lo": lo, "hi": hi},
|
||||
)
|
||||
)
|
||||
except Exception as exc:
|
||||
report.add(
|
||||
CheckResult(
|
||||
name=check_name,
|
||||
category="parity",
|
||||
passed=False,
|
||||
message=f"{name}: sanity check failed ({exc})",
|
||||
)
|
||||
)
|
||||
report.add(await _check_parity_value(session, prometheus_url, entry))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -53,6 +53,28 @@ logger = logging.getLogger("workload_orchestrator")
|
||||
SCRIPT_DIR = Path(__file__).parent.resolve()
|
||||
PROFILES_FILE = SCRIPT_DIR / "workload-profiles.json"
|
||||
|
||||
# Wall-clock allowance for a generator on top of its phase's configured
|
||||
# duration. It has to cover the work the generators do outside their timed
|
||||
# loop: tx_submitter.py creates and funds 8 accounts (~25 WebSocket round
|
||||
# trips) and then waits a fixed 10s for those funding transactions to
|
||||
# validate, and both generators drain in-flight requests while shutting down.
|
||||
# A generator that outruns this is killed and the phase records the timeout as
|
||||
# an error, so one wedged process can no longer stall the whole profile.
|
||||
SUBPROCESS_GRACE_SEC = 90.0
|
||||
|
||||
# How long to keep reading a killed process's output before giving up on it.
|
||||
SUBPROCESS_DRAIN_TIMEOUT_SEC = 10.0
|
||||
|
||||
# Read size for the pipe readers. Only bounds one read() call, not the total.
|
||||
PIPE_READ_CHUNK_BYTES = 65536
|
||||
|
||||
# Error-rate ceilings for the exit gate. The TX ceiling is higher because
|
||||
# short-lived CI test environments lack pre-funded accounts, causing expected
|
||||
# failures for complex transactions (AMMCreate, EscrowFinish, etc.) that
|
||||
# require specific ledger state.
|
||||
RPC_ERROR_RATE_LIMIT_PCT = 50.0
|
||||
TX_ERROR_RATE_LIMIT_PCT = 95.0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Data classes
|
||||
@@ -146,15 +168,45 @@ def load_profile(profile_name: str) -> dict[str, Any]:
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
async def run_subprocess(cmd: list[str], label: str) -> tuple[int, str, str]:
|
||||
"""Run a subprocess and capture its stdout and stderr.
|
||||
async def _accumulate(stream: asyncio.StreamReader, chunks: list[bytes]) -> None:
|
||||
"""Read a subprocess pipe to EOF, appending as it goes.
|
||||
|
||||
Appending to a caller-owned list, rather than returning at EOF, means
|
||||
everything read so far survives even if this task never reaches EOF.
|
||||
|
||||
Args:
|
||||
cmd: Command and arguments.
|
||||
label: Human-readable label for logging.
|
||||
stream: Pipe to read.
|
||||
chunks: List the caller reads once the process has exited.
|
||||
"""
|
||||
while True:
|
||||
chunk = await stream.read(PIPE_READ_CHUNK_BYTES)
|
||||
if not chunk:
|
||||
return
|
||||
chunks.append(chunk)
|
||||
|
||||
|
||||
async def run_subprocess(
|
||||
cmd: list[str], label: str, timeout: float
|
||||
) -> tuple[int, str, str]:
|
||||
"""Run a subprocess to completion, or kill it once ``timeout`` expires.
|
||||
|
||||
A generator that wedges used to block its phase — and so the rest of the
|
||||
profile — until something outside the orchestrator killed the whole run,
|
||||
destroying the report with it. Bounding the wait lets the orchestrator kill
|
||||
the process, keep the output it had already produced, and report the phase
|
||||
as failed.
|
||||
|
||||
Both pipes are drained by separate tasks for the process's whole life, so a
|
||||
chatty generator can never fill a pipe buffer and stall waiting to write.
|
||||
|
||||
Args:
|
||||
cmd: Command and arguments.
|
||||
label: Human-readable label for logging.
|
||||
timeout: Wall-clock limit in seconds.
|
||||
|
||||
Returns:
|
||||
Tuple of (return_code, stdout_text, stderr_text).
|
||||
Tuple of (return_code, stdout_text, stderr_text). On timeout the return
|
||||
code is non-zero and the timeout is appended to the stderr text.
|
||||
"""
|
||||
logger.debug("Starting %s: %s", label, " ".join(cmd))
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
@@ -162,15 +214,49 @@ async def run_subprocess(cmd: list[str], label: str) -> tuple[int, str, str]:
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
)
|
||||
stdout, stderr = await proc.communicate()
|
||||
if proc.returncode != 0:
|
||||
|
||||
out_chunks: list[bytes] = []
|
||||
err_chunks: list[bytes] = []
|
||||
readers = [
|
||||
asyncio.create_task(_accumulate(proc.stdout, out_chunks)),
|
||||
asyncio.create_task(_accumulate(proc.stderr, err_chunks)),
|
||||
]
|
||||
|
||||
timed_out = False
|
||||
try:
|
||||
# asyncio.wait_for raises the builtin TimeoutError on Python 3.11+.
|
||||
await asyncio.wait_for(proc.wait(), timeout=timeout)
|
||||
except TimeoutError:
|
||||
timed_out = True
|
||||
logger.error("%s exceeded its %.0fs budget — killing it", label, timeout)
|
||||
proc.kill()
|
||||
try:
|
||||
await asyncio.wait_for(proc.wait(), timeout=SUBPROCESS_DRAIN_TIMEOUT_SEC)
|
||||
except TimeoutError:
|
||||
logger.error("%s did not exit after being killed", label)
|
||||
|
||||
# The pipes reach EOF once the process is gone, which ends both readers.
|
||||
_, pending = await asyncio.wait(readers, timeout=SUBPROCESS_DRAIN_TIMEOUT_SEC)
|
||||
for task in pending:
|
||||
logger.error("%s output pipe stayed open — captured output truncated", label)
|
||||
task.cancel()
|
||||
|
||||
stderr_text = b"".join(err_chunks).decode(errors="replace")
|
||||
if timed_out:
|
||||
# Appended, not prepended: callers keep only the tail of stderr.
|
||||
stderr_text += f"\ntimed out after {timeout:.0f}s and was killed"
|
||||
|
||||
# A process whose exit was never collected reports no code; call it SIGKILL
|
||||
# so the status is still non-zero and the phase records an error.
|
||||
returncode = proc.returncode if proc.returncode is not None else -9
|
||||
if returncode != 0:
|
||||
logger.warning(
|
||||
"%s exited with code %d: %s",
|
||||
label,
|
||||
proc.returncode,
|
||||
stderr.decode().strip()[-500:],
|
||||
returncode,
|
||||
stderr_text.strip()[-500:],
|
||||
)
|
||||
return proc.returncode, stdout.decode(), stderr.decode()
|
||||
return returncode, b"".join(out_chunks).decode(errors="replace"), stderr_text
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -259,6 +345,49 @@ def _build_tx_cmd(
|
||||
return cmd
|
||||
|
||||
|
||||
def _launch_phase_tasks(
|
||||
phase: dict[str, Any],
|
||||
endpoints: list[str],
|
||||
report_dir: Path,
|
||||
prefix: str,
|
||||
) -> list[tuple[str, Path, asyncio.Task]]:
|
||||
"""Start the generators this phase configures.
|
||||
|
||||
Each generator is given the phase duration plus SUBPROCESS_GRACE_SEC, so a
|
||||
wedged one is killed instead of stalling the phase.
|
||||
|
||||
Args:
|
||||
phase: Phase dict from the profile.
|
||||
endpoints: List of WebSocket endpoint URLs.
|
||||
report_dir: Directory for per-phase JSON reports.
|
||||
prefix: Report filename prefix for this phase.
|
||||
|
||||
Returns:
|
||||
List of (label, report_path, task) for every generator started; empty
|
||||
when the phase configures no workload.
|
||||
"""
|
||||
name = phase["name"]
|
||||
duration = phase["duration_sec"]
|
||||
timeout = duration + SUBPROCESS_GRACE_SEC
|
||||
tasks: list[tuple[str, Path, asyncio.Task]] = []
|
||||
|
||||
rpc_cfg = phase.get("rpc")
|
||||
if rpc_cfg:
|
||||
rpc_out = report_dir / f"{prefix}-rpc.json"
|
||||
cmd = _build_rpc_cmd(endpoints, rpc_cfg, duration, rpc_out)
|
||||
task = asyncio.create_task(run_subprocess(cmd, f"RPC [{name}]", timeout))
|
||||
tasks.append(("rpc", rpc_out, task))
|
||||
|
||||
tx_cfg = phase.get("tx")
|
||||
if tx_cfg:
|
||||
tx_out = report_dir / f"{prefix}-tx.json"
|
||||
cmd = _build_tx_cmd(endpoints[0], tx_cfg, duration, tx_out)
|
||||
task = asyncio.create_task(run_subprocess(cmd, f"TX [{name}]", timeout))
|
||||
tasks.append(("tx", tx_out, task))
|
||||
|
||||
return tasks
|
||||
|
||||
|
||||
async def run_phase(
|
||||
phase: dict[str, Any],
|
||||
endpoints: list[str],
|
||||
@@ -292,24 +421,8 @@ async def run_phase(
|
||||
phase.get("description", ""),
|
||||
)
|
||||
|
||||
tasks: list[tuple[str, Path, asyncio.Task]] = []
|
||||
t0 = time.monotonic()
|
||||
|
||||
rpc_cfg = phase.get("rpc")
|
||||
if rpc_cfg:
|
||||
rpc_out = report_dir / f"{prefix}-rpc.json"
|
||||
cmd = _build_rpc_cmd(endpoints, rpc_cfg, duration, rpc_out)
|
||||
tasks.append(
|
||||
("rpc", rpc_out, asyncio.create_task(run_subprocess(cmd, f"RPC [{name}]")))
|
||||
)
|
||||
|
||||
tx_cfg = phase.get("tx")
|
||||
if tx_cfg:
|
||||
tx_out = report_dir / f"{prefix}-tx.json"
|
||||
cmd = _build_tx_cmd(endpoints[0], tx_cfg, duration, tx_out)
|
||||
tasks.append(
|
||||
("tx", tx_out, asyncio.create_task(run_subprocess(cmd, f"TX [{name}]")))
|
||||
)
|
||||
tasks = _launch_phase_tasks(phase, endpoints, report_dir, prefix)
|
||||
|
||||
if not tasks:
|
||||
logger.warning(
|
||||
@@ -406,6 +519,58 @@ async def run_profile(
|
||||
return report
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Exit gate
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def evaluate_exit_gate(report: dict[str, Any]) -> list[str]:
|
||||
"""Decide whether a finished run should fail, and say why.
|
||||
|
||||
Three independent conditions fail a run:
|
||||
* a phase recorded an error — a generator exited non-zero, was killed on
|
||||
timeout, or wrote a report that could not be parsed,
|
||||
* the RPC error rate exceeded RPC_ERROR_RATE_LIMIT_PCT,
|
||||
* the TX error rate exceeded TX_ERROR_RATE_LIMIT_PCT.
|
||||
|
||||
The phase errors have to be judged separately from the two rates. A
|
||||
generator that crashes writes no report, so its totals stay 0, both rates
|
||||
short-circuit to 0, and a rate-only gate passes a run in which no traffic
|
||||
was generated at all.
|
||||
|
||||
Args:
|
||||
report: Combined report produced by run_profile.
|
||||
|
||||
Returns:
|
||||
One human-readable reason per failure; empty when the run passed.
|
||||
"""
|
||||
reasons: list[str] = []
|
||||
|
||||
for phase in report.get("phases", []):
|
||||
for error in phase.get("errors", []):
|
||||
reasons.append(f"phase '{phase.get('name', '?')}': {error}")
|
||||
|
||||
totals = report.get("totals", {})
|
||||
rpc_sent = totals.get("rpc_sent", 0)
|
||||
tx_submitted = totals.get("tx_submitted", 0)
|
||||
rpc_err_rate = totals.get("rpc_errors", 0) / rpc_sent * 100 if rpc_sent > 0 else 0.0
|
||||
tx_err_rate = (
|
||||
totals.get("tx_errors", 0) / tx_submitted * 100 if tx_submitted > 0 else 0.0
|
||||
)
|
||||
|
||||
if rpc_err_rate > RPC_ERROR_RATE_LIMIT_PCT:
|
||||
reasons.append(
|
||||
f"RPC error rate {rpc_err_rate:.1f}% exceeds "
|
||||
f"{RPC_ERROR_RATE_LIMIT_PCT}% of {rpc_sent} requests"
|
||||
)
|
||||
if tx_err_rate > TX_ERROR_RATE_LIMIT_PCT:
|
||||
reasons.append(
|
||||
f"TX error rate {tx_err_rate:.1f}% exceeds "
|
||||
f"{TX_ERROR_RATE_LIMIT_PCT}% of {tx_submitted} submissions"
|
||||
)
|
||||
return reasons
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CLI entry point
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -482,23 +647,12 @@ def main() -> None:
|
||||
json.dump(report, f, indent=2)
|
||||
logger.info("Combined report written to %s", args.report)
|
||||
|
||||
# Exit with error if either generator had high error rates.
|
||||
totals = report["totals"]
|
||||
rpc_err_rate = (
|
||||
totals["rpc_errors"] / totals["rpc_sent"] * 100 if totals["rpc_sent"] > 0 else 0
|
||||
)
|
||||
tx_err_rate = (
|
||||
totals["tx_errors"] / totals["tx_submitted"] * 100
|
||||
if totals["tx_submitted"] > 0
|
||||
else 0
|
||||
)
|
||||
# TX threshold is higher because short-lived CI test environments lack
|
||||
# pre-funded accounts, causing expected failures for complex transactions
|
||||
# (AMMCreate, EscrowFinish, etc.) that require specific ledger state.
|
||||
if rpc_err_rate > 50 or tx_err_rate > 95:
|
||||
logger.error(
|
||||
"High error rates: RPC=%.1f%%, TX=%.1f%%", rpc_err_rate, tx_err_rate
|
||||
)
|
||||
# Fail on any phase error as well as on high error rates.
|
||||
failures = evaluate_exit_gate(report)
|
||||
if failures:
|
||||
logger.error("Workload failed %d gate condition(s):", len(failures))
|
||||
for reason in failures:
|
||||
logger.error(" %s", reason)
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
|
||||
@@ -1,6 +1,10 @@
|
||||
# xrpld validator node configuration template for workload harness.
|
||||
#
|
||||
# Placeholders (replaced by docker-compose entrypoint):
|
||||
# Not consumed by anything today: run-full-validation.sh writes each node's
|
||||
# cfg inline. Kept as the reference layout for a validator run as a container,
|
||||
# whose entrypoint would substitute the placeholders below.
|
||||
#
|
||||
# Placeholders:
|
||||
# {{NODE_INDEX}} — Node number (1-based)
|
||||
# {{RPC_PORT}} — HTTP RPC port
|
||||
# {{WS_PORT}} — WebSocket port
|
||||
@@ -18,16 +22,19 @@ port_rpc
|
||||
port_ws
|
||||
port_peer
|
||||
|
||||
# RPC and WebSocket stay on loopback, matching the cfg that
|
||||
# run-full-validation.sh generates and the rest of the repo's node configs.
|
||||
# Only the peer port below needs to listen on all interfaces.
|
||||
[port_rpc]
|
||||
port = {{RPC_PORT}}
|
||||
ip = 0.0.0.0
|
||||
admin = 0.0.0.0
|
||||
ip = 127.0.0.1
|
||||
admin = 127.0.0.1
|
||||
protocol = http
|
||||
|
||||
[port_ws]
|
||||
port = {{WS_PORT}}
|
||||
ip = 0.0.0.0
|
||||
admin = 0.0.0.0
|
||||
ip = 127.0.0.1
|
||||
admin = 127.0.0.1
|
||||
protocol = ws
|
||||
|
||||
[port_peer]
|
||||
|
||||
Reference in New Issue
Block a user