diff --git a/.gitignore b/.gitignore index 5dc3e0ef86..13b59a7e2c 100644 --- a/.gitignore +++ b/.gitignore @@ -89,5 +89,3 @@ target/ # clangd cache /.cache -docker/telemetry/workload/__pycache__/ -.claude/ diff --git a/OpenTelemetryPlan/06-implementation-phases.md b/OpenTelemetryPlan/06-implementation-phases.md index 1fec8164b1..1007dd9794 100644 --- a/OpenTelemetryPlan/06-implementation-phases.md +++ b/OpenTelemetryPlan/06-implementation-phases.md @@ -1023,11 +1023,11 @@ See [Phase10_taskList.md](./Phase10_taskList.md) for detailed per-task breakdown ### Validation Check Inventory `validate_telemetry.py` derives its check count at run time from -`expected_spans.json` (40 catalogued span types, 27 of them required, 67 unique -required attributes) and `expected_metrics.json` (36 metric entries, 14 dashboard -uids). No fixed total is asserted here, because any manifest edit changes it; the -historical "71 checks" figure predates the Phase 7-9 metric families. The -categories are: +`expected_spans.json` (span types and their required attributes) and +`expected_metrics.json` (metric names and dashboard uids). Per the inventory note +above, no counts are quoted here — those two manifests are the only authority and +any edit to them changes the total. The historical "71 checks" figure predates the +later metric families. The categories are: - **Service registration** — `xrpld` exists in Tempo - **Span existence** — every required entry in `expected_spans.json`. Note that diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md index 0e843ebd36..a66f71487d 100644 --- a/OpenTelemetryPlan/09-data-collection-reference.md +++ b/OpenTelemetryPlan/09-data-collection-reference.md @@ -828,7 +828,9 @@ for how the tier attributes are set and reach metrics. 1. Open Grafana at **http://localhost:3000** 2. Navigate to **Dashboards → xrpld** folder 3. All 15 dashboards are auto-provisioned from `docker/telemetry/grafana/dashboards/` - (the Phase-10 harness asserts 14 of them — `log-derived-insights` is unasserted) + (the workload harness checks that all 15 provision and load; 14 of them also have + metric-data assertions — `log-derived-insights` is Loki-backed, so only its + provisioning is checked) --- @@ -1175,17 +1177,18 @@ docker/telemetry/workload/benchmark.sh --xrpld .build/xrpld --duration 300 | Call-site `XRPL_METRIC_*` | 7 instruments | Prometheus query | `expected_metrics.json` | | Per-job-type gauges | 105 (35 types × 3) | Prometheus `__name__` query | `expected_metrics.json` | | SpanMetrics RED | 4 per span | Prometheus query | `expected_metrics.json` | -| Grafana dashboards | 14 of 15 on disk | Dashboard API "no data" check | `expected_metrics.json` | +| Grafana dashboards | all 15 on disk | Dashboard API load + panel count | `expected_metrics.json` | | Log-trace links | Present | Loki query + Tempo reverse check | — | -> **These are the harness's numbers, not the code's, and three of them differ.** +> **These are the harness's numbers, not the code's, and two of them differ.** > `docker/telemetry/workload/expected_spans.json` carries 40 span entries against > the **41** families the code emits ([§1.1](#11-complete-span-inventory-41-spans)) — > `rpc.ws_upgrade` has no entry — and 67 distinct required attributes (the > manifest's own `total_unique_attributes: 58` field is stale). -> `expected_metrics.json` asserts 36 metric entries and 14 dashboard uids against -> the **15** dashboard JSONs in `docker/telemetry/grafana/dashboards/`; -> `log-derived-insights` is the unasserted one. The 35 native instruments match +> `expected_metrics.json` lists all **15** dashboard uids in +> `docker/telemetry/grafana/dashboards/`, so dashboard coverage does not differ; +> `log-derived-insights` is listed for the provisioning check only, and its panel +> data is asserted nowhere. The 35 native instruments match > the tables in > [§Phase 9: OTel SDK-Exported Metrics](#phase-9-otel-sdk-exported-metrics-metricsregistry) > and the Phase 7+ section exactly, counting each labeled gauge family diff --git a/docker/telemetry/docker-compose.workload.yaml b/docker/telemetry/docker-compose.workload.yaml index 818c65f4c5..b59dada459 100644 --- a/docker/telemetry/docker-compose.workload.yaml +++ b/docker/telemetry/docker-compose.workload.yaml @@ -1,24 +1,32 @@ -# Docker Compose workload harness for Phase 10 telemetry validation. +# Docker Compose workload harness for telemetry validation. # -# Runs a 5-node validator cluster with full OTel telemetry stack: -# - 5 rippled validator nodes (consensus network) +# Runs the OTel telemetry backend only. There are no validator services here: # - OTel Collector (traces + native OTLP metrics) # - Tempo (trace backend + search API) # - Prometheus (metrics) # - Loki (log aggregation for log-trace correlation) # - Grafana (dashboards + trace/log exploration) # +# The validator cluster runs as host processes, not containers. +# run-full-validation.sh starts NUM_NODES (default 5) xrpld instances on +# 127.0.0.1, each with a cfg it generates inline, peered to each other via +# [ips_fixed]. They reach the collector through the published ports below and +# write their logs into the bind-mounted workdir for the filelog receiver. +# # Usage: -# # Start the harness (requires pre-built xrpld image or mount binary): +# # Start the telemetry backend on its own: # docker compose -f docker/telemetry/docker-compose.workload.yaml up -d # -# # Or use the orchestrator: +# # Or let the orchestrator start this stack and the node cluster together: # docker/telemetry/workload/run-full-validation.sh # -# Prerequisites: +# Prerequisites (for the orchestrator, not for this stack): # - xrpld binary built with -DXRPL_ENABLE_TELEMETRY=ON # - Validator keys generated via generate-validator-keys.sh -# - Node configs generated by run-full-validation.sh +# +# Image tags are pinned to the same versions as docker-compose.yml, which +# mounts these same collector, Tempo and Prometheus config files. Floating +# tags would let an upstream release change the harness result. # # Note: No Docker healthchecks are defined here. The orchestrator script # (run-full-validation.sh) polls each service endpoint directly from the @@ -30,7 +38,7 @@ services: # --------------------------------------------------------------------------- otel-collector: - image: otel/opentelemetry-collector-contrib:latest + image: otel/opentelemetry-collector-contrib:0.158.0 command: ["--config=/etc/otel-collector-config.yaml"] ports: - "4317:4317" # OTLP gRPC @@ -49,7 +57,7 @@ services: - workload-net tempo: - image: grafana/tempo:2.7.2 + image: grafana/tempo:2.9.4 command: ["-config.file=/etc/tempo.yaml"] ports: - "3200:3200" # Tempo HTTP API @@ -60,7 +68,7 @@ services: - workload-net prometheus: - image: prom/prometheus:latest + image: prom/prometheus:v3.13.2 ports: - "9090:9090" volumes: @@ -71,7 +79,7 @@ services: - workload-net loki: - image: grafana/loki:3.4.2 + image: grafana/loki:3.7.6 ports: - "3100:3100" # Loki HTTP API command: ["-config.file=/etc/loki/local-config.yaml"] @@ -79,7 +87,7 @@ services: - workload-net grafana: - image: grafana/grafana:latest + image: grafana/grafana:13.1.2 environment: - GF_AUTH_ANONYMOUS_ENABLED=true - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin diff --git a/docker/telemetry/workload/README.md b/docker/telemetry/workload/README.md index 44bbdf3aac..53f4b17415 100644 --- a/docker/telemetry/workload/README.md +++ b/docker/telemetry/workload/README.md @@ -56,6 +56,7 @@ run-full-validation.sh (shell orchestrator) | -> validation-report.json | |-- benchmark.sh (baseline vs telemetry comparison) + |-- collect_system_metrics.sh (per-leg CPU/RSS/latency/TPS sampling) -> benchmark-report-*.md ``` @@ -132,7 +133,7 @@ Orchestrates the complete validation pipeline. Starts the telemetry stack, start # Stress test with benchmarks ./run-full-validation.sh --xrpld /path/to/xrpld --profile stress --with-benchmark -# Skip Loki checks (if Phase 8 not deployed) +# Skip Loki checks (if log export is not deployed) ./run-full-validation.sh --xrpld /path/to/xrpld --skip-loki ``` @@ -213,7 +214,7 @@ python3 tx_submitter.py --endpoint ws://localhost:6006 \ Automated validation that all expected telemetry data exists. Every metric in `expected_metrics.json` is required — if it doesn't fire, the validation fails. Spans are required unless the entry carries `"optional": true`. - **Span validation**: All span types from `expected_spans.json` with required attributes and parent-child hierarchies. Entries marked `"optional": true` only fire under traffic the harness may not produce (HTTP/JSON-RPC client, gRPC client, missing-ledger fetch, mode transitions); their absence is recorded as a passing skip, not a failure. -- **Metric validation**: All metrics from `expected_metrics.json` — SpanMetrics, `beast::insight` gauges/counters/histograms, Phase 9 OTLP metrics. Every listed metric must have > 0 series. Uses the Prometheus `/api/v1/series` endpoint (not instant queries), polled until the metric appears or the poll window elapses, so a late-populating or quiet series is not a false negative. +- **Metric validation**: All metrics from `expected_metrics.json` — SpanMetrics, `beast::insight` gauges/counters/histograms, `MetricsRegistry` OTLP metrics. Every listed metric must have > 0 series. Uses the Prometheus `/api/v1/series` endpoint (not instant queries), polled until the metric appears or the poll window elapses, so a late-populating or quiet series is not a false negative. - **Log-trace correlation**: trace_id/span_id in Loki logs (requires Loki) - **Dashboard validation**: Every dashboard uid listed under `grafana_dashboards.uids` in `expected_metrics.json` loads with panels. That list currently covers **all 15** dashboards provisioned in `docker/telemetry/grafana/dashboards/`. Note the scope of this check: it asks the Grafana API whether the dashboard exists and returns a panel count — it does **not** run the panels' queries, so a dashboard can pass here while individual panels render empty. @@ -286,6 +287,65 @@ Thresholds (configurable via environment): | Throughput impact | < 5% | BENCH_TPS_IMPACT_PCT | | Consensus impact | < 1% | BENCH_CONSENSUS_IMPACT_PCT | +Each report row is `PASS`, `FAIL`, or `INCONCLUSIVE`. The throughput and +consensus rows are ratios of the baseline, so they have nothing to report when +the baseline run measured zero — that row becomes `INCONCLUSIVE` and **counts +as a failure**, because an undefined result must never read as a pass. + +Exit codes: + +| Code | Meaning | +| ---- | --------------------------------------------------------------------------------------------------------------------------- | +| 0 | Every metric was measured and is within its threshold | +| 1 | Every metric was measured and at least one exceeded its threshold | +| 2 | The overhead could not be measured — missing prerequisite, cluster never reached consensus, or incomplete metric collection | + +`run-full-validation.sh` keeps the last two apart: 1 folds into its own +"checks failed" exit, 2 into its "infrastructure error" exit. A run that +measured nothing is therefore never reported as a performance regression. + +### collect_system_metrics.sh + +Samples CPU, peak RSS, RPC p99 latency, TPS and the mean inter-ledger interval +from the running nodes, and writes them as JSON. `benchmark.sh` calls it once +per leg; it is rarely run by hand. + +```bash +./collect_system_metrics.sh 5020,5021,5022 300 /tmp/metrics.json +``` + +Processes are selected by matching `argv[0]`'s basename against `xrpld` or +`rippled`. A wrapper that merely names the binary in its arguments, and +unrelated tools whose command line happens to contain the string, are not +sampled — including them diluted the CPU average and attributed a foreign +process's RSS to the node. `ps -C xrpld` is not usable for this: xrpld renames +itself, so its `comm` is `xrpld-main`. + +Selection covers the whole host, so a second xrpld from another checkout is +sampled as well. Benchmark on a machine running one cluster only. + +The output carries a `metrics_complete` flag. It is `false` when any +measurement source came back empty — no matching process, no successful RPC +probe, or a ledger sequence that never advanced — and the affected metrics are +then `0` placeholders. Since `0` clears every threshold, a `false` flag must be +read as inconclusive, never as a pass. + +Exit codes: + +| Code | Meaning | +| ---- | --------------------------------------------------------------------------------------------------------- | +| 0 | Every metric was measured; `metrics_complete` is `true` | +| 1 | Cannot run: bad arguments, no GNU `date` with `%N`, or a failed process sample. No output file is written | +| 3 | The output file was written, but `metrics_complete` is `false` | + +`benchmark.sh` treats either non-zero code — and an explicit +`"metrics_complete": false` in an otherwise successful run — as fatal, and +exits 2 rather than comparing an incomplete run. + +A nanosecond clock is required. RPC latency is graded against a 2 ms +threshold, and GNU `date +%s%N` is the only source cheap enough that the clock +does not dominate what it measures, so the script refuses to start without it. + ## Reading Validation Reports The validation report (`validation-report.json`) is structured as follows. The diff --git a/docker/telemetry/workload/baselines/README.md b/docker/telemetry/workload/baselines/README.md index 2f379ba713..fffea2e34d 100644 --- a/docker/telemetry/workload/baselines/README.md +++ b/docker/telemetry/workload/baselines/README.md @@ -15,7 +15,10 @@ declared in [`../regression-metrics.json`](../regression-metrics.json) and write exits 0 without gating. This is how we bootstrap the baseline. - **Populated baseline**: the comparator diffs per-metric, enforces the thresholds (regression = current exceeds baseline on BOTH the percentage AND absolute bound), - and exits non-zero on any regression. + and exits non-zero on any regression. The single exception is a baseline that is + not positive: the percentage change is undefined there, so the absolute bound + decides alone. Without that fallback the AND gate would be unreachable and a + 0 ms → 500 ms jump would be reported as "within bounds". The regression gate runs against whatever workload profile `run-full-validation.sh` was invoked with. Capture and comparison are profile-agnostic — they only read @@ -102,3 +105,24 @@ captured. Two independent blockers: Closing this needs **both** an `rpc_methods` group in `regression-metrics.json` and a `defaults.rpc_method` block in `regression-thresholds.json`. Adding only the first produces metrics that look gated in the report but are not. + +## Known exclusion: `rpc.process` is not captured + +`rpc.process` is deliberately absent from the `spans.names` list in +`regression-metrics.json`, so no `span.rpc.process.*` key appears in this +baseline. The span is created only in `ServerHandler::processRequest()` +(`src/xrpld/rpc/detail/ServerHandler.cpp:705`), which is reached only from the +HTTP/JSON-RPC session path. The harness load generator is WebSocket-only and +that path never calls `processRequest`, so the span is never emitted under any +workload profile here — `expected_spans.json` marks it `"optional": true` for +the same reason. + +While it was listed, the three quantiles were captured as `null` on every run +and the comparator short-circuited them as `"new metric (not in baseline)"` — +so a 9999 ms value would still have reported `regressed: false`. Three keys +that can never gate are worse than no keys: they inflate `summary.total` and +read as covered. + +If per-request HTTP timings are wanted, the fix is to give the harness an +HTTP/JSON-RPC load path first, then re-add `rpc.process` and bootstrap a real +baseline for it. diff --git a/docker/telemetry/workload/baselines/baseline-timings.json b/docker/telemetry/workload/baselines/baseline-timings.json index cafd683e3c..4784953609 100644 --- a/docker/telemetry/workload/baselines/baseline-timings.json +++ b/docker/telemetry/workload/baselines/baseline-timings.json @@ -78,18 +78,6 @@ "unit": "ms", "value": 6.699999999999978 }, - "span.rpc.process.p50": { - "unit": "ms", - "value": null - }, - "span.rpc.process.p95": { - "unit": "ms", - "value": null - }, - "span.rpc.process.p99": { - "unit": "ms", - "value": null - }, "span.rpc.ws_message.p50": { "unit": "ms", "value": 0.5026522773001647 diff --git a/docker/telemetry/workload/benchmark.sh b/docker/telemetry/workload/benchmark.sh index 954322f714..6c87a6b121 100755 --- a/docker/telemetry/workload/benchmark.sh +++ b/docker/telemetry/workload/benchmark.sh @@ -17,6 +17,20 @@ # BENCH_RPC_LATENCY_IMPACT_MS=2 RPC p99 latency impact < 2ms # BENCH_TPS_IMPACT_PCT=5 Throughput impact < 5% # BENCH_CONSENSUS_IMPACT_PCT=1 Consensus round time impact < 1% +# +# Exit codes: +# 0 Every overhead metric was measured and is within its threshold. +# 1 Every overhead metric was measured and at least one exceeded its +# threshold. This is the only "telemetry is too expensive" signal. +# Also returned for a command-line usage error, which the pipeline +# cannot trigger (run-full-validation.sh passes a fixed flag list). +# 2 The overhead could not be measured at all — a missing prerequisite, a +# cluster that never reached consensus, or an incomplete metric +# collection. Nothing was compared, so nothing was breached. +# +# run-full-validation.sh depends on that split: it folds 1 into its own +# "checks failed" exit and 2 into its "infrastructure error" exit. Reporting a +# run that measured nothing as a threshold breach would be a false regression. set -euo pipefail @@ -27,11 +41,21 @@ log() { printf "\033[1;34m[BENCH]\033[0m %s\n" "$*"; } ok() { printf "\033[1;32m[BENCH]\033[0m %s\n" "$*"; } warn() { printf "\033[1;33m[BENCH]\033[0m %s\n" "$*"; } fail() { printf "\033[1;31m[BENCH]\033[0m %s\n" "$*"; } + +# Usage error. Exit 1 by shell convention; see the exit-code block above. die() { - printf "\033[1;31m[BENCH]\033[0m %s\n" "$*" >&2 + fail "$*" >&2 exit 1 } +# Fatal, and no overhead figure was produced. Exit 2 keeps exit 1 exclusively +# for a measured threshold breach, so the caller never grades a run that +# measured nothing as a performance regression. +cannot_measure() { + fail "$*" >&2 + exit 2 +} + # --------------------------------------------------------------------------- # Defaults and thresholds # --------------------------------------------------------------------------- @@ -91,11 +115,12 @@ while [ $# -gt 0 ]; do esac done -# Validate prerequisites. -[ -x "$XRPLD" ] || die "xrpld not found at $XRPLD" -command -v jq >/dev/null 2>&1 || die "jq not found" -command -v bc >/dev/null 2>&1 || die "bc not found" -command -v curl >/dev/null 2>&1 || die "curl not found" +# Validate prerequisites. A missing binary or tool means no measurement can be +# taken, which is "cannot measure", not "too slow". +[ -x "$XRPLD" ] || cannot_measure "xrpld not found at $XRPLD" +command -v jq >/dev/null 2>&1 || cannot_measure "jq not found" +command -v bc >/dev/null 2>&1 || cannot_measure "bc not found" +command -v curl >/dev/null 2>&1 || cannot_measure "curl not found" mkdir -p "$RESULTS_DIR" TIMESTAMP=$(date +%Y%m%d_%H%M%S) @@ -103,6 +128,11 @@ TIMESTAMP=$(date +%Y%m%d_%H%M%S) # --------------------------------------------------------------------------- # Node cluster management # --------------------------------------------------------------------------- + +# True while xrpld children spawned by start_cluster may still be alive. +# Read by stop_cluster so it can be called any number of times. +CLUSTER_RUNNING=false + start_cluster() { local telemetry_enabled="$1" local label="$2" @@ -115,6 +145,10 @@ start_cluster() { # Generate keys using first node. bash "$SCRIPT_DIR/generate-validator-keys.sh" "$XRPLD" "$NUM_NODES" "$WORKDIR" + # Set before the spawn loop so a failure part-way through it still gets + # cleaned up by the EXIT trap. + CLUSTER_RUNNING=true + # Build per-node configs. for i in $(seq 1 "$NUM_NODES"); do local node_dir="$WORKDIR/node$i" @@ -214,9 +248,12 @@ EOCFG echo $! >"$node_dir/xrpld.pid" done - # Wait for consensus. - log "Waiting for consensus..." - for attempt in $(seq 1 120); do + # Wait for consensus. Reaching the limit is fatal: numbers taken from a + # cluster that never got to "proposing" would silently corrupt the + # baseline-vs-telemetry comparison. + local max_wait=120 + log "Waiting for consensus (up to ${max_wait}s)..." + for attempt in $(seq 1 "$max_wait"); do local ready=0 for i in $(seq 1 "$NUM_NODES"); do local port @@ -233,8 +270,8 @@ EOCFG ok "All $NUM_NODES nodes proposing (attempt $attempt)" break fi - if [ "$attempt" -eq 120 ]; then - warn "Consensus timeout — $ready/$NUM_NODES nodes ready" + if [ "$attempt" -eq "$max_wait" ]; then + cannot_measure "Consensus timeout — only $ready/$NUM_NODES nodes proposing after ${max_wait}s" fi sleep 1 done @@ -244,6 +281,11 @@ EOCFG } stop_cluster() { + # Idempotent. The happy path calls this directly and the EXIT trap calls + # it again, so a second call must not re-kill or log a misleading message. + [ "$CLUSTER_RUNNING" = true ] || return 0 + CLUSTER_RUNNING=false + log "Stopping cluster..." for i in $(seq 1 "$NUM_NODES"); do local pidfile="$WORKDIR/node$i/xrpld.pid" @@ -251,10 +293,30 @@ stop_cluster() { kill "$(cat "$pidfile")" 2>/dev/null || true fi done - pkill -f "$WORKDIR" 2>/dev/null || true - sleep 3 + # Belt and braces for a node whose pidfile is missing or stale. Matched on + # the per-node config path — the shape start_cluster launches nodes with + # (`--conf $WORKDIR/nodeN/xrpld.cfg`) — rather than on the workdir alone. + # The loose form killed anything whose command line merely mentioned the + # workdir, including a developer's `tail -f $WORKDIR/node1/debug.log`, and + # the EXIT trap now makes this run on every exit path. + pkill -f "$WORKDIR/node[0-9]+/xrpld\.cfg" 2>/dev/null || true + + # Guarded on purpose. This runs as the EXIT trap, where any unguarded + # failure makes `set -e` exit with that command's status and discard the + # status the script meant to report — a threshold breach would surface as + # a plain 1 and "cannot measure" would lose its 2. With every command here + # guarded, the explicit `return 0` below is reachable and authoritative. + sleep 3 || true + + return 0 } +# Reap the cluster on every exit path. Installed here rather than straight +# after argument parsing so the handler name always resolves. Without it, any +# failure between start_cluster and stop_cluster leaks the xrpld children +# along with their RPC ports (5020+) and peer ports (51250+). +trap stop_cluster EXIT + # Build RPC ports CSV string. rpc_ports_csv() { local ports="" @@ -265,6 +327,30 @@ rpc_ports_csv() { echo "$ports" } +# Collects one leg of the benchmark. +# +# The collector exits non-zero when it cannot run (1) or when a measurement +# source came back empty (3). An all-zero or partial sample set clears every +# threshold, so an incomplete leg aborts with "cannot measure" instead of being +# compared and passed. +collect_metrics() { + local label="$1" + local out_file="$2" + + local status=0 + bash "$SCRIPT_DIR/collect_system_metrics.sh" \ + "$(rpc_ports_csv)" "$DURATION" "$out_file" || status=$? + [ "$status" -eq 0 ] || + cannot_measure "$label metric collection failed (exit $status) — refusing to compare an incomplete run" + + # Only an explicit false counts. The flag is absent from older artifacts, + # and jq's "//" operator would turn a real false into the default. + local complete + complete=$(jq -r '.metrics_complete' "$out_file" 2>/dev/null || echo "null") + [ "$complete" != "false" ] || + cannot_measure "$label metrics are flagged incomplete — refusing to compare an incomplete run" +} + # --------------------------------------------------------------------------- # Run benchmark # --------------------------------------------------------------------------- @@ -276,13 +362,13 @@ log "=" # --- Baseline run --- BASELINE_FILE="$RESULTS_DIR/baseline-${TIMESTAMP}.json" start_cluster "0" "baseline" -bash "$SCRIPT_DIR/collect_system_metrics.sh" "$(rpc_ports_csv)" "$DURATION" "$BASELINE_FILE" +collect_metrics "baseline" "$BASELINE_FILE" stop_cluster # --- Telemetry run --- TELEMETRY_FILE="$RESULTS_DIR/telemetry-${TIMESTAMP}.json" start_cluster "1" "telemetry" -bash "$SCRIPT_DIR/collect_system_metrics.sh" "$(rpc_ports_csv)" "$DURATION" "$TELEMETRY_FILE" +collect_metrics "telemetry" "$TELEMETRY_FILE" stop_cluster # --------------------------------------------------------------------------- @@ -290,6 +376,10 @@ stop_cluster # --------------------------------------------------------------------------- log "Comparing results..." +# Written into an impact variable when its baseline could not be used. +# check_threshold turns this into an INCONCLUSIVE verdict. +INCONCLUSIVE="n/a" + read_metric() { local file="$1" local key="$2" @@ -308,20 +398,30 @@ BASE_RPC=$(read_metric "$BASELINE_FILE" "rpc_p99_ms") TELE_RPC=$(read_metric "$TELEMETRY_FILE" "rpc_p99_ms") RPC_DELTA=$(echo "scale=2; $TELE_RPC - $BASE_RPC" | bc 2>/dev/null || echo "0") +# Both impacts below are ratios of the baseline, so a non-positive baseline +# leaves them undefined. The collector writes tps=0 whenever no ledger +# advanced and read_metric defaults a missing key to 0, so this is a routine +# outcome rather than an edge case. Reporting it as "0% impact" would clear +# the threshold and hide a failed baseline run. +# +# Both expressions scale by 100 before dividing. bc truncates at "scale" after +# every operation, so dividing first would floor the ratio to 2 decimals and +# then multiply the lost precision by 100 — a real 1.25% consensus impact came +# out as exactly 1.00 and passed the 1% threshold. BASE_TPS=$(read_metric "$BASELINE_FILE" "tps") TELE_TPS=$(read_metric "$TELEMETRY_FILE" "tps") if [[ "$(echo "$BASE_TPS > 0" | bc 2>/dev/null)" = "1" ]]; then - TPS_IMPACT=$(echo "scale=2; ($BASE_TPS - $TELE_TPS) / $BASE_TPS * 100" | bc 2>/dev/null || echo "0") + TPS_IMPACT=$(echo "scale=2; ($BASE_TPS - $TELE_TPS) * 100 / $BASE_TPS" | bc 2>/dev/null || echo "0") else - TPS_IMPACT="0" + TPS_IMPACT="$INCONCLUSIVE" fi -BASE_CONS=$(read_metric "$BASELINE_FILE" "consensus_round_p95_ms") -TELE_CONS=$(read_metric "$TELEMETRY_FILE" "consensus_round_p95_ms") +BASE_CONS=$(read_metric "$BASELINE_FILE" "consensus_round_mean_ms") +TELE_CONS=$(read_metric "$TELEMETRY_FILE" "consensus_round_mean_ms") if [[ "$(echo "$BASE_CONS > 0" | bc 2>/dev/null)" = "1" ]]; then - CONS_IMPACT=$(echo "scale=2; ($TELE_CONS - $BASE_CONS) / $BASE_CONS * 100" | bc 2>/dev/null || echo "0") + CONS_IMPACT=$(echo "scale=2; ($TELE_CONS - $BASE_CONS) * 100 / $BASE_CONS" | bc 2>/dev/null || echo "0") else - CONS_IMPACT="0" + CONS_IMPACT="$INCONCLUSIVE" fi # --------------------------------------------------------------------------- @@ -329,30 +429,63 @@ fi # --------------------------------------------------------------------------- PASS_COUNT=0 FAIL_COUNT=0 +INCONCLUSIVE_COUNT=0 +# Records the verdict for one row of the report. +# +# Arguments: metric name, measured value, threshold, unit, and the name of the +# variable to write the bare verdict into. +# +# The verdict travels through that named variable and every diagnostic goes to +# stderr. Calling this through a command substitution would run it in a +# subshell, which drops the counter updates and captures the colored log line +# into the caller's variable. check_threshold() { local name="$1" local actual="$2" local threshold="$3" local unit="$4" + local result_var="$5" + + # Unusable measurement. Counted as a failure so the exit gate fires: an + # undefined result must never read as a pass. + if [ "$actual" = "$INCONCLUSIVE" ]; then + fail "$name: INCONCLUSIVE — baseline was zero or missing" >&2 + FAIL_COUNT=$((FAIL_COUNT + 1)) + INCONCLUSIVE_COUNT=$((INCONCLUSIVE_COUNT + 1)) + printf -v "$result_var" 'INCONCLUSIVE' + return + fi # Compare: actual <= threshold if [[ "$(echo "$actual <= $threshold" | bc 2>/dev/null)" = "1" ]]; then - ok "$name: ${actual}${unit} <= ${threshold}${unit} PASS" + ok "$name: ${actual}${unit} <= ${threshold}${unit} PASS" >&2 PASS_COUNT=$((PASS_COUNT + 1)) - echo "PASS" + printf -v "$result_var" 'PASS' else - fail "$name: ${actual}${unit} > ${threshold}${unit} FAIL" + fail "$name: ${actual}${unit} > ${threshold}${unit} FAIL" >&2 FAIL_COUNT=$((FAIL_COUNT + 1)) - echo "FAIL" + printf -v "$result_var" 'FAIL' fi } -CPU_RESULT=$(check_threshold "CPU overhead" "$CPU_DELTA" "$CPU_THRESHOLD" "%") -MEM_RESULT=$(check_threshold "Memory overhead" "$MEM_DELTA" "$MEM_THRESHOLD" "MB") -RPC_RESULT=$(check_threshold "RPC p99 impact" "$RPC_DELTA" "$RPC_THRESHOLD" "ms") -TPS_RESULT=$(check_threshold "TPS impact" "$TPS_IMPACT" "$TPS_THRESHOLD" "%") -CONS_RESULT=$(check_threshold "Consensus impact" "$CONS_IMPACT" "$CONSENSUS_THRESHOLD" "%") +# Formats a delta for the report table: appends the unit to a real number and +# leaves the INCONCLUSIVE placeholder bare. +fmt_delta() { + local value="$1" + local unit="$2" + if [ "$value" = "$INCONCLUSIVE" ]; then + printf '%s' "$value" + else + printf '%s%s' "$value" "$unit" + fi +} + +check_threshold "CPU overhead" "$CPU_DELTA" "$CPU_THRESHOLD" "%" CPU_RESULT +check_threshold "Memory overhead" "$MEM_DELTA" "$MEM_THRESHOLD" "MB" MEM_RESULT +check_threshold "RPC p99 impact" "$RPC_DELTA" "$RPC_THRESHOLD" "ms" RPC_RESULT +check_threshold "TPS impact" "$TPS_IMPACT" "$TPS_THRESHOLD" "%" TPS_RESULT +check_threshold "Consensus impact" "$CONS_IMPACT" "$CONSENSUS_THRESHOLD" "%" CONS_RESULT # --------------------------------------------------------------------------- # Output Markdown table @@ -373,13 +506,20 @@ cat >"$REPORT_FILE" <&2; } die() { printf "\033[1;31m[METRICS]\033[0m %s\n" "$*" >&2 exit 1 @@ -56,10 +67,53 @@ OUTPUT_FILE="$3" IFS=',' read -ra RPC_PORTS <<<"$RPC_PORTS_CSV" SAMPLE_INTERVAL=5 -SAMPLES=$((DURATION / SAMPLE_INTERVAL)) + +# Reject anything the sample arithmetic cannot use, instead of silently +# treating it as 0. +case "$DURATION" in + '' | *[!0-9]*) die "duration_seconds must be a positive integer, got '$DURATION'" ;; +esac + +# Normalise to base 10 once, right after the digits check. Bash arithmetic +# reads a leading zero as octal, so "08" aborted with "value too great for +# base" and "0100" was silently taken as 64. Doing it here also keeps the +# value a valid JSON number in the output below, where "08" is not. +DURATION=$((10#$DURATION)) + +# Round up, so a duration shorter than one interval still takes one sample +# rather than truncating to zero and emitting an all-zero JSON. +SAMPLES=$(((DURATION + SAMPLE_INTERVAL - 1) / SAMPLE_INTERVAL)) +if [ "$SAMPLES" -lt 1 ]; then + die "duration_seconds=$DURATION yields $SAMPLES samples; need at least 1" +fi log "Collecting metrics for ${DURATION}s (${SAMPLES} samples, ${#RPC_PORTS[@]} nodes)..." +# --------------------------------------------------------------------------- +# Nanosecond clock +# +# GNU date supports "+%s%N". BSD/macOS date has no %N and echoes it back +# literally, which used to abort the sampling loop under `set -e` and still +# exit 0 with an all-zero JSON — a silent false pass. +# +# The clock has to be cheap as well as precise, because the latency it +# measures is compared against a 2 ms threshold. Measured on a dev box: `date +# +%s%N` costs ~1.2 ms per call, forking python3 for the same value ~13 ms. +# Two calls bracket every request, so a python3 fallback would add ~26 ms of +# its own overhead to a 2 ms budget and make the number meaningless. There is +# no cheap alternative worth having, so probe once and refuse to run without +# GNU date rather than report a figure that is quietly an order of magnitude +# wrong. +# --------------------------------------------------------------------------- +if [[ ! "$(date +%s%N 2>/dev/null)" =~ ^[0-9]+$ ]]; then + die "GNU coreutils date with %N is required for RPC latency timing; this date does not support it" +fi + +# Echo the current time in nanoseconds since the epoch. +now_ns() { + date +%s%N +} + # --------------------------------------------------------------------------- # Temporary files for aggregation # --------------------------------------------------------------------------- @@ -95,40 +149,48 @@ log "Initial validated ledger seq: $INITIAL_SEQ" # Sampling loop # --------------------------------------------------------------------------- for sample in $(seq 1 "$SAMPLES"); do - # Collect CPU usage for xrpld processes. - # Uses ps to find all xrpld processes and average their CPU%. - cpu_sum=0 - cpu_count=0 - while IFS= read -r line; do - cpu_val=$(echo "$line" | awk '{print $1}') - if [ -n "$cpu_val" ] && [ "$cpu_val" != "0.0" ]; then - cpu_sum=$(echo "$cpu_sum + $cpu_val" | bc 2>/dev/null || echo "$cpu_sum") - cpu_count=$((cpu_count + 1)) - fi - done < <(ps aux 2>/dev/null | grep '[x]rpld' | awk '{print $3}') + # Sample CPU% and RSS for the xrpld processes. One ps pass feeds both + # files, so the two numbers always come from the same instant. + # + # Selection is on argv[0]'s basename, NOT on "the command line mentions + # xrpld". The loose form matched every bystander whose command line + # happened to contain the string: this harness's own launcher (invoked as + # `--xrpld .build/xrpld`, whose argv[0] is bash), an editor's clangd, and + # any shell sitting in a directory with xrpld in its path. Measured + # against a live 5-node cluster, that pulled in 8-15 processes instead of + # 5, halved the CPU average with idle bystanders, and reported clangd's + # 1.1 GB RSS as xrpld's peak against a 5 MB threshold. `ps -C xrpld` is + # not an alternative: xrpld renames itself, so its comm is "xrpld-main" + # and -C matches nothing. "rippled" is accepted alongside "xrpld" so a + # rename of the binary cannot silently zero the collector. + # + # Scope is the whole host, as it always was: a second xrpld from another + # checkout is sampled too. Only run a benchmark on a box with one cluster. + # + # A %cpu of exactly 0.0 is a real reading and is counted — dropping idle + # samples would inflate the average — while non-numeric output is + # rejected by the pattern. An RSS of 0 is not a live process, so it + # contributes no memory sample; counting it would leave the file non-empty + # and mark a dead cluster's 0 MB peak as a complete measurement. + ps -eo %cpu=,rss=,args= | + awk -v cpu_file="$CPU_FILE" -v mem_file="$MEM_FILE" ' + $3 !~ /(^|\/)(xrpld|rippled)$/ { next } + $1 ~ /^[0-9]+(\.[0-9]+)?$/ { cpu_sum += $1; cpu_n++ } + $2 ~ /^[0-9]+$/ && $2 + 0 > 0 { printf("%.2f\n", $2 / 1024) >> mem_file } + END { if (cpu_n > 0) printf("%.2f\n", cpu_sum / cpu_n) >> cpu_file } + ' || die "process sampling failed on sample $sample/$SAMPLES" - if [ "$cpu_count" -gt 0 ]; then - cpu_avg=$(echo "scale=2; $cpu_sum / $cpu_count" | bc 2>/dev/null || echo "0") - echo "$cpu_avg" >>"$CPU_FILE" - fi - - # Collect memory RSS for xrpld processes. - while IFS= read -r line; do - rss_kb=$(echo "$line" | awk '{print $1}') - if [ -n "$rss_kb" ] && [ "$rss_kb" != "0" ]; then - rss_mb=$(echo "scale=2; $rss_kb / 1024" | bc 2>/dev/null || echo "0") - echo "$rss_mb" >>"$MEM_FILE" - fi - done < <(ps aux 2>/dev/null | grep '[x]rpld' | awk '{print $6}') - - # Collect RPC latency from each node. + # Collect RPC latency from each node. Only a successful call is a latency + # measurement: a refused connection returns in well under a millisecond, + # and recording that as ~0 ms would pull the reported p99 down. for port in "${RPC_PORTS[@]}"; do - start_ms=$(date +%s%N) - curl -sf "http://localhost:$port" \ - -d '{"method":"server_info"}' >/dev/null 2>&1 || true - end_ms=$(date +%s%N) - latency_ms=$(((end_ms - start_ms) / 1000000)) - echo "$latency_ms" >>"$RPC_FILE" + start_ns=$(now_ns) + if curl -sf "http://localhost:$port" \ + -d '{"method":"server_info"}' >/dev/null 2>&1; then + end_ns=$(now_ns) + latency_ms=$(((end_ns - start_ns) / 1000000)) + echo "$latency_ms" >>"$RPC_FILE" + fi done # Record current validated ledger seq. @@ -153,28 +215,55 @@ done # --------------------------------------------------------------------------- log "Computing aggregated metrics..." +# Cleared by any empty measurement source. A 0 metric is otherwise +# indistinguishable from a real reading, so the flag is exported in the JSON +# and drives the exit-3 contract documented in the header. +METRICS_COMPLETE=true + # CPU average. if [ -s "$CPU_FILE" ]; then CPU_AVG=$(awk '{ sum += $1; n++ } END { if (n>0) printf "%.2f", sum/n; else print "0" }' "$CPU_FILE") else + # Now that the selector cannot match this harness's own processes, an + # empty file means no xrpld process was running for any sample. + warn "No CPU samples collected (no xrpld process matched); cpu_pct_avg is a 0 placeholder" CPU_AVG="0" + METRICS_COMPLETE=false fi # Memory peak RSS (MB). if [ -s "$MEM_FILE" ]; then MEM_PEAK=$(sort -n "$MEM_FILE" | tail -1) else + warn "No memory samples collected (no xrpld process matched); memory_rss_mb_peak is a 0 placeholder" MEM_PEAK="0" + METRICS_COMPLETE=false fi # RPC latency p99 (ms). if [ -s "$RPC_FILE" ]; then RPC_COUNT=$(wc -l <"$RPC_FILE") - P99_INDEX=$(echo "scale=0; $RPC_COUNT * 99 / 100" | bc) + # Nearest-rank p99: ceil(count * 99 / 100), clamped into [1, count]. + # Integer arithmetic avoids both the floor bias of the old bc expression + # and the bc dependency, and the lower clamp keeps sed off line address 0 + # (a file with no trailing newline makes wc -l report 0). + P99_INDEX=$(((RPC_COUNT * 99 + 99) / 100)) + if [ "$P99_INDEX" -lt 1 ]; then + P99_INDEX=1 + fi + if [ "$P99_INDEX" -gt "$RPC_COUNT" ]; then + P99_INDEX="$RPC_COUNT" + fi RPC_P99=$(sort -n "$RPC_FILE" | sed -n "${P99_INDEX}p") - [ -z "$RPC_P99" ] && RPC_P99="0" + if [ -z "$RPC_P99" ]; then + warn "RPC latency file has no line $P99_INDEX; rpc_p99_ms is a 0 placeholder" + RPC_P99="0" + METRICS_COMPLETE=false + fi else + warn "No successful RPC probes; rpc_p99_ms is a 0 placeholder" RPC_P99="0" + METRICS_COMPLETE=false fi # TPS calculation from ledger sequence advancement. @@ -198,20 +287,26 @@ else TPS="0" fi -# Consensus round time p95 (from ledger close interval). -# Approximate by looking at ledger sequence progression intervals. +# Mean inter-ledger interval in ms: DURATION / (distinct ledgers - 1) * 1000. +# +# This is a MEAN, not a percentile — the JSON key says so. It is also aliased +# by the sample loop: LEDGER_FILE gets one sequence per sample, so at a +# SAMPLE_INTERVAL of 5 s the series cannot resolve a close interval faster +# than that (a ~4 s close is invisible). Read it as a coarse trend only. if [ -s "$LEDGER_FILE" ]; then - # Calculate intervals between consecutive ledger sequences. - LEDGER_COUNT=$(wc -l <"$LEDGER_FILE") - # Rough estimate: DURATION / number_of_distinct_ledgers * 1000 ms UNIQUE_LEDGERS=$(sort -u "$LEDGER_FILE" | wc -l) + # The > 1 test also keeps the divisor below at 1 or more. if [ "$UNIQUE_LEDGERS" -gt 1 ]; then - CONSENSUS_P95=$(echo "scale=0; $DURATION * 1000 / ($UNIQUE_LEDGERS - 1)" | bc 2>/dev/null || echo "0") + CONSENSUS_MEAN=$(echo "scale=0; $DURATION * 1000 / ($UNIQUE_LEDGERS - 1)" | bc 2>/dev/null || echo "0") else - CONSENSUS_P95="0" + warn "Ledger seq never advanced ($UNIQUE_LEDGERS distinct); consensus_round_mean_ms is a 0 placeholder" + CONSENSUS_MEAN="0" + METRICS_COMPLETE=false fi else - CONSENSUS_P95="0" + warn "No ledger samples collected; consensus_round_mean_ms is a 0 placeholder" + CONSENSUS_MEAN="0" + METRICS_COMPLETE=false fi # --------------------------------------------------------------------------- @@ -223,7 +318,8 @@ cat >"$OUTPUT_FILE" < str: + """Classify one comparison outcome for the report and the table.""" + if regressed: + note = "REGRESSION" + elif delta < 0: + note = "improved" + else: + note = "within bounds" + if pct_change is None: + note += " (absolute bound only; baseline not positive)" + return note + + +# The regression rule, applied by compute_delta below. +# +# A regression normally requires BOTH bounds to be breached simultaneously. +# That tolerates small-value noise: a 100% increase on a 0.5 ms metric (to +# 1.0 ms) is not a regression under a 5 ms absolute bound. +# +# A non-positive baseline has no defined percentage change, so there the +# absolute bound decides alone. Requiring both bounds in that case would make +# the gate unreachable and let a 0 -> 500 ms jump pass as "within bounds". def compute_delta( key: str, baseline_entry: dict | None, @@ -183,9 +209,8 @@ def compute_delta( ) -> MetricDelta: """Compute a MetricDelta for one metric key. - A regression requires BOTH bounds to be breached simultaneously. This - tolerates small-value noise: a 100% increase on a 0.5 ms metric - (to 1.0 ms) is not a regression under a 5 ms absolute bound. + Follows the regression rule set out in the comment above, including the + non-positive-baseline exception. """ baseline = baseline_entry.get("value") if baseline_entry else None current = current_entry.get("value") if current_entry else None @@ -224,16 +249,13 @@ def compute_delta( note="no threshold configured", ) - pct_breach = pct_change is not None and pct_change > pct_threshold abs_breach = delta > abs_threshold - regressed = pct_breach and abs_breach - - if regressed: - note = "REGRESSION" - elif delta < 0: - note = "improved" + if pct_change is None: + # Baseline is not positive, so there is no percentage to compare. + # The absolute bound is the only usable signal here. + regressed = abs_breach else: - note = "within bounds" + regressed = pct_change > pct_threshold and abs_breach return MetricDelta( key=key, @@ -245,7 +267,7 @@ def compute_delta( threshold_pct=pct_threshold, threshold_abs=abs_threshold, regressed=regressed, - note=note, + note=_delta_note(regressed, delta, pct_change), ) @@ -265,7 +287,10 @@ def print_summary(deltas: list[MetricDelta]) -> None: print("=" * 72) if regressions: - print("\nRegressions (breached BOTH pct AND absolute bounds):") + print( + "\nRegressions (breached BOTH pct AND absolute bounds, or the " + "absolute bound alone where the baseline is not positive):" + ) _print_table(regressions) if improvements: diff --git a/docker/telemetry/workload/expected_metrics.json b/docker/telemetry/workload/expected_metrics.json index 3d0d4b6ddf..2aa1d0fcb2 100644 --- a/docker/telemetry/workload/expected_metrics.json +++ b/docker/telemetry/workload/expected_metrics.json @@ -55,20 +55,20 @@ "total_messages_out" ] }, - "phase9_nodestore": { - "description": "Phase 9 NodeStore I/O observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label distinguishing sub-metrics.", + "nodestore_io": { + "description": "NodeStore I/O observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label distinguishing sub-metrics.", "metrics": ["nodestore_state"] }, - "phase9_cache": { - "description": "Phase 9 cache hit rate observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label.", + "cache_hit_rates": { + "description": "Cache hit rate observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label.", "metrics": ["cache_metrics"] }, - "phase9_txq": { - "description": "Phase 9 transaction queue observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label.", + "transaction_queue": { + "description": "Transaction queue observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label.", "metrics": ["txq_metrics"] }, - "phase9_rpc_method": { - "description": "Phase 9 per-RPC-method counters and duration histogram (MetricsRegistry.cpp:351-357). rpc_method_errored_total is deliberately absent — see not_asserted below. rpc_method_us is a Histogram, so the Prometheus exporter emits only the _bucket/_count/_sum triple and there is no bare rpc_method_us series to match — same convention as span_duration_milliseconds in the spanmetrics group above.", + "rpc_method_detail": { + "description": "Per-RPC-method counters and duration histogram (MetricsRegistry.cpp:351-357). rpc_method_errored_total is deliberately absent — see not_asserted below. rpc_method_us is a Histogram, so the Prometheus exporter emits only the _bucket/_count/_sum triple and there is no bare rpc_method_us series to match — same convention as span_duration_milliseconds in the spanmetrics group above.", "metrics": [ "rpc_method_started_total", "rpc_method_finished_total", @@ -77,8 +77,8 @@ "rpc_method_us_sum" ] }, - "phase9_job_queue": { - "description": "Phase 9 job-queue counters and latency histograms (MetricsRegistry.cpp:360-366). Every xrpld job passes through these, so they populate under any workload. Both histograms are recorded in the same function bodies as job_started_total / job_finished_total, under the same guard and with the same labels, so their presence is equally guaranteed. They are named with the _bucket/_count/_sum suffixes the Prometheus exporter emits: regression-metrics.json and the job-queue dashboard both query job_queued_us_bucket / job_running_us_bucket, and no bare series exists.", + "job_queue": { + "description": "Job-queue counters and latency histograms (MetricsRegistry.cpp:360-366). Every xrpld job passes through these, so they populate under any workload. Both histograms are recorded in the same function bodies as job_started_total / job_finished_total, under the same guard and with the same labels, so their presence is equally guaranteed. They are named with the _bucket/_count/_sum suffixes the Prometheus exporter emits: regression-metrics.json and the job-queue dashboard both query job_queued_us_bucket / job_running_us_bucket, and no bare series exists.", "metrics": [ "job_queued_total", "job_started_total", @@ -95,12 +95,12 @@ "description": "In-flight RPC gauge via the XRPL_METRIC_UPDOWN_ADD call-site macro (PerfLogImp.cpp, +1 rpcStart / -1 rpcEnd). UpDownCounter: no _total suffix.", "metrics": ["rpc_in_flight_requests"] }, - "phase9_objects": { - "description": "Phase 9 counted object instances observable gauge (MetricsRegistry via OTLP).", + "object_counts": { + "description": "Counted object instances observable gauge (MetricsRegistry via OTLP).", "metrics": ["object_count"] }, - "phase9_load": { - "description": "Phase 9 fee escalation and load factor observable gauge (MetricsRegistry via OTLP).", + "load_factors": { + "description": "Fee escalation and load factor observable gauge (MetricsRegistry via OTLP).", "metrics": ["load_factor_metrics"] }, "parity_validation_agreement": { @@ -125,7 +125,7 @@ ] }, "parity_ledger_economy": { - "description": "External dashboard parity: ledger economy metrics (MetricsRegistry).", + "description": "External dashboard parity: ledger economy metrics (MetricsRegistry.cpp:1401). transaction_rate is observed on every export, in both branches of the ledger-age test (MetricsRegistry.cpp:1444-1451). base_fee_xrp is observed only inside the 'if (ledger)' guard on getValidatedLedger() (MetricsRegistry.cpp:1418-1423), and that returns validLedger_ (LedgerMaster.cpp:1569-1572), which stays null until a ledger validates — the same precondition complete_ledgers has. Both are asserted because run-full-validation.sh waits for a validated ledger before running the workload. base_fee_xrp absent while transaction_rate is present is the signature of a cluster that never validated, not of a missing metric.", "metrics": [ "ledger_economy{metric=\"base_fee_xrp\"}", "ledger_economy{metric=\"transaction_rate\"}" @@ -149,7 +149,7 @@ "metrics": ["storage_detail{metric=\"stored_object_bytes\"}"] }, "node_health_gauges": { - "description": "Node-health observable gauges (MetricsRegistry.cpp:997, :1081, :1102, :1161). All four are registered with callbacks that fire on every periodic export and Observe unconditionally (build_info observes a literal 1; server_info and db_metrics read live services; complete_ledgers observes the parsed ledger range, which is non-empty once the cluster has closed a ledger), so their series exist regardless of workload shape.", + "description": "Node-health observable gauges (MetricsRegistry.cpp:997, :1081, :1102, :1161). server_info, build_info and db_metrics Observe unconditionally on every periodic export (build_info observes a literal 1; server_info and db_metrics read live services), so their series exist regardless of workload shape. complete_ledgers is the exception and is asserted on a narrower guarantee: its callback returns without observing when the range is empty (MetricsRegistry.cpp:1113-1114) and skips any segment that carries no '-' (:1122-1127), and a one-sequence range renders with no '-' (RangeSet.h:70-71), so it needs a complete range spanning at least two sequences. completeLedgers_ is filled by setFullLedger (LedgerMaster.cpp:862-863), which on a peered node is reached only from the publish path in doAdvance (LedgerMaster.cpp:1972) — closing a ledger is not enough, it has to validate. run-full-validation.sh waits for that before the workload starts, so on a healthy cluster the series always exists — a 5-node run yields 10 series, one start and one end per node. If this check ever fails, read the Step 3 output first: a run that logged 'No validated ledger' cannot produce this series and the cluster, not the exporter, is what broke.", "metrics": ["server_info", "build_info", "complete_ledgers", "db_metrics"] }, "overlay_reduce_relay": { @@ -169,7 +169,7 @@ "metrics_excluded": { "rpc_method_errored_total": "MetricsRegistry.cpp:354, push counter — needs an RPC that returns an error. rpc_load_generator.py issues only well-formed server_info / fee / ledger / ripple_path_find calls, so no series may ever be created.", "ledger_history_mismatch_total": "MetricsRegistry.cpp:377, incremented only from LedgerHistory.cpp:332 on a built-vs-validated ledger mismatch. On a healthy run it never fires — asserting it would mean asserting a defect.", - "txq_expired_total": "MetricsRegistry.cpp:379, incremented only at TxQ.cpp:1428 when a queued tx expires past its LastLedgerSequence. Requires sustained fee escalation plus expiry; the CI job drives rpc_load_generator/tx_submitter directly with --rpc-rate/--tx-tps and never runs the workload-profiles.json txq-burst phase, so this is not reachable in CI.", + "txq_expired_total": "MetricsRegistry.cpp:379, incremented only at TxQ.cpp:1428 when a queued tx expires past its LastLedgerSequence. CI does run a txq-burst phase (workload-profiles.json:41, 30 s of single-type Payment at 60 TPS), but that does not guarantee sustained fee escalation followed by expiry: a run in which every other check passed still exposed only txq_metrics and no txq_expired_total.", "txq_dropped_total": "MetricsRegistry.cpp:381, incremented only at TxQ.cpp:1302 / :1347 on queue-full admission refusal. Same reason as txq_expired_total.", "getobject_rejected_total": "GetObjectMetricNames.h:81, emitted from PeerImp.cpp:2725/:2743 only for a TMGetObjectByHash message refused as oversize or malformed_ledgerhash. A cooperating cluster never sends one.", "getobject_request_objects": "GetObjectMetricNames.h:86, emitted from PeerImp.cpp:2926 only while serving an inbound TMGetObjectByHash. The XRPL_METRIC_* macros create their instrument lazily on first use (MetricMacros.h:174-285), so no series exists until a peer actually requests objects by hash — which a 5-node cluster started at genesis and already in sync may never do.", diff --git a/docker/telemetry/workload/generate-validator-keys.sh b/docker/telemetry/workload/generate-validator-keys.sh index 7324a8bf61..48ebda8495 100755 --- a/docker/telemetry/workload/generate-validator-keys.sh +++ b/docker/telemetry/workload/generate-validator-keys.sh @@ -123,8 +123,14 @@ for i in $(seq 1 "$NUM_NODES"); do seed=$(echo "$result" | jq -r '.result.validation_seed') pubkey=$(echo "$result" | jq -r '.result.validation_public_key') + # Both fields must be present. jq -r prints the literal string "null" for + # a missing field, so an unvalidated pubkey would be written verbatim into + # validators.txt and xrpld would reject the file at startup. if [ -z "$seed" ] || [ "$seed" = "null" ]; then - die "Failed to generate key pair for node $i" + die "Failed to generate key pair for node $i: no validation_seed in response" + fi + if [ -z "$pubkey" ] || [ "$pubkey" = "null" ]; then + die "Failed to generate key pair for node $i: no validation_public_key in response" fi log " Node $i: ${pubkey:0:20}..." diff --git a/docker/telemetry/workload/prom_queries.py b/docker/telemetry/workload/prom_queries.py index b257c61241..a26a2b5459 100644 --- a/docker/telemetry/workload/prom_queries.py +++ b/docker/telemetry/workload/prom_queries.py @@ -28,6 +28,7 @@ Usage:: from __future__ import annotations +import asyncio import json import logging from dataclasses import dataclass @@ -38,6 +39,13 @@ import aiohttp logger = logging.getLogger("prom_queries") +# Instant queries run in parallel, but not all at once: a single-node +# Prometheus is easily saturated by a burst of the whole plan. With this cap +# the worst case (every query hitting the 30 s timeout) is +# ceil(len(plan) / 8) * 30 s rather than len(plan) * 30 s, which for a +# ~30-entry plan is ~2 min instead of ~15 min of a 30 min CI job. +MAX_CONCURRENT_QUERIES = 8 + @dataclass(frozen=True) class QueryEntry: @@ -147,6 +155,17 @@ async def run_query_plan( as "not yet observed" rather than as a regression. This keeps the baseline schema stable across runs with different load levels. + Queries run concurrently, at most MAX_CONCURRENT_QUERIES at a time. Keys + are emitted in plan order regardless of which query answers first. + + ``return_exceptions=True`` is what keeps the fan-out self-contained. + Without it the first escaping exception returns from ``gather`` while its + siblings keep running against a session the caller is about to close, so + the capture ends with dozens of orphaned queries and warnings from a + closed session. With it, every query is awaited before this returns, and + an unexpected failure is logged and recorded as no data -- the same + outcome _instant_query already produces for a query that fails. + Args: session: Shared aiohttp session. prom_url: Base URL of Prometheus (e.g. ``http://localhost:9090``). @@ -155,11 +174,32 @@ async def run_query_plan( Returns: Mapping from metric key to ``{"value": float|None, "unit": str}``. """ - results: dict[str, dict[str, Any]] = {} - for entry in plan: - value = await _instant_query(session, prom_url, entry.promql) - results[entry.key] = {"value": value, "unit": entry.unit} - return results + gate = asyncio.Semaphore(MAX_CONCURRENT_QUERIES) + + async def fetch(entry: QueryEntry) -> float | None: + """Run one plan entry once a query slot is free.""" + async with gate: + return await _instant_query(session, prom_url, entry.promql) + + results = await asyncio.gather( + *(fetch(entry) for entry in plan), return_exceptions=True + ) + + captured: dict[str, dict[str, Any]] = {} + for entry, result in zip(plan, results, strict=True): + value: float | None + if isinstance(result, BaseException): + logger.error( + "query for %s raised %s: %s", + entry.key, + type(result).__name__, + result, + ) + value = None + else: + value = result + captured[entry.key] = {"value": value, "unit": entry.unit} + return captured async def _instant_query( @@ -181,7 +221,10 @@ async def _instant_query( logger.warning("query HTTP %d: %s", resp.status, promql) return None body = await resp.json() - except (aiohttp.ClientError, TimeoutError) as exc: + # JSONDecodeError covers a 200 response whose body is not JSON: without it + # one malformed reply aborts the whole capture, since no caller of + # run_query_plan wraps it. + except (aiohttp.ClientError, TimeoutError, json.JSONDecodeError) as exc: logger.warning("query failed: %s — %s", promql, exc) return None diff --git a/docker/telemetry/workload/regression-metrics.json b/docker/telemetry/workload/regression-metrics.json index 0ea9dcbe86..050ce10a4f 100644 --- a/docker/telemetry/workload/regression-metrics.json +++ b/docker/telemetry/workload/regression-metrics.json @@ -1,13 +1,13 @@ { "_description": "Metric surface for the OTel-driven regression gate. Each entry names a metric, the quantiles to capture, and how to query Prometheus. The comparator compares current run against baseline-timings.json under these exact keys.", "_key_format": "{category}.{name}.p{quantile} (e.g. span.tx.process.p99, job.transaction.queued.p95). Only the categories defined below are captured; there is no rpc_methods group, so no rpc.* key is produced or gated (FU-4).", + "_excluded_spans": "rpc.process is deliberately absent from spans.names. It is created only in ServerHandler::processRequest() on the HTTP/JSON-RPC path, which the workload load generators, being WebSocket-only, never reach, so its quantiles were captured as null every run and could never gate. (The harness shell scripts do issue a few HTTP JSON-RPC health polls, far too few to produce a meaningful quantile.) See baselines/README.md.", "spans": { "_query_template": "histogram_quantile({quantile}, sum by (le) (rate(span_duration_milliseconds_bucket{span_name=\"{name}\"}[{window}])))", "_unit": "ms", "_quantiles": [0.5, 0.95, 0.99], "names": [ "rpc.ws_message", - "rpc.process", "tx.process", "tx.apply", "ledger.build", diff --git a/docker/telemetry/workload/requirements.txt b/docker/telemetry/workload/requirements.txt index f115de082b..062f216034 100644 --- a/docker/telemetry/workload/requirements.txt +++ b/docker/telemetry/workload/requirements.txt @@ -1,6 +1,786 @@ -# Python dependencies for Phase 10 workload tools. +# Python dependencies for the telemetry workload tools. +# +# cspell:ignore aiohappyeyeballs # # Install: pip install -r requirements.txt - -websockets>=12.0 -aiohttp>=3.9.0 +# +# Pinned with hashes so every install resolves to exactly these releases. +# Direct dependencies are websockets and aiohttp; the rest is aiohttp's +# transitive closure, which pip requires once any hash is present. +# +# Floors to keep when bumping: aiohttp >= 3.10.11 (earliest release carrying +# every 2024 aiohttp fix), websockets >= 14 (rejects a concurrent recv() with +# ConcurrencyError instead of a bare RuntimeError). +# +# Regenerate (python >= 3.11, the repo minimum, matching the universal floor): +# printf 'websockets>=17.0\naiohttp>=3.14.3\n' > requirements.in +# uv pip compile requirements.in --generate-hashes --universal \ +# --python-version 3.11 --output-file requirements.txt +aiohappyeyeballs==2.7.1 \ + --hash=sha256:065665c041c42a5938ed220bdcd7230f22527fbec085e1853d2402c8a3615d9d \ + --hash=sha256:9243213661e29250eb41368e5daa826fc017156c3b8a11440826b2e3ed376472 + # via aiohttp +aiohttp==3.14.3 \ + --hash=sha256:03cd2bde3d7f085b64e549c985f4bb928cad7e8ecf5323bfca320db548d81b39 \ + --hash=sha256:041badb8f84396357c4d3ad26de6afd7a32b112f43d3c63045c0c8278cfd2043 \ + --hash=sha256:0a5ff2dfbb9ce645fa5b8ef3e02c6c0b9cc3f6030ff863d0c51fffc50cb5541b \ + --hash=sha256:0fdea2281997af69da84c77ffa6f5938a0285f21fb3887c249d67419ca865b3d \ + --hash=sha256:11fb37ef075669eee52ab1928fbf6e1741fada40409fa309ebde9607a962aebf \ + --hash=sha256:134ac5ddcf61c6fad984b9a5727d83492ada43d63471db20fb73042c13fca62f \ + --hash=sha256:152516815ef926786a0b6ae2b8f1fd2e0c71582dee0b435636865316fd4891b7 \ + --hash=sha256:1576145bdceeb92382d899751e12743a3a5b8e460a841e3e50543859e54864dc \ + --hash=sha256:16100ad3ab8d649fdfbee87602d9d2dcdca9df0b9eda8a1b5fdc0d41f96da559 \ + --hash=sha256:16ea7e24c309fb7c0bbd505d149abe4fe4dccfb8db911db7dbec0921bc889a6f \ + --hash=sha256:18c441d0a8fca6de8d1f546849b9f0ab20d435993e2c5b59562b2fae6be2f929 \ + --hash=sha256:18cb43369747b2ae007bd2655fb8e63a099c2ff1d207962943636dac989b3147 \ + --hash=sha256:1b59533861b70a2185c8f4f350f791f39d64358ef6944ce71c5240c9ec0982c9 \ + --hash=sha256:1c5281acc88b92396f88c7e1e2748f8466689df22b80170e4f51efa712fb47a8 \ + --hash=sha256:1c5ec8fb1bcc31a8466f74aaf26c345d5c386fa4bd08a3f0eb9c7a4a3fe8b5bf \ + --hash=sha256:1caa7b0d05f3e3a36f87788c59e970a7ee1cefcfcbb924a9f138c4a6551c9cb7 \ + --hash=sha256:21c016079415ed3fd676963e9793700a566d85dbbd6bfc564b9b2d209147dcc8 \ + --hash=sha256:2498f0fe69ead802f9675beca44a7c21c62fdaa4ec5145ea1c3ad6edbee29f85 \ + --hash=sha256:25bd2708db6bdf6a6630dd37bdcdfcb47c4434d22ac69c64665b802910140b30 \ + --hash=sha256:270d3dace9ca2f10f0da5d8ebe519b7a310fc6112ed916e32df5866df0888553 \ + --hash=sha256:2e1161602f45a54de2ce0905243a95f58cb42dcd378402f3697f5e0b21e9d2e7 \ + --hash=sha256:2e9878ae68e4a5f1c0abe4dd497dbc3d51946f5837b56759e2a02e78fa90ef86 \ + --hash=sha256:30402d03a7c0ff52bce290b57e564e9079fd9d0cb545c8aba73f86a103162d2e \ + --hash=sha256:33a2d7c28d33797a2e99923dffa63f83d908a19b6bf26cfe80fa790aa5e1a75a \ + --hash=sha256:362a3fd481769cac1a824514bcd86fda51c65e8fe6e051099e008fddde6db17c \ + --hash=sha256:38901a84da3ce22249f6e860bf8f90d141bcab7da090cc398f8bb58c0e44b7da \ + --hash=sha256:39aded8c7f3b935b54aab1d8d73c70ec0ee2d3ec3b943e0e86611bc150ba47f5 \ + --hash=sha256:3a26434dafe408229ff3403458ca58de24fb51936504decac49ce6755f77e59d \ + --hash=sha256:3ae5b3a59436d089b5395d910121a390feed4d00578eb95a0fd1a329fe963100 \ + --hash=sha256:3d4f72af88ac2474bb5bca640030320e3d38a0163a1d7533500e87be458eef71 \ + --hash=sha256:3f42e9b78301f11c8f861746175d8b9c1ccef713fcad9eab396e2f6db8ed4a22 \ + --hash=sha256:42a67efc36300d052fb4508a53e8b6901b9284b599ae63945c377569c5fcc1e1 \ + --hash=sha256:48d67b87db6279c044760787eb01f6413032c2e6f3ba1cafaa492b1c8e578479 \ + --hash=sha256:498c6c623134f8e09a3c4e60bcd607a0b4590dd7dbf08dd40851b27cbb520ccb \ + --hash=sha256:49f7325beb0f85ef4aef5f48f490269575f83e6e2acad00a1d80b807eb027062 \ + --hash=sha256:4e3ac92d90e92773b2362d506068e9a948192bd553e743c5b2429e28527c8661 \ + --hash=sha256:530125ee1163c4219af35dc3aa1206e541e7b31b6efc1a3f93b70a136f65d427 \ + --hash=sha256:5373dc80ad1aa2fb9ad95c83f24eef418bbda3a61375f128e5b0192e4f3f9b32 \ + --hash=sha256:53e5179d8abb5710f8e83ba207c41c8d1261fcffd4616500e15ca2b7a33be10a \ + --hash=sha256:53e7b4ce82b54a8bcc71b3b67a5cbd177ca1d7f592cbc92cd38b7349f73482db \ + --hash=sha256:543906c127fb1d929b95076db19b83fa2d46751006ff1e23b093aa5ac4d8db42 \ + --hash=sha256:54cfcdee2770dac994417cbb0ee1f3eb0e7cb6b30c79bf44f2c02ff79ec5124a \ + --hash=sha256:55bdcc472aafe2de4a253045cc128007a64f1e0264fb675791e132ea5edaa3bd \ + --hash=sha256:56f355e79f71aef2a85c80305cc915f894b170dba76de5fe84f6351939b83c06 \ + --hash=sha256:5895ef58c4620afe02fa16044f023dc4dafec08158f9d08874a46a7dbc0341b8 \ + --hash=sha256:5bcb6ff3fdab1258a192679ff1a05d44f59626430aa05cd1a9d2447423599228 \ + --hash=sha256:5f08ec777f35ee70720233b8b9811d3bb5d728137f30ac91b7457709c3261ac0 \ + --hash=sha256:614c61d478b83953e261d02bb2df750f17227cd33ef8002945bf5aebbde21919 \ + --hash=sha256:617105e2c3018ee38d0c8ce5ee3c84f621a6d8b9f723202aacaff28449ca91ee \ + --hash=sha256:6debfa7312ff9d4c124dc71d72e9a0a4b9e0879e48ba6fcb42bef5c3300289e2 \ + --hash=sha256:7041d52c3a7fa20c9e8c182b534704abb19502c8bdcbde7ab23bfda6f642394f \ + --hash=sha256:70c987b27534f9ae1a723f47ae921571d616da21d3208282bf4c52af5164ac43 \ + --hash=sha256:74ab5b6a9fb13e873e5a90946588baecaf488745e1db1a4a5c433f971f035098 \ + --hash=sha256:78253b573e6ffab5028924fc98bc281aae05445969982a10864bc360dea2016c \ + --hash=sha256:7a75aa63cbf9b21cfaf60dc2657e19df2c2867d91707d653fee171ffeedd1371 \ + --hash=sha256:8800c996b01c2772a783e3e46f3e1abd5823029adca0df54231960de9bfefa5b \ + --hash=sha256:89176250f686cb9853c0fb7ead90e639e915b84a6f43eedc2a4e7ec21f1037f0 \ + --hash=sha256:8a5fd34f7f7410d1730d5c2ba873cacb2eed3fede366feb268a70ba22581ed8f \ + --hash=sha256:8b3b60de05f3dcb6f6a00f818bb2ec781cee4de0645f59ccaf99b1d1823b6100 \ + --hash=sha256:8f2f1c4c032c7cedd7d8da6f54c97b70266c6570c3108d3fdffee7188bb70529 \ + --hash=sha256:9491196535a88924a60afd5b5f434b5b203b6cc616250878dbdb223a8f7844bc \ + --hash=sha256:9aa6e61fdf20105c4144e755bd586008ff450791d67b1c8146fdc15959c4d51c \ + --hash=sha256:9d9edccfe496b476db5f398d97b865e9a6752bcf8aec4eef8390ce20fb64bb41 \ + --hash=sha256:9fc7b5bfec6573f3ae844f457fdde5adeb713f8b8e4a81ad64fc207b49383716 \ + --hash=sha256:a0dc483c00da8b673abbb367eb6f8d8f4bcec30eb58529ea13cb42e7fd2dfa33 \ + --hash=sha256:a3a8296e7ab5c295f53f1041487cb088e1480775aafbf7fe545d93b770a0f96f \ + --hash=sha256:a3e22975f905b89a55a488c2a08f2fdb2186175349e917d48985cc468a3d4c6e \ + --hash=sha256:a4af35c443e0b1a1bd6a8af3f3485d7fda15c142751a00f3ff8090f0b93346fa \ + --hash=sha256:a94dbaae5ae27bd849c93570669bff91e0510f33a80805738e3de72a7be0447b \ + --hash=sha256:ac74facc01463f138b0da5580329cfcc82818dea5656e83ddcd11268fc12ff80 \ + --hash=sha256:ad4c8b7488d745d2ca4838ebd8ae5ba9b56341d30b1da43640e4ce87f9f49646 \ + --hash=sha256:b014a6ed7cf912e787149fdc529166d3ceabac23f26efeea3158c9aba2354e7e \ + --hash=sha256:b20032766aedf6261c7a566585a40867d092ac03a0d81592d5370ef9b054f99b \ + --hash=sha256:b2466434105a4e03113c36ec775cc2ebe6676b62eae326fa670bb607ef788c1c \ + --hash=sha256:b304db572b4368edd8dda8a2274f73156fe15558fca4a917cb8a09fc47af5963 \ + --hash=sha256:ba59d59aba08ac02fc03b0c8983ccd5ee39a199d0552ce9e6d2b4845b34d59ae \ + --hash=sha256:bd52f811e65f6fb634b1047159657c98f52b407f8efec907bcfc09da9a4c0a25 \ + --hash=sha256:bdd0e2834dce1a26c1bbe26464861e16bbe217042cbff619247c11594472518c \ + --hash=sha256:c23ec8ee9d5ab2f5421f9c7fffce208435607af27fd46d4a44e031954352838f \ + --hash=sha256:c39846c3aad97a8530c89d7a3869a8f8e9e3762c6ac0504481e5c80948f7e807 \ + --hash=sha256:c3c200cf9757edd785051dc699c7ecbec22110dbfcb3fefc7a9f9695eda8ea7a \ + --hash=sha256:c7d3a97c678d34fc5b59da671ee9cd630096ddc643e7b5a30d54a2a6f3574d3f \ + --hash=sha256:c8653fd547c93a61aadc612007790f5555cdd18946fa48cf45e26d8ea4ea473d \ + --hash=sha256:cc7cb243a68167172f48c1fd43cee91ec4b1d40cefd190edd43369d1a6bc9c82 \ + --hash=sha256:ccd4893707b3e2a13e39c90d43cf80edf2e4d0457935bcc103bf2346214c3f15 \ + --hash=sha256:cd817772b2fcf2b8c0905795318485f9ec16eae60b29feb7f4c77085311637f0 \ + --hash=sha256:cda5fd5c95ad7a125a2e8464acc78b98b94c475a3780d6aa0aa157c93f470f4d \ + --hash=sha256:cef89a58e628c4efcac3275c2d68083f82426dcdc89c1492a6f654f9f7ea6ab9 \ + --hash=sha256:d1558173930a5a8d3069cee5c92fc91c87c4dbcb099debbb3622053717145a19 \ + --hash=sha256:d6088ec9894113802bddb3c09e974929aed2c7b3a8c456219b8aab4481f1a239 \ + --hash=sha256:d6218d92e450824e9b4881f44e8c09f1853b490f9a64130801024a4793b1b3b0 \ + --hash=sha256:d77640cc618c1d99fc4f8589c0f24a730adfa54eb1e57ef7bf0c8dfb78da898c \ + --hash=sha256:d7d2deec16eeedf55f2c7cf75b521ea3856a5177e123844f8fd0f114ce252cb5 \ + --hash=sha256:db332af25642007330fca8be5c4d194caf2bea7a7fc84415aff3497af5dfee6b \ + --hash=sha256:dd54d0e8717de95939766febac482ac0474d8ac3b048115f9f2b1d23a16e7db4 \ + --hash=sha256:ddcac3c6b382e81f1dd0499199d4136b877beb4cb5ef770bbbfba56c4b8f55d2 \ + --hash=sha256:df82f3787c940c94986b34222d59c9e38843fba85139f36e85255a82ad5355a9 \ + --hash=sha256:dfa68deb2a443bdaa3ea5297b0699c1464f08aef3812b486d1348eee61b07dc0 \ + --hash=sha256:dff9461ec275f22135650d5ba4b4931a11f3958df7dfbb8db630000d4dee0883 \ + --hash=sha256:e1e74298bab6ee0d6e749ed4fd1901c7e604bdda32c03d787a2cc71c46d0433d \ + --hash=sha256:e2667f0bbe7eb6c74eae5e9691441ad186e5845ca3cff63230fc09c4e7514f5d \ + --hash=sha256:e3be98a7c30b8c25d573dafba7171d66dfb05ee6a9070fc46535464ff97700a6 \ + --hash=sha256:e568e14940c09955aa51f4e645b6daa18a581c5dcfcd73744dcc86a856e3ced3 \ + --hash=sha256:e72ee89e28d907a18f46959b4eb0bb06701cc7f8cf4366e00029e2ccfaaf5924 \ + --hash=sha256:e92eb8acc45eb6a9f4935071a77edf5b85cc6f8dfad5cd99e97653c26593cdde \ + --hash=sha256:ea05e1f97ceea523942d9b2a7d7c0359d781d683d6b043f5943a602b14da4787 \ + --hash=sha256:eac645b09bcfdf73df7536331f0678c1086ea250981118ddb5199e17ccef72bb \ + --hash=sha256:eb0495d778817619273c108784292be161a924b9f5ae5cbbc70a2caa6838250b \ + --hash=sha256:ebe8e504f058fe91223351cecd2d9d6946c9d241bb0250d898ffbdf584cc72b0 \ + --hash=sha256:ed099d105449c4f9e84f24af203cd131349d4761d8813fa7e02c32e7128cd910 \ + --hash=sha256:f0f177d1b195b9e06376cfd7d308d8a1b920909a609d03ac82a8c73bbb16d3b9 \ + --hash=sha256:f3d2669fe7dec7fc359ecdb5984b29b50d85d5d00f8c1cb61de4f4a24ee42627 \ + --hash=sha256:f4e05329faa0ea1a404b37de4f034fd2c2defcca06a68dc6745e4e56c88e8a48 \ + --hash=sha256:f53bcd52f585e1ac3e590d61434eb61f9a88c38df041b4ea126d97144344a77b \ + --hash=sha256:f55119f7bf25f49ed210f6096090715da24f2943c62102448915fde3c62877ce \ + --hash=sha256:f631fe87a6f30df5fbe6d79640b25e4cffb38c31c7fb6f10871517b84b0f8c1a \ + --hash=sha256:f8fb78a83c9e5f741ca3a68cfb455c1f5bb83b4e7249a3848b3cd78d0a8563b0 \ + --hash=sha256:fa9467a8113aa69d3d7c55a70ef0b7c636010a40993f3df9d9d0d73b3eb7ef24 \ + --hash=sha256:fd51ebf9d3a00c074df4ede271023f4d2dba289bcc740b88191872716014e3c5 + # via -r requirements.in +aiosignal==1.4.0 \ + --hash=sha256:053243f8b92b990551949e63930a839ff0cf0b0ebbe0597b0f3fb19e1a0fe82e \ + --hash=sha256:f47eecd9468083c2029cc99945502cb7708b082c232f9aca65da147157b251c7 + # via aiohttp +attrs==26.1.0 \ + --hash=sha256:c647aa4a12dfbad9333ca4e71fe62ddc36f4e63b2d260a37a8b83d2f043ac309 \ + --hash=sha256:d03ceb89cb322a8fd706d4fb91940737b6642aa36998fe130a9bc96c985eff32 + # via aiohttp +frozenlist==1.8.0 \ + --hash=sha256:0325024fe97f94c41c08872db482cf8ac4800d80e79222c6b0b7b162d5b13686 \ + --hash=sha256:032efa2674356903cd0261c4317a561a6850f3ac864a63fc1583147fb05a79b0 \ + --hash=sha256:03ae967b4e297f58f8c774c7eabcce57fe3c2434817d4385c50661845a058121 \ + --hash=sha256:06be8f67f39c8b1dc671f5d83aaefd3358ae5cdcf8314552c57e7ed3e6475bdd \ + --hash=sha256:073f8bf8becba60aa931eb3bc420b217bb7d5b8f4750e6f8b3be7f3da85d38b7 \ + --hash=sha256:07cdca25a91a4386d2e76ad992916a85038a9b97561bf7a3fd12d5d9ce31870c \ + --hash=sha256:09474e9831bc2b2199fad6da3c14c7b0fbdd377cce9d3d77131be28906cb7d84 \ + --hash=sha256:0c18a16eab41e82c295618a77502e17b195883241c563b00f0aa5106fc4eaa0d \ + --hash=sha256:0f96534f8bfebc1a394209427d0f8a63d343c9779cda6fc25e8e121b5fd8555b \ + --hash=sha256:102e6314ca4da683dca92e3b1355490fed5f313b768500084fbe6371fddfdb79 \ + --hash=sha256:11847b53d722050808926e785df837353bd4d75f1d494377e59b23594d834967 \ + --hash=sha256:119fb2a1bd47307e899c2fac7f28e85b9a543864df47aa7ec9d3c1b4545f096f \ + --hash=sha256:13d23a45c4cebade99340c4165bd90eeb4a56c6d8a9d8aa49568cac19a6d0dc4 \ + --hash=sha256:154e55ec0655291b5dd1b8731c637ecdb50975a2ae70c606d100750a540082f7 \ + --hash=sha256:168c0969a329b416119507ba30b9ea13688fafffac1b7822802537569a1cb0ef \ + --hash=sha256:17c883ab0ab67200b5f964d2b9ed6b00971917d5d8a92df149dc2c9779208ee9 \ + --hash=sha256:1a7607e17ad33361677adcd1443edf6f5da0ce5e5377b798fba20fae194825f3 \ + --hash=sha256:1a7fa382a4a223773ed64242dbe1c9c326ec09457e6b8428efb4118c685c3dfd \ + --hash=sha256:1aa77cb5697069af47472e39612976ed05343ff2e84a3dcf15437b232cbfd087 \ + --hash=sha256:1b9290cf81e95e93fdf90548ce9d3c1211cf574b8e3f4b3b7cb0537cf2227068 \ + --hash=sha256:20e63c9493d33ee48536600d1a5c95eefc870cd71e7ab037763d1fbb89cc51e7 \ + --hash=sha256:21900c48ae04d13d416f0e1e0c4d81f7931f73a9dfa0b7a8746fb2fe7dd970ed \ + --hash=sha256:229bf37d2e4acdaf808fd3f06e854a4a7a3661e871b10dc1f8f1896a3b05f18b \ + --hash=sha256:2552f44204b744fba866e573be4c1f9048d6a324dfe14475103fd51613eb1d1f \ + --hash=sha256:27c6e8077956cf73eadd514be8fb04d77fc946a7fe9f7fe167648b0b9085cc25 \ + --hash=sha256:28bd570e8e189d7f7b001966435f9dac6718324b5be2990ac496cf1ea9ddb7fe \ + --hash=sha256:294e487f9ec720bd8ffcebc99d575f7eff3568a08a253d1ee1a0378754b74143 \ + --hash=sha256:29548f9b5b5e3460ce7378144c3010363d8035cea44bc0bf02d57f5a685e084e \ + --hash=sha256:2c5dcbbc55383e5883246d11fd179782a9d07a986c40f49abe89ddf865913930 \ + --hash=sha256:2dc43a022e555de94c3b68a4ef0b11c4f747d12c024a520c7101709a2144fb37 \ + --hash=sha256:2f05983daecab868a31e1da44462873306d3cbfd76d1f0b5b69c473d21dbb128 \ + --hash=sha256:33139dc858c580ea50e7e60a1b0ea003efa1fd42e6ec7fdbad78fff65fad2fd2 \ + --hash=sha256:332db6b2563333c5671fecacd085141b5800cb866be16d5e3eb15a2086476675 \ + --hash=sha256:33f48f51a446114bc5d251fb2954ab0164d5be02ad3382abcbfe07e2531d650f \ + --hash=sha256:34187385b08f866104f0c0617404c8eb08165ab1272e884abc89c112e9c00746 \ + --hash=sha256:342c97bf697ac5480c0a7ec73cd700ecfa5a8a40ac923bd035484616efecc2df \ + --hash=sha256:3462dd9475af2025c31cc61be6652dfa25cbfb56cbbf52f4ccfe029f38decaf8 \ + --hash=sha256:39ecbc32f1390387d2aa4f5a995e465e9e2f79ba3adcac92d68e3e0afae6657c \ + --hash=sha256:3e0761f4d1a44f1d1a47996511752cf3dcec5bbdd9cc2b4fe595caf97754b7a0 \ + --hash=sha256:3ede829ed8d842f6cd48fc7081d7a41001a56f1f38603f9d49bf3020d59a31ad \ + --hash=sha256:3ef2d026f16a2b1866e1d86fc4e1291e1ed8a387b2c333809419a2f8b3a77b82 \ + --hash=sha256:405e8fe955c2280ce66428b3ca55e12b3c4e9c336fb2103a4937e891c69a4a29 \ + --hash=sha256:42145cd2748ca39f32801dad54aeea10039da6f86e303659db90db1c4b614c8c \ + --hash=sha256:4314debad13beb564b708b4a496020e5306c7333fa9a3ab90374169a20ffab30 \ + --hash=sha256:433403ae80709741ce34038da08511d4a77062aa924baf411ef73d1146e74faf \ + --hash=sha256:44389d135b3ff43ba8cc89ff7f51f5a0bb6b63d829c8300f79a2fe4fe61bcc62 \ + --hash=sha256:48e6d3f4ec5c7273dfe83ff27c91083c6c9065af655dc2684d2c200c94308bb5 \ + --hash=sha256:494a5952b1c597ba44e0e78113a7266e656b9794eec897b19ead706bd7074383 \ + --hash=sha256:4970ece02dbc8c3a92fcc5228e36a3e933a01a999f7094ff7c23fbd2beeaa67c \ + --hash=sha256:4e0c11f2cc6717e0a741f84a527c52616140741cd812a50422f83dc31749fb52 \ + --hash=sha256:50066c3997d0091c411a66e710f4e11752251e6d2d73d70d8d5d4c76442a199d \ + --hash=sha256:517279f58009d0b1f2e7c1b130b377a349405da3f7621ed6bfae50b10adf20c1 \ + --hash=sha256:54b2077180eb7f83dd52c40b2750d0a9f175e06a42e3213ce047219de902717a \ + --hash=sha256:5500ef82073f599ac84d888e3a8c1f77ac831183244bfd7f11eaa0289fb30714 \ + --hash=sha256:581ef5194c48035a7de2aefc72ac6539823bb71508189e5de01d60c9dcd5fa65 \ + --hash=sha256:59a6a5876ca59d1b63af8cd5e7ffffb024c3dc1e9cf9301b21a2e76286505c95 \ + --hash=sha256:5a3a935c3a4e89c733303a2d5a7c257ea44af3a56c8202df486b7f5de40f37e1 \ + --hash=sha256:5c1c8e78426e59b3f8005e9b19f6ff46e5845895adbde20ece9218319eca6506 \ + --hash=sha256:5d63a068f978fc69421fb0e6eb91a9603187527c86b7cd3f534a5b77a592b888 \ + --hash=sha256:667c3777ca571e5dbeb76f331562ff98b957431df140b54c85fd4d52eea8d8f6 \ + --hash=sha256:6da155091429aeba16851ecb10a9104a108bcd32f6c1642867eadaee401c1c41 \ + --hash=sha256:6dc4126390929823e2d2d9dc79ab4046ed74680360fc5f38b585c12c66cdf459 \ + --hash=sha256:7398c222d1d405e796970320036b1b563892b65809d9e5261487bb2c7f7b5c6a \ + --hash=sha256:74c51543498289c0c43656701be6b077f4b265868fa7f8a8859c197006efb608 \ + --hash=sha256:776f352e8329135506a1d6bf16ac3f87bc25b28e765949282dcc627af36123aa \ + --hash=sha256:778a11b15673f6f1df23d9586f83c4846c471a8af693a22e066508b77d201ec8 \ + --hash=sha256:78f7b9e5d6f2fdb88cdde9440dc147259b62b9d3b019924def9f6478be254ac1 \ + --hash=sha256:799345ab092bee59f01a915620b5d014698547afd011e691a208637312db9186 \ + --hash=sha256:7bf6cdf8e07c8151fba6fe85735441240ec7f619f935a5205953d58009aef8c6 \ + --hash=sha256:8009897cdef112072f93a0efdce29cd819e717fd2f649ee3016efd3cd885a7ed \ + --hash=sha256:80f85f0a7cc86e7a54c46d99c9e1318ff01f4687c172ede30fd52d19d1da1c8e \ + --hash=sha256:8585e3bb2cdea02fc88ffa245069c36555557ad3609e83be0ec71f54fd4abb52 \ + --hash=sha256:878be833caa6a3821caf85eb39c5ba92d28e85df26d57afb06b35b2efd937231 \ + --hash=sha256:8a76ea0f0b9dfa06f254ee06053d93a600865b3274358ca48a352ce4f0798450 \ + --hash=sha256:8b7b94a067d1c504ee0b16def57ad5738701e4ba10cec90529f13fa03c833496 \ + --hash=sha256:8d92f1a84bb12d9e56f818b3a746f3efba93c1b63c8387a73dde655e1e42282a \ + --hash=sha256:908bd3f6439f2fef9e85031b59fd4f1297af54415fb60e4254a95f75b3cab3f3 \ + --hash=sha256:92db2bf818d5cc8d9c1f1fc56b897662e24ea5adb36ad1f1d82875bd64e03c24 \ + --hash=sha256:940d4a017dbfed9daf46a3b086e1d2167e7012ee297fef9e1c545c4d022f5178 \ + --hash=sha256:957e7c38f250991e48a9a73e6423db1bb9dd14e722a10f6b8bb8e16a0f55f695 \ + --hash=sha256:96153e77a591c8adc2ee805756c61f59fef4cf4073a9275ee86fe8cba41241f7 \ + --hash=sha256:96f423a119f4777a4a056b66ce11527366a8bb92f54e541ade21f2374433f6d4 \ + --hash=sha256:97260ff46b207a82a7567b581ab4190bd4dfa09f4db8a8b49d1a958f6aa4940e \ + --hash=sha256:974b28cf63cc99dfb2188d8d222bc6843656188164848c4f679e63dae4b0708e \ + --hash=sha256:9ff15928d62a0b80bb875655c39bf517938c7d589554cbd2669be42d97c2cb61 \ + --hash=sha256:a6483e309ca809f1efd154b4d37dc6d9f61037d6c6a81c2dc7a15cb22c8c5dca \ + --hash=sha256:a88f062f072d1589b7b46e951698950e7da00442fc1cacbe17e19e025dc327ad \ + --hash=sha256:ac913f8403b36a2c8610bbfd25b8013488533e71e62b4b4adce9c86c8cea905b \ + --hash=sha256:adbeebaebae3526afc3c96fad434367cafbfd1b25d72369a9e5858453b1bb71a \ + --hash=sha256:b2a095d45c5d46e5e79ba1e5b9cb787f541a8dee0433836cea4b96a2c439dcd8 \ + --hash=sha256:b3210649ee28062ea6099cfda39e147fa1bc039583c8ee4481cb7811e2448c51 \ + --hash=sha256:b37f6d31b3dcea7deb5e9696e529a6aa4a898adc33db82da12e4c60a7c4d2011 \ + --hash=sha256:b4dec9482a65c54a5044486847b8a66bf10c9cb4926d42927ec4e8fd5db7fed8 \ + --hash=sha256:b4f3b365f31c6cd4af24545ca0a244a53688cad8834e32f56831c4923b50a103 \ + --hash=sha256:b6db2185db9be0a04fecf2f241c70b63b1a242e2805be291855078f2b404dd6b \ + --hash=sha256:b9be22a69a014bc47e78072d0ecae716f5eb56c15238acca0f43d6eb8e4a5bda \ + --hash=sha256:bac9c42ba2ac65ddc115d930c78d24ab8d4f465fd3fc473cdedfccadb9429806 \ + --hash=sha256:bf0a7e10b077bf5fb9380ad3ae8ce20ef919a6ad93b4552896419ac7e1d8e042 \ + --hash=sha256:c23c3ff005322a6e16f71bf8692fcf4d5a304aaafe1e262c98c6d4adc7be863e \ + --hash=sha256:c4c800524c9cd9bac5166cd6f55285957fcfc907db323e193f2afcd4d9abd69b \ + --hash=sha256:c7366fe1418a6133d5aa824ee53d406550110984de7637d65a178010f759c6ef \ + --hash=sha256:c8d1634419f39ea6f5c427ea2f90ca85126b54b50837f31497f3bf38266e853d \ + --hash=sha256:c9a63152fe95756b85f31186bddf42e4c02c6321207fd6601a1c89ebac4fe567 \ + --hash=sha256:cb89a7f2de3602cfed448095bab3f178399646ab7c61454315089787df07733a \ + --hash=sha256:cba69cb73723c3f329622e34bdbf5ce1f80c21c290ff04256cff1cd3c2036ed2 \ + --hash=sha256:cee686f1f4cadeb2136007ddedd0aaf928ab95216e7691c63e50a8ec066336d0 \ + --hash=sha256:cf253e0e1c3ceb4aaff6df637ce033ff6535fb8c70a764a8f46aafd3d6ab798e \ + --hash=sha256:d1eaff1d00c7751b7c6662e9c5ba6eb2c17a2306ba5e2a37f24ddf3cc953402b \ + --hash=sha256:d3bb933317c52d7ea5004a1c442eef86f426886fba134ef8cf4226ea6ee1821d \ + --hash=sha256:d4d3214a0f8394edfa3e303136d0575eece0745ff2b47bd2cb2e66dd92d4351a \ + --hash=sha256:d6a5df73acd3399d893dafc71663ad22534b5aa4f94e8a2fabfe856c3c1b6a52 \ + --hash=sha256:d8b7138e5cd0647e4523d6685b0eac5d4be9a184ae9634492f25c6eb38c12a47 \ + --hash=sha256:db1e72ede2d0d7ccb213f218df6a078a9c09a7de257c2fe8fcef16d5925230b1 \ + --hash=sha256:e25ac20a2ef37e91c1b39938b591457666a0fa835c7783c3a8f33ea42870db94 \ + --hash=sha256:e2de870d16a7a53901e41b64ffdf26f2fbb8917b3e6ebf398098d72c5b20bd7f \ + --hash=sha256:e4a3408834f65da56c83528fb52ce7911484f0d1eaf7b761fc66001db1646eff \ + --hash=sha256:eaa352d7047a31d87dafcacbabe89df0aa506abb5b1b85a2fb91bc3faa02d822 \ + --hash=sha256:eab8145831a0d56ec9c4139b6c3e594c7a83c2c8be25d5bcf2d86136a532287a \ + --hash=sha256:ec3cc8c5d4084591b4237c0a272cc4f50a5b03396a47d9caaf76f5d7b38a4f11 \ + --hash=sha256:edee74874ce20a373d62dc28b0b18b93f645633c2943fd90ee9d898550770581 \ + --hash=sha256:eefdba20de0d938cec6a89bd4d70f346a03108a19b9df4248d3cf0d88f1b0f51 \ + --hash=sha256:ef2b7b394f208233e471abc541cc6991f907ffd47dc72584acee3147899d6565 \ + --hash=sha256:f21f00a91358803399890ab167098c131ec2ddd5f8f5fd5fe9c9f2c6fcd91e40 \ + --hash=sha256:f4be2e3d8bc8aabd566f8d5b8ba7ecc09249d74ba3c9ed52e54dc23a293f0b92 \ + --hash=sha256:f57fb59d9f385710aa7060e89410aeb5058b99e62f4d16b08b91986b9a2140c2 \ + --hash=sha256:f6292f1de555ffcc675941d65fffffb0a5bcd992905015f85d0592201793e0e5 \ + --hash=sha256:f833670942247a14eafbb675458b4e61c82e002a148f49e68257b79296e865c4 \ + --hash=sha256:fa47e444b8ba08fffd1c18e8cdb9a75db1b6a27f17507522834ad13ed5922b93 \ + --hash=sha256:fb30f9626572a76dfe4293c7194a09fb1fe93ba94c7d4f720dfae3b646b45027 \ + --hash=sha256:fe3c58d2f5db5fbd18c2987cba06d51b0529f52bc3a6cdc33d3f4eab725104bd + # via + # aiohttp + # aiosignal +idna==3.18 \ + --hash=sha256:7f952cbe720b688055e3f87de14f5c3e5fdaa8bc3928985c4077ca689de849a2 \ + --hash=sha256:ffb385a7e039654cef1ab9ef32c6fafe283c0c0467bba1d9029738ce4a14a848 + # via yarl +multidict==6.7.1 \ + --hash=sha256:026d264228bcd637d4e060844e39cdc60f86c479e463d49075dedc21b18fbbe0 \ + --hash=sha256:03ede2a6ffbe8ef936b92cb4529f27f42be7f56afcdab5ab739cd5f27fb1cbf9 \ + --hash=sha256:0458c978acd8e6ea53c81eefaddbbee9c6c5e591f41b3f5e8e194780fe026581 \ + --hash=sha256:067343c68cd6612d375710f895337b3a98a033c94f14b9a99eff902f205424e2 \ + --hash=sha256:08ccb2a6dc72009093ebe7f3f073e5ec5964cba9a706fa94b1a1484039b87941 \ + --hash=sha256:0b38ebffd9be37c1170d33bc0f36f4f262e0a09bc1aac1c34c7aa51a7293f0b3 \ + --hash=sha256:0b4c48648d7649c9335cf1927a8b87fa692de3dcb15faa676c6a6f1f1aabda43 \ + --hash=sha256:0d17522c37d03e85c8098ec8431636309b2682cf12e58f4dbc76121fb50e4962 \ + --hash=sha256:0e161ddf326db5577c3a4cc2d8648f81456e8a20d40415541587a71620d7a7d1 \ + --hash=sha256:0e697826df7eb63418ee190fd06ce9f1803593bb4b9517d08c60d9b9a7f69d8f \ + --hash=sha256:10ae39c9cfe6adedcdb764f5e8411d4a92b055e35573a2eaa88d3323289ef93c \ + --hash=sha256:121a34e5bfa410cdf2c8c49716de160de3b1dbcd86b49656f5681e4543bcd1a8 \ + --hash=sha256:128441d052254f42989ef98b7b6a6ecb1e6f708aa962c7984235316db59f50fa \ + --hash=sha256:12fad252f8b267cc75b66e8fc51b3079604e8d43a75428ffe193cd9e2195dfd6 \ + --hash=sha256:14525a5f61d7d0c94b368a42cff4c9a4e7ba2d52e2672a7b23d84dc86fb02b0c \ + --hash=sha256:17207077e29342fdc2c9a82e4b306f1127bf1ea91f8b71e02d4798a70bb99991 \ + --hash=sha256:17307b22c217b4cf05033dabefe68255a534d637c6c9b0cc8382718f87be4262 \ + --hash=sha256:1b99af4d9eec0b49927b4402bcbb58dea89d3e0db8806a4086117019939ad3dd \ + --hash=sha256:1d540e51b7e8e170174555edecddbd5538105443754539193e3e1061864d444d \ + --hash=sha256:1e3a8bb24342a8201d178c3b4984c26ba81a577c80d4d525727427460a50c22d \ + --hash=sha256:1fa6609d0364f4f6f58351b4659a1f3e0e898ba2a8c5cac04cb2c7bc556b0bc5 \ + --hash=sha256:21f830fe223215dffd51f538e78c172ed7c7f60c9b96a2bf05c4848ad49921c3 \ + --hash=sha256:233b398c29d3f1b9676b4b6f75c518a06fcb2ea0b925119fb2c1bc35c05e1601 \ + --hash=sha256:24c0cf81544ca5e17cfcb6e482e7a82cd475925242b308b890c9452a074d4505 \ + --hash=sha256:25167cc263257660290fba06b9318d2026e3c910be240a146e1f66dd114af2b0 \ + --hash=sha256:253282d70d67885a15c8a7716f3a73edf2d635793ceda8173b9ecc21f2fb8292 \ + --hash=sha256:273d23f4b40f3dce4d6c8a821c741a86dec62cded82e1175ba3d99be128147ed \ + --hash=sha256:283ddac99f7ac25a4acadbf004cb5ae34480bbeb063520f70ce397b281859362 \ + --hash=sha256:28ca5ce2fd9716631133d0e9a9b9a745ad7f60bac2bccafb56aa380fc0b6c511 \ + --hash=sha256:2b41f5fed0ed563624f1c17630cb9941cf2309d4df00e494b551b5f3e3d67a23 \ + --hash=sha256:2bbd113e0d4af5db41d5ebfe9ccaff89de2120578164f86a5d17d5a576d1e5b2 \ + --hash=sha256:2e1425e2f99ec5bd36c15a01b690a1a2456209c5deed58f95469ffb46039ccbb \ + --hash=sha256:2e2d2ed645ea29f31c4c7ea1552fcfd7cb7ba656e1eafd4134a6620c9f5fdd9e \ + --hash=sha256:3758692429e4e32f1ba0df23219cd0b4fc0a52f476726fff9337d1a57676a582 \ + --hash=sha256:38fb49540705369bab8484db0689d86c0a33a0a9f2c1b197f506b71b4b6c19b0 \ + --hash=sha256:3943debf0fbb57bdde5901695c11094a9a36723e5c03875f87718ee15ca2f4d2 \ + --hash=sha256:398c1478926eca669f2fd6a5856b6de9c0acf23a2cb59a14c0ba5844fa38077e \ + --hash=sha256:3ab8b9d8b75aef9df299595d5388b14530839f6422333357af1339443cff777d \ + --hash=sha256:3bd231490fa7217cc832528e1cd8752a96f0125ddd2b5749390f7c3ec8721b65 \ + --hash=sha256:3d51ff4785d58d3f6c91bdbffcb5e1f7ddfda557727043aa20d20ec4f65e324a \ + --hash=sha256:3fccb473e87eaa1382689053e4a4618e7ba7b9b9b8d6adf2027ee474597128cd \ + --hash=sha256:401c5a650f3add2472d1d288c26deebc540f99e2fb83e9525007a74cd2116f1d \ + --hash=sha256:41f2952231456154ee479651491e94118229844dd7226541788be783be2b5108 \ + --hash=sha256:432feb25a1cb67fe82a9680b4d65fb542e4635cb3166cd9c01560651ad60f177 \ + --hash=sha256:439cbebd499f92e9aa6793016a8acaa161dfa749ae86d20960189f5398a19144 \ + --hash=sha256:4885cb0e817aef5d00a2e8451d4665c1808378dc27c2705f1bf4ef8505c0d2e5 \ + --hash=sha256:497394b3239fc6f0e13a78a3e1b61296e72bf1c5f94b4c4eb80b265c37a131cd \ + --hash=sha256:497bde6223c212ba11d462853cfa4f0ae6ef97465033e7dc9940cdb3ab5b48e5 \ + --hash=sha256:4cfb48c6ea66c83bcaaf7e4dfa7ec1b6bbcf751b7db85a328902796dfde4c060 \ + --hash=sha256:538cec1e18c067d0e6103aa9a74f9e832904c957adc260e61cd9d8cf0c3b3d37 \ + --hash=sha256:55d97cc6dae627efa6a6e548885712d4864b81110ac76fa4e534c03819fa4a56 \ + --hash=sha256:563fe25c678aaba333d5399408f5ec3c383ca5b663e7f774dd179a520b8144df \ + --hash=sha256:57b46b24b5d5ebcc978da4ec23a819a9402b4228b8a90d9c656422b4bdd8a963 \ + --hash=sha256:5884a04f4ff56c6120f6ccf703bdeb8b5079d808ba604d4d53aec0d55dc33568 \ + --hash=sha256:59bc83d3f66b41dac1e7460aac1d196edc70c9ba3094965c467715a70ecb46db \ + --hash=sha256:5a37ca18e360377cfda1d62f5f382ff41f2b8c4ccb329ed974cc2e1643440118 \ + --hash=sha256:5c4b9bfc148f5a91be9244d6264c53035c8a0dcd2f51f1c3c6e30e30ebaa1c84 \ + --hash=sha256:5e01429a929600e7dab7b166062d9bb54a5eed752384c7384c968c2afab8f50f \ + --hash=sha256:5fa6a95dfee63893d80a34758cd0e0c118a30b8dcb46372bf75106c591b77889 \ + --hash=sha256:619e5a1ac57986dbfec9f0b301d865dddf763696435e2962f6d9cf2fdff2bb71 \ + --hash=sha256:65573858d27cdeaca41893185677dc82395159aa28875a8867af66532d413a8f \ + --hash=sha256:6704fa2b7453b2fb121740555fa1ee20cd98c4d011120caf4d2b8d4e7c76eec0 \ + --hash=sha256:6aac4f16b472d5b7dc6f66a0d49dd57b0e0902090be16594dc9ebfd3d17c47e7 \ + --hash=sha256:6b10359683bd8806a200fd2909e7c8ca3a7b24ec1d8132e483d58e791d881048 \ + --hash=sha256:6b83cabdc375ffaaa15edd97eb7c0c672ad788e2687004990074d7d6c9b140c8 \ + --hash=sha256:6d3bc717b6fe763b8be3f2bee2701d3c8eb1b2a8ae9f60910f1b2860c82b6c49 \ + --hash=sha256:6f77ce314a29263e67adadc7e7c1bc699fcb3a305059ab973d038f87caa42ed0 \ + --hash=sha256:749aa54f578f2e5f439538706a475aa844bfa8ef75854b1401e6e528e4937cf9 \ + --hash=sha256:7a7e590ff876a3eaf1c02a4dfe0724b6e69a9e9de6d8f556816f29c496046e59 \ + --hash=sha256:7dfb78d966b2c906ae1d28ccf6e6712a3cd04407ee5088cd276fe8cb42186190 \ + --hash=sha256:7eee46ccb30ff48a1e35bb818cc90846c6be2b68240e42a78599166722cea709 \ + --hash=sha256:7ff981b266af91d7b4b3793ca3382e53229088d193a85dfad6f5f4c27fc73e5d \ + --hash=sha256:841189848ba629c3552035a6a7f5bf3b02eb304e9fea7492ca220a8eda6b0e5c \ + --hash=sha256:844c5bca0b5444adb44a623fb0a1310c2f4cd41f402126bb269cd44c9b3f3e1e \ + --hash=sha256:84e61e3af5463c19b67ced91f6c634effb89ef8bfc5ca0267f954451ed4bb6a2 \ + --hash=sha256:8affcf1c98b82bc901702eb73b6947a1bfa170823c153fe8a47b5f5f02e48e40 \ + --hash=sha256:8be1802715a8e892c784c0197c2ace276ea52702a0ede98b6310c8f255a5afb3 \ + --hash=sha256:8f333ec9c5eb1b7105e3b84b53141e66ca05a19a605368c55450b6ba208cb9ee \ + --hash=sha256:9004d8386d133b7e6135679424c91b0b854d2d164af6ea3f289f8f2761064609 \ + --hash=sha256:90efbcf47dbe33dcf643a1e400d67d59abeac5db07dc3f27d6bdeae497a2198c \ + --hash=sha256:935434b9853c7c112eee7ac891bc4cb86455aa631269ae35442cb316790c1445 \ + --hash=sha256:93b1818e4a6e0930454f0f2af7dfce69307ca03cdcfb3739bf4d91241967b6c1 \ + --hash=sha256:95922cee9a778659e91db6497596435777bd25ed116701a4c034f8e46544955a \ + --hash=sha256:960c83bf01a95b12b08fd54324a4eb1d5b52c88932b5cba5d6e712bb3ed12eb5 \ + --hash=sha256:97231140a50f5d447d3164f994b86a0bed7cd016e2682f8650d6a9158e14fd31 \ + --hash=sha256:974e72a2474600827abaeda71af0c53d9ebbc3c2eb7da37b37d7829ae31232d8 \ + --hash=sha256:97891f3b1b3ffbded884e2916cacf3c6fc87b66bb0dde46f7357404750559f33 \ + --hash=sha256:98655c737850c064a65e006a3df7c997cd3b220be4ec8fe26215760b9697d4d7 \ + --hash=sha256:98bc624954ec4d2c7cb074b8eefc2b5d0ce7d482e410df446414355d158fe4ca \ + --hash=sha256:98c5787b0a0d9a41d9311eae44c3b76e6753def8d8870ab501320efe75a6a5f8 \ + --hash=sha256:9b0d9b91d1aa44db9c1f1ecd0d9d2ae610b2f4f856448664e01a3b35899f3f92 \ + --hash=sha256:9c90fed18bffc0189ba814749fdcc102b536e83a9f738a9003e569acd540a733 \ + --hash=sha256:9d624335fd4fa1c08a53f8b4be7676ebde19cd092b3895c421045ca87895b429 \ + --hash=sha256:9f9af11306994335398293f9958071019e3ab95e9a707dc1383a35613f6abcb9 \ + --hash=sha256:a0543217a6a017692aa6ae5cc39adb75e587af0f3a82288b1492eb73dd6cc2a4 \ + --hash=sha256:a088b62bd733e2ad12c50dad01b7d0166c30287c166e137433d3b410add807a6 \ + --hash=sha256:a407f13c188f804c759fc6a9f88286a565c242a76b27626594c133b82883b5c2 \ + --hash=sha256:a90f75c956e32891a4eda3639ce6dd86e87105271f43d43442a3aedf3cddf172 \ + --hash=sha256:a9fc4caa29e2e6ae408d1c450ac8bf19892c5fca83ee634ecd88a53332c59981 \ + --hash=sha256:aa23b001d968faef416ff70dc0f1ab045517b9b42a90edd3e9bcdb06479e31d5 \ + --hash=sha256:ac1c665bad8b5d762f5f85ebe4d94130c26965f11de70c708c75671297c776de \ + --hash=sha256:af959b9beeb66c822380f222f0e0a1889331597e81f1ded7f374f3ecb0fd6c52 \ + --hash=sha256:b0fa96985700739c4c7853a43c0b3e169360d6855780021bfc6d0f1ce7c123e7 \ + --hash=sha256:b26684587228afed0d50cf804cc71062cc9c1cdf55051c4c6345d372947b268c \ + --hash=sha256:b4938326284c4f1224178a560987b6cf8b4d38458b113d9b8c1db1a836e640a2 \ + --hash=sha256:b8c990b037d2fff2f4e33d3f21b9b531c5745b33a49a7d6dbe7a177266af44f6 \ + --hash=sha256:ba0a9fb644d0c1a2194cf7ffb043bd852cea63a57f66fbd33959f7dae18517bf \ + --hash=sha256:bb08271280173720e9fea9ede98e5231defcbad90f1624bea26f32ec8a956e2f \ + --hash=sha256:bdbf9f3b332abd0cdb306e7c2113818ab1e922dc84b8f8fd06ec89ed2a19ab8b \ + --hash=sha256:bfde23ef6ed9db7eaee6c37dcec08524cb43903c60b285b172b6c094711b3961 \ + --hash=sha256:c0abd12629b0af3cf590982c0b413b1e7395cd4ec026f30986818ab95bfaa94a \ + --hash=sha256:c102791b1c4f3ab36ce4101154549105a53dc828f016356b3e3bcae2e3a039d3 \ + --hash=sha256:c3a32d23520ee37bf327d1e1a656fec76a2edd5c038bf43eddfa0572ec49c60b \ + --hash=sha256:c524c6fb8fc342793708ab111c4dbc90ff9abd568de220432500e47e990c0358 \ + --hash=sha256:c5f0c21549ab432b57dcc82130f388d84ad8179824cc3f223d5e7cfbfd4143f6 \ + --hash=sha256:c6b3228e1d80af737b72925ce5fb4daf5a335e49cd7ab77ed7b9fdfbf58c526e \ + --hash=sha256:c76c4bec1538375dad9d452d246ca5368ad6e1c9039dadcf007ae59c70619ea1 \ + --hash=sha256:c9035dde0f916702850ef66460bc4239d89d08df4d02023a5926e7446724212c \ + --hash=sha256:c93c3db7ea657dd4637d57e74ab73de31bccefe144d3d4ce370052035bc85fb5 \ + --hash=sha256:cb2a55f408c3043e42b40cc8eecd575afa27b7e0b956dfb190de0f8499a57a53 \ + --hash=sha256:cdea2e7b2456cfb6694fb113066fd0ec7ea4d67e3a35e1f4cbeea0b448bf5872 \ + --hash=sha256:ce1bbd7d780bb5a0da032e095c951f7014d6b0a205f8318308140f1a6aba159e \ + --hash=sha256:cf37cbe5ced48d417ba045aca1b21bafca67489452debcde94778a576666a1df \ + --hash=sha256:d4f49cb5661344764e4c7c7973e92a47a59b8fc19b6523649ec9dc4960e58a03 \ + --hash=sha256:d54ecf9f301853f2c5e802da559604b3e95bb7a3b01a9c295c6ee591b9882de8 \ + --hash=sha256:d62b7f64ffde3b99d06b707a280db04fb3855b55f5a06df387236051d0668f4a \ + --hash=sha256:d82dd730a95e6643802f4454b8fdecdf08667881a9c5670db85bc5a56693f122 \ + --hash=sha256:da62917e6076f512daccfbbde27f46fed1c98fee202f0559adec8ee0de67f71a \ + --hash=sha256:dd96c01a9dcd4889dcfcf9eb5544ca0c77603f239e3ffab0524ec17aea9a93ee \ + --hash=sha256:df9f19c28adcb40b6aae30bbaa1478c389efd50c28d541d76760199fc1037c32 \ + --hash=sha256:e1c5988359516095535c4301af38d8a8838534158f649c05dd1050222321bcb3 \ + --hash=sha256:e628ef0e6859ffd8273c69412a2465c4be4a9517d07261b33334b5ec6f3c7489 \ + --hash=sha256:e82d14e3c948952a1a85503817e038cba5905a3352de76b9a465075d072fba23 \ + --hash=sha256:e954b24433c768ce78ab7929e84ccf3422e46deb45a4dc9f93438f8217fa2d34 \ + --hash=sha256:eb0ce7b2a32d09892b3dd6cc44877a0d02a33241fafca5f25c8b6b62374f8b75 \ + --hash=sha256:eb304767bca2bb92fb9c5bd33cedc95baee5bb5f6c88e63706533a1c06ad08c8 \ + --hash=sha256:eb351f72c26dc9abe338ca7294661aa22969ad8ffe7ef7d5541d19f368dc854a \ + --hash=sha256:ec6652a1bee61c53a3e5776b6049172c53b6aaba34f18c9ad04f82712bac623d \ + --hash=sha256:f2a0a924d4c2e9afcd7ec64f9de35fcd96915149b2216e1cb2c10a56df483855 \ + --hash=sha256:f33dc2a3abe9249ea5d8360f969ec7f4142e7ac45ee7014d8f8d5acddf178b7b \ + --hash=sha256:f537b55778cd3cbee430abe3131255d3a78202e0f9ea7ffc6ada893a4bcaeea4 \ + --hash=sha256:f5dd81c45b05518b9aa4da4aa74e1c93d715efa234fd3e8a179df611cc85e5f4 \ + --hash=sha256:f99fe611c312b3c1c0ace793f92464d8cd263cc3b26b5721950d977b006b6c4d \ + --hash=sha256:fa263a02f4f2dd2d11a7b1bb4362aa7cb1049f84a9235d31adf63f30143469a0 \ + --hash=sha256:fc5907494fccf3e7d3f94f95c91d6336b092b5fc83811720fae5e2765890dfba \ + --hash=sha256:fcee94dfbd638784645b066074b338bc9cc155d4b4bffa4adce1615c5a426c19 + # via + # aiohttp + # yarl +propcache==0.5.2 \ + --hash=sha256:01c4fc7480cd0598bb4b57022df55b9ca296da7fc5a8760bd8451a7e63a7d427 \ + --hash=sha256:04dc2390d9edbbaef7461f33322555976ffddf0b650a038649d026358714e6c5 \ + --hash=sha256:06187263ddad280d05b4d8a8b3bb7d164cbebd469236544a42e6d9b28ac6a4fa \ + --hash=sha256:0958834041a0166d343b8d2cedcd8bcbaeb4fdbe0cf08320c5379f143c3be6e7 \ + --hash=sha256:099aaf4b4d1a02265b92a977edf00b5c4f63b3b17ac6de39b0d637c9cac0188a \ + --hash=sha256:0d2c9bf8528f135dbb805ce027567e09164f7efa51a2be07458a2c0420f292d0 \ + --hash=sha256:0fd59b5af35f74da48d905dcbad55449ba13be91823cb05a9bd590bbf5b61660 \ + --hash=sha256:10734b5484ea113152ee25a91dccedf81631791805d2c9ccb054958e51842c94 \ + --hash=sha256:13fef48778b5a2a756523fdb781326b028ca75e32858b04f2cdd19f394564917 \ + --hash=sha256:178b4a2cdaac1818e2bf1c5a99b94383fa73ea5382e032a48dec07dc5668dc42 \ + --hash=sha256:196913dea116aeb5a2ba95af4ddcb7ea85559ae07d8eee8751688310d09168c3 \ + --hash=sha256:1b31822f4474c4036bae62de9402710051d431a606d6a0f907fec79935a071aa \ + --hash=sha256:1ca071adabaab6e9219924bbe00af821f1ee7de113a9eca1cdc292de3d120f4d \ + --hash=sha256:1d1ad32d9d4355e2be65574fd0bfd3677e7066b009cd5b9b2dee8aa6a6393b33 \ + --hash=sha256:1dbcf7675229b35d31abb6547d8ebc8c27a830ac3f9a794edff6254873ec7c0a \ + --hash=sha256:2293949b855ce597f2826452d17c2d545fb5622379c4ea6fdf525e9b8e8a2511 \ + --hash=sha256:26a4dca084132874e639895c3135dfad5eb20bae209f62d1aeb31b03e601c3c0 \ + --hash=sha256:2800a4a8ead6b28cccd1ec54b59346f0def7922ee1c7598e8499c733cfbb7c84 \ + --hash=sha256:29cbaac5ea0212663e6845e04b5e188d5a6ae6dd919810ac835bf1d3b42c3f4c \ + --hash=sha256:29f9309a2e42b0d273be006fdb4be2d6c39a47f6f57d8fb1cf9f81481df81b66 \ + --hash=sha256:2d7aa89ebca5acc98cba9d1472d976e394782f587bad6661003602a619fd1821 \ + --hash=sha256:2f22cbbac9e26a8e864c0985ff1268d5d939d53d9d9411a9824279097e03a2cb \ + --hash=sha256:2f8ea531c794b9d6274acd4e8d2c2ebcac590a4361d27482edd3010b79f1325e \ + --hash=sha256:3115559b8effafd63b142ea5ed53d63a16ea6469cbc63dce4ee194b42db5d853 \ + --hash=sha256:32775082acd2d807ee3db715c7770d38767b817870acfa08c29e057f3c4d5b56 \ + --hash=sha256:3430bb2bfe1331885c427745a751e774ee679fd4344f80b97bf879815fe8fa55 \ + --hash=sha256:3b199b9b2b3d6a7edf3183ba8a9a137a22b97f7df525feb5ae1eccf026d2a9c6 \ + --hash=sha256:40314bca9ac559716fe374094fc81c11dcc34b64fd6c585360f5775690505704 \ + --hash=sha256:44e488ef40dbb452700b2b1f8188934121f6648f52c295055662d2191959ff82 \ + --hash=sha256:452b5065457eb9991ec5eb38ff41d6cd4c991c9ac7c531c4d5849ae473a9a13f \ + --hash=sha256:45f11346f884bc47444f6e6647131055844134c3175b629f84952e2b5cd62b64 \ + --hash=sha256:46088abff4cba581dea21ae0467a480526cb25aa5f3c269e909f800328bc3999 \ + --hash=sha256:4621064bbf28fa77ff64dd5d94367c04684c67d3a5bf1dff25f0cd0d98a38f3b \ + --hash=sha256:4bc8ff1feffc6a61c7002ffe84634c41b822e104990ae009f44a0834430070bb \ + --hash=sha256:4db0ba63d693afd40d249bd93f842b5f144f8fcbb83de05660373bcf30517b1d \ + --hash=sha256:51f96d685ab16e88cab128cd37a52c5da540809c8b879fa047731bfcb4ad35a4 \ + --hash=sha256:54adaa85a22078d1e306304a40984dc5be99d599bf3dc0a24dc98f7daeab89ab \ + --hash=sha256:552ffadf6ad409844bc5919c42a0a83d88314cedddaea0e41e80a8b8fffe881f \ + --hash=sha256:5538d2c13d93e4698af7e092b57bc7298fd35d1d58e656ae18f23ee0d0378e03 \ + --hash=sha256:5570dbcc97571c15f68068e529c92715a12f8d54030e272d264b377e22bd17a5 \ + --hash=sha256:5671d09a36b06d0fd4a3da0fccbcae360e9b1570924171a15e9e0997f0249fba \ + --hash=sha256:583c19759d9eec1e5b69e2fbef36a7d9c326041be9746cb822d335c8cedc2979 \ + --hash=sha256:5aaa2b923c1944ac8febd6609cb373540a5563e7cbcb0fd770f75dace2eb817b \ + --hash=sha256:5dbc581d2814337da56222fab8dc5f161cd798a434e49bac27930aaef798e144 \ + --hash=sha256:5fcb98e7598b1ee0addab320d90f65b530297a867dbfe9de52ea838077e16e3d \ + --hash=sha256:6041d31504dc1779d700e1edcfb08eea334b357620b06681a4eabb57a74e574e \ + --hash=sha256:66ea454f095ddf5b6b14f56c064c0941c4788be11e18d2464cf643bf7203ff67 \ + --hash=sha256:68ce1c44c7a813a7f71ea04315a8c7b330b63db99d059a797a4651bb6f69f117 \ + --hash=sha256:6a997d0489e9668a384fcfd5061b857aa5361de73191cac204d04b889cfbbafa \ + --hash=sha256:6bf3be92233808fcd338eba0fb4d0b59ec5772af4f4ecfcec450d1bfc0f8b5eb \ + --hash=sha256:6de8bd93ddde9b992cf2b2e0d796d501a19026b5b9fd87356d7d0779531a8d96 \ + --hash=sha256:6e7b8719005dd1175be4ab1cd25e9b98659a5e0347331506ec6760d2773a7fb5 \ + --hash=sha256:6f328175a2cde1f0ff2c4ed8ce968b9dcfb55f3a7153f39e2957ed994da13476 \ + --hash=sha256:72d61e16dd78228b58c5d47be830ff3da7e5f139abdf0aef9d86cde1c5cf2191 \ + --hash=sha256:74b70780220e2dd89175ca24b81b68b67c83db499ae611e7f2313cb329801c78 \ + --hash=sha256:79aa3ff0a9b566633b642fa9caf7e21ed1c13d6feca718187873f199e1514078 \ + --hash=sha256:7afa37062e6650640e932e4cc9297d81f9f42d9944029cc386b8247dea4da837 \ + --hash=sha256:80168e2ebe4d3ec6599d10ad8f520304ae1cad9b6c5a95372aef1b66b7bfb53a \ + --hash=sha256:806719138ecd720339a12410fb9614ac9b2b2d3a5fdf8235d56981c36f4039ba \ + --hash=sha256:8114f28879e0904748e831c3a7774261bd9e75f49be089f389a76f959dcd13fe \ + --hash=sha256:81e3a30b0bb60caa22033dd0f8a3618d1d67356212514f62c57db75cb0ef410c \ + --hash=sha256:823581fd5cb08b12a48bfa11fe962a7916766b6170c17b028fbdf762b85eb9bf \ + --hash=sha256:85341b12b9d55bad0bded24cac341bb34289469e03a11f3f583ea1cc1db0326c \ + --hash=sha256:857187f381f88c8e2fa2fe56ab94879d011b883d5a2ee5a1b60a8cd2a06846d9 \ + --hash=sha256:8a90efd5777e996e42d568db9ac740b944d691e565cbfd31b2f7832f9184b2b8 \ + --hash=sha256:8b73ab70f1a3351fbc71f663b3e645af6dd0329100c353081cf69c37433fc6fe \ + --hash=sha256:8c7972d8f193740d9175f0998ab38717e6cd322d5935c5b0fef8c0d323fd9031 \ + --hash=sha256:8e778ebd44ef4f66ed60a0416b06b489687db264a9c0b3620362f26489492913 \ + --hash=sha256:9282fb1a3bccd038da9f768b927b24a0c753e466c086b7c4f3c6982851eefb2d \ + --hash=sha256:949c91d1a990cf3b2e8188dfcfb25005e0b834a06c63fa4ef9f360878ce21ecf \ + --hash=sha256:95f1e3f4760d404b13c9976c0229b2b49a3c8e2c62a9ce92efdd2b11ada75e3f \ + --hash=sha256:97797ebb098e670a2f92dd66f32897e30d7615b14e7f59711de23e30a9072539 \ + --hash=sha256:a0e399a2eccb91ed18721f86aa85757727400b6865c89e88934781deb9c8498b \ + --hash=sha256:a473b3440261e0c60706e732b2ed2f517857344fc21bf48fdfe211e2d98eb285 \ + --hash=sha256:a4840ab0ae0216d952f4b53dc6d0b992bfc2bedbfe360bdd9b548bc184c08959 \ + --hash=sha256:a592f5f3da71c8691c788c13cb6734b6d17663d2e1cb8caddf0673d01ef8847d \ + --hash=sha256:a6ae2198be502c10f09b2516e7b5d019816924bc3183a43ce792a7bd6625e6f4 \ + --hash=sha256:a6ddc6ac9e25de626c1f129c1b467d7ecd33ce2237d3fd0c4e429feef0a7ee1f \ + --hash=sha256:acd2c8edba48e31e58a363b8cf4e5c7db3b04b3f9e371f601df30d9b0d244836 \ + --hash=sha256:b05d643f944a8c3c4bd86d65ffd87bf3264b617f87791940302bc474d2ff5274 \ + --hash=sha256:b96db7141a592cbc968daf1feea83a118e6ab378af4abbc72b248c895414c22d \ + --hash=sha256:ba338430e87ceb9c8f0cf754de38a9860560261e56c00376debd628698a7364f \ + --hash=sha256:ba57fffe4ac99c5d30076161b5866336d97600769bad35cc68f7774b15298a4e \ + --hash=sha256:be1ddfcbb376e3de5d2e2db1d58d6d67463e6b4f9f040c000de8e300295465fe \ + --hash=sha256:c0cb9ed24c8964e172768d455a38254c2dd8a552905729ce006cad3d3dda59b1 \ + --hash=sha256:c60462af8e6dc30c35407c7237ea908d777b22862bbee27bc4699c0d8bcdc45a \ + --hash=sha256:c66afea89b1e43725731d2004732a046fe6fe955d51f952c3e95a7314a284a39 \ + --hash=sha256:c6844ba6364fb12f403928a82cfd295ab103a2b315c77c747b2dbe4a41894ea7 \ + --hash=sha256:c80f4ba3e8f00189165999a742ee526ebeccedf6c3f7beb0c7df821e9772435a \ + --hash=sha256:cafca7e56c12bb02ae16d283742bef25a61122e9dab2b5b3f2ccbe589ce32164 \ + --hash=sha256:cc1177027eda740fdb152706bd215a3f124e3eea15afc39f2cb9fe351b50619e \ + --hash=sha256:cc49723e2f60d6b32a0f0b08a3fd6d13203c07f1cd9566cfce0f12a917c967a2 \ + --hash=sha256:cc6fc3cc62e8501d3ed62894425040d2728ecddb1ed072737a5c70bd537aa9f0 \ + --hash=sha256:cd416c1de191973c52ff1a12a57446bfc7642797b282d7caf2162d7d1b8aa9a0 \ + --hash=sha256:cd645f03898405cabe694fb8bc35241e3a9c332ec85627584fe3de201452b335 \ + --hash=sha256:cef6cea3922890dd6c9654971001fa797b526c16ab5e1e46c05fd6f877be7568 \ + --hash=sha256:cfa21e036ce1e1db2be04ba3b85d2df1bb1702fa01932d984c5464c665228ff4 \ + --hash=sha256:d0326e2e5e1f3163fa306c834e48e8d490e5fae607a097a40c0648109b47ba80 \ + --hash=sha256:d310c013aad2c72f1c3f2f8dd3279d460a858c551f97aeb8c63e4693cca7b4d2 \ + --hash=sha256:d447bb0b3054be5818458fbb171208b1d9ff11eba14e18ca18b90cbb45767370 \ + --hash=sha256:d4dc37dec6c6cdad0b57881a5658fd14fbf53e333b1a86cf86559f190e1d9ec4 \ + --hash=sha256:d5a81be28596d6559f6131ef33e10200de6e17643b3c74ce03f9eb103be6ae8b \ + --hash=sha256:d9ee8826a7d47863a08ac44e1a5f611a462eefc3a194b492da242128bec75b42 \ + --hash=sha256:db2b80ea58eab4f86b2beec3cc8b39e8ff9276ac20e96b7cce43c8ae84cd6b5a \ + --hash=sha256:decfca4c79dd53ebab484b00cc4b6717d8c369f86e74aa4ca395a64ac651495e \ + --hash=sha256:dfed59d0a5aeb01e242e66ff0300bc4a265a7c05f612d30016f0b60b1017d757 \ + --hash=sha256:e00820e192c8dbebcafb383ebbf99030895f09905e7a0eb2e0340a0bcc2bc825 \ + --hash=sha256:e4294d04a94dcab1b3bccd8b66d962dcad411a1d19414b2a41d1445f1de32ad0 \ + --hash=sha256:e59bc9e66329185b93dab73f210f1a37f81cb40f321501db8017c9aea15dba27 \ + --hash=sha256:e5cbfac9f61484f7e9f3597775500cd3ebe8274e9b050c38f9525c77c97520bf \ + --hash=sha256:f064f8d2b59177878b7615df1735cd8fe3462ed6be8c7b217d17a276489c2b7f \ + --hash=sha256:f156a3529f38063b6dbaf356e15602a7f95f8055b1295a438433a6386f10463d \ + --hash=sha256:f19bb891234d72535764d703bfed1153cc34f4214d5bd7150aee1eec9e8f4366 \ + --hash=sha256:f7467da8a9822bf1a55336f877340c5bcbd3c482afc43a99771169f74a26dedc \ + --hash=sha256:f78abfa8dfc32376fd1aacf597b2f2fbbe0ea751419aee718af5d4f82537ef8c \ + --hash=sha256:f7eabc04151c78a9f4d5bbb5f1faf571e4defeb4b585e0fe95b60ff2dbe4d3d7 \ + --hash=sha256:f814362777a9f841adddb200ecdf8f5cb1e5a3c4b7a86378edbd6ccb26edd702 \ + --hash=sha256:fc299c129490f55f254cd90be0deca4764e36e9a7c08b4aa588479a3bbed3098 \ + --hash=sha256:fc76378c62a0f04d0cd82fbb1a2cd2d7e28fcb40d5873f28a6c44e388aaa2751 \ + --hash=sha256:fc88b26f08d634f7bc819a7852e5214f5802641ab8d9fd5326892292eee1993e \ + --hash=sha256:fe67a3d11cd9b4efabfa45c3d00ffba2b26811442a73a581a94b67c2b5faccf6 + # via + # aiohttp + # yarl +typing-extensions==4.16.0 ; python_full_version < '3.13' \ + --hash=sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8 \ + --hash=sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5 + # via + # aiohttp + # aiosignal +websockets==17.0.1 \ + --hash=sha256:02ed63bf26dda9fa27df730a41f6664586c4ee05972c8fb667ce1725b3fd13d3 \ + --hash=sha256:02f0b037a737d0cb0c33866c97bcd1a0b73170dfbf42d69d8fb86f51002fd5ae \ + --hash=sha256:038cfad5d5417f8bb09295abe986029a26d22f34bda622ccc79b670efd4dab56 \ + --hash=sha256:07abc3bd196a48af476a82fd47f3f79a6a3f70937a9f930cef703cfa0c9d83b6 \ + --hash=sha256:07d78a509c3333f5908c83d7f78144ea68a6c9ec28110f5c54d81d8fcdc262c4 \ + --hash=sha256:0b52c76b8a870b141b7ca0705289452183ce7a523101954ccfe29a25986a673f \ + --hash=sha256:10b1587c599fa0f2c89154587c80e0fda98ade6c9fa8c0260a2823fb1800b685 \ + --hash=sha256:10f461191125c63902ea7394ae9e752b1b5785641850c1d365bb30b0f88bc53f \ + --hash=sha256:15920057a6b723f84734f0641403bca163a4b176e5af809ee4f0c4a1e75e9fed \ + --hash=sha256:17ac37716c0244e82c9e384c41653c090b1864c6610224ca3857e7f7b58fce10 \ + --hash=sha256:18ded646ce98cdd3c0235825b3252f1df55765ba49b616bb10282f758667b4d0 \ + --hash=sha256:1b363bfd72a52c0658a3154a4cff219f15a474b35a235057d38853bf151acce7 \ + --hash=sha256:1bdd8c4be420905dd732e00dcd669852d8128cc723efa585a0c0e51adb00a28a \ + --hash=sha256:1d4cf7e8e5b8b1fa40758ac7524843a00237b124ab217e227542cafcfeb7a946 \ + --hash=sha256:1df81d174c1561292de9e40b141cafc04f69077272f6c352afe1d743e20810df \ + --hash=sha256:20a92f78ac8250984ed459faa9ca48c285adbfc0038ddc3fdac6046990a9c9ed \ + --hash=sha256:22bd00f8bae2bccdb5dbe41e20f58ba44ca9fff0b4b561aaf39099c35da762ed \ + --hash=sha256:2437d4ca208cc0f246d3a2297ae7474b4ba18261aaf5b9c79c84c031ecf348e1 \ + --hash=sha256:246927ae9ae06ca0d42a483a4bdb80d4862e1ee5b4cab37c354a5e1ad8356448 \ + --hash=sha256:2503c7e2a5049a12d5dac917a46d5d52591283a766165b8176bb167560421b38 \ + --hash=sha256:2604de7228506b13a44a256a9d223943340c0e725af5d367dc068e192b027761 \ + --hash=sha256:28012a54510fe8301bb893ef143cec30a2780a2d3bc20b7bbdf4379d7a63945d \ + --hash=sha256:2a855b6dfe21c4d3420be265ae031829ba8ba0be0ea350d9f7c3ef30ae63ebe2 \ + --hash=sha256:2abb1ba0a5133b7d2ef3c1c9f4b0c1e8a101012dce0b594ab2b2888d9a64820e \ + --hash=sha256:2b3f3020171202b135ca078e20434977c6b2b02af647130d6980c9e39b9462e3 \ + --hash=sha256:2bc14b481e05e331811108daa1aeb41a5e237a5564ef2f02ec5a356a0f102f78 \ + --hash=sha256:2fa2cb465a131c347ba6717a78c887746e73edb1c131d01c982d6ef0d68b82e0 \ + --hash=sha256:409d93efcaa14f7a99592c5baaef5ec6ca94fba0f5aec1a86f693977c69c9c1c \ + --hash=sha256:41d6aa06b5ab832aee72fedf47a149535b121ac900b6bb4d3fe14712afac9a79 \ + --hash=sha256:49266e4488309b38783257293a38298942b9a03aa106fcb45195377a77c0c1e2 \ + --hash=sha256:4d1d99db29b5444e3982f1ce2ba8a833508ad44b2f1fbd0bd99e81d825c0b461 \ + --hash=sha256:4d41c0a1d47a478bc432b3b9068097bee1ce0c5b19327ea6f75c2ab34ab1f2fb \ + --hash=sha256:5033ffe6804dd53afafa7d08e8c3eef2d2431f34d58ca30507a8442dd04a033a \ + --hash=sha256:53b90c00bc6201ab6695c7ff51a04d0e425514c37515e9eeecd2c1b978ac6c0e \ + --hash=sha256:54cdcaa56f5d3eafd57058f0fa4a3de93a310b43a3c4699f06efc4c0bd054a5a \ + --hash=sha256:5508f38c98ac29def9e747b87543b008a58b075df6da70b2cf2e0b47073d33bb \ + --hash=sha256:55383d8177b3c99fd873ee5db0e0193f4c1dd4a3feaccf1a4a03c1b7cf539cac \ + --hash=sha256:55b12e47dcee83673a40d07686cfb6f9d6dfc285976ade9463f61d2bef3fad22 \ + --hash=sha256:5661f868ef191d33dfc6a0cc7c5b3d495f0cc8bb3f8b30d87bda8755c61c95f5 \ + --hash=sha256:57d2ee9b24b404ce75f3814f92073c0ed88106c950148d2427fe8d25ca254d1f \ + --hash=sha256:599b03beb77633bffc095334338fad79cafc2b01fbd58953838130a9ae967d7b \ + --hash=sha256:5baa9bc0dfbae8c507e51c8cf1b6d4628086f7a87bbd3a9952bd5f035451f1cc \ + --hash=sha256:5f33a649bfcb8312524173cc4bbafa7dbb236e18eee9aa31a1d324ca0ddda28c \ + --hash=sha256:6740be6d1bab69f08ab52cb15b08f76c143b6fe61c580ba62bd929f3ab7a1d42 \ + --hash=sha256:6a434e59962a4fb9016bea327e1d14d6cd67670ecfb8942b4f4a0c24036634ce \ + --hash=sha256:6db9e5bf3649ab506c6ae8a3ac85a00fb1ae3816d75962771b2df8adbc5d40d2 \ + --hash=sha256:6fd88365da261c53d3e943fb37e0d0721b9cde119f6b2e3fc84369b6ab234d63 \ + --hash=sha256:7002d5f9e1c3ddd991cdfdbfee18cc8c8b196b2445022892badacd6cb338bbbc \ + --hash=sha256:70d438268e49f1a4bd096b6b6f7010f3ab48b5db2574dbf7d8c864c46ce7a06a \ + --hash=sha256:72d7f2a5aeb4e82daa4ee18f125b4277f427033359be5c745ad709608446cc2c \ + --hash=sha256:733e3cc7171fa1b899edbe725ef9382d0e960657dc1fd933f3281ae910c01dab \ + --hash=sha256:734d20364dc2cfe03674883cafcf580b6e431c5ce42b476312b9285310230cf9 \ + --hash=sha256:759adeb5b0c5775b563254ec63b5b79089fc0045b479143a0b1b8c0ebaae1253 \ + --hash=sha256:769ce7e2acfd9a89f2bed3a9c0da229459516bbc00bd4c9e2ca492c613ae4861 \ + --hash=sha256:810cb3fb5fa6e447216f4e82d9a85cb8aed0929ae3538153ddfe8a6e3121a58d \ + --hash=sha256:81ce19c6046ace11da7001781be7317bb1dc389f399af4b2ed962190f76f9add \ + --hash=sha256:846a4a8b0833e3cad57523d9e3bd50ec8ea05ab9d06c582f82a1340ba096af5f \ + --hash=sha256:872273e629ca7e3d35f16a2dc6ede84e1d5c831e616b8277de6e4f83114e7c58 \ + --hash=sha256:8848c207049ad49d318e5f64a3d4d7bb189f8328d0d98e65647788f2a085785c \ + --hash=sha256:884af729b8ab50486acd94d9768c2b60914bf39b579ebba0a5cb73bfdfd61fd2 \ + --hash=sha256:8c07f145d0b9e90cbd96035f31fb79199aef4da1872854e36ebeb258e3d57594 \ + --hash=sha256:8cd3369e42c0246afaf9d669cfc19797e3a49e8c0a639544459c57597108b966 \ + --hash=sha256:8e387adb0c692c6b5571bdeafc8ac9d1901ea30f10309134780b16ecd35e6605 \ + --hash=sha256:90246fa9e6cb192a778ce6ce024057ec54317a894db7899c922dcdc1f4cbf6a5 \ + --hash=sha256:90973a3a00f23afdfd1c9b06fb84289bf0220f247ef8a62501a1967c7af54f7b \ + --hash=sha256:9493314a99e599163c854fb5900ad7f7ea38c5cb9d9103aa30b3c6b8181c01fa \ + --hash=sha256:9f7747d3daa41a11f25f7cca5dc988fc51da97b311bed4c9d843860f79779283 \ + --hash=sha256:a39ce3a7b0e6059be093213d637963101380157bcbad355916738fafb490698d \ + --hash=sha256:a60fa1a25cca1bcc2bf87b8d6be37a741f0a3239fb5e9cfb7a37173b68ffcf87 \ + --hash=sha256:a68e604c6d1b0338e46652e2688cbce8096ad9c03548b075fda9e2ea19a9b7dd \ + --hash=sha256:a8af570fc29cd998a921c7131c8ac81d9434466d6d25300cb12a690fb56a8a08 \ + --hash=sha256:aadc298969ad229d8e3029fc5cc751fdad286696230f9cf014e90ff9cd8e6ea0 \ + --hash=sha256:ab56439c9f74c52770690c7b2f616b3bf775cb3920453ee355ac765c032d8bbf \ + --hash=sha256:ab9f962a5b64a5c3c845d556b7dc4e6fb683f7b67179f8205e814bb2e0213ffe \ + --hash=sha256:afbce6e3f0fac32dc87c2a0d84869d1a706460d64f39f3889386413e6e4d3d26 \ + --hash=sha256:b3ff0ad440ad52dda64138f16895f66403f40192365e39b1010e889f289746b0 \ + --hash=sha256:b580794e926cab7ff42ee4371ef14e0b22cb2bb722a607f77769136468f49a3f \ + --hash=sha256:b85b960a4507b0714c0a1246d031be9118d908ee974dc085257297a955205f1d \ + --hash=sha256:b98860aefbd3d9bc8e3c7f0eefb83b11142b16110739c68cd33d3b4d6e84e536 \ + --hash=sha256:bb31f42ea095ea826463c770829aa188a86c9a5c976b1467cbbf583c811de833 \ + --hash=sha256:bc0bca48ba24c6c866847fd20478a51dd547fa0ad258dab9615c414ec534bbc0 \ + --hash=sha256:bd1470d2c53fe53269bf5619da7725d30dd9b9693f1689f7a85eab8dea734442 \ + --hash=sha256:c09e097d0e46e3c289bedab9a475ae344b70c30ff5646e46af22b4e6fdc97b21 \ + --hash=sha256:c1bec5d6a19f5fbe87e4940739cfc65e7bb53d8b353e1029b8037a1653b321bc \ + --hash=sha256:c1c118a6b0e25bfc9a6802075d748fa6321714ffbdf3c88d29d9a0e3c7386c75 \ + --hash=sha256:c23e532c8a2325a1e7486de8763a60dc43e83f01bcaeca07e3ba79652c156db1 \ + --hash=sha256:c356dbddab0a529ed7574f78f559d75a223735c321c28f6f587fbf02b11ed301 \ + --hash=sha256:c38515cb54902f7e97d0239e81ef46c4444f9475f4807fb9bbdb789b4089abcf \ + --hash=sha256:c395bda8e7d8f51a02e80261fb57127979e5c472675d9a96b2860619ad47da48 \ + --hash=sha256:c6be9cba65c65cc76dfa3d4619e359ff02a4476c74e179b215236c11a0b32345 \ + --hash=sha256:cd526c8228e759c1006c4b7c9ac71dc4e925ced1a6a6a5a8e94643709738f63e \ + --hash=sha256:cddc675ec31bca65473321f9a9794e488b43b3b8de5d02c8ef4810c5d5792163 \ + --hash=sha256:cffc84ddec6da7f447677266fee2a3c40ecc78172f00752aa1150b8a8d65df1d \ + --hash=sha256:d41e9845514754a42d1d83b2fca9d27fee2ca7b3b0bee6843ba5a9bb2b6e25ac \ + --hash=sha256:d69fd559f9f0e8a52d2fce6f04ee143f86e70df0a189cd95164eddac599e810f \ + --hash=sha256:d7d72843691f50b91127c50688df10cb72ec6f4c4b1d7e2c11ab33b16acf8e51 \ + --hash=sha256:d9aac6081513f02eac3f8caace800dbfc5c608b69e4a7bef69e414eabfc95aa1 \ + --hash=sha256:dbfae8e75b342e31fc6fd1a8bbb393b7cbb91d6cfd581650300a94381e7b7e2b \ + --hash=sha256:e8208f2729cba030ff872a92064c97584eeb9502f53d32a05a0f05d5a17ca6c6 \ + --hash=sha256:e95e321d0d763f2b6633512605f6112ebd70d5746f3ce05c941909d4a25233f2 \ + --hash=sha256:e98ec9ec61cce5bc4b8b218322ad090b0994eb060bb04da704c62ef0a3d864e6 \ + --hash=sha256:eab6de8a98b9a7772cf686d00b4de439fc7efb8ab05ae106ef227291d06f87c5 \ + --hash=sha256:efe0ae052a8d023b87198921e8a7ce1dc7768816bcd2fbc20df171ac73a04891 \ + --hash=sha256:f11a398d8170b7ac5000baf7f258dcda579ef3ea744e0cc6a165e0dfbc0d3198 \ + --hash=sha256:f3fd9a1f87f8f0f3f8e9f9bd0195f7516562d13f5b178db8c5784d1f60b60bed \ + --hash=sha256:f47b0815af3948ec6a440b3afa02f05b18cc0939549e91b5c677b5d9c2c8472a \ + --hash=sha256:f991247276797d0c61ab7770bc9791eadc16f683b4d83517f624932adc1a8bab \ + --hash=sha256:ffad64ce7ad3703d652a3fd9af26238377d24ce52c6ad8ff35d26d82f61f493f + # via -r requirements.in +yarl==1.24.5 \ + --hash=sha256:0055afc45e864b92729ac7600e2d102c17bef060647e74bca75fa84d66b9ff36 \ + --hash=sha256:0465ec8cedc2349b97a6b595ace64084a50c6e839eca40aa0626f38b8350e331 \ + --hash=sha256:0ebfaffe1a16cb72141c8e09f18cc76856dbe58639f393a4f2b26e474b96b871 \ + --hash=sha256:16a2f5010280020e90f5330257e6944bc33e73593b136cc5a241e6c1dc292498 \ + --hash=sha256:17f57620f5475b3c69109376cc87e42a7af5db13c9398e4292772a706ff10780 \ + --hash=sha256:2120b96872df4a117cde97d270bac96aea7cc52205d305cf4611df694a487027 \ + --hash=sha256:240cbec09667c1fed4c6cd0060b9ec57332427d7441289a2ed8875dc9fb2b224 \ + --hash=sha256:24e861e9630e0daddcb9191fb187f60f034e17a4426f8101279f0c475cd74144 \ + --hash=sha256:2729fcfc4f6a596fb0c50f32090400aa9367774ac296a00387e65098c0befa76 \ + --hash=sha256:2c1fe720934a16ea8e7146175cba2126f87f54912c8c5435e7f7c7a51ef808d3 \ + --hash=sha256:2cabe6546e41dabe439999a23fcb5246e0c3b595b4315b96ef755252be90caeb \ + --hash=sha256:2dbe06fc16bc91502bca713704022182e5729861ae00277c3a23354b40929740 \ + --hash=sha256:3363fcc96e665878946ad7a106b9a13eac0541766a690ef287c0232ac768b6ec \ + --hash=sha256:377fe3732edbaf78ee74efdf2c9f49f6e99f20e7f9d2649fda3eb4badd77d76e \ + --hash=sha256:3ac6aff147deb9c09461b2d4bbdf6256831198f5d8a23f5d37138213090b6d8a \ + --hash=sha256:3f45789ce415a7ec0820dc4f82925f9b5f7732070be1dec1f5f23ec381435a24 \ + --hash=sha256:4103b77b8a8225e413107d2349b65eb3c1c52627b5cc5c3c4c1c6a798b218950 \ + --hash=sha256:4377407001ca3c057773f44d8ddd6358fa5f691407c1ba92210bd3cf8d9e4c95 \ + --hash=sha256:46c2f213e23a04b93a392942d782eb9e413e6ef6bf7c8c53884e599a5c174dcb \ + --hash=sha256:47e98aab9d8d82ff682e7b0b5dded33bf138a32b817fcf7fa3b27b2d7c412928 \ + --hash=sha256:4a36f9becdd4c5c52a20c3e9484128b070b1dcfc8944c006f3a528295a359a9c \ + --hash=sha256:4af7b7e1be0a69bee8210735fe6dcfc38879adfac6d62e789d53ba432d1ffa41 \ + --hash=sha256:4d97a951a81039050e45f04e96689b58b8243fa5e62aa14fe67cb6075300885e \ + --hash=sha256:4db9aecb141cb7a5447171b57aa1ed3a8fee06af40b992ffc31206c0b0121550 \ + --hash=sha256:53e549287ef628fecba270045c9701b0c564563a9b0577d24a4ec75b8ab8040f \ + --hash=sha256:56b149b22de33b23b0c6077ab9518c6dcb538ad462e1830e68d06591ccf6e38b \ + --hash=sha256:570fec8fbd22b032733625f03f10b7ff023bc399213db15e72a7acaef28c2f4e \ + --hash=sha256:5b8ee53be440a0cffc991a27be3057e0530122548dbe7c0892df08822fce5ede \ + --hash=sha256:5ba4f78df2bcc19f764a4b26a8a4f5049c110090ad5825993aacb052bf8003ad \ + --hash=sha256:5c55256dee8f4b27bfbf636c8363383c7c8db7890c7cba5217d7bd5f5f21dab6 \ + --hash=sha256:5c88e5815a49d289e599f3513aa7fde0bc2092ff188f99c940f007f90f53d104 \ + --hash=sha256:5fede79c6f73ff2c3ef822864cb1ada23196e62756df53bc6231d351a49516a2 \ + --hash=sha256:65be18ec59496c13908f02a2472751d9ef840b4f3fb5726f129306bf6a2a7bba \ + --hash=sha256:66410eb6345d467151934b49bfa70fb32f5b35a6140baa40ad97d6436abea2e9 \ + --hash=sha256:665b0a2c463cc9423dd647e0bfd9f4ccc9b50f768c55304d5e9f80b177c1de12 \ + --hash=sha256:6b8536851f9f65e7f00c7a1d49ba7f2be0ffe2c11555367fc9f50d9f842410a1 \ + --hash=sha256:6c95b17fe34ed802f17e205112e6e10db92275c34fee290aa9bdc55a9c724027 \ + --hash=sha256:6e73e7fe93f17a7b191f52ec9da9dd8c06a8fe735a1ecbd13b97d1c723bff385 \ + --hash=sha256:6efbccc3d7f75d5b03105172a8dc86d82ba4da86817952529dd93185f4a88be2 \ + --hash=sha256:709f1efed56c4a145793c046cd4939f9959bcd818979a787b77d8e09c57a0840 \ + --hash=sha256:79af890482fc94648e8cde4c68620378f7fef60932710fa17a66abc039244da2 \ + --hash=sha256:7bcbe0fcf850eae67b6b01749815a4f7161c560a844c769ad7b48fcd99f791c4 \ + --hash=sha256:7c0494a31a1ac5461a226e7947a9c9b78c44e1dc7185164fa7e9651557a5d9bc \ + --hash=sha256:7ce27823052e2013b597e0c738b13e7e36b8ccb9400df8959417b052ab0fd92c \ + --hash=sha256:7f72c74aa99359e27a2ee8d6613fefa28b5f76a983c083074dfc2aaa4ab46213 \ + --hash=sha256:7fa5e51397466ea7e98de493fa2ff1b8193cfef8a7b0f9b4842f92d342df0dba \ + --hash=sha256:82632daed195dcc8ea664e8556dc9bdbd671960fb3776bd92806ce05792c2448 \ + --hash=sha256:82f75e05912e84b7a0fe57075d9c59de3cb352b928330f2eb69b2e1f54c3e1f0 \ + --hash=sha256:841f0852f48fefea3b12c9dfec00704dfa3aef5215d0e3ce564bb3d7cd8d57c6 \ + --hash=sha256:874019bd513008b009f58657134e5d0c5e030b3559bd0553976837adf52fe966 \ + --hash=sha256:88f50c94e21a0a7f14042c015b0eba1881af78562e7bf007e0033e624da59750 \ + --hash=sha256:89a1bbb58e0e3f7a283653d854b1e95d65e5cfd4af224dac5f02629ec1a3e621 \ + --hash=sha256:8a6987eaad834cb32dd57d9d582225f0054a5d1af706ccfbbdba735af4927e13 \ + --hash=sha256:8ac73abdc7ab75610f95a8fd994c6457e87752b02a63987e188f937a1fc180f0 \ + --hash=sha256:8ccf9aca873b767977c73df497a85dbedee4ee086ae9ae49dc461333b9b79f58 \ + --hash=sha256:90333fd89b43c0d08ac85f3f1447593fc2c66de18c3d6378d7125ea118dc7a54 \ + --hash=sha256:92ab3e11448f2ff7bf53c5a26eff0edc086898ec8b21fb154b85839ce1d88075 \ + --hash=sha256:9335a099ad87287c37fe5d1a982ff392fa5efe5d14b40a730b1ec1d6a41382b4 \ + --hash=sha256:96d30286dd02679e32a39aa8f0b7498fc847fcda46cfc09df5513e82ce252440 \ + --hash=sha256:9baafc71b04f8f4bb0703b21d6fc9f0c30b346c636a532ff16ec8491a5ea4b1f \ + --hash=sha256:9d1216a7f6f77836617dba35687c5b78a4170afc3c3f18fc788f785ba26565c4 \ + --hash=sha256:9d399bdcfb4a0f659b9b3788bbc89babe63d9a6a65aacdf4d4e7065ff2e6316c \ + --hash=sha256:9e4e16c73d717c5cf27626c524d0a2e261ad20e46932b2670f64ad5dde23e26f \ + --hash=sha256:9f4d8cf085a4c6a40fb97ea0f46938a8df43c85d31f9d45e2a8867ea9293790d \ + --hash=sha256:a33700d13d9b7d84fd10947b09ff69fb9a792e519c8cb9764a3ca70baa6c23a7 \ + --hash=sha256:a3732e66413163e72508da9eff9ce9d2846fde51fae45d3605393d3e6cd303e9 \ + --hash=sha256:a4582acf7ef76482f6f511ebaf1946dae7f2e85ec4728b81a678c01df63bd723 \ + --hash=sha256:a61834fb15d81322d872eaafd333838ae7c9cea84067f232656f75965933d047 \ + --hash=sha256:a7cff474ab7cd149765bb784cf6d78b32e18e20473fb7bda860bce98ab58e9da \ + --hash=sha256:a8fe66b8f300da93798025a785a5b90b42f3810dc2b72283ff84a41aaaebc293 \ + --hash=sha256:a929d878fec099030c292803b31e5d5540a7b6a31e6a3cc76cb4685fc2a2f51b \ + --hash=sha256:ad5d8201d310b031e6cd839d9bac2d4e5a01533ce5d3d5b50b7de1ef3af1de61 \ + --hash=sha256:af3aefa655adb5869491fa907e652290386800ae99cc50095cba71e2c6aefdca \ + --hash=sha256:c0ebc836c47a6477e182169c6a476fc691d12b518894bf7dd2572f0d59f1c7ed \ + --hash=sha256:c687ed078e145f5fd53a14854beff320e1d2ab76df03e2009c98f39a0f68f39a \ + --hash=sha256:cbb833ccacdb5519eff9b8b71ee618cc2801c878e77e288775d77c3a2ced858a \ + --hash=sha256:cf139c02f5f23ef6532040a30ff662c00a318c952334f211046b8e60b7f17688 \ + --hash=sha256:d46b86567dd4e248c6c159fcbcdcce01e0a5c8a7cd2334a0fff759d0fa075b16 \ + --hash=sha256:d693396e5aea78db03decd60aec9ece16c9b40ba00a587f089615ff4e718a81d \ + --hash=sha256:d897129df1a22b12aeed2c2c98df0785a2e8e6e0bde87b389491d0025c187077 \ + --hash=sha256:daba5e594f06114e37db186efd2dd916609071e59daca901a0a2e71f02b142ce \ + --hash=sha256:dd625535328fd9882374356269227670189adfcc6a2d90284f323c05862eecbd \ + --hash=sha256:e006d3a974c4ee19512e5f058abedb6eef36a5e553c14812bdeba1758d812e6d \ + --hash=sha256:e1ae548a9d901adca07899a4147a7c826bbcc06239d3ce9a59f57886a28a4c88 \ + --hash=sha256:e2935f8c39e3b03e83519292d78f075189978f3f4adc15a78144c7c8e2a1cba5 \ + --hash=sha256:e42d75862735da90e7fc5a7b23db0c976f737113a54b3c9777a9b665e9cbff75 \ + --hash=sha256:e7d42c531243450ef0d4d9c172e7ed6ef052640f195629065041b5add4e058d1 \ + --hash=sha256:e81b83143bee16329c23db3c1b2d82b29892fcbcb849186d2f6e98a5abe9a57f \ + --hash=sha256:e8ffa78582120024f476a611d7befc123cee59e47e8309d470cf667d806e613b \ + --hash=sha256:ebb0ec7f17803063d5aeb982f3b1bd2b2f4e4fae6751226cbd6ba1fcfe9e63ff \ + --hash=sha256:f08c7513ecef5aad65687bfdf6bc601ae9fccd04a42904501f8f7141abad9eb9 \ + --hash=sha256:f0a658a6d3fafee5c6f63c58f3e785c8c43c93fbc02bf9f2b6663f8185e0971f \ + --hash=sha256:f0e466ed7511fe9d459a819edbc6c2585c0b6eabde9fa8a8947552468a7a6ef0 \ + --hash=sha256:f141474e85b7e54998ec5180530a7cda99ab29e282fa50e0756d89981a9b43c5 \ + --hash=sha256:f4239bbec5a3577ddb49e4b50aeb32d8e5792098262ae2f63723f916a29b1a25 \ + --hash=sha256:f540c013589084679a6c7fac07096b10159737918174f5dfc5e11bf5bca4dfe6 \ + --hash=sha256:f9f3e9c8a9ecffa57bef8fb4fa19e5fa4d2d8307cf6bac5b1fca5e5860f4ba00 \ + --hash=sha256:fa139875ff98ab97da323cfadfaff08900d1ad42f1b5087b0b812a55c5a06373 \ + --hash=sha256:fcd3b77e2f17bbe4ca56ec7bcb07992647d19d0b9c05d84886dcd6f9eb810afd \ + --hash=sha256:fd8c81f346b58f45818d09ea11db69a8d5fd34a224b79871f6d44f12cd7977b1 \ + --hash=sha256:fe7b7bb170daccbba19ad33012d2b15f1e7942296fd4d45fc1b79013da8cc0f2 \ + --hash=sha256:ff330d3c30db4eb6b01d79e29d2d0b407a7ecad39cfd9ec993ece57396a2ec0d \ + --hash=sha256:ff405d91509d88e8d44129cd87b18d70acd1f0c1aeabd7bc3c46792b1fe2acba \ + --hash=sha256:ffcd54362564dc1a30fb74d8b8a6e5a6b11ebd5e27266adc3b7427a21a6c9104 + # via aiohttp diff --git a/docker/telemetry/workload/rpc_load_generator.py b/docker/telemetry/workload/rpc_load_generator.py index 7b5e22f631..26962420f4 100644 --- a/docker/telemetry/workload/rpc_load_generator.py +++ b/docker/telemetry/workload/rpc_load_generator.py @@ -30,6 +30,7 @@ import argparse import asyncio import json import logging +import math import random import sys import time @@ -39,6 +40,11 @@ from typing import Any import websockets +# websockets loads its submodules lazily, so websockets.exceptions is not +# reachable through the package until something imports it. REQUEST_FAILURES +# is built at import time and needs it now. +import websockets.exceptions + # --------------------------------------------------------------------------- # Configuration # --------------------------------------------------------------------------- @@ -71,9 +77,81 @@ DEFAULT_WEIGHTS: dict[str, int] = { # Well-known genesis account for queries that require an account parameter. GENESIS_ACCOUNT = "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh" +# How long a single request waits for its reply. +RECV_TIMEOUT_S = 10.0 + +# Teardown budget for requests still in flight. Only the at-most-one request +# per connection that already holds the gate can still finish, and its worst +# case is the full receive timeout, so that plus a small grace is the whole +# useful wait. Requests still queued behind the gate would need +# queue_depth x round-trip, which is unbounded; they are cancelled instead. +DRAIN_TIMEOUT_S = RECV_TIMEOUT_S + 2.0 + +# Requests allowed to own one connection's recv() at a time. websockets +# rejects a second concurrent recv() on the same socket, so this must stay 1 +# unless request/response correlation by id is added. Concurrency comes from +# spreading requests round-robin over the endpoints instead. +MAX_INFLIGHT_PER_CONNECTION = 1 + +# Fraction of dispatched requests that must reach the server for a run to +# count as a measurement. The gate is one connection deep, so the ceiling is +# len(connections) / round-trip requests per second; asking for more than +# that silently drops the excess instead of erroring, which would report a +# perfect score on a run that generated a fraction of its intended load. +# Fifty percent mirrors the error-rate limit below, on the same reasoning: a +# run in which most requests did not happen is not a valid baseline. +MIN_DELIVERY_PCT = 50.0 + +# Error rate above which the run is treated as a failure. +MAX_ERROR_RATE_PCT = 50.0 + +# Failures that mean "this request failed", not "the generator is broken": +# asyncio.TimeoutError - no reply within RECV_TIMEOUT_S. Same class as the +# builtin TimeoutError on Python 3.11+. +# WebSocketException - transport failure, including the connection being +# closed while a receive was outstanding, and the +# ConcurrencyError raised for a rejected concurrent +# recv() (it subclasses WebSocketException). +# json.JSONDecodeError - reply body was not valid JSON. +# AttributeError - reply parsed to something with no .get(), e.g. a +# JSON array. +REQUEST_FAILURES: tuple[type[BaseException], ...] = ( + asyncio.TimeoutError, + websockets.exceptions.WebSocketException, + json.JSONDecodeError, + AttributeError, +) + logger = logging.getLogger("rpc_load_generator") +# --------------------------------------------------------------------------- +# Latency helpers +# --------------------------------------------------------------------------- + + +def _percentile(sorted_values: list[float], quantile: float) -> float: + """Return the nearest-rank percentile of an ascending list of values. + + The nearest-rank index is ``ceil(n * q) - 1``, clamped to the last + element. Plain ``int(n * q)`` truncation selects one rank too high + whenever ``n * q`` is a whole number — at n=100 it picks index 99, the + maximum, so the reported p99 was really p100 (n=20 for p95). + + Args: + sorted_values: Values sorted ascending. + quantile: Quantile to select, in (0, 1] — e.g. 0.99. + + Returns: + The selected value, or 0.0 when the list is empty. + """ + n = len(sorted_values) + if n == 0: + return 0.0 + idx = min(math.ceil(n * quantile) - 1, n - 1) + return sorted_values[max(idx, 0)] + + # --------------------------------------------------------------------------- # Data classes # --------------------------------------------------------------------------- @@ -83,17 +161,29 @@ logger = logging.getLogger("rpc_load_generator") class LoadStats: """Tracks request counts and latencies during a load run. + ``total_dispatched`` counts intent and ``total_sent`` counts outcome, so + the difference is the load that never happened. They diverge whenever the + requested rate exceeds what the connections can carry: the dispatch loop + keeps pace, the requests queue behind the per-connection gate, and + teardown cancels whatever never got its turn. Without the two extra + counters that shortfall shows up as nothing at all -- no error, no + warning, and an error_rate_pct of 0 over a fraction of the traffic. + Attributes: - total_sent: Total RPC requests dispatched. - total_success: Requests that returned a valid result. - total_errors: Requests that returned an error or timed out. - latencies: Per-command list of round-trip times in seconds. - command_counts: Per-command request count. + total_dispatched: Requests the dispatch loop created a task for. + total_sent: Requests that completed and were recorded. + total_success: Requests that returned a valid result. + total_errors: Requests that returned an error or timed out. + total_cancelled: Requests cancelled at teardown, never recorded. + latencies: Per-command list of round-trip times in seconds. + command_counts: Per-command request count. """ + total_dispatched: int = 0 total_sent: int = 0 total_success: int = 0 total_errors: int = 0 + total_cancelled: int = 0 latencies: dict[str, list[float]] = field(default_factory=dict) command_counts: dict[str, int] = field(default_factory=dict) @@ -108,30 +198,71 @@ class LoadStats: self.command_counts[command] = self.command_counts.get(command, 0) + 1 def summary(self) -> dict[str, Any]: - """Return a summary dict suitable for JSON serialization.""" + """Return a summary dict suitable for JSON serialization. + + ``total_sent``, ``total_success``, ``total_errors``, + ``error_rate_pct`` and ``per_command`` keep their names and meanings; + workload_orchestrator.py reads the first and third of those. The three + delivery keys are additions. ``delivery_pct`` is 0.0 when nothing was + dispatched at all -- a run that opened no connection delivered none of + its load, and reporting 100% for it would be the same blind spot the + key exists to close. + """ per_command: dict[str, Any] = {} for cmd, lats in self.latencies.items(): sorted_lats = sorted(lats) - n = len(sorted_lats) per_command[cmd] = { "count": self.command_counts.get(cmd, 0), - "p50_ms": round(sorted_lats[n // 2] * 1000, 2) if n else 0, - "p95_ms": (round(sorted_lats[int(n * 0.95)] * 1000, 2) if n else 0), - "p99_ms": (round(sorted_lats[int(n * 0.99)] * 1000, 2) if n else 0), + "p50_ms": round(_percentile(sorted_lats, 0.50) * 1000, 2), + "p95_ms": round(_percentile(sorted_lats, 0.95) * 1000, 2), + "p99_ms": round(_percentile(sorted_lats, 0.99) * 1000, 2), } return { + "total_dispatched": self.total_dispatched, "total_sent": self.total_sent, "total_success": self.total_success, "total_errors": self.total_errors, + "total_cancelled": self.total_cancelled, "error_rate_pct": ( round(self.total_errors / self.total_sent * 100, 2) if self.total_sent else 0 ), + "delivery_pct": ( + round(self.total_sent / self.total_dispatched * 100, 2) + if self.total_dispatched + else 0.0 + ), "per_command": per_command, } +@dataclass +class Connection: + """One open WebSocket endpoint together with its request gate. + + websockets raises rather than mis-delivering when two coroutines call + ``recv()`` on the same socket, so every request holds ``gate`` across its + send and its matching receive. Parallelism comes from the round-robin + spread over endpoints: N endpoints allow N requests in flight. + + run_load ──round-robin──> Connection[0] ─gate─> send_rpc (1 at a time) + └─> Connection[1] ─gate─> send_rpc (1 at a time) + + Attributes: + url: WebSocket URL this connection was opened against, for logging. + ws: The open connection. + gate: Limits concurrent send/receive pairs on ``ws`` to + MAX_INFLIGHT_PER_CONNECTION. + """ + + url: str + ws: websockets.ClientConnection + gate: asyncio.Semaphore = field( + default_factory=lambda: asyncio.Semaphore(MAX_INFLIGHT_PER_CONNECTION) + ) + + # --------------------------------------------------------------------------- # RPC command builders # --------------------------------------------------------------------------- @@ -167,7 +298,9 @@ def build_rpc_request(command: str) -> dict[str, Any]: req["limit"] = 5 elif command == "tx": # Use a dummy hash — returns "txnNotFound" error but still exercises - # the full RPC span pipeline (rpc.ws_message -> rpc.process -> rpc.command.tx). + # the full RPC span pipeline for this transport (rpc.ws_message -> + # rpc.command.tx). rpc.process is not in that chain: it is created + # only on the HTTP/JSON-RPC path, which this client never uses. req["transaction"] = "0" * 64 req["binary"] = False elif command == "account_tx": @@ -220,15 +353,23 @@ def choose_command(weights: dict[str, int]) -> str: async def send_rpc( - ws: websockets.WebSocketClientProtocol, + conn: Connection, command: str, stats: LoadStats, inject_traceparent: bool = True, ) -> None: """Send a single RPC request over WebSocket and record the result. + Holds ``conn.gate`` across the send and the matching receive so only one + request at a time owns the connection's ``recv()``. The latency clock + starts after the gate is acquired, so time spent queued behind a busy + connection is not charged to the server. + + Every outcome is recorded, including failures, so error_rate_pct in the + summary reflects every request that was actually sent. + Args: - ws: Open WebSocket connection. + conn: Target connection and its request gate. command: RPC command name. stats: LoadStats instance to record results. inject_traceparent: If True, add a W3C traceparent header field @@ -238,26 +379,173 @@ async def send_rpc( # Inject W3C traceparent for context propagation testing. # The rippled WebSocket handler extracts this from the JSON body - # when present (Phase 2 context propagation). + # when present. if inject_traceparent: trace_id = uuid.uuid4().hex span_id = uuid.uuid4().hex[:16] request["traceparent"] = f"00-{trace_id}-{span_id}-01" - t0 = time.monotonic() - try: - await ws.send(json.dumps(request)) - raw = await asyncio.wait_for(ws.recv(), timeout=10.0) - latency = time.monotonic() - t0 - response = json.loads(raw) - # Native WS responses have {"status": "success", "result": {...}} - # or {"status": "error", "error": "...", "error_message": "..."}. - success = response.get("status") == "success" + async with conn.gate: + t0 = time.monotonic() + # The try covers the I/O and the parse only. Recording sits outside it + # so a bug in record() surfaces as the task failure it is, instead of + # being counted as one more failed request. + try: + await conn.ws.send(json.dumps(request)) + raw = await asyncio.wait_for(conn.ws.recv(), timeout=RECV_TIMEOUT_S) + latency = time.monotonic() - t0 + # Native WS responses have {"status": "success", "result": {...}} + # or {"status": "error", "error": "...", "error_message": "..."}. + success = json.loads(raw).get("status") == "success" + except REQUEST_FAILURES as exc: + logger.debug("RPC %s failed: %s", command, exc) + stats.record(command, time.monotonic() - t0, False) + return stats.record(command, latency, success) - except (asyncio.TimeoutError, websockets.exceptions.WebSocketException) as exc: - latency = time.monotonic() - t0 - stats.record(command, latency, False) - logger.debug("RPC %s failed: %s", command, exc) + + +async def open_connections(endpoints: list[str]) -> list[Connection]: + """Open one persistent WebSocket connection per endpoint. + + Endpoints that refuse the connection are logged and skipped, so a partly + reachable cluster still produces load. + + Args: + endpoints: List of WebSocket URLs (ws://host:port). + + Returns: + The connections that were established, possibly empty. + """ + connections: list[Connection] = [] + for ep in endpoints: + try: + ws = await websockets.connect(ep, ping_interval=20, ping_timeout=10) + connections.append(Connection(url=ep, ws=ws)) + logger.info("Connected to %s", ep) + except Exception as exc: + logger.error("Failed to connect to %s: %s", ep, exc) + return connections + + +async def drain_requests(inflight: set[asyncio.Task[None]]) -> int: + """Let in-flight requests finish, then cancel whatever is still stuck. + + A request may wait up to RECV_TIMEOUT_S for its reply, so closing the + connections straight away would turn late replies into errors. Anything + still unfinished after DRAIN_TIMEOUT_S is cancelled, and the count is + returned so the caller can put the shortfall in the summary instead of + losing it to a log line. + + Args: + inflight: Tasks still tracked as unfinished. Finished tasks remove + themselves, so this is the outstanding set. + + Returns: + Number of requests cancelled without being recorded. + """ + pending = {task for task in inflight if not task.done()} + if not pending: + return 0 + + logger.info( + "Draining %d in-flight request(s), up to %.0fs...", + len(pending), + DRAIN_TIMEOUT_S, + ) + _, stuck = await asyncio.wait(pending, timeout=DRAIN_TIMEOUT_S) + if not stuck: + logger.info("All in-flight requests completed.") + return 0 + + for task in stuck: + task.cancel() + await asyncio.gather(*stuck, return_exceptions=True) + # Most of these never left the client: they were still waiting for the + # per-connection gate. Up to one per connection had already been sent and + # was waiting on recv() when it was cancelled, so the server may have + # handled it. Either way CancelledError is not a REQUEST_FAILURES member, + # so none of them reached stats.record and none are in total_sent. + logger.warning( + "Cancelled %d request(s) unfinished after %.0fs — not counted in " + "total_sent; see total_cancelled and delivery_pct", + len(stuck), + DRAIN_TIMEOUT_S, + ) + return len(stuck) + + +def log_progress(stats: LoadStats, elapsed: float) -> None: + """Log throughput every 100 recorded requests. + + Args: + stats: Live counters. + elapsed: Seconds since the run started. + """ + if stats.total_sent % 100 != 0 or stats.total_sent == 0: + return + logger.info( + "Progress: %d sent, %d errors, %.1f RPS (%.0fs elapsed)", + stats.total_sent, + stats.total_errors, + stats.total_sent / elapsed if elapsed > 0 else 0, + elapsed, + ) + + +async def dispatch_requests( + connections: list[Connection], + rate: float, + duration: float, + weights: dict[str, int], + stats: LoadStats, + inject_traceparent: bool, +) -> None: + """Fire requests round-robin at the target rate, then drain them. + + Each request runs as its own task so the dispatch loop keeps its pace + regardless of reply latency. Tasks are tracked, not forgotten, so + teardown can drain them and so an exception can never escape unseen. + + Every task created counts towards ``stats.total_dispatched`` and every one + cancelled at teardown towards ``stats.total_cancelled``, which is what + makes an under-delivering run visible in the summary. + + Args: + connections: Open connections to spread requests over. + rate: Target requests per second. + duration: Total run time in seconds. + weights: Command distribution weights. + stats: LoadStats instance to record results in. + inject_traceparent: Whether to inject W3C traceparent headers. + """ + interval = 1.0 / rate if rate > 0 else 0.1 + start = time.monotonic() + conn_idx = 0 + inflight: set[asyncio.Task[None]] = set() + + def reap(task: asyncio.Task[None]) -> None: + """Untrack a finished request and report anything that escaped it.""" + inflight.discard(task) + if not task.cancelled() and task.exception() is not None: + logger.error("RPC task failed unexpectedly: %s", task.exception()) + + try: + while (time.monotonic() - start) < duration: + conn = connections[conn_idx % len(connections)] + conn_idx += 1 + task = asyncio.create_task( + send_rpc(conn, choose_command(weights), stats, inject_traceparent) + ) + inflight.add(task) + task.add_done_callback(reap) + stats.total_dispatched += 1 + + await asyncio.sleep(interval) + log_progress(stats, time.monotonic() - start) + except asyncio.CancelledError: + logger.info("Load generation cancelled.") + finally: + stats.total_cancelled += await drain_requests(inflight) async def run_load( @@ -270,7 +558,9 @@ async def run_load( """Run the RPC load generator against the given endpoints. Distributes requests round-robin across endpoints at the specified - rate (requests per second) for the given duration. + rate (requests per second) for the given duration. Each connection + serves one request at a time, so the ceiling per connection is + 1 / round-trip-latency requests per second; add endpoints to raise it. Args: endpoints: List of WebSocket URLs (ws://host:port). @@ -283,18 +573,8 @@ async def run_load( LoadStats with aggregated results. """ stats = LoadStats() - interval = 1.0 / rate if rate > 0 else 0.1 - - # Open persistent connections to all endpoints. - connections: list[websockets.WebSocketClientProtocol] = [] - for ep in endpoints: - try: - ws = await websockets.connect(ep, ping_interval=20, ping_timeout=10) - connections.append(ws) - logger.info("Connected to %s", ep) - except Exception as exc: - logger.error("Failed to connect to %s: %s", ep, exc) + connections = await open_connections(endpoints) if not connections: logger.error("No connections established. Aborting.") return stats @@ -307,43 +587,26 @@ async def run_load( ) start = time.monotonic() - conn_idx = 0 - try: - while (time.monotonic() - start) < duration: - command = choose_command(weights) - ws = connections[conn_idx % len(connections)] - conn_idx += 1 - - # Fire-and-forget style with bounded concurrency via sleep. - asyncio.create_task(send_rpc(ws, command, stats, inject_traceparent)) - await asyncio.sleep(interval) - - # Periodic progress log. - elapsed = time.monotonic() - start - if stats.total_sent % 100 == 0 and stats.total_sent > 0: - actual_rps = stats.total_sent / elapsed if elapsed > 0 else 0 - logger.info( - "Progress: %d sent, %d errors, %.1f RPS (%.0fs elapsed)", - stats.total_sent, - stats.total_errors, - actual_rps, - elapsed, - ) - except asyncio.CancelledError: - logger.info("Load generation cancelled.") + await dispatch_requests( + connections, rate, duration, weights, stats, inject_traceparent + ) finally: - # Allow in-flight requests to complete. - await asyncio.sleep(2) - for ws in connections: - await ws.close() + # Only reached once dispatch_requests has drained: closing a + # connection under an outstanding receive raises ConnectionClosed, + # which would be recorded as a failed request. + for conn in connections: + await conn.ws.close() elapsed = time.monotonic() - start logger.info( - "Load complete: %d sent, %d success, %d errors in %.1fs (%.1f RPS)", + "Load complete: %d of %d dispatched sent, %d success, %d errors, " + "%d cancelled in %.1fs (%.1f RPS)", stats.total_sent, + stats.total_dispatched, stats.total_success, stats.total_errors, + stats.total_cancelled, elapsed, stats.total_sent / elapsed if elapsed > 0 else 0, ) @@ -455,9 +718,38 @@ def main() -> None: json.dump(summary, f, indent=2) logger.info("Summary written to %s", args.output) - # Exit with error if error rate exceeds 50%. - if summary["error_rate_pct"] > 50: - logger.error("High error rate: %.1f%%", summary["error_rate_pct"]) + # Both gates are evaluated and reported before exiting, and the summary is + # already on disk, so the caller sees every reason plus the numbers behind + # it. A run that under-delivers has to fail as loudly as one that errors: + # every downstream span and metric assertion would otherwise be checked + # against a fraction of the intended traffic and still look healthy. + failures: list[str] = [] + if summary["error_rate_pct"] > MAX_ERROR_RATE_PCT: + failures.append( + "error rate %.2f%% exceeds %.0f%% (%d of %d requests failed)" + % ( + summary["error_rate_pct"], + MAX_ERROR_RATE_PCT, + summary["total_errors"], + summary["total_sent"], + ) + ) + if summary["delivery_pct"] < MIN_DELIVERY_PCT: + failures.append( + "delivered %.2f%% of dispatched requests, below %.0f%% " + "(%d sent, %d cancelled, of %d dispatched) — lower --rate or add " + "endpoints" + % ( + summary["delivery_pct"], + MIN_DELIVERY_PCT, + summary["total_sent"], + summary["total_cancelled"], + summary["total_dispatched"], + ) + ) + for reason in failures: + logger.error("%s", reason) + if failures: sys.exit(1) diff --git a/docker/telemetry/workload/run-full-validation.sh b/docker/telemetry/workload/run-full-validation.sh index 3e9021fc0c..d633b463fc 100755 --- a/docker/telemetry/workload/run-full-validation.sh +++ b/docker/telemetry/workload/run-full-validation.sh @@ -19,7 +19,14 @@ # Exit codes: # 0 — All validation checks and the regression gate passed # 1 — Validation checks failed OR the regression gate detected a regression -# 2 — Infrastructure error (cluster/stack failed to start, timing capture failed) +# OR the benchmark exceeded its overhead thresholds +# 2 — Infrastructure error (cluster/stack failed to start, workload +# orchestration failed, timing capture failed, overhead could not be +# measured) +# +# Every step below records its status and folds it into FINAL_EXIT; the first +# non-zero status in pipeline order is the one returned, so the earliest +# failure — the one that explains the later ones — is what the caller sees. set -euo pipefail @@ -35,6 +42,17 @@ die() { exit 2 } +# Overall run status, folded step by step (see the exit-code table above). +FINAL_EXIT=0 + +# fold_exit STATUS — record a step's status in FINAL_EXIT. +# First non-zero wins, so FINAL_EXIT names the earliest failing step. +fold_exit() { + if [ "$1" -ne 0 ] && [ "$FINAL_EXIT" -eq 0 ]; then + FINAL_EXIT="$1" + fi +} + # --------------------------------------------------------------------------- # Configuration # --------------------------------------------------------------------------- @@ -49,6 +67,9 @@ NUM_NODES=5 RPC_PORT_BASE=5005 WS_PORT_BASE=6006 PEER_PORT_BASE=51235 +# Inert: parsed from --rpc-rate/--rpc-duration/--tx-tps/--tx-duration and never +# read again. Load shape comes from the workload profile instead. Kept because +# the CI workflow still passes the four flags. RPC_RATE=50 RPC_DURATION=120 TX_TPS=5 @@ -78,16 +99,22 @@ usage() { echo "Options:" echo " --xrpld PATH Path to xrpld binary" echo " --nodes NUM Number of validator nodes (default: 5)" - echo " --rpc-rate RPS RPC load rate (default: 50)" - echo " --rpc-duration SECS RPC load duration (default: 120)" - echo " --tx-tps TPS Transaction submit rate (default: 5)" - echo " --tx-duration SECS Transaction submit duration (default: 120)" echo " --profile NAME Workload profile (default: full-validation)" echo " --with-benchmark Also run performance overhead benchmark (telemetry off vs on)" echo " --skip-loki Skip Loki log-trace correlation checks" echo " --skip-regression Skip the OTel-baseline regression gate" echo " --cleanup Tear down everything and exit" echo " -h, --help Show this help" + echo "" + echo "Accepted but INERT (parsed for compatibility, then ignored):" + echo " --rpc-rate RPS no effect" + echo " --rpc-duration SECS no effect" + echo " --tx-tps TPS no effect" + echo " --tx-duration SECS no effect" + echo "" + echo " Load shape comes from the workload profile (--profile), which sets" + echo " the rate and duration of every phase in workload-profiles.json." + echo " These four flags stay accepted because the CI workflow passes them." exit 0 } @@ -101,6 +128,7 @@ while [ $# -gt 0 ]; do NUM_NODES="$2" shift 2 ;; + # The next four are inert — see the RPC_RATE default above. --rpc-rate) RPC_RATE="$2" shift 2 @@ -135,7 +163,10 @@ while [ $# -gt 0 ]; do ;; --cleanup) # Cleanup mode log "Cleaning up..." - pkill -f "$WORKDIR" 2>/dev/null || true + # Match the node config path, not the bare workdir: a plain + # "$WORKDIR" pattern also matches any shell, editor or log tail + # whose command line merely mentions that path. + pkill -f "$WORKDIR/node[0-9]+/xrpld\.cfg" 2>/dev/null || true docker compose -f "$COMPOSE_FILE" down 2>/dev/null || true rm -rf "$WORKDIR" ok "Cleanup complete." @@ -170,7 +201,8 @@ ok "Prerequisites verified." # Cleanup previous run # --------------------------------------------------------------------------- log "Cleaning up previous run..." -pkill -f "$WORKDIR" 2>/dev/null || true +# Narrowed for the same reason as the --cleanup branch above. +pkill -f "$WORKDIR/node[0-9]+/xrpld\.cfg" 2>/dev/null || true sleep 2 rm -rf "$WORKDIR" mkdir -p "$WORKDIR" "$REPORT_DIR" @@ -339,7 +371,13 @@ for attempt in $(seq 1 120); do break fi if [ "$attempt" -eq 120 ]; then - warn "Consensus timeout — $ready/$NUM_NODES nodes ready" + # Fatal, not a warning. A partial cluster still answers queries, so the + # run would complete and report unrelated span/metric failures: series + # counts scale with the number of live nodes, and spans that need a + # quorum are simply never emitted. One infrastructure error here is + # worth more than a pile of misleading assertion failures later. + echo "" + die "Consensus timeout — only $ready/$NUM_NODES nodes proposing after ${attempt}s. Check $WORKDIR/node*/debug.log, then '$0 --cleanup'." fi printf "\r %d/%d nodes proposing..." "$ready" "$NUM_NODES" sleep 1 @@ -356,7 +394,13 @@ for attempt in $(seq 1 60); do ok "Validated ledger: seq $val_seq" break fi - [ "$attempt" -eq 60 ] && warn "No validated ledger after 60s" + # Fatal for the same reason as the consensus timeout above, and because + # several assertions are gated on a validated ledger existing at all: + # ledger_economy{metric="base_fee_xrp"} is only observed from a validated + # ledger, and complete_ledgers stays absent while the range is empty. + if [ "$attempt" -eq 60 ]; then + die "No validated ledger after ${attempt}s (last seq: $val_seq). Check $WORKDIR/node*/debug.log, then '$0 --cleanup'." + fi sleep 1 done @@ -370,14 +414,21 @@ for i in $(seq 1 "$NUM_NODES"); do WS_ENDPOINTS="$WS_ENDPOINTS ws://localhost:$((WS_PORT_BASE + i - 1))" done +ORCHESTRATOR_EXIT=0 python3 "$SCRIPT_DIR/workload_orchestrator.py" \ --profile "$WORKLOAD_PROFILE" \ --endpoints $WS_ENDPOINTS \ --report "$REPORT_DIR/workload-report.json" \ - --report-dir "$REPORT_DIR" || - warn "Workload orchestrator returned non-zero exit" + --report-dir "$REPORT_DIR" || ORCHESTRATOR_EXIT=$? -ok "Workload orchestration complete." +if [ "$ORCHESTRATOR_EXIT" -eq 0 ]; then + ok "Workload orchestration complete." +else + # Treated as an infrastructure error: the span and metric assertions below + # would be graded against traffic that was never generated. + fail "Workload orchestrator failed (exit $ORCHESTRATOR_EXIT) — the checks below run against incomplete traffic" + fold_exit 2 +fi # --------------------------------------------------------------------------- # Step 5: Run telemetry validation suite @@ -397,6 +448,7 @@ if [ "$VALIDATION_EXIT" -eq 0 ]; then else fail "Some telemetry validation checks failed (exit $VALIDATION_EXIT)" fi +fold_exit "$VALIDATION_EXIT" # --------------------------------------------------------------------------- # Step 6: Capture OTel timings and run the regression comparison @@ -443,18 +495,33 @@ if [ "$SKIP_REGRESSION" != true ]; then else warn "Regression gate skipped." fi +fold_exit "$REGRESSION_EXIT" # --------------------------------------------------------------------------- # Step 7: (Optional) Run overhead benchmark # --------------------------------------------------------------------------- +BENCHMARK_EXIT=0 if [ "$WITH_BENCHMARK" = true ]; then log "Step 7: Running performance benchmark..." bash "$SCRIPT_DIR/benchmark.sh" \ --xrpld "$XRPLD" \ --duration 120 \ --nodes 3 \ - --output "$REPORT_DIR" || - warn "Benchmark returned non-zero exit" + --output "$REPORT_DIR" || BENCHMARK_EXIT=$? + + if [ "$BENCHMARK_EXIT" -eq 0 ]; then + ok "Benchmark within overhead thresholds." + elif [ "$BENCHMARK_EXIT" -eq 1 ]; then + # A measured threshold breach — same class as a failed check. + fail "Benchmark exceeded overhead thresholds (exit 1)" + fold_exit 1 + else + # benchmark.sh could not produce a usable measurement (e.g. incomplete + # system metrics). Reported as an infrastructure error, not a perf + # regression: nothing was measured, so nothing was breached. + fail "Benchmark could not measure overhead (exit $BENCHMARK_EXIT) — treated as an infrastructure error" + fold_exit 2 + fi fi # --------------------------------------------------------------------------- @@ -485,15 +552,13 @@ echo "" echo " To tear down:" echo " $0 --cleanup" echo "" +echo " Step statuses (0 = ok):" +echo " Workload orchestration: $ORCHESTRATOR_EXIT" +echo " Telemetry validation: $VALIDATION_EXIT" +echo " Regression gate: $REGRESSION_EXIT" +echo " Overhead benchmark: $BENCHMARK_EXIT" +echo "" echo "===========================================================" -# Fail the run if EITHER validation or the regression gate failed. The -# `[ "$VAR" -gt N ]` comparison works here because exit codes are numeric. -FINAL_EXIT=0 -if [ "$VALIDATION_EXIT" -ne 0 ]; then - FINAL_EXIT="$VALIDATION_EXIT" -fi -if [ "$REGRESSION_EXIT" -ne 0 ] && [ "$FINAL_EXIT" -eq 0 ]; then - FINAL_EXIT="$REGRESSION_EXIT" -fi +# FINAL_EXIT already holds the first non-zero step status (see fold_exit). exit "$FINAL_EXIT" diff --git a/docker/telemetry/workload/tx_submitter.py b/docker/telemetry/workload/tx_submitter.py index 4d779ec98e..745ce0ae9a 100644 --- a/docker/telemetry/workload/tx_submitter.py +++ b/docker/telemetry/workload/tx_submitter.py @@ -90,6 +90,57 @@ DEFAULT_TX_WEIGHTS: dict[str, int] = { # Number of test accounts to create. NUM_TEST_ACCOUNTS = 8 +# Minimum number of funded accounts the transaction builders need: TX_BUILDERS +# indexes positions 0..5 of the account list. +MIN_FUNDED_ACCOUNTS = 6 + +# Engine results that tie up the submitted sequence number, other than the +# tec* family which is matched by prefix. See consumes_sequence(). +SEQ_CONSUMING_RESULTS = frozenset({"tesSUCCESS", "terQUEUED"}) + +# Consecutive non-consuming submit results from one account before its +# sequence is re-read from the ledger. The periodic refresh only ever raises +# the counter, so a counter that leads the ledger -- a queued transaction that +# was later dropped, say -- would otherwise make every further submit from +# that account fail forever. Five is above the two or three rejections a +# single bad transaction type can produce in a row, and at the default 5 TPS +# it triggers within a few seconds, well before the periodic refresh below. +SEQ_REFETCH_AFTER_FAILURES = 5 + +# How often the submission loop re-reads every account's sequence from the +# ledger, to stay close to sequences other submitters have advanced. +SEQ_REFRESH_INTERVAL_S = 10.0 + + +def consumes_sequence(engine_result: str | None) -> bool: + """Report whether an engine result tied up the submitted sequence number. + + Two outcomes do. ``tesSUCCESS`` is applied (TER.h:243) and every ``tec*`` + claims the fee "to use the sequence number" (TER.h:265-266), so both + advance the account root. ``terQUEUED`` is the one ``ter*`` code rippled + forwards (TER.h:205); the transaction waits in the queue still holding + that sequence, so the next submit needs the following one. + + Everything else leaves the sequence free. ``tem*``, ``tef*`` and ``tel*`` + are neither applied nor forwarded, and neither are the remaining ``ter*`` + codes (TER.h:202-206) -- including ``terPRE_SEQ``, which reports that the + submitted sequence is already past the account root (TER.h:217). + Advancing the local counter on those widens the gap TER.h:209 calls a + "hole in sequence which jams transactions", and it is what + SEQ_REFETCH_AFTER_FAILURES exists to recover from. + + Args: + engine_result: The ``engine_result`` field of a submit response. A + missing or JSON-null field reads as non-consuming + rather than raising, so the caller records the + transaction exactly once. + + Returns: + True if the submitted sequence number is spoken for. + """ + result = str(engine_result or "") + return result in SEQ_CONSUMING_RESULTS or result.startswith("tec") + # --------------------------------------------------------------------------- # Data classes @@ -98,19 +149,28 @@ NUM_TEST_ACCOUNTS = 8 @dataclass class Account: - """Represents a funded XRPL test account. + """Represents an XRPL test account, funded or not. Attributes: name: Human-readable name (e.g., "alice"). account: Classic address (rXXX...). seed: Secret seed for signing. sequence: Next available sequence number. + funded: True once the account root exists on the ledger. False + accounts cannot submit, so they are excluded from the + submission loop rather than silently counted. + stalled: Consecutive submit results from this account that consumed + no sequence. Reset by any consuming result; at + SEQ_REFETCH_AFTER_FAILURES the sequence is re-read from + the ledger. """ name: str account: str seed: str sequence: int = 0 + funded: bool = False + stalled: int = 0 @dataclass @@ -163,7 +223,7 @@ class TxStats: async def ws_request( - ws: websockets.WebSocketClientProtocol, + ws: websockets.ClientConnection, command: str, params: dict[str, Any] | None = None, ) -> dict[str, Any]: @@ -204,7 +264,7 @@ async def ws_request( return resp.get("result", resp) -async def create_account(ws: websockets.WebSocketClientProtocol, name: str) -> Account: +async def create_account(ws: websockets.ClientConnection, name: str) -> Account: """Create a new account via wallet_propose RPC. Args: @@ -227,7 +287,7 @@ async def create_account(ws: websockets.WebSocketClientProtocol, name: str) -> A async def fund_account( - ws: websockets.WebSocketClientProtocol, + ws: websockets.ClientConnection, dest: Account, genesis_seq: int, ) -> tuple[bool, int]: @@ -239,7 +299,8 @@ async def fund_account( genesis_seq: Current genesis account sequence number. Returns: - Tuple of (success: bool, next_sequence: int). + Tuple of (funded: bool, next_genesis_sequence: int). The sequence is + unchanged when the ledger did not consume it. """ resp = await ws_request( ws, @@ -265,12 +326,16 @@ async def fund_account( engine_result, json.dumps(resp, indent=None)[:500], ) - return success, genesis_seq + 1 + # Advance the genesis sequence only when the ledger consumed it. The + # caller reads it once and threads it through every funding submit, so + # advancing past a tem*/tef*/tel* rejection would put every remaining + # submit on a future sequence and fund nothing. + if consumes_sequence(engine_result): + genesis_seq += 1 + return success, genesis_seq -async def get_account_sequence( - ws: websockets.WebSocketClientProtocol, account: str -) -> int: +async def get_account_sequence(ws: websockets.ClientConnection, account: str) -> int: """Get the current sequence number for an account. Args: @@ -564,7 +629,7 @@ TX_BUILDERS: dict[str, Any] = { async def setup_accounts( - ws: websockets.WebSocketClientProtocol, + ws: websockets.ClientConnection, ) -> list[Account]: """Create and fund test accounts from genesis. @@ -575,7 +640,8 @@ async def setup_accounts( ws: Open WebSocket connection to a rippled node. Returns: - List of funded Account instances. + Every created Account. ``funded`` marks the ones the ledger accepted, + so the caller must filter on it rather than on the list length. """ account_names = ["alice", "bob", "carol", "dave", "eve", "frank", "grace", "heidi"] @@ -593,8 +659,8 @@ async def setup_accounts( # Fund all accounts. logger.info("Funding test accounts...") for acct in accounts: - success, genesis_seq = await fund_account(ws, acct, genesis_seq) - if success: + acct.funded, genesis_seq = await fund_account(ws, acct, genesis_seq) + if acct.funded: logger.info(" Funded %s", acct.name) else: logger.warning(" Failed to fund %s", acct.name) @@ -603,19 +669,34 @@ async def setup_accounts( logger.info("Waiting 10s for funding transactions to validate...") await asyncio.sleep(10) - # Refresh sequence numbers for all accounts. + # Refresh sequence numbers, and confirm funding against the ledger rather + # than trusting the submit result. get_account_sequence returns 0 when + # account_info reports no account_data, which means the account root was + # never created; a sequence we cannot read also makes the account + # unusable, so either way it must not be submitted from. for acct in accounts: try: acct.sequence = await get_account_sequence(ws, acct.account) - logger.info(" %s sequence: %d", acct.name, acct.sequence) except Exception as exc: logger.warning(" Failed to get sequence for %s: %s", acct.name, exc) + if acct.sequence > 0: + logger.info(" %s sequence: %d", acct.name, acct.sequence) + else: + acct.funded = False + logger.warning( + " %s has no ledger sequence — treating as unfunded", acct.name + ) + logger.info( + "Funded %d of %d created accounts", + sum(1 for a in accounts if a.funded), + len(accounts), + ) return accounts async def submit_transaction( - ws: websockets.WebSocketClientProtocol, + ws: websockets.ClientConnection, tx_type: str, accounts: list[Account], stats: TxStats, @@ -652,8 +733,13 @@ async def submit_transaction( ) stats.record(tx_type, success) + # The sequence gate is deliberately not `success`: that tuple is + # narrower (every tec* consumes a sequence, only two are listed) and + # wider (a tem*/tef*/tel*/ter* rejection consumes none) than the set + # of results that actually consume one. _track_sequence also owns the + # recovery path for a counter that has drifted ahead of the ledger. if sender: - sender.sequence += 1 + await _track_sequence(ws, sender, engine_result) if not success: # First occurrence of each distinct result at WARNING, the rest at @@ -675,15 +761,77 @@ async def submit_transaction( _log_first_failure("exc:%s" % type(exc).__name__, "%s error: %s", tx_type, exc) +async def _track_sequence( + ws: websockets.ClientConnection, + sender: Account, + engine_result: str | None, +) -> None: + """Advance, or re-read from the ledger, one sender's sequence number. + + A consuming result moves the counter on by one and clears the stall + streak. A non-consuming one leaves the counter where it is, so the same + sequence is offered again -- correct when the rejection was about the + transaction, but a livelock when the counter itself is the problem, since + _refresh_sequences never lowers it. After SEQ_REFETCH_AFTER_FAILURES + consecutive non-consuming results the ledger's own value is taken + instead, in either direction. + + A zero from get_account_sequence means account_info returned no + account_data, so it is ignored rather than written over a usable counter. + + Args: + ws: Open WebSocket connection. + sender: The account the transaction was submitted from. + engine_result: The ``engine_result`` of that submit. + """ + if consumes_sequence(engine_result): + sender.sequence += 1 + sender.stalled = 0 + return + + sender.stalled += 1 + if sender.stalled < SEQ_REFETCH_AFTER_FAILURES: + return + + sender.stalled = 0 + try: + seq = await get_account_sequence(ws, sender.account) + except Exception as exc: + logger.warning("Sequence re-fetch for %s failed: %s", sender.name, exc) + return + if seq > 0 and seq != sender.sequence: + logger.warning( + "Re-syncing %s sequence %d -> %d after %d non-consuming results", + sender.name, + sender.sequence, + seq, + SEQ_REFETCH_AFTER_FAILURES, + ) + sender.sequence = seq + + async def _refresh_sequences( - ws: websockets.WebSocketClientProtocol, + ws: websockets.ClientConnection, accounts: list[Account], ) -> None: """Re-sync account sequences from the validated ledger. In a consensus network, other nodes' transactions advance sequences - beyond the submitter's local tracking. Refreshing every ~10 s keeps - the local counter close to the ledger and prevents tefPAST_SEQ storms. + beyond the submitter's local tracking. Refreshing every + SEQ_REFRESH_INTERVAL_S keeps the local counter close to the ledger and + prevents tefPAST_SEQ storms. + + The counter is only ever raised here, never lowered, because it may + legitimately lead the ledger: a queued transaction holds its sequence + without having applied yet, and lowering the counter would reuse it. + Raising it fixes the opposite case, where the ledger has moved on -- a + submit response that was lost while the transaction applied, or another + submitter using the same account. + + That leaves one case this cannot fix: a counter that leads the ledger and + never catches up, because the transaction it was advanced for was dropped + rather than applied. _track_sequence handles that one by re-reading the + ledger value in either direction. """ for acct in accounts: try: @@ -694,6 +842,59 @@ async def _refresh_sequences( pass +async def _submission_loop( + ws: websockets.ClientConnection, + accounts: list[Account], + weights: dict[str, int], + duration: float, + interval: float, + stats: TxStats, +) -> float: + """Submit a weighted transaction mix until ``duration`` elapses. + + Args: + ws: Open WebSocket connection. + accounts: Funded accounts to submit from. + weights: Transaction type distribution weights. + duration: Run time in seconds. + interval: Delay between submissions, i.e. 1 / target TPS. + stats: TxStats instance to record results in. + + Returns: + Seconds actually spent in the loop. + """ + tx_types = list(weights.keys()) + tx_weights = [weights[t] for t in tx_types] + + start = time.monotonic() + last_seq_refresh = start + while (time.monotonic() - start) < duration: + # Periodically re-sync account sequences from the ledger so + # locally-tracked sequences don't drift behind consensus. + if (time.monotonic() - last_seq_refresh) >= SEQ_REFRESH_INTERVAL_S: + await _refresh_sequences(ws, accounts) + last_seq_refresh = time.monotonic() + + tx_type = random.choices(tx_types, weights=tx_weights, k=1)[0] + await submit_transaction(ws, tx_type, accounts, stats) + await asyncio.sleep(interval) + + # Progress logging every 50 transactions. + if stats.total_submitted % 50 == 0 and stats.total_submitted > 0: + elapsed = time.monotonic() - start + logger.info( + "Progress: %d submitted, %d success, %d errors, " + "%.1f TPS (%.0fs elapsed)", + stats.total_submitted, + stats.total_success, + stats.total_errors, + stats.total_submitted / elapsed if elapsed > 0 else 0, + elapsed, + ) + + return time.monotonic() - start + + async def run_submitter( endpoint: str, tps: float, @@ -713,60 +914,40 @@ async def run_submitter( """ stats = TxStats() interval = 1.0 / tps if tps > 0 else 0.5 + elapsed = 0.0 ws = await websockets.connect(endpoint, ping_interval=20, ping_timeout=10) logger.info("Connected to %s", endpoint) try: - # Setup test accounts. - accounts = await setup_accounts(ws) - if len(accounts) < 6: - logger.error("Need at least 6 funded accounts, got %d", len(accounts)) + # Setup test accounts. Every created account is returned whether or + # not funding worked, so submit only from the funded ones — the + # builders address accounts by position and an unfunded account there + # would fail every transaction it is picked for. + created = await setup_accounts(ws) + accounts = [acct for acct in created if acct.funded] + if len(accounts) < MIN_FUNDED_ACCOUNTS: + logger.error( + "Need at least %d funded accounts, only %d of %d created " + "accounts were funded", + MIN_FUNDED_ACCOUNTS, + len(accounts), + len(created), + ) return stats - # Build weighted command list. - tx_types = list(weights.keys()) - tx_weights = [weights[t] for t in tx_types] - logger.info( "Starting TX submission: tps=%s, duration=%ss, types=%d", tps, duration, - len(tx_types), + len(weights), + ) + elapsed = await _submission_loop( + ws, accounts, weights, duration, interval, stats ) - - start = time.monotonic() - last_seq_refresh = start - seq_refresh_interval = 10.0 - while (time.monotonic() - start) < duration: - # Periodically re-sync account sequences from the ledger so - # locally-tracked sequences don't drift behind consensus. - if (time.monotonic() - last_seq_refresh) >= seq_refresh_interval: - await _refresh_sequences(ws, accounts) - last_seq_refresh = time.monotonic() - - tx_type = random.choices(tx_types, weights=tx_weights, k=1)[0] - await submit_transaction(ws, tx_type, accounts, stats) - await asyncio.sleep(interval) - - # Progress logging every 50 transactions. - if stats.total_submitted % 50 == 0 and stats.total_submitted > 0: - elapsed = time.monotonic() - start - actual_tps = stats.total_submitted / elapsed if elapsed > 0 else 0 - logger.info( - "Progress: %d submitted, %d success, %d errors, " - "%.1f TPS (%.0fs elapsed)", - stats.total_submitted, - stats.total_success, - stats.total_errors, - actual_tps, - elapsed, - ) - finally: await ws.close() - elapsed = time.monotonic() - start logger.info( "Submission complete: %d submitted, %d success, %d errors " "in %.1fs (%.1f TPS)", diff --git a/docker/telemetry/workload/validate_telemetry.py b/docker/telemetry/workload/validate_telemetry.py index 40f7e507a8..6f2c74ebfb 100644 --- a/docker/telemetry/workload/validate_telemetry.py +++ b/docker/telemetry/workload/validate_telemetry.py @@ -6,10 +6,13 @@ a workload run. Queries Tempo (spans), Prometheus (metrics), Loki (logs), and Grafana (dashboards) APIs to produce a pass/fail report. Validation categories: - 1. Span validation — All 16+ span types present with required attributes - 2. Metric validation — SpanMetrics, StatsD, and Phase 9 metrics are non-zero + 1. Span validation — Every required span type in expected_spans.json, each + carrying its required attributes + 2. Metric validation — SpanMetrics, StatsD, and MetricsRegistry OTLP metrics + are non-zero 3. Log-trace correlation — Loki logs contain trace_id/span_id fields - 4. Dashboard validation — All 14 Grafana dashboards render data + 4. Dashboard validation — Every dashboard uid in expected_metrics.json + provisions and loads (panel count only, not panel data) 5. External parity — Span attrs, metric existence, and value sanity for external dashboard parity (validator-health, peer-quality, node-health) @@ -27,6 +30,7 @@ Usage: import argparse import asyncio +import fnmatch import json import logging import sys @@ -62,6 +66,12 @@ EXPECTED_METRICS_FILE = SCRIPT_DIR / "expected_metrics.json" METRIC_POLL_TIMEOUT_SEC = 45.0 METRIC_POLL_INTERVAL_SEC = 5.0 +# All metrics are polled concurrently against ONE shared deadline, so the +# metric phase costs a single poll window instead of one per metric. This caps +# how many /api/v1/series requests are in flight at a time, so the fan-out does +# not hammer the single-container Prometheus the harness runs. +METRIC_POLL_CONCURRENCY = 8 + # --------------------------------------------------------------------------- # Data classes @@ -218,6 +228,26 @@ def _otlp_span_attr_keys(span: dict[str, Any]) -> set[str]: return {a["key"] for a in span.get("attributes", []) if "key" in a} +def _span_name_matches(emitted_name: str, expected_name: str) -> bool: + """Test an emitted span name against a name from expected_spans.json. + + Contract names are either literals or globs containing "*" (for example + "rpc.command.*"). Literals are compared for exact equality so a longer + emitted name cannot satisfy a shorter contract: "consensus.accept.apply" + must not stand in for "consensus.accept". + + Args: + emitted_name: Span name as reported by Tempo. + expected_name: Span name or glob pattern from expected_spans.json. + + Returns: + True when the emitted name satisfies the expected name. + """ + if "*" in expected_name: + return fnmatch.fnmatchcase(emitted_name, expected_name) + return emitted_name == expected_name + + # --------------------------------------------------------------------------- # Span Validation (Tempo API) # --------------------------------------------------------------------------- @@ -404,10 +434,20 @@ async def _validate_span_attributes_otlp( span_def: dict[str, Any], report: ValidationReport, ) -> None: - """Check that OTLP spans contain expected attributes. + """Check that the contract's own span carries its required attributes. + + Only spans whose name matches ``span_def["name"]`` are inspected. + Attributes are never borrowed from siblings: many span types share keys + such as ledger_seq or tx_hash, so a trace-wide scan would satisfy every + one of those contracts from a single carrier span and make the per-span + contract unenforceable. + + A span type passes when at least one instance of it carries every required + attribute. When none does, the closest instance's missing keys are + reported. Args: - spans: List of OTLP span dicts from Tempo. + spans: Every OTLP span dict in the fetched trace. span_def: Span definition from expected_spans.json. report: ValidationReport to accumulate results. """ @@ -416,26 +456,53 @@ async def _validate_span_attributes_otlp( return span_name = span_def["name"] - # Collect all attribute keys from all spans. - found_attrs: set[str] = set() - for span in spans: - found_attrs.update(_otlp_span_attr_keys(span)) + check_name = f"span.attrs.{span_name}" + matching = [s for s in spans if _span_name_matches(s.get("name", ""), span_name)] + + if not matching: + report.add( + CheckResult( + name=check_name, + category="span", + passed=False, + message=( + f"{span_name}: no span named '{span_name}' in the fetched " + "trace, cannot verify its attributes" + ), + details={"required": required_attrs, "instances": 0}, + ) + ) + return + + # Keep the instance that is missing the fewest required attributes, so the + # failure message names the closest witness rather than an arbitrary one. + best_found: set[str] = set() + best_missing: list[str] = list(required_attrs) + for span in matching: + found = _otlp_span_attr_keys(span) + missing = [a for a in required_attrs if a not in found] + if len(missing) < len(best_missing): + best_found, best_missing = found, missing + if not best_missing: + break - missing = [a for a in required_attrs if a not in found_attrs] report.add( CheckResult( - name=f"span.attrs.{span_name}", + name=check_name, category="span", - passed=len(missing) == 0, + passed=not best_missing, message=( f"{span_name}: all {len(required_attrs)} attributes present" - if not missing - else f"{span_name}: missing attributes: {missing}" + if not best_missing + else f"{span_name}: no '{span_name}' span carried all " + f"{len(required_attrs)} required attributes; closest of " + f"{len(matching)} instance(s) missing {best_missing}" ), details={ "required": required_attrs, - "found": list(found_attrs), - "missing": missing, + "found": sorted(best_found), + "missing": best_missing, + "instances": len(matching), }, ) ) @@ -474,21 +541,21 @@ async def _validate_parent_child( ) return - # Check if child spans exist within parent traces. - # Use the concrete child name for wildcard patterns. - concrete_child = child_name.replace("*", "server_info") + # Check if child spans exist within parent traces. Names are matched + # exactly (globs for wildcard contracts) — a substring test let a + # longer emitted name satisfy a shorter contract, so + # consensus.round -> consensus.accept passed on a + # consensus.accept.apply span alone. found_child = False for trace_summary in traces: trace_id = trace_summary.get("traceID", "") if not trace_id: continue spans = await _tempo_get_trace(session, tempo_url, trace_id) - for span in spans: - op = span.get("name", "") - if concrete_child in op or ("*" not in child_name and op == child_name): - found_child = True - break - if found_child: + if any( + _span_name_matches(span.get("name", ""), child_name) for span in spans + ): + found_child = True break report.add( @@ -519,29 +586,25 @@ async def _validate_parent_child( # --------------------------------------------------------------------------- -async def validate_metrics( - session: aiohttp.ClientSession, - prometheus_url: str, - report: ValidationReport, +async def _log_prometheus_metric_names( + session: aiohttp.ClientSession, prometheus_url: str ) -> None: - """Validate that expected metrics appear in Prometheus with non-zero values. + """Log the harness-relevant metric names Prometheus currently knows. + + Diagnostic only — this output appears in CI logs and helps debug name + mismatches between expected_metrics.json and actual emissions. Failures + are warnings, never check failures. Args: session: aiohttp client session. - prometheus_url: Base URL for Prometheus API (e.g., http://localhost:9090). - report: ValidationReport to accumulate results. + prometheus_url: Prometheus base URL. """ - logger.info("--- Metric Validation (Prometheus) ---") - - # Diagnostic: list all metric names in Prometheus. Helps debug name - # mismatches between expected_metrics.json and actual emissions. try: async with session.get( f"{prometheus_url}/api/v1/label/__name__/values" ) as resp: label_data = await resp.json() all_metrics = label_data.get("data", []) - # Log relevant metrics for debugging. relevant = [ m for m in all_metrics @@ -577,19 +640,99 @@ async def validate_metrics( except Exception as exc: logger.warning("Failed to fetch Prometheus metric names: %s", exc) + +async def validate_metrics( + session: aiohttp.ClientSession, + prometheus_url: str, + report: ValidationReport, +) -> None: + """Validate that expected metrics appear in Prometheus with non-zero values. + + Args: + session: aiohttp client session. + prometheus_url: Base URL for Prometheus API (e.g., http://localhost:9090). + report: ValidationReport to accumulate results. + """ + logger.info("--- Metric Validation (Prometheus) ---") + + await _log_prometheus_metric_names(session, prometheus_url) + with open(EXPECTED_METRICS_FILE) as f: expected = json.load(f) - # Check each metric category. - for category_key, category_data in expected.items(): - if category_key in ("description", "grafana_dashboards"): - continue + # Flatten every (category, metric) pair the contract asserts, then poll + # them concurrently against ONE shared deadline. Polling them serially made + # each metric own its own timeout, so the waits were additive: 58 metrics x + # 45 s = 43.5 min, which overran the CI job budget and lost the + # artifact-upload and summary diagnostics. Sharing the deadline bounds the + # whole phase to a single poll window. + targets = [ + (category_key, metric_name) + for category_key, category_data in expected.items() + if category_key not in ("description", "grafana_dashboards") + for metric_name in category_data.get("metrics", []) + ] - metrics = category_data.get("metrics", []) - for metric_name in metrics: - await _check_prometheus_metric( - session, prometheus_url, metric_name, category_key, report + deadline = time.monotonic() + METRIC_POLL_TIMEOUT_SEC + sem = asyncio.Semaphore(METRIC_POLL_CONCURRENCY) + checks = await asyncio.gather( + *( + _check_prometheus_metric( + session, prometheus_url, metric_name, category, deadline, sem ) + for category, metric_name in targets + ) + ) + + # Add in contract order, not completion order, so the report and its log + # lines stay deterministic across runs. + for check in checks: + report.add(check) + + +async def _poll_series_count( + session: aiohttp.ClientSession, + prometheus_url: str, + metric_name: str, + deadline: float, + sem: asyncio.Semaphore, +) -> int: + """Poll Prometheus until a metric has series or the deadline passes. + + Uses the /api/v1/series endpoint instead of an instant query. + Beast::insight StatsD gauges only mark dirty on value *changes*, so a gauge + that stabilizes (e.g. peer count stays at 1) may go stale in Prometheus and + disappear from instant queries. The series endpoint returns any metric + that existed in the window, regardless of staleness. + + Polls rather than querying once: late-populating gauges/counters may not + have completed the export+scrape pipeline when this runs, so a single query + races. A metric that never appears still fails once the deadline passes. + + Args: + session: aiohttp client session. + prometheus_url: Prometheus base URL. + metric_name: Prometheus metric name. + deadline: Monotonic deadline shared by every metric in the run. + sem: Bounds how many requests reach Prometheus at once. It + is held only across the request, never across the + sleep, so one absent metric cannot starve the others. + + Returns: + Number of series found, or 0 if the metric never appeared. + """ + params: dict[str, str] = {"match[]": metric_name} + while True: + async with sem: + async with session.get( + f"{prometheus_url}/api/v1/series", params=params + ) as resp: + data = await resp.json() + series_count = len(data.get("data", [])) + if series_count > 0 or time.monotonic() >= deadline: + return series_count + # Never sleep past the shared deadline. + await asyncio.sleep(min(METRIC_POLL_INTERVAL_SEC, deadline - time.monotonic())) async def _check_prometheus_metric( @@ -597,8 +740,9 @@ async def _check_prometheus_metric( prometheus_url: str, metric_name: str, category: str, - report: ValidationReport, -) -> None: + deadline: float, + sem: asyncio.Semaphore, +) -> CheckResult: """Query Prometheus for a specific metric and check it exists. Args: @@ -606,54 +750,34 @@ async def _check_prometheus_metric( prometheus_url: Prometheus base URL. metric_name: Prometheus metric name. category: Metric category for the report. - report: ValidationReport to accumulate results. + deadline: Monotonic deadline shared by every metric in the run. + sem: Bounds how many requests reach Prometheus at once. + + Returns: + The CheckResult for this metric. The caller adds it to the report so + report order follows the contract file rather than completion order. """ try: - # Use the /api/v1/series endpoint instead of an instant query. - # Beast::insight StatsD gauges only mark dirty on value *changes*, - # so a gauge that stabilizes (e.g. peer count stays at 1) may go - # stale in Prometheus and disappear from instant queries. The - # series endpoint returns any metric that existed in the window, - # regardless of staleness. - # - # Poll rather than query once: late-populating gauges/counters may - # not have completed the export+scrape pipeline when this runs, so a - # single query races. Re-query until the metric appears or the poll - # window elapses; a metric that never appears still fails after the - # timeout. - params: dict[str, str] = {"match[]": metric_name} - series_count = 0 - deadline = time.monotonic() + METRIC_POLL_TIMEOUT_SEC - while True: - async with session.get( - f"{prometheus_url}/api/v1/series", params=params - ) as resp: - data = await resp.json() - series_count = len(data.get("data", [])) - if series_count > 0 or time.monotonic() >= deadline: - break - await asyncio.sleep(METRIC_POLL_INTERVAL_SEC) - report.add( - CheckResult( - name=f"metric.{category}.{metric_name}", - category="metric", - passed=series_count > 0, - message=( - f"{metric_name}: {series_count} series" - if series_count > 0 - else f"{metric_name}: 0 series (expected > 0)" - ), - details={"series_count": series_count}, - ) + series_count = await _poll_series_count( + session, prometheus_url, metric_name, deadline, sem + ) + return CheckResult( + name=f"metric.{category}.{metric_name}", + category="metric", + passed=series_count > 0, + message=( + f"{metric_name}: {series_count} series" + if series_count > 0 + else f"{metric_name}: 0 series (expected > 0)" + ), + details={"series_count": series_count}, ) except Exception as exc: - report.add( - CheckResult( - name=f"metric.{category}.{metric_name}", - category="metric", - passed=False, - message=f"{metric_name}: query failed ({exc})", - ) + return CheckResult( + name=f"metric.{category}.{metric_name}", + category="metric", + passed=False, + message=f"{metric_name}: query failed ({exc})", ) @@ -1069,6 +1193,131 @@ async def validate_parity_span_attrs( ) +def _series_label(series: dict[str, Any]) -> str: + """Name a Prometheus series for use in a failure message. + + Args: + series: One entry from a Prometheus query result. + + Returns: + The series' service_instance_id when it carries one (the label that + tells harness cluster nodes apart), else its full label set. + """ + metric = series.get("metric", {}) + instance = metric.get("service_instance_id") + if instance: + return f"service_instance_id={instance}" + return str(metric) if metric else "" + + +def _value_in_bounds( + value: float, lo: float, hi: float | None, exclusive_lo: bool +) -> bool: + """Test one sample against a sanity range. + + Args: + value: Sample value. + lo: Lower bound. + hi: Upper bound, or None when unbounded above. + exclusive_lo: True when the lower bound is exclusive. + + Returns: + True when the value is inside the range. + """ + lo_ok = value > lo if exclusive_lo else value >= lo + return lo_ok and (hi is None or value <= hi) + + +def _bounds_description(lo: float, hi: float | None, exclusive_lo: bool) -> str: + """Build the human-readable bound text used in check messages. + + Args: + lo: Lower bound. + hi: Upper bound, or None when unbounded above. + exclusive_lo: True when the lower bound is exclusive. + + Returns: + A phrase such as "> 0 and <= 100". + """ + desc = f"{'>' if exclusive_lo else '>='} {lo}" + if hi is not None: + desc += f" and <= {hi}" + return desc + + +async def _check_parity_value( + session: aiohttp.ClientSession, + prometheus_url: str, + entry: dict[str, Any], +) -> CheckResult: + """Bounds-check every series returned by one parity sanity query. + + Args: + session: aiohttp client session. + prometheus_url: Prometheus API base URL. + entry: One PARITY_VALUE_SANITY entry. + + Returns: + A CheckResult that fails if any series is out of bounds, naming each + offending series. + """ + name = entry["name"] + lo = entry["lo"] + hi = entry["hi"] + exclusive_lo = entry.get("exclusive_lo", False) + check_name = f"parity.value_sanity.{name}" + + try: + async with session.get( + f"{prometheus_url}/api/v1/query", params={"query": entry["query"]} + ) as resp: + data = await resp.json() + results = data.get("data", {}).get("result", []) + + if not results: + return CheckResult( + name=check_name, + category="parity", + passed=False, + message=f"{name}: no data returned from Prometheus", + ) + + values: list[float] = [] + offenders: list[str] = [] + for series in results: + value = float(series["value"][1]) + values.append(value) + if not _value_in_bounds(value, lo, hi, exclusive_lo): + offenders.append(f"{_series_label(series)} value {value}") + + bound_desc = _bounds_description(lo, hi, exclusive_lo) + return CheckResult( + name=check_name, + category="parity", + passed=not offenders, + message=( + f"{name}: all {len(values)} series within bounds ({bound_desc})" + if not offenders + else f"{name}: {len(offenders)} of {len(values)} series out of " + f"bounds (expected {bound_desc}): " + "; ".join(offenders) + ), + details={ + "values": values, + "series_count": len(values), + "out_of_bounds": offenders, + "lo": lo, + "hi": hi, + }, + ) + except Exception as exc: + return CheckResult( + name=check_name, + category="parity", + passed=False, + message=f"{name}: sanity check failed ({exc})", + ) + + async def validate_parity_value_sanity( session: aiohttp.ClientSession, prometheus_url: str, @@ -1076,8 +1325,11 @@ async def validate_parity_value_sanity( ) -> None: """Validate that external-parity metric values fall within sane bounds. - For each entry in PARITY_VALUE_SANITY, queries the current value from - Prometheus and checks it against the specified [lo, hi] range. + For each entry in PARITY_VALUE_SANITY, queries Prometheus and checks + *every* returned series against the specified [lo, hi] range. These + queries are bare selectors with no aggregation, so a multi-node harness + cluster returns one series per service_instance_id; checking only the + first would let an out-of-range node pass silently. Args: session: aiohttp client session. @@ -1087,73 +1339,7 @@ async def validate_parity_value_sanity( logger.info("--- External Parity: Value Sanity Checks ---") for entry in PARITY_VALUE_SANITY: - name = entry["name"] - query = entry["query"] - lo = entry["lo"] - hi = entry["hi"] - exclusive_lo = entry.get("exclusive_lo", False) - check_name = f"parity.value_sanity.{name}" - - try: - params = {"query": query} - async with session.get( - f"{prometheus_url}/api/v1/query", params=params - ) as resp: - data = await resp.json() - results = data.get("data", {}).get("result", []) - - if not results: - report.add( - CheckResult( - name=check_name, - category="parity", - passed=False, - message=f"{name}: no data returned from Prometheus", - ) - ) - continue - - # Use the first result's value. - value = float(results[0]["value"][1]) - - # Check bounds. - in_range = True - if exclusive_lo: - in_range = in_range and (value > lo) - else: - in_range = in_range and (value >= lo) - if hi is not None: - in_range = in_range and (value <= hi) - - # Build human-readable bound description. - lo_op = ">" if exclusive_lo else ">=" - bound_desc = f"{lo_op} {lo}" - if hi is not None: - bound_desc += f" and <= {hi}" - - report.add( - CheckResult( - name=check_name, - category="parity", - passed=in_range, - message=( - f"{name}: value {value} is within bounds ({bound_desc})" - if in_range - else f"{name}: value {value} out of bounds " - f"(expected {bound_desc})" - ), - details={"value": value, "lo": lo, "hi": hi}, - ) - ) - except Exception as exc: - report.add( - CheckResult( - name=check_name, - category="parity", - passed=False, - message=f"{name}: sanity check failed ({exc})", - ) - ) + report.add(await _check_parity_value(session, prometheus_url, entry)) # --------------------------------------------------------------------------- diff --git a/docker/telemetry/workload/workload_orchestrator.py b/docker/telemetry/workload/workload_orchestrator.py index d9697d4943..5978c28e20 100755 --- a/docker/telemetry/workload/workload_orchestrator.py +++ b/docker/telemetry/workload/workload_orchestrator.py @@ -53,6 +53,28 @@ logger = logging.getLogger("workload_orchestrator") SCRIPT_DIR = Path(__file__).parent.resolve() PROFILES_FILE = SCRIPT_DIR / "workload-profiles.json" +# Wall-clock allowance for a generator on top of its phase's configured +# duration. It has to cover the work the generators do outside their timed +# loop: tx_submitter.py creates and funds 8 accounts (~25 WebSocket round +# trips) and then waits a fixed 10s for those funding transactions to +# validate, and both generators drain in-flight requests while shutting down. +# A generator that outruns this is killed and the phase records the timeout as +# an error, so one wedged process can no longer stall the whole profile. +SUBPROCESS_GRACE_SEC = 90.0 + +# How long to keep reading a killed process's output before giving up on it. +SUBPROCESS_DRAIN_TIMEOUT_SEC = 10.0 + +# Read size for the pipe readers. Only bounds one read() call, not the total. +PIPE_READ_CHUNK_BYTES = 65536 + +# Error-rate ceilings for the exit gate. The TX ceiling is higher because +# short-lived CI test environments lack pre-funded accounts, causing expected +# failures for complex transactions (AMMCreate, EscrowFinish, etc.) that +# require specific ledger state. +RPC_ERROR_RATE_LIMIT_PCT = 50.0 +TX_ERROR_RATE_LIMIT_PCT = 95.0 + # --------------------------------------------------------------------------- # Data classes @@ -146,15 +168,45 @@ def load_profile(profile_name: str) -> dict[str, Any]: # --------------------------------------------------------------------------- -async def run_subprocess(cmd: list[str], label: str) -> tuple[int, str, str]: - """Run a subprocess and capture its stdout and stderr. +async def _accumulate(stream: asyncio.StreamReader, chunks: list[bytes]) -> None: + """Read a subprocess pipe to EOF, appending as it goes. + + Appending to a caller-owned list, rather than returning at EOF, means + everything read so far survives even if this task never reaches EOF. Args: - cmd: Command and arguments. - label: Human-readable label for logging. + stream: Pipe to read. + chunks: List the caller reads once the process has exited. + """ + while True: + chunk = await stream.read(PIPE_READ_CHUNK_BYTES) + if not chunk: + return + chunks.append(chunk) + + +async def run_subprocess( + cmd: list[str], label: str, timeout: float +) -> tuple[int, str, str]: + """Run a subprocess to completion, or kill it once ``timeout`` expires. + + A generator that wedges used to block its phase — and so the rest of the + profile — until something outside the orchestrator killed the whole run, + destroying the report with it. Bounding the wait lets the orchestrator kill + the process, keep the output it had already produced, and report the phase + as failed. + + Both pipes are drained by separate tasks for the process's whole life, so a + chatty generator can never fill a pipe buffer and stall waiting to write. + + Args: + cmd: Command and arguments. + label: Human-readable label for logging. + timeout: Wall-clock limit in seconds. Returns: - Tuple of (return_code, stdout_text, stderr_text). + Tuple of (return_code, stdout_text, stderr_text). On timeout the return + code is non-zero and the timeout is appended to the stderr text. """ logger.debug("Starting %s: %s", label, " ".join(cmd)) proc = await asyncio.create_subprocess_exec( @@ -162,15 +214,49 @@ async def run_subprocess(cmd: list[str], label: str) -> tuple[int, str, str]: stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, ) - stdout, stderr = await proc.communicate() - if proc.returncode != 0: + + out_chunks: list[bytes] = [] + err_chunks: list[bytes] = [] + readers = [ + asyncio.create_task(_accumulate(proc.stdout, out_chunks)), + asyncio.create_task(_accumulate(proc.stderr, err_chunks)), + ] + + timed_out = False + try: + # asyncio.wait_for raises the builtin TimeoutError on Python 3.11+. + await asyncio.wait_for(proc.wait(), timeout=timeout) + except TimeoutError: + timed_out = True + logger.error("%s exceeded its %.0fs budget — killing it", label, timeout) + proc.kill() + try: + await asyncio.wait_for(proc.wait(), timeout=SUBPROCESS_DRAIN_TIMEOUT_SEC) + except TimeoutError: + logger.error("%s did not exit after being killed", label) + + # The pipes reach EOF once the process is gone, which ends both readers. + _, pending = await asyncio.wait(readers, timeout=SUBPROCESS_DRAIN_TIMEOUT_SEC) + for task in pending: + logger.error("%s output pipe stayed open — captured output truncated", label) + task.cancel() + + stderr_text = b"".join(err_chunks).decode(errors="replace") + if timed_out: + # Appended, not prepended: callers keep only the tail of stderr. + stderr_text += f"\ntimed out after {timeout:.0f}s and was killed" + + # A process whose exit was never collected reports no code; call it SIGKILL + # so the status is still non-zero and the phase records an error. + returncode = proc.returncode if proc.returncode is not None else -9 + if returncode != 0: logger.warning( "%s exited with code %d: %s", label, - proc.returncode, - stderr.decode().strip()[-500:], + returncode, + stderr_text.strip()[-500:], ) - return proc.returncode, stdout.decode(), stderr.decode() + return returncode, b"".join(out_chunks).decode(errors="replace"), stderr_text # --------------------------------------------------------------------------- @@ -259,6 +345,49 @@ def _build_tx_cmd( return cmd +def _launch_phase_tasks( + phase: dict[str, Any], + endpoints: list[str], + report_dir: Path, + prefix: str, +) -> list[tuple[str, Path, asyncio.Task]]: + """Start the generators this phase configures. + + Each generator is given the phase duration plus SUBPROCESS_GRACE_SEC, so a + wedged one is killed instead of stalling the phase. + + Args: + phase: Phase dict from the profile. + endpoints: List of WebSocket endpoint URLs. + report_dir: Directory for per-phase JSON reports. + prefix: Report filename prefix for this phase. + + Returns: + List of (label, report_path, task) for every generator started; empty + when the phase configures no workload. + """ + name = phase["name"] + duration = phase["duration_sec"] + timeout = duration + SUBPROCESS_GRACE_SEC + tasks: list[tuple[str, Path, asyncio.Task]] = [] + + rpc_cfg = phase.get("rpc") + if rpc_cfg: + rpc_out = report_dir / f"{prefix}-rpc.json" + cmd = _build_rpc_cmd(endpoints, rpc_cfg, duration, rpc_out) + task = asyncio.create_task(run_subprocess(cmd, f"RPC [{name}]", timeout)) + tasks.append(("rpc", rpc_out, task)) + + tx_cfg = phase.get("tx") + if tx_cfg: + tx_out = report_dir / f"{prefix}-tx.json" + cmd = _build_tx_cmd(endpoints[0], tx_cfg, duration, tx_out) + task = asyncio.create_task(run_subprocess(cmd, f"TX [{name}]", timeout)) + tasks.append(("tx", tx_out, task)) + + return tasks + + async def run_phase( phase: dict[str, Any], endpoints: list[str], @@ -292,24 +421,8 @@ async def run_phase( phase.get("description", ""), ) - tasks: list[tuple[str, Path, asyncio.Task]] = [] t0 = time.monotonic() - - rpc_cfg = phase.get("rpc") - if rpc_cfg: - rpc_out = report_dir / f"{prefix}-rpc.json" - cmd = _build_rpc_cmd(endpoints, rpc_cfg, duration, rpc_out) - tasks.append( - ("rpc", rpc_out, asyncio.create_task(run_subprocess(cmd, f"RPC [{name}]"))) - ) - - tx_cfg = phase.get("tx") - if tx_cfg: - tx_out = report_dir / f"{prefix}-tx.json" - cmd = _build_tx_cmd(endpoints[0], tx_cfg, duration, tx_out) - tasks.append( - ("tx", tx_out, asyncio.create_task(run_subprocess(cmd, f"TX [{name}]"))) - ) + tasks = _launch_phase_tasks(phase, endpoints, report_dir, prefix) if not tasks: logger.warning( @@ -406,6 +519,58 @@ async def run_profile( return report +# --------------------------------------------------------------------------- +# Exit gate +# --------------------------------------------------------------------------- + + +def evaluate_exit_gate(report: dict[str, Any]) -> list[str]: + """Decide whether a finished run should fail, and say why. + + Three independent conditions fail a run: + * a phase recorded an error — a generator exited non-zero, was killed on + timeout, or wrote a report that could not be parsed, + * the RPC error rate exceeded RPC_ERROR_RATE_LIMIT_PCT, + * the TX error rate exceeded TX_ERROR_RATE_LIMIT_PCT. + + The phase errors have to be judged separately from the two rates. A + generator that crashes writes no report, so its totals stay 0, both rates + short-circuit to 0, and a rate-only gate passes a run in which no traffic + was generated at all. + + Args: + report: Combined report produced by run_profile. + + Returns: + One human-readable reason per failure; empty when the run passed. + """ + reasons: list[str] = [] + + for phase in report.get("phases", []): + for error in phase.get("errors", []): + reasons.append(f"phase '{phase.get('name', '?')}': {error}") + + totals = report.get("totals", {}) + rpc_sent = totals.get("rpc_sent", 0) + tx_submitted = totals.get("tx_submitted", 0) + rpc_err_rate = totals.get("rpc_errors", 0) / rpc_sent * 100 if rpc_sent > 0 else 0.0 + tx_err_rate = ( + totals.get("tx_errors", 0) / tx_submitted * 100 if tx_submitted > 0 else 0.0 + ) + + if rpc_err_rate > RPC_ERROR_RATE_LIMIT_PCT: + reasons.append( + f"RPC error rate {rpc_err_rate:.1f}% exceeds " + f"{RPC_ERROR_RATE_LIMIT_PCT}% of {rpc_sent} requests" + ) + if tx_err_rate > TX_ERROR_RATE_LIMIT_PCT: + reasons.append( + f"TX error rate {tx_err_rate:.1f}% exceeds " + f"{TX_ERROR_RATE_LIMIT_PCT}% of {tx_submitted} submissions" + ) + return reasons + + # --------------------------------------------------------------------------- # CLI entry point # --------------------------------------------------------------------------- @@ -482,23 +647,12 @@ def main() -> None: json.dump(report, f, indent=2) logger.info("Combined report written to %s", args.report) - # Exit with error if either generator had high error rates. - totals = report["totals"] - rpc_err_rate = ( - totals["rpc_errors"] / totals["rpc_sent"] * 100 if totals["rpc_sent"] > 0 else 0 - ) - tx_err_rate = ( - totals["tx_errors"] / totals["tx_submitted"] * 100 - if totals["tx_submitted"] > 0 - else 0 - ) - # TX threshold is higher because short-lived CI test environments lack - # pre-funded accounts, causing expected failures for complex transactions - # (AMMCreate, EscrowFinish, etc.) that require specific ledger state. - if rpc_err_rate > 50 or tx_err_rate > 95: - logger.error( - "High error rates: RPC=%.1f%%, TX=%.1f%%", rpc_err_rate, tx_err_rate - ) + # Fail on any phase error as well as on high error rates. + failures = evaluate_exit_gate(report) + if failures: + logger.error("Workload failed %d gate condition(s):", len(failures)) + for reason in failures: + logger.error(" %s", reason) sys.exit(1) diff --git a/docker/telemetry/workload/xrpld-validator.cfg.template b/docker/telemetry/workload/xrpld-validator.cfg.template index 725faf550c..083c2eaf25 100644 --- a/docker/telemetry/workload/xrpld-validator.cfg.template +++ b/docker/telemetry/workload/xrpld-validator.cfg.template @@ -1,6 +1,10 @@ # xrpld validator node configuration template for workload harness. # -# Placeholders (replaced by docker-compose entrypoint): +# Not consumed by anything today: run-full-validation.sh writes each node's +# cfg inline. Kept as the reference layout for a validator run as a container, +# whose entrypoint would substitute the placeholders below. +# +# Placeholders: # {{NODE_INDEX}} — Node number (1-based) # {{RPC_PORT}} — HTTP RPC port # {{WS_PORT}} — WebSocket port @@ -18,16 +22,19 @@ port_rpc port_ws port_peer +# RPC and WebSocket stay on loopback, matching the cfg that +# run-full-validation.sh generates and the rest of the repo's node configs. +# Only the peer port below needs to listen on all interfaces. [port_rpc] port = {{RPC_PORT}} -ip = 0.0.0.0 -admin = 0.0.0.0 +ip = 127.0.0.1 +admin = 127.0.0.1 protocol = http [port_ws] port = {{WS_PORT}} -ip = 0.0.0.0 -admin = 0.0.0.0 +ip = 127.0.0.1 +admin = 127.0.0.1 protocol = ws [port_peer] diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index 1953138beb..0affd68fc6 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -3253,7 +3253,7 @@ Two more pairs from the same family: cumulative object-payload bytes this process has written — the same value as `node_written_bytes`, from the same accessor — so it excludes keys, padding and the log, and it resets with the process. A ratio of the two is a constant 1.0 and - measures nothing. This label value was called `nudb_bytes` before Phase 9; it + measures nothing. This label value was called `nudb_bytes` in earlier revisions; it comes from `node_store::Database` rather than the NuDB backend, so it is not part of the `nudb_*` family above and reads the same on RocksDB. - These gauges are sampled on the `MetricsRegistry` reader's 10 s cadence, while @@ -3338,7 +3338,7 @@ When telemetry is compiled out, all trace macros expand to no-ops with zero over ## Validating Telemetry Stack -After deploying telemetry, use the Phase 10 workload tools to validate the full stack end-to-end. +After deploying telemetry, use the workload tools in `docker/telemetry/workload/` to validate the full stack end-to-end. ### Quick Validation @@ -3379,13 +3379,13 @@ The counts are not hard-coded in the validator — it iterates the inventory fil so those files are authoritative. The figures below are the inventory as it stands today. -| Category | Checks | Description | -| ---------- | ----------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| Spans | Every **required** entry in `expected_spans.json` — 41 span types at the time of writing: 26 required, 15 marked `"optional": true` | Span name found in Tempo carrying its `required_attributes`, plus the declared parent-child relationships. An `"optional": true` entry that does not fire is recorded as a skip, not a failure — it needs traffic the harness may not generate (HTTP/JSON-RPC client, gRPC client, missing-ledger fetch, mode transitions). | -| Metrics | Every entry in every asserted category of `expected_metrics.json` — 52 metrics in 23 categories at the time of writing | SpanMetrics, `beast::insight` gauges/counters exported over OTLP, and the Phase 9 metrics. Each must have > 0 Prometheus series; none are optional. The separate `not_asserted` group lists metrics deliberately left out of the gate because they are workload-gated or defect-gated; it has no `metrics` key, so the validator skips it. | -| Logs | 2 checks | `trace_id`/`span_id` present in Loki, and a Tempo trace id resolves in Loki. Skipped in CI, which runs `--skip-loki`. | -| Parity | 10 checks | 6 span attributes the external-parity dashboard panels read, plus 4 metric value-sanity bounds. | -| Dashboards | Every uid in `expected_metrics.json` under `grafana_dashboards.uids` — currently all 15 provisioned dashboards | Each listed dashboard loads and reports a panel count. This is a provisioning check only: it does **not** execute the panels' queries, so a dashboard can pass while individual panels render empty. `log-derived-insights` is Loki-backed, so under `--skip-loki` only its provisioning is meaningfully covered. | +| Category | Checks | Description | +| ---------- | ----------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Spans | Every **required** entry in `expected_spans.json` — 41 span types at the time of writing: 26 required, 15 marked `"optional": true` | Span name found in Tempo carrying its `required_attributes`, plus the declared parent-child relationships. An `"optional": true` entry that does not fire is recorded as a skip, not a failure — it needs traffic the harness may not generate (HTTP/JSON-RPC client, gRPC client, missing-ledger fetch, mode transitions). | +| Metrics | Every entry in every asserted category of `expected_metrics.json` — 58 metrics in 23 categories at the time of writing | SpanMetrics, `beast::insight` gauges/counters exported over OTLP, and the `MetricsRegistry` OTLP metrics. Each must have > 0 Prometheus series; none are optional. The separate `not_asserted` group lists metrics deliberately left out of the gate because they are workload-gated or defect-gated; it has no `metrics` key, so the validator skips it. | +| Logs | 2 checks | `trace_id`/`span_id` present in Loki, and a Tempo trace id resolves in Loki. Skipped in CI, which runs `--skip-loki`. | +| Parity | 10 checks | 6 span attributes the external-parity dashboard panels read, plus 4 metric value-sanity bounds. | +| Dashboards | Every uid in `expected_metrics.json` under `grafana_dashboards.uids` — currently all 15 provisioned dashboards | Each listed dashboard loads and reports a panel count. This is a provisioning check only: it does **not** execute the panels' queries, so a dashboard can pass while individual panels render empty. `log-derived-insights` is Loki-backed, so under `--skip-loki` only its provisioning is meaningfully covered. | ### Running Individual Tools