# Telemetry Workload Tools Synthetic workload generation and validation tools for xrpld's OpenTelemetry telemetry stack. These tools validate that all spans, metrics, dashboards, and log-trace correlation work end-to-end under controlled load. ## Quick Start ```bash # Build xrpld with telemetry enabled (see BUILD.md for the full flow) mkdir -p .build && cd .build conan install .. --output-folder . --build missing \ --settings build_type=Release -o telemetry=True cmake -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake \ -DCMAKE_BUILD_TYPE=Release -Dtelemetry=ON .. cmake --build . --parallel "$(nproc)" --target xrpld cd .. # Run full validation (starts everything, runs load, validates) docker/telemetry/workload/run-full-validation.sh --xrpld .build/xrpld # Cleanup when done docker/telemetry/workload/run-full-validation.sh --cleanup ``` ## Architecture The validation suite runs a multi-node xrpld cluster as local processes alongside a Docker Compose telemetry stack. The cluster exercises consensus, peer-to-peer spans (proposals, validations), and all metric pipelines. ``` run-full-validation.sh (shell orchestrator) | |-- docker-compose.workload.yaml | |-- otel-collector (otlp receiver: traces + beast::insight metrics; | | filelog receiver: node debug.log -> Loki) | |-- tempo (trace backend + TraceQL search API) | |-- prometheus (metrics scraping) | |-- loki (log aggregation for log-trace correlation) | |-- grafana (dashboards, provisioned automatically) | |-- generate-validator-keys.sh | -> validator-keys.json, validators.txt | |-- Nx xrpld nodes (local processes, full telemetry) | - Each node: [telemetry] enabled=1, all 5 trace_* categories on | - [insight] server=otel (beast::insight metrics over OTLP, no StatsD) | - [signing_support] true (server-side signing for tx_submitter) | - Peer discovery via [ips] (not [ips_fixed]) for active peer counts | |-- workload_orchestrator.py (phased load execution) | |-- rpc_load_generator.py (WebSocket RPC traffic) | |-- tx_submitter.py (transaction diversity) | -> workload-report.json + per-phase reports | |-- validate_telemetry.py (pass/fail checks) | -> validation-report.json | |-- benchmark.sh (baseline vs telemetry comparison) |-- collect_system_metrics.sh (per-leg CPU/RSS/latency/TPS sampling) -> benchmark-report-*.md ``` ## Workload Profiles The workload orchestrator (`workload_orchestrator.py`) reads named profiles from `workload-profiles.json` and executes sequential load phases. Within each phase, the RPC generator and TX submitter run concurrently. ### Available Profiles | Profile | Phases | Duration | Purpose | | ----------------- | ------ | --------------------------- | ------------------------------------------------------------------------------------------------ | | `full-validation` | 7 | 4.5 min + 1 min propagation | Coverage for the full asserted span/metric/dashboard inventory, with burst/idle/plateau patterns | | `quick-smoke` | 1 | 30s + 30s propagation | Fast CI smoke test | | `stress` | 3 | 3.5 min + 1 min propagation | Heavy sustained load for benchmarking | Durations are the sum of the phase `duration_sec` values in `workload-profiles.json` plus that profile's `propagation_wait_sec`; they exclude cluster startup and the validation pass itself. ### full-validation Phases | Phase | RPC Rate | TX TPS | Duration | Dashboard Coverage | | ------------ | ------------------ | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | warmup | 5 RPS | — | 30s | Node Health, Validator Health (baseline gauges) | | steady-state | 30 RPS | 3 TPS | 60s | All dashboards (plateau data) | | rpc-burst | 100 RPS | — | 30s | Job Queue, RPC Performance (latency spikes) | | tx-flood | 5 RPS | 20 TPS | 30s | Fee Market & TxQ, Transaction Overview | | txq-burst | 5 RPS (100% `fee`) | 60 TPS | 30s | Fee Market & TxQ — single-type Payment burst that forces open-ledger fee escalation and TxQ queueing, exercising the `txq.*` spans (`txq.enqueue`, `txq.accept`, `txq.accept_tx`, `txq.cleanup`) | | mixed-peak | 50 RPS | 10 TPS | 60s | Consensus Health, Ledger Operations | | cooldown | 5 RPS | — | 30s | Recovery patterns, state transitions | ### Custom Profiles Add profiles to `workload-profiles.json`: ```json { "profiles": { "my-custom": { "description": "Custom profile for specific testing", "phases": [ { "name": "phase-name", "description": "What this phase exercises", "duration_sec": 60, "rpc": { "rate": 50, "weights": { "server_info": 80, "fee": 20 } }, "tx": { "tps": 5, "weights": { "Payment": 100 } } } ], "propagation_wait_sec": 30 } } } ``` Set `"rpc"` or `"tx"` to `null` to skip that generator for a phase. Custom `"weights"` override the default command/transaction distribution. ## Tools Reference ### run-full-validation.sh Orchestrates the complete validation pipeline. Starts the telemetry stack, starts a multi-node xrpld cluster, generates load, and validates the results. ```bash # Full validation with defaults (uses full-validation profile) ./run-full-validation.sh --xrpld /path/to/xrpld # Quick smoke test ./run-full-validation.sh --xrpld /path/to/xrpld --profile quick-smoke # Stress test with benchmarks ./run-full-validation.sh --xrpld /path/to/xrpld --profile stress --with-benchmark # Skip Loki checks (if log export is not deployed) ./run-full-validation.sh --xrpld /path/to/xrpld --skip-loki ``` ### workload_orchestrator.py Reads a named profile from `workload-profiles.json` and executes sequential load phases. Within each phase, `rpc_load_generator.py` and `tx_submitter.py` run as concurrent subprocesses. Produces per-phase reports and a combined summary. ```bash # Run with a specific profile python3 workload_orchestrator.py --profile full-validation # Multiple endpoints python3 workload_orchestrator.py --profile full-validation \ --endpoints ws://localhost:6006 ws://localhost:6007 # Save combined report python3 workload_orchestrator.py --profile stress --report /tmp/report.json ``` ### rpc_load_generator.py Generates RPC traffic matching realistic production distribution. Uses xrpld's **native WebSocket command format** (`{"command": ...}`) with flat parameters — the same format as `tx_submitter.py`. - 40% health checks (server_info, fee) - 30% wallet queries (account_info, account_lines, account_objects) - 15% explorer queries (ledger, ledger_data) - 10% transaction lookups (tx, account_tx) - 5% DEX queries (book_offers, amm_info) ```bash # Basic usage python3 rpc_load_generator.py --endpoints ws://localhost:6006 --rate 50 --duration 120 # Multiple endpoints (round-robin) python3 rpc_load_generator.py \ --endpoints ws://localhost:6006 ws://localhost:6007 \ --rate 100 --duration 300 # Custom weights python3 rpc_load_generator.py --endpoints ws://localhost:6006 \ --weights '{"server_info": 80, "account_info": 20}' ``` ### tx_submitter.py Submits diverse transaction types to exercise the full span and metric surface. Uses xrpld's **native WebSocket command format** (`{"command": ...}`) rather than JSON-RPC format. The response payload is inside the `"result"` key, with `"status"` at the top level. Supported transaction types: - Payment (XRP transfers) — exercises `tx.process`, `tx.receive`, `tx.apply` - OfferCreate / OfferCancel (DEX activity) - TrustSet (trust line creation) - NFTokenMint / NFTokenCreateOffer (NFT activity) - EscrowCreate / EscrowFinish (escrow lifecycle) - AMMCreate / AMMDeposit (AMM pool operations) Requires `[signing_support] true` in the node config for server-side signing. ```bash # Basic usage python3 tx_submitter.py --endpoint ws://localhost:6006 --tps 5 --duration 120 # Custom mix python3 tx_submitter.py --endpoint ws://localhost:6006 \ --weights '{"Payment": 60, "OfferCreate": 20, "TrustSet": 20}' ``` ### validate_telemetry.py Automated validation that all expected telemetry data exists. Every metric in `expected_metrics.json` is required — if it doesn't fire, the validation fails. Spans are required unless the entry carries `"optional": true`. - **Span validation**: All span types from `expected_spans.json` with required attributes and parent-child hierarchies. Entries marked `"optional": true` only fire under traffic the harness may not produce (HTTP/JSON-RPC client, gRPC client, missing-ledger fetch, mode transitions); their absence is recorded as a passing skip, not a failure. - **Metric validation**: All metrics from `expected_metrics.json` — SpanMetrics, `beast::insight` gauges/counters/histograms, `MetricsRegistry` OTLP metrics. Every listed metric must have > 0 series. Uses the Prometheus `/api/v1/series` endpoint (not instant queries), polled until the metric appears or the poll window elapses, so a late-populating or quiet series is not a false negative. - **Log-trace correlation**: trace_id/span_id in Loki logs (requires Loki) - **Dashboard validation**: Every dashboard uid listed under `grafana_dashboards.uids` in `expected_metrics.json` loads with panels. That list currently covers **all 16** dashboards provisioned in `docker/telemetry/grafana/dashboards/`. Note the scope of this check: it asks the Grafana API whether the dashboard exists and returns a panel count — it does **not** run the panels' queries, so a dashboard can pass here while individual panels render empty. ```bash # Run all validations python3 validate_telemetry.py --report /tmp/report.json # Skip Loki checks python3 validate_telemetry.py --skip-loki --report /tmp/report.json ``` ### OTel Timings Regression Gate `capture_timings.py` + `compare_to_baseline.py` implement a regression gate that compares OTel-derived per-span/per-RPC/per-job timings against a committed baseline. Unlike `benchmark.sh` (which measures the overhead of enabling telemetry on the current binary), this gate catches **xrpld performance regressions over time** by diffing against a stored baseline from a prior run. How it runs inside the validation pipeline: 1. `run-full-validation.sh` executes the normal workload and validation suite. 2. After validation, `capture_timings.py` queries Prometheus for every metric in `regression-metrics.json` and writes `reports/timings.json`. 3. `compare_to_baseline.py` reads `timings.json`, `baselines/baseline-timings.json`, and `regression-thresholds.json`, then either: - Prints the paste-me JSON block (when the baseline is a placeholder or empty) and exits 0. - Prints a delta table, writes `reports/regression-report.json`, and exits non-zero if any metric breached both the percentage AND absolute bound. Bootstrapping a baseline: 1. Push the branch. The `Telemetry Validation` CI run prints the full timings JSON under "Paste into `baselines/baseline-timings.json`" in the workflow Step Summary. 2. Open a PR copying that JSON block verbatim into `baselines/baseline-timings.json`. Reviewer approval is the audit gate. 3. Subsequent runs compare against it; the gate fails on regression. Per-run tuning: - `--skip-regression` disables the gate (local exploration only). - `REGRESSION_WINDOW` env var overrides the default Prometheus `rate()` window (`3m`). Keep close to the workload duration. - Metric surface lives in `regression-metrics.json`; thresholds in `regression-thresholds.json`; both are reviewed changes. See [`baselines/README.md`](./baselines/README.md) for the baseline lifecycle and refresh process. ### benchmark.sh Compares baseline (no telemetry) vs telemetry-enabled performance: ```bash ./benchmark.sh --xrpld /path/to/xrpld --duration 300 ``` Thresholds (configurable via environment): | Metric | Threshold | Env Variable | | ----------------- | --------- | --------------------------- | | CPU overhead | < 3% | BENCH_CPU_OVERHEAD_PCT | | Memory overhead | < 5MB | BENCH_MEM_OVERHEAD_MB | | RPC p99 latency | < 2ms | BENCH_RPC_LATENCY_IMPACT_MS | | Throughput impact | < 5% | BENCH_TPS_IMPACT_PCT | | Consensus impact | < 1% | BENCH_CONSENSUS_IMPACT_PCT | Each report row is `PASS`, `FAIL`, or `INCONCLUSIVE`. The throughput and consensus rows are ratios of the baseline, so they have nothing to report when the baseline run measured zero — that row becomes `INCONCLUSIVE` and **counts as a failure**, because an undefined result must never read as a pass. Exit codes: | Code | Meaning | | ---- | --------------------------------------------------------------------------------------------------------------------------- | | 0 | Every metric was measured and is within its threshold | | 1 | Every metric was measured and at least one exceeded its threshold | | 2 | The overhead could not be measured — missing prerequisite, cluster never reached consensus, or incomplete metric collection | `run-full-validation.sh` keeps the last two apart: 1 folds into its own "checks failed" exit, 2 into its "infrastructure error" exit. A run that measured nothing is therefore never reported as a performance regression. ### collect_system_metrics.sh Samples CPU, peak RSS, RPC p99 latency, TPS and the mean inter-ledger interval from the running nodes, and writes them as JSON. `benchmark.sh` calls it once per leg; it is rarely run by hand. ```bash ./collect_system_metrics.sh 5020,5021,5022 300 /tmp/metrics.json ``` Processes are selected by matching `argv[0]`'s basename against the daemon binary name; the pre-rename spelling is accepted too, so the sampler still works against an older deployment. A wrapper that merely names the binary in its arguments, and unrelated tools whose command line happens to contain the string, are not sampled — including them diluted the CPU average and attributed a foreign process's RSS to the node. `ps -C xrpld` is not usable for this: xrpld renames itself, so its `comm` is `xrpld-main`. Selection covers the whole host, so a second xrpld from another checkout is sampled as well. Benchmark on a machine running one cluster only. The output carries a `metrics_complete` flag. It is `false` when any measurement source came back empty — no matching process, no successful RPC probe, or a ledger sequence that never advanced — and the affected metrics are then `0` placeholders. Since `0` clears every threshold, a `false` flag must be read as inconclusive, never as a pass. Exit codes: | Code | Meaning | | ---- | --------------------------------------------------------------------------------------------------------- | | 0 | Every metric was measured; `metrics_complete` is `true` | | 1 | Cannot run: bad arguments, no GNU `date` with `%N`, or a failed process sample. No output file is written | | 3 | The output file was written, but `metrics_complete` is `false` | `benchmark.sh` treats either non-zero code — and an explicit `"metrics_complete": false` in an otherwise successful run — as fatal, and exits 2 rather than comparing an incomplete run. A nanosecond clock is required. RPC latency is graded against a 2 ms threshold, and GNU `date +%s%N` is the only source cheap enough that the clock does not dominate what it measures, so the script refuses to start without it. ## Reading Validation Reports The validation report (`validation-report.json`) is structured as follows. The counts below are illustrative — the real total is the sum of the span, metric, log, dashboard and parity checks for the run. ```json { "summary": { "total": 45, "passed": 42, "failed": 3, "all_passed": false }, "checks": [ { "name": "span.rpc.ws_message", "category": "span", "passed": true, "message": "rpc.ws_message: 15 traces found", "details": { "trace_count": 15 } } ] } ``` Categories: - **span**: Span type existence and attribute validation - **metric**: Prometheus metric existence - **log**: Log-trace correlation checks - **dashboard**: Grafana dashboard accessibility - **parity**: Span attributes required by the external-parity dashboard panels (validator-health, peer-quality, and friends) ## CI Integration The validation runs as a GitHub Actions workflow (`.github/workflows/telemetry-validation.yml`): - Triggered manually (`workflow_dispatch`) or on pushes to telemetry branches. There is no cron schedule. - Builds xrpld, starts the full stack, runs load, validates - Uploads reports as artifacts (and node logs when validation did not succeed) - Writes the validation summary and the regression-gate summary to the workflow **Step Summary** (`$GITHUB_STEP_SUMMARY`). It does **not** comment on the PR — the workflow declares no `permissions:` block and calls no GitHub API, so read the summary on the run page. Of the five `workflow_dispatch` inputs, only `run_benchmark` changes behaviour. `rpc_rate`, `rpc_duration`, `tx_tps` and `tx_duration` are forwarded to `run-full-validation.sh`, which parses them into shell variables and never reads them again — load shape comes entirely from `--profile` and `workload-profiles.json`. Their `description:` fields say so. ## Configuration Files | File | Purpose | | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | `workload-profiles.json` | Named load profiles with phase definitions | | `expected_spans.json` | Span inventory (names, attributes, hierarchies, config flags) | | `expected_metrics.json` | Metric inventory — every listed metric must be present — plus the `grafana_dashboards.uids` list the dashboard check iterates | | `test_accounts.json` | Test account roles (keys generated at runtime) | | `regression-metrics.json` | Metric surface for the OTel regression gate | | `regression-thresholds.json` | Per-metric regression bounds (pct AND abs) | | `baselines/baseline-timings.json` | Committed baseline — populated from first CI run | | `requirements.txt` | Python dependencies | ### expected_metrics.json Format ```json { "description": "Top-level doc string — skipped by the validator.", "category_name": { "description": "Human-readable description.", "metrics": ["metric_1", "metric_2"] }, "grafana_dashboards": { "uids": ["rpc-performance", "node-health"] }, "not_asserted": { "description": "Why these are excluded.", "metrics_excluded": { "metric_3": "reason" } } } ``` Every metric listed under a `metrics` array must produce > 0 Prometheus series during the validation run. If a metric doesn't fire, the workload generators need to produce enough load to trigger it. Three top-level keys are not metric categories: - `description` and `grafana_dashboards` are skipped explicitly by `validate_metrics`. `grafana_dashboards.uids` drives the dashboard check, so adding a dashboard to `docker/telemetry/grafana/dashboards/` does **not** put it under the gate until its uid is added here too. - `not_asserted` is skipped structurally: the loop reads `category_data.get("metrics", [])`, and this group deliberately has no `metrics` key — its entries live under `metrics_excluded` as a name-to-reason map. It documents metrics that are emitted and dashboarded but left unasserted because they are workload-gated or defect-gated (a check that fails on a healthy run is worse than no check). Promote an entry into an asserted group only after the workload is changed to guarantee it fires. ### expected_spans.json Format Each span entry defines its name, category, parent (for hierarchy validation), required attributes, and the `config_flag` that must be enabled. A trailing `*` in `name` is a wildcard. The optional `"optional": true` field marks a span whose absence is a skip rather than a failure: ```json { "name": "rpc.command.*", "category": "rpc", "parent": "rpc.process", "required_attributes": ["command", "version", "rpc_role", "rpc_status"], "config_flag": "trace_rpc" } ``` ## Node Configuration Notes The orchestrator (`run-full-validation.sh`) generates node configs with: - `[telemetry] enabled=1` with all five trace categories: `trace_rpc`, `trace_transactions`, `trace_consensus`, `trace_peer`, `trace_ledger` - `[insight] server=otel` with `endpoint=http://localhost:4318/v1/metrics` and `prefix=xrpld` — `beast::insight` metrics reach Prometheus over OTLP, because the collector declares no `statsd` receiver - `[signing_support] true` — required for `tx_submitter.py` to submit signed transactions via WebSocket - `[ips]` (not `[ips_fixed]`) — ensures peer connections are counted in the PeerFinder active-peer gauges, exported as `peer_finder_active_inbound_peers` / `peer_finder_active_outbound_peers` (fixed peers are excluded from these counters by design). The `beast::insight` group/name pair is `Peer_Finder` / `Active_Inbound_Peers`; `formatName()` lowercases it for export. ## Gauge Export Behaviour The harness configures each node with `[insight] server=otel` (see the `[insight]` block generated by `run-full-validation.sh`), so `beast::insight` gauges go through `OTelGaugeImpl` in `src/libxrpl/beast/insight/OTelCollector.cpp`, not through the StatsD collector. That matters for how the validator queries Prometheus. **How `OTelGaugeImpl` exports.** It wraps an OTel **observable** (asynchronous) gauge. `set()` and `increment()` only store into an `std::atomic`; nothing is exported at call time. The SDK's collection thread invokes `gaugeCallback`, which runs the collector's hooks and then `Observe()`s whatever the atomic currently holds. So the gauge reports **every collection cycle, whether or not the value changed** — including a gauge that sits at 0 from startup. There is no dirty flag on this path, and no first-flush special case is needed. **Why the validator still uses `/api/v1/series`.** Two reasons survive the move to OTLP: 1. **Late-populating series.** A gauge or counter may not have completed the export → collector → Prometheus-scrape pipeline by the time validation runs. `_check_prometheus_metric` in `validate_telemetry.py` therefore polls `/api/v1/series` (which returns anything that existed anywhere in the query window) until the metric appears or the poll window elapses, instead of racing a single instant query. 2. **Staleness robustness.** `/api/v1/series` does not care whether the newest sample is inside Prometheus's ~5-minute staleness horizon, so the check cannot be defeated by a quiet series. > **Note — the StatsD path is still in the tree but unused here.** If a node is > configured with `server=statsd`, `StatsDGaugeImpl` (in > `src/libxrpl/beast/insight/StatsDCollector.cpp`) does gate emission on a > `dirty_` flag that is only set by `set()`/`increment()`, and it is > initialised to `true` so the initial value is emitted on the first flush. The > collector configs shipped in `docker/telemetry/` declare no `statsd` receiver > (the metrics pipeline is `[otlp, spanmetrics]`) and the base > `docker-compose.yml` keeps its StatsD UDP port commented out, so nothing in > this harness can receive StatsD.