rippled/docker/telemetry/workload

Telemetry Workload Tools

Synthetic workload generation and validation tools for xrpld's OpenTelemetry telemetry stack. These tools validate that all spans, metrics, dashboards, and log-trace correlation work end-to-end under controlled load.

Quick Start

# Build xrpld with telemetry enabled (see BUILD.md for the full flow)
mkdir -p .build && cd .build
conan install .. --output-folder . --build missing \
    --settings build_type=Release -o telemetry=True
cmake -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake \
    -DCMAKE_BUILD_TYPE=Release -Dtelemetry=ON ..
cmake --build . --parallel "$(nproc)" --target xrpld
cd ..

# Run full validation (starts everything, runs load, validates)
docker/telemetry/workload/run-full-validation.sh --xrpld .build/xrpld

# Cleanup when done
docker/telemetry/workload/run-full-validation.sh --cleanup

Architecture

The validation suite runs a multi-node xrpld cluster as local processes alongside a Docker Compose telemetry stack. The cluster exercises consensus, peer-to-peer spans (proposals, validations), and all metric pipelines.

run-full-validation.sh (shell orchestrator)
  |
  |-- docker-compose.workload.yaml
  |     |-- otel-collector (otlp receiver: traces + beast::insight metrics;
  |     |                  filelog receiver: node debug.log -> Loki)
  |     |-- tempo (trace backend + TraceQL search API)
  |     |-- prometheus (metrics scraping)
  |     |-- loki (log aggregation for log-trace correlation)
  |     |-- grafana (dashboards, provisioned automatically)
  |
  |-- generate-validator-keys.sh
  |     -> validator-keys.json, validators.txt
  |
  |-- Nx xrpld nodes (local processes, full telemetry)
  |     - Each node: [telemetry] enabled=1, all 5 trace_* categories on
  |     - [insight] server=otel (beast::insight metrics over OTLP, no StatsD)
  |     - [signing_support] true (server-side signing for tx_submitter)
  |     - Peer discovery via [ips] (not [ips_fixed]) for active peer counts
  |
  |-- workload_orchestrator.py (phased load execution)
  |     |-- rpc_load_generator.py (WebSocket RPC traffic)
  |     |-- tx_submitter.py (transaction diversity)
  |     -> workload-report.json + per-phase reports
  |
  |-- validate_telemetry.py (pass/fail checks)
  |     -> validation-report.json
  |
  |-- benchmark.sh (baseline vs telemetry comparison)
        |-- collect_system_metrics.sh (per-leg CPU/RSS/latency/TPS sampling)
        -> benchmark-report-*.md

Workload Profiles

The workload orchestrator (workload_orchestrator.py) reads named profiles from workload-profiles.json and executes sequential load phases. Within each phase, the RPC generator and TX submitter run concurrently.

Available Profiles

Profile	Phases	Duration	Purpose
full-validation	7	4.5 min + 1 min propagation	Coverage for the full asserted span/metric/dashboard inventory, with burst/idle/plateau patterns
quick-smoke	1	30s + 30s propagation	Fast CI smoke test
stress	3	3.5 min + 1 min propagation	Heavy sustained load for benchmarking

Durations are the sum of the phase duration_sec values in workload-profiles.json plus that profile's propagation_wait_sec; they exclude cluster startup and the validation pass itself.

full-validation Phases

Phase	RPC Rate	TX TPS	Duration	Dashboard Coverage
warmup	5 RPS	—	30s	Node Health, Validator Health (baseline gauges)
steady-state	30 RPS	3 TPS	60s	All dashboards (plateau data)
rpc-burst	100 RPS	—	30s	Job Queue, RPC Performance (latency spikes)
tx-flood	5 RPS	20 TPS	30s	Fee Market & TxQ, Transaction Overview
txq-burst	5 RPS (100% fee)	60 TPS	30s	Fee Market & TxQ — single-type Payment burst that forces open-ledger fee escalation and TxQ queueing, exercising the txq.* spans (txq.enqueue, txq.accept, txq.accept_tx, txq.cleanup)
mixed-peak	50 RPS	10 TPS	60s	Consensus Health, Ledger Operations
cooldown	5 RPS	—	30s	Recovery patterns, state transitions

Custom Profiles

Add profiles to workload-profiles.json:

{
  "profiles": {
    "my-custom": {
      "description": "Custom profile for specific testing",
      "phases": [
        {
          "name": "phase-name",
          "description": "What this phase exercises",
          "duration_sec": 60,
          "rpc": { "rate": 50, "weights": { "server_info": 80, "fee": 20 } },
          "tx": { "tps": 5, "weights": { "Payment": 100 } }
        }
      ],
      "propagation_wait_sec": 30
    }
  }
}

Set "rpc" or "tx" to null to skip that generator for a phase. Custom "weights" override the default command/transaction distribution.

Tools Reference

run-full-validation.sh

Orchestrates the complete validation pipeline. Starts the telemetry stack, starts a multi-node xrpld cluster, generates load, and validates the results.

# Full validation with defaults (uses full-validation profile)
./run-full-validation.sh --xrpld /path/to/xrpld

# Quick smoke test
./run-full-validation.sh --xrpld /path/to/xrpld --profile quick-smoke

# Stress test with benchmarks
./run-full-validation.sh --xrpld /path/to/xrpld --profile stress --with-benchmark

# Skip Loki checks (if log export is not deployed)
./run-full-validation.sh --xrpld /path/to/xrpld --skip-loki

workload_orchestrator.py

Reads a named profile from workload-profiles.json and executes sequential load phases. Within each phase, rpc_load_generator.py and tx_submitter.py run as concurrent subprocesses. Produces per-phase reports and a combined summary.

# Run with a specific profile
python3 workload_orchestrator.py --profile full-validation

# Multiple endpoints
python3 workload_orchestrator.py --profile full-validation \
    --endpoints ws://localhost:6006 ws://localhost:6007

# Save combined report
python3 workload_orchestrator.py --profile stress --report /tmp/report.json

rpc_load_generator.py

Generates RPC traffic matching realistic production distribution. Uses xrpld's native WebSocket command format ({"command": ...}) with flat parameters — the same format as tx_submitter.py.

• 40% health checks (server_info, fee)
• 30% wallet queries (account_info, account_lines, account_objects)
• 15% explorer queries (ledger, ledger_data)
• 10% transaction lookups (tx, account_tx)
• 5% DEX queries (book_offers, amm_info)

# Basic usage
python3 rpc_load_generator.py --endpoints ws://localhost:6006 --rate 50 --duration 120

# Multiple endpoints (round-robin)
python3 rpc_load_generator.py \
    --endpoints ws://localhost:6006 ws://localhost:6007 \
    --rate 100 --duration 300

# Custom weights
python3 rpc_load_generator.py --endpoints ws://localhost:6006 \
    --weights '{"server_info": 80, "account_info": 20}'

tx_submitter.py

Submits diverse transaction types to exercise the full span and metric surface. Uses xrpld's native WebSocket command format ({"command": ...}) rather than JSON-RPC format. The response payload is inside the "result" key, with "status" at the top level.

Supported transaction types:

• Payment (XRP transfers) — exercises tx.process, tx.receive, tx.apply
• OfferCreate / OfferCancel (DEX activity)
• TrustSet (trust line creation)
• NFTokenMint / NFTokenCreateOffer (NFT activity)
• EscrowCreate / EscrowFinish (escrow lifecycle)
• AMMCreate / AMMDeposit (AMM pool operations)

Requires [signing_support] true in the node config for server-side signing.

# Basic usage
python3 tx_submitter.py --endpoint ws://localhost:6006 --tps 5 --duration 120

# Custom mix
python3 tx_submitter.py --endpoint ws://localhost:6006 \
    --weights '{"Payment": 60, "OfferCreate": 20, "TrustSet": 20}'

validate_telemetry.py

Automated validation that all expected telemetry data exists. Every metric in expected_metrics.json is required — if it doesn't fire, the validation fails. Spans are required unless the entry carries "optional": true.

• Span validation: All span types from expected_spans.json with required attributes and parent-child hierarchies. Entries marked "optional": true only fire under traffic the harness may not produce (HTTP/JSON-RPC client, gRPC client, missing-ledger fetch, mode transitions); their absence is recorded as a passing skip, not a failure.
• Metric validation: All metrics from expected_metrics.json — SpanMetrics, beast::insight gauges/counters/histograms, MetricsRegistry OTLP metrics. Every listed metric must have > 0 series. Uses the Prometheus /api/v1/series endpoint (not instant queries), polled until the metric appears or the poll window elapses, so a late-populating or quiet series is not a false negative.
• Log-trace correlation: trace_id/span_id in Loki logs (requires Loki)
• Dashboard validation: Every dashboard uid listed under grafana_dashboards.uids in expected_metrics.json loads with panels. That list currently covers all 15 dashboards provisioned in docker/telemetry/grafana/dashboards/. Note the scope of this check: it asks the Grafana API whether the dashboard exists and returns a panel count — it does not run the panels' queries, so a dashboard can pass here while individual panels render empty.

# Run all validations
python3 validate_telemetry.py --report /tmp/report.json

# Skip Loki checks
python3 validate_telemetry.py --skip-loki --report /tmp/report.json

OTel Timings Regression Gate

capture_timings.py + compare_to_baseline.py implement a regression gate that compares OTel-derived per-span/per-RPC/per-job timings against a committed baseline. Unlike benchmark.sh (which measures the overhead of enabling telemetry on the current binary), this gate catches xrpld performance regressions over time by diffing against a stored baseline from a prior run.

How it runs inside the validation pipeline:

1. run-full-validation.sh executes the normal workload and validation suite.
2. After validation, capture_timings.py queries Prometheus for every metric in regression-metrics.json and writes reports/timings.json.
3. compare_to_baseline.py reads timings.json, baselines/baseline-timings.json, and regression-thresholds.json, then either:
   • Prints the paste-me JSON block (when the baseline is a placeholder or empty) and exits 0.
   • Prints a delta table, writes reports/regression-report.json, and exits non-zero if any metric breached both the percentage AND absolute bound.

Bootstrapping a baseline:

1. Push the branch. The Telemetry Validation CI run prints the full timings JSON under "Paste into baselines/baseline-timings.json" in the workflow Step Summary.
2. Open a PR copying that JSON block verbatim into baselines/baseline-timings.json. Reviewer approval is the audit gate.
3. Subsequent runs compare against it; the gate fails on regression.

Per-run tuning:

• --skip-regression disables the gate (local exploration only).
• REGRESSION_WINDOW env var overrides the default Prometheus rate() window (3m). Keep close to the workload duration.
• Metric surface lives in regression-metrics.json; thresholds in regression-thresholds.json; both are reviewed changes.

See baselines/README.md for the baseline lifecycle and refresh process.

benchmark.sh

Compares baseline (no telemetry) vs telemetry-enabled performance:

./benchmark.sh --xrpld /path/to/xrpld --duration 300

Thresholds (configurable via environment):

Metric	Threshold	Env Variable
CPU overhead	< 3%	BENCH_CPU_OVERHEAD_PCT
Memory overhead	< 5MB	BENCH_MEM_OVERHEAD_MB
RPC p99 latency	< 2ms	BENCH_RPC_LATENCY_IMPACT_MS
Throughput impact	< 5%	BENCH_TPS_IMPACT_PCT
Consensus impact	< 1%	BENCH_CONSENSUS_IMPACT_PCT

Each report row is PASS, FAIL, or INCONCLUSIVE. The throughput and consensus rows are ratios of the baseline, so they have nothing to report when the baseline run measured zero — that row becomes INCONCLUSIVE and counts as a failure, because an undefined result must never read as a pass.

Exit codes:

Code	Meaning
0	Every metric was measured and is within its threshold
1	Every metric was measured and at least one exceeded its threshold
2	The overhead could not be measured — missing prerequisite, cluster never reached consensus, or incomplete metric collection

run-full-validation.sh keeps the last two apart: 1 folds into its own "checks failed" exit, 2 into its "infrastructure error" exit. A run that measured nothing is therefore never reported as a performance regression.

collect_system_metrics.sh

Samples CPU, peak RSS, RPC p99 latency, TPS and the mean inter-ledger interval from the running nodes, and writes them as JSON. benchmark.sh calls it once per leg; it is rarely run by hand.

./collect_system_metrics.sh 5020,5021,5022 300 /tmp/metrics.json

Processes are selected by matching argv[0]'s basename against the daemon binary name; the pre-rename spelling is accepted too, so the sampler still works against an older deployment. A wrapper that merely names the binary in its arguments, and unrelated tools whose command line happens to contain the string, are not sampled — including them diluted the CPU average and attributed a foreign process's RSS to the node. ps -C xrpld is not usable for this: xrpld renames itself, so its comm is xrpld-main.

Selection covers the whole host, so a second xrpld from another checkout is sampled as well. Benchmark on a machine running one cluster only.

The output carries a metrics_complete flag. It is false when any measurement source came back empty — no matching process, no successful RPC probe, or a ledger sequence that never advanced — and the affected metrics are then 0 placeholders. Since 0 clears every threshold, a false flag must be read as inconclusive, never as a pass.

Exit codes:

Code	Meaning
0	Every metric was measured; metrics_complete is true
1	Cannot run: bad arguments, no GNU date with %N, or a failed process sample. No output file is written
3	The output file was written, but metrics_complete is false

benchmark.sh treats either non-zero code — and an explicit "metrics_complete": false in an otherwise successful run — as fatal, and exits 2 rather than comparing an incomplete run.

A nanosecond clock is required. RPC latency is graded against a 2 ms threshold, and GNU date +%s%N is the only source cheap enough that the clock does not dominate what it measures, so the script refuses to start without it.

Reading Validation Reports

The validation report (validation-report.json) is structured as follows. The counts below are illustrative — the real total is the sum of the span, metric, log, dashboard and parity checks for the run.

{
  "summary": {
    "total": 45,
    "passed": 42,
    "failed": 3,
    "all_passed": false
  },
  "checks": [
    {
      "name": "span.rpc.ws_message",
      "category": "span",
      "passed": true,
      "message": "rpc.ws_message: 15 traces found",
      "details": { "trace_count": 15 }
    }
  ]
}

Categories:

• span: Span type existence and attribute validation
• metric: Prometheus metric existence
• log: Log-trace correlation checks
• dashboard: Grafana dashboard accessibility
• parity: Span attributes required by the external-parity dashboard panels (validator-health, peer-quality, and friends)

CI Integration

The validation runs as a GitHub Actions workflow (.github/workflows/telemetry-validation.yml):

• Triggered manually (workflow_dispatch) or on pushes to telemetry branches. There is no cron schedule.
• Builds xrpld, starts the full stack, runs load, validates
• Uploads reports as artifacts (and node logs when validation did not succeed)
• Writes the validation summary and the regression-gate summary to the workflow Step Summary ($GITHUB_STEP_SUMMARY). It does not comment on the PR — the workflow declares no permissions: block and calls no GitHub API, so read the summary on the run page.

Of the five workflow_dispatch inputs, only run_benchmark changes behaviour. rpc_rate, rpc_duration, tx_tps and tx_duration are forwarded to run-full-validation.sh, which parses them into shell variables and never reads them again — load shape comes entirely from --profile and workload-profiles.json. Their description: fields say so.

Configuration Files

File	Purpose
workload-profiles.json	Named load profiles with phase definitions
expected_spans.json	Span inventory (names, attributes, hierarchies, config flags)
expected_metrics.json	Metric inventory — every listed metric must be present — plus the grafana_dashboards.uids list the dashboard check iterates
test_accounts.json	Test account roles (keys generated at runtime)
regression-metrics.json	Metric surface for the OTel regression gate
regression-thresholds.json	Per-metric regression bounds (pct AND abs)
baselines/baseline-timings.json	Committed baseline — populated from first CI run
requirements.txt	Python dependencies

expected_metrics.json Format

{
  "description": "Top-level doc string — skipped by the validator.",
  "category_name": {
    "description": "Human-readable description.",
    "metrics": ["metric_1", "metric_2"]
  },
  "grafana_dashboards": {
    "uids": ["rpc-performance", "node-health"]
  },
  "not_asserted": {
    "description": "Why these are excluded.",
    "metrics_excluded": { "metric_3": "reason" }
  }
}

Every metric listed under a metrics array must produce > 0 Prometheus series during the validation run. If a metric doesn't fire, the workload generators need to produce enough load to trigger it.

Three top-level keys are not metric categories:

• description and grafana_dashboards are skipped explicitly by validate_metrics. grafana_dashboards.uids drives the dashboard check, so adding a dashboard to docker/telemetry/grafana/dashboards/ does not put it under the gate until its uid is added here too.
• not_asserted is skipped structurally: the loop reads category_data.get("metrics", []), and this group deliberately has no metrics key — its entries live under metrics_excluded as a name-to-reason map. It documents metrics that are emitted and dashboarded but left unasserted because they are workload-gated or defect-gated (a check that fails on a healthy run is worse than no check). Promote an entry into an asserted group only after the workload is changed to guarantee it fires.

expected_spans.json Format

Each span entry defines its name, category, parent (for hierarchy validation), required attributes, and the config_flag that must be enabled. A trailing * in name is a wildcard. The optional "optional": true field marks a span whose absence is a skip rather than a failure:

{
  "name": "rpc.command.*",
  "category": "rpc",
  "parent": "rpc.process",
  "required_attributes": ["command", "version", "rpc_role", "rpc_status"],
  "config_flag": "trace_rpc"
}

Node Configuration Notes

The orchestrator (run-full-validation.sh) generates node configs with:

• [telemetry] enabled=1 with all five trace categories: trace_rpc, trace_transactions, trace_consensus, trace_peer, trace_ledger
• [insight] server=otel with endpoint=http://localhost:4318/v1/metrics and prefix=xrpld — beast::insight metrics reach Prometheus over OTLP, because the collector declares no statsd receiver
• [signing_support] true — required for tx_submitter.py to submit signed transactions via WebSocket
• [ips] (not [ips_fixed]) — ensures peer connections are counted in the PeerFinder active-peer gauges, exported as peer_finder_active_inbound_peers / peer_finder_active_outbound_peers (fixed peers are excluded from these counters by design). The beast::insight group/name pair is Peer_Finder / Active_Inbound_Peers; formatName() lowercases it for export.

Gauge Export Behaviour

The harness configures each node with [insight] server=otel (see the [insight] block generated by run-full-validation.sh), so beast::insight gauges go through OTelGaugeImpl in src/libxrpl/beast/insight/OTelCollector.cpp, not through the StatsD collector. That matters for how the validator queries Prometheus.

How OTelGaugeImpl exports. It wraps an OTel observable (asynchronous) gauge. set() and increment() only store into an std::atomic<int64_t>; nothing is exported at call time. The SDK's collection thread invokes gaugeCallback, which runs the collector's hooks and then Observe()s whatever the atomic currently holds. So the gauge reports every collection cycle, whether or not the value changed — including a gauge that sits at 0 from startup. There is no dirty flag on this path, and no first-flush special case is needed.

Why the validator still uses /api/v1/series. Two reasons survive the move to OTLP:

1. Late-populating series. A gauge or counter may not have completed the export → collector → Prometheus-scrape pipeline by the time validation runs. _check_prometheus_metric in validate_telemetry.py therefore polls /api/v1/series (which returns anything that existed anywhere in the query window) until the metric appears or the poll window elapses, instead of racing a single instant query.
2. Staleness robustness. /api/v1/series does not care whether the newest sample is inside Prometheus's ~5-minute staleness horizon, so the check cannot be defeated by a quiet series.

Note — the StatsD path is still in the tree but unused here. If a node is configured with server=statsd, StatsDGaugeImpl (in src/libxrpl/beast/insight/StatsDCollector.cpp) does gate emission on a dirty_ flag that is only set by set()/increment(), and it is initialised to true so the initial value is emitted on the first flush. The collector configs shipped in docker/telemetry/ declare no statsd receiver (the metrics pipeline is [otlp, spanmetrics]) and the base docker-compose.yml keeps its StatsD UDP port commented out, so nothing in this harness can receive StatsD.