rippled/docker/telemetry/workload/README.md

# Telemetry Workload Tools

Synthetic workload generation and validation tools for xrpld's OpenTelemetry telemetry stack. These tools validate that all spans, metrics, dashboards, and log-trace correlation work end-to-end under controlled load.

## Quick Start

```bash
# Build xrpld with telemetry enabled (see BUILD.md for the full flow)
mkdir -p .build && cd .build
conan install .. --output-folder . --build missing \
    --settings build_type=Release -o telemetry=True
cmake -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake \
    -DCMAKE_BUILD_TYPE=Release -Dtelemetry=ON ..
cmake --build . --parallel "$(nproc)" --target xrpld
cd ..

# Run full validation (starts everything, runs load, validates)
docker/telemetry/workload/run-full-validation.sh --xrpld .build/xrpld

# Cleanup when done
docker/telemetry/workload/run-full-validation.sh --cleanup
```

## Architecture

The validation suite runs a multi-node xrpld cluster as local processes alongside
a Docker Compose telemetry stack. The cluster exercises consensus, peer-to-peer
spans (proposals, validations), and all metric pipelines.

```
run-full-validation.sh (shell orchestrator)
  |
  |-- docker-compose.workload.yaml
  |     |-- otel-collector (otlp receiver: traces + beast::insight metrics;
  |     |                  filelog receiver: node debug.log -> Loki)
  |     |-- tempo (trace backend + TraceQL search API)
  |     |-- prometheus (metrics scraping)
  |     |-- loki (log aggregation for log-trace correlation)
  |     |-- grafana (dashboards, provisioned automatically)
  |
  |-- generate-validator-keys.sh
  |     -> validator-keys.json, validators.txt
  |
  |-- Nx xrpld nodes (local processes, full telemetry)
  |     - Each node: [telemetry] enabled=1, all 5 trace_* categories on
  |     - [insight] server=otel (beast::insight metrics over OTLP, no StatsD)
  |     - [signing_support] true (server-side signing for tx_submitter)
  |     - Peer discovery via [ips] (not [ips_fixed]) for active peer counts
  |
  |-- workload_orchestrator.py (phased load execution)
  |     |-- rpc_load_generator.py (WebSocket RPC traffic)
  |     |-- tx_submitter.py (transaction diversity)
  |     -> workload-report.json + per-phase reports
  |
  |-- validate_telemetry.py (pass/fail checks)
  |     -> validation-report.json
  |
  |-- benchmark.sh (baseline vs telemetry comparison)
        |-- collect_system_metrics.sh (per-leg CPU/RSS/latency/TPS sampling)
        -> benchmark-report-*.md
```

## Workload Profiles

The workload orchestrator (`workload_orchestrator.py`) reads named profiles
from `workload-profiles.json` and executes sequential load phases. Within
each phase, the RPC generator and TX submitter run concurrently.

### Available Profiles

| Profile           | Phases | Duration                    | Purpose                                                                                          |
| ----------------- | ------ | --------------------------- | ------------------------------------------------------------------------------------------------ |
| `full-validation` | 7      | 4.5 min + 1 min propagation | Coverage for the full asserted span/metric/dashboard inventory, with burst/idle/plateau patterns |
| `quick-smoke`     | 1      | 30s + 30s propagation       | Fast CI smoke test                                                                               |
| `stress`          | 3      | 3.5 min + 1 min propagation | Heavy sustained load for benchmarking                                                            |

Durations are the sum of the phase `duration_sec` values in
`workload-profiles.json` plus that profile's `propagation_wait_sec`; they exclude
cluster startup and the validation pass itself.

### full-validation Phases

| Phase        | RPC Rate           | TX TPS | Duration | Dashboard Coverage                                                                                                                                                                               |
| ------------ | ------------------ | ------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| warmup       | 5 RPS              | —      | 30s      | Node Health, Validator Health (baseline gauges)                                                                                                                                                  |
| steady-state | 30 RPS             | 3 TPS  | 60s      | All dashboards (plateau data)                                                                                                                                                                    |
| rpc-burst    | 100 RPS            | —      | 30s      | Job Queue, RPC Performance (latency spikes)                                                                                                                                                      |
| tx-flood     | 5 RPS              | 20 TPS | 30s      | Fee Market & TxQ, Transaction Overview                                                                                                                                                           |
| txq-burst    | 5 RPS (100% `fee`) | 60 TPS | 30s      | Fee Market & TxQ — single-type Payment burst that forces open-ledger fee escalation and TxQ queueing, exercising the `txq.*` spans (`txq.enqueue`, `txq.accept`, `txq.accept_tx`, `txq.cleanup`) |
| mixed-peak   | 50 RPS             | 10 TPS | 60s      | Consensus Health, Ledger Operations                                                                                                                                                              |
| cooldown     | 5 RPS              | —      | 30s      | Recovery patterns, state transitions                                                                                                                                                             |

### Custom Profiles

Add profiles to `workload-profiles.json`:

```json
{
  "profiles": {
    "my-custom": {
      "description": "Custom profile for specific testing",
      "phases": [
        {
          "name": "phase-name",
          "description": "What this phase exercises",
          "duration_sec": 60,
          "rpc": { "rate": 50, "weights": { "server_info": 80, "fee": 20 } },
          "tx": { "tps": 5, "weights": { "Payment": 100 } }
        }
      ],
      "propagation_wait_sec": 30
    }
  }
}
```

Set `"rpc"` or `"tx"` to `null` to skip that generator for a phase.
Custom `"weights"` override the default command/transaction distribution.

## Tools Reference

### run-full-validation.sh

Orchestrates the complete validation pipeline. Starts the telemetry stack, starts a multi-node xrpld cluster, generates load, and validates the results.

```bash
# Full validation with defaults (uses full-validation profile)
./run-full-validation.sh --xrpld /path/to/xrpld

# Quick smoke test
./run-full-validation.sh --xrpld /path/to/xrpld --profile quick-smoke

# Stress test with benchmarks
./run-full-validation.sh --xrpld /path/to/xrpld --profile stress --with-benchmark

# Skip Loki checks (if log export is not deployed)
./run-full-validation.sh --xrpld /path/to/xrpld --skip-loki
```

### workload_orchestrator.py

Reads a named profile from `workload-profiles.json` and executes sequential
load phases. Within each phase, `rpc_load_generator.py` and `tx_submitter.py`
run as concurrent subprocesses. Produces per-phase reports and a combined
summary.

```bash
# Run with a specific profile
python3 workload_orchestrator.py --profile full-validation

# Multiple endpoints
python3 workload_orchestrator.py --profile full-validation \
    --endpoints ws://localhost:6006 ws://localhost:6007

# Save combined report
python3 workload_orchestrator.py --profile stress --report /tmp/report.json
```

### rpc_load_generator.py

Generates RPC traffic matching realistic production distribution. Uses
xrpld's **native WebSocket command format** (`{"command": ...}`) with flat
parameters — the same format as `tx_submitter.py`.

- 40% health checks (server_info, fee)
- 30% wallet queries (account_info, account_lines, account_objects)
- 15% explorer queries (ledger, ledger_data)
- 10% transaction lookups (tx, account_tx)
- 5% DEX queries (book_offers, amm_info)

```bash
# Basic usage
python3 rpc_load_generator.py --endpoints ws://localhost:6006 --rate 50 --duration 120

# Multiple endpoints (round-robin)
python3 rpc_load_generator.py \
    --endpoints ws://localhost:6006 ws://localhost:6007 \
    --rate 100 --duration 300

# Custom weights
python3 rpc_load_generator.py --endpoints ws://localhost:6006 \
    --weights '{"server_info": 80, "account_info": 20}'
```

### tx_submitter.py

Submits diverse transaction types to exercise the full span and metric surface.
Uses xrpld's **native WebSocket command format** (`{"command": ...}`) rather
than JSON-RPC format. The response payload is inside the `"result"` key, with
`"status"` at the top level.

Supported transaction types:

- Payment (XRP transfers) — exercises `tx.process`, `tx.receive`, `tx.apply`
- OfferCreate / OfferCancel (DEX activity)
- TrustSet (trust line creation)
- NFTokenMint / NFTokenCreateOffer (NFT activity)
- EscrowCreate / EscrowFinish (escrow lifecycle)
- AMMCreate / AMMDeposit (AMM pool operations)

Requires `[signing_support] true` in the node config for server-side signing.

```bash
# Basic usage
python3 tx_submitter.py --endpoint ws://localhost:6006 --tps 5 --duration 120

# Custom mix
python3 tx_submitter.py --endpoint ws://localhost:6006 \
    --weights '{"Payment": 60, "OfferCreate": 20, "TrustSet": 20}'
```

### validate_telemetry.py

Automated validation that all expected telemetry data exists. Every metric in `expected_metrics.json` is required — if it doesn't fire, the validation fails. Spans are required unless the entry carries `"optional": true`.

- **Span validation**: All span types from `expected_spans.json` with required attributes and parent-child hierarchies. Entries marked `"optional": true` only fire under traffic the harness may not produce (HTTP/JSON-RPC client, gRPC client, missing-ledger fetch, mode transitions); their absence is recorded as a passing skip, not a failure.
- **Metric validation**: All metrics from `expected_metrics.json` — SpanMetrics, `beast::insight` gauges/counters/histograms, `MetricsRegistry` OTLP metrics. Every listed metric must have > 0 series. Uses the Prometheus `/api/v1/series` endpoint (not instant queries), polled until the metric appears or the poll window elapses, so a late-populating or quiet series is not a false negative.
- **Log-trace correlation**: trace_id/span_id in Loki logs (requires Loki)
- **Dashboard validation**: Every dashboard uid listed under `grafana_dashboards.uids` in `expected_metrics.json` loads with panels. That list currently covers **all 16** dashboards provisioned in `docker/telemetry/grafana/dashboards/`. Note the scope of this check: it asks the Grafana API whether the dashboard exists and returns a panel count — it does **not** run the panels' queries, so a dashboard can pass here while individual panels render empty.

```bash
# Run all validations
python3 validate_telemetry.py --report /tmp/report.json

# Skip Loki checks
python3 validate_telemetry.py --skip-loki --report /tmp/report.json
```

### OTel Timings Regression Gate

`capture_timings.py` + `compare_to_baseline.py` implement a regression gate
that compares OTel-derived per-span/per-RPC/per-job timings against a
committed baseline. Unlike `benchmark.sh` (which measures the overhead of
enabling telemetry on the current binary), this gate catches **xrpld
performance regressions over time** by diffing against a stored baseline
from a prior run.

How it runs inside the validation pipeline:

1. `run-full-validation.sh` executes the normal workload and validation suite.
2. After validation, `capture_timings.py` queries Prometheus for every
   metric in `regression-metrics.json` and writes `reports/timings.json`.
3. `compare_to_baseline.py` reads `timings.json`,
   `baselines/baseline-timings.json`, and `regression-thresholds.json`,
   then either:
   - Prints the paste-me JSON block (when the baseline is a placeholder
     or empty) and exits 0.
   - Prints a delta table, writes `reports/regression-report.json`, and
     exits non-zero if any metric breached both the percentage AND
     absolute bound.

Bootstrapping a baseline:

1. Push the branch. The `Telemetry Validation` CI run prints the full
   timings JSON under "Paste into `baselines/baseline-timings.json`" in
   the workflow Step Summary.
2. Open a PR copying that JSON block verbatim into
   `baselines/baseline-timings.json`. Reviewer approval is the audit gate.
3. Subsequent runs compare against it; the gate fails on regression.

Per-run tuning:

- `--skip-regression` disables the gate (local exploration only).
- `REGRESSION_WINDOW` env var overrides the default Prometheus `rate()`
  window (`3m`). Keep close to the workload duration.
- Metric surface lives in `regression-metrics.json`; thresholds in
  `regression-thresholds.json`; both are reviewed changes.

See [`baselines/README.md`](./baselines/README.md) for the baseline
lifecycle and refresh process.

### benchmark.sh

Compares baseline (no telemetry) vs telemetry-enabled performance:

```bash
./benchmark.sh --xrpld /path/to/xrpld --duration 300
```

Thresholds (configurable via environment):

| Metric            | Threshold | Env Variable                |
| ----------------- | --------- | --------------------------- |
| CPU overhead      | < 3%      | BENCH_CPU_OVERHEAD_PCT      |
| Memory overhead   | < 5MB     | BENCH_MEM_OVERHEAD_MB       |
| RPC p99 latency   | < 2ms     | BENCH_RPC_LATENCY_IMPACT_MS |
| Throughput impact | < 5%      | BENCH_TPS_IMPACT_PCT        |
| Consensus impact  | < 1%      | BENCH_CONSENSUS_IMPACT_PCT  |

Each report row is `PASS`, `FAIL`, or `INCONCLUSIVE`. The throughput and
consensus rows are ratios of the baseline, so they have nothing to report when
the baseline run measured zero — that row becomes `INCONCLUSIVE` and **counts
as a failure**, because an undefined result must never read as a pass.

Exit codes:

| Code | Meaning                                                                                                                     |
| ---- | --------------------------------------------------------------------------------------------------------------------------- |
| 0    | Every metric was measured and is within its threshold                                                                       |
| 1    | Every metric was measured and at least one exceeded its threshold                                                           |
| 2    | The overhead could not be measured — missing prerequisite, cluster never reached consensus, or incomplete metric collection |

`run-full-validation.sh` keeps the last two apart: 1 folds into its own
"checks failed" exit, 2 into its "infrastructure error" exit. A run that
measured nothing is therefore never reported as a performance regression.

### collect_system_metrics.sh

Samples CPU, peak RSS, RPC p99 latency, TPS and the mean inter-ledger interval
from the running nodes, and writes them as JSON. `benchmark.sh` calls it once
per leg; it is rarely run by hand.

```bash
./collect_system_metrics.sh 5020,5021,5022 300 /tmp/metrics.json
```

Processes are selected by matching `argv[0]`'s basename against the daemon
binary name; the pre-rename spelling is accepted too, so the sampler still
works against an older deployment. A wrapper that merely names the binary in
its arguments, and unrelated tools whose command line happens to contain the
string, are not sampled — including them diluted the CPU average and
attributed a foreign process's RSS to the node. `ps -C xrpld` is not usable
for this: xrpld renames itself, so its `comm` is `xrpld-main`.

Selection covers the whole host, so a second xrpld from another checkout is
sampled as well. Benchmark on a machine running one cluster only.

The output carries a `metrics_complete` flag. It is `false` when any
measurement source came back empty — no matching process, no successful RPC
probe, or a ledger sequence that never advanced — and the affected metrics are
then `0` placeholders. Since `0` clears every threshold, a `false` flag must be
read as inconclusive, never as a pass.

Exit codes:

| Code | Meaning                                                                                                   |
| ---- | --------------------------------------------------------------------------------------------------------- |
| 0    | Every metric was measured; `metrics_complete` is `true`                                                   |
| 1    | Cannot run: bad arguments, no GNU `date` with `%N`, or a failed process sample. No output file is written |
| 3    | The output file was written, but `metrics_complete` is `false`                                            |

`benchmark.sh` treats either non-zero code — and an explicit
`"metrics_complete": false` in an otherwise successful run — as fatal, and
exits 2 rather than comparing an incomplete run.

A nanosecond clock is required. RPC latency is graded against a 2 ms
threshold, and GNU `date +%s%N` is the only source cheap enough that the clock
does not dominate what it measures, so the script refuses to start without it.

## Reading Validation Reports

The validation report (`validation-report.json`) is structured as follows. The
counts below are illustrative — the real total is the sum of the span, metric,
log, dashboard and parity checks for the run.

```json
{
  "summary": {
    "total": 45,
    "passed": 42,
    "failed": 3,
    "all_passed": false
  },
  "checks": [
    {
      "name": "span.rpc.ws_message",
      "category": "span",
      "passed": true,
      "message": "rpc.ws_message: 15 traces found",
      "details": { "trace_count": 15 }
    }
  ]
}
```

Categories:

- **span**: Span type existence and attribute validation
- **metric**: Prometheus metric existence
- **log**: Log-trace correlation checks
- **dashboard**: Grafana dashboard accessibility
- **parity**: Span attributes required by the external-parity dashboard panels (validator-health, peer-quality, and friends)

## CI Integration

The validation runs as a GitHub Actions workflow (`.github/workflows/telemetry-validation.yml`):

- Triggered manually (`workflow_dispatch`) or on pushes to telemetry branches. There is no cron schedule.
- Builds xrpld, starts the full stack, runs load, validates
- Uploads reports as artifacts (and node logs when validation did not succeed)
- Writes the validation summary and the regression-gate summary to the workflow **Step Summary** (`$GITHUB_STEP_SUMMARY`). It does **not** comment on the PR — the workflow declares no `permissions:` block and calls no GitHub API, so read the summary on the run page.

Of the five `workflow_dispatch` inputs, only `run_benchmark` changes behaviour.
`rpc_rate`, `rpc_duration`, `tx_tps` and `tx_duration` are forwarded to
`run-full-validation.sh`, which parses them into shell variables and never reads
them again — load shape comes entirely from `--profile` and
`workload-profiles.json`. Their `description:` fields say so.

## Configuration Files

| File                              | Purpose                                                                                                                       |
| --------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `workload-profiles.json`          | Named load profiles with phase definitions                                                                                    |
| `expected_spans.json`             | Span inventory (names, attributes, hierarchies, config flags)                                                                 |
| `expected_metrics.json`           | Metric inventory — every listed metric must be present — plus the `grafana_dashboards.uids` list the dashboard check iterates |
| `test_accounts.json`              | Test account roles (keys generated at runtime)                                                                                |
| `regression-metrics.json`         | Metric surface for the OTel regression gate                                                                                   |
| `regression-thresholds.json`      | Per-metric regression bounds (pct AND abs)                                                                                    |
| `baselines/baseline-timings.json` | Committed baseline — populated from first CI run                                                                              |
| `requirements.txt`                | Python dependencies                                                                                                           |

### expected_metrics.json Format

```json
{
  "description": "Top-level doc string — skipped by the validator.",
  "category_name": {
    "description": "Human-readable description.",
    "metrics": ["metric_1", "metric_2"]
  },
  "grafana_dashboards": {
    "uids": ["rpc-performance", "node-health"]
  },
  "not_asserted": {
    "description": "Why these are excluded.",
    "metrics_excluded": { "metric_3": "reason" }
  }
}
```

Every metric listed under a `metrics` array must produce > 0 Prometheus series during the validation run. If a metric doesn't fire, the workload generators need to produce enough load to trigger it.

Three top-level keys are not metric categories:

- `description` and `grafana_dashboards` are skipped explicitly by
  `validate_metrics`. `grafana_dashboards.uids` drives the dashboard check, so
  adding a dashboard to `docker/telemetry/grafana/dashboards/` does **not** put
  it under the gate until its uid is added here too.
- `not_asserted` is skipped structurally: the loop reads
  `category_data.get("metrics", [])`, and this group deliberately has no
  `metrics` key — its entries live under `metrics_excluded` as a name-to-reason
  map. It documents metrics that are emitted and dashboarded but left unasserted
  because they are workload-gated or defect-gated (a check that fails on a
  healthy run is worse than no check). Promote an entry into an asserted group
  only after the workload is changed to guarantee it fires.

### expected_spans.json Format

Each span entry defines its name, category, parent (for hierarchy validation),
required attributes, and the `config_flag` that must be enabled. A trailing `*`
in `name` is a wildcard. The optional `"optional": true` field marks a span whose
absence is a skip rather than a failure:

```json
{
  "name": "rpc.command.*",
  "category": "rpc",
  "parent": "rpc.process",
  "required_attributes": ["command", "version", "rpc_role", "rpc_status"],
  "config_flag": "trace_rpc"
}
```

## Node Configuration Notes

The orchestrator (`run-full-validation.sh`) generates node configs with:

- `[telemetry] enabled=1` with all five trace categories: `trace_rpc`, `trace_transactions`, `trace_consensus`, `trace_peer`, `trace_ledger`
- `[insight] server=otel` with `endpoint=http://localhost:4318/v1/metrics` and `prefix=xrpld` — `beast::insight` metrics reach Prometheus over OTLP, because the collector declares no `statsd` receiver
- `[signing_support] true` — required for `tx_submitter.py` to submit signed transactions via WebSocket
- `[ips]` (not `[ips_fixed]`) — ensures peer connections are counted in the PeerFinder active-peer gauges, exported as `peer_finder_active_inbound_peers` / `peer_finder_active_outbound_peers` (fixed peers are excluded from these counters by design). The `beast::insight` group/name pair is `Peer_Finder` / `Active_Inbound_Peers`; `formatName()` lowercases it for export.

## Gauge Export Behaviour

The harness configures each node with `[insight] server=otel` (see the
`[insight]` block generated by `run-full-validation.sh`), so `beast::insight`
gauges go through `OTelGaugeImpl` in
`src/libxrpl/beast/insight/OTelCollector.cpp`, not through the StatsD collector.
That matters for how the validator queries Prometheus.

**How `OTelGaugeImpl` exports.** It wraps an OTel **observable** (asynchronous)
gauge. `set()` and `increment()` only store into an `std::atomic<int64_t>`;
nothing is exported at call time. The SDK's collection thread invokes
`gaugeCallback`, which runs the collector's hooks and then `Observe()`s whatever
the atomic currently holds. So the gauge reports **every collection cycle,
whether or not the value changed** — including a gauge that sits at 0 from
startup. There is no dirty flag on this path, and no first-flush special case is
needed.

**Why the validator still uses `/api/v1/series`.** Two reasons survive the move
to OTLP:

1. **Late-populating series.** A gauge or counter may not have completed the
   export → collector → Prometheus-scrape pipeline by the time validation runs.
   `_check_prometheus_metric` in `validate_telemetry.py` therefore polls
   `/api/v1/series` (which returns anything that existed anywhere in the query
   window) until the metric appears or the poll window elapses, instead of
   racing a single instant query.
2. **Staleness robustness.** `/api/v1/series` does not care whether the newest
   sample is inside Prometheus's ~5-minute staleness horizon, so the check
   cannot be defeated by a quiet series.

> **Note — the StatsD path is still in the tree but unused here.** If a node is
> configured with `server=statsd`, `StatsDGaugeImpl` (in
> `src/libxrpl/beast/insight/StatsDCollector.cpp`) does gate emission on a
> `dirty_` flag that is only set by `set()`/`increment()`, and it is
> initialised to `true` so the initial value is emitted on the first flush. The
> collector configs shipped in `docker/telemetry/` declare no `statsd` receiver
> (the metrics pipeline is `[otlp, spanmetrics]`) and the base
> `docker-compose.yml` keeps its StatsD UDP port commented out, so nothing in
> this harness can receive StatsD.