# OpenTelemetry Integration Testing Guide This document describes how to verify the xrpld OpenTelemetry telemetry pipeline end-to-end, from span generation through the observability stack (otel-collector, Tempo, Prometheus, Grafana). --- ## Prerequisites ### Build xrpld with telemetry ```bash conan install . --build=missing -o telemetry=True cmake --preset default -Dtelemetry=ON cmake --build --preset default --target xrpld ``` The binary is at `.build/xrpld`. ### Required tools - **Docker** with `docker compose` (v2) - **curl** - **jq** (JSON processor) ### Verify binary ```bash .build/xrpld --version ``` --- ## Test 1: Single-Node Standalone (Quick Verification) This test verifies RPC and transaction spans in standalone mode, plus the consensus spans that a simulated round still produces. The proposal, voting and peer-facing consensus spans do not fire — see the expected-spans table at the end of this test for which do and which do not. ### Step 1: Start the observability stack ```bash docker compose -f docker/telemetry/docker-compose.yml up -d ``` Wait for services to be ready: ```bash # otel-collector health curl -sf http://localhost:13133/ && echo "collector ready" # Tempo readiness curl -sf http://localhost:3200/ready >/dev/null && echo "tempo ready" ``` ### Step 2: Start xrpld in standalone mode ```bash .build/xrpld --conf docker/telemetry/xrpld-telemetry.cfg -a --start ``` Wait a few seconds for the node to initialize. ### Step 3: Exercise RPC spans ```bash # server_info curl -s http://localhost:5005 \ -d '{"method":"server_info"}' | jq .result.info.server_state # server_state curl -s http://localhost:5005 \ -d '{"method":"server_state"}' | jq .result.state.server_state # ledger curl -s http://localhost:5005 \ -d '{"method":"ledger","params":[{"ledger_index":"current"}]}' | jq .result.ledger_current_index ``` ### Step 4: Submit a transaction Close the ledger to drive a simulated consensus round — that round is what produces the `consensus.*` spans. It is not required for `submit` itself: standalone puts the node in `OperatingMode::FULL` at startup (`NetworkOPsImp::setStandAlone()`), and the one validated-ledger-age gate on the submit path is skipped when `config.standalone()` is set (`checkTxJsonFields()` in `src/xrpld/rpc/detail/TransactionSign.cpp`). ```bash curl -s http://localhost:5005 -d '{"method":"ledger_accept"}' ``` Submit a Payment from the genesis account: ```bash curl -s http://localhost:5005 -d '{ "method": "submit", "params": [{ "secret": "snoPBrXtMeMyMHUVTgbuqAfg1SUTb", "tx_json": { "TransactionType": "Payment", "Account": "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "Destination": "rN7n7otQDd6FczFgLdSqtcsAUxDkw6fzRH", "Amount": "10000000" } }] }' | jq .result.engine_result ``` Expected result: `"tesSUCCESS"`. The destination does not have to exist yet. 10 XRP is exactly the default base reserve (`FeeSetup::accountReserve` in `src/xrpld/core/Config.h`), so the payment creates and funds the account. `integration-test.sh` does not hardcode a destination at all — it calls `wallet_propose` and uses the `account_id` that comes back. Close the ledger again to finalize: ```bash curl -s http://localhost:5005 -d '{"method":"ledger_accept"}' ``` ### Step 5: Verify traces in Tempo Wait 5 seconds for the batch export, then see the "Verification Queries" section below. Its span loop is a superset of what standalone mode produces, so compare its output against the "Expected spans (standalone mode)" table above rather than running a second, narrower set of queries here. Or open Grafana Explore with Tempo datasource: http://localhost:3000 ### Step 6: Teardown ```bash # Kill xrpld (Ctrl+C or) pkill -f 'xrpld --conf docker/telemetry/xrpld-telemetry\.cfg' # Stop observability stack docker compose -f docker/telemetry/docker-compose.yml down # Clean xrpld data rm -rf docker/telemetry/data/ ``` The pattern is anchored on the whole `--conf ` argument with the `.` escaped, so it matches this node and not another xrpld run or an editor whose command line happens to name the same file. `pkill` is also a no-op when nothing matches, where `kill $(pgrep ...)` errors out with no arguments. ### Expected spans (standalone mode) | Span Name | Expected | Notes | | ----------------------------------------------------------------------------- | -------- | ------------------------------------------ | | `rpc.http_request` | Yes | Every HTTP RPC call | | `rpc.process` | Yes | Every RPC processing | | `rpc.command.server_info` | Yes | server_info RPC | | `rpc.command.server_state` | Yes | server_state RPC | | `rpc.command.ledger` | Yes | ledger RPC | | `rpc.command.submit` | Yes | submit RPC | | `rpc.command.ledger_accept` | Yes | ledger_accept RPC | | `rpc.ws_upgrade`, `rpc.ws_message` | No | Need a WebSocket client | | `tx.process` | Yes | Transaction submission | | `tx.preflight`, `tx.preclaim`, `tx.transactor` | Yes | Apply stages of the Payment | | `tx.apply` | Yes | Ledger build applies the tx set | | `tx.receive` | No | No peers in standalone | | `txq.enqueue`, `txq.apply_direct` | Yes | `TxQ::apply` on the submit path | | `txq.accept`, `txq.cleanup` | Yes | Run on every ledger close | | `txq.accept_tx`, `txq.batch_clear` | No | Nothing is ever queued here | | `ledger.build`, `ledger.store` | Yes | `buildLCL` builds, then stores | | `ledger.validate` | No | `checkAccept` is unreachable in standalone | | `consensus.round`, `.phase.open`, `.ledger_close`, `.accept`, `.accept.apply` | Yes | `ledger_accept` drives a simulated round | | `consensus.mode_change` | Yes | Fires once per round start | | `consensus.establish`, `.update_positions`, `.check` | No | `phaseEstablish()` never runs | | `consensus.proposal.send`, `.validation.send` | No | The config carries no validator key | | `consensus.proposal.receive`, `.validation.receive` | No | No peers | | `peer.proposal.receive`, `peer.validation.receive` | No | No peers | | `pathfind.*` | No | No path request, no path subscription | | `grpc.*` | No | No `[port_grpc]` in the config | Four of the "No" rows have a reason worth spelling out. - `ledger.validate` belongs to `LedgerMaster::checkAccept`, and standalone never reaches it: `consensusBuilt` returns early when standalone, and `switchLCL` takes its standalone branch instead of calling `checkAccept`. That `getNeededValidations()` returns 0 in standalone is therefore not enough on its own. - `consensus.establish`, `.update_positions` and `.check` are started from `phaseEstablish()`. `simulate` does call `closeLedger({})` — which is exactly why `.phase.open` and `.ledger_close` do fire — and then sets the phase to `Accepted` itself, so `phaseEstablish()` is never entered. - `.proposal.send` and `.validation.send` are absent for a different reason again: `xrpld-telemetry.cfg` carries no `validation_seed` or `validator_token`, so `preStartRound` leaves `validating_` false. The node observes rather than proposes, and `validate()` — the owner of `.validation.send` — is never called. - `pathfind.update_all` is emitted only while at least one path subscription is active, and this test makes no `path_find` or `ripple_path_find` call. `.mode_change` is in the "Yes" rows because it does not depend on the mode actually changing. `startRoundInternal` calls `mode_.set()`, `MonitoredMode::set` calls `onModeChange` with no equality test, and `onModeChange` creates the span before the `before != after` check — that check guards only the censorship-detector reset. One `consensus.round` span reaches Tempo, not two. `roundSpan_` is reset only at the top of the next `startRoundTracing()`, so after the two `ledger_accept` calls the first round's span has ended and been exported while the second is still open. Only ended spans are exported. --- ## Test 2: 6-Node Consensus Network (Full Verification) This test verifies ALL span categories including consensus and peer transaction relay, using a 6-node validator network. ### Automated Run the integration test script: ```bash bash docker/telemetry/integration-test.sh ``` It checks prerequisites, clears the previous run, brings up the observability stack, generates six validator key pairs and their node configs, starts the nodes, waits for consensus and then for a validated ledger, exercises RPC and submits a transaction, verifies traces in Tempo and both the span_metrics and the native `beast::insight` metrics that arrive over OTLP in Prometheus, checks that no StatsD listener is needed, then prints a summary and leaves the stack running. The authoritative sequence is the 14 `# Step N:` banner comments in the script source, so read the file rather than the console — none of the script's 48 runtime `log` lines print a step number. The sequence is not restated here, because a numbered copy of it drifts as soon as a step is added. Its Tempo checks cover the RPC, transaction, consensus, ledger and peer span categories from a fixed list, which is narrower than the loop in the "Verification Queries" section below. ### Manual If you prefer to run the steps manually: #### Step 1: Start observability stack ```bash docker compose -f docker/telemetry/docker-compose.yml up -d ``` #### Step 2: Generate validator keys Start a temporary standalone xrpld: ```bash .build/xrpld --conf docker/telemetry/xrpld-telemetry.cfg -a --start & TEMP_PID=$! sleep 5 ``` Generate 6 key pairs: ```bash for i in $(seq 1 6); do curl -s http://localhost:5005 \ -d '{"method":"validation_create"}' | jq '.result' done ``` Record the `validation_seed` and `validation_public_key` for each. Kill the temporary node: ```bash kill $TEMP_PID rm -rf docker/telemetry/data/ ``` #### Step 3: Create node configs For each node (1-6), create a config file. Template: ```ini [server] port_rpc port_peer [port_rpc] port = {5004 + node_number} ip = 127.0.0.1 admin = 127.0.0.1 protocol = http [port_peer] port = {51234 + node_number} ip = 0.0.0.0 protocol = peer [node_db] type=NuDB path=/tmp/xrpld-integration/node{N}/nudb online_delete=256 [database_path] /tmp/xrpld-integration/node{N}/db [debug_logfile] /tmp/xrpld-integration/node{N}/debug.log [validation_seed] {seed from step 2} [validators_file] /tmp/xrpld-integration/validators.txt [ips_fixed] {one "127.0.0.1 " line for each port in 51235-51240 except this node's own 51234 + node_number — a node must not list itself as a fixed peer, so each config carries five lines, not six} [peer_private] 1 [telemetry] enabled=1 service_instance_id=Node-{N} traces_endpoint=http://localhost:4318/v1/traces metrics_endpoint=http://localhost:4318/v1/metrics batch_size=512 batch_delay_ms=2000 max_queue_size=2048 trace_rpc=1 trace_transactions=1 trace_consensus=1 trace_peer=1 trace_ledger=1 [insight] # server=otel is the only load-bearing key here -- it selects OTelCollector. # The export endpoint comes from [telemetry] metrics_endpoint, and [insight]'s # own service_instance_id/service_name keys are ignored. server=otel [rpc_startup] { "command": "log_level", "severity": "warning" } [ssl_verify] 0 ``` #### Step 4: Create validators.txt ```ini [validators] {public_key_1} {public_key_2} {public_key_3} {public_key_4} {public_key_5} {public_key_6} ``` #### Step 5: Start all 6 nodes ```bash for i in $(seq 1 6); do .build/xrpld --conf /tmp/xrpld-integration/node$i/xrpld.cfg --start & echo $! >/tmp/xrpld-integration/node$i/xrpld.pid done ``` #### Step 6: Wait for consensus Poll each node until `server_state` = `"proposing"`: ```bash for port in 5005 5006 5007 5008 5009 5010; do while true; do state=$(curl -s http://localhost:$port \ -d '{"method":"server_info"}' | jq -r '.result.info.server_state') echo "Port $port: $state" [ "$state" = "proposing" ] && break sleep 5 done done ``` #### Step 7: Exercise RPC and submit transaction ```bash # RPC calls curl -s http://localhost:5005 -d '{"method":"server_info"}' curl -s http://localhost:5005 -d '{"method":"server_state"}' curl -s http://localhost:5005 -d '{"method":"ledger","params":[{"ledger_index":"current"}]}' # Submit transaction curl -s http://localhost:5005 -d '{ "method": "submit", "params": [{ "secret": "snoPBrXtMeMyMHUVTgbuqAfg1SUTb", "tx_json": { "TransactionType": "Payment", "Account": "rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh", "Destination": "rN7n7otQDd6FczFgLdSqtcsAUxDkw6fzRH", "Amount": "10000000" } }] }' | jq .result.engine_result ``` Expected result: `"tesSUCCESS"`, the same as Test 1 Step 4. Wait 15 seconds for the consensus round and the trace batch export. Prometheus needs longer: `integration-test.sh` waits a further 20 s before its span_metrics queries and another 20 s before its StatsD queries, so 35 s and 55 s after the submit. Querying the metrics block at 15 s returns no series, which looks like a broken pipeline and is not one. #### Step 8: Verify in Tempo and Prometheus See the "Verification Queries" section below. --- ## Expected Span Catalog A smoke-test subset: the spans a short local run reliably produces, and how to trigger each. This is not the full catalog — the authoritative span list, with the attributes each span carries, is [docs/telemetry-runbook.md § Span Reference](../../docs/telemetry-runbook.md#span-reference). Attributes are deliberately not repeated here. Keeping a second copy is how this table came to list attribute keys that no longer exist anywhere in the code. | Span Name | Source File | How to Trigger | | --------------------------- | ----------------- | ------------------------- | | `rpc.http_request` | ServerHandler.cpp | Any HTTP RPC call | | `rpc.ws_upgrade` | ServerHandler.cpp | WebSocket upgrade | | `rpc.ws_message` | ServerHandler.cpp | WebSocket RPC message | | `rpc.process` | ServerHandler.cpp | RPC processing | | `rpc.command.` | RPCHandler.cpp | Any RPC command | | `tx.process` | NetworkOPs.cpp | Submit transaction | | `tx.receive` | PeerImp.cpp | Peer relays transaction | | `consensus.proposal.send` | RCLConsensus.cpp | Consensus proposing phase | | `consensus.ledger_close` | RCLConsensus.cpp | Ledger close event | | `consensus.accept` | RCLConsensus.cpp | Ledger accepted | | `consensus.validation.send` | RCLConsensus.cpp | Validation sent | | `consensus.accept.apply` | RCLConsensus.cpp | Ledger apply + close time | | `tx.apply` | BuildLedger.cpp | Ledger close (tx set) | | `ledger.build` | BuildLedger.cpp | Ledger build | | `ledger.validate` | LedgerMaster.cpp | Ledger validated | | `ledger.store` | LedgerMaster.cpp | Ledger stored | | `peer.proposal.receive` | PeerImp.cpp | Peer sends proposal | | `peer.validation.receive` | PeerImp.cpp | Peer sends validation | --- ## Verification Queries ### Tempo API Base URL: `http://localhost:3200` Run `RUN_START=$(date +%s)` **before** starting xrpld (Test 1 Step 2, Test 2 Step 5), in the same shell you will run the block below in. Tempo keeps blocks for `block_retention` (`tempo.yaml`, 1h) on a named volume, so a search with no time bound is answered by the previous run's traces. ```bash TEMPO="http://localhost:3200" # Refuse to run unbounded rather than report a previous run's traces. : "${RUN_START:?record RUN_START=\$(date +%s) before starting xrpld}" # List all services curl -s "$TEMPO/api/v2/search/tag/resource.service.name/values" | jq '.tagValues[].value' # Count traces per span name. Test 1 produces a subset of this list — read it # against the "Expected spans (standalone mode)" table above, not as pass/fail. # # -G is required: it moves the urlencoded parameters into the query string. # Without it curl POSTs them as a request body, Tempo answers 200 and ignores # the query, and every span name comes back non-zero. start/end bound the # search to this run; the end margin covers spans exported while the query is # in flight. for op in "rpc.http_request" "rpc.process" \ "rpc.command.server_info" "rpc.command.server_state" "rpc.command.ledger" \ "rpc.command.submit" "rpc.command.ledger_accept" \ "tx.process" "tx.receive" "tx.apply" \ "tx.preflight" "tx.preclaim" "tx.transactor" \ "txq.enqueue" "txq.apply_direct" "txq.accept" "txq.cleanup" \ "consensus.round" "consensus.phase.open" "consensus.ledger_close" \ "consensus.establish" "consensus.update_positions" "consensus.check" \ "consensus.accept" "consensus.accept.apply" \ "consensus.proposal.send" "consensus.validation.send" \ "consensus.mode_change" \ "consensus.proposal.receive" "consensus.validation.receive" \ "ledger.build" "ledger.validate" "ledger.store" \ "peer.proposal.receive" "peer.validation.receive"; do count=$(curl -sfG "$TEMPO/api/search" \ --data-urlencode "q={resource.service.name=\"xrpld\" && name=\"$op\"}" \ --data-urlencode "start=$RUN_START" \ --data-urlencode "end=$(($(date +%s) + 60))" \ --data-urlencode "limit=5" | jq '.traces | length') printf "%-35s %s traces\n" "$op" "$count" done ``` Eight more span families exist but need a trigger neither test performs, so they are counted separately — a zero here is the expected answer, not a failure. `rpc.ws_*` need a WebSocket client, the `pathfind.*` family needs a `path_find` or `ripple_path_find` call, and the two `txq` names need a transaction sitting in the queue. ```bash for op in "rpc.ws_upgrade" "rpc.ws_message" \ "pathfind.request" "pathfind.compute" "pathfind.discover" "pathfind.update_all" \ "txq.accept_tx" "txq.batch_clear"; do count=$(curl -sfG "$TEMPO/api/search" \ --data-urlencode "q={resource.service.name=\"xrpld\" && name=\"$op\"}" \ --data-urlencode "start=$RUN_START" \ --data-urlencode "end=$(($(date +%s) + 60))" \ --data-urlencode "limit=5" | jq '.traces | length') printf "%-35s %s traces\n" "$op" "$count" done ``` The remaining family is `grpc.`, whose span name is the gRPC method, so it has no fixed string to query and needs a `[port_grpc]` stanza neither test configures. ### Prometheus API Base URL: `http://localhost:9090` ```bash PROM="http://localhost:9090" # Span call counts (from the span_metrics connector). The span_ prefix is the # connector's `namespace: "span"` in otel-collector-config.yaml; drop that # setting and these become traces_span_metrics_*. curl -s "$PROM/api/v1/query?query=span_calls_total" | jq '.data.result[] | {span: .metric.span_name, count: .value[1]}' # Latency histogram curl -s "$PROM/api/v1/query?query=span_duration_milliseconds_count" | jq '.data.result[] | {span: .metric.span_name, count: .value[1]}' # RPC calls by command curl -s "$PROM/api/v1/query?query=span_calls_total{span_name=~\"rpc.command.*\"}" | jq '.data.result[] | {command: .metric.command, count: .value[1]}' # Deployment-tier labels present on metrics (set by the collector's # resource/tier processor and promoted via resource_to_telemetry_conversion). # Expect deployment_environment and xrpl_network_type on each series. curl -s "$PROM/api/v1/query?query=span_calls_total" | jq '.data.result[0].metric | {deployment_environment, xrpl_network_type, service_name}' ``` ### Grafana Open http://localhost:3000 (anonymous admin access enabled). Pre-configured dashboards: - **RPC Performance**: Request rates, latency percentiles by command, top commands, WebSocket rate - **Transaction Overview**: Transaction processing rates, apply duration, peer relay, failed tx rate - **Consensus Health**: Consensus round duration, proposer counts, mode tracking, accept heatmap - **Ledger Operations**: Build/validate/store rates and durations, TX apply metrics - **Peer Network**: Proposal/validation receive rates, trusted vs untrusted breakdown (requires `trace_peer=1`) Pre-configured datasources: - **Tempo**: Trace data at `http://tempo:3200` - **Prometheus**: Metrics at `http://prometheus:9090` --- ## Troubleshooting ### No traces in Tempo 1. Check otel-collector logs: ```bash docker compose -f docker/telemetry/docker-compose.yml logs otel-collector ``` 2. Verify xrpld telemetry config has `enabled=1` and correct endpoint 3. Check that otel-collector port 4318 is accessible: ```bash curl -sf http://localhost:4318 && echo "reachable" ``` 4. Increase `batch_delay_ms` or decrease `batch_size` in xrpld config ### Nodes not reaching "proposing" state 1. Check that all peer ports (51235-51240) are not in use: ```bash for p in 51235 51236 51237 51238 51239 51240; do ss -tlnp | grep ":$p " && echo "port $p in use" done ``` 2. Verify `[ips_fixed]` lists the 5 other peer ports, and not the node's own 3. Verify `validators.txt` has all 6 public keys 4. Check node debug logs: `tail -50 /tmp/xrpld-integration/node1/debug.log` 5. Ensure `[peer_private]` is set to `1`. In `src/libxrpl/peerfinder/Config.cpp` it sets both `autoConnect = !standalone && !peerPrivate` and `wantIncoming = (!config.peerPrivate) && (port != 0)`, so it stops the node reaching out to the public network **and** stops it accepting inbound peers. The nodes here find each other through `[ips_fixed]`, which is unaffected. ### Transaction not processing 1. Verify genesis account exists: ```bash curl -s http://localhost:5005 \ -d '{"method":"account_info","params":[{"account":"rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh"}]}' | jq .result.account_data.Balance ``` 2. Check submit response for error codes 3. In standalone mode, remember to call `ledger_accept` after submitting ### Spanmetrics not appearing in Prometheus 1. Verify otel-collector config has `span_metrics` connector 2. Check that the metrics pipeline is configured: ```yaml service: pipelines: metrics: receivers: [span_metrics] exporters: [prometheus] ``` 3. Verify Prometheus can reach collector: ```bash curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets' ```