From 5f53745fbfc6016425109e1d1bc7c9306aca9727 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Thu, 3 Sep 2026 10:52:04 +0100 Subject: [PATCH] docs(telemetry): correct the TESTING guide and the span attribute tables Review feedback on the testing guide: - rm -rf targeted data/, but this config writes under docker/telemetry/data/, so teardown did nothing and a second run reused the old NuDB and SQLite state. Corrected at both sites, including the Test 2 keygen node, which launches with the same config. - The standalone span table said consensus.* does not fire. It does: ledger_accept drives a simulated round, so consensus.round, .phase.open, .ledger_close, .accept and .accept.apply all appear. Only .establish, .update_positions, .check, .proposal.* , .validation.receive and .mode_change cannot. The test intro claimed the same thing and now agrees with the table. - Three blocks duplicated content the file already had. Test 1 now points at the shared Verification Queries section as Test 2 already did, and the Test 2 submit block checks engine_result like Test 1 does. - The numbered step list was a copy of the script's own Step N headers and had drifted by four entries, so it now points at those headers instead. Also corrects the runbook's ledger and peer span tables against the code: ledger.build was credited with tx_count and tx_failed, which tx.apply sets, and was missing its three close-time attributes; peer.validation.receive was missing ledger_hash and full_validation. The five source line numbers in those two tables were stale, so they now name the file only, as the other nineteen rows do. --- docker/telemetry/TESTING.md | 80 +++++++++++++------------------------ docs/telemetry-runbook.md | 20 +++++----- 2 files changed, 37 insertions(+), 63 deletions(-) diff --git a/docker/telemetry/TESTING.md b/docker/telemetry/TESTING.md index 098002309e..5c181b109a 100644 --- a/docker/telemetry/TESTING.md +++ b/docker/telemetry/TESTING.md @@ -34,8 +34,10 @@ The binary is at `.build/xrpld`. ## Test 1: Single-Node Standalone (Quick Verification) -This test verifies RPC and transaction spans in standalone mode. Consensus -spans will not fire because standalone mode does not run consensus. +This test verifies RPC and transaction spans in standalone mode, plus the +consensus spans that a simulated round still produces. The proposal, voting +and peer-facing consensus spans do not fire — see the expected-spans table at +the end of this test for which do and which do not. ### Step 1: Start the observability stack @@ -113,32 +115,7 @@ curl -s http://localhost:5005 -d '{"method":"ledger_accept"}' ### Step 5: Verify traces in Tempo -Wait 5 seconds for the batch export, then: - -```bash -TEMPO="http://localhost:3200" - -# Check xrpld service is registered -curl -s "$TEMPO/api/v2/search/tag/resource.service.name/values" | jq '.tagValues[].value' - -# Check RPC spans -curl -s "$TEMPO/api/search" \ - --data-urlencode 'q={resource.service.name="xrpld" && name="rpc.http_request"}' \ - --data-urlencode 'limit=5' | jq '.traces | length' - -curl -s "$TEMPO/api/search" \ - --data-urlencode 'q={resource.service.name="xrpld" && name="rpc.process"}' \ - --data-urlencode 'limit=5' | jq '.traces | length' - -curl -s "$TEMPO/api/search" \ - --data-urlencode 'q={resource.service.name="xrpld" && name="rpc.command.server_info"}' \ - --data-urlencode 'limit=5' | jq '.traces | length' - -# Check transaction spans -curl -s "$TEMPO/api/search" \ - --data-urlencode 'q={resource.service.name="xrpld" && name="tx.process"}' \ - --data-urlencode 'limit=5' | jq '.traces | length' -``` +Wait 5 seconds for the batch export, then see the "Verification Queries" section below. Its span loop is a superset of what standalone mode produces, so compare its output against the "Expected spans (standalone mode)" table above rather than running a second, narrower set of queries here. Or open Grafana Explore with Tempo datasource: http://localhost:3000 @@ -152,23 +129,24 @@ kill $(pgrep -f 'xrpld.*xrpld-telemetry') docker compose -f docker/telemetry/docker-compose.yml down # Clean xrpld data -rm -rf data/ +rm -rf docker/telemetry/data/ ``` ### Expected spans (standalone mode) -| Span Name | Expected | Notes | -| --------------------------- | -------- | ----------------------------- | -| `rpc.http_request` | Yes | Every HTTP RPC call | -| `rpc.process` | Yes | Every RPC processing | -| `rpc.command.server_info` | Yes | server_info RPC | -| `rpc.command.server_state` | Yes | server_state RPC | -| `rpc.command.ledger` | Yes | ledger RPC | -| `rpc.command.submit` | Yes | submit RPC | -| `rpc.command.ledger_accept` | Yes | ledger_accept RPC | -| `tx.process` | Yes | Transaction submission | -| `tx.receive` | No | No peers in standalone | -| `consensus.*` | No | Consensus disabled standalone | +| Span Name | Expected | Notes | +| ---------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------- | +| `rpc.http_request` | Yes | Every HTTP RPC call | +| `rpc.process` | Yes | Every RPC processing | +| `rpc.command.server_info` | Yes | server_info RPC | +| `rpc.command.server_state` | Yes | server_state RPC | +| `rpc.command.ledger` | Yes | ledger RPC | +| `rpc.command.submit` | Yes | submit RPC | +| `rpc.command.ledger_accept` | Yes | ledger_accept RPC | +| `tx.process` | Yes | Transaction submission | +| `tx.receive` | No | No peers in standalone | +| `consensus.round`, `.phase.open`, `.ledger_close`, `.accept`, `.accept.apply` | Yes | `ledger_accept` drives a simulated round | +| `consensus.establish`, `.update_positions`, `.check`, `.proposal.*`, `.validation.receive`, `.mode_change` | No | `simulate` jumps straight to `Accepted`; no peers | --- @@ -185,17 +163,11 @@ Run the integration test script: bash docker/telemetry/integration-test.sh ``` -The script will: +It checks prerequisites, clears the previous run, brings up the observability stack, generates six validator key pairs and their node configs, starts the nodes, waits for consensus and then for a validated ledger, exercises RPC and submits a transaction, verifies traces in Tempo and both the spanmetrics and the StatsD-derived metrics in Prometheus, then prints a summary and leaves the stack running. -1. Start the observability stack -2. Generate 6 validator key pairs -3. Create config files for each node -4. Start all 6 nodes -5. Wait for consensus ("proposing" state) -6. Exercise RPC, submit transactions -7. Verify all span categories in Tempo -8. Verify spanmetrics in Prometheus -9. Print results and leave the stack running +The script announces each step as it runs, so read its `Step N:` headers for the authoritative sequence — they are not restated here, because a numbered copy of them drifts as soon as a step is added. + +Its Tempo checks cover the RPC, transaction, consensus, ledger and peer span categories from a fixed list, which is narrower than the loop in the "Verification Queries" section below. ### Manual @@ -231,7 +203,7 @@ Kill the temporary node: ```bash kill $TEMP_PID -rm -rf data/ +rm -rf docker/telemetry/data/ ``` #### Step 3: Create node configs @@ -359,9 +331,11 @@ curl -s http://localhost:5005 -d '{ "Amount": "10000000" } }] -}' +}' | jq .result.engine_result ``` +Expected result: `"tesSUCCESS"`, the same as Test 1 Step 4. + Wait 15 seconds for consensus and batch export. #### Step 8: Verify in Tempo diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index 208bc700de..86f326ec3e 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -172,18 +172,18 @@ hash); `tx.preflight` is stateless and omits both. ### Ledger Spans -| Span Name | Source File | Attributes | Description | -| ----------------- | -------------------- | ------------------------------------- | ----------------------------- | -| `ledger.build` | BuildLedger.cpp:31 | `ledger_seq`, `tx_count`, `tx_failed` | Ledger build during consensus | -| `ledger.validate` | LedgerMaster.cpp:915 | `ledger_seq`, `validations` | Ledger promoted to validated | -| `ledger.store` | LedgerMaster.cpp:409 | `ledger_seq` | Ledger stored in history | +| Span Name | Source File | Attributes | Description | +| ----------------- | ---------------- | ----------------------------------------------------------------------- | ----------------------------- | +| `ledger.build` | BuildLedger.cpp | `ledger_seq`, `close_time`, `close_time_correct`, `close_resolution_ms` | Ledger build during consensus | +| `ledger.validate` | LedgerMaster.cpp | `ledger_seq`, `validations` | Ledger promoted to validated | +| `ledger.store` | LedgerMaster.cpp | `ledger_seq` | Ledger stored in history | ### Peer Spans -| Span Name | Source File | Attributes | Description | -| ------------------------- | ---------------- | ------------------------------- | ----------------------------- | -| `peer.proposal.receive` | PeerImp.cpp:1667 | `peer_id`, `proposal_trusted` | Proposal received from peer | -| `peer.validation.receive` | PeerImp.cpp:2264 | `peer_id`, `validation_trusted` | Validation received from peer | +| Span Name | Source File | Attributes | Description | +| ------------------------- | ----------- | ----------------------------------------------------------------- | ----------------------------- | +| `peer.proposal.receive` | PeerImp.cpp | `peer_id`, `proposal_trusted` | Proposal received from peer | +| `peer.validation.receive` | PeerImp.cpp | `peer_id`, `ledger_hash`, `full_validation`, `validation_trusted` | Validation received from peer | Both peer receive spans are `kConsumer` inbound entry points started as fresh trace roots. They never inherit an ambient span left active on the peer thread, @@ -459,7 +459,7 @@ all its normal attributes, it just lacks a cross-node parent link. {name=~"tx\\..*"} | tx_hash = "" # Find all spans in a cross-node consensus trace -{rootServiceName="xrpld"} | consensus_round_id = 92345679 +{rootServiceName="xrpld"} | consensus_round_id = "" # Compare latency between sender and receiver for validations {name="consensus.validation.send" || name="consensus.validation.receive"}