fix(telemetry): correct integration-test span checks and telemetry docs

The integration test's span assertions never actually ran. check_span()
built a Tempo /api/search call with --data-urlencode but no -G, so curl
POSTed the params as a body; Tempo answers 200 and ignores the query, so
every span name looked present. Verified against a live Tempo 2.9.4: the
buggy form returns the store's total trace count for any name, including
"zzz.does.not.exist"; with -G a real name returns 1 and a bogus one 0.

Fixed alongside it: the RPC check asserted "rpc.request", which is never
emitted (ServerHandler.cpp builds "rpc.http_request"). These two had to
change together, since -G turns the bogus name from a silent pass into a
hard failure.

Also in the script: a consensus timeout logged two failures and counted
two, because a post-loop else re-reported what the timeout branch had
already reported; and three unguarded curl calls aborted the whole script
under set -euo pipefail, making the ACCOUNT_ZERO fallback dead code with
no cleanup. Guarded the curls and wired an EXIT trap to the existing
cleanup(). The trap deliberately fires only before the summary, so a
completed run still leaves the stack up as the header documents.

Docs corrections, all re-derived from code:
- span inventory heading 35 -> 38, attribute heading 83 -> 89 rows
  (78 unique keys), and the section 6 header table now carries the
  missing TxApplySpanNames.h row so its columns sum to the same figures
- two stale paths: ConsensusSpanNames.h is under include/xrpl/consensus/,
  TxSpanNames.h under src/xrpld/telemetry/
- consensus_round_id is int64, not string (RCLConsensus.cpp sets
  prevLgr.seq() + 1); the runbook's TraceQL examples now use a numeric
  literal instead of an unparseable bare <round_id>
- state-accounting duration gauges are cumulative MICROSECONDS, not
  seconds (NetworkOPs.cpp declares std::chrono::microseconds and
  publishes dur.count() raw)
- sampling_ratio is not a config key; head sampling is fixed at 1.0 and
  the shipped collector has no tail sampling, so the caveat was rewritten
- the plan blurb referenced Jaeger; this stack is Tempo
This commit is contained in:
Pratik Mankawde
2026-08-15 17:24:14 +01:00
parent 773a5cc0fb
commit 18a40f86fb
4 changed files with 84 additions and 42 deletions

View File

@@ -160,8 +160,9 @@ hash); `tx.preflight` is stateless and omits both.
# Find specific ledger's consensus details
{name="consensus.accept.apply"} | ledger_seq = 92345678
# Find all spans in a consensus round (deterministic trace strategy)
{name="consensus.round"} | consensus_round_id = <round_id>
# Find all spans in a consensus round (deterministic trace strategy).
# consensus_round_id is an int64 — the previous ledger sequence plus one.
{name="consensus.round"} | consensus_round_id = 92345679
# Find dispute resolutions
{name="consensus.update_positions"} >> {event:name="dispute.resolve"}
@@ -249,10 +250,14 @@ sum by (stage) (rate(traces_span_metrics_calls_total{span_name=~"tx.preflight|tx
> a rising `tx.transactor` failure rate points to apply-time problems. Alert per
> stage rather than on a single aggregate so the failing stage is obvious.
> **Sampling caveat**: these stage metrics are span-derived and inherit the
> **tracer head-sampling** ratio (`sampling_ratio`). At `sampling_ratio < 1.0`
> they undercount proportionally — treat them as relative trends, not absolute
> transaction counts. Native StatsD metrics are unsampled.
> **Sampling caveat**: these stage metrics are span-derived, so they count only
> the spans the collector's spanmetrics connector sees. Head sampling at the node
> is fixed at 1.0 and is not configurable (`Telemetry.h`), and the shipped
> collector pipeline has no tail sampling, so today nothing is dropped and the
> counts are absolute. Volume reduction is delegated to the collector: adding a
> tail-sampling processor to the traces pipeline puts it ahead of the spanmetrics
> connector, and these metrics would then undercount proportionally — treat them
> as relative trends in that case. Native StatsD metrics are never sampled.
### Transaction Queue Health
@@ -452,7 +457,7 @@ all its normal attributes, it just lacks a cross-node parent link.
{name=~"tx\\..*"} | tx_hash = "<hash>"
# Find all spans in a cross-node consensus trace
{rootServiceName="xrpld"} | consensus_round_id = <round_id>
{rootServiceName="xrpld"} | consensus_round_id = 92345679
# Compare latency between sender and receiver for validations
{name="consensus.validation.send" || name="consensus.validation.receive"}
@@ -656,7 +661,7 @@ Ten dashboards are pre-provisioned in `docker/telemetry/grafana/dashboards/`:
| RPC Latency p95 by Command | timeseries | `histogram_quantile(0.95, sum by (le, command) (rate(traces_span_metrics_duration_milliseconds_bucket{span_name=~"rpc.command.*"}[5m])))` | `command` |
| RPC Error Rate | bargauge | Error spans / total spans × 100, grouped by `command` | `command`, `status_code` |
| RPC Latency Heatmap | heatmap | `sum(increase(traces_span_metrics_duration_milliseconds_bucket{span_name=~"rpc.command.*"}[5m])) by (le)` | `le` (bucket boundaries) |
| Overall RPC Throughput | timeseries | `rpc.request` + `rpc.process` rate | — |
| Overall RPC Throughput | timeseries | `rpc.http_request` + `rpc.process` rate | — |
| RPC Success vs Error | timeseries | by `status_code` (UNSET vs ERROR) | `status_code` |
| Top Commands by Volume | bargauge | `topk(10, ...)` by `command` | `command` |
| WebSocket Message Rate | stat | `rpc.ws_message` rate | — |