From 8418d474a7a63aa981a4854aa94388a556517b0d Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Wed, 26 Aug 2026 12:37:22 +0100 Subject: [PATCH] fix(telemetry): read span names from the Tempo name intrinsic The span reverse-coverage check has never evaluated. It reported "no span names were reported (backend unreachable or empty)" on a run where Tempo demonstrably held data -- the same run resolved a logged trace id to 32 spans. Root cause: the tag-values query asked for `span.name`. A span's name is a TraceQL intrinsic, not a span-scoped attribute, so `span.name` resolves to an attribute nothing sets. Tempo answers 200 with an empty tagValues list, which is indistinguishable from an empty backend and never raises, so the surrounding try/except stayed silent. Verified against tempo 2.9.4 holding exactly one span named probe.reverse.coverage, with the collector in front of it: /api/v2/search/tag/span.name/values -> {"tagValues":[]} /api/v2/search/tag/name/values -> that span's name /api/v2/search/tag/resource.service.name/values -> xrpld The third line is the control: the span was in Tempo, so the first line's emptiness was the wrong tag rather than no data. Cross-checked against a populated Tempo elsewhere, whose span scope lists real attributes (command, ledger_seq, tx_hash) and no name tag at all, while the bare intrinsic returns the whole span inventory. This is pre-existing, not a regression in the reverse check: the same URL fed the operations diagnostic before that check existed, and the last green run before it also logged "Tempo operations (0 total)". The check faithfully reported an empty input; the input was broken. The neighbouring resource.service.name query is correctly scoped and is left alone. --- docker/telemetry/workload/validate_telemetry.py | 11 ++++++++--- docs/telemetry-runbook.md | 16 ++++++++-------- 2 files changed, 16 insertions(+), 11 deletions(-) diff --git a/docker/telemetry/workload/validate_telemetry.py b/docker/telemetry/workload/validate_telemetry.py index ef990dd8c3..a6d1974ecd 100644 --- a/docker/telemetry/workload/validate_telemetry.py +++ b/docker/telemetry/workload/validate_telemetry.py @@ -465,11 +465,16 @@ async def validate_spans( # function. Note the tag-values API is not service-scoped, so in a stack # where something other than xrpld also sent traces this list would be a # superset; on the harness only xrpld exports spans. + # + # The tag is the bare intrinsic `name`, NOT `span.name`. A span's name is a + # TraceQL intrinsic, not a span-scoped attribute, so `span.name` resolves to + # an attribute nothing sets and Tempo answers 200 with an empty tagValues + # list -- indistinguishable from an empty backend, which is how the wrong + # tag went unnoticed. Verified against tempo 2.9.4 holding one span: + # `span.name` -> {"tagValues":[]}, `name` -> that span's name. emitted_span_names: list[str] = [] try: - async with session.get( - f"{tempo_url}/api/v2/search/tag/span.name/values" - ) as resp: + async with session.get(f"{tempo_url}/api/v2/search/tag/name/values") as resp: ops_data = await resp.json() tag_values = ops_data.get("tagValues", []) emitted_span_names = [tv.get("value", "") for tv in tag_values] diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index 17303ac9d3..aecbad1c52 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -3621,14 +3621,14 @@ The counts are not hard-coded in the validator — it iterates the inventory fil so those files are authoritative. The figures below are the inventory as it stands today. -| Category | Checks | Description | -| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Spans | Every **required** entry in `expected_spans.json` — 41 span types at the time of writing: 25 required, 16 marked `"optional": true` | Span name found in Tempo carrying its `required_attributes`, plus the declared parent-child relationships. An `"optional": true` entry that does not fire is recorded as a skip, not a failure — it needs traffic the harness may not generate (HTTP/JSON-RPC client, gRPC client, missing-ledger fetch, mode transitions) or that it deliberately no longer generates (path-finding RPC — see "Pathfinding is not exercised" in [the workload README](../docker/telemetry/workload/README.md)). | -| Metrics | Every entry in every asserted category of `expected_metrics.json` — 84 checks across 25 asserting categories at the time of writing: 79 metric names plus 5 `required_labels` checks | SpanMetrics, `beast::insight` gauges/counters exported over OTLP, and the `MetricsRegistry` OTLP metrics. Each must have > 0 Prometheus series; none are optional. A category may also declare `required_labels`, and each label there becomes one additional check that at least one of that category's series carries it with a non-empty value (matched as `