These inbound peer-message entry points (kConsumer) used span(), which
inherits whatever span is active on the peer thread — including a leaked
tx.receive scope — so validations/proposals were wrongly nested under
unrelated transaction traces. rootSpan() starts a fresh trace root.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The spanmetrics connector had no namespace, so it emitted traces_span_metrics_*
metric names by default. The span dashboards and docs are renamed to query
span_* names; this is only correct if the connector emits them too, so add
namespace: "span" to the spanmetrics connector. Both sides change together:
renaming the dashboards without the namespace (or vice versa) would break the
pipeline. Matches the phase9 collector config.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The phase7 OTelCollector::formatName lowercases and strips names, emitting
snake_case metrics with no xrpld_ prefix. The native Grafana dashboards and
the telemetry docs still queried the old xrpld_CamelCase names, so they were
broken against their own pipeline. Rename every metric name to match what the
code emits: drop the xrpld_ prefix and lowercase the remainder. The two job
histograms also drop the redundant 'duration' word (job_queued_us,
job_running_us) to match the phase9 forms. Add havetxset to the cspell
dictionary since the lowercased metric name no longer word-splits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rename the 5 system-* dashboard files to bare domain names (uid ==
filename stem) and update all doc references. The system- prefix was
redundant now that titles no longer carry a recorder-pipeline suffix.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make dashboard uids bare domain names (uid == filename stem) and update
doc references. Grafana links/bookmarks to the old uids will need
refreshing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Match the reader-facing dashboard catalog to the actual dashboard titles,
which no longer carry the recorder-pipeline suffix.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- 09-data-collection-reference: add $service_name, $deployment_environment,
$xrpl_network_type to the template-variable table; note they appear on all
dashboards and point to the runbook's Deployment Tiers section.
- TESTING.md: add a Prometheus check verifying the tier labels
(deployment_environment, xrpl_network_type, service_name) are present on
metric series.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The peer.validation.receive span now records the ledger hash and full-validation
flag under the shared bare keys ledger_hash / full_validation (same keys the
consensus validation spans use), so there is no longer a dotted span attribute.
Remove the "only dotted span attribute" note and update the peer attribute table.
Head sampling is fixed at 1.0 and not configurable. Drop sampling_ratio
from the docker sample/test configs and the data-collection reference;
rewrite the span-metric sampling caveat and volume guidance to point at
collector-side tail sampling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update the Peer Spans note, the known-gaps caveat, and the trace category
toggle table to show trace_peer enabled by default. The Production Setup
example keeps trace_peer=0 as a deliberate high-volume opt-out alongside
sampling_ratio=0.01.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The §1 span and attribute inventory had regressed to an older 16-span snapshot
that uses the pre-2026-05-13 dotted attribute keys, while phase-7's code emits
~36 spans with bare/underscore attribute keys. The §Data Flow Overview and §2
System Metrics sections (native OTLP transport — phase-7's migration) were already
correct and are left unchanged.
- §1.1: expand the span inventory to the full surface — add gRPC (grpc.<MethodName>),
TxQ (txq.*), PathFind (pathfind.*), and the full consensus set (round/phase.open/
establish/update_positions/check/mode_change/proposal.receive/validation.receive).
Fix the phantom rpc.request -> rpc.http_request, add rpc.ws_upgrade. No grpc.request,
no pathfind.rank, no ledger.acquire (the latter is added in phase-9, not yet present here).
- §1.2: convert every span-attribute key from dotted xrpl.<domain>.<field> to the
bare/underscore form. The sole span-attr dotted exception is xrpl.ledger.hash on
peer.validation.receive (shared constant); consensus.validation.send uses bare
ledger_hash. Resource attrs xrpl.network.id/type stay dotted. Fix tx_count/tx_failed
placement (on tx.apply, not ledger.build). Add attribute tables for the new families.
- §1.3: list the full set of spanmetrics dimension labels (bare keys, from the collector
config) instead of the stale xrpl_rpc_command-style names.
- §4/§5: convert Tempo TraceQL and PromQL examples to the bare attribute/label forms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the apply-pipeline stage spans (tx.preflight, tx.preclaim,
tx.transactor) added on phase-3 through the observability stack so the
spanmetrics connector produces per-stage RED metrics without any native
instruments.
- collector: add the `stage` dimension to the spanmetrics connector so
the three stages split into separate metric series (3 bounded values).
- dashboard: add a "Tx Apply Pipeline" section to transaction-overview
with rate, p95 latency, and failure-rate panels grouped by stage, plus
a `stage` template variable. Panels follow the existing config (node
filter, exported_instance legends, Title Case, axis labels).
- The failure panel filters ter_result != tesSUCCESS rather than span
status, because a failing ter code completes the span normally — only
thrown exceptions set an error status. This matches the existing
"Transaction Results by Type" panel convention.
- docs: document the spans, attributes, and stage dimension in the data
collection reference and runbook, including the sampling caveat that
span-derived metrics inherit tracer head-sampling and undercount at
sampling_ratio < 1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Build fixes in PeerImp.cpp:
- Rename duplicate `span` variable to `consSpan` in proposal and
validation handlers to avoid redefinition error
- Fix `->` on non-pointer SpanGuard (now correctly on shared_ptr)
- Fix move-only type copy in lambda capture
Clang-tidy fixes:
- Concatenate nested namespaces in LedgerSpanNames.h and PeerSpanNames.h
- Add missing SpanNames.h includes in BuildLedger.cpp, LedgerMaster.cpp,
PeerImp.cpp for direct seg:: symbol usage
- Add missing <chrono> and <cstdint> includes in BuildLedger.cpp
- Remove unused Feature.h include from BuildLedger.cpp
Rename check fix:
- Run docs.sh to rename rippled_ metric prefixes to xrpld_ in
09-data-collection-reference.md and telemetry-runbook.md
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Prettier formatting for markdown docs and OTelCollector header
- docs.sh rippled→xrpld renames in OTelCollector.cpp comments/strings
- Updated levelization ordering with new dependency edges
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The StatsD receiver config was lost during a branch rebase (--ours
conflict resolution dropped it). Re-add the statsd receiver to the
OTel Collector config and wire it into the metrics pipeline so
beast::insight UDP metrics flow to Prometheus.
Also fixes:
- Metric prefix mismatch: docs used xrpld_ but dashboards/tests use
rippled_ — align all documentation to match the runnable stack
- Remove phantom Peer_Disconnects_Charges from docs (plain atomic,
not a beast::insight gauge)
- Remove premature .codecov.yml exclusions for Phase 7 OTelCollector
files that don't exist on this branch
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace remaining rippled/Ripple references with xrpld/XRPL in
data collection reference, implementation phases, and runbook docs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Integrate the existing StatsD metrics pipeline (beast::insight) into
the OpenTelemetry observability stack and add new trace spans for
ledger build/store/validate and peer proposal/validation receive.
Phase 5b — Ledger, peer, and transaction spans:
- Add ledger.build span with close time attributes in BuildLedger.cpp
- Add tx.apply span with tx_count/tx_failed in BuildLedger.cpp
- Add ledger.store and ledger.validate spans in LedgerMaster.cpp
- Add peer.proposal.receive span with trusted attribute in PeerImp.cpp
- Add peer.validation.receive span with ledger_hash, full, trusted
attributes in PeerImp.cpp
- Add ledger-operations and peer-network Grafana dashboards
Phase 6 — StatsD metrics integration:
- Add StatsD UDP receiver (port 8125) to OTel Collector
- Add 5 StatsD Grafana dashboards: node health, network traffic,
overlay traffic detail, ledger data sync, RPC pathfinding
- Add 09-data-collection-reference.md cataloging all metrics/spans
- Update existing dashboards with new span panels
- Expand telemetry runbook and integration test script
- Add codecov exclusions for telemetry modules
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bridge the existing beast::insight gauge for resource-limit peer
disconnects (peerDisconnectsCharges_) into the StatsD metric inventory.
Part of the external dashboard parity initiative.
See docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
