Add a Deployment Tiers section covering the four filtering dimensions
(node, service, network, environment), who owns each attribute, the
collector's upsert-vs-insert rule, per-tier collector values, and how the
tier labels reach metrics. This is the operator-facing home for the
deployment-tier feature.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The consensus.proposal.receive and consensus.validation.receive spans record
the trust flag under proposal_trusted / validation_trusted (shared with the
peer spans of the same message type), not a bare `trusted`. Update the span
reference table to match.
Bring phase-4 forward into phase 5 (docs & deployment). Phase 5 owns the
operator runbook (docs/telemetry-runbook.md).
Conflict resolution:
- 05-configuration-reference.md: took phase-4's code-block-free prose for the
config-parser section; the mTLS options (tls_client_cert/tls_client_key)
remain documented in the §5.1.2 options table.
Phase-5-owned naming fix:
- docs/telemetry-runbook.md: converted 20 dotted xrpl.<domain>. attribute keys
in the Span Reference tables to the underscore convention (tx_hash, peer_id,
ledger_seq, consensus_mode/round/round_id/ledger_id, tx_id). Span NAMES stay
dotted (rule 5). `trusted` left as-is — verified it matches the code constant
ConsensusSpanNames.h (the proposal_trusted/validation_trusted split is a
separate, not-yet-applied code change).
End-to-end reconciliation: every span-attribute token in the runbook (L5) now
matches an L1 *SpanNames.h constant. The xrpl_<domain>_<field> tokens are the
spanmetrics-derived Prometheus labels (documented mapping), not span attrs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Head sampling is now fixed at 1.0 and not configurable. Drop the
sampling_ratio config row and redirect volume-reduction guidance to
collector-side tail sampling. trace_peer=0 stays as the high-volume opt-out.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reflect the new on-by-default behavior in the config defaults table. The
volume-reduction guidance (High memory usage, Production mainnet tuning)
intentionally keeps trace_peer=0 as an opt-out recommendation for
high-traffic deployments.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Flip the tracePeer Setup default and the trace_peer config parser
default from off to on, and update the example config and build doc to
match. Peer spans record only peer_id (a node-local numeric connection
id) plus trust/ledger metadata — no IP addresses or public keys — so
the privacy concern behind disabling it does not apply. The high-volume
characteristic is retained in the docs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the apply-pipeline stage spans (tx.preflight, tx.preclaim,
tx.transactor) added on phase-3 through the observability stack so the
spanmetrics connector produces per-stage RED metrics without any native
instruments.
- collector: add the `stage` dimension to the spanmetrics connector so
the three stages split into separate metric series (3 bounded values).
- dashboard: add a "Tx Apply Pipeline" section to transaction-overview
with rate, p95 latency, and failure-rate panels grouped by stage, plus
a `stage` template variable. Panels follow the existing config (node
filter, exported_instance legends, Title Case, axis labels).
- The failure panel filters ter_result != tesSUCCESS rather than span
status, because a failing ter code completes the span normally — only
thrown exceptions set an error status. This matches the existing
"Transaction Results by Type" panel convention.
- docs: document the spans, attributes, and stage dimension in the data
collection reference and runbook, including the sampling caveat that
span-derived metrics inherit tracer head-sampling and undercount at
sampling_ratio < 1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The spanmetrics connector dimension was `xrpl.consensus.mode`, but the code
emits the span attribute under the bare key `consensus_mode` (matching every
other dimension after the Phase 6 rename). The mismatch left the
`xrpl_consensus_mode` Prometheus label empty, so the Consensus Health
"Consensus Mode Over Time" panel and the `$consensus_mode` template variable
(which filters every panel) matched no live series.
- otel-collector-config.yaml: dimension `xrpl.consensus.mode` -> `consensus_mode`
- consensus-health.json: 11 label refs `xrpl_consensus_mode` -> `consensus_mode`
(the `$consensus_mode` Grafana variable name is unchanged)
- telemetry-runbook.md: refresh the stale spanmetrics label table to the bare
names actually emitted (command/rpc_status/consensus_mode/local/
proposal_trusted/validation_trusted), fix dotted->bare attribute names in
span tables and TraceQL examples (tx_hash, ledger_seq, consensus_round_id,
consensus_ledger_id, consensus_round, tx_id event attr), correct the
consensus_round_id query to int (not quoted string), and fix the
load_type value query ("exception_rpc" -> "exceptioned RPC").
Verified against the live stack: Tempo span tags confirm bare attribute keys
(consensus_mode, ledger_seq, tx_hash, ...); the populated xrpl_consensus_mode
series in Prometheus is stale retained data from an older build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve runbook conflict: keep both phase 6 ledger/peer span tables
AND new insights/sample queries section from the enrichment work.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds comprehensive "Insights and Sample Queries" section showing operators
what questions they can answer with the newly-added span attributes:
- Transaction workflow analysis (filter by tx_type, fee, ter_result)
- TxQ health (txq_status, ledger_changed)
- RPC debugging (is_batch, request_payload_size, load_type)
- PathFinding performance (dest_currency, num_source_assets)
- Consensus health (consensus_state, is_bow_out, disputes_count)
- Cross-subsystem correlation examples
Also updates all span reference tables with the new attributes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
