# xrpld Telemetry Operator Runbook ## Overview xrpld supports OpenTelemetry distributed tracing to provide visibility into RPC requests, transaction processing, and consensus rounds. This runbook covers operating a running node and querying its traces. For building xrpld with telemetry support and the internal architecture, see [build/telemetry.md](build/telemetry.md). ## Quick Start ### 1. Start the observability stack ```bash docker compose -f docker/telemetry/docker-compose.yml up -d ``` This starts: - **OTel Collector** on ports 4317 (gRPC), 4318 (HTTP), and 13133 (health) - **Tempo** trace storage on http://localhost:3200 - **Grafana** on http://localhost:3000 (Tempo pre-configured as datasource) ### 2. Enable telemetry in xrpld Add to your `xrpld.cfg`: ```ini [telemetry] enabled=1 endpoint=http://localhost:4318/v1/traces ``` ### 3. Build with telemetry support ```bash conan install . --build=missing -o telemetry=True cmake --preset default -Dtelemetry=ON cmake --build --preset default ``` ## Configuration Reference | Option | Default | Description | | -------------------------- | --------------------------------- | --------------------------------------------------------- | | `enabled` | `0` | Master switch for telemetry | | `endpoint` | `http://localhost:4318/v1/traces` | OTLP/HTTP endpoint | | `service_name` | `xrpld` | OpenTelemetry service name resource attribute | | `service_instance_id` | node public key | OpenTelemetry service instance ID resource attribute | | `trace_rpc` | `1` | Enable RPC request tracing | | `trace_transactions` | `1` | Enable transaction tracing | | `trace_consensus` | `1` | Enable consensus tracing | | `trace_peer` | `1` | Enable peer message tracing (high volume) | | `trace_ledger` | `1` | Enable ledger tracing | | `consensus_trace_strategy` | `deterministic` | Consensus trace ID strategy (`deterministic` or `random`) | | `batch_size` | `512` | Max spans per batch export | | `batch_delay_ms` | `5000` | Delay between batch exports | | `max_queue_size` | `2048` | Max spans queued before dropping | | `use_tls` | `0` | Use TLS for exporter connection | | `tls_ca_cert` | (empty) | Path to CA certificate bundle | | `tls_client_cert` | (empty) | Client cert (PEM) for mutual TLS; empty = one-way TLS | | `tls_client_key` | (empty) | Private key (PEM) for `tls_client_cert` | ## Span Reference All spans instrumented in xrpld, grouped by subsystem: ### RPC Spans | Span Name | Source File | Attributes | Description | | -------------------- | ----------------- | ----------------------------------------------------------- | ----------------------------------------------------- | | `rpc.http_request` | ServerHandler.cpp | `request_payload_size` | Top-level HTTP RPC request | | `rpc.ws_upgrade` | ServerHandler.cpp | — | WebSocket upgrade handshake | | `rpc.ws_message` | ServerHandler.cpp | `command` | WebSocket RPC message | | `rpc.process` | ServerHandler.cpp | `is_batch`, `batch_size` | RPC processing (child of rpc.http_request/ws_message) | | `rpc.command.` | RPCHandler.cpp | `command`, `version`, `rpc_role`, `rpc_status`, `load_type` | Per-command span (e.g., `rpc.command.server_info`) | ### Transaction Spans | Span Name | Source File | Attributes | Description | | ------------ | -------------- | --------------------------------------------------------------------------------- | ------------------------------------- | | `tx.process` | NetworkOPs.cpp | `tx_hash`, `local`, `path`, `tx_type`, `fee`, `sequence`, `ter_result`, `applied` | Transaction submission and processing | | `tx.receive` | PeerImp.cpp | `peer_id`, `tx_hash`, `tx_type`, `peer_version`, `suppressed`, `tx_status` | Transaction received from peer relay | ### Transaction Queue Spans | Span Name | Source File | Attributes | Description | | ------------------ | ----------- | -------------------------------------------------------- | -------------------------------------------------- | | `txq.enqueue` | TxQ.cpp | `tx_hash`, `tx_type` | Transaction enqueue decision (child of tx.process) | | `txq.apply_direct` | TxQ.cpp | -- | Direct apply attempt (bypassing queue) | | `txq.batch_clear` | TxQ.cpp | -- | Batch clear of queued transactions for an account | | `txq.accept` | TxQ.cpp | `queue_size`, `ledger_changed` | Ledger-close accept loop over queued transactions | | `txq.accept_tx` | TxQ.cpp | `tx_hash`, `retries_remaining`, `ter_code`, `txq_status` | Per-transaction apply during accept | | `txq.cleanup` | TxQ.cpp | `ledger_seq` | Post-close cleanup of expired queue entries | ### Consensus Spans | Span Name | Source File | Attributes | Description | | ------------------------------ | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `consensus.round` | RCLConsensus.cpp | `consensus_ledger_id`, `ledger_seq`, `consensus_mode`, `trace_strategy`, `consensus_round_id` | Root span for a consensus round (deterministic or random trace ID) | | `consensus.phase.open` | Consensus.h | -- | Open phase duration (child of round) | | `consensus.proposal.send` | RCLConsensus.cpp | `consensus_round`, `is_bow_out` | Consensus proposal broadcast | | `consensus.ledger_close` | RCLConsensus.cpp | `ledger_seq`, `consensus_mode` | Ledger close event | | `consensus.establish` | Consensus.h | `converge_percent`, `establish_count`, `proposers` | Establish phase duration (child of round) | | `consensus.update_positions` | Consensus.h | `converge_percent`, `proposers`, `disputes_count` | Position update and dispute resolution (see Events below) | | `consensus.check` | Consensus.h | `agree_count`, `disagree_count`, `converge_percent`, `have_close_time_consensus`, `threshold_percent`, `consensus_result` | Consensus threshold check | | `consensus.accept` | RCLConsensus.cpp | `proposers`, `round_time_ms`, `quorum`, `disputes_count`, `consensus_state` | Ledger accepted by consensus | | `consensus.accept.apply` | RCLConsensus.cpp | `ledger_seq`, `close_time`, `close_time_correct`, `close_resolution_ms`, `consensus_state`, `proposing`, `round_time_ms`, `parent_close_time`, `close_time_self`, `close_time_vote_bins`, `resolution_direction`, `tx_count` | Ledger application with close time details (see Events below) | | `consensus.validation.send` | RCLConsensus.cpp | `ledger_seq`, `proposing`, `ledger_hash`, `full_validation`, `validation_sign_time` | Validation sent after accept (follows-from link) | | `consensus.mode_change` | RCLConsensus.cpp | `mode_old`, `mode_new` | Consensus mode transition | | `consensus.proposal.receive` | PeerImp.cpp | `proposal_trusted`, `consensus_round` | Proposal received from peer (extracts parent context from TraceContext when present; falls back to standalone span for older peers) | | `consensus.validation.receive` | PeerImp.cpp | `validation_trusted`, `ledger_seq` | Validation received from peer (extracts parent context from TraceContext when present; falls back to standalone span for older peers) | #### Consensus Span Events | Parent Span | Event Name | Event Attributes | Description | | ---------------------------- | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------- | | `consensus.update_positions` | `dispute.resolve` | `tx_id`, `dispute_our_vote`, `dispute_yays`, `dispute_nays` | Emitted per dispute when votes are tallied | | `consensus.accept.apply` | `tx.included` | `tx_id` | Emitted per transaction included in the accepted ledger | #### Close Time Queries (Tempo TraceQL) ``` # Find rounds where validators disagreed on close time {name="consensus.accept.apply"} | close_time_correct = false # Find consensus failures (moved_on) {name="consensus.accept.apply"} | consensus_state = "moved_on" # Find slow ledger applications (>5s) {name="consensus.accept.apply"} | duration > 5s # Find specific ledger's consensus details {name="consensus.accept.apply"} | ledger_seq = 92345678 # Find all spans in a consensus round (deterministic trace strategy) {name="consensus.round"} | consensus_round_id = "" # Find dispute resolutions {name="consensus.update_positions"} >> {event:name="dispute.resolve"} ``` ## Insights and Sample Queries This section shows what questions you can now answer using the enriched span attributes, with example Tempo TraceQL queries. ### Transaction Workflow Analysis ``` # Find all AMM transactions (AMMDeposit, AMMWithdraw, AMMCreate, etc.) {name="tx.process"} | tx_type =~ "AMM.*" # Find Payment transactions that failed {name="tx.process"} | tx_type = "Payment" && ter_result != "tesSUCCESS" # Compare latency of different transaction types {name="tx.process"} | tx_type = "OfferCreate" {name="tx.process"} | tx_type = "Payment" # Find high-fee transactions (fee > 1 XRP = 1000000 drops) {name="tx.process"} | fee > 1000000 # Find transactions that were not applied {name="tx.process"} | applied = false # Trace a specific transaction by type across the network {name=~"tx\\..*"} | tx_type = "NFTokenMint" ``` ### Transaction Queue Health ``` # Find transactions rejected from the queue {name="txq.accept_tx"} | txq_status = "failed" # Which transaction types get queued most often? {name="txq.enqueue"} | tx_type = "Payment" {name="txq.enqueue"} | tx_type = "OfferCreate" # Find ledger closes that applied queued transactions {name="txq.accept"} | ledger_changed = true # Find transactions that exhausted retries {name="txq.accept_tx"} | txq_status = "retried" && retries_remaining = 0 ``` ### RPC Debugging ``` # Find batch RPC requests {name="rpc.process"} | is_batch = true # Find large RPC payloads (>100KB) {name="rpc.http_request"} | request_payload_size > 100000 # Find resource-heavy RPC commands (by load_type) {name=~"rpc.command.*"} | load_type = "exception_rpc" # Find a specific WebSocket command {name="rpc.ws_message"} | command = "subscribe" # Find slow pathfinding with many source assets {name="pathfind.discover"} | pathfind_num_source_assets > 10 ``` ### PathFinding Performance ``` # Find pathfinding for specific currencies {name="pathfind.compute"} | pathfind_dest_currency = "USD" # Find expensive pathfinding (many source assets to explore) {name="pathfind.discover"} | pathfind_num_source_assets > 20 # Find large pathfinding requests {name="pathfind.compute"} | duration > 1s ``` ### Consensus Health ``` # Find rounds where consensus timed out (expired) {name="consensus.accept"} | consensus_state = "expired" # Find rounds where we moved on without full agreement {name="consensus.accept"} | consensus_state = "moved_on" # Find rounds with many disputes {name="consensus.accept"} | disputes_count > 5 # Find bow-out proposals (node resigned from round) {name="consensus.proposal.send"} | is_bow_out = true # Correlate validation with its ledger {name="consensus.validation.send"} | ledger_hash = "" # Find rounds where validators disagreed on close time {name="consensus.accept.apply"} | close_time_correct = false ``` ### Cross-Subsystem Correlation ``` # Follow a transaction from receive through queue to ledger {name=~"tx\\..*|txq\\..*"} | tx_type = "Payment" && duration > 500ms # Find all NFT-related activity {name=~"tx\\..*|txq\\..*"} | tx_type =~ "NFToken.*" # Find consensus rounds with slow transactions {name="consensus.accept"} | round_time_ms > 5000 ``` ### Where to Look (Quick Reference) | Question | Span | Key Attributes | | ----------------------------------- | --------------------------- | ------------------------------ | | "Which tx type is slowest?" | `tx.process` | `tx_type` + duration | | "Why was my tx rejected?" | `tx.process` | `ter_result`, `applied` | | "Is the TxQ backing up?" | `txq.accept` | `queue_size`, `ledger_changed` | | "Why was my tx dropped from queue?" | `txq.accept_tx` | `txq_status`, `ter_code` | | "Are batch requests a problem?" | `rpc.process` | `is_batch`, `batch_size` | | "Which RPC is expensive?" | `rpc.command.*` | `load_type`, duration | | "Did consensus stall?" | `consensus.check` | `consensus_stalled` | | "Was consensus outcome normal?" | `consensus.accept` | `consensus_state` | | "Did a validator bow out?" | `consensus.proposal.send` | `is_bow_out` | | "Which ledger was validated?" | `consensus.validation.send` | `ledger_hash` | --- ## Cross-Node Trace Propagation xrpld propagates trace context across nodes via protobuf `TraceContext` fields embedded in peer-to-peer messages. When Node A sends a transaction, proposal, or validation, it injects its active span's trace/span IDs into the protobuf message. Node B extracts that context on receipt and creates a child span, linking the two nodes into a single distributed trace. ### How It Works ``` Node A (sender) Node B (receiver) +-----------------------------+ +-------------------------------+ | tx.process / consensus.* | | PeerImp::onMessage() | | | | | | | | v | | v | | SpanGuard::getTraceBytes() | | extract TraceContext from | | | | | protobuf message | | v | send | | | | injectSpanContext() --------|--------->| v | | sets TraceContext fields | proto | txReceiveSpan() | | (trace_id, span_id, flags) | msg | proposalReceiveSpan() | +-----------------------------+ | validationReceiveSpan() | | | | | v | | child span with parent link | +-------------------------------+ ``` ### Send-Side Injection | Message Type | Injection Point | Mechanism | | ------------- | -------------------------- | ------------------------------------------ | | TMTransaction | `NetworkOPs::apply()` | Injects `tx.process` span into relay msg | | TMProposeSet | `RCLConsensus::propose()` | Injects active context into proposal msg | | TMValidation | `RCLConsensus::validate()` | Injects active context into validation msg | ### Receive-Side Extraction | Message Type | Extraction Point | Helper Function | | ------------- | ----------------------------------- | -------------------------------------------------- | | TMTransaction | `PeerImp::onMessage(TMTransaction)` | `TxTracing::txReceiveSpan()` | | TMProposeSet | `PeerImp::onMessage(TMProposeSet)` | `ConsensusReceiveTracing::proposalReceiveSpan()` | | TMValidation | `PeerImp::onMessage(TMValidation)` | `ConsensusReceiveTracing::validationReceiveSpan()` | ### Key Files | File | Role | | ------------------------------------------------- | ----------------------------------------------- | | `src/xrpld/telemetry/PropagationHelpers.h` | `injectSpanContext()` — SpanGuard to protobuf | | `include/xrpl/telemetry/TraceContextPropagator.h` | OTel context <-> protobuf conversion primitives | | `src/xrpld/telemetry/ConsensusReceiveTracing.h` | Proposal/validation receive span factories | | `src/xrpld/telemetry/TxTracing.h` | Transaction receive span factory | ### Backwards Compatibility Older peers that do not populate `TraceContext` fields in their messages will simply produce empty trace bytes on the receive side. The extraction helpers detect this and create standalone (root) spans instead of child spans. No errors are logged and no data is lost — the receive span is still created with all its normal attributes, it just lacks a cross-node parent link. ### Example Tempo Queries ``` # Find cross-node transaction traces (tx.process -> tx.receive across nodes) {name="tx.receive"} && status != error # Find proposals received with cross-node parent context {name="consensus.proposal.receive"} && nestedSetParent > 0 # Trace a transaction across the network by its hash {name=~"tx\\..*"} | tx_hash = "" # Find all spans in a cross-node consensus trace {rootServiceName="xrpld"} | consensus_round_id = "" # Compare latency between sender and receiver for validations {name="consensus.validation.send" || name="consensus.validation.receive"} ``` ## Prometheus Metrics (Spanmetrics) The OTel Collector's spanmetrics connector automatically derives RED (Rate, Errors, Duration) metrics from every span. No custom metrics code is needed in xrpld. ### Generated Metric Names | Prometheus Metric | Type | Description | | -------------------------------------------------- | --------- | ---------------------------- | | `traces_span_metrics_calls_total` | Counter | Total span invocations | | `traces_span_metrics_duration_milliseconds_bucket` | Histogram | Latency distribution buckets | | `traces_span_metrics_duration_milliseconds_count` | Histogram | Latency observation count | | `traces_span_metrics_duration_milliseconds_sum` | Histogram | Cumulative latency | ### Metric Labels Every metric carries these standard labels: | Label | Source | Example | | -------------- | ------------------ | ---------------------------------------- | | `span_name` | Span name | `rpc.command.server_info` | | `status_code` | Span status | `STATUS_CODE_UNSET`, `STATUS_CODE_ERROR` | | `service_name` | Resource attribute | `xrpld` | | `span_kind` | Span kind | `SPAN_KIND_INTERNAL` | Additionally, span attributes configured as dimensions in the collector become metric labels (dots → underscores): | Span Attribute | Metric Label | Applies To | | ---------------- | --------------------- | ------------------------------ | | `command` | `xrpl_rpc_command` | `rpc.command.*` spans | | `rpc_status` | `xrpl_rpc_status` | `rpc.command.*` spans | | `consensus_mode` | `xrpl_consensus_mode` | `consensus.ledger_close` spans | | `local` | `xrpl_tx_local` | `tx.process` spans | ### Histogram Buckets Configured in `otel-collector-config.yaml`: ``` 1ms, 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 5s ``` ## Deployment Tiers Multiple xrpld instances can send telemetry to per-tier collectors that all forward to one Grafana stack. Four resource attributes segregate the data so one dashboard set serves every deployment: | Dimension | Attribute | Set by | Example values | | ----------- | ------------------------ | ---------- | ------------------------------ | | Node | `service.instance.id` | xrpld cfg | `alice-laptop`, `ci-runner-7` | | Service | `service.name` | xrpld cfg | `xrpld`, `xrpld-validator` | | Network | `xrpl.network.type` | xrpld node | `mainnet`, `testnet`, `devnet` | | Environment | `deployment.environment` | collector | `local`, `test`, `ci`, `prod` | Dashboards expose these as the template variables `$node`, `$service_name`, `$xrpl_network_type`, and `$deployment_environment` (each variable name matches its Prometheus label). Select them top-down — environment → network → service → node. Selecting **All** matches every value, including series lacking the label, so mixed old/new data never disappears. ### Who owns which attribute - **Node and service** come from xrpld config (`service_instance_id`, `service_name`). Unique per process. - **Network** is a property of the chain the node joined; the node derives it from `[network_id]` and stamps `xrpl.network.type` on all three signals. - **Environment** is a property of where the collector runs; each collector serves one environment and stamps it. ### The upsert vs insert rule The collector's `resource/tier` processor uses two actions on purpose: - `deployment.environment` → **`upsert`** (overwrite). The collector _is_ the environment, so it is authoritative. - `xrpl.network.type` → **`insert`** (fill only if absent). The node knows its real network, so the collector must not overwrite it — `insert` only supplies a value when the source did not (e.g. an older xrpld build). This is what lets a local node connected to mainnet report `network=mainnet`, not the collector's default. ### Configuring a collector for a tier Each tier runs its own collector. Set the two values in the `resource/tier` processor of the collector config (`otel-collector-config.yaml` for local backends, `otel-collector-config.grafanacloud.yaml` for Grafana Cloud): ```yaml processors: resource/tier: attributes: - key: deployment.environment value: # local | test | ci | prod action: upsert - key: xrpl.network.type value: # mainnet | testnet | devnet (fallback only) action: insert ``` Suggested per-tier values: | Collector | `deployment.environment` | `xrpl.network.type` (fallback) | | ------------------- | ------------------------ | ------------------------------ | | Developer laptop | `local` | `devnet` | | Test machines | `test` | `testnet` | | CI runs | `ci` | `testnet` | | Production observer | `prod` | `mainnet` | The `xrpl.network.type` value is only a fallback: when the node stamps its own network (all current builds do), the node's value wins. Set it to the network the collector most commonly serves. ### How the tier labels reach metrics Resource attributes do not become Prometheus labels automatically. Two collector settings make it work, both already enabled: - `prometheus.resource_to_telemetry_conversion: enabled: true` promotes resource attributes to metric labels on the local scrape surface. - `spanmetrics.resource_metrics_key_attributes` lists the tier attributes so span-derived series stay grouped per node and tier. Traces and logs carry resource attributes natively; Grafana Cloud ingests all three signals' attributes over OTLP directly. ## Grafana Dashboards Three dashboards are pre-provisioned in `docker/telemetry/grafana/dashboards/`: ### RPC Performance (`rpc-performance`) | Panel | Type | PromQL | Labels Used | | --------------------------- | ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- | | RPC Request Rate by Command | timeseries | `sum by (xrpl_rpc_command) (rate(traces_span_metrics_calls_total{span_name=~"rpc.command.*"}[5m]))` | `xrpl_rpc_command` | | RPC Latency p95 by Command | timeseries | `histogram_quantile(0.95, sum by (le, xrpl_rpc_command) (rate(traces_span_metrics_duration_milliseconds_bucket{span_name=~"rpc.command.*"}[5m])))` | `xrpl_rpc_command` | | RPC Error Rate | bargauge | Error spans / total spans × 100, grouped by `xrpl_rpc_command` | `xrpl_rpc_command`, `status_code` | | RPC Latency Heatmap | heatmap | `sum(increase(traces_span_metrics_duration_milliseconds_bucket{span_name=~"rpc.command.*"}[5m])) by (le)` | `le` (bucket boundaries) | ### Transaction Overview (`transaction-overview`) | Panel | Type | PromQL | Labels Used | | --------------------------------- | ---------- | -------------------------------------------------------------------------------------------- | --------------- | | Transaction Processing Rate | timeseries | `rate(traces_span_metrics_calls_total{span_name="tx.process"}[5m])` and `tx.receive` | `span_name` | | Transaction Processing Latency | timeseries | `histogram_quantile(0.95 / 0.50, ... {span_name="tx.process"})` | — | | Transaction Path Distribution | piechart | `sum by (xrpl_tx_local) (rate(traces_span_metrics_calls_total{span_name="tx.process"}[5m]))` | `xrpl_tx_local` | | Transaction Receive vs Suppressed | timeseries | `rate(traces_span_metrics_calls_total{span_name="tx.receive"}[5m])` | — | ### Consensus Health (`consensus-health`) | Panel | Type | PromQL | Labels Used | | ----------------------------- | ---------- | ---------------------------------------------------------------------------------- | ----------- | | Consensus Round Duration | timeseries | `histogram_quantile(0.95 / 0.50, ... {span_name="consensus.accept"})` | — | | Consensus Proposals Sent Rate | timeseries | `rate(traces_span_metrics_calls_total{span_name="consensus.proposal.send"}[5m])` | — | | Ledger Close Duration | timeseries | `histogram_quantile(0.95, ... {span_name="consensus.ledger_close"})` | — | | Validation Send Rate | stat | `rate(traces_span_metrics_calls_total{span_name="consensus.validation.send"}[5m])` | — | | Ledger Apply Duration | timeseries | `histogram_quantile(0.95 / 0.50, ... {span_name="consensus.accept.apply"})` | — | | Close Time Agreement | timeseries | `rate(traces_span_metrics_calls_total{span_name="consensus.accept.apply"}[5m])` | — | ### Span → Metric → Dashboard Summary | Span Name | Prometheus Metric Filter | Grafana Dashboard | | ------------------------------ | -------------------------------------------- | --------------------------------------------- | | `rpc.http_request` | `{span_name="rpc.http_request"}` | -- (available but not paneled) | | `rpc.ws_upgrade` | `{span_name="rpc.ws_upgrade"}` | -- (available but not paneled) | | `rpc.ws_message` | `{span_name="rpc.ws_message"}` | -- (available but not paneled) | | `rpc.process` | `{span_name="rpc.process"}` | -- (available but not paneled) | | `rpc.command.*` | `{span_name=~"rpc.command.*"}` | RPC Performance (all 4 panels) | | `tx.process` | `{span_name="tx.process"}` | Transaction Overview (3 panels) | | `tx.receive` | `{span_name="tx.receive"}` | Transaction Overview (2 panels) | | `txq.enqueue` | `{span_name="txq.enqueue"}` | -- (available but not paneled) | | `txq.apply_direct` | `{span_name="txq.apply_direct"}` | -- (available but not paneled) | | `txq.batch_clear` | `{span_name="txq.batch_clear"}` | -- (available but not paneled) | | `txq.accept` | `{span_name="txq.accept"}` | -- (available but not paneled) | | `txq.accept_tx` | `{span_name="txq.accept_tx"}` | -- (available but not paneled) | | `txq.cleanup` | `{span_name="txq.cleanup"}` | -- (available but not paneled) | | `consensus.round` | `{span_name="consensus.round"}` | -- (available but not paneled) | | `consensus.phase.open` | `{span_name="consensus.phase.open"}` | -- (available but not paneled) | | `consensus.establish` | `{span_name="consensus.establish"}` | -- (available but not paneled) | | `consensus.update_positions` | `{span_name="consensus.update_positions"}` | -- (available but not paneled) | | `consensus.check` | `{span_name="consensus.check"}` | -- (available but not paneled) | | `consensus.accept` | `{span_name="consensus.accept"}` | Consensus Health (Round Duration) | | `consensus.proposal.send` | `{span_name="consensus.proposal.send"}` | Consensus Health (Proposals Rate) | | `consensus.ledger_close` | `{span_name="consensus.ledger_close"}` | Consensus Health (Close Duration) | | `consensus.validation.send` | `{span_name="consensus.validation.send"}` | Consensus Health (Validation Rate) | | `consensus.accept.apply` | `{span_name="consensus.accept.apply"}` | Consensus Health (Apply Duration, Close Time) | | `consensus.mode_change` | `{span_name="consensus.mode_change"}` | -- (available but not paneled) | | `consensus.proposal.receive` | `{span_name="consensus.proposal.receive"}` | -- (available but not paneled) | | `consensus.validation.receive` | `{span_name="consensus.validation.receive"}` | -- (available but not paneled) | ## Troubleshooting ### No traces appearing in Tempo 1. Check xrpld logs for `Telemetry starting` message 2. Verify `enabled=1` in the `[telemetry]` config section 3. Test collector connectivity: `curl -v http://localhost:4318/v1/traces` 4. Check collector logs: `docker compose -f docker/telemetry/docker-compose.yml logs otel-collector` 5. Verify Tempo is receiving data: open Grafana → Explore → select Tempo datasource → search by `service.name = xrpld` 6. Check Tempo logs: `docker compose -f docker/telemetry/docker-compose.yml logs tempo` ### High memory usage - Reduce trace volume with collector-side tail sampling (xrpld head sampling is fixed at 1.0 and is not configurable) - Reduce `max_queue_size` and `batch_size` - Disable high-volume trace categories: `trace_peer=0` ### Collector connection failures - Verify endpoint URL matches collector address - Check firewall rules for ports 4317/4318 - If using TLS, verify certificate path with `tls_ca_cert` ## Performance Tuning | Scenario | Recommendation | | ------------------------ | --------------------------------------------------------- | | Production mainnet | `trace_peer=0`; reduce volume via collector tail sampling | | Testnet/devnet | Full tracing (head sampling fixed at 1.0) | | Debugging specific issue | Full tracing (head sampling fixed at 1.0) | | High-throughput node | Increase `batch_size=1024`, `max_queue_size=4096` | ## Disabling Telemetry Set `enabled=0` in config (runtime disable) or build without the flag: ```bash cmake --preset default -Dtelemetry=OFF ``` When telemetry is compiled out, all trace macros expand to no-ops with zero overhead.