rippled/OpenTelemetryPlan/09-data-collection-reference.md

Observability Data Collection Reference

Audience: Developers and operators. This is the single source of truth for all telemetry data collected by xrpld's observability stack.

Related docs: docs/telemetry-runbook.md (operator runbook with alerting and troubleshooting) | 03-implementation-strategy.md (code structure and performance optimization) | 04-code-samples.md (C++ instrumentation examples)

Data Flow Overview

graph LR
    subgraph xrpldNode["xrpld Node"]
        A["Trace Macros<br/>XRPL_TRACE_SPAN<br/>(OTLP/HTTP exporter)"]
        B["beast::insight<br/>StatsD metrics<br/>(UDP sender)"]
    end

    subgraph collector["OTel Collector  :4317 / :4318 / :8125"]
        direction TB
        R1["OTLP Receiver<br/>:4317 gRPC  |  :4318 HTTP"]
        R2["StatsD Receiver<br/>:8125 UDP"]
        BP["Batch Processor<br/>timeout 1s, batch 100"]
        SM["SpanMetrics Connector<br/>derives RED metrics<br/>from trace spans"]

        R1 --> BP
        BP --> SM
    end

    subgraph backends["Trace Backend"]
        D["Grafana Tempo  :3200<br/>TraceQL search &<br/>S3/GCS long-term storage"]
    end

    subgraph metrics["Metrics Stack"]
        E["Prometheus  :9090<br/>scrapes :8889<br/>span-derived + StatsD metrics"]
    end

    subgraph viz["Visualization"]
        F["Grafana  :3000<br/>10 dashboards"]
    end

    A -->|"OTLP/HTTP :4318<br/>(traces + attributes)"| R1
    B -->|"UDP :8125<br/>(gauges, counters, timers)"| R2

    BP -->|"OTLP/gRPC :4317"| D

    SM -->|"span_calls_total<br/>span_duration_ms<br/>(6 dimension labels)"| E
    R2 -->|"xrpld_* gauges<br/>xrpld_* counters<br/>xrpld_* summaries"| E

    E -->|"Prometheus<br/>data source"| F
    D -->|"Tempo<br/>data source"| F

    style A fill:#4a90d9,color:#fff,stroke:#2a6db5
    style B fill:#d9534f,color:#fff,stroke:#b52d2d
    style R1 fill:#5cb85c,color:#fff,stroke:#3d8b3d
    style R2 fill:#5cb85c,color:#fff,stroke:#3d8b3d
    style BP fill:#449d44,color:#fff,stroke:#2d6e2d
    style SM fill:#449d44,color:#fff,stroke:#2d6e2d
    style D fill:#f0ad4e,color:#000,stroke:#c78c2e
    style E fill:#f0ad4e,color:#000,stroke:#c78c2e
    style F fill:#5bc0de,color:#000,stroke:#3aa8c1
    style xrpldNode fill:#1a2633,color:#ccc,stroke:#4a90d9
    style collector fill:#1a3320,color:#ccc,stroke:#5cb85c
    style backends fill:#332a1a,color:#ccc,stroke:#f0ad4e
    style metrics fill:#332a1a,color:#ccc,stroke:#f0ad4e
    style viz fill:#1a2d33,color:#ccc,stroke:#5bc0de

There are two independent telemetry pipelines entering a single OTel Collector:

1. OpenTelemetry Traces — Distributed spans with attributes, exported via OTLP/HTTP (:4318) to the collector's OTLP Receiver. The Batch Processor groups spans (1s timeout, batch size 100) before forwarding to trace backends. The SpanMetrics Connector derives RED metrics (rate, errors, duration) from every span and feeds them into the metrics pipeline.
2. beast::insight StatsD — System-level gauges, counters, and timers emitted as StatsD UDP packets to port :8125, ingested by the collector's StatsD Receiver, and exported alongside span-derived metrics to Prometheus.

A third, narrower metrics path exists for instruments created at their call site through the XRPL_METRIC_* macros. These use the OTel Metrics SDK directly and reach the collector's OTLP receiver rather than the StatsD receiver, so their names carry no xrpld_ prefix. See §2a. Code in libxrpl cannot use these macros and always goes through beast::insight instead.

Trace backend — The collector exports traces via OTLP/gRPC to:

• Grafana Tempo — Preferred trace backend. Supports TraceQL queries at :3200, S3/GCS object storage for cost-effective long-term trace retention, and integrates natively with Grafana.

Further reading: 00-tracing-fundamentals.md for core OpenTelemetry concepts (traces, spans, context propagation, sampling). 07-observability-backends.md for production backend selection, collector placement, and sampling strategies.

---

1. OpenTelemetry Spans

1.1 Complete Span Inventory (35 spans)

See also: 02-design-decisions.md §2.3 for naming conventions and the full span catalog with rationale. 04-code-samples.md §4.6 for span flow diagrams.

RPC Spans

Controlled by trace_rpc=1 in [telemetry] config.

Span Name	Parent	Source File	Description
rpc.http_request	—	ServerHandler.cpp	Top-level HTTP RPC request entry point
rpc.process	rpc.http_request	ServerHandler.cpp	RPC processing pipeline
rpc.ws_message	—	ServerHandler.cpp	WebSocket message handling
rpc.ws_upgrade	—	ServerHandler.cpp	WebSocket upgrade handshake (error path)
rpc.command.<name>	rpc.process	RPCHandler.cpp	Per-command span (e.g., rpc.command.server_info, rpc.command.ledger)

Where to find: Tempo → TraceQL: {resource.service.name="xrpld" && name=~"rpc.http_request|rpc.command.*"}

Grafana dashboard: RPC Performance (rpc-performance)

Transaction Spans

Controlled by trace_transactions=1 in [telemetry] config.

Span Name	Parent	Source File	Description
tx.process	—	NetworkOPs.cpp	Transaction submission entry point (local or peer-relayed)
tx.receive	—	PeerImp.cpp	Raw transaction received from peer overlay (before deduplication)
tx.apply	ledger.build	BuildLedger.cpp	Transaction set applied to new ledger during consensus
tx.preflight	—	applySteps.cpp	Stateless checks stage (stage=preflight)
tx.preclaim	—	applySteps.cpp	Ledger-aware checks stage before fee claim (stage=preclaim)
tx.transactor	—	Transactor.cpp	Apply stage — the transactor runs (stage=apply)

The three apply-pipeline spans share a deterministic trace_id derived from txID[0:16], so preflight, preclaim, and transactor for one transaction group under a single trace even though they run sequentially and often on different threads. A transaction that hard-fails preflight or preclaim never reaches the later spans — the stage attribute identifies where it stopped.

Deterministic roots are true roots. Spans with a deterministic trace_id (the tx.* apply pipeline, tx.process, tx.receive, and consensus.round) are emitted as genuine trace roots with an empty parent_span_id. The chosen trace_id is injected through a custom DeterministicIdGenerator on the SDK's no-parent branch, so there is no synthetic placeholder parent — Tempo shows a clean root, not a "root span not yet received" warning. Cross-node correlation still works because every node derives the same trace_id from the shared hash.

Log-trace correlation is retained across coroutines. OTel context storage is coroutine-aware (backed by LocalValue), so the active span travels with a coroutine across yield() and resumes on whatever thread the scheduler picks. RPC, consensus, and transaction spans therefore keep per-line log-trace correlation, and their scopes are safe across coroutine yields. Job-handoff spans — transaction apply and receive, and consensus accept — are activated inside their worker bodies rather than at enqueue, so each worker's log lines carry the span's trace context.

Where to find: Tempo → TraceQL: {resource.service.name="xrpld" && name=~"tx.process|tx.receive"} or, for the apply pipeline: {resource.service.name="xrpld" && name=~"tx.preflight|tx.preclaim|tx.transactor"}

Grafana dashboard: Transaction Overview (transaction-overview)

PathFind Spans

Controlled by trace_rpc=1 in [telemetry] config (pathfinding spans fire within RPC request handling).

Span Name	Parent	Source File	Description
pathfind.request	rpc.command.<name>	PathFind.cpp	RPC entry for path_find (doPathFind)
pathfind.compute	pathfind.request	PathRequest.cpp	Single path computation (doUpdate)
pathfind.update_all	—	PathRequests.cpp	Async recomputation of all active path requests on close
pathfind.discover	pathfind.compute	Pathfinder.cpp	Graph exploration phase (Pathfinder::find)
pathfind.rank	pathfind.compute	Pathfinder.cpp	Path ranking and selection phase

Note

: pathfind.request nests under the active rpc.command.<name> span. Because OTel context storage is coroutine-aware (backed by LocalValue), the rpc.command.* scope stays correct even though its generic dispatch (callMethod) also wraps handlers such as doRipplePathFind whose span is held across a coroutine yield() — the ambient context travels with the coroutine when it resumes, so there is no wrong-thread scope pop. The pathfind.request → compute → discover sub-tree therefore parents to rpc.command.<name>, giving an exact request-to-pathfind nesting.

Where to find: Tempo → TraceQL: {resource.service.name="xrpld" && name=~"pathfind.*"}

Grafana dashboard: RPC & Pathfinding (StatsD) (xrpld-statsd-rpc) for StatsD timers; span-derived metrics via RPC Performance (rpc-performance)

TxQ Spans

Controlled by trace_transactions=1 in [telemetry] config.

Span Name	Parent	Source File	Description
txq.enqueue	tx.process (submission path; root on open-ledger rebuild)	TxQ.cpp	Queue admission decision (apply/queue/reject). Parents to tx.process via explicit context on submit; correlates via current_ledger_seq on all paths
txq.apply_direct	txq.enqueue	TxQ.cpp	Direct application attempt (bypassing queue)
txq.batch_clear	txq.enqueue	TxQ.cpp	Batch clear of account's queued transactions
txq.accept	—	TxQ.cpp	Ledger-close accept loop (drain queued transactions)
txq.accept.tx	txq.accept	TxQ.cpp	Per-transaction apply within accept loop
txq.cleanup	—	TxQ.cpp	Post-close cleanup (expire old transactions)

Where to find: Tempo → TraceQL: {resource.service.name="xrpld" && name=~"txq.*"}

Grafana dashboard: Transaction Overview (transaction-overview)

gRPC Spans

Controlled by trace_rpc=1 in [telemetry] config.

Span Name	Parent	Source File	Description
grpc.request	—	GRPCServer.cpp	Single gRPC request (GetLedger, GetLedgerData, GetLedgerDiff, GetLedgerEntry)

Where to find: Tempo → TraceQL: {resource.service.name="xrpld" && name="grpc.request"}

Consensus Spans

Controlled by trace_consensus=1 in [telemetry] config.

Span Name	Parent	Source File	Description
consensus.round	—	RCLConsensus.cpp	Top-level round span (deterministic trace ID)
consensus.proposal.send	consensus.round	RCLConsensus.cpp	Node broadcasts its transaction set proposal
consensus.ledger_close	consensus.round	RCLConsensus.cpp	Ledger close event triggered by consensus
consensus.establish	consensus.round	Consensus.h	Establish phase — convergence loop
consensus.update_positions	consensus.round	Consensus.h	Update positions during establish phase
consensus.check	consensus.round	Consensus.h	Check for consensus agreement
consensus.accept	consensus.round	RCLConsensus.cpp	Consensus accepts a ledger (round complete)
consensus.accept.apply	consensus.round	RCLConsensus.cpp	Ledger application with close time details
consensus.validation.send	consensus.round	RCLConsensus.cpp	Validation message sent after ledger accepted
consensus.mode_change	consensus.round	RCLConsensus.cpp	Consensus mode transition (e.g., tracking->proposing)

Note

: toDisplayString(ConsensusMode) (in ConsensusTypes.h) provides Title Case display names for mode attribute values: "Proposing", "Observing", "Wrong Ledger", "Switched Ledger". This is separate from to_string() which returns stable log-format strings.

Where to find: Tempo → TraceQL: {resource.service.name="xrpld" && name=~"consensus.*"}

Grafana dashboard: Consensus Health (consensus-health)

Ledger Spans

Controlled by trace_ledger=1 in [telemetry] config.

Span Name	Parent	Source File	Description
ledger.build	—	BuildLedger.cpp	Build new ledger from accepted transaction set
ledger.validate	—	LedgerMaster.cpp	Ledger promoted to validated status
ledger.store	—	LedgerMaster.cpp	Ledger stored to database/history

Where to find: Tempo → TraceQL: {resource.service.name="xrpld" && name=~"ledger.*"}

Grafana dashboard: Ledger Operations (ledger-operations)

Peer Spans

Controlled by trace_peer in [telemetry] config. Enabled by default (high volume).

Span Name	Parent	Source File	Description
peer.proposal.receive	—	PeerImp.cpp	Consensus proposal received from peer
peer.validation.receive	—	PeerImp.cpp	Validation message received from peer

A — parent means the span is a fresh trace root (kConsumer): it is started via ScopedSpanGuard::freshRoot() at the inbound-message entry point and never inherits an ambient span left active on the peer thread, so it does not nest under an unrelated transaction's trace.

Where to find: Tempo → TraceQL: {resource.service.name="xrpld" && name=~"peer.*"}

Grafana dashboard: Peer Network (peer-network)

---

1.2 Complete Attribute Inventory (83 attributes)

See also: 02-design-decisions.md §2.4.2 for attribute design rationale and privacy considerations.

Every span can carry key-value attributes that provide context for filtering and aggregation.

RPC Attributes

Attribute	Type	Set On	Description
command	string	rpc.command.*	RPC command name (e.g., server_info, ledger)
version	int64	rpc.command.*	API version number
rpc_role	string	rpc.command.*	Caller role: "admin" or "user"
rpc_status	string	rpc.command.*	Result: "success" or "error"
request_payload_size	int64	rpc.command.*	Request payload size in bytes

Tempo query: {span.command="server_info"} to find all server_info calls.

Prometheus label: xrpl_rpc_command (dots converted to underscores by SpanMetrics).

Transaction Attributes

Attribute	Type	Set On	Description
tx_hash	string	tx.process, tx.receive	Transaction hash (hex-encoded)
local	boolean	tx.process	true if locally submitted, false if peer-relayed
path	string	tx.process	Submission path: "sync" or "async"
suppressed	boolean	tx.receive	true if transaction was suppressed (duplicate)
tx_status	string	tx.receive	Transaction status (e.g., "known_bad")
peer_id	int64	tx.receive	Peer identifier (also set on peer spans)
peer_version	string	tx.receive	Peer protocol version string
stage	string	tx.preflight, tx.preclaim, tx.transactor	Apply-pipeline stage: preflight, preclaim, or apply
tx_type	string	tx.preflight, tx.preclaim, tx.transactor	Transaction type name (e.g., Payment)
ter_result	string	tx.preflight, tx.preclaim, tx.transactor	Engine result token for that stage (e.g., tesSUCCESS, terPRE_SEQ)
applied	boolean	tx.transactor	true if the transaction was applied to the ledger
current_ledger_seq	int64	tx.process, tx.receive, tx.preclaim, tx.transactor	Seq of the ledger being worked on (open/in-flight, not established) — joins the txID-keyed spans to the ledger trace
current_ledger_hash	string	tx.preclaim, tx.transactor	Parent hash of that ledger (= consensus.round trace-id seed on the build path). View-bearing stages only; tx.preflight omits both

Tempo query: {span.tx_hash="<hash>"} to trace a specific transaction across nodes. Join a transaction's work to its ledger with {span.current_ledger_seq=<N>}.

Prometheus label: xrpl_tx_local (used as SpanMetrics dimension).

PathFind Attributes

Attribute	Type	Set On	Description
source_account	string	pathfind.request	Source account address
dest_account	string	pathfind.request	Destination account address
fast	boolean	pathfind.compute	Whether this is a fast (non-full) pathfind
search_level	int64	pathfind.compute	Search depth level
num_complete_paths	int64	pathfind.compute	Number of complete paths found
num_paths	int64	pathfind.compute	Total number of paths explored
num_requests	int64	pathfind.update_all	Number of active path requests being recomputed
pathfind_ledger_index	int64	pathfind.update_all	Ledger index used for recomputation

Tempo query: {span.source_account="rHb9..."} to find pathfind requests from a specific account.

TxQ Attributes

Attribute	Type	Set On	Description
tx_hash	string	txq.enqueue, txq.accept.tx	Transaction hash in the queue
current_ledger_seq	int64	txq.enqueue	Seq of the ledger being worked on — correlates the enqueue to the ledger trace
current_ledger_hash	string	txq.enqueue	Parent hash of that ledger (= consensus.round trace-id seed on the build path)
txq_status	string	txq.enqueue	Queue result: "queued", "applied_direct", "rejected"
fee_level_paid	int64	txq.enqueue	Fee level paid by the transaction
required_fee_level	int64	txq.enqueue	Minimum fee level required for queue admission
queue_size	int64	txq.accept	Queue depth at start of accept
ledger_changed	boolean	txq.accept	Whether the open ledger changed since last accept
ledger_seq	int64	txq.cleanup	Ledger sequence for cleanup
expired_count	int64	txq.cleanup	Number of expired transactions removed
ter_code	string	txq.accept.tx	Transaction engine result code
retries_remaining	int64	txq.accept.tx	Remaining retry attempts for this transaction
num_cleared	int64	txq.batch_clear	Number of transactions cleared in batch

Tempo query: {span.txq_status="rejected"} to find rejected queue attempts.

gRPC Attributes

Attribute	Type	Set On	Description
method	string	grpc.request	gRPC method name (e.g., GetLedger, GetLedgerData)
rpc_role	string	grpc.request	Caller role: "admin" or "user"
rpc_status	string	grpc.request	Result: "success", "error", "resource_exhausted", etc.

Tempo query: {span.method="GetLedger"} to find gRPC ledger requests.

Consensus Attributes

Attribute	Type	Set On	Description
consensus_ledger_id	string	consensus.round	Previous ledger hash (used for deterministic trace ID)
ledger_seq	int64	consensus.round, consensus.ledger_close, consensus.accept, consensus.validation.send, consensus.accept.apply	Ledger sequence number
consensus_mode	string	consensus.round, consensus.proposal.send, consensus.ledger_close	Node mode via toDisplayString(): "Proposing", "Observing", etc.
consensus_round	int64	consensus.proposal.send	Consensus round number
proposers	int64	consensus.proposal.send, consensus.accept	Number of proposers in the round
round_time_ms	int64	consensus.accept, consensus.accept.apply	Total consensus round duration in milliseconds
proposing	boolean	consensus.validation.send	Whether this node was a proposer
consensus_state	string	consensus.accept.apply	Consensus outcome: "finished" or "moved_on"
close_time	int64	consensus.accept.apply	Agreed-upon ledger close time (epoch seconds)
close_time_correct	boolean	consensus.accept.apply	Whether validators reached agreement on close time
close_resolution_ms	int64	consensus.accept.apply	Close time rounding granularity in milliseconds
parent_close_time	int64	consensus.accept.apply	Parent ledger's close time (epoch seconds)
close_time_self	int64	consensus.accept.apply	This node's proposed close time
close_time_vote_bins	string	consensus.accept.apply	Histogram of close time votes from validators
resolution_direction	string	consensus.accept.apply	Resolution change: "increased", "decreased", or "unchanged"
converge_percent	int64	consensus.establish	Convergence percentage threshold
establish_count	int64	consensus.establish	Number of establish iterations completed
proposers_agreed	int64	consensus.establish	Number of proposers that agreed on this round
avalanche_threshold	int64	consensus.update_positions	Avalanche threshold for dispute resolution
close_time_threshold	int64	consensus.update_positions	Close time agreement threshold
have_close_time_consensus	boolean	consensus.update_positions	Whether close time consensus has been reached
agree_count	int64	consensus.check	Number of proposers that agree with our position
disagree_count	int64	consensus.check	Number of proposers that disagree with our position
threshold_percent	int64	consensus.check	Required agreement threshold percentage
consensus_result	string	consensus.check	Check result: "yes", "no", or "expired"
quorum	int64	consensus.check	Required quorum for validation
validation_count	int64	consensus.check	Number of validations received
trace_strategy	string	consensus.round	Trace sampling strategy used for this round
consensus_round_id	string	consensus.round	Deterministic round identifier
mode_old	string	consensus.mode_change	Previous consensus mode
mode_new	string	consensus.mode_change	New consensus mode
tx_id	string	consensus.update_positions	Disputed transaction ID
dispute_our_vote	boolean	consensus.update_positions	Our vote on the disputed transaction
dispute_yays	int64	consensus.update_positions	Number of proposers voting to include
dispute_nays	int64	consensus.update_positions	Number of proposers voting to exclude

Tempo query: {span.consensus_mode="Proposing"} to find rounds where node was proposing.

Prometheus label: xrpl_consensus_mode (used as SpanMetrics dimension).

Ledger Attributes

Attribute	Type	Set On	Description
ledger_seq	int64	ledger.build, ledger.validate, ledger.store, tx.apply	Ledger sequence number
close_time	int64	ledger.build	Ledger close time (epoch seconds)
close_time_correct	boolean	ledger.build	Whether close time was agreed upon by validators
close_resolution_ms	int64	ledger.build	Close time rounding granularity in milliseconds
tx_count	int64	ledger.build, tx.apply	Transactions in the ledger
tx_failed	int64	ledger.build, tx.apply	Failed transactions in the ledger
validations	int64	ledger.validate	Number of validations received for this ledger

Tempo query: {span.ledger_seq=12345} to find all spans for a specific ledger.

Peer Attributes

Attribute	Type	Set On	Description
peer_id	int64	tx.receive, peer.proposal.receive, peer.validation.receive	Peer identifier
proposal_trusted	boolean	peer.proposal.receive	Whether the proposal came from a trusted validator
ledger_hash	string	peer.validation.receive	Ledger hash the validation refers to
validation_full	boolean	peer.validation.receive	Whether this is a full (not partial) validation
validation_trusted	boolean	peer.validation.receive	Whether the validation came from a trusted validator

Prometheus labels: xrpl_peer_proposal_trusted, xrpl_peer_validation_trusted (SpanMetrics dimensions).

---

1.3 SpanMetrics — Derived Prometheus Metrics

See also: 01-architecture-analysis.md §1.8.2 for how span-derived metrics map to operational insights.

The OTel Collector's SpanMetrics connector automatically generates RED (Rate, Errors, Duration) metrics from every span. No custom metrics code in xrpld is needed.

Prometheus Metric	Type	Description
traces_span_metrics_calls_total	Counter	Total span invocations
traces_span_metrics_duration_milliseconds_bucket	Histogram	Latency distribution (buckets: 1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000 ms)
traces_span_metrics_duration_milliseconds_count	Histogram	Observation count
traces_span_metrics_duration_milliseconds_sum	Histogram	Cumulative latency

Standard labels on every metric: span_name, status_code, service_name, span_kind

Additional dimension labels (configured in otel-collector-config.yaml):

Span Attribute	Prometheus Label	Applies To
command	xrpl_rpc_command	rpc.command.*
rpc_status	xrpl_rpc_status	rpc.command.*
consensus_mode	xrpl_consensus_mode	consensus.ledger_close
local	xrpl_tx_local	tx.process
proposal_trusted	xrpl_peer_proposal_trusted	peer.proposal.receive
validation_trusted	xrpl_peer_validation_trusted	peer.validation.receive
stage	stage	tx.preflight, tx.preclaim, tx.transactor

The stage dimension (3 values: preflight, preclaim, apply) turns the apply-pipeline spans into per-stage RED metrics with no native instruments — the Transaction Overview dashboard charts rate, p95 latency, and failure rate by stage.

Sampling caveat: xrpld head sampling is fixed at 1.0 (every trace is recorded), so span-derived metrics are not undercounted at the node. If the collector is configured with tail sampling, span-derived metrics reflect only the retained traces, whereas native StatsD/meter metrics do not sample. Account for any collector-side tail sampling when reading absolute stage rates.

Where to query: Prometheus → traces_span_metrics_calls_total{span_name="rpc.command.server_info"}

---

2. StatsD Metrics (beast::insight)

See also: 02-design-decisions.md for the beast::insight coexistence design. 06-implementation-phases.md for the Phase 6 metric inventory.

These are system-level metrics emitted by xrpld's beast::insight framework via StatsD UDP. They cover operational data that doesn't map to individual trace spans.

Configuration

[insight]
server=statsd
address=127.0.0.1:8125
prefix=xrpld

Note

: The prefix value is user-configurable — all metric names in the tables below assume prefix=xrpld (matching the integration test and Grafana dashboards). If you change the prefix, replace xrpld_ with {your_prefix}_ in all PromQL queries.

2.1 Gauges

Prometheus Metric	Source File	Description	Typical Range
xrpld_LedgerMaster_Validated_Ledger_Age	LedgerMaster.h	Seconds since last validated ledger	0–10 (healthy), >30 (stale)
xrpld_LedgerMaster_Published_Ledger_Age	LedgerMaster.h	Seconds since last published ledger	0–10 (healthy)
xrpld_State_Accounting_Disconnected_duration	NetworkOPs.cpp	Cumulative seconds in Disconnected state	Monotonic
xrpld_State_Accounting_Connected_duration	NetworkOPs.cpp	Cumulative seconds in Connected state	Monotonic
xrpld_State_Accounting_Syncing_duration	NetworkOPs.cpp	Cumulative seconds in Syncing state	Monotonic
xrpld_State_Accounting_Tracking_duration	NetworkOPs.cpp	Cumulative seconds in Tracking state	Monotonic
xrpld_State_Accounting_Full_duration	NetworkOPs.cpp	Cumulative seconds in Full state	Monotonic (should dominate)
xrpld_State_Accounting_Disconnected_transitions	NetworkOPs.cpp	Count of transitions to Disconnected	Low
xrpld_State_Accounting_Connected_transitions	NetworkOPs.cpp	Count of transitions to Connected	Low
xrpld_State_Accounting_Syncing_transitions	NetworkOPs.cpp	Count of transitions to Syncing	Low
xrpld_State_Accounting_Tracking_transitions	NetworkOPs.cpp	Count of transitions to Tracking	Low
xrpld_State_Accounting_Full_transitions	NetworkOPs.cpp	Count of transitions to Full	Low (should be 1 after startup)
xrpld_Peer_Finder_Active_Inbound_Peers	PeerfinderManager.cpp	Active inbound peer connections	0–85
xrpld_Peer_Finder_Active_Outbound_Peers	PeerfinderManager.cpp	Active outbound peer connections	10–21
xrpld_Overlay_Peer_Disconnects	OverlayImpl.cpp	Cumulative peer disconnection count	Low growth
xrpld_jobq_job_count	JobQueue.cpp	Current job queue depth (all types)	0–100 (healthy)
xrpld_Node_family_full_below_cache_size	TaggedCache.h	FullBelowCache entry count	Varies
xrpld_Node_family_full_below_cache_hit_rate	TaggedCache.h	FullBelowCache hit rate percentage	0–100

Grafana dashboard: Node Health (StatsD) (xrpld-statsd-node-health)

2.2 Counters

Prometheus Metric	Source File	Description
xrpld_rpc_requests	ServerHandler.cpp	Total RPC requests received
xrpld_ledger_fetches	InboundLedgers.cpp	Inbound ledger fetch attempts
xrpld_ledger_history_mismatch	LedgerHistory.cpp	Ledger hash mismatches detected
xrpld_warn	Logic.h	Resource manager warnings issued
xrpld_drop	Logic.h	Resource manager drops (connections rejected)

Note: xrpld_warn and xrpld_drop use non-standard StatsD meter type (|m). The OTel StatsD receiver only recognizes |c, |g, |ms, |h, |s — these metrics may be silently dropped. See Known Issues below.

Grafana dashboard: RPC & Pathfinding (StatsD) (xrpld-statsd-rpc)

2.3 Histograms (from StatsD timers)

Prometheus Metric	Source File	Unit	Description
xrpld_rpc_time	ServerHandler.cpp	ms	RPC response time distribution
xrpld_rpc_size	ServerHandler.cpp	bytes	RPC response size distribution
xrpld_ios_latency	Application.cpp	ms	I/O service loop latency
xrpld_pathfind_fast	PathRequests.h	ms	Fast pathfinding duration
xrpld_pathfind_full	PathRequests.h	ms	Full pathfinding duration

Quantiles collected: 0th, 50th, 90th, 95th, 99th, 100th percentile.

Grafana dashboards: Node Health (ios_latency), RPC & Pathfinding (rpc_time, rpc_size, pathfind_*)

2.4 Overlay Traffic Metrics

For each of the 45+ overlay traffic categories (defined in TrafficCount.h), four gauges are emitted:

• xrpld_{category}_Bytes_In
• xrpld_{category}_Bytes_Out
• xrpld_{category}_Messages_In
• xrpld_{category}_Messages_Out

Key categories:

Category	Description
total	All traffic aggregated
overhead / overhead_overlay	Protocol overhead
transactions / transactions_duplicate	Transaction relay
proposals / proposals_untrusted / proposals_duplicate	Consensus proposals
validations / validations_untrusted / validations_duplicate	Consensus validations
ledger_data_get / ledger_data_share	Ledger data exchange
ledger_data_Transaction_Node_get/share	Transaction node data
ledger_data_Account_State_Node_get/share	Account state node data
ledger_data_Transaction_Set_candidate_get/share	Transaction set candidates
getObject / haveTxSet / ledgerData	Object requests
ping / status	Keepalive and status
set_get	Set requests

Grafana dashboards: Network Traffic (xrpld-statsd-network), Overlay Traffic Detail (xrpld-statsd-overlay-detail), Ledger Data & Sync (xrpld-statsd-ledger-sync)

2.5 Per-Job Timer Events

For each of the 35 non-special job types (defined in JobTypes.h), two StatsD timer events are emitted:

• xrpld_jobq_{jobName} — execution duration
• xrpld_jobq_{jobName}_q — dequeue wait time

The jobq segment is added at runtime by the collector group the JobQueue is constructed with — see §2.6 for the derivation, which applies to all jobq_* metrics including the queue-depth gauge in §2.1.

These produce summary metrics with quantiles (0th, 50th, 90th, 95th, 99th, 100th).

Key job types (most operationally relevant):

Job Name	Source Enum	Description
acceptLedger	jtACCEPT	Consensus round acceptance
advanceLedger	jtADVANCE	Ledger advancement
transaction	jtTRANSACTION	Transaction processing
writeObjects	jtWRITE	Database object writes
publishNewLedger	jtPUBLEDGER	New ledger publication
trustedValidation	jtVALIDATION_t	Trusted validation processing
trustedProposal	jtPROPOSAL_t	Trusted proposal processing
clientRPC	jtCLIENT_RPC	Client RPC request handling
heartbeat	jtNETOP_TIMER	Network heartbeat timer
sweep	jtSWEEP	Cache sweep / cleanup
ledgerData	jtLEDGER_DATA	Ledger data processing

Special job types (limit=0: peerCommand, diskAccess, processTransaction, orderBookSetup, pathFind, nodeRead, nodeWrite, generic, SyncReadNode, AsyncReadNode, WriteNode) do not emit timer events.

Grafana dashboard: Node Health (StatsD) (xrpld-statsd-node-health) — Key Jobs and All Jobs panels

2.6 Per-Job-Type Saturation Gauges

JobTypeData has always tracked three per-type counters — waiting, running, and deferred (include/xrpl/core/JobTypeData.h) — but only the process-wide queue depth was exported. These three gauge families export them per job type, so queue pressure can be attributed to a type instead of only being visible as a single total.

For each of the same 35 non-special job types, three gauges are created in the JobTypeData constructor and assigned in JobQueue::collect(). Note that the gauge members do not exist on this branch — they are introduced on phase-9, which is where the creation and assignment live.

Metric	Source File	Description
xrpld_jobq_{jobName}_waiting	JobTypeData.h	Jobs enqueued for this type but not yet started
xrpld_jobq_{jobName}_running	JobTypeData.h	Jobs of this type currently executing
xrpld_jobq_{jobName}_deferred	JobTypeData.h	Jobs held back by this type's concurrency limit

Why deferred matters most. JobQueue::addJob() never rejects — when a type is at its concurrency limit it increments deferred and returns true (JobQueue.cpp, addRefCountedJob). Backpressure on a capped type therefore shows up only as latency, after the delay has already happened. A non-zero deferred reading precedes that, which makes it the leading indicator for a saturating type. It is structurally always zero for special types, which is why they are excluded — the same !info.special() guard that already gates the timer events.

Name derivation. These names are not literals anywhere in the source; the jobq segment is added at runtime. Application.cpp constructs the JobQueue with collectorManager_->group("jobq"), and GroupImp::makeName() (Groups.cpp) joins the group name and the metric name with a .. The StatsD sender then prefixes the configured prefix the same way, so JtLedgerReq (whose type name is ledgerRequest) is sent as xrpld.jobq.ledgerRequest_deferred and arrives in Prometheus as xrpld_jobq_ledgerRequest_deferred. The same jobq. prefix applies to the §2.5 timer events and to the queue-depth gauge in §2.1.

Sampling granularity: these are gauges read by the collector hook once per second, not counters incremented at the event. A deferred spike shorter than the sampling interval can be missed. Saturation long enough to delay ledger sync persists across a sample, which is the case these gauges are for.

Pipeline: JobQueue.cpp is in libxrpl and reaches the exporter through beast::insight::Collector, like every other metric in this section. It cannot use the xrpld-only call-site metric macros used by §2a.

---

2a. Call-Site OTel Metrics (MetricsRegistry)

See also: §2 for the beast::insight metrics that use the StatsD transport instead.

Every metric in §2 goes through beast::insight. The metrics below are created directly at their call site through the XRPL_METRIC_* macros (src/xrpld/telemetry/MetricMacros.h) and exported by the OTel Metrics SDK rather than over StatsD. The macros create each instrument once and expand to an empty statement when telemetry is compiled out, so a call site needs no conditional compilation.

These names carry no xrpld_ prefix: the prefix belongs to the StatsD sender, and this path identifies the node through OTel resource attributes instead.

Note

: this family is emitted by MetricsRegistry, which arrives with the native-metrics work in a later phase than this document's own StatsD pipeline. It is catalogued here because this document is the single source of truth for the whole telemetry surface.

2a.1 Job Lifecycle Metrics

Recorded from the three PerfLog job hooks that JobQueue already calls.

Metric	Type	Unit	Labels	Description
job_queued_total	Counter	—	job_type, handler	Jobs enqueued
job_started_total	Counter	—	job_type, handler	Jobs started
job_finished_total	Counter	—	job_type, handler	Jobs completed
job_queued_us	Histogram	µs	job_type, handler	Queue wait time distribution
job_running_us	Histogram	µs	job_type, handler	Execution time distribution

job_queued_us and job_running_us register explicit microsecond buckets spanning 100 µs to 60 s. Without that view the SDK default buckets stop at 10 ms and every quantile reads as 10 ms.

The handler Label

job_type names the queue a job ran on, not who produced it. Several producers share one type — RcvGetLedger and RcvGetObjByHash both run as JtLedgerReq, and JtLedgerData has five (ProcessLData and GotStaleData in InboundLedgers.cpp, AcqDone and the InboundLedger TimeoutCounter job name in InboundLedger.cpp, and GotFetchPack in LedgerMaster.cpp) — so a queue-wait spike cannot be pinned to one of them by job_type alone. The handler label carries the name passed to addJob, which does distinguish them.

The value is not the raw name. It passes through MetricsRegistry::sanitiseHandler, which keeps the name only when it is non-empty and every character is an ASCII letter, and otherwise returns the literal other. The letter test is an explicit ASCII range check rather than a locale-dependent one, so it cannot admit non-ASCII bytes.

Why reduce the name. Two addJob names embed a ledger sequence number: LedgerPersistence.cpp builds "Pub" + std::to_string(ledger->seq()), and OrderBookDBImpl.cpp builds "OB" + std::to_string(ledger->seq() % 1000000000). Used raw, either would mint a new time series for every ledger. Both always contain digits, so both fold to other.

Every other production job name is a compile-time literal, which makes the label domain a function of the source rather than of traffic — it cannot grow at runtime. Names that are literals but still fail the rule also fold to other: GetConsL1 and GetConsL2 contain digits, and the coroutine names RPC-Client, WS-Client, and gRPC-Client contain a hyphen. A job name added later that fails the rule degrades to other rather than becoming unbounded, which is a stronger guarantee than a hand-maintained allowlist.

2a.2 GetObject Request Metrics

Emitted from the TMGetObjectByHash handler in src/xrpld/overlay/detail/PeerImp.cpp. Together with job_queued_us and job_running_us filtered to handler="RcvGetObjByHash", they split the handler's end-to-end time into queue wait, NodeStore lookup, and everything else — so slowness is attributable to a stage rather than merely observed.

Metric	Type	Unit	Labels	Description
getobject_lookup_us	Histogram	µs	—	Time in the NodeStore fetch loop
getobject_request_objects	Histogram	—	—	Objects requested per message
getobject_lookups_total	Counter	—	result	NodeStore lookup outcomes
getobject_rejected_total	Counter	—	reason	Requests refused before any NodeStore access
getobject_charge	Histogram	—	—	Dynamic resource charge applied to the request

Label	Values	Meaning
result	hit, miss	Whether the requested object was found in the NodeStore
reason	oversize, malformed_ledgerhash	Which admission check refused the request

Aggregation. getobject_lookup_us times the fetch loop once per request, and getobject_lookups_total is added once per request carrying the batch hit and miss totals. The loop is bounded by Tuning::kHardMaxReplyNodes (12288), so timing or counting each iteration would cost more than the lookups it measures. getobject_request_objects records the requested count, which is what the charge bands price on. getobject_charge records only the dynamic component returned by computeGetObjectByHashFee(); the admission-time base charge is a constant.

Bucket view required: getobject_lookup_us needs an explicit microsecond-bucket view like the two job histograms above. Without one the SDK default buckets stop at 10 ms, and a lookup loop running toward its 12288 cap exceeds that routinely — so the metric would saturate exactly when it is needed. The view is registered alongside the other three in MetricsRegistry.cpp. Note that MetricsRegistry does not exist on this branch — it is introduced on phase-9, which is where the registration lives.

Expected to read zero: getobject_rejected_total counts non-conforming requests, which a healthy peer does not send. A zero series does not prove the metric works — verify it with a deliberately oversized request.

---

3. Grafana Dashboard Reference

See also: 05-configuration-reference.md §5.8 for Grafana data source provisioning (Tempo, Prometheus) and TraceQL query examples.

3.1 Span-Derived Dashboards (5)

Dashboard	UID	Data Source	Key Panels
RPC Performance	rpc-performance	Prometheus (SpanMetrics)	Request rate by command, p95 latency by command, error rate, heatmap, top commands
Transaction Overview	transaction-overview	Prometheus (SpanMetrics)	Processing rate, latency p95/p50, local vs relay split, apply duration, heatmap
Consensus Health	consensus-health	Prometheus (SpanMetrics)	Round duration p95/p50, proposals rate, close duration, mode timeline, heatmap, close time correctness, resolution direction, close time drift, resolution change timeline, close time vote distribution
Ledger Operations	ledger-operations	Prometheus (SpanMetrics)	Build rate, build duration, validation rate, store rate, build vs close comparison
Peer Network	peer-network	Prometheus (SpanMetrics)	Proposal receive rate, validation receive rate, trusted vs untrusted breakdown

3.2 StatsD Dashboards (5)

Dashboard	UID	Data Source	Key Panels
Node Health	xrpld-statsd-node-health	Prometheus (StatsD)	Ledger age, operating mode, I/O latency, job queue, fetch rate, key/all jobs execution time, cache size/hit rate, publish gap, state duration rate
Network Traffic	xrpld-statsd-network	Prometheus (StatsD)	Active peers, disconnects, bytes in/out, messages in/out, traffic by category, duplicate traffic, all traffic categories detail
RPC & Pathfinding	xrpld-statsd-rpc	Prometheus (StatsD)	RPC rate, response time/size, pathfinding duration, resource warnings/drops
Overlay Traffic Detail	xrpld-statsd-overlay-detail	Prometheus (StatsD)	Squelch, overhead, validator lists, set get/share, have/requested tx, proof paths
Ledger Data & Sync	xrpld-statsd-ledger-sync	Prometheus (StatsD)	Ledger data exchange, legacy ledger share/get, getobject by type, traffic heatmap

3.3 Consensus Close-Time Panels

The Consensus Health dashboard includes 5 close-time panels added in Phase 4:

Panel	Metric / Attribute	Description
Close Time Correctness	close_time_correct	Percentage of rounds with agreed-upon close time
Resolution Direction	resolution_direction	Rate of resolution increases, decreases, and unchanged per time interval
Close Time Drift	close_time vs close_time_self	Difference between agreed close time and node's own proposed close time
Resolution Change Timeline	close_resolution_ms	Close time resolution granularity over time
Close Time Vote Distribution	close_time_vote_bins	Histogram of validator close time votes per round

Template variables (Consensus Health dashboard):

Variable	Source Attribute	Description
$node	exported_instance	Filter by xrpld node instance
$service_name	service_name	Filter by service (service.name, e.g. xrpld)
$deployment_environment	deployment_environment	Filter by deployment tier (local / test / ci / prod)
$xrpl_network_type	xrpl_network_type	Filter by network (mainnet / testnet / devnet)
$close_time_correct	xrpl_consensus_close_time_correct	Filter by close time correctness (true / false)
$resolution_direction	xrpl_consensus_resolution_direction	Filter by resolution direction (increased / decreased / unchanged)

The $service_name, $deployment_environment, and $xrpl_network_type variables are present on all dashboards; each variable name matches its Prometheus label. See telemetry-runbook.md "Deployment Tiers" for how the tier attributes are set and reach metrics.

3.4 Accessing the Dashboards

1. Open Grafana at http://localhost:3000
2. Navigate to Dashboards → xrpld folder
3. All 10 dashboards are auto-provisioned from docker/telemetry/grafana/dashboards/

---

4. Tempo Trace Search Guide

See also: 08-appendix.md §8.2 for span hierarchy visualizations. 05-configuration-reference.md §5.8.5 for TraceQL query examples.

Finding Traces by Type

What to Find	Tempo TraceQL Query
All RPC calls	{resource.service.name="xrpld" && name="rpc.http_request"}
Specific RPC command	{resource.service.name="xrpld" && name="rpc.command.server_info"}
Slow RPC calls	{resource.service.name="xrpld" && name=~"rpc.command.*"} | duration > 100ms
Failed RPC calls	{span.rpc_status="error"}
Specific transaction	{span.tx_hash="<hex_hash>"}
Local transactions only	{span.local=true}
Consensus rounds	{resource.service.name="xrpld" && name="consensus.accept"}
Rounds by mode	{span.consensus_mode="proposing"}
Specific ledger	{span.ledger_seq=12345}
Peer proposals (trusted)	{span.proposal_trusted=true}

Trace Structure

A typical RPC trace shows the span hierarchy:

rpc.http_request (ServerHandler)
  └── rpc.process (ServerHandler)
       └── rpc.command.server_info (RPCHandler)

A consensus round groups child spans under a deterministic trace ID:

consensus.round               (top-level, deterministic trace ID from ledger hash)
  ├── consensus.ledger_close        (close event)
  ├── consensus.proposal.send       (broadcast proposal)
  ├── consensus.establish           (convergence loop)
  │     ├── consensus.update_positions  (update disputes)
  │     └── consensus.check             (check agreement)
  ├── consensus.accept              (accept result)
  ├── consensus.accept.apply        (apply with close time details)
  ├── consensus.validation.send     (send validation)
  └── consensus.mode_change         (mode transition, if any)
ledger.build                  (build new ledger)
  └── tx.apply                (apply transaction set)
ledger.validate               (promote to validated)
ledger.store                  (persist to DB)

---

5. Prometheus Query Examples

See also: 05-configuration-reference.md §5.8.7 for correlating Prometheus StatsD metrics with trace-derived metrics.

Span-Derived Metrics

# RPC request rate by command (last 5 minutes)
sum by (xrpl_rpc_command) (rate(traces_span_metrics_calls_total{span_name=~"rpc.command.*"}[5m]))

# RPC p95 latency by command
histogram_quantile(0.95, sum by (le, xrpl_rpc_command) (rate(traces_span_metrics_duration_milliseconds_bucket{span_name=~"rpc.command.*"}[5m])))

# Consensus round duration p95
histogram_quantile(0.95, sum by (le) (rate(traces_span_metrics_duration_milliseconds_bucket{span_name="consensus.accept"}[5m])))

# Transaction processing rate (local vs relay)
sum by (xrpl_tx_local) (rate(traces_span_metrics_calls_total{span_name="tx.process"}[5m]))

# Trusted vs untrusted proposal rate
sum by (xrpl_peer_proposal_trusted) (rate(traces_span_metrics_calls_total{span_name="peer.proposal.receive"}[5m]))

StatsD Metrics

# Validated ledger age (should be < 10s)
xrpld_LedgerMaster_Validated_Ledger_Age

# Active peer count
xrpld_Peer_Finder_Active_Inbound_Peers + xrpld_Peer_Finder_Active_Outbound_Peers

# RPC response time p95
histogram_quantile(0.95, xrpld_rpc_time_bucket)

# Total network bytes in (rate)
rate(xrpld_total_Bytes_In[5m])

# Operating mode (should be "Full" after startup)
xrpld_State_Accounting_Full_duration

---

6. SpanNames Header File Inventory

All span names and attributes are defined as compile-time constants in colocated SpanNames.h headers. Each header lives next to its subsystem's implementation.

Header File	Subsystem	Span Count	Attribute Count	Notes
src/xrpld/rpc/detail/RpcSpanNames.h	RPC (HTTP/WS)	5	5	Includes rpc.ws_upgrade error path
src/xrpld/rpc/detail/PathFindSpanNames.h	PathFind	5	8	Covers one-shot and subscription paths
src/xrpld/app/main/GrpcSpanNames.h	gRPC	1	3	Flat single-span structure per request
src/xrpld/app/misc/TxSpanNames.h	Transaction	2	7	Includes peer context attributes
src/xrpld/app/misc/detail/TxQSpanNames.h	TxQ	6	11	Queue lifecycle: enqueue through cleanup
src/xrpld/app/consensus/ConsensusSpanNames.h	Consensus	10	35	Deterministic trace IDs, close-time details
src/xrpld/app/ledger/detail/LedgerSpanNames.h	Ledger	4	7	Build, store, validate, tx.apply
src/xrpld/overlay/detail/PeerSpanNames.h	Peer Overlay	2	5	Proposal and validation receive

Design convention: SpanNames headers are colocated with their subsystem classes rather than centralized in telemetry/. See memory/feedback_span-names-colocation.md for rationale.

---

7. Known Issues

Issue	Impact	Status
warn and drop metrics use non-standard StatsD |m meter type	Metrics silently dropped by OTel StatsD receiver	Phase 6 Task 6.1 — needs |m → |c change in StatsDCollector.cpp
xrpld_jobq_job_count may not emit in standalone mode	Missing from Prometheus in some test configs	Requires active job queue activity
xrpld_rpc_requests depends on [insight] config	Zero series if StatsD not configured	Requires [insight] server=statsd in xrpld.cfg
Peer tracing enabled by default	peer.* spans emit unless trace_peer=0	High volume — set trace_peer=0 to opt out on busy mainnet nodes

---

8. Privacy and Data Collection

The telemetry system is designed with privacy in mind:

• No private keys are ever included in spans or metrics
• No account balances or financial data is traced
• Transaction hashes are included (public on-ledger data) but not transaction contents
• Peer IDs are internal identifiers, not IP addresses
• All telemetry is opt-in — disabled by default at build time (-Dtelemetry=OFF)
• Sampling — head sampling is fixed at 1.0 (sample everything); reduce data volume with collector-side tail sampling
• Data stays local — the default stack sends data to localhost only

---

9. Configuration Quick Reference

Full reference: 05-configuration-reference.md §5.1 for all [telemetry] options with defaults, the config parser implementation, and collector YAML configurations (dev and production).

Minimal Setup (development)

[telemetry]
enabled=1

[insight]
server=statsd
address=127.0.0.1:8125
prefix=xrpld

Production Setup

[telemetry]
enabled=1
endpoint=http://otel-collector:4318/v1/traces
trace_peer=0
batch_size=1024
max_queue_size=4096

[insight]
server=statsd
address=otel-collector:8125
prefix=xrpld

Trace Category Toggle

Config Key	Default	Controls
trace_rpc	1	rpc.* spans
trace_transactions	1	tx.* spans
trace_consensus	1	consensus.* spans
trace_ledger	1	ledger.* spans
trace_peer	1	peer.* spans (high volume)