rippled/OpenTelemetryPlan/05-configuration-reference.md

Configuration Reference

Parent Document: OpenTelemetryPlan.md Related: Implementation Phases

---

5.1 xrpld Configuration

OTLP = OpenTelemetry Protocol | TxQ = Transaction Queue

5.1.1 Configuration File Section

The authoritative [telemetry] example lives in cfg/xrpld-example.cfg. Telemetry is disabled by default (enabled=0); enabling it turns on distributed tracing for transaction flow, consensus, and RPC calls, with traces exported to an OpenTelemetry Collector over OTLP. Head sampling is intentionally fixed at 1.0 (sample everything) and is not configurable — per-node head-sampling would produce broken/partial distributed traces, so volume reduction is delegated to the collector's tail sampling (see Section 7.4.2). The full option reference follows.

5.1.2 Configuration Options Summary

Option	Type	Default	Description
enabled	bool	false	Enable/disable telemetry
endpoint	string	http://localhost:4318/v1/traces	OTLP/HTTP collector endpoint
use_tls	bool	false	Enable TLS for exporter connection
tls_ca_cert	string	""	Path to CA certificate file
tls_client_cert	string	""	Path to node's client certificate (PEM) for mutual TLS; empty = one-way TLS
tls_client_key	string	""	Path to private key (PEM) for tls_client_cert; required when it is set
batch_size	uint	512	Spans per export batch
batch_delay_ms	uint	5000	Max delay before sending batch (ms)
max_queue_size	uint	2048	Maximum queued spans
trace_transactions	bool	true	Enable transaction tracing
trace_consensus	bool	true	Enable consensus tracing
trace_rpc	bool	true	Enable RPC tracing
trace_peer	bool	true	Enable peer message tracing (high volume)
trace_ledger	bool	true	Enable ledger tracing
tx_trace_strategy	string	"deterministic"	TX trace ID strategy: "deterministic" (trace_id = txHash[0:16]) or "attribute" (random)
consensus_trace_strategy	string	"deterministic"	Consensus trace ID strategy: "deterministic" (trace_id = prevLedgerHash[0:16]) or "attribute" (random)
service_name	string	"xrpld"	Service name for traces
service_instance_id	string	<node_pubkey>	Instance identifier

Planned (not yet implemented): the following options appear in the design documents but are not parsed by TelemetryConfig.cpp in Phase 1b and later phases. They will be added as the corresponding subsystems are instrumented:

Option	Planned Phase	Purpose
exporter	Future	Select between OTLP/HTTP and OTLP/gRPC
trace_pathfind	Phase 2	Path computation tracing toggle
trace_txq	Phase 3	Transaction queue tracing toggle
trace_validator	Future	Validator list / manifest update tracing
trace_amendment	Future	Amendment voting tracing
consensus_trace_strategy	Phase 4	Trace ID strategy for consensus rounds (deterministic | attribute)

---

5.2 Configuration Parser

TxQ = Transaction Queue

The parser setupTelemetry() in src/libxrpl/telemetry/TelemetryConfig.cpp reads the [telemetry] Section and populates a Telemetry::Setup struct, applying the defaults listed in Section 5.1.2 via section.value_or(...). It derives serviceInstanceId from the node public key when not overridden, selects the exporter endpoint default by exporter type, and leaves the sampling ratio at its fixed 1.0 default (not read from config — see Section 7.4.2).

---

5.3 Application Integration

5.3.1 ApplicationImp Changes

Deferred identity: The node public key (nodeIdentity_) is not available during ApplicationImp's member initializer list — it is resolved later in setup(). The Telemetry object is therefore constructed with an empty serviceInstanceId and patched via setServiceInstanceId() once setup() has called getNodeIdentity().

ApplicationImp (in src/xrpld/app/main/Application.cpp) owns a std::unique_ptr<telemetry::Telemetry> telemetry_. It is built in the member initializer list via makeTelemetry(setupTelemetry(...)) with an empty serviceInstanceId, then patched in setup() by calling setServiceInstanceId() with the Base58 node public key (unless the user supplied a custom service_instance_id). start() and run() forward to telemetry_->start() / telemetry_->stop(), and getTelemetry() returns the owned instance.

5.3.2 ServiceRegistry Interface Addition

include/xrpl/core/ServiceRegistry.h gains a pure-virtual telemetry::Telemetry& getTelemetry() (with a forward declaration of telemetry::Telemetry), giving every component a uniform accessor for the tracing subsystem.

Note: Application extends ServiceRegistry, so getTelemetry() is available on both. Components that hold a ServiceRegistry& (e.g. NetworkOPsImp) call registry_.get().getTelemetry(). Components that still hold an Application& (e.g. ServerHandler, PeerImp, RCLConsensusAdaptor) call app_.getTelemetry() directly.

---

5.4 CMake Integration

OTLP = OpenTelemetry Protocol

5.4.1 Find OpenTelemetry Module

A cmake/FindOpenTelemetry.cmake module locates the OpenTelemetry C++ SDK. It first tries find_package(opentelemetry-cpp CONFIG), aliasing the imported targets OpenTelemetry::api, OpenTelemetry::sdk, and OpenTelemetry::otlp_grpc_exporter, and falls back to pkg-config when no CMake config package is present.

5.4.2 CMakeLists.txt Changes

The top-level CMakeLists.txt adds an XRPL_ENABLE_TELEMETRY option (default OFF). When enabled, it runs find_package(OpenTelemetry REQUIRED), defines the XRPL_ENABLE_TELEMETRY compile flag, and builds the xrpl_telemetry library from the real telemetry sources linked against the OpenTelemetry targets; when disabled, it builds the same target from a no-op NullTelemetry.cpp so call sites compile unchanged.

---

5.5 OpenTelemetry Collector Configuration

OTLP = OpenTelemetry Protocol | APM = Application Performance Monitoring

Production hardening: The configurations in this section are starting points. For production deployments where xrpld ships telemetry across a network to a centrally-hosted collector, see Securing the OTel Pipeline for the required mTLS receiver config, NetworkPolicy, and peer trace-context validation.

The authoritative collector config lives in the repo at docker/telemetry/otel-collector-config.yaml (with Tempo backend config in docker/telemetry/tempo.yaml). The sections below summarize the development and production shapes of that pipeline.

5.5.1 Development Configuration

The development collector enables an OTLP receiver on both gRPC (0.0.0.0:4317) and HTTP (0.0.0.0:4318), a single batch processor (1s timeout, batch size 100), and two exporters: a logging exporter for console debugging and otlp/tempo (insecure) for trace visualization. The single traces pipeline wires receiver → batch → both exporters.

5.5.2 Production Configuration

The production collector adds TLS on the OTLP gRPC receiver and a richer processor chain: a memory_limiter (OOM guard), batch (5s timeout, size 512), tail_sampling, and an attributes processor that hashes sensitive fields (e.g. tx_account) and stamps deployment.environment. Tail sampling keeps all ERROR traces, slow consensus rounds (>5s) and slow RPC requests (>1s), and probabilistically samples the remainder at 10%. Exporters target Grafana Tempo (TLS) and Elastic APM; health_check and zpages extensions are enabled for operability.

---

5.6 Docker Compose Development Environment

OTLP = OpenTelemetry Protocol

The authoritative development stack lives in the repo at docker/telemetry/docker-compose.yml. It brings up four services on a shared xrpld-telemetry network: an otel-collector (otel/opentelemetry-collector-contrib) exposing OTLP gRPC 4317, OTLP HTTP 4318, and health check 13133; tempo for trace storage/visualization; grafana with provisioned datasources and dashboards (anonymous admin enabled); and an optional prometheus for metric correlation.

---

5.7 Configuration Architecture

OTLP = OpenTelemetry Protocol

flowchart TB
    subgraph config["Configuration Sources"]
        cfgFile["xrpld.cfg<br/>[telemetry] section"]
        cmake["CMake<br/>XRPL_ENABLE_TELEMETRY"]
    end

    subgraph init["Initialization"]
        parse["setupTelemetry()"]
        factory["makeTelemetry()"]
    end

    subgraph runtime["Runtime Components"]
        tracer["TracerProvider"]
        exporter["OTLP Exporter"]
        processor["BatchProcessor"]
    end

    subgraph collector["Collector Pipeline"]
        recv["Receivers"]
        proc["Processors"]
        exp["Exporters"]
    end

    cfgFile --> parse
    cmake -->|"compile flag"| parse
    parse --> factory
    factory --> tracer
    tracer --> processor
    processor --> exporter
    exporter -->|"OTLP"| recv
    recv --> proc
    proc --> exp

    style config fill:#e3f2fd,stroke:#1976d2
    style runtime fill:#e8f5e9,stroke:#388e3c
    style collector fill:#fff3e0,stroke:#ff9800

Reading the diagram:

• Configuration Sources: xrpld.cfg provides runtime settings (endpoint, sampling) while the CMake flag controls whether telemetry is compiled in at all.
• Initialization: setupTelemetry() parses config values, then makeTelemetry() constructs the provider, processor, and exporter objects.
• Runtime Components: The TracerProvider creates spans, the BatchProcessor buffers them, and the OTLP Exporter serializes and sends them over the wire.
• OTLP arrow to Collector: Trace data leaves the xrpld process via OTLP (gRPC or HTTP) and enters the external Collector pipeline.
• Collector Pipeline: Receivers ingest OTLP data, Processors apply sampling/filtering/enrichment, and Exporters forward traces to storage backends (Tempo, etc.).

---

5.8 Grafana Integration

APM = Application Performance Monitoring

Step-by-step instructions for integrating xrpld traces with Grafana.

5.8.1 Data Source Configuration

Tempo (Recommended)

A Tempo datasource (grafana/provisioning/datasources/tempo.yaml, provisioned from docker/telemetry/grafana/) points at http://tempo:3200 and enables tracesToLogs (linking to Loki on service.name/tx_hash and mapping trace_id → traceID), serviceMap against Prometheus, the node graph, and Loki search.

Elastic APM

Alternatively, an Elasticsearch datasource (grafana/provisioning/datasources/elastic-apm.yaml) of type elasticsearch points at http://elasticsearch:9200 against the apm-* index, using @timestamp as the time field and mapping the log message/level fields.

5.8.2 Dashboard Provisioning

A dashboard provider (grafana/provisioning/dashboards/dashboards.yaml) loads the xrpld dashboard folder from disk (/var/lib/grafana/dashboards/rippled), polling for changes every 30s with deletion disabled.

5.8.3 Example Dashboard: RPC Performance

An example xrpld RPC Performance dashboard (uid xrpld-rpc-performance) sourced from Tempo via TraceQL provides four panels: RPC latency by command (heatmap), RPC error rate by command (timeseries), the top 10 slowest RPC commands by average duration (table), and a recent-traces table.

5.8.4 Example Dashboard: Transaction Tracing

An example xrpld Transaction Tracing dashboard (uid xrpld-tx-tracing) over Tempo provides three panels: transaction throughput (tx.receive rate, stat), cross-node relay count (average span.relay_count on tx.relay, timeseries), and a table of transaction validation errors (tx.validate with status.code=error).

5.8.5 TraceQL Query Examples

Common queries for xrpld traces:

# Find all traces for a specific transaction hash
{resource.service.name="xrpld" && span.tx_hash="ABC123..."}

# Find slow RPC commands (>100ms)
{resource.service.name="xrpld" && name=~"rpc.command.*"} | duration > 100ms

# Find consensus rounds taking >5 seconds
{resource.service.name="xrpld" && name="consensus.round"} | duration > 5s

# Find failed transactions with error details
{resource.service.name="xrpld" && name="tx.validate" && status.code=error}

# Find transactions relayed to many peers
{resource.service.name="xrpld" && name="tx.relay"} | span.relay_count > 10

# Compare latency across nodes
{resource.service.name="xrpld" && name="rpc.command.account_info"} | avg(duration) by (resource.service.instance.id)

5.8.6 Correlation with PerfLog

To correlate OpenTelemetry traces with existing PerfLog data:

Step 1: Configure Loki to ingest PerfLog

Configure a Promtail scrape job (promtail-config.yaml) that tails /var/log/rippled/perf*.log, parses each JSON line, and promotes trace_id, ledger_seq, and tx_hash to Loki labels.

Step 2: Add trace_id to PerfLog entries

Modify PerfLog so its JSON output includes a trace_id field whenever a valid span is active: fetch the current span from the OpenTelemetry runtime context, and if its context is valid, render the trace ID as a 32-character lowercase hex string into the log entry.

Step 3: Configure Grafana trace-to-logs link

In the Tempo datasource, set the tracesToLogs derived field to link to Loki on the trace_id and tx_hash tags, with filterByTraceID: true.

5.8.7 Correlation with Insight/OTel System Metrics

To correlate traces with Beast Insight system metrics:

Step 1: Export Insight metrics to Prometheus

Beast Insight metrics are exported natively via OTLP to the OTel Collector, which exposes them on the Prometheus endpoint alongside spanmetrics. Configure the [insight] section of xrpld.cfg with server=otel, endpoint=http://localhost:4318/v1/metrics, and prefix=xrpld; no separate StatsD exporter or Prometheus scrape job is needed when using server=otel.

Step 2: Add exemplars to metrics

The OpenTelemetry SDK automatically adds exemplars (trace IDs) to metrics when using the Prometheus exporter, linking metric spikes to specific traces.

Step 3: Configure Grafana metric-to-trace link

In the Prometheus datasource, set exemplarTraceIdDestinations to map the trace_id exemplar to the Tempo datasource.

Step 4: Dashboard panel with exemplars

Add a timeseries panel over Prometheus (e.g. histogram_quantile(0.99, rate(xrpld_rpc_duration_seconds_bucket[5m]))) with exemplar: true enabled.

This allows clicking on metric data points to jump directly to the related trace.

---

Previous: Implementation Strategy | Next: Implementation Phases | Back to: Overview