rippled/OpenTelemetryPlan/05-configuration-reference.md

Configuration Reference

Parent Document: OpenTelemetryPlan.md Related: Implementation Phases

---

5.1 xrpld Configuration

OTLP = OpenTelemetry Protocol | TxQ = Transaction Queue

5.1.1 Configuration File Section

The authoritative [telemetry] example lives in cfg/xrpld-example.cfg. Telemetry is disabled by default (enabled=0); enabling it turns on distributed tracing for transaction flow, consensus, and RPC calls, with traces exported to an OpenTelemetry Collector over OTLP. Head sampling is intentionally fixed at 1.0 (sample everything) and is not configurable — per-node head-sampling would produce broken/partial distributed traces, so volume reduction is delegated to the collector's tail sampling (see Section 7.4.2). Transaction trace IDs are always deterministic (trace_id = txHash[0:16]); there is no strategy switch for the transaction path. The full option reference follows.

service_instance_id defaults to the node public key for every producer. Main.cpp resolves the identity before ApplicationImp is constructed and it is never empty, so all three producers below stamp the node key when the config key is unset. They still build their resources at three different times, which is what decides how each behaves if the identity ever changes mid-run:

Producer	Resource built by	Unset service_instance_id yields
Traces (and therefore all span_* metrics)	Telemetry::start(), during setup()	Base58 node public key
Native XRPL_METRIC_* (MetricsRegistry)	MetricsRegistry::initExporterAndProvider()	Base58 node public key
beast::insight ([insight] server=otel)	TelemetryImpl constructor	Base58 node public key

• Traces: the tracer resource is built in Telemetry::start(), which runs inside setup() after getNodeIdentity() has persisted the identity, so it is the one resource that can still be corrected by setServiceInstanceId(). The spanmetrics connector derives span_calls_total / span_duration_milliseconds_* from those spans, so span metrics inherit the same id.
• Native XRPL_METRIC_* metrics build their own MeterProvider resource in MetricsRegistry::initExporterAndProvider(), called from the registry's constructor. makeMetricsRegistryOptions() in Application.cpp supplies the id: the config key when set, else the resolved node key.
• beast::insight metrics use the global MeterProvider, whose resource is built in the TelemetryImpl constructor, because insight instruments are created eagerly in subsystem constructors and would otherwise bind to the noop provider forever. It receives the same resolved key. Note it writes the attribute unconditionally, where MetricsRegistry guards the write with if (!options.serviceInstanceId.empty()), so a deliberately blank service_instance_id yields service.instance.id="" there and an absent attribute here.

The one case where the three can disagree: a node whose wallet already holds an identity different from the resolved one (another process wrote it between construction and setup()). setServiceInstanceId() then corrects the tracer, and both metric resources, already frozen, need a restart to follow. Every shipped Grafana dashboard filters on service_instance_id=~"$node", so that run shows its metrics under the resolved key and its traces under the stored one.

5.1.2 Configuration Options Summary

Option	Type	Default	Description
enabled	0 or 1	0	Enable/disable telemetry
traces_endpoint	string	http://localhost:4318/v1/traces	OTLP/HTTP collector endpoint for traces
metrics_endpoint	string	http://localhost:4318/v1/metrics	OTLP/HTTP collector endpoint for the native metrics pipeline (MetricsRegistry). Read by makeMetricsRegistryOptions() in Application.cpp
use_tls	0 or 1	0	Enable TLS for exporter connection
tls_ca_cert	string	""	Path to CA certificate file
tls_client_cert	string	""	Client cert (PEM) for mTLS; empty = one-way; if enabled=1, needs key + use_tls=1 or startup fails
tls_client_key	string	""	Private key (PEM) for tls_client_cert; if set with enabled=1, needs the cert + use_tls=1 or fails
batch_size	uint	512	Spans per export batch
batch_delay_ms	uint	5000	Max delay before sending batch (ms)
max_queue_size	uint	2048	Maximum queued spans
trace_transactions	0 or 1	1	Enable transaction tracing
trace_consensus	0 or 1	1	Enable consensus tracing
trace_rpc	0 or 1	1	Enable RPC tracing
trace_peer	0 or 1	1	Enable peer message tracing (high volume)
trace_ledger	0 or 1	1	Enable ledger tracing
consensus_trace_strategy	string	"deterministic"	Consensus trace ID strategy: "deterministic" (trace_id = prevLedgerHash[0:16]) or "random" (experimental, not used). Rejected at startup if it is neither spelling
service_name	string	"xrpld"	Service name (service.name) for traces and metrics
service_instance_id	string	node public key (base58)	Instance identifier (service.instance.id). Traces, span metrics and native XRPL_METRIC_* metrics all fall back to the node key; beast::insight metrics do not — see the note in §5.1.1

Not a config key — deterministic transaction trace IDs are unconditional. Earlier drafts of this document listed a tx_trace_strategy option ("deterministic" | "attribute"). No such key exists: TelemetryConfig.cpp parses no transaction-strategy key, and the transaction trace ID is always derived from the transaction hash. Only the consensus path has a switchable strategy.

Planned (not yet implemented): the following options appear in the design documents but are not parsed by TelemetryConfig.cpp. They will be added as the corresponding subsystems are instrumented:

Option	Planned Phase	Purpose
exporter	Future	Select between OTLP/HTTP and OTLP/gRPC
trace_pathfind	Phase 2	Path computation tracing toggle
trace_txq	Phase 3	Transaction queue tracing toggle
trace_validator	Future	Validator list / manifest update tracing
trace_amendment	Future	Amendment voting tracing

exporter is not read, so do not set it. No shipped sample config carries it. OTLP/HTTP is the only transport that exists (§2.2.1), so traces_endpoint and metrics_endpoint are the only transport knobs until the §2.2.2 gRPC work lands.

---

5.2 Configuration Parser

TxQ = Transaction Queue

The parser makeTelemetrySetup() in src/libxrpl/telemetry/TelemetryConfig.cpp reads the [telemetry] Section and populates a Telemetry::Setup struct, applying the defaults listed in Section 5.1.2 via section.valueOr(...). It takes serviceInstanceId from the nodePublicKey argument when the key is absent, applies one unconditional traces_endpoint default (dflt::tracesEndpoint) — the parser has no notion of exporter type — and leaves the sampling ratio at its fixed 1.0 default (a static constexpr member, so there is nothing to parse). It also rejects two contradictory mTLS configurations outright (tls_client_cert without tls_client_key, and either without use_tls=1) rather than failing open at handshake time.

metrics_endpoint reaches MetricsRegistry by a second route: makeMetricsRegistryOptions() in Application.cpp reads it from the same Section and passes it to the registry's constructor, because Telemetry does not expose the Setup it parsed. Both metric exporters resolve to that one key:

Metric source	Exporter built by	URL comes from
beast::insight ([insight] server=otel)	Telemetry::initMetrics() (global provider)	metrics_endpoint, used verbatim
Native XRPL_METRIC_* (MetricsRegistry)	MetricsRegistry::initExporterAndProvider()	metrics_endpoint, defaulting to http://localhost:4318/v1/metrics

Setting traces_endpoint therefore moves traces only; both metric pipelines follow metrics_endpoint.

---

5.3 Application Integration

5.3.1 ApplicationImp Changes

Identity before construction: telemetry stamps the node public key into resources that are immutable once built, and it builds them during ApplicationImp's member initializer list. So Main.cpp calls resolveNodeIdentity() first, from the config and command line alone, and passes the keypair to makeApplication(). It never comes back empty: a configured [node_seed] decides it, else the wallet database supplies it if one already exists, else it is minted. Every producer therefore carries the node key from the start — the tracer, the global MeterProvider the beast::insight metrics use, and MetricsRegistry's separate one. ApplicationImp::setup() then calls getNodeIdentity(), which stores that keypair when the wallet holds none and otherwise adopts what the wallet holds. Only in that second case can the two differ, and then setServiceInstanceId() corrects the tracer alone: both metric resources are frozen once their providers are built, so those metrics need a restart to report the stored key.

ApplicationImp (in src/xrpld/app/main/Application.cpp) owns a std::pair<PublicKey, SecretKey> nodeIdentity_, declared before std::unique_ptr<telemetry::Telemetry> telemetry_ and std::unique_ptr<telemetry::MetricsRegistry> metricsRegistry_ so both resources can be built from it. All three are built in the member initializer list, before every subsystem. setup() persists the identity, starts tracing with startTelemetry() before the first consensus round, and arms the registry's observable gauges with startTelemetryGauges() once overlay_ exists. run() stops both observers before any service, then stops telemetry last; ~ApplicationImp repeats those stops for the paths that never reach run(). getTelemetry() and getMetricsRegistry() return the owned instances.

5.3.2 ServiceRegistry Interface Addition

include/xrpl/core/ServiceRegistry.h gains a pure-virtual telemetry::Telemetry& getTelemetry() (with a forward declaration of telemetry::Telemetry), giving every component a uniform accessor for the tracing subsystem.

Note: Application extends ServiceRegistry, so getTelemetry() is available on both. Components that hold a ServiceRegistry& (e.g. NetworkOPsImp) call registry_.get().getTelemetry(). Components that still hold an Application& (e.g. ServerHandler, PeerImp, RCLConsensus::Adaptor) call app_.getTelemetry() directly.

---

5.4 CMake Integration

OTLP = OpenTelemetry Protocol

5.4.1 Locating the OpenTelemetry SDK

Superseded design. Earlier drafts described a hand-written cmake/FindOpenTelemetry.cmake module that aliased OpenTelemetry::api, OpenTelemetry::sdk and OpenTelemetry::otlp_grpc_exporter with a pkg-config fallback. That module was never written — it exists in no commit — and the aliasing approach it described does not work with the package the build actually consumes.

The SDK is located by the Conan-generated CMake config package, nothing else:

• CMakeLists.txt — find_package(opentelemetry-cpp CONFIG REQUIRED), guarded by the telemetry option (§5.4.2). The dependency itself is declared in conanfile.py:153 (opentelemetry-cpp/1.28.0), also guarded — requirements() adds it only if self.options.telemetry (:152), so with the option off the package never enters the dependency graph.
• Linking goes through the umbrella target opentelemetry-cpp::opentelemetry-cpp, never the per-component targets. cmake/XrplCore.cmake:221-225 and :83-91 record why: the Conan package under-declares its inter-component dependencies, so naming ::api / ::sdk individually produces the wrong static-link order and fails at executable link time. The umbrella target supplies both the trace and metrics components with the correct ordering.

5.4.2 CMakeLists.txt Changes

The build flag is telemetry:

option(telemetry "Enable OpenTelemetry tracing" ON)   # top-level CMakeLists.txt

The declared value is ON temporarily, so that CI compiles the telemetry code paths while the feature branches are in review. OFF is the intended default once merged, and the flip is a separate change. Set the value explicitly rather than relying on the default:

To …	Use (CMake)	Use (Conan)
Build telemetry in	-Dtelemetry=ON	-o telemetry=True
Build it out (all no-ops)	-Dtelemetry=OFF	-o telemetry=False

When the option is ON, the guarded block below it runs find_package(opentelemetry-cpp CONFIG REQUIRED) and adds the compile definition XRPL_ENABLE_TELEMETRY.

XRPL_ENABLE_TELEMETRY is not a CMake option. It is only ever added as a compile definition by add_compile_definitions(XRPL_ENABLE_TELEMETRY) in that same block. Passing -DXRPL_ENABLE_TELEMETRY=OFF on the CMake command line disables nothing — it defines an unused cache variable and telemetry stays compiled in. CMake does report it, at the end of configuration under Manually-specified variables were not used by the project, so it is not literally silent — but that line is easy to scroll past. Any procedure that relies on it (including the rollback path in §3.9.6) must use -Dtelemetry=OFF.

The target is xrpl.libxrpl.telemetry, created by add_module(xrpl telemetry) at cmake/XrplCore.cmake:231 from include/xrpl/telemetry/ + src/libxrpl/telemetry/. There is no xrpl_telemetry target.

Selection between the real and the no-op implementation is an in-source #ifdef, not a source swap: NullTelemetry.cpp is compiled into the target unconditionally (see its header comment, NullTelemetry.cpp:1-12). It provides the makeTelemetry() factory when XRPL_ENABLE_TELEMETRY is undefined; when the macro is defined, Telemetry.cpp provides the factory instead and NullTelemetry's virtuals only serve as noop tracer/span fallbacks. Call sites compile unchanged either way.

---

5.5 OpenTelemetry Collector Configuration

OTLP = OpenTelemetry Protocol | APM = Application Performance Monitoring

Production hardening: The configurations in this section are starting points. For production deployments where xrpld ships telemetry across a network to a centrally-hosted collector, see Securing the OTel Pipeline for the required mTLS receiver config, NetworkPolicy, and peer trace-context validation.

The authoritative collector config lives in the repo at docker/telemetry/otel-collector-config.yaml (with Tempo backend config in docker/telemetry/tempo.yaml). The sections below summarize the development and production shapes of that pipeline.

5.5.1 Development / Base Configuration

docker/telemetry/otel-collector-config.yaml is the base config used by the local stack and by CI. It carries three pipelines, not one:

Pipeline	Receivers	Processors	Exporters
traces	otlp	resource/tier, resource/stripsdk, attributes/hash, batch	debug, otlp_grpc/tempo, span_metrics
metrics	otlp, span_metrics	resource/tier, resource/stripsdk, batch	prometheus
logs	file_log	resource/logs, resource/tier, resource/stripsdk, batch	otlp_http/loki

Component detail:

• Receivers. otlp on gRPC 0.0.0.0:4317 and HTTP 0.0.0.0:4318 (both traces and native metrics arrive on 4318). file_log tails /var/log/xrpld/*/debug.log and runs a regex_parser that lifts timestamp, partition, severity and the optional trace_id/span_id emitted by the journal sink (§5.8.5).
• Processors. batch (1s timeout, send_batch_size: 100); resource/tier (action: upsert on deployment.environment, action: insert on xrpl.network.type only when absent); resource/stripsdk (drops the telemetry.sdk.* attributes); resource/logs (action: upsert on service.name and job — only the former becomes a Loki stream label, see the known issue in §5.8.5); attributes/hash (hashes pathfind_source_account and pathfind_dest_account).
• Connector. span_metrics with namespace: "span" (otel-collector-config.yaml:114) — this is why the derived RED metrics are span_calls_total / span_duration_milliseconds_*. The connector's own default namespace is empty, so without this setting the names would be the bare calls_total / duration_milliseconds_*. The traces_span_metrics_* family is not the connector's default and is not produced here at all — it comes from a different producer, Tempo's metrics_generator span-metrics processor (tempo.yaml:75), whose remote_write is commented out in this repo (see §5.8.6). Histogram unit: ms with sub-millisecond buckets from 0.01ms, plus explicit 2s–30s boundaries for consensus and ledger.acquire. ~25 low-cardinality dimensions are promoted to labels (command, rpc_status, tx_type, ter_result, stage, consensus_mode, outcome, …).
• Exporters. debug (console, verbosity: detailed), otlp_grpc/tempo (tempo:4317, tls.insecure: true), otlp_http/loki (http://loki:3100/otlp — Loki 3.x native OTLP; the old loki exporter was removed in collector-contrib v0.147.0), and prometheus on 0.0.0.0:8889 with resource_to_telemetry_conversion.enabled: true so the tier and instance resource attributes become Prometheus labels.
• Extensions. health_check on 0.0.0.0:13133 only. There is no zpages extension.

Deliberately absent from the base config — do not document them as present: no memory_limiter, no tail_sampling, no Elastic APM exporter, and no tx_account attribute rule (the hashed keys are the two pathfind_*_account ones).

5.5.2 Production Configuration

There is no separate "production" collector config in this repo. The one overlay that exists is docker/telemetry/otel-collector-config.grafanacloud.yaml. It is not the base config plus one processor — it restructures the service graph. The full delta:

Added by the overlay	Where	Purpose
basicauth/grafanacloud	:29	Extension; instance id / API token from the container environment
tail_sampling	:60	One probabilistic policy at 0.5%, decision_wait: 10s
transform/cloudlabels	:119	Copies three resource attrs onto datapoint labels for Cloud (OTLP) ingest
otlp_http/grafanacloud	:236	Single OTLP/HTTP exporter fanning all three signals to Grafana Cloud
metrics_flush_interval	:136	spanmetrics flushes every 15s instead of the 60s default

Removed by the overlay	Consequence
attributes/hash	Pathfinding account attributes are not hashed on this config — see below
debug	No console span dump; collector logs alone when diagnosing ingest

Pipelines go from three (traces, metrics, logs) to five (:253-280): traces/metrics, traces/store, metrics/local, metrics/cloud, logs. tail_sampling is applied in traces/store (:259-261) — the branch feeding Tempo and Grafana Cloud — not in a pipeline named traces, which does not exist in the overlay. The traces/metrics branch feeds spanmetrics unsampled, so the derived RED metrics stay exact while stored traces are ~1/200 of ingested ones.

Known issue — the cloud path does not hash pathfinding accounts. The base config runs attributes/hash on its traces pipeline (otel-collector-config.yaml:105-110), hashing pathfind_source_account and pathfind_dest_account as defense in depth behind the node-side hashing. The overlay declares no such processor and lists none on any of its five pipelines, so on the Grafana Cloud config those two attributes reach both Grafana Cloud and the local Tempo with whatever value the node sent. Any node that emits raw addresses loses its second line of defense. Adding attributes/hash to traces/store and traces/metrics would close the gap.

Hardening a collector for a real deployment (TLS/mTLS on the receiver, NetworkPolicy, peer trace-context validation) is covered in Securing the OTel Pipeline — not by any config file in docker/telemetry/.

---

5.6 Docker Compose Development Environment

OTLP = OpenTelemetry Protocol

The authoritative development stack lives in the repo at docker/telemetry/docker-compose.yml. It brings up six services on a shared xrpld-telemetry bridge network. All images are pinned to exact tags.

Service	Image	Published ports	Role
otel-collector	otel/opentelemetry-collector-contrib:0.158.0	4317, 4318, 8889	OTLP ingest, spanmetrics, file_log tail, Prometheus scrape target
tempo	grafana/tempo:2.9.4	3200	Trace storage and TraceQL
loki	grafana/loki:3.7.6	3100	Log storage for log↔trace correlation
prometheus	prom/prometheus:v3.13.2	9090	Scrapes the collector's :8889
grafana	grafana/grafana:13.1.2	3000	Dashboards + provisioned datasources/alerts, anonymous admin
renderer	grafana/grafana-image-renderer:v5.12.0	none	Panel→PNG rendering for image export and alert screenshots

Two corrections to earlier drafts:

• prometheus is not optional. grafana lists it in depends_on (along with tempo, loki and renderer), and 7 of the 15 dashboards query span_calls_total from it. Removing it blanks most panels.
• Port 13133 is not published. The collector's health_check extension listens on 13133 inside the container, but the base compose file publishes only 4317, 4318 and 8889. Health checks from the host must either add a port mapping or run docker compose exec.

The collector also bind-mounts the xrpld log root read-only (${XRPLD_LOG_DIR:-./data/logs} → /var/log/xrpld) for the file_log receiver, and the grafana service reads Slack/email alert secrets from an optional gitignored .env.alerting.

---

5.7 Configuration Architecture

OTLP = OpenTelemetry Protocol

flowchart TB
    subgraph config["Configuration Sources"]
        cfgFile["xrpld.cfg<br/>[telemetry] section"]
        cmake["CMake option: telemetry<br/>ON today for CI, OFF once merged<br/>when ON, defines XRPL_ENABLE_TELEMETRY"]
    end

    subgraph init["Initialization"]
        parse["makeTelemetrySetup()"]
        factory["makeTelemetry()"]
    end

    subgraph runtime["Runtime Components"]
        tracer["TracerProvider"]
        exporter["OTLP Exporter"]
        processor["BatchProcessor"]
    end

    subgraph collector["Collector Pipeline"]
        recv["Receivers"]
        proc["Processors"]
        exp["Exporters"]
    end

    cfgFile --> parse
    cmake -->|"compile flag"| parse
    parse --> factory
    factory --> tracer
    tracer --> processor
    processor --> exporter
    exporter -->|"OTLP"| recv
    recv --> proc
    proc --> exp

    style config fill:#e3f2fd,stroke:#1976d2
    style runtime fill:#e8f5e9,stroke:#388e3c
    style collector fill:#fff3e0,stroke:#ff9800

Reading the diagram:

• Configuration Sources: xrpld.cfg provides runtime settings (endpoint, per-component trace toggles) while the CMake telemetry option controls whether telemetry is compiled in at all. That option is declared ON today only so CI compiles the instrumented paths; OFF is the intended default once merged, so treat the build gate as something to pass explicitly, and the runtime gate is opt-in either way (enabled=0 by default). Head sampling is fixed at 1.0 and is not a config option; volume reduction happens via tail sampling in the collector.
• Initialization: makeTelemetrySetup() parses config values, then makeTelemetry() constructs the provider, processor, and exporter objects.
• Runtime Components: The TracerProvider creates spans, the BatchProcessor buffers them, and the OTLP Exporter serializes and sends them over the wire.
• OTLP arrow to Collector: Trace data leaves the xrpld process via OTLP/HTTP and enters the external Collector pipeline. (OTLP/gRPC is future work — see design decisions §2.2.2.)
• Collector Pipeline: Receivers ingest OTLP data, Processors apply sampling/filtering/enrichment, and Exporters forward traces to storage backends (Tempo, etc.).

---

5.8 Grafana Integration

APM = Application Performance Monitoring

Step-by-step instructions for integrating xrpld traces with Grafana.

5.8.1 Data Source Configuration

Three datasources are provisioned from docker/telemetry/grafana/provisioning/datasources/. There is no Elastic APM datasource — elastic-apm.yaml was described in an earlier draft but never existed. Elastic remains a possible backend (§7.2); nothing in this repo provisions it.

File	Type	URL	uid	Notes
tempo.yaml	tempo	http://tempo:3200	tempo	nodeGraph, serviceMap/tracesToMetrics → prometheus, tracesToLogs → loki, plus ~30 Explore search filters
prometheus.yaml	prometheus	http://prometheus:9090	prometheus	Backs every span-metric and native-metric panel
loki.yaml	loki	http://loki:3100	loki	Backs log-derived-insights; derived fields jump back to Tempo

The Tempo tracesToLogs block is configured as filterByTraceID: true, filterBySpanID: false, tags: []. The empty tag list is deliberate: the correlation is by trace ID alone, so no span attribute needs to exist on both sides. Earlier drafts claimed trace_id + tx_hash tags — that is not what ships, and adding a tag Tempo cannot resolve blanks the link.

The search-filter list is the practical index of queryable span attributes: resource scope (service.name, service.instance.id, service.version, xrpl.network.id, xrpl.network.type), intrinsics (name, status, duration), and span scope (command, rpc_status, rpc_role, tx_hash, tx_type, tx_status, local, path, suppressed, peer_version, consensus_*, ledger_seq, ledger_hash, close_time_correct, close_resolution_ms, proposers, mode_old, mode_new, txq_status, ter_code).

5.8.2 Dashboard Provisioning

grafana/provisioning/dashboards/dashboards.yaml declares a single file provider named xrpld-telemetry, orgId: 1, targeting Grafana folder xrpld from path /var/lib/grafana/dashboards (no /rippled suffix), with disableDeletion: false, editable: true, foldersFromFilesStructure: false. It sets no poll interval — Grafana's updateIntervalSeconds default applies; the "every 30s" figure in earlier drafts was invented.

docker-compose.yml mounts ./grafana/dashboards read-only at that path, so the 15 JSON files in docker/telemetry/grafana/dashboards/ are what gets provisioned.

5.8.3 Shipped Dashboards

The dashboards are Prometheus-first, not TraceQL-first, and their uids are bare (no xrpld- prefix). The full inventory and per-panel query reference is 09-data-collection-reference.md; the uids are:

consensus-health, fee-market, job-queue, ledger-data-sync, ledger-operations, log-derived-insights, network-traffic, node-health, overlay-traffic-detail, peer-network, peer-quality, rpc-pathfinding, rpc-performance, transaction-overview, validator-health.

Panel-count convention used in these docs: counts are of data panels only — type: "row" collapsible headers are excluded, because a row is a layout element with no query. A board's raw panels array is therefore longer than its stated count (e.g. rpc-performance has 19 array entries: 2 rows + 17 data panels).

Two examples described in earlier drafts do not exist and should not be looked for: xrpld-rpc-performance (the real board is rpc-performance, 17 data panels in 2 rows, mostly Prometheus span metrics) and xrpld-tx-tracing (the transaction board is transaction-overview, 18 data panels in 3 rows; its error panel filters span_calls_total{span_name="tx.process", ter_result!~"tesSUCCESS|"}, since no tx.validate span was ever built — see 02 §2.3.2).

Why !~"tesSUCCESS|" and not !="tesSUCCESS". An absent Prometheus label compares equal to the empty string, and tx.process can end without a ter_result attribute: processTransaction() returns early when preProcessTransaction() rejects the transaction (NetworkOPs.cpp:1437-1438) and doTransactionAsync() returns early when the transaction is already applying (:1461-1462); the only setter runs later, at :1674. Those series carry ter_result="", which !="tesSUCCESS" counts as an error. The regex form excludes the empty value explicitly (the trailing | alternative), which is the form docs/telemetry-runbook.md:1198 and two of the three transaction-overview.json failure panels already use.

Every dashboard exposes a $node template variable bound to service_instance_id; see the §5.1.1 note on why service_instance_id must be set for metric panels to split per node.

5.8.4 TraceQL Query Examples

Common queries for xrpld traces. Every span name and attribute below is one that the code actually emits — check against the *SpanNames.h constants before adding more.

# Find all traces for a specific transaction hash
{resource.service.name="xrpld" && span.tx_hash="ABC123..."}

# Find slow RPC commands (>100ms)
{resource.service.name="xrpld" && name=~"rpc.command.*"} | { duration > 100ms }

# Find consensus rounds taking >5 seconds
{resource.service.name="xrpld" && name="consensus.round"} | { duration > 5s }

# Find failed transaction processing
{resource.service.name="xrpld" && name="tx.process" && span.ter_result!="tesSUCCESS"}

# Find failed apply-pipeline stages (preflight / preclaim / transactor)
{resource.service.name="xrpld" && name=~"tx\\.(preflight|preclaim|transactor)" && status=error}

# Find transactions that arrived from a peer rather than a local client.
# The `local` attribute lives on tx.process, NOT on tx.receive (see the note
# below).
{resource.service.name="xrpld" && name="tx.process" && span.local=false}

# Compare latency across nodes
{resource.service.name="xrpld" && name="rpc.command.account_info"} | avg_over_time(duration) by (resource.service.instance.id)

Queries in earlier drafts used tx.validate, tx.relay and span.relay_count. None of the three exists: signature/format validation ships as tx.preflight/tx.preclaim, and no relay span or relay-count attribute was ever built. See 02 §2.3.2.

TraceQL silently returns nothing for an absent attribute. Unlike PromQL, where a missing label compares equal to "", a TraceQL attribute predicate matches only spans that actually carry the attribute — including negated forms such as != and =~".*". So filtering on the wrong span name yields zero rows with no error. local has exactly one set-site, NetworkOPs.cpp:1417, and it is on tx.process: an earlier draft paired it with name="tx.receive", which can never match. Check the attribute's owning span in 09 §1.2 before combining a name= and a span. predicate.

5.8.5 Correlation with Logs

Log↔trace correlation is implemented (Phase 8) and needs no Promtail, Fluentd or PerfLog change. Two pieces:

1. The node stamps the IDs. The journal sink Logs::format() (src/libxrpl/basics/Log.cpp:304-338, guarded by XRPL_ENABLE_TELEMETRY) reads the thread-local OTel context and, when a valid span is active, prefixes the message with trace_id=<32 hex> span_id=<16 hex>. It reads the context value directly rather than calling GetSpan() to avoid a heap allocation on the (common) no-span path. This is the ordinary debug.log stream — PerfLog is not involved, and the setTraceId hook described in earlier drafts was never built.
2. The collector ingests them. The file_log receiver tails /var/log/xrpld/*/debug.log and its regex_parser lifts trace_id and span_id as optional capture groups (§5.5.1). resource/logs applies an upsert of service.name=xrpld, which Loki promotes to the stream label service_name, so the canonical selector is {service_name="xrpld"}. Logs land in Loki via otlp_http/loki.

Known issue — the collector's job upsert is ineffective for stream selection. resource/logs also applies an upsert of a job=xrpld attribute (otel-collector-config.yaml:62-70) with the stated intent that operators could paste {job="xrpld"}. That does not work. On OTLP ingest Loki promotes only an allow-listed set of resource attributes to indexed stream labels (service.name, service.namespace, service.instance.id, deployment.environment, the k8s.*/cloud.* keys); job is not on that list, and this repo ships no Loki config override — docker-compose.yml:116 starts Loki with the image's built-in /etc/loki/local-config.yaml. job therefore lands in structured metadata, which cannot appear in a stream selector, so {job="xrpld"} returns an empty result rather than an error. Corroboration in-repo: docs/telemetry-runbook.md:2533 states the same ("service_name="xrpld" (not job="xrpld")"), and all 38 Loki queries in the shipped dashboards (35 panel targets + 3 Loki-backed template variables) select on service_name — zero use job. Either drop the job upsert or add job to Loki's distributor.otlp_config.resource_attributes allow-list via a mounted Loki config; until then, use service_name.

Grafana then links the two directions: the Tempo datasource's tracesToLogs (filterByTraceID: true, tags: []) jumps trace → logs, and loki.yaml's derived fields jump log → trace.

5.8.6 Correlation with Insight/OTel System Metrics

To correlate traces with Beast Insight system metrics:

Step 1: Export Insight metrics to Prometheus

Beast Insight metrics are exported natively via OTLP to the OTel Collector, which exposes them on its Prometheus endpoint (:8889) alongside spanmetrics. Set server=otel in the [insight] section of xrpld.cfg; no separate StatsD exporter or Prometheus scrape job is needed.

makeCollectorManager() (src/xrpld/app/main/CollectorManager.cpp) reads these [insight] keys:

Key	Read at	Effect when server=otel
server	:35	Live. statsd | otel | anything else. Selects the collector implementation.
address	:39	StatsD only — the UDP endpoint.
prefix	:41, :53	Inert. Stored on the OTel collector but formatName() prepends nothing (OTelCollector.cpp:855-866); only StatsD applies it.
endpoint	:50	Inert. Logged for diagnostics (OTelCollector.cpp:730), then unused.
service_instance_id	:58	Inert. (void)-discarded (OTelCollector.cpp:722).
service_name	:64	Inert. (void)-discarded (OTelCollector.cpp:723).

Where the identity and endpoint actually come from. OTelCollector deliberately does not own a pipeline: it fetches the Meter from the global MeterProvider that Telemetry::initMetrics() published (OTelCollector.cpp:726-745). So the resource attributes — including service.instance.id, which every dashboard filters on — and the exporter URL both come from the [telemetry] section, not [insight]. The four inert keys above are back-compat leftovers from the StatsD-era signature; setting them has no effect. Set [telemetry] service_instance_id instead (§5.1.1).

server=otel is not the default. CollectorManager.cpp:72-75 falls through to NullCollector for any unrecognised or absent server value, so a node with no [insight] section emits no metrics at all.

Step 2: Correlate metrics to traces

Today this is a time-range correlation, not a click-through one: note the window from the metric panel, then search Tempo over the same window filtered by service.instance.id.

Exemplars are NOT implemented. Earlier drafts of this section instructed operators to rely on automatic exemplars, set exemplarTraceIdDestinations on the Prometheus datasource, and enable exemplar: true on panels. None of that is wired up: the string exemplar appears nowhere in src/libxrpl/telemetry/, src/xrpld/telemetry/, or docker/telemetry/. Concretely, three things are missing —

1. the SDK's exemplar filter is left at its default and no reservoir is configured in Telemetry::initMetrics() or MetricsRegistry;
2. the collector's prometheus exporter has no exemplar settings;
3. grafana/provisioning/datasources/prometheus.yaml has no exemplarTraceIdDestinations block.

Note also that the query used as an example, rpc_duration_seconds_bucket, does not exist — RPC latency histograms are span_duration_milliseconds_bucket (spanmetrics, unit: ms) and rpc_method_us (native). Wiring exemplars end to end is genuine open work; until it lands, do not document a click-through that operators cannot perform.

Step 3: Jump the other way instead

Trace → metrics is available now: the Tempo datasource sets tracesToMetrics.datasourceUid: prometheus with a ±1h time shift, so the span-metric queries it builds resolve against the span_* families the collector's spanmetrics connector produces. Trace → logs and log → trace are both live (§5.8.5).

Known gap — Service Map is configured but inactive. The Tempo datasource declares serviceMap.datasourceUid: prometheus, and tempo.yaml:70-76 enables the service-graphs metrics-generator processor, but the generator has nowhere to write: its remote_write block is commented out (tempo.yaml:53-56), and prometheus.yml:6-9 defines a single scrape job against otel-collector:8889 — it never scrapes or accepts writes from Tempo. traces_service_graph_request_total and its siblings are therefore never stored, so the Service Map / Node Graph tab renders empty. The same gap means Tempo's span-metrics processor never lands traces_spanmetrics_* either (§5.5.1) — every span metric the dashboards use comes from the collector's connector instead. Closing it needs both halves: uncomment remote_write in tempo.yaml and enable --web.enable-remote-write-receiver on the Prometheus service (or add a scrape job for Tempo).

---

Previous: Implementation Strategy | Next: Implementation Phases | Back to: Overview