rippled/OpenTelemetryPlan/02-design-decisions.md

Design Decisions

Parent Document: OpenTelemetryPlan.md Related: Architecture Analysis

---

2.1 OpenTelemetry Components

OTLP = OpenTelemetry Protocol

2.1.1 SDK Selection

Primary Choice: OpenTelemetry C++ SDK (opentelemetry-cpp)

Component	Purpose	Required
opentelemetry-cpp::api	Tracing API headers	Yes
opentelemetry-cpp::sdk	SDK implementation	Yes
opentelemetry-cpp::ext	Extensions (exporters)	Yes
opentelemetry-cpp::otlp_http_exporter	OTLP/HTTP export	Yes (shipped in Phase 1b)
opentelemetry-cpp::otlp_grpc_exporter	OTLP/gRPC export	Future (not yet wired up)

2.1.2 Instrumentation Strategy

Manual Instrumentation (recommended):

Approach	Pros	Cons
Manual	Precise control, optimized placement, xrpld-specific attributes	More development effort
Auto	Less code, automatic coverage	Less control, potential overhead, limited customization

---

2.2 Exporter Configuration

OTLP = OpenTelemetry Protocol

flowchart TB
    subgraph nodes["xrpld Nodes"]
        node1["xrpld<br/>Node 1"]
        node2["xrpld<br/>Node 2"]
        node3["xrpld<br/>Node 3"]
    end

    collector["OpenTelemetry<br/>Collector<br/>(sidecar or standalone)"]

    subgraph backends["Observability Backends"]
        tempo["Tempo"]
        elastic["Elastic<br/>APM"]
    end

    node1 -->|"OTLP/HTTP<br/>:4318"| collector
    node2 -->|"OTLP/HTTP<br/>:4318"| collector
    node3 -->|"OTLP/HTTP<br/>:4318"| collector

    collector --> tempo
    collector --> elastic

    style nodes fill:#0d47a1,stroke:#082f6a,color:#ffffff
    style backends fill:#1b5e20,stroke:#0d3d14,color:#ffffff
    style collector fill:#bf360c,stroke:#8c2809,color:#ffffff

Reading the diagram:

• xrpld Nodes (blue): The source of telemetry data. Each xrpld node exports spans via OTLP/HTTP on port 4318 (the only exporter shipped in Phase 1b).
• OpenTelemetry Collector (red): The central aggregation point that receives spans from all nodes. Can run as a sidecar (per-node) or standalone (shared). Handles batching, filtering, and routing.
• Observability Backends (green): The storage and visualization destinations. Tempo is the recommended backend for both development and production, and Elastic APM is an alternative. The Collector routes to one or more backends.
• Arrows (nodes to collector to backends): The data pipeline -- spans flow from nodes to the Collector over HTTP, then the Collector fans out to the configured backends.

2.2.1 OTLP/HTTP (Shipped in Phase 1b)

OTLP/HTTP is the only exporter wired up in Phase 1b. It is configured via OtlpHttpExporterOptions with the collector traces endpoint (http://localhost:4318/v1/traces by default) and a JSON content type (binary protobuf is also available).

2.2.2 OTLP/gRPC (Future Work — Planned Upgrade)

OTLP/gRPC is planned as a future upgrade from the HTTP exporter. The gRPC transport offers lower per-span overhead and tighter back-pressure semantics than HTTP/JSON, making it attractive for production deployments once the HTTP path is validated in earlier phases.

Required to land this upgrade:

1. Add opentelemetry-cpp::otlp_grpc_exporter to the Conan recipe (the dependency already exists but is not linked in Phase 1b builds).
2. Extend TelemetryConfig.cpp to parse an exporter key (otlp_http default, otlp_grpc opt-in) and a gRPC endpoint override.
3. In Telemetry::start() branch on the parsed exporter type and construct either OtlpHttpExporterFactory::Create(httpOpts) or OtlpGrpcExporterFactory::Create(grpcOpts) accordingly.
4. Update the runbook and dashboards to document the alternate port and TLS settings.

When wired up, the gRPC path will use OtlpGrpcExporterOptions configured with the collector endpoint (host on port 4317), TLS credentials enabled, and a CA certificate path.

Until that work lands, OtlpGrpcExporterOptions is not used by any code path in Phase 1b through Phase 5.

---

2.3 Span Naming Conventions

TxQ = Transaction Queue | UNL = Unique Node List | WS = WebSocket

2.3.1 Naming Schema

<component>.<operation>[.<sub-operation>]

Examples:

• tx.receive - Transaction received from peer
• consensus.phase.establish - Consensus establish phase
• rpc.command.server_info - server_info RPC command

2.3.2 Complete Span Catalog

Span name	Description
tx.receive	Transaction received from network
tx.validate	Transaction signature/format validation
tx.process	Full transaction processing
tx.relay	Transaction relay to peers
tx.apply	Apply transaction to ledger
consensus.round	Complete consensus round
consensus.phase.open	Open phase - collecting transactions
consensus.phase.establish	Establish phase - reaching agreement
consensus.phase.accept	Accept phase - applying consensus
consensus.proposal.receive	Receive peer proposal
consensus.proposal.send	Send our proposal
consensus.validation.receive	Receive peer validation
consensus.validation.send	Send our validation
rpc.request	HTTP/WebSocket request handling
rpc.command.*	Specific RPC command (dynamic)
peer.connect	Peer connection establishment
peer.disconnect	Peer disconnection
peer.message.send	Send protocol message
peer.message.receive	Receive protocol message
ledger.acquire	Ledger acquisition from network
ledger.build	Build new ledger
ledger.validate	Ledger validation
ledger.close	Close ledger
ledger.replay	Ledger replay executed
ledger.delta	Delta-based ledger acquired
pathfind.request	Path request initiated
pathfind.compute	Path computation executed
txq.enqueue	Transaction queued
txq.apply	Queued transaction applied
fee.escalate	Fee escalation triggered
validator.list.fetch	UNL list fetched
validator.manifest	Manifest update processed
amendment.vote	Amendment voting executed
shamap.sync	State tree synchronization
job.enqueue	Job added to queue
job.execute	Job execution

2.3.3 Attribute Naming Conventions

Span names follow §2.3.1 (dotted <component>.<operation>). Span attribute keys follow the rules below. The constants in the *SpanNames.h headers are the single source of truth; the collector, Tempo, the Grafana dashboards, and the runbook all consume these exact keys, so every layer must agree with the code. A CI check enforces this end to end.

1. Per-span unique attribute → bare field name, allowed when the field is recorded by a single span/workflow so the span name already supplies the domain (e.g. command, version, local on rpc.command).
2. Shared attribute (same concept on more than one span) → ONE key, reused verbatim on every span that records it; the span name tells the occurrences apart, so no per-emitter prefix is added. Name it by the field's meaning: a property of a domain object keeps that object's bare field name (ledger_hash, ledger_seq, tx_hash, peer_id, full_validation); a field already qualified by a sub-kind keeps that qualifier on every emitter (proposal_trusted on both consensus.proposal.receive and peer.proposal.receive; validation_trusted likewise). Defined once in the base SpanNames.h namespace attr block and re-exported (using) by each domain header.
3. Collision qualifier → <domain>_<field>, only when a bare name would collide with a DIFFERENT concept in the shared spanmetrics label space or with the OTel-reserved status key (e.g. rpc_status, grpc_status, consensus_state, consensus_round, consensus_mode). This disambiguates distinct concepts that share a word; it is NOT used to tag the same concept with its emitting workflow — that is rule 2 (one shared name).
4. Resource attribute → dotted xrpl.<subsystem>.<field>, reserved ONLY for process/network identity set once at startup (xrpl.network.id, xrpl.network.type). Span attributes are never dotted in the xrpl. form — it blurs the resource/span scope boundary and parses awkwardly in TraceQL.
5. Span names use <subsystem>[.<component>] (dotted, per §2.3.1). Only attribute keys follow rules 1–4.

Standard OpenTelemetry semantic-convention keys keep their canonical dotted form (e.g. service.* resource attributes, http.* span attributes); the "no dotted form" rule applies to xrpl-custom keys only.

The same rules are recorded in CONTRIBUTING.md (the permanent home, since OpenTelemetryPlan/ is removed once the rollout completes). The attribute examples in §2.4 below follow these rules.

---

2.4 Attribute Schema

TxQ = Transaction Queue | UNL = Unique Node List | OTLP = OpenTelemetry Protocol

2.4.1 Resource Attributes (Set Once at Startup)

Resource attributes identify the process and are set once at startup. They use the standard OpenTelemetry semantic conventions plus custom dotted xrpl.* keys (the dotted form is reserved for resource scope per §2.3.3).

Key	Type / value	Description
service.name	"xrpld"	Standard SERVICE_NAME
service.version	BuildInfo::getVersionString()	Standard SERVICE_VERSION
service.instance.id	node public key (base58)	Standard SERVICE_INSTANCE_ID
xrpl.network.id	network id (e.g. 0 for mainnet)	Network identifier
xrpl.network.type	"mainnet" | "testnet" | "devnet" | "unknown"	Network kind
xrpl.node.type	"validator" | "stock" | "reporting"	Node role
xrpl.node.cluster	cluster name	Cluster name, if clustered

2.4.2 Span Attributes by Category

Span attribute keys use the underscore form from §2.3.3 (shared/qualified keys are <domain>_<field>; per-span unique keys are bare). The dotted form is reserved for the resource attributes in §2.4.1 above. This catalog lists the planned attribute set by category; the exact emitted key for each implemented span is defined by the *SpanNames.h constants, which are the single source of truth where the two differ.

Transaction Attributes

Key	Type	Description
tx_hash	string	Transaction hash (hex)
tx_type	string	"Payment", "OfferCreate", etc.
tx_account	string	Source account (redacted in prod)
tx_sequence	int64	Account sequence number
tx_fee	int64	Fee in drops
tx_result	string	"tesSUCCESS", "tecPATH_DRY", etc.
ledger_index	int64	Ledger containing transaction

Consensus Attributes

Key	Type	Description
consensus_round	int64	Round number
consensus_phase	string	"open", "establish", "accept"
consensus_mode	string	"proposing", "observing", etc.
proposers	int64	Number of proposers
prev_ledger_prefix	string	Previous ledger hash prefix
ledger_seq	int64	Ledger sequence
tx_count	int64	Transactions in consensus set
round_time_ms	float64	Round duration

Establish-phase gap fill and cross-node correlation attributes (Phase 4a):

Key	Type	Description
consensus_round_id	int64	Consensus round number
consensus_ledger_id	string	previousLedger.id() — shared across nodes
trace_strategy	string	"deterministic" or "attribute"
converge_percent	int64	Convergence % (0-100+)
establish_count	int64	Number of establish iterations
disputes_count	int64	Active disputed transactions
agree_count	int64	Peers that agree (haveConsensus)
disagree_count	int64	Peers that disagree
threshold_percent	int64	Close-time consensus threshold (avCT_CONSENSUS_PCT=75%)
consensus_result	string	"yes", "no", "moved_on", "expired"
mode_old	string	Previous consensus mode
mode_new	string	New consensus mode

RPC Attributes

Key	Type	Description
command	string	Command name (per-span unique on rpc.command)
version	int64	API version
rpc_role	string	"admin" or "user" (qualified — role is generic)
params	string	Sanitized parameters (optional)

Peer & Message Attributes

Key	Type	Description
peer_id	string	Peer public key (base58)
peer_address	string	IP:port
peer_latency_ms	float64	Measured latency
peer_cluster	string	Cluster name if clustered
message_type	string	Protocol message type name
message_size_bytes	int64	Message size
message_compressed	bool	Whether compressed

Ledger & Job Attributes

Key	Type	Description
ledger_hash	string	Ledger hash
ledger_index	int64	Ledger sequence/index
close_time	int64	Close time (epoch)
ledger_tx_count	int64	Transaction count
job_type	string	Job type name
job_queue_ms	float64	Time spent in queue
job_worker	int64	Worker thread ID

PathFinding Attributes

Key	Type	Description
pathfind_source_currency	string	Source currency code
pathfind_dest_currency	string	Destination currency code
pathfind_path_count	int64	Number of paths found
pathfind_cache_hit	bool	RippleLineCache hit

TxQ Attributes

Key	Type	Description
txq_queue_depth	int64	Current queue depth
txq_fee_level	int64	Fee level of transaction
txq_eviction_reason	string	Why transaction was evicted

Fee Attributes

Key	Type	Description
fee_load_factor	int64	Current load factor
fee_escalation_level	int64	Fee escalation multiplier

Validator Attributes

Key	Type	Description
validator_list_size	int64	UNL size
validator_list_age_sec	int64	Seconds since last update

Amendment Attributes

Key	Type	Description
amendment_name	string	Amendment name
amendment_status	string	"enabled", "vetoed", "supported"

SHAMap Attributes

Key	Type	Description
shamap_type	string	"transaction", "state", "account_state"
shamap_missing_nodes	int64	Number of missing nodes during sync
shamap_duration_ms	float64	Sync duration

2.4.3 Data Collection Summary

The following table summarizes what data is collected by category:

Category	Attributes Collected	Purpose
Transaction	tx_hash, tx_type, tx_result, tx_fee, ledger_index	Trace transaction lifecycle
Consensus	consensus_round, consensus_phase, consensus_mode, proposers, round_time_ms	Analyze consensus timing
RPC	command, version, rpc_status, duration_ms	Monitor RPC performance
Peer	peer_id (public key), peer_latency_ms, message_type, message_size_bytes	Network topology analysis
Ledger	ledger_hash, ledger_index, close_time, ledger_tx_count	Ledger progression tracking
Job	job_type, job_queue_ms, job_worker	JobQueue performance
PathFinding	pathfind_fast, pathfind_search_level, pathfind_num_paths, pathfind_ledger_index, pathfind_num_requests	Payment path analysis
TxQ	txq_queue_depth, txq_fee_level, txq_eviction_reason	Queue depth and fee tracking
Fee	fee_load_factor, fee_escalation_level	Fee escalation monitoring
Validator	validator_list_size, validator_list_age_sec	UNL health monitoring
Amendment	amendment_name, amendment_status	Protocol upgrade tracking
SHAMap	shamap_type, shamap_missing_nodes, shamap_duration_ms	State tree sync performance

2.4.4 Privacy & Sensitive Data Policy

PII = Personally Identifiable Information

OpenTelemetry instrumentation is designed to collect operational metadata only, never sensitive content.

Data NOT Collected

The following data is explicitly excluded from telemetry collection:

Excluded Data	Reason
Private Keys	Never exposed; not relevant to tracing
Account Balances	Financial data; privacy sensitive
Transaction Amounts	Financial data; privacy sensitive
Raw TX Payloads	May contain sensitive memo/data fields
Personal Data	No PII collected
IP Addresses	Configurable; excluded by default in prod

Privacy Protection Mechanisms

Mechanism	Description
Account Hashing	tx_account is hashed at collector level before storage
Configurable Redaction	Sensitive fields can be excluded via [telemetry] config section
Sampling	Only 10% of traces recorded by default, reducing data exposure
Local Control	Node operators have full control over what gets exported
No Raw Payloads	Transaction content is never recorded, only metadata (hash, type, result)
Collector-Level Filtering	Additional redaction/hashing can be configured at OTel Collector

Collector-Level Data Protection

The OpenTelemetry Collector can be configured (via an attributes processor) to hash or redact sensitive attributes before export — for example, hashing tx_account, deleting peer_address to drop IP addresses, and deleting params to redact request parameters.

Configuration Options for Privacy

In xrpld.cfg, operators control data collection granularity through the [telemetry] section. Besides enabled, per-component toggles (trace_transactions, trace_consensus, trace_rpc, trace_peer — the last often disabled due to high volume) select which spans are emitted, and redaction flags (redact_account to hash account addresses, redact_peer_address to remove peer IP addresses) control SDK-level redaction before export.

Note

: The redact_account configuration in xrpld.cfg controls SDK-level redaction before export, while collector-level filtering (see Collector-Level Data Protection above) provides an additional defense-in-depth layer. Both can operate independently.

Key Principle: Telemetry collects operational metadata (timing, counts, hashes) — never sensitive content (keys, balances, amounts, raw payloads).

See also: Securing the OTel Pipeline covers transport-level protection for telemetry leaving the node — mTLS to the collector and validation of incoming peer trace context. Privacy controls in this section keep sensitive data out of spans; the security doc keeps the spans themselves out of untrusted hands.

---

2.5 Context Propagation Design

WS = WebSocket

2.5.0 Deterministic Trace ID Strategy

Both transaction and consensus tracing use deterministic trace IDs derived from a globally known hash, so all nodes handling the same workflow independently produce spans under the same trace_id. This is combined with protobuf span_id propagation for parent-child relay ordering when available.

Transactions — trace_id = txHash[0:16]

Every node that handles a transaction knows its txID (the uint256 transaction hash). The first 16 bytes of this hash are used as the OTel trace_id:

uint256 txHash:  A1B2C3D4 E5F6A7B8 C9D0E1F2 A3B4C5D6  E7F8A9B0 C1D2E3F4 A5B6C7D8 E9F0A1B2
                 |---------- trace_id (16 bytes) ---------|  (remaining 16 bytes unused)

Each node generates a random 8-byte span_id so its span is unique within the shared trace. When protobuf TraceContext is present in the incoming TMTransaction, the sender's span_id is extracted and used as the parent — preserving the relay chain as a parent-child tree. When absent (older peers, first hop from client), the span appears as a root in the same trace — correlation is preserved, only the tree structure degrades.

Node A (submitter)        Node B (relay)          Node C (relay)
trace_id: A1B2...         trace_id: A1B2...       trace_id: A1B2...
span_id:  1234 (random)   span_id:  5678 (random) span_id:  9ABC (random)
parent:   (none)          parent:   1234 (proto)   parent:   5678 (proto)
                               ↑                        ↑
                       protobuf propagation      protobuf propagation

If protobuf propagation fails at Node B (old peer):

Node A                    Node B (old peer)       Node C
trace_id: A1B2...         trace_id: A1B2...       trace_id: A1B2...
span_id:  1234            span_id:  5678          span_id:  9ABC
parent:   (none)          parent:   (none)        parent:   5678 (proto)
                          ↑ no parent, but same trace_id — still grouped

Consensus — trace_id = prevLedgerHash[0:16]

All validators in the same consensus round share the same previousLedger.id(). The first 16 bytes are used as trace_id. See Phase 4a implementation status and createDeterministicContext() in RCLConsensus.cpp for the implementation.

Switchable via consensus_trace_strategy config: "deterministic" (default) or "attribute" (random trace_id, correlation via attribute queries).

Why Not Random IDs with Propagation Only?

Random trace IDs require unbroken context propagation across every hop. In a mixed-version network (common during upgrades), older peers silently drop the trace_context protobuf field. The trace splits and downstream spans become impossible to find. Deterministic IDs make correlation propagation-resilient — the trace backend groups all spans for the same transaction/round regardless of whether propagation succeeded.

Why Keep Protobuf Propagation?

Deterministic trace IDs alone provide correlation (all spans grouped) but not causality (which node relayed to which). Protobuf span_id propagation adds parent-child ordering that shows the exact relay path. The two mechanisms complement each other:

Mechanism	Provides	Fails when
Deterministic trace_id	Cross-node correlation	Never (hash is always known)
Protobuf span_id propagation	Parent-child relay ordering	Older peer drops trace_context field

Implementation Reference

The utility function createDeterministicTxContext(uint256 const& txHash) follows the same pattern as createDeterministicContext(uint256 const& ledgerId) in RCLConsensus.cpp. See Phase 3 Task 3.9 for the full spec.

2.5.1 Propagation Boundaries

flowchart TB
    subgraph http["HTTP/WebSocket (RPC)"]
        w3c["W3C Trace Context Headers:<br/>traceparent:<br/>00-trace_id-span_id-flags<br/>tracestate: xrpld=..."]
    end

    subgraph protobuf["Protocol Buffers (P2P)"]
        proto["message TraceContext {<br/>  bytes trace_id = 1;  // 16 bytes<br/>  bytes span_id = 2;   // 8 bytes<br/>  uint32 trace_flags = 3;<br/>  string trace_state = 4;<br/>}"]
    end

    subgraph jobqueue["JobQueue (Internal Async)"]
        job["Context captured at job creation,<br/>restored at execution<br/><br/>class Job {<br/>  otel::context::Context<br/>    traceContext_;<br/>};"]
    end

    style http fill:#0d47a1,stroke:#082f6a,color:#ffffff
    style protobuf fill:#1b5e20,stroke:#0d3d14,color:#ffffff
    style jobqueue fill:#bf360c,stroke:#8c2809,color:#ffffff

Reading the diagram:

• HTTP/WebSocket - RPC (blue): For client-facing RPC requests, trace context is propagated using the W3C traceparent header. This is the standard approach and works with any OTel-compatible client.
• Protocol Buffers - P2P (green): For peer-to-peer messages between xrpld nodes, trace context is embedded as a protobuf TraceContext message carrying trace_id, span_id, flags, and optional trace_state.
• JobQueue - Internal Async (red): For asynchronous work within a single node, the OTel context is captured when a job is created and restored when the job executes on a worker thread. This bridges the async gap so spans remain linked.

---

2.6 Integration with Existing Observability

OTLP = OpenTelemetry Protocol | WS = WebSocket

2.6.1 Existing Frameworks Comparison

xrpld already has two observability mechanisms. OpenTelemetry complements (not replaces) them:

Aspect	PerfLog	Beast Insight (StatsD)	OpenTelemetry
Type	Logging	Metrics	Distributed Tracing
Data	JSON log entries	Counters, gauges, histograms	Spans with context
Scope	Single node	Single node	Cross-node
Output	perf.log file	StatsD server	OTLP Collector
Question answered	"What happened on this node?"	"How many? How fast?"	"What was the journey?"
Correlation	By timestamp	By metric name	By trace_id
Overhead	Low (file I/O)	Low (UDP packets)	Low-Medium (configurable)

2.6.2 What Each Framework Does Best

PerfLog

• Purpose: Detailed local event logging for RPC and job execution
• Strengths:
  • Rich JSON output with timing data
  • Already integrated in RPC handlers
  • File-based, no external dependencies
• Limitations:
  • Single-node only (no cross-node correlation)
  • No parent-child relationships between events
  • Manual log parsing required

A PerfLog entry is a JSON object with fields such as time, method, duration_us, and result.

Beast Insight (StatsD)

• Purpose: Real-time metrics for monitoring dashboards
• Strengths:
  • Aggregated metrics (counters, gauges, histograms)
  • Low overhead (UDP, fire-and-forget)
  • Good for alerting thresholds
• Limitations:
  • No request-level detail
  • No causal relationships
  • Single-node perspective

In xrpld, Beast Insight is used through increment (counters), gauge (point-in-time values), and timing (durations) calls.

OpenTelemetry (NEW)

• Purpose: Distributed request tracing across nodes
• Strengths:
  • Cross-node correlation via trace_id
  • Parent-child span relationships
  • Rich attributes per span
  • Industry standard (CNCF)
• Limitations:
  • Requires collector infrastructure
  • Higher complexity than logging

A span is created via startSpan (e.g. "tx.relay"), annotated with attributes such as tx_hash and peer_id, and is automatically linked to its parent through the active context.

2.6.3 When to Use Each

Scenario	PerfLog	StatsD	OpenTelemetry
"How many TXs per second?"	❌	✅	✅
"What's the p99 RPC latency?"	❌	✅	✅
"Why was this specific TX slow?"	⚠️ partial	❌	✅
"Which node delayed consensus?"	❌	❌	✅
"What happened on node X at time T?"	✅	❌	✅
"Show me the TX journey across 5 nodes"	❌	❌	✅

2.6.4 Coexistence Strategy

flowchart TB
    subgraph xrpld["xrpld Process"]
        perflog["PerfLog<br/>(JSON to file)"]
        insight["Beast Insight<br/>(StatsD)"]
        otel["OpenTelemetry<br/>(Tracing)"]
    end

    perflog --> perffile["perf.log"]
    insight --> statsd["StatsD Server"]
    otel --> collector["OTLP Collector"]

    perffile --> grafana["Grafana<br/>(Unified UI)"]
    statsd --> grafana
    collector --> grafana

    style xrpld fill:#212121,stroke:#0a0a0a,color:#ffffff
    style grafana fill:#bf360c,stroke:#8c2809,color:#ffffff

Reading the diagram:

• xrpld Process (dark gray): The single xrpld node running all three observability frameworks side by side. Each framework operates independently with no interference.
• PerfLog to perf.log: PerfLog writes JSON-formatted event logs to a local file. Grafana can ingest these via Loki or a file-based datasource.
• Beast Insight to StatsD Server: Insight sends aggregated metrics (counters, gauges) over UDP to a StatsD server. Grafana reads from StatsD-compatible backends like Graphite or Prometheus (via StatsD exporter).
• OpenTelemetry to OTLP Collector: OTel exports spans over OTLP/gRPC to a Collector, which then forwards to a trace backend (Tempo).
• Grafana (red, unified UI): All three data streams converge in Grafana, enabling operators to correlate logs, metrics, and traces in a single dashboard.

2.6.5 Correlation with PerfLog

Trace IDs can be correlated with existing PerfLog entries for comprehensive debugging. The design is for RPCHandler.cpp to start an rpc.command.<method> span alongside the existing PerfLog rpcStart/rpcFinish/rpcError calls, extract the span's trace_id (when valid), and eventually stamp it onto the PerfLog entry (a planned setTraceId hook) so logs and traces share a key. The span status is set to OK on success or to error (recording the exception) on failure.

---

Previous: Architecture Analysis | Next: Implementation Strategy | Back to: Overview