From d6450631bf94cc5ed450a27681fa200b7d6fb7d2 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Thu, 11 Jun 2026 14:29:01 +0100 Subject: [PATCH] removed code blocks from plan docs Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> --- OpenTelemetryPlan/00-tracing-fundamentals.md | 11 +- OpenTelemetryPlan/02-design-decisions.md | 423 +++--- .../03-implementation-strategy.md | 62 +- OpenTelemetryPlan/04-code-samples.md | 1129 ----------------- .../05-configuration-reference.md | 751 +---------- .../07-observability-backends.md | 317 +---- OpenTelemetryPlan/08-appendix.md | 2 - OpenTelemetryPlan/OpenTelemetryPlan.md | 33 +- OpenTelemetryPlan/POC_taskList.md | 636 ---------- 9 files changed, 225 insertions(+), 3139 deletions(-) delete mode 100644 OpenTelemetryPlan/04-code-samples.md delete mode 100644 OpenTelemetryPlan/POC_taskList.md diff --git a/OpenTelemetryPlan/00-tracing-fundamentals.md b/OpenTelemetryPlan/00-tracing-fundamentals.md index 7839273ff2..9c6f96d7af 100644 --- a/OpenTelemetryPlan/00-tracing-fundamentals.md +++ b/OpenTelemetryPlan/00-tracing-fundamentals.md @@ -494,16 +494,7 @@ traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 ### Protocol Buffers (xrpld P2P messages) -```protobuf -message TMTransaction { - bytes rawTransaction = 1; - // ... existing fields ... - - // Trace context extension - bytes trace_parent = 100; // W3C traceparent - bytes trace_state = 101; // W3C tracestate -} -``` +xrpld P2P messages such as `TMTransaction` carry the trace context in two added byte fields alongside the existing payload: `trace_parent` holds the W3C traceparent (`trace_id`, `span_id`, and `trace_flags`), and `trace_state` holds the optional W3C tracestate. Together they propagate the trace across the P2P boundary so a receiving node can attach its spans to the sender's span. --- diff --git a/OpenTelemetryPlan/02-design-decisions.md b/OpenTelemetryPlan/02-design-decisions.md index 58ef3abdc1..732eda3661 100644 --- a/OpenTelemetryPlan/02-design-decisions.md +++ b/OpenTelemetryPlan/02-design-decisions.md @@ -1,7 +1,7 @@ # Design Decisions > **Parent Document**: [OpenTelemetryPlan.md](./OpenTelemetryPlan.md) -> **Related**: [Architecture Analysis](./01-architecture-analysis.md) | [Code Samples](./04-code-samples.md) +> **Related**: [Architecture Analysis](./01-architecture-analysis.md) --- @@ -72,14 +72,10 @@ flowchart TB ### 2.2.1 OTLP/HTTP (Shipped in Phase 1b) -```cpp -// Configuration for OTLP over HTTP (the only exporter currently wired up). -namespace otlp = opentelemetry::exporter::otlp; - -otlp::OtlpHttpExporterOptions opts; -opts.url = "http://localhost:4318/v1/traces"; -opts.content_type = otlp::HttpRequestContentType::kJson; // or kBinary -``` +OTLP/HTTP is the only exporter wired up in Phase 1b. It is configured via +`OtlpHttpExporterOptions` with the collector traces endpoint +(`http://localhost:4318/v1/traces` by default) and a JSON content type +(binary protobuf is also available). ### 2.2.2 OTLP/gRPC (Future Work — Planned Upgrade) @@ -100,17 +96,9 @@ Required to land this upgrade: 4. Update the runbook and dashboards to document the alternate port and TLS settings. -Example Phase 1b+ gRPC configuration (when wired up): - -```cpp -// Configuration for OTLP over gRPC (future work). -namespace otlp = opentelemetry::exporter::otlp; - -otlp::OtlpGrpcExporterOptions opts; -opts.endpoint = ":4317"; -opts.use_ssl_credentials = true; -opts.ssl_credentials_cacert_path = "/path/to/ca.crt"; -``` +When wired up, the gRPC path will use `OtlpGrpcExporterOptions` configured with +the collector endpoint (host on port 4317), TLS credentials enabled, and a CA +certificate path. Until that work lands, `OtlpGrpcExporterOptions` is **not** used by any code path in Phase 1b through Phase 5. @@ -135,85 +123,44 @@ path in Phase 1b through Phase 5. ### 2.3.2 Complete Span Catalog -```yaml -# Transaction Spans -tx: - receive: "Transaction received from network" - validate: "Transaction signature/format validation" - process: "Full transaction processing" - relay: "Transaction relay to peers" - apply: "Apply transaction to ledger" - -# Consensus Spans -consensus: - round: "Complete consensus round" - phase: - open: "Open phase - collecting transactions" - establish: "Establish phase - reaching agreement" - accept: "Accept phase - applying consensus" - proposal: - receive: "Receive peer proposal" - send: "Send our proposal" - validation: - receive: "Receive peer validation" - send: "Send our validation" - -# RPC Spans -rpc: - request: "HTTP/WebSocket request handling" - command: - "*": "Specific RPC command (dynamic)" - -# Peer Spans -peer: - connect: "Peer connection establishment" - disconnect: "Peer disconnection" - message: - send: "Send protocol message" - receive: "Receive protocol message" - -# Ledger Spans -ledger: - acquire: "Ledger acquisition from network" - build: "Build new ledger" - validate: "Ledger validation" - close: "Close ledger" - replay: "Ledger replay executed" - delta: "Delta-based ledger acquired" - -# PathFinding Spans -pathfind: - request: "Path request initiated" - compute: "Path computation executed" - -# TxQ Spans -txq: - enqueue: "Transaction queued" - apply: "Queued transaction applied" - -# Fee/Load Spans -fee: - escalate: "Fee escalation triggered" - -# Validator Spans -validator: - list: - fetch: "UNL list fetched" - manifest: "Manifest update processed" - -# Amendment Spans -amendment: - vote: "Amendment voting executed" - -# SHAMap Spans -shamap: - sync: "State tree synchronization" - -# Job Spans -job: - enqueue: "Job added to queue" - execute: "Job execution" -``` +| Span name | Description | +| ------------------------------ | --------------------------------------- | +| `tx.receive` | Transaction received from network | +| `tx.validate` | Transaction signature/format validation | +| `tx.process` | Full transaction processing | +| `tx.relay` | Transaction relay to peers | +| `tx.apply` | Apply transaction to ledger | +| `consensus.round` | Complete consensus round | +| `consensus.phase.open` | Open phase - collecting transactions | +| `consensus.phase.establish` | Establish phase - reaching agreement | +| `consensus.phase.accept` | Accept phase - applying consensus | +| `consensus.proposal.receive` | Receive peer proposal | +| `consensus.proposal.send` | Send our proposal | +| `consensus.validation.receive` | Receive peer validation | +| `consensus.validation.send` | Send our validation | +| `rpc.request` | HTTP/WebSocket request handling | +| `rpc.command.*` | Specific RPC command (dynamic) | +| `peer.connect` | Peer connection establishment | +| `peer.disconnect` | Peer disconnection | +| `peer.message.send` | Send protocol message | +| `peer.message.receive` | Receive protocol message | +| `ledger.acquire` | Ledger acquisition from network | +| `ledger.build` | Build new ledger | +| `ledger.validate` | Ledger validation | +| `ledger.close` | Close ledger | +| `ledger.replay` | Ledger replay executed | +| `ledger.delta` | Delta-based ledger acquired | +| `pathfind.request` | Path request initiated | +| `pathfind.compute` | Path computation executed | +| `txq.enqueue` | Transaction queued | +| `txq.apply` | Queued transaction applied | +| `fee.escalate` | Fee escalation triggered | +| `validator.list.fetch` | UNL list fetched | +| `validator.manifest` | Manifest update processed | +| `amendment.vote` | Amendment voting executed | +| `shamap.sync` | State tree synchronization | +| `job.enqueue` | Job added to queue | +| `job.execute` | Job execution | ### 2.3.3 Attribute Naming Conventions @@ -256,18 +203,19 @@ examples in §2.4 below follow these rules. ### 2.4.1 Resource Attributes (Set Once at Startup) -```cpp -// Standard OpenTelemetry semantic conventions -resource::SemanticConventions::SERVICE_NAME = "xrpld" -resource::SemanticConventions::SERVICE_VERSION = BuildInfo::getVersionString() -resource::SemanticConventions::SERVICE_INSTANCE_ID = +Resource attributes identify the process and are set once at startup. They use +the standard OpenTelemetry semantic conventions plus custom dotted `xrpl.*` +keys (the dotted form is reserved for resource scope per §2.3.3). -// Custom xrpld attributes -"xrpl.network.id" = // e.g., 0 for mainnet -"xrpl.network.type" = "mainnet" | "testnet" | "devnet" | "standalone" -"xrpl.node.type" = "validator" | "stock" | "reporting" -"xrpl.node.cluster" = // If clustered -``` +| Key | Type / value | Description | +| --------------------- | ---------------------------------------------------------- | ------------------------------ | +| `service.name` | `"xrpld"` | Standard `SERVICE_NAME` | +| `service.version` | `BuildInfo::getVersionString()` | Standard `SERVICE_VERSION` | +| `service.instance.id` | node public key (base58) | Standard `SERVICE_INSTANCE_ID` | +| `xrpl.network.id` | network id (e.g. 0 for mainnet) | Network identifier | +| `xrpl.network.type` | `"mainnet"` \| `"testnet"` \| `"devnet"` \| `"standalone"` | Network kind | +| `xrpl.node.type` | `"validator"` \| `"stock"` \| `"reporting"` | Node role | +| `xrpl.node.cluster` | cluster name | Cluster name, if clustered | ### 2.4.2 Span Attributes by Category @@ -280,107 +228,107 @@ resource::SemanticConventions::SERVICE_INSTANCE_ID = #### Transaction Attributes -```cpp -"tx_hash" = string // Transaction hash (hex) -"tx_type" = string // "Payment", "OfferCreate", etc. -"tx_account" = string // Source account (redacted in prod) -"tx_sequence" = int64 // Account sequence number -"tx_fee" = int64 // Fee in drops -"tx_result" = string // "tesSUCCESS", "tecPATH_DRY", etc. -"ledger_index" = int64 // Ledger containing transaction -``` +| Key | Type | Description | +| -------------- | ------ | ------------------------------------- | +| `tx_hash` | string | Transaction hash (hex) | +| `tx_type` | string | `"Payment"`, `"OfferCreate"`, etc. | +| `tx_account` | string | Source account (redacted in prod) | +| `tx_sequence` | int64 | Account sequence number | +| `tx_fee` | int64 | Fee in drops | +| `tx_result` | string | `"tesSUCCESS"`, `"tecPATH_DRY"`, etc. | +| `ledger_index` | int64 | Ledger containing transaction | #### Consensus Attributes -```cpp -"consensus_round" = int64 // Round number -"consensus_phase" = string // "open", "establish", "accept" -"consensus_mode" = string // "proposing", "observing", etc. -"proposers" = int64 // Number of proposers -"prev_ledger_prefix" = string // Previous ledger hash prefix -"ledger_seq" = int64 // Ledger sequence -"tx_count" = int64 // Transactions in consensus set -"round_time_ms" = float64 // Round duration -``` +| Key | Type | Description | +| -------------------- | ------- | ----------------------------------- | +| `consensus_round` | int64 | Round number | +| `consensus_phase` | string | `"open"`, `"establish"`, `"accept"` | +| `consensus_mode` | string | `"proposing"`, `"observing"`, etc. | +| `proposers` | int64 | Number of proposers | +| `prev_ledger_prefix` | string | Previous ledger hash prefix | +| `ledger_seq` | int64 | Ledger sequence | +| `tx_count` | int64 | Transactions in consensus set | +| `round_time_ms` | float64 | Round duration | #### RPC Attributes -```cpp -"command" = string // Command name (per-span unique on rpc.command) -"version" = int64 // API version -"rpc_role" = string // "admin" or "user" (qualified — "role" is generic) -"params" = string // Sanitized parameters (optional) -``` +| Key | Type | Description | +| ---------- | ------ | ----------------------------------------------------- | +| `command` | string | Command name (per-span unique on `rpc.command`) | +| `version` | int64 | API version | +| `rpc_role` | string | `"admin"` or `"user"` (qualified — `role` is generic) | +| `params` | string | Sanitized parameters (optional) | #### Peer & Message Attributes -```cpp -"peer_id" = string // Peer public key (base58) -"peer_address" = string // IP:port -"peer_latency_ms" = float64 // Measured latency -"peer_cluster" = string // Cluster name if clustered -"message_type" = string // Protocol message type name -"message_size_bytes" = int64 // Message size -"message_compressed" = bool // Whether compressed -``` +| Key | Type | Description | +| -------------------- | ------- | -------------------------- | +| `peer_id` | string | Peer public key (base58) | +| `peer_address` | string | IP:port | +| `peer_latency_ms` | float64 | Measured latency | +| `peer_cluster` | string | Cluster name if clustered | +| `message_type` | string | Protocol message type name | +| `message_size_bytes` | int64 | Message size | +| `message_compressed` | bool | Whether compressed | #### Ledger & Job Attributes -```cpp -"ledger_hash" = string // Ledger hash -"ledger_index" = int64 // Ledger sequence/index -"close_time" = int64 // Close time (epoch) -"ledger_tx_count" = int64 // Transaction count -"job_type" = string // Job type name -"job_queue_ms" = float64 // Time spent in queue -"job_worker" = int64 // Worker thread ID -``` +| Key | Type | Description | +| ----------------- | ------- | --------------------- | +| `ledger_hash` | string | Ledger hash | +| `ledger_index` | int64 | Ledger sequence/index | +| `close_time` | int64 | Close time (epoch) | +| `ledger_tx_count` | int64 | Transaction count | +| `job_type` | string | Job type name | +| `job_queue_ms` | float64 | Time spent in queue | +| `job_worker` | int64 | Worker thread ID | #### PathFinding Attributes -```cpp -"pathfind_source_currency" = string // Source currency code -"pathfind_dest_currency" = string // Destination currency code -"pathfind_path_count" = int64 // Number of paths found -"pathfind_cache_hit" = bool // RippleLineCache hit -``` +| Key | Type | Description | +| -------------------------- | ------ | ------------------------- | +| `pathfind_source_currency` | string | Source currency code | +| `pathfind_dest_currency` | string | Destination currency code | +| `pathfind_path_count` | int64 | Number of paths found | +| `pathfind_cache_hit` | bool | RippleLineCache hit | #### TxQ Attributes -```cpp -"txq_queue_depth" = int64 // Current queue depth -"txq_fee_level" = int64 // Fee level of transaction -"txq_eviction_reason" = string // Why transaction was evicted -``` +| Key | Type | Description | +| --------------------- | ------ | --------------------------- | +| `txq_queue_depth` | int64 | Current queue depth | +| `txq_fee_level` | int64 | Fee level of transaction | +| `txq_eviction_reason` | string | Why transaction was evicted | #### Fee Attributes -```cpp -"fee_load_factor" = int64 // Current load factor -"fee_escalation_level" = int64 // Fee escalation multiplier -``` +| Key | Type | Description | +| ---------------------- | ----- | ------------------------- | +| `fee_load_factor` | int64 | Current load factor | +| `fee_escalation_level` | int64 | Fee escalation multiplier | #### Validator Attributes -```cpp -"validator_list_size" = int64 // UNL size -"validator_list_age_sec" = int64 // Seconds since last update -``` +| Key | Type | Description | +| ------------------------ | ----- | ------------------------- | +| `validator_list_size` | int64 | UNL size | +| `validator_list_age_sec` | int64 | Seconds since last update | #### Amendment Attributes -```cpp -"amendment_name" = string // Amendment name -"amendment_status" = string // "enabled", "vetoed", "supported" -``` +| Key | Type | Description | +| ------------------ | ------ | -------------------------------------- | +| `amendment_name` | string | Amendment name | +| `amendment_status` | string | `"enabled"`, `"vetoed"`, `"supported"` | #### SHAMap Attributes -```cpp -"shamap_type" = string // "transaction", "state", "account_state" -"shamap_missing_nodes" = int64 // Number of missing nodes during sync -"shamap_duration_ms" = float64 // Sync duration -``` +| Key | Type | Description | +| ---------------------- | ------- | --------------------------------------------- | +| `shamap_type` | string | `"transaction"`, `"state"`, `"account_state"` | +| `shamap_missing_nodes` | int64 | Number of missing nodes during sync | +| `shamap_duration_ms` | float64 | Sync duration | ### 2.4.3 Data Collection Summary @@ -433,41 +381,19 @@ The following data is explicitly **excluded** from telemetry collection: #### Collector-Level Data Protection -The OpenTelemetry Collector can be configured to hash or redact sensitive attributes before export: - -```yaml -processors: - attributes: - actions: - # Hash account addresses before storage - - key: tx_account - action: hash - # Remove IP addresses entirely - - key: peer_address - action: delete - # Redact specific fields - - key: params - action: delete -``` +The OpenTelemetry Collector can be configured (via an `attributes` processor) +to hash or redact sensitive attributes before export — for example, hashing +`tx_account`, deleting `peer_address` to drop IP addresses, and deleting +`params` to redact request parameters. #### Configuration Options for Privacy -In `xrpld.cfg`, operators can control data collection granularity: - -```ini -[telemetry] -enabled=1 - -# Disable collection of specific components -trace_transactions=1 -trace_consensus=1 -trace_rpc=1 -trace_peer=0 # Disable peer tracing (high volume) - -# Redact specific attributes -redact_account=1 # Hash account addresses before export -redact_peer_address=1 # Remove peer IP addresses -``` +In `xrpld.cfg`, operators control data collection granularity through the +`[telemetry]` section. Besides `enabled`, per-component toggles +(`trace_transactions`, `trace_consensus`, `trace_rpc`, `trace_peer` — the last +often disabled due to high volume) select which spans are emitted, and +redaction flags (`redact_account` to hash account addresses, `redact_peer_address` +to remove peer IP addresses) control SDK-level redaction before export. > **Note**: The `redact_account` configuration in `xrpld.cfg` controls SDK-level redaction before export, while collector-level filtering (see [Collector-Level Data Protection](#collector-level-data-protection) above) provides an additional defense-in-depth layer. Both can operate independently. @@ -540,15 +466,8 @@ xrpld already has two observability mechanisms. OpenTelemetry complements (not r - No parent-child relationships between events - Manual log parsing required -```json -// Example PerfLog entry -{ - "time": "2024-01-15T10:30:00.123Z", - "method": "submit", - "duration_us": 1523, - "result": "tesSUCCESS" -} -``` +A PerfLog entry is a JSON object with fields such as `time`, `method`, +`duration_us`, and `result`. #### Beast Insight (StatsD) @@ -562,12 +481,8 @@ xrpld already has two observability mechanisms. OpenTelemetry complements (not r - No causal relationships - Single-node perspective -```cpp -// Example StatsD usage in xrpld -insight.increment("rpc.submit.count"); -insight.gauge("ledger.age", age); -insight.timing("consensus.round", duration); -``` +In xrpld, Beast Insight is used through `increment` (counters), `gauge` +(point-in-time values), and `timing` (durations) calls. #### OpenTelemetry (NEW) @@ -581,13 +496,9 @@ insight.timing("consensus.round", duration); - Requires collector infrastructure - Higher complexity than logging -```cpp -// Example OpenTelemetry span -auto span = telemetry.startSpan("tx.relay"); -span->SetAttribute("tx_hash", hash); -span->SetAttribute("peer_id", peerId); -// Span automatically linked to parent via context -``` +A span is created via `startSpan` (e.g. `"tx.relay"`), annotated with +attributes such as `tx_hash` and `peer_id`, and is automatically linked to its +parent through the active context. ### 2.6.3 When to Use Each @@ -632,41 +543,13 @@ flowchart TB ### 2.6.5 Correlation with PerfLog -Trace IDs can be correlated with existing PerfLog entries for comprehensive debugging: - -```cpp -// In RPCHandler.cpp - correlate trace with PerfLog -Status doCommand(RPC::JsonContext& context, Json::Value& result) -{ - // Start OpenTelemetry span - auto span = context.app.getTelemetry().startSpan( - "rpc.command." + context.method); - - // Get trace ID for correlation - auto traceId = span->GetContext().trace_id().IsValid() - ? toHex(span->GetContext().trace_id()) - : ""; - - // Use existing PerfLog with trace correlation - auto const curId = context.app.getPerfLog().currentId(); - context.app.getPerfLog().rpcStart(context.method, curId); - - // Future: Add trace ID to PerfLog entry - // context.app.getPerfLog().setTraceId(curId, traceId); - - try { - auto ret = handler(context, result); - context.app.getPerfLog().rpcFinish(context.method, curId); - span->SetStatus(opentelemetry::trace::StatusCode::kOk); - return ret; - } catch (std::exception const& e) { - context.app.getPerfLog().rpcError(context.method, curId); - span->RecordException(e); - span->SetStatus(opentelemetry::trace::StatusCode::kError, e.what()); - throw; - } -} -``` +Trace IDs can be correlated with existing PerfLog entries for comprehensive +debugging. The design is for `RPCHandler.cpp` to start an `rpc.command.` +span alongside the existing PerfLog `rpcStart`/`rpcFinish`/`rpcError` calls, +extract the span's `trace_id` (when valid), and eventually stamp it onto the +PerfLog entry (a planned `setTraceId` hook) so logs and traces share a key. The +span status is set to OK on success or to error (recording the exception) on +failure. --- diff --git a/OpenTelemetryPlan/03-implementation-strategy.md b/OpenTelemetryPlan/03-implementation-strategy.md index 8a3eda6338..697e8af019 100644 --- a/OpenTelemetryPlan/03-implementation-strategy.md +++ b/OpenTelemetryPlan/03-implementation-strategy.md @@ -1,7 +1,7 @@ # Implementation Strategy > **Parent Document**: [OpenTelemetryPlan.md](./OpenTelemetryPlan.md) -> **Related**: [Code Samples](./04-code-samples.md) | [Configuration Reference](./05-configuration-reference.md) +> **Related**: [Configuration Reference](./05-configuration-reference.md) --- @@ -314,27 +314,12 @@ flowchart TD ### 3.7.3 Conditional Instrumentation -```cpp -// Compile-time feature flag -#ifndef XRPL_ENABLE_TELEMETRY -// Zero-cost when disabled -#define XRPL_TRACE_SPAN(t, n) ((void)0) -#endif - -// Runtime component filtering -if (telemetry.shouldTracePeer()) -{ - XRPL_TRACE_SPAN(telemetry, "peer.message.receive"); - // ... instrumentation -} -// No overhead when component tracing disabled -``` +Instrumentation is gated on two levels. A compile-time feature flag (`XRPL_ENABLE_TELEMETRY`) reduces the trace macros to no-ops when telemetry is built out, so disabled builds carry zero cost. At runtime, per-component guards (e.g. `shouldTracePeer()`) skip span creation for components whose tracing is turned off, incurring no overhead beyond a single boolean check. --- ## 3.8 Links to Detailed Documentation -- **[Code Samples](./04-code-samples.md)**: Complete implementation code for all components - **[Configuration Reference](./05-configuration-reference.md)**: Configuration options and collector setup - **[Implementation Phases](./06-implementation-phases.md)**: Detailed timeline and milestones @@ -482,47 +467,10 @@ If issues are discovered after deployment: ### 3.9.7 Code Change Examples -**Minimal RPC Instrumentation (Low Intrusiveness):** +**Minimal RPC Instrumentation (Low Intrusiveness):** Instrumenting an RPC handler adds roughly 3-4 lines: one macro to start the span and one or two `setAttribute` calls (command name, status). The span ends automatically via RAII, so the existing control flow — process the request, send the result — is untouched. -```cpp -// Before -void ServerHandler::onRequest(...) { - auto result = processRequest(req); - send(result); -} - -// After (only ~10 lines added) -void ServerHandler::onRequest(...) { - XRPL_TRACE_RPC(app_.getTelemetry(), "rpc.request"); // +1 line - XRPL_TRACE_SET_ATTR("command", command); // +1 line - - auto result = processRequest(req); - - XRPL_TRACE_SET_ATTR("rpc_status", status); // +1 line - send(result); -} -``` - -**Consensus Instrumentation (Medium Intrusiveness):** - -```cpp -// Before -void RCLConsensusAdaptor::startRound(...) { - // ... existing logic -} - -// After (context storage required) -void RCLConsensusAdaptor::startRound(...) { - XRPL_TRACE_CONSENSUS(app_.getTelemetry(), "consensus.round"); - XRPL_TRACE_SET_ATTR("ledger_seq", seq); - - // Store context for child spans in phase transitions - currentRoundContext_ = _xrpl_guard_->context(); // New member variable - - // ... existing logic unchanged -} -``` +**Consensus Instrumentation (Medium Intrusiveness):** Consensus is slightly more intrusive because child spans in later phase transitions need the round's context. Beyond the span-start and attribute macros, this requires storing the active context in a new member variable (`currentRoundContext_`) at round start. The existing round logic itself remains unchanged. --- -_Previous: [Design Decisions](./02-design-decisions.md)_ | _Next: [Code Samples](./04-code-samples.md)_ | _Back to: [Overview](./OpenTelemetryPlan.md)_ +_Previous: [Design Decisions](./02-design-decisions.md)_ | _Next: [Configuration Reference](./05-configuration-reference.md)_ | _Back to: [Overview](./OpenTelemetryPlan.md)_ diff --git a/OpenTelemetryPlan/04-code-samples.md b/OpenTelemetryPlan/04-code-samples.md deleted file mode 100644 index 497ff9d5ea..0000000000 --- a/OpenTelemetryPlan/04-code-samples.md +++ /dev/null @@ -1,1129 +0,0 @@ -# Code Samples - -> **Parent Document**: [OpenTelemetryPlan.md](./OpenTelemetryPlan.md) -> **Related**: [Implementation Strategy](./03-implementation-strategy.md) | [Configuration Reference](./05-configuration-reference.md) - ---- - -## 4.1 Core Interfaces - -> **OTLP** = OpenTelemetry Protocol - -### 4.1.1 Main Telemetry Interface - -```cpp -// include/xrpl/telemetry/Telemetry.h -#pragma once - -#include -#include -#include -#include - -#include -#include -#include - -namespace xrpl { -namespace telemetry { - -/** - * Main telemetry interface for OpenTelemetry integration. - * - * This class provides the primary API for distributed tracing in xrpld. - * It manages the OpenTelemetry SDK lifecycle and provides convenience - * methods for creating spans and propagating context. - */ -class Telemetry -{ -public: - /** - * Configuration for the telemetry system. - */ - struct Setup - { - bool enabled = false; - std::string serviceName = "xrpld"; - std::string serviceVersion; - std::string serviceInstanceId; // Node public key - - // Exporter configuration - std::string exporterType = "otlp_grpc"; // "otlp_grpc", "otlp_http", "none" - std::string exporterEndpoint = "localhost:4317"; - bool useTls = false; - std::string tlsCertPath; - - // Head sampling: fixed at 1.0 (sample everything), not config-driven. - // Keeps trace keep/drop decisions coherent across nodes; volume - // reduction is delegated to the collector's tail sampling. - double samplingRatio = 1.0; - - // Batch processor settings - std::uint32_t batchSize = 512; - std::chrono::milliseconds batchDelay{5000}; - std::uint32_t maxQueueSize = 2048; - - // Network attributes - std::uint32_t networkId = 0; - std::string networkType = "mainnet"; - - // Component filtering - bool traceTransactions = true; - bool traceConsensus = true; - bool traceRpc = true; - bool tracePeer = true; // High volume, enabled by default - bool traceLedger = true; - bool tracePathfind = true; - bool traceTxQ = true; - bool traceValidator = false; // Low volume, disabled by default - bool traceAmendment = false; // Very low volume, disabled by default - }; - - virtual ~Telemetry() = default; - - // ═══════════════════════════════════════════════════════════════════════ - // LIFECYCLE - // ═══════════════════════════════════════════════════════════════════════ - - /** Start the telemetry system (call after configuration) */ - virtual void start() = 0; - - /** Stop the telemetry system (flushes pending spans) */ - virtual void stop() = 0; - - /** Check if telemetry is enabled */ - virtual bool isEnabled() const = 0; - - // ═══════════════════════════════════════════════════════════════════════ - // TRACER ACCESS - // ═══════════════════════════════════════════════════════════════════════ - - /** Get the tracer for creating spans */ - virtual opentelemetry::nostd::shared_ptr - getTracer(std::string_view name = "xrpld") = 0; - - // ═══════════════════════════════════════════════════════════════════════ - // SPAN CREATION (Convenience Methods) - // ═══════════════════════════════════════════════════════════════════════ - - /** Start a new span with default options */ - virtual opentelemetry::nostd::shared_ptr - startSpan( - std::string_view name, - opentelemetry::trace::SpanKind kind = - opentelemetry::trace::SpanKind::kInternal) = 0; - - /** Start a span as child of given context */ - virtual opentelemetry::nostd::shared_ptr - startSpan( - std::string_view name, - opentelemetry::context::Context const& parentContext, - opentelemetry::trace::SpanKind kind = - opentelemetry::trace::SpanKind::kInternal) = 0; - - // ═══════════════════════════════════════════════════════════════════════ - // CONTEXT PROPAGATION - // ═══════════════════════════════════════════════════════════════════════ - - /** Serialize context for network transmission */ - virtual std::string serializeContext( - opentelemetry::context::Context const& ctx) = 0; - - /** Deserialize context from network data */ - virtual opentelemetry::context::Context deserializeContext( - std::string const& serialized) = 0; - - // ═══════════════════════════════════════════════════════════════════════ - // COMPONENT FILTERING - // ═══════════════════════════════════════════════════════════════════════ - - /** Check if transaction tracing is enabled */ - virtual bool shouldTraceTransactions() const = 0; - - /** Check if consensus tracing is enabled */ - virtual bool shouldTraceConsensus() const = 0; - - /** Check if RPC tracing is enabled */ - virtual bool shouldTraceRpc() const = 0; - - /** Check if peer message tracing is enabled */ - virtual bool shouldTracePeer() const = 0; - - /** Check if ledger tracing is enabled */ - virtual bool shouldTraceLedger() const = 0; - - /** Check if path finding tracing is enabled */ - virtual bool shouldTracePathfind() const = 0; - - /** Check if transaction queue tracing is enabled */ - virtual bool shouldTraceTxQ() const = 0; - - /** Check if validator list/manifest tracing is enabled */ - virtual bool shouldTraceValidator() const = 0; - - /** Check if amendment voting tracing is enabled */ - virtual bool shouldTraceAmendment() const = 0; -}; - -// Factory functions -std::unique_ptr -make_Telemetry( - Telemetry::Setup const& setup, - beast::Journal journal); - -Telemetry::Setup -setup_Telemetry( - Section const& section, - std::string const& nodePublicKey, - std::string const& version); - -} // namespace telemetry -} // namespace xrpl -``` - ---- - -## 4.2 RAII Span Guard - -```cpp -// include/xrpl/telemetry/SpanGuard.h -#pragma once - -#include -#include -#include - -#include -#include - -namespace xrpl { -namespace telemetry { - -/** - * RAII guard for OpenTelemetry spans. - * - * Automatically ends the span on destruction and makes it the current - * span in the thread-local context. - */ -class SpanGuard -{ - opentelemetry::nostd::shared_ptr span_; - opentelemetry::trace::Scope scope_; - -public: - /** - * Construct guard with span. - * The span becomes the current span in thread-local context. - * - * @note If span is nullptr (e.g., telemetry disabled), the guard - * becomes a no-op. All methods safely check for null before access. - */ - explicit SpanGuard( - opentelemetry::nostd::shared_ptr span) - : span_(span ? std::move(span) : nullptr) - , scope_(span_ ? opentelemetry::trace::Scope(span_) - : opentelemetry::trace::Scope( - opentelemetry::nostd::shared_ptr< - opentelemetry::trace::Span>(nullptr))) - { - } - - // Non-copyable, non-movable - SpanGuard(SpanGuard const&) = delete; - SpanGuard& operator=(SpanGuard const&) = delete; - SpanGuard(SpanGuard&&) = delete; - SpanGuard& operator=(SpanGuard&&) = delete; - - ~SpanGuard() - { - if (span_) - span_->End(); - } - - /** Access the underlying span */ - opentelemetry::trace::Span& span() { return *span_; } - opentelemetry::trace::Span const& span() const { return *span_; } - - /** Set span status to OK */ - void setOk() - { - span_->SetStatus(opentelemetry::trace::StatusCode::kOk); - } - - /** Set span status with code and description */ - void setStatus( - opentelemetry::trace::StatusCode code, - std::string_view description = "") - { - span_->SetStatus(code, std::string(description)); - } - - /** Set an attribute on the span */ - template - void setAttribute(std::string_view key, T&& value) - { - span_->SetAttribute( - opentelemetry::nostd::string_view(key.data(), key.size()), - std::forward(value)); - } - - /** Add an event to the span */ - void addEvent(std::string_view name) - { - span_->AddEvent(std::string(name)); - } - - /** Record an exception on the span */ - void recordException(std::exception const& e) - { - span_->RecordException(e); - span_->SetStatus( - opentelemetry::trace::StatusCode::kError, - e.what()); - } - - /** Get the current trace context */ - opentelemetry::context::Context context() const - { - return opentelemetry::context::RuntimeContext::GetCurrent(); - } -}; - -/** - * No-op span guard for when tracing is disabled. - * Provides the same interface but does nothing. - */ -class NullSpanGuard -{ -public: - NullSpanGuard() = default; - - void setOk() {} - void setStatus(opentelemetry::trace::StatusCode, std::string_view = "") {} - - template - void setAttribute(std::string_view, T&&) {} - - void addEvent(std::string_view) {} - void recordException(std::exception const&) {} - - /** Return a default empty context (matches SpanGuard interface) */ - opentelemetry::context::Context context() const - { - return opentelemetry::context::Context{}; - } -}; - -} // namespace telemetry -} // namespace xrpl -``` - ---- - -## 4.3 Instrumentation Macros - -```cpp -// src/xrpld/telemetry/TracingInstrumentation.h -#pragma once - -#include -#include - -namespace xrpl { -namespace telemetry { - -// ═══════════════════════════════════════════════════════════════════════════ -// INSTRUMENTATION MACROS -// ═══════════════════════════════════════════════════════════════════════════ - -#ifdef XRPL_ENABLE_TELEMETRY - -// Start a span that is automatically ended when guard goes out of scope -#define XRPL_TRACE_SPAN(telemetry, name) \ - auto _xrpl_span_ = (telemetry).startSpan(name); \ - ::xrpl::telemetry::SpanGuard _xrpl_guard_(_xrpl_span_) - -// Start a span with specific kind -#define XRPL_TRACE_SPAN_KIND(telemetry, name, kind) \ - auto _xrpl_span_ = (telemetry).startSpan(name, kind); \ - ::xrpl::telemetry::SpanGuard _xrpl_guard_(_xrpl_span_) - -// Conditional span based on component -#define XRPL_TRACE_TX(telemetry, name) \ - std::optional<::xrpl::telemetry::SpanGuard> _xrpl_guard_; \ - if ((telemetry).shouldTraceTransactions()) { \ - _xrpl_guard_.emplace((telemetry).startSpan(name)); \ - } - -#define XRPL_TRACE_CONSENSUS(telemetry, name) \ - std::optional<::xrpl::telemetry::SpanGuard> _xrpl_guard_; \ - if ((telemetry).shouldTraceConsensus()) { \ - _xrpl_guard_.emplace((telemetry).startSpan(name)); \ - } - -#define XRPL_TRACE_RPC(telemetry, name) \ - std::optional<::xrpl::telemetry::SpanGuard> _xrpl_guard_; \ - if ((telemetry).shouldTraceRpc()) { \ - _xrpl_guard_.emplace((telemetry).startSpan(name)); \ - } - -#define XRPL_TRACE_PEER(telemetry, name) \ - std::optional<::xrpl::telemetry::SpanGuard> _xrpl_guard_; \ - if ((telemetry).shouldTracePeer()) { \ - _xrpl_guard_.emplace((telemetry).startSpan(name)); \ - } - -#define XRPL_TRACE_LEDGER(telemetry, name) \ - std::optional<::xrpl::telemetry::SpanGuard> _xrpl_guard_; \ - if ((telemetry).shouldTraceLedger()) { \ - _xrpl_guard_.emplace((telemetry).startSpan(name)); \ - } - -#define XRPL_TRACE_PATHFIND(telemetry, name) \ - std::optional<::xrpl::telemetry::SpanGuard> _xrpl_guard_; \ - if ((telemetry).shouldTracePathfind()) { \ - _xrpl_guard_.emplace((telemetry).startSpan(name)); \ - } - -#define XRPL_TRACE_TXQ(telemetry, name) \ - std::optional<::xrpl::telemetry::SpanGuard> _xrpl_guard_; \ - if ((telemetry).shouldTraceTxQ()) { \ - _xrpl_guard_.emplace((telemetry).startSpan(name)); \ - } - -#define XRPL_TRACE_VALIDATOR(telemetry, name) \ - std::optional<::xrpl::telemetry::SpanGuard> _xrpl_guard_; \ - if ((telemetry).shouldTraceValidator()) { \ - _xrpl_guard_.emplace((telemetry).startSpan(name)); \ - } - -#define XRPL_TRACE_AMENDMENT(telemetry, name) \ - std::optional<::xrpl::telemetry::SpanGuard> _xrpl_guard_; \ - if ((telemetry).shouldTraceAmendment()) { \ - _xrpl_guard_.emplace((telemetry).startSpan(name)); \ - } - -// Set attribute on current span (if exists). -// Works with both std::optional (from conditional macros) -// and bare SpanGuard (from XRPL_TRACE_SPAN). Uses 'if constexpr'-like -// dispatch via a helper that checks for .has_value(). -#define XRPL_TRACE_SET_ATTR(key, value) \ - do { \ - if constexpr (requires { _xrpl_guard_.has_value(); }) { \ - if (_xrpl_guard_.has_value()) \ - _xrpl_guard_->setAttribute(key, value); \ - } else { \ - _xrpl_guard_.setAttribute(key, value); \ - } \ - } while(0) - -// Record exception on current span -#define XRPL_TRACE_EXCEPTION(e) \ - do { \ - if constexpr (requires { _xrpl_guard_.has_value(); }) { \ - if (_xrpl_guard_.has_value()) \ - _xrpl_guard_->recordException(e); \ - } else { \ - _xrpl_guard_.recordException(e); \ - } \ - } while(0) - -#else // XRPL_ENABLE_TELEMETRY not defined - -#define XRPL_TRACE_SPAN(telemetry, name) ((void)0) -#define XRPL_TRACE_SPAN_KIND(telemetry, name, kind) ((void)0) -#define XRPL_TRACE_TX(telemetry, name) ((void)0) -#define XRPL_TRACE_CONSENSUS(telemetry, name) ((void)0) -#define XRPL_TRACE_RPC(telemetry, name) ((void)0) -#define XRPL_TRACE_PEER(telemetry, name) ((void)0) -#define XRPL_TRACE_LEDGER(telemetry, name) ((void)0) -#define XRPL_TRACE_PATHFIND(telemetry, name) ((void)0) -#define XRPL_TRACE_TXQ(telemetry, name) ((void)0) -#define XRPL_TRACE_VALIDATOR(telemetry, name) ((void)0) -#define XRPL_TRACE_AMENDMENT(telemetry, name) ((void)0) -#define XRPL_TRACE_SET_ATTR(key, value) ((void)0) -#define XRPL_TRACE_EXCEPTION(e) ((void)0) - -#endif // XRPL_ENABLE_TELEMETRY - -} // namespace telemetry -} // namespace xrpl -``` - ---- - -## 4.4 Protocol Buffer Extensions - -### 4.4.1 TraceContext Message Definition - -Add to `src/xrpld/overlay/detail/ripple.proto`: - -```protobuf -// Note: xrpld uses proto2 syntax. The 'optional' keyword below is valid -// in proto2 (it is the default field rule) and is included for clarity. - -// Trace context for distributed tracing across nodes -// Uses W3C Trace Context format internally -message TraceContext { - // 16-byte trace identifier (required for valid context) - bytes trace_id = 1; - - // 8-byte span identifier of parent span - bytes span_id = 2; - - // Trace flags (bit 0 = sampled) - uint32 trace_flags = 3; - - // W3C tracestate header value for vendor-specific data - string trace_state = 4; -} - -// Extend existing messages with optional trace context -// High field numbers (1000+) to avoid conflicts - -message TMTransaction { - // ... existing fields ... - - // Optional trace context for distributed tracing - optional TraceContext trace_context = 1001; -} - -message TMProposeSet { - // ... existing fields ... - optional TraceContext trace_context = 1001; -} - -message TMValidation { - // ... existing fields ... - optional TraceContext trace_context = 1001; -} - -message TMGetLedger { - // ... existing fields ... - optional TraceContext trace_context = 1001; -} - -message TMLedgerData { - // ... existing fields ... - optional TraceContext trace_context = 1001; -} -``` - -### 4.4.2 Context Serialization/Deserialization - -```cpp -// include/xrpl/telemetry/TraceContext.h -#pragma once - -#include -#include -#include -#include -#include // Generated protobuf - -#include -#include - -namespace xrpl { -namespace telemetry { - -/** - * Utilities for trace context serialization and propagation. - */ -class TraceContextPropagator -{ -public: - /** - * Extract trace context from Protocol Buffer message. - * Returns empty context if no trace info present. - */ - static opentelemetry::context::Context - extract(protocol::TraceContext const& proto); - - /** - * Inject current trace context into Protocol Buffer message. - */ - static void - inject( - opentelemetry::context::Context const& ctx, - protocol::TraceContext& proto); - - /** - * Extract trace context from HTTP headers (for RPC). - * Supports W3C Trace Context (traceparent, tracestate). - */ - static opentelemetry::context::Context - extractFromHeaders( - std::function(std::string_view)> headerGetter); - - /** - * Inject trace context into HTTP headers (for RPC responses). - */ - static void - injectToHeaders( - opentelemetry::context::Context const& ctx, - std::function headerSetter); -}; - -// ═══════════════════════════════════════════════════════════════════════════ -// IMPLEMENTATION -// ═══════════════════════════════════════════════════════════════════════════ - -inline opentelemetry::context::Context -TraceContextPropagator::extract(protocol::TraceContext const& proto) -{ - using namespace opentelemetry::trace; - - if (proto.trace_id().size() != 16 || proto.span_id().size() != 8) - { - // Log malformed trace context for debugging. Silent failures in - // context extraction make distributed tracing issues hard to diagnose. - JLOG(j_.warn()) << "Malformed trace context: trace_id size=" - << proto.trace_id().size() - << " span_id size=" << proto.span_id().size(); - return opentelemetry::context::Context{}; - } - - // Construct TraceId and SpanId from bytes - TraceId traceId(reinterpret_cast(proto.trace_id().data())); - SpanId spanId(reinterpret_cast(proto.span_id().data())); - TraceFlags flags(static_cast(proto.trace_flags())); - - // Create SpanContext from extracted data - SpanContext spanContext(traceId, spanId, flags, /* remote = */ true); - - // DefaultSpan wraps SpanContext for use as a non-recording parent. - // This is the standard OTel C++ pattern for remote context propagation. - // DefaultSpan carries the remote SpanContext without recording any data. - auto parentCtx = opentelemetry::trace::SetSpan( - opentelemetry::context::Context{}, - opentelemetry::nostd::shared_ptr( - new DefaultSpan(spanContext))); - - return parentCtx; -} - -inline void -TraceContextPropagator::inject( - opentelemetry::context::Context const& ctx, - protocol::TraceContext& proto) -{ - using namespace opentelemetry::trace; - - // Get current span from context - auto span = GetSpan(ctx); - if (!span) - return; - - auto const& spanCtx = span->GetContext(); - if (!spanCtx.IsValid()) - return; - - // Serialize trace_id (16 bytes) - auto const& traceId = spanCtx.trace_id(); - proto.set_trace_id(traceId.Id().data(), TraceId::kSize); - - // Serialize span_id (8 bytes) - auto const& spanId = spanCtx.span_id(); - proto.set_span_id(spanId.Id().data(), SpanId::kSize); - - // Serialize flags - proto.set_trace_flags(spanCtx.trace_flags().flags()); - - // Note: tracestate not implemented yet -} - -} // namespace telemetry -} // namespace xrpl -``` - ---- - -## 4.5 Module-Specific Instrumentation - -### 4.5.1 Transaction Relay Instrumentation - -```cpp -// src/xrpld/overlay/detail/PeerImp.cpp (modified) - -#include - -void -PeerImp::handleTransaction( - std::shared_ptr const& m) -{ - // Extract trace context from incoming message - opentelemetry::context::Context parentCtx; - if (m->has_trace_context()) - { - parentCtx = telemetry::TraceContextPropagator::extract( - m->trace_context()); - } - - // Start span as child of remote span (cross-node link) - auto span = app_.getTelemetry().startSpan( - "tx.receive", - parentCtx, - opentelemetry::trace::SpanKind::kServer); - telemetry::SpanGuard guard(span); - - try - { - // Parse and validate transaction - SerialIter sit(makeSlice(m->rawtransaction())); - auto stx = std::make_shared(sit); - - // Add transaction attributes - guard.setAttribute("tx_hash", to_string(stx->getTransactionID())); - guard.setAttribute("tx_type", stx->getTxnType()); - guard.setAttribute("peer_id", remote_address_.to_string()); - - // Check if we've seen this transaction (HashRouter) - auto const [flags, suppressed] = - app_.getHashRouter().addSuppressionPeer( - stx->getTransactionID(), - id_); - - if (suppressed) - { - guard.setAttribute("suppressed", true); - guard.addEvent("tx.duplicate"); - return; // Already processing this transaction - } - - // Create child span for validation - { - auto validateSpan = app_.getTelemetry().startSpan("tx.validate"); - telemetry::SpanGuard validateGuard(validateSpan); - - auto [validity, reason] = checkTransaction(stx); - validateGuard.setAttribute("validity", - validity == Validity::Valid ? "valid" : "invalid"); - - if (validity != Validity::Valid) - { - validateGuard.setStatus( - opentelemetry::trace::StatusCode::kError, - reason); - return; - } - } - - // Relay to other peers (capture context for propagation) - auto ctx = guard.context(); - - // Create child span for relay - auto relaySpan = app_.getTelemetry().startSpan( - "tx.relay", - ctx, - opentelemetry::trace::SpanKind::kClient); - { - telemetry::SpanGuard relayGuard(relaySpan); - - // Inject context into outgoing message - protocol::TraceContext protoCtx; - telemetry::TraceContextPropagator::inject( - relayGuard.context(), protoCtx); - - // Relay to other peers - app_.getOverlay().relay( - stx->getTransactionID(), - *m, - protoCtx, // Pass trace context - exclusions); - - relayGuard.setAttribute("relay_count", - static_cast(relayCount)); - } - - guard.setOk(); - } - catch (std::exception const& e) - { - guard.recordException(e); - JLOG(journal_.warn()) << "Transaction handling failed: " << e.what(); - } -} -``` - -### 4.5.2 Consensus Instrumentation - -```cpp -// src/xrpld/app/consensus/RCLConsensus.cpp (modified) - -#include - -void -RCLConsensusAdaptor::startRound( - NetClock::time_point const& now, - RCLCxLedger::ID const& prevLedgerHash, - RCLCxLedger const& prevLedger, - hash_set const& peers, - bool proposing) -{ - XRPL_TRACE_CONSENSUS(app_.getTelemetry(), "consensus.round"); - - XRPL_TRACE_SET_ATTR("prev_ledger_prefix", to_string(prevLedgerHash)); - XRPL_TRACE_SET_ATTR("ledger_seq", - static_cast(prevLedger.seq() + 1)); - XRPL_TRACE_SET_ATTR("proposers", - static_cast(peers.size())); - XRPL_TRACE_SET_ATTR("consensus_mode", - proposing ? "proposing" : "observing"); - - // Store trace context for use in phase transitions - currentRoundContext_ = _xrpl_guard_.has_value() - ? _xrpl_guard_->context() - : opentelemetry::context::Context{}; - - // ... existing implementation ... -} - -ConsensusPhase -RCLConsensusAdaptor::phaseTransition(ConsensusPhase newPhase) -{ - // Create span for phase transition - auto span = app_.getTelemetry().startSpan( - "consensus.phase." + to_string(newPhase), - currentRoundContext_); - telemetry::SpanGuard guard(span); - - guard.setAttribute("consensus_phase", to_string(newPhase)); - guard.addEvent("phase.enter"); - - auto const startTime = std::chrono::steady_clock::now(); - - try - { - auto result = doPhaseTransition(newPhase); - - auto const duration = std::chrono::steady_clock::now() - startTime; - guard.setAttribute("consensus_phase_duration_ms", - std::chrono::duration(duration).count()); - - guard.setOk(); - return result; - } - catch (std::exception const& e) - { - guard.recordException(e); - throw; - } -} - -void -RCLConsensusAdaptor::peerProposal( - NetClock::time_point const& now, - RCLCxPeerPos const& proposal) -{ - // Extract trace context from proposal message - opentelemetry::context::Context parentCtx; - if (proposal.hasTraceContext()) - { - parentCtx = telemetry::TraceContextPropagator::extract( - proposal.traceContext()); - } - - auto span = app_.getTelemetry().startSpan( - "consensus.proposal.receive", - parentCtx, - opentelemetry::trace::SpanKind::kServer); - telemetry::SpanGuard guard(span); - - guard.setAttribute("proposer", - toBase58(TokenType::NodePublic, proposal.nodeId())); - guard.setAttribute("consensus_round", - static_cast(proposal.proposal().proposeSeq())); - - // ... existing implementation ... - - guard.setOk(); -} -``` - -### 4.5.3 RPC Handler Instrumentation - -```cpp -// src/xrpld/rpc/detail/ServerHandler.cpp (modified) - -#include - -void -ServerHandler::onRequest( - http_request_type&& req, - std::function&& send) -{ - // Extract trace context from HTTP headers (W3C Trace Context) - auto parentCtx = telemetry::TraceContextPropagator::extractFromHeaders( - [&req](std::string_view name) -> std::optional { - // Beast's find() accepts a string_view for custom header lookup - auto it = req.find(name); - if (it != req.end()) - return std::string(it->value()); - return std::nullopt; - }); - - // Start request span - auto span = app_.getTelemetry().startSpan( - "rpc.request", - parentCtx, - opentelemetry::trace::SpanKind::kServer); - telemetry::SpanGuard guard(span); - - // Add HTTP attributes - guard.setAttribute("http.method", std::string(req.method_string())); - guard.setAttribute("http.target", std::string(req.target())); - guard.setAttribute("http.user_agent", - std::string(req[boost::beast::http::field::user_agent])); - - auto const startTime = std::chrono::steady_clock::now(); - - try - { - // Parse and process request - auto const& body = req.body(); - Json::Value jv; - Json::Reader reader; - - if (!reader.parse(body, jv)) - { - guard.setStatus( - opentelemetry::trace::StatusCode::kError, - "Invalid JSON"); - sendError(send, "Invalid JSON"); - return; - } - - // Extract command name - std::string command = jv.isMember("command") - ? jv["command"].asString() - : jv.isMember("method") - ? jv["method"].asString() - : "unknown"; - - guard.setAttribute("command", command); - - // Create child span for command execution - auto cmdSpan = app_.getTelemetry().startSpan( - "rpc.command." + command); - { - telemetry::SpanGuard cmdGuard(cmdSpan); - - // Execute RPC command - auto result = processRequest(jv); - - // Record result attributes - if (result.isMember("status")) - { - cmdGuard.setAttribute("rpc_status", - result["status"].asString()); - } - - if (result["status"].asString() == "error") - { - cmdGuard.setStatus( - opentelemetry::trace::StatusCode::kError, - result.isMember("error_message") - ? result["error_message"].asString() - : "RPC error"); - } - else - { - cmdGuard.setOk(); - } - } - - auto const duration = std::chrono::steady_clock::now() - startTime; - guard.setAttribute("http.duration_ms", - std::chrono::duration(duration).count()); - - // Inject trace context into response headers - http_response_type resp; - telemetry::TraceContextPropagator::injectToHeaders( - guard.context(), - [&resp](std::string_view name, std::string_view value) { - resp.set(std::string(name), std::string(value)); - }); - - guard.setOk(); - send(std::move(resp)); - } - catch (std::exception const& e) - { - guard.recordException(e); - JLOG(journal_.error()) << "RPC request failed: " << e.what(); - sendError(send, e.what()); - } -} -``` - -### 4.5.4 JobQueue Context Propagation - -> **Architecture note**: `JobQueue` and its inner `Workers` class do not -> hold an `Application&` or `ServiceRegistry&`. They receive a -> `perf::PerfLog*` at construction. To instrument job execution, a -> `telemetry::Telemetry&` must be threaded into `JobQueue`'s constructor -> alongside the existing `PerfLog&`, or the trace context can be -> captured/restored without starting new spans inside the worker itself. -> -> The approach below captures trace context at job-creation time and -> restores it when the job executes, so that any spans created _inside_ -> the job body automatically become children of the original caller's -> trace. This requires adding a `telemetry::Telemetry&` to `JobQueue`. - -```cpp -// include/xrpl/core/JobQueue.h (modified) - -#ifdef XRPL_ENABLE_TELEMETRY -#include -#endif - -class JobQueue : private Workers::Callback -{ - // ... existing members ... - - // Telemetry reference for job execution spans (added alongside - // the existing perf::PerfLog& member). - telemetry::Telemetry& telemetry_; - -#ifdef XRPL_ENABLE_TELEMETRY - // Per-job trace context captured at addJob() time and restored - // on the worker thread when the job runs. - struct JobContext - { - opentelemetry::context::Context traceCtx; - }; -#endif - -public: - JobQueue( - int threadCount, - beast::insight::Collector::ptr const& collector, - beast::Journal journal, - Logs& logs, - perf::PerfLog& perfLog, - telemetry::Telemetry& telemetry); // New parameter - // ... -}; -``` - -```cpp -// src/libxrpl/core/detail/JobQueue.cpp (modified — processTask) - -void -JobQueue::processTask(int instance) -{ - // ... existing job dequeue logic ... - -#ifdef XRPL_ENABLE_TELEMETRY - // Restore the trace context that was captured when the job was - // enqueued. Any spans created inside the job body will become - // children of the original caller's trace. - auto token = opentelemetry::context::RuntimeContext::Attach( - job.traceContext()); - - // Start an execution span if telemetry is enabled at runtime - std::optional guard; - if (telemetry_.isEnabled()) - { - guard.emplace(telemetry_.startSpan("job.execute")); - guard->setAttribute("job_type", to_string(job.type())); - guard->setAttribute("job_worker", - static_cast(instance)); - } -#endif - - try - { - job.execute(); -#ifdef XRPL_ENABLE_TELEMETRY - if (guard) - guard->setOk(); -#endif - } - catch (std::exception const& e) - { -#ifdef XRPL_ENABLE_TELEMETRY - if (guard) - guard->recordException(e); -#endif - JLOG(journal_.error()) << "Job execution failed: " << e.what(); - } -} -``` - ---- - -## 4.6 Span Flow Visualization - -
- -```mermaid -flowchart TB - subgraph Client["External Client"] - submit["Submit TX"] - end - - subgraph NodeA["xrpld Node A"] - rpcA["rpc.request"] - cmdA["rpc.command.submit"] - txRecvA["tx.receive"] - txValA["tx.validate"] - txRelayA["tx.relay"] - end - - subgraph NodeB["xrpld Node B"] - txRecvB["tx.receive"] - txValB["tx.validate"] - txRelayB["tx.relay"] - end - - subgraph NodeC["xrpld Node C"] - txRecvC["tx.receive"] - consensusC["consensus.round"] - phaseC["consensus.phase.establish"] - end - - submit --> rpcA - rpcA --> cmdA - cmdA --> txRecvA - txRecvA --> txValA - txValA --> txRelayA - txRelayA -.->|"TraceContext"| txRecvB - txRecvB --> txValB - txValB --> txRelayB - txRelayB -.->|"TraceContext"| txRecvC - txRecvC --> consensusC - consensusC --> phaseC - - style Client fill:#334155,stroke:#1e293b,color:#fff - style NodeA fill:#1e3a8a,stroke:#172554,color:#fff - style NodeB fill:#064e3b,stroke:#022c22,color:#fff - style NodeC fill:#78350f,stroke:#451a03,color:#fff - style submit fill:#e2e8f0,stroke:#cbd5e1,color:#1e293b - style rpcA fill:#1d4ed8,stroke:#1e40af,color:#fff - style cmdA fill:#1d4ed8,stroke:#1e40af,color:#fff - style txRecvA fill:#047857,stroke:#064e3b,color:#fff - style txValA fill:#047857,stroke:#064e3b,color:#fff - style txRelayA fill:#047857,stroke:#064e3b,color:#fff - style txRecvB fill:#047857,stroke:#064e3b,color:#fff - style txValB fill:#047857,stroke:#064e3b,color:#fff - style txRelayB fill:#047857,stroke:#064e3b,color:#fff - style txRecvC fill:#047857,stroke:#064e3b,color:#fff - style consensusC fill:#fef3c7,stroke:#fde68a,color:#1e293b - style phaseC fill:#fef3c7,stroke:#fde68a,color:#1e293b -``` - -
- -**Reading the diagram:** - -- **Client / Submit TX**: An external client submits a transaction, creating the root span that initiates the trace. -- **Node A (RPC layer)**: The receiving node processes the submission through `rpc.request` and `rpc.command.submit`, then hands off to the transaction pipeline (`tx.receive` → `tx.validate` → `tx.relay`). -- **Dashed arrows (TraceContext)**: Cross-node boundaries where trace context is propagated via the protobuf protocol extension, linking spans across independent processes. -- **Node B (relay hop)**: A peer node that receives, validates, and relays the transaction further, demonstrating multi-hop propagation. -- **Node C (consensus)**: The final node where the transaction enters consensus (`consensus.round` → `consensus.phase.establish`), showing how a single client action produces an end-to-end distributed trace. - ---- - -_Previous: [Implementation Strategy](./03-implementation-strategy.md)_ | _Next: [Configuration Reference](./05-configuration-reference.md)_ | _Back to: [Overview](./OpenTelemetryPlan.md)_ diff --git a/OpenTelemetryPlan/05-configuration-reference.md b/OpenTelemetryPlan/05-configuration-reference.md index 9702b1af6c..2eeca72dcb 100644 --- a/OpenTelemetryPlan/05-configuration-reference.md +++ b/OpenTelemetryPlan/05-configuration-reference.md @@ -1,7 +1,7 @@ # Configuration Reference > **Parent Document**: [OpenTelemetryPlan.md](./OpenTelemetryPlan.md) -> **Related**: [Code Samples](./04-code-samples.md) | [Implementation Phases](./06-implementation-phases.md) +> **Related**: [Implementation Phases](./06-implementation-phases.md) --- @@ -11,63 +11,7 @@ ### 5.1.1 Configuration File Section -Add to `cfg/xrpld-example.cfg`: - -```ini -# ═══════════════════════════════════════════════════════════════════════════════ -# TELEMETRY (OpenTelemetry Distributed Tracing) -# ═══════════════════════════════════════════════════════════════════════════════ -# -# Enables distributed tracing for transaction flow, consensus, and RPC calls. -# Traces are exported to an OpenTelemetry Collector using OTLP protocol. -# -# [telemetry] -# -# # Enable/disable telemetry (default: 0 = disabled) -# enabled=1 -# -# # OTLP endpoint (default: http://localhost:4318/v1/traces - OTLP/HTTP) -# # Note: only OTLP/HTTP is shipped in Phase 1b. OTLP/gRPC support is -# # planned as future work and is not yet parsed by TelemetryConfig.cpp. -# endpoint=http://localhost:4318/v1/traces -# -# # Use TLS for exporter connection (default: 0) -# use_tls=0 -# -# # Path to CA certificate for TLS (optional) -# # tls_ca_cert=/path/to/ca.crt -# -# # Head sampling is intentionally fixed at 1.0 (sample everything) and is -# # NOT configurable. A per-node head-sampling ratio would let nodes make -# # divergent keep/drop decisions for the same distributed trace, producing -# # broken/partial traces across the network. Volume reduction is delegated -# # to the collector's tail sampling instead. See Section 7.4.2. -# -# # Batch processor settings -# batch_size=512 # Spans per batch (default: 512) -# batch_delay_ms=5000 # Max delay before sending batch (default: 5000) -# max_queue_size=2048 # Max queued spans (default: 2048) -# -# # Component-specific tracing (default: all enabled except peer) -# trace_transactions=1 # Transaction relay and processing -# trace_consensus=1 # Consensus rounds and proposals -# trace_rpc=1 # RPC request handling -# trace_peer=1 # Peer messages (high volume, enabled by default) -# trace_ledger=1 # Ledger acquisition and building -# -# # Planned (not yet parsed by TelemetryConfig.cpp): -# # trace_pathfind=1 # Path computation (Phase 2) -# # trace_txq=1 # Transaction queue (Phase 3) -# # trace_validator=0 # Validator list / manifest (future) -# # trace_amendment=0 # Amendment voting (future) -# -# # Service identification (automatically detected if not specified) -# # service_name=xrpld -# # service_instance_id= - -[telemetry] -enabled=0 -``` +The authoritative `[telemetry]` example lives in `cfg/xrpld-example.cfg`. Telemetry is disabled by default (`enabled=0`); enabling it turns on distributed tracing for transaction flow, consensus, and RPC calls, with traces exported to an OpenTelemetry Collector over OTLP. Head sampling is intentionally fixed at 1.0 (sample everything) and is not configurable — per-node head-sampling would produce broken/partial distributed traces, so volume reduction is delegated to the collector's tail sampling (see Section 7.4.2). The full option reference follows. ### 5.1.2 Configuration Options Summary @@ -106,67 +50,7 @@ phases. They will be added as the corresponding subsystems are instrumented: > **TxQ** = Transaction Queue -```cpp -// src/libxrpl/telemetry/TelemetryConfig.cpp - -#include -#include - -namespace xrpl { -namespace telemetry { - -Telemetry::Setup -setup_Telemetry( - Section const& section, - std::string const& nodePublicKey, - std::string const& version) -{ - Telemetry::Setup setup; - - // Basic settings - setup.enabled = section.value_or("enabled", false); - setup.serviceName = section.value_or("service_name", "xrpld"); - setup.serviceVersion = version; - setup.serviceInstanceId = section.value_or( - "service_instance_id", nodePublicKey); - - // Exporter settings - setup.exporterType = section.value_or("exporter", "otlp_grpc"); - - if (setup.exporterType == "otlp_grpc") - setup.exporterEndpoint = section.value_or("endpoint", "localhost:4317"); - else if (setup.exporterType == "otlp_http") - setup.exporterEndpoint = section.value_or("endpoint", "localhost:4318"); - - setup.useTls = section.value_or("use_tls", false); - setup.tlsCertPath = section.value_or("tls_ca_cert", ""); - - // Head sampling is fixed at 1.0 (sample everything) and is not read from - // config — see Section 7.4.2. setup.samplingRatio stays at its 1.0 default. - - // Batch processor - setup.batchSize = section.value_or("batch_size", 512u); - setup.batchDelay = std::chrono::milliseconds{ - section.value_or("batch_delay_ms", 5000u)}; - setup.maxQueueSize = section.value_or("max_queue_size", 2048u); - - // Component filtering - setup.traceTransactions = section.value_or("trace_transactions", true); - setup.traceConsensus = section.value_or("trace_consensus", true); - setup.traceRpc = section.value_or("trace_rpc", true); - setup.tracePeer = section.value_or("trace_peer", true); - setup.traceLedger = section.value_or("trace_ledger", true); - setup.tracePathfind = section.value_or("trace_pathfind", true); - setup.traceTxQ = section.value_or("trace_txq", true); - setup.traceValidator = section.value_or("trace_validator", false); - setup.traceAmendment = section.value_or("trace_amendment", false); - - return setup; -} - -} // namespace telemetry -} // namespace xrpl -``` +The parser `setup_Telemetry()` in `src/libxrpl/telemetry/TelemetryConfig.cpp` reads the `[telemetry]` `Section` and populates a `Telemetry::Setup` struct, applying the defaults listed in Section 5.1.2 via `section.value_or(...)`. It derives `serviceInstanceId` from the node public key when not overridden, selects the exporter endpoint default by exporter type, and leaves the sampling ratio at its fixed 1.0 default (not read from config — see Section 7.4.2). --- @@ -180,84 +64,11 @@ setup_Telemetry( > constructed with an empty `serviceInstanceId` and patched via > `setServiceInstanceId()` once `setup()` has called `getNodeIdentity()`. -```cpp -// src/xrpld/app/main/Application.cpp (modified) - -#include - -class ApplicationImp : public Application, public BasicApp -{ - // ... existing members (perfLog_, etc.) ... - - // Telemetry — constructed in the member initializer list with - // an empty serviceInstanceId, patched in setup(). - std::unique_ptr telemetry_; - - // Member initializer list (excerpt): - // ... - // , telemetry_( - // telemetry::make_Telemetry( - // telemetry::setup_Telemetry( - // config_->section("telemetry"), - // "", // Updated later via setServiceInstanceId() - // BuildInfo::getVersionString()), - // logs_->journal("Telemetry"))) - // ... - - bool setup(...) override - { - // ... existing setup code ... - - nodeIdentity_ = getNodeIdentity(*this, cmdline); - - // Inject node identity into telemetry resource attributes, - // unless the user already set a custom service_instance_id. - if (!config_->section("telemetry").exists("service_instance_id")) - telemetry_->setServiceInstanceId( - toBase58(TokenType::NodePublic, nodeIdentity_->first)); - - // ... rest of setup ... - } - - void start(bool withTimers) override - { - // ... existing start code ... - telemetry_->start(); - } - - void run() override - { - // ... existing run/shutdown code ... - telemetry_->stop(); - } - - telemetry::Telemetry& - getTelemetry() override - { - return *telemetry_; - } -}; -``` +`ApplicationImp` (in `src/xrpld/app/main/Application.cpp`) owns a `std::unique_ptr telemetry_`. It is built in the member initializer list via `make_Telemetry(setup_Telemetry(...))` with an empty `serviceInstanceId`, then patched in `setup()` by calling `setServiceInstanceId()` with the Base58 node public key (unless the user supplied a custom `service_instance_id`). `start()` and `run()` forward to `telemetry_->start()` / `telemetry_->stop()`, and `getTelemetry()` returns the owned instance. ### 5.3.2 ServiceRegistry Interface Addition -```cpp -// include/xrpl/core/ServiceRegistry.h (modified) - -namespace telemetry { -class Telemetry; -} // namespace telemetry - -class ServiceRegistry -{ -public: - // ... existing virtual methods ... - - /** Get the telemetry system for distributed tracing. */ - virtual telemetry::Telemetry& - getTelemetry() = 0; -}; -``` +`include/xrpl/core/ServiceRegistry.h` gains a pure-virtual `telemetry::Telemetry& getTelemetry()` (with a forward declaration of `telemetry::Telemetry`), giving every component a uniform accessor for the tracing subsystem. > **Note:** `Application` extends `ServiceRegistry`, so `getTelemetry()` is > available on both. Components that hold a `ServiceRegistry&` (e.g. @@ -273,114 +84,11 @@ public: ### 5.4.1 Find OpenTelemetry Module -```cmake -# cmake/FindOpenTelemetry.cmake - -# Find OpenTelemetry C++ SDK -# -# This module defines: -# OpenTelemetry_FOUND - System has OpenTelemetry -# OpenTelemetry::api - API library target -# OpenTelemetry::sdk - SDK library target -# OpenTelemetry::otlp_grpc_exporter - OTLP gRPC exporter target -# OpenTelemetry::otlp_http_exporter - OTLP HTTP exporter target - -find_package(opentelemetry-cpp CONFIG QUIET) - -if(opentelemetry-cpp_FOUND) - set(OpenTelemetry_FOUND TRUE) - - # Create imported targets if not already created by config - if(NOT TARGET OpenTelemetry::api) - add_library(OpenTelemetry::api ALIAS opentelemetry-cpp::api) - endif() - if(NOT TARGET OpenTelemetry::sdk) - add_library(OpenTelemetry::sdk ALIAS opentelemetry-cpp::sdk) - endif() - if(NOT TARGET OpenTelemetry::otlp_grpc_exporter) - add_library(OpenTelemetry::otlp_grpc_exporter ALIAS - opentelemetry-cpp::otlp_grpc_exporter) - endif() -else() - # Try pkg-config fallback - find_package(PkgConfig QUIET) - if(PKG_CONFIG_FOUND) - pkg_check_modules(OTEL opentelemetry-cpp QUIET) - if(OTEL_FOUND) - set(OpenTelemetry_FOUND TRUE) - # Create imported targets from pkg-config - add_library(OpenTelemetry::api INTERFACE IMPORTED) - target_include_directories(OpenTelemetry::api INTERFACE - ${OTEL_INCLUDE_DIRS}) - endif() - endif() -endif() - -include(FindPackageHandleStandardArgs) -find_package_handle_standard_args(OpenTelemetry - REQUIRED_VARS OpenTelemetry_FOUND) -``` +A `cmake/FindOpenTelemetry.cmake` module locates the OpenTelemetry C++ SDK. It first tries `find_package(opentelemetry-cpp CONFIG)`, aliasing the imported targets `OpenTelemetry::api`, `OpenTelemetry::sdk`, and `OpenTelemetry::otlp_grpc_exporter`, and falls back to `pkg-config` when no CMake config package is present. ### 5.4.2 CMakeLists.txt Changes -```cmake -# CMakeLists.txt (additions) - -# ═══════════════════════════════════════════════════════════════════════════════ -# TELEMETRY OPTIONS -# ═══════════════════════════════════════════════════════════════════════════════ - -option(XRPL_ENABLE_TELEMETRY - "Enable OpenTelemetry distributed tracing support" OFF) - -if(XRPL_ENABLE_TELEMETRY) - find_package(OpenTelemetry REQUIRED) - - # Define compile-time flag - add_compile_definitions(XRPL_ENABLE_TELEMETRY) - - message(STATUS "OpenTelemetry tracing: ENABLED") -else() - message(STATUS "OpenTelemetry tracing: DISABLED") -endif() - -# ═══════════════════════════════════════════════════════════════════════════════ -# TELEMETRY LIBRARY -# ═══════════════════════════════════════════════════════════════════════════════ - -if(XRPL_ENABLE_TELEMETRY) - add_library(xrpl_telemetry - src/libxrpl/telemetry/Telemetry.cpp - src/libxrpl/telemetry/TelemetryConfig.cpp - src/libxrpl/telemetry/TraceContext.cpp - ) - - target_include_directories(xrpl_telemetry - PUBLIC - ${CMAKE_CURRENT_SOURCE_DIR}/include - ) - - target_link_libraries(xrpl_telemetry - PUBLIC - OpenTelemetry::api - OpenTelemetry::sdk - OpenTelemetry::otlp_grpc_exporter - PRIVATE - xrpl_basics - ) - - # Add to main library dependencies - target_link_libraries(xrpld PRIVATE xrpl_telemetry) -else() - # Create null implementation library - add_library(xrpl_telemetry - src/libxrpl/telemetry/NullTelemetry.cpp - ) - target_include_directories(xrpl_telemetry - PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include - ) -endif() -``` +The top-level `CMakeLists.txt` adds an `XRPL_ENABLE_TELEMETRY` option (default `OFF`). When enabled, it runs `find_package(OpenTelemetry REQUIRED)`, defines the `XRPL_ENABLE_TELEMETRY` compile flag, and builds the `xrpl_telemetry` library from the real telemetry sources linked against the OpenTelemetry targets; when disabled, it builds the same target from a no-op `NullTelemetry.cpp` so call sites compile unchanged. --- @@ -388,151 +96,15 @@ endif() > **OTLP** = OpenTelemetry Protocol | **APM** = Application Performance Monitoring +The authoritative collector config lives in the repo at `docker/telemetry/otel-collector-config.yaml` (with Tempo backend config in `docker/telemetry/tempo.yaml`). The sections below summarize the development and production shapes of that pipeline. + ### 5.5.1 Development Configuration -```yaml -# otel-collector-dev.yaml -# Minimal configuration for local development - -receivers: - otlp: - protocols: - grpc: - endpoint: 0.0.0.0:4317 - http: - endpoint: 0.0.0.0:4318 - -processors: - batch: - timeout: 1s - send_batch_size: 100 - -exporters: - # Console output for debugging - logging: - verbosity: detailed - sampling_initial: 5 - sampling_thereafter: 200 - - # Tempo for trace visualization - otlp/tempo: - endpoint: tempo:4317 - tls: - insecure: true - -service: - pipelines: - traces: - receivers: [otlp] - processors: [batch] - exporters: [logging, otlp/tempo] -``` +The development collector enables an OTLP receiver on both gRPC (`0.0.0.0:4317`) and HTTP (`0.0.0.0:4318`), a single `batch` processor (1s timeout, batch size 100), and two exporters: a `logging` exporter for console debugging and `otlp/tempo` (insecure) for trace visualization. The single `traces` pipeline wires receiver → batch → both exporters. ### 5.5.2 Production Configuration -```yaml -# otel-collector-prod.yaml -# Production configuration with filtering, sampling, and multiple backends - -receivers: - otlp: - protocols: - grpc: - endpoint: 0.0.0.0:4317 - tls: - cert_file: /etc/otel/server.crt - key_file: /etc/otel/server.key - ca_file: /etc/otel/ca.crt - -processors: - # Memory limiter to prevent OOM - memory_limiter: - check_interval: 1s - limit_mib: 1000 - spike_limit_mib: 200 - - # Batch processing for efficiency - batch: - timeout: 5s - send_batch_size: 512 - send_batch_max_size: 1024 - - # Tail-based sampling (keep errors and slow traces) - tail_sampling: - decision_wait: 10s - num_traces: 100000 - expected_new_traces_per_sec: 1000 - policies: - # Always keep error traces - - name: errors - type: status_code - status_code: - status_codes: [ERROR] - # Keep slow consensus rounds (>5s) - - name: slow-consensus - type: latency - latency: - threshold_ms: 5000 - # Keep slow RPC requests (>1s) - - name: slow-rpc - type: and - and: - and_sub_policy: - - name: rpc-spans - type: string_attribute - string_attribute: - key: command - values: [".*"] - enabled_regex_matching: true - - name: latency - type: latency - latency: - threshold_ms: 1000 - # Probabilistic sampling for the rest - - name: probabilistic - type: probabilistic - probabilistic: - sampling_percentage: 10 - - # Attribute processing - attributes: - actions: - # Hash sensitive data - - key: tx_account - action: hash - # Add deployment info - - key: deployment.environment - value: production - action: upsert - -exporters: - # Grafana Tempo for long-term storage - otlp/tempo: - endpoint: tempo.monitoring:4317 - tls: - insecure: false - ca_file: /etc/otel/tempo-ca.crt - - # Elastic APM for correlation with logs - otlp/elastic: - endpoint: apm.elastic:8200 - headers: - Authorization: "Bearer ${ELASTIC_APM_TOKEN}" - -extensions: - health_check: - endpoint: 0.0.0.0:13133 - zpages: - endpoint: 0.0.0.0:55679 - -service: - extensions: [health_check, zpages] - pipelines: - traces: - receivers: [otlp] - processors: [memory_limiter, tail_sampling, attributes, batch] - exporters: [otlp/tempo, otlp/elastic] -``` +The production collector adds TLS on the OTLP gRPC receiver and a richer processor chain: a `memory_limiter` (OOM guard), `batch` (5s timeout, size 512), `tail_sampling`, and an `attributes` processor that hashes sensitive fields (e.g. `tx_account`) and stamps `deployment.environment`. Tail sampling keeps all `ERROR` traces, slow consensus rounds (>5s) and slow RPC requests (>1s), and probabilistically samples the remainder at 10%. Exporters target Grafana Tempo (TLS) and Elastic APM; `health_check` and `zpages` extensions are enabled for operability. --- @@ -540,61 +112,7 @@ service: > **OTLP** = OpenTelemetry Protocol -```yaml -# docker-compose-telemetry.yaml -version: "3.8" - -services: - # OpenTelemetry Collector - otel-collector: - image: otel/opentelemetry-collector-contrib:0.92.0 - container_name: otel-collector - command: ["--config=/etc/otel-collector-config.yaml"] - volumes: - - ./otel-collector-dev.yaml:/etc/otel-collector-config.yaml:ro - ports: - - "4317:4317" # OTLP gRPC - - "4318:4318" # OTLP HTTP - - "13133:13133" # Health check - depends_on: - - tempo - - # Tempo for trace visualization - tempo: - image: grafana/tempo:2.6.1 - container_name: tempo - ports: - - "3200:3200" # Tempo HTTP API - - "4317" # OTLP gRPC (internal) - - # Grafana for dashboards - grafana: - image: grafana/grafana:10.2.3 - container_name: grafana - environment: - - GF_AUTH_ANONYMOUS_ENABLED=true - - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin - volumes: - - ./grafana/provisioning:/etc/grafana/provisioning:ro - - ./grafana/dashboards:/var/lib/grafana/dashboards:ro - ports: - - "3000:3000" - depends_on: - - tempo - - # Prometheus for metrics (optional, for correlation) - prometheus: - image: prom/prometheus:v2.48.1 - container_name: prometheus - volumes: - - ./prometheus.yaml:/etc/prometheus/prometheus.yml:ro - ports: - - "9090:9090" - -networks: - default: - name: xrpld-telemetry -``` +The authoritative development stack lives in the repo at `docker/telemetry/docker-compose.yml`. It brings up four services on a shared `xrpld-telemetry` network: an `otel-collector` (otel/opentelemetry-collector-contrib) exposing OTLP gRPC `4317`, OTLP HTTP `4318`, and health check `13133`; `tempo` for trace storage/visualization; `grafana` with provisioned datasources and dashboards (anonymous admin enabled); and an optional `prometheus` for metric correlation. --- @@ -661,175 +179,23 @@ Step-by-step instructions for integrating xrpld traces with Grafana. #### Tempo (Recommended) -```yaml -# grafana/provisioning/datasources/tempo.yaml -apiVersion: 1 - -datasources: - - name: Tempo - type: tempo - access: proxy - url: http://tempo:3200 - jsonData: - httpMethod: GET - tracesToLogs: - datasourceUid: loki - tags: ["service.name", "tx_hash"] - mappedTags: [{ key: "trace_id", value: "traceID" }] - mapTagNamesEnabled: true - filterByTraceID: true - serviceMap: - datasourceUid: prometheus - nodeGraph: - enabled: true - search: - hide: false - lokiSearch: - datasourceUid: loki -``` +A Tempo datasource (`grafana/provisioning/datasources/tempo.yaml`, provisioned from `docker/telemetry/grafana/`) points at `http://tempo:3200` and enables `tracesToLogs` (linking to Loki on `service.name`/`tx_hash` and mapping `trace_id` → `traceID`), `serviceMap` against Prometheus, the node graph, and Loki search. #### Elastic APM -```yaml -# grafana/provisioning/datasources/elastic-apm.yaml -apiVersion: 1 - -datasources: - - name: Elasticsearch-APM - type: elasticsearch - access: proxy - url: http://elasticsearch:9200 - database: "apm-*" - jsonData: - esVersion: "8.0.0" - timeField: "@timestamp" - logMessageField: message - logLevelField: log.level -``` +Alternatively, an Elasticsearch datasource (`grafana/provisioning/datasources/elastic-apm.yaml`) of type `elasticsearch` points at `http://elasticsearch:9200` against the `apm-*` index, using `@timestamp` as the time field and mapping the log message/level fields. ### 5.8.2 Dashboard Provisioning -```yaml -# grafana/provisioning/dashboards/dashboards.yaml -apiVersion: 1 - -providers: - - name: "xrpld-dashboards" - orgId: 1 - folder: "xrpld" - folderUid: "xrpld" - type: file - disableDeletion: false - updateIntervalSeconds: 30 - options: - path: /var/lib/grafana/dashboards/rippled -``` +A dashboard provider (`grafana/provisioning/dashboards/dashboards.yaml`) loads the `xrpld` dashboard folder from disk (`/var/lib/grafana/dashboards/rippled`), polling for changes every 30s with deletion disabled. ### 5.8.3 Example Dashboard: RPC Performance -```json -{ - "title": "xrpld RPC Performance", - "uid": "xrpld-rpc-performance", - "panels": [ - { - "title": "RPC Latency by Command", - "type": "heatmap", - "datasource": "Tempo", - "targets": [ - { - "queryType": "traceql", - "query": "{resource.service.name=\"xrpld\" && span.command != \"\"} | histogram_over_time(duration) by (span.command)" - } - ], - "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 } - }, - { - "title": "RPC Error Rate", - "type": "timeseries", - "datasource": "Tempo", - "targets": [ - { - "queryType": "traceql", - "query": "{resource.service.name=\"xrpld\" && status.code=error} | rate() by (span.command)" - } - ], - "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 } - }, - { - "title": "Top 10 Slowest RPC Commands", - "type": "table", - "datasource": "Tempo", - "targets": [ - { - "queryType": "traceql", - "query": "{resource.service.name=\"xrpld\" && span.command != \"\"} | avg(duration) by (span.command) | topk(10)" - } - ], - "gridPos": { "h": 8, "w": 24, "x": 0, "y": 8 } - }, - { - "title": "Recent Traces", - "type": "table", - "datasource": "Tempo", - "targets": [ - { - "queryType": "traceql", - "query": "{resource.service.name=\"xrpld\"}" - } - ], - "gridPos": { "h": 8, "w": 24, "x": 0, "y": 16 } - } - ] -} -``` +An example `xrpld RPC Performance` dashboard (uid `xrpld-rpc-performance`) sourced from Tempo via TraceQL provides four panels: RPC latency by command (heatmap), RPC error rate by command (timeseries), the top 10 slowest RPC commands by average duration (table), and a recent-traces table. ### 5.8.4 Example Dashboard: Transaction Tracing -```json -{ - "title": "xrpld Transaction Tracing", - "uid": "xrpld-tx-tracing", - "panels": [ - { - "title": "Transaction Throughput", - "type": "stat", - "datasource": "Tempo", - "targets": [ - { - "queryType": "traceql", - "query": "{resource.service.name=\"xrpld\" && name=\"tx.receive\"} | rate()" - } - ], - "gridPos": { "h": 4, "w": 6, "x": 0, "y": 0 } - }, - { - "title": "Cross-Node Relay Count", - "type": "timeseries", - "datasource": "Tempo", - "targets": [ - { - "queryType": "traceql", - "query": "{resource.service.name=\"xrpld\" && name=\"tx.relay\"} | avg(span.relay_count)" - } - ], - "gridPos": { "h": 8, "w": 12, "x": 0, "y": 4 } - }, - { - "title": "Transaction Validation Errors", - "type": "table", - "datasource": "Tempo", - "targets": [ - { - "queryType": "traceql", - "query": "{resource.service.name=\"xrpld\" && name=\"tx.validate\" && status.code=error}" - } - ], - "gridPos": { "h": 8, "w": 12, "x": 12, "y": 4 } - } - ] -} -``` +An example `xrpld Transaction Tracing` dashboard (uid `xrpld-tx-tracing`) over Tempo provides three panels: transaction throughput (`tx.receive` rate, stat), cross-node relay count (average `span.relay_count` on `tx.relay`, timeseries), and a table of transaction validation errors (`tx.validate` with `status.code=error`). ### 5.8.5 TraceQL Query Examples @@ -861,58 +227,15 @@ To correlate OpenTelemetry traces with existing PerfLog data: **Step 1: Configure Loki to ingest PerfLog** -```yaml -# promtail-config.yaml -scrape_configs: - - job_name: xrpld-perflog - static_configs: - - targets: - - localhost - labels: - job: xrpld - __path__: /var/log/rippled/perf*.log - pipeline_stages: - - json: - expressions: - trace_id: trace_id - ledger_seq: ledger_seq - tx_hash: tx_hash - - labels: - trace_id: - ledger_seq: - tx_hash: -``` +Configure a Promtail scrape job (`promtail-config.yaml`) that tails `/var/log/rippled/perf*.log`, parses each JSON line, and promotes `trace_id`, `ledger_seq`, and `tx_hash` to Loki labels. **Step 2: Add trace_id to PerfLog entries** -Modify PerfLog to include trace_id when available: - -```cpp -// In PerfLog output, add trace_id from current span context -void logPerf(Json::Value& entry) { - auto span = opentelemetry::trace::GetSpan( - opentelemetry::context::RuntimeContext::GetCurrent()); - if (span && span->GetContext().IsValid()) { - char traceIdHex[33]; - span->GetContext().trace_id().ToLowerBase16(traceIdHex); - entry["trace_id"] = std::string(traceIdHex, 32); - } - // ... existing logging -} -``` +Modify PerfLog so its JSON output includes a `trace_id` field whenever a valid span is active: fetch the current span from the OpenTelemetry runtime context, and if its context is valid, render the trace ID as a 32-character lowercase hex string into the log entry. **Step 3: Configure Grafana trace-to-logs link** -In Tempo data source configuration, set up the derived field: - -```yaml -jsonData: - tracesToLogs: - datasourceUid: loki - tags: ["trace_id", "tx_hash"] - filterByTraceID: true - filterBySpanID: false -``` +In the Tempo datasource, set the `tracesToLogs` derived field to link to Loki on the `trace_id` and `tx_hash` tags, with `filterByTraceID: true`. ### 5.8.7 Correlation with Insight/StatsD Metrics @@ -920,46 +243,22 @@ To correlate traces with existing Beast Insight metrics: **Step 1: Export Insight metrics to Prometheus** -```yaml -# prometheus.yaml -scrape_configs: - - job_name: "xrpld-statsd" - static_configs: - - targets: ["statsd-exporter:9102"] -``` +Add a Prometheus scrape job (`prometheus.yaml`) named `xrpld-statsd` targeting the StatsD exporter at `statsd-exporter:9102`. **Step 2: Add exemplars to metrics** -OpenTelemetry SDK automatically adds exemplars (trace IDs) to metrics when using the Prometheus exporter. This links metrics spikes to specific traces. +The OpenTelemetry SDK automatically adds exemplars (trace IDs) to metrics when using the Prometheus exporter, linking metric spikes to specific traces. **Step 3: Configure Grafana metric-to-trace link** -```yaml -# In Prometheus data source -jsonData: - exemplarTraceIdDestinations: - - name: trace_id - datasourceUid: tempo -``` +In the Prometheus datasource, set `exemplarTraceIdDestinations` to map the `trace_id` exemplar to the Tempo datasource. **Step 4: Dashboard panel with exemplars** -```json -{ - "title": "RPC Latency with Trace Links", - "type": "timeseries", - "datasource": "Prometheus", - "targets": [ - { - "expr": "histogram_quantile(0.99, rate(xrpld_rpc_duration_seconds_bucket[5m]))", - "exemplar": true - } - ] -} -``` +Add a timeseries panel over Prometheus (e.g. `histogram_quantile(0.99, rate(xrpld_rpc_duration_seconds_bucket[5m]))`) with `exemplar: true` enabled. This allows clicking on metric data points to jump directly to the related trace. --- -_Previous: [Code Samples](./04-code-samples.md)_ | _Next: [Implementation Phases](./06-implementation-phases.md)_ | _Back to: [Overview](./OpenTelemetryPlan.md)_ +_Previous: [Implementation Strategy](./03-implementation-strategy.md)_ | _Next: [Implementation Phases](./06-implementation-phases.md)_ | _Back to: [Overview](./OpenTelemetryPlan.md)_ diff --git a/OpenTelemetryPlan/07-observability-backends.md b/OpenTelemetryPlan/07-observability-backends.md index b8e35d91f5..a8bd897723 100644 --- a/OpenTelemetryPlan/07-observability-backends.md +++ b/OpenTelemetryPlan/07-observability-backends.md @@ -230,205 +230,47 @@ Pre-built dashboards for xrpld observability. ### 7.6.1 Consensus Health Dashboard -```json -{ - "title": "xrpld Consensus Health", - "uid": "xrpld-consensus-health", - "tags": ["xrpld", "consensus", "tracing"], - "panels": [ - { - "title": "Consensus Round Duration", - "type": "timeseries", - "datasource": "Tempo", - "targets": [ - { - "queryType": "traceql", - "query": "{resource.service.name=\"xrpld\" && name=\"consensus.round\"} | avg(duration) by (resource.service.instance.id)" - } - ], - "fieldConfig": { - "defaults": { - "unit": "ms", - "thresholds": { - "steps": [ - { "color": "green", "value": null }, - { "color": "yellow", "value": 4000 }, - { "color": "red", "value": 5000 } - ] - } - } - }, - "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 } - }, - { - "title": "Phase Duration Breakdown", - "type": "barchart", - "datasource": "Tempo", - "targets": [ - { - "queryType": "traceql", - "query": "{resource.service.name=\"xrpld\" && name=~\"consensus.phase.*\"} | avg(duration) by (name)" - } - ], - "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 } - }, - { - "title": "Proposers per Round", - "type": "stat", - "datasource": "Tempo", - "targets": [ - { - "queryType": "traceql", - "query": "{resource.service.name=\"xrpld\" && name=\"consensus.round\"} | avg(span.proposers)" - } - ], - "gridPos": { "h": 4, "w": 6, "x": 0, "y": 8 } - }, - { - "title": "Recent Slow Rounds (>5s)", - "type": "table", - "datasource": "Tempo", - "targets": [ - { - "queryType": "traceql", - "query": "{resource.service.name=\"xrpld\" && name=\"consensus.round\"} | duration > 5s" - } - ], - "gridPos": { "h": 8, "w": 24, "x": 0, "y": 12 } - } - ] -} -``` +A Tempo-backed dashboard (uid `xrpld-consensus-health`) with four panels, all driven by TraceQL: + +- **Consensus Round Duration** (timeseries, ms): average `consensus.round` span duration per node instance, with yellow/red thresholds at 4s/5s. +- **Phase Duration Breakdown** (barchart): average duration of `consensus.phase.*` spans grouped by span name. +- **Proposers per Round** (stat): average of the `span.proposers` attribute on `consensus.round` spans. +- **Recent Slow Rounds (>5s)** (table): `consensus.round` spans filtered to `duration > 5s`. + +The underlying TraceQL queries are listed in section 7.7.3 and used throughout this doc. ### 7.6.2 Node Overview Dashboard -```json -{ - "title": "xrpld Node Overview", - "uid": "xrpld-node-overview", - "panels": [ - { - "title": "Active Nodes", - "type": "stat", - "datasource": "Tempo", - "targets": [ - { - "queryType": "traceql", - "query": "{resource.service.name=\"xrpld\"} | count_over_time() by (resource.service.instance.id) | count()" - } - ], - "gridPos": { "h": 4, "w": 4, "x": 0, "y": 0 } - }, - { - "title": "Total Transactions (1h)", - "type": "stat", - "datasource": "Tempo", - "targets": [ - { - "queryType": "traceql", - "query": "{resource.service.name=\"xrpld\" && name=\"tx.receive\"} | count()" - } - ], - "gridPos": { "h": 4, "w": 4, "x": 4, "y": 0 } - }, - { - "title": "Error Rate", - "type": "gauge", - "datasource": "Tempo", - "targets": [ - { - "queryType": "traceql", - "query": "{resource.service.name=\"xrpld\" && status.code=error} | rate() / {resource.service.name=\"xrpld\"} | rate() * 100" - } - ], - "fieldConfig": { - "defaults": { - "unit": "percent", - "max": 10, - "thresholds": { - "steps": [ - { "color": "green", "value": null }, - { "color": "yellow", "value": 1 }, - { "color": "red", "value": 5 } - ] - } - } - }, - "gridPos": { "h": 4, "w": 4, "x": 8, "y": 0 } - }, - { - "title": "Service Map", - "type": "nodeGraph", - "datasource": "Tempo", - "gridPos": { "h": 12, "w": 12, "x": 12, "y": 0 } - } - ] -} -``` +A Tempo-backed dashboard (uid `xrpld-node-overview`) with four panels: + +- **Active Nodes** (stat): count of distinct `resource.service.instance.id` values seen for the `xrpld` service. +- **Total Transactions (1h)** (stat): count of `tx.receive` spans. +- **Error Rate** (gauge, percent): ratio of `status.code=error` spans to all spans, with yellow/red thresholds at 1%/5%. +- **Service Map** (nodeGraph): Tempo-generated service dependency graph. ### 7.6.3 Alert Rules -```yaml -# grafana/provisioning/alerting/rippled-alerts.yaml -apiVersion: 1 +Grafana provisions three TraceQL-based alert rules (group `xrpld-tracing-alerts`, evaluated every 1m) against the Tempo datasource: -groups: - - name: xrpld-tracing-alerts - folder: xrpld - interval: 1m - rules: - - uid: consensus-slow - title: Consensus Round Slow - condition: A - data: - - refId: A - datasourceUid: tempo - model: - queryType: traceql - query: '{resource.service.name="xrpld" && name="consensus.round"} | avg(duration) > 5s' - # Note: Verify TraceQL aggregate queries are supported by your - # Tempo version. Aggregate alerting (e.g., avg(duration)) requires - # Tempo 2.3+ with TraceQL metrics enabled. - for: 5m - annotations: - summary: Consensus rounds taking >5 seconds - description: "Consensus duration: {{ $value }}ms" - labels: - severity: warning +- **Consensus Round Slow** (warning, `for: 5m`): fires when average `consensus.round` duration exceeds 5s. - - uid: rpc-error-spike - title: RPC Error Rate Spike - condition: B - data: - - refId: B - datasourceUid: tempo - model: - queryType: traceql - query: '{resource.service.name="xrpld" && name=~"rpc.command.*" && status.code=error} | rate() > 0.05' - # Note: Verify TraceQL aggregate queries are supported by your - # Tempo version. Aggregate alerting (e.g., rate()) requires - # Tempo 2.3+ with TraceQL metrics enabled. - for: 2m - annotations: - summary: RPC error rate >5% - labels: - severity: critical + ``` + {resource.service.name="xrpld" && name="consensus.round"} | avg(duration) > 5s + ``` - - uid: tx-throughput-drop - title: Transaction Throughput Drop - condition: C - data: - - refId: C - datasourceUid: tempo - model: - queryType: traceql - query: '{resource.service.name="xrpld" && name="tx.receive"} | rate() < 10' - for: 10m - annotations: - summary: Transaction throughput below threshold - labels: - severity: warning -``` +- **RPC Error Rate Spike** (critical, `for: 2m`): fires when the error rate across `rpc.command.*` spans exceeds 5%. + + ``` + {resource.service.name="xrpld" && name=~"rpc.command.*" && status.code=error} | rate() > 0.05 + ``` + +- **Transaction Throughput Drop** (warning, `for: 10m`): fires when the `tx.receive` span rate falls below 10/s. + + ``` + {resource.service.name="xrpld" && name="tx.receive"} | rate() < 10 + ``` + +> **Note**: The first two rules use TraceQL aggregates (`avg(duration)`, `rate()`), which require Tempo 2.3+ with TraceQL metrics enabled. Verify aggregate query support in your Tempo version before provisioning. --- @@ -548,93 +390,14 @@ rate(xrpld_tx_applied_total[1m]) ### 7.7.4 Unified Dashboard Example -```json -{ - "title": "xrpld Unified Observability", - "uid": "xrpld-unified", - "panels": [ - { - "title": "Transaction Latency (Traces)", - "type": "timeseries", - "datasource": "Tempo", - "targets": [ - { - "queryType": "traceql", - "query": "{resource.service.name=\"xrpld\" && name=\"tx.receive\"} | histogram_over_time(duration)" - } - ], - "gridPos": { "h": 6, "w": 8, "x": 0, "y": 0 } - }, - { - "title": "Transaction Rate (Metrics)", - "type": "timeseries", - "datasource": "Prometheus", - "targets": [ - { - "expr": "rate(xrpld_tx_received_total[5m])", - "legendFormat": "{{ instance }}" - } - ], - "fieldConfig": { - "defaults": { - "links": [ - { - "title": "View traces", - "url": "/explore?left={\"datasource\":\"Tempo\",\"query\":\"{resource.service.name=\\\"xrpld\\\" && name=\\\"tx.receive\\\"}\"}" - } - ] - } - }, - "gridPos": { "h": 6, "w": 8, "x": 8, "y": 0 } - }, - { - "title": "Recent Logs", - "type": "logs", - "datasource": "Loki", - "targets": [ - { - "expr": "{job=\"xrpld\"} | json" - } - ], - "gridPos": { "h": 6, "w": 8, "x": 16, "y": 0 } - }, - { - "title": "Trace Search", - "type": "table", - "datasource": "Tempo", - "targets": [ - { - "queryType": "traceql", - "query": "{resource.service.name=\"xrpld\"}" - } - ], - "fieldConfig": { - "overrides": [ - { - "matcher": { "id": "byName", "options": "traceID" }, - "properties": [ - { - "id": "links", - "value": [ - { - "title": "View trace", - "url": "/explore?left={\"datasource\":\"Tempo\",\"query\":\"${__value.raw}\"}" - }, - { - "title": "View logs", - "url": "/explore?left={\"datasource\":\"Loki\",\"query\":\"{job=\\\"xrpld\\\"} |= \\\"${__value.raw}\\\"\"}" - } - ] - } - ] - } - ] - }, - "gridPos": { "h": 12, "w": 24, "x": 0, "y": 6 } - } - ] -} -``` +A single dashboard (uid `xrpld-unified`) that ties traces, metrics, and logs together across the Tempo, Prometheus, and Loki datasources: + +- **Transaction Latency (Traces)** (timeseries, Tempo): `histogram_over_time(duration)` of `tx.receive` spans. +- **Transaction Rate (Metrics)** (timeseries, Prometheus): `rate(xrpld_tx_received_total[5m])` per instance, with a data link that opens the matching `tx.receive` traces in Tempo. +- **Recent Logs** (logs, Loki): `{job="xrpld"} | json`. +- **Trace Search** (table, Tempo): all `xrpld` traces, with per-row data links on `traceID` that jump to the trace in Tempo and to the correlated logs in Loki (`{job="xrpld"} |= ""`). + +The cross-datasource data links are what make this a single-pane debugging view; the correlation fields they rely on are listed in section 7.7.2. --- diff --git a/OpenTelemetryPlan/08-appendix.md b/OpenTelemetryPlan/08-appendix.md index 33ec8d3e02..ddff360695 100644 --- a/OpenTelemetryPlan/08-appendix.md +++ b/OpenTelemetryPlan/08-appendix.md @@ -177,7 +177,6 @@ flowchart TB | [01-architecture-analysis.md](./01-architecture-analysis.md) | xrpld architecture and trace points | | [02-design-decisions.md](./02-design-decisions.md) | SDK selection, exporters, span conventions | | [03-implementation-strategy.md](./03-implementation-strategy.md) | Directory structure, performance analysis | -| [04-code-samples.md](./04-code-samples.md) | C++ code examples for all components | | [05-configuration-reference.md](./05-configuration-reference.md) | xrpld config, CMake, Collector configs | | [06-implementation-phases.md](./06-implementation-phases.md) | Timeline, tasks, risks, success metrics | | [07-observability-backends.md](./07-observability-backends.md) | Backend selection and architecture | @@ -187,7 +186,6 @@ flowchart TB | Document | Description | | ------------------------------------ | --------------------------------------------------- | -| [POC_taskList.md](./POC_taskList.md) | Proof-of-concept telemetry integration | | [presentation.md](./presentation.md) | Presentation slides for OpenTelemetry plan overview | --- diff --git a/OpenTelemetryPlan/OpenTelemetryPlan.md b/OpenTelemetryPlan/OpenTelemetryPlan.md index 3974d79481..2836070485 100644 --- a/OpenTelemetryPlan/OpenTelemetryPlan.md +++ b/OpenTelemetryPlan/OpenTelemetryPlan.md @@ -46,7 +46,6 @@ flowchart TB subgraph impl["Implementation"] strategy["03-implementation-strategy.md"] - code["04-code-samples.md"] config["05-configuration-reference.md"] end @@ -54,7 +53,6 @@ flowchart TB phases["06-implementation-phases.md"] backends["07-observability-backends.md"] appendix["08-appendix.md"] - poc["POC_taskList.md"] end overview --> fundamentals @@ -65,12 +63,10 @@ flowchart TB fund --> arch arch --> design design --> strategy - strategy --> code - code --> config + strategy --> config config --> phases phases --> backends backends --> appendix - phases --> poc style overview fill:#1b5e20,stroke:#0d3d14,color:#fff,stroke-width:2px style fundamentals fill:#00695c,stroke:#004d40,color:#fff @@ -81,12 +77,10 @@ flowchart TB style arch fill:#0d47a1,stroke:#082f6a,color:#fff style design fill:#0d47a1,stroke:#082f6a,color:#fff style strategy fill:#bf360c,stroke:#8c2809,color:#fff - style code fill:#bf360c,stroke:#8c2809,color:#fff style config fill:#bf360c,stroke:#8c2809,color:#fff style phases fill:#4a148c,stroke:#2e0d57,color:#fff style backends fill:#4a148c,stroke:#2e0d57,color:#fff style appendix fill:#4a148c,stroke:#2e0d57,color:#fff - style poc fill:#4a148c,stroke:#2e0d57,color:#fff ``` @@ -101,12 +95,10 @@ flowchart TB | **1** | [Architecture Analysis](./01-architecture-analysis.md) | xrpld component analysis, trace points, instrumentation priorities | | **2** | [Design Decisions](./02-design-decisions.md) | SDK selection, exporters, span naming, attributes, context propagation | | **3** | [Implementation Strategy](./03-implementation-strategy.md) | Directory structure, key principles, performance optimization | -| **4** | [Code Samples](./04-code-samples.md) | C++ implementation examples for core infrastructure and key modules | | **5** | [Configuration Reference](./05-configuration-reference.md) | xrpld config, CMake integration, Collector configurations | | **6** | [Implementation Phases](./06-implementation-phases.md) | 5-phase timeline, tasks, risks, success metrics | | **7** | [Observability Backends](./07-observability-backends.md) | Backend selection guide and production architecture | | **8** | [Appendix](./08-appendix.md) | Glossary, references, version history | -| **POC** | [POC Task List](./POC_taskList.md) | Proof of concept tasks for RPC tracing end-to-end demo | --- @@ -154,21 +146,6 @@ Performance optimization strategies include head sampling fixed at 100% (intenti --- -## 4. Code Samples - -C++ implementation examples are provided for the core telemetry infrastructure and key modules: - -- `Telemetry.h` - Core interface for tracer access and span creation -- `SpanGuard.h` - RAII wrapper for automatic span lifecycle management -- `TracingInstrumentation.h` - Macros for conditional instrumentation -- Protocol Buffer extensions for trace context propagation -- Module-specific instrumentation (RPC, Consensus, P2P, JobQueue) -- Remaining modules (PathFinding, TxQ, Validator, etc.) follow the same patterns - -➡️ **[View all Code Samples](./04-code-samples.md)** - ---- - ## 5. Configuration Reference > **OTLP** = OpenTelemetry Protocol | **APM** = Application Performance Monitoring @@ -219,12 +196,4 @@ The appendix contains a glossary of OpenTelemetry and xrpld-specific terms, refe --- -## POC Task List - -A step-by-step task list for building a minimal end-to-end proof of concept that demonstrates distributed tracing in xrpld. The POC scope is limited to RPC tracing — showing request traces flowing from xrpld through an OpenTelemetry Collector into Tempo, viewable in Grafana. - -➡️ **[View POC Task List](./POC_taskList.md)** - ---- - _This document provides a comprehensive implementation plan for integrating OpenTelemetry distributed tracing into the xrpld XRP Ledger node software. For detailed information on any section, follow the links to the corresponding sub-documents._ diff --git a/OpenTelemetryPlan/POC_taskList.md b/OpenTelemetryPlan/POC_taskList.md deleted file mode 100644 index 3e2b96d86a..0000000000 --- a/OpenTelemetryPlan/POC_taskList.md +++ /dev/null @@ -1,636 +0,0 @@ -# OpenTelemetry POC Task List - -> **Goal**: Build a minimal end-to-end proof of concept that demonstrates distributed tracing in xrpld. A successful POC will show RPC request traces flowing from xrpld through an OTel Collector into Tempo, viewable in Grafana. -> -> **Scope**: RPC tracing only (highest value, lowest risk per the [CRAWL phase](./06-implementation-phases.md#6102-quick-wins-immediate-value) in the implementation phases). No cross-node P2P context propagation or consensus tracing in the POC. - -### Related Plan Documents - -| Document | Relevance to POC | -| ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [00-tracing-fundamentals.md](./00-tracing-fundamentals.md) | Core concepts: traces, spans, context propagation, sampling | -| [01-architecture-analysis.md](./01-architecture-analysis.md) | RPC request flow (§1.5), key trace points (§1.6), instrumentation priority (§1.7) | -| [02-design-decisions.md](./02-design-decisions.md) | SDK selection (§2.1), exporter config (§2.2), span naming (§2.3), attribute schema (§2.4), coexistence with PerfLog/Insight (§2.6) | -| [03-implementation-strategy.md](./03-implementation-strategy.md) | Directory structure (§3.1), key principles (§3.2), performance overhead (§3.3-3.6), conditional compilation (§3.7.3), code intrusiveness (§3.9) | -| [04-code-samples.md](./04-code-samples.md) | Telemetry interface (§4.1), SpanGuard (§4.2), macros (§4.3), RPC instrumentation (§4.5.3) | -| [05-configuration-reference.md](./05-configuration-reference.md) | xrpld config (§5.1), config parser (§5.2), Application integration (§5.3), CMake (§5.4), Collector config (§5.5), Docker Compose (§5.6), Grafana (§5.8) | -| [06-implementation-phases.md](./06-implementation-phases.md) | Phase 1 core tasks (§6.2), Phase 2 RPC tasks (§6.3), quick wins (§6.10), definition of done (§6.11) | -| [07-observability-backends.md](./07-observability-backends.md) | Tempo dev setup (§7.1), Grafana dashboards (§7.6), alert rules (§7.6.3) | - ---- - -## Task 0: Docker Observability Stack Setup - -> **OTLP** = OpenTelemetry Protocol - -**Objective**: Stand up the backend infrastructure to receive, store, and display traces. - -**What to do**: - -- Create `docker/telemetry/docker-compose.yml` in the repo with three services: - 1. **OpenTelemetry Collector** (`otel/opentelemetry-collector-contrib:0.92.0`) - - Expose ports `4317` (OTLP gRPC) and `4318` (OTLP HTTP) - - Expose port `13133` (health check) - - Mount a config file `docker/telemetry/otel-collector-config.yaml` - 2. **Tempo** (`grafana/tempo:2.6.1`) - - Expose port `3200` (HTTP API) and `4317` (OTLP gRPC, internal) - 3. **Grafana** (`grafana/grafana:latest`) — optional but useful - - Expose port `3000` - - Enable anonymous admin access for local dev (`GF_AUTH_ANONYMOUS_ENABLED=true`, `GF_AUTH_ANONYMOUS_ORG_ROLE=Admin`) - - Provision Tempo as a data source via `docker/telemetry/grafana/provisioning/datasources/tempo.yaml` - -- Create `docker/telemetry/otel-collector-config.yaml`: - - ```yaml - receivers: - otlp: - protocols: - grpc: - endpoint: 0.0.0.0:4317 - http: - endpoint: 0.0.0.0:4318 - - processors: - batch: - timeout: 1s - send_batch_size: 100 - - exporters: - logging: - verbosity: detailed - otlp/tempo: - endpoint: tempo:4317 - tls: - insecure: true - - service: - pipelines: - traces: - receivers: [otlp] - processors: [batch] - exporters: [logging, otlp/tempo] - ``` - -- Create Grafana Tempo datasource provisioning file at `docker/telemetry/grafana/provisioning/datasources/tempo.yaml`: - ```yaml - apiVersion: 1 - datasources: - - name: Tempo - type: tempo - access: proxy - url: http://tempo:3200 - ``` - -**Verification**: Run `docker compose -f docker/telemetry/docker-compose.yml up -d`, then: - -- `curl http://localhost:13133` returns healthy (Collector) -- `http://localhost:3000` opens Grafana (Tempo datasource available, no traces yet) - -**Reference**: - -- [05-configuration-reference.md §5.5](./05-configuration-reference.md) — Collector config (dev YAML with Tempo exporter) -- [05-configuration-reference.md §5.6](./05-configuration-reference.md) — Docker Compose development environment -- [07-observability-backends.md §7.1](./07-observability-backends.md) — Tempo quick start and backend selection -- [05-configuration-reference.md §5.8](./05-configuration-reference.md) — Grafana datasource provisioning and dashboards - ---- - -## Task 1: Add OpenTelemetry C++ SDK Dependency - -**Objective**: Make `opentelemetry-cpp` available to the build system. - -**What to do**: - -- Edit `conanfile.py` to add `opentelemetry-cpp` as an **optional** dependency. The gRPC otel plugin flag (`"grpc/*:otel_plugin": False`) in the existing conanfile may need to remain false — we pull the OTel SDK separately. - - Add a Conan option: `with_telemetry = [True, False]` defaulting to `False` - - When `with_telemetry` is `True`, add `opentelemetry-cpp` to `self.requires()` - - Required OTel Conan components: `opentelemetry-cpp` (which bundles api, sdk, and exporters). If the package isn't in Conan Center, consider using `FetchContent` in CMake or building from source as a fallback. -- Edit `CMakeLists.txt`: - - Add option: `option(XRPL_ENABLE_TELEMETRY "Enable OpenTelemetry tracing" OFF)` - - When ON, `find_package(opentelemetry-cpp CONFIG REQUIRED)` and add compile definition `XRPL_ENABLE_TELEMETRY` - - When OFF, do nothing (zero build impact) -- Verify the build succeeds with `-DXRPL_ENABLE_TELEMETRY=OFF` (no regressions) and with `-DXRPL_ENABLE_TELEMETRY=ON` (SDK links successfully). - -**Key files**: - -- `conanfile.py` -- `CMakeLists.txt` - -**Reference**: - -- [05-configuration-reference.md §5.4](./05-configuration-reference.md) — CMake integration, `FindOpenTelemetry.cmake`, `XRPL_ENABLE_TELEMETRY` option -- [03-implementation-strategy.md §3.2](./03-implementation-strategy.md) — Key principle: zero-cost when disabled via compile-time flags -- [02-design-decisions.md §2.1](./02-design-decisions.md) — SDK selection rationale and required OTel components - ---- - -## Task 2: Create Core Telemetry Interface and NullTelemetry - -**Objective**: Define the `Telemetry` abstract interface and a no-op implementation so the rest of the codebase can reference telemetry without hard-depending on the OTel SDK. - -**What to do**: - -- Create `include/xrpl/telemetry/Telemetry.h`: - - Define `namespace xrpl::telemetry` - - Define `struct Telemetry::Setup` holding: `enabled`, `exporterEndpoint`, `samplingRatio`, `serviceName`, `serviceVersion`, `serviceInstanceId`, `traceRpc`, `traceTransactions`, `traceConsensus`, `tracePeer` - - Define abstract `class Telemetry` with: - - `virtual void start() = 0;` - - `virtual void stop() = 0;` - - `virtual bool isEnabled() const = 0;` - - `virtual nostd::shared_ptr getTracer(string_view name = "xrpld") = 0;` - - `virtual nostd::shared_ptr startSpan(string_view name, SpanKind kind = kInternal) = 0;` - - `virtual nostd::shared_ptr startSpan(string_view name, Context const& parentContext, SpanKind kind = kInternal) = 0;` - - `virtual bool shouldTraceRpc() const = 0;` - - `virtual bool shouldTraceTransactions() const = 0;` - - `virtual bool shouldTraceConsensus() const = 0;` - - Factory: `std::unique_ptr make_Telemetry(Setup const&, beast::Journal);` - - Config parser: `Telemetry::Setup setup_Telemetry(Section const&, std::string const& nodePublicKey, std::string const& version);` - -- Create `include/xrpl/telemetry/SpanGuard.h`: - - RAII guard that takes an `nostd::shared_ptr`, creates a `Scope`, and calls `span->End()` in destructor. - - Convenience: `setAttribute()`, `setOk()`, `setStatus()`, `addEvent()`, `recordException()`, `context()` - - See [04-code-samples.md](./04-code-samples.md) §4.2 for the full implementation. - -- Create `src/libxrpl/telemetry/NullTelemetry.cpp`: - - Implements `Telemetry` with all no-ops. - - `isEnabled()` returns `false`, `startSpan()` returns a noop span. - - This is used when `XRPL_ENABLE_TELEMETRY` is OFF or `enabled=0` in config. - -- Guard all OTel SDK headers behind `#ifdef XRPL_ENABLE_TELEMETRY`. The `NullTelemetry` implementation should compile without the OTel SDK present. - -**Key new files**: - -- `include/xrpl/telemetry/Telemetry.h` -- `include/xrpl/telemetry/SpanGuard.h` -- `src/libxrpl/telemetry/NullTelemetry.cpp` - -**Reference**: - -- [04-code-samples.md §4.1](./04-code-samples.md) — Full `Telemetry` interface with `Setup` struct, lifecycle, tracer access, span creation, and component filtering methods -- [04-code-samples.md §4.2](./04-code-samples.md) — Full `SpanGuard` RAII implementation and `NullSpanGuard` no-op class -- [03-implementation-strategy.md §3.1](./03-implementation-strategy.md) — Directory structure: `include/xrpl/telemetry/` for headers, `src/libxrpl/telemetry/` for implementation -- [03-implementation-strategy.md §3.7.3](./03-implementation-strategy.md) — Conditional instrumentation and zero-cost compile-time disabled pattern - ---- - -## Task 3: Implement OTel-Backed Telemetry - -> **OTLP** = OpenTelemetry Protocol - -**Objective**: Implement the real `Telemetry` class that initializes the OTel SDK, configures the OTLP exporter and batch processor, and creates tracers/spans. - -**What to do**: - -- Create `src/libxrpl/telemetry/Telemetry.cpp` (compiled only when `XRPL_ENABLE_TELEMETRY=ON`): - - `class TelemetryImpl : public Telemetry` that: - - In `start()`: creates a `TracerProvider` with: - - Resource attributes: `service.name`, `service.version`, `service.instance.id` - - An `OtlpHttpExporter` pointed at `setup.exporterEndpoint` (default `localhost:4318`) - - A `BatchSpanProcessor` with configurable batch size and delay - - A `TraceIdRatioBasedSampler` using `setup.samplingRatio` - - Sets the global `TracerProvider` - - In `stop()`: calls `ForceFlush()` then shuts down the provider - - In `startSpan()`: delegates to `getTracer()->StartSpan(name, ...)` - - `shouldTraceRpc()` etc. read from `Setup` fields - -- Create `src/libxrpl/telemetry/TelemetryConfig.cpp`: - - `setup_Telemetry()` parses the `[telemetry]` config section from `xrpld.cfg` - - Maps config keys: `enabled`, `exporter`, `endpoint`, `sampling_ratio`, `trace_rpc`, `trace_transactions`, `trace_consensus`, `trace_peer` - -- Wire `make_Telemetry()` factory: - - If `setup.enabled` is true AND `XRPL_ENABLE_TELEMETRY` is defined: return `TelemetryImpl` - - Otherwise: return `NullTelemetry` - -- Add telemetry source files to CMake. When `XRPL_ENABLE_TELEMETRY=ON`, compile `Telemetry.cpp` and `TelemetryConfig.cpp` and link against `opentelemetry-cpp::api`, `opentelemetry-cpp::sdk`, `opentelemetry-cpp::otlp_grpc_exporter`. When OFF, compile only `NullTelemetry.cpp`. - -**Key new files**: - -- `src/libxrpl/telemetry/Telemetry.cpp` -- `src/libxrpl/telemetry/TelemetryConfig.cpp` - -**Key modified files**: - -- `CMakeLists.txt` (add telemetry library target) - -**Reference**: - -- [04-code-samples.md §4.1](./04-code-samples.md) — `Telemetry` interface that `TelemetryImpl` must implement -- [05-configuration-reference.md §5.2](./05-configuration-reference.md) — `setup_Telemetry()` config parser implementation -- [02-design-decisions.md §2.2](./02-design-decisions.md) — OTLP/gRPC exporter config (endpoint, TLS options) -- [02-design-decisions.md §2.4.1](./02-design-decisions.md) — Resource attributes: `service.name`, `service.version`, `service.instance.id`, `xrpl.network.id` -- [03-implementation-strategy.md §3.4](./03-implementation-strategy.md) — Per-operation CPU costs and overhead budget for span creation -- [03-implementation-strategy.md §3.5](./03-implementation-strategy.md) — Memory overhead: static (~456 KB) and dynamic (~1.2 MB) budgets - ---- - -## Task 4: Integrate Telemetry into Application Lifecycle - -**Objective**: Wire the `Telemetry` object into the `ServiceRegistry` / `Application` so all components can access it. - -**What to do**: - -- Edit `include/xrpl/core/ServiceRegistry.h`: - - Forward-declare `namespace telemetry { class Telemetry; }` inside `namespace xrpl` - - Add pure virtual method: `virtual telemetry::Telemetry& getTelemetry() = 0;` - - (`Application` extends `ServiceRegistry`, so this is automatically available on `Application` too) - -- Edit `src/xrpld/app/main/Application.cpp` (the `ApplicationImp` class): - - Add member: `std::unique_ptr telemetry_;` - - In the member initializer list, construct telemetry with an empty - `serviceInstanceId` (node identity is not yet known): - ```cpp - , telemetry_( - telemetry::make_Telemetry( - telemetry::setup_Telemetry( - config_->section("telemetry"), - "", // Updated later via setServiceInstanceId() - BuildInfo::getVersionString()), - logs_->journal("Telemetry"))) - ``` - - In `setup()`, after `nodeIdentity_` is resolved, inject the node - public key as the service instance ID: - ```cpp - if (!config_->section("telemetry").exists("service_instance_id")) - telemetry_->setServiceInstanceId( - toBase58(TokenType::NodePublic, nodeIdentity_->first)); - ``` - - In `start()`: call `telemetry_->start()` - - In `run()` (shutdown path): call `telemetry_->stop()` (to flush pending spans) - - Implement `getTelemetry()` override: return `*telemetry_` - -- Add `[telemetry]` section to the example config `cfg/xrpld-example.cfg`: - ```ini - # [telemetry] - # enabled=1 - # endpoint=http://localhost:4318/v1/traces - # sampling_ratio=1.0 - # trace_rpc=1 - ``` - -> **Access patterns**: Components holding `ServiceRegistry&` (e.g. -> `NetworkOPsImp`) call `registry_.get().getTelemetry()`. Components -> holding `Application&` (e.g. `ServerHandler`, `PeerImp`, -> `RCLConsensusAdaptor`) call `app_.getTelemetry()` directly. Both -> resolve to the same `Telemetry` instance. - -**Key modified files**: - -- `include/xrpl/core/ServiceRegistry.h` -- `src/xrpld/app/main/Application.cpp` -- `cfg/xrpld-example.cfg` (example config) - -**Reference**: - -- [05-configuration-reference.md §5.3](./05-configuration-reference.md) — `ApplicationImp` changes: member declaration, constructor init, `start()`/`stop()` wiring, `getTelemetry()` override -- [05-configuration-reference.md §5.1](./05-configuration-reference.md) — `[telemetry]` config section format and all option defaults -- [03-implementation-strategy.md §3.9.2](./03-implementation-strategy.md) — File impact assessment: `Application.cpp` ~15 lines added, ~3 changed (Low risk) - ---- - -## Task 5: Create Instrumentation Macros - -**Objective**: Define convenience macros that make instrumenting code one-liners, and that compile to zero-cost no-ops when telemetry is disabled. - -**What to do**: - -- Create `src/xrpld/telemetry/TracingInstrumentation.h`: - - When `XRPL_ENABLE_TELEMETRY` is defined: - - ```cpp - #define XRPL_TRACE_SPAN(telemetry, name) \ - auto _xrpl_span_ = (telemetry).startSpan(name); \ - ::xrpl::telemetry::SpanGuard _xrpl_guard_(_xrpl_span_) - - #define XRPL_TRACE_RPC(telemetry, name) \ - std::optional<::xrpl::telemetry::SpanGuard> _xrpl_guard_; \ - if ((telemetry).shouldTraceRpc()) { \ - _xrpl_guard_.emplace((telemetry).startSpan(name)); \ - } - - #define XRPL_TRACE_SET_ATTR(key, value) \ - if (_xrpl_guard_.has_value()) { \ - _xrpl_guard_->setAttribute(key, value); \ - } - - #define XRPL_TRACE_EXCEPTION(e) \ - if (_xrpl_guard_.has_value()) { \ - _xrpl_guard_->recordException(e); \ - } - ``` - - - When `XRPL_ENABLE_TELEMETRY` is NOT defined, all macros expand to `((void)0)` - -**Key new file**: - -- `src/xrpld/telemetry/TracingInstrumentation.h` - -**Reference**: - -- [04-code-samples.md §4.3](./04-code-samples.md) — Full macro definitions for `XRPL_TRACE_SPAN`, `XRPL_TRACE_RPC`, `XRPL_TRACE_CONSENSUS`, `XRPL_TRACE_SET_ATTR`, `XRPL_TRACE_EXCEPTION` with both enabled and disabled branches -- [03-implementation-strategy.md §3.7.3](./03-implementation-strategy.md) — Conditional instrumentation pattern: compile-time `#ifndef` and runtime `shouldTrace*()` checks -- [03-implementation-strategy.md §3.9.7](./03-implementation-strategy.md) — Before/after code examples showing minimal intrusiveness (~1-3 lines per instrumentation point) - ---- - -## Task 6: Instrument RPC ServerHandler - -> **WS** = WebSocket - -**Objective**: Add tracing to the HTTP RPC entry point so every incoming RPC request creates a span. - -**What to do**: - -- Edit `src/xrpld/rpc/detail/ServerHandler.cpp`: - - `#include` the `TracingInstrumentation.h` header - - In `ServerHandler::onRequest(Session& session)`: - - At the top of the method, add: `XRPL_TRACE_RPC(app_.getTelemetry(), "rpc.request");` - - After the RPC command name is extracted, set attribute: `XRPL_TRACE_SET_ATTR("command", command);` - - After the response status is known, set: `XRPL_TRACE_SET_ATTR("http.status_code", static_cast(statusCode));` - - Wrap error paths with: `XRPL_TRACE_EXCEPTION(e);` - - In `ServerHandler::processRequest(...)`: - - Add a child span: `XRPL_TRACE_RPC(app_.getTelemetry(), "rpc.process");` - - Set method attribute: `XRPL_TRACE_SET_ATTR("rpc_method", request_method);` - - In `ServerHandler::onWSMessage(...)` (WebSocket path): - - Add: `XRPL_TRACE_RPC(app_.getTelemetry(), "rpc.ws.message");` - -- The goal is to see spans like: - ``` - rpc.request - └── rpc.process - ``` - in Tempo/Grafana for every HTTP RPC call. - -**Key modified file**: - -- `src/xrpld/rpc/detail/ServerHandler.cpp` (~15-25 lines added) - -**Reference**: - -- [04-code-samples.md §4.5.3](./04-code-samples.md) — Complete `ServerHandler::onRequest()` instrumented code sample with W3C header extraction, span creation, attribute setting, and error handling -- [01-architecture-analysis.md §1.5](./01-architecture-analysis.md) — RPC request flow diagram: HTTP request -> attributes -> jobqueue.enqueue -> rpc.command -> response -- [01-architecture-analysis.md §1.6](./01-architecture-analysis.md) — Key trace points table: `rpc.request` in `ServerHandler.cpp::onRequest()` (Priority: High) -- [02-design-decisions.md §2.3](./02-design-decisions.md) — Span naming convention: `rpc.request`, `rpc.command.*` -- [02-design-decisions.md §2.4.2](./02-design-decisions.md) — RPC span attributes: `command`, `version`, `rpc_role`, `params` -- [03-implementation-strategy.md §3.9.2](./03-implementation-strategy.md) — File impact: `ServerHandler.cpp` ~40 lines added, ~10 changed (Low risk) - ---- - -## Task 7: Instrument RPC Command Execution - -**Objective**: Add per-command tracing inside the RPC handler so each command (e.g., `submit`, `account_info`, `server_info`) gets its own child span. - -**What to do**: - -- Edit `src/xrpld/rpc/detail/RPCHandler.cpp`: - - `#include` the `TracingInstrumentation.h` header - - In `doCommand(RPC::JsonContext& context, Json::Value& result)`: - - At the top: `XRPL_TRACE_RPC(context.app.getTelemetry(), "rpc.command." + context.method);` - - Set attributes: - - `XRPL_TRACE_SET_ATTR("command", context.method);` - - `XRPL_TRACE_SET_ATTR("version", static_cast(context.apiVersion));` - - `XRPL_TRACE_SET_ATTR("rpc_role", (context.role == Role::ADMIN) ? "admin" : "user");` - - On success: `XRPL_TRACE_SET_ATTR("rpc_status", "success");` - - On error: `XRPL_TRACE_SET_ATTR("rpc_status", "error");` and set the error message - -- After this, traces in Tempo/Grafana should look like: - ``` - rpc.request (command=account_info) - └── rpc.process - └── rpc.command.account_info (version=2, rpc_role=user, rpc_status=success) - ``` - -**Key modified file**: - -- `src/xrpld/rpc/detail/RPCHandler.cpp` (~15-20 lines added) - -**Reference**: - -- [04-code-samples.md §4.5.3](./04-code-samples.md) — `ServerHandler::onRequest()` code sample (includes child span pattern for `rpc.command.*`) -- [02-design-decisions.md §2.3](./02-design-decisions.md) — Span naming: `rpc.command.*` pattern with dynamic command name (e.g., `rpc.command.server_info`) -- [02-design-decisions.md §2.4.2](./02-design-decisions.md) — RPC attribute schema: `command`, `version`, `rpc_role`, `rpc_status` -- [01-architecture-analysis.md §1.6](./01-architecture-analysis.md) — Key trace points table: `rpc.command.*` in `RPCHandler.cpp::doCommand()` (Priority: High) -- [02-design-decisions.md §2.6.5](./02-design-decisions.md) — Correlation with PerfLog: how `doCommand()` can link trace_id with existing PerfLog entries -- [03-implementation-strategy.md §3.4.4](./03-implementation-strategy.md) — RPC request overhead budget: ~1.75 μs total per request - ---- - -## Task 8: Build, Run, and Verify End-to-End - -> **OTLP** = OpenTelemetry Protocol - -**Objective**: Prove the full pipeline works: xrpld emits traces -> OTel Collector receives them -> Tempo stores them for Grafana visualization. - -**What to do**: - -1. **Start the Docker stack**: - - ```bash - docker compose -f docker/telemetry/docker-compose.yml up -d - ``` - - Verify Collector health: `curl http://localhost:13133` - -2. **Build xrpld with telemetry**: - - ```bash - # Adjust for your actual build workflow - conan install . --build=missing -o with_telemetry=True - cmake --preset default -DXRPL_ENABLE_TELEMETRY=ON - cmake --build --preset default - ``` - -3. **Configure xrpld**: - Add to `xrpld.cfg` (or your local test config): - - ```ini - [telemetry] - enabled=1 - endpoint=localhost:4317 - sampling_ratio=1.0 - trace_rpc=1 - ``` - -4. **Start xrpld** in standalone mode: - - ```bash - ./rippled --conf xrpld.cfg -a --start - ``` - -5. **Generate RPC traffic**: - - ```bash - # server_info - curl -s -X POST http://localhost:5005 \ - -H "Content-Type: application/json" \ - -d '{"method":"server_info","params":[{}]}' - - # ledger - curl -s -X POST http://localhost:5005 \ - -H "Content-Type: application/json" \ - -d '{"method":"ledger","params":[{"ledger_index":"current"}]}' - - # account_info (will error in standalone, that's fine — we trace errors too) - curl -s -X POST http://localhost:5005 \ - -H "Content-Type: application/json" \ - -d '{"method":"account_info","params":[{"account":"rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh"}]}' - ``` - -6. **Verify in Grafana (Tempo)**: - - Open `http://localhost:3000` - - Navigate to Explore → select Tempo datasource - - Search for service `xrpld` - - Confirm you see traces with spans: `rpc.request` -> `rpc.process` -> `rpc.command.server_info` - - Click into a trace and verify attributes: `command`, `rpc_status`, `version` - -7. **Verify zero-overhead when disabled**: - - Rebuild with `XRPL_ENABLE_TELEMETRY=OFF`, or set `enabled=0` in config - - Run the same RPC calls - - Confirm no new traces appear and no errors in xrpld logs - -**Verification Checklist**: - -- [ ] Docker stack starts without errors -- [ ] xrpld builds with `-DXRPL_ENABLE_TELEMETRY=ON` -- [ ] xrpld starts and connects to OTel Collector (check xrpld logs for telemetry messages) -- [ ] Traces appear in Grafana/Tempo under service "xrpld" -- [ ] Span hierarchy is correct (parent-child relationships) -- [ ] Span attributes are populated (`command`, `rpc_status`, etc.) -- [ ] Error spans show error status and message -- [ ] Building with `XRPL_ENABLE_TELEMETRY=OFF` produces no regressions -- [ ] Setting `enabled=0` at runtime produces no traces and no errors - -**Reference**: - -- [06-implementation-phases.md §6.11.1](./06-implementation-phases.md) — Phase 1 definition of done: SDK compiles, runtime toggle works, span creation verified in Tempo, config validation passes -- [06-implementation-phases.md §6.11.2](./06-implementation-phases.md#6112-phase-2-rpc-tracing) — Phase 2 definition of done: 100% RPC coverage, traceparent propagation, <1ms p99 overhead, dashboard deployed -- [06-implementation-phases.md §6.8](./06-implementation-phases.md) — Success metrics: trace coverage >95%, CPU overhead <3%, memory <5 MB, latency impact <2% -- [03-implementation-strategy.md §3.9.5](./03-implementation-strategy.md) — Backward compatibility: config optional, protocol unchanged, `XRPL_ENABLE_TELEMETRY=OFF` produces identical binary -- [01-architecture-analysis.md §1.8](./01-architecture-analysis.md) — Observable outcomes: what traces, metrics, and dashboards to expect - ---- - -## Task 9: Document POC Results and Next Steps - -> **OTLP** = OpenTelemetry Protocol | **WS** = WebSocket - -**Objective**: Capture findings, screenshots, and remaining work for the team. - -**What to do**: - -- Take screenshots of Grafana/Tempo showing: - - The service list with "xrpld" - - A trace with the full span tree - - Span detail view showing attributes -- Document any issues encountered (build issues, SDK quirks, missing attributes) -- Note performance observations (build time impact, any noticeable runtime overhead) -- Write a short summary of what the POC proves and what it doesn't cover yet: - - **Proves**: OTel SDK integrates with xrpld, OTLP export works, RPC traces visible - - **Doesn't cover**: Cross-node P2P context propagation, consensus tracing, protobuf trace context, W3C traceparent header extraction, tail-based sampling, production deployment -- Outline next steps (mapping to the full plan phases): - - [Phase 2](./06-implementation-phases.md) completion: [W3C header extraction](./02-design-decisions.md) (§2.5), WebSocket tracing, all [RPC handlers](./01-architecture-analysis.md) (§1.6) - - [Phase 3](./06-implementation-phases.md): [Protobuf `TraceContext` message](./04-code-samples.md) (§4.4), [transaction relay tracing](./04-code-samples.md) (§4.5.1) across nodes - - [Phase 4](./06-implementation-phases.md): [Consensus round and phase tracing](./04-code-samples.md) (§4.5.2) - - [Phase 5](./06-implementation-phases.md): [Production collector config](./05-configuration-reference.md) (§5.5.2), [Grafana dashboards](./07-observability-backends.md) (§7.6), [alerting](./07-observability-backends.md) (§7.6.3) - -**Reference**: - -- [06-implementation-phases.md §6.1](./06-implementation-phases.md) — Full 5-phase timeline overview and Gantt chart -- [06-implementation-phases.md §6.10](./06-implementation-phases.md) — Crawl-Walk-Run strategy: POC is the CRAWL phase, next steps are WALK and RUN -- [06-implementation-phases.md §6.12](./06-implementation-phases.md) — Recommended implementation order (14 steps across 9 weeks) -- [03-implementation-strategy.md §3.9](./03-implementation-strategy.md) — Code intrusiveness assessment and risk matrix for each remaining component -- [07-observability-backends.md §7.2](./07-observability-backends.md) — Production backend selection (Tempo, Elastic APM, Honeycomb, Datadog) -- [02-design-decisions.md §2.5](./02-design-decisions.md) — Context propagation design: W3C HTTP headers, protobuf P2P, JobQueue internal -- [00-tracing-fundamentals.md](./00-tracing-fundamentals.md) — Reference for team onboarding on distributed tracing concepts - ---- - -## Summary - -| Task | Description | New Files | Modified Files | Depends On | -| ---- | ------------------------------------ | --------- | -------------- | ---------- | -| 0 | Docker observability stack | 4 | 0 | — | -| 1 | OTel C++ SDK dependency | 0 | 2 | — | -| 2 | Core Telemetry interface + NullImpl | 3 | 0 | 1 | -| 3 | OTel-backed Telemetry implementation | 2 | 1 | 1, 2 | -| 4 | Application lifecycle integration | 0 | 3 | 2, 3 | -| 5 | Instrumentation macros | 1 | 0 | 2 | -| 6 | Instrument RPC ServerHandler | 0 | 1 | 4, 5 | -| 7 | Instrument RPC command execution | 0 | 1 | 4, 5 | -| 8 | End-to-end verification | 0 | 0 | 0-7 | -| 9 | Document results and next steps | 1 | 0 | 8 | - -**Parallel work**: Tasks 0 and 1 can run in parallel. Tasks 2 and 5 have no dependency on each other. Tasks 6 and 7 can be done in parallel once Tasks 4 and 5 are complete. - ---- - -## Next Steps (Post-POC) - -> **OTLP** = OpenTelemetry Protocol | **WS** = WebSocket - -### Metrics Pipeline for Grafana Dashboards - -The current POC exports **traces only**. Grafana's Explore view can query Tempo for individual traces, but time-series charts (latency histograms, request throughput, error rates) require a **metrics pipeline**. To enable this: - -1. **Add a `spanmetrics` connector** to the OTel Collector config that derives RED metrics (Rate, Errors, Duration) from trace spans automatically: - - ```yaml - connectors: - spanmetrics: - histogram: - explicit: - buckets: [1ms, 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 5s] - dimensions: - - name: command - - name: rpc_status - - exporters: - prometheus: - endpoint: 0.0.0.0:8889 - - service: - pipelines: - traces: - receivers: [otlp] - processors: [batch] - exporters: [debug, otlp/tempo, spanmetrics] - metrics: - receivers: [spanmetrics] - exporters: [prometheus] - ``` - -2. **Add Prometheus** to the Docker Compose stack to scrape the collector's metrics endpoint. - -3. **Add Prometheus as a Grafana datasource** and build dashboards for: - - RPC request latency (p50/p95/p99) by command - - RPC throughput (requests/sec) by command - - Error rate by command - - Span duration distribution - -### Additional Instrumentation - -- **W3C `traceparent` header extraction** in `ServerHandler` to support cross-service context propagation from external callers -- **WebSocket RPC tracing** in `ServerHandler::onWSMessage()` -- **Transaction relay tracing** across nodes using protobuf `TraceContext` messages -- **Consensus round and phase tracing** for validator coordination visibility -- **Ledger close tracing** to measure close-to-validated latency - -### Production Hardening - -- **Tail-based sampling** in the OTel Collector to reduce volume while retaining error/slow traces -- **TLS configuration** for the OTLP exporter in production deployments -- **Resource limits** on the batch processor queue to prevent unbounded memory growth -- **Health monitoring** for the telemetry pipeline itself (collector lag, export failures) - -### POC Lessons Learned - -Issues encountered during POC implementation that inform future work: - -| Issue | Resolution | Impact on Future Work | -| -------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | ---------------------------------------------------------------- | -| Conan lockfile rejected `opentelemetry-cpp/1.18.0` | Used `--lockfile=""` to bypass | Lockfile must be regenerated when adding new dependencies | -| Conan package only builds OTLP HTTP exporter, not gRPC | Switched from gRPC to HTTP exporter (`localhost:4318/v1/traces`) | HTTP exporter is the default; gRPC requires custom Conan profile | -| CMake target `opentelemetry-cpp::api` etc. don't exist in Conan package | Use umbrella target `opentelemetry-cpp::opentelemetry-cpp` | Conan targets differ from upstream CMake targets | -| OTel Collector `logging` exporter deprecated | Renamed to `debug` exporter | Use `debug` in all collector configs going forward | -| Macro parameter `telemetry` collided with `::xrpl::telemetry::` namespace | Renamed macro params to `_tel_obj_`, `_span_name_` | Avoid common words as macro parameter names | -| `opentelemetry::trace::Scope` creates new context on move | Store scope as member, create once in constructor | SpanGuard move semantics need care with Scope lifecycle | -| `TracerProviderFactory::Create` returns `unique_ptr`, not `nostd::shared_ptr` | Use `std::shared_ptr` member, wrap in `nostd::shared_ptr` for global provider | OTel SDK factory return types don't match API provider types |