diff --git a/OpenTelemetryPlan/00-tracing-fundamentals.md b/OpenTelemetryPlan/00-tracing-fundamentals.md index 0dfac46e72..24322bdd09 100644 --- a/OpenTelemetryPlan/00-tracing-fundamentals.md +++ b/OpenTelemetryPlan/00-tracing-fundamentals.md @@ -406,7 +406,7 @@ flowchart TB ## Distributed Traces Across Nodes -In distributed systems like rippled, traces span **multiple independent nodes**. The trace context must be propagated in network messages: +In distributed systems like xrpld, traces span **multiple independent nodes**. The trace context must be propagated in network messages: ```mermaid sequenceDiagram @@ -492,7 +492,7 @@ traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 └── Version ``` -### Protocol Buffers (rippled P2P messages) +### Protocol Buffers (xrpld P2P messages) ```protobuf message TMTransaction { @@ -534,7 +534,7 @@ Trace completes → Collector evaluates: --- -## Key Benefits for rippled +## Key Benefits for xrpld | Challenge | How Tracing Helps | | ---------------------------------- | ---------------------------------------- | diff --git a/OpenTelemetryPlan/01-architecture-analysis.md b/OpenTelemetryPlan/01-architecture-analysis.md index 4424744e09..c62ac3454c 100644 --- a/OpenTelemetryPlan/01-architecture-analysis.md +++ b/OpenTelemetryPlan/01-architecture-analysis.md @@ -5,15 +5,15 @@ --- -## 1.1 Current rippled Architecture Overview +## 1.1 Current xrpld Architecture Overview > **WS** = WebSocket | **UNL** = Unique Node List | **TxQ** = Transaction Queue | **StatsD** = Statistics Daemon -The rippled node software consists of several interconnected components that need instrumentation for distributed tracing: +The xrpld node software consists of several interconnected components that need instrumentation for distributed tracing: ```mermaid flowchart TB - subgraph rippled["rippled Node"] + subgraph xrpld["xrpld Node"] subgraph services["Core Services"] RPC["RPC Server
(HTTP/WS/gRPC)"] Overlay["Overlay
(P2P Network)"] @@ -47,7 +47,7 @@ flowchart TB JobQueue --> appservices end - style rippled fill:#424242,stroke:#212121,color:#ffffff + style xrpld fill:#424242,stroke:#212121,color:#ffffff style services fill:#1565c0,stroke:#0d47a1,color:#ffffff style processing fill:#2e7d32,stroke:#1b5e20,color:#ffffff style appservices fill:#6a1b9a,stroke:#4a148c,color:#ffffff @@ -56,7 +56,7 @@ flowchart TB **Reading the diagram:** -- **Core Services (blue)**: The entry points into rippled -- RPC Server handles client requests, Overlay manages peer-to-peer networking, Consensus drives agreement, and ValidatorList manages trusted validators. +- **Core Services (blue)**: The entry points into xrpld -- RPC Server handles client requests, Overlay manages peer-to-peer networking, Consensus drives agreement, and ValidatorList manages trusted validators. - **JobQueue (center)**: The asynchronous thread pool that decouples Core Services from the Processing and Application layers. All work flows through it. - **Processing Layer (green)**: Core business logic -- NetworkOPs processes transactions, LedgerMaster manages ledger state, NodeStore handles persistence, and InboundLedgers synchronizes missing data. - **Application Services (purple)**: Higher-level features -- PathFinding computes payment routes, TxQ manages fee-based queuing, and LoadManager tracks server load. @@ -71,7 +71,7 @@ flowchart TB | Who (Plain English) | Technical Term | | ----------------------------------------- | -------------------------- | -| Network node running XRPL software | rippled node | +| Network node running XRPL software | xrpld node | | External client submitting requests | RPC Client | | Network neighbor sharing data | Peer (PeerImp) | | Request handler for client queries | RPC Server (ServerHandler) | @@ -354,17 +354,17 @@ After implementing OpenTelemetry, operators and developers will gain visibility ### 1.8.1 What You Will See: Traces -| Trace Type | Description | Example Query in Grafana/Tempo | -| -------------------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------------ | -| **Transaction Lifecycle** | Full journey from RPC submission through validation, relay, consensus, and ledger inclusion | `{service.name="rippled" && xrpl.tx.hash="ABC123..."}` | -| **Cross-Node Propagation** | Transaction path across multiple rippled nodes with timing | `{xrpl.tx.relay_count > 0}` | -| **Consensus Rounds** | Complete round with all phases (open, establish, accept) | `{span.name=~"consensus.round.*"}` | -| **RPC Request Processing** | Individual command execution with timing breakdown | `{xrpl.rpc.command="account_info"}` | -| **Ledger Acquisition** | Peer-to-peer ledger data requests and responses | `{span.name="ledger.acquire"}` | -| **PathFinding Latency** | Path computation time and cache effectiveness for payment RPCs | `{span.name="pathfind.compute"}` | -| **TxQ Behavior** | Queue depth, eviction patterns, fee escalation during congestion | `{span.name=~"txq.*"}` | -| **Ledger Sync** | Full acquisition timeline including delta and transaction fetches | `{span.name=~"ledger.acquire.*"}` | -| **Validator Health** | UNL fetch success, manifest updates, stale list detection | `{span.name=~"validator.*"}` | +| Trace Type | Description | Example Query in Grafana/Tempo | +| -------------------------- | ------------------------------------------------------------------------------------------- | ---------------------------------------------------- | +| **Transaction Lifecycle** | Full journey from RPC submission through validation, relay, consensus, and ledger inclusion | `{service.name="xrpld" && xrpl.tx.hash="ABC123..."}` | +| **Cross-Node Propagation** | Transaction path across multiple xrpld nodes with timing | `{xrpl.tx.relay_count > 0}` | +| **Consensus Rounds** | Complete round with all phases (open, establish, accept) | `{span.name=~"consensus.round.*"}` | +| **RPC Request Processing** | Individual command execution with timing breakdown | `{xrpl.rpc.command="account_info"}` | +| **Ledger Acquisition** | Peer-to-peer ledger data requests and responses | `{span.name="ledger.acquire"}` | +| **PathFinding Latency** | Path computation time and cache effectiveness for payment RPCs | `{span.name="pathfind.compute"}` | +| **TxQ Behavior** | Queue depth, eviction patterns, fee escalation during congestion | `{span.name=~"txq.*"}` | +| **Ledger Sync** | Full acquisition timeline including delta and transaction fetches | `{span.name=~"ledger.acquire.*"}` | +| **Validator Health** | UNL fetch success, manifest updates, stale list detection | `{span.name=~"validator.*"}` | ### 1.8.2 What You Will See: Metrics (Derived from Traces) diff --git a/OpenTelemetryPlan/02-design-decisions.md b/OpenTelemetryPlan/02-design-decisions.md index 8ff6eaa983..fe87fc78db 100644 --- a/OpenTelemetryPlan/02-design-decisions.md +++ b/OpenTelemetryPlan/02-design-decisions.md @@ -25,10 +25,10 @@ **Manual Instrumentation** (recommended): -| Approach | Pros | Cons | -| ---------- | ----------------------------------------------------------------- | ------------------------------------------------------- | -| **Manual** | Precise control, optimized placement, rippled-specific attributes | More development effort | -| **Auto** | Less code, automatic coverage | Less control, potential overhead, limited customization | +| Approach | Pros | Cons | +| ---------- | --------------------------------------------------------------- | ------------------------------------------------------- | +| **Manual** | Precise control, optimized placement, xrpld-specific attributes | More development effort | +| **Auto** | Less code, automatic coverage | Less control, potential overhead, limited customization | --- @@ -38,10 +38,10 @@ ```mermaid flowchart TB - subgraph nodes["rippled Nodes"] - node1["rippled
Node 1"] - node2["rippled
Node 2"] - node3["rippled
Node 3"] + subgraph nodes["xrpld Nodes"] + node1["xrpld
Node 1"] + node2["xrpld
Node 2"] + node3["xrpld
Node 3"] end collector["OpenTelemetry
Collector
(sidecar or standalone)"] @@ -65,7 +65,7 @@ flowchart TB **Reading the diagram:** -- **rippled Nodes (blue)**: The source of telemetry data. Each rippled node exports spans via OTLP/gRPC on port 4317. +- **xrpld Nodes (blue)**: The source of telemetry data. Each xrpld node exports spans via OTLP/gRPC on port 4317. - **OpenTelemetry Collector (red)**: The central aggregation point that receives spans from all nodes. Can run as a sidecar (per-node) or standalone (shared). Handles batching, filtering, and routing. - **Observability Backends (green)**: The storage and visualization destinations. Tempo is the recommended backend for both development and production, and Elastic APM is an alternative. The Collector routes to one or more backends. - **Arrows (nodes to collector to backends)**: The data pipeline -- spans flow from nodes to the Collector over gRPC, then the Collector fans out to the configured backends. @@ -203,11 +203,11 @@ job: ```cpp // Standard OpenTelemetry semantic conventions -resource::SemanticConventions::SERVICE_NAME = "rippled" +resource::SemanticConventions::SERVICE_NAME = "xrpld" resource::SemanticConventions::SERVICE_VERSION = BuildInfo::getVersionString() resource::SemanticConventions::SERVICE_INSTANCE_ID = -// Custom rippled attributes +// Custom xrpld attributes "xrpl.network.id" = // e.g., 0 for mainnet "xrpl.network.type" = "mainnet" | "testnet" | "devnet" | "standalone" "xrpl.node.type" = "validator" | "stock" | "reporting" @@ -390,7 +390,7 @@ processors: #### Configuration Options for Privacy -In `rippled.cfg`, operators can control data collection granularity: +In `xrpld.cfg`, operators can control data collection granularity: ```ini [telemetry] @@ -407,7 +407,7 @@ redact_account=1 # Hash account addresses before export redact_peer_address=1 # Remove peer IP addresses ``` -> **Note**: The `redact_account` configuration in `rippled.cfg` controls SDK-level redaction before export, while collector-level filtering (see [Collector-Level Data Protection](#collector-level-data-protection) above) provides an additional defense-in-depth layer. Both can operate independently. +> **Note**: The `redact_account` configuration in `xrpld.cfg` controls SDK-level redaction before export, while collector-level filtering (see [Collector-Level Data Protection](#collector-level-data-protection) above) provides an additional defense-in-depth layer. Both can operate independently. > **Key Principle**: Telemetry collects **operational metadata** (timing, counts, hashes) — never **sensitive content** (keys, balances, amounts, raw payloads). @@ -422,7 +422,7 @@ redact_peer_address=1 # Remove peer IP addresses ```mermaid flowchart TB subgraph http["HTTP/WebSocket (RPC)"] - w3c["W3C Trace Context Headers:
traceparent:
00-trace_id-span_id-flags
tracestate: rippled=..."] + w3c["W3C Trace Context Headers:
traceparent:
00-trace_id-span_id-flags
tracestate: xrpld=..."] end subgraph protobuf["Protocol Buffers (P2P)"] @@ -441,7 +441,7 @@ flowchart TB **Reading the diagram:** - **HTTP/WebSocket - RPC (blue)**: For client-facing RPC requests, trace context is propagated using the W3C `traceparent` header. This is the standard approach and works with any OTel-compatible client. -- **Protocol Buffers - P2P (green)**: For peer-to-peer messages between rippled nodes, trace context is embedded as a protobuf `TraceContext` message carrying trace_id, span_id, flags, and optional trace_state. +- **Protocol Buffers - P2P (green)**: For peer-to-peer messages between xrpld nodes, trace context is embedded as a protobuf `TraceContext` message carrying trace_id, span_id, flags, and optional trace_state. - **JobQueue - Internal Async (red)**: For asynchronous work within a single node, the OTel context is captured when a job is created and restored when the job executes on a worker thread. This bridges the async gap so spans remain linked. --- @@ -452,7 +452,7 @@ flowchart TB ### 2.6.1 Existing Frameworks Comparison -rippled already has two observability mechanisms. OpenTelemetry complements (not replaces) them: +xrpld already has two observability mechanisms. OpenTelemetry complements (not replaces) them: | Aspect | PerfLog | Beast Insight (StatsD) | OpenTelemetry | | --------------------- | ----------------------------- | ---------------------------- | ------------------------- | @@ -501,7 +501,7 @@ rippled already has two observability mechanisms. OpenTelemetry complements (not - Single-node perspective ```cpp -// Example StatsD usage in rippled +// Example StatsD usage in xrpld insight.increment("rpc.submit.count"); insight.gauge("ledger.age", age); insight.timing("consensus.round", duration); @@ -542,7 +542,7 @@ span->SetAttribute("peer.id", peerId); ```mermaid flowchart TB - subgraph rippled["rippled Process"] + subgraph xrpld["xrpld Process"] perflog["PerfLog
(JSON to file)"] insight["Beast Insight
(StatsD)"] otel["OpenTelemetry
(Tracing)"] @@ -556,13 +556,13 @@ flowchart TB statsd --> grafana collector --> grafana - style rippled fill:#212121,stroke:#0a0a0a,color:#ffffff + style xrpld fill:#212121,stroke:#0a0a0a,color:#ffffff style grafana fill:#bf360c,stroke:#8c2809,color:#ffffff ``` **Reading the diagram:** -- **rippled Process (dark gray)**: The single rippled node running all three observability frameworks side by side. Each framework operates independently with no interference. +- **xrpld Process (dark gray)**: The single xrpld node running all three observability frameworks side by side. Each framework operates independently with no interference. - **PerfLog to perf.log**: PerfLog writes JSON-formatted event logs to a local file. Grafana can ingest these via Loki or a file-based datasource. - **Beast Insight to StatsD Server**: Insight sends aggregated metrics (counters, gauges) over UDP to a StatsD server. Grafana reads from StatsD-compatible backends like Graphite or Prometheus (via StatsD exporter). - **OpenTelemetry to OTLP Collector**: OTel exports spans over OTLP/gRPC to a Collector, which then forwards to a trace backend (Tempo). diff --git a/OpenTelemetryPlan/03-implementation-strategy.md b/OpenTelemetryPlan/03-implementation-strategy.md index 8e9311639d..4f6beb61d8 100644 --- a/OpenTelemetryPlan/03-implementation-strategy.md +++ b/OpenTelemetryPlan/03-implementation-strategy.md @@ -7,7 +7,7 @@ ## 3.1 Directory Structure -The telemetry implementation follows rippled's existing code organization pattern: +The telemetry implementation follows xrpld's existing code organization pattern: ``` include/xrpl/ @@ -344,7 +344,7 @@ if (telemetry.shouldTracePeer()) > **TxQ** = Transaction Queue -This section provides a detailed assessment of how intrusive the OpenTelemetry integration is to the existing rippled codebase. +This section provides a detailed assessment of how intrusive the OpenTelemetry integration is to the existing xrpld codebase. ### 3.9.1 Files Modified Summary @@ -390,7 +390,7 @@ pie title Code Changes by Component | `src/libxrpl/telemetry/TelemetryConfig.cpp` | ~60 | Config parsing | | `src/libxrpl/telemetry/NullTelemetry.cpp` | ~40 | No-op implementation | -#### Modified Files (Existing Rippled Code) +#### Modified Files (Existing Xrpld Code) | File | Lines Added | Lines Changed | Risk Level | | ------------------------------------------------- | ----------- | ------------- | ---------- | diff --git a/OpenTelemetryPlan/04-code-samples.md b/OpenTelemetryPlan/04-code-samples.md index 5dfdbc32c1..44827586d7 100644 --- a/OpenTelemetryPlan/04-code-samples.md +++ b/OpenTelemetryPlan/04-code-samples.md @@ -30,7 +30,7 @@ namespace telemetry { /** * Main telemetry interface for OpenTelemetry integration. * - * This class provides the primary API for distributed tracing in rippled. + * This class provides the primary API for distributed tracing in xrpld. * It manages the OpenTelemetry SDK lifecycle and provides convenience * methods for creating spans and propagating context. */ @@ -43,7 +43,7 @@ public: struct Setup { bool enabled = false; - std::string serviceName = "rippled"; + std::string serviceName = "xrpld"; std::string serviceVersion; std::string serviceInstanceId; // Node public key @@ -98,7 +98,7 @@ public: /** Get the tracer for creating spans */ virtual opentelemetry::nostd::shared_ptr - getTracer(std::string_view name = "rippled") = 0; + getTracer(std::string_view name = "xrpld") = 0; // ═══════════════════════════════════════════════════════════════════════ // SPAN CREATION (Convenience Methods) @@ -457,7 +457,7 @@ namespace telemetry { Add to `src/xrpld/overlay/detail/ripple.proto`: ```protobuf -// Note: rippled uses proto2 syntax. The 'optional' keyword below is valid +// Note: xrpld uses proto2 syntax. The 'optional' keyword below is valid // in proto2 (it is the default field rule) and is included for clarity. // Trace context for distributed tracing across nodes @@ -1062,7 +1062,7 @@ flowchart TB submit["Submit TX"] end - subgraph NodeA["rippled Node A"] + subgraph NodeA["xrpld Node A"] rpcA["rpc.request"] cmdA["rpc.command.submit"] txRecvA["tx.receive"] @@ -1070,13 +1070,13 @@ flowchart TB txRelayA["tx.relay"] end - subgraph NodeB["rippled Node B"] + subgraph NodeB["xrpld Node B"] txRecvB["tx.receive"] txValB["tx.validate"] txRelayB["tx.relay"] end - subgraph NodeC["rippled Node C"] + subgraph NodeC["xrpld Node C"] txRecvC["tx.receive"] consensusC["consensus.round"] phaseC["consensus.phase.establish"] diff --git a/OpenTelemetryPlan/05-configuration-reference.md b/OpenTelemetryPlan/05-configuration-reference.md index 04239fb246..56627c3b6c 100644 --- a/OpenTelemetryPlan/05-configuration-reference.md +++ b/OpenTelemetryPlan/05-configuration-reference.md @@ -5,7 +5,7 @@ --- -## 5.1 rippled Configuration +## 5.1 xrpld Configuration > **OTLP** = OpenTelemetry Protocol | **TxQ** = Transaction Queue @@ -62,7 +62,7 @@ Add to `cfg/xrpld-example.cfg`: # trace_amendment=0 # Amendment voting (very low volume) # # # Service identification (automatically detected if not specified) -# # service_name=rippled +# # service_name=xrpld # # service_instance_id= [telemetry] @@ -91,7 +91,7 @@ enabled=0 | `trace_txq` | bool | `true` | Enable transaction queue tracing | | `trace_validator` | bool | `false` | Enable validator list/manifest tracing | | `trace_amendment` | bool | `false` | Enable amendment voting tracing | -| `service_name` | string | `"rippled"` | Service name for traces | +| `service_name` | string | `"xrpld"` | Service name for traces | | `service_instance_id` | string | `` | Instance identifier | --- @@ -119,7 +119,7 @@ setup_Telemetry( // Basic settings setup.enabled = section.value_or("enabled", false); - setup.serviceName = section.value_or("service_name", "rippled"); + setup.serviceName = section.value_or("service_name", "xrpld"); setup.serviceVersion = version; setup.serviceInstanceId = section.value_or( "service_instance_id", nodePublicKey); @@ -592,7 +592,7 @@ services: networks: default: - name: rippled-telemetry + name: xrpld-telemetry ``` --- @@ -645,7 +645,7 @@ flowchart TB - **Configuration Sources**: `xrpld.cfg` provides runtime settings (endpoint, sampling) while the CMake flag controls whether telemetry is compiled in at all. - **Initialization**: `setup_Telemetry()` parses config values, then `make_Telemetry()` constructs the provider, processor, and exporter objects. - **Runtime Components**: The `TracerProvider` creates spans, the `BatchProcessor` buffers them, and the `OTLP Exporter` serializes and sends them over the wire. -- **OTLP arrow to Collector**: Trace data leaves the rippled process via OTLP (gRPC or HTTP) and enters the external Collector pipeline. +- **OTLP arrow to Collector**: Trace data leaves the xrpld process via OTLP (gRPC or HTTP) and enters the external Collector pipeline. - **Collector Pipeline**: `Receivers` ingest OTLP data, `Processors` apply sampling/filtering/enrichment, and `Exporters` forward traces to storage backends (Tempo, etc.). --- @@ -654,7 +654,7 @@ flowchart TB > **APM** = Application Performance Monitoring -Step-by-step instructions for integrating rippled traces with Grafana. +Step-by-step instructions for integrating xrpld traces with Grafana. ### 5.8.1 Data Source Configuration @@ -713,10 +713,10 @@ datasources: apiVersion: 1 providers: - - name: "rippled-dashboards" + - name: "xrpld-dashboards" orgId: 1 - folder: "rippled" - folderUid: "rippled" + folder: "xrpld" + folderUid: "xrpld" type: file disableDeletion: false updateIntervalSeconds: 30 @@ -728,8 +728,8 @@ providers: ```json { - "title": "rippled RPC Performance", - "uid": "rippled-rpc-performance", + "title": "xrpld RPC Performance", + "uid": "xrpld-rpc-performance", "panels": [ { "title": "RPC Latency by Command", @@ -738,7 +738,7 @@ providers: "targets": [ { "queryType": "traceql", - "query": "{resource.service.name=\"rippled\" && span.xrpl.rpc.command != \"\"} | histogram_over_time(duration) by (span.xrpl.rpc.command)" + "query": "{resource.service.name=\"xrpld\" && span.xrpl.rpc.command != \"\"} | histogram_over_time(duration) by (span.xrpl.rpc.command)" } ], "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 } @@ -750,7 +750,7 @@ providers: "targets": [ { "queryType": "traceql", - "query": "{resource.service.name=\"rippled\" && status.code=error} | rate() by (span.xrpl.rpc.command)" + "query": "{resource.service.name=\"xrpld\" && status.code=error} | rate() by (span.xrpl.rpc.command)" } ], "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 } @@ -762,7 +762,7 @@ providers: "targets": [ { "queryType": "traceql", - "query": "{resource.service.name=\"rippled\" && span.xrpl.rpc.command != \"\"} | avg(duration) by (span.xrpl.rpc.command) | topk(10)" + "query": "{resource.service.name=\"xrpld\" && span.xrpl.rpc.command != \"\"} | avg(duration) by (span.xrpl.rpc.command) | topk(10)" } ], "gridPos": { "h": 8, "w": 24, "x": 0, "y": 8 } @@ -774,7 +774,7 @@ providers: "targets": [ { "queryType": "traceql", - "query": "{resource.service.name=\"rippled\"}" + "query": "{resource.service.name=\"xrpld\"}" } ], "gridPos": { "h": 8, "w": 24, "x": 0, "y": 16 } @@ -787,8 +787,8 @@ providers: ```json { - "title": "rippled Transaction Tracing", - "uid": "rippled-tx-tracing", + "title": "xrpld Transaction Tracing", + "uid": "xrpld-tx-tracing", "panels": [ { "title": "Transaction Throughput", @@ -797,7 +797,7 @@ providers: "targets": [ { "queryType": "traceql", - "query": "{resource.service.name=\"rippled\" && name=\"tx.receive\"} | rate()" + "query": "{resource.service.name=\"xrpld\" && name=\"tx.receive\"} | rate()" } ], "gridPos": { "h": 4, "w": 6, "x": 0, "y": 0 } @@ -809,7 +809,7 @@ providers: "targets": [ { "queryType": "traceql", - "query": "{resource.service.name=\"rippled\" && name=\"tx.relay\"} | avg(span.xrpl.tx.relay_count)" + "query": "{resource.service.name=\"xrpld\" && name=\"tx.relay\"} | avg(span.xrpl.tx.relay_count)" } ], "gridPos": { "h": 8, "w": 12, "x": 0, "y": 4 } @@ -821,7 +821,7 @@ providers: "targets": [ { "queryType": "traceql", - "query": "{resource.service.name=\"rippled\" && name=\"tx.validate\" && status.code=error}" + "query": "{resource.service.name=\"xrpld\" && name=\"tx.validate\" && status.code=error}" } ], "gridPos": { "h": 8, "w": 12, "x": 12, "y": 4 } @@ -832,26 +832,26 @@ providers: ### 5.8.5 TraceQL Query Examples -Common queries for rippled traces: +Common queries for xrpld traces: ``` # Find all traces for a specific transaction hash -{resource.service.name="rippled" && span.xrpl.tx.hash="ABC123..."} +{resource.service.name="xrpld" && span.xrpl.tx.hash="ABC123..."} # Find slow RPC commands (>100ms) -{resource.service.name="rippled" && name=~"rpc.command.*"} | duration > 100ms +{resource.service.name="xrpld" && name=~"rpc.command.*"} | duration > 100ms # Find consensus rounds taking >5 seconds -{resource.service.name="rippled" && name="consensus.round"} | duration > 5s +{resource.service.name="xrpld" && name="consensus.round"} | duration > 5s # Find failed transactions with error details -{resource.service.name="rippled" && name="tx.validate" && status.code=error} +{resource.service.name="xrpld" && name="tx.validate" && status.code=error} # Find transactions relayed to many peers -{resource.service.name="rippled" && name="tx.relay"} | span.xrpl.tx.relay_count > 10 +{resource.service.name="xrpld" && name="tx.relay"} | span.xrpl.tx.relay_count > 10 # Compare latency across nodes -{resource.service.name="rippled" && name="rpc.command.account_info"} | avg(duration) by (resource.service.instance.id) +{resource.service.name="xrpld" && name="rpc.command.account_info"} | avg(duration) by (resource.service.instance.id) ``` ### 5.8.6 Correlation with PerfLog @@ -863,12 +863,12 @@ To correlate OpenTelemetry traces with existing PerfLog data: ```yaml # promtail-config.yaml scrape_configs: - - job_name: rippled-perflog + - job_name: xrpld-perflog static_configs: - targets: - localhost labels: - job: rippled + job: xrpld __path__: /var/log/rippled/perf*.log pipeline_stages: - json: @@ -922,7 +922,7 @@ To correlate traces with existing Beast Insight metrics: ```yaml # prometheus.yaml scrape_configs: - - job_name: "rippled-statsd" + - job_name: "xrpld-statsd" static_configs: - targets: ["statsd-exporter:9102"] ``` @@ -950,7 +950,7 @@ jsonData: "datasource": "Prometheus", "targets": [ { - "expr": "histogram_quantile(0.99, rate(rippled_rpc_duration_seconds_bucket[5m]))", + "expr": "histogram_quantile(0.99, rate(xrpld_rpc_duration_seconds_bucket[5m]))", "exemplar": true } ] diff --git a/OpenTelemetryPlan/07-observability-backends.md b/OpenTelemetryPlan/07-observability-backends.md index 2877333a41..94124a62fe 100644 --- a/OpenTelemetryPlan/07-observability-backends.md +++ b/OpenTelemetryPlan/07-observability-backends.md @@ -92,13 +92,13 @@ flowchart TD ```mermaid flowchart TB subgraph validators["Validator Nodes"] - v1[rippled
Validator 1] - v2[rippled
Validator 2] + v1[xrpld
Validator 1] + v2[xrpld
Validator 2] end subgraph stock["Stock Nodes"] - s1[rippled
Stock 1] - s2[rippled
Stock 2] + s1[xrpld
Stock 1] + s2[xrpld
Stock 2] end subgraph collector["OTel Collector Cluster"] @@ -140,7 +140,7 @@ flowchart TB **Reading the diagram:** -- **Validator / Stock Nodes**: All rippled nodes emit trace data via OTLP. Validators and stock nodes are grouped separately because they may reside in different network zones. +- **Validator / Stock Nodes**: All xrpld nodes emit trace data via OTLP. Validators and stock nodes are grouped separately because they may reside in different network zones. - **Collector Cluster (DC1, DC2)**: Regional collectors receive OTLP from nodes in their datacenter, apply processing (sampling, enrichment), and fan out to multiple backends. - **Storage Backends**: Tempo and Elastic provide queryable trace storage; S3/GCS Archive provides long-term cold storage for compliance or post-incident analysis. - **Grafana Dashboards**: The single visualization layer that queries both Tempo and Elastic, giving operators a unified view of all traces. @@ -160,7 +160,7 @@ flowchart TB | **DaemonSet** | Collector per host | Shared resources | Complexity | | **Gateway** | Central collector(s) | Centralized processing | Single point of failure | -**Recommendation**: Use **Gateway** pattern with regional collectors for rippled networks: +**Recommendation**: Use **Gateway** pattern with regional collectors for xrpld networks: - One collector cluster per datacenter/region - Tail-based sampling at collector level @@ -197,7 +197,7 @@ flowchart LR **Reading the diagram:** -- **Head Sampling (Node)**: The first filter -- each rippled node decides whether to sample a trace at creation time (default 100%, recommended 10% in production). This controls the volume leaving the node. +- **Head Sampling (Node)**: The first filter -- each xrpld node decides whether to sample a trace at creation time (default 100%, recommended 10% in production). This controls the volume leaving the node. - **Tail Sampling (Collector)**: The second filter -- the collector inspects completed traces and applies rules: keep all errors, keep anything slower than 5 seconds, and keep 10% of the remainder. - **Arrow head → tail**: All head-sampled traces flow to the collector, where tail sampling further reduces volume while preserving the most valuable data. - **Final Traces**: The output after both sampling stages; this is what gets stored and queried. The two-stage approach balances cost with debuggability. @@ -226,15 +226,15 @@ flowchart LR ## 7.6 Grafana Dashboard Examples -Pre-built dashboards for rippled observability. +Pre-built dashboards for xrpld observability. ### 7.6.1 Consensus Health Dashboard ```json { - "title": "rippled Consensus Health", - "uid": "rippled-consensus-health", - "tags": ["rippled", "consensus", "tracing"], + "title": "xrpld Consensus Health", + "uid": "xrpld-consensus-health", + "tags": ["xrpld", "consensus", "tracing"], "panels": [ { "title": "Consensus Round Duration", @@ -243,7 +243,7 @@ Pre-built dashboards for rippled observability. "targets": [ { "queryType": "traceql", - "query": "{resource.service.name=\"rippled\" && name=\"consensus.round\"} | avg(duration) by (resource.service.instance.id)" + "query": "{resource.service.name=\"xrpld\" && name=\"consensus.round\"} | avg(duration) by (resource.service.instance.id)" } ], "fieldConfig": { @@ -267,7 +267,7 @@ Pre-built dashboards for rippled observability. "targets": [ { "queryType": "traceql", - "query": "{resource.service.name=\"rippled\" && name=~\"consensus.phase.*\"} | avg(duration) by (name)" + "query": "{resource.service.name=\"xrpld\" && name=~\"consensus.phase.*\"} | avg(duration) by (name)" } ], "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 } @@ -279,7 +279,7 @@ Pre-built dashboards for rippled observability. "targets": [ { "queryType": "traceql", - "query": "{resource.service.name=\"rippled\" && name=\"consensus.round\"} | avg(span.xrpl.consensus.proposers)" + "query": "{resource.service.name=\"xrpld\" && name=\"consensus.round\"} | avg(span.xrpl.consensus.proposers)" } ], "gridPos": { "h": 4, "w": 6, "x": 0, "y": 8 } @@ -291,7 +291,7 @@ Pre-built dashboards for rippled observability. "targets": [ { "queryType": "traceql", - "query": "{resource.service.name=\"rippled\" && name=\"consensus.round\"} | duration > 5s" + "query": "{resource.service.name=\"xrpld\" && name=\"consensus.round\"} | duration > 5s" } ], "gridPos": { "h": 8, "w": 24, "x": 0, "y": 12 } @@ -304,8 +304,8 @@ Pre-built dashboards for rippled observability. ```json { - "title": "rippled Node Overview", - "uid": "rippled-node-overview", + "title": "xrpld Node Overview", + "uid": "xrpld-node-overview", "panels": [ { "title": "Active Nodes", @@ -314,7 +314,7 @@ Pre-built dashboards for rippled observability. "targets": [ { "queryType": "traceql", - "query": "{resource.service.name=\"rippled\"} | count_over_time() by (resource.service.instance.id) | count()" + "query": "{resource.service.name=\"xrpld\"} | count_over_time() by (resource.service.instance.id) | count()" } ], "gridPos": { "h": 4, "w": 4, "x": 0, "y": 0 } @@ -326,7 +326,7 @@ Pre-built dashboards for rippled observability. "targets": [ { "queryType": "traceql", - "query": "{resource.service.name=\"rippled\" && name=\"tx.receive\"} | count()" + "query": "{resource.service.name=\"xrpld\" && name=\"tx.receive\"} | count()" } ], "gridPos": { "h": 4, "w": 4, "x": 4, "y": 0 } @@ -338,7 +338,7 @@ Pre-built dashboards for rippled observability. "targets": [ { "queryType": "traceql", - "query": "{resource.service.name=\"rippled\" && status.code=error} | rate() / {resource.service.name=\"rippled\"} | rate() * 100" + "query": "{resource.service.name=\"xrpld\" && status.code=error} | rate() / {resource.service.name=\"xrpld\"} | rate() * 100" } ], "fieldConfig": { @@ -373,8 +373,8 @@ Pre-built dashboards for rippled observability. apiVersion: 1 groups: - - name: rippled-tracing-alerts - folder: rippled + - name: xrpld-tracing-alerts + folder: xrpld interval: 1m rules: - uid: consensus-slow @@ -385,7 +385,7 @@ groups: datasourceUid: tempo model: queryType: traceql - query: '{resource.service.name="rippled" && name="consensus.round"} | avg(duration) > 5s' + query: '{resource.service.name="xrpld" && name="consensus.round"} | avg(duration) > 5s' # Note: Verify TraceQL aggregate queries are supported by your # Tempo version. Aggregate alerting (e.g., avg(duration)) requires # Tempo 2.3+ with TraceQL metrics enabled. @@ -404,7 +404,7 @@ groups: datasourceUid: tempo model: queryType: traceql - query: '{resource.service.name="rippled" && name=~"rpc.command.*" && status.code=error} | rate() > 0.05' + query: '{resource.service.name="xrpld" && name=~"rpc.command.*" && status.code=error} | rate() > 0.05' # Note: Verify TraceQL aggregate queries are supported by your # Tempo version. Aggregate alerting (e.g., rate()) requires # Tempo 2.3+ with TraceQL metrics enabled. @@ -422,7 +422,7 @@ groups: datasourceUid: tempo model: queryType: traceql - query: '{resource.service.name="rippled" && name="tx.receive"} | rate() < 10' + query: '{resource.service.name="xrpld" && name="tx.receive"} | rate() < 10' for: 10m annotations: summary: Transaction throughput below threshold @@ -436,13 +436,13 @@ groups: > **OTLP** = OpenTelemetry Protocol -How to correlate OpenTelemetry traces with existing rippled observability. +How to correlate OpenTelemetry traces with existing xrpld observability. ### 7.7.1 Correlation Architecture ```mermaid flowchart TB - subgraph rippled["rippled Node"] + subgraph xrpld["xrpld Node"] otel[OpenTelemetry
Spans] perflog[PerfLog
JSON Logs] insight[Beast Insight
StatsD Metrics] @@ -479,7 +479,7 @@ flowchart TB logs --> corr metrics --> corr - style rippled fill:#0d47a1,stroke:#082f6a,color:#fff + style xrpld fill:#0d47a1,stroke:#082f6a,color:#fff style collectors fill:#bf360c,stroke:#8c2809,color:#fff style storage fill:#1b5e20,stroke:#0d3d14,color:#fff style grafana fill:#4a148c,stroke:#2e0d57,color:#fff @@ -500,7 +500,7 @@ flowchart TB **Reading the diagram:** -- **rippled Node (three sources)**: A single node emits three independent data streams -- OpenTelemetry spans, PerfLog JSON logs, and Beast Insight StatsD metrics. +- **xrpld Node (three sources)**: A single node emits three independent data streams -- OpenTelemetry spans, PerfLog JSON logs, and Beast Insight StatsD metrics. - **Data Collection layer**: Each stream has its own collector -- OTel Collector for spans, Promtail/Fluentd for logs, and a StatsD exporter for metrics. They operate independently. - **Storage layer (Tempo, Loki, Prometheus)**: Each data type lands in a purpose-built store optimized for its query patterns (trace search, log grep, metric aggregation). - **Grafana Correlation Panel**: The key integration point -- Grafana queries all three stores and links them via shared fields (`trace_id`, `xrpl.tx.hash`, `ledger_seq`), enabling a single-pane debugging experience. @@ -522,7 +522,7 @@ flowchart TB ``` # In Grafana Explore with Tempo -{resource.service.name="rippled" && span.xrpl.tx.hash="ABC123..."} +{resource.service.name="xrpld" && span.xrpl.tx.hash="ABC123..."} ``` **Step 2: Get the trace_id from the trace view** @@ -535,14 +535,14 @@ Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736 ``` # In Grafana Explore with Loki -{job="rippled"} |= "4bf92f3577b34da6a3ce929d0e0e4736" +{job="xrpld"} |= "4bf92f3577b34da6a3ce929d0e0e4736" ``` **Step 4: Check Insight metrics for the time window** ``` # In Grafana with Prometheus -rate(rippled_tx_applied_total[1m]) +rate(xrpld_tx_applied_total[1m]) @ timestamp_from_trace ``` @@ -550,8 +550,8 @@ rate(rippled_tx_applied_total[1m]) ```json { - "title": "rippled Unified Observability", - "uid": "rippled-unified", + "title": "xrpld Unified Observability", + "uid": "xrpld-unified", "panels": [ { "title": "Transaction Latency (Traces)", @@ -560,7 +560,7 @@ rate(rippled_tx_applied_total[1m]) "targets": [ { "queryType": "traceql", - "query": "{resource.service.name=\"rippled\" && name=\"tx.receive\"} | histogram_over_time(duration)" + "query": "{resource.service.name=\"xrpld\" && name=\"tx.receive\"} | histogram_over_time(duration)" } ], "gridPos": { "h": 6, "w": 8, "x": 0, "y": 0 } @@ -571,7 +571,7 @@ rate(rippled_tx_applied_total[1m]) "datasource": "Prometheus", "targets": [ { - "expr": "rate(rippled_tx_received_total[5m])", + "expr": "rate(xrpld_tx_received_total[5m])", "legendFormat": "{{ instance }}" } ], @@ -580,7 +580,7 @@ rate(rippled_tx_applied_total[1m]) "links": [ { "title": "View traces", - "url": "/explore?left={\"datasource\":\"Tempo\",\"query\":\"{resource.service.name=\\\"rippled\\\" && name=\\\"tx.receive\\\"}\"}" + "url": "/explore?left={\"datasource\":\"Tempo\",\"query\":\"{resource.service.name=\\\"xrpld\\\" && name=\\\"tx.receive\\\"}\"}" } ] } @@ -593,7 +593,7 @@ rate(rippled_tx_applied_total[1m]) "datasource": "Loki", "targets": [ { - "expr": "{job=\"rippled\"} | json" + "expr": "{job=\"xrpld\"} | json" } ], "gridPos": { "h": 6, "w": 8, "x": 16, "y": 0 } @@ -605,7 +605,7 @@ rate(rippled_tx_applied_total[1m]) "targets": [ { "queryType": "traceql", - "query": "{resource.service.name=\"rippled\"}" + "query": "{resource.service.name=\"xrpld\"}" } ], "fieldConfig": { @@ -622,7 +622,7 @@ rate(rippled_tx_applied_total[1m]) }, { "title": "View logs", - "url": "/explore?left={\"datasource\":\"Loki\",\"query\":\"{job=\\\"rippled\\\"} |= \\\"${__value.raw}\\\"\"}" + "url": "/explore?left={\"datasource\":\"Loki\",\"query\":\"{job=\\\"xrpld\\\"} |= \\\"${__value.raw}\\\"\"}" } ] } diff --git a/OpenTelemetryPlan/08-appendix.md b/OpenTelemetryPlan/08-appendix.md index 2e3d2f5d72..33ec8d3e02 100644 --- a/OpenTelemetryPlan/08-appendix.md +++ b/OpenTelemetryPlan/08-appendix.md @@ -26,7 +26,7 @@ | **Resource** | Entity producing telemetry (service, host, etc.) | | **Instrumentation** | Code that creates telemetry data | -### rippled-Specific Terms +### xrpld-Specific Terms | Term | Definition | | ----------------- | ------------------------------------------------------------- | @@ -36,8 +36,8 @@ | **Validation** | Validator's signature on a closed ledger | | **HashRouter** | Component for transaction deduplication | | **JobQueue** | Thread pool for asynchronous task execution | -| **PerfLog** | Existing performance logging system in rippled | -| **Beast Insight** | Existing metrics framework in rippled | +| **PerfLog** | Existing performance logging system in xrpld | +| **Beast Insight** | Existing metrics framework in xrpld | | **PathFinding** | Payment path computation engine for cross-currency payments | | **TxQ** | Transaction queue managing fee-based prioritization | | **LoadManager** | Dynamic fee escalation based on network load | @@ -146,13 +146,13 @@ flowchart TB 6. [W3C Baggage](https://www.w3.org/TR/baggage/) 7. [Protocol Buffers](https://protobuf.dev/) -### rippled Resources +### xrpld Resources -8. [rippled Source Code](https://github.com/XRPLF/rippled) +8. [xrpld Source Code](https://github.com/XRPLF/rippled) 9. [XRP Ledger Documentation](https://xrpl.org/docs/) -10. [rippled Overlay README](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/README.md) -11. [rippled RPC README](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/rpc/README.md) -12. [rippled Consensus README](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/consensus/README.md) +10. [xrpld Overlay README](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/README.md) +11. [xrpld RPC README](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/rpc/README.md) +12. [xrpld Consensus README](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/consensus/README.md) --- @@ -174,11 +174,11 @@ flowchart TB | ---------------------------------------------------------------- | -------------------------------------------- | | [OpenTelemetryPlan.md](./OpenTelemetryPlan.md) | Master overview and executive summary | | [00-tracing-fundamentals.md](./00-tracing-fundamentals.md) | Distributed tracing concepts and OTel primer | -| [01-architecture-analysis.md](./01-architecture-analysis.md) | rippled architecture and trace points | +| [01-architecture-analysis.md](./01-architecture-analysis.md) | xrpld architecture and trace points | | [02-design-decisions.md](./02-design-decisions.md) | SDK selection, exporters, span conventions | | [03-implementation-strategy.md](./03-implementation-strategy.md) | Directory structure, performance analysis | | [04-code-samples.md](./04-code-samples.md) | C++ code examples for all components | -| [05-configuration-reference.md](./05-configuration-reference.md) | rippled config, CMake, Collector configs | +| [05-configuration-reference.md](./05-configuration-reference.md) | xrpld config, CMake, Collector configs | | [06-implementation-phases.md](./06-implementation-phases.md) | Timeline, tasks, risks, success metrics | | [07-observability-backends.md](./07-observability-backends.md) | Backend selection and architecture | | [08-appendix.md](./08-appendix.md) | Glossary, references, version history | diff --git a/OpenTelemetryPlan/OpenTelemetryPlan.md b/OpenTelemetryPlan/OpenTelemetryPlan.md index fb9f037c00..8f7476753b 100644 --- a/OpenTelemetryPlan/OpenTelemetryPlan.md +++ b/OpenTelemetryPlan/OpenTelemetryPlan.md @@ -1,10 +1,10 @@ -# [OpenTelemetry](00-tracing-fundamentals.md) Distributed Tracing Implementation Plan for rippled (xrpld) +# [OpenTelemetry](00-tracing-fundamentals.md) Distributed Tracing Implementation Plan for xrpld (xrpld) ## Executive Summary > **OTLP** = OpenTelemetry Protocol -This document provides a comprehensive implementation plan for integrating OpenTelemetry distributed tracing into the rippled XRP Ledger node software. The plan addresses the unique challenges of a decentralized peer-to-peer system where trace context must propagate across network boundaries between independent nodes. +This document provides a comprehensive implementation plan for integrating OpenTelemetry distributed tracing into the xrpld XRP Ledger node software. The plan addresses the unique challenges of a decentralized peer-to-peer system where trace context must propagate across network boundaries between independent nodes. ### Key Benefits @@ -98,11 +98,11 @@ flowchart TB | Section | Document | Description | | ------- | ---------------------------------------------------------- | ---------------------------------------------------------------------- | | **0** | [Tracing Fundamentals](./00-tracing-fundamentals.md) | Distributed tracing concepts, span relationships, context propagation | -| **1** | [Architecture Analysis](./01-architecture-analysis.md) | rippled component analysis, trace points, instrumentation priorities | +| **1** | [Architecture Analysis](./01-architecture-analysis.md) | xrpld component analysis, trace points, instrumentation priorities | | **2** | [Design Decisions](./02-design-decisions.md) | SDK selection, exporters, span naming, attributes, context propagation | | **3** | [Implementation Strategy](./03-implementation-strategy.md) | Directory structure, key principles, performance optimization | | **4** | [Code Samples](./04-code-samples.md) | C++ implementation examples for core infrastructure and key modules | -| **5** | [Configuration Reference](./05-configuration-reference.md) | rippled config, CMake integration, Collector configurations | +| **5** | [Configuration Reference](./05-configuration-reference.md) | xrpld config, CMake integration, Collector configurations | | **6** | [Implementation Phases](./06-implementation-phases.md) | 5-phase timeline, tasks, risks, success metrics | | **7** | [Observability Backends](./07-observability-backends.md) | Backend selection guide and production architecture | | **8** | [Appendix](./08-appendix.md) | Glossary, references, version history | @@ -112,7 +112,7 @@ flowchart TB ## 0. Tracing Fundamentals -This document introduces distributed tracing concepts for readers unfamiliar with the domain. It covers what traces and spans are, how parent-child and follows-from relationships model causality, how context propagates across service boundaries, and how sampling controls data volume. It also maps these concepts to rippled-specific scenarios like transaction relay and consensus. +This document introduces distributed tracing concepts for readers unfamiliar with the domain. It covers what traces and spans are, how parent-child and follows-from relationships model causality, how context propagates across service boundaries, and how sampling controls data volume. It also maps these concepts to xrpld-specific scenarios like transaction relay and consensus. ➡️ **[Read Tracing Fundamentals](./00-tracing-fundamentals.md)** @@ -122,7 +122,7 @@ This document introduces distributed tracing concepts for readers unfamiliar wit > **WS** = WebSocket | **TxQ** = Transaction Queue -The rippled node consists of several key components that require instrumentation for comprehensive distributed tracing. The main areas include the RPC server (HTTP/WebSocket), Overlay P2P network, Consensus mechanism (RCLConsensus), JobQueue for async task execution, PathFinding, Transaction Queue (TxQ), fee escalation (LoadManager), ledger acquisition, validator management, and existing observability infrastructure (PerfLog, Insight/StatsD, Journal logging). +The xrpld node consists of several key components that require instrumentation for comprehensive distributed tracing. The main areas include the RPC server (HTTP/WebSocket), Overlay P2P network, Consensus mechanism (RCLConsensus), JobQueue for async task execution, PathFinding, Transaction Queue (TxQ), fee escalation (LoadManager), ledger acquisition, validator management, and existing observability infrastructure (PerfLog, Insight/StatsD, Journal logging). Key trace points span across transaction submission via RPC, peer-to-peer message propagation, consensus round execution, ledger building, path computation, transaction queue behavior, fee escalation, and validator health. The implementation prioritizes high-value, low-risk components first: RPC handlers provide immediate value with minimal risk, while consensus tracing requires careful implementation to avoid timing impacts. @@ -213,7 +213,7 @@ The recommended production architecture uses a gateway collector pattern with re ## 8. Appendix -The appendix contains a glossary of OpenTelemetry and rippled-specific terms, references to external documentation and specifications, version history for this implementation plan, and a complete document index. +The appendix contains a glossary of OpenTelemetry and xrpld-specific terms, references to external documentation and specifications, version history for this implementation plan, and a complete document index. ➡️ **[View Appendix](./08-appendix.md)** @@ -221,10 +221,10 @@ The appendix contains a glossary of OpenTelemetry and rippled-specific terms, re ## POC Task List -A step-by-step task list for building a minimal end-to-end proof of concept that demonstrates distributed tracing in rippled. The POC scope is limited to RPC tracing — showing request traces flowing from rippled through an OpenTelemetry Collector into Tempo, viewable in Grafana. +A step-by-step task list for building a minimal end-to-end proof of concept that demonstrates distributed tracing in xrpld. The POC scope is limited to RPC tracing — showing request traces flowing from xrpld through an OpenTelemetry Collector into Tempo, viewable in Grafana. ➡️ **[View POC Task List](./POC_taskList.md)** --- -_This document provides a comprehensive implementation plan for integrating OpenTelemetry distributed tracing into the rippled XRP Ledger node software. For detailed information on any section, follow the links to the corresponding sub-documents._ +_This document provides a comprehensive implementation plan for integrating OpenTelemetry distributed tracing into the xrpld XRP Ledger node software. For detailed information on any section, follow the links to the corresponding sub-documents._ diff --git a/OpenTelemetryPlan/POC_taskList.md b/OpenTelemetryPlan/POC_taskList.md index decb9f1738..cab5e365fe 100644 --- a/OpenTelemetryPlan/POC_taskList.md +++ b/OpenTelemetryPlan/POC_taskList.md @@ -1,21 +1,21 @@ # OpenTelemetry POC Task List -> **Goal**: Build a minimal end-to-end proof of concept that demonstrates distributed tracing in rippled. A successful POC will show RPC request traces flowing from rippled through an OTel Collector into Tempo, viewable in Grafana. +> **Goal**: Build a minimal end-to-end proof of concept that demonstrates distributed tracing in xrpld. A successful POC will show RPC request traces flowing from xrpld through an OTel Collector into Tempo, viewable in Grafana. > > **Scope**: RPC tracing only (highest value, lowest risk per the [CRAWL phase](./06-implementation-phases.md#6102-quick-wins-immediate-value) in the implementation phases). No cross-node P2P context propagation or consensus tracing in the POC. ### Related Plan Documents -| Document | Relevance to POC | -| ---------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | -| [00-tracing-fundamentals.md](./00-tracing-fundamentals.md) | Core concepts: traces, spans, context propagation, sampling | -| [01-architecture-analysis.md](./01-architecture-analysis.md) | RPC request flow (§1.5), key trace points (§1.6), instrumentation priority (§1.7) | -| [02-design-decisions.md](./02-design-decisions.md) | SDK selection (§2.1), exporter config (§2.2), span naming (§2.3), attribute schema (§2.4), coexistence with PerfLog/Insight (§2.6) | -| [03-implementation-strategy.md](./03-implementation-strategy.md) | Directory structure (§3.1), key principles (§3.2), performance overhead (§3.3-3.6), conditional compilation (§3.7.3), code intrusiveness (§3.9) | -| [04-code-samples.md](./04-code-samples.md) | Telemetry interface (§4.1), SpanGuard (§4.2), macros (§4.3), RPC instrumentation (§4.5.3) | -| [05-configuration-reference.md](./05-configuration-reference.md) | rippled config (§5.1), config parser (§5.2), Application integration (§5.3), CMake (§5.4), Collector config (§5.5), Docker Compose (§5.6), Grafana (§5.8) | -| [06-implementation-phases.md](./06-implementation-phases.md) | Phase 1 core tasks (§6.2), Phase 2 RPC tasks (§6.3), quick wins (§6.10), definition of done (§6.11) | -| [07-observability-backends.md](./07-observability-backends.md) | Tempo dev setup (§7.1), Grafana dashboards (§7.6), alert rules (§7.6.3) | +| Document | Relevance to POC | +| ---------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | +| [00-tracing-fundamentals.md](./00-tracing-fundamentals.md) | Core concepts: traces, spans, context propagation, sampling | +| [01-architecture-analysis.md](./01-architecture-analysis.md) | RPC request flow (§1.5), key trace points (§1.6), instrumentation priority (§1.7) | +| [02-design-decisions.md](./02-design-decisions.md) | SDK selection (§2.1), exporter config (§2.2), span naming (§2.3), attribute schema (§2.4), coexistence with PerfLog/Insight (§2.6) | +| [03-implementation-strategy.md](./03-implementation-strategy.md) | Directory structure (§3.1), key principles (§3.2), performance overhead (§3.3-3.6), conditional compilation (§3.7.3), code intrusiveness (§3.9) | +| [04-code-samples.md](./04-code-samples.md) | Telemetry interface (§4.1), SpanGuard (§4.2), macros (§4.3), RPC instrumentation (§4.5.3) | +| [05-configuration-reference.md](./05-configuration-reference.md) | xrpld config (§5.1), config parser (§5.2), Application integration (§5.3), CMake (§5.4), Collector config (§5.5), Docker Compose (§5.6), Grafana (§5.8) | +| [06-implementation-phases.md](./06-implementation-phases.md) | Phase 1 core tasks (§6.2), Phase 2 RPC tasks (§6.3), quick wins (§6.10), definition of done (§6.11) | +| [07-observability-backends.md](./07-observability-backends.md) | Tempo dev setup (§7.1), Grafana dashboards (§7.6), alert rules (§7.6.3) | --- @@ -137,7 +137,7 @@ - `virtual void start() = 0;` - `virtual void stop() = 0;` - `virtual bool isEnabled() const = 0;` - - `virtual nostd::shared_ptr getTracer(string_view name = "rippled") = 0;` + - `virtual nostd::shared_ptr getTracer(string_view name = "xrpld") = 0;` - `virtual nostd::shared_ptr startSpan(string_view name, SpanKind kind = kInternal) = 0;` - `virtual nostd::shared_ptr startSpan(string_view name, Context const& parentContext, SpanKind kind = kInternal) = 0;` - `virtual bool shouldTraceRpc() const = 0;` @@ -418,7 +418,7 @@ > **OTLP** = OpenTelemetry Protocol -**Objective**: Prove the full pipeline works: rippled emits traces -> OTel Collector receives them -> Tempo stores them for Grafana visualization. +**Objective**: Prove the full pipeline works: xrpld emits traces -> OTel Collector receives them -> Tempo stores them for Grafana visualization. **What to do**: @@ -430,7 +430,7 @@ Verify Collector health: `curl http://localhost:13133` -2. **Build rippled with telemetry**: +2. **Build xrpld with telemetry**: ```bash # Adjust for your actual build workflow @@ -439,8 +439,8 @@ cmake --build --preset default ``` -3. **Configure rippled**: - Add to `rippled.cfg` (or your local test config): +3. **Configure xrpld**: + Add to `xrpld.cfg` (or your local test config): ```ini [telemetry] @@ -450,10 +450,10 @@ trace_rpc=1 ``` -4. **Start rippled** in standalone mode: +4. **Start xrpld** in standalone mode: ```bash - ./rippled --conf rippled.cfg -a --start + ./rippled --conf xrpld.cfg -a --start ``` 5. **Generate RPC traffic**: @@ -478,21 +478,21 @@ 6. **Verify in Grafana (Tempo)**: - Open `http://localhost:3000` - Navigate to Explore → select Tempo datasource - - Search for service `rippled` + - Search for service `xrpld` - Confirm you see traces with spans: `rpc.request` -> `rpc.process` -> `rpc.command.server_info` - Click into a trace and verify attributes: `xrpl.rpc.command`, `xrpl.rpc.status`, `xrpl.rpc.version` 7. **Verify zero-overhead when disabled**: - Rebuild with `XRPL_ENABLE_TELEMETRY=OFF`, or set `enabled=0` in config - Run the same RPC calls - - Confirm no new traces appear and no errors in rippled logs + - Confirm no new traces appear and no errors in xrpld logs **Verification Checklist**: - [ ] Docker stack starts without errors -- [ ] rippled builds with `-DXRPL_ENABLE_TELEMETRY=ON` -- [ ] rippled starts and connects to OTel Collector (check rippled logs for telemetry messages) -- [ ] Traces appear in Grafana/Tempo under service "rippled" +- [ ] xrpld builds with `-DXRPL_ENABLE_TELEMETRY=ON` +- [ ] xrpld starts and connects to OTel Collector (check xrpld logs for telemetry messages) +- [ ] Traces appear in Grafana/Tempo under service "xrpld" - [ ] Span hierarchy is correct (parent-child relationships) - [ ] Span attributes are populated (`xrpl.rpc.command`, `xrpl.rpc.status`, etc.) - [ ] Error spans show error status and message @@ -518,13 +518,13 @@ **What to do**: - Take screenshots of Grafana/Tempo showing: - - The service list with "rippled" + - The service list with "xrpld" - A trace with the full span tree - Span detail view showing attributes - Document any issues encountered (build issues, SDK quirks, missing attributes) - Note performance observations (build time impact, any noticeable runtime overhead) - Write a short summary of what the POC proves and what it doesn't cover yet: - - **Proves**: OTel SDK integrates with rippled, OTLP export works, RPC traces visible + - **Proves**: OTel SDK integrates with xrpld, OTLP export works, RPC traces visible - **Doesn't cover**: Cross-node P2P context propagation, consensus tracing, protobuf trace context, W3C traceparent header extraction, tail-based sampling, production deployment - Outline next steps (mapping to the full plan phases): - [Phase 2](./06-implementation-phases.md) completion: [W3C header extraction](./02-design-decisions.md) (§2.5), WebSocket tracing, all [RPC handlers](./01-architecture-analysis.md) (§1.6) diff --git a/OpenTelemetryPlan/presentation.md b/OpenTelemetryPlan/presentation.md index 799accda86..479aa8fa55 100644 --- a/OpenTelemetryPlan/presentation.md +++ b/OpenTelemetryPlan/presentation.md @@ -1,4 +1,4 @@ -# OpenTelemetry Distributed Tracing for rippled +# OpenTelemetry Distributed Tracing for xrpld --- @@ -10,7 +10,7 @@ OpenTelemetry is an open-source, CNCF-backed observability framework for distributed tracing, metrics, and logs. -### Why OpenTelemetry for rippled? +### Why OpenTelemetry for xrpld? - **End-to-End Transaction Visibility**: Track transactions from submission → consensus → ledger inclusion - **Cross-Node Correlation**: Follow requests across multiple independent nodes using a unique `trace_id` @@ -59,13 +59,13 @@ flowchart LR ## Slide 3: Adoption Scope — Traces Only (Current Plan) -OpenTelemetry supports three signal types: **Traces**, **Metrics**, and **Logs**. rippled already captures metrics (StatsD via Beast Insight) and logs (Journal/PerfLog). The question is: how much of OTel do we adopt? +OpenTelemetry supports three signal types: **Traces**, **Metrics**, and **Logs**. xrpld already captures metrics (StatsD via Beast Insight) and logs (Journal/PerfLog). The question is: how much of OTel do we adopt? > **Scenario A**: Add distributed tracing. Keep StatsD for metrics and Journal for logs. ```mermaid flowchart LR - subgraph rippled["rippled Process"] + subgraph xrpld["xrpld Process"] direction TB OTel["OTel SDK
(Traces)"] Insight["Beast Insight
(StatsD Metrics)"] @@ -80,7 +80,7 @@ flowchart LR StatsD --> Graphite["Graphite / Grafana"] LogFile --> Loki["Loki (optional)"] - style rippled fill:#424242,stroke:#212121,color:#fff + style xrpld fill:#424242,stroke:#212121,color:#fff style OTel fill:#2e7d32,stroke:#1b5e20,color:#fff style Insight fill:#1565c0,stroke:#0d47a1,color:#fff style Journal fill:#e65100,stroke:#bf360c,color:#fff @@ -106,7 +106,7 @@ flowchart LR ```mermaid flowchart LR - subgraph rippled["rippled Process"] + subgraph xrpld["xrpld Process"] direction TB OTel["OTel SDK
(Traces + Metrics)"] Journal["Journal + PerfLog
(Logging)"] @@ -119,7 +119,7 @@ flowchart LR Collector --> Prom["Prometheus
(Metrics)"] LogFile --> Loki["Loki (optional)"] - style rippled fill:#424242,stroke:#212121,color:#fff + style xrpld fill:#424242,stroke:#212121,color:#fff style OTel fill:#2e7d32,stroke:#1b5e20,color:#fff style Journal fill:#e65100,stroke:#bf360c,color:#fff style Collector fill:#2e7d32,stroke:#1b5e20,color:#fff @@ -136,7 +136,7 @@ flowchart LR ```mermaid flowchart LR - subgraph rippled["rippled Process"] + subgraph xrpld["xrpld Process"] OTel["OTel SDK
(Traces + Metrics + Logs)"] end @@ -146,7 +146,7 @@ flowchart LR Collector --> Prom["Prometheus
(Metrics)"] Collector --> Loki["Loki / Elastic
(Logs)"] - style rippled fill:#424242,stroke:#212121,color:#fff + style xrpld fill:#424242,stroke:#212121,color:#fff style OTel fill:#2e7d32,stroke:#1b5e20,color:#fff style Collector fill:#2e7d32,stroke:#1b5e20,color:#fff ``` @@ -177,7 +177,7 @@ flowchart LR --- -## Slide 5: Comparison with rippled's Existing Solutions +## Slide 5: Comparison with xrpld's Existing Solutions ### Current Observability Stack @@ -211,7 +211,7 @@ flowchart LR ```mermaid flowchart TB - subgraph rippled["rippled Node"] + subgraph xrpld["xrpld Node"] subgraph services["Core Services"] direction LR RPC["RPC Server
(HTTP/WS)"] ~~~ Overlay["Overlay
(P2P Network)"] ~~~ Consensus["Consensus
(RCLConsensus)"] @@ -227,7 +227,7 @@ flowchart TB Collector --> Tempo["Grafana Tempo"] Collector --> Elastic["Elastic APM"] - style rippled fill:#424242,stroke:#212121,color:#fff + style xrpld fill:#424242,stroke:#212121,color:#fff style services fill:#1565c0,stroke:#0d47a1,color:#fff style Telemetry fill:#2e7d32,stroke:#1b5e20,color:#fff style Collector fill:#e65100,stroke:#bf360c,color:#fff @@ -236,9 +236,9 @@ flowchart TB **Reading the diagram:** - **Core Services (blue, top)**: RPC Server, Overlay, and Consensus are the three primary components that generate trace data — they represent the entry points for client requests, peer messages, and consensus rounds respectively. -- **Telemetry Module (green, middle)**: The OpenTelemetry SDK sits below the core services and receives span data from all three; it acts as a single collection point within the rippled process. -- **OTel Collector (orange, center)**: An external process that receives spans over OTLP/gRPC from the Telemetry Module; it decouples rippled from backend choices and handles batching, sampling, and routing. -- **Backends (bottom row)**: Tempo and Elastic APM are interchangeable — the Collector fans out to any combination, so operators can switch backends without modifying rippled code. +- **Telemetry Module (green, middle)**: The OpenTelemetry SDK sits below the core services and receives span data from all three; it acts as a single collection point within the xrpld process. +- **OTel Collector (orange, center)**: An external process that receives spans over OTLP/gRPC from the Telemetry Module; it decouples xrpld from backend choices and handles batching, sampling, and routing. +- **Backends (bottom row)**: Tempo and Elastic APM are interchangeable — the Collector fans out to any combination, so operators can switch backends without modifying xrpld code. - **Top-to-bottom flow**: Data flows from instrumented code down through the SDK, out over the network to the Collector, and finally into storage/visualization backends. ### Context Propagation @@ -496,7 +496,7 @@ flowchart LR | Aspect | Details | | ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Where it runs** | Inside rippled (SDK-level). Configured via `sampling_ratio` in `rippled.cfg`. | +| **Where it runs** | Inside xrpld (SDK-level). Configured via `sampling_ratio` in `xrpld.cfg`. | | **When the decision happens** | At trace creation time — before the first span is even populated. | | **How it works** | `sampling_ratio=0.1` means each trace has a 10% probability of being recorded. Dropped traces incur near-zero overhead (no spans created, no attributes set, no export). | | **Propagation** | Once a trace is sampled, the `trace_flags` field (1 byte in the context header) tells downstream nodes to also sample it. Unsampled traces propagate `trace_flags=0`, so downstream nodes skip them too. | @@ -504,7 +504,7 @@ flowchart LR | **Cons** | **Blind** — it doesn't know if the trace will be interesting. A rare error or slow consensus round has only a 10% chance of being captured. | | **Best for** | High-volume, steady-state traffic where most traces look similar (e.g., routine RPC requests). | -**rippled configuration**: +**xrpld configuration**: ```ini [telemetry] @@ -538,16 +538,16 @@ flowchart TB style E fill:#4a148c,stroke:#2e0d57,color:#fff ``` -| Aspect | Details | -| ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Where it runs** | In the **OTel Collector** (external process), not inside rippled. rippled exports 100% of traces; the Collector decides what to keep. | -| **When the decision happens** | After the Collector has received all spans for a trace (waits `decision_wait=10s` for stragglers). | -| **How it works** | Policy rules evaluate the completed trace: keep all errors, keep slow operations above a threshold, keep all consensus rounds, then probabilistically sample the rest at 10%. | -| **Pros** | **Never misses important traces**. Errors, slow requests, and consensus anomalies are always captured regardless of probability. | -| **Cons** | Higher resource usage — rippled must export 100% of spans to the Collector, which buffers them in memory before deciding. The Collector needs more RAM (configured via `num_traces` and `decision_wait`). | -| **Best for** | Production troubleshooting where you can't afford to miss errors or anomalies. | +| Aspect | Details | +| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Where it runs** | In the **OTel Collector** (external process), not inside xrpld. xrpld exports 100% of traces; the Collector decides what to keep. | +| **When the decision happens** | After the Collector has received all spans for a trace (waits `decision_wait=10s` for stragglers). | +| **How it works** | Policy rules evaluate the completed trace: keep all errors, keep slow operations above a threshold, keep all consensus rounds, then probabilistically sample the rest at 10%. | +| **Pros** | **Never misses important traces**. Errors, slow requests, and consensus anomalies are always captured regardless of probability. | +| **Cons** | Higher resource usage — xrpld must export 100% of spans to the Collector, which buffers them in memory before deciding. The Collector needs more RAM (configured via `num_traces` and `decision_wait`). | +| **Best for** | Production troubleshooting where you can't afford to miss errors or anomalies. | -**Collector configuration** (tail sampling rules for rippled): +**Collector configuration** (tail sampling rules for xrpld): ```yaml processors: @@ -576,22 +576,22 @@ processors: | | Head Sampling | Tail Sampling | | ----------------------------- | ---------------------------------------- | ------------------------------------------------ | -| **Decision point** | Trace start (inside rippled) | Trace end (in OTel Collector) | +| **Decision point** | Trace start (inside xrpld) | Trace end (in OTel Collector) | | **Knows trace content?** | No (random coin flip) | Yes (evaluates completed trace) | -| **Overhead on rippled** | Lowest (dropped traces = no-op) | Higher (must export 100% to Collector) | +| **Overhead on xrpld** | Lowest (dropped traces = no-op) | Higher (must export 100% to Collector) | | **Collector resource usage** | Low (receives only sampled traces) | Higher (buffers all traces before deciding) | | **Captures all errors?** | No (only if trace was randomly selected) | **Yes** (error policy catches them) | | **Captures slow operations?** | No (random) | **Yes** (latency policy catches them) | -| **Configuration** | `rippled.cfg`: `sampling_ratio=0.1` | `otel-collector.yaml`: `tail_sampling` processor | +| **Configuration** | `xrpld.cfg`: `sampling_ratio=0.1` | `otel-collector.yaml`: `tail_sampling` processor | | **Best for** | High-throughput steady-state | Troubleshooting & anomaly detection | -### Recommended Strategy for rippled +### Recommended Strategy for xrpld Use **both** in a layered approach: ```mermaid flowchart LR - subgraph rippled["rippled (Head Sampling)"] + subgraph xrpld["xrpld (Head Sampling)"] HS["sampling_ratio=1.0
(export everything)"] end @@ -603,14 +603,14 @@ flowchart LR ST["Only interesting traces
stored long-term"] end - rippled -->|"100% of spans"| collector -->|"~15-20% kept"| storage + xrpld -->|"100% of spans"| collector -->|"~15-20% kept"| storage - style rippled fill:#424242,stroke:#212121,color:#fff + style xrpld fill:#424242,stroke:#212121,color:#fff style collector fill:#1565c0,stroke:#0d47a1,color:#fff style storage fill:#2e7d32,stroke:#1b5e20,color:#fff ``` -> **Why this works**: rippled exports everything (no blind drops), the Collector applies intelligent filtering (keep errors/slow/anomalies, sample the rest), and only ~15-20% of traces reach storage. If Collector resource usage becomes a concern, add head sampling at `sampling_ratio=0.5` to halve the export volume while still giving the Collector enough data for good tail-sampling decisions. +> **Why this works**: xrpld exports everything (no blind drops), the Collector applies intelligent filtering (keep errors/slow/anomalies, sample the rest), and only ~15-20% of traces reach storage. If Collector resource usage becomes a concern, add head sampling at `sampling_ratio=0.5` to halve the export volume while still giving the Collector enough data for good tail-sampling decisions. ---