ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-06-02 16:26:48 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment

Browse Source
This commit is contained in:
Pratik Mankawde
2026-05-29 17:53:29 +01:00
parent c3c980e858 a44d91ec27
commit e5fae351d6
1646 changed files with 79758 additions and 65546 deletions
Download Patch File Download Diff File Expand all files Collapse all files

30
OpenTelemetryPlan/02-design-decisions.md
View File

@@ -362,20 +362,20 @@ resource::SemanticConventions::SERVICE_INSTANCE_ID = <node_public_key_base58>
The following table summarizes what data is collected by category:
| Category | Attributes Collected | Purpose |
| --------------- | ---------------------------------------------------------------------- | ---------------------------- |
| **Transaction** | `tx.hash`, `tx.type`, `tx.result`, `tx.fee`, `ledger_index` | Trace transaction lifecycle |
| **Consensus** | `round`, `phase`, `mode`, `proposers` (public keys), `duration_ms` | Analyze consensus timing |
| **RPC** | `command`, `version`, `status`, `duration_ms` | Monitor RPC performance |
| **Peer** | `peer.id` (public key), `latency_ms`, `message.type`, `message.size` | Network topology analysis |
| **Ledger** | `ledger.hash`, `ledger.index`, `close_time`, `tx_count` | Ledger progression tracking |
| **Job** | `job.type`, `queue_ms`, `worker` | JobQueue performance |
| **PathFinding** | `pathfind.source_currency`, `dest_currency`, `path_count`, `cache_hit` | Payment path analysis |
| **TxQ** | `txq.queue_depth`, `fee_level`, `eviction_reason` | Queue depth and fee tracking |
| **Fee** | `fee.load_factor`, `escalation_level` | Fee escalation monitoring |
| **Validator** | `validator.list_size`, `list_age_sec` | UNL health monitoring |
| **Amendment** | `amendment.name`, `status` | Protocol upgrade tracking |
| **SHAMap** | `shamap.type`, `missing_nodes`, `duration_ms` | State tree sync performance |
| Category | Attributes Collected | Purpose |
| --------------- | ---------------------------------------------------------------------------------------------------------------- | ---------------------------- |
| **Transaction** | `tx.hash`, `tx.type`, `tx.result`, `tx.fee`, `ledger_index` | Trace transaction lifecycle |
| **Consensus** | `round`, `phase`, `mode`, `proposers` (public keys), `duration_ms` | Analyze consensus timing |
| **RPC** | `command`, `version`, `status`, `duration_ms` | Monitor RPC performance |
| **Peer** | `peer.id` (public key), `latency_ms`, `message.type`, `message.size` | Network topology analysis |
| **Ledger** | `ledger.hash`, `ledger.index`, `close_time`, `tx_count` | Ledger progression tracking |
| **Job** | `job.type`, `queue_ms`, `worker` | JobQueue performance |
| **PathFinding** | `pathfind_fast`, `pathfind_search_level`, `pathfind_num_paths`, `pathfind_ledger_index`, `pathfind_num_requests` | Payment path analysis |
| **TxQ** | `txq.queue_depth`, `fee_level`, `eviction_reason` | Queue depth and fee tracking |
| **Fee** | `fee.load_factor`, `escalation_level` | Fee escalation monitoring |
| **Validator** | `validator.list_size`, `list_age_sec` | UNL health monitoring |
| **Amendment** | `amendment.name`, `status` | Protocol upgrade tracking |
| **SHAMap** | `shamap.type`, `missing_nodes`, `duration_ms` | State tree sync performance |
### 2.4.4 Privacy & Sensitive Data Policy
@@ -449,6 +449,8 @@ redact_peer_address=1 # Remove peer IP addresses
> **Key Principle**: Telemetry collects **operational metadata** (timing, counts, hashes) — never **sensitive content** (keys, balances, amounts, raw payloads).
> **See also**: [Securing the OTel Pipeline](./secure-OTel.md) covers transport-level protection for telemetry leaving the node — mTLS to the collector and validation of incoming peer trace context. Privacy controls in this section keep sensitive data out of spans; the security doc keeps the spans themselves out of untrusted hands.
---
## 2.5 Context Propagation Design

4
OpenTelemetryPlan/04-code-samples.md
View File

@@ -165,12 +165,12 @@ public:
// Factory functions
std::unique_ptr<Telemetry>
make_Telemetry(
makeTelemetry(
Telemetry::Setup const& setup,
beast::Journal journal);
Telemetry::Setup
setup_Telemetry(
setupTelemetry(
Section const& section,
std::string const& nodePublicKey,
std::string const& version);

14
OpenTelemetryPlan/05-configuration-reference.md
View File

@@ -128,7 +128,7 @@ namespace xrpl {
namespace telemetry {
Telemetry::Setup
setup_Telemetry(
setupTelemetry(
Section const& section,
std::string const& nodePublicKey,
std::string const& version)
@@ -213,8 +213,8 @@ class ApplicationImp : public Application, public BasicApp
// Member initializer list (excerpt):
// ...
// , telemetry_(
// telemetry::make_Telemetry(
// telemetry::setup_Telemetry(
// telemetry::makeTelemetry(
// telemetry::setupTelemetry(
// config_->section("telemetry"),
// "", // Updated later via setServiceInstanceId()
// BuildInfo::getVersionString()),
@@ -405,6 +405,8 @@ endif()
> **OTLP** = OpenTelemetry Protocol | **APM** = Application Performance Monitoring
> **Production hardening**: The configurations in this section are starting points. For production deployments where xrpld ships telemetry across a network to a centrally-hosted collector, see [Securing the OTel Pipeline](./secure-OTel.md) for the required mTLS receiver config, NetworkPolicy, and peer trace-context validation.
### 5.5.1 Development Configuration
```yaml
@@ -627,8 +629,8 @@ flowchart TB
end
subgraph init["Initialization"]
parse["setup_Telemetry()"]
factory["make_Telemetry()"]
parse["setupTelemetry()"]
factory["makeTelemetry()"]
end
subgraph runtime["Runtime Components"]
@@ -661,7 +663,7 @@ flowchart TB
**Reading the diagram:**
- **Configuration Sources**: `xrpld.cfg` provides runtime settings (endpoint, sampling) while the CMake flag controls whether telemetry is compiled in at all.
- **Initialization**: `setup_Telemetry()` parses config values, then `make_Telemetry()` constructs the provider, processor, and exporter objects.
- **Initialization**: `setupTelemetry()` parses config values, then `makeTelemetry()` constructs the provider, processor, and exporter objects.
- **Runtime Components**: The `TracerProvider` creates spans, the `BatchProcessor` buffers them, and the `OTLP Exporter` serializes and sends them over the wire.
- **OTLP arrow to Collector**: Trace data leaves the xrpld process via OTLP (gRPC or HTTP) and enters the external Collector pipeline.
- **Collector Pipeline**: `Receivers` ingest OTLP data, `Processors` apply sampling/filtering/enrichment, and `Exporters` forward traces to storage backends (Tempo, etc.).

8
OpenTelemetryPlan/07-observability-backends.md
View File

@@ -19,10 +19,10 @@
```bash
# Start Tempo with OTLP support
docker run -d --name tempo \
-p 3200:3200 \
-p 4317:4317 \
-p 4318:4318 \
grafana/tempo:2.6.1
-p 3200:3200 \
-p 4317:4317 \
-p 4318:4318 \
grafana/tempo:2.6.1
```
---

27
OpenTelemetryPlan/08-appendix.md
View File

@@ -170,19 +170,20 @@ flowchart TB
### Plan Documents
| Document | Description |
| ---------------------------------------------------------------- | -------------------------------------------- |
| [OpenTelemetryPlan.md](./OpenTelemetryPlan.md) | Master overview and executive summary |
| [00-tracing-fundamentals.md](./00-tracing-fundamentals.md) | Distributed tracing concepts and OTel primer |
| [01-architecture-analysis.md](./01-architecture-analysis.md) | xrpld architecture and trace points |
| [02-design-decisions.md](./02-design-decisions.md) | SDK selection, exporters, span conventions |
| [03-implementation-strategy.md](./03-implementation-strategy.md) | Directory structure, performance analysis |
| [04-code-samples.md](./04-code-samples.md) | C++ code examples for all components |
| [05-configuration-reference.md](./05-configuration-reference.md) | xrpld config, CMake, Collector configs |
| [06-implementation-phases.md](./06-implementation-phases.md) | Timeline, tasks, risks, success metrics |
| [07-observability-backends.md](./07-observability-backends.md) | Backend selection and architecture |
| [08-appendix.md](./08-appendix.md) | Glossary, references, version history |
| [presentation.md](./presentation.md) | Slide deck for OTel plan overview |
| Document | Description |
| ---------------------------------------------------------------- | -------------------------------------------------- |
| [OpenTelemetryPlan.md](./OpenTelemetryPlan.md) | Master overview and executive summary |
| [00-tracing-fundamentals.md](./00-tracing-fundamentals.md) | Distributed tracing concepts and OTel primer |
| [01-architecture-analysis.md](./01-architecture-analysis.md) | xrpld architecture and trace points |
| [02-design-decisions.md](./02-design-decisions.md) | SDK selection, exporters, span conventions |
| [03-implementation-strategy.md](./03-implementation-strategy.md) | Directory structure, performance analysis |
| [04-code-samples.md](./04-code-samples.md) | C++ code examples for all components |
| [05-configuration-reference.md](./05-configuration-reference.md) | xrpld config, CMake, Collector configs |
| [06-implementation-phases.md](./06-implementation-phases.md) | Timeline, tasks, risks, success metrics |
| [07-observability-backends.md](./07-observability-backends.md) | Backend selection and architecture |
| [08-appendix.md](./08-appendix.md) | Glossary, references, version history |
| [secure-OTel.md](./secure-OTel.md) | Threat model and hardening (mTLS, peer validation) |
| [presentation.md](./presentation.md) | Slide deck for OTel plan overview |
### Task Lists

12
OpenTelemetryPlan/OpenTelemetryPlan.md
View File

@@ -54,6 +54,7 @@ flowchart TB
phases["06-implementation-phases.md"]
backends["07-observability-backends.md"]
appendix["08-appendix.md"]
secure["secure-OTel.md"]
poc["POC_taskList.md"]
end
@@ -70,6 +71,7 @@ flowchart TB
config --> phases
phases --> backends
backends --> appendix
backends --> secure
phases --> poc
style overview fill:#1b5e20,stroke:#0d3d14,color:#fff,stroke-width:2px
@@ -86,6 +88,7 @@ flowchart TB
style phases fill:#4a148c,stroke:#2e0d57,color:#fff
style backends fill:#4a148c,stroke:#2e0d57,color:#fff
style appendix fill:#4a148c,stroke:#2e0d57,color:#fff
style secure fill:#4a148c,stroke:#2e0d57,color:#fff
style poc fill:#4a148c,stroke:#2e0d57,color:#fff
```
@@ -106,6 +109,7 @@ flowchart TB
| **6** | [Implementation Phases](./06-implementation-phases.md) | 5-phase timeline, tasks, risks, success metrics |
| **7** | [Observability Backends](./07-observability-backends.md) | Backend selection guide and production architecture |
| **8** | [Appendix](./08-appendix.md) | Glossary, references, version history |
| **Sec** | [Securing the OTel Pipeline](./secure-OTel.md) | Threat model and hardening (mTLS, peer trace-context validation) |
| **POC** | [POC Task List](./POC_taskList.md) | Proof of concept tasks for RPC tracing end-to-end demo |
---
@@ -220,6 +224,14 @@ The appendix contains a glossary of OpenTelemetry and xrpld-specific terms, refe
---
## Securing the OTel Pipeline
Threat model and hardening guidance for production deployments where xrpld nodes ship telemetry to a centrally-hosted collector across an untrusted network. Covers the two attack surfaces (collector ingress and peer trace-context spoofing) and the chosen defenses: mTLS as primary collector auth, NetworkPolicy as defense-in-depth, and source-side validation plus per-peer rate limiting for the `protocol::TraceContext` field on peer messages.
➡️ **[View Securing the OTel Pipeline](./secure-OTel.md)**
---
## POC Task List
A step-by-step task list for building a minimal end-to-end proof of concept that demonstrates distributed tracing in xrpld. The POC scope is limited to RPC tracing — showing request traces flowing from xrpld through an OpenTelemetry Collector into Tempo, viewable in Grafana.

26
OpenTelemetryPlan/POC_taskList.md
View File

@@ -143,8 +143,8 @@
- `virtual bool shouldTraceRpc() const = 0;`
- `virtual bool shouldTraceTransactions() const = 0;`
- `virtual bool shouldTraceConsensus() const = 0;`
- Factory: `std::unique_ptr<Telemetry> make_Telemetry(Setup const&, beast::Journal);`
- Config parser: `Telemetry::Setup setup_Telemetry(Section const&, std::string const& nodePublicKey, std::string const& version);`
- Factory: `std::unique_ptr<Telemetry> makeTelemetry(Setup const&, beast::Journal);`
- Config parser: `Telemetry::Setup setupTelemetry(Section const&, std::string const& nodePublicKey, std::string const& version);`
- Create `include/xrpl/telemetry/SpanGuard.h`:
- RAII guard with static factory methods (`rpcSpan()`, `txSpan()`, `consensusSpan()`, etc.) that access the global `Telemetry::getInstance()` singleton internally.
@@ -196,10 +196,10 @@
- `shouldTraceRpc()` etc. read from `Setup` fields
- Create `src/libxrpl/telemetry/TelemetryConfig.cpp`:
- `setup_Telemetry()` parses the `[telemetry]` config section from `xrpld.cfg`
- `setupTelemetry()` parses the `[telemetry]` config section from `xrpld.cfg`
- Maps config keys: `enabled`, `exporter`, `endpoint`, `sampling_ratio`, `trace_rpc`, `trace_transactions`, `trace_consensus`, `trace_peer`
- Wire `make_Telemetry()` factory:
- Wire `makeTelemetry()` factory:
- If `setup.enabled` is true AND `XRPL_ENABLE_TELEMETRY` is defined: return `TelemetryImpl`
- Otherwise: return `NullTelemetry`
@@ -217,7 +217,7 @@
**Reference**:
- [04-code-samples.md §4.1](./04-code-samples.md) — `Telemetry` interface that `TelemetryImpl` must implement
- [05-configuration-reference.md §5.2](./05-configuration-reference.md) — `setup_Telemetry()` config parser implementation
- [05-configuration-reference.md §5.2](./05-configuration-reference.md) — `setupTelemetry()` config parser implementation
- [02-design-decisions.md §2.2](./02-design-decisions.md) — OTLP/gRPC exporter config (endpoint, TLS options)
- [02-design-decisions.md §2.4.1](./02-design-decisions.md) — Resource attributes: `service.name`, `service.version`, `service.instance.id`, `xrpl.network.id`
- [03-implementation-strategy.md §3.4](./03-implementation-strategy.md) — Per-operation CPU costs and overhead budget for span creation
@@ -242,8 +242,8 @@
`serviceInstanceId` (node identity is not yet known):
```cpp
, telemetry_(
telemetry::make_Telemetry(
telemetry::setup_Telemetry(
telemetry::makeTelemetry(
telemetry::setupTelemetry(
config_->section("telemetry"),
"", // Updated later via setServiceInstanceId()
BuildInfo::getVersionString()),
@@ -453,18 +453,18 @@
```bash
# server_info
curl -s -X POST http://localhost:5005 \
-H "Content-Type: application/json" \
-d '{"method":"server_info","params":[{}]}'
-H "Content-Type: application/json" \
-d '{"method":"server_info","params":[{}]}'
# ledger
curl -s -X POST http://localhost:5005 \
-H "Content-Type: application/json" \
-d '{"method":"ledger","params":[{"ledger_index":"current"}]}'
-H "Content-Type: application/json" \
-d '{"method":"ledger","params":[{"ledger_index":"current"}]}'
# account_info (will error in standalone, that's fine — we trace errors too)
curl -s -X POST http://localhost:5005 \
-H "Content-Type: application/json" \
-d '{"method":"account_info","params":[{"account":"rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh"}]}'
-H "Content-Type: application/json" \
-d '{"method":"account_info","params":[{"account":"rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh"}]}'
```
6. **Verify in Grafana (Tempo)**:

59
OpenTelemetryPlan/Phase2_taskList.md
View File

@@ -1,8 +1,8 @@
# Phase 2: RPC Tracing Completion Task List
> **Goal**: Complete RPC tracing coverage with unit tests, Grafana search filters, node health attributes, and config hardening. Build on the Phase 1c SpanGuard factory foundation to achieve production-quality RPC observability.
> **Goal**: Complete RPC tracing coverage with unit tests, Grafana search filters, PathFind instrumentation, and config hardening. Build on the Phase 1c SpanGuard factory foundation to achieve production-quality RPC observability.
>
> **Scope**: Unit tests for core telemetry, Grafana Tempo search filters, node health span attributes, config validation (`std::clamp`).
> **Scope**: Unit tests for core telemetry, Grafana Tempo search filters, PathFind RPC tracing, config validation (`std::clamp`).
>
> **Branch**: `pratik/otel-phase2-rpc-tracing` (from `pratik/otel-phase1c-rpc-integration`)
@@ -67,7 +67,7 @@
- `src/tests/libxrpl/telemetry/TelemetryConfig.cpp`:
- Test Setup defaults (all fields have correct initial values)
- Test `setup_Telemetry` config parser (empty section, full section, edge cases)
- Test `setupTelemetry` config parser (empty section, full section, edge cases)
- Test `samplingRatio` clamping (values outside 0.0-1.0)
- `src/tests/libxrpl/telemetry/SpanGuardFactory.cpp`:
@@ -121,42 +121,9 @@ These can be added later if dashboard queries specifically need them. The node h
## Task 2.8: RPC Span Attribute Enrichment — Node Health Context
> **Source**: [External Dashboard Parity](../docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md) — adds node-level health context inspired by the community [xrpl-validator-dashboard](https://github.com/realgrapedrop/xrpl-validator-dashboard).
>
> **Downstream**: Phase 7 (MetricsRegistry uses these attributes for alerting context), Phase 10 (validation checks for these attributes).
**Status**: DROPPED.
**Objective**: Add node-level health state to every `rpc.command.*` span so operators can correlate RPC behavior with node state in Tempo.
**What to do**:
- Edit `src/xrpld/rpc/detail/RPCHandler.cpp`:
- In the `rpc.command.*` span creation block (after existing `setAttribute` calls for `command`, `version`, etc.):
- Node health attrs (`xrpl.node.amendment_blocked`, `xrpl.node.server_state`) are now resource-level attrs, not per-span. They are set at Tracer init.
**New span attributes**:
| Attribute | Type | Source | Example |
| ----------------------------- | ------ | ------------------------------------------- | -------- |
| `xrpl.node.amendment_blocked` | bool | `context.app.getOPs().isAmendmentBlocked()` | `true` |
| `xrpl.node.server_state` | string | `context.app.getOPs().strOperatingMode()` | `"full"` |
**Rationale**: When a node is amendment-blocked or in a degraded state, every RPC response is suspect. Tagging spans with this state enables Tempo TraceQL queries like:
```
{name=~"rpc.command.*"} | xrpl.node.amendment_blocked = true
```
This surfaces all RPCs served during a blocked period — critical for post-incident analysis.
**Key modified files**:
- `src/xrpld/rpc/detail/RPCHandler.cpp`
**Exit Criteria**:
- [ ] `rpc.command.server_info` spans carry `xrpl.node.amendment_blocked` and `xrpl.node.server_state` attributes
- [ ] No measurable latency impact (attribute values are cached atomics, not computed per-call)
- [ ] Attributes appear in Tempo trace detail view
Node health (`amendment_blocked`, `server_state`) is not part of the telemetry surface. Operators consume the same data via the existing `server_info` / `server_state` RPC commands, so duplicating it on traces adds storage and cardinality cost without new value. The OTel C++ SDK 1.18.0 also does not support runtime updates to the resource, ruling out resource-level emission of these dynamic-by-nature flags.
---
@@ -169,10 +136,11 @@ This surfaces all RPCs served during a blocked period — critical for post-inci
**Spans added**:
- `pathfind.request` — wraps `doPathFind()` and `doRipplePathFind()` RPC handlers
- `pathfind.compute` — wraps `PathRequest::doUpdate()` (fast/normal attr)
- `pathfind.update_all` — wraps `PathRequestManager::updateAll()` on ledger close (ledger_index attr)
- `pathfind.discover` — wraps `Pathfinder::findPaths()` graph exploration (search_level attr)
- `pathfind.rank` — wraps `Pathfinder::computePathRanks()` liquidity validation (num_paths attr)
- `pathfind.compute` — wraps `PathRequest::doUpdate()` (`pathfind_fast` attr)
- `pathfind.update_all` — wraps `PathRequestManager::updateAll()` on ledger close (`pathfind_ledger_index`, `pathfind_num_requests` attrs; emitted only when active subscriptions exist)
- `pathfind.discover` — wraps the entire per-source-asset loop in `PathRequest::findPaths()` (`pathfind_search_level`, `pathfind_num_paths` attrs). One span per RPC call instead of N (one per source asset). Trade-off: per-asset breakdown is lost; storage and cardinality bounded.
**Attribute namespacing**: All pathfind attributes use the `pathfind_*` underscore form per the Phase 1c naming-spec rule 5.
**New file**: `src/xrpld/rpc/detail/PathFindSpanNames.h`
@@ -197,9 +165,10 @@ This surfaces all RPCs served during a blocked period — critical for post-inci
| 2.5 | Enhanced RPC span attributes (HTTP-level) | Deferred | Low value; span duration covers timing natively |
| 2.6 | Build verification and performance baseline | Complete | Verified in CI on Phase 1c |
| 2.7 | Grafana Tempo search filters | Complete | rpc-command, rpc-status, rpc-role filters |
| 2.8 | RPC span attribute enrichment (node health) | Complete | amendment_blocked + server_state |
| 2.9 | PathFind RPC instrumentation (5 spans) | Complete | request, compute, update_all, discover, rank |
| 2.8 | RPC span attribute enrichment (node health) | Dropped | Available via `server_info`/`server_state` RPC |
| 2.9 | PathFind RPC instrumentation | Complete | request, compute, update_all, discover |
**Delivered in this branch**: Tasks 2.4, 2.7, 2.8, 2.9.
**Delivered in this branch**: Tasks 2.4, 2.7, 2.9.
**Deferred with rationale**: Tasks 2.1 (→Phase 3), 2.5 (low priority).
**Dropped**: Task 2.8 (node health not duplicated on traces).
**Superseded**: Task 2.2 (Phase 1c SpanGuard factory covers this).

6
OpenTelemetryPlan/Phase3_taskList.md
View File

@@ -89,7 +89,7 @@
- In `onMessage(TMTransaction)` / `handleTransaction()`:
- Extract parent trace context from incoming `TMTransaction::trace_context` field (if present)
- Create `tx.receive` span as child of extracted context (or new root if none)
- Set attributes: `xrpl.tx.hash`, `xrpl.peer.id`, `tx_status`
- Set attributes: `tx_hash`, `peer_id`, `tx_status`
- On HashRouter suppression (duplicate): set `suppressed=true`, add `tx.duplicate` event
- Wrap validation call with child span `tx.validate`
- Wrap relay with `tx.relay` span
@@ -121,7 +121,7 @@
- Edit `src/xrpld/app/misc/NetworkOPs.cpp`:
- In `processTransaction()`:
- Create `tx.process` span
- Set attributes: `xrpl.tx.hash`, `tx_type`, `local` (whether from RPC or peer)
- Set attributes: `tx_hash`, `tx_type`, `local` (whether from RPC or peer)
- Record whether sync or async path is taken
- In `doTransactionAsync()`:
@@ -256,7 +256,7 @@
**What to do**:
- Edit `src/xrpld/overlay/detail/PeerImp.cpp`:
- In the `tx.receive` span block (after existing `xrpl.peer.id` setAttribute call):
- In the `tx.receive` span block (after existing `peer_id` setAttribute call):
- Add `peer_version` (string) — from `this->getVersion()`
- Only set if `getVersion()` returns a non-empty string (avoid empty-string attributes)

16
OpenTelemetryPlan/Phase4_taskList.md
View File

@@ -445,8 +445,8 @@ direct method calls, which is cleaner and avoids macro control-flow issues.
// Actual usage in Consensus.h::updateOurPositions():
span.addEvent(
"dispute.resolve",
{{cons_span::attr::txId, to_string(txId)},
{cons_span::attr::disputeOurVote, dispute.getOurVote() ? "yes" : "no"}});
{{consensus::span::attr::txId, to_string(txId)},
{consensus::span::attr::disputeOurVote, dispute.getOurVote() ? "yes" : "no"}});
```
2. **Span link support** — implemented via `SpanGuard::linkedSpan()` static factory
@@ -624,10 +624,10 @@ details.
```cpp
span.addEvent(
"dispute.resolve",
{{cons_span::attr::txId, to_string(txId)},
{cons_span::attr::disputeOurVote, dispute.getOurVote() ? "yes" : "no"},
{cons_span::attr::disputeYays, std::to_string(dispute.getYays())},
{cons_span::attr::disputeNays, std::to_string(dispute.getNays())}});
{{consensus::span::attr::txId, to_string(txId)},
{consensus::span::attr::disputeOurVote, dispute.getOurVote() ? "yes" : "no"},
{consensus::span::attr::disputeYays, std::to_string(dispute.getYays())},
{consensus::span::attr::disputeNays, std::to_string(dispute.getNays())}});
```
**Not implemented**:
@@ -687,8 +687,8 @@ wrongLedger, switchedLedger).
```cpp
auto span = telemetry::SpanGuard::span(
telemetry::TraceCategory::Consensus, telemetry::seg::consensus, "mode_change");
span.setAttribute(cons_span::attr::modeOld, to_string(before).c_str()); // "mode_old"
span.setAttribute(cons_span::attr::modeNew, to_string(after).c_str()); // "mode_new"
span.setAttribute(consensus::span::attr::modeOld, to_string(before).c_str()); // "mode_old"
span.setAttribute(consensus::span::attr::modeNew, to_string(after).c_str()); // "mode_new"
```
- `MonitoredMode::set()` in `Consensus.h` calls `adaptor_.onModeChange(before, after)`.

243
OpenTelemetryPlan/secure-OTel.md Normal file
View File

@@ -0,0 +1,243 @@
# Securing OpenTelemetry Against Trace Context Spoofing
> **Part of**: [OpenTelemetry Implementation Plan](./OpenTelemetryPlan.md) — see also [Design Decisions § Privacy](./02-design-decisions.md#244-privacy--sensitive-data-policy) (what we don't collect) and [Configuration Reference § 5.5](./05-configuration-reference.md#55-opentelemetry-collector-configuration) (collector base config).
Trace context spoofing (or poisoning) occurs when untrusted actors inject tampered or stale trace IDs into your system. If these requests are processed, the spans are appended to historical trace buckets, stretching trace durations, ruining p99 latency metrics, and breaking Grafana dashboards.
This guide outlines two categories of defense: mitigating tampered contexts and locking down the OpenTelemetry (OTel) Collector to trusted clients only.
---
## Part 1: Mitigating Tampered Trace Contexts
### 1. Perimeter Defense: Strip Headers at the API Gateway
The most effective way to prevent spoofing from external sources is to treat your API Gateway (Envoy, NGINX, AWS ALB) as a hard boundary. Strip incoming W3C tracing headers (`traceparent`, `tracestate`) from public traffic so the gateway is forced to generate a fresh, legitimate `trace_id`.
**NGINX Example (Stripping Headers):**
Code output
File generated successfully.
```nginx
server {
listen 80;
location {
# Clear out untrusted incoming trace headers
proxy_set_header traceparent "";
proxy_set_header tracestate "";
proxy_pass http://backend_service;
}
}
```
### **2. Timestamp-Anchored Trace IDs and OTTL Filtering**
If you use a custom trace ID generator that embeds a timestamp in the first few bytes (like AWS X-Ray or UUIDv7), you can use the OTel Collector's OpenTelemetry Transform Language (OTTL) to detect anomalies.
**Collector Configuration (Conceptual OTTL Filter):**
```yaml
processors:
filter/stale_traces:
error_mode: ignore
traces:
span:
# Example: Drop spans where the start time is significantly different
# from an expected parameter or embedded timestamp logic.
# Note: Standard W3C trace IDs do not contain timestamps by default.
- 'Keep out-of-bounds spans: time.sub(start_time, now()) > duration("1h")'
```
## **Part 2: Restricting Access to the OTel Collector**
Locking down the Collector ensures that only authenticated, trusted clients can submit telemetry data.
### **Approach A: Network Layer Security (Kubernetes Network Policies)**
Ensure your Collector is not exposed to the public internet. If running in Kubernetes, use a NetworkPolicy to restrict ingress traffic to specific namespaces.
**Kubernetes NetworkPolicy Example:**
```yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-internal-otel
namespace: observability
spec:
podSelector:
matchLabels:
app: opentelemetry-collector
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
environment: production
ports:
- protocol: TCP
port: 4317 # gRPC
- protocol: TCP
port: 4318 # HTTP
```
### **Approach B: Transport Layer Security (Mutual TLS / mTLS)**
Require clients to present a valid cryptographic certificate to connect to the Collector.
**Collector Configuration (mTLS):**
```yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
tls:
client_ca_file: /certs/client_ca.pem # CA that signs trusted client certs
cert_file: /certs/collector.pem
key_file: /certs/collector.key
auth_type: require_and_verify_client_cert # Rejects unauthorized clients
```
### **Approach C: Application Layer Authentication (Basic Auth Extension)**
Use the Collector's extension system to require an API key or Basic Auth credentials.
**Collector Configuration (Basic Auth):**
```yaml
extensions:
basicauth/collector:
htpasswd:
inline: |
# username:trusted-client, password:SecurePassword123
trusted-client:$apr1$4v8p76o6$DMTX5Wv6uOmrFAZp2X1N1.
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
auth:
authenticator: basicauth/collector
processors:
batch:
exporters:
otlp:
endpoint: my-backend-storage:4317
service:
extensions: [basicauth/collector]
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp]
```
**Client Setup (Environment Variables):**
Developers must pass the authentication header using the standard OTel SDK environment variables:
```bash
# Base64 encoded "trusted-client:SecurePassword123"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic dHJ1c3RlZC1jbGllbnQ6U2VjdXJlUGFzc3dvcmQxMjM="
```
---
Available routes to build on top of: https://github.com/XRPLF/rippled/pull/6425#discussion_r3234751995
---
# Analysis: Applying the Guide to xrpld
The guide above is written for HTTP-fronted web services. xrpld is a P2P node daemon, so the threat model and the applicable defenses differ. This section captures how each approach maps to xrpld and the chosen direction.
## Threat Model
xrpld has **two distinct attack surfaces**, not one. The original guide conflates them under "trace context spoofing"; for xrpld they need separate defenses.
| Surface | Attacker | Vector | Defense |
| ----------------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------- |
| **Collector ingress** (xrpld → collector) | Anyone who can reach `4317`/`4318` on the collector host | Forged OTLP traffic, telemetry exfiltration, DoS on collector | mTLS + network policy |
| **Peer trace context** (peer → xrpld) | Malicious peer in the XRPL overlay | Crafted `protocol::TraceContext` field inside peer protobuf messages (TMTransaction, consensus, etc.) — used to forge `trace_id`/`span_id`, pollute p99, attach spans to historical traces | Validate + rate-limit at the receive boundary |
**Deployment context:** Across-network. xrpld nodes (potentially run by external operators or in different DCs) ship telemetry to a centrally-hosted collector across an untrusted network. The collector is NOT on the same host or private VPC as every node.
```
┌── peer (untrusted) ── TMTransaction{trace_context} ──▶ xrpld
│ │
│ [validate + rate-limit]
│ │
│ ▼
│ SpanGuard (clean)
│ │
│ │ OTLP/gRPC
│ │ + mTLS
│ ▼
└───────────────────────────────────────── [require_and_verify_client_cert]
OTel Collector
(in private subnet, NetPol)
```
## Part 1 Applicability — Peer Trace-Context Validation
The guide's NGINX header stripping and OTTL stale-span filtering target HTTP gateways and post-hoc cleanup. Neither fits xrpld directly:
- **NGINX header stripping** — N/A. There is no HTTP gateway between peers and xrpld; trace context arrives inside protobuf peer messages (`protocol::TraceContext`), not as W3C `traceparent` headers. See [src/xrpld/telemetry/PropagationHelpers.h](../src/xrpld/telemetry/PropagationHelpers.h).
- **OTTL stale-span filtering** — Weak fit. Post-hoc cleanup at the collector loses peer identity (you can't tell _which_ peer poisoned the trace). Validation at the receive site is stronger.
**xrpld-specific Part 1 mitigations:**
1. **Validate extracted context at the boundary** in [src/xrpld/telemetry/ConsensusReceiveTracing.h](../src/xrpld/telemetry/ConsensusReceiveTracing.h) and any other peer-message receive site. Reject if `trace_id` is all-zero, wrong length, or fails W3C format checks. Treat invalid context as "no propagated context" — start a fresh span — rather than dropping the message.
2. **Per-peer sample rate limiting** so a hostile peer cannot flood the collector with spans bearing a fabricated `trace_id`. Use probabilistic sampling on the receive path keyed by peer identity.
## Part 2 — Comparison of Collector Hardening Approaches
Evaluated for the across-network deployment shape:
| Approach | Across-network fit | Cost | Verdict |
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | ---------------------------------- |
| **A. NetworkPolicy / firewall** | Necessary baseline (don't expose `4317`/`4318` to the internet), but insufficient on its own when traffic genuinely crosses networks — you cannot NetworkPolicy the public internet. | Cheap. | **Defense-in-depth, not primary.** |
| **B. mTLS** | Strongest fit. Every xrpld node holds a client cert; collector verifies with `require_and_verify_client_cert`. Encrypts in transit (raw OTLP over the internet leaks transaction patterns and validator identity). Compromised node = revoke one cert, no shared secret to rotate everywhere. | Cert issuance + rotation pipeline. | **Primary.** |
| **C. Basic Auth** | Worst shape for this topology. Single shared password across all xrpld nodes — one leaked node config compromises the whole fleet. Doesn't encrypt; you'd need TLS underneath anyway, at which point you're 80% of the way to mTLS. | Cheap to set up, expensive to operate (rotation across N operators). | **Skip.** |
## Decision
**Primary defense:** mTLS (Approach B) on the collector's OTLP receivers, with `auth_type: require_and_verify_client_cert`.
**Defense-in-depth:** NetworkPolicy / firewall rules (Approach A) so `4317`/`4318` are never reachable from outside the expected operator subnets even if mTLS were misconfigured.
**Skipped:** Basic Auth (Approach C) — wrong shape for an across-network, multi-operator topology.
**Plus xrpld-specific Part 1 work:** trace-context validation and per-peer rate limiting at peer-message receive sites.
## Decisions Made
| Decision | Choice | Rationale |
| -------------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Cert source for mTLS | **Reuse XRPL node identity key** | One identity per node, no separate PKI to operate. Fits XRPL's existing trust model; requires small CA tooling step to derive/sign the OTel client cert from the node key. |
| Part 1 scope | **Include in this spec** | Collector hardening and peer trace-context validation share one threat model. Coherent design doc; can still be split into multiple PRs at implementation. |
| Dev impact | **Production-only** | Local `docker/telemetry/docker-compose.yml` keeps `insecure: true` and no auth for fast iteration. Only production deployment manifests gain mTLS. Accepted risk: minor dev/prod drift, mitigated by integration tests against a TLS-enabled collector in CI. |
## Out of Scope
- NGINX/Envoy header stripping (no HTTP gateway in front of xrpld-to-collector traffic).
- OTTL stale-span filtering at the collector (weaker than source validation; loses peer identity).
- Local development docker-compose hardening.
- Telemetry backend (Tempo) hardening — separate concern, downstream of the collector.
## Next Step
Write this up as a design doc with full sections covering:
1. Threat model & architecture (this section, expanded)
2. Collector hardening — mTLS config, NetworkPolicy
3. Cert pipeline — deriving OTel client cert from XRPL node key
4. Peer trace-context validation — receive-site checks in `ConsensusReceiveTracing.h`
5. Per-peer span rate limiting
6. Testing & rollout