ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-07-29 18:10:34 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill

Browse Source 
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
This commit is contained in:
Pratik Mankawde
2026-06-26 16:23:16 +01:00
parent 3e375d1034 dc3d5107fd
commit fa46aa55fc
345 changed files with 6574 additions and 4975 deletions
Download Patch File Download Diff File Expand all files Collapse all files

99
OpenTelemetryPlan/03-implementation-strategy.md
View File

@@ -229,6 +229,14 @@ xychart-beta
- Maximum memory is bounded: ~8.3 MB static (dominated by worker thread stack) + 2048 queued spans x ~500 bytes (~1 MB) + active spans (~0.8 MB) ≈ **~10 MB ceiling**
- The worker thread stack (~8 MB) is virtual memory; actual RSS depends on stack usage (typically much less)
> **Measured outcome**: A perf-iac comparison (telemetry compiled-in + enabled vs compiled-out,
> 9 nodes — validators and client-handlers — under sustained payment load) recorded **no measurable
> RSS increase over the telemetry-off baseline** (~15 GiB mean / ~18–19 GiB peak on both sides),
> with no OOM, no swap, and no leak across the run. The ~10 MB ceiling above is therefore a
> provisioning safety margin (dominated by virtual thread-stack address space), not an expected
> resident-memory increase. Steady-state cost shows up as throughput (~3–4% at head sampling 1.0),
> not memory.
### 3.5.4 Performance Data Sources
The overhead estimates in Sections 3.3-3.5 are derived from the following sources:
@@ -329,19 +337,19 @@ This section provides a detailed assessment of how intrusive the OpenTelemetry i
### 3.9.1 Files Modified Summary
| Component | Files Modified | Lines Added | Lines Changed | Architectural Impact |
| --------------------- | -------------- | ----------- | ------------- | -------------------- |
| **Core Telemetry** | 7 new files | ~800 | 0 | None (new module) |
| **Application Init** | 2 files | ~30 | ~5 | Minimal |
| **RPC Layer** | 3 files | ~80 | ~20 | Minimal |
| **Transaction Relay** | 4 files | ~120 | ~40 | Low |
| **Consensus** | 3 files | ~100 | ~30 | Low-Medium |
| **Protocol Buffers** | 1 file | ~25 | 0 | Low |
| **CMake/Build** | 3 files | ~50 | ~10 | Minimal |
| **PathFinding** | 2 | ~80 | ~5 | Minimal |
| **TxQ/Fee** | 2 | ~60 | ~5 | Minimal |
| **Validator/Amend** | 3 | ~40 | ~5 | Minimal |
| **Total** | **~27 files** | **~1,490** | **~120** | **Low** |
| Component | Files Modified | Architectural Impact |
| --------------------- | -------------- | -------------------- |
| **Core Telemetry** | 10 new files | None (new module) |
| **Application Init** | 2 files | Minimal |
| **RPC Layer** | 3 files | Minimal |
| **Transaction Relay** | 4 files | Low |
| **Consensus** | 3 files | Low-Medium |
| **Protocol Buffers** | 1 file | Low |
| **CMake/Build** | 3 files | Minimal |
| **PathFinding** | 2 | Minimal |
| **TxQ/Fee** | 2 | Minimal |
| **Validator/Amend** | 3 | Minimal |
| **Total** | **~33 files** | **Low** |
### 3.9.2 Detailed File Impact
@@ -361,40 +369,43 @@ pie title Code Changes by Component
#### New Files (No Impact on Existing Code)
| File | Lines | Purpose |
| ------------------------------------------- | ----- | ----------------------------------------------------- |
| `include/xrpl/telemetry/Telemetry.h` | ~160 | Main interface (global singleton) |
| `include/xrpl/telemetry/SpanGuard.h` | ~250 | RAII wrapper + factory methods + discard + no-op stub |
| `include/xrpl/telemetry/DiscardFlag.h` | ~28 | Thread-local discard flag |
| `include/xrpl/telemetry/TraceContext.h` | ~80 | Context propagation |
| `src/libxrpl/telemetry/Telemetry.cpp` | ~400 | Implementation + FilteringSpanProcessor |
| `src/libxrpl/telemetry/TelemetryConfig.cpp` | ~60 | Config parsing |
| `src/libxrpl/telemetry/NullTelemetry.cpp` | ~40 | No-op implementation |
| File | Purpose |
| ------------------------------------------- | ------------------------- |
| `include/xrpl/telemetry/Telemetry.h` | Main interface |
| `include/xrpl/telemetry/TelemetryConfig.h` | Configuration structures |
| `include/xrpl/telemetry/TraceContext.h` | Context propagation |
| `include/xrpl/telemetry/SpanGuard.h` | RAII wrapper |
| `include/xrpl/telemetry/DiscardFlag.h` | Thread-local discard flag |
| `include/xrpl/telemetry/SpanAttributes.h` | Attribute helpers |
| `src/libxrpl/telemetry/Telemetry.cpp` | Implementation |
| `src/libxrpl/telemetry/TelemetryConfig.cpp` | Config parsing |
| `src/libxrpl/telemetry/TraceContext.cpp` | Context serialization |
| `src/libxrpl/telemetry/NullTelemetry.cpp` | No-op implementation |
#### Modified Files (Existing Xrpld Code)
| File | Lines Added | Lines Changed | Risk Level |
| ------------------------------------------------- | ----------- | ------------- | ---------- |
| `src/xrpld/app/main/Application.cpp` | ~15 | ~3 | Low |
| `include/xrpl/core/ServiceRegistry.h` | ~5 | ~2 | Low |
| `src/xrpld/rpc/detail/ServerHandler.cpp` | ~40 | ~10 | Low |
| `src/xrpld/rpc/handlers/*.cpp` | ~30 | ~8 | Low |
| `src/xrpld/overlay/detail/PeerImp.cpp` | ~60 | ~15 | Medium |
| `src/xrpld/overlay/detail/OverlayImpl.cpp` | ~30 | ~10 | Medium |
| `src/xrpld/app/consensus/RCLConsensus.cpp` | ~50 | ~15 | Medium |
| `src/xrpld/app/consensus/RCLConsensusAdaptor.cpp` | ~40 | ~12 | Medium |
| `src/xrpld/core/JobQueue.cpp` | ~20 | ~5 | Low |
| `src/xrpld/app/paths/PathRequest.cpp` | ~40 | ~3 | Low |
| `src/xrpld/app/paths/Pathfinder.cpp` | ~40 | ~2 | Low |
| `src/xrpld/app/misc/TxQ.cpp` | ~40 | ~3 | Low |
| `src/xrpld/app/main/LoadManager.cpp` | ~20 | ~2 | Low |
| `src/xrpld/app/misc/ValidatorList.cpp` | ~20 | ~2 | Low |
| `src/xrpld/app/misc/AmendmentTable.cpp` | ~10 | ~2 | Low |
| `src/xrpld/app/misc/Manifest.cpp` | ~10 | ~1 | Low |
| `src/xrpld/shamap/SHAMap.cpp` | ~20 | ~3 | Low |
| `src/xrpld/overlay/detail/ripple.proto` | ~25 | 0 | Low |
| `CMakeLists.txt` | ~40 | ~8 | Low |
| `cmake/FindOpenTelemetry.cmake` | ~50 | 0 | None (new) |
| File | Risk Level |
| ------------------------------------------------- | ---------- |
| `src/xrpld/app/main/Application.cpp` | Low |
| `include/xrpl/core/ServiceRegistry.h` | Low |
| `src/xrpld/rpc/detail/ServerHandler.cpp` | Low |
| `src/xrpld/rpc/handlers/*.cpp` | Low |
| `src/xrpld/overlay/detail/PeerImp.cpp` | Medium |
| `src/xrpld/overlay/detail/OverlayImpl.cpp` | Medium |
| `src/xrpld/app/consensus/RCLConsensus.cpp` | Medium |
| `src/xrpld/app/consensus/RCLConsensusAdaptor.cpp` | Medium |
| `src/xrpld/core/JobQueue.cpp` | Low |
| `src/xrpld/app/paths/PathRequest.cpp` | Low |
| `src/xrpld/app/paths/Pathfinder.cpp` | Low |
| `src/xrpld/app/misc/TxQ.cpp` | Low |
| `src/xrpld/app/main/LoadManager.cpp` | Low |
| `src/xrpld/app/misc/ValidatorList.cpp` | Low |
| `src/xrpld/app/misc/AmendmentTable.cpp` | Low |
| `src/xrpld/app/misc/Manifest.cpp` | Low |
| `src/xrpld/shamap/SHAMap.cpp` | Low |
| `src/xrpld/overlay/detail/ripple.proto` | Low |
| `CMakeLists.txt` | Low |
| `cmake/FindOpenTelemetry.cmake` | None (new) |
### 3.9.3 Risk Assessment by Component

15
OpenTelemetryPlan/05-configuration-reference.md
View File

@@ -40,13 +40,14 @@ The authoritative `[telemetry]` example lives in `cfg/xrpld-example.cfg`. Teleme
documents but are not parsed by `TelemetryConfig.cpp` in Phase 1b and later
phases. They will be added as the corresponding subsystems are instrumented:
| Option | Planned Phase | Purpose |
| ----------------- | ------------- | ---------------------------------------- |
| `exporter` | Future | Select between OTLP/HTTP and OTLP/gRPC |
| `trace_pathfind` | Phase 2 | Path computation tracing toggle |
| `trace_txq` | Phase 3 | Transaction queue tracing toggle |
| `trace_validator` | Future | Validator list / manifest update tracing |
| `trace_amendment` | Future | Amendment voting tracing |
| Option | Planned Phase | Purpose |
| -------------------------- | ------------- | ----------------------------------------------------------------------- |
| `exporter` | Future | Select between OTLP/HTTP and OTLP/gRPC |
| `trace_pathfind` | Phase 2 | Path computation tracing toggle |
| `trace_txq` | Phase 3 | Transaction queue tracing toggle |
| `trace_validator` | Future | Validator list / manifest update tracing |
| `trace_amendment` | Future | Amendment voting tracing |
| `consensus_trace_strategy` | Phase 4 | Trace ID strategy for consensus rounds (`deterministic` \| `attribute`) |
---

9
OpenTelemetryPlan/07-observability-backends.md
View File

@@ -258,10 +258,13 @@ Grafana provisions three TraceQL-based alert rules (group `xrpld-tracing-alerts`
{resource.service.name="xrpld" && name="consensus.round"} | avg(duration) > 5s
```
- **RPC Error Rate Spike** (critical, `for: 2m`): fires when the error rate across `rpc.command.*` spans exceeds 5%.
- **RPC Error Rate Spike** (critical, `for: 2m`): fires when the error rate across `rpc.command.*` spans exceeds 5%. Error _rate_ is a ratio, so it must divide the error-span rate by the total-span rate — a single TraceQL `rate()` returns spans/second, not a percentage, and would fire on traffic volume alone. This uses span metrics emitted by the collector's `spanmetrics` connector (Prometheus datasource), not a TraceQL query:
```
{resource.service.name="xrpld" && name=~"rpc.command.*" && status.code=error} | rate() > 0.05
sum(rate(calls_total{service_name="xrpld", span_name=~"rpc.command.*", status_code="STATUS_CODE_ERROR"}[5m]))
/
sum(rate(calls_total{service_name="xrpld", span_name=~"rpc.command.*"}[5m]))
> 0.05
```
- **Transaction Throughput Drop** (warning, `for: 10m`): fires when the `tx.receive` span rate falls below 10/s.
@@ -270,7 +273,7 @@ Grafana provisions three TraceQL-based alert rules (group `xrpld-tracing-alerts`
{resource.service.name="xrpld" && name="tx.receive"} | rate() < 10
```
> **Note**: The first two rules use TraceQL aggregates (`avg(duration)`, `rate()`), which require Tempo 2.3+ with TraceQL metrics enabled. Verify aggregate query support in your Tempo version before provisioning.
> **Note**: The Consensus Round Slow and Transaction Throughput Drop rules use TraceQL aggregates (`avg(duration)`, `rate()`), which require Tempo 2.3+ with TraceQL metrics enabled. Verify aggregate query support in your Tempo version before provisioning. The RPC Error Rate Spike rule instead queries Prometheus span metrics (collector `spanmetrics` connector), so it needs that connector enabled in the collector pipeline.
---

12
OpenTelemetryPlan/OpenTelemetryPlan.md
View File

@@ -16,12 +16,12 @@ This document provides a comprehensive implementation plan for integrating OpenT
### Estimated Performance Overhead
| Metric | Overhead | Notes |
| ------------- | ---------- | ----------------------------------- |
| CPU | 1-3% | Span creation and attribute setting |
| Memory | 2-5 MB | Batch buffer for pending spans |
| Network | 10-50 KB/s | Compressed OTLP export to collector |
| Latency (p99) | <2% | With proper sampling configuration |
| Metric | Overhead | Notes |
| ------------- | ---------- | ------------------------------------------------ |
| CPU | 1-3% | Span creation and attribute setting |
| Memory | <10 MB | SDK statics + batch buffer + worker thread stack |
| Network | 10-50 KB/s | Compressed OTLP export to collector |
| Latency (p99) | <2% | With proper sampling configuration |
---