mirror of
https://github.com/XRPLF/rippled.git
synced 2026-07-24 23:50:22 +00:00
Merge branch 'pratik/otel-phase10-workload-validation' into pratik/perf-test-otel-off
This commit is contained in:
Pratik Mankawde
@@ -185,7 +185,7 @@ agree with the code. A CI check enforces this end to end.
|
||||
3. **Collision qualifier** → `<domain>_<field>`, only when a bare name would
|
||||
collide with a DIFFERENT concept in the shared spanmetrics label space or with
|
||||
the OTel-reserved `status` key (e.g. `rpc_status`, `grpc_status`,
|
||||
`consensus_state`, `consensus_round`, `consensus_mode`). This disambiguates
|
||||
`consensus_phase`, `consensus_round`, `consensus_mode`). This disambiguates
|
||||
distinct concepts that share a word; it is NOT used to tag the same concept
|
||||
with its emitting workflow — that is rule 2 (one shared name).
|
||||
4. **Resource attribute** → dotted `xrpl.<subsystem>.<field>`, reserved ONLY
|
||||
@@ -245,6 +245,8 @@ keys (the dotted form is reserved for resource scope per §2.3.3).
|
||||
| `tx_fee` | int64 | Fee in drops |
|
||||
| `tx_result` | string | `"tesSUCCESS"`, `"tecPATH_DRY"`, etc. |
|
||||
| `ledger_index` | int64 | Ledger containing transaction |
|
||||
| `relay_count` | int64 | Peers the transaction was relayed to |
|
||||
| `suppressed` | bool | `true` when HashRouter dropped a dup |
|
||||
|
||||
#### Consensus Attributes
|
||||
|
||||
@@ -278,12 +280,14 @@ Establish-phase gap fill and cross-node correlation attributes (Phase 4a):
|
||||
|
||||
#### RPC Attributes
|
||||
|
||||
| Key | Type | Description |
|
||||
| ---------- | ------ | ----------------------------------------------------- |
|
||||
| `command` | string | Command name (per-span unique on `rpc.command`) |
|
||||
| `version` | int64 | API version |
|
||||
| `rpc_role` | string | `"admin"` or `"user"` (qualified — `role` is generic) |
|
||||
| `params` | string | Sanitized parameters (optional) |
|
||||
| Key | Type | Description |
|
||||
| ------------- | ------- | ----------------------------------------------------------------------------- |
|
||||
| `command` | string | Command name (per-span unique on `rpc.command`) |
|
||||
| `version` | int64 | API version |
|
||||
| `rpc_role` | string | `"admin"` or `"user"` (qualified — `role` is generic) |
|
||||
| `params` | string | Sanitized parameters (optional) |
|
||||
| `rpc_status` | string | Response status: `success` \| `error` (qualified — `status` is OTel-reserved) |
|
||||
| `duration_ms` | float64 | Request duration in milliseconds |
|
||||
|
||||
#### Peer & Message Attributes
|
||||
|
||||
@@ -395,32 +399,48 @@ The following data is explicitly **excluded** from telemetry collection:
|
||||
|
||||
#### Privacy Protection Mechanisms
|
||||
|
||||
| Mechanism | Description |
|
||||
| ----------------------------- | ------------------------------------------------------------------------- |
|
||||
| **Account Hashing** | `tx_account` is hashed at collector level before storage |
|
||||
| **Configurable Redaction** | Sensitive fields can be excluded via `[telemetry]` config section |
|
||||
| **Sampling** | Only 10% of traces recorded by default, reducing data exposure |
|
||||
| **Local Control** | Node operators have full control over what gets exported |
|
||||
| **No Raw Payloads** | Transaction content is never recorded, only metadata (hash, type, result) |
|
||||
| **Collector-Level Filtering** | Additional redaction/hashing can be configured at OTel Collector |
|
||||
| Mechanism | Description |
|
||||
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
||||
| **Account Hashing** | Account addresses are hashed both SDK-side (`pathfind_source_account`, `pathfind_dest_account` — always hashed before emission) and again at the collector level, so raw addresses never reach storage |
|
||||
| **Configurable Redaction** | Sensitive fields can be excluded via `[telemetry]` config section |
|
||||
| **Collector Tail Sampling** | xrpld head sampling is fixed at 1.0 (every span emitted); the collector retains ~10% of non-error traces, reducing stored data exposure |
|
||||
| **Sampling** | Only 10% of traces recorded by default, reducing data exposure |
|
||||
| **Local Control** | Node operators have full control over what gets exported |
|
||||
| **No Raw Payloads** | Transaction content is never recorded, only metadata (hash, type, result) |
|
||||
| **Collector-Level Filtering** | Additional redaction/hashing can be configured at OTel Collector |
|
||||
|
||||
#### Account Address Hashing
|
||||
|
||||
Account addresses are **always** hashed before they reach the telemetry
|
||||
backend — there is no opt-out flag and therefore no insecure-by-default
|
||||
failure mode. Protection is applied in two independent layers:
|
||||
|
||||
1. **SDK-side** (this node): the path-finding RPC handlers call
|
||||
`redactAccount()` (`xrpl::telemetry`, `Redaction.h`) before setting the
|
||||
`pathfind_source_account` / `pathfind_dest_account` span attributes. The
|
||||
helper emits the first 16 characters of `sha512Half(address)` as
|
||||
lowercase hex — deterministic (spans for one account still correlate)
|
||||
but non-reversible.
|
||||
2. **Collector-side** (defense-in-depth): an `attributes/hash` processor in
|
||||
the OpenTelemetry Collector re-hashes those same attributes, so any node
|
||||
that emitted a raw value is still redacted before storage.
|
||||
|
||||
#### Collector-Level Data Protection
|
||||
|
||||
The OpenTelemetry Collector can be configured (via an `attributes` processor)
|
||||
to hash or redact sensitive attributes before export — for example, hashing
|
||||
`tx_account`, deleting `peer_address` to drop IP addresses, and deleting
|
||||
`params` to redact request parameters.
|
||||
`pathfind_source_account` / `pathfind_dest_account`, deleting `peer_address`
|
||||
to drop IP addresses, and deleting `params` to redact request parameters.
|
||||
|
||||
#### Configuration Options for Privacy
|
||||
|
||||
In `xrpld.cfg`, operators control data collection granularity through the
|
||||
`[telemetry]` section. Besides `enabled`, per-component toggles
|
||||
(`trace_transactions`, `trace_consensus`, `trace_rpc`, `trace_peer` — the last
|
||||
often disabled due to high volume) select which spans are emitted, and
|
||||
redaction flags (`redact_account` to hash account addresses, `redact_peer_address`
|
||||
to remove peer IP addresses) control SDK-level redaction before export.
|
||||
|
||||
> **Note**: The `redact_account` configuration in `xrpld.cfg` controls SDK-level redaction before export, while collector-level filtering (see [Collector-Level Data Protection](#collector-level-data-protection) above) provides an additional defense-in-depth layer. Both can operate independently.
|
||||
often disabled due to high volume) select which spans are emitted. Account
|
||||
address hashing is not configurable: addresses are hashed unconditionally by
|
||||
the SDK helper described above, with collector-level hashing as a second
|
||||
layer.
|
||||
|
||||
> **Key Principle**: Telemetry collects **operational metadata** (timing, counts, hashes) — never **sensitive content** (keys, balances, amounts, raw payloads).
|
||||
|
||||
@@ -646,7 +666,7 @@ flowchart TB
|
||||
- **xrpld Process (dark gray)**: The single xrpld node running all three observability frameworks side by side. Each framework operates independently with no interference.
|
||||
- **PerfLog to perf.log**: PerfLog writes JSON-formatted event logs to a local file. Grafana can ingest these via Loki or a file-based datasource.
|
||||
- **Beast Insight to StatsD Server**: Insight sends aggregated metrics (counters, gauges) over UDP to a StatsD server. Grafana reads from StatsD-compatible backends like Graphite or Prometheus (via StatsD exporter).
|
||||
- **OpenTelemetry to OTLP Collector**: OTel exports spans over OTLP/gRPC to a Collector, which then forwards to a trace backend (Tempo).
|
||||
- **OpenTelemetry to OTLP Collector**: OTel exports spans over OTLP/HTTP to a Collector, which then forwards to a trace backend (Tempo). (OTLP/gRPC is future work — §2.2.2.)
|
||||
- **Grafana (red, unified UI)**: All three data streams converge in Grafana, enabling operators to correlate logs, metrics, and traces in a single dashboard.
|
||||
|
||||
**Phase 7 target state**: Beast Insight routes to `OTelCollector` (new `Collector` implementation) which exports via OTLP/HTTP to the same collector endpoint as traces. StatsD UDP path becomes a deprecated fallback (`[insight] server=statsd`). See [06-implementation-phases.md §6.8](./06-implementation-phases.md) and [Phase7_taskList.md](./Phase7_taskList.md) for details.
|
||||
|
||||
@@ -180,14 +180,14 @@ pie showData
|
||||
| TracerProvider singleton | ~64 KB | At startup |
|
||||
| BatchSpanProcessor (circular buffer) | ~16 KB | At startup |
|
||||
| BatchSpanProcessor (worker thread) | ~8 MB | At startup |
|
||||
| OTLP exporter (gRPC channel init) | ~256 KB | At startup |
|
||||
| OTLP/HTTP exporter (client init) | ~64 KB | At startup |
|
||||
| Propagator registry | ~8 KB | At startup |
|
||||
| **Total static** | **~8.3 MB** | |
|
||||
| **Total static** | **~8.1 MB** | |
|
||||
|
||||
> **Why higher than earlier estimate**: The BatchSpanProcessor's circular buffer itself is only ~16 KB
|
||||
> (2049 x 8-byte `AtomicUniquePtr` entries), but it spawns a dedicated worker thread whose default
|
||||
> stack size on Linux is ~8 MB. The OTLP gRPC exporter allocates memory for channel stubs and TLS
|
||||
> initialization. The worker thread stack dominates the static footprint.
|
||||
> stack size on Linux is ~8 MB. The OTLP/HTTP exporter allocates a small client and TLS
|
||||
> initialization buffer. The worker thread stack dominates the static footprint.
|
||||
|
||||
### 3.5.2 Dynamic Memory
|
||||
|
||||
|
||||
@@ -21,8 +21,8 @@ The authoritative `[telemetry]` example lives in `cfg/xrpld-example.cfg`. Teleme
|
||||
| `endpoint` | string | `http://localhost:4318/v1/traces` | OTLP/HTTP collector endpoint |
|
||||
| `use_tls` | bool | `false` | Enable TLS for exporter connection |
|
||||
| `tls_ca_cert` | string | `""` | Path to CA certificate file |
|
||||
| `tls_client_cert` | string | `""` | Path to node's client certificate (PEM) for mutual TLS; empty = one-way TLS |
|
||||
| `tls_client_key` | string | `""` | Path to private key (PEM) for `tls_client_cert`; required when it is set |
|
||||
| `tls_client_cert` | string | `""` | Path to node's client certificate (PEM) for mutual TLS; requires `use_tls=1`; empty = one-way TLS |
|
||||
| `tls_client_key` | string | `""` | Path to private key (PEM) for `tls_client_cert`; requires `use_tls=1`; required when the cert is set |
|
||||
| `batch_size` | uint | `512` | Spans per export batch |
|
||||
| `batch_delay_ms` | uint | `5000` | Max delay before sending batch (ms) |
|
||||
| `max_queue_size` | uint | `2048` | Maximum queued spans |
|
||||
@@ -168,10 +168,10 @@ flowchart TB
|
||||
|
||||
**Reading the diagram:**
|
||||
|
||||
- **Configuration Sources**: `xrpld.cfg` provides runtime settings (endpoint, sampling) while the CMake flag controls whether telemetry is compiled in at all.
|
||||
- **Configuration Sources**: `xrpld.cfg` provides runtime settings (endpoint, per-component trace toggles) while the CMake flag controls whether telemetry is compiled in at all. Head sampling is fixed at 1.0 and is not a config option; volume reduction happens via tail sampling in the collector.
|
||||
- **Initialization**: `setupTelemetry()` parses config values, then `makeTelemetry()` constructs the provider, processor, and exporter objects.
|
||||
- **Runtime Components**: The `TracerProvider` creates spans, the `BatchProcessor` buffers them, and the `OTLP Exporter` serializes and sends them over the wire.
|
||||
- **OTLP arrow to Collector**: Trace data leaves the xrpld process via OTLP (gRPC or HTTP) and enters the external Collector pipeline.
|
||||
- **OTLP arrow to Collector**: Trace data leaves the xrpld process via OTLP/HTTP and enters the external Collector pipeline. (OTLP/gRPC is future work — see design decisions §2.2.2.)
|
||||
- **Collector Pipeline**: `Receivers` ingest OTLP data, `Processors` apply sampling/filtering/enrichment, and `Exporters` forward traces to storage backends (Tempo, etc.).
|
||||
|
||||
---
|
||||
|
||||
@@ -296,9 +296,11 @@ See [Phase4_taskList.md](./Phase4_taskList.md) for full task details.
|
||||
**Objective**: Wire `TraceContextPropagator` for P2P messages (proposals,
|
||||
validations) to enable true distributed tracing between nodes.
|
||||
|
||||
**Status**: Design documented, NOT implemented. Protobuf fields (field 1001)
|
||||
and `TraceContextPropagator` free functions exist. Wiring deferred until Phase 4a is
|
||||
validated in a multi-node environment.
|
||||
**Status**: Partially implemented. Send-side injection (proposals and
|
||||
validations) and receive-side extraction (`consensus.{proposal,validation}.
|
||||
receive` spans parented on the sender's context) are wired in Phase 4a.
|
||||
Remaining Phase 4b work: relay spans in `share(RCLCxPeerPos)` and multi-node
|
||||
validation of the propagation path.
|
||||
|
||||
**Prerequisites**: Phase 4a complete and validated.
|
||||
|
||||
|
||||
@@ -237,7 +237,7 @@ A Tempo-backed dashboard (uid `xrpld-consensus-health`) with four panels, all dr
|
||||
- **Proposers per Round** (stat): average of the `span.proposers` attribute on `consensus.round` spans.
|
||||
- **Recent Slow Rounds (>5s)** (table): `consensus.round` spans filtered to `duration > 5s`.
|
||||
|
||||
The underlying TraceQL queries are listed in section 7.7.3 and used throughout this doc.
|
||||
Each panel's TraceQL query is described inline in its bullet above.
|
||||
|
||||
### 7.6.2 Node Overview Dashboard
|
||||
|
||||
@@ -261,9 +261,9 @@ Grafana provisions three TraceQL-based alert rules (group `xrpld-tracing-alerts`
|
||||
- **RPC Error Rate Spike** (critical, `for: 2m`): fires when the error rate across `rpc.command.*` spans exceeds 5%. Error _rate_ is a ratio, so it must divide the error-span rate by the total-span rate — a single TraceQL `rate()` returns spans/second, not a percentage, and would fire on traffic volume alone. This uses span metrics emitted by the collector's `spanmetrics` connector (Prometheus datasource), not a TraceQL query:
|
||||
|
||||
```
|
||||
sum(rate(calls_total{service_name="xrpld", span_name=~"rpc.command.*", status_code="STATUS_CODE_ERROR"}[5m]))
|
||||
sum(rate(traces_spanmetrics_calls_total{service_name="xrpld", span_name=~"rpc.command.*", status_code="STATUS_CODE_ERROR"}[5m]))
|
||||
/
|
||||
sum(rate(calls_total{service_name="xrpld", span_name=~"rpc.command.*"}[5m]))
|
||||
sum(rate(traces_spanmetrics_calls_total{service_name="xrpld", span_name=~"rpc.command.*"}[5m]))
|
||||
> 0.05
|
||||
```
|
||||
|
||||
|
||||
@@ -211,7 +211,6 @@ flowchart TB
|
||||
| [Phase9_taskList.md](./Phase9_taskList.md) | Internal metric instrumentation gap fill (future) |
|
||||
| [Phase10_taskList.md](./Phase10_taskList.md) | Synthetic workload generation & validation (future) |
|
||||
| [Phase11_taskList.md](./Phase11_taskList.md) | Third-party data collection pipelines (future) |
|
||||
| [presentation.md](./presentation.md) | Presentation slides for OpenTelemetry plan overview |
|
||||
|
||||
> **Note**: Phases 1 and 6 do not have separate task list files. Phase 1 tasks are documented in [06-implementation-phases.md §6.2](./06-implementation-phases.md). Phase 6 tasks are documented in [06-implementation-phases.md §6.7](./06-implementation-phases.md).
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# [OpenTelemetry](00-tracing-fundamentals.md) Distributed Tracing Implementation Plan for xrpld (xrpld)
|
||||
# [OpenTelemetry](00-tracing-fundamentals.md) Distributed Tracing Implementation Plan for xrpld
|
||||
|
||||
## Executive Summary
|
||||
|
||||
@@ -134,7 +134,7 @@ Key trace points span across transaction submission via RPC, peer-to-peer messag
|
||||
|
||||
> **OTLP** = OpenTelemetry Protocol | **CNCF** = Cloud Native Computing Foundation
|
||||
|
||||
The OpenTelemetry C++ SDK is selected for its CNCF backing, active development, and native performance characteristics. Traces are exported via OTLP/gRPC (primary) or OTLP/HTTP (fallback) to an OpenTelemetry Collector, which provides flexible routing and sampling.
|
||||
The OpenTelemetry C++ SDK is selected for its CNCF backing, active development, and native performance characteristics. Traces are exported via OTLP/HTTP to an OpenTelemetry Collector, which provides flexible routing and sampling. OTLP/gRPC is planned future work (see design decisions §2.2.2).
|
||||
|
||||
Span naming follows a hierarchical `<component>.<operation>` convention (e.g., `rpc.submit`, `tx.relay`, `consensus.round`). Context propagation uses W3C Trace Context headers for HTTP and embedded Protocol Buffer fields for P2P messages. The implementation coexists with existing PerfLog and Insight observability systems through correlation IDs.
|
||||
|
||||
@@ -158,7 +158,7 @@ Performance optimization strategies include head sampling fixed at 100% (intenti
|
||||
|
||||
> **OTLP** = OpenTelemetry Protocol | **APM** = Application Performance Monitoring
|
||||
|
||||
Configuration is handled through the `[telemetry]` section in `xrpld.cfg` with options for enabling/disabling, exporter selection, endpoint configuration, sampling ratios, and component-level filtering. CMake integration includes a `XRPL_ENABLE_TELEMETRY` option for compile-time control.
|
||||
Configuration is handled through the `[telemetry]` section in `xrpld.cfg` with options for enabling/disabling, exporter selection, endpoint configuration, and component-level filtering. Head sampling is fixed at 1.0 (not operator-configurable); volume reduction is done by tail sampling in the collector. CMake integration includes a `XRPL_ENABLE_TELEMETRY` option for compile-time control.
|
||||
|
||||
OpenTelemetry Collector configurations are provided for development and production (with tail-based sampling, Tempo, and Elastic APM). Docker Compose examples enable quick local development environment setup.
|
||||
|
||||
|
||||
@@ -169,11 +169,12 @@ Node health (`amendment_blocked`, `server_state`) is not part of the telemetry s
|
||||
| `rpc.process` | `batch_size` | int64 | `params.size()` (only when batch) |
|
||||
| `rpc.ws_message` | `command` | string | `jv[command]` or `jv[method]` |
|
||||
| `rpc.command.*` | `load_type` | string | `context.loadType.label()` |
|
||||
| `pathfind.compute` | `pathfind_dest_amount` | string | `saDstAmount_.getFullText()` |
|
||||
| `pathfind.compute` | `pathfind_dest_currency` | string | `to_string(saDstAmount_.asset())` |
|
||||
| `pathfind.discover` | `pathfind_num_source_assets` | int64 | `sourceAssets.size()` |
|
||||
|
||||
**New attr keys**: `RpcSpanNames.h` (`isBatch`, `batchSize`, `loadType`), `PathFindSpanNames.h` (`destAmount`, `destCurrency`, `numSourceAssets`).
|
||||
_Note: `pathfind_dest_amount` was removed — the destination amount is a financial value excluded by the privacy policy (design §2.4.4)._
|
||||
|
||||
**New attr keys**: `RpcSpanNames.h` (`isBatch`, `batchSize`, `loadType`), `PathFindSpanNames.h` (`destCurrency`, `numSourceAssets`).
|
||||
|
||||
**Modified files**:
|
||||
|
||||
|
||||
@@ -166,7 +166,8 @@
|
||||
|
||||
## Task 3.6: Context Propagation in Transaction Relay
|
||||
|
||||
**Status**: COMPLETE
|
||||
**Status**: COMPLETE (transaction relay). Consensus proposal/validation
|
||||
propagation is deferred to Phase 4 — see "Planned (Phase 4)" below.
|
||||
|
||||
**Objective**: Ensure trace context flows correctly when transactions are relayed between peers, creating linked spans across nodes.
|
||||
|
||||
@@ -178,17 +179,6 @@
|
||||
wired in PeerImp) extracts the parent span_id and creates the tx.receive span
|
||||
as a child of the sender's tx.process span.
|
||||
|
||||
- **Proposal send/receive**: `RCLConsensus::Adaptor::propose()` injects the
|
||||
current thread's active span context into the `TMProposeSet` protobuf via
|
||||
`telemetry::injectToProtobuf()`. PeerImp creates a
|
||||
`consensus.proposal.receive` span that extracts the sender's trace context
|
||||
as parent (via `ConsensusReceiveTracing.h`).
|
||||
|
||||
- **Validation send/receive**: `RCLConsensus::Adaptor::validate()` injects
|
||||
the current thread's active span context into the `TMValidation` protobuf.
|
||||
PeerImp creates a `consensus.validation.receive` span that extracts the
|
||||
sender's trace context as parent.
|
||||
|
||||
- **Edge cases**: Missing trace context (older peers) degrades gracefully to
|
||||
standalone spans. Invalid/corrupted context is treated as absent. Trace
|
||||
flags are propagated and respected.
|
||||
@@ -200,20 +190,36 @@
|
||||
- `PropagationHelpers.h` — `injectSpanContext(SpanGuard&, proto)` bridge
|
||||
between SpanGuard and protobuf TraceContext.
|
||||
- `TraceContextPropagator.h` — `injectToProtobuf(ctx, proto)` for
|
||||
same-thread injection via OTel RuntimeContext (used in propose/validate).
|
||||
- `ConsensusReceiveTracing.h` — `proposalReceiveSpan()` and
|
||||
`validationReceiveSpan()` helper functions that create receive spans with
|
||||
optional parent context extraction from incoming protobuf messages.
|
||||
same-thread injection via OTel RuntimeContext.
|
||||
|
||||
**Key modified files**:
|
||||
|
||||
- `src/xrpld/app/misc/NetworkOPs.cpp` — tx relay injection
|
||||
- `src/xrpld/app/consensus/RCLConsensus.cpp` — proposal/validation send injection
|
||||
- `src/xrpld/overlay/detail/PeerImp.cpp` — proposal/validation receive spans
|
||||
- `include/xrpl/telemetry/SpanGuard.h` — `TraceBytes` struct, `getTraceBytes()`
|
||||
- `src/libxrpl/telemetry/SpanGuard.cpp` — `getTraceBytes()` implementation
|
||||
- `src/xrpld/telemetry/PropagationHelpers.h` — inject helpers (new file)
|
||||
- `src/xrpld/telemetry/ConsensusReceiveTracing.h` — receive span helpers (new file)
|
||||
|
||||
**Planned (Phase 4 — not in this PR)**:
|
||||
|
||||
The consensus proposal/validation propagation below is Phase 4 scope and is
|
||||
not implemented on this branch. It is listed here only to record the intended
|
||||
design.
|
||||
|
||||
- **Proposal send/receive**: `RCLConsensus::Adaptor::propose()` injects the
|
||||
current thread's active span context into the `TMProposeSet` protobuf via
|
||||
`telemetry::injectToProtobuf()`. PeerImp creates a
|
||||
`consensus.proposal.receive` span that extracts the sender's trace context
|
||||
as parent (via `ConsensusReceiveTracing.h`).
|
||||
|
||||
- **Validation send/receive**: `RCLConsensus::Adaptor::validate()` injects
|
||||
the current thread's active span context into the `TMValidation` protobuf.
|
||||
PeerImp creates a `consensus.validation.receive` span that extracts the
|
||||
sender's trace context as parent.
|
||||
|
||||
- Planned files: `src/xrpld/app/consensus/RCLConsensus.cpp` (send injection),
|
||||
`src/xrpld/overlay/detail/PeerImp.cpp` (receive spans),
|
||||
`src/xrpld/telemetry/ConsensusReceiveTracing.h` (receive span helpers,
|
||||
new file).
|
||||
|
||||
**Reference**:
|
||||
|
||||
@@ -396,7 +402,7 @@ This gives the best of both worlds: guaranteed cross-node correlation via determ
|
||||
|
||||
- `src/xrpld/overlay/detail/PeerImp.cpp` — restructured span creation
|
||||
- `src/xrpld/app/misc/NetworkOPs.cpp` — deterministic context for tx.process
|
||||
- `src/xrpld/app/misc/TxSpanNames.h` — new `traceStrategy` attribute constant
|
||||
- `src/xrpld/telemetry/TxSpanNames.h` — new `traceStrategy` attribute constant
|
||||
- New or shared utility for `createDeterministicTxContext()` (location TBD: could be
|
||||
a shared header like `include/xrpl/telemetry/DeterministicContext.h`, or file-local
|
||||
if only used in two places)
|
||||
@@ -431,7 +437,7 @@ This gives the best of both worlds: guaranteed cross-node correlation via determ
|
||||
- `txSpan(category, group, name, txHash, parentCtx)` — child span (deterministic
|
||||
trace_id combined with protobuf-extracted parent span_id for relay ordering).
|
||||
|
||||
- **`TxTracing.h` helper functions** (`src/xrpld/overlay/detail/TxTracing.h`):
|
||||
- **`TxTracing.h` helper functions** (`src/xrpld/telemetry/TxTracing.h`):
|
||||
File-local helpers that wrap `SpanGuard::txSpan()` for the two main PeerImp call
|
||||
sites:
|
||||
- `txReceiveSpan(txHash, parentCtx)` — creates `tx.receive` span with
|
||||
@@ -495,7 +501,7 @@ This gives the best of both worlds: guaranteed cross-node correlation via determ
|
||||
|
||||
**Modified files**:
|
||||
|
||||
- `src/xrpld/app/misc/TxSpanNames.h`
|
||||
- `src/xrpld/telemetry/TxSpanNames.h`
|
||||
- `src/xrpld/app/misc/detail/TxQSpanNames.h`
|
||||
- `src/xrpld/app/misc/NetworkOPs.cpp`
|
||||
- `src/xrpld/overlay/detail/PeerImp.cpp`
|
||||
|
||||
@@ -95,10 +95,16 @@
|
||||
- Creates `consensus.proposal.receive` span
|
||||
- Sets `trusted` attribute (bool)
|
||||
|
||||
**Not implemented** (deferred to Phase 4b — cross-node propagation):
|
||||
**Done here** (cross-node propagation, send + receive):
|
||||
|
||||
- `consensus.proposal.relay` span in `share(RCLCxPeerPos)` — requires trace context injection
|
||||
- Trace context injection/extraction for `TMProposeSet::trace_context`
|
||||
- Trace context injection for `TMProposeSet::trace_context` in `propose()`
|
||||
- Receive-side extraction in `PeerImp::onMessage(TMProposeSet)` via
|
||||
`telemetry::proposalReceiveSpan()` (parents the receive span on the
|
||||
sender's context when a valid `trace_context` is present)
|
||||
|
||||
**Not implemented** (deferred to Phase 4b):
|
||||
|
||||
- `consensus.proposal.relay` span in `share(RCLCxPeerPos)`
|
||||
|
||||
**Key modified files**:
|
||||
|
||||
|
||||
@@ -107,13 +107,13 @@ Tempo/Prometheus.
|
||||
- `consensus.validation.send` (Adaptor::validate)
|
||||
- Verify attributes:
|
||||
- `xrpl.consensus.mode` on `consensus.ledger_close`
|
||||
- `xrpl.consensus.proposers` on `consensus.accept`
|
||||
- `proposers` on `consensus.accept`
|
||||
- `xrpl.consensus.ledger.seq` on `consensus.validation.send`
|
||||
|
||||
**Verification**:
|
||||
|
||||
- [ ] Tempo shows `consensus.ledger_close` traces with `xrpl.consensus.mode`
|
||||
- [ ] Tempo shows `consensus.accept` traces with `xrpl.consensus.proposers`
|
||||
- [ ] Tempo shows `consensus.accept` traces with `proposers`
|
||||
- [ ] Tempo shows `consensus.proposal.send` traces
|
||||
- [ ] Tempo shows `consensus.validation.send` traces
|
||||
|
||||
@@ -147,7 +147,7 @@ Tempo/Prometheus.
|
||||
- Prerequisites section
|
||||
- Single-node standalone test (quick verification)
|
||||
- 6-node consensus test (full verification)
|
||||
- Expected span catalog (all 12 span names with attributes)
|
||||
- Expected span catalog (all 11 span names with attributes)
|
||||
- Verification queries (Tempo API, Prometheus API)
|
||||
- Troubleshooting guide
|
||||
|
||||
@@ -156,7 +156,7 @@ Tempo/Prometheus.
|
||||
**Verification**:
|
||||
|
||||
- [ ] Document covers both single-node and multi-node testing
|
||||
- [ ] All 12 span names documented with source file and attributes
|
||||
- [ ] All 11 span names documented with source file and attributes
|
||||
- [ ] Troubleshooting section covers common failure modes
|
||||
|
||||
---
|
||||
|
||||
@@ -1,436 +0,0 @@
|
||||
# OpenTelemetry Observability for xrpld
|
||||
|
||||
> Status: Phases 1-8 shipped. Traces, metrics, logs all live via OTel.
|
||||
|
||||
---
|
||||
|
||||
## Slide 1: Introduction
|
||||
|
||||
> **CNCF** = Cloud Native Computing Foundation | **OTel** = OpenTelemetry
|
||||
|
||||
### What is OpenTelemetry?
|
||||
|
||||
CNCF-backed, vendor-neutral framework for **traces, metrics, and logs** with a single SDK and wire protocol (OTLP).
|
||||
|
||||
### Why OTel for xrpld?
|
||||
|
||||
- **End-to-end TX visibility** — submission → consensus → ledger inclusion
|
||||
- **Cross-node correlation** — shared `trace_id` stitches hops without a central coordinator
|
||||
- **Consensus round analysis** — phase timing across validators
|
||||
- **Incident debugging** — correlated traces, metrics, logs for one query
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A["Node A<br/>tx.receive<br/>trace_id: abc123"] --> B["Node B<br/>tx.relay<br/>trace_id: abc123"] --> C["Node C<br/>tx.validate<br/>trace_id: abc123"] --> D["Node D<br/>ledger.apply<br/>trace_id: abc123"]
|
||||
|
||||
style A fill:#1565c0,stroke:#0d47a1,color:#fff
|
||||
style B fill:#2e7d32,stroke:#1b5e20,color:#fff
|
||||
style C fill:#2e7d32,stroke:#1b5e20,color:#fff
|
||||
style D fill:#e65100,stroke:#bf360c,color:#fff
|
||||
```
|
||||
|
||||
> One trace, four nodes, full lifecycle.
|
||||
|
||||
---
|
||||
|
||||
## Slide 2: Old Stack vs New OTel Stack
|
||||
|
||||
### Side-by-Side
|
||||
|
||||
| Aspect | Before (StatsD + Debug Logs) | After (OTel: Traces + Metrics + Logs) |
|
||||
| ------------------------- | --------------------------------------------------------------------------------- | ------------------------------------------------------------ |
|
||||
| **Metrics** | Beast Insight → StatsD UDP → Graphite | `MetricsRegistry` → OTLP/HTTP → Prometheus |
|
||||
| **Metric inventory** | **~250 metric series** at runtime (28 registrations × overlay traffic categories) | **23 native instruments** × dimensions + RED via spanmetrics |
|
||||
| **Logs** | `beast::Journal` → `debug.log` (grep / tail) | Journal → filelog tail → Loki (structured, queryable) |
|
||||
| **Traces** | None | Telemetry SDK → OTLP → Tempo (cross-node) |
|
||||
| **Correlation** | Timestamp + grep across files | Shared `trace_id` across all 3 signals |
|
||||
| **Format** | Counter/gauge names; free-form log lines | OTLP protobuf; structured records |
|
||||
| **Backend choice** | Locked to StatsD daemon + log files | Vendor-neutral via Collector exporters |
|
||||
| **Cross-node view** | ❌ Not possible | ✅ Native via trace context propagation |
|
||||
| **Histogram p50/p95/p99** | ❌ Counters/gauges only | ✅ Native histograms + spanmetrics |
|
||||
|
||||
### Legacy StatsD Metric Series (~250 total)
|
||||
|
||||
| Category | Series | Notes |
|
||||
| --------------------------- | -------- | ----------------------------------------------------------------------------------- |
|
||||
| **Overlay traffic gauges** | ~224 | 56 `TrafficCount::category` enum × 4 gauges (`Bytes_{In,Out}`, `Messages_{In,Out}`) |
|
||||
| **Peer Finder** | 2 | `Active_{In,Out}bound_Peers` |
|
||||
| **State Accounting** | 10 | `{Disconnected,Connected,Syncing,Tracking,Full}_{duration,transitions}` |
|
||||
| **Ledger** | 4 | `Validated/Published_Ledger_Age`, `mismatch`, `ledger_fetches` |
|
||||
| **RPC / Pathfinding** | 5 | `requests`, `size`, `time`, `pathfind_{fast,full}` |
|
||||
| **JobQueue / IO / Disconn** | 3 | `job_count`, `ios_latency`, `Peer_Disconnects` |
|
||||
| **Total** | **~248** | 28 `make_*` call sites; series count balloons via overlay-category fan-out |
|
||||
|
||||
### Use Case Matrix
|
||||
|
||||
| Scenario | StatsD | Debug Logs | OTel Traces | OTel Metrics | OTel Logs |
|
||||
| ---------------------------------- | ------ | ---------- | ----------- | ------------ | --------- |
|
||||
| "TXs per second?" | ✅ | ❌ | ❌ | ✅ | ❌ |
|
||||
| "Why was this specific TX slow?" | ❌ | ⚠️ | ✅ | ❌ | ⚠️ |
|
||||
| "Which node delayed consensus?" | ❌ | ❌ | ✅ | ❌ | ❌ |
|
||||
| "TX journey across 5 nodes" | ❌ | ❌ | ✅ | ❌ | ❌ |
|
||||
| "Validator error at 14:02" | ❌ | ✅ | ⚠️ | ❌ | ✅ |
|
||||
| "Reproduce rare assertion / crash" | ❌ | ✅ | ❌ | ❌ | ✅ |
|
||||
| "p99 RPC latency by method" | ⚠️ | ❌ | ⚠️ | ✅ | ❌ |
|
||||
|
||||
> Old stack: 2 signals, no correlation, single node. New stack: 3 signals, `trace_id` everywhere, cross-node native.
|
||||
|
||||
---
|
||||
|
||||
## Slide 3: OTel vs Open-Source Alternatives
|
||||
|
||||
| Feature | OpenTelemetry | Jaeger | Zipkin | SkyWalking | Pinpoint | Prometheus |
|
||||
| ------------------- | --------------- | ------------- | --------------- | ---------- | ---------- | ---------- |
|
||||
| **Tracing** | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ |
|
||||
| **Metrics** | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ |
|
||||
| **Logs** | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ |
|
||||
| **C++ SDK** | ✅ Official | ⚠️ Deprecated | ⚠️ Unmaintained | ❌ | ❌ | ✅ |
|
||||
| **Vendor neutral** | ✅ Primary goal | ❌ | ❌ | ❌ | ❌ | ❌ |
|
||||
| **Instrumentation** | Manual + Auto | Manual | Manual | Auto-first | Auto-first | Manual |
|
||||
| **Backend** | Any (exporters) | Self | Self | Self | Self | Self |
|
||||
| **CNCF Status** | Incubating | Graduated | — | Incubating | — | Graduated |
|
||||
|
||||
> Only actively maintained, full-signal C++ option. Backend-agnostic — Tempo/Prometheus/Loki/Elastic/commercial all work without code change.
|
||||
|
||||
---
|
||||
|
||||
## Slide 4: Architecture (Current)
|
||||
|
||||
> **OTLP** = OpenTelemetry Protocol over HTTP/gRPC
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph xrpld["xrpld Node"]
|
||||
direction TB
|
||||
Surfaces["RPC · TX · Consensus · Peer · Ledger · Job"]
|
||||
SDK["Telemetry SDK + MetricsRegistry"]
|
||||
Journal["beast::Journal → debug.log<br/>(trace_id/span_id injected)"]
|
||||
Surfaces --> SDK
|
||||
Surfaces --> Journal
|
||||
end
|
||||
|
||||
SDK -->|"OTLP/HTTP :4318<br/>traces + metrics"| Collector["OTel Collector"]
|
||||
Journal -->|"filelog tail"| Collector
|
||||
|
||||
Collector --> Tempo["Tempo<br/>(traces)"]
|
||||
Collector --> Prom["Prometheus<br/>(metrics)"]
|
||||
Collector --> Loki["Loki<br/>(logs)"]
|
||||
|
||||
Tempo --> Grafana["Grafana<br/>(15 dashboards)"]
|
||||
Prom --> Grafana
|
||||
Loki --> Grafana
|
||||
|
||||
style xrpld fill:#424242,stroke:#212121,color:#fff
|
||||
style SDK fill:#2e7d32,stroke:#1b5e20,color:#fff
|
||||
style Journal fill:#1565c0,stroke:#0d47a1,color:#fff
|
||||
style Collector fill:#e65100,stroke:#bf360c,color:#fff
|
||||
style Grafana fill:#4a148c,stroke:#2e0d57,color:#fff
|
||||
```
|
||||
|
||||
| Component | Role |
|
||||
| ---------------------- | --------------------------------------------------- |
|
||||
| Telemetry SDK | Span creation, trace context, OTLP traces export |
|
||||
| MetricsRegistry | RPC/job/peer/consensus counters, gauges, histograms |
|
||||
| beast::Journal filelog | `debug.log` tailed by Collector, parsed → Loki |
|
||||
| OTel Collector | Receive OTLP + filelog; route to Tempo/Prom/Loki |
|
||||
| Spanmetrics connector | Derives RED metrics from spans (Prometheus) |
|
||||
|
||||
---
|
||||
|
||||
## Slide 5: Signal Coverage
|
||||
|
||||
| Surface | Traces (Spans) | Metrics (OTLP) | Logs (Journal Partition) |
|
||||
| ------------------ | --------------------------------------------------------------- | ---------------------------------------------- | ------------------------------ |
|
||||
| **RPC** | `rpc.request` + handler spans | request count, latency p50/p95/p99, error rate | `RPC*` |
|
||||
| **Transactions** | `tx.receive`, `tx.validate`, `tx.relay`, `tx.apply` | TX/sec by result, fee escalation gauges | `TxQ`, `LedgerMaster` |
|
||||
| **Consensus** | `consensus.round`, `proposal.send/recv`, `validation.send/recv` | round duration, phase histograms, mode gauge | `Consensus`, `LedgerConsensus` |
|
||||
| **Peer / Overlay** | `peer.send`, `peer.receive` per message type | peer count, bytes/sec by msg type, suppression | `Overlay`, `PeerImp` |
|
||||
| **Ledger** | `ledger.close`, `ledger.apply` | close time, TX count, ledger index gauge | `LedgerMaster` |
|
||||
| **Job Queue** | (sampled per type) | queue depth, queue/run duration histograms | `JobQueue` |
|
||||
|
||||
> ~30 distinct span kinds, ~80 metric series, structured logs from 50+ partitions.
|
||||
|
||||
---
|
||||
|
||||
## Slide 6: Context Propagation
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client
|
||||
participant NodeA as Node A
|
||||
participant NodeB as Node B
|
||||
|
||||
Client->>NodeA: Submit TX (no context)
|
||||
Note over NodeA: Create trace_id: abc123<br/>span: tx.receive
|
||||
NodeA->>NodeB: Relay TX (TraceContext field, ~29B)
|
||||
Note over NodeB: Link trace_id: abc123<br/>span: tx.relay (parent: A)
|
||||
```
|
||||
|
||||
| Carrier | Mechanism |
|
||||
| --------------------- | ------------------------------------------ |
|
||||
| HTTP / WebSocket RPC | W3C `traceparent` header |
|
||||
| P2P protobuf | `TraceContext` extension field per message |
|
||||
| Internal job dispatch | Thread-local context + `SpanGuard` |
|
||||
|
||||
| Field | Size | Description |
|
||||
| ------------- | --------- | ------------------------------------- |
|
||||
| `trace_id` | 16 bytes | Trace correlation key |
|
||||
| `span_id` | 8 bytes | Parent span on receiver |
|
||||
| `trace_flags` | 1 byte | Sampling decision |
|
||||
| `trace_state` | 0-4 bytes | Optional vendor data |
|
||||
| **Total** | **~29 B** | Per traced P2P message (~1-6% of msg) |
|
||||
|
||||
---
|
||||
|
||||
## Slide 7: Performance Overhead
|
||||
|
||||
| Metric | Overhead | Driver |
|
||||
| ----------------- | ---------- | --------------------------------------------------- |
|
||||
| **CPU** | 1-3% | ~4 μs/TX span work (~2% at 25 TPS baseline) |
|
||||
| **Memory** | ~10 MB | SDK statics + worker stack + 2048-span export queue |
|
||||
| **Network** | 10-50 KB/s | OTLP export + 29 B P2P context per traced msg |
|
||||
| **Latency (p99)** | <2% | TX path dominates; RPC and consensus negligible |
|
||||
|
||||
### Kill Switches
|
||||
|
||||
1. `enabled=0` in `xrpld.cfg` → instant disable, no restart
|
||||
2. Build with `XRPL_ENABLE_TELEMETRY=OFF` → zero overhead (no-op stubs)
|
||||
3. Reduce `sampling_ratio` → linear export reduction
|
||||
|
||||
> Derivations and per-component cost tables: see [03-implementation-strategy.md §3.5.4](./03-implementation-strategy.md#354-performance-data-sources).
|
||||
|
||||
---
|
||||
|
||||
## Slide 8: Sampling — Head vs Tail
|
||||
|
||||
| | Head Sampling | Tail Sampling |
|
||||
| ------------------------ | --------------------------------- | -------------------------------------- |
|
||||
| **Where** | Inside xrpld (SDK) | OTel Collector (external) |
|
||||
| **Decision time** | Trace start (random coin flip) | Trace end (after all spans buffered) |
|
||||
| **Knows trace content?** | No | Yes — error, latency, span kind |
|
||||
| **xrpld overhead** | Lowest (drop = no-op) | Higher (export 100%) |
|
||||
| **Captures all errors?** | No | **Yes** (status_code policy) |
|
||||
| **Captures slow ops?** | No | **Yes** (latency policy) |
|
||||
| **Config** | `xrpld.cfg`: `sampling_ratio=0.1` | `tail_sampling` processor in collector |
|
||||
| **Best for** | Steady-state high volume | Anomaly + error retention |
|
||||
|
||||
### Recommended Layered Strategy
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
xrpld["xrpld<br/>sampling_ratio=1.0<br/>(export all)"] -->|"100%"| col["Collector<br/>tail_sampling:<br/>errors + slow + 10% random"]
|
||||
col -->|"~15-20% kept"| tempo["Tempo storage"]
|
||||
|
||||
style xrpld fill:#424242,stroke:#212121,color:#fff
|
||||
style col fill:#1565c0,stroke:#0d47a1,color:#fff
|
||||
style tempo fill:#2e7d32,stroke:#1b5e20,color:#fff
|
||||
```
|
||||
|
||||
> If Collector resource pressure: drop `sampling_ratio` to 0.5 — still enough trace volume for tail decisions.
|
||||
|
||||
---
|
||||
|
||||
## Slide 9: Data Collection & Privacy
|
||||
|
||||
### Collected (operational metadata)
|
||||
|
||||
| Category | Attributes |
|
||||
| ----------- | -------------------------------------------------------------------- |
|
||||
| Transaction | `tx.hash`, `tx.type`, `tx.result`, `tx.fee`, `ledger_index` |
|
||||
| Consensus | `round`, `phase`, `mode`, `proposers`, `duration_ms` |
|
||||
| RPC | `command`, `version`, `status`, `duration_ms` |
|
||||
| Peer | `peer.id` (public key), `latency_ms`, `message.type`, `message.size` |
|
||||
| Ledger | `ledger.hash`, `ledger.index`, `close_time`, `tx_count` |
|
||||
| Job | `job.type`, `queue_ms`, `worker` |
|
||||
|
||||
### NOT Collected (hard exclusions)
|
||||
|
||||
> ❌ Private keys · ❌ Account balances · ❌ Transaction amounts · ❌ Raw payloads · ❌ Personal data · ⚙️ IP addresses (configurable)
|
||||
|
||||
### Privacy Mechanisms
|
||||
|
||||
| Mechanism | Description |
|
||||
| ---------------------- | --------------------------------------------------------- |
|
||||
| Account hashing | `xrpl.tx.account` hashed at Collector before storage |
|
||||
| Configurable redaction | Sensitive attributes excluded via Collector config |
|
||||
| Sampling | 10% default reduces exposure |
|
||||
| Local control | Operator owns Collector → backend pipeline |
|
||||
| No raw payloads | Span attributes are metadata only, never message contents |
|
||||
|
||||
> Principle: telemetry records **operational metadata** — never financial or personal content.
|
||||
|
||||
---
|
||||
|
||||
## Slide 10: Implementation Timeline
|
||||
|
||||
```mermaid
|
||||
gantt
|
||||
title OpenTelemetry Rollout
|
||||
dateFormat YYYY-MM-DD
|
||||
axisFormat Week %W
|
||||
|
||||
section Done
|
||||
Phase 1 Core Infra :done, p1, 2024-01-01, 2w
|
||||
Phase 2 RPC Tracing :done, p2, after p1, 2w
|
||||
Phase 3 TX Tracing :done, p3, after p2, 2w
|
||||
Phase 4 Consensus :done, p4, after p3, 2w
|
||||
Phase 5 Docs/Deploy :done, p5, after p4, 1w
|
||||
Phase 6 StatsD Bridge :done, p6, after p5, 1w
|
||||
Phase 7 Native OTel Metrics :done, p7, after p6, 2w
|
||||
Phase 8 Log-Trace Correlation :done, p8, after p7, 1w
|
||||
Phase 9 Metric Gap Fill :active, p9, after p8, 2w
|
||||
|
||||
section Future
|
||||
Phase 10 Workload Validation :p10, after p9, 2w
|
||||
Phase 11 3rd-Party Pipelines :p11, after p10, 3w
|
||||
```
|
||||
|
||||
| Phase | Focus | Status |
|
||||
| ----- | ------------------------------------------- | ------- |
|
||||
| 1 | SDK integration, Telemetry, Config | ✅ Done |
|
||||
| 2 | RPC handler spans, HTTP context | ✅ Done |
|
||||
| 3 | TX spans, P2P protobuf context | ✅ Done |
|
||||
| 4 | Consensus rounds, proposal/validation | ✅ Done |
|
||||
| 5 | Runbook, dashboards, deployment | ✅ Done |
|
||||
| 6 | StatsD bridge (interim) | ✅ Done |
|
||||
| 7 | Native OTel metrics (replace Beast Insight) | ✅ Done |
|
||||
| 8 | Log-trace correlation (Loki) | ✅ Done |
|
||||
| 9 | Internal metric gap fill | ✅ Done |
|
||||
|
||||
---
|
||||
|
||||
## Slide 11: Current State — What Shipped
|
||||
|
||||
### By Signal
|
||||
|
||||
| Signal | Backend | Status | Notes |
|
||||
| ----------- | ---------- | ------ | -------------------------------------------------------- |
|
||||
| **Traces** | Tempo | ✅ | All 6 surfaces instrumented; cross-node propagation live |
|
||||
| **Metrics** | Prometheus | ✅ | Native OTLP; Beast Insight retired |
|
||||
| **Logs** | Loki | ✅ | filelog tailing `debug.log`; `trace_id` injected |
|
||||
|
||||
### By Surface
|
||||
|
||||
| Surface | Spans Live | Metrics Live | Notes |
|
||||
| -------------- | ---------- | ------------ | --------------------------------------------------- |
|
||||
| RPC | ✅ | ✅ | Handler + pathfinding + TxQ |
|
||||
| Transactions | ✅ | ✅ | Receive, validate, relay, apply |
|
||||
| Consensus | ✅ | ✅ | Round + proposal/validation send+receive (Phase 4a) |
|
||||
| Peer / Overlay | ✅ | ✅ | Per-msg-type send/receive |
|
||||
| Ledger | ✅ | ✅ | Close + apply |
|
||||
| Job Queue | ✅ | ✅ | Queue depth + duration histograms |
|
||||
|
||||
### Stack Live
|
||||
|
||||
| Component | Version |
|
||||
| -------------------------- | ------- |
|
||||
| OTel Collector (contrib) | 0.121.0 |
|
||||
| Grafana Tempo | 2.7.2 |
|
||||
| Grafana Loki | 3.4.2 |
|
||||
| Prometheus | latest |
|
||||
| Grafana | 11.5.2 |
|
||||
| **Dashboards provisioned** | **15** |
|
||||
|
||||
---
|
||||
|
||||
## Slide 12: Future Phases
|
||||
|
||||
### Phase 10 — Synthetic Workload Validation
|
||||
|
||||
| Aspect | Detail |
|
||||
| ----------- | ------------------------------------------------------------------ |
|
||||
| Goal | Drive instrumented surfaces under reproducible load |
|
||||
| Why | Validate dashboards, catch regressions, measure overhead at scale |
|
||||
| Deliverable | Workload generator + assertion suite (RPC/TX/peer churn scenarios) |
|
||||
| Effort | ~2 weeks |
|
||||
|
||||
### Phase 11 — Admin-RPC Receiver (`xrpl_*` metrics)
|
||||
|
||||
| Aspect | Detail |
|
||||
| ----------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Goal | Custom Go OTel Collector receiver polls xrpld admin RPC, emits `xrpl_*` Prometheus metrics |
|
||||
| Why | Admin-RPC-only data has no native export — every consumer reinvents JSON-RPC polling |
|
||||
| Scope | `validators` (UNL, listed keys), `feature` (amendments), `peers` (per-peer detail), `amm_info`, `book_offers`, `fee` (detail tiers) |
|
||||
| Excluded | `server_info` / `get_counts` basics — Phase 9 (#6513) already ships `xrpld_server_info` + 14 gauges/histograms natively from in-process state |
|
||||
| Deliverable | Go receiver plugin + custom Collector binary + 4 Grafana dashboards (UNL, amendments, AMM, DEX) + Prometheus alerts |
|
||||
| Effort | ~3 weeks |
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
rpc["xrpld admin RPC<br/>(validators, feature, peers,<br/>amm_info, book_offers, fee)"] -->|JSON-RPC poll| recv["Custom Go receiver<br/>(in Collector)"]
|
||||
recv -->|xrpl_* metrics| prom["Prometheus"]
|
||||
prom --> graf["Grafana dashboards"]
|
||||
|
||||
style rpc fill:#2e7d32,stroke:#1b5e20,color:#fff
|
||||
style recv fill:#1565c0,stroke:#0d47a1,color:#fff
|
||||
style prom fill:#e65100,stroke:#bf360c,color:#fff
|
||||
style graf fill:#6a1b9a,stroke:#4a148c,color:#fff
|
||||
```
|
||||
|
||||
> Phase 11 fills the gap above Phase 9 — data only reachable via admin RPC, not via in-process metric callbacks.
|
||||
|
||||
---
|
||||
|
||||
## Slide 11: External Dashboard Parity (Phase 7+)
|
||||
|
||||
### Bridging Community Monitoring into Native OTel
|
||||
|
||||
The community [xrpl-validator-dashboard](https://github.com/realgrapedrop/xrpl-validator-dashboard) provides 86 metrics for validator operators. We integrated the 29 missing metrics natively into the OTel pipeline.
|
||||
|
||||
### New Metric Categories
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph "New Observable Gauges"
|
||||
VH["Validator Health<br/>amendment_blocked, UNL expiry,<br/>quorum"]
|
||||
PQ["Peer Quality<br/>P90 latency, insane peers,<br/>version awareness"]
|
||||
LE["Ledger Economy<br/>fees, reserves, tx rate,<br/>ledger age"]
|
||||
ST["State Tracking<br/>state value 0-6,<br/>time in state"]
|
||||
VA["Validation Agreement<br/>1h/24h agreement %,<br/>agreements, misses"]
|
||||
end
|
||||
|
||||
subgraph "Counters"
|
||||
C1["ledgers_closed_total"]
|
||||
C2["validations_sent_total"]
|
||||
C3["state_changes_total"]
|
||||
end
|
||||
|
||||
style VH fill:#1565c0,color:#fff
|
||||
style PQ fill:#2e7d32,color:#fff
|
||||
style LE fill:#e65100,color:#fff
|
||||
style ST fill:#6a1b9a,color:#fff
|
||||
style VA fill:#c62828,color:#fff
|
||||
style C1 fill:#37474f,color:#fff
|
||||
style C2 fill:#37474f,color:#fff
|
||||
style C3 fill:#37474f,color:#fff
|
||||
```
|
||||
|
||||
### ValidationTracker — Agreement Computation
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as RCLConsensus
|
||||
participant VT as ValidationTracker
|
||||
participant MR as MetricsRegistry
|
||||
participant P as Prometheus
|
||||
|
||||
C->>VT: recordOurValidation(hash, seq)
|
||||
Note over VT: Stores pending event
|
||||
C->>VT: recordNetworkValidation(hash, seq)
|
||||
Note over VT: Marks network validated
|
||||
MR->>VT: reconcile() [every 10s]
|
||||
Note over VT: After 8s grace period:<br/>both validated → agreed<br/>only one → missed<br/>5min late repair window
|
||||
MR->>P: Export agreement_pct_1h/24h
|
||||
```
|
||||
|
||||
### New Grafana Dashboards
|
||||
|
||||
| Dashboard | Key Panels |
|
||||
| ---------------- | --------------------------------------------------- |
|
||||
| Validator Health | Agreement %, amendment blocked, quorum, state value |
|
||||
| Peer Quality | P90 latency, version awareness, upgrade recommended |
|
||||
| Ledger Economy | Base fee, reserves, ledger age, transaction rate |
|
||||
|
||||
---
|
||||
|
||||
_End of Presentation_
|
||||
@@ -20,7 +20,7 @@ The most effective way to prevent spoofing from external sources is to treat you
|
||||
server {
|
||||
listen 80;
|
||||
|
||||
location {
|
||||
location / {
|
||||
# Clear out untrusted incoming trace headers
|
||||
proxy_set_header traceparent "";
|
||||
proxy_set_header tracestate "";
|
||||
@@ -92,10 +92,11 @@ receivers:
|
||||
grpc:
|
||||
endpoint: 0.0.0.0:4317
|
||||
tls:
|
||||
# Setting client_ca_file makes the collector require and verify a
|
||||
# client cert, rejecting connections without a trusted one.
|
||||
client_ca_file: /certs/client_ca.pem # CA that signs trusted client certs
|
||||
cert_file: /certs/collector.pem
|
||||
key_file: /certs/collector.key
|
||||
auth_type: require_and_verify_client_cert # Rejects unauthorized clients
|
||||
```
|
||||
|
||||
### **Approach C: Application Layer Authentication (Basic Auth Extension)**
|
||||
@@ -175,7 +176,7 @@ xrpld has **two distinct attack surfaces**, not one. The original guide conflate
|
||||
│ │ OTLP/gRPC
|
||||
│ │ + mTLS
|
||||
│ ▼
|
||||
└───────────────────────────────────────── [require_and_verify_client_cert]
|
||||
└───────────────────────────────────────── [client_ca_file: verify client cert]
|
||||
OTel Collector
|
||||
(in private subnet, NetPol)
|
||||
```
|
||||
@@ -196,15 +197,15 @@ The guide's NGINX header stripping and OTTL stale-span filtering target HTTP gat
|
||||
|
||||
Evaluated for the across-network deployment shape:
|
||||
|
||||
| Approach | Across-network fit | Cost | Verdict |
|
||||
| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | ---------------------------------- |
|
||||
| **A. NetworkPolicy / firewall** | Necessary baseline (don't expose `4317`/`4318` to the internet), but insufficient on its own when traffic genuinely crosses networks — you cannot NetworkPolicy the public internet. | Cheap. | **Defense-in-depth, not primary.** |
|
||||
| **B. mTLS** | Strongest fit. Every xrpld node holds a client cert; collector verifies with `require_and_verify_client_cert`. Encrypts in transit (raw OTLP over the internet leaks transaction patterns and validator identity). Compromised node = revoke one cert, no shared secret to rotate everywhere. | Cert issuance + rotation pipeline. | **Primary.** |
|
||||
| **C. Basic Auth** | Worst shape for this topology. Single shared password across all xrpld nodes — one leaked node config compromises the whole fleet. Doesn't encrypt; you'd need TLS underneath anyway, at which point you're 80% of the way to mTLS. | Cheap to set up, expensive to operate (rotation across N operators). | **Skip.** |
|
||||
| Approach | Across-network fit | Cost | Verdict |
|
||||
| ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | ---------------------------------- |
|
||||
| **A. NetworkPolicy / firewall** | Necessary baseline (don't expose `4317`/`4318` to the internet), but insufficient on its own when traffic genuinely crosses networks — you cannot NetworkPolicy the public internet. | Cheap. | **Defense-in-depth, not primary.** |
|
||||
| **B. mTLS** | Strongest fit. Every xrpld node holds a client cert; the collector verifies it via `client_ca_file` in the receiver's `tls` block. Encrypts in transit (raw OTLP over the internet leaks transaction patterns and validator identity). Compromised node = revoke one cert, no shared secret to rotate everywhere. | Cert issuance + rotation pipeline. | **Primary.** |
|
||||
| **C. Basic Auth** | Worst shape for this topology. Single shared password across all xrpld nodes — one leaked node config compromises the whole fleet. Doesn't encrypt; you'd need TLS underneath anyway, at which point you're 80% of the way to mTLS. | Cheap to set up, expensive to operate (rotation across N operators). | **Skip.** |
|
||||
|
||||
## Decision
|
||||
|
||||
**Primary defense:** mTLS (Approach B) on the collector's OTLP receivers, with `auth_type: require_and_verify_client_cert`.
|
||||
**Primary defense:** mTLS (Approach B) on the collector's OTLP receivers. The collector requires and verifies each client certificate when `client_ca_file` is set in the receiver's `tls` block (there is no `auth_type` field — setting `client_ca_file` is what enforces client-cert verification).
|
||||
|
||||
**Defense-in-depth:** NetworkPolicy / firewall rules (Approach A) so `4317`/`4318` are never reachable from outside the expected operator subnets even if mTLS were misconfigured.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user