mirror of
https://github.com/XRPLF/rippled.git
synced 2026-04-29 15:37:57 +00:00
301 lines
13 KiB
Markdown
301 lines
13 KiB
Markdown
# Phase 3: Transaction Tracing Task List
|
|
|
|
> **Goal**: Trace the full transaction lifecycle from RPC submission through peer relay, including cross-node context propagation via Protocol Buffer extensions. This is the WALK phase that demonstrates true distributed tracing.
|
|
>
|
|
> **Scope**: Protocol Buffer `TraceContext` message, context serialization, PeerImp transaction instrumentation, NetworkOPs processing instrumentation, HashRouter visibility, and multi-node relay context propagation.
|
|
>
|
|
> **Branch**: `pratik/otel-phase3-tx-tracing` (from `pratik/otel-phase2-rpc-tracing`)
|
|
|
|
### Related Plan Documents
|
|
|
|
| Document | Relevance |
|
|
| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------ |
|
|
| [04-code-samples.md](./04-code-samples.md) | TraceContext protobuf (§4.4.1), PeerImp instrumentation (§4.5.1), context serialization (§4.4.2) |
|
|
| [01-architecture-analysis.md](./01-architecture-analysis.md) | Transaction flow (§1.3), key trace points (§1.6) |
|
|
| [06-implementation-phases.md](./06-implementation-phases.md) | Phase 3 tasks (§6.4), definition of done (§6.11.3) |
|
|
| [02-design-decisions.md](./02-design-decisions.md) | Context propagation design (§2.5), attribute schema (§2.4.3) |
|
|
|
|
---
|
|
|
|
## Task 3.1: Define TraceContext Protocol Buffer Message
|
|
|
|
**Objective**: Add trace context fields to the P2P protocol messages so trace IDs can propagate across nodes.
|
|
|
|
**What to do**:
|
|
|
|
- Edit `include/xrpl/proto/xrpl.proto` (or `src/ripple/proto/ripple.proto`, wherever the proto is):
|
|
- Add `TraceContext` message definition:
|
|
```protobuf
|
|
message TraceContext {
|
|
bytes trace_id = 1; // 16-byte trace identifier
|
|
bytes span_id = 2; // 8-byte span identifier
|
|
uint32 trace_flags = 3; // bit 0 = sampled
|
|
string trace_state = 4; // W3C tracestate value
|
|
}
|
|
```
|
|
- Add `optional TraceContext trace_context = 1001;` to:
|
|
- `TMTransaction`
|
|
- `TMProposeSet` (for Phase 4 use)
|
|
- `TMValidation` (for Phase 4 use)
|
|
- Use high field numbers (1001+) to avoid conflicts with existing fields
|
|
|
|
- Regenerate protobuf C++ code
|
|
|
|
**Key modified files**:
|
|
|
|
- `include/xrpl/proto/xrpl.proto` (or equivalent)
|
|
|
|
**Reference**:
|
|
|
|
- [04-code-samples.md §4.4.1](./04-code-samples.md) — TraceContext message definition
|
|
- [02-design-decisions.md §2.5.2](./02-design-decisions.md) — Protocol buffer context propagation design
|
|
|
|
---
|
|
|
|
## Task 3.2: Implement Protobuf Context Serialization
|
|
|
|
**Objective**: Create utilities to serialize/deserialize OTel trace context to/from protobuf `TraceContext` messages.
|
|
|
|
**What to do**:
|
|
|
|
- Create `include/xrpl/telemetry/TraceContextPropagator.h` (extend from Phase 2 if exists, or add protobuf methods):
|
|
- Add protobuf-specific methods:
|
|
- `static Context extractFromProtobuf(protocol::TraceContext const& proto)` — reconstruct OTel context from protobuf fields
|
|
- `static void injectToProtobuf(Context const& ctx, protocol::TraceContext& proto)` — serialize current span context into protobuf fields
|
|
- Both methods guard behind `#ifdef XRPL_ENABLE_TELEMETRY`
|
|
|
|
- Create/extend `src/libxrpl/telemetry/TraceContextPropagator.cpp`:
|
|
- Implement extraction: read trace_id (16 bytes), span_id (8 bytes), trace_flags from protobuf, construct `SpanContext`, wrap in `Context`
|
|
- Implement injection: get current span from context, serialize its TraceId, SpanId, and TraceFlags into protobuf fields
|
|
|
|
**Key new/modified files**:
|
|
|
|
- `include/xrpl/telemetry/TraceContextPropagator.h`
|
|
- `src/libxrpl/telemetry/TraceContextPropagator.cpp`
|
|
|
|
**Reference**:
|
|
|
|
- [04-code-samples.md §4.4.2](./04-code-samples.md) — Full extract/inject implementation
|
|
|
|
---
|
|
|
|
## Task 3.3: Instrument PeerImp Transaction Handling
|
|
|
|
**Objective**: Add trace spans to the peer-level transaction receive and relay path.
|
|
|
|
**What to do**:
|
|
|
|
- Edit `src/xrpld/overlay/detail/PeerImp.cpp`:
|
|
- In `onMessage(TMTransaction)` / `handleTransaction()`:
|
|
- Extract parent trace context from incoming `TMTransaction::trace_context` field (if present)
|
|
- Create `tx.receive` span as child of extracted context (or new root if none)
|
|
- Set attributes: `xrpl.tx.hash`, `xrpl.peer.id`, `xrpl.tx.status`
|
|
- On HashRouter suppression (duplicate): set `xrpl.tx.suppressed=true`, add `tx.duplicate` event
|
|
- Wrap validation call with child span `tx.validate`
|
|
- Wrap relay with `tx.relay` span
|
|
- When relaying to peers:
|
|
- Inject current trace context into outgoing `TMTransaction::trace_context`
|
|
- Set `xrpl.tx.relay_count` attribute
|
|
|
|
- Include `TracingInstrumentation.h` and use `XRPL_TRACE_TX` macro
|
|
|
|
**Key modified files**:
|
|
|
|
- `src/xrpld/overlay/detail/PeerImp.cpp`
|
|
|
|
**Reference**:
|
|
|
|
- [04-code-samples.md §4.5.1](./04-code-samples.md) — Full PeerImp instrumentation example
|
|
- [01-architecture-analysis.md §1.3](./01-architecture-analysis.md) — Transaction flow diagram
|
|
- [01-architecture-analysis.md §1.6](./01-architecture-analysis.md) — tx.receive trace point
|
|
|
|
---
|
|
|
|
## Task 3.4: Instrument NetworkOPs Transaction Processing
|
|
|
|
**Objective**: Trace the transaction processing pipeline in NetworkOPs, covering both sync and async paths.
|
|
|
|
**What to do**:
|
|
|
|
- Edit `src/xrpld/app/misc/NetworkOPs.cpp`:
|
|
- In `processTransaction()`:
|
|
- Create `tx.process` span
|
|
- Set attributes: `xrpl.tx.hash`, `xrpl.tx.type`, `xrpl.tx.local` (whether from RPC or peer)
|
|
- Record whether sync or async path is taken
|
|
|
|
- In `doTransactionAsync()`:
|
|
- Capture parent context before queuing
|
|
- Create `tx.queue` span with queue depth attribute
|
|
- Add event when transaction is dequeued for processing
|
|
|
|
- In `doTransactionSync()`:
|
|
- Create `tx.process_sync` span
|
|
- Record result (applied, queued, rejected)
|
|
|
|
**Key modified files**:
|
|
|
|
- `src/xrpld/app/misc/NetworkOPs.cpp`
|
|
|
|
**Reference**:
|
|
|
|
- [01-architecture-analysis.md §1.6](./01-architecture-analysis.md) — tx.validate and tx.process trace points
|
|
- [02-design-decisions.md §2.4.3](./02-design-decisions.md) — Transaction attribute schema
|
|
|
|
---
|
|
|
|
## Task 3.5: Instrument HashRouter for Dedup Visibility
|
|
|
|
**Objective**: Make transaction deduplication visible in traces by recording HashRouter decisions as span attributes/events.
|
|
|
|
**What to do**:
|
|
|
|
- Edit `src/xrpld/overlay/detail/PeerImp.cpp` (in handleTransaction):
|
|
- After calling `HashRouter::shouldProcess()` or `addSuppressionPeer()`:
|
|
- Record `xrpl.tx.suppressed` attribute (true/false)
|
|
- Record `xrpl.tx.flags` showing current HashRouter state (SAVED, TRUSTED, etc.)
|
|
- Add `tx.first_seen` or `tx.duplicate` event
|
|
|
|
- This is NOT a modification to HashRouter itself — just recording its decisions as span attributes in the existing PeerImp instrumentation from Task 3.3.
|
|
|
|
**Key modified files**:
|
|
|
|
- `src/xrpld/overlay/detail/PeerImp.cpp` (same changes as 3.3, logically grouped)
|
|
|
|
---
|
|
|
|
## Task 3.6: Context Propagation in Transaction Relay
|
|
|
|
**Objective**: Ensure trace context flows correctly when transactions are relayed between peers, creating linked spans across nodes.
|
|
|
|
**What to do**:
|
|
|
|
- Verify the relay path injects trace context:
|
|
- When `PeerImp` relays a transaction, the `TMTransaction` message should carry `trace_context`
|
|
- When a remote peer receives it, the context is extracted and used as parent
|
|
|
|
- Test context propagation:
|
|
- Manually verify with 2+ node setup that trace IDs match across nodes
|
|
- Confirm parent-child span relationships are correct in Tempo
|
|
|
|
- Handle edge cases:
|
|
- Missing trace context (older peers): create new root span
|
|
- Corrupted trace context: log warning, create new root span
|
|
- Sampled-out traces: respect trace flags
|
|
|
|
**Key modified files**:
|
|
|
|
- `src/xrpld/overlay/detail/PeerImp.cpp`
|
|
- `src/xrpld/overlay/detail/OverlayImpl.cpp` (if relay method needs context param)
|
|
|
|
**Reference**:
|
|
|
|
- [02-design-decisions.md §2.5](./02-design-decisions.md) — Context propagation design
|
|
- [04-code-samples.md §4.5.1](./04-code-samples.md) — Relay context injection pattern
|
|
|
|
---
|
|
|
|
## Task 3.7: Build Verification and Testing
|
|
|
|
**Objective**: Verify all Phase 3 changes compile and work correctly.
|
|
|
|
**What to do**:
|
|
|
|
1. Build with `telemetry=ON` — verify no compilation errors
|
|
2. Build with `telemetry=OFF` — verify no regressions
|
|
3. Run existing unit tests
|
|
4. Verify protobuf regeneration produces correct C++ code
|
|
5. Document any issues encountered
|
|
|
|
**Verification Checklist**:
|
|
|
|
- [ ] Protobuf changes generate valid C++
|
|
- [ ] Build succeeds with telemetry ON
|
|
- [ ] Build succeeds with telemetry OFF
|
|
- [ ] Existing tests pass
|
|
- [ ] No undefined symbols from new telemetry calls
|
|
|
|
---
|
|
|
|
## Task 3.8: Transaction Span Peer Version Attribute
|
|
|
|
> **Source**: [External Dashboard Parity](../docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md) — adds peer version context inspired by the community [xrpl-validator-dashboard](https://github.com/realgrapedrop/xrpl-validator-dashboard).
|
|
>
|
|
> **Upstream**: Phase 2 (RPC span infrastructure must exist).
|
|
> **Downstream**: Phase 10 (validation checks for this attribute).
|
|
|
|
**Objective**: Add the relaying peer's rippled version to `tx.receive` spans so operators can correlate transaction issues with peer version mismatches during network upgrades.
|
|
|
|
**What to do**:
|
|
|
|
- Edit `src/xrpld/overlay/detail/PeerImp.cpp`:
|
|
- In the `tx.receive` span block (after existing `xrpl.peer.id` setAttribute call):
|
|
- Add `xrpl.peer.version` (string) — from `this->getVersion()`
|
|
- Only set if `getVersion()` returns a non-empty string (avoid empty-string attributes)
|
|
|
|
**New span attribute**:
|
|
|
|
| Attribute | Type | Source | Example |
|
|
| ------------------- | ------ | -------------------- | ----------------- |
|
|
| `xrpl.peer.version` | string | `peer->getVersion()` | `"rippled-2.4.0"` |
|
|
|
|
**Rationale**: Transaction relay is where version mismatches cause subtle serialization or validation bugs. Tracing "this tx came from a v2.3.0 peer" helps diagnose compatibility issues. The community dashboard tracks peer versions externally; this brings version awareness into the trace itself.
|
|
|
|
**Key modified files**:
|
|
|
|
- `src/xrpld/overlay/detail/PeerImp.cpp`
|
|
|
|
**Exit Criteria**:
|
|
|
|
- [ ] `tx.receive` spans carry `xrpl.peer.version` attribute with a non-empty version string
|
|
- [ ] Attribute is omitted (not set to empty string) when `getVersion()` returns empty
|
|
- [ ] Attribute visible in Jaeger span detail view
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
| Task | Description | New Files | Modified Files | Depends On |
|
|
| ---- | ----------------------------------- | --------- | -------------- | ---------- |
|
|
| 3.1 | TraceContext protobuf message | 0 | 1 | Phase 2 |
|
|
| 3.2 | Protobuf context serialization | 1-2 | 0 | 3.1 |
|
|
| 3.3 | PeerImp transaction instrumentation | 0 | 1 | 3.2 |
|
|
| 3.4 | NetworkOPs transaction processing | 0 | 1 | Phase 2 |
|
|
| 3.5 | HashRouter dedup visibility | 0 | 1 | 3.3 |
|
|
| 3.6 | Relay context propagation | 0 | 1-2 | 3.3, 3.5 |
|
|
| 3.7 | Build verification and testing | 0 | 0 | 3.1-3.6 |
|
|
| 3.8 | TX span peer version attribute | 0 | 1 | 3.3 |
|
|
|
|
**Parallel work**: Tasks 3.1 and 3.4 can start in parallel. Task 3.2 depends on 3.1. Tasks 3.3 and 3.5 depend on 3.2. Task 3.6 depends on 3.3 and 3.5. Task 3.8 depends on 3.3 (span must exist).
|
|
|
|
**Exit Criteria** (from [06-implementation-phases.md §6.11.3](./06-implementation-phases.md)):
|
|
|
|
- [ ] Transaction traces span across nodes
|
|
- [ ] Trace context in Protocol Buffer messages
|
|
- [ ] HashRouter deduplication visible in traces
|
|
- [ ] <5% overhead on transaction throughput
|
|
|
|
---
|
|
|
|
## Known Issues / Future Work
|
|
|
|
### Propagation utilities not yet wired into P2P flow
|
|
|
|
`extractFromProtobuf()` and `injectToProtobuf()` in `TraceContextPropagator.h`
|
|
are implemented and tested but not called from production code. To enable
|
|
cross-node distributed traces:
|
|
|
|
- Call `injectToProtobuf()` in `PeerImp` when sending `TMTransaction` /
|
|
`TMProposeSet` messages
|
|
- Call `extractFromProtobuf()` in the corresponding message handlers to
|
|
reconstruct the parent span context, then pass it to `startSpan()` as the
|
|
parent
|
|
|
|
This was deferred to validate single-node tracing performance first.
|
|
|
|
### Unused trace_state proto field
|
|
|
|
The `TraceContext.trace_state` field (field 4) in `xrpl.proto` is reserved for
|
|
W3C `tracestate` vendor-specific key-value pairs but is not read or written by
|
|
`TraceContextPropagator`. Wire it when cross-vendor trace propagation is needed.
|
|
No wire cost since proto `optional` fields are zero-cost when absent.
|