mirror of
https://github.com/XRPLF/rippled.git
synced 2026-06-03 08:46:46 +00:00
docs(telemetry): add deterministic TX trace ID design (Task 3.9)
Add trace_id = txHash[0:16] strategy so all nodes handling the same transaction independently produce spans under the same trace_id, combined with protobuf span_id propagation for parent-child ordering. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pratik Mankawde
@@ -253,6 +253,149 @@
|
||||
|
||||
---
|
||||
|
||||
## Task 3.9: Deterministic Transaction Trace ID
|
||||
|
||||
> **Upstream**: Task 3.2 (protobuf serialization), Task 3.3 (PeerImp span exists).
|
||||
> **Downstream**: Phase 10 (workload validation can query by tx hash directly).
|
||||
> **Pattern**: Mirrors the consensus deterministic trace ID in Phase 4a
|
||||
> (`createDeterministicContext` in `RCLConsensus.cpp`), adapted for transactions.
|
||||
|
||||
**Objective**: Derive the trace_id for transaction spans deterministically from the
|
||||
transaction hash so that all nodes handling the same transaction independently produce
|
||||
spans under the same trace_id — regardless of whether protobuf context propagation
|
||||
succeeds.
|
||||
|
||||
**Why**: The current approach creates spans with random trace_ids and relies entirely
|
||||
on protobuf `TraceContext` propagation to link them. If any hop in the relay chain
|
||||
drops the context (older peers, message corruption, mixed-version networks), the trace
|
||||
splits and downstream spans become impossible to find. With deterministic trace_ids,
|
||||
correlation is guaranteed because every node derives the same trace_id from the same
|
||||
`txID`.
|
||||
|
||||
**Approach — deterministic trace_id + protobuf span_id propagation**:
|
||||
|
||||
1. Derive `trace_id = txHash[0:16]` (first 16 bytes of the 32-byte transaction hash).
|
||||
2. Generate a random 8-byte `span_id` per node (each node's span is unique within
|
||||
the shared trace).
|
||||
3. Create the span under this deterministic context as parent.
|
||||
4. **Additionally**, if protobuf `TraceContext` is present in the incoming
|
||||
`TMTransaction` message, extract the sender's `span_id` and use it as the span's
|
||||
parent — this preserves parent-child ordering in the trace tree.
|
||||
5. If protobuf context is absent (older peer, first hop), the span still has the
|
||||
correct deterministic `trace_id` — it appears as a sibling root in the same trace
|
||||
rather than being lost.
|
||||
|
||||
This gives the best of both worlds: guaranteed cross-node correlation via deterministic
|
||||
`trace_id`, plus parent-child relay ordering via protobuf `span_id` when available.
|
||||
|
||||
**What to do**:
|
||||
|
||||
- Create `createDeterministicTxContext(uint256 const& txHash)` utility function:
|
||||
- Location: shared header or file-local in `PeerImp.cpp` and `NetworkOPs.cpp`
|
||||
(or a shared telemetry utility if both need it).
|
||||
- Pattern: identical to `createDeterministicContext(uint256 const& ledgerId)` in
|
||||
`RCLConsensus.cpp` — take `txHash[0:16]` as trace_id, random span_id via
|
||||
`crypto_prng()`, sampled flag set, `remote=false`.
|
||||
- Guard behind `#ifdef XRPL_ENABLE_TELEMETRY`.
|
||||
|
||||
```cpp
|
||||
opentelemetry::context::Context
|
||||
createDeterministicTxContext(uint256 const& txHash)
|
||||
{
|
||||
namespace trace = opentelemetry::trace;
|
||||
|
||||
// First 16 bytes of the 32-byte tx hash as trace ID.
|
||||
trace::TraceId traceId(
|
||||
opentelemetry::nostd::span<uint8_t const, 16>(txHash.data(), 16));
|
||||
|
||||
// Random span_id so each node's span is unique within the trace.
|
||||
uint8_t spanIdBytes[8];
|
||||
crypto_prng()(spanIdBytes, sizeof(spanIdBytes));
|
||||
trace::SpanId spanId(
|
||||
opentelemetry::nostd::span<uint8_t const, 8>(spanIdBytes, 8));
|
||||
|
||||
trace::SpanContext syntheticCtx(
|
||||
traceId, spanId, trace::TraceFlags(1), /* remote = */ false);
|
||||
|
||||
return opentelemetry::context::Context{}.SetValue(
|
||||
trace::kSpanKey,
|
||||
opentelemetry::nostd::shared_ptr<trace::Span>(
|
||||
new trace::DefaultSpan(syntheticCtx)));
|
||||
}
|
||||
```
|
||||
|
||||
- Edit `src/xrpld/overlay/detail/PeerImp.cpp` — restructure `handleTransaction()`:
|
||||
- **Move span creation after deserialization** (txID must be known first):
|
||||
1. Deserialize `STTx` and get `txID` (existing code at line ~1382).
|
||||
2. Create deterministic parent context: `auto detCtx = createDeterministicTxContext(txID)`.
|
||||
3. If `m->has_trace_context()`: extract protobuf context via `extractFromProtobuf()`,
|
||||
**combine** with deterministic trace_id — use the protobuf span_id as parent
|
||||
to preserve relay ordering, but override trace_id with the deterministic one.
|
||||
4. If no protobuf context: create span under `detCtx` directly.
|
||||
5. Set all existing attributes (`hash`, `peerId`, `peerVersion`, `suppressed`, etc.).
|
||||
|
||||
- **Combining deterministic trace_id with protobuf parent span_id**:
|
||||
When both are available, construct a synthetic `SpanContext` with:
|
||||
- `trace_id` = `txHash[0:16]` (deterministic)
|
||||
- `span_id` = extracted from protobuf (sender's span_id → becomes parent)
|
||||
- `trace_flags` = from protobuf
|
||||
- `remote` = true (came from another node)
|
||||
|
||||
```cpp
|
||||
// Pseudo-code for the combined context:
|
||||
auto detTraceId = trace::TraceId(txHash.data(), 16);
|
||||
auto remoteSpanId = /* from extractFromProtobuf */;
|
||||
auto remoteFlags = /* from extractFromProtobuf */;
|
||||
|
||||
trace::SpanContext combinedCtx(
|
||||
detTraceId, remoteSpanId, remoteFlags, /* remote = */ true);
|
||||
// Use as parent context for the new span.
|
||||
```
|
||||
|
||||
- Edit `src/xrpld/app/misc/NetworkOPs.cpp` — update `processTransaction()`:
|
||||
- `transaction->getID()` is already available at the top of the function.
|
||||
- Create deterministic parent context from `txID`.
|
||||
- Create `tx.process` span under this context.
|
||||
- No protobuf context to extract here (NetworkOPs is intra-node), so
|
||||
deterministic context alone is sufficient.
|
||||
|
||||
- Add `tx_trace_strategy` attribute to spans:
|
||||
- Add `inline constexpr auto traceStrategy = join(xrplTx, makeStr("trace_strategy"));`
|
||||
to `TxSpanNames.h`.
|
||||
- Set on each tx span: `span.setAttribute(tx_span::attr::traceStrategy, "deterministic")`.
|
||||
|
||||
**Key new/modified files**:
|
||||
|
||||
- `src/xrpld/overlay/detail/PeerImp.cpp` — restructured span creation
|
||||
- `src/xrpld/app/misc/NetworkOPs.cpp` — deterministic context for tx.process
|
||||
- `src/xrpld/telemetry/TxSpanNames.h` — new `traceStrategy` attribute constant
|
||||
- New or shared utility for `createDeterministicTxContext()` (location TBD: could be
|
||||
a shared header like `include/xrpl/telemetry/DeterministicContext.h`, or file-local
|
||||
if only used in two places)
|
||||
|
||||
**Interaction with existing tasks**:
|
||||
|
||||
- **Task 3.3 (PeerImp instrumentation)**: The span creation in `handleTransaction()`
|
||||
must be restructured — the span currently starts before `txID` is known. This task
|
||||
moves it after deserialization.
|
||||
- **Task 3.6 (Relay context propagation)**: Protobuf injection at the relay site
|
||||
remains the same — `injectToProtobuf()` serializes the current span's `span_id`.
|
||||
The receiver extracts it and combines with the deterministic `trace_id`.
|
||||
- **Phase 4a (Consensus deterministic trace ID)**: This task follows the same pattern.
|
||||
Consider extracting a shared utility (e.g., `createDeterministicContext(uint256)`)
|
||||
that both consensus and transaction tracing use.
|
||||
|
||||
**Exit Criteria**:
|
||||
|
||||
- [ ] `tx.receive` and `tx.process` spans have deterministic trace_id = `txHash[0:16]`
|
||||
- [ ] All nodes handling the same transaction produce spans under the same trace_id
|
||||
- [ ] Protobuf `span_id` propagation still works when available (parent-child ordering)
|
||||
- [ ] Missing protobuf context (old peer) degrades gracefully to sibling spans, not lost traces
|
||||
- [ ] `xrpl.tx.trace_strategy` attribute set to `"deterministic"` on all tx spans
|
||||
- [ ] Trace queryable by tx hash (truncate hash → trace_id → direct lookup in Tempo)
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Task | Description | New Files | Modified Files | Depends On |
|
||||
@@ -265,8 +408,9 @@
|
||||
| 3.6 | Relay context propagation | 0 | 1-2 | 3.3, 3.5 |
|
||||
| 3.7 | Build verification and testing | 0 | 0 | 3.1-3.6 |
|
||||
| 3.8 | TX span peer version attribute | 0 | 1 | 3.3 |
|
||||
| 3.9 | Deterministic transaction trace ID | 0-1 | 3 | 3.2, 3.3 |
|
||||
|
||||
**Parallel work**: Tasks 3.1 and 3.4 can start in parallel. Task 3.2 depends on 3.1. Tasks 3.3 and 3.5 depend on 3.2. Task 3.6 depends on 3.3 and 3.5. Task 3.8 depends on 3.3 (span must exist).
|
||||
**Parallel work**: Tasks 3.1 and 3.4 can start in parallel. Task 3.2 depends on 3.1. Tasks 3.3 and 3.5 depend on 3.2. Task 3.6 depends on 3.3 and 3.5. Task 3.8 depends on 3.3 (span must exist). Task 3.9 depends on 3.2 and 3.3.
|
||||
|
||||
**Exit Criteria** (from [06-implementation-phases.md §6.11.3](./06-implementation-phases.md)):
|
||||
|
||||
@@ -274,3 +418,5 @@
|
||||
- [ ] Trace context in Protocol Buffer messages
|
||||
- [ ] HashRouter deduplication visible in traces
|
||||
- [ ] <5% overhead on transaction throughput
|
||||
- [ ] Deterministic trace_id: same trace_id for same tx across all nodes
|
||||
- [ ] Protobuf span_id propagation preserves parent-child ordering when available
|
||||
|
||||
Reference in New Issue
Block a user