ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-22 14:50:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(telemetry): remove residual Jaeger references across chain

Browse Source 
Fix remaining Jaeger references that accumulated across intermediate
branches in the stacked PR chain. These were in files modified by
multiple phases where the per-branch fixes didn't cover all additions.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pratik Mankawde
2026-03-31 22:35:04 +01:00
parent a142a700e8
commit 7e149f7773
11 changed files with 37 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
OpenTelemetryPlan/05-configuration-reference.md
View File

@@ -420,7 +420,7 @@ service:
traces:
receivers: [otlp]
processors: [batch]
exporters: [logging, jaeger, otlp/tempo]
exporters: [logging, otlp/tempo]
```
### 5.5.2 Production Configuration
@@ -586,7 +586,6 @@ services:
ports:
- "3000:3000"
depends_on:
- jaeger
- tempo
# Prometheus for metrics (optional, for correlation)

14
OpenTelemetryPlan/06-implementation-phases.md
View File

@@ -464,7 +464,7 @@ graph LR
end
subgraph backends["Trace Backends"]
D["Jaeger / Tempo"]
D["Tempo"]
end
subgraph metrics["Metrics Stack"]
@@ -561,11 +561,11 @@ See [Phase7_taskList.md](./Phase7_taskList.md) for detailed per-task breakdown.
### Motivation
rippled's `beast::Journal` logs and OpenTelemetry traces are currently two disjoint observability signals. When investigating an issue, operators must manually correlate timestamps between log files and Jaeger/Tempo traces. Phase 8 bridges this gap by injecting trace context (`trace_id`, `span_id`) into every log line emitted within an active span, and ingesting those logs into Grafana Loki via the OTel Collector's filelog receiver.
rippled's `beast::Journal` logs and OpenTelemetry traces are currently two disjoint observability signals. When investigating an issue, operators must manually correlate timestamps between log files and Tempo traces. Phase 8 bridges this gap by injecting trace context (`trace_id`, `span_id`) into every log line emitted within an active span, and ingesting those logs into Grafana Loki via the OTel Collector's filelog receiver.
#### Gains
1. **One-click trace-to-log navigation** — Click a trace in Tempo/Jaeger and immediately see the corresponding log lines in Loki, filtered by `trace_id`.
1. **One-click trace-to-log navigation** — Click a trace in Tempo and immediately see the corresponding log lines in Loki, filtered by `trace_id`.
2. **Reverse lookup (log-to-trace)** — Loki derived fields make `trace_id` values clickable links back to Tempo.
3. **Unified observability** — All three pillars (traces, metrics, logs) flow through the same OTel Collector pipeline and are visible in a single Grafana instance.
4. **Zero new dependencies in rippled** — Uses existing OTel SDK headers (`GetSpan`, `GetContext`) already linked in Phase 1.
@@ -774,7 +774,7 @@ Before the telemetry stack (Phases 1-9) can be considered production-ready, we n
### Architecture
The validation uses a **2-node** validator cluster running as local processes alongside a Docker Compose telemetry stack (Collector, Jaeger, Prometheus, Grafana). Two nodes are sufficient for consensus rounds and peer-to-peer span validation while minimizing CI resource usage.
The validation uses a **2-node** validator cluster running as local processes alongside a Docker Compose telemetry stack (Collector, Tempo, Prometheus, Grafana). Two nodes are sufficient for consensus rounds and peer-to-peer span validation while minimizing CI resource usage.
```mermaid
flowchart LR
@@ -786,7 +786,7 @@ flowchart LR
subgraph telemetry["Docker Compose Telemetry Stack"]
direction TB
COL["OTel Collector<br/>(OTLP + StatsD)"]
JAE["Jaeger<br/>(trace search)"]
JAE["Tempo<br/>(trace search)"]
PROM["Prometheus<br/>(metrics)"]
GRAF["Grafana<br/>(dashboards)"]
end
@@ -797,7 +797,7 @@ flowchart LR
end
subgraph validation["Validation Suite"]
SV["Span Validator<br/>(Jaeger API)"]
SV["Span Validator<br/>(Tempo API)"]
MV["Metric Validator<br/>(Prometheus API,<br/>all 26 metrics required)"]
DV["Dashboard Validator<br/>(Grafana API)"]
BM["Benchmark Suite<br/>(CPU, memory, latency<br/>ON vs OFF comparison)"]
@@ -852,7 +852,7 @@ See [Phase10_taskList.md](./Phase10_taskList.md) for detailed per-task breakdown
The validation suite (`validate_telemetry.py`) runs exactly 71 checks, broken down as:
- **1 service registration** — `rippled` exists in Jaeger
- **1 service registration** — `rippled` exists in Tempo
- **17 span existence** — `rpc.request`, `rpc.process`, `rpc.ws_message`, `rpc.command.*`, `tx.process`, `tx.receive`, `tx.apply`, `consensus.proposal.send`, `consensus.ledger_close`, `consensus.accept`, `consensus.validation.send`, `consensus.accept.apply`, `ledger.build`, `ledger.validate`, `ledger.store`, `peer.proposal.receive`, `peer.validation.receive`
- **14 span attribute** — required attributes on the 14 spans that define them (22 unique attributes total)
- **2 span hierarchies** — `rpc.process` -> `rpc.command.*`, `ledger.build` -> `tx.apply` (1 skipped: `rpc.request` -> `rpc.process`, cross-thread)

4
OpenTelemetryPlan/09-data-collection-reference.md
View File

@@ -511,7 +511,7 @@ Example:
2024-01-15T10:30:45.123Z LedgerMaster:NFO trace_id=abc123def456789012345678abcdef01 span_id=0123456789abcdef Validated ledger 42
```
- **`trace_id=<hex32>`** — 32-character lowercase hex trace identifier. Links to the distributed trace in Tempo/Jaeger.
- **`trace_id=<hex32>`** — 32-character lowercase hex trace identifier. Links to the distributed trace in Tempo.
- **`span_id=<hex16>`** — 16-character lowercase hex span identifier. Identifies the specific span within the trace.
- **Only present** when the log is emitted within an active OTel span. Log lines outside of traced code paths have no trace context fields.
@@ -734,7 +734,7 @@ docker/telemetry/workload/benchmark.sh --xrpld .build/xrpld --duration 300
| Category | Expected Count | Validation Method | Config File |
| ------------------ | -------------- | -------------------------------- | ----------------------- |
| Trace spans | 17 | Jaeger/Tempo API query | `expected_spans.json` |
| Trace spans | 17 | Tempo API query | `expected_spans.json` |
| Span attributes | 22 | Per-span attribute assertion | `expected_spans.json` |
| StatsD metrics | 255+ | Prometheus query | `expected_metrics.json` |
| Phase 9 metrics | 68+ | Prometheus query | `expected_metrics.json` |

2
OpenTelemetryPlan/OpenTelemetryPlan.md
View File

@@ -228,7 +228,7 @@ The appendix contains a glossary of OpenTelemetry and rippled-specific terms, re
## 9. Data Collection Reference
A single-source-of-truth reference documenting every piece of telemetry data collected by rippled. Covers all 16 OpenTelemetry spans with their 22 attributes, all StatsD metrics (gauges, counters, histograms, overlay traffic), SpanMetrics-derived Prometheus metrics, and all 8 Grafana dashboards. Includes Jaeger search guides and Prometheus query examples.
A single-source-of-truth reference documenting every piece of telemetry data collected by rippled. Covers all 16 OpenTelemetry spans with their 22 attributes, all StatsD metrics (gauges, counters, histograms, overlay traffic), SpanMetrics-derived Prometheus metrics, and all 8 Grafana dashboards. Includes Tempo search guides and Prometheus query examples.
➡️ **[View Data Collection Reference](./09-data-collection-reference.md)**

2
OpenTelemetryPlan/Phase3_taskList.md
View File

@@ -248,7 +248,7 @@
- [ ] `tx.receive` spans carry `xrpl.peer.version` attribute with a non-empty version string
- [ ] Attribute is omitted (not set to empty string) when `getVersion()` returns empty
- [ ] Attribute visible in Jaeger span detail view
- [ ] Attribute visible in Tempo trace detail view
---

4
OpenTelemetryPlan/Phase4_taskList.md
View File

@@ -334,7 +334,7 @@ Two strategies for cross-node trace correlation, switchable via config:
Derive `trace_id = SHA256(previousLedger.id())[0:16]` so all nodes in the same
consensus round share the same trace_id without P2P context propagation.
- **Pros**: All nodes appear in the same trace in Tempo/Jaeger automatically.
- **Pros**: All nodes appear in the same trace in Tempo automatically.
No collector-side post-processing needed.
- **Cons**: Overrides OTel's random trace_id generation; requires custom
`IdGenerator` or manual span context construction.
@@ -876,7 +876,7 @@ Received messages use **span links** (follows-from), NOT parent-child:
- The receiver's processing span links to the sender's context
- This preserves each node's independent trace tree
- Cross-node correlation visible via linked traces in Tempo/Jaeger
- Cross-node correlation visible via linked traces in Tempo
## Interaction with Deterministic Trace ID (Strategy A)

10
docker/telemetry/TESTING.md
View File

@@ -482,17 +482,17 @@ severity code and the message. Example:
Lines emitted outside of an active span (background tasks, startup) will
NOT have trace context — this is expected.
### Step 2: Cross-check trace_id in Jaeger
### Step 2: Cross-check trace_id in Tempo
Extract a `trace_id` from the log and verify it exists in Jaeger:
Extract a `trace_id` from the log and verify it exists in Tempo:
```bash
TRACE_ID=$(grep -o 'trace_id=[a-f0-9]\{32\}' /path/to/debug.log | head -1 | cut -d= -f2)
echo "Checking trace: $TRACE_ID"
curl -s "http://localhost:16686/api/traces/$TRACE_ID" | jq '.data | length'
curl -s "http://localhost:3200/api/traces/$TRACE_ID" | jq '.batches | length'
```
Expected result: `1` (the trace exists in Jaeger).
Expected result: `> 0` (the trace exists in Tempo).
### Step 3: Verify Loki log ingestion
@@ -530,7 +530,7 @@ Expected: > 0 results.
| `trace_id=` in debug.log | Present in log lines within active spans |
| `span_id=` in debug.log | Present alongside trace_id |
| Logs without active span | No trace_id/span_id fields |
| trace_id in Jaeger | Matches a valid trace |
| trace_id in Tempo | Matches a valid trace |
| Loki log ingestion | Logs visible via LogQL |
| Tempo -> Loki "Logs for trace" | Shows correlated log lines |
| Loki -> Tempo TraceID link | Navigates to correct trace |

14
docker/telemetry/integration-test.sh
View File

@@ -67,7 +67,7 @@ check_span() {
# Phase 8: Verify trace_id injection in rippled log output.
# Greps all node debug.log files for the "trace_id=<hex> span_id=<hex>"
# pattern that Logs::format() injects when an active OTel span exists.
# Also cross-checks that a trace_id found in logs matches a trace in Jaeger.
# Also cross-checks that a trace_id found in logs matches a trace in Tempo.
check_log_correlation() {
log "Checking log-trace correlation..."
@@ -82,7 +82,7 @@ check_log_correlation() {
local matches
matches=$(grep -c 'trace_id=[a-f0-9]\{32\} span_id=[a-f0-9]\{16\}' "$logfile" 2>/dev/null || echo 0)
total_matches=$((total_matches + matches))
# Capture the first trace_id we find for cross-referencing with Jaeger
# Capture the first trace_id we find for cross-referencing with Tempo
if [ -z "$sample_trace_id" ] && [ "$matches" -gt 0 ]; then
sample_trace_id=$(grep -o 'trace_id=[a-f0-9]\{32\}' "$logfile" | head -1 | cut -d= -f2)
fi
@@ -94,15 +94,15 @@ check_log_correlation() {
fail "Log correlation: no trace_id found in any node debug.log"
fi
# Cross-check: verify the sample trace_id exists in Jaeger
# Cross-check: verify the sample trace_id exists in Tempo
if [ -n "$sample_trace_id" ]; then
local trace_found
trace_found=$(curl -sf "$JAEGER/api/traces/$sample_trace_id" \
| jq '.data | length' 2>/dev/null || echo 0)
trace_found=$(curl -sf "$TEMPO/api/traces/$sample_trace_id" \
| jq '.batches | length' 2>/dev/null || echo 0)
if [ "$trace_found" -gt 0 ]; then
ok "Log-Jaeger cross-check: trace_id=$sample_trace_id found in Jaeger"
ok "Log-Tempo cross-check: trace_id=$sample_trace_id found in Tempo"
else
fail "Log-Jaeger cross-check: trace_id=$sample_trace_id NOT found in Jaeger"
fail "Log-Tempo cross-check: trace_id=$sample_trace_id NOT found in Tempo"
fi
fi
}

2
docker/telemetry/xrpld-telemetry.cfg
View File

@@ -7,7 +7,7 @@
# ./xrpld --conf docker/telemetry/xrpld-telemetry.cfg -a --start
# 3. Send RPC commands to exercise tracing:
# curl -s http://localhost:5005 -d '{"method":"server_info"}'
# 4. View traces in Jaeger UI: http://localhost:16686
# 4. View traces in Grafana Explore -> Tempo: http://localhost:3000
[server]
port_rpc_admin_local

12
docs/telemetry-runbook.md
View File

@@ -526,12 +526,12 @@ cat /tmp/xrpld-validation/reports/validation-report.json | jq '.summary'
### What Gets Validated
| Category | Checks | Description |
| ---------- | -------------- | -------------------------------------------------------- |
| Spans | 16+ span types | All span names appear in Jaeger with required attributes |
| Metrics | 30+ metrics | SpanMetrics, StatsD gauges/counters, Phase 9 metrics |
| Logs | 2 checks | trace_id/span_id present in Loki, cross-reference works |
| Dashboards | 10 dashboards | All Grafana dashboards load without errors |
| Category | Checks | Description |
| ---------- | -------------- | ------------------------------------------------------- |
| Spans | 16+ span types | All span names appear in Tempo with required attributes |
| Metrics | 30+ metrics | SpanMetrics, StatsD gauges/counters, Phase 9 metrics |
| Logs | 2 checks | trace_id/span_id present in Loki, cross-reference works |
| Dashboards | 10 dashboards | All Grafana dashboards load without errors |
### Running Individual Tools

8
tasks/fix-validation-checks.md
View File

@@ -80,13 +80,13 @@ Likely causes (investigate in order):
2. **Code path not triggered:** `tx.process` fires in `NetworkOPs::processTransaction()`.
The tx_submitter submits via RPC `submit` command which calls this path. But if the
transactions fail validation before reaching `processTransaction()`, no span is emitted.
3. **Span naming mismatch:** The validation queries Jaeger for exact operation name
`tx.process`. Verify Jaeger stores the span with this exact name.
3. **Span naming mismatch:** The validation queries Tempo for exact operation name
`tx.process`. Verify Tempo stores the span with this exact name.
**Investigation:**
- Check the tx_submitter output in CI logs — are transactions actually succeeding?
- Query Jaeger API locally for all span names to see what's actually emitted.
- Query Tempo API locally for all span names to see what's actually emitted.
**Files to modify:**
@@ -103,7 +103,7 @@ Likely causes (investigate in order):
[FAIL] span.hierarchy.rpc.request->rpc.process: rpc.process not found in rpc.request traces
```
**Root Cause:** The validator fetches traces containing `rpc.request` from Jaeger and
**Root Cause:** The validator fetches traces containing `rpc.request` from Tempo and
checks if any child span is named `rpc.process`. Both spans are emitted (they pass
individual checks), but the parent-child relationship isn't established.