From 7e149f77737670bc4a620636bac7401b368cbb9f Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Tue, 31 Mar 2026 22:35:04 +0100 Subject: [PATCH] refactor(telemetry): remove residual Jaeger references across chain Fix remaining Jaeger references that accumulated across intermediate branches in the stacked PR chain. These were in files modified by multiple phases where the per-branch fixes didn't cover all additions. Co-Authored-By: Claude Opus 4.6 (1M context) --- OpenTelemetryPlan/05-configuration-reference.md | 3 +-- OpenTelemetryPlan/06-implementation-phases.md | 14 +++++++------- OpenTelemetryPlan/09-data-collection-reference.md | 4 ++-- OpenTelemetryPlan/OpenTelemetryPlan.md | 2 +- OpenTelemetryPlan/Phase3_taskList.md | 2 +- OpenTelemetryPlan/Phase4_taskList.md | 4 ++-- docker/telemetry/TESTING.md | 10 +++++----- docker/telemetry/integration-test.sh | 14 +++++++------- docker/telemetry/xrpld-telemetry.cfg | 2 +- docs/telemetry-runbook.md | 12 ++++++------ tasks/fix-validation-checks.md | 8 ++++---- 11 files changed, 37 insertions(+), 38 deletions(-) diff --git a/OpenTelemetryPlan/05-configuration-reference.md b/OpenTelemetryPlan/05-configuration-reference.md index a69e60cb5e..7a7c2b90c6 100644 --- a/OpenTelemetryPlan/05-configuration-reference.md +++ b/OpenTelemetryPlan/05-configuration-reference.md @@ -420,7 +420,7 @@ service: traces: receivers: [otlp] processors: [batch] - exporters: [logging, jaeger, otlp/tempo] + exporters: [logging, otlp/tempo] ``` ### 5.5.2 Production Configuration @@ -586,7 +586,6 @@ services: ports: - "3000:3000" depends_on: - - jaeger - tempo # Prometheus for metrics (optional, for correlation) diff --git a/OpenTelemetryPlan/06-implementation-phases.md b/OpenTelemetryPlan/06-implementation-phases.md index 94bdd7c8ae..6cc15beb14 100644 --- a/OpenTelemetryPlan/06-implementation-phases.md +++ b/OpenTelemetryPlan/06-implementation-phases.md @@ -464,7 +464,7 @@ graph LR end subgraph backends["Trace Backends"] - D["Jaeger / Tempo"] + D["Tempo"] end subgraph metrics["Metrics Stack"] @@ -561,11 +561,11 @@ See [Phase7_taskList.md](./Phase7_taskList.md) for detailed per-task breakdown. ### Motivation -rippled's `beast::Journal` logs and OpenTelemetry traces are currently two disjoint observability signals. When investigating an issue, operators must manually correlate timestamps between log files and Jaeger/Tempo traces. Phase 8 bridges this gap by injecting trace context (`trace_id`, `span_id`) into every log line emitted within an active span, and ingesting those logs into Grafana Loki via the OTel Collector's filelog receiver. +rippled's `beast::Journal` logs and OpenTelemetry traces are currently two disjoint observability signals. When investigating an issue, operators must manually correlate timestamps between log files and Tempo traces. Phase 8 bridges this gap by injecting trace context (`trace_id`, `span_id`) into every log line emitted within an active span, and ingesting those logs into Grafana Loki via the OTel Collector's filelog receiver. #### Gains -1. **One-click trace-to-log navigation** — Click a trace in Tempo/Jaeger and immediately see the corresponding log lines in Loki, filtered by `trace_id`. +1. **One-click trace-to-log navigation** — Click a trace in Tempo and immediately see the corresponding log lines in Loki, filtered by `trace_id`. 2. **Reverse lookup (log-to-trace)** — Loki derived fields make `trace_id` values clickable links back to Tempo. 3. **Unified observability** — All three pillars (traces, metrics, logs) flow through the same OTel Collector pipeline and are visible in a single Grafana instance. 4. **Zero new dependencies in rippled** — Uses existing OTel SDK headers (`GetSpan`, `GetContext`) already linked in Phase 1. @@ -774,7 +774,7 @@ Before the telemetry stack (Phases 1-9) can be considered production-ready, we n ### Architecture -The validation uses a **2-node** validator cluster running as local processes alongside a Docker Compose telemetry stack (Collector, Jaeger, Prometheus, Grafana). Two nodes are sufficient for consensus rounds and peer-to-peer span validation while minimizing CI resource usage. +The validation uses a **2-node** validator cluster running as local processes alongside a Docker Compose telemetry stack (Collector, Tempo, Prometheus, Grafana). Two nodes are sufficient for consensus rounds and peer-to-peer span validation while minimizing CI resource usage. ```mermaid flowchart LR @@ -786,7 +786,7 @@ flowchart LR subgraph telemetry["Docker Compose Telemetry Stack"] direction TB COL["OTel Collector
(OTLP + StatsD)"] - JAE["Jaeger
(trace search)"] + JAE["Tempo
(trace search)"] PROM["Prometheus
(metrics)"] GRAF["Grafana
(dashboards)"] end @@ -797,7 +797,7 @@ flowchart LR end subgraph validation["Validation Suite"] - SV["Span Validator
(Jaeger API)"] + SV["Span Validator
(Tempo API)"] MV["Metric Validator
(Prometheus API,
all 26 metrics required)"] DV["Dashboard Validator
(Grafana API)"] BM["Benchmark Suite
(CPU, memory, latency
ON vs OFF comparison)"] @@ -852,7 +852,7 @@ See [Phase10_taskList.md](./Phase10_taskList.md) for detailed per-task breakdown The validation suite (`validate_telemetry.py`) runs exactly 71 checks, broken down as: -- **1 service registration** — `rippled` exists in Jaeger +- **1 service registration** — `rippled` exists in Tempo - **17 span existence** — `rpc.request`, `rpc.process`, `rpc.ws_message`, `rpc.command.*`, `tx.process`, `tx.receive`, `tx.apply`, `consensus.proposal.send`, `consensus.ledger_close`, `consensus.accept`, `consensus.validation.send`, `consensus.accept.apply`, `ledger.build`, `ledger.validate`, `ledger.store`, `peer.proposal.receive`, `peer.validation.receive` - **14 span attribute** — required attributes on the 14 spans that define them (22 unique attributes total) - **2 span hierarchies** — `rpc.process` -> `rpc.command.*`, `ledger.build` -> `tx.apply` (1 skipped: `rpc.request` -> `rpc.process`, cross-thread) diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md index 95664e3313..a4e15af34c 100644 --- a/OpenTelemetryPlan/09-data-collection-reference.md +++ b/OpenTelemetryPlan/09-data-collection-reference.md @@ -511,7 +511,7 @@ Example: 2024-01-15T10:30:45.123Z LedgerMaster:NFO trace_id=abc123def456789012345678abcdef01 span_id=0123456789abcdef Validated ledger 42 ``` -- **`trace_id=`** — 32-character lowercase hex trace identifier. Links to the distributed trace in Tempo/Jaeger. +- **`trace_id=`** — 32-character lowercase hex trace identifier. Links to the distributed trace in Tempo. - **`span_id=`** — 16-character lowercase hex span identifier. Identifies the specific span within the trace. - **Only present** when the log is emitted within an active OTel span. Log lines outside of traced code paths have no trace context fields. @@ -734,7 +734,7 @@ docker/telemetry/workload/benchmark.sh --xrpld .build/xrpld --duration 300 | Category | Expected Count | Validation Method | Config File | | ------------------ | -------------- | -------------------------------- | ----------------------- | -| Trace spans | 17 | Jaeger/Tempo API query | `expected_spans.json` | +| Trace spans | 17 | Tempo API query | `expected_spans.json` | | Span attributes | 22 | Per-span attribute assertion | `expected_spans.json` | | StatsD metrics | 255+ | Prometheus query | `expected_metrics.json` | | Phase 9 metrics | 68+ | Prometheus query | `expected_metrics.json` | diff --git a/OpenTelemetryPlan/OpenTelemetryPlan.md b/OpenTelemetryPlan/OpenTelemetryPlan.md index 7be3cb57ed..89f5ab0b47 100644 --- a/OpenTelemetryPlan/OpenTelemetryPlan.md +++ b/OpenTelemetryPlan/OpenTelemetryPlan.md @@ -228,7 +228,7 @@ The appendix contains a glossary of OpenTelemetry and rippled-specific terms, re ## 9. Data Collection Reference -A single-source-of-truth reference documenting every piece of telemetry data collected by rippled. Covers all 16 OpenTelemetry spans with their 22 attributes, all StatsD metrics (gauges, counters, histograms, overlay traffic), SpanMetrics-derived Prometheus metrics, and all 8 Grafana dashboards. Includes Jaeger search guides and Prometheus query examples. +A single-source-of-truth reference documenting every piece of telemetry data collected by rippled. Covers all 16 OpenTelemetry spans with their 22 attributes, all StatsD metrics (gauges, counters, histograms, overlay traffic), SpanMetrics-derived Prometheus metrics, and all 8 Grafana dashboards. Includes Tempo search guides and Prometheus query examples. ➡️ **[View Data Collection Reference](./09-data-collection-reference.md)** diff --git a/OpenTelemetryPlan/Phase3_taskList.md b/OpenTelemetryPlan/Phase3_taskList.md index cf86b737de..49468d0805 100644 --- a/OpenTelemetryPlan/Phase3_taskList.md +++ b/OpenTelemetryPlan/Phase3_taskList.md @@ -248,7 +248,7 @@ - [ ] `tx.receive` spans carry `xrpl.peer.version` attribute with a non-empty version string - [ ] Attribute is omitted (not set to empty string) when `getVersion()` returns empty -- [ ] Attribute visible in Jaeger span detail view +- [ ] Attribute visible in Tempo trace detail view --- diff --git a/OpenTelemetryPlan/Phase4_taskList.md b/OpenTelemetryPlan/Phase4_taskList.md index 3817183a22..96e4213bbc 100644 --- a/OpenTelemetryPlan/Phase4_taskList.md +++ b/OpenTelemetryPlan/Phase4_taskList.md @@ -334,7 +334,7 @@ Two strategies for cross-node trace correlation, switchable via config: Derive `trace_id = SHA256(previousLedger.id())[0:16]` so all nodes in the same consensus round share the same trace_id without P2P context propagation. -- **Pros**: All nodes appear in the same trace in Tempo/Jaeger automatically. +- **Pros**: All nodes appear in the same trace in Tempo automatically. No collector-side post-processing needed. - **Cons**: Overrides OTel's random trace_id generation; requires custom `IdGenerator` or manual span context construction. @@ -876,7 +876,7 @@ Received messages use **span links** (follows-from), NOT parent-child: - The receiver's processing span links to the sender's context - This preserves each node's independent trace tree -- Cross-node correlation visible via linked traces in Tempo/Jaeger +- Cross-node correlation visible via linked traces in Tempo ## Interaction with Deterministic Trace ID (Strategy A) diff --git a/docker/telemetry/TESTING.md b/docker/telemetry/TESTING.md index 1e6e654e7b..8b5bfdade1 100644 --- a/docker/telemetry/TESTING.md +++ b/docker/telemetry/TESTING.md @@ -482,17 +482,17 @@ severity code and the message. Example: Lines emitted outside of an active span (background tasks, startup) will NOT have trace context — this is expected. -### Step 2: Cross-check trace_id in Jaeger +### Step 2: Cross-check trace_id in Tempo -Extract a `trace_id` from the log and verify it exists in Jaeger: +Extract a `trace_id` from the log and verify it exists in Tempo: ```bash TRACE_ID=$(grep -o 'trace_id=[a-f0-9]\{32\}' /path/to/debug.log | head -1 | cut -d= -f2) echo "Checking trace: $TRACE_ID" -curl -s "http://localhost:16686/api/traces/$TRACE_ID" | jq '.data | length' +curl -s "http://localhost:3200/api/traces/$TRACE_ID" | jq '.batches | length' ``` -Expected result: `1` (the trace exists in Jaeger). +Expected result: `> 0` (the trace exists in Tempo). ### Step 3: Verify Loki log ingestion @@ -530,7 +530,7 @@ Expected: > 0 results. | `trace_id=` in debug.log | Present in log lines within active spans | | `span_id=` in debug.log | Present alongside trace_id | | Logs without active span | No trace_id/span_id fields | -| trace_id in Jaeger | Matches a valid trace | +| trace_id in Tempo | Matches a valid trace | | Loki log ingestion | Logs visible via LogQL | | Tempo -> Loki "Logs for trace" | Shows correlated log lines | | Loki -> Tempo TraceID link | Navigates to correct trace | diff --git a/docker/telemetry/integration-test.sh b/docker/telemetry/integration-test.sh index 79a6bcedf4..dc123395b1 100755 --- a/docker/telemetry/integration-test.sh +++ b/docker/telemetry/integration-test.sh @@ -67,7 +67,7 @@ check_span() { # Phase 8: Verify trace_id injection in rippled log output. # Greps all node debug.log files for the "trace_id= span_id=" # pattern that Logs::format() injects when an active OTel span exists. -# Also cross-checks that a trace_id found in logs matches a trace in Jaeger. +# Also cross-checks that a trace_id found in logs matches a trace in Tempo. check_log_correlation() { log "Checking log-trace correlation..." @@ -82,7 +82,7 @@ check_log_correlation() { local matches matches=$(grep -c 'trace_id=[a-f0-9]\{32\} span_id=[a-f0-9]\{16\}' "$logfile" 2>/dev/null || echo 0) total_matches=$((total_matches + matches)) - # Capture the first trace_id we find for cross-referencing with Jaeger + # Capture the first trace_id we find for cross-referencing with Tempo if [ -z "$sample_trace_id" ] && [ "$matches" -gt 0 ]; then sample_trace_id=$(grep -o 'trace_id=[a-f0-9]\{32\}' "$logfile" | head -1 | cut -d= -f2) fi @@ -94,15 +94,15 @@ check_log_correlation() { fail "Log correlation: no trace_id found in any node debug.log" fi - # Cross-check: verify the sample trace_id exists in Jaeger + # Cross-check: verify the sample trace_id exists in Tempo if [ -n "$sample_trace_id" ]; then local trace_found - trace_found=$(curl -sf "$JAEGER/api/traces/$sample_trace_id" \ - | jq '.data | length' 2>/dev/null || echo 0) + trace_found=$(curl -sf "$TEMPO/api/traces/$sample_trace_id" \ + | jq '.batches | length' 2>/dev/null || echo 0) if [ "$trace_found" -gt 0 ]; then - ok "Log-Jaeger cross-check: trace_id=$sample_trace_id found in Jaeger" + ok "Log-Tempo cross-check: trace_id=$sample_trace_id found in Tempo" else - fail "Log-Jaeger cross-check: trace_id=$sample_trace_id NOT found in Jaeger" + fail "Log-Tempo cross-check: trace_id=$sample_trace_id NOT found in Tempo" fi fi } diff --git a/docker/telemetry/xrpld-telemetry.cfg b/docker/telemetry/xrpld-telemetry.cfg index 2a96dd6ab5..2d5adf3692 100644 --- a/docker/telemetry/xrpld-telemetry.cfg +++ b/docker/telemetry/xrpld-telemetry.cfg @@ -7,7 +7,7 @@ # ./xrpld --conf docker/telemetry/xrpld-telemetry.cfg -a --start # 3. Send RPC commands to exercise tracing: # curl -s http://localhost:5005 -d '{"method":"server_info"}' -# 4. View traces in Jaeger UI: http://localhost:16686 +# 4. View traces in Grafana Explore -> Tempo: http://localhost:3000 [server] port_rpc_admin_local diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index 3afc40409f..f588c963c9 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -526,12 +526,12 @@ cat /tmp/xrpld-validation/reports/validation-report.json | jq '.summary' ### What Gets Validated -| Category | Checks | Description | -| ---------- | -------------- | -------------------------------------------------------- | -| Spans | 16+ span types | All span names appear in Jaeger with required attributes | -| Metrics | 30+ metrics | SpanMetrics, StatsD gauges/counters, Phase 9 metrics | -| Logs | 2 checks | trace_id/span_id present in Loki, cross-reference works | -| Dashboards | 10 dashboards | All Grafana dashboards load without errors | +| Category | Checks | Description | +| ---------- | -------------- | ------------------------------------------------------- | +| Spans | 16+ span types | All span names appear in Tempo with required attributes | +| Metrics | 30+ metrics | SpanMetrics, StatsD gauges/counters, Phase 9 metrics | +| Logs | 2 checks | trace_id/span_id present in Loki, cross-reference works | +| Dashboards | 10 dashboards | All Grafana dashboards load without errors | ### Running Individual Tools diff --git a/tasks/fix-validation-checks.md b/tasks/fix-validation-checks.md index 096920c524..44cacc102a 100644 --- a/tasks/fix-validation-checks.md +++ b/tasks/fix-validation-checks.md @@ -80,13 +80,13 @@ Likely causes (investigate in order): 2. **Code path not triggered:** `tx.process` fires in `NetworkOPs::processTransaction()`. The tx_submitter submits via RPC `submit` command which calls this path. But if the transactions fail validation before reaching `processTransaction()`, no span is emitted. -3. **Span naming mismatch:** The validation queries Jaeger for exact operation name - `tx.process`. Verify Jaeger stores the span with this exact name. +3. **Span naming mismatch:** The validation queries Tempo for exact operation name + `tx.process`. Verify Tempo stores the span with this exact name. **Investigation:** - Check the tx_submitter output in CI logs — are transactions actually succeeding? -- Query Jaeger API locally for all span names to see what's actually emitted. +- Query Tempo API locally for all span names to see what's actually emitted. **Files to modify:** @@ -103,7 +103,7 @@ Likely causes (investigate in order): [FAIL] span.hierarchy.rpc.request->rpc.process: rpc.process not found in rpc.request traces ``` -**Root Cause:** The validator fetches traces containing `rpc.request` from Jaeger and +**Root Cause:** The validator fetches traces containing `rpc.request` from Tempo and checks if any child span is named `rpc.process`. Both spans are emitted (they pass individual checks), but the parent-child relationship isn't established.