diff --git a/OpenTelemetryPlan/05-configuration-reference.md b/OpenTelemetryPlan/05-configuration-reference.md index a69e60cb5e..7a7c2b90c6 100644 --- a/OpenTelemetryPlan/05-configuration-reference.md +++ b/OpenTelemetryPlan/05-configuration-reference.md @@ -420,7 +420,7 @@ service: traces: receivers: [otlp] processors: [batch] - exporters: [logging, jaeger, otlp/tempo] + exporters: [logging, otlp/tempo] ``` ### 5.5.2 Production Configuration @@ -586,7 +586,6 @@ services: ports: - "3000:3000" depends_on: - - jaeger - tempo # Prometheus for metrics (optional, for correlation) diff --git a/OpenTelemetryPlan/06-implementation-phases.md b/OpenTelemetryPlan/06-implementation-phases.md index 94bdd7c8ae..6cc15beb14 100644 --- a/OpenTelemetryPlan/06-implementation-phases.md +++ b/OpenTelemetryPlan/06-implementation-phases.md @@ -464,7 +464,7 @@ graph LR end subgraph backends["Trace Backends"] - D["Jaeger / Tempo"] + D["Tempo"] end subgraph metrics["Metrics Stack"] @@ -561,11 +561,11 @@ See [Phase7_taskList.md](./Phase7_taskList.md) for detailed per-task breakdown. ### Motivation -rippled's `beast::Journal` logs and OpenTelemetry traces are currently two disjoint observability signals. When investigating an issue, operators must manually correlate timestamps between log files and Jaeger/Tempo traces. Phase 8 bridges this gap by injecting trace context (`trace_id`, `span_id`) into every log line emitted within an active span, and ingesting those logs into Grafana Loki via the OTel Collector's filelog receiver. +rippled's `beast::Journal` logs and OpenTelemetry traces are currently two disjoint observability signals. When investigating an issue, operators must manually correlate timestamps between log files and Tempo traces. Phase 8 bridges this gap by injecting trace context (`trace_id`, `span_id`) into every log line emitted within an active span, and ingesting those logs into Grafana Loki via the OTel Collector's filelog receiver. #### Gains -1. **One-click trace-to-log navigation** — Click a trace in Tempo/Jaeger and immediately see the corresponding log lines in Loki, filtered by `trace_id`. +1. **One-click trace-to-log navigation** — Click a trace in Tempo and immediately see the corresponding log lines in Loki, filtered by `trace_id`. 2. **Reverse lookup (log-to-trace)** — Loki derived fields make `trace_id` values clickable links back to Tempo. 3. **Unified observability** — All three pillars (traces, metrics, logs) flow through the same OTel Collector pipeline and are visible in a single Grafana instance. 4. **Zero new dependencies in rippled** — Uses existing OTel SDK headers (`GetSpan`, `GetContext`) already linked in Phase 1. @@ -774,7 +774,7 @@ Before the telemetry stack (Phases 1-9) can be considered production-ready, we n ### Architecture -The validation uses a **2-node** validator cluster running as local processes alongside a Docker Compose telemetry stack (Collector, Jaeger, Prometheus, Grafana). Two nodes are sufficient for consensus rounds and peer-to-peer span validation while minimizing CI resource usage. +The validation uses a **2-node** validator cluster running as local processes alongside a Docker Compose telemetry stack (Collector, Tempo, Prometheus, Grafana). Two nodes are sufficient for consensus rounds and peer-to-peer span validation while minimizing CI resource usage. ```mermaid flowchart LR @@ -786,7 +786,7 @@ flowchart LR subgraph telemetry["Docker Compose Telemetry Stack"] direction TB COL["OTel Collector
(OTLP + StatsD)"] - JAE["Jaeger
(trace search)"] + JAE["Tempo
(trace search)"] PROM["Prometheus
(metrics)"] GRAF["Grafana
(dashboards)"] end @@ -797,7 +797,7 @@ flowchart LR end subgraph validation["Validation Suite"] - SV["Span Validator
(Jaeger API)"] + SV["Span Validator
(Tempo API)"] MV["Metric Validator
(Prometheus API,
all 26 metrics required)"] DV["Dashboard Validator
(Grafana API)"] BM["Benchmark Suite
(CPU, memory, latency
ON vs OFF comparison)"] @@ -852,7 +852,7 @@ See [Phase10_taskList.md](./Phase10_taskList.md) for detailed per-task breakdown The validation suite (`validate_telemetry.py`) runs exactly 71 checks, broken down as: -- **1 service registration** — `rippled` exists in Jaeger +- **1 service registration** — `rippled` exists in Tempo - **17 span existence** — `rpc.request`, `rpc.process`, `rpc.ws_message`, `rpc.command.*`, `tx.process`, `tx.receive`, `tx.apply`, `consensus.proposal.send`, `consensus.ledger_close`, `consensus.accept`, `consensus.validation.send`, `consensus.accept.apply`, `ledger.build`, `ledger.validate`, `ledger.store`, `peer.proposal.receive`, `peer.validation.receive` - **14 span attribute** — required attributes on the 14 spans that define them (22 unique attributes total) - **2 span hierarchies** — `rpc.process` -> `rpc.command.*`, `ledger.build` -> `tx.apply` (1 skipped: `rpc.request` -> `rpc.process`, cross-thread) diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md index 95664e3313..a4e15af34c 100644 --- a/OpenTelemetryPlan/09-data-collection-reference.md +++ b/OpenTelemetryPlan/09-data-collection-reference.md @@ -511,7 +511,7 @@ Example: 2024-01-15T10:30:45.123Z LedgerMaster:NFO trace_id=abc123def456789012345678abcdef01 span_id=0123456789abcdef Validated ledger 42 ``` -- **`trace_id=`** — 32-character lowercase hex trace identifier. Links to the distributed trace in Tempo/Jaeger. +- **`trace_id=`** — 32-character lowercase hex trace identifier. Links to the distributed trace in Tempo. - **`span_id=`** — 16-character lowercase hex span identifier. Identifies the specific span within the trace. - **Only present** when the log is emitted within an active OTel span. Log lines outside of traced code paths have no trace context fields. @@ -734,7 +734,7 @@ docker/telemetry/workload/benchmark.sh --xrpld .build/xrpld --duration 300 | Category | Expected Count | Validation Method | Config File | | ------------------ | -------------- | -------------------------------- | ----------------------- | -| Trace spans | 17 | Jaeger/Tempo API query | `expected_spans.json` | +| Trace spans | 17 | Tempo API query | `expected_spans.json` | | Span attributes | 22 | Per-span attribute assertion | `expected_spans.json` | | StatsD metrics | 255+ | Prometheus query | `expected_metrics.json` | | Phase 9 metrics | 68+ | Prometheus query | `expected_metrics.json` | diff --git a/OpenTelemetryPlan/OpenTelemetryPlan.md b/OpenTelemetryPlan/OpenTelemetryPlan.md index 7be3cb57ed..89f5ab0b47 100644 --- a/OpenTelemetryPlan/OpenTelemetryPlan.md +++ b/OpenTelemetryPlan/OpenTelemetryPlan.md @@ -228,7 +228,7 @@ The appendix contains a glossary of OpenTelemetry and rippled-specific terms, re ## 9. Data Collection Reference -A single-source-of-truth reference documenting every piece of telemetry data collected by rippled. Covers all 16 OpenTelemetry spans with their 22 attributes, all StatsD metrics (gauges, counters, histograms, overlay traffic), SpanMetrics-derived Prometheus metrics, and all 8 Grafana dashboards. Includes Jaeger search guides and Prometheus query examples. +A single-source-of-truth reference documenting every piece of telemetry data collected by rippled. Covers all 16 OpenTelemetry spans with their 22 attributes, all StatsD metrics (gauges, counters, histograms, overlay traffic), SpanMetrics-derived Prometheus metrics, and all 8 Grafana dashboards. Includes Tempo search guides and Prometheus query examples. ➡️ **[View Data Collection Reference](./09-data-collection-reference.md)** diff --git a/OpenTelemetryPlan/Phase3_taskList.md b/OpenTelemetryPlan/Phase3_taskList.md index cf86b737de..49468d0805 100644 --- a/OpenTelemetryPlan/Phase3_taskList.md +++ b/OpenTelemetryPlan/Phase3_taskList.md @@ -248,7 +248,7 @@ - [ ] `tx.receive` spans carry `xrpl.peer.version` attribute with a non-empty version string - [ ] Attribute is omitted (not set to empty string) when `getVersion()` returns empty -- [ ] Attribute visible in Jaeger span detail view +- [ ] Attribute visible in Tempo trace detail view --- diff --git a/OpenTelemetryPlan/Phase4_taskList.md b/OpenTelemetryPlan/Phase4_taskList.md index 3817183a22..96e4213bbc 100644 --- a/OpenTelemetryPlan/Phase4_taskList.md +++ b/OpenTelemetryPlan/Phase4_taskList.md @@ -334,7 +334,7 @@ Two strategies for cross-node trace correlation, switchable via config: Derive `trace_id = SHA256(previousLedger.id())[0:16]` so all nodes in the same consensus round share the same trace_id without P2P context propagation. -- **Pros**: All nodes appear in the same trace in Tempo/Jaeger automatically. +- **Pros**: All nodes appear in the same trace in Tempo automatically. No collector-side post-processing needed. - **Cons**: Overrides OTel's random trace_id generation; requires custom `IdGenerator` or manual span context construction. @@ -876,7 +876,7 @@ Received messages use **span links** (follows-from), NOT parent-child: - The receiver's processing span links to the sender's context - This preserves each node's independent trace tree -- Cross-node correlation visible via linked traces in Tempo/Jaeger +- Cross-node correlation visible via linked traces in Tempo ## Interaction with Deterministic Trace ID (Strategy A) diff --git a/docker/telemetry/TESTING.md b/docker/telemetry/TESTING.md index 1e6e654e7b..8b5bfdade1 100644 --- a/docker/telemetry/TESTING.md +++ b/docker/telemetry/TESTING.md @@ -482,17 +482,17 @@ severity code and the message. Example: Lines emitted outside of an active span (background tasks, startup) will NOT have trace context — this is expected. -### Step 2: Cross-check trace_id in Jaeger +### Step 2: Cross-check trace_id in Tempo -Extract a `trace_id` from the log and verify it exists in Jaeger: +Extract a `trace_id` from the log and verify it exists in Tempo: ```bash TRACE_ID=$(grep -o 'trace_id=[a-f0-9]\{32\}' /path/to/debug.log | head -1 | cut -d= -f2) echo "Checking trace: $TRACE_ID" -curl -s "http://localhost:16686/api/traces/$TRACE_ID" | jq '.data | length' +curl -s "http://localhost:3200/api/traces/$TRACE_ID" | jq '.batches | length' ``` -Expected result: `1` (the trace exists in Jaeger). +Expected result: `> 0` (the trace exists in Tempo). ### Step 3: Verify Loki log ingestion @@ -530,7 +530,7 @@ Expected: > 0 results. | `trace_id=` in debug.log | Present in log lines within active spans | | `span_id=` in debug.log | Present alongside trace_id | | Logs without active span | No trace_id/span_id fields | -| trace_id in Jaeger | Matches a valid trace | +| trace_id in Tempo | Matches a valid trace | | Loki log ingestion | Logs visible via LogQL | | Tempo -> Loki "Logs for trace" | Shows correlated log lines | | Loki -> Tempo TraceID link | Navigates to correct trace | diff --git a/docker/telemetry/integration-test.sh b/docker/telemetry/integration-test.sh index 79a6bcedf4..dc123395b1 100755 --- a/docker/telemetry/integration-test.sh +++ b/docker/telemetry/integration-test.sh @@ -67,7 +67,7 @@ check_span() { # Phase 8: Verify trace_id injection in rippled log output. # Greps all node debug.log files for the "trace_id= span_id=" # pattern that Logs::format() injects when an active OTel span exists. -# Also cross-checks that a trace_id found in logs matches a trace in Jaeger. +# Also cross-checks that a trace_id found in logs matches a trace in Tempo. check_log_correlation() { log "Checking log-trace correlation..." @@ -82,7 +82,7 @@ check_log_correlation() { local matches matches=$(grep -c 'trace_id=[a-f0-9]\{32\} span_id=[a-f0-9]\{16\}' "$logfile" 2>/dev/null || echo 0) total_matches=$((total_matches + matches)) - # Capture the first trace_id we find for cross-referencing with Jaeger + # Capture the first trace_id we find for cross-referencing with Tempo if [ -z "$sample_trace_id" ] && [ "$matches" -gt 0 ]; then sample_trace_id=$(grep -o 'trace_id=[a-f0-9]\{32\}' "$logfile" | head -1 | cut -d= -f2) fi @@ -94,15 +94,15 @@ check_log_correlation() { fail "Log correlation: no trace_id found in any node debug.log" fi - # Cross-check: verify the sample trace_id exists in Jaeger + # Cross-check: verify the sample trace_id exists in Tempo if [ -n "$sample_trace_id" ]; then local trace_found - trace_found=$(curl -sf "$JAEGER/api/traces/$sample_trace_id" \ - | jq '.data | length' 2>/dev/null || echo 0) + trace_found=$(curl -sf "$TEMPO/api/traces/$sample_trace_id" \ + | jq '.batches | length' 2>/dev/null || echo 0) if [ "$trace_found" -gt 0 ]; then - ok "Log-Jaeger cross-check: trace_id=$sample_trace_id found in Jaeger" + ok "Log-Tempo cross-check: trace_id=$sample_trace_id found in Tempo" else - fail "Log-Jaeger cross-check: trace_id=$sample_trace_id NOT found in Jaeger" + fail "Log-Tempo cross-check: trace_id=$sample_trace_id NOT found in Tempo" fi fi } diff --git a/docker/telemetry/xrpld-telemetry.cfg b/docker/telemetry/xrpld-telemetry.cfg index 2a96dd6ab5..2d5adf3692 100644 --- a/docker/telemetry/xrpld-telemetry.cfg +++ b/docker/telemetry/xrpld-telemetry.cfg @@ -7,7 +7,7 @@ # ./xrpld --conf docker/telemetry/xrpld-telemetry.cfg -a --start # 3. Send RPC commands to exercise tracing: # curl -s http://localhost:5005 -d '{"method":"server_info"}' -# 4. View traces in Jaeger UI: http://localhost:16686 +# 4. View traces in Grafana Explore -> Tempo: http://localhost:3000 [server] port_rpc_admin_local diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index 3afc40409f..f588c963c9 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -526,12 +526,12 @@ cat /tmp/xrpld-validation/reports/validation-report.json | jq '.summary' ### What Gets Validated -| Category | Checks | Description | -| ---------- | -------------- | -------------------------------------------------------- | -| Spans | 16+ span types | All span names appear in Jaeger with required attributes | -| Metrics | 30+ metrics | SpanMetrics, StatsD gauges/counters, Phase 9 metrics | -| Logs | 2 checks | trace_id/span_id present in Loki, cross-reference works | -| Dashboards | 10 dashboards | All Grafana dashboards load without errors | +| Category | Checks | Description | +| ---------- | -------------- | ------------------------------------------------------- | +| Spans | 16+ span types | All span names appear in Tempo with required attributes | +| Metrics | 30+ metrics | SpanMetrics, StatsD gauges/counters, Phase 9 metrics | +| Logs | 2 checks | trace_id/span_id present in Loki, cross-reference works | +| Dashboards | 10 dashboards | All Grafana dashboards load without errors | ### Running Individual Tools diff --git a/tasks/fix-validation-checks.md b/tasks/fix-validation-checks.md index 096920c524..44cacc102a 100644 --- a/tasks/fix-validation-checks.md +++ b/tasks/fix-validation-checks.md @@ -80,13 +80,13 @@ Likely causes (investigate in order): 2. **Code path not triggered:** `tx.process` fires in `NetworkOPs::processTransaction()`. The tx_submitter submits via RPC `submit` command which calls this path. But if the transactions fail validation before reaching `processTransaction()`, no span is emitted. -3. **Span naming mismatch:** The validation queries Jaeger for exact operation name - `tx.process`. Verify Jaeger stores the span with this exact name. +3. **Span naming mismatch:** The validation queries Tempo for exact operation name + `tx.process`. Verify Tempo stores the span with this exact name. **Investigation:** - Check the tx_submitter output in CI logs — are transactions actually succeeding? -- Query Jaeger API locally for all span names to see what's actually emitted. +- Query Tempo API locally for all span names to see what's actually emitted. **Files to modify:** @@ -103,7 +103,7 @@ Likely causes (investigate in order): [FAIL] span.hierarchy.rpc.request->rpc.process: rpc.process not found in rpc.request traces ``` -**Root Cause:** The validator fetches traces containing `rpc.request` from Jaeger and +**Root Cause:** The validator fetches traces containing `rpc.request` from Tempo and checks if any child span is named `rpc.process`. Both spans are emitted (they pass individual checks), but the parent-child relationship isn't established.