diff --git a/OpenTelemetryPlan/02-design-decisions.md b/OpenTelemetryPlan/02-design-decisions.md index f7a7ee7022..1ccf7c9285 100644 --- a/OpenTelemetryPlan/02-design-decisions.md +++ b/OpenTelemetryPlan/02-design-decisions.md @@ -220,7 +220,7 @@ agree with the code. A CI check enforces this end to end. `validation_trusted` likewise). Defined once in the base `SpanNames.h` `namespace attr` block and re-exported (`using`) by each domain header. 3. **Collision qualifier** → `_`, only when a bare name would - collide with a DIFFERENT concept in the shared spanmetrics label space or with + collide with a DIFFERENT concept in the shared span_metrics label space or with the OTel-reserved `status` key (e.g. `rpc_status`, `grpc_status`, `consensus_phase`, `consensus_round`, `consensus_mode`). This disambiguates distinct concepts that share a word; it is NOT used to tag the same concept diff --git a/OpenTelemetryPlan/05-configuration-reference.md b/OpenTelemetryPlan/05-configuration-reference.md index 13078518ea..41796b3590 100644 --- a/OpenTelemetryPlan/05-configuration-reference.md +++ b/OpenTelemetryPlan/05-configuration-reference.md @@ -247,11 +247,11 @@ The authoritative collector config lives in the repo at `docker/telemetry/otel-c `docker/telemetry/otel-collector-config.yaml` is the base config used by the local stack and by CI. It carries **three** pipelines, not one: -| Pipeline | Receivers | Processors | Exporters | -| --------- | --------------------- | ---------------------------------------------------------------- | ------------------------------------ | -| `traces` | `otlp` | `resource/tier`, `resource/stripsdk`, `attributes/hash`, `batch` | `debug`, `otlp/tempo`, `spanmetrics` | -| `metrics` | `otlp`, `spanmetrics` | `resource/tier`, `resource/stripsdk`, `batch` | `prometheus` | -| `logs` | `file_log` | `resource/logs`, `resource/tier`, `resource/stripsdk`, `batch` | `otlp_http/loki` | +| Pipeline | Receivers | Processors | Exporters | +| --------- | ---------------------- | ---------------------------------------------------------------- | ------------------------------------------ | +| `traces` | `otlp` | `resource/tier`, `resource/stripsdk`, `attributes/hash`, `batch` | `debug`, `otlp_grpc/tempo`, `span_metrics` | +| `metrics` | `otlp`, `span_metrics` | `resource/tier`, `resource/stripsdk`, `batch` | `prometheus` | +| `logs` | `file_log` | `resource/logs`, `resource/tier`, `resource/stripsdk`, `batch` | `otlp_http/loki` | Component detail: @@ -267,12 +267,12 @@ Component detail: `service.name` and `job` — only the former becomes a Loki stream label, see the known issue in §5.8.5); `attributes/hash` (hashes `pathfind_source_account` and `pathfind_dest_account`). -- **Connector.** `spanmetrics` with `namespace: "span"` +- **Connector.** `span_metrics` with `namespace: "span"` (`otel-collector-config.yaml:114`) — this is why the derived RED metrics are `span_calls_total` / `span_duration_milliseconds_*`. The connector's own default namespace is **empty**, so without this setting the names would be the bare `calls_total` / `duration_milliseconds_*`. The - `traces_spanmetrics_*` family is **not** the connector's default and is not + `traces_span_metrics_*` family is **not** the connector's default and is not produced here at all — it comes from a different producer, Tempo's `metrics_generator` `span-metrics` processor (`tempo.yaml:75`), whose `remote_write` is commented out in this repo (see §5.8.6). Histogram @@ -281,7 +281,7 @@ Component detail: boundaries for consensus and `ledger.acquire`. ~25 low-cardinality dimensions are promoted to labels (`command`, `rpc_status`, `tx_type`, `ter_result`, `stage`, `consensus_mode`, `outcome`, …). -- **Exporters.** `debug` (console, `verbosity: detailed`), `otlp/tempo` +- **Exporters.** `debug` (console, `verbosity: detailed`), `otlp_grpc/tempo` (`tempo:4317`, `tls.insecure: true`), `otlp_http/loki` (`http://loki:3100/otlp` — Loki 3.x native OTLP; the old `loki` exporter was removed in collector-contrib v0.147.0), and `prometheus` on diff --git a/OpenTelemetryPlan/07-observability-backends.md b/OpenTelemetryPlan/07-observability-backends.md index 993d5b994a..9029e2ea76 100644 --- a/OpenTelemetryPlan/07-observability-backends.md +++ b/OpenTelemetryPlan/07-observability-backends.md @@ -364,7 +364,7 @@ alert catalogue, not here. ### 7.6.3 Writing New Rules: the metric name If you add a span-metric alert, the metric is **`span_calls_total`**. This stack -sets the `spanmetrics` connector's `namespace: "span"` +sets the `span_metrics` connector's `namespace: "span"` (`otel-collector-config.yaml:114`); the connector's own default namespace is **empty**, so without that setting the names would be the bare `calls_total` / `duration_milliseconds_*`. 7 of the 15 dashboards already query the `span_` diff --git a/docker/telemetry/TESTING.md b/docker/telemetry/TESTING.md index 44b29a1e87..738fe71fac 100644 --- a/docker/telemetry/TESTING.md +++ b/docker/telemetry/TESTING.md @@ -184,7 +184,7 @@ Run the integration test script: bash docker/telemetry/integration-test.sh ``` -It checks prerequisites, clears the previous run, brings up the observability stack, generates six validator key pairs and their node configs, starts the nodes, waits for consensus and then for a validated ledger, exercises RPC and submits a transaction, verifies traces in Tempo and both the spanmetrics and the native `beast::insight` metrics that arrive over OTLP in Prometheus, checks that no StatsD listener is needed, then prints a summary and leaves the stack running. +It checks prerequisites, clears the previous run, brings up the observability stack, generates six validator key pairs and their node configs, starts the nodes, waits for consensus and then for a validated ledger, exercises RPC and submits a transaction, verifies traces in Tempo and both the span_metrics and the native `beast::insight` metrics that arrive over OTLP in Prometheus, checks that no StatsD listener is needed, then prints a summary and leaves the stack running. The script announces each step as it runs, so read its `Step N:` headers for the authoritative sequence — they are not restated here, because a numbered copy of them drifts as soon as a step is added. @@ -465,7 +465,9 @@ Base URL: `http://localhost:9090` ```bash PROM="http://localhost:9090" -# Span call counts (from spanmetrics connector) +# Span call counts (from the span_metrics connector). The span_ prefix is the +# connector's `namespace: "span"` in otel-collector-config.yaml; drop that +# setting and these become traces_span_metrics_*. curl -s "$PROM/api/v1/query?query=span_calls_total" | jq '.data.result[] | {span: .metric.span_name, count: .value[1]}' @@ -789,14 +791,14 @@ Counting `.data.result | length` would count streams, not log lines. ### Spanmetrics not appearing in Prometheus -1. Verify otel-collector config has `spanmetrics` connector +1. Verify otel-collector config has `span_metrics` connector 2. Check that the metrics pipeline matches `otel-collector-config.yaml` verbatim: ```yaml service: pipelines: metrics: - receivers: [otlp, spanmetrics] + receivers: [otlp, span_metrics] processors: [resource/tier, resource/stripsdk, batch] exporters: [prometheus] ``` diff --git a/docker/telemetry/docker-compose.yml b/docker/telemetry/docker-compose.yml index 68e019d759..1cd02c9647 100644 --- a/docker/telemetry/docker-compose.yml +++ b/docker/telemetry/docker-compose.yml @@ -84,10 +84,11 @@ services: ports: - "4317:4317" # OTLP gRPC - "4318:4318" # OTLP HTTP (traces + native OTel metrics) - - "8889:8889" # Prometheus metrics (spanmetrics + OTLP) + - "8889:8889" # Prometheus metrics (span_metrics + OTLP) # StatsD UDP port removed — beast::insight now uses native OTLP. # Uncomment if using server=statsd fallback: # - "8125:8125/udp" + - "13133:13133" # Health check volumes: # Mount collector pipeline config (receivers → processors → exporters) - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:ro diff --git a/docker/telemetry/integration-test.sh b/docker/telemetry/integration-test.sh index 01578b8d5a..71747301bb 100755 --- a/docker/telemetry/integration-test.sh +++ b/docker/telemetry/integration-test.sh @@ -207,8 +207,9 @@ cleanup() { done # Also kill any straggling xrpld processes from our workdir pkill -f "$WORKDIR" 2>/dev/null || true - # Stop docker stack - docker compose -f "$COMPOSE_FILE" down 2>/dev/null || true + # Stop docker stack. -v also drops the tempo-data volume: plain `down` + # keeps it, and retained traces would then answer a later run's searches. + docker compose -f "$COMPOSE_FILE" down -v 2>/dev/null || true # Remove workdir rm -rf "$WORKDIR" log "Cleanup complete." @@ -250,6 +251,10 @@ pkill -f "$WORKDIR" 2>/dev/null || true pkill -f "xrpld-telemetry.cfg" 2>/dev/null || true sleep 2 rm -rf "$WORKDIR" +# A run that reached the summary left the stack up, so nothing has torn it +# down. Do it here, with -v: Tempo's traces and Prometheus' samples must not +# survive into this run, or an assertion can pass on the previous run's data. +docker compose -f "$COMPOSE_FILE" down -v 2>/dev/null || true mkdir -p "$WORKDIR" # --------------------------------------------------------------------------- @@ -687,7 +692,7 @@ log "--- Log-Trace Correlation ---" check_log_correlation # --------------------------------------------------------------------------- -# Step 10: Verify Prometheus spanmetrics +# Step 10: Verify Prometheus span_metrics # --------------------------------------------------------------------------- log "" log "--- Spanmetrics ---" diff --git a/docker/telemetry/otel-collector-config.yaml b/docker/telemetry/otel-collector-config.yaml index e9f04fab6e..5d238cdae7 100644 --- a/docker/telemetry/otel-collector-config.yaml +++ b/docker/telemetry/otel-collector-config.yaml @@ -1,12 +1,12 @@ # OpenTelemetry Collector configuration for xrpld development. # # Pipelines: -# traces: OTLP receiver -> batch processor -> debug + Tempo + spanmetrics -# metrics: OTLP receiver + spanmetrics connector -> Prometheus exporter +# traces: OTLP receiver -> batch processor -> debug + Tempo + span_metrics +# metrics: OTLP receiver + span_metrics connector -> Prometheus exporter # logs: file_log receiver -> batch processor -> otlp_http/Loki # # xrpld sends traces via OTLP/HTTP to port 4318. The collector batches -# them, forwards to Tempo, and derives RED metrics via the spanmetrics +# them, forwards to Tempo, and derives RED metrics via the span_metrics # connector, which Prometheus scrapes on port 8889. # # xrpld sends beast::insight metrics natively via OTLP/HTTP to port 4318 @@ -147,7 +147,7 @@ processors: action: hash connectors: - spanmetrics: + span_metrics: namespace: "span" # Resource attributes that define the resource-metrics grouping key. # All resource attributes are propagated onto the output metrics @@ -240,7 +240,7 @@ connectors: exporters: debug: verbosity: detailed - otlp/tempo: + otlp_grpc/tempo: endpoint: tempo:4317 tls: insecure: true @@ -268,9 +268,9 @@ service: traces: receivers: [otlp] processors: [resource/tier, resource/stripsdk, attributes/hash, batch] - exporters: [debug, otlp/tempo, spanmetrics] + exporters: [debug, otlp_grpc/tempo, span_metrics] metrics: - receivers: [otlp, spanmetrics] + receivers: [otlp, span_metrics] # batch keeps the OTLP metric path from exporting one request per # instrument. It delays a sample by at most the batch timeout, which # is well under the Prometheus scrape interval. diff --git a/docker/telemetry/prometheus.yml b/docker/telemetry/prometheus.yml index d99d919a55..7d87649c51 100644 --- a/docker/telemetry/prometheus.yml +++ b/docker/telemetry/prometheus.yml @@ -1,4 +1,4 @@ -# Prometheus configuration for scraping spanmetrics from OTel Collector. +# Prometheus configuration for scraping span_metrics from OTel Collector. global: scrape_interval: 15s evaluation_interval: 15s diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index 691bb4e2c2..e1098a30b4 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -284,7 +284,7 @@ this span: count successes as total minus error, or filter on `status_code`. The three apply-pipeline spans (`tx.preflight`, `tx.preclaim`, `tx.transactor`) share a deterministic `trace_id` from `txID[0:16]`, so they group under one trace per transaction. The `stage` attribute (`preflight` / `preclaim` / -`apply`) drives the collector spanmetrics `stage` dimension, giving per-stage +`apply`) drives the collector span_metrics `stage` dimension, giving per-stage RED metrics on the _Transaction Overview_ dashboard. `current_ledger_seq` is the current (open/in-flight) ledger index a span acted on @@ -1333,7 +1333,7 @@ sum by (stage) (rate(span_calls_total{span_name=~"tx.preflight|tx.preclaim|tx.tr > in `otel-collector-config.grafanacloud.yaml` — the base > `otel-collector-config.yaml` has no tail sampling at all, so a stock local > stack retains every trace. Where that Cloud policy is in force it applies to -> the trace-storage branch only; spanmetrics run on a separate branch and still +> the trace-storage branch only; span_metrics run on a separate branch and still > see 100% of spans, so the derived RED metrics stay exact either way. ### Transaction Queue Health @@ -1569,7 +1569,7 @@ all its normal attributes, it just lacks a cross-node parent link. ## Prometheus Metrics (Spanmetrics) -The OTel Collector's spanmetrics connector automatically derives RED (Rate, Errors, Duration) metrics from every span. No custom metrics code is needed in xrpld. +The OTel Collector's span_metrics connector automatically derives RED (Rate, Errors, Duration) metrics from every span. No custom metrics code is needed in xrpld. ### Generated Metric Names @@ -2162,7 +2162,7 @@ collector settings make it work, both already enabled: - `prometheus.resource_to_telemetry_conversion: enabled: true` promotes resource attributes to metric labels on the local scrape surface. -- `spanmetrics.resource_metrics_key_attributes` lists the tier attributes so +- `span_metrics.resource_metrics_key_attributes` lists the tier attributes so span-derived series stay grouped per node and tier. Traces and logs carry resource attributes natively; Grafana Cloud ingests all