merge: bring the component renames forward from phase8-log-correlation

Four doc conflicts, all where this branch had rewritten a passage that upstream
had only renamed. This branch's text is kept in seven of the eight hunks and
the spanmetrics -> span_metrics and otlp/tempo -> otlp_grpc/tempo spellings
carried into it, so the rewrite is not lost and the names stay current.

The exception is the TESTING.md span-call-count comment, where the incoming
side is the fuller text: it explains that the span_ prefix comes from the
connector's namespace setting. That side is taken.

Metric names are untouched — span_calls_total and traces_span_metrics_* are
produced by the connector's namespace, not by its component name.
This commit is contained in:
Pratik Mankawde
2026-09-09 19:22:16 +01:00
9 changed files with 38 additions and 30 deletions

View File

@@ -184,7 +184,7 @@ Run the integration test script:
bash docker/telemetry/integration-test.sh
```
It checks prerequisites, clears the previous run, brings up the observability stack, generates six validator key pairs and their node configs, starts the nodes, waits for consensus and then for a validated ledger, exercises RPC and submits a transaction, verifies traces in Tempo and both the spanmetrics and the native `beast::insight` metrics that arrive over OTLP in Prometheus, checks that no StatsD listener is needed, then prints a summary and leaves the stack running.
It checks prerequisites, clears the previous run, brings up the observability stack, generates six validator key pairs and their node configs, starts the nodes, waits for consensus and then for a validated ledger, exercises RPC and submits a transaction, verifies traces in Tempo and both the span_metrics and the native `beast::insight` metrics that arrive over OTLP in Prometheus, checks that no StatsD listener is needed, then prints a summary and leaves the stack running.
The script announces each step as it runs, so read its `Step N:` headers for the authoritative sequence — they are not restated here, because a numbered copy of them drifts as soon as a step is added.
@@ -465,7 +465,9 @@ Base URL: `http://localhost:9090`
```bash
PROM="http://localhost:9090"
# Span call counts (from spanmetrics connector)
# Span call counts (from the span_metrics connector). The span_ prefix is the
# connector's `namespace: "span"` in otel-collector-config.yaml; drop that
# setting and these become traces_span_metrics_*.
curl -s "$PROM/api/v1/query?query=span_calls_total" |
jq '.data.result[] | {span: .metric.span_name, count: .value[1]}'
@@ -789,14 +791,14 @@ Counting `.data.result | length` would count streams, not log lines.
### Spanmetrics not appearing in Prometheus
1. Verify otel-collector config has `spanmetrics` connector
1. Verify otel-collector config has `span_metrics` connector
2. Check that the metrics pipeline matches `otel-collector-config.yaml`
verbatim:
```yaml
service:
pipelines:
metrics:
receivers: [otlp, spanmetrics]
receivers: [otlp, span_metrics]
processors: [resource/tier, resource/stripsdk, batch]
exporters: [prometheus]
```

View File

@@ -84,10 +84,11 @@ services:
ports:
- "4317:4317" # OTLP gRPC
- "4318:4318" # OTLP HTTP (traces + native OTel metrics)
- "8889:8889" # Prometheus metrics (spanmetrics + OTLP)
- "8889:8889" # Prometheus metrics (span_metrics + OTLP)
# StatsD UDP port removed — beast::insight now uses native OTLP.
# Uncomment if using server=statsd fallback:
# - "8125:8125/udp"
- "13133:13133" # Health check
volumes:
# Mount collector pipeline config (receivers → processors → exporters)
- ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:ro

View File

@@ -207,8 +207,9 @@ cleanup() {
done
# Also kill any straggling xrpld processes from our workdir
pkill -f "$WORKDIR" 2>/dev/null || true
# Stop docker stack
docker compose -f "$COMPOSE_FILE" down 2>/dev/null || true
# Stop docker stack. -v also drops the tempo-data volume: plain `down`
# keeps it, and retained traces would then answer a later run's searches.
docker compose -f "$COMPOSE_FILE" down -v 2>/dev/null || true
# Remove workdir
rm -rf "$WORKDIR"
log "Cleanup complete."
@@ -250,6 +251,10 @@ pkill -f "$WORKDIR" 2>/dev/null || true
pkill -f "xrpld-telemetry.cfg" 2>/dev/null || true
sleep 2
rm -rf "$WORKDIR"
# A run that reached the summary left the stack up, so nothing has torn it
# down. Do it here, with -v: Tempo's traces and Prometheus' samples must not
# survive into this run, or an assertion can pass on the previous run's data.
docker compose -f "$COMPOSE_FILE" down -v 2>/dev/null || true
mkdir -p "$WORKDIR"
# ---------------------------------------------------------------------------
@@ -687,7 +692,7 @@ log "--- Log-Trace Correlation ---"
check_log_correlation
# ---------------------------------------------------------------------------
# Step 10: Verify Prometheus spanmetrics
# Step 10: Verify Prometheus span_metrics
# ---------------------------------------------------------------------------
log ""
log "--- Spanmetrics ---"

View File

@@ -1,12 +1,12 @@
# OpenTelemetry Collector configuration for xrpld development.
#
# Pipelines:
# traces: OTLP receiver -> batch processor -> debug + Tempo + spanmetrics
# metrics: OTLP receiver + spanmetrics connector -> Prometheus exporter
# traces: OTLP receiver -> batch processor -> debug + Tempo + span_metrics
# metrics: OTLP receiver + span_metrics connector -> Prometheus exporter
# logs: file_log receiver -> batch processor -> otlp_http/Loki
#
# xrpld sends traces via OTLP/HTTP to port 4318. The collector batches
# them, forwards to Tempo, and derives RED metrics via the spanmetrics
# them, forwards to Tempo, and derives RED metrics via the span_metrics
# connector, which Prometheus scrapes on port 8889.
#
# xrpld sends beast::insight metrics natively via OTLP/HTTP to port 4318
@@ -147,7 +147,7 @@ processors:
action: hash
connectors:
spanmetrics:
span_metrics:
namespace: "span"
# Resource attributes that define the resource-metrics grouping key.
# All resource attributes are propagated onto the output metrics
@@ -240,7 +240,7 @@ connectors:
exporters:
debug:
verbosity: detailed
otlp/tempo:
otlp_grpc/tempo:
endpoint: tempo:4317
tls:
insecure: true
@@ -268,9 +268,9 @@ service:
traces:
receivers: [otlp]
processors: [resource/tier, resource/stripsdk, attributes/hash, batch]
exporters: [debug, otlp/tempo, spanmetrics]
exporters: [debug, otlp_grpc/tempo, span_metrics]
metrics:
receivers: [otlp, spanmetrics]
receivers: [otlp, span_metrics]
# batch keeps the OTLP metric path from exporting one request per
# instrument. It delays a sample by at most the batch timeout, which
# is well under the Prometheus scrape interval.

View File

@@ -1,4 +1,4 @@
# Prometheus configuration for scraping spanmetrics from OTel Collector.
# Prometheus configuration for scraping span_metrics from OTel Collector.
global:
scrape_interval: 15s
evaluation_interval: 15s