mirror of
https://github.com/XRPLF/rippled.git
synced 2026-09-26 23:19:07 +00:00
merge: bring the component renames forward from phase8-log-correlation
Four doc conflicts, all where this branch had rewritten a passage that upstream had only renamed. This branch's text is kept in seven of the eight hunks and the spanmetrics -> span_metrics and otlp/tempo -> otlp_grpc/tempo spellings carried into it, so the rewrite is not lost and the names stay current. The exception is the TESTING.md span-call-count comment, where the incoming side is the fuller text: it explains that the span_ prefix comes from the connector's namespace setting. That side is taken. Metric names are untouched — span_calls_total and traces_span_metrics_* are produced by the connector's namespace, not by its component name.
This commit is contained in:
Pratik Mankawde
@@ -220,7 +220,7 @@ agree with the code. A CI check enforces this end to end.
|
||||
`validation_trusted` likewise). Defined once in the base `SpanNames.h`
|
||||
`namespace attr` block and re-exported (`using`) by each domain header.
|
||||
3. **Collision qualifier** → `<domain>_<field>`, only when a bare name would
|
||||
collide with a DIFFERENT concept in the shared spanmetrics label space or with
|
||||
collide with a DIFFERENT concept in the shared span_metrics label space or with
|
||||
the OTel-reserved `status` key (e.g. `rpc_status`, `grpc_status`,
|
||||
`consensus_phase`, `consensus_round`, `consensus_mode`). This disambiguates
|
||||
distinct concepts that share a word; it is NOT used to tag the same concept
|
||||
|
||||
@@ -247,11 +247,11 @@ The authoritative collector config lives in the repo at `docker/telemetry/otel-c
|
||||
`docker/telemetry/otel-collector-config.yaml` is the base config used by the
|
||||
local stack and by CI. It carries **three** pipelines, not one:
|
||||
|
||||
| Pipeline | Receivers | Processors | Exporters |
|
||||
| --------- | --------------------- | ---------------------------------------------------------------- | ------------------------------------ |
|
||||
| `traces` | `otlp` | `resource/tier`, `resource/stripsdk`, `attributes/hash`, `batch` | `debug`, `otlp/tempo`, `spanmetrics` |
|
||||
| `metrics` | `otlp`, `spanmetrics` | `resource/tier`, `resource/stripsdk`, `batch` | `prometheus` |
|
||||
| `logs` | `file_log` | `resource/logs`, `resource/tier`, `resource/stripsdk`, `batch` | `otlp_http/loki` |
|
||||
| Pipeline | Receivers | Processors | Exporters |
|
||||
| --------- | ---------------------- | ---------------------------------------------------------------- | ------------------------------------------ |
|
||||
| `traces` | `otlp` | `resource/tier`, `resource/stripsdk`, `attributes/hash`, `batch` | `debug`, `otlp_grpc/tempo`, `span_metrics` |
|
||||
| `metrics` | `otlp`, `span_metrics` | `resource/tier`, `resource/stripsdk`, `batch` | `prometheus` |
|
||||
| `logs` | `file_log` | `resource/logs`, `resource/tier`, `resource/stripsdk`, `batch` | `otlp_http/loki` |
|
||||
|
||||
Component detail:
|
||||
|
||||
@@ -267,12 +267,12 @@ Component detail:
|
||||
`service.name` and `job` — only the former becomes a Loki stream label, see
|
||||
the known issue in §5.8.5); `attributes/hash` (hashes
|
||||
`pathfind_source_account` and `pathfind_dest_account`).
|
||||
- **Connector.** `spanmetrics` with `namespace: "span"`
|
||||
- **Connector.** `span_metrics` with `namespace: "span"`
|
||||
(`otel-collector-config.yaml:114`) — this is why the derived RED metrics are
|
||||
`span_calls_total` / `span_duration_milliseconds_*`. The connector's own
|
||||
default namespace is **empty**, so without this setting the names would be
|
||||
the bare `calls_total` / `duration_milliseconds_*`. The
|
||||
`traces_spanmetrics_*` family is **not** the connector's default and is not
|
||||
`traces_span_metrics_*` family is **not** the connector's default and is not
|
||||
produced here at all — it comes from a different producer, Tempo's
|
||||
`metrics_generator` `span-metrics` processor (`tempo.yaml:75`), whose
|
||||
`remote_write` is commented out in this repo (see §5.8.6). Histogram
|
||||
@@ -281,7 +281,7 @@ Component detail:
|
||||
boundaries for consensus and `ledger.acquire`. ~25 low-cardinality
|
||||
dimensions are promoted to labels (`command`, `rpc_status`, `tx_type`,
|
||||
`ter_result`, `stage`, `consensus_mode`, `outcome`, …).
|
||||
- **Exporters.** `debug` (console, `verbosity: detailed`), `otlp/tempo`
|
||||
- **Exporters.** `debug` (console, `verbosity: detailed`), `otlp_grpc/tempo`
|
||||
(`tempo:4317`, `tls.insecure: true`), `otlp_http/loki`
|
||||
(`http://loki:3100/otlp` — Loki 3.x native OTLP; the old `loki` exporter was
|
||||
removed in collector-contrib v0.147.0), and `prometheus` on
|
||||
|
||||
@@ -364,7 +364,7 @@ alert catalogue, not here.
|
||||
### 7.6.3 Writing New Rules: the metric name
|
||||
|
||||
If you add a span-metric alert, the metric is **`span_calls_total`**. This stack
|
||||
sets the `spanmetrics` connector's `namespace: "span"`
|
||||
sets the `span_metrics` connector's `namespace: "span"`
|
||||
(`otel-collector-config.yaml:114`); the connector's own default namespace is
|
||||
**empty**, so without that setting the names would be the bare `calls_total` /
|
||||
`duration_milliseconds_*`. 7 of the 15 dashboards already query the `span_`
|
||||
|
||||
@@ -184,7 +184,7 @@ Run the integration test script:
|
||||
bash docker/telemetry/integration-test.sh
|
||||
```
|
||||
|
||||
It checks prerequisites, clears the previous run, brings up the observability stack, generates six validator key pairs and their node configs, starts the nodes, waits for consensus and then for a validated ledger, exercises RPC and submits a transaction, verifies traces in Tempo and both the spanmetrics and the native `beast::insight` metrics that arrive over OTLP in Prometheus, checks that no StatsD listener is needed, then prints a summary and leaves the stack running.
|
||||
It checks prerequisites, clears the previous run, brings up the observability stack, generates six validator key pairs and their node configs, starts the nodes, waits for consensus and then for a validated ledger, exercises RPC and submits a transaction, verifies traces in Tempo and both the span_metrics and the native `beast::insight` metrics that arrive over OTLP in Prometheus, checks that no StatsD listener is needed, then prints a summary and leaves the stack running.
|
||||
|
||||
The script announces each step as it runs, so read its `Step N:` headers for the authoritative sequence — they are not restated here, because a numbered copy of them drifts as soon as a step is added.
|
||||
|
||||
@@ -465,7 +465,9 @@ Base URL: `http://localhost:9090`
|
||||
```bash
|
||||
PROM="http://localhost:9090"
|
||||
|
||||
# Span call counts (from spanmetrics connector)
|
||||
# Span call counts (from the span_metrics connector). The span_ prefix is the
|
||||
# connector's `namespace: "span"` in otel-collector-config.yaml; drop that
|
||||
# setting and these become traces_span_metrics_*.
|
||||
curl -s "$PROM/api/v1/query?query=span_calls_total" |
|
||||
jq '.data.result[] | {span: .metric.span_name, count: .value[1]}'
|
||||
|
||||
@@ -789,14 +791,14 @@ Counting `.data.result | length` would count streams, not log lines.
|
||||
|
||||
### Spanmetrics not appearing in Prometheus
|
||||
|
||||
1. Verify otel-collector config has `spanmetrics` connector
|
||||
1. Verify otel-collector config has `span_metrics` connector
|
||||
2. Check that the metrics pipeline matches `otel-collector-config.yaml`
|
||||
verbatim:
|
||||
```yaml
|
||||
service:
|
||||
pipelines:
|
||||
metrics:
|
||||
receivers: [otlp, spanmetrics]
|
||||
receivers: [otlp, span_metrics]
|
||||
processors: [resource/tier, resource/stripsdk, batch]
|
||||
exporters: [prometheus]
|
||||
```
|
||||
|
||||
@@ -84,10 +84,11 @@ services:
|
||||
ports:
|
||||
- "4317:4317" # OTLP gRPC
|
||||
- "4318:4318" # OTLP HTTP (traces + native OTel metrics)
|
||||
- "8889:8889" # Prometheus metrics (spanmetrics + OTLP)
|
||||
- "8889:8889" # Prometheus metrics (span_metrics + OTLP)
|
||||
# StatsD UDP port removed — beast::insight now uses native OTLP.
|
||||
# Uncomment if using server=statsd fallback:
|
||||
# - "8125:8125/udp"
|
||||
- "13133:13133" # Health check
|
||||
volumes:
|
||||
# Mount collector pipeline config (receivers → processors → exporters)
|
||||
- ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:ro
|
||||
|
||||
@@ -207,8 +207,9 @@ cleanup() {
|
||||
done
|
||||
# Also kill any straggling xrpld processes from our workdir
|
||||
pkill -f "$WORKDIR" 2>/dev/null || true
|
||||
# Stop docker stack
|
||||
docker compose -f "$COMPOSE_FILE" down 2>/dev/null || true
|
||||
# Stop docker stack. -v also drops the tempo-data volume: plain `down`
|
||||
# keeps it, and retained traces would then answer a later run's searches.
|
||||
docker compose -f "$COMPOSE_FILE" down -v 2>/dev/null || true
|
||||
# Remove workdir
|
||||
rm -rf "$WORKDIR"
|
||||
log "Cleanup complete."
|
||||
@@ -250,6 +251,10 @@ pkill -f "$WORKDIR" 2>/dev/null || true
|
||||
pkill -f "xrpld-telemetry.cfg" 2>/dev/null || true
|
||||
sleep 2
|
||||
rm -rf "$WORKDIR"
|
||||
# A run that reached the summary left the stack up, so nothing has torn it
|
||||
# down. Do it here, with -v: Tempo's traces and Prometheus' samples must not
|
||||
# survive into this run, or an assertion can pass on the previous run's data.
|
||||
docker compose -f "$COMPOSE_FILE" down -v 2>/dev/null || true
|
||||
mkdir -p "$WORKDIR"
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
@@ -687,7 +692,7 @@ log "--- Log-Trace Correlation ---"
|
||||
check_log_correlation
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Step 10: Verify Prometheus spanmetrics
|
||||
# Step 10: Verify Prometheus span_metrics
|
||||
# ---------------------------------------------------------------------------
|
||||
log ""
|
||||
log "--- Spanmetrics ---"
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
# OpenTelemetry Collector configuration for xrpld development.
|
||||
#
|
||||
# Pipelines:
|
||||
# traces: OTLP receiver -> batch processor -> debug + Tempo + spanmetrics
|
||||
# metrics: OTLP receiver + spanmetrics connector -> Prometheus exporter
|
||||
# traces: OTLP receiver -> batch processor -> debug + Tempo + span_metrics
|
||||
# metrics: OTLP receiver + span_metrics connector -> Prometheus exporter
|
||||
# logs: file_log receiver -> batch processor -> otlp_http/Loki
|
||||
#
|
||||
# xrpld sends traces via OTLP/HTTP to port 4318. The collector batches
|
||||
# them, forwards to Tempo, and derives RED metrics via the spanmetrics
|
||||
# them, forwards to Tempo, and derives RED metrics via the span_metrics
|
||||
# connector, which Prometheus scrapes on port 8889.
|
||||
#
|
||||
# xrpld sends beast::insight metrics natively via OTLP/HTTP to port 4318
|
||||
@@ -147,7 +147,7 @@ processors:
|
||||
action: hash
|
||||
|
||||
connectors:
|
||||
spanmetrics:
|
||||
span_metrics:
|
||||
namespace: "span"
|
||||
# Resource attributes that define the resource-metrics grouping key.
|
||||
# All resource attributes are propagated onto the output metrics
|
||||
@@ -240,7 +240,7 @@ connectors:
|
||||
exporters:
|
||||
debug:
|
||||
verbosity: detailed
|
||||
otlp/tempo:
|
||||
otlp_grpc/tempo:
|
||||
endpoint: tempo:4317
|
||||
tls:
|
||||
insecure: true
|
||||
@@ -268,9 +268,9 @@ service:
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [resource/tier, resource/stripsdk, attributes/hash, batch]
|
||||
exporters: [debug, otlp/tempo, spanmetrics]
|
||||
exporters: [debug, otlp_grpc/tempo, span_metrics]
|
||||
metrics:
|
||||
receivers: [otlp, spanmetrics]
|
||||
receivers: [otlp, span_metrics]
|
||||
# batch keeps the OTLP metric path from exporting one request per
|
||||
# instrument. It delays a sample by at most the batch timeout, which
|
||||
# is well under the Prometheus scrape interval.
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Prometheus configuration for scraping spanmetrics from OTel Collector.
|
||||
# Prometheus configuration for scraping span_metrics from OTel Collector.
|
||||
global:
|
||||
scrape_interval: 15s
|
||||
evaluation_interval: 15s
|
||||
|
||||
@@ -284,7 +284,7 @@ this span: count successes as total minus error, or filter on `status_code`.
|
||||
The three apply-pipeline spans (`tx.preflight`, `tx.preclaim`, `tx.transactor`)
|
||||
share a deterministic `trace_id` from `txID[0:16]`, so they group under one
|
||||
trace per transaction. The `stage` attribute (`preflight` / `preclaim` /
|
||||
`apply`) drives the collector spanmetrics `stage` dimension, giving per-stage
|
||||
`apply`) drives the collector span_metrics `stage` dimension, giving per-stage
|
||||
RED metrics on the _Transaction Overview_ dashboard.
|
||||
|
||||
`current_ledger_seq` is the current (open/in-flight) ledger index a span acted on
|
||||
@@ -1333,7 +1333,7 @@ sum by (stage) (rate(span_calls_total{span_name=~"tx.preflight|tx.preclaim|tx.tr
|
||||
> in `otel-collector-config.grafanacloud.yaml` — the base
|
||||
> `otel-collector-config.yaml` has no tail sampling at all, so a stock local
|
||||
> stack retains every trace. Where that Cloud policy is in force it applies to
|
||||
> the trace-storage branch only; spanmetrics run on a separate branch and still
|
||||
> the trace-storage branch only; span_metrics run on a separate branch and still
|
||||
> see 100% of spans, so the derived RED metrics stay exact either way.
|
||||
|
||||
### Transaction Queue Health
|
||||
@@ -1569,7 +1569,7 @@ all its normal attributes, it just lacks a cross-node parent link.
|
||||
|
||||
## Prometheus Metrics (Spanmetrics)
|
||||
|
||||
The OTel Collector's spanmetrics connector automatically derives RED (Rate, Errors, Duration) metrics from every span. No custom metrics code is needed in xrpld.
|
||||
The OTel Collector's span_metrics connector automatically derives RED (Rate, Errors, Duration) metrics from every span. No custom metrics code is needed in xrpld.
|
||||
|
||||
### Generated Metric Names
|
||||
|
||||
@@ -2162,7 +2162,7 @@ collector settings make it work, both already enabled:
|
||||
|
||||
- `prometheus.resource_to_telemetry_conversion: enabled: true` promotes
|
||||
resource attributes to metric labels on the local scrape surface.
|
||||
- `spanmetrics.resource_metrics_key_attributes` lists the tier attributes so
|
||||
- `span_metrics.resource_metrics_key_attributes` lists the tier attributes so
|
||||
span-derived series stay grouped per node and tier.
|
||||
|
||||
Traces and logs carry resource attributes natively; Grafana Cloud ingests all
|
||||
|
||||
Reference in New Issue
Block a user