mirror of
https://github.com/XRPLF/rippled.git
synced 2026-08-21 14:20:56 +00:00
d059f21bf3
Fixes the review findings on this PR that belong to files it owns, plus several defects found while verifying those fixes. Findings in files owned by upstream branches are routed there and left untouched here. Correctness: - tx_submitter: advance the account sequence only on results that actually consume one (tes*, tec*, terQUEUED). tem*/tef*/tel* never reach the ledger, so advancing left a permanent gap that every later submit from that account inherited. Add a re-fetch hatch so a repeated non-consuming failure cannot livelock on the same sequence, and gate the account check on funded-ness rather than list length. - validate_telemetry: filter spans by name before collecting attributes, so a per-span attribute contract can no longer be satisfied by a sibling span; require exact name equality for non-wildcard children and glob matching for wildcards; bounds-check every returned series instead of only the first. - collect_system_metrics: select xrpld by argv[0] rather than a substring match on the whole command line, which averaged in unrelated processes and reported their RSS as xrpld's. Count genuine 0.0 CPU readings, use a clamped nearest-rank p99 index, and record RPC latency only on success. - benchmark: return each verdict through a named variable instead of a command substitution, so the pass/fail counters survive and the exit gate can fire. Scale before dividing in the percentage math, which truncated a 1.26% impact to 1.00% and cleared a 1% threshold. - compare_to_baseline: fall back to the absolute bound when the baseline is not positive, so a 0 -> 500 ms jump is no longer "within bounds". - rpc_load_generator: bound each connection to one in-flight recv(), drain in-flight requests before closing, use a nearest-rank percentile, and report delivery shortfall so an under-delivered run cannot pass with a 0% error rate. Fail loudly instead of silently: - run-full-validation: treat a consensus timeout and a missing validated ledger as fatal infrastructure errors, and fold the orchestrator and benchmark exit codes into the final status. A degraded cluster previously ran a full validation pass and reported misleading downstream failures. - collect_system_metrics: warn per empty measurement source, emit metrics_complete, and exit non-zero instead of substituting zeros that pass every threshold. Require GNU date with %N rather than falling back to a per-sample python3 fork that costs more than the threshold it is measured against. - benchmark: distinguish "could not measure" from "exceeded thresholds", install a cleanup trap so a failure cannot leak nodes and ports, and report an unusable baseline as inconclusive. - workload_orchestrator: bound subprocess communicate() and fail the exit gate on per-phase errors. Also pins the workload compose images to the versions the sibling stack already uses, hash-pins the Python dependencies, restricts the validator config template to loopback, corrects the dashboard and metric counts in the reference docs, drops a span from the regression gate that cannot fire under a WebSocket-only workload, and narrows the teardown pkill pattern so it no longer matches processes that merely mention the work directory. Verified with a full harness run against a local five-node cluster: 158 of 158 checks passed with no regressions detected.
112 lines
3.6 KiB
YAML
112 lines
3.6 KiB
YAML
# Docker Compose workload harness for telemetry validation.
|
|
#
|
|
# Runs the OTel telemetry backend only. There are no validator services here:
|
|
# - OTel Collector (traces + native OTLP metrics)
|
|
# - Tempo (trace backend + search API)
|
|
# - Prometheus (metrics)
|
|
# - Loki (log aggregation for log-trace correlation)
|
|
# - Grafana (dashboards + trace/log exploration)
|
|
#
|
|
# The validator cluster runs as host processes, not containers.
|
|
# run-full-validation.sh starts NUM_NODES (default 5) xrpld instances on
|
|
# 127.0.0.1, each with a cfg it generates inline, peered to each other via
|
|
# [ips_fixed]. They reach the collector through the published ports below and
|
|
# write their logs into the bind-mounted workdir for the filelog receiver.
|
|
#
|
|
# Usage:
|
|
# # Start the telemetry backend on its own:
|
|
# docker compose -f docker/telemetry/docker-compose.workload.yaml up -d
|
|
#
|
|
# # Or let the orchestrator start this stack and the node cluster together:
|
|
# docker/telemetry/workload/run-full-validation.sh
|
|
#
|
|
# Prerequisites (for the orchestrator, not for this stack):
|
|
# - xrpld binary built with -DXRPL_ENABLE_TELEMETRY=ON
|
|
# - Validator keys generated via generate-validator-keys.sh
|
|
#
|
|
# Image tags are pinned to the same versions as docker-compose.yml, which
|
|
# mounts these same collector, Tempo and Prometheus config files. Floating
|
|
# tags would let an upstream release change the harness result.
|
|
#
|
|
# Note: No Docker healthchecks are defined here. The orchestrator script
|
|
# (run-full-validation.sh) polls each service endpoint directly from the
|
|
# host, which avoids issues with missing curl/wget in container images.
|
|
|
|
services:
|
|
# ---------------------------------------------------------------------------
|
|
# Telemetry Backend Stack
|
|
# ---------------------------------------------------------------------------
|
|
|
|
otel-collector:
|
|
image: otel/opentelemetry-collector-contrib:0.158.0
|
|
command: ["--config=/etc/otel-collector-config.yaml"]
|
|
ports:
|
|
- "4317:4317" # OTLP gRPC
|
|
- "4318:4318" # OTLP HTTP (traces + beast::insight metrics)
|
|
- "8889:8889" # Prometheus metrics endpoint
|
|
- "13133:13133" # Health check
|
|
volumes:
|
|
- ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:ro
|
|
# Mount the validation workdir so the filelog receiver can tail node
|
|
# logs. run-full-validation.sh sets XRPLD_LOG_DIR to its workdir; the
|
|
# default matches that workdir so a bare `docker compose up` also works.
|
|
- ${XRPLD_LOG_DIR:-/tmp/xrpld-validation}:/var/log/xrpld:ro
|
|
depends_on:
|
|
- tempo
|
|
networks:
|
|
- workload-net
|
|
|
|
tempo:
|
|
image: grafana/tempo:2.9.4
|
|
command: ["-config.file=/etc/tempo.yaml"]
|
|
ports:
|
|
- "3200:3200" # Tempo HTTP API
|
|
volumes:
|
|
- ./tempo.yaml:/etc/tempo.yaml:ro
|
|
- tempo-data:/var/tempo
|
|
networks:
|
|
- workload-net
|
|
|
|
prometheus:
|
|
image: prom/prometheus:v3.13.2
|
|
ports:
|
|
- "9090:9090"
|
|
volumes:
|
|
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
|
depends_on:
|
|
- otel-collector
|
|
networks:
|
|
- workload-net
|
|
|
|
loki:
|
|
image: grafana/loki:3.7.6
|
|
ports:
|
|
- "3100:3100" # Loki HTTP API
|
|
command: ["-config.file=/etc/loki/local-config.yaml"]
|
|
networks:
|
|
- workload-net
|
|
|
|
grafana:
|
|
image: grafana/grafana:13.1.2
|
|
environment:
|
|
- GF_AUTH_ANONYMOUS_ENABLED=true
|
|
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
|
|
ports:
|
|
- "3000:3000"
|
|
volumes:
|
|
- ./grafana/provisioning:/etc/grafana/provisioning:ro
|
|
- ./grafana/dashboards:/var/lib/grafana/dashboards:ro
|
|
depends_on:
|
|
- tempo
|
|
- prometheus
|
|
- loki
|
|
networks:
|
|
- workload-net
|
|
|
|
volumes:
|
|
tempo-data:
|
|
|
|
networks:
|
|
workload-net:
|
|
driver: bridge
|