ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-09-27 15:28:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(telemetry): name the harness log directories after their instance ids

Browse Source 
The collector reads the per-node directory off the log file path and stamps it
as the Loki label service_instance_id, so the directory name has to equal the
node's own [telemetry] service_instance_id or log lines carry a node name that
no trace or metric shares and nothing joins.

Both harness scripts disagreed with themselves: run-full-validation.sh wrote to
node$i while setting validator-${i}, and benchmark.sh wrote to node$i while
setting bench-node-${i}. Rename the directories to match the ids rather than
the reverse, so no existing trace or metric label value moves and no harness
expectation has to be re-checked. Only path references are renamed; the
human-readable "node$i" in log and error messages is left as prose.

The config template is not rendered by any script, so its DATA_DIR
documentation gains a note about the same constraint instead.

Also rename the deprecated otlphttp/filelog collector component names in the
harness scripts and docs.
This commit is contained in:
Pratik Mankawde
2026-09-09 15:12:36 +01:00
parent 478b3e4b07
commit ac07e1345f
7 changed files with 41 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docker/telemetry/docker-compose.workload.yaml
View File

@@ -11,7 +11,7 @@
# run-full-validation.sh starts NUM_NODES (default 5) xrpld instances on
# 127.0.0.1, each with a cfg it generates inline, peered to each other via
# [ips_fixed]. They reach the collector through the published ports below and
# write their logs into the bind-mounted workdir for the filelog receiver.
# write their logs into the bind-mounted workdir for the file_log receiver.
#
# Usage:
# # Start the telemetry backend on its own:
@@ -47,7 +47,7 @@ services:
- "13133:13133" # Health check
volumes:
- ./otel-collector-config.yaml:/etc/otel-collector-config.yaml:ro
# Mount the validation workdir so the filelog receiver can tail node
# Mount the validation workdir so the file_log receiver can tail node
# logs. run-full-validation.sh sets XRPLD_LOG_DIR to its workdir; the
# default matches that workdir so a bare `docker compose up` also works.
- ${XRPLD_LOG_DIR:-/tmp/xrpld-validation}:/var/log/xrpld:ro

8
docker/telemetry/workload/README.md
View File

@@ -32,7 +32,7 @@ run-full-validation.sh (shell orchestrator)
|
|-- docker-compose.workload.yaml
| |-- otel-collector (otlp receiver: traces + beast::insight metrics;
| | filelog receiver: node debug.log -> Loki)
| | file_log receiver: node debug.log -> Loki)
| |-- tempo (trace backend + TraceQL search API)
| |-- prometheus (metrics scraping)
| |-- loki (log aggregation for log-trace correlation)
@@ -458,7 +458,7 @@ its own `check_log_correlation()`, but no workflow runs that script.
Correlation depends on four independent legs, and a failed check on its own names
none of them: the node must write a `debug.log` line carrying trace ids, the
collector container must see that file, its `filelog` receiver must parse and
collector container must see that file, its `file_log` receiver must parse and
export the line, and Loki must return it for the validator's LogQL.
`run-full-validation.sh` prints a per-leg diagnostic after the suite whenever the
Loki checks are enabled — per-node correlated-line counts and severity mix, the
@@ -467,7 +467,7 @@ internal log-record counters, and Loki's own entry counts for the selector with
and without the line filter. Read that block first; it identifies the broken leg
without reproducing anything.
Those two entry counts **must** be wrapped in `sum()`. The `filelog` receiver's
Those two entry counts **must** be wrapped in `sum()`. The `file_log` receiver's
`regex_parser` leaves `message` and `timestamp` as log-record attributes, and
Loki's OTLP path stores them as structured metadata that joins the label set of a
metric query — so an unaggregated `count_over_time` returns one series per log
@@ -510,7 +510,7 @@ docker/telemetry/workload/run-full-validation.sh --xrpld .build/xrpld
```
Re-run it after any change to log formatting, span activation, the collector's
`filelog` receiver, or the Loki exporter.
`file_log` receiver, or the Loki exporter.
### Pathfinding is not exercised

6
docker/telemetry/workload/benchmark.sh
View File

@@ -189,7 +189,7 @@ start_cluster() {
# Build per-node configs.
for i in $(seq 1 "$NUM_NODES"); do
local node_dir="$WORKDIR/node$i"
local node_dir="$WORKDIR/bench-node-$i"
mkdir -p "$node_dir/nudb" "$node_dir/db" ||
cannot_measure "Could not create node$i directories under $node_dir"
@@ -361,7 +361,7 @@ stop_cluster() {
log "Stopping cluster..."
for i in $(seq 1 "$NUM_NODES"); do
local pidfile="$WORKDIR/node$i/xrpld.pid"
local pidfile="$WORKDIR/bench-node-$i/xrpld.pid"
if [ -f "$pidfile" ]; then
kill "$(cat "$pidfile")" 2>/dev/null || true
fi
@@ -422,7 +422,7 @@ ws_endpoints() {
node_pids_csv() {
local i out="" pid
for i in $(seq 1 "$NUM_NODES"); do
pid=$(cat "$WORKDIR/node$i/xrpld.pid" 2>/dev/null) || continue
pid=$(cat "$WORKDIR/bench-node-$i/xrpld.pid" 2>/dev/null) || continue
[ -n "$pid" ] && out="$out,$pid"
done
printf '%s' "${out#,}"

28
docker/telemetry/workload/run-full-validation.sh
View File

@@ -266,7 +266,7 @@ mkdir -p "$WORKDIR" "$REPORT_DIR" || die "Could not create $WORKDIR and $REPORT_
# Step 1: Start observability stack
# ---------------------------------------------------------------------------
log "Step 1: Starting observability stack..."
# Point the collector's log mount at this run's workdir so the filelog
# Point the collector's log mount at this run's workdir so the file_log
# receiver tails the per-node debug.log files generated below.
XRPLD_LOG_DIR="$WORKDIR" docker compose -f "$COMPOSE_FILE" up -d ||
die "docker compose up failed for $COMPOSE_FILE — the observability stack did not start"
@@ -311,7 +311,7 @@ bash "$SCRIPT_DIR/generate-validator-keys.sh" "$XRPLD" "$NUM_NODES" "$WORKDIR" |
die "generate-validator-keys.sh failed — no validator keys for the $NUM_NODES-node cluster"
for i in $(seq 1 "$NUM_NODES"); do
NODE_DIR="$WORKDIR/node$i"
NODE_DIR="$WORKDIR/validator-$i"
mkdir -p "$NODE_DIR/nudb" "$NODE_DIR/db" || die "Could not create node$i directories under $NODE_DIR"
RPC_PORT=$((RPC_PORT_BASE + i - 1))
@@ -478,15 +478,15 @@ node_running() {
report_stopped_nodes() {
local i pid status
for i in $(seq 1 "$NUM_NODES"); do
pid=$(cat "$WORKDIR/node$i/xrpld.pid" 2>/dev/null || echo "")
pid=$(cat "$WORKDIR/validator-$i/xrpld.pid" 2>/dev/null || echo "")
[ -n "$pid" ] || continue
node_running "$pid" && continue
status=0
wait "$pid" 2>/dev/null || status=$?
warn "node$i (pid $pid) is not running — wait status $status"
if [ -s "$WORKDIR/node$i/stdout.log" ]; then
if [ -s "$WORKDIR/validator-$i/stdout.log" ]; then
warn "node$i last output:"
tail -n 15 "$WORKDIR/node$i/stdout.log" | sed 's/^/ /' >&2
tail -n 15 "$WORKDIR/validator-$i/stdout.log" | sed 's/^/ /' >&2
else
warn "node$i wrote no stdout at all"
fi
@@ -606,7 +606,7 @@ fi
# ---------------------------------------------------------------------------
# Log-trace correlation has four legs and a failed check names none of them:
# the node must write a debug.log line carrying trace ids, the collector
# container must see that file, its filelog receiver must parse and export the
# container must see that file, its file_log receiver must parse and export the
# line, and Loki must return it for the validator's own LogQL. Each leg below
# reports what it observed, so a reader with only the CI log can tell which one
# broke instead of guessing.
@@ -704,7 +704,7 @@ diag_node_logs() {
local i log bytes total correlated sample
echo " [leg 1/4 node] debug.log lines matching '$DIAG_TRACE_RE'"
for i in $(seq 1 "$NUM_NODES"); do
log="$WORKDIR/node$i/debug.log"
log="$WORKDIR/validator-$i/debug.log"
if [ ! -f "$log" ]; then
echo " node$i: no debug.log at $log — the node never opened its log sink"
continue
@@ -786,10 +786,10 @@ diag_collector_mount() {
sed 's/^/ /' || echo " (container-side listing failed)"
}
# Leg 3 — collector: did the filelog receiver parse and export those lines?
# Leg 3 — collector: did the file_log receiver parse and export those lines?
#
# Two independent readings. The collector's own stderr names every file the
# receiver opened and carries any filelog parse or Loki export error. Its
# receiver opened and carries any file_log parse or Loki export error. Its
# internal telemetry counts log records in and out: accepted>0 with sent=0 is
# an export failure, accepted=0 while files are being watched is a parse
# failure.
@@ -801,7 +801,7 @@ diag_collector_mount() {
# exists; when it reports nothing matching, the leg says so.
diag_collector_pipeline() {
local cid img watched problems metrics
echo " [leg 3/4 collector] filelog receiver state"
echo " [leg 3/4 collector] file_log receiver state"
if ! command -v docker >/dev/null 2>&1; then
echo " docker is not on PATH — leg skipped"
return 0
@@ -822,12 +822,12 @@ diag_collector_pipeline() {
# Second filter keys on the collector's own logs-pipeline markers so this
# does not report warnings from the trace or metric pipelines. Nothing is
# excluded beyond that: the collector's benign config-alias deprecation
# notices ("filelog" -> "file_log") do surface here, and suppressing lines
# notices ("file_log" -> "file_log") do surface here, and suppressing lines
# because they are usually harmless is how a diagnostic hides the one that
# was not.
problems=$(diag_run docker logs "$cid" 2>&1 |
grep -iE '(warn|error)' |
grep -iE 'filelog|fileconsumer|loki|signal": *"logs' |
grep -iE 'file_log|fileconsumer|loki|signal": *"logs' |
tail -n 20 || true)
if [ -n "$problems" ]; then
echo " logs-pipeline warnings and errors (last 20):"
@@ -869,7 +869,7 @@ diag_loki_stream() {
[ -n "$selector" ] || selector="$DIAG_LOG_SELECTOR"
[ -n "$correlation" ] || correlation="$DIAG_LOG_SELECTOR $DIAG_LOG_FILTER"
# sum() is required, for the reason recorded at _log_loki_diagnostics in
# validate_telemetry.py: the filelog regex_parser leaves message/timestamp
# validate_telemetry.py: the file_log regex_parser leaves message/timestamp
# as log-record attributes, Loki's OTLP path turns those into structured
# metadata that joins a metric query's label set, so an unaggregated
# count_over_time yields one series per log line and Loki rejects the query
@@ -1076,7 +1076,7 @@ echo " xrpld nodes ($NUM_NODES) are running:"
for i in $(seq 1 "$NUM_NODES"); do
rpc=$((RPC_PORT_BASE + i - 1))
ws=$((WS_PORT_BASE + i - 1))
pid=$(cat "$WORKDIR/node$i/xrpld.pid" 2>/dev/null || echo 'unknown')
pid=$(cat "$WORKDIR/validator-$i/xrpld.pid" 2>/dev/null || echo 'unknown')
echo " Node $i: RPC=$rpc WS=$ws PID=$pid"
done
echo ""

2
docker/telemetry/workload/validate_telemetry.py
View File

@@ -1680,7 +1680,7 @@ async def _log_loki_diagnostics(session: aiohttp.ClientSession, loki_url: str) -
"Loki diagnostic: service_name values: %s", ", ".join(found) or "(none)"
)
# sum() is load-bearing, not cosmetic. The filelog receiver's regex_parser
# sum() is load-bearing, not cosmetic. The file_log receiver's regex_parser
# leaves message, timestamp, trace_id and span_id as log-record attributes,
# and Loki's OTLP path stores those as structured metadata, which joins the
# label set of a metric query. Because `message` and `timestamp` are unique

7
docker/telemetry/workload/xrpld-validator.cfg.template
View File

@@ -14,7 +14,12 @@
# {{RPC_PORT}} — HTTP RPC port
# {{WS_PORT}} — WebSocket port
# {{PEER_PORT}} — Peer protocol port
# {{DATA_DIR}} — Node data directory
# {{DATA_DIR}} — Node data directory. Its last path segment must
# equal service_instance_id below: the collector's
# file_log receiver reads that segment off the log
# file path and stamps it as the Loki label
# service_instance_id, so a mismatch gives log lines
# a node name no trace or metric shares.
# {{VALIDATION_SEED}} — Validator seed from key generation
# {{VALIDATORS_FILE}} — Path to shared validators.txt
# {{IPS_FIXED}} — Peer addresses (one per line)