fix(telemetry): correct the phase-10 validation harness against the code

The harness manifests asserted things the code cannot produce and missed most
of what it does. Two assertions were failing every run, and the metric set
covered 16 of the ~41 emitted names.

expected_spans.json: rpc.process was required with rpc.ws_message as its
parent, but it is created only in ServerHandler::processRequest() on the HTTP
path, so a WebSocket-only workload never produces it -- it is now optional and
parented to rpc.http_request, and the rpc.process -> rpc.command.* edge is
skipped with the real reason instead of a coroutine-context-loss diagnosis that
was never the cause. Adds the missing rpc.ws_upgrade span, corrects four
parents (consensus.mode_change, pathfind.request, and update_positions/check,
which are children of consensus.establish rather than consensus.round), and
demotes conditionally-set attributes out of required_attributes so a healthy
run stops failing. Counts recomputed from the file: 41 span types, 62 unique
required attributes.

expected_metrics.json: 16 -> 52 asserted entries across the job-queue, RPC
method, reduce-relay, overflow and validation families, plus the fifteenth
dashboard uid. Metrics the harness workload cannot exercise -- erroring RPC,
ledger-mismatch, TxQ overflow, and the lazily-created getobject_* instruments
-- are listed in a not_asserted group the validator skips, rather than as
assertions that would fail on a healthy node.

The workflow's push trigger listed two globs matching nothing
(include/xrpl/basics/Telemetry*.h, src/xrpld/app/misc/Telemetry*), so no C++
telemetry change ever triggered validation. Replaced with the paths the code
actually lives in, including src/libxrpl/beast/insight/** for the insight
export path the harness depends on. The four inert workflow_dispatch inputs are
now labelled UNUSED rather than looking like working knobs.

Docs: the workload README described a StatsD dirty-flag mechanism under a
member name that does not exist, on a code path the harness never uses -- it
sets [insight] server=otel, so gauges export through an observable-gauge
callback every cycle. Adds the missing txq-burst phase, reconciles three
different dashboard counts, and drops "posts summary to PR", which the workflow
has no permission to do. The runbook's phase-10 section loses the last
sampling_ratio reference (not a config key), gains a Regression Gate and CI
subsection covering the gate that can fail CI, and its compose-logs command now
names the workload compose file. cmake --preset default is left for a separate
change: no CMakePresets.json is tracked, so it is wrong everywhere it appears.

Also drops the dead exporter=otlp_http key the harness wrote into every node
config, and stops capture_timings.py defaulting --profile to a profile that
does not exist.
This commit is contained in:
Pratik Mankawde
2026-08-14 12:34:33 +01:00
parent 0825ac8ecc
commit 22e440aee1
11 changed files with 535 additions and 140 deletions

View File

@@ -26,25 +26,32 @@ name: Telemetry Validation
on:
workflow_dispatch:
# NOTE: rpc_rate / rpc_duration / tx_tps / tx_duration have NO effect.
# They are forwarded to run-full-validation.sh, which parses them into
# shell variables and never reads them again — load shape comes entirely
# from --profile and docker/telemetry/workload/workload-profiles.json.
# They are kept (and labelled) rather than removed so existing dispatch
# bookmarks and any saved input sets do not break. To change the load,
# edit or add a profile in workload-profiles.json.
inputs:
rpc_rate:
description: "RPC load rate (requests per second)"
description: "UNUSED — has no effect. Load shape comes from the workload profile."
required: false
default: "50"
rpc_duration:
description: "RPC load duration (seconds)"
description: "UNUSED — has no effect. Load shape comes from the workload profile."
required: false
default: "120"
tx_tps:
description: "Transaction submit rate (TPS)"
description: "UNUSED — has no effect. Load shape comes from the workload profile."
required: false
default: "5"
tx_duration:
description: "Transaction submit duration (seconds)"
description: "UNUSED — has no effect. Load shape comes from the workload profile."
required: false
default: "120"
run_benchmark:
description: "Run performance benchmarks"
description: "Run performance benchmarks (the only input that changes behaviour)"
required: false
type: boolean
default: false
@@ -54,11 +61,19 @@ on:
- "pratik/otel-phase*"
- "feature/otel-*"
- "feature/telemetry-*"
# Keep these globs pointing at paths that actually exist. Two earlier
# entries (include/xrpl/basics/Telemetry*.h, src/xrpld/app/misc/Telemetry*)
# matched zero tracked files, so a pure C++ telemetry change never
# triggered this workflow on push — only edits under docker/telemetry/**
# or to this file did. The telemetry sources live in the three telemetry
# module directories below.
paths:
- ".github/workflows/telemetry-validation.yml"
- "docker/telemetry/**"
- "include/xrpl/basics/Telemetry*.h"
- "src/xrpld/app/misc/Telemetry*"
- "include/xrpl/telemetry/**"
- "src/libxrpl/telemetry/**"
- "src/libxrpl/beast/insight/**"
- "src/xrpld/telemetry/**"
concurrency:
group: telemetry-validation-${{ github.ref }}
@@ -206,6 +221,11 @@ jobs:
TX_DURATION: ${{ github.event.inputs.tx_duration || '120' }}
RUN_BENCHMARK: ${{ github.event.inputs.run_benchmark }}
run: |
# The four rate/duration flags below are inert (see the
# workflow_dispatch inputs note): run-full-validation.sh parses them
# and never reads them. Load shape comes from the default
# --profile full-validation. They are still passed so the flags stay
# exercised if they are ever wired up.
ARGS="--xrpld ${{ env.BUILD_DIR }}/xrpld --skip-loki"
ARGS="$ARGS --rpc-rate $RPC_RATE"
ARGS="$ARGS --rpc-duration $RPC_DURATION"