mirror of
https://github.com/XRPLF/rippled.git
synced 2026-09-27 07:26:51 +00:00
efb4cc2edd94fb3976efeefbc405ae7ed498d228
20 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Pratik Mankawde
|
85fc1c5399 |
fix(telemetry): keep the workload cluster on [ips], not [ips_fixed]
The previous commit switched the generated node config from [ips] to [ips_fixed] on the grounds that the variable, the comment and the sibling cfg template all named ips_fixed, and that ips_fixed is the section whose documented meaning fits a private cluster. Both of those are still true. The switch is reverted anyway, because it is a workload change rather than a naming fix. Measured on CI, parent commit against this branch's previous tip, one functional config line apart: span.consensus.ledger_close.p95 0.57 ms -> 6.43 ms (tripped the gate) span.consensus.ledger_close.p99 0.94 ms -> 9.50 ms span.consensus.accept.p50 0.97 ms -> 2.63 ms span.tx.process.p50 0.36 ms -> 0.18 ms (faster) job.acceptLedger.running.p95 21157 us -> 10938 us (faster) Every consensus-path span rose and every transaction-path metric fell, which is the shape a denser always-connected mesh produces and not the shape of run-to-run variance. [ips_fixed] holds connections open to all four peers instead of treating the list as a discovery hint, so each node processes proposals and validations from the full mesh every round. Nothing else in that commit touches the consensus path: the emitted config differed in exactly three lines, of which one is a die message and one expands to an identical string. The committed baseline describes the [ips] topology. Adopting [ips_fixed] therefore needs a refreshed baseline and re-derived bounds, which is the process baselines/README.md already documents for a workload change. Left as its own work item rather than smuggled in behind a section rename, and the reason is now recorded beside the line so it is not repeated. This also falsified a claim the previous commit had written into baselines/README.md and regression-thresholds.json: that none of the six weakly-guarded keys fires on any observed run. Corrected in both, and the measurement above is cited in place of the absolute. |
||
|
Pratik Mankawde
|
ffc4ee40fe |
fix(telemetry): correct node directory naming and stale gates in the workload harness
The harness killed and probed node directories named `node<N>`, but the
directories it creates are `validator-<N>` in run-full-validation.sh and
`bench-node-<N>` in benchmark.sh. Verified with pgrep against processes whose
command lines mimic the real ones: the pattern matched nothing either script
produces. Three consequences, all live:
- `--cleanup` deleted the workdir and left the xrpld processes running. They
are host processes, so the compose teardown does not reach them.
- The pre-run cleanup could not free the previous run's RPC, WS and peer
ports, which surfaces much later as a cluster that never reaches consensus.
- The startup crash fast-fail read a pid path that never exists, so its
`stopped > 0` branch was unreachable and a dead node waited out the full
120-attempt window.
Rather than patch four literals, derive every node path, kill pattern and log
glob from one NODE_PREFIX per script. The directory name is also the node's
identity: the collector's file_log receiver lifts that segment into
service.instance.id, so the directory and the [telemetry] service_instance_id
must agree. Deriving both from one value is what stops them drifting again.
Also in the same files, each confirmed by test rather than inspection:
- The collector readiness probe could never fail. curl -w '%{http_code}'
prints 000 on a refused connection and then exits non-zero, so the
`|| echo 000` inside the substitution appended a second 000 and the
"not ready" comparison never matched. Move the fallback outside.
- The generated config wrote [ips], the starter-list section. A loopback mesh
that must reach quorum is the [ips_fixed] case, which is what the variable,
the comment and the sibling cfg template already said.
- benchmark.sh returned exit 1 for a row it could not measure, though the
exit-code table reserves 1 for "every metric was measured and one breached".
Report 2 there instead.
- Five bc computations fell back to 0, which clears every threshold. The
guards beside them already fall back to the inconclusive token; these now
do too.
- A comment claimed a `|| guard` after a heredoc lands in the heredoc, and
that claim had removed a real guard from the config write. It does not: the
guard runs, and fires when cat fails.
- The EXIT trap was installed 88 lines before stop_workload was defined. If it
fired in that window, errexit aborted the handler on "command not found" and
the cluster reap never ran. Install it below both handlers.
- jq exits 5 on malformed JSON, outside this script's documented codes, so
read_metric now routes that through cannot_measure.
- --nodes and --duration were unvalidated, and --nodes 0 made the pid-count
guard compare 0 with 0 and pass, handing the sampler no pids at all.
- --cleanup now passes -v so the named tempo-data volume goes with it.
Otherwise the next run's Tempo still serves the previous run's traces and a
span assertion can be satisfied by them.
- Five messages reported an attempt count as seconds, though each attempt is
a sleep plus every node's probe.
The baselines README and the two regression JSON files had gone stale when the
baseline was refreshed to a three-run median: they described 20 gated keys and
five exclusions, against an actual 19 and six, and cited the superseded run,
date and commit. Re-derive every affected figure from the committed files. The
detection floors are recomputed (2.00x to 7.41x, so a 10x regression is now
caught on all 19 keys), the newly excluded span.ledger.build.p99 is documented,
and figures that no committed artifact can verify are either replaced with
derivable ones or labelled with their numerator.
No baseline value, threshold bound or derivation entry changes.
|
||
|
Pratik Mankawde
|
00c0265cc6 |
test(telemetry): refresh the timing baseline from three clean runs
The committed baseline was captured 2026-08-26, before the account-funding race was detectable. Phases whose funding silently failed submitted no transactions, so the capture recorded artificially low ledger and transaction timings, and job.transaction.queued.p95 and job.transaction.running.p95 could not be captured at all. Once funding worked, span.ledger.build.p99 read 29.00 ms against a 9.11 ms baseline and turned the gate red on a run whose 200 span and metric checks all passed. Refresh every value to the median of CI runs 34495527952, 34505215266 and 34507425933, the first three with the fix in place, and re-derive each absolute bound as hi_next - baseline from that median. Exclude span.ledger.build.p99. Across those three runs it read 29.00, 7.06 and 8.94 ms, a 4.11x spread whose maximum is 1.16x its 25 ms trip point, so a healthy run reddens CI. Widening cannot fix it: a bound tolerating 29.00 ms would reach into the bucket above and restore the single-crossing false positive the derivation rule removes. span.ledger.build.p95 stays gated at 0.48 of its trip point, so ledger construction keeps coverage. The other 19 keys sit between 0.17 and 0.76 of their trip points. span.tx.process.p95 is the tightest and is the first to re-measure if the gate reddens again. Repoint one bounds-checker test at span.ledger.build.p95, since it mutated the p99 override this commit removes. |
||
|
Pratik Mankawde
|
478b3e4b07 |
docs(telemetry): correct stale claims and citations in the harness docs
The workload README contradicted itself on --skip-loki: one bullet said CI always passes it and so the two log-correlation checks are never exercised, another said the workflow no longer passes it. The workflow mentions the flag nowhere, so the first was the stale half. Other claims checked against the tree and corrected: - both the README and the plan doc described the push trigger as filtered on branch names. The workflow has no branches filter, deliberately, because GitHub ANDs branches with paths - the plan doc printed 6 of the workflow's 12 paths globs, and claimed the workflow was 367 lines against an actual 451. The glob block is now generated from the workflow, and the line count dropped rather than restated - rpcNOT_SUPPORTED does not exist anywhere in the tree. The symbol is RpcNotSupported, and the refusal sites are RipplePathFind.cpp:59-60 and PathFind.cpp:50-51, not :48-49 and :39 - RCLConsensus.cpp:666 and :663 are not log or event lines; the tx.included event is at :720 and the per-transaction debug log at :715 - LedgerMaster.cpp:463 is fixIndex, not the ledger.store span, which is at :470 - ServerHandler.cpp:705 is inside makeJsonError; processRequest is at :718 - file counts: docker/telemetry/workload/ is 25 files, include/xrpl/telemetry/ 13 - the optional-span bullet named five causes covering 10 of 16 entries, omitting the txq.* family and the WebSocket handshake - the /api/v1/series choice was attributed to stale StatsD gauges; this harness runs no StatsD A line number in run-full-validation.sh was cited in five places and drifts on every edit to that file, so those now name the file only. The keygen helper's header records what production does instead -- validator-keys-tool create_keys then create_token, keeping the master key off the node -- and why a disposable cluster does not. |
||
|
Pratik Mankawde
|
3a63a17548 |
docs(telemetry): stop the harness contract narrating its own revisions
Notes across the workload contract described earlier versions of themselves, or cited commits that only exist inside this chain. A squash merge publishes none of it, so each reference resolves nowhere. Notes that described their own earlier text: - expected_spans.json: 'this note previously concluded', 'this note previously said', 'Un-skipped 2026-08-26', 'the reason had simply gone stale for two weeks' and 'the claim this entry carried' are replaced by the standing reason each entry holds. The wildcard pairs now say the validator globs the child via _span_name_matches(), and state the literal-collapse failure as what a different validator WOULD do rather than as history. - regression-thresholds.json: 'an earlier version of this note wrongly claimed', 'the earlier version oversold it' and 'an earlier note called that' become the cautions themselves -- do not reason from 'every ladder step is at least 2x', do not oversell the backstop, do not read a false fire as a missing override. - test_check_regression_bounds.py: the docstring gives the reason a literal is wrong here, not the story of two tests that once hard-coded one. Baseline-refresh history rewritten as measurement: - README.md, baselines/README.md, telemetry-runbook.md and regression-metrics.json no longer attribute threshold moves to 'the 2026-08-26 refresh'. The evidence is kept as measurement -- span.tx.apply.p50 has read 0.7917 ms and 0.00597 ms on the same workload, 132x apart; job.acceptLedger.running.p95 has measured a 5.74x floor on one baseline and 16.28x on another -- which is what supports the claim that a single-run baseline cannot bound these keys. Two chain-only commit ids removed, |
||
|
Pratik Mankawde
|
8521b96d85 |
fix(telemetry): stop an incomplete capture becoming the committed baseline
The regression baseline is bootstrapped by copying a CI artifact. The workflow
tested only that timings.json existed, then printed it verbatim under a heading
inviting the reader to paste it in as the new baseline.
capture_timings.py writes that file and only then enforces --min-capture-ratio,
so an incomplete capture leaves a file that exists but covers fewer keys than
the contract declares. The verdict lived in CAPTURE_EXIT, a shell variable local
to run-full-validation.sh that no other program could read. So on a placeholder
baseline plus a thin capture, CI offered an incomplete artifact as the next
baseline, and pasting it narrowed the gate with nothing reporting that it had.
That is the failure shape this harness keeps producing: a degraded result that
looks exactly like a good one.
The artifact now carries its own completeness, next to metrics:
"capture": { "declared": 20, "captured": 20, "min_ratio": 0.5, "complete": true }
complete is the same condition the producer exits 0 on, computed once with the
exit code read off it, so the flag and the status cannot drift apart. Any
consumer can now tell a complete capture from a thin one, not just CI.
Both paste-me paths refuse rather than warn: the workflow prints the counts and
an error annotation with no JSON, and the comparator explains on stderr while
leaving stdout empty, so a redirect cannot produce a plausible-looking file. A
warning above a copyable block is still a copyable block, and a reader who has
just hit a red gate is already predisposed to re-baseline. A missing capture
block fails closed.
Refusal is scoped to bootstrapping a baseline, not to comparing against one, so
artifacts captured before this change still replay: verified against the run the
current baseline came from, which carries no capture block and still reports 0
regressions. An injected regression is still caught, and the gated surface is
unchanged at 20 keys with 5 excluded.
|
||
|
Pratik Mankawde
|
a734da8b33 |
test(telemetry): recapture the baseline and stop gating what variance dominates
Refreshes baselines/baseline-timings.json from run 32964262700 at |
||
|
Pratik Mankawde
|
3836078a78 |
fix(telemetry): stop gating ledger.validate p95 and p99, which vary too much
The regression gate has been red on runs with no code change. Only two of the 25 gated keys ever tripped, both on the same span and never together: run 32862589645 failed p99 at 25.8750 ms against a 1.0600 ms baseline (+2341%), run 32867433073 failed p95 at 0.7500 ms against 0.2404 ms (+212%), and in each run the other quantile sat well inside its own bound. A real slowdown would move both. This is variance, not a defect. Measured across four CI runs: span.ledger.validate.p50 0.0484 to 0.0778 ms 1.6x spread kept span.ledger.validate.p95 0.1281 to 0.7500 ms 5.9x spread excluded span.ledger.validate.p99 0.3875 to 25.8750 ms 66.8x spread excluded Both excluded quantiles reach past their trip point on a healthy run. The mechanism is arrival timing, not slow code: the span opens only once a quorum-completing validation arrives (LedgerMaster.cpp:987, inside checkAccept, past the early return) and wraps the promotion work that follows, so one slow consensus round dominates the tail of a 3m rate window and which round that is differs every run. Widening is not available and must not be attempted later: tolerating 25.8750 ms against a 1.0600 ms baseline needs a bound of about 24.8 ms, which gates nothing. A bound admitting every healthy run's worst case admits every regression too. p50 stays gated; it is stable. THE GENERAL RULE, recorded so this does not recur: an absolute bound derived as hi_next minus baseline comes from the histogram ladder, so it budgets for quantization noise and for nothing else. It knows nothing about how far a metric moves between runs on identical code. Before gating any key, check its observed maximum across several runs against its trip point and gate it only with margin. Spread alone proves nothing: tx.apply.p50 swings 364x and never fires, because its 5 ms trip point absorbs the range. Of the 23 keys still gated the worst reaches 0.67 of its trip point. Mechanism: spans.names lists span names while _quantiles is shared, so dropping two quantiles of one span cannot be expressed by deleting a name. regression-metrics.json gains an excluded_keys map from a flat key to the reason it is not gated, subtracted by both prom_queries.py (so the key is never queried) and check_regression_bounds.py rule A. A per-name quantile override was rejected: a typo there leaves the key gating, whereas a typo in an exclusion subtracts nothing and new rule F rejects it, along with an empty reason, a leftover threshold override and a leftover baseline value. Derived figures recomputed from the committed baseline: 25 gated keys to 23, detection floor 2.02x-9.43x to 2.02x-9.42x, weakly guarded keys ten to nine, bound over baseline 102%-843% to 102%-842%. The baseline edit is a deletion of two entries only, with no value rewritten. Verified: both previously failing runs replay to zero regressions and exit 0; a tenfold increase injected into each of the 23 remaining keys in turn is still caught in all 23 cases; rule F was confirmed load-bearing by stubbing it out, which lets a stale exclusion pass. |
||
|
Pratik Mankawde
|
e4926f55be |
fix(telemetry): derive workload gate bounds from the bucket above the baseline
The gate could not catch a regression on any sub-millisecond span. compare_to_baseline.py requires both the percentage and the absolute bound to breach, and every span shared one flat absolute bound of 10 ms (15 ms for p99) calibrated for a 5-25 ms band the spans do not occupy. Against the baseline captured on 2026-08-24, where 18 of the 28 quantiles gated at the time sat below 1 ms, that bound sat 1.15x to 2000x above the metric it guarded, so the AND never fired: a 100x regression injected into span.ledger.store.p95 reported 0 regressions and exit 0. Injecting a 10x regression into each key in turn was caught on only 5 of 28. Give every gated key its own absolute bound, equal to the distance from its baseline to hi_next, the edge above the top of the bucket the baseline sits in. The trip point is then exactly hi_next, so the gate fires only once the reading clears the bucket above the baseline's own. That is the property a multiple of the enclosing bucket width cannot provide: after the quantile crosses hi, the interpolation happens across the next bucket, which on this ladder is up to eight times wider, so no multiple of the enclosing width bounds the excursion. Measured with a model-free reachability test, a single bucket crossing can produce a false regression on 2 of 25 keys under the old flat bound and 0 of 25 under this rule. The smallest catchable regression is 2.02x to 9.43x per key. The job queue bound had the same shape of problem on three of its four keys (42x, 47x, 220x before). Defaults now sit at each ladder floor, leaving the percentage bound operative for a metric that somehow reaches them. Drop span.ledger.store from the gated surface. Its captured quantiles were 0.005, 0.0095 and 0.0099 ms, which is the ladder's 0.01 ms floor times the quantile: every sample lands under 10 us, so the reported value does not move even if each store slows from 2 us to 9 us. No bound can gate it. Presence is still asserted by expected_spans.json and the integration test, and the rate is still on the ledger-operations dashboard. Add check_regression_bounds.py, wired into the same workflow step as the bucket parity check. It fails when a bound is not the one its own baseline implies, when a gated key has no override, when the baseline and metric surface disagree, when the percentage bound would become operative, and when a baseline carries the ladder floor signature. This gate has now broken three times through the same drift between ladder, baseline and bounds, so documentation alone is not enough. compare_to_baseline.py is unchanged: its existing per-metric override mechanism already expresses all of this. A missing, unreadable or malformed input makes that check exit 1 naming the input, rather than reporting success without having checked anything; only a placeholder baseline, the documented bootstrap state, still exits 0. Its own tests cover both halves of that contract plus one case per rule, and run in the workflow before the check so a broken rule reads as a broken rule. |
||
|
Pratik Mankawde
|
c4b8df9de1 |
test(telemetry): recapture span baselines on the current ladder
Copied verbatim from the timings.json produced by the telemetry-validation
run at
|
||
|
Pratik Mankawde
|
98ba282854 |
fix(telemetry): make the integration test correlate by construction, record the baseline log-level coupling
Three related follow-ups to running the workload at info. integration-test.sh has its own log-trace correlation check that the workload validator knows nothing about: check_log_correlation() greps each node's debug.log for "trace_id=<hex> span_id=<hex>" and fails when it finds none, then cross-checks a sample id against Tempo. At warning it had no guaranteed source. The only warn-or-worse statement inside the activated accept scope is RCLConsensus.cpp:671, which fires solely when a transaction throws, so the check was passing incidentally -- helped by scanning whole files with no time window. Raising it to info gives it the same guarantee the workload now has: the consensus accept pair, one branch of which fires every accepted round. Safe here because this script captures no latency baseline, so there is nothing for the extra log I/O to contaminate. baselines/README.md now records that the committed baseline is only valid at the log level the harness generates. Logging is synchronous and several gated spans contain log statements -- ledger.build has BuildLedger.cpp:81, and consensus.accept has RCLConsensus.cpp:655/663/686 with :663 logging once per transaction -- so the configured level is part of the measurement. Moving it inflates or deflates the quantiles the gate reads without ever reporting a regression, because the baseline moves with it. Changing the level therefore requires re-capturing the baseline. benchmark.sh keeps warning and keeps prefix=xrpld, and now says why. It measures telemetry overhead as a delta between a telemetry-off and a telemetry-on arm, so extra synchronous log I/O would inflate both arms and the thresholds gate the result. The comment exists to stop a future reader "aligning" it with the workload harness and quietly degrading the measurement. |
||
|
Pratik Mankawde
|
d1b80e47a2 | test(telemetry): void span baselines captured on the old span ladder | ||
|
Pratik Mankawde
|
1282645289 |
test(telemetry): invalidate job-queue baselines captured on the old ladder
The workload harness gates regressions on histogram_quantile over job_queued_us / job_running_us, so re-cutting the microsecond ladder changes what those queries return and the stored baselines no longer describe the same measurement. baseline-timings.json's job.acceptLedger.queued.p95 was 96.79us, which is 0.95 / 0.9926 x 100 -- the old 100us bucket edge scaled by the quantile, with 99.3% of samples beneath it. It was never a latency. Keeping it would make the gate LESS sensitive rather than more: a genuine regression from a real 40us to 90us would still sit under 96.79us + 50% and pass. Removes the four job.* entries and records why, including their values. The comparer reports a metric absent from the baseline as "new metric (not in baseline)" and skips it, so the span baselines stay live and gating continues for everything unaffected. is_placeholder() still returns False, so this does not disable the gate wholesale. Recapture the job.* numbers on a node running the re-cut ladder. Also corrects _bucket_note in regression-thresholds.json. It described the spanmetrics ladder as 15 edges starting at 1ms; the collector config has 20, including five sub-millisecond edges. The note's own reasoning was void too -- it justified the 10ms absolute span bound as "~2 low-end bucket widths", but the low-end bucket width is 0.01ms, not 5ms. The bound is kept and justified on the band where span quantiles actually sit, rather than on a derivation from a ladder that no longer exists. |
||
|
Pratik Mankawde
|
d059f21bf3 |
fix(telemetry): address review findings in the workload validation harness
Fixes the review findings on this PR that belong to files it owns, plus several defects found while verifying those fixes. Findings in files owned by upstream branches are routed there and left untouched here. Correctness: - tx_submitter: advance the account sequence only on results that actually consume one (tes*, tec*, terQUEUED). tem*/tef*/tel* never reach the ledger, so advancing left a permanent gap that every later submit from that account inherited. Add a re-fetch hatch so a repeated non-consuming failure cannot livelock on the same sequence, and gate the account check on funded-ness rather than list length. - validate_telemetry: filter spans by name before collecting attributes, so a per-span attribute contract can no longer be satisfied by a sibling span; require exact name equality for non-wildcard children and glob matching for wildcards; bounds-check every returned series instead of only the first. - collect_system_metrics: select xrpld by argv[0] rather than a substring match on the whole command line, which averaged in unrelated processes and reported their RSS as xrpld's. Count genuine 0.0 CPU readings, use a clamped nearest-rank p99 index, and record RPC latency only on success. - benchmark: return each verdict through a named variable instead of a command substitution, so the pass/fail counters survive and the exit gate can fire. Scale before dividing in the percentage math, which truncated a 1.26% impact to 1.00% and cleared a 1% threshold. - compare_to_baseline: fall back to the absolute bound when the baseline is not positive, so a 0 -> 500 ms jump is no longer "within bounds". - rpc_load_generator: bound each connection to one in-flight recv(), drain in-flight requests before closing, use a nearest-rank percentile, and report delivery shortfall so an under-delivered run cannot pass with a 0% error rate. Fail loudly instead of silently: - run-full-validation: treat a consensus timeout and a missing validated ledger as fatal infrastructure errors, and fold the orchestrator and benchmark exit codes into the final status. A degraded cluster previously ran a full validation pass and reported misleading downstream failures. - collect_system_metrics: warn per empty measurement source, emit metrics_complete, and exit non-zero instead of substituting zeros that pass every threshold. Require GNU date with %N rather than falling back to a per-sample python3 fork that costs more than the threshold it is measured against. - benchmark: distinguish "could not measure" from "exceeded thresholds", install a cleanup trap so a failure cannot leak nodes and ports, and report an unusable baseline as inconclusive. - workload_orchestrator: bound subprocess communicate() and fail the exit gate on per-phase errors. Also pins the workload compose images to the versions the sibling stack already uses, hash-pins the Python dependencies, restricts the validator config template to loopback, corrects the dashboard and metric counts in the reference docs, drops a span from the regression gate that cannot fire under a WebSocket-only workload, and narrows the teardown pkill pattern so it no longer matches processes that merely mention the work directory. Verified with a full harness run against a local five-node cluster: 158 of 158 checks passed with no regressions detected. |
||
|
Pratik Mankawde
|
22e440aee1 |
fix(telemetry): correct the phase-10 validation harness against the code
The harness manifests asserted things the code cannot produce and missed most of what it does. Two assertions were failing every run, and the metric set covered 16 of the ~41 emitted names. expected_spans.json: rpc.process was required with rpc.ws_message as its parent, but it is created only in ServerHandler::processRequest() on the HTTP path, so a WebSocket-only workload never produces it -- it is now optional and parented to rpc.http_request, and the rpc.process -> rpc.command.* edge is skipped with the real reason instead of a coroutine-context-loss diagnosis that was never the cause. Adds the missing rpc.ws_upgrade span, corrects four parents (consensus.mode_change, pathfind.request, and update_positions/check, which are children of consensus.establish rather than consensus.round), and demotes conditionally-set attributes out of required_attributes so a healthy run stops failing. Counts recomputed from the file: 41 span types, 62 unique required attributes. expected_metrics.json: 16 -> 52 asserted entries across the job-queue, RPC method, reduce-relay, overflow and validation families, plus the fifteenth dashboard uid. Metrics the harness workload cannot exercise -- erroring RPC, ledger-mismatch, TxQ overflow, and the lazily-created getobject_* instruments -- are listed in a not_asserted group the validator skips, rather than as assertions that would fail on a healthy node. The workflow's push trigger listed two globs matching nothing (include/xrpl/basics/Telemetry*.h, src/xrpld/app/misc/Telemetry*), so no C++ telemetry change ever triggered validation. Replaced with the paths the code actually lives in, including src/libxrpl/beast/insight/** for the insight export path the harness depends on. The four inert workflow_dispatch inputs are now labelled UNUSED rather than looking like working knobs. Docs: the workload README described a StatsD dirty-flag mechanism under a member name that does not exist, on a code path the harness never uses -- it sets [insight] server=otel, so gauges export through an observable-gauge callback every cycle. Adds the missing txq-burst phase, reconciles three different dashboard counts, and drops "posts summary to PR", which the workflow has no permission to do. The runbook's phase-10 section loses the last sampling_ratio reference (not a config key), gains a Regression Gate and CI subsection covering the gate that can fail CI, and its compose-logs command now names the workload compose file. cmake --preset default is left for a separate change: no CMakePresets.json is tracked, so it is wrong everywhere it appears. Also drops the dead exporter=otlp_http key the harness wrote into every node config, and stops capture_timings.py defaulting --profile to a profile that does not exist. |
||
|
Pratik Mankawde
|
d83cb0bdb3 |
fix(telemetry): refresh regression baseline + widen bucket-noise thresholds
With validation now passing 133/133, the only remaining job failure was the regression gate flagging 4 timing "regressions". Two compounding causes: 1. Stale baseline: the committed baseline was captured (2026-04-24) under the old, lighter workload — before the new txq-burst phase (60 TPS) existed. The heavier per-ledger work genuinely raises ledger.build / tx.apply / ledger.validate / acceptLedger timings, so every run regressed against it. Refreshed the baseline from the latest CI-measured timings (same workload). 2. Histogram quantization: SpanMetrics latency buckets are [1,5,10,25,...]ms, so a sub-millisecond quantile near a low-end boundary can jump a full bucket (1ms->5ms) between runs with no real change. The old absolute bounds (2-5ms) were narrower than one bucket width, so that jitter tripped the gate. Widened the default span bounds to 10-15ms (~2 low-end buckets) and pct to 50%, and the job_queue running bound to 20ms, to tolerate quantization noise while still catching genuine multi-bucket regressions. The consensus.* overrides (tight pct, large abs) are unchanged. The refreshed baseline also picks up real rpc.ws_message timings (previously null under the phantom rpc.request key). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Pratik Mankawde
|
cb9fce6890 |
fix(telemetry): align Phase 10 workload harness with current OTel recording surface + fix CI
The Phase 10 validation harness had drifted from the code's recording surface and the telemetry-validation CI job was failing before it could build. CI fix (telemetry-validation.yml): - Replace nonexistent local action ./.github/actions/print-env with the remote XRPLF/actions/print-build-env (the build-xrpld job failed in 56s on this). - Sync prepare-runner and upload-artifact action SHAs to the canonical workflow. Recording-surface reconciliation (docker/telemetry/workload/): - Migrate span attributes from dotted xrpl.<domain>.<field> to the bare/underscore form introduced by the 2026-05-13 span-attr naming redesign (tx_hash, peer_id, ledger_seq, consensus_mode, consensus_round, full_validation, quorum, ...). Dotted xrpl.ledger.hash is retained only on peer.validation.receive (shared constant), while consensus.validation.send uses bare ledger_hash. - Fix attribute placement: tx.apply carries tx_count/tx_failed (not ledger_seq); ledger.build carries ledger_seq/close_* (not tx_count/tx_failed). - Replace the phantom rpc.request span with the real WS root rpc.ws_message; drop the never-emitted duration_ms; rebuild the parent-child map accordingly. - Add the new spans the code emits: apply-pipeline stage spans (tx.preflight/preclaim/transactor with stage/tx_type/ter_result), txq.*, consensus sub-spans (round/establish/update_positions/check/phase.open), ledger.acquire, grpc.*, pathfind.*. Conditional spans are marked optional so they are skipped (not failed) when the workload does not exercise them. - validate_telemetry.py: service.name and Loki job label rippled -> xrpld; fix PARITY_SPAN_ATTRS (rename the 4 real attrs, drop the 3 that are metrics not span attrs); add optional-span handling that skips missing optional spans while still validating attributes when present. - expected_metrics.json: rippled_ -> xrpld_ on all beast::insight/overlay metrics, xrpld_job_count, the 15 on-disk xrpld-* dashboard UIDs, and the real bare spanmetrics dimension labels. - regression-metrics.json + baseline-timings.json: rpc.request -> rpc.ws_message. Metrics pipeline fix: - Switch node [insight] config from server=statsd/prefix=rippled to server=otel + /v1/metrics endpoint + prefix=xrpld across run-full-validation.sh, xrpld-validator.cfg.template, benchmark.sh and the workload compose. The collector has no StatsD receiver, so system metrics only reach Prometheus over OTLP. Synthetic load for new spans: - Add ripple_path_find to the RPC load generator (drives pathfind.* spans). - Add a high-TPS txq-burst workload phase to force fee escalation (drives txq.*). All facts verified against the *SpanNames.h headers and a live xrpld node + collector (Tempo service.name=xrpld, tx.preflight attrs [stage,ter_result,tx_type], 279 xrpld_ Prometheus metrics and zero rippled_). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Pratik Mankawde
|
dc13e9d680 |
fix: populate baseline from CI run, remove dead rpc_methods metrics
Populate baselines/baseline-timings.json from the green CI run
(24906110133, commit
|
||
|
Pratik Mankawde
|
577d1f8a21 |
fix: address review findings in regression gate
- capture_timings.py: fail when captured/total ratio < 50% (--min-capture-ratio). Prevents silent pass on unreachable Prometheus. - run-full-validation.sh: set REGRESSION_EXIT=2 on capture failure so the final exit code reflects it. Update exit code docs in header. - compare_to_baseline.py: extract _skip_delta helper to bring compute_delta under 80 lines. Fix 0.0-as-falsy bug in abs_bound resolution (use explicit None check instead of `or`). Remove dead variable override_prefix_key. - prom_queries.py: extract _build_simple_entries and _build_job_entries to bring build_query_plan under 80 lines. Fix module docstring return type example. Use aiohttp.ClientTimeout instead of bare int. - telemetry-validation.yml: add set -euo pipefail to regression summary step; guard jq calls with -e flag and fallback; fail on missing baseline file; emit ::warning annotation when timings.json missing. - baselines/README.md: document the placeholder field. |
||
|
Pratik Mankawde
|
df79d5e74b |
feat: add OTel-driven regression gate for Phase 10 telemetry validation
Captures per-span / per-RPC / per-job timings from Prometheus after the workload run and diffs them against a committed baseline. Regression requires breaching both a percentage and an absolute bound, tolerating small-value noise. When the baseline is a placeholder, the comparator emits the captured JSON in the exact schema for one-time paste into baselines/baseline-timings.json, and the CI Step Summary surfaces that block for the reviewer. Scope: gate only — automated baseline persistence, benchmark.sh PromQL migration, and the historical trend dashboard remain follow-ups. |