17 Commits

Author	SHA1	Message	Date
Pratik Mankawde	478b3e4b07	docs(telemetry): correct stale claims and citations in the harness docs ... The workload README contradicted itself on --skip-loki: one bullet said CI always passes it and so the two log-correlation checks are never exercised, another said the workflow no longer passes it. The workflow mentions the flag nowhere, so the first was the stale half. Other claims checked against the tree and corrected: - both the README and the plan doc described the push trigger as filtered on branch names. The workflow has no branches filter, deliberately, because GitHub ANDs branches with paths - the plan doc printed 6 of the workflow's 12 paths globs, and claimed the workflow was 367 lines against an actual 451. The glob block is now generated from the workflow, and the line count dropped rather than restated - rpcNOT_SUPPORTED does not exist anywhere in the tree. The symbol is RpcNotSupported, and the refusal sites are RipplePathFind.cpp:59-60 and PathFind.cpp:50-51, not :48-49 and :39 - RCLConsensus.cpp:666 and :663 are not log or event lines; the tx.included event is at :720 and the per-transaction debug log at :715 - LedgerMaster.cpp:463 is fixIndex, not the ledger.store span, which is at :470 - ServerHandler.cpp:705 is inside makeJsonError; processRequest is at :718 - file counts: docker/telemetry/workload/ is 25 files, include/xrpl/telemetry/ 13 - the optional-span bullet named five causes covering 10 of 16 entries, omitting the txq.* family and the WebSocket handshake - the /api/v1/series choice was attributed to stale StatsD gauges; this harness runs no StatsD A line number in run-full-validation.sh was cited in five places and drifts on every edit to that file, so those now name the file only. The keygen helper's header records what production does instead -- validator-keys-tool create_keys then create_token, keeping the master key off the node -- and why a disposable cluster does not.	2026-09-09 13:16:13 +01:00
Pratik Mankawde	3a63a17548	docs(telemetry): stop the harness contract narrating its own revisions ... Notes across the workload contract described earlier versions of themselves, or cited commits that only exist inside this chain. A squash merge publishes none of it, so each reference resolves nowhere. Notes that described their own earlier text: - expected_spans.json: 'this note previously concluded', 'this note previously said', 'Un-skipped 2026-08-26', 'the reason had simply gone stale for two weeks' and 'the claim this entry carried' are replaced by the standing reason each entry holds. The wildcard pairs now say the validator globs the child via _span_name_matches(), and state the literal-collapse failure as what a different validator WOULD do rather than as history. - regression-thresholds.json: 'an earlier version of this note wrongly claimed', 'the earlier version oversold it' and 'an earlier note called that' become the cautions themselves -- do not reason from 'every ladder step is at least 2x', do not oversell the backstop, do not read a false fire as a missing override. - test_check_regression_bounds.py: the docstring gives the reason a literal is wrong here, not the story of two tests that once hard-coded one. Baseline-refresh history rewritten as measurement: - README.md, baselines/README.md, telemetry-runbook.md and regression-metrics.json no longer attribute threshold moves to 'the 2026-08-26 refresh'. The evidence is kept as measurement -- span.tx.apply.p50 has read 0.7917 ms and 0.00597 ms on the same workload, 132x apart; job.acceptLedger.running.p95 has measured a 5.74x floor on one baseline and 16.28x on another -- which is what supports the claim that a single-run baseline cannot bound these keys. Two chain-only commit ids removed, d059f21bf3 and 3860c93db2. Neither is reachable from develop, so both cease to exist on merge; the second is chain bookkeeping. The facts they were cited for (the validator globs wildcards; the span ladder's floor is 0.01 ms) are stated directly instead. Capture provenance is deliberately kept: baseline-timings.json 'captured_at', the 2026-08-26 baseline heading, and the 2026-08-24 figures cited as data. Documentation, JSON note strings and one docstring only, no behaviour change.	2026-09-02 20:32:13 +01:00
Pratik Mankawde	8521b96d85	fix(telemetry): stop an incomplete capture becoming the committed baseline ... The regression baseline is bootstrapped by copying a CI artifact. The workflow tested only that timings.json existed, then printed it verbatim under a heading inviting the reader to paste it in as the new baseline. capture_timings.py writes that file and only then enforces --min-capture-ratio, so an incomplete capture leaves a file that exists but covers fewer keys than the contract declares. The verdict lived in CAPTURE_EXIT, a shell variable local to run-full-validation.sh that no other program could read. So on a placeholder baseline plus a thin capture, CI offered an incomplete artifact as the next baseline, and pasting it narrowed the gate with nothing reporting that it had. That is the failure shape this harness keeps producing: a degraded result that looks exactly like a good one. The artifact now carries its own completeness, next to metrics: "capture": { "declared": 20, "captured": 20, "min_ratio": 0.5, "complete": true } complete is the same condition the producer exits 0 on, computed once with the exit code read off it, so the flag and the status cannot drift apart. Any consumer can now tell a complete capture from a thin one, not just CI. Both paste-me paths refuse rather than warn: the workflow prints the counts and an error annotation with no JSON, and the comparator explains on stderr while leaving stdout empty, so a redirect cannot produce a plausible-looking file. A warning above a copyable block is still a copyable block, and a reader who has just hit a red gate is already predisposed to re-baseline. A missing capture block fails closed. Refusal is scoped to bootstrapping a baseline, not to comparing against one, so artifacts captured before this change still replay: verified against the run the current baseline came from, which carries no capture block and still reports 0 regressions. An injected regression is still caught, and the gated surface is unchanged at 20 keys with 5 excluded.	2026-08-27 09:52:48 +01:00
Pratik Mankawde	a734da8b33	test(telemetry): recapture the baseline and stop gating what variance dominates ... Refreshes baselines/baseline-timings.json from run 32964262700 at 8418d474a7, byte-identical to the CI artifact. The previous baseline was captured at 6a82fc6f37, before the path-finding load was removed from the workload, so it described a load shape the harness no longer runs. Every absolute bound is re-derived, because the rule is hi_next minus baseline and the baselines moved. Three more keys stop being gated: span.tx.apply.p50, span.ledger.build.p50 and span.consensus.ledger_close.p50. This is the rule the previous commit recorded being applied, not a new exception -- a key is gateable only when its run-to-run spread fits inside its bound. The evidence is span.tx.apply.p50, which read 0.7917 ms in the old baseline and 0.00597 ms in this one. That is a 132x move between two runs of the SAME workload. The old value happened to land mid-distribution, so hi_next minus baseline gave a 4.21 ms bound that absorbed the spread; the new value lands in the ladder's first bucket, so the same rule gives 0.0440 ms and cannot survive one. Whether the gate functioned was decided by where in the distribution the captured run happened to fall, which is not a threshold in need of tuning. Measured spreads across four runs agree: 364x, 25.3x and 5.9x respectively. All five excluded keys share one shape -- a baseline landing in the ladder's low buckets, where the derived bound is tiny, together with large run-to-run spread. Single-run baselines cannot support them; a multi-run baseline, or a spread measurement captured alongside the baseline, is what would let them be gated again. Not attempted here. Both runs that would have reddened CI now replay clean, and an injected 10x regression is still caught on 19 of the 20 remaining keys, 20 of 20 at 20x. The exception is job.acceptLedger.running.p95, whose baseline fell while its hi_next did not, moving its floor to 16.28x. It stays gated with that floor recorded beside the other weak keys. Also makes the bounds checker report a zero or negative baseline as a named rule failure instead of dividing by it and raising.	2026-08-26 14:38:16 +01:00
Pratik Mankawde	3836078a78	fix(telemetry): stop gating ledger.validate p95 and p99, which vary too much ... The regression gate has been red on runs with no code change. Only two of the 25 gated keys ever tripped, both on the same span and never together: run 32862589645 failed p99 at 25.8750 ms against a 1.0600 ms baseline (+2341%), run 32867433073 failed p95 at 0.7500 ms against 0.2404 ms (+212%), and in each run the other quantile sat well inside its own bound. A real slowdown would move both. This is variance, not a defect. Measured across four CI runs: span.ledger.validate.p50 0.0484 to 0.0778 ms 1.6x spread kept span.ledger.validate.p95 0.1281 to 0.7500 ms 5.9x spread excluded span.ledger.validate.p99 0.3875 to 25.8750 ms 66.8x spread excluded Both excluded quantiles reach past their trip point on a healthy run. The mechanism is arrival timing, not slow code: the span opens only once a quorum-completing validation arrives (LedgerMaster.cpp:987, inside checkAccept, past the early return) and wraps the promotion work that follows, so one slow consensus round dominates the tail of a 3m rate window and which round that is differs every run. Widening is not available and must not be attempted later: tolerating 25.8750 ms against a 1.0600 ms baseline needs a bound of about 24.8 ms, which gates nothing. A bound admitting every healthy run's worst case admits every regression too. p50 stays gated; it is stable. THE GENERAL RULE, recorded so this does not recur: an absolute bound derived as hi_next minus baseline comes from the histogram ladder, so it budgets for quantization noise and for nothing else. It knows nothing about how far a metric moves between runs on identical code. Before gating any key, check its observed maximum across several runs against its trip point and gate it only with margin. Spread alone proves nothing: tx.apply.p50 swings 364x and never fires, because its 5 ms trip point absorbs the range. Of the 23 keys still gated the worst reaches 0.67 of its trip point. Mechanism: spans.names lists span names while _quantiles is shared, so dropping two quantiles of one span cannot be expressed by deleting a name. regression-metrics.json gains an excluded_keys map from a flat key to the reason it is not gated, subtracted by both prom_queries.py (so the key is never queried) and check_regression_bounds.py rule A. A per-name quantile override was rejected: a typo there leaves the key gating, whereas a typo in an exclusion subtracts nothing and new rule F rejects it, along with an empty reason, a leftover threshold override and a leftover baseline value. Derived figures recomputed from the committed baseline: 25 gated keys to 23, detection floor 2.02x-9.43x to 2.02x-9.42x, weakly guarded keys ten to nine, bound over baseline 102%-843% to 102%-842%. The baseline edit is a deletion of two entries only, with no value rewritten. Verified: both previously failing runs replay to zero regressions and exit 0; a tenfold increase injected into each of the 23 remaining keys in turn is still caught in all 23 cases; rule F was confirmed load-bearing by stubbing it out, which lets a stale exclusion pass.	2026-08-25 18:21:35 +01:00
Pratik Mankawde	e4926f55be	fix(telemetry): derive workload gate bounds from the bucket above the baseline ... The gate could not catch a regression on any sub-millisecond span. compare_to_baseline.py requires both the percentage and the absolute bound to breach, and every span shared one flat absolute bound of 10 ms (15 ms for p99) calibrated for a 5-25 ms band the spans do not occupy. Against the baseline captured on 2026-08-24, where 18 of the 28 quantiles gated at the time sat below 1 ms, that bound sat 1.15x to 2000x above the metric it guarded, so the AND never fired: a 100x regression injected into span.ledger.store.p95 reported 0 regressions and exit 0. Injecting a 10x regression into each key in turn was caught on only 5 of 28. Give every gated key its own absolute bound, equal to the distance from its baseline to hi_next, the edge above the top of the bucket the baseline sits in. The trip point is then exactly hi_next, so the gate fires only once the reading clears the bucket above the baseline's own. That is the property a multiple of the enclosing bucket width cannot provide: after the quantile crosses hi, the interpolation happens across the next bucket, which on this ladder is up to eight times wider, so no multiple of the enclosing width bounds the excursion. Measured with a model-free reachability test, a single bucket crossing can produce a false regression on 2 of 25 keys under the old flat bound and 0 of 25 under this rule. The smallest catchable regression is 2.02x to 9.43x per key. The job queue bound had the same shape of problem on three of its four keys (42x, 47x, 220x before). Defaults now sit at each ladder floor, leaving the percentage bound operative for a metric that somehow reaches them. Drop span.ledger.store from the gated surface. Its captured quantiles were 0.005, 0.0095 and 0.0099 ms, which is the ladder's 0.01 ms floor times the quantile: every sample lands under 10 us, so the reported value does not move even if each store slows from 2 us to 9 us. No bound can gate it. Presence is still asserted by expected_spans.json and the integration test, and the rate is still on the ledger-operations dashboard. Add check_regression_bounds.py, wired into the same workflow step as the bucket parity check. It fails when a bound is not the one its own baseline implies, when a gated key has no override, when the baseline and metric surface disagree, when the percentage bound would become operative, and when a baseline carries the ladder floor signature. This gate has now broken three times through the same drift between ladder, baseline and bounds, so documentation alone is not enough. compare_to_baseline.py is unchanged: its existing per-metric override mechanism already expresses all of this. A missing, unreadable or malformed input makes that check exit 1 naming the input, rather than reporting success without having checked anything; only a placeholder baseline, the documented bootstrap state, still exits 0. Its own tests cover both halves of that contract plus one case per rule, and run in the workflow before the check so a broken rule reads as a broken rule.	2026-08-25 13:02:12 +01:00
Pratik Mankawde	c4b8df9de1	test(telemetry): recapture span baselines on the current ladder ... Copied verbatim from the timings.json produced by the telemetry-validation run at 6a82fc6f37 (166/166 checks passed), which is the hand-off the workflow prints for a placeholder baseline. The numbers confirm why the previous baseline had to be voided. It was captured 2026-06-05, before the collector's spanmetrics ladder gained sub-millisecond edges, and its sub-1ms entries were arithmetic on the old 1ms first edge rather than latencies: span.ledger.store p50/p95/p99 0.5 / 0.95 / 0.99 -> 0.005 / 0.0095 / 0.0099 Exactly 100x, because the old values were quantile x 1ms and the real ones are quantile x 0.01ms. Since the gate only trips on increases, every sub-millisecond span was unguarded against a 100x regression. The job.* pair is back too, recaptured on the re-cut microsecond ladder (floor 1us): job.acceptLedger.queued.p95 now reads 91.1us as a measurement, where the voided value of 96.79us was 0.95/0.9926 x 100.	2026-08-24 22:33:22 +01:00
Pratik Mankawde	98ba282854	fix(telemetry): make the integration test correlate by construction, record the baseline log-level coupling ... Three related follow-ups to running the workload at info. integration-test.sh has its own log-trace correlation check that the workload validator knows nothing about: check_log_correlation() greps each node's debug.log for "trace_id=<hex> span_id=<hex>" and fails when it finds none, then cross-checks a sample id against Tempo. At warning it had no guaranteed source. The only warn-or-worse statement inside the activated accept scope is RCLConsensus.cpp:671, which fires solely when a transaction throws, so the check was passing incidentally -- helped by scanning whole files with no time window. Raising it to info gives it the same guarantee the workload now has: the consensus accept pair, one branch of which fires every accepted round. Safe here because this script captures no latency baseline, so there is nothing for the extra log I/O to contaminate. baselines/README.md now records that the committed baseline is only valid at the log level the harness generates. Logging is synchronous and several gated spans contain log statements -- ledger.build has BuildLedger.cpp:81, and consensus.accept has RCLConsensus.cpp:655/663/686 with :663 logging once per transaction -- so the configured level is part of the measurement. Moving it inflates or deflates the quantiles the gate reads without ever reporting a regression, because the baseline moves with it. Changing the level therefore requires re-capturing the baseline. benchmark.sh keeps warning and keeps prefix=xrpld, and now says why. It measures telemetry overhead as a delta between a telemetry-off and a telemetry-on arm, so extra synchronous log I/O would inflate both arms and the thresholds gate the result. The comment exists to stop a future reader "aligning" it with the workload harness and quietly degrading the measurement.	2026-08-24 20:50:34 +01:00
Pratik Mankawde	d1b80e47a2	test(telemetry): void span baselines captured on the old span ladder	2026-08-24 19:35:00 +01:00
Pratik Mankawde	1282645289	test(telemetry): invalidate job-queue baselines captured on the old ladder ... The workload harness gates regressions on histogram_quantile over job_queued_us / job_running_us, so re-cutting the microsecond ladder changes what those queries return and the stored baselines no longer describe the same measurement. baseline-timings.json's job.acceptLedger.queued.p95 was 96.79us, which is 0.95 / 0.9926 x 100 -- the old 100us bucket edge scaled by the quantile, with 99.3% of samples beneath it. It was never a latency. Keeping it would make the gate LESS sensitive rather than more: a genuine regression from a real 40us to 90us would still sit under 96.79us + 50% and pass. Removes the four job.* entries and records why, including their values. The comparer reports a metric absent from the baseline as "new metric (not in baseline)" and skips it, so the span baselines stay live and gating continues for everything unaffected. is_placeholder() still returns False, so this does not disable the gate wholesale. Recapture the job.* numbers on a node running the re-cut ladder. Also corrects _bucket_note in regression-thresholds.json. It described the spanmetrics ladder as 15 edges starting at 1ms; the collector config has 20, including five sub-millisecond edges. The note's own reasoning was void too -- it justified the 10ms absolute span bound as "~2 low-end bucket widths", but the low-end bucket width is 0.01ms, not 5ms. The bound is kept and justified on the band where span quantiles actually sit, rather than on a derivation from a ladder that no longer exists.	2026-08-21 12:49:56 +01:00
Pratik Mankawde	d059f21bf3	fix(telemetry): address review findings in the workload validation harness ... Fixes the review findings on this PR that belong to files it owns, plus several defects found while verifying those fixes. Findings in files owned by upstream branches are routed there and left untouched here. Correctness: - tx_submitter: advance the account sequence only on results that actually consume one (tes*, tec*, terQUEUED). tem*/tef*/tel* never reach the ledger, so advancing left a permanent gap that every later submit from that account inherited. Add a re-fetch hatch so a repeated non-consuming failure cannot livelock on the same sequence, and gate the account check on funded-ness rather than list length. - validate_telemetry: filter spans by name before collecting attributes, so a per-span attribute contract can no longer be satisfied by a sibling span; require exact name equality for non-wildcard children and glob matching for wildcards; bounds-check every returned series instead of only the first. - collect_system_metrics: select xrpld by argv[0] rather than a substring match on the whole command line, which averaged in unrelated processes and reported their RSS as xrpld's. Count genuine 0.0 CPU readings, use a clamped nearest-rank p99 index, and record RPC latency only on success. - benchmark: return each verdict through a named variable instead of a command substitution, so the pass/fail counters survive and the exit gate can fire. Scale before dividing in the percentage math, which truncated a 1.26% impact to 1.00% and cleared a 1% threshold. - compare_to_baseline: fall back to the absolute bound when the baseline is not positive, so a 0 -> 500 ms jump is no longer "within bounds". - rpc_load_generator: bound each connection to one in-flight recv(), drain in-flight requests before closing, use a nearest-rank percentile, and report delivery shortfall so an under-delivered run cannot pass with a 0% error rate. Fail loudly instead of silently: - run-full-validation: treat a consensus timeout and a missing validated ledger as fatal infrastructure errors, and fold the orchestrator and benchmark exit codes into the final status. A degraded cluster previously ran a full validation pass and reported misleading downstream failures. - collect_system_metrics: warn per empty measurement source, emit metrics_complete, and exit non-zero instead of substituting zeros that pass every threshold. Require GNU date with %N rather than falling back to a per-sample python3 fork that costs more than the threshold it is measured against. - benchmark: distinguish "could not measure" from "exceeded thresholds", install a cleanup trap so a failure cannot leak nodes and ports, and report an unusable baseline as inconclusive. - workload_orchestrator: bound subprocess communicate() and fail the exit gate on per-phase errors. Also pins the workload compose images to the versions the sibling stack already uses, hash-pins the Python dependencies, restricts the validator config template to loopback, corrects the dashboard and metric counts in the reference docs, drops a span from the regression gate that cannot fire under a WebSocket-only workload, and narrows the teardown pkill pattern so it no longer matches processes that merely mention the work directory. Verified with a full harness run against a local five-node cluster: 158 of 158 checks passed with no regressions detected.	2026-08-14 19:59:19 +01:00
Pratik Mankawde	22e440aee1	fix(telemetry): correct the phase-10 validation harness against the code ... The harness manifests asserted things the code cannot produce and missed most of what it does. Two assertions were failing every run, and the metric set covered 16 of the ~41 emitted names. expected_spans.json: rpc.process was required with rpc.ws_message as its parent, but it is created only in ServerHandler::processRequest() on the HTTP path, so a WebSocket-only workload never produces it -- it is now optional and parented to rpc.http_request, and the rpc.process -> rpc.command.* edge is skipped with the real reason instead of a coroutine-context-loss diagnosis that was never the cause. Adds the missing rpc.ws_upgrade span, corrects four parents (consensus.mode_change, pathfind.request, and update_positions/check, which are children of consensus.establish rather than consensus.round), and demotes conditionally-set attributes out of required_attributes so a healthy run stops failing. Counts recomputed from the file: 41 span types, 62 unique required attributes. expected_metrics.json: 16 -> 52 asserted entries across the job-queue, RPC method, reduce-relay, overflow and validation families, plus the fifteenth dashboard uid. Metrics the harness workload cannot exercise -- erroring RPC, ledger-mismatch, TxQ overflow, and the lazily-created getobject_* instruments -- are listed in a not_asserted group the validator skips, rather than as assertions that would fail on a healthy node. The workflow's push trigger listed two globs matching nothing (include/xrpl/basics/Telemetry*.h, src/xrpld/app/misc/Telemetry*), so no C++ telemetry change ever triggered validation. Replaced with the paths the code actually lives in, including src/libxrpl/beast/insight/** for the insight export path the harness depends on. The four inert workflow_dispatch inputs are now labelled UNUSED rather than looking like working knobs. Docs: the workload README described a StatsD dirty-flag mechanism under a member name that does not exist, on a code path the harness never uses -- it sets [insight] server=otel, so gauges export through an observable-gauge callback every cycle. Adds the missing txq-burst phase, reconciles three different dashboard counts, and drops "posts summary to PR", which the workflow has no permission to do. The runbook's phase-10 section loses the last sampling_ratio reference (not a config key), gains a Regression Gate and CI subsection covering the gate that can fail CI, and its compose-logs command now names the workload compose file. cmake --preset default is left for a separate change: no CMakePresets.json is tracked, so it is wrong everywhere it appears. Also drops the dead exporter=otlp_http key the harness wrote into every node config, and stops capture_timings.py defaulting --profile to a profile that does not exist.	2026-08-14 12:34:33 +01:00
Pratik Mankawde	d83cb0bdb3	fix(telemetry): refresh regression baseline + widen bucket-noise thresholds ... With validation now passing 133/133, the only remaining job failure was the regression gate flagging 4 timing "regressions". Two compounding causes: 1. Stale baseline: the committed baseline was captured (2026-04-24) under the old, lighter workload — before the new txq-burst phase (60 TPS) existed. The heavier per-ledger work genuinely raises ledger.build / tx.apply / ledger.validate / acceptLedger timings, so every run regressed against it. Refreshed the baseline from the latest CI-measured timings (same workload). 2. Histogram quantization: SpanMetrics latency buckets are [1,5,10,25,...]ms, so a sub-millisecond quantile near a low-end boundary can jump a full bucket (1ms->5ms) between runs with no real change. The old absolute bounds (2-5ms) were narrower than one bucket width, so that jitter tripped the gate. Widened the default span bounds to 10-15ms (~2 low-end buckets) and pct to 50%, and the job_queue running bound to 20ms, to tolerate quantization noise while still catching genuine multi-bucket regressions. The consensus.* overrides (tight pct, large abs) are unchanged. The refreshed baseline also picks up real rpc.ws_message timings (previously null under the phantom rpc.request key). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-05 19:58:07 +01:00
Pratik Mankawde	cb9fce6890	fix(telemetry): align Phase 10 workload harness with current OTel recording surface + fix CI ... The Phase 10 validation harness had drifted from the code's recording surface and the telemetry-validation CI job was failing before it could build. CI fix (telemetry-validation.yml): - Replace nonexistent local action ./.github/actions/print-env with the remote XRPLF/actions/print-build-env (the build-xrpld job failed in 56s on this). - Sync prepare-runner and upload-artifact action SHAs to the canonical workflow. Recording-surface reconciliation (docker/telemetry/workload/): - Migrate span attributes from dotted xrpl.<domain>.<field> to the bare/underscore form introduced by the 2026-05-13 span-attr naming redesign (tx_hash, peer_id, ledger_seq, consensus_mode, consensus_round, full_validation, quorum, ...). Dotted xrpl.ledger.hash is retained only on peer.validation.receive (shared constant), while consensus.validation.send uses bare ledger_hash. - Fix attribute placement: tx.apply carries tx_count/tx_failed (not ledger_seq); ledger.build carries ledger_seq/close_* (not tx_count/tx_failed). - Replace the phantom rpc.request span with the real WS root rpc.ws_message; drop the never-emitted duration_ms; rebuild the parent-child map accordingly. - Add the new spans the code emits: apply-pipeline stage spans (tx.preflight/preclaim/transactor with stage/tx_type/ter_result), txq.*, consensus sub-spans (round/establish/update_positions/check/phase.open), ledger.acquire, grpc.*, pathfind.*. Conditional spans are marked optional so they are skipped (not failed) when the workload does not exercise them. - validate_telemetry.py: service.name and Loki job label rippled -> xrpld; fix PARITY_SPAN_ATTRS (rename the 4 real attrs, drop the 3 that are metrics not span attrs); add optional-span handling that skips missing optional spans while still validating attributes when present. - expected_metrics.json: rippled_ -> xrpld_ on all beast::insight/overlay metrics, xrpld_job_count, the 15 on-disk xrpld-* dashboard UIDs, and the real bare spanmetrics dimension labels. - regression-metrics.json + baseline-timings.json: rpc.request -> rpc.ws_message. Metrics pipeline fix: - Switch node [insight] config from server=statsd/prefix=rippled to server=otel + /v1/metrics endpoint + prefix=xrpld across run-full-validation.sh, xrpld-validator.cfg.template, benchmark.sh and the workload compose. The collector has no StatsD receiver, so system metrics only reach Prometheus over OTLP. Synthetic load for new spans: - Add ripple_path_find to the RPC load generator (drives pathfind.* spans). - Add a high-TPS txq-burst workload phase to force fee escalation (drives txq.*). All facts verified against the *SpanNames.h headers and a live xrpld node + collector (Tempo service.name=xrpld, tx.preflight attrs [stage,ter_result,tx_type], 279 xrpld_ Prometheus metrics and zero rippled_). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-05 17:08:58 +01:00
Pratik Mankawde	dc13e9d680	fix: populate baseline from CI run, remove dead rpc_methods metrics ... Populate baselines/baseline-timings.json from the green CI run (24906110133, commit f11ebc1253). 25/31 metrics have non-null values; 6 span.rpc.* are null due to sparse data in the 3m window. Remove the rpc_methods section from regression-metrics.json and its thresholds. rippled_rpc_method_duration_us_bucket is never populated because PerfLogImp::rpcEnd never calls MetricsRegistry::recordRpcFinished — only recordRpcStarted is wired up (Phase 9 instrumentation gap). The span-based rpc.request/rpc.process metrics via spanmetrics already cover RPC latency.	2026-04-24 20:08:52 +01:00
Pratik Mankawde	577d1f8a21	fix: address review findings in regression gate ... - capture_timings.py: fail when captured/total ratio < 50% (--min-capture-ratio). Prevents silent pass on unreachable Prometheus. - run-full-validation.sh: set REGRESSION_EXIT=2 on capture failure so the final exit code reflects it. Update exit code docs in header. - compare_to_baseline.py: extract _skip_delta helper to bring compute_delta under 80 lines. Fix 0.0-as-falsy bug in abs_bound resolution (use explicit None check instead of `or`). Remove dead variable override_prefix_key. - prom_queries.py: extract _build_simple_entries and _build_job_entries to bring build_query_plan under 80 lines. Fix module docstring return type example. Use aiohttp.ClientTimeout instead of bare int. - telemetry-validation.yml: add set -euo pipefail to regression summary step; guard jq calls with -e flag and fallback; fail on missing baseline file; emit ::warning annotation when timings.json missing. - baselines/README.md: document the placeholder field.	2026-04-24 19:36:15 +01:00
Pratik Mankawde	df79d5e74b	feat: add OTel-driven regression gate for Phase 10 telemetry validation ... Captures per-span / per-RPC / per-job timings from Prometheus after the workload run and diffs them against a committed baseline. Regression requires breaching both a percentage and an absolute bound, tolerating small-value noise. When the baseline is a placeholder, the comparator emits the captured JSON in the exact schema for one-time paste into baselines/baseline-timings.json, and the CI Step Summary surfaces that block for the reviewer. Scope: gate only — automated baseline persistence, benchmark.sh PromQL migration, and the historical trend dashboard remain follow-ups.	2026-04-24 18:53:44 +01:00