mirror of https://github.com/XRPLF/rippled.git synced 2026-06-06 18:26:51 +00:00

Files

Pratik Mankawde d83cb0bdb3 fix(telemetry): refresh regression baseline + widen bucket-noise thresholds

With validation now passing 133/133, the only remaining job failure was the
regression gate flagging 4 timing "regressions". Two compounding causes:

1. Stale baseline: the committed baseline was captured (2026-04-24) under the
   old, lighter workload — before the new txq-burst phase (60 TPS) existed. The
   heavier per-ledger work genuinely raises ledger.build / tx.apply /
   ledger.validate / acceptLedger timings, so every run regressed against it.
   Refreshed the baseline from the latest CI-measured timings (same workload).
2. Histogram quantization: SpanMetrics latency buckets are
   [1,5,10,25,...]ms, so a sub-millisecond quantile near a low-end boundary can
   jump a full bucket (1ms->5ms) between runs with no real change. The old
   absolute bounds (2-5ms) were narrower than one bucket width, so that jitter
   tripped the gate. Widened the default span bounds to 10-15ms (~2 low-end
   buckets) and pct to 50%, and the job_queue running bound to 20ms, to tolerate
   quantization noise while still catching genuine multi-bucket regressions. The
   consensus.* overrides (tight pct, large abs) are unchanged.

The refreshed baseline also picks up real rpc.ws_message timings (previously null
under the phantom rpc.request key).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-05 19:58:07 +01:00

baseline-timings.json

fix(telemetry): refresh regression baseline + widen bucket-noise thresholds

2026-06-05 19:58:07 +01:00

README.md

fix: address review findings in regression gate

2026-04-24 19:36:15 +01:00

README.md

Performance Baselines

This directory holds the committed baseline file used by the OTel-driven regression gate.

How the gate works

After the validation suite runs, capture_timings.py queries Prometheus for the timings declared in ../regression-metrics.json and writes a timings.json. Then compare_to_baseline.py reads baseline-timings.json, ../regression-thresholds.json, and the captured timings.json. The comparator picks one of two modes automatically:

Placeholder baseline ("placeholder": true or empty metrics): the comparator prints the captured timings JSON in exactly the format expected for this file, then exits 0 without gating. This is how we bootstrap the baseline.
Populated baseline: the comparator diffs per-metric, enforces the thresholds (regression = current exceeds baseline on BOTH the percentage AND absolute bound), and exits non-zero on any regression.

The regression gate runs against whatever workload profile run-full-validation.sh was invoked with. Capture and comparison are profile-agnostic — they only read Prometheus — so all existing profiles (full-validation, quick-smoke, stress) continue to work unchanged.

Bootstrapping the baseline

Merge a CI run with a "placeholder": true baseline. The telemetry-validation workflow runs, fails no gate, and prints the captured timings block to the workflow Step Summary under the heading ### Paste into baselines/baseline-timings.json.
Open a new PR. Copy the full JSON block from the Step Summary (or download the timings.json artifact) into this file, replacing the placeholder contents. The JSON is emitted in the exact byte-for-byte format this file expects — sorted keys, 2-space indent, trailing newline.
The committed baseline PR needs reviewer approval just like any other code change. This is the primary audit point for "who moved the performance bar."

Refreshing the baseline

Refresh when a legitimate performance change lands on develop (for example, a deliberate rewrite that changes a span's structure). The process is identical to bootstrapping: run CI with the current baseline, inspect the delta, and if the new numbers should become the norm, open a PR pasting the fresh timings into baseline-timings.json. The reviewer decides whether the new baseline is acceptable.

Do not edit baseline-timings.json by hand outside of this process — every entry should trace back to a real CI run so variance characteristics are preserved.

Schema

{
  "schema_version": 1,
  "captured_at": "2026-04-24T17:30:00Z",
  "window": "3m",
  "git_sha": "<SHA of the commit that produced these numbers>",
  "profile": "<workload profile used>",
  "metrics": {
    "span.tx.process.p99": { "value": 12.4, "unit": "ms" },
    "rpc.server_info.p95": { "value": 850.0, "unit": "us" },
    "job.transaction.queued.p95": { "value": 1500.0, "unit": "us" }
  }
}

Placeholder baselines additionally include "placeholder": true. The comparator detects this field (or an empty metrics object) to switch into "populate" mode instead of enforcing thresholds. Remove the placeholder key when pasting real captured timings.

Missing metrics (value null) in a captured run do not count as regressions — they are reported separately in regression-report.json under missing_in_current. This keeps the gate robust when a profile doesn't exercise every span on every run.