ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-21 22:30:57 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
0b9684d47204d397dfe4bc02278616f857ee7140
rippled/docker/telemetry/workload/baselines
History
Pratik Mankawde 1282645289 test(telemetry): invalidate job-queue baselines captured on the old ladder 
The workload harness gates regressions on histogram_quantile over
job_queued_us / job_running_us, so re-cutting the microsecond ladder changes
what those queries return and the stored baselines no longer describe the
same measurement.

baseline-timings.json's job.acceptLedger.queued.p95 was 96.79us, which is
0.95 / 0.9926 x 100 -- the old 100us bucket edge scaled by the quantile, with
99.3% of samples beneath it. It was never a latency. Keeping it would make the
gate LESS sensitive rather than more: a genuine regression from a real 40us to
90us would still sit under 96.79us + 50% and pass.

Removes the four job.* entries and records why, including their values. The
comparer reports a metric absent from the baseline as "new metric (not in
baseline)" and skips it, so the span baselines stay live and gating continues
for everything unaffected. is_placeholder() still returns False, so this does
not disable the gate wholesale. Recapture the job.* numbers on a node running
the re-cut ladder.

Also corrects _bucket_note in regression-thresholds.json. It described the
spanmetrics ladder as 15 edges starting at 1ms; the collector config has 20,
including five sub-millisecond edges. The note's own reasoning was void too --
it justified the 10ms absolute span bound as "~2 low-end bucket widths", but
the low-end bucket width is 0.01ms, not 5ms. The bound is kept and justified
on the band where span quantiles actually sit, rather than on a derivation
from a ladder that no longer exists.
2026-08-21 12:49:56 +01:00
..
baseline-timings.json
test(telemetry): invalidate job-queue baselines captured on the old ladder
2026-08-21 12:49:56 +01:00
README.md
fix(telemetry): address review findings in the workload validation harness
2026-08-14 19:59:19 +01:00

README.md

Performance Baselines

This directory holds the committed baseline file used by the OTel-driven regression gate.

How the gate works

After the validation suite runs, capture_timings.py queries Prometheus for the timings declared in ../regression-metrics.json and writes a timings.json. Then compare_to_baseline.py reads baseline-timings.json, ../regression-thresholds.json, and the captured timings.json. The comparator picks one of two modes automatically:

  • Placeholder baseline ("placeholder": true or empty metrics): the comparator prints the captured timings JSON in exactly the format expected for this file, then exits 0 without gating. This is how we bootstrap the baseline.
  • Populated baseline: the comparator diffs per-metric, enforces the thresholds (regression = current exceeds baseline on BOTH the percentage AND absolute bound), and exits non-zero on any regression. The single exception is a baseline that is not positive: the percentage change is undefined there, so the absolute bound decides alone. Without that fallback the AND gate would be unreachable and a 0 ms → 500 ms jump would be reported as "within bounds".

The regression gate runs against whatever workload profile run-full-validation.sh was invoked with. Capture and comparison are profile-agnostic — they only read Prometheus — so all existing profiles (full-validation, quick-smoke, stress) continue to work unchanged.

Bootstrapping the baseline

  1. Merge a CI run with a "placeholder": true baseline. The telemetry-validation workflow runs, fails no gate, and prints the captured timings block to the workflow Step Summary under the heading ### Paste into baselines/baseline-timings.json.
  2. Open a new PR. Copy the full JSON block from the Step Summary (or download the timings.json artifact) into this file, replacing the placeholder contents. The JSON is emitted in the exact byte-for-byte format this file expects — sorted keys, 2-space indent, trailing newline.
  3. The committed baseline PR needs reviewer approval just like any other code change. This is the primary audit point for "who moved the performance bar."

Refreshing the baseline

Refresh when a legitimate performance change lands on develop (for example, a deliberate rewrite that changes a span's structure). The process is identical to bootstrapping: run CI with the current baseline, inspect the delta, and if the new numbers should become the norm, open a PR pasting the fresh timings into baseline-timings.json. The reviewer decides whether the new baseline is acceptable.

Do not edit baseline-timings.json by hand outside of this process — every entry should trace back to a real CI run so variance characteristics are preserved.

Schema

{
  "schema_version": 1,
  "captured_at": "2026-04-24T17:30:00Z",
  "window": "3m",
  "git_sha": "<SHA of the commit that produced these numbers>",
  "profile": "<workload profile used>",
  "metrics": {
    "span.tx.process.p99": { "value": 12.4, "unit": "ms" },
    "job.transaction.queued.p95": { "value": 1500.0, "unit": "us" }
  }
}

Keys follow {category}.{name}.p{quantile}. Only two categories are actually produced today — span.* and job.* — because build_query_plan() in prom_queries.py reads the spans and job_queue groups of regression-metrics.json, and that file defines only those two.

Placeholder baselines additionally include "placeholder": true. The comparator detects this field (or an empty metrics object) to switch into "populate" mode instead of enforcing thresholds. Remove the placeholder key when pasting real captured timings.

Missing metrics (value null) in a captured run do not count as regressions. In regression-report.json, summary.missing_in_current is a count only; the identities are in the metrics[] array, as the entries whose note is "not captured in current run". Filter for those to see which keys went missing:

jq -r '.metrics[] | select(.note == "not captured in current run") | .key' \
    /tmp/xrpld-validation/reports/regression-report.json

This keeps the gate robust when a profile doesn't exercise every span on every run.

Known gap: no rpc.* metric can gate (FU-4)

Per-RPC-method timings are not gated, and would not gate even if they were captured. Two independent blockers:

  1. Nothing emits an rpc.* key. build_query_plan() in prom_queries.py builds rpc.* entries from cfg.get("rpc_methods", {}), and regression-metrics.json has no rpc_methods block — so the group resolves to empty and no rpc.* key ever reaches timings.json or this baseline.
  2. Even a captured rpc.* key would silently not gate. resolve_thresholds() in compare_to_baseline.py maps the rpc category to the threshold group rpc_method, but regression-thresholds.json defines only defaults.span and defaults.job_queue. With no rpc_method block the lookup returns (None, None), which the comparator treats as "no threshold configured" — the metric is reported but can never fail the build.

Closing this needs both an rpc_methods group in regression-metrics.json and a defaults.rpc_method block in regression-thresholds.json. Adding only the first produces metrics that look gated in the report but are not.

Known exclusion: rpc.process is not captured

rpc.process is deliberately absent from the spans.names list in regression-metrics.json, so no span.rpc.process.* key appears in this baseline. The span is created only in ServerHandler::processRequest() (src/xrpld/rpc/detail/ServerHandler.cpp:705), which is reached only from the HTTP/JSON-RPC session path. The harness load generator is WebSocket-only and that path never calls processRequest, so the span is never emitted under any workload profile here — expected_spans.json marks it "optional": true for the same reason.

While it was listed, the three quantiles were captured as null on every run and the comparator short-circuited them as "new metric (not in baseline)" — so a 9999 ms value would still have reported regressed: false. Three keys that can never gate are worse than no keys: they inflate summary.total and read as covered.

If per-request HTTP timings are wanted, the fix is to give the harness an HTTP/JSON-RPC load path first, then re-add rpc.process and bootstrap a real baseline for it.