Dashboards and alert rules reference 186 metrics; the harness asserted 57.
Excluding the 107 per-category overlay-traffic expansions, the meaningful
gap was 20 names. This closes it under the contract file's own doctrine:
assert only what the workload guarantees, and record the rest with a
precise reason.
Asserted 18, taking the metric checks from 61 to 79 and the whole metric
phase from 66 to 84. No pre-existing check name or position changes.
statsd_gauges gains the nine state_accounting_* siblings of the one member
already asserted, plus the two NodeFamily full-below-cache gauges and
overlay_peer_disconnects. All twelve rest on one mechanism the group
description now spells out: on the OTel path a beast gauge is an
Int64ObservableGauge, every instance self-registers in its constructor,
onCollectionReady arms all of them unconditionally, and the armed callback
Observes on every export cycle whether or not set was ever called, so the
series exist at 0. The state_accounting family is set in one unconditional
block in NetworkOPsImp::collectMetrics, and full_transitions is the input
to the NodeStateFlapping alert rule, so the alert's own signal had been
going unverified.
A new job_queue_per_type_gauges group asserts the six per-job-type gauges
that a panel or a rule names literally, jobq_manifest_waiting among them
as the ManifestJobQueueConvoy rule's input. The description records why
those six and not all 105: the guarantee is identical for every
non-special job type, so the discriminator is consumer coverage, and the
remaining names are only reached through topk queries over the family that
do not depend on any single type being present.
Recorded four more in not_asserted rather than asserting them.
pathfind_fast_milliseconds is unreachable for this workload, not merely
rare: reportFast fires only from the doCreate fast pass, which is guarded
by !hasCompletion(), and both ripple_path_find entry points construct the
request with a completion function. Only the path_find subscription
reaches it, and the generator does not use it.
pathfind_full_milliseconds is reachable but only one ledger close after
the request, through PathRequestManager::updateAll, and nothing in the
harness arranges or checks that, so the guarantee is probabilistic.
warn_total and drop_total are resource-manager meters gated on a consumer
crossing the warn or drop threshold; their rpc-pathfinding panels are
correct and render empty only because the condition has not occurred,
which is worth stating because both were briefly mis-read as phantoms.
Runbook check counts updated to match.
The gate could not catch a regression on any sub-millisecond span.
compare_to_baseline.py requires both the percentage and the absolute bound to
breach, and every span shared one flat absolute bound of 10 ms (15 ms for p99)
calibrated for a 5-25 ms band the spans do not occupy. Against the baseline
captured on 2026-08-24, where 18 of the 28 quantiles gated at the time sat
below 1 ms, that bound sat 1.15x to 2000x above the metric it guarded, so the
AND never fired: a 100x regression injected into span.ledger.store.p95 reported
0 regressions and exit 0. Injecting a 10x regression into each key in turn was
caught on only 5 of 28.
Give every gated key its own absolute bound, equal to the distance from its
baseline to hi_next, the edge above the top of the bucket the baseline sits in.
The trip point is then exactly hi_next, so the gate fires only once the reading
clears the bucket above the baseline's own. That is the property a multiple of
the enclosing bucket width cannot provide: after the quantile crosses hi, the
interpolation happens across the next bucket, which on this ladder is up to
eight times wider, so no multiple of the enclosing width bounds the excursion.
Measured with a model-free reachability test, a single bucket crossing can
produce a false regression on 2 of 25 keys under the old flat bound and 0 of 25
under this rule. The smallest catchable regression is 2.02x to 9.43x per key.
The job queue bound had the same shape of problem on three of its four keys
(42x, 47x, 220x before). Defaults now sit at each ladder floor, leaving the
percentage bound operative for a metric that somehow reaches them.
Drop span.ledger.store from the gated surface. Its captured quantiles were
0.005, 0.0095 and 0.0099 ms, which is the ladder's 0.01 ms floor times the
quantile: every sample lands under 10 us, so the reported value does not move
even if each store slows from 2 us to 9 us. No bound can gate it. Presence is
still asserted by expected_spans.json and the integration test, and the rate is
still on the ledger-operations dashboard.
Add check_regression_bounds.py, wired into the same workflow step as the bucket
parity check. It fails when a bound is not the one its own baseline implies,
when a gated key has no override, when the baseline and metric surface disagree,
when the percentage bound would become operative, and when a baseline carries
the ladder floor signature. This gate has now broken three times through the
same drift between ladder, baseline and bounds, so documentation alone is not
enough.
compare_to_baseline.py is unchanged: its existing per-metric override mechanism
already expresses all of this.
A missing, unreadable or malformed input makes that check exit 1 naming the
input, rather than reporting success without having checked anything; only a
placeholder baseline, the documented bootstrap state, still exits 0. Its own
tests cover both halves of that contract plus one case per rule, and run in the
workflow before the check so a broken rule reads as a broken rule.
The runbook said a correlated log line was "guaranteed" at info severity.
Info is necessary but not sufficient. Replaced the flat claim with the four
real preconditions, each with the code that enforces it and the failure mode
it produces: telemetry enabled, trace_consensus=1, a valid roundSpanContext_
(SpanGuard::childSpan returns a null guard on an invalid parent), and a valid
plus sampled span context (Log.cpp gates injection on IsValid and IsSampled).
Also noted which harness cfgs satisfy them -- run-full-validation.sh and
integration-test.sh set all three config keys; benchmark.sh deliberately
stays at warning and runs no correlation check.
Second, the two checks this work exists to make pass are not exercised by
CI. The workflow hardcodes --skip-loki, and validate_telemetry.py builds
log.trace_id_present and log.trace_id_cross_reference only inside an
"if not skip_loki" branch, so they are never constructed rather than merely
skipped, and never appear in the report. No workflow runs integration-test.sh
either, so its own check_log_correlation() never runs in CI. Recorded that in
the runbook's CI workflow section and in the workload README, with the local
command that does cover it: run-full-validation.sh without --skip-loki.
The workflow itself is unchanged on purpose. Dropping the flag would make CI
exercise Loki ingestion and filelog mounting for the first time on the same
run that must produce a clean regression baseline, so a red result would not
be attributable.
The "What Gets Validated" table claimed "58 metrics in 23 categories". Both
numbers were stale: generalising required_labels and adding the asserted
io_latency group changed the gate to 66 checks across 24 asserting categories
(61 metric names plus 5 required_labels checks).
Derived from the inventory rather than counted by hand -- ran the validator's
own _metric_check_targets() against expected_metrics.json:
metric names = 61 | label checks = 5 | TOTAL = 66
asserting groups = 24
label checks per group = {'spanmetrics': 4, 'job_queue': 1}
Also records that labels are gated at all. They were declared in the contract
but nothing read the key until the check was generalised, so four spanmetrics
labels were documented as required while going unverified -- the table
described existence checks only and gave no hint that a label regression was
now catchable.
The other rows in the same table were re-derived and are still accurate:
spans 41 total / 26 required / 15 optional, parity 6 span attributes + 4 value
bounds = 10, dashboards 15 uids against 15 provisioned files, logs 2.
The row listed only the two end-of-phase attributes. Add the four set at span
creation and the three set at the close decision.
Records what the previous wording implied but did not state: the end-of-phase
attributes are absent when the round is recovered by handleWrongLedger or
driven by simulate(), because neither reaches closeLedger(). Any average over
open_duration_ms silently excludes those rounds.
Edited here rather than on phase 5, where this cell is empty and would conflict
on the way up.
The two log.trace_id_* checks have failed on every run -- they were the only
failures in the 2026-08-20 run (158/160). The workload never satisfied their
precondition, because warning suppressed the one line that is correlated by
construction.
trace_id is injected in Log.cpp from RuntimeContext::GetCurrent(). Severity
does not affect injection, but JLOG filters on severity before format() runs,
so what matters is which severity emits a line while a span is current.
A span becomes current in either of two ways: as a ScopedSpanGuard, or by
activating a plain SpanGuard via activate() / activateIfLive(). activate()
returns a ScopedActivation holding an otel_trace::Scope built from the span,
which pushes onto the same RuntimeContext store Log.cpp reads. A plain
SpanGuard that is never activated makes no span current.
The guaranteed correlated line at info is the consensus accept pair at
RCLConsensus.cpp:736/740 -- an if/else, so exactly one fires on every accepted
round. doAccept activates the accept span as ambient over its whole body at
:565 via activateIfLive(acceptSpan), and that activation lives to the end of
the function, so both branches are inside it. At roughly one round every 4 s
this gives dozens of correlated lines per run, well inside the validator's 4 h
window. LOG_QUERY_WINDOW_SECONDS stays at 4 h deliberately -- a wider window
would let the check pass on logs from a previous run.
info is the minimum that works, which is what the task asked for. debug would
correlate strictly more, additionally covering BuildLedger.cpp:81 and
RPCHandler.cpp:188, but it is the wrong default: it puts synchronous log I/O
inside ledger.build, consensus.accept (RCLConsensus.cpp:663 logs per
transaction) and tx.apply, which are exactly the spans whose latency
regression-metrics.json gates. The next run reprints the voided baseline, so
capturing at debug would bake log I/O into the latency numbers permanently --
the same class of defect this plan exists to remove. The runbook records how to
get the broader coverage per partition, after a baseline exists.
The dashboard inventory listed "Peer TX Receive Rate", but this branch
deletes that panel from transaction-overview.json, so an operator
following the runbook finds no such chart. The row is stale from
phase-7 onward and is still present at the tip of the chain, so
nothing downstream repairs it.
Remove the row rather than retarget it: the neighbouring
"Transaction Receive vs Suppressed" row already documents the
tx.receive rate and its panel exists on every branch, so retargeting
would have produced two rows for one chart.
The microsecond ladder's first edge was 100us, which sat ABOVE the mass of
every instrument using it. Measured on devnet: 99.3% of job_queued_us
samples, 92.5% of job_running_us and 90.4% of getobject_lookup_us fell in
that first bucket. histogram_quantile then interpolated inside bucket 0 and
returned `quantile / fraction_in_bucket_0 x first_edge` -- p75/p95/p99 of
job_queued_us read 75.52/95.66/99.69us against a prediction of
75.53/95.67/99.70. Three-decimal agreement: those panels were reporting
arithmetic on the bucket edge, not latency.
The fix was already half-written. kSubMillisecondBoundaries had been parked
in MetricsRegistry.cpp as [[maybe_unused]] with a comment noting exactly this
problem for nodestore reads. Its edges are now folded into kMicrosecondBuckets
rather than deleted, so the parked intent is carried forward: 1..1000us
resolution where the mass is, upper edges unchanged so multi-second stalls
stay measurable.
Also moves the GetObject count and charge ladders into HistogramBuckets.h, so
all five ladders have one owner and one set of invariant tests (29 now).
Adds check_bucket_parity.py, wired into the existing OTel naming workflow.
The C++ millisecond ladder and the collector's spanmetrics ladder are
specified to agree over their shared range; they were identical when shipped,
then the collector side alone was extended and nothing noticed for eleven
phases. The check asserts containment rather than equality, because jobs
outlive spans -- jobq_updatepaths averages ~60s, which no span approaches, so
demanding equality would force a ceiling that censors it. Verified it rejects
a missing collector edge, a bogus in-range edge, and a return to the 5s
ceiling.
ledger-data-sync's "Job Queue Wait p95 By Type" moves off the beast
jobq_*_q_milliseconds pair onto job_queued_us filtered by job_type. Those
beast metrics are ms-quantised at the source (Event rounds up to a whole
millisecond), so 94-100% of their samples sat in the first bucket and no
ladder change could fix them. Note the label values are camelCase
(job_type="ledgerData"), not the lowercase metric-name fragments.
Both histogram-fed alert thresholds re-validated and left unchanged, with the
measured basis recorded so neither gets tuned against the old artefact: only
0.0022% of job_queued_us samples exceed the 1s threshold, and every edge
bracketing the 1000ms ios_latency threshold survived the ladder change.
Docs: the rpc_size "known issue -- tracked separately" notes in the runbook
and 09-data-collection-reference are now resolved notes, the stale 10-edge
span_duration bucket list is corrected to the collector's real 20, and the
runbook gains a "Reading A Histogram Percentile" section covering both
saturation traps and the expected discontinuity after a ladder change.
Bring the three documentation surfaces in line with the new parse-time check:
- The @throws clause on makeTelemetrySetup now names the third failure
condition and records that an empty path is skipped.
- cfg/xrpld-example.cfg states, under all three TLS keys, that with enabled=1
and use_tls=1 a path that does not exist or cannot be read stops startup. The
tls_ca_cert wording still says that empty selects the system CA store, since
only a path that is set is checked.
- The runbook troubleshooting entry gains a third bullet for the "cannot be
read" message, whose remedy is the path or its permissions rather than the
certificate and key pairing.
Documentation only; no behaviour change.
Use `rgb(15, 122, 102)` instead of `rgb(25, 158, 112)`: the brighter step drew
too much attention for a background band.
This is the darkest teal that still separates from the JMeter grey by a readable
margin -- normal-vision dE 15.6 against a floor of 15, CVD dE 12.3 against a
target of 8, and at least 3:1 on the dark surface. Dimmer steps fail: rgb(25,
100, 90) lands at dE 9.5, and a grey-derived rgb(25, 70, 70) at dE 5.8, which is
indistinguishable from the JMeter grey even with full colour vision.
The driver split changed the existing perf-run regions from grey to violet,
which was not asked for. Restore `rgb(70, 70, 70)` on `Perf Runs (JMeter)` so
every region that rendered before keeps its colour; `Perf Runs (Locust)` stays
aqua, since it is new.
Grey separates from aqua well (dE 22.8 deutan, 25.9 tritan, 26.1 normal), but it
sits at 1.98:1 against the dark-theme surface, below the 3:1 floor, so its region
edges read faint there. Noted in the runbook.
A single "Annotate perf-iac runs" layer matched only `perf-iac`, so a Locust
load window was indistinguishable from a JMeter one. perf-iac now tags every
region with its load driver, so each driver can have its own layer and colour.
- Replace that layer with `Perf Runs (JMeter)` and `Perf Runs (Locust)`, each
matching ["perf-iac", "<driver>"] with matchAny:false, on 12 dashboards.
- job-queue, ledger-data-sync and log-derived-insights had an empty annotations
list and drew no perf regions at all; they now carry the builtIn layer plus
both driver layers.
- Grafana tag matching is a superset AND with no negation, so a generic
`perf-iac` layer also matches every driver region. Keeping one alongside the
driver layers would draw each load window twice, so it is replaced, not kept.
- Document the layers in the telemetry runbook, including two rendering limits:
annotations draw only on timeseries, state-timeline and candlestick panels,
and the shaded fill is 10% opacity so the region edges carry the colour.
- Add `jmeter` to the cspell dictionary; the hook rejects the bare word.
makeTelemetrySetup() rejects a contradictory [telemetry] mutual-TLS
setup by throwing, but it is called from ApplicationImp's
member-initializer list. A try/catch in the constructor body cannot
reach a throw from there, and nothing further up the stack caught it
either, so a config mistake reached std::terminate: the default handler
printed a terminate dump and raised SIGABRT, leaving a core file
instead of a startup error.
Catch std::exception around makeApplication() in run(), report the
reason on stderr and return -1, so the failure is a clean non-zero exit
with a message an operator can act on. Only the construction is
wrapped. setup() starts subsystems whose shutdown order is delicate and
is left outside deliberately, because unwinding a half-started
Application would skip the normal stop sequence.
Gate both validation guards on enabled. A node with telemetry switched
off previously refused to start over certificate paths that nothing
would read.
Document both throws on makeTelemetrySetup(), state in
cfg/xrpld-example.cfg and the configuration reference that a partial
mutual-TLS setup is fatal and that the checks apply only when
enabled=1, and add a runbook troubleshooting entry keyed on the two
error messages.
Tests cover both guards with the message asserted so the two are told
apart, both enabled=0 paths, and the default plaintext configuration.
Node identity reached the OTel resource only as service.instance.id, which is
config-overridable and carries a deployment-chosen label rather than the node's
own identity. Add xrpl.node.id, set unconditionally from the node public key
(base58, TokenType::NodePublic), so traces and metrics share a stable per-node
key independent of [telemetry] service_instance_id.
Set on the tracer resource via Telemetry::setNodeId(), called from
ApplicationImp::setup() once nodeIdentity_ is known, and on the MetricsRegistry
resource via an added start() parameter. The beast::insight meter provider is
built in TelemetryImpl's constructor, before the wallet DB exists, so its
resource cannot carry the value; that path is left for later and the attribute
is omitted rather than stamped blank.
Also drops the transform/spanidentity collector processor added in
4a361a496d: per-node identity belongs on the resource, not copied onto every
span.
Consensus spans share one deterministic, ledger-derived trace_id, so a
single trace holds spans from every node and the resource-level node id is
not a reliable per-span discriminator in stored traces.
Add transform/spanidentity to both collector configs, copying
service.instance.id onto every span as service_instance_id so TraceQL can
filter per node with the same value the $node dashboard variable already
uses on the metrics side. Wired into the traces pipeline locally and into
traces/store (after tail_sampling) on the Grafana Cloud variant.
The transitions panel used increase(...[$__rate_interval]). $__rate_interval is
defined as max($__interval + scrape, 4 * scrape), i.e. deliberately one scrape
longer than the step so rate() windows overlap and lose no counter increase.
That overlap is harmless for rate(), but this panel reads the value as a count
of discrete events, and the overlap counts each event in more than one bucket.
Measured against a log-derived ground truth of 106 syncing transitions on
devnet-otel-usw2-01 over 2026-08-11T11:05Z..2026-08-12T23:04Z, the old query
reported 111.3 at a 300s step and 133.7 at a 60s step -- the error grew to +26%
as you zoomed in, because the overlap is a larger fraction of a smaller step.
Switch to $__interval so the buckets tile exactly, and wrap in round() because
increase() extrapolates to the window edges and so reports fractional counts for
an integer counter. The same measurement now gives 106 at 300s, 105 at 60s and
107 at 900s. Every state and both nodes land within a few counts of truth at any
zoom, and the legend Total is now a meaningful figure.
Pin Min step to 1m: the real scrape interval is 60s while the datasource
declares 15s, so without a floor $__interval can fall below one sample.
Draw as bars with 0 decimals -- the value is a discrete count per bucket, and a
line implies interpolation between counts that does not exist.
