The workload harness gates regressions on histogram_quantile over
job_queued_us / job_running_us, so re-cutting the microsecond ladder changes
what those queries return and the stored baselines no longer describe the
same measurement.
baseline-timings.json's job.acceptLedger.queued.p95 was 96.79us, which is
0.95 / 0.9926 x 100 -- the old 100us bucket edge scaled by the quantile, with
99.3% of samples beneath it. It was never a latency. Keeping it would make the
gate LESS sensitive rather than more: a genuine regression from a real 40us to
90us would still sit under 96.79us + 50% and pass.
Removes the four job.* entries and records why, including their values. The
comparer reports a metric absent from the baseline as "new metric (not in
baseline)" and skips it, so the span baselines stay live and gating continues
for everything unaffected. is_placeholder() still returns False, so this does
not disable the gate wholesale. Recapture the job.* numbers on a node running
the re-cut ladder.
Also corrects _bucket_note in regression-thresholds.json. It described the
spanmetrics ladder as 15 edges starting at 1ms; the collector config has 20,
including five sub-millisecond edges. The note's own reasoning was void too --
it justified the 10ms absolute span bound as "~2 low-end bucket widths", but
the low-end bucket width is 0.01ms, not 5ms. The bound is kept and justified
on the band where span quantiles actually sit, rather than on a derivation
from a ladder that no longer exists.
The microsecond ladder's first edge was 100us, which sat ABOVE the mass of
every instrument using it. Measured on devnet: 99.3% of job_queued_us
samples, 92.5% of job_running_us and 90.4% of getobject_lookup_us fell in
that first bucket. histogram_quantile then interpolated inside bucket 0 and
returned `quantile / fraction_in_bucket_0 x first_edge` -- p75/p95/p99 of
job_queued_us read 75.52/95.66/99.69us against a prediction of
75.53/95.67/99.70. Three-decimal agreement: those panels were reporting
arithmetic on the bucket edge, not latency.
The fix was already half-written. kSubMillisecondBoundaries had been parked
in MetricsRegistry.cpp as [[maybe_unused]] with a comment noting exactly this
problem for nodestore reads. Its edges are now folded into kMicrosecondBuckets
rather than deleted, so the parked intent is carried forward: 1..1000us
resolution where the mass is, upper edges unchanged so multi-second stalls
stay measurable.
Also moves the GetObject count and charge ladders into HistogramBuckets.h, so
all five ladders have one owner and one set of invariant tests (29 now).
Adds check_bucket_parity.py, wired into the existing OTel naming workflow.
The C++ millisecond ladder and the collector's spanmetrics ladder are
specified to agree over their shared range; they were identical when shipped,
then the collector side alone was extended and nothing noticed for eleven
phases. The check asserts containment rather than equality, because jobs
outlive spans -- jobq_updatepaths averages ~60s, which no span approaches, so
demanding equality would force a ceiling that censors it. Verified it rejects
a missing collector edge, a bogus in-range edge, and a return to the 5s
ceiling.
ledger-data-sync's "Job Queue Wait p95 By Type" moves off the beast
jobq_*_q_milliseconds pair onto job_queued_us filtered by job_type. Those
beast metrics are ms-quantised at the source (Event rounds up to a whole
millisecond), so 94-100% of their samples sat in the first bucket and no
ladder change could fix them. Note the label values are camelCase
(job_type="ledgerData"), not the lowercase metric-name fragments.
Both histogram-fed alert thresholds re-validated and left unchanged, with the
measured basis recorded so neither gets tuned against the old artefact: only
0.0022% of job_queued_us samples exceed the 1s threshold, and every edge
bracketing the 1000ms ios_latency threshold survived the ladder change.
Docs: the rpc_size "known issue -- tracked separately" notes in the runbook
and 09-data-collection-reference are now resolved notes, the stale 10-edge
span_duration bucket list is corrected to the collector's real 20, and the
runbook gains a "Reading A Histogram Percentile" section covering both
saturation traps and the expected discontinuity after a ladder change.
This is the change that actually lifts the 5 s ceiling. Until now the
millisecond ladder and the Unit type existed but nothing consumed them.
Telemetry.cpp registered ONE histogram view: instrument name pattern "*",
unit exactly "ms", boundaries {1, 5, ..., 1000, 5000}. Verified against the
installed SDK, "*" matches every name and "ms" matches exactly, so that view
governed every beast::insight Event -- all 54 of them, whatever they measure.
Measured on devnet: 24.9% of rpc_size samples and 100% of jobq_updatepaths
samples fell above 5000. A quantile landing in the `+Inf` bucket reads back
as the second-highest edge, so those p95s reported a flat 5000 rather than a
measurement, and the 1 s to 5 s span was a single four-second-wide bucket
that any quantile inside it had to interpolate across.
Replaces it with one view per unit, keyed on the unit an instrument declares:
- `ms` gets kMillisecondBuckets: every representable edge of the collector's
spanmetrics ladder, plus 60 s and 120 s. The extensions are deliberate --
jobq_updatepaths was measured averaging 59,956 ms, which no span
approaches, so parity alone would still censor it.
- `By` gets kByteBuckets, placed from the measured response distribution
(mean 2131 B, half under 1 kB, tail mean bounded at 7538 B).
OTelEventImpl now derives its declared unit AND its description from unit()
instead of hardcoding "Duration in ms"/"ms", so rpc_size exports as
rpc_size_bytes on the byte ladder. rpc-pathfinding's "RPC Response Size"
panel follows the rename; its unit was already decbytes and is now truthful.
Also corrects Phase7_taskList.md, which still specified the 5000 ladder as
"matching SpanMetrics". That was true when written and became false when the
collector ladder was extended on its own -- implementing the plan as written
reproduced the bug, so the spec is where the defect had come to live. The
edges now have exactly one owner and the plan points at it.
Match the other dashboards: `rgb(15, 122, 102)` instead of the brighter
`rgb(25, 158, 112)`, still clear of the separation floors against the JMeter grey.
Use `rgb(15, 122, 102)` instead of `rgb(25, 158, 112)`: the brighter step drew
too much attention for a background band.
This is the darkest teal that still separates from the JMeter grey by a readable
margin -- normal-vision dE 15.6 against a floor of 15, CVD dE 12.3 against a
target of 8, and at least 3:1 on the dark surface. Dimmer steps fail: rgb(25,
100, 90) lands at dE 9.5, and a grey-derived rgb(25, 70, 70) at dE 5.8, which is
indistinguishable from the JMeter grey even with full colour vision.
Both checks selected on {job="xrpld"}. Loki's OTLP ingestion promotes
service.name to the label `service_name` and keeps a `job` attribute as
structured metadata, which a stream selector cannot match, so the selector
returned zero streams whatever had been ingested. The collector config and
TESTING.md already say to select on `service_name`.
Invert the cross-reference. Picking an arbitrary trace from Tempo and
expecting it in Loki fails even when correlation works, because a log line
carries a trace_id only when emitted inside a sampled span and most spans
log nothing at `warning` level. Start from a logged trace_id instead and
resolve it in Tempo, which is the invariant worth asserting, and try every
id found so one unexported trace does not fail the check.
Bound the log queries in time. Nothing here set start/end, so every query
relied on Loki's one-hour default and returned nothing when re-run later to
investigate a result.
Match the other dashboards: restore `rgb(70, 70, 70)` on `Perf Runs (JMeter)`
so regions that rendered before keep their colour. `Perf Runs (Locust)` stays
aqua.
The driver split changed the existing perf-run regions from grey to violet,
which was not asked for. Restore `rgb(70, 70, 70)` on `Perf Runs (JMeter)` so
every region that rendered before keeps its colour; `Perf Runs (Locust)` stays
aqua, since it is new.
Grey separates from aqua well (dE 22.8 deutan, 25.9 tritan, 26.1 normal), but it
sits at 1.98:1 against the dark-theme surface, below the 3:1 floor, so its region
edges read faint there. Noted in the runbook.
Apply the same two-layer split as the other dashboards to the sync-health board,
which is introduced on this branch: `Perf Runs (JMeter)` and
`Perf Runs (Locust)`, each matching ["perf-iac", "<driver>"] with
matchAny:false, in place of the single generic `perf-iac` layer.
A single "Annotate perf-iac runs" layer matched only `perf-iac`, so a Locust
load window was indistinguishable from a JMeter one. perf-iac now tags every
region with its load driver, so each driver can have its own layer and colour.
- Replace that layer with `Perf Runs (JMeter)` and `Perf Runs (Locust)`, each
matching ["perf-iac", "<driver>"] with matchAny:false, on 12 dashboards.
- job-queue, ledger-data-sync and log-derived-insights had an empty annotations
list and drew no perf regions at all; they now carry the builtIn layer plus
both driver layers.
- Grafana tag matching is a superset AND with no negation, so a generic
`perf-iac` layer also matches every driver region. Keeping one alongside the
driver layers would draw each load window twice, so it is replaced, not kept.
- Document the layers in the telemetry runbook, including two rendering limits:
annotations draw only on timeseries, state-timeline and candlestick panels,
and the shaded fill is 10% opacity so the region edges carry the colour.
- Add `jmeter` to the cspell dictionary; the hook rejects the bare word.
beast::insight instruments are created during ApplicationImp's member-init
list, and opentelemetry-cpp 1.28 never rebinds an already-vended Meter, so an
instrument created before the MeterProvider is published records nothing for
the rest of the process. Observable instruments carry the opposite constraint:
registering one arms the SDK reader thread, and its callbacks run hook handlers
that read services which do not exist that early.
Publish the provider in Telemetry's constructor, ahead of every producer, and
defer only the observables. Collector gains onCollectionReady() and
onCollectionStopping(); OTelCollector arms and disarms its gauges in response.
StatsDCollector starts its polling thread in its own constructor and had the
same hazard, so it uses the pair to gate that thread.
The metrics resource carries service.instance.id and is immutable once built,
so the node public key is resolved in Main.cpp, where a config error can still
be reported, and passed to makeApplication(). getNodeIdentity() remains
authoritative; both paths now share readNodeIdentity(), so telemetry cannot
report a key the node has abandoned.
An explicit ~ApplicationImp stops observing and stops telemetry, covering the
setup() failure paths that never reach run(). Telemetry::stop() is once-only
and no longer clears another instance's global pointer. The histogram view's
meter selector now matches the meter actually in use, so its bucket boundaries
apply for the first time.
Node identity reached the OTel resource only as service.instance.id, which is
config-overridable and carries a deployment-chosen label rather than the node's
own identity. Add xrpl.node.id, set unconditionally from the node public key
(base58, TokenType::NodePublic), so traces and metrics share a stable per-node
key independent of [telemetry] service_instance_id.
Set on the tracer resource via Telemetry::setNodeId(), called from
ApplicationImp::setup() once nodeIdentity_ is known, and on the MetricsRegistry
resource via an added start() parameter. The beast::insight meter provider is
built in TelemetryImpl's constructor, before the wallet DB exists, so its
resource cannot carry the value; that path is left for later and the attribute
is omitted rather than stamped blank.
Also drops the transform/spanidentity collector processor added in
4a361a496d: per-node identity belongs on the resource, not copied onto every
span.
Consensus spans share one deterministic, ledger-derived trace_id, so a
single trace holds spans from every node and the resource-level node id is
not a reliable per-span discriminator in stored traces.
Add transform/spanidentity to both collector configs, copying
service.instance.id onto every span as service_instance_id so TraceQL can
filter per node with the same value the $node dashboard variable already
uses on the metrics side. Wired into the traces pipeline locally and into
traces/store (after tail_sampling) on the Grafana Cloud variant.
The collector bind-mounts one log root, data/logs, and identifies each node by
the subdirectory inside it. The second instance wrote to data2/logs, which is not
mounted at all, so its logs have never reached the pipeline under any name --
while its metrics and traces flowed normally, making it look like a quiet node
rather than an uncollected one. Its own config comment already claimed the
directory was "the same dir the compose stack bind-mounts into the collector",
so the intent was the single root and only the path was wrong.
Both nodes now log under data/logs. The nodestore stays split across data/ and
data2/, which is what those directories are for; the logs gain nothing from the
split because both sit on the same disk, and lose collection entirely.
The installer's log-directory check was wrong in the same way: it verified a path
under the host's /var/log/xrpld, which nothing reads -- that path exists only
inside the container, where the root is mounted. It now checks the directory the
collector actually reads, and the docs no longer ask for host symlinks that have
no effect.
Also drop the trailing slash from the data ignore rules and add data2. A
trailing-slash pattern matches only real directories, so on a host that follows
the runbook and symlinks both to a fast disk, neither was ignored -- leaving them
permanently untracked in a tree the runbook says should read clean, and one
`git add` away from committing a nodestore.
A node's service_instance_id should say which box the data came from, so a
dashboard can tell two otherwise-identical instances apart. A machine name is
also exactly what this public repository should not carry, and the value cannot
come from the environment: it is read only from the [telemetry] section, and the
config parser has no include directive.
So the tracked configs keep a generic identity and name no host, and the
installer renders each into a .host.cfg beside it with the id substituted from
.env.devbox. The units run the rendered copies, which are gitignored. The
tracked configs are never edited on the host, so an update cannot conflict and a
rebuild loses nothing.
The identity is substituted in two places from one value -- the setting and the
log directory name -- because they have to agree: the collector derives identity
for the logs pipeline from the log path, so a mismatch costs that node's logs
their service_instance_id label while its metrics keep theirs, which reads as
"no logs" rather than as a misconfiguration. The installer counts the
occurrences it expects to replace and verifies the result, so a config reshuffle
fails loudly instead of yielding a copy that quietly kept the generic identity;
on the dashboards that would look like the node had disappeared. It also rejects
an id that is not a safe directory name, and rejects two nodes sharing one id.
Also record what the last deployment needed and the runbook did not say: the
session-bus variable a rootless container runtime needs before its user units
will install, the link from the unit's expected binary path to the preset's
build directory, and that a config merge which redeclares service.extensions
drops the cloud authenticator and stops the collector exporting anything at all.
These files also had not been through the formatting hooks, which want 4-space
indentation and reflowed tables; that is fixed here too.
The collector exits at startup whenever the Grafana Cloud stack is brought up:
cannot start pipelines: failed to start "otlphttp/grafanacloud" exporter:
failed to resolve authenticator "basicauth/grafanacloud": authenticator not found
The developer stack passes otel-collector-filestorage.yaml as a second --config to
persist filelog read offsets. The collector replaces lists on merge rather than
appending them, which that file notes, and it repeats the BASE config's
service.extensions -- [health_check, file_storage/filelog]. Layer the cloud
overlay on top and the base config is swapped for the cloud one, whose list also
carries basicauth/grafanacloud; the storage overlay still comes last, so the
authenticator is dropped and the exporter that needs it cannot start.
Nothing degrades: the collector refuses to start at all, so a host brought up this
way exports no telemetry anywhere, local or cloud.
Adds a cloud variant of the overlay naming all three extensions, mounted by
docker-compose.grafanacloud.yaml over the base one so the command line is
unchanged. The base variant stays as it is for stacks that do not use the cloud
exporter.
Following the previous version on a freshly provisioned host failed three times,
each in a way that pointed away from the actual cause. Replacing the build section
with what actually works, and recording why each wrong turn misleads.
The daemon is not built by default: the Conan recipe defaults its xrpld option to
off. Omit the option and both Conan and CMake report success, then the build fails
with "No rule to make target 'xrpld'" because the target was never created.
Configure through the Conan-generated preset rather than a hand-written toolchain
path. The toolchain does not sit where the single- versus multi-config layouts
suggest, and a wrong path fails with "Could not find toolchain file" followed by
"CMAKE_CXX_COMPILER not set", which reads as a broken compiler.
The build directory is .build/build/Release, not .build. Building the wrong one
reports "Generator: execution of make failed", which reads as a toolchain problem.
The CMake cache is sticky: re-running conan install with a changed option does not
update an already-cached CMake variable, so the target stays absent even though
the option was accepted. Added the check and the cache-clearing steps.
Standing up and updating one of these hosts involves a handful of steps that fail
quietly when missed, and the knowledge was living in people's heads and in an
untracked file on the host itself. That file died with the last rebuild.
Covers updating the checkout, re-applying the host overlays a checkout can
disturb, building, and starting the collector and both nodes, then verifying all
three signals actually arrive rather than trusting the configuration.
The failure modes it calls out are ones already hit here: bringing the collector
up without the cloud overlay, which exports nothing and logs no error; omitting
--force-recreate, so a config change appears applied but is not; a log directory
whose name does not match the instance id, which leaves logs unlabelled while
metrics stay labelled; and reading the container command line to check which
config is active, which cannot distinguish the two cases because the overlay
mounts over the same path.
It also records two signals that read backwards: a higher acquisition timeout
count can mean a node is retrying forward more cheaply rather than stalling, and
outbound byte counters are recorded at the decision to send rather than at the
wire, so under a flood they overstate egress by orders of magnitude.
Deliberately names no credential or environment files and carries no host
identity, so it is safe in a public tree. Which file holds what is documented
where it is needed, next to the installer that reads it.
Running two nodes on one host is how the storage backends get compared with
everything else equal, but only the first instance's config was ever committed.
The second node's config, both systemd units, and the collector's per-node log
identity lived on the host alone and were lost when it was rebuilt.
Adds xrpld-telemetry-mainnet2.cfg, derived from the first and differing only
where two nodes on one host must differ: ports, data and log paths,
service_instance_id, and [node_db] type. Ports continue the offset-by-ten scheme
already in use -- devnet on 5005, Mainnet on 5015, so this one on 5025 -- so all
three can bind together.
Adds the units as templates plus install-units.sh, which fills them from an
untracked .env.devbox. This repository is public and f64f4b35e7 already removed a
personal home directory from shipped config, so the run user and checkout path
must not come back into git. The installer refuses an env file that is not mode
600 and refuses to install a unit still holding an unsubstituted placeholder, so
a half-configured host fails loudly rather than producing a unit systemd never
starts.
Both units carry RequiresMountsFor for the data mount. That mount's fstab entry
normally uses 'nofail' so a missing disk does not block boot, which also means it
fails silently; without the guard a node starts anyway and writes its nodestore
to the root filesystem until it fills.
Re-adds the collector's per-node log identity: include_file_path plus operators
lifting the log directory basename into service.instance.id. Without it the logs
pipeline carries no service_instance_id label while the metrics pipeline does, so
the dashboards' $node filter matches nothing for logs and reads as "no logs"
rather than as a misconfiguration.
Renames the first instance's log directory from data/logs/mainnet to
data/logs/xrpld-mainnet, the one change to that file: the derivation above reads
the basename, so it has to equal the service_instance_id for a node's logs and
metrics to carry the same label.
No credentials or host-identifying values here. .env.devbox and .env.grafanacloud
are both covered by the .env.* ignore rule; only the examples are tracked.
The transitions panel used increase(...[$__rate_interval]). $__rate_interval is
defined as max($__interval + scrape, 4 * scrape), i.e. deliberately one scrape
longer than the step so rate() windows overlap and lose no counter increase.
That overlap is harmless for rate(), but this panel reads the value as a count
of discrete events, and the overlap counts each event in more than one bucket.
Measured against a log-derived ground truth of 106 syncing transitions on
devnet-otel-usw2-01 over 2026-08-11T11:05Z..2026-08-12T23:04Z, the old query
reported 111.3 at a 300s step and 133.7 at a 60s step -- the error grew to +26%
as you zoomed in, because the overlap is a larger fraction of a smaller step.
Switch to $__interval so the buckets tile exactly, and wrap in round() because
increase() extrapolates to the window edges and so reports fractional counts for
an integer counter. The same measurement now gives 106 at 300s, 105 at 60s and
107 at 900s. Every state and both nodes land within a few counts of truth at any
zoom, and the legend Total is now a meaningful figure.
Pin Min step to 1m: the real scrape interval is 60s while the datasource
declares 15s, so without a floor $__interval can fall below one sample.
Draw as bars with 0 decimals -- the value is a discrete count per bucket, and a
line implies interpolation between counts that does not exist.
Nine conflicts, resolved as follows.
src/xrpld/app/ledger/detail/InboundLedger.cpp -- kept this branch's version.
phase10 sets the span's outcome/timeouts/peer_count attributes inline at each
exit; this branch replaced that with the idempotent finalizeAcquireSpan(), called
on all four exits (init, done, give-up, destructor). Taking phase10's blocks
would have set the outcome twice against a helper documented as not overwriting
what the real exit recorded. phase10's comment explains why peer_count must not
be read in a destructor; the helper solves that structurally by taking
std::optional<std::size_t> and being passed std::nullopt from there.
src/xrpld/telemetry/MetricsRegistry.cpp -- kept metric::ledgerEconomy over
phase10's "ledger_economy" literal. This branch added the naming check that
requires constants for converted families, so the literal would regress it. Took
phase10's comment cleanup.
src/xrpld/telemetry/MetricsRegistry.h -- kept registerRotationStateGauge(), which
only exists here, and took phase10's removal of the stale task-number comment.
validate_telemetry.py -- combined both. phase10 replaced serial metric polling
with a concurrent fan-out on one shared deadline, because 58 metrics x 45 s of
additive timeout overran the CI budget; that is kept. Its target list filters on
SKIPPED_METRIC_GROUPS rather than the two literals it hardcoded, so the
sync_diagnostics group stays owned by assert_sync_diagnostics_metrics() instead
of being polled and reported twice. Both SYNC_DIAGNOSTICS_GROUP and
METRIC_POLL_CONCURRENCY are needed and both are kept.
check_otel_naming.py -- both sides extend the rule docstring. Took phase10's
fuller Rule E text (doc discovery, allow-dotted markers) and re-appended rules
I/J/K/L, which exist only here.
expected_metrics.json -- the two sides add disjoint sibling groups, so both are
kept: sync_diagnostics alongside node_health_gauges, overlay_reduce_relay,
overlay_overflow, validation_lifetime_counters and not_asserted. Both dashboard
uids are kept, giving 16 asserted uids against 16 dashboards on disk.
expected_spans.json -- kept this branch's span set, a superset that adds the
acquire phase spans, ledger.serve, txset.acquire and peer.dial, and expands
ledger.acquire's required attributes. Took phase10's description, which documents
what the totals mean, and its note on how the RPC wildcard span is created.
total_span_types and total_unique_attributes are recomputed for the union: 48 and
74, since each side's figure counted only its own spans.
Docs: took phase10's more accurate wording on what the dashboard check actually
covers, and corrected the dashboard count from 15 to 16 where the merge made it
stale.
Verified: no conflict markers remain, both JSON contracts parse, both Python
files compile, asserted dashboard uids match the dashboards on disk exactly, and
the OTel naming check reports all layers consistent.
The Operating Mode Transitions panel queried state_accounting_*_transitions
directly. Those are monotonic counters, so the panel drew a slowly rising line
and a few transitions per hour were invisible against a total in the hundreds.
It also fell off a cliff whenever xrpld restarted and the counters reset to 0,
which reads as missing data rather than a restart.
Wrap each target in increase(...[$__rate_interval]) so each point is the number
of transitions in that bucket and the series survives a counter reset. This is
what the sibling panels on the same row (Operating Mode (Time Share), State
Duration Rate) already do.
Verified against devnet-otel-usw2-01/02 over 2026-08-11T11:01Z..2026-08-12T16:23Z:
the fixed expression reports 107 and 123 syncing transitions, matching the
counter deltas, and stays continuous across the 12:07 restart where the raw
counter dropped 630 -> 1.
Brief mode flaps remain invisible on Operating Mode (State Timeline) because a
~2 s dwell cannot be captured by a 60 s scrape; this panel is the place to read
them.
This branch had already made the same corrections independently, and in
richer form, so the resolution keeps this branch's version nearly throughout:
- 09-data-collection-reference.md: this branch already documents the
state-accounting gauges as cumulative **microseconds** with an explanatory
callout, and already names `jobq_job_count` with its `jobq` group. Kept.
- telemetry-runbook.md: already carries `jobq_job_count` in both tables. Kept,
along with this branch's larger additions.
- OpenTelemetryPlan.md: kept this branch's rewritten section 9 blurb, which
describes the inventory without hardcoding counts and so cannot drift.
- consensus-health.json: kept this branch's rewrite. It deliberately removed
the four TraceQL close-time detail panels and renamed the agreement panel;
the incoming side would have resurrected them. Panel count unchanged at 26.
- integration-test.sh: this branch's unprefixed native metric names were kept,
but it still asserted `job_count`, so the `jobq_job_count` correction was
carried over. That check would otherwise always fail.
