Resolved nine dashboards. This branch re-exported all of them, so the upstream
unit changes land in lines this branch had already rewritten. Took this
branch's files from the clean baseline merge, then applied si: and decimals 0
to every count-unit panel left in them. Every panel set matches that baseline
merge exactly.
The three si:drops panels keep no decimals setting, as before.
Resolved consensus-health.json. This branch re-exported that dashboard, so
phase-6's unit changes land in lines this branch had already rewritten. Took
this branch's file from the clean baseline merge, then applied si: and
decimals 0 to every count-unit panel left in it. The panel set matches that
baseline merge exactly.
Panels whose unit counts whole things switch from a custom `suffix:` unit to
`si:`, so a large value reads 15 k nodes/s instead of 15000 nodes/s. A custom
suffix unit is formatted by toFixedUnit, which never scales; si: goes through
SIPrefix and does.
decimals is set to 0 on the same panels, because a fraction of a ledger or a
node says nothing. The two settings need each other: on its own decimals 0
would round a slow rate down to zero, and si: shifts it to the milli step
instead.
Left alone: the one panel whose unit is a seconds-per-second ratio.
42 panels across 10 dashboards, plus 5 name placements and one title.
Panels whose unit counts whole things switch from a custom `suffix:` unit to
`si:`, so a large value reads 15 k nodes/s instead of 15000 nodes/s. A custom
suffix unit is formatted by toFixedUnit, which never scales; si: goes through
SIPrefix and does.
decimals is set to 0 on the same panels, because a fraction of a ledger or a
node says nothing. The two settings need each other: on its own decimals 0
would round a slow rate down to zero, and si: shifts it to the milli step
instead.
Left alone: the one panel whose unit is a seconds-per-second ratio.
26 panels across 9 dashboards, plus 3 legends and one name placement.
Panels whose unit counts whole things switch from a custom `suffix:` unit to
`si:`, so a large value reads 15 k nodes/s instead of 15000 nodes/s. A custom
suffix unit is formatted by toFixedUnit, which never scales; si: goes through
SIPrefix and does.
decimals is set to 0 on the same panels, because a fraction of a ledger or a
node says nothing. The two settings need each other: on its own decimals 0
would round a slow rate down to zero, and si: shifts it to the milli step
instead.
Left alone: the one panel whose unit is a seconds-per-second ratio.
20 panels across 5 dashboards.
Resolved network-traffic.json. Both branches own a version of the Peer
Disconnects legend: phase-7 had the older "[instance branch role work_item]"
identity form, and this branch rewrote the same line to use xrpl_ident. Kept
this branch's line, then applied the same change to it, so the legend drops
the series token and the query drops the label_replace that set it.
The incoming side's only edit to that file was those two lines, so nothing
else was superseded.
A panel legend drops ${__field.labels.series} where the panel title already
carries that name, and the query drops the label_replace that set the label,
since nothing reads it any more.
Horizontal bar gauges put the series name above the bar.
40 legends across 5 dashboards, 2 bar gauges to the LCD style, 5 name placements.
A panel legend drops ${__field.labels.series} where the panel title already
carries that name, and the query drops the label_replace that set the label,
since nothing reads it any more.
Horizontal bar gauges put the series name above the bar.
3 legends across 2 dashboards, and one bar gauge name placement.
Resolved four dashboards. This branch re-exported consensus-health, node-health,
rpc-performance and transaction-overview, so both sides touched the same option
blocks: upstream added the annotations block where this branch's export already
carried it set to false. Rebuilt each file from the clean pre-change merge of the
two branches, then set the option on every panel. Panel sets match that baseline
merge exactly, so the panels this branch's export drops are unchanged by the
resolution.
All 189 timeseries and state-timeline panels on this branch now set multiLane,
and all four state timelines set perPage.
Resolved three dashboards. phase-7 rewrote consensus-health, rpc-performance
and transaction-overview, so both sides added the annotations block next to
lines that had already diverged. Rebuilt each file from the clean pre-change
merge of the two branches, then set the option on every panel, which is what
both sides intended. Panel sets match that baseline merge exactly.
All 107 timeseries and state-timeline panels on this branch now set multiLane,
and both state timelines set perPage.
Every timeseries and state-timeline panel sets
options.annotations.multiLane. The state-timeline panels set
options.perPage to 5.
99 panels across 13 dashboards. Seven of those already carried the option from
this branch's node-health export, so they flip here and not upstream.
Every timeseries and state-timeline panel sets
options.annotations.multiLane. The state-timeline panels set
options.perPage to 5.
62 panels across 5 dashboards.
Every timeseries and state-timeline panel sets
options.annotations.multiLane. The state-timeline panels set
options.perPage to 5.
46 panels across 5 dashboards.
The list mirrors the collector's spanmetrics dimensions, and that dimension is
gone. The reduce-relay and squelch counters keep their own names, which are a
different thing.
Alloy and the Grafana Cloud collector still declared a spanmetrics dimension
for an attribute tx.receive no longer sets, which only widens the label set.
The runbook's drop-path diagram still listed two tx_status values that no
longer exist, because the paths that set them now run before the span starts.
transaction-overview.json conflicted only because this branch reordered every
panel, so the diff could not line the two sides up. The incoming side changed
one panel and nothing else, so this keeps every panel and rewrite this branch
made and re-applies that single change in this branch's own style: the receive
panel drops the suppressed grouping, names its series like the other
single-series panels, and says where the duplicate count now lives.
Resolved transaction-overview.json by composing both sides: this branch's
renamed span_calls_total metric and its 15s interval, with phase-6's removal of
the suppressed grouping and its new legend.
tx.receive is created after the duplicate check, so it never carries a
suppressed attribute. The Tempo tag filter and the dashboard panel still
referenced it, and check_otel_naming rules C and D fail on a reference with no
constant behind it.
The panel keeps its remaining signal as a plain receive rate. How many relayed
copies were dropped is the transactions_duplicate traffic category, which does
not depend on a span.
The spanmetrics dimension goes too; a dimension for an attribute nothing sets
only widens the label set.
tx.receive no longer carries a suppressed attribute: the span is created only
once the node has decided to process the transaction, so there is no dropped
copy for the attribute to describe.
The validator fails a span that is missing a required attribute, so leaving it
listed turns the telemetry-validation leg red.
The previous commit switched the generated node config from [ips] to
[ips_fixed] on the grounds that the variable, the comment and the sibling cfg
template all named ips_fixed, and that ips_fixed is the section whose
documented meaning fits a private cluster. Both of those are still true. The
switch is reverted anyway, because it is a workload change rather than a
naming fix.
Measured on CI, parent commit against this branch's previous tip, one
functional config line apart:
span.consensus.ledger_close.p95 0.57 ms -> 6.43 ms (tripped the gate)
span.consensus.ledger_close.p99 0.94 ms -> 9.50 ms
span.consensus.accept.p50 0.97 ms -> 2.63 ms
span.tx.process.p50 0.36 ms -> 0.18 ms (faster)
job.acceptLedger.running.p95 21157 us -> 10938 us (faster)
Every consensus-path span rose and every transaction-path metric fell, which
is the shape a denser always-connected mesh produces and not the shape of
run-to-run variance. [ips_fixed] holds connections open to all four peers
instead of treating the list as a discovery hint, so each node processes
proposals and validations from the full mesh every round. Nothing else in that
commit touches the consensus path: the emitted config differed in exactly
three lines, of which one is a die message and one expands to an identical
string.
The committed baseline describes the [ips] topology. Adopting [ips_fixed]
therefore needs a refreshed baseline and re-derived bounds, which is the
process baselines/README.md already documents for a workload change. Left as
its own work item rather than smuggled in behind a section rename, and the
reason is now recorded beside the line so it is not repeated.
This also falsified a claim the previous commit had written into
baselines/README.md and regression-thresholds.json: that none of the six
weakly-guarded keys fires on any observed run. Corrected in both, and the
measurement above is cited in place of the absolute.
The harness killed and probed node directories named `node<N>`, but the
directories it creates are `validator-<N>` in run-full-validation.sh and
`bench-node-<N>` in benchmark.sh. Verified with pgrep against processes whose
command lines mimic the real ones: the pattern matched nothing either script
produces. Three consequences, all live:
- `--cleanup` deleted the workdir and left the xrpld processes running. They
are host processes, so the compose teardown does not reach them.
- The pre-run cleanup could not free the previous run's RPC, WS and peer
ports, which surfaces much later as a cluster that never reaches consensus.
- The startup crash fast-fail read a pid path that never exists, so its
`stopped > 0` branch was unreachable and a dead node waited out the full
120-attempt window.
Rather than patch four literals, derive every node path, kill pattern and log
glob from one NODE_PREFIX per script. The directory name is also the node's
identity: the collector's file_log receiver lifts that segment into
service.instance.id, so the directory and the [telemetry] service_instance_id
must agree. Deriving both from one value is what stops them drifting again.
Also in the same files, each confirmed by test rather than inspection:
- The collector readiness probe could never fail. curl -w '%{http_code}'
prints 000 on a refused connection and then exits non-zero, so the
`|| echo 000` inside the substitution appended a second 000 and the
"not ready" comparison never matched. Move the fallback outside.
- The generated config wrote [ips], the starter-list section. A loopback mesh
that must reach quorum is the [ips_fixed] case, which is what the variable,
the comment and the sibling cfg template already said.
- benchmark.sh returned exit 1 for a row it could not measure, though the
exit-code table reserves 1 for "every metric was measured and one breached".
Report 2 there instead.
- Five bc computations fell back to 0, which clears every threshold. The
guards beside them already fall back to the inconclusive token; these now
do too.
- A comment claimed a `|| guard` after a heredoc lands in the heredoc, and
that claim had removed a real guard from the config write. It does not: the
guard runs, and fires when cat fails.
- The EXIT trap was installed 88 lines before stop_workload was defined. If it
fired in that window, errexit aborted the handler on "command not found" and
the cluster reap never ran. Install it below both handlers.
- jq exits 5 on malformed JSON, outside this script's documented codes, so
read_metric now routes that through cannot_measure.
- --nodes and --duration were unvalidated, and --nodes 0 made the pid-count
guard compare 0 with 0 and pass, handing the sampler no pids at all.
- --cleanup now passes -v so the named tempo-data volume goes with it.
Otherwise the next run's Tempo still serves the previous run's traces and a
span assertion can be satisfied by them.
- Five messages reported an attempt count as seconds, though each attempt is
a sleep plus every node's probe.
The baselines README and the two regression JSON files had gone stale when the
baseline was refreshed to a three-run median: they described 20 gated keys and
five exclusions, against an actual 19 and six, and cited the superseded run,
date and commit. Re-derive every affected figure from the committed files. The
detection floors are recomputed (2.00x to 7.41x, so a 10x regression is now
caught on all 19 keys), the newly excluded span.ledger.build.p99 is documented,
and figures that no committed artifact can verify are either replaced with
derivable ones or labelled with their numerator.
No baseline value, threshold bound or derivation entry changes.
MetricsRegistry did two jobs. It owned the OTel metrics pipeline, and it
registered the observable gauges whose callbacks read live application
services. The second job is what made the whole class xrpld-tier, so the
pipeline's lifecycle -- the recording() gate and the stop() teardown that
closes a use-after-free window -- could not be unit-tested in xrpl_tests.
Split it in two:
- xrpl::telemetry::MetricsRegistry (libxrpl) owns the exporter, provider,
meter, the 16 synchronous instruments, recording(), stop(), and the
record*/increment* methods.
- xrpl::telemetry::AppMetricGauges (xrpld) owns the 19 observable gauges
and their callbacks, holding a reference to the core and to the
ServiceRegistry.
MetricMacros.h and ValidationTracker move with the core. The macros need
only recording() and meter(), both core members; the core holds a tracker
by value, and a libxrpl header cannot include one from src/.
ApplicationImp owns both objects and sequences them. The core is built in
the member-init list, so every synchronous instrument exists before any
subsystem can record one. The gauges are armed once overlay_ exists, the
last service their callbacks read. Shutdown detaches the gauge callbacks
before the core drops the provider, and each shutdown step is isolated so
a failure in one cannot skip the others.
That detach call is new. detachCallbacks() had no callers, and the flag it
sets is read by the gauge callbacks but can no longer be written by the
core, so the caller now has to make the ordering explicit.
The telemetry module links xrpl.libxrpl.core and xrpl.libxrpl.protocol
PUBLIC: ValidationTracker.h takes a LedgerIndex and MetricMacros.h takes a
ServiceRegistry, both in interfaces a consumer compiles against.
Adds a MetricsRegistry gtest that drives an enabled core with telemetry on
and pins the recording() gate, stop() leaving the registry inert, and
stop() being idempotent. The libxrpl test tree no longer depends on
xrpld.telemetry at all, and the two CMake workarounds that compiled xrpld
sources into xrpl_tests are gone.
Documentation and dashboard source links follow the code to their new
paths, split between the two classes by which one now defines each metric.
The rule watched state_accounting_full_transitions > 3 per hour, so a node
that flaps once (one full -> syncing -> full round, e.g. per online-delete
rotation) never tripped it. Lower the threshold to > 0 so a single re-entry
into FULL, past the one-hour uptime gate, alerts.
Keep the state_accounting_full_transitions metric: it is a cumulative gauge
every node always reports, so increase() yields a real series (0 when
healthy) and the rule never evaluates to NoData. A sparse counter would
raise a false DatasourceNoData on a healthy node. Set noDataState: OK so a
scrape gap cannot page either.
Merges pratik/otel-phase9-metric-gap-fill into
pratik/otel-phase10-workload-validation.
Auto-merged. Carries the weak_ptr gauges_ change into OTelCollector, the
Test 1 standalone-store fix in TESTING.md, and the collection-lifecycle
calls in the StatsD test that phase-7 requires.
Merges pratik/otel-phase8-log-correlation into pratik/otel-phase9-metric-gap-fill.
Auto-merged. TESTING.md picked up phase-8's Test 4 rewrite alongside this
branch's Step 2 standalone-store fix. OTelCollector.cpp gained the weak_ptr
gauges_ list; the StatsD test gained its onCollectionReady() calls.
Merges pratik/otel-phase7-native-metrics into pratik/otel-phase8-log-correlation.
Conflict was one TESTING.md hunk under "Nodes not reaching proposing state":
this branch renamed the node directories to Node-N in integration-test.sh,
phase-7 kept nodeN and expanded the [peer_private] explanation. Resolution
keeps this branch's Node-1 path (its own script uses that naming) and
phase-7's fuller prose citing peerfinder/Config.cpp.
Non-conflicting phase-7 changes come through: OTelCollector's gauges_ list
becomes weak_ptr, matching the earlier hooks_ change; the phase-6 revert of
the StatsD-test onCollectionReady() calls resolved against phase-7's version
that keeps them.
Merges pratik/otel-phase6-statsd into pratik/otel-phase7-native-metrics.
Phase-6 dropped the three onCollectionReady() calls that had been added to
its StatsD test, because that method is only declared here on phase-7.
This branch's own copy of the file was unchanged from the merge base, so
the default merge would have silently deleted the calls from here too —
where they are needed, because this branch gates polling behind
onCollectionReady() in OTelCollectorImp::onTimer.
Resolution keeps both sides: phase-6's two new include lines
(Counter.h, Gauge.h) and phase-7's three onCollectionReady() calls plus
their doxygen and inline explanations. The merged file is exactly
phase-7's tip plus those two includes.
TESTING.md auto-merged cleanly; both sides added text under Test 1 in
different regions.
Three conflicts, all from both branches editing the same passage:
- Main.cpp: kept phase-9's wording. The metrics registry only exists here, so
"unwinding destroys little: the metrics registry, whose destructor joins its
export thread" is the true statement on this branch.
- TESTING.md: kept both paragraphs. They document different things (the
private [network_id], and the log path plus log_level).
- 05-configuration-reference.md: composed both. The identity is now resolved
before construction and never empty, so every producer stamps the node key
from the start; the only divergence left is a wallet that already holds a
different key, which corrects the tracer alone. Rewrote the earlier
"three producers" blockquote too: its "no fallback", "first boot ... left
off" and "Known issue" claims are what this change removes.
One silent break the merge could not flag: makeMetricsRegistryOptions() took
the std::optional<std::string> node key that used to be a constructor
parameter, and that parameter is now the resolved keypair. It takes the base58
string directly, and the constructor derives it from nodeIdentity_, which is
declared before both telemetry_ and metricsRegistry_.
Test 1 pointed standalone at docker/telemetry/xrpld-telemetry.cfg, whose
[node_db], [database_path] and [debug_logfile] all resolve under
docker/telemetry/data. Standalone builds its own private chain, so that
left one NuDB holding two unrelated chains. This file already states the
rule for the key generation node in Test 2, and the sibling mainnet config
keeps its store under data/mainnet/ for the same reason.
Derive a standalone config with the three paths redirected under
data/standalone/, and run from that.
Note why the flag is not the problem: --start selects StartUpType::Fresh,
but the default Normal reaches startGenesisLedger() through the same branch
chain in ApplicationImp::setup, so any standalone run writes a genesis
ledger into whichever store the config names.
The collector readiness note claimed docker-compose.yml publishes only 4317,
4318 and 8889 and that 13133 comes from a workload stack. It publishes 13133,
and that stack is not part of this branch. Probe health_check on 13133 and drop
the note; the troubleshooting entry now points at the same check instead of
carrying a second, weaker copy.
Stop restating BUILD.md. The hardcoded conan and cmake lines had drifted from
it, -Dtelemetry=ON is redundant because the Conan toolchain carries it, and the
conan-release preset resolves only from the repo root, builds into
.build/build/Release rather than .build, and sets no -Dxrpld=ON. Defer to
BUILD.md and docs/build/telemetry.md.
Test 2's keygen step reused the Devnet config with -a --start, which wrote a
genesis chain into the Devnet store, took RPC port 5005 from node 1, and was
followed by an rm -rf that also destroyed the mainnet node's store and every
log. Give it its own config under the test's temp root, as the script does.
The manual path also needs XRPLD_LOG_DIR, or the collector tails the wrong root
and Test 3 finds nothing without erroring.
Neither the template nor the script set [network_id], so a local cluster
stamped xrpl.network.type=mainnet and shared dashboard series with real mainnet
data. Set a private id in both, and say which label it produces. Also drop a
duplicate metrics_endpoint from the generated config.
Split the consensus trigger row: six families fire on a standalone
ledger_accept, and the remaining seven need the establish phase, a validator
key, or a peer. ledger.validate needs peers too, because checkAccept is
unreachable in standalone. Correct the trace-id note to 16 bytes, and name the
strategy it depends on.
The pathfinding bullet said raw account values reach Grafana Cloud. Both
accounts are already tokens when they leave the node; what differs is that the
base config hashes them a second time, so one account carries two tokens across
configs and traces must not be joined across them.
Also: the Loki allow-list is a fixed 18 keys on the pinned image with k8s and
cloud enumerated rather than wildcarded, the runbook documents 9 of 15
dashboards, and the spanmetrics block now uses one spelling with a note that
the cloud config uses the other.
The config template wrote each node's log to a lowercase node{N} directory
while setting service_instance_id=Node-{N}. The collector takes the node name
from the log file's parent directory and stamps it as the Loki
service_instance_id label, so the logs carried a name no trace or metric
shared and nothing joined. Use Node-{N} and state the rule.
The template also set log_level to warning. Nothing in the pipeline filters on
severity; the constraint is that a log line carries trace context only when it
is emitted inside an active span. At warning the only such statements in the
consensus accept span are a catch path a healthy round never takes and a
periodic censorship warning. At info the CNF Val / CNF buildLCL pair writes one
line per accepted ledger, which is what makes this test's Step 1 findable.
Grafana 13 offers the link per span, labelled "Logs for this span", in the
span's Links row — not per trace. Fix the step and the expected-results row.
The example log line quoted a message that does not exist. The real in-span
RPC statement logs at debug, so the severity code is DBG; say which line to
look for under each test, since Test 2 now logs at info.
Drop the reference to workload/validate_telemetry.py: that file is not part of
this branch, and its instant-endpoint call uses seconds, so the nanoseconds
claim applied only to query_range.
On the OTel path only [insight] server is load-bearing. CollectorManager reads
endpoint and hands it to OTelCollector, which logs it at startup and routes
nothing with it; the real export endpoint is [telemetry] metrics_endpoint,
which the template already sets. service_instance_id and service_name in that
section are read and discarded.
Leaving the line invited an operator to reconcile a mismatch that has no
effect. integration-test.sh already emits only server=otel with the same
explanation, so the two now agree.
The Payment destination was not a valid XRPL address — its base58 checksum
does not match — so Test 1 Step 4 and Test 2 Step 7 could never have returned
the tesSUCCESS they claim. Use a valid one and note that the destination does
not need to exist.
The Tempo search loop had no -G, so curl posted the parameters as a body,
Tempo answered 200 while ignoring the query, and every span name came back
non-zero. It also had no time bound, and Tempo keeps blocks for an hour, so a
re-run was answered by the previous run's traces. Add -G, RUN_START, and
start/end, matching what integration-test.sh already does.
Split the query list in two: 35 names that should be present, and 8 that need
a trigger neither test performs, where zero is the expected answer. Previously
two of the latter sat in the pass/fail list and read as failures.
Correct the standalone span table. consensus.mode_change fires once per round
start whether or not the mode changes, ledger.validate cannot fire because
checkAccept is unreachable in standalone, and the apply-stage, TxQ and ledger
families were missing rows. Give each "No" row the reason that actually
applies: the establish phase, a missing validator key, or no peers.
Also: ledger_accept is not required before submit, the teardown pgrep matched
more than this node, [peer_private] also disables the inbound listener, and
the 15-second wait covers Tempo but not Prometheus.
The committed baseline was captured 2026-08-26, before the account-funding
race was detectable. Phases whose funding silently failed submitted no
transactions, so the capture recorded artificially low ledger and transaction
timings, and job.transaction.queued.p95 and job.transaction.running.p95 could
not be captured at all. Once funding worked, span.ledger.build.p99 read
29.00 ms against a 9.11 ms baseline and turned the gate red on a run whose
200 span and metric checks all passed.
Refresh every value to the median of CI runs 34495527952, 34505215266 and
34507425933, the first three with the fix in place, and re-derive each
absolute bound as hi_next - baseline from that median.
Exclude span.ledger.build.p99. Across those three runs it read 29.00, 7.06
and 8.94 ms, a 4.11x spread whose maximum is 1.16x its 25 ms trip point, so a
healthy run reddens CI. Widening cannot fix it: a bound tolerating 29.00 ms
would reach into the bucket above and restore the single-crossing false
positive the derivation rule removes. span.ledger.build.p95 stays gated at
0.48 of its trip point, so ledger construction keeps coverage.
The other 19 keys sit between 0.17 and 0.76 of their trip points.
span.tx.process.p95 is the tightest and is the first to re-measure if the gate
reddens again.
Repoint one bounds-checker test at span.ledger.build.p95, since it mutated the
p99 override this commit removes.
Account setup submitted the funding Payments, slept a flat 10 seconds, then
read each sequence once. The txq-burst and mixed-peak phases escalate the
open-ledger fee on purpose, so the funding transactions were queued, every
account read Sequence 0, and the phase aborted with "only 0 of 8 created
accounts were funded". The run then reddened on a workload gate rather than on
anything telemetry had done.
Poll the ledger until each account has a sequence, with a deadline, so a late
confirmation is still seen and a healthy cluster pays no waiting cost. Pay a
multiple of the current open-ledger fee, so funding is not queued behind the
load a phase creates deliberately. terQUEUED no longer marks an account funded:
only a ledger read does.
Retry the accounts that never confirmed, once, after re-reading the genesis
sequence from the ledger. consumes_sequence advances the local counter on
terQUEUED, so a dropped funding transaction leaves it ahead of the ledger and
every resubmit would otherwise land on a future sequence.
The funding wait can run twice, so raise the orchestrator's grace above twice
the timeout. A test pins that relationship, since the two constants live in
different files.
Also save each generator's full stdout and stderr beside its JSON report. Only
the last 200 characters of stderr reached the phase error and stdout was
dropped, so none of the per-account funding results appeared in CI.
The `service_name`, not `job` note pointed at three file:line locations.
All three had drifted, because the cited files move on every merge forward
and nothing checks the references. Name the `resource/logs` processor and
the `loki` service instead — those survive line drift and a rename breaks
a grep loudly.
