Resolved five files.
PerfLogImp.cpp: PerfLog now takes its method names as string_view, so the
methods map stores the map key rather than a char const*. Kept this branch's
scoped lock so the OTel recording still runs after methodsMutex is released,
and kept both the NetworkOPs and metric-macro includes.
PeerImp.h: took develop's earlier `protected:`, which also covers
processLedgerRequest, and kept recordGetObjectMetrics inside it. Dropped the
comment naming one test, since four test files now rely on that access.
TMGetObjectByHash_test.cpp: kept this branch's suite. It already runs develop's
three bound cases (limit+1, limit, limit-1) as testReplyLimit, plus the charge
and hit/miss cases. develop's extracted PeerTest.h stays, used by its four new
overlay tests.
ordering.txt: regenerated. Both perflog edges are real, to xrpl.server and to
xrpl.telemetry.
.cspell.config.yaml: kept both added words.
Resolved src/libxrpl/tx/applySteps.cpp. calculateBaseFee now returns
std::expected<XRPAmount, TER>, so invokePreclaim rejects the transaction when
the fee cannot be computed. Kept that inside this branch's preclaimTer wrapper,
so the returned error also reaches the span's ter_result attribute.
The validation owned its own push trigger and paths filter, which made it the one
build-and-test workflow outside the on-pr.yml plus reusable-*.yml pair the rest of
CI uses. Rename it to reusable-telemetry-validation.yml, take workflow_call, and
let on-pr.yml decide when it runs.
on-pr.yml gains a second changed-files list for the telemetry paths. Its existing
`go` flag is true for nearly every pull request, so gating on `go` alone would run
a self-hosted build and a 30-minute cluster on all of them.
The configure step no longer repeats the telemetry option. A command-line define
is written to the cache before the toolchain runs and wins over it, so it could
turn telemetry on with the package never fetched. build-xrpld reads the value the
build actually configured out of CMakeCache.txt and publishes it as a job output;
validate-telemetry runs only when that says telemetry is on, and every value CMake
does not accept as a boolean fails the job rather than skipping silently.
Brings the MetricsRegistry split onto this branch. The pipeline half is now
xrpl::telemetry::MetricsRegistry in libxrpl; the observable gauges are
xrpl::telemetry::AppMetricGauges in xrpld.
This branch had added its own instrumentation to the pre-split class, so the
merge had to route each addition to the correct half:
- The thirteen gauges added here -- amendment block, cache hit-rate detail,
clock skew, job-queue saturation, ledger quorum publish, peer ledger supply,
rotation state, slot census, stall events, sync acquire, sync state, UNL
quorum, and the cache lock-hold observer -- all land on AppMetricGauges,
reading the core's meter and validation tracker through it.
- The pipeline additions stay in libxrpl: the consensus round-duration and
rotation-phase histogram views, the malloc-trim and dns/dial latency bucket
ladders, the job-stall counter, and the switch from literal metric names to
the MetricNames.h constants.
Git detected the pre-split MetricsRegistry.cpp and .h as renames of the gauge
files, so both sides' pipeline changes initially landed in the gauge half. They
were moved back, and the result was audited by inventory: every method
definition, instrument creation, view registration, and emitted string from
either side is present, with identical multiplicity.
MetricNames.h moves to include/xrpl/telemetry/ alongside the core. It has no
includes of its own and its two sibling name headers already live there, so
keeping it under src/ would leave an xrpld path in libxrpl's dependency
surface. Nineteen files follow it.
incrementStateChanges() stays removed. The labelled state_changes_total{from,to}
counter this branch introduced replaces it, and the test asserting the method is
absent is kept -- an unlabelled instrument alongside the labelled one would give
Prometheus two conflicting versions of one metric name.
Two tests that drove startAsyncGauges() against a mock ServiceRegistry are
dropped: xrpl_tests links only xrpl.libxrpl and cannot reach the gauge class.
Levelization regenerated. Both xrpld.telemetry loops become bidirectional
rather than one-way; neither is new.
MetricsRegistry did two jobs. It owned the OTel metrics pipeline, and it
registered the observable gauges whose callbacks read live application
services. The second job is what made the whole class xrpld-tier, so the
pipeline's lifecycle -- the recording() gate and the stop() teardown that
closes a use-after-free window -- could not be unit-tested in xrpl_tests.
Split it in two:
- xrpl::telemetry::MetricsRegistry (libxrpl) owns the exporter, provider,
meter, the 16 synchronous instruments, recording(), stop(), and the
record*/increment* methods.
- xrpl::telemetry::AppMetricGauges (xrpld) owns the 19 observable gauges
and their callbacks, holding a reference to the core and to the
ServiceRegistry.
MetricMacros.h and ValidationTracker move with the core. The macros need
only recording() and meter(), both core members; the core holds a tracker
by value, and a libxrpl header cannot include one from src/.
ApplicationImp owns both objects and sequences them. The core is built in
the member-init list, so every synchronous instrument exists before any
subsystem can record one. The gauges are armed once overlay_ exists, the
last service their callbacks read. Shutdown detaches the gauge callbacks
before the core drops the provider, and each shutdown step is isolated so
a failure in one cannot skip the others.
That detach call is new. detachCallbacks() had no callers, and the flag it
sets is read by the gauge callbacks but can no longer be written by the
core, so the caller now has to make the ordering explicit.
The telemetry module links xrpl.libxrpl.core and xrpl.libxrpl.protocol
PUBLIC: ValidationTracker.h takes a LedgerIndex and MetricMacros.h takes a
ServiceRegistry, both in interfaces a consumer compiles against.
Adds a MetricsRegistry gtest that drives an enabled core with telemetry on
and pins the recording() gate, stop() leaving the registry inert, and
stop() being idempotent. The libxrpl test tree no longer depends on
xrpld.telemetry at all, and the two CMake workarounds that compiled xrpld
sources into xrpl_tests are gone.
Documentation and dashboard source links follow the code to their new
paths, split between the two classes by which one now defines each metric.
Resolutions:
- MetricsRegistry.cpp: keep both <exception> and <limits>; drop
incrementStateChanges(), which this branch removed on purpose (the
labelled state_changes_total call site in NetworkOPsImp::setMode
replaces it, and a compile-time test guards that).
- tests/MetricsRegistry.cpp: constructor-built pipeline wording from
phase-10, this branch's test list and gauge paragraphs kept; the two
lifecycle tests now call startAsyncGauges() and pass kTestOptions.
- tests/MetricMacros.cpp: comments name the recording() gate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Three conflicts, all from both branches editing the same passage:
- Main.cpp: kept phase-9's wording. The metrics registry only exists here, so
"unwinding destroys little: the metrics registry, whose destructor joins its
export thread" is the true statement on this branch.
- TESTING.md: kept both paragraphs. They document different things (the
private [network_id], and the log path plus log_level).
- 05-configuration-reference.md: composed both. The identity is now resolved
before construction and never empty, so every producer stamps the node key
from the start; the only divergence left is a wallet that already holds a
different key, which corrects the tracer alone. Rewrote the earlier
"three producers" blockquote too: its "no fallback", "first boot ... left
off" and "Known issue" claims are what this change removes.
One silent break the merge could not flag: makeMetricsRegistryOptions() took
the std::optional<std::string> node key that used to be a constructor
parameter, and that parameter is now the resolved keypair. It takes the base58
string directly, and the constructor derives it from nodeIdentity_, which is
declared before both telemetry_ and metricsRegistry_.
resolveNodePublicKey() returned std::nullopt in three real cases: a first boot
with no wallet database, a standalone run (its wallet is a private temporary
database), and --newnodeid. Telemetry's resources are built during
ApplicationImp's member-init list and are immutable, so on those runs the node
reported an empty service.instance.id and no xrpl.node.id for the whole run,
while setup() minted a key moments later and patched only the tracer.
Replace it with resolveNodeIdentity(), which always returns a keypair: derived
from a configured seed, else read from an existing wallet database, else
minted. Main.cpp passes that pair to makeApplication(), ApplicationImp stores
it in nodeIdentity_ -- now declared before telemetry_ and no longer an optional,
because it is always set -- and builds the telemetry resource from it.
setup() calls getNodeIdentity(), which now persists rather than mints: it
stores the resolved pair when the wallet holds no identity, adopts the stored
one when it does, and clears first for --newnodeid. The write stays in setup()
because that is where the database exists; a standalone run has no persistent
wallet to write to, which is why the pair has to be decided before
construction rather than read back afterwards. Wallet gains storeNodeIdentity()
for that write, and getNodeIdentity(session) now uses it instead of repeating
the insert.
The three-argument makeApplication() mints a keypair, so jtx::Env and any other
test Application behave as a standalone run always did.
Also fold the three hand-rolled "meter from a NoopMeterProvider" copies into
telemetry::noopMeter(): the base-pointer call and the kMeterVersion argument are
both easy to get wrong alone, and the meter identity has to match the one the
histogram views select on.
The new gtest covers the wallet half: store-then-read, store not replacing an
existing identity, clear-then-store, and that the mint path persists. It adds
the tests.libxrpl > xrpl.rdb levelization edge, regenerated here.
The committed baseline was captured 2026-08-26, before the account-funding
race was detectable. Phases whose funding silently failed submitted no
transactions, so the capture recorded artificially low ledger and transaction
timings, and job.transaction.queued.p95 and job.transaction.running.p95 could
not be captured at all. Once funding worked, span.ledger.build.p99 read
29.00 ms against a 9.11 ms baseline and turned the gate red on a run whose
200 span and metric checks all passed.
Refresh every value to the median of CI runs 34495527952, 34505215266 and
34507425933, the first three with the fix in place, and re-derive each
absolute bound as hi_next - baseline from that median.
Exclude span.ledger.build.p99. Across those three runs it read 29.00, 7.06
and 8.94 ms, a 4.11x spread whose maximum is 1.16x its 25 ms trip point, so a
healthy run reddens CI. Widening cannot fix it: a bound tolerating 29.00 ms
would reach into the bucket above and restore the single-crossing false
positive the derivation rule removes. span.ledger.build.p95 stays gated at
0.48 of its trip point, so ledger construction keeps coverage.
The other 19 keys sit between 0.17 and 0.76 of their trip points.
span.tx.process.p95 is the tightest and is the first to re-measure if the gate
reddens again.
Repoint one bounds-checker test at span.ledger.build.p95, since it mutated the
p99 override this commit removes.
Account setup submitted the funding Payments, slept a flat 10 seconds, then
read each sequence once. The txq-burst and mixed-peak phases escalate the
open-ledger fee on purpose, so the funding transactions were queued, every
account read Sequence 0, and the phase aborted with "only 0 of 8 created
accounts were funded". The run then reddened on a workload gate rather than on
anything telemetry had done.
Poll the ledger until each account has a sequence, with a deadline, so a late
confirmation is still seen and a healthy cluster pays no waiting cost. Pay a
multiple of the current open-ledger fee, so funding is not queued behind the
load a phase creates deliberately. terQUEUED no longer marks an account funded:
only a ledger read does.
Retry the accounts that never confirmed, once, after re-reading the genesis
sequence from the ledger. consumes_sequence advances the local counter on
terQUEUED, so a dropped funding transaction leaves it ahead of the ledger and
every resubmit would otherwise land on a future sequence.
The funding wait can run twice, so raise the orchestrator's grace above twice
the timeout. A test pins that relationship, since the two constants live in
different files.
Also save each generator's full stdout and stderr beside its JSON report. Only
the last 200 characters of stderr reached the phase error and stdout was
dropped, so none of the per-account funding results appeared in CI.
compare_to_baseline took the unit from the baseline entry and dropped the current
run's, and nothing compared the two, so a us -> ms change was scored as a numeric
delta: four keys rewritten to the same physical durations reported 99.9%
improvements and the gate exited 0. prom_queries.py says the baseline preserves
the unit "so the comparator can sanity-check unit drift"; it never did. A unit
mismatch now fails and names both units.
The workflow's step summary printed total, regressions and improvements. total is
every key in the report -- the union of baseline and current -- so it was neither
the baseline count nor what was gated, and missing_in_current was computed and
never printed. A run that gated 16 of 20 keys read as a full comparison. The
comparator now reports a real "compared" count and the summary prints it beside
the not-captured count, with a warning when any key was missed. The table also
refused nothing on a truncated report; existence is not readability.
check_regression_bounds told the operator to add max_abs_increase while reading
max_abs_increase_ms / _us, so following the message added a key nothing reads and
the gate kept failing with no explanation. The committed thresholds use only the
suffixed spelling, so the message was the defect. Its three JSON inputs were also
unchecked: a top-level null, list or number parsed and then died on the first
.get, and a string "metrics" survived the placeholder test and reported its own
characters as gated keys -- wrong advice rather than a crash.
Four tests cover these; all four fail against the previous checker.
