Resolved five files.
PerfLogImp.cpp: PerfLog now takes its method names as string_view, so the
methods map stores the map key rather than a char const*. Kept this branch's
scoped lock so the OTel recording still runs after methodsMutex is released,
and kept both the NetworkOPs and metric-macro includes.
PeerImp.h: took develop's earlier `protected:`, which also covers
processLedgerRequest, and kept recordGetObjectMetrics inside it. Dropped the
comment naming one test, since four test files now rely on that access.
TMGetObjectByHash_test.cpp: kept this branch's suite. It already runs develop's
three bound cases (limit+1, limit, limit-1) as testReplyLimit, plus the charge
and hit/miss cases. develop's extracted PeerTest.h stays, used by its four new
overlay tests.
ordering.txt: regenerated. Both perflog edges are real, to xrpl.server and to
xrpl.telemetry.
.cspell.config.yaml: kept both added words.
Resolved src/libxrpl/tx/applySteps.cpp. calculateBaseFee now returns
std::expected<XRPAmount, TER>, so invokePreclaim rejects the transaction when
the fee cannot be computed. Kept that inside this branch's preclaimTer wrapper,
so the returned error also reaches the span's ter_result attribute.
The table conflicted because both sides edited it: this branch had dropped
ledger_seq from the tx.apply row, and the incoming side updated the tx.receive
row. Kept both.
Alloy and the Grafana Cloud collector still declared a spanmetrics dimension
for an attribute tx.receive no longer sets, which only widens the label set.
The runbook's drop-path diagram still listed two tx_status values that no
longer exist, because the paths that set them now run before the span starts.
The validation owned its own push trigger and paths filter, which made it the one
build-and-test workflow outside the on-pr.yml plus reusable-*.yml pair the rest of
CI uses. Rename it to reusable-telemetry-validation.yml, take workflow_call, and
let on-pr.yml decide when it runs.
on-pr.yml gains a second changed-files list for the telemetry paths. Its existing
`go` flag is true for nearly every pull request, so gating on `go` alone would run
a self-hosted build and a 30-minute cluster on all of them.
The configure step no longer repeats the telemetry option. A command-line define
is written to the cache before the toolchain runs and wins over it, so it could
turn telemetry on with the package never fetched. build-xrpld reads the value the
build actually configured out of CMakeCache.txt and publishes it as a job output;
validate-telemetry runs only when that says telemetry is on, and every value CMake
does not accept as a boolean fails the job rather than skipping silently.
This branch rewrote the CMakeLists telemetry block and the build doc. Both now
describe a CMake option that no longer exists: the Conan option is the switch and
the generated toolchain carries it into CMake. The comment also recorded the
state of the change rather than the behaviour of the code.
The doc's "Building without telemetry" section told readers to pass
-Dtelemetry=OFF to CMake as well, which would override the toolchain rather than
follow it.
CMakeLists declared option(telemetry) with a default of its own, so the setting
had two homes and they were free to disagree. The Conan option already reaches
CMake without it: conanfile.py forwards the option into the generated toolchain,
which sets the variable this file reads, and every build here goes through Conan.
Drop the option() call and keep the if(telemetry) test. docs/build/telemetry.md
told readers a CMake option had to be set as well, so that goes with it.
Brings the MetricsRegistry split onto this branch. The pipeline half is now
xrpl::telemetry::MetricsRegistry in libxrpl; the observable gauges are
xrpl::telemetry::AppMetricGauges in xrpld.
This branch had added its own instrumentation to the pre-split class, so the
merge had to route each addition to the correct half:
- The thirteen gauges added here -- amendment block, cache hit-rate detail,
clock skew, job-queue saturation, ledger quorum publish, peer ledger supply,
rotation state, slot census, stall events, sync acquire, sync state, UNL
quorum, and the cache lock-hold observer -- all land on AppMetricGauges,
reading the core's meter and validation tracker through it.
- The pipeline additions stay in libxrpl: the consensus round-duration and
rotation-phase histogram views, the malloc-trim and dns/dial latency bucket
ladders, the job-stall counter, and the switch from literal metric names to
the MetricNames.h constants.
Git detected the pre-split MetricsRegistry.cpp and .h as renames of the gauge
files, so both sides' pipeline changes initially landed in the gauge half. They
were moved back, and the result was audited by inventory: every method
definition, instrument creation, view registration, and emitted string from
either side is present, with identical multiplicity.
MetricNames.h moves to include/xrpl/telemetry/ alongside the core. It has no
includes of its own and its two sibling name headers already live there, so
keeping it under src/ would leave an xrpld path in libxrpl's dependency
surface. Nineteen files follow it.
incrementStateChanges() stays removed. The labelled state_changes_total{from,to}
counter this branch introduced replaces it, and the test asserting the method is
absent is kept -- an unlabelled instrument alongside the labelled one would give
Prometheus two conflicting versions of one metric name.
Two tests that drove startAsyncGauges() against a mock ServiceRegistry are
dropped: xrpl_tests links only xrpl.libxrpl and cannot reach the gauge class.
Levelization regenerated. Both xrpld.telemetry loops become bidirectional
rather than one-way; neither is new.
MetricsRegistry did two jobs. It owned the OTel metrics pipeline, and it
registered the observable gauges whose callbacks read live application
services. The second job is what made the whole class xrpld-tier, so the
pipeline's lifecycle -- the recording() gate and the stop() teardown that
closes a use-after-free window -- could not be unit-tested in xrpl_tests.
Split it in two:
- xrpl::telemetry::MetricsRegistry (libxrpl) owns the exporter, provider,
meter, the 16 synchronous instruments, recording(), stop(), and the
record*/increment* methods.
- xrpl::telemetry::AppMetricGauges (xrpld) owns the 19 observable gauges
and their callbacks, holding a reference to the core and to the
ServiceRegistry.
MetricMacros.h and ValidationTracker move with the core. The macros need
only recording() and meter(), both core members; the core holds a tracker
by value, and a libxrpl header cannot include one from src/.
ApplicationImp owns both objects and sequences them. The core is built in
the member-init list, so every synchronous instrument exists before any
subsystem can record one. The gauges are armed once overlay_ exists, the
last service their callbacks read. Shutdown detaches the gauge callbacks
before the core drops the provider, and each shutdown step is isolated so
a failure in one cannot skip the others.
That detach call is new. detachCallbacks() had no callers, and the flag it
sets is read by the gauge callbacks but can no longer be written by the
core, so the caller now has to make the ordering explicit.
The telemetry module links xrpl.libxrpl.core and xrpl.libxrpl.protocol
PUBLIC: ValidationTracker.h takes a LedgerIndex and MetricMacros.h takes a
ServiceRegistry, both in interfaces a consumer compiles against.
Adds a MetricsRegistry gtest that drives an enabled core with telemetry on
and pins the recording() gate, stop() leaving the registry inert, and
stop() being idempotent. The libxrpl test tree no longer depends on
xrpld.telemetry at all, and the two CMake workarounds that compiled xrpld
sources into xrpl_tests are gone.
Documentation and dashboard source links follow the code to their new
paths, split between the two classes by which one now defines each metric.
Review follow-up on the freshen lock-hold fix:
- Drop host names, dates and one-site figures from the new comments,
harness notes and docs; explain the mechanism in general terms.
- RotationPhase stores its stage and cache labels as owned std::string,
not std::string_view: the ctor still takes views so the label
constants pass without a copy, but a member view would dangle if a
caller ever passed a temporary. freshenCache/recordFreshen take the
cache name by std::string_view (read-only, call-scoped).
- The new DatabaseRotating test called fetchNodeObject through the
derived type, whose private override hides the public base method;
call it through Database& instead. This was the dev-box build break.
- freshenCache reports the exact fetched count when a health abort cuts
it short, and stops labelling the per-partition hold 'getKeys'.
- Remove a [[maybe_unused]] that silenced no warning (the build sets
-Wno-unused-parameter and disables misc-unused-parameters).
The runbook provenance paragraph named the internal AWS dev box and a
build hash and dates. State what was measured (one mainnet node, same
host and binary, differing only in store state) without the deployment
detail, which belongs in an internal runbook, not the public repo.
The online-delete rotation's cache freshen called TaggedCache::getKeys(),
which held the cache mutex while copying every key. On the dev box's 26
million entry tree-node cache that hold lasted 5-6 s, froze every job
that touches the cache, and dropped the RocksDB node out of sync once per
rotation: each "getKeys held the lock" warning was followed within 1-5 s
by "View of consensus changed" (5 of 5 rotations on 2026-09-15).
Copy the keys one map partition at a time instead. TaggedCache gains
forEachKeyPartition(), which holds the mutex only while one partition's
keys are copied and runs the callback with the mutex released, so the
longest hold shrinks by the partition count (8 on the dev box). The
freshen.keys rotation phase no longer exists as one step, so its span,
stage value, harness entries and docs are removed; the per-partition hold
still shows on the cache lock-hold peak gauge.
Measure what the freshen achieves, which no existing signal did.
DatabaseRotating gains duplicateCopyForwardTotal(), counting archive
copies made on duplicate fetches (the rotation's own copy walk and
freshen); copyForwardTotal() deliberately excludes those. The freshen
phase records rotation_freshen_keys_total{cache,outcome} and stamps
key_count, cache and keys_copied on its span; the copy phase stamps
nodes_copied. A warn log line per freshen reports the same numbers, and
the ledger-sync-health dashboard gets a Rotation Freshen Yield panel.
Log the "STATE->" operating-mode change at warn instead of info. It is
the only record of a mode change with an exact timestamp; the
state_changes_total counter is scraped once a minute and cannot order a
flap against a multi-second event.
Tests: five GTests for forEachKeyPartition (every key once, empty cache,
mutex free during the callback, concurrent insert, lock-hold peak), three
for duplicateCopyForwardTotal over two memory backends, one for the new
counter's series, and the new name literals.
The counter and event landed in the same commit as Task 6, so the runbook
recipe's step 5 stands on its own now.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `consensus_view_change_total{consensus_mode}` and a `view.change` event
on the round span, both fired from `RCLConsensus::Adaptor::getPrevLedger`
on the same transition-into-WrongLedger edge that already calls
`consensusViewChange()`. The counter is the exact detector for
"consensus disagreed with this node's view this minute"; the event lands the
disagreement on the same trace that carries the round.
`net_ledger_prefix` (16 hex chars) joins `prev_ledger_prefix` on the event,
so a Tempo view of one flap shows both ledger identities on a single line.
`consensus_mode` labels the mode being left (never WrongLedger itself).
Together with the rotation-phase spans, this closes the proof chain a
rotation-driven `full`->`syncing` flap needs — the runbook's step 5
resolves now that the counter and event exist.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
