MetricsRegistry did two jobs. It owned the OTel metrics pipeline, and it
registered the observable gauges whose callbacks read live application
services. The second job is what made the whole class xrpld-tier, so the
pipeline's lifecycle -- the recording() gate and the stop() teardown that
closes a use-after-free window -- could not be unit-tested in xrpl_tests.
Split it in two:
- xrpl::telemetry::MetricsRegistry (libxrpl) owns the exporter, provider,
meter, the 16 synchronous instruments, recording(), stop(), and the
record*/increment* methods.
- xrpl::telemetry::AppMetricGauges (xrpld) owns the 19 observable gauges
and their callbacks, holding a reference to the core and to the
ServiceRegistry.
MetricMacros.h and ValidationTracker move with the core. The macros need
only recording() and meter(), both core members; the core holds a tracker
by value, and a libxrpl header cannot include one from src/.
ApplicationImp owns both objects and sequences them. The core is built in
the member-init list, so every synchronous instrument exists before any
subsystem can record one. The gauges are armed once overlay_ exists, the
last service their callbacks read. Shutdown detaches the gauge callbacks
before the core drops the provider, and each shutdown step is isolated so
a failure in one cannot skip the others.
That detach call is new. detachCallbacks() had no callers, and the flag it
sets is read by the gauge callbacks but can no longer be written by the
core, so the caller now has to make the ordering explicit.
The telemetry module links xrpl.libxrpl.core and xrpl.libxrpl.protocol
PUBLIC: ValidationTracker.h takes a LedgerIndex and MetricMacros.h takes a
ServiceRegistry, both in interfaces a consumer compiles against.
Adds a MetricsRegistry gtest that drives an enabled core with telemetry on
and pins the recording() gate, stop() leaving the registry inert, and
stop() being idempotent. The libxrpl test tree no longer depends on
xrpld.telemetry at all, and the two CMake workarounds that compiled xrpld
sources into xrpl_tests are gone.
Documentation and dashboard source links follow the code to their new
paths, split between the two classes by which one now defines each metric.
The three tracker files conflicted because both sides had rewritten them. Took
the incoming lock-free class, then put this branch's two monotonic accessors
back on top of it: totalAgreementsEver() and totalMissedEver(), backed by a
gross pair incremented at first classification and left alone by the repair
branch. MetricsRegistry reads both, so dropping them would not compile.
The test file kept the incoming suite, which renames every case, and gained
this branch's two gross-counter cases adapted to the injected clock.
Three CI failures, one cause each.
macOS could not compile the tracker: Apple's libc++ has no std::atomic for a
shared_ptr, so the primary template's trivially-copyable assert fired. Publish
through boost::atomic_shared_ptr instead, which every standard library the
matrix covers can build. boost/smart_ptr is already used in this tree. The
header's note no longer claims libstdc++ as the assumption.
Four tests that predate the metrics_endpoint scheme guard set tls_client_cert
and only put traces_endpoint on https, so the new guard threw before the check
each one asserts. They now set both endpoints. Nine tests in the two files set a
client cert; the other five stay correct because the pairing, use_tls and traces
checks all run ahead of the metrics one.
Eleven clang-tidy findings: redundant member initialisers, two aggregate
initialisations that wanted designated form, two unbraced bodies, three
unparenthesised multiplications, and a reserve before a loop that emplaces.
Dropping std::make_shared also left <memory> unused in both files.
The two writer paths run on the consensus and ledger-publish threads, and both
took a single mutex that the nine window getters also held while scanning their
deques. At a week of 4-second ledgers those deques hold about 151,000 records,
the same record stored three times, so a read walked half a million entries and
the writers waited behind it.
Each writer now owns a 128-slot ring and shares nothing with the other, so a
record is one timestamp read and one store. A full ring drops the event and
bumps droppedEvents() rather than blocking a consensus thread; at 4-second
ledgers that needs about eight and a half minutes with no drain, against a
worst normal drain gap of one export interval plus one export timeout.
The windows become a grid of one-minute buckets with a running agreed and total
per span, so a getter reads a published snapshot instead of counting. Missed is
derived from the two, which leaves a late repair touching only the agreed side.
Steady-state footprint drops from about 8 MB to about 91 KB, and window edges
quantise to a minute.
Whichever thread is inside reconcile() owns the decision state; a second caller
returns instead of waiting. Readers take a shared_ptr to the snapshot, so the
values they read cannot be overwritten underneath them.
The clock is now injected, which is what makes bucket expiry reachable in a
test. The suite covers pairing, single-sided misses, the grace boundary, late
repair across a bucket edge and past the window, each window edge exactly, two
grid lengths of steady traffic, ring overflow, and three real-thread cases.
It no longer sleeps: nine sleep_for calls totalling 81 seconds are gone.
Every public method keeps its name and signature.
Both record entry points reached their record with pending_[ledgerHash], and
operator[] default-constructs on a missing key. An all-zero recordTime was the
only signal for "first time seeing this ledger", so it could not tell a new
ledger apart from one already counted and evicted.
Reconciled entries are removed on purpose once they leave the late-repair
window, and reconcile() adds a ledger to the totals only on its first
reconcile. A validation arriving after its entry was counted and removed
therefore rebuilt a blank record, looked new, and was counted a second time -
in the lifetime totals and in all three rolling windows, so the agreement
percentages skewed too. A re-arrival carrying only one side manufactured a miss
for a ledger first counted as an agreement.
Both entry points now go through pendingEvent(), which looks in pending_
first so a ledger still awaiting late repair keeps updating its record, then
checks tallied_ and records nothing for a ledger already counted. That retires
the all-zero sentinel, which was the defect itself.
tallied_ holds hashes only, since the record is gone by then, and is bounded at
kMaxTalliedEvents with oldest-first eviction to match evictOldPending. A
validation for a ledger counted more than that many ledgers ago is counted
again; the comment says so.
The message counters keep incrementing on every call, including for an ignored
ledger. They count messages rather than ledgers.
kMaxPendingEvents becomes public so the new test sizes its fill from the
production constant instead of copying the number. Its value and meaning are
unchanged.
A ledger reaches totalAgreements or totalMissed only when its event is
classified, which normally happens in reconcile() once the grace period has
elapsed. The insert-path bound erased the oldest entry outright, so an event
dropped before it was classified was counted as neither an agreement nor a
miss and both lifetime totals under-reported. MaxPendingEventsTrimming shows
it: 1100 recorded ledgers yielded 1000 agreements.
Move the classification into classifyPending() and call it from both places,
so a ledger reaches the totals exactly once whether reconcile() or the bound
resolves it. Add a test that records past the cap without reconciling and
asserts every evicted ledger still has a verdict -- the property the earlier
bound test missed, because it only checked the record-time counter.
Classifying on eviction fixes the verdict on whichever validations have
arrived, so one still in flight can no longer complete or repair it. That
only happens while over the bound, which means reconcile() is not running.
ValidationTracker::pending_ was pruned only by evictOldPending(), which is
reachable only from reconcile(), which runs only from the observable-gauge
callbacks. Those callbacks need telemetry compiled in and enabled, so any node
without both recorded one entry per validated ledger and freed none -- roughly
2.4 MB a day, plus a mutex acquisition per ledger. kMaxPendingEvents did not
help: that check lives inside the function that never ran.
This also affected ordinary builds, not just telemetry-off ones, because
startAsyncGauges() returns early when [telemetry] enabled=0 and so never
registers the callbacks.
Bound the map where the bound can actually be guaranteed -- the insert path --
by dropping the oldest entry once it is full. A container has to be bounded by
its writer, not by whoever happens to read it.
Also stop doing the work when nothing will consume it: hold the tracker, its
accessor and its header behind the telemetry guard, and check isEnabled() at
both record sites, which the metric macros do but these direct calls did not.
Adds pendingCount() so the bound is observable, and a test that records 8000
ledgers without ever reconciling and asserts the size settles at a fixed cap.
- OTelCollector::formatName made static (no member state); braces added;
<cctype> added for std::tolower.
- Telemetry.cpp getMeter(): braces around single-statement if.
- GetMeter.cpp: drop unused <opentelemetry/metrics/meter.h> and
sync_instruments.h; add <xrpl/beast/utility/Journal.h> and <memory>;
make sdkProvider const.
- ValidationTracker.h: add <cstddef> for std::size_t.
- detail/ValidationTracker.cpp: use std::views::reverse range-based loop;
add <ranges>.
- test ValidationTracker.cpp: use auto for cast result.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Replace unused observer_result.h with nostd/shared_ptr.h and
nostd/unique_ptr.h in MetricsRegistry.h (misc-include-cleaner)
- Use auto for cast initializations in MetricsRegistry.cpp and
ValidationTracker test (modernize-use-auto)
- Run rename scripts: rippled -> xrpld in comments and docs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire the two previously-registered-but-never-incremented validation
counters to ValidationTracker's gross lifetime tallies, exported as
monotonic ObservableCounters. New gross atomics count each ledger once at
first classification and are never adjusted on late repair, keeping the
_total counters monotonic and additive (agreements_total + missed_total ==
ledgers reconciled); the repair-aware windowed view stays on the existing
xrpld_validation_agreement gauge. The validator-health dashboard panels
that already query these names now render data instead of "No data".
Also de-stale 09-data-collection-reference.md: §5b documented flat metric
names (xrpld_cache_SLE_hit_rate, ...) that the code never emits — it emits
labeled gauges (xrpld_cache_metrics{metric="SLE_hit_rate"}). Replace the
stale flat-name tables with a pointer to the canonical labeled section,
reconcile the contradictory headline counts, and correct xrpld_job_count
to its real exported name xrpld_jobq_job_count.
Adds two GTests asserting gross tallies stay frozen on repair while net
totals move, plus the additive invariant.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Clang-tidy fixes:
- Concatenate nested namespaces (modernize-concat-nested-namespaces)
in OTelCollector.h, OTelCollector.cpp, ValidationTracker.h/.cpp
- Add missing direct includes (misc-include-cleaner) in
ValidationTracker.cpp, test, CollectorManager.cpp, OTelCollector.cpp
- Make lock_guard variables const (misc-const-correctness)
- Add braces around single-line if/else (readability-braces-around-statements)
- Use designated initializer for WindowEvent (modernize-use-designated-initializers)
- Initialize LedgerEvent::seq field (cppcoreguidelines-pro-type-member-init)
Linker fix:
- Add ValidationTracker.cpp as source to xrpl.test.telemetry target
(it lives in src/xrpld/ but the test links against libxrpl only)
Levelization fix:
- Remove stale dependency edges from ordering.txt that were introduced
by the erroneous develop-merge commit
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cover normal agreement, missed validation, late repair, empty window,
grace period boundary, max pending trimming, mixed results, duplicate
recording, and only-we-validated scenarios.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
