A -DXRPL_ENABLE_TELEMETRY=0 build expands every XRPL_METRIC_* macro to
nothing, so anything whose only use sits inside one looks unused.
- MetricsRegistry::hasPipeline touches no member in that build, so
clang-tidy asks for it to be static. Making it static would give the two
builds different signatures, so suppress it the way Recording.h already
does for the same reason.
- PathRequest.cpp's only use of MetricMacros.h is such a macro, so mark the
include as kept.
- AppMetricGauges.cpp names MetricsRegistry and beast::Journal in a
constructor signature that is compiled either way, so their includes move
outside the telemetry guard.
MetricsRegistry did two jobs. It owned the OTel metrics pipeline, and it
registered the observable gauges whose callbacks read live application
services. The second job is what made the whole class xrpld-tier, so the
pipeline's lifecycle -- the recording() gate and the stop() teardown that
closes a use-after-free window -- could not be unit-tested in xrpl_tests.
Split it in two:
- xrpl::telemetry::MetricsRegistry (libxrpl) owns the exporter, provider,
meter, the 16 synchronous instruments, recording(), stop(), and the
record*/increment* methods.
- xrpl::telemetry::AppMetricGauges (xrpld) owns the 19 observable gauges
and their callbacks, holding a reference to the core and to the
ServiceRegistry.
MetricMacros.h and ValidationTracker move with the core. The macros need
only recording() and meter(), both core members; the core holds a tracker
by value, and a libxrpl header cannot include one from src/.
ApplicationImp owns both objects and sequences them. The core is built in
the member-init list, so every synchronous instrument exists before any
subsystem can record one. The gauges are armed once overlay_ exists, the
last service their callbacks read. Shutdown detaches the gauge callbacks
before the core drops the provider, and each shutdown step is isolated so
a failure in one cannot skip the others.
That detach call is new. detachCallbacks() had no callers, and the flag it
sets is read by the gauge callbacks but can no longer be written by the
core, so the caller now has to make the ordering explicit.
The telemetry module links xrpl.libxrpl.core and xrpl.libxrpl.protocol
PUBLIC: ValidationTracker.h takes a LedgerIndex and MetricMacros.h takes a
ServiceRegistry, both in interfaces a consumer compiles against.
Adds a MetricsRegistry gtest that drives an enabled core with telemetry on
and pins the recording() gate, stop() leaving the registry inert, and
stop() being idempotent. The libxrpl test tree no longer depends on
xrpld.telemetry at all, and the two CMake workarounds that compiled xrpld
sources into xrpl_tests are gone.
Documentation and dashboard source links follow the code to their new
paths, split between the two classes by which one now defines each metric.
clang-tidy's bugprone-unchecked-optional-access does not model GTest's
ASSERT_TRUE(x.has_value()), so it flags every deref that follows one.
The reads are guarded; mark them NOLINT, matching the same suppression
in src/tests/libxrpl/consensus/LedgerTrie.cpp. .value() does not help --
the checker treats it as an unchecked access too.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
OTelCollectorHooks.cpp named nothing from Hook.h, took ResourceMetrics
without a direct include, and left the overriding
GetAggregationTemporality() unmarked. NodeIdentity.h carried <optional>
and <string> it no longer uses. All fail CI under warnings-as-errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SecretKey deletes operator==, so EXPECT_EQ on the pair's second member
does not compile; compare the byte ranges instead. Application.cpp no
longer calls logicError(), so clang-tidy's include-cleaner rejects
<xrpl/basics/contract.h> there.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Static constants take the k prefix (readability-identifier-naming), and
SpanGuardScope.cpp no longer names anything from <opentelemetry/metrics/noop.h>
since it calls noopMeter(). Both fail CI under warnings-as-errors.
The helper's docstring also claimed NoopMeterProvider hides the base
two-argument GetMeter; it declares that overload itself, so the only
detail worth sharing is the version.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
clang-tidy's include-cleaner flags Counter.h and Gauge.h here: the test
names neither type, so the includes fail CI under warnings-as-errors.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
stop() destroys the MeterProvider, and with it every View's
AggregationConfig. The SDK's SyncMetricStorage keeps a raw pointer to
that config, and the call-site statics keep the storage alive, so a
histogram record with a first-seen attribute set during the shutdown
drain would dereference freed memory. Application::run() stops the
registry before the job queue and server handler, so that window is
real.
phase_ is now atomic and stop() stores Stopped before tearing down.
Every XRPL_METRIC_* macro and every record*/increment* method checks
recording() (enabled and not stopped) instead of isEnabled(). meter_ is
never written after construction, so record threads read it without a
lock.
Also: an empty [telemetry] service_instance_id now falls back to the
node key on both the trace and the metrics side, so one node reports one
identity; disablePipeline() uses telemetry::noopMeter(); comments
corrected.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Lift the seed parsing and the stored-vs-mint choice into two libxrpl
helpers, parseNodeIdentitySeed() and selectNodeIdentity(), so xrpl_tests
can drive each branch without an xrpld Config. resolveNodeIdentity() now
marshals Config and the cmdline into them; behaviour is unchanged.
Also pin that storeNodeIdentity() appends (row count, not SQLite row
order), fix the test header that described getNodeIdentity()'s property
as the store's, and route NullTelemetry::getMeter() through noopMeter().
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Merges pratik/otel-phase8-log-correlation into pratik/otel-phase9-metric-gap-fill.
Auto-merged. TESTING.md picked up phase-8's Test 4 rewrite alongside this
branch's Step 2 standalone-store fix. OTelCollector.cpp gained the weak_ptr
gauges_ list; the StatsD test gained its onCollectionReady() calls.
Merges pratik/otel-phase7-native-metrics into pratik/otel-phase8-log-correlation.
Conflict was one TESTING.md hunk under "Nodes not reaching proposing state":
this branch renamed the node directories to Node-N in integration-test.sh,
phase-7 kept nodeN and expanded the [peer_private] explanation. Resolution
keeps this branch's Node-1 path (its own script uses that naming) and
phase-7's fuller prose citing peerfinder/Config.cpp.
Non-conflicting phase-7 changes come through: OTelCollector's gauges_ list
becomes weak_ptr, matching the earlier hooks_ change; the phase-6 revert of
the StatsD-test onCollectionReady() calls resolved against phase-7's version
that keeps them.
Merges pratik/otel-phase6-statsd into pratik/otel-phase7-native-metrics.
Phase-6 dropped the three onCollectionReady() calls that had been added to
its StatsD test, because that method is only declared here on phase-7.
This branch's own copy of the file was unchanged from the merge base, so
the default merge would have silently deleted the calls from here too —
where they are needed, because this branch gates polling behind
onCollectionReady() in OTelCollectorImp::onTimer.
Resolution keeps both sides: phase-6's two new include lines
(Counter.h, Gauge.h) and phase-7's three onCollectionReady() calls plus
their doxygen and inline explanations. The merged file is exactly
phase-7's tip plus those two includes.
TESTING.md auto-merged cleanly; both sides added text under Test 1 in
different regions.
nodeIdentity_ is no longer a std::optional, so setNodeId() must read it
directly. The merge could not flag this: phase-8 changed the member's type and
this line lives only on phase-9, so neither side of the merge touched the same
file region.
Three conflicts, all from both branches editing the same passage:
- Main.cpp: kept phase-9's wording. The metrics registry only exists here, so
"unwinding destroys little: the metrics registry, whose destructor joins its
export thread" is the true statement on this branch.
- TESTING.md: kept both paragraphs. They document different things (the
private [network_id], and the log path plus log_level).
- 05-configuration-reference.md: composed both. The identity is now resolved
before construction and never empty, so every producer stamps the node key
from the start; the only divergence left is a wallet that already holds a
different key, which corrects the tracer alone. Rewrote the earlier
"three producers" blockquote too: its "no fallback", "first boot ... left
off" and "Known issue" claims are what this change removes.
One silent break the merge could not flag: makeMetricsRegistryOptions() took
the std::optional<std::string> node key that used to be a constructor
parameter, and that parameter is now the resolved keypair. It takes the base58
string directly, and the constructor derives it from nodeIdentity_, which is
declared before both telemetry_ and metricsRegistry_.
gauges_ was still a vector of raw pointers after the hook list moved to
weak references. onCollectionReady() and onCollectionStopping() copy that
list under mutex_ and then call arm()/disarm() with the lock released,
because both enter the SDK's observable registry lock. ~OTelGaugeImpl only
re-acquires mutex_ to erase its own entry, so nothing stopped a gauge from
being destroyed between the snapshot and the dereference.
Store weak references, for the reason the hook list already does: locking
an entry keeps that gauge alive for exactly its own arm or disarm call, and
one destroyed since the snapshot locks to null and is skipped. Registration
moves from the OTelGaugeImpl constructor to makeGauge(), because no
weak_ptr to the object exists until the owning shared_ptr does, and the
destructor now prunes by expiry rather than by address.
The startup log line counts the gauges that were still live rather than the
snapshot length, so a skipped entry is not reported as a failed
registration.
The SDK callback path is unchanged and does not need this: gaugeCallback
takes a void* and is guarded by RemoveCallback being synchronous, as
~OTelGaugeImpl already documents.
resolveNodePublicKey() returned std::nullopt in three real cases: a first boot
with no wallet database, a standalone run (its wallet is a private temporary
database), and --newnodeid. Telemetry's resources are built during
ApplicationImp's member-init list and are immutable, so on those runs the node
reported an empty service.instance.id and no xrpl.node.id for the whole run,
while setup() minted a key moments later and patched only the tracer.
Replace it with resolveNodeIdentity(), which always returns a keypair: derived
from a configured seed, else read from an existing wallet database, else
minted. Main.cpp passes that pair to makeApplication(), ApplicationImp stores
it in nodeIdentity_ -- now declared before telemetry_ and no longer an optional,
because it is always set -- and builds the telemetry resource from it.
setup() calls getNodeIdentity(), which now persists rather than mints: it
stores the resolved pair when the wallet holds no identity, adopts the stored
one when it does, and clears first for --newnodeid. The write stays in setup()
because that is where the database exists; a standalone run has no persistent
wallet to write to, which is why the pair has to be decided before
construction rather than read back afterwards. Wallet gains storeNodeIdentity()
for that write, and getNodeIdentity(session) now uses it instead of repeating
the insert.
The three-argument makeApplication() mints a keypair, so jtx::Env and any other
test Application behave as a standalone run always did.
Also fold the three hand-rolled "meter from a NoopMeterProvider" copies into
telemetry::noopMeter(): the base-pointer call and the kMeterVersion argument are
both easy to get wrong alone, and the meter identity has to match the one the
histogram views select on.
The new gtest covers the wallet half: store-then-read, store not replacing an
existing identity, clear-then-store, and that the mint path persists. It adds
the tests.libxrpl > xrpl.rdb levelization edge, regenerated here.
MetricsRegistry created its provider and synchronous instruments in start(),
called from setup() after the node identity was read. Every XRPL_METRIC_*
call site creates its instrument on first use, so any site that ran before
that point found no meter and never recorded again. The start was moved three
times to chase the newest early caller; nothing guaranteed the order.
Build the pipeline in the constructor instead. ApplicationImp declares
metricsRegistry_ right after telemetry_ and before every subsystem, so
declaration order now guarantees the instruments exist before any producer.
start() is gone and its config parsing moves to makeMetricsRegistryOptions().
The registry now guarantees a meter whenever it is enabled: the real one, or
the OTel no-op meter if the pipeline failed to build. disablePipeline() owns
that fallback and its one error log, and the constructor routes both
std::exception and a non-std throw through it, because the SDK is third-party
code. So the macros shrink to one function-local static built from meter()
plus the record call: no once-flag, no null check, and no path for a call
that arrives before the meter, because that state no longer exists. The three
observable macros drop the same now-dead meter check.
The lifecycle is three explicit phases with a Phase enum: Ready at
construction, GaugesArmed by startAsyncGauges() once overlay_ exists, and
Stopped by stop(). startAsyncGauges() checks the phase before the pipeline,
so a second call and a call after stop() are each reported as what they are.
run() stops both observers (insight collector, registry) before any service,
and ~ApplicationImp repeats the stop for the setup() failure paths that never
reach run(). stop() already detaches the callbacks, so it is the only call.
Meter name and version come from kMeterName/kMeterVersion, and the endpoint
default from Telemetry::Setup, so the two metric pipelines share one source
for both.
0eb4291688 added collector->onCollectionReady() to both StatsD tests. That
method does not exist on this branch: it is introduced later, alongside the
polling gate it belongs to, so all four build legs and clang-tidy failed.
Nothing gates polling here. The StatsDCollectorImp constructor starts its
thread, run() calls setTimer() unconditionally, and onTimer polls metrics_
and drains the buffers every second. The gauge test's expectation already
holds without any lifecycle call, because StatsDGaugeImpl starts dirty so a
untouched gauge emits its zero on the first flush.
Restores the two includes that commit also dropped.
callHooks() copied the hook list into a vector of raw pointers, released
mutex_, then dereferenced them. It has to release the lock: a handler may
drop the last reference to a hook, and ~OTelHookImpl re-acquires mutex_,
so invoking handlers under the non-recursive lock would deadlock. That
left a window in which an entry could be freed before it was used.
The window is not reachable today. Every hook belongs to a long-lived
ApplicationImp member, and onCollectionStopping() runs before those
members are destroyed, both from stop() and from the destructor body.
That call disarms each gauge via RemoveCallback, which blocks until an
in-flight callback finishes, because the SDK holds its registry mutex
across the callback. Safety therefore rests on four separate facts, none
of them enforced by a test, one of them internal to a vendored library.
Store weak references instead, so the code is correct by construction:
locking an entry keeps that hook alive for exactly its own handler call,
and a hook destroyed since the snapshot locks to null and is skipped.
Registration moves from the OTelHookImpl constructor to makeHook(),
because no weak_ptr to the object exists until the owning shared_ptr
does, and the destructor now prunes by expiry rather than by address.
Add three GTests over the real collector. A test-local MetricReader
drives one synchronous collection pass, since the SDK ships only a
threaded periodic reader. They assert a live hook runs, a destroyed hook
is skipped, and destroying one hook leaves its siblings registered --
the last pairing both directions so neither can pass vacuously.
Not yet run: verifying them needs a telemetry-enabled build of
xrpl_tests. Compile, clang-tidy and the pre-commit gates are clean.
The three tracker files conflicted because both sides had rewritten them. Took
the incoming lock-free class, then put this branch's two monotonic accessors
back on top of it: totalAgreementsEver() and totalMissedEver(), backed by a
gross pair incremented at first classification and left alone by the repair
branch. MetricsRegistry reads both, so dropping them would not compile.
The test file kept the incoming suite, which renames every case, and gained
this branch's two gross-counter cases adapted to the injected clock.
Three CI failures, one cause each.
macOS could not compile the tracker: Apple's libc++ has no std::atomic for a
shared_ptr, so the primary template's trivially-copyable assert fired. Publish
through boost::atomic_shared_ptr instead, which every standard library the
matrix covers can build. boost/smart_ptr is already used in this tree. The
header's note no longer claims libstdc++ as the assumption.
Four tests that predate the metrics_endpoint scheme guard set tls_client_cert
and only put traces_endpoint on https, so the new guard threw before the check
each one asserts. They now set both endpoints. Nine tests in the two files set a
client cert; the other five stay correct because the pairing, use_tls and traces
checks all run ahead of the metrics one.
Eleven clang-tidy findings: redundant member initialisers, two aggregate
initialisations that wanted designated form, two unbraced bodies, three
unparenthesised multiplications, and a reserve before a loop that emplaces.
Dropping std::make_shared also left <memory> unused in both files.
The two writer paths run on the consensus and ledger-publish threads, and both
took a single mutex that the nine window getters also held while scanning their
deques. At a week of 4-second ledgers those deques hold about 151,000 records,
the same record stored three times, so a read walked half a million entries and
the writers waited behind it.
Each writer now owns a 128-slot ring and shares nothing with the other, so a
record is one timestamp read and one store. A full ring drops the event and
bumps droppedEvents() rather than blocking a consensus thread; at 4-second
ledgers that needs about eight and a half minutes with no drain, against a
worst normal drain gap of one export interval plus one export timeout.
The windows become a grid of one-minute buckets with a running agreed and total
per span, so a getter reads a published snapshot instead of counting. Missed is
derived from the two, which leaves a late repair touching only the agreed side.
Steady-state footprint drops from about 8 MB to about 91 KB, and window edges
quantise to a minute.
Whichever thread is inside reconcile() owns the decision state; a second caller
returns instead of waiting. Readers take a shared_ptr to the snapshot, so the
values they read cannot be overwritten underneath them.
The clock is now injected, which is what makes bucket expiry reachable in a
test. The suite covers pairing, single-sided misses, the grace boundary, late
repair across a bucket edge and past the window, each window edge exactly, two
grid lengths of steady traffic, ring overflow, and three real-thread cases.
It no longer sleeps: nine sleep_for calls totalling 81 seconds are gone.
Every public method keeps its name and signature.
MetricsRegistry::start() takes a StartOptions aggregate. Six call sites in
the #ifndef XRPL_ENABLE_TELEMETRY block still passed a std::string, so the
block did not compile. Nothing in CI compiles it: telemetry defaults ON, and
the block is skipped whenever the macro is defined.
Add one shared kTestStartOptions carrying just the endpoint -- the other
fields are never read on the no-op path -- and correct two comments that
still described the old signature and a #else stub that does not exist.
