MetricsRegistry observable-gauge callbacks run on the OTel reader thread
and read live state from nodeStore_, overlay_, networkOPs_, ledgerMaster,
inboundLedgers, loadManager, and others. The old shutdown sequence called
metricsRegistry_->stop() AFTER all those services were already stopped,
which left a race window between each service's stop() and the final
provider_->ForceFlush() during which a callback could dereference
already-stopped service state. The try/catch guards in each callback
mitigated crashes but not reads from freed members.
- Add MetricsRegistry::detachCallbacks() that sets an atomic<bool>
callbacksDetached_ with release ordering. Idempotent.
- Guard every ObservableGauge callback entry with an acquire-load of the
same flag and return early if it is set. Covers all 15 registered
callbacks (cacheHitRate, txq, objectCount, loadFactor, nodeStore,
serverInfo, buildInfo, completeLedgers, dbMetrics, validatorHealth,
peerQuality, ledgerEconomy, stateTracking, storageDetail,
validationAgreement).
- Application::run() shutdown sequence now calls
metricsRegistry_->detachCallbacks() right after m_loadManager->stop()
and BEFORE m_shaMapStore, m_jobQueue, overlay_, grpcServer_,
m_networkOPs, serverHandler_, m_ledgerReplayer, m_inboundTransactions,
m_inboundLedgers, ledgerCleaner_, m_nodeStore, perfLog_ are stopped.
The acquire/release pair guarantees subsequent reader-thread ticks see
the detach before they dereference stopped services.
- metricsRegistry_->stop() keeps setting the flag as a belt-and-suspenders
defense in case a future caller forgets to detach first.
- Drop the misleading "No explicit RemoveCallback is needed" comment
from stop(); provider destruction alone does not beat the reader
thread to already-freed state.
The objectCountGauge callback previously discarded its state pointer
via `void* /* state */`; restore the state argument so it can access
self->callbacksDetached_ too.
- MetricsRegistry.cpp: concatenate nested namespaces, add missing
direct includes (Journal.h, string, string_view, cstdint), suppress
readability-convert-member-functions-to-static in #else stubs by
referencing enabled_ member, void unused instanceId parameter.
- MetricsRegistry test: add missing direct includes (Log.h, Journal.h,
uint256.h, io_context.hpp, optional, stdexcept, string), make
throwUnimplemented() static, add [[nodiscard]] to getOpenLedger/
isStopping/getTrapTxID overrides, make const-eligible registry const.
- PerfLogImp.cpp: add braces around if/else body per
readability-braces-around-statements.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
NetworkOPs.h and SpanNames.h were only needed for per-span
nodeAmendmentBlocked/nodeServerState calls, which were removed
in the attr naming simplification. Fixes clang-tidy CI failure.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Drop xrpl.pathfind.* prefix from per-span attrs (source_account,
dest_account, fast, search_level, num_complete_paths, num_paths,
num_requests).
- Keep xrpl.pathfind.ledger_index qualified (rule 5: distinct from
xrpl.ledger.seq).
- Remove per-span nodeAmendmentBlocked/nodeServerState calls from
RPCHandler — promoted to resource-level attrs.
- Mark node-health attrs in SpanNames.h as RESOURCE-ONLY with doc.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two fixes so gauges register in Prometheus (via StatsD) even when their
initial/steady-state value is 0:
1. StatsDGaugeImpl m_dirty: default-init to true so the initial value
(0) is emitted on the first flush. Previously, gauges whose value
never changed from 0 were never flushed and never appeared
downstream.
2. io_latency_sampler firstSample_: new atomic<bool>, init true.
m_event.notify now fires when either firstSample_ is true (exchanged
to false) or lastSample >= 10 ms. This guarantees the io_latency
metric is registered on startup; subsequent sub-10 ms samples are
still suppressed to avoid flooding.
Set defaults for tx_span::attr::suppressed (false) and
tx_span::attr::status ("new") immediately after creating the txReceive
span. Without defaults, spans whose suppressed/status attributes would
only be set in the HashRouter-suppressed branch lacked these attributes
entirely, producing incomplete span data in downstream stores.
The suppressed branch still overrides these when the transaction has
already been seen via HashRouter.
Two build failures surfaced by CI on the Phase 9 branch:
1. NetworkOPsImp stores the ServiceRegistry as
std::reference_wrapper<ServiceRegistry> registry_, so calls must go
through registry_.get().<method>(). The MetricsRegistry hooks added
in setMode() and recvValidation() dereferenced the wrapper directly,
which compiles against a pre-existing accessor on the wrapper type
on some toolchains but fails on clang 16/17/20 and gcc 13/15 with
"no member named 'getMetricsRegistry' in
std::reference_wrapper<xrpl::ServiceRegistry>".
2. MetricsRegistry::app_ and MetricsRegistry::journal_ are only used
inside XRPL_ENABLE_TELEMETRY-guarded code paths (gauge callbacks
and JLOG). When telemetry is disabled, clang's
-Werror=-Wunused-private-field tripped. Move the two fields under
the same #ifdef and guard the constructor initialisers with
[[maybe_unused]] so the no-op build continues to compile cleanly.
Addresses code review findings on PR #6513:
1. registerAsyncGauges() was ~730 lines, violating the CLAUDE.md
rule "No function longer than 80 lines." Split into fifteen
per-domain helpers (cache, TxQ, object count, load factor,
NodeStore, server info, build info, complete ledgers, DB,
validator health, peer quality, ledger economy, state tracking,
storage detail, validation agreement) dispatched from a thin
shell. Each helper now stays at or below the 80-line limit.
2. PerfLogImp::rpcEnd() only updated the in-memory counter and
never advanced the OTel xrpld_rpc_method_finished_total,
xrpld_rpc_method_errored_total, or xrpld_rpc_method_duration_us
instruments. rpcStart() was already wired up, so the finished
and errored counters stayed at zero for every RPC call.
rpcEnd() now computes the duration once, records it under the
existing mutex, and forwards finish/error events to
MetricsRegistry::recordRpcFinished / recordRpcErrored outside
the counter mutex to avoid lock nesting with the OTel SDK.
3. Added class-level Doxygen for MetricsRegistry with an ASCII
collaborator diagram and explicit @note tags covering
thread-safety, lifetime, and extension guidance.
Consensus.h (Phase 4 tracing) depends on DisputedTx::getYays()/getNays()
to build disputeResolve span events. Both accessors were removed by
earlier 'duplicate accessor' cleanup commits on this branch, leaving
Consensus.h referencing non-existent members. CI caught this on
macOS/clang-17/gcc-13/Windows builds.
Restore the accessors on the branch where they were dropped so downstream
phase branches inherit a compiling DisputedTx.h via merge.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Clang-tidy fixes:
- Concatenate nested namespaces (modernize-concat-nested-namespaces)
in OTelCollector.h, OTelCollector.cpp, ValidationTracker.h/.cpp
- Add missing direct includes (misc-include-cleaner) in
ValidationTracker.cpp, test, CollectorManager.cpp, OTelCollector.cpp
- Make lock_guard variables const (misc-const-correctness)
- Add braces around single-line if/else (readability-braces-around-statements)
- Use designated initializer for WindowEvent (modernize-use-designated-initializers)
- Initialize LedgerEvent::seq field (cppcoreguidelines-pro-type-member-init)
Linker fix:
- Add ValidationTracker.cpp as source to xrpl.test.telemetry target
(it lives in src/xrpld/ but the test links against libxrpl only)
Levelization fix:
- Remove stale dependency edges from ordering.txt that were introduced
by the erroneous develop-merge commit
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
These were already added in an earlier phase branch. The duplicate
with slightly different Doxygen wording was introduced by the
erroneous merge/revert cycle.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
