Two suspects from the 3.3.0 slowdown investigation had no signal. Both were
already computing the numbers and throwing them away, so this exposes them
rather than adding measurement.
Per-sweep heap trim. The trim runs after every cache sweep, and its cost
scales with resident heap, so it is the leading explanation for a node with
a populated database syncing slower than a fresh one. The report already
carried duration, fault deltas and reclaimed pages, but the whole
measurement sat behind a debug-journal check, so an ordinary node measured
nothing, and the call site discarded the result. The measurement now always
runs and only the log line stays gated. Records trim duration, minor faults
and reclaimed kilobytes. Measured cost of the always-on path is about six
microseconds per sweep against a trim costing milliseconds, at a cadence of
ten to a hundred and twenty seconds.
Honest limit, stated in the runbook: the fault delta spans only the trim
call, so it shows the trim itself faulting but not the faults that follow as
caches refill. The duration is the signal to correlate against sweep-job
queueing.
Rotation writes. Rotation copies archive-served reads forward and re-stores
nodes missing from both backends, both of which compete with sync I/O and
only happen on a populated online_delete database. The copy-forward count
existed but was reset by the rotation's own log line, so a metric reading it
would drop to zero on every swap; a never-reset total sits beside it now.
The re-store count was not measured at all. Rotation duration is
deliberately not recorded: the health throttle sleeps at eight points inside
the sequence and dominates exactly when the node is unhealthy, so the number
would conflate work with waiting.
Nothing added for the other two suspects. Get-object serving is already
covered by the handler label, the lookup histogram and the deferred and
saturation gauges; peer churn by the disconnect-reason counter.
Also replaces nine per-file cspell ignores with one ignoreRegExpList entry
for the telemetry macro names, and picks up the levelization baseline for the
consensus span-name test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brings in the telemetry/metrics init-order fix (start before [rpc_startup]).
Conflict in Application.cpp: phase10 had reordered start() to run
telemetry+metrics first; phase9 relocated that block into a new
startTelemetry() called from setup() before the [rpc_startup] loop.
Resolved by taking the fix — the block lives only in startTelemetry()
(called once from setup(), after overlay_ is built), removed from start().
telemetry_->start() and metricsRegistry_->start() each appear exactly once.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Metric-emitting code that first runs during ApplicationImp::setup() (the
[rpc_startup] loop) executed while the OTel meter was still empty, because
telemetry_ and metricsRegistry_ were started later in start(). A call-site
metric macro caches its instrument on first use via std::call_once, so that
first pre-meter use latched a null instrument for the process lifetime and the
metric silently never recorded (observed: rpc_in_flight_requests, emitted from
PerfLogImp::rpcStart, never appeared).
Extract telemetry_->start() and the metricsRegistry_->start() block into a new
private ApplicationImp::startTelemetry() and call it from setup() just before
the [rpc_startup] loop, so the meter is live before any metric-emitting code
runs. The call sits after overlay_ (and the other subsystems the observable-
gauge callbacks read) is constructed, since starting the metrics reader thread
earlier would let its callbacks call getOverlay() before overlay_ exists.
perfLog_->start() stays in start() — the macro call sites do not depend on it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the xrpld_ prefix from beast::insight metric names — the OTel
resource (service.name=xrpld) already identifies the service. formatName()
now lowercases and sanitizes spaces/dots to underscores for
Prometheus-compatible names.
Also fix 4 source strings containing spaces that produced invalid OTel
instrument names (silently rejected by the SDK).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update the transform/cloudlabels processor to set service_instance_id
instead of exported_instance, matching the local Prometheus path where
resource_to_telemetry_conversion promotes service.instance.id with that
exact label name. Also update source comments and documentation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The native OTel metrics path hard-coded service.name="xrpld" and stamped
no network attribute, while traces stamped a configurable service.name
and xrpl.network.type. Metrics therefore could not be filtered by service
or network. Align the two paths:
- OTelCollector::New / OTelCollectorImp gain serviceName + networkType
params. service.name uses the configured value (default "xrpld" when
unset, preserving today's behavior); xrpl.network.type is stamped when
provided. The key is a string literal because beast/insight sits below
the telemetry module and cannot include its SpanNames const.
- CollectorManager reads service_name from [insight], falling back to the
[telemetry] value, and receives the network type from the caller.
- Application derives the network type once via the shared
telemetry::networkTypeFromId, now declared in Telemetry.h and moved out
of an anonymous namespace so the trace and metric paths reuse a single
0/1/2 -> mainnet/testnet/devnet mapping (no duplication).
Dashboards (5 system-* files): add $service_name, $deployment_environment,
$xrpl_network_type template variables and wire them into every panel query
that filters by $node.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
MetricsRegistry observable-gauge callbacks run on the OTel reader thread
and read live state from nodeStore_, overlay_, networkOPs_, ledgerMaster,
inboundLedgers, loadManager, and others. The old shutdown sequence called
metricsRegistry_->stop() AFTER all those services were already stopped,
which left a race window between each service's stop() and the final
provider_->ForceFlush() during which a callback could dereference
already-stopped service state. The try/catch guards in each callback
mitigated crashes but not reads from freed members.
- Add MetricsRegistry::detachCallbacks() that sets an atomic<bool>
callbacksDetached_ with release ordering. Idempotent.
- Guard every ObservableGauge callback entry with an acquire-load of the
same flag and return early if it is set. Covers all 15 registered
callbacks (cacheHitRate, txq, objectCount, loadFactor, nodeStore,
serverInfo, buildInfo, completeLedgers, dbMetrics, validatorHealth,
peerQuality, ledgerEconomy, stateTracking, storageDetail,
validationAgreement).
- Application::run() shutdown sequence now calls
metricsRegistry_->detachCallbacks() right after m_loadManager->stop()
and BEFORE m_shaMapStore, m_jobQueue, overlay_, grpcServer_,
m_networkOPs, serverHandler_, m_ledgerReplayer, m_inboundTransactions,
m_inboundLedgers, ledgerCleaner_, m_nodeStore, perfLog_ are stopped.
The acquire/release pair guarantees subsequent reader-thread ticks see
the detach before they dereference stopped services.
- metricsRegistry_->stop() keeps setting the flag as a belt-and-suspenders
defense in case a future caller forgets to detach first.
- Drop the misleading "No explicit RemoveCallback is needed" comment
from stop(); provider destruction alone does not beat the reader
thread to already-freed state.
The objectCountGauge callback previously discarded its state pointer
via `void* /* state */`; restore the state argument so it can access
self->callbacksDetached_ too.
Two fixes so gauges register in Prometheus (via StatsD) even when their
initial/steady-state value is 0:
1. StatsDGaugeImpl m_dirty: default-init to true so the initial value
(0) is emitted on the first flush. Previously, gauges whose value
never changed from 0 were never flushed and never appeared
downstream.
2. io_latency_sampler firstSample_: new atomic<bool>, init true.
m_event.notify now fires when either firstSample_ is true (exchanged
to false) or lastSample >= 10 ms. This guarantees the io_latency
metric is registered on startup; subsequent sub-10 ms samples are
still suppressed to avoid flooding.
