mirror of
https://github.com/XRPLF/rippled.git
synced 2026-07-23 23:20:33 +00:00
fix(telemetry): start telemetry + metrics before [rpc_startup]
Metric-emitting code that first runs during ApplicationImp::setup() (the [rpc_startup] loop) executed while the OTel meter was still empty, because telemetry_ and metricsRegistry_ were started later in start(). A call-site metric macro caches its instrument on first use via std::call_once, so that first pre-meter use latched a null instrument for the process lifetime and the metric silently never recorded (observed: rpc_in_flight_requests, emitted from PerfLogImp::rpcStart, never appeared). Extract telemetry_->start() and the metricsRegistry_->start() block into a new private ApplicationImp::startTelemetry() and call it from setup() just before the [rpc_startup] loop, so the meter is live before any metric-emitting code runs. The call sits after overlay_ (and the other subsystems the observable- gauge callbacks read) is constructed, since starting the metrics reader thread earlier would let its callbacks call getOverlay() before overlay_ exists. perfLog_->start() stays in start() — the macro call sites do not depend on it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1163,6 +1163,29 @@ private:
|
||||
void
|
||||
startGenesisLedger();
|
||||
|
||||
/**
|
||||
* Start the tracing and metrics pipelines.
|
||||
*
|
||||
* Called once from setup(), just before the [rpc_startup] loop. Starting
|
||||
* here (rather than in start()) guarantees the OTel MeterProvider is live
|
||||
* before any metric-emitting code runs — including startup RPCs, whose
|
||||
* PerfLog instrumentation records a call-site metric. A call-site metric
|
||||
* macro caches its instrument on first use via std::call_once; if that
|
||||
* first use happens while the meter is still empty, the instrument latches
|
||||
* null for the process lifetime and the metric silently never records.
|
||||
*
|
||||
* The call site sits after overlay_ and the other subsystems are
|
||||
* constructed, because the metrics reader thread starts here and its
|
||||
* observable-gauge callbacks read that state (e.g. getOverlay()); starting
|
||||
* earlier would let the reader observe a half-built application.
|
||||
*
|
||||
* @pre nodeIdentity_ is populated (needed for the service_instance_id
|
||||
* fallback), metricsRegistry_ is constructed, and overlay_ (and the other
|
||||
* subsystems read by observable-gauge callbacks) are constructed.
|
||||
*/
|
||||
void
|
||||
startTelemetry();
|
||||
|
||||
std::shared_ptr<Ledger>
|
||||
getLastFullLedger();
|
||||
|
||||
@@ -1347,7 +1370,8 @@ ApplicationImp::setup(boost::program_options::variables_map const& cmdline)
|
||||
telemetry_->setServiceInstanceId(toBase58(TokenType::NodePublic, nodeIdentity_->first));
|
||||
|
||||
// Create the OTel MetricsRegistry for gap-fill metrics (counters,
|
||||
// histograms, observable gauges). It is started later in start().
|
||||
// histograms, observable gauges). It is started later, just before the
|
||||
// [rpc_startup] loop (see startTelemetry()).
|
||||
metricsRegistry_ = std::make_unique<telemetry::MetricsRegistry>(
|
||||
telemetry_->isEnabled(), *this, logs_->journal("MetricsRegistry"));
|
||||
|
||||
@@ -1492,6 +1516,16 @@ ApplicationImp::setup(boost::program_options::variables_map const& cmdline)
|
||||
JLOG(journal_.warn()) << "*** standalone signing solution as soon as possible.";
|
||||
}
|
||||
|
||||
// Start telemetry and metrics now — before the [rpc_startup] loop below —
|
||||
// so the OTel meter is live before any metric-emitting code runs. Startup
|
||||
// RPCs invoke PerfLog instrumentation that records a call-site metric; a
|
||||
// metric macro caches its instrument on first use, so a first use before
|
||||
// the meter exists would latch null for the process lifetime. Placed here,
|
||||
// after overlay_ and the other subsystems the observable-gauge callbacks
|
||||
// read are constructed, so the metrics reader thread never observes a
|
||||
// half-built application.
|
||||
startTelemetry();
|
||||
|
||||
//
|
||||
// Execute start up rpc commands.
|
||||
//
|
||||
@@ -1563,6 +1597,12 @@ ApplicationImp::start(bool withTimers)
|
||||
|
||||
ledgerCleaner_->start();
|
||||
perfLog_->start();
|
||||
}
|
||||
|
||||
void
|
||||
ApplicationImp::startTelemetry()
|
||||
{
|
||||
// Start tracing first so subsequent startup/early activity can be traced.
|
||||
telemetry_->start();
|
||||
|
||||
// Start the metrics pipeline after telemetry; the endpoint uses the
|
||||
|
||||
Reference in New Issue
Block a user