fix(telemetry): start telemetry + metrics before [rpc_startup]

Metric-emitting code that first runs during ApplicationImp::setup() (the
[rpc_startup] loop) executed while the OTel meter was still empty, because
telemetry_ and metricsRegistry_ were started later in start(). A call-site
metric macro caches its instrument on first use via std::call_once, so that
first pre-meter use latched a null instrument for the process lifetime and the
metric silently never recorded (observed: rpc_in_flight_requests, emitted from
PerfLogImp::rpcStart, never appeared).

Extract telemetry_->start() and the metricsRegistry_->start() block into a new
private ApplicationImp::startTelemetry() and call it from setup() just before
the [rpc_startup] loop, so the meter is live before any metric-emitting code
runs. The call sits after overlay_ (and the other subsystems the observable-
gauge callbacks read) is constructed, since starting the metrics reader thread
earlier would let its callbacks call getOverlay() before overlay_ exists.
perfLog_->start() stays in start() — the macro call sites do not depend on it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pratik Mankawde
2026-07-22 14:35:55 +01:00
parent f59e34192a
commit 93e95bdc03

View File

@@ -1163,6 +1163,29 @@ private:
void
startGenesisLedger();
/**
* Start the tracing and metrics pipelines.
*
* Called once from setup(), just before the [rpc_startup] loop. Starting
* here (rather than in start()) guarantees the OTel MeterProvider is live
* before any metric-emitting code runs — including startup RPCs, whose
* PerfLog instrumentation records a call-site metric. A call-site metric
* macro caches its instrument on first use via std::call_once; if that
* first use happens while the meter is still empty, the instrument latches
* null for the process lifetime and the metric silently never records.
*
* The call site sits after overlay_ and the other subsystems are
* constructed, because the metrics reader thread starts here and its
* observable-gauge callbacks read that state (e.g. getOverlay()); starting
* earlier would let the reader observe a half-built application.
*
* @pre nodeIdentity_ is populated (needed for the service_instance_id
* fallback), metricsRegistry_ is constructed, and overlay_ (and the other
* subsystems read by observable-gauge callbacks) are constructed.
*/
void
startTelemetry();
std::shared_ptr<Ledger>
getLastFullLedger();
@@ -1347,7 +1370,8 @@ ApplicationImp::setup(boost::program_options::variables_map const& cmdline)
telemetry_->setServiceInstanceId(toBase58(TokenType::NodePublic, nodeIdentity_->first));
// Create the OTel MetricsRegistry for gap-fill metrics (counters,
// histograms, observable gauges). It is started later in start().
// histograms, observable gauges). It is started later, just before the
// [rpc_startup] loop (see startTelemetry()).
metricsRegistry_ = std::make_unique<telemetry::MetricsRegistry>(
telemetry_->isEnabled(), *this, logs_->journal("MetricsRegistry"));
@@ -1492,6 +1516,16 @@ ApplicationImp::setup(boost::program_options::variables_map const& cmdline)
JLOG(journal_.warn()) << "*** standalone signing solution as soon as possible.";
}
// Start telemetry and metrics now — before the [rpc_startup] loop below —
// so the OTel meter is live before any metric-emitting code runs. Startup
// RPCs invoke PerfLog instrumentation that records a call-site metric; a
// metric macro caches its instrument on first use, so a first use before
// the meter exists would latch null for the process lifetime. Placed here,
// after overlay_ and the other subsystems the observable-gauge callbacks
// read are constructed, so the metrics reader thread never observes a
// half-built application.
startTelemetry();
//
// Execute start up rpc commands.
//
@@ -1563,6 +1597,12 @@ ApplicationImp::start(bool withTimers)
ledgerCleaner_->start();
perfLog_->start();
}
void
ApplicationImp::startTelemetry()
{
// Start tracing first so subsequent startup/early activity can be traced.
telemetry_->start();
// Start the metrics pipeline after telemetry; the endpoint uses the