Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill

Resolves the telemetry-startup conflict between the two branches. Both
sides move the telemetry start earlier in setup(); they disagree only on
how far the pipeline had been split at that point.

phase-1b (arriving) moved nodeIdentity_, setServiceInstanceId() and the
telemetry start up to just after the wallet DB is proven usable. phase-9
had split the metrics pipeline in two and left its copy of that block at
the old, later position.

Kept both intentions: the block stays at phase-1b's early position, and
metricsRegistry_ construction moves up with it so it precedes
startTelemetry() -- the metrics half is guarded on the registry existing,
so leaving the construction behind would have started tracing while
silently skipping metrics. phase-9's later copy is dropped as the stale
duplicate. The two-phase split is preserved: startTelemetryGauges() still
runs after overlay_ is constructed, because the observable callbacks read
it and getOverlay() asserts.

Net effect is that the metrics provider now starts earlier than on either
branch, and still before beginConsensus() emits the first spans and the
only operating-mode transition.
This commit is contained in:
Pratik Mankawde
2026-07-30 20:00:04 +01:00
9 changed files with 145 additions and 59 deletions

View File

@@ -1312,6 +1312,39 @@ ApplicationImp::setup(boost::program_options::variables_map const& cmdline)
return false;
}
nodeIdentity_ = getNodeIdentity(*this, cmdline);
// Now that the node identity is known, inject it into the telemetry
// resource attributes — but only if the user didn't already set a
// custom service_instance_id in [telemetry]. The Telemetry object
// was constructed with an empty serviceInstanceId because
// nodeIdentity_ is not available in the member initializer list.
if (!config_->section("telemetry").exists("service_instance_id"))
telemetry_->setServiceInstanceId(toBase58(TokenType::NodePublic, nodeIdentity_->first));
// Create the OTel MetricsRegistry for gap-fill metrics (counters,
// histograms, observable gauges). It must exist before startTelemetry(),
// which starts the metrics half of the pipeline.
metricsRegistry_ = std::make_unique<telemetry::MetricsRegistry>(
telemetry_->isEnabled(), *this, logs_->journal("MetricsRegistry"));
// Start telemetry here, not in start(). Spans and metrics are both emitted
// during the rest of setup() — the first consensus round in
// beginConsensus() below emits spans and records the process's only
// operating-mode transition — and both are dropped unless the pipeline is
// already live.
//
// The position is bounded on both sides:
// - After initRelationalDatabase(): the wallet DB must exist for the node
// identity above, and a DB failure aborts setup(), so starting earlier
// would export a partial stream for a run that never comes up.
// - Before beginConsensus(): that call emits the first consensus spans
// and the only mode-transition counter increment.
//
// Only the observable instruments have to wait for their subsystems; they
// are registered separately by startTelemetryGauges() once overlay_ exists.
startTelemetry();
if (validatorKeys_.keys)
setMaxDisallowedLedger();
@@ -1399,29 +1432,6 @@ ApplicationImp::setup(boost::program_options::variables_map const& cmdline)
orderBookDB_->setup(getLedgerMaster().getCurrentLedger());
nodeIdentity_ = getNodeIdentity(*this, cmdline);
// Now that the node identity is known, inject it into the telemetry
// resource attributes — but only if the user didn't already set a
// custom service_instance_id in [telemetry]. The Telemetry object
// was constructed with an empty serviceInstanceId because
// nodeIdentity_ is not available in the member initializer list.
if (!config_->section("telemetry").exists("service_instance_id"))
telemetry_->setServiceInstanceId(toBase58(TokenType::NodePublic, nodeIdentity_->first));
// Create the OTel MetricsRegistry for gap-fill metrics (counters,
// histograms, observable gauges).
metricsRegistry_ = std::make_unique<telemetry::MetricsRegistry>(
telemetry_->isEnabled(), *this, logs_->journal("MetricsRegistry"));
// Start tracing and the metrics provider right away, so the meter exists
// before anything records a metric. beginConsensus() below emits a
// mode-transition counter, and it is the only mode transition the process
// ever makes — a meter created after it would lose that series entirely.
// Only the observable gauges have to wait; they are registered by
// startTelemetryGauges() once overlay_ exists.
startTelemetry();
if (!cluster_->load(config().section(Sections::kClusterNodes)))
{
JLOG(journal_.fatal()) << "Invalid entry in cluster configuration.";