fix(telemetry): detach metrics gauge callbacks before Application services stop

MetricsRegistry observable-gauge callbacks run on the OTel reader thread
and read live state from nodeStore_, overlay_, networkOPs_, ledgerMaster,
inboundLedgers, loadManager, and others. The old shutdown sequence called
metricsRegistry_->stop() AFTER all those services were already stopped,
which left a race window between each service's stop() and the final
provider_->ForceFlush() during which a callback could dereference
already-stopped service state. The try/catch guards in each callback
mitigated crashes but not reads from freed members.

- Add MetricsRegistry::detachCallbacks() that sets an atomic<bool>
  callbacksDetached_ with release ordering. Idempotent.
- Guard every ObservableGauge callback entry with an acquire-load of the
  same flag and return early if it is set. Covers all 15 registered
  callbacks (cacheHitRate, txq, objectCount, loadFactor, nodeStore,
  serverInfo, buildInfo, completeLedgers, dbMetrics, validatorHealth,
  peerQuality, ledgerEconomy, stateTracking, storageDetail,
  validationAgreement).
- Application::run() shutdown sequence now calls
  metricsRegistry_->detachCallbacks() right after m_loadManager->stop()
  and BEFORE m_shaMapStore, m_jobQueue, overlay_, grpcServer_,
  m_networkOPs, serverHandler_, m_ledgerReplayer, m_inboundTransactions,
  m_inboundLedgers, ledgerCleaner_, m_nodeStore, perfLog_ are stopped.
  The acquire/release pair guarantees subsequent reader-thread ticks see
  the detach before they dereference stopped services.
- metricsRegistry_->stop() keeps setting the flag as a belt-and-suspenders
  defense in case a future caller forgets to detach first.
- Drop the misleading "No explicit RemoveCallback is needed" comment
  from stop(); provider destruction alone does not beat the reader
  thread to already-freed state.

The objectCountGauge callback previously discarded its state pointer
via `void* /* state */`; restore the state argument so it can access
self->callbacksDetached_ too.
This commit is contained in:
Pratik Mankawde
2026-05-14 17:20:52 +01:00
parent 145b1469d6
commit 2735e4ac78
3 changed files with 93 additions and 4 deletions

View File

@@ -1686,6 +1686,19 @@ ApplicationImp::run()
// The order of these stop calls is delicate.
// Re-ordering them risks undefined behavior.
m_loadManager->stop();
// Detach MetricsRegistry observable-gauge callbacks BEFORE stopping
// any service the callbacks read from. The callbacks run on the OTel
// reader thread and touch nodeStore_, overlay_, networkOPs_,
// ledgerMaster, inboundLedgers, etc. A final tick that fires after
// one of those services has shut down would dereference dangling
// state. detachCallbacks() flips an atomic flag every callback
// acquire-loads at its entry, so subsequent ticks become no-ops.
// The final provider teardown still happens in metricsRegistry_->stop()
// farther down.
if (metricsRegistry_)
metricsRegistry_->detachCallbacks();
m_shaMapStore->stop();
m_jobQueue->stop();
if (overlay_)