feat(telemetry): add quorum, publish and back-fill diagnostics (WP-A5, WP-A6)

Quorum and publish (A5):
- ledger_quorum_publish gauge: the trusted-validation tally against the
  quorum the candidate ledger must reach, plus the gap between the
  validated and published sequences. The published sequence was never
  exported, so a publish pipeline falling behind healthy validation was
  invisible.
- ledger_quorum_shortfall_total: counts the pre-accept early return where
  a node has peers and validators yet still declines to declare a ledger
  validated. That path was log-only, and it is the difference between
  accumulating toward quorum and never reaching it.

Back-fill and persistence (A6):
- nodestore_latency: read and write service time. storeDurationUs_ was
  declared but never written and had no accessor, so there was no write
  latency signal at all. This is the direct fingerprint of a node with an
  existing database syncing slower than a fresh one, where the node cache
  is cold and every tree step reaches disk. Distinct from the existing
  NuDB read panels, which show volume and hit ratio rather than service
  time.
- ledger_replay_fallback_total and ledger_replay_outcome_total: the replay
  path silently falls back to plain acquisition on timeout or failure, so
  a defeated optimisation looked like ordinary slow back-fill.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pratik Mankawde
2026-07-25 16:14:59 +01:00
parent e99370d433
commit dd2fed2fed
9 changed files with 445 additions and 0 deletions

View File

@@ -11,6 +11,7 @@
#include <xrpl/nodestore/Scheduler.h>
#include <atomic>
#include <chrono>
#include <condition_variable>
#include <cstdint>
#include <functional>
@@ -197,6 +198,54 @@ public:
return fetchSz_;
}
/**
* Cumulative microseconds spent inside store() calls.
*
* Pairs with getStoreCount() to derive mean write latency
* (`duration / count`), mirroring how the read side pairs
* getFetchDurationUs() with getFetchTotalCount().
*
* This is the "an existing DB syncs slower than a fresh one" signal: the
* read counters cannot show it, because back-fill is write-bound. Until
* now `storeDurationUs_` was declared but never written, so no write-side
* latency existed anywhere.
*
* @return Total microseconds accumulated across every completed store.
*
* @note Thread-safe: a single relaxed atomic load. Cheap enough for a
* periodic observer (the telemetry reader ticks every ~10 s). Relaxed is
* sufficient because the value is a monotonic statistic, not a
* synchronization signal — a reader that observes a slightly stale total
* simply reports a slightly stale mean.
* @note Monotonic and never reset, so a dashboard must take a rate or a
* delta of both this and getStoreCount() over the same window to see
* current latency rather than the since-boot average.
*/
[[nodiscard]] std::uint64_t
getStoreDurationUs() const noexcept
{
return storeDurationUs_.load(std::memory_order_relaxed);
}
/**
* Cumulative microseconds spent inside fetchNodeObject() calls.
*
* Pairs with getFetchTotalCount() to derive mean read latency. The same
* total is already published as the `node_reads_duration_us` field of
* getCountsJson(); this accessor exposes it directly so a caller need not
* build a json::Value and parse a decimal string back to an integer.
*
* @return Total microseconds accumulated across every completed fetch.
*
* @note Same threading and monotonicity contract as
* getStoreDurationUs().
*/
[[nodiscard]] std::uint64_t
getFetchDurationUs() const noexcept
{
return fetchDurationUs_.load(std::memory_order_relaxed);
}
void
getCountsJson(json::Value& obj);
@@ -253,6 +302,36 @@ protected:
storeSz_ += sz;
}
/**
* Accumulate the time one completed store took.
*
* The write counterpart of the timing fetchNodeObject() already does for
* reads. `store()` is pure virtual, so unlike the read path there is no
* non-virtual wrapper in this class to time — each concrete database calls
* this once per store it completes, and the single conversion to
* microseconds lives here rather than being repeated per subclass.
*
* @param elapsed Wall time the store took, as measured by the caller.
*
* @note Call once per store operation, never inside a per-tree-node loop:
* a ledger write walks thousands of SHAMap nodes and this must stay a
* single atomic add on the whole write, matching the one-sample-per-fetch
* cost on the read side.
* @note Thread-safe: one relaxed atomic add, no lock. Relaxed ordering is
* correct because the total is a statistic that is only ever read by a
* periodic observer, never used to order other memory operations.
* @note A negative duration cannot occur (steady_clock is monotonic), but
* a caller passing one would be clamped to zero rather than wrapping the
* unsigned total to a huge value.
*/
void
recordStoreDuration(std::chrono::steady_clock::duration elapsed) noexcept
{
auto const us = std::chrono::duration_cast<std::chrono::microseconds>(elapsed).count();
if (us > 0)
storeDurationUs_.fetch_add(static_cast<std::uint64_t>(us), std::memory_order_relaxed);
}
// Called by the public import function
void
importInternal(Backend& dstBackend, Database& srcDB);