mirror of
https://github.com/XRPLF/rippled.git
synced 2026-08-21 22:30:57 +00:00
feat(telemetry): add quorum, publish and back-fill diagnostics (WP-A5, WP-A6)
Quorum and publish (A5): - ledger_quorum_publish gauge: the trusted-validation tally against the quorum the candidate ledger must reach, plus the gap between the validated and published sequences. The published sequence was never exported, so a publish pipeline falling behind healthy validation was invisible. - ledger_quorum_shortfall_total: counts the pre-accept early return where a node has peers and validators yet still declines to declare a ledger validated. That path was log-only, and it is the difference between accumulating toward quorum and never reaching it. Back-fill and persistence (A6): - nodestore_latency: read and write service time. storeDurationUs_ was declared but never written and had no accessor, so there was no write latency signal at all. This is the direct fingerprint of a node with an existing database syncing slower than a fresh one, where the node cache is cold and every tree step reaches disk. Distinct from the existing NuDB read panels, which show volume and hit ratio rather than service time. - ledger_replay_fallback_total and ledger_replay_outcome_total: the replay path silently falls back to plain acquisition on timeout or failure, so a defeated optimisation looked like ordinary slow back-fill. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -11,6 +11,7 @@
|
||||
#include <xrpl/nodestore/Scheduler.h>
|
||||
|
||||
#include <atomic>
|
||||
#include <chrono>
|
||||
#include <condition_variable>
|
||||
#include <cstdint>
|
||||
#include <functional>
|
||||
@@ -197,6 +198,54 @@ public:
|
||||
return fetchSz_;
|
||||
}
|
||||
|
||||
/**
|
||||
* Cumulative microseconds spent inside store() calls.
|
||||
*
|
||||
* Pairs with getStoreCount() to derive mean write latency
|
||||
* (`duration / count`), mirroring how the read side pairs
|
||||
* getFetchDurationUs() with getFetchTotalCount().
|
||||
*
|
||||
* This is the "an existing DB syncs slower than a fresh one" signal: the
|
||||
* read counters cannot show it, because back-fill is write-bound. Until
|
||||
* now `storeDurationUs_` was declared but never written, so no write-side
|
||||
* latency existed anywhere.
|
||||
*
|
||||
* @return Total microseconds accumulated across every completed store.
|
||||
*
|
||||
* @note Thread-safe: a single relaxed atomic load. Cheap enough for a
|
||||
* periodic observer (the telemetry reader ticks every ~10 s). Relaxed is
|
||||
* sufficient because the value is a monotonic statistic, not a
|
||||
* synchronization signal — a reader that observes a slightly stale total
|
||||
* simply reports a slightly stale mean.
|
||||
* @note Monotonic and never reset, so a dashboard must take a rate or a
|
||||
* delta of both this and getStoreCount() over the same window to see
|
||||
* current latency rather than the since-boot average.
|
||||
*/
|
||||
[[nodiscard]] std::uint64_t
|
||||
getStoreDurationUs() const noexcept
|
||||
{
|
||||
return storeDurationUs_.load(std::memory_order_relaxed);
|
||||
}
|
||||
|
||||
/**
|
||||
* Cumulative microseconds spent inside fetchNodeObject() calls.
|
||||
*
|
||||
* Pairs with getFetchTotalCount() to derive mean read latency. The same
|
||||
* total is already published as the `node_reads_duration_us` field of
|
||||
* getCountsJson(); this accessor exposes it directly so a caller need not
|
||||
* build a json::Value and parse a decimal string back to an integer.
|
||||
*
|
||||
* @return Total microseconds accumulated across every completed fetch.
|
||||
*
|
||||
* @note Same threading and monotonicity contract as
|
||||
* getStoreDurationUs().
|
||||
*/
|
||||
[[nodiscard]] std::uint64_t
|
||||
getFetchDurationUs() const noexcept
|
||||
{
|
||||
return fetchDurationUs_.load(std::memory_order_relaxed);
|
||||
}
|
||||
|
||||
void
|
||||
getCountsJson(json::Value& obj);
|
||||
|
||||
@@ -253,6 +302,36 @@ protected:
|
||||
storeSz_ += sz;
|
||||
}
|
||||
|
||||
/**
|
||||
* Accumulate the time one completed store took.
|
||||
*
|
||||
* The write counterpart of the timing fetchNodeObject() already does for
|
||||
* reads. `store()` is pure virtual, so unlike the read path there is no
|
||||
* non-virtual wrapper in this class to time — each concrete database calls
|
||||
* this once per store it completes, and the single conversion to
|
||||
* microseconds lives here rather than being repeated per subclass.
|
||||
*
|
||||
* @param elapsed Wall time the store took, as measured by the caller.
|
||||
*
|
||||
* @note Call once per store operation, never inside a per-tree-node loop:
|
||||
* a ledger write walks thousands of SHAMap nodes and this must stay a
|
||||
* single atomic add on the whole write, matching the one-sample-per-fetch
|
||||
* cost on the read side.
|
||||
* @note Thread-safe: one relaxed atomic add, no lock. Relaxed ordering is
|
||||
* correct because the total is a statistic that is only ever read by a
|
||||
* periodic observer, never used to order other memory operations.
|
||||
* @note A negative duration cannot occur (steady_clock is monotonic), but
|
||||
* a caller passing one would be clamped to zero rather than wrapping the
|
||||
* unsigned total to a huge value.
|
||||
*/
|
||||
void
|
||||
recordStoreDuration(std::chrono::steady_clock::duration elapsed) noexcept
|
||||
{
|
||||
auto const us = std::chrono::duration_cast<std::chrono::microseconds>(elapsed).count();
|
||||
if (us > 0)
|
||||
storeDurationUs_.fetch_add(static_cast<std::uint64_t>(us), std::memory_order_relaxed);
|
||||
}
|
||||
|
||||
// Called by the public import function
|
||||
void
|
||||
importInternal(Backend& dstBackend, Database& srcDB);
|
||||
|
||||
Reference in New Issue
Block a user