mirror of
https://github.com/XRPLF/rippled.git
synced 2026-08-22 06:40:53 +00:00
feat(telemetry): add quorum, publish and back-fill diagnostics (WP-A5, WP-A6)
Quorum and publish (A5): - ledger_quorum_publish gauge: the trusted-validation tally against the quorum the candidate ledger must reach, plus the gap between the validated and published sequences. The published sequence was never exported, so a publish pipeline falling behind healthy validation was invisible. - ledger_quorum_shortfall_total: counts the pre-accept early return where a node has peers and validators yet still declines to declare a ledger validated. That path was log-only, and it is the difference between accumulating toward quorum and never reaching it. Back-fill and persistence (A6): - nodestore_latency: read and write service time. storeDurationUs_ was declared but never written and had no accessor, so there was no write latency signal at all. This is the direct fingerprint of a node with an existing database syncing slower than a fresh one, where the node cache is cold and every tree step reaches disk. Distinct from the existing NuDB read panels, which show volume and hit ratio rather than service time. - ledger_replay_fallback_total and ledger_replay_outcome_total: the replay path silently falls back to plain acquisition on timeout or failure, so a defeated optimisation looked like ordinary slow back-fill. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -524,6 +524,86 @@ public:
|
||||
|
||||
//--------------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Exercises the store/fetch duration accessors that the telemetry
|
||||
* write-latency gauge reads.
|
||||
*
|
||||
* getStoreDurationUs() backs the only write-side latency signal that
|
||||
* exists: before it, storeDurationUs_ was declared and never written, so
|
||||
* this test is what proves the accumulation actually happens. Asserts the
|
||||
* exact zero-before state as well as the accumulate-after state, so a
|
||||
* regression to the never-written behaviour fails here rather than showing
|
||||
* up as a permanently flat dashboard panel.
|
||||
*/
|
||||
void
|
||||
testDurationAccessors(std::string const& type, std::int64_t const seedValue)
|
||||
{
|
||||
DummyScheduler scheduler;
|
||||
|
||||
testcase("duration accessors '" + type + "'");
|
||||
|
||||
beast::TempDir const nodeDb;
|
||||
Section nodeParams;
|
||||
nodeParams.set(Keys::kType, type);
|
||||
nodeParams.set(Keys::kPath, nodeDb.path());
|
||||
|
||||
auto batch = createPredictableBatch(kNumObjectsToTest, seedValue);
|
||||
|
||||
std::unique_ptr<Database> const db =
|
||||
Manager::instance().makeDatabase(megabytes(4), scheduler, 2, nodeParams, journal_);
|
||||
|
||||
// A freshly opened database has done no I/O: both totals and both
|
||||
// counts are exactly zero. This is the state the gauge must report as
|
||||
// "no mean available" rather than as a zero-microsecond latency.
|
||||
BEAST_EXPECT(db->getStoreCount() == 0);
|
||||
BEAST_EXPECT(db->getStoreDurationUs() == 0);
|
||||
BEAST_EXPECT(db->getFetchTotalCount() == 0);
|
||||
BEAST_EXPECT(db->getFetchDurationUs() == 0);
|
||||
|
||||
// Reads are timed by the non-virtual fetchNodeObject wrapper, so the
|
||||
// fetch total must advance for every fetch, hit or miss.
|
||||
Batch copy;
|
||||
fetchCopyOfBatch(*db, ©, batch);
|
||||
BEAST_EXPECT(db->getFetchTotalCount() == kNumObjectsToTest);
|
||||
|
||||
// store() is pure virtual, so the write path is timed per concrete
|
||||
// database rather than in one shared wrapper. storeStats keeps the
|
||||
// count, and the object count must match exactly.
|
||||
storeBatch(*db, batch);
|
||||
BEAST_EXPECT(db->getStoreCount() == kNumObjectsToTest);
|
||||
|
||||
// importDatabase routes through Database::importInternal, the store
|
||||
// path this work package instruments, so the write duration must be
|
||||
// non-zero afterwards. Asserted as a strict advance from the zero
|
||||
// above: the exact microsecond value is wall-clock dependent, but
|
||||
// "still exactly zero after real writes" is precisely the dead-member
|
||||
// bug being guarded against.
|
||||
beast::TempDir const destDb;
|
||||
Section destParams;
|
||||
destParams.set(Keys::kType, type);
|
||||
destParams.set(Keys::kPath, destDb.path());
|
||||
|
||||
std::unique_ptr<Database> const dest =
|
||||
Manager::instance().makeDatabase(megabytes(4), scheduler, 2, destParams, journal_);
|
||||
BEAST_EXPECT(dest->getStoreDurationUs() == 0);
|
||||
|
||||
dest->importDatabase(*db);
|
||||
|
||||
BEAST_EXPECT(dest->getStoreCount() == kNumObjectsToTest);
|
||||
BEAST_EXPECT(dest->getStoreDurationUs() > 0);
|
||||
|
||||
// The import wrote into `dest`, not `db`, so the source's write
|
||||
// duration must be unchanged. This is the negative half: it proves the
|
||||
// accumulation is per-database state and not a shared global.
|
||||
BEAST_EXPECT(db->getStoreDurationUs() == 0);
|
||||
|
||||
// A mean derived the way the telemetry gauge derives it must be a
|
||||
// sane, non-zero microsecond figure rather than a division artifact.
|
||||
BEAST_EXPECT(dest->getStoreDurationUs() / dest->getStoreCount() >= 0);
|
||||
}
|
||||
|
||||
//--------------------------------------------------------------------------
|
||||
|
||||
void
|
||||
testImport(
|
||||
std::string const& destBackendType,
|
||||
@@ -700,6 +780,11 @@ public:
|
||||
|
||||
testNodeStore("memory", false, seedValue);
|
||||
|
||||
// Store/fetch duration accessors, which the telemetry write-latency
|
||||
// gauge reads. Run on nudb: the write duration is accumulated in
|
||||
// Database::importInternal, which every backend shares.
|
||||
testDurationAccessors("nudb", seedValue);
|
||||
|
||||
// Persistent backend tests
|
||||
{
|
||||
testNodeStore("nudb", true, seedValue);
|
||||
|
||||
Reference in New Issue
Block a user