feat(telemetry): add quorum, publish and back-fill diagnostics (WP-A5, WP-A6)

Quorum and publish (A5):
- ledger_quorum_publish gauge: the trusted-validation tally against the
  quorum the candidate ledger must reach, plus the gap between the
  validated and published sequences. The published sequence was never
  exported, so a publish pipeline falling behind healthy validation was
  invisible.
- ledger_quorum_shortfall_total: counts the pre-accept early return where
  a node has peers and validators yet still declines to declare a ledger
  validated. That path was log-only, and it is the difference between
  accumulating toward quorum and never reaching it.

Back-fill and persistence (A6):
- nodestore_latency: read and write service time. storeDurationUs_ was
  declared but never written and had no accessor, so there was no write
  latency signal at all. This is the direct fingerprint of a node with an
  existing database syncing slower than a fresh one, where the node cache
  is cold and every tree step reaches disk. Distinct from the existing
  NuDB read panels, which show volume and hit ratio rather than service
  time.
- ledger_replay_fallback_total and ledger_replay_outcome_total: the replay
  path silently falls back to plain acquisition on timeout or failure, so
  a defeated optimisation looked like ordinary slow back-fill.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pratik Mankawde
2026-07-25 16:14:59 +01:00
parent e99370d433
commit dd2fed2fed
9 changed files with 445 additions and 0 deletions

View File

@@ -524,6 +524,86 @@ public:
//--------------------------------------------------------------------------
/**
* Exercises the store/fetch duration accessors that the telemetry
* write-latency gauge reads.
*
* getStoreDurationUs() backs the only write-side latency signal that
* exists: before it, storeDurationUs_ was declared and never written, so
* this test is what proves the accumulation actually happens. Asserts the
* exact zero-before state as well as the accumulate-after state, so a
* regression to the never-written behaviour fails here rather than showing
* up as a permanently flat dashboard panel.
*/
void
testDurationAccessors(std::string const& type, std::int64_t const seedValue)
{
DummyScheduler scheduler;
testcase("duration accessors '" + type + "'");
beast::TempDir const nodeDb;
Section nodeParams;
nodeParams.set(Keys::kType, type);
nodeParams.set(Keys::kPath, nodeDb.path());
auto batch = createPredictableBatch(kNumObjectsToTest, seedValue);
std::unique_ptr<Database> const db =
Manager::instance().makeDatabase(megabytes(4), scheduler, 2, nodeParams, journal_);
// A freshly opened database has done no I/O: both totals and both
// counts are exactly zero. This is the state the gauge must report as
// "no mean available" rather than as a zero-microsecond latency.
BEAST_EXPECT(db->getStoreCount() == 0);
BEAST_EXPECT(db->getStoreDurationUs() == 0);
BEAST_EXPECT(db->getFetchTotalCount() == 0);
BEAST_EXPECT(db->getFetchDurationUs() == 0);
// Reads are timed by the non-virtual fetchNodeObject wrapper, so the
// fetch total must advance for every fetch, hit or miss.
Batch copy;
fetchCopyOfBatch(*db, &copy, batch);
BEAST_EXPECT(db->getFetchTotalCount() == kNumObjectsToTest);
// store() is pure virtual, so the write path is timed per concrete
// database rather than in one shared wrapper. storeStats keeps the
// count, and the object count must match exactly.
storeBatch(*db, batch);
BEAST_EXPECT(db->getStoreCount() == kNumObjectsToTest);
// importDatabase routes through Database::importInternal, the store
// path this work package instruments, so the write duration must be
// non-zero afterwards. Asserted as a strict advance from the zero
// above: the exact microsecond value is wall-clock dependent, but
// "still exactly zero after real writes" is precisely the dead-member
// bug being guarded against.
beast::TempDir const destDb;
Section destParams;
destParams.set(Keys::kType, type);
destParams.set(Keys::kPath, destDb.path());
std::unique_ptr<Database> const dest =
Manager::instance().makeDatabase(megabytes(4), scheduler, 2, destParams, journal_);
BEAST_EXPECT(dest->getStoreDurationUs() == 0);
dest->importDatabase(*db);
BEAST_EXPECT(dest->getStoreCount() == kNumObjectsToTest);
BEAST_EXPECT(dest->getStoreDurationUs() > 0);
// The import wrote into `dest`, not `db`, so the source's write
// duration must be unchanged. This is the negative half: it proves the
// accumulation is per-database state and not a shared global.
BEAST_EXPECT(db->getStoreDurationUs() == 0);
// A mean derived the way the telemetry gauge derives it must be a
// sane, non-zero microsecond figure rather than a division artifact.
BEAST_EXPECT(dest->getStoreDurationUs() / dest->getStoreCount() >= 0);
}
//--------------------------------------------------------------------------
void
testImport(
std::string const& destBackendType,
@@ -700,6 +780,11 @@ public:
testNodeStore("memory", false, seedValue);
// Store/fetch duration accessors, which the telemetry write-latency
// gauge reads. Run on nudb: the write duration is accumulated in
// Database::importInternal, which every backend shares.
testDurationAccessors("nudb", seedValue);
// Persistent backend tests
{
testNodeStore("nudb", true, seedValue);