ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-23 07:10:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(telemetry): emit xrpld_validation_{agreements,missed}_total counters

Browse Source 
Wire the two previously-registered-but-never-incremented validation
counters to ValidationTracker's gross lifetime tallies, exported as
monotonic ObservableCounters. New gross atomics count each ledger once at
first classification and are never adjusted on late repair, keeping the
_total counters monotonic and additive (agreements_total + missed_total ==
ledgers reconciled); the repair-aware windowed view stays on the existing
xrpld_validation_agreement gauge. The validator-health dashboard panels
that already query these names now render data instead of "No data".

Also de-stale 09-data-collection-reference.md: §5b documented flat metric
names (xrpld_cache_SLE_hit_rate, ...) that the code never emits — it emits
labeled gauges (xrpld_cache_metrics{metric="SLE_hit_rate"}). Replace the
stale flat-name tables with a pointer to the canonical labeled section,
reconcile the contradictory headline counts, and correct xrpld_job_count
to its real exported name xrpld_jobq_job_count.

Adds two GTests asserting gross tallies stay frozen on repair while net
totals move, plus the additive invariant.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pratik Mankawde
2026-06-05 18:29:29 +01:00
parent 0d1d1aa0e1
commit dc5bb4b35c
6 changed files with 288 additions and 119 deletions
Download Patch File Download Diff File Expand all files Collapse all files

90
src/tests/libxrpl/telemetry/ValidationTracker.cpp
View File

@@ -132,6 +132,8 @@ TEST_F(ValidationTrackerTest, EmptyWindowReturnsZero)
EXPECT_EQ(tracker_.missed24h(), 0u);
EXPECT_EQ(tracker_.totalAgreements(), 0u);
EXPECT_EQ(tracker_.totalMissed(), 0u);
EXPECT_EQ(tracker_.totalAgreementsEver(), 0u);
EXPECT_EQ(tracker_.totalMissedEver(), 0u);
EXPECT_EQ(tracker_.totalValidationsSent(), 0u);
EXPECT_EQ(tracker_.totalValidationsChecked(), 0u);
}
@@ -282,3 +284,91 @@ TEST_F(ValidationTrackerTest, OnlyWeValidated)
EXPECT_EQ(tracker_.missed1h(), 1u);
EXPECT_DOUBLE_EQ(tracker_.agreementPct1h(), 0.0);
}
// ---------------------------------------------------------------
// 10. Gross miss tally is monotonic across a late repair
// The gross lifetime tallies (totalAgreementsEver/totalMissedEver)
// back the monotonic Prometheus _total counters. A late repair must
// move the NET totals (miss -> agreement) but must NOT move the gross
// tallies: a miss already counted stays counted, and the repair does
// not add a second (agreement) count for the same ledger.
// ---------------------------------------------------------------
TEST_F(ValidationTrackerTest, GrossMissedNeverDecrementsOnRepair)
{
auto const hash = makeHash(10);
LedgerIndex const seq = 1000;
// Network validates, we do not (yet).
tracker_.recordNetworkValidation(hash, seq);
// Grace period elapses -- reconciled as a miss.
std::this_thread::sleep_for(std::chrono::seconds(9));
tracker_.reconcile();
// Net and gross both show exactly one initial miss, zero agreements.
EXPECT_EQ(tracker_.totalMissed(), 1u);
EXPECT_EQ(tracker_.totalMissedEver(), 1u);
EXPECT_EQ(tracker_.totalAgreements(), 0u);
EXPECT_EQ(tracker_.totalAgreementsEver(), 0u);
// Late arrival of our validation repairs the miss to an agreement.
tracker_.recordOurValidation(hash, seq);
tracker_.reconcile();
// Net totals reflect the repair...
EXPECT_EQ(tracker_.totalMissed(), 0u);
EXPECT_EQ(tracker_.totalAgreements(), 1u);
// ...but the gross tallies are frozen at first classification: the miss
// stays counted and no agreement was added (repair path excluded).
EXPECT_EQ(tracker_.totalMissedEver(), 1u);
EXPECT_EQ(tracker_.totalAgreementsEver(), 0u);
}
// ---------------------------------------------------------------
// 11. Gross tallies count initial classification only (additive)
// With a mix of initial agreements and misses the gross tallies equal
// the net totals. A subsequent repair shifts the net totals but leaves
// the gross tallies unchanged, and the gross sum equals the number of
// reconciled ledgers (the additive invariant the _total counters rely on).
// ---------------------------------------------------------------
TEST_F(ValidationTrackerTest, GrossAgreementsCountInitialOnly)
{
// 3 initial agreements: both sides validate.
for (int i = 1; i <= 3; ++i)
{
auto const h = makeHash(static_cast<std::uint64_t>(i));
tracker_.recordOurValidation(h, static_cast<LedgerIndex>(i));
tracker_.recordNetworkValidation(h, static_cast<LedgerIndex>(i));
}
// 2 initial misses: only network validates.
for (int i = 4; i <= 5; ++i)
{
auto const h = makeHash(static_cast<std::uint64_t>(i));
tracker_.recordNetworkValidation(h, static_cast<LedgerIndex>(i));
}
// Grace period elapses -- all five reconciled at first classification.
std::this_thread::sleep_for(std::chrono::seconds(9));
tracker_.reconcile();
// Before any repair, gross equals net.
EXPECT_EQ(tracker_.totalAgreements(), 3u);
EXPECT_EQ(tracker_.totalAgreementsEver(), 3u);
EXPECT_EQ(tracker_.totalMissed(), 2u);
EXPECT_EQ(tracker_.totalMissedEver(), 2u);
// Repair one of the misses (hash 4) within the repair window.
tracker_.recordOurValidation(makeHash(4), 4);
tracker_.reconcile();
// Net totals shift by the repair...
EXPECT_EQ(tracker_.totalAgreements(), 4u);
EXPECT_EQ(tracker_.totalMissed(), 1u);
// ...gross tallies stay at the initial classification.
EXPECT_EQ(tracker_.totalAgreementsEver(), 3u);
EXPECT_EQ(tracker_.totalMissedEver(), 2u);
// Additive invariant: gross agree + gross miss == ledgers reconciled.
EXPECT_EQ(tracker_.totalAgreementsEver() + tracker_.totalMissedEver(), 5u);
}

74
src/xrpld/telemetry/MetricsRegistry.cpp
View File

@@ -244,10 +244,9 @@ MetricsRegistry::start(std::string const& endpoint, std::string const& instanceI
"xrpld_txq_expired_total", "Total transactions expired out of the transaction queue");
txqDroppedCounter_ = meter_->CreateUInt64Counter(
"xrpld_txq_dropped_total", "Total transactions refused admission to the queue by reason");
validationAgreementsCounter_ = meter_->CreateUInt64Counter(
"xrpld_validation_agreements_total", "Total validation agreements");
validationMissedCounter_ =
meter_->CreateUInt64Counter("xrpld_validation_missed_total", "Total validation misses");
// Note: xrpld_validation_agreements_total / xrpld_validation_missed_total
// are monotonic ObservableCounters created in registerValidationTotalsCounters()
// (below), observed from ValidationTracker's gross lifetime tallies.
// Register all observable (async) gauges.
registerAsyncGauges();
@@ -441,6 +440,7 @@ MetricsRegistry::registerAsyncGauges()
registerStateTrackingGauge();
registerStorageDetailGauge();
registerValidationAgreementGauge();
registerValidationTotalsCounters();
}
void
@@ -1325,13 +1325,67 @@ MetricsRegistry::registerValidationAgreementGauge()
}
},
this);
}
// Note: validationAgreementsCounter_ and validationMissedCounter_ are
// created above but not currently incremented. The
// xrpld_validation_agreement gauge already provides agreement and miss
// counts from ValidationTracker's rolling windows and lifetime totals.
// These counters are reserved for future use if a push-style counter
// integration with ValidationTracker is desired.
void
MetricsRegistry::registerValidationTotalsCounters()
{
// Lifetime validation agreement/miss counters.
//
// These are monotonic ObservableCounters (not the sync Counters they used
// to be): a Prometheus _total must never decrease, but ValidationTracker's
// NET totals are non-monotonic (a late repair decrements the net miss
// count). We therefore observe the tracker's GROSS lifetime tallies, which
// count each ledger once at first classification and are never adjusted on
// repair (initial-classification semantics — see ValidationTracker). The
// repaired/agreement view remains available from xrpld_validation_agreement.
//
// reconcile() is called first so pending events are resolved before the
// tallies are read; the callback fires every ~10 s from the
// PeriodicExportingMetricReader thread.
validationAgreementsObservable_ = meter_->CreateInt64ObservableCounter(
"xrpld_validation_agreements_total",
"Lifetime validations that initially agreed with network consensus");
validationAgreementsObservable_->AddCallback(
[](opentelemetry::metrics::ObserverResult result, void* state) {
auto* self = static_cast<MetricsRegistry*>(state);
if (self->callbacksDetached_.load(std::memory_order_acquire))
return;
try
{
self->validationTracker_.reconcile();
opentelemetry::nostd::get<opentelemetry::nostd::shared_ptr<
opentelemetry::metrics::ObserverResultT<int64_t>>>(result)
->Observe(static_cast<int64_t>(self->validationTracker_.totalAgreementsEver()));
}
catch (...) // NOLINT(bugprone-empty-catch)
{
// Silently skip on error.
}
},
this);
validationMissedObservable_ = meter_->CreateInt64ObservableCounter(
"xrpld_validation_missed_total",
"Lifetime validations that initially missed network consensus");
validationMissedObservable_->AddCallback(
[](opentelemetry::metrics::ObserverResult result, void* state) {
auto* self = static_cast<MetricsRegistry*>(state);
if (self->callbacksDetached_.load(std::memory_order_acquire))
return;
try
{
self->validationTracker_.reconcile();
opentelemetry::nostd::get<opentelemetry::nostd::shared_ptr<
opentelemetry::metrics::ObserverResultT<int64_t>>>(result)
->Observe(static_cast<int64_t>(self->validationTracker_.totalMissedEver()));
}
catch (...) // NOLINT(bugprone-empty-catch)
{
// Silently skip on error.
}
},
this);
}
#endif // XRPL_ENABLE_TELEMETRY

19
src/xrpld/telemetry/MetricsRegistry.h
View File

@@ -529,13 +529,16 @@ private:
/// Counter: xrpld_txq_dropped_total{reason} — incremented when a transaction is refused
/// admission to the queue.
opentelemetry::nostd::unique_ptr<opentelemetry::metrics::Counter<uint64_t>> txqDroppedCounter_;
/// Counter: xrpld_validation_agreements_total — incremented by ValidationTracker on
/// agreement.
opentelemetry::nostd::unique_ptr<opentelemetry::metrics::Counter<uint64_t>>
validationAgreementsCounter_;
/// Counter: xrpld_validation_missed_total — incremented by ValidationTracker on miss.
opentelemetry::nostd::unique_ptr<opentelemetry::metrics::Counter<uint64_t>>
validationMissedCounter_;
/// ObservableCounter: xrpld_validation_agreements_total — observed from
/// ValidationTracker::totalAgreementsEver() (monotonic gross lifetime
/// tally, initial-classification semantics).
opentelemetry::nostd::shared_ptr<opentelemetry::metrics::ObservableInstrument>
validationAgreementsObservable_;
/// ObservableCounter: xrpld_validation_missed_total — observed from
/// ValidationTracker::totalMissedEver() (monotonic gross lifetime tally,
/// initial-classification semantics).
opentelemetry::nostd::shared_ptr<opentelemetry::metrics::ObservableInstrument>
validationMissedObservable_;
/** Register all observable gauge callbacks with the OTel SDK.
Dispatches to one helper per metric domain so that each helper
@@ -580,6 +583,8 @@ private:
registerStorageDetailGauge(); // Task 7.13
void
registerValidationAgreementGauge(); // Task 7.15
void
registerValidationTotalsCounters(); // gap-fill: lifetime agree/miss _total
#endif // XRPL_ENABLE_TELEMETRY
};

45
src/xrpld/telemetry/ValidationTracker.h
View File

@@ -186,6 +186,26 @@ public:
uint64_t
totalMissed() const;
/** Lifetime agreements counted at first classification only.
* @note Unlike totalAgreements(), this is strictly monotonic: it is
* incremented only when a ledger is first reconciled as an agreement and
* is never adjusted by a late repair. It backs the monotonic Prometheus
* counter xrpld_validation_agreements_total. See the counting-semantics
* note in detail/ValidationTracker.cpp.
*/
uint64_t
totalAgreementsEver() const;
/** Lifetime misses counted at first classification only.
* @note Unlike totalMissed(), this is strictly monotonic: it is
* incremented only when a ledger is first reconciled as a miss and is
* never decremented by a late repair. It backs the monotonic Prometheus
* counter xrpld_validation_missed_total. See the counting-semantics note
* in detail/ValidationTracker.cpp.
*/
uint64_t
totalMissedEver() const;
/** Total validations this node sent. */
uint64_t
totalValidationsSent() const;
@@ -254,12 +274,33 @@ private:
/// Sliding window of reconciled events (last 7 days).
std::deque<WindowEvent> window7d_;
/// Lifetime count of agreements.
/// Lifetime count of agreements (net: incremented on agree, also on
/// repair). May be read via totalAgreements(); feeds the windowed gauge.
std::atomic<uint64_t> totalAgreements_{0};
/// Lifetime count of misses.
/// Lifetime count of misses (net: incremented on miss, decremented on
/// repair). NON-monotonic. May be read via totalMissed().
std::atomic<uint64_t> totalMissed_{0};
// Monotonic "gross" lifetime tallies for the Prometheus _total counters.
//
// Counting decision (initial-classification only): each reconciled ledger
// is counted exactly once, at its first classification, into exactly one
// of the two tallies below. A later late-repair (miss -> agreement) does
// NOT move either tally. This keeps both strictly monotonic (a Prometheus
// _total must never decrease) and additive:
// totalAgreementsGross_ + totalMissedGross_ == ledgers reconciled.
// The repaired/agreement view is still available from the windowed gauge
// (xrpld_validation_agreement) and the net totals above.
/// Monotonic lifetime initial agreements; backs
/// xrpld_validation_agreements_total. Never adjusted on repair.
std::atomic<uint64_t> totalAgreementsGross_{0};
/// Monotonic lifetime initial misses; backs xrpld_validation_missed_total.
/// Never decremented on repair.
std::atomic<uint64_t> totalMissedGross_{0};
/// Lifetime count of validations this node sent.
std::atomic<uint64_t> totalValidationsSent_{0};

29
src/xrpld/telemetry/detail/ValidationTracker.cpp
View File

@@ -63,10 +63,16 @@ ValidationTracker::reconcile()
if (evt.agreed)
{
totalAgreements_.fetch_add(1, std::memory_order_relaxed);
// Gross tally: count the initial agreement once. See the
// counting-decision note below (repair branch).
totalAgreementsGross_.fetch_add(1, std::memory_order_relaxed);
}
else
{
totalMissed_.fetch_add(1, std::memory_order_relaxed);
// Gross tally: count the initial miss once. See the
// counting-decision note below (repair branch).
totalMissedGross_.fetch_add(1, std::memory_order_relaxed);
}
WindowEvent const we{.time = now, .ledgerHash = evt.ledgerHash, .agreed = evt.agreed};
@@ -78,11 +84,20 @@ ValidationTracker::reconcile()
evt.reconciled && !evt.agreed && evt.weValidated && evt.networkValidated &&
(now - evt.recordTime) <= kLateRepairWindow)
{
// Late repair: was a miss, now both flags set.
// Late repair: was a miss, now both flags set. Adjust the NET
// totals (used by the windowed agreement gauge) so the live view
// reflects the repair.
evt.agreed = true;
totalMissed_.fetch_sub(1, std::memory_order_relaxed);
totalAgreements_.fetch_add(1, std::memory_order_relaxed);
// Counting decision (initial-classification only): the gross
// tallies (totalAgreementsGross_ / totalMissedGross_) that back the
// monotonic Prometheus _total counters are deliberately NOT touched
// here. Each ledger is counted once, at first classification; a
// repair must not decrement missed (a _total may never decrease)
// nor add a second agreement (which would double-count the ledger).
// Flip the corresponding window entries from miss to agreement.
repairWindowEntry(window1h_, evt.ledgerHash);
repairWindowEntry(window24h_, evt.ledgerHash);
@@ -253,6 +268,18 @@ ValidationTracker::totalMissed() const
return totalMissed_.load(std::memory_order_relaxed);
}
uint64_t
ValidationTracker::totalAgreementsEver() const
{
return totalAgreementsGross_.load(std::memory_order_relaxed);
}
uint64_t
ValidationTracker::totalMissedEver() const
{
return totalMissedGross_.load(std::memory_order_relaxed);
}
uint64_t
ValidationTracker::totalValidationsSent() const
{