ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-21 22:30:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(telemetry): expose the sweep-trim and rotation costs (WP-B5)

Browse Source 
Two suspects from the 3.3.0 slowdown investigation had no signal. Both were
already computing the numbers and throwing them away, so this exposes them
rather than adding measurement.

Per-sweep heap trim. The trim runs after every cache sweep, and its cost
scales with resident heap, so it is the leading explanation for a node with
a populated database syncing slower than a fresh one. The report already
carried duration, fault deltas and reclaimed pages, but the whole
measurement sat behind a debug-journal check, so an ordinary node measured
nothing, and the call site discarded the result. The measurement now always
runs and only the log line stays gated. Records trim duration, minor faults
and reclaimed kilobytes. Measured cost of the always-on path is about six
microseconds per sweep against a trim costing milliseconds, at a cadence of
ten to a hundred and twenty seconds.

Honest limit, stated in the runbook: the fault delta spans only the trim
call, so it shows the trim itself faulting but not the faults that follow as
caches refill. The duration is the signal to correlate against sweep-job
queueing.

Rotation writes. Rotation copies archive-served reads forward and re-stores
nodes missing from both backends, both of which compete with sync I/O and
only happen on a populated online_delete database. The copy-forward count
existed but was reset by the rotation's own log line, so a metric reading it
would drop to zero on every swap; a never-reset total sits beside it now.
The re-store count was not measured at all. Rotation duration is
deliberately not recorded: the health throttle sleeps at eight points inside
the sequence and dominates exactly when the node is unhealthy, so the number
would conflate work with waiting.

Nothing added for the other two suspects. Get-object serving is already
covered by the handler label, the lookup histogram and the deferred and
saturation gauges; peer churn by the disconnect-reason counter.

Also replaces nine per-file cspell ignores with one ignoreRegExpList entry
for the telemetry macro names, and picks up the levelization baseline for the
consensus span-name test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pratik Mankawde
2026-07-27 16:39:09 +01:00
parent 371f10934e
commit 8633df7a3e
25 changed files with 1402 additions and 111 deletions
Download Patch File Download Diff File Expand all files Collapse all files

147
src/libxrpl/basics/MallocTrim.cpp
View File

@@ -15,7 +15,6 @@
#include <chrono>
#include <cstddef>
#include <cstdint>
#include <fstream>
#include <ios>
#include <sstream>
@@ -69,6 +68,73 @@ parseStatmRSSkB(std::string const& statm)
return (resident * pageSize) / 1024;
}
/**
* Read a whole /proc pseudo-file into a string.
*
* /proc files are frequently not seekable, so the contents are streamed rather
* than sized-then-read.
*
* @param path Absolute path of the pseudo-file.
* @return The file contents, or an empty string if it could not be opened.
*/
std::string
readProcFile(std::string const& path)
{
std::ifstream ifs(path, std::ios::in | std::ios::binary);
if (!ifs.is_open())
return {};
std::ostringstream oss;
oss << ifs.rdbuf();
return oss.str();
}
/**
* Run malloc_trim and measure what it cost.
*
* Split out of mallocTrim() so the always-on measurement is one testable unit
* and the caller is left with only the logging decision.
*
* Measurement order brackets the trim as tightly as possible: the two RSS
* samples are outermost, the two fault samples inside them, and the clock pair
* innermost, so the reported duration contains the trim and nothing else.
*
* @param padBytes glibc trim padding, passed straight to ::malloc_trim.
* @return A fully populated report. Fields whose source syscall failed keep
* their -1 "not measured" sentinel.
*/
MallocTrimReport
measuredTrim(std::size_t padBytes)
{
MallocTrimReport report;
report.supported = true;
std::string const statmPath = "/proc/self/statm";
report.rssBeforeKB = detail::parseStatmRSSkB(readProcFile(statmPath));
struct rusage ru0{};
bool const haveRu0 = getRusageThread(ru0);
auto const t0 = std::chrono::steady_clock::now();
report.trimResult = detail::mallocTrimWithPad(padBytes);
auto const t1 = std::chrono::steady_clock::now();
struct rusage ru1{};
bool const haveRu1 = getRusageThread(ru1);
report.rssAfterKB = detail::parseStatmRSSkB(readProcFile(statmPath));
report.durationUs = std::chrono::duration_cast<std::chrono::microseconds>(t1 - t0);
if (haveRu0 && haveRu1)
{
report.minfltDelta = ru1.ru_minflt - ru0.ru_minflt;
report.majfltDelta = ru1.ru_majflt - ru0.ru_majflt;
}
return report;
}
#endif // __GLIBC__ && BOOST_OS_LINUX
} // namespace detail
@@ -88,70 +154,37 @@ mallocTrim(std::string_view tag, beast::Journal journal)
// of RSS reduction and trim-latency stability without adding a tuning surface.
static constexpr std::size_t kTrimPad = 0;
report.supported = true;
// The measurement is unconditional. It used to sit inside
// `if (journal.debug())`, which meant an ordinary node -- which does not run
// at debug level -- measured nothing, so the caller had no duration to
// record and the per-sweep trim cost was invisible in production, the one
// place it matters.
//
// Cost of measuring on every sweep, measured on this platform: two
// /proc/self/statm reads at ~2.8 us each and two getrusage(RUSAGE_THREAD)
// calls at ~0.17 us each, so about 6 us in total. The trim it brackets
// costs milliseconds on a large heap, and the sweep that calls it runs once
// per SizedItem::SweepInterval (10 s at the fastest, tiny-node setting).
// That is a duty cycle under 1e-6 percent, and under 1 percent of the
// measured operation, so nothing here is worth making conditional -- a
// debug-only RSS read would only reintroduce the blind spot it costs
// nothing to remove.
report = detail::measuredTrim(kTrimPad);
// Only the LOG stays gated: the string formatting is what an ordinary node
// genuinely should not pay for, and the numbers now reach the metrics
// pipeline through the return value instead.
if (journal.debug())
{
auto readFile = [](std::string const& path) -> std::string {
std::ifstream ifs(path, std::ios::in | std::ios::binary);
if (!ifs.is_open())
return {};
// /proc files are often not seekable; read as a stream.
std::ostringstream oss;
oss << ifs.rdbuf();
return oss.str();
};
std::string const tagStr{tag};
std::string const statmPath = "/proc/self/statm";
auto const statmBefore = readFile(statmPath);
long const rssBeforeKB = detail::parseStatmRSSkB(statmBefore);
struct rusage ru0{};
bool const haveRu0 = getRusageThread(ru0);
auto const t0 = std::chrono::steady_clock::now();
report.trimResult = detail::mallocTrimWithPad(kTrimPad);
auto const t1 = std::chrono::steady_clock::now();
struct rusage ru1{};
bool const haveRu1 = getRusageThread(ru1);
auto const statmAfter = readFile(statmPath);
long const rssAfterKB = detail::parseStatmRSSkB(statmAfter);
// Populate report fields
report.rssBeforeKB = rssBeforeKB;
report.rssAfterKB = rssAfterKB;
report.durationUs = std::chrono::duration_cast<std::chrono::microseconds>(t1 - t0);
if (haveRu0 && haveRu1)
{
report.minfltDelta = ru1.ru_minflt - ru0.ru_minflt;
report.majfltDelta = ru1.ru_majflt - ru0.ru_majflt;
}
std::int64_t const deltaKB = (rssBeforeKB < 0 || rssAfterKB < 0)
? 0
: (static_cast<std::int64_t>(rssAfterKB) - static_cast<std::int64_t>(rssBeforeKB));
JLOG(journal.debug()) << "malloc_trim tag=" << tagStr << " result=" << report.trimResult
JLOG(journal.debug()) << "malloc_trim tag=" << tag << " result=" << report.trimResult
<< " pad=" << kTrimPad << " bytes"
<< " rss_before=" << rssBeforeKB << "kB"
<< " rss_after=" << rssAfterKB << "kB"
<< " delta=" << deltaKB << "kB"
<< " rss_before=" << report.rssBeforeKB << "kB"
<< " rss_after=" << report.rssAfterKB << "kB"
<< " delta=" << report.deltaKB() << "kB"
<< " duration_us=" << report.durationUs.count()
<< " minflt_delta=" << report.minfltDelta
<< " majflt_delta=" << report.majfltDelta;
}
else
{
report.trimResult = detail::mallocTrimWithPad(kTrimPad);
}
#endif

6
src/libxrpl/nodestore/DatabaseRotatingImp.cpp
View File

@@ -211,7 +211,13 @@ DatabaseRotatingImp::fetchNodeObject(
if (duplicate || rotationInFlight_.load(std::memory_order_acquire))
{
if (!duplicate)
{
// Two counters, one event: the per-rotation tally that
// rotate() resets for its log line, and the monotonic total
// the metrics gauge reads, which must never go backwards.
copyForwardCount_.fetch_add(1, std::memory_order_relaxed);
copyForwardTotal_.fetch_add(1, std::memory_order_relaxed);
}
writable->store(nodeObject);
}
}

39
src/tests/libxrpl/basics/MallocTrim.cpp
View File

@@ -121,7 +121,12 @@ TEST(parseStatmRSSkB, standard_format)
}
#endif
TEST(mallocTrim, without_debug_logging)
// The measurement must NOT depend on the journal's severity. It used to sit
// inside `if (journal.debug())`, so an ordinary node -- which does not run at
// debug level -- measured nothing and the caller had no duration to record.
// This is the regression test for that: with a null sink (nothing is even
// loggable) every field must still be populated.
TEST(mallocTrim, measures_without_debug_logging)
{
beast::Journal const journal{beast::Journal::getNullSink()};
@@ -130,10 +135,29 @@ TEST(mallocTrim, without_debug_logging)
#if defined(__GLIBC__) && BOOST_OS_LINUX
EXPECT_EQ(report.supported, true);
EXPECT_GE(report.trimResult, 0);
EXPECT_EQ(report.durationUs, std::chrono::microseconds{-1});
EXPECT_EQ(report.minfltDelta, -1);
EXPECT_EQ(report.majfltDelta, -1);
// The three measured fields are populated, NOT left at their -1
// "not measured" sentinel. Asserting >= 0 rather than == a fixed number
// because these are real timings; the sentinel is what the test excludes.
EXPECT_GE(report.durationUs.count(), 0);
EXPECT_GE(report.minfltDelta, 0);
EXPECT_GE(report.majfltDelta, 0);
// RSS is read on both sides of the trim, so both are real page counts.
// A live process always has resident pages, so these are strictly > 0.
EXPECT_GT(report.rssBeforeKB, 0);
EXPECT_GT(report.rssAfterKB, 0);
// deltaKB() is now derived from two real readings rather than from the
// sentinel pair, so it is the genuine change: a trim never grows RSS by
// more than another thread could allocate concurrently, and this test is
// single-threaded, so the reading cannot be positive.
EXPECT_LE(report.deltaKB(), 0);
EXPECT_EQ(report.deltaKB(), report.rssAfterKB - report.rssBeforeKB);
#else
// NEGATIVE PLATFORM PATH: not Linux/glibc, so there is no trim at all and
// every field must keep its sentinel. A zero here would falsely claim a
// free trim happened.
EXPECT_EQ(report.supported, false);
EXPECT_EQ(report.trimResult, -1);
EXPECT_EQ(report.rssBeforeKB, -1);
@@ -141,6 +165,7 @@ TEST(mallocTrim, without_debug_logging)
EXPECT_EQ(report.durationUs, std::chrono::microseconds{-1});
EXPECT_EQ(report.minfltDelta, -1);
EXPECT_EQ(report.majfltDelta, -1);
EXPECT_EQ(report.deltaKB(), 0);
#endif
}
@@ -185,6 +210,12 @@ TEST(mallocTrim, with_debug_logging)
EXPECT_GE(report.durationUs.count(), 0);
EXPECT_GE(report.minfltDelta, 0);
EXPECT_GE(report.majfltDelta, 0);
// Same fields as the null-sink case above: raising the severity adds the
// log line and changes nothing about what is measured. The two tests
// together are what prove the severity no longer gates the measurement.
EXPECT_GT(report.rssBeforeKB, 0);
EXPECT_GT(report.rssAfterKB, 0);
#else
EXPECT_EQ(report.supported, false);
EXPECT_EQ(report.trimResult, -1);

377
src/tests/libxrpl/telemetry/MetricMacros.cpp
View File

@@ -17,10 +17,6 @@
* MetricsRegistry.cpp is only compiled into this binary on the no-op path.
*/
// cspell:ignore ISTOGRAM
// The all-caps macro name XRPL_METRIC_HISTOGRAM_RECORD trips cspell's
// compound-word splitter, which emits the subword "ISTOGRAM"; ignore it here.
#ifdef XRPL_ENABLE_TELEMETRY
#include <xrpld/telemetry/MetricMacros.h>
@@ -29,6 +25,7 @@
#include <xrpld/peerfinder/PeerfinderManager.h>
#include <xrpld/telemetry/MetricNames.h>
#include <xrpl/basics/MallocTrim.h>
#include <xrpl/core/JobQueue.h>
#include <gtest/gtest.h>
@@ -2921,4 +2918,376 @@ TEST(MetricMacros, consensus_round_duration_emits_nothing_when_registry_disabled
EXPECT_EQ(app.registry().meterCalls(), 0);
}
// -----------------------------------------------------------------
// WP-B5: per-sweep malloc_trim, and online_delete rotation writes
//
// Suspect 3. malloc_trim runs after EVERY cache sweep and its cost scales with
// the resident heap, so a node with a large existing database pays a per-sweep
// penalty a fresh one does not. The MallocTrimReport already carried every
// number; the emit site discarded it.
//
// Suspect 4. An online_delete rotation performs writes an ordinary fetch would
// not -- copy-forward from the doomed archive, plus copyNode re-stores -- which
// compete with sync I/O and only exist on a populated, already-rotated
// database.
// -----------------------------------------------------------------
// The three sweep instruments must stay three separate series carrying their own
// exact values, and the guards at the emit site must drop a reading that was
// never measured rather than publishing a zero for it.
//
// The 45000 us sample is above the SDK's 10,000 default top boundary on
// purpose: that is why MetricsRegistry registers the microsecond ladder for
// this instrument, and a 45 ms trim is exactly the large-heap case the signal
// exists to catch. The sum must carry the real value however it is bucketed.
TEST(MetricMacros, sweep_malloc_trim_records_exact_duration_faults_and_reclaim)
{
CollectingProvider const provider;
FakeApp app;
wire(app, /*enabled=*/true, provider.meter());
// Mirrors ApplicationImp::trimHeapAndRecord for three sweeps: a cheap trim
// on a small heap, an expensive one on a large heap, and one that reclaimed
// nothing. Distinct values so a collapsed instrument cannot look correct.
struct Sweep
{
std::int64_t durationUs;
std::int64_t minfltDelta;
std::int64_t reclaimedKb;
};
for (auto const& sweep : {
Sweep{.durationUs = 120, .minfltDelta = 3, .reclaimedKb = 512},
Sweep{.durationUs = 45'000, .minfltDelta = 900, .reclaimedKb = 262'144},
Sweep{.durationUs = 80, .minfltDelta = 0, .reclaimedKb = 0},
})
{
XRPL_METRIC_HISTOGRAM_RECORD(
app,
telemetry::metric::sweepMallocTrimUs,
"Duration of the malloc_trim call ending each cache sweep (microseconds)",
sweep.durationUs);
if (sweep.minfltDelta > 0)
{
XRPL_METRIC_COUNTER_ADD(
app,
telemetry::metric::sweepMallocTrimMinorFaultsTotal,
"Minor page faults taken inside the sweep's malloc_trim call",
static_cast<std::uint64_t>(sweep.minfltDelta));
}
if (sweep.reclaimedKb > 0)
{
XRPL_METRIC_COUNTER_ADD(
app,
telemetry::metric::sweepMallocTrimReclaimedKbTotal,
"Resident kilobytes returned to the OS by the sweep's malloc_trim",
static_cast<std::uint64_t>(sweep.reclaimedKb));
}
}
auto const data = provider.collect();
// Histogram: all three sweeps recorded, including the one whose fault and
// reclaim readings were skipped -- a trim always has a duration.
auto const [count, sum] = histogramCountAndSum(data, "sweep_malloc_trim_us");
EXPECT_EQ(count, 3u);
EXPECT_NEAR(sum, 45'200.0, 1e-9);
// Counters: cumulative totals, so the panel can rate() them. The zero-value
// third sweep contributed to neither.
EXPECT_EQ(
counterValue(data, "sweep_malloc_trim_minor_faults_total", otel_sdk::PointAttributes{}),
903);
EXPECT_EQ(
counterValue(data, "sweep_malloc_trim_reclaimed_kb_total", otel_sdk::PointAttributes{}),
262'656);
// All three are unlabelled: one series each, per node. A label here would be
// unbounded (there is no bounded dimension a sweep varies over) and the
// dashboard reads one line per node instead.
ASSERT_EQ(data.at("sweep_malloc_trim_us").size(), 1u);
EXPECT_TRUE(data.at("sweep_malloc_trim_us").begin()->first.empty());
ASSERT_EQ(data.at("sweep_malloc_trim_minor_faults_total").size(), 1u);
EXPECT_TRUE(data.at("sweep_malloc_trim_minor_faults_total").begin()->first.empty());
ASSERT_EQ(data.at("sweep_malloc_trim_reclaimed_kb_total").size(), 1u);
EXPECT_TRUE(data.at("sweep_malloc_trim_reclaimed_kb_total").begin()->first.empty());
// Exactly the three instruments, so neither counter absorbed the other's
// adds and the histogram did not spawn a sibling.
EXPECT_EQ(data.size(), 3u);
}
// EDGE CASE: nothing was measured. On a non-glibc platform mallocTrim() returns
// a report whose fields are all at their -1 sentinel, and the emit site's
// guards must publish NO series for it. A zero-valued series would claim the
// trim was instantaneous and reclaimed nothing, which is a different (and
// false) statement from "this platform has no trim".
//
// A FRESH provider, because the reader is cumulative: a second collect() on the
// provider used above would still show the earlier series and the absence
// assertions could not fail.
TEST(MetricMacros, sweep_malloc_trim_publishes_nothing_when_unmeasured)
{
CollectingProvider const provider;
FakeApp app;
wire(app, /*enabled=*/true, provider.meter());
// The sentinel report, exactly as MallocTrimReport default-constructs it.
MallocTrimReport const unsupported;
ASSERT_FALSE(unsupported.supported);
ASSERT_EQ(unsupported.durationUs.count(), -1);
ASSERT_EQ(unsupported.minfltDelta, -1);
ASSERT_EQ(unsupported.deltaKB(), 0);
// trimHeapAndRecord's first guard: an unsupported report emits nothing at
// all, so the three statements below are never reached on such a platform.
if (unsupported.supported)
{
if (unsupported.durationUs.count() >= 0)
{
XRPL_METRIC_HISTOGRAM_RECORD(
app,
telemetry::metric::sweepMallocTrimUs,
"Duration of the malloc_trim call ending each cache sweep (microseconds)",
unsupported.durationUs.count());
}
}
auto const data = provider.collect();
// State: not one of the three series exists.
EXPECT_EQ(data.count("sweep_malloc_trim_us"), 0u);
EXPECT_EQ(data.count("sweep_malloc_trim_minor_faults_total"), 0u);
EXPECT_EQ(data.count("sweep_malloc_trim_reclaimed_kb_total"), 0u);
EXPECT_EQ(data.size(), 0u);
}
// EDGE CASE: RSS GREW across the trim, because another thread allocated faster
// than the trim released. deltaKB() is then positive, and the reclaimed counter
// must skip it -- a counter cannot decrease, and there is no such thing as
// reclaiming a negative number of kilobytes. The duration is still recorded,
// because the trim did happen and did cost time.
TEST(MetricMacros, sweep_malloc_trim_skips_reclaim_when_rss_grew)
{
CollectingProvider const provider;
FakeApp app;
wire(app, /*enabled=*/true, provider.meter());
// A report whose after-RSS is ABOVE its before-RSS.
MallocTrimReport grew;
grew.supported = true;
grew.trimResult = 1;
grew.rssBeforeKB = 1'000'000;
grew.rssAfterKB = 1'000'800;
grew.durationUs = std::chrono::microseconds{9'000};
grew.minfltDelta = 11;
// Cause: the sign convention is after-minus-before, so growth is positive
// and a naive negation would add 800 to a "reclaimed" total.
ASSERT_EQ(grew.deltaKB(), 800);
XRPL_METRIC_HISTOGRAM_RECORD(
app,
telemetry::metric::sweepMallocTrimUs,
"Duration of the malloc_trim call ending each cache sweep (microseconds)",
grew.durationUs.count());
XRPL_METRIC_COUNTER_ADD(
app,
telemetry::metric::sweepMallocTrimMinorFaultsTotal,
"Minor page faults taken inside the sweep's malloc_trim call",
static_cast<std::uint64_t>(grew.minfltDelta));
if (auto const deltaKB = grew.deltaKB(); deltaKB < 0)
{
XRPL_METRIC_COUNTER_ADD(
app,
telemetry::metric::sweepMallocTrimReclaimedKbTotal,
"Resident kilobytes returned to the OS by the sweep's malloc_trim",
static_cast<std::uint64_t>(-deltaKB));
}
auto const data = provider.collect();
// The duration and the faults are real and are recorded.
auto const [count, sum] = histogramCountAndSum(data, "sweep_malloc_trim_us");
EXPECT_EQ(count, 1u);
EXPECT_NEAR(sum, 9'000.0, 1e-9);
EXPECT_EQ(
counterValue(data, "sweep_malloc_trim_minor_faults_total", otel_sdk::PointAttributes{}),
11);
// NEGATIVE: the reclaim counter has no series at all. Absence, not a zero
// and emphatically not 800.
EXPECT_EQ(data.count("sweep_malloc_trim_reclaimed_kb_total"), 0u);
EXPECT_EQ(data.size(), 2u);
}
// rotation_state is polled from the node store, so the production callback in
// MetricsRegistry::registerRotationStateGauge cannot be linked into this
// binary. The derivation it performs is asserted here against the same two
// inputs, mirroring how nodestore_latency is tested above.
TEST(MetricMacros, rotation_state_gauge_observes_in_flight_window_and_copy_forward_total)
{
// Each scenario gets a FRESH provider: the reader is cumulative, so a
// series observed once would persist into the next collect() and the
// "no rotating store means no series" assertion below could not fail.
struct RotationState
{
bool isRotating;
std::uint64_t copyForwardTotal;
bool hasRotatingStore;
};
auto collectWith = [](RotationState& state) {
CollectingProvider const provider;
auto gauge = provider.meter()->CreateInt64ObservableGauge(
telemetry::metric::rotationState,
"Online-delete rotation state and copy-forward write total");
gauge->AddCallback(
[](opentelemetry::metrics::ObserverResult result, void* state) {
auto const* self = static_cast<RotationState const*>(state);
// The production dynamic_cast: a node without online_delete has
// a non-rotating store and publishes nothing.
if (!self->hasRotatingStore)
return;
auto observe = [&](char const* field, std::int64_t value) {
opentelemetry::nostd::get<opentelemetry::nostd::shared_ptr<
opentelemetry::metrics::ObserverResultT<std::int64_t>>>(result)
->Observe(value, {{telemetry::label::metric, field}});
};
observe(
telemetry::lval::rotation_state::inFlight,
static_cast<std::int64_t>(self->isRotating ? 1 : 0));
observe(
telemetry::lval::rotation_state::copyForward,
static_cast<std::int64_t>(self->copyForwardTotal));
},
&state);
return provider.collect();
};
// A rotation is running and has already forced 4096 copy-forward writes:
// the shape an operator should see, and the one that explains sync I/O
// contention on a populated online_delete database.
RotationState state{.isRotating = true, .copyForwardTotal = 4096, .hasRotatingStore = true};
auto const rotating = collectWith(state);
ASSERT_EQ(rotating.at("rotation_state").size(), 2u);
EXPECT_EQ(gaugeValue(rotating, "rotation_state", attrs("metric", "in_flight")), 1);
EXPECT_EQ(gaugeValue(rotating, "rotation_state", attrs("metric", "copy_forward")), 4096);
// Single fixed-cardinality label group, keyed exactly `metric`.
auto const& firstKey = rotating.at("rotation_state").begin()->first;
ASSERT_EQ(firstKey.size(), 1u);
EXPECT_EQ(firstKey.begin()->first, "metric");
// Between rotations: in_flight drops to 0 while the total HOLDS its value.
// That combination is the whole point of pairing them -- the extra writes
// are not happening now, but they did, and the total must not reset (a
// counter that drops cannot be rated).
state = RotationState{.isRotating = false, .copyForwardTotal = 4096, .hasRotatingStore = true};
auto const idle = collectWith(state);
EXPECT_EQ(gaugeValue(idle, "rotation_state", attrs("metric", "in_flight")), 0);
EXPECT_EQ(gaugeValue(idle, "rotation_state", attrs("metric", "copy_forward")), 4096);
// Both series still exist while idle: a zero in_flight is a real reading,
// and absence would be the regression.
EXPECT_EQ(idle.at("rotation_state").size(), 2u);
// EDGE CASE: a fresh node that has online_delete configured but has never
// rotated. Both readings are 0 and BOTH series still exist -- 0 copy-forward
// writes is the healthy answer to "what did rotation cost", not missing data.
state = RotationState{.isRotating = false, .copyForwardTotal = 0, .hasRotatingStore = true};
auto const neverRotated = collectWith(state);
EXPECT_EQ(gaugeValue(neverRotated, "rotation_state", attrs("metric", "in_flight")), 0);
EXPECT_EQ(gaugeValue(neverRotated, "rotation_state", attrs("metric", "copy_forward")), 0);
EXPECT_EQ(neverRotated.at("rotation_state").size(), 2u);
// NEGATIVE: no rotating store at all -- online_delete is not configured, so
// the node store is a DatabaseNodeImp and the cast fails. NO series is
// published, deliberately: an absent series means "rotation is not
// configured", which a zero would misreport as "rotation is free".
state = RotationState{.isRotating = false, .copyForwardTotal = 0, .hasRotatingStore = false};
auto const notConfigured = collectWith(state);
EXPECT_EQ(notConfigured.count("rotation_state"), 0u);
EXPECT_EQ(notConfigured.size(), 0u);
}
// The copyNode re-store counter: one increment per node rescued from neither
// backend, on ONE unlabelled series. The node hash must never become a label --
// it is unbounded runtime data and would mint one series per rescued node.
TEST(MetricMacros, rotation_copy_node_restore_accumulates_on_one_unlabelled_series)
{
CollectingProvider const provider;
FakeApp app;
wire(app, /*enabled=*/true, provider.meter());
// Seven rescued nodes across one rotation's state-map walk.
for (int i = 0; i < 7; ++i)
{
XRPL_METRIC_COUNTER_INC(
app,
telemetry::metric::rotationCopyNodeRestoreTotal,
"Nodes re-stored during rotation because they were missing from both backends");
}
auto const data = provider.collect();
ASSERT_EQ(data.at("rotation_copy_node_restore_total").size(), 1u);
EXPECT_EQ(
counterValue(data, "rotation_copy_node_restore_total", otel_sdk::PointAttributes{}), 7);
// The single series carries NO labels, which is what bounds its cardinality
// to one per node however many distinct hashes were rescued.
EXPECT_TRUE(data.at("rotation_copy_node_restore_total").begin()->first.empty());
// The rotation gauge is a separate instrument, so the counter cannot inflate
// the copy-forward total: they measure two different extra writes.
EXPECT_EQ(data.count("rotation_state"), 0u);
EXPECT_EQ(data.size(), 1u);
}
// RUNTIME-DISABLED no-op proof for all four WP-B5 push instruments. With the
// registry disabled nothing is emitted -- total absence, not zero-valued series
// -- and no macro ever asks for a meter.
TEST(MetricMacros, sweep_and_rotation_metrics_emit_nothing_when_registry_disabled)
{
CollectingProvider const provider;
FakeApp app;
wire(app, /*enabled=*/false, provider.meter());
XRPL_METRIC_HISTOGRAM_RECORD(
app,
telemetry::metric::sweepMallocTrimUs,
"Duration of the malloc_trim call ending each cache sweep (microseconds)",
45'000);
XRPL_METRIC_COUNTER_ADD(
app,
telemetry::metric::sweepMallocTrimMinorFaultsTotal,
"Minor page faults taken inside the sweep's malloc_trim call",
900);
XRPL_METRIC_COUNTER_ADD(
app,
telemetry::metric::sweepMallocTrimReclaimedKbTotal,
"Resident kilobytes returned to the OS by the sweep's malloc_trim",
262'144);
XRPL_METRIC_COUNTER_INC(
app,
telemetry::metric::rotationCopyNodeRestoreTotal,
"Nodes re-stored during rotation because they were missing from both backends");
auto const data = provider.collect();
// State: no series under any of the four names, and nothing else leaked in.
EXPECT_EQ(data.count("sweep_malloc_trim_us"), 0u);
EXPECT_EQ(data.count("sweep_malloc_trim_minor_faults_total"), 0u);
EXPECT_EQ(data.count("sweep_malloc_trim_reclaimed_kb_total"), 0u);
EXPECT_EQ(data.count("rotation_copy_node_restore_total"), 0u);
EXPECT_EQ(data.size(), 0u);
// Cause, not just state: the isEnabled() gate short-circuited before any
// macro asked for a meter, so no instrument was ever created.
EXPECT_EQ(app.registry().meterCalls(), 0);
}
#endif // XRPL_ENABLE_TELEMETRY

3
src/xrpld/app/consensus/RCLConsensus.cpp
View File

@@ -1,6 +1,3 @@
// cspell:ignore ISTOGRAM
// The all-caps macro name XRPL_METRIC_HISTOGRAM_RECORD trips cspell's
// compound-word splitter, which emits the subword "ISTOGRAM"; ignore it here.
#include <xrpld/app/consensus/RCLConsensus.h>

75
src/xrpld/app/main/Application.cpp
View File

@@ -1,3 +1,4 @@
#include <xrpld/app/main/Application.h>
#include <xrpld/app/consensus/RCLValidations.h>
@@ -38,6 +39,8 @@
#include <xrpld/rpc/detail/PathRequestManager.h>
#include <xrpld/rpc/detail/Pathfinder.h>
#include <xrpld/shamap/NodeFamily.h>
#include <xrpld/telemetry/MetricMacros.h>
#include <xrpld/telemetry/MetricNames.h>
#include <xrpld/telemetry/MetricsRegistry.h>
#include <xrpl/basics/ByteUtilities.h>
@@ -1131,12 +1134,82 @@ public:
<< "; size after: " << cachedSLEs_.size();
}
mallocTrim("doSweep", journal_);
trimHeapAndRecord();
// Set timer to do another sweep later.
setSweepTimer();
}
/**
* Return free heap pages to the OS at the end of a sweep, and record what
* that cost.
*
* Split out of doSweep() so the metric emit site is one small unit rather
* than a further three statements on an already-long function.
*
* Why this is instrumented: `malloc_trim` runs after EVERY cache sweep, and
* its cost scales with the resident heap, so a node with a large existing
* database pays a per-sweep penalty a fresh one does not -- the leading
* explanation for "an existing database syncs slower than an empty one" on
* glibc. The report used to be discarded here, so none of it was visible.
*
* doSweep() -> trimHeapAndRecord() -> mallocTrim()
* | |
* | MallocTrimReport (duration,
* | minor faults, RSS before/after)
* v
* 3 OTel instruments
*
* Cost: one histogram Record and two counter Adds per sweep, at a cadence
* of SizedItem::SweepInterval (10-120 s), so this is free.
*
* @note The minor-fault count covers the trim call only. It cannot show the
* faults taken later, as the caches refill and touch the pages the
* trim handed back -- see the runbook branch for how to read it.
*/
void
trimHeapAndRecord()
{
MallocTrimReport const report = mallocTrim("doSweep", journal_);
// Nothing was measured: not Linux/glibc, so there is no trim to report
// and a zero would falsely claim a free one.
if (!report.supported)
return;
if (report.durationUs.count() >= 0)
{
XRPL_METRIC_HISTOGRAM_RECORD(
*this,
telemetry::metric::sweepMallocTrimUs,
"Duration of the malloc_trim call ending each cache sweep (microseconds)",
report.durationUs.count());
}
if (report.minfltDelta > 0)
{
XRPL_METRIC_COUNTER_ADD(
*this,
telemetry::metric::sweepMallocTrimMinorFaultsTotal,
"Minor page faults taken inside the sweep's malloc_trim call",
static_cast<std::uint64_t>(report.minfltDelta));
}
// deltaKB() is after-minus-before, so a successful trim is NEGATIVE.
// Publish the reclaimed amount as a positive cumulative total and drop
// the case where RSS grew across the call (another thread allocating
// faster than the trim released): a counter cannot go down, and "grew"
// is not a reclaim of a negative size.
if (auto const deltaKB = report.deltaKB(); deltaKB < 0)
{
XRPL_METRIC_COUNTER_ADD(
*this,
telemetry::metric::sweepMallocTrimReclaimedKbTotal,
"Resident kilobytes returned to the OS by the sweep's malloc_trim",
static_cast<std::uint64_t>(-deltaKB));
}
}
LedgerIndex
getMaxDisallowedLedger() override
{

12
src/xrpld/app/misc/SHAMapStoreImp.cpp
View File

@@ -4,6 +4,8 @@
#include <xrpld/app/misc/SHAMapStore.h>
#include <xrpld/app/rdb/backend/SQLiteDatabase.h>
#include <xrpld/core/Config.h>
#include <xrpld/telemetry/MetricMacros.h>
#include <xrpld/telemetry/MetricNames.h>
#include <xrpl/basics/ByteUtilities.h>
#include <xrpl/basics/Log.h>
@@ -273,6 +275,16 @@ SHAMapStoreImp::copyNode(std::uint64_t& nodeCount, SHAMapTreeNode const& node)
dbRotating_->store(NodeObjectType::AccountNode, std::move(s.modData()), hash, 0);
JLOG(journal_.warn()) << "copyNode: re-stored node missing from both backends, hash="
<< hash << " type=" << static_cast<int>(node.getType());
// One extra write per rescued node, on top of the whole-state-map walk
// the rotation already performs. Rotation-time writes compete with sync
// I/O, and this branch was warn-log-only, so the volume was invisible
// unless an operator was reading logs. The node hash is deliberately NOT
// a label: it is unbounded runtime data and would mint one series per
// node. Correlate a spike against the log line by node and time.
XRPL_METRIC_COUNTER_INC(
app_,
telemetry::metric::rotationCopyNodeRestoreTotal,
"Nodes re-stored during rotation because they were missing from both backends");
}
if ((++nodeCount % checkHealthInterval_) == 0u)
{

3
src/xrpld/overlay/detail/ConnectAttempt.cpp
View File

@@ -1,6 +1,3 @@
// cspell:ignore ISTOGRAM
// The all-caps macro name XRPL_METRIC_HISTOGRAM_RECORD trips cspell's
// compound-word splitter, which emits the subword "ISTOGRAM"; ignore it here.
#include <xrpld/overlay/detail/ConnectAttempt.h>

3
src/xrpld/overlay/detail/OverlayImpl.cpp
View File

@@ -1,6 +1,3 @@
// cspell:ignore ISTOGRAM
// The all-caps macro name XRPL_METRIC_HISTOGRAM_RECORD trips cspell's
// compound-word splitter, which emits the subword "ISTOGRAM"; ignore it here.
#include <xrpld/overlay/detail/OverlayImpl.h>

3
src/xrpld/overlay/detail/PeerImp.cpp
View File

@@ -1,6 +1,3 @@
// cspell:ignore ISTOGRAM
// The all-caps macro name XRPL_METRIC_HISTOGRAM_RECORD trips cspell's
// compound-word splitter, which emits the subword "ISTOGRAM"; ignore it here.
#include <xrpld/overlay/detail/PeerImp.h>

4
src/xrpld/telemetry/MetricMacros.h
View File

@@ -1,9 +1,5 @@
#pragma once
// cspell:ignore ISTOGRAM
// The all-caps macro name XRPL_METRIC_HISTOGRAM_RECORD trips cspell's
// compound-word splitter, which emits the subword "ISTOGRAM"; ignore it here.
/**
* Call-site OTel metric macros.
*

66
src/xrpld/telemetry/MetricNames.h
View File

@@ -268,6 +268,58 @@ inline constexpr char nodestoreLatency[] = "nodestore_latency";
*/
inline constexpr char consensusRoundDurationMs[] = "consensus_round_duration_ms";
// ===== Sweep: what the periodic cache sweep costs ============================
//
// The sweep runs every `SizedItem::SweepInterval` seconds (10 s on a tiny node
// through 120 s on a huge one), so these three emit sites are cold: one
// histogram Record and two counter Adds per sweep is free at that cadence.
/**
* Wall-clock duration of the `malloc_trim` call that ends every cache sweep.
*
* The cost of returning free heap pages to the kernel scales with the resident
* heap, so this is the signal that a node with a large existing database pays a
* per-sweep penalty a fresh node does not.
*/
inline constexpr char sweepMallocTrimUs[] = "sweep_malloc_trim_us";
/**
* Minor page faults taken *inside* the `malloc_trim` call.
*
* Cumulative, so `rate()` gives faults/sec. Scoped to the trim call only -- see
* the limitation noted on the runbook branch: this proves the trim itself
* faults, not that the trim causes later faults as the caches refill.
*/
inline constexpr char sweepMallocTrimMinorFaultsTotal[] = "sweep_malloc_trim_minor_faults_total";
/**
* Resident kilobytes the trim actually returned to the kernel.
*
* Cumulative and clamped at zero per sweep: a trim that reclaimed nothing, or
* during which another thread grew the heap faster than the trim shrank it,
* contributes 0 rather than a negative amount.
*/
inline constexpr char sweepMallocTrimReclaimedKbTotal[] = "sweep_malloc_trim_reclaimed_kb_total";
// ===== Rotation: the extra writes an online_delete rotation performs =========
/**
* Nodes re-stored by `copyNode` because they were missing from both backends.
*
* The genuinely unmeasured extra write of a rotation: a clean node reachable
* from the validated state map whose only on-disk copy lived in a backend an
* earlier rotation removed. Was warn-log-only.
*/
inline constexpr char rotationCopyNodeRestoreTotal[] = "rotation_copy_node_restore_total";
/**
* Rotation state: whether one is running, and the copy-forward write total.
*
* A gauge, not a counter, because the two readings are polled from the node
* store rather than pushed: `in_flight` is current state and `copy_forward` is a
* cumulative total the nodestore already keeps. Observed from the existing
* `registerNodeStoreGauge` callback, which is how everything else reads the node
* store from xrpld without libxrpl having to know about telemetry.
*/
inline constexpr char rotationState[] = "rotation_state";
// ===== Pre-existing instruments pulled in by the family ratchet ==============
//
// These predate the sync-diagnostics work. They are declared here because the
@@ -592,6 +644,20 @@ inline constexpr char warned[] = "warned";
inline constexpr char secondsToBlock[] = "seconds_to_block";
} // namespace amendment_block
/**
* `rotation_state` sub-metrics: is a rotation running, and how many extra
* writes have rotations caused.
*
* Read together: a `copy_forward` total that climbs while `in_flight` is 1 is
* the rotation doing its extra writes, which is the expected shape. The same
* total climbing while `in_flight` is 0 would mean the flag leaked, not that
* rotation is cheap.
*/
namespace rotation_state {
inline constexpr char inFlight[] = "in_flight";
inline constexpr char copyForward[] = "copy_forward";
} // namespace rotation_state
/**
* `nodestore_latency` sub-metrics: mean latency per direction, with counts.
*/

119
src/xrpld/telemetry/MetricsRegistry.cpp
View File

@@ -1,6 +1,3 @@
// cspell:ignore ISTOGRAM
// The all-caps macro name XRPL_METRIC_HISTOGRAM_RECORD trips cspell's
// compound-word splitter, which emits the subword "ISTOGRAM"; ignore it here.
/**
* MetricsRegistry implementation — OpenTelemetry metric instruments for xrpld.
@@ -47,6 +44,7 @@
#include <xrpl/json/json_value.h>
#include <xrpl/ledger/AmendmentTable.h>
#include <xrpl/nodestore/Database.h>
#include <xrpl/nodestore/DatabaseRotating.h>
#include <xrpl/protocol/BuildInfo.h>
#include <xrpl/protocol/jss.h>
#include <xrpl/rdb/RelationalDatabase.h>
@@ -328,6 +326,19 @@ MetricsRegistry::initExporterAndProvider(std::string const& endpoint, std::strin
// comes from the shared constant both sites use.
addMicrosecondHistogramView(*views, kGetObjectLookupUs);
// Sweep malloc_trim duration. Shares the microsecond ladder rather than
// getting a bespoke one, and the ladder is what makes it readable: a trim on
// a small heap lands in the tens-of-microseconds buckets, while a trim on a
// multi-gigabyte resident heap runs well past 10 ms -- which is exactly the
// large-existing-database case this signal exists to catch. With the SDK
// default ceiling of 10,000 every one of those would collapse into the
// overflow bucket and p95 would read exactly 10 ms however bad it got. The
// shared ladder's upper reaches (25 ms, 50 ms, 100 ms, 250 ms, 500 ms, 1 s
// and beyond) resolve those, and its lower reaches (100 us, 500 us) resolve
// the healthy fresh-node case, so a per-instrument ladder would add a second
// thing to maintain for no extra resolution.
addMicrosecondHistogramView(*views, metric::sweepMallocTrimUs);
// Millisecond dial/resolve latencies. Both exceed the SDK default ceiling
// of 10,000: the dial timer is 15 s, so without an explicit ladder every
// timed-out dial lands in the overflow bucket and p95 reads exactly 10 s
@@ -336,13 +347,39 @@ MetricsRegistry::initExporterAndProvider(std::string const& endpoint, std::strin
addHistogramView(
*views,
metric::dnsResolveLatencyMs,
{1.0, 5.0, 10.0, 25.0, 50.0, 100.0, 250.0, 500.0, 1'000.0, 2'500.0, 5'000.0, 10'000.0,
15'000.0, 20'000.0, 30'000.0});
{1.0,
5.0,
10.0,
25.0,
50.0,
100.0,
250.0,
500.0,
1'000.0,
2'500.0,
5'000.0,
10'000.0,
15'000.0,
20'000.0,
30'000.0});
addHistogramView(
*views,
metric::overlayDialLatencyMs,
{1.0, 5.0, 10.0, 25.0, 50.0, 100.0, 250.0, 500.0, 1'000.0, 2'500.0, 5'000.0, 10'000.0,
15'000.0, 20'000.0, 30'000.0});
{1.0,
5.0,
10.0,
25.0,
50.0,
100.0,
250.0,
500.0,
1'000.0,
2'500.0,
5'000.0,
10'000.0,
15'000.0,
20'000.0,
30'000.0});
// The remaining two GetObject histograms are not durations, so the
// microsecond ladder above does not fit them. Both still need explicit
@@ -635,6 +672,7 @@ MetricsRegistry::registerAsyncGauges()
registerObjectCountGauge();
registerLoadFactorGauge();
registerNodeStoreGauge();
registerRotationStateGauge();
registerServerInfoGauge();
registerBuildInfoGauge();
registerCompleteLedgersGauge();
@@ -967,6 +1005,70 @@ MetricsRegistry::registerNodeStoreGauge()
this);
}
void
MetricsRegistry::registerRotationStateGauge()
{
// --- Sync diagnostics: what an online_delete rotation costs ---
// A rotation performs writes an ordinary fetch would not: the archive
// backend is about to be deleted, so any node body it serves during the
// rotation window has to be rewritten into the writable backend to survive.
// That work scales with the archive, competes with sync I/O, and appears
// ONLY on a populated, already-rotated online_delete database -- which is
// why it never shows up on a fresh node and why it was never measured. The
// count existed as copyForwardCount_ but was log-only and reset per
// rotation.
//
// Polled from the node store rather than pushed, matching
// registerNodeStoreGauge above: DatabaseRotatingImp lives in libxrpl and
// cannot include xrpld/telemetry, so the counters are read through the
// DatabaseRotating accessors on each collection tick instead.
rotationStateGauge_ = meter_->CreateInt64ObservableGauge(
metric::rotationState, "Online-delete rotation state and copy-forward write total");
rotationStateGauge_->AddCallback(
[](opentelemetry::metrics::ObserverResult result, void* state) {
auto* self = static_cast<MetricsRegistry*>(state);
if (self->callbacksDetached_.load(std::memory_order_acquire))
return;
auto& app = self->app_;
try
{
// Only a rotating store has a rotation to report. On a node
// without online_delete the node store is a DatabaseNodeImp, so
// the cast fails and NO series is published -- deliberately, so
// that an absent series means "rotation is not configured"
// rather than a zero that would read as "rotation is free".
auto* rotating = dynamic_cast<NodeStore::DatabaseRotating*>(&app.getNodeStore());
if (rotating == nullptr)
return;
auto observe = [&](char const* field, int64_t value) {
opentelemetry::nostd::get<opentelemetry::nostd::shared_ptr<
opentelemetry::metrics::ObserverResultT<int64_t>>>(result)
->Observe(value, {{label::metric, field}});
};
// The window the extra writes happen in. A panel needs this to
// know when the copy-forward total is expected to move.
observe(
lval::rotation_state::inFlight,
static_cast<int64_t>(rotating->isRotationInFlight() ? 1 : 0));
// Monotonic, so the panel takes rate() over it. The per-rotation
// tally that rotate() logs is reset on every swap and is
// therefore unusable here.
observe(
lval::rotation_state::copyForward,
static_cast<int64_t>(rotating->copyForwardTotal()));
}
catch (...) // NOLINT(bugprone-empty-catch)
{
// Silently skip if services are not yet ready.
}
},
this);
}
void
MetricsRegistry::registerServerInfoGauge()
{
@@ -2183,8 +2285,7 @@ MetricsRegistry::registerNodeStoreLatencyGauge()
lval::nodestore_latency::writeDurationUs,
static_cast<int64_t>(storeDurationUs));
observe(
lval::nodestore_latency::readDurationUs,
static_cast<int64_t>(fetchDurationUs));
lval::nodestore_latency::readDurationUs, static_cast<int64_t>(fetchDurationUs));
if (storeCount > 0 && storeDurationUs > 0)
{

9
src/xrpld/telemetry/MetricsRegistry.h
View File

@@ -645,6 +645,13 @@ private:
* Observable gauges for NodeStore write_load and read_queue.
*/
opentelemetry::nostd::shared_ptr<opentelemetry::metrics::ObservableInstrument> nodeStoreGauge_;
/**
* Observable gauge for online-delete rotation state and its copy-forward
* write total. Publishes nothing on a node without `online_delete`, where
* the node store is not a rotating one.
*/
opentelemetry::nostd::shared_ptr<opentelemetry::metrics::ObservableInstrument>
rotationStateGauge_;
/**
* Observable gauge for server-level health metrics (state, uptime, peers, etc.).
*/
@@ -872,6 +879,8 @@ private:
void
registerNodeStoreGauge(); // Task 9.1
void
registerRotationStateGauge(); // Sync diagnostics: online_delete rotation
void
registerServerInfoGauge(); // Task 9.7a
void
registerBuildInfoGauge(); // Task 9.7b