mirror of
https://github.com/XRPLF/rippled.git
synced 2026-08-22 23:00:55 +00:00
feat(telemetry): expose the sweep-trim and rotation costs (WP-B5)
Two suspects from the 3.3.0 slowdown investigation had no signal. Both were already computing the numbers and throwing them away, so this exposes them rather than adding measurement. Per-sweep heap trim. The trim runs after every cache sweep, and its cost scales with resident heap, so it is the leading explanation for a node with a populated database syncing slower than a fresh one. The report already carried duration, fault deltas and reclaimed pages, but the whole measurement sat behind a debug-journal check, so an ordinary node measured nothing, and the call site discarded the result. The measurement now always runs and only the log line stays gated. Records trim duration, minor faults and reclaimed kilobytes. Measured cost of the always-on path is about six microseconds per sweep against a trim costing milliseconds, at a cadence of ten to a hundred and twenty seconds. Honest limit, stated in the runbook: the fault delta spans only the trim call, so it shows the trim itself faulting but not the faults that follow as caches refill. The duration is the signal to correlate against sweep-job queueing. Rotation writes. Rotation copies archive-served reads forward and re-stores nodes missing from both backends, both of which compete with sync I/O and only happen on a populated online_delete database. The copy-forward count existed but was reset by the rotation's own log line, so a metric reading it would drop to zero on every swap; a never-reset total sits beside it now. The re-store count was not measured at all. Rotation duration is deliberately not recorded: the health throttle sleeps at eight points inside the sequence and dominates exactly when the node is unhealthy, so the number would conflate work with waiting. Nothing added for the other two suspects. Get-object serving is already covered by the handler label, the lookup histogram and the deferred and saturation gauges; peer churn by the disconnect-reason counter. Also replaces nine per-file cspell ignores with one ignoreRegExpList entry for the telemetry macro names, and picks up the levelization baseline for the consensus span-name test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,3 +1,4 @@
|
||||
|
||||
#include <xrpld/app/main/Application.h>
|
||||
|
||||
#include <xrpld/app/consensus/RCLValidations.h>
|
||||
@@ -38,6 +39,8 @@
|
||||
#include <xrpld/rpc/detail/PathRequestManager.h>
|
||||
#include <xrpld/rpc/detail/Pathfinder.h>
|
||||
#include <xrpld/shamap/NodeFamily.h>
|
||||
#include <xrpld/telemetry/MetricMacros.h>
|
||||
#include <xrpld/telemetry/MetricNames.h>
|
||||
#include <xrpld/telemetry/MetricsRegistry.h>
|
||||
|
||||
#include <xrpl/basics/ByteUtilities.h>
|
||||
@@ -1131,12 +1134,82 @@ public:
|
||||
<< "; size after: " << cachedSLEs_.size();
|
||||
}
|
||||
|
||||
mallocTrim("doSweep", journal_);
|
||||
trimHeapAndRecord();
|
||||
|
||||
// Set timer to do another sweep later.
|
||||
setSweepTimer();
|
||||
}
|
||||
|
||||
/**
|
||||
* Return free heap pages to the OS at the end of a sweep, and record what
|
||||
* that cost.
|
||||
*
|
||||
* Split out of doSweep() so the metric emit site is one small unit rather
|
||||
* than a further three statements on an already-long function.
|
||||
*
|
||||
* Why this is instrumented: `malloc_trim` runs after EVERY cache sweep, and
|
||||
* its cost scales with the resident heap, so a node with a large existing
|
||||
* database pays a per-sweep penalty a fresh one does not -- the leading
|
||||
* explanation for "an existing database syncs slower than an empty one" on
|
||||
* glibc. The report used to be discarded here, so none of it was visible.
|
||||
*
|
||||
* doSweep() -> trimHeapAndRecord() -> mallocTrim()
|
||||
* | |
|
||||
* | MallocTrimReport (duration,
|
||||
* | minor faults, RSS before/after)
|
||||
* v
|
||||
* 3 OTel instruments
|
||||
*
|
||||
* Cost: one histogram Record and two counter Adds per sweep, at a cadence
|
||||
* of SizedItem::SweepInterval (10-120 s), so this is free.
|
||||
*
|
||||
* @note The minor-fault count covers the trim call only. It cannot show the
|
||||
* faults taken later, as the caches refill and touch the pages the
|
||||
* trim handed back -- see the runbook branch for how to read it.
|
||||
*/
|
||||
void
|
||||
trimHeapAndRecord()
|
||||
{
|
||||
MallocTrimReport const report = mallocTrim("doSweep", journal_);
|
||||
|
||||
// Nothing was measured: not Linux/glibc, so there is no trim to report
|
||||
// and a zero would falsely claim a free one.
|
||||
if (!report.supported)
|
||||
return;
|
||||
|
||||
if (report.durationUs.count() >= 0)
|
||||
{
|
||||
XRPL_METRIC_HISTOGRAM_RECORD(
|
||||
*this,
|
||||
telemetry::metric::sweepMallocTrimUs,
|
||||
"Duration of the malloc_trim call ending each cache sweep (microseconds)",
|
||||
report.durationUs.count());
|
||||
}
|
||||
|
||||
if (report.minfltDelta > 0)
|
||||
{
|
||||
XRPL_METRIC_COUNTER_ADD(
|
||||
*this,
|
||||
telemetry::metric::sweepMallocTrimMinorFaultsTotal,
|
||||
"Minor page faults taken inside the sweep's malloc_trim call",
|
||||
static_cast<std::uint64_t>(report.minfltDelta));
|
||||
}
|
||||
|
||||
// deltaKB() is after-minus-before, so a successful trim is NEGATIVE.
|
||||
// Publish the reclaimed amount as a positive cumulative total and drop
|
||||
// the case where RSS grew across the call (another thread allocating
|
||||
// faster than the trim released): a counter cannot go down, and "grew"
|
||||
// is not a reclaim of a negative size.
|
||||
if (auto const deltaKB = report.deltaKB(); deltaKB < 0)
|
||||
{
|
||||
XRPL_METRIC_COUNTER_ADD(
|
||||
*this,
|
||||
telemetry::metric::sweepMallocTrimReclaimedKbTotal,
|
||||
"Resident kilobytes returned to the OS by the sweep's malloc_trim",
|
||||
static_cast<std::uint64_t>(-deltaKB));
|
||||
}
|
||||
}
|
||||
|
||||
LedgerIndex
|
||||
getMaxDisallowedLedger() override
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user