mirror of
https://github.com/XRPLF/rippled.git
synced 2026-09-29 08:18:01 +00:00
614c1a39ad
The online-delete rotation's cache freshen called TaggedCache::getKeys(),
which held the cache mutex while copying every key. On the dev box's 26
million entry tree-node cache that hold lasted 5-6 s, froze every job
that touches the cache, and dropped the RocksDB node out of sync once per
rotation: each "getKeys held the lock" warning was followed within 1-5 s
by "View of consensus changed" (5 of 5 rotations on 2026-09-15).
Copy the keys one map partition at a time instead. TaggedCache gains
forEachKeyPartition(), which holds the mutex only while one partition's
keys are copied and runs the callback with the mutex released, so the
longest hold shrinks by the partition count (8 on the dev box). The
freshen.keys rotation phase no longer exists as one step, so its span,
stage value, harness entries and docs are removed; the per-partition hold
still shows on the cache lock-hold peak gauge.
Measure what the freshen achieves, which no existing signal did.
DatabaseRotating gains duplicateCopyForwardTotal(), counting archive
copies made on duplicate fetches (the rotation's own copy walk and
freshen); copyForwardTotal() deliberately excludes those. The freshen
phase records rotation_freshen_keys_total{cache,outcome} and stamps
key_count, cache and keys_copied on its span; the copy phase stamps
nodes_copied. A warn log line per freshen reports the same numbers, and
the ledger-sync-health dashboard gets a Rotation Freshen Yield panel.
Log the "STATE->" operating-mode change at warn instead of info. It is
the only record of a mode change with an exact timestamp; the
state_changes_total counter is scraped once a minute and cannot order a
flap against a multi-second event.
Tests: five GTests for forEachKeyPartition (every key once, empty cache,
mutex free during the callback, concurrent insert, lock-hold peak), three
for duplicateCopyForwardTotal over two memory backends, one for the new
counter's series, and the new name literals.
113 lines
4.0 KiB
C++
113 lines
4.0 KiB
C++
#pragma once
|
|
|
|
#include <xrpl/beast/utility/Journal.h>
|
|
#include <xrpl/nodestore/Backend.h>
|
|
#include <xrpl/nodestore/Database.h>
|
|
#include <xrpl/nodestore/Scheduler.h>
|
|
|
|
#include <cstdint>
|
|
#include <functional>
|
|
#include <memory>
|
|
#include <string>
|
|
|
|
namespace xrpl::node_store {
|
|
|
|
/* This class has two key-value store Backend objects for persisting SHAMap
|
|
* records. This facilitates online deletion of data. New backends are
|
|
* rotated in. Old ones are rotated out and deleted.
|
|
*/
|
|
|
|
class DatabaseRotating : public Database
|
|
{
|
|
public:
|
|
DatabaseRotating(
|
|
Scheduler& scheduler,
|
|
int readThreads,
|
|
Section const& config,
|
|
beast::Journal journal)
|
|
: Database(scheduler, readThreads, config, journal)
|
|
{
|
|
}
|
|
|
|
/**
|
|
* Rotates the backends.
|
|
*
|
|
* @param newBackend New writable backend
|
|
* @param f A function executed after the rotation outside of lock. The
|
|
* values passed to f will be the new backend database names _after_
|
|
* rotation.
|
|
*/
|
|
virtual void
|
|
rotate(
|
|
std::unique_ptr<node_store::Backend>&& newBackend,
|
|
std::function<void(std::string const& writableName, std::string const& archiveName)> const&
|
|
f) = 0;
|
|
|
|
/**
|
|
* Marks an online-delete rotation as in progress (or completed).
|
|
*
|
|
* While in flight, a read served by the archive backend is copied
|
|
* forward into the writable backend even for ordinary
|
|
* (duplicate == false) fetches: the archive is about to be deleted,
|
|
* and a node body canonicalized into caches during the rotation
|
|
* window would otherwise survive only in RAM once the archive is
|
|
* dropped.
|
|
*/
|
|
virtual void
|
|
setRotationInFlight(bool inFlight) = 0;
|
|
|
|
/**
|
|
* Whether an online-delete rotation is in progress right now.
|
|
*
|
|
* A rotation's extra writes only happen inside this window, so a panel
|
|
* reading the copy-forward total needs this to know when to expect it to
|
|
* move. Outside the window a flat total is correct, not a broken signal.
|
|
*
|
|
* @return true between the cache-freshen phase starting and rotate()
|
|
* completing.
|
|
*/
|
|
[[nodiscard]] virtual bool
|
|
isRotationInFlight() const = 0;
|
|
|
|
/**
|
|
* Nodes copied forward from the archive backend into the writable one
|
|
* during rotation windows, since this process started.
|
|
*
|
|
* These are writes an ordinary fetch would not have performed: the archive
|
|
* is about to be deleted, so a body it served has to be rewritten to
|
|
* survive. The count therefore scales with how much of the archive is read
|
|
* during a rotation, which is why it appears only on a populated,
|
|
* already-rotated online_delete database.
|
|
*
|
|
* Cumulative for the lifetime of the process, deliberately: the per-rotation
|
|
* tally that `rotate()` logs is reset on every swap, and a counter that goes
|
|
* backwards cannot be rated. A panel takes the rate of this instead.
|
|
*
|
|
* @return Monotonic count of copy-forward writes.
|
|
*/
|
|
[[nodiscard]] virtual std::uint64_t
|
|
copyForwardTotal() const = 0;
|
|
|
|
/**
|
|
* Nodes copied forward from the archive because the caller asked for it.
|
|
*
|
|
* The rotation's copy walk and its cache freshen fetch every node with
|
|
* `duplicate == true`, which copies an archive-served body into the
|
|
* writable backend. copyForwardTotal() does not count those; it counts
|
|
* only ordinary reads copied while a rotation is in flight. This total is
|
|
* the other half: how many of the nodes the rotation asked for were found
|
|
* only in the archive. That is the yield of the copy walk and of the
|
|
* freshen, and the caller cannot measure it any other way because it
|
|
* discards the fetched object.
|
|
*
|
|
* Cumulative for the lifetime of the process. The phase code reads it
|
|
* before and after a step and takes the difference.
|
|
*
|
|
* @return Monotonic count of copy-forward writes made on duplicate fetches.
|
|
*/
|
|
[[nodiscard]] virtual std::uint64_t
|
|
duplicateCopyForwardTotal() const = 0;
|
|
};
|
|
|
|
} // namespace xrpl::node_store
|