ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-21 14:20:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(ledger): stop the sweeper deleting acquires that are still being served

Browse Source 
Two metric-level defects the sync analysis identified, fixed at the source rather
than worked around on the dashboard.

1. InboundLedgers::sweep() destroys any acquire idle for more than a minute, and
   that destruction is what telemetry reports as outcome=abandoned. But
   lastAction_ was only refreshed by the constructor, update() and done() --
   never by the receive path. With JtLedgerData capped at 3 concurrent jobs and
   33 acquires in flight, an acquire whose peers were answering normally could
   wait past the cutoff for its turn to apply data and be deleted for looking
   idle. Measured on a fresh mainnet sync: 490 abandoned acquires against ZERO
   expired retry budgets, so every one was a sweep, not a give-up.

   gotData() now calls touch(). The sweeper's idle test measures real inactivity
   instead of queue wait.

   lastAction_ had to become atomic to allow this. It was a plain
   clock_type::time_point written by the acquiring thread and read by sweep() on
   the timer thread; adding a third writer on peer threads would have been a data
   race. It is now std::atomic<clock_type::duration::rep> with relaxed ordering on
   both sides -- the sweeper compares against a 60-second threshold, so a value
   one tick stale cannot change its decision.

2. getLedgersBehindNetwork() returned the entire ledger sequence space on a fresh
   node. The existing floor only guarded being ahead of every peer; it did not
   guard having validated nothing at all, so validated=0 against a live tip gave
   105,892,534 -- an accurate subtraction of a meaningless quantity. It
   auto-scaled every consumer's axis and would trip any threshold. Distance to
   tip is undefined before the first validated ledger, so it now reports 0 until
   there is one, and the sync-state signals carry the initial-acquire progress.

   The clamp_max(1e6) added to the Ledgers Behind Network panel as a stopgap is
   removed: the metric is correct now, and leaving the clamp would hide a real
   large backlog.

Verified: clang-tidy over the full compile database reports no finding on any
changed line in the three files (the pre-existing misc-include-cleaner and
misc-const-correctness findings elsewhere in InboundLedger.cpp are untouched by
this change). pre-commit passes including clang-format and the Doxygen style
check; validate_dashboards passes.

Not verified: not compiled -- per instructions.md the build needs approval, so CI
is the first real compile of the atomic change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pratik Mankawde
2026-07-28 14:30:08 +01:00
parent fddf78567d
commit df01ba5d3d
4 changed files with 54 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docker/telemetry/grafana/dashboards/ledger-sync-health.json
View File

@@ -1729,7 +1729,7 @@
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "label_replace(label_join(label_replace(clamp_max(sync_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"ledgers_behind\"}, 1000000), \"series\", \"Ledgers Behind\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")",
"expr": "label_replace(label_join(label_replace(sync_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"ledgers_behind\"}, \"series\", \"Ledgers Behind\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")",
"refId": "A"
}
],

37
src/xrpld/app/ledger/InboundLedger.h
View File

@@ -113,16 +113,37 @@ public:
void
runData();
/**
* Mark this acquire as still alive, so the sweeper does not reclaim it.
*
* InboundLedgers::sweep() destroys any acquire whose last action is more
* than a minute old, and that destruction is what the telemetry reports as
* `outcome=abandoned`. Anything that represents real progress must therefore
* call this, or a fetch that is working normally can be deleted for looking
* idle.
*
* @note Thread-safe and lock-free: a relaxed store of the clock's tick
* count. Called from peer threads on the receive path, from the
* acquiring thread, and read by the sweeper on the timer thread, so it
* cannot be a plain member.
*/
void
touch()
{
lastAction_ = clock_.now();
lastAction_.store(clock_.now().time_since_epoch().count(), std::memory_order_relaxed);
}
/**
* When this acquire last made progress, for the sweeper's age check.
*
* @note Thread-safe and lock-free: a relaxed load. A value one tick stale is
* acceptable against a 60-second sweep interval.
*/
clock_type::time_point
getLastAction() const
{
return lastAction_;
return clock_type::time_point{
clock_type::duration{lastAction_.load(std::memory_order_relaxed)}};
}
/**
@@ -354,7 +375,17 @@ private:
std::optional<int> missingNodes) noexcept;
clock_type& clock_;
clock_type::time_point lastAction_;
/**
* Tick count of the last action, as the clock's duration rep.
*
* Stored as the raw rep rather than a `time_point` so it can be atomic: the
* receive path writes it from peer threads while InboundLedgers::sweep()
* reads it from the timer thread. Relaxed on both sides -- the sweeper
* compares against a 60-second threshold, so a value one tick out of date
* cannot change its decision.
*/
std::atomic<clock_type::duration::rep> lastAction_;
std::shared_ptr<Ledger> ledger_;
bool haveHeader_{false};

9
src/xrpld/app/ledger/detail/InboundLedger.cpp
View File

@@ -1430,6 +1430,15 @@ InboundLedger::gotData(
// Mirror the depth for the telemetry gauge, which must not take this lock.
receivedDataDepth_.store(receivedData_.size(), std::memory_order_relaxed);
// A peer just answered, so this acquire is making progress even if its turn
// to apply the data has not come up yet. Without this the sweeper's one
// minute idle test measures the wait for a JtLedgerData slot rather than
// real inactivity, and deletes fetches that are still being served: on a
// fresh mainnet sync that produced 490 abandoned acquires against zero
// expired retry budgets, because only the constructor, update() and done()
// ever refreshed the timestamp.
touch();
if (receiveDispatched_)
return false;

10
src/xrpld/app/misc/NetworkOPs.cpp
View File

@@ -1055,6 +1055,16 @@ NetworkOPsImp::getLedgersBehindNetwork() const
auto const validated = registry_.get().getLedgerMaster().getValidLedgerIndex();
// A node that has validated nothing is not "behind" by the whole sequence
// space; the distance is undefined until there is a validated ledger to
// measure from. Returning the raw difference here reported ~105.9 million on
// a fresh mainnet start, which is a real reading of a meaningless quantity:
// it auto-scaled every consumer's axis and would trip any threshold. Report
// zero until the first ledger is validated, and let the sync-state signals
// say how far along the initial acquire is.
if (validated == 0)
return 0;
// Floor at zero: we can legitimately be ahead of every peer's reported
// range, and a peer that has reported nothing yet leaves the target at 0.
return networkTarget > validated ? networkTarget - validated : 0;