Compare commits

..

9 Commits

Author SHA1 Message Date
Bart
029a330c91 fix: Make NodePathStack fail closed when assertions are compiled out
The stack asserted its preconditions and then went ahead regardless. Asserts
expand to `assert`, so in a release build every one of those was a no-op in
front of the operation it was guarding: reading or popping an empty
`std::stack` is undefined, `pushChild`'s out-of-range branch check was missing
entirely, and `getChildNodeID` throws `std::logic_error` at leaf depth. None
of these conditions are reachable through any of `SHAMap`'s public entry
points today, but the failure modes if they ever did happen would be
disproportionate: an out-of-range branch would silently corrupt a node ID
instead of failing loudly, and a `logic_error` reaching an unguarded call
chain would abort the process, since nothing in this codebase catches it.

Pushes now return false instead of throwing or silently corrupting the ID, and
reads degrade to a null node rather than undefined behavior. `[[nodiscard]]`
makes an unchecked push a compile error. Each new guard is marked
`UNREACHABLE` rather than left implicitly untested, since no test fixture in
this suite can reach these paths without building a deliberately corrupt map.
`pushRoot` gets the same conversion as every sibling method; it was the one
push still asserting instead of returning false.

`walkTowardsKey`'s two modes (with and without a caller-supplied stack) must
fail at the same node and leave the stack in a state every caller already
knows how to handle; on failure the stack is now cleared via a restored
`clear()`, and a restored `pushCurrent` lambda keeps the loop-entry and
post-loop push-and-clear logic from being duplicated. It also stops deriving
each node ID twice: a caller-supplied stack now reads the ID `pushNode` just
computed off `stack->top().second`, instead of a redundant local copy that
additionally went stale once the loop exited. `belowHelper` and
`peekNextItem` finally get the fallback `top()`'s own docstring promises: both
read `stack.top()` right after an assert-only emptiness check, with no
fallback for release builds, so both now return early on an empty stack
instead of dereferencing a null `SHAMapTreeNodePtr`.

`pushChild`'s hard guard also only checked the parent's depth against
`kLeafDepth`, one level too permissive for an inner child: a parent at 63
passed the check, then pushed an inner child at 64 with only a debug-only
assert catching it, the exact gap this commit exists to close. Tightened to
require depth + 1 below `kLeafDepth` for an inner child, with
`walkTowardsKey`'s no-stack path given the identical tightening so a
malformed map fails at the same node in both modes.
2026-08-21 17:35:53 -04:00
Bart
e7915badd4 refactor: Unify upperBound and lowerBound into boundHelper
The two functions were near duplicates: walk to the key, then look for the
nearest leaf on one side. Only the scan direction, the comparison deciding a
leaf qualifies, and whether to take the first or last leaf below the subtree
differed, exactly the distinction `BelowDirection` already draws for
`belowHelper`, so the pair collapse into one parameterised walk. Also drops
the stale `// TODO: what to return here?` above `lowerBound`'s `return end()`:
no predecessor is the correct answer for the smallest key, and the tests pin
it.

Existing coverage only exercised `boundHelper`'s inner-node branch, every test
map had at least three items, so the root was always an inner node and the
leaf branch at the top of the function was never reached with a real answer to
give. Adds coverage for a single-item map, where the leaf branch alone decides
the outcome, and an empty map, where the scan must find nothing on every
branch before falling through to `end()`.

Fixes the single-item test's own comment, which claimed `root_` becomes a
leaf, when in fact `root_` stays the inner node it was constructed with for
any map built via `addItem`; only a single-item map synced from a peer
(`addRootNode`) ever replaces `root_` with a leaf directly.
2026-08-21 17:35:49 -04:00
Bart
ac8b5e2c90 fix: Derive traversal node IDs from the branch actually descended
`belowHelper` built each stack entry's `SHAMapNodeID` from `branch`, the branch
used to reach the subtree root, rather than `childBranch`, the branch it had
just descended. The resulting IDs carried a correct depth but named a
different subtree, and nothing rejected them: such an ID has a legal depth and
a legal mask, so only comparing it against an actual leaf key exposes the
mismatch. The affected stacks feed read-only traversals whose consumers use
only the depth, so no ledger state, hash, or peer message was affected, but
any future consumer of `getNodeID()` would have silently received the wrong
position.

Rather than fix the one call, make the mistake unrepresentable.
`NodePathStack` replaces the bare `std::stack` and refuses to accept an ID at
all: every push takes the branch being descended and derives the ID itself, so
a node and its ID cannot disagree. `isPrefixOf` assertions on each push catch a
wrong branch at the point it happens rather than wherever the ID is later
read. Leaf entries now keep the depth they were reached at instead of a
normalized `kLeafDepth`, which is what lets those assertions hold:
`addGiveItem` splits a leaf from the depth it actually sits at.

The new traversal tests fail on the previous code: reverting the branch
derivation trips the leaf-key assertion on the first iteration. Also adds a
`deepFanOutKeysAtLeafDepth` helper and mirrors them against it, since the
existing `deepFanOutKeys`'s fan-out at the 6th nibble keeps its tree only
about 6 levels deep and never exercised the depth-63/64 code these tests are
meant to protect, plus a case that collapses the entire depth-63 chain of
single-child inner nodes into a leaf on the final delete, which the
every-other-key deletion pattern the other new tests use never triggers.
2026-08-21 17:35:43 -04:00
Bart
2ab143cb9c fix: Clamp the depth used to index selectBranch's key byte
`selectBranch` reads the key byte at `depth / 2`, which is out of bounds for a
32-byte key once the depth reaches 64. Every branch selection in the map
funnels through here, so this is the one place a stray depth can turn into a
bad read. Assert the precondition for callers, then clamp anyway: a wrong
answer for an input that should never occur is better than reading past the
buffer. Verified under ASan that the unclamped form reads one byte past a
32-byte allocation while the clamped form does not.

`depthMask`'s own 65-entry table had the same exposure one level up, reachable
through the public `createID` factory rather than only from inside the map. A
depth past `kLeafDepth` indexed that table out of bounds, confirmed under ASan
as a 4-byte `global-buffer-overflow` immediately after `kMasks`. Both places
that can set a depth now clamp it: the constructor, which is the single point
every `SHAMapNodeID`'s `depth_` passes through, and `createID`, which needs its
own bound because it picks the mask while evaluating the constructor's
argument, before the constructor body could correct anything.

Clamping rather than throwing, which is what `getChildNodeID` does for the
analogous case: `createID` is reached from `getSHAMapNodeID` with a
peer-supplied depth, and two of that function's three callers
(`InboundTransactions::gotData`, `PeerImp::onMessage`) sit on paths with no
handler between them and a thread boundary, so a throw there would end the
process rather than the message. Clamping also has to fix up `id_` alongside
`depth_`, since a node ID whose id and depth disagree fails the invariant every
read of `id_` relies on. Leaving the depth unclamped would additionally let
`getRawString` narrow it to a byte, turning depth 256 into a node claiming to
be the root.

`deserializeSHAMapNodeID` gets the same mask check `isPrefixOf` already
performs, pulled into a shared `isPrefixOfAtDepth` helper, and the masking both
it and `createID` perform is now a named `maskedToDepth` rather than a repeated
bitwise-and.

Tests cover the depth-sensitivity of `isPrefixOf`, the guards that must hold
with asserts stripped, that `deserializeSHAMapNodeID` rejects an out-of-range
depth, and the clamp itself under both build configurations (`EXPECT_DEATH`
in a forked process when the assert is live, and the clamped result compared
against depth 63 when it is not). The clamp test also gates on
`ENABLE_VOIDSTAR`, not just `NDEBUG`: under Antithesis instrumentation
`XRPL_ASSERT` routes to a handler that records the hit but never aborts, so a
Debug build with voidstar enabled has `NDEBUG` undefined yet still hits the
same non-fatal assert as a release build, and without this gate would send
that configuration into the `EXPECT_DEATH` arm, where the forked child never
dies and the test fails, breaking the CI job that runs this suite under
`-Dvoidstar=ON`.
2026-08-21 17:35:39 -04:00
Bart
5891da909d docs: Correct the reachability claims on the leaf depth guards
The `visitDifferences` guard said an inner node at leaf depth could be seeded by a peer through an
earlier fetch-pack exchange, while the code marks the branch `UNREACHABLE`, which is documented as a
line that must not be reached in normal use or under fuzzing. Both cannot hold: a peer-reachable
assert is a peer-triggered abort. The code is the correct half. `addKnownNode` marks the map invalid
instead of hooking such a node in, fetch-pack blobs are checked against their content hash in
`LedgerMaster::getFetchPack`, and the parent child-hash slots they attach to chain up to a validated
root, so provoking this needs a preimage rather than a crafted message.

The comment now attributes the branch to a defect or a corrupt store, and records why the node is
still reported into the pack before its children are skipped: the wire form of an inner node carries
only child hashes, so the bad depth is never transmitted, and the recipient hooks blobs in by hash at
positions its own traversal picks. Withholding it would instead surface as a peer that cannot
complete a ledger, with nothing to diagnose locally.

The `hasLeafNode` comment credited its caller with a bound that does not apply. That guard limits the
depth of the caller's own traversal, whereas `hasLeafNode` runs on the map passed in and descends
from that map's root, so the check here is the only thing between a malformed map and the throw in
`getChildNodeID`, not a second line of defense. `hasInnerNode` is the one bounded by its caller.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-21 17:35:11 -04:00
Bart
90ef662bc7 test: Cover the leaf key check in verifyProofPath
The check that the terminal leaf of a proof path actually holds the key being proved had no test
covering it, because the substitution it guards against cannot be built by swapping a leaf under a
genuine root: the hash comparison one level up rejects that first.

This test forges the root instead. It puts a genuine leaf from elsewhere in the map at the bottom of
a two-element path and wraps it in an inner node whose branch for the proved key carries that leaf's
hash, so the chain up to the root hash validates even though the leaf belongs to another key. A
control case runs the same forged root over the proved key's own leaf and expects acceptance, which
pins the leaf key comparison as the only reason the substituted path is rejected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-21 17:04:43 -04:00
Bart
d854982fd7 fix: Reject an inner node claimed at leaf depth in verifyProofPath
A SHAMap has 65 levels, and nibbles run out at level 64: `selectBranch` indexes
the key byte at `depth / 2`, so at depth 64 it reads byte 32 of a 32-byte key.
Only the leaf terminating a path may sit at that depth, but proof path nodes
come off the wire, so a peer could send 65 hash-chained inner nodes and drive
that read past the end of the buffer.

The existing length bound cannot be tightened to catch this: a path for two
keys sharing all 63 leading nibbles legitimately holds 64 inner nodes plus a
leaf, so 65 elements is valid. The claimed node type at the final depth is the
thing to reject, using `>=` rather than `==` to match the convention every
other `kLeafDepth` comparison in this codebase already follows.

Reachable from `TMProofPathResponse` via `LedgerReplayMsgHandler`; confirmed
under ASan with asserts compiled out that the unguarded read lands one byte
past a 32-byte heap allocation. Also closes a second gap: nothing checked the
terminal leaf's own key against `key`, so the hash chain alone let a peer
substitute any leaf whose subtree hashes matched at every level above it.
Pinned by tests: the 65-element path that must verify for both keys sharing
the deep prefix, and the forged all-inner path that must not.

Also guards `visitDifferences` against an inner node claimed at leaf depth:
`hasLeafNode` only checked the comparison map, not the map being walked, so a
corrupt node in the map under `visitDifferences` itself could still throw
uncaught. Skip such a node's children instead.
2026-08-18 10:17:53 -04:00
Bart
ca39bff3c8 refactor: Add SHAMapNodeID::isPrefixOf (#7939)
Co-authored-by: Bart <11445373+bthomee@users.noreply.github.com>
2026-08-18 12:35:32 +00:00
Vito Tumas
dd0edc19a0 fix: Conserve funds correctly when LoanPay fee payee is below reserve (#7843) 2026-08-18 11:09:33 +00:00
15 changed files with 1304 additions and 354 deletions

View File

@@ -39,7 +39,6 @@ This section contains changes targeting a future version.
- `TRANSACTION_FLAGS`: Maps transaction type names to their supported flags and flag values.
- `LEDGER_ENTRY_FLAGS`: Maps ledger entry type names to their flags and flag values.
- `ACCOUNT_SET_FLAGS`: Maps AccountSet flag names (asf flags) to their numeric values.
- `submit`: Augmented response fields (`accepted`, `applied`, `broadcast`, `queued`, `kept`, `account_sequence_next`, `account_sequence_available`, `open_ledger_cost`, `validated_ledger_index`) are now included in sign-and-submit mode. Previously, these fields were only returned when submitting a binary transaction blob.
### Bugfixes

View File

@@ -420,7 +420,155 @@ public:
invariants() const;
private:
using SharedPtrNodeStack = std::stack<std::pair<SHAMapTreeNodePtr, SHAMapNodeID>>;
/**
* A path from the root of the map down to some node, pairing each node with the ID naming its
* position.
*
* The two halves of an entry must agree, and the only way to get that wrong is to compute an ID
* from the wrong branch. So this type does not accept an ID at all: every push takes the branch
* being descended and derives the ID itself, so a node and its ID cannot disagree. Reads are
* exposed through the same accessors a std::stack would offer.
*/
class NodePathStack
{
public:
[[nodiscard]] bool
empty() const
{
return stack_.empty();
}
[[nodiscard]] std::size_t
size() const
{
return stack_.size();
}
/**
* The node at the end of the path, paired with its ID.
*
* Reading an empty stack would be undefined, and the assert alone is stripped in release,
* so an empty path yields a null node the caller can test instead.
*/
[[nodiscard]] std::pair<SHAMapTreeNodePtr, SHAMapNodeID> const&
top() const
{
if (stack_.empty())
{
// LCOV_EXCL_START
UNREACHABLE("xrpl::SHAMap::NodePathStack::top : empty stack");
static std::pair<SHAMapTreeNodePtr, SHAMapNodeID> const kEmpty;
return kEmpty;
// LCOV_EXCL_STOP
}
return stack_.top();
}
void
pop()
{
if (stack_.empty())
{
// LCOV_EXCL_START
UNREACHABLE("xrpl::SHAMap::NodePathStack::pop : empty stack");
return;
// LCOV_EXCL_STOP
}
stack_.pop();
}
/**
* Discard the whole path.
*
* For a walk that pushed a node it then found unusable: the node never became a
* meaningful path entry, so it must not be mistaken for one by whatever the caller
* does next with an empty-vs-nonempty check.
*/
void
clear()
{
stack_ = {};
}
/**
* Start a path at the root of the map, whose ID is the zero-depth ID by definition.
*
* @return false, leaving the path unchanged, if a path was already started. A malformed
* call must not abort a release build, so callers stop rather than overwrite it.
*/
[[nodiscard]] bool
pushRoot(SHAMapTreeNodePtr node)
{
if (!stack_.empty())
{
// LCOV_EXCL_START
UNREACHABLE("xrpl::SHAMap::NodePathStack::pushRoot : non-empty stack");
return false;
// LCOV_EXCL_STOP
}
stack_.emplace(std::move(node), SHAMapNodeID{});
return true;
}
/**
* Extend the path to the child of the current node reached by `branch`.
*
* A node keeps the depth it was reached at, never a normalized kLeafDepth. Only a leaf may
* sit at kLeafDepth, since an inner node there would have no branch left to select.
*
* @return false, leaving the path unchanged, if the current node can have no child. A
* malformed map must not abort a release build, so callers stop walking instead.
*/
[[nodiscard]] bool
pushChild(SHAMapTreeNodePtr node, unsigned int branch)
{
if (stack_.empty() || !node || branch >= kBranchFactor)
{
// LCOV_EXCL_START
UNREACHABLE("xrpl::SHAMap::NodePathStack::pushChild : no child to push");
return false;
// LCOV_EXCL_STOP
}
// Only a leaf may sit at kLeafDepth, so an inner child must land one level short of
// it, tighter than the plain depth bound a leaf child needs.
auto const& parentID = stack_.top().second;
auto const parentDepth = parentID.getDepth();
if (node->isInner() ? parentDepth + 1u >= kLeafDepth : parentDepth >= kLeafDepth)
{
// LCOV_EXCL_START
UNREACHABLE("xrpl::SHAMap::NodePathStack::pushChild : no child to push");
return false;
// LCOV_EXCL_STOP
}
auto childID = parentID.getChildNodeID(branch);
XRPL_ASSERT_IF(
node->isLeaf(),
childID.isPrefixOf(leafKey(*node)),
"xrpl::SHAMap::NodePathStack::pushChild : leaf key below branch");
stack_.emplace(std::move(node), std::move(childID));
return true;
}
/**
* Extend the path to a node lying on the path to `target`.
*
* For nodes not reached by descending a known branch: the walk tracks only the key it is
* heading for, or the node is newly created. Either way `target` selects the branch.
*/
[[nodiscard]] bool
pushNode(SHAMapTreeNodePtr node, uint256 const& target)
{
if (stack_.empty())
return pushRoot(std::move(node));
return pushChild(std::move(node), selectBranch(stack_.top().second, target));
}
private:
std::stack<std::pair<SHAMapTreeNodePtr, SHAMapNodeID>> stack_;
};
using DeltaRef =
std::pair<boost::intrusive_ptr<SHAMapItem const>, boost::intrusive_ptr<SHAMapItem const>>;
@@ -447,7 +595,7 @@ private:
* Update hashes up to the root
*/
void
dirtyUp(SharedPtrNodeStack& stack, uint256 const& target, SHAMapTreeNodePtr terminal);
dirtyUp(NodePathStack& stack, uint256 const& target, SHAMapTreeNodePtr terminal);
/**
* Walk towards the specified id, returning the node. Caller must check
@@ -455,7 +603,7 @@ private:
* id
*/
SHAMapLeafNode*
walkTowardsKey(uint256 const& id, SharedPtrNodeStack* stack = nullptr) const;
walkTowardsKey(uint256 const& id, NodePathStack* stack = nullptr) const;
/**
* Return nullptr if key not found
*/
@@ -482,27 +630,19 @@ private:
SHAMapTreeNodePtr
writeNode(NodeObjectType t, SHAMapTreeNodePtr node) const;
// returns the first item at or below this node
SHAMapLeafNode*
firstBelow(SHAMapTreeNodePtr node, SharedPtrNodeStack& stack, unsigned int branch = 0u) const;
// returns the last item at or below this node
SHAMapLeafNode*
lastBelow(
SHAMapTreeNodePtr node,
SharedPtrNodeStack& stack,
unsigned int branch = kBranchFactor) const;
// direction in which belowHelper scans an inner node's branches
// direction in which a scan walks an inner node's branches
enum class BelowDirection { First, Last };
// helper function for firstBelow and lastBelow
/**
* Returns the first or last item at or below the node already on top of `stack`, extending
* `stack` with the path walked to reach it.
*/
SHAMapLeafNode*
belowHelper(
SHAMapTreeNodePtr node,
SharedPtrNodeStack& stack,
unsigned int branch,
BelowDirection direction) const;
belowHelper(NodePathStack& stack, BelowDirection direction) const;
// helper function for upperBound and lowerBound
ConstIterator
boundHelper(uint256 const& id, BelowDirection direction) const;
// Simple descent
// Get a child of the specified node
@@ -550,9 +690,9 @@ private:
hasLeafNode(uint256 const& tag, SHAMapHash const& hash) const;
SHAMapLeafNode const*
peekFirstItem(SharedPtrNodeStack& stack) const;
peekFirstItem(NodePathStack& stack) const;
SHAMapLeafNode const*
peekNextItem(uint256 const& id, SharedPtrNodeStack& stack) const;
peekNextItem(uint256 const& id, NodePathStack& stack) const;
bool
walkBranch(
SHAMapTreeNode* node,
@@ -697,7 +837,7 @@ public:
using pointer = value_type const*;
private:
SharedPtrNodeStack stack_;
NodePathStack stack_;
SHAMap const* map_ = nullptr;
pointer item_ = nullptr;
@@ -723,7 +863,7 @@ public:
private:
explicit ConstIterator(SHAMap const* map);
ConstIterator(SHAMap const* map, std::nullptr_t);
ConstIterator(SHAMap const* map, pointer item, SharedPtrNodeStack&& stack);
ConstIterator(SHAMap const* map, pointer item, NodePathStack&& stack);
friend bool
operator==(ConstIterator const& x, ConstIterator const& y);
@@ -742,10 +882,7 @@ inline SHAMap::ConstIterator::ConstIterator(SHAMap const* map, std::nullptr_t) :
{
}
inline SHAMap::ConstIterator::ConstIterator(
SHAMap const* map,
pointer item,
SharedPtrNodeStack&& stack)
inline SHAMap::ConstIterator::ConstIterator(SHAMap const* map, pointer item, NodePathStack&& stack)
: stack_(std::move(stack)), map_(map), item_(item)
{
}

View File

@@ -55,6 +55,20 @@ public:
[[nodiscard]] SHAMapNodeID
getChildNodeID(unsigned int branch) const;
/**
* Test whether this node ID lies on the path to the given leaf key
*
* A node at depth d identifies the tree path spelled by the first d
* nibbles of its key, so any leaf beneath it must agree on that prefix.
* A node ID that fails this test names a different subtree than the one
* it was built for.
*
* @param key the key of a leaf below this node
* @return whether this node ID is a prefix of the leaf key
*/
[[nodiscard]] bool
isPrefixOf(uint256 const& key) const;
/**
* Create a SHAMapNodeID of a node with the depth of the node and
* the key of a leaf

View File

@@ -97,7 +97,7 @@ SHAMap::snapShot(bool isMutable) const
}
void
SHAMap::dirtyUp(SharedPtrNodeStack& stack, uint256 const& target, SHAMapTreeNodePtr child)
SHAMap::dirtyUp(NodePathStack& stack, uint256 const& target, SHAMapTreeNodePtr child)
{
// walk the tree up from through the inner nodes to the root_
// update hashes and links
@@ -126,29 +126,49 @@ SHAMap::dirtyUp(SharedPtrNodeStack& stack, uint256 const& target, SHAMapTreeNode
}
SHAMapLeafNode*
SHAMap::walkTowardsKey(uint256 const& id, SharedPtrNodeStack* stack) const
SHAMap::walkTowardsKey(uint256 const& id, NodePathStack* stack) const
{
XRPL_ASSERT(
stack == nullptr || stack->empty(), "xrpl::SHAMap::walkTowardsKey : empty stack input");
auto inNode = root_;
SHAMapNodeID nodeID;
// Without a caller-supplied stack, `nodeID` is the only record of position, so it is derived
// directly here instead of read back from a push. A failure below means the map is malformed
// (an inner node one level too deep), not that `id` is merely absent; the stack is cleared
// rather than left holding a node that never became a real path entry.
auto pushCurrent = [&]() -> bool {
if (stack == nullptr || stack->pushNode(inNode, id))
return true;
stack->clear();
return false;
};
while (inNode->isInner())
{
if (stack != nullptr)
stack->emplace(inNode, nodeID);
auto const inner = intr_ptr::staticPointerCast<SHAMapInnerNode>(inNode);
auto const branch = selectBranch(nodeID, id);
if (inner->isEmptyBranch(branch))
if (!pushCurrent())
return nullptr;
inNode = descendThrow(*inner, branch);
nodeID = nodeID.getChildNodeID(branch);
auto& inner = safeDowncast<SHAMapInnerNode&>(*inNode);
auto const branch = selectBranch(stack != nullptr ? stack->top().second : nodeID, id);
if (inner.isEmptyBranch(branch))
return nullptr;
inNode = descendThrow(inner, branch);
if (stack == nullptr)
{
// Only a leaf may sit at kLeafDepth, so an inner child needs the tighter bound: this
// must mirror pushChild's guard exactly, or a malformed map fails one mode earlier
// than the other and stack/no-stack callers disagree on the outcome.
auto const depth = nodeID.getDepth();
if (inNode->isInner() ? depth + 1u >= kLeafDepth : depth >= kLeafDepth)
return nullptr;
nodeID = nodeID.getChildNodeID(branch);
}
}
if (stack != nullptr)
stack->emplace(inNode, nodeID);
if (!pushCurrent())
return nullptr;
return safeDowncast<SHAMapLeafNode*>(inNode.get());
}
@@ -428,65 +448,44 @@ SHAMap::unshareNode(intr_ptr::SharedPtr<Node> node, SHAMapNodeID const& nodeID)
}
SHAMapLeafNode*
SHAMap::belowHelper(
SHAMapTreeNodePtr node,
SharedPtrNodeStack& stack,
unsigned int branch,
BelowDirection direction) const
SHAMap::belowHelper(NodePathStack& stack, BelowDirection direction) const
{
if (node->isLeaf())
{
auto n = intr_ptr::staticPointerCast<SHAMapLeafNode>(node);
stack.push({node, {kLeafDepth, n->peekItem()->key()}});
return n.get();
}
auto inner = intr_ptr::staticPointerCast<SHAMapInnerNode>(node);
XRPL_ASSERT(!stack.empty(), "xrpl::SHAMap::belowHelper : non-empty stack input");
if (stack.empty())
{
stack.emplace(inner, SHAMapNodeID{});
}
else
{
stack.emplace(inner, stack.top().second.getChildNodeID(branch));
}
// `scanned` counts how many branches of `inner` we have examined; the branch we look at is
// derived from it, so no index ever goes out of range.
return nullptr;
if (auto const& top = stack.top().first; top->isLeaf())
return safeDowncast<SHAMapLeafNode*>(top.get());
// The stack owns the node/ID pairing, so descending is only ever "push the branch we took".
// `scanned` counts how many branches of the current node we have examined; the branch we look
// at is derived from it, so no index ever goes out of range. `inner` tracks the node on top of
// the stack, which keeps it alive, so it only needs recomputing after a push.
auto* inner = safeDowncast<SHAMapInnerNode*>(stack.top().first.get());
for (auto scanned = 0u; scanned < kBranchFactor;)
{
auto const childBranch =
(direction == BelowDirection::Last) ? (kBranchFactor - 1u - scanned) : scanned;
if (!inner->isEmptyBranch(childBranch))
{
node.adopt(descendThrow(inner.get(), childBranch));
XRPL_ASSERT(!stack.empty(), "xrpl::SHAMap::belowHelper : non-empty stack");
if (node->isLeaf())
{
auto n = intr_ptr::staticPointerCast<SHAMapLeafNode>(node);
stack.push({n, {kLeafDepth, n->peekItem()->key()}});
return n.get();
}
inner = intr_ptr::staticPointerCast<SHAMapInnerNode>(node);
stack.emplace(inner, stack.top().second.getChildNodeID(branch));
scanned = 0u; // descend and restart the scan on the new node
}
else
if (inner->isEmptyBranch(childBranch))
{
++scanned; // scan next branch
continue;
}
auto descended = descendThrow(*inner, childBranch);
if (!stack.pushChild(std::move(descended), childBranch))
return nullptr;
auto const& child = stack.top().first;
if (child->isLeaf())
return safeDowncast<SHAMapLeafNode*>(child.get());
inner = safeDowncast<SHAMapInnerNode*>(child.get());
scanned = 0u; // descend and restart the scan on the new node
}
return nullptr;
}
SHAMapLeafNode*
SHAMap::lastBelow(SHAMapTreeNodePtr node, SharedPtrNodeStack& stack, unsigned int branch) const
{
return belowHelper(node, stack, branch, BelowDirection::Last);
}
SHAMapLeafNode*
SHAMap::firstBelow(SHAMapTreeNodePtr node, SharedPtrNodeStack& stack, unsigned int branch) const
{
return belowHelper(node, stack, branch, BelowDirection::First);
}
static boost::intrusive_ptr<SHAMapItem const> const kNoItem;
boost::intrusive_ptr<SHAMapItem const> const&
@@ -529,36 +528,42 @@ SHAMap::onlyBelow(SHAMapTreeNode* node) const
}
SHAMapLeafNode const*
SHAMap::peekFirstItem(SharedPtrNodeStack& stack) const
SHAMap::peekFirstItem(NodePathStack& stack) const
{
XRPL_ASSERT(stack.empty(), "xrpl::SHAMap::peekFirstItem : empty stack input");
SHAMapLeafNode const* node = firstBelow(root_, stack);
if (!stack.pushRoot(root_))
return nullptr;
SHAMapLeafNode const* node = belowHelper(stack, BelowDirection::First);
if (node == nullptr)
{
while (!stack.empty())
stack.pop();
// An empty map leaves only the root behind; a failed walk leaves the path it got to.
stack.clear();
return nullptr;
}
return node;
}
SHAMapLeafNode const*
SHAMap::peekNextItem(uint256 const& id, SharedPtrNodeStack& stack) const
SHAMap::peekNextItem(uint256 const& id, NodePathStack& stack) const
{
XRPL_ASSERT(!stack.empty(), "xrpl::SHAMap::peekNextItem : non-empty stack input");
if (stack.empty())
return nullptr;
XRPL_ASSERT(stack.top().first->isLeaf(), "xrpl::SHAMap::peekNextItem : stack starts with leaf");
stack.pop();
while (!stack.empty())
{
auto [node, nodeID] = stack.top();
auto const [node, nodeID] = stack.top();
XRPL_ASSERT(!node->isLeaf(), "xrpl::SHAMap::peekNextItem : another node is not leaf");
auto inner = intr_ptr::staticPointerCast<SHAMapInnerNode>(node);
auto& inner = safeDowncast<SHAMapInnerNode&>(*node);
for (auto i = selectBranch(nodeID, id) + 1; i < kBranchFactor; ++i)
{
if (!inner->isEmptyBranch(i))
if (!inner.isEmptyBranch(i))
{
node = descendThrow(*inner, i);
auto leaf = firstBelow(node, stack, i);
auto child = descendThrow(inner, i);
if (!stack.pushChild(std::move(child), i))
Throw<SHAMapMissingNode>(type_, id);
auto leaf = belowHelper(stack, BelowDirection::First);
if (leaf == nullptr)
Throw<SHAMapMissingNode>(type_, id);
XRPL_ASSERT(leaf->isLeaf(), "xrpl::SHAMap::peekNextItem : leaf is valid");
@@ -595,72 +600,61 @@ SHAMap::peekItem(uint256 const& id, SHAMapHash& hash) const
}
SHAMap::ConstIterator
SHAMap::upperBound(uint256 const& id) const
SHAMap::boundHelper(uint256 const& id, BelowDirection direction) const
{
SharedPtrNodeStack stack;
// Walk back up the path to `id` looking for the nearest leaf on the requested side. At each
// inner node the branches beyond the one `id` takes hold the candidates; the first non-empty
// one is the closest, and the extreme leaf below it is the answer.
auto const searchingForward = direction == BelowDirection::First;
NodePathStack stack;
walkTowardsKey(id, &stack);
while (!stack.empty())
{
auto [node, nodeID] = stack.top();
auto const [node, nodeID] = stack.top();
if (node->isLeaf())
{
auto leaf = safeDowncast<SHAMapLeafNode*>(node.get());
if (leaf->peekItem()->key() > id)
return ConstIterator(this, leaf->peekItem().get(), std::move(stack));
auto const& item = safeDowncast<SHAMapLeafNode const&>(*node).peekItem();
if (searchingForward ? (item->key() > id) : (item->key() < id))
return ConstIterator(this, item.get(), std::move(stack));
}
else
{
auto inner = intr_ptr::staticPointerCast<SHAMapInnerNode>(node);
for (auto branch = selectBranch(nodeID, id) + 1; branch < kBranchFactor; ++branch)
auto& inner = safeDowncast<SHAMapInnerNode&>(*node);
auto const taken = selectBranch(nodeID, id);
auto const remaining = searchingForward ? (kBranchFactor - 1u - taken) : taken;
for (auto scanned = 0u; scanned < remaining; ++scanned)
{
if (!inner->isEmptyBranch(branch))
{
node = descendThrow(*inner, branch);
auto leaf = firstBelow(node, stack, branch);
if (leaf == nullptr)
Throw<SHAMapMissingNode>(type_, id);
return ConstIterator(this, leaf->peekItem().get(), std::move(stack));
}
auto const branch =
searchingForward ? (taken + 1u + scanned) : (taken - 1u - scanned);
if (inner.isEmptyBranch(branch))
continue;
auto child = descendThrow(inner, branch);
if (!stack.pushChild(std::move(child), branch))
Throw<SHAMapMissingNode>(type_, id);
auto const leaf = belowHelper(stack, direction);
if (leaf == nullptr)
Throw<SHAMapMissingNode>(type_, id);
return ConstIterator(this, leaf->peekItem().get(), std::move(stack));
}
}
stack.pop();
}
return end();
}
SHAMap::ConstIterator
SHAMap::upperBound(uint256 const& id) const
{
return boundHelper(id, BelowDirection::First);
}
SHAMap::ConstIterator
SHAMap::lowerBound(uint256 const& id) const
{
SharedPtrNodeStack stack;
walkTowardsKey(id, &stack);
while (!stack.empty())
{
auto [node, nodeID] = stack.top();
if (node->isLeaf())
{
auto leaf = safeDowncast<SHAMapLeafNode*>(node.get());
if (leaf->peekItem()->key() < id)
return ConstIterator(this, leaf->peekItem().get(), std::move(stack));
}
else
{
auto inner = intr_ptr::staticPointerCast<SHAMapInnerNode>(node);
for (auto branch = selectBranch(nodeID, id); branch > 0u;)
{
--branch;
if (!inner->isEmptyBranch(branch))
{
node = descendThrow(*inner, branch);
auto leaf = lastBelow(node, stack, branch);
if (leaf == nullptr)
Throw<SHAMapMissingNode>(type_, id);
return ConstIterator(this, leaf->peekItem().get(), std::move(stack));
}
}
}
stack.pop();
}
// TODO: what to return here?
return end();
return boundHelper(id, BelowDirection::Last);
}
bool
@@ -675,7 +669,7 @@ SHAMap::delItem(uint256 const& id)
// delete the item with this ID
XRPL_ASSERT(state_ != SHAMapState::Immutable, "xrpl::SHAMap::delItem : not immutable");
SharedPtrNodeStack stack;
NodePathStack stack;
walkTowardsKey(id, &stack);
if (stack.empty())
@@ -761,7 +755,7 @@ SHAMap::addGiveItem(SHAMapNodeType type, boost::intrusive_ptr<SHAMapItem const>
// add the specified item, does not update
uint256 const tag = item->key();
SharedPtrNodeStack stack;
NodePathStack stack;
walkTowardsKey(tag, &stack);
if (stack.empty())
@@ -801,7 +795,8 @@ SHAMap::addGiveItem(SHAMapNodeType type, boost::intrusive_ptr<SHAMapItem const>
while ((b1 = selectBranch(nodeID, tag)) == (b2 = selectBranch(nodeID, otherItem->key())))
{
stack.emplace(node, nodeID);
if (!stack.pushNode(node, tag))
Throw<SHAMapMissingNode>(type_, tag);
// we need a new inner node, since both go on same branch at this
// level
@@ -848,7 +843,7 @@ SHAMap::updateGiveItem(SHAMapNodeType type, boost::intrusive_ptr<SHAMapItem cons
XRPL_ASSERT(state_ != SHAMapState::Immutable, "xrpl::SHAMap::updateGiveItem : not immutable");
SharedPtrNodeStack stack;
NodePathStack stack;
walkTowardsKey(tag, &stack);
if (stack.empty())
@@ -1170,7 +1165,7 @@ SHAMap::invariants() const
auto node = root_.get();
XRPL_ASSERT(node, "xrpl::SHAMap::invariants : non-null root node");
XRPL_ASSERT(!node->isLeaf(), "xrpl::SHAMap::invariants : root node is not leaf");
SharedPtrNodeStack stack;
NodePathStack stack;
for (auto leaf = peekFirstItem(stack); leaf != nullptr;
leaf = peekNextItem(leaf->peekItem()->key(), stack))
;

View File

@@ -6,6 +6,7 @@
#include <xrpl/protocol/Serializer.h>
#include <xrpl/shamap/SHAMap.h>
#include <algorithm>
#include <cstddef>
#include <optional>
#include <stdexcept>
@@ -40,14 +41,43 @@ depthMask(unsigned int depth)
return kMasks.entry[depth];
}
// The prefix of `key` at `depth`: the leading nibbles naming the subtree a node at that depth
// identifies, with the remainder of the key masked off.
static uint256
maskedToDepth(uint256 const& key, unsigned int depth)
{
return key & depthMask(depth);
}
// Whether `id` at `depth` is what `key` looks like once masked down to that depth, i.e.
// whether an ID with this depth and id names a subtree that `key` falls under.
static bool
isPrefixOfAtDepth(uint256 const& id, unsigned int depth, uint256 const& key)
{
return maskedToDepth(key, depth) == id;
}
// canonicalize the hash to a node ID for this depth
SHAMapNodeID::SHAMapNodeID(unsigned int depth, uint256 const& hash) : id_(hash), depth_(depth)
{
// Every SHAMapNodeID's depth is stored here, so this is the one place that can stop an
// out-of-range one from being kept: a depth past kLeafDepth would go on to index depthMask
// out of bounds, and getRawString would narrow it to a byte, silently renaming the node.
// Clamp rather than throw, since node IDs are built from peer-supplied depths on the ledger
// data path, where no caller catches an exception before it reaches a thread boundary.
if (depth_ > SHAMap::kLeafDepth)
{
// LCOV_EXCL_START
UNREACHABLE("xrpl::SHAMapNodeID::SHAMapNodeID : depth within tree");
depth_ = SHAMap::kLeafDepth;
id_ = maskedToDepth(id_, depth_);
// LCOV_EXCL_STOP
}
XRPL_ASSERT(
depth <= SHAMap::kLeafDepth, "xrpl::SHAMapNodeID::SHAMapNodeID : maximum depth input");
XRPL_ASSERT(
id_ == (id_ & depthMask(depth)),
"xrpl::SHAMapNodeID::SHAMapNodeID : hash and depth inputs do match");
isPrefixOf(id_), "xrpl::SHAMapNodeID::SHAMapNodeID : hash and depth inputs do match");
}
std::string
@@ -79,7 +109,7 @@ SHAMapNodeID::getChildNodeID(unsigned int branch) const
if (depth_ >= SHAMap::kLeafDepth)
Throw<std::logic_error>("Request for child node ID of " + to_string(*this));
if (id_ != (id_ & depthMask(depth_)))
if (!isPrefixOf(id_))
Throw<std::logic_error>("Incorrect mask for " + to_string(*this));
SHAMapNodeID node{depth_ + 1, id_};
@@ -87,6 +117,12 @@ SHAMapNodeID::getChildNodeID(unsigned int branch) const
return node;
}
bool
SHAMapNodeID::isPrefixOf(uint256 const& key) const
{
return isPrefixOfAtDepth(id_, depth_, key);
}
[[nodiscard]] std::optional<SHAMapNodeID>
deserializeSHAMapNodeID(void const* data, std::size_t size)
{
@@ -97,9 +133,9 @@ deserializeSHAMapNodeID(void const* data, std::size_t size)
unsigned int const depth = *(static_cast<unsigned char const*>(data) + 32);
if (depth <= SHAMap::kLeafDepth)
{
auto const id = uint256::fromVoid(data);
if (id == (id & depthMask(depth)))
// Reject a serialized ID carrying bits below its own depth. Checked before
// constructing, since the constructor asserts that same property.
if (auto const id = uint256::fromVoid(data); isPrefixOfAtDepth(id, depth, id))
ret.emplace(depth, id);
}
}
@@ -110,7 +146,11 @@ deserializeSHAMapNodeID(void const* data, std::size_t size)
[[nodiscard]] unsigned int
selectBranch(SHAMapNodeID const& id, uint256 const& hash)
{
auto const depth = id.getDepth();
XRPL_ASSERT(id.getDepth() < SHAMap::kLeafDepth, "xrpl::selectBranch : depth below leaf depth");
// A depth-64 ID has no nibble left to select. Callers must not ask, but clamp anyway to keep
// the read below the end of the 32-byte key.
auto const depth = std::min(id.getDepth(), SHAMap::kLeafDepth - 1u);
auto branch = static_cast<unsigned int>(*(hash.begin() + (depth / 2)));
if ((depth & 1) != 0u)
@@ -129,8 +169,18 @@ selectBranch(SHAMapNodeID const& id, uint256 const& hash)
SHAMapNodeID
SHAMapNodeID::createID(unsigned int depth, uint256 const& key)
{
XRPL_ASSERT(depth <= SHAMap::kLeafDepth, "xrpl::SHAMapNodeID::createID : valid depth");
return SHAMapNodeID(depth, key & depthMask(depth));
// The mask is chosen here, before the constructor runs, so the clamp there cannot cover this
// call: an out-of-range depth would index depthMask's table while still evaluating this
// argument. A public factory has to hold its own bound.
if (depth > SHAMap::kLeafDepth)
{
// LCOV_EXCL_START
UNREACHABLE("xrpl::SHAMapNodeID::createID : depth within tree");
depth = SHAMap::kLeafDepth;
// LCOV_EXCL_STOP
}
return SHAMapNodeID(depth, maskedToDepth(key, depth));
}
} // namespace xrpl

View File

@@ -143,6 +143,20 @@ SHAMap::visitDifferences(
if (!function(*node))
return;
// Nibbles run out at kLeafDepth, so only a leaf belongs there. A well-formed map never
// holds an inner node at that depth: addKnownNode marks the map invalid rather than hooking
// one in, and fetch-pack data is hash-verified against a validated root, so reaching this
// means a defect or a corrupt store, not something a peer can provoke. Report the node
// anyway - the wire form carries no depth, and the recipient hooks blobs in by hash - but
// skip the children rather than letting getChildNodeID throw on them.
if (nodeID.getDepth() >= kLeafDepth)
{
// LCOV_EXCL_START
UNREACHABLE("xrpl::SHAMap::visitDifferences : inner node at leaf depth");
continue;
// LCOV_EXCL_STOP
}
// 2) push non-matching child inner nodes
for (auto i = 0u; i < kBranchFactor; ++i)
{
@@ -555,10 +569,9 @@ SHAMap::addKnownNode(
{
XRPL_ASSERT(!nodeID.isRoot(), "xrpl::SHAMap::addKnownNode : valid node");
XRPL_ASSERT(treeNode, "xrpl::SHAMap::addKnownNode : non-null tree node");
XRPL_ASSERT(
!treeNode->isLeaf() ||
SHAMapNodeID::createID(nodeID.getDepth(), leafKey(*treeNode)).getNodeID() ==
nodeID.getNodeID(),
XRPL_ASSERT_IF(
treeNode->isLeaf(),
nodeID.isPrefixOf(leafKey(*treeNode)),
"xrpl::SHAMap::addKnownNode : leaf position consistent with node ID");
if (!isSynching())
@@ -750,11 +763,9 @@ SHAMap::hasLeafNode(uint256 const& tag, SHAMapHash const& targetNodeHash) const
do
{
// An inner node is only reachable here at a depth below kLeafDepth in a well-formed map,
// where the loop always finds a leaf first. A malformed map could still have an inner
// node claiming kLeafDepth, and getChildNodeID below throws in that case: reject rather
// than let the throw escape uncaught. Not reachable through any public entry point,
// since addKnownNode already marks such a map invalid, so no test can cover this.
// Same kLeafDepth hazard as in visitDifferences above. That guard bounds the caller's own
// traversal, not the map queried here, and the loop below descends from this map's root
// independently, so this check is what keeps a malformed map from reaching getChildNodeID.
if (nodeID.getDepth() >= kLeafDepth)
{
// LCOV_EXCL_START
@@ -782,7 +793,7 @@ SHAMap::hasLeafNode(uint256 const& tag, SHAMapHash const& targetNodeHash) const
std::optional<std::vector<Blob>>
SHAMap::getProofPath(uint256 const& key) const
{
SharedPtrNodeStack stack;
NodePathStack stack;
walkTowardsKey(key, &stack);
if (stack.empty())
@@ -831,15 +842,30 @@ SHAMap::verifyProofPath(uint256 const& rootHash, uint256 const& key, std::vector
if (node->getHash() != hash)
return false;
auto const depth = std::distance(path.rbegin(), rit);
auto const depth = static_cast<unsigned int>(std::distance(path.rbegin(), rit));
if (node->isInner())
{
auto nodeId = SHAMapNodeID::createID(static_cast<unsigned int>(depth), key);
// Nibbles run out at kLeafDepth, so only the leaf terminating the path may sit
// there. These nodes come off the wire, so a peer can still claim an inner one;
// reject it rather than passing this depth to selectBranch.
SOMETIMES(
depth >= kLeafDepth, "xrpl::SHAMap::verifyProofPath : inner at leaf depth");
if (depth >= kLeafDepth)
return false;
auto nodeId = SHAMapNodeID::createID(depth, key);
hash = safeDowncast<SHAMapInnerNode*>(node.get())
->getChildHash(selectBranch(nodeId, key));
}
else
{
// The hash chain up to rootHash only proves this leaf sits where the path claims,
// not that it is the leaf for `key`: a peer could substitute any other leaf whose
// subtree hashes to the same value at every level above it. Checking the terminal
// leaf's own key is what ties the proof to `key` specifically.
if (leafKey(*node) != key)
return false;
// should exhaust all the blobs now
return depth + 1 == path.size();
}

View File

@@ -2,6 +2,7 @@
#include <xrpl/basics/Log.h>
#include <xrpl/basics/Number.h>
#include <xrpl/beast/utility/Journal.h>
#include <xrpl/beast/utility/Zero.h>
#include <xrpl/beast/utility/instrumentation.h>
#include <xrpl/json/to_string.h>
@@ -9,6 +10,8 @@
#include <xrpl/ledger/View.h>
#include <xrpl/ledger/helpers/LendingHelpers.h>
#include <xrpl/ledger/helpers/TokenHelpers.h>
#include <xrpl/protocol/AccountID.h>
#include <xrpl/protocol/Asset.h>
#include <xrpl/protocol/Feature.h>
#include <xrpl/protocol/Indexes.h>
#include <xrpl/protocol/LedgerFormats.h>
@@ -33,6 +36,34 @@
namespace xrpl {
namespace {
// Returns the account's true, unclamped balance in `asset`, for use only in
// fund-conservation checks. accountHolds(..., SpendableHandling::FullBalance)
// cannot be used for this: for XRP it always defers to xrpLiquid, which
// subtracts the account's reserve, so a payee sitting below its own reserve
// would appear to receive nothing even though its raw ledger balance grew.
// That mismatch is exactly what a conservation check must not see.
STAmount
conservationBalance(ReadView const& view, AccountID const& id, Asset const& asset, beast::Journal j)
{
if (isXRP(asset))
{
auto const sle = view.read(keylet::account(id));
if (!sle)
return STAmount{asset}; // LCOV_EXCL_LINE
return view.balanceHookIOU(id, xrpAccount(), sle->getFieldAmount(sfBalance));
}
return accountHolds(
view,
id,
asset,
FreezeHandling::IgnoreFreeze,
AuthHandling::IgnoreAuth,
j,
SpendableHandling::FullBalance);
}
} // namespace
bool
LoanPay::checkExtraFeatures(PreflightContext const& ctx)
{
@@ -581,34 +612,13 @@ LoanPay::doApply()
}
// These three values are used to check that funds are conserved after the transfers
auto const accountBalanceBefore = accountHolds(
view,
accountID_,
asset,
FreezeHandling::IgnoreFreeze,
AuthHandling::IgnoreAuth,
j_,
SpendableHandling::FullBalance);
auto const accountBalanceBefore = conservationBalance(view, accountID_, asset, j_);
auto const vaultBalanceBefore = accountID_ == vaultPseudoAccount
? STAmount{asset, 0}
: accountHolds(
view,
vaultPseudoAccount,
asset,
FreezeHandling::IgnoreFreeze,
AuthHandling::IgnoreAuth,
j_,
SpendableHandling::FullBalance);
: conservationBalance(view, vaultPseudoAccount, asset, j_);
auto const brokerBalanceBefore = accountID_ == brokerPayee
? STAmount{asset, 0}
: accountHolds(
view,
brokerPayee,
asset,
FreezeHandling::IgnoreFreeze,
AuthHandling::IgnoreAuth,
j_,
SpendableHandling::FullBalance);
: conservationBalance(view, brokerPayee, asset, j_);
if (totalPaidToVaultRounded != beast::kZero)
{
@@ -664,33 +674,13 @@ LoanPay::doApply()
#endif
// Check that funds are conserved
auto const accountBalanceAfter = accountHolds(
view,
accountID_,
asset,
FreezeHandling::IgnoreFreeze,
AuthHandling::IgnoreAuth,
j_,
SpendableHandling::FullBalance);
auto const accountBalanceAfter = conservationBalance(view, accountID_, asset, j_);
auto const vaultBalanceAfter = accountID_ == vaultPseudoAccount
? STAmount{asset, 0}
: accountHolds(
view,
vaultPseudoAccount,
asset,
FreezeHandling::IgnoreFreeze,
AuthHandling::IgnoreAuth,
j_,
SpendableHandling::FullBalance);
auto const brokerBalanceAfter = accountID_ == brokerPayee ? STAmount{asset, 0}
: accountHolds(
view,
brokerPayee,
asset,
FreezeHandling::IgnoreFreeze,
AuthHandling::IgnoreAuth,
j_,
SpendableHandling::FullBalance);
: conservationBalance(view, vaultPseudoAccount, asset, j_);
auto const brokerBalanceAfter = accountID_ == brokerPayee
? STAmount{asset, 0}
: conservationBalance(view, brokerPayee, asset, j_);
auto const balanceScale = [&]() {
// Find a reasonable scale to use for the balance comparisons.
//

View File

@@ -5,6 +5,7 @@
#include <test/jtx/amount.h>
#include <test/jtx/fee.h>
#include <test/jtx/jtx_json.h>
#include <test/jtx/noop.h>
#include <test/jtx/pay.h>
#include <test/jtx/ter.h>
#include <test/jtx/trust.h>
@@ -13,6 +14,7 @@
#include <xrpl/beast/unit_test/suite.h>
#include <xrpl/beast/utility/Zero.h>
#include <xrpl/json/json_value.h>
#include <xrpl/ledger/helpers/LendingHelpers.h>
#include <xrpl/protocol/Feature.h>
#include <xrpl/protocol/Indexes.h>
#include <xrpl/protocol/Issue.h>
@@ -728,6 +730,110 @@ private:
}
}
void
testLoanPayFundsConservedPayeeBelowReserve(FeatureBitset features)
{
// Regression test: LoanPay::doApply's fund-conservation check used to
// read XRP balances via accountHolds(..., SpendableHandling::
// FullBalance), which for XRP always defers to xrpLiquid (balance
// minus reserve, clamped at zero). When the broker fee landed on a
// payee sitting below its own reserve, that payee's clamped balance
// stayed zero and the fee vanished from the conservation sum,
// tripping "funds are conserved (with rounding)".
testcase("LoanPay funds conserved: broker fee payee below reserve");
using namespace jtx;
Env env(*this, features);
Account const issuer{"issuer"};
Account const lender{"lender"};
Account const borrower{"borrower"};
// Broker defaults match the fuzz workload: ManagementFeeRate = 100
// tenth-bips. The service fee guarantees feePaid > 0 on the first
// regular payment.
BrokerParameters const brokerParams;
Number const serviceFeeValue{2};
LoanParameters const loanParams{
.account = borrower,
.counter = lender,
.principalRequest = 1000,
.serviceFee = serviceFeeValue,
.interest = TenthBips32{percentageToTenthBips(12)},
.payTotal = 12,
.payInterval = 3600};
auto const loanOpt =
createLoan(env, AssetType::XRP, brokerParams, loanParams, issuer, lender, borrower);
if (BEAST_EXPECT(loanOpt); !loanOpt.has_value())
return;
auto const& [broker, loanKeylet, brokerPseudo] = *loanOpt;
auto const vaultPseudo = [&]() {
auto const vaultSle = env.le(keylet::vault(broker.vaultID));
if (!BEAST_EXPECT(vaultSle))
return AccountID{};
return vaultSle->at(sfAccount);
}();
// Raw AccountRoot balance, matching LoanPay::doApply's conservation
// check (not the reserve-clamped accountHolds()/xrpLiquid() value).
auto rawBalance = [&](AccountID const& id) -> STAmount {
auto const sle = env.le(keylet::account(id));
if (!BEAST_EXPECT(sle))
return STAmount{};
return sle->getFieldAmount(sfBalance);
};
auto lenderReserve = [&] {
return env.current()->fees().accountReserve(ownerCount(env, lender), 1);
};
STAmount const baseFee{env.current()->fees().base};
// Park the lender (broker owner, fee payee) exactly at its reserve,
// then burn part of the reserve with an oversized transaction fee.
// Fees are exempt from the reserve check, so the balance ends up
// below the reserve.
env(pay(lender, issuer, rawBalance(lender.id()) - lenderReserve() - baseFee));
env(noop(lender), Fee(XRP(100)));
env.close();
BEAST_EXPECT(env.balance(lender) < lenderReserve());
// First regular payment, exactly the amount due.
auto const state = getCurrentState(env, broker, loanKeylet);
STAmount const serviceFee = broker.asset(serviceFeeValue);
STAmount const roundedPeriodicPayment{
broker.asset,
roundPeriodicPayment(broker.asset, state.periodicPayment, state.loanScale)};
STAmount const totalDue = roundToScale(
roundedPeriodicPayment + serviceFee, state.loanScale, Number::RoundingMode::Upward);
auto const borrowerBefore = rawBalance(borrower.id());
auto const vaultBefore = rawBalance(vaultPseudo);
auto const lenderBefore = rawBalance(lender.id());
// Before the fix, this aborted inside LoanPay::doApply on
// XRPL_ASSERT_PARTS(goodRounding, "xrpl::LoanPay::doApply", "funds
// are conserved (with rounding)").
env(loan::pay(borrower, loanKeylet.key, totalDue));
env.close();
auto const borrowerAfter = rawBalance(borrower.id());
auto const vaultAfter = rawBalance(vaultPseudo);
auto const lenderAfter = rawBalance(lender.id());
// The broker fee reached the lender's AccountRoot, even though the
// lender's balance remains below its reserve.
BEAST_EXPECT(lenderAfter > lenderBefore);
BEAST_EXPECT(lenderAfter < lenderReserve());
// Total funds conserved across the payer, vault, and fee payee.
BEAST_EXPECT(
borrowerBefore - baseFee + vaultBefore + lenderBefore ==
borrowerAfter + vaultAfter + lenderAfter);
}
void
runAmendmentIndependent()
{
@@ -741,6 +847,7 @@ private:
#if LOAN_TODO
testLoanPayLateFullPaymentBypassesPenalties(features);
#endif
testLoanPayFundsConservedPayeeBelowReserve(features);
testOverpaymentManagementFee(features);
testDosLoanPay(features);
testLoanNextPaymentDueDateOverflow(features);

View File

@@ -2,104 +2,20 @@
#include <test/jtx/Env.h>
#include <test/jtx/JTx.h>
#include <test/jtx/amount.h>
#include <test/jtx/envconfig.h>
#include <test/jtx/pay.h>
#include <xrpld/core/Config.h>
#include <xrpl/basics/strHex.h>
#include <xrpl/beast/unit_test/suite.h>
#include <xrpl/config/Constants.h>
#include <xrpl/json/json_value.h>
#include <xrpl/json/to_string.h>
#include <xrpl/protocol/Seed.h>
#include <xrpl/protocol/Serializer.h>
#include <xrpl/protocol/jss.h>
#include <memory>
namespace xrpl::test {
class Submit_test : public beast::unit_test::Suite
{
public:
void
testAugmentedFields()
{
testcase("Augmented fields in sign-and-submit mode");
using namespace jtx;
// Enable signing support in config
Env env{*this, envconfig([](std::unique_ptr<Config> cfg) {
static std::string const kSigningSupportCfg =
std::string("[") + Sections::kSigningSupport + "]\ntrue";
cfg->loadFromString(kSigningSupportCfg);
return cfg;
})};
Account const alice{"alice"};
Account const bob{"bob"};
env.fund(XRP(10000), alice, bob);
env.close();
// Test 1: Sign-and-submit mode should return augmented fields
{
json::Value jv;
jv[jss::tx_json][jss::TransactionType] = jss::Payment;
jv[jss::tx_json][jss::Account] = alice.human();
jv[jss::tx_json][jss::Destination] = bob.human();
jv[jss::tx_json][jss::Amount] = XRP(100).value().getJson();
jv[jss::secret] = alice.name();
auto const result = env.rpc("json", "submit", to_string(jv))[jss::result];
// These are the augmented fields that should be present
BEAST_EXPECT(result.isMember(jss::engine_result));
BEAST_EXPECT(result.isMember(jss::engine_result_code));
BEAST_EXPECT(result.isMember(jss::engine_result_message));
// New augmented fields from issue #3125
BEAST_EXPECT(result.isMember(jss::accepted));
BEAST_EXPECT(result.isMember(jss::applied));
BEAST_EXPECT(result.isMember(jss::broadcast));
BEAST_EXPECT(result.isMember(jss::queued));
BEAST_EXPECT(result.isMember(jss::kept));
// Current ledger state fields
BEAST_EXPECT(result.isMember(jss::account_sequence_next));
BEAST_EXPECT(result.isMember(jss::account_sequence_available));
BEAST_EXPECT(result.isMember(jss::open_ledger_cost));
BEAST_EXPECT(result.isMember(jss::validated_ledger_index));
// Verify basic transaction fields
BEAST_EXPECT(result.isMember(jss::tx_blob));
BEAST_EXPECT(result.isMember(jss::tx_json));
}
// Test 2: Binary blob mode should also return augmented fields (regression test)
{
auto jt = env.jt(pay(alice, bob, XRP(100)));
Serializer s;
jt.stx->add(s);
auto const result = env.rpc("submit", strHex(s.slice()))[jss::result];
// Verify augmented fields are present in binary mode too
BEAST_EXPECT(result.isMember(jss::engine_result));
BEAST_EXPECT(result.isMember(jss::accepted));
BEAST_EXPECT(result.isMember(jss::applied));
BEAST_EXPECT(result.isMember(jss::broadcast));
BEAST_EXPECT(result.isMember(jss::queued));
BEAST_EXPECT(result.isMember(jss::kept));
BEAST_EXPECT(result.isMember(jss::account_sequence_next));
BEAST_EXPECT(result.isMember(jss::account_sequence_available));
BEAST_EXPECT(result.isMember(jss::open_ledger_cost));
BEAST_EXPECT(result.isMember(jss::validated_ledger_index));
}
}
void
testFailHardValidation()
{
@@ -173,7 +89,6 @@ public:
void
run() override
{
testAugmentedFields();
testFailHardValidation();
}
};

View File

@@ -3,19 +3,23 @@
#include <xrpl/basics/Blob.h>
#include <xrpl/basics/Buffer.h>
#include <xrpl/basics/SHAMapHash.h>
#include <xrpl/basics/Slice.h>
#include <xrpl/basics/base_uint.h>
#include <xrpl/beast/utility/Journal.h>
#include <xrpl/beast/utility/Zero.h>
#include <xrpl/protocol/Serializer.h>
#include <xrpl/shamap/SHAMapInnerNode.h>
#include <xrpl/shamap/SHAMapItem.h>
#include <xrpl/shamap/SHAMapLeafNode.h>
#include <xrpl/shamap/SHAMapMissingNode.h>
#include <xrpl/shamap/SHAMapNodeID.h>
#include <xrpl/shamap/SHAMapTreeNode.h>
#include <gtest/gtest.h>
#include <helpers/TestSink.h>
#include <shamap/common.h>
#include <algorithm>
#include <array>
#include <cstddef>
#include <cstdint>
@@ -268,6 +272,411 @@ INSTANTIATE_TEST_SUITE_P(
::testing::Values(kBackedMode, kUnbackedMode),
shamapBackingModeName);
// Exercises the traversal stacks built by firstBelow/lastBelow. Each stack entry pairs a node with
// the ID naming its position, and SHAMap asserts that pairing on every push, so these traversals
// fail loudly in a Debug build if a node ID is ever derived from the wrong branch.
class SHAMapTraversal : public ::testing::Test
{
protected:
beast::Journal const j_{TestSink::instance()};
// Keys that share a long prefix and then fan out across distinct branches, so the deeper inner
// nodes have several children and traversal must descend many levels.
static std::vector<uint256>
deepFanOutKeys()
{
std::vector<uint256> keys;
for (unsigned int branch = 0; branch < SHAMap::kBranchFactor; ++branch)
{
// Vary the 6th nibble, keeping the first five identical.
auto text = std::string("abcde") + "0123456789abcdef"[branch];
text.append(64 - text.size(), '7');
keys.emplace_back(std::string_view{text});
}
return keys;
}
// Keys that share all 63 leading nibbles and fan out only at the last one, so the tree is a
// chain of single-child inner nodes down to depth 63 with the leaves as siblings at depth 64.
// This exercises kLeafDepth directly, unlike deepFanOutKeys() above, whose fan-out at the 6th
// nibble keeps the tree only about 6 levels deep.
static std::vector<uint256>
deepFanOutKeysAtLeafDepth()
{
std::vector<uint256> keys;
for (unsigned int branch = 0; branch < SHAMap::kBranchFactor; ++branch)
{
auto text = std::string(63, 'a') + "0123456789abcdef"[branch];
keys.emplace_back(std::string_view{text});
}
return keys;
}
static void
fillMap(SHAMap& map, std::vector<uint256> const& keys)
{
map.setUnbacked();
for (auto const& k : keys)
{
Buffer vuc{32};
std::fill_n(vuc.data(), vuc.size(), std::uint8_t{1});
map.addItem(SHAMapNodeType::TnAccountState, makeShamapitem(k, std::move(vuc)));
map.invariants();
}
}
};
TEST_F(SHAMapTraversal, forward_iteration_visits_every_key_in_order)
{
tests::TestNodeFamily f{j_};
auto keys = deepFanOutKeys();
SHAMap map{SHAMapType::FREE, f};
fillMap(map, keys);
std::sort(keys.begin(), keys.end());
std::vector<uint256> visited;
for (auto const& item : map)
visited.push_back(item.key());
EXPECT_EQ(visited, keys);
}
TEST_F(SHAMapTraversal, upper_bound_walks_the_whole_map)
{
tests::TestNodeFamily f{j_};
auto keys = deepFanOutKeys();
SHAMap map{SHAMapType::FREE, f};
fillMap(map, keys);
std::sort(keys.begin(), keys.end());
// upperBound from each key must land on its successor, driving firstBelow across every subtree.
for (std::size_t k = 0; k + 1 < keys.size(); ++k)
{
auto it = map.upperBound(keys[k]);
ASSERT_NE(it, map.end()) << "no successor for key " << k;
EXPECT_EQ(it->key(), keys[k + 1]) << "wrong successor for key " << k;
}
EXPECT_EQ(map.upperBound(keys.back()), map.end());
}
TEST_F(SHAMapTraversal, lower_bound_walks_the_whole_map)
{
tests::TestNodeFamily f{j_};
auto keys = deepFanOutKeys();
SHAMap map{SHAMapType::FREE, f};
fillMap(map, keys);
std::sort(keys.begin(), keys.end());
// lowerBound is the lastBelow counterpart: it descends to the greatest key below a subtree.
for (std::size_t k = 1; k < keys.size(); ++k)
{
auto it = map.lowerBound(keys[k]);
ASSERT_NE(it, map.end()) << "no predecessor for key " << k;
EXPECT_EQ(it->key(), keys[k - 1]) << "wrong predecessor for key " << k;
}
EXPECT_EQ(map.lowerBound(keys.front()), map.end());
}
TEST_F(SHAMapTraversal, bounds_agree_with_iteration_for_absent_keys)
{
tests::TestNodeFamily f{j_};
auto keys = deepFanOutKeys();
SHAMap map{SHAMapType::FREE, f};
fillMap(map, keys);
std::sort(keys.begin(), keys.end());
// Probe keys that are not in the map, so the traversal starts mid-tree rather than at a leaf.
for (unsigned char c : {0x00, 0x40, 0x80, 0xc0, 0xff})
{
uint256 probe;
std::fill_n(probe.begin(), probe.size(), c);
auto const expectedUpper = std::upper_bound(keys.begin(), keys.end(), probe);
auto const upper = map.upperBound(probe);
if (expectedUpper == keys.end())
{
EXPECT_EQ(upper, map.end()) << "probe " << static_cast<unsigned>(c);
}
else
{
ASSERT_NE(upper, map.end()) << "probe " << static_cast<unsigned>(c);
EXPECT_EQ(upper->key(), *expectedUpper) << "probe " << static_cast<unsigned>(c);
}
auto const lowerCount = std::lower_bound(keys.begin(), keys.end(), probe) - keys.begin();
auto const lower = map.lowerBound(probe);
if (lowerCount == 0)
{
EXPECT_EQ(lower, map.end()) << "probe " << static_cast<unsigned>(c);
}
else
{
ASSERT_NE(lower, map.end()) << "probe " << static_cast<unsigned>(c);
EXPECT_EQ(lower->key(), keys[lowerCount - 1]) << "probe " << static_cast<unsigned>(c);
}
}
}
TEST_F(SHAMapTraversal, bounds_on_empty_map_return_end)
{
tests::TestNodeFamily f{j_};
SHAMap map{SHAMapType::FREE, f};
map.setUnbacked();
// The root is a childless inner node, so boundHelper's inner-node branch must scan all 16
// branches, find every one empty, and fall through to end() rather than dereference a child.
EXPECT_EQ(map.upperBound(uint256{}), map.end());
EXPECT_EQ(map.lowerBound(uint256{}), map.end());
uint256 probe;
std::fill_n(probe.begin(), probe.size(), std::uint8_t{0xff});
EXPECT_EQ(map.upperBound(probe), map.end());
EXPECT_EQ(map.lowerBound(probe), map.end());
}
TEST_F(SHAMapTraversal, bounds_on_single_item_map_use_the_leaf_root)
{
tests::TestNodeFamily f{j_};
SHAMap map{SHAMapType::FREE, f};
auto const key = deepFanOutKeys().front();
fillMap(map, {key});
// root_ can be a leaf, but only after syncing a single-item map from a peer (addRootNode);
// fillMap builds this map in-process via addItem, which always leaves root_ as the inner node
// it was constructed with, with the single leaf one level below it. So here the stack holds
// that inner root plus the leaf, and boundHelper's leaf branch, examined first, decides the
// outcome before root_'s own inner-node scan would ever run.
uint256 below = key;
--below;
uint256 above = key;
++above;
EXPECT_EQ(map.upperBound(below)->key(), key);
EXPECT_EQ(map.upperBound(key), map.end());
EXPECT_EQ(map.upperBound(above), map.end());
EXPECT_EQ(map.lowerBound(above)->key(), key);
EXPECT_EQ(map.lowerBound(key), map.end());
EXPECT_EQ(map.lowerBound(below), map.end());
}
TEST_F(SHAMapTraversal, iteration_survives_deletions)
{
tests::TestNodeFamily f{j_};
auto keys = deepFanOutKeys();
SHAMap map{SHAMapType::FREE, f};
fillMap(map, keys);
std::sort(keys.begin(), keys.end());
// Deleting every other key drops the fan-out node's branch count from 16 to 8, never the 1
// that would make delItem collapse it into a leaf. So this pins that iteration survives
// deletions that reshape the map without collapsing any inner node; the case that does
// collapse one is iteration_survives_a_collapsed_inner_node below.
for (std::size_t k = 0; k < keys.size(); k += 2)
{
ASSERT_TRUE(map.delItem(keys[k]));
map.invariants();
}
std::vector<uint256> expected;
for (std::size_t k = 1; k < keys.size(); k += 2)
expected.push_back(keys[k]);
std::vector<uint256> visited;
for (auto const& item : map)
visited.push_back(item.key());
EXPECT_EQ(visited, expected);
for (std::size_t k = 0; k + 1 < expected.size(); ++k)
{
auto it = map.upperBound(expected[k]);
ASSERT_NE(it, map.end());
EXPECT_EQ(it->key(), expected[k + 1]);
}
}
TEST_F(SHAMapTraversal, iteration_survives_a_collapsed_inner_node)
{
tests::TestNodeFamily f{j_};
SHAMap map{SHAMapType::FREE, f};
// One key in a separate subtree, diverging from the fan-out group at the very first nibble, so
// it survives untouched while the fan-out group below is collapsed.
auto const sentinel = uint256{std::string_view{std::string(64, '0')}};
auto fanOutKeys = deepFanOutKeysAtLeafDepth();
fillMap(map, fanOutKeys);
Buffer vuc{32};
std::fill_n(vuc.data(), vuc.size(), std::uint8_t{1});
ASSERT_TRUE(
map.addItem(SHAMapNodeType::TnAccountState, makeShamapitem(sentinel, std::move(vuc))));
map.invariants();
std::sort(fanOutKeys.begin(), fanOutKeys.end());
// Delete all but the last fan-out key. The fan-out node's branch count drops to 1 on the final
// delete, which delItem collapses by pulling the sole remaining leaf up in its place; every
// ancestor above it has exactly one child by construction, so each of those also drops to
// branch count 1 and collapses in turn, all the way up to (but not including) the root. That
// final delete replaces the entire 63-level chain with the root pointing straight at the one
// remaining leaf, so the surviving traversal stack is rebuilt over a drastically different tree
// shape, not just missing one inner node.
for (std::size_t k = 0; k + 1 < fanOutKeys.size(); ++k)
{
ASSERT_TRUE(map.delItem(fanOutKeys[k]));
map.invariants();
}
std::vector<uint256> const expected{sentinel, fanOutKeys.back()};
std::vector<uint256> visited;
for (auto const& item : map)
visited.push_back(item.key());
EXPECT_EQ(visited, expected);
auto it = map.upperBound(sentinel);
ASSERT_NE(it, map.end());
EXPECT_EQ(it->key(), fanOutKeys.back());
EXPECT_EQ(map.upperBound(fanOutKeys.back()), map.end());
}
// The tests below mirror the ones above but use deepFanOutKeysAtLeafDepth(), whose keys share all
// 63 leading nibbles and fan out only at the last one. That puts the leaves at depth
// SHAMap::kLeafDepth, so these traversals walk a chain of single-child inner nodes all the way down
// and exercise the kLeafDepth guards that deepFanOutKeys() alone (fanning out at the 6th nibble)
// never reaches.
TEST_F(SHAMapTraversal, forward_iteration_visits_every_key_in_order_at_leaf_depth)
{
tests::TestNodeFamily f{j_};
auto keys = deepFanOutKeysAtLeafDepth();
SHAMap map{SHAMapType::FREE, f};
fillMap(map, keys);
std::sort(keys.begin(), keys.end());
std::vector<uint256> visited;
for (auto const& item : map)
visited.push_back(item.key());
EXPECT_EQ(visited, keys);
}
TEST_F(SHAMapTraversal, upper_bound_walks_the_whole_map_at_leaf_depth)
{
tests::TestNodeFamily f{j_};
auto keys = deepFanOutKeysAtLeafDepth();
SHAMap map{SHAMapType::FREE, f};
fillMap(map, keys);
std::sort(keys.begin(), keys.end());
// upperBound from each key must land on its successor, driving firstBelow down to depth
// kLeafDepth for every subtree.
for (std::size_t k = 0; k + 1 < keys.size(); ++k)
{
auto it = map.upperBound(keys[k]);
ASSERT_NE(it, map.end()) << "no successor for key " << k;
EXPECT_EQ(it->key(), keys[k + 1]) << "wrong successor for key " << k;
}
EXPECT_EQ(map.upperBound(keys.back()), map.end());
}
TEST_F(SHAMapTraversal, lower_bound_walks_the_whole_map_at_leaf_depth)
{
tests::TestNodeFamily f{j_};
auto keys = deepFanOutKeysAtLeafDepth();
SHAMap map{SHAMapType::FREE, f};
fillMap(map, keys);
std::sort(keys.begin(), keys.end());
// lowerBound is the lastBelow counterpart: it descends to depth kLeafDepth to find the greatest
// key below a subtree.
for (std::size_t k = 1; k < keys.size(); ++k)
{
auto it = map.lowerBound(keys[k]);
ASSERT_NE(it, map.end()) << "no predecessor for key " << k;
EXPECT_EQ(it->key(), keys[k - 1]) << "wrong predecessor for key " << k;
}
EXPECT_EQ(map.lowerBound(keys.front()), map.end());
}
TEST_F(SHAMapTraversal, bounds_agree_with_iteration_for_absent_keys_at_leaf_depth)
{
tests::TestNodeFamily f{j_};
auto keys = deepFanOutKeysAtLeafDepth();
SHAMap map{SHAMapType::FREE, f};
fillMap(map, keys);
std::sort(keys.begin(), keys.end());
// The keys fill all 16 branches of the last nibble, so an absent key must diverge from the
// shared 'a' prefix earlier than that. Diverging at increasingly deep nibbles forces
// walkTowardsKey to descend through more single-child inner nodes before it finds the empty
// branch, right up to the one just above kLeafDepth.
for (unsigned int divergeAt : {0u, 31u, 61u, 62u})
{
auto text = std::string(divergeAt, 'a') + "b";
text.append(64 - text.size(), '0');
uint256 const probe{std::string_view{text}};
auto const expectedUpper = std::upper_bound(keys.begin(), keys.end(), probe);
auto const upper = map.upperBound(probe);
if (expectedUpper == keys.end())
{
EXPECT_EQ(upper, map.end()) << "divergeAt " << divergeAt;
}
else
{
ASSERT_NE(upper, map.end()) << "divergeAt " << divergeAt;
EXPECT_EQ(upper->key(), *expectedUpper) << "divergeAt " << divergeAt;
}
auto const lowerCount = std::lower_bound(keys.begin(), keys.end(), probe) - keys.begin();
auto const lower = map.lowerBound(probe);
if (lowerCount == 0)
{
EXPECT_EQ(lower, map.end()) << "divergeAt " << divergeAt;
}
else
{
ASSERT_NE(lower, map.end()) << "divergeAt " << divergeAt;
EXPECT_EQ(lower->key(), keys[lowerCount - 1]) << "divergeAt " << divergeAt;
}
}
}
TEST_F(SHAMapTraversal, iteration_survives_deletions_at_leaf_depth)
{
tests::TestNodeFamily f{j_};
auto keys = deepFanOutKeysAtLeafDepth();
SHAMap map{SHAMapType::FREE, f};
fillMap(map, keys);
std::sort(keys.begin(), keys.end());
// Deleting every other key drops the fan-out node's branch count from 16 to 8, the same
// non-collapsing case as iteration_survives_deletions above, but reached by descending through
// a chain of single-child inner nodes down to kLeafDepth instead of a shallow one.
for (std::size_t k = 0; k < keys.size(); k += 2)
{
ASSERT_TRUE(map.delItem(keys[k]));
map.invariants();
}
std::vector<uint256> expected;
for (std::size_t k = 1; k < keys.size(); k += 2)
expected.push_back(keys[k]);
std::vector<uint256> visited;
for (auto const& item : map)
visited.push_back(item.key());
EXPECT_EQ(visited, expected);
for (std::size_t k = 0; k + 1 < expected.size(); ++k)
{
auto it = map.upperBound(expected[k]);
ASSERT_NE(it, map.end());
EXPECT_EQ(it->key(), expected[k + 1]);
}
}
class SHAMapPathProof : public ::testing::Test
{
protected:
@@ -346,4 +755,149 @@ TEST_F(SHAMapPathProof, verify_proof_path)
EXPECT_FALSE(map.verifyProofPath(rootHash, key, badPath));
}
// A legitimate proof path for two keys sharing all 63 leading nibbles is 65 elements: inner nodes
// at depths 0..63 plus the leaf at depth 64. This pins that the 65 bound is real, so the fix for
// the forged-path case below must not simply tighten the length limit.
TEST_F(SHAMapPathProof, legitimate_deep_path_is_sixty_five_elements)
{
tests::TestNodeFamily f{j_};
SHAMap map{SHAMapType::FREE, f};
map.setUnbacked();
auto const kA = uint256{std::string_view{std::string(63, 'a') + "1"}};
auto const kB = uint256{std::string_view{std::string(63, 'a') + "2"}};
for (auto const& k : {kA, kB})
{
Buffer vuc{32};
std::fill_n(vuc.data(), vuc.size(), std::uint8_t{1});
ASSERT_TRUE(map.addItem(SHAMapNodeType::TnAccountState, makeShamapitem(k, std::move(vuc))));
}
map.invariants();
auto const pathA = map.getProofPath(kA);
ASSERT_TRUE(pathA.has_value());
// NOLINTBEGIN(bugprone-unchecked-optional-access) has_value() checked above
EXPECT_EQ(pathA->size(), 65u);
EXPECT_TRUE(SHAMap::verifyProofPath(map.getHash().asUInt256(), kA, *pathA));
// NOLINTEND(bugprone-unchecked-optional-access)
auto const pathB = map.getProofPath(kB);
ASSERT_TRUE(pathB.has_value());
// NOLINTBEGIN(bugprone-unchecked-optional-access) has_value() checked above
EXPECT_EQ(pathB->size(), 65u);
EXPECT_TRUE(SHAMap::verifyProofPath(map.getHash().asUInt256(), kB, *pathB));
// NOLINTEND(bugprone-unchecked-optional-access)
}
// A forged path of 65 hash-chained inner nodes reaches depth kLeafDepth, where only the leaf
// terminating the path may sit. Such a path must be rejected.
TEST_F(SHAMapPathProof, all_inner_path_at_leaf_depth_is_rejected)
{
// An arbitrary well-formed key; the test does not care about its specific value.
constexpr uint256 kTestKey("b92891fe4ef6cee585fdc6fda1e09eb4d386363158ec3321b8123e5a772c6ca8");
// Build upwards from the deepest node so each parent's selected branch carries its child's hash
// and the hash chain validates at every level.
std::vector<Blob> path;
SHAMapHash childHash{uint256{1}};
for (auto depth = SHAMap::kLeafDepth + 1u; depth-- > 0;)
{
auto const id = SHAMapNodeID::createID(std::min(depth, SHAMap::kLeafDepth - 1u), kTestKey);
auto const branch = selectBranch(id, kTestKey);
Serializer s;
for (auto i = 0u; i < SHAMap::kBranchFactor; ++i)
s.addBitString(i == branch ? childHash.asUInt256() : uint256{});
s.add8(kWireTypeInner);
path.push_back(s.getData());
auto node = SHAMapTreeNode::makeFromWire(makeSlice(path.back()));
ASSERT_TRUE(node);
node->updateHash();
childHash = node->getHash();
}
ASSERT_EQ(path.size(), 65u);
EXPECT_FALSE(SHAMap::verifyProofPath(childHash.asUInt256(), kTestKey, path));
}
/**
* Wrap a leaf blob in a forged root inner node whose branch for `key` carries that leaf's hash.
*
* The resulting two-element path hash-chains for `key` no matter which leaf sits at the bottom,
* which is exactly the substitution a peer could attempt.
*
* @param leafBlob the wire form of the leaf to place at the bottom of the path.
* @param key the key the forged path claims to prove.
* @return the path (deepest element first) and the forged root hash, or an empty path if the leaf
* blob does not parse.
*/
static std::pair<std::vector<Blob>, uint256>
forgeRootOverLeaf(Blob const& leafBlob, uint256 const& key)
{
auto leaf = SHAMapTreeNode::makeFromWire(makeSlice(leafBlob));
if (!leaf || !leaf->isLeaf())
return {};
leaf->updateHash();
auto const branch = selectBranch(SHAMapNodeID::createID(0, key), key);
Serializer s;
for (auto i = 0u; i < SHAMap::kBranchFactor; ++i)
s.addBitString(i == branch ? leaf->getHash().asUInt256() : uint256{});
s.add8(kWireTypeInner);
auto root = SHAMapTreeNode::makeFromWire(makeSlice(s.peekData()));
if (!root)
return {};
root->updateHash();
return {std::vector<Blob>{leafBlob, s.getData()}, root->getHash().asUInt256()};
}
// The hash chain above a leaf proves nothing about which key that leaf holds, so a peer can graft a
// genuine leaf from elsewhere in the map onto a path forged for another key. Comparing the terminal
// leaf's own key against the key being proved is what rejects it.
TEST_F(SHAMapPathProof, substituted_leaf_for_other_key_is_rejected)
{
tests::TestNodeFamily f{j_};
SHAMap map{SHAMapType::FREE, f};
map.setUnbacked();
// Two arbitrary keys differing in their first nibble, so each leaf hangs off the root directly.
constexpr uint256 kKey("1c8cec8e5e9b0e5e0e0f5b3e2c9f7a1d6b4e8c2a0d7f3b9e5c1a8d4f2b6e0c93");
constexpr uint256 kOtherKey("e3f1a7d5b9c2e8f406a1d3b5c7e9f2a4d6b8c0e2f4a6d8b0c2e4f6a8d0b2c4e6");
for (auto const& k : {kKey, kOtherKey})
{
ASSERT_TRUE(map.addItem(
SHAMapNodeType::TnAccountState, makeShamapitem(k, Slice{k.data(), k.size()})));
}
map.invariants();
auto const ownPath = map.getProofPath(kKey);
auto const otherPath = map.getProofPath(kOtherKey);
ASSERT_TRUE(ownPath.has_value());
ASSERT_TRUE(otherPath.has_value());
// NOLINTBEGIN(bugprone-unchecked-optional-access) has_value() checked above
// The genuine leaf blobs, deepest element first.
auto const& ownLeaf = ownPath->front();
auto const& otherLeaf = otherPath->front();
// NOLINTEND(bugprone-unchecked-optional-access)
// Control: the forged root is accepted when the leaf below it really is kKey's leaf, so the
// rejection below can only come from the leaf key comparison.
auto const [goodPath, goodRoot] = forgeRootOverLeaf(ownLeaf, kKey);
ASSERT_EQ(goodPath.size(), 2u);
EXPECT_TRUE(SHAMap::verifyProofPath(goodRoot, kKey, goodPath));
// Same forged root, but kOtherKey's leaf substituted at the bottom: the hash chain still
// validates, yet the path does not prove anything about kKey.
auto const [badPath, badRoot] = forgeRootOverLeaf(otherLeaf, kKey);
ASSERT_EQ(badPath.size(), 2u);
EXPECT_FALSE(SHAMap::verifyProofPath(badRoot, kKey, badPath));
}
} // namespace xrpl::tests

View File

@@ -0,0 +1,190 @@
#include <xrpl/shamap/SHAMapNodeID.h>
#include <xrpl/basics/base_uint.h>
#include <xrpl/protocol/Serializer.h>
#include <xrpl/shamap/SHAMap.h>
#include <gtest/gtest.h>
#include <stdexcept>
namespace xrpl::tests {
// An arbitrary 32-byte key reused across tests below that don't care about its specific value,
// only that it is a well-formed key.
constexpr uint256 kTestKey("b92891fe4ef6cee585fdc6fda1e09eb4d386363158ec3321b8123e5a772c6ca8");
TEST(SHAMapNodeIDTest, root_is_prefix_of_every_key)
{
SHAMapNodeID const root;
EXPECT_EQ(root.getDepth(), 0u);
EXPECT_TRUE(root.isPrefixOf(uint256{}));
EXPECT_TRUE(root.isPrefixOf(kTestKey));
}
TEST(SHAMapNodeIDTest, child_id_is_prefix_of_keys_in_that_branch)
{
// Walking the branches spelled by the key's own nibbles must keep every
// intermediate ID a prefix of that key.
SHAMapNodeID id;
for (auto depth = 0u; depth < SHAMap::kLeafDepth; ++depth)
{
id = id.getChildNodeID(selectBranch(id, kTestKey));
EXPECT_EQ(id.getDepth(), depth + 1);
EXPECT_TRUE(id.isPrefixOf(kTestKey)) << "depth " << id.getDepth();
}
}
TEST(SHAMapNodeIDTest, wrong_branch_is_not_prefix_of_key)
{
SHAMapNodeID const root;
auto const correct = selectBranch(root, kTestKey);
ASSERT_EQ(correct, 0xbu);
// An ID built from the wrong branch still has a valid depth and a self-consistent mask, so
// isPrefixOf(kTestKey) below is what actually distinguishes the correct branch from the rest.
for (auto branch = 0u; branch < SHAMap::kBranchFactor; ++branch)
{
auto const child = root.getChildNodeID(branch);
EXPECT_EQ(child.getDepth(), 1u);
EXPECT_EQ(child.isPrefixOf(kTestKey), branch == correct) << "branch " << branch;
}
}
TEST(SHAMapNodeIDTest, prefix_check_is_depth_sensitive)
{
// kTestKey and kOther agree on the first two nibbles ("b9") and then diverge.
constexpr uint256 kOther("b99891fe4ef6cee585fdc6fda1e09eb4d386363158ec3321b8123e5a772c6ca8");
auto id = SHAMapNodeID{}.getChildNodeID(selectBranch(SHAMapNodeID{}, kTestKey));
EXPECT_TRUE(id.isPrefixOf(kTestKey));
EXPECT_TRUE(id.isPrefixOf(kOther)) << "shared first nibble";
id = id.getChildNodeID(selectBranch(id, kTestKey));
EXPECT_TRUE(id.isPrefixOf(kTestKey));
EXPECT_TRUE(id.isPrefixOf(kOther)) << "shared second nibble";
// Third nibble differs, so the deeper ID no longer covers kOther.
id = id.getChildNodeID(selectBranch(id, kTestKey));
EXPECT_TRUE(id.isPrefixOf(kTestKey));
EXPECT_FALSE(id.isPrefixOf(kOther));
}
TEST(SHAMapNodeIDTest, leaf_id_from_key_is_prefix_of_that_key)
{
SHAMapNodeID const leaf{SHAMap::kLeafDepth, kTestKey};
EXPECT_TRUE(leaf.isPrefixOf(kTestKey));
// At full depth the prefix is the whole key, so nothing else matches.
constexpr uint256 kOther("b92891fe4ef6cee585fdc6fda1e09eb4d386363158ec3321b8123e5a772c6ca9");
EXPECT_FALSE(leaf.isPrefixOf(kOther));
}
TEST(SHAMapNodeIDTest, create_id_masks_key_to_depth)
{
for (auto depth = 0u; depth <= SHAMap::kLeafDepth; ++depth)
{
auto const id = SHAMapNodeID::createID(depth, kTestKey);
EXPECT_EQ(id.getDepth(), depth);
EXPECT_TRUE(id.isPrefixOf(kTestKey)) << "depth " << depth;
}
}
// The guards below must hold with XRPL_ASSERT compiled out (NDEBUG), so each one
// has to be a real runtime check rather than an assert.
TEST(SHAMapNodeIDTest, child_of_leaf_depth_id_throws)
{
auto const leafDepthID = SHAMapNodeID::createID(SHAMap::kLeafDepth, kTestKey);
ASSERT_EQ(leafDepthID.getDepth(), SHAMap::kLeafDepth);
EXPECT_THROW((void)leafDepthID.getChildNodeID(0), std::logic_error);
}
TEST(SHAMapNodeIDTest, out_of_range_depth_is_clamped)
{
// A depth past kLeafDepth has no mask in depthMask's 65-entry table, so both the constructor
// and createID clamp it. createID needs its own clamp: it picks the mask while evaluating the
// constructor's argument, so the constructor's clamp cannot cover that read.
//
// Both clamps are marked UNREACHABLE, which is an assert and therefore fatal wherever asserts
// are live. Only a build with them compiled out (or routed to Antithesis's non-fatal handler)
// reaches the clamp itself, so that is the only configuration that can assert on the result.
#if defined(NDEBUG) || defined(ENABLE_VOIDSTAR)
for (auto const depth : {SHAMap::kLeafDepth + 1u, 100u, 255u, 256u, 320u})
{
auto const id = SHAMapNodeID::createID(depth, kTestKey);
// Clamped to a real depth, not the depth asked for, and not a byte-narrowed version of it:
// 256 would otherwise become 0 and name the root, 320 would become 64.
EXPECT_EQ(id.getDepth(), SHAMap::kLeafDepth) << "depth " << depth;
// id_ and depth_ still agree, so the object is usable rather than merely non-crashing.
EXPECT_TRUE(id.isPrefixOf(kTestKey)) << "depth " << depth;
EXPECT_EQ(id, SHAMapNodeID::createID(SHAMap::kLeafDepth, kTestKey)) << "depth " << depth;
// The clamp holds through the wire format too, which encodes the depth in one byte.
auto const roundTripped = deserializeSHAMapNodeID(id.getRawString());
ASSERT_TRUE(roundTripped.has_value()) << "depth " << depth;
EXPECT_EQ(roundTripped->getDepth(), SHAMap::kLeafDepth) << "depth " << depth;
}
// The constructor clamps on its own, for the paths that do not go through createID.
SHAMapNodeID const direct{SHAMap::kLeafDepth + 1u, uint256{}};
EXPECT_EQ(direct.getDepth(), SHAMap::kLeafDepth);
#else
EXPECT_DEATH(
(void)SHAMapNodeID::createID(SHAMap::kLeafDepth + 1u, kTestKey), "depth within tree");
#endif
}
TEST(SHAMapNodeIDTest, select_branch_clamps_leaf_depth)
{
// selectBranch's own precondition is depth < kLeafDepth: a depth-64 ID has no nibble left
// to select. That makes it unlike the guards above, which have a throw/return reachable
// even with XRPL_ASSERT compiled out; selectBranch has no such path, so the two build
// configurations have to be tested differently.
//
// Under ENABLE_VOIDSTAR, XRPL_ASSERT routes to Antithesis's assert_impl, which only records
// the hit and returns rather than aborting, even though NDEBUG is undefined there (voidstar
// requires a Debug build). So the assert is live in name but never fatal, the same as the
// NDEBUG case below.
auto const leafDepthID = SHAMapNodeID::createID(SHAMap::kLeafDepth, kTestKey);
#if defined(NDEBUG) || defined(ENABLE_VOIDSTAR)
// With the assert compiled out or routed to a non-fatal handler, the clamp is what stands
// between this call and reading past the end of the 32-byte key. Clamping means it reads the
// same byte, and returns the same branch, as the deepest ID that still has one: depth 63.
auto const deepestWithBranchID = SHAMapNodeID::createID(SHAMap::kLeafDepth - 1u, kTestKey);
auto const branch = selectBranch(leafDepthID, kTestKey);
EXPECT_LT(branch, SHAMap::kBranchFactor);
EXPECT_EQ(branch, selectBranch(deepestWithBranchID, kTestKey));
#else
// In a debug build the assert is live and must reject this call outright, in a forked
// process so a failure here cannot take down the rest of the suite.
EXPECT_DEATH((void)selectBranch(leafDepthID, kTestKey), "depth below leaf depth");
#endif
}
TEST(SHAMapNodeIDTest, deserialize_rejects_out_of_range_depth)
{
// getRawString() only serializes a depth already accepted by the constructor's own
// assertion, so an out-of-range depth here is built by hand instead.
auto serializeWithRawDepth = [](unsigned int depth) {
Serializer s;
s.addBitString(uint256{});
s.add8(static_cast<unsigned char>(depth));
return s.getString();
};
for (auto const depth : {65u, 100u, 255u})
EXPECT_FALSE(deserializeSHAMapNodeID(serializeWithRawDepth(depth)).has_value())
<< "depth " << depth;
// A depth-64 ID is legal, since leaves live there, but it has no children.
auto const id =
deserializeSHAMapNodeID(SHAMapNodeID{SHAMap::kLeafDepth, uint256{}}.getRawString());
ASSERT_TRUE(id.has_value());
EXPECT_THROW((void)id->getChildNodeID(0), std::logic_error);
}
} // namespace xrpl::tests

View File

@@ -75,11 +75,10 @@ getSHAMapNodeID(protocol::TMLedgerNode const& ledgerNode, SHAMapTreeNode const&
if (treeNode.isLeaf())
{
auto const key = leafKey(treeNode);
auto const expectedID = SHAMapNodeID::createID(nodeID->getDepth(), key);
SOMETIMES(
nodeID->getNodeID() != expectedID.getNodeID(),
!nodeID->isPrefixOf(key),
"xrpl::getSHAMapNodeID : legacy leaf ID inconsistent with key");
if (nodeID->getNodeID() != expectedID.getNodeID())
if (!nodeID->isPrefixOf(key))
return std::nullopt;
}

View File

@@ -19,7 +19,6 @@
#include <xrpl/basics/Slice.h>
#include <xrpl/basics/base_uint.h>
#include <xrpl/basics/contract.h>
#include <xrpl/basics/safe_cast.h>
#include <xrpl/basics/strHex.h>
#include <xrpl/beast/utility/instrumentation.h>
#include <xrpl/core/NetworkIDService.h>
@@ -805,8 +804,6 @@ transactionFormatResultImpl(Transaction::pointer tpTrans, unsigned apiVersion)
jvResult[jss::engine_result] = sToken;
jvResult[jss::engine_result_code] = tpTrans->getResult();
jvResult[jss::engine_result_message] = sHuman;
rpc::populateAugmentedSubmitFields(jvResult, tpTrans);
}
}
catch (std::exception&)
@@ -820,33 +817,6 @@ transactionFormatResultImpl(Transaction::pointer tpTrans, unsigned apiVersion)
//------------------------------------------------------------------------------
void
populateAugmentedSubmitFields(
json::Value& jvResult,
std::shared_ptr<Transaction> const& transaction)
{
auto const submitResult = transaction->getSubmitResult();
jvResult[jss::accepted] = submitResult.any();
jvResult[jss::applied] = submitResult.applied;
jvResult[jss::broadcast] = submitResult.broadcast;
jvResult[jss::queued] = submitResult.queued;
jvResult[jss::kept] = submitResult.kept;
if (auto currentLedgerState = transaction->getCurrentLedgerState())
{
jvResult[jss::account_sequence_next] =
safeCast<json::Value::UInt>(currentLedgerState->accountSeqNext);
jvResult[jss::account_sequence_available] =
safeCast<json::Value::UInt>(currentLedgerState->accountSeqAvail);
jvResult[jss::open_ledger_cost] = to_string(currentLedgerState->minFeeRequired);
jvResult[jss::validated_ledger_index] =
safeCast<json::Value::UInt>(currentLedgerState->validatedLedger);
}
}
//------------------------------------------------------------------------------
[[nodiscard]] static XRPAmount
getTxFee(Application const& app, Config const& config, json::Value tx)
{

View File

@@ -22,21 +22,6 @@ class TxQ;
namespace rpc {
/**
* Populate augmented submit fields into a JSON result.
* This helper populates the submit result flags (accepted, applied,
* broadcast, queued, kept) and current ledger state fields
* (account_sequence_next, account_sequence_available, open_ledger_cost,
* validated_ledger_index) from a Transaction pointer.
*
* @param jvResult The JSON result to populate
* @param transaction The transaction containing the submit result and state
*/
void
populateAugmentedSubmitFields(
json::Value& jvResult,
std::shared_ptr<Transaction> const& transaction);
json::Value
getCurrentNetworkFee(
Role const role,

View File

@@ -6,6 +6,7 @@
#include <xrpl/basics/Slice.h>
#include <xrpl/basics/StringUtilities.h>
#include <xrpl/basics/safe_cast.h>
#include <xrpl/basics/strHex.h>
#include <xrpl/json/json_value.h>
#include <xrpl/protocol/ErrorCodes.h>
@@ -13,6 +14,7 @@
#include <xrpl/protocol/STTx.h>
#include <xrpl/protocol/Serializer.h>
#include <xrpl/protocol/TER.h>
#include <xrpl/protocol/XRPAmount.h>
#include <xrpl/protocol/jss.h>
#include <xrpl/resource/Fees.h>
#include <xrpl/tx/apply.h>
@@ -153,7 +155,24 @@ doSubmit(rpc::JsonContext& context)
jvResult[jss::engine_result_code] = transaction->getResult();
jvResult[jss::engine_result_message] = sHuman;
rpc::populateAugmentedSubmitFields(jvResult, transaction);
auto const submitResult = transaction->getSubmitResult();
jvResult[jss::accepted] = submitResult.any();
jvResult[jss::applied] = submitResult.applied;
jvResult[jss::broadcast] = submitResult.broadcast;
jvResult[jss::queued] = submitResult.queued;
jvResult[jss::kept] = submitResult.kept;
if (auto currentLedgerState = transaction->getCurrentLedgerState())
{
jvResult[jss::account_sequence_next] =
safeCast<json::Value::UInt>(currentLedgerState->accountSeqNext);
jvResult[jss::account_sequence_available] =
safeCast<json::Value::UInt>(currentLedgerState->accountSeqAvail);
jvResult[jss::open_ledger_cost] = to_string(currentLedgerState->minFeeRequired);
jvResult[jss::validated_ledger_index] =
safeCast<json::Value::UInt>(currentLedgerState->validatedLedger);
}
}
return jvResult;