Commit Graph

14318 Commits

Author SHA1 Message Date
Nicholas Dudfield
de43ca2385 refactor(export): store multisigned tx as sfExportedTxn object in metadata
Use sfExportedTxn (OBJECT) instead of sfBlob (VL) for the multisigned
transaction in sfExportResult metadata. This renders as readable JSON
with all fields visible (Account, Signers, etc.) instead of an opaque
hex blob.

Also compute the tx hash directly via getHash(HashPrefix::transactionID)
on the STObject instead of serializing/deserializing through STTx.
2026-03-18 14:16:25 +07:00
Nicholas Dudfield
8c747a1916 feat(export): produce multisigned blob in export metadata
Export::doApply now builds the fully multisigned inner tx and stores
it as sfBlob in sfExportResult metadata. In standalone mode, the node
signs directly with its own validator keys (no consensus needed).

Key changes:
- ExportSigCollector stores actual multisign signatures, not just pubkeys
- RCLConsensus proposal attachment computes real multisign sigs over inner tx
- PeerImp harvests variable-length sig entries from proposals
- Export::doApply assembles Signers array, builds multisigned blob first,
  then uses its hash for the shadow ticket (getTransactionID includes all
  fields including Signers)
- Import skips OperationLimit and signing key checks for export callback
  path (sfTicketSequence present) — shadow ticket proves the relationship
- Full Export→XRPL→Import round-trip test: export on Xahau, submit
  multisigned blob to XRPL (with matching SignerList), build XPOP,
  import back, verify shadow ticket consumed
2026-03-18 13:54:29 +07:00
Nicholas Dudfield
cea110f29a feat: add XPOP test helper and XPOP_test suite
- src/test/jtx/xpop.h: test utilities for building XPOPs from Env ledgers
  (TestValidator, TestVLPublisher, TestXPOPContext, buildTestXPOP)
- src/test/app/XPOP_test.cpp: 4 tests (173 assertions)
  - LedgerProof construction from payment tx
  - XPOP v1 JSON structure verification
  - Merkle proof verification for multi-tx ledgers
  - Full Import round-trip: source Env payment → XPOP → dest Env Import → tesSUCCESS
2026-03-18 11:59:34 +07:00
Nicholas Dudfield
3ca056a94b feat: add XPOP test helper and XPOP_test suite
- src/test/jtx/xpop.h: test utilities for building XPOPs from Env ledgers
  (TestValidator, TestVLPublisher, buildTestXPOP)
- src/test/app/XPOP_test.cpp: 3 tests (133 assertions)
  - LedgerProof construction from payment tx
  - XPOP v1 JSON structure verification
  - Merkle proof verification for multi-tx ledgers
2026-03-18 11:41:16 +07:00
Nicholas Dudfield
705d8400db feat: add proof module for XPOP construction
New module at src/xrpld/app/proof/ with layered design:

- ProofBuilder: SHAMap merkle proof extraction (extractProofV1)
  Binary trie proof with 16-way branching, root hash verification.
- LedgerProof: proof-of-ledger (header fields + tx blob/meta + merkle proof)
  buildLedgerProof() extracts everything from a closed Ledger.
- XPOPv1: JSON format builder matching Import.cpp expectations
  buildXPOPv1() creates complete XPOP with validation signatures.

Designed for versioning: v1 JSON (current Import compat), future v2
binary proofs and account state proofs layer on the same core.
2026-03-18 11:29:29 +07:00
Nicholas Dudfield
655b751698 chore: regenerate hook/sfcodes.h for new sfields
Adds sfCancelTicketSequence (UINT32 101) and sfExportResult (OBJECT 98).
2026-03-18 11:06:04 +07:00
Nicholas Dudfield
f324081277 fix(import): verify XPOP tx hash matches shadow ticket
Shadow tickets store the exported transaction hash. Import now verifies
the XPOP's inner tx hash matches, preventing use of a different XPOP
with the same TicketSequence.
2026-03-18 10:52:19 +07:00
Nicholas Dudfield
24a284180a fix(export): address review findings from code audit
- Validate incoming export sig pubkeys against trusted UNL (PeerImp)
- Fix handleAcquiredRngSet misclassifying export sets when local map is null
- Add stale-entry cleanup (cleanupStale with 256-ledger timeout)
- Gate sig attachment on featureExport amendment (RCLConsensus)
- Fail with tefINTERNAL if ExportResult metadata can't be written
- Make export and cancel mutually exclusive (reject both in preflight)
- Remove dead ExportLimits.h include
2026-03-18 10:30:41 +07:00
Nicholas Dudfield
6f003cc983 feat(export): add SHAMap-based export sig convergence
Add deterministic export sig set convergence using CE infrastructure:
- exportSigSetHash in ExtendedPosition (flag 0x10)
- buildExportSigSet() builds SHAMap from collected sigs
- hasPendingExportSigs() gates convergence in phaseEstablish
- Parallel convergence gate alongside RNG sub-states
- handleAcquiredRngSet extended to merge export sig sets
- Tiered quorum: 80% with CE, 100% unanimity without CE

Scenario tests for all 3 feature combos:
- CE+Export, 1 node down: 4/5=80% → tesSUCCESS
- Export only, all up: 5/5=100% → tesSUCCESS
- Export only, 1 node down: 4/5≠100% → tecEXPORT_EXPIRED
2026-03-18 10:17:28 +07:00
Nicholas Dudfield
3a58020388 fix(export): tiered quorum threshold based on CE availability
Without CE: require unanimity (100% UNL) to avoid non-deterministic
quorum disagreement. With CE: use standard 80% quorum via
calculateQuorumThreshold (SHAMap convergence will ensure agreement).
Standalone/unit tests: require 1 sig.
2026-03-18 09:46:36 +07:00
Nicholas Dudfield
829441b52e fix(export): deduplicate export sigs across proposals within a round 2026-03-18 09:32:27 +07:00
Nicholas Dudfield
3a055663cc chore: add export-sig-attachment marker for projected-source 2026-03-18 09:26:33 +07:00
Nicholas Dudfield
985a194bdc feat(export): migrate to retriable ttEXPORT with proposal-based sigs
Replace the old ltEXPORTED_TXN + ttEXPORT_FINALIZE (validation-based
sigs, TxQ injection) approach with a retriable ttEXPORT that collects
validator signatures via TMProposeSet during consensus.

Added:
- terRETRY_EXPORT: keeps tx in retry set across ledger boundaries
- tecEXPORT_EXPIRED (200): LLS expiry frees sequence cleanly
- sfExportResult (OBJECT 98): signed export result in tx metadata
- ExportSigCollector: minimal thread-safe sig tracker
- Proposal sig attachment (RCLConsensus) + harvesting (PeerImp)
- exportSignatures field in TMProposeSet (ripple.proto)
- Metadata plumbing (TxMeta, ApplyViewImpl, ApplyStateTable)
- Hook xport() now emits ttEXPORT via normal emitted txn path

Removed:
- ttEXPORT_FINALIZE (type 90) pseudo-tx and Change::applyExportFinalize
- ltEXPORTED_TXN ledger entry and exportedDir/exportedTxn keylets
- ExportSignatureCollector (replaced by ExportSigCollector)
- TxQ export injection (quorum check + rawTxInsert)
- Validation-based export signing in RCLConsensus
- Application::getExportSignatureCollector

Verified on 5-node testnet: golden path (same-ledger finalization with
ExportResult in metadata), degraded path (tecEXPORT_EXPIRED on sub-quorum),
and hook xport() path (emitted ttEXPORT with shadow ticket creation).
2026-03-18 09:23:52 +07:00
Nicholas Dudfield
869f366d8a feat(export): add sfCancelTicketSequence for shadow ticket cancellation
Add sfCancelTicketSequence (UINT32 field 101) to ttEXPORT, allowing
users to cancel shadow tickets via a transaction. Both sfExportedTxn
and sfCancelTicketSequence are optional — at least one must be present.
This allows export, cancel, or both in a single transaction.

Test: create shadow ticket via export, cancel via sfCancelTicketSequence,
verify ticket is gone and owner reserve is freed.
2026-03-17 14:13:57 +07:00
Nicholas Dudfield
03936aa928 fix(export): require TicketSequence on exported transactions
Exported transactions must use TicketSequence (with Sequence=0)
because a bounced tx on the destination chain would jam sequential
sequence numbers. This is enforced in both the hook xport() API
and the Export transactor via ExportLedgerOps::validateTicketSequence().

Adds test: ttEXPORT rejects export without TicketSequence.
Updates existing test hooks to include TicketSequence in exported txns.
2026-03-17 12:30:55 +07:00
Nicholas Dudfield
6d180307ad feat(export): split Import into B2M and export callback paths
When the inner XPOP transaction has sfTicketSequence, Import now
takes the export callback path: consume the shadow ticket via
ExportLedgerOps::cancelShadowTicket() and return. No B2M balance
crediting, no account creation. Hooks fire normally and can inspect
the result via xpop_slot().

The B2M path is unchanged for non-ticket imports.

Also migrates the shadow ticket check in preclaim from the old
hookState namespace approach to keylet::shadowTicket().

Removes the unused shadowTicketNamespace constant.
2026-03-17 12:23:27 +07:00
Nicholas Dudfield
f2ca499c97 feat(export): add ltSHADOW_TICKET and xport_cancel hook API
Introduce shadow tickets for export replay protection:

- ltSHADOW_TICKET ledger entry: account-owned, keyed by
  account + ticket sequence. Fields: sfAccount, sfTicketSequence,
  sfTransactionHash, sfLedgerSequence, sfOwnerNode.

- ExportLedgerOps::createShadowTicket(): creates shadow ticket
  when exported tx has sfTicketSequence. Charges owner reserve.
  Called from both hook xport() path and Export transactor.

- ExportLedgerOps::cancelShadowTicket(): deletes shadow ticket,
  frees reserve. Used by xport_cancel hook API.

- xport_cancel(ticket_seq) hook API: allows hooks to cancel
  shadow tickets for exports that will never get a callback.

- InvariantCheck: add ltSHADOW_TICKET to valid entry types.

- Test: verify shadow ticket creation with correct fields and
  owner count bump via ttEXPORT with TicketSequence.
2026-03-17 12:13:41 +07:00
Nicholas Dudfield
bd68364f25 feat(export): add ttEXPORT user transaction and extract ExportLedgerOps
Rename the existing ttEXPORT pseudo-tx to ttEXPORT_FINALIZE (type 90)
to make room for a user-submittable ttEXPORT (type 91).

ttEXPORT allows non-hook users to submit export transactions directly,
creating the same ltEXPORTED_TXN entries that the hook xport() API
creates inline.

Extract shared logic into ExportLedgerOps.h:
- createExportedTxn(): creates ltEXPORTED_TXN, enforces directory cap
- validateNetworkID(): self-target and unconfigured guards
- validateExportAccount(): account ownership check

Both the hook API (HookAPI.cpp) and the Export transactor now call
into ExportLedgerOps, eliminating duplicated validation and ledger
mutation code.
2026-03-17 11:43:45 +07:00
Nicholas Dudfield
42a6407815 fix(export): reject exports when NETWORK_ID is unconfigured
If the node's NETWORK_ID is 0 (default/unconfigured) and the exported
transaction has no sfNetworkID field, we can't distinguish self-targeting
from legitimate cross-chain export. Reject to be safe.

Also adds exportTestConfig() helper and test for the unconfigured case.
2026-03-17 07:41:36 +07:00
Nicholas Dudfield
a387c853ab test(export): add NetworkID self-target guard test
Verify that xport() rejects exported transactions whose sfNetworkID
matches the local network. The hook builds a Payment with
NetworkID=21337 (matching the test env), and the guard correctly
returns EXPORT_FAILURE causing tecHOOK_REJECTED.

Also fix log level for the guard rejection to warn (not trace).
2026-03-17 07:31:35 +07:00
Nicholas Dudfield
9311e567d3 fix(export): reject exports targeting the local network
Explicitly forbid exported transactions whose sfNetworkID matches the
local network's ID. An exported txn re-executing on its origin chain
could cause exploits or logic issues.

The check is intentionally non-mandatory: XRPL mainnet (the primary
export target) doesn't use NetworkID, so absent = allowed.
2026-03-16 17:30:14 +07:00
Nicholas Dudfield
c26582bdf9 fix(export): move ExportLimits.h to xrpl/protocol
Both xrpld.overlay and xrpl.hook depend on xrpl.protocol, so placing
the header there avoids introducing a new xrpld.overlay > xrpl.hook
levelization dependency.
2026-03-16 15:58:58 +07:00
Nicholas Dudfield
417b999c7f chore(levelization): add xrpld.overlay > xrpl.hook dependency
New include of ExportLimits.h in PeerImp.cpp introduces this
module dependency (from feat(export) commit 89274b538).
2026-03-16 15:47:45 +07:00
Nicholas Dudfield
0205be4500 chore: add testnet scenario scripts
Entropy and export scenario scripts for local testnet validation.
2026-03-16 15:17:32 +07:00
Nicholas Dudfield
89274b5387 feat(export): wip export system limits
- max_export per hook: 4 → 2
- maxPendingExports: cap exported directory at 8 entries (tecDIR_FULL)
- clamp inbound signature processing in PeerImp to directory cap

The directory cap is the root DoS constraint: each pending export
requires every validator to sign and broadcast every round. Inbound
processing and signing throughput are transitively bounded by it.
2026-03-16 13:59:06 +07:00
Nicholas Dudfield
b65d9faf12 docs(consensus): add MERGE NOTE comments for upstream 86ef16dbeb resolution
Extends merge guidance to cover the empty-disputes bugfix (not yet in
sync-2.5.0): !disputes.empty() guard, stalled() j/clog params,
"should be rare" doc wording, debug→warn promotion, and auto-merged
testDisputes duplicate warning.
2026-03-11 10:45:04 +07:00
Nicholas Dudfield
aa1a7e5320 docs(consensus): add MERGE NOTE comments for sync-2.5.0 resolution
Inline comments at all 6 conflict points guiding the maintainer
through the expected merge conflicts when sync-2.5.0 lands:
ledgerMAX_CONSENSUS const, bootstrap params, calculateQuorumThreshold,
effectiveParms+stalled in haveConsensus, DisputedTx::stalled() j/clog
params, and testDisputes placement.
2026-03-11 10:09:02 +07:00
Nicholas Dudfield
6f0f17aad9 fix(consensus): cherry-pick upstream 86ef16dbeb empty-disputes stall fix
Cherry-pick of ripple/rippled@86ef16dbeb ("Fix: Don't flag consensus
as stalled prematurely (#5627)"). Not yet in any xahau sync branch.

Fixes false stall detection when there are no disputed transactions:
std::ranges::all_of on an empty set is vacuously true, so consensus
was incorrectly flagged as stalled. Adds !result_->disputes.empty()
guard.

Also adds diagnostic logging to DisputedTx::stalled() and the
stall detection path in haveConsensus(), and promotes the
"Need validated ledger" log from debug to warn.
2026-03-11 09:47:11 +07:00
Nicholas Dudfield
407bfa1467 feat(consensus): cherry-pick dd085e5d8 (upstream d22a5057b9) anti-stall mechanisms
Cherry-pick of ripple/rippled@d22a5057b9 / xahau dd085e5d8 ("Prevent
consensus from getting stuck in the establish phase (#5277)"), resolved
against our RNG pipeline and bootstrap fast-start changes.

Upstream adds three layered anti-stall mechanisms:
- Stateful per-dispute avalanche state machine (init→mid→late→stuck)
- Stall detection: declares consensus when all disputes individually settled
- Hard expiration: clamp(10× prev round, 15s, 120s) wall-clock safety net

Conflict resolution:
- ConsensusParms.h: kept both avalanche state machine (const members,
  avMIN_ROUNDS, avSTALLED_ROUNDS, getNeededWeight) and our bootstrap
  params (bootstrapRoundTimeSeed, bootstrapStableRoundsRequired).
  ledgerMAX_CONSENSUS left non-const for bootstrap override.
- Consensus.h: pass both stalled flag and effectiveParms to checkConsensus.
  Stall check uses original parms, bootstrap override only affects max
  consensus timeout.
- Consensus_test.cpp: kept all 12 RNG tests and new testDisputes test.
2026-03-11 09:36:38 +07:00
Nicholas Dudfield
f0dfcf6b81 fix(consensus): cap bootstrap ledgerMAX_CONSENSUS at 5s
Use an explicit 5s cap instead of dividing the default 15s.
5s is the sweet spot: long enough for peers to exchange proposals
and converge naturally, short enough to avoid wasted time.
Shorter values (e.g. 3.75s) cause nodes to hit reachedMax before
peers converge, cascading into slower subsequent rounds.
2026-03-10 14:30:20 +07:00
Nicholas Dudfield
503d2ebf98 feat(consensus): add XAHAUD_BOOTSTRAP_FAST_START for faster cold-start
Seed prevRoundTime_ to 3s instead of 15s on first round, override
idle interval to bypass closeTimeResolution (10-30s on early ledgers),
and halve ledgerMAX_CONSENSUS during bootstrap. Auto-disables after 3
consecutive rounds with UNL quorum participation.

Cuts 5-node testnet cold-start from ~28s to ~13s.

Also adds projected-source markers to TxQ, NetworkOPs, and Submit for
the transaction-submission documentation template.
2026-03-10 12:52:56 +07:00
Nicholas Dudfield
e52bc51384 refactor(consensus): extract shouldZeroEntropy() for quorum-gated entropy
Consolidate the repeated entropy fallback condition
(entropyFailed || no reveals || sub-quorum reveals) into a single
method. Fixes EntropyCount field reporting non-zero when the digest
was correctly zeroed due to sub-quorum reveals.
2026-03-10 08:42:10 +07:00
Nicholas Dudfield
91860db578 fix(consensus): require quorum-many reveals for non-zero entropy
Sub-quorum reveals (e.g. 3/4 threshold) were producing real entropy,
allowing a minority of validators to disproportionately influence the
output. Both injectEntropyPseudoTx and buildExplicitFinalProposalTxSet
now fall back to zero entropy when reveals < quorumThreshold().
2026-03-09 17:13:02 +07:00
Nicholas Dudfield
0b317a8e7a fix(consensus): skip rng pipeline during bootstrap convergence
When prevProposers < quorumThreshold, the network is still converging
and RNG can only produce zero entropy. Skip the commit/reveal pipeline
to avoid PIPELINE_TIMEOUT and conflict-wait delays that compound across
staggered startup rounds.
2026-03-09 16:27:36 +07:00
Nicholas Dudfield
dbd230b695 feat(rpc): add rng state to consensus_info response 2026-03-09 16:05:42 +07:00
Nicholas Dudfield
30cefcba85 chore: clang-format alignment fixes 2026-03-06 18:39:37 +07:00
Nicholas Dudfield
94edb5759d fix(export): gate pre-quorum on verified signature count
hasQuorum() and getExportsWithQuorum() were using raw signerMap.size()
which includes unverified signatures. TxQ could inject a ttEXPORT
pseudo-tx that then fails the stricter verified-signature check in
Change::applyExport(). Use verifiedSignatureCount() instead so TxQ
only injects when cryptographically verified quorum is actually met.

Also add cmake plumbing for enhanced logging: link date::date-tz when
available and enable BEAST_ENHANCED_LOGGING for Debug builds.
2026-03-06 18:38:54 +07:00
Nicholas Dudfield
ce57b6a3a0 fix(consensus): fix rng quorum to active UNL and demote rng log noise
Quorum fix:
- Rename expectedProposers_ → likelyParticipants_ to clarify role
- Fix commit quorum to 80% of active UNL snapshot (not shrinkable by
  recent proposer count, which was allowing 2/3 to pass as quorum)
- hasQuorumOfCommits() now uses simple threshold check only
- Add CSF test: persistent loss does not shrink quorum

Log level cleanup:
- Demote ~30 RNG/STALLDIAG per-peer/per-tick lines from info/debug to
  debug/trace across Consensus.h and RCLConsensus.cpp
- Principle: per-peer/per-tick → trace; state transitions → debug;
  milestones → info
- Reduces testnet log volume by ~93%
2026-03-06 18:36:43 +07:00
Nicholas Dudfield
fca5cad470 fix(log): catch tzdb exception in local-time formatter
date::current_zone() can throw if the timezone database is unavailable
or misconfigured (e.g. minimal container images). Fall back to UTC
formatting so enhanced logging does not make startup fatal.
2026-03-06 18:36:22 +07:00
Nicholas Dudfield
bb77c2090b consensus: gate RNG substates by amendment state 2026-03-06 14:09:06 +07:00
Nicholas Dudfield
90a94294e4 protocol: split export and consensus entropy amendments 2026-03-06 14:08:15 +07:00
Nicholas Dudfield
c2209b4472 docs(consensus): explain why seq=3 may mirror seq=2
Clarify inline that seq=3 publish can carry unchanged txSetHash while still providing extra entropySetHash delivery/fetch opportunity under packet loss or reordering.
2026-03-03 17:41:55 +07:00
Nicholas Dudfield
8fcb2ed336 docs(consensus): annotate implicit entropy injection rationale
Document why synthetic entropy pseudo-tx is canonically injected at onAccept/buildLCL and why explicit-final proposal remains experimental/default-off.
2026-03-03 17:31:04 +07:00
Nicholas Dudfield
fd1567d1ba consensus: document explicit-final tradeoffs and tighten rng diagnostics
Keep explicit final proposal as an opt-in experimental path with implicit mode as default.

Add inline rationale/TBD notes, extend stall diagnostics, and cover runtime-config + CSF txn-path behavior with tests.
2026-03-03 17:08:38 +07:00
Nicholas Dudfield
d32f34d3bf build(levelization): add fast python generator with CI parity check
Add Builds/levelization/levelization.py for fast local iteration and semantic comparison against canonical shell output via --compare-to.

Keep Builds/levelization/levelization.sh as canonical path, and update levelization workflow to fail if python output diverges from shell-generated results.

Also harden interactive-shell detection in levelization.sh for portability and document local usage in README.
2026-03-03 10:17:46 +07:00
Nicholas Dudfield
c491c5c82f refactor(consensus): reduce header fanout for faster iteration
Decouple RCLConsensus.h from Consensus.h by forward-declaring Consensus and storing Consensus<Adaptor> behind std::unique_ptr, moving thin wrappers out-of-line into RCLConsensus.cpp.

Also remove direct RCLConsensus.h include from NetworkOPs.h (forward declare), and add explicit includes in DatagramMonitor.h and ServerDefinitions.cpp to replace transitive dependencies.

Keep RNG fast-path behavior unchanged in Consensus.h; build and ripple.consensus.Consensus remain green.
2026-03-03 09:49:59 +07:00
Nicholas Dudfield
74817765ae consensus: restore full entropySet broadcast and document fanout tradeoffs 2026-03-03 08:32:09 +07:00
Nicholas Dudfield
fc23fa8535 consensus: reduce entropy-set proposal fanout
Keep entropy-set recovery path but elect a deterministic single broadcaster (lowest NodeID among tx-converged participants) instead of every proposer broadcasting entropySetHash.

This lowers steady-state proposal chatter while preserving liveness for lagging peers that need entropy-set fetch/merge.
2026-03-03 07:42:27 +07:00
Nicholas Dudfield
34c0f17b6b runtimeconfig: add rng_claim_drop_pct testing control
Expose rng_claim_drop_pct in runtime config (RPC + env) as a clamped 0-100 percentage used by RNG claim-drop testing.

Include RuntimeConfig RPC tests for round-trip and clamping behavior.
2026-03-03 07:20:32 +07:00
Nicholas Dudfield
765ad6a278 consensus: harden RNG set convergence under dropped claims
Track active RNG round sequence for fetched set validation so lagging observers can merge current-round commit sets instead of rejecting them as closed+1 out-of-round.

Refresh/re-publish commitSetHash after fetch-merge conflicts and publish entropySetHash in ConvergingReveal so peers can recover reveal sets.

Add inline tradeoff notes: extra proposal traffic is accepted to preserve consensus safety/liveness under packet loss or drop injection.
2026-03-03 07:14:46 +07:00