Compare commits

...

69 Commits

Author SHA1 Message Date
Mayukha Vadari
744355c1d9 fix: Keep GFM alert markers on their own line
proseWrap:never collapsed each `> [!NOTE]`-style blockquote onto its
body line, which breaks GitHub/GitLab's alert-callout rendering (the
marker must be alone on the blockquote's first line). Mark each one
with a prettier-ignore comment so prettier leaves it split across two
lines, and fix two more instances the review missed (CONTRIBUTING.md,
docs/build/advanced_conan.md).
2026-09-01 17:33:45 -04:00
Mayukha Vadari
9dde6c907f style: Collapse hard-wrapped Markdown prose to one line per paragraph
Set prettier's proseWrap to never for Markdown files so paragraphs are
written and formatted as a single line rather than manually wrapped at
a fixed column, matching the convention already used in AGENTS.md.
2026-09-01 17:22:18 -04:00
Vito Tumas
7863ac8cf6 fix: Add 60s buffer before closed-ended vault redemption (#8151) 2026-09-01 18:27:05 +00:00
Vito Tumas
b3b38e4416 fix: Correct fee-payer XRP delta in ValidVault for sponsored VaultWithdraw (#8141) 2026-09-01 14:43:11 +00:00
Vito Tumas
ccd5dc5e06 fix: Keep LoanBrokerDelete valid for auth-required MPT cover (#8144) 2026-09-01 14:40:53 +00:00
Vito Tumas
b2453b626e fix: Unblock VaultSet and cash-basis LoanSet at AssetsMaximum (#8143) 2026-09-01 14:06:17 +00:00
Vito Tumas
de6e5d3a94 fix: Keep VaultDeposit share count after the assetsTotal clamp (#8140) 2026-09-01 13:51:39 +00:00
Copilot
fac20a06f3 refactor: Add common helper function for injected metadata fields in RPCs (#5706)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Mayukha Vadari <mvadari@gmail.com>
Co-authored-by: Mayukha Vadari <mvadari@ripple.com>
Co-authored-by: Timur Yalymov <36795566+tyalymov@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Vito Tumas <5780819+Tapanito@users.noreply.github.com>
Co-authored-by: Ayaz Salikhov <mathbunnyru@users.noreply.github.com>
2026-08-31 17:21:25 +00:00
Ayaz Salikhov
9a7c5ea593 chore: Bump version to 3.4.0-b3 (#8132) 2026-08-27 17:26:55 +00:00
Jingchen
7281e0606a feat: Add vault invariants (#7732)
Co-authored-by: Vito Tumas <5780819+Tapanito@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-08-27 17:15:17 +00:00
Jingchen
71f5555873 feat: Remove pseudo account field filter (#8042)
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-08-27 13:52:23 +00:00
Ayaz Salikhov
3967ed6d54 build: Update release-info to get better pkg_release (#8131) 2026-08-27 13:34:38 +00:00
Vito Tumas
e0151229b6 fix: Waive unrealized-loss discount on sole-holder VaultClawback (#8119) 2026-08-27 08:15:38 +00:00
Ayaz Salikhov
b6a899583b build: Make packaging reusable (#8126) 2026-08-26 19:44:31 +00:00
Gregory Tsipenyuk
dc3bd9cf00 fix: Fix MPT/DEX Audit/Attackathon reports (Phase 1) (#7334) 2026-08-26 18:09:46 +00:00
Vito Tumas
1e8b136bfb feat: Enable LendingProtocolV1_1 amendment (#8125) 2026-08-26 17:35:54 +00:00
Vito Tumas
3c47af779c fix: Clamp Vault Deposit, Withdraw, and Clawback to assetsTotal grid (#8057)
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-08-26 17:02:05 +00:00
Ayaz Salikhov
c28d389e0e build: Refactor generate.py to make packaging_config part of config (#8115) 2026-08-26 14:04:58 +00:00
Ayaz Salikhov
d83a84510e chore: Bump version to 3.4.0-b2 (#8120) 2026-08-26 14:03:01 +00:00
Timur Yalymov
f8fba079fe fix: Refuse a pseudo-account as the vault clawback holder (#8111)
Co-authored-by: Vito Tumas <5780819+Tapanito@users.noreply.github.com>
2026-08-26 13:55:04 +00:00
Sergey Kuznetsov
42502e4263 ci: Update CI image (#8121) 2026-08-26 13:42:19 +00:00
Vito Tumas
36c165f74d fix: Prevent early loan impairment and due-date manipulation (#6557)
Co-authored-by: Ed Hennis <ed@ripple.com>
Co-authored-by: Timur Yalymov <36795566+tyalymov@users.noreply.github.com>
2026-08-26 13:38:24 +00:00
Peter Chen
f7ea645bf4 fix: AMMClawback exact LP token boundary (#7373) 2026-08-26 13:14:53 +00:00
Ayaz Salikhov
50527485d3 build: Rename release channels: unstable->rc, experimental->beta (#8116) 2026-08-26 13:13:59 +00:00
Vito Tumas
421af6db79 fix: Reject open-ended vaults at LoanBrokerSet (#8076)
Co-authored-by: Kenny Lei <3003853+kennyzlei@users.noreply.github.com>
2026-08-26 00:08:25 +00:00
Ayaz Salikhov
fee4bfc22e build: Implement packaging in Python (#8109) 2026-08-25 19:32:03 +00:00
Vito Tumas
5e3d20b3ed fix: Prevent vault clawback and withdraw overrun (#8075) 2026-08-25 17:59:37 +00:00
Ayaz Salikhov
9e2aaf6f60 build: Add rust-toolchain.toml to .envrc (#8107) 2026-08-25 17:09:03 +00:00
Vito Tumas
ec042fefee fix: Absorb Vault invariant rounding noise (#8055) 2026-08-25 17:07:58 +00:00
Ayaz Salikhov
45e4b8899d build: Update packaging images; add Python (#8106) 2026-08-25 14:34:41 +00:00
Jingchen
c5dc408596 fix: Remove explicit from std/boost hash specialisation default constructors (#8100) 2026-08-25 14:13:02 +00:00
Sergey Kuznetsov
473fe44a85 chore: Upgrade rust toolchain to 1.97.1 (#8105) 2026-08-25 13:48:26 +00:00
Bart
0fdaf69e2c chore: Bump version to 3.4.0-b1 (#8102) 2026-08-24 20:41:58 +00:00
Timur Yalymov
9d41b1bd1c fix: Exempt vault and loan broker accounts from IOU authorization (#8013)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-24 16:23:43 +00:00
Jingchen
f137d71510 test: Split Invariants_test.cpp into per-topic files (#8077) 2026-08-24 16:17:33 +00:00
Bart
8bc6e81c5f fix: Reject an inner node claimed at leaf depth in verifyProofPath (#7940)
Co-authored-by: Bart <11445373+bthomee@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-24 16:17:12 +00:00
Ed Hennis
764cbe7c29 perf: Pause online delete if there any gaps in recent ledger history (#5531)
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-08-24 14:16:03 +00:00
Timur Yalymov
520650081b fix: Remove credentials pinned to Vault, LoanBroker, and AMM pseudo-accounts (#7877)
Co-authored-by: Vito Tumas <5780819+Tapanito@users.noreply.github.com>
2026-08-24 13:06:44 +00:00
Timur Yalymov
a097ccebae fix: Tighten destination checks on vault withdrawal (#7977)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Vito Tumas <5780819+Tapanito@users.noreply.github.com>
2026-08-24 12:50:57 +00:00
Mayukha Vadari
fe4ccdf750 fix: Add assert for account_info flags (#7987) 2026-08-21 15:20:44 +00:00
Timur Yalymov
046d4dd4af fix: Reject vault deposits that move nothing from the depositor (#8014)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-21 12:30:17 +00:00
Ed Hennis
d27beef500 perf: Speed up addition time for drastically different exponents (#7825) 2026-08-20 19:40:42 +00:00
Mayukha Vadari
85512541ad refactor: Collapse transactions.macro settings into a TxSettings struct (#8001)
Co-authored-by: Timur Yalymov <36795566+tyalymov@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Vito Tumas <5780819+Tapanito@users.noreply.github.com>
Co-authored-by: Ayaz Salikhov <mathbunnyru@users.noreply.github.com>
2026-08-20 19:25:27 +00:00
Gregory Tsipenyuk
3ab5288ef2 fix: Enforce MPT balance invariants under the latest cleanup amendment (#7889) 2026-08-20 19:05:28 +00:00
Kassaking7
d0dbf9163c fix: Prevent AMM auction slots from being acquired at zero cost when trading fee is zero (#7430) 2026-08-20 19:04:04 +00:00
Ayaz Salikhov
a1478fac39 docs: Fix yum installation baseurl (#8066) 2026-08-20 16:28:43 +00:00
Timur Yalymov
cc76708563 fix: Return specific and consistent errors from vault_info (#8015)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-20 16:05:18 +00:00
Sergey Kuznetsov
422e5245e4 ci: Save cargo cache only from develop by default (#8063) 2026-08-20 16:02:26 +00:00
Shawn Xie
e3ba569187 fix: Check credential for LoanBrokerCoverWithdraw and VaultWithdraw (#7107)
Co-authored-by: Peter Chen <ychen@ripple.com>
Co-authored-by: Ayaz Salikhov <mathbunnyru@users.noreply.github.com>
2026-08-20 15:33:46 +00:00
Jingchen
959a186a0f build: Suppress MSVC linker warning LNK4099 (#8049) 2026-08-20 13:19:07 +00:00
dependabot[bot]
3b1c9e4320 chore: [DEPENDABOT] Bump cxx from 1.0.198 to 1.0.199 in /crates in the rust-dependencies group across 1 directory (#8050)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-20 13:12:14 +00:00
Ayaz Salikhov
a3a2c85c41 build: Use debian any distribution & signed (hosted) rpm repo (#8053) 2026-08-20 13:05:18 +00:00
Ayaz Salikhov
b1ac891210 ci: Do not cache cargo binaries (#8062) 2026-08-20 11:47:27 +00:00
Ayaz Salikhov
da57183e0c build: Compress the RPM payload with zstd (#8047) 2026-08-19 15:05:04 +00:00
Sergey Kuznetsov
f370289733 chore: Rust-C++ cmake and CI integration (#7034) 2026-08-19 14:30:06 +00:00
Vito Tumas
d1dc7a6ccf refactor: Extract invariant invocation into free checkInvariants runner (#7404)
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-08-19 14:10:11 +00:00
Ayaz Salikhov
5639863715 docs: Rewrite the install guide (#8048) 2026-08-19 14:02:42 +00:00
Ayaz Salikhov
1be4868875 build: Sign RPM packages (#8046) 2026-08-19 13:46:11 +00:00
Timur Yalymov
368ff1afce fix: Exempt loan default from asset freeze (#7932)
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Vito Tumas <5780819+Tapanito@users.noreply.github.com>
Co-authored-by: Ayaz Salikhov <mathbunnyru@users.noreply.github.com>
2026-08-19 13:43:40 +00:00
Vito Tumas
3adf2d40b5 fix: Reject VaultWithdraw fixed-share amounts that round to zero (#7950) 2026-08-19 13:09:38 +00:00
Ayaz Salikhov
a6983f8bf3 build: Use AlmaLinux for the RHEL packaging image (#8045) 2026-08-19 00:25:46 +00:00
Ayaz Salikhov
4113b105a5 build: Run nix macos builds in CI; deny nix store references (#8023) 2026-08-18 23:34:16 +00:00
Olek
7442ff2dec fix: Enable reserve checking on ending sponsorship (#8044) 2026-08-18 22:39:32 +00:00
Kassaking7
666e77b22c fix: Add ValidPermissionedDEX invariant track for fully consumed offer (#6736) 2026-08-18 21:08:02 +00:00
Vito Tumas
8c12de6c56 test: Split Vault_test into topical suites under src/test/app/vault/ (#8041) 2026-08-18 17:57:12 +00:00
Shawn Xie
b21fd86f6e fix: Fix assorted NFT and pDEX bugs (#7749) 2026-08-18 17:56:33 +00:00
Ayaz Salikhov
f5f47f1cf5 chore: Publish debian/rpm packages from GitHub directly (#8031) 2026-08-18 15:03:45 +00:00
Bart
ca39bff3c8 refactor: Add SHAMapNodeID::isPrefixOf (#7939)
Co-authored-by: Bart <11445373+bthomee@users.noreply.github.com>
2026-08-18 12:35:32 +00:00
Vito Tumas
dd0edc19a0 fix: Conserve funds correctly when LoanPay fee payee is below reserve (#7843) 2026-08-18 11:09:33 +00:00
369 changed files with 36885 additions and 22297 deletions

View File

@@ -1,10 +1,32 @@
codecov:
require_ci_to_pass: true
# The C++ and Rust uploads land minutes apart; without this gate Codecov
# publishes a near-zero total from whichever one arrives first.
notify:
after_n_builds: 2
wait_for_ci: true
comment:
behavior: default
layout: reach,diff,flags,tree,reach
show_carryforward_flags: false
show_carryforward_flags: true
after_n_builds: 2
# C++ and Rust coverage upload from independent workflows under the `cpp` and
# `rust` flags; carryforward keeps one language's total when only the other reran.
flag_management:
default_rules:
carryforward: true
individual_flags:
- name: cpp
carryforward: true
paths:
- include/
- src/
- name: rust
carryforward: true
paths:
- crates/
coverage:
range: "70..85"

View File

@@ -69,6 +69,7 @@ words:
- Buildx
- canonicality
- canonicalised
- cctools
- changespq
- checkme
- choco
@@ -110,6 +111,7 @@ words:
- disablerepo
- distro
- doxyfile
- dsymutil
- dxrpl
- elgamal
- enabled
@@ -168,6 +170,7 @@ words:
- LOCALGOOD
- logwstream
- Lombrozo
- lresolv
- lseq
- lsmf
- ltype
@@ -221,6 +224,7 @@ words:
- Nyffenegger
- onlatest
- ostr
- otool
- oxalica
- pargs
- partitioner
@@ -250,11 +254,15 @@ words:
- Raphson
- rcflags
- replayer
- repodata
- repomd
- rerandomize
- rerandomization
- rerandomized
- rerandomizes
- rerere
- retargeted
- retargets
- retriable
- RIPD
- ripdtop
@@ -290,6 +298,7 @@ words:
- sles
- soci
- socidb
- Sonatype
- sponsee
- sponsees
- SRPMS
@@ -309,6 +318,7 @@ words:
- summands
- superpeer
- superpeers
- Swatinem
- takergets
- takerpays
- ters
@@ -356,6 +366,7 @@ words:
- venv
- vfalco
- vinnie
- vkeylet
- wasmi
- wextra
- wptr
@@ -364,6 +375,7 @@ words:
- wthread
- xbridge
- xchain
- xcrun
- ximinez
- XMACRO
- xored
@@ -373,4 +385,6 @@ words:
- xrplf
- xxhash
- xxhasher
- zstdio
- CGNAT
- ungated

3
.envrc
View File

@@ -1,5 +1,8 @@
watch_file nix/*.nix
# Pinned Rust toolchain, read by nix/packages.nix via fromRustupToolchainFile.
watch_file rust-toolchain.toml
# The dev shell derivation includes all of conan/ (see nix/devshell.nix), so any
# change in there has to invalidate direnv's cached environment.
watch_dir conan

39
.github/actions/cargo-cache/action.yml vendored Normal file
View File

@@ -0,0 +1,39 @@
name: Use cargo artifacts cache
description: >
Cache the cargo build artifacts with rust-cache. Never caches ~/.cargo/bin:
when saving the cache, rust-cache deletes all binaries that were already
present there, which on persistent self-hosted runners wipes the tools
installed by prepare-runner. Harmless on ephemeral runners, but kept
consistent everywhere.
inputs:
workspaces:
description: "Workspaces to cache, as 'workspace -> target' lines."
required: false
default: crates
key:
description: "Additional part of the cache key."
required: false
default: ""
cache-directories:
description: "Additional non-workspace directories to cache."
required: false
default: ""
save-if:
description: >
Condition for saving the cache after the job. Defaults to save only from develop branch
required: false
default: ${{ github.ref == 'refs/heads/develop' }}
runs:
using: composite
steps:
- name: Use cargo artifacts cache
uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
with:
cache-bin: "false"
cache-directories: ${{ inputs.cache-directories }}
key: ${{ inputs.key }}
save-if: ${{ inputs.save-if }}
workspaces: ${{ inputs.workspaces }}

View File

@@ -1,44 +0,0 @@
name: Generate build version number
description: "Generate build version number."
outputs:
version:
description: "The generated build version number."
value: ${{ steps.version.outputs.version }}
runs:
using: composite
steps:
# When a tag is pushed, the version is used as-is.
- name: Generate version for tag event
if: ${{ startsWith(github.ref, 'refs/tags/') }}
shell: bash
env:
VERSION: ${{ github.ref_name }}
run: echo "VERSION=${VERSION}" >>"${GITHUB_ENV}"
# When a tag is not pushed, then the version (e.g. 1.2.3-b0) is extracted
# from the BuildInfo.cpp file and the shortened commit hash appended to it.
# We use a plus sign instead of a hyphen because Conan recipe versions do
# not support two hyphens.
- name: Generate version for non-tag event
if: ${{ !startsWith(github.ref, 'refs/tags/') }}
shell: bash
run: |
echo 'Extracting version from BuildInfo.cpp.'
VERSION="$(cat src/libxrpl/protocol/BuildInfo.cpp | grep "versionString =" | awk -F '"' '{print $2}')"
if [[ -z "${VERSION}" ]]; then
echo 'Unable to extract version from BuildInfo.cpp.'
exit 1
fi
echo 'Appending shortened commit hash to version.'
SHA='${{ github.sha }}'
VERSION="${VERSION}+${SHA:0:7}"
echo "VERSION=${VERSION}" >>"${GITHUB_ENV}"
- name: Output version
id: version
shell: bash
run: echo "version=${VERSION}" >>"${GITHUB_OUTPUT}"

44
.github/actions/release-info/action.yml vendored Normal file
View File

@@ -0,0 +1,44 @@
name: Release info
description: "Derive the version, release channel and package release number for this build."
outputs:
version:
description: "The build version number."
value: ${{ steps.version.outputs.version }}
channel:
description: "The release channel this build belongs to."
value: ${{ steps.release_info.outputs.channel }}
pkg_release:
description: "The package release number: 1 for a tag, <run number>.<commit date>git<short commit hash> otherwise."
value: ${{ steps.release_info.outputs.pkg_release }}
runs:
using: composite
steps:
# A tag names its own version. Anything else takes it from BuildInfo.cpp and
# appends the commit hash as build metadata, joined with a plus sign because a
# Conan version cannot contain two hyphens.
- name: Determine version
id: version
shell: bash
env:
IS_TAG: ${{ startsWith(github.ref, 'refs/tags/') }}
REF_NAME: ${{ github.ref_name }}
SHA: ${{ github.sha }}
run: |
if [[ "${IS_TAG}" == "true" ]]; then
version="${REF_NAME}"
else
version="$(awk -F'"' '/versionString =/ { print $2 }' src/libxrpl/protocol/BuildInfo.cpp)"
if [[ -z "${version}" ]]; then
echo "Unable to read versionString from BuildInfo.cpp." >&2
exit 1
fi
version="${version}+${SHA:0:7}"
fi
echo "version=${version}" | tee -a "${GITHUB_OUTPUT}"
- name: Determine release channel and package release
id: release_info
uses: XRPLF/actions/release-info@7cc0e4a8d9d0b838f92c48d312856b190341bbba

View File

@@ -0,0 +1,69 @@
name: Setup Nix environment
description: "Build the flake's CI environment and put its tools on PATH."
# The environment from nix/ci-env.nix, the same one the Linux CI images bake in
# (see nix/docker). Exported onto PATH rather than entered with `nix develop`:
# the composite actions below run plain `bash` and would escape a dev shell.
runs:
using: composite
steps:
- name: Build the CI environment
id: build
shell: bash
env:
# --out-link doubles as a GC root for the length of the job.
OUT_LINK: ${{ runner.temp }}/xrpld-ci-env
run: |
# --extra-experimental-features: flakes may not be on in the runner's nix.conf.
nix --extra-experimental-features "nix-command flakes" \
build .#default --out-link "${OUT_LINK}" --print-build-logs
echo "path=$(readlink -f "${OUT_LINK}")" >>"${GITHUB_OUTPUT}"
- name: Export the environment
shell: bash
env:
ENV_PATH: ${{ steps.build.outputs.path }}
run: |
echo "${ENV_PATH}/bin" >>"${GITHUB_PATH}"
# Already KEY=VALUE per line. See `darwinEnv` in nix/ci-env.nix.
ENV_FILE="${ENV_PATH}/share/xrpld-ci-env/env"
if [ -f "${ENV_FILE}" ]; then
cat "${ENV_FILE}" >>"${GITHUB_ENV}"
fi
# XrplSanity.cmake otherwise rejects a Nix compiler as one that leaked.
echo "XRPL_DEVSHELL=ci-env" >>"${GITHUB_ENV}"
# Unlike the Linux nix images, macOS needs no SSL_CERT_FILE: it has its
# own trust store, and pinning would break TLS to hosts relying on it.
# Workspace-local, so `cleanup-workspace` clears it, but not the
# `.conan2` prepare-runner hands the system toolchain: that Conan is a
# different version, and the two would migrate each other's cache.
echo "CONAN_HOME=${{ github.workspace }}/.conan2-nix" >>"${GITHUB_ENV}"
# Config, profiles and remote, exactly as the dev shell sets them up on
# entry; the `setup-conan` action is skipped for this toolchain.
- name: Setup Conan
shell: bash
run: ./conan/init.sh
# `Check tools` runs later but swallows failures; a bad export would just
# build with the system toolchain.
- name: Verify the toolchain resolves into the Nix store
shell: bash
run: |
for tool in clang clang++ cmake ninja conan; do
path="$(command -v "${tool}" || true)"
echo "${tool} -> ${path:-<not found>}"
case "${path}" in
/nix/store/*) ;;
*)
echo "::error::${tool} does not resolve into the Nix store"
exit 1
;;
esac
done

View File

@@ -4,7 +4,8 @@ updates:
directories:
- /
- .github/actions/build-deps/
- .github/actions/generate-version/
- .github/actions/cargo-cache/
- .github/actions/release-info/
- .github/actions/set-compiler-env/
- .github/actions/setup-conan/
schedule:
@@ -19,3 +20,19 @@ updates:
github-actions:
patterns:
- "*"
- package-ecosystem: cargo
directory: /crates
schedule:
interval: weekly
day: monday
time: "04:00"
timezone: Etc/GMT
commit-message:
prefix: "chore: [DEPENDABOT] "
target-branch: develop
open-pull-requests-limit: 10
groups:
rust-dependencies:
patterns:
- "*"

View File

@@ -1,34 +1,19 @@
# Levelization
Levelization is the term used to describe efforts to prevent xrpld from
having or creating cyclic dependencies.
Levelization is the term used to describe efforts to prevent xrpld from having or creating cyclic dependencies.
xrpld code is organized into directories under `src/xrpld`, `src/libxrpl` (and
`src/test`) representing modules. The modules are intended to be
organized into "tiers" or "levels" such that a module from one level can
only include code from lower levels. Additionally, a module
in one level should never include code in an `impl` or `detail` folder of any level
other than its own.
xrpld code is organized into directories under `src/xrpld`, `src/libxrpl` (and `src/test`) representing modules. The modules are intended to be organized into "tiers" or "levels" such that a module from one level can only include code from lower levels. Additionally, a module in one level should never include code in an `impl` or `detail` folder of any level other than its own.
The codebase is split into two main areas:
- **libxrpl** (`src/libxrpl`, `include/xrpl`): Reusable library modules with public interfaces
- **xrpld** (`src/xrpld`): Application-specific implementation code
Unfortunately, over time, enforcement of levelization has been
inconsistent, so the current state of the code doesn't necessarily
reflect these rules. Whenever possible, developers should refactor any
levelization violations they find (by moving files or individual
classes). At the very least, don't make things worse.
Unfortunately, over time, enforcement of levelization has been inconsistent, so the current state of the code doesn't necessarily reflect these rules. Whenever possible, developers should refactor any levelization violations they find (by moving files or individual classes). At the very least, don't make things worse.
The table below summarizes the _desired_ division of modules, based on the current
state of the xrpld code. The levels are numbered from
the bottom up with the lower level, lower numbered, more independent
modules listed first, and the higher level, higher numbered modules with
more dependencies listed later.
The table below summarizes the _desired_ division of modules, based on the current state of the xrpld code. The levels are numbered from the bottom up with the lower level, lower numbered, more independent modules listed first, and the higher level, higher numbered modules with more dependencies listed later.
**tl;dr:** The modules listed first are more independent than the modules
listed later.
**tl;dr:** The modules listed first are more independent than the modules listed later.
## libxrpl Modules (Reusable Libraries)
@@ -55,80 +40,37 @@ listed later.
## Test Modules
| Level / Tier | Module(s) |
| ------------ | -------------------------------------------------------------------------------------------------------- |
| 11 | test/jtx test/beast test/csf |
| 12 | test/unit_test |
| 13 | test/crypto test/conditions test/json test/resource test/shamap test/peerfinder test/basics test/overlay |
| 14 | test |
| 15 | test/net test/protocol test/ledger test/consensus test/core test/server test/nodestore |
| 16 | test/rpc test/app |
| Level / Tier | Module(s) |
| --- | --- |
| 11 | test/jtx test/beast test/csf |
| 12 | test/unit_test |
| 13 | test/crypto test/conditions test/json test/resource test/shamap test/peerfinder test/basics test/overlay |
| 14 | test |
| 15 | test/net test/protocol test/ledger test/consensus test/core test/server test/nodestore |
| 16 | test/rpc test/app |
(Note that `test` levelization is _much_ less important and _much_ less
strictly enforced than `xrpl`/`xrpld` levelization, other than the requirement
that `test` code should _never_ be included in `xrpl` or `xrpld` code.)
(Note that `test` levelization is _much_ less important and _much_ less strictly enforced than `xrpl`/`xrpld` levelization, other than the requirement that `test` code should _never_ be included in `xrpl` or `xrpld` code.)
## Validation
The [levelization](generate.py) script takes no parameters,
reads no environment variables, and can be run from any directory,
as long as it is in the expected location in the xrpld repo.
It can be run at any time from within a checked out repo, and will
do an analysis of all the `#include`s in
the xrpld source. The only caveat is that it runs much slower
under Windows than in Linux. It hasn't yet been tested under MacOS.
It generates many files of [results](results):
The [levelization](generate.py) script takes no parameters, reads no environment variables, and can be run from any directory, as long as it is in the expected location in the xrpld repo. It can be run at any time from within a checked out repo, and will do an analysis of all the `#include`s in the xrpld source. The only caveat is that it runs much slower under Windows than in Linux. It hasn't yet been tested under MacOS. It generates many files of [results](results):
- `rawincludes.txt`: The raw dump of the `#includes`
- `paths.txt`: A second dump grouping the source module
to the destination module, de-duped, and with frequency counts.
- `includes/`: A directory where each file represents a module and
contains a list of modules and counts that the module _includes_.
- `included_by/`: Similar to `includes/`, but the other way around. Each
file represents a module and contains a list of modules and counts
that _include_ the module.
- [`loops.txt`](results/loops.txt): A list of direct loops detected
between modules as they actually exist, as opposed to how they are
desired as described above. In a perfect repo, this file will be
empty.
This file is committed to the repo, and is used by the [levelization
Github workflow](../../workflows/reusable-check-levelization.yml) to validate
that nothing changed.
- [`ordering.txt`](results/ordering.txt): A list showing relationships
between modules where there are no loops as they actually exist, as
opposed to how they are desired as described above.
This file is committed to the repo, and is used by the [levelization
Github workflow](../../workflows/reusable-check-levelization.yml) to validate
that nothing changed.
- [`levelization.yml`](../../workflows/reusable-check-levelization.yml)
Github Actions workflow to test that levelization loops haven't
changed. Unfortunately, if changes are detected, it can't tell if
they are improvements or not, so if you have resolved any issues or
done anything else to improve levelization, run `generate.py`,
and commit the updated results.
- `paths.txt`: A second dump grouping the source module to the destination module, de-duped, and with frequency counts.
- `includes/`: A directory where each file represents a module and contains a list of modules and counts that the module _includes_.
- `included_by/`: Similar to `includes/`, but the other way around. Each file represents a module and contains a list of modules and counts that _include_ the module.
- [`loops.txt`](results/loops.txt): A list of direct loops detected between modules as they actually exist, as opposed to how they are desired as described above. In a perfect repo, this file will be empty. This file is committed to the repo, and is used by the [levelization Github workflow](../../workflows/reusable-check-levelization.yml) to validate that nothing changed.
- [`ordering.txt`](results/ordering.txt): A list showing relationships between modules where there are no loops as they actually exist, as opposed to how they are desired as described above. This file is committed to the repo, and is used by the [levelization Github workflow](../../workflows/reusable-check-levelization.yml) to validate that nothing changed.
- [`levelization.yml`](../../workflows/reusable-check-levelization.yml) Github Actions workflow to test that levelization loops haven't changed. Unfortunately, if changes are detected, it can't tell if they are improvements or not, so if you have resolved any issues or done anything else to improve levelization, run `generate.py`, and commit the updated results.
The `loops.txt` and `ordering.txt` files relate the modules
using comparison signs, which indicate the number of times each
module is included in the other.
The `loops.txt` and `ordering.txt` files relate the modules using comparison signs, which indicate the number of times each module is included in the other.
- `A > B` means that A should probably be at a higher level than B,
because B is included in A significantly more than A is included in B.
These results can be included in both `loops.txt` and `ordering.txt`.
Because `ordering.txt`only includes relationships where B is not
included in A at all, it will only include these types of results.
- `A ~= B` means that A and B are included in each other a different
number of times, but the values are so close that the script can't
definitively say that one should be above the other. These results
will only be included in `loops.txt`.
- `A == B` means that A and B include each other the same number of
times, so the script has no clue which should be higher. These results
will only be included in `loops.txt`.
- `A > B` means that A should probably be at a higher level than B, because B is included in A significantly more than A is included in B. These results can be included in both `loops.txt` and `ordering.txt`. Because `ordering.txt`only includes relationships where B is not included in A at all, it will only include these types of results.
- `A ~= B` means that A and B are included in each other a different number of times, but the values are so close that the script can't definitively say that one should be above the other. These results will only be included in `loops.txt`.
- `A == B` means that A and B include each other the same number of times, so the script has no clue which should be higher. These results will only be included in `loops.txt`.
The committed files hide the detailed values intentionally, to
prevent false alarms and merging issues, and because it's easy to
get those details locally.
The committed files hide the detailed values intentionally, to prevent false alarms and merging issues, and because it's easy to get those details locally.
1. Run `generate.py`
2. Grep the modules in `paths.txt`.
- For example, if a cycle is found `A ~= B`, simply `grep -w
A .github/scripts/levelization/results/paths.txt | grep -w B`
- For example, if a cycle is found `A ~= B`, simply `grep -w A .github/scripts/levelization/results/paths.txt | grep -w B`

View File

@@ -1,41 +1,20 @@
## Renaming ripple(d) to xrpl(d)
In the initial phases of development of the XRPL, the open source codebase was
called "rippled" and it remains with that name even today. Today, over 1000
nodes run the application, and code contributions have been submitted by
developers located around the world. The XRPL community is larger than ever.
In light of the decentralized and diversified nature of XRPL, we will rename any
references to `ripple` and `rippled` to `xrpl` and `xrpld`, when appropriate.
In the initial phases of development of the XRPL, the open source codebase was called "rippled" and it remains with that name even today. Today, over 1000 nodes run the application, and code contributions have been submitted by developers located around the world. The XRPL community is larger than ever. In light of the decentralized and diversified nature of XRPL, we will rename any references to `ripple` and `rippled` to `xrpl` and `xrpld`, when appropriate.
See [here](https://xls.xrpl.org/xls/XLS-0095-rename-rippled-to-xrpld.html) for
more information.
See [here](https://xls.xrpl.org/xls/XLS-0095-rename-rippled-to-xrpld.html) for more information.
### Scripts
To facilitate this transition, there will be multiple scripts that developers
can run on their own PRs and forks to minimize conflicts. Each script should be
run from the repository root.
To facilitate this transition, there will be multiple scripts that developers can run on their own PRs and forks to minimize conflicts. Each script should be run from the repository root.
1. `.github/scripts/rename/definitions.sh`: This script will rename all
definitions, such as include guards, from `RIPPLE_XXX` and `RIPPLED_XXX` to
`XRPL_XXX`.
2. `.github/scripts/rename/copyright.sh`: This script will remove superfluous
copyright notices.
3. `.github/scripts/rename/cmake.sh`: This script will rename all CMake files
from `RippleXXX.cmake` or `RippledXXX.cmake` to `XrplXXX.cmake`, and any
references to `ripple` and `rippled` (with or without capital letters) to
`xrpl` and `xrpld`, respectively. The name of the binary will remain as-is,
and will only be renamed to `xrpld` by a later script.
4. `.github/scripts/rename/binary.sh`: This script will rename the binary from
`rippled` to `xrpld`, and reverses the symlink so that `rippled` points to
the `xrpld` binary.
5. `.github/scripts/rename/namespace.sh`: This script will rename the C++
namespaces from `ripple` to `xrpl`.
6. `.github/scripts/rename/config.sh`: This script will rename the config from
`rippled.cfg` to `xrpld.cfg`, and updating the code accordingly. The old
filename will still be accepted.
7. `.github/scripts/rename/docs.sh`: This script will rename any lingering
references of `ripple(d)` to `xrpl(d)` in code, comments, and documentation.
1. `.github/scripts/rename/definitions.sh`: This script will rename all definitions, such as include guards, from `RIPPLE_XXX` and `RIPPLED_XXX` to `XRPL_XXX`.
2. `.github/scripts/rename/copyright.sh`: This script will remove superfluous copyright notices.
3. `.github/scripts/rename/cmake.sh`: This script will rename all CMake files from `RippleXXX.cmake` or `RippledXXX.cmake` to `XrplXXX.cmake`, and any references to `ripple` and `rippled` (with or without capital letters) to `xrpl` and `xrpld`, respectively. The name of the binary will remain as-is, and will only be renamed to `xrpld` by a later script.
4. `.github/scripts/rename/binary.sh`: This script will rename the binary from `rippled` to `xrpld`, and reverses the symlink so that `rippled` points to the `xrpld` binary.
5. `.github/scripts/rename/namespace.sh`: This script will rename the C++ namespaces from `ripple` to `xrpl`.
6. `.github/scripts/rename/config.sh`: This script will rename the config from `rippled.cfg` to `xrpld.cfg`, and updating the code accordingly. The old filename will still be accepted.
7. `.github/scripts/rename/docs.sh`: This script will rename any lingering references of `ripple(d)` to `xrpl(d)` in code, comments, and documentation.
You can run all these scripts from the repository root as follows:

View File

@@ -49,7 +49,7 @@ ${SED_COMMAND} -i -E 's@ripple/xrpld@XRPLF/rippled@g' BUILD.md
${SED_COMMAND} -i -E 's@XRPLF/xrpld@XRPLF/rippled@g' BUILD.md
${SED_COMMAND} -i -E 's@xrpld \(`xrpld`\)@xrpld@g' BUILD.md
${SED_COMMAND} -i -E 's@XRPLF/xrpld@XRPLF/rippled@g' CONTRIBUTING.md
${SED_COMMAND} -i -E 's@XRPLF/xrpld@XRPLF/rippled@g' docs/build/install.md
${SED_COMMAND} -i -E 's@XRPLF/xrpld@XRPLF/rippled@g' docs/install.md
popd
echo "Processing complete."

View File

@@ -77,8 +77,8 @@ ${SED_COMMAND} -i 's/Ripple integrators/XRPL developers/' README.md
${SED_COMMAND} -i 's/sanitizer-configuration-for-rippled/sanitizer-configuration-for-xrpld/' docs/build/sanitizers.md
${SED_COMMAND} -i 's/rippled/xrpld/g' .github/scripts/levelization/README.md
${SED_COMMAND} -i 's/rippled/xrpld/g' .github/scripts/strategy-matrix/generate.py
${SED_COMMAND} -i 's@/rippled@/xrpld@g' docs/build/install.md
${SED_COMMAND} -i 's@github.com/XRPLF/xrpld@github.com/XRPLF/rippled@g' docs/build/install.md
${SED_COMMAND} -i 's@/rippled@/xrpld@g' docs/install.md
${SED_COMMAND} -i 's@github.com/XRPLF/xrpld@github.com/XRPLF/rippled@g' docs/install.md
${SED_COMMAND} -i 's/rippled/xrpld/g' docs/Doxyfile
${SED_COMMAND} -i 's/ripple_basics/basics/' include/xrpl/basics/CountedObject.h
${SED_COMMAND} -i 's/<ripple/<xrpl/' include/xrpl/protocol/AccountID.h

View File

@@ -7,7 +7,13 @@ from pathlib import Path
THIS_DIR = Path(__file__).parent.resolve()
_BASE_CMAKE_ARGS = ["-Dtests=ON", "-Dwerr=ON", "-Dxrpld=ON", "-Dwextra=ON"]
_BASE_CMAKE_ARGS = [
"-Dtests=ON",
"-Dwerr=ON",
"-Dxrpld=ON",
"-Dwextra=ON",
"-Drust=ON",
]
# Maps sanitizer names (as used in cmake) to short config-name suffixes.
_SANITIZER_SUFFIX: dict[str, str] = {
@@ -17,6 +23,19 @@ _SANITIZER_SUFFIX: dict[str, str] = {
}
def config_name(
distro: str,
compiler: str,
build_type: str,
arch: str,
suffix: str = "",
sanitizer: str = "",
) -> str:
"""Name a config. Its artifacts are named after it, so packaging reuses this."""
parts = [s for s in [suffix, _SANITIZER_SUFFIX.get(sanitizer, "")] if s]
return "-".join([f"{distro}-{compiler}-{build_type.lower()}-{arch}", *parts])
def get_cmake_args(build_type: str, extra_args: str) -> str:
"""Get the full list of CMake arguments for a config."""
args = _BASE_CMAKE_ARGS.copy()
@@ -31,17 +50,27 @@ def get_cmake_args(build_type: str, extra_args: str) -> str:
# Every config must declare 'minimal'. Minimal configs form the reduced matrix
# built for pull requests by default; the full matrix adds the rest. Packaging
# configs declare it too, but packaging is gated in the workflow, not by it.
# built for pull requests by default; the full matrix adds the rest.
#
# Configs may also opt into 'benchmark' to smoke-run the benchmarks. Note that
# the flag applies to every entry a config expands into, so only set it on
# configs that expand to a single combination.
# Configs may also opt into 'benchmark' to smoke-run the benchmarks, or carry a
# 'package' map to be packaged as well. Note that either applies to every entry
# a config expands into, so only set them on configs that expand to a single
# combination.
@dataclasses.dataclass
class PackageConfig:
"""The 'package' map of a config whose binaries are also packaged."""
type: str # "deb" or "rpm"; has to match what the image provides
# The packaging container image: a vanilla distro image, not the nix image
# the config itself builds in.
image: str
@dataclasses.dataclass
class LinuxConfig:
"""One entry in linux.json's 'configs' or 'package_configs' arrays."""
"""One entry in a linux.json 'configs' array."""
compiler: list[str]
build_type: list[str]
@@ -51,7 +80,11 @@ class LinuxConfig:
sanitizers: list[str] = dataclasses.field(default_factory=list)
suffix: str = ""
extra_cmake_args: str = ""
image: str = "" # only used by package_configs entries
package: PackageConfig | None = None # set to also package this config
def __post_init__(self) -> None:
if isinstance(self.package, dict):
self.package = PackageConfig(**self.package)
@dataclasses.dataclass
@@ -60,22 +93,16 @@ class LinuxFile:
image_tag: str
configs: dict[str, list[LinuxConfig]] # distro → configs
package_configs: dict[str, list[LinuxConfig]] # distro → packaging configs
@classmethod
def load(cls, path: Path) -> "LinuxFile":
data = json.loads(path.read_text())
def parse(section: dict) -> dict[str, list[LinuxConfig]]:
return {
distro: [LinuxConfig(**c) for c in cfgs]
for distro, cfgs in section.items()
}
return cls(
image_tag=data["image_tag"],
configs=parse(data["configs"]),
package_configs=parse(data.get("package_configs", {})),
configs={
distro: [LinuxConfig(**c) for c in cfgs]
for distro, cfgs in data["configs"].items()
},
)
@@ -88,6 +115,9 @@ class PlatformConfig:
build_only: bool = False # if true, skip tests (e.g. macos/Windows Debug)
benchmark: bool = False # if true, smoke-run the benchmarks after testing
extra_cmake_args: str = ""
# "" is the runner's system compiler, "nix" the flake's CI environment.
# macOS only: Linux always builds in a Nix image, Windows has no Nix.
toolchain: str = ""
def __post_init__(self) -> None:
if isinstance(self.build_type, str):
@@ -137,6 +167,7 @@ class MatrixEntry:
sanitizers: str
image: str = "" # container image; empty for macOS/Windows (runs natively)
compiler: str = "" # compiler name ("gcc" or "clang"); empty for macOS/Windows
toolchain: str = "" # "nix" for the flake's CI environment; see PlatformConfig
@dataclasses.dataclass
@@ -146,7 +177,7 @@ class PackagingEntry:
xrpld_artifact_name: str
validator_keys_artifact_name: str
image: str
distro: str # e.g. "debian" or "rhel"; drives package-format-specific steps
package_type: str # "deb" or "rpm"; drives the format-specific steps
# ---------------------------------------------------------------------------
@@ -187,13 +218,9 @@ def expand_linux_matrix(linux: LinuxFile, minimal: bool) -> list[MatrixEntry]:
effective_sanitizers,
effective_archs.items(),
):
name = f"{distro}-{compiler}-{build_type.lower()}-{arch}"
suffix_parts = [
s for s in [cfg.suffix, _SANITIZER_SUFFIX.get(sanitizer, "")] if s
]
if suffix_parts:
name += "-" + "-".join(suffix_parts)
name = config_name(
distro, compiler, build_type, arch, cfg.suffix, sanitizer
)
entries.append(
MatrixEntry(
config_name=name,
@@ -213,27 +240,33 @@ def expand_linux_matrix(linux: LinuxFile, minimal: bool) -> list[MatrixEntry]:
def expand_linux_packaging(linux: LinuxFile) -> list[PackagingEntry]:
"""Generate the packaging matrix from a LinuxFile's package_configs section.
"""Generate the packaging matrix from the configs that carry a 'package' map.
Packaging uses vanilla distro images (debian:bookworm, ubi9, …) instead of
the nix-based build images, because deb/rpm tooling (debhelper, rpm-build)
is taken from the distro's archive rather than from nixpkgs. Each config
entry carries its own 'image'.
Packaging consumes the binaries that config's build job uploaded, so the
artifact names come from the same config name, and a packaged config is one
that passes -Dvalidator_keys=ON.
The artifact names must match what the build job uploads: one artifact per
binary, each named after the build config.
Packaging itself runs in vanilla distro images (debian:trixie, almalinux:10)
instead of the nix-based build images, because deb/rpm tooling (debhelper,
rpm-build) is taken from the distro's archive rather than from nixpkgs.
"""
entries = []
for distro, configs in linux.package_configs.items():
for distro, configs in linux.configs.items():
for cfg in configs:
for compiler, build_type in itertools.product(cfg.compiler, cfg.build_type):
config_name = f"{distro}-{compiler}-{build_type.lower()}-amd64"
if cfg.package is None:
continue
for compiler, build_type, arch in itertools.product(
cfg.compiler, cfg.build_type, cfg.arch
):
# The packaging workflow hardcodes an amd64 runner.
assert arch == "amd64", f"cannot package {distro} on {arch}"
name = config_name(distro, compiler, build_type, arch, cfg.suffix)
entries.append(
PackagingEntry(
xrpld_artifact_name=f"xrpld-{config_name}",
validator_keys_artifact_name=f"validator-keys-{config_name}",
image=cfg.image,
distro=distro,
xrpld_artifact_name=f"xrpld-{name}",
validator_keys_artifact_name=f"validator-keys-{name}",
image=cfg.package.image,
package_type=cfg.package.type,
)
)
@@ -253,9 +286,12 @@ def expand_platform_matrix(pf: PlatformFile, minimal: bool) -> list[MatrixEntry]
if minimal and not cfg.minimal:
continue
for build_type in cfg.build_type:
name = f"{platform_name}-{arch}-{build_type.lower()}"
if cfg.toolchain:
name += f"-{cfg.toolchain}"
entries.append(
MatrixEntry(
config_name=f"{platform_name}-{arch}-{build_type.lower()}",
config_name=name,
cmake_args=get_cmake_args(build_type, cfg.extra_cmake_args),
cmake_target="install" if is_windows else "all",
build_only=cfg.build_only,
@@ -263,6 +299,7 @@ def expand_platform_matrix(pf: PlatformFile, minimal: bool) -> list[MatrixEntry]
build_type=build_type,
architecture=Architecture(platform=pf.platform, runner=pf.runner),
sanitizers="",
toolchain=cfg.toolchain,
)
)
return entries

View File

@@ -1,5 +1,5 @@
{
"image_tag": "sha-a0074f8",
"image_tag": "sha-473fe44",
"configs": {
"ubuntu": [
{
@@ -71,7 +71,11 @@
"build_type": ["Release"],
"arch": ["amd64"],
"minimal": false,
"extra_cmake_args": "-Dvalidator_keys=ON"
"extra_cmake_args": "-Dvalidator_keys=ON",
"package": {
"type": "deb",
"image": "ghcr.io/xrplf/xrpld/packaging-debian:sha-b6a8995"
}
}
],
@@ -81,28 +85,11 @@
"build_type": ["Release"],
"arch": ["amd64"],
"minimal": false,
"extra_cmake_args": "-Dvalidator_keys=ON"
}
]
},
"package_configs": {
"debian": [
{
"compiler": ["gcc"],
"build_type": ["Release"],
"arch": ["amd64"],
"minimal": false,
"image": "ghcr.io/xrplf/xrpld/packaging-debian:sha-577d745"
}
],
"rhel": [
{
"compiler": ["gcc"],
"build_type": ["Release"],
"arch": ["amd64"],
"minimal": false,
"image": "ghcr.io/xrplf/xrpld/packaging-rhel:sha-577d745"
"extra_cmake_args": "-Dvalidator_keys=ON",
"package": {
"type": "rpm",
"image": "ghcr.io/xrplf/xrpld/packaging-rhel:sha-b6a8995"
}
}
]
}

View File

@@ -12,6 +12,19 @@
"extra_cmake_args": "-DCMAKE_POLICY_VERSION_MINIMUM=3.5",
"build_only": true,
"minimal": false
},
{
"build_type": "Release",
"extra_cmake_args": "-DCMAKE_POLICY_VERSION_MINIMUM=3.5",
"toolchain": "nix",
"minimal": false
},
{
"build_type": "Debug",
"extra_cmake_args": "-DCMAKE_POLICY_VERSION_MINIMUM=3.5",
"toolchain": "nix",
"build_only": true,
"minimal": false
}
]
}

View File

@@ -12,6 +12,7 @@ on:
- "nix/**"
- "!nix/docker/README.md"
- "!nix/devshell.nix"
- "!nix/check-tools/*.txt"
- "bin/check-tools.sh"
- "bin/default-loader-path.sh"
- "bin/install-sanitizer-libs.sh"
@@ -24,6 +25,7 @@ on:
- "nix/**"
- "!nix/docker/README.md"
- "!nix/devshell.nix"
- "!nix/check-tools/*.txt"
- "bin/check-tools.sh"
- "bin/default-loader-path.sh"
- "bin/install-sanitizer-libs.sh"
@@ -58,7 +60,7 @@ jobs:
base_image: debian:bookworm
- name: rhel
base_image: registry.access.redhat.com/ubi9/ubi:latest
uses: XRPLF/actions/.github/workflows/build-multiarch-image.yml@9e7e4e80af9e684c116b38369add8eea64451f32
uses: XRPLF/actions/.github/workflows/build-multiarch-image.yml@65d5a0bd72be4ecea95cff0673a6e0672ab5243a
with:
image_name: xrpld/nix-${{ matrix.distro.name }}
dockerfile: nix/docker/Dockerfile

View File

@@ -6,13 +6,13 @@ on:
- develop
paths:
- ".github/workflows/build-packaging-images.yml"
- "package/Dockerfile"
- "package/install-packaging-tools.sh"
- "bin/install-packaging-tools.sh"
- "package/docker/**"
pull_request:
paths:
- ".github/workflows/build-packaging-images.yml"
- "package/Dockerfile"
- "package/install-packaging-tools.sh"
- "bin/install-packaging-tools.sh"
- "package/docker/**"
workflow_dispatch:
concurrency:
@@ -33,14 +33,17 @@ jobs:
strategy:
fail-fast: false
matrix:
# Newest of each distro: these images only wrap pre-built binaries, so
# they set no floor for consumers. build_pkg.py pins the RPM dist tag.
distro:
- name: debian
base_image: debian:bookworm
base_image: debian:trixie
# AlmaLinux rather than UBI, which does not ship rpm-sign.
- name: rhel
base_image: registry.access.redhat.com/ubi9/ubi:latest
uses: XRPLF/actions/.github/workflows/build-multiarch-image.yml@9e7e4e80af9e684c116b38369add8eea64451f32
base_image: almalinux:10
uses: XRPLF/actions/.github/workflows/build-multiarch-image.yml@65d5a0bd72be4ecea95cff0673a6e0672ab5243a
with:
image_name: xrpld/packaging-${{ matrix.distro.name }}
dockerfile: package/Dockerfile
dockerfile: package/docker/Dockerfile
base_image: ${{ matrix.distro.base_image }}
push: ${{ github.event_name == 'push' }}

View File

@@ -30,7 +30,7 @@ jobs:
permissions:
contents: read
packages: write
uses: XRPLF/actions/.github/workflows/build-multiarch-image.yml@9e7e4e80af9e684c116b38369add8eea64451f32
uses: XRPLF/actions/.github/workflows/build-multiarch-image.yml@65d5a0bd72be4ecea95cff0673a6e0672ab5243a
with:
image_name: xrpld/pre-commit
dockerfile: bin/pre-commit/Dockerfile

80
.github/workflows/cargo-audit.yml vendored Normal file
View File

@@ -0,0 +1,80 @@
name: Cargo audit
on:
schedule:
# 06:32 UTC every Monday.
- cron: "32 6 * * 1"
push:
branches:
- "develop"
- "release/*"
paths:
- "crates/**/Cargo.toml"
- "crates/Cargo.lock"
- ".github/workflows/cargo-audit.yml"
pull_request:
paths:
- "crates/**/Cargo.toml"
- "crates/Cargo.lock"
- ".github/workflows/cargo-audit.yml"
workflow_dispatch:
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
defaults:
run:
shell: bash
working-directory: crates
permissions:
contents: read
jobs:
audit:
runs-on: ubuntu-latest
container: ghcr.io/xrplf/xrpld/nix-ubuntu:sha-473fe44
permissions:
contents: read
# Needed to open an issue on scheduled failures.
issues: write
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Run cargo audit
id: audit
continue-on-error: true
run: |
set -o pipefail
cargo audit | tee /tmp/cargo-audit.txt
- name: Prepare issue body
if: ${{ steps.audit.outcome != 'success' && github.event_name == 'schedule' }}
run: |
{
echo "## \`cargo audit\` found advisories"
echo
echo '```'
cat /tmp/cargo-audit.txt
echo '```'
echo
echo "---"
echo "*This issue was automatically created by the cargo-audit workflow.*"
} >/tmp/cargo-audit-issue.md
- name: Create issue
if: ${{ steps.audit.outcome != 'success' && github.event_name == 'schedule' }}
uses: XRPLF/actions/create-issue@2b8bc36af85b88bca0dd7bfac2e2dc05f94ad712
with:
title: "cargo audit found vulnerabilities"
body_file: /tmp/cargo-audit-issue.md
labels: "Bug,Security"
- name: Fail if advisories were found
if: ${{ steps.audit.outcome != 'success' }}
run: |
echo "cargo audit found advisories!"
cat /tmp/cargo-audit.txt
exit 1

View File

@@ -79,7 +79,7 @@ jobs:
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Prepare runner
uses: XRPLF/actions/prepare-runner@c00c22ada3bd6bcda48fcb0d62fbbab49fec8a0f
uses: XRPLF/actions/prepare-runner@7bf7ceca5932114abdd0d43493c3c30c5a654e13
with:
enable_ccache: false

View File

@@ -77,24 +77,28 @@ jobs:
# Keep the paths below in sync with those in `on-trigger.yml`.
.github/actions/build-deps/**
.github/actions/generate-version/**
.github/actions/release-info/**
.github/actions/setup-conan/**
.github/actions/setup-nix-env/**
.github/scripts/strategy-matrix/**
.github/workflows/reusable-build-test-config.yml
.github/workflows/reusable-build-test.yml
.github/workflows/reusable-check-autogen.yml
.github/workflows/reusable-clang-tidy.yml
.github/workflows/reusable-package.yml
.github/workflows/reusable-rust.yml
.github/workflows/reusable-strategy-matrix.yml
.github/workflows/reusable-test.yml
.github/workflows/reusable-upload-recipe.yml
.clang-tidy
.codecov.yml
bin/check-nix-store-refs.sh
bin/check-tools.sh
bin/default-loader-path.sh
cfg/**
cmake/**
conan/**
crates/**
external/**
include/**
src/**
@@ -102,6 +106,9 @@ jobs:
CMakeLists.txt
conanfile.py
conan.lock
flake.lock
flake.nix
nix/**
LICENSE.md
package/**
README.md
@@ -168,6 +175,13 @@ jobs:
secrets:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
rust:
needs: should-run
if: ${{ needs.should-run.outputs.go == 'true' }}
uses: ./.github/workflows/reusable-rust.yml
secrets:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
package:
needs: [should-run, build-test]
# Packaging consumes the debian/rhel release binaries, which are only built
@@ -211,6 +225,7 @@ jobs:
- check-rename
- clang-tidy
- build-test
- rust
- package
- upload-recipe
- notify-clio

View File

@@ -1,5 +1,9 @@
# This workflow uploads the libxrpl recipe to the Conan remote and builds
# release packages when a versioned tag is pushed.
# When a versioned tag is pushed, this workflow:
#
# - uploads the libxrpl recipe to the Conan remote
# - builds and tests the release binaries
# - builds the DEB and RPM packages
# - publishes those packages to the XRPLF package repositories
name: Tag
on:
@@ -24,7 +28,7 @@ jobs:
remote_password: ${{ secrets.NEXUS_REMOTE_PASSWORD }}
build-test:
if: ${{ github.repository == 'XRPLF/rippled' }}
if: ${{ github.repository_owner == 'XRPLF' }}
uses: ./.github/workflows/reusable-build-test.yml
strategy:
fail-fast: true
@@ -37,6 +41,12 @@ jobs:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
package:
if: ${{ github.repository == 'XRPLF/rippled' }}
if: ${{ github.repository_owner == 'XRPLF' }}
needs: build-test
uses: ./.github/workflows/reusable-package.yml
with:
publish: true
secrets:
remote_username: ${{ secrets.NEXUS_REMOTE_USERNAME }}
remote_password: ${{ secrets.NEXUS_REMOTE_PASSWORD }}
signing_key: ${{ secrets.NEXUS_PACKAGES_PRIVATE_KEY }}

View File

@@ -15,24 +15,28 @@ on:
# Keep the paths below in sync with those in `on-pr.yml`.
- ".github/actions/build-deps/**"
- ".github/actions/generate-version/**"
- ".github/actions/release-info/**"
- ".github/actions/setup-conan/**"
- ".github/actions/setup-nix-env/**"
- ".github/scripts/strategy-matrix/**"
- ".github/workflows/reusable-build-test-config.yml"
- ".github/workflows/reusable-build-test.yml"
- ".github/workflows/reusable-check-autogen.yml"
- ".github/workflows/reusable-clang-tidy.yml"
- ".github/workflows/reusable-package.yml"
- ".github/workflows/reusable-rust.yml"
- ".github/workflows/reusable-strategy-matrix.yml"
- ".github/workflows/reusable-test.yml"
- ".github/workflows/reusable-upload-recipe.yml"
- ".clang-tidy"
- ".codecov.yml"
- "bin/check-nix-store-refs.sh"
- "bin/check-tools.sh"
- "bin/default-loader-path.sh"
- "cfg/**"
- "cmake/**"
- "conan/**"
- "crates/**"
- "external/**"
- "include/**"
- "src/**"
@@ -40,6 +44,9 @@ on:
- "CMakeLists.txt"
- "conanfile.py"
- "conan.lock"
- "flake.lock"
- "flake.nix"
- "nix/**"
- "LICENSE.md"
- "package/**"
- "README.md"
@@ -96,6 +103,11 @@ jobs:
secrets:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
rust:
uses: ./.github/workflows/reusable-rust.yml
secrets:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
upload-recipe:
needs: build-test
# Only run when pushing to the develop branch.
@@ -108,3 +120,11 @@ jobs:
package:
needs: build-test
uses: ./.github/workflows/reusable-package.yml
with:
# Packages are built on every trigger; only develop pushes in XRPLF/rippled
# publish them, matching upload-recipe above.
publish: ${{ github.repository == 'XRPLF/rippled' && github.event_name == 'push' && github.ref == 'refs/heads/develop' }}
secrets:
remote_username: ${{ secrets.NEXUS_REMOTE_USERNAME }}
remote_password: ${{ secrets.NEXUS_REMOTE_PASSWORD }}
signing_key: ${{ secrets.NEXUS_PACKAGES_PRIVATE_KEY }}

View File

@@ -14,7 +14,7 @@ on:
jobs:
# Call the workflow in the XRPLF/actions repo that runs the pre-commit hooks.
run-hooks:
uses: XRPLF/actions/.github/workflows/pre-commit.yml@3ba08d6ddf114092891d48491fc2e26c3ba15552
uses: XRPLF/actions/.github/workflows/pre-commit.yml@f1952595d212e86169935135efc66294b4574131
with:
runs_on: ubuntu-latest
container: '{ "image": "ghcr.io/xrplf/xrpld/pre-commit:sha-f56b79f" }'
container: '{ "image": "ghcr.io/xrplf/xrpld/pre-commit:sha-473fe44" }'

View File

@@ -41,13 +41,13 @@ env:
jobs:
build:
runs-on: ubuntu-latest
container: ghcr.io/xrplf/xrpld/nix-ubuntu:sha-a0074f8
container: ghcr.io/xrplf/xrpld/nix-ubuntu:sha-473fe44
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Prepare runner
uses: XRPLF/actions/prepare-runner@c00c22ada3bd6bcda48fcb0d62fbbab49fec8a0f
uses: XRPLF/actions/prepare-runner@7bf7ceca5932114abdd0d43493c3c30c5a654e13
with:
enable_ccache: false

View File

@@ -69,6 +69,12 @@ on:
type: string
default: ""
toolchain:
description: 'Where the toolchain comes from ("nix" to build the flake CI environment on the runner, empty for the system one). macOS only: Linux always builds in a Nix image, and Nix has no Windows support.'
required: false
type: string
default: ""
secrets:
CODECOV_TOKEN:
description: "The Codecov token to use for uploading coverage reports."
@@ -111,6 +117,9 @@ jobs:
VOIDSTAR_ENABLED: ${{ contains(inputs.cmake_args, '-Dvoidstar=ON') }}
VALIDATOR_KEYS_ENABLED: ${{ contains(inputs.cmake_args, '-Dvalidator_keys=ON') }}
SANITIZERS_ENABLED: ${{ inputs.sanitizers != '' }}
# The binaries reusable-package.yml consumes. A private repository skips
# them except on a tag push, which is what produces its release packages.
PACKAGING_ARTIFACTS_ENABLED: ${{ github.event.repository.visibility == 'public' || startsWith(github.ref, 'refs/tags/') }}
steps:
- name: Cleanup workspace (macOS and Windows)
if: ${{ runner.os == 'macOS' || runner.os == 'Windows' }}
@@ -120,10 +129,15 @@ jobs:
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Prepare runner
uses: XRPLF/actions/prepare-runner@c00c22ada3bd6bcda48fcb0d62fbbab49fec8a0f
uses: XRPLF/actions/prepare-runner@7bf7ceca5932114abdd0d43493c3c30c5a654e13
with:
enable_ccache: ${{ inputs.ccache_enabled }}
# Before any step that uses a build tool, composite actions included.
- name: Setup Nix environment
if: ${{ inputs.toolchain == 'nix' }}
uses: ./.github/actions/setup-nix-env
- name: Set ccache log file
if: ${{ inputs.ccache_enabled && runner.debug == '1' }}
run: echo "CCACHE_LOGFILE=${{ runner.temp }}/ccache.log" >>"${GITHUB_ENV}"
@@ -148,7 +162,21 @@ jobs:
with:
compiler: ${{ inputs.compiler }}
- name: Use cargo artifacts cache
uses: ./.github/actions/cargo-cache
with:
cache-directories: ${{ env.BUILD_DIR }}/corrosion
key: ${{ inputs.config_name }}
# two workspaces here because build artifacts are located in 2 places:
# - crates/target when cargo is called directly
# - build/cargo when cargo is called by cmake
workspaces: |
crates
crates -> ${{ runner.os == 'Windows' && format('../{0}/x64/{1}/cargo', env.BUILD_DIR, inputs.build_type) || format('../{0}/cargo', env.BUILD_DIR) }}
# `setup-nix-env` already did this for the Nix toolchain.
- name: Setup Conan
if: ${{ inputs.toolchain != 'nix' }}
env:
SANITIZERS: ${{ inputs.sanitizers }}
uses: ./.github/actions/setup-conan
@@ -212,6 +240,24 @@ jobs:
--target "${CMAKE_TARGET}" \
2>&1 | tee "${GITHUB_WORKSPACE}/build.log"
# Nothing may reference the store, so whole trees are checked - the Conan
# cache included, since what it holds is what gets uploaded and reused.
- name: Check the build output for Nix store references (Nix toolchain)
if: ${{ inputs.toolchain == 'nix' }}
run: ./bin/check-nix-store-refs.sh "${BUILD_DIR}"
- name: Check the Conan cache for Nix store references (Nix toolchain)
if: ${{ inputs.toolchain == 'nix' }}
run: ./bin/check-nix-store-refs.sh "${CONAN_HOME}"
# Only what PatchNixBinary.cmake retargets: the toolchain in the Linux
# images always references the store. Same condition it uses.
- name: Check for Nix store references (Linux)
if: ${{ runner.os == 'Linux' && env.SANITIZERS_ENABLED == 'false' }}
run: |
./bin/check-nix-store-refs.sh "${BUILD_DIR}/xrpld"
./bin/check-nix-store-refs.sh "${BUILD_DIR}/xrpl_tests"
- name: Show ccache statistics
if: ${{ inputs.ccache_enabled }}
run: |
@@ -222,7 +268,7 @@ jobs:
fi
- name: Upload the binary (Linux)
if: ${{ github.event.repository.visibility == 'public' && runner.os == 'Linux' }}
if: ${{ env.PACKAGING_ARTIFACTS_ENABLED == 'true' && runner.os == 'Linux' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: xrpld-${{ inputs.config_name }}
@@ -236,7 +282,7 @@ jobs:
run: ./validator-keys --unittest
- name: Upload the validator-keys binary
if: ${{ github.event.repository.visibility == 'public' && env.VALIDATOR_KEYS_ENABLED == 'true' }}
if: ${{ env.PACKAGING_ARTIFACTS_ENABLED == 'true' && env.VALIDATOR_KEYS_ENABLED == 'true' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: validator-keys-${{ inputs.config_name }}
@@ -323,6 +369,11 @@ jobs:
LD_PRELOAD="$PRELOAD" ./xrpld --unittest --unittest-jobs "${BUILD_NPROC}" 2>&1 | tee "${GITHUB_WORKSPACE}/unittest.log"
- name: Run Rust tests
if: ${{ !inputs.build_only }}
working-directory: crates
run: cargo nextest run --workspace --all-features --locked --no-tests=warn
# Smoke-run every benchmark module with a single repetition to confirm the
# benchmarks still build and execute. This is a correctness check, not a
# performance measurement, so there is nothing to gain from repeating it
@@ -394,6 +445,7 @@ jobs:
disable_telem: true
fail_ci_if_error: true
files: ${{ env.BUILD_DIR }}/coverage.xml
flags: cpp
plugins: noop
token: ${{ secrets.CODECOV_TOKEN }}
verbose: true

View File

@@ -51,5 +51,6 @@ jobs:
config_name: ${{ matrix.config_name }}
sanitizers: ${{ matrix.sanitizers }}
compiler: ${{ matrix.compiler || '' }}
toolchain: ${{ matrix.toolchain || '' }}
secrets:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}

View File

@@ -27,14 +27,14 @@ jobs:
determine-files:
permissions:
contents: read
uses: XRPLF/actions/.github/workflows/determine-tidy-files.yml@d041ac9f1fa9f07a4ba335eb4c1c82233fb3fef6
uses: XRPLF/actions/.github/workflows/determine-tidy-files.yml@70145243b905dc3e040a61d39c00e178cfb96f71
run-clang-tidy:
name: Run clang tidy
needs: [determine-files]
if: ${{ needs.determine-files.outputs.cpp_changed_files != '' || needs.determine-files.outputs.need_full_run == 'true' }}
runs-on: ["self-hosted", "Linux", "X64", "heavy"]
container: "ghcr.io/xrplf/xrpld/nix-debian:sha-a0074f8"
container: "ghcr.io/xrplf/xrpld/nix-debian:sha-473fe44"
permissions:
contents: read
issues: write
@@ -43,7 +43,7 @@ jobs:
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Prepare runner
uses: XRPLF/actions/prepare-runner@c00c22ada3bd6bcda48fcb0d62fbbab49fec8a0f
uses: XRPLF/actions/prepare-runner@7bf7ceca5932114abdd0d43493c3c30c5a654e13
with:
enable_ccache: false
@@ -59,6 +59,12 @@ jobs:
with:
compiler: ${{ env.COMPILER }}
- name: Use cargo artifacts cache
uses: ./.github/actions/cargo-cache
with:
cache-directories: ${{ env.BUILD_DIR }}/corrosion
workspaces: crates -> ../${{ env.BUILD_DIR }}/cargo
- name: Setup Conan
uses: ./.github/actions/setup-conan
@@ -80,13 +86,13 @@ jobs:
-Dwerr=ON \
-Dxrpld=ON \
-Dverify_headers=ON \
-Drust=ON \
..
# clang-tidy needs headers generated from proto files
- name: Build libxrpl.libpb
- name: Build clang-tidy prerequisites
working-directory: ${{ env.BUILD_DIR }}
run: |
ninja -j ${{ steps.nproc.outputs.nproc }} xrpl.libpb
ninja -j ${{ steps.nproc.outputs.nproc }} tidy_prerequisites
- name: Run clang tidy
id: run_clang_tidy

View File

@@ -1,17 +1,37 @@
# Build Linux packages (DEB and RPM) from pre-built binary artifacts (xrpld and
# validator-keys). Discovers which configurations to package from linux.json
# (configs in "package_configs") and fans out one job per distro. Only
# linux/amd64 is supported; the runner is hardcoded in the job below.
# Build Linux packages from the pre-built xrpld and validator-keys artifacts:
#
# - one job per config that carries a "package" map in linux.json
# - that map names the container image and the format it builds there
# - every job ends with the image's publish_pkg.py, uploading what it built
# with 'publish: true' and doing a --dry-run otherwise
#
# Only linux/amd64 is supported; the runner is hardcoded in the job below.
name: Package
on:
workflow_call:
inputs:
pkg_release:
description: "Package release number. Increment when repackaging the same executable."
publish:
description: "Whether to publish the packages after building them."
required: false
type: boolean
default: false
nexus_url:
description: "The base URL of the Nexus instance hosting the deb and rpm repositories."
required: false
type: string
default: "1"
default: https://packages.xrplf.org
secrets:
remote_username:
description: "The username of a Nexus account with write access to the repositories."
required: false
remote_password:
description: "The password or token for that Nexus account."
required: false
signing_key:
description: "Armoured PGP private key used to sign the RPMs. Required when publishing."
required: false
defaults:
run:
@@ -41,7 +61,7 @@ jobs:
package:
needs: [generate-matrix]
if: ${{ github.event.repository.visibility == 'public' }}
if: ${{ github.event.repository.visibility == 'public' || startsWith(github.ref, 'refs/tags/') }}
strategy:
fail-fast: false
matrix: ${{ fromJson(needs.generate-matrix.outputs.matrix) }}
@@ -56,6 +76,11 @@ jobs:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Prepare runner
uses: XRPLF/actions/prepare-runner@7bf7ceca5932114abdd0d43493c3c30c5a654e13
with:
enable_ccache: false
- name: Download pre-built xrpld binary
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
@@ -71,10 +96,29 @@ jobs:
- name: Make binaries executable
run: chmod +x "${BUILD_DIR}/xrpld" "${BUILD_DIR}/validator-keys"
- name: Determine release info
id: release_info
uses: ./.github/actions/release-info
- name: Build package
env:
PKG_RELEASE: ${{ inputs.pkg_release }}
run: ./package/build_pkg.sh
PACKAGE_TYPE: ${{ matrix.package_type }}
PKG_RELEASE: ${{ steps.release_info.outputs.pkg_release }}
CHANNEL: ${{ steps.release_info.outputs.channel }}
run: |
./package/build_pkg.py \
--package-type "${PACKAGE_TYPE}" \
--build-dir "${BUILD_DIR}" \
--pkg-release "${PKG_RELEASE}" \
--channel "${CHANNEL}"
# Before the upload, so the artifact and the published package are the
# same bytes. DEBs are not signed, so the key is never set on that job.
- name: Sign RPM
if: ${{ inputs.publish && matrix.package_type == 'rpm' }}
env:
PKG_SIGNING_KEY: ${{ secrets.signing_key }}
run: ./package/sign_rpm.py --package-dir "${BUILD_DIR}"
- name: Upload package artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
@@ -85,3 +129,17 @@ jobs:
${{ env.BUILD_DIR }}/debbuild/*.ddeb
${{ env.BUILD_DIR }}/rpmbuild/RPMS/**/*.rpm
if-no-files-found: error
- name: Publish package
env:
CHANNEL: ${{ steps.release_info.outputs.channel }}
DRY_RUN_OPTION: ${{ !inputs.publish && '--dry-run' || '' }}
NEXUS_URL: ${{ inputs.nexus_url }}
NEXUS_USERNAME: ${{ inputs.publish && secrets.remote_username || '' }}
NEXUS_PASSWORD: ${{ inputs.publish && secrets.remote_password || '' }}
run: |
publish_pkg.py \
--channel "${CHANNEL}" \
--package-dir "${BUILD_DIR}" \
--nexus-url "${NEXUS_URL}" \
${DRY_RUN_OPTION}

80
.github/workflows/reusable-rust.yml vendored Normal file
View File

@@ -0,0 +1,80 @@
# Clippy, coverage and documentation for the Rust crates in crates/. Each runs
# as an independent job on a GitHub-hosted runner, but inside the same container
# image used to build the crates in the C++/Corrosion path, so the toolchain
# (and therefore the lints, coverage instrumentation and the cargo cache) matches
# what production builds use.
#
# Rust unit tests are deliberately NOT run here. They run as part of the C++
# build (reusable-build-test-config.yml), which already compiles the crates on a
# self-hosted runner, so there is no need to provision a toolchain again.
name: Rust
on:
workflow_call:
secrets:
CODECOV_TOKEN:
description: "The Codecov token to use for uploading coverage reports."
required: true
defaults:
run:
shell: bash
working-directory: crates
permissions:
contents: read
jobs:
clippy:
runs-on: ubuntu-latest
container: ghcr.io/xrplf/xrpld/nix-ubuntu:sha-473fe44
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Use cargo artifacts cache
uses: ./.github/actions/cargo-cache
- name: Run clippy
run: cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
coverage:
runs-on: ubuntu-latest
container: ghcr.io/xrplf/xrpld/nix-ubuntu:sha-473fe44
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Use cargo artifacts cache
uses: ./.github/actions/cargo-cache
- name: Generate coverage report
run: cargo llvm-cov nextest --workspace --all-features --locked --no-tests=warn --lcov --output-path lcov.info
- name: Upload coverage report
if: ${{ github.repository == 'XRPLF/rippled' }}
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
with:
disable_search: true
disable_telem: true
fail_ci_if_error: true
files: crates/lcov.info
flags: rust
plugins: noop
token: ${{ secrets.CODECOV_TOKEN }}
verbose: true
doc:
runs-on: ubuntu-latest
container: ghcr.io/xrplf/xrpld/nix-ubuntu:sha-473fe44
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Use cargo artifacts cache
uses: ./.github/actions/cargo-cache
- name: Build documentation
env:
RUSTDOCFLAGS: "-D warnings"
run: cargo doc --workspace --no-deps --all-features --locked

View File

@@ -40,7 +40,7 @@ defaults:
jobs:
upload:
runs-on: ubuntu-latest
container: ghcr.io/xrplf/xrpld/nix-ubuntu:sha-a0074f8
container: ghcr.io/xrplf/xrpld/nix-ubuntu:sha-473fe44
env:
REMOTE_NAME: ${{ inputs.remote_name }}
CONAN_LOGIN_USERNAME_XRPLF: ${{ secrets.remote_username }}
@@ -49,9 +49,14 @@ jobs:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Generate build version number
id: version
uses: ./.github/actions/generate-version
- name: Prepare runner
uses: XRPLF/actions/prepare-runner@7bf7ceca5932114abdd0d43493c3c30c5a654e13
with:
enable_ccache: false
- name: Determine release info
id: release_info
uses: ./.github/actions/release-info
- name: Set up Conan
uses: ./.github/actions/setup-conan
@@ -64,8 +69,8 @@ jobs:
- name: Upload Conan recipe (version)
run: |
conan export . --version=${{ steps.version.outputs.version }}
conan upload --confirm --check --remote="${REMOTE_NAME}" xrpl/${{ steps.version.outputs.version }}
conan export . --version=${{ steps.release_info.outputs.version }}
conan upload --confirm --check --remote="${REMOTE_NAME}" xrpl/${{ steps.release_info.outputs.version }}
# When this workflow is triggered by a push event, it will always be when merging into the
# 'develop' branch, see on-trigger.yml.
@@ -92,4 +97,4 @@ jobs:
conan upload --confirm --check --remote="${REMOTE_NAME}" xrpl/release
outputs:
ref: xrpl/${{ steps.version.outputs.version }}
ref: xrpl/${{ steps.release_info.outputs.version }}

View File

@@ -68,10 +68,15 @@ jobs:
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Prepare runner
uses: XRPLF/actions/prepare-runner@c00c22ada3bd6bcda48fcb0d62fbbab49fec8a0f
uses: XRPLF/actions/prepare-runner@7bf7ceca5932114abdd0d43493c3c30c5a654e13
with:
enable_ccache: false
# Before any step that uses a build tool, composite actions included.
- name: Setup Nix environment
if: ${{ matrix.toolchain == 'nix' }}
uses: ./.github/actions/setup-nix-env
- name: Print build environment
uses: XRPLF/actions/print-build-env@59dec886e4afb05a1724443af08baccbc045b574
@@ -87,7 +92,9 @@ jobs:
with:
compiler: ${{ matrix.compiler }}
# `setup-nix-env` already did this for the Nix toolchain.
- name: Setup Conan
if: ${{ matrix.toolchain != 'nix' }}
env:
SANITIZERS: ${{ matrix.sanitizers }}
uses: ./.github/actions/setup-conan
@@ -106,6 +113,10 @@ jobs:
log_verbosity: ${{ runner.os == 'Windows' && 'quiet' || 'verbose' }}
sanitizers: ${{ matrix.sanitizers }}
- name: Check the Conan cache for Nix store references (Nix toolchain)
if: ${{ matrix.toolchain == 'nix' }}
run: ./bin/check-nix-store-refs.sh "${CONAN_HOME}"
- name: Log into Conan remote
if: ${{ github.repository == 'XRPLF/rippled' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') }}
run: conan remote login "${CONAN_REMOTE_NAME}" "${{ secrets.NEXUS_REMOTE_USERNAME }}" --password "${{ secrets.NEXUS_REMOTE_PASSWORD }}"

3
.gitignore vendored
View File

@@ -89,3 +89,6 @@ target/
# clangd cache
/.cache
# Rust build directory
crates/target

View File

@@ -62,6 +62,15 @@ repos:
types_or: [c++, c, proto]
exclude: ^include/xrpl/protocol_autogen/(transactions|ledger_entries)/
- repo: local
hooks:
- id: cargo-fmt
name: cargo fmt
entry: cargo fmt --manifest-path crates/Cargo.toml --all
language: system
types: [rust]
pass_filenames: false # rustfmt formats the whole workspace
- repo: https://github.com/BlankSpruce/gersemi-pre-commit
rev: e98930bdc210d3387007f9252d8c1694ea7e410f # frozen: 0.27.7
hooks:
@@ -73,11 +82,27 @@ repos:
- id: prettier
args: [--end-of-line=auto]
# Scoped to package/: the rest of the repo's Python has pre-existing findings,
# so widening these is its own change.
- repo: https://github.com/astral-sh/ruff-pre-commit
rev: 7c55798a78262d14b2074abf623d8a992ebb70d4 # frozen: v0.16.2
hooks:
- id: ruff-check
args: [--fix]
files: ^package/.*\.py$
- repo: https://github.com/psf/black-pre-commit-mirror
rev: 4160603246a6b365d4a2af661c6d71b0a0f50478 # frozen: 26.5.1
hooks:
- id: black
- repo: https://github.com/pre-commit/mirrors-mypy
rev: 41e691678310dfd3833f7ab4e180ddb014310356 # frozen: v2.3.0
hooks:
- id: mypy
args: [--strict]
files: ^package/.*\.py$
- repo: https://github.com/scop/pre-commit-shfmt
rev: 05c1426671b9237fb5e1444dd63aa5731bec0dfb # frozen: v3.13.1-1
hooks:

7
.prettierrc.yaml Normal file
View File

@@ -0,0 +1,7 @@
# Collapse hard-wrapped Markdown prose to one line per paragraph instead of
# preserving manual line breaks (the prettier default). Scoped to Markdown
# only, since proseWrap also reflows YAML block scalars.
overrides:
- files: "*.md"
options:
proseWrap: never

View File

@@ -28,17 +28,15 @@ This section contains changes targeting a future version.
### Additions
- `account_tx`: Added an optional `delegate` request object to filter delegated transactions. The object requires `delegate_filter`, which must be either `actor` for transactions owned by the requested account but signed by another account, or `authorizer` for transactions signed by the requested account on behalf of another account. The optional `counter_party` account narrows the results to a specific signer/delegate for `actor` or a specific owner/delegator for `authorizer`. Malformed `delegate`, `delegate_filter`, and `counter_party` values return standard invalid field errors, and invalid account IDs return `actMalformed`.
When paginating delegate-filtered queries, a marker from a delegate-filtered query includes a `delegate` flag and is only valid for follow-up requests that also supply `delegate` (mixing marker conventions returns `invalidParams`). Because filtering is applied after the ledger scan, a page may contain fewer results than `limit` (possibly zero) while still returning a marker, so callers must continue until no marker is present.
- `account_tx`: Added an optional `delegate` request object to filter delegated transactions. The object requires `delegate_filter`, which must be either `actor` for transactions owned by the requested account but signed by another account, or `authorizer` for transactions signed by the requested account on behalf of another account. The optional `counter_party` account narrows the results to a specific signer/delegate for `actor` or a specific owner/delegator for `authorizer`. Malformed `delegate`, `delegate_filter`, and `counter_party` values return standard invalid field errors, and invalid account IDs return `actMalformed`. When paginating delegate-filtered queries, a marker from a delegate-filtered query includes a `delegate` flag and is only valid for follow-up requests that also supply `delegate` (mixing marker conventions returns `invalidParams`). Because filtering is applied after the ledger scan, a page may contain fewer results than `limit` (possibly zero) while still returning a marker, so callers must continue until no marker is present.
- `ledger_entry`, `account_objects`: The `Delegate` ledger entry now includes an optional `DestinationNode` field, which stores the index into the authorized account's owner directory. This field is present on entries created after bidirectional directory tracking was introduced and may appear in RPC responses for those entries. ([#6681](https://github.com/XRPLF/rippled/pull/6681))
- `server_definitions`: Added the following new sections to the response ([#6321](https://github.com/XRPLF/rippled/pull/6321)):
- `TRANSACTION_FORMATS`: Describes the fields and their optionality for each transaction type, including common fields shared across all transactions.
- `LEDGER_ENTRY_FORMATS`: Describes the fields and their optionality for each ledger entry type, including common fields shared across all ledger entries.
- `TRANSACTION_FLAGS`: Maps transaction type names to their supported flags and flag values.
- `LEDGER_ENTRY_FLAGS`: Maps ledger entry type names to their flags and flag values.
- `ACCOUNT_SET_FLAGS`: Maps AccountSet flag names (asf flags) to their numeric values.
- `ledger`: `nftoken_id`, `nftoken_ids`, and `offer_id` are now included in transaction metadata when transactions are expanded (`expand`, or admin-only `full`), matching the `tx`, `account_tx`, and `subscribe` (`transactions` stream) responses. ([#5706](https://github.com/XRPLF/rippled/pull/5706))
### Bugfixes
@@ -54,8 +52,12 @@ This section contains changes targeting a future version.
- `submit`: The `fail_hard` field now returns an error if the value is not a boolean. [#6529](https://github.com/XRPLF/rippled/pull/6529)
- `subscribe`: The `taker` field in the `books` array now returns `actMalformed` instead of `badIssuer` if the value is not a valid account. [#6529](https://github.com/XRPLF/rippled/pull/6529)
- Fixed a bug in `Forwarded` HTTP header parsing where the extracted IP address could be incorrect when no comma or semicolon delimiter follows the address. This could cause the server to misidentify a client's IP address when operating behind a reverse proxy. [#6529](https://github.com/XRPLF/rippled/pull/6529)
- `vault_info`: Errors now identify what the request got wrong instead of reporting every failure as the unregistered token `malformedRequest`, and the `error`, `error_code` and `error_message` fields now agree with each other. An invalid `vault_id` or `seq` returns `invalidParams`, an invalid `owner` returns `actMalformed`, and a request that mixes `vault_id` with `owner`/`seq` or supplies neither returns `invalidParams` with a message naming the accepted combinations. [#8015](https://github.com/XRPLF/rippled/pull/8015)
- `vault_info`: A well-formed all-zero `vault_id` now returns `entryNotFound` instead of being rejected as malformed, and `entryNotFound` responses now include `error_code` and `error_message`. Clients that request `ripplerpc` 3.0 or above therefore receive HTTP 400 with that error rather than HTTP 200. [#8015](https://github.com/XRPLF/rippled/pull/8015)
- `vault_info`: `vault_id` and `owner` must now be strings, matching how `ledger_entry` reads the same fields. An object or an array in either field previously produced an internal error, and a number was silently converted to its decimal text; `vault_id` now returns `invalidParams` and `owner` returns `actMalformed`. [#8015](https://github.com/XRPLF/rippled/pull/8015)
- `gateway_balances`: The `account` and `ident` fields now return an `invalidParams` error if the value is not a string, instead of an `internal` error. [#7655](https://github.com/XRPLF/rippled/pull/7655)
- `account_lines`: The `peer` field now returns an error if the value is not a string. [#7728](https://github.com/XRPLF/rippled/pull/7728)
- `ledger`: `delivered_amount` is now included in the metadata of successful `AccountDelete` transactions when transactions are expanded (`expand`, or admin-only `full`). Previously it was only added for `Payment` and `CheckCash`, which made `ledger` inconsistent with `tx` and `account_tx`. [#5706](https://github.com/XRPLF/rippled/pull/5706)
## XRP Ledger server version 3.1.0
@@ -205,9 +207,7 @@ This release contains bug fixes only and no API changes.
- Adds `AMMDelete` transaction type to delete `AMM` instance.
- Adds `sfAMMID` to `AccountRoot` to indicate that the account is `AMM`'s account. `AMMID` is used to fetch `ltAMM`.
- Adds `lsfAMMNode` `TrustLine` flag to indicate that one side of the `TrustLine` is `AMM` account.
- Adds `tfLPToken`, `tfSingleAsset`, `tfTwoAsset`, `tfOneAssetLPToken`, `tfLimitLPToken`, `tfTwoAssetIfEmpty`,
`tfWithdrawAll`, `tfOneAssetWithdrawAll` which allow a trader to specify different fields combination
for `AMMDeposit` and `AMMWithdraw` transactions.
- Adds `tfLPToken`, `tfSingleAsset`, `tfTwoAsset`, `tfOneAssetLPToken`, `tfLimitLPToken`, `tfTwoAssetIfEmpty`, `tfWithdrawAll`, `tfOneAssetWithdrawAll` which allow a trader to specify different fields combination for `AMMDeposit` and `AMMWithdraw` transactions.
- Adds new transaction result codes:
- tecUNFUNDED_AMM: insufficient balance to fund AMM. The account does not have funds for liquidity provision.
- tecAMM_BALANCE: AMM has invalid balance. Calculated balances greater than the current pool balances.
@@ -250,14 +250,11 @@ This release contains bug fixes only and no API changes.
## XRP Ledger server version 1.10.0
[Version 1.10.0](https://github.com/XRPLF/rippled/releases/tag/1.10.0)
was released on Mar 14, 2023.
[Version 1.10.0](https://github.com/XRPLF/rippled/releases/tag/1.10.0) was released on Mar 14, 2023.
### Breaking changes in 1.10
- If the `XRPFees` feature is enabled, the `fee_ref` field will be
removed from the [ledger subscription stream](https://xrpl.org/subscribe.html#ledger-stream), because it will no longer
have any meaning.
- If the `XRPFees` feature is enabled, the `fee_ref` field will be removed from the [ledger subscription stream](https://xrpl.org/subscribe.html#ledger-stream), because it will no longer have any meaning.
# Unit tests for API changes

View File

@@ -13,8 +13,7 @@ In API version 2, the following deprecated methods are no longer available: ([#4
## Modifications to JSON transaction element in API version 2
In API version 2, JSON elements for transaction output have been changed and made consistent for all methods which output transactions. ([#4775](https://github.com/XRPLF/rippled/pull/4775))
This helps to unify the JSON serialization format of transactions. ([clio#722](https://github.com/XRPLF/clio/issues/722), [#4727](https://github.com/XRPLF/rippled/issues/4727))
In API version 2, JSON elements for transaction output have been changed and made consistent for all methods which output transactions. ([#4775](https://github.com/XRPLF/rippled/pull/4775)) This helps to unify the JSON serialization format of transactions. ([clio#722](https://github.com/XRPLF/clio/issues/722), [#4727](https://github.com/XRPLF/rippled/issues/4727))
- JSON transaction element is named `tx_json`
- Binary transaction element is named `tx_blob`

211
BUILD.md
View File

@@ -1,91 +1,70 @@
| :warning: **WARNING** :warning: |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| :warning: **WARNING** :warning: |
| --- |
| These instructions assume you have a C++ development environment ready with Git, Python, Conan, CMake, and a C++ compiler. For help setting one up on Linux, macOS, or Windows, [see this guide](./docs/build/environment.md).<br><br>These instructions also assume a basic familiarity with Conan and CMake. If you are unfamiliar with Conan, you can read our [crash course](./docs/build/conan.md) or the official [Getting Started][conan-getting-started] walkthrough. |
## Minimum Requirements
For the hardware needed to run a node, see
[System Requirements](https://xrpl.org/system-requirements.html).
For the hardware needed to run a node, see [System Requirements](https://xrpl.org/system-requirements.html).
For the software needed to build xrpld, see the
[environment setup guide](./docs/build/environment.md).
For the software needed to build xrpld, see the [environment setup guide](./docs/build/environment.md).
## Operating Systems
### Linux
The Ubuntu Linux distribution has received the highest level of quality
assurance, testing, and support. We also support Red Hat and use Debian
internally.
Our Linux CI tooling is distro-independent and uses a Nix-based environment, so it should be possible to build on other Linux distributions as well, although we have not tested them.
The Ubuntu Linux distribution has received the highest level of quality assurance, testing, and support. We also support Red Hat and use Debian internally. Our Linux CI tooling is distro-independent and uses a Nix-based environment, so it should be possible to build on other Linux distributions as well, although we have not tested them.
### macOS
Many `xrpld` engineers use macOS for development.
The minimum supported version is macOS 15 (Sequoia).
CI testing is done in macOS 26 (Tahoe), but the build defaults `CMAKE_OSX_DEPLOYMENT_TARGET` to 15.
Many `xrpld` engineers use macOS for development. The minimum supported version is macOS 15 (Sequoia). CI testing is done in macOS 26 (Tahoe), but the build defaults `CMAKE_OSX_DEPLOYMENT_TARGET` to 15.
### Windows
Windows is used by some engineers for development only, and is not recommended
for production use.
Windows is used by some engineers for development only, and is not recommended for production use.
## Steps
### Branches
For the latest set of untested features, or to contribute, choose the `develop`
branch.
For the latest set of untested features, or to contribute, choose the `develop` branch.
```bash
git checkout develop
```
For a release candidate, choose the relevant release branch, e.g.
`release/3.2.x`.
For a release candidate, choose the relevant release branch, e.g. `release/3.2.x`.
```bash
git checkout release/3.2.x
```
For a stable release, choose one of the [tagged
releases](https://github.com/XRPLF/rippled/releases).
For a stable release, choose one of the [tagged releases](https://github.com/XRPLF/rippled/releases).
### Set Up Conan
Once your [development environment](./docs/build/environment.md) is ready, set
Conan up for this repository:
Once your [development environment](./docs/build/environment.md) is ready, set Conan up for this repository:
```bash
./conan/init.sh
```
That installs our [`global.conf`](./conan/global.conf), our Conan
[profiles](./conan/profiles), and the `xrplf` remote that hosts some of our
dependencies. It honours `CONAN_HOME` and never deletes an existing Conan home,
so it is safe to re-run — it only overwrites the files it manages.
That installs our [`global.conf`](./conan/global.conf), our Conan [profiles](./conan/profiles), and the `xrplf` remote that hosts some of our dependencies. It honours `CONAN_HOME` and never deletes an existing Conan home, so it is safe to re-run — it only overwrites the files it manages.
<!-- prettier-ignore -->
> [!TIP]
> In the [Nix development shell](./docs/build/nix.md#conan-configuration) this is
> already done for you: the script runs on entry.
> In the [Nix development shell](./docs/build/nix.md#conan-configuration) this is already done for you: the script runs on entry.
You can inspect the resulting profile with `conan profile show`. If it is not
suitable for your environment, create a custom profile and pass it to Conan — see
[Advanced Conan configuration](./docs/build/advanced_conan.md).
You can inspect the resulting profile with `conan profile show`. If it is not suitable for your environment, create a custom profile and pass it to Conan — see [Advanced Conan configuration](./docs/build/advanced_conan.md).
### Set Up Ccache
To speed up repeated compilations, we recommend that you install
[ccache](https://ccache.dev), a tool that wraps your compiler so that it can
cache build objects locally.
To speed up repeated compilations, we recommend that you install [ccache](https://ccache.dev), a tool that wraps your compiler so that it can cache build objects locally.
On Linux and macOS, `ccache` is included in the [Nix development shell](./docs/build/nix.md).
#### Windows
You can install it using Chocolatey, i.e. `choco install ccache`. If you already
have Ccache installed, then `choco upgrade ccache` will update it to the latest
version. However, if you see an error such as:
You can install it using Chocolatey, i.e. `choco install ccache`. If you already have Ccache installed, then `choco upgrade ccache` will update it to the latest version. However, if you see an error such as:
```
terminate called after throwing an instance of 'std::bad_alloc'
@@ -93,8 +72,7 @@ terminate called after throwing an instance of 'std::bad_alloc'
C:\Program Files\Microsoft Visual Studio\2022\Community\MSBuild\Microsoft\VC\v170\Microsoft.CppCommon.targets(617,5): error MSB6006: "cl.exe" exited with code 3.
```
then please install a specific version of Ccache that we know works, via: `choco
install ccache --version 4.11.3 --allow-downgrade`.
then please install a specific version of Ccache that we know works, via: `choco install ccache --version 4.11.3 --allow-downgrade`.
### Build and Test
@@ -105,14 +83,9 @@ install ccache --version 4.11.3 --allow-downgrade`.
cd .build
```
You can use any directory name. Conan treats your working directory as an
install folder and generates files with implementation details.
You don't need to worry about these files, but make sure to change
your working directory to your build directory before calling Conan.
You can use any directory name. Conan treats your working directory as an install folder and generates files with implementation details. You don't need to worry about these files, but make sure to change your working directory to your build directory before calling Conan.
**Note:** You can specify a directory for the installation files by adding
the `install-folder` or `-if` option to every `conan install` command
in the next step.
**Note:** You can specify a directory for the installation files by adding the `install-folder` or `-if` option to every `conan install` command in the next step.
2. Use conan to generate CMake files for every configuration you want to build:
@@ -123,25 +96,15 @@ install ccache --version 4.11.3 --allow-downgrade`.
To build Debug, in the next step, be sure to set `-DCMAKE_BUILD_TYPE=Debug`
For a single-configuration generator, e.g. `Unix Makefiles` or `Ninja`,
you only need to run this command once.
For a multi-configuration generator, e.g. `Visual Studio`, you may want to
run it more than once.
For a single-configuration generator, e.g. `Unix Makefiles` or `Ninja`, you only need to run this command once. For a multi-configuration generator, e.g. `Visual Studio`, you may want to run it more than once.
Each of these commands should also have a different `build_type` setting.
A second command with the same `build_type` setting will overwrite the files
generated by the first. You can pass the build type on the command line with
`--settings build_type=$BUILD_TYPE` or in the profile itself,
under the section `[settings]` with the key `build_type`.
Each of these commands should also have a different `build_type` setting. A second command with the same `build_type` setting will overwrite the files generated by the first. You can pass the build type on the command line with `--settings build_type=$BUILD_TYPE` or in the profile itself, under the section `[settings]` with the key `build_type`.
3. Configure CMake and pass the toolchain file generated by Conan, located at
`$OUTPUT_FOLDER/build/generators/conan_toolchain.cmake`.
3. Configure CMake and pass the toolchain file generated by Conan, located at `$OUTPUT_FOLDER/build/generators/conan_toolchain.cmake`.
Single-config generators:
Pass the CMake variable [`CMAKE_BUILD_TYPE`][build_type]
and make sure it matches the one of the `build_type` settings
you chose in the previous step.
Pass the CMake variable [`CMAKE_BUILD_TYPE`][build_type] and make sure it matches the one of the `build_type` settings you chose in the previous step.
For example, to build Debug, in the next command, replace "Release" with "Debug"
@@ -159,9 +122,7 @@ install ccache --version 4.11.3 --allow-downgrade`.
4. Build `xrpld`.
For a single-configuration generator, it will build whatever configuration
you passed for `CMAKE_BUILD_TYPE`. For a multi-configuration generator, you
must pass the option `--config` to select the build configuration.
For a single-configuration generator, it will build whatever configuration you passed for `CMAKE_BUILD_TYPE`. For a multi-configuration generator, you must pass the option `--config` to select the build configuration.
Single-config generators:
@@ -176,8 +137,7 @@ install ccache --version 4.11.3 --allow-downgrade`.
cmake --build . --config Debug --parallel N
```
Replace the `--parallel` parameter N with the desired number of parallel jobs. A common starting point is half of the number of available CPU
cores.
Replace the `--parallel` parameter N with the desired number of parallel jobs. A common starting point is half of the number of available CPU cores.
5. Test xrpld.
@@ -194,28 +154,20 @@ install ccache --version 4.11.3 --allow-downgrade`.
./Debug/xrpld --unittest --unittest-jobs N
```
Replace the `--unittest-jobs` parameter N with the desired unit tests
concurrency. Recommended setting is half of the number of available CPU
cores.
Replace the `--unittest-jobs` parameter N with the desired unit tests concurrency. Recommended setting is half of the number of available CPU cores.
The location of `xrpld` binary in your build directory depends on your
CMake generator. Pass `--help` to see the rest of the command line options.
The location of `xrpld` binary in your build directory depends on your CMake generator. Pass `--help` to see the rest of the command line options.
## Code generation
The protocol wrapper classes in `include/xrpl/protocol_autogen/` are generated
from macro definition files in `include/xrpl/protocol/detail/`. If you modify
the macro files (e.g. `transactions.macro`, `ledger_entries.macro`) or the
generation scripts/templates in `cmake/scripts/codegen/`, you need to regenerate the
files:
The protocol wrapper classes in `include/xrpl/protocol_autogen/` are generated from macro definition files in `include/xrpl/protocol/detail/`. If you modify the macro files (e.g. `transactions.macro`, `ledger_entries.macro`) or the generation scripts/templates in `cmake/scripts/codegen/`, you need to regenerate the files:
```
cmake --build . --target setup_code_gen # create venv and install dependencies (once)
cmake --build . --target code_gen # regenerate code
```
The same targets are also available as a standalone project, which does not
need the dependencies to be configured first:
The same targets are also available as a standalone project, which does not need the dependencies to be configured first:
```
cmake -S cmake/codegen -B build/codegen
@@ -223,15 +175,11 @@ cmake --build build/codegen --target setup_code_gen
cmake --build build/codegen --target code_gen
```
The regenerated files should be committed alongside your changes. CI verifies
that they are up-to-date.
The regenerated files should be committed alongside your changes. CI verifies that they are up-to-date.
## Coverage report
The coverage report is intended for developers using compilers GCC
or Clang (including Apple Clang). It is generated by the build target `coverage`,
which is only enabled when the `coverage` option is set, e.g. with
`--options coverage=True` in `conan` or `-Dcoverage=ON` variable in `cmake`
The coverage report is intended for developers using compilers GCC or Clang (including Apple Clang). It is generated by the build target `coverage`, which is only enabled when the `coverage` option is set, e.g. with `--options coverage=True` in `conan` or `-Dcoverage=ON` variable in `cmake`
Prerequisites for the coverage report:
@@ -239,34 +187,19 @@ Prerequisites for the coverage report:
- `gcov` for GCC or `llvm-cov` for Clang, usually installed with the compiler
- `Debug` build type
<!-- prettier-ignore -->
> [!NOTE]
> Clang coverage is not available in the [Nix development shell](./docs/build/nix.md#building-xrpld-in-the-nix-shell):
> its `clang` shells do not ship `llvm-cov`. Use a `gcc` shell instead (`.#gcc`,
> or `.#gcc-plain` on Linux), which provides a `gcov` matching its compiler.
> Clang coverage is not available in the [Nix development shell](./docs/build/nix.md#building-xrpld-in-the-nix-shell): its `clang` shells do not ship `llvm-cov`. Use a `gcc` shell instead (`.#gcc`, or `.#gcc-plain` on Linux), which provides a `gcov` matching its compiler.
A coverage report is created when the following steps are completed, in order:
1. `xrpld` binary built with instrumentation data, enabled by the `coverage`
option mentioned above
1. `xrpld` binary built with instrumentation data, enabled by the `coverage` option mentioned above
2. completed one or more run of the unit tests, which populates coverage capture data
3. completed run of the `gcovr` tool (which internally invokes either `gcov` or `llvm-cov`)
to assemble both instrumentation data and the coverage capture data into a coverage report
3. completed run of the `gcovr` tool (which internally invokes either `gcov` or `llvm-cov`) to assemble both instrumentation data and the coverage capture data into a coverage report
The last step of the above is automated into a single target `coverage`. The instrumented
`xrpld` binary can also be used for regular development or testing work, at
the cost of extra disk space utilization and a small performance hit
(to store coverage capture data). Since `xrpld` binary is simply a dependency of the
coverage report target, it is possible to re-run the `coverage` target without
rebuilding the `xrpld` binary. Note, running of the unit tests before the `coverage`
target is left to the developer. Each such run will append to the coverage data
collected in the build directory.
The last step of the above is automated into a single target `coverage`. The instrumented `xrpld` binary can also be used for regular development or testing work, at the cost of extra disk space utilization and a small performance hit (to store coverage capture data). Since `xrpld` binary is simply a dependency of the coverage report target, it is possible to re-run the `coverage` target without rebuilding the `xrpld` binary. Note, running of the unit tests before the `coverage` target is left to the developer. Each such run will append to the coverage data collected in the build directory.
The default coverage report format is `html-details`, but the user
can override it to any of the formats listed in `Builds/CMake/CodeCoverage.cmake`
by setting the `coverage_format` variable in `cmake`. It is also possible
to generate more than one format at a time by setting the `coverage_extra_args`
variable in `cmake`. The specific command line used to run the `gcovr` tool will be
displayed if the `CODE_COVERAGE_VERBOSE` variable is set.
The default coverage report format is `html-details`, but the user can override it to any of the formats listed in `Builds/CMake/CodeCoverage.cmake` by setting the `coverage_format` variable in `cmake`. It is also possible to generate more than one format at a time by setting the `coverage_extra_args` variable in `cmake`. The specific command line used to run the `gcovr` tool will be displayed if the `CODE_COVERAGE_VERBOSE` variable is set.
Example use with some cmake variables set:
@@ -277,16 +210,14 @@ cmake -DCMAKE_BUILD_TYPE=Debug -Dcoverage=ON -Dxrpld=ON -Dtests=ON -Dcoverage_te
cmake --build . --target coverage
```
After the `coverage` target is completed, the generated coverage report will be
stored inside the build directory, as either of:
After the `coverage` target is completed, the generated coverage report will be stored inside the build directory, as either of:
- file named `coverage.`_extension_, with a suitable extension for the report format, or
- directory named `coverage`, with the `index.html` and other files inside, for the `html-details` or `html-nested` report formats.
## Sanitizers
To build dependencies and xrpld with sanitizer instrumentation, set the
`SANITIZERS` environment variable when running `conan install` and use the `sanitizers` profile:
To build dependencies and xrpld with sanitizer instrumentation, set the `SANITIZERS` environment variable when running `conan install` and use the `sanitizers` profile:
```bash
export SANITIZERS=address,undefinedbehavior
@@ -300,40 +231,41 @@ See [Sanitizers docs](./docs/build/sanitizers.md) for more details.
## Options
| Option | Default Value | Description |
| ---------------- | ------------- | ----------------------------------------------------------------------------- |
| `assert` | OFF | Force enabling assertions. |
| `coverage` | OFF | Prepare the coverage report. |
| `tests` | OFF | Build tests. |
| `unity` | OFF | Configure a unity build. |
| `verify_headers` | ON | Make the `verify-headers` target available to compile each header on its own. |
| `xrpld` | OFF | Build the xrpld application, and not just the libxrpl library. |
| `werr` | OFF | Treat compilation warnings as errors |
| `wextra` | OFF | Enable additional compilation warnings |
| Option | Default Value | Description |
| --- | --- | --- |
| `assert` | OFF | Force enabling assertions. |
| `coverage` | OFF | Prepare the coverage report. |
| `rust` | OFF | Build the Rust crates and the C++ code that depends on them. |
| `tests` | OFF | Build tests. |
| `unity` | OFF | Configure a unity build. |
| `verify_headers` | ON | Make the `verify-headers` target available to compile each header on its own. |
| `xrpld` | OFF | Build the xrpld application, and not just the libxrpl library. |
| `werr` | OFF | Treat compilation warnings as errors |
| `wextra` | OFF | Enable additional compilation warnings |
[Unity builds][unity-build] may be faster for the first build (at the cost of much more
memory) since they concatenate sources into fewer translation units. Non-unity
builds may be faster for incremental builds, and can be helpful for detecting
`#include` omissions.
[Unity builds][unity-build] may be faster for the first build (at the cost of much more memory) since they concatenate sources into fewer translation units. Non-unity builds may be faster for incremental builds, and can be helpful for detecting `#include` omissions.
### Rust crates
The Rust crates in `crates/` are only part of the build when `rust` is ON. With `-Drust=OFF` (the default) the `crates` directory is not added to the build, no cxxbridge bindings are generated, and the C++ tests that exercise the Rust interop are not compiled — so no Rust toolchain is needed. CI builds always pass `-Drust=ON`.
With `-Drust=ON` you need one extra dependency: a Rust toolchain (`cargo`, `rustc`) matching the channel pinned in [`rust-toolchain.toml`](./rust-toolchain.toml), which compiles the crates and generates the cxxbridge bindings. It is provided by the [Nix development shell](./docs/build/nix.md), so `-Drust=ON` works there without any extra setup; otherwise install it as described in [Rust](./docs/build/environment.md#rust).
The crates also have their own Rust unit tests. Those are run with `cargo` and need only the Rust toolchain, independently of CMake and of the `rust` option (CI runs them with `cargo nextest`):
```bash
cargo test --manifest-path crates/Cargo.toml --workspace
```
### Verifying headers
The regular build only compiles `.cpp` files, so a header is only ever checked
through whatever translation unit happens to include it. A header that forgets
an `#include` is not caught as long as every `.cpp` that uses it includes its
missing dependency first. The `verify_headers` option (ON by default) adds a
`verify-headers` target that compiles every header on its own, which fails if a
header is not self-contained:
The regular build only compiles `.cpp` files, so a header is only ever checked through whatever translation unit happens to include it. A header that forgets an `#include` is not caught as long as every `.cpp` that uses it includes its missing dependency first. The `verify_headers` option (ON by default) adds a `verify-headers` target that compiles every header on its own, which fails if a header is not self-contained:
```bash
cmake --build . --target verify-headers
```
The per-header objects are excluded from the `all` target, so a normal build
never compiles them; they are built only through `verify-headers`. The generated
translation units do appear in `compile_commands.json`, so clang-tidy (and
clangd and IDEs) can lint each header on its own. Pass `-Dverify_headers=OFF` to
omit them entirely.
The per-header objects are excluded from the `all` target, so a normal build never compiles them; they are built only through `verify-headers`. The generated translation units do appear in `compile_commands.json`, so clang-tidy (and clangd and IDEs) can lint each header on its own. Pass `-Dverify_headers=OFF` to omit them entirely.
## Troubleshooting
@@ -360,20 +292,15 @@ After any updates or changes to dependencies, you may need to do the following:
4. [Regenerate lockfile](./docs/build/advanced_conan.md#conan-lockfile).
5. Re-run [conan install](#build-and-test).
If you are using the Nix development shell, whether prebuilt Conan binaries apply
depends on your platform — see
[Prebuilt packages](./docs/build/nix.md#prebuilt-packages).
If you are using the Nix development shell, whether prebuilt Conan binaries apply depends on your platform — see [Prebuilt packages](./docs/build/nix.md#prebuilt-packages).
#### ERROR: Package not resolved
If you're seeing an error like `ERROR: Package 'snappy/1.1.10' not resolved: Unable to find 'snappy/1.1.10#968fef506ff261592ec30c574d4a7809%1756234314.246' in remotes.`,
please [set Conan up](#set-up-conan) so the `xrplf` remote is configured, or re-run `conan export` for [patched recipes](./docs/build/advanced_conan.md#patched-recipes).
If you're seeing an error like `ERROR: Package 'snappy/1.1.10' not resolved: Unable to find 'snappy/1.1.10#968fef506ff261592ec30c574d4a7809%1756234314.246' in remotes.`, please [set Conan up](#set-up-conan) so the `xrplf` remote is configured, or re-run `conan export` for [patched recipes](./docs/build/advanced_conan.md#patched-recipes).
### `protobuf/port_def.inc` file not found
If `cmake --build .` results in an error due to a missing a protobuf file, then
you might have generated CMake files for a different `build_type` than the
`CMAKE_BUILD_TYPE` you passed to Conan.
If `cmake --build .` results in an error due to a missing a protobuf file, then you might have generated CMake files for a different `build_type` than the `CMAKE_BUILD_TYPE` you passed to Conan.
```
/xrpld/.build/pb-xrpl.libpb/xrpl/proto/xrpl.pb.h:10:10: fatal error: 'google/protobuf/port_def.inc' file not found

View File

@@ -158,7 +158,13 @@ if(coverage)
include(XrplCov)
endif()
add_custom_target(tidy_prerequisites)
if(rust)
add_subdirectory(crates)
endif()
include(XrplCore)
include(XrplProtocolAutogen)
include(XrplInstall)
include(XrplValidatorKeys)

View File

@@ -1,31 +1,19 @@
The XRP Ledger has many and diverse stakeholders, and everyone deserves
a chance to contribute meaningful changes to the code that runs the
XRPL.
The XRP Ledger has many and diverse stakeholders, and everyone deserves a chance to contribute meaningful changes to the code that runs the XRPL.
# Contributing
We assume you are familiar with the general practice of [making
contributions on GitHub][contrib]. This file includes only special
instructions specific to this project.
We assume you are familiar with the general practice of [making contributions on GitHub][contrib]. This file includes only special instructions specific to this project.
## Before you start
The following branches exist in the main project repository:
- `develop`: The latest set of unreleased features, and the most common
starting point for contributions.
- `release/*` (e.g. `release/3.2.x`): Release branches, one per release line,
holding the latest release candidate, or stable release for that line.
Stable releases are published as [tagged releases](https://github.com/XRPLF/rippled/releases).
- `develop`: The latest set of unreleased features, and the most common starting point for contributions.
- `release/*` (e.g. `release/3.2.x`): Release branches, one per release line, holding the latest release candidate, or stable release for that line. Stable releases are published as [tagged releases](https://github.com/XRPLF/rippled/releases).
The tip of each branch must be signed. In order for GitHub to sign a
squashed commit that it builds from your pull request, GitHub must know
your verifying key. Please set up [signature verification][signing].
The tip of each branch must be signed. In order for GitHub to sign a squashed commit that it builds from your pull request, GitHub must know your verifying key. Please set up [signature verification][signing].
In general, external contributions should be developed in your personal
[fork][forking]. Contributions from developers with write permissions
should be done in [the main repository][xrpld] in a branch with
a permitted prefix. Permitted prefixes are:
In general, external contributions should be developed in your personal [fork][forking]. Contributions from developers with write permissions should be done in [the main repository][xrpld] in a branch with a permitted prefix. Permitted prefixes are:
- XLS-[a-zA-Z0-9]+/.+
- e.g. XLS-0033d/mpt-clarify-STEitherAmount
@@ -34,87 +22,47 @@ a permitted prefix. Permitted prefixes are:
- [Organization name]/.+
- e.g. ripple/antithesis
Regardless of where the branch is created, please open a _draft_ pull
request as soon as possible after pushing the branch to Github, to
increase visibility, and ease feedback during the development process.
Regardless of where the branch is created, please open a _draft_ pull request as soon as possible after pushing the branch to Github, to increase visibility, and ease feedback during the development process.
## Major contributions
If your contribution is a major feature or breaking change, then you
must first write an XRP Ledger Standard (XLS) describing it. Go to
[XRPL-Standards](https://github.com/XRPLF/XRPL-Standards/discussions),
choose the next available standard number, and open a discussion with an
appropriate title to propose your draft standard.
If your contribution is a major feature or breaking change, then you must first write an XRP Ledger Standard (XLS) describing it. Go to [XRPL-Standards](https://github.com/XRPLF/XRPL-Standards/discussions), choose the next available standard number, and open a discussion with an appropriate title to propose your draft standard.
When you submit a pull request, please link the corresponding XLS in the
description. An XLS still in `Draft` status is considered a
work-in-progress and open for discussion. Please allow time for
questions, suggestions, and changes to the XLS draft. It is the
responsibility of the XLS author to update the draft to match the final
implementation when its corresponding pull request is merged, unless the
author delegates that responsibility to others.
When you submit a pull request, please link the corresponding XLS in the description. An XLS still in `Draft` status is considered a work-in-progress and open for discussion. Please allow time for questions, suggestions, and changes to the XLS draft. It is the responsibility of the XLS author to update the draft to match the final implementation when its corresponding pull request is merged, unless the author delegates that responsibility to others.
Any amendment or major RPC change requires either a new XLS or an update
to an existing XLS. Neither change will be released (in an amendment's
case, marked as `Supported::yes`) until the corresponding XLS's status
is `Final`.
Any amendment or major RPC change requires either a new XLS or an update to an existing XLS. Neither change will be released (in an amendment's case, marked as `Supported::yes`) until the corresponding XLS's status is `Final`.
## Before making a pull request
(Or marking a draft pull request as ready.)
Changes that alter transaction processing must be guarded by an
[Amendment](https://xrpl.org/amendments.html).
All other changes that maintain the existing behavior do not need an
Amendment.
Changes that alter transaction processing must be guarded by an [Amendment](https://xrpl.org/amendments.html). All other changes that maintain the existing behavior do not need an Amendment.
Ensure that your code compiles according to the build instructions in
[`BUILD.md`](./BUILD.md).
Ensure that your code compiles according to the build instructions in [`BUILD.md`](./BUILD.md).
Please write tests for your code.
If your test can be run offline, in under 60 seconds, then it can be an
automatic test run by `xrpld --unittest`.
Otherwise, it must be a manual test.
Please write tests for your code. If your test can be run offline, in under 60 seconds, then it can be an automatic test run by `xrpld --unittest`. Otherwise, it must be a manual test.
If you create new source files, they must be organized as follows:
- If the files are in any of the `libxrpl` modules, the headers (`.h`) must go
under `include/xrpl`, and source (`.cpp`) files must go under
`src/libxrpl`.
- If the files are in any of the `libxrpl` modules, the headers (`.h`) must go under `include/xrpl`, and source (`.cpp`) files must go under `src/libxrpl`.
- All other non-test files must go under `src/xrpld`.
- All test source files must go under `src/test`.
- All benchmark source files must go under `src/benchmarks`.
The source must be formatted according to the style guide below. The easiest
way to satisfy this is to install the [`pre-commit`](#pre-commit-hooks) hooks,
which format and lint your changes automatically on every commit.
The source must be formatted according to the style guide below. The easiest way to satisfy this is to install the [`pre-commit`](#pre-commit-hooks) hooks, which format and lint your changes automatically on every commit.
Header includes must be [levelized](.github/scripts/levelization).
Changes should be usually squashed down into a single commit.
Some larger or more complicated change sets make more sense,
and are easier to review if organized into multiple logical commits.
Either way, all commits should fit the following criteria:
Changes should be usually squashed down into a single commit. Some larger or more complicated change sets make more sense, and are easier to review if organized into multiple logical commits. Either way, all commits should fit the following criteria:
- Changes should be presented in a single commit or a logical
sequence of commits.
Specifically, chronological commits that simply
reflect the history of how the author implemented
the change, "warts and all", are not useful to
reviewers.
- Every commit should have a [good message](#good-commit-messages).
to explain a specific aspects of the change.
- Changes should be presented in a single commit or a logical sequence of commits. Specifically, chronological commits that simply reflect the history of how the author implemented the change, "warts and all", are not useful to reviewers.
- Every commit should have a [good message](#good-commit-messages). to explain a specific aspects of the change.
- Every commit should be signed.
- Every commit should be well-formed (builds successfully,
unit tests passing), as this helps to resolve merge
conflicts, and makes it easier to use `git bisect`
to find bugs.
- Every commit should be well-formed (builds successfully, unit tests passing), as this helps to resolve merge conflicts, and makes it easier to use `git bisect` to find bugs.
### Good commit messages
Refer to
["How to Write a Git Commit Message"](https://cbea.ms/git-commit/)
for general rules on writing a good commit message.
Refer to ["How to Write a Git Commit Message"](https://cbea.ms/git-commit/) for general rules on writing a good commit message.
tl;dr
@@ -124,9 +72,7 @@ tl;dr
> 3. Capitalize the subject line.
> 4. Do not end the subject line with a period.
> 5. Use the imperative mood in the subject line.
> - A properly formed Git commit subject line should always be able
> to complete the following sentence: "If applied, this commit will
> _your subject line here_".
> - A properly formed Git commit subject line should always be able to complete the following sentence: "If applied, this commit will _your subject line here_".
> 6. Wrap the body at 72 characters.
> 7. Use the body to explain what and why vs. how.
@@ -134,32 +80,15 @@ tl;dr
In general, pull requests use `develop` as the base branch.
The exceptions are fixes, improvements, and hotfixes for an existing release,
which use that release's branch (e.g. `release/3.2.x`) as the base.
The exceptions are fixes, improvements, and hotfixes for an existing release, which use that release's branch (e.g. `release/3.2.x`) as the base.
If your changes are not quite ready, but you want to make it easily available
for preliminary examination or review, you can create a "Draft" pull request.
While a pull request is marked as a "Draft", you can rebase or reorganize the
commits in the pull request as desired.
If your changes are not quite ready, but you want to make it easily available for preliminary examination or review, you can create a "Draft" pull request. While a pull request is marked as a "Draft", you can rebase or reorganize the commits in the pull request as desired.
Github pull requests are created as "Ready" by default, or you can mark
a "Draft" pull request as "Ready".
Once a pull request is marked as "Ready",
any changes must be added as new commits. Do not
force-push to a branch in a pull request under review.
(This includes rebasing your branch onto the updated base branch.
Use a merge operation, instead or hit the "Update branch" button
at the bottom of the Github PR page.)
This preserves the ability for reviewers to filter changes since their last
review.
Github pull requests are created as "Ready" by default, or you can mark a "Draft" pull request as "Ready". Once a pull request is marked as "Ready", any changes must be added as new commits. Do not force-push to a branch in a pull request under review. (This includes rebasing your branch onto the updated base branch. Use a merge operation, instead or hit the "Update branch" button at the bottom of the Github PR page.) This preserves the ability for reviewers to filter changes since their last review.
A pull request must obtain **approvals from at least two reviewers**
before it can be considered for merge by a Maintainer.
Maintainers retain discretion to require more approvals if they feel the
credibility of the existing approvals is insufficient.
A pull request must obtain **approvals from at least two reviewers** before it can be considered for merge by a Maintainer. Maintainers retain discretion to require more approvals if they feel the credibility of the existing approvals is insufficient.
Pull requests must be merged by [squash-and-merge][squash]
to preserve a linear history for the `develop` branch.
Pull requests must be merged by [squash-and-merge][squash] to preserve a linear history for the `develop` branch.
### Type of Change
@@ -180,53 +109,26 @@ First letter after the type prefix should be capitalized, and the type prefix sh
### "Ready to merge"
A pull request should only have the "Ready to merge" label added when it
meets a few criteria:
A pull request should only have the "Ready to merge" label added when it meets a few criteria:
1. It must have two approving reviews [as described
above](#pull-requests). (Exception: PRs that are deemed "trivial"
only need one approval.)
2. All CI checks must be complete and passed. (One-off failures may
be acceptable if they are related to a known issue.)
1. It must have two approving reviews [as described above](#pull-requests). (Exception: PRs that are deemed "trivial" only need one approval.)
2. All CI checks must be complete and passed. (One-off failures may be acceptable if they are related to a known issue.)
3. The PR must have a [good commit message](#good-commit-messages).
- If the PR started with a good commit message, and it doesn't
need to be updated, the author can indicate that in a comment.
- Any contributor, preferably the author, can leave a comment
suggesting a commit message.
- If the author squashes and rebases the code in preparation for
merge, they should also ensure the commit message(s) are updated
as well.
4. The PR branch must be up to date with the base branch (usually
`develop`). This is usually accomplished by merging the base branch
into the feature branch, but if the other criteria are met, the
changes can be squashed and rebased on top of the base branch.
5. Finally, and most importantly, the author of the PR must
positively indicate that the PR is ready to merge. That can be
accomplished by adding the "Ready to merge" label if their role
allows, or by leaving a comment to the effect that the PR is ready to
merge.
- If the PR started with a good commit message, and it doesn't need to be updated, the author can indicate that in a comment.
- Any contributor, preferably the author, can leave a comment suggesting a commit message.
- If the author squashes and rebases the code in preparation for merge, they should also ensure the commit message(s) are updated as well.
4. The PR branch must be up to date with the base branch (usually `develop`). This is usually accomplished by merging the base branch into the feature branch, but if the other criteria are met, the changes can be squashed and rebased on top of the base branch.
5. Finally, and most importantly, the author of the PR must positively indicate that the PR is ready to merge. That can be accomplished by adding the "Ready to merge" label if their role allows, or by leaving a comment to the effect that the PR is ready to merge.
Once the "Ready to merge" label is added, a maintainer may merge the PR
at any time, so don't use it lightly.
Once the "Ready to merge" label is added, a maintainer may merge the PR at any time, so don't use it lightly.
# Style guide
This is a non-exhaustive list of recommended style guidelines. These are
not always strictly enforced and serve as a way to keep the codebase
coherent rather than a set of _thou shalt not_ commandments.
This is a non-exhaustive list of recommended style guidelines. These are not always strictly enforced and serve as a way to keep the codebase coherent rather than a set of _thou shalt not_ commandments.
## Pre-commit hooks
We use the [`pre-commit`](https://pre-commit.com/) framework to run the
formatting and linting tools that keep the codebase consistent. `pre-commit`
runs each tool configured in
[`.pre-commit-config.yaml`](./.pre-commit-config.yaml) in its own isolated
environment, so you don't need to install most of the individual tools
yourself. The version of each hook sourced from an external repository
(`clang-format`, `gersemi`, etc.) is pinned in that file, so running the hooks
locally uses exactly the same versions as CI. A few `local` hooks — most notably
`clang-tidy` — run tools from your own environment; see
[Installing clang-tidy](#installing-clang-tidy) for how to get those.
We use the [`pre-commit`](https://pre-commit.com/) framework to run the formatting and linting tools that keep the codebase consistent. `pre-commit` runs each tool configured in [`.pre-commit-config.yaml`](./.pre-commit-config.yaml) in its own isolated environment, so you don't need to install most of the individual tools yourself. The version of each hook sourced from an external repository (`clang-format`, `gersemi`, etc.) is pinned in that file, so running the hooks locally uses exactly the same versions as CI. A few `local` hooks — most notably `clang-tidy` and `cargo fmt` — run tools from your own environment; see [Installing clang-tidy](#installing-clang-tidy) and [Rust](./docs/build/environment.md#rust) for how to get those.
To get started, install `pre-commit` and enable the git hook scripts:
@@ -235,8 +137,7 @@ pip install pre-commit
pre-commit install
```
Once installed, the hooks run automatically on your staged files every time you
`git commit`. You can also run them on demand:
Once installed, the hooks run automatically on your staged files every time you `git commit`. You can also run them on demand:
```bash
# Run all hooks against only the staged files
@@ -255,21 +156,15 @@ The hooks configured in this repository include, among others:
- `clang-tidy` — C++ static analysis (see [Clang-tidy](#clang-tidy)); opt in with `TIDY=1`
- `fix-include-style`, `fix-pragma-once`, `check-doxygen-style` — C++ hygiene
- `gersemi` — CMake formatting
- `cargo fmt` — Rust formatting for the crates in `crates/`
- `prettier`, `black`, `shfmt` — formatting for JavaScript/JSON/Markdown, Python, and shell
- `cspell` — spell checking
The same hooks run in CI on every pull request, so running them locally before
you push helps you avoid CI failures.
The same hooks run in CI on every pull request, so running them locally before you push helps you avoid CI failures.
## Formatting
All code must conform to `clang-format`, according to the settings in
[`.clang-format`](./.clang-format), unless the result would be unreasonably
difficult to read or maintain. The `clang-format` version is pinned in
[`.pre-commit-config.yaml`](./.pre-commit-config.yaml), so the
[`pre-commit`](#pre-commit-hooks) hook always formats with the same version as
CI. To demarcate lines that should be left as-is, surround them with comments
like this:
All code must conform to `clang-format`, according to the settings in [`.clang-format`](./.clang-format), unless the result would be unreasonably difficult to read or maintain. The `clang-format` version is pinned in [`.pre-commit-config.yaml`](./.pre-commit-config.yaml), so the [`pre-commit`](#pre-commit-hooks) hook always formats with the same version as CI. To demarcate lines that should be left as-is, surround them with comments like this:
```
// clang-format off
@@ -277,20 +172,17 @@ like this:
// clang-format on
```
The easiest way to format your changes is to let the `pre-commit` hook run
automatically on commit, or to run it manually:
The easiest way to format your changes is to let the `pre-commit` hook run automatically on commit, or to run it manually:
```bash
pre-commit run clang-format --all-files
```
You can also format individual files in place by running `clang-format -i <file>...`
from any directory within this project.
You can also format individual files in place by running `clang-format -i <file>...` from any directory within this project.
<!-- prettier-ignore -->
> [!NOTE]
> This uses whatever `clang-format` version is installed locally, which may
> differ from the pinned version used by `pre-commit` and CI, so the results
> can vary.
> This uses whatever `clang-format` version is installed locally, which may differ from the pinned version used by `pre-commit` and CI, so the results can vary.
There is a Continuous Integration job that runs clang-format on pull requests. If the code doesn't comply, a patch file that corrects auto-fixable formatting issues is generated.
@@ -319,7 +211,11 @@ See the [environment setup guide](./docs/build/environment.md#clang-tidy) for ho
### Running clang-tidy locally
Before running clang-tidy, you must build the project to generate required files (particularly protobuf headers). Refer to [`BUILD.md`](./BUILD.md) for build instructions.
Before running clang-tidy, you must generate the files it depends on (protobuf headers, and, when the project is configured with `-Drust=ON`, the cxxbridge headers from the Rust crates). Configure the project as described in [`BUILD.md`](./BUILD.md), then build the `tidy_prerequisites` target, which generates all of them:
```bash
cmake --build build --target tidy_prerequisites
```
#### Via pre-commit (recommended)
@@ -347,8 +243,7 @@ Then run clang-tidy on your local changes:
run-clang-tidy -p build -allow-no-checks src tests
```
This will check all source files in the `src`, `include` and `tests` directories using the compile commands from your `build` directory.
If you wish to automatically fix whatever clang-tidy finds _and_ is capable of fixing, add `-fix -format` to the above command:
This will check all source files in the `src`, `include` and `tests` directories using the compile commands from your `build` directory. If you wish to automatically fix whatever clang-tidy finds _and_ is capable of fixing, add `-fix -format` to the above command:
```
run-clang-tidy -p build -quiet -fix -format -allow-no-checks src tests
@@ -358,27 +253,11 @@ run-clang-tidy -p build -quiet -fix -format -allow-no-checks src tests
## Contracts and instrumentation
We are using [Antithesis](https://antithesis.com/) for continuous fuzzing,
and keep a copy of [Antithesis C++ SDK](https://github.com/antithesishq/antithesis-sdk-cpp/)
in `external/antithesis-sdk`. One of the aims of fuzzing is to identify bugs
by finding external conditions which cause contracts violations inside `xrpld`.
The contracts are expressed as `XRPL_ASSERT` or `UNREACHABLE` (defined in
`include/xrpl/beast/utility/instrumentation.h`), which are effectively (outside
of Antithesis) wrappers for `assert(...)` with added name. The purpose of name
is to provide contracts with stable identity which does not rely on line numbers.
We are using [Antithesis](https://antithesis.com/) for continuous fuzzing, and keep a copy of [Antithesis C++ SDK](https://github.com/antithesishq/antithesis-sdk-cpp/) in `external/antithesis-sdk`. One of the aims of fuzzing is to identify bugs by finding external conditions which cause contracts violations inside `xrpld`. The contracts are expressed as `XRPL_ASSERT` or `UNREACHABLE` (defined in `include/xrpl/beast/utility/instrumentation.h`), which are effectively (outside of Antithesis) wrappers for `assert(...)` with added name. The purpose of name is to provide contracts with stable identity which does not rely on line numbers.
When `xrpld` is built with the Antithesis instrumentation enabled
(using `voidstar` CMake option) and ran on the Antithesis platform, the
contracts become
[test properties](https://antithesis.com/docs/using_antithesis/properties.html);
otherwise they are just like a regular `assert`.
To learn more about Antithesis, see
[How Antithesis Works](https://antithesis.com/docs/introduction/how_antithesis_works.html)
and [C++ SDK](https://antithesis.com/docs/using_antithesis/sdk/cpp/overview.html#)
When `xrpld` is built with the Antithesis instrumentation enabled (using `voidstar` CMake option) and ran on the Antithesis platform, the contracts become [test properties](https://antithesis.com/docs/using_antithesis/properties.html); otherwise they are just like a regular `assert`. To learn more about Antithesis, see [How Antithesis Works](https://antithesis.com/docs/introduction/how_antithesis_works.html) and [C++ SDK](https://antithesis.com/docs/using_antithesis/sdk/cpp/overview.html#)
We continue to use the old style `assert` or `assert(false)` in certain
locations, where the reporting of contract violations on the Antithesis
platform is either not possible or not useful.
We continue to use the old style `assert` or `assert(false)` in certain locations, where the reporting of contract violations on the Antithesis platform is either not possible or not useful.
For this reason:
@@ -386,38 +265,17 @@ For this reason:
- `constexpr` functions
- unit tests i.e. files under `src/test`
- unit tests-related modules (files under `beast/test` and `beast/unit_test`)
- Outside of the listed locations, do not use `assert`; use `XRPL_ASSERT` instead,
giving it unique name, with the short description of the contract.
- Outside of the listed locations, do not use `assert(false)`; use
`UNREACHABLE` instead, giving it unique name, with the description of the
condition being violated
- The contract name should start with a full name (including scope) of the
function, optionally a named lambda, followed by a colon `:` and a brief
(typically at most five words) description. `UNREACHABLE` contracts
can use slightly longer descriptions. If there are multiple overloads of the
function, use common sense to balance both brevity and unambiguity of the
function name. NOTE: the purpose of name is to provide stable means of
unique identification of every contract; for this reason try to avoid elements
which can change in some obvious refactors or when reinforcing the condition.
- Contract description typically (except for `UNREACHABLE`) should describe the
_expected_ condition, as in "I assert that _expected_ is true".
- Contract description for `UNREACHABLE` should describe the _unexpected_
situation which caused the line to have been reached.
- Example good name for an
`UNREACHABLE` macro `"json::operator==(Value, Value) : invalid type"`; example
good name for an `XRPL_ASSERT` macro `"json::Value::asCString : valid type"`.
- Example **bad** name
`"RFC1751::insert(char* s, int x, int start, int length) : length is greater than or equal zero"`
(missing namespace, unnecessary full function signature, description too verbose).
Good name: `"xrpl::RFC1751::insert : minimum length"`.
- In **few** well-justified cases a non-standard name can be used, in which case a
comment should be placed to explain the rationale (example in `contract.cpp`)
- Do **not** rename a contract without a good reason (e.g. the name no longer
reflects the location or the condition being checked)
- Outside of the listed locations, do not use `assert`; use `XRPL_ASSERT` instead, giving it unique name, with the short description of the contract.
- Outside of the listed locations, do not use `assert(false)`; use `UNREACHABLE` instead, giving it unique name, with the description of the condition being violated
- The contract name should start with a full name (including scope) of the function, optionally a named lambda, followed by a colon `:` and a brief (typically at most five words) description. `UNREACHABLE` contracts can use slightly longer descriptions. If there are multiple overloads of the function, use common sense to balance both brevity and unambiguity of the function name. NOTE: the purpose of name is to provide stable means of unique identification of every contract; for this reason try to avoid elements which can change in some obvious refactors or when reinforcing the condition.
- Contract description typically (except for `UNREACHABLE`) should describe the _expected_ condition, as in "I assert that _expected_ is true".
- Contract description for `UNREACHABLE` should describe the _unexpected_ situation which caused the line to have been reached.
- Example good name for an `UNREACHABLE` macro `"json::operator==(Value, Value) : invalid type"`; example good name for an `XRPL_ASSERT` macro `"json::Value::asCString : valid type"`.
- Example **bad** name `"RFC1751::insert(char* s, int x, int start, int length) : length is greater than or equal zero"` (missing namespace, unnecessary full function signature, description too verbose). Good name: `"xrpl::RFC1751::insert : minimum length"`.
- In **few** well-justified cases a non-standard name can be used, in which case a comment should be placed to explain the rationale (example in `contract.cpp`)
- Do **not** rename a contract without a good reason (e.g. the name no longer reflects the location or the condition being checked)
- Do not use `std::unreachable`
- Do not put contracts where they can be violated by an external condition
(e.g. timing, data payload before mandatory validation etc.) as this creates
bogus bug reports (and causes crashes of Debug builds)
- Do not put contracts where they can be violated by an external condition (e.g. timing, data payload before mandatory validation etc.) as this creates bogus bug reports (and causes crashes of Debug builds)
## Unit Tests
@@ -425,8 +283,7 @@ To execute all unit tests:
`xrpld --unittest --unittest-jobs=<number of cores>`
(Note: Using multiple cores on a Mac M1 can cause spurious test failures. The
cause is still under investigation. If you observe this problem, try specifying fewer jobs.)
(Note: Using multiple cores on a Mac M1 can cause spurious test failures. The cause is still under investigation. If you observe this problem, try specifying fewer jobs.)
To run a specific set of test suites:
@@ -434,11 +291,7 @@ To run a specific set of test suites:
xrpld --unittest TestSuiteName
```
Note: In this example, all tests with prefix `TestSuiteName` will be run, so if
`TestSuiteName1` and `TestSuiteName2` both exist, then both tests will run.
Alternatively, if the unit test name finds an exact match, it will stop
doing partial matches, i.e. if a unit test with a title of `TestSuiteName`
exists, then no other unit test will be executed, apart from `TestSuiteName`.
Note: In this example, all tests with prefix `TestSuiteName` will be run, so if `TestSuiteName1` and `TestSuiteName2` both exist, then both tests will run. Alternatively, if the unit test name finds an exact match, it will stop doing partial matches, i.e. if a unit test with a title of `TestSuiteName` exists, then no other unit test will be executed, apart from `TestSuiteName`.
## Avoid
@@ -448,44 +301,27 @@ exists, then no other unit test will be executed, apart from `TestSuiteName`.
4. Unmanaged memory allocation and raw pointers.
5. Macros and non-trivial templates (unless they add significant value).
6. Lambda patterns (unless these add significant value).
7. CPU or architecture-specific code unless there is a good reason to
include it, and where it is used, guard it with macros and provide
explanatory comments.
7. CPU or architecture-specific code unless there is a good reason to include it, and where it is used, guard it with macros and provide explanatory comments.
8. Importing new libraries unless there is a very good reason to do so.
## Seek to
9. Extend functionality of existing code rather than creating new code.
10. Prefer readability over terseness where important logic is
concerned.
11. Inline functions that are not used or are not likely to be used
elsewhere in the codebase.
12. Use clear and self-explanatory names for functions, variables,
structs and classes.
13. Use TitleCase for classes, structs and filenames, camelCase for
function and variable names, lower case for namespaces and folders.
14. Provide as many comments as you feel that a competent programmer
would need to understand what your code does.
10. Prefer readability over terseness where important logic is concerned.
11. Inline functions that are not used or are not likely to be used elsewhere in the codebase.
12. Use clear and self-explanatory names for functions, variables, structs and classes.
13. Use TitleCase for classes, structs and filenames, camelCase for function and variable names, lower case for namespaces and folders.
14. Provide as many comments as you feel that a competent programmer would need to understand what your code does.
# Maintainers
Maintainers are ecosystem participants with elevated access to the repository.
They are able to push new code, make decisions on when a release should be
made, etc.
Maintainers are ecosystem participants with elevated access to the repository. They are able to push new code, make decisions on when a release should be made, etc.
## Adding and removing
New maintainers can be proposed by two existing maintainers, subject to a vote
by a quorum of the existing maintainers.
A minimum of 50% support and a 50% participation is required.
In the event of a tie vote, the addition of the new maintainer will be
rejected.
New maintainers can be proposed by two existing maintainers, subject to a vote by a quorum of the existing maintainers. A minimum of 50% support and a 50% participation is required. In the event of a tie vote, the addition of the new maintainer will be rejected.
Existing maintainers can resign, or be subject to a vote for removal at the
behest of two existing maintainers.
A minimum of 60% agreement and 50% participation are required.
The XRP Ledger Foundation will have the ability, for cause, to remove an
existing maintainer without a vote.
Existing maintainers can resign, or be subject to a vote for removal at the behest of two existing maintainers. A minimum of 60% agreement and 50% participation are required. The XRP Ledger Foundation will have the ability, for cause, to remove an existing maintainer without a vote.
## Current Maintainers
@@ -501,8 +337,7 @@ Maintainers are users with maintain or admin access to the repo.
## Current Code Reviewers
Code Reviewers are developers who have the ability to review, approve, and
in some cases merge source code changes.
Code Reviewers are developers who have the ability to review, approve, and in some cases merge source code changes.
- [a1q123456](https://github.com/a1q123456) (Ripple)
- [Bronek](https://github.com/Bronek) (Ripple)
@@ -523,14 +358,11 @@ in some cases merge source code changes.
- [Tapanito](https://github.com/Tapanito) (Ripple)
- [ximinez](https://github.com/ximinez) (Ripple)
Developers not on this list are able and encouraged to submit feedback
on pending code changes (open pull requests).
Developers not on this list are able and encouraged to submit feedback on pending code changes (open pull requests).
## Instructions for maintainers
These instructions assume you have your git upstream remotes configured
to avoid accidental pushes to the main repo, and a remote group
specifying both of them. e.g.
These instructions assume you have your git upstream remotes configured to avoid accidental pushes to the main repo, and a remote group specifying both of them. e.g.
```
$ git remote -v | grep upstream
@@ -555,104 +387,58 @@ $ git config user.signingkey
### When and how to merge pull requests
The maintainer should double-check that the PR has met all the
necessary criteria, and can request additional information from the
owner, or additional reviews, and can always feel free to remove the
"Ready to merge" label if appropriate. The maintainer has final say on
whether a PR gets merged, and are encouraged to communicate and issues
or concerns to other maintainers.
The maintainer should double-check that the PR has met all the necessary criteria, and can request additional information from the owner, or additional reviews, and can always feel free to remove the "Ready to merge" label if appropriate. The maintainer has final say on whether a PR gets merged, and are encouraged to communicate and issues or concerns to other maintainers.
#### Most pull requests: "Squash and merge"
Most pull requests don't need special handling, and can simply be
merged using the "Squash and merge" button on the Github UI. Update
the suggested commit message, or modify it as needed.
Most pull requests don't need special handling, and can simply be merged using the "Squash and merge" button on the Github UI. Update the suggested commit message, or modify it as needed.
#### Slightly more complicated pull requests
Some pull requests need to be pushed to `develop` as more than one
commit. A PR author may _request_ to merge as separate commits. They
must _justify_ why separate commits are needed, and _specify_ how they
would like the commits to be merged. If you disagree with the author,
discuss it with them directly.
Some pull requests need to be pushed to `develop` as more than one commit. A PR author may _request_ to merge as separate commits. They must _justify_ why separate commits are needed, and _specify_ how they would like the commits to be merged. If you disagree with the author, discuss it with them directly.
If the process is reasonable, follow it. The simplest option is to do a
fast forward only merge (`--ff-only`) on the command line and push to
`develop`.
If the process is reasonable, follow it. The simplest option is to do a fast forward only merge (`--ff-only`) on the command line and push to `develop`.
Some examples of when separate commits are worthwhile are:
1. PRs where source files are reorganized in multiple steps.
2. PRs where the commits are mostly independent and _could_ be separate
PRs, but are pulled together into one PR under a commit theme or
issue.
3. PRs that are complicated enough that `git bisect` would not be much
help if it determined this PR introduced a problem.
2. PRs where the commits are mostly independent and _could_ be separate PRs, but are pulled together into one PR under a commit theme or issue.
3. PRs that are complicated enough that `git bisect` would not be much help if it determined this PR introduced a problem.
Either way, check that:
- The commits are based on the current tip of `develop`.
- The commits are clean: No merge commits (except when reverse
merging), no "[FOLD]" or "fixup!" messages.
- All commits are signed. If the commits are not signed by the author, use
`git commit --amend -S` to sign them yourself.
- At least one (but preferably all) of the commits has the PR number
in the commit message.
- The commits are clean: No merge commits (except when reverse merging), no "[FOLD]" or "fixup!" messages.
- All commits are signed. If the commits are not signed by the author, use `git commit --amend -S` to sign them yourself.
- At least one (but preferably all) of the commits has the PR number in the commit message.
The "Create a merge commit" and "Rebase and merge" options should be
disabled in the Github UI, but if you ever find them available **Do not
use them!**
The "Create a merge commit" and "Rebase and merge" options should be disabled in the Github UI, but if you ever find them available **Do not use them!**
### Releases
All releases, including release candidates and betas, are handled
differently from typical PRs. Most importantly, never use
the Github UI to merge a release.
All releases, including release candidates and betas, are handled differently from typical PRs. Most importantly, never use the Github UI to merge a release.
Xrpld uses a linear workflow model that can be summarized as:
1. In between releases, developers work against the `develop` branch.
2. Periodically, a maintainer will build and tag a beta version from
`develop`, which is pushed to `release`.
- Betas are usually released every two to three weeks, though that
schedule can vary depending on progress, availability, and other
factors.
3. When the changes in `develop` are considered stable and mature enough
to be ready to release, a release candidate (RC) is built and tagged
from `develop`, and merged to `release`.
- Further development for that release (primarily fixes) then
continues against `release`, while other development continues on
`develop`. Effectively, `release` is forked from `develop`. Changes
to `release` must be reverse merged to `develop`.
4. When the candidate has passed testing and is ready for release, the
final release is merged to `master`.
5. If any issues are found post-release, a hotfix / point release may be
created, which is merged to `master`, and then reverse merged to
`develop`.
2. Periodically, a maintainer will build and tag a beta version from `develop`, which is pushed to `release`.
- Betas are usually released every two to three weeks, though that schedule can vary depending on progress, availability, and other factors.
3. When the changes in `develop` are considered stable and mature enough to be ready to release, a release candidate (RC) is built and tagged from `develop`, and merged to `release`.
- Further development for that release (primarily fixes) then continues against `release`, while other development continues on `develop`. Effectively, `release` is forked from `develop`. Changes to `release` must be reverse merged to `develop`.
4. When the candidate has passed testing and is ready for release, the final release is merged to `master`.
5. If any issues are found post-release, a hotfix / point release may be created, which is merged to `master`, and then reverse merged to `develop`.
#### Betas, and the first release candidate
##### Preparing the `develop` branch
1. Optimally, the `develop` branch will be ready to go, with all
relevant PRs already merged.
1. Optimally, the `develop` branch will be ready to go, with all relevant PRs already merged.
2. If there are any PRs pending, merge them **BEFORE** preparing the beta.
1. If only one or two PRs need to be merged, merge those PRs [as
normal](#when-and-how-to-merge-pull-requests), updating the second
one, and waiting for CI to finish in between.
2. If there are several pending PRs, do not use the Github UI,
because the delays waiting for CI in between each merge will be
unnecessarily onerous. (Incidentally, this process can also be
used to merge if the Github UI has issues.) Merge each PR branch
directly to a `release-next` on your local machine and create a single
PR, then push your branch to `develop`.
1. Squash the changes from each PR, one commit each (unless more
are needed), being sure to sign each commit and update the
commit message to include the PR number. You may be able to use
a fast-forward merge for the first PR.
1. If only one or two PRs need to be merged, merge those PRs [as normal](#when-and-how-to-merge-pull-requests), updating the second one, and waiting for CI to finish in between.
2. If there are several pending PRs, do not use the Github UI, because the delays waiting for CI in between each merge will be unnecessarily onerous. (Incidentally, this process can also be used to merge if the Github UI has issues.) Merge each PR branch directly to a `release-next` on your local machine and create a single PR, then push your branch to `develop`.
1. Squash the changes from each PR, one commit each (unless more are needed), being sure to sign each commit and update the commit message to include the PR number. You may be able to use a fast-forward merge for the first PR.
2. Push your branch.
3. Continue to [Making the release](#making-the-release) to update
the version number, etc.
3. Continue to [Making the release](#making-the-release) to update the version number, etc.
The workflow may look something like:
@@ -685,17 +471,13 @@ git push --set-upstream origin
You can also use the [squash-branches] script.
You may also need to manually close the open PRs after the changes are
merged to `develop`. Be sure to include the commit ID.
You may also need to manually close the open PRs after the changes are merged to `develop`. Be sure to include the commit ID.
##### Making the release
This includes, betas, and the first release candidate (RC).
1. If you didn't create one [preparing the `develop`
branch](#preparing-the-develop-branch), Ensure there is no old
`release-next` branch hanging around. Then make a `release-next`
branch that only changes the version number. e.g.
1. If you didn't create one [preparing the `develop` branch](#preparing-the-develop-branch), Ensure there is no old `release-next` branch hanging around. Then make a `release-next` branch that only changes the version number. e.g.
```
git fetch upstreams
@@ -718,8 +500,7 @@ git fetch upstreams
git branch --set-upstream-to=upstream/release-next
```
You can also use the [update-version] script. 2. Create a Pull Request for `release-next` with **`develop`** as
the base branch.
You can also use the [update-version] script. 2. Create a Pull Request for `release-next` with **`develop`** as the base branch.
1. Use the title "[TRIVIAL] Set version to X.X.X-bX".
2. Instead of the default description template, use the following:
@@ -731,27 +512,17 @@ This PR only changes the version number. It will be merged as
soon as Github CI actions successfully complete.
```
3. Wait for CI to successfully complete, and get someone to approve
the PR. (It is safe to ignore known CI issues.)
4. Push the updated `develop` branch using your `release-next`
branch. **Do not use the Github UI. It's important to preserve
commit IDs.**
3. Wait for CI to successfully complete, and get someone to approve the PR. (It is safe to ignore known CI issues.)
4. Push the updated `develop` branch using your `release-next` branch. **Do not use the Github UI. It's important to preserve commit IDs.**
```
git push upstream-push release-next:develop
```
5. In the unlikely event that the push fails because someone has merged
something else in the meantime, rebase your branch onto the updated
`develop` branch, push again, and go back to step 3.
6. Ensure that your PR against `develop` is closed. Github should do it
automatically.
7. Once this is done, forward progress on `develop` can continue
(other PRs may be merged).
8. Now create a Pull Request for `release-next` with **`release`** as
the base branch. Instead of the default template, reuse and update
the message from the previous release. Include the following verbiage
somewhere in the description:
5. In the unlikely event that the push fails because someone has merged something else in the meantime, rebase your branch onto the updated `develop` branch, push again, and go back to step 3.
6. Ensure that your PR against `develop` is closed. Github should do it automatically.
7. Once this is done, forward progress on `develop` can continue (other PRs may be merged).
8. Now create a Pull Request for `release-next` with **`release`** as the base branch. Instead of the default template, reuse and update the message from the previous release. Include the following verbiage somewhere in the description:
```
The base branch is `release`. [All releases (including
@@ -760,12 +531,8 @@ go in `release`. This PR branch will be pushed directly to `release` (not
squashed or rebased, and not using the GitHub UI).
```
7. Sign-offs for the three platforms (Linux, Mac, Windows) usually occur
offline, but at least one approval will be needed on the PR.
- If issues are discovered during testing, simply abandon the
release. It's easy to start a new release, it should be easy to
abandon one. **DO NOT REUSE THE VERSION NUMBER.** e.g. If you
abandon 2.4.0-b1, the next attempt will be 2.4.0-b2.
7. Sign-offs for the three platforms (Linux, Mac, Windows) usually occur offline, but at least one approval will be needed on the PR.
- If issues are discovered during testing, simply abandon the release. It's easy to start a new release, it should be easy to abandon one. **DO NOT REUSE THE VERSION NUMBER.** e.g. If you abandon 2.4.0-b1, the next attempt will be 2.4.0-b2.
8. Once everything is ready to go, push to `release`.
```
@@ -802,44 +569,25 @@ git push upstream-push <version number>
git push --delete upstream-push release-next
```
11. Finally [create a new release on
Github](https://github.com/XRPLF/rippled/releases).
11. Finally [create a new release on Github](https://github.com/XRPLF/rippled/releases).
#### Release candidates after the first
Once the first release candidate is [merged into
release](#making-the-release), then `release` and `develop` _are allowed
to diverge_.
Once the first release candidate is [merged into release](#making-the-release), then `release` and `develop` _are allowed to diverge_.
If a bug or issue is discovered in a version that has a release
candidate being tested, any fix and new version will need to be applied
against `release`, then reverse-merged to `develop`. This helps keep git
history as linear as possible.
If a bug or issue is discovered in a version that has a release candidate being tested, any fix and new version will need to be applied against `release`, then reverse-merged to `develop`. This helps keep git history as linear as possible.
A `release-next` branch will be created from `release`, and any further
work for that release must be based on `release-next`. Specifically,
PRs must use `release-next` as the base, and those PRs will be merged
directly to `release-next` when approved. Changes should be restricted
to bug fixes, but other changes may be necessary from time to time.
A `release-next` branch will be created from `release`, and any further work for that release must be based on `release-next`. Specifically, PRs must use `release-next` as the base, and those PRs will be merged directly to `release-next` when approved. Changes should be restricted to bug fixes, but other changes may be necessary from time to time.
1. Open any PRs for the pending release using `release-next` as the base,
so they can be merged directly in to it. Unlike `develop`, though,
`release-next` can be thrown away and recreated if necessary.
2. Once a new release candidate is ready, create a version commit as in
step 1 [above](#making-the-release) on `release-next`. You can use
the [update-version] script for this, too.
3. Jump to step 8 ("Now create a Pull Request for `release-next` with
**`release`** as the base") from the process
[above](#making-the-release) to merge `release-next` into `release`.
1. Open any PRs for the pending release using `release-next` as the base, so they can be merged directly in to it. Unlike `develop`, though, `release-next` can be thrown away and recreated if necessary.
2. Once a new release candidate is ready, create a version commit as in step 1 [above](#making-the-release) on `release-next`. You can use the [update-version] script for this, too.
3. Jump to step 8 ("Now create a Pull Request for `release-next` with **`release`** as the base") from the process [above](#making-the-release) to merge `release-next` into `release`.
##### Follow up: reverse merge
Once the RC is merged and tagged, it needs to be reverse merged into
`develop` as soon as possible.
Once the RC is merged and tagged, it needs to be reverse merged into `develop` as soon as possible.
1. Create a branch, based on `upstream/develop`.
The branch name is not important, but could include "mergeNNNrcN".
E.g. For release A.B.C-rcD, use `mergeABCrcD`.
1. Create a branch, based on `upstream/develop`. The branch name is not important, but could include "mergeNNNrcN". E.g. For release A.B.C-rcD, use `mergeABCrcD`.
```
git fetch upstreams
@@ -855,24 +603,15 @@ git checkout --no-track -b mergeABCrcD upstream/develop
git merge upstream/release
```
3. `BuildInfo.cpp` will have a conflict with the version number.
Resolve it with the version from `develop` - the higher version.
4. Push your branch to your repo (or `upstream` if you have permission),
and open a normal PR against `develop`. The "High level overview" can
simply indicate that this is a merge of the RC. The "Context" should
summarize the changes from the RC. Include the following text
prominently:
3. `BuildInfo.cpp` will have a conflict with the version number. Resolve it with the version from `develop` - the higher version.
4. Push your branch to your repo (or `upstream` if you have permission), and open a normal PR against `develop`. The "High level overview" can simply indicate that this is a merge of the RC. The "Context" should summarize the changes from the RC. Include the following text prominently:
```
This PR must be merged manually using a push. Do not use the Github UI.
```
5. Depending on the complexity of the changes, and/or merge conflicts,
the PR may need a thorough review, or just a sign-off that the
merge was done correctly.
6. If `develop` is updated before this PR is merged, do not merge
`develop` back into your branch. Instead rebase preserving merges,
or do the merge again. (See also the `rerere` git config setting.)
5. Depending on the complexity of the changes, and/or merge conflicts, the PR may need a thorough review, or just a sign-off that the merge was done correctly.
6. If `develop` is updated before this PR is merged, do not merge `develop` back into your branch. Instead rebase preserving merges, or do the merge again. (See also the `rerere` git config setting.)
```
git rebase --rebase-merges upstream/develop
@@ -900,30 +639,14 @@ Development on `develop` can proceed as normal.
A final release is any release that is not a beta or RC, such as 2.2.0.
Only code that has already been tested and vetted across all three
platforms should be included in a final release. Most of the time, that
means that the commit immediately preceding the commit setting the
version number will be an RC. Occasionally, there may be last-minute bug
fixes included as well. If so, those bug fixes must have been tested
internally as if they were RCs (at minimum, ensuring unit tests pass,
and the app starts, syncs, and stops cleanly across all three
platforms.)
Only code that has already been tested and vetted across all three platforms should be included in a final release. Most of the time, that means that the commit immediately preceding the commit setting the version number will be an RC. Occasionally, there may be last-minute bug fixes included as well. If so, those bug fixes must have been tested internally as if they were RCs (at minimum, ensuring unit tests pass, and the app starts, syncs, and stops cleanly across all three platforms.)
_If in doubt, make an RC first._
The process for building a final release is very similar to [the process
for building a beta](#making-the-release), except the code will be
moving from `release` to `master` instead of from `develop` to
`release`, and both branches will be pushed at the same time.
The process for building a final release is very similar to [the process for building a beta](#making-the-release), except the code will be moving from `release` to `master` instead of from `develop` to `release`, and both branches will be pushed at the same time.
1. Ensure there is no old `master-next` branch hanging around.
Then make a `master-next` branch that only changes the version
number. As above, or using the
[update-version] script.
2. Create a Pull Request for `master-next` with **`master`** as
the base branch. Instead of the default template, reuse and update
the message from the previous final release. Include the following verbiage
somewhere in the description:
1. Ensure there is no old `master-next` branch hanging around. Then make a `master-next` branch that only changes the version number. As above, or using the [update-version] script.
2. Create a Pull Request for `master-next` with **`master`** as the base branch. Instead of the default template, reuse and update the message from the previous final release. Include the following verbiage somewhere in the description:
```
The base branch is `master`. This PR branch will be pushed directly to
@@ -931,11 +654,8 @@ The base branch is `master`. This PR branch will be pushed directly to
GitHub UI).
```
7. Sign-offs for the three platforms (Linux, Mac, Windows) usually occur
offline, but at least one approval will be needed on the PR.
- If issues are discovered during testing, close the PR, delete
`master-next`, and move development back to `release`, [issuing
more RCs as necessary](#release-candidates-after-the-first)
7. Sign-offs for the three platforms (Linux, Mac, Windows) usually occur offline, but at least one approval will be needed on the PR.
- If issues are discovered during testing, close the PR, delete `master-next`, and move development back to `release`, [issuing more RCs as necessary](#release-candidates-after-the-first)
8. Once everything is ready to go, push to `release` and `master`.
```
@@ -974,31 +694,19 @@ git push upstream-push <version number>
git push --delete upstream-push master-next
```
11. [Create a new release on
Github](https://github.com/XRPLF/rippled/releases). Be sure that
"Set as the latest release" is checked.
11. [Create a new release on Github](https://github.com/XRPLF/rippled/releases). Be sure that "Set as the latest release" is checked.
12. Open a PR to update the [API-CHANGELOG](API-CHANGELOG.md) and `API-VERSION-[n].md` with the changes for this release (if any are missing).
13. Finally, [reverse merge the release into `develop`](#follow-up-reverse-merge).
#### Special cases: point releases, hotfixes, etc.
On occasion, a bug or issue is discovered in a version that already
had a final release. Most of the time, development will have started
on the next version, and will usually have changes in `develop`
and often in `release`.
On occasion, a bug or issue is discovered in a version that already had a final release. Most of the time, development will have started on the next version, and will usually have changes in `develop` and often in `release`.
Because git history is kept as linear as possible, any fix and new
version will need to be applied against `master`.
Because git history is kept as linear as possible, any fix and new version will need to be applied against `master`.
The process for building a hotfix release is very similar to [the
process for building release candidates after the
first](#release-candidates-after-the-first) and [for building a final
release](#final-releases), except the changes will be done against
`master` instead of `release`.
The process for building a hotfix release is very similar to [the process for building release candidates after the first](#release-candidates-after-the-first) and [for building a final release](#final-releases), except the changes will be done against `master` instead of `release`.
If there is only a single issue for the hotfix, the work can be done in
any branch. When it's ready to merge, jump to step 3 using your branch
instead of `master-next`.
If there is only a single issue for the hotfix, the work can be done in any branch. When it's ready to merge, jump to step 3 using your branch instead of `master-next`.
1. Create a `master-next` branch from `master`.
@@ -1008,27 +716,17 @@ git push upstream-push
git fetch upstreams
```
2. Open any PRs for the pending hotfix using `master-next` as the base,
so they can be merged directly in to it. Unlike `develop`, though,
`master-next` can be thrown away and recreated if necessary.
3. Once the hotfix is ready, create a version commit using the same
steps as above, or use the
[update-version] script.
4. Create a Pull Request for `master-next` with **`master`** as
the base branch. Instead of the default template, reuse and update
the message from the previous final release. Include the following verbiage
somewhere in the description:
2. Open any PRs for the pending hotfix using `master-next` as the base, so they can be merged directly in to it. Unlike `develop`, though, `master-next` can be thrown away and recreated if necessary.
3. Once the hotfix is ready, create a version commit using the same steps as above, or use the [update-version] script.
4. Create a Pull Request for `master-next` with **`master`** as the base branch. Instead of the default template, reuse and update the message from the previous final release. Include the following verbiage somewhere in the description:
```
The base branch is `master`. This PR branch will be pushed directly to
`master` (not squashed or rebased, and not using the GitHub UI).
```
7. Sign-offs for the three platforms (Linux, Mac, Windows) usually occur
offline, but at least one approval will be needed on the PR.
- If issues are discovered during testing, update `master-next` as
needed, but ensure that the changes are properly squashed, and the
version setting commit remains last
7. Sign-offs for the three platforms (Linux, Mac, Windows) usually occur offline, but at least one approval will be needed on the PR.
- If issues are discovered during testing, update `master-next` as needed, but ensure that the changes are properly squashed, and the version setting commit remains last
8. Once everything is ready to go, push to `master` **only**.
```
@@ -1065,17 +763,11 @@ git push upstream-push <version number>
git push --delete upstream-push master-next
```
10. [Create a new release on
Github](https://github.com/XRPLF/rippled/releases). Be sure that
"Set as the latest release" is checked.
10. [Create a new release on Github](https://github.com/XRPLF/rippled/releases). Be sure that "Set as the latest release" is checked.
Once the hotfix is released, it needs to be reverse merged into
`develop` as soon as possible. It may also need to be merged into
`release` if a release candidate is under development.
Once the hotfix is released, it needs to be reverse merged into `develop` as soon as possible. It may also need to be merged into `release` if a release candidate is under development.
1. Create a branch in your own repo, based on `upstream/develop`.
The branch name is not important, but could include "mergeNNN".
E.g. For release 2.2.3, use `merge223`.
1. Create a branch in your own repo, based on `upstream/develop`. The branch name is not important, but could include "mergeNNN". E.g. For release 2.2.3, use `merge223`.
```
git fetch upstreams
@@ -1091,24 +783,15 @@ git checkout --no-track -b merge223 upstream/develop
git merge upstream/master
```
3. `BuildInfo.cpp` will have a conflict with the version number.
Resolve it with the version from `develop` - the higher version.
4. Push your branch to your repo, and open a normal PR against
`develop`. The "High level overview" can simply indicate that this
is a merge of the hotfix version. The "Context" should summarize
the changes from the hotfix. Include the following text
prominently:
3. `BuildInfo.cpp` will have a conflict with the version number. Resolve it with the version from `develop` - the higher version.
4. Push your branch to your repo, and open a normal PR against `develop`. The "High level overview" can simply indicate that this is a merge of the hotfix version. The "Context" should summarize the changes from the hotfix. Include the following text prominently:
```
This PR must be merged manually using a --ff-only merge. Do not use the Github UI.
```
5. Depending on the complexity of the hotfix, and/or merge conflicts,
the PR may need a thorough review, or just a sign-off that the
merge was done correctly.
6. If `develop` is updated before this PR is merged, do not merge
`develop` back into your branch. Instead rebase preserving merges,
or do the merge again. (See also the `rerere` git config setting.)
5. Depending on the complexity of the hotfix, and/or merge conflicts, the PR may need a thorough review, or just a sign-off that the merge was done correctly.
6. If `develop` is updated before this PR is merged, do not merge `develop` back into your branch. Instead rebase preserving merges, or do the merge again. (See also the `rerere` git config setting.)
```
git rebase --rebase-merges upstream/develop
@@ -1128,22 +811,13 @@ git log --show-signature "upstream/develop..HEAD"
git push upstream-push HEAD:develop
```
Development on `develop` can proceed as normal. It is recommended to
create a beta (or RC) immediately to ensure that everything worked as
expected.
Development on `develop` can proceed as normal. It is recommended to create a beta (or RC) immediately to ensure that everything worked as expected.
##### An even rarer scenario: A hotfix on an old release
Historically, once a final release is tagged and packages are released,
versions older than the latest final release are no longer supported.
However, there is a possibility that a very high severity bug may occur
in a non-amendment blocked version that is still being run by
a significant fraction of users, which would necessitate a hotfix / point
release to that version as well as any later versions.
Historically, once a final release is tagged and packages are released, versions older than the latest final release are no longer supported. However, there is a possibility that a very high severity bug may occur in a non-amendment blocked version that is still being run by a significant fraction of users, which would necessitate a hotfix / point release to that version as well as any later versions.
This scenario would follow the same basic procedure as above,
except that _none_ of `develop`, `release`, or `master`
would be touched during the release process.
This scenario would follow the same basic procedure as above, except that _none_ of `develop`, `release`, or `master` would be touched during the release process.
In this example, consider if version 2.1.1 needed to be patched.
@@ -1163,20 +837,9 @@ git push upstream-push
git fetch upstreams
```
2. Work continues as above, except using `master-2.1.2`as
the base branch for any merging, packaging, etc.
3. After the release is tagged and packages are built, you could
potentially delete both branches, e.g. `master-2.1.2` and
`master212-next`. However, it may be useful to keep `master-2.1.2`
around indefinitely for reference.
4. Assuming that a hotfix is also released for the latest
version in parallel with this one, or if the issue is
already fixed in the latest version, do no do any
reverse merges. However, if it is not, it probably makes
sense to reverse merge `master-2.1.2` into `master`,
release a hotfix for _that_ version, then reverse merge
from `master` to `develop`. (Please don't do this unless absolutely
necessary.)
2. Work continues as above, except using `master-2.1.2`as the base branch for any merging, packaging, etc.
3. After the release is tagged and packages are built, you could potentially delete both branches, e.g. `master-2.1.2` and `master212-next`. However, it may be useful to keep `master-2.1.2` around indefinitely for reference.
4. Assuming that a hotfix is also released for the latest version in parallel with this one, or if the issue is already fixed in the latest version, do no do any reverse merges. However, if it is not, it probably makes sense to reverse merge `master-2.1.2` into `master`, release a hotfix for _that_ version, then reverse merge from `master` to `develop`. (Please don't do this unless absolutely necessary.)
[contrib]: https://docs.github.com/en/get-started/quickstart/contributing-to-projects
[squash]: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/incorporating-changes-from-a-pull-request/about-pull-request-merges#squash-and-merge-your-commits

View File

@@ -1,16 +1,7 @@
ISC License
Copyright (c) 2011, Arthur Britto, David Schwartz, Jed McCaleb, Vinnie Falco, Bob Way, Eric Lombrozo, Nikolaos D. Bougalis, Howard Hinnant.
Copyright (c) 2012-present, the XRP Ledger developers.
Copyright (c) 2011, Arthur Britto, David Schwartz, Jed McCaleb, Vinnie Falco, Bob Way, Eric Lombrozo, Nikolaos D. Bougalis, Howard Hinnant. Copyright (c) 2012-present, the XRP Ledger developers.
Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
Permission to use, copy, modify, and distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

View File

@@ -54,9 +54,9 @@ Here are some good places to start learning the source code:
| `./docs` | Source documentation files and doxygen config. |
| `./cfg` | Example configuration files. |
| `./src` | Source code. |
| `./crates` | Rust source code. |
Some of the directories under `src` are external repositories included using
git-subtree. See those directories' README files for more details.
Some of the directories under `src` are external repositories included using git-subtree. See those directories' README files for more details.
## Additional Documentation

111
bin/check-nix-store-refs.sh Executable file
View File

@@ -0,0 +1,111 @@
#!/usr/bin/env bash
# Fail if a binary under <path> records a /nix/store path it resolves at run
# time. See docs/build/nix.md#prebuilt-packages for why that matters.
#
# <path> is a file or a directory. macOS: nothing may reference the store, so
# point it at whole trees. Linux: the toolchain always writes the store into
# PT_INTERP and RUNPATH, so only at what cmake/PatchNixBinary.cmake retargets.
#
# Only Mach-O / ELF is inspected. Static archives hold store paths in debug info
# alone; the scripts in a Conan cache are all git hook samples and autotools
# scratch, 36 false positives to 0 real.
#
# Usage: bin/check-nix-store-refs.sh <path>
set -euo pipefail
if [ "$#" -ne 1 ]; then
echo "usage: $0 <path>" >&2
exit 2
fi
if [ ! -e "$1" ]; then
echo "$0: no such path: $1" >&2
exit 2
fi
case "$(uname -s)" in
Darwin)
format=Mach-O
recorded_paths=macho_recorded_paths
tool=otool
;;
Linux)
format=ELF
recorded_paths=elf_recorded_paths
tool=readelf
;;
*)
echo "Unsupported OS - skipping the Nix store reference check."
exit 0
;;
esac
# `pipefail` would catch this too, but only as a bare nonzero exit.
if ! command -v "${tool}" >/dev/null; then
echo "$0: ${tool} not found; cannot inspect binaries" >&2
exit 2
fi
# Both list what the file records. `ldd` would answer what this machine resolves
# now, which is wrong both ways: store paths for a correctly patched binary,
# silence for a store RUNPATH that resolves nowhere.
# `name` covers LC_ID_DYLIB and LC_LOAD*_DYLIB, `path` covers LC_RPATH.
macho_recorded_paths() {
otool -l "$1" | sed -nE 's#^ *(name|path) ([^ ]*).*#\2#p'
}
# RPATH and RUNPATH are colon-separated.
elf_recorded_paths() {
readelf -ldW "$1" |
sed -nE \
-e 's#.*program interpreter: ([^]]*)\].*#\1#p' \
-e 's#.*\((RPATH|RUNPATH|NEEDED)\).*\[([^]]*)\].*#\2#p' |
tr ':' '\n'
}
checked=0
skipped=0
leaked=0
while IFS= read -r file; do
case "$(file -b "${file}" 2>/dev/null)" in
*"${format}"*) ;;
*)
skipped=$((skipped + 1))
continue
;;
esac
checked=$((checked + 1))
# Filter after extracting, or a search path starting elsewhere ($ORIGIN)
# hides the rest. `sed` not `grep`: grep calls "no matches" a failure, and
# the `|| true` that would need masks a broken pipeline too.
refs="$("${recorded_paths}" "${file}" | sed -n '\#^/nix/store/#p' | sort -u)"
if [ -n "${refs}" ]; then
leaked=$((leaked + 1))
echo "::error file=${file}::references the Nix store at run time"
echo "${file}"
echo "${refs}" | sed 's/^/ /'
fi
done < <(find "$1" -type f \( -perm -u+x -o -name '*.dylib' -o -name '*.so*' \))
echo "$1: checked ${checked}, skipped ${skipped}, ${leaked} with Nix store references."
if [ "${leaked}" -ne 0 ]; then
cat >&2 <<'EOF'
Fixes, in order of preference:
- A Conan package built before this check existed: drop it
(`conan remove '<name>/*'`) and rebuild.
- A binary that should have been retargeted to the system loader: check that
cmake/PatchNixBinary.cmake ran for it.
- Link the macOS system library instead of the Nix one - see
libresolvSystemStub in nix/darwin.nix.
- No system library exists (libstdc++): link it statically.
- None of the above: pin the toolchain into the package ID, following
`user.package:libc_version` in conan/profiles/ci.
EOF
exit 1
fi

View File

@@ -28,29 +28,31 @@ esac
# - debhelper and dpkg-dev build the DEB
# - rpm-build builds the RPM, with systemd-rpm-macros and redhat-rpm-config
# supplying the systemd and find-debuginfo macros the spec uses
# - git gives build_pkg.sh a real history to read SOURCE_DATE_EPOCH from;
# without one the timestamp falls back to the wall clock
# - curl uploads the finished packages in publish_pkg.sh
# - ca-certificates lets curl and git verify TLS
# - rpm-sign and gnupg2 sign the built RPM
# - python3 runs the packaging scripts
# - git gives build_pkg.py the commit timestamp it stamps files with
# - ca-certificates lets git and the packaging scripts verify TLS
function install() {
case "${ID}" in
debian | ubuntu)
apt-get update -y
apt-get install -y --no-install-recommends \
ca-certificates \
curl \
debhelper \
debhelper-compat \
dpkg-dev \
git
git \
python3
;;
rhel | centos | rocky | almalinux)
dnf install -y --setopt=install_weak_deps=False \
curl-minimal \
git \
rpm-build \
gnupg2 \
python3 \
redhat-rpm-config \
rpm-build \
rpm-sign \
systemd-rpm-macros
;;
esac

View File

@@ -1094,8 +1094,8 @@
# Default is 100.
#
# back_off_milliseconds
# Number of milliseconds to wait between
# online_delete batches to allow other functions
# Number of milliseconds to wait between online_delete
# SQL deletion batches to allow other functions
# to catch up.
# Default is 100.
#
@@ -1109,10 +1109,22 @@
# The online delete process checks periodically
# that xrpld is still in sync with the network,
# and that the validated ledger is less than
# 'age_threshold_seconds' old. If not, then continue
# 'age_threshold_seconds' old, and that all
# recent ledgers are available. If not, then continue
# sleeping for this number of seconds and
# checking until healthy.
# Default is 5.
# Default is 2.
#
# max_waiting_ledgers
# The maximum number of ledgers that may be validated
# while online deletion is waiting for the node to get
# fully synced with the rest of the network. If more than
# this number of ledgers are validated while waiting, then
# online deletion gives up on the current ledger and tries
# again later. Note this only affects situations that cause
# rotation to wait, such as going out of sync, or missing
# ledgers. Forward progress is not penalized. Minimum is 64.
# Default is the online_delete value.
#
# Notes:
# The 'node_db' entry configures the primary, persistent storage.

View File

@@ -120,7 +120,10 @@ if(MSVC)
_SILENCE_ALL_CXX17_DEPRECATION_WARNINGS
$<$<AND:$<COMPILE_LANGUAGE:CXX>,$<CONFIG:Debug>>:_CRTDBG_MAP_ALLOC>
)
target_link_libraries(common INTERFACE -errorreport:none -machine:X64)
target_link_libraries(
common
INTERFACE -errorreport:none -machine:X64 -ignore:4099
)
else()
target_compile_options(
common

View File

@@ -51,6 +51,8 @@ target_compile_options(
target_link_libraries(xrpl.libpb PUBLIC protobuf::libprotobuf gRPC::grpc++)
add_dependencies(tidy_prerequisites xrpl.libpb)
# TODO: Clean up the number of library targets later.
add_library(xrpl.imports.main INTERFACE)

View File

@@ -1,7 +1,7 @@
#[===================================================================[
Linux packaging support: 'package' target.
The packaging script (package/build_pkg.sh) installs to FHS-standard
The packaging script (package/build_pkg.py) installs to FHS-standard
paths (/usr/bin, /etc/xrpld, etc.) regardless of CMAKE_INSTALL_PREFIX,
so no prefix guard is needed here.
#]===================================================================]
@@ -38,19 +38,20 @@ if(NOT TARGET validator-keys)
return()
endif()
set(package_env
SRC_DIR=${CMAKE_SOURCE_DIR}
BUILD_DIR=${CMAKE_BINARY_DIR}
PKG_RELEASE=${pkg_release}
)
if(DPKG_BUILDPACKAGE_EXECUTABLE)
set(pkg_type deb)
else()
set(pkg_type rpm)
endif()
add_custom_target(
package
COMMAND
${CMAKE_COMMAND} -E env ${package_env}
${CMAKE_SOURCE_DIR}/package/build_pkg.sh
${CMAKE_SOURCE_DIR}/package/build_pkg.py --package-type=${pkg_type}
--build-dir=${CMAKE_BINARY_DIR} --pkg-release=${pkg_release}
--channel=UNRELEASED
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
DEPENDS xrpld validator-keys
COMMENT "Building Linux package (deb/rpm inferred from host tooling)"
COMMENT "Building Linux ${pkg_type} package"
VERBATIM
)

View File

@@ -32,6 +32,11 @@ endif()
option(benchmark "Build benchmarks" ON)
# When OFF, the crates directory is not added to the build at all: no Rust
# toolchain is required, no cxxbridge bindings are generated, and the C++ tests
# that consume those bindings are left out of the build tree.
option(rust "Build the Rust crates and the C++ code that depends on them" OFF)
# Enabled by default so every header is compiled on its own as the main file of
# its own compile_commands.json entry - this is what lets clang-tidy (and clangd
# and IDEs) analyse a header's own includes directly. The per-header objects are

View File

@@ -8,6 +8,7 @@ Uses pcpp to preprocess the macro file and pyparsing to parse the DSL.
import io
import argparse
import re
from pathlib import Path
import pyparsing as pp
@@ -53,28 +54,89 @@ def create_transaction_parser():
return macro_parser
# Defaults for xrpl::TxSettings members, mirroring
# include/xrpl/protocol/TxSettings.h. A transaction's settings blob only names
# the members that differ from these.
SETTING_DEFAULTS = {
"delegable": "Delegation::NotDelegable",
"amendment": "uint256{}",
"privileges": "Privilege::NoPriv",
}
def parse_settings(settings_str):
"""Parse a TxSettings blob into a dict, filling in defaults.
Args:
settings_str: A string like '({.delegable = Delegation::NotDelegable,
.privileges = Privilege::CreateAcct})', or '({})'.
Returns:
A dict with a value for every key in SETTING_DEFAULTS.
"""
body = settings_str.strip()
if not (body.startswith("(") and body.endswith(")")):
raise ValueError(
f"Malformed settings blob, expected '({{...}})': {settings_str!r}"
)
body = body[1:-1].strip()
if not (body.startswith("{") and body.endswith("}")):
raise ValueError(
f"Malformed settings blob, expected '({{...}})': {settings_str!r}"
)
body = body[1:-1]
# Strip comments, which may be interleaved with the designated initializers.
body = re.sub(r"//[^\n]*", "", body)
settings = dict(SETTING_DEFAULTS)
seen = set()
# Each entry runs from '.key =' up to the next '.key =' or the end.
for key, value in re.findall(
r"\.(\w+)\s*=\s*(.*?)(?=,\s*\.\w+\s*=|,?\s*$)", body, re.S
):
if key not in SETTING_DEFAULTS:
raise ValueError(f"Unknown TxSettings member '.{key}' in {settings_str!r}")
settings[key] = " ".join(value.split()).rstrip(",")
seen.add(key)
# Catch a typo'd or unparsed initializer rather than silently defaulting it.
# Every '.member' in the blob must have been consumed above.
if len(re.findall(r"\.\w+", body)) != len(seen):
raise ValueError(f"Could not parse every setting in {settings_str!r}")
# A blob with content but no designated initializer is positional, which
# would otherwise be read as "all defaults" and silently generate the
# wrong output.
if body.strip() and not seen:
raise ValueError(
"TxSettings requires designated initializers (.member = value), "
f"got {settings_str!r}"
)
return settings
def parse_transaction_args(args_list):
"""Parse the arguments of a TRANSACTION macro call.
Args:
args_list: A list of parsed arguments from pyparsing, e.g.,
['ttPAYMENT', '0', 'Payment', 'Delegation::delegable',
'uint256{}', 'createAcct', '({...})']
['ttPAYMENT', '0', 'Payment',
'({.privileges = Privilege::CreateAcct})', '({...})']
Returns:
A dict with parsed transaction information.
"""
if len(args_list) < 7:
if len(args_list) < 5:
raise ValueError(
f"Expected at least 7 parts in TRANSACTION, got {len(args_list)}: {args_list}"
f"Expected at least 5 parts in TRANSACTION, got {len(args_list)}: {args_list}"
)
tag = args_list[0]
value = args_list[1]
name = args_list[2]
delegable = args_list[3]
amendments = args_list[4]
privileges = args_list[5]
settings = parse_settings(args_list[3])
fields_str = args_list[-1]
# Parse fields: ({field1, field2, ...})
@@ -84,9 +146,9 @@ def parse_transaction_args(args_list):
"tag": tag,
"value": value,
"name": name,
"delegable": delegable,
"amendments": amendments,
"privileges": privileges,
"delegable": settings["delegable"],
"amendments": settings["amendment"],
"privileges": settings["privileges"],
"fields": fields,
}

View File

@@ -23,6 +23,7 @@
"fast_float/8.2.10#f6f28d6bb22112078e7dbda611caf681%1782494504.298",
"ed25519/2015.03#ae761bdc52730a843f0809bdf6c1b1f6%1782307148.15562",
"date/3.0.4#862e11e80030356b53c2c38599ceb32b%1782392402.538492",
"corrosion/0.6.1#bfa292df0a957bc70a450ff316cd9435%1786119416.131296",
"c-ares/1.34.6#545240bb1c40e2cacd4362d6b8967650%1782392402.681654",
"bzip2/1.0.8#c470882369c2d95c5c77e970c0c7e321%1782392402.296732",
"boost/1.91.0#ea540ca2133d831b560036aa24dece3c%1782392419.475605",

View File

@@ -2,11 +2,8 @@
To achieve reproducible dependencies, we use a [Conan lockfile](https://docs.conan.io/2/tutorial/versioning/lockfiles.html).
The `conan.lock` file in the repository contains a "snapshot" of the current
dependencies. It is implicitly used when running `conan` commands, so you don't
need to specify it.
The `conan.lock` file in the repository contains a "snapshot" of the current dependencies. It is implicitly used when running `conan` commands, so you don't need to specify it.
You have to update this file every time you add a new dependency or change a
revision or version of an existing dependency.
You have to update this file every time you add a new dependency or change a revision or version of an existing dependency.
To update a lockfile, run from the repository root: `./conan/lockfile/regenerate.sh`

View File

@@ -28,6 +28,7 @@ class Xrpl(ConanFile):
}
requires = [
"corrosion/0.6.1",
"ed25519/2015.03",
"fast_float/8.2.10",
"grpc/1.81.1",

17
crates/.cargo/config.toml Normal file
View File

@@ -0,0 +1,17 @@
# The Rust static libraries are linked into C++ targets, so the runtime linkage
# here has to match what the C++ build uses (see cmake/XrplCompiler.cmake).
#
# macOS needs nothing: AppleClang cannot link libgcc/libc++ statically, so the
# C++ build skips those flags on Apple as well.
# Both amd64 and arm64 Linux builds link libgcc statically. This only affects
# links that rustc itself drives (`cargo test` binaries and the like) — the
# `staticlib` crates consumed by CMake are archived, not linked, so rustc
# silently ignores link args for them. Keeping libgcc_s.so.1 off the xrpld link
# line is handled in crates/CMakeLists.txt instead.
[target.'cfg(target_os = "linux")']
rustflags = ["-C", "link-args=-static-libgcc"]
# Windows builds use the static MSVC runtime.
[target.'cfg(windows)']
rustflags = ["-C", "target-feature=+crt-static"]

104
crates/CMakeLists.txt Normal file
View File

@@ -0,0 +1,104 @@
find_package(Corrosion REQUIRED)
corrosion_import_crate(MANIFEST_PATH ${CMAKE_CURRENT_SOURCE_DIR}/Cargo.toml)
# The generated C++ lands in the build tree, so put a .clang-tidy next to it to
# keep clang-tidy from analyzing code we don't own.
configure_file(
generated.clang-tidy
"${CMAKE_CURRENT_BINARY_DIR}/.clang-tidy"
COPYONLY
)
add_custom_target(xrpl_crates)
add_dependencies(tidy_prerequisites xrpl_crates)
# On macOS, ld warns `ignoring duplicate libraries` when linking a crate.
# Corrosion is the source of both duplicates it names:
#
# * The crate archive and its cxxbridge archive, because
# `corrosion_add_cxxbridge` makes the two depend on each other, and CMake
# repeats a static library cycle on the link line so single-pass linkers can
# resolve it. (LINK_INTERFACE_MULTIPLICITY can only raise that count.)
# * `-lSystem`, which Corrosion copies from rustc's `native-static-libs` even
# though the compiler driver always links libSystem.
#
# ld needs neither: it resolves the cycle from one copy of each archive and
# links libSystem once. So silence the warning rather than rewrite Corrosion's
# link interface, which the cycle is also part of. The option itself is old —
# Xcode 15 is only where the warning became the default — and the check below
# leaves it out on a linker that does not know it.
if(is_macos)
include(CheckLinkerFlag)
check_linker_flag(
CXX
-Wl,-no_warn_duplicate_libraries
have_no_warn_duplicate_libraries
)
endif()
function(_unlink_libgcc_s crate)
if(NOT (is_linux AND static))
return()
endif()
# Corrosion exposes a crate's staticlib as an imported `<crate>-static`
# target and puts the native libs in its INTERFACE_LINK_LIBRARIES. If either
# of those changes, warn instead of silently letting libgcc_s.so.1 return.
set(imported "${crate}-static")
if(NOT TARGET ${imported})
message(
FATAL_ERROR
"Corrosion did not create the imported target '${imported}', so "
"libgcc_s cannot be removed from the link interface of '${crate}'. "
"xrpld will link libgcc_s.so.1 dynamically. Check where Corrosion "
"${CORROSION_VERSION} now records `native-static-libs`."
)
return()
endif()
get_target_property(libs ${imported} INTERFACE_LINK_LIBRARIES)
if(NOT "gcc_s" IN_LIST libs)
message(
WARNING
"'gcc_s' was not in the link interface of '${imported}' as "
"expected. If the Rust toolchain stopped reporting it this "
"workaround is obsolete and can be deleted; otherwise xrpld may "
"link libgcc_s.so.1 dynamically. Verify with: "
"objdump -p xrpld | grep NEEDED"
)
return()
endif()
list(REMOVE_ITEM libs gcc_s)
set_property(TARGET ${imported} PROPERTY INTERFACE_LINK_LIBRARIES ${libs})
endfunction()
function(add_xrpl_crate name)
cmake_parse_arguments(ARG "" "CRATE" "FILES" ${ARGN})
_unlink_libgcc_s(${ARG_CRATE})
# `cc` picks its runtime flag from `crt-static` alone, so it compiles a
# crate's C++ with `-MT`; Debug needs `-MTd` (to match cmake/XrplCompiler.cmake).
if(is_msvc)
corrosion_set_env_vars(
${ARG_CRATE}
"$<$<CONFIG:Debug>:CXXFLAGS=-MTd>"
)
endif()
corrosion_add_cxxbridge(${name}_cxxbridge CRATE ${ARG_CRATE} FILES
${ARG_FILES}
)
# Generated cxxbridge headers don't exist at configure time; CMake 3.28+
# validates INTERFACE_SOURCES on consuming targets. Clear it to skip the
# existence check — build-time ordering is enforced by the custom commands.
set_target_properties(${name}_cxxbridge PROPERTIES INTERFACE_SOURCES "")
if(have_no_warn_duplicate_libraries)
target_link_options(
${name}_cxxbridge
INTERFACE -Wl,-no_warn_duplicate_libraries
)
endif()
add_dependencies(xrpl_crates ${name}_cxxbridge)
endfunction()
add_xrpl_crate(rs_hello_world CRATE rs_hello_world FILES lib.rs)

301
crates/Cargo.lock generated Normal file
View File

@@ -0,0 +1,301 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "anstyle"
version = "1.0.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000"
[[package]]
name = "cc"
version = "1.2.61"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d16d90359e986641506914ba71350897565610e87ce0ad9e6f28569db3dd5c6d"
dependencies = [
"find-msvc-tools",
"shlex",
]
[[package]]
name = "clap"
version = "4.6.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1ddb117e43bbf7dacf0a4190fef4d345b9bad68dfc649cb349e7d17d28428e51"
dependencies = [
"clap_builder",
]
[[package]]
name = "clap_builder"
version = "4.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "714a53001bf66416adb0e2ef5ac857140e7dc3a0c48fb28b2f10762fc4b5069f"
dependencies = [
"anstyle",
"clap_lex",
"strsim",
]
[[package]]
name = "clap_lex"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9"
[[package]]
name = "codespan-reporting"
version = "0.13.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "af491d569909a7e4dee0ad7db7f5341fef5c614d5b8ec8cf765732aba3cff681"
dependencies = [
"serde",
"termcolor",
"unicode-width",
]
[[package]]
name = "cxx"
version = "1.0.199"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "824894a4a85dca76d4c95c2b9098c036f5a29f627b30c12780774f6654e60974"
dependencies = [
"cc",
"cxx-build",
"cxxbridge-cmd",
"cxxbridge-flags",
"cxxbridge-macro",
"foldhash",
"link-cplusplus",
]
[[package]]
name = "cxx-build"
version = "1.0.199"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f1ae0b651ea5b0000b19513aef5a03f194d7e3486f2d9258b658da8677fe9036"
dependencies = [
"cc",
"codespan-reporting",
"indexmap",
"proc-macro2",
"quote",
"scratch",
"syn 3.0.3",
]
[[package]]
name = "cxxbridge-cmd"
version = "1.0.199"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "fb05f91d3fb8435d9bab6ac5ce6ac1868be774325fb7fb2a91be39393b21388e"
dependencies = [
"clap",
"codespan-reporting",
"indexmap",
"proc-macro2",
"quote",
"syn 3.0.3",
]
[[package]]
name = "cxxbridge-flags"
version = "1.0.199"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bf293202e0e3e98495785745389e8d0755b217e66f19194a5c695c25e03282ef"
[[package]]
name = "cxxbridge-macro"
version = "1.0.199"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ca001d746947c7249ed9d332a10f7a59daedbafeb0ec68c5c18a7db7a93f6ccc"
dependencies = [
"indexmap",
"proc-macro2",
"quote",
"syn 3.0.3",
]
[[package]]
name = "equivalent"
version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f"
[[package]]
name = "find-msvc-tools"
version = "0.1.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582"
[[package]]
name = "foldhash"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb"
[[package]]
name = "hashbrown"
version = "0.17.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4f467dd6dccf739c208452f8014c75c18bb8301b050ad1cfb27153803edb0f51"
[[package]]
name = "indexmap"
version = "2.14.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d466e9454f08e4a911e14806c24e16fba1b4c121d1ea474396f396069cf949d9"
dependencies = [
"equivalent",
"hashbrown",
]
[[package]]
name = "link-cplusplus"
version = "1.0.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7f78c730aaa7d0b9336a299029ea49f9ee53b0ed06e9202e8cb7db9bae7b8c82"
dependencies = [
"cc",
]
[[package]]
name = "proc-macro2"
version = "1.0.106"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8fd00f0bb2e90d81d1044c2b32617f68fcb9fa3bb7640c23e9c748e53fb30934"
dependencies = [
"unicode-ident",
]
[[package]]
name = "quote"
version = "1.0.45"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924"
dependencies = [
"proc-macro2",
]
[[package]]
name = "rs-hello_world"
version = "0.1.0"
dependencies = [
"cxx",
]
[[package]]
name = "scratch"
version = "1.0.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d68f2ec51b097e4c1a75b681a8bec621909b5e91f15bb7b840c4f2f7b01148b2"
[[package]]
name = "serde"
version = "1.0.228"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e"
dependencies = [
"serde_core",
"serde_derive",
]
[[package]]
name = "serde_core"
version = "1.0.228"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad"
dependencies = [
"serde_derive",
]
[[package]]
name = "serde_derive"
version = "1.0.228"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.117",
]
[[package]]
name = "shlex"
version = "1.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0fda2ff0d084019ba4d7c6f371c95d8fd75ce3524c3cb8fb653a3023f6323e64"
[[package]]
name = "strsim"
version = "0.11.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7da8b5736845d9f2fcb837ea5d9e2628564b3b043a70948a3f0b778838c5fb4f"
[[package]]
name = "syn"
version = "2.0.117"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "syn"
version = "3.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3"
dependencies = [
"proc-macro2",
"quote",
"unicode-ident",
]
[[package]]
name = "termcolor"
version = "1.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "06794f8f6c5c898b3275aebefa6b8a1cb24cd2c6c79397ab15774837a0bc5755"
dependencies = [
"winapi-util",
]
[[package]]
name = "unicode-ident"
version = "1.0.24"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
[[package]]
name = "unicode-width"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254"
[[package]]
name = "winapi-util"
version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
dependencies = [
"windows-sys",
]
[[package]]
name = "windows-link"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
[[package]]
name = "windows-sys"
version = "0.61.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc"
dependencies = [
"windows-link",
]

15
crates/Cargo.toml Normal file
View File

@@ -0,0 +1,15 @@
[workspace]
members = ["hello_world"]
resolver = "3"
[workspace.dependencies]
cxx = { version = "1.0.198", features = ["c++20"] }
[workspace.package]
edition = "2024"
[profile.release]
opt-level = 3
overflow-checks = true
lto = true
debug = true

View File

@@ -0,0 +1,10 @@
---
# Neutralizes clang-tidy for the corrosion/cxxbridge-generated C++. Copied into
# the crates build directory by crates/CMakeLists.txt, next to the generated
# sources, so clang-tidy picks it up instead of the top-level configuration.
#
# One check is kept enabled to avoid clang-tidy's "no checks enabled" error.
Checks: "-*,google-readability-todo"
WarningsAsErrors: ""
HeaderFilterRegex: ""
InheritParentConfig: false

View File

@@ -0,0 +1,10 @@
[package]
name = "rs-hello_world"
version = "0.1.0"
edition.workspace = true
[lib]
crate-type = ["staticlib"]
[dependencies]
cxx.workspace = true

View File

@@ -0,0 +1,10 @@
#[cxx::bridge(namespace = "rs::hello_world")]
mod ffi {
extern "Rust" {
fn hello_world() -> String;
}
}
pub fn hello_world() -> String {
"hello_world".to_string()
}

View File

@@ -2,61 +2,21 @@
## The Problem Statement
The moment-to-moment health of the XRP Ledger network depends on the health and
connectivity of a small number of computers (nodes). The most important nodes
are validators, specifically ones listed on the unique node list
([UNL](#Question-What-are-UNLs)). Ripple publishes a recommended UNL that most
network nodes use to determine which peers in the network are trusted. Although
most validators use the same list, they are not required to. The XRP Ledger
network progresses to the next ledger when enough validators reach agreement
(above the minimum quorum of 80%) about what transactions to include in the next
ledger.
The moment-to-moment health of the XRP Ledger network depends on the health and connectivity of a small number of computers (nodes). The most important nodes are validators, specifically ones listed on the unique node list ([UNL](#Question-What-are-UNLs)). Ripple publishes a recommended UNL that most network nodes use to determine which peers in the network are trusted. Although most validators use the same list, they are not required to. The XRP Ledger network progresses to the next ledger when enough validators reach agreement (above the minimum quorum of 80%) about what transactions to include in the next ledger.
As an example, if there are 10 validators on the UNL, at least 8 validators have
to agree with the latest ledger for it to become validated. But what if enough
of those validators are offline to drop the network below the 80% quorum? The
XRP Ledger network favors safety/correctness over advancing the ledger. Which
means if enough validators are offline, the network will not be able to validate
ledgers.
As an example, if there are 10 validators on the UNL, at least 8 validators have to agree with the latest ledger for it to become validated. But what if enough of those validators are offline to drop the network below the 80% quorum? The XRP Ledger network favors safety/correctness over advancing the ledger. Which means if enough validators are offline, the network will not be able to validate ledgers.
Unfortunately validators can go offline at any time for many different reasons.
Power outages, network connectivity issues, and hardware failures are just a few
scenarios where a validator would appear "offline". Given that most of these
events are temporary, it would make sense to temporarily remove that validator
from the UNL. But the UNL is updated infrequently and not every node uses the
same UNL. So instead of removing the unreliable validator from the Ripple
recommended UNL, we can create a second negative UNL which is stored directly on
the ledger (so the entire network has the same view). This will help the network
see which validators are **currently** unreliable, and adjust their quorum
calculation accordingly.
Unfortunately validators can go offline at any time for many different reasons. Power outages, network connectivity issues, and hardware failures are just a few scenarios where a validator would appear "offline". Given that most of these events are temporary, it would make sense to temporarily remove that validator from the UNL. But the UNL is updated infrequently and not every node uses the same UNL. So instead of removing the unreliable validator from the Ripple recommended UNL, we can create a second negative UNL which is stored directly on the ledger (so the entire network has the same view). This will help the network see which validators are **currently** unreliable, and adjust their quorum calculation accordingly.
_Improving the liveness of the network is the main motivation for the negative UNL._
### Targeted Faults
In order to determine which validators are unreliable, we need clearly define
what kind of faults to measure and analyze. We want to deal with the faults we
frequently observe in the production network. Hence we will only monitor for
validators that do not reliably respond to network messages or send out
validations disagreeing with the locally generated validations. We will not
target other byzantine faults.
In order to determine which validators are unreliable, we need clearly define what kind of faults to measure and analyze. We want to deal with the faults we frequently observe in the production network. Hence we will only monitor for validators that do not reliably respond to network messages or send out validations disagreeing with the locally generated validations. We will not target other byzantine faults.
To track whether or not a validator is responding to the network, we could
monitor them with a “heartbeat” protocol. Instead of creating a new heartbeat
protocol, we can leverage some existing protocol messages to mimic the
heartbeat. We picked validation messages because validators should send one and
only one validation message per ledger. In addition, we only count the
validation messages that agree with the local node's validations.
To track whether or not a validator is responding to the network, we could monitor them with a “heartbeat” protocol. Instead of creating a new heartbeat protocol, we can leverage some existing protocol messages to mimic the heartbeat. We picked validation messages because validators should send one and only one validation message per ledger. In addition, we only count the validation messages that agree with the local node's validations.
With the negative UNL, the network could keep making forward progress safely
even if the number of remaining validators gets to 60%. Say we have a network
with 10 validators on the UNL and everything is operating correctly. The quorum
required for this network would be 8 (80% of 10). When validators fail, the
quorum required would be as low as 6 (60% of 10), which is the absolute
**_minimum quorum_**. We need the absolute minimum quorum to be strictly greater
than 50% of the original UNL so that there cannot be two partitions of
well-behaved nodes headed in different directions. We arbitrarily choose 60% as
the minimum quorum to give a margin of safety.
With the negative UNL, the network could keep making forward progress safely even if the number of remaining validators gets to 60%. Say we have a network with 10 validators on the UNL and everything is operating correctly. The quorum required for this network would be 8 (80% of 10). When validators fail, the quorum required would be as low as 6 (60% of 10), which is the absolute **_minimum quorum_**. We need the absolute minimum quorum to be strictly greater than 50% of the original UNL so that there cannot be two partitions of well-behaved nodes headed in different directions. We arbitrarily choose 60% as the minimum quorum to give a margin of safety.
Consider these events in the absence of negative UNL:
@@ -65,10 +25,7 @@ Consider these events in the absence of negative UNL:
1. 5:00pm - validator3 fails, votes vs. quorum: 7 < 8, we dont have quorum
- **network cannot validate new ledgers with 3 failed validators**
We're below 80% agreement, so new ledgers cannot be validated. This is how the
XRP Ledger operates today, but if the negative UNL was enabled, the events would
happen as follows. (Please note that the events below are from a simplified
version of our protocol.)
We're below 80% agreement, so new ledgers cannot be validated. This is how the XRP Ledger operates today, but if the negative UNL was enabled, the events would happen as follows. (Please note that the events below are from a simplified version of our protocol.)
1. 1:00pm - validator1 fails, votes vs. quorum: 9 >= 8, we have quorum
1. 1:40pm - network adds validator1 to negative UNL, quorum changes to ceil(9 \* 0.8), or 8
@@ -88,8 +45,7 @@ This proposal will:
1. add a new pseudo-transaction type
1. add the negative UNL to the ledger data structure.
Any tools or systems that rely on the format of this data will have to be
updated.
Any tools or systems that rely on the format of this data will have to be updated.
### Amendment
@@ -105,122 +61,70 @@ This section discusses the following topics about the Negative UNL design:
- [Negative UNL maintenance](#Negative-UNL-Maintenance)
- [Quorum size calculation](#Quorum-Size-Calculation)
- [Filter validation messages](#Filter-Validation-Messages)
- [High level sequence diagram of code
changes](#High-Level-Sequence-Diagram-of-Code-Changes)
- [High level sequence diagram of code changes](#High-Level-Sequence-Diagram-of-Code-Changes)
### Negative UNL Protocol Overview
Every ledger stores a list of zero or more unreliable validators. Updates to the
list must be approved by the validators using the consensus mechanism that
validators use to agree on the set of transactions. The list is used only when
checking if a ledger is fully validated. If a validator V is in the list, nodes
with V in their UNL adjust the quorum and Vs validation message is not counted
when verifying if a ledger is fully validated. Vs flow of messages and network
interactions, however, will remain the same.
Every ledger stores a list of zero or more unreliable validators. Updates to the list must be approved by the validators using the consensus mechanism that validators use to agree on the set of transactions. The list is used only when checking if a ledger is fully validated. If a validator V is in the list, nodes with V in their UNL adjust the quorum and Vs validation message is not counted when verifying if a ledger is fully validated. Vs flow of messages and network interactions, however, will remain the same.
We define the **\*effective UNL** = original UNL - negative UNL\*, and the
**_effective quorum_** as the quorum of the _effective UNL_. And we set
_effective quorum = Ceiling(80% _ effective UNL)\*.
We define the **\*effective UNL** = original UNL - negative UNL\*, and the **_effective quorum_** as the quorum of the _effective UNL_. And we set _effective quorum = Ceiling(80% _ effective UNL)\*.
### Validator Reliability Measurement
A node only measures the reliability of validators on its own UNL, and only
proposes based on local observations. There are many metrics that a node can
measure about its validators, but we have chosen ledger validation messages.
This is because every validator shall send one and only one signed validation
message per ledger. This keeps the measurement simple and removes
timing/clock-sync issues. A node will measure the percentage of agreeing
validation messages (_PAV_) received from each validator on the node's UNL. Note
that the node will only count the validation messages that agree with its own
validations.
A node only measures the reliability of validators on its own UNL, and only proposes based on local observations. There are many metrics that a node can measure about its validators, but we have chosen ledger validation messages. This is because every validator shall send one and only one signed validation message per ledger. This keeps the measurement simple and removes timing/clock-sync issues. A node will measure the percentage of agreeing validation messages (_PAV_) received from each validator on the node's UNL. Note that the node will only count the validation messages that agree with its own validations.
We define the **PAV** as the Percentage of Agreed Validation
messages received for the last N ledgers, where N = 256 by default.
We define the **PAV** as the Percentage of Agreed Validation messages received for the last N ledgers, where N = 256 by default.
When the PAV drops below the **_low-water mark_**, the validator is considered
unreliable, and is a candidate to be disabled by being added to the negative
UNL. A validator must have a PAV higher than the **_high-water mark_** to be
re-enabled. The validator is re-enabled by removing it from the negative UNL. In
the implementation, we plan to set the low-water mark as 50% and the high-water
mark as 80%.
When the PAV drops below the **_low-water mark_**, the validator is considered unreliable, and is a candidate to be disabled by being added to the negative UNL. A validator must have a PAV higher than the **_high-water mark_** to be re-enabled. The validator is re-enabled by removing it from the negative UNL. In the implementation, we plan to set the low-water mark as 50% and the high-water mark as 80%.
### Format Changes
The negative UNL component in a ledger contains three fields.
- **_NegativeUNL_**: The current negative UNL, a list of unreliable validators.
- **_ToDisable_**: The validator to be added to the negative UNL on the next
flag ledger.
- **_ToReEnable_**: The validator to be removed from the negative UNL on the
next flag ledger.
- **_ToDisable_**: The validator to be added to the negative UNL on the next flag ledger.
- **_ToReEnable_**: The validator to be removed from the negative UNL on the next flag ledger.
All three fields are optional. When the _ToReEnable_ field exists, the
_NegativeUNL_ field cannot be empty.
All three fields are optional. When the _ToReEnable_ field exists, the _NegativeUNL_ field cannot be empty.
A new pseudo-transaction, **_UNLModify_**, is added. It has three fields
- **_Disabling_**: A flag indicating whether the modification is to disable or
to re-enable a validator.
- **_Disabling_**: A flag indicating whether the modification is to disable or to re-enable a validator.
- **_Seq_**: The ledger sequence number.
- **_Validator_**: The validator to be disabled or re-enabled.
There would be at most one _disable_ `UNLModify` and one _re-enable_ `UNLModify`
transaction per flag ledger. The full machinery is described further on.
There would be at most one _disable_ `UNLModify` and one _re-enable_ `UNLModify` transaction per flag ledger. The full machinery is described further on.
### Negative UNL Maintenance
The negative UNL can only be modified on the flag ledgers. If a validator's
reliability status changes, it takes two flag ledgers to modify the negative
UNL. Let's see an example of the algorithm:
The negative UNL can only be modified on the flag ledgers. If a validator's reliability status changes, it takes two flag ledgers to modify the negative UNL. Let's see an example of the algorithm:
- Ledger seq = 100: A validator V goes offline.
- Ledger seq = 256: This is a flag ledger, and V's reliability measurement _PAV_
is lower than the low-water mark. Other validators add `UNLModify`
pseudo-transactions `{true, 256, V}` to the transaction set which goes through
the consensus. Then the pseudo-transaction is applied to the negative UNL
ledger component by setting `ToDisable = V`.
- Ledger seq = 257 ~ 511: The negative UNL ledger component is copied from the
parent ledger.
- Ledger seq=512: This is a flag ledger, and the negative UNL is updated
`NegativeUNL = NegativeUNL + ToDisable`.
- Ledger seq = 256: This is a flag ledger, and V's reliability measurement _PAV_ is lower than the low-water mark. Other validators add `UNLModify` pseudo-transactions `{true, 256, V}` to the transaction set which goes through the consensus. Then the pseudo-transaction is applied to the negative UNL ledger component by setting `ToDisable = V`.
- Ledger seq = 257 ~ 511: The negative UNL ledger component is copied from the parent ledger.
- Ledger seq=512: This is a flag ledger, and the negative UNL is updated `NegativeUNL = NegativeUNL + ToDisable`.
The negative UNL may have up to `MaxNegativeListed = floor(original UNL * 25%)`
validators. The 25% is because of 75% \* 80% = 60%, where 75% = 100% - 25%, 80%
is the quorum of the effective UNL, and 60% is the absolute minimum quorum of
the original UNL. Adding more than 25% validators to the negative UNL does not
improve the liveness of the network, because adding more validators to the
negative UNL cannot lower the effective quorum.
The negative UNL may have up to `MaxNegativeListed = floor(original UNL * 25%)` validators. The 25% is because of 75% \* 80% = 60%, where 75% = 100% - 25%, 80% is the quorum of the effective UNL, and 60% is the absolute minimum quorum of the original UNL. Adding more than 25% validators to the negative UNL does not improve the liveness of the network, because adding more validators to the negative UNL cannot lower the effective quorum.
The following is the detailed algorithm:
- **If** the ledger seq = x is a flag ledger
1. Compute `NegativeUNL = NegativeUNL + ToDisable - ToReEnable` if they
exist in the parent ledger
1. Compute `NegativeUNL = NegativeUNL + ToDisable - ToReEnable` if they exist in the parent ledger
1. Try to find a candidate to disable if `sizeof NegativeUNL < MaxNegativeListed`
1. Find a validator V that has a _PAV_ lower than the low-water
mark, but is not in `NegativeUNL`.
1. Find a validator V that has a _PAV_ lower than the low-water mark, but is not in `NegativeUNL`.
1. If two or more are found, their public keys are XORed with the hash
of the parent ledger and the one with the lowest XOR result is chosen.
1. If V is found, create a `UNLModify` pseudo-transaction
`TxDisableValidator = {true, x, V}`
1. If two or more are found, their public keys are XORed with the hash of the parent ledger and the one with the lowest XOR result is chosen.
1. If V is found, create a `UNLModify` pseudo-transaction `TxDisableValidator = {true, x, V}`
1. Try to find a candidate to re-enable if `sizeof NegativeUNL > 0`:
1. Find a validator U that is in `NegativeUNL` and has a _PAV_ higher
than the high-water mark.
1. If U is not found, try to find one in `NegativeUNL` but not in the
local _UNL_.
1. If two or more are found, their public keys are XORed with the hash
of the parent ledger and the one with the lowest XOR result is chosen.
1. If U is found, create a `UNLModify` pseudo-transaction
`TxReEnableValidator = {false, x, U}`
1. Find a validator U that is in `NegativeUNL` and has a _PAV_ higher than the high-water mark.
1. If U is not found, try to find one in `NegativeUNL` but not in the local _UNL_.
1. If two or more are found, their public keys are XORed with the hash of the parent ledger and the one with the lowest XOR result is chosen.
1. If U is found, create a `UNLModify` pseudo-transaction `TxReEnableValidator = {false, x, U}`
1. If any `UNLModify` pseudo-transactions are created, add them to the
transaction set. The transaction set goes through the consensus algorithm.
1. If have enough support, the `UNLModify` pseudo-transactions remain in the
transaction set agreed by the validators. Then the pseudo-transactions are
applied to the ledger:
1. If any `UNLModify` pseudo-transactions are created, add them to the transaction set. The transaction set goes through the consensus algorithm.
1. If have enough support, the `UNLModify` pseudo-transactions remain in the transaction set agreed by the validators. Then the pseudo-transactions are applied to the ledger:
1. If have `TxDisableValidator`, set `ToDisable=TxDisableValidator.V`.
Else clear `ToDisable`.
@@ -231,61 +135,33 @@ The following is the detailed algorithm:
- **Else** (not a flag ledger)
1. Copy the negative UNL ledger component from the parent ledger
The negative UNL is stored on each ledger because we don't know when a validator
may reconnect to the network. If the negative UNL was stored only on every flag
ledger, then a new validator would have to wait until it acquires the latest
flag ledger to know the negative UNL. So any new ledgers created that are not
flag ledgers copy the negative UNL from the parent ledger.
The negative UNL is stored on each ledger because we don't know when a validator may reconnect to the network. If the negative UNL was stored only on every flag ledger, then a new validator would have to wait until it acquires the latest flag ledger to know the negative UNL. So any new ledgers created that are not flag ledgers copy the negative UNL from the parent ledger.
Note that when we have a validator to disable and a validator to re-enable at
the same flag ledger, we create two separate `UNLModify` pseudo-transactions. We
want either one or the other or both to make it into the ledger on their own
merits.
Note that when we have a validator to disable and a validator to re-enable at the same flag ledger, we create two separate `UNLModify` pseudo-transactions. We want either one or the other or both to make it into the ledger on their own merits.
Readers may have noticed that we defined several rules of creating the
`UNLModify` pseudo-transactions but did not describe how to enforce the rules.
The rules are actually enforced by the existing consensus algorithm. Unless
enough validators propose the same pseudo-transaction it will not be included in
the transaction set of the ledger.
Readers may have noticed that we defined several rules of creating the `UNLModify` pseudo-transactions but did not describe how to enforce the rules. The rules are actually enforced by the existing consensus algorithm. Unless enough validators propose the same pseudo-transaction it will not be included in the transaction set of the ledger.
### Quorum Size Calculation
The effective quorum is 80% of the effective UNL. Note that because at most 25%
of the original UNL can be on the negative UNL, the quorum should not be lower
than the absolute minimum quorum (i.e. 60%) of the original UNL. However,
considering that different nodes may have different UNLs, to be safe we compute
`quorum = Ceiling(max(60% * original UNL, 80% * effective UNL))`.
The effective quorum is 80% of the effective UNL. Note that because at most 25% of the original UNL can be on the negative UNL, the quorum should not be lower than the absolute minimum quorum (i.e. 60%) of the original UNL. However, considering that different nodes may have different UNLs, to be safe we compute `quorum = Ceiling(max(60% * original UNL, 80% * effective UNL))`.
### Filter Validation Messages
If a validator V is in the negative UNL, it still participates in consensus
sessions in the same way, i.e. V still follows the protocol and publishes
proposal and validation messages. The messages from V are still stored the same
way by everyone, used to calculate the new PAV for V, and could be used in
future consensus sessions if needed. However V's ledger validation message is
not counted when checking if the ledger is fully validated.
If a validator V is in the negative UNL, it still participates in consensus sessions in the same way, i.e. V still follows the protocol and publishes proposal and validation messages. The messages from V are still stored the same way by everyone, used to calculate the new PAV for V, and could be used in future consensus sessions if needed. However V's ledger validation message is not counted when checking if the ledger is fully validated.
### High Level Sequence Diagram of Code Changes
The diagram below is the sequence of one round of consensus. Classes and
components with non-trivial changes are colored green.
The diagram below is the sequence of one round of consensus. Classes and components with non-trivial changes are colored green.
- The `ValidatorList` class is modified to compute the quorum of the effective
UNL.
- The `ValidatorList` class is modified to compute the quorum of the effective UNL.
- The `Validations` class provides an interface for querying the validation
messages from trusted validators.
- The `Validations` class provides an interface for querying the validation messages from trusted validators.
- The `ConsensusAdaptor` component:
- The `RCLConsensus::Adaptor` class is modified for creating `UNLModify`
Pseudo-Transactions.
- The `Change` class is modified for applying `UNLModify`
Pseudo-Transactions.
- The `Ledger` class is modified for creating and adjusting the negative UNL
ledger component.
- The `LedgerMaster` class is modified for filtering out validation messages
from negative UNL validators when verifying if a ledger is fully
validated.
- The `RCLConsensus::Adaptor` class is modified for creating `UNLModify` Pseudo-Transactions.
- The `Change` class is modified for applying `UNLModify` Pseudo-Transactions.
- The `Ledger` class is modified for creating and adjusting the negative UNL ledger component.
- The `LedgerMaster` class is modified for filtering out validation messages from negative UNL validators when verifying if a ledger is fully validated.
![Sequence diagram](./negativeUNL_highLevel_sequence.png?raw=true "Negative UNL
Changes")
@@ -294,63 +170,29 @@ Changes")
### Use a Mechanism Like Fee Voting to Process UNLModify Pseudo-Transactions
The previous version of the negative UNL specification used the same mechanism
as the [fee voting](https://xrpl.org/fee-voting.html#voting-process.) for
creating the negative UNL, and used the negative UNL as soon as the ledger was
fully validated. However the timing of fully validation can differ among nodes,
so different negative UNLs could be used, resulting in different effective UNLs
and different quorums for the same ledger. As a result, the network's safety is
impacted.
The previous version of the negative UNL specification used the same mechanism as the [fee voting](https://xrpl.org/fee-voting.html#voting-process.) for creating the negative UNL, and used the negative UNL as soon as the ledger was fully validated. However the timing of fully validation can differ among nodes, so different negative UNLs could be used, resulting in different effective UNLs and different quorums for the same ledger. As a result, the network's safety is impacted.
This updated version does not impact safety though operates a bit more slowly.
The negative UNL modifications in the _UNLModify_ pseudo-transaction approved by
the consensus will take effect at the next flag ledger. The extra time of the
256 ledgers should be enough for nodes to be in sync of the negative UNL
modifications.
This updated version does not impact safety though operates a bit more slowly. The negative UNL modifications in the _UNLModify_ pseudo-transaction approved by the consensus will take effect at the next flag ledger. The extra time of the 256 ledgers should be enough for nodes to be in sync of the negative UNL modifications.
### Use an Expiration Approach to Re-enable Validators
After a validator disabled by the negative UNL becomes reliable, other
validators explicitly vote for re-enabling it. An alternative approach to
re-enable a validator is the expiration approach, which was considered in the
previous version of the specification. In the expiration approach, every entry
in the negative UNL has a fixed expiration time. One flag ledger interval was
chosen as the expiration interval. Once expired, the other validators must
continue voting to keep the unreliable validator on the negative UNL. The
advantage of this approach is its simplicity. But it has a requirement. The
negative UNL protocol must be able to vote multiple unreliable validators to be
disabled at the same flag ledger. In this version of the specification, however,
only one unreliable validator can be disabled at a flag ledger. So the
expiration approach cannot be simply applied.
After a validator disabled by the negative UNL becomes reliable, other validators explicitly vote for re-enabling it. An alternative approach to re-enable a validator is the expiration approach, which was considered in the previous version of the specification. In the expiration approach, every entry in the negative UNL has a fixed expiration time. One flag ledger interval was chosen as the expiration interval. Once expired, the other validators must continue voting to keep the unreliable validator on the negative UNL. The advantage of this approach is its simplicity. But it has a requirement. The negative UNL protocol must be able to vote multiple unreliable validators to be disabled at the same flag ledger. In this version of the specification, however, only one unreliable validator can be disabled at a flag ledger. So the expiration approach cannot be simply applied.
### Validator Reliability Measurement and Flag Ledger Frequency
If the ledger time is about 4.5 seconds and the low-water mark is 50%, then in
the worst case, it takes 48 minutes _((0.5 _ 256 + 256 + 256) _ 4.5 / 60 = 48)_
to put an offline validator on the negative UNL. We considered lowering the flag
ledger frequency so that the negative UNL can be more responsive. We also
considered decoupling the reliability measurement and flag ledger frequency to
be more flexible. In practice, however, their benefits are not clear.
If the ledger time is about 4.5 seconds and the low-water mark is 50%, then in the worst case, it takes 48 minutes _((0.5 _ 256 + 256 + 256) _ 4.5 / 60 = 48)_ to put an offline validator on the negative UNL. We considered lowering the flag ledger frequency so that the negative UNL can be more responsive. We also considered decoupling the reliability measurement and flag ledger frequency to be more flexible. In practice, however, their benefits are not clear.
## New Attack Vectors
A group of malicious validators may try to frame a reliable validator and put it
on the negative UNL. But they cannot succeed. Because:
A group of malicious validators may try to frame a reliable validator and put it on the negative UNL. But they cannot succeed. Because:
1. A reliable validator sends a signed validation message every ledger. A
sufficient peer-to-peer network will propagate the validation messages to other
validators. The validators will decide if another validator is reliable or not
only by its local observation of the validation messages received. So an honest
validators vote on another validators reliability is accurate.
1. A reliable validator sends a signed validation message every ledger. A sufficient peer-to-peer network will propagate the validation messages to other validators. The validators will decide if another validator is reliable or not only by its local observation of the validation messages received. So an honest validators vote on another validators reliability is accurate.
1. Given the votes are accurate, and one vote per validator, an honest validator
will not create a UNLModify transaction of a reliable validator.
1. Given the votes are accurate, and one vote per validator, an honest validator will not create a UNLModify transaction of a reliable validator.
1. A validator can be added to a negative UNL only through a UNLModify
transaction.
1. A validator can be added to a negative UNL only through a UNLModify transaction.
Assuming the group of malicious validators is less than the quorum, they cannot
frame a reliable validator.
Assuming the group of malicious validators is less than the quorum, they cannot frame a reliable validator.
## Summary
@@ -358,16 +200,13 @@ The bullet points below briefly summarize the current proposal:
- The motivation of the negative UNL is to improve the liveness of the network.
- The targeted faults are the ones frequently observed in the production
network.
- The targeted faults are the ones frequently observed in the production network.
- Validators propose negative UNL candidates based on their local measurements.
- The absolute minimum quorum is 60% of the original UNL.
- The format of the ledger is changed, and a new _UNLModify_ pseudo-transaction
is added. Any tools or systems that rely on the format of these data will have
to be updated.
- The format of the ledger is changed, and a new _UNLModify_ pseudo-transaction is added. Any tools or systems that rely on the format of these data will have to be updated.
- The negative UNL can only be modified on the flag ledgers.
@@ -375,59 +214,39 @@ The bullet points below briefly summarize the current proposal:
- At most one validator can be removed from the negative UNL at a flag ledger.
- If a validator's reliability status changes, it takes two flag ledgers to
modify the negative UNL.
- If a validator's reliability status changes, it takes two flag ledgers to modify the negative UNL.
- The quorum is the larger of 80% of the effective UNL and 60% of the original
UNL.
- The quorum is the larger of 80% of the effective UNL and 60% of the original UNL.
- If a validator is on the negative UNL, its validation messages are ignored
when the local node verifies if a ledger is fully validated.
- If a validator is on the negative UNL, its validation messages are ignored when the local node verifies if a ledger is fully validated.
## FAQ
### Question: What are UNLs?
Quote from the [Technical FAQ](https://xrpl.org/technical-faq.html): "They are
the lists of transaction validators a given participant believes will not
conspire to defraud them."
Quote from the [Technical FAQ](https://xrpl.org/technical-faq.html): "They are the lists of transaction validators a given participant believes will not conspire to defraud them."
### Question: How does the negative UNL proposal affect network liveness?
The network can make forward progress when more than a quorum of the trusted
validators agree with the progress. The lower the quorum size is, the easier for
the network to progress. If the quorum is too low, however, the network is not
safe because nodes may have different results. So the quorum size used in the
consensus protocol is a balance between the safety and the liveness of the
network. The negative UNL reduces the size of the effective UNL, resulting in a
lower quorum size while keeping the network safe.
The network can make forward progress when more than a quorum of the trusted validators agree with the progress. The lower the quorum size is, the easier for the network to progress. If the quorum is too low, however, the network is not safe because nodes may have different results. So the quorum size used in the consensus protocol is a balance between the safety and the liveness of the network. The negative UNL reduces the size of the effective UNL, resulting in a lower quorum size while keeping the network safe.
<h3> Question: How does a validator get into the negative UNL? How is a
validator removed from the negative UNL? </h3>
A validators reliability is measured by other validators. If a validator
becomes unreliable, at a flag ledger, other validators propose _UNLModify_
pseudo-transactions which vote the validator to add to the negative UNL during
the consensus session. If agreed, the validator is added to the negative UNL at
the next flag ledger. The mechanism of removing a validator from the negative
UNL is the same.
A validators reliability is measured by other validators. If a validator becomes unreliable, at a flag ledger, other validators propose _UNLModify_ pseudo-transactions which vote the validator to add to the negative UNL during the consensus session. If agreed, the validator is added to the negative UNL at the next flag ledger. The mechanism of removing a validator from the negative UNL is the same.
### Question: Given a negative UNL, what happens if the UNL changes?
Answer: Lets consider the cases:
1. A validator is added to the UNL, and it is already in the negative UNL. This
case could happen when not all the nodes have the same UNL. Note that the
negative UNL on the ledger lists unreliable nodes that are not necessarily the
validators for everyone.
1. A validator is added to the UNL, and it is already in the negative UNL. This case could happen when not all the nodes have the same UNL. Note that the negative UNL on the ledger lists unreliable nodes that are not necessarily the validators for everyone.
In this case, the liveness is affected negatively. Because the minimum
quorum could be larger but the usable validators are not increased.
1. A validator is removed from the UNL, and it is in the negative UNL.
In this case, the liveness is affected positively. Because the quorum could
be smaller but the usable validators are not reduced.
In this case, the liveness is affected positively. Because the quorum could be smaller but the usable validators are not reduced.
1. A validator is added to the UNL, and it is not in the negative UNL.
1. A validator is removed from the UNL, and it is not in the negative UNL.
@@ -438,59 +257,23 @@ Answer: Lets consider the cases:
Answer: No, because the negative UNL approach is safer.
First lets compare the two approaches intuitively, (1) the _negative UNL_
approach, and (2) _lower quorum_: simply lowering the quorum from 80% to 60%
without the negative UNL. The negative UNL approach uses consensus to come up
with a list of unreliable validators, which are then removed from the effective
UNL temporarily. With this approach, the list of unreliable validators is agreed
to by a quorum of validators and will be used by every node in the network to
adjust its UNL. The quorum is always 80% of the effective UNL. The lower quorum
approach is a tradeoff between safety and liveness and against our principle of
preferring safety over liveness. Note that different validators don't have to
agree on which validation sources they are ignoring.
First lets compare the two approaches intuitively, (1) the _negative UNL_ approach, and (2) _lower quorum_: simply lowering the quorum from 80% to 60% without the negative UNL. The negative UNL approach uses consensus to come up with a list of unreliable validators, which are then removed from the effective UNL temporarily. With this approach, the list of unreliable validators is agreed to by a quorum of validators and will be used by every node in the network to adjust its UNL. The quorum is always 80% of the effective UNL. The lower quorum approach is a tradeoff between safety and liveness and against our principle of preferring safety over liveness. Note that different validators don't have to agree on which validation sources they are ignoring.
Next we compare the two approaches quantitatively with examples, and apply
Theorem 8 of [Analysis of the XRP Ledger Consensus
Protocol](https://arxiv.org/abs/1802.07242) paper:
Next we compare the two approaches quantitatively with examples, and apply Theorem 8 of [Analysis of the XRP Ledger Consensus Protocol](https://arxiv.org/abs/1802.07242) paper:
_XRP LCP guarantees fork safety if **O<sub>i,j</sub> > n<sub>j</sub> / 2 +
n<sub>i</sub> q<sub>i</sub> + t<sub>i,j</sub>** for every pair of nodes
P<sub>i</sub>, P<sub>j</sub>,_
_XRP LCP guarantees fork safety if **O<sub>i,j</sub> > n<sub>j</sub> / 2 + n<sub>i</sub> q<sub>i</sub> + t<sub>i,j</sub>** for every pair of nodes P<sub>i</sub>, P<sub>j</sub>,_
where _O<sub>i,j</sub>_ is the overlapping requirement, n<sub>j</sub> and
n<sub>i</sub> are UNL sizes, q<sub>i</sub> is the quorum size of P<sub>i</sub>,
_t<sub>i,j</sub> = min(t<sub>i</sub>, t<sub>j</sub>, O<sub>i,j</sub>)_, and
t<sub>i</sub> and t<sub>j</sub> are the number of faults can be tolerated by
P<sub>i</sub> and P<sub>j</sub>.
where _O<sub>i,j</sub>_ is the overlapping requirement, n<sub>j</sub> and n<sub>i</sub> are UNL sizes, q<sub>i</sub> is the quorum size of P<sub>i</sub>, _t<sub>i,j</sub> = min(t<sub>i</sub>, t<sub>j</sub>, O<sub>i,j</sub>)_, and t<sub>i</sub> and t<sub>j</sub> are the number of faults can be tolerated by P<sub>i</sub> and P<sub>j</sub>.
We denote _UNL<sub>i</sub>_ as _P<sub>i</sub>'s UNL_, and _|UNL<sub>i</sub>|_ as
the size of _P<sub>i</sub>'s UNL_.
We denote _UNL<sub>i</sub>_ as _P<sub>i</sub>'s UNL_, and _|UNL<sub>i</sub>|_ as the size of _P<sub>i</sub>'s UNL_.
Assuming _|UNL<sub>i</sub>| = |UNL<sub>j</sub>|_, let's consider the following
three cases:
Assuming _|UNL<sub>i</sub>| = |UNL<sub>j</sub>|_, let's consider the following three cases:
1. With 80% quorum and 20% faults, _O<sub>i,j</sub> > 100% / 2 + 100% - 80% +
20% = 90%_. I.e. fork safety requires > 90% UNL overlaps. This is one of the
results in the analysis paper.
1. With 80% quorum and 20% faults, _O<sub>i,j</sub> > 100% / 2 + 100% - 80% + 20% = 90%_. I.e. fork safety requires > 90% UNL overlaps. This is one of the results in the analysis paper.
1. If the quorum is 60%, the relationship between the overlapping requirement
and the faults that can be tolerated is _O<sub>i,j</sub> > 90% +
t<sub>i,j</sub>_. Under the same overlapping condition (i.e. 90%), to guarantee
the fork safety, the network cannot tolerate any faults. So under the same
overlapping condition, if the quorum is simply lowered, the network can tolerate
fewer faults.
1. If the quorum is 60%, the relationship between the overlapping requirement and the faults that can be tolerated is _O<sub>i,j</sub> > 90% + t<sub>i,j</sub>_. Under the same overlapping condition (i.e. 90%), to guarantee the fork safety, the network cannot tolerate any faults. So under the same overlapping condition, if the quorum is simply lowered, the network can tolerate fewer faults.
1. With the negative UNL approach, we want to argue that the inequation
_O<sub>i,j</sub> > n<sub>j</sub> / 2 + n<sub>i</sub> q<sub>i</sub> +
t<sub>i,j</sub>_ is always true to guarantee fork safety, while the negative UNL
protocol runs, i.e. the effective quorum is lowered without weakening the
network's fault tolerance. To make the discussion easier, we rewrite the
inequation as _O<sub>i,j</sub> > n<sub>j</sub> / 2 + (n<sub>i</sub>
q<sub>i</sub>) + min(t<sub>i</sub>, t<sub>j</sub>)_, where O<sub>i,j</sub> is
dropped from the definition of t<sub>i,j</sub> because _O<sub>i,j</sub> >
min(t<sub>i</sub>, t<sub>j</sub>)_ always holds under the parameters we will
use. Assuming a validator V is added to the negative UNL, now let's consider the
4 cases:
1. With the negative UNL approach, we want to argue that the inequation _O<sub>i,j</sub> > n<sub>j</sub> / 2 + n<sub>i</sub> q<sub>i</sub> + t<sub>i,j</sub>_ is always true to guarantee fork safety, while the negative UNL protocol runs, i.e. the effective quorum is lowered without weakening the network's fault tolerance. To make the discussion easier, we rewrite the inequation as _O<sub>i,j</sub> > n<sub>j</sub> / 2 + (n<sub>i</sub> q<sub>i</sub>) + min(t<sub>i</sub>, t<sub>j</sub>)_, where O<sub>i,j</sub> is dropped from the definition of t<sub>i,j</sub> because _O<sub>i,j</sub> > min(t<sub>i</sub>, t<sub>j</sub>)_ always holds under the parameters we will use. Assuming a validator V is added to the negative UNL, now let's consider the 4 cases:
1. V is not on UNL<sub>i</sub> nor UNL<sub>j</sub>
@@ -526,64 +309,35 @@ three cases:
<h3> Question: We have observed that occasionally a validator wanders off on its
own chain. How is this case handled by the negative UNL algorithm? </h3>
Answer: The case that a validator wanders off on its own chain can be measured
with the validations agreement. Because the validations by this validator must
be different from other validators' validations of the same sequence numbers.
When there are enough disagreed validations, other validators will vote this
validator onto the negative UNL.
Answer: The case that a validator wanders off on its own chain can be measured with the validations agreement. Because the validations by this validator must be different from other validators' validations of the same sequence numbers. When there are enough disagreed validations, other validators will vote this validator onto the negative UNL.
In general by measuring the agreement of validations, we also measured the
"sanity". If two validators have too many disagreements, one of them could be
insane. When enough validators think a validator is insane, that validator is
put on the negative UNL.
In general by measuring the agreement of validations, we also measured the "sanity". If two validators have too many disagreements, one of them could be insane. When enough validators think a validator is insane, that validator is put on the negative UNL.
<h3> Question: Why would there be at most one disable UNLModify and one
re-enable UNLModify transaction per flag ledger? </h3>
Answer: It is a design choice so that the effective UNL does not change too
quickly. A typical targeted scenario is several validators go offline slowly
during a long weekend. The current design can handle this kind of cases well
without changing the effective UNL too quickly.
Answer: It is a design choice so that the effective UNL does not change too quickly. A typical targeted scenario is several validators go offline slowly during a long weekend. The current design can handle this kind of cases well without changing the effective UNL too quickly.
## Appendix
### Confidence Test
We will use two test networks, a single machine test network with multiple IP
addresses and the QE test network with multiple machines. The single machine
network will be used to test all the test cases and to debug. The QE network
will be used after that. We want to see the test cases still pass with real
network delay. A test case specifies:
We will use two test networks, a single machine test network with multiple IP addresses and the QE test network with multiple machines. The single machine network will be used to test all the test cases and to debug. The QE network will be used after that. We want to see the test cases still pass with real network delay. A test case specifies:
1. a UNL with different number of validators for different test cases,
1. a network with zero or more non-validator nodes,
1. a sequence of validator reliability change events (by killing/restarting
nodes, or by running modified xrpld that does not send all validation
messages),
1. a sequence of validator reliability change events (by killing/restarting nodes, or by running modified xrpld that does not send all validation messages),
1. the correct outcomes.
For all the test cases, the correct outcomes are verified by examining logs. We
will grep the log to see if the correct negative UNLs are generated, and whether
or not the network is making progress when it should be. The ripdtop tool will
be helpful for monitoring validators' states and ledger progress. Some of the
timing parameters of xrpld will be changed to have faster ledger time. Most if
not all test cases do not need client transactions.
For all the test cases, the correct outcomes are verified by examining logs. We will grep the log to see if the correct negative UNLs are generated, and whether or not the network is making progress when it should be. The ripdtop tool will be helpful for monitoring validators' states and ledger progress. Some of the timing parameters of xrpld will be changed to have faster ledger time. Most if not all test cases do not need client transactions.
For example, the test cases for the prototype:
1. A 10-validator UNL.
1. The network does not have other nodes.
1. The validators will be started from the genesis. Once they start to produce
ledgers, we kill five validators, one every flag ledger interval. Then we
will restart them one by one.
1. A sequence of events (or the lack of events) such as a killed validator is
added to the negative UNL.
1. The validators will be started from the genesis. Once they start to produce ledgers, we kill five validators, one every flag ledger interval. Then we will restart them one by one.
1. A sequence of events (or the lack of events) such as a killed validator is added to the negative UNL.
#### Roads Not Taken: Test with Extended CSF
We considered testing with the current unit test framework, specifically the
[Consensus Simulation
Framework](https://github.com/XRPLF/rippled/blob/develop/src/test/csf/README.md)
(CSF). However, the CSF currently can only test the generic consensus algorithm
as in the paper: [Analysis of the XRP Ledger Consensus
Protocol](https://arxiv.org/abs/1802.07242).
We considered testing with the current unit test framework, specifically the [Consensus Simulation Framework](https://github.com/XRPLF/rippled/blob/develop/src/test/csf/README.md) (CSF). However, the CSF currently can only test the generic consensus algorithm as in the paper: [Analysis of the XRP Ledger Consensus Protocol](https://arxiv.org/abs/1802.07242).

View File

@@ -1,63 +1,20 @@
# Ledger Replay
`LedgerReplayer` is a new `Stoppable` for replaying ledgers.
Patterned after two other `Stoppable`s under `JobQueue`---`InboundLedgers`
and `InboundTransactions`---it acts like a factory for creating
state-machine workers, and a network message demultiplexer for those workers.
Think of these workers like asynchronous functions.
Like functions, they each take a set of parameters.
The `Stoppable` memoizes these functions. It maintains a table for each
worker type, mapping sets of arguments to the worker currently working
on that argument set.
Whenever the `Stoppable` is asked to construct a worker, it first searches its
table to see if there is an existing worker with the same or overlapping
argument set.
If one exists, then it is used. If not, then a new one is created,
initialized, and added to the table.
`LedgerReplayer` is a new `Stoppable` for replaying ledgers. Patterned after two other `Stoppable`s under `JobQueue`---`InboundLedgers` and `InboundTransactions`---it acts like a factory for creating state-machine workers, and a network message demultiplexer for those workers. Think of these workers like asynchronous functions. Like functions, they each take a set of parameters. The `Stoppable` memoizes these functions. It maintains a table for each worker type, mapping sets of arguments to the worker currently working on that argument set. Whenever the `Stoppable` is asked to construct a worker, it first searches its table to see if there is an existing worker with the same or overlapping argument set. If one exists, then it is used. If not, then a new one is created, initialized, and added to the table.
For `LedgerReplayer`, there are three worker types: `LedgerReplayTask`,
`SkipListAcquire`, and `LedgerDeltaAcquire`.
Each is derived from `TimeoutCounter` to give it a timeout.
For `LedgerReplayTask`, the parameter set
is {reason, finish ledger ID, number of ledgers}. For `SkipListAcquire` and
`LedgerDeltaAcquire`, there is just one parameter: a ledger ID.
For `LedgerReplayer`, there are three worker types: `LedgerReplayTask`, `SkipListAcquire`, and `LedgerDeltaAcquire`. Each is derived from `TimeoutCounter` to give it a timeout. For `LedgerReplayTask`, the parameter set is {reason, finish ledger ID, number of ledgers}. For `SkipListAcquire` and `LedgerDeltaAcquire`, there is just one parameter: a ledger ID.
Each `Stoppable` has an entry point. For `LedgerReplayer`, it is `replay`.
`replay` creates two workers: a `LedgerReplayTask` and a `SkipListAcquire`.
`LedgerDeltaAcquire`s are created in the callback for when the skip list
returns.
Each `Stoppable` has an entry point. For `LedgerReplayer`, it is `replay`. `replay` creates two workers: a `LedgerReplayTask` and a `SkipListAcquire`. `LedgerDeltaAcquire`s are created in the callback for when the skip list returns.
For `SkipListAcquire` and `LedgerDeltaAcquire`, initialization fires off the
underlying asynchronous network request and starts the timeout. The argument
set identifying the worker is included in the network request, and copied to
the network response. `SkipListAcquire` sends a request for a proof path for
the skip list of the desired ledger. `LedgerDeltaAcquire` sends a request for
the transaction set of the desired ledger.
For `SkipListAcquire` and `LedgerDeltaAcquire`, initialization fires off the underlying asynchronous network request and starts the timeout. The argument set identifying the worker is included in the network request, and copied to the network response. `SkipListAcquire` sends a request for a proof path for the skip list of the desired ledger. `LedgerDeltaAcquire` sends a request for the transaction set of the desired ledger.
`LedgerReplayer` is also a network message demultiplexer.
When a response arrives for a request that was sent by a `SkipListAcquire` or
`LedgerDeltaAcquire` worker, the `Peer` object knows to send it to the
`LedgerReplayer`, which looks up the worker waiting for that response based on
the identifying argument set included in the response.
`LedgerReplayer` is also a network message demultiplexer. When a response arrives for a request that was sent by a `SkipListAcquire` or `LedgerDeltaAcquire` worker, the `Peer` object knows to send it to the `LedgerReplayer`, which looks up the worker waiting for that response based on the identifying argument set included in the response.
`LedgerReplayTask` may ask `InboundLedgers` to send requests to acquire
the start ledger, but there is no way to attach a callback or be notified when
the `InboundLedger` worker completes. All the responses for its messages will
be directed to `InboundLedgers`, not `LedgerReplayer`. Instead,
`LedgerReplayTask` checks whether the start ledger has arrived every time its
timeout expires.
`LedgerReplayTask` may ask `InboundLedgers` to send requests to acquire the start ledger, but there is no way to attach a callback or be notified when the `InboundLedger` worker completes. All the responses for its messages will be directed to `InboundLedgers`, not `LedgerReplayer`. Instead, `LedgerReplayTask` checks whether the start ledger has arrived every time its timeout expires.
Like a promise, each worker keeps track of whether it is pending (`!isDone()`)
or whether it has resolved successfully (`complete_ == true`) or unsuccessfully
(`failed_ == true`). It will never exist in both resolved states at once, nor
will it return to a pending state after reaching a resolved state.
Like a promise, each worker keeps track of whether it is pending (`!isDone()`) or whether it has resolved successfully (`complete_ == true`) or unsuccessfully (`failed_ == true`). It will never exist in both resolved states at once, nor will it return to a pending state after reaching a resolved state.
Like promises, some workers can accept continuations to be called when they
reach a resolved state, or immediately if they are already resolved.
`SkipListAcquire` and `LedgerDeltaAcquire` both accept continuations of a type
specific to their payload, both via a method named `addDataCallback()`. Continuations
cannot be removed explicitly, but they are held by `std::weak_ptr` so they can
be removed implicitly.
Like promises, some workers can accept continuations to be called when they reach a resolved state, or immediately if they are already resolved. `SkipListAcquire` and `LedgerDeltaAcquire` both accept continuations of a type specific to their payload, both via a method named `addDataCallback()`. Continuations cannot be removed explicitly, but they are held by `std::weak_ptr` so they can be removed implicitly.
`LedgerReplayTask` is simultaneously:
@@ -73,13 +30,7 @@ Each of these roles corresponds to different entry points:
1. the callback added to `LedgerDeltaAcquire`, which calls `deltaReady(...)` or `cancel()`
1. `onTimer()`
Each of these entry points does something unique to that entry point. They
either (a) transition `LedgerReplayTask` to a terminal failed resolved state
(`cancel()` and `onTimer()`) or (b) try to make progress toward the successful
resolved state. `init()` and `updateSkipList(...)` call `trigger()` while
`deltaReady(...)` calls `tryAdvance()`. There's a similarity between this
pattern and the way coroutines are implemented, where every yield saves the spot
in the code where it left off and every resume jumps back to that spot.
Each of these entry points does something unique to that entry point. They either (a) transition `LedgerReplayTask` to a terminal failed resolved state (`cancel()` and `onTimer()`) or (b) try to make progress toward the successful resolved state. `init()` and `updateSkipList(...)` call `trigger()` while `deltaReady(...)` calls `tryAdvance()`. There's a similarity between this pattern and the way coroutines are implemented, where every yield saves the spot in the code where it left off and every resume jumps back to that spot.
### Sequence Diagram

View File

@@ -1,13 +1,10 @@
# Coding Standards
Coding standards used here gradually evolve and propagate through
code reviews. Some aspects are enforced more strictly than others.
Coding standards used here gradually evolve and propagate through code reviews. Some aspects are enforced more strictly than others.
## Rules
These rules only apply to our own code. We can't enforce any sort of
style on the external repositories and libraries we include. The best
guideline is to maintain the standards that are used in those libraries.
These rules only apply to our own code. We can't enforce any sort of style on the external repositories and libraries we include. The best guideline is to maintain the standards that are used in those libraries.
- Tab inserts 4 spaces. No tab characters.
- Braces are indented in the [Allman style][1].
@@ -16,67 +13,37 @@ guideline is to maintain the standards that are used in those libraries.
## Guidelines
If you want to do something contrary to these guidelines, understand
why you're doing it. Think, use common sense, and consider that these
changes will probably need to be maintained long after you've
moved on to other projects.
If you want to do something contrary to these guidelines, understand why you're doing it. Think, use common sense, and consider that these changes will probably need to be maintained long after you've moved on to other projects.
- Use white space and blank lines to guide the eye and keep your intent clear.
- Put private data members at the top of a class, and the 6 public special
members immediately after, in the following order:
- Put private data members at the top of a class, and the 6 public special members immediately after, in the following order:
- Destructor
- Default constructor
- Copy constructor
- Copy assignment
- Move constructor
- Move assignment
- Don't over-inline by defining large functions within the class
declaration, not even for template classes.
- Don't over-inline by defining large functions within the class declaration, not even for template classes.
## Formatting
The goal of source code formatting should always be to make things as easy to
read as possible. White space is used to guide the eye so that details are not
overlooked. Blank lines are used to separate code into "paragraphs."
The goal of source code formatting should always be to make things as easy to read as possible. White space is used to guide the eye so that details are not overlooked. Blank lines are used to separate code into "paragraphs."
- Always place a space before and after all binary operators,
especially assignments (`operator=`).
- Always place a space before and after all binary operators, especially assignments (`operator=`).
- The `!` operator should be preceded by a space, but not followed by one.
- The `~` operator should be preceded by a space, but not followed by one.
- The `++` and `--` operators should have no spaces between the operator and
the operand.
- The `++` and `--` operators should have no spaces between the operator and the operand.
- A space never appears before a comma, and always appears after a comma.
- Don't put spaces after a parenthesis. A typical member function call might
look like this: `foobar (1, 2, 3);`
- Don't put spaces after a parenthesis. A typical member function call might look like this: `foobar (1, 2, 3);`
- In general, leave a blank line before an `if` statement.
- In general, leave a blank line after a closing brace `}`.
- Do not place code on the same line as any opening or
closing brace.
- Do not write `if` statements all-on-one-line. The exception to this is when
you've got a sequence of similar `if` statements, and are aligning them all
vertically to highlight their similarities.
- In an `if-else` statement, if you surround one half of the statement with
braces, you also need to put braces around the other half, to match.
- When writing a pointer type, use this spacing: `SomeObject* myObject`.
Technically, a more correct spacing would be `SomeObject *myObject`, but
it makes more sense for the asterisk to be grouped with the type name,
since being a pointer is part of the type, not the variable name. The only
time that this can lead to any problems is when you're declaring multiple
pointers of the same type in the same statement - which leads on to the next
rule:
- When declaring multiple pointers, never do so in a single statement, e.g.
`SomeObject* p1, *p2;` - instead, always split them out onto separate lines
and write the type name again, to make it quite clear what's going on, and
avoid the danger of missing out any vital asterisks.
- The previous point also applies to references, so always put the `&` next to
the type rather than the variable, e.g. `void foo (Thing const& thing)`. And
don't put a space on both sides of the `*` or `&` - always put a space after
it, but never before it.
- The word `const` should be placed to the right of the thing that it modifies,
for consistency. For example `int const` refers to an int which is const.
`int const*` is a pointer to an int which is const. `int *const` is a const
pointer to an int.
- Always place a space in between the template angle brackets and the type
name. Template code is already hard enough to read!
- Do not place code on the same line as any opening or closing brace.
- Do not write `if` statements all-on-one-line. The exception to this is when you've got a sequence of similar `if` statements, and are aligning them all vertically to highlight their similarities.
- In an `if-else` statement, if you surround one half of the statement with braces, you also need to put braces around the other half, to match.
- When writing a pointer type, use this spacing: `SomeObject* myObject`. Technically, a more correct spacing would be `SomeObject *myObject`, but it makes more sense for the asterisk to be grouped with the type name, since being a pointer is part of the type, not the variable name. The only time that this can lead to any problems is when you're declaring multiple pointers of the same type in the same statement - which leads on to the next rule:
- When declaring multiple pointers, never do so in a single statement, e.g. `SomeObject* p1, *p2;` - instead, always split them out onto separate lines and write the type name again, to make it quite clear what's going on, and avoid the danger of missing out any vital asterisks.
- The previous point also applies to references, so always put the `&` next to the type rather than the variable, e.g. `void foo (Thing const& thing)`. And don't put a space on both sides of the `*` or `&` - always put a space after it, but never before it.
- The word `const` should be placed to the right of the thing that it modifies, for consistency. For example `int const` refers to an int which is const. `int const*` is a pointer to an int which is const. `int *const` is a const pointer to an int.
- Always place a space in between the template angle brackets and the type name. Template code is already hard enough to read!
[1]: http://en.wikipedia.org/wiki/Indent_style#Allman_style

View File

@@ -1,33 +1,18 @@
## Heap profiling of xrpld with jemalloc
The jemalloc library provides a good API for doing heap analysis,
including a mechanism to dump a description of the heap from within the
running application via a function call. Details on how to perform this
activity in general, as well as how to acquire the software, are available on
the jemalloc site:
[https://github.com/jemalloc/jemalloc/wiki/Use-Case:-Heap-Profiling](https://github.com/jemalloc/jemalloc/wiki/Use-Case:-Heap-Profiling)
The jemalloc library provides a good API for doing heap analysis, including a mechanism to dump a description of the heap from within the running application via a function call. Details on how to perform this activity in general, as well as how to acquire the software, are available on the jemalloc site: [https://github.com/jemalloc/jemalloc/wiki/Use-Case:-Heap-Profiling](https://github.com/jemalloc/jemalloc/wiki/Use-Case:-Heap-Profiling)
jemalloc is acquired separately from xrpld, and is not affiliated
with Ripple Labs. If you compile and install jemalloc from the
source release with default options, it will install the library and header
under `/usr/local/lib` and `/usr/local/include`, respectively. Heap
profiling has been tested with xrpld on a Linux platform. It should
work on platforms on which both xrpld and jemalloc are available.
jemalloc is acquired separately from xrpld, and is not affiliated with Ripple Labs. If you compile and install jemalloc from the source release with default options, it will install the library and header under `/usr/local/lib` and `/usr/local/include`, respectively. Heap profiling has been tested with xrpld on a Linux platform. It should work on platforms on which both xrpld and jemalloc are available.
To link xrpld with jemalloc, the argument
`profile-jemalloc=<jemalloc_dir>` is provided after the optional target.
The `<jemalloc_dir>` argument should be the same as that of the
`--prefix` parameter passed to the jemalloc configure script when building.
To link xrpld with jemalloc, the argument `profile-jemalloc=<jemalloc_dir>` is provided after the optional target. The `<jemalloc_dir>` argument should be the same as that of the `--prefix` parameter passed to the jemalloc configure script when building.
## Examples:
Build xrpld with jemalloc library under /usr/local/lib and
header under /usr/local/include:
Build xrpld with jemalloc library under /usr/local/lib and header under /usr/local/include:
$ scons profile-jemalloc=/usr/local
Build xrpld using clang with the jemalloc library under /opt/local/lib
and header under /opt/local/include:
Build xrpld using clang with the jemalloc library under /opt/local/lib and header under /opt/local/include:
$ scons clang profile-jemalloc=/opt/local
@@ -35,10 +20,7 @@ and header under /opt/local/include:
## Using the jemalloc library from within the code
The `profile-jemalloc` parameter enables a macro definition called
`PROFILE_JEMALLOC`. Include the jemalloc header file as
well as the api call(s) that you wish to make within preprocessor
conditional groups, such as:
The `profile-jemalloc` parameter enables a macro definition called `PROFILE_JEMALLOC`. Include the jemalloc header file as well as the api call(s) that you wish to make within preprocessor conditional groups, such as:
In global scope:
@@ -52,11 +34,6 @@ And later, within a function scope:
mallctl("prof.dump", NULL, NULL, NULL, 0);
#endif
Fuller descriptions of how to acquire and use jemalloc's api to do memory
analysis are available at the [jemalloc
site.](http://www.canonware.com/jemalloc/)
Fuller descriptions of how to acquire and use jemalloc's api to do memory analysis are available at the [jemalloc site.](http://www.canonware.com/jemalloc/)
Linking against the jemalloc library will override
the system's default `malloc()` and related functions with jemalloc's
implementation. This is the case even if the code is not instrumented
to use jemalloc's specific API.
Linking against the jemalloc library will override the system's default `malloc()` and related functions with jemalloc's implementation. This is the case even if the code is not instrumented to use jemalloc's specific API.

View File

@@ -4,34 +4,25 @@ This document provides advanced instructions for setting up and configuring Cona
## Custom profile
If the default profile does not work for you and you do not yet have a Conan
profile, you can create one by running:
If the default profile does not work for you and you do not yet have a Conan profile, you can create one by running:
```bash
conan profile detect
```
You may need to make changes to the profile to suit your environment. You can
refer to the provided `conan/profiles/default` profile for inspiration, and you
may also need to apply the required [tweaks](#conan-profile-tweaks) to this
default profile.
You may need to make changes to the profile to suit your environment. You can refer to the provided `conan/profiles/default` profile for inspiration, and you may also need to apply the required [tweaks](#conan-profile-tweaks) to this default profile.
## Conan lockfile
To achieve reproducible dependencies, we use a [Conan lockfile](https://docs.conan.io/2/tutorial/versioning/lockfiles.html),
which has to be updated every time dependencies change.
To achieve reproducible dependencies, we use a [Conan lockfile](https://docs.conan.io/2/tutorial/versioning/lockfiles.html), which has to be updated every time dependencies change.
Please see the [instructions on how to regenerate the lockfile](../../conan/lockfile/README.md).
## Patched recipes
Occasionally, we need patched recipes or recipes not present in Conan Center.
We maintain a fork of the Conan Center Index
[here](https://github.com/XRPLF/conan-center-index/) containing the modified and newly added recipes.
Occasionally, we need patched recipes or recipes not present in Conan Center. We maintain a fork of the Conan Center Index [here](https://github.com/XRPLF/conan-center-index/) containing the modified and newly added recipes.
To ensure our patched recipes are used, you must add our Conan remote at a
higher index than the default Conan Center remote, so it is consulted first. You
can do this by running:
To ensure our patched recipes are used, you must add our Conan remote at a higher index than the default Conan Center remote, so it is consulted first. You can do this by running:
```bash
conan remote add --index 0 --force xrplf https://conan.xrplf.org/repository/conan/
@@ -61,16 +52,11 @@ git checkout master
cd ../../
```
In the case we switch to a newer version of a dependency that still requires a
patch or add a new dependency, it will be necessary for you to pull in the changes and re-export the
updated dependencies with the newer version. However, if we switch to a newer
version that no longer requires a patch, no action is required on your part, as
the new recipe will be automatically pulled from the official Conan Center.
In the case we switch to a newer version of a dependency that still requires a patch or add a new dependency, it will be necessary for you to pull in the changes and re-export the updated dependencies with the newer version. However, if we switch to a newer version that no longer requires a patch, no action is required on your part, as the new recipe will be automatically pulled from the official Conan Center.
<!-- prettier-ignore -->
> [!NOTE]
> You might need to add `--lockfile=""` to your `conan install` command
> to avoid automatic use of the existing `conan.lock` file when you run
> `conan export` manually on your machine
> You might need to add `--lockfile=""` to your `conan install` command to avoid automatic use of the existing `conan.lock` file when you run `conan export` manually on your machine
>
> This is not recommended though, as you might end up using different revisions of recipes.
@@ -88,8 +74,7 @@ Possible values are ['5.0', '5.1', '6.0', '6.1', '7.0', '7.3', '8.0', '8.1',
Read "http://docs.conan.io/2/knowledge/faq.html#error-invalid-setting"
```
you need to create `$(conan config home)/settings_user.yml` file if it doesn't exist and add the required version number(s)
to the `version` array specific for your compiler. For example:
you need to create `$(conan config home)/settings_user.yml` file if it doesn't exist and add the required version number(s) to the `version` array specific for your compiler. For example:
```yaml
compiler:
@@ -99,13 +84,9 @@ compiler:
### Multiple compilers
If you have multiple compilers installed, make sure to select the one to use in
your default Conan configuration **before** running `conan profile detect`, by
setting the `CC` and `CXX` environment variables.
If you have multiple compilers installed, make sure to select the one to use in your default Conan configuration **before** running `conan profile detect`, by setting the `CC` and `CXX` environment variables.
For example, if you are running MacOS and have [homebrew
LLVM@18](https://formulae.brew.sh/formula/llvm@18), and want to use it as a
compiler in the new Conan profile:
For example, if you are running MacOS and have [homebrew LLVM@18](https://formulae.brew.sh/formula/llvm@18), and want to use it as a compiler in the new Conan profile:
```bash
export CC=$(brew --prefix llvm@18)/bin/clang
@@ -113,9 +94,7 @@ export CXX=$(brew --prefix llvm@18)/bin/clang++
conan profile detect
```
You should also explicitly set the path to the compiler in the profile file,
which helps to avoid errors when `CC` and/or `CXX` are set and disagree with the
selected Conan profile. For example:
You should also explicitly set the path to the compiler in the profile file, which helps to avoid errors when `CC` and/or `CXX` are set and disagree with the selected Conan profile. For example:
```text
[conf]
@@ -124,15 +103,11 @@ tools.build:compiler_executables={'c':'/usr/bin/gcc','cpp':'/usr/bin/g++'}
### Multiple profiles
You can manage multiple Conan profiles in the directory
`$(conan config home)/profiles`, for example renaming `default` to a different
name and then creating a new `default` profile for a different compiler.
You can manage multiple Conan profiles in the directory `$(conan config home)/profiles`, for example renaming `default` to a different name and then creating a new `default` profile for a different compiler.
### Select language
The default profile created by Conan will typically select different C++ dialect
than C++23 used by this project. You should set `23` in the profile line
starting with `compiler.cppstd=`. For example:
The default profile created by Conan will typically select different C++ dialect than C++23 used by this project. You should set `23` in the profile line starting with `compiler.cppstd=`. For example:
```bash
sed -i.bak -e 's|^compiler\.cppstd=.*$|compiler.cppstd=23|' $(conan config home)/profiles/default
@@ -140,10 +115,7 @@ sed -i.bak -e 's|^compiler\.cppstd=.*$|compiler.cppstd=23|' $(conan config home)
### Select standard library in Linux
**Linux** developers will commonly have a default Conan [profile][] that
compiles with GCC and links with libstdc++. If you are linking with libstdc++
(see profile setting `compiler.libcxx`), then you will need to choose the
`libstdc++11` ABI:
**Linux** developers will commonly have a default Conan [profile][] that compiles with GCC and links with libstdc++. If you are linking with libstdc++ (see profile setting `compiler.libcxx`), then you will need to choose the `libstdc++11` ABI:
```bash
sed -i.bak -e 's|^compiler\.libcxx=.*$|compiler.libcxx=libstdc++11|' $(conan config home)/profiles/default
@@ -151,12 +123,9 @@ sed -i.bak -e 's|^compiler\.libcxx=.*$|compiler.libcxx=libstdc++11|' $(conan con
### Select architecture and runtime in Windows
**Windows** developers may need to use the x64 native build tools. An easy way
to do that is to run the shortcut "x64 Native Tools Command Prompt" for the
version of Visual Studio that you have installed.
**Windows** developers may need to use the x64 native build tools. An easy way to do that is to run the shortcut "x64 Native Tools Command Prompt" for the version of Visual Studio that you have installed.
Windows developers must also build `xrpld` and its dependencies for the x64
architecture:
Windows developers must also build `xrpld` and its dependencies for the x64 architecture:
```bash
sed -i.bak -e 's|^arch=.*$|arch=x86_64|' $(conan config home)/profiles/default
@@ -175,10 +144,8 @@ If you want to experiment with a new package, follow these steps:
1. Search for the package on [Conan Center](https://conan.io/center/).
2. Modify [`conanfile.py`](../../conanfile.py):
- Add a version of the package to the `requires` property.
- Change any default options for the package by adding them to the
`default_options` property (with syntax `'$package:$option': $value`).
3. Regenerate the [Conan lockfile](../../conan/lockfile/README.md) so the new
dependency is captured:
- Change any default options for the package by adding them to the `default_options` property (with syntax `'$package:$option': $value`).
3. Regenerate the [Conan lockfile](../../conan/lockfile/README.md) so the new dependency is captured:
```bash
./conan/lockfile/regenerate.sh
@@ -186,8 +153,7 @@ If you want to experiment with a new package, follow these steps:
4. Modify [`CMakeLists.txt`](../../CMakeLists.txt):
- Add a call to `find_package($package REQUIRED)`.
- Link a library from the package to the target `xrpl_libs`
(search for the existing call to `target_link_libraries(xrpl_libs INTERFACE ...)`).
- Link a library from the package to the target `xrpl_libs` (search for the existing call to `target_link_libraries(xrpl_libs INTERFACE ...)`).
5. Start coding! Don't forget to include whatever headers you need from the package.
[profile]: https://docs.conan.io/2/reference/config_files/profiles.html

103
docs/build/conan.md vendored
View File

@@ -1,115 +1,44 @@
## A crash course in CMake and Conan
To better understand how to use Conan,
we should first understand _why_ we use Conan,
and to understand that,
we need to understand how we use CMake.
To better understand how to use Conan, we should first understand _why_ we use Conan, and to understand that, we need to understand how we use CMake.
### CMake
Technically, you don't need CMake to build this project.
You could manually compile every translation unit into an object file,
using the right compiler options,
and then manually link all those objects together,
using the right linker options.
However, that is very tedious and error-prone,
which is why we lean on tools like CMake.
Technically, you don't need CMake to build this project. You could manually compile every translation unit into an object file, using the right compiler options, and then manually link all those objects together, using the right linker options. However, that is very tedious and error-prone, which is why we lean on tools like CMake.
We have written CMake configuration files
([`CMakeLists.txt`](./CMakeLists.txt) and friends)
for this project so that CMake can be used to correctly compile and link
all of the translation units in it.
Or rather, CMake will generate files for a separate build system
(e.g. Make, Ninja, Visual Studio, Xcode, etc.)
that compile and link all of the translation units.
Even then, CMake has parameters, some of which are platform-specific.
In CMake's parlance, parameters are specially-named **variables** like
[`CMAKE_BUILD_TYPE`][build_type] or
[`CMAKE_MSVC_RUNTIME_LIBRARY`][runtime].
Parameters include:
We have written CMake configuration files ([`CMakeLists.txt`](./CMakeLists.txt) and friends) for this project so that CMake can be used to correctly compile and link all of the translation units in it. Or rather, CMake will generate files for a separate build system (e.g. Make, Ninja, Visual Studio, Xcode, etc.) that compile and link all of the translation units. Even then, CMake has parameters, some of which are platform-specific. In CMake's parlance, parameters are specially-named **variables** like [`CMAKE_BUILD_TYPE`][build_type] or [`CMAKE_MSVC_RUNTIME_LIBRARY`][runtime]. Parameters include:
- what build system to generate files for
- where to find the compiler and linker
- where to find dependencies, e.g. libraries and headers
- how to link dependencies, e.g. any special compiler or linker flags that
need to be used with them, including preprocessor definitions
- how to compile translation units, e.g. with optimizations, debug symbols,
position-independent code, etc.
- how to link dependencies, e.g. any special compiler or linker flags that need to be used with them, including preprocessor definitions
- how to compile translation units, e.g. with optimizations, debug symbols, position-independent code, etc.
- on Windows, which runtime library to link with
For some of these parameters, like the build system and compiler,
CMake goes through a complicated search process to choose default values.
For others, like the dependencies,
_we_ had written in the CMake configuration files of this project
our own complicated process to choose defaults.
For most developers, things "just worked"... until they didn't, and then
you were left trying to debug one of these complicated processes, instead of
choosing and manually passing the parameter values yourself.
For some of these parameters, like the build system and compiler, CMake goes through a complicated search process to choose default values. For others, like the dependencies, _we_ had written in the CMake configuration files of this project our own complicated process to choose defaults. For most developers, things "just worked"... until they didn't, and then you were left trying to debug one of these complicated processes, instead of choosing and manually passing the parameter values yourself.
You can pass every parameter to CMake on the command line,
but writing out these parameters every time we want to configure CMake is
a pain.
Most humans prefer to put them into a configuration file, once, that
CMake can read every time it is configured.
For CMake, that file is a [toolchain file][toolchain].
You can pass every parameter to CMake on the command line, but writing out these parameters every time we want to configure CMake is a pain. Most humans prefer to put them into a configuration file, once, that CMake can read every time it is configured. For CMake, that file is a [toolchain file][toolchain].
### Conan
These next few paragraphs on Conan are going to read much like the ones above
for CMake.
These next few paragraphs on Conan are going to read much like the ones above for CMake.
Technically, you don't need Conan to build this project.
You could manually download, configure, build, and install all of the
dependencies yourself, and then pass all of the parameters necessary for
CMake to link to those dependencies.
To guarantee ABI compatibility, you must be sure to use the same set of
compiler and linker options for all dependencies _and_ this project.
However, that is very tedious and error-prone, which is why we lean on tools
like Conan.
Technically, you don't need Conan to build this project. You could manually download, configure, build, and install all of the dependencies yourself, and then pass all of the parameters necessary for CMake to link to those dependencies. To guarantee ABI compatibility, you must be sure to use the same set of compiler and linker options for all dependencies _and_ this project. However, that is very tedious and error-prone, which is why we lean on tools like Conan.
We have written a Conan configuration file ([`conanfile.py`](../../conanfile.py))
so that Conan can be used to correctly download, configure, build, and install
all of the dependencies for this project,
using a single set of compiler and linker options for all of them.
It generates files that contain almost all of the parameters that CMake
expects.
Those files include:
We have written a Conan configuration file ([`conanfile.py`](../../conanfile.py)) so that Conan can be used to correctly download, configure, build, and install all of the dependencies for this project, using a single set of compiler and linker options for all of them. It generates files that contain almost all of the parameters that CMake expects. Those files include:
- A single toolchain file.
- For every dependency, a CMake [package configuration file][pcf],
[package version file][pvf], and for every build type, a package
targets file.
Together, these files implement version checking and define `IMPORTED`
targets for the dependencies.
- For every dependency, a CMake [package configuration file][pcf], [package version file][pvf], and for every build type, a package targets file. Together, these files implement version checking and define `IMPORTED` targets for the dependencies.
The toolchain file itself amends the search path
([`CMAKE_PREFIX_PATH`][prefix_path]) so that [`find_package()`][find_package]
will [discover][search] the generated package configuration files.
The toolchain file itself amends the search path ([`CMAKE_PREFIX_PATH`][prefix_path]) so that [`find_package()`][find_package] will [discover][search] the generated package configuration files.
**Nearly all we must do to properly configure CMake is pass the toolchain
file.**
What CMake parameters are left out?
You'll still need to pick a build system generator,
and if you choose a single-configuration generator,
you'll need to pass the `CMAKE_BUILD_TYPE`,
which should match the `build_type` setting you gave to Conan.
**Nearly all we must do to properly configure CMake is pass the toolchain file.** What CMake parameters are left out? You'll still need to pick a build system generator, and if you choose a single-configuration generator, you'll need to pass the `CMAKE_BUILD_TYPE`, which should match the `build_type` setting you gave to Conan.
Even then, Conan has parameters, some of which are platform-specific.
In Conan's parlance, parameters are either settings or options.
**Settings** are shared by all packages, e.g. the build type.
**Options** are specific to a given package, e.g. whether to build and link
OpenSSL as a shared library.
Even then, Conan has parameters, some of which are platform-specific. In Conan's parlance, parameters are either settings or options. **Settings** are shared by all packages, e.g. the build type. **Options** are specific to a given package, e.g. whether to build and link OpenSSL as a shared library.
For settings, Conan goes through a complicated search process to choose
defaults.
For options, each package recipe defines its own defaults.
For settings, Conan goes through a complicated search process to choose defaults. For options, each package recipe defines its own defaults.
You can pass every parameter to Conan on the command line,
but it is more convenient to put them in a configuration file, once, that
Conan can read every time it is configured.
For Conan, that file is a [profile][].
**All we must do to properly configure Conan is edit and pass the profile.**
By default, Conan will use the profile named "default".
You can pass every parameter to Conan on the command line, but it is more convenient to put them in a configuration file, once, that Conan can read every time it is configured. For Conan, that file is a [profile][]. **All we must do to properly configure Conan is edit and pass the profile.** By default, Conan will use the profile named "default".
[build_type]: https://cmake.org/cmake/help/latest/variable/CMAKE_BUILD_TYPE.html
[find_package]: https://cmake.org/cmake/help/latest/command/find_package.html

19
docs/build/depend.md vendored
View File

@@ -1,12 +1,8 @@
We recommend two different methods to depend on libxrpl in your own [CMake][]
project.
Both methods add a CMake library target named `xrpl::libxrpl`.
We recommend two different methods to depend on libxrpl in your own [CMake][] project. Both methods add a CMake library target named `xrpl::libxrpl`.
## Conan requirement
The first method adds libxrpl as a [Conan][] requirement.
With this method, there is no need for a Git [submodule][].
It is good for when you just need a dependency on libxrpl as-is.
The first method adds libxrpl as a [Conan][] requirement. With this method, there is no need for a Git [submodule][]. It is good for when you just need a dependency on libxrpl as-is.
```
# This conanfile.txt is just an example.
@@ -49,16 +45,7 @@ cmake --build . --parallel
## CMake subdirectory
The second method adds the [xrpld][] project as a CMake
[subdirectory][add_subdirectory].
This method works well when you keep the xrpld project as a Git
[submodule][].
It's good for when you want to make changes to libxrpl as part of your own
project.
Be careful, though.
Your project will inherit all of the same CMake options,
so watch out for name collisions.
We still recommend using [Conan][] to download, build, and connect dependencies.
The second method adds the [xrpld][] project as a CMake [subdirectory][add_subdirectory]. This method works well when you keep the xrpld project as a Git [submodule][]. It's good for when you want to make changes to libxrpl as part of your own project. Be careful, though. Your project will inherit all of the same CMake options, so watch out for name collisions. We still recommend using [Conan][] to download, build, and connect dependencies.
```
# Add the project as a Git submodule.

View File

@@ -1,14 +1,10 @@
Our [build instructions][BUILD.md] assume you have a C++ development
environment complete with Git, Python, Conan, CMake, and a C++ compiler.
This document explains how to set one up.
Our [build instructions][BUILD.md] assume you have a C++ development environment complete with Git, Python, Conan, CMake, and a C++ compiler. This document explains how to set one up.
[BUILD.md]: ../../BUILD.md
## Tested compiler versions
`xrpld` is built in the **C++23** dialect by default, so your toolchain has to
support it — see [compiler support for C++23][cpp23-support].
The versions currently tested in CI are:
`xrpld` is built in the **C++23** dialect by default, so your toolchain has to support it — see [compiler support for C++23][cpp23-support]. The versions currently tested in CI are:
| Compiler | Version |
| ----------- | ------------------ |
@@ -21,16 +17,9 @@ LLVM tools (`clang-tidy` and `clang-format`) are also pinned to version 22.
### Older compilers
Older compilers may fail to build the latest `develop` code: the codebase now
relies on C++23 features and has been adjusted for `clang-tidy`.
If the latest code doesn't build for you, update your build toolchain first.
Older compilers may fail to build the latest `develop` code: the codebase now relies on C++23 features and has been adjusted for `clang-tidy`. If the latest code doesn't build for you, update your build toolchain first.
If updating isn't an option for you, we do accept pull requests that fix builds
on older compilers, as long as the change is small and doesn't make the code
harder to read. What we can't promise is that older compilers will keep working:
only the versions in the table above are tested in CI, and we won't hold back
the use of C++23 features or add invasive workarounds to keep an untested
compiler building. Treat support for anything outside the table as best-effort.
If updating isn't an option for you, we do accept pull requests that fix builds on older compilers, as long as the change is small and doesn't make the code harder to read. What we can't promise is that older compilers will keep working: only the versions in the table above are tested in CI, and we won't hold back the use of C++23 features or add invasive workarounds to keep an untested compiler building. Treat support for anything outside the table as best-effort.
## Required tools
@@ -43,8 +32,9 @@ Besides a compiler, building `xrpld` requires:
| [Conan](https://conan.io/downloads.html) | 2.17 |
| [CMake](https://cmake.org/download/) | 3.16 |
On Linux and macOS, the [Nix development shell](./nix.md) provides all of them
(see below). On Windows they have to be installed manually.
On Linux and macOS, the [Nix development shell](./nix.md) provides all of them (see below). On Windows they have to be installed manually.
Building with `-Drust=ON` additionally requires a Rust toolchain, see [Rust](#rust). A default build does not, so it is not in the table above.
Once they are in place, verify that everything is installed and runnable with:
@@ -54,38 +44,25 @@ Once they are in place, verify that everything is installed and runnable with:
## Linux and macOS
The **recommended way** to get a development environment on Linux and macOS is
the Nix development shell. It provides the exact tooling used in CI — `git`,
`python`, `conan`, `cmake`, `clang-tidy`, `clang-format`, and everything else —
with a single command and without installing anything system-wide:
The **recommended way** to get a development environment on Linux and macOS is the Nix development shell. It provides the exact tooling used in CI — `git`, `python`, `conan`, `cmake`, `clang-tidy`, `clang-format`, and everything else — with a single command and without installing anything system-wide:
```bash
nix --experimental-features 'nix-command flakes' develop
```
On **Linux**, Nix also provides the compiler (GCC); on **macOS**, it provides
Clang. If you instead opt to use your system-wide Apple Clang (via
`nix develop .#apple-clang`), you need to manage its version yourself (see
below).
On **Linux**, Nix also provides the compiler (GCC); on **macOS**, it provides Clang. If you instead opt to use your system-wide Apple Clang (via `nix develop .#apple-clang`), you need to manage its version yourself (see below).
See [Using the Nix development shell](./nix.md) for installation and usage
details, including how to select a different compiler and why we recommend Nix
over a hand-maintained environment.
See [Using the Nix development shell](./nix.md) for installation and usage details, including how to select a different compiler and why we recommend Nix over a hand-maintained environment.
### macOS: managing the Apple Clang version
If you use your system-wide Apple Clang on macOS (via `nix develop .#apple-clang`),
the compiler version is whatever your installed Xcode (or Command Line Tools)
provides. The following command should return a version greater than or equal to
the [tested one](#tested-compiler-versions):
If you use your system-wide Apple Clang on macOS (via `nix develop .#apple-clang`), the compiler version is whatever your installed Xcode (or Command Line Tools) provides. The following command should return a version greater than or equal to the [tested one](#tested-compiler-versions):
```bash
clang --version
```
If you develop other applications using Xcode, you might be consistently
updating to the newest version of Apple Clang, which will likely cause issues
building xrpld. You may want to install and pin a specific version of Xcode:
If you develop other applications using Xcode, you might be consistently updating to the newest version of Apple Clang, which will likely cause issues building xrpld. You may want to install and pin a specific version of Xcode:
1. **Download Xcode**
- Visit [Apple Developer Downloads](https://developer.apple.com/download/more/)
@@ -111,26 +88,25 @@ building xrpld. You may want to install and pin a specific version of Xcode:
## Windows
Nix is not available on Windows, so the required tools have to be installed
manually:
Nix is not available on Windows, so the required tools have to be installed manually:
- [Visual Studio 2026](https://visualstudio.microsoft.com/) with the
**"Desktop development with C++"** workload — this provides MSVC and the
"x64 Native Tools Command Prompt". CI configures CMake with the
`Visual Studio 18 2026` generator.
- [Visual Studio 2026](https://visualstudio.microsoft.com/) with the **"Desktop development with C++"** workload — this provides MSVC and the "x64 Native Tools Command Prompt". CI configures CMake with the `Visual Studio 18 2026` generator.
- [Git for Windows](https://git-scm.com/download/win)
- Python, Conan, and CMake, at the versions listed in
[Required tools](#required-tools).
- Python, Conan, and CMake, at the versions listed in [Required tools](#required-tools).
- a [Rust toolchain](https://rustup.rs) — only needed to build with `-Drust=ON`, see [Rust](#rust)
## Rust
The repository contains a Rust workspace in [`crates/`](../../crates), whose crates are exposed to C++ through [cxx](https://cxx.rs) bindings. It is **not** part of a default build: the CMake `rust` option is OFF by default, and with it off no Rust toolchain is needed. It is only required when configuring with `-Drust=ON` (which is what CI does), see [Options](../../BUILD.md#options).
The toolchain (`cargo`, `rustc`) is pinned to the channel in [`rust-toolchain.toml`](../../rust-toolchain.toml) at the repository root. If you install Rust with [rustup](https://rustup.rs), that file is picked up automatically, and `cargo`/`rustc` in the repository will use the pinned version.
Everything else the Rust build needs on the CMake side comes from Conan along with the rest of the dependencies, so there is nothing further to install.
## Clang-tidy
`clang-tidy` is required to run static analysis checks locally (see
[CONTRIBUTING.md](../../CONTRIBUTING.md)). It is not required to build the
project. The version this project uses is listed in
[Tested compiler versions](#tested-compiler-versions).
`clang-tidy` is required to run static analysis checks locally (see [CONTRIBUTING.md](../../CONTRIBUTING.md)). It is not required to build the project. The version this project uses is listed in [Tested compiler versions](#tested-compiler-versions).
On Linux and macOS, the [Nix development shell](./nix.md) provides that exact
version out of the box — run it via `run-clang-tidy`. No separate installation
is needed.
On Linux and macOS, the [Nix development shell](./nix.md) provides that exact version out of the box — run it via `run-clang-tidy`. No separate installation is needed.
[cpp23-support]: https://en.cppreference.com/w/cpp/compiler_support/23

91
docs/build/nix.md vendored
View File

@@ -7,7 +7,7 @@ This guide explains how to use Nix to set up a reproducible development environm
## Benefits of Using Nix
- **Reproducible environment**: Everyone gets the same versions of tools and compilers
- **Matches CI**: The Linux CI runs in Docker images built from this exact Nix environment
- **Matches CI**: The Linux CI runs in Docker images built from this exact Nix environment, and CI builds some macOS configurations in it as well
- **No system pollution**: Dependencies are isolated and don't affect your system packages
- **Consistent compilers**: The GCC and Clang shells use the same versions as CI
- **Quick setup**: Get started with a single command
@@ -38,17 +38,10 @@ The first time you run this command, it will take a few minutes to download and
### Platform notes
- **Linux**: `nix develop` gives you a shell with all the tooling necessary to develop xrpld
and with the same GCC/glibc toolchain that Nix builds for CI.
See [Choosing a different compiler](#choosing-a-different-compiler)
for the custom-vs-plain toolchain trade-off.
- **macOS**: `nix develop` gives you a full environment too, with Clang (and
every other tool, including Conan) provided by Nix. To use your system-wide
Apple Clang instead, enter `nix develop .#apple-clang`. Conan has no binary in
the Nix cache for macOS, so it is built from source the first time you enter
the shell, which makes the initial setup slower (this is handled
automatically; see [`nix/devshell.nix`](../../nix/devshell.nix)).
- **Linux**: `nix develop` gives you a shell with all the tooling necessary to develop xrpld and with the same GCC/glibc toolchain that Nix builds for CI. See [Choosing a different compiler](#choosing-a-different-compiler) for the custom-vs-plain toolchain trade-off.
- **macOS**: `nix develop` gives you a full environment too, with Clang (and every other tool, including Conan) provided by Nix. To use your system-wide Apple Clang instead, enter `nix develop .#apple-clang`. Conan has no binary in the Nix cache for macOS, so it is built from source the first time you enter the shell, which makes the initial setup slower (this is handled automatically; see [`nix/devshell.nix`](../../nix/devshell.nix)).
<!-- prettier-ignore -->
> [!TIP]
> To avoid typing `--experimental-features 'nix-command flakes'` every time, you can permanently enable flakes by creating `~/.config/nix/nix.conf`:
>
@@ -59,6 +52,7 @@ The first time you run this command, it will take a few minutes to download and
>
> After this, you can simply use `nix develop` instead.
<!-- prettier-ignore -->
> [!NOTE]
> The examples below assume you've enabled flakes in your config. If you haven't, add `--experimental-features 'nix-command flakes'` after each `nix` command.
@@ -66,14 +60,7 @@ The first time you run this command, it will take a few minutes to download and
A compiler can be chosen by providing its name with the `.#` prefix, e.g. `nix develop .#clang`.
On Linux, `.#gcc` and `.#clang` provide the exact toolchain CI uses:
the compiler (pinned in [`nix/packages.nix`](../../nix/packages.nix))
rebuilt against the pinned custom glibc (see [`nix/compilers.nix`](../../nix/compilers.nix)).
Building that toolchain the first time is slow unless it is fetched from a Nix binary cache.
If you don't need the custom glibc, the Linux-only `.#gcc-plain` and `.#clang-plain`
give you the stock nixpkgs compilers of the same versions.
On macOS there is no custom glibc, so `.#gcc` and `.#clang` are already the plain nixpkgs toolchain,
and the `-plain` variants do not exist.
On Linux, `.#gcc` and `.#clang` provide the exact toolchain CI uses: the compiler (pinned in [`nix/packages.nix`](../../nix/packages.nix)) rebuilt against the pinned custom glibc (see [`nix/linux.nix`](../../nix/linux.nix)). Building that toolchain the first time is slow unless it is fetched from a Nix binary cache. If you don't need the custom glibc, the Linux-only `.#gcc-plain` and `.#clang-plain` give you the stock nixpkgs compilers of the same versions. On macOS there is no custom glibc, so `.#gcc` and `.#clang` are already the plain nixpkgs toolchain, and the `-plain` variants do not exist.
Use `nix flake show` to see all the available development shells.
@@ -111,6 +98,7 @@ nix develop -c fish
nix develop -c "$SHELL"
```
<!-- prettier-ignore -->
> [!WARNING]
> Your shell's interactive startup files (e.g. `config.fish`, `.zshrc`) may prepend other directories — most commonly Homebrew — to `$PATH`, which can shadow the tools provided by the Nix shell. After entering, verify that tools resolve into the Nix store:
>
@@ -124,31 +112,52 @@ nix develop -c "$SHELL"
Once inside the Nix development shell, follow the standard [build instructions](../../BUILD.md#steps). The Nix shell provides all necessary tools (CMake, Ninja, Conan, etc.).
Coverage builds (`-Dcoverage=ON`) work in the `gcc` shell (and `gcc-plain` on Linux):
each ships a `gcov` matching its compiler, since Nix's cc-wrapper does not expose one.
The `clang` shells do not include `llvm-cov`, so use a `gcc` shell for coverage.
Coverage builds (`-Dcoverage=ON`) work in the `gcc` shell (and `gcc-plain` on Linux): each ships a `gcov` matching its compiler, since Nix's cc-wrapper does not expose one. The `clang` shells do not include `llvm-cov`, so use a `gcc` shell for coverage.
Builds of the Rust crates (`-Drust=ON`) also work out of the box: every shell provides the Rust toolchain pinned in [`rust-toolchain.toml`](../../rust-toolchain.toml) (see [Rust](./environment.md#rust)), plus the `cargo-audit`, `cargo-llvm-cov` and `cargo-nextest` plugins.
## Conan configuration
The shell runs [`conan/init.sh`](../../conan/init.sh) on entry, so
[Set Up Conan](../../BUILD.md#set-up-conan) is already done for you. It installs
into the shell's own Conan home: `CONAN_HOME=~/.conan2-nix`.
The shell runs [`conan/init.sh`](../../conan/init.sh) on entry, so [Set Up Conan](../../BUILD.md#set-up-conan) is already done for you. It installs into the shell's own Conan home: `CONAN_HOME=~/.conan2-nix`.
### Prebuilt packages
On **Linux**, the binaries on the `xrplf` remote are built in this same Nix
environment — CI runs in Docker images that bundle the dev shell's toolchain (see
[`nix/docker`](../../nix/docker)) — so `.#gcc` and `.#clang` can reuse them. The
`-plain` shells do not match that toolchain's glibc, so binaries from the remote
are not a reliable match there.
On **Linux**, the binaries on the `xrplf` remote are built in this same Nix environment — CI runs in Docker images that bundle the dev shell's toolchain (see [`nix/docker`](../../nix/docker)) — so `.#gcc` and `.#clang` can reuse them. The `-plain` shells do not match that toolchain's glibc, so binaries from the remote are not a reliable match there.
On **macOS**, CI builds with Apple Clang, so the remote holds nothing for the Nix
`clang` toolchain and dependencies are compiled locally. We do not publish
Nix-built macOS binaries because a Conan package ID records the compiler version
but not the nixpkgs revision.
On **macOS**, CI also builds in this Nix environment, in Debug and Release (the `macos-arm64-*-nix` configurations — Debug because the profile defaults to it). The Nix build resolves to `compiler=clang`, so it gets its own package IDs, separate from the Apple Clang ones. The [dependency upload](../../.github/workflows/upload-conan-deps.yml) publishes them on pushes to `develop` and on manual runs — its nightly run rebuilds everything from source but uploads nothing — so once a set has been published `nix develop` can reuse it instead of compiling every dependency locally. These configurations run outside the reduced pull-request matrix, so label a PR `Full CI build` when it touches `flake.lock` or `nix/`.
To compile everything from source, add `--build '*'` to the `conan install`
command.
To compile everything from source, add `--build '*'` to the `conan install` command.
### Why the nixpkgs revision is not part of the package ID
A Conan package ID records the compiler and its major version, but nothing about the nixpkgs revision the toolchain came from — and `flake.lock` moves far more often than the toolchain meaningfully changes, so folding it in would rebuild every dependency on every bump for nothing.
That is safe as long as no cached artifact resolves a `/nix/store` path at run time, because store paths change on every update and the old ones disappear with `nix-collect-garbage`. With the `clang` toolchain macOS CI and the dev shell use, they do not: it links against `/usr/lib/libc++` and `/usr/lib/libSystem`, and store paths reach the `.a` files only through debug info, which nothing resolves at link or run time.
<!-- prettier-ignore -->
> [!WARNING]
> This does not hold for `nix develop .#gcc` on macOS. There is no system libstdc++, so GCC links its own from the store and every binary keeps a `/nix/store` reference. That shell is fine for tooling, but it is not a build configuration CI covers, and no dependency binaries are published for it.
This is checked rather than assumed. [`bin/check-nix-store-refs.sh`](../../bin/check-nix-store-refs.sh) takes one file or directory and fails if a binary under it resolves a store path at run time. CI runs it over the build output and the Conan cache, and again in the upload job before anything is published. You can run it yourself:
```bash
bin/check-nix-store-refs.sh build
bin/check-nix-store-refs.sh ~/.conan2-nix
```
It works on Linux too, but asserts something narrower there: the toolchain always writes the store into `PT_INTERP` and `RUNPATH`, and CI builds inside an image whose store is fixed for its lifetime, so that is fine. Only the binaries [`PatchNixBinary.cmake`](../../cmake/PatchNixBinary.cmake) retargets to the system loader have to be clean, and those are what CI checks:
```bash
bin/check-nix-store-refs.sh build/xrpld
```
### The libresolv stub
This is not hypothetical: `xrpld` used to be caught by it. The c-ares package tells the linker to pass `-lresolv`, and nixpkgs keeps `libresolv` out of the macOS SDK and ships it as an ordinary store dylib — so every Nix-built `xrpld` recorded a `/nix/store/…-libresolv-93/lib/libresolv.9.dylib` load command and stopped running once that path was collected. Nothing in the link uses a single symbol from it.
Both environments now put a stub on the linker search path (`libresolvSystemStub` in [`nix/darwin.nix`](../../nix/darwin.nix)): the same library with its install name set to `/usr/lib/libresolv.9.dylib`, which is exactly the load command the Apple Clang build records.
Package IDs did not change, so Conan keeps serving anything built before the stub landed. If a binary fails to start with `Library not loaded: /nix/store/…`, see [that entry](./nix_troubleshooting.md#library-not-loaded-nixstore-from-a-binary-that-used-to-work) in the troubleshooting guide.
## Automatic Activation with direnv
@@ -161,6 +170,7 @@ The repository already ships an `.envrc` at its root that activates the Nix flak
1. [Install direnv](https://direnv.net/docs/installation.html) and [hook it into your shell](https://direnv.net/docs/hook.html) (bash, zsh, fish, …). Installing [nix-direnv](https://github.com/nix-community/nix-direnv) as well is recommended: it caches the shell so that activation is near-instant after the first run.
2. Run `direnv allow` once in the repository root. direnv will then load (and reload) the Nix development shell automatically whenever you enter the directory.
<!-- prettier-ignore -->
> [!NOTE]
> direnv only caches the `.direnv` directory (already listed in `.gitignore`); no other repository files are affected.
@@ -170,13 +180,8 @@ To update `flake.lock` to the latest revision use `nix flake update` command.
## Tooling snapshots
The tool versions in each Nix environment are recorded in
[`nix/check-tools/`](../../nix/check-tools) and verified by CI. If you change the
environment (bump the CI image tag, update `flake.lock`, or edit the tool list in
`bin/check-tools.sh`), CI fails until you regenerate and commit the affected
snapshot — see [`nix/check-tools/README.md`](../../nix/check-tools/README.md).
The tool versions in each Nix environment are recorded in [`nix/check-tools/`](../../nix/check-tools) and verified by CI. If you change the environment (bump the CI image tag, update `flake.lock`, or edit the tool list in `bin/check-tools.sh`), CI fails until you regenerate and commit the affected snapshot — see [`nix/check-tools/README.md`](../../nix/check-tools/README.md).
## Troubleshooting
See [Troubleshooting Nix problems](./nix_troubleshooting.md) for common issues,
such as `nix develop` failing inside Git worktrees.
See [Troubleshooting Nix problems](./nix_troubleshooting.md) for common issues, such as `nix develop` failing inside Git worktrees.

View File

@@ -1,7 +1,6 @@
# Troubleshooting Nix problems
Common issues encountered when using the [Nix development shell](./nix.md), and
how to resolve them.
Common issues encountered when using the [Nix development shell](./nix.md), and how to resolve them.
## `command not found: nix` after a macOS update
@@ -12,8 +11,7 @@ $ nix develop
zsh: command not found: nix
```
then Nix is almost certainly still installed — only the shell hook that puts it
on your `PATH` is gone. Confirm that first:
then Nix is almost certainly still installed — only the shell hook that puts it on your `PATH` is gone. Confirm that first:
```bash
ls -l /nix/var/nix/profiles/default/bin/nix
@@ -23,8 +21,7 @@ If that exists, the installation is fine and this is purely a `PATH` problem.
### Why it happens
The installer does not touch your dotfiles. Instead it sources a setup script
from the Nix store by editing **system-wide** rc files:
The installer does not touch your dotfiles. Instead it sources a setup script from the Nix store by editing **system-wide** rc files:
| Shell | File the installer edits |
| ----- | ------------------------------------- |
@@ -32,17 +29,13 @@ from the Nix store by editing **system-wide** rc files:
| zsh | `/etc/zshrc` |
| fish | `$__fish_sysconf_dir/conf.d/nix.fish` |
macOS manages `/etc/zshrc`, so an OS update can replace it with the vendor copy
and silently drop the Nix block. `/etc/bashrc` and the fish file usually survive,
which is why the breakage often shows up in zsh only. You can verify this by
diffing against the backup the installer left behind:
macOS manages `/etc/zshrc`, so an OS update can replace it with the vendor copy and silently drop the Nix block. `/etc/bashrc` and the fish file usually survive, which is why the breakage often shows up in zsh only. You can verify this by diffing against the backup the installer left behind:
```bash
diff /etc/zshrc /etc/zshrc.backup-before-nix
```
If they are identical, the Nix snippet was wiped. This is upstream issue
[NixOS/nix#3616](https://github.com/NixOS/nix/issues/3616).
If they are identical, the Nix snippet was wiped. This is upstream issue [NixOS/nix#3616](https://github.com/NixOS/nix/issues/3616).
### Fix
@@ -52,8 +45,7 @@ To unblock the current shell:
. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh
```
For a permanent fix, add the snippet to your **user** rc file rather than
restoring `/etc/zshrc` — user dotfiles are not clobbered by OS updates:
For a permanent fix, add the snippet to your **user** rc file rather than restoring `/etc/zshrc` — user dotfiles are not clobbered by OS updates:
```bash
cat >>~/.zshrc <<'EOF'
@@ -66,14 +58,11 @@ fi
EOF
```
The scripts guard against double-sourcing via `__ETC_PROFILE_NIX_SOURCED`, so
this is safe even if a system-wide hook is later restored.
The scripts guard against double-sourcing via `__ETC_PROFILE_NIX_SOURCED`, so this is safe even if a system-wide hook is later restored.
<!-- prettier-ignore -->
> [!NOTE]
> `/etc/zshrc` and `~/.zshrc` are only read by **interactive** zsh. If the
> snippet is present but `zsh -c '…'`, a script, or an IDE terminal still can't
> find `nix`, that shell is non-interactive — put the snippet in `~/.zshenv`
> instead.
> `/etc/zshrc` and `~/.zshrc` are only read by **interactive** zsh. If the snippet is present but `zsh -c '…'`, a script, or an IDE terminal still can't find `nix`, that shell is non-interactive — put the snippet in `~/.zshenv` instead.
## Git worktrees
@@ -86,48 +75,99 @@ error:
error: opening Git repository "/path/to/rippled": unsupported extension name extensions.relativeworktrees (libgit2 error code = 6)
```
then your Nix is linked against a libgit2 older than **1.9.4**. Git 2.48+ writes
the `extensions.relativeWorktrees` config entry when a worktree is created with
relative paths (`git worktree add --relative-paths`, or with
`worktree.useRelativePaths=true`), and older libgit2 versions refuse to open a
repository that uses it. Nix uses libgit2 to read the flake, so evaluation
fails.
then your Nix is linked against a libgit2 older than **1.9.4**. Git 2.48+ writes the `extensions.relativeWorktrees` config entry when a worktree is created with relative paths (`git worktree add --relative-paths`, or with `worktree.useRelativePaths=true`), and older libgit2 versions refuse to open a repository that uses it. Nix uses libgit2 to read the flake, so evaluation fails.
<!-- prettier-ignore -->
> [!IMPORTANT]
> This entry is written to the **shared** repository config, so once any
> relative worktree exists, `nix develop` fails in the main checkout too — not
> just inside the worktree.
> This entry is written to the **shared** repository config, so once any relative worktree exists, `nix develop` fails in the main checkout too — not just inside the worktree.
### Workarounds
These work today, with any Nix version:
- bypass libgit2 with a `path:` flakeref: `nix develop "path:$PWD"`
(note: this copies the working tree to the store and ignores `.gitignore`); or
- bypass libgit2 with a `path:` flakeref: `nix develop "path:$PWD"` (note: this copies the working tree to the store and ignores `.gitignore`); or
- create worktrees with absolute paths (omit `--relative-paths`); or
- clear the extension if you don't need relative worktrees:
`git config --unset extensions.relativeWorktrees`.
- clear the extension if you don't need relative worktrees: `git config --unset extensions.relativeWorktrees`.
### Permanent fix
The fix is in [libgit2 1.9.4](https://github.com/libgit2/libgit2/releases/tag/v1.9.4),
so the real solution is a Nix that links against libgit2 `1.9.4` or newer. Check
which version yours links against:
The fix is in [libgit2 1.9.4](https://github.com/libgit2/libgit2/releases/tag/v1.9.4), so the real solution is a Nix that links against libgit2 `1.9.4` or newer. Check which version yours links against:
```bash
nix-store -qR "$(readlink -f "$(command -v nix)")" | grep libgit2
```
<!-- prettier-ignore -->
> [!WARNING]
> `nix upgrade-nix` does **not** help yet. It installs the build from the
> official [`nix-fallback-paths`](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/installer/tools/nix-fallback-paths.nix),
> which is still linked against libgit2 `1.9.2` — there is no new upstream Nix
> release with the fix. (On some systems that build is even the exact store path
> you already have, making the upgrade a no-op.)
> `nix upgrade-nix` does **not** help yet. It installs the build from the official [`nix-fallback-paths`](https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/installer/tools/nix-fallback-paths.nix), which is still linked against libgit2 `1.9.2` — there is no new upstream Nix release with the fix. (On some systems that build is even the exact store path you already have, making the upgrade a no-op.)
nixpkgs has already rebuilt Nix against the fixed libgit2 (e.g. `nix-2.34.7+1`),
so the cleanest path is to reinstall Nix using your usual installation method
once it picks up that rebuild, then re-run the `grep libgit2` check above to
confirm it reports `1.9.4` or newer.
nixpkgs has already rebuilt Nix against the fixed libgit2 (e.g. `nix-2.34.7+1`), so the cleanest path is to reinstall Nix using your usual installation method once it picks up that rebuild, then re-run the `grep libgit2` check above to confirm it reports `1.9.4` or newer.
Until then, prefer the workarounds above.
## `wint_t` / `uint32_t` errors from the Nix libc++ headers
A build that mixes the Nix toolchain with the system SDK fails in libc++ itself, with errors that look nothing like your code:
```
/nix/store/...-libcxx-.../include/c++/v1/cwchar:136:9: error: target of using declaration conflicts with declaration already in scope
136 | using ::wint_t _LIBCPP_USING_IF_EXISTS;
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_wint_t.h:32:25: note: target of using declaration
...
error: use of undeclared identifier 'UINT32_C'
```
The give-away is the second path: Nix's libc++ headers are being combined with the **Xcode Command Line Tools** SDK instead of the Nix one.
### Why it happens
`SDKROOT` and `DEVELOPER_DIR` are what point the toolchain at the Nix SDK, and they are not baked into the compiler — a dev shell gets them from the `apple-sdk` setup hook. CMake, finding neither, asks `xcrun`, which answers with the system SDK. Nix's `libc++` and Apple's headers then declare the same types twice.
### Fix
Run the build from inside the dev shell (`nix develop`), or from an environment that exports both variables. To confirm which SDK a configured build is using:
```bash
grep -o '\-isysroot [^ ]*' build/compile_commands.json | sort -u
```
It should print a `/nix/store/...-apple-sdk-*` path. If it prints `/Library/Developer/CommandLineTools/...`, re-configure from within the shell — CMake caches the sysroot, so an existing `build/` directory keeps the wrong one.
## `Library not loaded: /nix/store/…` from a binary that used to work
A binary stops starting after a `nix flake update`, or after `nix-collect-garbage` removes the paths the previous toolchain used:
```
dyld[57271]: Library not loaded: /nix/store/…-libresolv-93/lib/libresolv.9.dylib
```
[`bin/check-nix-store-refs.sh`](../../bin/check-nix-store-refs.sh) finds the same thing without having to run anything, and names the file:
```
$ bin/check-nix-store-refs.sh ~/.conan2-nix
::error file=/Users/you/.conan2-nix/p/b/c-area24ded30c388c/p/bin/adig::references the Nix store at run time
/Users/you/.conan2-nix/p/b/c-area24ded30c388c/p/bin/adig
/nix/store/p4lp3xq4imd1qzqh08x8vcq2zfhi7rca-libresolv-93/lib/libresolv.9.dylib
/Users/you/.conan2-nix: checked 135, skipped 2495, 1 with Nix store references.
```
Conan's cache folders are named after a truncated package name plus a hash, so ask Conan which package the offending one belongs to — pass the folder holding the hash, not the file itself:
```
$ conan cache ref ~/.conan2-nix/p/b/c-area24ded30c388c
c-ares/1.34.6#545240bb1c40e2cacd4362d6b8967650:dab5992496abe6d219defb7986ecbf367615a5e5#…
```
### Why it happens
The binary records a store path that no longer exists. Nothing we build should: see [Prebuilt packages](./nix.md#prebuilt-packages) for why, and `libresolvSystemStub` in [`nix/darwin.nix`](../../nix/darwin.nix) for the one dependency that needed help to comply.
A Conan package ID does not encode the nixpkgs revision, so a package built before that stub existed stays in your local cache and keeps being reused. The dev shell is also what tends to produce one: it is a slightly _less_ isolated build environment than CI's, because `mkShell` puts every tool's headers and libraries on the compiler's search path — which is how c-ares found the Nix `libresolv` in the first place.
### Fix
Drop that package and let Conan refetch or rebuild it:
```bash
conan remove 'c-ares/*'
```

View File

@@ -1,11 +1,10 @@
# Sanitizer Configuration for Xrpld
This document explains how to properly configure and run sanitizers (`AddressSanitizer`, `UndefinedBehaviorSanitizer`, `ThreadSanitizer`) with the xrpld project.
Corresponding suppression files are located in the `sanitizers/suppressions` directory.
This document explains how to properly configure and run sanitizers (`AddressSanitizer`, `UndefinedBehaviorSanitizer`, `ThreadSanitizer`) with the xrpld project. Corresponding suppression files are located in the `sanitizers/suppressions` directory.
<!-- prettier-ignore -->
> [!CAUTION]
> Do not mix Address and Thread sanitizers - they are incompatible.
> Also, we don't yet support MSVC sanitizers, so this is only for Clang/GCC builds.
> Do not mix Address and Thread sanitizers - they are incompatible. Also, we don't yet support MSVC sanitizers, so this is only for Clang/GCC builds.
- [Sanitizer Configuration for Xrpld](#sanitizer-configuration-for-xrpld)
- [Building with Sanitizers](#building-with-sanitizers)
@@ -35,10 +34,10 @@ Corresponding suppression files are located in the `sanitizers/suppressions` dir
Follow the same instructions as mentioned in [BUILD.md](../../BUILD.md) but with the following changes:
1. Make sure you have a clean build directory.
2. Set the `SANITIZERS` environment variable before calling `conan install`. Only set it once.
Example: `export SANITIZERS=address,undefinedbehavior`
2. Set the `SANITIZERS` environment variable before calling `conan install`. Only set it once. Example: `export SANITIZERS=address,undefinedbehavior`
3. Use `--profile:all sanitizers` with Conan to build dependencies with sanitizer instrumentation.
<!-- prettier-ignore -->
> [!NOTE]
> Building with sanitizer-instrumented dependencies is slower but produces fewer false positives.

View File

@@ -2,277 +2,116 @@
**This section is a work in progress!!**
Consensus is the task of reaching agreement within a distributed system in the
presence of faulty or even malicious participants. This document outlines the
[XRP Ledger Consensus Algorithm](https://arxiv.org/abs/1802.07242)
as implemented in [xrpld](https://github.com/XRPLF/rippled), but
focuses on its utility as a generic consensus algorithm independent of the
detailed mechanics of the XRPL consensus Ledger. Most notably, the algorithm
does not require fully synchronous communication between all nodes in the
network, or even a fixed network topology, but instead achieves consensus via
collectively trusted subnetworks.
Consensus is the task of reaching agreement within a distributed system in the presence of faulty or even malicious participants. This document outlines the [XRP Ledger Consensus Algorithm](https://arxiv.org/abs/1802.07242) as implemented in [xrpld](https://github.com/XRPLF/rippled), but focuses on its utility as a generic consensus algorithm independent of the detailed mechanics of the XRPL consensus Ledger. Most notably, the algorithm does not require fully synchronous communication between all nodes in the network, or even a fixed network topology, but instead achieves consensus via collectively trusted subnetworks.
## Distributed Agreement
A challenge for distributed systems is reaching agreement on changes in shared
state. For the XRPL network, the shared state is the current ledger--account
information, account balances, order books and other financial data. We will
refer to shared distributed state as a /ledger/ throughout the remainder of this
document.
A challenge for distributed systems is reaching agreement on changes in shared state. For the XRPL network, the shared state is the current ledger--account information, account balances, order books and other financial data. We will refer to shared distributed state as a /ledger/ throughout the remainder of this document.
![Ledger Chain](images/consensus/ledger_chain.png "Ledger Chain")
As shown above, new ledgers are made by applying a set of transactions to the
prior ledger. For the XRPL network, transactions include payments,
modification of account settings, updates to offers and more.
As shown above, new ledgers are made by applying a set of transactions to the prior ledger. For the XRPL network, transactions include payments, modification of account settings, updates to offers and more.
In a centralized system, generating the next ledger is trivial since there is a
single unique arbiter of which transactions to include and how to apply them to
a ledger. For decentralized systems, participants must resolve disagreements on
the set of transactions to include, the order to apply those transactions, and
even the resulting ledger after applying the transactions. This is even more
difficult when some participants are faulty or malicious.
In a centralized system, generating the next ledger is trivial since there is a single unique arbiter of which transactions to include and how to apply them to a ledger. For decentralized systems, participants must resolve disagreements on the set of transactions to include, the order to apply those transactions, and even the resulting ledger after applying the transactions. This is even more difficult when some participants are faulty or malicious.
The XRPL network is a decentralized and **trust-full** network. Anyone is free
to join and participants are free to choose a subset of peers that are
collectively trusted to not collude in an attempt to defraud the participant.
Leveraging this network of trust, the XRPL algorithm has two main components.
The XRPL network is a decentralized and **trust-full** network. Anyone is free to join and participants are free to choose a subset of peers that are collectively trusted to not collude in an attempt to defraud the participant. Leveraging this network of trust, the XRPL algorithm has two main components.
- _Consensus_ in which network participants agree on the transactions to apply
to a prior ledger, based on the positions of their chosen peers.
- _Validation_ in which network participants agree on what ledger was
generated, based on the ledgers generated by chosen peers.
- _Consensus_ in which network participants agree on the transactions to apply to a prior ledger, based on the positions of their chosen peers.
- _Validation_ in which network participants agree on what ledger was generated, based on the ledgers generated by chosen peers.
These phases are continually repeated to process transactions submitted to the
network, generating successive ledgers and giving rise to the blockchain ledger
history depicted below. In this diagram, time is flowing to the right, but
links between ledgers point backward to the parent. Also note the alternate
Ledger 2 that was generated by some participants, but which failed validation
and was abandoned.
These phases are continually repeated to process transactions submitted to the network, generating successive ledgers and giving rise to the blockchain ledger history depicted below. In this diagram, time is flowing to the right, but links between ledgers point backward to the parent. Also note the alternate Ledger 2 that was generated by some participants, but which failed validation and was abandoned.
![Block Chain](images/consensus/block_chain.png "Block Chain")
The remainder of this section describes the Consensus and Validation algorithms
in more detail and is meant as a companion guide to understanding the generic
implementation in `xrpld`. The document **does not** discuss correctness,
fault-tolerance or liveness properties of the algorithms or the full details of
how they integrate within `xrpld` to support the XRPL consensus Ledger.
The remainder of this section describes the Consensus and Validation algorithms in more detail and is meant as a companion guide to understanding the generic implementation in `xrpld`. The document **does not** discuss correctness, fault-tolerance or liveness properties of the algorithms or the full details of how they integrate within `xrpld` to support the XRPL consensus Ledger.
## Consensus Overview
### Definitions
- The _ledger_ is the shared distributed state. Each ledger has a unique ID to
distinguish it from all other ledgers. During consensus, the _previous_,
_prior_ or _last-closed_ ledger is the most recent ledger seen by consensus
and is the basis upon which it will build the next ledger.
- A _transaction_ is an instruction for an atomic change in the ledger state. A
unique ID distinguishes a transaction from other transactions.
- A _transaction set_ is a set of transactions under consideration by consensus.
The goal of consensus is to reach agreement on this set. The generic
consensus algorithm does not rely on an ordering of transactions within the
set, nor does it specify how to apply a transaction set to a ledger to
generate a new ledger. A unique ID distinguishes a set of transactions from
all other sets of transactions.
- A _node_ is one of the distributed actors running the consensus algorithm. It
has a unique ID to distinguish it from all other nodes.
- A _peer_ of a node is another node that it has chosen to follow and which it
believes will not collude with other chosen peers. The choice of peers is not
symmetric, since participants can decide on their chosen sets independently.
- A /position/ is the current belief of the next ledger's transaction set and
close time. Position can refer to the node's own position or the position of a
peer.
- A _proposal_ is one of a sequence of positions a node shares during consensus.
An initial proposal contains the starting position taken by a node before it
considers any peer positions. If a node subsequently updates its position in
response to its peers, it will issue an updated proposal. A proposal is
uniquely identified by the ID of the proposing node, the ID of the position
taken, the ID of the prior ledger the proposal is for, and the sequence number
of the proposal.
- A _dispute_ is a transaction that is either not part of a node's position or
not in a peer's position. During consensus, the node will add or remove
disputed transactions from its position based on that transaction's support
amongst its peers.
- The _ledger_ is the shared distributed state. Each ledger has a unique ID to distinguish it from all other ledgers. During consensus, the _previous_, _prior_ or _last-closed_ ledger is the most recent ledger seen by consensus and is the basis upon which it will build the next ledger.
- A _transaction_ is an instruction for an atomic change in the ledger state. A unique ID distinguishes a transaction from other transactions.
- A _transaction set_ is a set of transactions under consideration by consensus. The goal of consensus is to reach agreement on this set. The generic consensus algorithm does not rely on an ordering of transactions within the set, nor does it specify how to apply a transaction set to a ledger to generate a new ledger. A unique ID distinguishes a set of transactions from all other sets of transactions.
- A _node_ is one of the distributed actors running the consensus algorithm. It has a unique ID to distinguish it from all other nodes.
- A _peer_ of a node is another node that it has chosen to follow and which it believes will not collude with other chosen peers. The choice of peers is not symmetric, since participants can decide on their chosen sets independently.
- A /position/ is the current belief of the next ledger's transaction set and close time. Position can refer to the node's own position or the position of a peer.
- A _proposal_ is one of a sequence of positions a node shares during consensus. An initial proposal contains the starting position taken by a node before it considers any peer positions. If a node subsequently updates its position in response to its peers, it will issue an updated proposal. A proposal is uniquely identified by the ID of the proposing node, the ID of the position taken, the ID of the prior ledger the proposal is for, and the sequence number of the proposal.
- A _dispute_ is a transaction that is either not part of a node's position or not in a peer's position. During consensus, the node will add or remove disputed transactions from its position based on that transaction's support amongst its peers.
Note that most types have an ID as a lightweight identifier of instances of that
type. Consensus often operates on the IDs directly since the underlying type is
potentially expensive to share over the network. For example, proposal's only
contain the ID of the position of a peer. Since many peers likely have the same
position, this reduces the need to send the full transaction set multiple times.
Instead, a node can request the transaction set from the network if necessary.
Note that most types have an ID as a lightweight identifier of instances of that type. Consensus often operates on the IDs directly since the underlying type is potentially expensive to share over the network. For example, proposal's only contain the ID of the position of a peer. Since many peers likely have the same position, this reduces the need to send the full transaction set multiple times. Instead, a node can request the transaction set from the network if necessary.
### Overview
![Consensus Overview](images/consensus/consensus_overview.png "Consensus Overview")
The diagram above is an overview of the consensus process from the perspective
of a single participant. Recall that during a single consensus round, a node is
trying to agree with its peers on which transactions to apply to its prior
ledger when generating the next ledger. It also attempts to agree on the
[network time when the ledger closed](#effective_close_time). There are
3 main phases to a consensus round:
The diagram above is an overview of the consensus process from the perspective of a single participant. Recall that during a single consensus round, a node is trying to agree with its peers on which transactions to apply to its prior ledger when generating the next ledger. It also attempts to agree on the [network time when the ledger closed](#effective_close_time). There are 3 main phases to a consensus round:
- A call to `startRound` places the node in the `Open` phase. In this phase,
the node is waiting for transactions to include in its open ledger.
- At some point, the node will `Close` the open ledger and transition to the
`Establish` phase. In this phase, the node shares/receives peer proposals on
which transactions should be accepted in the closed ledger.
- At some point, the node determines it has reached consensus with its peers on
which transactions to include. It transitions to the `Accept` phase. In this
phase, the node works on applying the transactions to the prior ledger to
generate a new closed ledger. Once the new ledger is completed, the node shares
the validated ledger hash with the network and makes a call to `startRound` to
start the cycle again for the next ledger.
- A call to `startRound` places the node in the `Open` phase. In this phase, the node is waiting for transactions to include in its open ledger.
- At some point, the node will `Close` the open ledger and transition to the `Establish` phase. In this phase, the node shares/receives peer proposals on which transactions should be accepted in the closed ledger.
- At some point, the node determines it has reached consensus with its peers on which transactions to include. It transitions to the `Accept` phase. In this phase, the node works on applying the transactions to the prior ledger to generate a new closed ledger. Once the new ledger is completed, the node shares the validated ledger hash with the network and makes a call to `startRound` to start the cycle again for the next ledger.
Throughout, a heartbeat timer calls `timerEntry` at a regular frequency to drive
the process forward. Although the `startRound` call occurs at arbitrary times
based on when the initial round began and the time it takes to apply
transactions, the transitions from `Open` to `Establish` and `Establish` to
`Accept` only occur during calls to `timerEntry`. Similarly, transactions can
arrive at arbitrary times, independent of the heartbeat timer. Transactions
received after the `Open` to `Close` transition and not part of peer proposals
won't be considered until the next consensus round. They are represented above
by the light green triangles.
Throughout, a heartbeat timer calls `timerEntry` at a regular frequency to drive the process forward. Although the `startRound` call occurs at arbitrary times based on when the initial round began and the time it takes to apply transactions, the transitions from `Open` to `Establish` and `Establish` to `Accept` only occur during calls to `timerEntry`. Similarly, transactions can arrive at arbitrary times, independent of the heartbeat timer. Transactions received after the `Open` to `Close` transition and not part of peer proposals won't be considered until the next consensus round. They are represented above by the light green triangles.
Peer proposals are issued by a node during a `timerEntry` call, but since peers
do not synchronize `timerEntry` calls, they are received by other peers at
arbitrary times. Peer proposals are only considered if received prior to the
`Establish` to `Accept` transition, and only if the peer is working on the same
prior ledger. Peer proposals received after consensus is reached will not be
meaningful and are represented above by the circle with the X in it. Only
proposals from chosen peers are considered.
Peer proposals are issued by a node during a `timerEntry` call, but since peers do not synchronize `timerEntry` calls, they are received by other peers at arbitrary times. Peer proposals are only considered if received prior to the `Establish` to `Accept` transition, and only if the peer is working on the same prior ledger. Peer proposals received after consensus is reached will not be meaningful and are represented above by the circle with the X in it. Only proposals from chosen peers are considered.
### Effective Close Time ### {#effective_close_time}
In addition to agreeing on a transaction set, each consensus round tries to
agree on the time the ledger closed. Each node calculates its own close time
when it closes the open ledger. This exact close time is rounded to the nearest
multiple of the current _effective close time resolution_. It is this
_effective close time_ that nodes seek to agree on. This allows servers to
derive a common time for a ledger without the need for perfectly synchronized
clocks. As depicted below, the 3 pink arrows represent exact close times from 3
consensus nodes that round to the same effective close time given the current
resolution. The purple arrow represents a peer whose estimate rounds to a
different effective close time given the current resolution.
In addition to agreeing on a transaction set, each consensus round tries to agree on the time the ledger closed. Each node calculates its own close time when it closes the open ledger. This exact close time is rounded to the nearest multiple of the current _effective close time resolution_. It is this _effective close time_ that nodes seek to agree on. This allows servers to derive a common time for a ledger without the need for perfectly synchronized clocks. As depicted below, the 3 pink arrows represent exact close times from 3 consensus nodes that round to the same effective close time given the current resolution. The purple arrow represents a peer whose estimate rounds to a different effective close time given the current resolution.
![Effective Close Time](images/consensus/EffCloseTime.png "Effective Close Time")
The effective close time is part of the node's position and is shared with peers
in its proposals. Just like the position on the consensus transaction set, a
node will update its close time position in response to its peers' effective
close time positions. Peers can agree to disagree on the close time, in which
case the effective close time is taken as 1 second past the prior close.
The effective close time is part of the node's position and is shared with peers in its proposals. Just like the position on the consensus transaction set, a node will update its close time position in response to its peers' effective close time positions. Peers can agree to disagree on the close time, in which case the effective close time is taken as 1 second past the prior close.
The close time resolution is itself dynamic, decreasing (coarser) resolution in
subsequent consensus rounds if nodes are unable to reach consensus on an
effective close time and increasing (finer) resolution if nodes consistently
reach close time consensus.
The close time resolution is itself dynamic, decreasing (coarser) resolution in subsequent consensus rounds if nodes are unable to reach consensus on an effective close time and increasing (finer) resolution if nodes consistently reach close time consensus.
### Modes
Internally, a node operates under one of the following consensus modes. Either
of the first two modes may be chosen when a consensus round starts.
Internally, a node operates under one of the following consensus modes. Either of the first two modes may be chosen when a consensus round starts.
- _Proposing_ indicates the node is a full-fledged consensus participant. It
takes on positions and sends proposals to its peers.
- _Observing_ indicates the node is a passive consensus participant. It
maintains a position internally, but does not propose that position to its
peers. Instead, it receives peer proposals and updates its position
to track the majority of its peers. This may be preferred if the node is only
being used to track the state of the network or during a start-up phase while
it is still synchronizing with the network.
- _Proposing_ indicates the node is a full-fledged consensus participant. It takes on positions and sends proposals to its peers.
- _Observing_ indicates the node is a passive consensus participant. It maintains a position internally, but does not propose that position to its peers. Instead, it receives peer proposals and updates its position to track the majority of its peers. This may be preferred if the node is only being used to track the state of the network or during a start-up phase while it is still synchronizing with the network.
The other two modes are set internally during the consensus round when the node
believes it is no longer working on the dominant ledger chain based on peer
validations. It checks this on every call to `timerEntry`.
The other two modes are set internally during the consensus round when the node believes it is no longer working on the dominant ledger chain based on peer validations. It checks this on every call to `timerEntry`.
- _Wrong Ledger_ indicates the node is not working on the correct prior ledger
and does not have it available. It requests that ledger from the network, but
continues to work towards consensus this round while waiting. If it had been
_proposing_, it will send a special "bow-out" proposal to its peers to indicate
its change in mode for the rest of this round. For the duration of the round,
it defers to peer positions for determining the consensus outcome as if it
were just _observing_.
- _Switch Ledger_ indicates that the node has acquired the correct prior ledger
from the network. Although it now has the correct prior ledger, the fact that
it had the wrong one at some point during this round means it is likely behind
and should defer to peer positions for determining the consensus outcome.
- _Wrong Ledger_ indicates the node is not working on the correct prior ledger and does not have it available. It requests that ledger from the network, but continues to work towards consensus this round while waiting. If it had been _proposing_, it will send a special "bow-out" proposal to its peers to indicate its change in mode for the rest of this round. For the duration of the round, it defers to peer positions for determining the consensus outcome as if it were just _observing_.
- _Switch Ledger_ indicates that the node has acquired the correct prior ledger from the network. Although it now has the correct prior ledger, the fact that it had the wrong one at some point during this round means it is likely behind and should defer to peer positions for determining the consensus outcome.
![Consensus Modes](images/consensus/consensus_modes.png "Consensus Modes")
Once either wrong ledger or switch ledger are reached, the node cannot
return to proposing or observing until the next consensus round. However,
the node could change its view of the correct prior ledger, so going from
switch ledger to wrong ledger and back again is possible.
Once either wrong ledger or switch ledger are reached, the node cannot return to proposing or observing until the next consensus round. However, the node could change its view of the correct prior ledger, so going from switch ledger to wrong ledger and back again is possible.
The distinction between the wrong and switched ledger modes arises because a
ledger's unique identifier may be known by a node before the ledger itself. This
reflects that fact that the data corresponding to a ledger may be large and take
time to share over the network, whereas the smaller ID could be shared in a peer
validation much more quickly. Distinguishing the two states allows the node to
decide how best to generate the next ledger once it declares consensus.
The distinction between the wrong and switched ledger modes arises because a ledger's unique identifier may be known by a node before the ledger itself. This reflects that fact that the data corresponding to a ledger may be large and take time to share over the network, whereas the smaller ID could be shared in a peer validation much more quickly. Distinguishing the two states allows the node to decide how best to generate the next ledger once it declares consensus.
### Phases
As depicted in the overview diagram, consensus is best viewed as a progression
through 3 phases. There are 4 public methods of the generic consensus algorithm
that determine this progression
As depicted in the overview diagram, consensus is best viewed as a progression through 3 phases. There are 4 public methods of the generic consensus algorithm that determine this progression
- `startRound` begins a consensus round.
- `timerEntry` is called at a regular frequency (`LEDGER_MIN_CLOSE`) and is the
only call to consensus that can change the phase from `Open` to `Establish`
or `Accept`.
- `peerProposal` is called whenever a peer proposal is received and is what
allows a node to update its position in a subsequent `timerEntry` call.
- `gotTxSet` is called when a transaction set is received from the network. This
is typically in response to a prior request from the node to acquire the
transaction set corresponding to a disagreeing peer's position.
- `timerEntry` is called at a regular frequency (`LEDGER_MIN_CLOSE`) and is the only call to consensus that can change the phase from `Open` to `Establish` or `Accept`.
- `peerProposal` is called whenever a peer proposal is received and is what allows a node to update its position in a subsequent `timerEntry` call.
- `gotTxSet` is called when a transaction set is received from the network. This is typically in response to a prior request from the node to acquire the transaction set corresponding to a disagreeing peer's position.
The following subsections describe each consensus phase in more detail and what
actions are taken in response to these calls.
The following subsections describe each consensus phase in more detail and what actions are taken in response to these calls.
#### Open
The `Open` phase is a quiescent period to allow transactions to build up in the
node's open ledger. The duration is a trade-off between latency and throughput.
A shorter window reduces the latency to generating the next ledger, but also
reduces transaction throughput due to fewer transactions accepted into the
ledger.
The `Open` phase is a quiescent period to allow transactions to build up in the node's open ledger. The duration is a trade-off between latency and throughput. A shorter window reduces the latency to generating the next ledger, but also reduces transaction throughput due to fewer transactions accepted into the ledger.
A call to `startRound` would forcibly begin the next consensus round, skipping
completion of the current round. This is not expected during normal operation.
Calls to `peerProposal` or `gotTxSet` simply store the proposal or transaction
set for use in the coming `Establish` phase.
A call to `startRound` would forcibly begin the next consensus round, skipping completion of the current round. This is not expected during normal operation. Calls to `peerProposal` or `gotTxSet` simply store the proposal or transaction set for use in the coming `Establish` phase.
A call to `timerEntry` first checks that the node is working on the correct
prior ledger. If not, it will update the mode and request the correct ledger.
Otherwise, the node checks whether to switch to the `Establish` phase and close
the ledger.
A call to `timerEntry` first checks that the node is working on the correct prior ledger. If not, it will update the mode and request the correct ledger. Otherwise, the node checks whether to switch to the `Establish` phase and close the ledger.
##### Ledger Close
Under normal circumstances, the open ledger period ends when one of the following
is true
Under normal circumstances, the open ledger period ends when one of the following is true
- if there are transactions in the open ledger and more than `LEDGER_MIN_CLOSE`
have elapsed. This is the typical behavior.
- if there are no open transactions and a suitably longer idle interval has
elapsed. This increases the opportunity to get some transaction into
the next ledger and avoids doing useless work closing an empty ledger.
- if more than half the number of prior round peers have already closed or finished
this round. This indicates the node is falling behind and needs to catch up.
- if there are transactions in the open ledger and more than `LEDGER_MIN_CLOSE` have elapsed. This is the typical behavior.
- if there are no open transactions and a suitably longer idle interval has elapsed. This increases the opportunity to get some transaction into the next ledger and avoids doing useless work closing an empty ledger.
- if more than half the number of prior round peers have already closed or finished this round. This indicates the node is falling behind and needs to catch up.
When closing the ledger, the node takes its initial position based on the
transactions in the open ledger and uses the current time as
its initial close time estimate. If in the proposing mode, the node shares its
initial position with peers. Now that the node has taken a position, it will
consider any peer positions for this round that arrived earlier. The node
generates disputed transactions for each transaction not in common with a peer's
position. The node also records the vote of each peer for each disputed
transaction.
When closing the ledger, the node takes its initial position based on the transactions in the open ledger and uses the current time as its initial close time estimate. If in the proposing mode, the node shares its initial position with peers. Now that the node has taken a position, it will consider any peer positions for this round that arrived earlier. The node generates disputed transactions for each transaction not in common with a peer's position. The node also records the vote of each peer for each disputed transaction.
In the example below, we suppose our node has closed with transactions 1,2 and 3. It creates disputes
for transactions 2,3 and 4, since at least one peer position differs on each.
In the example below, we suppose our node has closed with transactions 1,2 and 3. It creates disputes for transactions 2,3 and 4, since at least one peer position differs on each.
##### disputes ##### {#disputes_image}
@@ -280,119 +119,55 @@ for transactions 2,3 and 4, since at least one peer position differs on each.
#### Establish
The establish phase is the active period of consensus in which the node
exchanges proposals with peers in an attempt to reach agreement on the consensus
transactions and effective close time.
The establish phase is the active period of consensus in which the node exchanges proposals with peers in an attempt to reach agreement on the consensus transactions and effective close time.
A call to `startRound` would forcibly begin the next consensus round, skipping
completion of the current round. This is not expected during normal operation.
Calls to `peerProposal` or `gotTxSet` that reflect new positions will generate
disputed transactions for any new disagreements and will update the peer's vote
for all disputed transactions.
A call to `startRound` would forcibly begin the next consensus round, skipping completion of the current round. This is not expected during normal operation. Calls to `peerProposal` or `gotTxSet` that reflect new positions will generate disputed transactions for any new disagreements and will update the peer's vote for all disputed transactions.
A call to `timerEntry` first checks that the node is working from the correct
prior ledger. If not, the node will update the mode and request the correct
ledger. Otherwise, the node updates the node's position and considers whether
to switch to the `Accepted` phase and declare consensus reached. However, at
least `LEDGER_MIN_CONSENSUS` time must have elapsed before doing either. This
allows peers an opportunity to take an initial position and share it.
A call to `timerEntry` first checks that the node is working from the correct prior ledger. If not, the node will update the mode and request the correct ledger. Otherwise, the node updates the node's position and considers whether to switch to the `Accepted` phase and declare consensus reached. However, at least `LEDGER_MIN_CONSENSUS` time must have elapsed before doing either. This allows peers an opportunity to take an initial position and share it.
##### Update Position
In order to achieve consensus, the node is looking for a transaction set that is
supported by a super-majority of peers. The node works towards this set by
adding or removing disputed transactions from its position based on an
increasing threshold for inclusion.
In order to achieve consensus, the node is looking for a transaction set that is supported by a super-majority of peers. The node works towards this set by adding or removing disputed transactions from its position based on an increasing threshold for inclusion.
![Threshold](images/consensus/threshold.png "Threshold")
By starting with a lower threshold, a node initially allows a wide set of
transactions into its position. If the establish round continues and the node is
"stuck", a higher threshold can focus on accepting transactions with the most
support. The constants that define the thresholds and durations at which the
thresholds change are given by `AV_XXX_CONSENSUS_PCT` and
`AV_XXX_CONSENSUS_TIME` respectively, where `XXX` is `INIT`,`MID`,`LATE` and
`STUCK`. The effective close time position is updated using the same
thresholds.
By starting with a lower threshold, a node initially allows a wide set of transactions into its position. If the establish round continues and the node is "stuck", a higher threshold can focus on accepting transactions with the most support. The constants that define the thresholds and durations at which the thresholds change are given by `AV_XXX_CONSENSUS_PCT` and `AV_XXX_CONSENSUS_TIME` respectively, where `XXX` is `INIT`,`MID`,`LATE` and `STUCK`. The effective close time position is updated using the same thresholds.
Given the [example disputes above](#disputes_image) and an initial threshold
of 50%, our node would retain its position since transaction 1 was not in
dispute and transactions 2 and 3 have 75% support. Since its position did not
change, it would not need to send a new proposal to peers. Peer C would not
change either. Peer A would add transaction 3 to its position and Peer B would
remove transaction 4 from its position; both would then send an updated
position.
Given the [example disputes above](#disputes_image) and an initial threshold of 50%, our node would retain its position since transaction 1 was not in dispute and transactions 2 and 3 have 75% support. Since its position did not change, it would not need to send a new proposal to peers. Peer C would not change either. Peer A would add transaction 3 to its position and Peer B would remove transaction 4 from its position; both would then send an updated position.
Conversely, if the diagram reflected a later call to =timerEntry= that occurs in
the stuck region with a threshold of say 95%, our node would remove transactions
2 and 3 from its candidate set and send an updated position. Likewise, all the
other peers would end up with only transaction 1 in their position.
Conversely, if the diagram reflected a later call to =timerEntry= that occurs in the stuck region with a threshold of say 95%, our node would remove transactions 2 and 3 from its candidate set and send an updated position. Likewise, all the other peers would end up with only transaction 1 in their position.
Lastly, if our node were not in the proposing mode, it would not include its own
vote and just take the majority (>50%) position of its peers. In this example,
our node would maintain its position of transactions 1, 2 and 3.
Lastly, if our node were not in the proposing mode, it would not include its own vote and just take the majority (>50%) position of its peers. In this example, our node would maintain its position of transactions 1, 2 and 3.
##### Checking Consensus
After updating its position, the node checks for supermajority agreement with
its peers on its current position. This agreement is of the exact transaction
set, not just the support of individual transactions. That is, if our position
is a subset of a peer's position, that counts as a disagreement. Also recall
that effective close time agreement allows a supermajority of participants
agreeing to disagree.
After updating its position, the node checks for supermajority agreement with its peers on its current position. This agreement is of the exact transaction set, not just the support of individual transactions. That is, if our position is a subset of a peer's position, that counts as a disagreement. Also recall that effective close time agreement allows a supermajority of participants agreeing to disagree.
Consensus is declared when the following 3 clauses are true:
- `LEDGER_MIN_CONSENSUS` time has elapsed in the establish phase
- At least 75% of the prior round proposers have proposed OR this establish
phase is `LEDGER_MIN_CONSENSUS` longer than the last round's establish phase
- At least 75% of the prior round proposers have proposed OR this establish phase is `LEDGER_MIN_CONSENSUS` longer than the last round's establish phase
- `minimumConsensusPercentage` of ourself and our peers share the same position
The middle condition ensures slower peers have a chance to share positions, but
prevents waiting too long on peers that have disconnected. Additionally, a node
can declare that consensus has moved on if `minimumConsensusPercentage` peers
have sent validations and moved on to the next ledger. This outcome indicates
the node has fallen behind its peers and needs to catch up.
The middle condition ensures slower peers have a chance to share positions, but prevents waiting too long on peers that have disconnected. Additionally, a node can declare that consensus has moved on if `minimumConsensusPercentage` peers have sent validations and moved on to the next ledger. This outcome indicates the node has fallen behind its peers and needs to catch up.
If a node is not proposing, it does not include its own position when
calculating the percent of agreeing participants but otherwise follows the above
logic.
If a node is not proposing, it does not include its own position when calculating the percent of agreeing participants but otherwise follows the above logic.
##### Accepting Consensus
Once consensus is reached (or moved on), the node switches to the `Accept` phase
and signals to the implementing code that the round is complete. That code is
responsible for using the consensus transaction set to generate the next ledger
and calling `startRound` to begin the next round. The implementation has total
freedom on ordering transactions, deciding what to do if consensus moved on,
determining whether to retry or abandon local transactions that did not make the
consensus set and updating any internal state based on the consensus progress.
Once consensus is reached (or moved on), the node switches to the `Accept` phase and signals to the implementing code that the round is complete. That code is responsible for using the consensus transaction set to generate the next ledger and calling `startRound` to begin the next round. The implementation has total freedom on ordering transactions, deciding what to do if consensus moved on, determining whether to retry or abandon local transactions that did not make the consensus set and updating any internal state based on the consensus progress.
#### Accept
The `Accept` phase is the terminal phase of the consensus algorithm. Calls to
`timerEntry`, `peerProposal` and `gotTxSet` will not change the internal
consensus state while in the accept phase. The expectation is that the
application specific code is working to generate the new ledger based on the
consensus outcome. Once complete, that code should make a call to `startRound`
to kick off the next consensus round. The `startRound` call includes the new
prior ledger, prior ledger ID and whether the round should begin in the
proposing or observing mode. After setting some initial state, the phase
transitions to `Open`. The node will also check if the provided prior ledger
and ID are correct, updating the mode and requesting the proper ledger from the
network if necessary.
The `Accept` phase is the terminal phase of the consensus algorithm. Calls to `timerEntry`, `peerProposal` and `gotTxSet` will not change the internal consensus state while in the accept phase. The expectation is that the application specific code is working to generate the new ledger based on the consensus outcome. Once complete, that code should make a call to `startRound` to kick off the next consensus round. The `startRound` call includes the new prior ledger, prior ledger ID and whether the round should begin in the proposing or observing mode. After setting some initial state, the phase transitions to `Open`. The node will also check if the provided prior ledger and ID are correct, updating the mode and requesting the proper ledger from the network if necessary.
## Consensus Type Requirements
The consensus type requirements are given below as minimal implementation stubs.
Actual implementations would augment these stubs with members appropriate for
managing the details of transactions and ledgers within the larger application
framework.
The consensus type requirements are given below as minimal implementation stubs. Actual implementations would augment these stubs with members appropriate for managing the details of transactions and ledgers within the larger application framework.
### Transaction
The transaction type `Tx` encapsulates a single transaction under consideration
by consensus.
The transaction type `Tx` encapsulates a single transaction under consideration by consensus.
```{.cpp}
struct Tx
@@ -406,10 +181,7 @@ struct Tx
### Transaction Set
The transaction set type `TxSet` represents a set of `Tx`s that are collectively
under consideration by consensus. A `TxSet` can be compared against other `TxSet`s
(typically from peers) and can be modified to add or remove transactions via
the mutable subtype.
The transaction set type `TxSet` represents a set of `Tx`s that are collectively under consideration by consensus. A `TxSet` can be compared against other `TxSet`s (typically from peers) and can be modified to add or remove transactions via the mutable subtype.
```{.cpp}
struct TxSet
@@ -446,12 +218,7 @@ struct TxSet
### Ledger
The `Ledger` type represents the state shared amongst the
distributed participants. Notice that the details of how the next ledger is
generated from the prior ledger and the consensus accepted transaction set is
not part of the interface. Within the generic code, this type is primarily used
to know that peers are working on the same tip of the ledger chain and to
provide some basic timing data for consensus.
The `Ledger` type represents the state shared amongst the distributed participants. Notice that the details of how the next ledger is generated from the prior ledger and the consensus accepted transaction set is not part of the interface. Within the generic code, this type is primarily used to know that peers are working on the same tip of the ledger chain and to provide some basic timing data for consensus.
```{.cpp}
struct Ledger
@@ -485,9 +252,7 @@ struct Ledger
### PeerProposal
The `PeerProposal` type represents the signed position taken
by a peer during consensus. The only type requirement is owning an instance of a
generic `ConsensusProposal`.
The `PeerProposal` type represents the signed position taken by a peer during consensus. The only type requirement is owning an instance of a generic `ConsensusProposal`.
```{.cpp}
// Represents our proposed position or a peer's proposed position
@@ -508,11 +273,7 @@ struct PeerPosition
### Generic Consensus Interface
The generic `Consensus` relies on `Adaptor` template class to implement a set
of helper functions that plug the consensus algorithm into a specific application.
The `Adaptor` class also defines the types above needed by the algorithm. Below
are excerpts of the generic consensus implementation and of helper types that will
interact with the concrete implementing class.
The generic `Consensus` relies on `Adaptor` template class to implement a set of helper functions that plug the consensus algorithm into a specific application. The `Adaptor` class also defines the types above needed by the algorithm. Below are excerpts of the generic consensus implementation and of helper types that will interact with the concrete implementing class.
```{.cpp}
// Represents a transaction under dispute this round
@@ -653,26 +414,11 @@ struct Adaptor
};
```
The implementing class hides many details of the peer communication
model from the generic code.
The implementing class hides many details of the peer communication model from the generic code.
- The `share` member functions are responsible for sharing the given type with a
node's peers, but are agnostic to the mechanism. Ideally, messages are delivered
faster than `LEDGER_GRANULARITY`.
- The generic code does not specify how transactions are submitted by clients,
propagated through the network or stored in the open ledger. Indeed, the open
ledger is only conceptual from the perspective of the generic code---the
initial position and transaction set are opaquely generated in a
`Consensus::Result` instance returned from the `onClose` callback.
- The calls to `acquireLedger` and `acquireTxSet` only have non-trivial return
if the ledger or transaction set of interest is available. The implementing
class is free to block while acquiring, or return the empty option while
servicing the request asynchronously. Due to legacy reasons, the two calls
are not symmetric. `acquireTxSet` requires the host application to call
`gotTxSet` when an asynchronous `acquire` completes. Conversely,
`acquireLedger` will be called again later by the consensus code if it still
desires the ledger with the hope that the asynchronous acquisition is
complete.
- The `share` member functions are responsible for sharing the given type with a node's peers, but are agnostic to the mechanism. Ideally, messages are delivered faster than `LEDGER_GRANULARITY`.
- The generic code does not specify how transactions are submitted by clients, propagated through the network or stored in the open ledger. Indeed, the open ledger is only conceptual from the perspective of the generic code---the initial position and transaction set are opaquely generated in a `Consensus::Result` instance returned from the `onClose` callback.
- The calls to `acquireLedger` and `acquireTxSet` only have non-trivial return if the ledger or transaction set of interest is available. The implementing class is free to block while acquiring, or return the empty option while servicing the request asynchronously. Due to legacy reasons, the two calls are not symmetric. `acquireTxSet` requires the host application to call `gotTxSet` when an asynchronous `acquire` completes. Conversely, `acquireLedger` will be called again later by the consensus code if it still desires the ledger with the hope that the asynchronous acquisition is complete.
## Validation

View File

@@ -1,22 +1,20 @@
This document contains instructions for installing xrpld.
The APT package manager is common on Debian-based Linux distributions like
Ubuntu,
while the YUM package manager is common on Red Hat-based Linux distributions
like CentOS.
Installing from source is an option for all platforms,
and the only supported option for installing custom builds.
# Installing xrpld 3.3.0 and earlier
<!-- prettier-ignore -->
> [!IMPORTANT]
> These instructions apply to xrpld 3.3.0 and earlier, published to repos.ripple.com. For later releases see [install.md](./install.md).
This document contains instructions for installing xrpld. The APT package manager is common on Debian-based Linux distributions like Ubuntu, while the YUM package manager is common on Red Hat-based Linux distributions like CentOS. Installing from source is an option for all platforms, and the only supported option for installing custom builds.
## From source
From a source build, you can install xrpld and libxrpl using CMake's
`--install` mode:
From a source build, you can install xrpld and libxrpl using CMake's `--install` mode:
```
cmake --install . --prefix /opt/local
```
The default [prefix][1] is typically `/usr/local` on Linux and macOS and
`C:/Program Files/xrpld` on Windows.
The default [prefix][1] is typically `/usr/local` on Linux and macOS and `C:/Program Files/xrpld` on Windows.
[1]: https://cmake.org/cmake/help/latest/variable/CMAKE_INSTALL_PREFIX.html
@@ -52,7 +50,7 @@ The default [prefix][1] is typically `/usr/local` on Linux and macOS and
5. Add the appropriate XRPL repository for your operating system version:
echo "deb [signed-by=/usr/local/share/keyrings/ripple-key.gpg] https://repos.ripple.com/repos/xrpld-deb focal stable" | \
echo "deb [signed-by=/usr/local/share/keyrings/ripple-key.gpg] https://repos.ripple.com/repos/rippled-deb focal stable" | \
sudo tee -a /etc/apt/sources.list.d/ripple.list
The above example is appropriate for **Ubuntu 20.04 Focal Fossa**. For other operating systems, replace the word `focal` with one of the following:
@@ -106,8 +104,8 @@ The default [prefix][1] is typically `/usr/local` on Linux and macOS and
enabled=1
gpgcheck=0
repo_gpgcheck=1
baseurl=https://repos.ripple.com/repos/xrpld-rpm/stable/
gpgkey=https://repos.ripple.com/repos/xrpld-rpm/stable/repodata/repomd.xml.key
baseurl=https://repos.ripple.com/repos/rippled-rpm/stable/
gpgkey=https://repos.ripple.com/repos/rippled-rpm/stable/repodata/repomd.xml.key
REPOFILE
_Unstable_
@@ -118,8 +116,8 @@ The default [prefix][1] is typically `/usr/local` on Linux and macOS and
enabled=1
gpgcheck=0
repo_gpgcheck=1
baseurl=https://repos.ripple.com/repos/xrpld-rpm/unstable/
gpgkey=https://repos.ripple.com/repos/xrpld-rpm/unstable/repodata/repomd.xml.key
baseurl=https://repos.ripple.com/repos/rippled-rpm/unstable/
gpgkey=https://repos.ripple.com/repos/rippled-rpm/unstable/repodata/repomd.xml.key
REPOFILE
_Nightly_
@@ -130,8 +128,8 @@ The default [prefix][1] is typically `/usr/local` on Linux and macOS and
enabled=1
gpgcheck=0
repo_gpgcheck=1
baseurl=https://repos.ripple.com/repos/xrpld-rpm/nightly/
gpgkey=https://repos.ripple.com/repos/xrpld-rpm/nightly/repodata/repomd.xml.key
baseurl=https://repos.ripple.com/repos/rippled-rpm/nightly/
gpgkey=https://repos.ripple.com/repos/rippled-rpm/nightly/repodata/repomd.xml.key
REPOFILE
2. Fetch the latest repo updates:

136
docs/install.md Normal file
View File

@@ -0,0 +1,136 @@
# Installing xrpld
<!-- prettier-ignore -->
> [!NOTE]
> These instructions apply to packages published from 2026-08-19 onwards. For xrpld 3.3.0 and earlier see [install-legacy.md](./install-legacy.md).
`xrpld` is published as DEB and RPM packages for 64-bit x86 Linux. Use APT on Debian-based distributions such as Debian and Ubuntu, and YUM on Red Hat-based distributions such as RHEL, AlmaLinux, and Rocky Linux. To build from source instead, see [BUILD.md](../BUILD.md).
## Release channels
Packages are published to four channels:
- `stable` - production releases
- `rc` - release candidates
- `beta` - beta builds
- `develop` - every push to the [`develop` branch](https://github.com/XRPLF/rippled/tree/develop)
See [Publishing packages](../package/README.md#publishing-packages) for how channels are produced.
The instructions below use `stable`. To follow another channel, replace `stable` with its name wherever it appears in the repository configuration.
<!-- prettier-ignore -->
> [!WARNING]
> Channels other than `stable` may be broken at any time. Do not use them for production servers.
## Install the xrpld package
### With the APT package manager
1. Install utilities:
```bash
sudo apt update -y
sudo apt install -y apt-transport-https ca-certificates curl gnupg
```
2. Add the XRPL Foundation package-signing key to your list of trusted keys:
```bash
sudo install -d -m 0755 /etc/apt/keyrings
sudo curl -fsS https://packages.xrplf.org/xrplf.asc -o /etc/apt/keyrings/xrplf.asc
```
3. Check the fingerprint of the newly-added key:
```bash
gpg --show-keys /etc/apt/keyrings/xrplf.asc
```
The output should be:
```text
pub rsa4096 2026-08-18 [SC]
B655416741221F780FBCFBC9AA84D41A11D29FA9
uid XRPLF Packages <distribution@xrplf.org>
```
In particular, make sure that the fingerprint matches.
4. Add the repository, using the channel you picked in [Release channels](#release-channels):
```bash
echo "deb [signed-by=/etc/apt/keyrings/xrplf.asc] https://packages.xrplf.org/repository/deb-stable any main" | \
sudo tee /etc/apt/sources.list.d/xrplf.list
```
5. Fetch the repository:
```bash
sudo apt -y update
```
6. Install the `xrpld` software package:
```bash
sudo apt -y install xrpld
```
### With the YUM package manager
1. Add the XRPL Foundation package-signing key:
```bash
sudo rpm --import https://packages.xrplf.org/xrplf.asc
```
2. Add the repository, using the channel you picked in [Release channels](#release-channels):
```bash
cat << 'REPOFILE' | sudo tee /etc/yum.repos.d/xrplf.repo
[xrplf-stable]
name=XRP Ledger Packages
enabled=1
baseurl=https://packages.xrplf.org/repository/rpm-stable/$basearch/
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.xrplf.org/xrplf.asc
REPOFILE
```
`gpgcheck=1` verifies each package against the key above. `repo_gpgcheck=1` verifies the repository metadata, which the server signs with the same key.
3. Install the `xrpld` package:
```bash
sudo yum install -y xrpld
```
## The xrpld service
Both package managers install a systemd unit and enable it, so `xrpld` starts on boot. Check whether it is already running:
```bash
systemctl status xrpld.service
```
The APT packages start it immediately as well; the YUM packages do not, so start it yourself:
```bash
sudo systemctl start xrpld.service
```
### Optional: binding to privileged ports
To serve incoming API requests on port 80 or 443, grant the service the capability to bind them. You must also update the config file's port settings.
```bash
sudo install -d -m 0755 /etc/systemd/system/xrpld.service.d
sudo tee /etc/systemd/system/xrpld.service.d/privileged-ports.conf >/dev/null <<'EOF'
[Service]
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
EOF
sudo systemctl daemon-reload
sudo systemctl restart xrpld.service
```

View File

@@ -15,8 +15,7 @@ The module xrpl/basics should contain no dependencies on other modules.
- `std::list`
- For ordered containers with inserts and erases to the middle.
- For containers with iterators stable over insert and erase.
- Generally slower and bigger than `std::vector` or `std::deque` except for
those cases.
- Generally slower and bigger than `std::vector` or `std::deque` except for those cases.
- `std::set`
- For sorted containers.
@@ -26,9 +25,7 @@ The module xrpl/basics should contain no dependencies on other modules.
- For "small" sets, `std::set` might be faster and smaller.
- `xrpl::hardened_hash_set`
- For data sets where the key could be manipulated by an attacker
in an attempt to mount an algorithmic complexity attack: see
http://en.wikipedia.org/wiki/Algorithmic_complexity_attack
- For data sets where the key could be manipulated by an attacker in an attempt to mount an algorithmic complexity attack: see http://en.wikipedia.org/wiki/Algorithmic_complexity_attack
The following container is deprecated

View File

@@ -103,7 +103,7 @@ namespace boost {
template <>
struct hash<::beast::ip::Address>
{
explicit hash() = default;
hash() = default;
std::size_t
operator()(::beast::ip::Address const& addr) const

View File

@@ -125,6 +125,7 @@ struct Keys
static constexpr auto kMaximumTxnInLedger = "maximum_txn_in_ledger";
static constexpr auto kMaximumTxnPerAccount = "maximum_txn_per_account";
static constexpr auto kMemoryLevel = "memory_level";
static constexpr auto kMaxWaitingLedgers = "max_waiting_ledgers";
static constexpr auto kMinLedgersToComputeSizeLimit = "min_ledgers_to_compute_size_limit";
static constexpr auto kMinimumEscalationMultiplier = "minimum_escalation_multiplier";
static constexpr auto kMinimumLastLedgerBuffer = "minimum_last_ledger_buffer";

View File

@@ -1,8 +1,3 @@
# Consensus
This directory contains the implementation of a
generic consensus algorithm. The implementation
follows a CRTP design, requiring client code to implement
specific functions and types to use consensus in their
application. The interface is undergoing refactoring and
is not yet finalized.
This directory contains the implementation of a generic consensus algorithm. The implementation follows a CRTP design, requiring client code to implement specific functions and types to use consensus in their application. The interface is undergoing refactoring and is not yet finalized.

View File

@@ -1,4 +1,3 @@
# SSLUtil
This module exposes the OpenSSL headers and provides utilities to
operate with OpenSSL / BIGNUM objects.
This module exposes the OpenSSL headers and provides utilities to operate with OpenSSL / BIGNUM objects.

View File

@@ -24,6 +24,7 @@
#include <optional>
#include <set>
#include <utility>
#include <vector>
namespace xrpl {
@@ -198,7 +199,10 @@ dirLink(
* if withdrawing to self.
* - If withdrawing to self, succeed.
* - If not, checks if the receiver requires deposit authorization, and if
* the sender has it.
* the sender has it (account-based or credential-based).
* - Expects any credentials passed in to already exist in the ledger, and
* returns an internal error otherwise. Validate them beforehand with
* credentials::valid().
* - Checks that the receiver will not exceed the limit (IOU trustline limit
* or MPT MaximumAmount).
*/
@@ -209,7 +213,8 @@ canWithdraw(
AccountID const& to,
SLE::const_ref toSle,
STAmount const& amount,
bool hasDestinationTag);
bool hasDestinationTag,
std::optional<std::vector<uint256>> const& credentialIDs = std::nullopt);
/**
* Checks that can withdraw funds from an object to itself or a destination.
@@ -222,7 +227,10 @@ canWithdraw(
* if withdrawing to self.
* - If withdrawing to self, succeed.
* - If not, checks if the receiver requires deposit authorization, and if
* the sender has it.
* the sender has it (account-based or credential-based).
* - Expects any credentials passed in to already exist in the ledger, and
* returns an internal error otherwise. Validate them beforehand with
* credentials::valid().
* - Checks that the receiver will not exceed the limit (IOU trustline limit
* or MPT MaximumAmount).
*/
@@ -232,20 +240,25 @@ canWithdraw(
AccountID const& from,
AccountID const& to,
STAmount const& amount,
bool hasDestinationTag);
bool hasDestinationTag,
std::optional<std::vector<uint256>> const& credentialIDs = std::nullopt);
/**
* Checks that can withdraw funds from an object to itself or a destination.
*
* The receiver may be either the submitting account (sfAccount) or a different
* destination account (sfDestination).
* destination account (sfDestination). Credentials, if any, are taken from the
* transaction's sfCredentialIDs field.
*
* - Checks that the receiver account exists.
* - If the receiver requires a destination tag, check that one exists, even
* if withdrawing to self.
* - If withdrawing to self, succeed.
* - If not, checks if the receiver requires deposit authorization, and if
* the sender has it.
* the sender has it (account-based or credential-based).
* - Expects any credentials in sfCredentialIDs to already exist in the
* ledger, and returns an internal error otherwise. Validate them
* beforehand with credentials::valid().
* - Checks that the receiver will not exceed the limit (IOU trustline limit
* or MPT MaximumAmount).
*/

View File

@@ -15,7 +15,6 @@
#include <cstdint>
#include <expected>
#include <optional>
#include <set>
#include <vector>
namespace xrpl {
@@ -353,14 +352,14 @@ pseudoAccountAddress(ReadView const& view, uint256 const& pseudoOwnerKey);
*
* The list is constructed during initialization and is const after that.
* Pseudo-account designator fields MUST be maintained by including the
* SField::sMD_PseudoAccount flag in the SField definition.
* SField::kSmdPseudoAccount flag in the SField definition.
*/
[[nodiscard]] std::vector<SField const*> const&
getPseudoAccountFields();
/**
* Returns true if and only if sleAcct is a pseudo-account or specific
* pseudo-accounts in pseudoFieldFilter.
* Returns true if and only if sleAcct is a pseudo-account of any kind
* (i.e. carries at least one field flagged with SField::kSmdPseudoAccount).
*
* Returns false if sleAcct is:
* - NOT a pseudo-account OR
@@ -368,18 +367,15 @@ getPseudoAccountFields();
* - null pointer
*/
[[nodiscard]] bool
isPseudoAccount(SLE::const_pointer sleAcct, std::set<SField const*> const& pseudoFieldFilter = {});
isPseudoAccount(SLE::const_pointer sleAcct);
/**
* Convenience overload that reads the account from the view.
*/
[[nodiscard]] inline bool
isPseudoAccount(
ReadView const& view,
AccountID const& accountId,
std::set<SField const*> const& pseudoFieldFilter = {})
isPseudoAccount(ReadView const& view, AccountID const& accountId)
{
return isPseudoAccount(view.read(keylet::account(accountId)), pseudoFieldFilter);
return isPseudoAccount(view.read(keylet::account(accountId)));
}
/**

View File

@@ -14,6 +14,7 @@
#include <xrpl/protocol/STVector256.h>
#include <xrpl/protocol/TER.h>
#include <cstdint>
#include <memory>
#include <set>
#include <utility>
@@ -33,6 +34,32 @@ checkExpired(SLE const& sleCredential, NetClock::time_point const& closed);
[[nodiscard]] TER
deleteSLE(ApplyView& view, SLE::ref sleCredential, beast::Journal j);
/**
* @brief Remove credentials pinned to a pseudo-account's owner directory.
*
* Cleans up credentials that were linked to a pseudo-account (Vault, LoanBroker,
* AMM), which such an account can neither accept nor delete. Only credentials
* are removed; every other object is left in place. The walk visits at most
* @p maxNodesToDelete directory entries and charges the ones it leaves alone
* against that budget too, so a directory holding other objects yields fewer
* than @p maxNodesToDelete deletions. On reaching the bound the result is
* `tecINCOMPLETE` and the caller must propagate it so a later transaction
* resumes.
*
* @param view Mutable ledger view.
* @param pseudoAcct The pseudo-account whose directory is cleaned.
* @param maxNodesToDelete Upper bound on directory entries processed in one call.
* @param j Journal for diagnostics.
* @return tesSUCCESS once no credentials remain, tecINCOMPLETE if the bound was
* reached, or a deletion error.
*/
[[nodiscard]] TER
deletePseudoAccountCredentials(
ApplyView& view,
AccountID const& pseudoAcct,
std::uint16_t maxNodesToDelete,
beast::Journal j);
// Amendment and parameters checks for sfCredentialIDs field
NotTEC
checkFields(STTx const& tx, Rules const& rules, beast::Journal j);

View File

@@ -7,6 +7,7 @@
#include <xrpl/beast/utility/instrumentation.h>
#include <xrpl/ledger/ApplyView.h>
#include <xrpl/ledger/ReadView.h>
#include <xrpl/protocol/AccountID.h>
#include <xrpl/protocol/Asset.h>
#include <xrpl/protocol/LedgerFormats.h> // IWYU pragma: keep
#include <xrpl/protocol/Protocol.h>
@@ -21,6 +22,7 @@
#include <cstdint>
#include <expected>
#include <optional>
#include <string_view>
#include <utility>
@@ -58,6 +60,42 @@ canApplyToBrokerCover(
bool
checkLendingProtocolDependencies(Rules const& rules, STTx const& tx);
/**
* The accounts and asset that LoanManage::defaultLoan's fixCleanup3_4_0
* freeze/lock exemption applies to.
*
* `defaultLoan` moves funds from the LoanBroker pseudo-account to the Vault
* pseudo-account via `accountSend`. Since neither is the vault asset's
* issuer, this is a third-party transfer that transits through the issuer in
* two hops (broker -> issuer, issuer -> vault; see
* `directSendNoLimitIOU`/`directSendNoLimitMPT`), so the exemption must cover
* both the issuer/broker and issuer/vault pairs, not a direct broker/vault
* pair. `asset` scopes it further to the vault's own currency/MPT issuance,
* so an unrelated one the same accounts happen to hold is still protected.
*/
struct LoanDefaultFreezeExemptAccounts
{
AccountID issuer;
AccountID broker;
AccountID vault;
Asset asset;
};
/**
* Resolves the accounts and asset a LoanManage default transaction is
* exempt from freeze/lock for.
*
* @param view Ledger view used to resolve the Loan -> LoanBroker -> Vault
* chain.
* @param tx The transaction under invariant review.
* @return The exempt accounts and asset if `tx` is a `ttLOAN_MANAGE`
* transaction with the `tfLoanDefault` flag set, `fixCleanup3_4_0` is
* enabled, and the loan/broker/vault objects it references can all be
* resolved; `std::nullopt` otherwise.
*/
[[nodiscard]] std::optional<LoanDefaultFreezeExemptAccounts>
getLoanDefaultFreezeExemptAccounts(ReadView const& view, STTx const& tx);
static constexpr std::uint32_t kSecondsInYear = 365 * 24 * 60 * 60;
Number
@@ -286,6 +324,12 @@ computeFullPaymentInterest(
std::uint32_t startDate,
TenthBips32 closeInterestRate);
// Returns true if the loan's next payment is late per protocol rules. The
// boundary is amendment-gated: with fixCleanup3_4_0 the due date must be
// strictly in the past, otherwise the exact due-date instant counts as late.
[[nodiscard]] bool
isPaymentLate(ReadView const& view, SLE::const_ref loanSle);
// Deltas applied to Vault.AssetsTotal and LoanBroker.DebtTotal at a single
// accounting touch point (origination, payment, impair/unimpair/default).
struct AccountingDeltas

View File

@@ -294,6 +294,14 @@ accountFunds(
AuthHandling authHandling,
beast::Journal j);
/**
* Returns the transfer fee as Rate based on the type of token
* @param view The ledger view
* @param asset The asset being transferred
*/
[[nodiscard]] Rate
transferRate(ReadView const& view, Asset const& asset);
/**
* Returns the transfer fee as Rate based on the type of token
* @param view The ledger view

View File

@@ -1,12 +1,16 @@
#pragma once
#include <xrpl/basics/Number.h>
#include <xrpl/ledger/ReadView.h>
#include <xrpl/protocol/AccountID.h>
#include <xrpl/protocol/Asset.h>
#include <xrpl/protocol/Protocol.h>
#include <xrpl/protocol/STAmount.h>
#include <xrpl/protocol/STLedgerEntry.h>
#include <xrpl/protocol/TER.h>
#include <cstdint>
#include <expected>
#include <optional>
namespace xrpl {
@@ -41,6 +45,32 @@ assetsToSharesDeposit(SLE::const_ref vault, SLE::const_ref issuance, STAmount co
[[nodiscard]] std::optional<STAmount>
sharesToAssetsDeposit(SLE::const_ref vault, SLE::const_ref issuance, STAmount const& shares);
/**
* Adjusts a requested asset change (`delta`) to match the decimal scale of the
* updated total vault assets. This ensures `sfAssetsTotal`, `sfAssetsAvailable`,
* and the actual asset transfer change by the exact same representable amount.
*
* Rounding strategy:
* - Debits (withdrawals): Rounds down `|delta|` on the new scale to prevent
* paying out more than requested.
* - Credits (deposits): Floors the resulting total asset balance and returns the
* difference from the current total. This prevents crediting the vault with
* more assets than the user deposited.
*
* Key rules:
* - The returned magnitude never exceeds `|delta|`.
* - Returns `tecPRECISION_LOSS` if the change is smaller than 1 ULP of the target scale
* (prevents share operations when totals cannot change).
* - For integer assets (XRP, MPT), rounding is a no-op.
*
* @param vault The vault ledger entry.
* @param delta The requested signed change to sfAssetsTotal.
* @return The rounded, positive magnitude, or `tecPRECISION_LOSS` if the
* change is below representable precision.
*/
[[nodiscard]] std::expected<STAmount, TER>
clampToAssetsTotalScale(SLE::const_ref vault, STAmount const& delta);
/**
* Controls whether to truncate shares instead of rounding.
*/
@@ -55,6 +85,35 @@ enum class TruncateShares : bool { No = false, Yes = true };
*/
enum class WaiveUnrealizedLoss : bool { No = false, Yes = true };
/**
* Returns the assets backing outstanding shares for a withdrawal:
* sfAssetsTotal minus sfLossUnrealized, or sfAssetsTotal alone when the
* unrealized loss is waived. Used by assetsToSharesWithdraw and
* sharesToAssetsWithdraw as the numerator of the share/asset exchange rate.
*
* @param vault The vault SLE.
* @param waive Whether to skip subtracting the unrealized loss.
*/
[[nodiscard]] Number
assetsTotalForWithdrawal(SLE::const_ref vault, WaiveUnrealizedLoss waive);
/**
* Returns true if debiting `amount` from `total` (the current value of a
* vault's sfAssetsTotal or sfAssetsAvailable) would canonicalize to the
* same STAmount value. This happens when `amount` is non-zero but too small
* to change the stored total at STAmount's precision. Shares would still
* move, so the ValidVault invariant would fail after apply; callers use
* this to reject the transaction upfront instead.
*
* @param asset The vault's underlying asset, used to canonicalize both
* sides the same way the ledger will when the field is stored.
* @param total The field's current value.
* @param amount The amount to debit. Zero always returns false; that case
* is rejected separately.
*/
[[nodiscard]] bool
debitIsNonZeroDust(Asset const& asset, Number const& total, Number const& amount);
/**
* From the perspective of a vault, return the number of shares to demand from
* the depositor when they ask to withdraw a fixed amount of assets. Since
@@ -204,4 +263,40 @@ getVaultPhase(
std::optional<std::uint32_t> subscriptionDate,
std::optional<std::uint32_t> redemptionDate);
/**
* Controls whether checkVaultDomain reports an expired credential as an
* error. A caller that deletes expired credentials later, in doApply, passes
* Yes and treats the subject as authorized; a caller with no such cleanup
* step must keep the error.
*/
enum class SuppressExpired : bool { No = false, Yes = true };
/**
* Checks that subject belongs to the permissioned domain governing a vault's
* shares.
*
* The domain is read from the share issuance rather than from the vault. Vault
* shares are issued by the vault's pseudo-account, which cannot grant an
* authorization explicitly, so domain membership is the only route to being
* authorized: a vault with no domain set has no authorized participants at
* all, and every subject fails with tecNO_AUTH.
*
* Which accounts to check, and whether to check at all, is left to the caller.
* This says nothing about vault privacy or about the roles of the accounts.
*
* @param view The ledger view.
* @param issuance The MPTokenIssuance SLE for the vault's shares.
* @param subject The account whose domain membership is checked.
* @param suppressExpired Whether an expired credential counts as authorized.
*
* @return tesSUCCESS if the subject is a domain member, otherwise the reason
* it is not.
*/
[[nodiscard]] TER
checkVaultDomain(
ReadView const& view,
SLE::const_ref issuance,
AccountID const& subject,
SuppressExpired suppressExpired);
} // namespace xrpl

View File

@@ -7,16 +7,11 @@
## Introduction
A `NodeObject` is a simple object that the Ledger uses to store entries. It is
comprised of a type, a hash and a blob. It can be uniquely
identified by the hash, which is a 256 bit hash of the blob. The blob is a
variable length block of serialized data. The type identifies what the blob
contains. The fields are as follows:
A `NodeObject` is a simple object that the Ledger uses to store entries. It is comprised of a type, a hash and a blob. It can be uniquely identified by the hash, which is a 256 bit hash of the blob. The blob is a variable length block of serialized data. The type identifies what the blob contains. The fields are as follows:
- `mType`
An enumeration that determines what the blob holds. There are four
different types of objects stored.
An enumeration that determines what the blob holds. There are four different types of objects stored.
- **ledger**
@@ -50,19 +45,11 @@ A blob containing the payload. Stored in the following format.
---
The `NodeStore` provides an interface that stores, in a persistent database, a
collection of NodeObjects that xrpld uses as its primary representation of
ledger entries. All ledger entries are stored as NodeObjects and as such, need
to be persisted between launches. If a NodeObject is accessed and is not in
memory, it will be retrieved from the database.
The `NodeStore` provides an interface that stores, in a persistent database, a collection of NodeObjects that xrpld uses as its primary representation of ledger entries. All ledger entries are stored as NodeObjects and as such, need to be persisted between launches. If a NodeObject is accessed and is not in memory, it will be retrieved from the database.
## Backend
The `NodeStore` implementation provides the `Backend` abstract interface,
which lets different key/value databases to be chosen at run-time. This allows
experimentation with different engines. Improvements in the performance of the
NodeStore are a constant area of research. The database can be specified in
the configuration file [node_db] section as follows.
The `NodeStore` implementation provides the `Backend` abstract interface, which lets different key/value databases to be chosen at run-time. This allows experimentation with different engines. Improvements in the performance of the NodeStore are a constant area of research. The database can be specified in the configuration file [node_db] section as follows.
One or more lines of key / value pairs
@@ -106,75 +93,30 @@ Choices for 'compression'
# Benchmarks
The `NodeStore.Timing` test is used to execute a set of read/write workloads to
compare current available nodestore backends. It can be executed with:
The `NodeStore.Timing` test is used to execute a set of read/write workloads to compare current available nodestore backends. It can be executed with:
```
$xrpld --unittest=NodeStoreTiming
```
It is also possible to use alternate DB config params by passing config strings
as `--unittest-arg`.
It is also possible to use alternate DB config params by passing config strings as `--unittest-arg`.
## Addendum
The discussion below refers to a `RocksDBQuick` backend that has since been
removed from the code as it was not working and not maintained. That backend
primarily used one of the several rocks `Optimize*` methods to setup the
majority of the DB options/params, whereas the primary RocksDB backend exposes
many of the available config options directly. The code for RocksDBQuick can be
found in versions of this repo 1.2 and earlier if you need to refer back to it.
The conclusions below date from about 2014 and may need revisiting based on
newer versions of RocksDB (TBD).
The discussion below refers to a `RocksDBQuick` backend that has since been removed from the code as it was not working and not maintained. That backend primarily used one of the several rocks `Optimize*` methods to setup the majority of the DB options/params, whereas the primary RocksDB backend exposes many of the available config options directly. The code for RocksDBQuick can be found in versions of this repo 1.2 and earlier if you need to refer back to it. The conclusions below date from about 2014 and may need revisiting based on newer versions of RocksDB (TBD).
## Discussion
RocksDBQuickFactory is intended to provide a testbed for comparing potential
rocksdb performance with the existing recommended configuration in xrpld.cfg.
Through various executions and profiling some conclusions are presented below.
RocksDBQuickFactory is intended to provide a testbed for comparing potential rocksdb performance with the existing recommended configuration in xrpld.cfg. Through various executions and profiling some conclusions are presented below.
- If the write ahead log is enabled, insert speed soon clogs up under load. The
BatchWriter class intends to stop this from blocking the main threads by queuing
up writes and running them in a separate thread. However, rocksdb already has
separate threads dedicated to flushing the memtable to disk and the memtable is
itself an in-memory queue. The result is two queues with a guarantee of
durability in between. However if the memtable was used as the sole queue and
the rocksdb::Flush() call was manually triggered at opportune moments, possibly
just after ledger close, then that would provide similar, but more predictable
guarantees. It would also remove an unneeded thread and unnecessary memory
usage. An alternative point of view is that because there will always be many
other xrpld instances running there is no need for such guarantees. The nodes
will always be available from another peer.
- If the write ahead log is enabled, insert speed soon clogs up under load. The BatchWriter class intends to stop this from blocking the main threads by queuing up writes and running them in a separate thread. However, rocksdb already has separate threads dedicated to flushing the memtable to disk and the memtable is itself an in-memory queue. The result is two queues with a guarantee of durability in between. However if the memtable was used as the sole queue and the rocksdb::Flush() call was manually triggered at opportune moments, possibly just after ledger close, then that would provide similar, but more predictable guarantees. It would also remove an unneeded thread and unnecessary memory usage. An alternative point of view is that because there will always be many other xrpld instances running there is no need for such guarantees. The nodes will always be available from another peer.
- Lookup in a block was previously using binary search. With xrpld's use case
it is highly unlikely that two adjacent key/values will ever be requested one
after the other. Therefore hash indexing of blocks makes much more sense.
Rocksdb has a number of options for hash indexing both memtables and blocks and
these need more testing to find the best choice.
- Lookup in a block was previously using binary search. With xrpld's use case it is highly unlikely that two adjacent key/values will ever be requested one after the other. Therefore hash indexing of blocks makes much more sense. Rocksdb has a number of options for hash indexing both memtables and blocks and these need more testing to find the best choice.
- The current Database implementation has two forms of caching, so the LRU cache
of blocks at Factory level does not make any sense. However, if the hash
indexing and potentially the new [bloom
filter](http://rocksdb.org/blog/1427/new-bloom-filter-format/) can provide
faster lookup for non-existent keys, then potentially the caching could exist at
Factory level.
- The current Database implementation has two forms of caching, so the LRU cache of blocks at Factory level does not make any sense. However, if the hash indexing and potentially the new [bloom filter](http://rocksdb.org/blog/1427/new-bloom-filter-format/) can provide faster lookup for non-existent keys, then potentially the caching could exist at Factory level.
- Multiple runs of the benchmarks can yield surprisingly different results. This
can perhaps be attributed to the asynchronous nature of rocksdb's compaction
process. The benchmarks are artificial and create highly unlikely write load to
create the dataset to measure different read access patterns. Therefore multiple
runs of the benchmarks are required to get a feel for the effectiveness of the
changes. This contrasts sharply with the keyvadb benchmarking were highly
repeatable timings were discovered. Also realistically sized datasets are
required to get a correct insight. The number of 2,000,000 key/values (actually
4,000,000 after the two insert benchmarks complete) is too low to get a full
picture.
- Multiple runs of the benchmarks can yield surprisingly different results. This can perhaps be attributed to the asynchronous nature of rocksdb's compaction process. The benchmarks are artificial and create highly unlikely write load to create the dataset to measure different read access patterns. Therefore multiple runs of the benchmarks are required to get a feel for the effectiveness of the changes. This contrasts sharply with the keyvadb benchmarking were highly repeatable timings were discovered. Also realistically sized datasets are required to get a correct insight. The number of 2,000,000 key/values (actually 4,000,000 after the two insert benchmarks complete) is too low to get a full picture.
- An interesting side effect of running the benchmarks in a profiler was that a
clear pattern of what RocksDB does under the hood was observable. This led to
the decision to trial hash indexing and also the discovery of the native CRC32
instruction not being used.
- An interesting side effect of running the benchmarks in a profiler was that a clear pattern of what RocksDB does under the hood was observable. This led to the decision to trial hash indexing and also the discovery of the native CRC32 instruction not being used.
- Important point to note that is if this factory is tested with an existing set
of sst files none of the old sst files will benefit from indexing changes until
they are compacted at a future point in time.
- Important point to note that is if this factory is tested with an existing set of sst files none of the old sst files will benefit from indexing changes until they are compacted at a future point in time.

View File

@@ -1,4 +1,3 @@
# Proto
This holds protocol buffers source code. The protoc tool stores the output
of the .proto files in the build directory.
This holds protocol buffers source code. The protoc tool stores the output of the .proto files in the build directory.

View File

@@ -1,81 +1,35 @@
# Protocol buffer definitions for gRPC
This folder contains the protocol buffer definitions used by the xrpld gRPC API.
The gRPC API attempts to mimic the JSON/Websocket API as much as possible.
As of April 2020, the gRPC API supports a subset of the full xrpld API:
tx, account_tx, account_info, fee and submit.
This folder contains the protocol buffer definitions used by the xrpld gRPC API. The gRPC API attempts to mimic the JSON/Websocket API as much as possible. As of April 2020, the gRPC API supports a subset of the full xrpld API: tx, account_tx, account_info, fee and submit.
### Making Changes
#### Wire Format and Backwards Compatibility
When making changes to the protocol buffer definitions in this folder, care must
be taken to ensure the changes do not break the wire format, which would break
backwards compatibility. At a high level, do not change any existing fields.
This includes the field's name, type and field number. Do not remove any
existing fields. It is always safe to add fields; just remember to give each of
the new fields a unique field number. The field numbers don't have to be in any
particular order and there can be gaps. More info about what changes break the
wire format can be found
[here](https://developers.google.com/protocol-buffers/docs/proto3#updating).
When making changes to the protocol buffer definitions in this folder, care must be taken to ensure the changes do not break the wire format, which would break backwards compatibility. At a high level, do not change any existing fields. This includes the field's name, type and field number. Do not remove any existing fields. It is always safe to add fields; just remember to give each of the new fields a unique field number. The field numbers don't have to be in any particular order and there can be gaps. More info about what changes break the wire format can be found [here](https://developers.google.com/protocol-buffers/docs/proto3#updating).
#### Conventions
For fields that are reused across different message types, we define the field as a unique
message type in common.proto. The name of the message type is the same as the
field name, with the exception that the field name itself is snake case, whereas
the message type is in Pascal case. The message type has one field, called
`value`. This pattern does not need to be strictly followed across the entire API,
but should be followed for transactions and ledger objects, since there is a high rate
of field reuse across different transactions and ledger objects.
The motivation for this pattern is two-fold. First, we ensure the field has the
same type everywhere that the field is used. Second, wrapping primitive types in
their own message type prevents default initialization of those primitive types.
For example, `uint32` is initialized to `0` if not explicitly set;
there is no way to tell if the client or server set the field to `0` (which may be
a valid value for the field) or the field was default initialized.
For fields that are reused across different message types, we define the field as a unique message type in common.proto. The name of the message type is the same as the field name, with the exception that the field name itself is snake case, whereas the message type is in Pascal case. The message type has one field, called `value`. This pattern does not need to be strictly followed across the entire API, but should be followed for transactions and ledger objects, since there is a high rate of field reuse across different transactions and ledger objects. The motivation for this pattern is two-fold. First, we ensure the field has the same type everywhere that the field is used. Second, wrapping primitive types in their own message type prevents default initialization of those primitive types. For example, `uint32` is initialized to `0` if not explicitly set; there is no way to tell if the client or server set the field to `0` (which may be a valid value for the field) or the field was default initialized.
#### Name Collisions
Each message type must have a unique name. To resolve collisions, add a suffix
to one or more message types. For instance, ledger objects and transaction types
often have the same name (`DepositPreauth` for example). To resolve this, the
`DepositPreauth` ledger object is named `DepositPreauthObject`.
Each message type must have a unique name. To resolve collisions, add a suffix to one or more message types. For instance, ledger objects and transaction types often have the same name (`DepositPreauth` for example). To resolve this, the `DepositPreauth` ledger object is named `DepositPreauthObject`.
#### To add a field or message type
To add a field to a message, define the fields type, name and unique index.
To add a new message type, give the message type a unique name.
Then, add the appropriate C++ code in GRPCHelpers.cpp, or in the handler itself,
to serialize/deserialize the new field or message type.
To add a field to a message, define the fields type, name and unique index. To add a new message type, give the message type a unique name. Then, add the appropriate C++ code in GRPCHelpers.cpp, or in the handler itself, to serialize/deserialize the new field or message type.
#### To add a new gRPC method
To add a new gRPC method, add the gRPC method in xrp_ledger.proto. The method name
should begin with a verb. Define the request and response types in their own
file. The name of the request type should be the method name suffixed with `Request`, and
the response type name should be the method name suffixed with `Response`. For
example, the `GetAccountInfo` method has request type `GetAccountInfoRequest` and
response type `GetAccountInfoResponse`.
To add a new gRPC method, add the gRPC method in xrp_ledger.proto. The method name should begin with a verb. Define the request and response types in their own file. The name of the request type should be the method name suffixed with `Request`, and the response type name should be the method name suffixed with `Response`. For example, the `GetAccountInfo` method has request type `GetAccountInfoRequest` and response type `GetAccountInfoResponse`.
After defining the protobuf messages for the new method, add an instantiation of the
templated `CallData` class in GRPCServerImpl::setupListeners(). The template
parameters should be the request type and the response type.
After defining the protobuf messages for the new method, add an instantiation of the templated `CallData` class in GRPCServerImpl::setupListeners(). The template parameters should be the request type and the response type.
Finally, define the handler itself in the appropriate file under the
src/xrpld/rpc/handlers folder. If the method already has a JSON/Websocket
equivalent, write the gRPC handler in the same file, and abstract common logic
into helper functions (see Tx.cpp or AccountTx.cpp for an example).
Finally, define the handler itself in the appropriate file under the src/xrpld/rpc/handlers folder. If the method already has a JSON/Websocket equivalent, write the gRPC handler in the same file, and abstract common logic into helper functions (see Tx.cpp or AccountTx.cpp for an example).
#### Testing
When modifying an existing gRPC method, be sure to test that modification in the
corresponding, existing unit test. When creating a new gRPC method, create a
client stub with `XRPLedgerAPIService::NewStub` and `grpc::CreateChannel`, and
use it to call the new method. See `GRPCServerTLS_test.cpp` for an example.
The gRPC tests are paired with their JSON counterpart, and the tests should
mirror the JSON test as much as possible.
When modifying an existing gRPC method, be sure to test that modification in the corresponding, existing unit test. When creating a new gRPC method, create a client stub with `XRPLedgerAPIService::NewStub` and `grpc::CreateChannel`, and use it to call the new method. See `GRPCServerTLS_test.cpp` for an example. The gRPC tests are paired with their JSON counterpart, and the tests should mirror the JSON test as much as possible.
Refer to the Protocol Buffers [language
guide](https://developers.google.com/protocol-buffers/docs/proto3)
for more detailed information about Protocol Buffers.
Refer to the Protocol Buffers [language guide](https://developers.google.com/protocol-buffers/docs/proto3) for more detailed information about Protocol Buffers.

View File

@@ -91,6 +91,17 @@ getFee(std::uint16_t tfee)
return Number{tfee} / kAuctionSlotFeeScaleFactor;
}
/**
* Minimum auction slot price: LPTokens * TradingFee / kAuctionSlotMinFeeFraction
* @param lptAMMBalance AMM LP token balance
* @param tradingFee trading fee in {0, 1000}
*/
inline Number
ammAuctionMinSlotPrice(Number const& lptAMMBalance, std::uint16_t tradingFee)
{
return lptAMMBalance * getFee(tradingFee) / kAuctionSlotMinFeeFraction;
}
/**
* Get fee multiplier (1 - tfee)
* @tfee trading fee in basis points

View File

@@ -133,7 +133,7 @@ private:
using id_hash_type = boost::base_from_member<std::hash<xrpl::MPTID>, 0>;
public:
explicit hash() = default;
hash() = default;
using value_type = std::size_t;
using argument_type = xrpl::MPTIssue;
@@ -160,7 +160,7 @@ private:
mptissue_hasher mMptissueHasher_;
public:
explicit hash() = default;
hash() = default;
value_type
operator()(argument_type const& asset) const
@@ -227,7 +227,7 @@ struct hash<xrpl::Issue> : std::hash<xrpl::Issue>
template <>
struct hash<xrpl::MPTIssue> : std::hash<xrpl::MPTIssue>
{
explicit hash() = default;
hash() = default;
using Base = std::hash<xrpl::MPTIssue>;
};
@@ -235,7 +235,7 @@ struct hash<xrpl::MPTIssue> : std::hash<xrpl::MPTIssue>
template <>
struct hash<xrpl::Asset> : std::hash<xrpl::Asset>
{
explicit hash() = default;
hash() = default;
using Base = std::hash<xrpl::Asset>;
};

View File

@@ -9,6 +9,7 @@
#include <cstdint>
#include <limits>
#include <optional>
#include <ostream>
#include <stdexcept>
#include <string>
@@ -174,4 +175,17 @@ mulRatio(MPTAmount const& amt, std::uint32_t num, std::uint32_t den, bool roundU
return MPTAmount(r.convert_to<MPTAmount::value_type>());
}
inline std::optional<MPTAmount>
tryMulRatio(MPTAmount const& amt, std::uint32_t num, std::uint32_t den, bool roundUp)
{
try
{
return mulRatio(amt, num, den, roundUp);
}
catch (std::overflow_error const&)
{
return std::nullopt;
}
}
} // namespace xrpl

View File

@@ -151,7 +151,7 @@ namespace std {
template <>
struct hash<xrpl::MPTID> : xrpl::MPTID::hasher
{
explicit hash() = default;
hash() = default;
};
} // namespace std

View File

@@ -1,19 +0,0 @@
#pragma once
#include <xrpl/json/json_forwards.h>
#include <xrpl/protocol/STTx.h>
#include <xrpl/protocol/TxMeta.h>
#include <memory>
namespace xrpl::rpc {
/**
* Adds common synthetic fields to transaction-related JSON responses
*/
/** @{ */
void
insertNFTSyntheticInJson(json::Value&, std::shared_ptr<STTx const> const&, TxMeta const&);
/** @} */
} // namespace xrpl::rpc

View File

@@ -4,6 +4,7 @@
#include <xrpl/protocol/Rules.h>
#include <xrpl/protocol/SOTemplate.h>
#include <xrpl/protocol/TxFormats.h>
#include <xrpl/protocol/TxSettings.h>
#include <cstdint>
#include <functional>
@@ -38,11 +39,6 @@ enum GranularPermissionType : std::uint32_t {
#pragma pop_macro("GRANULAR_PERMISSION")
};
// Injected bare enumerators (xrpl::delegable / xrpl::notDelegable) are required by preprocessor
// tricks in tests and macro-generated code; enum class would break that.
// NOLINTNEXTLINE(cppcoreguidelines-use-enum-class)
enum Delegation { Delegable, NotDelegable };
class Permission
{
private:
@@ -65,7 +61,7 @@ private:
struct TxDelegationEntry
{
uint256 amendment;
Delegation delegable{NotDelegable};
Delegation delegable{Delegation::NotDelegable};
};
std::unordered_set<TxType> granularTxTypes_;

View File

@@ -348,13 +348,24 @@ enum class VaultPhase : std::uint8_t {
Redemption,
};
/**
* Minimum gap between a closed-ended loan's final scheduled payment and the
* vault's RedemptionDate. LoanSet rejects a schedule whose final payment is
* fewer than this many seconds before RedemptionDate.
*/
constexpr std::uint32_t kLoanRedemptionBuffer = std::chrono::seconds{60}.count();
/**
* Bounds on the length of a closed-ended vault's Investment phase
* (RedemptionDate - SubscriptionDate). At vault creation the gap must satisfy
* kMinInvestmentPeriod <= gap < kMaxInvestmentPeriod.
*
* 180s is enough to originate a loan that uses the minimum payment interval
* and kLoanRedemptionBuffer after StartDate, which is strictly after
* SubscriptionDate. The interval and buffer need not be equal; only their
* sum plus one second must fit in this floor.
*/
constexpr std::uint32_t kMinInvestmentPeriod =
std::chrono::seconds{std::chrono::minutes{1}}.count();
constexpr std::uint32_t kMinInvestmentPeriod = std::chrono::seconds{180}.count();
// This is 946708560 seconds which 30 x 365.2425 days (the average length of a Gregorian year).
constexpr std::uint32_t kMaxInvestmentPeriod = std::chrono::seconds{std::chrono::years{30}}.count();
@@ -396,6 +407,16 @@ using TxID = uint256;
*/
constexpr std::uint16_t kMaxDeletableAmmTrustLines = 512;
/**
* The maximum number of owner-directory entries to walk when clearing
* credentials pinned to a pseudo-account, in a single transaction.
*
* The walk stops after this many entries whether or not each one turns out to
* be a credential, so a directory that also holds other objects yields fewer
* deletions per transaction.
*/
constexpr std::uint16_t kMaxDeletablePseudoAccountCredentials = 512;
/**
* The maximum length of a URI inside an Oracle
*/

View File

@@ -1,42 +1,24 @@
# protocol
Classes and functions for handling data and
values associated with the XRP Ledger protocol.
Classes and functions for handling data and values associated with the XRP Ledger protocol.
## Serialized Objects
Objects transmitted over the network must be
serialized into a canonical format. The prefix "ST" refers
to classes that deal with the serialized format.
Objects transmitted over the network must be serialized into a canonical format. The prefix "ST" refers to classes that deal with the serialized format.
The term "Tx" or "tx" is an abbreviation for "Transaction",
a commonly occurring object type.
The term "Tx" or "tx" is an abbreviation for "Transaction", a commonly occurring object type.
### Optional Fields
Our serialized fields have some "type magic" to make
optional fields easier to read:
Our serialized fields have some "type magic" to make optional fields easier to read:
- The operation `x[sfFoo]` means "return the value of 'Foo'
if it exists, or the default value if it doesn't."
- The operation `x[~sfFoo]` means "return the value of 'Foo'
if it exists, or nothing if it doesn't." This usage of the
tilde/bitwise NOT operator is not standard outside of the
`xrpld` codebase.
- As a consequence of this, `x[~sfFoo] = y[~sfFoo]`
assigns the value of Foo from y to x, including omitting
Foo from x if it doesn't exist in y.
- The operation `x[sfFoo]` means "return the value of 'Foo' if it exists, or the default value if it doesn't."
- The operation `x[~sfFoo]` means "return the value of 'Foo' if it exists, or nothing if it doesn't." This usage of the tilde/bitwise NOT operator is not standard outside of the `xrpld` codebase.
- As a consequence of this, `x[~sfFoo] = y[~sfFoo]` assigns the value of Foo from y to x, including omitting Foo from x if it doesn't exist in y.
Typically, for things that are guaranteed to exist, you use
`x[sfFoo]` and avoid having to deal with a container that may
or may not hold a value. For things not guaranteed to exist,
you use `x[~sfFoo]` because you want such a container. It
avoids having to look something up twice, once just to see if
it exists and a second time to get/set its value.
([Real example](https://github.com/XRPLF/rippled/blob/35f4698aed5dce02f771b34cfbb690495cb5efcc/src/ripple/app/tx/impl/PayChan.cpp#L229-L236))
Typically, for things that are guaranteed to exist, you use `x[sfFoo]` and avoid having to deal with a container that may or may not hold a value. For things not guaranteed to exist, you use `x[~sfFoo]` because you want such a container. It avoids having to look something up twice, once just to see if it exists and a second time to get/set its value. ([Real example](https://github.com/XRPLF/rippled/blob/35f4698aed5dce02f771b34cfbb690495cb5efcc/src/ripple/app/tx/impl/PayChan.cpp#L229-L236))
The source of this "type magic" is in
[SField.h](./SField.h#L296-L302).
The source of this "type magic" is in [SField.h](./SField.h#L296-L302).
### Related Resources

View File

@@ -19,7 +19,7 @@ namespace xrpl {
class Rules;
namespace test {
class Invariants_test;
class InvariantsMisc_test;
} // namespace test
class STLedgerEntry final : public STObject, public CountedObject<STLedgerEntry>
@@ -83,8 +83,8 @@ private:
void
setSLEType();
friend test::Invariants_test; // this test wants access to the private
// type_
friend test::InvariantsMisc_test; // this test wants access to the
// private type_
STBase*
copy(std::size_t n, void* buf) const override;

View File

@@ -0,0 +1,96 @@
#pragma once
#include <xrpl/basics/base_uint.h>
#include <xrpl/basics/safe_cast.h>
#include <cstdint>
#include <type_traits>
namespace xrpl {
enum class Delegation { Delegable, NotDelegable };
/**
* Operations a transaction is permitted to perform, as a bitfield.
*
* These are declared per-transaction in transactions.macro (via
* TxSettings::privileges) and enforced in InvariantCheck.cpp.
*/
enum class Privilege : std::uint16_t {
NoPriv = 0x0000, // The transaction can not do any of the enumerated operations
CreateAcct = 0x0001, // The transaction can create a new ACCOUNT_ROOT object.
CreatePseudoAcct = 0x0002, // The transaction can create a pseudo account,
// which implies createAcct
MustDeleteAcct = 0x0004, // The transaction must delete an ACCOUNT_ROOT object
MayDeleteAcct = 0x0008, // The transaction may delete an ACCOUNT_ROOT
// object, but does not have to
OverrideFreeze = 0x0010, // The transaction can override some freeze rules
ChangeNftCounts = 0x0020, // The transaction can mint or burn an NFT
CreateMptIssuance = 0x0040, // The transaction can create a new MPT issuance
DestroyMptIssuance = 0x0080, // The transaction can destroy an MPT issuance
MustAuthorizeMpt = 0x0100, // The transaction MUST create or delete an MPT
// object (except by issuer)
MayAuthorizeMpt = 0x0200, // The transaction MAY create or delete an MPT
// object (except by issuer)
MayDeleteMpt = 0x0400, // The transaction MAY delete an MPT object. May not create.
MustModifyVault = 0x0800, // The transaction must modify, delete or create, a vault
MayModifyVault = 0x1000, // The transaction MAY modify, delete or create, a vault
MayCreateMpt = 0x2000, // The transaction MAY create an MPT object, except for issuer.
};
// The inner static_cast is not redundant: the underlying type is narrower than
// `int`, so the operands integer-promote and the result has to be narrowed back.
// safeCast rejects that narrowing, but every input bit is a Privilege bit by
// construction, so the result is always representable.
constexpr Privilege
operator|(Privilege lhs, Privilege rhs)
{
using Underlying = std::underlying_type_t<Privilege>;
return static_cast<Privilege>(
static_cast<Underlying>(safeCast<Underlying>(lhs) | safeCast<Underlying>(rhs)));
}
constexpr Privilege
operator&(Privilege lhs, Privilege rhs)
{
using Underlying = std::underlying_type_t<Privilege>;
return static_cast<Privilege>(
static_cast<Underlying>(safeCast<Underlying>(lhs) & safeCast<Underlying>(rhs)));
}
/**
* Per-transaction metadata declared in transactions.macro.
*
* Every member has a default, so a transaction only needs to name the settings
* that differ from the common case. See the documentation at the top of
* transactions.macro for the authoring syntax.
*
* This is deliberately not a constexpr-friendly type: amendment identifiers are
* runtime-initialized `extern uint256 const` globals (see Feature.h), so a
* TxSettings can only be built at runtime.
*/
struct TxSettings
{
/**
* Whether an account may delegate this transaction to another account.
*/
Delegation delegable{Delegation::NotDelegable};
/**
* The amendment gating this transaction, or uint256{} if always available.
*/
// The `{}` looks redundant, because BaseUInt's default constructor already
// zeroes the value. It is not: without a default member initializer here,
// every partial designated initializer in transactions.macro trips the
// missing-designated-field-initializers warning, which the build treats as
// an error.
// NOLINTNEXTLINE(readability-redundant-member-init)
uint256 amendment{};
/**
* Operations this transaction is permitted to perform.
*/
Privilege privileges{Privilege::NoPriv};
};
} // namespace xrpl

View File

@@ -18,7 +18,7 @@
XRPL_FIX (Cleanup3_4_0, Supported::Yes, VoteBehavior::DefaultNo)
XRPL_FEATURE(Sponsor, Supported::Yes, VoteBehavior::DefaultNo)
XRPL_FEATURE(BatchV1_1, Supported::Yes, VoteBehavior::DefaultNo)
XRPL_FEATURE(LendingProtocolV1_1, Supported::No, VoteBehavior::DefaultNo)
XRPL_FEATURE(LendingProtocolV1_1, Supported::Yes, VoteBehavior::DefaultNo)
XRPL_FEATURE(ConfidentialTransfer, Supported::Yes, VoteBehavior::DefaultNo)
XRPL_FIX (Cleanup3_3_0, Supported::Yes, VoteBehavior::DefaultNo)
XRPL_FIX (Cleanup3_2_0, Supported::Yes, VoteBehavior::DefaultNo)

File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More