Compare commits

..

42 Commits

Author SHA1 Message Date
Ayaz Salikhov
00a178fb92 chore: Bump version to 3.3.0 2026-08-06 17:34:39 +01:00
Bart
39c8c293b3 chore: Bump version to 3.3.0-rc7 2026-08-04 17:11:00 -04:00
Pratik Mankawde
54cfdda00b fix: Increase manifest protocol message size cap and fix manifests relay
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-08-04 17:08:43 -04:00
Ed Hennis
15d08770bf Merge branch 'release/3.2.x' into ximinez/merge-321-to-330
* release/3.2.x:
  chore: Bump version to 3.2.1
  chore: Bump version to 3.2.1-rc1
  fix: Cap untrusted manifests per message and drop oversized ones
  fix: Reject oversized validator manifest before decoding
  fix: Reduce untrusted manifest cache cap to 100
  fix: Bound untrusted manifest cache
2026-08-04 13:28:52 -04:00
Pratik Mankawde
8461ded0d8 fix: Cap untrusted manifests per message and drop oversized ones 2026-08-03 12:00:11 -04:00
Ed Hennis
d4c1359921 chore: Bump version to 3.2.1 2026-07-31 19:46:17 -04:00
Bart
a88ba66fce chore: Bump version to 3.2.1-rc1 2026-07-31 19:43:19 -04:00
Valentin Balaschenko
4bd1d1ca2f fix: Cap untrusted manifests per message and drop oversized ones
Bound the number of manifests carried in a single TMManifests message
(kMaxManifestsPerMessage). Trusted manifests are always included and
processed; untrusted gossip is capped per message on both send and
receive, and the sender is charged only when untrusted entries are
actually skipped. Oversized TMManifests messages are dropped without
penalty at the protocol layer so an unpatched peer is not disconnected.

Complements the cache bound from #276/#323.
2026-07-31 19:43:18 -04:00
Bart
0cce5a06d9 fix: Reject oversized validator manifest before decoding 2026-07-31 19:43:18 -04:00
Pratik Mankawde
32a9cc4038 fix: Reduce untrusted manifest cache cap to 100 2026-07-31 19:43:15 -04:00
Pratik Mankawde
587505ef18 fix: Bound untrusted manifest cache 2026-07-31 19:43:04 -04:00
Bart
4eece4003d chore: Bump version to 3.3.0-rc6 2026-07-30 16:34:46 +01:00
Ayaz Salikhov
3ad6ce236e feat: Package validator-keys inside rippled 2026-07-30 11:29:37 -04:00
Bart
ccb9db0bc7 chore: Bump version to 3.3.0-rc5 2026-07-29 14:25:13 -04:00
Mayukha Vadari
24b6dad287 fix: Switch SponsorshipSet to use a delta for sfFeeAmount 2026-07-29 14:24:55 -04:00
Bart
e290005db5 fix: Re-revert "fix: Set request size limits and differential pricing for get-object-by-hash calls" 2026-07-28 14:02:49 -04:00
Bart
6668b7e8d0 chore: Bump version to 3.3.0-rc4 2026-07-24 18:40:02 -04:00
Bart
a5cc339d7b fix: Revert "fix: Set request size limits and differential pricing for get-object-by-hash calls" 2026-07-24 18:39:35 -04:00
Bart
1653f0c80f chore: Bump version to 3.3.0-rc3 2026-07-23 16:39:50 -04:00
Pratik Mankawde
c50edf507c fix: Reduce untrusted manifest cache cap to 100 2026-07-23 16:27:45 -04:00
Valentin Balaschenko
9cd531659a fix: Revert "fix: Reject oversized SHAMap nodes in gotStaleData and fetch-pack path" 2026-07-23 14:53:15 -04:00
Bart
982bf36dd8 chore: Bump version to 3.3.0-rc2 2026-07-17 17:09:13 -04:00
Denis Angell
faca302adf fix: Check transaction type before RawTransactions 2026-07-17 17:08:57 -04:00
Shawn Xie
6b3eaf091b fix: Change ConfidentialMPTConvert to no delegate 2026-07-17 16:06:17 -04:00
Pratik Mankawde
68a765d929 fix: Bound untrusted manifest cache 2026-07-17 14:07:57 -04:00
yinyiqian1
033dca2f0e feat: Make DynamicMPT opt-in-immutable 2026-07-17 14:02:35 -04:00
Ayaz Salikhov
bf65e5fa7b chore: Upload codecov for whole XRPLF org 2026-07-17 11:20:28 -04:00
Valentin Balaschenko
90b2a68da8 fix: Reject oversized SHAMap nodes in gotStaleData and fetch-pack path 2026-07-17 08:07:42 -04:00
Ayaz Salikhov
06a9b1b617 chore: Update mpt-crypto to 1.0.2 2026-07-17 08:07:01 -04:00
Jingchen
7d3611df2a fix: Compute validation suppression key over canonical serialisation 2026-07-17 07:33:44 -04:00
Bart
7877ee42a0 fix: Reject oversized validator manifest before decoding 2026-07-16 16:33:03 -04:00
Bart
1dcaf4b54e fix: Bound and offload per-connection subscription cleanup 2026-07-16 16:33:03 -04:00
Bart
5ab95748d4 refactor: Clean up pong replies 2026-07-16 16:33:03 -04:00
Bart
4a9ee54c88 fix: Handle malformed ledger replay responses 2026-07-16 16:33:03 -04:00
Bart
981c256933 fix: Use weighted median for close-time offset aggregation 2026-07-16 16:33:03 -04:00
Bart
a5af6b4e4a fix: Charge kFeeHeavyBurdenRpc in doChannelVerify 2026-07-16 16:33:03 -04:00
Bart
e1dae5f775 refactor: Use tecPSEUDO_ACCOUNT instead of tecNO_PERMISSION where relevant 2026-07-16 16:32:54 -04:00
Timothy Banks
6c793edf72 fix: Reject oversized TMPing messages 2026-07-16 15:07:36 -04:00
Timothy Banks
d60955e2fc fix: Acquire lock on getClosedLedgerHash 2026-07-16 12:41:34 -04:00
Timothy Banks
846369cbe7 fix: Use hashmap for quicker lookup in assembleAdd 2026-07-16 12:16:09 -04:00
Mayukha Vadari
d569f7db9e fix: Prevent credentials from being created for pseudo-accounts 2026-07-16 11:42:27 -04:00
Mayukha Vadari
a12ab0496c fix: Add missing pseudo-account checks 2026-07-16 11:18:26 -04:00
270 changed files with 11943 additions and 10317 deletions

View File

@@ -30,9 +30,7 @@ ignoreRegExpList:
- ABCDEFGHIJKLMNOPQRSTUVWXYZ
- ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz
overrides:
- filename:
- "**/*_test.cpp" # legacy boost.test files
- "src/tests/**/*.cpp" # gtest test files
- filename: "**/*_test.cpp" # all test files
ignoreRegExpList:
- /"[^"]*"/g # double-quoted strings
- /'[^']*'/g # single-quoted strings
@@ -132,6 +130,7 @@ words:
- godexsoft
- gpgcheck
- gpgkey
- Hinnant
- hotwallet
- hwaddress
- hwrap
@@ -165,6 +164,7 @@ words:
- llection
- LOCALGOOD
- logwstream
- Lombrozo
- lseq
- lsmf
- ltype
@@ -202,6 +202,7 @@ words:
- nftokens
- nftpage
- nikb
- Nikolaos
- nixfmt
- nixos
- nixpkgs
@@ -218,7 +219,6 @@ words:
- Nyffenegger
- onlatest
- ostr
- oxalica
- pargs
- partitioner
- paychan
@@ -265,8 +265,6 @@ words:
- Rohrs
- roundings
- rustc
- rustfmt
- rustup
- sahyadri
- Satoshi
- scons
@@ -310,7 +308,6 @@ words:
- takerpays
- ters
- TMEndpointv2
- toolchain
- tparam
- trixie
- tx
@@ -343,7 +340,6 @@ words:
- unsquelch
- unsquelched
- unsquelching
- unsuffixed
- unvalidated
- unveto
- unvetoed

View File

@@ -11,9 +11,6 @@ endfunction()
function(create_symbolic_link target link)
endfunction()
function(xrpl_add_benchmark name)
endfunction()
macro(exclude_from_default target_)
endmacro()

View File

@@ -1,10 +1,10 @@
<!--
This PR template helps you write a good pull request description.
This PR template helps you to write a good pull request description.
Please feel free to include additional useful information even beyond what is requested below.
If your branch is on a personal fork and has a name that allows it to
run CI build/test jobs (e.g. "ci/foo"), remember to rename it BEFORE
opening the PR. This avoids redundant test runs. Renaming
opening the PR. This avoids unnecessary redundant test runs. Renaming
the branch after opening the PR will close the PR.
https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-branches-in-your-repository/renaming-a-branch
-->
@@ -15,7 +15,7 @@ https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-
Please include a summary of the changes.
This may be a direct input to the release notes.
If too broad, please consider splitting into multiple PRs.
If there is a relevant task or issue, please link it here.
If a relevant task or issue, please link it here.
-->
### Context of Change
@@ -65,5 +65,5 @@ This section may not be needed if your change includes thoroughly commented unit
<!--
## Future Tasks
For future tasks related to this PR.
For future tasks related to PR.
-->

View File

@@ -40,18 +40,18 @@ listed later.
| 04 | xrpl/protocol |
| 05 | xrpl/core xrpl/resource xrpl/server |
| 06 | xrpl/ledger xrpl/nodestore xrpl/net |
| 07 | xrpl/shamap xrpl/consensus |
| 07 | xrpl/shamap |
## xrpld Modules (Application Implementation)
| Level / Tier | Module(s) |
| ------------ | --------------------------- |
| 05 | xrpld/conditions |
| 06 | xrpld/core xrpld/peerfinder |
| 07 | xrpld/shamap xrpld/overlay |
| 08 | xrpld/app |
| 09 | xrpld/rpc |
| 10 | xrpld/perflog |
| Level / Tier | Module(s) |
| ------------ | -------------------------------- |
| 05 | xrpld/conditions xrpld/consensus |
| 06 | xrpld/core xrpld/peerfinder |
| 07 | xrpld/shamap xrpld/overlay |
| 08 | xrpld/app |
| 09 | xrpld/rpc |
| 10 | xrpld/perflog |
## Test Modules

View File

@@ -1,6 +1,9 @@
Loop: xrpld.app xrpld.overlay
xrpld.app > xrpld.overlay
Loop: xrpld.app xrpld.peerfinder
xrpld.peerfinder ~= xrpld.app
Loop: xrpld.app xrpld.rpc
xrpld.rpc > xrpld.app

View File

@@ -1,13 +1,8 @@
benchmarks.libxrpl > xrpl.basics
benchmarks.libxrpl > xrpl.config
benchmarks.libxrpl > xrpl.nodestore
libxrpl.basics > xrpl.basics
libxrpl.conditions > xrpl.basics
libxrpl.conditions > xrpl.conditions
libxrpl.config > xrpl.basics
libxrpl.config > xrpl.config
libxrpl.consensus > xrpl.basics
libxrpl.consensus > xrpl.consensus
libxrpl.core > xrpl.basics
libxrpl.core > xrpl.core
libxrpl.core > xrpl.json
@@ -27,9 +22,6 @@ libxrpl.nodestore > xrpl.config
libxrpl.nodestore > xrpl.json
libxrpl.nodestore > xrpl.nodestore
libxrpl.nodestore > xrpl.protocol
libxrpl.peerfinder > xrpl.basics
libxrpl.peerfinder > xrpl.peerfinder
libxrpl.peerfinder > xrpl.protocol
libxrpl.protocol > xrpl.basics
libxrpl.protocol > xrpl.json
libxrpl.protocol > xrpl.protocol
@@ -65,9 +57,9 @@ test.app > test.jtx
test.app > test.unit_test
test.app > xrpl.basics
test.app > xrpl.config
test.app > xrpl.consensus
test.app > xrpl.core
test.app > xrpld.app
test.app > xrpld.consensus
test.app > xrpld.core
test.app > xrpld.overlay
test.app > xrpld.rpc
@@ -88,9 +80,12 @@ test.basics > xrpl.protocol
test.beast > xrpl.basics
test.conditions > xrpl.basics
test.conditions > xrpl.conditions
test.consensus > test.csf
test.consensus > test.jtx
test.consensus > test.unit_test
test.consensus > xrpl.basics
test.consensus > xrpld.app
test.consensus > xrpld.consensus
test.consensus > xrpl.ledger
test.consensus > xrpl.protocol
test.consensus > xrpl.shamap
@@ -105,6 +100,10 @@ test.core > xrpl.json
test.core > xrpl.protocol
test.core > xrpl.rdb
test.core > xrpl.server
test.csf > xrpl.basics
test.csf > xrpld.consensus
test.csf > xrpl.json
test.csf > xrpl.ledger
test.json > test.jtx
test.json > xrpl.json
test.jtx > test.unit_test
@@ -144,13 +143,19 @@ test.overlay > xrpl.config
test.overlay > xrpld.app
test.overlay > xrpld.core
test.overlay > xrpld.overlay
test.overlay > xrpld.peerfinder
test.overlay > xrpl.json
test.overlay > xrpl.nodestore
test.overlay > xrpl.peerfinder
test.overlay > xrpl.protocol
test.overlay > xrpl.resource
test.overlay > xrpl.server
test.overlay > xrpl.shamap
test.peerfinder > test.beast
test.peerfinder > test.unit_test
test.peerfinder > xrpl.basics
test.peerfinder > xrpld.core
test.peerfinder > xrpld.peerfinder
test.peerfinder > xrpl.protocol
test.protocol > test.jtx
test.protocol > test.unit_test
test.protocol > xrpl.basics
@@ -184,13 +189,11 @@ test.unit_test > xrpl.basics
test.unit_test > xrpl.protocol
tests.libxrpl > xrpl.basics
tests.libxrpl > xrpl.config
tests.libxrpl > xrpl.consensus
tests.libxrpl > xrpl.core
tests.libxrpl > xrpl.json
tests.libxrpl > xrpl.ledger
tests.libxrpl > xrpl.net
tests.libxrpl > xrpl.nodestore
tests.libxrpl > xrpl.peerfinder
tests.libxrpl > xrpl.protocol
tests.libxrpl > xrpl.protocol_autogen
tests.libxrpl > xrpl.resource
@@ -200,10 +203,6 @@ tests.libxrpl > xrpl.tx
xrpl.conditions > xrpl.basics
xrpl.conditions > xrpl.protocol
xrpl.config > xrpl.basics
xrpl.consensus > xrpl.basics
xrpl.consensus > xrpl.json
xrpl.consensus > xrpl.ledger
xrpl.consensus > xrpl.protocol
xrpl.core > xrpl.basics
xrpl.core > xrpl.json
xrpl.core > xrpl.protocol
@@ -218,8 +217,6 @@ xrpl.nodestore > xrpl.basics
xrpl.nodestore > xrpl.config
xrpl.nodestore > xrpl.json
xrpl.nodestore > xrpl.protocol
xrpl.peerfinder > xrpl.basics
xrpl.peerfinder > xrpl.protocol
xrpl.protocol > xrpl.basics
xrpl.protocol > xrpl.json
xrpl.protocol_autogen > xrpl.json
@@ -246,20 +243,23 @@ xrpl.tx > xrpl.protocol
xrpld.app > test.unit_test
xrpld.app > xrpl.basics
xrpld.app > xrpl.config
xrpld.app > xrpl.consensus
xrpld.app > xrpl.core
xrpld.app > xrpld.consensus
xrpld.app > xrpld.core
xrpld.app > xrpl.json
xrpld.app > xrpl.ledger
xrpld.app > xrpl.net
xrpld.app > xrpl.nodestore
xrpld.app > xrpl.peerfinder
xrpld.app > xrpl.protocol
xrpld.app > xrpl.rdb
xrpld.app > xrpl.resource
xrpld.app > xrpl.server
xrpld.app > xrpl.shamap
xrpld.app > xrpl.tx
xrpld.consensus > xrpl.basics
xrpld.consensus > xrpl.json
xrpld.consensus > xrpl.ledger
xrpld.consensus > xrpl.protocol
xrpld.core > xrpl.basics
xrpld.core > xrpl.config
xrpld.core > xrpl.core
@@ -268,22 +268,21 @@ xrpld.core > xrpl.protocol
xrpld.core > xrpl.rdb
xrpld.overlay > xrpl.basics
xrpld.overlay > xrpl.config
xrpld.overlay > xrpl.consensus
xrpld.overlay > xrpl.core
xrpld.overlay > xrpld.consensus
xrpld.overlay > xrpld.core
xrpld.overlay > xrpld.peerfinder
xrpld.overlay > xrpl.json
xrpld.overlay > xrpl.ledger
xrpld.overlay > xrpl.peerfinder
xrpld.overlay > xrpl.protocol
xrpld.overlay > xrpl.resource
xrpld.overlay > xrpl.server
xrpld.overlay > xrpl.shamap
xrpld.overlay > xrpl.tx
xrpld.peerfinder > xrpl.basics
xrpld.peerfinder > xrpld.app
xrpld.peerfinder > xrpl.config
xrpld.peerfinder > xrpld.core
xrpld.peerfinder > xrpl.peerfinder
xrpld.peerfinder > xrpl.protocol
xrpld.peerfinder > xrpl.rdb
xrpld.perflog > xrpl.basics
xrpld.perflog > xrpl.config

View File

@@ -136,7 +136,8 @@ class MatrixEntry:
class PackagingEntry:
"""One entry in the generated packaging strategy matrix."""
artifact_name: str
xrpld_artifact_name: str
validator_keys_artifact_name: str
image: str
distro: str # e.g. "debian" or "rhel"; drives package-format-specific steps
@@ -210,14 +211,19 @@ def expand_linux_packaging(linux: LinuxFile) -> list[PackagingEntry]:
the nix-based build images, because deb/rpm tooling (debhelper, rpm-build)
is taken from the distro's archive rather than from nixpkgs. Each config
entry carries its own 'image'.
The artifact names must match what the build job uploads: one artifact per
binary, each named after the build config.
"""
entries = []
for distro, configs in linux.package_configs.items():
for cfg in configs:
for compiler, build_type in itertools.product(cfg.compiler, cfg.build_type):
config_name = f"{distro}-{compiler}-{build_type.lower()}-amd64"
entries.append(
PackagingEntry(
artifact_name=f"xrpld-{distro}-{compiler}-{build_type.lower()}-amd64",
xrpld_artifact_name=f"xrpld-{config_name}",
validator_keys_artifact_name=f"validator-keys-{config_name}",
image=cfg.image,
distro=distro,
)

View File

@@ -1,5 +1,5 @@
{
"image_tag": "sha-3122de8",
"image_tag": "sha-e29b523",
"configs": {
"ubuntu": [
{
@@ -68,7 +68,8 @@
"compiler": ["gcc"],
"build_type": ["Release"],
"arch": ["amd64"],
"minimal": false
"minimal": false,
"extra_cmake_args": "-Dvalidator_keys=ON"
}
],
@@ -77,7 +78,8 @@
"compiler": ["gcc"],
"build_type": ["Release"],
"arch": ["amd64"],
"minimal": false
"minimal": false,
"extra_cmake_args": "-Dvalidator_keys=ON"
}
]
},

View File

@@ -1,4 +1,4 @@
name: Build `nix` Docker images
name: Build Nix Docker images
on:
push:
@@ -8,7 +8,6 @@ on:
- ".github/workflows/build-nix-images.yml"
- "flake.nix"
- "flake.lock"
- "rust-toolchain.toml"
- "nix/**"
- "!nix/docker/README.md"
- "!nix/devshell.nix"
@@ -19,7 +18,6 @@ on:
- ".github/workflows/build-nix-images.yml"
- "flake.nix"
- "flake.lock"
- "rust-toolchain.toml"
- "nix/**"
- "!nix/docker/README.md"
- "!nix/devshell.nix"
@@ -38,7 +36,7 @@ defaults:
jobs:
build-merge:
name: Build and push `nix-${{ matrix.distro.name }}` image
name: Build and push nix-${{ matrix.distro.name }}
permissions:
contents: read
packages: write
@@ -56,7 +54,7 @@ jobs:
base_image: debian:bookworm
- name: rhel
base_image: registry.access.redhat.com/ubi9/ubi:latest
uses: XRPLF/actions/.github/workflows/build-multiarch-image.yml@9e7e4e80af9e684c116b38369add8eea64451f32
uses: XRPLF/actions/.github/workflows/build-multiarch-image.yml@ee03d31bcc4501d7599dc1b1ecd7a34af582ad1c
with:
image_name: xrpld/nix-${{ matrix.distro.name }}
dockerfile: nix/docker/Dockerfile

View File

@@ -1,4 +1,4 @@
name: Build `packaging` Docker images
name: Build packaging Docker images
on:
push:
@@ -26,7 +26,7 @@ defaults:
jobs:
build-merge:
name: Build and push `packaging-${{ matrix.distro.name }}` image
name: Build and push packaging-${{ matrix.distro.name }}
permissions:
contents: read
packages: write
@@ -38,7 +38,7 @@ jobs:
base_image: debian:bookworm
- name: rhel
base_image: registry.access.redhat.com/ubi9/ubi:latest
uses: XRPLF/actions/.github/workflows/build-multiarch-image.yml@9e7e4e80af9e684c116b38369add8eea64451f32
uses: XRPLF/actions/.github/workflows/build-multiarch-image.yml@ee03d31bcc4501d7599dc1b1ecd7a34af582ad1c
with:
image_name: xrpld/packaging-${{ matrix.distro.name }}
dockerfile: package/Dockerfile

View File

@@ -1,38 +0,0 @@
name: Build `pre-commit` Docker image
on:
push:
branches:
- develop
paths:
- ".github/workflows/build-pre-commit-image.yml"
- "bin/pre-commit/Dockerfile"
- "rust-toolchain.toml"
pull_request:
paths:
- ".github/workflows/build-pre-commit-image.yml"
- "bin/pre-commit/Dockerfile"
- "rust-toolchain.toml"
workflow_dispatch:
concurrency:
# Read `on-trigger.yml` for the rationale behind this concurrency group name.
group: ${{ github.workflow }}-${{ github.event_name == 'push' && github.ref == 'refs/heads/develop' && github.sha || github.ref }}
cancel-in-progress: true
defaults:
run:
shell: bash
jobs:
build-merge:
name: Build and push `pre-commit` image
permissions:
contents: read
packages: write
uses: XRPLF/actions/.github/workflows/build-multiarch-image.yml@9e7e4e80af9e684c116b38369add8eea64451f32
with:
image_name: xrpld/pre-commit
dockerfile: bin/pre-commit/Dockerfile
base_image: ubuntu:26.04
push: ${{ github.event_name == 'push' }}

View File

@@ -20,4 +20,4 @@ on:
jobs:
check_title:
if: ${{ github.event.pull_request.draft != true }}
uses: XRPLF/actions/.github/workflows/check-pr-title.yml@d7c65e49225a38f6d8010eacf017bb5a98d7476c
uses: XRPLF/actions/.github/workflows/check-pr-title.yml@cba1f0891650baf1a9c88624dc2d72573be2eb81

View File

@@ -14,7 +14,7 @@ on:
jobs:
# Call the workflow in the XRPLF/actions repo that runs the pre-commit hooks.
run-hooks:
uses: XRPLF/actions/.github/workflows/pre-commit.yml@3ba08d6ddf114092891d48491fc2e26c3ba15552
uses: XRPLF/actions/.github/workflows/pre-commit.yml@1bde119a1ab71305ba5d3716e7a82cea1c7bdede
with:
runs_on: ubuntu-latest
container: '{ "image": "ghcr.io/xrplf/xrpld/pre-commit:sha-f56b79f" }'
container: '{ "image": "ghcr.io/xrplf/ci/tools-rippled-pre-commit:sha-41ec7c1" }'

View File

@@ -41,13 +41,13 @@ env:
jobs:
build:
runs-on: ubuntu-latest
container: ghcr.io/xrplf/xrpld/nix-ubuntu:sha-3122de8
container: ghcr.io/xrplf/xrpld/nix-ubuntu:sha-e29b523
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Prepare runner
uses: XRPLF/actions/prepare-runner@e4b6449d55a61c002d7c3fdfa6c20f721ede0606
uses: XRPLF/actions/prepare-runner@64ec3cf3b152b4444638f470bbd6df7a7a30c81c
with:
enable_ccache: false

View File

@@ -100,9 +100,10 @@ jobs:
# header files are copied into separate directories by CMake, which will
# otherwise result in cache misses.
CCACHE_SLOPPINESS: include_file_ctime,include_file_mtime
# Determine if coverage and voidstar should be enabled.
# Determine if coverage, voidstar and validator-keys should be enabled.
COVERAGE_ENABLED: ${{ contains(inputs.cmake_args, '-Dcoverage=ON') }}
VOIDSTAR_ENABLED: ${{ contains(inputs.cmake_args, '-Dvoidstar=ON') }}
VALIDATOR_KEYS_ENABLED: ${{ contains(inputs.cmake_args, '-Dvalidator_keys=ON') }}
SANITIZERS_ENABLED: ${{ inputs.sanitizers != '' }}
steps:
- name: Cleanup workspace (macOS and Windows)
@@ -113,7 +114,7 @@ jobs:
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Prepare runner
uses: XRPLF/actions/prepare-runner@e4b6449d55a61c002d7c3fdfa6c20f721ede0606
uses: XRPLF/actions/prepare-runner@64ec3cf3b152b4444638f470bbd6df7a7a30c81c
with:
enable_ccache: ${{ inputs.ccache_enabled }}
@@ -223,13 +224,11 @@ jobs:
BUILD_TYPE: ${{ inputs.build_type }}
CMAKE_TARGET: ${{ inputs.cmake_target }}
run: |
set -o pipefail
cmake \
--build . \
--config "${BUILD_TYPE}" \
--parallel "${BUILD_NPROC}" \
--target "${CMAKE_TARGET}" \
2>&1 | tee "${GITHUB_WORKSPACE}/build.log"
--target "${CMAKE_TARGET}"
- name: Show ccache statistics
if: ${{ inputs.ccache_enabled }}
@@ -249,6 +248,22 @@ jobs:
retention-days: 3
if-no-files-found: error
- name: Run the validator-keys tests
if: ${{ env.VALIDATOR_KEYS_ENABLED == 'true' }}
working-directory: ${{ env.BUILD_DIR }}
run: ./validator-keys --unittest
- name: Upload the validator-keys binary
if: ${{ github.event.repository.visibility == 'public' && env.VALIDATOR_KEYS_ENABLED == 'true' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: validator-keys-${{ inputs.config_name }}
path: |
${{ env.BUILD_DIR }}/validator-keys
${{ env.BUILD_DIR }}/validator-keys-LICENSE
retention-days: 3
if-no-files-found: error
- name: Upload the test binary (Linux)
if: ${{ github.event.repository.visibility == 'public' && runner.os == 'Linux' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
@@ -324,46 +339,27 @@ jobs:
PRELOAD=""
fi
LD_PRELOAD="$PRELOAD" ./xrpld --unittest --unittest-jobs "${BUILD_NPROC}" 2>&1 | tee "${GITHUB_WORKSPACE}/unittest.log"
LD_PRELOAD="$PRELOAD" ./xrpld --unittest --unittest-jobs "${BUILD_NPROC}" 2>&1 | tee unittest.log
# Smoke-run every benchmark module with a single repetition to confirm the
# benchmarks still build and execute. This is a correctness check, not a
# performance measurement, so it is skipped for instrumented builds
# (sanitizers/coverage/voidstar), where it would be slow and meaningless,
# and on Windows, where the `install` target does not build them.
- name: Run the benchmarks
if: ${{ !inputs.build_only && runner.os != 'Windows' && env.SANITIZERS_ENABLED == 'false' && env.COVERAGE_ENABLED != 'true' && env.VOIDSTAR_ENABLED != 'true' }}
working-directory: ${{ env.BUILD_DIR }}
- name: Show test failure summary
if: ${{ failure() && !inputs.build_only }}
env:
WORKING_DIR: ${{ runner.os == 'Windows' && format('{0}\{1}', env.BUILD_DIR, inputs.build_type) || env.BUILD_DIR }}
run: |
rc=0
while IFS= read -r bench; do
echo "::group::${bench}"
"./${bench}" --benchmark_repetitions=1 || rc=1
echo "::endgroup::"
done < <(find src/benchmarks -type f -perm -u+x -name 'xrpl.bench.*')
exit "${rc}"
if [ ! -d "${WORKING_DIR}" ]; then
echo "Working directory '${WORKING_DIR}' does not exist."
exit 0
fi
- name: Show build/test failure summary
if: ${{ failure() }}
run: |
cd "${GITHUB_WORKSPACE}"
cd "${WORKING_DIR}"
if [ -f unittest.log ]; then
if ! grep -E "failed" unittest.log | grep -vE "^I[0-9]|^[0-9]+> (ERR:|FTL:)"; then
echo "unittest.log present but no failure lines found."
fi
elif [ -f build.log ]; then
# GCC/Clang emit "error:" (covers "fatal error:"); MSVC emits
# "error C####:", "error LNK####:", and "fatal error LNK####:".
# -A6 prints the lines that follow each match (source line, caret,
# notes, and the "N errors generated" tally) to capture the whole
# diagnostic block.
if ! grep -E -A6 "error:|error C[0-9]{4}|error LNK[0-9]{4}|fatal error" build.log; then
echo "build.log present but no compile errors found."
fi
else
echo "unittest.log/build.log not found; something went wrong."
exit 1
if [ ! -f unittest.log ]; then
echo "unittest.log not found; embedded tests may not have run."
exit 0
fi
if ! grep -E "failed" unittest.log; then
echo "Log present but no failure lines found in unittest.log."
fi
- name: Debug failure (Linux)
if: ${{ failure() && runner.os == 'Linux' && !inputs.build_only }}
@@ -387,7 +383,7 @@ jobs:
--target coverage
- name: Upload coverage report
if: ${{ github.repository == 'XRPLF/rippled' && !inputs.build_only && env.COVERAGE_ENABLED == 'true' }}
if: ${{ github.repository_owner == 'XRPLF' && !inputs.build_only && env.COVERAGE_ENABLED == 'true' }}
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
with:
disable_search: true

View File

@@ -34,7 +34,7 @@ jobs:
needs: [determine-files]
if: ${{ needs.determine-files.outputs.cpp_changed_files != '' || needs.determine-files.outputs.need_full_run == 'true' }}
runs-on: ["self-hosted", "Linux", "X64", "heavy"]
container: "ghcr.io/xrplf/xrpld/nix-debian:sha-3122de8"
container: "ghcr.io/xrplf/xrpld/nix-debian:sha-e29b523"
permissions:
contents: read
issues: write
@@ -43,7 +43,7 @@ jobs:
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Prepare runner
uses: XRPLF/actions/prepare-runner@e4b6449d55a61c002d7c3fdfa6c20f721ede0606
uses: XRPLF/actions/prepare-runner@64ec3cf3b152b4444638f470bbd6df7a7a30c81c
with:
enable_ccache: false

View File

@@ -1,7 +1,7 @@
# Build Linux packages (DEB and RPM) from pre-built binary artifacts.
# Discovers which configurations to package from linux.json (configs in
# "package_configs") and fans out one job per distro. Only linux/amd64 is
# supported; the runner is hardcoded in the job below.
# Build Linux packages (DEB and RPM) from pre-built binary artifacts (xrpld and
# validator-keys). Discovers which configurations to package from linux.json
# (configs in "package_configs") and fans out one job per distro. Only
# linux/amd64 is supported; the runner is hardcoded in the job below.
name: Package
on:
@@ -30,7 +30,7 @@ jobs:
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: "3.13"
@@ -45,7 +45,7 @@ jobs:
strategy:
fail-fast: false
matrix: ${{ fromJson(needs.generate-matrix.outputs.matrix) }}
name: "${{ matrix.artifact_name }}"
name: "${{ matrix.xrpld_artifact_name }}"
permissions:
contents: read
runs-on: ["self-hosted", "Linux", "X64", "heavy"]
@@ -56,14 +56,20 @@ jobs:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Download pre-built binary
- name: Download pre-built xrpld binary
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: ${{ matrix.artifact_name }}
name: ${{ matrix.xrpld_artifact_name }}
path: ${{ env.BUILD_DIR }}
- name: Make binary executable
run: chmod +x "${BUILD_DIR}/xrpld"
- name: Download pre-built validator-keys binary
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: ${{ matrix.validator_keys_artifact_name }}
path: ${{ env.BUILD_DIR }}
- name: Make binaries executable
run: chmod +x "${BUILD_DIR}/xrpld" "${BUILD_DIR}/validator-keys"
- name: Build package
env:
@@ -73,7 +79,7 @@ jobs:
- name: Upload package artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ matrix.artifact_name }}-pkg
name: ${{ matrix.xrpld_artifact_name }}-pkg
path: |
${{ env.BUILD_DIR }}/debbuild/*.deb
${{ env.BUILD_DIR }}/debbuild/*.ddeb

View File

@@ -26,7 +26,7 @@ jobs:
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
with:
python-version: "3.13"

View File

@@ -40,7 +40,7 @@ defaults:
jobs:
upload:
runs-on: ubuntu-latest
container: ghcr.io/xrplf/xrpld/nix-ubuntu:sha-3122de8
container: ghcr.io/xrplf/xrpld/nix-ubuntu:sha-e29b523
env:
REMOTE_NAME: ${{ inputs.remote_name }}
CONAN_LOGIN_USERNAME_XRPLF: ${{ secrets.remote_username }}

View File

@@ -68,7 +68,7 @@ jobs:
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: Prepare runner
uses: XRPLF/actions/prepare-runner@e4b6449d55a61c002d7c3fdfa6c20f721ede0606
uses: XRPLF/actions/prepare-runner@64ec3cf3b152b4444638f470bbd6df7a7a30c81c
with:
enable_ccache: false

3
.gitignore vendored
View File

@@ -81,9 +81,6 @@ DerivedData
# Python
__pycache__
# Rust build artifacts.
target/
# Direnv's directory
/.direnv

View File

@@ -55,7 +55,7 @@ repos:
types_or: [c++, c]
- repo: https://github.com/pre-commit/mirrors-clang-format
rev: f4d7745e17a28aad7eed2f4874ca8d1568c11c4c # frozen: v22.1.8
rev: dd18dad857d6133e90bbe478f4f2f22ec0030269 # frozen: v22.1.5
hooks:
- id: clang-format
args: [--style=file]
@@ -68,7 +68,7 @@ repos:
- id: gersemi
- repo: https://github.com/rbubley/mirrors-prettier
rev: 9337a74165b178ae2c766f60bee7252a0f06f3e8 # frozen: v3.9.5
rev: 39e2973981e6d2f9b6c543b0086a2d2393abdc89 # frozen: v3.9.4
hooks:
- id: prettier
args: [--end-of-line=auto]

View File

@@ -131,10 +131,6 @@ else()
endif()
target_link_libraries(xrpl_libs INTERFACE ${nudb})
if(benchmark)
find_package(benchmark REQUIRED)
endif()
if(coverage)
include(XrplCov)
endif()
@@ -142,14 +138,12 @@ endif()
include(XrplCore)
include(XrplProtocolAutogen)
include(XrplInstall)
include(XrplPackaging)
include(XrplValidatorKeys)
# Must come after XrplValidatorKeys: the 'package' target depends on the
# validator-keys target existing.
include(XrplPackaging)
if(tests)
include(CTest)
add_subdirectory(src/tests/libxrpl)
endif()
if(benchmark)
add_subdirectory(src/benchmarks/libxrpl)
endif()

View File

@@ -1,4 +0,0 @@
# By default, anyone can review changes.
# The CI tooling team should review changes to the CI configuration.
/.github/ @XRPLF/ci-tooling

View File

@@ -83,7 +83,6 @@ If you create new source files, they must be organized as follows:
`src/libxrpl`.
- All other non-test files must go under `src/xrpld`.
- All test source files must go under `src/test`.
- All benchmark source files must go under `src/benchmarks`.
The source must be formatted according to the style guide below. The easiest
way to satisfy this is to install the [`pre-commit`](#pre-commit-hooks) hooks,

View File

@@ -1,55 +0,0 @@
ARG BASE_IMAGE=ubuntu:26.04
FROM ${BASE_IMAGE}
SHELL ["/bin/bash", "-e", "-o", "pipefail", "-c"]
ENTRYPOINT ["/bin/bash"]
ARG DEBIAN_FRONTEND=noninteractive
RUN <<EOF
pkgs=()
pkgs+=(curl) # Required to install nix.
pkgs+=(doxygen) # Needed for Clio's check-doxygen-docs.sh.
pkgs+=(git) # Required for prepare-runner.
pkgs+=(libatomic1) # Required to run pre-commit provided `node`.
pkgs+=(python3) # Python 3 interpreter.
pkgs+=(python3-pip) # Package manager for Python applications.
pkgs+=(xz-utils) # Required to install nix
apt-get update
apt-get install -y --no-install-recommends "${pkgs[@]}"
apt-get clean
rm -rf /var/lib/apt/lists/*
EOF
ARG PRE_COMMIT_VERSION=4.6.0
RUN pip install --no-cache --break-system-packages \
pre-commit==${PRE_COMMIT_VERSION}
RUN sh <(curl --proto '=https' --tlsv1.2 -L https://nixos.org/nix/install) --daemon --yes
# Add nix to PATH and set NIX environment variables,
# so nix is available in all shells including non-interactive shells (e.g., GitHub Actions).
ENV PATH="/nix/var/nix/profiles/default/bin:${PATH}"
ENV NIX_PROFILES="/nix/var/nix/profiles/default"
ENV NIX_SSL_CERT_FILE="/nix/var/nix/profiles/default/etc/ssl/certs/ca-bundle.crt"
# Verify nix installation
RUN nix --version
ENV RUSTUP_HOME="/opt/rust/rustup"
ENV CARGO_HOME="/opt/rust/cargo"
ENV PATH="/opt/rust/cargo/bin:${PATH}"
WORKDIR /tmp
COPY rust-toolchain.toml /tmp/rust-toolchain.toml
RUN <<EOF
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs \
| sh -s -- -y --no-modify-path --profile minimal --default-toolchain none
rustup toolchain install
rustup show
cargo fmt --version
EOF
WORKDIR /

View File

@@ -488,6 +488,17 @@
# Must be a number between 100 and 1000, defaults to 250
#
#
# [max_subscriptions_per_connection]
#
# Maximum number of account, real-time account, and account-history
# subscriptions a single client connection may hold at once. Bounds the
# per-connection state torn down when the connection disconnects. Book
# subscriptions are tracked separately and are not counted here.
#
# Defaults to 100000 if not set; large enough for legitimate power users
# such as block explorers.
#
#
# [overlay]
#
# Controls settings related to the peer to peer overlay.
@@ -538,6 +549,45 @@
# only be used for local testing and debugging. Do not disable
# on mainnet.
#
# max_untrusted_count = <number>
#
# The number of manifests the server keeps for validators it does not
# list, and the number it sends and processes in a single peer protocol
# message. Once the server holds this many, a manifest for a new
# unlisted validator is rejected, so peer gossip cannot grow the cache
# without end.
#
# This option can take any value between 50 and 1000, inclusive. If
# the option is not present the server uses its built-in value.
#
# The current default (which is subject to change) is 300.
#
# max_trusted_count = <number>
#
# The number of manifests for listed validators to allow for when
# sizing peer protocol messages. Manifests for listed validators are
# never dropped, whether sending or receiving, because doing so would
# delay a validator key change reaching this server. Set this above the
# number of validators the server lists.
#
# Together the two counts above set the largest manifest message the
# server accepts: bigger messages are discarded without reading them,
# and without penalising the sender. Raising either means the server
# accepts and sends bigger messages than a peer using the defaults, and
# those peers will discard what this server sends. Lowering either below
# what peers send makes this server discard their manifest messages,
# which it does without recording anything.
#
# This option can take any value between 50 and 1000, inclusive. If
# the option is not present the server uses its built-in value.
#
# The current default (which is subject to change) is 300.
#
# NOTE: These two options (max_untrusted_count and max_trusted_count)
# are transitional. They exist to bound manifest-message size and cache
# growth during the network upgrade. They may be removed in a future
# release once the fleet has upgraded, and should not be relied upon as
# stable configuration.
#
# [transaction_queue] EXPERIMENTAL
#

View File

@@ -3,9 +3,9 @@
The Nix-based CI image links binaries against an ELF interpreter (loader)
that lives in the Nix store, so the resulting binaries don't run elsewhere
(including once installed from the .deb package). `patch_nix_binary` adds a
POST_BUILD step that resets the interpreter to the system default loader and
drops the rpath.
(including once installed from the .deb package). `patch_nix_binary` resets
the interpreter to the system default loader and drops the rpath, once the
binary has been linked.
This is only active inside the Nix-based image, detected by the presence of
/tmp/loader-path.sh (shipped by that image, resolves the default loader). It
@@ -41,13 +41,38 @@ function(patch_nix_binary target)
if(NOT PATCH_NIX_BINARIES)
return()
endif()
add_custom_command(
TARGET ${target}
POST_BUILD
COMMAND
"${PATCHELF_COMMAND}" --set-interpreter "${DEFAULT_LOADER_PATH}"
--remove-rpath "$<TARGET_FILE:${target}>"
COMMENT "Patching ${target}: set default loader, remove rpath"
VERBATIM
set(patch_command
"${PATCHELF_COMMAND}"
--set-interpreter
"${DEFAULT_LOADER_PATH}"
--remove-rpath
"$<TARGET_FILE:${target}>"
)
set(comment "Patching ${target}: set default loader, remove rpath")
# POST_BUILD is the cheap way to do this: it runs only when the binary is
# relinked. It is also only available in the directory that defined the
# target, so for a target from elsewhere (e.g. a FetchContent subproject)
# fall back to a custom target that runs after the binary is linked. That
# one runs on every build, which is harmless because patchelf is idempotent.
get_target_property(target_source_dir ${target} SOURCE_DIR)
if("${target_source_dir}" STREQUAL "${CMAKE_CURRENT_SOURCE_DIR}")
add_custom_command(
TARGET ${target}
POST_BUILD
COMMAND ${patch_command}
COMMENT "${comment}"
VERBATIM
)
else()
add_custom_target(
${target}-patch-nix
ALL
COMMAND ${patch_command}
COMMENT "${comment}"
VERBATIM
)
add_dependencies(${target}-patch-nix ${target})
endif()
endfunction()

View File

@@ -1,36 +0,0 @@
include(isolate_headers)
# Define a benchmark executable for the module `name`.
#
# This follows the same general pattern as other build helpers in this repo
# (e.g. `add_module`): create a target and isolate headers, but here the target
# is a benchmark executable and no `add_test(...)` is registered.
#
# `isolate_headers` exposes only `${CMAKE_CURRENT_SOURCE_DIR}/${name}` on the
# include path, rooted at `src`, so a benchmark's own headers are reached as
# `<benchmarks/.../${name}/...>` and nothing else in the tree leaks in.
function(xrpl_add_benchmark name)
set(target ${PROJECT_NAME}.bench.${name})
file(
GLOB_RECURSE sources
CONFIGURE_DEPENDS
"${CMAKE_CURRENT_SOURCE_DIR}/${name}/*.cpp"
"${CMAKE_CURRENT_SOURCE_DIR}/${name}.cpp"
)
add_executable(${target} ${ARGN} ${sources})
# Benchmark sources register cases through Google Benchmark's static
# registrars (anonymous-namespace lambdas). Merging several such files into
# one unity translation unit collides those internal-linkage entities, so
# keep benchmarks out of the unity build - mirroring xrpl.libpb in
# XrplCore.cmake. Each file compiles fine on its own.
set_target_properties(${target} PROPERTIES UNITY_BUILD OFF)
isolate_headers(
${target}
"${CMAKE_SOURCE_DIR}/src"
"${CMAKE_CURRENT_SOURCE_DIR}/${name}"
PRIVATE
)
endfunction()

View File

@@ -133,12 +133,6 @@ target_link_libraries(
add_module(xrpl resource)
target_link_libraries(xrpl.libxrpl.resource PUBLIC xrpl.libxrpl.protocol)
add_module(xrpl peerfinder)
target_link_libraries(
xrpl.libxrpl.peerfinder
PUBLIC xrpl.libxrpl.basics xrpl.libxrpl.protocol
)
# Level 08
add_module(xrpl net)
target_link_libraries(
@@ -207,16 +201,6 @@ target_link_libraries(
add_module(xrpl tx)
target_link_libraries(xrpl.libxrpl.tx PUBLIC xrpl.libxrpl.ledger)
add_module(xrpl consensus)
target_link_libraries(
xrpl.libxrpl.consensus
PUBLIC
xrpl.libxrpl.basics
xrpl.libxrpl.json
xrpl.libxrpl.protocol
xrpl.libxrpl.ledger
)
add_library(xrpl.libxrpl)
set_target_properties(xrpl.libxrpl PROPERTIES OUTPUT_NAME xrpl)
@@ -236,7 +220,6 @@ target_link_modules(
beast
conditions
config
consensus
core
crypto
git
@@ -244,7 +227,6 @@ target_link_modules(
ledger
net
nodestore
peerfinder
protocol
protocol_autogen
rdb

View File

@@ -25,6 +25,19 @@ if(NOT (RPMBUILD_EXECUTABLE OR DPKG_BUILDPACKAGE_EXECUTABLE))
return()
endif()
if(NOT TARGET xrpld)
message(STATUS "xrpld=ON is required; 'package' target not available")
return()
endif()
if(NOT TARGET validator-keys)
message(
STATUS
"validator_keys=ON is required; 'package' target not available"
)
return()
endif()
set(package_env
SRC_DIR=${CMAKE_SOURCE_DIR}
BUILD_DIR=${CMAKE_BINARY_DIR}
@@ -37,7 +50,7 @@ add_custom_target(
${CMAKE_COMMAND} -E env ${package_env}
${CMAKE_SOURCE_DIR}/package/build_pkg.sh
WORKING_DIRECTORY ${CMAKE_BINARY_DIR}
DEPENDS xrpld
DEPENDS xrpld validator-keys
COMMENT "Building Linux package (deb/rpm inferred from host tooling)"
VERBATIM
)

View File

@@ -30,8 +30,6 @@ if(tests)
endif()
endif()
option(benchmark "Build benchmarks" ON)
# Enabled by default so every header is compiled on its own as the main file of
# its own compile_commands.json entry - this is what lets clang-tidy (and clangd
# and IDEs) analyse a header's own includes directly. The per-header objects are

View File

@@ -5,22 +5,39 @@ option(
)
if(validator_keys)
git_branch(current_branch)
# default to tracking VK master branch unless we are on release
if(NOT (current_branch STREQUAL "release"))
set(current_branch "master")
endif()
message(STATUS "Tracking ValidatorKeys branch: ${current_branch}")
# Own the install destination below rather than relying on another module
# having pulled this in first.
include(GNUInstallDirs)
# Pinned to an exact commit, not a branch: the tool ships inside our
# packages, so the same xrpld version must always package the same
# validator-keys. Bump this deliberately.
set(validator_keys_commit "4c0fb75eec9601c711645998c904507e87e910ae")
message(STATUS "Using ValidatorKeys commit: ${validator_keys_commit}")
FetchContent_Declare(
validator_keys
GIT_REPOSITORY https://github.com/ripple/validator-keys-tool.git
GIT_TAG "${current_branch}"
GIT_TAG "${validator_keys_commit}"
)
FetchContent_MakeAvailable(validator_keys)
# The tool's own CMakeLists excludes the target from 'all' when it is built
# as a subproject. Undo that, so validator_keys=ON really does build it.
set_target_properties(
validator-keys
PROPERTIES RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}"
PROPERTIES
RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}"
EXCLUDE_FROM_ALL OFF
EXCLUDE_FROM_DEFAULT_BUILD OFF
)
# We ship this binary, so like xrpld it must not keep the Nix store's ELF
# loader, or it cannot run on the target distro at all.
patch_nix_binary(validator-keys)
configure_file(
"${validator_keys_SOURCE_DIR}/LICENSE"
"${CMAKE_BINARY_DIR}/validator-keys-LICENSE"
COPYONLY
)
install(TARGETS validator-keys RUNTIME DESTINATION ${CMAKE_INSTALL_BINDIR})
endif()

View File

@@ -12,7 +12,7 @@
"protobuf/6.33.5#ff253ead763bd8d9904a52979cd21e81%1782392410.233933",
"openssl/3.6.3#f806de8933e3bf6f01016c6a888cee2e%1783945160.863288",
"nudb/2.0.9#11149c73f8f2baff9a0198fe25971fc7%1782392402.297166",
"mpt-crypto/0.4.0-rc4#ffdba12f2332357f0d8b0ae944cfff52%1784138702.932355",
"mpt-crypto/1.0.2#b313cef0c1a493eb970ad185b2e9bab7%1784285108.866483",
"lz4/1.10.0#982d9b673900f665a1da109e09c17cab%1782392402.164188",
"libiconv/1.17#9923bc6dc6f106646d6967e0039a5ada%1782392792.775744",
"libbacktrace/cci.20210118#a7691bfccd8caaf66309df196790a5a1%1782392402.420732",
@@ -25,7 +25,6 @@
"c-ares/1.34.6#545240bb1c40e2cacd4362d6b8967650%1782392402.681654",
"bzip2/1.0.8#c470882369c2d95c5c77e970c0c7e321%1782392402.296732",
"boost/1.91.0#ea540ca2133d831b560036aa24dece3c%1782392419.475605",
"benchmark/1.9.5#b885dc73ad67b40a55d45684d1c88ad1%1782736613.864841",
"abseil/20250127.0#9ef01c1451a8340f9022e46238c0fbb6%1783945159.651047"
],
"build_requires": [

View File

@@ -20,22 +20,6 @@ compiler.libcxx={{ detect_api.detect_libcxx(compiler, version, compiler_exe) }}
{% endif %}
[conf]
{# The Boost recipe builds with b2, which doesn't use Conan's toolchain files. #}
{# Instead it hand-rolls the compiler for user-config.jam, #}
{# and its fallback probes a version-suffixed binary (e.g. `g++-15`) before plain `g++`. #}
{# Inside the Nix shell the wrapper only provides `g++`/`gcc` (no `-15` suffix), #}
{# so on a host that also has a system `g++-15` the probe escapes Nix #}
{# and picks the system compiler, which is mismatched with the Nix libraries #}
{# and breaks the build (e.g. Boost.Stacktrace link checks fail). #}
{# Pinning the executables here short-circuits that probe so Boost (and the rest of the toolchain) #}
{# resolve the same compiler. #}
{# Not part of the package ID, so binaries stay shareable. #}
{% if os != "Windows" %}
{% set cc_exe = {"gcc": "gcc", "clang": "clang", "apple-clang": "clang"}.get(compiler) %}
{% set cxx_exe = {"gcc": "g++", "clang": "clang++", "apple-clang": "clang++"}.get(compiler) %}
tools.build:compiler_executables={'c':'{{ cc_exe }}','cpp':'{{ cxx_exe }}'}
{% endif %}
{# By default, Conan tries to reuse binaries built with different cppstd versions. #}
{# We want to avoid that to improve reproduceability, so we add the cppstd version to the package ID. #}
{# More info: https://docs.conan.io/2/reference/extensions/binary_compatibility.html #}

View File

@@ -15,7 +15,6 @@ class Xrpl(ConanFile):
settings = "os", "compiler", "build_type", "arch"
options = {
"assertions": [True, False],
"benchmark": [True, False],
"coverage": [True, False],
"fPIC": [True, False],
"jemalloc": [True, False],
@@ -47,7 +46,6 @@ class Xrpl(ConanFile):
default_options = {
"assertions": False,
"benchmark": True,
"coverage": False,
"fPIC": True,
"jemalloc": False,
@@ -131,14 +129,12 @@ class Xrpl(ConanFile):
self.options["boost"].without_cobalt = True
def requirements(self):
if self.options.benchmark:
self.requires("benchmark/1.9.5")
self.requires("boost/1.91.0", force=True, transitive_headers=True)
self.requires("date/3.0.4", transitive_headers=True)
if self.options.jemalloc:
self.requires("jemalloc/5.3.1")
self.requires("lz4/1.10.0", force=True)
self.requires("mpt-crypto/0.4.0-rc4", transitive_headers=True)
self.requires("mpt-crypto/1.0.2", transitive_headers=True)
self.requires("protobuf/6.33.5", force=True)
if self.options.rocksdb:
self.requires("rocksdb/10.5.1")
@@ -166,7 +162,6 @@ class Xrpl(ConanFile):
def generate(self):
tc = CMakeToolchain(self)
tc.variables["tests"] = self.options.tests
tc.variables["benchmark"] = self.options.benchmark
tc.variables["assert"] = self.options.assertions
tc.variables["coverage"] = self.options.coverage
tc.variables["jemalloc"] = self.options.jemalloc

30
docs/build/nix.md vendored
View File

@@ -38,10 +38,8 @@ The first time you run this command, it will take a few minutes to download and
### Platform notes
- **Linux**: `nix develop` gives you a shell with all the tooling necessary to develop xrpld
and with the same GCC/glibc toolchain that Nix builds for CI.
See [Choosing a different compiler](#choosing-a-different-compiler)
for the custom-vs-plain toolchain trade-off.
- **Linux**: `nix develop` gives you a shell with all the tooling necessary to
develop xrpld and with GCC 15.2 (also provided by Nix). There are no caveats.
- **macOS**: `nix develop` gives you a full environment too, with Clang (and
every other tool, including Conan) provided by Nix. To use your system-wide
Apple Clang instead, enter `nix develop .#apple-clang`. Conan has no binary in
@@ -65,16 +63,8 @@ The first time you run this command, it will take a few minutes to download and
### Choosing a different compiler
A compiler can be chosen by providing its name with the `.#` prefix, e.g. `nix develop .#clang`.
On Linux, `.#gcc` and `.#clang` provide the exact toolchain CI uses:
the compiler (pinned in [`nix/packages.nix`](../../nix/packages.nix))
rebuilt against the pinned custom glibc (see [`nix/compilers.nix`](../../nix/compilers.nix)).
Building that toolchain the first time is slow unless it is fetched from a Nix binary cache.
If you don't need the custom glibc, the Linux-only `.#gcc-plain` and `.#clang-plain`
give you the stock nixpkgs compilers of the same versions.
On macOS there is no custom glibc, so `.#gcc` and `.#clang` are already the plain nixpkgs toolchain,
and the `-plain` variants do not exist.
The `.#gcc` and `.#clang` shells provide the same GCC and Clang versions used in CI
(pinned in [`nix/packages.nix`](../../nix/packages.nix)).
Use `nix flake show` to see all the available development shells.
Use `nix develop .#no-compiler` to use the compiler from your system.
@@ -82,18 +72,14 @@ Use `nix develop .#no-compiler` to use the compiler from your system.
### Example Usage
```bash
# Use GCC same toolchain as CI (custom glibc on Linux)
# Use GCC (same version as CI)
nix develop .#gcc
# Use Clang same toolchain as CI (custom glibc on Linux)
# Use Clang (same version as CI)
nix develop .#clang
# Use default for your platform
nix develop
# Stock nixpkgs GCC/Clang, Linux only — skips the custom-glibc build, but does not match CI
nix develop .#gcc-plain
nix develop .#clang-plain
```
### Using a different shell
@@ -124,10 +110,6 @@ nix develop -c "$SHELL"
Once inside the Nix development shell, follow the standard [build instructions](../../BUILD.md#steps). The Nix shell provides all necessary tools (CMake, Ninja, Conan, etc.).
Coverage builds (`-Dcoverage=ON`) work in the `gcc` shell (and `gcc-plain` on Linux):
each ships a `gcov` matching its compiler, since Nix's cc-wrapper does not expose one.
The `clang` shells do not include `llvm-cov`, so use a `gcc` shell for coverage.
## Automatic Activation with direnv
[direnv](https://direnv.net/) or [nix-direnv](https://github.com/nix-community/nix-direnv) can automatically activate the Nix development shell when you enter the repository directory.

23
flake.lock generated
View File

@@ -36,28 +36,7 @@
"root": {
"inputs": {
"nixpkgs": "nixpkgs",
"nixpkgs-custom-glibc": "nixpkgs-custom-glibc",
"rust-overlay": "rust-overlay"
}
},
"rust-overlay": {
"inputs": {
"nixpkgs": [
"nixpkgs"
]
},
"locked": {
"lastModified": 1784611586,
"narHash": "sha256-OfqgY+0hp/zseZB7uyH0U8kIDPS4scZZCyAurEplvG0=",
"owner": "oxalica",
"repo": "rust-overlay",
"rev": "14f58845249f3552a89b07772626b8d3c632fa86",
"type": "github"
},
"original": {
"owner": "oxalica",
"repo": "rust-overlay",
"type": "github"
"nixpkgs-custom-glibc": "nixpkgs-custom-glibc"
}
}
},

View File

@@ -10,25 +10,12 @@
url = "github:NixOS/nixpkgs/9cd98386a38891d1074fc18036b842dc4416f562";
flake = false;
};
# Pinned Rust toolchains, delivered from the Nix store. Lets the Nix CI
# image and dev shell honour the single `rust-toolchain.toml` pin (shared
# with the rustup-based non-Nix runners) while staying hermetic — the
# toolchain lands in the image's Nix closure and is locked by flake.lock.
rust-overlay = {
url = "github:oxalica/rust-overlay";
inputs.nixpkgs.follows = "nixpkgs";
};
};
outputs =
{
nixpkgs,
nixpkgs-custom-glibc,
rust-overlay,
...
}:
{ nixpkgs, nixpkgs-custom-glibc, ... }:
let
forEachSystem = import ./nix/utils.nix { inherit nixpkgs nixpkgs-custom-glibc rust-overlay; };
forEachSystem = import ./nix/utils.nix { inherit nixpkgs nixpkgs-custom-glibc; };
in
{
devShells = forEachSystem (import ./nix/devshell.nix);

View File

@@ -41,6 +41,35 @@
namespace xrpl {
namespace base64 {
/**
* Returns the maximum number of characters needed to base64-encode @p nBytes bytes.
*
* @param nBytes Number of input bytes.
* @return Size of the encoded string, including padding.
*/
constexpr std::size_t
encodedSize(std::size_t const nBytes)
{
return 4 * ((nBytes + 2) / 3);
}
/**
* Returns the maximum number of bytes a base64 string of @p numChars characters
* decodes to.
*
* @param numChars Number of base64 characters.
* @return Upper bound on the number of decoded bytes.
*/
constexpr std::size_t
decodedSize(std::size_t const numChars)
{
return ((numChars / 4) * 3) + 2;
}
} // namespace base64
std::string
base64Encode(std::uint8_t const* data, std::size_t len);

View File

@@ -5,7 +5,6 @@
#include <xrpl/beast/container/detail/aged_associative_container.h>
#include <xrpl/beast/container/detail/aged_container_iterator.h>
#include <xrpl/beast/container/detail/empty_base_optimization.h>
#include <xrpl/beast/utility/instrumentation.h>
#include <boost/intrusive/list.hpp>
#include <boost/intrusive/unordered_set.hpp>

View File

@@ -25,6 +25,7 @@ struct Sections
static constexpr auto kLedgerHistory = "ledger_history";
static constexpr auto kLedgerReplay = "ledger_replay";
static constexpr auto kLedgerTxTables = "ledger_tx_tables";
static constexpr auto kMaxSubscriptionsPerConnection = "max_subscriptions_per_connection";
static constexpr auto kMaxTransactions = "max_transactions";
static constexpr auto kNetworkId = "network_id";
static constexpr auto kNetworkQuorum = "network_quorum";
@@ -118,7 +119,9 @@ struct Keys
static constexpr auto kLogInterval = "log_interval";
static constexpr auto kMaxDivergedTime = "max_diverged_time";
static constexpr auto kMaxLedgerCountsToStore = "max_ledger_counts_to_store";
static constexpr auto kMaxTrustedCount = "max_trusted_count";
static constexpr auto kMaxUnknownTime = "max_unknown_time";
static constexpr auto kMaxUntrustedCount = "max_untrusted_count";
static constexpr auto kMaximumTxnInLedger = "maximum_txn_in_ledger";
static constexpr auto kMaximumTxnPerAccount = "maximum_txn_per_account";
static constexpr auto kMemoryLevel = "memory_level";

View File

@@ -263,11 +263,10 @@ constructLoanState(
Number const& principalOutstanding,
Number const& managementFeeOutstanding);
// Overload of constructLoanState() that reads the three tracked fields
// directly from a Loan ledger object, which always holds rounded values,
// rather than taking them as separate Number arguments.
// Constructs a valid LoanState object from a Loan object, which always has
// rounded values
LoanState
constructLoanState(SLE::const_ref loan);
constructRoundedLoanState(SLE::const_ref loan);
Number
computeManagementFee(

View File

@@ -1,163 +0,0 @@
#pragma once
#include <xrpl/beast/utility/PropertyStream.h>
#include <xrpl/peerfinder/detail/Tuning.h>
#include <cstddef>
#include <cstdint>
#include <optional>
#include <string>
#include <string_view>
namespace xrpl::PeerFinder {
struct PeerLimitConfig
{
std::optional<std::size_t> maxPeers;
std::optional<std::size_t> inPeers;
std::optional<std::size_t> outPeers;
};
/**
* PeerFinder configuration settings.
*/
struct Config
{
/**
* The largest number of public peer slots to allow.
* This includes both inbound and outbound, but does not include
* fixed peers.
*/
std::size_t maxPeers{Tuning::kDefaultMaxPeers};
/**
* The number of automatic outbound connections to maintain.
* Outbound connections are only maintained if autoConnect
* is `true`.
*/
std::size_t outPeers = calcOutPeers(); // Note: relies on `maxPeers` being initialized
/**
* The number of automatic inbound connections to maintain.
* Inbound connections are only maintained if wantIncoming
* is `true`.
*/
std::size_t inPeers{0};
/**
* `true` if we want our IP address kept private.
*/
bool peerPrivate = true;
/**
* `true` if we want to accept incoming connections.
*/
bool wantIncoming{true};
/**
* `true` if we want to establish connections automatically
*/
bool autoConnect{true};
/**
* The listening port number.
*/
std::uint16_t listeningPort{0};
/**
* The set of features we advertise.
*/
std::string features;
/**
* Limit how many incoming connections we allow per IP
*/
int ipLimit{0};
/**
* `true` if we want to verify endpoints in TMEndpoints messages
*/
bool verifyEndpoints = true;
//--------------------------------------------------------------------------
/**
* Returns a suitable value for outPeers according to the rules.
*/
[[nodiscard]] std::size_t
calcOutPeers() const;
/**
* Adjusts the values so they follow the business rules.
*/
void
applyTuning();
/**
* Write the configuration into a property stream
*/
void
onWrite(beast::PropertyStream::Map& map) const;
/**
* Make PeerFinder::Config from peer limit and server mode parameters.
*/
static Config
makeConfig(
bool peerPrivate,
bool standalone,
PeerLimitConfig const& limits,
std::uint16_t port,
bool validationPublicKey,
int ipLimit,
bool verifyEndpoints);
/**
* Compares two configurations for equality field by field.
*/
friend bool
operator==(Config const& lhs, Config const& rhs) = default;
};
//------------------------------------------------------------------------------
/**
* Possible results from activating a slot.
*/
enum class Result { InboundDisabled, DuplicatePeer, IpLimitExceeded, Full, Success };
/**
* @brief Converts a `Result` enum value to its string representation.
*
* This function provides a human-readable string for a given `Result` enum,
* which is useful for logging, debugging, or displaying status messages.
*
* @param result The `Result` enum value to convert.
* @return A `std::string_view` representing the enum value. Returns "unknown"
* if the enum value is not explicitly handled.
*
* @note This function returns a `std::string_view` for performance.
* A `std::string` would need to allocate memory on the heap and copy the
* string literal into it every time the function is called.
*/
inline std::string_view
to_string(Result result) noexcept
{
switch (result)
{
case Result::InboundDisabled:
return "inbound disabled";
case Result::DuplicatePeer:
return "peer already connected";
case Result::IpLimitExceeded:
return "ip limit exceeded";
case Result::Full:
return "slots full";
case Result::Success:
return "success";
}
return "unknown";
}
} // namespace xrpl::PeerFinder

View File

@@ -1,179 +0,0 @@
#pragma once
#include <xrpl/beast/net/IPEndpoint.h>
#include <xrpl/beast/utility/PropertyStream.h>
#include <xrpl/peerfinder/Config.h>
#include <xrpl/peerfinder/Slot.h>
#include <xrpl/peerfinder/Types.h>
#include <xrpl/protocol/PublicKey.h>
#include <boost/asio/ip/tcp.hpp>
#include <memory>
#include <string>
#include <string_view>
#include <utility>
#include <vector>
namespace xrpl::PeerFinder {
/**
* Maintains a set of IP addresses used for getting into the network.
*/
class Manager : public beast::PropertyStream::Source
{
protected:
Manager() noexcept;
public:
/**
* Destroy the object.
* Any pending source fetch operations are aborted.
* There may be some listener calls made before the
* destructor returns.
*/
~Manager() override = default;
/**
* Set the configuration for the manager.
* The new settings will be applied asynchronously.
* Thread safety:
* Can be called from any threads at any time.
*/
virtual void
setConfig(Config const& config) = 0;
/**
* Transition to the started state, synchronously.
*/
virtual void
start() = 0;
/**
* Transition to the stopped state, synchronously.
*/
virtual void
stop() = 0;
/**
* Returns the configuration for the manager.
*/
virtual Config
config() = 0;
/**
* Add a peer that should always be connected.
* This is useful for maintaining a private cluster of peers.
* The string is the name as specified in the configuration
* file, along with the set of corresponding IP addresses.
*/
virtual void
addFixedPeer(std::string_view name, std::vector<beast::IP::Endpoint> const& addresses) = 0;
/**
* Add a set of strings as fallback IP::Endpoint sources.
* @param name A label used for diagnostics.
*/
virtual void
addFallbackStrings(std::string const& name, std::vector<std::string> const& strings) = 0;
/**
* Add a URL as a fallback location to obtain IP::Endpoint sources.
* @param name A label used for diagnostics.
*/
/* VFALCO NOTE Unimplemented
virtual void addFallbackURL (std::string const& name,
std::string const& url) = 0;
*/
//--------------------------------------------------------------------------
/**
* Create a new inbound slot with the specified remote endpoint.
* If nullptr is returned, then the slot could not be assigned.
* Usually this is because of a detected self-connection.
*/
virtual std::pair<std::shared_ptr<Slot>, Result>
newInboundSlot(
beast::IP::Endpoint const& localEndpoint,
beast::IP::Endpoint const& remoteEndpoint) = 0;
/**
* Create a new outbound slot with the specified remote endpoint.
* If nullptr is returned, then the slot could not be assigned.
* Usually this is because of a duplicate connection.
*/
virtual std::pair<std::shared_ptr<Slot>, Result>
newOutboundSlot(beast::IP::Endpoint const& remoteEndpoint) = 0;
/**
* Called when mtENDPOINTS is received.
*/
virtual void
onEndpoints(std::shared_ptr<Slot> const& slot, Endpoints const& endpoints) = 0;
/**
* Called when the slot is closed.
* This always happens when the socket is closed, unless the socket
* was canceled.
*/
virtual void
onClosed(std::shared_ptr<Slot> const& slot) = 0;
/**
* Called when an outbound connection is deemed to have failed
*/
virtual void
onFailure(std::shared_ptr<Slot> const& slot) = 0;
/**
* Called when we received redirect IPs from a busy peer.
*/
virtual void
onRedirects(
boost::asio::ip::tcp::endpoint const& remoteAddress,
std::vector<boost::asio::ip::tcp::endpoint> const& eps) = 0;
//--------------------------------------------------------------------------
/**
* Called when an outbound connection attempt succeeds.
* The local endpoint must be valid. If the caller receives an error
* when retrieving the local endpoint from the socket, it should
* proceed as if the connection attempt failed by calling on_closed
* instead of on_connected.
* @return `true` if the connection should be kept
*/
virtual bool
onConnected(std::shared_ptr<Slot> const& slot, beast::IP::Endpoint const& localEndpoint) = 0;
/**
* Request an active slot type.
*/
virtual Result
activate(std::shared_ptr<Slot> const& slot, PublicKey const& key, bool reserved) = 0;
/**
* Returns a set of endpoints suitable for redirection.
*/
virtual std::vector<Endpoint>
redirect(std::shared_ptr<Slot> const& slot) = 0;
/**
* Return a set of addresses we should connect to.
*/
virtual std::vector<beast::IP::Endpoint>
autoconnect() = 0;
virtual std::vector<std::pair<std::shared_ptr<Slot>, std::vector<Endpoint>>>
buildEndpointsForPeers() = 0;
/**
* Perform periodic activity.
* This should be called once per second.
*/
virtual void
oncePerSecond() = 0;
};
} // namespace xrpl::PeerFinder

View File

@@ -1,46 +0,0 @@
#pragma once
#include <xrpl/beast/clock/abstract_clock.h>
#include <xrpl/beast/net/IPEndpoint.h>
#include <xrpl/peerfinder/detail/Tuning.h>
#include <chrono>
#include <cstdint>
#include <vector>
namespace xrpl::PeerFinder {
using clock_type = beast::AbstractClock<std::chrono::steady_clock>;
/**
* Represents a set of addresses.
*/
using IPAddresses = std::vector<beast::IP::Endpoint>;
//------------------------------------------------------------------------------
/**
* Describes a connectable peer address along with some metadata.
*/
struct Endpoint
{
Endpoint() = default;
Endpoint(beast::IP::Endpoint ep, std::uint32_t hops);
std::uint32_t hops = 0;
beast::IP::Endpoint address;
};
inline bool
operator<(Endpoint const& lhs, Endpoint const& rhs)
{
return lhs.address < rhs.address;
}
/**
* A set of Endpoint used for connecting.
*/
using Endpoints = std::vector<Endpoint>;
} // namespace xrpl::PeerFinder

View File

@@ -1,36 +0,0 @@
#pragma once
#include <xrpl/beast/insight/Collector.h>
#include <xrpl/beast/utility/Journal.h>
#include <xrpl/peerfinder/PeerfinderManager.h>
#include <xrpl/peerfinder/Types.h>
#include <xrpl/peerfinder/detail/Store.h>
#include <boost/asio/io_context.hpp>
#include <memory>
namespace xrpl::PeerFinder {
/**
* @brief Create a new Manager.
*
* @param ioContext The io_context used to schedule asynchronous work.
* @param clock The clock used for timekeeping.
* @param journal The journal used for logging.
* @param store The persistence backend for the bootstrap cache. The caller
* retains ownership and must keep it alive (and opened) for the lifetime of
* the returned Manager. This lets consumers supply their own Store
* implementation (e.g. the SQLite-backed StoreSqdb in xrpld).
* @param collector The collector used to report metrics.
* @return The newly created Manager.
*/
std::unique_ptr<Manager>
makeManager(
boost::asio::io_context& ioContext,
clock_type& clock,
beast::Journal journal,
Store& store,
beast::insight::Collector::ptr const& collector);
} // namespace xrpl::PeerFinder

View File

@@ -293,14 +293,15 @@ message TMLedgerData {
}
message TMPing {
// Previously used - don't reuse.
reserved 3, 4;
enum pingType {
ptPING = 0; // we want a reply
ptPONG = 1; // this is a reply
}
required pingType type = 1;
optional uint32 seq = 2; // detect stale replies, ensure other side is reading
optional uint64 pingTime = 3; // know when we think we sent the ping
optional uint64 netTime = 4;
optional uint32 seq = 2; // detect stale replies, ensure other side is reading
}
message TMSquelch {

View File

@@ -190,17 +190,6 @@ enum LedgerEntryType : std::uint16_t {
LSF_FLAG(lsfMPTCanClawback, 0x00000040) \
LSF_FLAG(lsfMPTCanHoldConfidentialBalance, 0x00000080)) \
\
LEDGER_OBJECT(MPTokenIssuanceMutable, \
LSF_FLAG(lsmfMPTCanEnableCanLock, 0x00000002) \
LSF_FLAG(lsmfMPTCanEnableRequireAuth, 0x00000004) \
LSF_FLAG(lsmfMPTCanEnableCanEscrow, 0x00000008) \
LSF_FLAG(lsmfMPTCanEnableCanTrade, 0x00000010) \
LSF_FLAG(lsmfMPTCanEnableCanTransfer, 0x00000020) \
LSF_FLAG(lsmfMPTCanEnableCanClawback, 0x00000040) \
LSF_FLAG(lsmfMPTCannotEnableCanHoldConfidentialBalance, 0x00000080) \
LSF_FLAG(lsmfMPTCanMutateMetadata, 0x00010000) \
LSF_FLAG(lsmfMPTCanMutateTransferFee, 0x00020000)) \
\
LEDGER_OBJECT(MPToken, \
LSF_FLAG2(lsfMPTLocked, 0x00000001) \
LSF_FLAG(lsfMPTAuthorized, 0x00000002) \
@@ -294,6 +283,17 @@ getAllLedgerFlags()
#pragma pop_macro("TO_MAP")
#pragma pop_macro("ALL_LEDGER_FLAGS")
// MPTokenIssuance ImmutableFlags (sfImmutableFlags)
inline constexpr std::uint32_t lsifMPTCanLock = 0x00000002;
inline constexpr std::uint32_t lsifMPTRequireAuth = 0x00000004;
inline constexpr std::uint32_t lsifMPTCanEscrow = 0x00000008;
inline constexpr std::uint32_t lsifMPTCanTrade = 0x00000010;
inline constexpr std::uint32_t lsifMPTCanTransfer = 0x00000020;
inline constexpr std::uint32_t lsifMPTCanClawback = 0x00000040;
inline constexpr std::uint32_t lsifMPTCanHoldConfidentialBalance = 0x00000080;
inline constexpr std::uint32_t lsifMPTMetadata = 0x00010000;
inline constexpr std::uint32_t lsifMPTTransferFee = 0x00020000;
//------------------------------------------------------------------------------
/**

View File

@@ -90,7 +90,11 @@ public:
operator=(STObject&& other);
STObject(SOTemplate const& type, SField const& name);
STObject(SOTemplate const& type, SerialIter& sit, SField const& name);
STObject(
SOTemplate const& type,
SerialIter& sit,
SField const& name,
bool requireCanonicalOrder = false);
STObject(SerialIter& sit, SField const& name, int depth = 0);
STObject(SerialIter&& sit, SField const& name);
explicit STObject(SField const& name);
@@ -123,7 +127,7 @@ public:
set(SOTemplate const&);
bool
set(SerialIter& u, int depth = 0);
set(SerialIter& u, int depth = 0, bool requireCanonicalOrder = false);
[[nodiscard]] SerializedTypeID
getSType() const override;

View File

@@ -1,6 +1,7 @@
#pragma once
#include <xrpl/basics/CountedObject.h>
#include <xrpl/basics/UnorderedContainers.h>
#include <xrpl/beast/utility/instrumentation.h>
#include <xrpl/json/json_value.h>
#include <xrpl/protocol/AccountID.h>
@@ -108,6 +109,9 @@ public:
[[nodiscard]] bool
isType(Type const& pe) const;
[[nodiscard]] size_t
getHash() const;
bool
operator==(STPathElement const& t) const;
@@ -171,12 +175,23 @@ public:
reserve(size_t s);
};
template <class Hasher>
void
hash_append(Hasher& h, STPath const& p) noexcept
{
for (auto const& e : p)
{
beast::hash_append(h, e.getHash());
}
}
//------------------------------------------------------------------------------
// A set of zero or more payment paths
class STPathSet final : public STBase, public CountedObject<STPathSet>
{
std::vector<STPath> value_;
xrpl::hardened_hash_set<STPath> seenHashes_;
public:
STPathSet() = default;
@@ -205,9 +220,6 @@ public:
std::vector<STPath>::const_reference
operator[](std::vector<STPath>::size_type n) const;
std::vector<STPath>::reference
operator[](std::vector<STPath>::size_type n);
[[nodiscard]] std::vector<STPath>::const_iterator
begin() const;
@@ -227,6 +239,9 @@ public:
void
emplaceBack(Args&&... args);
[[nodiscard]] bool
contains(STPath const& path) const;
private:
STBase*
copy(std::size_t n, void* buf) const override;
@@ -515,12 +530,6 @@ STPathSet::operator[](std::vector<STPath>::size_type n) const
return value_[n];
}
inline std::vector<STPath>::reference
STPathSet::operator[](std::vector<STPath>::size_type n)
{
return value_[n];
}
inline std::vector<STPath>::const_iterator
STPathSet::begin() const
{
@@ -549,6 +558,7 @@ inline void
STPathSet::pushBack(STPath const& e)
{
value_.push_back(e);
seenHashes_.emplace(value_.back());
}
template <typename... Args>
@@ -556,6 +566,13 @@ inline void
STPathSet::emplaceBack(Args&&... args)
{
value_.emplace_back(std::forward<Args>(args)...);
seenHashes_.emplace(value_.back());
}
inline bool
STPathSet::contains(STPath const& path) const
{
return seenHashes_.contains(path);
}
} // namespace xrpl

View File

@@ -54,6 +54,22 @@ class STValidation final : public STObject, public CountedObject<STValidation>
NetClock::time_point seenTime_;
public:
/**
* @struct DeserializeOptions
* @brief Options controlling deserialization of a STValidation.
* @var DeserializeOptions::checkSignature
* Whether to verify the data was signed properly
*
* @var DeserializeOptions::requireCanonicalOrder
* Whether to require the fields to be in canonical order
*/
struct DeserializeOptions
{
bool checkSignature;
bool requireCanonicalOrder;
};
/**
* Construct a STValidation from a peer from serialized data.
*
@@ -64,12 +80,12 @@ public:
* that signed the validation. For manifest based
* validators, this should be the NodeID of the master
* public key.
* @param checkSignature Whether to verify the data was signed properly
* @param options Options controlling deserialization
*
* @note Throws if the object is not valid
*/
template <class LookupNodeID>
STValidation(SerialIter& sit, LookupNodeID&& lookupNodeID, bool checkSignature);
STValidation(SerialIter& sit, LookupNodeID&& lookupNodeID, DeserializeOptions options);
/**
* Construct, sign and trust a new STValidation issued by this node.
@@ -163,8 +179,8 @@ private:
};
template <class LookupNodeID>
STValidation::STValidation(SerialIter& sit, LookupNodeID&& lookupNodeID, bool checkSignature)
: STObject(validationFormat(), sit, sfValidation)
STValidation::STValidation(SerialIter& sit, LookupNodeID&& lookupNodeID, DeserializeOptions options)
: STObject(validationFormat(), sit, sfValidation, options.requireCanonicalOrder)
, signingPubKey_([this]() {
auto const spk = getFieldVL(sfSigningPubKey);
@@ -175,7 +191,7 @@ STValidation::STValidation(SerialIter& sit, LookupNodeID&& lookupNodeID, bool ch
}())
, nodeID_(lookupNodeID(signingPubKey_))
{
if (checkSignature && !isValid())
if (options.checkSignature && !isValid())
{
JLOG(debugLog().error()) << "Invalid signature in validation: "
<< getJson(JsonOptions::Values::None);

View File

@@ -152,7 +152,14 @@ inline constexpr FlagValue tfUniversalMask = ~tfUniversal;
\
TRANSACTION(MPTokenIssuanceSet, \
TF_FLAG(tfMPTLock, 0x00000001) \
TF_FLAG(tfMPTUnlock, 0x00000002), \
TF_FLAG(tfMPTUnlock, 0x00000002) \
TF_FLAG(tfMPTSetCanLock, 0x00000004) \
TF_FLAG(tfMPTSetRequireAuth, 0x00000008) \
TF_FLAG(tfMPTSetCanEscrow, 0x00000010) \
TF_FLAG(tfMPTSetCanTrade, 0x00000020) \
TF_FLAG(tfMPTSetCanTransfer, 0x00000040) \
TF_FLAG(tfMPTSetCanClawback, 0x00000080) \
TF_FLAG(tfMPTSetCanHoldConfidentialBalance, 0x00000100), \
MASK_ADJ(0)) \
\
TRANSACTION(NFTokenCreateOffer, \
@@ -356,38 +363,26 @@ inline constexpr FlagValue tfMPTPaymentMask = ~(tfUniversal | tfPartialPayment);
inline constexpr FlagValue tfTrustSetPermissionMask =
~(tfUniversal | tfSetfAuth | tfSetFreeze | tfClearFreeze);
// MPTokenIssuanceCreate MutableFlags:
// Indicating specific fields or flags may be changed after issuance.
inline constexpr FlagValue tmfMPTCanEnableCanLock = lsmfMPTCanEnableCanLock;
inline constexpr FlagValue tmfMPTCanEnableRequireAuth = lsmfMPTCanEnableRequireAuth;
inline constexpr FlagValue tmfMPTCanEnableCanEscrow = lsmfMPTCanEnableCanEscrow;
inline constexpr FlagValue tmfMPTCanEnableCanTrade = lsmfMPTCanEnableCanTrade;
inline constexpr FlagValue tmfMPTCanEnableCanTransfer = lsmfMPTCanEnableCanTransfer;
inline constexpr FlagValue tmfMPTCanEnableCanClawback = lsmfMPTCanEnableCanClawback;
inline constexpr FlagValue tmfMPTCanMutateMetadata = lsmfMPTCanMutateMetadata;
inline constexpr FlagValue tmfMPTCanMutateTransferFee = lsmfMPTCanMutateTransferFee;
inline constexpr FlagValue tmfMPTCannotEnableCanHoldConfidentialBalance =
lsmfMPTCannotEnableCanHoldConfidentialBalance;
inline constexpr FlagValue tmfMPTokenIssuanceCreateMutableMask =
~(tmfMPTCanEnableCanLock | tmfMPTCanEnableRequireAuth | tmfMPTCanEnableCanEscrow |
tmfMPTCanEnableCanTrade | tmfMPTCanEnableCanTransfer | tmfMPTCanEnableCanClawback |
tmfMPTCanMutateMetadata | tmfMPTCanMutateTransferFee |
tmfMPTCannotEnableCanHoldConfidentialBalance);
// MPTokenIssuanceCreate / MPTokenIssuanceSet ImmutableFlags:
// Defines the immutable fields and flags specific to MPTokenIssuance.
inline constexpr FlagValue tifMPTCanLock = lsifMPTCanLock;
inline constexpr FlagValue tifMPTRequireAuth = lsifMPTRequireAuth;
inline constexpr FlagValue tifMPTCanEscrow = lsifMPTCanEscrow;
inline constexpr FlagValue tifMPTCanTrade = lsifMPTCanTrade;
inline constexpr FlagValue tifMPTCanTransfer = lsifMPTCanTransfer;
inline constexpr FlagValue tifMPTCanClawback = lsifMPTCanClawback;
inline constexpr FlagValue tifMPTMetadata = lsifMPTMetadata;
inline constexpr FlagValue tifMPTTransferFee = lsifMPTTransferFee;
inline constexpr FlagValue tifMPTCanHoldConfidentialBalance = lsifMPTCanHoldConfidentialBalance;
inline constexpr FlagValue tifMPTokenIssuanceImmutableMask =
~(tifMPTCanLock | tifMPTRequireAuth | tifMPTCanEscrow | tifMPTCanTrade | tifMPTCanTransfer |
tifMPTCanClawback | tifMPTMetadata | tifMPTTransferFee | tifMPTCanHoldConfidentialBalance);
// MPTokenIssuanceSet MutableFlags:
// Enable mutable capability flags. These flags are one-way: once enabled,
// the corresponding capability cannot be disabled by MPTokenIssuanceSet.
inline constexpr FlagValue tmfMPTSetCanLock = 0x00000001;
inline constexpr FlagValue tmfMPTSetRequireAuth = 0x00000002;
inline constexpr FlagValue tmfMPTSetCanEscrow = 0x00000004;
inline constexpr FlagValue tmfMPTSetCanTrade = 0x00000008;
inline constexpr FlagValue tmfMPTSetCanTransfer = 0x00000010;
inline constexpr FlagValue tmfMPTSetCanClawback = 0x00000020;
inline constexpr FlagValue tmfMPTSetCanHoldConfidentialBalance = 0x00000040;
inline constexpr FlagValue tmfMPTokenIssuanceSetMutableMask =
~(tmfMPTSetCanLock | tmfMPTSetRequireAuth | tmfMPTSetCanEscrow | tmfMPTSetCanTrade |
tmfMPTSetCanTransfer | tmfMPTSetCanClawback | tmfMPTSetCanHoldConfidentialBalance);
// MPTokenIssuanceSet set of flags that is used to enable capabilities on an MPTokenIssuance.
// Used as `txFlags & tfMPTokenIssuanceSetEnableFlagMask` to extract the capability-enabling bits.
inline constexpr FlagValue tfMPTokenIssuanceSetEnableFlagMask = tfMPTSetCanLock |
tfMPTSetRequireAuth | tfMPTSetCanEscrow | tfMPTSetCanTrade | tfMPTSetCanTransfer |
tfMPTSetCanClawback | tfMPTSetCanHoldConfidentialBalance;
// Prior to fixRemoveNFTokenAutoTrustLine, transfer of an NFToken between accounts allowed a
// TrustLine to be added to the issuer of that token without explicit permission from that issuer.

View File

@@ -15,8 +15,6 @@
// Add new amendments to the top of this list.
// Keep it sorted in reverse chronological order.
XRPL_FEATURE(DefragDirectories, Supported::No, VoteBehavior::DefaultNo)
XRPL_FIX (Cleanup3_4_0, Supported::Yes, VoteBehavior::DefaultNo)
XRPL_FEATURE(Sponsor, Supported::Yes, VoteBehavior::DefaultNo)
XRPL_FEATURE(BatchV1_1, Supported::Yes, VoteBehavior::DefaultNo)
XRPL_FEATURE(LendingProtocolV1_1, Supported::No, VoteBehavior::DefaultNo)

View File

@@ -404,7 +404,7 @@ LEDGER_ENTRY(ltMPTOKEN_ISSUANCE, 0x007e, MPTokenIssuance, mpt_issuance, ({
{sfPreviousTxnID, SoeRequired},
{sfPreviousTxnLgrSeq, SoeRequired},
{sfDomainID, SoeOptional},
{sfMutableFlags, SoeDefault},
{sfImmutableFlags, SoeDefault},
{sfReferenceHolding, SoeOptional},
{sfIssuerEncryptionKey, SoeOptional},
{sfAuditorEncryptionKey, SoeOptional},

View File

@@ -97,7 +97,7 @@ TYPED_SFIELD(sfVoteWeight, UINT32, 48)
TYPED_SFIELD(sfFirstNFTokenSequence, UINT32, 50)
TYPED_SFIELD(sfOracleDocumentID, UINT32, 51)
TYPED_SFIELD(sfPermissionValue, UINT32, 52)
TYPED_SFIELD(sfMutableFlags, UINT32, 53)
TYPED_SFIELD(sfImmutableFlags, UINT32, 53)
TYPED_SFIELD(sfStartDate, UINT32, 54)
TYPED_SFIELD(sfPaymentInterval, UINT32, 55)
TYPED_SFIELD(sfGracePeriod, UINT32, 56)
@@ -238,6 +238,7 @@ TYPED_SFIELD(sfManagementFeeOutstanding, NUMBER, 17, SField::kSmdNeedsAsset
// int32
TYPED_SFIELD(sfLoanScale, INT32, 1)
TYPED_SFIELD(sfRemainingOwnerCountDelta, INT32, 2)
// currency amount (common)
TYPED_SFIELD(sfAmount, AMOUNT, 1)
@@ -277,6 +278,7 @@ TYPED_SFIELD(sfMinAccountCreateAmount, AMOUNT, 30)
TYPED_SFIELD(sfLPTokenBalance, AMOUNT, 31)
TYPED_SFIELD(sfFeeAmount, AMOUNT, 32)
TYPED_SFIELD(sfMaxFee, AMOUNT, 33)
TYPED_SFIELD(sfFeeAmountDelta, AMOUNT, 34)
// variable length (common)
TYPED_SFIELD(sfPublicKey, VL, 1)

View File

@@ -705,7 +705,7 @@ TRANSACTION(ttMPTOKEN_ISSUANCE_CREATE, 54, MPTokenIssuanceCreate,
{sfMaximumAmount, SoeOptional},
{sfMPTokenMetadata, SoeOptional},
{sfDomainID, SoeOptional},
{sfMutableFlags, SoeOptional},
{sfImmutableFlags, SoeOptional},
}))
/** This transaction type destroys a MPTokensIssuance instance */
@@ -734,7 +734,7 @@ TRANSACTION(ttMPTOKEN_ISSUANCE_SET, 56, MPTokenIssuanceSet,
{sfDomainID, SoeOptional},
{sfMPTokenMetadata, SoeOptional},
{sfTransferFee, SoeOptional},
{sfMutableFlags, SoeOptional},
{sfImmutableFlags, SoeOptional},
{sfIssuerEncryptionKey, SoeOptional},
{sfAuditorEncryptionKey, SoeOptional},
}))
@@ -1085,7 +1085,7 @@ TRANSACTION(ttLOAN_PAY, 84, LoanPay,
# include <xrpl/tx/transactors/token/ConfidentialMPTConvert.h>
#endif
TRANSACTION(ttCONFIDENTIAL_MPT_CONVERT, 85, ConfidentialMPTConvert,
Delegation::Delegable,
Delegation::NotDelegable,
featureConfidentialTransfer,
NoPriv,
({
@@ -1189,9 +1189,9 @@ TRANSACTION(ttSPONSORSHIP_SET, 91, SponsorshipSet,
({
{sfCounterpartySponsor, SoeOptional},
{sfSponsee, SoeOptional},
{sfFeeAmount, SoeOptional},
{sfFeeAmountDelta, SoeOptional},
{sfMaxFee, SoeOptional},
{sfRemainingOwnerCount, SoeOptional},
{sfRemainingOwnerCountDelta, SoeOptional},
}))
/** This system-generated transaction type is used to update the status of the various amendments.

View File

@@ -256,27 +256,27 @@ public:
}
/**
* @brief Get sfMutableFlags (SoeDefault)
* @brief Get sfImmutableFlags (SoeDefault)
* @return The field value, or std::nullopt if not present.
*/
[[nodiscard]]
protocol_autogen::Optional<SF_UINT32::type::value_type>
getMutableFlags() const
getImmutableFlags() const
{
if (hasMutableFlags())
return this->sle_->at(sfMutableFlags);
if (hasImmutableFlags())
return this->sle_->at(sfImmutableFlags);
return std::nullopt;
}
/**
* @brief Check if sfMutableFlags is present.
* @brief Check if sfImmutableFlags is present.
* @return True if the field is present, false otherwise.
*/
[[nodiscard]]
bool
hasMutableFlags() const
hasImmutableFlags() const
{
return this->sle_->isFieldPresent(sfMutableFlags);
return this->sle_->isFieldPresent(sfImmutableFlags);
}
/**
@@ -557,13 +557,13 @@ public:
}
/**
* @brief Set sfMutableFlags (SoeDefault)
* @brief Set sfImmutableFlags (SoeDefault)
* @return Reference to this builder for method chaining.
*/
MPTokenIssuanceBuilder&
setMutableFlags(std::decay_t<typename SF_UINT32::type::value_type> const& value)
setImmutableFlags(std::decay_t<typename SF_UINT32::type::value_type> const& value)
{
object_[sfMutableFlags] = value;
object_[sfImmutableFlags] = value;
return *this;
}

View File

@@ -19,7 +19,7 @@ class ConfidentialMPTConvertBuilder;
* @brief Transaction: ConfidentialMPTConvert
*
* Type: ttCONFIDENTIAL_MPT_CONVERT (85)
* Delegable: Delegation::Delegable
* Delegable: Delegation::NotDelegable
* Amendment: featureConfidentialTransfer
* Privileges: NoPriv
*

View File

@@ -178,29 +178,29 @@ public:
}
/**
* @brief Get sfMutableFlags (SoeOptional)
* @brief Get sfImmutableFlags (SoeOptional)
* @return The field value, or std::nullopt if not present.
*/
[[nodiscard]]
protocol_autogen::Optional<SF_UINT32::type::value_type>
getMutableFlags() const
getImmutableFlags() const
{
if (hasMutableFlags())
if (hasImmutableFlags())
{
return this->tx_->at(sfMutableFlags);
return this->tx_->at(sfImmutableFlags);
}
return std::nullopt;
}
/**
* @brief Check if sfMutableFlags is present.
* @brief Check if sfImmutableFlags is present.
* @return True if the field is present, false otherwise.
*/
[[nodiscard]]
bool
hasMutableFlags() const
hasImmutableFlags() const
{
return this->tx_->isFieldPresent(sfMutableFlags);
return this->tx_->isFieldPresent(sfImmutableFlags);
}
};
@@ -302,13 +302,13 @@ public:
}
/**
* @brief Set sfMutableFlags (SoeOptional)
* @brief Set sfImmutableFlags (SoeOptional)
* @return Reference to this builder for method chaining.
*/
MPTokenIssuanceCreateBuilder&
setMutableFlags(std::decay_t<typename SF_UINT32::type::value_type> const& value)
setImmutableFlags(std::decay_t<typename SF_UINT32::type::value_type> const& value)
{
object_[sfMutableFlags] = value;
object_[sfImmutableFlags] = value;
return *this;
}

View File

@@ -163,29 +163,29 @@ public:
}
/**
* @brief Get sfMutableFlags (SoeOptional)
* @brief Get sfImmutableFlags (SoeOptional)
* @return The field value, or std::nullopt if not present.
*/
[[nodiscard]]
protocol_autogen::Optional<SF_UINT32::type::value_type>
getMutableFlags() const
getImmutableFlags() const
{
if (hasMutableFlags())
if (hasImmutableFlags())
{
return this->tx_->at(sfMutableFlags);
return this->tx_->at(sfImmutableFlags);
}
return std::nullopt;
}
/**
* @brief Check if sfMutableFlags is present.
* @brief Check if sfImmutableFlags is present.
* @return True if the field is present, false otherwise.
*/
[[nodiscard]]
bool
hasMutableFlags() const
hasImmutableFlags() const
{
return this->tx_->isFieldPresent(sfMutableFlags);
return this->tx_->isFieldPresent(sfImmutableFlags);
}
/**
@@ -341,13 +341,13 @@ public:
}
/**
* @brief Set sfMutableFlags (SoeOptional)
* @brief Set sfImmutableFlags (SoeOptional)
* @return Reference to this builder for method chaining.
*/
MPTokenIssuanceSetBuilder&
setMutableFlags(std::decay_t<typename SF_UINT32::type::value_type> const& value)
setImmutableFlags(std::decay_t<typename SF_UINT32::type::value_type> const& value)
{
object_[sfMutableFlags] = value;
object_[sfImmutableFlags] = value;
return *this;
}

View File

@@ -100,29 +100,29 @@ public:
}
/**
* @brief Get sfFeeAmount (SoeOptional)
* @brief Get sfFeeAmountDelta (SoeOptional)
* @return The field value, or std::nullopt if not present.
*/
[[nodiscard]]
protocol_autogen::Optional<SF_AMOUNT::type::value_type>
getFeeAmount() const
getFeeAmountDelta() const
{
if (hasFeeAmount())
if (hasFeeAmountDelta())
{
return this->tx_->at(sfFeeAmount);
return this->tx_->at(sfFeeAmountDelta);
}
return std::nullopt;
}
/**
* @brief Check if sfFeeAmount is present.
* @brief Check if sfFeeAmountDelta is present.
* @return True if the field is present, false otherwise.
*/
[[nodiscard]]
bool
hasFeeAmount() const
hasFeeAmountDelta() const
{
return this->tx_->isFieldPresent(sfFeeAmount);
return this->tx_->isFieldPresent(sfFeeAmountDelta);
}
/**
@@ -152,29 +152,29 @@ public:
}
/**
* @brief Get sfRemainingOwnerCount (SoeOptional)
* @brief Get sfRemainingOwnerCountDelta (SoeOptional)
* @return The field value, or std::nullopt if not present.
*/
[[nodiscard]]
protocol_autogen::Optional<SF_UINT32::type::value_type>
getRemainingOwnerCount() const
protocol_autogen::Optional<SF_INT32::type::value_type>
getRemainingOwnerCountDelta() const
{
if (hasRemainingOwnerCount())
if (hasRemainingOwnerCountDelta())
{
return this->tx_->at(sfRemainingOwnerCount);
return this->tx_->at(sfRemainingOwnerCountDelta);
}
return std::nullopt;
}
/**
* @brief Check if sfRemainingOwnerCount is present.
* @brief Check if sfRemainingOwnerCountDelta is present.
* @return True if the field is present, false otherwise.
*/
[[nodiscard]]
bool
hasRemainingOwnerCount() const
hasRemainingOwnerCountDelta() const
{
return this->tx_->isFieldPresent(sfRemainingOwnerCount);
return this->tx_->isFieldPresent(sfRemainingOwnerCountDelta);
}
};
@@ -243,13 +243,13 @@ public:
}
/**
* @brief Set sfFeeAmount (SoeOptional)
* @brief Set sfFeeAmountDelta (SoeOptional)
* @return Reference to this builder for method chaining.
*/
SponsorshipSetBuilder&
setFeeAmount(std::decay_t<typename SF_AMOUNT::type::value_type> const& value)
setFeeAmountDelta(std::decay_t<typename SF_AMOUNT::type::value_type> const& value)
{
object_[sfFeeAmount] = value;
object_[sfFeeAmountDelta] = value;
return *this;
}
@@ -265,13 +265,13 @@ public:
}
/**
* @brief Set sfRemainingOwnerCount (SoeOptional)
* @brief Set sfRemainingOwnerCountDelta (SoeOptional)
* @return Reference to this builder for method chaining.
*/
SponsorshipSetBuilder&
setRemainingOwnerCount(std::decay_t<typename SF_UINT32::type::value_type> const& value)
setRemainingOwnerCountDelta(std::decay_t<typename SF_INT32::type::value_type> const& value)
{
object_[sfRemainingOwnerCount] = value;
object_[sfRemainingOwnerCountDelta] = value;
return *this;
}

View File

@@ -13,6 +13,7 @@ extern Charge const kFeeRequestNoReply; // A request that we cannot satisfy.
extern Charge const kFeeInvalidSignature; // An object whose signature we had to check that failed.
extern Charge const kFeeUselessData; // Data we have no use for.
extern Charge const kFeeInvalidData; // Data we have to verify before rejecting.
extern Charge const kFeeMalformedData; // Data that no honest peer would send.
// RPC loads
extern Charge const kFeeMalformedRpc; // An RPC request that we can immediately tell is invalid.

View File

@@ -11,6 +11,7 @@
#include <xrpl/server/Manifest.h>
#include <atomic>
#include <cstddef>
#include <cstdint>
#include <functional>
#include <memory>
@@ -22,6 +23,39 @@ namespace xrpl {
// Operations that clients may wish to perform against the network
// Master operational handler, server sequencer, network tracker
/**
* Maximum number of subscriptions a single client connection may hold at once.
*
* Applies to the account, real-time account, and account-history subscriptions
* tracked on one InfoSub (the sets counted by totalSubscriptionCount), bounding
* the disconnect-time cleanup of those sets. Book subscriptions are tracked
* separately (OrderBookDB) and are not counted here. Generous enough for
* legitimate power users such as block explorers.
*/
constexpr std::size_t kMaxSubscriptionsPerConnection = 100'000;
/**
* Whether adding @p additional subscriptions to a connection already holding
* @p current would exceed the cap.
*
* Pure arithmetic split out so it can be unit-tested without a live
* connection. The first term avoids underflow in the subtraction.
*
* @param current Subscriptions already tracked on the connection.
* @param additional Subscriptions a request would add.
* @param cap The effective per-connection cap. Defaults to the
* built-in limit; callers may pass a configured override.
* @return true if the request must be rejected to stay within the cap.
*/
[[nodiscard]] constexpr bool
exceedsSubscriptionCap(
std::size_t current,
std::size_t additional,
std::size_t cap = kMaxSubscriptionsPerConnection)
{
return additional > cap || current > cap - additional;
}
class InfoSubRequest : public CountedObject<InfoSubRequest>
{
public:
@@ -44,12 +78,12 @@ public:
* map.
*
* @note Lifetime contract: every `InfoSub` instance MUST be destroyed
* before the backing `Source`. NetworkOPsImp shutdown drops all
* subscriber strong refs before its own teardown to satisfy this.
* before the backing `Source`. NetworkOPsImp shutdown drops all
* subscriber strong refs before its own teardown to satisfy this.
* @note Thread-safety: per-instance state is guarded by `lock_`. The
* destructor reads tracking sets without taking `lock_` because
* the strong-pointer ref-count is zero at destruction time, so
* no other thread can be calling the public mutators.
* destructor reads tracking sets without taking `lock_` because
* the strong-pointer ref-count is zero at destruction time, so
* no other thread can be calling the public mutators.
*/
class InfoSub : public CountedObject<InfoSub>
{
@@ -117,6 +151,34 @@ public:
AccountID const& account,
bool historyOnly) = 0;
/**
* Schedule the server-side teardown of a disconnecting connection's
* account subscriptions off the destructor thread.
*
* The implementation posts a low-priority JobQueue task that erases the
* entries in bounded chunks, so `~InfoSub` returns immediately instead
* of running the erase loop inline. The sets are taken by value so the
* job owns its copies and never references the destroyed `InfoSub`.
* Cleanup is keyed on `seq` (unique per connection), so deferring it
* cannot disturb a reconnected client reusing the same accounts.
*
* @param seq The disconnecting connection's unique subscription id.
* @param rtAccounts Real-time account subscriptions to remove.
* @param normalAccounts Normal account subscriptions to remove.
* @param historyAccounts Account-history subscriptions to remove.
*
* @note The implementing `Source` must outlive any job it posts. If the
* JobQueue is already stopping (process shutdown), the job is not
* enqueued; the cleanup is skipped because the server-side maps
* are about to be destroyed and no publishing can run.
*/
virtual void
scheduleAccountCleanup(
std::uint64_t seq,
hash_set<AccountID> rtAccounts,
hash_set<AccountID> normalAccounts,
hash_set<AccountID> historyAccounts) = 0;
// VFALCO TODO Document the bool return value
virtual bool
subLedger(ref ispListener, json::Value& jvResult) = 0;
@@ -153,12 +215,12 @@ public:
* @param ispListener The subscriber requesting removal.
* @param book The order book to unsubscribe from.
* @return true if the entry was present and removed, false if the
* subscriber was not subscribed to @p book.
* subscriber was not subscribed to @p book.
*
* @note Thread-safety: acquires subLock_ internally.
* @note Thread-safety: acquires bookLock_ internally.
* @note Do NOT call from ~InfoSub(). Use unsubBookInternal instead
* to avoid a redundant write-back to bookSubscriptions_ on a
* partially-destroyed object.
* to avoid a redundant write-back to bookSubscriptions_ on a
* partially-destroyed object.
*/
virtual bool
unsubBook(ref ispListener, Book const&) = 0;
@@ -173,9 +235,9 @@ public:
* @param uListener The sequence number of the subscriber being torn down.
* @param book The order book entry to remove.
* @return true if the entry was present and removed, false otherwise
* (e.g., already removed by a concurrent RPC unsubscribe).
* (e.g., already removed by a concurrent RPC unsubscribe).
*
* @note Thread-safety: acquires subLock_ internally.
* @note Thread-safety: acquires bookLock_ internally.
*/
virtual bool
unsubBookInternal(std::uint64_t uListener, Book const&) = 0;
@@ -221,8 +283,8 @@ public:
/**
* Journal used by InfoSub for diagnostics that occur after the
* owning subsystem (e.g. application-level Logs) is the only
* surviving sink — primarily destructor-time cleanup failures.
* owning subsystem (e.g. application-level Logs) is the only
* surviving sink — primarily destructor-time cleanup failures.
*/
[[nodiscard]] virtual beast::Journal const&
journal() const = 0;
@@ -243,6 +305,56 @@ public:
[[nodiscard]] std::uint64_t
getSeq() const;
/**
* Return the number of subscriptions currently tracked on this
* connection.
*
* The combined size of the per-connection account, real-time account, and
* account-history subscription sets. `doSubscribe` reads this to enforce
* the per-connection subscription cap before admitting more.
*
* @return The total tracked subscription count for this connection.
*
* @note Thread-safe: takes `lock_` for the read; read-only.
*/
[[nodiscard]] std::size_t
totalSubscriptionCount() const;
/**
* Enforce the cap and reserve a request's net-new accounts, atomically.
*
* Under one hold of `lock_`: count the net-new entries in the two sets,
* check the total against @p cap, and insert them only if it fits.
* All-or-nothing. Doing check and insert together stops two concurrent
* requests sharing an InfoSub (the admin subscribe-by-url path) from both
* passing the check before either records its accounts. The server-side
* maps are populated afterwards by subAccount, whose re-insert is a no-op.
*
* @param proposedAccounts Real-time (accounts_proposed) ids to reserve.
* @param normalAccounts Normal (accounts) ids to reserve.
* @param cap The effective per-connection cap.
* @return true if reserved; false if the request must be rejected.
* @note Thread-safe: takes `lock_`.
*/
[[nodiscard]] bool
tryReserveAccountSubscriptions(
hash_set<AccountID> const& proposedAccounts,
hash_set<AccountID> const& normalAccounts,
std::size_t cap);
/**
* Whether this connection already tracks an account-history for @p account.
*
* `doSubscribe` reads this to charge the cap for an account_history_tx_stream
* only when it is net-new, matching the account branches.
*
* @param account The account an account_history_tx_stream would add.
* @return true if @p account is already in the account-history set.
* @note Thread-safe: takes `lock_`; read-only.
*/
[[nodiscard]] bool
hasAccountHistorySubscription(AccountID const& account) const;
void
onSendEmpty();
@@ -302,7 +414,9 @@ public:
getApiVersion() const noexcept;
protected:
std::mutex lock_;
// Mutable so the read-only totalSubscriptionCount() accessor can lock it
// from a const method; locking semantics are otherwise unchanged.
mutable std::mutex lock_;
private:
Consumer consumer_;

View File

@@ -3,12 +3,14 @@
#include <xrpl/basics/Blob.h>
#include <xrpl/basics/Slice.h>
#include <xrpl/basics/UnorderedContainers.h>
#include <xrpl/basics/base64.h>
#include <xrpl/basics/base_uint.h>
#include <xrpl/beast/utility/Journal.h>
#include <xrpl/protocol/PublicKey.h>
#include <xrpl/protocol/SecretKey.h>
#include <atomic>
#include <cstddef>
#include <cstdint>
#include <functional>
#include <optional>
@@ -43,12 +45,15 @@ namespace xrpl {
dynamically generates the signatureless form when it needs to verify
the signature.
An instance of ManifestCache stores, for each trusted validator, (a) its
An instance of ManifestCache stores, for each known validator, (a) its
master public key, and (b) the most senior of all valid manifests it has
seen for that validator, if any. On startup, the [validator_token] config
entry (which contains the manifest for this validator) is decoded and
added to the manifest cache. Other manifests are added as "gossip"
received from xrpld peers.
received from xrpld peers, including ones for validators this node does not
trust. Manifests for untrusted validators are capped (kMaxUntrustedCount)
so peer gossip cannot grow the cache without bound; trusted validators are
not capped. Entries are never evicted, so a stored revocation is permanent.
When an ephemeral key is compromised, a new signing key pair is created,
along with a new manifest vouching for it (with a higher sequence number),
@@ -164,6 +169,100 @@ struct Manifest
std::string
to_string(Manifest const& m);
/**
* Largest a valid manifest can be, in decoded bytes.
*
* A manifest has a fixed set of fields. Each is serialized as a field header
* (1-2 bytes), an optional length prefix (1 byte for these sizes), and the
* field body. Taking every field at its largest gives the maximum below, so
* anything larger cannot be a valid manifest.
*
* Field header + length + body = bytes
* sfVersion (U16) 2 0 2 4
* sfSequence (U32) 1 0 4 5
* sfPublicKey (33) 1 1 33 35
* sfSigningPubKey (33) 1 1 33 35
* sfSignature (72) 1 1 72 74
* sfMasterSignature (72) 2 1 72 75
* sfDomain (128) 1 1 128 130
* -----
* 358
*/
constexpr std::size_t kMaxManifestBytes = 358;
/**
* Largest a valid manifest can be, in base64 characters.
*
* base64 encodes 3 bytes as 4 characters, so this is the encoded form of
* @ref kMaxManifestBytes. Callers that receive a base64 manifest should
* reject anything longer than this before decoding, to avoid allocating
* memory for an oversized input.
*/
constexpr std::size_t kMaxManifestBase64 = base64::encodedSize(kMaxManifestBytes);
/**
* Default number of untrusted manifests to store in cache and allowed
* in one Manifest message.
*
* Bounds unlisted validators two ways. In the cache, a manifest for a
* brand-new unlisted key is rejected once this many are held, so peer gossip
* cannot grow the cache without end. In a TMManifests message, this many are
* sent and processed, so a peer sending its whole cache cannot force unbounded
* work.
*
* Operators can override this with `[overlay] max_untrusted_count`. Both users
* read the configured value and fall back to this default.
*/
constexpr std::size_t kMaxUntrustedCount = 300;
/**
* Default number of trusted manifests allowed in a Manifest message.
* Not used atm while creating the message, but used to calculate the higher limit on
* received message size. Introduced to maintain consistency. Future implementation
* will use this limit.
*
* Trusted manifests are never dropped: every one this node holds is sent, and
* every one received is processed, since dropping one would delay a validator
* key rotation. This count only sizes the largest message accepted, so it must
* stay above any realistic validator list. Cap can be increased in the config
* file if messages get rejected with actual trusted manifest count crossing
* configured(or else default) value.
* Operators can override this with `[overlay] max_trusted_count`.
*/
constexpr std::size_t kMaxTrustedCount = 300;
/**
* Number of untrusted manifests to store in cache and allowed
* in one Manifest message..
*
* Returns the operator's override when one is configured, otherwise
* @ref kMaxUntrustedCount. Config stores an override rather than the default
* itself because the core module cannot depend on this module.
*
* @param configured The value from `[overlay] max_untrusted_count`, or
* `std::nullopt` when the operator did not set it.
*/
constexpr std::size_t
untrustedManifestCount(std::optional<std::size_t> const& configured)
{
return configured.value_or(kMaxUntrustedCount);
}
/**
* Number of trusted manifests allowed in a Manifest message.
*
* Not a cap on how many are sent or processed; see @ref kMaxTrustedCount.
* but used to calculate the higher limit on received message size.
*
* @param configured The value from `[overlay] max_trusted_count`, or
* `std::nullopt` when the operator did not set it.
*/
constexpr std::size_t
trustedManifestCount(std::optional<std::size_t> const& configured)
{
return configured.value_or(kMaxTrustedCount);
}
/**
* Constructs Manifest from serialized string
*
@@ -172,7 +271,7 @@ to_string(Manifest const& m);
* @return `std::nullopt` if string is invalid
*
* @note This does not verify manifest signatures.
* `Manifest::verify` should be called after constructing manifest.
* `Manifest::verify` should be called after constructing manifest.
*/
/** @{ */
std::optional<Manifest>
@@ -225,30 +324,17 @@ loadValidatorToken(
beast::Journal journal = beast::Journal(beast::Journal::getNullSink()));
enum class ManifestDisposition {
/**
* Manifest is valid
*/
Accepted = 0,
Accepted = 0, ///< Manifest is valid
/**
* Sequence is too old
*/
Stale,
Stale, ///< Sequence is too old
/**
* The master key is not acceptable to us
*/
BadMasterKey,
BadMasterKey, ///< The master key is not acceptable to us
/**
* The ephemeral key is not acceptable to us
*/
BadEphemeralKey,
BadEphemeralKey, ///< The ephemeral key is not acceptable to us
/**
* Timely, but invalid signature
*/
Invalid
Invalid, ///< Timely, but invalid signature
UntrustedCapacity ///< Unlisted and limit reached
};
inline std::string
@@ -266,11 +352,25 @@ to_string(ManifestDisposition m)
return "badEphemeralKey";
case ManifestDisposition::Invalid:
return "invalid";
case ManifestDisposition::UntrustedCapacity:
return "untrustedCapacity";
default:
return "unknown";
}
}
/**
* Whether a manifest counts against the 'untrusted' cache cap.
*
* Passed to `ManifestCache::applyManifest` with no default, so every caller
* must choose. `Capped` is the safe, flood-resistant value; only listed or
* configured keys should use `Uncapped`.
*/
enum class ManifestRateLimitCapPolicy : std::uint8_t {
Capped, ///< Subject to the untrusted cap (unlisted peer gossip)
Uncapped ///< Bypasses the cap (listed/trusted or config manifests)
};
class DatabaseCon;
/**
@@ -294,8 +394,51 @@ private:
std::atomic<std::uint32_t> seq_{0};
/**
* Master keys of cached manifests for validators this node does not list.
*
* One entry per capped key in `map_`; its size enforces the cap below.
* A key is added when first cached under `Capped` and removed when it
* becomes listed (see `promoteToTrusted`) or an `Uncapped` update arrives,
* never re-added on de-listing. Uncapped keys are not tracked here.
*/
hash_set<PublicKey> untrustedKeys_;
/**
* Maximum number of untrusted master keys kept in the cache.
*
* Once reached, a manifest for a brand-new unlisted key is rejected. Set
* from the config, defaulting to @ref kMaxUntrustedCount.
*/
std::size_t const maxUntrustedCount_;
/**
* Running count of manifests rejected because the untrusted cap was full.
*
* Drives throttled logging (see `kUntrustedRejectCount`). Atomic because
* `applyManifest` may run concurrently.
*/
std::atomic<std::uint64_t> untrustedRejectCount_{0};
/**
* Number of cap rejections between summary warnings.
*
* @see untrustedRejectCount_
*/
static constexpr std::uint64_t kUntrustedRejectCount = 10000;
public:
explicit ManifestCache(beast::Journal j = beast::Journal(beast::Journal::getNullSink())) : j_(j)
/**
* @param j Journal for logging.
*
* @param maxUntrustedCount Untrusted master keys to keep. Pass the
* configured value; defaults to @ref kMaxUntrustedCount. Taken as a
* parameter because this module cannot depend on the config.
*/
explicit ManifestCache(
beast::Journal j = beast::Journal(beast::Journal::getNullSink()),
std::size_t maxUntrustedCount = kMaxUntrustedCount)
: j_(j), maxUntrustedCount_(maxUntrustedCount)
{
}
@@ -378,17 +521,44 @@ public:
/**
* Add manifest to cache.
*
* A brand-new unlisted key is rejected once the untrusted cap is full;
* updates to a cached key and `Uncapped` manifests bypass the cap. The
* caller decides `cap` before calling so the cache lock is not held while
* consulting the validator list, which would risk a lock-ordering deadlock.
*
* @param m Manifest to add
*
* @return `ManifestDisposition::accepted` if successful, or
* `stale` or `invalid` otherwise
* @param cap `Uncapped` skips the untrusted cap; use it for keys that are
* listed, configured, or loaded from the DB. Note `Uncapped` does not
* assert the key is currently trusted (a DB entry may predate a
* de-listing). Callers must state this explicitly so a manifest is
* never left uncapped by omission.
*
* @return `Accepted` if stored, `Stale` if superseded, `Invalid`/
* `BadEphemeralKey` if malformed, or `UntrustedCapacity` if the
* untrusted cap is full.
*
* @par Thread Safety
*
* May be called concurrently
*/
ManifestDisposition
applyManifest(Manifest m);
applyManifest(Manifest m, ManifestRateLimitCapPolicy cap);
/**
* Stop counting a master key against the untrusted cap.
*
* Called when a cached untrusted key becomes listed, freeing its slot.
* Idempotent and a no-op for keys that were never counted.
*
* @param pk Master public key that is now listed/trusted
*
* @par Thread Safety
*
* May be called concurrently
*/
void
promoteToTrusted(PublicKey const& pk);
/**
* Populate manifest cache with manifests in database and config.

View File

@@ -17,8 +17,7 @@ class ValidPermissionedDEX
bool regularOffers_ = false; // post-fixCleanup3_2_0: excludes deleted offers
bool badHybridsOld_ = false; // pre-fixCleanup3_1_3: missing field/domain or size > 1
bool badHybrids_ = false; // post-fixCleanup3_1_3: also catches size == 0 (size != 1)
hash_set<uint256> domainsOld_; // pre-fixCleanup3_4_0: also flags deleted domains
hash_set<uint256> domains_; // post-fixCleanup3_4_0: excludes deleted domains
hash_set<uint256> domains_;
public:
void

View File

@@ -2,6 +2,8 @@
#include <xrpl/beast/utility/Journal.h>
#include <xrpl/ledger/ReadView.h>
#include <xrpl/protocol/AccountID.h>
#include <xrpl/protocol/Keylet.h>
#include <xrpl/protocol/STLedgerEntry.h>
#include <xrpl/protocol/STTx.h>
#include <xrpl/protocol/TER.h>
@@ -16,7 +18,7 @@ namespace xrpl {
class SponsorshipSet : public Transactor
{
public:
static constexpr auto kConsequencesFactory = ConsequencesFactoryType::Normal;
static constexpr auto kConsequencesFactory = ConsequencesFactoryType::Custom;
explicit SponsorshipSet(ApplyContext& ctx) : Transactor(ctx)
{
@@ -47,6 +49,15 @@ public:
XRPAmount fee,
ReadView const& view,
beast::Journal const& j) override;
private:
TER
createSponsorship(
Keylet const& sponsorshipKeylet,
AccountID const& sponsorID,
AccountID const& sponseeID,
SLE::ref sponsorAccSle,
SLE::ref reserveSponsorAccSle);
};
} // namespace xrpl

View File

@@ -32,7 +32,7 @@ struct MPTCreateArgs
std::optional<std::uint16_t> transferFee = std::nullopt;
std::optional<Slice> const& metadata{};
std::optional<uint256> domainId = std::nullopt;
std::optional<std::uint32_t> mutableFlags = std::nullopt;
std::optional<std::uint32_t> immutableFlags = std::nullopt;
// Set only by callers that issue an MPT representing a wrapped asset
// (e.g. VaultCreate's share token). The keylet must point to an
// existing MPToken or RippleState owned by `account`. Surfaces on

View File

@@ -3,12 +3,15 @@
#include <xrpl/beast/utility/Journal.h>
#include <xrpl/core/ServiceRegistry.h>
#include <xrpl/ledger/ReadView.h>
#include <xrpl/protocol/LedgerFormats.h>
#include <xrpl/protocol/STTx.h>
#include <xrpl/protocol/TER.h>
#include <xrpl/protocol/TxFlags.h>
#include <xrpl/protocol/XRPAmount.h>
#include <xrpl/tx/ApplyContext.h>
#include <xrpl/tx/Transactor.h>
#include <array>
#include <cstdint>
namespace xrpl {
@@ -22,6 +25,37 @@ public:
{
}
// Maps each MPTokenIssuanceSet set flag(e.g., tfMPTSetCanLock), to the issuance's
// corresponding immutable flag (e.g., lsifMPTCanLock) and the target ledger flag (e.g.,
// lsfMPTCanLock).
struct FlagMapping
{
std::uint32_t setFlag;
std::uint32_t immutableFlag;
std::uint32_t ledgerFlag;
};
static constexpr std::array<FlagMapping, 7> flagMapping = {
{{.setFlag = tfMPTSetCanLock, .immutableFlag = lsifMPTCanLock, .ledgerFlag = lsfMPTCanLock},
{.setFlag = tfMPTSetRequireAuth,
.immutableFlag = lsifMPTRequireAuth,
.ledgerFlag = lsfMPTRequireAuth},
{.setFlag = tfMPTSetCanEscrow,
.immutableFlag = lsifMPTCanEscrow,
.ledgerFlag = lsfMPTCanEscrow},
{.setFlag = tfMPTSetCanTrade,
.immutableFlag = lsifMPTCanTrade,
.ledgerFlag = lsfMPTCanTrade},
{.setFlag = tfMPTSetCanTransfer,
.immutableFlag = lsifMPTCanTransfer,
.ledgerFlag = lsfMPTCanTransfer},
{.setFlag = tfMPTSetCanClawback,
.immutableFlag = lsifMPTCanClawback,
.ledgerFlag = lsfMPTCanClawback},
{.setFlag = tfMPTSetCanHoldConfidentialBalance,
.immutableFlag = lsifMPTCanHoldConfidentialBalance,
.ledgerFlag = lsfMPTCanHoldConfidentialBalance}}};
static bool
checkExtraFeatures(PreflightContext const& ctx);

View File

@@ -6,18 +6,106 @@
let
inherit (import ./packages.nix { inherit pkgs; })
commonPackages
gccVersion
gccPackage
llvmPackages
llvmVersion
mkVersionedToolLinks
;
# Custom-glibc toolchain, shared with the Linux dev shell (see compilers.nix).
inherit (import ./compilers.nix { inherit pkgs customGlibc; })
customGcc
customClang
customBinutils
customGcov
;
# Underlying compiler toolchains to wrap (versions pinned in packages.nix).
customGccPackage = gccPackage;
customLlvmPackages = llvmPackages;
# binutils wrapped to emit binaries that reference the custom glibc
# (dynamic linker path, library search path, RPATH).
customBinutils = pkgs.wrapBintoolsWith {
bintools = pkgs.binutils-unwrapped;
libc = customGlibc;
};
# Rebuild gcc (specifically libstdc++ / libgcc_s) against the custom
# glibc. The override swaps gcc.cc's bootstrap stdenv for one that uses
# the existing gcc binary but links against the custom glibc, so the
# resulting compiler ships runtime libraries that only reference symbols
# available in that glibc.
customGccCc = customGccPackage.cc.override {
stdenv = pkgs.stdenvAdapters.overrideCC pkgs.stdenv (
pkgs.wrapCCWith {
cc = customGccPackage.cc;
libc = customGlibc;
bintools = customBinutils;
}
);
};
# cc-wrapper around the rebuilt compiler, pointing at the custom glibc
# headers and libraries. This is what we actually expose to users.
customGcc = pkgs.wrapCCWith {
cc = customGccCc;
libc = customGlibc;
bintools = customBinutils;
};
# gcov ships in gcc's `cc` output, but the cc-wrapper doesn't expose it.
# Surface the gcov from our rebuilt gcc (linked against the custom glibc, so
# it runs under the loader installed in the image) and matching the exact
# compiler version, so gcovr can produce coverage reports in the CI env.
customGcov = pkgs.runCommand "gcov-custom-for-ci-env" { } ''
mkdir -p "$out/bin"
ln -s "${customGccCc}/bin/gcov" "$out/bin/gcov"
'';
# stdenv built around the rebuilt gcc / custom glibc. Used to rebuild
# compiler-rt below so its sanitizer runtimes see the custom glibc
# headers.
customStdenv = pkgs.stdenvAdapters.overrideCC pkgs.stdenv customGcc;
# Rebuild compiler-rt against the custom glibc so the sanitizer runtimes
# don't use glibc symbols (or sysconf constants like _SC_SIGSTKSZ) that
# only exist in newer glibc versions. scudo is dropped because its CMake
# includes CheckAtomic with -nostdinc++ in CMAKE_REQUIRED_FLAGS, which
# makes std::atomic unfindable in our stdenv; we don't use scudo (only
# asan/ubsan/tsan etc.).
customCompilerRt =
(customLlvmPackages.compiler-rt.override {
stdenv = customStdenv;
}).overrideAttrs
(old: {
postPatch = (old.postPatch or "") + ''
substituteInPlace lib/CMakeLists.txt \
--replace-quiet 'add_subdirectory(scudo/standalone)' \
'# scudo/standalone disabled in xrpld ci-env'
'';
});
# cc-wrapper around clang, pointing at the custom glibc headers and
# libraries. Reuses the rebuilt gcc for libstdc++ / libgcc_s so that
# C++ binaries produced by clang also only reference symbols available
# in the custom glibc. compiler-rt is wired into a resource-root so
# sanitizer runtimes (libclang_rt.*.a) are found at link time; this
# mirrors what nixpkgs does internally when building llvmPackages.clang.
customClang = pkgs.wrapCCWith {
cc = customLlvmPackages.clang-unwrapped;
libc = customGlibc;
bintools = customBinutils;
gccForLibs = customGccCc;
extraPackages = [ customCompilerRt ];
extraBuildCommands = ''
rsrc="$out/resource-root"
mkdir "$rsrc"
ln -s "${customLlvmPackages.clang-unwrapped.lib}/lib/clang/${toString llvmVersion}/include" "$rsrc/include"
ln -s "${customCompilerRt.out}/lib" "$rsrc/lib"
ln -s "${customCompilerRt.out}/share" "$rsrc/share" || true
echo "-resource-dir=$rsrc" >> $out/nix-support/cc-cflags
# compiler-rt ships the sanitizer/profile/xray interface headers (e.g.
# <sanitizer/lsan_interface.h>) in its `dev` output. In a normal Nix
# build these reach the include path because compiler-rt is propagated
# via depsTargetTargetPropagated and stdenv's setup hooks add its
# dev/include. The CI image runs clang outside a Nix stdenv (binaries
# on PATH, no setup hooks), so that never happens; add the headers
# explicitly. gcc ships its own copy, which is why this is clang-only.
echo "-isystem ${customCompilerRt.dev}/include" >> $out/nix-support/cc-cflags
'';
};
# Strip the generic cc/c++/cpp symlinks from the clang wrapper so it can
# coexist with the gcc wrapper in buildEnv. gcc remains the default
@@ -39,25 +127,6 @@ in
customGcov
customClangForCiEnv
customBinutils
(mkVersionedToolLinks {
name = "gcc";
package = customGcc;
version = gccVersion;
tools = [
"gcc"
"g++"
"cpp"
];
})
(mkVersionedToolLinks {
name = "clang";
package = customClang;
version = llvmVersion;
tools = [
"clang"
"clang++"
];
})
# CA certificate bundle so HTTPS clients (git, curl, conan) can verify
# TLS connections without ca-certificates being installed in the system.
pkgs.cacert

View File

@@ -1,117 +0,0 @@
# Custom-glibc compiler toolchain shared by the CI environment (ci-env.nix) and
# the Linux dev shell (devshell.nix): gcc / clang / binutils rebuilt to target
# the pinned custom glibc. Linux only — the pinned glibc snapshot does not build
# on darwin, so callers must not evaluate this on macOS.
{
pkgs,
customGlibc,
}:
let
inherit (import ./packages.nix { inherit pkgs; })
gccPackage
llvmPackages
llvmVersion
mkGcov
;
# binutils wrapped to emit binaries that reference the custom glibc
# (dynamic linker path, library search path, RPATH).
customBinutils = pkgs.wrapBintoolsWith {
bintools = pkgs.binutils-unwrapped;
libc = customGlibc;
};
# Rebuild gcc (specifically libstdc++ / libgcc_s) against the custom
# glibc. The override swaps gcc.cc's bootstrap stdenv for one that uses
# the existing gcc binary but links against the custom glibc, so the
# resulting compiler ships runtime libraries that only reference symbols
# available in that glibc.
customGccCc = gccPackage.cc.override {
stdenv = pkgs.stdenvAdapters.overrideCC pkgs.stdenv (
pkgs.wrapCCWith {
cc = gccPackage.cc;
libc = customGlibc;
bintools = customBinutils;
}
);
};
# cc-wrapper around the rebuilt compiler, pointing at the custom glibc
# headers and libraries. This is what we actually expose to users.
customGcc = pkgs.wrapCCWith {
cc = customGccCc;
libc = customGlibc;
bintools = customBinutils;
};
# gcov matching the rebuilt gcc (linked against the custom glibc), so gcovr
# can produce coverage reports both in CI and in the dev shell.
customGcov = mkGcov {
name = "custom";
cc = customGccCc;
};
# stdenv built around the rebuilt gcc / custom glibc. Exported as the dev
# shell's gcc stdenv, and used below to rebuild compiler-rt so its sanitizer
# runtimes see the custom glibc headers.
customStdenv = pkgs.stdenvAdapters.overrideCC pkgs.stdenv customGcc;
# Rebuild compiler-rt against the custom glibc so the sanitizer runtimes
# don't use glibc symbols (or sysconf constants like _SC_SIGSTKSZ) that
# only exist in newer glibc versions. scudo is dropped because its CMake
# includes CheckAtomic with -nostdinc++ in CMAKE_REQUIRED_FLAGS, which
# makes std::atomic unfindable in our stdenv; we don't use scudo (only
# asan/ubsan/tsan etc.).
customCompilerRt =
(llvmPackages.compiler-rt.override {
stdenv = customStdenv;
}).overrideAttrs
(old: {
postPatch = (old.postPatch or "") + ''
substituteInPlace lib/CMakeLists.txt \
--replace-quiet 'add_subdirectory(scudo/standalone)' \
'# scudo/standalone disabled in xrpld ci-env'
'';
});
# cc-wrapper around clang, pointing at the custom glibc headers and
# libraries. Reuses the rebuilt gcc for libstdc++ / libgcc_s so that
# C++ binaries produced by clang also only reference symbols available
# in the custom glibc. compiler-rt is wired into a resource-root so
# sanitizer runtimes (libclang_rt.*.a) are found at link time; this
# mirrors what nixpkgs does internally when building llvmPackages.clang.
customClang = pkgs.wrapCCWith {
cc = llvmPackages.clang-unwrapped;
libc = customGlibc;
bintools = customBinutils;
gccForLibs = customGccCc;
extraPackages = [ customCompilerRt ];
extraBuildCommands = ''
rsrc="$out/resource-root"
mkdir "$rsrc"
ln -s "${llvmPackages.clang-unwrapped.lib}/lib/clang/${toString llvmVersion}/include" "$rsrc/include"
ln -s "${customCompilerRt.out}/lib" "$rsrc/lib"
ln -s "${customCompilerRt.out}/share" "$rsrc/share" || true
echo "-resource-dir=$rsrc" >> $out/nix-support/cc-cflags
# compiler-rt ships the sanitizer/profile/xray interface headers (e.g.
# <sanitizer/lsan_interface.h>) in its `dev` output. In a normal Nix
# build these reach the include path because compiler-rt is propagated
# via depsTargetTargetPropagated and stdenv's setup hooks add its
# dev/include. The CI image runs clang outside a Nix stdenv (binaries
# on PATH, no setup hooks), so that never happens; add the headers
# explicitly. gcc ships its own copy, which is why this is clang-only.
echo "-isystem ${customCompilerRt.dev}/include" >> $out/nix-support/cc-cflags
'';
};
in
{
inherit
customGcc
customClang
customBinutils
customStdenv
customGcov
;
customClangStdenv = pkgs.stdenvAdapters.overrideCC pkgs.stdenv customClang;
}

View File

@@ -1,64 +1,23 @@
{ pkgs, customGlibc, ... }:
{ pkgs, ... }:
let
inherit (import ./packages.nix { inherit pkgs; })
commonPackages
gccPackage
gccVersion
llvmVersion
llvmPackages
mkVersionedToolLinks
mkGcov
;
# Plain nixpkgs stdenvs — no custom glibc.
plainGccStdenv = pkgs."gcc${toString gccVersion}Stdenv";
plainClangStdenv = llvmPackages.stdenv;
# Custom-glibc stdenvs, matching the CI environment (see compilers.nix). The
# pinned glibc snapshot only builds on Linux, so on darwin these fall back to
# the plain stdenvs; the `if isLinux` guard keeps `customGlibc` from being
# forced (and erroring) on macOS.
customCompilers = import ./compilers.nix { inherit pkgs customGlibc; };
customGccStdenv = if pkgs.stdenv.isLinux then customCompilers.customStdenv else plainGccStdenv;
customClangStdenv =
if pkgs.stdenv.isLinux then customCompilers.customClangStdenv else plainClangStdenv;
# gcov matching each gcc shell, so `-Dcoverage=ON` builds work in the shell.
plainGcov = mkGcov {
name = "plain";
cc = gccPackage.cc;
};
customGccGcov = if pkgs.stdenv.isLinux then customCompilers.customGcov else plainGcov;
# Shown when entering a *-plain shell. These exist only on Linux (see below),
# where the stock toolchain diverges from CI.
plainWarningHook = ''
echo " WARNING: this is the stock nixpkgs toolchain and does not match CI's glibc. Prefer 'nix develop .#gcc' / '.#clang' unless you need to skip the custom-glibc build."
'';
# Tools to expose under version-suffixed names (see mkVersionedToolLinks).
gccVersionedTools = [
"gcc"
"g++"
"cpp"
];
clangVersionedTools = [
"clang"
"clang++"
];
# Plain nixpkgs stdenvs — no custom glibc, unlike ci-env.nix.
gccStdenv = pkgs."gcc${toString gccVersion}Stdenv";
clangStdenv = llvmPackages.stdenv;
# compilerName is the command used to print the version, or null for none.
makeShell =
{
stdenv,
compilerName,
version ? null,
versionedTools ? [ ],
extraPackages ? [ ],
warningHook ? "",
}:
let
compilerVersionHook =
compilerVersion =
if compilerName == null then
''echo "No compiler specified - using system compiler"''
else
@@ -66,19 +25,12 @@ let
echo "Compiler: "
${compilerName} --version
'';
versionedLinks = pkgs.lib.optional (version != null) (mkVersionedToolLinks {
name = compilerName;
package = stdenv.cc;
inherit version;
tools = versionedTools;
});
in
(pkgs.mkShell.override { inherit stdenv; }) {
packages = commonPackages ++ versionedLinks ++ extraPackages;
packages = commonPackages;
shellHook = ''
echo "Welcome to xrpld development shell";
${compilerVersionHook}
${warningHook}
${compilerVersion}
'';
};
in
@@ -86,21 +38,14 @@ rec {
# macOS: Nix Clang. Linux: Nix GCC.
default = if pkgs.stdenv.isDarwin then clang else gcc;
# gcc/clang use the custom-glibc toolchain, matching CI. On darwin there is no
# custom glibc, so they fall back to the plain nixpkgs toolchain.
gcc = makeShell {
stdenv = customGccStdenv;
stdenv = gccStdenv;
compilerName = "gcc";
version = gccVersion;
versionedTools = gccVersionedTools;
extraPackages = [ customGccGcov ];
};
clang = makeShell {
stdenv = customClangStdenv;
stdenv = clangStdenv;
compilerName = "clang";
version = llvmVersion;
versionedTools = clangVersionedTools;
};
# Nix provides no compiler; use the one from your system (e.g. Apple Clang).
@@ -110,24 +55,3 @@ rec {
};
apple-clang = no-compiler;
}
# The *-plain shells (stock nixpkgs toolchain) exist only on Linux: on darwin
# gcc/clang are already plain, so these would be redundant and are omitted, which
# makes `nix develop .#gcc-plain` fail there rather than silently aliasing gcc.
// pkgs.lib.optionalAttrs pkgs.stdenv.isLinux {
gcc-plain = makeShell {
stdenv = plainGccStdenv;
compilerName = "gcc";
version = gccVersion;
versionedTools = gccVersionedTools;
extraPackages = [ plainGcov ];
warningHook = plainWarningHook;
};
clang-plain = makeShell {
stdenv = plainClangStdenv;
compilerName = "clang";
version = llvmVersion;
versionedTools = clangVersionedTools;
warningHook = plainWarningHook;
};
}

View File

@@ -8,12 +8,10 @@ RUN mkdir -p ~/.config/nix && \
# Copy our source and setup our working dir.
COPY nix/ci-env.nix /tmp/build/nix/ci-env.nix
COPY nix/compilers.nix /tmp/build/nix/compilers.nix
COPY nix/packages.nix /tmp/build/nix/packages.nix
COPY nix/utils.nix /tmp/build/nix/utils.nix
COPY flake.nix /tmp/build/
COPY flake.lock /tmp/build/
COPY rust-toolchain.toml /tmp/build/
WORKDIR /tmp/build
FROM builder-source AS builder

View File

@@ -47,9 +47,7 @@ work without `ca-certificates` being installed in the base image.
[`test_files/cpp/sources/`](./test_files/cpp/sources) with both `g++` and
`clang++`, and sanitizers, and
- compiles the Rust test programs in
[`test_files/rust/sources/`](./test_files/rust/sources) with `rustc`, and
builds the [`test_files/rust/proc_macro/`](./test_files/rust/proc_macro)
workspace with `cargo` to exercise proc-macro dylib loading.
[`test_files/rust/sources/`](./test_files/rust/sources) with `rustc`.
3. **`tester`** — Start again from a clean `BASE_IMAGE` (no Nix toolchain),
install only the sanitizer runtime libraries
([`install-sanitizer-libs.sh`](./install-sanitizer-libs.sh)), and run the
@@ -78,23 +76,20 @@ toolchain being present at runtime. Two pieces make that work:
[`loader-path.sh`](./loader-path.sh) reports the expected loader path for the
current architecture, so we can patch the binaries to use the correct loader.
The build then verifies all of this end to end, and the C++ and Rust programs
go through the same pipeline: each is compiled in `final`, has its `PT_INTERP`
patched to the target loader, and is then run in the clean `tester` stage to
confirm it emits the expected diagnostic on a stock base image. The C++ programs
are in `test_files/cpp/sources/` (a regular binary plus ASan/TSan/UBSan
variants); the Rust programs are in `test_files/rust/sources/` (a hello binary
plus panic and overflow-check variants), plus the `test_files/rust/proc_macro/`
workspace — a crate whose compilation additionally loads a proc-macro dylib, and
whose resulting binary is patched and run like the others.
The build then verifies all of this end to end: the C++ test programs in
`test_files/cpp/sources/` (a regular binary plus ASan/TSan/UBSan variants) and
the Rust test programs in `test_files/rust/sources/` (a hello binary plus panic
and overflow-check variants) are compiled in `final`, their `PT_INTERP` is
patched to the target loader, and they are run in the clean `tester` stage to
confirm each emits the expected diagnostic on a stock base image.
## Files
| File | Purpose |
| ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------ |
| [`./Dockerfile`](./Dockerfile) | Multi-stage build described above. |
| [`./loader-path.sh`](./loader-path.sh) | Print the dynamic-linker (`PT_INTERP`) path for the current architecture. |
| [`./test_files/cpp/`](./test_files/cpp) | C++ sanitizer smoke test: sources + compile/run scripts. |
| [`./test_files/rust/`](./test_files/rust) | Rust smoke test: rustc sources + a cargo proc-macro workspace + compile/run scripts. |
| [`/bin/check-tools.sh`](../../bin/check-tools.sh) | Verify every expected tools are present and runnable. |
| [`/bin/install-sanitizer-libs.sh`](../../bin/install-sanitizer-libs.sh) | Install `libasan`/`libtsan`/`libubsan` runtimes on the supported base images. |
| File | Purpose |
| ----------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| [`./Dockerfile`](./Dockerfile) | Multi-stage build described above. |
| [`./loader-path.sh`](./loader-path.sh) | Print the dynamic-linker (`PT_INTERP`) path for the current architecture. |
| [`./test_files/cpp/`](./test_files/cpp) | C++ sanitizer smoke test: sources + compile/run scripts. |
| [`./test_files/rust/`](./test_files/rust) | Rust rustc smoke test: sources + compile/run scripts. |
| [`/bin/check-tools.sh`](../../bin/check-tools.sh) | Verify every expected tools are present and runnable. |
| [`/bin/install-sanitizer-libs.sh`](../../bin/install-sanitizer-libs.sh) | Install `libasan`/`libtsan`/`libubsan` runtimes on the supported base images. |

View File

@@ -40,29 +40,6 @@ compile hello
compile panic
compile overflow "-C overflow-checks=on"
function compile_proc_macro() {
local proj="${src_dir}/../proc_macro"
echo "=== Building proc-macro workspace (cargo) ==="
cargo build --manifest-path "${proj}/Cargo.toml" --offline
local built="${proj}/target/debug/test_macro"
if [ ! -f "${built}" ]; then
echo "ERROR: built test_macro binary not found at ${built}" >&2
exit 1
fi
local binary="${dst_dir}/proc_macro"
cp "${built}" "${binary}"
echo "=== Patching ${binary} to use ${loader} as PT_INTERP ==="
patchelf --set-interpreter "${loader}" --remove-rpath "${binary}"
rm -rf "${proj}/target"
}
compile_proc_macro
echo "=== All binaries compiled ==="
ls -la "${dst_dir}"

View File

@@ -1,14 +0,0 @@
# This file is automatically @generated by Cargo.
# It is not intended for manual editing.
version = 4
[[package]]
name = "echo_macro"
version = "0.0.0"
[[package]]
name = "test_macro"
version = "0.0.0"
dependencies = [
"echo_macro",
]

View File

@@ -1,3 +0,0 @@
[workspace]
resolver = "2"
members = ["echo_macro", "test_macro"]

View File

@@ -1,8 +0,0 @@
[package]
name = "echo_macro"
version = "0.0.0"
edition = "2024"
publish = false
[lib]
proc-macro = true

View File

@@ -1,6 +0,0 @@
use proc_macro::TokenStream;
#[proc_macro]
pub fn define_echo(item: TokenStream) -> TokenStream {
format!("fn echo() -> u32 {{ {item} }}").parse().unwrap()
}

View File

@@ -1,8 +0,0 @@
[package]
name = "test_macro"
version = "0.0.0"
edition = "2024"
publish = false
[dependencies]
echo_macro = { path = "../echo_macro" }

View File

@@ -1,9 +0,0 @@
use echo_macro::define_echo;
define_echo!(42);
fn main() {
let a = echo();
println!("proc-macro answer = {a}");
assert_eq!(a, 42, "proc-macro expansion produced the wrong value");
}

View File

@@ -1,7 +1,7 @@
#!/bin/bash
# Run pre-compiled Rust binaries and confirm each emits its expected diagnostic.
# Binaries must already exist in <bins_dir> as <name> for name in
# {hello,panic,overflow,proc_macro}.
# {hello,panic,overflow}.
set -eo pipefail
@@ -54,13 +54,12 @@ declare -A expect=(
[hello]="Hello from main thread"
[panic]="explicit panic from test"
[overflow]="attempt to add with overflow"
[proc_macro]="proc-macro answer = 42"
)
for name in hello panic overflow proc_macro; do
for name in hello panic overflow; do
binary="${bins_dir}/${name}"
if [ "${name}" = "hello" ] || [ "${name}" = "proc_macro" ]; then
if [ "${name}" = "hello" ]; then
expected_rc=0
else
expected_rc=nonzero

View File

@@ -15,66 +15,6 @@ let
runClangTidy = pkgs.writeShellScriptBin "run-clang-tidy" ''
exec ${pkgs.python3}/bin/python3 ${llvmPackages.clang-unwrapped}/bin/run-clang-tidy "$@"
'';
rustToolchain = pkgs.rust-bin.fromRustupToolchainFile ../rust-toolchain.toml;
# Nix wraps its toolchain so that binaries are exposed only under unsuffixed
# names (gcc, g++, clang-tidy, ...). Several tools probe for a
# version-suffixed name first and fall back to a system binary on the PATH
# when Nix doesn't provide it:
# - Conan's Boost recipe looks up `g++-<major>` before plain `g++`.
# - bin/pre-commit/clang_tidy_check.py looks up `run-clang-tidy-<v>` and
# `clang-apply-replacements-<v>` before the unsuffixed names.
# On a host that also has the matching system binary (e.g. Ubuntu's
# `/usr/bin/g++-15` or `clang-tidy-22`) the probe escapes Nix and mixes a
# system tool into the Nix environment. Generate version-suffixed symlinks
# next to a package's tools so those probes resolve to the Nix ones.
#
# Compiler links must point at whichever compiler is active in a given
# environment (the plain stdenv compiler in the dev shell, the custom-glibc
# wrappers in ci-env.nix), so those callers pass their own `package`; the
# clang tooling is environment-independent and is linked in commonPackages.
mkVersionedToolLinks =
{
name,
package,
version,
tools,
}:
pkgs.linkFarm "${name}-${toString version}-versioned-links" (
map (tool: {
name = "bin/${tool}-${toString version}";
path = "${package}/bin/${tool}";
}) tools
);
# The cc-wrapper doesn't re-export gcov, but coverage tooling (gcovr) needs a
# gcov that exactly matches the compiler. Surface it from a gcc `cc` output.
mkGcov =
{ name, cc }:
pkgs.linkFarm "gcov-${name}" [
{
name = "bin/gcov";
path = "${cc}/bin/gcov";
}
];
clangToolLinks = mkVersionedToolLinks {
name = "clang-tools";
package = clangTools;
version = llvmVersion;
tools = [
"clang-tidy"
"clang-apply-replacements"
"clang-format"
];
};
runClangTidyLink = mkVersionedToolLinks {
name = "run-clang-tidy";
package = runClangTidy;
version = llvmVersion;
tools = [ "run-clang-tidy" ];
};
in
{
inherit
@@ -82,13 +22,9 @@ in
llvmVersion
gccPackage
llvmPackages
mkVersionedToolLinks
mkGcov
;
commonPackages = with pkgs; [
clangToolLinks
runClangTidyLink
ccache
clangbuildanalyzer
clangTools
@@ -127,10 +63,14 @@ in
vim
zip
# Rust packages
cargo
cargo-audit
cargo-llvm-cov
cargo-nextest
clippy
corrosion
rustToolchain
rust-analyzer
rustc
rustfmt
];
}

View File

@@ -1,8 +1,4 @@
{
nixpkgs,
nixpkgs-custom-glibc,
rust-overlay,
}:
{ nixpkgs, nixpkgs-custom-glibc }:
function:
nixpkgs.lib.genAttrs
[
@@ -14,12 +10,7 @@ nixpkgs.lib.genAttrs
(
system:
function {
# rust-overlay adds `pkgs.rust-bin`, from which we build the pinned Rust
# toolchain (see packages.nix). Consumed by both the CI image and dev shell.
pkgs = import nixpkgs {
inherit system;
overlays = [ (import rust-overlay) ];
};
pkgs = import nixpkgs { inherit system; };
# glibc 2.31 — matches the system libc on Ubuntu 20.04 LTS. Sourced
# from the nixpkgs snapshot pinned via the `nixpkgs-custom-glibc`
# flake input, so the build uses the compiler from that snapshot

View File

@@ -1,6 +1,8 @@
# Linux Packaging
This directory contains all files needed to build RPM and Debian packages for `xrpld`.
This directory contains all files needed to build RPM and Debian packages for
`xrpld`. The packages also ship the `validator-keys` tool, so packaging requires
a build configured with `-Dvalidator_keys=ON`.
## Directory layout
@@ -46,17 +48,28 @@ To print the full packaging matrix (artifact names and images) for the current
Caller workflows (`on-pr.yml`, `on-tag.yml`, `on-trigger.yml`) call
`reusable-package.yml`. That workflow generates its own packaging matrix from
`package_configs` in `linux.json` (via `generate.py --packaging`) and fans out
one job per distro. Each job downloads the pre-built `xrpld` binary artifact and
runs in that distro's container, so the package format follows from the
container's package manager. The packaging script derives the package version
from the downloaded binary's `xrpld --version` output; no CMake configure or
build step is needed inside the packaging job.
one job per distro. Each job downloads the pre-built `xrpld` and `validator-keys`
binary artifacts and runs in that distro's container, so the package format
follows from the container's package manager. The packaging script derives the
package version from the downloaded binary's `xrpld --version` output; no CMake
configure or build step is needed inside the packaging job.
The binaries come from the `debian` and `rhel` build configurations in
`linux.json`'s `configs` section, which pass `-Dvalidator_keys=ON` so that the
build job produces `validator-keys` next to `xrpld` and uploads it as the
`validator-keys-<config name>` artifact. The packaging entry for a distro names
both artifacts (`xrpld_artifact_name` and `validator_keys_artifact_name`), so a
packaged configuration must keep `-Dvalidator_keys=ON`.
`validator-keys` is fetched from an exact commit pinned in
[`cmake/XrplValidatorKeys.cmake`](../cmake/XrplValidatorKeys.cmake), so a given
`xrpld` version always packages the same tool; bump that commit deliberately.
### Locally (mirrors CI)
With an `xrpld` binary already built at `build/xrpld`, run the packaging step
inside the same container CI uses. The image tag is derived from `linux.json`
so you don't need to hardcode a SHA.
With `xrpld` and `validator-keys` binaries already built at `build/xrpld` and
`build/validator-keys`, run the packaging step inside the same container CI uses.
The image tag is derived from `linux.json` so you don't need to hardcode a SHA.
```bash
# From the repo root. Each distro's container image is the `image` field of its
@@ -87,6 +100,7 @@ needed, but the host toolchain replaces the pinned CI image:
```bash
cmake \
-Dxrpld=ON \
-Dvalidator_keys=ON \
-Dpkg_release=1 \
-Dtests=OFF \
..
@@ -95,9 +109,11 @@ cmake --build . --target package # deb on Debian/Ubuntu, rpm on RHEL
```
The `cmake/XrplPackaging.cmake` module defines the `package` target only if at
least one of `rpmbuild` / `dpkg-buildpackage` is present; `build_pkg.sh` then
infers the package format from the host's package manager. The packaging script
installs to FHS-standard paths (`/usr/bin`, `/etc/xrpld`, etc.) regardless of
least one of `rpmbuild` / `dpkg-buildpackage` is present and both the `xrpld` and
`validator-keys` targets exist (`-Dxrpld=ON -Dvalidator_keys=ON`); the target
builds both binaries before packaging. `build_pkg.sh` then infers the package
format from the host's package manager. The packaging script installs to
FHS-standard paths (`/usr/bin`, `/etc/xrpld`, etc.) regardless of
`CMAKE_INSTALL_PREFIX`.
The package version is not a CMake input on this path: `build_pkg.sh` derives it
@@ -156,13 +172,17 @@ CMake/CI integration. The CI workflow and the CMake `package` target both invoke
and lets the script use defaults for the rest.
It resolves `SRC_DIR` and `BUILD_DIR` to absolute paths, then calls
`stage_common()` to copy the binary, config files, and shared support files
into the staging area, and invokes the platform build tool.
`stage_common()` to copy the `xrpld` and `validator-keys` binaries, config files,
and shared support files into the staging area, and invokes the platform build
tool. Both binaries must be present in `BUILD_DIR` and must run in the packaging
environment; a missing or non-runnable one fails early. That runtime check is
what catches a binary still linked against the Nix store's ELF loader (see
`patch_nix_binary` in `cmake/PatchNixBinary.cmake`).
### RPM
1. Creates the standard `rpmbuild/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}` tree inside the build directory.
2. Copies `xrpld.spec` and all shared source files (binary, configs, service files) into `SOURCES/`.
2. Copies `xrpld.spec` and all shared source files (binaries, configs, service files) into `SOURCES/`.
3. Runs `rpmbuild -bb`, passing the normalized package metadata version as the
`pkg_version` RPM macro and `PKG_RELEASE` as the `pkg_release` RPM macro.
The spec uses manual `install` commands to place files, disables `dwz`, and
@@ -182,7 +202,8 @@ service restart.
### DEB
1. Creates a staging source tree at `debbuild/source/` inside the build directory.
2. Stages the binary, configs, `README.md`, and `LICENSE.md`.
2. Stages the binaries, configs, `README.md`, `LICENSE.md`, and
`validator-keys-LICENSE`.
3. Copies `package/debian/` control files into `debbuild/source/debian/`.
4. Copies shared service/sysusers/tmpfiles into `debian/` where `dh_installsystemd`, `dh_installsysusers`, and `dh_installtmpfiles` pick them up automatically.
5. Generates a minimal `debian/changelog` using `${pkg_version}-${PKG_RELEASE}`,

View File

@@ -1,7 +1,8 @@
#!/usr/bin/env bash
set -euo pipefail
# Build an RPM or Debian package from a pre-built xrpld binary.
# Build an RPM or Debian package from the pre-built xrpld and validator-keys
# binaries.
#
# Flags override env vars; env vars override defaults.
@@ -11,7 +12,9 @@ Usage: build_pkg.sh [options]
Options (each can also be set via the env var shown):
--src-dir DIR repo root [SRC_DIR; default: ${PWD}]
--build-dir DIR directory holding xrpld [BUILD_DIR; default: ${PWD}/build]
--build-dir DIR directory holding the
xrpld and validator-keys
binaries [BUILD_DIR; default: ${PWD}/build]
--pkg-release N package release iteration [PKG_RELEASE; default: 1]
--source-date-epoch SECS reproducibility timestamp [SOURCE_DATE_EPOCH; latest git ctime; fallback: current time]
-h, --help show this help and exit
@@ -69,15 +72,44 @@ SRC_DIR="$(cd "${SRC_DIR:-${PWD}}" && pwd)"
BUILD_DIR="${BUILD_DIR:-${PWD}/build}"
if [[ ! -d "${BUILD_DIR}" ]]; then
echo "build_pkg.sh: build directory not found: ${BUILD_DIR}" >&2
echo "Build xrpld before packaging, or set BUILD_DIR to the directory containing xrpld." >&2
echo "Build the binaries before packaging, or set BUILD_DIR to the directory containing them." >&2
exit 1
fi
BUILD_DIR="$(cd "${BUILD_DIR}" && pwd)"
xrpld_binary="${BUILD_DIR}/xrpld"
if [[ ! -x "${xrpld_binary}" ]]; then
echo "build_pkg.sh: expected executable xrpld binary at ${xrpld_binary}." >&2
echo "Build xrpld before packaging, or set BUILD_DIR to the directory containing xrpld." >&2
validator_keys_binary="${BUILD_DIR}/validator-keys"
# Report both binaries at once: they share a single BUILD_DIR, so telling the
# reader to point it at one of them in isolation is advice they cannot follow.
missing=()
[[ -x "${xrpld_binary}" ]] || missing+=(xrpld)
[[ -x "${validator_keys_binary}" ]] || missing+=(validator-keys)
if [[ ${#missing[@]} -gt 0 ]]; then
echo "build_pkg.sh: missing or not executable in ${BUILD_DIR}: ${missing[*]}" >&2
echo "Both binaries come from a single CMake build directory configured with" >&2
echo "-Dxrpld=ON -Dvalidator_keys=ON. Build them, then point BUILD_DIR at that" >&2
echo "directory." >&2
exit 1
fi
# Shipping validator-keys means shipping its notice, so treat it as required
# rather than letting a package go out without the attribution.
validator_keys_license="${BUILD_DIR}/validator-keys-LICENSE"
if [[ ! -f "${validator_keys_license}" ]]; then
echo "build_pkg.sh: missing ${validator_keys_license}." >&2
echo "cmake/XrplValidatorKeys.cmake copies it out of the fetched" >&2
echo "validator-keys-tool source, so reconfigure with -Dvalidator_keys=ON." >&2
exit 1
fi
# The binary must also *run* here. Packaging happens in a vanilla distro
# container, so this is what catches a binary still pointing at the Nix store's
# ELF loader (see patch_nix_binary in cmake/PatchNixBinary.cmake); xrpld is
# covered implicitly by the version query below.
if ! "${validator_keys_binary}" --version >/dev/null; then
echo "build_pkg.sh: ${validator_keys_binary} exists but does not run here." >&2
exit 1
fi
@@ -150,7 +182,9 @@ stage_common() {
local dest="$1"
mkdir -p "${dest}"
cp "${BUILD_DIR}/xrpld" "${dest}/xrpld"
cp "${xrpld_binary}" "${dest}/xrpld"
cp "${validator_keys_binary}" "${dest}/validator-keys"
cp "${validator_keys_license}" "${dest}/validator-keys-LICENSE"
cp "${SRC_DIR}/cfg/xrpld-example.cfg" "${dest}/xrpld.cfg"
cp "${SRC_DIR}/cfg/validators-example.txt" "${dest}/validators.txt"
cp "${SRC_DIR}/LICENSE.md" "${dest}/LICENSE.md"

View File

@@ -18,6 +18,8 @@ Depends:
${shlibs:Depends},
${misc:Depends}
Description: XRP Ledger daemon
Reference implementation of the XRP Ledger protocol.
Participates in the peer-to-peer network, processes transactions,
and maintains a local ledger copy.
xrpld is the reference implementation of the XRP Ledger protocol. It
participates in the peer-to-peer XRP Ledger network, processes
transactions, and maintains the ledger database.
This package also includes the validator-keys tool for validator key
management.

View File

@@ -4,6 +4,25 @@ Source: https://github.com/XRPLF/rippled
Files: *
Copyright: 2011-present, the XRP Ledger developers
License: ISC
Files: validator-keys
Copyright: 2016, Ripple Labs Inc.
2011, Arthur Britto, David Schwartz, Jed McCaleb, Vinnie Falco, Bob Way,
Eric Lombrozo, Nikolaos D. Bougalis, Howard Hinnant
2013, Raw Material Software Ltd.
2003-2011, Christopher M. Kohlhoff
2009-2010, Satoshi Nakamoto
2011, The Bitcoin developers
2003-2005, Tom Wu
License: ISC
Comment: Built from https://github.com/ripple/validator-keys-tool at the commit
pinned in cmake/XrplValidatorKeys.cmake. Besides ISC-licensed code it
incorporates work under the Boost Software License 1.0 (ASIO), the MIT/X11
license (Bitcoin) and Tom Wu's license, whose terms require its notice to be
retained intact. The complete upstream notice is therefore shipped verbatim as
/usr/share/doc/xrpld/validator-keys-LICENSE.
License: ISC
Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above

View File

@@ -18,6 +18,7 @@ override_dh_installsysusers:
override_dh_install:
install -D -m 0755 xrpld debian/xrpld/usr/bin/xrpld
install -D -m 0755 validator-keys debian/xrpld/usr/bin/validator-keys
install -D -m 0644 xrpld.cfg debian/xrpld/etc/xrpld/xrpld.cfg
install -D -m 0644 validators.txt debian/xrpld/etc/xrpld/validators.txt

View File

@@ -1 +1,2 @@
README.md
validator-keys-LICENSE

View File

@@ -32,6 +32,8 @@ BuildRequires: systemd-rpm-macros
xrpld is the reference implementation of the XRP Ledger protocol. It
participates in the peer-to-peer XRP Ledger network, processes
transactions, and maintains the ledger database.
This package also includes the validator-keys tool for validator key
management.
%prep
:
@@ -41,6 +43,7 @@ transactions, and maintains the ledger database.
%install
install -Dm0755 %{_sourcedir}/xrpld %{buildroot}%{_bindir}/%{name}
install -Dm0755 %{_sourcedir}/validator-keys %{buildroot}%{_bindir}/validator-keys
install -Dm0644 %{_sourcedir}/xrpld.cfg %{buildroot}%{_sysconfdir}/%{name}/xrpld.cfg
install -Dm0644 %{_sourcedir}/validators.txt %{buildroot}%{_sysconfdir}/%{name}/validators.txt
@@ -59,6 +62,8 @@ install -Dm0644 %{_sourcedir}/xrpld.logrotate %{buildroot}%{_sysconfdir}/lo
# Docs
install -Dm0644 %{_sourcedir}/LICENSE.md %{buildroot}%{_docdir}/%{name}/LICENSE.md
install -Dm0644 %{_sourcedir}/README.md %{buildroot}%{_docdir}/%{name}/README.md
# Upstream notice for the bundled validator-keys tool.
install -Dm0644 %{_sourcedir}/validator-keys-LICENSE %{buildroot}%{_docdir}/%{name}/validator-keys-LICENSE
# Legacy compatibility for pre-FHS package layouts.
# TODO: remove after rippled fully deprecated.
@@ -80,11 +85,13 @@ systemd-tmpfiles --create %{_tmpfilesdir}/xrpld.conf || :
%files
%license %{_docdir}/%{name}/LICENSE.md
%license %{_docdir}/%{name}/validator-keys-LICENSE
%doc %{_docdir}/%{name}/README.md
%dir %{_sysconfdir}/%{name}
%{_bindir}/%{name}
%{_bindir}/validator-keys
%config(noreplace) %{_sysconfdir}/%{name}/xrpld.cfg
%config(noreplace) %{_sysconfdir}/%{name}/validators.txt

View File

@@ -1,4 +0,0 @@
[toolchain]
channel = "1.95"
components = ["rustfmt", "clippy", "rust-analyzer", "llvm-tools-preview"]
profile = "minimal"

View File

@@ -1,22 +0,0 @@
include(XrplAddBenchmark)
# Benchmark requirements.
find_package(benchmark REQUIRED)
# Custom target for all benchmarks defined in this file.
add_custom_target(xrpl.benchmarks)
# Common library dependencies for every benchmark module. `benchmark_main`
# supplies a `main()` that parses the standard Google Benchmark CLI flags
# (`--benchmark_filter`, `--benchmark_format`, ...), so no per-module main.cpp
# is needed.
add_library(xrpl.imports.bench INTERFACE)
target_link_libraries(
xrpl.imports.bench
INTERFACE benchmark::benchmark_main xrpl.libxrpl
)
# One benchmark executable for each module.
xrpl_add_benchmark(nodestore)
target_link_libraries(xrpl.bench.nodestore PRIVATE xrpl.imports.bench)
add_dependencies(xrpl.benchmarks xrpl.bench.nodestore)

View File

@@ -1,329 +0,0 @@
#include <xrpl/nodestore/Backend.h>
#include <xrpl/basics/base_uint.h>
#include <xrpl/nodestore/NodeObject.h>
#include <xrpl/nodestore/Types.h>
#include <benchmark/benchmark.h>
#include <benchmarks/libxrpl/nodestore/NodeStoreBench.h>
#include <array>
#include <cstddef>
#include <cstdint>
#include <functional>
#include <memory>
#include <string>
#include <string_view>
#include <utility>
#include <vector>
namespace xrpl::NodeStore {
namespace {
constexpr std::size_t kPoolSizes[] = {1000, 10000, 100000};
constexpr int kThreadCounts[] = {1, 4, 8};
constexpr std::size_t kBatchSize = 256;
constexpr std::string_view kNamePrefix = "BM_Backend_";
constexpr std::string_view kNameSeparator = "/";
struct RunState
{
std::unique_ptr<BackendHarness> harness;
Batch present; // prefix-1 objects, eligible to be stored
Batch recent; // prefix-1 objects in the "future" key space
std::vector<uint256> missing; // prefix-2 keys that are never stored
std::vector<std::size_t> shuffle; // [0, poolSize) permutation for random-like access
std::size_t avgPayload = 0; // mean getData().size() over `present`
void
release()
{
harness.reset();
Batch{}.swap(present);
Batch{}.swap(recent);
std::vector<uint256>{}.swap(missing);
std::vector<std::size_t>{}.swap(shuffle);
}
};
struct SetupContext
{
RunState& rs;
Backend& backend;
std::size_t poolSize;
};
struct IterateContext
{
RunState& rs;
Backend& backend;
std::size_t index;
std::size_t poolSize;
};
struct Workload
{
std::string_view name;
std::function<void(SetupContext const&)> setup;
std::function<void(IterateContext const&)> iterate;
bool reportBytes = false; // SetBytesProcessed from rs.avgPayload
bool clobber = true; // ClobberMemory after the loop (false for pure stores)
bool pinToPool = false; // pin iterations to one pool sweep instead of autotuning
};
// One store() per iteration. Iterations are pinned to one pool sweep (per
// thread) so the index never wraps past the pool - otherwise NuDB::doInsert
// swallows key_exists and the workload degenerates into duplicate-detection
// no-ops.
Workload const kInsert{
.name = "Insert",
.setup =
[](SetupContext const& ctx) {
ctx.rs.present = makePool(1, ctx.poolSize);
ctx.rs.avgPayload = averagePayload(ctx.rs.present);
},
.iterate =
[](IterateContext const& ctx) {
auto& [rs, backend, index, poolSize] = ctx;
backend.store(rs.present[index % poolSize]);
},
.reportBytes = true,
.clobber = false,
.pinToPool = true,
};
// One fetch() of a present key (a hit) per iteration.
Workload const kFetch{
.name = "Fetch",
.setup =
[](SetupContext const& ctx) {
ctx.rs.present = makePool(1, ctx.poolSize);
ctx.rs.avgPayload = averagePayload(ctx.rs.present);
prepopulate(ctx.backend, ctx.rs.present);
},
.iterate =
[](IterateContext const& ctx) {
auto& [rs, backend, index, poolSize] = ctx;
std::shared_ptr<NodeObject> result;
backend.fetch(rs.present[index % poolSize]->getHash(), &result);
benchmark::DoNotOptimize(result);
},
.reportBytes = true,
};
// One fetch() of a never-stored key (a miss); the backend is left empty.
Workload const kMissing{
.name = "Missing",
.setup = [](SetupContext const& ctx) { ctx.rs.missing = makeMissingKeys(ctx.poolSize); },
.iterate =
[](IterateContext const& ctx) {
auto& [rs, backend, index, poolSize] = ctx;
std::shared_ptr<NodeObject> result;
backend.fetch(rs.missing[index % poolSize], &result);
benchmark::DoNotOptimize(result);
},
};
// 80% hits / 20% misses. The fetch index comes from a shuffle table so access
// is random-like without per-iteration RNG cost; sequential `index % poolSize`
// would be artificially cache-friendly to RocksDB's block cache.
Workload const kMixed{
.name = "Mixed",
.setup =
[](SetupContext const& ctx) {
ctx.rs.present = makePool(1, ctx.poolSize);
ctx.rs.missing = makeMissingKeys(ctx.poolSize);
ctx.rs.shuffle = makeShuffle(ctx.poolSize, /*seed=*/1);
prepopulate(ctx.backend, ctx.rs.present);
},
.iterate =
[](IterateContext const& ctx) {
auto& [rs, backend, index, poolSize] = ctx;
std::shared_ptr<NodeObject> result;
auto const pick = rs.shuffle[index % poolSize];
if (index % 5 == 0)
{
backend.fetch(rs.missing[pick], &result);
}
else
{
backend.fetch(rs.present[pick]->getHash(), &result);
}
benchmark::DoNotOptimize(result);
},
};
// An xrpld-like cycle: a hit, a maybe-miss recent fetch, and a store. The
// recent fetch uses the shuffle table (not `slot`) so it doesn't fetch the item
// it's about to store this iteration - which would give an all-miss-then-hit
// step instead of a smooth ramp. The store walks sequentially so each recent
// object is stored once.
Workload const kWork{
.name = "Work",
.setup =
[](SetupContext const& ctx) {
ctx.rs.present = makePool(1, ctx.poolSize);
ctx.rs.recent = makePool(1, ctx.poolSize, ctx.poolSize);
ctx.rs.shuffle = makeShuffle(ctx.poolSize, /*seed=*/2);
prepopulate(ctx.backend, ctx.rs.present);
},
.iterate =
[](IterateContext const& ctx) {
auto& [rs, backend, index, poolSize] = ctx;
auto const slot = index % poolSize;
auto const pick = rs.shuffle[slot];
std::shared_ptr<NodeObject> historical;
backend.fetch(rs.present[pick]->getHash(), &historical);
benchmark::DoNotOptimize(historical);
std::shared_ptr<NodeObject> recent;
backend.fetch(rs.recent[pick]->getHash(), &recent);
benchmark::DoNotOptimize(recent);
backend.store(rs.recent[slot]);
},
.clobber = true,
.pinToPool = true,
};
auto
makeRunner(Workload w, std::string cfg, std::shared_ptr<RunState> rs)
{
return [w = std::move(w), cfg = std::move(cfg), rs = std::move(rs)](benchmark::State& state) {
auto const poolSize = static_cast<std::size_t>(state.range(0));
if (state.thread_index() == 0)
{
rs->harness = std::make_unique<BackendHarness>(cfg);
w.setup(
SetupContext{.rs = *rs, .backend = *rs->harness->backend, .poolSize = poolSize});
}
std::size_t index = state.thread_index();
for (auto _ : state)
{
w.iterate(
IterateContext{
.rs = *rs,
.backend = *rs->harness->backend,
.index = index,
.poolSize = poolSize});
index += state.threads();
}
if (w.clobber)
benchmark::ClobberMemory();
state.SetItemsProcessed(state.iterations());
if (w.reportBytes)
state.SetBytesProcessed(static_cast<std::int64_t>(state.iterations() * rs->avgPayload));
if (state.thread_index() == 0)
rs->release();
};
}
// Register workload `w` against backend `bc`, choosing the registration shape
// from `w.pinToPool`.
void
registerWorkload(BackendConfig const& bc, Workload const& w)
{
std::string const cfg = bc.config;
std::string name{kNamePrefix};
name += w.name;
name += kNameSeparator;
name += bc.name;
if (!w.pinToPool)
{
auto rs = std::make_shared<RunState>();
auto* b = benchmark::RegisterBenchmark(name, makeRunner(w, cfg, rs));
b->RangeMultiplier(10)->Range(kPoolSizes[0], kPoolSizes[std::size(kPoolSizes) - 1]);
b->Threads(1)->Threads(4)->Threads(8)->UseRealTime();
return;
}
for (auto const poolSize : kPoolSizes)
{
for (auto const threads : kThreadCounts)
{
if (poolSize % static_cast<std::size_t>(threads) != 0)
continue;
auto rs = std::make_shared<RunState>();
benchmark::RegisterBenchmark(name, makeRunner(w, cfg, rs))
->Arg(poolSize)
->Iterations(poolSize / static_cast<std::size_t>(threads))
->Threads(threads)
->UseRealTime();
}
}
}
// One storeBatch() of kBatchSize objects per iteration. Single-threaded:
// Backend::storeBatch must not run concurrently with itself or store().
// Iterations are pinned to the batch count so the index never wraps into
// key_exists no-ops. Kept separate from Workload: batch slicing and the
// per-batch item/byte accounting don't fit the thread-axis mold.
void
registerStoreBatch(BackendConfig const& bc)
{
std::string const cfg = bc.config;
std::string name{kNamePrefix};
name += "StoreBatch";
name += kNameSeparator;
name += bc.name;
for (auto const poolSize : kPoolSizes)
{
auto const numBatches = poolSize / kBatchSize;
if (numBatches == 0)
continue;
auto rs = std::make_shared<RunState>();
benchmark::RegisterBenchmark(
name,
[rs, cfg](benchmark::State& state) {
auto const poolSize = static_cast<std::size_t>(state.range(0));
rs->harness = std::make_unique<BackendHarness>(cfg);
rs->present = makePool(1, poolSize);
rs->avgPayload = averagePayload(rs->present);
std::vector<Batch> const batches = sliceBatches(rs->present, kBatchSize);
if (batches.empty())
{
state.SkipWithError("pool smaller than one batch");
return;
}
std::size_t index = 0;
for (auto _ : state)
{
rs->harness->backend->storeBatch(batches[index % batches.size()]);
++index;
}
state.SetItemsProcessed(static_cast<std::int64_t>(state.iterations() * kBatchSize));
state.SetBytesProcessed(
static_cast<std::int64_t>(state.iterations() * kBatchSize * rs->avgPayload));
rs->release();
})
->Arg(poolSize)
->Iterations(numBatches);
}
}
[[maybe_unused]] bool const kRegistered = [] {
auto const workloads = std::to_array({&kInsert, &kFetch, &kMissing, &kMixed, &kWork});
for (auto const& bc : backendConfigs())
{
for (auto const* w : workloads)
registerWorkload(bc, *w);
registerStoreBatch(bc);
}
return true;
}();
} // namespace
} // namespace xrpl::NodeStore

View File

@@ -1,243 +0,0 @@
#include <xrpl/nodestore/Database.h>
#include <xrpl/basics/Blob.h>
#include <xrpl/basics/base_uint.h>
#include <xrpl/nodestore/NodeObject.h>
#include <xrpl/nodestore/Types.h>
#include <benchmark/benchmark.h>
#include <benchmarks/libxrpl/nodestore/NodeStoreBench.h>
#include <array>
#include <cstddef>
#include <cstdint>
#include <functional>
#include <memory>
#include <string>
#include <string_view>
#include <utility>
#include <vector>
namespace xrpl::NodeStore {
namespace {
// Number of distinct objects pre-generated per run.
constexpr std::size_t kDefaultPoolSize = 100000;
// Async read threads the Database spawns. Unused by the synchronous fetch path
// these benchmarks take; kept fixed so runs are comparable.
constexpr int kReadThreads = 4;
constexpr std::string_view kNamePrefix = "BM_Database_";
constexpr std::string_view kNameSeparator = "/";
struct RunState
{
std::unique_ptr<DatabaseHarness> harness;
Batch present; // prefix-1 objects, eligible to be stored
Batch recent; // prefix-1 objects in the "future" key space
std::vector<uint256> missing; // prefix-2 keys that are never stored
std::vector<std::size_t> shuffle; // [0, poolSize) permutation for random-like access
std::size_t avgPayload = 0; // mean getData().size() over `present`
};
struct SetupContext
{
RunState& rs;
Database& db;
std::size_t poolSize;
};
struct IterateContext
{
RunState& rs;
Database& db;
std::uint32_t seq;
std::size_t index;
std::size_t poolSize;
};
struct Workload
{
std::string_view name;
std::function<void(SetupContext const&)> setup;
std::function<void(IterateContext const&)> iterate;
bool reportBytes = false;
bool pinIterations = false;
};
void
prepopulate(Database& db, Batch const& objects)
{
auto const seq = db.earliestLedgerSeq();
for (auto const& obj : objects)
{
Blob data(obj->getData());
db.store(obj->getType(), std::move(data), obj->getHash(), seq);
}
db.sync();
}
// One store() per iteration; a fresh Blob copy is handed over each time.
Workload const kStore{
.name = "Store",
.setup =
[](SetupContext const& ctx) {
ctx.rs.present = makePool(1, ctx.poolSize);
ctx.rs.avgPayload = averagePayload(ctx.rs.present);
},
.iterate =
[](IterateContext const& ctx) {
auto& [rs, db, seq, index, poolSize] = ctx;
auto const& obj = rs.present[index % poolSize];
Blob data(obj->getData());
db.store(obj->getType(), std::move(data), obj->getHash(), seq);
},
.reportBytes = true,
.pinIterations = true,
};
// One fetchNodeObject() of a stored key (a hit) per iteration.
Workload const kFetch{
.name = "Fetch",
.setup =
[](SetupContext const& ctx) {
ctx.rs.present = makePool(1, ctx.poolSize);
ctx.rs.avgPayload = averagePayload(ctx.rs.present);
prepopulate(ctx.db, ctx.rs.present);
},
.iterate =
[](IterateContext const& ctx) {
auto& [rs, db, seq, index, poolSize] = ctx;
auto obj = db.fetchNodeObject(rs.present[index % poolSize]->getHash(), seq);
benchmark::DoNotOptimize(obj);
},
.reportBytes = true,
};
// One fetchNodeObject() of a never-stored key (a miss) per iteration.
Workload const kMissing{
.name = "Missing",
.setup = [](SetupContext const& ctx) { ctx.rs.missing = makeMissingKeys(ctx.poolSize); },
.iterate =
[](IterateContext const& ctx) {
auto& [rs, db, seq, index, poolSize] = ctx;
auto obj = db.fetchNodeObject(rs.missing[index % poolSize], seq);
benchmark::DoNotOptimize(obj);
},
};
// 80% hits / 20% misses. The fetch index comes from a shuffle table so access
// is random-like without per-iteration RNG cost; sequential `index % poolSize`
// would be artificially cache-friendly.
Workload const kMixed{
.name = "Mixed",
.setup =
[](SetupContext const& ctx) {
ctx.rs.present = makePool(1, ctx.poolSize);
ctx.rs.missing = makeMissingKeys(ctx.poolSize);
ctx.rs.shuffle = makeShuffle(ctx.poolSize, /*seed=*/1);
prepopulate(ctx.db, ctx.rs.present);
},
.iterate =
[](IterateContext const& ctx) {
auto& [rs, db, seq, index, poolSize] = ctx;
auto const pick = rs.shuffle[index % poolSize];
std::shared_ptr<NodeObject> obj;
if (index % 5 == 0)
{
obj = db.fetchNodeObject(rs.missing[pick], seq);
}
else
{
obj = db.fetchNodeObject(rs.present[pick]->getHash(), seq);
}
benchmark::DoNotOptimize(obj);
},
};
// An xrpld-like cycle: a hit, a maybe-miss recent fetch, and a store. The
// recent fetch uses the shuffle table (not `slot`) so it doesn't fetch the item
// it's about to store this iteration - which would give an all-miss-then-hit
// step instead of a smooth ramp. The store walks sequentially so each recent
// object is stored once.
Workload const kWork{
.name = "Work",
.setup =
[](SetupContext const& ctx) {
ctx.rs.present = makePool(1, ctx.poolSize);
ctx.rs.recent = makePool(1, ctx.poolSize, ctx.poolSize);
ctx.rs.shuffle = makeShuffle(ctx.poolSize, /*seed=*/2);
prepopulate(ctx.db, ctx.rs.present);
},
.iterate =
[](IterateContext const& ctx) {
auto& [rs, db, seq, index, poolSize] = ctx;
auto const slot = index % poolSize;
auto const pick = rs.shuffle[slot];
auto historical = db.fetchNodeObject(rs.present[pick]->getHash(), seq);
benchmark::DoNotOptimize(historical);
auto recent = db.fetchNodeObject(rs.recent[pick]->getHash(), seq);
benchmark::DoNotOptimize(recent);
auto const& obj = rs.recent[slot];
Blob data(obj->getData());
db.store(obj->getType(), std::move(data), obj->getHash(), seq);
},
.pinIterations = true,
};
void
registerWorkload(BackendConfig const& bc, Workload const& w)
{
auto rs = std::make_shared<RunState>();
std::string const cfg = bc.config;
std::string name{kNamePrefix};
name += w.name;
name += kNameSeparator;
name += bc.name;
auto* b = benchmark::RegisterBenchmark(name, [rs, cfg, w](benchmark::State& state) {
auto const poolSize = static_cast<std::size_t>(state.range(0));
rs->harness = std::make_unique<DatabaseHarness>(cfg, kReadThreads);
auto& db = *rs->harness->db;
w.setup(SetupContext{.rs = *rs, .db = db, .poolSize = poolSize});
auto const seq = db.earliestLedgerSeq();
std::size_t index = 0;
for (auto _ : state)
{
w.iterate(
IterateContext{
.rs = *rs, .db = db, .seq = seq, .index = index, .poolSize = poolSize});
++index;
}
benchmark::ClobberMemory();
state.SetItemsProcessed(state.iterations());
if (w.reportBytes)
{
state.SetBytesProcessed(static_cast<std::int64_t>(state.iterations() * rs->avgPayload));
}
rs->harness.reset();
});
b->Arg(kDefaultPoolSize);
if (w.pinIterations)
b->Iterations(kDefaultPoolSize);
}
[[maybe_unused]] bool const kRegistered = [] {
auto const workloads = std::to_array({&kStore, &kFetch, &kMissing, &kMixed, &kWork});
for (auto const& bc : backendConfigs())
{
for (auto const* w : workloads)
registerWorkload(bc, *w);
}
return true;
}();
} // namespace
} // namespace xrpl::NodeStore

View File

@@ -1,318 +0,0 @@
#pragma once
#include <xrpl/basics/Blob.h>
#include <xrpl/basics/ByteUtilities.h>
#include <xrpl/basics/base_uint.h>
#include <xrpl/basics/safe_cast.h>
#include <xrpl/beast/utility/Journal.h>
#include <xrpl/beast/utility/temp_dir.h>
#include <xrpl/beast/xor_shift_engine.h>
#include <xrpl/config/BasicConfig.h>
#include <xrpl/nodestore/Backend.h>
#include <xrpl/nodestore/Database.h>
#include <xrpl/nodestore/DummyScheduler.h>
#include <xrpl/nodestore/Manager.h>
#include <xrpl/nodestore/NodeObject.h>
#include <xrpl/nodestore/Scheduler.h>
#include <xrpl/nodestore/Types.h>
#include <boost/algorithm/string/classification.hpp>
#include <boost/algorithm/string/split.hpp>
#include <algorithm>
#include <cstddef>
#include <cstdint>
#include <cstring>
#include <memory>
#include <numeric>
#include <random>
#include <string>
#include <utility>
#include <vector>
// Shared helpers for the NodeStore benchmarks.
//
namespace xrpl::NodeStore {
// Fill `bytes` of memory at `buffer` with random bits drawn from `g`.
template <class Generator>
inline void
rngcpy(void* buffer, std::size_t bytes, Generator& g)
{
using result_type = typename Generator::result_type;
while (bytes >= sizeof(result_type))
{
auto const v = g();
std::memcpy(buffer, &v, sizeof(v));
buffer = reinterpret_cast<std::uint8_t*>(buffer) + sizeof(v);
bytes -= sizeof(v);
}
if (bytes > 0)
{
auto const v = g();
std::memcpy(buffer, &v, bytes);
}
}
/**
* @brief Deterministic generator of a reproducible sequence of random NodeObjects.
*
* Indexing is stable: `obj(n)` and `key(n)` always return the same value for a
* given `n`, regardless of call order, because the engine is reseeded from `n`
* on every call.
*
* Using different prefixes guarantees the two key spaces are disjoint for the fetch-miss
* workloads.
*/
class Sequence
{
private:
static constexpr auto kMinSize = 250;
static constexpr auto kMaxSize = 1250;
beast::xor_shift_engine gen_;
std::uint8_t prefix_;
std::discrete_distribution<std::uint32_t> dType_;
std::uniform_int_distribution<std::uint32_t> dSize_;
public:
explicit Sequence(std::uint8_t prefix)
: prefix_(prefix)
// uniform distribution over hotLEDGER - hotTRANSACTION_NODE
// but exclude hotTRANSACTION = 2 (removed)
, dType_({1, 1, 0, 1, 1})
, dSize_(kMinSize, kMaxSize)
{
}
// Returns the n-th key. Used to generate keys that are never stored.
// The layout mirrors obj()'s: prefix at byte 0, RNG over the rest, so the
// two key spaces stay disjoint by construction (not by coincidence).
uint256
key(std::size_t n)
{
gen_.seed(n + 1);
uint256 result;
auto const data = static_cast<std::uint8_t*>(&*result.begin());
*data = prefix_;
rngcpy(data + 1, result.size() - 1, gen_);
return result;
}
// Returns the n-th complete NodeObject.
std::shared_ptr<NodeObject>
obj(std::size_t n)
{
gen_.seed(n + 1);
uint256 key;
auto const data = static_cast<std::uint8_t*>(&*key.begin());
*data = prefix_;
rngcpy(data + 1, key.size() - 1, gen_);
Blob value(dSize_(gen_));
rngcpy(&value[0], value.size(), gen_);
return NodeObject::createObject(
safeCast<NodeObjectType>(dType_(gen_)), std::move(value), key);
}
// Fills `b` with `size` consecutive NodeObjects starting at index `n`.
void
batch(std::size_t n, Batch& b, std::size_t size)
{
b.clear();
b.reserve(size);
while ((size--) != 0u)
b.push_back(obj(n++));
}
};
// Parse a comma-separated "key=value,key=value" string into a config Section.
inline Section
parseConfig(std::string const& s)
{
Section section;
std::vector<std::string> values;
boost::split(values, s, boost::algorithm::is_any_of(","));
section.append(values);
return section;
}
// Pre-generate `count` distinct objects from key space `prefix`, starting at
// sequence index `start`.
inline Batch
makePool(std::uint8_t prefix, std::size_t count, std::size_t start = 0)
{
Sequence seq(prefix);
Batch pool;
pool.reserve(count);
for (std::size_t i = 0; i < count; ++i)
pool.push_back(seq.obj(start + i));
return pool;
}
// Pre-generate `count` keys disjoint from every `makePool(...)` object, for
// measuring fetches that miss.
inline std::vector<uint256>
makeMissingKeys(std::size_t count)
{
Sequence seq(2);
std::vector<uint256> keys;
keys.reserve(count);
for (std::size_t i = 0; i < count; ++i)
keys.push_back(seq.key(i));
return keys;
}
// Mean payload size across a pool, used for SetBytesProcessed throughput.
inline std::size_t
averagePayload(Batch const& pool)
{
if (pool.empty())
return 0;
std::size_t total = 0;
for (auto const& obj : pool)
total += obj->getData().size();
return total / pool.size();
}
// Store every object and flush, so a following fetch exercises the real read
// path rather than an in-memory write buffer.
//
// We chunk the write at kBatchWriteLimitSize because Types.h documents that as
// the maximum allowed batch size. NuDB happens to tolerate larger batches
// today, but the benchmark should not rely on that.
//
// sync() is a no-op for both NuDB and RocksDB at the moment (NuDB has a small
// internal burst buffer that the timed loop will warm up). That is a contract
// hint, not a guarantee; if either backend ever grows a real flush we get it
// here for free.
inline void
prepopulate(Backend& backend, Batch const& objects)
{
for (std::size_t i = 0; i < objects.size(); i += kBatchWriteLimitSize)
{
auto const end = std::min(i + kBatchWriteLimitSize, objects.size());
backend.storeBatch(Batch(objects.begin() + i, objects.begin() + end));
}
backend.sync();
}
// A deterministic permutation of [0, size). Lets the timed loop visit the
// pre-generated pool in a random-like order with zero RNG cost per iteration -
// the Timing_test workloads it replaces used uniform_int_distribution per
// fetch, and a shuffle table reproduces that access pattern without paying for
// the distribution inside the timed region.
inline std::vector<std::size_t>
makeShuffle(std::size_t size, std::uint64_t seed)
{
std::vector<std::size_t> v(size);
std::iota(v.begin(), v.end(), std::size_t{0});
beast::xor_shift_engine gen(seed);
std::shuffle(v.begin(), v.end(), gen);
return v;
}
// Partition a pool into fixed-size batches. Any trailing remainder shorter than
// `batchSize` is dropped, so every returned batch has exactly `batchSize`.
inline std::vector<Batch>
sliceBatches(Batch const& pool, std::size_t batchSize)
{
std::vector<Batch> batches;
if (batchSize == 0)
return batches;
batches.reserve(pool.size() / batchSize);
for (std::size_t i = 0; i + batchSize <= pool.size(); i += batchSize)
batches.emplace_back(pool.begin() + i, pool.begin() + i + batchSize);
return batches;
}
/**
* @brief RAII owner of a NodeStore Backend opened on a private temporary directory.
*
* Member declaration order matters: `tempDir` is declared first so it is
* destroyed last, after the backend has closed and released its files.
*/
struct BackendHarness
{
beast::TempDir tempDir;
DummyScheduler scheduler;
beast::Journal journal{beast::Journal::getNullSink()};
std::unique_ptr<Backend> backend;
explicit BackendHarness(std::string const& configString)
{
Section config = parseConfig(configString);
// A private, unique path per harness, so concurrent or repeated runs
// never share on-disk state.
config.set("path", tempDir.path());
backend =
Manager::instance().makeBackend(config, megabytes(std::size_t{4}), scheduler, journal);
backend->setDeletePath();
backend->open();
}
~BackendHarness()
{
if (backend)
backend->close();
}
};
/**
* RAII owner of a NodeStore Database - the application-facing wrapper around a
* Backend, which adds fetch/store accounting and the async read-thread pool.
*/
struct DatabaseHarness
{
beast::TempDir tempDir;
DummyScheduler scheduler;
beast::Journal journal{beast::Journal::getNullSink()};
std::unique_ptr<Database> db;
DatabaseHarness(std::string const& configString, int readThreads)
{
Section config = parseConfig(configString);
config.set("path", tempDir.path());
db = Manager::instance().makeDatabase(
megabytes(std::size_t{4}), scheduler, readThreads, config, journal);
}
~DatabaseHarness()
{
if (db)
db->stop();
}
};
// A NodeStore backend to benchmark, named for the --benchmark_filter CLI flag.
struct BackendConfig
{
char const* name; // short label, e.g. "nudb"
char const* config; // parseConfig() string, e.g. "type=nudb"
};
// The backends every workload is registered against.
//
// The in-memory backend is intentionally excluded. It keeps its table in a
// process-global map keyed by path, with no removal API, so building a fresh
// backend per run - as a microbenchmark must - would leak the whole dataset on
// every run. Timing_test, the suite this benchmark replaces, excluded it for
// the same reason. NuDB and RocksDB are the production backends worth timing.
//
// RocksDB is included only when it was compiled in (xrpl.libxrpl carries
// XRPL_ROCKSDB_AVAILABLE transitively).
inline std::vector<BackendConfig> const&
backendConfigs()
{
static std::vector<BackendConfig> const kConfigs = {
{.name = "nudb", .config = "type=nudb"},
#if XRPL_ROCKSDB_AVAILABLE
{.name = "rocksdb",
.config = "type=rocksdb,open_files=2000,filter_bits=12,cache_mb=256,"
"file_size_mb=8,file_size_mult=2"},
#endif
};
return kConfigs;
}
} // namespace xrpl::NodeStore

View File

@@ -76,24 +76,6 @@ getInverse()
return &kTab[0];
}
/**
* Returns max chars needed to encode a base64 string
*/
constexpr std::size_t
encodedSize(std::size_t n)
{
return 4 * ((n + 2) / 3);
}
/**
* Returns max bytes needed to decode a base64 string
*/
constexpr std::size_t
decodedSize(std::size_t n)
{
return ((n / 4) * 3) + 2;
}
/**
* Encode a series of octets as a padded, base64 string.
*

View File

@@ -26,14 +26,6 @@ namespace xrpl {
namespace directory {
struct Gap
{
uint64_t const page;
SLE::pointer node;
uint64_t const nextPage;
SLE::pointer next;
};
std::uint64_t
createRoot(
ApplyView& view,
@@ -134,9 +126,7 @@ insertPage(
if (page == 0)
return std::nullopt;
if (!view.rules().enabled(fixDirectoryLimit) && page >= kDirNodeMaxPages) // Old pages limit
{
return std::nullopt;
}
// We are about to create a new node; we'll link it to
// the chain first:
@@ -157,8 +147,12 @@ insertPage(
// Save some space by not specifying the value 0 since it's the default.
if (page != 1)
node->setFieldU64(sfIndexPrevious, page - 1);
XRPL_ASSERT_PARTS(!nextPage, "xrpl::directory::insertPage", "nextPage has default value");
/* Reserved for future use when directory pages may be inserted in
* between two other pages instead of only at the end of the chain.
if (nextPage)
node->setFieldU64(sfIndexNext, nextPage);
*/
describe(node);
view.insert(node);
@@ -174,7 +168,7 @@ ApplyView::dirAdd(
uint256 const& key,
std::function<void(SLE::ref)> const& describe)
{
auto const root = peek(directory);
auto root = peek(directory);
if (!root)
{
@@ -184,43 +178,6 @@ ApplyView::dirAdd(
auto [page, node, indexes] = directory::findPreviousPage(*this, directory, root);
if (rules().enabled(featureDefragDirectories))
{
// If there are more nodes than just the root, and there's no space in
// the last one, walk backwards to find one with space, or to find one
// missing.
std::optional<directory::Gap> gapPages;
while (page && indexes.size() >= kDIR_NODE_MAX_PAGES)
{
// Find a page with space, or a gap in pages.
auto [prevPage, prevNode, prevIndexes] =
directory::findPreviousPage(*this, directory, node);
if (!gapPages && prevPage != page - 1)
gapPages.emplace(prevPage, prevNode, page, node);
page = prevPage;
node = prevNode;
indexes = prevIndexes;
}
// We looped through all the pages back to the root.
if (!page)
{
// If we found a gap, use it.
if (gapPages)
{
return directory::insertPage(
*this,
gapPages->page,
gapPages->node,
gapPages->nextPage,
gapPages->next,
key,
directory,
describe);
}
std::tie(page, node, indexes) = directory::findPreviousPage(*this, directory, root);
}
}
// If there's space, we use it:
if (indexes.size() < kDirNodeMaxEntries)
{

View File

@@ -312,25 +312,6 @@ computeInterestAndFeeParts(
return std::make_pair(interest - fee, fee);
}
/* Rounds a raw (unrounded) interest amount to the loan's scale, then splits
* the rounded amount into net interest (to the vault) and management fee (to
* the broker).
*
* This is the common "round then split" step shared by late payment, full
* payment, and overpayment interest calculations.
*/
std::pair<Number, Number>
roundAndSplitInterest(
Asset const& asset,
Number const& rawInterest,
TenthBips16 managementFeeRate,
std::int32_t loanScale,
Number::RoundingMode mode = Number::getround())
{
auto const interest = roundToAsset(asset, rawInterest, loanScale, mode);
return computeInterestAndFeeParts(asset, interest, managementFeeRate, loanScale);
}
/* Calculates penalty interest accrued on overdue payments.
* Returns 0 if payment is not late.
*
@@ -406,18 +387,22 @@ loanAccruedInterest(
*
* This is the core function that updates the Loan ledger object fields based on
* a computed payment.
*/
LoanPaymentParts
doPayment(ExtendedPaymentComponents const& payment, SLE::ref loan)
{
auto totalValueOutstandingProxy = loan->at(sfTotalValueOutstanding);
auto principalOutstandingProxy = loan->at(sfPrincipalOutstanding);
auto managementFeeOutstandingProxy = loan->at(sfManagementFeeOutstanding);
auto paymentRemainingProxy = loan->at(sfPaymentRemaining);
auto prevPaymentDateProxy = loan->at(sfPreviousPaymentDueDate);
auto nextDueDateProxy = loan->at(sfNextPaymentDueDate);
std::uint32_t const paymentInterval = loan->at(sfPaymentInterval);
* The function is templated to work with both direct Number/uint32_t values
* (for testing/simulation) and ValueProxy types (for actual ledger updates).
*/
template <class NumberProxy, class UInt32Proxy, class UInt32OptionalProxy>
LoanPaymentParts
doPayment(
ExtendedPaymentComponents const& payment,
NumberProxy& totalValueOutstandingProxy,
NumberProxy& principalOutstandingProxy,
NumberProxy& managementFeeOutstandingProxy,
UInt32Proxy& paymentRemainingProxy,
UInt32Proxy& prevPaymentDateProxy,
UInt32OptionalProxy& nextDueDateProxy,
std::uint32_t paymentInterval)
{
XRPL_ASSERT_PARTS(nextDueDateProxy, "xrpl::detail::doPayment", "Next due date proxy set");
if (payment.specialCase == PaymentSpecialCase::Final)
@@ -485,12 +470,16 @@ doPayment(ExtendedPaymentComponents const& payment, SLE::ref loan)
// Principal can never exceed total value (principal is part of total value)
XRPL_ASSERT_PARTS(
// Use an explicit cast because the template parameter can be
// ValueProxy<Number> or Number
static_cast<Number>(principalOutstandingProxy) <=
static_cast<Number>(totalValueOutstandingProxy),
"xrpl::detail::doPayment",
"principal does not exceed total");
XRPL_ASSERT_PARTS(
// Use an explicit cast because the template parameter can be
// ValueProxy<Number> or Number
static_cast<Number>(managementFeeOutstandingProxy) >= beast::kZero,
"xrpl::detail::doPayment",
"fee outstanding stays valid");
@@ -728,23 +717,22 @@ tryOverpayment(
* overpayment would leave the loan in an invalid state, we can reject it
* gracefully without corrupting the ledger data.
*/
template <class NumberProxy>
std::expected<LoanPaymentParts, TER>
doOverpayment(
Rules const& rules,
Asset const& asset,
std::int32_t loanScale,
ExtendedPaymentComponents const& overpaymentComponents,
SLE::ref loan,
NumberProxy& totalValueOutstandingProxy,
NumberProxy& principalOutstandingProxy,
NumberProxy& managementFeeOutstandingProxy,
NumberProxy& periodicPaymentProxy,
Number const& periodicRate,
std::uint32_t const paymentRemaining,
TenthBips16 const managementFeeRate,
beast::Journal j)
{
auto totalValueOutstandingProxy = loan->at(sfTotalValueOutstanding);
auto principalOutstandingProxy = loan->at(sfPrincipalOutstanding);
auto managementFeeOutstandingProxy = loan->at(sfManagementFeeOutstanding);
auto periodicPaymentProxy = loan->at(sfPeriodicPayment);
auto const paymentsRemaining = loan->at(sfPaymentRemaining);
auto const loanState = constructLoanState(
totalValueOutstandingProxy, principalOutstandingProxy, managementFeeOutstandingProxy);
auto const periodicPayment = periodicPaymentProxy;
@@ -756,7 +744,7 @@ doOverpayment(
<< ", interestPart: " << overpaymentComponents.trackedInterestPart()
<< ", untrackedInterest: " << overpaymentComponents.untrackedInterest
<< ", totalDue: " << overpaymentComponents.totalDue
<< ", payments remaining :" << paymentsRemaining;
<< ", payments remaining :" << paymentRemaining;
// Attempt to re-amortize the loan with the overpayment applied.
// This modifies the temporary copies, leaving the proxies unchanged.
@@ -768,7 +756,7 @@ doOverpayment(
loanState,
periodicPayment,
periodicRate,
paymentsRemaining,
paymentRemaining,
managementFeeRate,
j);
if (!ret)
@@ -876,15 +864,16 @@ std::expected<ExtendedPaymentComponents, TER>
computeLatePayment(
Asset const& asset,
ApplyView const& view,
SLE::const_ref loan,
Number const& principalOutstanding,
std::int32_t nextDueDate,
ExtendedPaymentComponents const& periodic,
TenthBips32 lateInterestRate,
std::int32_t loanScale,
Number const& latePaymentFee,
STAmount const& amount,
TenthBips16 managementFeeRate,
beast::Journal j)
{
std::int32_t const nextDueDate = loan->at(sfNextPaymentDueDate);
std::int32_t const loanScale = loan->at(sfLoanScale);
// Check if the due date has passed. If not, reject the payment as
// being too soon
if (!hasExpired(view, nextDueDate))
@@ -892,15 +881,15 @@ computeLatePayment(
// Calculate the penalty interest based on how long the payment is overdue.
auto const latePaymentInterest = loanLatePaymentInterest(
loan->at(sfPrincipalOutstanding),
TenthBips32{loan->at(sfLateInterestRate)},
view.parentCloseTime(),
nextDueDate);
principalOutstanding, lateInterestRate, view.parentCloseTime(), nextDueDate);
// Round the late interest and split it between the vault (net interest)
// and the broker (management fee portion).
auto const [roundedLateInterest, roundedLateManagementFee] =
roundAndSplitInterest(asset, latePaymentInterest, managementFeeRate, loanScale);
// and the broker (management fee portion). This lambda ensures we
// round before splitting to maintain precision.
auto const [roundedLateInterest, roundedLateManagementFee] = [&]() {
auto const interest = roundToAsset(asset, latePaymentInterest, loanScale);
return computeInterestAndFeeParts(asset, interest, managementFeeRate, loanScale);
}();
XRPL_ASSERT(roundedLateInterest >= 0, "xrpl::detail::computeLatePayment : valid late interest");
XRPL_ASSERT_PARTS(
@@ -919,7 +908,7 @@ computeLatePayment(
// 1. Regular service fee (from periodic.untrackedManagementFee)
// 2. Late payment fee (fixed penalty)
// 3. Management fee portion of late interest
periodic.untrackedManagementFee + loan->at(sfLatePaymentFee) + roundedLateManagementFee,
periodic.untrackedManagementFee + latePaymentFee + roundedLateManagementFee,
// Untracked interest includes:
// 1. Any untracked interest from the regular payment (usually 0)
@@ -969,15 +958,22 @@ std::expected<ExtendedPaymentComponents, TER>
computeFullPayment(
Asset const& asset,
ApplyView& view,
SLE::const_ref loan,
Number const& principalOutstanding,
Number const& managementFeeOutstanding,
Number const& periodicPayment,
std::uint32_t paymentRemaining,
std::uint32_t prevPaymentDate,
std::uint32_t const startDate,
std::uint32_t const paymentInterval,
TenthBips32 const closeInterestRate,
std::int32_t loanScale,
Number const& totalInterestOutstanding,
Number const& periodicRate,
Number const& closePaymentFee,
STAmount const& amount,
TenthBips16 managementFeeRate,
beast::Journal j)
{
std::uint32_t const paymentRemaining = loan->at(sfPaymentRemaining);
std::int32_t const loanScale = loan->at(sfLoanScale);
// Full payment must be made before the final scheduled payment.
if (paymentRemaining <= 1)
{
@@ -990,7 +986,7 @@ computeFullPayment(
// This theoretical (unrounded) value is used to compute interest and
// penalties accurately.
Number const theoreticalPrincipalOutstanding = loanPrincipalFromPeriodicPayment(
view.rules(), loan->at(sfPeriodicPayment), periodicRate, paymentRemaining);
view.rules(), periodicPayment, periodicRate, paymentRemaining);
// Full payment interest includes both accrued interest (time since last
// payment) and prepayment penalty (for closing early).
@@ -998,21 +994,18 @@ computeFullPayment(
theoreticalPrincipalOutstanding,
periodicRate,
view.parentCloseTime(),
loan->at(sfPaymentInterval),
loan->at(sfPreviousPaymentDueDate),
loan->at(sfStartDate),
TenthBips32{loan->at(sfCloseInterestRate)});
paymentInterval,
prevPaymentDate,
startDate,
closeInterestRate);
// Split the full payment interest into net interest (to vault) and management fee (to broker),
// applying proper rounding.
auto const [roundedFullInterest, roundedFullManagementFee] = roundAndSplitInterest(
asset, fullPaymentInterest, managementFeeRate, loanScale, Number::RoundingMode::Downward);
LoanState const loanState = constructLoanState(loan);
Number const principalOutstanding = loanState.principalOutstanding;
Number const managementFeeOutstanding = loanState.managementFeeDue;
Number const totalInterestOutstanding = loanState.interestDue;
Number const closePaymentFee = roundToAsset(asset, loan->at(sfClosePaymentFee), loanScale);
// Split the full payment interest into net interest (to vault) and
// management fee (to broker), applying proper rounding.
auto const [roundedFullInterest, roundedFullManagementFee] = [&]() {
auto const interest =
roundToAsset(asset, fullPaymentInterest, loanScale, Number::RoundingMode::Downward);
return computeInterestAndFeeParts(asset, interest, managementFeeRate, loanScale);
}();
ExtendedPaymentComponents const full{
PaymentComponents{
@@ -1053,7 +1046,8 @@ computeFullPayment(
"xrpl::detail::computeFullPayment",
"total due is rounded");
JLOG(j.trace()) << "computeFullPayment result: periodicRate: " << periodicRate
JLOG(j.trace()) << "computeFullPayment result: periodicPayment: " << periodicPayment
<< ", periodicRate: " << periodicRate
<< ", paymentRemaining: " << paymentRemaining
<< ", theoreticalPrincipalOutstanding: " << theoreticalPrincipalOutstanding
<< ", fullPaymentInterest: " << fullPaymentInterest
@@ -1304,34 +1298,6 @@ computePaymentComponents(
};
}
/* Thin overload of computePaymentComponents() that unwraps the tracked
* fields directly from the Loan ledger object. `periodicRate` is derived
* rather than stored, and `managementFeeRate` comes from the LoanBroker, not
* the Loan, so both remain explicit parameters. Kept separate from the
* value-based overload above, which is exercised directly by unit tests
* against simulated (non-ledger) loan states.
*/
PaymentComponents
computePaymentComponents(
Rules const& rules,
Asset const& asset,
SLE::ref loan,
Number const& periodicRate,
TenthBips16 managementFeeRate)
{
return computePaymentComponents(
rules,
asset,
loan->at(sfLoanScale),
loan->at(sfTotalValueOutstanding),
loan->at(sfPrincipalOutstanding),
loan->at(sfManagementFeeOutstanding),
loan->at(sfPeriodicPayment),
periodicRate,
loan->at(sfPaymentRemaining),
managementFeeRate);
}
/* Computes payment components for an overpayment scenario.
*
* An overpayment occurs when a borrower pays more than the scheduled periodic
@@ -1376,12 +1342,11 @@ computeOverpaymentComponents(
// This interest doesn't follow the normal amortization schedule - it's
// a one-time charge for paying early.
// Equation (20) and (21) from XLS-66 spec, Section A-2 Equation Glossary
auto const [roundedOverpaymentInterest, roundedOverpaymentManagementFee] =
roundAndSplitInterest(
asset,
tenthBipsOfValue(overpayment, overpaymentInterestRate),
managementFeeRate,
loanScale);
auto const [roundedOverpaymentInterest, roundedOverpaymentManagementFee] = [&]() {
auto const interest =
roundToAsset(asset, tenthBipsOfValue(overpayment, overpaymentInterestRate), loanScale);
return detail::computeInterestAndFeeParts(asset, interest, managementFeeRate, loanScale);
}();
auto const result = detail::ExtendedPaymentComponents{
// Build the payment components, after fees and penalty
@@ -1408,265 +1373,6 @@ computeOverpaymentComponents(
return result;
}
/* Derives the two rate values every make*Payment() helper needs: the
* broker's management fee rate, and the loan's periodic (per-payment-period)
* interest rate.
*/
std::pair<TenthBips16, Number>
loanRatesFor(SLE::const_ref loan, SLE::const_ref brokerSle)
{
TenthBips16 const managementFeeRate{brokerSle->at(sfManagementFeeRate)};
TenthBips32 const interestRate{loan->at(sfInterestRate)};
Number const periodicRate = loanPeriodicRate(interestRate, loan->at(sfPaymentInterval));
XRPL_ASSERT(interestRate == 0 || periodicRate > 0, "xrpl::detail::loanRatesFor : valid rate");
return {managementFeeRate, periodicRate};
}
/* Handles a full (early payoff) payment. Implements the "full payment"
* branch of the make_payment function from the XLS-66 spec, Section
* 3.2.4.4.
*/
std::expected<LoanPaymentParts, TER>
makeFullPayment(
Asset const& asset,
ApplyView& view,
SLE::ref loan,
SLE::const_ref brokerSle,
STAmount const& amount,
beast::Journal j)
{
auto const [managementFeeRate, periodicRate] = loanRatesFor(loan, brokerSle);
auto const fullPaymentComponents =
computeFullPayment(asset, view, loan, periodicRate, amount, managementFeeRate, j);
// computeFullPayment only ever fails with a genuine error TER (never
// tesSUCCESS), so there is no separate "no-op" outcome to handle here.
if (fullPaymentComponents.has_value())
return doPayment(*fullPaymentComponents, loan);
return std::unexpected(fullPaymentComponents.error());
}
/* Handles a late payment (past due date, with the late-payment flag set).
* Implements the "late payment" branch of the make_payment function from
* the XLS-66 spec, Section 3.2.4.4.
*/
std::expected<LoanPaymentParts, TER>
makeLatePayment(
Asset const& asset,
ApplyView const& view,
SLE::ref loan,
SLE::const_ref brokerSle,
STAmount const& amount,
beast::Journal j)
{
auto const [managementFeeRate, periodicRate] = loanRatesFor(loan, brokerSle);
Number const serviceFee = loan->at(sfLoanServiceFee);
ExtendedPaymentComponents const periodic{
computePaymentComponents(view.rules(), asset, loan, periodicRate, managementFeeRate),
serviceFee};
XRPL_ASSERT_PARTS(
periodic.trackedPrincipalDelta >= 0,
"xrpl::detail::makeLatePayment",
"regular payment valid principal");
auto const latePaymentComponents =
computeLatePayment(asset, view, loan, periodic, amount, managementFeeRate, j);
// computeLatePayment only ever fails with a genuine error TER (never
// tesSUCCESS), so there is no separate "no-op" outcome to handle here.
if (latePaymentComponents.has_value())
return doPayment(*latePaymentComponents, loan);
return std::unexpected(latePaymentComponents.error());
}
/* Handles regular scheduled payments, including an optional overpayment tail.
* Implements the "regular" and "overpayment" branches of the make_payment
* function from the XLS-66 spec, Section 3.2.4.4.
*/
std::expected<LoanPaymentParts, TER>
makeRegularPayment(
Asset const& asset,
ApplyView const& view,
SLE::ref loan,
SLE::const_ref brokerSle,
STAmount const& amount,
LoanPaymentType const paymentType,
beast::Journal j)
{
using namespace Lending;
XRPL_ASSERT_PARTS(
paymentType == LoanPaymentType::Regular || paymentType == LoanPaymentType::Overpayment,
"xrpl::detail::makeRegularPayment",
"regular payment type");
auto const [managementFeeRate, periodicRate] = loanRatesFor(loan, brokerSle);
std::int32_t const loanScale = loan->at(sfLoanScale);
Number const serviceFee = loan->at(sfLoanServiceFee);
ExtendedPaymentComponents periodic{
computePaymentComponents(view.rules(), asset, loan, periodicRate, managementFeeRate),
serviceFee};
XRPL_ASSERT_PARTS(
periodic.trackedPrincipalDelta >= 0,
"xrpl::detail::makeRegularPayment",
"regular payment valid principal");
// Keep a running total of the actual parts paid
LoanPaymentParts totalParts;
Number totalPaid = kNumZero;
std::size_t numPayments = 0;
// Cached here (rather than re-looking up loan->at(sfPaymentRemaining) at each use) since it's
// read multiple times below. It's a write-through proxy, so it still reflects doPayment's
// mutations each iteration.
auto paymentRemainingProxy = loan->at(sfPaymentRemaining);
while ((amount >= (totalPaid + periodic.totalDue)) && paymentRemainingProxy > 0 &&
numPayments < kLoanMaximumPaymentsPerTransaction)
{
// Try to make more payments
XRPL_ASSERT_PARTS(
periodic.trackedPrincipalDelta >= 0,
"xrpl::detail::makeRegularPayment",
"payment pays non-negative principal");
totalPaid += periodic.totalDue;
totalParts += doPayment(periodic, loan);
++numPayments;
XRPL_ASSERT_PARTS(
(periodic.specialCase == PaymentSpecialCase::Final) == (paymentRemainingProxy == 0),
"xrpl::detail::makeRegularPayment",
"final payment is the final payment");
// Don't compute the next payment if this was the last payment
if (periodic.specialCase == PaymentSpecialCase::Final)
break;
periodic = ExtendedPaymentComponents{
computePaymentComponents(view.rules(), asset, loan, periodicRate, managementFeeRate),
serviceFee};
}
if (numPayments == 0)
{
JLOG(j.warn()) << "Regular loan payment amount is insufficient. Due: " << periodic.totalDue
<< ", paid: " << amount;
return std::unexpected(tecINSUFFICIENT_PAYMENT);
}
XRPL_ASSERT_PARTS(
totalParts.principalPaid + totalParts.interestPaid + totalParts.feePaid == totalPaid,
"xrpl::detail::makeRegularPayment",
"payment parts add up");
XRPL_ASSERT_PARTS(
totalParts.valueChange == 0, "xrpl::detail::makeRegularPayment", "no value change");
// -------------------------------------------------------------
// overpayment handling
//
// If the "fixCleanup3_1_3" amendment is enabled, truncate "amount",
// at the loan scale. If the raw value is used, the overpayment
// amount could be meaningless dust. Trying to process such a small
// amount will, at best, waste time when all the result values round
// to zero. At worst, it can cause logical errors with tiny amounts
// of interest that don't add up correctly.
auto const roundedAmount = view.rules().enabled(fixCleanup3_1_3)
? roundToAsset(asset, amount, loanScale, Number::RoundingMode::TowardsZero)
: amount;
bool const overpaymentSupported =
paymentType == LoanPaymentType::Overpayment && loan->isFlag(lsfLoanOverpayment);
bool const overpaymentAllowed = //
paymentRemainingProxy > 0 && //
totalPaid < roundedAmount && //
numPayments < kLoanMaximumPaymentsPerTransaction;
if (overpaymentSupported && overpaymentAllowed)
{
TenthBips32 const overpaymentInterestRate{loan->at(sfOverpaymentInterestRate)};
TenthBips32 const overpaymentFeeRate{loan->at(sfOverpaymentFee)};
// It shouldn't be possible for the overpayment to be greater than
// totalValueOutstanding, because that would have been processed as
// another normal payment. But cap it just in case.
Number const overpaymentRaw =
std::min(roundedAmount - totalPaid, *loan->at(sfTotalValueOutstanding));
bool const fixEnabled = view.rules().enabled(fixCleanup3_2_0);
Number const overpayment = fixEnabled
? roundToAsset(asset, overpaymentRaw, loanScale, Number::RoundingMode::Downward)
: overpaymentRaw;
// Post-amendment, the rounded overpayment can be zero; pre-amendment
// it's always positive given the surrounding guards.
if (!fixEnabled || overpayment > 0)
{
ExtendedPaymentComponents const overpaymentComponents = computeOverpaymentComponents(
view.rules(),
asset,
loanScale,
overpayment,
overpaymentInterestRate,
overpaymentFeeRate,
managementFeeRate);
// Don't process an overpayment if the whole amount (or more!)
// gets eaten by fees and interest.
if (overpaymentComponents.trackedPrincipalDelta > 0)
{
XRPL_ASSERT_PARTS(
overpaymentComponents.untrackedInterest >= beast::kZero,
"xrpl::detail::makeRegularPayment",
"overpayment penalty did not reduce value of loan");
if (auto const overResult = doOverpayment(
view.rules(),
asset,
loanScale,
overpaymentComponents,
loan,
periodicRate,
managementFeeRate,
j))
{
totalParts += *overResult;
}
else if (overResult.error())
{
// error() will be the TER returned if a payment is not
// made. It will only evaluate to true if it's unsuccessful.
// Otherwise, tesSUCCESS means nothing was done, so
// continue.
return std::unexpected(overResult.error());
}
}
}
}
// Check the final results are rounded, to double-check that the
// intermediate steps were rounded.
XRPL_ASSERT(
isRounded(asset, totalParts.principalPaid, loanScale) &&
totalParts.principalPaid >= beast::kZero,
"xrpl::detail::makeRegularPayment : total principal paid is valid");
XRPL_ASSERT(
isRounded(asset, totalParts.interestPaid, loanScale) &&
totalParts.interestPaid >= beast::kZero,
"xrpl::detail::makeRegularPayment : total interest paid is valid");
XRPL_ASSERT(
isRounded(asset, totalParts.valueChange, loanScale),
"xrpl::detail::makeRegularPayment : loan value change is valid");
XRPL_ASSERT(
isRounded(asset, totalParts.feePaid, loanScale) && totalParts.feePaid >= beast::kZero,
"xrpl::detail::makeRegularPayment : fee paid is valid");
return totalParts;
}
} // namespace detail
detail::LoanStateDeltas
@@ -1926,10 +1632,8 @@ constructLoanState(
}
LoanState
constructLoanState(SLE::const_ref loan)
constructRoundedLoanState(SLE::const_ref loan)
{
XRPL_ASSERT(loan && loan->getType() == ltLOAN, "xrpl::constructLoanState : valid loan SLE");
return constructLoanState(
loan->at(sfTotalValueOutstanding),
loan->at(sfPrincipalOutstanding),
@@ -2086,7 +1790,12 @@ loanMakePayment(
LoanPaymentType const paymentType,
beast::Journal j)
{
if (loan->at(sfPaymentRemaining) == 0 || loan->at(sfPrincipalOutstanding) == 0)
using namespace Lending;
auto principalOutstandingProxy = loan->at(sfPrincipalOutstanding);
auto paymentRemainingProxy = loan->at(sfPaymentRemaining);
if (paymentRemainingProxy == 0 || principalOutstandingProxy == 0)
{
// Loan complete this is already checked in LoanPay::preclaim()
// LCOV_EXCL_START
@@ -2095,6 +1804,9 @@ loanMakePayment(
// LCOV_EXCL_STOP
}
auto totalValueOutstandingProxy = loan->at(sfTotalValueOutstanding);
auto managementFeeOutstandingProxy = loan->at(sfManagementFeeOutstanding);
// Next payment due date must be set unless the loan is complete
auto nextDueDateProxy = loan->at(sfNextPaymentDueDate);
if (*nextDueDateProxy == 0)
@@ -2103,8 +1815,26 @@ loanMakePayment(
return std::unexpected(tecINTERNAL);
}
XRPL_ASSERT(
*loan->at(sfTotalValueOutstanding) > 0, "xrpl::loanMakePayment : valid total value");
std::int32_t const loanScale = loan->at(sfLoanScale);
TenthBips32 const interestRate{loan->at(sfInterestRate)};
Number const serviceFee = loan->at(sfLoanServiceFee);
TenthBips16 const managementFeeRate{brokerSle->at(sfManagementFeeRate)};
Number const periodicPayment = loan->at(sfPeriodicPayment);
auto prevPaymentDateProxy = loan->at(sfPreviousPaymentDueDate);
std::uint32_t const startDate = loan->at(sfStartDate);
std::uint32_t const paymentInterval = loan->at(sfPaymentInterval);
// Compute the periodic rate that will be used for calculations
// throughout
Number const periodicRate = loanPeriodicRate(interestRate, paymentInterval);
XRPL_ASSERT(interestRate == 0 || periodicRate > 0, "xrpl::loanMakePayment : valid rate");
XRPL_ASSERT(*totalValueOutstandingProxy > 0, "xrpl::loanMakePayment : valid total value");
view.update(loan);
@@ -2114,29 +1844,311 @@ loanMakePayment(
{
// If the payment is late, and the late flag was not set, it's not
// valid
JLOG(j.warn()) << "Loan payment is overdue. Use the tfLoanLatePayment transaction flag to "
"make a late payment. Loan was created on "
<< loan->at(sfStartDate) << ", prev payment due date is "
<< loan->at(sfPreviousPaymentDueDate) << ", next payment due date is "
<< nextDueDateProxy << ", ledger time is "
JLOG(j.warn()) << "Loan payment is overdue. Use the tfLoanLatePayment "
"transaction "
"flag to make a late payment. Loan was created on "
<< startDate << ", prev payment due date is " << prevPaymentDateProxy
<< ", next payment due date is " << nextDueDateProxy << ", ledger time is "
<< view.parentCloseTime().time_since_epoch().count();
return std::unexpected(tecEXPIRED);
}
switch (paymentType)
// -------------------------------------------------------------
// full payment handling
if (paymentType == LoanPaymentType::Full)
{
case LoanPaymentType::Full:
return detail::makeFullPayment(asset, view, loan, brokerSle, amount, j);
case LoanPaymentType::Late:
return detail::makeLatePayment(asset, view, loan, brokerSle, amount, j);
case LoanPaymentType::Regular:
case LoanPaymentType::Overpayment:
return detail::makeRegularPayment(asset, view, loan, brokerSle, amount, paymentType, j);
TenthBips32 const closeInterestRate{loan->at(sfCloseInterestRate)};
Number const closePaymentFee = roundToAsset(asset, loan->at(sfClosePaymentFee), loanScale);
LoanState const roundedLoanState = constructLoanState(
totalValueOutstandingProxy, principalOutstandingProxy, managementFeeOutstandingProxy);
auto const fullPaymentComponents = detail::computeFullPayment(
asset,
view,
principalOutstandingProxy,
managementFeeOutstandingProxy,
periodicPayment,
paymentRemainingProxy,
prevPaymentDateProxy,
startDate,
paymentInterval,
closeInterestRate,
loanScale,
roundedLoanState.interestDue,
periodicRate,
closePaymentFee,
amount,
managementFeeRate,
j);
if (fullPaymentComponents.has_value())
{
return doPayment(
*fullPaymentComponents,
totalValueOutstandingProxy,
principalOutstandingProxy,
managementFeeOutstandingProxy,
paymentRemainingProxy,
prevPaymentDateProxy,
nextDueDateProxy,
paymentInterval);
}
if (fullPaymentComponents.error())
{
// error() will be the TER returned if a payment is not made. It
// will only evaluate to true if it's unsuccessful. Otherwise,
// tesSUCCESS means nothing was done, so continue.
return std::unexpected(fullPaymentComponents.error());
}
// LCOV_EXCL_START
UNREACHABLE("xrpl::loanMakePayment : invalid full payment result");
JLOG(j.error()) << "Full payment computation failed unexpectedly.";
return std::unexpected(tecINTERNAL);
// LCOV_EXCL_STOP
}
// LCOV_EXCL_START
UNREACHABLE("xrpl::loanMakePayment : invalid payment type");
return std::unexpected(tecINTERNAL);
// LCOV_EXCL_STOP
// -------------------------------------------------------------
// compute the periodic payment info that will be needed whether the
// payment is late or regular
detail::ExtendedPaymentComponents periodic{
detail::computePaymentComponents(
view.rules(),
asset,
loanScale,
totalValueOutstandingProxy,
principalOutstandingProxy,
managementFeeOutstandingProxy,
periodicPayment,
periodicRate,
paymentRemainingProxy,
managementFeeRate),
serviceFee};
XRPL_ASSERT_PARTS(
periodic.trackedPrincipalDelta >= 0,
"xrpl::loanMakePayment",
"regular payment valid principal");
// -------------------------------------------------------------
// late payment handling
if (paymentType == LoanPaymentType::Late)
{
TenthBips32 const lateInterestRate{loan->at(sfLateInterestRate)};
Number const latePaymentFee = loan->at(sfLatePaymentFee);
auto const latePaymentComponents = detail::computeLatePayment(
asset,
view,
principalOutstandingProxy,
nextDueDateProxy,
periodic,
lateInterestRate,
loanScale,
latePaymentFee,
amount,
managementFeeRate,
j);
if (latePaymentComponents.has_value())
{
return doPayment(
*latePaymentComponents,
totalValueOutstandingProxy,
principalOutstandingProxy,
managementFeeOutstandingProxy,
paymentRemainingProxy,
prevPaymentDateProxy,
nextDueDateProxy,
paymentInterval);
}
if (latePaymentComponents.error())
{
// error() will be the TER returned if a payment is not made. It
// will only evaluate to true if it's unsuccessful.
return std::unexpected(latePaymentComponents.error());
}
// LCOV_EXCL_START
UNREACHABLE("xrpl::loanMakePayment : invalid late payment result");
JLOG(j.error()) << "Late payment computation failed unexpectedly.";
return std::unexpected(tecINTERNAL);
// LCOV_EXCL_STOP
}
// -------------------------------------------------------------
// regular periodic payment handling
XRPL_ASSERT_PARTS(
paymentType == LoanPaymentType::Regular || paymentType == LoanPaymentType::Overpayment,
"xrpl::loanMakePayment",
"regular payment type");
// Keep a running total of the actual parts paid
LoanPaymentParts totalParts;
Number totalPaid;
std::size_t numPayments = 0;
while ((amount >= (totalPaid + periodic.totalDue)) && paymentRemainingProxy > 0 &&
numPayments < kLoanMaximumPaymentsPerTransaction)
{
// Try to make more payments
XRPL_ASSERT_PARTS(
periodic.trackedPrincipalDelta >= 0,
"xrpl::loanMakePayment",
"payment pays non-negative principal");
totalPaid += periodic.totalDue;
totalParts += detail::doPayment(
periodic,
totalValueOutstandingProxy,
principalOutstandingProxy,
managementFeeOutstandingProxy,
paymentRemainingProxy,
prevPaymentDateProxy,
nextDueDateProxy,
paymentInterval);
++numPayments;
XRPL_ASSERT_PARTS(
(periodic.specialCase == detail::PaymentSpecialCase::Final) ==
(paymentRemainingProxy == 0),
"xrpl::loanMakePayment",
"final payment is the final payment");
// Don't compute the next payment if this was the last payment
if (periodic.specialCase == detail::PaymentSpecialCase::Final)
break;
periodic = detail::ExtendedPaymentComponents{
detail::computePaymentComponents(
view.rules(),
asset,
loanScale,
totalValueOutstandingProxy,
principalOutstandingProxy,
managementFeeOutstandingProxy,
periodicPayment,
periodicRate,
paymentRemainingProxy,
managementFeeRate),
serviceFee};
}
if (numPayments == 0)
{
JLOG(j.warn()) << "Regular loan payment amount is insufficient. Due: " << periodic.totalDue
<< ", paid: " << amount;
return std::unexpected(tecINSUFFICIENT_PAYMENT);
}
XRPL_ASSERT_PARTS(
totalParts.principalPaid + totalParts.interestPaid + totalParts.feePaid == totalPaid,
"xrpl::loanMakePayment",
"payment parts add up");
XRPL_ASSERT_PARTS(totalParts.valueChange == 0, "xrpl::loanMakePayment", "no value change");
// -------------------------------------------------------------
// overpayment handling
//
// If the "fixCleanup3_1_3" amendment is enabled, truncate "amount",
// at the loan scale. If the raw value is used, the overpayment
// amount could be meaningless dust. Trying to process such a small
// amount will, at best, waste time when all the result values round
// to zero. At worst, it can cause logical errors with tiny amounts
// of interest that don't add up correctly.
auto const roundedAmount = view.rules().enabled(fixCleanup3_1_3)
? roundToAsset(asset, amount, loanScale, Number::RoundingMode::TowardsZero)
: amount;
if (paymentType == LoanPaymentType::Overpayment && loan->isFlag(lsfLoanOverpayment) &&
paymentRemainingProxy > 0 && totalPaid < roundedAmount &&
numPayments < kLoanMaximumPaymentsPerTransaction)
{
TenthBips32 const overpaymentInterestRate{loan->at(sfOverpaymentInterestRate)};
TenthBips32 const overpaymentFeeRate{loan->at(sfOverpaymentFee)};
// It shouldn't be possible for the overpayment to be greater than
// totalValueOutstanding, because that would have been processed as
// another normal payment. But cap it just in case.
Number const overpaymentRaw =
std::min(roundedAmount - totalPaid, *totalValueOutstandingProxy);
bool const fixEnabled = view.rules().enabled(fixCleanup3_2_0);
Number const overpayment = fixEnabled
? roundToAsset(asset, overpaymentRaw, loanScale, Number::RoundingMode::Downward)
: overpaymentRaw;
// Post-amendment, the rounded overpayment can be zero; pre-amendment
// it's always positive given the surrounding guards.
if (!fixEnabled || overpayment > 0)
{
detail::ExtendedPaymentComponents const overpaymentComponents =
detail::computeOverpaymentComponents(
view.rules(),
asset,
loanScale,
overpayment,
overpaymentInterestRate,
overpaymentFeeRate,
managementFeeRate);
// Don't process an overpayment if the whole amount (or more!)
// gets eaten by fees and interest.
if (overpaymentComponents.trackedPrincipalDelta > 0)
{
XRPL_ASSERT_PARTS(
overpaymentComponents.untrackedInterest >= beast::kZero,
"xrpl::loanMakePayment",
"overpayment penalty did not reduce value of loan");
// Can't just use `periodicPayment` here, because it might
// change
auto periodicPaymentProxy = loan->at(sfPeriodicPayment);
if (auto const overResult = detail::doOverpayment(
view.rules(),
asset,
loanScale,
overpaymentComponents,
totalValueOutstandingProxy,
principalOutstandingProxy,
managementFeeOutstandingProxy,
periodicPaymentProxy,
periodicRate,
paymentRemainingProxy,
managementFeeRate,
j))
{
totalParts += *overResult;
}
else if (overResult.error())
{
// error() will be the TER returned if a payment is not
// made. It will only evaluate to true if it's unsuccessful.
// Otherwise, tesSUCCESS means nothing was done, so
// continue.
return std::unexpected(overResult.error());
}
}
}
}
// Check the final results are rounded, to double-check that the
// intermediate steps were rounded.
XRPL_ASSERT(
isRounded(asset, totalParts.principalPaid, loanScale) &&
totalParts.principalPaid >= beast::kZero,
"xrpl::loanMakePayment : total principal paid is valid");
XRPL_ASSERT(
isRounded(asset, totalParts.interestPaid, loanScale) &&
totalParts.interestPaid >= beast::kZero,
"xrpl::loanMakePayment : total interest paid is valid");
XRPL_ASSERT(
isRounded(asset, totalParts.valueChange, loanScale),
"xrpl::loanMakePayment : loan value change is valid");
XRPL_ASSERT(
isRounded(asset, totalParts.feePaid, loanScale) && totalParts.feePaid >= beast::kZero,
"xrpl::loanMakePayment : fee paid is valid");
return totalParts;
}
} // namespace xrpl

View File

@@ -1,135 +0,0 @@
#include <xrpl/peerfinder/Config.h>
#include <xrpl/beast/utility/PropertyStream.h>
#include <algorithm>
#include <cstddef>
#include <cstdint>
#include <stdexcept>
namespace xrpl::PeerFinder {
std::size_t
Config::calcOutPeers() const
{
return std::max(
((maxPeers * Tuning::kOutPercent) + 50) / 100, std::size_t(Tuning::kMinOutCount));
}
void
Config::applyTuning()
{
if (ipLimit == 0)
{
// Unless a limit is explicitly set, we allow between
// 2 and 5 connections from non RFC-1918 "private"
// IP addresses.
ipLimit = 2;
if (inPeers > Tuning::kDefaultMaxPeers)
ipLimit += std::min(5, static_cast<int>(inPeers / Tuning::kDefaultMaxPeers));
}
// We don't allow a single IP to consume all incoming slots,
// unless we only have one incoming slot available.
ipLimit = std::max(1, std::min(ipLimit, static_cast<int>(inPeers / 2)));
}
void
Config::onWrite(beast::PropertyStream::Map& map) const
{
map["max_peers"] = maxPeers;
map["out_peers"] = outPeers;
map["want_incoming"] = wantIncoming;
map["auto_connect"] = autoConnect;
map["port"] = listeningPort;
map["features"] = features;
map["ip_limit"] = ipLimit;
map["verify_endpoints"] = verifyEndpoints;
}
Config
Config::makeConfig(
bool peerPrivate,
bool standalone,
PeerLimitConfig const& limits,
std::uint16_t port,
bool validationPublicKey,
int ipLimit,
bool verifyEndpoints)
{
PeerFinder::Config config;
if (!limits.maxPeers)
{
if (limits.inPeers && !limits.outPeers)
throw std::runtime_error("Both inbound and outbound peer limits must be configured");
if (limits.outPeers && !limits.inPeers)
throw std::runtime_error("Both inbound and outbound peer limits must be configured");
if (limits.inPeers && *limits.inPeers > 1000)
throw std::runtime_error("Inbound peer limit must be less than or equal to 1000");
if (limits.outPeers && (*limits.outPeers < 10 || *limits.outPeers > 1000))
throw std::runtime_error("Outbound peer limit must be in the range 10-1000");
}
config.peerPrivate = peerPrivate;
// Servers with peer privacy don't want to allow incoming connections
config.wantIncoming = (!config.peerPrivate) && (port != 0);
if (limits.maxPeers || (!limits.inPeers && !limits.outPeers))
{
if (limits.maxPeers && *limits.maxPeers != 0)
config.maxPeers = *limits.maxPeers;
config.maxPeers = std::max<std::size_t>(config.maxPeers, Tuning::kMinOutCount);
config.outPeers = config.calcOutPeers();
// Calculate the number of outbound peers we want. If we dont want
// or can't accept incoming, this will simply be equal to maxPeers.
if (!config.wantIncoming)
config.outPeers = config.maxPeers;
// Calculate the largest number of inbound connections we could
// take.
if (config.maxPeers >= config.outPeers)
{
config.inPeers = config.maxPeers - config.outPeers;
}
else
{
config.inPeers = 0;
}
}
else
{
config.outPeers = *limits.outPeers;
config.inPeers = *limits.inPeers;
config.maxPeers = 0;
}
// This will cause servers configured as validators to request that
// peers they connect to never report their IP address. We set this
// after we set the 'wantIncoming' because we want a "soft" version
// of peer privacy unless the operator explicitly asks for it.
if (validationPublicKey)
config.peerPrivate = true;
// if it's a private peer or we are running as standalone
// automatic connections would defeat the purpose.
config.autoConnect = !standalone && !peerPrivate;
config.listeningPort = port;
config.features = "";
config.ipLimit = ipLimit;
config.verifyEndpoints = verifyEndpoints;
// Enforce business rules
config.applyTuning();
return config;
}
} // namespace xrpl::PeerFinder

Some files were not shown because too many files have changed in this diff Show More