ARCHIVE/xahaud
3
0
Fork 0
mirror of https://github.com/Xahau/xahaud.git synced 2026-06-03 16:56:36 +00:00
Code Issues Packages Projects Releases Wiki Activity

chore: Run prettier on all files (#5657)

Browse Source
This commit is contained in:
Mayukha Vadari
2025-08-11 12:15:42 -04:00
committed by tequ
parent f512fbec5d
commit 6a18c92c5c
45 changed files with 5608 additions and 5529 deletions
Download Patch File Download Diff File Expand all files Collapse all files

81
.clang-format
View File

@@ -1,5 +1,5 @@
---
Language: Cpp
Language: Cpp
AccessModifierOffset: -4
AlignAfterOpenBracket: AlwaysBreak
AlignConsecutiveAssignments: false
@@ -19,52 +19,52 @@ AlwaysBreakTemplateDeclarations: true
BinPackArguments: false
BinPackParameters: false
BraceWrapping:
AfterClass: true
AfterClass: true
AfterControlStatement: true
AfterEnum: false
AfterFunction: true
AfterNamespace: false
AfterEnum: false
AfterFunction: true
AfterNamespace: false
AfterObjCDeclaration: true
AfterStruct: true
AfterUnion: true
BeforeCatch: true
BeforeElse: true
IndentBraces: false
AfterStruct: true
AfterUnion: true
BeforeCatch: true
BeforeElse: true
IndentBraces: false
BreakBeforeBinaryOperators: false
BreakBeforeBraces: Custom
BreakBeforeTernaryOperators: true
BreakConstructorInitializersBeforeComma: true
ColumnLimit: 80
CommentPragmas: '^ IWYU pragma:'
ColumnLimit: 80
CommentPragmas: "^ IWYU pragma:"
ConstructorInitializerAllOnOneLineOrOnePerLine: true
ConstructorInitializerIndentWidth: 4
ContinuationIndentWidth: 4
Cpp11BracedListStyle: true
DerivePointerAlignment: false
DisableFormat: false
DisableFormat: false
ExperimentalAutoDetectBinPacking: false
ForEachMacros: [ Q_FOREACH, BOOST_FOREACH ]
IncludeBlocks: Regroup
ForEachMacros: [Q_FOREACH, BOOST_FOREACH]
IncludeBlocks: Regroup
IncludeCategories:
- Regex: '^<(test)/'
Priority: 0
- Regex: '^<(xrpld)/'
Priority: 1
- Regex: '^<(xrpl)/'
Priority: 2
- Regex: '^<(boost)/'
Priority: 3
- Regex: '^.*/'
Priority: 4
- Regex: '^.*\.h'
Priority: 5
- Regex: '.*'
Priority: 6
IncludeIsMainRegex: '$'
- Regex: "^<(test)/"
Priority: 0
- Regex: "^<(xrpld)/"
Priority: 1
- Regex: "^<(xrpl)/"
Priority: 2
- Regex: "^<(boost)/"
Priority: 3
- Regex: "^.*/"
Priority: 4
- Regex: '^.*\.h'
Priority: 5
- Regex: ".*"
Priority: 6
IncludeIsMainRegex: "$"
IndentCaseLabels: true
IndentFunctionDeclarationAfterType: false
IndentRequiresClause: true
IndentWidth: 4
IndentWidth: 4
IndentWrappedFunctionNames: false
KeepEmptyLinesAtTheStartOfBlocks: false
MaxEmptyLinesToKeep: 1
@@ -78,20 +78,25 @@ PenaltyBreakString: 1000
PenaltyExcessCharacter: 1000000
PenaltyReturnTypeOnItsOwnLine: 200
PointerAlignment: Left
ReflowComments: true
ReflowComments: true
RequiresClausePosition: OwnLine
SortIncludes: true
SortIncludes: true
SpaceAfterCStyleCast: false
SpaceBeforeAssignmentOperators: true
SpaceBeforeParens: ControlStatements
SpaceInEmptyParentheses: false
SpacesBeforeTrailingComments: 2
SpacesInAngles: false
SpacesInAngles: false
SpacesInContainerLiterals: true
SpacesInCStyleCastParentheses: false
SpacesInParentheses: false
SpacesInSquareBrackets: false
Standard: Cpp11
TabWidth: 8
UseTab: Never
QualifierAlignment: Right
Standard: Cpp11
TabWidth: 8
UseTab: Never
QualifierAlignment: Right
---
Language: JavaScript
---
Language: Json
IndentWidth: 2

20
.codecov.yml
View File

@@ -4,3 +4,23 @@ coverage:
default:
target: 60%
threshold: 2%
patch:
default:
target: auto
threshold: 2%
changes: false
github_checks:
annotations: true
parsers:
cobertura:
partials_as_hits: true
handle_missing_conditions: true
slack_app: false
ignore:
- "src/test/"
- "include/xrpl/beast/test/"
- "include/xrpl/beast/unit_test/"

13
.github/ISSUE_TEMPLATE/bug_report.md vendored
View File

@@ -2,30 +2,35 @@
name: Bug Report
about: Create a report to help us improve rippled
title: "[Title with short description] (Version: [rippled version])"
labels: ''
assignees: ''
labels: ""
assignees: ""
---
<!-- Please search existing issues to avoid creating duplicates.-->
## Issue Description
<!--Provide a summary for your issue/bug.-->
## Steps to Reproduce
<!--List in detail the exact steps to reproduce the unexpected behavior of the software.-->
## Expected Result
<!--Explain in detail what behavior you expected to happen.-->
## Actual Result
<!--Explain in detail what behavior actually happened.-->
## Environment
<!--Please describe your environment setup (such as Ubuntu 18.04 with Boost 1.70).-->
<!-- If you are using a formal release, please use the version returned by './rippled --version' as the version number-->
<!-- If you are working off of develop, please add the git hash via 'git rev-parse HEAD'-->
## Supporting Files
<!--If you have supporting files such as a log, feel free to post a link here using Github Gist.-->
<!--Consider adding configuration files with private information removed via Github Gist. -->

8
.github/ISSUE_TEMPLATE/feature_request.md vendored
View File

@@ -3,19 +3,23 @@ name: Feature Request
about: Suggest a new feature for the rippled project
title: "[Title with short description] (Version: [rippled version])"
labels: Feature Request
assignees: ''
assignees: ""
---
<!-- Please search existing issues to avoid creating duplicates.-->
## Summary
<!-- Provide a summary to the feature request-->
## Motivation
<!-- Why do we need this feature?-->
## Solution
<!-- What is the solution?-->
## Paths Not Taken
<!-- What other alternatives have been considered?-->

8
.pre-commit-config.yaml
View File

@@ -1,6 +1,6 @@
# .pre-commit-config.yaml
repos:
- repo: https://github.com/pre-commit/mirrors-clang-format
rev: v18.1.8
hooks:
- id: clang-format
- repo: https://github.com/pre-commit/mirrors-clang-format
rev: v18.1.8
hooks:
- id: clang-format

113
BUILD.md
View File

@@ -84,9 +84,9 @@ If you are unfamiliar with Conan, then please read [this crash course](./docs/bu
You'll need at least one Conan profile:
```
conan profile detect --force
```
```
conan profile detect --force
```
Update the compiler settings:
@@ -215,60 +215,60 @@ It patches their CMake to correctly import its dependencies.
2. Use conan to generate CMake files for every configuration you want to build:
```
conan install .. --output-folder . --build missing --settings build_type=Release
conan install .. --output-folder . --build missing --settings build_type=Debug
```
```
conan install .. --output-folder . --build missing --settings build_type=Release
conan install .. --output-folder . --build missing --settings build_type=Debug
```
To build Debug, in the next step, be sure to set `-DCMAKE_BUILD_TYPE=Debug`
To build Debug, in the next step, be sure to set `-DCMAKE_BUILD_TYPE=Debug`
For a single-configuration generator, e.g. `Unix Makefiles` or `Ninja`,
you only need to run this command once.
For a multi-configuration generator, e.g. `Visual Studio`, you may want to
run it more than once.
For a single-configuration generator, e.g. `Unix Makefiles` or `Ninja`,
you only need to run this command once.
For a multi-configuration generator, e.g. `Visual Studio`, you may want to
run it more than once.
Each of these commands should also have a different `build_type` setting.
A second command with the same `build_type` setting will overwrite the files
generated by the first. You can pass the build type on the command line with
`--settings build_type=$BUILD_TYPE` or in the profile itself,
under the section `[settings]` with the key `build_type`.
Each of these commands should also have a different `build_type` setting.
A second command with the same `build_type` setting will overwrite the files
generated by the first. You can pass the build type on the command line with
`--settings build_type=$BUILD_TYPE` or in the profile itself,
under the section `[settings]` with the key `build_type`.
If you are using a Microsoft Visual C++ compiler,
then you will need to ensure consistency between the `build_type` setting
and the `compiler.runtime` setting.
If you are using a Microsoft Visual C++ compiler,
then you will need to ensure consistency between the `build_type` setting
and the `compiler.runtime` setting.
When `build_type` is `Release`, `compiler.runtime` should be `MT`.
When `build_type` is `Release`, `compiler.runtime` should be `MT`.
When `build_type` is `Debug`, `compiler.runtime` should be `MTd`.
When `build_type` is `Debug`, `compiler.runtime` should be `MTd`.
```
conan install .. --output-folder . --build missing --settings build_type=Release --settings compiler.runtime=MT
conan install .. --output-folder . --build missing --settings build_type=Debug --settings compiler.runtime=MTd
```
```
conan install .. --output-folder . --build missing --settings build_type=Release --settings compiler.runtime=MT
conan install .. --output-folder . --build missing --settings build_type=Debug --settings compiler.runtime=MTd
```
3. Configure CMake and pass the toolchain file generated by Conan, located at
`$OUTPUT_FOLDER/build/generators/conan_toolchain.cmake`.
Single-config generators:
Single-config generators:
Pass the CMake variable [`CMAKE_BUILD_TYPE`][build_type]
and make sure it matches the one of the `build_type` settings
you chose in the previous step.
Pass the CMake variable [`CMAKE_BUILD_TYPE`][build_type]
and make sure it matches the one of the `build_type` settings
you chose in the previous step.
For example, to build Debug, in the next command, replace "Release" with "Debug"
For example, to build Debug, in the next command, replace "Release" with "Debug"
```
cmake -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake -DCMAKE_BUILD_TYPE=Release -Dxrpld=ON -Dtests=ON ..
```
```
cmake -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake -DCMAKE_BUILD_TYPE=Release -Dxrpld=ON -Dtests=ON ..
```
Multi-config generators:
Multi-config generators:
```
cmake -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake -Dxrpld=ON -Dtests=ON ..
```
```
cmake -DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake -Dxrpld=ON -Dtests=ON ..
```
**Note:** You can pass build options for `xahaud` in this step.
**Note:** You can pass build options for `xahaud` in this step.
4. Build `xahaud`.
@@ -308,7 +308,6 @@ It patches their CMake to correctly import its dependencies.
The location of `xahaud` in your build directory depends on your CMake
generator. Pass `--help` to see the rest of the command line options.
## Coverage report
The coverage report is intended for developers using compilers GCC
@@ -348,7 +347,7 @@ variable in `cmake`. The specific command line used to run the `gcovr` tool will
displayed if the `CODE_COVERAGE_VERBOSE` variable is set.
By default, the code coverage tool runs parallel unit tests with `--unittest-jobs`
set to the number of available CPU cores. This may cause spurious test
set to the number of available CPU cores. This may cause spurious test
errors on Apple. Developers can override the number of unit test jobs with
the `coverage_test_parallelism` variable in `cmake`.
@@ -367,24 +366,24 @@ stored inside the build directory, as either of:
- file named `coverage.`_extension_ , with a suitable extension for the report format, or
- directory named `coverage`, with the `index.html` and other files inside, for the `html-details` or `html-nested` report formats.
## Options
| Option | Default Value | Description |
| --- | ---| ---|
| `assert` | OFF | Enable assertions.
| `coverage` | OFF | Prepare the coverage report. |
| `san` | N/A | Enable a sanitizer with Clang. Choices are `thread` and `address`. |
| `tests` | OFF | Build tests. |
| `unity` | ON | Configure a unity build. |
| `xrpld` | OFF | Build the xrpld (`rippled`) application, and not just the libxrpl library. |
| Option | Default Value | Description |
| ---------- | ------------- | -------------------------------------------------------------------------- |
| `assert` | OFF | Enable assertions. |
| `coverage` | OFF | Prepare the coverage report. |
| `san` | N/A | Enable a sanitizer with Clang. Choices are `thread` and `address`. |
| `tests` | OFF | Build tests. |
| `unity` | OFF | Configure a unity build. |
| `xrpld` | OFF | Build the xrpld (`rippled`) application, and not just the libxrpl library. |
| `werr` | OFF | Treat compilation warnings as errors |
| `wextra` | OFF | Enable additional compilation warnings |
[Unity builds][5] may be faster for the first build
(at the cost of much more memory) since they concatenate sources into fewer
translation units. Non-unity builds may be faster for incremental builds,
and can be helpful for detecting `#include` omissions.
## Troubleshooting
@@ -454,13 +453,13 @@ If you want to experiment with a new package, follow these steps:
1. Search for the package on [Conan Center](https://conan.io/center/).
2. Modify [`conanfile.py`](./conanfile.py):
- Add a version of the package to the `requires` property.
- Change any default options for the package by adding them to the
`default_options` property (with syntax `'$package:$option': $value`).
- Add a version of the package to the `requires` property.
- Change any default options for the package by adding them to the
`default_options` property (with syntax `'$package:$option': $value`).
3. Modify [`CMakeLists.txt`](./CMakeLists.txt):
- Add a call to `find_package($package REQUIRED)`.
- Link a library from the package to the target `ripple_libs`
(search for the existing call to `target_link_libraries(ripple_libs INTERFACE ...)`).
- Add a call to `find_package($package REQUIRED)`.
- Link a library from the package to the target `ripple_libs`
(search for the existing call to `target_link_libraries(ripple_libs INTERFACE ...)`).
4. Start coding! Don't forget to include whatever headers you need from the package.

68
Builds/levelization/README.md
View File

@@ -25,28 +25,28 @@ more dependencies listed later.
**tl;dr:** The modules listed first are more independent than the modules
listed later.
| Level / Tier | Module(s) |
|--------------|-----------------------------------------------|
| 01 | ripple/beast ripple/unity
| 02 | ripple/basics
| 03 | ripple/json ripple/crypto
| 04 | ripple/protocol
| 05 | ripple/core ripple/conditions ripple/consensus ripple/resource ripple/server
| 06 | ripple/peerfinder ripple/ledger ripple/nodestore ripple/net
| 07 | ripple/shamap ripple/overlay
| 08 | ripple/app
| 09 | ripple/rpc
| 10 | ripple/perflog
| 11 | test/jtx test/beast test/csf
| 12 | test/unit_test
| 13 | test/crypto test/conditions test/json test/resource test/shamap test/peerfinder test/basics test/overlay
| 14 | test
| 15 | test/net test/protocol test/ledger test/consensus test/core test/server test/nodestore
| 16 | test/rpc test/app
| Level / Tier | Module(s) |
| ------------ | -------------------------------------------------------------------------------------------------------- |
| 01 | ripple/beast ripple/unity |
| 02 | ripple/basics |
| 03 | ripple/json ripple/crypto |
| 04 | ripple/protocol |
| 05 | ripple/core ripple/conditions ripple/consensus ripple/resource ripple/server |
| 06 | ripple/peerfinder ripple/ledger ripple/nodestore ripple/net |
| 07 | ripple/shamap ripple/overlay |
| 08 | ripple/app |
| 09 | ripple/rpc |
| 10 | ripple/perflog |
| 11 | test/jtx test/beast test/csf |
| 12 | test/unit_test |
| 13 | test/crypto test/conditions test/json test/resource test/shamap test/peerfinder test/basics test/overlay |
| 14 | test |
| 15 | test/net test/protocol test/ledger test/consensus test/core test/server test/nodestore |
| 16 | test/rpc test/app |
(Note that `test` levelization is *much* less important and *much* less
(Note that `test` levelization is _much_ less important and _much_ less
strictly enforced than `ripple` levelization, other than the requirement
that `test` code should *never* be included in `ripple` code.)
that `test` code should _never_ be included in `ripple` code.)
## Validation
@@ -59,48 +59,48 @@ the rippled source. The only caveat is that it runs much slower
under Windows than in Linux. It hasn't yet been tested under MacOS.
It generates many files of [results](results):
* `rawincludes.txt`: The raw dump of the `#includes`
* `paths.txt`: A second dump grouping the source module
- `rawincludes.txt`: The raw dump of the `#includes`
- `paths.txt`: A second dump grouping the source module
to the destination module, deduped, and with frequency counts.
* `includes/`: A directory where each file represents a module and
- `includes/`: A directory where each file represents a module and
contains a list of modules and counts that the module _includes_.
* `includedby/`: Similar to `includes/`, but the other way around. Each
- `includedby/`: Similar to `includes/`, but the other way around. Each
file represents a module and contains a list of modules and counts
that _include_ the module.
* [`loops.txt`](results/loops.txt): A list of direct loops detected
- [`loops.txt`](results/loops.txt): A list of direct loops detected
between modules as they actually exist, as opposed to how they are
desired as described above. In a perfect repo, this file will be
empty.
This file is committed to the repo, and is used by the [levelization
Github workflow](../../.github/workflows/levelization.yml) to validate
that nothing changed.
* [`ordering.txt`](results/ordering.txt): A list showing relationships
- [`ordering.txt`](results/ordering.txt): A list showing relationships
between modules where there are no loops as they actually exist, as
opposed to how they are desired as described above.
This file is committed to the repo, and is used by the [levelization
Github workflow](../../.github/workflows/levelization.yml) to validate
that nothing changed.
* [`levelization.yml`](../../.github/workflows/levelization.yml)
- [`levelization.yml`](../../.github/workflows/levelization.yml)
Github Actions workflow to test that levelization loops haven't
changed. Unfortunately, if changes are detected, it can't tell if
changed. Unfortunately, if changes are detected, it can't tell if
they are improvements or not, so if you have resolved any issues or
done anything else to improve levelization, run `levelization.py`,
and commit the updated results.
The `loops.txt` and `ordering.txt` files relate the modules
The `loops.txt` and `ordering.txt` files relate the modules
using comparison signs, which indicate the number of times each
module is included in the other.
* `A > B` means that A should probably be at a higher level than B,
- `A > B` means that A should probably be at a higher level than B,
because B is included in A significantly more than A is included in B.
These results can be included in both `loops.txt` and `ordering.txt`.
Because `ordering.txt`only includes relationships where B is not
included in A at all, it will only include these types of results.
* `A ~= B` means that A and B are included in each other a different
- `A ~= B` means that A and B are included in each other a different
number of times, but the values are so close that the script can't
definitively say that one should be above the other. These results
will only be included in `loops.txt`.
* `A == B` means that A and B include each other the same number of
- `A == B` means that A and B include each other the same number of
times, so the script has no clue which should be higher. These results
will only be included in `loops.txt`.
@@ -110,5 +110,5 @@ get those details locally.
1. Run `levelization.py`
2. Grep the modules in `paths.txt`.
* For example, if a cycle is found `A ~= B`, simply `grep -w
A Builds/levelization/results/paths.txt | grep -w B`
- For example, if a cycle is found `A ~= B`, simply `grep -w
A Builds/levelization/results/paths.txt | grep -w B`

89
CONTRIBUTING.md
View File

@@ -7,7 +7,6 @@ We assume you are familiar with the general practice of [making
contributions on GitHub][1]. This file includes only special
instructions specific to this project.
## Before you start
In general, contributions should be developed in your personal
@@ -28,7 +27,6 @@ your verifying key. Please set up [signature verification][signing].
[signing]:
https://docs.github.com/en/authentication/managing-commit-signature-verification/about-commit-signature-verification
## Major contributions
If your contribution is a major feature or breaking change, then you
@@ -45,7 +43,6 @@ responsibility of the XLS author to update the draft to match the final
implementation when its corresponding pull request is merged, unless the
author delegates that responsibility to others.
## Before making a pull request
Changes that alter transaction processing must be guarded by an
@@ -75,16 +72,17 @@ Changes should be usually squashed down into a single commit.
Some larger or more complicated change sets make more sense,
and are easier to review if organized into multiple logical commits.
Either way, all commits should fit the following criteria:
* Changes should be presented in a single commit or a logical
- Changes should be presented in a single commit or a logical
sequence of commits.
Specifically, chronological commits that simply
reflect the history of how the author implemented
the change, "warts and all", are not useful to
reviewers.
* Every commit should have a [good message](#good-commit-messages).
- Every commit should have a [good message](#good-commit-messages).
to explain a specific aspects of the change.
* Every commit should be signed.
* Every commit should be well-formed (builds successfully,
- Every commit should be signed.
- Every commit should be well-formed (builds successfully,
unit tests passing), as this helps to resolve merge
conflicts, and makes it easier to use `git bisect`
to find bugs.
@@ -96,13 +94,14 @@ Refer to
for general rules on writing a good commit message.
tl;dr
> 1. Separate subject from body with a blank line.
> 2. Limit the subject line to 50 characters.
> * [...]shoot for 50 characters, but consider 72 the hard limit.
> - [...]shoot for 50 characters, but consider 72 the hard limit.
> 3. Capitalize the subject line.
> 4. Do not end the subject line with a period.
> 5. Use the imperative mood in the subject line.
> * A properly formed Git commit subject line should always be able
> - A properly formed Git commit subject line should always be able
> to complete the following sentence: "If applied, this commit will
> _your subject line here_".
> 6. Wrap the body at 72 characters.
@@ -110,16 +109,17 @@ tl;dr
In addition to those guidelines, please add one of the following
prefixes to the subject line if appropriate.
* `fix:` - The primary purpose is to fix an existing bug.
* `perf:` - The primary purpose is performance improvements.
* `refactor:` - The changes refactor code without affecting
- `fix:` - The primary purpose is to fix an existing bug.
- `perf:` - The primary purpose is performance improvements.
- `refactor:` - The changes refactor code without affecting
functionality.
* `test:` - The changes _only_ affect unit tests.
* `docs:` - The changes _only_ affect documentation. This can
- `test:` - The changes _only_ affect unit tests.
- `docs:` - The changes _only_ affect documentation. This can
include code comments in addition to `.md` files like this one.
* `build:` - The changes _only_ affect the build process,
- `build:` - The changes _only_ affect the build process,
including CMake and/or Conan settings.
* `chore:` - Other tasks that don't affect the binary, but don't fit
- `chore:` - Other tasks that don't affect the binary, but don't fit
any of the other cases. e.g. formatting, git settings, updating
Github Actions jobs.
@@ -169,11 +169,11 @@ meets a few criteria:
2. All CI checks must be complete and passed. (One-off failures may
be acceptable if they are related to a known issue.)
3. The PR must have a [good commit message](#good-commit-messages).
* If the PR started with a good commit message, and it doesn't
- If the PR started with a good commit message, and it doesn't
need to be updated, the author can indicate that in a comment.
* Any contributor, preferably the author, can leave a comment
- Any contributor, preferably the author, can leave a comment
suggesting a commit message.
* If the author squashes and rebases the code in preparation for
- If the author squashes and rebases the code in preparation for
merge, they should also ensure the commit message(s) are updated
as well.
4. The PR branch must be up to date with the base branch (usually
@@ -320,7 +320,6 @@ This is a non-exhaustive list of recommended style guidelines. These are
not always strictly enforced and serve as a way to keep the codebase
coherent rather than a set of _thou shalt not_ commandments.
## Formatting
All code must conform to `clang-format` version 10,
@@ -349,6 +348,7 @@ To download the patch file:
5. Commit and push.
You can install a pre-commit hook to automatically run `clang-format` before every commit:
```
pip3 install pre-commit
pre-commit install
@@ -379,49 +379,51 @@ locations, where the reporting of contract violations on the Antithesis
platform is either not possible or not useful.
For this reason:
* The locations where `assert` or `assert(false)` contracts should continue to be used:
* `constexpr` functions
* unit tests i.e. files under `src/test`
* unit tests-related modules (files under `beast/test` and `beast/unit_test`)
* Outside of the listed locations, do not use `assert`; use `XRPL_ASSERT` instead,
- The locations where `assert` or `assert(false)` contracts should continue to be used:
- `constexpr` functions
- unit tests i.e. files under `src/test`
- unit tests-related modules (files under `beast/test` and `beast/unit_test`)
- Outside of the listed locations, do not use `assert`; use `XRPL_ASSERT` instead,
giving it unique name, with the short description of the contract.
* Outside of the listed locations, do not use `assert(false)`; use
- Outside of the listed locations, do not use `assert(false)`; use
`UNREACHABLE` instead, giving it unique name, with the description of the
condition being violated
* The contract name should start with a full name (including scope) of the
function, optionally a named lambda, followed by a colon ` : ` and a brief
- The contract name should start with a full name (including scope) of the
function, optionally a named lambda, followed by a colon `:` and a brief
(typically at most five words) description. `UNREACHABLE` contracts
can use slightly longer descriptions. If there are multiple overloads of the
function, use common sense to balance both brevity and unambiguity of the
function name. NOTE: the purpose of name is to provide stable means of
unique identification of every contract; for this reason try to avoid elements
which can change in some obvious refactors or when reinforcing the condition.
* Contract description typically (except for `UNREACHABLE`) should describe the
- Contract description typically (except for `UNREACHABLE`) should describe the
_expected_ condition, as in "I assert that _expected_ is true".
* Contract description for `UNREACHABLE` should describe the _unexpected_
- Contract description for `UNREACHABLE` should describe the _unexpected_
situation which caused the line to have been reached.
* Example good name for an
- Example good name for an
`UNREACHABLE` macro `"Json::operator==(Value, Value) : invalid type"`; example
good name for an `XRPL_ASSERT` macro `"Json::Value::asCString : valid type"`.
* Example **bad** name
- Example **bad** name
`"RFC1751::insert(char* s, int x, int start, int length) : length is greater than or equal zero"`
(missing namespace, unnecessary full function signature, description too verbose).
Good name: `"ripple::RFC1751::insert : minimum length"`.
* In **few** well-justified cases a non-standard name can be used, in which case a
- In **few** well-justified cases a non-standard name can be used, in which case a
comment should be placed to explain the rationale (example in `contract.cpp`)
* Do **not** rename a contract without a good reason (e.g. the name no longer
- Do **not** rename a contract without a good reason (e.g. the name no longer
reflects the location or the condition being checked)
* Do not use `std::unreachable`
* Do not put contracts where they can be violated by an external condition
- Do not use `std::unreachable`
- Do not put contracts where they can be violated by an external condition
(e.g. timing, data payload before mandatory validation etc.) as this creates
bogus bug reports (and causes crashes of Debug builds)
## Unit Tests
To execute all unit tests:
```rippled --unittest --unittest-jobs=<number of cores>```
`rippled --unittest --unittest-jobs=<number of cores>`
(Note: Using multiple cores on a Mac M1 can cause spurious test failures. The
(Note: Using multiple cores on a Mac M1 can cause spurious test failures. The
cause is still under investigation. If you observe this problem, try specifying fewer jobs.)
To run a specific set of test suites:
@@ -429,10 +431,11 @@ To run a specific set of test suites:
```
rippled --unittest TestSuiteName
```
Note: In this example, all tests with prefix `TestSuiteName` will be run, so if
`TestSuiteName1` and `TestSuiteName2` both exist, then both tests will run.
Alternatively, if the unit test name finds an exact match, it will stop
doing partial matches, i.e. if a unit test with a title of `TestSuiteName`
`TestSuiteName1` and `TestSuiteName2` both exist, then both tests will run.
Alternatively, if the unit test name finds an exact match, it will stop
doing partial matches, i.e. if a unit test with a title of `TestSuiteName`
exists, then no other unit test will be executed, apart from `TestSuiteName`.
## Avoid
@@ -448,7 +451,6 @@ exists, then no other unit test will be executed, apart from `TestSuiteName`.
explanatory comments.
8. Importing new libraries unless there is a very good reason to do so.
## Seek to
9. Extend functionality of existing code rather than creating new code.
@@ -463,14 +465,12 @@ exists, then no other unit test will be executed, apart from `TestSuiteName`.
14. Provide as many comments as you feel that a competent programmer
would need to understand what your code does.
# Maintainers
Maintainers are ecosystem participants with elevated access to the repository.
They are able to push new code, make decisions on when a release should be
made, etc.
## Adding and removing
New maintainers can be proposed by two existing maintainers, subject to a vote
@@ -485,7 +485,6 @@ A minimum of 60% agreement and 50% participation are required.
The XRP Ledger Foundation will have the ability, for cause, to remove an
existing maintainer without a vote.
## Current Maintainers
* [Richard Holland](https://github.com/RichardAH) (XRPL Labs + INFTF)

3
LICENSE.md
View File

@@ -1,4 +1,4 @@
ISC License
ISC License
Copyright (c) 2011, Arthur Britto, David Schwartz, Jed McCaleb, Vinnie Falco, Bob Way, Eric Lombrozo, Nikolaos D. Bougalis, Howard Hinnant.
Copyright (c) 2012-2020, the XRP Ledger developers.
@@ -15,4 +15,3 @@ ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

2
README.md
View File

@@ -5,9 +5,11 @@
[Xahau](https://xahau.network/) is a decentralized cryptographic ledger that builds upon the robust foundation of the XRP Ledger. It inherits the XRP Ledger's Byzantine Fault Tolerant consensus algorithm and enhances it with additional features and functionalities. Developers and users familiar with the XRP Ledger will find that most documentation and tutorials available on [xrpl.org](https://xrpl.org) are relevant and applicable to Xahau, including those related to running validators and managing validator keys. For Xahau specific documentation you can visit our [documentation](https://xahau.network/)
## XAH
XAH is the public, counterparty-free asset native to Xahau and functions primarily as network gas. Transactions submitted to the Xahau network must supply an appropriate amount of XAH, to be burnt by the network as a fee, in order to be successfully included in a validated ledger. In addition, XAH also acts as a bridge currency within the Xahau DEX. XAH is traded on the open-market and is available for anyone to access. Xahau was created in 2023 with a supply of 600 million units of XAH.
## xahaud
The server software that powers Xahau is called `xahaud` and is available in this repository under the permissive [ISC open-source license](LICENSE.md). The `xahaud` server software is written primarily in C++ and runs on a variety of platforms. The `xahaud` server software can run in several modes depending on its configuration.
### Build from Source

1
SECURITY.md
View File

@@ -2,7 +2,6 @@
For more details on operating the Xahau server securely, please visit https://docs.xahau.network/infrastructure/building-xahau.
# Security Policy
## Supported Versions

1
docs/build/environment.md vendored
View File

@@ -5,7 +5,6 @@ platforms: Linux, macOS, or Windows.
[BUILD.md]: ../../BUILD.md
## Linux
Package ecosystems vary across Linux distributions,

13
external/README.md vendored
View File

@@ -2,11 +2,8 @@ The subdirectories in this directory contain either copies or Conan recipes
of external libraries used by rippled.
The Conan recipes include patches we have not yet pushed upstream.
| Folder | Upstream | Description |
|:----------------|:---------------------------------------------|:------------|
| `antithesis-sdk`| [Project](https://github.com/antithesishq/antithesis-sdk-cpp/) | [Antithesis](https://antithesis.com/docs/using_antithesis/sdk/cpp/overview.html) SDK for C++ |
| `ed25519-donna` | [Project](https://github.com/floodyberry/ed25519-donna) | [Ed25519](http://ed25519.cr.yp.to/) digital signatures |
| `rocksdb` | [Recipe](https://github.com/conan-io/conan-center-index/tree/master/recipes/rocksdb) | Fast key/value database. (Supports rotational disks better than NuDB.) |
| `secp256k1` | [Project](https://github.com/bitcoin-core/secp256k1) | ECDSA digital signatures using the **secp256k1** curve |
| `snappy` | [Recipe](https://github.com/conan-io/conan-center-index/tree/master/recipes/snappy) | "Snappy" lossless compression algorithm. |
| `soci` | [Recipe](https://github.com/conan-io/conan-center-index/tree/master/recipes/soci) | Abstraction layer for database access. |
| Folder | Upstream | Description |
| :--------------- | :------------------------------------------------------------- | :------------------------------------------------------------------------------------------- |
| `antithesis-sdk` | [Project](https://github.com/antithesishq/antithesis-sdk-cpp/) | [Antithesis](https://antithesis.com/docs/using_antithesis/sdk/cpp/overview.html) SDK for C++ |
| `ed25519-donna` | [Project](https://github.com/floodyberry/ed25519-donna) | [Ed25519](http://ed25519.cr.yp.to/) digital signatures |
| `secp256k1` | [Project](https://github.com/bitcoin-core/secp256k1) | ECDSA digital signatures using the **secp256k1** curve |

7
external/antithesis-sdk/README.md vendored
View File

@@ -1,8 +1,9 @@
# Antithesis C++ SDK
This library provides methods for C++ programs to configure the [Antithesis](https://antithesis.com) platform. It contains three kinds of functionality:
* Assertion macros that allow you to define test properties about your software or workload.
* Randomness functions for requesting both structured and unstructured randomness from the Antithesis platform.
* Lifecycle functions that inform the Antithesis environment that particular test phases or milestones have been reached.
- Assertion macros that allow you to define test properties about your software or workload.
- Randomness functions for requesting both structured and unstructured randomness from the Antithesis platform.
- Lifecycle functions that inform the Antithesis environment that particular test phases or milestones have been reached.
For general usage guidance see the [Antithesis C++ SDK Documentation](https://antithesis.com/docs/using_antithesis/sdk/cpp/overview/)

97
external/ed25519-donna/README.md vendored
View File

@@ -1,12 +1,12 @@
[ed25519](http://ed25519.cr.yp.to/) is an
[Elliptic Curve Digital Signature Algortithm](http://en.wikipedia.org/wiki/Elliptic_Curve_DSA),
developed by [Dan Bernstein](http://cr.yp.to/djb.html),
[Niels Duif](http://www.nielsduif.nl/),
[Tanja Lange](http://hyperelliptic.org/tanja),
[Peter Schwabe](http://www.cryptojedi.org/users/peter/),
[ed25519](http://ed25519.cr.yp.to/) is an
[Elliptic Curve Digital Signature Algortithm](http://en.wikipedia.org/wiki/Elliptic_Curve_DSA),
developed by [Dan Bernstein](http://cr.yp.to/djb.html),
[Niels Duif](http://www.nielsduif.nl/),
[Tanja Lange](http://hyperelliptic.org/tanja),
[Peter Schwabe](http://www.cryptojedi.org/users/peter/),
and [Bo-Yin Yang](http://www.iis.sinica.edu.tw/pages/byyang/).
This project provides performant, portable 32-bit & 64-bit implementations. All implementations are
This project provides performant, portable 32-bit & 64-bit implementations. All implementations are
of course constant time in regard to secret data.
#### Performance
@@ -52,35 +52,35 @@ are made.
#### Compilation
No configuration is needed **if you are compiling against OpenSSL**.
No configuration is needed **if you are compiling against OpenSSL**.
##### Hash Options
If you are not compiling aginst OpenSSL, you will need a hash function.
To use a simple/**slow** implementation of SHA-512, use `-DED25519_REFHASH` when compiling `ed25519.c`.
To use a simple/**slow** implementation of SHA-512, use `-DED25519_REFHASH` when compiling `ed25519.c`.
This should never be used except to verify the code works when OpenSSL is not available.
To use a custom hash function, use `-DED25519_CUSTOMHASH` when compiling `ed25519.c` and put your
To use a custom hash function, use `-DED25519_CUSTOMHASH` when compiling `ed25519.c` and put your
custom hash implementation in ed25519-hash-custom.h. The hash must have a 512bit digest and implement
struct ed25519_hash_context;
struct ed25519_hash_context;
void ed25519_hash_init(ed25519_hash_context *ctx);
void ed25519_hash_update(ed25519_hash_context *ctx, const uint8_t *in, size_t inlen);
void ed25519_hash_final(ed25519_hash_context *ctx, uint8_t *hash);
void ed25519_hash(uint8_t *hash, const uint8_t *in, size_t inlen);
void ed25519_hash_init(ed25519_hash_context *ctx);
void ed25519_hash_update(ed25519_hash_context *ctx, const uint8_t *in, size_t inlen);
void ed25519_hash_final(ed25519_hash_context *ctx, uint8_t *hash);
void ed25519_hash(uint8_t *hash, const uint8_t *in, size_t inlen);
##### Random Options
If you are not compiling aginst OpenSSL, you will need a random function for batch verification.
To use a custom random function, use `-DED25519_CUSTOMRANDOM` when compiling `ed25519.c` and put your
To use a custom random function, use `-DED25519_CUSTOMRANDOM` when compiling `ed25519.c` and put your
custom hash implementation in ed25519-randombytes-custom.h. The random function must implement:
void ED25519_FN(ed25519_randombytes_unsafe) (void *p, size_t len);
void ED25519_FN(ed25519_randombytes_unsafe) (void *p, size_t len);
Use `-DED25519_TEST` when compiling `ed25519.c` to use a deterministically seeded, non-thread safe CSPRNG
Use `-DED25519_TEST` when compiling `ed25519.c` to use a deterministically seeded, non-thread safe CSPRNG
variant of Bob Jenkins [ISAAC](http://en.wikipedia.org/wiki/ISAAC_%28cipher%29)
##### Minor options
@@ -91,80 +91,79 @@ Use `-DED25519_FORCE_32BIT` to force the use of 32 bit routines even when compil
##### 32-bit
gcc ed25519.c -m32 -O3 -c
gcc ed25519.c -m32 -O3 -c
##### 64-bit
gcc ed25519.c -m64 -O3 -c
gcc ed25519.c -m64 -O3 -c
##### SSE2
gcc ed25519.c -m32 -O3 -c -DED25519_SSE2 -msse2
gcc ed25519.c -m64 -O3 -c -DED25519_SSE2
gcc ed25519.c -m32 -O3 -c -DED25519_SSE2 -msse2
gcc ed25519.c -m64 -O3 -c -DED25519_SSE2
clang and icc are also supported
#### Usage
To use the code, link against `ed25519.o -mbits` and:
#include "ed25519.h"
#include "ed25519.h"
Add `-lssl -lcrypto` when using OpenSSL (Some systems don't need -lcrypto? It might be trial and error).
To generate a private key, simply generate 32 bytes from a secure
cryptographic source:
ed25519_secret_key sk;
randombytes(sk, sizeof(ed25519_secret_key));
ed25519_secret_key sk;
randombytes(sk, sizeof(ed25519_secret_key));
To generate a public key:
ed25519_public_key pk;
ed25519_publickey(sk, pk);
ed25519_public_key pk;
ed25519_publickey(sk, pk);
To sign a message:
ed25519_signature sig;
ed25519_sign(message, message_len, sk, pk, signature);
ed25519_signature sig;
ed25519_sign(message, message_len, sk, pk, signature);
To verify a signature:
int valid = ed25519_sign_open(message, message_len, pk, signature) == 0;
int valid = ed25519_sign_open(message, message_len, pk, signature) == 0;
To batch verify signatures:
const unsigned char *mp[num] = {message1, message2..}
size_t ml[num] = {message_len1, message_len2..}
const unsigned char *pkp[num] = {pk1, pk2..}
const unsigned char *sigp[num] = {signature1, signature2..}
int valid[num]
const unsigned char *mp[num] = {message1, message2..}
size_t ml[num] = {message_len1, message_len2..}
const unsigned char *pkp[num] = {pk1, pk2..}
const unsigned char *sigp[num] = {signature1, signature2..}
int valid[num]
/* valid[i] will be set to 1 if the individual signature was valid, 0 otherwise */
int all_valid = ed25519_sign_open_batch(mp, ml, pkp, sigp, num, valid) == 0;
/* valid[i] will be set to 1 if the individual signature was valid, 0 otherwise */
int all_valid = ed25519_sign_open_batch(mp, ml, pkp, sigp, num, valid) == 0;
**Note**: Batch verification uses `ed25519_randombytes_unsafe`, implemented in
`ed25519-randombytes.h`, to generate random scalars for the verification code.
**Note**: Batch verification uses `ed25519_randombytes_unsafe`, implemented in
`ed25519-randombytes.h`, to generate random scalars for the verification code.
The default implementation now uses OpenSSLs `RAND_bytes`.
Unlike the [SUPERCOP](http://bench.cr.yp.to/supercop.html) version, signatures are
not appended to messages, and there is no need for padding in front of messages.
Additionally, the secret key does not contain a copy of the public key, so it is
not appended to messages, and there is no need for padding in front of messages.
Additionally, the secret key does not contain a copy of the public key, so it is
32 bytes instead of 64 bytes, and the public key must be provided to the signing
function.
##### Curve25519
Curve25519 public keys can be generated thanks to
[Adam Langley](http://www.imperialviolet.org/2013/05/10/fastercurve25519.html)
Curve25519 public keys can be generated thanks to
[Adam Langley](http://www.imperialviolet.org/2013/05/10/fastercurve25519.html)
leveraging Ed25519's precomputed basepoint scalar multiplication.
curved25519_key sk, pk;
randombytes(sk, sizeof(curved25519_key));
curved25519_scalarmult_basepoint(pk, sk);
curved25519_key sk, pk;
randombytes(sk, sizeof(curved25519_key));
curved25519_scalarmult_basepoint(pk, sk);
Note the name is curved25519, a combination of curve and ed25519, to prevent
Note the name is curved25519, a combination of curve and ed25519, to prevent
name clashes. Performance is slightly faster than short message ed25519
signing due to both using the same code for the scalar multiply.
@@ -180,4 +179,4 @@ with extreme values to ensure they function correctly. SSE2 is now supported.
#### Papers
[Available on the Ed25519 website](http://ed25519.cr.yp.to/papers.html)
[Available on the Ed25519 website](http://ed25519.cr.yp.to/papers.html)

99
external/ed25519-donna/fuzz/README.md vendored
View File

@@ -1,78 +1,78 @@
This code fuzzes ed25519-donna (and optionally ed25519-donna-sse2) against the ref10 implementations of
[curve25519](https://github.com/floodyberry/supercop/tree/master/crypto_scalarmult/curve25519/ref10) and
[curve25519](https://github.com/floodyberry/supercop/tree/master/crypto_scalarmult/curve25519/ref10) and
[ed25519](https://github.com/floodyberry/supercop/tree/master/crypto_sign/ed25519/ref10).
Curve25519 tests that generating a public key from a secret key
# Building
## *nix + PHP
## \*nix + PHP
`php build-nix.php (required parameters) (optional parameters)`
Required parameters:
* `--function=[curve25519,ed25519]`
* `--bits=[32,64]`
- `--function=[curve25519,ed25519]`
- `--bits=[32,64]`
Optional parameters:
* `--with-sse2`
- `--with-sse2`
Also fuzz against ed25519-donna-sse2
* `--with-openssl`
Also fuzz against ed25519-donna-sse2
Build with OpenSSL's SHA-512.
- `--with-openssl`
Default: Reference SHA-512 implementation (slow!)
Build with OpenSSL's SHA-512.
* `--compiler=[gcc,clang,icc]`
Default: Reference SHA-512 implementation (slow!)
Default: gcc
- `--compiler=[gcc,clang,icc]`
* `--no-asm`
Default: gcc
Do not use platform specific assembler
- `--no-asm`
Do not use platform specific assembler
example:
php build-nix.php --bits=64 --function=ed25519 --with-sse2 --compiler=icc
php build-nix.php --bits=64 --function=ed25519 --with-sse2 --compiler=icc
## Windows
Create a project with access to the ed25519 files.
If you are not using OpenSSL, add the `ED25519_REFHASH` define to the projects
If you are not using OpenSSL, add the `ED25519_REFHASH` define to the projects
"Properties/Preprocessor/Preprocessor Definitions" option
Add the following files to the project:
* `fuzz/curve25519-ref10.c`
* `fuzz/ed25519-ref10.c`
* `fuzz/ed25519-donna.c`
* `fuzz/ed25519-donna-sse2.c` (optional)
* `fuzz-[curve25519/ed25519].c` (depending on which you want to fuzz)
- `fuzz/curve25519-ref10.c`
- `fuzz/ed25519-ref10.c`
- `fuzz/ed25519-donna.c`
- `fuzz/ed25519-donna-sse2.c` (optional)
- `fuzz-[curve25519/ed25519].c` (depending on which you want to fuzz)
If you are also fuzzing against ed25519-donna-sse2, add the `ED25519_SSE2` define for `fuzz-[curve25519/ed25519].c` under
If you are also fuzzing against ed25519-donna-sse2, add the `ED25519_SSE2` define for `fuzz-[curve25519/ed25519].c` under
its "Properties/Preprocessor/Preprocessor Definitions" option.
# Running
If everything agrees, the program will only output occasional status dots (every 0x1000 passes)
If everything agrees, the program will only output occasional status dots (every 0x1000 passes)
and a 64bit progress count (every 0x20000 passes):
fuzzing: ref10 curved25519 curved25519-sse2
................................ [0000000000020000]
................................ [0000000000040000]
................................ [0000000000060000]
................................ [0000000000080000]
................................ [00000000000a0000]
................................ [00000000000c0000]
If any of the implementations do not agree with the ref10 implementation, the program will dump
the random data that was used, the data generated by the ref10 implementation, and diffs of the
the random data that was used, the data generated by the ref10 implementation, and diffs of the
ed25519-donna data against the ref10 data.
## Example errors
@@ -83,21 +83,21 @@ These are example error dumps (with intentionally introduced errors).
Random data:
* sk, or Secret Key
* m, or Message
- sk, or Secret Key
- m, or Message
Generated data:
* pk, or Public Key
* sig, or Signature
* valid, or if the signature of the message is valid with the public key
- pk, or Public Key
- sig, or Signature
- valid, or if the signature of the message is valid with the public key
Dump:
sk:
0x3b,0xb7,0x17,0x7a,0x66,0xdc,0xb7,0x9a,0x90,0x25,0x07,0x99,0x96,0xf3,0x92,0xef,
0x78,0xf8,0xad,0x6c,0x35,0x87,0x81,0x67,0x03,0xe6,0x95,0xba,0x06,0x18,0x7c,0x9c,
m:
0x7c,0x8d,0x3d,0xe1,0x92,0xee,0x7a,0xb8,0x4d,0xc9,0xfb,0x02,0x34,0x1e,0x5a,0x91,
0xee,0x01,0xa6,0xb8,0xab,0x37,0x3f,0x3d,0x6d,0xa2,0x47,0xe3,0x27,0x93,0x7c,0xb7,
@@ -107,67 +107,66 @@ Dump:
0x63,0x14,0xe0,0x81,0x52,0xec,0xcd,0xcf,0x70,0x54,0x7d,0xa3,0x49,0x8b,0xf0,0x89,
0x70,0x07,0x12,0x2a,0xd9,0xaa,0x16,0x01,0xb2,0x16,0x3a,0xbb,0xfc,0xfa,0x13,0x5b,
0x69,0x83,0x92,0x70,0x95,0x76,0xa0,0x8e,0x16,0x79,0xcc,0xaa,0xb5,0x7c,0xf8,0x7a,
ref10:
pk:
0x71,0xb0,0x5e,0x62,0x1b,0xe3,0xe7,0x36,0x91,0x8b,0xc0,0x13,0x36,0x0c,0xc9,0x04,
0x16,0xf5,0xff,0x48,0x0c,0x83,0x6b,0x88,0x53,0xa2,0xc6,0x0f,0xf7,0xac,0x42,0x04,
sig:
0x3e,0x05,0xc5,0x37,0x16,0x0b,0x29,0x30,0x89,0xa3,0xe7,0x83,0x08,0x16,0xdd,0x96,
0x02,0xfa,0x0d,0x44,0x2c,0x43,0xaa,0x80,0x93,0x04,0x58,0x22,0x09,0xbf,0x11,0xa5,
0xcc,0xa5,0x3c,0x9f,0xa0,0xa4,0x64,0x5a,0x4a,0xdb,0x20,0xfb,0xc7,0x9b,0xfd,0x3f,
0x08,0xae,0xc4,0x3c,0x1e,0xd8,0xb6,0xb4,0xd2,0x6d,0x80,0x92,0xcb,0x71,0xf3,0x02,
valid: yes
ed25519-donna:
pk diff:
____,____,____,____,____,____,____,____,____,____,____,____,____,____,____,____,
____,____,____,____,____,____,____,____,____,____,____,____,____,____,____,____,
sig diff:
0x2c,0xb9,0x25,0x14,0xd0,0x94,0xeb,0xfe,0x46,0x02,0xc2,0xe8,0xa3,0xeb,0xbf,0xb5,
0x72,0x84,0xbf,0xc1,0x8a,0x32,0x30,0x99,0xf7,0x58,0xfe,0x06,0xa8,0xdc,0xdc,0xab,
0xb5,0x57,0x03,0x33,0x87,0xce,0x54,0x55,0x6a,0x69,0x8a,0xc4,0xb7,0x2a,0xed,0x97,
0xb4,0x68,0xe7,0x52,0x7a,0x07,0x55,0x3b,0xa2,0x94,0xd6,0x5e,0xa1,0x61,0x80,0x08,
valid: no
In this case, the generated public key matches, but the generated signature is completely
In this case, the generated public key matches, but the generated signature is completely
different and does not validate.
### Curve25519
Random data:
* sk, or Secret Key
- sk, or Secret Key
Generated data:
* pk, or Public Key
- pk, or Public Key
Dump:
sk:
0x44,0xec,0x0b,0x0e,0xa2,0x0e,0x9c,0x5b,0x8c,0xce,0x7b,0x1d,0x68,0xae,0x0f,0x9e,
0x81,0xe2,0x04,0x76,0xda,0x87,0xa4,0x9e,0xc9,0x4f,0x3b,0xf9,0xc3,0x89,0x63,0x70,
ref10:
0x24,0x55,0x55,0xc0,0xf9,0x80,0xaf,0x02,0x43,0xee,0x8c,0x7f,0xc1,0xad,0x90,0x95,
0x57,0x91,0x14,0x2e,0xf2,0x14,0x22,0x80,0xdd,0x4e,0x3c,0x85,0x71,0x84,0x8c,0x62,
curved25519 diff:
0x12,0xd1,0x61,0x2b,0x16,0xb3,0xd8,0x29,0xf8,0xa3,0xba,0x70,0x4e,0x49,0x4f,0x43,
0xa1,0x3c,0x6b,0x42,0x11,0x61,0xcc,0x30,0x87,0x73,0x46,0xfb,0x85,0xc7,0x9a,0x35,
curved25519-sse2 diff:
____,____,____,____,____,____,____,____,____,____,____,____,____,____,____,____,
____,____,____,____,____,____,____,____,____,____,____,____,____,____,____,____,
In this case, curved25519 is totally wrong, while curved25519-sse2 matches the reference
implementation.
In this case, curved25519 is totally wrong, while curved25519-sse2 matches the reference
implementation.

144
external/secp256k1/CHANGELOG.md vendored
View File

@@ -8,153 +8,189 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [0.6.0] - 2024-11-04
#### Added
- New module `musig` implements the MuSig2 multisignature scheme according to the [BIP 327 specification](https://github.com/bitcoin/bips/blob/master/bip-0327.mediawiki). See:
- Header file `include/secp256k1_musig.h` which defines the new API.
- Document `doc/musig.md` for further notes on API usage.
- Usage example `examples/musig.c`.
- New CMake variable `SECP256K1_APPEND_LDFLAGS` for appending linker flags to the build command.
- New module `musig` implements the MuSig2 multisignature scheme according to the [BIP 327 specification](https://github.com/bitcoin/bips/blob/master/bip-0327.mediawiki). See:
- Header file `include/secp256k1_musig.h` which defines the new API.
- Document `doc/musig.md` for further notes on API usage.
- Usage example `examples/musig.c`.
- New CMake variable `SECP256K1_APPEND_LDFLAGS` for appending linker flags to the build command.
#### Changed
- API functions now use a significantly more robust method to clear secrets from the stack before returning. However, secret clearing remains a best-effort security measure and cannot guarantee complete removal.
- Any type `secp256k1_foo` can now be forward-declared using `typedef struct secp256k1_foo secp256k1_foo;` (or also `struct secp256k1_foo;` in C++).
- Organized CMake build artifacts into dedicated directories (`bin/` for executables, `lib/` for libraries) to improve build output structure and Windows shared library compatibility.
- API functions now use a significantly more robust method to clear secrets from the stack before returning. However, secret clearing remains a best-effort security measure and cannot guarantee complete removal.
- Any type `secp256k1_foo` can now be forward-declared using `typedef struct secp256k1_foo secp256k1_foo;` (or also `struct secp256k1_foo;` in C++).
- Organized CMake build artifacts into dedicated directories (`bin/` for executables, `lib/` for libraries) to improve build output structure and Windows shared library compatibility.
#### Removed
- Removed the `secp256k1_scratch_space` struct and its associated functions `secp256k1_scratch_space_create` and `secp256k1_scratch_space_destroy` because the scratch space was unused in the API.
- Removed the `secp256k1_scratch_space` struct and its associated functions `secp256k1_scratch_space_create` and `secp256k1_scratch_space_destroy` because the scratch space was unused in the API.
#### ABI Compatibility
The symbols `secp256k1_scratch_space_create` and `secp256k1_scratch_space_destroy` were removed.
Otherwise, the library maintains backward compatibility with versions 0.3.x through 0.5.x.
## [0.5.1] - 2024-08-01
#### Added
- Added usage example for an ElligatorSwift key exchange.
- Added usage example for an ElligatorSwift key exchange.
#### Changed
- The default size of the precomputed table for signing was changed from 22 KiB to 86 KiB. The size can be changed with the configure option `--ecmult-gen-kb` (`SECP256K1_ECMULT_GEN_KB` for CMake).
- "auto" is no longer an accepted value for the `--with-ecmult-window` and `--with-ecmult-gen-kb` configure options (this also applies to `SECP256K1_ECMULT_WINDOW_SIZE` and `SECP256K1_ECMULT_GEN_KB` in CMake). To achieve the same configuration as previously provided by the "auto" value, omit setting the configure option explicitly.
- The default size of the precomputed table for signing was changed from 22 KiB to 86 KiB. The size can be changed with the configure option `--ecmult-gen-kb` (`SECP256K1_ECMULT_GEN_KB` for CMake).
- "auto" is no longer an accepted value for the `--with-ecmult-window` and `--with-ecmult-gen-kb` configure options (this also applies to `SECP256K1_ECMULT_WINDOW_SIZE` and `SECP256K1_ECMULT_GEN_KB` in CMake). To achieve the same configuration as previously provided by the "auto" value, omit setting the configure option explicitly.
#### Fixed
- Fixed compilation when the extrakeys module is disabled.
- Fixed compilation when the extrakeys module is disabled.
#### ABI Compatibility
The ABI is backward compatible with versions 0.5.0, 0.4.x and 0.3.x.
## [0.5.0] - 2024-05-06
#### Added
- New function `secp256k1_ec_pubkey_sort` that sorts public keys using lexicographic (of compressed serialization) order.
- New function `secp256k1_ec_pubkey_sort` that sorts public keys using lexicographic (of compressed serialization) order.
#### Changed
- The implementation of the point multiplication algorithm used for signing and public key generation was changed, resulting in improved performance for those operations.
- The related configure option `--ecmult-gen-precision` was replaced with `--ecmult-gen-kb` (`SECP256K1_ECMULT_GEN_KB` for CMake).
- This changes the supported precomputed table sizes for these operations. The new supported sizes are 2 KiB, 22 KiB, or 86 KiB (while the old supported sizes were 32 KiB, 64 KiB, or 512 KiB).
- The implementation of the point multiplication algorithm used for signing and public key generation was changed, resulting in improved performance for those operations.
- The related configure option `--ecmult-gen-precision` was replaced with `--ecmult-gen-kb` (`SECP256K1_ECMULT_GEN_KB` for CMake).
- This changes the supported precomputed table sizes for these operations. The new supported sizes are 2 KiB, 22 KiB, or 86 KiB (while the old supported sizes were 32 KiB, 64 KiB, or 512 KiB).
#### ABI Compatibility
The ABI is backward compatible with versions 0.4.x and 0.3.x.
## [0.4.1] - 2023-12-21
#### Changed
- The point multiplication algorithm used for ECDH operations (module `ecdh`) was replaced with a slightly faster one.
- Optional handwritten x86_64 assembly for field operations was removed because modern C compilers are able to output more efficient assembly. This change results in a significant speedup of some library functions when handwritten x86_64 assembly is enabled (`--with-asm=x86_64` in GNU Autotools, `-DSECP256K1_ASM=x86_64` in CMake), which is the default on x86_64. Benchmarks with GCC 10.5.0 show a 10% speedup for `secp256k1_ecdsa_verify` and `secp256k1_schnorrsig_verify`.
- The point multiplication algorithm used for ECDH operations (module `ecdh`) was replaced with a slightly faster one.
- Optional handwritten x86_64 assembly for field operations was removed because modern C compilers are able to output more efficient assembly. This change results in a significant speedup of some library functions when handwritten x86_64 assembly is enabled (`--with-asm=x86_64` in GNU Autotools, `-DSECP256K1_ASM=x86_64` in CMake), which is the default on x86_64. Benchmarks with GCC 10.5.0 show a 10% speedup for `secp256k1_ecdsa_verify` and `secp256k1_schnorrsig_verify`.
#### ABI Compatibility
The ABI is backward compatible with versions 0.4.0 and 0.3.x.
## [0.4.0] - 2023-09-04
#### Added
- New module `ellswift` implements ElligatorSwift encoding for public keys and x-only Diffie-Hellman key exchange for them.
ElligatorSwift permits representing secp256k1 public keys as 64-byte arrays which cannot be distinguished from uniformly random. See:
- Header file `include/secp256k1_ellswift.h` which defines the new API.
- Document `doc/ellswift.md` which explains the mathematical background of the scheme.
- The [paper](https://eprint.iacr.org/2022/759) on which the scheme is based.
- We now test the library with unreleased development snapshots of GCC and Clang. This gives us an early chance to catch miscompilations and constant-time issues introduced by the compiler (such as those that led to the previous two releases).
- New module `ellswift` implements ElligatorSwift encoding for public keys and x-only Diffie-Hellman key exchange for them.
ElligatorSwift permits representing secp256k1 public keys as 64-byte arrays which cannot be distinguished from uniformly random. See:
- Header file `include/secp256k1_ellswift.h` which defines the new API.
- Document `doc/ellswift.md` which explains the mathematical background of the scheme.
- The [paper](https://eprint.iacr.org/2022/759) on which the scheme is based.
- We now test the library with unreleased development snapshots of GCC and Clang. This gives us an early chance to catch miscompilations and constant-time issues introduced by the compiler (such as those that led to the previous two releases).
#### Fixed
- Fixed symbol visibility in Windows DLL builds, where three internal library symbols were wrongly exported.
- Fixed symbol visibility in Windows DLL builds, where three internal library symbols were wrongly exported.
#### Changed
- When consuming libsecp256k1 as a static library on Windows, the user must now define the `SECP256K1_STATIC` macro before including `secp256k1.h`.
- When consuming libsecp256k1 as a static library on Windows, the user must now define the `SECP256K1_STATIC` macro before including `secp256k1.h`.
#### ABI Compatibility
This release is backward compatible with the ABI of 0.3.0, 0.3.1, and 0.3.2. Symbol visibility is now believed to be handled properly on supported platforms and is now considered to be part of the ABI. Please report any improperly exported symbols as a bug.
## [0.3.2] - 2023-05-13
We strongly recommend updating to 0.3.2 if you use or plan to use GCC >=13 to compile libsecp256k1. When in doubt, check the GCC version using `gcc -v`.
#### Security
- Module `ecdh`: Fix "constant-timeness" issue with GCC 13.1 (and potentially future versions of GCC) that could leave applications using libsecp256k1's ECDH module vulnerable to a timing side-channel attack. The fix avoids secret-dependent control flow during ECDH computations when libsecp256k1 is compiled with GCC 13.1.
- Module `ecdh`: Fix "constant-timeness" issue with GCC 13.1 (and potentially future versions of GCC) that could leave applications using libsecp256k1's ECDH module vulnerable to a timing side-channel attack. The fix avoids secret-dependent control flow during ECDH computations when libsecp256k1 is compiled with GCC 13.1.
#### Fixed
- Fixed an old bug that permitted compilers to potentially output bad assembly code on x86_64. In theory, it could lead to a crash or a read of unrelated memory, but this has never been observed on any compilers so far.
- Fixed an old bug that permitted compilers to potentially output bad assembly code on x86_64. In theory, it could lead to a crash or a read of unrelated memory, but this has never been observed on any compilers so far.
#### Changed
- Various improvements and changes to CMake builds. CMake builds remain experimental.
- Made API versioning consistent with GNU Autotools builds.
- Switched to `BUILD_SHARED_LIBS` variable for controlling whether to build a static or a shared library.
- Added `SECP256K1_INSTALL` variable for the controlling whether to install the build artefacts.
- Renamed asm build option `arm` to `arm32`. Use `--with-asm=arm32` instead of `--with-asm=arm` (GNU Autotools), and `-DSECP256K1_ASM=arm32` instead of `-DSECP256K1_ASM=arm` (CMake).
- Various improvements and changes to CMake builds. CMake builds remain experimental.
- Made API versioning consistent with GNU Autotools builds.
- Switched to `BUILD_SHARED_LIBS` variable for controlling whether to build a static or a shared library.
- Added `SECP256K1_INSTALL` variable for the controlling whether to install the build artefacts.
- Renamed asm build option `arm` to `arm32`. Use `--with-asm=arm32` instead of `--with-asm=arm` (GNU Autotools), and `-DSECP256K1_ASM=arm32` instead of `-DSECP256K1_ASM=arm` (CMake).
#### ABI Compatibility
The ABI is compatible with versions 0.3.0 and 0.3.1.
## [0.3.1] - 2023-04-10
We strongly recommend updating to 0.3.1 if you use or plan to use Clang >=14 to compile libsecp256k1, e.g., Xcode >=14 on macOS has Clang >=14. When in doubt, check the Clang version using `clang -v`.
#### Security
- Fix "constant-timeness" issue with Clang >=14 that could leave applications using libsecp256k1 vulnerable to a timing side-channel attack. The fix avoids secret-dependent control flow and secret-dependent memory accesses in conditional moves of memory objects when libsecp256k1 is compiled with Clang >=14.
- Fix "constant-timeness" issue with Clang >=14 that could leave applications using libsecp256k1 vulnerable to a timing side-channel attack. The fix avoids secret-dependent control flow and secret-dependent memory accesses in conditional moves of memory objects when libsecp256k1 is compiled with Clang >=14.
#### Added
- Added tests against [Project Wycheproof's](https://github.com/google/wycheproof/) set of ECDSA test vectors (Bitcoin "low-S" variant), a fixed set of test cases designed to trigger various edge cases.
- Added tests against [Project Wycheproof's](https://github.com/google/wycheproof/) set of ECDSA test vectors (Bitcoin "low-S" variant), a fixed set of test cases designed to trigger various edge cases.
#### Changed
- Increased minimum required CMake version to 3.13. CMake builds remain experimental.
- Increased minimum required CMake version to 3.13. CMake builds remain experimental.
#### ABI Compatibility
The ABI is compatible with version 0.3.0.
## [0.3.0] - 2023-03-08
#### Added
- Added experimental support for CMake builds. Traditional GNU Autotools builds (`./configure` and `make`) remain fully supported.
- Usage examples: Added a recommended method for securely clearing sensitive data, e.g., secret keys, from memory.
- Tests: Added a new test binary `noverify_tests`. This binary runs the tests without some additional checks present in the ordinary `tests` binary and is thereby closer to production binaries. The `noverify_tests` binary is automatically run as part of the `make check` target.
- Added experimental support for CMake builds. Traditional GNU Autotools builds (`./configure` and `make`) remain fully supported.
- Usage examples: Added a recommended method for securely clearing sensitive data, e.g., secret keys, from memory.
- Tests: Added a new test binary `noverify_tests`. This binary runs the tests without some additional checks present in the ordinary `tests` binary and is thereby closer to production binaries. The `noverify_tests` binary is automatically run as part of the `make check` target.
#### Fixed
- Fixed declarations of API variables for MSVC (`__declspec(dllimport)`). This fixes MSVC builds of programs which link against a libsecp256k1 DLL dynamically and use API variables (and not only API functions). Unfortunately, the MSVC linker now will emit warning `LNK4217` when trying to link against libsecp256k1 statically. Pass `/ignore:4217` to the linker to suppress this warning.
- Fixed declarations of API variables for MSVC (`__declspec(dllimport)`). This fixes MSVC builds of programs which link against a libsecp256k1 DLL dynamically and use API variables (and not only API functions). Unfortunately, the MSVC linker now will emit warning `LNK4217` when trying to link against libsecp256k1 statically. Pass `/ignore:4217` to the linker to suppress this warning.
#### Changed
- Forbade cloning or destroying `secp256k1_context_static`. Create a new context instead of cloning the static context. (If this change breaks your code, your code is probably wrong.)
- Forbade randomizing (copies of) `secp256k1_context_static`. Randomizing a copy of `secp256k1_context_static` did not have any effect and did not provide defense-in-depth protection against side-channel attacks. Create a new context if you want to benefit from randomization.
- Forbade cloning or destroying `secp256k1_context_static`. Create a new context instead of cloning the static context. (If this change breaks your code, your code is probably wrong.)
- Forbade randomizing (copies of) `secp256k1_context_static`. Randomizing a copy of `secp256k1_context_static` did not have any effect and did not provide defense-in-depth protection against side-channel attacks. Create a new context if you want to benefit from randomization.
#### Removed
- Removed the configuration header `src/libsecp256k1-config.h`. We recommend passing flags to `./configure` or `cmake` to set configuration options (see `./configure --help` or `cmake -LH`). If you cannot or do not want to use one of the supported build systems, pass configuration flags such as `-DSECP256K1_ENABLE_MODULE_SCHNORRSIG` manually to the compiler (see the file `configure.ac` for supported flags).
- Removed the configuration header `src/libsecp256k1-config.h`. We recommend passing flags to `./configure` or `cmake` to set configuration options (see `./configure --help` or `cmake -LH`). If you cannot or do not want to use one of the supported build systems, pass configuration flags such as `-DSECP256K1_ENABLE_MODULE_SCHNORRSIG` manually to the compiler (see the file `configure.ac` for supported flags).
#### ABI Compatibility
Due to changes in the API regarding `secp256k1_context_static` described above, the ABI is *not* compatible with previous versions.
Due to changes in the API regarding `secp256k1_context_static` described above, the ABI is _not_ compatible with previous versions.
## [0.2.0] - 2022-12-12
#### Added
- Added usage examples for common use cases in a new `examples/` directory.
- Added `secp256k1_selftest`, to be used in conjunction with `secp256k1_context_static`.
- Added support for 128-bit wide multiplication on MSVC for x86_64 and arm64, giving roughly a 20% speedup on those platforms.
- Added usage examples for common use cases in a new `examples/` directory.
- Added `secp256k1_selftest`, to be used in conjunction with `secp256k1_context_static`.
- Added support for 128-bit wide multiplication on MSVC for x86_64 and arm64, giving roughly a 20% speedup on those platforms.
#### Changed
- Enabled modules `schnorrsig`, `extrakeys` and `ecdh` by default in `./configure`.
- The `secp256k1_nonce_function_rfc6979` nonce function, used by default by `secp256k1_ecdsa_sign`, now reduces the message hash modulo the group order to match the specification. This only affects improper use of ECDSA signing API.
- Enabled modules `schnorrsig`, `extrakeys` and `ecdh` by default in `./configure`.
- The `secp256k1_nonce_function_rfc6979` nonce function, used by default by `secp256k1_ecdsa_sign`, now reduces the message hash modulo the group order to match the specification. This only affects improper use of ECDSA signing API.
#### Deprecated
- Deprecated context flags `SECP256K1_CONTEXT_VERIFY` and `SECP256K1_CONTEXT_SIGN`. Use `SECP256K1_CONTEXT_NONE` instead.
- Renamed `secp256k1_context_no_precomp` to `secp256k1_context_static`.
- Module `schnorrsig`: renamed `secp256k1_schnorrsig_sign` to `secp256k1_schnorrsig_sign32`.
- Deprecated context flags `SECP256K1_CONTEXT_VERIFY` and `SECP256K1_CONTEXT_SIGN`. Use `SECP256K1_CONTEXT_NONE` instead.
- Renamed `secp256k1_context_no_precomp` to `secp256k1_context_static`.
- Module `schnorrsig`: renamed `secp256k1_schnorrsig_sign` to `secp256k1_schnorrsig_sign32`.
#### ABI Compatibility
Since this is the first release, we do not compare application binary interfaces.
However, there are earlier unreleased versions of libsecp256k1 that are *not* ABI compatible with this version.
However, there are earlier unreleased versions of libsecp256k1 that are _not_ ABI compatible with this version.
## [0.1.0] - 2013-03-05 to 2021-12-25

6
external/secp256k1/CMakePresets.json vendored
View File

@@ -1,5 +1,9 @@
{
"cmakeMinimumRequired": {"major": 3, "minor": 21, "patch": 0},
"cmakeMinimumRequired": {
"major": 3,
"minor": 21,
"patch": 0
},
"version": 3,
"configurePresets": [
{

74
external/secp256k1/CONTRIBUTING.md vendored
View File

@@ -12,15 +12,15 @@ The libsecp256k1 project welcomes contributions in the form of new functionality
It is the responsibility of the contributors to convince the maintainers that the proposed functionality is within the project's scope, high-quality and maintainable.
Contributors are recommended to provide the following in addition to the new code:
* **Specification:**
A specification can help significantly in reviewing the new code as it provides documentation and context.
It may justify various design decisions, give a motivation and outline security goals.
If the specification contains pseudocode, a reference implementation or test vectors, these can be used to compare with the proposed libsecp256k1 code.
* **Security Arguments:**
In addition to a defining the security goals, it should be argued that the new functionality meets these goals.
Depending on the nature of the new functionality, a wide range of security arguments are acceptable, ranging from being "obviously secure" to rigorous proofs of security.
* **Relevance Arguments:**
The relevance of the new functionality for the Bitcoin ecosystem should be argued by outlining clear use cases.
- **Specification:**
A specification can help significantly in reviewing the new code as it provides documentation and context.
It may justify various design decisions, give a motivation and outline security goals.
If the specification contains pseudocode, a reference implementation or test vectors, these can be used to compare with the proposed libsecp256k1 code.
- **Security Arguments:**
In addition to a defining the security goals, it should be argued that the new functionality meets these goals.
Depending on the nature of the new functionality, a wide range of security arguments are acceptable, ranging from being "obviously secure" to rigorous proofs of security.
- **Relevance Arguments:**
The relevance of the new functionality for the Bitcoin ecosystem should be argued by outlining clear use cases.
These are not the only factors taken into account when considering to add new functionality.
The proposed new libsecp256k1 code must be of high quality, including API documentation and tests, as well as featuring a misuse-resistant API design.
@@ -44,36 +44,36 @@ The Contributor Workflow & Peer Review in libsecp256k1 are similar to Bitcoin Co
In addition, libsecp256k1 tries to maintain the following coding conventions:
* No runtime heap allocation (e.g., no `malloc`) unless explicitly requested by the caller (via `secp256k1_context_create` or `secp256k1_scratch_space_create`, for example). Moreover, it should be possible to use the library without any heap allocations.
* The tests should cover all lines and branches of the library (see [Test coverage](#coverage)).
* Operations involving secret data should be tested for being constant time with respect to the secrets (see [src/ctime_tests.c](src/ctime_tests.c)).
* Local variables containing secret data should be cleared explicitly to try to delete secrets from memory.
* Use `secp256k1_memcmp_var` instead of `memcmp` (see [#823](https://github.com/bitcoin-core/secp256k1/issues/823)).
* As a rule of thumb, the default values for configuration options should target standard desktop machines and align with Bitcoin Core's defaults, and the tests should mostly exercise the default configuration (see [#1549](https://github.com/bitcoin-core/secp256k1/issues/1549#issuecomment-2200559257)).
- No runtime heap allocation (e.g., no `malloc`) unless explicitly requested by the caller (via `secp256k1_context_create` or `secp256k1_scratch_space_create`, for example). Moreover, it should be possible to use the library without any heap allocations.
- The tests should cover all lines and branches of the library (see [Test coverage](#coverage)).
- Operations involving secret data should be tested for being constant time with respect to the secrets (see [src/ctime_tests.c](src/ctime_tests.c)).
- Local variables containing secret data should be cleared explicitly to try to delete secrets from memory.
- Use `secp256k1_memcmp_var` instead of `memcmp` (see [#823](https://github.com/bitcoin-core/secp256k1/issues/823)).
- As a rule of thumb, the default values for configuration options should target standard desktop machines and align with Bitcoin Core's defaults, and the tests should mostly exercise the default configuration (see [#1549](https://github.com/bitcoin-core/secp256k1/issues/1549#issuecomment-2200559257)).
#### Style conventions
* Commits should be atomic and diffs should be easy to read. For this reason, do not mix any formatting fixes or code moves with actual code changes. Make sure each individual commit is hygienic: that it builds successfully on its own without warnings, errors, regressions, or test failures.
* New code should adhere to the style of existing, in particular surrounding, code. Other than that, we do not enforce strict rules for code formatting.
* The code conforms to C89. Most notably, that means that only `/* ... */` comments are allowed (no `//` line comments). Moreover, any declarations in a `{ ... }` block (e.g., a function) must appear at the beginning of the block before any statements. When you would like to declare a variable in the middle of a block, you can open a new block:
```C
void secp256k_foo(void) {
unsigned int x; /* declaration */
int y = 2*x; /* declaration */
x = 17; /* statement */
{
int a, b; /* declaration */
a = x + y; /* statement */
secp256k_bar(x, &b); /* statement */
}
}
```
* Use `unsigned int` instead of just `unsigned`.
* Use `void *ptr` instead of `void* ptr`.
* Arguments of the publicly-facing API must have a specific order defined in [include/secp256k1.h](include/secp256k1.h).
* User-facing comment lines in headers should be limited to 80 chars if possible.
* All identifiers in file scope should start with `secp256k1_`.
* Avoid trailing whitespace.
- Commits should be atomic and diffs should be easy to read. For this reason, do not mix any formatting fixes or code moves with actual code changes. Make sure each individual commit is hygienic: that it builds successfully on its own without warnings, errors, regressions, or test failures.
- New code should adhere to the style of existing, in particular surrounding, code. Other than that, we do not enforce strict rules for code formatting.
- The code conforms to C89. Most notably, that means that only `/* ... */` comments are allowed (no `//` line comments). Moreover, any declarations in a `{ ... }` block (e.g., a function) must appear at the beginning of the block before any statements. When you would like to declare a variable in the middle of a block, you can open a new block:
```C
void secp256k_foo(void) {
unsigned int x; /* declaration */
int y = 2*x; /* declaration */
x = 17; /* statement */
{
int a, b; /* declaration */
a = x + y; /* statement */
secp256k_bar(x, &b); /* statement */
}
}
```
- Use `unsigned int` instead of just `unsigned`.
- Use `void *ptr` instead of `void* ptr`.
- Arguments of the publicly-facing API must have a specific order defined in [include/secp256k1.h](include/secp256k1.h).
- User-facing comment lines in headers should be limited to 80 chars if possible.
- All identifiers in file scope should start with `secp256k1_`.
- Avoid trailing whitespace.
### Tests
@@ -101,7 +101,7 @@ To create a HTML report with coloured and annotated source code:
#### Exhaustive tests
There are tests of several functions in which a small group replaces secp256k1.
These tests are *exhaustive* since they provide all elements and scalars of the small group as input arguments (see [src/tests_exhaustive.c](src/tests_exhaustive.c)).
These tests are _exhaustive_ since they provide all elements and scalars of the small group as input arguments (see [src/tests_exhaustive.c](src/tests_exhaustive.c)).
### Benchmarks

130
external/secp256k1/README.md vendored
View File

@@ -1,5 +1,4 @@
libsecp256k1
============
# libsecp256k1
![Dependencies: None](https://img.shields.io/badge/dependencies-none-success)
[![irc.libera.chat #secp256k1](https://img.shields.io/badge/irc.libera.chat-%23secp256k1-success)](https://web.libera.chat/#secp256k1)
@@ -9,60 +8,59 @@ High-performance high-assurance C library for digital signatures and other crypt
This library is intended to be the highest quality publicly available library for cryptography on the secp256k1 curve. However, the primary focus of its development has been for usage in the Bitcoin system and usage unlike Bitcoin's may be less well tested, verified, or suffer from a less well thought out interface. Correct usage requires some care and consideration that the library is fit for your application's purpose.
Features:
* secp256k1 ECDSA signing/verification and key generation.
* Additive and multiplicative tweaking of secret/public keys.
* Serialization/parsing of secret keys, public keys, signatures.
* Constant time, constant memory access signing and public key generation.
* Derandomized ECDSA (via RFC6979 or with a caller provided function.)
* Very efficient implementation.
* Suitable for embedded systems.
* No runtime dependencies.
* Optional module for public key recovery.
* Optional module for ECDH key exchange.
* Optional module for Schnorr signatures according to [BIP-340](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki).
* Optional module for ElligatorSwift key exchange according to [BIP-324](https://github.com/bitcoin/bips/blob/master/bip-0324.mediawiki).
* Optional module for MuSig2 Schnorr multi-signatures according to [BIP-327](https://github.com/bitcoin/bips/blob/master/bip-0327.mediawiki).
Implementation details
----------------------
- secp256k1 ECDSA signing/verification and key generation.
- Additive and multiplicative tweaking of secret/public keys.
- Serialization/parsing of secret keys, public keys, signatures.
- Constant time, constant memory access signing and public key generation.
- Derandomized ECDSA (via RFC6979 or with a caller provided function.)
- Very efficient implementation.
- Suitable for embedded systems.
- No runtime dependencies.
- Optional module for public key recovery.
- Optional module for ECDH key exchange.
- Optional module for Schnorr signatures according to [BIP-340](https://github.com/bitcoin/bips/blob/master/bip-0340.mediawiki).
- Optional module for ElligatorSwift key exchange according to [BIP-324](https://github.com/bitcoin/bips/blob/master/bip-0324.mediawiki).
- Optional module for MuSig2 Schnorr multi-signatures according to [BIP-327](https://github.com/bitcoin/bips/blob/master/bip-0327.mediawiki).
* General
* No runtime heap allocation.
* Extensive testing infrastructure.
* Structured to facilitate review and analysis.
* Intended to be portable to any system with a C89 compiler and uint64_t support.
* No use of floating types.
* Expose only higher level interfaces to minimize the API surface and improve application security. ("Be difficult to use insecurely.")
* Field operations
* Optimized implementation of arithmetic modulo the curve's field size (2^256 - 0x1000003D1).
* Using 5 52-bit limbs
* Using 10 26-bit limbs (including hand-optimized assembly for 32-bit ARM, by Wladimir J. van der Laan).
* This is an experimental feature that has not received enough scrutiny to satisfy the standard of quality of this library but is made available for testing and review by the community.
* Scalar operations
* Optimized implementation without data-dependent branches of arithmetic modulo the curve's order.
* Using 4 64-bit limbs (relying on __int128 support in the compiler).
* Using 8 32-bit limbs.
* Modular inverses (both field elements and scalars) based on [safegcd](https://gcd.cr.yp.to/index.html) with some modifications, and a variable-time variant (by Peter Dettman).
* Group operations
* Point addition formula specifically simplified for the curve equation (y^2 = x^3 + 7).
* Use addition between points in Jacobian and affine coordinates where possible.
* Use a unified addition/doubling formula where necessary to avoid data-dependent branches.
* Point/x comparison without a field inversion by comparison in the Jacobian coordinate space.
* Point multiplication for verification (a*P + b*G).
* Use wNAF notation for point multiplicands.
* Use a much larger window for multiples of G, using precomputed multiples.
* Use Shamir's trick to do the multiplication with the public key and the generator simultaneously.
* Use secp256k1's efficiently-computable endomorphism to split the P multiplicand into 2 half-sized ones.
* Point multiplication for signing
* Use a precomputed table of multiples of powers of 16 multiplied with the generator, so general multiplication becomes a series of additions.
* Intended to be completely free of timing sidechannels for secret-key operations (on reasonable hardware/toolchains)
* Access the table with branch-free conditional moves so memory access is uniform.
* No data-dependent branches
* Optional runtime blinding which attempts to frustrate differential power analysis.
* The precomputed tables add and eventually subtract points for which no known scalar (secret key) is known, preventing even an attacker with control over the secret key used to control the data internally.
## Implementation details
Building with Autotools
-----------------------
- General
- No runtime heap allocation.
- Extensive testing infrastructure.
- Structured to facilitate review and analysis.
- Intended to be portable to any system with a C89 compiler and uint64_t support.
- No use of floating types.
- Expose only higher level interfaces to minimize the API surface and improve application security. ("Be difficult to use insecurely.")
- Field operations
- Optimized implementation of arithmetic modulo the curve's field size (2^256 - 0x1000003D1).
- Using 5 52-bit limbs
- Using 10 26-bit limbs (including hand-optimized assembly for 32-bit ARM, by Wladimir J. van der Laan).
- This is an experimental feature that has not received enough scrutiny to satisfy the standard of quality of this library but is made available for testing and review by the community.
- Scalar operations
- Optimized implementation without data-dependent branches of arithmetic modulo the curve's order.
- Using 4 64-bit limbs (relying on \_\_int128 support in the compiler).
- Using 8 32-bit limbs.
- Modular inverses (both field elements and scalars) based on [safegcd](https://gcd.cr.yp.to/index.html) with some modifications, and a variable-time variant (by Peter Dettman).
- Group operations
- Point addition formula specifically simplified for the curve equation (y^2 = x^3 + 7).
- Use addition between points in Jacobian and affine coordinates where possible.
- Use a unified addition/doubling formula where necessary to avoid data-dependent branches.
- Point/x comparison without a field inversion by comparison in the Jacobian coordinate space.
- Point multiplication for verification (a*P + b*G).
- Use wNAF notation for point multiplicands.
- Use a much larger window for multiples of G, using precomputed multiples.
- Use Shamir's trick to do the multiplication with the public key and the generator simultaneously.
- Use secp256k1's efficiently-computable endomorphism to split the P multiplicand into 2 half-sized ones.
- Point multiplication for signing
- Use a precomputed table of multiples of powers of 16 multiplied with the generator, so general multiplication becomes a series of additions.
- Intended to be completely free of timing sidechannels for secret-key operations (on reasonable hardware/toolchains)
- Access the table with branch-free conditional moves so memory access is uniform.
- No data-dependent branches
- Optional runtime blinding which attempts to frustrate differential power analysis.
- The precomputed tables add and eventually subtract points for which no known scalar (secret key) is known, preventing even an attacker with control over the secret key used to control the data internally.
## Building with Autotools
$ ./autogen.sh
$ ./configure
@@ -72,8 +70,7 @@ Building with Autotools
To compile optional modules (such as Schnorr signatures), you need to run `./configure` with additional flags (such as `--enable-module-schnorrsig`). Run `./configure --help` to see the full list of available flags.
Building with CMake (experimental)
----------------------------------
## Building with CMake (experimental)
To maintain a pristine source tree, CMake encourages to perform an out-of-source build by using a separate dedicated build tree.
@@ -109,18 +106,19 @@ In "Developer Command Prompt for VS 2022":
>cmake -G "Visual Studio 17 2022" -A x64 -S . -B build
>cmake --build build --config RelWithDebInfo
Usage examples
-----------
## Usage examples
Usage examples can be found in the [examples](examples) directory. To compile them you need to configure with `--enable-examples`.
* [ECDSA example](examples/ecdsa.c)
* [Schnorr signatures example](examples/schnorr.c)
* [Deriving a shared secret (ECDH) example](examples/ecdh.c)
* [ElligatorSwift key exchange example](examples/ellswift.c)
- [ECDSA example](examples/ecdsa.c)
- [Schnorr signatures example](examples/schnorr.c)
- [Deriving a shared secret (ECDH) example](examples/ecdh.c)
- [ElligatorSwift key exchange example](examples/ellswift.c)
To compile the Schnorr signature and ECDH examples, you also need to configure with `--enable-module-schnorrsig` and `--enable-module-ecdh`.
Benchmark
------------
## Benchmark
If configured with `--enable-benchmark` (which is the default), binaries for benchmarking the libsecp256k1 functions will be present in the root directory after the build.
To print the benchmark result to the command line:
@@ -131,12 +129,10 @@ To create a CSV file for the benchmark result :
$ ./bench_name | sed '2d;s/ \{1,\}//g' > bench_name.csv
Reporting a vulnerability
------------
## Reporting a vulnerability
See [SECURITY.md](SECURITY.md)
Contributing to libsecp256k1
------------
## Contributing to libsecp256k1
See [CONTRIBUTING.md](CONTRIBUTING.md)

10
external/secp256k1/SECURITY.md vendored
View File

@@ -6,10 +6,10 @@ To report security issues send an email to secp256k1-security@bitcoincore.org (n
The following keys may be used to communicate sensitive information to developers:
| Name | Fingerprint |
|------|-------------|
| Pieter Wuille | 133E AC17 9436 F14A 5CF1 B794 860F EB80 4E66 9320 |
| Jonas Nick | 36C7 1A37 C9D9 88BD E825 08D9 B1A7 0E4F 8DCD 0366 |
| Tim Ruffing | 09E0 3F87 1092 E40E 106E 902B 33BC 86AB 80FF 5516 |
| Name | Fingerprint |
| ------------- | ------------------------------------------------- |
| Pieter Wuille | 133E AC17 9436 F14A 5CF1 B794 860F EB80 4E66 9320 |
| Jonas Nick | 36C7 1A37 C9D9 88BD E825 08D9 B1A7 0E4F 8DCD 0366 |
| Tim Ruffing | 09E0 3F87 1092 E40E 106E 902B 33BC 86AB 80FF 5516 |
You can import a key by running the following command with that individual’s fingerprint: `gpg --keyserver hkps://keys.openpgp.org --recv-keys "<fingerprint>"` Ensure that you put quotes around fingerprints containing spaces.

410
external/secp256k1/doc/ellswift.md vendored
View File

@@ -5,17 +5,17 @@ construction in the
["SwiftEC: Shallue–van de Woestijne Indifferentiable Function To Elliptic Curves"](https://eprint.iacr.org/2022/759)
paper by Jorge Chávez-Saab, Francisco Rodríguez-Henríquez, and Mehdi Tibouchi.
* [1. Introduction](#1-introduction)
* [2. The decoding function](#2-the-decoding-function)
+ [2.1 Decoding for `secp256k1`](#21-decoding-for-secp256k1)
* [3. The encoding function](#3-the-encoding-function)
+ [3.1 Switching to *v, w* coordinates](#31-switching-to-v-w-coordinates)
+ [3.2 Avoiding computing all inverses](#32-avoiding-computing-all-inverses)
+ [3.3 Finding the inverse](#33-finding-the-inverse)
+ [3.4 Dealing with special cases](#34-dealing-with-special-cases)
+ [3.5 Encoding for `secp256k1`](#35-encoding-for-secp256k1)
* [4. Encoding and decoding full *(x, y)* coordinates](#4-encoding-and-decoding-full-x-y-coordinates)
+ [4.1 Full *(x, y)* coordinates for `secp256k1`](#41-full-x-y-coordinates-for-secp256k1)
- [1. Introduction](#1-introduction)
- [2. The decoding function](#2-the-decoding-function)
- [2.1 Decoding for `secp256k1`](#21-decoding-for-secp256k1)
- [3. The encoding function](#3-the-encoding-function)
- [3.1 Switching to _v, w_ coordinates](#31-switching-to-v-w-coordinates)
- [3.2 Avoiding computing all inverses](#32-avoiding-computing-all-inverses)
- [3.3 Finding the inverse](#33-finding-the-inverse)
- [3.4 Dealing with special cases](#34-dealing-with-special-cases)
- [3.5 Encoding for `secp256k1`](#35-encoding-for-secp256k1)
- [4. Encoding and decoding full _(x, y)_ coordinates](#4-encoding-and-decoding-full-x-y-coordinates)
- [4.1 Full _(x, y)_ coordinates for `secp256k1`](#41-full-x-y-coordinates-for-secp256k1)
## 1. Introduction
@@ -34,13 +34,14 @@ are taken modulo $p$), and then evaluating $F_u(t)$, which for every $u$ and $t$
x-coordinate on the curve. The functions $F_u$ will be defined in [Section 2](#2-the-decoding-function).
**Encoding** a given $x$ coordinate is conceptually done as follows:
* Loop:
* Pick a uniformly random field element $u.$
* Compute the set $L = F_u^{-1}(x)$ of $t$ values for which $F_u(t) = x$, which may have up to *8* elements.
* With probability $1 - \dfrac{\\#L}{8}$, restart the loop.
* Select a uniformly random $t \in L$ and return $(u, t).$
This is the *ElligatorSwift* algorithm, here given for just x-coordinates. An extension to full
- Loop:
- Pick a uniformly random field element $u.$
- Compute the set $L = F_u^{-1}(x)$ of $t$ values for which $F_u(t) = x$, which may have up to _8_ elements.
- With probability $1 - \dfrac{\\#L}{8}$, restart the loop.
- Select a uniformly random $t \in L$ and return $(u, t).$
This is the _ElligatorSwift_ algorithm, here given for just x-coordinates. An extension to full
$(x, y)$ points will be given in [Section 4](#4-encoding-and-decoding-full-x-y-coordinates).
The algorithm finds a uniformly random $(u, t)$ among (almost all) those
for which $F_u(t) = x.$ Section 3.2 in the paper proves that the number of such encodings for
@@ -50,37 +51,40 @@ almost all x-coordinates on the curve (all but at most 39) is close to two times
## 2. The decoding function
First some definitions:
* $\mathbb{F}$ is the finite field of size $q$, of characteristic 5 or more, and $q \equiv 1 \mod 3.$
* For `secp256k1`, $q = 2^{256} - 2^{32} - 977$, which satisfies that requirement.
* Let $E$ be the elliptic curve of points $(x, y) \in \mathbb{F}^2$ for which $y^2 = x^3 + ax + b$, with $a$ and $b$
- $\mathbb{F}$ is the finite field of size $q$, of characteristic 5 or more, and $q \equiv 1 \mod 3.$
- For `secp256k1`, $q = 2^{256} - 2^{32} - 977$, which satisfies that requirement.
- Let $E$ be the elliptic curve of points $(x, y) \in \mathbb{F}^2$ for which $y^2 = x^3 + ax + b$, with $a$ and $b$
public constants, for which $\Delta_E = -16(4a^3 + 27b^2)$ is a square, and at least one of $(-b \pm \sqrt{-3 \Delta_E} / 36)/2$ is a square.
This implies that the order of $E$ is either odd, or a multiple of *4*.
This implies that the order of $E$ is either odd, or a multiple of _4_.
If $a=0$, this condition is always fulfilled.
* For `secp256k1`, $a=0$ and $b=7.$
* Let the function $g(x) = x^3 + ax + b$, so the $E$ curve equation is also $y^2 = g(x).$
* Let the function $h(x) = 3x^3 + 4a.$
* Define $V$ as the set of solutions $(x_1, x_2, x_3, z)$ to $z^2 = g(x_1)g(x_2)g(x_3).$
* Define $S_u$ as the set of solutions $(X, Y)$ to $X^2 + h(u)Y^2 = -g(u)$ and $Y \neq 0.$
* $P_u$ is a function from $\mathbb{F}$ to $S_u$ that will be defined below.
* $\psi_u$ is a function from $S_u$ to $V$ that will be defined below.
- For `secp256k1`, $a=0$ and $b=7.$
- Let the function $g(x) = x^3 + ax + b$, so the $E$ curve equation is also $y^2 = g(x).$
- Let the function $h(x) = 3x^3 + 4a.$
- Define $V$ as the set of solutions $(x_1, x_2, x_3, z)$ to $z^2 = g(x_1)g(x_2)g(x_3).$
- Define $S_u$ as the set of solutions $(X, Y)$ to $X^2 + h(u)Y^2 = -g(u)$ and $Y \neq 0.$
- $P_u$ is a function from $\mathbb{F}$ to $S_u$ that will be defined below.
- $\psi_u$ is a function from $S_u$ to $V$ that will be defined below.
**Note**: In the paper:
* $F_u$ corresponds to $F_{0,u}$ there.
* $P_u(t)$ is called $P$ there.
* All $S_u$ sets together correspond to $S$ there.
* All $\psi_u$ functions together (operating on elements of $S$) correspond to $\psi$ there.
- $F_u$ corresponds to $F_{0,u}$ there.
- $P_u(t)$ is called $P$ there.
- All $S_u$ sets together correspond to $S$ there.
- All $\psi_u$ functions together (operating on elements of $S$) correspond to $\psi$ there.
Note that for $V$, the left hand side of the equation $z^2$ is square, and thus the right
hand must also be square. As multiplying non-squares results in a square in $\mathbb{F}$,
out of the three right-hand side factors an even number must be non-squares.
This implies that exactly *1* or exactly *3* out of
This implies that exactly _1_ or exactly _3_ out of
$\\{g(x_1), g(x_2), g(x_3)\\}$ must be square, and thus that for any $(x_1,x_2,x_3,z) \in V$,
at least one of $\\{x_1, x_2, x_3\\}$ must be a valid x-coordinate on $E.$ There is one exception
to this, namely when $z=0$, but even then one of the three values is a valid x-coordinate.
**Define** the decoding function $F_u(t)$ as:
* Let $(x_1, x_2, x_3, z) = \psi_u(P_u(t)).$
* Return the first element $x$ of $(x_3, x_2, x_1)$ which is a valid x-coordinate on $E$ (i.e., $g(x)$ is square).
- Let $(x_1, x_2, x_3, z) = \psi_u(P_u(t)).$
- Return the first element $x$ of $(x_3, x_2, x_1)$ which is a valid x-coordinate on $E$ (i.e., $g(x)$ is square).
$P_u(t) = (X(u, t), Y(u, t))$, where:
@@ -98,12 +102,13 @@ Y(u, t) & = & \left\\{\begin{array}{ll}
$$
$P_u(t)$ is defined:
* For $a=0$, unless:
* $u = 0$ or $t = 0$ (division by zero)
* $g(u) = -t^2$ (would give $Y=0$).
* For $a \neq 0$, unless:
* $X_0(u) = 0$ or $h(u)t^2 = -1$ (division by zero)
* $Y_0(u) (1 - h(u)t^2) = 2X_0(u)t$ (would give $Y=0$).
- For $a=0$, unless:
- $u = 0$ or $t = 0$ (division by zero)
- $g(u) = -t^2$ (would give $Y=0$).
- For $a \neq 0$, unless:
- $X_0(u) = 0$ or $h(u)t^2 = -1$ (division by zero)
- $Y_0(u) (1 - h(u)t^2) = 2X_0(u)t$ (would give $Y=0$).
The functions $X_0(u)$ and $Y_0(u)$ are defined in Appendix A of the paper, and depend on various properties of $E.$
@@ -123,20 +128,22 @@ $$
Put together and specialized for $a=0$ curves, decoding $(u, t)$ to an x-coordinate is:
**Define** $F_u(t)$ as:
* Let $X = \dfrac{u^3 + b - t^2}{2t}.$
* Let $Y = \dfrac{X + t}{u\sqrt{-3}}.$
* Return the first $x$ in $(u + 4Y^2, \dfrac{-X}{2Y} - \dfrac{u}{2}, \dfrac{X}{2Y} - \dfrac{u}{2})$ for which $g(x)$ is square.
- Let $X = \dfrac{u^3 + b - t^2}{2t}.$
- Let $Y = \dfrac{X + t}{u\sqrt{-3}}.$
- Return the first $x$ in $(u + 4Y^2, \dfrac{-X}{2Y} - \dfrac{u}{2}, \dfrac{X}{2Y} - \dfrac{u}{2})$ for which $g(x)$ is square.
To make sure that every input decodes to a valid x-coordinate, we remap the inputs in case
$P_u$ is not defined (when $u=0$, $t=0$, or $g(u) = -t^2$):
**Define** $F_u(t)$ as:
* Let $u'=u$ if $u \neq 0$; $1$ otherwise (guaranteeing $u' \neq 0$).
* Let $t'=t$ if $t \neq 0$; $1$ otherwise (guaranteeing $t' \neq 0$).
* Let $t''=t'$ if $g(u') \neq -t'^2$; $2t'$ otherwise (guaranteeing $t'' \neq 0$ and $g(u') \neq -t''^2$).
* Let $X = \dfrac{u'^3 + b - t''^2}{2t''}.$
* Let $Y = \dfrac{X + t''}{u'\sqrt{-3}}.$
* Return the first $x$ in $(u' + 4Y^2, \dfrac{-X}{2Y} - \dfrac{u'}{2}, \dfrac{X}{2Y} - \dfrac{u'}{2})$ for which $x^3 + b$ is square.
- Let $u'=u$ if $u \neq 0$; $1$ otherwise (guaranteeing $u' \neq 0$).
- Let $t'=t$ if $t \neq 0$; $1$ otherwise (guaranteeing $t' \neq 0$).
- Let $t''=t'$ if $g(u') \neq -t'^2$; $2t'$ otherwise (guaranteeing $t'' \neq 0$ and $g(u') \neq -t''^2$).
- Let $X = \dfrac{u'^3 + b - t''^2}{2t''}.$
- Let $Y = \dfrac{X + t''}{u'\sqrt{-3}}.$
- Return the first $x$ in $(u' + 4Y^2, \dfrac{-X}{2Y} - \dfrac{u'}{2}, \dfrac{X}{2Y} - \dfrac{u'}{2})$ for which $x^3 + b$ is square.
The choices here are not strictly necessary. Just returning a fixed constant in any of the undefined cases would suffice,
but the approach here is simple enough and gives fairly uniform output even in these cases.
@@ -150,10 +157,11 @@ in `secp256k1_ellswift_xswiftec_var` (which outputs the actual x-coordinate).
## 3. The encoding function
To implement $F_u^{-1}(x)$, the function to find the set of inverses $t$ for which $F_u(t) = x$, we have to reverse the process:
* Find all the $(X, Y) \in S_u$ that could have given rise to $x$, through the $x_1$, $x_2$, or $x_3$ formulas in $\psi_u.$
* Map those $(X, Y)$ solutions to $t$ values using $P_u^{-1}(X, Y).$
* For each of the found $t$ values, verify that $F_u(t) = x.$
* Return the remaining $t$ values.
- Find all the $(X, Y) \in S_u$ that could have given rise to $x$, through the $x_1$, $x_2$, or $x_3$ formulas in $\psi_u.$
- Map those $(X, Y)$ solutions to $t$ values using $P_u^{-1}(X, Y).$
- For each of the found $t$ values, verify that $F_u(t) = x.$
- Return the remaining $t$ values.
The function $P_u^{-1}$, which finds $t$ given $(X, Y) \in S_u$, is significantly simpler than $P_u:$
@@ -185,13 +193,14 @@ precedence over both. Because of this, the $g(-u-x)$ being square test for $x_1$
values round-trip back to the input $x$ correctly. This is the reason for choosing the $(x_3, x_2, x_1)$ precedence order in the decoder;
any order which does not place $x_3$ first requires more complicated round-trip checks in the encoder.
### 3.1 Switching to *v, w* coordinates
### 3.1 Switching to _v, w_ coordinates
Before working out the formulas for all this, we switch to different variables for $S_u.$ Let $v = (X/Y - u)/2$, and
$w = 2Y.$ Or in the other direction, $X = w(u/2 + v)$ and $Y = w/2:$
* $S_u'$ becomes the set of $(v, w)$ for which $w^2 (u^2 + uv + v^2 + a) = -g(u)$ and $w \neq 0.$
* For $a=0$ curves, $P_u^{-1}$ can be stated for $(v,w)$ as $P_u^{'-1}(v, w) = w\left(\frac{\sqrt{-3}-1}{2}u - v\right).$
* $\psi_u$ can be stated for $(v, w)$ as $\psi_u'(v, w) = (x_1, x_2, x_3, z)$, where
- $S_u'$ becomes the set of $(v, w)$ for which $w^2 (u^2 + uv + v^2 + a) = -g(u)$ and $w \neq 0.$
- For $a=0$ curves, $P_u^{-1}$ can be stated for $(v,w)$ as $P_u^{'-1}(v, w) = w\left(\frac{\sqrt{-3}-1}{2}u - v\right).$
- $\psi_u$ can be stated for $(v, w)$ as $\psi_u'(v, w) = (x_1, x_2, x_3, z)$, where
$$
\begin{array}{lcl}
@@ -204,34 +213,37 @@ $$
We can now write the expressions for finding $(v, w)$ given $x$ explicitly, by solving each of the $\\{x_1, x_2, x_3\\}$
expressions for $v$ or $w$, and using the $S_u'$ equation to find the other variable:
* Assuming $x = x_1$, we find $v = x$ and $w = \pm\sqrt{-g(u)/(u^2 + uv + v^2 + a)}$ (two solutions).
* Assuming $x = x_2$, we find $v = -u-x$ and $w = \pm\sqrt{-g(u)/(u^2 + uv + v^2 + a)}$ (two solutions).
* Assuming $x = x_3$, we find $w = \pm\sqrt{x-u}$ and $v = -u/2 \pm \sqrt{-w^2(4g(u) + w^2h(u))}/(2w^2)$ (four solutions).
- Assuming $x = x_1$, we find $v = x$ and $w = \pm\sqrt{-g(u)/(u^2 + uv + v^2 + a)}$ (two solutions).
- Assuming $x = x_2$, we find $v = -u-x$ and $w = \pm\sqrt{-g(u)/(u^2 + uv + v^2 + a)}$ (two solutions).
- Assuming $x = x_3$, we find $w = \pm\sqrt{x-u}$ and $v = -u/2 \pm \sqrt{-w^2(4g(u) + w^2h(u))}/(2w^2)$ (four solutions).
### 3.2 Avoiding computing all inverses
The *ElligatorSwift* algorithm as stated in Section 1 requires the computation of $L = F_u^{-1}(x)$ (the
The _ElligatorSwift_ algorithm as stated in Section 1 requires the computation of $L = F_u^{-1}(x)$ (the
set of all $t$ such that $(u, t)$ decode to $x$) in full. This is unnecessary.
Observe that the procedure of restarting with probability $(1 - \frac{\\#L}{8})$ and otherwise returning a
uniformly random element from $L$ is actually equivalent to always padding $L$ with $\bot$ values up to length 8,
picking a uniformly random element from that, restarting whenever $\bot$ is picked:
**Define** *ElligatorSwift(x)* as:
* Loop:
* Pick a uniformly random field element $u.$
* Compute the set $L = F_u^{-1}(x).$
* Let $T$ be the 8-element vector consisting of the elements of $L$, plus $8 - \\#L$ times $\\{\bot\\}.$
* Select a uniformly random $t \in T.$
* If $t \neq \bot$, return $(u, t)$; restart loop otherwise.
**Define** _ElligatorSwift(x)_ as:
- Loop:
- Pick a uniformly random field element $u.$
- Compute the set $L = F_u^{-1}(x).$
- Let $T$ be the 8-element vector consisting of the elements of $L$, plus $8 - \\#L$ times $\\{\bot\\}.$
- Select a uniformly random $t \in T.$
- If $t \neq \bot$, return $(u, t)$; restart loop otherwise.
Now notice that the order of elements in $T$ does not matter, as all we do is pick a uniformly
random element in it, so we do not need to have all $\bot$ values at the end.
As we have 8 distinct formulas for finding $(v, w)$ (taking the variants due to $\pm$ into account),
we can associate every index in $T$ with exactly one of those formulas, making sure that:
* Formulas that yield no solutions (due to division by zero or non-existing square roots) or invalid solutions are made to return $\bot.$
* For the $x_1$ and $x_2$ cases, if $g(-u-x)$ is a square, $\bot$ is returned instead (the round-trip check).
* In case multiple formulas would return the same non- $\bot$ result, all but one of those must be turned into $\bot$ to avoid biasing those.
- Formulas that yield no solutions (due to division by zero or non-existing square roots) or invalid solutions are made to return $\bot.$
- For the $x_1$ and $x_2$ cases, if $g(-u-x)$ is a square, $\bot$ is returned instead (the round-trip check).
- In case multiple formulas would return the same non- $\bot$ result, all but one of those must be turned into $\bot$ to avoid biasing those.
The last condition above only occurs with negligible probability for cryptographically-sized curves, but is interesting
to take into account as it allows exhaustive testing in small groups. See [Section 3.4](#34-dealing-with-special-cases)
@@ -240,12 +252,13 @@ for an analysis of all the negligible cases.
If we define $T = (G_{0,u}(x), G_{1,u}(x), \ldots, G_{7,u}(x))$, with each $G_{i,u}$ matching one of the formulas,
the loop can be simplified to only compute one of the inverses instead of all of them:
**Define** *ElligatorSwift(x)* as:
* Loop:
* Pick a uniformly random field element $u.$
* Pick a uniformly random integer $c$ in $[0,8).$
* Let $t = G_{c,u}(x).$
* If $t \neq \bot$, return $(u, t)$; restart loop otherwise.
**Define** _ElligatorSwift(x)_ as:
- Loop:
- Pick a uniformly random field element $u.$
- Pick a uniformly random integer $c$ in $[0,8).$
- Let $t = G_{c,u}(x).$
- If $t \neq \bot$, return $(u, t)$; restart loop otherwise.
This is implemented in `secp256k1_ellswift_xelligatorswift_var`.
@@ -256,18 +269,19 @@ Those are then repeated as $c=4$ through $c=7$ for the other sign of $w$ (noting
Ignoring the negligible cases, we get:
**Define** $G_{c,u}(x)$ as:
* If $c \in \\{0, 1, 4, 5\\}$ (for $x_1$ and $x_2$ formulas):
* If $g(-u-x)$ is square, return $\bot$ (as $x_3$ would be valid and take precedence).
* If $c \in \\{0, 4\\}$ (the $x_1$ formula) let $v = x$, otherwise let $v = -u-x$ (the $x_2$ formula)
* Let $s = -g(u)/(u^2 + uv + v^2 + a)$ (using $s = w^2$ in what follows).
* Otherwise, when $c \in \\{2, 3, 6, 7\\}$ (for $x_3$ formulas):
* Let $s = x-u.$
* Let $r = \sqrt{-s(4g(u) + sh(u))}.$
* Let $v = (r/s - u)/2$ if $c \in \\{3, 7\\}$; $(-r/s - u)/2$ otherwise.
* Let $w = \sqrt{s}.$
* Depending on $c:$
* If $c \in \\{0, 1, 2, 3\\}:$ return $P_u^{'-1}(v, w).$
* If $c \in \\{4, 5, 6, 7\\}:$ return $P_u^{'-1}(v, -w).$
- If $c \in \\{0, 1, 4, 5\\}$ (for $x_1$ and $x_2$ formulas):
- If $g(-u-x)$ is square, return $\bot$ (as $x_3$ would be valid and take precedence).
- If $c \in \\{0, 4\\}$ (the $x_1$ formula) let $v = x$, otherwise let $v = -u-x$ (the $x_2$ formula)
- Let $s = -g(u)/(u^2 + uv + v^2 + a)$ (using $s = w^2$ in what follows).
- Otherwise, when $c \in \\{2, 3, 6, 7\\}$ (for $x_3$ formulas):
- Let $s = x-u.$
- Let $r = \sqrt{-s(4g(u) + sh(u))}.$
- Let $v = (r/s - u)/2$ if $c \in \\{3, 7\\}$; $(-r/s - u)/2$ otherwise.
- Let $w = \sqrt{s}.$
- Depending on $c:$
- If $c \in \\{0, 1, 2, 3\\}:$ return $P_u^{'-1}(v, w).$
- If $c \in \\{4, 5, 6, 7\\}:$ return $P_u^{'-1}(v, -w).$
Whenever a square root of a non-square is taken, $\bot$ is returned; for both square roots this happens with roughly
50% on random inputs. Similarly, when a division by 0 would occur, $\bot$ is returned as well; this will only happen
@@ -284,20 +298,21 @@ transformation. Furthermore, that transformation has no effect on $s$ in the fir
as $u^2 + ux + x^2 + a = u^2 + u(-u-x) + (-u-x)^2 + a.$ Thus we can extract it out and move it down:
**Define** $G_{c,u}(x)$ as:
* If $c \in \\{0, 1, 4, 5\\}:$
* If $g(-u-x)$ is square, return $\bot.$
* Let $s = -g(u)/(u^2 + ux + x^2 + a).$
* Let $v = x.$
* Otherwise, when $c \in \\{2, 3, 6, 7\\}:$
* Let $s = x-u.$
* Let $r = \sqrt{-s(4g(u) + sh(u))}.$
* Let $v = (r/s - u)/2.$
* Let $w = \sqrt{s}.$
* Depending on $c:$
* If $c \in \\{0, 2\\}:$ return $P_u^{'-1}(v, w).$
* If $c \in \\{1, 3\\}:$ return $P_u^{'-1}(-u-v, w).$
* If $c \in \\{4, 6\\}:$ return $P_u^{'-1}(v, -w).$
* If $c \in \\{5, 7\\}:$ return $P_u^{'-1}(-u-v, -w).$
- If $c \in \\{0, 1, 4, 5\\}:$
- If $g(-u-x)$ is square, return $\bot.$
- Let $s = -g(u)/(u^2 + ux + x^2 + a).$
- Let $v = x.$
- Otherwise, when $c \in \\{2, 3, 6, 7\\}:$
- Let $s = x-u.$
- Let $r = \sqrt{-s(4g(u) + sh(u))}.$
- Let $v = (r/s - u)/2.$
- Let $w = \sqrt{s}.$
- Depending on $c:$
- If $c \in \\{0, 2\\}:$ return $P_u^{'-1}(v, w).$
- If $c \in \\{1, 3\\}:$ return $P_u^{'-1}(-u-v, w).$
- If $c \in \\{4, 6\\}:$ return $P_u^{'-1}(v, -w).$
- If $c \in \\{5, 7\\}:$ return $P_u^{'-1}(-u-v, -w).$
This shows there will always be exactly 0, 4, or 8 $t$ values for a given $(u, x)$ input.
There can be 0, 1, or 2 $(v, w)$ pairs before invoking $P_u^{'-1}$, and each results in 4 distinct $t$ values.
@@ -310,58 +325,60 @@ we analyse them here. They generally fall into two categories: cases in which th
do not decode back to $x$ (or at least cannot guarantee that they do), and cases in which the encoder might produce the same
$t$ value for multiple $c$ inputs (thereby biasing that encoding):
* In the branch for $x_1$ and $x_2$ (where $c \in \\{0, 1, 4, 5\\}$):
* When $g(u) = 0$, we would have $s=w=Y=0$, which is not on $S_u.$ This is only possible on even-ordered curves.
- In the branch for $x_1$ and $x_2$ (where $c \in \\{0, 1, 4, 5\\}$):
- When $g(u) = 0$, we would have $s=w=Y=0$, which is not on $S_u.$ This is only possible on even-ordered curves.
Excluding this also removes the one condition under which the simplified check for $x_3$ on the curve
fails (namely when $g(x_1)=g(x_2)=0$ but $g(x_3)$ is not square).
This does exclude some valid encodings: when both $g(u)=0$ and $u^2+ux+x^2+a=0$ (also implying $g(x)=0$),
the $S_u'$ equation degenerates to $0 = 0$, and many valid $t$ values may exist. Yet, these cannot be targeted uniformly by the
encoder anyway as there will generally be more than 8.
* When $g(x) = 0$, the same $t$ would be produced as in the $x_3$ branch (where $c \in \\{2, 3, 6, 7\\}$) which we give precedence
- When $g(x) = 0$, the same $t$ would be produced as in the $x_3$ branch (where $c \in \\{2, 3, 6, 7\\}$) which we give precedence
as it can deal with $g(u)=0$.
This is again only possible on even-ordered curves.
* In the branch for $x_3$ (where $c \in \\{2, 3, 6, 7\\}$):
* When $s=0$, a division by zero would occur.
* When $v = -u-v$ and $c \in \\{3, 7\\}$, the same $t$ would be returned as in the $c \in \\{2, 6\\}$ cases.
- In the branch for $x_3$ (where $c \in \\{2, 3, 6, 7\\}$):
- When $s=0$, a division by zero would occur.
- When $v = -u-v$ and $c \in \\{3, 7\\}$, the same $t$ would be returned as in the $c \in \\{2, 6\\}$ cases.
It is equivalent to checking whether $r=0$.
This cannot occur in the $x_1$ or $x_2$ branches, as it would trigger the $g(-u-x)$ is square condition.
A similar concern for $w = -w$ does not exist, as $w=0$ is already impossible in both branches: in the first
it requires $g(u)=0$ which is already outlawed on even-ordered curves and impossible on others; in the second it would trigger division by zero.
* Curve-specific special cases also exist that need to be rejected, because they result in $(u,t)$ which is invalid to the decoder, or because of division by zero in the encoder:
* For $a=0$ curves, when $u=0$ or when $t=0$. The latter can only be reached by the encoder when $g(u)=0$, which requires an even-ordered curve.
* For $a \neq 0$ curves, when $X_0(u)=0$, when $h(u)t^2 = -1$, or when $w(u + 2v) = 2X_0(u)$ while also either $w \neq 2Y_0(u)$ or $h(u)=0$.
- Curve-specific special cases also exist that need to be rejected, because they result in $(u,t)$ which is invalid to the decoder, or because of division by zero in the encoder:
- For $a=0$ curves, when $u=0$ or when $t=0$. The latter can only be reached by the encoder when $g(u)=0$, which requires an even-ordered curve.
- For $a \neq 0$ curves, when $X_0(u)=0$, when $h(u)t^2 = -1$, or when $w(u + 2v) = 2X_0(u)$ while also either $w \neq 2Y_0(u)$ or $h(u)=0$.
**Define** a version of $G_{c,u}(x)$ which deals with all these cases:
* If $a=0$ and $u=0$, return $\bot.$
* If $a \neq 0$ and $X_0(u)=0$, return $\bot.$
* If $c \in \\{0, 1, 4, 5\\}:$
* If $g(u) = 0$ or $g(x) = 0$, return $\bot$ (even curves only).
* If $g(-u-x)$ is square, return $\bot.$
* Let $s = -g(u)/(u^2 + ux + x^2 + a)$ (cannot cause division by zero).
* Let $v = x.$
* Otherwise, when $c \in \\{2, 3, 6, 7\\}:$
* Let $s = x-u.$
* Let $r = \sqrt{-s(4g(u) + sh(u))}$; return $\bot$ if not square.
* If $c \in \\{3, 7\\}$ and $r=0$, return $\bot.$
* If $s = 0$, return $\bot.$
* Let $v = (r/s - u)/2.$
* Let $w = \sqrt{s}$; return $\bot$ if not square.
* If $a \neq 0$ and $w(u+2v) = 2X_0(u)$ and either $w \neq 2Y_0(u)$ or $h(u) = 0$, return $\bot.$
* Depending on $c:$
* If $c \in \\{0, 2\\}$, let $t = P_u^{'-1}(v, w).$
* If $c \in \\{1, 3\\}$, let $t = P_u^{'-1}(-u-v, w).$
* If $c \in \\{4, 6\\}$, let $t = P_u^{'-1}(v, -w).$
* If $c \in \\{5, 7\\}$, let $t = P_u^{'-1}(-u-v, -w).$
* If $a=0$ and $t=0$, return $\bot$ (even curves only).
* If $a \neq 0$ and $h(u)t^2 = -1$, return $\bot.$
* Return $t.$
- If $a=0$ and $u=0$, return $\bot.$
- If $a \neq 0$ and $X_0(u)=0$, return $\bot.$
- If $c \in \\{0, 1, 4, 5\\}:$
- If $g(u) = 0$ or $g(x) = 0$, return $\bot$ (even curves only).
- If $g(-u-x)$ is square, return $\bot.$
- Let $s = -g(u)/(u^2 + ux + x^2 + a)$ (cannot cause division by zero).
- Let $v = x.$
- Otherwise, when $c \in \\{2, 3, 6, 7\\}:$
- Let $s = x-u.$
- Let $r = \sqrt{-s(4g(u) + sh(u))}$; return $\bot$ if not square.
- If $c \in \\{3, 7\\}$ and $r=0$, return $\bot.$
- If $s = 0$, return $\bot.$
- Let $v = (r/s - u)/2.$
- Let $w = \sqrt{s}$; return $\bot$ if not square.
- If $a \neq 0$ and $w(u+2v) = 2X_0(u)$ and either $w \neq 2Y_0(u)$ or $h(u) = 0$, return $\bot.$
- Depending on $c:$
- If $c \in \\{0, 2\\}$, let $t = P_u^{'-1}(v, w).$
- If $c \in \\{1, 3\\}$, let $t = P_u^{'-1}(-u-v, w).$
- If $c \in \\{4, 6\\}$, let $t = P_u^{'-1}(v, -w).$
- If $c \in \\{5, 7\\}$, let $t = P_u^{'-1}(-u-v, -w).$
- If $a=0$ and $t=0$, return $\bot$ (even curves only).
- If $a \neq 0$ and $h(u)t^2 = -1$, return $\bot.$
- Return $t.$
Given any $u$, using this algorithm over all $x$ and $c$ values, every $t$ value will be reached exactly once,
for an $x$ for which $F_u(t) = x$ holds, except for these cases that will not be reached:
* All cases where $P_u(t)$ is not defined:
* For $a=0$ curves, when $u=0$, $t=0$, or $g(u) = -t^2.$
* For $a \neq 0$ curves, when $h(u)t^2 = -1$, $X_0(u) = 0$, or $Y_0(u) (1 - h(u) t^2) = 2X_0(u)t.$
* When $g(u)=0$, the potentially many $t$ values that decode to an $x$ satisfying $g(x)=0$ using the $x_2$ formula. These were excluded by the $g(u)=0$ condition in the $c \in \\{0, 1, 4, 5\\}$ branch.
- All cases where $P_u(t)$ is not defined:
- For $a=0$ curves, when $u=0$, $t=0$, or $g(u) = -t^2.$
- For $a \neq 0$ curves, when $h(u)t^2 = -1$, $X_0(u) = 0$, or $Y_0(u) (1 - h(u) t^2) = 2X_0(u)t.$
- When $g(u)=0$, the potentially many $t$ values that decode to an $x$ satisfying $g(x)=0$ using the $x_2$ formula. These were excluded by the $g(u)=0$ condition in the $c \in \\{0, 1, 4, 5\\}$ branch.
These cases form a negligible subset of all $(u, t)$ for cryptographically sized curves.
@@ -370,40 +387,42 @@ These cases form a negligible subset of all $(u, t)$ for cryptographically sized
Specialized for odd-ordered $a=0$ curves:
**Define** $G_{c,u}(x)$ as:
* If $u=0$, return $\bot.$
* If $c \in \\{0, 1, 4, 5\\}:$
* If $(-u-x)^3 + b$ is square, return $\bot$
* Let $s = -(u^3 + b)/(u^2 + ux + x^2)$ (cannot cause division by 0).
* Let $v = x.$
* Otherwise, when $c \in \\{2, 3, 6, 7\\}:$
* Let $s = x-u.$
* Let $r = \sqrt{-s(4(u^3 + b) + 3su^2)}$; return $\bot$ if not square.
* If $c \in \\{3, 7\\}$ and $r=0$, return $\bot.$
* If $s = 0$, return $\bot.$
* Let $v = (r/s - u)/2.$
* Let $w = \sqrt{s}$; return $\bot$ if not square.
* Depending on $c:$
* If $c \in \\{0, 2\\}:$ return $w(\frac{\sqrt{-3}-1}{2}u - v).$
* If $c \in \\{1, 3\\}:$ return $w(\frac{\sqrt{-3}+1}{2}u + v).$
* If $c \in \\{4, 6\\}:$ return $w(\frac{-\sqrt{-3}+1}{2}u + v).$
* If $c \in \\{5, 7\\}:$ return $w(\frac{-\sqrt{-3}-1}{2}u - v).$
- If $u=0$, return $\bot.$
- If $c \in \\{0, 1, 4, 5\\}:$
- If $(-u-x)^3 + b$ is square, return $\bot$
- Let $s = -(u^3 + b)/(u^2 + ux + x^2)$ (cannot cause division by 0).
- Let $v = x.$
- Otherwise, when $c \in \\{2, 3, 6, 7\\}:$
- Let $s = x-u.$
- Let $r = \sqrt{-s(4(u^3 + b) + 3su^2)}$; return $\bot$ if not square.
- If $c \in \\{3, 7\\}$ and $r=0$, return $\bot.$
- If $s = 0$, return $\bot.$
- Let $v = (r/s - u)/2.$
- Let $w = \sqrt{s}$; return $\bot$ if not square.
- Depending on $c:$
- If $c \in \\{0, 2\\}:$ return $w(\frac{\sqrt{-3}-1}{2}u - v).$
- If $c \in \\{1, 3\\}:$ return $w(\frac{\sqrt{-3}+1}{2}u + v).$
- If $c \in \\{4, 6\\}:$ return $w(\frac{-\sqrt{-3}+1}{2}u + v).$
- If $c \in \\{5, 7\\}:$ return $w(\frac{-\sqrt{-3}-1}{2}u - v).$
This is implemented in `secp256k1_ellswift_xswiftec_inv_var`.
And the x-only ElligatorSwift encoding algorithm is still:
**Define** *ElligatorSwift(x)* as:
* Loop:
* Pick a uniformly random field element $u.$
* Pick a uniformly random integer $c$ in $[0,8).$
* Let $t = G_{c,u}(x).$
* If $t \neq \bot$, return $(u, t)$; restart loop otherwise.
**Define** _ElligatorSwift(x)_ as:
- Loop:
- Pick a uniformly random field element $u.$
- Pick a uniformly random integer $c$ in $[0,8).$
- Let $t = G_{c,u}(x).$
- If $t \neq \bot$, return $(u, t)$; restart loop otherwise.
Note that this logic does not take the remapped $u=0$, $t=0$, and $g(u) = -t^2$ cases into account; it just avoids them.
While it is not impossible to make the encoder target them, this would increase the maximum number of $t$ values for a given $(u, x)$
combination beyond 8, and thereby slow down the ElligatorSwift loop proportionally, for a negligible gain in uniformity.
## 4. Encoding and decoding full *(x, y)* coordinates
## 4. Encoding and decoding full _(x, y)_ coordinates
So far we have only addressed encoding and decoding x-coordinates, but in some cases an encoding
for full points with $(x, y)$ coordinates is desirable. It is possible to encode this information
@@ -422,30 +441,32 @@ four distinct $P_u^{'-1}$ calls in the definition of $G_{u,c}.$
To encode the sign of $y$ in the sign of $Y:$
**Define** *Decode(u, t)* for full $(x, y)$ as:
* Let $(X, Y) = P_u(t).$
* Let $x$ be the first value in $(u + 4Y^2, \frac{-X}{2Y} - \frac{u}{2}, \frac{X}{2Y} - \frac{u}{2})$ for which $g(x)$ is square.
* Let $y = \sqrt{g(x)}.$
* If $sign(y) = sign(Y)$, return $(x, y)$; otherwise return $(x, -y).$
**Define** _Decode(u, t)_ for full $(x, y)$ as:
- Let $(X, Y) = P_u(t).$
- Let $x$ be the first value in $(u + 4Y^2, \frac{-X}{2Y} - \frac{u}{2}, \frac{X}{2Y} - \frac{u}{2})$ for which $g(x)$ is square.
- Let $y = \sqrt{g(x)}.$
- If $sign(y) = sign(Y)$, return $(x, y)$; otherwise return $(x, -y).$
And encoding would be done using a $G_{c,u}(x, y)$ function defined as:
**Define** $G_{c,u}(x, y)$ as:
* If $c \in \\{0, 1\\}:$
* If $g(u) = 0$ or $g(x) = 0$, return $\bot$ (even curves only).
* If $g(-u-x)$ is square, return $\bot.$
* Let $s = -g(u)/(u^2 + ux + x^2 + a)$ (cannot cause division by zero).
* Let $v = x.$
* Otherwise, when $c \in \\{2, 3\\}:$
* Let $s = x-u.$
* Let $r = \sqrt{-s(4g(u) + sh(u))}$; return $\bot$ if not square.
* If $c = 3$ and $r = 0$, return $\bot.$
* Let $v = (r/s - u)/2.$
* Let $w = \sqrt{s}$; return $\bot$ if not square.
* Let $w' = w$ if $sign(w/2) = sign(y)$; $-w$ otherwise.
* Depending on $c:$
* If $c \in \\{0, 2\\}:$ return $P_u^{'-1}(v, w').$
* If $c \in \\{1, 3\\}:$ return $P_u^{'-1}(-u-v, w').$
- If $c \in \\{0, 1\\}:$
- If $g(u) = 0$ or $g(x) = 0$, return $\bot$ (even curves only).
- If $g(-u-x)$ is square, return $\bot.$
- Let $s = -g(u)/(u^2 + ux + x^2 + a)$ (cannot cause division by zero).
- Let $v = x.$
- Otherwise, when $c \in \\{2, 3\\}:$
- Let $s = x-u.$
- Let $r = \sqrt{-s(4g(u) + sh(u))}$; return $\bot$ if not square.
- If $c = 3$ and $r = 0$, return $\bot.$
- Let $v = (r/s - u)/2.$
- Let $w = \sqrt{s}$; return $\bot$ if not square.
- Let $w' = w$ if $sign(w/2) = sign(y)$; $-w$ otherwise.
- Depending on $c:$
- If $c \in \\{0, 2\\}:$ return $P_u^{'-1}(v, w').$
- If $c \in \\{1, 3\\}:$ return $P_u^{'-1}(-u-v, w').$
Note that $c$ now only ranges $[0,4)$, as the sign of $w'$ is decided based on that of $y$, rather than on $c.$
This change makes some valid encodings unreachable: when $y = 0$ and $sign(Y) \neq sign(0)$.
@@ -454,22 +475,23 @@ In the above logic, $sign$ can be implemented in several ways, such as parity of
of the input field element (for prime-sized fields) or the quadratic residuosity (for fields where
$-1$ is not square). The choice does not matter, as long as it only takes on two possible values, and for $x \neq 0$ it holds that $sign(x) \neq sign(-x)$.
### 4.1 Full *(x, y)* coordinates for `secp256k1`
### 4.1 Full _(x, y)_ coordinates for `secp256k1`
For $a=0$ curves, there is another option. Note that for those,
the $P_u(t)$ function translates negations of $t$ to negations of (both) $X$ and $Y.$ Thus, we can use $sign(t)$ to
encode the y-coordinate directly. Combined with the earlier remapping to guarantee all inputs land on the curve, we get
as decoder:
**Define** *Decode(u, t)* as:
* Let $u'=u$ if $u \neq 0$; $1$ otherwise.
* Let $t'=t$ if $t \neq 0$; $1$ otherwise.
* Let $t''=t'$ if $u'^3 + b + t'^2 \neq 0$; $2t'$ otherwise.
* Let $X = \dfrac{u'^3 + b - t''^2}{2t''}.$
* Let $Y = \dfrac{X + t''}{u'\sqrt{-3}}.$
* Let $x$ be the first element of $(u' + 4Y^2, \frac{-X}{2Y} - \frac{u'}{2}, \frac{X}{2Y} - \frac{u'}{2})$ for which $g(x)$ is square.
* Let $y = \sqrt{g(x)}.$
* Return $(x, y)$ if $sign(y) = sign(t)$; $(x, -y)$ otherwise.
**Define** _Decode(u, t)_ as:
- Let $u'=u$ if $u \neq 0$; $1$ otherwise.
- Let $t'=t$ if $t \neq 0$; $1$ otherwise.
- Let $t''=t'$ if $u'^3 + b + t'^2 \neq 0$; $2t'$ otherwise.
- Let $X = \dfrac{u'^3 + b - t''^2}{2t''}.$
- Let $Y = \dfrac{X + t''}{u'\sqrt{-3}}.$
- Let $x$ be the first element of $(u' + 4Y^2, \frac{-X}{2Y} - \frac{u'}{2}, \frac{X}{2Y} - \frac{u'}{2})$ for which $g(x)$ is square.
- Let $y = \sqrt{g(x)}.$
- Return $(x, y)$ if $sign(y) = sign(t)$; $(x, -y)$ otherwise.
This is implemented in `secp256k1_ellswift_swiftec_var`. The used $sign(x)$ function is the parity of $x$ when represented as in integer in $[0,q).$

3
external/secp256k1/doc/musig.md vendored
View File

@@ -1,5 +1,4 @@
Notes on the musig module API
===========================
# Notes on the musig module API
The following sections contain additional notes on the API of the musig module (`include/secp256k1_musig.h`).
A usage example can be found in `examples/musig.c`.

40
external/secp256k1/doc/release-process.md vendored
View File

@@ -2,7 +2,7 @@
This document outlines the process for releasing versions of the form `$MAJOR.$MINOR.$PATCH`.
We distinguish between two types of releases: *regular* and *maintenance* releases.
We distinguish between two types of releases: _regular_ and _maintenance_ releases.
Regular releases are releases of a new major or minor version as well as patches of the most recent release.
Maintenance releases, on the other hand, are required for patches of older releases.
@@ -15,6 +15,7 @@ This process also assumes that there will be no minor releases for old major rel
We aim to cut a regular release every 3-4 months, approximately twice as frequent as major Bitcoin Core releases. Every second release should be published one month before the feature freeze of the next major Bitcoin Core release, allowing sufficient time to update the library in Core.
## Sanity checks
Perform these checks when reviewing the release PR (see below):
1. Ensure `make distcheck` doesn't fail.
@@ -42,15 +43,15 @@ Perform these checks when reviewing the release PR (see below):
## Regular release
1. Open a PR to the master branch with a commit (using message `"release: prepare for $MAJOR.$MINOR.$PATCH"`, for example) that
* finalizes the release notes in [CHANGELOG.md](../CHANGELOG.md) by
* adding a section for the release (make sure that the version number is a link to a diff between the previous and new version),
* removing the `[Unreleased]` section header,
* ensuring that the release notes are not missing entries (check the `needs-changelog` label on github), and
* including an entry for `### ABI Compatibility` if it doesn't exist,
* sets `_PKG_VERSION_IS_RELEASE` to `true` in `configure.ac`, and,
* if this is not a patch release,
* updates `_PKG_VERSION_*` and `_LIB_VERSION_*` in `configure.ac`, and
* updates `project(libsecp256k1 VERSION ...)` and `${PROJECT_NAME}_LIB_VERSION_*` in `CMakeLists.txt`.
- finalizes the release notes in [CHANGELOG.md](../CHANGELOG.md) by
- adding a section for the release (make sure that the version number is a link to a diff between the previous and new version),
- removing the `[Unreleased]` section header,
- ensuring that the release notes are not missing entries (check the `needs-changelog` label on github), and
- including an entry for `### ABI Compatibility` if it doesn't exist,
- sets `_PKG_VERSION_IS_RELEASE` to `true` in `configure.ac`, and,
- if this is not a patch release,
- updates `_PKG_VERSION_*` and `_LIB_VERSION_*` in `configure.ac`, and
- updates `project(libsecp256k1 VERSION ...)` and `${PROJECT_NAME}_LIB_VERSION_*` in `CMakeLists.txt`.
2. Perform the [sanity checks](#sanity-checks) on the PR branch.
3. After the PR is merged, tag the commit, and push the tag:
```
@@ -59,11 +60,12 @@ Perform these checks when reviewing the release PR (see below):
git push git@github.com:bitcoin-core/secp256k1.git v$MAJOR.$MINOR.$PATCH
```
4. Open a PR to the master branch with a commit (using message `"release cleanup: bump version after $MAJOR.$MINOR.$PATCH"`, for example) that
* sets `_PKG_VERSION_IS_RELEASE` to `false` and increments `_PKG_VERSION_PATCH` and `_LIB_VERSION_REVISION` in `configure.ac`,
* increments the `$PATCH` component of `project(libsecp256k1 VERSION ...)` and `${PROJECT_NAME}_LIB_VERSION_REVISION` in `CMakeLists.txt`, and
* adds an `[Unreleased]` section header to the [CHANGELOG.md](../CHANGELOG.md).
- sets `_PKG_VERSION_IS_RELEASE` to `false` and increments `_PKG_VERSION_PATCH` and `_LIB_VERSION_REVISION` in `configure.ac`,
- increments the `$PATCH` component of `project(libsecp256k1 VERSION ...)` and `${PROJECT_NAME}_LIB_VERSION_REVISION` in `CMakeLists.txt`, and
- adds an `[Unreleased]` section header to the [CHANGELOG.md](../CHANGELOG.md).
If other maintainers are not present to approve the PR, it can be merged without ACKs.
5. Create a new GitHub release with a link to the corresponding entry in [CHANGELOG.md](../CHANGELOG.md).
6. Send an announcement email to the bitcoin-dev mailing list.
@@ -77,9 +79,9 @@ Note that bug fixes need to be backported only to releases for which no compatib
git push git@github.com:bitcoin-core/secp256k1.git $MAJOR.$MINOR
```
2. Open a pull request to the `$MAJOR.$MINOR` branch that
* includes the bug fixes,
* finalizes the release notes similar to a regular release,
* increments `_PKG_VERSION_PATCH` and `_LIB_VERSION_REVISION` in `configure.ac`
- includes the bug fixes,
- finalizes the release notes similar to a regular release,
- increments `_PKG_VERSION_PATCH` and `_LIB_VERSION_REVISION` in `configure.ac`
and the `$PATCH` component of `project(libsecp256k1 VERSION ...)` and `${PROJECT_NAME}_LIB_VERSION_REVISION` in `CMakeLists.txt`
(with commit message `"release: bump versions for $MAJOR.$MINOR.$PATCH"`, for example).
3. Perform the [sanity checks](#sanity-checks) on the PR branch.
@@ -89,6 +91,6 @@ Note that bug fixes need to be backported only to releases for which no compatib
git tag -s v$MAJOR.$MINOR.$PATCH -m "libsecp256k1 $MAJOR.$MINOR.$PATCH"
git push git@github.com:bitcoin-core/secp256k1.git v$MAJOR.$MINOR.$PATCH
```
6. Create a new GitHub release with a link to the corresponding entry in [CHANGELOG.md](../CHANGELOG.md).
7. Send an announcement email to the bitcoin-dev mailing list.
8. Open PR to the master branch that includes a commit (with commit message `"release notes: add $MAJOR.$MINOR.$PATCH"`, for example) that adds release notes to [CHANGELOG.md](../CHANGELOG.md).
5. Create a new GitHub release with a link to the corresponding entry in [CHANGELOG.md](../CHANGELOG.md).
6. Send an announcement email to the bitcoin-dev mailing list.
7. Open PR to the master branch that includes a commit (with commit message `"release notes: add $MAJOR.$MINOR.$PATCH"`, for example) that adds release notes to [CHANGELOG.md](../CHANGELOG.md).

301
external/secp256k1/doc/safegcd_implementation.md vendored
View File

@@ -29,65 +29,67 @@ def gcd(f, g):
return abs(f)
```
It computes the greatest common divisor of an odd integer *f* and any integer *g*. Its inner loop
keeps rewriting the variables *f* and *g* alongside a state variable *&delta;* that starts at *1*, until
*g=0* is reached. At that point, *|f|* gives the GCD. Each of the transitions in the loop is called a
It computes the greatest common divisor of an odd integer _f_ and any integer _g_. Its inner loop
keeps rewriting the variables _f_ and _g_ alongside a state variable _&delta;_ that starts at _1_, until
_g=0_ is reached. At that point, _|f|_ gives the GCD. Each of the transitions in the loop is called a
"division step" (referred to as divstep in what follows).
For example, *gcd(21, 14)* would be computed as:
- Start with *&delta;=1 f=21 g=14*
- Take the third branch: *&delta;=2 f=21 g=7*
- Take the first branch: *&delta;=-1 f=7 g=-7*
- Take the second branch: *&delta;=0 f=7 g=0*
- The answer *|f| = 7*.
For example, _gcd(21, 14)_ would be computed as:
- Start with _&delta;=1 f=21 g=14_
- Take the third branch: _&delta;=2 f=21 g=7_
- Take the first branch: _&delta;=-1 f=7 g=-7_
- Take the second branch: _&delta;=0 f=7 g=0_
- The answer _|f| = 7_.
Why it works:
- Divsteps can be decomposed into two steps (see paragraph 8.2 in the paper):
- (a) If *g* is odd, replace *(f,g)* with *(g,g-f)* or (f,g+f), resulting in an even *g*.
- (b) Replace *(f,g)* with *(f,g/2)* (where *g* is guaranteed to be even).
- (a) If _g_ is odd, replace _(f,g)_ with _(g,g-f)_ or (f,g+f), resulting in an even _g_.
- (b) Replace _(f,g)_ with _(f,g/2)_ (where _g_ is guaranteed to be even).
- Neither of those two operations change the GCD:
- For (a), assume *gcd(f,g)=c*, then it must be the case that *f=a&thinsp;c* and *g=b&thinsp;c* for some integers *a*
and *b*. As *(g,g-f)=(b&thinsp;c,(b-a)c)* and *(f,f+g)=(a&thinsp;c,(a+b)c)*, the result clearly still has
common factor *c*. Reasoning in the other direction shows that no common factor can be added by
- For (a), assume _gcd(f,g)=c_, then it must be the case that _f=a&thinsp;c_ and _g=b&thinsp;c_ for some integers _a_
and _b_. As _(g,g-f)=(b&thinsp;c,(b-a)c)_ and _(f,f+g)=(a&thinsp;c,(a+b)c)_, the result clearly still has
common factor _c_. Reasoning in the other direction shows that no common factor can be added by
doing so either.
- For (b), we know that *f* is odd, so *gcd(f,g)* clearly has no factor *2*, and we can remove
it from *g*.
- The algorithm will eventually converge to *g=0*. This is proven in the paper (see theorem G.3).
- It follows that eventually we find a final value *f'* for which *gcd(f,g) = gcd(f',0)*. As the
gcd of *f'* and *0* is *|f'|* by definition, that is our answer.
- For (b), we know that _f_ is odd, so _gcd(f,g)_ clearly has no factor _2_, and we can remove
it from _g_.
- The algorithm will eventually converge to _g=0_. This is proven in the paper (see theorem G.3).
- It follows that eventually we find a final value _f'_ for which _gcd(f,g) = gcd(f',0)_. As the
gcd of _f'_ and _0_ is _|f'|_ by definition, that is our answer.
Compared to more [traditional GCD algorithms](https://en.wikipedia.org/wiki/Euclidean_algorithm), this one has the property of only ever looking at
the low-order bits of the variables to decide the next steps, and being easy to make
constant-time (in more low-level languages than Python). The *&delta;* parameter is necessary to
constant-time (in more low-level languages than Python). The _&delta;_ parameter is necessary to
guide the algorithm towards shrinking the numbers' magnitudes without explicitly needing to look
at high order bits.
Properties that will become important later:
- Performing more divsteps than needed is not a problem, as *f* does not change anymore after *g=0*.
- Only even numbers are divided by *2*. This means that when reasoning about it algebraically we
do not need to worry about rounding.
- At every point during the algorithm's execution the next *N* steps only depend on the bottom *N*
bits of *f* and *g*, and on *&delta;*.
- Performing more divsteps than needed is not a problem, as _f_ does not change anymore after _g=0_.
- Only even numbers are divided by _2_. This means that when reasoning about it algebraically we
do not need to worry about rounding.
- At every point during the algorithm's execution the next _N_ steps only depend on the bottom _N_
bits of _f_ and _g_, and on _&delta;_.
## 2. From GCDs to modular inverses
We want an algorithm to compute the inverse *a* of *x* modulo *M*, i.e. the number a such that *a&thinsp;x=1
mod M*. This inverse only exists if the GCD of *x* and *M* is *1*, but that is always the case if *M* is
prime and *0 < x < M*. In what follows, assume that the modular inverse exists.
We want an algorithm to compute the inverse _a_ of _x_ modulo _M_, i.e. the number a such that _a&thinsp;x=1
mod M_. This inverse only exists if the GCD of _x_ and _M_ is _1_, but that is always the case if _M_ is
prime and _0 < x < M_. In what follows, assume that the modular inverse exists.
It turns out this inverse can be computed as a side effect of computing the GCD by keeping track
of how the internal variables can be written as linear combinations of the inputs at every step
(see the [extended Euclidean algorithm](https://en.wikipedia.org/wiki/Extended_Euclidean_algorithm)).
Since the GCD is *1*, such an algorithm will compute numbers *a* and *b* such that a&thinsp;x + b&thinsp;M = 1*.
Since the GCD is _1_, such an algorithm will compute numbers _a_ and _b_ such that a&thinsp;x + b&thinsp;M = 1*.
Taking that expression *mod M* gives *a&thinsp;x mod M = 1*, and we see that *a* is the modular inverse of *x
mod M*.
mod M\*.
A similar approach can be used to calculate modular inverses using the divsteps-based GCD
algorithm shown above, if the modulus *M* is odd. To do so, compute *gcd(f=M,g=x)*, while keeping
track of extra variables *d* and *e*, for which at every step *d = f/x (mod M)* and *e = g/x (mod M)*.
*f/x* here means the number which multiplied with *x* gives *f mod M*. As *f* and *g* are initialized to *M*
and *x* respectively, *d* and *e* just start off being *0* (*M/x mod M = 0/x mod M = 0*) and *1* (*x/x mod M
= 1*).
algorithm shown above, if the modulus _M_ is odd. To do so, compute _gcd(f=M,g=x)_, while keeping
track of extra variables _d_ and _e_, for which at every step _d = f/x (mod M)_ and _e = g/x (mod M)_.
_f/x_ here means the number which multiplied with _x_ gives _f mod M_. As _f_ and _g_ are initialized to _M_
and _x_ respectively, _d_ and _e_ just start off being _0_ (_M/x mod M = 0/x mod M = 0_) and _1_ (_x/x mod M
= 1_).
```python
def div2(M, x):
@@ -119,17 +121,16 @@ def modinv(M, x):
return (d * f) % M
```
Also note that this approach to track *d* and *e* throughout the computation to determine the inverse
Also note that this approach to track _d_ and _e_ throughout the computation to determine the inverse
is different from the paper. There (see paragraph 12.1 in the paper) a transition matrix for the
entire computation is determined (see section 3 below) and the inverse is computed from that.
The approach here avoids the need for 2x2 matrix multiplications of various sizes, and appears to
be faster at the level of optimization we're able to do in C.
## 3. Batching multiple divsteps
Every divstep can be expressed as a matrix multiplication, applying a transition matrix *(1/2 t)*
to both vectors *[f, g]* and *[d, e]* (see paragraph 8.1 in the paper):
Every divstep can be expressed as a matrix multiplication, applying a transition matrix _(1/2 t)_
to both vectors _[f, g]_ and _[d, e]_ (see paragraph 8.1 in the paper):
```
t = [ u, v ]
@@ -142,15 +143,15 @@ to both vectors *[f, g]* and *[d, e]* (see paragraph 8.1 in the paper):
[ out_e ] [ in_e ]
```
where *(u, v, q, r)* is *(0, 2, -1, 1)*, *(2, 0, 1, 1)*, or *(2, 0, 0, 1)*, depending on which branch is
taken. As above, the resulting *f* and *g* are always integers.
where _(u, v, q, r)_ is _(0, 2, -1, 1)_, _(2, 0, 1, 1)_, or _(2, 0, 0, 1)_, depending on which branch is
taken. As above, the resulting _f_ and _g_ are always integers.
Performing multiple divsteps corresponds to a multiplication with the product of all the
individual divsteps' transition matrices. As each transition matrix consists of integers
divided by *2*, the product of these matrices will consist of integers divided by *2<sup>N</sup>* (see also
theorem 9.2 in the paper). These divisions are expensive when updating *d* and *e*, so we delay
them: we compute the integer coefficients of the combined transition matrix scaled by *2<sup>N</sup>*, and
do one division by *2<sup>N</sup>* as a final step:
divided by _2_, the product of these matrices will consist of integers divided by _2<sup>N</sup>_ (see also
theorem 9.2 in the paper). These divisions are expensive when updating _d_ and _e_, so we delay
them: we compute the integer coefficients of the combined transition matrix scaled by _2<sup>N</sup>_, and
do one division by _2<sup>N</sup>_ as a final step:
```python
def divsteps_n_matrix(delta, f, g):
@@ -166,13 +167,13 @@ def divsteps_n_matrix(delta, f, g):
return delta, (u, v, q, r)
```
As the branches in the divsteps are completely determined by the bottom *N* bits of *f* and *g*, this
As the branches in the divsteps are completely determined by the bottom _N_ bits of _f_ and _g_, this
function to compute the transition matrix only needs to see those bottom bits. Furthermore all
intermediate results and outputs fit in *(N+1)*-bit numbers (unsigned for *f* and *g*; signed for *u*, *v*,
*q*, and *r*) (see also paragraph 8.3 in the paper). This means that an implementation using 64-bit
integers could set *N=62* and compute the full transition matrix for 62 steps at once without any
intermediate results and outputs fit in _(N+1)_-bit numbers (unsigned for _f_ and _g_; signed for _u_, _v_,
_q_, and _r_) (see also paragraph 8.3 in the paper). This means that an implementation using 64-bit
integers could set _N=62_ and compute the full transition matrix for 62 steps at once without any
big integer arithmetic at all. This is the reason why this algorithm is efficient: it only needs
to update the full-size *f*, *g*, *d*, and *e* numbers once every *N* steps.
to update the full-size _f_, _g_, _d_, and _e_ numbers once every _N_ steps.
We still need functions to compute:
@@ -184,8 +185,8 @@ We still need functions to compute:
[ out_e ] ( [ q, r ]) [ in_e ]
```
Because the divsteps transformation only ever divides even numbers by two, the result of *t&thinsp;[f,g]* is always even. When *t* is a composition of *N* divsteps, it follows that the resulting *f*
and *g* will be multiple of *2<sup>N</sup>*, and division by *2<sup>N</sup>* is simply shifting them down:
Because the divsteps transformation only ever divides even numbers by two, the result of _t&thinsp;[f,g]_ is always even. When _t_ is a composition of _N_ divsteps, it follows that the resulting _f_
and _g_ will be multiple of _2<sup>N</sup>_, and division by _2<sup>N</sup>_ is simply shifting them down:
```python
def update_fg(f, g, t):
@@ -199,8 +200,8 @@ def update_fg(f, g, t):
return cf >> N, cg >> N
```
The same is not true for *d* and *e*, and we need an equivalent of the `div2` function for division by *2<sup>N</sup> mod M*.
This is easy if we have precomputed *1/M mod 2<sup>N</sup>* (which always exists for odd *M*):
The same is not true for _d_ and _e_, and we need an equivalent of the `div2` function for division by _2<sup>N</sup> mod M_.
This is easy if we have precomputed _1/M mod 2<sup>N</sup>_ (which always exists for odd _M_):
```python
def div2n(M, Mi, x):
@@ -224,7 +225,7 @@ def update_de(d, e, t, M, Mi):
return div2n(M, Mi, cd), div2n(M, Mi, ce)
```
With all of those, we can write a version of `modinv` that performs *N* divsteps at once:
With all of those, we can write a version of `modinv` that performs _N_ divsteps at once:
```python3
def modinv(M, Mi, x):
@@ -242,20 +243,19 @@ def modinv(M, Mi, x):
return (d * f) % M
```
This means that in practice we'll always perform a multiple of *N* divsteps. This is not a problem
because once *g=0*, further divsteps do not affect *f*, *g*, *d*, or *e* anymore (only *&delta;* keeps
This means that in practice we'll always perform a multiple of _N_ divsteps. This is not a problem
because once _g=0_, further divsteps do not affect _f_, _g_, _d_, or _e_ anymore (only _&delta;_ keeps
increasing). For variable time code such excess iterations will be mostly optimized away in later
sections.
## 4. Avoiding modulus operations
So far, there are two places where we compute a remainder of big numbers modulo *M*: at the end of
`div2n` in every `update_de`, and at the very end of `modinv` after potentially negating *d* due to the
sign of *f*. These are relatively expensive operations when done generically.
So far, there are two places where we compute a remainder of big numbers modulo _M_: at the end of
`div2n` in every `update_de`, and at the very end of `modinv` after potentially negating _d_ due to the
sign of _f_. These are relatively expensive operations when done generically.
To deal with the modulus operation in `div2n`, we simply stop requiring *d* and *e* to be in range
*[0,M)* all the time. Let's start by inlining `div2n` into `update_de`, and dropping the modulus
To deal with the modulus operation in `div2n`, we simply stop requiring _d_ and _e_ to be in range
_[0,M)_ all the time. Let's start by inlining `div2n` into `update_de`, and dropping the modulus
operation at the end:
```python
@@ -272,15 +272,15 @@ def update_de(d, e, t, M, Mi):
return cd >> N, ce >> N
```
Let's look at bounds on the ranges of these numbers. It can be shown that *|u|+|v|* and *|q|+|r|*
never exceed *2<sup>N</sup>* (see paragraph 8.3 in the paper), and thus a multiplication with *t* will have
outputs whose absolute values are at most *2<sup>N</sup>* times the maximum absolute input value. In case the
inputs *d* and *e* are in *(-M,M)*, which is certainly true for the initial values *d=0* and *e=1* assuming
*M > 1*, the multiplication results in numbers in range *(-2<sup>N</sup>M,2<sup>N</sup>M)*. Subtracting less than *2<sup>N</sup>*
times *M* to cancel out *N* bits brings that up to *(-2<sup>N+1</sup>M,2<sup>N</sup>M)*, and
dividing by *2<sup>N</sup>* at the end takes it to *(-2M,M)*. Another application of `update_de` would take that
to *(-3M,2M)*, and so forth. This progressive expansion of the variables' ranges can be
counteracted by incrementing *d* and *e* by *M* whenever they're negative:
Let's look at bounds on the ranges of these numbers. It can be shown that _|u|+|v|_ and _|q|+|r|_
never exceed _2<sup>N</sup>_ (see paragraph 8.3 in the paper), and thus a multiplication with _t_ will have
outputs whose absolute values are at most _2<sup>N</sup>_ times the maximum absolute input value. In case the
inputs _d_ and _e_ are in _(-M,M)_, which is certainly true for the initial values _d=0_ and _e=1_ assuming
_M > 1_, the multiplication results in numbers in range _(-2<sup>N</sup>M,2<sup>N</sup>M)_. Subtracting less than _2<sup>N</sup>_
times _M_ to cancel out _N_ bits brings that up to _(-2<sup>N+1</sup>M,2<sup>N</sup>M)_, and
dividing by _2<sup>N</sup>_ at the end takes it to _(-2M,M)_. Another application of `update_de` would take that
to _(-3M,2M)_, and so forth. This progressive expansion of the variables' ranges can be
counteracted by incrementing _d_ and _e_ by _M_ whenever they're negative:
```python
...
@@ -293,12 +293,12 @@ counteracted by incrementing *d* and *e* by *M* whenever they're negative:
...
```
With inputs in *(-2M,M)*, they will first be shifted into range *(-M,M)*, which means that the
output will again be in *(-2M,M)*, and this remains the case regardless of how many `update_de`
With inputs in _(-2M,M)_, they will first be shifted into range _(-M,M)_, which means that the
output will again be in _(-2M,M)_, and this remains the case regardless of how many `update_de`
invocations there are. In what follows, we will try to make this more efficient.
Note that increasing *d* by *M* is equal to incrementing *cd* by *u&thinsp;M* and *ce* by *q&thinsp;M*. Similarly,
increasing *e* by *M* is equal to incrementing *cd* by *v&thinsp;M* and *ce* by *r&thinsp;M*. So we could instead write:
Note that increasing _d_ by _M_ is equal to incrementing _cd_ by _u&thinsp;M_ and _ce_ by _q&thinsp;M_. Similarly,
increasing _e_ by _M_ is equal to incrementing _cd_ by _v&thinsp;M_ and _ce_ by _r&thinsp;M_. So we could instead write:
```python
...
@@ -318,10 +318,10 @@ increasing *e* by *M* is equal to incrementing *cd* by *v&thinsp;M* and *ce* by
...
```
Now note that we have two steps of corrections to *cd* and *ce* that add multiples of *M*: this
Now note that we have two steps of corrections to _cd_ and _ce_ that add multiples of _M_: this
increment, and the decrement that cancels out bottom bits. The second one depends on the first
one, but they can still be efficiently combined by only computing the bottom bits of *cd* and *ce*
at first, and using that to compute the final *md*, *me* values:
one, but they can still be efficiently combined by only computing the bottom bits of _cd_ and _ce_
at first, and using that to compute the final _md_, _me_ values:
```python
def update_de(d, e, t, M, Mi):
@@ -346,8 +346,8 @@ def update_de(d, e, t, M, Mi):
return cd >> N, ce >> N
```
One last optimization: we can avoid the *md&thinsp;M* and *me&thinsp;M* multiplications in the bottom bits of *cd*
and *ce* by moving them to the *md* and *me* correction:
One last optimization: we can avoid the _md&thinsp;M_ and _me&thinsp;M_ multiplications in the bottom bits of _cd_
and _ce_ by moving them to the _md_ and _me_ correction:
```python
...
@@ -362,10 +362,10 @@ and *ce* by moving them to the *md* and *me* correction:
...
```
The resulting function takes *d* and *e* in range *(-2M,M)* as inputs, and outputs values in the same
range. That also means that the *d* value at the end of `modinv` will be in that range, while we want
a result in *[0,M)*. To do that, we need a normalization function. It's easy to integrate the
conditional negation of *d* (based on the sign of *f*) into it as well:
The resulting function takes _d_ and _e_ in range _(-2M,M)_ as inputs, and outputs values in the same
range. That also means that the _d_ value at the end of `modinv` will be in that range, while we want
a result in _[0,M)_. To do that, we need a normalization function. It's easy to integrate the
conditional negation of _d_ (based on the sign of _f_) into it as well:
```python
def normalize(sign, v, M):
@@ -391,22 +391,21 @@ And calling it in `modinv` is simply:
return normalize(f, d, M)
```
## 5. Constant-time operation
The primary selling point of the algorithm is fast constant-time operation. What code flow still
depends on the input data so far?
- the number of iterations of the while *g &ne; 0* loop in `modinv`
- the number of iterations of the while _g &ne; 0_ loop in `modinv`
- the branches inside `divsteps_n_matrix`
- the sign checks in `update_de`
- the sign checks in `normalize`
To make the while loop in `modinv` constant time it can be replaced with a constant number of
iterations. The paper proves (Theorem 11.2) that *741* divsteps are sufficient for any *256*-bit
inputs, and [safegcd-bounds](https://github.com/sipa/safegcd-bounds) shows that the slightly better bound *724* is
sufficient even. Given that every loop iteration performs *N* divsteps, it will run a total of
*&lceil;724/N&rceil;* times.
iterations. The paper proves (Theorem 11.2) that _741_ divsteps are sufficient for any _256_-bit
inputs, and [safegcd-bounds](https://github.com/sipa/safegcd-bounds) shows that the slightly better bound _724_ is
sufficient even. Given that every loop iteration performs _N_ divsteps, it will run a total of
_&lceil;724/N&rceil;_ times.
To deal with the branches in `divsteps_n_matrix` we will replace them with constant-time bitwise
operations (and hope the C compiler isn't smart enough to turn them back into branches; see
@@ -425,10 +424,10 @@ divstep can be written instead as (compare to the inner loop of `gcd` in section
```
To convert the above to bitwise operations, we rely on a trick to negate conditionally: per the
definition of negative numbers in two's complement, (*-v == ~v + 1*) holds for every number *v*. As
*-1* in two's complement is all *1* bits, bitflipping can be expressed as xor with *-1*. It follows
that *-v == (v ^ -1) - (-1)*. Thus, if we have a variable *c* that takes on values *0* or *-1*, then
*(v ^ c) - c* is *v* if *c=0* and *-v* if *c=-1*.
definition of negative numbers in two's complement, (_-v == ~v + 1_) holds for every number _v_. As
_-1_ in two's complement is all _1_ bits, bitflipping can be expressed as xor with _-1_. It follows
that _-v == (v ^ -1) - (-1)_. Thus, if we have a variable _c_ that takes on values _0_ or _-1_, then
_(v ^ c) - c_ is _v_ if _c=0_ and _-v_ if _c=-1_.
Using this we can write:
@@ -444,13 +443,13 @@ in constant-time form as:
x = (f ^ c1) - c1
```
To use that trick, we need a helper mask variable *c1* that resolves the condition *&delta;>0* to *-1*
(if true) or *0* (if false). We compute *c1* using right shifting, which is equivalent to dividing by
the specified power of *2* and rounding down (in Python, and also in C under the assumption of a typical two's complement system; see
`assumptions.h` for tests that this is the case). Right shifting by *63* thus maps all
numbers in range *[-2<sup>63</sup>,0)* to *-1*, and numbers in range *[0,2<sup>63</sup>)* to *0*.
To use that trick, we need a helper mask variable _c1_ that resolves the condition _&delta;>0_ to _-1_
(if true) or _0_ (if false). We compute _c1_ using right shifting, which is equivalent to dividing by
the specified power of _2_ and rounding down (in Python, and also in C under the assumption of a typical two's complement system; see
`assumptions.h` for tests that this is the case). Right shifting by _63_ thus maps all
numbers in range _[-2<sup>63</sup>,0)_ to _-1_, and numbers in range _[0,2<sup>63</sup>)_ to _0_.
Using the facts that *x&0=0* and *x&(-1)=x* (on two's complement systems again), we can write:
Using the facts that _x&0=0_ and _x&(-1)=x_ (on two's complement systems again), we can write:
```python
if g & 1:
@@ -498,8 +497,8 @@ becomes:
```
It turns out that this can be implemented more efficiently by applying the substitution
*&eta;=-&delta;*. In this representation, negating *&delta;* corresponds to negating *&eta;*, and incrementing
*&delta;* corresponds to decrementing *&eta;*. This allows us to remove the negation in the *c1*
_&eta;=-&delta;_. In this representation, negating _&delta;_ corresponds to negating _&eta;_, and incrementing
_&delta;_ corresponds to decrementing _&eta;_. This allows us to remove the negation in the _c1_
computation:
```python
@@ -519,12 +518,12 @@ computation:
g >>= 1
```
A variant of divsteps with better worst-case performance can be used instead: starting *&delta;* at
*1/2* instead of *1*. This reduces the worst case number of iterations to *590* for *256*-bit inputs
(which can be shown using convex hull analysis). In this case, the substitution *&zeta;=-(&delta;+1/2)*
is used instead to keep the variable integral. Incrementing *&delta;* by *1* still translates to
decrementing *&zeta;* by *1*, but negating *&delta;* now corresponds to going from *&zeta;* to *-(&zeta;+1)*, or
*~&zeta;*. Doing that conditionally based on *c3* is simply:
A variant of divsteps with better worst-case performance can be used instead: starting _&delta;_ at
_1/2_ instead of _1_. This reduces the worst case number of iterations to _590_ for _256_-bit inputs
(which can be shown using convex hull analysis). In this case, the substitution _&zeta;=-(&delta;+1/2)_
is used instead to keep the variable integral. Incrementing _&delta;_ by _1_ still translates to
decrementing _&zeta;_ by _1_, but negating _&delta;_ now corresponds to going from _&zeta;_ to _-(&zeta;+1)_, or
_~&zeta;_. Doing that conditionally based on _c3_ is simply:
```python
...
@@ -534,13 +533,12 @@ decrementing *&zeta;* by *1*, but negating *&delta;* now corresponds to going fr
```
By replacing the loop in `divsteps_n_matrix` with a variant of the divstep code above (extended to
also apply all *f* operations to *u*, *v* and all *g* operations to *q*, *r*), a constant-time version of
also apply all _f_ operations to _u_, _v_ and all _g_ operations to _q_, _r_), a constant-time version of
`divsteps_n_matrix` is obtained. The full code will be in section 7.
These bit fiddling tricks can also be used to make the conditional negations and additions in
`update_de` and `normalize` constant-time.
## 6. Variable-time optimizations
In section 5, we modified the `divsteps_n_matrix` function (and a few others) to be constant time.
@@ -550,7 +548,7 @@ faster non-constant time `divsteps_n_matrix` function.
To do so, first consider yet another way of writing the inner loop of divstep operations in
`gcd` from section 1. This decomposition is also explained in the paper in section 8.2. We use
the original version with initial *&delta;=1* and *&eta;=-&delta;* here.
the original version with initial _&delta;=1_ and _&eta;=-&delta;_ here.
```python
for _ in range(N):
@@ -562,7 +560,7 @@ for _ in range(N):
g >>= 1
```
Whenever *g* is even, the loop only shifts *g* down and decreases *&eta;*. When *g* ends in multiple zero
Whenever _g_ is even, the loop only shifts _g_ down and decreases _&eta;_. When _g_ ends in multiple zero
bits, these iterations can be consolidated into one step. This requires counting the bottom zero
bits efficiently, which is possible on most platforms; it is abstracted here as the function
`count_trailing_zeros`.
@@ -595,20 +593,20 @@ while True:
# g is even now, and the eta decrement and g shift will happen in the next loop.
```
We can now remove multiple bottom *0* bits from *g* at once, but still need a full iteration whenever
there is a bottom *1* bit. In what follows, we will get rid of multiple *1* bits simultaneously as
We can now remove multiple bottom _0_ bits from _g_ at once, but still need a full iteration whenever
there is a bottom _1_ bit. In what follows, we will get rid of multiple _1_ bits simultaneously as
well.
Observe that as long as *&eta; &geq; 0*, the loop does not modify *f*. Instead, it cancels out bottom
bits of *g* and shifts them out, and decreases *&eta;* and *i* accordingly - interrupting only when *&eta;*
becomes negative, or when *i* reaches *0*. Combined, this is equivalent to adding a multiple of *f* to
*g* to cancel out multiple bottom bits, and then shifting them out.
Observe that as long as _&eta; &geq; 0_, the loop does not modify _f_. Instead, it cancels out bottom
bits of _g_ and shifts them out, and decreases _&eta;_ and _i_ accordingly - interrupting only when _&eta;_
becomes negative, or when _i_ reaches _0_. Combined, this is equivalent to adding a multiple of _f_ to
_g_ to cancel out multiple bottom bits, and then shifting them out.
It is easy to find what that multiple is: we want a number *w* such that *g+w&thinsp;f* has a few bottom
zero bits. If that number of bits is *L*, we want *g+w&thinsp;f mod 2<sup>L</sup> = 0*, or *w = -g/f mod 2<sup>L</sup>*. Since *f*
is odd, such a *w* exists for any *L*. *L* cannot be more than *i* steps (as we'd finish the loop before
doing more) or more than *&eta;+1* steps (as we'd run `eta, f, g = -eta, g, -f` at that point), but
apart from that, we're only limited by the complexity of computing *w*.
It is easy to find what that multiple is: we want a number _w_ such that _g+w&thinsp;f_ has a few bottom
zero bits. If that number of bits is _L_, we want _g+w&thinsp;f mod 2<sup>L</sup> = 0_, or _w = -g/f mod 2<sup>L</sup>_. Since _f_
is odd, such a _w_ exists for any _L_. _L_ cannot be more than _i_ steps (as we'd finish the loop before
doing more) or more than _&eta;+1_ steps (as we'd run `eta, f, g = -eta, g, -f` at that point), but
apart from that, we're only limited by the complexity of computing _w_.
This code demonstrates how to cancel up to 4 bits per step:
@@ -642,26 +640,25 @@ some can be found in Hacker's Delight second edition by Henry S. Warren, Jr. pag
Here we need the negated modular inverse, which is a simple transformation of those:
- Instead of a 3-bit table:
- *-f* or *f ^ 6*
- _-f_ or _f ^ 6_
- Instead of a 4-bit table:
- *1 - f(f + 1)*
- *-(f + (((f + 1) & 4) << 1))*
- For larger tables the following technique can be used: if *w=-1/f mod 2<sup>L</sup>*, then *w(w&thinsp;f+2)* is
*-1/f mod 2<sup>2L</sup>*. This allows extending the previous formulas (or tables). In particular we
- _1 - f(f + 1)_
- _-(f + (((f + 1) & 4) << 1))_
- For larger tables the following technique can be used: if _w=-1/f mod 2<sup>L</sup>_, then _w(w&thinsp;f+2)_ is
_-1/f mod 2<sup>2L</sup>_. This allows extending the previous formulas (or tables). In particular we
have this 6-bit function (based on the 3-bit function above):
- *f(f<sup>2</sup> - 2)*
- _f(f<sup>2</sup> - 2)_
This loop, again extended to also handle *u*, *v*, *q*, and *r* alongside *f* and *g*, placed in
This loop, again extended to also handle _u_, _v_, _q_, and _r_ alongside _f_ and _g_, placed in
`divsteps_n_matrix`, gives a significantly faster, but non-constant time version.
## 7. Final Python version
All together we need the following functions:
- A way to compute the transition matrix in constant time, using the `divsteps_n_matrix` function
from section 2, but with its loop replaced by a variant of the constant-time divstep from
section 5, extended to handle *u*, *v*, *q*, *r*:
section 5, extended to handle _u_, _v_, _q_, _r_:
```python
def divsteps_n_matrix(zeta, f, g):
@@ -684,7 +681,7 @@ def divsteps_n_matrix(zeta, f, g):
return zeta, (u, v, q, r)
```
- The functions to update *f* and *g*, and *d* and *e*, from section 2 and section 4, with the constant-time
- The functions to update _f_ and _g_, and _d_ and _e_, from section 2 and section 4, with the constant-time
changes to `update_de` from section 5:
```python
@@ -723,7 +720,7 @@ def normalize(sign, v, M):
return v
```
- And finally the `modinv` function too, adapted to use *&zeta;* instead of *&delta;*, and using the fixed
- And finally the `modinv` function too, adapted to use _&zeta;_ instead of _&delta;_, and using the fixed
iteration count from section 5:
```python
@@ -772,20 +769,21 @@ def modinv_var(M, Mi, x):
## 8. From GCDs to Jacobi symbol
We can also use a similar approach to calculate Jacobi symbol *(x | M)* by keeping track of an
extra variable *j*, for which at every step *(x | M) = j (g | f)*. As we update *f* and *g*, we
make corresponding updates to *j* using
We can also use a similar approach to calculate Jacobi symbol _(x | M)_ by keeping track of an
extra variable _j_, for which at every step _(x | M) = j (g | f)_. As we update _f_ and _g_, we
make corresponding updates to _j_ using
[properties of the Jacobi symbol](https://en.wikipedia.org/wiki/Jacobi_symbol#Properties):
* *((g/2) | f)* is either *(g | f)* or *-(g | f)*, depending on the value of *f mod 8* (negating if it's *3* or *5*).
* *(f | g)* is either *(g | f)* or *-(g | f)*, depending on *f mod 4* and *g mod 4* (negating if both are *3*).
These updates depend only on the values of *f* and *g* modulo *4* or *8*, and can thus be applied
very quickly, as long as we keep track of a few additional bits of *f* and *g*. Overall, this
- _((g/2) | f)_ is either _(g | f)_ or _-(g | f)_, depending on the value of _f mod 8_ (negating if it's _3_ or _5_).
- _(f | g)_ is either _(g | f)_ or _-(g | f)_, depending on _f mod 4_ and _g mod 4_ (negating if both are _3_).
These updates depend only on the values of _f_ and _g_ modulo _4_ or _8_, and can thus be applied
very quickly, as long as we keep track of a few additional bits of _f_ and _g_. Overall, this
calculation is slightly simpler than the one for the modular inverse because we no longer need to
keep track of *d* and *e*.
keep track of _d_ and _e_.
However, one difficulty of this approach is that the Jacobi symbol *(a | n)* is only defined for
positive odd integers *n*, whereas in the original safegcd algorithm, *f, g* can take negative
However, one difficulty of this approach is that the Jacobi symbol _(a | n)_ is only defined for
positive odd integers _n_, whereas in the original safegcd algorithm, _f, g_ can take negative
values. We resolve this by using the following modified steps:
```python
@@ -799,15 +797,16 @@ values. We resolve this by using the following modified steps:
```
The algorithm is still correct, since the changed divstep, called a "posdivstep" (see section 8.4
and E.5 in the paper) preserves *gcd(f, g)*. However, there's no proof that the modified algorithm
and E.5 in the paper) preserves _gcd(f, g)_. However, there's no proof that the modified algorithm
will converge. The justification for posdivsteps is completely empirical: in practice, it appears
that the vast majority of nonzero inputs converge to *f=g=gcd(f<sub>0</sub>, g<sub>0</sub>)* in a
that the vast majority of nonzero inputs converge to _f=g=gcd(f<sub>0</sub>, g<sub>0</sub>)_ in a
number of steps proportional to their logarithm.
Note that:
- We require inputs to satisfy *gcd(x, M) = 1*, as otherwise *f=1* is not reached.
- We require inputs *x &neq; 0*, because applying posdivstep with *g=0* has no effect.
- We need to update the termination condition from *g=0* to *f=1*.
- We require inputs to satisfy _gcd(x, M) = 1_, as otherwise _f=1_ is not reached.
- We require inputs _x &neq; 0_, because applying posdivstep with _g=0_ has no effect.
- We need to update the termination condition from _g=0_ to _f=1_.
We account for the possibility of nonconvergence by only performing a bounded number of
posdivsteps, and then falling back to square-root based Jacobi calculation if a solution has not
@@ -815,5 +814,5 @@ yet been found.
The optimizations in sections 3-7 above are described in the context of the original divsteps, but
in the C implementation we also adapt most of them (not including "avoiding modulus operations",
since it's not necessary to track *d, e*, and "constant-time operation", since we never calculate
since it's not necessary to track _d, e_, and "constant-time operation", since we never calculate
Jacobi symbols for secret data) to the posdivsteps version.

8086
external/secp256k1/src/wycheproof/ecdsa_secp256k1_sha256_bitcoin_test.json vendored
View File

File diff suppressed because one or more lines are too long

43
include/xrpl/basics/README.md
View File

@@ -4,37 +4,34 @@ Utility functions and classes.
ripple/basic should contain no dependencies on other modules.
# Choosing a rippled container.
Choosing a rippled container.
=============================
- `std::vector`
- For ordered containers with most insertions or erases at the end.
* `std::vector`
* For ordered containers with most insertions or erases at the end.
- `std::deque`
- For ordered containers with most insertions or erases at the start or end.
* `std::deque`
* For ordered containers with most insertions or erases at the start or end.
* `std::list`
* For ordered containers with inserts and erases to the middle.
* For containers with iterators stable over insert and erase.
* Generally slower and bigger than `std::vector` or `std::deque` except for
- `std::list`
- For ordered containers with inserts and erases to the middle.
- For containers with iterators stable over insert and erase.
- Generally slower and bigger than `std::vector` or `std::deque` except for
those cases.
* `std::set`
* For sorted containers.
- `std::set`
- For sorted containers.
* `ripple::hash_set`
* Where inserts and contains need to be O(1).
* For "small" sets, `std::set` might be faster and smaller.
- `ripple::hash_set`
- Where inserts and contains need to be O(1).
- For "small" sets, `std::set` might be faster and smaller.
* `ripple::hardened_hash_set`
* For data sets where the key could be manipulated by an attacker
in an attempt to mount an algorithmic complexity attack: see
- `ripple::hardened_hash_set`
- For data sets where the key could be manipulated by an attacker
in an attempt to mount an algorithmic complexity attack: see
http://en.wikipedia.org/wiki/Algorithmic_complexity_attack
The following container is deprecated
* `std::unordered_set`
* Use `ripple::hash_set` instead, which uses a better hashing algorithm.
* Or use `ripple::hardened_hash_set` to prevent algorithmic complexity attacks.
- `std::unordered_set`
- Use `ripple::hash_set` instead, which uses a better hashing algorithm.
- Or use `ripple::hardened_hash_set` to prevent algorithmic complexity attacks.

1
include/xrpl/proto/org/xrpl/rpc/v1/README.md
View File

@@ -77,4 +77,3 @@ mirror the JSON test as much as possible.
Refer to the Protocol Buffers [language
guide](https://developers.google.com/protocol-buffers/docs/proto3)
for more detailed information about Protocol Buffers.

6
include/xrpl/protocol/README.md
View File

@@ -23,9 +23,9 @@ optional fields easier to read:
if it exists, or nothing if it doesn't." This usage of the
tilde/bitwise NOT operator is not standard outside of the
`rippled` codebase.
- As a consequence of this, `x[~sfFoo] = y[~sfFoo]`
assigns the value of Foo from y to x, including omitting
Foo from x if it doesn't exist in y.
- As a consequence of this, `x[~sfFoo] = y[~sfFoo]`
assigns the value of Foo from y to x, including omitting
Foo from x if it doesn't exist in y.
Typically, for things that are guaranteed to exist, you use
`x[sfFoo]` and avoid having to deal with a container that may

32
include/xrpl/resource/README.md
View File

@@ -1,4 +1,4 @@
# Resource::Manager #
# Resource::Manager
The ResourceManager module has these responsibilities:
@@ -7,7 +7,7 @@ The ResourceManager module has these responsibilities:
- Provide an interface to share load information in a cluster.
- Warn and/or disconnect endpoints for imposing load.
## Description ##
## Description
To prevent monopolization of server resources or attacks on servers,
resource consumption is monitored at each endpoint. When consumption
@@ -33,44 +33,44 @@ Although RPC connections consume resources, they are transient and
cannot be rate limited. It is advised not to expose RPC interfaces
to the general public.
## Consumer Types ##
## Consumer Types
Consumers are placed into three classifications (as identified by the
Resource::Kind enumeration):
- InBound,
- OutBound, and
- Admin
- InBound,
- OutBound, and
- Admin
Each caller determines for itself the classification of the Consumer it is
creating.
Each caller determines for itself the classification of the Consumer it is
creating.
## Resource Loading ##
## Resource Loading
It is expected that a client will impose a higher load on the server
when it first connects: the client may need to catch up on transactions
it has missed, or get trust lines, or transfer fees. The Manager must
it has missed, or get trust lines, or transfer fees. The Manager must
expect this initial peak load, but not allow that high load to continue
because over the long term that would unduly stress the server.
If a client places a sustained high load on the server, that client
is initially given a warning message. If that high load continues
is initially given a warning message. If that high load continues
the Manager may tell the heavily loaded server to drop the connection
entirely and not allow re-connection for some amount of time.
Each load is monitored by capturing peaks and then decaying those peak
values over time: this is implemented by the DecayingSample class.
## Gossip ##
## Gossip
Each server in a cluster creates a list of IP addresses of end points
that are imposing a significant load. This list is called Gossip, which
is passed to other nodes in that cluster. Gossip helps individual
that are imposing a significant load. This list is called Gossip, which
is passed to other nodes in that cluster. Gossip helps individual
servers in the cluster identify IP addreses that might be unduly loading
the entire cluster. Again the recourse of the individual servers is to
the entire cluster. Again the recourse of the individual servers is to
drop connections to those IP addresses that occur commonly in the gossip.
## Access ##
## Access
In rippled, the Application holds a unique instance of Resource::Manager,
which may be retrieved by calling the method

5
src/test/README.md
View File

@@ -1,4 +1,3 @@
# Unit Tests
## Running Tests
@@ -12,13 +11,13 @@ just `NoRippleCheckLimits`).
More than one suite or group of suites can be specified as a comma separated
list via the argument. For example, `--unittest=beast,OversizeMeta` will run
all suites in the `beast` library (root identifier) as well as the test suite
named `OversizeMeta`). All name matches are case sensitive.
named `OversizeMeta`). All name matches are case sensitive.
Tests can be executed in parallel using several child processes by specifying
the `--unittest-jobs=N` parameter. The default behavior is to execute serially
using a single process.
The order that suites are executed is determined by the suite priority that
The order that suites are executed is determined by the suite priority that
is optionally specified when the suite is declared in the code with one of the
`BEAST_DEFINE_TESTSUITE` macros. By default, suites have a priority of 0, and
other suites can choose to declare an integer priority value to make themselves

27
src/test/csf/README.md
View File

@@ -26,7 +26,7 @@ collect when running the simulation. The specification includes:
- A collection of [`Peer`s](./Peer.h) that represent the participants in the
network, with each independently running the consensus algorithm.
- The `Peer` trust relationships as a `TrustGraph`. This is a directed graph
whose edges define what other `Peer`s a given `Peer` trusts. In other words,
whose edges define what other `Peer`s a given `Peer` trusts. In other words,
the set of out edges for a `Peer` in the graph correspond to the UNL of that
`Peer`.
- The network communication layer as a `BasicNetwork`. This models the overlay
@@ -45,6 +45,7 @@ eventually fully validating the consensus history of accepted transactions. Each
the registered `Collector`s.
## Example Simulation
Below is a basic simulation we can walk through to get an understanding of the
framework. This simulation is for a set of 5 validators that aren't directly
connected but rely on a single hub node for communication.
@@ -98,12 +99,12 @@ center[0]->runAsValidator = false;
The simulation code starts by creating a single instance of the [`Sim`
class](./Sim.h). This class is used to manage the overall simulation and
internally owns most other components, including the `Peer`s, `Scheduler`,
`BasicNetwork` and `TrustGraph`. The next two lines create two differ
`BasicNetwork` and `TrustGraph`. The next two lines create two differ
`PeerGroup`s of size 5 and 1 . A [`PeerGroup`](./PeerGroup.h) is a convenient
way for configuring a set of related peers together and internally has a vector
of pointers to the `Peer`s which are owned by the `Sim`. `PeerGroup`s can be
combined using `+/-` operators to configure more complex relationships of nodes
as shown by `PeerGroup network`. Note that each call to `createGroup` adds that
as shown by `PeerGroup network`. Note that each call to `createGroup` adds that
many new `Peer`s to the simulation, but does not specify any trust or network
relationships for the new `Peer`s.
@@ -125,14 +126,14 @@ validators.connect(center, delay);
Although the `sim` object has accessible instances of
[TrustGraph](./TrustGraph.h) and [BasicNetwork](./BasicNetwork.h), it is more
convenient to manage the graphs via the `PeerGroup`s. The first two lines
create a trust topology in which all `Peer`s trust the 5 validating `Peer`s. Or
convenient to manage the graphs via the `PeerGroup`s. The first two lines
create a trust topology in which all `Peer`s trust the 5 validating `Peer`s. Or
in the UNL perspective, all `Peer`s are configured with the same UNL listing the
5 validating `Peer`s. The two lines could've been rewritten as
`network.trust(validators)`.
The next lines create the network communication topology. Each of the validating
`Peer`s connects to the central hub `Peer` with a fixed delay of 200ms. Note
`Peer`s connects to the central hub `Peer` with a fixed delay of 200ms. Note
that the network connections are really undirected, but are represented
internally in a directed graph using edge pairs of inbound and outbound connections.
@@ -143,11 +144,11 @@ SimDurationCollector simDur;
sim.collectors.add(simDur);
```
The next lines add a single collector to the simulation. The
The next lines add a single collector to the simulation. The
`SimDurationCollector` is a simple example collector which tracks the total
duration of the simulation. More generally, a collector is any class that
duration of the simulation. More generally, a collector is any class that
implements `void on(NodeID, SimTime, Event)` for all [Events](./events.h)
emitted by a Peer. Events are arbitrary types used to indicate some action or
emitted by a Peer. Events are arbitrary types used to indicate some action or
change of state of a `Peer`. Other [existing collectors](./collectors.h) measure
latencies of transaction submission to validation or the rate of ledger closing
and monitor any jumps in ledger history.
@@ -176,9 +177,9 @@ to send transactions in at fixed or random intervals to fixed or random `Peer`s.
## Run
The example has two calls to `sim.run(1)`. This call runs the simulation until
each `Peer` has closed one additional ledger. After closing the additional
ledger, the `Peer` stops participating in consensus. The first call is used to
The example has two calls to `sim.run(1)`. This call runs the simulation until
each `Peer` has closed one additional ledger. After closing the additional
ledger, the `Peer` stops participating in consensus. The first call is used to
ensure a more useful prior state of all `Peer`s. After the transaction
submission, the second call to `run` results in one additional ledger that
accepts those transactions.
@@ -188,4 +189,4 @@ Alternatively, you can specify a duration to run the simulation, e.g.
scheduler has elapsed 10 additional seconds. The `sim.scheduler.in` or
`sim.scheduler.at` methods can schedule arbitrary code to execute at a later
time in the simulation, for example removing a network connection or modifying
the trust graph.
the trust graph.

1
src/tests/README.md
View File

@@ -1,4 +1,5 @@
# Unit tests
This directory contains unit tests for the project. The difference from existing `src/test` folder
is that we switch to 3rd party testing framework (doctest). We intend to gradually move existing tests
from our own framework to doctest and such tests will be moved to this new folder.

15
src/xrpld/app/consensus/README.md
View File

@@ -1,13 +1,12 @@
# RCL Consensus
# RCL Consensus
This directory holds the types and classes needed
to connect the generic consensus algorithm to the
rippled-specific instance of consensus.
* `RCLCxTx` adapts a `SHAMapItem` transaction.
* `RCLCxTxSet` adapts a `SHAMap` to represent a set of transactions.
* `RCLCxLedger` adapts a `Ledger`.
* `RCLConsensus` is implements the requirements of the generic
`Consensus` class by connecting to the rest of the `rippled`
application.
- `RCLCxTx` adapts a `SHAMapItem` transaction.
- `RCLCxTxSet` adapts a `SHAMap` to represent a set of transactions.
- `RCLCxLedger` adapts a `Ledger`.
- `RCLConsensus` is implements the requirements of the generic
`Consensus` class by connecting to the rest of the `rippled`
application.

249
src/xrpld/app/ledger/README.md
View File

@@ -1,9 +1,8 @@
# Ledger Process
# Ledger Process #
## Introduction
## Introduction ##
## Life Cycle ##
## Life Cycle
Every server always has an open ledger. All received new transactions are
applied to the open ledger. The open ledger can't close until we reach
@@ -37,10 +36,11 @@ round. This is a "rebase": now that we know the real history, the current open
ledger is rebased against the last closed ledger.
The purpose of the open ledger is as follows:
- Forms the basis of the initial proposal during consensus
- Used to decide if we can reject the transaction without relaying it
## Byzantine Failures ##
## Byzantine Failures
Byzantine failures are resolved as follows. If there is a supermajority ledger,
then a minority of validators will discover that the consensus round is
@@ -52,167 +52,169 @@ If there is no majority ledger, then starting on the next consensus round there
will not be a consensus on the last closed ledger. Another avalanche process
is started.
## Validators ##
## Validators
The only meaningful difference between a validator and a 'regular' server is
that the validator sends its proposals and validations to the network.
---
# The Ledger Stream #
# The Ledger Stream
## Ledger Priorities ##
## Ledger Priorities
There are two ledgers that are the most important for a rippled server to have:
- The consensus ledger and
- The last validated ledger.
- The consensus ledger and
- The last validated ledger.
If we need either of those two ledgers they are fetched with the highest
priority. Also, when they arrive, they replace their earlier counterparts
priority. Also, when they arrive, they replace their earlier counterparts
(if they exist).
The `LedgerMaster` object tracks
- the last published ledger,
- the last validated ledger, and
- ledger history.
So the `LedgerMaster` is at the center of fetching historical ledger data.
- the last published ledger,
- the last validated ledger, and
- ledger history.
So the `LedgerMaster` is at the center of fetching historical ledger data.
In specific, the `LedgerMaster::doAdvance()` method triggers the code that
fetches historical data and controls the state machine for ledger acquisition.
The server tries to publish an on-going stream of consecutive ledgers to its
clients. After the server has started and caught up with network
clients. After the server has started and caught up with network
activity, say when ledger 500 is being settled, then the server puts its best
effort into publishing validated ledger 500 followed by validated ledger 501
and then 502. This effort continues until the server is shut down.
and then 502. This effort continues until the server is shut down.
But loading or network connectivity may sometimes interfere with that ledger
stream. So suppose the server publishes validated ledger 600 and then
receives validated ledger 603. Then the server wants to back fill its ledger
stream. So suppose the server publishes validated ledger 600 and then
receives validated ledger 603. Then the server wants to back fill its ledger
history with ledgers 601 and 602.
The server prioritizes keeping up with current ledgers. But if it is caught
The server prioritizes keeping up with current ledgers. But if it is caught
up on the current ledger, and there are no higher priority demands on the
server, then it will attempt to back fill its historical ledgers. It fills
server, then it will attempt to back fill its historical ledgers. It fills
in the historical ledger data first by attempting to retrieve it from the
local database. If the local database does not have all of the necessary data
local database. If the local database does not have all of the necessary data
then the server requests the remaining information from network peers.
Suppose the server is missing multiple historical ledgers. Take the previous
example where we have ledgers 603 and 600, but we're missing 601 and 602. In
Suppose the server is missing multiple historical ledgers. Take the previous
example where we have ledgers 603 and 600, but we're missing 601 and 602. In
that case the server requests information for ledger 602 first, before
back-filling ledger 601. We want to expand the contiguous range of
most-recent ledgers that the server has locally. There's also a limit to
how much historical ledger data is useful. So if we're on ledger 603, but
back-filling ledger 601. We want to expand the contiguous range of
most-recent ledgers that the server has locally. There's also a limit to
how much historical ledger data is useful. So if we're on ledger 603, but
we're missing ledger 4 we may not bother asking for ledger 4.
## Assembling a Ledger ##
## Assembling a Ledger
When data for a ledger arrives from a peer, it may take a while before the
server can apply that data. So when ledger data arrives we schedule a job
thread to apply that data. If more data arrives before the job starts we add
that data to the job. We defer requesting more ledger data until all of the
data we have for that ledger has been processed. Once all of that data is
server can apply that data. So when ledger data arrives we schedule a job
thread to apply that data. If more data arrives before the job starts we add
that data to the job. We defer requesting more ledger data until all of the
data we have for that ledger has been processed. Once all of that data is
processed we can intelligently request only the additional data that we need
to fill in the ledger. This reduces network traffic and minimizes the load
to fill in the ledger. This reduces network traffic and minimizes the load
on peers supplying the data.
If we receive data for a ledger that is not currently under construction,
we don't just throw the data away. In particular the AccountStateNodes
may be useful, since they can be re-used across ledgers. This data is
we don't just throw the data away. In particular the AccountStateNodes
may be useful, since they can be re-used across ledgers. This data is
stashed in memory (not the database) where the acquire process can find
it.
Peers deliver ledger data in the order in which the data can be validated.
Data arrives in the following order:
1. The hash of the ledger header
2. The ledger header
3. The root nodes of the transaction tree and state tree
4. The lower (non-root) nodes of the state tree
5. The lower (non-root) nodes of the transaction tree
1. The hash of the ledger header
2. The ledger header
3. The root nodes of the transaction tree and state tree
4. The lower (non-root) nodes of the state tree
5. The lower (non-root) nodes of the transaction tree
Inner-most nodes are supplied before outer nodes. This allows the
Inner-most nodes are supplied before outer nodes. This allows the
requesting server to hook things up (and validate) in the order in which
data arrives.
If this process fails, then a server can also ask for ledger data by hash,
rather than by asking for specific nodes in a ledger. Asking for information
rather than by asking for specific nodes in a ledger. Asking for information
by hash is less efficient, but it allows a peer to return the information
even if the information is not assembled into a tree. All the peer needs is
even if the information is not assembled into a tree. All the peer needs is
the raw data.
## Which Peer To Ask ##
## Which Peer To Ask
Peers go though state transitions as the network goes through its state
transitions. Peer's provide their state to their directly connected peers.
transitions. Peer's provide their state to their directly connected peers.
By monitoring the state of each connected peer a server can tell which of
its peers has the information that it needs.
Therefore if a server suffers a byzantine failure the server can tell which
of its peers did not suffer that same failure. So the server knows which
of its peers did not suffer that same failure. So the server knows which
peer(s) to ask for the missing information.
Peers also report their contiguous range of ledgers. This is another way that
Peers also report their contiguous range of ledgers. This is another way that
a server can determine which peer to ask for a particular ledger or piece of
a ledger.
There are also indirect peer queries. If there have been timeouts while
acquiring ledger data then a server may issue indirect queries. In that
There are also indirect peer queries. If there have been timeouts while
acquiring ledger data then a server may issue indirect queries. In that
case the server receiving the indirect query passes the query along to any
of its peers that may have the requested data. This is important if the
network has a byzantine failure. It also helps protect the validation
network. A validator may need to get a peer set from one of the other
of its peers that may have the requested data. This is important if the
network has a byzantine failure. It also helps protect the validation
network. A validator may need to get a peer set from one of the other
validators, and indirect queries improve the likelihood of success with
that.
## Kinds of Fetch Packs ##
## Kinds of Fetch Packs
A FetchPack is the way that peers send partial ledger data to other peers
so the receiving peer can reconstruct a ledger.
A 'normal' FetchPack is a bucket of nodes indexed by hash. The server
A 'normal' FetchPack is a bucket of nodes indexed by hash. The server
building the FetchPack puts information into the FetchPack that the
destination server is likely to need. Normally they contain all of the
destination server is likely to need. Normally they contain all of the
missing nodes needed to fill in a ledger.
A 'compact' FetchPack, on the other hand, contains only leaf nodes, no
inner nodes. Because there are no inner nodes, the ledger information that
it contains cannot be validated as the ledger is assembled. We have to,
inner nodes. Because there are no inner nodes, the ledger information that
it contains cannot be validated as the ledger is assembled. We have to,
initially, take the accuracy of the FetchPack for granted and assemble the
ledger. Once the entire ledger is assembled the entire ledger can be
validated. But if the ledger does not validate then there's nothing to be
ledger. Once the entire ledger is assembled the entire ledger can be
validated. But if the ledger does not validate then there's nothing to be
done but throw the entire FetchPack away; there's no way to save a portion
of the FetchPack.
The FetchPacks just described could be termed 'reverse FetchPacks.' They
only provide historical data. There may be a use for what could be called a
'forward FetchPack.' A forward FetchPack would contain the information that
The FetchPacks just described could be termed 'reverse FetchPacks.' They
only provide historical data. There may be a use for what could be called a
'forward FetchPack.' A forward FetchPack would contain the information that
is needed to build a new ledger out of the preceding ledger.
A forward compact FetchPack would need to contain:
- The header for the new ledger,
- The leaf nodes of the transaction tree (if there is one),
- The index of deleted nodes in the state tree,
- The index and data for new nodes in the state tree, and
- The index and new data of modified nodes in the state tree.
- The header for the new ledger,
- The leaf nodes of the transaction tree (if there is one),
- The index of deleted nodes in the state tree,
- The index and data for new nodes in the state tree, and
- The index and new data of modified nodes in the state tree.
---
# Definitions #
# Definitions
## Open Ledger ##
## Open Ledger
The open ledger is the ledger that the server applies all new incoming
transactions to.
## Last Validated Ledger ##
## Last Validated Ledger
The most recent ledger that the server is certain will always remain part
of the permanent, public history.
## Last Closed Ledger ##
## Last Closed Ledger
The most recent ledger that the server believes the network reached consensus
on. Different servers can arrive at a different conclusion about the last
@@ -220,29 +222,29 @@ closed ledger. This is a consequence of Byzantanine failure. The purpose of
validations is to resolve the differences between servers and come to a common
conclusion about which last closed ledger is authoritative.
## Consensus ##
## Consensus
A distributed agreement protocol. Ripple uses the consensus process to solve
the problem of double-spending.
## Validation ##
## Validation
A signed statement indicating that it built a particular ledger as a result
of the consensus process.
## Proposal ##
## Proposal
A signed statement of which transactions it believes should be included in
the next consensus ledger.
## Ledger Header ##
## Ledger Header
The "ledger header" is the chunk of data that hashes to the
ledger's hash. It contains the sequence number, parent hash,
hash of the previous ledger, hash of the root node of the
state tree, and so on.
## Ledger Base ##
## Ledger Base
The term "ledger base" refers to a particular type of query
and response used in the ledger fetch process that includes
@@ -251,9 +253,9 @@ such as the root node of the state tree.
---
# Ledger Structures #
# Ledger Structures
## Account Root ##
## Account Root
**Account:** A 160-bit account ID.
@@ -264,8 +266,8 @@ such as the root node of the state tree.
**LedgerEntryType:** "AccountRoot"
**OwnerCount:** The number of items the account owns that are charged to the
account. Offers are charged to the account. Trust lines may be charged to
the account (but not necessarily). The OwnerCount determines the reserve on
account. Offers are charged to the account. Trust lines may be charged to
the account (but not necessarily). The OwnerCount determines the reserve on
the account.
**PreviousTxnID:** 256-bit index of the previous transaction on this account.
@@ -274,43 +276,45 @@ the account.
transaction on this account.
**Sequence:** Must be a value of 1 for the account to process a valid
transaction. The value initially matches the sequence number of the state
tree of the account that signed the transaction. The process of executing
the transaction increments the sequence number. This is how ripple prevents
transaction. The value initially matches the sequence number of the state
tree of the account that signed the transaction. The process of executing
the transaction increments the sequence number. This is how ripple prevents
a transaction from executing more than once.
**index:** 256-bit hash of this AccountRoot.
## Trust Line ##
## Trust Line
The trust line acts as an edge connecting two accounts: the accounts
represented by the HighNode and the LowNode. Which account is "high" and
"low" is determined by the values of the two 160-bit account IDs. The
account with the smaller 160-bit ID is always the low account. This
represented by the HighNode and the LowNode. Which account is "high" and
"low" is determined by the values of the two 160-bit account IDs. The
account with the smaller 160-bit ID is always the low account. This
ordering makes the hash of a trust line between accounts A and B have the
same value as a trust line between accounts B and A.
**Balance:**
- **currency:** String identifying a valid currency, e.g., "BTC".
- **issuer:** There is no issuer, really, this entry is "NoAccount".
- **value:**
- **currency:** String identifying a valid currency, e.g., "BTC".
- **issuer:** There is no issuer, really, this entry is "NoAccount".
- **value:**
**Flags:** ???
**HighLimit:**
- **currency:** Same as for Balance.
- **issuer:** A 160-bit account ID.
- **value:** The largest amount this issuer will accept of the currency.
- **currency:** Same as for Balance.
- **issuer:** A 160-bit account ID.
- **value:** The largest amount this issuer will accept of the currency.
**HighNode:** A deletion hint.
**LedgerEntryType:** "RippleState".
**LowLimit:**
- **currency:** Same as for Balance.
- **issuer:** A 160-bit account ID.
- **value:** The largest amount of the currency this issuer will accept.
- **currency:** Same as for Balance.
- **issuer:** A 160-bit account ID.
- **value:** The largest amount of the currency this issuer will accept.
**LowNode:** A deletion hint
@@ -321,8 +325,7 @@ transaction on this account.
**index:** 256-bit hash of this RippleState.
## Ledger Hashes ##
## Ledger Hashes
**Flags:** ???
@@ -334,8 +337,7 @@ transaction on this account.
**index:** 256-bit hash of this LedgerHashes.
## Owner Directory ##
## Owner Directory
Lists all of the offers and trust lines that are associated with an account.
@@ -351,8 +353,7 @@ Lists all of the offers and trust lines that are associated with an account.
**index:** A hash of the owner account.
## Book Directory ##
## Book Directory
Lists one or more offers that have the same quality.
@@ -360,18 +361,18 @@ If a pair of Currency and Issuer fields are all zeros, then that pair is
dealing in XRP.
The code, at the moment, does not recognize that the Currency and Issuer
fields are currencies and issuers. So those values are presented in hex,
rather than as accounts and currencies. That's a bug and should be fixed
fields are currencies and issuers. So those values are presented in hex,
rather than as accounts and currencies. That's a bug and should be fixed
at some point.
**ExchangeRate:** A 64-bit value. The first 8-bits is the exponent and the
remaining bits are the mantissa. The format is such that a bigger 64-bit
**ExchangeRate:** A 64-bit value. The first 8-bits is the exponent and the
remaining bits are the mantissa. The format is such that a bigger 64-bit
value always represents a higher exchange rate.
Each type can compute its own hash. The hash of a book directory contains,
as its lowest 64 bits, the exchange rate. This means that if there are
multiple *almost* identical book directories, but with different exchange
rates, then these book directories will sit together in the ledger. The best
Each type can compute its own hash. The hash of a book directory contains,
as its lowest 64 bits, the exchange rate. This means that if there are
multiple _almost_ identical book directories, but with different exchange
rates, then these book directories will sit together in the ledger. The best
exchange rate will be the first in the sequence of Book Directories.
**Flags:** ???
@@ -392,14 +393,14 @@ currencies described by this BookDirectory.
**TakerPaysIssuer:** Issuer of the PaysCurrency.
**index:** A 256-bit hash computed using the TakerGetsCurrency, TakerGetsIssuer,
TakerPaysCurrency, and TakerPaysIssuer in the top 192 bits. The lower 64-bits
TakerPaysCurrency, and TakerPaysIssuer in the top 192 bits. The lower 64-bits
are occupied by the exchange rate.
---
# Ledger Publication #
# Ledger Publication
## Overview ##
## Overview
The Ripple server permits clients to subscribe to a continuous stream of
fully-validated ledgers. The publication code maintains this stream.
@@ -408,7 +409,7 @@ The server attempts to maintain this continuous stream unless it falls
too far behind, in which case it jumps to the current fully-validated
ledger and then attempts to resume a continuous stream.
## Implementation ##
## Implementation
`LedgerMaster::doAdvance` is invoked when work may need to be done to
publish ledgers to clients. This code loops until it cannot make further
@@ -430,17 +431,17 @@ the list of resident ledgers.
---
# The Ledger Cleaner #
# The Ledger Cleaner
## Overview ##
## Overview
The ledger cleaner checks and, if necessary, repairs the SQLite ledger and
transaction databases. It can also check for pieces of a ledger that should
be in the node back end but are missing. If it detects this case, it
triggers a fetch of the ledger. The ledger cleaner only operates by manual
transaction databases. It can also check for pieces of a ledger that should
be in the node back end but are missing. If it detects this case, it
triggers a fetch of the ledger. The ledger cleaner only operates by manual
request. It is never started automatically.
## Operations ##
## Operations
The ledger cleaner can operate on a single ledger or a range of ledgers. It
always validates the ledger chain itself, ensuring that the SQLite database
@@ -448,7 +449,7 @@ contains a consistent chain of ledgers from the last validated ledger as far
back as the database goes.
If requested, it can additionally repair the SQLite entries for transactions
in each checked ledger. This was primarily intended to repair incorrect
in each checked ledger. This was primarily intended to repair incorrect
entries created by a bug (since fixed) that could cause transasctions from a
ledger other than the fully-validated ledger to appear in the SQLite
databases in addition to the transactions from the correct ledger.
@@ -460,7 +461,7 @@ To prevent the ledger cleaner from saturating the available I/O bandwidth
and excessively polluting caches with ancient information, the ledger
cleaner paces itself and does not attempt to get its work done quickly.
## Commands ##
## Commands
The ledger cleaner can be controlled and monitored with the **ledger_cleaner**
RPC command. With no parameters, this command reports on the status of the
@@ -486,4 +487,4 @@ ledger(s) for missing nodes in the back end node store
---
# References #
# References

279
src/xrpld/app/misc/FeeEscalation.md
View File

@@ -17,15 +17,16 @@ transactions into the open ledger, even during unfavorable conditions.
How fees escalate:
1. There is a base [fee level](#fee-level) of 256,
which is the minimum that a typical transaction
is required to pay. For a [reference
transaction](#reference-transaction), that corresponds to the
network base fee, which is currently 10 drops.
which is the minimum that a typical transaction
is required to pay. For a [reference
transaction](#reference-transaction), that corresponds to the
network base fee, which is currently 10 drops.
2. However, there is a limit on the number of transactions that
can get into an open ledger for that base fee level. The limit
will vary based on the [health](#consensus-health) of the
consensus process, but will be at least [5](#other-constants).
* If consensus stays [healthy](#consensus-health), the limit will
can get into an open ledger for that base fee level. The limit
will vary based on the [health](#consensus-health) of the
consensus process, but will be at least [5](#other-constants).
- If consensus stays [healthy](#consensus-health), the limit will
be the max of the number of transactions in the validated ledger
plus [20%](#other-constants) or the current limit until it gets
to [50](#other-constants), at which point, the limit will be the
@@ -35,50 +36,56 @@ consensus process, but will be at least [5](#other-constants).
decreases (i.e. a large ledger is no longer recent), the limit will
decrease to the new largest value by 10% each time the ledger has
more than 50 transactions.
* If consensus does not stay [healthy](#consensus-health),
- If consensus does not stay [healthy](#consensus-health),
the limit will clamp down to the smaller of the number of
transactions in the validated ledger minus [50%](#other-constants)
or the previous limit minus [50%](#other-constants).
* The intended effect of these mechanisms is to allow as many base fee
- The intended effect of these mechanisms is to allow as many base fee
level transactions to get into the ledger as possible while the
network is [healthy](#consensus-health), but to respond quickly to
any condition that makes it [unhealthy](#consensus-health), including,
but not limited to, malicious attacks.
3. Once there are more transactions in the open ledger than indicated
by the limit, the required fee level jumps drastically.
* The formula is `( lastLedgerMedianFeeLevel *
TransactionsInOpenLedger^2 / limit^2 )`,
by the limit, the required fee level jumps drastically.
- The formula is `( lastLedgerMedianFeeLevel *
TransactionsInOpenLedger^2 / limit^2 )`,
and returns a [fee level](#fee-level).
4. That may still be pretty small, but as more transactions get
into the ledger, the fee level increases exponentially.
* For example, if the limit is 6, and the median fee is minimal,
into the ledger, the fee level increases exponentially.
- For example, if the limit is 6, and the median fee is minimal,
and assuming all [reference transactions](#reference-transaction),
the 8th transaction only requires a [level](#fee-level) of about 174,000
or about 6800 drops,
but the 20th transaction requires a [level](#fee-level) of about
1,283,000 or about 50,000 drops.
5. Finally, as each ledger closes, the median fee level of that ledger is
computed and used as `lastLedgerMedianFeeLevel` (with a
[minimum value of 128,000](#other-constants))
in the fee escalation formula for the next open ledger.
* Continuing the example above, if ledger consensus completes with
computed and used as `lastLedgerMedianFeeLevel` (with a
[minimum value of 128,000](#other-constants))
in the fee escalation formula for the next open ledger.
- Continuing the example above, if ledger consensus completes with
only those 20 transactions, and all of those transactions paid the
minimum required fee at each step, the limit will be adjusted from
6 to 24, and the `lastLedgerMedianFeeLevel` will be about 322,000,
which is 12,600 drops for a
[reference transaction](#reference-transaction).
* This will only require 10 drops for the first 25 transactions,
- This will only require 10 drops for the first 25 transactions,
but the 26th transaction will require a level of about 349,150
or about 13,649 drops.
* This example assumes a cold-start scenario, with a single, possibly
malicious, user willing to pay arbitrary amounts to get transactions
into the open ledger. It ignores the effects of the [Transaction
Queue](#transaction-queue). Any lower fee level transactions submitted
by other users at the same time as this user's transactions will go into
the transaction queue, and will have the first opportunity to be applied
to the _next_ open ledger. The next section describes how that works in
more detail.
- This example assumes a cold-start scenario, with a single, possibly
malicious, user willing to pay arbitrary amounts to get transactions
into the open ledger. It ignores the effects of the [Transaction
Queue](#transaction-queue). Any lower fee level transactions submitted
by other users at the same time as this user's transactions will go into
the transaction queue, and will have the first opportunity to be applied
to the _next_ open ledger. The next section describes how that works in
more detail.
## Transaction Queue
@@ -92,33 +99,34 @@ traffic periods, and give those transactions a much better chance to
succeed.
1. If an incoming transaction meets both the base [fee
level](#fee-level) and the [load fee](#load-fee) minimum, but does not have a high
enough [fee level](#fee-level) to immediately go into the open ledger,
it is instead put into the queue and broadcast to peers. Each peer will
then make an independent decision about whether to put the transaction
into its open ledger or the queue. In principle, peers with identical
open ledgers will come to identical decisions. Any discrepancies will be
resolved as usual during consensus.
level](#fee-level) and the [load fee](#load-fee) minimum, but does not have a high
enough [fee level](#fee-level) to immediately go into the open ledger,
it is instead put into the queue and broadcast to peers. Each peer will
then make an independent decision about whether to put the transaction
into its open ledger or the queue. In principle, peers with identical
open ledgers will come to identical decisions. Any discrepancies will be
resolved as usual during consensus.
2. When consensus completes, the open ledger limit is adjusted, and
the required [fee level](#fee-level) drops back to the base
[fee level](#fee-level). Before the ledger is made available to
external transactions, transactions are applied from the queue to the
ledger from highest [fee level](#fee-level) to lowest. These transactions
count against the open ledger limit, so the required [fee level](#fee-level)
may start rising during this process.
the required [fee level](#fee-level) drops back to the base
[fee level](#fee-level). Before the ledger is made available to
external transactions, transactions are applied from the queue to the
ledger from highest [fee level](#fee-level) to lowest. These transactions
count against the open ledger limit, so the required [fee level](#fee-level)
may start rising during this process.
3. Once the queue is empty, or the required [fee level](#fee-level)
rises too high for the remaining transactions in the queue, the ledger
is opened up for normal transaction processing.
rises too high for the remaining transactions in the queue, the ledger
is opened up for normal transaction processing.
4. A transaction in the queue can stay there indefinitely in principle,
but in practice, either
* it will eventually get applied to the ledger,
* it will attempt to apply to the ledger and fail,
* it will attempt to apply to the ledger and retry [10
but in practice, either
- it will eventually get applied to the ledger,
- it will attempt to apply to the ledger and fail,
- it will attempt to apply to the ledger and retry [10
times](#other-constants),
* its last ledger sequence number will expire,
* the user will replace it by submitting another transaction with the same
- its last ledger sequence number will expire,
- the user will replace it by submitting another transaction with the same
sequence number and at least a [25% higher fee](#other-constants), or
* it will get dropped when the queue fills up with more valuable transactions.
- it will get dropped when the queue fills up with more valuable transactions.
The size limit is computed dynamically, and can hold transactions for
the next [20 ledgers](#other-constants) (restricted to a minimum of
[2000 transactions](#other-constants)). The lower the transaction's
@@ -128,14 +136,15 @@ If a transaction is submitted for an account with one or more transactions
already in the queue, and a sequence number that is sequential with the other
transactions in the queue for that account, it will be considered
for the queue if it meets these additional criteria:
* the account has fewer than [10](#other-constants) transactions
- the account has fewer than [10](#other-constants) transactions
already in the queue.
* all other queued transactions for that account, in the case where
- all other queued transactions for that account, in the case where
they spend the maximum possible XRP, leave enough XRP balance to pay
the fee,
* the total fees for the other queued transactions are less than both
- the total fees for the other queued transactions are less than both
the network's minimum reserve and the account's XRP balance, and
* none of the prior queued transactions affect the ability of subsequent
- none of the prior queued transactions affect the ability of subsequent
transactions to claim a fee.
Currently, there is an additional restriction that the queue cannot work with
@@ -148,7 +157,7 @@ development will make the queue aware of `sfAccountTxnID` mechanisms.
### Fee Level
"Fee level" is used to allow the cost of different types of transactions
to be compared directly. For a [reference
to be compared directly. For a [reference
transaction](#reference-transaction), the base fee
level is 256. If a transaction is submitted with a higher `Fee` field,
the fee level is scaled appropriately.
@@ -157,16 +166,16 @@ Examples, assuming a [reference transaction](#reference-transaction)
base fee of 10 drops:
1. A single-signed [reference transaction](#reference-transaction)
with `Fee=20` will have a fee level of
`20 drop fee * 256 fee level / 10 drop base fee = 512 fee level`.
with `Fee=20` will have a fee level of
`20 drop fee * 256 fee level / 10 drop base fee = 512 fee level`.
2. A multi-signed [reference transaction](#reference-transaction) with
3 signatures (base fee = 40 drops) and `Fee=60` will have a fee level of
`60 drop fee * 256 fee level / ((1tx + 3sigs) * 10 drop base fee) = 384
3 signatures (base fee = 40 drops) and `Fee=60` will have a fee level of
`60 drop fee * 256 fee level / ((1tx + 3sigs) * 10 drop base fee) = 384
fee level`.
3. A hypothetical future non-reference transaction with a base
fee of 15 drops multi-signed with 5 signatures and `Fee=90` will
have a fee level of
`90 drop fee * 256 fee level / ((1tx + 5sigs) * 15 drop base fee) = 256
fee of 15 drops multi-signed with 5 signatures and `Fee=90` will
have a fee level of
`90 drop fee * 256 fee level / ((1tx + 5sigs) * 15 drop base fee) = 256
fee level`.
This demonstrates that a simpler transaction paying less XRP can be more
@@ -194,7 +203,7 @@ For consensus to be considered healthy, the peers on the network
should largely remain in sync with one another. It is particularly
important for the validators to remain in sync, because that is required
for participation in consensus. However, the network tolerates some
validators being out of sync. Fundamentally, network health is a
validators being out of sync. Fundamentally, network health is a
function of validators reaching consensus on sets of recently submitted
transactions.
@@ -214,61 +223,61 @@ often coincides with new ledgers with zero transactions.
### Other Constants
* *Base fee transaction limit per ledger*. The minimum value of 5 was
chosen to ensure the limit never gets so small that the ledger becomes
unusable. The "target" value of 50 was chosen so the limit never gets large
enough to invite abuse, but keeps up if the network stays healthy and
active. These exact values were chosen experimentally, and can easily
change in the future.
* *Expected ledger size growth and reduction percentages*. The growth
value of 20% was chosen to allow the limit to grow quickly as load
increases, but not so quickly as to allow bad actors to run unrestricted.
The reduction value of 50% was chosen to cause the limit to drop
significantly, but not so drastically that the limit cannot quickly
recover if the problem is temporary. These exact values were chosen
experimentally, and can easily change in the future.
* *Minimum `lastLedgerMedianFeeLevel`*. The value of 500 was chosen to
ensure that the first escalated fee was more significant and noticable
than what the default would allow. This exact value was chosen
experimentally, and can easily change in the future.
* *Transaction queue size limit*. The limit is computed based on the
base fee transaction limit per ledger, so that the queue can grow
automatically as the network's performance improves, allowing
more transactions per second, and thus more transactions per ledger
to process successfully. The limit of 20 ledgers was used to provide
a balance between resource (specifically memory) usage, and giving
transactions a realistic chance to be processed. The minimum size of
2000 transactions was chosen to allow a decent functional backlog during
network congestion conditions. These exact values were
chosen experimentally, and can easily change in the future.
* *Maximum retries*. A transaction in the queue can attempt to apply
to the open ledger, but get a retry (`ter`) code up to 10 times, at
which point, it will be removed from the queue and dropped. The
value was chosen to be large enough to allow temporary failures to clear
up, but small enough that the queue doesn't fill up with stale
transactions which prevent lower fee level, but more likely to succeed,
transactions from queuing.
* *Maximum transactions per account*. A single account can have up to 10
transactions in the queue at any given time. This is primarily to
mitigate the lost cost of broadcasting multiple transactions if one of
the earlier ones fails or is otherwise removed from the queue without
being applied to the open ledger. The value was chosen arbitrarily, and
can easily change in the future.
* *Minimum last ledger sequence buffer*. If a transaction has a
`LastLedgerSequence` value, and cannot be processed into the open
ledger, that `LastLedgerSequence` must be at least 2 more than the
sequence number of the open ledger to be considered for the queue. The
value was chosen to provide a balance between letting the user control
the lifespan of the transaction, and giving a queued transaction a
chance to get processed out of the queue before getting discarded,
particularly since it may have dependent transactions also in the queue,
which will never succeed if this one is discarded.
* *Replaced transaction fee increase*. Any transaction in the queue can be
replaced by another transaction with the same sequence number and at
least a 25% higher fee level. The 25% increase is intended to cover the
resource cost incurred by broadcasting the original transaction to the
network. This value was chosen experimentally, and can easily change in
the future.
- _Base fee transaction limit per ledger_. The minimum value of 5 was
chosen to ensure the limit never gets so small that the ledger becomes
unusable. The "target" value of 50 was chosen so the limit never gets large
enough to invite abuse, but keeps up if the network stays healthy and
active. These exact values were chosen experimentally, and can easily
change in the future.
- _Expected ledger size growth and reduction percentages_. The growth
value of 20% was chosen to allow the limit to grow quickly as load
increases, but not so quickly as to allow bad actors to run unrestricted.
The reduction value of 50% was chosen to cause the limit to drop
significantly, but not so drastically that the limit cannot quickly
recover if the problem is temporary. These exact values were chosen
experimentally, and can easily change in the future.
- _Minimum `lastLedgerMedianFeeLevel`_. The value of 500 was chosen to
ensure that the first escalated fee was more significant and noticable
than what the default would allow. This exact value was chosen
experimentally, and can easily change in the future.
- _Transaction queue size limit_. The limit is computed based on the
base fee transaction limit per ledger, so that the queue can grow
automatically as the network's performance improves, allowing
more transactions per second, and thus more transactions per ledger
to process successfully. The limit of 20 ledgers was used to provide
a balance between resource (specifically memory) usage, and giving
transactions a realistic chance to be processed. The minimum size of
2000 transactions was chosen to allow a decent functional backlog during
network congestion conditions. These exact values were
chosen experimentally, and can easily change in the future.
- _Maximum retries_. A transaction in the queue can attempt to apply
to the open ledger, but get a retry (`ter`) code up to 10 times, at
which point, it will be removed from the queue and dropped. The
value was chosen to be large enough to allow temporary failures to clear
up, but small enough that the queue doesn't fill up with stale
transactions which prevent lower fee level, but more likely to succeed,
transactions from queuing.
- _Maximum transactions per account_. A single account can have up to 10
transactions in the queue at any given time. This is primarily to
mitigate the lost cost of broadcasting multiple transactions if one of
the earlier ones fails or is otherwise removed from the queue without
being applied to the open ledger. The value was chosen arbitrarily, and
can easily change in the future.
- _Minimum last ledger sequence buffer_. If a transaction has a
`LastLedgerSequence` value, and cannot be processed into the open
ledger, that `LastLedgerSequence` must be at least 2 more than the
sequence number of the open ledger to be considered for the queue. The
value was chosen to provide a balance between letting the user control
the lifespan of the transaction, and giving a queued transaction a
chance to get processed out of the queue before getting discarded,
particularly since it may have dependent transactions also in the queue,
which will never succeed if this one is discarded.
- _Replaced transaction fee increase_. Any transaction in the queue can be
replaced by another transaction with the same sequence number and at
least a 25% higher fee level. The 25% increase is intended to cover the
resource cost incurred by broadcasting the original transaction to the
network. This value was chosen experimentally, and can easily change in
the future.
### `fee` command
@@ -287,6 +296,7 @@ ledger. It includes the sequence number of the current open ledger,
but may not make sense if rippled is not synced to the network.
Result format:
```
{
"result" : {
@@ -319,13 +329,13 @@ without warning.**
Up to two fields in `server_info` output are related to fee escalation.
1. `load_factor_fee_escalation`: The factor on base transaction cost
that a transaction must pay to get into the open ledger. This value can
change quickly as transactions are processed from the network and
ledgers are closed. If not escalated, the value is 1, so will not be
returned.
that a transaction must pay to get into the open ledger. This value can
change quickly as transactions are processed from the network and
ledgers are closed. If not escalated, the value is 1, so will not be
returned.
2. `load_factor_fee_queue`: If the queue is full, this is the factor on
base transaction cost that a transaction must pay to get into the queue.
If not full, the value is 1, so will not be returned.
base transaction cost that a transaction must pay to get into the queue.
If not full, the value is 1, so will not be returned.
In all cases, the transaction fee must be high enough to overcome both
`load_factor_fee_queue` and `load_factor` to be considered. It does not
@@ -341,22 +351,21 @@ without warning.**
Three fields in `server_state` output are related to fee escalation.
1. `load_factor_fee_escalation`: The factor on base transaction cost
that a transaction must pay to get into the open ledger. This value can
change quickly as transactions are processed from the network and
ledgers are closed. The ratio between this value and
`load_factor_fee_reference` determines the multiplier for transaction
fees to get into the current open ledger.
that a transaction must pay to get into the open ledger. This value can
change quickly as transactions are processed from the network and
ledgers are closed. The ratio between this value and
`load_factor_fee_reference` determines the multiplier for transaction
fees to get into the current open ledger.
2. `load_factor_fee_queue`: This is the factor on base transaction cost
that a transaction must pay to get into the queue. The ratio between
this value and `load_factor_fee_reference` determines the multiplier for
transaction fees to get into the transaction queue to be considered for
a later ledger.
that a transaction must pay to get into the queue. The ratio between
this value and `load_factor_fee_reference` determines the multiplier for
transaction fees to get into the transaction queue to be considered for
a later ledger.
3. `load_factor_fee_reference`: Like `load_base`, this is the baseline
that is used to scale fee escalation computations.
that is used to scale fee escalation computations.
In all cases, the transaction fee must be high enough to overcome both
`load_factor_fee_queue` and `load_factor` to be considered. It does not
need to overcome `load_factor_fee_escalation`, though if it does not, it
is more likely to be queued than immediately processed into the open
ledger.

34
src/xrpld/app/misc/README.md
View File

@@ -71,18 +71,18 @@ Amendment must receive at least an 80% approval rate from validating nodes for
a period of two weeks before being accepted. The following example outlines the
process of an Amendment from its conception to approval and usage.
* A community member proposes to change transaction processing in some way.
- A community member proposes to change transaction processing in some way.
The proposal is discussed amongst the community and receives its support
creating a community or human consensus.
* Some members contribute their time and work to develop the Amendment.
- Some members contribute their time and work to develop the Amendment.
* A pull request is created and the new code is folded into a rippled build
- A pull request is created and the new code is folded into a rippled build
and made available for use.
* The consensus process begins with the validating nodes.
- The consensus process begins with the validating nodes.
* If the Amendment holds an 80% majority for a two week period, nodes will begin
- If the Amendment holds an 80% majority for a two week period, nodes will begin
including the transaction to enable it in their initial sets.
Nodes may veto Amendments they consider undesirable by never announcing their
@@ -112,7 +112,7 @@ enabled.
Optional online deletion happens through the SHAMapStore. Records are deleted
from disk based on ledger sequence number. These records reside in the
key-value database as well as in the SQLite ledger and transaction databases.
key-value database as well as in the SQLite ledger and transaction databases.
Without online deletion storage usage grows without bounds. It can only
be pruned by stopping, manually deleting data, and restarting the server.
Online deletion requires less operator intervention to manage the server.
@@ -142,14 +142,14 @@ server restarts.
Configuration:
* In the [node_db] configuration section, an optional online_delete parameter is
set. If not set or if set to 0, online delete is disabled. Otherwise, the
setting defines number of ledgers between deletion cycles.
* Another optional parameter in [node_db] is that for advisory_delete. It is
disabled by default. If set to non-zero, requires an RPC call to activate the
deletion routine.
* online_delete must not be greater than the [ledger_history] parameter.
* [fetch_depth] will be silently set to equal the online_delete setting if
online_delete is greater than fetch_depth.
* In the [node_db] section, there is a performance tuning option, delete_batch,
which sets the maximum number of SQL rows deleted per batch.
- In the [node_db] configuration section, an optional online_delete parameter is
set. If not set or if set to 0, online delete is disabled. Otherwise, the
setting defines number of ledgers between deletion cycles.
- Another optional parameter in [node_db] is that for advisory_delete. It is
disabled by default. If set to non-zero, requires an RPC call to activate the
deletion routine.
- online_delete must not be greater than the [ledger_history] parameter.
- [fetch_depth] will be silently set to equal the online_delete setting if
online_delete is greater than fetch_depth.
- In the [node_db] section, there is a performance tuning option, delete_batch,
which sets the maximum number of SQL rows deleted per batch.

46
src/xrpld/app/rdb/README.md
View File

@@ -2,8 +2,8 @@
The guiding principles of the Relational Database Interface are summarized below:
* All hard-coded SQL statements should be stored in the [files](#source-files) under the `xrpld/app/rdb` directory. With the exception of test modules, no hard-coded SQL should be added to any other file in rippled.
* The base class `RelationalDatabase` is inherited by derived classes that each provide an interface for operating on distinct relational database systems.
- All hard-coded SQL statements should be stored in the [files](#source-files) under the `xrpld/app/rdb` directory. With the exception of test modules, no hard-coded SQL should be added to any other file in rippled.
- The base class `RelationalDatabase` is inherited by derived classes that each provide an interface for operating on distinct relational database systems.
## Overview
@@ -45,36 +45,34 @@ src/xrpld/app/rdb/
```
### File Contents
| File | Contents |
| ----------- | ----------- |
| `Node.[h\|cpp]` | Defines/Implements methods used by `SQLiteDatabase` for interacting with SQLite node databases|
|`SQLiteDatabase.[h\|cpp]`| Defines/Implements the class `SQLiteDatabase`/`SQLiteDatabaseImp` which inherits from `RelationalDatabase` and is used to operate on the main stores |
| `PeerFinder.[h\|cpp]` | Defines/Implements methods for interacting with the PeerFinder SQLite database |
|`RelationalDatabase.cpp`| Implements the static method `RelationalDatabase::init` which is used to initialize an instance of `RelationalDatabase` |
| `RelationalDatabase.h` | Defines the abstract class `RelationalDatabase`, the primary class of the Relational Database Interface |
| `State.[h\|cpp]` | Defines/Implements methods for interacting with the State SQLite database which concerns ledger deletion and database rotation |
| `Vacuum.[h\|cpp]` | Defines/Implements a method for performing the `VACUUM` operation on SQLite databases |
| `Wallet.[h\|cpp]` | Defines/Implements methods for interacting with Wallet SQLite databases |
| File | Contents |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| `Node.[h\|cpp]` | Defines/Implements methods used by `SQLiteDatabase` for interacting with SQLite node databases |
| `SQLiteDatabase.[h\|cpp]` | Defines/Implements the class `SQLiteDatabase`/`SQLiteDatabaseImp` which inherits from `RelationalDatabase` and is used to operate on the main stores |
| `PeerFinder.[h\|cpp]` | Defines/Implements methods for interacting with the PeerFinder SQLite database |
| `RelationalDatabase.cpp` | Implements the static method `RelationalDatabase::init` which is used to initialize an instance of `RelationalDatabase` |
| `RelationalDatabase.h` | Defines the abstract class `RelationalDatabase`, the primary class of the Relational Database Interface |
| `State.[h\|cpp]` | Defines/Implements methods for interacting with the State SQLite database which concerns ledger deletion and database rotation |
| `Vacuum.[h\|cpp]` | Defines/Implements a method for performing the `VACUUM` operation on SQLite databases |
| `Wallet.[h\|cpp]` | Defines/Implements methods for interacting with Wallet SQLite databases |
## Classes
The abstract class `RelationalDatabase` is the primary class of the Relational Database Interface and is defined in the eponymous header file. This class provides a static method `init()` which, when invoked, creates a concrete instance of a derived class whose type is specified by the system configuration. All other methods in the class are virtual. Presently there exist two classes that derive from `RelationalDatabase`, namely `SQLiteDatabase` and `PostgresDatabase`.
The abstract class `RelationalDatabase` is the primary class of the Relational Database Interface and is defined in the eponymous header file. This class provides a static method `init()` which, when invoked, creates a concrete instance of a derived class whose type is specified by the system configuration. All other methods in the class are virtual. Presently there exist two classes that derive from `RelationalDatabase`, namely `SQLiteDatabase` and `PostgresDatabase`.
## Database Methods
The Relational Database Interface provides three categories of methods for interacting with databases:
* Free functions for interacting with SQLite databases used by various components of the software. These methods feature a `soci::session` parameter which facilitates connecting to SQLite databases, and are defined and implemented in the following files:
- Free functions for interacting with SQLite databases used by various components of the software. These methods feature a `soci::session` parameter which facilitates connecting to SQLite databases, and are defined and implemented in the following files:
* `PeerFinder.[h\|cpp]`
* `State.[h\|cpp]`
* `Vacuum.[h\|cpp]`
* `Wallet.[h\|cpp]`
- `PeerFinder.[h\|cpp]`
- `State.[h\|cpp]`
- `Vacuum.[h\|cpp]`
- `Wallet.[h\|cpp]`
- Free functions used exclusively by `SQLiteDatabaseImp` for interacting with SQLite databases owned by the node store. Unlike the free functions in the files listed above, these are not intended to be invoked directly by clients. Rather, these methods are invoked by derived instances of `RelationalDatabase`. These methods are defined in the following files:
- `Node.[h|cpp]`
* Free functions used exclusively by `SQLiteDatabaseImp` for interacting with SQLite databases owned by the node store. Unlike the free functions in the files listed above, these are not intended to be invoked directly by clients. Rather, these methods are invoked by derived instances of `RelationalDatabase`. These methods are defined in the following files:
* `Node.[h|cpp]`
* Member functions of `RelationalDatabase`, `SQLiteDatabase`, and `PostgresDatabase` which are used to access the node store.
- Member functions of `RelationalDatabase`, `SQLiteDatabase`, and `PostgresDatabase` which are used to access the node store.

5
src/xrpld/consensus/README.md
View File

@@ -1,9 +1,8 @@
# Consensus
This directory contains the implementation of a
generic consensus algorithm. The implementation
generic consensus algorithm. The implementation
follows a CRTP design, requiring client code to implement
specific functions and types to use consensus in their
application. The interface is undergoing refactoring and
application. The interface is undergoing refactoring and
is not yet finalized.

154
src/xrpld/nodestore/README.md
View File

@@ -1,6 +1,7 @@
# Database Documentation
* [NodeStore](#nodestore)
* [Benchmarks](#benchmarks)
- [NodeStore](#nodestore)
- [Benchmarks](#benchmarks)
# NodeStore
@@ -12,41 +13,43 @@ identified by the hash, which is a 256 bit hash of the blob. The blob is a
variable length block of serialized data. The type identifies what the blob
contains. The fields are as follows:
* `mType`
- `mType`
An enumeration that determines what the blob holds. There are four
different types of objects stored.
An enumeration that determines what the blob holds. There are four
different types of objects stored.
* **ledger**
- **ledger**
A ledger header.
A ledger header.
* **transaction**
- **transaction**
A signed transaction.
A signed transaction.
* **account node**
- **account node**
A node in a ledger's account state tree.
A node in a ledger's account state tree.
* **transaction node**
- **transaction node**
A node in a ledger's transaction tree.
A node in a ledger's transaction tree.
* `mHash`
- `mHash`
A 256-bit hash of the blob.
A 256-bit hash of the blob.
* `mData`
- `mData`
A blob containing the payload. Stored in the following format.
A blob containing the payload. Stored in the following format.
| Byte | | |
| :------ | :----- | :------------------------- |
| 0...7 | unused | |
| 8 | type | NodeObjectType enumeration |
| 9...end | data | body of the object data |
|Byte | | |
|:------|:--------------------|:-------------------------|
|0...7 |unused | |
|8 |type |NodeObjectType enumeration|
|9...end|data |body of the object data |
---
The `NodeStore` provides an interface that stores, in a persistent database, a
collection of NodeObjects that rippled uses as its primary representation of
ledger entries. All ledger entries are stored as NodeObjects and as such, need
@@ -64,41 +67,42 @@ the configuration file [node_db] section as follows.
One or more lines of key / value pairs
Example:
```
type=RocksDB
path=rocksdb
compression=1
```
Choices for 'type' (not case-sensitive)
* **HyperLevelDB**
- **HyperLevelDB**
An improved version of LevelDB (preferred).
An improved version of LevelDB (preferred).
* **LevelDB**
- **LevelDB**
Google's LevelDB database (deprecated).
Google's LevelDB database (deprecated).
* **none**
- **none**
Use no backend.
Use no backend.
* **RocksDB**
- **RocksDB**
Facebook's RocksDB database, builds on LevelDB.
Facebook's RocksDB database, builds on LevelDB.
* **SQLite**
- **SQLite**
Use SQLite.
Use SQLite.
'path' speficies where the backend will store its data files.
Choices for 'compression'
* **0** off
* **1** on (default)
- **0** off
- **1** on (default)
# Benchmarks
@@ -129,48 +133,48 @@ RocksDBQuickFactory is intended to provide a testbed for comparing potential
rocksdb performance with the existing recommended configuration in rippled.cfg.
Through various executions and profiling some conclusions are presented below.
* If the write ahead log is enabled, insert speed soon clogs up under load. The
BatchWriter class intends to stop this from blocking the main threads by queuing
up writes and running them in a separate thread. However, rocksdb already has
separate threads dedicated to flushing the memtable to disk and the memtable is
itself an in-memory queue. The result is two queues with a guarantee of
durability in between. However if the memtable was used as the sole queue and
the rocksdb::Flush() call was manually triggered at opportune moments, possibly
just after ledger close, then that would provide similar, but more predictable
guarantees. It would also remove an unneeded thread and unnecessary memory
usage. An alternative point of view is that because there will always be many
other rippled instances running there is no need for such guarantees. The nodes
will always be available from another peer.
- If the write ahead log is enabled, insert speed soon clogs up under load. The
BatchWriter class intends to stop this from blocking the main threads by queuing
up writes and running them in a separate thread. However, rocksdb already has
separate threads dedicated to flushing the memtable to disk and the memtable is
itself an in-memory queue. The result is two queues with a guarantee of
durability in between. However if the memtable was used as the sole queue and
the rocksdb::Flush() call was manually triggered at opportune moments, possibly
just after ledger close, then that would provide similar, but more predictable
guarantees. It would also remove an unneeded thread and unnecessary memory
usage. An alternative point of view is that because there will always be many
other rippled instances running there is no need for such guarantees. The nodes
will always be available from another peer.
* Lookup in a block was previously using binary search. With rippled's use case
it is highly unlikely that two adjacent key/values will ever be requested one
after the other. Therefore hash indexing of blocks makes much more sense.
Rocksdb has a number of options for hash indexing both memtables and blocks and
these need more testing to find the best choice.
- Lookup in a block was previously using binary search. With rippled's use case
it is highly unlikely that two adjacent key/values will ever be requested one
after the other. Therefore hash indexing of blocks makes much more sense.
Rocksdb has a number of options for hash indexing both memtables and blocks and
these need more testing to find the best choice.
* The current Database implementation has two forms of caching, so the LRU cache
of blocks at Factory level does not make any sense. However, if the hash
indexing and potentially the new [bloom
filter](http://rocksdb.org/blog/1427/new-bloom-filter-format/) can provide
faster lookup for non-existent keys, then potentially the caching could exist at
Factory level.
- The current Database implementation has two forms of caching, so the LRU cache
of blocks at Factory level does not make any sense. However, if the hash
indexing and potentially the new [bloom
filter](http://rocksdb.org/blog/1427/new-bloom-filter-format/) can provide
faster lookup for non-existent keys, then potentially the caching could exist at
Factory level.
* Multiple runs of the benchmarks can yield surprisingly different results. This
can perhaps be attributed to the asynchronous nature of rocksdb's compaction
process. The benchmarks are artifical and create highly unlikely write load to
create the dataset to measure different read access patterns. Therefore multiple
runs of the benchmarks are required to get a feel for the effectiveness of the
changes. This contrasts sharply with the keyvadb benchmarking were highly
repeatable timings were discovered. Also realistically sized datasets are
required to get a correct insight. The number of 2,000,000 key/values (actually
4,000,000 after the two insert benchmarks complete) is too low to get a full
picture.
- Multiple runs of the benchmarks can yield surprisingly different results. This
can perhaps be attributed to the asynchronous nature of rocksdb's compaction
process. The benchmarks are artifical and create highly unlikely write load to
create the dataset to measure different read access patterns. Therefore multiple
runs of the benchmarks are required to get a feel for the effectiveness of the
changes. This contrasts sharply with the keyvadb benchmarking were highly
repeatable timings were discovered. Also realistically sized datasets are
required to get a correct insight. The number of 2,000,000 key/values (actually
4,000,000 after the two insert benchmarks complete) is too low to get a full
picture.
* An interesting side effect of running the benchmarks in a profiler was that a
clear pattern of what RocksDB does under the hood was observable. This led to
the decision to trial hash indexing and also the discovery of the native CRC32
instruction not being used.
- An interesting side effect of running the benchmarks in a profiler was that a
clear pattern of what RocksDB does under the hood was observable. This led to
the decision to trial hash indexing and also the discovery of the native CRC32
instruction not being used.
* Important point to note that is if this factory is tested with an existing set
of sst files none of the old sst files will benefit from indexing changes until
they are compacted at a future point in time.
- Important point to note that is if this factory is tested with an existing set
of sst files none of the old sst files will benefit from indexing changes until
they are compacted at a future point in time.

146
src/xrpld/overlay/README.md
View File

@@ -39,10 +39,10 @@ The HTTP [request](https://www.w3.org/Protocols/rfc2616/rfc2616-sec5.html) must:
- Use HTTP version 1.1.
- Specify a request URI consisting of a single forward slash character ("/")
indicating the server root. Requests using different URIs are reserved for
future protocol implementations.
indicating the server root. Requests using different URIs are reserved for
future protocol implementations.
- Use the [_HTTP/1.1 Upgrade_][upgrade_header] mechanism with additional custom
fields to communicate protocol specific information related to the upgrade.
fields to communicate protocol specific information related to the upgrade.
HTTP requests which do not conform to this requirements must generate an
appropriate HTTP error and result in the connection being closed.
@@ -72,7 +72,6 @@ Previous-Ledger: q4aKbP7sd5wv+EXArwCmQiWZhq9AwBl2p/hCtpGJNsc=
##### Example HTTP Upgrade Response (Success)
```
HTTP/1.1 101 Switching Protocols
Connection: Upgrade
@@ -102,9 +101,9 @@ Content-Type: application/json
#### Standard Fields
| Field Name | Request | Response |
|--------------------- |:-----------------: |:-----------------: |
| `User-Agent` | :heavy_check_mark: | |
| Field Name | Request | Response |
| ------------ | :----------------: | :------: |
| `User-Agent` | :heavy_check_mark: | |
The `User-Agent` field indicates the version of the software that the
peer that is making the HTTP request is using. No semantic meaning is
@@ -113,9 +112,9 @@ specify the version of the software that is used.
See [RFC2616 &sect;14.43](https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.43).
| Field Name | Request | Response |
|--------------------- |:-----------------: |:-----------------: |
| `Server` | | :heavy_check_mark: |
| Field Name | Request | Response |
| ---------- | :-----: | :----------------: |
| `Server` | | :heavy_check_mark: |
The `Server` field indicates the version of the software that the
peer that is processing the HTTP request is using. No semantic meaning is
@@ -124,18 +123,18 @@ specify the version of the software that is used.
See [RFC2616 &sect;14.38](https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.38).
| Field Name | Request | Response |
|--------------------- |:-----------------: |:-----------------: |
| `Connection` | :heavy_check_mark: | :heavy_check_mark: |
| Field Name | Request | Response |
| ------------ | :----------------: | :----------------: |
| `Connection` | :heavy_check_mark: | :heavy_check_mark: |
The `Connection` field should have a value of `Upgrade` to indicate that a
request to upgrade the connection is being performed.
See [RFC2616 &sect;14.10](https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.10).
| Field Name | Request | Response |
|--------------------- |:-----------------: |:-----------------: |
| `Upgrade` | :heavy_check_mark: | :heavy_check_mark: |
| Field Name | Request | Response |
| ---------- | :----------------: | :----------------: |
| `Upgrade` | :heavy_check_mark: | :heavy_check_mark: |
The `Upgrade` field is part of the standard connection upgrade mechanism and
must be present in both requests and responses. It is used to negotiate the
@@ -156,12 +155,11 @@ equal to 2 and the minor is greater than or equal to 0.
See [RFC 2616 &sect;14.42](https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.42)
#### Custom Fields
| Field Name | Request | Response |
|--------------------- |:-----------------: |:-----------------: |
| `Connect-As` | :heavy_check_mark: | :heavy_check_mark: |
| Field Name | Request | Response |
| ------------ | :----------------: | :----------------: |
| `Connect-As` | :heavy_check_mark: | :heavy_check_mark: |
The mandatory `Connect-As` field is used to specify that type of connection
that is being requested.
@@ -175,10 +173,9 @@ elements specified in the request. If a server processing a request does not
recognize any of the connection types, the request should fail with an
appropriate HTTP error code (e.g. by sending an HTTP 400 "Bad Request" response).
| Field Name | Request | Response |
|--------------------- |:-----------------: |:-----------------: |
| `Remote-IP` | :white_check_mark: | :white_check_mark: |
| Field Name | Request | Response |
| ----------- | :----------------: | :----------------: |
| `Remote-IP` | :white_check_mark: | :white_check_mark: |
The optional `Remote-IP` field contains the string representation of the IP
address of the remote end of the connection as seen from the peer that is
@@ -187,10 +184,9 @@ sending the field.
By observing values of this field from a sufficient number of different
servers, a peer making outgoing connections can deduce its own IP address.
| Field Name | Request | Response |
|--------------------- |:-----------------: |:-----------------: |
| `Local-IP` | :white_check_mark: | :white_check_mark: |
| Field Name | Request | Response |
| ---------- | :----------------: | :----------------: |
| `Local-IP` | :white_check_mark: | :white_check_mark: |
The optional `Local-IP` field contains the string representation of the IP
address that the peer sending the field believes to be its own.
@@ -198,10 +194,9 @@ address that the peer sending the field believes to be its own.
Servers receiving this field can detect IP address mismatches, which may
indicate a potential man-in-the-middle attack.
| Field Name | Request | Response |
|--------------------- |:-----------------: |:-----------------: |
| `Network-ID` | :white_check_mark: | :white_check_mark: |
| Field Name | Request | Response |
| ------------ | :----------------: | :----------------: |
| `Network-ID` | :white_check_mark: | :white_check_mark: |
The optional `Network-ID` can be used to identify to which of several
[parallel networks](https://xrpl.org/parallel-networks.html) the server
@@ -217,10 +212,9 @@ If a server configured to join one network receives a connection request from a
server configured to join another network, the request should fail with an
appropriate HTTP error code (e.g. by sending an HTTP 400 "Bad Request" response).
| Field Name | Request | Response |
|--------------------- |:-----------------: |:-----------------: |
| `Network-Time` | :white_check_mark: | :white_check_mark: |
| Field Name | Request | Response |
| -------------- | :----------------: | :----------------: |
| `Network-Time` | :white_check_mark: | :white_check_mark: |
The optional `Network-Time` field reports the current [time](https://xrpl.org/basic-data-types.html#specifying-time)
according to sender's internal clock.
@@ -232,20 +226,18 @@ each other with an appropriate HTTP error code (e.g. by sending an HTTP 400
It is highly recommended that servers synchronize their clocks using time
synchronization software. For more on this topic, please visit [ntp.org](http://www.ntp.org/).
| Field Name | Request | Response |
|--------------------- |:-----------------: |:-----------------: |
| `Public-Key` | :heavy_check_mark: | :heavy_check_mark: |
| Field Name | Request | Response |
| ------------ | :----------------: | :----------------: |
| `Public-Key` | :heavy_check_mark: | :heavy_check_mark: |
The mandatory `Public-Key` field identifies the sending server's public key,
encoded in base58 using the standard encoding for node public keys.
See: https://xrpl.org/base58-encodings.html
| Field Name | Request | Response |
|--------------------- |:-----------------: |:-----------------: |
| `Server-Domain` | :white_check_mark: | :white_check_mark: |
| Field Name | Request | Response |
| --------------- | :----------------: | :----------------: |
| `Server-Domain` | :white_check_mark: | :white_check_mark: |
The optional `Server-Domain` field allows a server to report the domain that
it is operating under. The value is configured by the server administrator in
@@ -259,10 +251,9 @@ under the specified domain and locating the public key of this server under the
Sending a malformed domain will prevent a connection from being established.
| Field Name | Request | Response |
|--------------------- |:-----------------: |:-----------------: |
| `Session-Signature` | :heavy_check_mark: | :heavy_check_mark: |
| Field Name | Request | Response |
| ------------------- | :----------------: | :----------------: |
| `Session-Signature` | :heavy_check_mark: | :heavy_check_mark: |
The `Session-Signature` field is mandatory and is used to secure the peer link
against certain types of attack. For more details see "Session Signature" below.
@@ -272,36 +263,35 @@ should support both **Base64** and **HEX** encoding for this value.
For more details on this field, please see **Session Signature** below.
| Field Name | Request | Response |
|--------------------- |:-----------------: |:-----------------: |
| `Crawl` | :white_check_mark: | :white_check_mark: |
| Field Name | Request | Response |
| ---------- | :----------------: | :----------------: |
| `Crawl` | :white_check_mark: | :white_check_mark: |
The optional `Crawl` field can be used by a server to indicate whether peers
should include it in crawl reports.
The field can take two values:
- **`Public`**: The server's IP address and port should be included in crawl
reports.
reports.
- **`Private`**: The server's IP address and port should not be included in
crawl reports. _This is the default, if the field is omitted._
crawl reports. _This is the default, if the field is omitted._
For more on the Peer Crawler, please visit https://xrpl.org/peer-crawler.html.
| Field Name | Request | Response |
|--------------------- |:-----------------: |:-----------------: |
| `Closed-Ledger` | :white_check_mark: | :white_check_mark: |
| Field Name | Request | Response |
| --------------- | :----------------: | :----------------: |
| `Closed-Ledger` | :white_check_mark: | :white_check_mark: |
If present, identifies the hash of the last ledger that the sending server
considers to be closed.
The value is encoded as **HEX**, but implementations should support both
**Base64** and **HEX** encoding for this value for legacy purposes.
| Field Name | Request | Response |
|--------------------- |:-----------------: |:-----------------: |
| `Previous-Ledger` | :white_check_mark: | :white_check_mark: |
| Field Name | Request | Response |
| ----------------- | :----------------: | :----------------: |
| `Previous-Ledger` | :white_check_mark: | :white_check_mark: |
If present, identifies the hash of the parent ledger that the sending server
considers to be closed.
@@ -317,7 +307,6 @@ and values in both requests and responses.
Implementations should not reject requests because of the presence of fields
that they do not understand.
### Session Signature
Even for SSL/TLS encrypted connections, it is possible for an attacker to mount
@@ -365,8 +354,7 @@ transferred between A and B and will not be able to intelligently tamper with th
message stream between Alice and Bob, although she may be still be able to inject
delays or terminate the link.
# Ripple Clustering #
# Ripple Clustering
A cluster consists of more than one Ripple server under common
administration that share load information, distribute cryptography
@@ -374,19 +362,19 @@ operations, and provide greater response consistency.
Cluster nodes are identified by their public node keys. Cluster nodes
exchange information about endpoints that are imposing load upon them.
Cluster nodes share information about their internal load status. Cluster
Cluster nodes share information about their internal load status. Cluster
nodes do not have to verify the cryptographic signatures on messages
received from other cluster nodes.
## Configuration ##
## Configuration
A server's public key can be determined from the output of the `server_info`
command. The key is in the `pubkey_node` value, and is a text string
beginning with the letter `n`. The key is maintained across runs in a
command. The key is in the `pubkey_node` value, and is a text string
beginning with the letter `n`. The key is maintained across runs in a
database.
Cluster members are configured in the `rippled.cfg` file under
`[cluster_nodes]`. Each member should be configured on a line beginning
`[cluster_nodes]`. Each member should be configured on a line beginning
with the node public key, followed optionally by a space and a friendly
name.
@@ -404,23 +392,23 @@ New spokes can be added as follows:
- Restart each hub, one by one
- Restart the spoke
## Transaction Behavior ##
## Transaction Behavior
When a transaction is received from a cluster member, several normal checks
are bypassed:
Signature checking is bypassed because we trust that a cluster member would
not relay a transaction with an incorrect signature. Validators may wish to
not relay a transaction with an incorrect signature. Validators may wish to
disable this feature, preferring the additional load to get the additional
security of having validators check each transaction.
Local checks for transaction checking are also bypassed. For example, a
server will not reject a transaction from a cluster peer because the fee
does not meet its current relay fee. It is preferable to keep the cluster
does not meet its current relay fee. It is preferable to keep the cluster
in agreement and permit confirmation from one cluster member to more
reliably indicate the transaction's acceptance by the cluster.
## Server Load Information ##
## Server Load Information
Cluster members exchange information on their server's load level. The load
level is essentially the amount by which the normal fee levels are multiplied
@@ -431,22 +419,22 @@ fee, is the highest of its local load level, the network load level, and the
cluster load level. The cluster load level is the median load level reported
by a cluster member.
## Gossip ##
## Gossip
Gossip is the mechanism by which cluster members share information about
endpoints (typically IPv4 addresses) that are imposing unusually high load
on them. The endpoint load manager takes into account gossip to reduce the
on them. The endpoint load manager takes into account gossip to reduce the
amount of load the endpoint is permitted to impose on the local server
before it is warned, disconnected, or banned.
Suppose, for example, that an attacker controls a large number of IP
addresses, and with these, he can send sufficient requests to overload a
server. Without gossip, he could use these same addresses to overload all
the servers in a cluster. With gossip, if he chooses to use the same IP
server. Without gossip, he could use these same addresses to overload all
the servers in a cluster. With gossip, if he chooses to use the same IP
address to impose load on more than one server, he will find that the amount
of load he can impose before getting disconnected is much lower.
## Monitoring ##
## Monitoring
The `peers` command will report on the status of the cluster. The `cluster`
object will contain one entry for each member of the cluster (either configured

76
src/xrpld/peerfinder/README.md
View File

@@ -1,4 +1,3 @@
# PeerFinder
## Introduction
@@ -31,23 +30,23 @@ slots_.
PeerFinder has these responsibilities
* Maintain a persistent set of endpoint addresses suitable for bootstrapping
- Maintain a persistent set of endpoint addresses suitable for bootstrapping
into the peer to peer overlay, ranked by relative locally observed utility.
* Send and receive protocol messages for discovery of endpoint addresses.
- Send and receive protocol messages for discovery of endpoint addresses.
* Provide endpoint addresses to new peers that need them.
- Provide endpoint addresses to new peers that need them.
* Maintain connections to a configured set of fixed peers.
- Maintain connections to a configured set of fixed peers.
* Impose limits on the various slots consumed by peer connections.
- Impose limits on the various slots consumed by peer connections.
* Initiate outgoing connection attempts to endpoint addresses to maintain the
- Initiate outgoing connection attempts to endpoint addresses to maintain the
overlay connectivity and fixed peer policies.
* Verify the connectivity of neighbors who advertise inbound connection slots.
- Verify the connectivity of neighbors who advertise inbound connection slots.
* Prevent duplicate connections and connections to self.
- Prevent duplicate connections and connections to self.
---
@@ -79,28 +78,28 @@ The `Config` structure defines the operational parameters of the PeerFinder.
Some values come from the configuration file while others are calculated via
tuned heuristics. The fields are as follows:
* `autoConnect`
- `autoConnect`
A flag indicating whether or not the Autoconnect feature is enabled.
* `wantIncoming`
- `wantIncoming`
A flag indicating whether or not the peer desires inbound connections. When
this flag is turned off, a peer will not advertise itself in Endpoint
messages.
* `listeningPort`
- `listeningPort`
The port number to use when creating the listening socket for peer
connections.
* `maxPeers`
- `maxPeers`
The largest number of active peer connections to allow. This includes inbound
and outbound connections, but excludes fixed and cluster peers. There is an
implementation defined floor on this value.
* `outPeers`
- `outPeers`
The number of automatic outbound connections that PeerFinder will maintain
when the Autoconnect feature is enabled. The value is computed with fractional
@@ -161,8 +160,8 @@ Endpoint messages are received from the overlay over time.
The `Bootcache` stores IP addresses useful for gaining initial connections.
Each address is associated with the following metadata:
* **Valence**
- **Valence**
A signed integer which represents the number of successful
consecutive connection attempts when positive, and the number of
@@ -202,30 +201,30 @@ a slot. Slots have properties and state associated with them:
The slot state represents the current stage of the connection as it passes
through the business logic for establishing peer connections.
* `accept`
- `accept`
The accept state is an initial state resulting from accepting an incoming
connection request on a listening socket. The remote IP address and port
are known, and a handshake is expected next.
* `connect`
- `connect`
The connect state is an initial state used when actively establishing outbound
connection attempts. The desired remote IP address and port are known.
* `connected`
- `connected`
When an outbound connection attempt succeeds, it moves to the connected state.
The handshake is initiated but not completed.
* `active`
- `active`
The state becomes Active when a connection in either the Accepted or Connected
state completes the handshake process, and a slot is available based on the
properties. If no slot is available when the handshake completes, the socket
is gracefully closed.
* `closing`
- `closing`
The Closing state represents a connected socket in the process of being
gracefully closed.
@@ -234,13 +233,13 @@ through the business logic for establishing peer connections.
Slot properties may be combined and are not mutually exclusive.
* **Inbound**
- **Inbound**
An inbound slot is the condition of a socket which has accepted an incoming
connection request. A connection which is not inbound is by definition
outbound.
* **Fixed**
- **Fixed**
A fixed slot is a desired connection to a known peer identified by IP address,
usually entered manually in the configuration file. For the purpose of
@@ -248,14 +247,14 @@ Slot properties may be combined and are not mutually exclusive.
although only the IP address is checked to determine if the fixed peer is
already connected. Fixed slots do not count towards connection limits.
* **Cluster**
- **Cluster**
A cluster slot is a connection which has completed the handshake stage, whose
public key matches a known public key usually entered manually in the
configuration file or learned through overlay messages from other trusted
peers. Cluster slots do not count towards connection limits.
* **Superpeer** (forthcoming)
- **Superpeer** (forthcoming)
A superpeer slot is a connection to a peer which can accept incoming
connections, meets certain resource availaibility requirements (such as
@@ -279,7 +278,7 @@ Cluster slots are identified by the public key and set up during the
initialization of the manager or discovered upon receipt of messages in the
overlay from trusted connections.
--------------------------------------------------------------------------------
---
# Algorithms
@@ -295,8 +294,8 @@ This stage is invoked when the number of active fixed connections is below the
number of fixed connections specified in the configuration, and one of the
following is true:
* There are eligible fixed addresses to try
* Any outbound connection attempts are in progress
- There are eligible fixed addresses to try
- Any outbound connection attempts are in progress
Each fixed address is associated with a retry timer. On a fixed connection
failure, the timer is reset so that the address is not tried for some amount
@@ -317,8 +316,8 @@ The Livecache is invoked when Stage 1 is not active, autoconnect is enabled,
and the number of active outbound connections is below the number desired. The
stage remains active while:
* The Livecache has addresses to try
* Any outbound connection attempts are in progress
- The Livecache has addresses to try
- Any outbound connection attempts are in progress
PeerFinder makes its best effort to exhaust addresses in the Livecache before
moving on to the Bootcache, because Livecache addresses are highly likely
@@ -333,7 +332,7 @@ The Bootcache is invoked when Stage 1 and Stage 2 are not active, autoconnect
is enabled, and the number of active outbound connections is below the number
desired. The stage remains active while:
* There are addresses in the cache that have not been tried recently.
- There are addresses in the cache that have not been tried recently.
Entries in the Bootcache are ranked, with highly connectible addresses preferred
over others. Connection attempts to Bootcache addresses are very likely to
@@ -342,7 +341,7 @@ not have open slots. Before the remote peer closes the connection it will send
a handful of addresses from its Livecache to help the new peer coming online
obtain connections.
--------------------------------------------------------------------------------
---
# References
@@ -352,10 +351,11 @@ Much of the work in PeerFinder was inspired by earlier work in Gnutella:
_By Christopher Rohrs and Vincent Falco_
[Gnutella 0.6 Protocol:](http://rfc-gnutella.sourceforge.net/src/rfc-0_6-draft.html) Sections:
* 2.2.2 Ping (0x00)
* 2.2.3 Pong (0x01)
* 2.2.4 Use of Ping and Pong messages
* 2.2.4.1 A simple pong caching scheme
* 2.2.4.2 Other pong caching schemes
- 2.2.2 Ping (0x00)
- 2.2.3 Pong (0x01)
- 2.2.4 Use of Ping and Pong messages
- 2.2.4.1 A simple pong caching scheme
- 2.2.4.2 Other pong caching schemes
[overlay_network]: http://en.wikipedia.org/wiki/Overlay_network

17
src/xrpld/rpc/README.md
View File

@@ -2,15 +2,16 @@
## Introduction.
By default, an RPC handler runs as an uninterrupted task on the JobQueue. This
By default, an RPC handler runs as an uninterrupted task on the JobQueue. This
is fine for commands that are fast to compute but might not be acceptable for
tasks that require multiple parts or are large, like a full ledger.
For this purpose, the rippled RPC handler allows *suspension with continuation*
For this purpose, the rippled RPC handler allows _suspension with continuation_
- a request to suspend execution of the RPC response and to continue it after
some function or job has been executed. A default continuation is supplied
which simply reschedules the job on the JobQueue, or the programmer can supply
their own.
some function or job has been executed. A default continuation is supplied
which simply reschedules the job on the JobQueue, or the programmer can supply
their own.
## The classes.
@@ -28,16 +29,16 @@ would prevent any other task from making forward progress when you call a
`Callback`.
A `Continuation` is a function that is given a `Callback` and promises to call
it later. A `Continuation` guarantees to call the `Callback` exactly once at
it later. A `Continuation` guarantees to call the `Callback` exactly once at
some point in the future, but it does not have to be immediately or even in the
current thread.
A `Suspend` is a function belonging to a `Coroutine`. A `Suspend` runs a
A `Suspend` is a function belonging to a `Coroutine`. A `Suspend` runs a
`Continuation`, passing it a `Callback` that continues execution of the
`Coroutine`.
And finally, a `Coroutine` is a `std::function` which is given a
`Suspend`. This is what the RPC handler gives to the coroutine manager,
`Suspend`. This is what the RPC handler gives to the coroutine manager,
expecting to get called back with a `Suspend` and to be able to start execution.
## The flow of control.

173
src/xrpld/shamap/README.md
View File

@@ -1,4 +1,4 @@
# SHAMap Introduction #
# SHAMap Introduction
March 2020
@@ -30,20 +30,20 @@ The root node is always a SHAMapInnerNode.
A given `SHAMap` always stores only one of three kinds of data:
* Transactions with metadata
* Transactions without metadata, or
* Account states.
- Transactions with metadata
- Transactions without metadata, or
- Account states.
So all of the leaf nodes of a particular `SHAMap` will always have a uniform type.
The inner nodes carry no data other than the hash of the nodes beneath them.
All nodes are owned by shared_ptrs resident in either other nodes, or in case of
the root node, a shared_ptr in the `SHAMap` itself. The use of shared_ptrs
permits more than one `SHAMap` at a time to share ownership of a node. This
the root node, a shared_ptr in the `SHAMap` itself. The use of shared_ptrs
permits more than one `SHAMap` at a time to share ownership of a node. This
occurs (for example), when a copy of a `SHAMap` is made.
Copies are made with the `snapShot` function as opposed to the `SHAMap` copy
constructor. See the section on `SHAMap` creation for more details about
constructor. See the section on `SHAMap` creation for more details about
`snapShot`.
Sequence numbers are used to further customize the node ownership strategy. See
@@ -51,62 +51,62 @@ the section on sequence numbers for details on sequence numbers.
![node diagram](https://user-images.githubusercontent.com/46455409/77350005-1ef12c80-6cf9-11ea-9c8d-56410f442859.png)
## Mutability ##
## Mutability
There are two different ways of building and using a `SHAMap`:
1. A mutable `SHAMap` and
2. An immutable `SHAMap`
1. A mutable `SHAMap` and
2. An immutable `SHAMap`
The distinction here is not of the classic C++ immutable-means-unchanging sense.
An immutable `SHAMap` contains *nodes* that are immutable. Also, once a node has
An immutable `SHAMap` contains _nodes_ that are immutable. Also, once a node has
been located in an immutable `SHAMap`, that node is guaranteed to persist in that
`SHAMap` for the lifetime of the `SHAMap`.
So, somewhat counter-intuitively, an immutable `SHAMap` may grow as new nodes are
introduced. But an immutable `SHAMap` will never get smaller (until it entirely
evaporates when it is destroyed). Nodes, once introduced to the immutable
`SHAMap`, also never change their location in memory. So nodes in an immutable
introduced. But an immutable `SHAMap` will never get smaller (until it entirely
evaporates when it is destroyed). Nodes, once introduced to the immutable
`SHAMap`, also never change their location in memory. So nodes in an immutable
`SHAMap` can be handled using raw pointers (if you're careful).
One consequence of this design is that an immutable `SHAMap` can never be
"trimmed". There is no way to identify unnecessary nodes in an immutable `SHAMap`
that could be removed. Once a node has been brought into the in-memory `SHAMap`,
"trimmed". There is no way to identify unnecessary nodes in an immutable `SHAMap`
that could be removed. Once a node has been brought into the in-memory `SHAMap`,
that node stays in memory for the life of the `SHAMap`.
Most `SHAMap`s are immutable, in the sense that they don't modify or remove their
contained nodes.
An example where a mutable `SHAMap` is required is when we want to apply
transactions to the last closed ledger. To do so we'd make a mutable snapshot
transactions to the last closed ledger. To do so we'd make a mutable snapshot
of the state trie and then start applying transactions to it. Because the
snapshot is mutable, changes to nodes in the snapshot will not affect nodes in
other `SHAMap`s.
An example using a immutable ledger would be when there's an open ledger and
some piece of code wishes to query the state of the ledger. In this case we
some piece of code wishes to query the state of the ledger. In this case we
don't wish to change the state of the `SHAMap`, so we'd use an immutable snapshot.
## Sequence numbers ##
## Sequence numbers
Both `SHAMap`s and their nodes carry a sequence number. This is simply an
Both `SHAMap`s and their nodes carry a sequence number. This is simply an
unsigned number that indicates ownership or membership, or a non-membership.
`SHAMap`s sequence numbers normally start out as 1. However when a snap-shot of
`SHAMap`s sequence numbers normally start out as 1. However when a snap-shot of
a `SHAMap` is made, the copy's sequence number is 1 greater than the original.
The nodes of a `SHAMap` have their own copy of a sequence number. If the `SHAMap`
The nodes of a `SHAMap` have their own copy of a sequence number. If the `SHAMap`
is mutable, meaning it can change, then all of its nodes must have the
same sequence number as the `SHAMap` itself. This enforces an invariant that none
same sequence number as the `SHAMap` itself. This enforces an invariant that none
of the nodes are shared with other `SHAMap`s.
When a `SHAMap` needs to have a private copy of a node, not shared by any other
`SHAMap`, it first clones it and then sets the new copy to have a sequence number
equal to the `SHAMap` sequence number. The `unshareNode` is a private utility
equal to the `SHAMap` sequence number. The `unshareNode` is a private utility
which automates the task of first checking if the node is already sharable, and
if so, cloning it and giving it the proper sequence number. An example case
if so, cloning it and giving it the proper sequence number. An example case
where a private copy is needed is when an inner node needs to have a child
pointer altered. Any modification to a node will require a non-shared node.
pointer altered. Any modification to a node will require a non-shared node.
When a `SHAMap` decides that it is safe to share a node of its own, it sets the
node's sequence number to 0 (a `SHAMap` never has a sequence number of 0). This
@@ -116,40 +116,40 @@ Note that other objects in rippled also have sequence numbers (e.g. ledgers).
The `SHAMap` and node sequence numbers should not be confused with these other
sequence numbers (no relation).
## SHAMap Creation ##
## SHAMap Creation
A `SHAMap` is usually not created from vacuum. Once an initial `SHAMap` is
A `SHAMap` is usually not created from vacuum. Once an initial `SHAMap` is
constructed, later `SHAMap`s are usually created by calling snapShot(bool
isMutable) on the original `SHAMap`. The returned `SHAMap` has the expected
isMutable) on the original `SHAMap`. The returned `SHAMap` has the expected
characteristics (mutable or immutable) based on the passed in flag.
It is cheaper to make an immutable snapshot of a `SHAMap` than to make a mutable
snapshot. If the `SHAMap` snapshot is mutable then sharable nodes must be
snapshot. If the `SHAMap` snapshot is mutable then sharable nodes must be
copied before they are placed in the mutable map.
A new `SHAMap` is created with each new ledger round. Transactions not executed
A new `SHAMap` is created with each new ledger round. Transactions not executed
in the previous ledger populate the `SHAMap` for the new ledger.
## Storing SHAMap data in the database ##
## Storing SHAMap data in the database
When consensus is reached, the ledger is closed. As part of this process, the
When consensus is reached, the ledger is closed. As part of this process, the
`SHAMap` is stored to the database by calling `SHAMap::flushDirty`.
Both `unshare()` and `flushDirty` walk the `SHAMap` by calling
`SHAMap::walkSubTree`. As `unshare()` walks the trie, nodes are not written to
`SHAMap::walkSubTree`. As `unshare()` walks the trie, nodes are not written to
the database, and as `flushDirty` walks the trie nodes are written to the
database. `walkSubTree` visits every node in the trie. This process must ensure
that each node is only owned by this trie, and so "unshares" as it walks each
node (from the root down). This is done in the `preFlushNode` function by
ensuring that the node has a sequence number equal to that of the `SHAMap`. If
node (from the root down). This is done in the `preFlushNode` function by
ensuring that the node has a sequence number equal to that of the `SHAMap`. If
the node doesn't, it is cloned.
For each inner node encountered (starting with the root node), each of the
children are inspected (from 1 to 16). For each child, if it has a non-zero
sequence number (unshareable), the child is first copied. Then if the child is
an inner node, we recurse down to that node's children. Otherwise we've found a
leaf node and that node is written to the database. A count of each leaf node
that is visited is kept. The hash of the data in the leaf node is computed at
children are inspected (from 1 to 16). For each child, if it has a non-zero
sequence number (unshareable), the child is first copied. Then if the child is
an inner node, we recurse down to that node's children. Otherwise we've found a
leaf node and that node is written to the database. A count of each leaf node
that is visited is kept. The hash of the data in the leaf node is computed at
this time, and the child is reassigned back into the parent inner node just in
case the COW operation created a new pointer to this leaf node.
@@ -157,22 +157,22 @@ After processing each node, the node is then marked as sharable again by setting
its sequence number to 0.
After all of an inner node's children are processed, then its hash is updated
and the inner node is written to the database. Then this inner node is assigned
and the inner node is written to the database. Then this inner node is assigned
back into it's parent node, again in case the COW operation created a new
pointer to it.
## Walking a SHAMap ##
## Walking a SHAMap
The private function `SHAMap::walkTowardsKey` is a good example of *how* to walk
The private function `SHAMap::walkTowardsKey` is a good example of _how_ to walk
a `SHAMap`, and the various functions that call `walkTowardsKey` are good examples
of *why* one would want to walk a `SHAMap` (e.g. `SHAMap::findKey`).
of _why_ one would want to walk a `SHAMap` (e.g. `SHAMap::findKey`).
`walkTowardsKey` always starts at the root of the `SHAMap` and traverses down
through the inner nodes, looking for a leaf node along a path in the trie
designated by a `uint256`.
As one walks the trie, one can *optionally* keep a stack of nodes that one has
passed through. This isn't necessary for walking the trie, but many clients
will use the stack after finding the desired node. For example if one is
As one walks the trie, one can _optionally_ keep a stack of nodes that one has
passed through. This isn't necessary for walking the trie, but many clients
will use the stack after finding the desired node. For example if one is
deleting a node from the trie, the stack is handy for repairing invariants in
the trie after the deletion.
@@ -189,10 +189,10 @@ how we use a `SHAMapNodeID` to select a "branch" (child) by indexing into a
path at a given depth.
While the current node is an inner node, traversing down the trie from the root
continues, unless the path indicates a child that does not exist. And in this
continues, unless the path indicates a child that does not exist. And in this
case, `nullptr` is returned to indicate no leaf node along the given path
exists. Otherwise a leaf node is found and a (non-owning) pointer to it is
returned. At each step, if a stack is requested, a
exists. Otherwise a leaf node is found and a (non-owning) pointer to it is
returned. At each step, if a stack is requested, a
`pair<shared_ptr<SHAMapTreeNode>, SHAMapNodeID>` is pushed onto the stack.
When a child node is found by `selectBranch`, the traversal to that node
@@ -210,35 +210,35 @@ The first step consists of several attempts to find the node in various places:
If the node is not found in the trie, then it is installed into the trie as part
of the traversal process.
## Late-arriving Nodes ##
## Late-arriving Nodes
As we noted earlier, `SHAMap`s (even immutable ones) may grow. If a `SHAMap` is
As we noted earlier, `SHAMap`s (even immutable ones) may grow. If a `SHAMap` is
searching for a node and runs into an empty spot in the trie, then the `SHAMap`
looks to see if the node exists but has not yet been made part of the map. This
operation is performed in the `SHAMap::fetchNodeNT()` method. The *NT*
looks to see if the node exists but has not yet been made part of the map. This
operation is performed in the `SHAMap::fetchNodeNT()` method. The _NT_
is this case stands for 'No Throw'.
The `fetchNodeNT()` method goes through three phases:
1. By calling `cacheLookup()` we attempt to locate the missing node in the
TreeNodeCache. The TreeNodeCache is a cache of immutable SHAMapTreeNodes
1. By calling `cacheLookup()` we attempt to locate the missing node in the
TreeNodeCache. The TreeNodeCache is a cache of immutable SHAMapTreeNodes
that are shared across all `SHAMap`s.
Any SHAMapLeafNode that is immutable has a sequence number of zero
(sharable). When a mutable `SHAMap` is created then its SHAMapTreeNodes are
given non-zero sequence numbers (unsharable). But all nodes in the
given non-zero sequence numbers (unsharable). But all nodes in the
TreeNodeCache are immutable, so if one is found here, its sequence number
will be 0.
2. If the node is not in the TreeNodeCache, we attempt to locate the node
in the historic data stored by the data base. The call to to
2. If the node is not in the TreeNodeCache, we attempt to locate the node
in the historic data stored by the data base. The call to to
`fetchNodeFromDB(hash)` does that work for us.
3. Finally if a filter exists, we check if it can supply the node. This is
3. Finally if a filter exists, we check if it can supply the node. This is
typically the LedgerMaster which tracks the current ledger and ledgers
in the process of closing.
## Canonicalize ##
## Canonicalize
`canonicalize()` is called every time a node is introduced into the `SHAMap`.
@@ -251,51 +251,50 @@ by favoring the copy already in the `TreeNodeCache`.
By using `canonicalize()` we manage a thread race condition where two different
threads might both recognize the lack of a SHAMapLeafNode at the same time
(during a fetch). If they both attempt to insert the node into the `SHAMap`, then
(during a fetch). If they both attempt to insert the node into the `SHAMap`, then
`canonicalize` makes sure that the first node in wins and the slower thread
receives back a pointer to the node inserted by the faster thread. Recall
receives back a pointer to the node inserted by the faster thread. Recall
that these two `SHAMap`s will share the same `TreeNodeCache`.
## `TreeNodeCache` ##
## `TreeNodeCache`
The `TreeNodeCache` is a `std::unordered_map` keyed on the hash of the
`SHAMap` node. The stored type consists of `shared_ptr<SHAMapTreeNode>`,
`SHAMap` node. The stored type consists of `shared_ptr<SHAMapTreeNode>`,
`weak_ptr<SHAMapTreeNode>`, and a time point indicating the most recent
access of this node in the cache. The time point is based on
access of this node in the cache. The time point is based on
`std::chrono::steady_clock`.
The container uses a cryptographically secure hash that is randomly seeded.
The `TreeNodeCache` also carries with it various data used for statistics
and logging, and a target age for the contained nodes. When the target age
and logging, and a target age for the contained nodes. When the target age
for a node is exceeded, and there are no more references to the node, the
node is removed from the `TreeNodeCache`.
## `FullBelowCache` ##
## `FullBelowCache`
This cache remembers which trie keys have all of their children resident in a
`SHAMap`. This optimizes the process of acquiring a complete trie. This is used
when creating the missing nodes list. Missing nodes are those nodes that a
`SHAMap`. This optimizes the process of acquiring a complete trie. This is used
when creating the missing nodes list. Missing nodes are those nodes that a
`SHAMap` refers to but that are not stored in the local database.
As a depth-first walk of a `SHAMap` is performed, if an inner node answers true to
`isFullBelow()` then it is known that none of this node's children are missing
nodes, and thus that subtree does not need to be walked. These nodes are stored
in the FullBelowCache. Subsequent walks check the FullBelowCache first when
nodes, and thus that subtree does not need to be walked. These nodes are stored
in the FullBelowCache. Subsequent walks check the FullBelowCache first when
encountering a node, and ignore that subtree if found.
## `SHAMapTreeNode` ##
## `SHAMapTreeNode`
This is an abstract base class for the concrete node types. It holds the
This is an abstract base class for the concrete node types. It holds the
following common data:
1. A hash
2. An identifier used to perform copy-on-write operations
### `SHAMapInnerNode`
### `SHAMapInnerNode` ###
`SHAMapInnerNode` publicly inherits directly from `SHAMapTreeNode`. It holds
`SHAMapInnerNode` publicly inherits directly from `SHAMapTreeNode`. It holds
the following data:
1. Up to 16 child nodes, each held with a shared_ptr.
@@ -304,36 +303,34 @@ the following data:
4. An identifier used to determine whether the map below this node is
fully populated
### `SHAMapLeafNode` ###
### `SHAMapLeafNode`
`SHAMapLeafNode` is an abstract class which publicly inherits directly from
`SHAMapTreeNode`. It isIt holds the
`SHAMapTreeNode`. It isIt holds the
following data:
1. A shared_ptr to a const SHAMapItem.
#### `SHAMapAccountStateLeafNode` ####
#### `SHAMapAccountStateLeafNode`
`SHAMapAccountStateLeafNode` is a class which publicly inherits directly from
`SHAMapLeafNode`. It is used to represent entries (i.e. account objects, escrow
objects, trust lines, etc.) in a state map.
#### `SHAMapTxLeafNode` ####
#### `SHAMapTxLeafNode`
`SHAMapTxLeafNode` is a class which publicly inherits directly from
`SHAMapLeafNode`. It is used to represent transactions in a state map.
#### `SHAMapTxPlusMetaLeafNode` ####
#### `SHAMapTxPlusMetaLeafNode`
`SHAMapTxPlusMetaLeafNode` is a class which publicly inherits directly from
`SHAMapLeafNode`. It is used to represent transactions along with metadata
associated with this transaction in a state map.
## SHAMapItem ##
## SHAMapItem
This holds the following data:
1. uint256. The hash of the data.
2. vector<unsigned char>. The data (transactions, account info).
1. uint256. The hash of the data.
2. vector<unsigned char>. The data (transactions, account info).

1
tests/README.md
View File

@@ -1,4 +1,5 @@
# Integration tests
This directory contains integration tests for the project. These tests are run
against the `libxrpl` library or `rippled` binary to verify they are working as
expected.