This change makes progress on the plan in #4371. It does not replicate
the full [matrix] implemented in #3851, but it does replicate the 1.ii
section of the Linux matrix. It leverages "heavy" self-hosted runners,
and demonstrates a repeatable pattern for future matrices.
[matrix]: d794a0f3f1/.github/README.md (continuous-integration)
Address issues related to the removal of `std::{u,bi}nary_function` in
C++17 and some warnings with Clang 16. Some warnings appeared with the
upgrade to Apple clang version 14.0.3 (clang-1403.0.22.14.1).
- `std::{u,bi}nary_function` were removed in C++17. They were empty
classes with a few associated types. We already have conditional code
to define the types. Just make it unconditional.
- libc++ checks a cast in an unevaluated context to see if a type
inherits from a binary function class in the standard library, e.g.
`std::equal_to`, and this causes an error when the type privately
inherits from such a class. Change these instances to public
inheritance.
- We don't need a middle-man for the empty base optimization. Prefer to
inherit directly from an empty class than from
`beast::detail::empty_base_optimization`.
- Clang warns when all the uses of a variable are removed by conditional
compilation of assertions. Add a `[[maybe_unused]]` annotation to
suppress it.
- As a drive-by clean-up, remove commented code.
See related work in #4486.
If `--quorum` setting is present on the command line, use the specified
value as the minimum quorum. This allows for the use of a potentially
fork-unsafe quorum, but it is sometimes necessary for small and test
networks.
Fix#4488.
---------
Co-authored-by: RichardAH <richard.holland@starstone.co.nz>
Add instructions for installing rippled using the package managers APT
and YUM. Some steps were adapted from xrpl.org.
---------
Co-authored-by: Michael Legleux <mlegleux@ripple.com>
Newer compilers, such as Apple Clang 15.0, have removed `std::result_of`
as part of C++20. The build instructions provided a fix for this (by
adding a preprocessor definition), but the fix was broken.
This fixes the fix by:
* Adding the `conf` prefix for tool configurations (which had been
forgotten).
* Passing `extra_b2_flags` to `boost` package to fix its build.
* Define `BOOST_ASIO_HAS_STD_INVOKE_RESULT` in order to build boost
1.77 with a newer compiler.
Add a `NetworkID` field to help prevent replay attacks on and from
side-chains.
The new field must be used when the server is using a network id > 1024.
To preserve legacy behavior, all chains with a network ID less than 1025
retain the existing behavior. This includes Mainnet, Testnet, Devnet,
and hooks-testnet. If `sfNetworkID` is present in any transaction
submitted to any of the nodes on one of these chains, then
`telNETWORK_ID_MAKES_TX_NON_CANONICAL` is returned.
Since chains with a network ID less than 1025, including Mainnet, retain
the existing behavior, there is no need for an amendment.
The `NetworkID` helps to prevent replay attacks because users specify a
`NetworkID` field in every transaction for that chain.
This change introduces a new UINT32 field, `sfNetworkID` ("NetworkID").
There are also three new local error codes for transaction results:
- `telNETWORK_ID_MAKES_TX_NON_CANONICAL`
- `telREQUIRES_NETWORK_ID`
- `telWRONG_NETWORK`
To learn about the other transaction result codes, see:
https://xrpl.org/transaction-results.html
Local error codes were chosen because a transaction is not necessarily
malformed if it is submitted to a node running on the incorrect chain.
This is a local error specific to that node and could be corrected by
switching to a different node or by changing the `network_id` on that
node. See:
https://xrpl.org/connect-your-rippled-to-the-xrp-test-net.html
In addition to using `NetworkID`, it is still generally recommended to
use different accounts and keys on side-chains. However, people will
undoubtedly use the same keys on multiple chains; for example, this is
common practice on other blockchain networks. There are also some
legitimate use cases for this.
A `app.NetworkID` test suite has been added, and `core.Config` was
updated to include some network_id tests.
The `Ledger` class contains two `SHAMap` instances: the state and
transaction maps. Previously, the maps were dynamically allocated using
`std::make_shared` despite the fact that they did not require lifetime
management separate from the lifetime of the `Ledger` instance to which
they belong.
The two `SHAMap` instances are now regular member variables. Some smart
pointers and dynamic memory allocation was avoided by using stack-based
alternatives.
Commit 3 of 3 in #4218.
The `SHAMapItem` class contains a variable-sized buffer that
holds the serialized data associated with a particular item
inside a `SHAMap`.
Prior to this commit, the buffer for the serialized data was
allocated separately. Coupled with the fact that most instances
of `SHAMapItem` were wrapped around a `std::shared_ptr` meant
that an instantiation might result in up to three separate
memory allocations.
This commit switches away from `std::shared_ptr` for `SHAMapItem`
and uses `boost::intrusive_ptr` instead, allowing the reference
count for an instance to live inside the instance itself. Coupled
with using a slab-based allocator to optimize memory allocation
for the most commonly sized buffers, the net result is significant
memory savings. In testing, the reduction in memory usage hovers
between 400MB and 650MB. Other scenarios might result in larger
savings.
In performance testing with NFTs, this commit reduces memory size by
about 15% sustained over long duration.
Commit 2 of 3 in #4218.
When instantiating a large amount of fixed-sized objects on the heap
the overhead that dynamic memory allocation APIs impose will quickly
become significant.
In some cases, allocating a large amount of memory at once and using
a slabbing allocator to carve the large block into fixed-sized units
that are used to service requests for memory out will help to reduce
memory fragmentation significantly and, potentially, improve overall
performance.
This commit introduces a new `SlabAllocator<>` class that exposes an
API that is _similar_ to the C++ concept of an `Allocator` but it is
not meant to be a general-purpose allocator.
It should not be used unless profiling and analysis of specific memory
allocation patterns indicates that the additional complexity introduced
will improve the performance of the system overall, and subsequent
profiling proves it.
A helper class, `SlabAllocatorSet<>` simplifies handling of variably
sized objects that benefit from slab allocations.
This commit incorporates improvements suggested by Greg Popovitch
(@greg7mdp).
Commit 1 of 3 in #4218.
Add Clio-specific JSS constants to ensure a common vocabulary of
keywords in Clio and this project. By providing visibility of the full
API keyword namespace, it reduces the likelihood of developers
introducing minor variations on names used by Clio, or unknowingly
claiming a keyword that Clio has already claimed. This change moves this
project slightly away from having only the code necessary for running
the core server, but it is a step toward the goal of keeping this
server's and Clio's APIs similar. The added JSS constants are annotated
to indicate their relevance to Clio.
Clio can be found here: https://github.com/XRPLF/clio
Signed-off-by: ledhed2222 <ledhed2222@users.noreply.github.com>
- Include NFTokenPages in account_objects to make it easier to
understand an account's Owner Reserve and simplify app development.
- Update related tests and documentation.
- Fix#4347.
For info about the Owner Reserve, see https://xrpl.org/reserves.html
---------
Co-authored-by: Scott Schurr <scott@ripple.com>
Co-authored-by: Ed Hennis <ed@ripple.com>
Change `ledger_data` to return an empty list when all entries are
filtered out.
When the `type` field is specified for the `ledger_data` method, it is
possible that no objects of the specified type are found. This can even
occur if those objects exist, but not in the section that the server
checked while serving your request. Previously, the `state` field of the
response has the value `null`, instead of an empty array `[]`. By
changing this to an empty array, the response is the same data type so
that clients can handle it consistently.
For example, in Python, `for entry in state` should now work correctly.
It would raise an exception if `state` is `null` (or `None`).
This could break client code that explicitly checks for null. However,
this fix aligns the response with the documentation, where the `state`
field is an array.
Fix#4392.
Previously, the object `account_data` in the `account_info` response
contained a single field `Flags` that contains flags of an account. API
consumers must perform bitwise operations on this field to retrieve the
account flags.
This change adds a new object, `account_flags`, at the top level of the
`account_info` response `result`. The object contains relevant flags of
the account. This makes it easier to write simple code to check a flag's
value.
The flags included may depend on the amendments that are enabled.
Fix#2457.
Log exception messages at several locations.
Previously, these were locations where an exception was caught, but the
exception message was not logged. Logging the exception messages can be
useful for analysis or debugging. The additional logging could have a
small negative performance impact.
Fix#3213.
In the release notes (current and historical), there is a link to the
`Builds` directory. By creating `Builds/README.md` with a link to
`BUILD.md`, it is easier to find the build instructions.
* Create the FeeSettings object in genesis ledger.
* Initialize with default values from the config. Removes the need to
pass a Config down into the Ledger initialization functions, including
setup().
* Drop the undocumented fee config settings in favor of the [voting]
section.
* Fix#3734.
* If you previously used fee_account_reserve and/or fee_owner_reserve,
you should change to using the [voting] section instead. Example:
```
[voting]
account_reserve=10000000
owner_reserve=2000000
```
* Because old Mainnet ledgers (prior to 562177 - yes, I looked it up)
don't have FeeSettings, some of the other ctors will default them to
the config values before setup() tries to load the object.
* Update default Config fee values to match Mainnet.
* Fix unit tests:
* Updated fees: Some tests are converted to use computed values of fee
object, but the default Env config was also updated to fix the rest.
* Unit tests that check the structure of the ledger have updated
hashes and counts.
Add the ability to mark amendments as obsolete. There are some known
amendments that should not be voted for because they are broken (or
similar reasons).
This commit marks four amendments as obsolete:
1. `CryptoConditionsSuite`
2. `NonFungibleTokensV1`
3. `fixNFTokenDirV1`
4. `fixNFTokenNegOffer`
When an amendment is `Obsolete`, voting for the amendment is prevented.
A determined operator can still vote for the amendment by changing the
source, and doing so does not break any protocol rules.
The "feature" command now does not modify the vote for obsolete
amendments.
Before this change, there were two options for an amendment's
`DefaultVote` behavior: yes and no.
After this change, there are three options for an amendment's
`VoteBehavior`: DefaultYes, DefaultNo, and Obsolete.
To be clear, if an obsolete amendment were to (somehow) be activated by
consensus, the server still has the code to process transactions
according to that amendment, and would not be amendment blocked. It
would function the same as if it had been voting "no" on the amendment.
Resolves#4014.
Incorporates review feedback from @scottschurr.
Fix a case where `ripple::Expected` returned a json array, not a value.
The problem was that `Expected` invoked the wrong constructor for the
expected type, which resulted in a constructor that took multiple
arguments being interpreted as an array.
A proposed fix was provided by @godexsoft, which involved a minor
adjustment to three constructors that replaces the use of curly braces
with parentheses. This makes `Expected` usable for
[Clio](https://github.com/XRPLF/clio).
A unit test is also included to ensure that the issue doesn't occur
again in the future.
Make it easy for projects to depend on libxrpl by adding an `ALIAS`
target named `xrpl::libxrpl` for projects to link.
The name was chosen because:
* The current library target is named `xrpl_core`. There is no other
"non-core" library target against which we need to distinguish the
"core" library. We only export one library target, and it should just
be named after the project to keep things simple and predictable.
* Underscores in target or library names are generally discouraged.
* Every target exported in CMake should be prefixed with the project
name.
By adding an `ALIAS` target, existing consumers who use the `xrpl_core`
target will not be affected.
* In the future, there can be a migration plan to make `xrpl_core` the
`ALIAS` target (and `libxrpl` the "real" target, which will affect the
filename of the compiled binary), and eventually remove it entirely.
Also:
* Fix the Conan recipe so that consumers using Conan import a target
named `xrpl::libxrpl`. This way, every consumer can use the same
instructions.
* Document the two easiest methods to depend on libxrpl. Both have been
tested.
* See #4443.
* Remove obsolete build instructions.
* By using Conan, builders can choose which dependencies specifically to
build and link as shared objects.
* Refactor the build instructions based on the plan in #4433.
Without the protocol amendment introduced by this commit, an NFT ID can
be reminted in this manner:
1. Alice creates an account and mints an NFT.
2. Alice burns the NFT with an `NFTokenBurn` transaction.
3. Alice deletes her account with an `AccountDelete` transaction.
4. Alice re-creates her account.
5. Alice mints an NFT with an `NFTokenMint` transaction with params:
`NFTokenTaxon` = 0, `Flags` = 9).
This will mint a NFT with the same `NFTokenID` as the one minted in step
1. The params that construct the NFT ID will cause a collision in
`NFTokenID` if their values are equal before and after the remint.
With the `fixNFTokenRemint` amendment, there is a new sequence number
construct which avoids this scenario:
- A new `AccountRoot` field, `FirstNFTSequence`, stays constant over
time.
- This field is set to the current account sequence when the account
issues their first NFT.
- Otherwise, it is not set.
- The sequence of a newly-minted NFT is computed by: `FirstNFTSequence +
MintedNFTokens`.
- `MintedNFTokens` is then incremented by 1 for each mint.
Furthermore, there is a new account deletion restriction:
- An account can only be deleted if `FirstNFTSequence + MintedNFTokens +
256` is less than the current ledger sequence.
- 256 was chosen because it already exists in the current account
deletion constraint.
Without this restriction, an NFT may still be remintable. Example
scenario:
1. Alice's account sequence is at 1.
2. Bob is Alice's authorized minter.
3. Bob mints 500 NFTs for Alice. The NFTs will have sequences 1-501, as
NFT sequence is computed by `FirstNFTokenSequence + MintedNFTokens`).
4. Alice deletes her account at ledger 257 (as required by the existing
`AccountDelete` amendment).
5. Alice re-creates her account at ledger 258.
6. Alice mints an NFT. `FirstNFTokenSequence` initializes to her account
sequence (258), and `MintedNFTokens` initializes as 0. This
newly-minted NFT would have a sequence number of 258, which is a
duplicate of what she issued through authorized minting before she
deleted her account.
---------
Signed-off-by: Shawn Xie <shawnxie920@gmail.com>
There were situations where `marker`s returned by `account_lines` did
not work on subsequent requests, returning "Invalid Parameters".
This was caused by the optimization implemented in "Enforce account RPC
limits by account objects traversed":
e28989638d
Previously, the ledger traversal would find up to `limit` account lines,
and if there were more, the marker would be derived from the key of the
next account line. After the change, ledger traversal would _consider_
up to `limit` account objects of any kind found in the account's
directory structure. If there were more, the marker would be derived
from the key of the next object, regardless of type.
With this optimization, it is expected that `account_lines` may return
fewer than `limit` account lines - even 0 - along with a marker
indicating that there are may be more available.
The problem is that this optimization did not update the
`RPC::isOwnedByAccount` helper function to handle those other object
types. Additionally, XLS-20 added `ltNFTOKEN_OFFER` ledger objects to
objects that have been added to the account's directory structure, but
did not update `RPC::isOwnedByAccount` to be able to handle those
objects. The `marker` provided in the example for #4354 includes the key
for an `ltNFTOKEN_OFFER`. When that `marker` is used on subsequent
calls, it is not recognized as valid, and so the request fails.
* Add unit test that walks all the object types and verifies that all of
their indexes can work as a marker.
* Fix#4340
* Fix#4354
When writing objects to the NodeStore, we need to convert them from
the in-memory format to the binary format used by the node store.
The conversion is handled by the `EncodedBlob` class, which is only
instantiated on the stack. Coupled with the fact that most objects
are under 1024 bytes in size, this presents an opportunity to elide
a memory allocation in a critical path.
This commit also simplifies the interface of `EncodedBlob` and
eliminates a subtle corner case that could result in dangling
pointers.
These changes are not expected to cause a significant reduction in
memory usage. The change avoids the use of a `std::shared_ptr` when
unnecessary and tries to use stack-based memory allocation instead
of the heap whenever possible.
This is a net gain both in terms of memory usage (lower
fragmentation) and performance (less work to do at runtime).
In rare circumstances, both `onRequestTimeout` and the response handler
(`onSiteFetch` or `onTextFetch`) can get queued and processed. In all
observed cases, the response handler processes a network error.
`onRequestTimeout` usually runs first, but on rare occasions, the
response handler runs first, which leaves `activeResource` empty.
* Prevent internal error by catching overflow exception in `gateway_balances`.
* Treat `gateway_balances` obligations overflow as max (largest valid) `STAmount`.
* Note that very large sums of STAmount are approximations regardless.
---------
Co-authored-by: Scott Schurr <scott@ripple.com>
- MSVC 19.x reported a warning about import paths in boost for
function_output_iterator class (boost::function_output_iterator).
- Eliminate that warning by updating the import paths, as suggested by
the compiler warnings.
Port numbers can now be specified using either a colon or a space.
Examples:
1.2.3.4:51235
1.2.3.4 51235
- In the configuration file, an annoying "gotcha" for node operators is
accidentally specifying IP:PORT combinations using a colon. The code
previously expected a space, not a colon. It also does not provide
good feedback when this operator error is made.
- This change simply allows this mistake (using a colon) to be fixed
automatically, preserving the intention of the operator.
- Add unit tests, which test the functionality when specifying IP:PORT
in the configuration file.
- The RPCCall test regime is not specific enough to test this
functionality, it has been tested by hand.
- Ensure IPv6 addresses are not confused for ip:port
---------
Co-authored-by: Elliot Lee <github.public@intelliot.com>
- Implement the `operator==` and the `operator<=>` (aka the spaceship
operator) in `base_uint`, `Issue`, and `Book`.
- C++20-compliant compilers automatically provide the remaining
comparison operators (e.g. `operator<`, `operator<=`, ...).
- Remove the function compare() because it is no longer needed.
- Maintain the same semantics as the existing code.
- Add some unit tests to gain further confidence.
- Fix#2525.
In Reporting Mode, a server would core dump when it is not able to read
from Cassandra. This patch prevents the core dump when Cassandra is down
for reporting mode servers. This does not fix the root cause, but it
cuts down on some of the resulting noise.
* Follow-up to #4336
* NFToken is the naming convention in the codebase (rather than NFT)
* Rename `lsfDisallowIncomingNFTOffer` to `lsfDisallowIncomingNFTokenOffer`
* Rename `asfDisallowIncomingNFTOffer` to `asfDisallowIncomingNFTokenOffer`
Partially revert the functionality introduced
with #4195 / 5a15229 (part of 1.10.0-b1).
Acknowledgements:
Aaron Hook for responsibly disclosing this issue.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to
responsibly disclose any issues they may find.
To report a bug, please send a detailed report to:
bugs@xrpl.org
---------
Co-authored-by: Nik Bougalis <nikb@bougalis.net>
- Copies the recipe for Snappy from Conan Center, but removes three
lines that explicitly link the standard library, which prevents
builders from statically linking it.
- Removes the recipe for RocksDB now that an official recipe for version
6.27.3 is in Conan Center.
Developers will likely need to remove cached versions of both RocksDB
and Snappy:
```
conan remove -f rocksdb
conan remove -f snappy
```
---------
Co-authored-by: John Freeman <jfreeman08@gmail.com>
