o Fixes an off-by-one when determining which NFTokenPage an
NFToken belongs on.
o Improves handling of packed sets of 32 NFTs with
identical low 96-bits.
o Fixes marker handling by the account_nfts RPC command.
o Tightens constraints of NFTokenPage invariant checks.
Adds unit tests to exercise the fixed cases as well as tests
for previously untested functionality.
The amendment increases the maximum sign of an account's signer
list from 8 to 32.
Like all new features, the associated amendment is configured with
a default vote of "no" and server operators will have to vote for
it explicitly if they believe it is useful.
* "A path is considered invalid if and only if it enters and exits an
address node through trust lines where No Ripple has been enabled for
that address." (https://xrpl.org/rippling.html#specifics)
* When loading trust lines for an account "Alice" which was reached
via a trust line that has the No Ripple flag set on Alice's side, do
not use or cache any of Alice's trust lines which have the No Ripple
flag set on Alice's side. For typical "end-user" accounts, this will
return no trust lines.
One of the two versions of the `rngfill` function accepts a pointer
to a buffer and a size (in bytes). The function aims to fill the
provided `buffer` with `size` random bytes. It does this in chunks
of 8 bytes, for long as possible, and then fills any left-over gap
one byte at a time.
To avoid an annoying and incorrect warning about a potential buffer
overflow in the "trailing write", commit 78bc2727f7
used a `#pragma` to instruct the compiler to not generate the incorrect
diagnostic. Unfortunately, this change _also_ eliminated the trailing
write code, which means that, under some cases, the `rngfill` function
would generate between 1 and 7 fewer random bytes than requested.
This problem would only manifest on builds that do not define `__GNUC__`
which, as of this writing, means MSVC.
Several hard-coded parameters control the behavior of the ledger
acquisition engine. The values of many of these parameters where
set by intuition and have complex and non-intuitive interactions
with each other and other parts of the code.
An earlier commit attempted to adjust several of these parameters
to improve syncing performance; initial testing was promising but
a number of operators reported experiencing syncing and stability
issues with their servers. As a result, this commit reverts parts
of commit 18235067af.
This commit further adjusts some tunables so as to increase the
aggressiveness of the ledger acquisition engine.
This commit addresses minor bugs introduced with commit
6faaa91850:
- The number of threads used by the database engine was
incorrectly clamped to the lower possible value, such
that the database was effectively operating in single
threaded mode.
- The number of requests to extract at once was so high
that it could result in increased latency. The bundle
size is now limited to 4 and can be adjusted by a new
configuration option `rq_bundle` in the `[node_db]`
stanza. This is an advanced tunable and adjusting it
should not be needed.
This commit modernizes the `AcceptedLedger` and `AcceptedLedgerTx`
classes, reduces their memory footprint and reduces unnecessary
dynamic memory allocations.
This commit optimizes the way asynchronous nodestore operations are
processed both by reducing the amount of time locks are held and by
minimizing the number of memory allocations and data copying.
Adds support to TaggedCache to support smart replacement
(Needed to avoid race conditions with negative caching.)
Create a "hotDUMMY" object that represents the absence
of an object.
Allow DatabaseNodeImp::asyncFetch to complete immediately
if object is in cache (positive or negative).
Fix a bug in asyncFetch where the object returned may not
be the correct canonical version because we stash the
object in the results array before we canonicalize it.
When fetching ledgers, the existing code would isolate the peer
that sent the most useful responses and issue follow up queries
only to that peer.
This commit increases the query aggressiveness, and changes the
mechanism used to select which peers to issue follow-up queries
to so as to more evenly spread the load along those peers which
provided useful responses.
The `SHAMapInnerNode` class had a global mutex to protect the
array of node children. Profiling suggested that around 4% of
all attempts to lock the global would block.
This commit removes that global mutex, and replaces it with a
new per-node 16-way spinlock (implemented so as not to effect
the size of an inner node objet), effectively eliminating the
lock contention.
* Txs with the same fee level will sort by TxID XORed with the parent
ledger hash.
* The TxQ is re-sorted after every ledger.
* Attempt to future-proof the TxQ tie breaking test
* Abort background path finding when closed or disconnected
* Exit pathfinding job thread if there are no requests left
* Don't bother creating the path find job if there are no requests
* Refactor to remove circular dependency between InfoSub and PathRequest
The existing trust line caching code was suboptimal in that it stored
redundant information, pinned SLEs into memory and required multiple
memory allocations per cached object.
This commit eliminates redundant data, reducing the size of cached
objects and unpinning SLEs from memory, and uses value_types to
avoid the need for `std::shared_ptr`. As a result of these changes, the
effective size of a cached object, includes the overhead of the memory
allocator and the `std::shared_ptr` should be reduced by at least 64
bytes. This is significant, as there can easily be tens of millions
of these objects.
This commit combines the `apply_mutex` and `read_mutex` into a single `mutex_`
var. This new `mutex_` var is a `shared_mutex`, and most operations only need to
lock it with a `shared_lock`. The only exception is `applyMutex`, which may need
a `unique_lock`.
One consequence of removing the `apply_mutex` is more than one `applyMutex`
function can run at the same time. To help reduce lock contention that a
`unique_lock` would cause, checks that only require reading data are run a
`shared_lock` (call these the "prewriteChecks"), then the lock is released, then
a `unique_lock` is acquired. Since a currently running `applyManifest` may write
data between the time a `shared_lock` is released and the `write_lock` is
acquired, the "prewriteChecks" need to be rerun. Duplicating this work isn't
ideal, but the "prewirteChecks" are relatively inexpensive.
A couple of other designs were considered. We could restrict more than one
`applyMutex` function from running concurrently - either with a `applyMutex` or
my setting the max number of manifest jobs on the job queue to one. The biggest
issue with this is if any other function ever adds a write lock for any reason,
`applyManifest` would not be broken - data could be written between the release
of the `shared_lock` and the acquisition of the `unique_lock`. Note: it is
tempting to solve this problem by not releasing the `shared_mutex` and simply
upgrading the lock. In the presence of concurrently running `applyManifest`
functions, this will deadlock (both function need to wait for the other to
release their read locks before they can acquire a write lock).
In order to preserve the Hooks ABI, it is important that field
values used for hooks be stable going forward.
This commit reserves the required codes so that they will not
be repurposed before Hooks can be proposed for inclusion in
the codebase.
* Remove Application & Database dependency in PerfLog. Replace it with
a callback passed into the constructor.
* Fixes the circular dependency between ripple/nodestore and ripple/basics
If fast loading is enabled but the last persisted ledger is not
entirely on disk, the server would fail to start without manual
intervention by the server operator.
This commit allows the server to detect this scenario and attempt
to automatically recover.
This is a refactor aimed at cleaning up and simplifying the existing
job queue.
As of now, all jobs are cancelled at the same time and in the same
way, so this commit removes the per-job cancellation token. If the
need for such support is demonstrated, support can be re-added.
* Revise documentation for ClosureCounter and Workers.
* Simplify code, removing unnecessary function arguments and
deduplicating expressions
* Restructure job handlers to no longer need to pass a job's
handle to the job.
A typographical error would mishandle the case where a caller explicitly
tries to remove a child that is not actually part of the node. This case
is never invoked in practice, and so the bug would will never trigger.
Commit bf013c02ad added support
for incorporating a commit ID into the compiled version string
but did so in a way that did not follow the semantic versioning
standard.
This commit corrects that flaw by moving the commit ID into the
"metadata" part of the version string and properly handles the
case where the commit hash cannot be retrieved.
The pathfinding engine built into the code has several configurable
parameters to adjust the depth of the paths indexed and explored.
These parameters can dramatically impact the performance and memory
consumption of a server; higher values can result in resource usage
increasing exponentially.
These default values were decided early and somewhat arbitrarily at
a time when the network and the size of the network state were much
smaller.
This commit adjusts the default values to reduce the depth of paths
to more reasonable levels; unless explicitly overriden, the changes
mean that pathfinding operations will return fewer, shallower paths
than previous versions of the software.
This commit corrects a technical flaw that was introduced with commit
7c12f01358: as written, a mutex that is
intended to help provide synchronization for multiple threads as they
are each walking the map, is declared so that each thread is passed a
dangling reference to a unique mutex.
This commit hoists the mutex outside the thread creation loop, so all
threads use a single mutex and eliminating the dangling reference.
The nodestore includes a built-in cache to reduce the disk I/O
load but, by default, this cache was not initialized unless it
was explicitly configured by the server operator.
This commit introduces sensible defaults based on the server's
configured node size.
It remains possible to completely disable the cache if desired
by explicitly configuring it the cache size and age parameters
to 0:
[node_db]
...
cache_size = 0
cache_age = 0
- Only duplicate records from archive to writable during online_delete.
- Log duration of nodestore reads.
- Include nodestore counters in perf_log output.
- Remove gratuitous nodestore activity counting.
- Report initial sync duration in server_info and perfLog.
- Report state_accounting in perfLog.
- Make state_accounting durations more accurate.
- Parallel ledger loader.
- Config parameter to load ledgers on start.
1) Don't acquire so many nodes per pass. It's likely
far more than we need.
2) Right-size the finishedReads_ vector on passes other
than just the first.
* Sort by fee level (which is the current behavior) then by transaction
ID (hash).
* Edge case when the account at the end of the queue submits a higher
paying transaction to walk backwards and compare against the cheapest
transaction from a different account.
* Use std::if_any to simplify the JobQueue::isOverloaded loop.
* Log load fee values (at debug) received from validations.
* Log remote and cluster fee values (at trace) when changed.
* Refactor JobQueue::isOverloaded to return sooner if overloaded.
* Refactor Transactor::checkFee to only compute fee if ledger is open.
This flag, if present, suppresses the output of incoming
trustlines in default state.
This is primarily motivated by observing that users of Xumm often
have many unwanted incoming trustlines in a default state, which are
not useful in the vast majority of cases.
Being able to suppress those when doing `account_lines` saves bandwidth
and resources.
This commit implements partitioned unordered maps and makes it possible
to traverse such a map in parallel, allowing for more efficient use of
CPU resources.
The `CachedSLEs`, `TaggedCache`, and `KeyCache` classes make use of the
new functionality, which should improve performance.
The pathfinding engine requires pre-building large tables which is a
resource-intensive operation. Typically, one would not expect that a
server configured as a validator would also support pathfinding APIs
and so, building those tables by default wastes resources.
This commit, if merged, will disable pathfinding on servers that are
configured as validators, unless the server operator opts to support
it explicitly, by configuring the `[path_search_max]` parameter.
Validator operators that wish to support pathfinding on a validator
and want to use the default values can add the following stanza to
their server's configuration file:
[path_search_max]
7
The priority of different types of jobs was set back in the early
days of development, based on insight and observations that don't
necessarily apply any longer.
Specifically, job types used by the server to sync to the network
were being treated as lower priority than client requests, making
it more difficult to regain sync.
This commit adjusts the priority of several jobs and should allow
servers to prioritize resynchronizing to the network over serving
clients.
The existing calculation would limit the maximum number of threads
that would be created by default to at most 6; this may have been
reasonable a few years ago, but given both the load on the network
as of today and the increase in the number of CPU cores, the value
should be revisited.
This commit, if merged, changes the default calculation for nodes
that are configured as `large` or `huge` to allow for up to twelve
threads.
The "sweep interval" is the amount of time between successive sweeps of
of various in-memory data structures to remove stale items.
Prior to this commit, the interval was automatically adjusted, based on
the value of the `[node_size]` option in a server's configuration file.
If merged, this commit introduces a new configuration option that makes
it possible for a server operator to adjust the sweep interval and make
a CPU/memory tradeoff:
[sweep_interval]
<integer>
The specified value represents the number of seconds between successive
sweeps. The range of valid values is between 10 and 600.
Important operator notes:
This is an advanced configuration option that should not be used unless
there is empirical data which suggests that the default sweep frequency
is either resulting in performance problems or is causing undue load to
the server.
Note that adjusting the sweep interval may not have the intended effect
on the server. Lower values will not always translate to a reduction of
memory usage and higher values will not always translate to a reduction
of CPU usage and/or load.
The performance characteristics of `std::unordered_map` are better
than `std::map` and the former should be preferred when the strict
ordering of the latter is not required.
* Only require adding the new feature names in one place. (Also need to
increment a counter, but a check on startup will catch that.)
* Allows rippled to have the code to support a given amendment, but
not vote for it by default. This allows the amendment to be enabled in
a future version without necessarily amendment blocking these older
versions.
* The default vote is carried with the amendment name in the list of
supported amendments.
* The amendment table is constructed with the amendment and default
vote.
* Also clean up some formatting in the Windows instructions
* Changed the recommended version for Windows to 1.1.1L after deeper
checking uncovered some build issues.
* Patch the soci unsigned-types.h file. If no changes are made, delete
the patched file and exit. If there are changes, backup the original
and replace it with the patched file.
* Fixes#3885
Patch Rocksdb only once:
* The repeated patches do not appear to affect build times, but avoiding
unnecessary copies is good for its own sake.
There are two mutexes in ValidatorSite: `site_mutex_` and `state_mutex_`. Some
function end up locking both mutexes. However, depending on the call, the
mutexes could be locked in different orders, resulting in deadlocks.
If both mutexes are locked, this patch always locks the `sites_mutex_` first.
The existing logic involves every server sending every transaction
that it receives to all its peers (except the one that it received
a transaction from).
This commit instead uses a randomized algorithm, where a node will
randomly select peers to relay a given transaction to, caching the
list of transaction hashes that are not relayed and forwading them
to peers once every second. Peers can then determine whether there
are transactions that they have not seen and can request them from
the node which has them.
It is expected that this feature will further reduce the bandwidth
needed to operate a server.
The existing license file contains copyright statements and license
snippets referencing code that has long since been removed from the
codebase and which are not necessary any longer.
The general copyright statement for the ISC License is also changed
from "Ripple Labs" which is one contributor to the codebase, to the
more general "XRP Ledger Developers".
Remaining code which was originally taken from Bitcoin includes the
relevant copyright statement(s) inline.
To aid in the automated detection of the license type by GitHub and
tools like `licensee`, the following statement which referenced the
ISC license, and which was listed at the bottom of the file, is now
incorporated at the top:
>The accompanying files incorporate work covered by the following copyright
>and previous license notice:
>
>Copyright (c) 2011 Arthur Britto, David Schwartz, Jed McCaleb,
>Vinnie Falco, Bob Way, Eric Lombrozo, Nikolaos D. Bougalis, Howard Hinnant
This commit removes the `ltINVALID` pseudo-type identifier from
`LedgerEntryType` and the `ttINVALID` pseudo-type identifier from
`TxType` and includes several small additional improvements that
help to simplify the code base.
It also improves the documentation `LedgerEntryType` and `TxType`,
which was all over the place, and highlights some important caveats
associated with making changes to the ledger and transaction type
identifiers.
The commit also adds a safety check to the `KnownFormats<>` class,
that will catch the the accidental reuse of format identifiers.
Ideally, this should be done at compile time but C++ does not (yet?)
allow for the sort of introspection that would enable this.
The legacy functions `cdirFirst` and `dirFirst` were mostly
identical; the differences were only type-related. The same
situation existed with `cdirNext` and `dirNext`.
This commit removes the duplicated code by introducing new
template functions that abstract away the differences that
are present between each pair of functions.
This commit also improves the naming of function arguments,
helping to elucidate their purpose & use and to make the
code self-documenting.
The Negative UNL is a feature of the XRP Ledger consensus protocol that
improves liveness (the network's ability to make forward progress) during
a partial outage. Using the Negative UNL, servers adjust their effective
UNLs based on which validators are currently online and operational, so
that a new ledger version can be declared validated even if several trusted
validators are offline.
The Negative UNL has no impact on how the network processes transactions
or what transactions' outcomes are, except that it improves the network's
ability to declare outcomes final during some types of partial outages.
The feature was originally introduced with version **1.6.0** but it was
only possible to manually enable this. If merged, this commit introduces
the amendment associated with the feature so that server operators can
vote on whether to enable this feature.
For more details, please see https://xrpl.org/negative-unl.html
This commit closes#3898.
Under some circumstances, it is possible to induce an out-of-bounds
memory read in the base58 decoder.
This commit addresses this issue.
Acknowledgements:
Guido Vranken for discovering and responsibly disclosing this issue.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to
responsibly disclose any issues they may find.
Ripple is generously sponsoring a bug bounty program for the
rippled project. For more information please visit:
https://ripple.com/bug-bounty
The HardenedValidations amendment introduces additional fields
in validations:
- `sfValidatedHash`, if present, is the hash the of last ledger that
the validator considers to be fully validated.
- `sfCookie`, if present, is a 64-bit cookie (the default
implementation selects it randomly at startup but other
implementations are possible), which can be used to improve the
detection and classification of duplicate validations.
- `sfServerVersion`, if present, reports the version of the software
that the validator is running. By surfacing this information,
server operators gain additional insight about variety of software
on the network.
If merged, this commit fixes#3797 by adding the fields to the
`validations` stream as shown below:
- `sfValidateHash` as `validated_hash`: a 256-bit hex string;
- `sfCookie` as `cookie`: a 64-bit integer as a string; and
- `sfServerVersion` as `server_version`: a 64-bit integer as
a string.
With this amendment, the CheckCash transaction creates a TrustLine
if needed. The change is modeled after offer crossing. And,
similar to offer crossing, cashing a check allows an account to
exceed its trust line limit.
The following changes were made:
- Removed dependency on template defined in beast detail namespace.
- Removed Section::find() method which had an obsolete interface.
- Made Section::get<>() easier to use for the common case of
retrieving a std::string. The revised get() method replaces old
calls to Section::find().
- Provided a default template parameter to free function
get<>(Section config, std::string name) so it stays similar to
Section::get<>().
Then the rest of the code was adapted to these changes.
- Calls to Section::find() were replaced with calls to Section::get.
- Unnecessary get<std::string>() arguments were reduced to get().
These changes dug up an interesting artifact in the SHAMap unit
tests. I'm not sure why the tests were working before, but there
was a problem with the case of a Section key. The unit test is
fixed.
The `[node_size]` configuration parameter is used to tune various
parameters based on the hardware that the code is running on. The
parameter can take five distinct values: `tiny`, `small`, `medium`,
`large` and `huge`.
The default value in the code is `tiny` but the default configuration
file sets the value to `medium`. This commit attempts to detect the
amount of RAM on the system and adjusts the node size default value
based on the amount of RAM and the number of hardware execution
threads on the system.
The decision matrix currently used is:
| | 1 | 2 or 3 | ≥ 4 |
|:-------:|:----:|:------:|:------:|
| > ~8GB | tiny | tiny | tiny |
| > ~12GB | tiny | small | small |
| > ~16GB | tiny | small | medium |
| > ~24GB | tiny | small | large |
| > ~32GB | tiny | small | huge |
Some systems exclude memory reserved by the the hardware, the kernel
or the underlying hypervisor so the automatic detection code may end
up determining the node_size to be one less than "appropriate" given
the above table.
The detection algorithm is simplistic and does not take into account
other relevant factors. Therefore, for production-quality servers it
is recommended that server operators examine the system holistically
and determine what the appropriate size is instead of relying on the
automatic detection code.
To aid server operators, the node size will now be reported in the
`server_info` API as `node_size` when the command is invoked in
'admin' mode.
A recent version of clang notes a number of places in range
for loops where the code base was making unnecessary copies
or using const lvalue references to extend lifetimes. This
fixes the places that clang identified.
* Create SQLite database for mapping transaction IDs to shard indexes
* Create SQLite database for mapping ledger hashes to shard indexes
* Create additional test cases for the shard database
* load_factor was missing from server_info when the server was running in
reporting mode. Now, the reporting mode server calls server_info on the p2p
node, and propagates the load_factor back to the client.
In order to effectively mitigate CVE-2021-3499 even when compiling
against versions of OpenSSL prior to 1.1.1k, this commit:
1) requires use of TLS 1.2 or later. Note that both TLS 1.0 and
TLS 1.1 have been officially deprecated for over a year.
2) disables renegotiation support for TLS 1.2 connections.
Lastly, this commit also changes the default list of ciphers that
the server offers, limiting it only to ciphers that are part of
TLS 1.2.
The `tx` command supports output in both "text" and "binary" modes,
controlled by the binary flag. For more details on the command and
the possible arguments, please see: https://xrpl.org/tx.html.
The existing handler would incorrectly deal with metadata when in
binary mode. This commit corrects this issue, ensuring that the
metadata is properly encoded, depending on the mode.
Typically, an RPC response contains a `result` field, which
contains details about the operation performed. For ease of
parsing, forwarded responses must look like a non-forwarded
response.
In some instances the response was incorrectly composed, so
that the actual `result` object would be encapsulated by an
outer `result` object, breaking existing code.
This commit, addresses this issue and correctly "folds" the
`result` field, ensuring a consistent schema for responses.
* Add instructions to the workflow at the point where the failure would
occur, which is where someone experiencing a failure is most likely to
look. To keep things simple, the instructions are always printed. The
assumption is that if the job succeeds, nobody is likely to look
anyway.
* Provides the diff of the failure as an artifact, so the user can apply
it directly to their repo.
* Also update the levelization/README.md to clarify the levels a little
bit.
This updates the build process to use the local Artifactory server as a docker image cache to avoid being rate limited by docker hub during the build process.
While most of the code associated with secp256k1 operations had
been migrated to libsecp256k1, the deterministic key derivation
code was still using calls to OpenSSL.
If merged, this commit replaces the OpenSSL-based routines with
new libsecp256k1-based implementations. No functional change is
expected and the change should be transparent.
This commit also removes several support classes and utility
functions that wrapped or adapted various OpenSSL types that
are no longer needed.
A tip of the hat to the original author of this truly superb
library, Dr. Pieter Wuille, and to all other contributors.
This commit expands the detection capabilities of the Byzantine
validation detector. Prior to this commit, only validators that
were on a server's UNL were monitored. Now, all the validations
that a server receives are passed through the detector.
The existing class offered several constructors which were mostly
unnecessary. This commit eliminates all existing constructors and
introduces a single new one, taking a `Slice`.
The internal buffer is switched from `std::vector` to `Buffer` to
save a minimum of 8 bytes (plus the buffer slack that is inherent
in `std::vector`) per SHAMapItem instance.
Add support to allow multiple indepedent nodes to produce a binary identical
shard for a given range of ledgers. The advantage is that servers can use
content-addressable storage, and can more efficiently retrieve shards by
downloading from multiple peers at once and then verifying the integrity of
a shard by cross-checking its checksum with the checksum other servers report.
Before this change any non-zero Sequence field was handled as
a non-ticketed transaction, even if a TicketSequence was
present. We learned that this could lead to user confusion.
So the rules are tightened up.
Now if any transaction contains both a non-zero Sequence
field and a TicketSequence field then that transaction
returns a temSEQ_AND_TICKET error code.
The (deprecated) "sign" and "submit" RPC commands are tuned
up so they auto-insert a Sequence field of zero if they see
a TicketSequence in the transaction.
No amendment is needed because this change is going into
the first release that supports the TicketBatch amendment.
* Fix bug where incorrect max amount was set for XRP
* Fix bug where incorrect source currencies were set when XRP was the dst and a
sendmax amount was set
The existing code that deserialized an STAmount was sub-optimal and performed
poorly. In some rare cases the operation could result in otherwise valid
serialized amounts overflowing during deserialization. This commit will help
detect error conditions more quickly and eliminate the problematic corner cases.
* Use theoretical quality to order the strands
* Do not use strands below the user specified quality limit
* Stop exploring strands (at the current quality iteration) once any strand is non-dry
The previous error description was focused on keys that are too long,
but this error can occur if the key is too short or does not contain
the correct prefix.
* Add a new operating mode to rippled called reporting mode
* Add ETL mechanism for a reporting node to extract data from a p2p node
* Add new gRPC methods to faciliate ETL
* Use Postgres in place of SQLite in reporting mode
* Add Cassandra as a nodestore option
* Update logic of RPC handlers when running in reporting mode
* Add ability to forward RPCs to a p2p node
- The changes to manifest relaying introduced with commit f74b469e68
will cause newly accepted manifests to be sent back to the peer from
which they were received. This no longer happens: a newly accepted
manifest is never sent back to the peer we received it from.
- When encountering a manifest without a domain set, the `manifest` and
`validator_info` commands would include an empty string as the domain
associated with the manifest. This no longer happens: if a domain is
not present, the `domain` field will not be.
The existing code attempts to validate the provided node public key
using a function that assumes that the encoded public key is for an
account. This causes the parsing to fail.
This commit fixes#3317 by letting the caller specify the type of
the public key being checked.
The manifest relay code would only ever relay manifests from validators
on a server's UNL which means that the manifests of validators that are
not broadly trusted can fail to propagate across the network, which can
make it difficult to detect and track such validators.
This commit, if merged, propagates all manifests on a best-effort basis
resulting in broader availability of manifests on the network and avoid
the need to introduce on-ledger manifest storage or to establish one or
more manifest repositories.
- Add validation/proposal reduce-relay feature negotiation to
the handshake
- Make squelch duration proportional to a number of peers that
can be squelched
- Refactor makeRequest()/makeResponse() to facilitate handshake
unit-testing
- Fix compression enable flag for inbound peer
- Fix compression algorithm parsing in the header parser
- Fix squelch duration in onMessage(TMSquelch)
This commit fixes 3624, fixes 3639 and fixes 3641
Support for 'out-of-sequence' transaction execution was introduced
in commit 7724cca384.
The changes in that commit were gated under a feature but there was
no corresponding amendment introduced that would allow the network
to vote on this amendment.
This commit introduces 'TicketBatch' amendment as the amendment
that is associated with the tickets feature. If the amendment is
enabled, it will activate support for tickets.
This commit also removes several workarounds that are no longer
needed in unit tests.
* Markdown explanation of what levelization is, the intended levels, as
well as the process used to determine dependencies
* Shell script finds all dependencies, groups them, and finds cyclic
dependencies and maps out non-cyclic dependencies.
* Github job to run the script and fail if anything changes. Should
catch introduction of new dependencies and new problems. Will also
detect changes if problems or dependencies are removed.
* Creates a version 2 of the UNL file format allowing publishers to
pre-publish the next UNL while the current one is still valid.
* Version 1 of the UNL file format is still valid and backward
compatible.
* Also causes rippled to lock down if it has no valid UNLs, similar to
being amendment blocked, except reversible.
* Resolves#3548
* Resolves#3470
* Move all the vcpkg windows dependency installations into one step.
* Move the unmodified `before_install` step above the matrix to improve
readability, because this step runs before any of the matrix steps.
Due to some quirky emergent behavior, the server can't really begin
synching until twice the default close time resolution of the genesis
ledger, which is 30 seconds, has passed. In effect, this causes a one
minute delay.
This commit adjusts the default close time resolution down to the
minimum allowed resoluion of 10 seconds, so the corresponding delay
is reduced by 67% down to 20 seconds. This should be enough time to
ensure the server has reasonable connectivity without unduly delaying
initial synch times.
This comment explains this patch and the associated patches
that should be folded into it. This paragraph should be removed
when the patches are folded after review.
This change significantly improves ledger sync and fetch
times while reducing memory consumption. The change affects
the code from that begins with SHAMap::getMissingNodes and runs
through to Database::threadEntry.
The existing code issues a number of async fetches which are then
handed off to the Database's pool of read threads to execute.
The results of each read are placed in the Database's positive
and negative caches. The caller waits for all reads to complete
and then retrieves the results out of these caches.
Among other issues, this means that the results of the first read
cannot be processed until the last read completes. Additionally,
all the results must sit in memory.
This patch changes the behavior so that each read operation has a
completion handler associated with it. The completion of the read
calls the handler, allowing the results of each read to be
processed as it completes. As this was the only reason the
negative and positive caches were needed, they can now be removed.
The read generation code is also no longer needed and is removed.
The batch fetch logic was never implemented or supported and is
removed.
gcc's implementation of `prm::synchronized_pool_resource` showed
extremely poor performance compared with
`boost::synchronized_pool_resouece`. Boost's implementation of pmr is
now used in all cases (previously it was only used when a standard
lib, like clang's, lacked an implementation of pmr).
This patch also makes a minor change where inner nodes are constructed
with sparse arrays, unless "dense" is explicitly requested.
Prior to this commit, the amendments that a server would vote in support
of or against could be configured both via the configuration file and
via the command line "feature" command. Changes made in the configuration
file would only be loaded once at server startup and changes made via the
command line take effect immediately but are not persisted across
restarts.
This commit deprecates management of amendments via the configuration
file and stores the relevant information in the `wallet.db` database
file.
1. On startup, the new code parses the configuration file.
2. If the `[veto_amendments]` or `[amendments]` sections are present,
we check if the `FeatureVotes` table is present in `wallet.db`.
3. If it is not, we create the `FeatureVotes` table and transfer the
settings from the config file.
4. Proceed normally but only reference the `FeatureVotes` table instead
of the config file.
5. Warns if the voting table already exists in `wallet.db` and there
exists voting sections in the config file. The config file is ignored
in this case.
This change addresses & closes#3366
* Found several functions called under lock that take a lock. Refactor
to require a lock as a parameter instead.
* Found several functions called under lock that don't take a lock, but
should. Refactored those as well to require a lock as a parameter.
Unit tests are counting test failures, process crashes, and process exit code
failures in the count. Since a failing tests causes the process exit code to
return failure, we get extra counts. This patch removes process exit code
failures from the count.
ReadViewFwdRange was storing a cached `end_` iterator that was lazily
created in an iterators `end()` function. When the cache is empty, and
the range it iterated from multiple threads, this creates a race
condition.
This change has performance consequences for "old style" for loops.
For example:
```
// don't do this
for(auto i = tx_range.begin(); i != tx_range.end(); ++i)
```
Can call the now expensive `end()` function more often than needed.
Range-based for loop (I.e. `for(auto const& t : tx_range)`) should be
used instead.
- Under some conditions, comparing `ReadViewFwdRange::iterators`
for equality could derefence an empty `std::unique_ptr` which
will result in a crash.
- Misuse of the `equal` API could result in a `std::bad_cast`
exception being thrown from when iterating transactions or
SLEs from the `OpenView`, `RawStateTable` and `Ledger` classes.
A large percentage of inner nodes only store a small number of children. Memory
can be saved by storing the inner node's children in sparse arrays. Measurements
show that on average a typical SHAMap's inner nodes can be stored using only 25%
of the original space.
This commit combines a number of cleanups, targeting both the
code structure and the code logic. Large changes include:
- Using more strongly-typed classes for SHAMap nodes, instead of relying
on runtime-time detection of class types. This change saves 16 bytes
of memory per node.
- Improving the interface of SHAMap::addGiveItem and SHAMap::addItem to
avoid the need for passing two bool arguments.
- Documenting the "copy-on-write" semantics that SHAMap uses to
efficiently track changes in individual nodes.
- Removing unused code and simplifying several APIs.
- Improving function naming.
- Simplify and consolidate code for parsing hex input.
- Replace beast::endian::order with boost::endian::order.
- Simplify CountedObject code.
- Remove pre-C++17 workarounds in favor of C++17 based solutions.
- Improve `base_uint` and simplify its hex-parsing interface by
consolidating the `SexHex` and `SetHexExact` methods into one
API: `parseHex` which forces callers to verify the result of
the operation; as a result some public-facing API endpoints
may now return errors when passed values that were previously
accepted.
- Remove the simple fallback implementations of SHA2 and RIPEMD
introduced to reduce our dependency on OpenSSL. The code is
slow and rarely, if ever, exercised and we rely on OpenSSL
functionality for Boost.ASIO as well.
- Provide separate functions for serializing depending on whether
one wants a "wire" version of a node, or one suitable for hashing.
- Remove unused functions
The existing SHAMapNodeID object has both a valid and an invalid state
and requirs callers to verify the state of an instance prior to using
it. A simple set of changes removes that restriction and ensures that
all instances are valid, making the code more robust.
This change also:
1. Introduces a new function to construct a SHAMapNodeID from a
serialized blob; and
2. Reduces the amount of constructors the class exposes.
- Limit the lifetime of a buffer that was only used in the early
phases of peer connection establishment but which lived on as
long as the peer was active.
- Cache the message used to transfer manifests, so it can be reused
instead of recreated for every peer connection.
- Improve the reading of partial messages by passing a hint to the
I/O layer if the number of bytes needed to complete the message
is known.
The existing code issues a PING to each peer every 8 seconds. While
frequent PINGs allow us to estimate a peer's latency with a high
degree of accuracy, this "inter-server polka dance" is inefficient
and not useful. This commit, if merged, reduces the PING frequency
to once every 60 seconds.
Additionally, this commit simplifies the PING handling logic and
merges the code used to check and disconnect peers which fail to
track the network directly into the timer callback.
When evaluating the fitness and usefulness of an outbound peer, the code
would incorrectly calculate the amount of time that the peer spent in
a non-useful state.
This commit, if merged, corrects the calculation and makes the timeout
values configurable by server operators.
Two new options are introduced in the 'overlay' stanza of the config
file. The default values, in seconds, are:
[overlay]
max_unknown_time = 600
max_diverged_time = 300
This commit replaces the `peers_max` configuration element which had
a predetermined split between incoming and outgoing connections with
two new configuration options, `peers_in_max` and `peers_out_max`,
which server operators can use to explicitly control the number of
incoming and outgoing peer slots.
There have been cases in the past where SFields have been defined
in such a way that they did not follow our conventions. In
particular, the string representation of an SField should match
the in-code name of the SField.
This change leverages the preprocessor to encourage SFields to
be properly constructed.
The suffixes of SField types are changed to be the same as
the suffixes of corresponding SerializedTypeIDs. This allows
The preprocessor to match types using simple name pasting.
Since the string representation of the SField is part of our
stable API, the name of sfPayChannel was changed to sfChannel.
This change allows sfChannel to follow our conventions while
making no changes to our external API.
* Remove DinD Service from container build template
DinD has changed how it works on GItLab due to recent docker changes such that the service no long needs to be called so long as the runner is being run on a `docker-X` tagged machine.
* refactor for docker service on normal node
* If multiple transactions are queued for the account, change the
account's sequence number in a temporary view before processing the
transaction.
* Adds a new "at()" interface to STObject which is identical to the
operator[], but easier to write and read when dealing with ptrs.
* Split the TxQ tests into two suites to speed up parallel run times.
This commit introduces a new configuration option that server
operators can set. The value is communicated to other servers
and is also reported via the `server_info` API.
The value is meant to allow third-party applications or tools
to group servers together. For example, a tool that visualizes
the network's topology can group servers together.
Similar to the "Domain" field in validator manifests, an operator
can claim any domain. Prior to relying on the value returned, the
domain should be verified by retrieving the xrp-ledger.toml file
from the domain and looking for the server's public key in the
`nodes` array.
* Increases hard-coded number of parallel unit test processes for
Windows and MacOS builds from 1 to 2.
* Reduces Travis job time to well under the timeout value of 1.5 hours.
* Continue using a hard-coded value rather than `nprocs` because higher
values cause some jobs to run out of memory.
* Jobs with no unit tests are counted as failures. Resolves#3474
* Crashed processes are counted as failures. Resolves#3600
* Any tests specified on the command line test do not have matching
suites are counted as failures.
* Remove unused CI manual test.
When processing the `tx` command, we will now load both the transaction
and its metadata directly from SQLite.
Previously the `tx` RPC call was querying SQLite for the transaction
and then separately querying the key-value store for the metadata.
Support for IPv6 messages was added with commit 08382d866b
and version 1.1.0. No peer presently connected to the network in a useful capacity fails
to understand v2 messages.
This commit removes the code that generates and processes v1 messages and deletes legacy
messages from the protocol buffer definition file.
Use C++17 constant expressions to calculate the inverse
alphabet map at compile time instead of at runtime.
Remove support for encoding & decoding tokens using the
Bitcoin alphabet.
The job queue can impose limits of how many jobs of a particular
type can be queued.
This commit makes the previously hard-coded limit associated with
transactions configurable by the server's operator. Servers that
have increased memory capacity or which expect to see an influx
of transactions can increase the number of transactions their
server will be able to queue.
This commit fixes#3556.
The "/vl" HTTP endpoint can be used to request a particular
UNL from a rippled instance.
This commit, if merged, includes the public key of the requested
list in the response.
This commit fixes#3392
* Distinguish between recent and historical shards
* Allow multiple storage paths for historical shards
* Add documentation for this feature
* Add unit tests
Some RPC commands return `ledger_index` as a quoted numeric
string. This change allows the returned value to be directly
copied and used for follow-on RPC commands.
This commit fixes#3533
When attempting to load a validator list from a configured
site, attempt to reuse the last IP that was successfully
used if that IP is still present in the DNS response.
Otherwise, randomly select an IP address from the list of
IPs provided by the DNS system.
This commit fixes#3494.
With few exceptions, servers will typically receive multiple copies
of any given message from its directly connected peers. For servers
with several peers this can impact the processing latency and force
it to do redundant work. Proposal and validation messages are often
relayed with extremely high redundancy.
This commit, if merged, introduces experimental code that attempts
to optimize the relaying of proposals and validations by allowing
servers to instruct their peers to "squelch" delivery of selected
proposals and validations. Servers making squelching decisions by
a process that evaluates the fitness and performance of a given
server and randomly selecting a subset of the best candidates.
The experimental code is presently disabled and must be explicitly
enabled by server operators that wish to test it.
Tickets are a mechanism to allow for the "out-of-order" execution of
transactions on the XRP Ledger.
This commit, if merged, reworks the existing support for tickets and
introduces support for 'ticket batching', completing the feature set
needed for tickets.
The code is gated under the newly-introduced `TicketBatch` amendment
and the `Tickets` amendment, which is not presently active on the
network, is being removed.
The specification for this change can be found at:
https://github.com/xrp-community/standards-drafts/issues/16
Commit 4dc08f8202 introduced support for
deterministic shards, which makes the sharding functionality provided
by rippled more useful.
After merging, several opportunities for further improvements to the
deterministic sharding implementation were identified and a significant
increase int memory usage during shard finalization was detected.
Because of these issues, the commit is being reverted and the feature is
being rolled back. It will be reintroduced in a future release.
* Builds Windows dependencies first.
* Builds ALL OSs in the last stage.
* Fix the MacOS builds.
* Windows dependency stages are allowed to fail so ALL configurations will
attempt to build. Windows builds will probably fail if dependencies fail
(caching may allow them to succeed), but they will at least be attempted.
* Remove broken AppVeyor config file, so it stops trying.
The checkpointer class had assumed that the database would exist for the
lifetime of the application. This is no long true. These changes resolve bugs
involving dangling pointers.
There was a race condition in `on_accept` where the object's destructor
could run while `on_accept` was called.
This patch ensures that if `on_accept` is called then the object remains
valid for the duration of the call.
* Fixes#3486
* load factor computation normalized by load_base.
* last validated ledger age set to -1 while syncing.
* Return status changed:
* healthy -> ok
* warning -> service_unavailable
* critical -> internal_server_error
This change can help improve the liveness of the network during periods of network
instability, by allowing the network to track which validators are presently not online
and to disregard them for the purposes of quorum calculations.
If the 'HardenedValidations' amendment is enabled, this commit will
track the version of the software that validators embed in their
validations.
If a server notices that at least 60% of the validators on its UNL
are running a newer version than it is running, it will periodically
print an informational message, reminding the operator to check for
update.
The tecUNFUNDED code is actively used when attempting to create payment
channels; the messages incorrectly list it as deprecated.
Meanwhile, the tecUNFUNDED_ADD code actually is an unused legacy code,
dating back to when there was a WalletAdd transactor. The terLAST and
terFUNDS_SPENT codes are also unused legacy codes.
Engine result messages are not part of the binary format and are
documented as subject to change without notice, so this should not
require an amendment nor a new API version.
Align error code table for human readability.
The amendment was partially complete, included no functional code
and, even if activated, it would result in no changes to transaction
proessing. Despite this, removing the amendment is the prudent course
of action and avoids the possibility of an accidental activation.
If additional cryptoconditions are implemented, they will be each
assigned a new, unique amendment code.
This commit, if merged, adds support to allow multiple indepedent nodes to
produce a binary identical shard for a given range of ledgers. The advantage
is that servers can use content-addressable storage, and can more efficiently
retrieve shards by downloading from multiple peers at once and then verifying
the integrity of a shard by cross-checking its checksum with the checksum
other servers report.
* Document delete_batch, back_off_milliseconds, age_threshold_seconds.
* Convert those time values to chrono types.
* Fix bug that ignored age_threshold_seconds.
* Add a "recovery buffer" to the config that gives the node a chance to
recover before aborting online delete.
* Add begin/end log messages around the SQL queries.
* Add a new configuration section: [sqlite] to allow tuning the sqlite
database operations. Ignored on full/large history servers.
* Update documentation of [node_db] and [sqlite] in the
rippled-example.cfg file.
Resolves#3321
* The amendment ballot counting code contained a minor technical
flaw, caused by the use of integer arithmetic and rounding
semantics, that could allow amendments to reach majority with
slightly less than 80% support. This commit introduces an
amendment which, if enabled, will ensure that activation
requires at least 80% support.
* This commit also introduces a configuration option to adjust
the amendment activation hysteresis. This option is useful on
test networks, but should not be used on the main network as
is a network-wide consensus parameter that should not be
changed on a per-server basis; doing so can result in a
hard-fork.
Fixes#3396
Work on a version 2 of the XRP Network API has begun. The new
API returns:
* `notSynced` in place of `noClosed`, `noCurrent`, and `noNetwork`;
* `invalidParams` in place of `lgrIdxInvalid`.
The new version 2 API cannot be selected yet, as it remains a work
in progress.
Fixes#3269
If a port number is not specified in the [ips] or [ips_fixed]
blocks, automatically add the new default peer port which was
registered with IANA: 2459. Also use 2459 if no port is specified
with manually using the `connect` command; previously it was
using 6561, which could have resulted in spurious failures.
This commit, if merged, fixes#2861.
* Gives a summary of the health of the node:
Healthy, Warning, or Critical
* Last validated ledger age:
<7s is Healthy,
7s to 20s is Warning
> 20s is Critcal
* If amendment blocked, Critical
* Number of peers:
> 7 is Healthy
1 to 7 is Warning
0 is Critical
* server state:
One of full, validating or proposing is Healthy
One of syncing, tracking or connected is Warning
All other states are Critical
* load factor:
<= 100 is Healthy
101 to 999 is Warning
>= 1000 is Critical
* If not Healthy, info field contains data that is considered not
Healthy.
Fixes: #2809
Commit e257a22 introduced changes in the logic used to acquire historical
ledgers. The logic could cause historical ledgers to be acquired only since
the last online deletion interval instead of the configured value to allow
deletion.
* Make sure variables are always initialized
* Use lround instead of adding .5 and casting
* Remove some unneeded vars
* Check for null before calling strcmp
* Remove redundant if conditions
* Remove make_TxQ factory function
* Improve documentation
* Make the ShardArchiveHandler rather than the DatabaseShardImp perform
LastLedgerHash verification for downloaded shards
* Remove ShardArchiveHandler's singleton implementation and make it an
Application member
* Have the Application invoke ShardArchiveHandler initialization
instead of clients
* Add RecoveryHandler as a ShardArchiveHandler derived class
* Improve commenting
* Add documentation for shard validation
* Retrieve last ledger hash for imported shards
* Verify the last ledger hash in Shard::finalize
* Limit last ledger hash retrieval attempts for imported shards
* Use a common function for removing failed shards
* Add new ShardInfo::State for imported shards
Identifiers for retired amendments should not generally be used
in the codebase.
This commit reduces their visibility down to one translation
unit and marks them as unused and deprecated to prevent
accidental reuse.
In deciding whether to relay a proposal or validation, a server would
consider whether it was issued by a validator on that server's UNL.
While both trusted proposals and validations were always relayed,
the code prioritized relaying of untrusted proposals over untrusted
validations. While not technically incorrect, validations are
generally more "valuable" because they are required during the
consensus process, whereas proposals are not, strictly, required.
The commit introduces two new configuration options, allowing server
operators to fine-tune the relaying behavior:
The `[relay_proposals]` option controls the relaying behavior for
proposals received by this server. It has two settings: "trusted"
and "all" and the default is "trusted".
The `[relay_validations]` options controls the relaying behavior for
validations received by this server. It has two settings: "trusted"
and "all" and the default is "all".
This change does not require an amendment as it does not affect
transaction processing.
The sfLedgerSequence field is designated as optional in the object
template but it is effectively required and validations which do not
include it were, correctly, rejected.
This commit migrates the check outside of the peer code and into the
constructor used for validations being deserialized for the network.
Furthermore, the code will generate an error if a validation that is
generated by a server does not include the field.
The existing code used std::deque along with a size check to constrain the
size of a buffer and, effectively, "hand rolled" a circular buffer. This
change simply migrates directly to boost::circular_buffer.
The unit test now verifies that if an account is not present in the
starting account_tx ledger, account_tx still iterates down and finds
the transaction that deletes the account (and earlier transactions).
This commit introduces no functional changes but cleans up the
code and shrinks the surface area by removing dead and unused
code, leveraging std:: alternatives to hand-rolled code and
improving comments and documentation.
The script, when invoked by a server operator can collect information
useful for debugging, while attempting to redact potentially sensitive
data.
It contained no explanation or other exposition to allow people who
look at the file but aren't familiar with shell scripts to understand
its purpose.
The built-in watchdog is simplistic and can, sometimes, cause problems
especially on systems that have the ability to automatically start and
monitor processes.
This commit removes the sustain system entirely, changes the handling
of the SIGTERM signal to properly terminate the process and improves
the error message reported to the user when the command line used to
start `rippled` is incorrect and malformed.
Entries in the ledger are located using 256-bit locators. The locators
are calculated using a wide range of parameters specific to the entry
whose locator we are calculating (e.g. an account's locator is derived
from the account's address, whereas the locator for an offer is derived
from the account and the offer sequence.)
Keylets enhance type safety during lookup and make the code more robust,
so this commit removes most of the earlier code, which used naked
uint256 values.
This commit removes obsolete comments, dead or no longer useful
code, and workarounds for several issues that were present in older
compilers that we no longer support.
Specifically:
- It improves the transaction metadata handling class, simplifying
its use and making it less error-prone.
- It reduces the footprint of the Serializer class by consolidating
code and leveraging templates.
- It cleanups the ST* class hierarchy, removing dead code, improving
and consolidating code to reduce complexity and code duplication.
- It shores up the handling of currency codes and the conversation
between 160-bit currency codes and their string representation.
- It migrates beast::secure_erase to the ripple namespace and uses
a call to OpenSSL_cleanse instead of the custom implementation.
A deliberately malformed token can cause the server to crash during
startup. This is not remotely exploitable and would require someone
with access to the configuration file of the server to make changes
and then restart the server.
Acknowledgements:
Guido Vranken for responsibly disclosing this issue.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to
responsibly disclose any issues they may find.
Ripple is generously sponsoring a bug bounty program for the
rippled project. For more information please visit:
https://ripple.com/bug-bounty
Currently there is no mechanism for a validator to report the
version of the software it is currently running. Such reports
can be useful for those who are developing network monitoring
dashboards and server operators in general.
This commit, if merged, defines an encoding scheme to encode
a version string into a 64-bit unsigned integer and adds an
additional optional field to validations.
This commit piggybacks on "HardenedValidations" amendment to
determine whether version information should be propagated
or not.
The general encoding scheme is:
XXXXXXXX-XXXXXXXX-YYYYYYYY-YYYYYYYY-YYYYYYYY-YYYYYYYY-YYYYYYYY-YYYYYYYY
X: 16 bits identifying the particular implementation
Y: 48 bits of data specific to the implementation
The rippled-specific format (implementation ID is: 0x18 0x3B) is:
00011000-00111011-MMMMMMMM-mmmmmmmm-pppppppp-TTNNNNNN-00000000-00000000
M: 8-bit major version (0-255)
m: 8-bit minor version (0-255)
p: 8-bit patch version (0-255)
T: 11 if neither an RC nor a beta
10 if an RC
01 if a beta
N: 6-bit rc/beta number (1-63)
This commit introduces the "HardenedValidations" amendment which,
if enabled, allows validators to include additional information in
their validations that can increase the robustness of consensus.
Specifically, the commit introduces a new optional field that can
be set in validation messages can be used to attest to the hash of
the latest ledger that a validator considers to be fully validated.
Additionally, the commit leverages the previously introduced "cookie"
field to improve the robustness of the network by making it possible
for servers to automatically detect accidental misconfiguration which
results in two or more validators using the same validation key.
- Add missing `#include` in `ripple/core/JobTypeInfo.h`
- Protect version string from clang-format in
`ripple/protocol/impl/BuildInfo.cpp`.
`Builds/CMake/RippledVersion.cmake` searches for this line by pattern.
Existing per-thread PRNGs are individually initialized using calls
to std::random_device.
If merged, this commit will use a single PRNG, initialized from
std::random_device on startup, to seed the thread-specific PRNGs.
Acknowledgements:
Thomas Snider, who reported this issue to Ripple on April 8, 2020.
Historically strand re-execute log messages have been treated as
errors. However in the vast majority of cases these log messages
are caused by well understood mechanics in the payment engine.
So usually these log messages should be treated as warnings.
The automated build system only builds packages signed with a list of
approved keys. This is a security measure to prevent someone who gains
push access to the repository from producing potentially malicious
packages that are signed by Ripple's trusted private keys.
Moving this list to the new location makes it easy to add and delete
new keys to the list.
* scoped_lock is now a std name with subtly different semantics
compared to lock_guard. Namely it can be used to lock 0 or
more mutexes. This is valuable, but can also be accidentally
used to lock 0 mutexes when 1 was intended, creating a
run-time error.
Therefore, if and when we use scoped_lock, extra care needs to
be taken in reviewing that code to ensure it doesn't
accidentally lock 0 mutexes when 1 was intended. To aid in
such careful reviewing, the use of the name scoped_lock should
be limited to those cases where the number of mutexes is not
exactly one.
* canonicalize_replace_cache
* canonicalize_replace_client
Now it is clear at the call site that if there are
duplicate copies of the data between the cache and
the caller, which copy gets replaced.
Additionally data parameter is now const-correct.
If it is not going to be replaced (canonicalize_replace_cache),
then the shared_ptr to the client data is const.
* The [network_id] option allows three string values:
- main: the XRP Ledger
- testnet: the Testnet operated by Ripple.
- devnet: the development test network operated by Ripple.
* Peers negotiate compression via HTTP Header "X-Offer-Compression: lz4"
* Messages greater than 70 bytes and protocol type messages MANIFESTS,
ENDPOINTS, TRANSACTION, GET_LEDGER, LEDGER_DATA, GET_OBJECT,
and VALIDATORLIST are compressed
* If the compressed message is larger than the uncompressed message
then the uncompressed message is sent
* Compression flag and the compression algorithm type are included
in the message header
* Only LZ4 block compression is currently supported
The payment engine restricts payment paths so two steps do not input the
same Currency/Issuer or output the same Currency/Issuer. This check was
skipped when the path started or ended with XRP. An example of a path
that was incorrectly accepted was: XRP -> //USD -> //XRP -> EUR
This patch enables the path loop check for paths that start or end with
XRP.
* Make ShardArchiveHandler a singleton.
* Add state database for ShardArchiveHandler.
* Use temporary database for SSLHTTPDownloader downloads.
* Make ShardArchiveHandler a Stoppable class.
* Automatically resume interrupted downloads at server start.
* Reduce lock scope on all public functions
* Use TaskQueue to process shard finalization in separate thread
* Store shard last ledger hash and other info in backend
* Use temp SQLite DB versus control file when acquiring
* Remove boost serialization from cmake files
When computing rates for offers, an STAmount's value can be out of can be out of
range (before canonicalizing). There was an assert that could incorrectly fire
in some cases. This patch removes that assert.
The newest MSVC 19.25.28610.4 does not build rocksdb. During the
Travis CI Windows job, the vs_BuildTools.exe automatically
downloads the newest version of the compiler. This fix forces the
install of MSVC 19.24.28314.0 to build rocksdb.
The fix1781 amendment was incorrectly introduced during conflict
resolution and support for it is not included at this time. This commit
removes the definition of the amendment identifier.
A review of the lag ratchet code revealed that we were using
the long-term master public keys of trusted validators, when
we should have been using the ephemeral public keys instead.
As a result, the lag ratchet code would be effectively
inoperable.
- Add support for all transaction types and ledger object types to gRPC
implementation of tx and account_tx.
- Create common handlers for tx and account_tx.
- Remove mutex and abort() from gRPC server. JobQueue is stopped before
gRPC server, with all coroutines executed to completion, so no need for
synchronization.
* Whenever a node downloads a new VL, send it to all peers that
haven't already sent or received it. It also saves it to the
database_dir as a Json text file named "cache." plus the public key of
the list signer. Any files that exist for public keys provided in
[validator_list_keys] will be loaded and processed if any download
from [validator_list_sites] fails or no [validator_list_sites] are
configured.
* Whenever a node receives a broadcast VL message, it treats it as if
it had downloaded it on it's own, broadcasting to other peers as
described above.
* Because nodes normally download the VL once every 5 minutes, a single
node downloading a VL with an updated sequence number could
potentially propagate across a large part of a well-connected network
before any other nodes attempt to download, decreasing the amount of
time that different parts of the network are using different VLs.
* Send all of our current valid VLs to new peers on connection.
This is probably the "noisiest" part of this change, but will give
poorly connected or poorly networked nodes the best chance of syncing
quickly. Nodes which have no http(s) access configured or available
can get a VL with no extra effort.
* Requests on the peer port to the /vl/<pubkey> endpoint will return
that VL in the same JSON format as is used to download now, IF the
node trusts and has a valid instance of that VL.
* Upgrade protocol version to 2.1. VLs will only be sent to 2.1 and
higher nodes.
* Resolves#2953
* Example: gcc.Debug will use the the default version of gcc installed on the
system. gcc-9.Debug will use version 9, regardless of the default. This will
be most useful when the default is older than required or desired.
* When an unknown amendment reaches majority, log an error-level
message, and return a `warnings` array on all successful
admin-level RPC calls to `server_info` and `server_state` with
a message describing the problem, and the expected deadline.
* In addition to the `amendment_blocked` flag returned by
`server_info` and `server_state`, return a warning with a more
verbose description when the server is amendment blocked.
* Check on every flag ledger to see if the amendment(s) lose majority.
Logs again if they don't, resumes normal operations if they did.
The intention is to give operators earlier warning that their
instances are in danger of being amendment blocked, which will
hopefully motivate them to update ahead of time.
* update EP and find package requirements
* minor protobuf/libarchive build fixes
* change travis release builds to nounity to
ameliorate vm memory exhaustion.
FIXES: #3223, #3232
* In and Out parameters were swapped when calculating the rate
* In and out qualities were not calculated correctly; use existing functions
to get the qualities
* Added tests to check that theoretical quality matches actual computed quality
* Remove in/out parameter from qualityUpperBound
* Rename an overload of qualityUpperBound to adjustQualityWithFees
* Add fix amendment
STAmount::soTime and soTime2 were time based "amendment like"
switches to control small changes in behavior for STAmount.
soTime2, which was the most recent, was dated Feb 27, 2016.
That's over 3 years ago.
The main reason to retain these soTimes would be to replay
old transactions. The likelihood of needing to replay a
transaction from over three years ago is pretty low. So it
makes sense to remove these soTime values.
In Flow_test the testZeroOutputStep() test is removed. That
test started to fail when the STAmount::soTimes were removed.
I checked with the original author of the test. He said
that the code being tested by that unit test has been removed,
so it makes sense to remove the test. That test is removed.
* use tagged containers for pkg build
* update build images
* continue to build container images in pipeline, but allow
failure (non-block)
* limit travis macos cache
* add vs2019 windows to travis
* remove xcode 9 travis build
* remove clang5/6 from CI and update min version of Clang required in
cmake
* break windows CI build into stages to reduce timeouts
* update datelib
* add if condition to travis builds to allow commit message to limit
builds by platform
FIXES: #2847
* Transactions that are submitted with the fail_hard flag
and that result in any TER code besides tesSUCCESS shall
be neither queued nor held.
[FOLD] Keep tec results out of the open ledger when fail_hard:
* Improve TransactionStatus const correctness, and remove redundant
`local` check
* Check open ledger tx count in fail_hard tests
* Fix some wrapping
* Remove duplicate test
Remove the implicit conversion from int64 to XRPAmount. The motivation for this
was noticing that many calls to `to_string` with an integer parameter type were
calling the wrong `to_string` function. Since the calls were not prefixed with
`std::`, and there is no ADL to call `std::to_string`, this was converting the
int to an `XRPAmount` and calling `to_string(XRPAmount)`.
Since `to_string(XRPAmount)` did the same thing as `to_string(int)` this error
went undetected.
Prior to this commit, the queue and execution times for individual jobs
were reported indepedently and could, potentially, be out of sync. This
change reports both values when either one of the exceeds the reporting
threshold.
It's possible an overloaded job queue is causing false alarms on the deadlock
detector. Log a fatal message after 90s, declare a logic error after 600s.
If merged, this commit will report additional information in the
response to the submit command; this will make it easier for developers
to accurately track the status of transaction submission.
Fixes#2851
Treat all `#` characters in config files as comments (and remove)
*unless* the `#` is immediately preceded by `\`. Write a warning
to log file when trailing comments are found/ignored in the config
to let operators know that the treatment of trailing `#` has changed.
Fixes#3121
The 'network_id' option allows an administrator to specify to which
network they intend a server to connect. Servers can leverage this
information to optimize routing and prune automatically discovered
cross-network connections.
This commit will, if merged:
- add support for the devnet keyword, which corresponds to network ID #2;
- report the network ID, if one is configured, in server_info
The `node_size` configuration option is used to automatically
configure various parameters (cache sizes, timeouts, etc) for
the server.
A previous commit included changes that caused incorrect values
to be returned which can result in sub-optimal performance that
can manifest as difficulty syncing to the network, or increased
disk I/O and/or memory usage. The problem was introduced with
commit 66fad62e66.
This commit, if merged, fixes the code to ensure that the correct
values are returned and introduces a compile-time check to prevent
this issue from reoccurring.
The existing platform detection code was derived from the old Beast
library, which was, itself, derived from JUCE.
This commit removes that code and replaces it with the Boost.Predef
library which defines a consistent set of compiler, architecture,
operating system, library, and other version numbers.
For more on Boost.Predef, please see the Boost documentation. The
documentation for the current version as of this writing is at:
https://www.boost.org/doc/libs/1_71_0/doc/html/predef.html
This commit restructures the HTTP based protocol negotiation that `rippled`
executes and introduces support for negotiation of compression for peer
links which, if implemented, should result in significant bandwidth savings
for some server roles.
This commit also introduces the new `[network_id]` configuration option
that administrators can use to specify which network the server is part of
and intends to join. This makes it possible for servers from different
networks to drop the link early.
The changeset also improves the log messages generated when negotiation
of a peer link upgrade fails. In the past, no useful information would
be logged, making it more difficult for admins to troubleshoot errors.
This commit also fixes RIPD-237 and RIPD-451
* The `tx` command now supports min_ledger and max_ledger fields.
* If the requested transaction isn't found and these fields are
provided, the error response indicates whether or not every
ledger in the the provided range was searched.
This fixes#2924
* adding package signing steps for rpm and deb
* first spike at GPG signing with CI and containers
* refine ubuntu portion
* get correct gpg package version
* adding CentOS support
* fixing errors in installing gpg on ubuntu
* base64 decode the GPG key
* fixing line continuations
* revised package signing, looking for package artifacts
* add dpkg-sig to ubuntu image
* sign all deb packges
* add passphrase to GPG process
* repeat yo slef on dpkg
* sign all the rpm packages too
* install rpm-sign in the CentOS docker image
* loop through rpm files
* no need for PIN on GPG signing
Collecting the returned and expected values in sets only works if there are no
duplicates. The implementation is changed to use sorted vectors to fix this case.
When the Env::AppBundle constructor throws an exception
it still needs to run ~AppBundle(), otherwise the JobQueue
isn't properly shut down. Specifically the JobQueue
can destruct without waiting on outstanding jobs in the
queue.
This change ensures that if Env::AppBundle constructor
throws, Env::AppBundle::~AppBundle() runs.
This fixes the unit test crash exposed by PR #3047.
FIXES: #3106
Different versions of protobuf produce subtly different
results when given invalid message payloads. This leads to
subtly different behavior when we try to deserialize these
invalid messages. As such, we can't tie success to a
particular exception.
The XRP Ledger utilizes an account model. Unlike systems based on a UTXO
model, XRP Ledger accounts are first-class objects. This design choice
allows the XRP Ledger to offer rich functionality, including the ability
to own objects (offers, escrows, checks, signer lists) as well as other
advanced features, such as key rotation and configurable multi-signing
without needing to change a destination address.
The trade-off is that accounts must be stored on ledger. The XRP Ledger
applies reserve requirements, in XRP, to protect the shared global ledger
from growing excessively large as the result of spam or malicious usage.
Prior to this commit, accounts had been permanent objects; once created,
they could never be deleted.
This commit introduces a new amendment "DeletableAccounts" which, if
enabled, will allow account objects to be deleted by executing the new
"AccountDelete" transaction. Any funds remaining in the account will
be transferred to an account specified in the deletion transaction.
The amendment changes the mechanics of account creation; previously
a new account would have an initial sequence number of 1. Accounts
created after the amendment will have an initial sequence number that
is equal to the ledger in which the account was created.
Accounts can only be deleted if they are not associated with any
obligations (like RippleStates, Escrows, or PayChannels) and if the
current ledger sequence number exceeds the account's sequence number
by at least 256 so that, if recreated, the account can be protected
from transaction replay.
* replace boost::beast::detail::iequals with boost::iequals
* replace deprecated `buffers` function with `make_printable`
* replace boost::beast::detail::ascii_tolower with lambda
* add missing includes
The validation stream only reported the ephemeral signing key for validators
which use manifests. This made tracking unnecessarily difficult for clients
processing the data stream.
With this change, the validator's long-term master public key is also
included.
This commit fixes#3005
* Provide proposing validator's master key in the validation stream
subscription JSON responses.
Implement code review changes.
FIXES: #3005
* Explain that Arch/Manjaro/etc. need `-Dstatic=OFF` during the configure step
* move configuration options closer to that step
* separate sub-headers for configuration and build
Different compilers and handling the shadow warning differently. In particular,
some are warning about types being shadowed by variables. Until these can be
resolved the shadow warning is being disabled.
Note: the shadow warning was originally enabled to help with the structured
bindings patch. As that is now complete, it's less important to keep this
warning.
FIXES: #2527
* define custom docker image for travis-linux builds based on
package build image
* add macos builds
* add windows builds (currently allowed to fail)
* improve build and shell scripts as required for the CI envs
* add asio timer latency workaround
* omit several manual tests from TravisCI which cause memory exhaustion
This commit allows server operators to reserve slots for specific
peers (identified by the peer's public node identity) and to make
changes to the reservations while the server is operating.
This commit closes#2938
- Add docker container tags for "latest_BRANCH"
- Prevent different branches from overwriting deb repo artifacts
- Manual approval always required before pushing to prod
The original intent was that RPC error codes were not stable.
But those codes were made available through the API, so some
users came to depend on the code values. This change adapts
to the current state of affairs.
Manifests which are revoked can include ephemeral keys although doing
so does not make sense: a revoked manifest isn't used for signing and
so don't need to define an ephemeral key.
A running instance of the server tracks the number of protocol messages
and the number of bytes it sends and receives.
This commit makes the counters more granular, allowing server operators
to better track and understand bandwidth usage.
* Add construction and assignment from a generic
contiguous container. Both compile-time and run time
safety checks are made to ensure the safety of this
conversion.
* Remove base_uint::copyFrom. The generic copy assignment
operator now does this functionality with enhanced
safety and better syntax.
* Remove construction from and dedendence on Blob.
The generic constructor and assignment now handle this
functionality.
* Fix client code to adhere to this new API.
* Removed the use of fromVoid in PeerImp.cpp as it was
an inappropriate use of this dangerous API. The
generic container constructors do it with enhanced
safety and better syntax.
* Rename data member pn to data_ and make it private.
* Remove constraint from hash_append
* Remove array_type alias
This PR addresses a problem where the server could hang indefinitely
on shutdown. The cause of the problem is the SNTPClock class was not
binding the socket to an endpoint on initialization. This can create
an error sent to the read handler. Unfortunately, the handler ignores
the error, reads again and enters into a loop preventing the
io_service from ever completing.
- Explain how to bind to both IPv4 and IPv6 interfaces
- Provide a hint in the default [port_peer] section
- Do not enable it by default
Note that on Linux, use of '::' and IPv4-mapped IPv6 depends on a sysctl value
setting 'net.ipv6.bindv6only = 0' which seems to be the default on most Linux
distributions.
- Use `std::lock` when grabbing multiple mutexes to ensure consistent
locking order and avoid deadlocks.
- Reduce the scope of the master mutex lock by relesing it prior to
calling setHeartbeatTimer
A tiny input amount to a payment step can cause this step to output zero. For
example, if a previous steps outputs a dust amount of 10^-80, and this step is a
IOU -> XRP offer, the offer may output zero drops. In this case, call the strand
dry. Before this patch, an error would be logged, the strand would be called
dry; in debug mode an assert triggered.
Note, this patch is not transaction breaking, as the caller did not user the ter
code. The caller only checked for success or failuer.
This patch addresses github issue issue reported here:
https://github.com/ripple/rippled/issues/2929
This patch removes calls to several deprecated asio functions.
* `io_service::post` becomes `post` (free function)
* `io_service::work` becomes `executor_work_guard`
* `io_service::wrap` becomes `bind_executor`
* `get_io_context` becomes `get_executor` or `get_executor().context()`
This patch was tested with boost 1.69 and 1.70. The functions
`ripple::get_lowest_layer` and `beast::create_waitable_timer` are required to
handle a breaking difference between these versions. When rippled no longer
needs to support pre 1.70 boost versions, both of these functions may be
removed, and the waitable timer injections may also be removed.
The new parse logic is more strict but handles more cases. If an exception
is thrown, just bail.
* Allow parsing unenclosed IPv6 addresses without port
* Improve string construction
* Reduce nesting levels of code
The XRP Ledger allows an account to authorize a secondary key pair,
called a regular key pair, to sign future transactions, while keeping
the master key pair offline.
The regular key pair can be changed as often as desired, without
requiring other changes on the account.
If merged, this commit corrects a minor technical flaw which would
allow an account holder to specify the master key as the account's
new regular key.
The change is controlled by the `fixMasterKeyAsRegularKey` amendment
which, if enabled, will:
1. Prevent specifying an account's master key as the account's
regular key.
2. Prevent the "Disable Master Key" flag from incorrectly affecting
regular keys.
Before this patch, jtx allowed non-invocable functions to be passed to
operator(). However, these arguments are ignored. This caused erronious code
code such as:
```
env (offer (account_to_test, BTC (250), XRP (1000)),
offers (account_to_test, 1));
```
While it looks like the number of offers are checked, they are not. The `offers`
funclet is never run. While we could modify jtx to make the above code correct,
a cleaner solution is to run post conditions in a `require` statement after a
transasction runs.
At this point all of the jss::* names are defined in the same
file. That file has been named JsonFields.h. That file name
has little to do with either JsonStaticStrings (which is what
jss is short for) or with jss. The file is renamed to jss.h
so the file name better reflects what the file contains.
All includes of that file are fixed. A few include order
issues are tidied up along the way.
Formerly an SOTemplate was default constructed and its elements
added using push_back(). This left open the possibility of a
malformed SOTemplate if adding one of the elements caused a throw.
With this commit the SOTemplate requires an initializer_list of
its elements at construction. Elements may not be added after
construction. With this approach either the SOTemplate is fully
constructed with all of its elements or the constructor throws,
which prevents an invalid SOTemplate from even existing.
This change requires all SOTemplate construction to be adjusted
at the call site. Those changes are also in this commit.
The SOE_Flags enum is also renamed to SOEStyle, which harmonizes
the name with other uses in the code base. SOEStyle elements
are renamed (slightly) to have an "soe" prefix rather than "SOE_".
This heads toward reserving identifiers with all upper case for
macros. The new style also aligns with other prominent enums in
the code base like the collection of TER identifiers.
SOElement is adjusted so it can be stored directly in an STL
container, rather than requiring storage in a unique_ptr.
Correspondingly, unique_ptr usage is removed from both
SOTemplate and KnownFormats.
The new 'Domain' field allows validator operators to associate a domain
name with their manifest in a transparent and independently verifiable
fashion.
It is important to point out that while this system can cryptographically
prove that a particular validator claims to be associated with a domain
it does *NOT* prove that the validator is, actually, associated with that
domain.
Domain owners will have to cryptographically attest to operating particular
validators that claim to be associated with that domain. One option for
doing so would be by making available a file over HTTPS under the domain
being claimed, which is verified separately (e.g. by ensuring that the
certificate used to serve the file matches the domain being claimed) and
which contains the long-term master public keys of validator(s) associated
with that domain.
Credit for an early prototype of this idea goes to GitHub user @cryptobrad
who introduced a PR that would allow a validator list publisher to attest
that a particular validator was associated with a domain. The idea may be
worth revisiting as a way of verifying the domain name claimed by the
validator's operator.
Resource limits were not properly applied to connections with
known IP addresses but no corresponding users.
Add unit tests for unlimited vs. limited ports.
An audit showed that a number of the RPC error codes in
ErrorCodes.h are no longer used in the code base. The unused
codes were removed from the file along with their support code
in ErrorCodes.cpp.
The ledger already declared a transaction that is both single-
and multi-signing malformed. This just adds some checking in
the signing RPC commands (like submit and sign_for) which allows
that sort of error to be identified a bit closer to the user.
In the process of adding this code a bug was found in the
RPCCall unit test. That bug is fixed as well.
If the number of peers a server has is below the configured
minimum peer limit, this commit will properly transition the
server's state to "disconnected".
The default limit for the minimum number of peers required was
0 meaning that a server that was connected but lost all its
peers would never transition to disconnected, since it could
never drop below zero peers.
This commit redefines the default minimum number of peers to 1
and produces a warning if the server is configured in a way
that will prevent it from ever achieving sufficient connectivity.
If a server is configured to support crawl, it will report the
IP addresses of all peers it is connected to, unless those peers
have explicitly opted out by setting the `peer_private` option
in their config file.
This commit makes servers that are configured as validators
opt out of crawling.
Several commands allow a user to retrieve a server's status. Commands
will typically limit disclosure of information that can reveal that a
particular server is a validator to connections that are not verified
to make it more difficult to determine validators via fingerprinting.
Prior to this commit, servers configured to operate as validators
would, instead of simply reporting their server state as 'full',
augment their state information to indicate whether they are
'proposing' or 'validating'.
Servers will only provide this enhanced state information for
connections that have elevated privileges.
Acknowledgements:
Ripple thanks Markus Teufelberger for responsibly disclosing this issue.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to responsibly
disclose any issues that they may find. For more on Ripple's Bug Bounty
program, please visit: https://ripple.com/bug-bounty
The /crawl API endpoint allows developers to examine the structure of
the XRP Ledger's overlay network.
This commit adds additional information about the local server to the
/crawl endpoint, making it possible for developers to create data-rich
network-wide status dashboards.
Related:
- https://developers.ripple.com/peer-protocol.html
- https://github.com/ripple/rippled-network-crawler
When deserializing specially crafted data, the code would ignore certain
types of errors. Reserializing objects created from such data results in
failures or generates a different serialization, which is not ideal.
Also addresses: RIPD-1677, RIPD-1682, RIPD-1686 and RIPD-1689.
Acknowledgements:
Ripple thanks Guido Vranken for responsibly disclosing these issues.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to responsibly
disclose any issues that they may find. For more on Ripple's Bug Bounty
program, please visit: https://ripple.com/bug-bounty
Specially crafted messages could cause the server to buffer large
amounts of memory which could increase memory pressure.
This commit changes how messages are buffered and imposes a limit
on the amount of data that the server is willing to buffer.
Acknowledgements:
Aaron Hook for responsibly disclosing this issue.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to
responsibly disclose any issues they may find. For information
on Ripple's Bug Bounty program, please visit:
https://ripple.com/bug-bounty
The constructor would previously assert that the specified buffer pointer
was non-null, even if the buffer size is specified as 0. While reasonable,
this also makes it more difficult to use this API.
* Using txnsExpected_, which is influenced by both the config
and network behavior, can reserve far too much or far too
little memory, wasting time and resources.
* Not an issue during normal operation, but a user could
cause problems on their local node with extreme configuration
settings.
* initFee was using a lot of logic that could look unclear. Add
some documentation explaining why certain values were used.
* Because initFee had side effects, callers needed to repeat the
max queue size computation, making the initial problem more
likely. Instead, return the max queue size value, so the caller
can reuse it.
* A newer test (testInFlightBalance()) was incorrectly using a
hard-coded queue limit. Fix it to use initFee's new return
value.
The --rpc_port command-line option is effectively ignored. We construct
an `Endpoint` with the given port, but then drop it on the floor.
(Perhaps the author thought the `Endpoint::at_port` method is a mutation
instead of a transformation.) This small change adds the missing
assignment to hold on to the new endpoint.
Fixes#2764
This changeset ensures the preferred ledger calculation
properly distinguishes the absence of trusted validations
from a preferred ledger which is the genesis ledger.
The `STObject` member function `setType()` has been renamed to
applyTemplate() and modified to throw if there is a template
mismatch.
The error description in the exception is, in certain cases,
used, to better indicate why a particular transaction was
considered ill formed.
Fixes#2585.
`Json::Value::isConvertibleTo` indicates that unsigned integers and
reals are convertible to string, but trying to do so (with
`Json::Value::asString`) throws an exception because its internal switch
is missing these cases. This change fills them in (and adds tests).
Acknowledgements:
Ripple thanks Guido Vranken for responsibly disclosing this issue.
Closes#2778
Although `parseURL` used a regex to pull the authority out of the URL
being parsed, it performed manual parsing of the hostname and port.
This commit rolls the parsing of the username and password, if any,
directly into the regex. The hostname can be a name, an IPv4 or an
IPv6 address.
Fixes#2751
* Adds local file:// URL support to the [validator_list_sites] stanza.
The file:// URL must not contain a hostname. Allows a rippled node
operator to "sideload" a new list if their node is unable to reach
a validator list's web site before an old list expires. Lists
loaded from a file will be validated in the same way a downloaded
list is validated.
* Generalize file/dir "guards" from Config test so they can be reused
in other tests.
* Check for error when reading validators.txt. Saves some parsing and
checking of an empty string, and will give a more meaningful error.
* Completes RIPD-1674.
* Relevant when deciding whether an account can queue multiple
transactions. If the potential spend of the already queued
transactions would dip into the reserve, the reserve is
preserved for fees.
* Also change several direct modifications of the owner count to
call adjustOwnerCount to preserve overflow checking.
* Update related unit testcase
* Resolves#2251
Perform some extra checks on the close time and sequence number
of a candidate for network consensus ledger. This tightens
defenses against some "insane/hostile supermajority" attacks.
Under certain conditions, we could call `memcpy` or `memcmp` with a null
source pointer. Even when specifying 0 as the amount of data to copy this
could result in undefined behavior under the C and C++ standards.
Acknowledgements:
Ripple thanks Guido Vranken for responsibly disclosing these issues.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to responsibly
disclose any issues that they may find. For more on Ripple's Bug Bounty
program, please visit: https://ripple.com/bug-bounty
The XRP Ledger is designed to be censorship resistant. Any attempt to
censor transactions would require coordinated action by a majority of
the system's validators.
Importantly, the design of the system is such that such an attempt is
detectable and can be easily proven since every validators must sign
the validations it publishes.
This commit adds an automated censorship detector. While the server is
in sync, the detector tracks all transactions that, in the view of the
server, should have been included and issues warnings of increasing
severity for any transactions which, have not after several rounds.
When Ed25519 support was added to ripple-lib, a way to specify
whether a seed should be used to derive a "classic" secp256k1
keypair or a "new" Ed25519 keypair was needed, and the
requirements were that:
1. previously seeds would, correctly, generate a secp256k1
keypair.
2. users would not have to know about whether the seed was
used to generate a secp256k1 or an Ed25519 keypair.
To address these requirements, the decision was made to encode
the type of key within the seed and a custom encoding was
designed.
The encoding uses a token type of 1 and prefixes the actual
seed with a 2 byte header, selected to ensure that all such
keypairs will, when encoded, begin with the string "sEd".
This custom encoding is non-standard and was not previously
documented; as a result, it is not widely supported and other
sofware may treat such keys as invalid. This can make it
difficult for users that have stored such a seed to use
wallets or other tooling that is not based on ripple-lib.
This commit adds support to rippled for automatically
detecting and properly handling such seeds.
The 'validation_seed' RPC command was used to change the validation
key used by a validator at runtime.
Its implementation was commented out with commit fa796a2eb5
which has been included in the codebase since the 0.30.0 release
and there are no plans to reintroduce the functionality at this
point.
Validator operators should migrate to using validator manifests
instead.
This fixes#2748.
The FeeEscalation amendment has been enabled on the XRP Ledger network
since May 19, 2016. The transaction which activated this amendment is:
5B1F1E8E791A9C243DD728680F108FEF1F28F21BA3B202B8F66E7833CA71D3C3.
This change removes all conditional code based around the FeeEscalation
amendment, but leaves the amendment definition itself since removing the
definition would cause nodes to think an unknown amendment was activate
causing them to become amendment blocked.
The commit also removes the redundant precomputed hashes from the
supportedAmendments vector.
Problem:
- There are only a few call sites to cachedRead, and all of them
currently do more work than is required since we know the type in each
case.
Solution:
- "Inline" the codepath to cachedRead, but do not check if the type is
valid. In all such call sites, we know the keylet to read directly.
This fixes#2550
The WaitableEvent class was a leftover from the pre-Boost
version of Beast and used Windows- and pthread-specific
APIs.
This refactor replaces that functionality by using only
interfaces provided by the C++ standard, making the code
more portable.
Closes#2402.
Many of the warnings on Windows were not resolved, just
silenced with _SILENCE_ALL_CXX17_DEPRECATION_WARNINGS.
They need to be resolved in a future commit.
* If rippled is not synced to the network, `fee` will return a
"no network" error instead of the possibly confusing "not enabled"
error.
* Resolves RIPD-1588
A validator that was configured to use a published validator list could
exhibit aberrent behavior if that validator list expired.
This commit introduces additional logic that makes validators operating
with an expired validator list bow out of the consensus process instead
of continuing to publish validations. Normal operation will resume once
a non-expired validator list becomes available.
This commit also enhances status reporting when using the `server_info`
and `validators` commands. Before, only the expiration time of the list
would be returned; now, its current status is also reported in a format
that is clearer.
A validator that was configured to use a published validator list could
exhibit aberrent behavior if that validator list expired.
This commit introduces additional logic that makes validators operating
with an expired validator list bow out of the consensus process instead
of continuing to publish validations. Normal operation will resume once
a non-expired validator list becomes available.
This commit also enhances status reporting when using the `server_info`
and `validators` commands. Before, only the expiration time of the list
would be returned; now, its current status is also reported in a format
that is clearer.
Problem:
- There are several specific overloads with some custom code that can be
easily replaced using Boost.Hex.
Solution:
- Introduce `strHex(itr, itr)` to return a string given a begin and end
iterator.
- Remove `strHex(itr, size)` in favor of the `strHex(T)` where T is
something that has a `begin()` member function. This allows us to
remove the strHex overloads for `std::string`, Blob, and Slice.
* For example Visual Studio, XCode. This will allow easily working with
any file in the IDE.
* Also ignore the file created by Visual Studio when using cmake
integration.
* Use conditional for unity/nounity sources (h/t @mellery451)
- allow private token for jenkins/codecov
- add custom targets for gcc/clang to generate codecov reports
- use CMake coverage target in jenkins build
- optional coverage_test argument when configuring the build
Reduces the account reserve for a multisigning SignerList from
(conditionally) 3 to 10 OwnerCounts to (unconditionally) 1
OwnerCount. Includes a transition process.
* When increasing the expected ledger size, add on an extra 20%.
* When decreasing the expected ledger size, take the minimum of the
validated ledger size or the old expected size, and subract another 50%.
* Update fee escalation documentation.
* Refactor the FeeMetrics object to use values from Setup
As described in #2314, when an offer executed with `Fill or Kill`
semantics, the server would return `tesSUCCESS` even if the order
couldn't be filled and was aborted. This would require additional
processing of metadata by users to determine the effects of the
transaction.
This commit introduces the `fix1578` amendment which, if enabled,
will cause the server to return the new `tecKILLED` error code
instead of `tesSUCCESS` for `Fill or Kill` orders that could not
be filled.
Additionally, the `fix1578` amendment will prevent the setting of
the `No Ripple` flag on trust lines with negative balance; trying
to set the flag on such a trust line will fail with the new error
code `tecNEGATIVE_BALANCE`.
Fixes: RIPD-1648
- use ExternalProject for snappy, lz4, SOCI, and sqlite3
- use FetchContent for NuDB
- update SOCI from 79e222e3c2278e6108137a2d26d3689418b37544 to
3a1f602b3021b925d38828e3ff95f9e7f8887ff7
- update lz4 from c10863b98e1503af90616ae99725ecd120265dfb to v1.8.2
- update sqlite3 from 3.21 to 3.24
- update snappy from b02bfa754ebf27921d8da3bd2517eab445b84ff9 to 1.1.7
- update NuDB from 00adc6a4f16679a376f40c967f77dfa544c179c1 to 1.0.0
When validators publish a proposal, they include the close time that they
believe the new ledger should have, and the network attempts to reach
consensus on that.
Instead of delaying consensus if no close time has the required majority
the servers can "agree to disagree"; if this happens, they switch to
proposing a close time of 0, and the network avalanches to that value.
If that occurs, determinstic rules record the new ledger's close time as
being one second later than its parent, and set a flag indicating that
no consensus on the close time was reached.
The wire protocol decoder would incorrectly filter such proposals, so
that they would not be seen by the higher level consensus engine.
This commit removes the low-level filtering, and allows higher level
code to filter out stale proposals instead.
In order to facilitate transaction signing, `rippled` offers the `sign` and
`sign_for` and `submit` commands, which, given a seed, can be used to sign or
sign-and-submit transactions. These commands are accessible from the command
line, as well as over the WebSocket and RPC interfaces that `rippled` can be
configured to provide.
These commands, unfortunately, have significant security implications:
1. They require divulging an account's seed (commonly known as a "secret
key") to the server.
2. When executing these commands against remote servers, the seeds can be
transported over clear-text links.
3. When executing these commands over the command line, the account
seed may be visible using common tools that show running processes
and may potentially be inadvertently stored by system monitoring
tools or facilities designed to maintain a history of previously
typed commands.
While this commit cannot prevent users from issuing these commands to a
server, whether locally or remotely, it restricts the `sign` and `sign_for`
commands, as well as the `submit` command when used to sign-and-submit,
so that they require administrative privileges on the server.
Server operators that want to allow unrestricted signing can do so by
adding the following stanza to their configuration file:
[signing_support]
true
Ripple discourages server operators from doing so and advises against using
these commands, which will be removed in a future release. If you rely on
these commands for signing, please migrate to a standalone signing solution
as soon as possible. One option is to use `ripple-lib`; documentation is
available at https://developers.ripple.com/rippleapi-reference.html#sign.
If the commands are administratively enabled, the server includes a warning
on startup and adds a new field in the resulting JSON, informing the caller
that the commands are deprecated and may become unavailable at any time.
Acknowledgements:
Jesper Wallin for reporting this issue to Ripple.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to responsibly
disclose any issues that they may find. For more on Ripple's Bug Bounty
program, please visit: https://ripple.com/bug-bounty
Exclude several libraries from build when we are included in a
super-project (this is the case when someone only wants to use
xrpl_core). Force several target (deprecated) params to be cache
variables since they are now exposed as options.
Increase page size for SQLite transaction database upon creation
Provide diagnostics for transaction db page usage.
Shut down rippled gracefullly if transaction db is running out of pages.
Add new rippled maintenance command line option to cause new page size
to take effect.
Switch to target-oriented dependencies. Use imported targets for
dependencies (openssl, boost). Localize FindBoost to remove cmake
version dependence for latest boost support. Logically separate
"ripple-libpp" core sources and add install targets.
Add ninja build for msvc. Add two clang sanitizer builds. Misc script
changes to work with latest modernized cmake.
Certain versions of the Beast HTTP & WebSocket library can
generate exceptions, which unless caught, will result in
unexpected behavior.
Acknowledgements:
Ripple thanks Thomas Snider for originally noticing this
issue and responsibly disclosing it to Ripple.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers
to responsibly disclose any issues that they may find. For
more on Ripple's Bug Bounty program, please visit:
https://ripple.com/bug-bounty
Based on a TODO comment in DisputedTX.h, it seems at one point the
data members of ConsensusParms were macros. Now that they are not,
we should spell them like other data members (without all uppercase).
After some discussion on https://github.com/ripple/rippled/pull/2595
we have decided that the allocator should not be plumbed through
the KeyCache class template. As such, remove the comment suggesting
to push the allocator through.
All listed validators are trusted and quorum is 80% of trusted
validators regardless of the number of:
* configured published lists
* listed or trusted validators
* recently seen validators
Exceptions:
* A listed validator whose master key has been revoked is not trusted
* Custom minimum quorum (specified with --quorum in the command line)
is used if the normal quorum appears unreachable based on the number
of recently received validators.
RIPD-1640
std::random_shuffle is deprecated in C++14 and removed completely
in C++17. The two-iterator version of std::random_shuffle usually
depends on std::rand and also on a global state. The preferred
replacement is to use std::shuffle with a pseudo-random number
generator.
- Since we require a min Boost version of 1.67 as of recently (for
Beast), we also remove the conditional checks that existed for us
to know whether Boost.Process is available or not. We can
always assume it is available now.
- Remove runtime checks for minimum Boost and OpenSSL versions
since they are checked at CMake configure time.
This changes the rules for payments in two ways:
1) It sets the maximum number of offers any book step can consume from
2000 to 1000.
2) When a strand contains a step that consumes too many offers,
currently the liquidity is not used at all and the strand will
be considered dry. This changes things so the liquidity is used,
however the strand will still be considered dry.
* mFeeDue is only used in one place by one derived class, so
only compute it as a local in that function.
* The baseFee needs to be calculated outside of the Transactor class
because, it can change during transaction processing, and the function
is static, so we need to be sure to call the right version
* Rename Transactor::calculateFee to minimumFee
Fixes: RIPD-1574
Alias beast address classes to the asio equivalents. Adjust users of
address classes accordingly. Fix resolver class so that it can support
ipv6 addresses. Make unit tests use ipv6 localhost network. Extend
endpoint peer message to support string endpoint
representations while also supporting the existing fields (both are
optional/repeated types). Expand test for Livecache and Endpoint.
Workaround some false positive ipaddr tests on windows (asio bug?)
Replaced usage of address::from_string(deprecated) with free function
make_address. Identified a remaining use of v4 address type and
replaced with the more appropriate IPEndpoint type (rpc_ip cmdline
option). Add CLI flag for using ipv4 with unit tests.
Release Notes
-------------
The optional rpc_port command line flag is deprecated. The rpc_ip
parameter now works as documented and accepts ip and port combined.
Fixes: RIPD-1575. Fix argument passing to runner. Allow multiple unit
test selectors to be passed via --unittest argument. Add optional
integer priority value to test suite list. Fix several failing manual
tests. Update CLI usage message to make it clearer.
The lsfDepositAuth flag limits the AccountIDs that can deposit into
the account that has the flag set. The original design only
allowed deposits to complete if the account with the flag set also
signed the transaction that caused the deposit.
The DepositPreauth ledger type allows an account with the
lsfDepositAuth flag set to preauthorize additional accounts.
This preauthorization allows them to sign deposits as well. An
account can add DepositPreauth objects to the ledger (and remove
them as well) using the DepositPreauth transaction.
Add a new invariant checker that verifies that we never charge a
fee higher than specified in the transaction; we will charge less
in some corner cases where the transacting account cannot afford
the fee.
Detect more anomalous conditions, and improve the logged error
messages.
Clarify the code flow associated with invoking the invariant checker
from `Transactor`, add extra comments and improve naming to make the
code self-documenting.
The six different ranges of TER codes are broken up into six
different enumerations. A template class allows subsets of
these enumerations to be aggregated. This technique allows
verification at compile time that no TEC codes are returned
before the signature is checked.
Conversion between TER instance and integer is provided by
named functions. This makes accidental conversion almost
impossible and makes type abuse easier to spot in the code
base.
* UptimeClock is a chrono-compatible seconds-precision clock.
* Like UptimeTimer, its purpose is to make it possible for clients
to query the uptime thousands of times per second without a
significant performance hit.
* UptimeClock decouples itself from LoadManager by managing its
own once-per-second update loop.
* Clients now traffic in chrono time_points and durations instead
of int.
Some classes had virtual methods, but were missing a virtual
destructor.
Technically, every unit test that inherits from the Beast test suite
would get flagged by `-Wnon-virtual-dtor` but I did not think it would
be a great idea to go sprinkle a virtual destructor for every Ripple
test suite.
# No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
# Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
