Log exception messages at several locations.
Previously, these were locations where an exception was caught, but the
exception message was not logged. Logging the exception messages can be
useful for analysis or debugging. The additional logging could have a
small negative performance impact.
Fix#3213.
This commit combines the `apply_mutex` and `read_mutex` into a single `mutex_`
var. This new `mutex_` var is a `shared_mutex`, and most operations only need to
lock it with a `shared_lock`. The only exception is `applyMutex`, which may need
a `unique_lock`.
One consequence of removing the `apply_mutex` is more than one `applyMutex`
function can run at the same time. To help reduce lock contention that a
`unique_lock` would cause, checks that only require reading data are run a
`shared_lock` (call these the "prewriteChecks"), then the lock is released, then
a `unique_lock` is acquired. Since a currently running `applyManifest` may write
data between the time a `shared_lock` is released and the `write_lock` is
acquired, the "prewriteChecks" need to be rerun. Duplicating this work isn't
ideal, but the "prewirteChecks" are relatively inexpensive.
A couple of other designs were considered. We could restrict more than one
`applyMutex` function from running concurrently - either with a `applyMutex` or
my setting the max number of manifest jobs on the job queue to one. The biggest
issue with this is if any other function ever adds a write lock for any reason,
`applyManifest` would not be broken - data could be written between the release
of the `shared_lock` and the acquisition of the `unique_lock`. Note: it is
tempting to solve this problem by not releasing the `shared_mutex` and simply
upgrading the lock. In the presence of concurrently running `applyManifest`
functions, this will deadlock (both function need to wait for the other to
release their read locks before they can acquire a write lock).
This commit expands the detection capabilities of the Byzantine
validation detector. Prior to this commit, only validators that
were on a server's UNL were monitored. Now, all the validations
that a server receives are passed through the detector.
- Simplify and consolidate code for parsing hex input.
- Replace beast::endian::order with boost::endian::order.
- Simplify CountedObject code.
- Remove pre-C++17 workarounds in favor of C++17 based solutions.
- Improve `base_uint` and simplify its hex-parsing interface by
consolidating the `SexHex` and `SetHexExact` methods into one
API: `parseHex` which forces callers to verify the result of
the operation; as a result some public-facing API endpoints
may now return errors when passed values that were previously
accepted.
- Remove the simple fallback implementations of SHA2 and RIPEMD
introduced to reduce our dependency on OpenSSL. The code is
slow and rarely, if ever, exercised and we rely on OpenSSL
functionality for Boost.ASIO as well.
- Limit the lifetime of a buffer that was only used in the early
phases of peer connection establishment but which lived on as
long as the peer was active.
- Cache the message used to transfer manifests, so it can be reused
instead of recreated for every peer connection.
- Improve the reading of partial messages by passing a hint to the
I/O layer if the number of bytes needed to complete the message
is known.
This commit introduces a new configuration option that server
operators can set. The value is communicated to other servers
and is also reported via the `server_info` API.
The value is meant to allow third-party applications or tools
to group servers together. For example, a tool that visualizes
the network's topology can group servers together.
Similar to the "Domain" field in validator manifests, an operator
can claim any domain. Prior to relying on the value returned, the
domain should be verified by retrieving the xrp-ledger.toml file
from the domain and looking for the server's public key in the
`nodes` array.
A deliberately malformed token can cause the server to crash during
startup. This is not remotely exploitable and would require someone
with access to the configuration file of the server to make changes
and then restart the server.
Acknowledgements:
Guido Vranken for responsibly disclosing this issue.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to
responsibly disclose any issues they may find.
Ripple is generously sponsoring a bug bounty program for the
rippled project. For more information please visit:
https://ripple.com/bug-bounty
Manifests which are revoked can include ephemeral keys although doing
so does not make sense: a revoked manifest isn't used for signing and
so don't need to define an ephemeral key.
Formerly an SOTemplate was default constructed and its elements
added using push_back(). This left open the possibility of a
malformed SOTemplate if adding one of the elements caused a throw.
With this commit the SOTemplate requires an initializer_list of
its elements at construction. Elements may not be added after
construction. With this approach either the SOTemplate is fully
constructed with all of its elements or the constructor throws,
which prevents an invalid SOTemplate from even existing.
This change requires all SOTemplate construction to be adjusted
at the call site. Those changes are also in this commit.
The SOE_Flags enum is also renamed to SOEStyle, which harmonizes
the name with other uses in the code base. SOEStyle elements
are renamed (slightly) to have an "soe" prefix rather than "SOE_".
This heads toward reserving identifiers with all upper case for
macros. The new style also aligns with other prominent enums in
the code base like the collection of TER identifiers.
SOElement is adjusted so it can be stored directly in an STL
container, rather than requiring storage in a unique_ptr.
Correspondingly, unique_ptr usage is removed from both
SOTemplate and KnownFormats.
The new 'Domain' field allows validator operators to associate a domain
name with their manifest in a transparent and independently verifiable
fashion.
It is important to point out that while this system can cryptographically
prove that a particular validator claims to be associated with a domain
it does *NOT* prove that the validator is, actually, associated with that
domain.
Domain owners will have to cryptographically attest to operating particular
validators that claim to be associated with that domain. One option for
doing so would be by making available a file over HTTPS under the domain
being claimed, which is verified separately (e.g. by ensuring that the
certificate used to serve the file matches the domain being claimed) and
which contains the long-term master public keys of validator(s) associated
with that domain.
Credit for an early prototype of this idea goes to GitHub user @cryptobrad
who introduced a PR that would allow a validator list publisher to attest
that a particular validator was associated with a domain. The idea may be
worth revisiting as a way of verifying the domain name claimed by the
validator's operator.
* RIPD-1617, RIPD-1619, RIPD-1621:
Verify serialized public keys more strictly before
using them.
* RIPD-1618:
* Simplify the base58 decoder logic.
* Reduce the complexity of the base58 encoder and
eliminate a potential out-of-bounds memory access.
* Improve type safety by using an `enum class` to
enforce strict type checking for token types.
* RIPD-1616:
Avoid calling `memcpy` with a null pointer even if the
size is specified as zero, since it results in undefined
behavior.
Acknowledgements:
Ripple thanks Guido Vranken for responsibly disclosing these
issues.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers
to responsibly disclose any issues that they may find. For
more on Ripple's Bug Bounty program, please visit:
https://ripple.com/bug-bounty
Allow manifest revoking validator keys to be stored in a separate
[validator_key_revocation] config field, so the validator can run
again with new keys and token.
Instead of specifying a static list of trusted validators in the config
or validators file, the configuration can now include trusted validator
list publisher keys.
The trusted validator list and quorum are now reset each consensus
round using the latest validator lists and the list of recent
validations seen. The minimum validation quorum is now only
configurable via the command line.
