* Peers negotiate compression via HTTP Header "X-Offer-Compression: lz4"
* Messages greater than 70 bytes and protocol type messages MANIFESTS,
ENDPOINTS, TRANSACTION, GET_LEDGER, LEDGER_DATA, GET_OBJECT,
and VALIDATORLIST are compressed
* If the compressed message is larger than the uncompressed message
then the uncompressed message is sent
* Compression flag and the compression algorithm type are included
in the message header
* Only LZ4 block compression is currently supported
* Make ShardArchiveHandler a singleton.
* Add state database for ShardArchiveHandler.
* Use temporary database for SSLHTTPDownloader downloads.
* Make ShardArchiveHandler a Stoppable class.
* Automatically resume interrupted downloads at server start.
* Reduce lock scope on all public functions
* Use TaskQueue to process shard finalization in separate thread
* Store shard last ledger hash and other info in backend
* Use temp SQLite DB versus control file when acquiring
* Remove boost serialization from cmake files
Prior to this commit, the queue and execution times for individual jobs
were reported indepedently and could, potentially, be out of sync. This
change reports both values when either one of the exceeds the reporting
threshold.
Treat all `#` characters in config files as comments (and remove)
*unless* the `#` is immediately preceded by `\`. Write a warning
to log file when trailing comments are found/ignored in the config
to let operators know that the treatment of trailing `#` has changed.
Fixes#3121
The `node_size` configuration option is used to automatically
configure various parameters (cache sizes, timeouts, etc) for
the server.
A previous commit included changes that caused incorrect values
to be returned which can result in sub-optimal performance that
can manifest as difficulty syncing to the network, or increased
disk I/O and/or memory usage. The problem was introduced with
commit 66fad62e66.
This commit, if merged, fixes the code to ensure that the correct
values are returned and introduces a compile-time check to prevent
this issue from reoccurring.
The existing platform detection code was derived from the old Beast
library, which was, itself, derived from JUCE.
This commit removes that code and replaces it with the Boost.Predef
library which defines a consistent set of compiler, architecture,
operating system, library, and other version numbers.
For more on Boost.Predef, please see the Boost documentation. The
documentation for the current version as of this writing is at:
https://www.boost.org/doc/libs/1_71_0/doc/html/predef.html
* replace boost::beast::detail::iequals with boost::iequals
* replace deprecated `buffers` function with `make_printable`
* replace boost::beast::detail::ascii_tolower with lambda
* add missing includes
This commit allows server operators to reserve slots for specific
peers (identified by the peer's public node identity) and to make
changes to the reservations while the server is operating.
This commit closes#2938
This PR addresses a problem where the server could hang indefinitely
on shutdown. The cause of the problem is the SNTPClock class was not
binding the socket to an endpoint on initialization. This can create
an error sent to the read handler. Unfortunately, the handler ignores
the error, reads again and enters into a loop preventing the
io_service from ever completing.
If the number of peers a server has is below the configured
minimum peer limit, this commit will properly transition the
server's state to "disconnected".
The default limit for the minimum number of peers required was
0 meaning that a server that was connected but lost all its
peers would never transition to disconnected, since it could
never drop below zero peers.
This commit redefines the default minimum number of peers to 1
and produces a warning if the server is configured in a way
that will prevent it from ever achieving sufficient connectivity.
* Adds local file:// URL support to the [validator_list_sites] stanza.
The file:// URL must not contain a hostname. Allows a rippled node
operator to "sideload" a new list if their node is unable to reach
a validator list's web site before an old list expires. Lists
loaded from a file will be validated in the same way a downloaded
list is validated.
* Generalize file/dir "guards" from Config test so they can be reused
in other tests.
* Check for error when reading validators.txt. Saves some parsing and
checking of an empty string, and will give a more meaningful error.
* Completes RIPD-1674.
The WaitableEvent class was a leftover from the pre-Boost
version of Beast and used Windows- and pthread-specific
APIs.
This refactor replaces that functionality by using only
interfaces provided by the C++ standard, making the code
more portable.
Closes#2402.
Fixes: RIPD-1648
- use ExternalProject for snappy, lz4, SOCI, and sqlite3
- use FetchContent for NuDB
- update SOCI from 79e222e3c2278e6108137a2d26d3689418b37544 to
3a1f602b3021b925d38828e3ff95f9e7f8887ff7
- update lz4 from c10863b98e1503af90616ae99725ecd120265dfb to v1.8.2
- update sqlite3 from 3.21 to 3.24
- update snappy from b02bfa754ebf27921d8da3bd2517eab445b84ff9 to 1.1.7
- update NuDB from 00adc6a4f16679a376f40c967f77dfa544c179c1 to 1.0.0
In order to facilitate transaction signing, `rippled` offers the `sign` and
`sign_for` and `submit` commands, which, given a seed, can be used to sign or
sign-and-submit transactions. These commands are accessible from the command
line, as well as over the WebSocket and RPC interfaces that `rippled` can be
configured to provide.
These commands, unfortunately, have significant security implications:
1. They require divulging an account's seed (commonly known as a "secret
key") to the server.
2. When executing these commands against remote servers, the seeds can be
transported over clear-text links.
3. When executing these commands over the command line, the account
seed may be visible using common tools that show running processes
and may potentially be inadvertently stored by system monitoring
tools or facilities designed to maintain a history of previously
typed commands.
While this commit cannot prevent users from issuing these commands to a
server, whether locally or remotely, it restricts the `sign` and `sign_for`
commands, as well as the `submit` command when used to sign-and-submit,
so that they require administrative privileges on the server.
Server operators that want to allow unrestricted signing can do so by
adding the following stanza to their configuration file:
[signing_support]
true
Ripple discourages server operators from doing so and advises against using
these commands, which will be removed in a future release. If you rely on
these commands for signing, please migrate to a standalone signing solution
as soon as possible. One option is to use `ripple-lib`; documentation is
available at https://developers.ripple.com/rippleapi-reference.html#sign.
If the commands are administratively enabled, the server includes a warning
on startup and adds a new field in the resulting JSON, informing the caller
that the commands are deprecated and may become unavailable at any time.
Acknowledgements:
Jesper Wallin for reporting this issue to Ripple.
Bug Bounties and Responsible Disclosures:
We welcome reviews of the rippled code and urge researchers to responsibly
disclose any issues that they may find. For more on Ripple's Bug Bounty
program, please visit: https://ripple.com/bug-bounty
Fixes: RIPD-1574
Alias beast address classes to the asio equivalents. Adjust users of
address classes accordingly. Fix resolver class so that it can support
ipv6 addresses. Make unit tests use ipv6 localhost network. Extend
endpoint peer message to support string endpoint
representations while also supporting the existing fields (both are
optional/repeated types). Expand test for Livecache and Endpoint.
Workaround some false positive ipaddr tests on windows (asio bug?)
Replaced usage of address::from_string(deprecated) with free function
make_address. Identified a remaining use of v4 address type and
replaced with the more appropriate IPEndpoint type (rpc_ip cmdline
option). Add CLI flag for using ipv4 with unit tests.
Release Notes
-------------
The optional rpc_port command line flag is deprecated. The rpc_ip
parameter now works as documented and accepts ip and port combined.
* UptimeClock is a chrono-compatible seconds-precision clock.
* Like UptimeTimer, its purpose is to make it possible for clients
to query the uptime thousands of times per second without a
significant performance hit.
* UptimeClock decouples itself from LoadManager by managing its
own once-per-second update loop.
* Clients now traffic in chrono time_points and durations instead
of int.
Some classes had virtual methods, but were missing a virtual
destructor.
Technically, every unit test that inherits from the Beast test suite
would get flagged by `-Wnon-virtual-dtor` but I did not think it would
be a great idea to go sprinkle a virtual destructor for every Ripple
test suite.
* Tally and duration counters for Job Queue tasks and RPC calls
optionally rendered by server_info and server_state, and
optionally printed to a distinct log file.
- Tally each Job Queue task as it is queued, starts, and
finishes running. Track total duration queued and running.
- Tally each RPC call as it starts and either finishes
successfully or throws an exception. Track total running
duration for each.
* Track currently executing Job Queue tasks and RPC methods
along with durations.
* Json-formatted performance log file written by a dedicated
thread, for above-described data.
* New optional parameter, "counters", for server_info and
server_state. If set, render Job Queue and RPC call counters
as well as currently executing tasks.
* New configuration section, "[perf]", to optionally control
performance logging to a file.
* Support optional sub-second periods when rendering human-readable
time points.
* The compiler can provide many non-explicit constructors for
aggregate types. This is sometimes desired, but it can
happen accidentally, resulting in run-time errors.
* This commit assures that no types are aggregates unless existing
code is using aggregate initialization.
