The existing code properly parses the network_id parameter from the
the configuration file, but it does not properly set up the code to
use the value correctly. As a result the configured `network_id` is
ignored.
The existing logic involves every server sending every transaction
that it receives to all its peers (except the one that it received
a transaction from).
This commit instead uses a randomized algorithm, where a node will
randomly select peers to relay a given transaction to, caching the
list of transaction hashes that are not relayed and forwading them
to peers once every second. Peers can then determine whether there
are transactions that they have not seen and can request them from
the node which has them.
It is expected that this feature will further reduce the bandwidth
needed to operate a server.
- Add validation/proposal reduce-relay feature negotiation to
the handshake
- Make squelch duration proportional to a number of peers that
can be squelched
- Refactor makeRequest()/makeResponse() to facilitate handshake
unit-testing
- Fix compression enable flag for inbound peer
- Fix compression algorithm parsing in the header parser
- Fix squelch duration in onMessage(TMSquelch)
This commit fixes 3624, fixes 3639 and fixes 3641
- Limit the lifetime of a buffer that was only used in the early
phases of peer connection establishment but which lived on as
long as the peer was active.
- Cache the message used to transfer manifests, so it can be reused
instead of recreated for every peer connection.
- Improve the reading of partial messages by passing a hint to the
I/O layer if the number of bytes needed to complete the message
is known.
The existing code issues a PING to each peer every 8 seconds. While
frequent PINGs allow us to estimate a peer's latency with a high
degree of accuracy, this "inter-server polka dance" is inefficient
and not useful. This commit, if merged, reduces the PING frequency
to once every 60 seconds.
Additionally, this commit simplifies the PING handling logic and
merges the code used to check and disconnect peers which fail to
track the network directly into the timer callback.
When evaluating the fitness and usefulness of an outbound peer, the code
would incorrectly calculate the amount of time that the peer spent in
a non-useful state.
This commit, if merged, corrects the calculation and makes the timeout
values configurable by server operators.
Two new options are introduced in the 'overlay' stanza of the config
file. The default values, in seconds, are:
[overlay]
max_unknown_time = 600
max_diverged_time = 300
With few exceptions, servers will typically receive multiple copies
of any given message from its directly connected peers. For servers
with several peers this can impact the processing latency and force
it to do redundant work. Proposal and validation messages are often
relayed with extremely high redundancy.
This commit, if merged, introduces experimental code that attempts
to optimize the relaying of proposals and validations by allowing
servers to instruct their peers to "squelch" delivery of selected
proposals and validations. Servers making squelching decisions by
a process that evaluates the fitness and performance of a given
server and randomly selecting a subset of the best candidates.
The experimental code is presently disabled and must be explicitly
enabled by server operators that wish to test it.
* Gives a summary of the health of the node:
Healthy, Warning, or Critical
* Last validated ledger age:
<7s is Healthy,
7s to 20s is Warning
> 20s is Critcal
* If amendment blocked, Critical
* Number of peers:
> 7 is Healthy
1 to 7 is Warning
0 is Critical
* server state:
One of full, validating or proposing is Healthy
One of syncing, tracking or connected is Warning
All other states are Critical
* load factor:
<= 100 is Healthy
101 to 999 is Warning
>= 1000 is Critical
* If not Healthy, info field contains data that is considered not
Healthy.
Fixes: #2809
In deciding whether to relay a proposal or validation, a server would
consider whether it was issued by a validator on that server's UNL.
While both trusted proposals and validations were always relayed,
the code prioritized relaying of untrusted proposals over untrusted
validations. While not technically incorrect, validations are
generally more "valuable" because they are required during the
consensus process, whereas proposals are not, strictly, required.
The commit introduces two new configuration options, allowing server
operators to fine-tune the relaying behavior:
The `[relay_proposals]` option controls the relaying behavior for
proposals received by this server. It has two settings: "trusted"
and "all" and the default is "trusted".
The `[relay_validations]` options controls the relaying behavior for
validations received by this server. It has two settings: "trusted"
and "all" and the default is "all".
This change does not require an amendment as it does not affect
transaction processing.
This commit introduces no functional changes but cleans up the
code and shrinks the surface area by removing dead and unused
code, leveraging std:: alternatives to hand-rolled code and
improving comments and documentation.
* Whenever a node downloads a new VL, send it to all peers that
haven't already sent or received it. It also saves it to the
database_dir as a Json text file named "cache." plus the public key of
the list signer. Any files that exist for public keys provided in
[validator_list_keys] will be loaded and processed if any download
from [validator_list_sites] fails or no [validator_list_sites] are
configured.
* Whenever a node receives a broadcast VL message, it treats it as if
it had downloaded it on it's own, broadcasting to other peers as
described above.
* Because nodes normally download the VL once every 5 minutes, a single
node downloading a VL with an updated sequence number could
potentially propagate across a large part of a well-connected network
before any other nodes attempt to download, decreasing the amount of
time that different parts of the network are using different VLs.
* Send all of our current valid VLs to new peers on connection.
This is probably the "noisiest" part of this change, but will give
poorly connected or poorly networked nodes the best chance of syncing
quickly. Nodes which have no http(s) access configured or available
can get a VL with no extra effort.
* Requests on the peer port to the /vl/<pubkey> endpoint will return
that VL in the same JSON format as is used to download now, IF the
node trusts and has a valid instance of that VL.
* Upgrade protocol version to 2.1. VLs will only be sent to 2.1 and
higher nodes.
* Resolves#2953
The 'network_id' option allows an administrator to specify to which
network they intend a server to connect. Servers can leverage this
information to optimize routing and prune automatically discovered
cross-network connections.
This commit will, if merged:
- add support for the devnet keyword, which corresponds to network ID #2;
- report the network ID, if one is configured, in server_info
This commit restructures the HTTP based protocol negotiation that `rippled`
executes and introduces support for negotiation of compression for peer
links which, if implemented, should result in significant bandwidth savings
for some server roles.
This commit also introduces the new `[network_id]` configuration option
that administrators can use to specify which network the server is part of
and intends to join. This makes it possible for servers from different
networks to drop the link early.
The changeset also improves the log messages generated when negotiation
of a peer link upgrade fails. In the past, no useful information would
be logged, making it more difficult for admins to troubleshoot errors.
This commit also fixes RIPD-237 and RIPD-451
The /crawl API endpoint allows developers to examine the structure of
the XRP Ledger's overlay network.
This commit adds additional information about the local server to the
/crawl endpoint, making it possible for developers to create data-rich
network-wide status dashboards.
Related:
- https://developers.ripple.com/peer-protocol.html
- https://github.com/ripple/rippled-network-crawler
Do not process a transaction received from a peer if it has
been processed within the past ten seconds.
Increase the number of transaction handlers that can be in
flight in the job queue and decrease the relative cost for
peers to share transaction and ledger data.
Additionally, make better use of resources by adjusting the
number of threads we initialize, by reverting commit
68b8ffdb63.
Performance counter modifications:
* Create and display counters to track:
1) Pending transaction limit overruns.
2) Total peer disconnections.
3) Peers disconnections due to resource consumption.
Avoid a potential double-free in Json library.
Instead of specifying a static list of trusted validators in the config
or validators file, the configuration can now include trusted validator
list publisher keys.
The trusted validator list and quorum are now reset each consensus
round using the latest validator lists and the list of recent
validations seen. The minimum validation quorum is now only
configurable via the command line.
* Use std::mutex instead of std::recursive_mutex
* Remove unnecessary type alias
* Use std::set instead of ripple::hash_map
* Don't reinvent virtual functions
New classes are introduced to represent HTTP messages and their
associated bodies. The parser interface is reworked to use CRTP,
error codes, and trait checks.
New classes:
* basic_headers
Models field/value pairs in a HTTP message.
* message
Models a HTTP message, body behavior defined by template argument.
Parsed message carries metadata generated during parsing.
* parser
Produces parsed messages.
* empty_body, string_body, basic_streambuf_body
Classes used to represent content bodies in various ways.
New functions:
* read, async_read, write, async_write
Read and write HTTP messages on a socket.
New concepts:
* Body: Represents the HTTP Content-Body.
* Field: A HTTP header field.
* FieldSequence: A forward sequence of fields.
* Reader: Parses a Body from a stream of bytes.
* Writer: Serializes a Body to buffers.
basic_parser changes:
* add write methods which throw exceptions instead
* error_code passed via parameter instead of return value
* fold private member calls into existing callbacks
* basic_parser uses CRTP instead of virtual members
* add documentation on Derived requirements for CRTP
impl/http-parser changes:
* joyent renamed to nodejs to reflect upstream changes
Calculate the number of file descriptors that are needed during
execution based on the configuration file, with a hard floor
of 1024, adjusting the limit if possible. Refuse to run if enough
fds are not available.
Additionally, allow administrators to limit the number of incoming
connections a configured port will accept. By default no limit is
imposed.
The RippleAddress class was used to represent a number of fundamentally
different types: account public keys, account secret keys, node public
keys, node secret keys, seeds and generators.
The class is replaced by the following types:
* PublicKey for account and node public keys
* SecretKey for account and node private keys
* Generator for generating secp256k1 accounts
* Seed for account, node and generator seeds
* Remove cxx14 compatibility layer from ripple
* Update travis to clang 3.6 and drop gcc 4.8
* Remove unneeded beast CXX14 defines
* Do not run clang build with gdb with travis
* Update circle ci to clang 3.6 & gcc-5
* Don't run rippled in gdb, clang builds crash gdb
* Staticly link libstdc++, boost, ssl, & protobuf
* Support builds on ubuntu 15.10
