911c0466c0
* Set version to 2.4.0 * refactor: Remove unused and add missing includes (#5293) The codebase is filled with includes that are unused, and which thus can be removed. At the same time, the files often do not include all headers that contain the definitions used in those files. This change uses clang-format and clang-tidy to clean up the includes, with minor manual intervention to ensure the code compiles on all platforms. * refactor: Calculate numFeatures automatically (#5324) Requiring manual updates of numFeatures is an annoying manual process that is easily forgotten, and leads to frequent merge conflicts. This change takes advantage of the `XRPL_FEATURE` and `XRPL_FIX` macros, and adds a new `XRPL_RETIRE` macro to automatically set `numFeatures`. * refactor: Improve ordering of headers with clang-format (#5343) Removes all manual header groupings from source and header files by leveraging clang-format options. * Rename "deadlock" to "stall" in `LoadManager` (#5341) What the LoadManager class does is stall detection, which is not the same as deadlock detection. In the condition of severe CPU starvation, LoadManager will currently intentionally crash rippled reporting `LogicError: Deadlock detected`. This error message is misleading as the condition being detected is not a deadlock. This change fixes and refactors the code in response. * Adds hub.xrpl-commons.org as a new Bootstrap Cluster (#5263) * fix: Error message for ledger_entry rpc (#5344) Changes the error to `malformedAddress` for `permissioned_domain` in the `ledger_entry` rpc, when the account is not a string. This change makes it more clear to a user what is wrong with their request. * fix: Handle invalid marker parameter in grpc call (#5317) The `end_marker` is used to limit the range of ledger entries to fetch. If `end_marker` is less than `marker`, a crash can occur. This change adds an additional check. * fix: trust line RPC no ripple flag (#5345) The Trustline RPC `no_ripple` flag gets set depending on `lsfDefaultRipple` flag, which is not a flag of a trustline but of the account root. The `lsfDefaultRipple` flag does not provide any insight if this particular trust line has `lsfLowNoRipple` or `lsfHighNoRipple` flag set, so it should not be used here at all. This change simplifies the logic. * refactor: Updates Conan dependencies: RocksDB (#5335) Updates RocksDB to version 9.7.3, the latest version supported in Conan 1.x. A patch for 9.7.4 that fixes a memory leak is included. * fix: Remove null pointer deref, just do abort (#5338) This change removes the existing undefined behavior from `LogicError`, so we can be certain that there will be always a stacktrace. De-referencing a null pointer is an old trick to generate `SIGSEGV`, which would typically also create a stacktrace. However it is also an undefined behaviour and compilers can do something else. A more robust way to create a stacktrace while crashing the program is to use `std::abort`, which we have also used in this location for a long time. If we combine the two, we might not get the expected behaviour - namely, the nullpointer deref followed by `std::abort`, as handled in certain compiler versions may not immediately cause a crash. We have observed stacktrace being wiped instead, and thread put in indeterminate state, then stacktrace created without any useful information. * chore: Add PR number to payload (#5310) This PR adds one more payload field to the libXRPL compatibility check workflow - the PR number itself. * chore: Update link to ripple-binary-codec (#5355) The link to ripple-binary-codec's definitions.json appears to be outdated. The updated link is also documented here: https://xrpl.org/docs/references/protocol/binary-format#definitions-file * Prevent consensus from getting stuck in the establish phase (#5277) - Detects if the consensus process is "stalled". If it is, then we can declare a consensus and end successfully even if we do not have 80% agreement on our proposal. - "Stalled" is defined as: - We have a close time consensus - Each disputed transaction is individually stalled: - It has been in the final "stuck" 95% requirement for at least 2 (avMIN_ROUNDS) "inner rounds" of phaseEstablish, - and either all of the other trusted proposers or this validator, if proposing, have had the same vote(s) for at least 4 (avSTALLED_ROUNDS) "inner rounds", and at least 80% of the validators (including this one, if appropriate) agree about the vote (whether yes or no). - If we have been in the establish phase for more than 10x the previous consensus establish phase's time, then consensus is considered "expired", and we will leave the round, which sends a partial validation (indicating that the node is moving on without validating). Two restrictions avoid prematurely exiting, or having an extended exit in extreme situations. - The 10x time is clamped to be within a range of 15s (ledgerMAX_CONSENSUS) to 120s (ledgerABANDON_CONSENSUS). - If consensus has not had an opportunity to walk through all avalanche states (defined as not going through 8 "inner rounds" of phaseEstablish), then ConsensusState::Expired is treated as ConsensusState::No. - When enough nodes leave the round, any remaining nodes will see they've fallen behind, and move on, too, generally before hitting the timeout. Any validations or partial validations sent during this time will help the consensus process bring the nodes back together. --------- Co-authored-by: Michael Legleux <mlegleux@ripple.com> Co-authored-by: Bart <bthomee@users.noreply.github.com> Co-authored-by: Ed Hennis <ed@ripple.com> Co-authored-by: Bronek Kozicki <brok@incorrekt.com> Co-authored-by: Darius Tumas <Tokeiito@users.noreply.github.com> Co-authored-by: Sergey Kuznetsov <skuznetsov@ripple.com> Co-authored-by: cyan317 <120398799+cindyyan317@users.noreply.github.com> Co-authored-by: Vlad <129996061+vvysokikh1@users.noreply.github.com> Co-authored-by: Alex Kremer <akremer@ripple.com>
Resource::Manager
The ResourceManager module has these responsibilities:
- Uniquely identify endpoints which impose load.
- Track the load used across endpoints.
- Provide an interface to share load information in a cluster.
- Warn and/or disconnect endpoints for imposing load.
Description
To prevent monopolization of server resources or attacks on servers, resource consumption is monitored at each endpoint. When consumption exceeds certain thresholds, costs are imposed. Costs could include charging additional XRP for transactions, requiring a proof of work to be performed, or simply disconnecting the endpoint.
Currently, consumption endpoints include websocket connections used to service clients, and peer connections used to create the peer to peer overlay network implementing the Ripple protocol.
The current "balance" of a Consumer represents resource consumption debt or credit. Debt is accrued when bad loads are imposed. Credit is granted when good loads are imposed. When the balance crosses heuristic thresholds, costs are increased on the endpoint. The balance is represented as a unitless relative quantity. This balance is currently held by the Entry struct in the impl/Entry.h file.
Costs associated with specific transactions are defined in the impl/Fees files.
Although RPC connections consume resources, they are transient and cannot be rate limited. It is advised not to expose RPC interfaces to the general public.
Consumer Types
Consumers are placed into three classifications (as identified by the Resource::Kind enumeration):
- InBound,
- OutBound, and
- Admin
Each caller determines for itself the classification of the Consumer it is creating.
Resource Loading
It is expected that a client will impose a higher load on the server when it first connects: the client may need to catch up on transactions it has missed, or get trust lines, or transfer fees. The Manager must expect this initial peak load, but not allow that high load to continue because over the long term that would unduly stress the server.
If a client places a sustained high load on the server, that client is initially given a warning message. If that high load continues the Manager may tell the heavily loaded server to drop the connection entirely and not allow re-connection for some amount of time.
Each load is monitored by capturing peaks and then decaying those peak values over time: this is implemented by the DecayingSample class.
Gossip
Each server in a cluster creates a list of IP addresses of end points that are imposing a significant load. This list is called Gossip, which is passed to other nodes in that cluster. Gossip helps individual servers in the cluster identify IP addreses that might be unduly loading the entire cluster. Again the recourse of the individual servers is to drop connections to those IP addresses that occur commonly in the gossip.
Access
In rippled, the Application holds a unique instance of Resource::Manager,
which may be retrieved by calling the method
Application::getResourceManager().