mirror of
https://github.com/XRPLF/rippled.git
synced 2025-11-19 10:35:50 +00:00
e919a25ecb
* Fix: Don't flag consensus as stalled prematurely (#5658) Fix stalled consensus detection to prevent false positives in situations where there are no disputed transactions. Stalled consensus detection was added to 2.5.0 in response to a network consensus halt that caused a round to run for over an hour. However, it has a flaw that makes it very easy to have false positives. Those false positives are usually mitigated by other checks that prevent them from having an effect, but there have been several instances of validators "running ahead" because there are circumstances where the other checks are "successful", allowing the stall state to be checked. * Set version to 2.5.1 * fix: Skip processing transaction batch if the batch is empty (#5670) Avoids an assertion failure in NetworkOPsImp::apply in the unlikely event that all incoming transactions are invalid. * Fix: EscrowTokenV1 (#5571) * resolves an accounting inconsistency in MPT escrows where transfer fees were not properly handled when unlocking escrowed tokens. * refactor: Wrap GitHub CI conditionals in curly braces (#5796) This change wraps all GitHub conditionals in `${{ .. }}`, both for consistency and to reduce unexpected failures, because it was previously noticed that not all conditionals work without those curly braces. * Only notify clio for PRs targeting the release and master branches (#5794) Clio should only be notified when releases are about to be made, instead of for all PR, so this change only notifies Clio when a PR targets the release or master branch. * Support DynamicMPT XLS-94d (#5705) * extends the functionality of the MPTokenIssuanceSet transaction, allowing the issuer to update fields or flags that were explicitly marked as mutable during creation. * Bugfix: Adds graceful peer disconnection (#5669) The XRPL establishes connections in three stages: first a TCP connection, then a TLS/SSL handshake to secure the connection, and finally an upgrade to the bespoke XRP Ledger peer-to-peer protocol. During connection termination, xrpld directly closes the TCP connection, bypassing the TLS/SSL shutdown handshake. This makes peer disconnection diagnostics more difficult - abrupt TCP termination appears as if the peer crashed rather than disconnected gracefully. This change refactors the connection lifecycle with the following changes: - Enhanced outgoing connection logic with granular timeouts for each connection stage (TCP, TLS, XRPL handshake) to improve diagnostic capabilities - Updated both PeerImp and ConnectAttempt to use proper asynchronous TLS shutdown procedures for graceful connection termination * Downgrade to boost 1.83 * Set version to 2.6.1-rc1 * chore: Use self hosted windows runners (#5780) This changes switches from the GitHub-managed Windows runners to self-hosted runners to significantly reduce build time. * Rename mutable flags (#5797) This is a minor change on top of #5705 * fix(amendment): Add missing fields for keylets to ledger objects (#5646) This change adds a fix amendment (`fixIncludeKeyletFields`) that adds: * `sfSequence` to `Escrow` and `PayChannel` * `sfOwner` to `SignerList` * `sfOracleDocumentID` to `Oracle` This ensures that all ledger entries hold all the information needed to determine their keylet. * chore: Limits CI build and test parallelism to reduce resource contention (#5799) GitHub runners have a limit on how many concurrent jobs they can actually process (even though they will try to run them all at the same time), and similarly the Conan remote cannot handle hundreds of concurrent requests. Previously, the Conan dependency uploading was already limited to max 10 jobs running in parallel, and this change makes the same change to the build+test workflow. * chore: Build and test all configs for daily scheduled run (#5801) This change re-enables building and testing all configurations, but only for the daily scheduled run. Previously all configurations were run for each merge into the develop branch, but that overwhelmed both the GitHub runners and the Conan remote, and thus they were limited to just a subset of configurations. Now that the number of jobs is limited via `max-parallel: 10`, we should be able to safely enable building all configurations again. However, building them all once a day instead of for each PR merge should be sufficient. * chore: Add unit tests dir to code coverage excludes (#5803) This change excludes unit test code from code coverage reporting. * refactor: Modularise ledger (#5493) This change moves the ledger code to libxrpl. * Mark PermissionDelegation as unsupported * Set version to 2.6.1-rc2 * Miscellaneous refactors and updates (#5590) - Added a new Invariant: `ValidPseudoAccounts` which checks that all pseudo-accounts behave consistently through creation and updates, and that no "real" accounts look like pseudo-accounts (which means they don't have a 0 sequence). - `to_short_string(base_uint)`. Like `to_string`, but only returns the first 8 characters. (Similar to how a git commit ID can be abbreviated.) Used as a wrapped sink to prefix most transaction-related messages. More can be added later. - `XRPL_ASSERT_PARTS`. Convenience wrapper for `XRPL_ASSERT`, which takes the `function` and `description` as separate parameters. - `SField::sMD_PseudoAccount`. Metadata option for `SField` definitions to indicate that the field, if set in an `AccountRoot` indicates that account is a pseudo-account. Removes the need for hard-coded field lists all over the place. Added the flag to `AMMID` and `VaultID`. - Added functionality to `SField` ctor to detect both code and name collisions using asserts. And require all SFields to have a name - Convenience type aliases `STLedgerEntry::const_pointer` and `STLedgerEntry::const_ref`. (`SLE` is an alias to `STLedgerEntry`.) - Generalized `feeunit.h` (`TaggedFee`) into `unit.h` (`ValueUnit`) and added new "BIPS"-related tags for future use. Also refactored the type restrictions to use Concepts. - Restructured `transactions.macro` to do two big things 1. Include the `#include` directives for transactor header files directly in the macro file. Removes the need to update `applySteps.cpp` and the resulting conflicts. 2. Added a `privileges` parameter to the `TRANSACTION` macro, which specifies some of the operations a transaction is allowed to do. These `privileges` are enforced by invariant checks. Again, removed the need to update scattered lists of transaction types in various checks. - Unit tests: 1. Moved more helper functions into `TestHelpers.h` and `.cpp`. 2. Cleaned up the namespaces to prevent / mitigate random collisions and ambiguous symbols, particularly in unity builds. 3. Generalized `Env::balance` to add support for `MPTIssue` and `Asset`. 4. Added a set of helper classes to simplify `Env` transaction parameter classes: `JTxField`, `JTxFieldWrapper`, and a bunch of classes derived or aliased from it. For an example of how awesome it is, check the changes `src/test/jtx/escrow.h` for how much simpler the definitions are for `finish_time`, `cancel_time`, `condition`, and `fulfillment`. 5. Generalized several of the amount-related helper classes to understand `Asset`s. 6. `env.balance` for an MPT issuer will return a negative number (or 0) for consistency with IOUs. * refactor: Simplify STParsedJSON with some helper functions (#5591) - Add code coverage for STParsedJSON edge cases Co-authored-by: Denis Angell <dangell@transia.co> * test: Add STInteger and STParsedJSON tests (#5726) This change is to improve code coverage (and to simplify #5720 and #5725); there is otherwise no change in functionality. The change adds basic tests for `STInteger` and `STParsedJSON`, so it becomes easier to test smaller changes to the types, as well as removes `STParsedJSONArray`, since it is not used anywhere (including in Clio). * Revert "Update Conan dependencies: OpenSSL" (#5807) This change reverts #5617, because it will require extensive testing that will take up more time than we have before the next scheduled release. Reverting this change does not mean we are abandoning it. We aim to pick it back up once there's a sufficient time window to allow for testing on multiple distros running a mixture of OpenSSL 1.x and 3.x. * docs: Add warning about using std::counting_semaphore (#5595) This adds a comment to avoid using `std::counting_semaphore` until the minimum compiler versions of GCC and Clang have been updated to no longer contain the bug that is present in older compilers. * Improve ValidatorList invalid UNL manifest logging (#5804) This change raises logging severity from `INFO` to `WARN` when handling UNL manifest signed with an unexpected / invalid key. It also changes the internal error code for an invalid format of UNL manifest to `invalid` (from `untrusted`). This is a follow up to problems experienced by an UNL node due to old manifest key configured in `validators.txt`, which would be easier to diagnose with improved logging. It also replaces a log line with `UNREACHABLE` for an impossible situation when we match UNL manifest key against a configured key which has an invalid type (we cannot configure such a key because of checks when loading configured keys). * chore: Pin all CI Docker tags (#5813) To avoid surprises and ensure reproducibility, this change pins all CI Docker image tags to the latest version in the XRPLF/CI repo. * change `fixPriceOracleOrder` to `Supported::yes` (#5749) * fix: Address http header case sensitivity (#5767) This change makes the regex in `HttpClient.cpp` that matches the content-length http header case insensitive to improve compatibility, as http headers are case insensitive. * test: add more comprehensive tests for `FeeVote` (#5746) This change adds more comprehensive tests for the `FeeVote` module, which previously only checked the basics, and not the more comprehensive flows in that class. * ci: Call all reusable workflows reusable (#5818) * Add `STInt32` as a new `SType` (#5788) This change adds `STInt32` as a new `SType` under the `STInteger` umbrella, with `SType` value `12`. This is the first and only `STInteger` type that supports negative values. * switch `fixIncludeKeyletFields` to `Supported::yes` (#5819) * refactor: Restructure Transactor::preflight to reduce boilerplate (#5592) * Restructures `Transactor::preflight` to create several functions that will remove the need for error-prone boilerplate code in derived classes' implementations of `preflight`. * refactor: Add support for extra transaction signatures (#5594) * Restructures Transactor signature checking code to be able to handle a `sigObject`, which may be the full transaction, or may be an object field containing a separate signature. Either way, the `sigObject` can be a single- or multi-sign signature. * ci: Upload artifacts during build and test in a separate job (#5817) * chore: Set free-form CI inputs as env vars (#5822) This change moves CI values that could be user-provided into environment variables. * Rename flags for DynamicMPT (#5820) * Set version to 2.6.1 * fix: FD/handle guarding + exponential backoff (#5823) * fix: Transaction sig checking functions do not get a full context (#5829) Fixes a (currently harmless) bug introduced by PR #5594 * Remove bogus coverage warning (#5838) * fix return type --------- Co-authored-by: Ed Hennis <ed@ripple.com> Co-authored-by: Jingchen <a1q123456@users.noreply.github.com> Co-authored-by: Denis Angell <dangell@transia.co> Co-authored-by: Bart <bthomee@users.noreply.github.com> Co-authored-by: yinyiqian1 <yqian@ripple.com> Co-authored-by: Vito Tumas <5780819+Tapanito@users.noreply.github.com> Co-authored-by: Bronek Kozicki <brok@incorrekt.com> Co-authored-by: Mayukha Vadari <mvadari@ripple.com> Co-authored-by: Valentin Balaschenko <13349202+vlntb@users.noreply.github.com> Co-authored-by: tequ <git@tequ.dev> Co-authored-by: Ayaz Salikhov <mathbunnyru@users.noreply.github.com>
Levelization
Levelization is the term used to describe efforts to prevent rippled from having or creating cyclic dependencies.
rippled code is organized into directories under src/rippled (and
src/test) representing modules. The modules are intended to be
organized into "tiers" or "levels" such that a module from one level can
only include code from lower levels. Additionally, a module
in one level should never include code in an impl folder of any level
other than it's own.
Unfortunately, over time, enforcement of levelization has been inconsistent, so the current state of the code doesn't necessarily reflect these rules. Whenever possible, developers should refactor any levelization violations they find (by moving files or individual classes). At the very least, don't make things worse.
The table below summarizes the desired division of modules, based on the state of the rippled code when it was created. The levels are numbered from the bottom up with the lower level, lower numbered, more independent modules listed first, and the higher level, higher numbered modules with more dependencies listed later.
tl;dr: The modules listed first are more independent than the modules listed later.
| Level / Tier | Module(s) |
|---|---|
| 01 | ripple/beast ripple/unity |
| 02 | ripple/basics |
| 03 | ripple/json ripple/crypto |
| 04 | ripple/protocol |
| 05 | ripple/core ripple/conditions ripple/consensus ripple/resource ripple/server |
| 06 | ripple/peerfinder ripple/ledger ripple/nodestore ripple/net |
| 07 | ripple/shamap ripple/overlay |
| 08 | ripple/app |
| 09 | ripple/rpc |
| 10 | ripple/perflog |
| 11 | test/jtx test/beast test/csf |
| 12 | test/unit_test |
| 13 | test/crypto test/conditions test/json test/resource test/shamap test/peerfinder test/basics test/overlay |
| 14 | test |
| 15 | test/net test/protocol test/ledger test/consensus test/core test/server test/nodestore |
| 16 | test/rpc test/app |
(Note that test levelization is much less important and much less
strictly enforced than ripple levelization, other than the requirement
that test code should never be included in ripple code.)
Validation
The levelization script takes no parameters,
reads no environment variables, and can be run from any directory,
as long as it is in the expected location in the rippled repo.
It can be run at any time from within a checked out repo, and will
do an analysis of all the #includes in
the rippled source. The only caveat is that it runs much slower
under Windows than in Linux. It hasn't yet been tested under MacOS.
It generates many files of results:
rawincludes.txt: The raw dump of the#includespaths.txt: A second dump grouping the source module to the destination module, deduped, and with frequency counts.includes/: A directory where each file represents a module and contains a list of modules and counts that the module includes.includedby/: Similar toincludes/, but the other way around. Each file represents a module and contains a list of modules and counts that include the module.loops.txt: A list of direct loops detected between modules as they actually exist, as opposed to how they are desired as described above. In a perfect repo, this file will be empty. This file is committed to the repo, and is used by the levelization Github workflow to validate that nothing changed.ordering.txt: A list showing relationships between modules where there are no loops as they actually exist, as opposed to how they are desired as described above. This file is committed to the repo, and is used by the levelization Github workflow to validate that nothing changed.levelization.ymlGithub Actions workflow to test that levelization loops haven't changed. Unfortunately, if changes are detected, it can't tell if they are improvements or not, so if you have resolved any issues or done anything else to improve levelization, runlevelization.sh, and commit the updated results.
The loops.txt and ordering.txt files relate the modules
using comparison signs, which indicate the number of times each
module is included in the other.
A > Bmeans that A should probably be at a higher level than B, because B is included in A significantly more than A is included in B. These results can be included in bothloops.txtandordering.txt. Becauseordering.txtonly includes relationships where B is not included in A at all, it will only include these types of results.A ~= Bmeans that A and B are included in each other a different number of times, but the values are so close that the script can't definitively say that one should be above the other. These results will only be included inloops.txt.A == Bmeans that A and B include each other the same number of times, so the script has no clue which should be higher. These results will only be included inloops.txt.
The committed files hide the detailed values intentionally, to prevent false alarms and merging issues, and because it's easy to get those details locally.
- Run
levelization.sh - Grep the modules in
paths.txt.- For example, if a cycle is found
A ~= B, simplygrep -w A .github/scripts/levelization/results/paths.txt | grep -w B
- For example, if a cycle is found