* Fix: Don't flag consensus as stalled prematurely (#5658)
Fix stalled consensus detection to prevent false positives in situations where there are no disputed transactions.
Stalled consensus detection was added to 2.5.0 in response to a network consensus halt that caused a round to run for over an hour. However, it has a flaw that makes it very easy to have false positives. Those false positives are usually mitigated by other checks that prevent them from having an effect, but there have been several instances of validators "running ahead" because there are circumstances where the other checks are "successful", allowing the stall state to be checked.
* Set version to 2.5.1
* fix: Skip processing transaction batch if the batch is empty (#5670)
Avoids an assertion failure in NetworkOPsImp::apply in the unlikely event that all incoming transactions are invalid.
* Fix: EscrowTokenV1 (#5571)
* resolves an accounting inconsistency in MPT escrows where transfer fees were not properly handled when unlocking escrowed tokens.
* refactor: Wrap GitHub CI conditionals in curly braces (#5796)
This change wraps all GitHub conditionals in `${{ .. }}`, both for consistency and to reduce unexpected failures, because it was previously noticed that not all conditionals work without those curly braces.
* Only notify clio for PRs targeting the release and master branches (#5794)
Clio should only be notified when releases are about to be made, instead of for all PR, so this change only notifies Clio when a PR targets the release or master branch.
* Support DynamicMPT XLS-94d (#5705)
* extends the functionality of the MPTokenIssuanceSet transaction, allowing the issuer to update fields or flags that were explicitly marked as mutable during creation.
* Bugfix: Adds graceful peer disconnection (#5669)
The XRPL establishes connections in three stages: first a TCP connection, then a TLS/SSL handshake to secure the connection, and finally an upgrade to the bespoke XRP Ledger peer-to-peer protocol. During connection termination, xrpld directly closes the TCP connection, bypassing the TLS/SSL shutdown handshake. This makes peer disconnection diagnostics more difficult - abrupt TCP termination appears as if the peer crashed rather than disconnected gracefully.
This change refactors the connection lifecycle with the following changes:
- Enhanced outgoing connection logic with granular timeouts for each connection stage (TCP, TLS, XRPL handshake) to improve diagnostic capabilities
- Updated both PeerImp and ConnectAttempt to use proper asynchronous TLS shutdown procedures for graceful connection termination
* Downgrade to boost 1.83
* Set version to 2.6.1-rc1
* chore: Use self hosted windows runners (#5780)
This changes switches from the GitHub-managed Windows runners to self-hosted runners to significantly reduce build time.
* Rename mutable flags (#5797)
This is a minor change on top of #5705
* fix(amendment): Add missing fields for keylets to ledger objects (#5646)
This change adds a fix amendment (`fixIncludeKeyletFields`) that adds:
* `sfSequence` to `Escrow` and `PayChannel`
* `sfOwner` to `SignerList`
* `sfOracleDocumentID` to `Oracle`
This ensures that all ledger entries hold all the information needed to determine their keylet.
* chore: Limits CI build and test parallelism to reduce resource contention (#5799)
GitHub runners have a limit on how many concurrent jobs they can actually process (even though they will try to run them all at the same time), and similarly the Conan remote cannot handle hundreds of concurrent requests. Previously, the Conan dependency uploading was already limited to max 10 jobs running in parallel, and this change makes the same change to the build+test workflow.
* chore: Build and test all configs for daily scheduled run (#5801)
This change re-enables building and testing all configurations, but only for the daily scheduled run. Previously all configurations were run for each merge into the develop branch, but that overwhelmed both the GitHub runners and the Conan remote, and thus they were limited to just a subset of configurations. Now that the number of jobs is limited via `max-parallel: 10`, we should be able to safely enable building all configurations again. However, building them all once a day instead of for each PR merge should be sufficient.
* chore: Add unit tests dir to code coverage excludes (#5803)
This change excludes unit test code from code coverage reporting.
* refactor: Modularise ledger (#5493)
This change moves the ledger code to libxrpl.
* Mark PermissionDelegation as unsupported
* Set version to 2.6.1-rc2
* Miscellaneous refactors and updates (#5590)
- Added a new Invariant: `ValidPseudoAccounts` which checks that all pseudo-accounts behave consistently through creation and updates, and that no "real" accounts look like pseudo-accounts (which means they don't have a 0 sequence).
- `to_short_string(base_uint)`. Like `to_string`, but only returns the first 8 characters. (Similar to how a git commit ID can be abbreviated.) Used as a wrapped sink to prefix most transaction-related messages. More can be added later.
- `XRPL_ASSERT_PARTS`. Convenience wrapper for `XRPL_ASSERT`, which takes the `function` and `description` as separate parameters.
- `SField::sMD_PseudoAccount`. Metadata option for `SField` definitions to indicate that the field, if set in an `AccountRoot` indicates that account is a pseudo-account. Removes the need for hard-coded field lists all over the place. Added the flag to `AMMID` and `VaultID`.
- Added functionality to `SField` ctor to detect both code and name collisions using asserts. And require all SFields to have a name
- Convenience type aliases `STLedgerEntry::const_pointer` and `STLedgerEntry::const_ref`. (`SLE` is an alias to `STLedgerEntry`.)
- Generalized `feeunit.h` (`TaggedFee`) into `unit.h` (`ValueUnit`) and added new "BIPS"-related tags for future use. Also refactored the type restrictions to use Concepts.
- Restructured `transactions.macro` to do two big things
1. Include the `#include` directives for transactor header files directly in the macro file. Removes the need to update `applySteps.cpp` and the resulting conflicts.
2. Added a `privileges` parameter to the `TRANSACTION` macro, which specifies some of the operations a transaction is allowed to do. These `privileges` are enforced by invariant checks. Again, removed the need to update scattered lists of transaction types in various checks.
- Unit tests:
1. Moved more helper functions into `TestHelpers.h` and `.cpp`.
2. Cleaned up the namespaces to prevent / mitigate random collisions and ambiguous symbols, particularly in unity builds.
3. Generalized `Env::balance` to add support for `MPTIssue` and `Asset`.
4. Added a set of helper classes to simplify `Env` transaction parameter classes: `JTxField`, `JTxFieldWrapper`, and a bunch of classes derived or aliased from it. For an example of how awesome it is, check the changes `src/test/jtx/escrow.h` for how much simpler the definitions are for `finish_time`, `cancel_time`, `condition`, and `fulfillment`.
5. Generalized several of the amount-related helper classes to understand `Asset`s.
6. `env.balance` for an MPT issuer will return a negative number (or 0) for consistency with IOUs.
* refactor: Simplify STParsedJSON with some helper functions (#5591)
- Add code coverage for STParsedJSON edge cases
Co-authored-by: Denis Angell <dangell@transia.co>
* test: Add STInteger and STParsedJSON tests (#5726)
This change is to improve code coverage (and to simplify #5720 and #5725); there is otherwise no change in functionality. The change adds basic tests for `STInteger` and `STParsedJSON`, so it becomes easier to test smaller changes to the types, as well as removes `STParsedJSONArray`, since it is not used anywhere (including in Clio).
* Revert "Update Conan dependencies: OpenSSL" (#5807)
This change reverts #5617, because it will require extensive testing that will take up more time than we have before the next scheduled release.
Reverting this change does not mean we are abandoning it. We aim to pick it back up once there's a sufficient time window to allow for testing on multiple distros running a mixture of OpenSSL 1.x and 3.x.
* docs: Add warning about using std::counting_semaphore (#5595)
This adds a comment to avoid using `std::counting_semaphore` until the minimum compiler versions of GCC and Clang have been updated to no longer contain the bug that is present in older compilers.
* Improve ValidatorList invalid UNL manifest logging (#5804)
This change raises logging severity from `INFO` to `WARN` when handling UNL manifest signed with an unexpected / invalid key. It also changes the internal error code for an invalid format of UNL manifest to `invalid` (from `untrusted`).
This is a follow up to problems experienced by an UNL node due to old manifest key configured in `validators.txt`, which would be easier to diagnose with improved logging.
It also replaces a log line with `UNREACHABLE` for an impossible situation when we match UNL manifest key against a configured key which has an invalid type (we cannot configure such a key because of checks when loading configured keys).
* chore: Pin all CI Docker tags (#5813)
To avoid surprises and ensure reproducibility, this change pins all CI Docker image tags to the latest version in the XRPLF/CI repo.
* change `fixPriceOracleOrder` to `Supported::yes` (#5749)
* fix: Address http header case sensitivity (#5767)
This change makes the regex in `HttpClient.cpp` that matches the content-length http header case insensitive to improve compatibility, as http headers are case insensitive.
* test: add more comprehensive tests for `FeeVote` (#5746)
This change adds more comprehensive tests for the `FeeVote` module, which previously only checked the basics, and not the more comprehensive flows in that class.
* ci: Call all reusable workflows reusable (#5818)
* Add `STInt32` as a new `SType` (#5788)
This change adds `STInt32` as a new `SType` under the `STInteger` umbrella, with `SType` value `12`. This is the first and only `STInteger` type that supports negative values.
* switch `fixIncludeKeyletFields` to `Supported::yes` (#5819)
* refactor: Restructure Transactor::preflight to reduce boilerplate (#5592)
* Restructures `Transactor::preflight` to create several functions that will remove the need for error-prone boilerplate code in derived classes' implementations of `preflight`.
* refactor: Add support for extra transaction signatures (#5594)
* Restructures Transactor signature checking code to be able to handle a `sigObject`, which may be the full transaction, or may be an object field containing a separate signature. Either way, the `sigObject` can be a single- or multi-sign signature.
* ci: Upload artifacts during build and test in a separate job (#5817)
* chore: Set free-form CI inputs as env vars (#5822)
This change moves CI values that could be user-provided into environment variables.
* Rename flags for DynamicMPT (#5820)
* Set version to 2.6.1
* fix: FD/handle guarding + exponential backoff (#5823)
* fix: Transaction sig checking functions do not get a full context (#5829)
Fixes a (currently harmless) bug introduced by PR #5594
* Remove bogus coverage warning (#5838)
* fix return type
---------
Co-authored-by: Ed Hennis <ed@ripple.com>
Co-authored-by: Jingchen <a1q123456@users.noreply.github.com>
Co-authored-by: Denis Angell <dangell@transia.co>
Co-authored-by: Bart <bthomee@users.noreply.github.com>
Co-authored-by: yinyiqian1 <yqian@ripple.com>
Co-authored-by: Vito Tumas <5780819+Tapanito@users.noreply.github.com>
Co-authored-by: Bronek Kozicki <brok@incorrekt.com>
Co-authored-by: Mayukha Vadari <mvadari@ripple.com>
Co-authored-by: Valentin Balaschenko <13349202+vlntb@users.noreply.github.com>
Co-authored-by: tequ <git@tequ.dev>
Co-authored-by: Ayaz Salikhov <mathbunnyru@users.noreply.github.com>
This change modifies the `build_only` check used to determine whether to run tests. For easier debugging in the future it also prints out the contents of the strategy matrix.
This change updates some incorrect Conan commands for Conan 2. As some flags do not exist in Conan 2, such as --settings build_type=[configuration], the commands have been adjusted accordingly. This change further uses the org-level variables and secrets rather than the repo-level ones.
This change uploads built Conan dependencies to the Conan remote upon merge into the develop branch.
At the moment, whenever Conan dependencies change, we need to remember to manually push them to our Conan remote, so they are cached for future reuse. If we forget to do so, these changed dependencies need to be rebuilt over and over again, which can take a long time.
This change:
* Removes the patched Conan recipes from the `external/` directory.
* Adds instructions for contributors how to obtain our patched recipes.
* Updates the Conan remote name and remote URL (the underlying package repository isn't changed).
* If the remote already exists, updates the URL instead of removing and re-adding.
* This is not done for the libXRPL job as it still uses Conan 1. This job will be switched to Conan 2 soon.
* Removes duplicate Conan remote CI pipeline steps.
* Overwrites the existing global.conf on MacOS and Windows machines, as those do not run CI pipelines in isolation but all share the same Conan installation; appending the same config over and over bloats the file.
This change updates RocksDB to its latest version. RocksDB is backward-compatible, so even though this is a major version bump, databases created with previous versions will continue to function.
The external RocksDB folder is removed, as the latest version available via Conan Center no longer needs custom patches.
This PR updates several dependencies to their latest versions. Not all dependencies have been updated, as some need to be patched and some require additional code changes due to backward incompatibilities introduced by the version bump.
- PR #5228 added assert=TRUE and werr=TRUE CMake flags to the
build/action.yml script which is used by all CI jobs to build rippled,
ensuring those flags were always set. The assumption was that only the
CI jobs used that script, so any extra time cost was offset by the
benefit of the extra checks. That assumption was incorrect. That
script is used by other downstream projects. Therefore, those flags
have been moved into the individual CI jobs' "cmake-args" parameter
passed to build/action.yml. This will have the same effect for CI jobs
without any side effects.
* Resolves an issue introduced in #5111, which inadvertently removed the
-Wno-maybe-uninitialized compiler option from some xrpl.libxrpl
modules. This resulted in new "may be used uninitialized" build
warnings, first noticed in the "protocol" module. When compiling with
derr=TRUE, those warnings became errors, which made the build fail.
* Github CI actions will build with the assert and werr options turned
on. This will cause CI jobs to fail if a developer introduces a new
compiler warning, or causes an assert to fail in release builds.
* Includes the OS and compiler version in the linux dependencies jobs in
the "check environment" step.
* Translates the `unity` build option into `CMAKE_UNITY_BUILD` setting.
Artifactory support was added to the `nix` builds with #4556. This
extends that support to the Windows build. Now the Windows build works;
CI will build and test a Windows release build. This only affects CI and
does not change any C++ code.
* Copy the remote setup step outcome fix from #4716 discussion
* Allow the Windows job to succeed if tests fail:
* Currently the tests do not always pass, even on a single threaded
run on the GitHub runners. So we are using parallel runs and mark
the test step as allowed to fail (continue-on-error).
* At this point, it's more important that the build succeeds than that
the tests succeed, because:
* We've got plenty of test coverage on the other jobs.
* Test failures are much rarer than build failures because of
cross-platform issues.
* Having a test failure locally doesn't interrupt a workflow nearly as
much as a build failure.
Note that Conan Center cannot hold the binaries we need. They do not
build the configurations we need, and they will not add them.
## Future Tasks
This introduces a new bottleneck since the build and test takes over an
hour. Speed up the job by:
* Making this job run on heavy Windows runners.
* Increasing the number of hardware threads.
Use the most recent versions in ConanCenter.
* Due to a bug in Clang 16, you may get a compile error:
"call to 'async_teardown' is ambiguous"
* A compiler flag workaround is documented in `BUILD.md`.
* At this time, building this with gcc 13 may require editing some files
in `.conan/data`
* A patch to support gcc13 may be added in a later PR.
---------
Co-authored-by: Scott Schurr <scott@ripple.com>
SOCI is the C++ database access library. The SOCI recipe was updated in
Conan Center Index (CCI), and it breaks for our choice of options. This
breakage occurs when you build with a fresh Conan cache (e.g. when you
submit a PR, or delete `~/.conan/data`).
* Add a custom Conan recipe for SOCI v4.0.3
* Update dependency building to handle exporting and installing Snappy
and SOCI
* Fix workflows to use custom SOCI recipe
* Update BUILD.md to include instruction for exporting the SOCI Conan
recipe:
* `conan export external/soci soci/4.0.3@`
This solution has been verified on Ubuntu 20.04 and macOS.
Context:
* There is a compiler error that the `sqlite3.h` header is not available
when building soci.
* When package B depends on package A, it finds the pieces it needs by
importing the Package Configuration File (PCF) that Conan generates
for package A.
* Read the CMake written by package B to check that it is importing
the PCF correctly and linking its exports correctly.
* Since this can be difficult, it is often more efficient to check
https://github.com/conan-io/conan-center-index/issues for package B
to see if anyone else has seen a similar problem.
* One of the issues points to a problem area in soci's CMake. To
confirm the diagnosis, review soci's CMake (after any patches are
applied) in the Conan build directory `build/$buildId/src/`.
* Review the Conan-generated PCF in
`build/$buildId/build/$buildType/generators/`.
* In this case, the problem was likely (re)introduced by
https://github.com/conan-io/conan-center-index/pull/17026
* If there is a problem in the source or in the Conan recipe, the
fastest fix is to copy the recipe and either:
* Add a source patch to fix any problems in the source.
* Change the recipe to fix any problems in the recipe.
* In this case, this can be done by finding soci's Conan recipe at
https://github.com/conan-io/conan-center-index/tree/master/recipes/soci
and then copying the `all` directory as `external/$packageName` in our
project. Then, make any changes.
* Test packages can be removed from the recipe folder as they are not
needed.
* If adding a patch in the `patches` directory, add a description for
it to `conandata.yml`.
* Since `conanfile.py` has no `version` property on the recipe class,
builders need to pass a version on the command line (like they do
for our `snappy` recipe).
* Add an example command to `BUILD.md`.
Future work: It may make sense to refer to recipes by revision, by
checking in a lockfile.
This change makes progress on the plan in #4371. It does not replicate
the full [matrix] implemented in #3851, but it does replicate the 1.ii
section of the Linux matrix. It leverages "heavy" self-hosted runners,
and demonstrates a repeatable pattern for future matrices.
[matrix]: d794a0f3f1/.github/README.md (continuous-integration)
