tsan.supp turned lock-order checking off wholesale: deadlock:pthread_create,
deadlock:pthread_rwlock_rdlock and deadlock:boost::asio name locking
primitives rather than source files, so between them they covered every
std::shared_mutex read lock and any inversion reached through a strand or a
thread start, and detect_deadlocks defaults to true. Seven more named source
files, among them ValidatorList.cpp, ValidatorSite.cpp and Manifest.cpp. Drop
all ten, because the job cannot fail a build, so a suppression here costs a
finding and buys nothing. The Manifest.cpp line alone was hiding a two-thread
deadlock between ManifestCache::save() and the validator list, which a run
without it reported 52 times on the shutdown path. Also repoint eight
patterns at files that moved into libxrpl and the gtest tree, and empty
sanitizer-ignorelist.txt of 24 entries that never matched anything, either
through suppression syntax a clang ignorelist never consults or through a
glob with no leading star. Both rules are now written down in
docs/build/sanitizers.md.
Add ubuntu-clang-debug-amd64-tsan to the Linux matrix, in a config of its
own so TSan stays on clang and Debug, and give the matrix a third tier so
it runs at night rather than on every labelled pull request. A Linux config
may now declare "extended", which holds it out of both the minimal and the
full matrix; generate.py emits those configs only for --extended, which the
workflow passes on a schedule or a manual run. The name says which matrix a
config belongs to, like "minimal", rather than naming a trigger, because the
trigger set already grew from the schedule to manual runs and a config
property outlives that. It avoids "maximal", which reads as a synonym for
the full matrix it is meant to be larger than. It is a whole hour of
runner time, which is too much to spend per pull request, and it reports
nothing back anyway, because the workflow appends exitcode=0 to
TSAN_OPTIONS. That belongs in the workflow rather than in
runtime-tsan-options.txt, which the documented local command also reads and
which must keep failing on a finding.
Cap the test jobs at 4 under TSan, which is measured rather than chosen.
One per core is 30 on the current runner and twice starved it until it lost
contact with the server; 12 was killed by the OOM killer with code 137
before a single suite finished; 4 completes the suite. The machine has 32
cores and no swap, so nothing absorbs the peak, and the runner is one pod
among several on a node, so the ceiling is not ours alone.
Check that a build carries the instrumentation it asked for, by the __asan,
__tsan and __ubsan symbols in the binary, since instrumented code
references its runtime however that runtime is linked. The version string
is checked too, but cannot stand alone, because cmake sets the SANITIZERS
macro separately from the flags. Widen the voidstar step for this rather
than add a second one, and add ASAN_ENABLED, TSAN_ENABLED and UBSAN_ENABLED
so a step needing one sanitizer does not parse the list. Define XRPL_ASAN,
XRPL_TSAN and XRPL_UBSAN so a test can skip when its sanitizer is inactive,
and drop the -Dcoverage_test_parallelism example from BUILD.md, which
neither cmake nor conanfile.py defines.
* upstream/release/3.3.x: (41 commits)
chore: Bump version to 3.3.0
chore: Bump version to 3.3.0-rc7
fix: Increase manifest protocol message size cap and fix manifests relay
fix: Cap untrusted manifests per message and drop oversized ones
chore: Bump version to 3.2.1
chore: Bump version to 3.2.1-rc1
fix: Cap untrusted manifests per message and drop oversized ones
fix: Reject oversized validator manifest before decoding
fix: Reduce untrusted manifest cache cap to 100
fix: Bound untrusted manifest cache
chore: Bump version to 3.3.0-rc6
feat: Package validator-keys inside rippled
chore: Bump version to 3.3.0-rc5
fix: Switch SponsorshipSet to use a delta for sfFeeAmount
fix: Re-revert "fix: Set request size limits and differential pricing for get-object-by-hash calls"
chore: Bump version to 3.3.0-rc4
fix: Revert "fix: Set request size limits and differential pricing for get-object-by-hash calls"
chore: Bump version to 3.3.0-rc3
fix: Reduce untrusted manifest cache cap to 100
fix: Revert "fix: Reject oversized SHAMap nodes in gotStaleData and fetch-pack path"
...
Unity builds were intended to speed up builds, by bundling multiple files into compilation units. However, now that ccache is available on all platforms, there is no need for unity builds anymore, as ccache stores compiled individual build objects for reuse. This change therefore removes the ability to make unity builds.
Right now, each pipeline invocation builds the source code from scratch. Although compiled Conan dependencies are cached in a remote server, the source build objects are not. We are able to further speed up our builds by leveraging `ccache`. This change enables caching of build objects using `ccache` on Linux, macOS, and Windows.
This updates the CI image hashes after following change: https://github.com/XRPLF/ci/pull/81. And, since we use latest Conan, we can have `conan.lock` with a newline at the end, and we don't need to exclude it from `pre-commit` hooks any longer.
To debug test failures we would like to use `netstat`, but that package wasn't installed yet in the CI images. This change uses the new CI images created by https://github.com/XRPLF/ci/pull/79.
This change uses the new RHEL 9 and 10 images to build and test the binary, and adds support for having different Docker image SHAs per distro-compiler combination.
Instead of supporting RHEL each minor version, we are simplifying our pipelines by only supporting RHEL major versions. Our CI Docker images have already been updated accordingly, and we recently added support for RHEL 10 as well. Up until now, the CI Docker images had all been rebuilt at the same time, but that is not necessarily true as the most recent push to the CI repo has shown where the RHEL images now have a different SHA than the Debian and Ubuntu ones.
Co-authored-by: Bart Thomee <11445373+bthomee@users.noreply.github.com>
The change updates how clang-format is called in CI and locally, and adds prettier to the pre-commit hook. Proto files are now also formatted, while external files are excluded.
