ci: Detect TSan lock-order inversions and add a TSan CI job

tsan.supp turned lock-order checking off wholesale: deadlock:pthread_create,
deadlock:pthread_rwlock_rdlock and deadlock:boost::asio name locking
primitives rather than source files, so between them they covered every
std::shared_mutex read lock and any inversion reached through a strand or a
thread start, and detect_deadlocks defaults to true. Seven more named source
files, among them ValidatorList.cpp, ValidatorSite.cpp and Manifest.cpp. Drop
all ten, because the job cannot fail a build, so a suppression here costs a
finding and buys nothing. The Manifest.cpp line alone was hiding a two-thread
deadlock between ManifestCache::save() and the validator list, which a run
without it reported 52 times on the shutdown path. Also repoint eight
patterns at files that moved into libxrpl and the gtest tree, and empty
sanitizer-ignorelist.txt of 24 entries that never matched anything, either
through suppression syntax a clang ignorelist never consults or through a
glob with no leading star. Both rules are now written down in
docs/build/sanitizers.md.

Add ubuntu-clang-debug-amd64-tsan to the Linux matrix, in a config of its
own so TSan stays on clang and Debug, and give the matrix a third tier so
it runs at night rather than on every labelled pull request. A Linux config
may now declare "extended", which holds it out of both the minimal and the
full matrix; generate.py emits those configs only for --extended, which the
workflow passes on a schedule or a manual run. The name says which matrix a
config belongs to, like "minimal", rather than naming a trigger, because the
trigger set already grew from the schedule to manual runs and a config
property outlives that. It avoids "maximal", which reads as a synonym for
the full matrix it is meant to be larger than. It is a whole hour of
runner time, which is too much to spend per pull request, and it reports
nothing back anyway, because the workflow appends exitcode=0 to
TSAN_OPTIONS. That belongs in the workflow rather than in
runtime-tsan-options.txt, which the documented local command also reads and
which must keep failing on a finding.

Cap the test jobs at 4 under TSan, which is measured rather than chosen.
One per core is 30 on the current runner and twice starved it until it lost
contact with the server; 12 was killed by the OOM killer with code 137
before a single suite finished; 4 completes the suite. The machine has 32
cores and no swap, so nothing absorbs the peak, and the runner is one pod
among several on a node, so the ceiling is not ours alone.

Check that a build carries the instrumentation it asked for, by the __asan,
__tsan and __ubsan symbols in the binary, since instrumented code
references its runtime however that runtime is linked. The version string
is checked too, but cannot stand alone, because cmake sets the SANITIZERS
macro separately from the flags. Widen the voidstar step for this rather
than add a second one, and add ASAN_ENABLED, TSAN_ENABLED and UBSAN_ENABLED
so a step needing one sanitizer does not parse the list. Define XRPL_ASAN,
XRPL_TSAN and XRPL_UBSAN so a test can skip when its sanitizer is inactive,
and drop the -Dcoverage_test_parallelism example from BUILD.md, which
neither cmake nor conanfile.py defines.
This commit is contained in:
Bart
2026-09-12 07:30:11 -04:00
parent d8870162eb
commit a91b2f4e25
10 changed files with 281 additions and 61 deletions

View File

@@ -60,6 +60,11 @@ def get_cmake_args(build_type: str, extra_args: str) -> str:
# Every config must declare 'minimal'. Minimal configs form the reduced matrix
# built for pull requests by default; the full matrix adds the rest.
#
# A Linux config may instead declare 'extended'. Neither the minimal nor the full
# matrix includes such a config; only the extended matrix does, which the nightly
# schedule and a manual run ask for. Use it for a config too expensive to run per
# pull request. It is Linux only because nothing else needs it yet.
#
# Configs may also opt into 'benchmark' to smoke-run the benchmarks, or carry a
# 'package' map to be packaged as well. Note that either applies to every entry
# a config expands into, so only set them on configs that expand to a single
@@ -94,6 +99,7 @@ class LinuxConfig:
build_type: list[str]
arch: list[str]
minimal: bool
extended: bool = False
benchmark: bool = False # if true, smoke-run the benchmarks after testing
sanitizers: list[str] = dataclasses.field(default_factory=list)
suffix: str = ""
@@ -215,12 +221,22 @@ _ARCHS: dict[str, Architecture] = {
}
def expand_linux_matrix(linux: LinuxFile, minimal: bool) -> list[MatrixEntry]:
def expand_linux_matrix(
linux: LinuxFile, minimal: bool, extended: bool = False
) -> list[MatrixEntry]:
"""Expand a LinuxFile into a flat list of matrix entries.
Each config entry is expanded over the cross-product of its
compiler, build_type, sanitizers, and architecture lists. When 'minimal' is
true, only configs flagged as minimal are included.
@param linux The parsed linux.json.
@param minimal Emit only the configs flagged 'minimal'.
@param extended Emit the configs flagged 'extended'. Both the minimal and
the full matrix leave those out, so this is the only way to get them.
@return One entry per combination the surviving configs expand into.
@note Do not set both 'minimal' and 'extended'. The command line rejects the
pair, but a direct caller gets the minimal matrix rather than an error.
"""
entries: list[MatrixEntry] = []
@@ -228,6 +244,8 @@ def expand_linux_matrix(linux: LinuxFile, minimal: bool) -> list[MatrixEntry]:
for cfg in configs:
if minimal and not cfg.minimal:
continue
if not extended and cfg.extended:
continue
# An empty sanitizers list means "one entry with no sanitizer".
effective_sanitizers = cfg.sanitizers or [""]
effective_archs = {arch: _ARCHS[arch] for arch in cfg.arch}
@@ -367,7 +385,12 @@ if __name__ == "__main__":
help="Emit the Linux packaging matrix instead of the build/test matrix.",
action="store_true",
)
parser.add_argument(
# Each flag picks a matrix size, and the sizes nest: minimal is a subset of
# the full matrix, which is a subset of extended. So one flag narrows and the
# other widens the same default, and asking for both has no answer. Argparse
# rejects the pair rather than letting the filters intersect into a surprise.
size = parser.add_mutually_exclusive_group()
size.add_argument(
"-m",
"--minimal",
help="Emit only the minimal matrix (the configs flagged 'minimal'), "
@@ -375,6 +398,14 @@ if __name__ == "__main__":
"emitted.",
action="store_true",
)
size.add_argument(
"-x",
"--extended",
help="Emit the extended matrix: the full one plus the configs flagged "
"'extended', which no other matrix includes. Used for the nightly "
"schedule and for a manual run.",
action="store_true",
)
args = parser.parse_args()
matrix: list[MatrixEntry] | list[PackagingEntry] = []
@@ -388,7 +419,7 @@ if __name__ == "__main__":
else:
if args.config in ("linux", None):
matrix += expand_linux_matrix(
LinuxFile.load(THIS_DIR / "linux.json"), args.minimal
LinuxFile.load(THIS_DIR / "linux.json"), args.minimal, args.extended
)
if args.config in ("macos", None):
matrix += expand_platform_matrix(

View File

@@ -38,6 +38,14 @@
"minimal": false,
"sanitizers": ["address", "undefinedbehavior"]
},
{
"compiler": ["clang"],
"build_type": ["Debug"],
"arch": ["amd64"],
"minimal": false,
"extended": true,
"sanitizers": ["thread"]
},
{
"compiler": ["clang"],