Merge branch 'develop' into ximinez/online-delete-gaps

- lastGoodValidatedLedger_ update is now trace.
- Log the first healthWait failure for a given validated ledger as
  trace.

.envrc

@@ -0,0 +1 @@
+use flake

cfg/xrpld-example.cfg

@@ -1059,10 +1059,11 @@
 #                           The online delete process checks periodically
 #                           that xrpld is still in sync with the network,
 #                           and that the validated ledger is less than
-#                           'age_threshold_seconds' old. If not, then continue
+#                           'age_threshold_seconds' old, and that all
+#                           recent ledgers are available. If not, then continue
 #                           sleeping for this number of seconds and
 #                           checking until healthy.
-#                           Default is 5.
+#                           Default is 2.
 #
 #   Notes:
 #       The 'node_db' entry configures the primary, persistent storage.

docs/build/environment.md

@@ -33,9 +33,10 @@ with a single command and without installing anything system-wide:
 nix --experimental-features 'nix-command flakes' develop
 ```
 
-On **Linux**, Nix also provides the compiler (GCC). On **macOS**, the shell uses
-your **system-wide Apple Clang** as the compiler, so you still need to manage
-its version (see below).
+On **Linux**, Nix also provides the compiler (GCC); on **macOS**, it provides
+Clang. If you instead opt to use your system-wide Apple Clang (via
+`nix develop .#apple-clang`), you need to manage its version yourself (see
+below).
 
 See [Using the Nix development shell](./nix.md) for installation and usage
 details, including how to select a different compiler.
@@ -48,10 +49,10 @@ details, including how to select a different compiler.
 
 ### macOS: managing the Apple Clang version
 
-Because the Nix shell uses the system-wide Apple Clang on macOS, the compiler
-version is whatever your installed Xcode (or Command Line Tools) provides. The
-following command should return a version greater than or equal to the
-[minimum required](#tested-compiler-versions):
+If you use your system-wide Apple Clang on macOS (via `nix develop .#apple-clang`),
+the compiler version is whatever your installed Xcode (or Command Line Tools)
+provides. The following command should return a version greater than or equal to
+the [minimum required](#tested-compiler-versions):
 
 ```bash
 clang --version

docs/build/nix.md

@@ -9,7 +9,7 @@ This guide explains how to use Nix to set up a reproducible development environm
 - **Reproducible environment**: Everyone gets the same versions of tools and compilers
 - **Matches CI**: The Linux CI runs in Docker images built from this exact Nix environment
 - **No system pollution**: Dependencies are isolated and don't affect your system packages
-- **Multiple compiler versions**: Easily switch between different GCC and Clang versions
+- **Consistent compilers**: The GCC and Clang shells use the same versions as CI
 - **Quick setup**: Get started with a single command
 - **Works on Linux and macOS**: Consistent experience across platforms
 
@@ -31,8 +31,8 @@ This will:
 
 - Download and set up all required development tools (CMake, Ninja, Conan, etc.)
 - Configure the appropriate compiler for your platform:
-  - **Linux**: GCC 15.2 (provided by Nix)
-  - **macOS**: Apple Clang (your system compiler)
+  - **Linux**: GCC (provided by Nix)
+  - **macOS**: Clang (provided by Nix)
 
 The first time you run this command, it will take a few minutes to download and build the environment. Subsequent runs will be much faster.
 
@@ -40,12 +40,12 @@ The first time you run this command, it will take a few minutes to download and
 
 - **Linux**: `nix develop` gives you a shell with all the tooling necessary to
   develop xrpld and with GCC 15.2 (also provided by Nix). There are no caveats.
-- **macOS**: `nix develop` gives you a full environment too. The compiler is
-  your system-wide Apple Clang, while every other tool — including Conan — is
-  provided by Nix. Conan has no binary in the Nix cache for macOS, so it is
-  built from source the first time you enter the shell, which makes the initial
-  setup slower (this is handled automatically; see
-  [`nix/devshell.nix`](../../nix/devshell.nix)).
+- **macOS**: `nix develop` gives you a full environment too, with Clang (and
+  every other tool, including Conan) provided by Nix. To use your system-wide
+  Apple Clang instead, enter `nix develop .#apple-clang`. Conan has no binary in
+  the Nix cache for macOS, so it is built from source the first time you enter
+  the shell, which makes the initial setup slower (this is handled
+  automatically; see [`nix/devshell.nix`](../../nix/devshell.nix)).
 
 > [!TIP]
 > To avoid typing `--experimental-features 'nix-command flakes'` every time, you can permanently enable flakes by creating `~/.config/nix/nix.conf`:
@@ -62,7 +62,9 @@ The first time you run this command, it will take a few minutes to download and
 
 ### Choosing a different compiler
 
-A compiler can be chosen by providing its name with the `.#` prefix, e.g. `nix develop .#gcc15`.
+A compiler can be chosen by providing its name with the `.#` prefix, e.g. `nix develop .#clang`.
+The `.#gcc` and `.#clang` shells provide the same GCC and Clang versions used in CI
+(pinned in [`nix/packages.nix`](../../nix/packages.nix)).
 Use `nix flake show` to see all the available development shells.
 
 Use `nix develop .#no-compiler` to use the compiler from your system.
@@ -70,11 +72,11 @@ Use `nix develop .#no-compiler` to use the compiler from your system.
 ### Example Usage
 
 ```bash
-# Use GCC 14
-nix develop .#gcc14
+# Use GCC (same version as CI)
+nix develop .#gcc
 
-# Use Clang 19
-nix develop .#clang19
+# Use Clang (same version as CI)
+nix develop .#clang
 
 # Use default for your platform
 nix develop
@@ -112,7 +114,15 @@ Once inside the Nix development shell, follow the standard [build instructions](
 
 [direnv](https://direnv.net/) or [nix-direnv](https://github.com/nix-community/nix-direnv) can automatically activate the Nix development shell when you enter the repository directory.
 
-This is also the most robust way to use the environment from **any shell** (bash, zsh, fish, …): direnv stays in your current shell and loads the environment _after_ your shell's startup files have run, so the Nix-provided tools take precedence over anything your shell configuration adds to `$PATH`. To use it, install direnv for your shell, then add an `.envrc` containing `use flake` at the repository root and run `direnv allow`.
+This is also the most robust way to use the environment from **any shell** (bash, zsh, fish, …): direnv stays in your current shell and loads the environment _after_ your shell's startup files have run, so the Nix-provided tools take precedence over anything your shell configuration adds to `$PATH`.
+
+The repository already ships an `.envrc` at its root that activates the Nix flake development shell, so you don't need to create one. To use it:
+
+1. [Install direnv](https://direnv.net/docs/installation.html) and [hook it into your shell](https://direnv.net/docs/hook.html) (bash, zsh, fish, …). Installing [nix-direnv](https://github.com/nix-community/nix-direnv) as well is recommended: it caches the shell so that activation is near-instant after the first run.
+2. Run `direnv allow` once in the repository root. direnv will then load (and reload) the Nix development shell automatically whenever you enter the directory.
+
+> [!NOTE]
+> direnv only caches the `.direnv` directory (already listed in `.gitignore`); no other repository files are affected.
 
 ## Conan and Prebuilt Packages
 

nix/ci-env.nix

@@ -4,14 +4,16 @@
   ...
 }:
 let
-  inherit (import ./packages.nix { inherit pkgs; }) commonPackages;
-  inherit (pkgs) lib;
+  inherit (import ./packages.nix { inherit pkgs; })
+    commonPackages
+    gccPackage
+    llvmPackages
+    llvmVersion
+    ;
 
-  # Underlying compiler toolchains to wrap. Bump these in one place to
-  # roll the whole environment forward.
-  customGccPackage = pkgs.gcc15;
-  customLlvmPackages = pkgs.llvmPackages_22;
-  customClangMajor = lib.versions.major (lib.getVersion customLlvmPackages.clang-unwrapped);
+  # Underlying compiler toolchains to wrap (versions pinned in packages.nix).
+  customGccPackage = gccPackage;
+  customLlvmPackages = llvmPackages;
 
   # binutils wrapped to emit binaries that reference the custom glibc
   # (dynamic linker path, library search path, RPATH).
@@ -90,7 +92,7 @@ let
     extraBuildCommands = ''
       rsrc="$out/resource-root"
       mkdir "$rsrc"
-      ln -s "${customLlvmPackages.clang-unwrapped.lib}/lib/clang/${customClangMajor}/include" "$rsrc/include"
+      ln -s "${customLlvmPackages.clang-unwrapped.lib}/lib/clang/${toString llvmVersion}/include" "$rsrc/include"
       ln -s "${customCompilerRt.out}/lib" "$rsrc/lib"
       ln -s "${customCompilerRt.out}/share" "$rsrc/share" || true
       echo "-resource-dir=$rsrc" >> $out/nix-support/cc-cflags

nix/devshell.nix

@@ -1,127 +1,57 @@
 { pkgs, ... }:
 let
-  inherit (import ./packages.nix { inherit pkgs; }) commonPackages;
+  inherit (import ./packages.nix { inherit pkgs; })
+    commonPackages
+    gccVersion
+    llvmPackages
+    ;
 
-  # Supported compiler versions
-  gccVersion = pkgs.lib.range 13 15;
-  clangVersions = pkgs.lib.range 18 21;
+  # Plain nixpkgs stdenvs — no custom glibc, unlike ci-env.nix.
+  gccStdenv = pkgs."gcc${toString gccVersion}Stdenv";
+  clangStdenv = llvmPackages.stdenv;
 
-  defaultCompiler = if pkgs.stdenv.isDarwin then "apple-clang" else "gcc";
-  defaultGccVersion = pkgs.lib.last gccVersion;
-  defaultClangVersion = pkgs.lib.last clangVersions;
-
-  strToCompilerEnv =
-    compiler: version:
-    (
-      if compiler == "gcc" then
-        let
-          gccPkg = pkgs."gcc${toString version}Stdenv" or null;
-        in
-        if gccPkg != null && builtins.elem version gccVersion then
-          gccPkg
-        else
-          throw "Invalid GCC version: ${toString version}. Must be one of: ${toString gccVersion}"
-      else if compiler == "clang" then
-        let
-          clangPkg = pkgs."llvmPackages_${toString version}".stdenv or null;
-        in
-        if clangPkg != null && builtins.elem version clangVersions then
-          clangPkg
-        else
-          throw "Invalid Clang version: ${toString version}. Must be one of: ${toString clangVersions}"
-      else if compiler == "apple-clang" || compiler == "none" then
-        pkgs.stdenvNoCC
-      else
-        throw "Invalid compiler: ${compiler}. Must be one of: gcc, clang, apple-clang, none"
-    );
-
-  # Helper function to create a shell with a specific compiler
+  # compilerName is the command used to print the version, or null for none.
   makeShell =
     {
-      compiler ? defaultCompiler,
-      version ? (
-        if compiler == "gcc" then
-          defaultGccVersion
-        else if compiler == "clang" then
-          defaultClangVersion
-        else
-          null
-      ),
+      stdenv,
+      compilerName,
     }:
     let
-      compilerStdEnv = strToCompilerEnv compiler version;
-
-      compilerName =
-        if compiler == "apple-clang" then
-          "clang"
-        else if compiler == "none" then
-          null
-        else
-          compiler;
-
-      gccOnMacWarning =
-        if pkgs.stdenv.isDarwin && compiler == "gcc" then
-          ''
-            echo "WARNING: Using GCC on macOS with Conan may not work."
-            echo "         Consider using 'nix develop .#clang' or the default shell instead."
-            echo ""
-          ''
-        else
-          "";
-
       compilerVersion =
-        if compilerName != null then
+        if compilerName == null then
+          ''echo "No compiler specified - using system compiler"''
+        else
           ''
             echo "Compiler: "
             ${compilerName} --version
-          ''
-        else
-          ''
-            echo "No compiler specified - using system compiler"
           '';
-
-      shellAttrs = {
-        packages = commonPackages;
-
-        shellHook = ''
-          echo "Welcome to xrpld development shell";
-          ${gccOnMacWarning}${compilerVersion}
-        '';
-      };
     in
-    pkgs.mkShell.override { stdenv = compilerStdEnv; } shellAttrs;
-
-  # Generate shells for each compiler version
-  gccShells = builtins.listToAttrs (
-    map (version: {
-      name = "gcc${toString version}";
-      value = makeShell {
-        compiler = "gcc";
-        version = version;
-      };
-    }) gccVersion
-  );
-
-  clangShells = builtins.listToAttrs (
-    map (version: {
-      name = "clang${toString version}";
-      value = makeShell {
-        compiler = "clang";
-        version = version;
-      };
-    }) clangVersions
-  );
-
+    (pkgs.mkShell.override { inherit stdenv; }) {
+      packages = commonPackages;
+      shellHook = ''
+        echo "Welcome to xrpld development shell";
+        ${compilerVersion}
+      '';
+    };
 in
-gccShells
-// clangShells
-// {
-  # Default shells
-  default = makeShell { };
-  gcc = makeShell { compiler = "gcc"; };
-  clang = makeShell { compiler = "clang"; };
+rec {
+  # macOS: Nix Clang. Linux: Nix GCC.
+  default = if pkgs.stdenv.isDarwin then clang else gcc;
 
-  # No compiler
-  no-compiler = makeShell { compiler = "none"; };
-  apple-clang = makeShell { compiler = "apple-clang"; };
+  gcc = makeShell {
+    stdenv = gccStdenv;
+    compilerName = "gcc";
+  };
+
+  clang = makeShell {
+    stdenv = clangStdenv;
+    compilerName = "clang";
+  };
+
+  # Nix provides no compiler; use the one from your system (e.g. Apple Clang).
+  no-compiler = makeShell {
+    stdenv = pkgs.stdenvNoCC;
+    compilerName = null;
+  };
+  apple-clang = no-compiler;
 }

nix/packages.nix

@@ -1,15 +1,33 @@
 { pkgs }:
 let
+  # Compiler versions used across the dev shell and the CI environment.
+  gccVersion = 15;
+  llvmVersion = 22;
+
+  gccPackage = pkgs."gcc${toString gccVersion}";
+  llvmPackages = pkgs."llvmPackages_${toString llvmVersion}";
+
+  # Bound explicitly so it tracks llvmPackages above, not the `with pkgs` default.
+  clangTools = llvmPackages.clang-tools;
+
   # In LLVM 22, run-clang-tidy.py moved from share/clang/ to bin/, so nixpkgs
   # clang-tools no longer links it. Wrap it manually.
   runClangTidy = pkgs.writeShellScriptBin "run-clang-tidy" ''
-    exec ${pkgs.python3}/bin/python3 ${pkgs.llvmPackages_22.clang-unwrapped}/bin/run-clang-tidy "$@"
+    exec ${pkgs.python3}/bin/python3 ${llvmPackages.clang-unwrapped}/bin/run-clang-tidy "$@"
   '';
 in
 {
+  inherit
+    gccVersion
+    llvmVersion
+    gccPackage
+    llvmPackages
+    ;
+
   commonPackages = with pkgs; [
     ccache
     clangbuildanalyzer
+    clangTools
     cmake
     conan
     curlMinimal # needed for codecov/codecov-action
@@ -23,7 +41,6 @@ in
     gnumake
     gnupg # needed for signing commits & codecov/codecov-action
     graphviz
-    llvmPackages_22.clang-tools
     less # needed for git diff
     mold
     nettools # provides netstat, used to debug failures in CI

src/test/app/LedgerMaster_test.cpp

@@ -5,17 +5,21 @@
 #include <test/jtx/noop.h>
 
 #include <xrpld/app/ledger/LedgerMaster.h>
+#include <xrpld/app/misc/SHAMapStore.h>
 #include <xrpld/core/Config.h>
 
+#include <xrpl/basics/ToString.h>
 #include <xrpl/basics/base_uint.h>
 #include <xrpl/beast/unit_test/suite.h>
 #include <xrpl/protocol/Feature.h>
+#include <xrpl/protocol/Protocol.h>
 #include <xrpl/protocol/SField.h>
 #include <xrpl/protocol/STObject.h>
 #include <xrpl/protocol/STTx.h>
 
 #include <cstdint>
 #include <memory>
+#include <sstream>
 #include <vector>
 
 namespace xrpl::test {
@@ -111,6 +115,73 @@ class LedgerMaster_test : public beast::unit_test::Suite
         }
     }
 
+    void
+    testCompleteLedgerRange(FeatureBitset features)
+    {
+        // Note that this test is intentionally very similar to
+        // SHAMapStore_test::testLedgerGaps, but has a different
+        // focus.
+
+        testcase("Complete Ledger operations");
+
+        using namespace test::jtx;
+
+        auto const deleteInterval = 8;
+
+        Env env{*this, envconfig([](auto cfg) {
+                    return onlineDelete(std::move(cfg), deleteInterval);
+                })};
+
+        auto const alice = Account("alice");
+        env.fund(XRP(1000), alice);
+        env.close();
+
+        auto& lm = env.app().getLedgerMaster();
+        LedgerIndex minSeq = 2;
+        LedgerIndex maxSeq = env.closed()->header().seq;
+        auto& store = env.app().getSHAMapStore();
+        BEAST_EXPECT(store.rendezvous());
+        LedgerIndex lastRotated = store.getLastRotated();
+        BEAST_EXPECTS(maxSeq == 3, to_string(maxSeq));
+        BEAST_EXPECTS(lm.getCompleteLedgers() == "2-3", lm.getCompleteLedgers());
+        BEAST_EXPECTS(lastRotated == 3, to_string(lastRotated));
+        BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq, maxSeq) == 0);
+        BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq + 1, maxSeq - 1) == 0);
+        BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq - 1, maxSeq + 1) == 2);
+        BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq - 2, maxSeq - 2) == 2);
+        BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq + 2, maxSeq + 2) == 2);
+
+        // Close enough ledgers to rotate a few times
+        for (int i = 0; i < 24; ++i)
+        {
+            for (int t = 0; t < 3; ++t)
+            {
+                env(noop(alice));
+            }
+            env.close();
+            BEAST_EXPECT(store.rendezvous());
+
+            ++maxSeq;
+
+            if (maxSeq == lastRotated + deleteInterval)
+            {
+                minSeq = lastRotated;
+                lastRotated = maxSeq;
+            }
+            BEAST_EXPECTS(
+                env.closed()->header().seq == maxSeq, to_string(env.closed()->header().seq));
+            BEAST_EXPECTS(store.getLastRotated() == lastRotated, to_string(store.getLastRotated()));
+            std::stringstream expectedRange;
+            expectedRange << minSeq << "-" << maxSeq;
+            BEAST_EXPECTS(lm.getCompleteLedgers() == expectedRange.str(), lm.getCompleteLedgers());
+            BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq, maxSeq) == 0);
+            BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq + 1, maxSeq - 1) == 0);
+            BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq - 1, maxSeq + 1) == 2);
+            BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq - 2, maxSeq - 2) == 2);
+            BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq + 2, maxSeq + 2) == 2);
+        }
+    }
+
 public:
     void
     run() override
@@ -124,6 +195,7 @@ public:
     testWithFeats(FeatureBitset features)
     {
         testTxnIdFromIndex(features);
+        testCompleteLedgerRange(features);
     }
 };
 

src/test/app/SHAMapStore_test.cpp

@@ -1,7 +1,9 @@
 #include <test/jtx/Env.h>
 #include <test/jtx/amount.h>
 #include <test/jtx/envconfig.h>
+#include <test/jtx/noop.h>
 
+#include <xrpld/app/ledger/LedgerMaster.h>
 #include <xrpld/app/main/Application.h>
 #include <xrpld/app/main/NodeStoreScheduler.h>
 #include <xrpld/app/misc/SHAMapStore.h>
@@ -9,6 +11,7 @@
 #include <xrpld/core/Config.h>
 
 #include <xrpl/basics/ByteUtilities.h>
+#include <xrpl/basics/Number.h>
 #include <xrpl/basics/base_uint.h>
 #include <xrpl/beast/unit_test/suite.h>
 #include <xrpl/config/BasicConfig.h>
@@ -26,12 +29,15 @@
 #include <boost/filesystem/path.hpp>
 
 #include <atomic>
+#include <cstddef>
 #include <cstdint>
 #include <limits>
 #include <map>
 #include <memory>
 #include <optional>
+#include <sstream>
 #include <string>
+#include <thread>
 #include <utility>
 
 namespace xrpl::test {
@@ -43,10 +49,7 @@ class SHAMapStore_test : public beast::unit_test::Suite
     static auto
     onlineDelete(std::unique_ptr<Config> cfg)
     {
-        cfg->ledgerHistory = kDeleteInterval;
-        auto& section = cfg->section(Sections::kNodeDatabase);
-        section.set(Keys::kOnlineDelete, std::to_string(kDeleteInterval));
-        return cfg;
+        return jtx::onlineDelete(std::move(cfg), kDeleteInterval);
     }
 
     static auto
@@ -144,11 +147,11 @@ class SHAMapStore_test : public beast::unit_test::Suite
         auto& store = env.app().getSHAMapStore();
 
         int ledgerSeq = 3;
-        store.rendezvous();
+        BEAST_EXPECT(store.rendezvous());
         BEAST_EXPECT(!store.getLastRotated());
 
         env.close();
-        store.rendezvous();
+        BEAST_EXPECT(store.rendezvous());
 
         auto ledger = env.rpc("ledger", "validated");
         BEAST_EXPECT(goodLedger(env, ledger, std::to_string(ledgerSeq++)));
@@ -228,7 +231,7 @@ public:
             BEAST_EXPECT(goodLedger(env, ledger, std::to_string(kDeleteInterval + 4)));
         }
 
-        store.rendezvous();
+        BEAST_EXPECT(store.rendezvous());
 
         BEAST_EXPECT(store.getLastRotated() == kDeleteInterval + 3);
         lastRotated = store.getLastRotated();
@@ -255,7 +258,7 @@ public:
                 !getHash(ledgers[i]).empty());
         }
 
-        store.rendezvous();
+        BEAST_EXPECT(store.rendezvous());
 
         BEAST_EXPECT(store.getLastRotated() == kDeleteInterval + lastRotated);
 
@@ -293,7 +296,7 @@ public:
             BEAST_EXPECT(goodLedger(env, ledger, std::to_string(ledgerSeq), true));
         }
 
-        store.rendezvous();
+        BEAST_EXPECT(store.rendezvous());
 
         // The database will always have back to ledger 2,
         // regardless of lastRotated.
@@ -308,7 +311,7 @@ public:
             BEAST_EXPECT(goodLedger(env, ledger, std::to_string(ledgerSeq++), true));
         }
 
-        store.rendezvous();
+        BEAST_EXPECT(store.rendezvous());
 
         ledgerCheck(env, ledgerSeq - lastRotated, lastRotated);
         BEAST_EXPECT(lastRotated != store.getLastRotated());
@@ -324,7 +327,7 @@ public:
             BEAST_EXPECT(goodLedger(env, ledger, std::to_string(ledgerSeq), true));
         }
 
-        store.rendezvous();
+        BEAST_EXPECT(store.rendezvous());
 
         ledgerCheck(env, kDeleteInterval + 1, lastRotated);
         BEAST_EXPECT(lastRotated != store.getLastRotated());
@@ -363,7 +366,7 @@ public:
             BEAST_EXPECT(goodLedger(env, ledger, std::to_string(ledgerSeq), true));
         }
 
-        store.rendezvous();
+        BEAST_EXPECT(store.rendezvous());
 
         ledgerCheck(env, ledgerSeq - 2, 2);
         BEAST_EXPECT(lastRotated == store.getLastRotated());
@@ -373,7 +376,7 @@ public:
         BEAST_EXPECT(!RPC::containsError(canDelete[jss::result]));
         BEAST_EXPECT(canDelete[jss::result][jss::can_delete] == ledgerSeq + (kDeleteInterval / 2));
 
-        store.rendezvous();
+        BEAST_EXPECT(store.rendezvous());
 
         ledgerCheck(env, ledgerSeq - 2, 2);
         BEAST_EXPECT(store.getLastRotated() == lastRotated);
@@ -386,7 +389,7 @@ public:
             BEAST_EXPECT(goodLedger(env, ledger, std::to_string(ledgerSeq++), true));
         }
 
-        store.rendezvous();
+        BEAST_EXPECT(store.rendezvous());
 
         ledgerCheck(env, ledgerSeq - lastRotated, lastRotated);
 
@@ -402,7 +405,7 @@ public:
             BEAST_EXPECT(goodLedger(env, ledger, std::to_string(ledgerSeq), true));
         }
 
-        store.rendezvous();
+        BEAST_EXPECT(store.rendezvous());
 
         BEAST_EXPECT(store.getLastRotated() == lastRotated);
 
@@ -414,7 +417,7 @@ public:
             BEAST_EXPECT(goodLedger(env, ledger, std::to_string(ledgerSeq++), true));
         }
 
-        store.rendezvous();
+        BEAST_EXPECT(store.rendezvous());
 
         ledgerCheck(env, ledgerSeq - firstBatch, firstBatch);
 
@@ -436,7 +439,7 @@ public:
             BEAST_EXPECT(goodLedger(env, ledger, std::to_string(ledgerSeq), true));
         }
 
-        store.rendezvous();
+        BEAST_EXPECT(store.rendezvous());
 
         BEAST_EXPECT(store.getLastRotated() == lastRotated);
 
@@ -448,7 +451,7 @@ public:
             BEAST_EXPECT(goodLedger(env, ledger, std::to_string(ledgerSeq++), true));
         }
 
-        store.rendezvous();
+        BEAST_EXPECT(store.rendezvous());
 
         ledgerCheck(env, ledgerSeq - lastRotated, lastRotated);
 
@@ -469,7 +472,7 @@ public:
             BEAST_EXPECT(goodLedger(env, ledger, std::to_string(ledgerSeq), true));
         }
 
-        store.rendezvous();
+        BEAST_EXPECT(store.rendezvous());
 
         BEAST_EXPECT(store.getLastRotated() == lastRotated);
 
@@ -481,7 +484,7 @@ public:
             BEAST_EXPECT(goodLedger(env, ledger, std::to_string(ledgerSeq++), true));
         }
 
-        store.rendezvous();
+        BEAST_EXPECT(store.rendezvous());
 
         ledgerCheck(env, ledgerSeq - lastRotated, lastRotated);
 
@@ -604,6 +607,173 @@ public:
         BEAST_EXPECT(dbr->getName() == "3");
     }
 
+    void
+    testLedgerGaps()
+    {
+        // Note that this test is intentionally very similar to
+        // LedgerMaster_test::testCompleteLedgerRange, but has a different
+        // focus.
+
+        testcase("Wait for ledger gaps to fill in");
+
+        using namespace test::jtx;
+
+        Env env{*this, envconfig(onlineDelete)};
+
+        auto failureMessage = [&](char const* label, auto expected, auto actual) {
+            std::stringstream ss;
+            ss << label << ": Expected: " << expected << ", Got: " << actual;
+            return ss.str();
+        };
+
+        auto const alice = Account("alice");
+        env.fund(XRP(1000), alice);
+        env.close();
+
+        auto& lm = env.app().getLedgerMaster();
+        LedgerIndex minSeq = 2;
+        LedgerIndex maxSeq = env.closed()->header().seq;
+        auto& store = env.app().getSHAMapStore();
+        LedgerIndex lastRotated = store.getLastRotated();
+        while (lastRotated != 3)
+        {
+            BEAST_EXPECT(store.rendezvous());
+            lastRotated = store.getLastRotated();
+        }
+        BEAST_EXPECTS(maxSeq == 3, std::to_string(maxSeq));
+        BEAST_EXPECTS(lm.getCompleteLedgers() == "2-3", lm.getCompleteLedgers());
+        BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq, maxSeq) == 0);
+        BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq + 1, maxSeq - 1) == 0);
+        BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq - 1, maxSeq + 1) == 2);
+        BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq - 2, maxSeq - 2) == 2);
+        BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq + 2, maxSeq + 2) == 2);
+
+        // Close enough ledgers to rotate a few times
+        while (maxSeq < 20)
+        {
+            for (int t = 0; t < 3; ++t)
+            {
+                env(noop(alice));
+            }
+            env.close();
+            BEAST_EXPECT(store.rendezvous());
+
+            ++maxSeq;
+
+            if (maxSeq + 1 == lastRotated + kDeleteInterval)
+            {
+                using namespace std::chrono_literals;
+
+                // The next ledger will trigger a rotation. Delete the
+                // current ledger from LedgerMaster.
+                std::this_thread::sleep_for(100ms);
+                LedgerIndex const deleteSeq = maxSeq;
+                {
+                    std::size_t iterations = 30;
+                    while (!lm.haveLedger(deleteSeq) && --iterations > 0)
+                    {
+                        std::this_thread::sleep_for(100ms);
+                    }
+                    // Even the slowest machines should be able to finalize deleteSeq within 4
+                    // loops (400ms). If this test ever actually fails feel free to lower this
+                    // cutoff.
+                    BEAST_EXPECTS(iterations > 25, to_string(iterations));
+                }
+                lm.clearLedger(deleteSeq);
+
+                auto expectedRange = [](auto minSeq, auto deleteSeq, auto maxSeq) {
+                    std::stringstream expectedRange;
+                    expectedRange << minSeq << "-" << (deleteSeq - 1);
+                    if (deleteSeq + 1 == maxSeq)
+                    {
+                        expectedRange << "," << maxSeq;
+                    }
+                    else if (deleteSeq < maxSeq)
+                    {
+                        expectedRange << "," << (deleteSeq + 1) << "-" << maxSeq;
+                    }
+                    return expectedRange.str();
+                };
+                BEAST_EXPECTS(
+                    lm.getCompleteLedgers() == expectedRange(minSeq, deleteSeq, maxSeq),
+                    failureMessage(
+                        "Complete ledgers",
+                        expectedRange(minSeq, deleteSeq, maxSeq),
+                        lm.getCompleteLedgers()));
+                BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq, maxSeq) == 1);
+
+                // Close another ledger, which will trigger a rotation, but the
+                // rotation will be stuck until the missing ledger is filled in.
+                env.close();
+                // Do not call rendezvous() here without a timeout; it will block until the missing
+                // ledger is backfilled. That will not happen automatically. It's a manual step that
+                // is done later in this test.
+                ++maxSeq;
+
+                // Nothing has changed
+                BEAST_EXPECTS(
+                    store.getLastRotated() == lastRotated,
+                    failureMessage("lastRotated", lastRotated, store.getLastRotated()));
+                BEAST_EXPECTS(
+                    lm.getCompleteLedgers() == expectedRange(minSeq, deleteSeq, maxSeq),
+                    failureMessage(
+                        "Complete ledgers",
+                        expectedRange(minSeq, deleteSeq, maxSeq),
+                        lm.getCompleteLedgers()));
+
+                // Close 5 more ledgers, waiting a little bit in between to
+                // simulate the ledger making progress while online delete waits
+                // for the missing ledger to be filled in.
+                // This ensures the healthWait check has time to run and
+                // detect the gap.
+                for (int l = 0; l < 5; ++l)
+                {
+                    env.close();
+                    ++maxSeq;
+                    // Nothing has changed
+                    BEAST_EXPECTS(
+                        store.getLastRotated() == lastRotated,
+                        failureMessage("lastRotated", lastRotated, store.getLastRotated()));
+                    BEAST_EXPECTS(
+                        lm.getCompleteLedgers() == expectedRange(minSeq, deleteSeq, maxSeq),
+                        failureMessage(
+                            "Complete Ledgers",
+                            expectedRange(minSeq, deleteSeq, maxSeq),
+                            lm.getCompleteLedgers()));
+                    // The Store is "stuck" in healthWait() and won't finish the run() loop until
+                    // it's backfilled
+                    BEAST_EXPECT(!store.rendezvous(100ms));
+                }
+
+                // Put the missing ledger back in LedgerMaster
+                lm.setLedgerRangePresent(deleteSeq, deleteSeq);
+
+                // Wait for the rotation to finish
+                BEAST_EXPECT(store.rendezvous());
+
+                minSeq = lastRotated;
+                lastRotated = deleteSeq + 1;
+            }
+            BEAST_EXPECT(maxSeq != lastRotated + kDeleteInterval);
+            BEAST_EXPECTS(
+                env.closed()->header().seq == maxSeq,
+                failureMessage("maxSeq", maxSeq, env.closed()->header().seq));
+            BEAST_EXPECTS(
+                store.getLastRotated() == lastRotated,
+                failureMessage("lastRotated", lastRotated, store.getLastRotated()));
+            std::stringstream expectedRange;
+            expectedRange << minSeq << "-" << maxSeq;
+            BEAST_EXPECTS(
+                lm.getCompleteLedgers() == expectedRange.str(),
+                failureMessage("CompleteLedgers", expectedRange.str(), lm.getCompleteLedgers()));
+            BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq, maxSeq) == 0);
+            BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq + 1, maxSeq - 1) == 0);
+            BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq - 1, maxSeq + 1) == 2);
+            BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq - 2, maxSeq - 2) == 2);
+            BEAST_EXPECT(lm.missingFromCompleteLedgerRange(minSeq + 2, maxSeq + 2) == 2);
+        }
+    }
+
     void
     run() override
     {
@@ -611,6 +781,7 @@ public:
         testAutomatic();
         testCanDelete();
         testRotate();
+        testLedgerGaps();
     }
 };
 

src/test/jtx/envconfig.h

@@ -3,6 +3,7 @@
 #include <xrpld/core/Config.h>
 
 #include <atomic>
+#include <cstdint>
 #include <map>
 #include <memory>
 #include <string>
@@ -56,6 +57,17 @@ envconfig(F&& modfunc, Args&&... args)
     return modfunc(envconfig(), std::forward<Args>(args)...);
 }
 
+/// @brief adjust config to enable online_delete
+///
+/// @param cfg config instance to be modified
+///
+/// @param deleteInterval how many new ledgers should be available before
+/// rotating. Defaults to 8, because the standalone minimum is 8.
+///
+/// @return unique_ptr to Config instance
+std::unique_ptr<Config>
+onlineDelete(std::unique_ptr<Config> cfg, std::uint32_t deleteInterval = 8);
+
 /// @brief adjust config so no admin ports are enabled
 ///
 /// this is intended for use with envconfig, as in

src/test/jtx/impl/envconfig.cpp

@@ -7,8 +7,10 @@
 #include <xrpl/config/Constants.h>
 
 #include <atomic>
+#include <cstdint>
 #include <map>
 #include <memory>
+#include <string>
 #include <vector>
 
 namespace xrpl::test {
@@ -60,6 +62,15 @@ setupConfigForUnitTests(Config& cfg)
 
 namespace jtx {
 
+std::unique_ptr<Config>
+onlineDelete(std::unique_ptr<Config> cfg, std::uint32_t deleteInterval)
+{
+    cfg->ledgerHistory = deleteInterval;
+    auto& section = cfg->section(Sections::kNodeDatabase);
+    section.set(Keys::kOnlineDelete, std::to_string(deleteInterval));
+    return cfg;
+}
+
 std::unique_ptr<Config>
 noAdmin(std::unique_ptr<Config> cfg)
 {

src/xrpld/app/ledger/LedgerMaster.h

@@ -122,7 +122,10 @@ public:
     failedSave(std::uint32_t seq, uint256 const& hash);
 
     std::string
-    getCompleteLedgers();
+    getCompleteLedgers() const;
+
+    std::size_t
+    missingFromCompleteLedgerRange(LedgerIndex first, LedgerIndex last) const;
 
     /** Apply held transactions to the open ledger
         This is normally called as we close the ledger.
@@ -339,7 +342,7 @@ private:
     // A set of transactions to replay during the next close
     std::unique_ptr<LedgerReplay> replayData_;
 
-    std::recursive_mutex completeLock_;
+    std::recursive_mutex mutable completeLock_;
     RangeSet<std::uint32_t> completeLedgers_;
 
     // Publish thread is running.

src/xrpld/app/ledger/detail/LedgerMaster.cpp

@@ -57,6 +57,7 @@
 #include <xrpl/shamap/SHAMapMissingNode.h>
 #include <xrpl/shamap/SHAMapTreeNode.h>
 
+#include <boost/icl/concept/interval_associator.hpp>
 #include <boost/icl/concept/interval_set.hpp>
 
 #include <xrpl.pb.h>
@@ -1571,12 +1572,25 @@ LedgerMaster::getPublishedLedger()
 }
 
 std::string
-LedgerMaster::getCompleteLedgers()
+LedgerMaster::getCompleteLedgers() const
 {
     std::scoped_lock const sl(completeLock_);
     return to_string(completeLedgers_);
 }
 
+std::size_t
+LedgerMaster::missingFromCompleteLedgerRange(LedgerIndex first, LedgerIndex last) const
+{
+    RangeSet<LedgerIndex> const target{range(first, last)};
+
+    auto const missing = [&target, this] {
+        std::scoped_lock const sl(completeLock_);
+        return target - completeLedgers_;
+    }();
+
+    return boost::icl::size(missing);
+}
+
 std::optional<NetClock::time_point>
 LedgerMaster::getCloseTimeBySeq(LedgerIndex ledgerIndex)
 {

src/xrpld/app/misc/SHAMapStore.h

@@ -8,6 +8,7 @@
 #include <xrpl/nodestore/Scheduler.h>
 #include <xrpl/protocol/Protocol.h>
 
+#include <chrono>
 #include <cstdint>
 #include <memory>
 #include <optional>
@@ -32,8 +33,8 @@ public:
     virtual void
     start() = 0;
 
-    virtual void
-    rendezvous() const = 0;
+    [[nodiscard]] virtual bool
+    rendezvous(std::optional<std::chrono::milliseconds> const& timeout = {}) const = 0;
 
     virtual void
     stop() = 0;

src/xrpld/app/misc/SHAMapStoreImp.cpp

@@ -8,6 +8,7 @@
 #include <xrpl/basics/ByteUtilities.h>
 #include <xrpl/basics/Log.h>
 #include <xrpl/basics/contract.h>
+#include <xrpl/basics/scope.h>
 #include <xrpl/beast/core/CurrentThreadName.h>
 #include <xrpl/beast/utility/Journal.h>
 #include <xrpl/beast/utility/instrumentation.h>
@@ -30,6 +31,8 @@
 #include <boost/filesystem/path.hpp>
 
 #include <algorithm>
+#include <chrono>
+#include <cstddef>
 #include <cstdint>
 #include <functional>
 #include <limits>
@@ -234,14 +237,21 @@ SHAMapStoreImp::onLedgerClosed(std::shared_ptr<Ledger const> const& ledger)
     cond_.notify_one();
 }
 
-void
-SHAMapStoreImp::rendezvous() const
+bool
+SHAMapStoreImp::rendezvous(std::optional<std::chrono::milliseconds> const& timeout) const
 {
     if (!working_)
-        return;
+        return true;
+
+    auto notWorking = [&] { return !working_; };
 
     std::unique_lock<std::mutex> lock(mutex_);
-    rendezvous_.wait(lock, [&] { return !working_; });
+    if (timeout)
+    {
+        return rendezvous_.wait_for(lock, *timeout, notWorking);
+    }
+    rendezvous_.wait(lock, notWorking);
+    return true;
 }
 
 int
@@ -312,6 +322,16 @@ SHAMapStoreImp::run()
         bool const readyToRotate = validatedSeq >= lastRotated + deleteInterval_ &&
             canDelete_ >= lastRotated - 1 && healthWait() == HealthResult::KeepGoing;
 
+        {
+            JLOG(journal_.trace()) << "run: Setting lastGoodValidatedLedger_ to " << validatedSeq;
+            // Note that this is set after the healthWait() check, so that we
+            // don't start the rotation until the validated ledger is fully
+            // processed. It is not guaranteed to be done at this point. It also
+            // allows the testLedgerGaps unit test to work.
+            std::unique_lock<std::mutex> const lock(mutex_);
+            lastGoodValidatedLedger_ = validatedSeq;
+        }
+
         // will delete up to (not including) lastRotated
         if (readyToRotate)
         {
@@ -319,7 +339,8 @@ SHAMapStoreImp::run()
                                   << lastRotated << " deleteInterval " << deleteInterval_
                                   << " canDelete_ " << canDelete_ << " state "
                                   << app_.getOPs().strOperatingMode(false) << " age "
-                                  << ledgerMaster_->getValidatedLedgerAge().count() << 's';
+                                  << ledgerMaster_->getValidatedLedgerAge().count()
+                                  << "s. Complete ledgers: " << ledgerMaster_->getCompleteLedgers();
 
             clearPrior(lastRotated);
             if (healthWait() == HealthResult::Stopping)
@@ -377,7 +398,9 @@ SHAMapStoreImp::run()
                     clearCaches(validatedSeq);
                 });
 
-            JLOG(journal_.warn()) << "finished rotation " << validatedSeq;
+            JLOG(journal_.warn()) << "finished rotation. validatedSeq: " << validatedSeq
+                                  << ", lastRotated: " << lastRotated
+                                  << ". Complete ledgers: " << ledgerMaster_->getCompleteLedgers();
         }
     }
 }
@@ -621,20 +644,70 @@ SHAMapStoreImp::clearPrior(LedgerIndex lastRotated)
 SHAMapStoreImp::HealthResult
 SHAMapStoreImp::healthWait()
 {
-    auto age = ledgerMaster_->getValidatedLedgerAge();
-    OperatingMode mode = netOPs_->getOperatingMode();
-    std::unique_lock lock(mutex_);
-    while (!stop_ && (mode != OperatingMode::FULL || age > ageThreshold_))
-    {
-        lock.unlock();
-        JLOG(journal_.warn()) << "Waiting " << recoveryWaitTime_.count()
-                              << "s for node to stabilize. state: "
-                              << app_.getOPs().strOperatingMode(mode, false) << ". age "
-                              << age.count() << 's';
-        std::this_thread::sleep_for(recoveryWaitTime_);
+    // Gets the current status of the server from ledgerMaster_ and netOPs_. Must be called
+    // while mutex_ is unlocked to avoid unlikely, but possible, deadlock with ledgerMaster_'s
+    // completeLock_.
+    // Releasing the lock may mean that status will be slightly out of date when the lock is
+    // reacquired, but it's close enough. In a normal rotation, healthWait() is called frequently,
+    // so a false positive will be detected on the next call, and a false negative will be detected
+    // in the next loop iteration. Database rotation is important, but not timely, so an extra
+    // delay is fine.
+    auto readServerStatus = [this](
+                                LedgerIndex& index,
+                                std::chrono::seconds& age,
+                                OperatingMode& mode,
+                                std::size_t& numMissing,
+                                LedgerIndex const lowerBound,
+                                ScopeUnlock<decltype(mutex_)> const&) {
+        index = ledgerMaster_->getValidLedgerIndex();
         age = ledgerMaster_->getValidatedLedgerAge();
         mode = netOPs_->getOperatingMode();
-        lock.lock();
+
+        numMissing =
+            lowerBound == 0 ? 0 : ledgerMaster_->missingFromCompleteLedgerRange(lowerBound, index);
+    };
+
+    // Tracked server status properties
+    LedgerIndex index = 0;
+    std::chrono::seconds age;
+    OperatingMode mode = OperatingMode::DISCONNECTED;
+    std::size_t numMissing = 0;
+    LedgerIndex lastLedger = 0;
+
+    std::unique_lock lock(mutex_);
+
+    auto const waitTime = recoveryWaitTime_;
+    auto const ageThreshold = ageThreshold_;
+    {
+        auto const lowerBound = lastGoodValidatedLedger_;
+
+        ScopeUnlock const unlock(lock);
+
+        readServerStatus(index, age, mode, numMissing, lowerBound, unlock);
+    }
+    while (!stop_ && (mode != OperatingMode::FULL || age > ageThreshold || numMissing > 0))
+    {
+        // this value shouldn't change, so grab it while we have the
+        // lock
+        auto const lowerBound = lastGoodValidatedLedger_;
+
+        ScopeUnlock const unlock(lock);
+
+        auto const stream = std::invoke([mode, age, ageThreshold, index, lastLedger, this]() {
+            if (mode != OperatingMode::FULL || age > ageThreshold)
+                return journal_.warn();
+            if (index != lastLedger)
+                return journal_.trace();
+            return journal_.info();
+        });
+        JLOG(stream) << "Waiting " << waitTime.count() << "s for node to stabilize. state: "
+                     << app_.getOPs().strOperatingMode(mode, false) << ". age " << age.count()
+                     << "s. Missing ledgers: " << numMissing << ".  Expect: " << lowerBound << "-"
+                     << index << ". Complete ledgers: " << ledgerMaster_->getCompleteLedgers();
+        std::this_thread::sleep_for(waitTime);
+
+        readServerStatus(index, age, mode, numMissing, lowerBound, unlock);
+        lastLedger = index;
     }
 
     return stop_ ? HealthResult::Stopping : HealthResult::KeepGoing;

src/xrpld/app/misc/SHAMapStoreImp.h

@@ -88,6 +88,11 @@ private:
     std::thread thread_;
     bool stop_ = false;
     bool healthy_ = true;
+    // Used to prevent ledger gaps from forming during online deletion. Keeps
+    // track of the last validated ledger that was processed without gaps. There
+    // are no guarantees about gaps while online delete is not running. For
+    // that, use advisory_delete and check for gaps externally.
+    LedgerIndex lastGoodValidatedLedger_ = 0;
     mutable std::condition_variable cond_;
     mutable std::condition_variable rendezvous_;
     mutable std::mutex mutex_;
@@ -101,11 +106,11 @@ private:
     std::uint32_t deleteBatch_ = 100;
     std::chrono::milliseconds backOff_{100};
     std::chrono::seconds ageThreshold_{60};
-    /// If  the node is out of sync during an online_delete healthWait()
-    /// call, sleep the thread for this time, and continue checking until
-    /// recovery.
+    /// If  the node is out of sync, or any recent ledgers are not
+    /// available during an online_delete healthWait() call, sleep
+    /// the thread for this time, and continue checking until recovery.
     /// See also: "recovery_wait_seconds" in xrpld-example.cfg
-    std::chrono::seconds recoveryWaitTime_{5};
+    std::chrono::seconds recoveryWaitTime_{2};
 
     // these do not exist upon SHAMapStore creation, but do exist
     // as of run() or before
@@ -161,8 +166,8 @@ public:
     void
     onLedgerClosed(std::shared_ptr<Ledger const> const& ledger) override;
 
-    void
-    rendezvous() const override;
+    bool
+    rendezvous(std::optional<std::chrono::milliseconds> const& timeout = {}) const override;
     int
     fdRequired() const override;