build: Run nix macos builds in CI; deny nix store references (#8023)

docs/build/nix.md

@@ -7,7 +7,7 @@ This guide explains how to use Nix to set up a reproducible development environm
 ## Benefits of Using Nix
 
 - **Reproducible environment**: Everyone gets the same versions of tools and compilers
-- **Matches CI**: The Linux CI runs in Docker images built from this exact Nix environment
+- **Matches CI**: The Linux CI runs in Docker images built from this exact Nix environment, and CI builds some macOS configurations in it as well
 - **No system pollution**: Dependencies are isolated and don't affect your system packages
 - **Consistent compilers**: The GCC and Clang shells use the same versions as CI
 - **Quick setup**: Get started with a single command
@@ -68,7 +68,7 @@ A compiler can be chosen by providing its name with the `.#` prefix, e.g. `nix d
 
 On Linux, `.#gcc` and `.#clang` provide the exact toolchain CI uses:
 the compiler (pinned in [`nix/packages.nix`](../../nix/packages.nix))
-rebuilt against the pinned custom glibc (see [`nix/compilers.nix`](../../nix/compilers.nix)).
+rebuilt against the pinned custom glibc (see [`nix/linux.nix`](../../nix/linux.nix)).
 Building that toolchain the first time is slow unless it is fetched from a Nix binary cache.
 If you don't need the custom glibc, the Linux-only `.#gcc-plain` and `.#clang-plain`
 give you the stock nixpkgs compilers of the same versions.
@@ -142,14 +142,80 @@ environment — CI runs in Docker images that bundle the dev shell's toolchain (
 `-plain` shells do not match that toolchain's glibc, so binaries from the remote
 are not a reliable match there.
 
-On **macOS**, CI builds with Apple Clang, so the remote holds nothing for the Nix
-`clang` toolchain and dependencies are compiled locally. We do not publish
-Nix-built macOS binaries because a Conan package ID records the compiler version
-but not the nixpkgs revision.
+On **macOS**, CI also builds in this Nix environment, in Debug and Release (the
+`macos-arm64-*-nix` configurations — Debug because the profile defaults to it).
+The Nix build resolves to `compiler=clang`, so it gets its own package IDs,
+separate from the Apple Clang ones. The
+[dependency upload](../../.github/workflows/upload-conan-deps.yml) publishes them
+on pushes to `develop` and on manual runs — its nightly run rebuilds everything
+from source but uploads nothing — so once a set has been published `nix develop`
+can reuse it instead of compiling every dependency locally. These configurations
+run outside the reduced pull-request matrix, so label a PR `Full CI build` when it
+touches `flake.lock` or `nix/`.
 
 To compile everything from source, add `--build '*'` to the `conan install`
 command.
 
+### Why the nixpkgs revision is not part of the package ID
+
+A Conan package ID records the compiler and its major version, but nothing about
+the nixpkgs revision the toolchain came from — and `flake.lock` moves far more
+often than the toolchain meaningfully changes, so folding it in would rebuild
+every dependency on every bump for nothing.
+
+That is safe as long as no cached artifact resolves a `/nix/store` path at run
+time, because store paths change on every update and the old ones disappear with
+`nix-collect-garbage`. With the `clang` toolchain macOS CI and the dev shell use,
+they do not: it links against `/usr/lib/libc++` and `/usr/lib/libSystem`, and
+store paths reach the `.a` files only through debug info, which nothing resolves
+at link or run time.
+
+> [!WARNING]
+> This does not hold for `nix develop .#gcc` on macOS. There is no system
+> libstdc++, so GCC links its own from the store and every binary keeps a
+> `/nix/store` reference. That shell is fine for tooling, but it is not a build
+> configuration CI covers, and no dependency binaries are published for it.
+
+This is checked rather than assumed.
+[`bin/check-nix-store-refs.sh`](../../bin/check-nix-store-refs.sh) takes one file
+or directory and fails if a binary under it resolves a store path at run time.
+CI runs it over the build output and the Conan cache, and again in the upload job
+before anything is published. You can run it yourself:
+
+```bash
+bin/check-nix-store-refs.sh build
+bin/check-nix-store-refs.sh ~/.conan2-nix
+```
+
+It works on Linux too, but asserts something narrower there: the toolchain always
+writes the store into `PT_INTERP` and `RUNPATH`, and CI builds inside an image
+whose store is fixed for its lifetime, so that is fine. Only the binaries
+[`PatchNixBinary.cmake`](../../cmake/PatchNixBinary.cmake) retargets to the
+system loader have to be clean, and those are what CI checks:
+
+```bash
+bin/check-nix-store-refs.sh build/xrpld
+```
+
+### The libresolv stub
+
+This is not hypothetical: `xrpld` used to be caught by it. The c-ares package
+tells the linker to pass `-lresolv`, and nixpkgs keeps `libresolv` out of the
+macOS SDK and ships it as an ordinary store dylib — so every Nix-built `xrpld`
+recorded a `/nix/store/…-libresolv-93/lib/libresolv.9.dylib` load command and
+stopped running once that path was collected. Nothing in the link uses a single
+symbol from it.
+
+Both environments now put a stub on the linker search path
+(`libresolvSystemStub` in [`nix/darwin.nix`](../../nix/darwin.nix)): the
+same library with its install name set to `/usr/lib/libresolv.9.dylib`, which is
+exactly the load command the Apple Clang build records.
+
+Package IDs did not change, so Conan keeps serving anything built before the
+stub landed. If a binary fails to start with `Library not loaded: /nix/store/…`,
+see [that entry](./nix_troubleshooting.md#library-not-loaded-nixstore-from-a-binary-that-used-to-work)
+in the troubleshooting guide.
+
 ## Automatic Activation with direnv
 
 [direnv](https://direnv.net/) or [nix-direnv](https://github.com/nix-community/nix-direnv) can automatically activate the Nix development shell when you enter the repository directory.

docs/build/nix_troubleshooting.md

@@ -131,3 +131,91 @@ once it picks up that rebuild, then re-run the `grep libgit2` check above to
 confirm it reports `1.9.4` or newer.
 
 Until then, prefer the workarounds above.
+
+## `wint_t` / `uint32_t` errors from the Nix libc++ headers
+
+A build that mixes the Nix toolchain with the system SDK fails in libc++ itself,
+with errors that look nothing like your code:
+
+```
+/nix/store/...-libcxx-.../include/c++/v1/cwchar:136:9: error: target of using declaration conflicts with declaration already in scope
+  136 | using ::wint_t _LIBCPP_USING_IF_EXISTS;
+/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/sys/_types/_wint_t.h:32:25: note: target of using declaration
+...
+error: use of undeclared identifier 'UINT32_C'
+```
+
+The give-away is the second path: Nix's libc++ headers are being combined with
+the **Xcode Command Line Tools** SDK instead of the Nix one.
+
+### Why it happens
+
+`SDKROOT` and `DEVELOPER_DIR` are what point the toolchain at the Nix SDK, and
+they are not baked into the compiler — a dev shell gets them from the
+`apple-sdk` setup hook. CMake, finding neither, asks `xcrun`, which answers with
+the system SDK. Nix's `libc++` and Apple's headers then declare the same types
+twice.
+
+### Fix
+
+Run the build from inside the dev shell (`nix develop`), or from an environment
+that exports both variables. To confirm which SDK a configured build is using:
+
+```bash
+grep -o '\-isysroot [^ ]*' build/compile_commands.json | sort -u
+```
+
+It should print a `/nix/store/...-apple-sdk-*` path. If it prints
+`/Library/Developer/CommandLineTools/...`, re-configure from within the shell —
+CMake caches the sysroot, so an existing `build/` directory keeps the wrong one.
+
+## `Library not loaded: /nix/store/…` from a binary that used to work
+
+A binary stops starting after a `nix flake update`, or after
+`nix-collect-garbage` removes the paths the previous toolchain used:
+
+```
+dyld[57271]: Library not loaded: /nix/store/…-libresolv-93/lib/libresolv.9.dylib
+```
+
+[`bin/check-nix-store-refs.sh`](../../bin/check-nix-store-refs.sh) finds the same
+thing without having to run anything, and names the file:
+
+```
+$ bin/check-nix-store-refs.sh ~/.conan2-nix
+::error file=/Users/you/.conan2-nix/p/b/c-area24ded30c388c/p/bin/adig::references the Nix store at run time
+/Users/you/.conan2-nix/p/b/c-area24ded30c388c/p/bin/adig
+    /nix/store/p4lp3xq4imd1qzqh08x8vcq2zfhi7rca-libresolv-93/lib/libresolv.9.dylib
+/Users/you/.conan2-nix: checked 135, skipped 2495, 1 with Nix store references.
+```
+
+Conan's cache folders are named after a truncated package name plus a hash, so
+ask Conan which package the offending one belongs to — pass the folder holding
+the hash, not the file itself:
+
+```
+$ conan cache ref ~/.conan2-nix/p/b/c-area24ded30c388c
+c-ares/1.34.6#545240bb1c40e2cacd4362d6b8967650:dab5992496abe6d219defb7986ecbf367615a5e5#…
+```
+
+### Why it happens
+
+The binary records a store path that no longer exists. Nothing we build should:
+see [Prebuilt packages](./nix.md#prebuilt-packages) for why, and
+`libresolvSystemStub` in [`nix/darwin.nix`](../../nix/darwin.nix) for the one
+dependency that needed help to comply.
+
+A Conan package ID does not encode the nixpkgs revision, so a package built
+before that stub existed stays in your local cache and keeps being reused. The
+dev shell is also what tends to produce one: it is a slightly _less_ isolated
+build environment than CI's, because `mkShell` puts every tool's headers and
+libraries on the compiler's search path — which is how c-ares found the Nix
+`libresolv` in the first place.
+
+### Fix
+
+Drop that package and let Conan refetch or rebuild it:
+
+```bash
+conan remove 'c-ares/*'
+```