Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill

docs/build/environment.md

@@ -33,9 +33,10 @@ with a single command and without installing anything system-wide:
 nix --experimental-features 'nix-command flakes' develop
 ```
 
-On **Linux**, Nix also provides the compiler (GCC). On **macOS**, the shell uses
-your **system-wide Apple Clang** as the compiler, so you still need to manage
-its version (see below).
+On **Linux**, Nix also provides the compiler (GCC); on **macOS**, it provides
+Clang. If you instead opt to use your system-wide Apple Clang (via
+`nix develop .#apple-clang`), you need to manage its version yourself (see
+below).
 
 See [Using the Nix development shell](./nix.md) for installation and usage
 details, including how to select a different compiler.
@@ -48,10 +49,10 @@ details, including how to select a different compiler.
 
 ### macOS: managing the Apple Clang version
 
-Because the Nix shell uses the system-wide Apple Clang on macOS, the compiler
-version is whatever your installed Xcode (or Command Line Tools) provides. The
-following command should return a version greater than or equal to the
-[minimum required](#tested-compiler-versions):
+If you use your system-wide Apple Clang on macOS (via `nix develop .#apple-clang`),
+the compiler version is whatever your installed Xcode (or Command Line Tools)
+provides. The following command should return a version greater than or equal to
+the [minimum required](#tested-compiler-versions):
 
 ```bash
 clang --version

docs/build/nix.md

@@ -9,7 +9,7 @@ This guide explains how to use Nix to set up a reproducible development environm
 - **Reproducible environment**: Everyone gets the same versions of tools and compilers
 - **Matches CI**: The Linux CI runs in Docker images built from this exact Nix environment
 - **No system pollution**: Dependencies are isolated and don't affect your system packages
-- **Multiple compiler versions**: Easily switch between different GCC and Clang versions
+- **Consistent compilers**: The GCC and Clang shells use the same versions as CI
 - **Quick setup**: Get started with a single command
 - **Works on Linux and macOS**: Consistent experience across platforms
 
@@ -31,8 +31,8 @@ This will:
 
 - Download and set up all required development tools (CMake, Ninja, Conan, etc.)
 - Configure the appropriate compiler for your platform:
-  - **Linux**: GCC 15.2 (provided by Nix)
-  - **macOS**: Apple Clang (your system compiler)
+  - **Linux**: GCC (provided by Nix)
+  - **macOS**: Clang (provided by Nix)
 
 The first time you run this command, it will take a few minutes to download and build the environment. Subsequent runs will be much faster.
 
@@ -40,12 +40,12 @@ The first time you run this command, it will take a few minutes to download and
 
 - **Linux**: `nix develop` gives you a shell with all the tooling necessary to
   develop xrpld and with GCC 15.2 (also provided by Nix). There are no caveats.
-- **macOS**: `nix develop` gives you a full environment too. The compiler is
-  your system-wide Apple Clang, while every other tool — including Conan — is
-  provided by Nix. Conan has no binary in the Nix cache for macOS, so it is
-  built from source the first time you enter the shell, which makes the initial
-  setup slower (this is handled automatically; see
-  [`nix/devshell.nix`](../../nix/devshell.nix)).
+- **macOS**: `nix develop` gives you a full environment too, with Clang (and
+  every other tool, including Conan) provided by Nix. To use your system-wide
+  Apple Clang instead, enter `nix develop .#apple-clang`. Conan has no binary in
+  the Nix cache for macOS, so it is built from source the first time you enter
+  the shell, which makes the initial setup slower (this is handled
+  automatically; see [`nix/devshell.nix`](../../nix/devshell.nix)).
 
 > [!TIP]
 > To avoid typing `--experimental-features 'nix-command flakes'` every time, you can permanently enable flakes by creating `~/.config/nix/nix.conf`:
@@ -62,7 +62,9 @@ The first time you run this command, it will take a few minutes to download and
 
 ### Choosing a different compiler
 
-A compiler can be chosen by providing its name with the `.#` prefix, e.g. `nix develop .#gcc15`.
+A compiler can be chosen by providing its name with the `.#` prefix, e.g. `nix develop .#clang`.
+The `.#gcc` and `.#clang` shells provide the same GCC and Clang versions used in CI
+(pinned in [`nix/packages.nix`](../../nix/packages.nix)).
 Use `nix flake show` to see all the available development shells.
 
 Use `nix develop .#no-compiler` to use the compiler from your system.
@@ -70,11 +72,11 @@ Use `nix develop .#no-compiler` to use the compiler from your system.
 ### Example Usage
 
 ```bash
-# Use GCC 14
-nix develop .#gcc14
+# Use GCC (same version as CI)
+nix develop .#gcc
 
-# Use Clang 19
-nix develop .#clang19
+# Use Clang (same version as CI)
+nix develop .#clang
 
 # Use default for your platform
 nix develop
@@ -112,7 +114,15 @@ Once inside the Nix development shell, follow the standard [build instructions](
 
 [direnv](https://direnv.net/) or [nix-direnv](https://github.com/nix-community/nix-direnv) can automatically activate the Nix development shell when you enter the repository directory.
 
-This is also the most robust way to use the environment from **any shell** (bash, zsh, fish, …): direnv stays in your current shell and loads the environment _after_ your shell's startup files have run, so the Nix-provided tools take precedence over anything your shell configuration adds to `$PATH`. To use it, install direnv for your shell, then add an `.envrc` containing `use flake` at the repository root and run `direnv allow`.
+This is also the most robust way to use the environment from **any shell** (bash, zsh, fish, …): direnv stays in your current shell and loads the environment _after_ your shell's startup files have run, so the Nix-provided tools take precedence over anything your shell configuration adds to `$PATH`.
+
+The repository already ships an `.envrc` at its root that activates the Nix flake development shell, so you don't need to create one. To use it:
+
+1. [Install direnv](https://direnv.net/docs/installation.html) and [hook it into your shell](https://direnv.net/docs/hook.html) (bash, zsh, fish, …). Installing [nix-direnv](https://github.com/nix-community/nix-direnv) as well is recommended: it caches the shell so that activation is near-instant after the first run.
+2. Run `direnv allow` once in the repository root. direnv will then load (and reload) the Nix development shell automatically whenever you enter the directory.
+
+> [!NOTE]
+> direnv only caches the `.direnv` directory (already listed in `.gitignore`); no other repository files are affected.
 
 ## Conan and Prebuilt Packages
 

docs/build/telemetry.md

@@ -15,6 +15,7 @@ This document explains how to build xrpld with OpenTelemetry distributed tracing
     - [Conan lockfile error](#conan-lockfile-error)
     - [CMake target not found](#cmake-target-not-found)
   - [Conditional compilation](#conditional-compilation)
+  - [Span lifetime and cross-thread handling](#span-lifetime-and-cross-thread-handling)
 
 ## Overview
 
@@ -127,3 +128,34 @@ no-op stub class with zero overhead and zero OTel dependencies.
 At runtime, if `enabled=0` is set in config (or the section is omitted), a
 `NullTelemetry` implementation is used that returns no-op spans.
 This two-layer approach ensures zero overhead when telemetry is not wanted.
+
+## Span lifetime and cross-thread handling
+
+`SpanGuard` is an RAII wrapper: it activates a span on the current thread when
+constructed and ends it when destroyed. A normal (scoped) guard binds an OTel
+`Scope` to the constructing thread's thread-local context stack. Because of
+this, a scoped guard must be used and destroyed on the thread that created it.
+
+Two entry points cover the cases where the default scoped behaviour is wrong:
+
+- **`SpanGuard::rootSpan(cat, prefix, name)`** starts a span as a fresh trace
+  root, ignoring whatever span is already active on the current thread. Use it
+  at an inbound entry point that runs on a shared worker thread (for example a
+  peer message received while unrelated work is still on the stack), so those
+  unrelated spans do not become parents of the new trace.
+
+- **`std::move(guard).detached()`** returns a new guard that holds the same
+  span with **no** thread-local `Scope`. Call it on the origin thread before
+  handing the guard to a job that runs on another thread. The returned guard
+  can be moved freely across threads and ended anywhere; only its final
+  destruction ends the span.
+
+### Why are unrelated spans in my trace?
+
+If a trace shows hundreds of unrelated spans nested under one operation, a
+scoped guard was likely moved to another thread (for example stored in a job
+and destroyed on a worker thread). Destroying a scoped guard off its origin
+thread pops the wrong context stack, leaving the origin thread's active span
+in place so later spans inherit it. Fix it by calling `.detached()` on the
+origin thread before the hand-off, or by starting the operation with
+`rootSpan()` so it never inherits an ambient parent.

docs/telemetry-runbook.md

@@ -179,6 +179,12 @@ Span attributes are filtered with `span.<attr>` inside `{}`. Combine conditions
 | `peer.proposal.receive`   | PeerImp.cpp:1667 | `peer_id`, `proposal_trusted`   | Proposal received from peer   |
 | `peer.validation.receive` | PeerImp.cpp:2264 | `peer_id`, `validation_trusted` | Validation received from peer |
 
+Both peer receive spans are `kConsumer` inbound entry points started as fresh
+trace roots. They never inherit an ambient span left active on the peer thread,
+so they do not nest under an unrelated transaction's trace. The distributed
+child span that links back to the sending node is the separate
+`consensus.*.receive` / `tx.receive` span (see Cross-Node Trace Propagation).
+
 ---
 
 ## Insights and Sample Queries