Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation

docs/build/nix_troubleshooting.md

@@ -3,6 +3,78 @@
 Common issues encountered when using the [Nix development shell](./nix.md), and
 how to resolve them.
 
+## `command not found: nix` after a macOS update
+
+If a shell suddenly can't find `nix` at all:
+
+```
+$ nix develop
+zsh: command not found: nix
+```
+
+then Nix is almost certainly still installed — only the shell hook that puts it
+on your `PATH` is gone. Confirm that first:
+
+```bash
+ls -l /nix/var/nix/profiles/default/bin/nix
+```
+
+If that exists, the installation is fine and this is purely a `PATH` problem.
+
+### Why it happens
+
+The installer does not touch your dotfiles. Instead it sources a setup script
+from the Nix store by editing **system-wide** rc files:
+
+| Shell | File the installer edits              |
+| ----- | ------------------------------------- |
+| bash  | `/etc/bashrc`, `/etc/bash.bashrc`     |
+| zsh   | `/etc/zshrc`                          |
+| fish  | `$__fish_sysconf_dir/conf.d/nix.fish` |
+
+macOS manages `/etc/zshrc`, so an OS update can replace it with the vendor copy
+and silently drop the Nix block. `/etc/bashrc` and the fish file usually survive,
+which is why the breakage often shows up in zsh only. You can verify this by
+diffing against the backup the installer left behind:
+
+```bash
+diff /etc/zshrc /etc/zshrc.backup-before-nix
+```
+
+If they are identical, the Nix snippet was wiped. This is upstream issue
+[NixOS/nix#3616](https://github.com/NixOS/nix/issues/3616).
+
+### Fix
+
+To unblock the current shell:
+
+```bash
+. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh
+```
+
+For a permanent fix, add the snippet to your **user** rc file rather than
+restoring `/etc/zshrc` — user dotfiles are not clobbered by OS updates:
+
+```bash
+cat >>~/.zshrc <<'EOF'
+
+# Nix
+if [ -e '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh' ]; then
+  . '/nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh'
+fi
+# End Nix
+EOF
+```
+
+The scripts guard against double-sourcing via `__ETC_PROFILE_NIX_SOURCED`, so
+this is safe even if a system-wide hook is later restored.
+
+> [!NOTE]
+> `/etc/zshrc` and `~/.zshrc` are only read by **interactive** zsh. If the
+> snippet is present but `zsh -c '…'`, a script, or an IDE terminal still can't
+> find `nix`, that shell is non-interactive — put the snippet in `~/.zshenv`
+> instead.
+
 ## Git worktrees
 
 If `nix develop` fails with an error like:

docs/build/telemetry.md

@@ -59,7 +59,7 @@ cd .build
 
 #### Install dependencies
 
-The `telemetry` option adds `opentelemetry-cpp/1.26.0` as a dependency.
+The `telemetry` option adds `opentelemetry-cpp/1.28.0` as a dependency.
 If the Conan lockfile does not yet include this package, bypass it with `--lockfile=""`.
 
 ```bash
@@ -112,7 +112,7 @@ The resulting binary is identical to one built before telemetry support was adde
 
 ### Conan lockfile error
 
-If you see `ERROR: Requirement 'opentelemetry-cpp/1.26.0' not in lockfile 'requires'`,
+If you see `ERROR: Requirement 'opentelemetry-cpp/1.28.0' not in lockfile 'requires'`,
 the lockfile was generated without the telemetry dependency.
 Pass `--lockfile=""` to bypass the lockfile, or regenerate it with telemetry enabled.
 

docs/telemetry-runbook.md

@@ -2386,8 +2386,10 @@ after the selector and cannot be discovered by `label_values()`.
 # Find all logs for a specific trace
 {service_name="xrpld"} |= "trace_id=abc123def456789012345678abcdef01"
 
-# Error logs with trace context (log lines with ERR severity that have a trace_id)
-{service_name="xrpld"} |= "ERR" |= "trace_id="
+# Error logs with trace context (log lines with ERR severity that have a trace_id).
+# Use the severity field, not `|= "ERR"`: a line filter also matches the literal
+# "ERR" anywhere in the message body (measured: 4 DBG lines per 6h on devnet).
+{service_name="xrpld"} | severity = `ERR` | trace_id != ""
 
 # All logs from a specific partition that were emitted during a span.
 # Prefer the structured-metadata filter over a line match: `|= "LedgerMaster"`
@@ -2397,11 +2399,14 @@ after the selector and cannot be discovered by `label_values()`.
 # Logs from a specific subsystem during a span (e.g. LedgerConsensus)
 {service_name="xrpld"} | partition = `LedgerConsensus` | trace_id != ""
 
-# Logs from the last hour containing trace context
-{service_name="xrpld"} |= "trace_id=" | regexp `(?P<partition>\S+):(?P<sev>\S+)\s+trace_id=(?P<tid>[a-f0-9]+)`
+# Logs from the last hour containing trace context. `partition`, `severity`, and
+# `trace_id` are already parsed into structured metadata by the collector's
+# filelog receiver, so re-extracting them with regexp is unnecessary work.
+{service_name="xrpld"} | trace_id != ""
 
 # Count of traced vs untraced log lines
-count_over_time({service_name="xrpld"} |= "trace_id=" [5m])
+sum(count_over_time({service_name="xrpld"} | trace_id != "" [5m]))
+sum(count_over_time({service_name="xrpld"} | trace_id = "" [5m]))
 ```
 
 ### Verifying Log Correlation
@@ -2409,7 +2414,7 @@ count_over_time({service_name="xrpld"} |= "trace_id=" [5m])
 1. Start the observability stack and xrpld with telemetry enabled.
 2. Send an RPC request: `curl http://localhost:5005 -d '{"method":"server_info"}'`
 3. Check the debug.log for `trace_id=` entries: `grep trace_id= /path/to/debug.log`
-4. Open Grafana at http://localhost:3000 -> Explore -> Loki and search for `{service_name="xrpld"} |= "trace_id="`.
+4. Open Grafana at http://localhost:3000 -> Explore -> Loki and search for `{service_name="xrpld"} | trace_id != ""`.
 5. Click the TraceID link to navigate to the corresponding trace in Tempo.
 
 ### Log-Derived Insights (`log-derived-insights`)