diff --git a/docker/telemetry/.env.devbox.example b/docker/telemetry/.env.devbox.example index 782808d2e8..a7928b9799 100644 --- a/docker/telemetry/.env.devbox.example +++ b/docker/telemetry/.env.devbox.example @@ -14,3 +14,15 @@ REPO_DIR= # Mount holding the node data directories. The units require it, so a missing # disk fails the unit instead of silently filling the root filesystem. DATA_MOUNT= + +# Telemetry identity of each node, reported as the service_instance_id label on +# every metric, span and log line, and used as the log directory name. +# +# Name them after the machine, so a dashboard shows which box the data came +# from -- a host running two nodes needs two distinct values. Changing them +# breaks continuity with data already stored under the old names, so reuse the +# host's established names rather than inventing new ones. +# +# Allowed characters: letters, digits, dot, underscore, hyphen. +NODE1_INSTANCE_ID= +NODE2_INSTANCE_ID= diff --git a/docker/telemetry/.gitignore b/docker/telemetry/.gitignore index dce518044f..366d329bf4 100644 --- a/docker/telemetry/.gitignore +++ b/docker/telemetry/.gitignore @@ -4,6 +4,10 @@ data/ # Env. file carrying environmental setup data for local or cloud runs. .env.* +# Host-local configs rendered by systemd/install-units.sh. They carry the +# machine's telemetry identity, which must not be committed. +*.host.cfg + # Keep examples !.env.alerting.example !.env.grafanacloud.example diff --git a/docker/telemetry/RUNBOOK-two-node-host.md b/docker/telemetry/RUNBOOK-two-node-host.md index 126469e8f0..8f40505c89 100644 --- a/docker/telemetry/RUNBOOK-two-node-host.md +++ b/docker/telemetry/RUNBOOK-two-node-host.md @@ -42,29 +42,56 @@ ln -sfn "$FAST_MOUNT"/xrpld/data docker/telemetry/data ln -sfn "$FAST_MOUNT"/xrpld/data2 docker/telemetry/data2 ``` -**Log directories where the collector looks.** The collector reads -`/var/log/xrpld//debug.log` and derives each node's identity from -that directory name, so the basename must match the instance id exactly: - -```sh -mkdir -p "$FAST_MOUNT"/xrpld/data/logs/xrpld-mainnet \ - "$FAST_MOUNT"/xrpld/data2/logs/xrpld-mainnet2 -sudo mkdir -p /var/log/xrpld -sudo ln -sfn "$FAST_MOUNT"/xrpld/data/logs/xrpld-mainnet /var/log/xrpld/xrpld-mainnet -sudo ln -sfn "$FAST_MOUNT"/xrpld/data2/logs/xrpld-mainnet2 /var/log/xrpld/xrpld-mainnet2 -``` - **Host-local settings and cloud credentials.** Both are provided out of band and are never committed; the ignore rules already exclude them. Copy each tracked example in this directory to its working name, fill it in, and set mode `600`. The unit installer refuses to run against a world-readable settings file. +The settings include each node's telemetry identity. Name those after the +machine, and **reuse the names this host has used before** — they are the join +key for every metric, trace and log already stored, so a new name starts a fresh +series and no dashboard will show the old and new data together. See +[systemd/README.md](systemd/README.md) for why they are not committed. + +**Log directories where the collector looks.** The collector reads +`/var/log/xrpld//debug.log` and derives each node's identity from +that directory name, so the basename must match the instance id exactly. Using +the ids from the settings file keeps the two in step: + +```sh +. docker/telemetry/.env.devbox # NODE1_INSTANCE_ID, NODE2_INSTANCE_ID + +mkdir -p "$FAST_MOUNT"/xrpld/data/logs/"$NODE1_INSTANCE_ID" \ + "$FAST_MOUNT"/xrpld/data2/logs/"$NODE2_INSTANCE_ID" +sudo mkdir -p /var/log/xrpld +sudo ln -sfn "$FAST_MOUNT"/xrpld/data/logs/"$NODE1_INSTANCE_ID" /var/log/xrpld/"$NODE1_INSTANCE_ID" +sudo ln -sfn "$FAST_MOUNT"/xrpld/data2/logs/"$NODE2_INSTANCE_ID" /var/log/xrpld/"$NODE2_INSTANCE_ID" +``` + +**Rootless Docker.** If the container runtime is rootless, its systemd user +units need a session bus to install, and the variable is absent over a plain +non-interactive SSH connection — the install appears to run and leaves nothing +behind: + +```sh +export XDG_RUNTIME_DIR=/run/user/$(id -u) +export DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$(id -u)/bus +dockerd-rootless-setuptool.sh install +``` + +`DOCKER_HOST` must then point at the rootless socket for every later `docker` +invocation, including non-interactive ones. + **Install the units:** ```sh sh docker/telemetry/systemd/install-units.sh ``` +This also renders the host-local configs the units run +(`xrpld-telemetry-mainnet{,2}.host.cfg`), so re-run it after changing either a +tracked config or the settings file. Nothing else regenerates them. + --- ## 2. Update to the latest code on the branch in use @@ -111,10 +138,15 @@ may have changed. After every update: # Confirm the data directories are still symlinks to the fast mount ls -ld docker/telemetry/data docker/telemetry/data2 -# Reinstall units in case the templates changed +# Reinstall units and re-render the host-local configs sh docker/telemetry/systemd/install-units.sh ``` +Re-running the installer is not optional after an update. It is what carries any +change to a tracked config into the `.host.cfg` the unit actually runs; skip it +and the node keeps running the previous rendering, so a config change appears to +have been applied and has not been. + Verify the ignored host files survived — they should, since a checkout does not touch ignored paths, but confirm their mode is still `600` before relying on them. @@ -177,11 +209,22 @@ Further notes from experience: a running node. - If Conan reports a missing default profile, its cache directory was created by a different user than the one running it. Fix ownership, then `conan profile - detect`. +detect`. - A dependency added upstream fails `cmake` configure with a missing package before any compilation starts. Re-run `conan install` rather than assuming the build itself broke. +**Expose the binary where the units expect it.** The build leaves `xrpld` in the +preset's build directory, while the units run `.build/xrpld`. Link the two, or +every start fails with the unit reporting only a missing executable: + +```sh +ln -sfn build/Release/xrpld .build/xrpld +``` + +The installer warns when that path is not executable, which is the cheapest +place to catch it — before the unit is ever started. + Confirm the binary is newer than the source you just pulled: ```sh @@ -276,14 +319,17 @@ per-instance identity: ## 8. When something looks wrong -| Symptom | First thing to check | -| --- | --- | -| Cloud dashboards empty, no errors anywhere | Collector started without the cloud overlay, or without `--force-recreate` after a config change | -| Metrics have instance labels, logs do not | Log directory basename does not equal the instance id | -| Unit refuses to start | The data mount is missing; the units require it deliberately, so a node cannot silently fill the root filesystem | -| `conan: command not found` under automation | Conan installed user-locally instead of on the system PATH | -| Node runs an old binary after rebuild | The unit did not restart; stop both, rebuild, start again | -| Update refuses to fast-forward | The host has local commits or edits to tracked files — see the guiding rule at the top | +| Symptom | First thing to check | +| ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Cloud dashboards empty, no errors anywhere | Collector started without the cloud overlay, or without `--force-recreate` after a config change | +| Collector exits at startup on an unresolved authenticator | An overlay redeclared `service.extensions`; the collector merges configs by **replacing** lists, not appending, so the cloud auth extension was dropped. Nothing exports at all in this state, local included | +| A node's data appears under the generic instance id, or a brand-new one | The installer was not re-run, so the `.host.cfg` still carries the old identity. Check the rendered file, not the tracked one | +| A node's history seems to have stopped | Its identity changed; the old data is intact under the old name. Query both names | +| Metrics have instance labels, logs do not | Log directory basename does not equal the instance id | +| Unit refuses to start | The data mount is missing; the units require it deliberately, so a node cannot silently fill the root filesystem | +| `conan: command not found` under automation | Conan installed user-locally instead of on the system PATH | +| Node runs an old binary after rebuild | The unit did not restart; stop both, rebuild, start again | +| Update refuses to fast-forward | The host has local commits or edits to tracked files — see the guiding rule at the top | Counter-intuitive signals worth knowing before drawing conclusions: diff --git a/docker/telemetry/systemd/README.md b/docker/telemetry/systemd/README.md index 0dd833c770..fe38197838 100644 --- a/docker/telemetry/systemd/README.md +++ b/docker/telemetry/systemd/README.md @@ -12,15 +12,16 @@ node's settings from notes. Everything else is identical on purpose — any other divergence would confound the backend comparison. -| | instance 1 | instance 2 | -| --- | --- | --- | -| Unit | `xrpld-mainnet` | `xrpld-mainnet2` | -| Config | `xrpld-telemetry-mainnet.cfg` | `xrpld-telemetry-mainnet2.cfg` | -| `service_instance_id` | `xrpld-mainnet` | `xrpld-mainnet2` | -| Backend | NuDB | RocksDB | -| rpc / ws-admin / ws-public / peer | 5015 / 6016 / 6015 / 51245 | 5025 / 6026 / 6025 / 51255 | -| Data | `data/mainnet` | `data2/mainnet` | -| Logs | `data/logs/xrpld-mainnet/` | `data2/logs/xrpld-mainnet2/` | +| | instance 1 | instance 2 | +| --------------------------------- | ---------------------------------- | ----------------------------------- | +| Unit | `xrpld-mainnet` | `xrpld-mainnet2` | +| Tracked config | `xrpld-telemetry-mainnet.cfg` | `xrpld-telemetry-mainnet2.cfg` | +| Config the unit runs | `xrpld-telemetry-mainnet.host.cfg` | `xrpld-telemetry-mainnet2.host.cfg` | +| `service_instance_id` | `$NODE1_INSTANCE_ID` | `$NODE2_INSTANCE_ID` | +| Backend | NuDB | RocksDB | +| rpc / ws-admin / ws-public / peer | 5015 / 6016 / 6015 / 51245 | 5025 / 6026 / 6025 / 51255 | +| Data | `data/mainnet` | `data2/mainnet` | +| Logs | `data/logs/$NODE1_INSTANCE_ID/` | `data2/logs/$NODE2_INSTANCE_ID/` | Ports continue the offset-by-ten scheme already in use — devnet on 5005, Mainnet on 5015 — so all three configs can bind on one host. @@ -34,13 +35,40 @@ and the real values live in an untracked `.env.devbox`: ```sh cp docker/telemetry/.env.devbox.example docker/telemetry/.env.devbox chmod 600 docker/telemetry/.env.devbox -$EDITOR docker/telemetry/.env.devbox # RUN_USER, REPO_DIR, DATA_MOUNT +$EDITOR docker/telemetry/.env.devbox ``` `.env.*` is gitignored, so the real file cannot be committed. The installer refuses to run if the file is not mode `600`, and refuses to install a unit that still contains an unsubstituted placeholder. +## Telemetry identity names the machine, so it is not committed + +Each node reports a `service_instance_id` on every metric, span and log line. +It should name the **machine**, so a dashboard shows which box the data came +from — but a machine name is exactly what a public repository should not carry. + +So the tracked configs keep a generic identity and name no host, and the +installer renders each one into a `.host.cfg` beside it with +`NODE1_INSTANCE_ID` / `NODE2_INSTANCE_ID` substituted in. The units run the +rendered copies; `*.host.cfg` is gitignored. The tracked configs are never +edited, so an update never conflicts and a rebuild loses nothing. + +The identity is substituted in two places at once — the `service_instance_id` +setting and the log directory name — because they must agree, for the reason in +the next section. The installer counts the occurrences it expects to replace and +verifies the result, so a config reshuffle fails loudly instead of yielding a +copy that quietly kept the generic identity: on the dashboards that reads as the +node having disappeared, not as a failed substitution. + +**Reuse the host's established names.** These values are the join key for +everything already stored. Renaming a node starts a fresh series and silently +breaks continuity with its own history — the old data is still there, under the +old name, and no dashboard will show both. + +Re-run the installer after changing either the tracked config or `.env.devbox`; +nothing else regenerates the rendered copies. + ## Install ```sh @@ -56,10 +84,12 @@ for the job pool. ## Two things that are easy to get wrong **The log directory basename must equal the `service_instance_id`.** The +installer substitutes both from one value so they cannot drift, but the +directory itself still has to exist under that name. The collector's filelog receiver derives per-node identity from the log path (`include_file_path` plus a regex on `/xrpld//debug.log`). Name the directory -anything else and that node's *logs* lose their `service_instance_id` label while -its *metrics* keep theirs — so the dashboards' `$node` filter matches nothing for +anything else and that node's _logs_ lose their `service_instance_id` label while +its _metrics_ keep theirs — so the dashboards' `$node` filter matches nothing for logs and reads as "no logs" rather than as a misconfiguration. The collector expects the logs under `/var/log/xrpld//`, so symlink or bind-mount each node's log directory there. diff --git a/docker/telemetry/systemd/install-units.sh b/docker/telemetry/systemd/install-units.sh index 96a1501fad..8a002e8e11 100755 --- a/docker/telemetry/systemd/install-units.sh +++ b/docker/telemetry/systemd/install-units.sh @@ -19,56 +19,174 @@ tel=$(dirname "$here") env_file="$tel/.env.devbox" if [ ! -f "$env_file" ]; then - echo "ERROR: $env_file not found." >&2 - echo " Copy $tel/.env.devbox.example to it and fill in the values." >&2 - exit 1 + echo "ERROR: $env_file not found." >&2 + echo " Copy $tel/.env.devbox.example to it and fill in the values." >&2 + exit 1 fi # Refuse a world-readable env file: it names the account xrpld runs as, and this # script is the only thing that should be reading it. mode=$(stat -c %a "$env_file" 2>/dev/null || echo "") case "$mode" in - 600|400) : ;; - "") echo "WARN: could not read permissions of $env_file" >&2 ;; - *) echo "ERROR: $env_file is mode $mode; expected 600. Run: chmod 600 $env_file" >&2; exit 1 ;; + 600 | 400) : ;; + "") echo "WARN: could not read permissions of $env_file" >&2 ;; + *) + echo "ERROR: $env_file is mode $mode; expected 600. Run: chmod 600 $env_file" >&2 + exit 1 + ;; esac # shellcheck disable=SC1090 . "$env_file" -for var in RUN_USER REPO_DIR DATA_MOUNT; do - eval "val=\${$var:-}" - if [ -z "$val" ]; then - echo "ERROR: $var is empty in $env_file" >&2 - exit 1 - fi +for var in RUN_USER REPO_DIR DATA_MOUNT NODE1_INSTANCE_ID NODE2_INSTANCE_ID; do + eval "val=\${$var:-}" + if [ -z "$val" ]; then + echo "ERROR: $var is empty in $env_file" >&2 + exit 1 + fi done # Fail early on values that would produce a unit systemd silently never starts. -id "$RUN_USER" >/dev/null 2>&1 || { echo "ERROR: user '$RUN_USER' does not exist" >&2; exit 1; } -[ -d "$REPO_DIR" ] || { echo "ERROR: REPO_DIR '$REPO_DIR' is not a directory" >&2; exit 1; } +id "$RUN_USER" >/dev/null 2>&1 || { + echo "ERROR: user '$RUN_USER' does not exist" >&2 + exit 1 +} +[ -d "$REPO_DIR" ] || { + echo "ERROR: REPO_DIR '$REPO_DIR' is not a directory" >&2 + exit 1 +} [ -x "$REPO_DIR/.build/xrpld" ] || echo "WARN: $REPO_DIR/.build/xrpld not built yet; the unit will fail to start until it is" >&2 [ -d "$DATA_MOUNT" ] || echo "WARN: DATA_MOUNT '$DATA_MOUNT' does not exist yet; the unit will refuse to start until it is mounted" >&2 -for unit in xrpld-mainnet xrpld-mainnet2; do - tpl="$here/$unit.service.template" - [ -f "$tpl" ] || { echo "ERROR: missing template $tpl" >&2; exit 1; } - out=$(mktemp) - sed -e "s|__RUN_USER__|$RUN_USER|g" \ - -e "s|__REPO_DIR__|$REPO_DIR|g" \ - -e "s|__DATA_MOUNT__|$DATA_MOUNT|g" \ - "$tpl" > "$out" +# The instance id becomes a directory name and a telemetry label, so reject +# anything that would need quoting in either. +for var in NODE1_INSTANCE_ID NODE2_INSTANCE_ID; do + eval "val=\$$var" + case "$val" in + *[!A-Za-z0-9._-]*) + echo "ERROR: $var '$val' has characters outside [A-Za-z0-9._-]" >&2 + exit 1 + ;; + esac +done - if grep -q '__[A-Z_]*__' "$out"; then - echo "ERROR: unsubstituted placeholder left in $unit:" >&2 - grep -o '__[A-Z_]*__' "$out" | sort -u | sed 's/^/ /' >&2 - rm -f "$out" +if [ "$NODE1_INSTANCE_ID" = "$NODE2_INSTANCE_ID" ]; then + echo "ERROR: both nodes are named '$NODE1_INSTANCE_ID'; their telemetry would be indistinguishable" >&2 exit 1 - fi +fi - sudo install -m 0644 "$out" "/etc/systemd/system/$unit.service" - rm -f "$out" - echo "installed /etc/systemd/system/$unit.service" +# Render a tracked config into a host-local copy carrying this host's instance +# id. The tracked file is never modified: this repository is public and must not +# name a machine, and a host that edits tracked configs in place conflicts on +# every update and loses those edits when the machine is rebuilt. +# +# The identity appears both as the service_instance_id setting and as the log +# directory name, and the two must agree: the collector derives each node's +# identity for the *logs* pipeline from the log path, so a mismatch costs the +# logs their service_instance_id label while metrics keep theirs. Both are +# rewritten here from one value so they cannot drift. +render_cfg() { + src=$1 + new_id=$2 + out=$3 + + [ -f "$src" ] || { + echo "ERROR: missing config $src" >&2 + exit 1 + } + + old_id=$(sed -n 's/^service_instance_id=\(.*\)$/\1/p' "$src" | head -1) + [ -n "$old_id" ] || { + echo "ERROR: no service_instance_id= setting in $src" >&2 + exit 1 + } + + # Escape for use as a sed pattern; the tracked ids contain no metacharacters + # today, but a future rename should not silently mis-substitute. + old_esc=$(printf '%s' "$old_id" | sed 's/[.[\*^$/]/\\&/g') + + # Count what should be replaced before replacing it. A config reshuffle that + # renamed or dropped one of these would otherwise yield a copy that quietly + # kept the tracked identity -- which reads on the dashboards as the node + # having vanished rather than as a failed substitution. + id_count=$(grep -c "^service_instance_id=$old_esc\$" "$src" || true) + if [ "$id_count" -ne 2 ]; then + echo "ERROR: $src has $id_count 'service_instance_id=$old_id' lines, expected 2 ([insight] and [telemetry])" >&2 + exit 1 + fi + log_count=$(grep -c "logs/$old_esc/" "$src" || true) + if [ "$log_count" -lt 1 ]; then + echo "ERROR: $src has no 'logs/$old_id/' path to rename" >&2 + exit 1 + fi + + { + echo "# GENERATED by install-units.sh from $(basename "$src") -- do not edit." + echo "# Host-local copy: the tracked config names no machine, this one does." + echo "# Identity '$old_id' replaced with '$new_id'. Re-run the installer" + echo "# after updating the config or .env.devbox." + sed -e "s|^service_instance_id=$old_esc\$|service_instance_id=$new_id|" \ + -e "s|logs/$old_esc/|logs/$new_id/|g" \ + "$src" + } >"$out" + + # Verify the copy, rather than trusting that sed did what was intended. + if grep -q "logs/$old_esc/" "$out" || grep -q "^service_instance_id=$old_esc\$" "$out"; then + echo "ERROR: $out still carries the tracked identity '$old_id'" >&2 + exit 1 + fi + new_count=$(grep -c "^service_instance_id=$new_id\$" "$out" || true) + if [ "$new_count" -ne 2 ]; then + echo "ERROR: $out has $new_count 'service_instance_id=$new_id' lines, expected 2" >&2 + exit 1 + fi + echo "rendered $(basename "$out") with service_instance_id=$new_id" +} + +render_cfg "$tel/xrpld-telemetry-mainnet.cfg" "$NODE1_INSTANCE_ID" "$tel/xrpld-telemetry-mainnet.host.cfg" +render_cfg "$tel/xrpld-telemetry-mainnet2.cfg" "$NODE2_INSTANCE_ID" "$tel/xrpld-telemetry-mainnet2.host.cfg" + +# The log directories have to exist under both the names the node writes and the +# path the collector reads, or one node's logs go unlabelled while everything +# else looks healthy. +check_log_dir() { + node_dir=$1 + id=$2 + [ -d "$tel/$node_dir/logs/$id" ] || cat >&2 <&2 +} + +check_log_dir data "$NODE1_INSTANCE_ID" +check_log_dir data2 "$NODE2_INSTANCE_ID" + +for unit in xrpld-mainnet xrpld-mainnet2; do + tpl="$here/$unit.service.template" + [ -f "$tpl" ] || { + echo "ERROR: missing template $tpl" >&2 + exit 1 + } + out=$(mktemp) + sed -e "s|__RUN_USER__|$RUN_USER|g" \ + -e "s|__REPO_DIR__|$REPO_DIR|g" \ + -e "s|__DATA_MOUNT__|$DATA_MOUNT|g" \ + "$tpl" >"$out" + + if grep -q '__[A-Z_]*__' "$out"; then + echo "ERROR: unsubstituted placeholder left in $unit:" >&2 + grep -o '__[A-Z_]*__' "$out" | sort -u | sed 's/^/ /' >&2 + rm -f "$out" + exit 1 + fi + + sudo install -m 0644 "$out" "/etc/systemd/system/$unit.service" + rm -f "$out" + echo "installed /etc/systemd/system/$unit.service" done sudo systemctl daemon-reload diff --git a/docker/telemetry/systemd/xrpld-mainnet.service.template b/docker/telemetry/systemd/xrpld-mainnet.service.template index 405a96b2e9..3ed33f9294 100644 --- a/docker/telemetry/systemd/xrpld-mainnet.service.template +++ b/docker/telemetry/systemd/xrpld-mainnet.service.template @@ -26,7 +26,10 @@ Type=simple User=__RUN_USER__ Group=__RUN_USER__ WorkingDirectory=__REPO_DIR__ -ExecStart=__REPO_DIR__/.build/xrpld --conf docker/telemetry/xrpld-telemetry-mainnet.cfg +# The .host.cfg is the tracked config rendered with this host's instance id, so +# the node's telemetry identifies the machine it runs on. install-units.sh +# generates it; it is gitignored and never committed. +ExecStart=__REPO_DIR__/.build/xrpld --conf docker/telemetry/xrpld-telemetry-mainnet.host.cfg Restart=on-failure RestartSec=5s # xrpld flushes its OpenTelemetry buffers on shutdown; killing it early loses the diff --git a/docker/telemetry/systemd/xrpld-mainnet2.service.template b/docker/telemetry/systemd/xrpld-mainnet2.service.template index b1cf042f18..a167dde087 100644 --- a/docker/telemetry/systemd/xrpld-mainnet2.service.template +++ b/docker/telemetry/systemd/xrpld-mainnet2.service.template @@ -26,7 +26,10 @@ Type=simple User=__RUN_USER__ Group=__RUN_USER__ WorkingDirectory=__REPO_DIR__ -ExecStart=__REPO_DIR__/.build/xrpld --conf docker/telemetry/xrpld-telemetry-mainnet2.cfg +# The .host.cfg is the tracked config rendered with this host's instance id, so +# the node's telemetry identifies the machine it runs on. install-units.sh +# generates it; it is gitignored and never committed. +ExecStart=__REPO_DIR__/.build/xrpld --conf docker/telemetry/xrpld-telemetry-mainnet2.host.cfg Restart=on-failure RestartSec=5s # xrpld flushes its OpenTelemetry buffers on shutdown; killing it early loses the