ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-21 22:30:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

Name each node's telemetry after its machine, without committing the name

Browse Source 
A node's service_instance_id should say which box the data came from, so a
dashboard can tell two otherwise-identical instances apart. A machine name is
also exactly what this public repository should not carry, and the value cannot
come from the environment: it is read only from the [telemetry] section, and the
config parser has no include directive.

So the tracked configs keep a generic identity and name no host, and the
installer renders each into a .host.cfg beside it with the id substituted from
.env.devbox. The units run the rendered copies, which are gitignored. The
tracked configs are never edited on the host, so an update cannot conflict and a
rebuild loses nothing.

The identity is substituted in two places from one value -- the setting and the
log directory name -- because they have to agree: the collector derives identity
for the logs pipeline from the log path, so a mismatch costs that node's logs
their service_instance_id label while its metrics keep theirs, which reads as
"no logs" rather than as a misconfiguration. The installer counts the
occurrences it expects to replace and verifies the result, so a config reshuffle
fails loudly instead of yielding a copy that quietly kept the generic identity;
on the dashboards that would look like the node had disappeared. It also rejects
an id that is not a safe directory name, and rejects two nodes sharing one id.

Also record what the last deployment needed and the runbook did not say: the
session-bus variable a rootless container runtime needs before its user units
will install, the link from the unit's expected binary path to the preset's
build directory, and that a config merge which redeclares service.extensions
drops the cloud authenticator and stops the collector exporting anything at all.

These files also had not been through the formatting hooks, which want 4-space
indentation and reflowed tables; that is fixed here too.
This commit is contained in:
Pratik Mankawde
2026-08-18 15:41:06 +01:00
parent eefadf43fd
commit 4988533d96
7 changed files with 282 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
docker/telemetry/.env.devbox.example
View File

@@ -14,3 +14,15 @@ REPO_DIR=
# Mount holding the node data directories. The units require it, so a missing
# disk fails the unit instead of silently filling the root filesystem.
DATA_MOUNT=
# Telemetry identity of each node, reported as the service_instance_id label on
# every metric, span and log line, and used as the log directory name.
#
# Name them after the machine, so a dashboard shows which box the data came
# from -- a host running two nodes needs two distinct values. Changing them
# breaks continuity with data already stored under the old names, so reuse the
# host's established names rather than inventing new ones.
#
# Allowed characters: letters, digits, dot, underscore, hyphen.
NODE1_INSTANCE_ID=
NODE2_INSTANCE_ID=

4
docker/telemetry/.gitignore vendored
View File

@@ -4,6 +4,10 @@ data/
# Env. file carrying environmental setup data for local or cloud runs.
.env.*
# Host-local configs rendered by systemd/install-units.sh. They carry the
# machine's telemetry identity, which must not be committed.
*.host.cfg
# Keep examples
!.env.alerting.example
!.env.grafanacloud.example

90
docker/telemetry/RUNBOOK-two-node-host.md
View File

@@ -42,29 +42,56 @@ ln -sfn "$FAST_MOUNT"/xrpld/data docker/telemetry/data
ln -sfn "$FAST_MOUNT"/xrpld/data2 docker/telemetry/data2
```
**Log directories where the collector looks.** The collector reads
`/var/log/xrpld/<instance-id>/debug.log` and derives each node's identity from
that directory name, so the basename must match the instance id exactly:
```sh
mkdir -p "$FAST_MOUNT"/xrpld/data/logs/xrpld-mainnet \
"$FAST_MOUNT"/xrpld/data2/logs/xrpld-mainnet2
sudo mkdir -p /var/log/xrpld
sudo ln -sfn "$FAST_MOUNT"/xrpld/data/logs/xrpld-mainnet /var/log/xrpld/xrpld-mainnet
sudo ln -sfn "$FAST_MOUNT"/xrpld/data2/logs/xrpld-mainnet2 /var/log/xrpld/xrpld-mainnet2
```
**Host-local settings and cloud credentials.** Both are provided out of band and
are never committed; the ignore rules already exclude them. Copy each tracked
example in this directory to its working name, fill it in, and set mode `600`.
The unit installer refuses to run against a world-readable settings file.
The settings include each node's telemetry identity. Name those after the
machine, and **reuse the names this host has used before** — they are the join
key for every metric, trace and log already stored, so a new name starts a fresh
series and no dashboard will show the old and new data together. See
[systemd/README.md](systemd/README.md) for why they are not committed.
**Log directories where the collector looks.** The collector reads
`/var/log/xrpld/<instance-id>/debug.log` and derives each node's identity from
that directory name, so the basename must match the instance id exactly. Using
the ids from the settings file keeps the two in step:
```sh
. docker/telemetry/.env.devbox # NODE1_INSTANCE_ID, NODE2_INSTANCE_ID
mkdir -p "$FAST_MOUNT"/xrpld/data/logs/"$NODE1_INSTANCE_ID" \
"$FAST_MOUNT"/xrpld/data2/logs/"$NODE2_INSTANCE_ID"
sudo mkdir -p /var/log/xrpld
sudo ln -sfn "$FAST_MOUNT"/xrpld/data/logs/"$NODE1_INSTANCE_ID" /var/log/xrpld/"$NODE1_INSTANCE_ID"
sudo ln -sfn "$FAST_MOUNT"/xrpld/data2/logs/"$NODE2_INSTANCE_ID" /var/log/xrpld/"$NODE2_INSTANCE_ID"
```
**Rootless Docker.** If the container runtime is rootless, its systemd user
units need a session bus to install, and the variable is absent over a plain
non-interactive SSH connection — the install appears to run and leaves nothing
behind:
```sh
export XDG_RUNTIME_DIR=/run/user/$(id -u)
export DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$(id -u)/bus
dockerd-rootless-setuptool.sh install
```
`DOCKER_HOST` must then point at the rootless socket for every later `docker`
invocation, including non-interactive ones.
**Install the units:**
```sh
sh docker/telemetry/systemd/install-units.sh
```
This also renders the host-local configs the units run
(`xrpld-telemetry-mainnet{,2}.host.cfg`), so re-run it after changing either a
tracked config or the settings file. Nothing else regenerates them.
---
## 2. Update to the latest code on the branch in use
@@ -111,10 +138,15 @@ may have changed. After every update:
# Confirm the data directories are still symlinks to the fast mount
ls -ld docker/telemetry/data docker/telemetry/data2
# Reinstall units in case the templates changed
# Reinstall units and re-render the host-local configs
sh docker/telemetry/systemd/install-units.sh
```
Re-running the installer is not optional after an update. It is what carries any
change to a tracked config into the `.host.cfg` the unit actually runs; skip it
and the node keeps running the previous rendering, so a config change appears to
have been applied and has not been.
Verify the ignored host files survived — they should, since a checkout does not
touch ignored paths, but confirm their mode is still `600` before relying on
them.
@@ -177,11 +209,22 @@ Further notes from experience:
a running node.
- If Conan reports a missing default profile, its cache directory was created by a
different user than the one running it. Fix ownership, then `conan profile
detect`.
detect`.
- A dependency added upstream fails `cmake` configure with a missing package
before any compilation starts. Re-run `conan install` rather than assuming the
build itself broke.
**Expose the binary where the units expect it.** The build leaves `xrpld` in the
preset's build directory, while the units run `.build/xrpld`. Link the two, or
every start fails with the unit reporting only a missing executable:
```sh
ln -sfn build/Release/xrpld .build/xrpld
```
The installer warns when that path is not executable, which is the cheapest
place to catch it — before the unit is ever started.
Confirm the binary is newer than the source you just pulled:
```sh
@@ -276,14 +319,17 @@ per-instance identity:
## 8. When something looks wrong
| Symptom | First thing to check |
| --- | --- |
| Cloud dashboards empty, no errors anywhere | Collector started without the cloud overlay, or without `--force-recreate` after a config change |
| Metrics have instance labels, logs do not | Log directory basename does not equal the instance id |
| Unit refuses to start | The data mount is missing; the units require it deliberately, so a node cannot silently fill the root filesystem |
| `conan: command not found` under automation | Conan installed user-locally instead of on the system PATH |
| Node runs an old binary after rebuild | The unit did not restart; stop both, rebuild, start again |
| Update refuses to fast-forward | The host has local commits or edits to tracked files — see the guiding rule at the top |
| Symptom | First thing to check |
| ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Cloud dashboards empty, no errors anywhere | Collector started without the cloud overlay, or without `--force-recreate` after a config change |
| Collector exits at startup on an unresolved authenticator | An overlay redeclared `service.extensions`; the collector merges configs by **replacing** lists, not appending, so the cloud auth extension was dropped. Nothing exports at all in this state, local included |
| A node's data appears under the generic instance id, or a brand-new one | The installer was not re-run, so the `.host.cfg` still carries the old identity. Check the rendered file, not the tracked one |
| A node's history seems to have stopped | Its identity changed; the old data is intact under the old name. Query both names |
| Metrics have instance labels, logs do not | Log directory basename does not equal the instance id |
| Unit refuses to start | The data mount is missing; the units require it deliberately, so a node cannot silently fill the root filesystem |
| `conan: command not found` under automation | Conan installed user-locally instead of on the system PATH |
| Node runs an old binary after rebuild | The unit did not restart; stop both, rebuild, start again |
| Update refuses to fast-forward | The host has local commits or edits to tracked files — see the guiding rule at the top |
Counter-intuitive signals worth knowing before drawing conclusions:

54
docker/telemetry/systemd/README.md
View File

@@ -12,15 +12,16 @@ node's settings from notes.
Everything else is identical on purpose — any other divergence would confound the
backend comparison.
| | instance 1 | instance 2 |
| --- | --- | --- |
| Unit | `xrpld-mainnet` | `xrpld-mainnet2` |
| Config | `xrpld-telemetry-mainnet.cfg` | `xrpld-telemetry-mainnet2.cfg` |
| `service_instance_id` | `xrpld-mainnet` | `xrpld-mainnet2` |
| Backend | NuDB | RocksDB |
| rpc / ws-admin / ws-public / peer | 5015 / 6016 / 6015 / 51245 | 5025 / 6026 / 6025 / 51255 |
| Data | `data/mainnet` | `data2/mainnet` |
| Logs | `data/logs/xrpld-mainnet/` | `data2/logs/xrpld-mainnet2/` |
| | instance 1 | instance 2 |
| --------------------------------- | ---------------------------------- | ----------------------------------- |
| Unit | `xrpld-mainnet` | `xrpld-mainnet2` |
| Tracked config | `xrpld-telemetry-mainnet.cfg` | `xrpld-telemetry-mainnet2.cfg` |
| Config the unit runs | `xrpld-telemetry-mainnet.host.cfg` | `xrpld-telemetry-mainnet2.host.cfg` |
| `service_instance_id` | `$NODE1_INSTANCE_ID` | `$NODE2_INSTANCE_ID` |
| Backend | NuDB | RocksDB |
| rpc / ws-admin / ws-public / peer | 5015 / 6016 / 6015 / 51245 | 5025 / 6026 / 6025 / 51255 |
| Data | `data/mainnet` | `data2/mainnet` |
| Logs | `data/logs/$NODE1_INSTANCE_ID/` | `data2/logs/$NODE2_INSTANCE_ID/` |
Ports continue the offset-by-ten scheme already in use — devnet on 5005, Mainnet
on 5015 — so all three configs can bind on one host.
@@ -34,13 +35,40 @@ and the real values live in an untracked `.env.devbox`:
```sh
cp docker/telemetry/.env.devbox.example docker/telemetry/.env.devbox
chmod 600 docker/telemetry/.env.devbox
$EDITOR docker/telemetry/.env.devbox # RUN_USER, REPO_DIR, DATA_MOUNT
$EDITOR docker/telemetry/.env.devbox
```
`.env.*` is gitignored, so the real file cannot be committed. The installer
refuses to run if the file is not mode `600`, and refuses to install a unit that
still contains an unsubstituted placeholder.
## Telemetry identity names the machine, so it is not committed
Each node reports a `service_instance_id` on every metric, span and log line.
It should name the **machine**, so a dashboard shows which box the data came
from — but a machine name is exactly what a public repository should not carry.
So the tracked configs keep a generic identity and name no host, and the
installer renders each one into a `.host.cfg` beside it with
`NODE1_INSTANCE_ID` / `NODE2_INSTANCE_ID` substituted in. The units run the
rendered copies; `*.host.cfg` is gitignored. The tracked configs are never
edited, so an update never conflicts and a rebuild loses nothing.
The identity is substituted in two places at once — the `service_instance_id`
setting and the log directory name — because they must agree, for the reason in
the next section. The installer counts the occurrences it expects to replace and
verifies the result, so a config reshuffle fails loudly instead of yielding a
copy that quietly kept the generic identity: on the dashboards that reads as the
node having disappeared, not as a failed substitution.
**Reuse the host's established names.** These values are the join key for
everything already stored. Renaming a node starts a fresh series and silently
breaks continuity with its own history — the old data is still there, under the
old name, and no dashboard will show both.
Re-run the installer after changing either the tracked config or `.env.devbox`;
nothing else regenerates the rendered copies.
## Install
```sh
@@ -56,10 +84,12 @@ for the job pool.
## Two things that are easy to get wrong
**The log directory basename must equal the `service_instance_id`.** The
installer substitutes both from one value so they cannot drift, but the
directory itself still has to exist under that name. The
collector's filelog receiver derives per-node identity from the log path
(`include_file_path` plus a regex on `/xrpld/<id>/debug.log`). Name the directory
anything else and that node's *logs* lose their `service_instance_id` label while
its *metrics* keep theirs — so the dashboards' `$node` filter matches nothing for
anything else and that node's _logs_ lose their `service_instance_id` label while
its _metrics_ keep theirs — so the dashboards' `$node` filter matches nothing for
logs and reads as "no logs" rather than as a misconfiguration. The collector
expects the logs under `/var/log/xrpld/<id>/`, so symlink or bind-mount each
node's log directory there.

178
docker/telemetry/systemd/install-units.sh
View File

@@ -19,56 +19,174 @@ tel=$(dirname "$here")
env_file="$tel/.env.devbox"
if [ ! -f "$env_file" ]; then
echo "ERROR: $env_file not found." >&2
echo " Copy $tel/.env.devbox.example to it and fill in the values." >&2
exit 1
echo "ERROR: $env_file not found." >&2
echo " Copy $tel/.env.devbox.example to it and fill in the values." >&2
exit 1
fi
# Refuse a world-readable env file: it names the account xrpld runs as, and this
# script is the only thing that should be reading it.
mode=$(stat -c %a "$env_file" 2>/dev/null || echo "")
case "$mode" in
600|400) : ;;
"") echo "WARN: could not read permissions of $env_file" >&2 ;;
*) echo "ERROR: $env_file is mode $mode; expected 600. Run: chmod 600 $env_file" >&2; exit 1 ;;
600 | 400) : ;;
"") echo "WARN: could not read permissions of $env_file" >&2 ;;
*)
echo "ERROR: $env_file is mode $mode; expected 600. Run: chmod 600 $env_file" >&2
exit 1
;;
esac
# shellcheck disable=SC1090
. "$env_file"
for var in RUN_USER REPO_DIR DATA_MOUNT; do
eval "val=\${$var:-}"
if [ -z "$val" ]; then
echo "ERROR: $var is empty in $env_file" >&2
exit 1
fi
for var in RUN_USER REPO_DIR DATA_MOUNT NODE1_INSTANCE_ID NODE2_INSTANCE_ID; do
eval "val=\${$var:-}"
if [ -z "$val" ]; then
echo "ERROR: $var is empty in $env_file" >&2
exit 1
fi
done
# Fail early on values that would produce a unit systemd silently never starts.
id "$RUN_USER" >/dev/null 2>&1 || { echo "ERROR: user '$RUN_USER' does not exist" >&2; exit 1; }
[ -d "$REPO_DIR" ] || { echo "ERROR: REPO_DIR '$REPO_DIR' is not a directory" >&2; exit 1; }
id "$RUN_USER" >/dev/null 2>&1 || {
echo "ERROR: user '$RUN_USER' does not exist" >&2
exit 1
}
[ -d "$REPO_DIR" ] || {
echo "ERROR: REPO_DIR '$REPO_DIR' is not a directory" >&2
exit 1
}
[ -x "$REPO_DIR/.build/xrpld" ] || echo "WARN: $REPO_DIR/.build/xrpld not built yet; the unit will fail to start until it is" >&2
[ -d "$DATA_MOUNT" ] || echo "WARN: DATA_MOUNT '$DATA_MOUNT' does not exist yet; the unit will refuse to start until it is mounted" >&2
for unit in xrpld-mainnet xrpld-mainnet2; do
tpl="$here/$unit.service.template"
[ -f "$tpl" ] || { echo "ERROR: missing template $tpl" >&2; exit 1; }
out=$(mktemp)
sed -e "s|__RUN_USER__|$RUN_USER|g" \
-e "s|__REPO_DIR__|$REPO_DIR|g" \
-e "s|__DATA_MOUNT__|$DATA_MOUNT|g" \
"$tpl" > "$out"
# The instance id becomes a directory name and a telemetry label, so reject
# anything that would need quoting in either.
for var in NODE1_INSTANCE_ID NODE2_INSTANCE_ID; do
eval "val=\$$var"
case "$val" in
*[!A-Za-z0-9._-]*)
echo "ERROR: $var '$val' has characters outside [A-Za-z0-9._-]" >&2
exit 1
;;
esac
done
if grep -q '__[A-Z_]*__' "$out"; then
echo "ERROR: unsubstituted placeholder left in $unit:" >&2
grep -o '__[A-Z_]*__' "$out" | sort -u | sed 's/^/ /' >&2
rm -f "$out"
if [ "$NODE1_INSTANCE_ID" = "$NODE2_INSTANCE_ID" ]; then
echo "ERROR: both nodes are named '$NODE1_INSTANCE_ID'; their telemetry would be indistinguishable" >&2
exit 1
fi
fi
sudo install -m 0644 "$out" "/etc/systemd/system/$unit.service"
rm -f "$out"
echo "installed /etc/systemd/system/$unit.service"
# Render a tracked config into a host-local copy carrying this host's instance
# id. The tracked file is never modified: this repository is public and must not
# name a machine, and a host that edits tracked configs in place conflicts on
# every update and loses those edits when the machine is rebuilt.
#
# The identity appears both as the service_instance_id setting and as the log
# directory name, and the two must agree: the collector derives each node's
# identity for the *logs* pipeline from the log path, so a mismatch costs the
# logs their service_instance_id label while metrics keep theirs. Both are
# rewritten here from one value so they cannot drift.
render_cfg() {
src=$1
new_id=$2
out=$3
[ -f "$src" ] || {
echo "ERROR: missing config $src" >&2
exit 1
}
old_id=$(sed -n 's/^service_instance_id=\(.*\)$/\1/p' "$src" | head -1)
[ -n "$old_id" ] || {
echo "ERROR: no service_instance_id= setting in $src" >&2
exit 1
}
# Escape for use as a sed pattern; the tracked ids contain no metacharacters
# today, but a future rename should not silently mis-substitute.
old_esc=$(printf '%s' "$old_id" | sed 's/[.[\*^$/]/\\&/g')
# Count what should be replaced before replacing it. A config reshuffle that
# renamed or dropped one of these would otherwise yield a copy that quietly
# kept the tracked identity -- which reads on the dashboards as the node
# having vanished rather than as a failed substitution.
id_count=$(grep -c "^service_instance_id=$old_esc\$" "$src" || true)
if [ "$id_count" -ne 2 ]; then
echo "ERROR: $src has $id_count 'service_instance_id=$old_id' lines, expected 2 ([insight] and [telemetry])" >&2
exit 1
fi
log_count=$(grep -c "logs/$old_esc/" "$src" || true)
if [ "$log_count" -lt 1 ]; then
echo "ERROR: $src has no 'logs/$old_id/' path to rename" >&2
exit 1
fi
{
echo "# GENERATED by install-units.sh from $(basename "$src") -- do not edit."
echo "# Host-local copy: the tracked config names no machine, this one does."
echo "# Identity '$old_id' replaced with '$new_id'. Re-run the installer"
echo "# after updating the config or .env.devbox."
sed -e "s|^service_instance_id=$old_esc\$|service_instance_id=$new_id|" \
-e "s|logs/$old_esc/|logs/$new_id/|g" \
"$src"
} >"$out"
# Verify the copy, rather than trusting that sed did what was intended.
if grep -q "logs/$old_esc/" "$out" || grep -q "^service_instance_id=$old_esc\$" "$out"; then
echo "ERROR: $out still carries the tracked identity '$old_id'" >&2
exit 1
fi
new_count=$(grep -c "^service_instance_id=$new_id\$" "$out" || true)
if [ "$new_count" -ne 2 ]; then
echo "ERROR: $out has $new_count 'service_instance_id=$new_id' lines, expected 2" >&2
exit 1
fi
echo "rendered $(basename "$out") with service_instance_id=$new_id"
}
render_cfg "$tel/xrpld-telemetry-mainnet.cfg" "$NODE1_INSTANCE_ID" "$tel/xrpld-telemetry-mainnet.host.cfg"
render_cfg "$tel/xrpld-telemetry-mainnet2.cfg" "$NODE2_INSTANCE_ID" "$tel/xrpld-telemetry-mainnet2.host.cfg"
# The log directories have to exist under both the names the node writes and the
# path the collector reads, or one node's logs go unlabelled while everything
# else looks healthy.
check_log_dir() {
node_dir=$1
id=$2
[ -d "$tel/$node_dir/logs/$id" ] || cat >&2 <<EOF
WARN: $tel/$node_dir/logs/$id does not exist; create it and expose it to the collector:
mkdir -p "$tel/$node_dir/logs/$id"
sudo mkdir -p /var/log/xrpld
sudo ln -sfn "$tel/$node_dir/logs/$id" /var/log/xrpld/$id
EOF
[ -e "/var/log/xrpld/$id" ] || echo "WARN: /var/log/xrpld/$id is missing; the collector will find no logs for '$id'" >&2
}
check_log_dir data "$NODE1_INSTANCE_ID"
check_log_dir data2 "$NODE2_INSTANCE_ID"
for unit in xrpld-mainnet xrpld-mainnet2; do
tpl="$here/$unit.service.template"
[ -f "$tpl" ] || {
echo "ERROR: missing template $tpl" >&2
exit 1
}
out=$(mktemp)
sed -e "s|__RUN_USER__|$RUN_USER|g" \
-e "s|__REPO_DIR__|$REPO_DIR|g" \
-e "s|__DATA_MOUNT__|$DATA_MOUNT|g" \
"$tpl" >"$out"
if grep -q '__[A-Z_]*__' "$out"; then
echo "ERROR: unsubstituted placeholder left in $unit:" >&2
grep -o '__[A-Z_]*__' "$out" | sort -u | sed 's/^/ /' >&2
rm -f "$out"
exit 1
fi
sudo install -m 0644 "$out" "/etc/systemd/system/$unit.service"
rm -f "$out"
echo "installed /etc/systemd/system/$unit.service"
done
sudo systemctl daemon-reload

5
docker/telemetry/systemd/xrpld-mainnet.service.template
View File

@@ -26,7 +26,10 @@ Type=simple
User=__RUN_USER__
Group=__RUN_USER__
WorkingDirectory=__REPO_DIR__
ExecStart=__REPO_DIR__/.build/xrpld --conf docker/telemetry/xrpld-telemetry-mainnet.cfg
# The .host.cfg is the tracked config rendered with this host's instance id, so
# the node's telemetry identifies the machine it runs on. install-units.sh
# generates it; it is gitignored and never committed.
ExecStart=__REPO_DIR__/.build/xrpld --conf docker/telemetry/xrpld-telemetry-mainnet.host.cfg
Restart=on-failure
RestartSec=5s
# xrpld flushes its OpenTelemetry buffers on shutdown; killing it early loses the

5
docker/telemetry/systemd/xrpld-mainnet2.service.template
View File

@@ -26,7 +26,10 @@ Type=simple
User=__RUN_USER__
Group=__RUN_USER__
WorkingDirectory=__REPO_DIR__
ExecStart=__REPO_DIR__/.build/xrpld --conf docker/telemetry/xrpld-telemetry-mainnet2.cfg
# The .host.cfg is the tracked config rendered with this host's instance id, so
# the node's telemetry identifies the machine it runs on. install-units.sh
# generates it; it is gitignored and never committed.
ExecStart=__REPO_DIR__/.build/xrpld --conf docker/telemetry/xrpld-telemetry-mainnet2.host.cfg
Restart=on-failure
RestartSec=5s
# xrpld flushes its OpenTelemetry buffers on shutdown; killing it early loses the