4988533d96
A node's service_instance_id should say which box the data came from, so a dashboard can tell two otherwise-identical instances apart. A machine name is also exactly what this public repository should not carry, and the value cannot come from the environment: it is read only from the [telemetry] section, and the config parser has no include directive. So the tracked configs keep a generic identity and name no host, and the installer renders each into a .host.cfg beside it with the id substituted from .env.devbox. The units run the rendered copies, which are gitignored. The tracked configs are never edited on the host, so an update cannot conflict and a rebuild loses nothing. The identity is substituted in two places from one value -- the setting and the log directory name -- because they have to agree: the collector derives identity for the logs pipeline from the log path, so a mismatch costs that node's logs their service_instance_id label while its metrics keep theirs, which reads as "no logs" rather than as a misconfiguration. The installer counts the occurrences it expects to replace and verifies the result, so a config reshuffle fails loudly instead of yielding a copy that quietly kept the generic identity; on the dashboards that would look like the node had disappeared. It also rejects an id that is not a safe directory name, and rejects two nodes sharing one id. Also record what the last deployment needed and the runbook did not say: the session-bus variable a rootless container runtime needs before its user units will install, the link from the unit's expected binary path to the preset's build directory, and that a config merge which redeclares service.extensions drops the cloud authenticator and stops the collector exporting anything at all. These files also had not been through the formatting hooks, which want 4-space indentation and reflowed tables; that is fixed here too.
Running two xrpld instances on one host
Templates and an installer for running two nodes side by side, one per storage backend, so NuDB and RocksDB can be compared with everything else equal.
These are committed deliberately. They previously existed only on the host and were lost when that machine was rebuilt, which meant reconstructing the RocksDB node's settings from notes.
What differs between the two
Everything else is identical on purpose — any other divergence would confound the backend comparison.
| instance 1 | instance 2 | |
|---|---|---|
| Unit | xrpld-mainnet |
xrpld-mainnet2 |
| Tracked config | xrpld-telemetry-mainnet.cfg |
xrpld-telemetry-mainnet2.cfg |
| Config the unit runs | xrpld-telemetry-mainnet.host.cfg |
xrpld-telemetry-mainnet2.host.cfg |
service_instance_id |
$NODE1_INSTANCE_ID |
$NODE2_INSTANCE_ID |
| Backend | NuDB | RocksDB |
| rpc / ws-admin / ws-public / peer | 5015 / 6016 / 6015 / 51245 | 5025 / 6026 / 6025 / 51255 |
| Data | data/mainnet |
data2/mainnet |
| Logs | data/logs/$NODE1_INSTANCE_ID/ |
data2/logs/$NODE2_INSTANCE_ID/ |
Ports continue the offset-by-ten scheme already in use — devnet on 5005, Mainnet on 5015 — so all three configs can bind on one host.
No host-specific values in git
This repository is public, and an earlier commit already removed a personal home
directory from shipped config. The unit templates therefore carry placeholders,
and the real values live in an untracked .env.devbox:
cp docker/telemetry/.env.devbox.example docker/telemetry/.env.devbox
chmod 600 docker/telemetry/.env.devbox
$EDITOR docker/telemetry/.env.devbox
.env.* is gitignored, so the real file cannot be committed. The installer
refuses to run if the file is not mode 600, and refuses to install a unit that
still contains an unsubstituted placeholder.
Telemetry identity names the machine, so it is not committed
Each node reports a service_instance_id on every metric, span and log line.
It should name the machine, so a dashboard shows which box the data came
from — but a machine name is exactly what a public repository should not carry.
So the tracked configs keep a generic identity and name no host, and the
installer renders each one into a .host.cfg beside it with
NODE1_INSTANCE_ID / NODE2_INSTANCE_ID substituted in. The units run the
rendered copies; *.host.cfg is gitignored. The tracked configs are never
edited, so an update never conflicts and a rebuild loses nothing.
The identity is substituted in two places at once — the service_instance_id
setting and the log directory name — because they must agree, for the reason in
the next section. The installer counts the occurrences it expects to replace and
verifies the result, so a config reshuffle fails loudly instead of yielding a
copy that quietly kept the generic identity: on the dashboards that reads as the
node having disappeared, not as a failed substitution.
Reuse the host's established names. These values are the join key for everything already stored. Renaming a node starts a fresh series and silently breaks continuity with its own history — the old data is still there, under the old name, and no dashboard will show both.
Re-run the installer after changing either the tracked config or .env.devbox;
nothing else regenerates the rendered copies.
Install
sh docker/telemetry/systemd/install-units.sh
sudo systemctl start xrpld-mainnet # wait for tracking/full
sudo systemctl start xrpld-mainnet2
Staggering the starts is a nicety rather than a requirement when the data directories are on fast local storage, but two bootstrapping nodes still contend for the job pool.
Two things that are easy to get wrong
The log directory basename must equal the service_instance_id. The
installer substitutes both from one value so they cannot drift, but the
directory itself still has to exist under that name. The
collector's filelog receiver derives per-node identity from the log path
(include_file_path plus a regex on /xrpld/<id>/debug.log). Name the directory
anything else and that node's logs lose their service_instance_id label while
its metrics keep theirs — so the dashboards' $node filter matches nothing for
logs and reads as "no logs" rather than as a misconfiguration. The collector
expects the logs under /var/log/xrpld/<id>/, so symlink or bind-mount each
node's log directory there.
Put the data directories on fast local storage. The configs use repo-relative paths so they stay portable; point them at the fast disk with symlinks:
mkdir -p "$DATA_MOUNT"/xrpld/data "$DATA_MOUNT"/xrpld/data2
ln -sfn "$DATA_MOUNT"/xrpld/data docker/telemetry/data
ln -sfn "$DATA_MOUNT"/xrpld/data2 docker/telemetry/data2
Measured on an i4i instance: moving the nodestore off EBS onto the instance-store
NVMe took time-to-full from 2234 s to 681 s (3.3x), because NuDB's roughly
tenfold key-file write amplification saturated EBS while the local NVMe sat near
idle.
If that storage is an instance store, note it is volatile — contents survive
a reboot but are lost on a stop/start. Both units carry RequiresMountsFor, so a
missing mount fails the unit loudly instead of silently filling the root
filesystem.
Telemetry
Both instances export OTLP to the collector on localhost:4318. Bring the
collector up with both compose files:
docker compose -f docker/telemetry/docker-compose.yml \
-f docker/telemetry/docker-compose.grafanacloud.yaml up -d --force-recreate
The base file alone yields a collector with only local exporters, so telemetry
silently never leaves the host and the Cloud dashboards read empty with no error
anywhere. --force-recreate is needed after a config change, or Compose reuses
the running container with its old config.
Grafana Cloud credentials go in .env.grafanacloud, also gitignored, also mode
600.