72d2c7d49f
Running two nodes on one host is how the storage backends get compared with
everything else equal, but only the first instance's config was ever committed.
The second node's config, both systemd units, and the collector's per-node log
identity lived on the host alone and were lost when it was rebuilt.
Adds xrpld-telemetry-mainnet2.cfg, derived from the first and differing only
where two nodes on one host must differ: ports, data and log paths,
service_instance_id, and [node_db] type. Ports continue the offset-by-ten scheme
already in use -- devnet on 5005, Mainnet on 5015, so this one on 5025 -- so all
three can bind together.
Adds the units as templates plus install-units.sh, which fills them from an
untracked .env.devbox. This repository is public and f64f4b35e7 already removed a
personal home directory from shipped config, so the run user and checkout path
must not come back into git. The installer refuses an env file that is not mode
600 and refuses to install a unit still holding an unsubstituted placeholder, so
a half-configured host fails loudly rather than producing a unit systemd never
starts.
Both units carry RequiresMountsFor for the data mount. That mount's fstab entry
normally uses 'nofail' so a missing disk does not block boot, which also means it
fails silently; without the guard a node starts anyway and writes its nodestore
to the root filesystem until it fills.
Re-adds the collector's per-node log identity: include_file_path plus operators
lifting the log directory basename into service.instance.id. Without it the logs
pipeline carries no service_instance_id label while the metrics pipeline does, so
the dashboards' $node filter matches nothing for logs and reads as "no logs"
rather than as a misconfiguration.
Renames the first instance's log directory from data/logs/mainnet to
data/logs/xrpld-mainnet, the one change to that file: the derivation above reads
the basename, so it has to equal the service_instance_id for a node's logs and
metrics to carry the same label.
No credentials or host-identifying values here. .env.devbox and .env.grafanacloud
are both covered by the .env.* ignore rule; only the examples are tracked.
4.2 KiB
Running two xrpld instances on one host
Templates and an installer for running two nodes side by side, one per storage backend, so NuDB and RocksDB can be compared with everything else equal.
These are committed deliberately. They previously existed only on the host and were lost when that machine was rebuilt, which meant reconstructing the RocksDB node's settings from notes.
What differs between the two
Everything else is identical on purpose — any other divergence would confound the backend comparison.
| instance 1 | instance 2 | |
|---|---|---|
| Unit | xrpld-mainnet |
xrpld-mainnet2 |
| Config | xrpld-telemetry-mainnet.cfg |
xrpld-telemetry-mainnet2.cfg |
service_instance_id |
xrpld-mainnet |
xrpld-mainnet2 |
| Backend | NuDB | RocksDB |
| rpc / ws-admin / ws-public / peer | 5015 / 6016 / 6015 / 51245 | 5025 / 6026 / 6025 / 51255 |
| Data | data/mainnet |
data2/mainnet |
| Logs | data/logs/xrpld-mainnet/ |
data2/logs/xrpld-mainnet2/ |
Ports continue the offset-by-ten scheme already in use — devnet on 5005, Mainnet on 5015 — so all three configs can bind on one host.
No host-specific values in git
This repository is public, and an earlier commit already removed a personal home
directory from shipped config. The unit templates therefore carry placeholders,
and the real values live in an untracked .env.devbox:
cp docker/telemetry/.env.devbox.example docker/telemetry/.env.devbox
chmod 600 docker/telemetry/.env.devbox
$EDITOR docker/telemetry/.env.devbox # RUN_USER, REPO_DIR, DATA_MOUNT
.env.* is gitignored, so the real file cannot be committed. The installer
refuses to run if the file is not mode 600, and refuses to install a unit that
still contains an unsubstituted placeholder.
Install
sh docker/telemetry/systemd/install-units.sh
sudo systemctl start xrpld-mainnet # wait for tracking/full
sudo systemctl start xrpld-mainnet2
Staggering the starts is a nicety rather than a requirement when the data directories are on fast local storage, but two bootstrapping nodes still contend for the job pool.
Two things that are easy to get wrong
The log directory basename must equal the service_instance_id. The
collector's filelog receiver derives per-node identity from the log path
(include_file_path plus a regex on /xrpld/<id>/debug.log). Name the directory
anything else and that node's logs lose their service_instance_id label while
its metrics keep theirs — so the dashboards' $node filter matches nothing for
logs and reads as "no logs" rather than as a misconfiguration. The collector
expects the logs under /var/log/xrpld/<id>/, so symlink or bind-mount each
node's log directory there.
Put the data directories on fast local storage. The configs use repo-relative paths so they stay portable; point them at the fast disk with symlinks:
mkdir -p "$DATA_MOUNT"/xrpld/data "$DATA_MOUNT"/xrpld/data2
ln -sfn "$DATA_MOUNT"/xrpld/data docker/telemetry/data
ln -sfn "$DATA_MOUNT"/xrpld/data2 docker/telemetry/data2
Measured on an i4i instance: moving the nodestore off EBS onto the instance-store
NVMe took time-to-full from 2234 s to 681 s (3.3x), because NuDB's roughly
tenfold key-file write amplification saturated EBS while the local NVMe sat near
idle.
If that storage is an instance store, note it is volatile — contents survive
a reboot but are lost on a stop/start. Both units carry RequiresMountsFor, so a
missing mount fails the unit loudly instead of silently filling the root
filesystem.
Telemetry
Both instances export OTLP to the collector on localhost:4318. Bring the
collector up with both compose files:
docker compose -f docker/telemetry/docker-compose.yml \
-f docker/telemetry/docker-compose.grafanacloud.yaml up -d --force-recreate
The base file alone yields a collector with only local exporters, so telemetry
silently never leaves the host and the Cloud dashboards read empty with no error
anywhere. --force-recreate is needed after a config change, or Compose reuses
the running container with its old config.
Grafana Cloud credentials go in .env.grafanacloud, also gitignored, also mode
600.