The collector bind-mounts one log root, data/logs, and identifies each node by
the subdirectory inside it. The second instance wrote to data2/logs, which is not
mounted at all, so its logs have never reached the pipeline under any name --
while its metrics and traces flowed normally, making it look like a quiet node
rather than an uncollected one. Its own config comment already claimed the
directory was "the same dir the compose stack bind-mounts into the collector",
so the intent was the single root and only the path was wrong.
Both nodes now log under data/logs. The nodestore stays split across data/ and
data2/, which is what those directories are for; the logs gain nothing from the
split because both sit on the same disk, and lose collection entirely.
The installer's log-directory check was wrong in the same way: it verified a path
under the host's /var/log/xrpld, which nothing reads -- that path exists only
inside the container, where the root is mounted. It now checks the directory the
collector actually reads, and the docs no longer ask for host symlinks that have
no effect.
Also drop the trailing slash from the data ignore rules and add data2. A
trailing-slash pattern matches only real directories, so on a host that follows
the runbook and symlinks both to a fast disk, neither was ignored -- leaving them
permanently untracked in a tree the runbook says should read clean, and one
`git add` away from committing a nodestore.
A node's service_instance_id should say which box the data came from, so a
dashboard can tell two otherwise-identical instances apart. A machine name is
also exactly what this public repository should not carry, and the value cannot
come from the environment: it is read only from the [telemetry] section, and the
config parser has no include directive.
So the tracked configs keep a generic identity and name no host, and the
installer renders each into a .host.cfg beside it with the id substituted from
.env.devbox. The units run the rendered copies, which are gitignored. The
tracked configs are never edited on the host, so an update cannot conflict and a
rebuild loses nothing.
The identity is substituted in two places from one value -- the setting and the
log directory name -- because they have to agree: the collector derives identity
for the logs pipeline from the log path, so a mismatch costs that node's logs
their service_instance_id label while its metrics keep theirs, which reads as
"no logs" rather than as a misconfiguration. The installer counts the
occurrences it expects to replace and verifies the result, so a config reshuffle
fails loudly instead of yielding a copy that quietly kept the generic identity;
on the dashboards that would look like the node had disappeared. It also rejects
an id that is not a safe directory name, and rejects two nodes sharing one id.
Also record what the last deployment needed and the runbook did not say: the
session-bus variable a rootless container runtime needs before its user units
will install, the link from the unit's expected binary path to the preset's
build directory, and that a config merge which redeclares service.extensions
drops the cloud authenticator and stops the collector exporting anything at all.
These files also had not been through the formatting hooks, which want 4-space
indentation and reflowed tables; that is fixed here too.
Running two nodes on one host is how the storage backends get compared with
everything else equal, but only the first instance's config was ever committed.
The second node's config, both systemd units, and the collector's per-node log
identity lived on the host alone and were lost when it was rebuilt.
Adds xrpld-telemetry-mainnet2.cfg, derived from the first and differing only
where two nodes on one host must differ: ports, data and log paths,
service_instance_id, and [node_db] type. Ports continue the offset-by-ten scheme
already in use -- devnet on 5005, Mainnet on 5015, so this one on 5025 -- so all
three can bind together.
Adds the units as templates plus install-units.sh, which fills them from an
untracked .env.devbox. This repository is public and f64f4b35e7 already removed a
personal home directory from shipped config, so the run user and checkout path
must not come back into git. The installer refuses an env file that is not mode
600 and refuses to install a unit still holding an unsubstituted placeholder, so
a half-configured host fails loudly rather than producing a unit systemd never
starts.
Both units carry RequiresMountsFor for the data mount. That mount's fstab entry
normally uses 'nofail' so a missing disk does not block boot, which also means it
fails silently; without the guard a node starts anyway and writes its nodestore
to the root filesystem until it fills.
Re-adds the collector's per-node log identity: include_file_path plus operators
lifting the log directory basename into service.instance.id. Without it the logs
pipeline carries no service_instance_id label while the metrics pipeline does, so
the dashboards' $node filter matches nothing for logs and reads as "no logs"
rather than as a misconfiguration.
Renames the first instance's log directory from data/logs/mainnet to
data/logs/xrpld-mainnet, the one change to that file: the derivation above reads
the basename, so it has to equal the service_instance_id for a node's logs and
metrics to carry the same label.
No credentials or host-identifying values here. .env.devbox and .env.grafanacloud
are both covered by the .env.* ignore rule; only the examples are tracked.
