Collect the second node's logs, which were never mounted

The collector bind-mounts one log root, data/logs, and identifies each node by
the subdirectory inside it. The second instance wrote to data2/logs, which is not
mounted at all, so its logs have never reached the pipeline under any name --
while its metrics and traces flowed normally, making it look like a quiet node
rather than an uncollected one. Its own config comment already claimed the
directory was "the same dir the compose stack bind-mounts into the collector",
so the intent was the single root and only the path was wrong.

Both nodes now log under data/logs. The nodestore stays split across data/ and
data2/, which is what those directories are for; the logs gain nothing from the
split because both sit on the same disk, and lose collection entirely.

The installer's log-directory check was wrong in the same way: it verified a path
under the host's /var/log/xrpld, which nothing reads -- that path exists only
inside the container, where the root is mounted. It now checks the directory the
collector actually reads, and the docs no longer ask for host symlinks that have
no effect.

Also drop the trailing slash from the data ignore rules and add data2. A
trailing-slash pattern matches only real directories, so on a host that follows
the runbook and symlinks both to a fast disk, neither was ignored -- leaving them
permanently untracked in a tree the runbook says should read clean, and one
`git add` away from committing a nodestore.
This commit is contained in:
Pratik Mankawde
2026-08-18 16:06:57 +01:00
parent 4988533d96
commit 953294f113
5 changed files with 51 additions and 31 deletions

View File

@@ -21,7 +21,7 @@ backend comparison.
| Backend | NuDB | RocksDB |
| rpc / ws-admin / ws-public / peer | 5015 / 6016 / 6015 / 51245 | 5025 / 6026 / 6025 / 51255 |
| Data | `data/mainnet` | `data2/mainnet` |
| Logs | `data/logs/$NODE1_INSTANCE_ID/` | `data2/logs/$NODE2_INSTANCE_ID/` |
| Logs | `data/logs/$NODE1_INSTANCE_ID/` | `data/logs/$NODE2_INSTANCE_ID/` |
Ports continue the offset-by-ten scheme already in use — devnet on 5005, Mainnet
on 5015 — so all three configs can bind on one host.
@@ -83,16 +83,23 @@ for the job pool.
## Two things that are easy to get wrong
**The log directory basename must equal the `service_instance_id`.** The
installer substitutes both from one value so they cannot drift, but the
directory itself still has to exist under that name. The
collector's filelog receiver derives per-node identity from the log path
(`include_file_path` plus a regex on `/xrpld/<id>/debug.log`). Name the directory
anything else and that node's _logs_ lose their `service_instance_id` label while
its _metrics_ keep theirs — so the dashboards' `$node` filter matches nothing for
logs and reads as "no logs" rather than as a misconfiguration. The collector
expects the logs under `/var/log/xrpld/<id>/`, so symlink or bind-mount each
node's log directory there.
**The log directory basename must equal the `service_instance_id`, and sit in the
one log root.** The installer substitutes the id into both the setting and the
path from one value so they cannot drift, but the directory itself still has to
exist under that name. The collector's filelog receiver derives per-node identity
from the log path (`include_file_path` plus a regex on `/xrpld/<id>/debug.log`).
Name the directory anything else and that node's _logs_ lose their
`service_instance_id` label while its _metrics_ keep theirs — so the dashboards'
`$node` filter matches nothing for logs and reads as "no logs" rather than as a
misconfiguration.
The root is `data/logs`, which compose mounts into the collector as
`/var/log/xrpld`. That is a path inside the container, not on the host, so
nothing needs creating in the host's `/var/log`. **Both** nodes log under
`data/logs`, including the one whose nodestore is under `data2/`: a log directory
outside the mounted root is never read at all, and since both data directories
sit on the same disk, splitting the logs would buy no I/O separation to pay for
the lost collection.
**Put the data directories on fast local storage.** The configs use
repo-relative paths so they stay portable; point them at the fast disk with

View File

@@ -147,23 +147,25 @@ render_cfg() {
render_cfg "$tel/xrpld-telemetry-mainnet.cfg" "$NODE1_INSTANCE_ID" "$tel/xrpld-telemetry-mainnet.host.cfg"
render_cfg "$tel/xrpld-telemetry-mainnet2.cfg" "$NODE2_INSTANCE_ID" "$tel/xrpld-telemetry-mainnet2.host.cfg"
# The log directories have to exist under both the names the node writes and the
# path the collector reads, or one node's logs go unlabelled while everything
# else looks healthy.
# Each node needs a log directory named after its instance id inside the log root
# the collector mounts (data/logs, exposed to the container as /var/log/xrpld).
# A directory missing here, or placed outside that root, costs that node its logs
# while its metrics keep flowing -- which reads as a quiet node, not as a
# collection gap.
check_log_dir() {
node_dir=$1
id=$2
[ -d "$tel/$node_dir/logs/$id" ] || cat >&2 <<EOF
WARN: $tel/$node_dir/logs/$id does not exist; create it and expose it to the collector:
WARN: $tel/$node_dir/logs/$id does not exist, so the collector will find no logs for '$id'. Create it:
mkdir -p "$tel/$node_dir/logs/$id"
sudo mkdir -p /var/log/xrpld
sudo ln -sfn "$tel/$node_dir/logs/$id" /var/log/xrpld/$id
EOF
[ -e "/var/log/xrpld/$id" ] || echo "WARN: /var/log/xrpld/$id is missing; the collector will find no logs for '$id'" >&2
}
# Both nodes log under data/, whatever their nodestore uses: the collector
# mounts that one directory as its log root and identifies each node by the
# subdirectory name, so a log directory outside it is never read.
check_log_dir data "$NODE1_INSTANCE_ID"
check_log_dir data2 "$NODE2_INSTANCE_ID"
check_log_dir data "$NODE2_INSTANCE_ID"
for unit in xrpld-mainnet xrpld-mainnet2; do
tpl="$here/$unit.service.template"