The collector bind-mounts one log root, data/logs, and identifies each node by
the subdirectory inside it. The second instance wrote to data2/logs, which is not
mounted at all, so its logs have never reached the pipeline under any name --
while its metrics and traces flowed normally, making it look like a quiet node
rather than an uncollected one. Its own config comment already claimed the
directory was "the same dir the compose stack bind-mounts into the collector",
so the intent was the single root and only the path was wrong.
Both nodes now log under data/logs. The nodestore stays split across data/ and
data2/, which is what those directories are for; the logs gain nothing from the
split because both sit on the same disk, and lose collection entirely.
The installer's log-directory check was wrong in the same way: it verified a path
under the host's /var/log/xrpld, which nothing reads -- that path exists only
inside the container, where the root is mounted. It now checks the directory the
collector actually reads, and the docs no longer ask for host symlinks that have
no effect.
Also drop the trailing slash from the data ignore rules and add data2. A
trailing-slash pattern matches only real directories, so on a host that follows
the runbook and symlinks both to a fast disk, neither was ignored -- leaving them
permanently untracked in a tree the runbook says should read clean, and one
`git add` away from committing a nodestore.
A node's service_instance_id should say which box the data came from, so a
dashboard can tell two otherwise-identical instances apart. A machine name is
also exactly what this public repository should not carry, and the value cannot
come from the environment: it is read only from the [telemetry] section, and the
config parser has no include directive.
So the tracked configs keep a generic identity and name no host, and the
installer renders each into a .host.cfg beside it with the id substituted from
.env.devbox. The units run the rendered copies, which are gitignored. The
tracked configs are never edited on the host, so an update cannot conflict and a
rebuild loses nothing.
The identity is substituted in two places from one value -- the setting and the
log directory name -- because they have to agree: the collector derives identity
for the logs pipeline from the log path, so a mismatch costs that node's logs
their service_instance_id label while its metrics keep theirs, which reads as
"no logs" rather than as a misconfiguration. The installer counts the
occurrences it expects to replace and verifies the result, so a config reshuffle
fails loudly instead of yielding a copy that quietly kept the generic identity;
on the dashboards that would look like the node had disappeared. It also rejects
an id that is not a safe directory name, and rejects two nodes sharing one id.
Also record what the last deployment needed and the runbook did not say: the
session-bus variable a rootless container runtime needs before its user units
will install, the link from the unit's expected binary path to the preset's
build directory, and that a config merge which redeclares service.extensions
drops the cloud authenticator and stops the collector exporting anything at all.
These files also had not been through the formatting hooks, which want 4-space
indentation and reflowed tables; that is fixed here too.
Following the previous version on a freshly provisioned host failed three times,
each in a way that pointed away from the actual cause. Replacing the build section
with what actually works, and recording why each wrong turn misleads.
The daemon is not built by default: the Conan recipe defaults its xrpld option to
off. Omit the option and both Conan and CMake report success, then the build fails
with "No rule to make target 'xrpld'" because the target was never created.
Configure through the Conan-generated preset rather than a hand-written toolchain
path. The toolchain does not sit where the single- versus multi-config layouts
suggest, and a wrong path fails with "Could not find toolchain file" followed by
"CMAKE_CXX_COMPILER not set", which reads as a broken compiler.
The build directory is .build/build/Release, not .build. Building the wrong one
reports "Generator: execution of make failed", which reads as a toolchain problem.
The CMake cache is sticky: re-running conan install with a changed option does not
update an already-cached CMake variable, so the target stays absent even though
the option was accepted. Added the check and the cache-clearing steps.
Standing up and updating one of these hosts involves a handful of steps that fail
quietly when missed, and the knowledge was living in people's heads and in an
untracked file on the host itself. That file died with the last rebuild.
Covers updating the checkout, re-applying the host overlays a checkout can
disturb, building, and starting the collector and both nodes, then verifying all
three signals actually arrive rather than trusting the configuration.
The failure modes it calls out are ones already hit here: bringing the collector
up without the cloud overlay, which exports nothing and logs no error; omitting
--force-recreate, so a config change appears applied but is not; a log directory
whose name does not match the instance id, which leaves logs unlabelled while
metrics stay labelled; and reading the container command line to check which
config is active, which cannot distinguish the two cases because the overlay
mounts over the same path.
It also records two signals that read backwards: a higher acquisition timeout
count can mean a node is retrying forward more cheaply rather than stalling, and
outbound byte counters are recorded at the decision to send rather than at the
wire, so under a flood they overstate egress by orders of magnitude.
Deliberately names no credential or environment files and carries no host
identity, so it is safe in a public tree. Which file holds what is documented
where it is needed, next to the installer that reads it.
