The collector bind-mounts one log root, data/logs, and identifies each node by
the subdirectory inside it. The second instance wrote to data2/logs, which is not
mounted at all, so its logs have never reached the pipeline under any name --
while its metrics and traces flowed normally, making it look like a quiet node
rather than an uncollected one. Its own config comment already claimed the
directory was "the same dir the compose stack bind-mounts into the collector",
so the intent was the single root and only the path was wrong.
Both nodes now log under data/logs. The nodestore stays split across data/ and
data2/, which is what those directories are for; the logs gain nothing from the
split because both sit on the same disk, and lose collection entirely.
The installer's log-directory check was wrong in the same way: it verified a path
under the host's /var/log/xrpld, which nothing reads -- that path exists only
inside the container, where the root is mounted. It now checks the directory the
collector actually reads, and the docs no longer ask for host symlinks that have
no effect.
Also drop the trailing slash from the data ignore rules and add data2. A
trailing-slash pattern matches only real directories, so on a host that follows
the runbook and symlinks both to a fast disk, neither was ignored -- leaving them
permanently untracked in a tree the runbook says should read clean, and one
`git add` away from committing a nodestore.
A node's service_instance_id should say which box the data came from, so a
dashboard can tell two otherwise-identical instances apart. A machine name is
also exactly what this public repository should not carry, and the value cannot
come from the environment: it is read only from the [telemetry] section, and the
config parser has no include directive.
So the tracked configs keep a generic identity and name no host, and the
installer renders each into a .host.cfg beside it with the id substituted from
.env.devbox. The units run the rendered copies, which are gitignored. The
tracked configs are never edited on the host, so an update cannot conflict and a
rebuild loses nothing.
The identity is substituted in two places from one value -- the setting and the
log directory name -- because they have to agree: the collector derives identity
for the logs pipeline from the log path, so a mismatch costs that node's logs
their service_instance_id label while its metrics keep theirs, which reads as
"no logs" rather than as a misconfiguration. The installer counts the
occurrences it expects to replace and verifies the result, so a config reshuffle
fails loudly instead of yielding a copy that quietly kept the generic identity;
on the dashboards that would look like the node had disappeared. It also rejects
an id that is not a safe directory name, and rejects two nodes sharing one id.
Also record what the last deployment needed and the runbook did not say: the
session-bus variable a rootless container runtime needs before its user units
will install, the link from the unit's expected binary path to the preset's
build directory, and that a config merge which redeclares service.extensions
drops the cloud authenticator and stops the collector exporting anything at all.
These files also had not been through the formatting hooks, which want 4-space
indentation and reflowed tables; that is fixed here too.
The collector exits at startup whenever the Grafana Cloud stack is brought up:
cannot start pipelines: failed to start "otlphttp/grafanacloud" exporter:
failed to resolve authenticator "basicauth/grafanacloud": authenticator not found
The developer stack passes otel-collector-filestorage.yaml as a second --config to
persist filelog read offsets. The collector replaces lists on merge rather than
appending them, which that file notes, and it repeats the BASE config's
service.extensions -- [health_check, file_storage/filelog]. Layer the cloud
overlay on top and the base config is swapped for the cloud one, whose list also
carries basicauth/grafanacloud; the storage overlay still comes last, so the
authenticator is dropped and the exporter that needs it cannot start.
Nothing degrades: the collector refuses to start at all, so a host brought up this
way exports no telemetry anywhere, local or cloud.
Adds a cloud variant of the overlay naming all three extensions, mounted by
docker-compose.grafanacloud.yaml over the base one so the command line is
unchanged. The base variant stays as it is for stacks that do not use the cloud
exporter.
Following the previous version on a freshly provisioned host failed three times,
each in a way that pointed away from the actual cause. Replacing the build section
with what actually works, and recording why each wrong turn misleads.
The daemon is not built by default: the Conan recipe defaults its xrpld option to
off. Omit the option and both Conan and CMake report success, then the build fails
with "No rule to make target 'xrpld'" because the target was never created.
Configure through the Conan-generated preset rather than a hand-written toolchain
path. The toolchain does not sit where the single- versus multi-config layouts
suggest, and a wrong path fails with "Could not find toolchain file" followed by
"CMAKE_CXX_COMPILER not set", which reads as a broken compiler.
The build directory is .build/build/Release, not .build. Building the wrong one
reports "Generator: execution of make failed", which reads as a toolchain problem.
The CMake cache is sticky: re-running conan install with a changed option does not
update an already-cached CMake variable, so the target stays absent even though
the option was accepted. Added the check and the cache-clearing steps.
Standing up and updating one of these hosts involves a handful of steps that fail
quietly when missed, and the knowledge was living in people's heads and in an
untracked file on the host itself. That file died with the last rebuild.
Covers updating the checkout, re-applying the host overlays a checkout can
disturb, building, and starting the collector and both nodes, then verifying all
three signals actually arrive rather than trusting the configuration.
The failure modes it calls out are ones already hit here: bringing the collector
up without the cloud overlay, which exports nothing and logs no error; omitting
--force-recreate, so a config change appears applied but is not; a log directory
whose name does not match the instance id, which leaves logs unlabelled while
metrics stay labelled; and reading the container command line to check which
config is active, which cannot distinguish the two cases because the overlay
mounts over the same path.
It also records two signals that read backwards: a higher acquisition timeout
count can mean a node is retrying forward more cheaply rather than stalling, and
outbound byte counters are recorded at the decision to send rather than at the
wire, so under a flood they overstate egress by orders of magnitude.
Deliberately names no credential or environment files and carries no host
identity, so it is safe in a public tree. Which file holds what is documented
where it is needed, next to the installer that reads it.
Running two nodes on one host is how the storage backends get compared with
everything else equal, but only the first instance's config was ever committed.
The second node's config, both systemd units, and the collector's per-node log
identity lived on the host alone and were lost when it was rebuilt.
Adds xrpld-telemetry-mainnet2.cfg, derived from the first and differing only
where two nodes on one host must differ: ports, data and log paths,
service_instance_id, and [node_db] type. Ports continue the offset-by-ten scheme
already in use -- devnet on 5005, Mainnet on 5015, so this one on 5025 -- so all
three can bind together.
Adds the units as templates plus install-units.sh, which fills them from an
untracked .env.devbox. This repository is public and f64f4b35e7 already removed a
personal home directory from shipped config, so the run user and checkout path
must not come back into git. The installer refuses an env file that is not mode
600 and refuses to install a unit still holding an unsubstituted placeholder, so
a half-configured host fails loudly rather than producing a unit systemd never
starts.
Both units carry RequiresMountsFor for the data mount. That mount's fstab entry
normally uses 'nofail' so a missing disk does not block boot, which also means it
fails silently; without the guard a node starts anyway and writes its nodestore
to the root filesystem until it fills.
Re-adds the collector's per-node log identity: include_file_path plus operators
lifting the log directory basename into service.instance.id. Without it the logs
pipeline carries no service_instance_id label while the metrics pipeline does, so
the dashboards' $node filter matches nothing for logs and reads as "no logs"
rather than as a misconfiguration.
Renames the first instance's log directory from data/logs/mainnet to
data/logs/xrpld-mainnet, the one change to that file: the derivation above reads
the basename, so it has to equal the service_instance_id for a node's logs and
metrics to carry the same label.
No credentials or host-identifying values here. .env.devbox and .env.grafanacloud
are both covered by the .env.* ignore rule; only the examples are tracked.
Nine conflicts, resolved as follows.
src/xrpld/app/ledger/detail/InboundLedger.cpp -- kept this branch's version.
phase10 sets the span's outcome/timeouts/peer_count attributes inline at each
exit; this branch replaced that with the idempotent finalizeAcquireSpan(), called
on all four exits (init, done, give-up, destructor). Taking phase10's blocks
would have set the outcome twice against a helper documented as not overwriting
what the real exit recorded. phase10's comment explains why peer_count must not
be read in a destructor; the helper solves that structurally by taking
std::optional<std::size_t> and being passed std::nullopt from there.
src/xrpld/telemetry/MetricsRegistry.cpp -- kept metric::ledgerEconomy over
phase10's "ledger_economy" literal. This branch added the naming check that
requires constants for converted families, so the literal would regress it. Took
phase10's comment cleanup.
src/xrpld/telemetry/MetricsRegistry.h -- kept registerRotationStateGauge(), which
only exists here, and took phase10's removal of the stale task-number comment.
validate_telemetry.py -- combined both. phase10 replaced serial metric polling
with a concurrent fan-out on one shared deadline, because 58 metrics x 45 s of
additive timeout overran the CI budget; that is kept. Its target list filters on
SKIPPED_METRIC_GROUPS rather than the two literals it hardcoded, so the
sync_diagnostics group stays owned by assert_sync_diagnostics_metrics() instead
of being polled and reported twice. Both SYNC_DIAGNOSTICS_GROUP and
METRIC_POLL_CONCURRENCY are needed and both are kept.
check_otel_naming.py -- both sides extend the rule docstring. Took phase10's
fuller Rule E text (doc discovery, allow-dotted markers) and re-appended rules
I/J/K/L, which exist only here.
expected_metrics.json -- the two sides add disjoint sibling groups, so both are
kept: sync_diagnostics alongside node_health_gauges, overlay_reduce_relay,
overlay_overflow, validation_lifetime_counters and not_asserted. Both dashboard
uids are kept, giving 16 asserted uids against 16 dashboards on disk.
expected_spans.json -- kept this branch's span set, a superset that adds the
acquire phase spans, ledger.serve, txset.acquire and peer.dial, and expands
ledger.acquire's required attributes. Took phase10's description, which documents
what the totals mean, and its note on how the RPC wildcard span is created.
total_span_types and total_unique_attributes are recomputed for the union: 48 and
74, since each side's figure counted only its own spans.
Docs: took phase10's more accurate wording on what the dashboard check actually
covers, and corrected the dashboard count from 15 to 16 where the merge made it
stale.
Verified: no conflict markers remain, both JSON contracts parse, both Python
files compile, asserted dashboard uids match the dashboards on disk exactly, and
the OTel naming check reports all layers consistent.
The Operating Mode Transitions panel queried state_accounting_*_transitions
directly. Those are monotonic counters, so the panel drew a slowly rising line
and a few transitions per hour were invisible against a total in the hundreds.
It also fell off a cliff whenever xrpld restarted and the counters reset to 0,
which reads as missing data rather than a restart.
Wrap each target in increase(...[$__rate_interval]) so each point is the number
of transitions in that bucket and the series survives a counter reset. This is
what the sibling panels on the same row (Operating Mode (Time Share), State
Duration Rate) already do.
Verified against devnet-otel-usw2-01/02 over 2026-08-11T11:01Z..2026-08-12T16:23Z:
the fixed expression reports 107 and 123 syncing transitions, matching the
counter deltas, and stays continuous across the 12:07 restart where the raw
counter dropped 630 -> 1.
Brief mode flaps remain invisible on Operating Mode (State Timeline) because a
~2 s dwell cannot be captured by a 60 s scrape; this panel is the place to read
them.
This branch had already made the same corrections independently, and in
richer form, so the resolution keeps this branch's version nearly throughout:
- 09-data-collection-reference.md: this branch already documents the
state-accounting gauges as cumulative **microseconds** with an explanatory
callout, and already names `jobq_job_count` with its `jobq` group. Kept.
- telemetry-runbook.md: already carries `jobq_job_count` in both tables. Kept,
along with this branch's larger additions.
- OpenTelemetryPlan.md: kept this branch's rewritten section 9 blurb, which
describes the inventory without hardcoding counts and so cannot drift.
- consensus-health.json: kept this branch's rewrite. It deliberately removed
the four TraceQL close-time detail panels and renamed the agreement panel;
the incoming side would have resurrected them. Panel count unchanged at 26.
- integration-test.sh: this branch's unprefixed native metric names were kept,
but it still asserted `job_count`, so the `jobq_job_count` correction was
carried over. That check would otherwise always fail.
The troubleshooting step queried `job_count`, which returns no series. The
gauge is registered as `makeGauge("job_count")` but `Application.cpp` passes
`collectorManager_->group("jobq")`, so the exported name carries the `jobq`
segment. The two metric tables in this file were corrected when phase-6
merged forward; this example was missed because it sits outside the tables.
Conflict resolution kept this branch's evolution and re-applied phase-6's
fixes on top of it, rather than taking either side wholesale:
- consensus-health.json: kept the native `span_calls_total` metric name and
the `interval: 15s` and point styling from this branch; added phase-6's
`close_time_correct` PromQL filter and the NetClock axis labels. The
TraceQL boolean-regex filter stays removed and the `byRegexp` overrides
carry over. Panel count unchanged at 27.
- 09-data-collection-reference.md: kept this branch's headings, its more
detailed consensus attribute table (which already types
`consensus_round_id` as int64) and its section numbering, including the
deliberate removal of the SpanNames inventory. Carried over only the
correction that the state-accounting duration gauges are cumulative
microseconds, not seconds.
- telemetry-runbook.md: kept this branch's native metric names
(`span_calls_total`, `span_duration_milliseconds_bucket`); carried the
`rpc.request` -> `rpc.http_request` span-name fix and the `jobq_` segment
on the job-queue depth metric.
- integration-test.sh: kept this branch's `check_otel_metric` form and
carried the `jobq_job_count` correction.
The integration test asserted `rippled_job_count`, which never reports any
series, so that check always failed. `JobQueue` registers the gauge as
`makeGauge("job_count")`, but `Application.cpp` passes it
`collectorManager_->group("jobq")`, so the emitted StatsD name is
`jobq.job_count` and the exported Prometheus name is
`<prefix>_jobq_job_count`.
Corrected the same name in two runbook tables that also dropped the `jobq`
segment. `09-data-collection-reference.md` already had it right, which is why
the two documents disagreed.
Routed here rather than to the phase-10 PR where it was reported: the wrong
name is present in `integration-test.sh` on every branch from phase 6
onward, and this is the branch that introduces the file.
Left alone deliberately:
- `statsd-node-health.json` still queries the old name, but that dashboard is
deleted at phase 7 in favour of `node-health.json`
- `06-implementation-phases.md` names `job_count`, which is accurate as the
code-level makeGauge argument rather than the exported metric name
The file_storage extension was added to otel-collector-config.yaml, which
every stack mounts. That made the extension mandatory: the collector image
runs as 10001:10001 and ships no writable directory, so any stack without a
prepared volume would fail to start rather than merely lose offsets. The
workload-validation stack mounts this same config and has no such volume.
Offset persistence is only useful where logs outlive a restart. The workload
harness creates a fresh log directory per run, so it has nothing to resume
from. Move the extension, the receiver's storage reference and the extended
service.extensions list into otel-collector-filestorage.yaml, layered as a
second --config by the developer stack alone. The base config keeps
start_at: beginning, which is what actually fixes the reported defect, and
stays self-sufficient for every other stack.
Verified against the pinned collector image: the base config validates and
runs on its own with no volume mounted and still ingests a line written
before startup; base plus overlay validates, preserves the base receiver's
operators through the merge, and re-ingests that line zero times on a second
run against the same volume.
Conflict resolution, OpenTelemetryPlan/09-data-collection-reference.md
Known Issues table: kept phase-9's expanded 17-row table rather than the
incoming 4-row version, and re-applied the incoming correction to the one
row it changed. The other three incoming rows are already present in
phase-9's table, and phase-9's version is more current (jobq_job_count,
not job_count).
The rpc_requests row now says [insight] server=otel; server=statsd
reaches a collector with no statsd receiver on this branch. The section 8
Quick Reference and the fallback caveat auto-merged cleanly.
