detachCallbacks() flips a flag that each observable callback checks on entry,
which leaves a callback already past that check running while the twelve
service stops below it tear down the state it reads. Stop the provider at the
same point instead: that joins the reader thread, so once it returns no
callback is running and none can start. Metrics recorded during the remaining
shutdown steps are no longer exported, which is the cost of the guarantee.
Build metricsRegistry_ in the member-init list rather than assigning it in
setup(). getMetricsRegistry() is read from the job queue and io threads, which
are running by then, so the later assignment was an unsynchronised write to the
handle those reads follow.
beast::insight instruments are created during ApplicationImp's member-init
list, and opentelemetry-cpp 1.28 never rebinds an already-vended Meter, so an
instrument created before the MeterProvider is published records nothing for
the rest of the process. Observable instruments carry the opposite constraint:
registering one arms the SDK reader thread, and its callbacks run hook handlers
that read services which do not exist that early.
Publish the provider in Telemetry's constructor, ahead of every producer, and
defer only the observables. Collector gains onCollectionReady() and
onCollectionStopping(); OTelCollector arms and disarms its gauges in response.
StatsDCollector starts its polling thread in its own constructor and had the
same hazard, so it uses the pair to gate that thread.
The metrics resource carries service.instance.id and is immutable once built,
so the node public key is resolved in Main.cpp, where a config error can still
be reported, and passed to makeApplication(). getNodeIdentity() remains
authoritative; both paths now share readNodeIdentity(), so telemetry cannot
report a key the node has abandoned.
An explicit ~ApplicationImp stops observing and stops telemetry, covering the
setup() failure paths that never reach run(). Telemetry::stop() is once-only
and no longer clears another instance's global pointer. The histogram view's
meter selector now matches the meter actually in use, so its bucket boundaries
apply for the first time.
makeTelemetrySetup() rejects a contradictory [telemetry] mutual-TLS
setup by throwing, but it is called from ApplicationImp's
member-initializer list. A try/catch in the constructor body cannot
reach a throw from there, and nothing further up the stack caught it
either, so a config mistake reached std::terminate: the default handler
printed a terminate dump and raised SIGABRT, leaving a core file
instead of a startup error.
Catch std::exception around makeApplication() in run(), report the
reason on stderr and return -1, so the failure is a clean non-zero exit
with a message an operator can act on. Only the construction is
wrapped. setup() starts subsystems whose shutdown order is delicate and
is left outside deliberately, because unwinding a half-started
Application would skip the normal stop sequence.
Gate both validation guards on enabled. A node with telemetry switched
off previously refused to start over certificate paths that nothing
would read.
Document both throws on makeTelemetrySetup(), state in
cfg/xrpld-example.cfg and the configuration reference that a partial
mutual-TLS setup is fatal and that the checks apply only when
enabled=1, and add a runbook troubleshooting entry keyed on the two
error messages.
Tests cover both guards with the message asserted so the two are told
apart, both enabled=0 paths, and the default plaintext configuration.
Companion to the same fix on phase-9. These two uses exist only on this
branch, so they survived the merge-forward: develop moved TempDir from
beast:: to xrpl:: and deleted xrpl/beast/utility/temp_dir.h.
Database.cpp already includes xrpl/basics/FileUtilities.h and already
spells TempDir unqualified at line 288, so only the qualifier was wrong.
develop moved TempDir from beast:: to xrpl::, deleting
include/xrpl/beast/utility/temp_dir.h in favour of
include/xrpl/basics/FileUtilities.h. The merge kept this branch's
references to the old API, so the tree no longer compiled: the missing
header is a fatal include, and because DatabaseConfig_test.cpp lands in
the xrpld unity blob it broke the xrpld target itself, not just the tests.
Swap the include and drop the stale beast:: qualifier on 17 uses. The
three src/tests/libxrpl/nodestore files already include FileUtilities.h
and already spell TempDir unqualified elsewhere, so only the qualifier
was wrong there. DatabaseConfig_test.cpp needed the include as well; it
sits in namespace xrpl::node_store, so unqualified TempDir resolves to
xrpl::TempDir through the enclosing namespace.
Two further uses exist only on the sync-diagnostics tip and are fixed
there rather than here.
Node identity reached the OTel resource only as service.instance.id, which is
config-overridable and carries a deployment-chosen label rather than the node's
own identity. Add xrpl.node.id, set unconditionally from the node public key
(base58, TokenType::NodePublic), so traces and metrics share a stable per-node
key independent of [telemetry] service_instance_id.
Set on the tracer resource via Telemetry::setNodeId(), called from
ApplicationImp::setup() once nodeIdentity_ is known, and on the MetricsRegistry
resource via an added start() parameter. The beast::insight meter provider is
built in TelemetryImpl's constructor, before the wallet DB exists, so its
resource cannot carry the value; that path is left for later and the attribute
is omitted rather than stamped blank.
Also drops the transform/spanidentity collector processor added in
4a361a496d: per-node identity belongs on the resource, not copied onto every
span.
Nine conflicts, resolved as follows.
src/xrpld/app/ledger/detail/InboundLedger.cpp -- kept this branch's version.
phase10 sets the span's outcome/timeouts/peer_count attributes inline at each
exit; this branch replaced that with the idempotent finalizeAcquireSpan(), called
on all four exits (init, done, give-up, destructor). Taking phase10's blocks
would have set the outcome twice against a helper documented as not overwriting
what the real exit recorded. phase10's comment explains why peer_count must not
be read in a destructor; the helper solves that structurally by taking
std::optional<std::size_t> and being passed std::nullopt from there.
src/xrpld/telemetry/MetricsRegistry.cpp -- kept metric::ledgerEconomy over
phase10's "ledger_economy" literal. This branch added the naming check that
requires constants for converted families, so the literal would regress it. Took
phase10's comment cleanup.
src/xrpld/telemetry/MetricsRegistry.h -- kept registerRotationStateGauge(), which
only exists here, and took phase10's removal of the stale task-number comment.
validate_telemetry.py -- combined both. phase10 replaced serial metric polling
with a concurrent fan-out on one shared deadline, because 58 metrics x 45 s of
additive timeout overran the CI budget; that is kept. Its target list filters on
SKIPPED_METRIC_GROUPS rather than the two literals it hardcoded, so the
sync_diagnostics group stays owned by assert_sync_diagnostics_metrics() instead
of being polled and reported twice. Both SYNC_DIAGNOSTICS_GROUP and
METRIC_POLL_CONCURRENCY are needed and both are kept.
check_otel_naming.py -- both sides extend the rule docstring. Took phase10's
fuller Rule E text (doc discovery, allow-dotted markers) and re-appended rules
I/J/K/L, which exist only here.
expected_metrics.json -- the two sides add disjoint sibling groups, so both are
kept: sync_diagnostics alongside node_health_gauges, overlay_reduce_relay,
overlay_overflow, validation_lifetime_counters and not_asserted. Both dashboard
uids are kept, giving 16 asserted uids against 16 dashboards on disk.
expected_spans.json -- kept this branch's span set, a superset that adds the
acquire phase spans, ledger.serve, txset.acquire and peer.dial, and expands
ledger.acquire's required attributes. Took phase10's description, which documents
what the totals mean, and its note on how the RPC wildcard span is created.
total_span_types and total_unique_attributes are recomputed for the union: 48 and
74, since each side's figure counted only its own spans.
Docs: took phase10's more accurate wording on what the dashboard check actually
covers, and corrected the dashboard count from 15 to 16 where the merge made it
stale.
Verified: no conflict markers remain, both JSON contracts parse, both Python
files compile, asserted dashboard uids match the dashboards on disk exactly, and
the OTel naming check reports all layers consistent.
Conflict resolutions:
- docker/telemetry/xrpld-telemetry.cfg: relocation conflict. phase-9 had
already moved [insight] to the end of the file with server=otel, so the
incoming block was dropped rather than inserted. Keeping both would have
produced two [insight] sections, which merge last-wins into a single
effective section, silently reviving the bug this branch just fixed.
phase-9's per-branch service_instance_id=xrpld-devnet is preserved.
- OpenTelemetryPlan/06-implementation-phases.md: kept both corrections.
phase-9's "Tempo" is right (no Jaeger anywhere in the stack) and
phase-8's "active, sampled span" is right: Log.cpp:328 injects only
when spanCtx.IsValid() && spanCtx.IsSampled().
- OpenTelemetryPlan/09-data-collection-reference.md and
docs/telemetry-runbook.md: kept phase-9's structured-metadata LogQL.
The collector's filelog regex_parser already extracts partition,
severity, trace_id and span_id, so phase-8's inline regexp forms are
redundant, and a line filter matches the literal text in a message body.
Six findings from the review of #6494 survived independent verification.
Each was checked against the branch tip, and where behaviour was in
question, against a live collector and Loki rather than from the
reviewer's claim or from documentation alone.
Plan-doc section numbering. 06-implementation-phases.md used "## 6.9"
twice: for the new Phase 8 section and for the pre-existing Risk
Assessment. Three references already pointed at 6.8.1 and none at 6.9,
and the later phases are numbered 6.8.2 through 6.8.4, so Phase 8
becomes 6.8.1 and the sequence is monotonic. Renumbering to 6.10, as
suggested on the PR, would have collided with Success Metrics.
filelog read position. The receiver relied on the upstream default
start_at=end, which skips everything a node wrote before the first poll
and reads nothing at all from a log that has stopped being written to.
Read from the beginning instead, paired with a file_storage extension so
a restart resumes at the last offset rather than re-ingesting the file.
The collector image runs as 10001:10001 and ships no writable directory,
and a fresh named volume is root-owned, so a one-shot init service
prepares the volume first. It reuses an image the stack already pulls,
adding no new dependency.
Loki log stream label. The job resource attribute did not become a Loki
index label, so the documented {job="xrpld"} queries matched nothing.
Verified against grafana/loki:3.4.2 with its default config: only
service_name and deployment_environment are indexed, and job arrives as
structured metadata, which a stream selector cannot match. Dropped the
attribute and moved the twelve queries this branch introduced to
{service_name="xrpld"}. Three further occurrences in
07-observability-backends.md originate on the phase-1a branch and are
left for a commit there.
Trace ids on unsampled spans. Logs::format emitted trace_id and span_id
whenever the span context was valid. A span dropped by the
ParentBasedSampler still carries its parent's ids, so log lines
advertised traces that were never exported and the log-to-trace link
resolved to nothing. Require the sampled flag as well, and correct the
task list and the documentation that promised the fields unconditionally.
The remaining two findings were refuted. The reported risk of signing
material reaching Loki does not hold: Logs::format already scrubs seven
sensitive fields, and there is a single write path to the log file, so
every JLOG site is covered. The suggestion to add internalLink to the
Loki derived field is not applicable, because that key is not part of
Grafana's schema.
Four files conflicted.
xrpl.proto and TraceContextPropagator.h: phase-9's note is a semantic
superset of phase-8's and already carries no plan-folder pointer, so
phase-9's text is kept and nothing phase-8 said is lost.
node-health.json: the only change phase-8 makes to this file is renaming a
row that phase-9 had already deleted, so phase-9's file stands. Panel count
holds at 59 and both Validated Ledger Seq panels survive.
TESTING.md: resolved per hunk rather than by side. Phase-9's replacement of
the drifted span table and its new Grafana Cloud section are kept, and
phase-8's de-numbered "Test 3: Log-Trace Correlation" heading is taken --
keeping phase-9 wholesale would have reinstated the phase number that
phase-8 removed.
These comments were indexed by task, use-case and limitation numbers that
are defined only in planning documents outside the shipped tree. Nothing
in the repository defined them, so the cross-references resolved nowhere.
Each comment now states what the code does.
Two comment-only notes.
MetricsRegistry.cpp: record why the app and overlay includes exist and
what they cost. They are what makes loops.txt carry
xrpld.app <-> xrpld.telemetry and xrpld.overlay <-> xrpld.telemetry, where
ordering.txt previously had telemetry strictly below both. The observable
gauges are pull-model, so their callbacks need the concrete types to
sample live state. The cycle is confined to this translation unit: no
telemetry header includes app or overlay, and all of src/xrpld builds
into one target, so there is no header or link cycle. Inverting it needs
a metrics-source interface below overlay, which is left as follow-up.
Also note loops.txt is generated and must never be hand-edited.
docker-compose.yml: state the Compose >= 2.24.0 floor. The grafana
service uses the long-form env_file mapping, which older Compose cannot
parse, and it fails for the whole file rather than that one service. The
long form is required because .env.alerting is gitignored and absent in a
fresh clone, and the short form treats a missing env file as an error.
The test carried both <xrpl/proto/xrpl.pb.h> and the bare <xrpl.pb.h>.
Both resolve to the same generated header, because the proto helper puts
the generated tree and its prefixed subdirectory on the target, so the
second include expanded to nothing behind the header guard.
The bare form arrived from merging two same-day clang-tidy commits that
added the include with different spellings. Keep the prefixed spelling,
which is what the telemetry headers and the upstream phase branches use.
