Add two Peer Quality panels reading peer_disconnect_total: Peer Disconnect
Rate, the per-second teardown rate per node, and Peer Disconnects By Reason
& Direction, the per-interval increase split by cause and by which side
opened the connection. Both sit in the existing Disconnects & Connection Mix
row beside Resource Disconnects, which counts only the resource-charge
subset and carries no reason label.
The Ledger Sync Health board already shows the same split as a window
total, so it says how much of each reason but not when. These give the
time-shaped view, letting a reason spike be lined up against a stall.
Add disconnect_reason and disconnect_direction template variables for the
two new label dimensions and wire both queries to them, so the panels
filter on every dimension their series carry.
Update the 09 reference panel column and the _a7_note panel list to name
the panels that now render this counter.
The Peer Disconnect Rate By Reason panel anchored its LogQL capture on
"\] ", which only matches a reason logged immediately after the [NNN]
peer-id prefix. PeerImp does not log that way: PeerImp::fail emits
"[NNN] <name> failed: <reason>" and the clean teardown emits
"close: Closed". Only ConnectAttempt::fail, which logs the bare reason,
ever matched. The panel's Timeout series was therefore connect-attempt
timeouts only, Ping Timeout was invisible, and PeerImp's own Closed was
uncounted.
Match all three prefixes and separate Ping Timeout from Connect Timeout.
Recorded as LogQL trap 11 in the runbook, alongside the other silent
failures this dashboard exposed.
Also document six overlay observability gaps found while auditing what
ping/pong and gossip traffic is actually tracked. All are pre-existing
and none is fixed here: the code fixes belong in develop-owned overlay
files (TrafficCount, OverlayImpl, PeerImp, PeerfinderManager), not on a
telemetry branch, and one of them needs a public signature change.
- 09 section 6: six known issues, each marked NOT IMPLEMENTED with
file:line evidence -- mtCLUSTER counted as unknown (overhead_cluster_*
always zero, 8 panels flatline), squelch_ignored byte counts always
zero, inbound/outbound byte-basis asymmetry plus a stale Total header
comment, ping/endpoints instrumentation absent, peer span coverage,
and PeerFinder exporting 2 of ~17 available readings.
- 02 section 2.3.2: add a Status column to the span catalog. Of 36
catalogued spans, 16 are live, 15 were never built, and 5 shipped
under different names (consensus.phase.establish became
consensus.establish, ledger.close became consensus.ledger_close,
rpc.request split into rpc.http_request/rpc.ws_message, txq.apply
became txq.apply_direct/txq.accept_tx). The catalog is a design
inventory; 09 section 1.1 remains authoritative for what emits.
- Phase9_taskList: tasks 9.14-9.17 tracking the deferred work, with
exit criteria checked only for what is actually done.
- Glossary: new Ping / pong keepalive term distinguishing ping timeout
from connect timeout. Correct the Cluster and Squelch entries, which
described behaviour the metrics cannot show.
The glossary header pointed at tasks/telemetry_terms.py as its
generator. That file is in no branch and nowhere on disk -- tasks/ is
gitignored one directory up -- so the header now states the file is
hand-maintained and gives the entry format.
Gates: check_otel_naming.py passes all 9 rules (Rule D over 555
dashboard queries, Rule E over the runbook); 19 doc anchors verified;
dashboard JSON valid with a one-line diff. No C++ changes.
Bumps the Loki pin this branch introduced, and updates the two docs that
name the version so they do not drift from the compose file:
- OpenTelemetryPlan/09-data-collection-reference.md (log backend section)
- OpenTelemetryPlan/Phase8_taskList.md (compose snippet)
The service runs the config bundled in the image rather than one from this
repo; `-verify-config` reports "config is valid" under 3.7.6, and native OTLP
ingestion at /otlp is unchanged across 3.4 -> 3.7.
The heading still read "Future:" while the section body documents the
validation suite this branch ships and gives commands to run it. Section 6.8.3
in 06-implementation-phases.md already drops its "Future Enhancement" marker
here; this makes 5c consistent with it.
The Phase 7 section existed twice in this file; the copies disagreed on this
one label. Consolidating on the single copy kept the older wording, but Jaeger
was removed from the project earlier in this chain, so Tempo is the only trace
backend the diagram should name.
These changes were developed on the phase-10 branch but belong to content this
branch and its upstreams introduced. Carrying them on phase-10 made its PR diff
report churn in files phase-10 does not own, and left each PR claiming a scope
that did not match its contents.
Moved here from phase-10 (identical content, no functional change):
- Dashboards: all 14 existing boards plus the new log-derived-insights board.
- Docs: telemetry-runbook.md (minus the workload/benchmark sections, which
describe phase-10 tooling) and the new telemetry-glossary.md.
- Grafana Cloud + Alloy export path: collector config, compose override, the
two .env examples and alloy/config.alloy.
- Local stack: otel-collector-config.yaml gains sub-millisecond and
second-scale spanmetrics buckets, pins unit=ms, and promotes
close_time_correct; integration-test.sh and TESTING.md follow.
- Node configs: exported_instance -> service_instance_id in comments; the
mainnet sample now logs at warning to bound log volume.
- Metrics code: Telemetry.cpp builds the metrics pipeline in the constructor
via initMetrics() so the global MeterProvider is published before any
subsystem creates a beast::insight instrument, and the histogram view keeps
each instrument's own name instead of collapsing them under one series.
MetricsRegistry gains a last_close_time gauge and skips negative job-queue
durations. OTelCollector drops an unused accessor.
- Naming CI: xrpl_work_item joins EXTERNAL_INFRA_LABELS and Rule E accepts the
dotted perf-iac resource-attribute form. This must travel with the
dashboards and runbook that reference those labels, or the rules fail.
- Doxygen input glob no longer recurses dot-directories.
Sections describing phase-10 tooling stay on phase-10 and keep their
"Future Enhancement" / "Planned, not yet implemented" markers here; phase-10
removes those markers when it lands the tooling.
Brings the telemetry startup-ordering fix forward. Two conflicts, both in
the metrics registry.
initSyncInstruments(): this branch had already replaced the registry-owned
state_changes_total with a call-site macro carrying {from,to} labels, and
the arriving branch moves the jq_trans_overflow_total observable out of
this function into registerJqTransOverflowCounter() so it is armed with
the other pull-model instruments. Kept both: the explanatory comment for
state_changes_total stays, the inline overflow block goes. The arriving
stateChangesCounter_ creation is dropped rather than merged -- this branch
removed that member, so keeping the line would not compile.
Test file: kept this branch's fuller header documentation, which records
what the disabled build can and cannot assert for the sync-diagnostics
gauges, and folded in the arriving branch's one new fact, that the
lifecycle is now two-phase. Include lists unioned.
Resolves the telemetry-startup conflict between the two branches. Both
sides move the telemetry start earlier in setup(); they disagree only on
how far the pipeline had been split at that point.
phase-1b (arriving) moved nodeIdentity_, setServiceInstanceId() and the
telemetry start up to just after the wallet DB is proven usable. phase-9
had split the metrics pipeline in two and left its copy of that block at
the old, later position.
Kept both intentions: the block stays at phase-1b's early position, and
metricsRegistry_ construction moves up with it so it precedes
startTelemetry() -- the metrics half is guarded on the registry existing,
so leaving the construction behind would have started tracing while
silently skipping metrics. phase-9's later copy is dropped as the stale
duplicate. The two-phase split is preserved: startTelemetryGauges() still
runs after overlay_ is constructed, because the observable callbacks read
it and getOverlay() asserts.
Net effect is that the metrics provider now starts earlier than on either
branch, and still before beginConsensus() emits the first spans and the
only operating-mode transition.
Telemetry must read state, never change it. Two defects here did change it,
plus three smaller correctness and privacy fixes.
doPathFind and doRipplePathFind read source_account / destination_account off
context.params to hash them into span attributes. context.params is non-const,
so those reads selected json::Value's non-const operator[], which inserts a
null for a missing key. The same object is later validated by
PathRequest::parseJson, whose first checks are isMember(source_account) and
isMember(destination_account) — so a request that omitted either field looked
present and the client received Malformed instead of Missing. Reads now go
through std::as_const, whose overload returns kNull without inserting.
PathRequest::doUpdate emitted pathfind_dest_currency as
to_string(saDstAmount_.asset()). For a non-XRP asset that renders as
"<issuer>/<currency>" with the issuer as a plaintext Base58 address, so a
plain account address reached the span pipeline even though every other
account here is hashed first. The issuer is now redacted and the currency
kept; an MPT asset renders as its issuance ID and carries no address.
PathRequestManager::updateAll created pathfind.update_all with an unscoped
SpanGuard. An unscoped guard takes the ambient span as its own parent but does
not itself become the ambient parent, so the pathfind.compute spans that
doUpdate creates never nested under it, contradicting the documented hierarchy.
It is now a scoped guard, held in std::optional because ScopedSpanGuard is
deliberately non-movable and so cannot be produced by a ternary. The skip when
there are no active subscriptions is preserved. updateAll is dispatched via
addJob and doUpdate runs synchronously, so the guard is constructed and
destroyed under the same context store, as ScopedSpanGuard requires.
The WebSocket entry point emitted the client-supplied command string directly.
That value becomes a Prometheus label, so arbitrary request input could drive
unbounded label cardinality. It is now resolved against the handler registry,
collapsing anything unrecognized to "unknown", matching what the HTTP path
already does.
Also: the pathfind.discover comment claimed future child spans could be
parented off it, which its unscoped guard cannot do — corrected to say what
would be required instead. Config-reference and task-list docs named the
parser setupTelemetry(); the API is makeTelemetrySetup().
Conflict in src/libxrpl/telemetry/SpanGuard.cpp: both sides added a different
include in the same alphabetical slot — phase-4 added <initializer_list>, the
incoming std::format change added <format>. Both are required, so both are
kept in sorted order.
Conflict in OpenTelemetryPlan/03-implementation-strategy.md §3.9: both
branches independently fixed the same defect (stale hand-maintained line
counts) in different ways.
- 1b (0ff947454c) stripped the Lines Added / Lines Changed columns but kept
the §3.9.1 and §3.9.2 tables.
- 1a (3ad525a48a) removed both sections outright.
Resolved in favour of 1a's deletion. The tables were pre-implementation
estimates with no source of truth, so trimming the columns only defers the
next drift; the file/component lists duplicated §3.1's directory tree, which
1b already keeps current (DiscardFlag.h, FilteringSpanProcessor). Keeping the
upstream resolution also stops the same conflict recurring on 1c..10.
1b-specific content verified intact after the merge: §3.1 retains the
DiscardFlag.h entry, the FilteringSpanProcessor and discard() annotations,
and the TracingInstrumentation removal. No references to the deleted
sections remain in any plan doc.
