unl_expiry_days subtracted two NetClock time points, whose rep is uint32_t, so
the subtraction wrapped before the duration_cast ran. A list expired by one day
read about +49709 days. The panel is green above 30 while its own description
promises red at expiry, so an expired validator list rendered healthy.
daysUntil() widens both endpoints to int64_t first, which makes the wrap
impossible rather than checked for. It deliberately does not clamp at zero: a
negative reading is the signal that expiry has passed. A config-listed list,
which uses time_point::max(), now reports positive infinity, because any finite
sentinel could not be told apart from the wrap this removes. -1 keeps its
existing meaning of no published list fetched.
The sweep counter told a second story it could not support. It counted every
entry the 1-minute sweep evicted, including acquisitions that had already
completed or failed and were merely still in the map. Those were counted when
they ended, so the metric buried the wasteful case in ordinary cleanup while
the runbook, the reference doc and the panel description all described only the
unfinished population. It now counts what those three already claimed.
isComplete()/isFailed() are used rather than isDone(), which is protected on
TimeoutCounter and not callable here.
The panel fused queue-wait and execution p99 into a single bargauge, so
every job type drew two bars and neither measure could be read on its
own. It is now two panels side by side, one measure each, with its own
title and description.
The panel also ran an instant query over a fixed 5 minute rate window.
An instant query always evaluates at "now", so the dashboard time picker
had no effect and a network with no samples in the last few minutes
rendered empty at every range. The window is now $__range, which ties it
to the picker.
Repeat direction is vertical. Horizontal repeat with maxPerRow set makes
Grafana widen each copy to the full row, which would break the pair.
Conflict in docker/telemetry/docker-compose.yml, on the collector service's
command key. This branch layers a second --config for file_log offset
persistence; upstream added a comment above the ports block explaining the
loopback binding.
Kept both. The two changes are unrelated: one is the command list, the other is
a comment. The layered --config and all nine loopback-bound ports are present,
and the file still parses.
Conflict in docker/telemetry/docker-compose.yml, in the collector's ports
block. Upstream bound every port to the host loopback, including the StatsD
UDP port. This branch had already retired that port, because beast::insight
moved to native OTLP, leaving it commented out as a fallback example.
Not resolved as a union, which would have resurrected the retired port.
Composed instead: this branch's shape is kept, so StatsD stays commented, and
the loopback binding is applied to the active ports and to the commented
example, so uncommenting it yields the safe form.
Conflict in docker/telemetry/docker-compose.yml, in the collector's ports
block. Both sides bound ports to the host loopback: upstream did 4317, 4318
and 13133, this branch did 8125, 8889 and 9090.
Resolved as the union. All eight published ports are loopback-bound, and both
explanatory comments are kept: upstream's, which covers the whole block and
cites the upstream guidance, and this branch's, which is specific to the
unauthenticated StatsD receiver.
Commit 54809960ff removed the site measurements from the three collector
comment blocks, but a later commit on this branch was built from an index
holding the pre-rewrite blobs, so the tree reverted while the commit stayed in
the log. History is append-only, so this re-applies the same three blocks.
No non-comment line changes. The bucket lists are untouched and still agree
across the two collector configs and the Alloy config.
The gauge showed a p99 pooled across every job type, so a slow rare job could
not surface: sweep sat at 5 s while the dial read 20 ms, because a
count-weighted p99 follows the frequent jobs and those are fast.
It is now a bar gauge over per-job-type p99, which is what the panel is for,
and a bar gauge holds tens of values where a dial holds one. The series keep
job_type so each bar names its job.
max is set to the red threshold, so every bar shares one 0 to 1 s scale
instead of each dial auto-scaling to its own data.
The targets are instant queries. Left as range queries the bar count followed
the dashboard's time range rather than the current series count, reaching 531
series over a week.
The repo is public, so a comment must not carry one-site measurements or
claim a prior state of the source. Three comments did both.
The tail-sampling comment called itself "Box-only (uncommitted)", which names
our deployment and was untrue. It now states what the sampling does, and adds
the consequence that matters: Log.cpp writes a trace id only for a sampled
span, so sampling after that decision leaves log-to-trace links resolving to
nothing.
The histogram bucket comments justified each edge with measured percentages
and durations, and said what the buckets "used to" do. The mechanism is
unchanged and now stated without the figures. Also drops a duplicated line in
the Alloy copy. The bucket lists themselves are untouched and still agree
across all three files.
The two xrpld-telemetry configs name a file in [validators_file], and xrpld
exits at startup when that file is missing. Neither list ships now: each
developer creates their own from cfg/validators-example.txt.
The runbook says so in "Run against a live network", ahead of the run command.
Sixteen template variables named a datasource by the literal uid "prometheus"
or "loki". Neither exists on the Grafana instance, so those variables resolved
to nothing and their pickers offered only All.
That also broke the panels. With an empty variable the selector becomes
job_type=~"" rather than job_type=~".*", which matches no series, so the
Current Job Latency gauge had no data and its renderer raised the plugin error
banner on every refresh.
Every working dashboard already references ${DS_PROMETHEUS} or ${DS_LOKI},
which the dashboard declares in templating.list. These now match.
job-queue 6, ledger-data-sync 5, log-derived-insights 3, overlay-traffic-detail 2.
Grafana reaches Loki as loki:3100 over the compose network and the collector
exports to it in-network, so the published port was reachable off-host for no
consumer.
The StatsD port beside them was already loopback-bound with a comment saying
why; the Prometheus exporter and Prometheus itself were not. Prometheus
scrapes the collector as otel-collector:8889 over the compose network, so the
published port had no consumer.
The compose ports published on every host interface, so the collector's
unauthenticated OTLP receivers, Tempo, and the anonymous-admin Grafana were
reachable by anything that could route to the host.
Nothing consumed the published ports from off-host: containers address each
other by compose service name, and every doc and script uses localhost.
Resolved nine dashboards. This branch re-exported all of them, so the upstream
unit changes land in lines this branch had already rewritten. Took this
branch's files from the clean baseline merge, then applied si: and decimals 0
to every count-unit panel left in them. Every panel set matches that baseline
merge exactly.
The three si:drops panels keep no decimals setting, as before.
Resolved consensus-health.json. This branch re-exported that dashboard, so
phase-6's unit changes land in lines this branch had already rewritten. Took
this branch's file from the clean baseline merge, then applied si: and
decimals 0 to every count-unit panel left in it. The panel set matches that
baseline merge exactly.
Panels whose unit counts whole things switch from a custom `suffix:` unit to
`si:`, so a large value reads 15 k nodes/s instead of 15000 nodes/s. A custom
suffix unit is formatted by toFixedUnit, which never scales; si: goes through
SIPrefix and does.
decimals is set to 0 on the same panels, because a fraction of a ledger or a
node says nothing. The two settings need each other: on its own decimals 0
would round a slow rate down to zero, and si: shifts it to the milli step
instead.
Left alone: the one panel whose unit is a seconds-per-second ratio.
42 panels across 10 dashboards, plus 5 name placements and one title.
Panels whose unit counts whole things switch from a custom `suffix:` unit to
`si:`, so a large value reads 15 k nodes/s instead of 15000 nodes/s. A custom
suffix unit is formatted by toFixedUnit, which never scales; si: goes through
SIPrefix and does.
decimals is set to 0 on the same panels, because a fraction of a ledger or a
node says nothing. The two settings need each other: on its own decimals 0
would round a slow rate down to zero, and si: shifts it to the milli step
instead.
Left alone: the one panel whose unit is a seconds-per-second ratio.
26 panels across 9 dashboards, plus 3 legends and one name placement.
Panels whose unit counts whole things switch from a custom `suffix:` unit to
`si:`, so a large value reads 15 k nodes/s instead of 15000 nodes/s. A custom
suffix unit is formatted by toFixedUnit, which never scales; si: goes through
SIPrefix and does.
decimals is set to 0 on the same panels, because a fraction of a ledger or a
node says nothing. The two settings need each other: on its own decimals 0
would round a slow rate down to zero, and si: shifts it to the milli step
instead.
Left alone: the one panel whose unit is a seconds-per-second ratio.
20 panels across 5 dashboards.
Resolved network-traffic.json. Both branches own a version of the Peer
Disconnects legend: phase-7 had the older "[instance branch role work_item]"
identity form, and this branch rewrote the same line to use xrpl_ident. Kept
this branch's line, then applied the same change to it, so the legend drops
the series token and the query drops the label_replace that set it.
The incoming side's only edit to that file was those two lines, so nothing
else was superseded.
A panel legend drops ${__field.labels.series} where the panel title already
carries that name, and the query drops the label_replace that set the label,
since nothing reads it any more.
Horizontal bar gauges put the series name above the bar.
40 legends across 5 dashboards, 2 bar gauges to the LCD style, 5 name placements.
A panel legend drops ${__field.labels.series} where the panel title already
carries that name, and the query drops the label_replace that set the label,
since nothing reads it any more.
Horizontal bar gauges put the series name above the bar.
3 legends across 2 dashboards, and one bar gauge name placement.
Resolved four dashboards. This branch re-exported consensus-health, node-health,
rpc-performance and transaction-overview, so both sides touched the same option
blocks: upstream added the annotations block where this branch's export already
carried it set to false. Rebuilt each file from the clean pre-change merge of the
two branches, then set the option on every panel. Panel sets match that baseline
merge exactly, so the panels this branch's export drops are unchanged by the
resolution.
All 189 timeseries and state-timeline panels on this branch now set multiLane,
and all four state timelines set perPage.
Resolved three dashboards. phase-7 rewrote consensus-health, rpc-performance
and transaction-overview, so both sides added the annotations block next to
lines that had already diverged. Rebuilt each file from the clean pre-change
merge of the two branches, then set the option on every panel, which is what
both sides intended. Panel sets match that baseline merge exactly.
All 107 timeseries and state-timeline panels on this branch now set multiLane,
and both state timelines set perPage.
Every timeseries and state-timeline panel sets
options.annotations.multiLane. The state-timeline panels set
options.perPage to 5.
99 panels across 13 dashboards. Seven of those already carried the option from
this branch's node-health export, so they flip here and not upstream.
Every timeseries and state-timeline panel sets
options.annotations.multiLane. The state-timeline panels set
options.perPage to 5.
62 panels across 5 dashboards.
Every timeseries and state-timeline panel sets
options.annotations.multiLane. The state-timeline panels set
options.perPage to 5.
46 panels across 5 dashboards.
Alloy and the Grafana Cloud collector still declared a spanmetrics dimension
for an attribute tx.receive no longer sets, which only widens the label set.
The runbook's drop-path diagram still listed two tx_status values that no
longer exist, because the paths that set them now run before the span starts.
transaction-overview.json conflicted only because this branch reordered every
panel, so the diff could not line the two sides up. The incoming side changed
one panel and nothing else, so this keeps every panel and rewrite this branch
made and re-applies that single change in this branch's own style: the receive
panel drops the suppressed grouping, names its series like the other
single-series panels, and says where the duplicate count now lives.
Resolved transaction-overview.json by composing both sides: this branch's
renamed span_calls_total metric and its 15s interval, with phase-6's removal of
the suppressed grouping and its new legend.
tx.receive is created after the duplicate check, so it never carries a
suppressed attribute. The Tempo tag filter and the dashboard panel still
referenced it, and check_otel_naming rules C and D fail on a reference with no
constant behind it.
The panel keeps its remaining signal as a plain receive rate. How many relayed
copies were dropped is the transactions_duplicate traffic category, which does
not depend on a span.
The spanmetrics dimension goes too; a dimension for an attribute nothing sets
only widens the label set.
MetricsRegistry did two jobs. It owned the OTel metrics pipeline, and it
registered the observable gauges whose callbacks read live application
services. The second job is what made the whole class xrpld-tier, so the
pipeline's lifecycle -- the recording() gate and the stop() teardown that
closes a use-after-free window -- could not be unit-tested in xrpl_tests.
Split it in two:
- xrpl::telemetry::MetricsRegistry (libxrpl) owns the exporter, provider,
meter, the 16 synchronous instruments, recording(), stop(), and the
record*/increment* methods.
- xrpl::telemetry::AppMetricGauges (xrpld) owns the 19 observable gauges
and their callbacks, holding a reference to the core and to the
ServiceRegistry.
MetricMacros.h and ValidationTracker move with the core. The macros need
only recording() and meter(), both core members; the core holds a tracker
by value, and a libxrpl header cannot include one from src/.
ApplicationImp owns both objects and sequences them. The core is built in
the member-init list, so every synchronous instrument exists before any
subsystem can record one. The gauges are armed once overlay_ exists, the
last service their callbacks read. Shutdown detaches the gauge callbacks
before the core drops the provider, and each shutdown step is isolated so
a failure in one cannot skip the others.
That detach call is new. detachCallbacks() had no callers, and the flag it
sets is read by the gauge callbacks but can no longer be written by the
core, so the caller now has to make the ordering explicit.
The telemetry module links xrpl.libxrpl.core and xrpl.libxrpl.protocol
PUBLIC: ValidationTracker.h takes a LedgerIndex and MetricMacros.h takes a
ServiceRegistry, both in interfaces a consumer compiles against.
Adds a MetricsRegistry gtest that drives an enabled core with telemetry on
and pins the recording() gate, stop() leaving the registry inert, and
stop() being idempotent. The libxrpl test tree no longer depends on
xrpld.telemetry at all, and the two CMake workarounds that compiled xrpld
sources into xrpl_tests are gone.
Documentation and dashboard source links follow the code to their new
paths, split between the two classes by which one now defines each metric.
The rule watched state_accounting_full_transitions > 3 per hour, so a node
that flaps once (one full -> syncing -> full round, e.g. per online-delete
rotation) never tripped it. Lower the threshold to > 0 so a single re-entry
into FULL, past the one-hour uptime gate, alerts.
Keep the state_accounting_full_transitions metric: it is a cumulative gauge
every node always reports, so increase() yields a real series (0 when
healthy) and the rule never evaluates to NoData. A sparse counter would
raise a false DatasourceNoData on a healthy node. Set noDataState: OK so a
scrape gap cannot page either.
Merges pratik/otel-phase8-log-correlation into pratik/otel-phase9-metric-gap-fill.
Auto-merged. TESTING.md picked up phase-8's Test 4 rewrite alongside this
branch's Step 2 standalone-store fix. OTelCollector.cpp gained the weak_ptr
gauges_ list; the StatsD test gained its onCollectionReady() calls.
Merges pratik/otel-phase7-native-metrics into pratik/otel-phase8-log-correlation.
Conflict was one TESTING.md hunk under "Nodes not reaching proposing state":
this branch renamed the node directories to Node-N in integration-test.sh,
phase-7 kept nodeN and expanded the [peer_private] explanation. Resolution
keeps this branch's Node-1 path (its own script uses that naming) and
phase-7's fuller prose citing peerfinder/Config.cpp.
Non-conflicting phase-7 changes come through: OTelCollector's gauges_ list
becomes weak_ptr, matching the earlier hooks_ change; the phase-6 revert of
the StatsD-test onCollectionReady() calls resolved against phase-7's version
that keeps them.
Merges pratik/otel-phase6-statsd into pratik/otel-phase7-native-metrics.
Phase-6 dropped the three onCollectionReady() calls that had been added to
its StatsD test, because that method is only declared here on phase-7.
This branch's own copy of the file was unchanged from the merge base, so
the default merge would have silently deleted the calls from here too —
where they are needed, because this branch gates polling behind
onCollectionReady() in OTelCollectorImp::onTimer.
Resolution keeps both sides: phase-6's two new include lines
(Counter.h, Gauge.h) and phase-7's three onCollectionReady() calls plus
their doxygen and inline explanations. The merged file is exactly
phase-7's tip plus those two includes.
TESTING.md auto-merged cleanly; both sides added text under Test 1 in
different regions.
Three conflicts, all from both branches editing the same passage:
- Main.cpp: kept phase-9's wording. The metrics registry only exists here, so
"unwinding destroys little: the metrics registry, whose destructor joins its
export thread" is the true statement on this branch.
- TESTING.md: kept both paragraphs. They document different things (the
private [network_id], and the log path plus log_level).
- 05-configuration-reference.md: composed both. The identity is now resolved
before construction and never empty, so every producer stamps the node key
from the start; the only divergence left is a wallet that already holds a
different key, which corrects the tracer alone. Rewrote the earlier
"three producers" blockquote too: its "no fallback", "first boot ... left
off" and "Known issue" claims are what this change removes.
One silent break the merge could not flag: makeMetricsRegistryOptions() took
the std::optional<std::string> node key that used to be a constructor
parameter, and that parameter is now the resolved keypair. It takes the base58
string directly, and the constructor derives it from nodeIdentity_, which is
declared before both telemetry_ and metricsRegistry_.
