Commit Graph

679 Commits

Author SHA1 Message Date
Pratik Mankawde
964b0d64bf docs(telemetry): drop the example ticket id from the work-item filter
The $xrpl_work_item variable's description carried a real ticket id as its
example. The filter needs no example, so the id is gone and the wording stays.
2026-09-23 12:00:37 +01:00
Pratik Mankawde
aeac95b1d7 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-09-22 21:17:58 +01:00
Pratik Mankawde
fcdc4f9f66 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-22 21:17:58 +01:00
Pratik Mankawde
e48b0286c2 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
StatsDCollector test kept both sides: this branch's onCollectionReady() call,
which enables polling and only exists from here on, followed by the upstream
branch's control assertion that reads the resulting datagram.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:17:48 +01:00
Pratik Mankawde
e9d9cffe3b Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd
Both doc indexes kept this branch's 09-data-collection-reference.md rows,
which only exist here, and dropped every secure-OTel.md reference because
the upstream branch removed that file. No dangling link remains.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:16:44 +01:00
Pratik Mankawde
aea4f56505 Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing
SpanGuardScope.cpp kept both includes: each side added one and both symbols
are used in the merged test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:15:05 +01:00
Pratik Mankawde
bba92767a6 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing
tempo.yaml kept both sides' filter blocks: phase-2's six path-finding
filters ahead of this branch's three transaction filters, matching chain
order, and both header comment lines.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:14:20 +01:00
Pratik Mankawde
3cfe8d139e feat(telemetry): Add Tempo search filters for the path-finding attributes
This branch emits nine pathfind attributes and the datasource offered a dropdown
for none of them, so the signal was there but not searchable in Explore. The
file's own header states that each phase adds filters for what it introduces.

Six filters, for the attributes that select a request. The three counts are
measurements read off a span rather than things an operator searches by, so they
get none. ledger_index is dynamic because it takes a new value every ledger.
2026-09-22 20:31:14 +01:00
Pratik Mankawde
c3673f51cd fix(telemetry): Read RPC success from the absence of an error, not from Ok
The Success series selected status_code="STATUS_CODE_OK". Successful
rpc.command spans no longer carry Ok, because the specification reserves it for
an operator asserting verified success and warns a tool may read it as
suppressing errors, so instrumentation leaves a successful span Unset.

Selecting on the absence of an error keeps the panel correct either way, rather
than pinning it to whichever status a successful span happens to have.
2026-09-22 20:15:18 +01:00
Pratik Mankawde
35c7f9be4c docs(telemetry): use a placeholder zone in the cloud env example
The example endpoint named a concrete zone, which reads as the stack this repo
pushes to. The two sibling files already write <zone>, so match them.
2026-09-22 19:09:16 +01:00
Pratik Mankawde
2d3eb7a981 fix(telemetry): Stop unl_expiry_days wrapping, and count only unfinished sweep evictions
unl_expiry_days subtracted two NetClock time points, whose rep is uint32_t, so
the subtraction wrapped before the duration_cast ran. A list expired by one day
read about +49709 days. The panel is green above 30 while its own description
promises red at expiry, so an expired validator list rendered healthy.

daysUntil() widens both endpoints to int64_t first, which makes the wrap
impossible rather than checked for. It deliberately does not clamp at zero: a
negative reading is the signal that expiry has passed. A config-listed list,
which uses time_point::max(), now reports positive infinity, because any finite
sentinel could not be told apart from the wrap this removes. -1 keeps its
existing meaning of no published list fetched.

The sweep counter told a second story it could not support. It counted every
entry the 1-minute sweep evicted, including acquisitions that had already
completed or failed and were merely still in the map. Those were counted when
they ended, so the metric buried the wasteful case in ordinary cleanup while
the runbook, the reference doc and the panel description all described only the
unfinished population. It now counts what those three already claimed.

isComplete()/isFailed() are used rather than isDone(), which is protected on
TimeoutCounter and not callable here.
2026-09-22 19:07:41 +01:00
Pratik Mankawde
e53569e696 fix(telemetry): Split the job queue p99 bargauge into wait and exec
The panel fused queue-wait and execution p99 into a single bargauge, so
every job type drew two bars and neither measure could be read on its
own. It is now two panels side by side, one measure each, with its own
title and description.

The panel also ran an instant query over a fixed 5 minute rate window.
An instant query always evaluates at "now", so the dashboard time picker
had no effect and a network with no samples in the last few minutes
rendered empty at every range. The window is now $__range, which ties it
to the picker.

Repeat direction is vertical. Horizontal repeat with maxPerRow set makes
Grafana widen each copy to the full row, which would break the pair.
2026-09-22 15:42:21 +01:00
Pratik Mankawde
e94e37b7c2 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-09-22 15:14:02 +01:00
Pratik Mankawde
87fad43d9a Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation
Conflict in docker/telemetry/docker-compose.yml, on the collector service's
command key. This branch layers a second --config for file_log offset
persistence; upstream added a comment above the ports block explaining the
loopback binding.

Kept both. The two changes are unrelated: one is the command list, the other is
a comment. The layered --config and all nine loopback-bound ports are present,
and the file still parses.
2026-09-22 15:13:54 +01:00
Pratik Mankawde
4308f967f3 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
Conflict in docker/telemetry/docker-compose.yml, in the collector's ports
block. Upstream bound every port to the host loopback, including the StatsD
UDP port. This branch had already retired that port, because beast::insight
moved to native OTLP, leaving it commented out as a fallback example.

Not resolved as a union, which would have resurrected the retired port.
Composed instead: this branch's shape is kept, so StatsD stays commented, and
the loopback binding is applied to the active ports and to the commented
example, so uncommenting it yields the safe form.
2026-09-22 15:13:14 +01:00
Pratik Mankawde
18df80c5ec Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd
Conflict in docker/telemetry/docker-compose.yml, in the collector's ports
block. Both sides bound ports to the host loopback: upstream did 4317, 4318
and 13133, this branch did 8125, 8889 and 9090.

Resolved as the union. All eight published ports are loopback-bound, and both
explanatory comments are kept: upstream's, which covers the whole block and
cites the upstream guidance, and this branch's, which is specific to the
unauthenticated StatsD receiver.
2026-09-22 15:12:32 +01:00
Pratik Mankawde
6ecd5f24a9 Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-09-22 15:11:47 +01:00
Pratik Mankawde
0e1e335f42 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing 2026-09-22 15:11:47 +01:00
Pratik Mankawde
5a148ff55e Merge branch 'pratik/otel-phase1c-rpc-integration' into pratik/otel-phase2-rpc-tracing 2026-09-22 15:11:47 +01:00
Pratik Mankawde
4bf76ce2ff metrics_flush_interval = "15s"
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-09-22 15:11:28 +01:00
Pratik Mankawde
6ed6189ba9 docs(telemetry): re-apply the comment rewrite lost to a concurrent commit
Commit 54809960ff removed the site measurements from the three collector
comment blocks, but a later commit on this branch was built from an index
holding the pre-rewrite blobs, so the tree reverted while the commit stayed in
the log. History is append-only, so this re-applies the same three blocks.

No non-comment line changes. The bucket lists are untouched and still agree
across the two collector configs and the Alloy config.
2026-09-22 15:07:10 +01:00
Pratik Mankawde
d3f9fa1f67 fix(telemetry): make the job latency panel readable at any time range
The gauge showed a p99 pooled across every job type, so a slow rare job could
not surface: sweep sat at 5 s while the dial read 20 ms, because a
count-weighted p99 follows the frequent jobs and those are fast.

It is now a bar gauge over per-job-type p99, which is what the panel is for,
and a bar gauge holds tens of values where a dial holds one. The series keep
job_type so each bar names its job.

max is set to the red threshold, so every bar shares one 0 to 1 s scale
instead of each dial auto-scaling to its own data.

The targets are instant queries. Left as range queries the bar count followed
the dashboard's time range rather than the current series count, reaching 531
series over a week.
2026-09-22 15:04:40 +01:00
Pratik Mankawde
54809960ff docs(telemetry): remove site measurements from the collector comments
The repo is public, so a comment must not carry one-site measurements or
claim a prior state of the source. Three comments did both.

The tail-sampling comment called itself "Box-only (uncommitted)", which names
our deployment and was untrue. It now states what the sampling does, and adds
the consequence that matters: Log.cpp writes a trace id only for a sampled
span, so sampling after that decision leaves log-to-trace links resolving to
nothing.

The histogram bucket comments justified each edge with measured percentages
and durations, and said what the buckets "used to" do. The mechanism is
unchanged and now stated without the figures. Also drops a duplicated line in
the Alloy copy. The bucket lists themselves are untouched and still agree
across all three files.
2026-09-22 15:00:30 +01:00
Pratik Mankawde
a1b3b4486a docs(telemetry): stop tracking the validator list files
The two xrpld-telemetry configs name a file in [validators_file], and xrpld
exits at startup when that file is missing. Neither list ships now: each
developer creates their own from cfg/validators-example.txt.

The runbook says so in "Run against a live network", ahead of the run command.
2026-09-22 14:56:57 +01:00
Pratik Mankawde
46ca2daf2f Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-22 14:07:04 +01:00
Pratik Mankawde
318fe0d10d Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-09-22 14:07:04 +01:00
Pratik Mankawde
2de992f334 fix(telemetry): give Ledger Data & Sync an auto-refresh interval
The dashboard had no top-level refresh key at all, so auto-refresh was off
while every other dashboard refreshes at 30s.
2026-09-22 14:07:02 +01:00
Pratik Mankawde
07ec48d717 fix(telemetry): point template variables at the datasource variable
Sixteen template variables named a datasource by the literal uid "prometheus"
or "loki". Neither exists on the Grafana instance, so those variables resolved
to nothing and their pickers offered only All.

That also broke the panels. With an empty variable the selector becomes
job_type=~"" rather than job_type=~".*", which matches no series, so the
Current Job Latency gauge had no data and its renderer raised the plugin error
banner on every refresh.

Every working dashboard already references ${DS_PROMETHEUS} or ${DS_LOKI},
which the dashboard declares in templating.list. These now match.

job-queue 6, ledger-data-sync 5, log-derived-insights 3, overlay-traffic-detail 2.
2026-09-22 14:06:49 +01:00
Pratik Mankawde
850ed7ec2e fix(telemetry): slow the Job Queue dashboard refresh to 30s
Every other dashboard refreshes at 30s. This one was at 5s, which re-ran all
of its queries twelve times a minute for no benefit.
2026-09-22 14:06:47 +01:00
Pratik Mankawde
0dccf29b64 fix(telemetry): publish the Loki port on the host loopback
Grafana reaches Loki as loki:3100 over the compose network and the collector
exports to it in-network, so the published port was reachable off-host for no
consumer.
2026-09-22 13:58:20 +01:00
Pratik Mankawde
3cc9f33041 fix(telemetry): publish the Prometheus ports on the host loopback
The StatsD port beside them was already loopback-bound with a comment saying
why; the Prometheus exporter and Prometheus itself were not. Prometheus
scrapes the collector as otel-collector:8889 over the compose network, so the
published port had no consumer.
2026-09-22 13:58:18 +01:00
Pratik Mankawde
3f510ecf44 fix(telemetry): publish local stack ports on the host loopback
The compose ports published on every host interface, so the collector's
unauthenticated OTLP receivers, Tempo, and the anonymous-admin Grafana were
reachable by anything that could route to the host.

Nothing consumed the published ports from off-host: containers address each
other by compose service name, and every doc and script uses localhost.
2026-09-22 13:58:17 +01:00
Pratik Mankawde
a42f68cf89 merge: bring the count-unit changes forward from phase-8
Resolved nine dashboards. This branch re-exported all of them, so the upstream
unit changes land in lines this branch had already rewritten. Took this
branch's files from the clean baseline merge, then applied si: and decimals 0
to every count-unit panel left in them. Every panel set matches that baseline
merge exactly.

The three si:drops panels keep no decimals setting, as before.
2026-09-21 19:09:03 +01:00
Pratik Mankawde
fb92c78c22 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-21 19:08:04 +01:00
Pratik Mankawde
826802a35f merge: bring the count-unit changes forward from phase-6
Resolved consensus-health.json. This branch re-exported that dashboard, so
phase-6's unit changes land in lines this branch had already rewritten. Took
this branch's file from the clean baseline merge, then applied si: and
decimals 0 to every count-unit panel left in it. The panel set matches that
baseline merge exactly.
2026-09-21 19:07:53 +01:00
Pratik Mankawde
7d38d1bff8 feat(telemetry): scale count units and drop meaningless decimals
Panels whose unit counts whole things switch from a custom `suffix:` unit to
`si:`, so a large value reads 15 k nodes/s instead of 15000 nodes/s. A custom
suffix unit is formatted by toFixedUnit, which never scales; si: goes through
SIPrefix and does.

decimals is set to 0 on the same panels, because a fraction of a ledger or a
node says nothing. The two settings need each other: on its own decimals 0
would round a slow rate down to zero, and si: shifts it to the milli step
instead.

Left alone: the one panel whose unit is a seconds-per-second ratio.

42 panels across 10 dashboards, plus 5 name placements and one title.
2026-09-21 19:06:05 +01:00
Pratik Mankawde
5c8495e230 feat(telemetry): scale count units and drop meaningless decimals
Panels whose unit counts whole things switch from a custom `suffix:` unit to
`si:`, so a large value reads 15 k nodes/s instead of 15000 nodes/s. A custom
suffix unit is formatted by toFixedUnit, which never scales; si: goes through
SIPrefix and does.

decimals is set to 0 on the same panels, because a fraction of a ledger or a
node says nothing. The two settings need each other: on its own decimals 0
would round a slow rate down to zero, and si: shifts it to the milli step
instead.

Left alone: the one panel whose unit is a seconds-per-second ratio.

26 panels across 9 dashboards, plus 3 legends and one name placement.
2026-09-21 19:06:03 +01:00
Pratik Mankawde
202fefe896 feat(telemetry): scale count units and drop meaningless decimals
Panels whose unit counts whole things switch from a custom `suffix:` unit to
`si:`, so a large value reads 15 k nodes/s instead of 15000 nodes/s. A custom
suffix unit is formatted by toFixedUnit, which never scales; si: goes through
SIPrefix and does.

decimals is set to 0 on the same panels, because a fraction of a ledger or a
node says nothing. The two settings need each other: on its own decimals 0
would round a slow rate down to zero, and si: shifts it to the milli step
instead.

Left alone: the one panel whose unit is a seconds-per-second ratio.

20 panels across 5 dashboards.
2026-09-21 19:06:01 +01:00
Pratik Mankawde
b69c7631de merge: bring the legend and bar gauge changes forward from phase-8
Resolved network-traffic.json. Both branches own a version of the Peer
Disconnects legend: phase-7 had the older "[instance branch role work_item]"
identity form, and this branch rewrote the same line to use xrpl_ident. Kept
this branch's line, then applied the same change to it, so the legend drops
the series token and the query drops the label_replace that set it.

The incoming side's only edit to that file was those two lines, so nothing
else was superseded.
2026-09-21 18:10:35 +01:00
Pratik Mankawde
b2aa831621 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-21 18:08:26 +01:00
Pratik Mankawde
3036ada2e7 feat(telemetry): tidy dashboard legends and bar gauge styling
A panel legend drops ${__field.labels.series} where the panel title already
carries that name, and the query drops the label_replace that set the label,
since nothing reads it any more.

Horizontal bar gauges put the series name above the bar.

40 legends across 5 dashboards, 2 bar gauges to the LCD style, 5 name placements.
2026-09-21 18:08:16 +01:00
Pratik Mankawde
6b8462cddf feat(telemetry): tidy dashboard legends and bar gauge styling
A panel legend drops ${__field.labels.series} where the panel title already
carries that name, and the query drops the label_replace that set the label,
since nothing reads it any more.

Horizontal bar gauges put the series name above the bar.

3 legends across 2 dashboards, and one bar gauge name placement.
2026-09-21 18:08:14 +01:00
Pratik Mankawde
81a2406092 merge: bring the dashboard annotation settings forward from phase-8
Resolved four dashboards. This branch re-exported consensus-health, node-health,
rpc-performance and transaction-overview, so both sides touched the same option
blocks: upstream added the annotations block where this branch's export already
carried it set to false. Rebuilt each file from the clean pre-change merge of the
two branches, then set the option on every panel. Panel sets match that baseline
merge exactly, so the panels this branch's export drops are unchanged by the
resolution.

All 189 timeseries and state-timeline panels on this branch now set multiLane,
and all four state timelines set perPage.
2026-09-21 17:07:06 +01:00
Pratik Mankawde
4cbb2199e5 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-21 17:06:18 +01:00
Pratik Mankawde
94bb5313a2 merge: bring the dashboard annotation settings forward from phase-6
Resolved three dashboards. phase-7 rewrote consensus-health, rpc-performance
and transaction-overview, so both sides added the annotations block next to
lines that had already diverged. Rebuilt each file from the clean pre-change
merge of the two branches, then set the option on every panel, which is what
both sides intended. Panel sets match that baseline merge exactly.

All 107 timeseries and state-timeline panels on this branch now set multiLane,
and both state timelines set perPage.
2026-09-21 17:06:07 +01:00
Pratik Mankawde
0e4db42a34 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-21 17:04:05 +01:00
Pratik Mankawde
73eb8336cc feat(telemetry): add state-timeline pagination and multi-row annotations
Every timeseries and state-timeline panel sets
options.annotations.multiLane. The state-timeline panels set
options.perPage to 5.

99 panels across 13 dashboards. Seven of those already carried the option from
this branch's node-health export, so they flip here and not upstream.
2026-09-21 17:03:53 +01:00
Pratik Mankawde
18e7333bbe feat(telemetry): add state-timeline pagination and multi-row annotations
Every timeseries and state-timeline panel sets
options.annotations.multiLane. The state-timeline panels set
options.perPage to 5.

62 panels across 5 dashboards.
2026-09-21 17:00:54 +01:00
Pratik Mankawde
5b4f328489 feat(telemetry): add state-timeline pagination and multi-row annotations
Every timeseries and state-timeline panel sets
options.annotations.multiLane. The state-timeline panels set
options.perPage to 5.

46 panels across 5 dashboards.
2026-09-21 17:00:52 +01:00
Pratik Mankawde
3b60a190de fix(telemetry): drop the remaining suppressed dimensions and diagram values
Alloy and the Grafana Cloud collector still declared a spanmetrics dimension
for an attribute tx.receive no longer sets, which only widens the label set.
The runbook's drop-path diagram still listed two tx_status values that no
longer exist, because the paths that set them now run before the span starts.
2026-09-18 09:32:29 +01:00