Commit Graph

271 Commits

Author SHA1 Message Date
Pratik Mankawde
e3452f9e30 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation
# Conflicts:
#	docker/telemetry/otel-collector-config.yaml
2026-07-15 11:33:24 +01:00
Pratik Mankawde
1f43baa0bb fix(telemetry): use raw-label legend Display name; drop collector group labels
The collector-built resource_group / iac_group labels did not survive the
dashboards' own PromQL aggregation: panels that sum by(service_instance_id,
xrpl_branch, xrpl_node_role, ...) drop any label not named in the by() clause,
so the derived group labels were aggregated away and the Display name rendered
empty ("Observing" with no node). Building them in the collector also coupled
every dashboard to a lockstep collector redeploy on every node.

Switch the legend Display name to reference the raw labels the panels already
group by, which are always present by construction and need no collector
support:
  ${__field.labels.series} ${__field.labels.service_instance_id}
  ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role}
  ${__field.labels.xrpl_work_item}
${__field.labels.X} renders empty for a label a series lacks (verified in
Grafana), so dev/mainnet nodes show just "<name> <node>" and perf nodes add the
branch/role/work-item, with no empty brackets and no double spaces. Verified
live across timeseries, stat, bargauge, piechart, gauge and cross-node
aggregation panels.

Revert the transform/legendgroups processor from the local collector config;
this file is now identical to its pre-change state.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-15 11:32:09 +01:00
Pratik Mankawde
c18f9cbf58 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-14 19:08:53 +01:00
Pratik Mankawde
53289203ed fix(telemetry): move legend brackets into collector label, drop empty []
The Display name templated the brackets: "${series} [${resource_group}]
[${iac_group}]". When a group label was absent the "${}" rendered empty,
leaving a literal "[]" in the legend, and the cleanup renameByRegex could not
remove it (Grafana runs that transform on the pre-interpolation template, which
has no "[]" -- it only appears after interpolation at render time).

Bake the brackets and comma separators into the group-label VALUES in the
collector instead (resource_group="[node, mainnet]",
iac_group="[branch, role, work-item]"), so the Display name is bracket-free
("${series} ${resource_group} ${iac_group}") and an absent group renders as
nothing. Separator changes from "-" to ", ". Drop the dead cleanup transform.

Covers this branch's local collector config and its three dashboards
(ledger-data-sync, network-traffic, overlay-traffic-detail).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 19:08:25 +01:00
Pratik Mankawde
61e6d0089f Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-14 18:33:35 +01:00
Pratik Mankawde
6ebb21d8a3 fix(telemetry): keep category names on Overlay Traffic bargauge
The "Overlay Traffic by Category (Bytes In)" bargauge names each bar by
category via 22 byName field-override displayNames (Transactions, Proposals,
...). The legend migration had also added a generic
${__field.labels.series} [...] Display name, which competed with those
curated names. Remove the generic Display name (and its now-unneeded
empty-bracket cleanup transform) from this panel so the category names stand;
this panel's identity is the traffic category, not the node/perf-iac grouping.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 18:33:16 +01:00
Pratik Mankawde
4f9c507c71 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-14 18:28:59 +01:00
Pratik Mankawde
72db55233e fix(telemetry): guard legend-group build against empty-string attributes
The transform/legendgroups guards checked only != nil. The perf-iac collector
stamps xrpl.branch / xrpl.node.role / xrpl.work.item with an empty-string
default (work_item | default('')) when a run has no work item, and an empty
string is not nil. OTTL Concat renders a stray separator for an empty element,
so an all-empty perf-iac stamp would have produced iac_group="--".

Add != "" to every presence guard (iac_group sources, service.instance.id,
xrpl.network.type) so a group is built only from genuinely-present values and
left unset otherwise. Verified on a live collector: empty-string perf-iac attrs
leave iac_group absent.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 18:28:39 +01:00
Pratik Mankawde
9b9e4787db Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation
# Conflicts:
#	docker/telemetry/otel-collector-config.yaml
2026-07-14 18:04:52 +01:00
Pratik Mankawde
6911963939 fix(telemetry): grouped legend labels via collector, drop regex chain
The per-panel legend used legendFormat "__auto" reshaped by an 11-rule
renameByRegex chain. __auto sorts labels alphabetically and cannot group,
so the chain existed only to bolt grouping onto its output -- unmaintainable
and non-compliant with the dashboard legend guideline.

Build the grouped labels once at ingestion instead. A new transform/legendgroups
processor derives two resource attributes, promoted to Prometheus labels by
resource_to_telemetry_conversion:
  - resource_group: service_instance_id[-xrpl_network_type]
  - iac_group:      xrpl_branch-xrpl_node_role-xrpl_work_item (perf runs only)
Each is left unset when its sources are absent, so Prometheus drops the empty
label and the legend shows nothing for it. Concat is presence-guarded because
OTTL Concat renders "<nil>" for an absent attribute.

Dashboards now use a Standard-option Display name that references the pre-built
labels (${__field.labels.series} [resource_group] [iac_group]); ${__field.labels.X}
renders empty for a missing label, so absent groups vanish. A single generic
renameByRegex strips the resulting empty "[]" bracket. legendFormat is dropped
(Display name supersedes it). Tooltip maxHeight set to 600 per guideline.

Applied here to the three dashboards this branch introduced (ledger-data-sync,
network-traffic, overlay-traffic-detail); value-mapping renameByRegex rules are
preserved. Verified end-to-end on a live local stack (collector -> Prometheus ->
Grafana): perf and dev nodes render correct, clutter-free legends.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 18:04:03 +01:00
Pratik Mankawde
af9a2e8bea Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-14 12:18:19 +01:00
Pratik Mankawde
8c34fd9f5a fix(telemetry): conditional filter-dimension legends + 30m span-nulls
Panel legends only showed the node (service_instance_id); the perf-iac
filter dimensions (xrpl_branch, xrpl_node_role) never appeared, so
baseline-vs-test series were indistinguishable.

Each node-referencing target injects its curated name via
label_replace(...,"series",...) and uses the "__auto" legend, which shows
only labels that vary across the displayed series. A word-boundary-anchored
renameByRegex chain reshapes the auto output into
"Name [node, role]-[branch, work_item]", collapsing empty groups so a
dimension appears only when the filter widens it.

Timeseries panels connect only null gaps shorter than 30m
(spanNulls=1800000): brief scrape gaps bridge, genuine outages stay broken.

Dashboards (stable panel set): ledger-data-sync, network-traffic,
overlay-traffic-detail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 12:17:39 +01:00
Pratik Mankawde
71686b0d52 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-13 20:51:34 +01:00
Pratik Mankawde
6f9319e319 refactor(telemetry): split ledger-data-sync panels by data type
Each family panel (7-8 series) was overloaded. Split by Ledger/Transaction/
Account State so each panel shows only related series. Get+Share stay
together within a type.

- Ledger Data Exchange: 8 targets -> 3 panels
- Ledger Share/Get Traffic: 8 targets -> 3 panels
- GetObject Traffic by Type: 8 targets -> 3 panels
- GetObject Messages by Type: 7 targets -> 3 + specials panel
- GetObject Aggregate & Special: renamed, kept as one
- Overlay Traffic Heatmap: unchanged

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-13 20:29:38 +01:00
Pratik Mankawde
94a02e8183 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-13 15:45:04 +01:00
Pratik Mankawde
de579f815e fix(telemetry): dashboard panel units and filter descriptions
Set display units on native-metric dashboard panels so counts render
SI-scaled (Active Peers, Peer Disconnects → short). Polish the shared
filter descriptions: spell out the deployment-tier and network values and
add the perf network value.

Dashboards: network-traffic, ledger-data-sync, overlay-traffic-detail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 15:44:17 +01:00
Pratik Mankawde
cd091bfe0e Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-13 14:32:19 +01:00
Pratik Mankawde
db3fc94cc8 feat(telemetry): add perf-iac work-item/branch/node-role dashboard filters
Add three template variables (Work Item, Branch, Node Role) and their
label matchers to the native-metric dashboards so perf-iac comparison
runs can be filtered by ticket, comparison side (baseline/test), and node
role. The labels (xrpl_work_item, xrpl_branch, xrpl_node_role) are stamped
by perf-iac's alloy config as OTLP resource attributes and promoted to
Prometheus labels by Grafana Cloud; no repo collector change is needed.

Each variable mirrors the existing xrpl_network_type config (multi-select,
includeAll, allValue ".*") so local and non-perf runs — where these labels
are absent — continue to render (an absent PromQL label matches ".*").

Dashboards: ledger-data-sync, network-traffic, overlay-traffic-detail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 14:31:40 +01:00
Pratik Mankawde
8ca7036a8b Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-10 18:36:48 +01:00
Pratik Mankawde
5f8ca9d84c Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
# Conflicts:
#	docker/telemetry/grafana/dashboards/transaction-overview.json
#	docs/telemetry-runbook.md
2026-07-10 18:36:39 +01:00
Pratik Mankawde
24542c46c5 feat(telemetry): TxQ accept applied-ratio state timeline
Replace the TxQ Accept Status piechart on the Transaction Overview
dashboard with a state-timeline showing each node's applied fraction of
TxQ accepts (applied / applied+failed) over time, colored by threshold
(green >=0.9, yellow >=0.7, red below). Remove the now-orphaned
txq_status template variable (the piechart was its only consumer) and
document the panel in the telemetry runbook.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:33:22 +01:00
Pratik Mankawde
0c9001d058 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-10 18:25:05 +01:00
Pratik Mankawde
e357cf743d Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
# Conflicts:
#	docker/telemetry/grafana/dashboards/consensus-health.json
#	docker/telemetry/grafana/dashboards/ledger-operations.json
#	docker/telemetry/grafana/dashboards/peer-network.json
#	docker/telemetry/grafana/dashboards/rpc-performance.json
#	docker/telemetry/grafana/dashboards/transaction-overview.json
2026-07-10 18:24:53 +01:00
Pratik Mankawde
086c85fb2c fix(telemetry): seed $node dashboard filter from target_info
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:21:59 +01:00
Pratik Mankawde
a998b80f7b fix(telemetry): seed $node dashboard filter from target_info
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:21:56 +01:00
Pratik Mankawde
4f2b7cfb45 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-10 17:19:01 +01:00
Pratik Mankawde
3143216ad1 docs(telemetry): flag RPC Response Size instrument mismatch pending byte-histogram fix
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 17:00:13 +01:00
Pratik Mankawde
9820ae8e38 fix(telemetry): use $__rate_interval + min step on all rate panels
Replace hardcoded [5m] rate windows with [$__rate_interval] inside
rate()/increase() on plain-counter panels so ops/s panels resample
correctly on zoom-out instead of under-sampling or flat-lining on
short windows. Histogram _bucket windows keep their fixed [5m] span.
Set Min step ("interval": "15s") on every rate target to match the
Prometheus 15s scrape interval.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:51:53 +01:00
Pratik Mankawde
ba9ac50559 fix(telemetry): drop duplicate peer-tx panel, repoint apply-failed KPI to tx.transactor
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:22:47 +01:00
Pratik Mankawde
b0f4370397 fix(telemetry): show operating mode as % time-in-mode on phase7 node-health
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:18:06 +01:00
Pratik Mankawde
a90d51c808 fix(telemetry): render phase7 traffic dashboards as per-second rates
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:06:28 +01:00
Pratik Mankawde
982e3cf9ef Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-10 14:31:07 +01:00
Pratik Mankawde
86bbe8f55b fix(telemetry): rename spanmetrics to span_ namespace on phase7 (dashboards + collector config)
The spanmetrics connector had no namespace, so it emitted traces_span_metrics_*
metric names by default. The span dashboards and docs are renamed to query
span_* names; this is only correct if the connector emits them too, so add
namespace: "span" to the spanmetrics connector. Both sides change together:
renaming the dashboards without the namespace (or vice versa) would break the
pipeline. Matches the phase9 collector config.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 14:30:01 +01:00
Pratik Mankawde
34a237fab6 fix(telemetry): strip xrpld_ prefix and snake_case native dashboard/doc metric names
The phase7 OTelCollector::formatName lowercases and strips names, emitting
snake_case metrics with no xrpld_ prefix. The native Grafana dashboards and
the telemetry docs still queried the old xrpld_CamelCase names, so they were
broken against their own pipeline. Rename every metric name to match what the
code emits: drop the xrpld_ prefix and lowercase the remainder. The two job
histograms also drop the redundant 'duration' word (job_queued_us,
job_running_us) to match the phase9 forms. Add havetxset to the cspell
dictionary since the lowercased metric name no longer word-splits.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 14:07:29 +01:00
Pratik Mankawde
d174d4a7d8 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-10 13:40:42 +01:00
Pratik Mankawde
6ae6e623a7 test(telemetry): add dashboard lint for raw-cumulative panels
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 13:21:52 +01:00
Pratik Mankawde
6530e7d76c Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-09 17:22:12 +01:00
Pratik Mankawde
dae3f6e4d1 refactor(telemetry): default the $service_name dashboard filter to 'xrpld'
Continuation of the same default change applied on phase-6-statsd
(50d7047abe): the 5 dashboards introduced on phase-7 (formerly
system-*, renamed to bare names on this branch) also get their
service_name filter pre-selected to 'xrpld' instead of 'All'.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-09 17:22:01 +01:00
Pratik Mankawde
f2d1ac5c78 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-09 17:21:40 +01:00
Pratik Mankawde
50d7047abe refactor(telemetry): default the $service_name dashboard filter to 'xrpld'
Change the pre-selected value of the service_name template variable from
'All' to 'xrpld'. On first load, dashboards now filter to service.name=xrpld
directly rather than showing every service the collector sees. The 'All'
option is still selectable — this only changes the default the widget
starts on.

Reduces the chance of a fresh dashboard link showing unrelated services
(e.g. non-xrpld exporters sharing the same Grafana Cloud stack) before
the user narrows the filter.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-09 17:21:24 +01:00
Pratik Mankawde
4ed9bd595d Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-09 17:03:56 +01:00
Pratik Mankawde
bab0c84793 refactor(telemetry): rename dashboard template variable label from 'Service' to 'Service Name'
Continuation of the same rename applied on phase-6-statsd (commit
91c6582bd5): 5 more dashboards introduced on phase-7 (formerly the
system-* set, renamed to bare names by 2c590a47c5) still carried the
older 'Service' UI label. Aligning them with the rest.

Same reasoning as the phase-6 commit: the underlying Prometheus label
is `service_name`, and the UI label 'Service' misled users into typing
`service=xrpld` in Grafana Metrics Drilldown.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-09 17:03:47 +01:00
Pratik Mankawde
46cff27b15 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-09 17:03:35 +01:00
Pratik Mankawde
91c6582bd5 refactor(telemetry): rename dashboard template variable label from 'Service' to 'Service Name'
The template variable's Prometheus label name is `service_name` but the
UI display label read 'Service', which mismatched what users typed into
Grafana's Metrics Drilldown UI (which defaults to filter on a bare
`service` label). Rename the display label to 'Service Name' so it
unambiguously points at the underlying `service_name` Prometheus label.

Behaviour unchanged; template-variable name (`service_name`) and PromQL
label are the same. Only the widget label shown at the top of the
dashboard changes from 'Service' to 'Service Name'.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-09 17:03:15 +01:00
Pratik Mankawde
a681eea6c4 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-08 15:26:06 +01:00
Pratik Mankawde
a1a9d45bff Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-08 15:22:52 +01:00
Pratik Mankawde
4826038068 fix(telemetry): run rename scripts for CI compliance
Rename rippled → xrpld in statsd config prefix.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-08 15:22:45 +01:00
Pratik Mankawde
a3053a34a7 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-08 14:51:55 +01:00
Pratik Mankawde
87f370a64d fix(telemetry): use service_instance_id label instead of exported_instance
exported_instance is a Prometheus scrape artifact that only exists when
Prometheus scrapes a target with a conflicting instance label. Users
running push-based pipelines (remote_write, OTLP) never get this label.
service_instance_id is always present via resource_to_telemetry_conversion
regardless of ingestion path.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-08 14:51:48 +01:00
Pratik Mankawde
ff92c3727c Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-08 14:51:36 +01:00