Commit Graph

388 Commits

Author SHA1 Message Date
Pratik Mankawde
f8bc8dee48 fix(telemetry): conditional filter-dimension legends + 30m span-nulls
Panel legends only showed the node (service_instance_id); the perf-iac
filter dimensions (xrpl_branch, xrpl_node_role) never appeared, so
baseline-vs-test series were indistinguishable.

Each node-referencing target injects its curated name via
label_replace(...,"series",...) and uses the "__auto" legend, which shows
only labels that vary across the displayed series. A word-boundary-anchored
renameByRegex chain reshapes the auto output into
"Name [node, role]-[branch, work_item]", collapsing empty groups so a
dimension appears only when the filter widens it. Aggregating queries gain
xrpl_branch/xrpl_node_role in their by() so those labels survive; multi-
dimension legends keep their rich label set and append branch/role.

Quantile panels surface the quantile via the panel title (q$quantile),
since {{quantile}} is a dashboard variable, not a series label.

Timeseries panels connect only null gaps shorter than 30m
(spanNulls=1800000): brief scrape gaps bridge, genuine outages stay broken.

Non-node panels (heatmaps, rows, tables, state timelines, category/bucket
series) are left unchanged. Applied to all 14 dashboards.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 12:18:04 +01:00
Pratik Mankawde
1a2b210214 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-07-13 20:51:40 +01:00
Pratik Mankawde
71686b0d52 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-13 20:51:34 +01:00
Pratik Mankawde
674c9a8496 fix(telemetry): correct NodeStore Cache Hit Ratio PromQL vector matching
The Cache Hit Ratio panel divided two nodestore_state series that carry
different `metric` label values (node_reads_hit / node_reads_total). Without
ignoring(metric) the two vectors share no common label set, so the division
matched nothing and the panel showed "No data".

Add `ignoring(metric)` to the division. Verified live against mainnet: now
returns a valid ratio (~0.11-0.23). Pre-existing bug, surfaced when the panel
was split out from NodeStore Read/Write Totals.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-13 20:48:51 +01:00
Pratik Mankawde
d06faec407 fix(telemetry): split mixed-unit node-health panels into single-unit panels
Three panels mixed incompatible units on one Y-axis (count vs ms, ops vs
percent), causing the small-magnitude series to flatline to zero.

- Last Close Info -> Last Close — Proposers (short) + Converge Time (ms)
- NodeStore Read/Write Totals -> Throughput (ops) + Cache Hit Ratio (%)
- NodeStore Read Threads & Duration -> Read Threads (short) + Busy Ratio (%)

Each new panel has a single unit; per-series unit overrides removed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-13 20:29:52 +01:00
Pratik Mankawde
6f9319e319 refactor(telemetry): split ledger-data-sync panels by data type
Each family panel (7-8 series) was overloaded. Split by Ledger/Transaction/
Account State so each panel shows only related series. Get+Share stay
together within a type.

- Ledger Data Exchange: 8 targets -> 3 panels
- Ledger Share/Get Traffic: 8 targets -> 3 panels
- GetObject Traffic by Type: 8 targets -> 3 panels
- GetObject Messages by Type: 7 targets -> 3 + specials panel
- GetObject Aggregate & Special: renamed, kept as one
- Overlay Traffic Heatmap: unchanged

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-13 20:29:38 +01:00
Pratik Mankawde
7f71ffe06e Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-07-13 15:45:10 +01:00
Pratik Mankawde
94a02e8183 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-13 15:45:04 +01:00
Pratik Mankawde
1b0426279c fix(telemetry): dashboard panel units, validator-health layout, filter descriptions
Set display units per panel so values render with correct scaling:
- counts/levels/depths → short
- per-minute rates (rate()*60) → cpm; genuine per-second → cps
- per-hour rate (rate()*3600) → suffix:/hr
- fee amounts → si:drops; UNL expiry → d (days)
Value-mapped and identity-number panels (server state, ledger seq, build
version) are left unitless so they are not abbreviated.

Lay out every validator-health panel at full width for readability.

Polish the shared filter descriptions (deployment tier, network values +
perf) across all eleven dashboards.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 15:44:30 +01:00
Pratik Mankawde
de579f815e fix(telemetry): dashboard panel units and filter descriptions
Set display units on native-metric dashboard panels so counts render
SI-scaled (Active Peers, Peer Disconnects → short). Polish the shared
filter descriptions: spell out the deployment-tier and network values and
add the perf network value.

Dashboards: network-traffic, ledger-data-sync, overlay-traffic-detail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 15:44:17 +01:00
Pratik Mankawde
29da7a2bdf Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-07-13 14:32:26 +01:00
Pratik Mankawde
cd091bfe0e Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-13 14:32:19 +01:00
Pratik Mankawde
3706f306b3 feat(telemetry): add perf-iac work-item/branch/node-role dashboard filters
Add three template variables (Work Item, Branch, Node Role) and their
label matchers to the remaining OTel dashboards so perf-iac comparison
runs can be filtered by ticket, comparison side (baseline/test), and node
role. The labels (xrpl_work_item, xrpl_branch, xrpl_node_role) are stamped
by perf-iac's alloy config as OTLP resource attributes and promoted to
Prometheus labels by Grafana Cloud; no repo collector change is needed.

Each variable mirrors the existing xrpl_network_type config (multi-select,
includeAll, allValue ".*") so local and non-perf runs — where these labels
are absent — continue to render (an absent PromQL label matches ".*").

Dashboards: consensus-health, ledger-operations, peer-network,
rpc-performance, transaction-overview, node-health, rpc-pathfinding,
fee-market, job-queue, peer-quality, validator-health.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 14:31:54 +01:00
Pratik Mankawde
db3fc94cc8 feat(telemetry): add perf-iac work-item/branch/node-role dashboard filters
Add three template variables (Work Item, Branch, Node Role) and their
label matchers to the native-metric dashboards so perf-iac comparison
runs can be filtered by ticket, comparison side (baseline/test), and node
role. The labels (xrpl_work_item, xrpl_branch, xrpl_node_role) are stamped
by perf-iac's alloy config as OTLP resource attributes and promoted to
Prometheus labels by Grafana Cloud; no repo collector change is needed.

Each variable mirrors the existing xrpl_network_type config (multi-select,
includeAll, allValue ".*") so local and non-perf runs — where these labels
are absent — continue to render (an absent PromQL label matches ".*").

Dashboards: ledger-data-sync, network-traffic, overlay-traffic-detail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 14:31:40 +01:00
Pratik Mankawde
3179763eb0 fix(telemetry): remove orphaned TxQ Accept Status piechart
The phase-6 change replaced this piechart with the applied-ratio state
timeline, but phase-9's independently-diverged dashboard kept its own
copy (id 17) which survived the merge-forward. Its query filtered on the
$txq_status template variable, which was removed alongside the swap, so
the panel resolved to txq_status=~"" and rendered empty. Delete it so
only the state timeline remains.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:51:00 +01:00
Pratik Mankawde
e0979a2080 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill
# Conflicts:
#	docker/telemetry/grafana/dashboards/transaction-overview.json
2026-07-10 18:42:46 +01:00
Pratik Mankawde
8ca7036a8b Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-10 18:36:48 +01:00
Pratik Mankawde
5f8ca9d84c Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
# Conflicts:
#	docker/telemetry/grafana/dashboards/transaction-overview.json
#	docs/telemetry-runbook.md
2026-07-10 18:36:39 +01:00
Pratik Mankawde
24542c46c5 feat(telemetry): TxQ accept applied-ratio state timeline
Replace the TxQ Accept Status piechart on the Transaction Overview
dashboard with a state-timeline showing each node's applied fraction of
TxQ accepts (applied / applied+failed) over time, colored by threshold
(green >=0.9, yellow >=0.7, red below). Remove the now-orphaned
txq_status template variable (the piechart was its only consumer) and
document the panel in the telemetry runbook.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:33:22 +01:00
Pratik Mankawde
b218763c8d fix(telemetry): repoint apply-failed KPI to tx.transactor (lost in merge)
The phase8->phase9 merge resolution took phase9's side of transaction-overview,
which reverted the Transaction Apply Failed Rate panel to the old
tx.apply/STATUS_CODE_ERROR query (constant zero). Re-apply the tx.transactor
stage=apply / ter_result!=tesSUCCESS repoint and raised thresholds.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:31:42 +01:00
Pratik Mankawde
efe1386440 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-07-10 18:25:11 +01:00
Pratik Mankawde
0c9001d058 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-10 18:25:05 +01:00
Pratik Mankawde
e357cf743d Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
# Conflicts:
#	docker/telemetry/grafana/dashboards/consensus-health.json
#	docker/telemetry/grafana/dashboards/ledger-operations.json
#	docker/telemetry/grafana/dashboards/peer-network.json
#	docker/telemetry/grafana/dashboards/rpc-performance.json
#	docker/telemetry/grafana/dashboards/transaction-overview.json
2026-07-10 18:24:53 +01:00
Pratik Mankawde
251b214f7b fix(telemetry): seed $node dashboard filter from target_info
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:22:02 +01:00
Pratik Mankawde
086c85fb2c fix(telemetry): seed $node dashboard filter from target_info
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:21:59 +01:00
Pratik Mankawde
a998b80f7b fix(telemetry): seed $node dashboard filter from target_info
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:21:56 +01:00
Pratik Mankawde
eb502d6eea Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill
# Conflicts:
#	docker/telemetry/grafana/dashboards/consensus-health.json
#	docker/telemetry/grafana/dashboards/ledger-operations.json
#	docker/telemetry/grafana/dashboards/node-health.json
#	docker/telemetry/grafana/dashboards/peer-network.json
#	docker/telemetry/grafana/dashboards/rpc-pathfinding.json
#	docker/telemetry/grafana/dashboards/rpc-performance.json
#	docker/telemetry/grafana/dashboards/transaction-overview.json
#	docs/telemetry-runbook.md
2026-07-10 17:28:57 +01:00
Pratik Mankawde
4f2b7cfb45 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-10 17:19:01 +01:00
Pratik Mankawde
851bc75dfc feat(telemetry): wire jq_trans_overflow_total as observable counter
Observe the overflow total from Overlay::getJqTransOverflow() via an
ObservableCounter instead of the unused push helper (which had no call
site, so the metric read flat zero). Keeps the increment path in the
overlay-owned code and removes the dead incrementJqTransOverflow() helper.
Update the job-queue panel to reflect the now-live metric, and refresh
stale xrpld_-prefixed metric names in the header comments.

NOTE: committed without a local build per instruction; needs compile +
clang-tidy verification before push.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 17:16:35 +01:00
Pratik Mankawde
3143216ad1 docs(telemetry): flag RPC Response Size instrument mismatch pending byte-histogram fix
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 17:00:13 +01:00
Pratik Mankawde
535a95363f fix(telemetry): use $__rate_interval on phase9 rate panels
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:59:19 +01:00
Pratik Mankawde
9820ae8e38 fix(telemetry): use $__rate_interval + min step on all rate panels
Replace hardcoded [5m] rate windows with [$__rate_interval] inside
rate()/increase() on plain-counter panels so ops/s panels resample
correctly on zoom-out instead of under-sampling or flat-lining on
short windows. Histogram _bucket windows keep their fixed [5m] span.
Set Min step ("interval": "15s") on every rate target to match the
Prometheus 15s scrape interval.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:51:53 +01:00
Pratik Mankawde
d7ecdacbdf docs(telemetry): flag uninstrumented tx-overflow panel; use rate-interval
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:39:26 +01:00
Pratik Mankawde
0fbe7f4e4f fix(telemetry): remove misleading stacked load-factor attribution panel
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:39:24 +01:00
Pratik Mankawde
40d49e3969 fix(telemetry): rate() nodestore panels + cache-hit ratio, dual-axis last-close, ledger-age trend, operating-mode state timeline
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:39:22 +01:00
Pratik Mankawde
ba9ac50559 fix(telemetry): drop duplicate peer-tx panel, repoint apply-failed KPI to tx.transactor
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:22:47 +01:00
Pratik Mankawde
b0f4370397 fix(telemetry): show operating mode as % time-in-mode on phase7 node-health
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:18:06 +01:00
Pratik Mankawde
a90d51c808 fix(telemetry): render phase7 traffic dashboards as per-second rates
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:06:28 +01:00
Pratik Mankawde
989c420a7a Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill
# Conflicts:
#	OpenTelemetryPlan/09-data-collection-reference.md
#	docker/telemetry/grafana/dashboards/consensus-health.json
#	docker/telemetry/grafana/dashboards/ledger-operations.json
#	docker/telemetry/grafana/dashboards/node-health.json
#	docker/telemetry/grafana/dashboards/rpc-pathfinding.json
#	docker/telemetry/grafana/dashboards/rpc-performance.json
#	docker/telemetry/grafana/dashboards/transaction-overview.json
#	docs/telemetry-runbook.md
2026-07-10 15:35:34 +01:00
Pratik Mankawde
982e3cf9ef Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-10 14:31:07 +01:00
Pratik Mankawde
86bbe8f55b fix(telemetry): rename spanmetrics to span_ namespace on phase7 (dashboards + collector config)
The spanmetrics connector had no namespace, so it emitted traces_span_metrics_*
metric names by default. The span dashboards and docs are renamed to query
span_* names; this is only correct if the connector emits them too, so add
namespace: "span" to the spanmetrics connector. Both sides change together:
renaming the dashboards without the namespace (or vice versa) would break the
pipeline. Matches the phase9 collector config.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 14:30:01 +01:00
Pratik Mankawde
34a237fab6 fix(telemetry): strip xrpld_ prefix and snake_case native dashboard/doc metric names
The phase7 OTelCollector::formatName lowercases and strips names, emitting
snake_case metrics with no xrpld_ prefix. The native Grafana dashboards and
the telemetry docs still queried the old xrpld_CamelCase names, so they were
broken against their own pipeline. Rename every metric name to match what the
code emits: drop the xrpld_ prefix and lowercase the remainder. The two job
histograms also drop the redundant 'duration' word (job_queued_us,
job_running_us) to match the phase9 forms. Add havetxset to the cspell
dictionary since the lowercased metric name no longer word-splits.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 14:07:29 +01:00
Pratik Mankawde
d174d4a7d8 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-10 13:40:42 +01:00
Pratik Mankawde
6ae6e623a7 test(telemetry): add dashboard lint for raw-cumulative panels
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 13:21:52 +01:00
Pratik Mankawde
74a4a4ab7a refactor(telemetry): default the $service_name dashboard filter to 'xrpld'
Completes the default change applied on phase-6-statsd and
phase-7-native-metrics. The four dashboards introduced on phase-9
(validator-health, peer-quality, fee-market, job-queue) now also
pre-select 'xrpld' rather than 'All' for the service_name filter.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-09 17:22:56 +01:00
Pratik Mankawde
6530e7d76c Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-09 17:22:12 +01:00
Pratik Mankawde
e5b2ce1696 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-07-09 17:22:12 +01:00
Pratik Mankawde
dae3f6e4d1 refactor(telemetry): default the $service_name dashboard filter to 'xrpld'
Continuation of the same default change applied on phase-6-statsd
(50d7047abe): the 5 dashboards introduced on phase-7 (formerly
system-*, renamed to bare names on this branch) also get their
service_name filter pre-selected to 'xrpld' instead of 'All'.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-09 17:22:01 +01:00
Pratik Mankawde
f2d1ac5c78 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-09 17:21:40 +01:00
Pratik Mankawde
50d7047abe refactor(telemetry): default the $service_name dashboard filter to 'xrpld'
Change the pre-selected value of the service_name template variable from
'All' to 'xrpld'. On first load, dashboards now filter to service.name=xrpld
directly rather than showing every service the collector sees. The 'All'
option is still selectable — this only changes the default the widget
starts on.

Reduces the chance of a fresh dashboard link showing unrelated services
(e.g. non-xrpld exporters sharing the same Grafana Cloud stack) before
the user narrows the filter.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-09 17:21:24 +01:00