Commit Graph

15393 Commits

Author SHA1 Message Date
Pratik Mankawde
f682d0dc98 fix(telemetry): migrate remaining dashboards to grouped Display-name legends
Apply the collector-built grouped-label legend approach (introduced on
phase-7) to the eleven dashboards this branch and its ancestors own:
consensus-health, fee-market, job-queue, ledger-operations, node-health,
peer-network, peer-quality, rpc-pathfinding, rpc-performance,
transaction-overview, validator-health.

Per legend panel: drop the 11-rule renameByRegex chain and legendFormat
"__auto"; set a Standard-option Display name referencing the pre-built
resource_group / iac_group labels plus, when the panel groups by a
distinguishing dimension (e.g. command, tx_type, consensus_mode, stage,
method), that dimension as a leading bracket; keep a single generic
empty-bracket cleanup rule; set tooltip maxHeight to 600.

Value-mapping renameByRegex rules (e.g. consensus close-time
increased->Coarser) are preserved. Only labels a panel groups by appear in
its legend; filter-only selectors ($command with no by(command)) narrow data
but do not become legend groups. All 14 dashboards validate; no legend
renameByRegex chain remains.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 18:06:02 +01:00
Pratik Mankawde
8296012f71 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-07-14 18:05:01 +01:00
Pratik Mankawde
9b9e4787db Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation
# Conflicts:
#	docker/telemetry/otel-collector-config.yaml
2026-07-14 18:04:52 +01:00
Pratik Mankawde
6911963939 fix(telemetry): grouped legend labels via collector, drop regex chain
The per-panel legend used legendFormat "__auto" reshaped by an 11-rule
renameByRegex chain. __auto sorts labels alphabetically and cannot group,
so the chain existed only to bolt grouping onto its output -- unmaintainable
and non-compliant with the dashboard legend guideline.

Build the grouped labels once at ingestion instead. A new transform/legendgroups
processor derives two resource attributes, promoted to Prometheus labels by
resource_to_telemetry_conversion:
  - resource_group: service_instance_id[-xrpl_network_type]
  - iac_group:      xrpl_branch-xrpl_node_role-xrpl_work_item (perf runs only)
Each is left unset when its sources are absent, so Prometheus drops the empty
label and the legend shows nothing for it. Concat is presence-guarded because
OTTL Concat renders "<nil>" for an absent attribute.

Dashboards now use a Standard-option Display name that references the pre-built
labels (${__field.labels.series} [resource_group] [iac_group]); ${__field.labels.X}
renders empty for a missing label, so absent groups vanish. A single generic
renameByRegex strips the resulting empty "[]" bracket. legendFormat is dropped
(Display name supersedes it). Tooltip maxHeight set to 600 per guideline.

Applied here to the three dashboards this branch introduced (ledger-data-sync,
network-traffic, overlay-traffic-detail); value-mapping renameByRegex rules are
preserved. Verified end-to-end on a live local stack (collector -> Prometheus ->
Grafana): perf and dev nodes render correct, clutter-free legends.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 18:04:03 +01:00
Pratik Mankawde
b1b9480c26 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-07-14 12:18:19 +01:00
Pratik Mankawde
af9a2e8bea Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-14 12:18:19 +01:00
Pratik Mankawde
f8bc8dee48 fix(telemetry): conditional filter-dimension legends + 30m span-nulls
Panel legends only showed the node (service_instance_id); the perf-iac
filter dimensions (xrpl_branch, xrpl_node_role) never appeared, so
baseline-vs-test series were indistinguishable.

Each node-referencing target injects its curated name via
label_replace(...,"series",...) and uses the "__auto" legend, which shows
only labels that vary across the displayed series. A word-boundary-anchored
renameByRegex chain reshapes the auto output into
"Name [node, role]-[branch, work_item]", collapsing empty groups so a
dimension appears only when the filter widens it. Aggregating queries gain
xrpl_branch/xrpl_node_role in their by() so those labels survive; multi-
dimension legends keep their rich label set and append branch/role.

Quantile panels surface the quantile via the panel title (q$quantile),
since {{quantile}} is a dashboard variable, not a series label.

Timeseries panels connect only null gaps shorter than 30m
(spanNulls=1800000): brief scrape gaps bridge, genuine outages stay broken.

Non-node panels (heatmaps, rows, tables, state timelines, category/bucket
series) are left unchanged. Applied to all 14 dashboards.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 12:18:04 +01:00
Pratik Mankawde
8c34fd9f5a fix(telemetry): conditional filter-dimension legends + 30m span-nulls
Panel legends only showed the node (service_instance_id); the perf-iac
filter dimensions (xrpl_branch, xrpl_node_role) never appeared, so
baseline-vs-test series were indistinguishable.

Each node-referencing target injects its curated name via
label_replace(...,"series",...) and uses the "__auto" legend, which shows
only labels that vary across the displayed series. A word-boundary-anchored
renameByRegex chain reshapes the auto output into
"Name [node, role]-[branch, work_item]", collapsing empty groups so a
dimension appears only when the filter widens it.

Timeseries panels connect only null gaps shorter than 30m
(spanNulls=1800000): brief scrape gaps bridge, genuine outages stay broken.

Dashboards (stable panel set): ledger-data-sync, network-traffic,
overlay-traffic-detail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 12:17:39 +01:00
Pratik Mankawde
1a2b210214 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-07-13 20:51:40 +01:00
Pratik Mankawde
71686b0d52 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-13 20:51:34 +01:00
Pratik Mankawde
674c9a8496 fix(telemetry): correct NodeStore Cache Hit Ratio PromQL vector matching
The Cache Hit Ratio panel divided two nodestore_state series that carry
different `metric` label values (node_reads_hit / node_reads_total). Without
ignoring(metric) the two vectors share no common label set, so the division
matched nothing and the panel showed "No data".

Add `ignoring(metric)` to the division. Verified live against mainnet: now
returns a valid ratio (~0.11-0.23). Pre-existing bug, surfaced when the panel
was split out from NodeStore Read/Write Totals.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-13 20:48:51 +01:00
Pratik Mankawde
d06faec407 fix(telemetry): split mixed-unit node-health panels into single-unit panels
Three panels mixed incompatible units on one Y-axis (count vs ms, ops vs
percent), causing the small-magnitude series to flatline to zero.

- Last Close Info -> Last Close — Proposers (short) + Converge Time (ms)
- NodeStore Read/Write Totals -> Throughput (ops) + Cache Hit Ratio (%)
- NodeStore Read Threads & Duration -> Read Threads (short) + Busy Ratio (%)

Each new panel has a single unit; per-series unit overrides removed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-13 20:29:52 +01:00
Pratik Mankawde
6f9319e319 refactor(telemetry): split ledger-data-sync panels by data type
Each family panel (7-8 series) was overloaded. Split by Ledger/Transaction/
Account State so each panel shows only related series. Get+Share stay
together within a type.

- Ledger Data Exchange: 8 targets -> 3 panels
- Ledger Share/Get Traffic: 8 targets -> 3 panels
- GetObject Traffic by Type: 8 targets -> 3 panels
- GetObject Messages by Type: 7 targets -> 3 + specials panel
- GetObject Aggregate & Special: renamed, kept as one
- Overlay Traffic Heatmap: unchanged

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-13 20:29:38 +01:00
Pratik Mankawde
7f71ffe06e Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-07-13 15:45:10 +01:00
Pratik Mankawde
94a02e8183 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-13 15:45:04 +01:00
Pratik Mankawde
1b0426279c fix(telemetry): dashboard panel units, validator-health layout, filter descriptions
Set display units per panel so values render with correct scaling:
- counts/levels/depths → short
- per-minute rates (rate()*60) → cpm; genuine per-second → cps
- per-hour rate (rate()*3600) → suffix:/hr
- fee amounts → si:drops; UNL expiry → d (days)
Value-mapped and identity-number panels (server state, ledger seq, build
version) are left unitless so they are not abbreviated.

Lay out every validator-health panel at full width for readability.

Polish the shared filter descriptions (deployment tier, network values +
perf) across all eleven dashboards.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 15:44:30 +01:00
Pratik Mankawde
de579f815e fix(telemetry): dashboard panel units and filter descriptions
Set display units on native-metric dashboard panels so counts render
SI-scaled (Active Peers, Peer Disconnects → short). Polish the shared
filter descriptions: spell out the deployment-tier and network values and
add the perf network value.

Dashboards: network-traffic, ledger-data-sync, overlay-traffic-detail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 15:44:17 +01:00
Pratik Mankawde
29da7a2bdf Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-07-13 14:32:26 +01:00
Pratik Mankawde
cd091bfe0e Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-13 14:32:19 +01:00
Pratik Mankawde
3706f306b3 feat(telemetry): add perf-iac work-item/branch/node-role dashboard filters
Add three template variables (Work Item, Branch, Node Role) and their
label matchers to the remaining OTel dashboards so perf-iac comparison
runs can be filtered by ticket, comparison side (baseline/test), and node
role. The labels (xrpl_work_item, xrpl_branch, xrpl_node_role) are stamped
by perf-iac's alloy config as OTLP resource attributes and promoted to
Prometheus labels by Grafana Cloud; no repo collector change is needed.

Each variable mirrors the existing xrpl_network_type config (multi-select,
includeAll, allValue ".*") so local and non-perf runs — where these labels
are absent — continue to render (an absent PromQL label matches ".*").

Dashboards: consensus-health, ledger-operations, peer-network,
rpc-performance, transaction-overview, node-health, rpc-pathfinding,
fee-market, job-queue, peer-quality, validator-health.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 14:31:54 +01:00
Pratik Mankawde
db3fc94cc8 feat(telemetry): add perf-iac work-item/branch/node-role dashboard filters
Add three template variables (Work Item, Branch, Node Role) and their
label matchers to the native-metric dashboards so perf-iac comparison
runs can be filtered by ticket, comparison side (baseline/test), and node
role. The labels (xrpl_work_item, xrpl_branch, xrpl_node_role) are stamped
by perf-iac's alloy config as OTLP resource attributes and promoted to
Prometheus labels by Grafana Cloud; no repo collector change is needed.

Each variable mirrors the existing xrpl_network_type config (multi-select,
includeAll, allValue ".*") so local and non-perf runs — where these labels
are absent — continue to render (an absent PromQL label matches ".*").

Dashboards: ledger-data-sync, network-traffic, overlay-traffic-detail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 14:31:40 +01:00
Pratik Mankawde
11c22bfac9 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-07-11 18:01:11 +01:00
Pratik Mankawde
5a1bbc5c2b Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-11 18:01:06 +01:00
Pratik Mankawde
919478fab0 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-11 18:01:01 +01:00
Pratik Mankawde
8a63b31c0e Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd 2026-07-11 18:00:57 +01:00
Pratik Mankawde
905061ce80 Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-07-11 18:00:46 +01:00
Pratik Mankawde
17c17901dc Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing
# Conflicts:
#	src/libxrpl/telemetry/SpanGuard.cpp
2026-07-11 18:00:46 +01:00
Pratik Mankawde
906780b622 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing 2026-07-11 18:00:11 +01:00
Pratik Mankawde
b2baefa907 Merge branch 'pratik/otel-phase1c-rpc-integration' into pratik/otel-phase2-rpc-tracing 2026-07-11 18:00:11 +01:00
Pratik Mankawde
868ee3eadd Merge branch 'pratik/otel-phase1b-telemetry-infra' into pratik/otel-phase1c-rpc-integration 2026-07-11 18:00:04 +01:00
Pratik Mankawde
a512009170 fix(telemetry): replace std::format with string concat for gcc-12
libstdc++ ships <format> starting with gcc-13, so std::format in
SpanGuard::span() fails to compile on gcc-12. Build the "<prefix>.<name>"
span name via std::string::append instead and drop the <format> include.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-11 17:59:33 +01:00
Pratik Mankawde
3179763eb0 fix(telemetry): remove orphaned TxQ Accept Status piechart
The phase-6 change replaced this piechart with the applied-ratio state
timeline, but phase-9's independently-diverged dashboard kept its own
copy (id 17) which survived the merge-forward. Its query filtered on the
$txq_status template variable, which was removed alongside the swap, so
the panel resolved to txq_status=~"" and rendered empty. Delete it so
only the state timeline remains.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:51:00 +01:00
Pratik Mankawde
505dd360ee Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-07-10 18:49:14 +01:00
Pratik Mankawde
e9a0b49c5b Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-10 18:49:05 +01:00
Pratik Mankawde
cb413c18d3 docs(telemetry): correct jq_trans_overflow_total to ObservableCounter in 09-doc
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:48:48 +01:00
Pratik Mankawde
ae47ab08f0 docs(telemetry): note rpc_size ms-histogram instrument mismatch in 09-doc
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:48:43 +01:00
Pratik Mankawde
e0979a2080 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill
# Conflicts:
#	docker/telemetry/grafana/dashboards/transaction-overview.json
2026-07-10 18:42:46 +01:00
Pratik Mankawde
8ca7036a8b Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-10 18:36:48 +01:00
Pratik Mankawde
5f8ca9d84c Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
# Conflicts:
#	docker/telemetry/grafana/dashboards/transaction-overview.json
#	docs/telemetry-runbook.md
2026-07-10 18:36:39 +01:00
Pratik Mankawde
24542c46c5 feat(telemetry): TxQ accept applied-ratio state timeline
Replace the TxQ Accept Status piechart on the Transaction Overview
dashboard with a state-timeline showing each node's applied fraction of
TxQ accepts (applied / applied+failed) over time, colored by threshold
(green >=0.9, yellow >=0.7, red below). Remove the now-orphaned
txq_status template variable (the piechart was its only consumer) and
document the panel in the telemetry runbook.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:33:22 +01:00
Pratik Mankawde
b218763c8d fix(telemetry): repoint apply-failed KPI to tx.transactor (lost in merge)
The phase8->phase9 merge resolution took phase9's side of transaction-overview,
which reverted the Transaction Apply Failed Rate panel to the old
tx.apply/STATUS_CODE_ERROR query (constant zero). Re-apply the tx.transactor
stage=apply / ter_result!=tesSUCCESS repoint and raised thresholds.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:31:42 +01:00
Pratik Mankawde
efe1386440 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-07-10 18:25:11 +01:00
Pratik Mankawde
0c9001d058 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-10 18:25:05 +01:00
Pratik Mankawde
e357cf743d Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
# Conflicts:
#	docker/telemetry/grafana/dashboards/consensus-health.json
#	docker/telemetry/grafana/dashboards/ledger-operations.json
#	docker/telemetry/grafana/dashboards/peer-network.json
#	docker/telemetry/grafana/dashboards/rpc-performance.json
#	docker/telemetry/grafana/dashboards/transaction-overview.json
2026-07-10 18:24:53 +01:00
Pratik Mankawde
251b214f7b fix(telemetry): seed $node dashboard filter from target_info
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:22:02 +01:00
Pratik Mankawde
086c85fb2c fix(telemetry): seed $node dashboard filter from target_info
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:21:59 +01:00
Pratik Mankawde
a998b80f7b fix(telemetry): seed $node dashboard filter from target_info
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:21:56 +01:00
Pratik Mankawde
eb502d6eea Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill
# Conflicts:
#	docker/telemetry/grafana/dashboards/consensus-health.json
#	docker/telemetry/grafana/dashboards/ledger-operations.json
#	docker/telemetry/grafana/dashboards/node-health.json
#	docker/telemetry/grafana/dashboards/peer-network.json
#	docker/telemetry/grafana/dashboards/rpc-pathfinding.json
#	docker/telemetry/grafana/dashboards/rpc-performance.json
#	docker/telemetry/grafana/dashboards/transaction-overview.json
#	docs/telemetry-runbook.md
2026-07-10 17:28:57 +01:00
Pratik Mankawde
4f2b7cfb45 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-10 17:19:01 +01:00
Pratik Mankawde
4aa2181b99 docs(telemetry): sync runbook panel tables with remediated dashboards
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 17:18:29 +01:00