Panel legends only showed the node (service_instance_id); the perf-iac
filter dimensions (xrpl_branch, xrpl_node_role) never appeared, so
baseline-vs-test series were indistinguishable.
Each node-referencing target now injects its curated name via
label_replace(...,"series",...) and uses the Prometheus "__auto" legend,
which shows only labels that vary across the displayed series. A
renameByRegex chain reshapes the auto output into
"Name [node, role]-[branch, work_item]", collapsing empty groups so a
dimension appears only when the filter widens it. Aggregating queries gain
xrpl_branch/xrpl_node_role in their by() so those labels survive; multi-
dimension legends keep their rich label set and append branch/role.
Quantile panels surfaced the quantile via the panel title (q$quantile),
since {{quantile}} is a dashboard variable, not a series label.
Timeseries panels now connect only null gaps shorter than 30m
(spanNulls=1800000): brief scrape gaps bridge, genuine outages stay broken.
Non-node panels (heatmaps, rows, tables, state timelines, category/bucket
series) are left unchanged. Applied to all 14 dashboards.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Grafana Cloud variants of the 14 xrpld dashboards. Identical to the local
grafana/dashboards/ originals except the DS_PROMETHEUS / DS_TEMPO datasource
template variables have their `current` value pinned to the Grafana Cloud
datasource UIDs (grafanacloud-ripplex-prometheus / -traces).
Includes the ledger-data-sync data-type split and node-health mixed-unit
split + Cache Hit Ratio fix merged forward from phases 7 and 9.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Cache Hit Ratio panel divided two nodestore_state series that carry
different `metric` label values (node_reads_hit / node_reads_total). Without
ignoring(metric) the two vectors share no common label set, so the division
matched nothing and the panel showed "No data".
Add `ignoring(metric)` to the division. Verified live against mainnet: now
returns a valid ratio (~0.11-0.23). Pre-existing bug, surfaced when the panel
was split out from NodeStore Read/Write Totals.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Three panels mixed incompatible units on one Y-axis (count vs ms, ops vs
percent), causing the small-magnitude series to flatline to zero.
- Last Close Info -> Last Close — Proposers (short) + Converge Time (ms)
- NodeStore Read/Write Totals -> Throughput (ops) + Cache Hit Ratio (%)
- NodeStore Read Threads & Duration -> Read Threads (short) + Busy Ratio (%)
Each new panel has a single unit; per-series unit overrides removed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Each family panel (7-8 series) was overloaded. Split by Ledger/Transaction/
Account State so each panel shows only related series. Get+Share stay
together within a type.
- Ledger Data Exchange: 8 targets -> 3 panels
- Ledger Share/Get Traffic: 8 targets -> 3 panels
- GetObject Traffic by Type: 8 targets -> 3 panels
- GetObject Messages by Type: 7 targets -> 3 + specials panel
- GetObject Aggregate & Special: renamed, kept as one
- Overlay Traffic Heatmap: unchanged
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Span-derived metrics (span_calls_total, span_duration_milliseconds_*) were
emitted at the spanmetrics connector default of 60s. Grafana Cloud stores one
datapoint per flush, but dashboard panels use rate(...[$__rate_interval]),
whose window at short ranges (<=1h) falls below 60s — leaving <2 samples per
window, so those panels rendered "No data" on the Cloud datasource. (Local
Prometheus scrapes at 15s, so local dashboards were unaffected.)
Set metrics_flush_interval: 15s so every rate window holds >=2 samples at any
dashboard range. Trade-off: ~4x span-metric datapoints pushed to Cloud. Native
insight metrics (ledgermaster_*, jobq_*, peer_quality) already push faster and
were never affected.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Brings dashboard panel units, validator-health full-width layout, and filter
description polish into phase 10. The validator-health conflict was resolved
by keeping phase 10's per-target datasource uids and re-applying the unit,
layout, and description changes on top.
Set display units per panel so values render with correct scaling:
- counts/levels/depths → short
- per-minute rates (rate()*60) → cpm; genuine per-second → cps
- per-hour rate (rate()*3600) → suffix:/hr
- fee amounts → si:drops; UNL expiry → d (days)
Value-mapped and identity-number panels (server state, ledger seq, build
version) are left unitless so they are not abbreviated.
Lay out every validator-health panel at full width for readability.
Polish the shared filter descriptions (deployment tier, network values +
perf) across all eleven dashboards.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Set display units on native-metric dashboard panels so counts render
SI-scaled (Active Peers, Peer Disconnects → short). Polish the shared
filter descriptions: spell out the deployment-tier and network values and
add the perf network value.
Dashboards: network-traffic, ledger-data-sync, overlay-traffic-detail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Brings the perf-iac dashboard filters (work item / branch / node role) into
phase 10. Conflicts in network-traffic and node-health were resolved by
keeping phase 10's query fixes (jobq_job_count rename; per-metric
label_replace identity for bytes-in breakdown) and re-applying the three
filter matchers on top.
Add three template variables (Work Item, Branch, Node Role) and their
label matchers to the remaining OTel dashboards so perf-iac comparison
runs can be filtered by ticket, comparison side (baseline/test), and node
role. The labels (xrpl_work_item, xrpl_branch, xrpl_node_role) are stamped
by perf-iac's alloy config as OTLP resource attributes and promoted to
Prometheus labels by Grafana Cloud; no repo collector change is needed.
Each variable mirrors the existing xrpl_network_type config (multi-select,
includeAll, allValue ".*") so local and non-perf runs — where these labels
are absent — continue to render (an absent PromQL label matches ".*").
Dashboards: consensus-health, ledger-operations, peer-network,
rpc-performance, transaction-overview, node-health, rpc-pathfinding,
fee-market, job-queue, peer-quality, validator-health.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add three template variables (Work Item, Branch, Node Role) and their
label matchers to the native-metric dashboards so perf-iac comparison
runs can be filtered by ticket, comparison side (baseline/test), and node
role. The labels (xrpl_work_item, xrpl_branch, xrpl_node_role) are stamped
by perf-iac's alloy config as OTLP resource attributes and promoted to
Prometheus labels by Grafana Cloud; no repo collector change is needed.
Each variable mirrors the existing xrpl_network_type config (multi-select,
includeAll, allValue ".*") so local and non-perf runs — where these labels
are absent — continue to render (an absent PromQL label matches ".*").
Dashboards: ledger-data-sync, network-traffic, overlay-traffic-detail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dashboard audit against the live datasource surfaced two broken panel queries
and applied UX polish across all 14 dashboards.
- node-health "Job Queue Depth": job_count -> jobq_job_count. The JobQueue
collector is wrapped in group("jobq") (Application.cpp), so the registered
job_count gauge is emitted with the jobq_ prefix; the panel queried the
unprefixed name and returned nothing.
- network-traffic "Overlay Traffic by Category" + "All Traffic Categories":
topk(N, rate({__name__=~".*_bytes_in"}[...])) errors on Mimir ("vector
cannot contain metrics with the same labelset") because rate() drops
__name__ and the many counters collapse. Replaced with an enumerated
label_replace form that re-attaches __name__ per metric, preserving the
{{__name__}} legend and per-series display-name overrides.
- All 14 dashboards: refresh set to 10s.
- peer-quality: each panel full screen width.
- validator-health: at most two panels per row (row headers preserved).
- docs: telemetry-runbook and 06-implementation-phases updated for the
jobq_ prefix and the network-traffic query pattern.
Verified end-to-end against a local mainnet xrpld node feeding the local
stack: jobq_job_count returns data (old job_count empty), both network-traffic
exprs execute (old form reproduces the labelset error), and panels render
through the Grafana proxy. All 14 pass validate_dashboards.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The phase-6 change replaced this piechart with the applied-ratio state
timeline, but phase-9's independently-diverged dashboard kept its own
copy (id 17) which survived the merge-forward. Its query filtered on the
$txq_status template variable, which was removed alongside the swap, so
the panel resolved to txq_status=~"" and rendered empty. Delete it so
only the state timeline remains.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the TxQ Accept Status piechart on the Transaction Overview
dashboard with a state-timeline showing each node's applied fraction of
TxQ accepts (applied / applied+failed) over time, colored by threshold
(green >=0.9, yellow >=0.7, red below). Remove the now-orphaned
txq_status template variable (the piechart was its only consumer) and
document the panel in the telemetry runbook.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The phase8->phase9 merge resolution took phase9's side of transaction-overview,
which reverted the Transaction Apply Failed Rate panel to the old
tx.apply/STATUS_CODE_ERROR query (constant zero). Re-apply the tx.transactor
stage=apply / ter_result!=tesSUCCESS repoint and raised thresholds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add an Alloy-based collector config as an alternative to the reference
OpenTelemetry Collector. It forwards xrpld OTLP traces and native metrics
to the Grafana Cloud OTLP gateway and, critically, runs a spanmetrics
connector so span-derived RED metrics (span_calls_total,
span_duration_milliseconds_*) are produced. Those metrics are not emitted
by xrpld; they are derived from spans, so an Alloy node without this
connector is missing from every span-based dashboard.
All credentials, endpoints, usernames and the per-node host label are read
from the environment (no secrets in the file). Adds a matching
.env.grafanacloud-alloy.example and gitignore exception, and the otelcol
term to the cspell dictionary.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Observe the overflow total from Overlay::getJqTransOverflow() via an
ObservableCounter instead of the unused push helper (which had no call
site, so the metric read flat zero). Keeps the increment path in the
overlay-owned code and removes the dead incrementJqTransOverflow() helper.
Update the job-queue panel to reflect the now-live metric, and refresh
stale xrpld_-prefixed metric names in the header comments.
NOTE: committed without a local build per instruction; needs compile +
clang-tidy verification before push.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace hardcoded [5m] rate windows with [$__rate_interval] inside
rate()/increase() on plain-counter panels so ops/s panels resample
correctly on zoom-out instead of under-sampling or flat-lining on
short windows. Histogram _bucket windows keep their fixed [5m] span.
Set Min step ("interval": "15s") on every rate target to match the
Prometheus 15s scrape interval.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
