Match the panel height the operator settled on in Grafana (18 rows) so
all 20 ranked bars render without an inner scrollbar, and reflow the
Sync Diagnostics row and the panels below it accordingly.
The bar labels on 'Overlay Traffic Heatmap (All Categories, Bytes In)'
were raw metric names carrying a redundant '_bytes_in' suffix, and the
half-width 8-row panel truncated both the category and the node
identity.
- Strip the '_bytes_in' suffix from the derived series label; the panel
title already states the metric is inbound bytes.
- Widen the panel to full width and grow it to 12 rows so 20 bars render
with their full category and node labels.
- Shift the Sync Diagnostics row and the panels below it down by 4 to
keep the layout gap-free.
- Note the label derivation in the panel description.
The four aggregated Sync Diagnostics panels grouped by xrpl_work_item,
which no layer in this repo emits (it is injected by the perf-iac
harness). That tripped Rule D of the telemetry naming check:
D ledger-data-sync.json xrpl_work_item
must exist in L1, a metric label, or be a builtin
Align with the convention used by every other aggregated panel in the
dashboard set: group by (service_instance_id, xrpl_branch,
xrpl_node_role) and leave the xrpl_ident label_join untouched. The
legend is unaffected -- label_join over a label dropped by the
aggregation contributes an empty segment, which the trailing
label_replace already strips.
Verified live: both the NuDB ratio and the job-queue p95 queries still
return per-node series rendering as "[aws-dev-xrpl-1]".
The 0-4 values now render as named states via value mappings, so the numeric
axis label is misleading. Rename to "Server State".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add value mappings (0 Disconnected, 1 Connected, 2 Syncing, 3 Tracking,
4 Full) and matching red->green thresholds to the Sync State panel, so the
line, tooltip, and legend render state names and colors instead of bare 0-4.
Keeps the minimal custom-key convention; only the Sync State panel changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rework the 10 Sync Diagnostics panels' legends to match the format used by the
existing panels: wrap each query in label_join + label_replace to build the
xrpl_ident label ([service_instance_id, xrpl_branch, xrpl_work_item] with empty
values stripped) and set displayName to "${series} ${xrpl_ident}". Verified
against a live node: renders as "Series [aws-dev-xrpl-1]" with no empty-label
gaps, identical mechanism to the original 15 panels. Only the new panels change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a 10-panel "Sync Diagnostics" row that shows ledger-sync slowdown and its
causes, laid out top-to-bottom as symptom -> latency -> root cause:
- Red flags: Sync State, Validated Ledger Age, Ledger Close Rate
- Latency: Job Queue Wait p95 by type, NuDB Read Latency, I/O Scheduler p95
- Root cause: NuDB Cache Hit Ratio, NuDB Read Pressure, Job Queue Depth,
Load Factor & Peers
All panels use native beast::insight metrics introduced on this branch
(nodestore_state, ledgermaster_*, jobq_*_q, ios_latency, load_factor_metrics,
peer_finder_*) plus the consensus.mode_change span metric. Queries filter by
the dashboard's $node and tier variables and were verified against a live
node. Nodestore ratio panels use sum-by(identity) matching to bridge the
metric= sub-label.
Format matches the dashboard convention: minimal custom keys, ${DS_PROMETHEUS}
datasource uid, and the standard What/How/Reading/Healthy/Watch/Source/Function
description structure. Pure-append; existing panels untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Group the node + perf-iac identity labels inside one pair of brackets in the
legend Display name:
${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]
so a series reads e.g. "Queue Depth [validator-0 test:pr:abc123 validator
RIPD-7455]" on a perf node and "Queue Depth [aws-dev-xrpl-1]" on a plain node.
Absent perf-iac labels render empty and collapse to a single space before the
closing bracket (HTML/SVG whitespace collapse) — the only cosmetic artifact on
non-perf nodes; no empty "[]" and no data impact. Verified on a live local
Grafana render.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The collector-built resource_group / iac_group labels did not survive the
dashboards' own PromQL aggregation: panels that sum by(service_instance_id,
xrpl_branch, xrpl_node_role, ...) drop any label not named in the by() clause,
so the derived group labels were aggregated away and the Display name rendered
empty ("Observing" with no node). Building them in the collector also coupled
every dashboard to a lockstep collector redeploy on every node.
Switch the legend Display name to reference the raw labels the panels already
group by, which are always present by construction and need no collector
support:
${__field.labels.series} ${__field.labels.service_instance_id}
${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role}
${__field.labels.xrpl_work_item}
${__field.labels.X} renders empty for a label a series lacks (verified in
Grafana), so dev/mainnet nodes show just "<name> <node>" and perf nodes add the
branch/role/work-item, with no empty brackets and no double spaces. Verified
live across timeseries, stat, bargauge, piechart, gauge and cross-node
aggregation panels.
Revert the transform/legendgroups processor from the local collector config;
this file is now identical to its pre-change state.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Display name templated the brackets: "${series} [${resource_group}]
[${iac_group}]". When a group label was absent the "${}" rendered empty,
leaving a literal "[]" in the legend, and the cleanup renameByRegex could not
remove it (Grafana runs that transform on the pre-interpolation template, which
has no "[]" -- it only appears after interpolation at render time).
Bake the brackets and comma separators into the group-label VALUES in the
collector instead (resource_group="[node, mainnet]",
iac_group="[branch, role, work-item]"), so the Display name is bracket-free
("${series} ${resource_group} ${iac_group}") and an absent group renders as
nothing. Separator changes from "-" to ", ". Drop the dead cleanup transform.
Covers this branch's local collector config and its three dashboards
(ledger-data-sync, network-traffic, overlay-traffic-detail).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "Overlay Traffic by Category (Bytes In)" bargauge names each bar by
category via 22 byName field-override displayNames (Transactions, Proposals,
...). The legend migration had also added a generic
${__field.labels.series} [...] Display name, which competed with those
curated names. Remove the generic Display name (and its now-unneeded
empty-bracket cleanup transform) from this panel so the category names stand;
this panel's identity is the traffic category, not the node/perf-iac grouping.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The transform/legendgroups guards checked only != nil. The perf-iac collector
stamps xrpl.branch / xrpl.node.role / xrpl.work.item with an empty-string
default (work_item | default('')) when a run has no work item, and an empty
string is not nil. OTTL Concat renders a stray separator for an empty element,
so an all-empty perf-iac stamp would have produced iac_group="--".
Add != "" to every presence guard (iac_group sources, service.instance.id,
xrpl.network.type) so a group is built only from genuinely-present values and
left unset otherwise. Verified on a live collector: empty-string perf-iac attrs
leave iac_group absent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-panel legend used legendFormat "__auto" reshaped by an 11-rule
renameByRegex chain. __auto sorts labels alphabetically and cannot group,
so the chain existed only to bolt grouping onto its output -- unmaintainable
and non-compliant with the dashboard legend guideline.
Build the grouped labels once at ingestion instead. A new transform/legendgroups
processor derives two resource attributes, promoted to Prometheus labels by
resource_to_telemetry_conversion:
- resource_group: service_instance_id[-xrpl_network_type]
- iac_group: xrpl_branch-xrpl_node_role-xrpl_work_item (perf runs only)
Each is left unset when its sources are absent, so Prometheus drops the empty
label and the legend shows nothing for it. Concat is presence-guarded because
OTTL Concat renders "<nil>" for an absent attribute.
Dashboards now use a Standard-option Display name that references the pre-built
labels (${__field.labels.series} [resource_group] [iac_group]); ${__field.labels.X}
renders empty for a missing label, so absent groups vanish. A single generic
renameByRegex strips the resulting empty "[]" bracket. legendFormat is dropped
(Display name supersedes it). Tooltip maxHeight set to 600 per guideline.
Applied here to the three dashboards this branch introduced (ledger-data-sync,
network-traffic, overlay-traffic-detail); value-mapping renameByRegex rules are
preserved. Verified end-to-end on a live local stack (collector -> Prometheus ->
Grafana): perf and dev nodes render correct, clutter-free legends.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Panel legends only showed the node (service_instance_id); the perf-iac
filter dimensions (xrpl_branch, xrpl_node_role) never appeared, so
baseline-vs-test series were indistinguishable.
Each node-referencing target injects its curated name via
label_replace(...,"series",...) and uses the "__auto" legend, which shows
only labels that vary across the displayed series. A word-boundary-anchored
renameByRegex chain reshapes the auto output into
"Name [node, role]-[branch, work_item]", collapsing empty groups so a
dimension appears only when the filter widens it.
Timeseries panels connect only null gaps shorter than 30m
(spanNulls=1800000): brief scrape gaps bridge, genuine outages stay broken.
Dashboards (stable panel set): ledger-data-sync, network-traffic,
overlay-traffic-detail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Each family panel (7-8 series) was overloaded. Split by Ledger/Transaction/
Account State so each panel shows only related series. Get+Share stay
together within a type.
- Ledger Data Exchange: 8 targets -> 3 panels
- Ledger Share/Get Traffic: 8 targets -> 3 panels
- GetObject Traffic by Type: 8 targets -> 3 panels
- GetObject Messages by Type: 7 targets -> 3 + specials panel
- GetObject Aggregate & Special: renamed, kept as one
- Overlay Traffic Heatmap: unchanged
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Set display units on native-metric dashboard panels so counts render
SI-scaled (Active Peers, Peer Disconnects → short). Polish the shared
filter descriptions: spell out the deployment-tier and network values and
add the perf network value.
Dashboards: network-traffic, ledger-data-sync, overlay-traffic-detail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add three template variables (Work Item, Branch, Node Role) and their
label matchers to the native-metric dashboards so perf-iac comparison
runs can be filtered by ticket, comparison side (baseline/test), and node
role. The labels (xrpl_work_item, xrpl_branch, xrpl_node_role) are stamped
by perf-iac's alloy config as OTLP resource attributes and promoted to
Prometheus labels by Grafana Cloud; no repo collector change is needed.
Each variable mirrors the existing xrpl_network_type config (multi-select,
includeAll, allValue ".*") so local and non-perf runs — where these labels
are absent — continue to render (an absent PromQL label matches ".*").
Dashboards: ledger-data-sync, network-traffic, overlay-traffic-detail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the TxQ Accept Status piechart on the Transaction Overview
dashboard with a state-timeline showing each node's applied fraction of
TxQ accepts (applied / applied+failed) over time, colored by threshold
(green >=0.9, yellow >=0.7, red below). Remove the now-orphaned
txq_status template variable (the piechart was its only consumer) and
document the panel in the telemetry runbook.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace hardcoded [5m] rate windows with [$__rate_interval] inside
rate()/increase() on plain-counter panels so ops/s panels resample
correctly on zoom-out instead of under-sampling or flat-lining on
short windows. Histogram _bucket windows keep their fixed [5m] span.
Set Min step ("interval": "15s") on every rate target to match the
Prometheus 15s scrape interval.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The spanmetrics connector had no namespace, so it emitted traces_span_metrics_*
metric names by default. The span dashboards and docs are renamed to query
span_* names; this is only correct if the connector emits them too, so add
namespace: "span" to the spanmetrics connector. Both sides change together:
renaming the dashboards without the namespace (or vice versa) would break the
pipeline. Matches the phase9 collector config.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The phase7 OTelCollector::formatName lowercases and strips names, emitting
snake_case metrics with no xrpld_ prefix. The native Grafana dashboards and
the telemetry docs still queried the old xrpld_CamelCase names, so they were
broken against their own pipeline. Rename every metric name to match what the
code emits: drop the xrpld_ prefix and lowercase the remainder. The two job
histograms also drop the redundant 'duration' word (job_queued_us,
job_running_us) to match the phase9 forms. Add havetxset to the cspell
dictionary since the lowercased metric name no longer word-splits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Continuation of the same default change applied on phase-6-statsd
(50d7047abe): the 5 dashboards introduced on phase-7 (formerly
system-*, renamed to bare names on this branch) also get their
service_name filter pre-selected to 'xrpld' instead of 'All'.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Change the pre-selected value of the service_name template variable from
'All' to 'xrpld'. On first load, dashboards now filter to service.name=xrpld
directly rather than showing every service the collector sees. The 'All'
option is still selectable — this only changes the default the widget
starts on.
Reduces the chance of a fresh dashboard link showing unrelated services
(e.g. non-xrpld exporters sharing the same Grafana Cloud stack) before
the user narrows the filter.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Continuation of the same rename applied on phase-6-statsd (commit
91c6582bd5): 5 more dashboards introduced on phase-7 (formerly the
system-* set, renamed to bare names by 2c590a47c5) still carried the
older 'Service' UI label. Aligning them with the rest.
Same reasoning as the phase-6 commit: the underlying Prometheus label
is `service_name`, and the UI label 'Service' misled users into typing
`service=xrpld` in Grafana Metrics Drilldown.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The template variable's Prometheus label name is `service_name` but the
UI display label read 'Service', which mismatched what users typed into
Grafana's Metrics Drilldown UI (which defaults to filter on a bare
`service` label). Rename the display label to 'Service Name' so it
unambiguously points at the underlying `service_name` Prometheus label.
Behaviour unchanged; template-variable name (`service_name`) and PromQL
label are the same. Only the widget label shown at the top of the
dashboard changes from 'Service' to 'Service Name'.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
exported_instance is a Prometheus scrape artifact that only exists when
Prometheus scrapes a target with a conflicting instance label. Users
running push-based pipelines (remote_write, OTLP) never get this label.
service_instance_id is always present via resource_to_telemetry_conversion
regardless of ingestion path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
exported_instance is a Prometheus scrape artifact that only exists when
Prometheus scrapes a target with a conflicting instance label. Users
running push-based pipelines (remote_write, OTLP) never get this label.
service_instance_id is always present via resource_to_telemetry_conversion
regardless of ingestion path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 3/4" token, keep the description.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 4" tokens, keep descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 3" tokens, keep descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
