Group the node + perf-iac identity labels inside one pair of brackets in the
legend Display name:
${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]
so a series reads e.g. "Queue Depth [validator-0 test:pr:abc123 validator
RIPD-7455]" on a perf node and "Queue Depth [aws-dev-xrpl-1]" on a plain node.
Absent perf-iac labels render empty and collapse to a single space before the
closing bracket (HTML/SVG whitespace collapse) — the only cosmetic artifact on
non-perf nodes; no empty "[]" and no data impact. Verified on a live local
Grafana render.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The collector-built resource_group / iac_group labels did not survive the
dashboards' own PromQL aggregation: panels that sum by(service_instance_id,
xrpl_branch, xrpl_node_role, ...) drop any label not named in the by() clause,
so the derived group labels were aggregated away and the Display name rendered
empty ("Observing" with no node). Building them in the collector also coupled
every dashboard to a lockstep collector redeploy on every node.
Switch the legend Display name to reference the raw labels the panels already
group by, which are always present by construction and need no collector
support:
${__field.labels.series} ${__field.labels.service_instance_id}
${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role}
${__field.labels.xrpl_work_item}
${__field.labels.X} renders empty for a label a series lacks (verified in
Grafana), so dev/mainnet nodes show just "<name> <node>" and perf nodes add the
branch/role/work-item, with no empty brackets and no double spaces. Verified
live across timeseries, stat, bargauge, piechart, gauge and cross-node
aggregation panels.
Revert the transform/legendgroups processor from the local collector config;
this file is now identical to its pre-change state.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Display name templated the brackets: "${series} [${resource_group}]
[${iac_group}]". When a group label was absent the "${}" rendered empty,
leaving a literal "[]" in the legend, and the cleanup renameByRegex could not
remove it (Grafana runs that transform on the pre-interpolation template, which
has no "[]" -- it only appears after interpolation at render time).
Bake the brackets and comma separators into the group-label VALUES in the
collector instead (resource_group="[node, mainnet]",
iac_group="[branch, role, work-item]"), so the Display name is bracket-free
("${series} ${resource_group} ${iac_group}") and an absent group renders as
nothing. Separator changes from "-" to ", ". Drop the dead cleanup transform.
Covers this branch's local collector config and its three dashboards
(ledger-data-sync, network-traffic, overlay-traffic-detail).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "Overlay Traffic by Category (Bytes In)" bargauge names each bar by
category via 22 byName field-override displayNames (Transactions, Proposals,
...). The legend migration had also added a generic
${__field.labels.series} [...] Display name, which competed with those
curated names. Remove the generic Display name (and its now-unneeded
empty-bracket cleanup transform) from this panel so the category names stand;
this panel's identity is the traffic category, not the node/perf-iac grouping.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The transform/legendgroups guards checked only != nil. The perf-iac collector
stamps xrpl.branch / xrpl.node.role / xrpl.work.item with an empty-string
default (work_item | default('')) when a run has no work item, and an empty
string is not nil. OTTL Concat renders a stray separator for an empty element,
so an all-empty perf-iac stamp would have produced iac_group="--".
Add != "" to every presence guard (iac_group sources, service.instance.id,
xrpl.network.type) so a group is built only from genuinely-present values and
left unset otherwise. Verified on a live collector: empty-string perf-iac attrs
leave iac_group absent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-panel legend used legendFormat "__auto" reshaped by an 11-rule
renameByRegex chain. __auto sorts labels alphabetically and cannot group,
so the chain existed only to bolt grouping onto its output -- unmaintainable
and non-compliant with the dashboard legend guideline.
Build the grouped labels once at ingestion instead. A new transform/legendgroups
processor derives two resource attributes, promoted to Prometheus labels by
resource_to_telemetry_conversion:
- resource_group: service_instance_id[-xrpl_network_type]
- iac_group: xrpl_branch-xrpl_node_role-xrpl_work_item (perf runs only)
Each is left unset when its sources are absent, so Prometheus drops the empty
label and the legend shows nothing for it. Concat is presence-guarded because
OTTL Concat renders "<nil>" for an absent attribute.
Dashboards now use a Standard-option Display name that references the pre-built
labels (${__field.labels.series} [resource_group] [iac_group]); ${__field.labels.X}
renders empty for a missing label, so absent groups vanish. A single generic
renameByRegex strips the resulting empty "[]" bracket. legendFormat is dropped
(Display name supersedes it). Tooltip maxHeight set to 600 per guideline.
Applied here to the three dashboards this branch introduced (ledger-data-sync,
network-traffic, overlay-traffic-detail); value-mapping renameByRegex rules are
preserved. Verified end-to-end on a live local stack (collector -> Prometheus ->
Grafana): perf and dev nodes render correct, clutter-free legends.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Panel legends only showed the node (service_instance_id); the perf-iac
filter dimensions (xrpl_branch, xrpl_node_role) never appeared, so
baseline-vs-test series were indistinguishable.
Each node-referencing target injects its curated name via
label_replace(...,"series",...) and uses the "__auto" legend, which shows
only labels that vary across the displayed series. A word-boundary-anchored
renameByRegex chain reshapes the auto output into
"Name [node, role]-[branch, work_item]", collapsing empty groups so a
dimension appears only when the filter widens it.
Timeseries panels connect only null gaps shorter than 30m
(spanNulls=1800000): brief scrape gaps bridge, genuine outages stay broken.
Dashboards (stable panel set): ledger-data-sync, network-traffic,
overlay-traffic-detail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Each family panel (7-8 series) was overloaded. Split by Ledger/Transaction/
Account State so each panel shows only related series. Get+Share stay
together within a type.
- Ledger Data Exchange: 8 targets -> 3 panels
- Ledger Share/Get Traffic: 8 targets -> 3 panels
- GetObject Traffic by Type: 8 targets -> 3 panels
- GetObject Messages by Type: 7 targets -> 3 + specials panel
- GetObject Aggregate & Special: renamed, kept as one
- Overlay Traffic Heatmap: unchanged
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Set display units on native-metric dashboard panels so counts render
SI-scaled (Active Peers, Peer Disconnects → short). Polish the shared
filter descriptions: spell out the deployment-tier and network values and
add the perf network value.
Dashboards: network-traffic, ledger-data-sync, overlay-traffic-detail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add three template variables (Work Item, Branch, Node Role) and their
label matchers to the native-metric dashboards so perf-iac comparison
runs can be filtered by ticket, comparison side (baseline/test), and node
role. The labels (xrpl_work_item, xrpl_branch, xrpl_node_role) are stamped
by perf-iac's alloy config as OTLP resource attributes and promoted to
Prometheus labels by Grafana Cloud; no repo collector change is needed.
Each variable mirrors the existing xrpl_network_type config (multi-select,
includeAll, allValue ".*") so local and non-perf runs — where these labels
are absent — continue to render (an absent PromQL label matches ".*").
Dashboards: ledger-data-sync, network-traffic, overlay-traffic-detail.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the TxQ Accept Status piechart on the Transaction Overview
dashboard with a state-timeline showing each node's applied fraction of
TxQ accepts (applied / applied+failed) over time, colored by threshold
(green >=0.9, yellow >=0.7, red below). Remove the now-orphaned
txq_status template variable (the piechart was its only consumer) and
document the panel in the telemetry runbook.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace hardcoded [5m] rate windows with [$__rate_interval] inside
rate()/increase() on plain-counter panels so ops/s panels resample
correctly on zoom-out instead of under-sampling or flat-lining on
short windows. Histogram _bucket windows keep their fixed [5m] span.
Set Min step ("interval": "15s") on every rate target to match the
Prometheus 15s scrape interval.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The spanmetrics connector had no namespace, so it emitted traces_span_metrics_*
metric names by default. The span dashboards and docs are renamed to query
span_* names; this is only correct if the connector emits them too, so add
namespace: "span" to the spanmetrics connector. Both sides change together:
renaming the dashboards without the namespace (or vice versa) would break the
pipeline. Matches the phase9 collector config.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The phase7 OTelCollector::formatName lowercases and strips names, emitting
snake_case metrics with no xrpld_ prefix. The native Grafana dashboards and
the telemetry docs still queried the old xrpld_CamelCase names, so they were
broken against their own pipeline. Rename every metric name to match what the
code emits: drop the xrpld_ prefix and lowercase the remainder. The two job
histograms also drop the redundant 'duration' word (job_queued_us,
job_running_us) to match the phase9 forms. Add havetxset to the cspell
dictionary since the lowercased metric name no longer word-splits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Continuation of the same default change applied on phase-6-statsd
(50d7047abe): the 5 dashboards introduced on phase-7 (formerly
system-*, renamed to bare names on this branch) also get their
service_name filter pre-selected to 'xrpld' instead of 'All'.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Change the pre-selected value of the service_name template variable from
'All' to 'xrpld'. On first load, dashboards now filter to service.name=xrpld
directly rather than showing every service the collector sees. The 'All'
option is still selectable — this only changes the default the widget
starts on.
Reduces the chance of a fresh dashboard link showing unrelated services
(e.g. non-xrpld exporters sharing the same Grafana Cloud stack) before
the user narrows the filter.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Continuation of the same rename applied on phase-6-statsd (commit
91c6582bd5): 5 more dashboards introduced on phase-7 (formerly the
system-* set, renamed to bare names by 2c590a47c5) still carried the
older 'Service' UI label. Aligning them with the rest.
Same reasoning as the phase-6 commit: the underlying Prometheus label
is `service_name`, and the UI label 'Service' misled users into typing
`service=xrpld` in Grafana Metrics Drilldown.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The template variable's Prometheus label name is `service_name` but the
UI display label read 'Service', which mismatched what users typed into
Grafana's Metrics Drilldown UI (which defaults to filter on a bare
`service` label). Rename the display label to 'Service Name' so it
unambiguously points at the underlying `service_name` Prometheus label.
Behaviour unchanged; template-variable name (`service_name`) and PromQL
label are the same. Only the widget label shown at the top of the
dashboard changes from 'Service' to 'Service Name'.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
exported_instance is a Prometheus scrape artifact that only exists when
Prometheus scrapes a target with a conflicting instance label. Users
running push-based pipelines (remote_write, OTLP) never get this label.
service_instance_id is always present via resource_to_telemetry_conversion
regardless of ingestion path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
exported_instance is a Prometheus scrape artifact that only exists when
Prometheus scrapes a target with a conflicting instance label. Users
running push-based pipelines (remote_write, OTLP) never get this label.
service_instance_id is always present via resource_to_telemetry_conversion
regardless of ingestion path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 3/4" token, keep the description.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 4" tokens, keep descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 3" tokens, keep descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 2" tokens, keep descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 1b" token, keep the filter
description.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- node-health.json: repoint the two "All Jobs" detail panels from the
__name__ regex over old xrpld_<job>_milliseconds_bucket names to the
native xrpld_job_running/queued_duration_us_bucket histograms, grouping
by the job_type label and legending on {{job_type}}.
- OTelCollector::callHooks: copy the hook list under mutex_ and invoke
handlers outside the lock. A handler can drop the last reference to an
OTelHookImpl, whose destructor calls removeHook() and re-acquires the
non-recursive mutex_ — invoking handlers under the lock could deadlock.
- ~OTelGaugeImpl: document that the SDK ObservableRegistry serializes
RemoveCallback against the Observe() callback pass with one mutex, so
callback removal is synchronous and gaugeCallback cannot run on a
dangling pointer after the destructor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Key Jobs execution/dequeue panels queried StatsD-era per-job metric
names (xrpld_<job>_milliseconds_bucket, xrpld_<job>_q_milliseconds_bucket)
that the native OTel path no longer emits. Phase 9's MetricsRegistry emits
job timings as two label-dimensioned histograms instead:
xrpld_job_running_duration_us_bucket{job_type="<job>"}
xrpld_job_queued_duration_us_bucket{job_type="<job>"}
Rewrite the 22 affected panel queries (11 job types x running/queued) to
the label-dimensioned form, preserving the histogram_quantile / rate /
sum-by structure and all template-variable filters.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- ValidationTracker: remove dead hard-trim Pass 1. The preceding
eviction loop already erases every reconciled entry older than the
late-repair cutoff using the same condition, so Pass 1 could never
match. The "drop any reconciled entry" fallback is retained.
- integration-test.sh: probe the StatsD port with a UDP-aware check
(ss -ulnp) instead of curl. StatsD is UDP-only on 8125, so the TCP
curl probe always reported "refused" — a false negative. Guarded by
a command -v ss check that logs a skip when ss is unavailable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
