Dashboard audit against the live datasource surfaced two broken panel queries
and applied UX polish across all 14 dashboards.
- node-health "Job Queue Depth": job_count -> jobq_job_count. The JobQueue
collector is wrapped in group("jobq") (Application.cpp), so the registered
job_count gauge is emitted with the jobq_ prefix; the panel queried the
unprefixed name and returned nothing.
- network-traffic "Overlay Traffic by Category" + "All Traffic Categories":
topk(N, rate({__name__=~".*_bytes_in"}[...])) errors on Mimir ("vector
cannot contain metrics with the same labelset") because rate() drops
__name__ and the many counters collapse. Replaced with an enumerated
label_replace form that re-attaches __name__ per metric, preserving the
{{__name__}} legend and per-series display-name overrides.
- All 14 dashboards: refresh set to 10s.
- peer-quality: each panel full screen width.
- validator-health: at most two panels per row (row headers preserved).
- docs: telemetry-runbook and 06-implementation-phases updated for the
jobq_ prefix and the network-traffic query pattern.
Verified end-to-end against a local mainnet xrpld node feeding the local
stack: jobq_job_count returns data (old job_count empty), both network-traffic
exprs execute (old form reproduces the labelset error), and panels render
through the Grafana proxy. All 14 pass validate_dashboards.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The phase-6 change replaced this piechart with the applied-ratio state
timeline, but phase-9's independently-diverged dashboard kept its own
copy (id 17) which survived the merge-forward. Its query filtered on the
$txq_status template variable, which was removed alongside the swap, so
the panel resolved to txq_status=~"" and rendered empty. Delete it so
only the state timeline remains.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the TxQ Accept Status piechart on the Transaction Overview
dashboard with a state-timeline showing each node's applied fraction of
TxQ accepts (applied / applied+failed) over time, colored by threshold
(green >=0.9, yellow >=0.7, red below). Remove the now-orphaned
txq_status template variable (the piechart was its only consumer) and
document the panel in the telemetry runbook.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The phase8->phase9 merge resolution took phase9's side of transaction-overview,
which reverted the Transaction Apply Failed Rate panel to the old
tx.apply/STATUS_CODE_ERROR query (constant zero). Re-apply the tx.transactor
stage=apply / ter_result!=tesSUCCESS repoint and raised thresholds.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add an Alloy-based collector config as an alternative to the reference
OpenTelemetry Collector. It forwards xrpld OTLP traces and native metrics
to the Grafana Cloud OTLP gateway and, critically, runs a spanmetrics
connector so span-derived RED metrics (span_calls_total,
span_duration_milliseconds_*) are produced. Those metrics are not emitted
by xrpld; they are derived from spans, so an Alloy node without this
connector is missing from every span-based dashboard.
All credentials, endpoints, usernames and the per-node host label are read
from the environment (no secrets in the file). Adds a matching
.env.grafanacloud-alloy.example and gitignore exception, and the otelcol
term to the cspell dictionary.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Observe the overflow total from Overlay::getJqTransOverflow() via an
ObservableCounter instead of the unused push helper (which had no call
site, so the metric read flat zero). Keeps the increment path in the
overlay-owned code and removes the dead incrementJqTransOverflow() helper.
Update the job-queue panel to reflect the now-live metric, and refresh
stale xrpld_-prefixed metric names in the header comments.
NOTE: committed without a local build per instruction; needs compile +
clang-tidy verification before push.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace hardcoded [5m] rate windows with [$__rate_interval] inside
rate()/increase() on plain-counter panels so ops/s panels resample
correctly on zoom-out instead of under-sampling or flat-lining on
short windows. Histogram _bucket windows keep their fixed [5m] span.
Set Min step ("interval": "15s") on every rate target to match the
Prometheus 15s scrape interval.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The spanmetrics connector had no namespace, so it emitted traces_span_metrics_*
metric names by default. The span dashboards and docs are renamed to query
span_* names; this is only correct if the connector emits them too, so add
namespace: "span" to the spanmetrics connector. Both sides change together:
renaming the dashboards without the namespace (or vice versa) would break the
pipeline. Matches the phase9 collector config.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The phase7 OTelCollector::formatName lowercases and strips names, emitting
snake_case metrics with no xrpld_ prefix. The native Grafana dashboards and
the telemetry docs still queried the old xrpld_CamelCase names, so they were
broken against their own pipeline. Rename every metric name to match what the
code emits: drop the xrpld_ prefix and lowercase the remainder. The two job
histograms also drop the redundant 'duration' word (job_queued_us,
job_running_us) to match the phase9 forms. Add havetxset to the cspell
dictionary since the lowercased metric name no longer word-splits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes the default change applied on phase-6-statsd and
phase-7-native-metrics. The four dashboards introduced on phase-9
(validator-health, peer-quality, fee-market, job-queue) now also
pre-select 'xrpld' rather than 'All' for the service_name filter.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Continuation of the same default change applied on phase-6-statsd
(50d7047abe): the 5 dashboards introduced on phase-7 (formerly
system-*, renamed to bare names on this branch) also get their
service_name filter pre-selected to 'xrpld' instead of 'All'.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Change the pre-selected value of the service_name template variable from
'All' to 'xrpld'. On first load, dashboards now filter to service.name=xrpld
directly rather than showing every service the collector sees. The 'All'
option is still selectable — this only changes the default the widget
starts on.
Reduces the chance of a fresh dashboard link showing unrelated services
(e.g. non-xrpld exporters sharing the same Grafana Cloud stack) before
the user narrows the filter.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Grafana Cloud's OTLP ingest already promotes the `service.name` resource
attribute to a `service_name` metric label. The `transform/cloudlabels`
processor was ALSO setting `service_name` as a datapoint attribute,
which Mimir treats as a duplicate label and concatenates as
`service_name="xrpld;xrpld"` on every series. Dashboards filtering on
`service_name=xrpld` then returned no data.
Remove the redundant `set(attributes["service_name"], ...)` line and
add a comment documenting the doubling. `service.instance.id` does not
exhibit the same behaviour, so that copy is retained for the $node
filter to keep resolving.
Reintroduces the pre-e76eed5296 shape (which was correct on this label)
without reverting that commit's other cloud-label fixes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Completes the rename applied on phase-6-statsd and phase-7-native-metrics.
These four dashboards were introduced on phase-9: peer-quality and
validator-health via b92354715d, fee-market and job-queue via 25868f2740.
Same reasoning: the underlying Prometheus label is `service_name`,
so the UI display label should say 'Service Name' rather than the
ambiguous 'Service'.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
