Add a Keywords definition-list and a References link line to 178 Grafana
panel descriptions across all 14 dashboards, defining the XRPL/rippled
domain terms each panel uses (tiers 1-9: ledger, consensus, transaction
pipeline, fees/queue, node state, peer/overlay, storage, validator,
RPC/pathfinding). Cross-cutting chart terms and job-queue internals are
intentionally excluded.
Add docs/telemetry-glossary.md (86 terms, 9 categories) as the deeper
reference, linked from each panel's References line and from the runbook.
Keywords are injected only where a term appears in that panel's prose
(matched over description text, not source-file citations).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "Ledger Close Duration" panel measured the consensus.ledger_close span
duration, which only wraps the sub-millisecond onClose() prologue — not the
ledger close. Repoint it to the consensus.round span (full round, open to
accept) so it reflects actual close time (~3-5s on mainnet). The consensus_mode
filter is preserved (consensus.round carries that attribute, verified live).
The sibling rate panels (Consensus Mode Over Time, Accept vs Close Rate,
Validation vs Close Rate) keep using consensus.ledger_close: a rate of that
span is a valid per-close event counter, only its duration was wrong.
Runbook updated to match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "ledger close time" was mis-derived and partly un-queryable:
- Build vs Close Duration derived close time from the consensus.ledger_close
span, which only wraps the onClose() prologue (~0.8ms live) — not the close.
Repoint the close series to consensus.round (full round, live P95 ~4.8s).
- The network close-time value (close_time) lived only as a span attribute,
un-queryable in Prometheus and unfit as a spanmetrics label (monotonic
timestamp -> unbounded cardinality). Expose it as last_close_time on the
existing server_info observable gauge (native gauge, no new instrument).
- Add a "Ledger Close Interval & Age" panel to ledger-operations and
node-health: interval = 1/rate(ledgers_closed_total) (counter-based,
scrape-independent); age = time() - (last_close_time + epoch_offset).
A gauge delta is deliberately NOT used for the interval — a timestamp gauge's
delta aliases to the scrape period, not the close cadence (verified live).
Guardrail comments in both collector configs record why close_time must never
become a spanmetrics dimension. Docs (09-reference) and the operator runbook
updated to match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sync the Node Health dashboard with panels and field settings refined in
Grafana, and document the ledger-convergence panels:
- Add descriptions for "Validated Ledger Seq — Current (Stat)" (per-node lag
behind the network tip) and "Validated Ledger Seq — Convergence (Max − Min)"
(seq spread across nodes), in the standard 7-field panel format.
- Peer Count: threshold colours (red < 5, yellow < 10, green >= 10) so a low
peer count reads red rather than using a value gradient.
- Minor field-config fixes on Last Close and NuDB Storage panels (axis label,
threshold cleanup) as adjusted in Grafana.
Add the Grafana duration unit "dthms" to the cspell dictionary. Panel content
stays at parity with the Grafana Cloud copy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Regroup all 47 panels under correct subsystem rows and fix the broken row
structure. Previously the "NodeStore I/O" row had no gridPos and collided at
y=32 with an appended "Extended Metrics" catch-all row, so NodeStore I/O
rendered empty and its panels landed under the wrong header.
- Rows: Overview (top), Operating Mode, NodeStore I/O, Jobs, Caches, Server
Info, Complete Ledgers & DB, Ledger Economy. Drop the stray "Extended
Metrics" row; each panel now sits under its subsystem.
- Normalize panel heights to three buckets: 12 (standard), 16 (emphasis /
detail plots), 24 (Operating Mode state timeline). Widths stay half (12) or
full (24).
- Contiguous, non-overlapping gridPos; all 47 panels preserved.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rework the legend identity bracket across all 14 dashboards so it reads
"<series> [instance, branch, work_item]" (comma-separated, Node Role dropped)
and collapses cleanly when perf-iac labels are absent -- "<series> [instance]"
with no empty "[, ]" gaps.
- Build one xrpl_ident label per query via label_join + a single label_replace
that trims leading/trailing empty segments and brackets the result. Done at
the outermost (post-aggregation) level so it survives each panel's by() clause.
- Add service_instance_id to the by() of 6 panels that grouped only by node_role,
so same-branch validator/peer series stay distinguishable after dropping the
node_role legend token.
- Fold the two legendFormat two-bracket panels (Transaction Results by Type,
Tx Apply Pipeline Latency by Type and Stage) into the same single-bracket form.
- Rename the annotation query label "Perf runs (perf-iac)" to
"Annotate perf-iac runs".
- Revert dashboard-level descriptions to plain text (the folder-list info tooltip
escapes HTML, so <br>/markdown render literally).
Verified: output byte-identical to the prior multi-call cleanup over a 2-day
live perf window; all 14 files valid JSON.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Group the node + perf-iac identity labels inside one pair of brackets in the
legend Display name so a series reads e.g. "Queue Depth [validator-0
test:pr:abc123 validator RIPD-7455]" on a perf node and "Queue Depth
[aws-dev-xrpl-1]" on a plain node. Absent perf-iac labels collapse to a single
space before the closing bracket; no empty "[]", no data impact. Covers the
eleven dashboards this branch owns.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The legend now uses raw labels in the dashboard Display name (see the paired
dashboard commit), so the collector no longer needs to build resource_group /
iac_group. Those derived labels also could not survive the dashboards' sum by()
aggregation, which is why the grouped approach was dropped.
Revert transform/legendgroups and its transform/cloudlabels datapoint copies
from the Grafana Cloud collector config and the in-repo Alloy config; both files
are now identical to their pre-change state.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The collector-built resource_group / iac_group labels did not survive the
dashboards' own PromQL aggregation: panels that sum by(service_instance_id,
xrpl_branch, xrpl_node_role, ...) drop any label not in the by() clause, so the
derived group labels were aggregated away and the Display name rendered empty.
Switch the legend Display name to the raw labels the panels already group by,
which are always present by construction:
${__field.labels.series} ${__field.labels.service_instance_id}
${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role}
${__field.labels.xrpl_work_item}
Absent labels render empty (no brackets, no gaps). Covers the eleven dashboards
this branch owns. The matching collector-config revert lands on the branches
that introduced those configs (phase-7 local, phase-10 cloud/alloy).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The collector-built resource_group / iac_group labels did not survive the
dashboards' own PromQL aggregation: panels that sum by(service_instance_id,
xrpl_branch, xrpl_node_role, ...) drop any label not named in the by() clause,
so the derived group labels were aggregated away and the Display name rendered
empty ("Observing" with no node). Building them in the collector also coupled
every dashboard to a lockstep collector redeploy on every node.
Switch the legend Display name to reference the raw labels the panels already
group by, which are always present by construction and need no collector
support:
${__field.labels.series} ${__field.labels.service_instance_id}
${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role}
${__field.labels.xrpl_work_item}
${__field.labels.X} renders empty for a label a series lacks (verified in
Grafana), so dev/mainnet nodes show just "<name> <node>" and perf nodes add the
branch/role/work-item, with no empty brackets and no double spaces. Verified
live across timeseries, stat, bargauge, piechart, gauge and cross-node
aggregation panels.
Revert the transform/legendgroups processor from the local collector config;
this file is now identical to its pre-change state.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update the config.alloy metric_statements comment to describe the current
bracket-baked, ", "-joined group-label values ("[node, mainnet]"), matching the
implementation. Comment-only; no behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror the local collector change on the Grafana Cloud collector config and the
in-repo Alloy config: build the group-label VALUES with brackets and comma
separators (resource_group="[node, mainnet]",
iac_group="[branch, role, work-item]") in both trace and metric resource
statements, so span-derived and native metrics carry the same bracketed values.
This lets the dashboards use a bracket-free Display name and avoids the empty
"[]" that a templated bracket would leave when a group is absent (Grafana's
renameByRegex runs pre-interpolation and cannot strip it). Separator changes
from "-" to ", ".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Display name templated the brackets: "${series} [${resource_group}]
[${iac_group}]". When a group label was absent the "${}" rendered empty,
leaving a literal "[]" in the legend, and the cleanup renameByRegex could not
remove it (Grafana runs that transform on the pre-interpolation template, which
has no "[]" -- it only appears after interpolation at render time).
Bake the brackets and comma separators into the group-label VALUES in the
collector instead (resource_group="[node, mainnet]",
iac_group="[branch, role, work-item]"), so the Display name is bracket-free
("${series} ${resource_group} ${iac_group}") and an absent group renders as
nothing. Separator changes from "-" to ", ". Drop the dead cleanup transform.
Covers this branch's local collector config and its three dashboards
(ledger-data-sync, network-traffic, overlay-traffic-detail).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Display name templated the brackets: "${series} [${resource_group}]
[${iac_group}]". When a group label was absent (e.g. iac_group on a non-perf
node) the "${}" rendered empty, leaving a literal "[]" in the legend. The
empty-bracket cleanup renameByRegex could not remove it: Grafana runs that
transform on the pre-interpolation Display name template, which contains no
"[]" -- the "[]" only appears after variable interpolation, at render time.
Fix it at the source: the collector now bakes the brackets and comma separators
into the group-label VALUES (e.g. resource_group="[node, mainnet]",
iac_group="[branch, role, work-item]"). The Display name becomes bracket-free --
"${series} ${resource_group} ${iac_group}" -- so an absent group is simply an
absent label and renders as nothing. Also switches the intra-group separator
from "-" to ", " (e.g. "[node, mainnet]"). Drop the now-dead cleanup transform
from every panel; value-mapping renameByRegex rules are preserved.
Collector-side label building lives in the paired commits on the collector
config branches. This commit is the dashboard half for the eleven dashboards
this branch owns.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Panels that group by a distinguishing dimension (consensus_mode, tx_type,
command, method, stage, ...) already fold that dimension's value into the
curated series name via label_replace(..., "series", "<prefix> $1", "<dim>",
"(.*)"). The migration also added a separate [${__field.labels.<dim>}] bracket,
so those legends rendered the value twice, e.g. "proposing [proposing] [...]"
on Consensus Mode Over Time.
Remove the dimension bracket from all 29 affected panel Display names, leaving
${__field.labels.series} [resource_group] [iac_group]; the series name already
conveys the dimension. The migration script no longer emits the bracket either.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In the Alloy config the spanmetrics connector derives span_calls_total /
span_duration_* from the tagged TRACES and sends them straight to the batch
processor, bypassing the metric resource statements. The grouped legend labels
were built only in metric_statements, so span-derived RED metrics carried no
resource_group / iac_group and span-based dashboard panels rendered empty group
legends.
Build the same labels in trace_statements too (mirroring how the tier attrs are
already set in both), so spans carry them before spanmetrics derives from them.
The reference otel collector is unaffected: there spanmetrics is a connector
feeding the metrics pipeline, which runs transform/legendgroups on all inputs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The docker/telemetry/alloy/config.alloy node config feeds Grafana Cloud via the
OTLP gateway (which promotes resource attributes to labels server-side). Add the
resource_group / iac_group build to its metric resource statements so nodes
using this Alloy path get the same grouped legend labels the dashboards expect.
This is a generic node config (no perf-iac attrs), so iac_group's guard never
fires here and only resource_group is built; the iac_group statement is kept for
parity and activates automatically on a perf-comparison node. Guards check != ""
as well as != nil since OTTL Concat emits a stray separator for an empty element.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Eighteen timeseries panels in consensus-health and transaction-overview
carried tooltip maxHeight 500; the dashboard guideline specifies 600. Normalize
them so every panel's tooltip matches Mode:All / Sort:Desc / Max Height:600.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "Overlay Traffic by Category (Bytes In)" bargauge names each bar by
category via 22 byName field-override displayNames (Transactions, Proposals,
...). The legend migration had also added a generic
${__field.labels.series} [...] Display name, which competed with those
curated names. Remove the generic Display name (and its now-unneeded
empty-bracket cleanup transform) from this panel so the category names stand;
this panel's identity is the traffic category, not the node/perf-iac grouping.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror the local-config guard hardening on the Grafana Cloud collector config:
add != "" to the transform/legendgroups presence guards so an empty-string
perf-iac attribute (work_item | default('') on a no-work-item run) does not
produce iac_group="--". Empty-string sources now leave the group unset.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The transform/legendgroups guards checked only != nil. The perf-iac collector
stamps xrpl.branch / xrpl.node.role / xrpl.work.item with an empty-string
default (work_item | default('')) when a run has no work item, and an empty
string is not nil. OTTL Concat renders a stray separator for an empty element,
so an all-empty perf-iac stamp would have produced iac_group="--".
Add != "" to every presence guard (iac_group sources, service.instance.id,
xrpl.network.type) so a group is built only from genuinely-present values and
left unset otherwise. Verified on a live collector: empty-string perf-iac attrs
leave iac_group absent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the transform/legendgroups processor (introduced on phase-7 for the local
config) to the dual-export grafanacloud collector config, and copy the two
group labels onto datapoint attributes in transform/cloudlabels.
The local Prometheus scrape promotes resource attributes to labels via
resource_to_telemetry_conversion, but Grafana Cloud's OTLP ingest does not, so
resource_group / iac_group must be copied onto datapoint attributes the same
way service_instance_id / deployment_environment / xrpl_network_type already
are. legendgroups runs before cloudlabels in both metrics pipelines so the
built resource attributes are available to copy; the copies are presence-guarded
so an absent group is not created as an empty datapoint label.
Verified on a live collector (debug exporter): a perf node emits datapoint
attributes resource_group and iac_group; a non-perf node emits resource_group
only, with iac_group correctly absent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Apply the collector-built grouped-label legend approach (introduced on
phase-7) to the eleven dashboards this branch and its ancestors own:
consensus-health, fee-market, job-queue, ledger-operations, node-health,
peer-network, peer-quality, rpc-pathfinding, rpc-performance,
transaction-overview, validator-health.
Per legend panel: drop the 11-rule renameByRegex chain and legendFormat
"__auto"; set a Standard-option Display name referencing the pre-built
resource_group / iac_group labels plus, when the panel groups by a
distinguishing dimension (e.g. command, tx_type, consensus_mode, stage,
method), that dimension as a leading bracket; keep a single generic
empty-bracket cleanup rule; set tooltip maxHeight to 600.
Value-mapping renameByRegex rules (e.g. consensus close-time
increased->Coarser) are preserved. Only labels a panel groups by appear in
its legend; filter-only selectors ($command with no by(command)) narrow data
but do not become legend groups. All 14 dashboards validate; no legend
renameByRegex chain remains.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-panel legend used legendFormat "__auto" reshaped by an 11-rule
renameByRegex chain. __auto sorts labels alphabetically and cannot group,
so the chain existed only to bolt grouping onto its output -- unmaintainable
and non-compliant with the dashboard legend guideline.
Build the grouped labels once at ingestion instead. A new transform/legendgroups
processor derives two resource attributes, promoted to Prometheus labels by
resource_to_telemetry_conversion:
- resource_group: service_instance_id[-xrpl_network_type]
- iac_group: xrpl_branch-xrpl_node_role-xrpl_work_item (perf runs only)
Each is left unset when its sources are absent, so Prometheus drops the empty
label and the legend shows nothing for it. Concat is presence-guarded because
OTTL Concat renders "<nil>" for an absent attribute.
Dashboards now use a Standard-option Display name that references the pre-built
labels (${__field.labels.series} [resource_group] [iac_group]); ${__field.labels.X}
renders empty for a missing label, so absent groups vanish. A single generic
renameByRegex strips the resulting empty "[]" bracket. legendFormat is dropped
(Display name supersedes it). Tooltip maxHeight set to 600 per guideline.
Applied here to the three dashboards this branch introduced (ledger-data-sync,
network-traffic, overlay-traffic-detail); value-mapping renameByRegex rules are
preserved. Verified end-to-end on a live local stack (collector -> Prometheus ->
Grafana): perf and dev nodes render correct, clutter-free legends.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The docker/telemetry/grafanacloud/*.json files are local deploy variants
(they pin the Grafana Cloud datasource UID) and are regenerated from the
canonical grafana/dashboards/*.json. Untrack them and gitignore so they
stay local-only and never drift in version control.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Panel legends only showed the node (service_instance_id); the perf-iac
filter dimensions (xrpl_branch, xrpl_node_role) never appeared, so
baseline-vs-test series were indistinguishable.
Each node-referencing target injects its curated name via
label_replace(...,"series",...) and uses the "__auto" legend, which shows
only labels that vary across the displayed series. A word-boundary-anchored
renameByRegex chain reshapes the auto output into
"Name [node, role]-[branch, work_item]", collapsing empty groups so a
dimension appears only when the filter widens it. Aggregating queries gain
xrpl_branch/xrpl_node_role in their by() so those labels survive; multi-
dimension legends keep their rich label set and append branch/role.
Quantile panels surface the quantile via the panel title (q$quantile),
since {{quantile}} is a dashboard variable, not a series label.
Timeseries panels connect only null gaps shorter than 30m
(spanNulls=1800000): brief scrape gaps bridge, genuine outages stay broken.
Non-node panels (heatmaps, rows, tables, state timelines, category/bucket
series) are left unchanged. Applied to all 14 dashboards.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Panel legends only showed the node (service_instance_id); the perf-iac
filter dimensions (xrpl_branch, xrpl_node_role) never appeared, so
baseline-vs-test series were indistinguishable.
Each node-referencing target injects its curated name via
label_replace(...,"series",...) and uses the "__auto" legend, which shows
only labels that vary across the displayed series. A word-boundary-anchored
renameByRegex chain reshapes the auto output into
"Name [node, role]-[branch, work_item]", collapsing empty groups so a
dimension appears only when the filter widens it. Aggregating queries gain
xrpl_branch/xrpl_node_role in their by() so those labels survive; multi-
dimension legends keep their rich label set and append branch/role.
Quantile panels surface the quantile via the panel title (q$quantile),
since {{quantile}} is a dashboard variable, not a series label.
Timeseries panels connect only null gaps shorter than 30m
(spanNulls=1800000): brief scrape gaps bridge, genuine outages stay broken.
Non-node panels (heatmaps, rows, tables, state timelines, category/bucket
series) are left unchanged. Applied to all 14 dashboards.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
