Run with no arguments the script iterated an empty list, found no
violations and printed "OK: 0 dashboard(s) passed" with exit 0 -- a clean
bill of health for reading no files, indistinguishable from a real pass.
A bare run now defaults to every dashboard beside the script, and a run
that still ends up with nothing to check exits 2 rather than reporting
success. Passing paths explicitly behaves as before.
Nine targets across four panels filtered on xrpl_work_item while leaving
it out of their sum by() grouping. PromQL keeps only the labels listed in
by(), so the label was dropped from the result and the xrpl_ident legend
these panels build from it lost that segment.
Perf-iac runs stamp xrpl_work_item and give each work item its own set of
nodes, so service_instance_id already separates the runs; the visible
effect is the legend rather than merged series. Adding the label makes the
work item identifiable, which is the comparison these panels exist for.
NuDB Read Latency and NuDB Read Found Ratio each divide two aggregations.
Both sides get the label so their label sets stay equal and vector
matching still works.
The row was titled after the rollout phase its metrics came from, a number
defined only in a planning folder outside the shipped tree. Grafana shows
this title to operators, so it named something no reader could look up.
Two panel descriptions pointed at a rollout phase and task number defined
only in a planning folder outside the shipped tree. The note still names
the file and the change it is waiting on, which is the part a reader can
act on.
The catalog tagged every span with a rollout phase number defined only in
a planning folder outside the shipped tree, so the column meant nothing
to a reader of the repository. The span name and source file identify
each entry.
The section labels carried rollout phase numbers that are defined only in
a planning folder outside the shipped tree, so they meant nothing to a
reader of the repository. The descriptive half of each label already
identifies the section.
The label carried a rollout phase number defined only in a planning
folder outside the shipped tree. The descriptive half already identifies
the section.
These comments pointed at a planning folder and at its rollout phase
numbering, neither of which is part of the shipped tree, so the
references would dangle for any reader of the repository. Each comment
now states the fact it was pointing at.
The trace_state comment pointed at a planning document that is not part
of the shipped tree, so the reference would dangle for any reader of the
repository. State the reserved-and-inert fact on its own.
A repeating panel expands into one copy per network at view time and, with
maxPerRow=2, claims the whole row. The non-repeating panel paired beside it was
pushed to the next line but kept its stored x=12, so it rendered on the right
against an empty gap.
Two changes to how the layout is planned:
- a repeating panel gets a row to itself. It keeps w=12, so its copies still
tile two across inside that row.
- single-value panels are grouped to the top of each row section, so the
charts that follow pair with each other instead of being split up by an
interleaved repeat. Without this the gaps just become wasted half-rows.
Verified per dashboard: no panel lost, every targets block byte-identical, ids
1..N, no overlaps, and no row left with a gap on its left.
Every heatmap carried only `tooltip` and `yAxis`, missing `calculate`, `color`
and `cellGap`. Grafana's heatmap plugin treats those as required, and without
them the panel fails to initialise: the dashboard opens with "An error occurred
within the plugin" rather than a chart.
The option values are taken from the one heatmap in this stack that does render
on Grafana Cloud (ledger-sync-health): calculate=false since the queries already
return histogram buckets, the Turbo 64-step scheme, and cellGap=1. Each panel
keeps its own yAxis label, unit and tooltip settings.
This is a long-standing defect rather than fallout from the recent layout work -
the same options are absent in origin/phase9 and in the cloud copies that were
already live.
Two problems showed up once these dashboards were live on Grafana Cloud.
Panels were too short. Charts at h=8 clipped their legends mid-row, and stats
at h=4 were cramped. Every visualisation is now h=10, with tables and logs at
h=12; the log-derived-insights instruction banner keeps h=12 for its prose.
Uniform height also means any two panels can pair side by side.
Repeat on a state-timeline broke the dashboard outright: ledger-data-sync
failed to open on Cloud at v45 and had to be restored to v44. Repeat is now
limited to single-value panels (stat, gauge, bargauge, table). Charts show
their networks as separate series instead, which is what a chart is for.
Repeat was also sticky: normalization only ever added the keys, so a panel that
picked them up in an earlier pass kept them even after its type stopped being
eligible. The keys and the title suffix are now removed from ineligible panels,
which is what actually cleared the two timeline panels here.
Verified per dashboard: no panel lost, every targets block byte-identical, ids
1..N, no grid overlaps, and repeat present only on stat/gauge/bargauge.
Guideline 8 asks for gauges and stats at the top. Seven dashboards had them
scattered below charts, so the reader met a wall of time series before the
at-a-glance numbers that give those series context.
Stats, gauges and bar gauges now come first within each row section. The move
is deliberately scoped to inside a section: shifting a panel across a row
boundary would change which category it belongs to. Panels keep their relative
order otherwise, so the reading sequence within each group is unchanged.
Whole panel objects are cut and re-spliced as raw text, so their contents stay
byte-identical and only gridPos and id are recomputed. Verified per dashboard:
panel count unchanged, no panel lost, every targets block byte-identical, ids
still 1..N, and no row section left with a stat below a chart.
These dashboards were hand-authored over time and had drifted apart: panel
heights spanned ten different values, nine dashboards had no row grouping,
line-chart styling was inconsistent, and no panel carried an id, so Grafana
assigned them positionally at load and every panelId deep link was only as
stable as the panel order.
Per panel, in document order:
- id written as 1..N so panelId links address a specific panel
- gridPos quantized to at most two panels across: charts h=8,
stats/gauges h=4, tables/logs h=12 full width. Panels are
paired only with an equal-height neighbour, so no row is left
with a ragged half-empty cell.
- line charts lineWidth=1, fillOpacity=0, pointSize=5, gradientMode=none
- repeat xrpl_network_type (horizontal, maxPerRow=2) with a
[$xrpl_network_type] title suffix, on the panel types where
overlaying two networks in one panel reads as noise
(stat/gauge/bargauge/table/state-timeline). Line charts keep
their networks as separate series, which is the point of a
line chart.
- decimals 0 where the value counts discrete things (threads, peers,
queue depths); a fractional thread count is meaningless.
- rows category rows added where a dashboard had none
Panels whose legend sits on the right stay full width: a side legend needs the
horizontal room, and squeezing it to half width clips the series names.
Edits were made as raw-text replacements, not a json.dump round-trip, so
formatting and escaping of untouched lines are byte-identical. Verified per
dashboard: panel count unchanged, every targets block byte-identical, all
descriptions unchanged, ids exactly 1..N, and no two panels overlapping on the
grid. The repo dashboard lint and the OTel naming check both pass.
The Transaction Overview panel "Queue Bypass Ratio (Direct Apply vs
Enqueue)" reported a confident 0.50 on every node while the true bypass
rate was zero. The two spans it divided are not disjoint alternatives:
txq.apply_direct is a child of txq.enqueue. TxQ.cpp creates the
apply_direct span as the first statement of tryDirectApply(), ahead of
the account, sequence and fee-level guards, and tryDirectApply() is
called from inside the live enqueue scope. The span therefore counts
attempts, so the denominator direct + enqueue counts each transaction
twice and pins the ratio to one half algebraically.
Measured on a four-node fleet: 6082443 direct against 6082877 enqueue
over the same population, panel output 0.5000170 on three nodes and
0.5000000 on the fourth. Grouping txq.enqueue by txq_status over seven
days returns only "rejected" -- no transaction has ever taken the
direct-apply path.
Remove the panel rather than repoint it. A correct expression using
txq_status as the disjoint discriminator would render permanently
empty on this fleet, which reads no better than a wrong number.
Widen the band partner "TxQ Enqueue Rate by Transaction Type" from 12
to 24 columns so the y=48 band still fills the grid. Every band in all
ten dashboards sums to 24 columns; leaving a half-width hole would be
the only exception. Panel order and every other panel's position,
width and height are unchanged.
The runbook already listed txq.apply_direct as available but not
paneled, so that row becomes accurate. Rows describing the span itself
are untouched -- the span and its metric are unchanged.
Panels 21 (NuDB Read Latency) and 23 (NuDB Read Found Ratio) on the
Ledger Data & Sync dashboard guarded their divisor with
clamp_min(<denominator>, 1). clamp_min raises the value, not just the
zero case, so any node reading fewer than 1 block per second was
divided by a fabricated 1 instead of by its real read count.
Replace the clamp with the filter (<denominator> > 0). A comparison
without the bool modifier drops the sample rather than rewriting it, so
these panels now show no data instead of a wrong number.
Measured over 7 days: five nodes fall below 1 read/s. On validator-0 the
clamp reported 2.726 us/read against a true 5.493, and on nonUNLmalloc-tc
it reported 0 us/read, which cannot occur. The error is largest exactly
when panel 21 is used as the bottleneck discriminator during a stall,
because that is when the read rate collapses toward zero.
Matches the existing idiom on the same nodestore_state metric family in
the NodeStore Write vs Read Latency panel.
Moves off the v2.53 LTS line, which this branch pinned, onto the current
v3 release, and corrects the accompanying comment that named the old line.
Verified against the new image rather than assumed, since this crosses a
major version:
- prometheus.yml passes `promtool check config` under v3.13.2
- all 391 unique dashboard PromQL queries parse under the v3 parser
- all 13 alert-rule expressions parse under the v3 parser
The config uses none of the surfaces v3 changed: no remote_write, no
holt_winters, no offset modifiers.
Bumps the three images this branch pins:
otel-collector-contrib 0.121.0 -> 0.158.0
tempo 2.7.2 -> 2.9.4
grafana 11.5.2 -> 13.1.2
Verified locally against the new images: the collector config passes
`validate` under 0.158.0 and tempo.yaml passes `-config.verify=true` under
2.9.4, both unchanged. The collector's log path uses the generic otlphttp
exporter (otlphttp/loki), not the dedicated loki exporter removed upstream in
v0.147.0, so the pipeline is unaffected by that removal.
Grafana crosses two majors. Operator-visible consequences are handled on the
branches that own the affected files: Grafana 13 enables the renderAuthJWT
feature toggle by default, so the image renderer now requires a matching
renderer_token on both the server and the renderer container.
