Carries the layout fix to the dashboards this branch owns. A repeating panel
claims the whole row once it expands per network, so the non-repeating panel
paired beside it was pushed down while keeping its stored x=12 - rendering on
the right with an empty gap on its left.
Repeating panels now get a row to themselves, keeping w=12 so their copies still
tile two across. Single-value panels are grouped to the top of each row section
so the charts that follow pair with each other rather than being separated by an
interleaved repeat.
Verified against origin/phase9: no panel lost, every targets block unchanged,
ids 1..N, no overlaps, no rows with a left-hand gap.
A repeating panel expands into one copy per network at view time and, with
maxPerRow=2, claims the whole row. The non-repeating panel paired beside it was
pushed to the next line but kept its stored x=12, so it rendered on the right
against an empty gap.
Two changes to how the layout is planned:
- a repeating panel gets a row to itself. It keeps w=12, so its copies still
tile two across inside that row.
- single-value panels are grouped to the top of each row section, so the
charts that follow pair with each other instead of being split up by an
interleaved repeat. Without this the gaps just become wasted half-rows.
Verified per dashboard: no panel lost, every targets block byte-identical, ids
1..N, no overlaps, and no row left with a gap on its left.
The GetObject Request Size Distribution heatmap, added on this branch, was
missing calculate, color and cellGap. Grafana's heatmap plugin needs them, and
without them the whole dashboard opens with "An error occurred within the
plugin" instead of rendering.
Same values as the other four heatmaps fixed upstream, taken from the one that
renders correctly on Grafana Cloud. The panel keeps its own axis label and unit.
Every heatmap carried only `tooltip` and `yAxis`, missing `calculate`, `color`
and `cellGap`. Grafana's heatmap plugin treats those as required, and without
them the panel fails to initialise: the dashboard opens with "An error occurred
within the plugin" rather than a chart.
The option values are taken from the one heatmap in this stack that does render
on Grafana Cloud (ledger-sync-health): calculate=false since the queries already
return histogram buckets, the Turbo 64-step scheme, and cellGap=1. Each panel
keeps its own yAxis label, unit and tooltip settings.
This is a long-standing defect rather than fallout from the recent layout work -
the same options are absent in origin/phase9 and in the cloud copies that were
already live.
Carries the sizing fix through to the dashboards this branch owns, after the
live Cloud copies showed charts clipping their legends at h=8 and stats too
cramped at h=4.
Every visualisation is h=10; tables and logs h=12; the log-derived-insights
instruction banner h=12 for its prose. A single height per visualisation also
means any two panels pair cleanly side by side.
Repeat stays on single-value panels only. The five state-timeline and heatmap
panels that had picked it up in an earlier pass are cleared, since repeating a
timeline broke ledger-data-sync on Cloud.
Verified against origin/phase9: no panel lost, every targets block unchanged,
ids 1..N, no grid overlaps.
Two problems showed up once these dashboards were live on Grafana Cloud.
Panels were too short. Charts at h=8 clipped their legends mid-row, and stats
at h=4 were cramped. Every visualisation is now h=10, with tables and logs at
h=12; the log-derived-insights instruction banner keeps h=12 for its prose.
Uniform height also means any two panels can pair side by side.
Repeat on a state-timeline broke the dashboard outright: ledger-data-sync
failed to open on Cloud at v45 and had to be restored to v44. Repeat is now
limited to single-value panels (stat, gauge, bargauge, table). Charts show
their networks as separate series instead, which is what a chart is for.
Repeat was also sticky: normalization only ever added the keys, so a panel that
picked them up in an earlier pass kept them even after its type stopped being
eligible. The keys and the title suffix are now removed from ineligible panels,
which is what actually cleared the two timeline panels here.
Verified per dashboard: no panel lost, every targets block byte-identical, ids
1..N, no grid overlaps, and repeat present only on stat/gauge/bargauge.
Applies the guideline-8 ordering to the two dashboards introduced on this
branch, and clears the last two labelling gaps found while auditing the set:
- peer-quality, validator-health: stats and bar gauges now lead their row
section instead of trailing the charts, matching the other dashboards.
- node-health "Peer Disconnects (Resources)": had an empty axisLabel, the
only chart in the set without one. Now reads "Disconnects".
- rpc-performance row "Aggregate RPC (all commands)" -> "(All Commands)",
the one title that was not in Title Case.
Reordering stays inside each row section so no panel changes category, and the
panel objects are cut and re-spliced as raw text, so their contents are
byte-identical. Verified against origin/phase9: no panel lost, every targets
block unchanged, ids 1..N, and every row section now leads with its
single-value panels.
Guideline 8 asks for gauges and stats at the top. Seven dashboards had them
scattered below charts, so the reader met a wall of time series before the
at-a-glance numbers that give those series context.
Stats, gauges and bar gauges now come first within each row section. The move
is deliberately scoped to inside a section: shifting a panel across a row
boundary would change which category it belongs to. Panels keep their relative
order otherwise, so the reading sequence within each group is unchanged.
Whole panel objects are cut and re-spliced as raw text, so their contents stay
byte-identical and only gridPos and id are recomputed. Verified per dashboard:
panel count unchanged, no panel lost, every targets block byte-identical, ids
still 1..N, and no row section left with a stat below a chart.
Applies the same normalization the phase-7-owned dashboards received, to the
five introduced on this branch: fee-market, job-queue, log-derived-insights,
peer-quality and validator-health.
- id written as 1..N so panelId deep links address a specific panel
rather than whichever panel happens to sit in that position
- gridPos at most two panels across; charts h=8, stats/gauges h=4,
tables/logs h=12 full width. Panels pair only with an
equal-height neighbour, so no row keeps a ragged empty cell.
validator-health and peer-quality were stacked single-file at
full width, which is why they were the longest to scroll.
- line charts lineWidth=1, fillOpacity=0, pointSize=5, gradientMode=none
- repeat xrpl_network_type (horizontal, maxPerRow=2) with a
[$xrpl_network_type] title suffix on stat/gauge/bargauge/
table/state-timeline panels, where two networks overlaid in
one panel reads as noise. Line charts keep their networks as
separate series.
- decimals 0 on panels counting discrete things (peers, quorum,
connection mix) - a fractional peer count is meaningless.
- rows category rows added to fee-market, job-queue and peer-quality
Panels with a right-hand legend stay full width; a side legend needs the room.
Raw-text edits, so untouched lines keep their formatting and escaping byte for
byte. Verified per dashboard against origin/phase9: panel count unchanged, no
panel lost, every targets block byte-identical, descriptions unchanged, ids
exactly 1..N, and no two panels overlapping on the grid.
These dashboards were hand-authored over time and had drifted apart: panel
heights spanned ten different values, nine dashboards had no row grouping,
line-chart styling was inconsistent, and no panel carried an id, so Grafana
assigned them positionally at load and every panelId deep link was only as
stable as the panel order.
Per panel, in document order:
- id written as 1..N so panelId links address a specific panel
- gridPos quantized to at most two panels across: charts h=8,
stats/gauges h=4, tables/logs h=12 full width. Panels are
paired only with an equal-height neighbour, so no row is left
with a ragged half-empty cell.
- line charts lineWidth=1, fillOpacity=0, pointSize=5, gradientMode=none
- repeat xrpl_network_type (horizontal, maxPerRow=2) with a
[$xrpl_network_type] title suffix, on the panel types where
overlaying two networks in one panel reads as noise
(stat/gauge/bargauge/table/state-timeline). Line charts keep
their networks as separate series, which is the point of a
line chart.
- decimals 0 where the value counts discrete things (threads, peers,
queue depths); a fractional thread count is meaningless.
- rows category rows added where a dashboard had none
Panels whose legend sits on the right stay full width: a side legend needs the
horizontal room, and squeezing it to half width clips the series names.
Edits were made as raw-text replacements, not a json.dump round-trip, so
formatting and escaping of untouched lines are byte-identical. Verified per
dashboard: panel count unchanged, every targets block byte-identical, all
descriptions unchanged, ids exactly 1..N, and no two panels overlapping on the
grid. The repo dashboard lint and the OTel naming check both pass.
Four defects found by a query-correctness audit of all 16 dashboards, all
scoped to panels this branch owns.
job-queue "Current Job Latency (p99 Gauge)": the histogram by-clause dropped
service_instance_id, collapsing every node into one fleet-wide p99 and hiding
a slow node. Measured: gauge read 1325us while the worst node was 1868us. The
panel's displayName already referenced service_instance_id, so it was also
rendering an empty label. Now groups by service_instance_id and xrpl_work_item,
matching sibling panels 5, 6 and 7.
ledger-data-sync "NodeStore Read Latency (Bottleneck Discriminator)": replaced
clamp_min(<denominator>, 1) with (<denominator> > 0). clamp_min clamps the
value, not just the zero case, so any node reading below 1/s got a fabricated
denominator. Demonstrated with a zero-rate denominator: clamp_min invents
40.2/40.3/11.2/9.2 where the > 0 guard correctly returns no data. This panel is
the bottleneck discriminator, read during a stall, which is exactly when the
read rate collapses and the clamp is most wrong. Panels 21 and 23 carry the
same defect but originate on phase-7 and are fixed there.
node-health thresholds: percentunit fields are compared against the raw value,
so a step of 80 needed 8000% and could never fire. Rescaled panels 74, 81 and
85 to 0.8. Panel 81 is a found-ratio where high is healthy, so its bands were
also inverted. Note these three panels use palette-classic with thresholdsStyle
off, so the steps are currently dormant rather than visibly wrong.
node-health panels 81 and 85 descriptions: both described the multi-series
panels they were split from. Panel 81 carried a byte-identical copy of panel
80's text, promising three plotted rate lines where it draws a single bounded
ratio; panel 85's text described read-thread gauges absent from its expression.
Rewritten to match the actual queries.
The Transaction Overview panel "Queue Bypass Ratio (Direct Apply vs
Enqueue)" reported a confident 0.50 on every node while the true bypass
rate was zero. The two spans it divided are not disjoint alternatives:
txq.apply_direct is a child of txq.enqueue. TxQ.cpp creates the
apply_direct span as the first statement of tryDirectApply(), ahead of
the account, sequence and fee-level guards, and tryDirectApply() is
called from inside the live enqueue scope. The span therefore counts
attempts, so the denominator direct + enqueue counts each transaction
twice and pins the ratio to one half algebraically.
Measured on a four-node fleet: 6082443 direct against 6082877 enqueue
over the same population, panel output 0.5000170 on three nodes and
0.5000000 on the fourth. Grouping txq.enqueue by txq_status over seven
days returns only "rejected" -- no transaction has ever taken the
direct-apply path.
Remove the panel rather than repoint it. A correct expression using
txq_status as the disjoint discriminator would render permanently
empty on this fleet, which reads no better than a wrong number.
Widen the band partner "TxQ Enqueue Rate by Transaction Type" from 12
to 24 columns so the y=48 band still fills the grid. Every band in all
ten dashboards sums to 24 columns; leaving a half-width hole would be
the only exception. Panel order and every other panel's position,
width and height are unchanged.
The runbook already listed txq.apply_direct as available but not
paneled, so that row becomes accurate. Rows describing the span itself
are untouched -- the span and its metric are unchanged.
Panels 21 (NuDB Read Latency) and 23 (NuDB Read Found Ratio) on the
Ledger Data & Sync dashboard guarded their divisor with
clamp_min(<denominator>, 1). clamp_min raises the value, not just the
zero case, so any node reading fewer than 1 block per second was
divided by a fabricated 1 instead of by its real read count.
Replace the clamp with the filter (<denominator> > 0). A comparison
without the bool modifier drops the sample rather than rewriting it, so
these panels now show no data instead of a wrong number.
Measured over 7 days: five nodes fall below 1 read/s. On validator-0 the
clamp reported 2.726 us/read against a true 5.493, and on nonUNLmalloc-tc
it reported 0 us/read, which cannot occur. The error is largest exactly
when panel 21 is used as the bottleneck discriminator during a stall,
because that is when the read rate collapses toward zero.
Matches the existing idiom on the same nodestore_state metric family in
the NodeStore Write vs Read Latency panel.
The renderer was the only service in the stack on a floating :latest tag, so
the stack was not reproducible: a renderer release could change under a
checkout that had not been touched. Pins it to v5.12.0, matching how every
other service here is pinned.
Also wires the shared render token. Grafana 13 enables the renderAuthJWT
feature toggle by default, and the renderer rejects requests unless both sides
present the same non-default token, so pinning alone would leave rendering
broken once the Grafana bump merges forward. Both values come from the same
GF_RENDERING_RENDERER_TOKEN variable and cannot drift apart; the local default
keeps `docker compose up` working with no setup.
Verified `docker compose config` resolves and both AUTH_TOKEN and
GF_RENDERING_RENDERER_TOKEN expand to the same value.
Bumps the Loki pin this branch introduced, and updates the two docs that
name the version so they do not drift from the compose file:
- OpenTelemetryPlan/09-data-collection-reference.md (log backend section)
- OpenTelemetryPlan/Phase8_taskList.md (compose snippet)
The service runs the config bundled in the image rather than one from this
repo; `-verify-config` reports "config is valid" under 3.7.6, and native OTLP
ingestion at /otlp is unchanged across 3.4 -> 3.7.
Moves off the v2.53 LTS line, which this branch pinned, onto the current
v3 release, and corrects the accompanying comment that named the old line.
Verified against the new image rather than assumed, since this crosses a
major version:
- prometheus.yml passes `promtool check config` under v3.13.2
- all 391 unique dashboard PromQL queries parse under the v3 parser
- all 13 alert-rule expressions parse under the v3 parser
The config uses none of the surfaces v3 changed: no remote_write, no
holt_winters, no offset modifiers.
Bumps the three images this branch pins:
otel-collector-contrib 0.121.0 -> 0.158.0
tempo 2.7.2 -> 2.9.4
grafana 11.5.2 -> 13.1.2
Verified locally against the new images: the collector config passes
`validate` under 0.158.0 and tempo.yaml passes `-config.verify=true` under
2.9.4, both unchanged. The collector's log path uses the generic otlphttp
exporter (otlphttp/loki), not the dedicated loki exporter removed upstream in
v0.147.0, so the pipeline is unaffected by that removal.
Grafana crosses two majors. Operator-visible consequences are handled on the
branches that own the affected files: Grafana 13 enables the renderAuthJWT
feature toggle by default, so the image renderer now requires a matching
renderer_token on both the server and the renderer container.
The Mainnet telemetry config carried ssl_verify=0, inherited when the file
was copied from the Devnet config. Mainnet fetches its validator lists from
public HTTPS publishers (vl.ripple.com, unl.xrplf.org) that present valid CA
certificates, so there is no self-signed certificate to work around.
With ssl_verify=0 both halves of TLS peer authentication are skipped on those
fetches: the certificate chain check and the RFC 6125 hostname match. The list
blob's own signature is still verified against the pinned publisher keys, so
this restores defense in depth rather than fixing a validator-injection path.
Restores the secure default and documents why it stays that way.
The stat block was relaid two-per-row in the previous commit, but the reflow
shifted subsequent panels by a delta computed against the block's old height.
The block shrank, so the delta left a 12-grid-line dead band between the stat
row and "Node Operating State Transitions".
Recompute every panel's y by walking the list in order and pairing half-width
panels, rather than patching offsets incrementally. The layout is now derived
from panel order and heights, so this class of drift cannot recur.
Verified: 192 grid lines, fully covered, no empty band, no overlaps, and no grid
line occupied by more than two panels.
Fixes found while validating the Log-Derived Insights dashboard against live
Loki and Tempo. Every change was verified by executing the query, not by
inspection.
loki.yaml — the ConsensusLedgerHash derived field never matched anything:
- The TraceQL query named `.xrpl.consensus.ledger_id`. The attribute is
`consensus_ledger_id` (ConsensusSpanNames.h); live Tempo lists no dotted or
xrpl-prefixed variant among its span attributes.
- The bare `.` intrinsic scope cannot match a span attribute regardless of
name; TraceQL requires `span.`.
Corrected to `{span.consensus_ledger_id="..."}`, which returns the expected
consensus.round trace for a real ledger hash.
log-derived-insights.json:
- Stat units `suffix: <noun>` rendered raw integers, so large values printed
in full. `si:<noun>` is unsafe because Grafana parses the leading letter as
an SI prefix (`si:fee` rendered "29.0 Mee"). Switched to `short`, the
convention already used by 42 panels; the noun is in each panel title.
- Stat panels showed a single fleet-wide tile labelled "Value #A". Grafana's
Loki backend does not name binary-operation frames from legendFormat, so the
ratio panel could not resolve per-node labels. Replaced it with a plain sum
of duplicate fetches; the ratio remains available as a timeseries. Added an
explicit legendFormat to all eight stat panels.
- The ledger-fetch regex required `good:N dupe:N` together, but
SHAMapAddNode::get() omits any zero counter. That dropped 32% of lines --
disproportionately the efficient fetches -- biasing the ratio upward. Each
field now matches independently.
- state-timeline used `spanNulls: true`, which spans gaps indefinitely and
would render an outage as continuous. Set to the 30-minute threshold.
- Stat panels relaid two-per-row and given a tooltip, per dashboard guidelines.
telemetry-runbook.md:
- LogQL examples used line filters where structured metadata exists.
`|= "ERR"` also matches the literal in a message body: measured 423 ERR
lines plus 4 DBG lines per 6h. Replaced with severity, partition, and
trace_id field filters.
- Documented the new dashboard, its debug-log requirement, and the LogQL traps
these fixes exposed.
- Corrected the dashboard count, which was already stale at eleven against
fifteen on disk, and named the six with no reference section.
These changes were developed on the phase-10 branch but belong to content this
branch and its upstreams introduced. Carrying them on phase-10 made its PR diff
report churn in files phase-10 does not own, and left each PR claiming a scope
that did not match its contents.
Moved here from phase-10 (identical content, no functional change):
- Dashboards: all 14 existing boards plus the new log-derived-insights board.
- Docs: telemetry-runbook.md (minus the workload/benchmark sections, which
describe phase-10 tooling) and the new telemetry-glossary.md.
- Grafana Cloud + Alloy export path: collector config, compose override, the
two .env examples and alloy/config.alloy.
- Local stack: otel-collector-config.yaml gains sub-millisecond and
second-scale spanmetrics buckets, pins unit=ms, and promotes
close_time_correct; integration-test.sh and TESTING.md follow.
- Node configs: exported_instance -> service_instance_id in comments; the
mainnet sample now logs at warning to bound log volume.
- Metrics code: Telemetry.cpp builds the metrics pipeline in the constructor
via initMetrics() so the global MeterProvider is published before any
subsystem creates a beast::insight instrument, and the histogram view keeps
each instrument's own name instead of collapsing them under one series.
MetricsRegistry gains a last_close_time gauge and skips negative job-queue
durations. OTelCollector drops an unused accessor.
- Naming CI: xrpl_work_item joins EXTERNAL_INFRA_LABELS and Rule E accepts the
dotted perf-iac resource-attribute form. This must travel with the
dashboards and runbook that reference those labels, or the rules fail.
- Doxygen input glob no longer recurses dot-directories.
Sections describing phase-10 tooling stay on phase-10 and keep their
"Future Enhancement" / "Planned, not yet implemented" markers here; phase-10
removes those markers when it lands the tooling.
ValidatedLedgerStale fired on every node, healthy included.
LedgerMaster::getValidatedLedgerAge() returns weeks{2} (1209600s) as a
SENTINEL when no validated ledger exists, not as a measurement. The rule
read that as "14 days stale". Measured over six days it produced sustained
firing on all nine nodes. Excluding the exact sentinel value drops that to
zero while still tracking real staleness.
ManifestFloodInbound at 50 kB/s was routine paging: ~41 sustained 5-minute
samples across six healthy nodes in six days. Healthy p99 is 1.0-1.8 kB/s
and real storms peak at 2.7 MB/s, so 512 kB/s sits ~280x above normal and
~5x below the peaks, cutting sustained samples to 2.
Both thresholds were previously justified from a 24-hour window, which was
too short to expose either problem.
Three dashboards had layout faults that Grafana renders as visible
artifacts.
On Node Health, two row headers carried the same title as the panel
directly beneath them -- "Cache Hit Rates" and "Object Instance Counts"
each appeared twice, once as a row bar and once as a panel. Rename the
rows to "Caches" and "Object Instances"; the panels keep their names.
Worse on the same dashboard, seven bands summed to 48 columns on a
24-column grid, because the Extended Metrics block reused the y-space
already occupied by NodeStore I/O. Grafana cannot lay that out
deterministically, which is what produced the artifacts.
RPC Performance and Transaction Overview each had blank bands -- one
row and eight rows respectively -- left behind by earlier panel moves.
Repack all three grids so every band fits 24 columns, each band starts
where the previous one ends, and rows stay full-width single-height
anchors. Panel order, panel count and every panel's own width and
height are unchanged; only position moves.
