Add two Peer Quality panels reading peer_disconnect_total: Peer Disconnect
Rate, the per-second teardown rate per node, and Peer Disconnects By Reason
& Direction, the per-interval increase split by cause and by which side
opened the connection. Both sit in the existing Disconnects & Connection Mix
row beside Resource Disconnects, which counts only the resource-charge
subset and carries no reason label.
The Ledger Sync Health board already shows the same split as a window
total, so it says how much of each reason but not when. These give the
time-shaped view, letting a reason spike be lined up against a stall.
Add disconnect_reason and disconnect_direction template variables for the
two new label dimensions and wire both queries to them, so the panels
filter on every dimension their series carry.
Update the 09 reference panel column and the _a7_note panel list to name
the panels that now render this counter.
Two runbook readings were misleading in ways that hid a real devnet issue.
The Online-Delete Rotation Window row said a healthy node shows the flag
"briefly 1 once per delete interval". That is true but not sufficient:
rotation_state{metric="in_flight"} is set immediately before
freshenCaches() and cleared by RotationExposureGuard on scope exit, so it
brackets only the freshen/swap phase. Measured on devnet-otel-usw2-01/02
with online_delete=256 and ~47.4M state nodes, the flag averaged
0.159/0.135 over 9 h while the node was inside a rotation ~93% of wall
clock, because the dominant visitNodes copy phase (median 651 s of an
~785 s cycle) emits nothing. The row read healthy on a rotation-bound
node. Add a scope note with the log triplet that does measure occupancy.
The Consensus Round Duration panel gained a p99 series, so retitle both
references and say when to read it: at ~19 rounds/min a single stalled
round is one sample in several hundred, so p95 stays at the normal close
time and can dip. An 11.4 s round measured p99 13400 ms while p95 read
3400 ms against a 2900-3787 ms baseline.
Instrumenting the copy phase is tracked in RIPD-7144; the underlying
rotation defects in RIPD-7720.
The "Consensus Round Duration" panel plotted only p50 and p95. At the
devnet rate of ~19 rounds/min, a single stalled round is one sample in
several hundred, so p95 over a rate interval stays near the normal 2s
close time and the stall is invisible.
Measured on devnet-otel-usw2-02 over a 40 min window: 761 rounds, of
which exactly one fell in the 7500-10000ms bucket while the mean never
exceeded 2270ms. The histogram already records the outlier; only the
panel could not show it.
Add a p99 series alongside p50/p95 and retitle the panel accordingly.
No metric or collector change: the existing
consensus_round_duration_ms_bucket histogram already has 20 boundaries
up to 120000ms.
The Peer Disconnect Rate By Reason panel anchored its LogQL capture on
"\] ", which only matches a reason logged immediately after the [NNN]
peer-id prefix. PeerImp does not log that way: PeerImp::fail emits
"[NNN] <name> failed: <reason>" and the clean teardown emits
"close: Closed". Only ConnectAttempt::fail, which logs the bare reason,
ever matched. The panel's Timeout series was therefore connect-attempt
timeouts only, Ping Timeout was invisible, and PeerImp's own Closed was
uncounted.
Match all three prefixes and separate Ping Timeout from Connect Timeout.
Recorded as LogQL trap 11 in the runbook, alongside the other silent
failures this dashboard exposed.
Also document six overlay observability gaps found while auditing what
ping/pong and gossip traffic is actually tracked. All are pre-existing
and none is fixed here: the code fixes belong in develop-owned overlay
files (TrafficCount, OverlayImpl, PeerImp, PeerfinderManager), not on a
telemetry branch, and one of them needs a public signature change.
- 09 section 6: six known issues, each marked NOT IMPLEMENTED with
file:line evidence -- mtCLUSTER counted as unknown (overhead_cluster_*
always zero, 8 panels flatline), squelch_ignored byte counts always
zero, inbound/outbound byte-basis asymmetry plus a stale Total header
comment, ping/endpoints instrumentation absent, peer span coverage,
and PeerFinder exporting 2 of ~17 available readings.
- 02 section 2.3.2: add a Status column to the span catalog. Of 36
catalogued spans, 16 are live, 15 were never built, and 5 shipped
under different names (consensus.phase.establish became
consensus.establish, ledger.close became consensus.ledger_close,
rpc.request split into rpc.http_request/rpc.ws_message, txq.apply
became txq.apply_direct/txq.accept_tx). The catalog is a design
inventory; 09 section 1.1 remains authoritative for what emits.
- Phase9_taskList: tasks 9.14-9.17 tracking the deferred work, with
exit criteria checked only for what is actually done.
- Glossary: new Ping / pong keepalive term distinguishing ping timeout
from connect timeout. Correct the Cluster and Squelch entries, which
described behaviour the metrics cannot show.
The glossary header pointed at tasks/telemetry_terms.py as its
generator. That file is in no branch and nowhere on disk -- tasks/ is
gitignored one directory up -- so the header now states the file is
hand-maintained and gives the entry format.
Gates: check_otel_naming.py passes all 9 rules (Rule D over 555
dashboard queries, Rule E over the runbook); 19 doc anchors verified;
dashboard JSON valid with a one-line diff. No C++ changes.
Resolves the nodestore latency-accumulator conflict by keeping this
branch's API surface and applying the incoming nanosecond fix to it.
This branch had renamed storeDurationStats() to recordStoreDuration() and
given it noexcept, an explicit relaxed fetch_add, and call-frequency and
thread-safety notes. The incoming change fixed the accumulators to hold
nanoseconds so sub-microsecond backend calls stop rounding to zero. Both
are kept: the name, signature and documentation from here, the nanosecond
accumulation from there.
The incoming duplicate getFetchDurationUs()/getStoreDurationUs() pair is
dropped in favour of this branch's noexcept versions, which now read the
nanosecond members and convert on read. recordStoreDuration()'s `if (us >
0)` guard is gone: it discarded every sub-microsecond store, which is the
same rounding the incoming change removes one level up.
Also updates a MetricMacros comment that explained the zero write mean in
terms of that removed guard; the injected totals it asserts on are
unchanged.
The fetch and store duration counters converted each sample to
microseconds before adding it, so any backend call finishing in under a
microsecond contributed zero. A warm nudb read answers in a few hundred
nanoseconds, so on fast hardware every read floored and the totals stayed
at zero no matter how many reads happened -- the same loss of resolution
the microsecond report was introduced to avoid, one decade lower.
Both accumulators now hold nanoseconds, the clock's own resolution, and
convert once in getFetchDurationUs() and getStoreDurationUs(). The public
accessors, the node_reads_duration_us and node_writes_duration_us JSON
fields, and the metrics that read them all keep microseconds, so nothing
downstream changes unit. storeDurationStats() takes the raw duration
instead of a pre-converted integer so no caller can round early, and
updateFetchMetrics() scales its microsecond input to match.
FetchReport::elapsed stays microseconds: it carries one fetch, not a
total, and that is the unit it declares. The reported sum is therefore the
accumulated total minus a sub-microsecond remainder per fetch, so the two
tests that asserted exact equality between them now assert that bound.
Both had assertions that depended on how fast the host reads; the bound
holds on any hardware.
The debug-log notice on Log-Derived Insights was two grid rows taller than its
content needs, pushing the first data row further down than necessary. Rendered
the panel to confirm the markdown still fits with no clipping.
It is the first panel in the dashboard, so no other panel's stored y needed to
change - Grafana's vertical compaction closes the gap.
The dashboard's sections and panels were rearranged directly in Grafana. The
in-repo copy still carried the old arrangement, so the next replace-upload would
have undone that work.
Panel and row positions (y, and x for non-repeating panels), plus panel ids, now
come from the live copy. Ids matter because the uploader keys on them: matching
them makes an upload an in-place update rather than a reshuffle.
Repeating panels stay stored at w=12, x=0 rather than adopting the live w=24. For
a horizontal repeat Grafana ignores the stored width and lays each copy out at
24/min(N, maxPerRow), so the two are indistinguishable on screen - verified by
rendering a single-network view, where a w=12 repeat still fills the row. w=12 is
the honest description of one panel at half width.
Verified: panel and row order identical to live, rendered geometry identical at
two networks (what the live stack runs), no panel rendering with an empty column
band to its left at one through four networks, and every query, description and
panel option untouched.
Carries the layout fix to the dashboards this branch owns. A repeating panel
claims the whole row once it expands per network, so the non-repeating panel
paired beside it was pushed down while keeping its stored x=12 - rendering on
the right with an empty gap on its left.
Repeating panels now get a row to themselves, keeping w=12 so their copies still
tile two across. Single-value panels are grouped to the top of each row section
so the charts that follow pair with each other rather than being separated by an
interleaved repeat.
Verified against origin/phase9: no panel lost, every targets block unchanged,
ids 1..N, no overlaps, no rows with a left-hand gap.
A repeating panel expands into one copy per network at view time and, with
maxPerRow=2, claims the whole row. The non-repeating panel paired beside it was
pushed to the next line but kept its stored x=12, so it rendered on the right
against an empty gap.
Two changes to how the layout is planned:
- a repeating panel gets a row to itself. It keeps w=12, so its copies still
tile two across inside that row.
- single-value panels are grouped to the top of each row section, so the
charts that follow pair with each other instead of being split up by an
interleaved repeat. Without this the gaps just become wasted half-rows.
Verified per dashboard: no panel lost, every targets block byte-identical, ids
1..N, no overlaps, and no row left with a gap on its left.
Panels were appearing on the right of a row with nothing to their left. A
repeating panel expands into one copy per network at view time and, with
maxPerRow=2, claims the whole row; the non-repeating panel paired beside it was
pushed to the next line but kept its stored x=12, so it rendered on the right
against an empty gap. Thirteen rows here were affected - more than every other
dashboard combined, which is why it showed up on this one first.
Two changes:
- a repeating panel is given a row to itself when the layout is planned. It
keeps w=12, so its copies still tile two across inside that row.
- single-value panels are grouped to the top of each row section, so the
charts that follow can pair with each other instead of being split up by an
interleaved repeat. Without this the gaps become wasted half-rows.
Result: no mixed rows, no left-hand gaps, 13 paired rows and only 3 panels
sitting alone, from sections with an odd number of charts.
Verified: 54 panels before and after, no panel lost, every targets block and
description byte-identical, ids 1..N, no overlaps.
Brings this dashboard in line with the other fifteen, which were normalized
together:
- id written as 1..N so panelId deep links address a specific panel
rather than whichever one happens to sit in that position
- gridPos at most two panels across; every visualisation h=10. Three
panels sat at w=8, the only three-across row in the set.
- line charts lineWidth=1, fillOpacity=0, pointSize=5, gradientMode=none
- repeat xrpl_network_type (horizontal, maxPerRow=2) with a
[$xrpl_network_type] title suffix on the stat and bargauge
panels, where two networks overlaid in one panel reads as
noise. Charts keep their networks as separate series.
The nine existing row groups are kept as they are. The heatmap already carried
the full plugin option set, so it is untouched.
Raw-text edits, so untouched lines keep their formatting byte for byte.
Verified: 54 panels before and after, no panel lost, every targets block and
description byte-identical, ids 1..N, and no two panels overlapping.
The GetObject Request Size Distribution heatmap, added on this branch, was
missing calculate, color and cellGap. Grafana's heatmap plugin needs them, and
without them the whole dashboard opens with "An error occurred within the
plugin" instead of rendering.
Same values as the other four heatmaps fixed upstream, taken from the one that
renders correctly on Grafana Cloud. The panel keeps its own axis label and unit.
Every heatmap carried only `tooltip` and `yAxis`, missing `calculate`, `color`
and `cellGap`. Grafana's heatmap plugin treats those as required, and without
them the panel fails to initialise: the dashboard opens with "An error occurred
within the plugin" rather than a chart.
The option values are taken from the one heatmap in this stack that does render
on Grafana Cloud (ledger-sync-health): calculate=false since the queries already
return histogram buckets, the Turbo 64-step scheme, and cellGap=1. Each panel
keeps its own yAxis label, unit and tooltip settings.
This is a long-standing defect rather than fallout from the recent layout work -
the same options are absent in origin/phase9 and in the cloud copies that were
already live.
Carries the sizing fix through to the dashboards this branch owns, after the
live Cloud copies showed charts clipping their legends at h=8 and stats too
cramped at h=4.
Every visualisation is h=10; tables and logs h=12; the log-derived-insights
instruction banner h=12 for its prose. A single height per visualisation also
means any two panels pair cleanly side by side.
Repeat stays on single-value panels only. The five state-timeline and heatmap
panels that had picked it up in an earlier pass are cleared, since repeating a
timeline broke ledger-data-sync on Cloud.
Verified against origin/phase9: no panel lost, every targets block unchanged,
ids 1..N, no grid overlaps.
Two problems showed up once these dashboards were live on Grafana Cloud.
Panels were too short. Charts at h=8 clipped their legends mid-row, and stats
at h=4 were cramped. Every visualisation is now h=10, with tables and logs at
h=12; the log-derived-insights instruction banner keeps h=12 for its prose.
Uniform height also means any two panels can pair side by side.
Repeat on a state-timeline broke the dashboard outright: ledger-data-sync
failed to open on Cloud at v45 and had to be restored to v44. Repeat is now
limited to single-value panels (stat, gauge, bargauge, table). Charts show
their networks as separate series instead, which is what a chart is for.
Repeat was also sticky: normalization only ever added the keys, so a panel that
picked them up in an earlier pass kept them even after its type stopped being
eligible. The keys and the title suffix are now removed from ineligible panels,
which is what actually cleared the two timeline panels here.
Verified per dashboard: no panel lost, every targets block byte-identical, ids
1..N, no grid overlaps, and repeat present only on stat/gauge/bargauge.
Applies the guideline-8 ordering to the two dashboards introduced on this
branch, and clears the last two labelling gaps found while auditing the set:
- peer-quality, validator-health: stats and bar gauges now lead their row
section instead of trailing the charts, matching the other dashboards.
- node-health "Peer Disconnects (Resources)": had an empty axisLabel, the
only chart in the set without one. Now reads "Disconnects".
- rpc-performance row "Aggregate RPC (all commands)" -> "(All Commands)",
the one title that was not in Title Case.
Reordering stays inside each row section so no panel changes category, and the
panel objects are cut and re-spliced as raw text, so their contents are
byte-identical. Verified against origin/phase9: no panel lost, every targets
block unchanged, ids 1..N, and every row section now leads with its
single-value panels.
Guideline 8 asks for gauges and stats at the top. Seven dashboards had them
scattered below charts, so the reader met a wall of time series before the
at-a-glance numbers that give those series context.
Stats, gauges and bar gauges now come first within each row section. The move
is deliberately scoped to inside a section: shifting a panel across a row
boundary would change which category it belongs to. Panels keep their relative
order otherwise, so the reading sequence within each group is unchanged.
Whole panel objects are cut and re-spliced as raw text, so their contents stay
byte-identical and only gridPos and id are recomputed. Verified per dashboard:
panel count unchanged, no panel lost, every targets block byte-identical, ids
still 1..N, and no row section left with a stat below a chart.
Applies the same normalization the phase-7-owned dashboards received, to the
five introduced on this branch: fee-market, job-queue, log-derived-insights,
peer-quality and validator-health.
- id written as 1..N so panelId deep links address a specific panel
rather than whichever panel happens to sit in that position
- gridPos at most two panels across; charts h=8, stats/gauges h=4,
tables/logs h=12 full width. Panels pair only with an
equal-height neighbour, so no row keeps a ragged empty cell.
validator-health and peer-quality were stacked single-file at
full width, which is why they were the longest to scroll.
- line charts lineWidth=1, fillOpacity=0, pointSize=5, gradientMode=none
- repeat xrpl_network_type (horizontal, maxPerRow=2) with a
[$xrpl_network_type] title suffix on stat/gauge/bargauge/
table/state-timeline panels, where two networks overlaid in
one panel reads as noise. Line charts keep their networks as
separate series.
- decimals 0 on panels counting discrete things (peers, quorum,
connection mix) - a fractional peer count is meaningless.
- rows category rows added to fee-market, job-queue and peer-quality
Panels with a right-hand legend stay full width; a side legend needs the room.
Raw-text edits, so untouched lines keep their formatting and escaping byte for
byte. Verified per dashboard against origin/phase9: panel count unchanged, no
panel lost, every targets block byte-identical, descriptions unchanged, ids
exactly 1..N, and no two panels overlapping on the grid.
