The Operating Mode Transitions panel queried state_accounting_*_transitions
directly. Those are monotonic counters, so the panel drew a slowly rising line
and a few transitions per hour were invisible against a total in the hundreds.
It also fell off a cliff whenever xrpld restarted and the counters reset to 0,
which reads as missing data rather than a restart.
Wrap each target in increase(...[$__rate_interval]) so each point is the number
of transitions in that bucket and the series survives a counter reset. This is
what the sibling panels on the same row (Operating Mode (Time Share), State
Duration Rate) already do.
Verified against devnet-otel-usw2-01/02 over 2026-08-11T11:01Z..2026-08-12T16:23Z:
the fixed expression reports 107 and 123 syncing transitions, matching the
counter deltas, and stays continuous across the 12:07 restart where the raw
counter dropped 630 -> 1.
Brief mode flaps remain invisible on Operating Mode (State Timeline) because a
~2 s dwell cannot be captured by a 60 s scrape; this panel is the place to read
them.
The troubleshooting step queried `job_count`, which returns no series. The
gauge is registered as `makeGauge("job_count")` but `Application.cpp` passes
`collectorManager_->group("jobq")`, so the exported name carries the `jobq`
segment. The two metric tables in this file were corrected when phase-6
merged forward; this example was missed because it sits outside the tables.
Conflict resolution kept this branch's evolution and re-applied phase-6's
fixes on top of it, rather than taking either side wholesale:
- consensus-health.json: kept the native `span_calls_total` metric name and
the `interval: 15s` and point styling from this branch; added phase-6's
`close_time_correct` PromQL filter and the NetClock axis labels. The
TraceQL boolean-regex filter stays removed and the `byRegexp` overrides
carry over. Panel count unchanged at 27.
- 09-data-collection-reference.md: kept this branch's headings, its more
detailed consensus attribute table (which already types
`consensus_round_id` as int64) and its section numbering, including the
deliberate removal of the SpanNames inventory. Carried over only the
correction that the state-accounting duration gauges are cumulative
microseconds, not seconds.
- telemetry-runbook.md: kept this branch's native metric names
(`span_calls_total`, `span_duration_milliseconds_bucket`); carried the
`rpc.request` -> `rpc.http_request` span-name fix and the `jobq_` segment
on the job-queue depth metric.
- integration-test.sh: kept this branch's `check_otel_metric` form and
carried the `jobq_job_count` correction.
The integration test asserted `rippled_job_count`, which never reports any
series, so that check always failed. `JobQueue` registers the gauge as
`makeGauge("job_count")`, but `Application.cpp` passes it
`collectorManager_->group("jobq")`, so the emitted StatsD name is
`jobq.job_count` and the exported Prometheus name is
`<prefix>_jobq_job_count`.
Corrected the same name in two runbook tables that also dropped the `jobq`
segment. `09-data-collection-reference.md` already had it right, which is why
the two documents disagreed.
Routed here rather than to the phase-10 PR where it was reported: the wrong
name is present in `integration-test.sh` on every branch from phase 6
onward, and this is the branch that introduces the file.
Left alone deliberately:
- `statsd-node-health.json` still queries the old name, but that dashboard is
deleted at phase 7 in favour of `node-health.json`
- `06-implementation-phases.md` names `job_count`, which is accurate as the
code-level makeGauge argument rather than the exported metric name
09-data-collection-reference.md contradicted itself. Section 2 presents
server=otel as the recommended transport with StatsD as a fallback, but
the Known Issues table and both Configuration Quick Reference examples
still prescribed server=statsd, which on this branch reaches a collector
with no statsd receiver and an unpublished 8125/udp.
Switch the Known Issues row and the Minimal and Production examples to
server=otel with the OTLP metrics endpoint. Keep the labelled fallback
block, and state what it actually requires: re-adding the statsd
receiver and republishing the port. Also record that StatsDCollector
applies prefix to metric names while OTelCollector does not, so the two
transports do not produce the same series.
The phase-6 copy is left alone; server=statsd is correct there.
check-otel-naming fails Rule D on this branch: ledger-data-sync.json
aggregates by xrpl_work_item, which nothing in-tree emits because
perf-iac's alloy pipeline stamps it, so it cannot be derived from
*SpanNames.h. The sibling perf-iac identities xrpl_branch and
xrpl_node_role are already allowlisted; this one was missed when the
by() clauses were reintroduced.
Add it to EXTERNAL_INFRA_LABELS alongside them. Verified by removing
the entry again, which restores the failure.
This branch removes the collector's StatsD receiver and un-publishes
8125/udp, but xrpld-telemetry.cfg still selected server=statsd, so the
sample config sent beast::insight metrics over UDP to a port nothing
listens on. Phase7_taskList.md:132 lists this switch as required work.
Select server=otel and replace address= with the OTLP metrics endpoint.
Document that endpoint and prefix are informational only, since
OTelCollector records on the global MeterProvider that [telemetry]
configures and formatName() applies no prefix, and note that beast
instruments are not exported yet because the collector is constructed
before the MeterProvider is registered.
All five TraceQL panels on this dashboard returned nothing, and did so
without any visible error: they filtered on
span.close_time_correct=~"$close_time_correct", but close_time_correct is
a boolean attribute (RCLConsensus.cpp:601 passes a raw bool), and Tempo
restricts the regex operator to string operands, so the spanset resolved
to false. With the variable defaulting to All the clause rendered as
=~".*", so the panels were empty out of the box and looked exactly like a
node with no consensus activity.
Note this is the opposite of PromQL, where an absent or empty label does
match ".*" — which is why the 17 Prometheus panels on this same board were
unaffected and the dashboard appeared healthy.
Dropped the clause from all six queries, matching phases 9 and 10 where it
is already gone. The $close_time_correct variable now filters the
Prometheus "Close Time Agreement" panel instead, which already grouped by
that label but never filtered on it, so the control stays useful rather
than becoming dead UI.
Two defects were masked behind the empty panels and are fixed too:
- "Close Time: Raw Proposals" and "Close Time: Effective / Quantized"
carried unit dateTimeFromNow over close_time_self/close_time, which are
NetClock seconds (Ripple epoch), while Grafana's dateTime formatters
expect a millisecond Unix epoch — every point would have rendered as
roughly 1970. They now plot as plain numbers with the axis labelled
"NetClock Seconds (Ripple Epoch)", and the descriptions give the
946684800 offset for converting to Unix time.
- "Close Time Vote Bins & Resolution" matched its unit and axis overrides
byName against "Vote Bins" and "Resolution", which are not field names;
TraceQL select() yields close_time_vote_bins and close_resolution_ms, so
neither override applied. Switched to byRegexp so the match holds
whichever scope prefix Grafana emits.
No panel was added or removed: the (type, title) multiset is unchanged at
22. resolution_direction keeps its regex filter, which is correct there —
it is set from a std::string whose values are exactly the variable's
increased/decreased/unchanged.
The integration test's span assertions never actually ran. check_span()
built a Tempo /api/search call with --data-urlencode but no -G, so curl
POSTed the params as a body; Tempo answers 200 and ignores the query, so
every span name looked present. Verified against a live Tempo 2.9.4: the
buggy form returns the store's total trace count for any name, including
"zzz.does.not.exist"; with -G a real name returns 1 and a bogus one 0.
Fixed alongside it: the RPC check asserted "rpc.request", which is never
emitted (ServerHandler.cpp builds "rpc.http_request"). These two had to
change together, since -G turns the bogus name from a silent pass into a
hard failure.
Also in the script: a consensus timeout logged two failures and counted
two, because a post-loop else re-reported what the timeout branch had
already reported; and three unguarded curl calls aborted the whole script
under set -euo pipefail, making the ACCOUNT_ZERO fallback dead code with
no cleanup. Guarded the curls and wired an EXIT trap to the existing
cleanup(). The trap deliberately fires only before the summary, so a
completed run still leaves the stack up as the header documents.
Docs corrections, all re-derived from code:
- span inventory heading 35 -> 38, attribute heading 83 -> 89 rows
(78 unique keys), and the section 6 header table now carries the
missing TxApplySpanNames.h row so its columns sum to the same figures
- two stale paths: ConsensusSpanNames.h is under include/xrpl/consensus/,
TxSpanNames.h under src/xrpld/telemetry/
- consensus_round_id is int64, not string (RCLConsensus.cpp sets
prevLgr.seq() + 1); the runbook's TraceQL examples now use a numeric
literal instead of an unparseable bare <round_id>
- state-accounting duration gauges are cumulative MICROSECONDS, not
seconds (NetworkOPs.cpp declares std::chrono::microseconds and
publishes dur.count() raw)
- sampling_ratio is not a config key; head sampling is fixed at 1.0 and
the shipped collector has no tail sampling, so the caveat was rewritten
- the plan blurb referenced Jaeger; this stack is Tempo
Run with no arguments the script iterated an empty list, found no
violations and printed "OK: 0 dashboard(s) passed" with exit 0 -- a clean
bill of health for reading no files, indistinguishable from a real pass.
A bare run now defaults to every dashboard beside the script, and a run
that still ends up with nothing to check exits 2 rather than reporting
success. Passing paths explicitly behaves as before.
Nine targets across four panels filtered on xrpl_work_item while leaving
it out of their sum by() grouping. PromQL keeps only the labels listed in
by(), so the label was dropped from the result and the xrpl_ident legend
these panels build from it lost that segment.
Perf-iac runs stamp xrpl_work_item and give each work item its own set of
nodes, so service_instance_id already separates the runs; the visible
effect is the legend rather than merged series. Adding the label makes the
work item identifiable, which is the comparison these panels exist for.
NuDB Read Latency and NuDB Read Found Ratio each divide two aggregations.
Both sides get the label so their label sets stay equal and vector
matching still works.
The row was titled after the rollout phase its metrics came from, a number
defined only in a planning folder outside the shipped tree. Grafana shows
this title to operators, so it named something no reader could look up.
Two panel descriptions pointed at a rollout phase and task number defined
only in a planning folder outside the shipped tree. The note still names
the file and the change it is waiting on, which is the part a reader can
act on.
The catalog tagged every span with a rollout phase number defined only in
a planning folder outside the shipped tree, so the column meant nothing
to a reader of the repository. The span name and source file identify
each entry.
The section labels carried rollout phase numbers that are defined only in
a planning folder outside the shipped tree, so they meant nothing to a
reader of the repository. The descriptive half of each label already
identifies the section.
The label carried a rollout phase number defined only in a planning
folder outside the shipped tree. The descriptive half already identifies
the section.
These comments pointed at a planning folder and at its rollout phase
numbering, neither of which is part of the shipped tree, so the
references would dangle for any reader of the repository. Each comment
now states the fact it was pointing at.
The trace_state comment pointed at a planning document that is not part
of the shipped tree, so the reference would dangle for any reader of the
repository. State the reserved-and-inert fact on its own.
A repeating panel expands into one copy per network at view time and, with
maxPerRow=2, claims the whole row. The non-repeating panel paired beside it was
pushed to the next line but kept its stored x=12, so it rendered on the right
against an empty gap.
Two changes to how the layout is planned:
- a repeating panel gets a row to itself. It keeps w=12, so its copies still
tile two across inside that row.
- single-value panels are grouped to the top of each row section, so the
charts that follow pair with each other instead of being split up by an
interleaved repeat. Without this the gaps just become wasted half-rows.
Verified per dashboard: no panel lost, every targets block byte-identical, ids
1..N, no overlaps, and no row left with a gap on its left.
Every heatmap carried only `tooltip` and `yAxis`, missing `calculate`, `color`
and `cellGap`. Grafana's heatmap plugin treats those as required, and without
them the panel fails to initialise: the dashboard opens with "An error occurred
within the plugin" rather than a chart.
The option values are taken from the one heatmap in this stack that does render
on Grafana Cloud (ledger-sync-health): calculate=false since the queries already
return histogram buckets, the Turbo 64-step scheme, and cellGap=1. Each panel
keeps its own yAxis label, unit and tooltip settings.
This is a long-standing defect rather than fallout from the recent layout work -
the same options are absent in origin/phase9 and in the cloud copies that were
already live.
Two problems showed up once these dashboards were live on Grafana Cloud.
Panels were too short. Charts at h=8 clipped their legends mid-row, and stats
at h=4 were cramped. Every visualisation is now h=10, with tables and logs at
h=12; the log-derived-insights instruction banner keeps h=12 for its prose.
Uniform height also means any two panels can pair side by side.
Repeat on a state-timeline broke the dashboard outright: ledger-data-sync
failed to open on Cloud at v45 and had to be restored to v44. Repeat is now
limited to single-value panels (stat, gauge, bargauge, table). Charts show
their networks as separate series instead, which is what a chart is for.
Repeat was also sticky: normalization only ever added the keys, so a panel that
picked them up in an earlier pass kept them even after its type stopped being
eligible. The keys and the title suffix are now removed from ineligible panels,
which is what actually cleared the two timeline panels here.
Verified per dashboard: no panel lost, every targets block byte-identical, ids
1..N, no grid overlaps, and repeat present only on stat/gauge/bargauge.
Guideline 8 asks for gauges and stats at the top. Seven dashboards had them
scattered below charts, so the reader met a wall of time series before the
at-a-glance numbers that give those series context.
Stats, gauges and bar gauges now come first within each row section. The move
is deliberately scoped to inside a section: shifting a panel across a row
boundary would change which category it belongs to. Panels keep their relative
order otherwise, so the reading sequence within each group is unchanged.
Whole panel objects are cut and re-spliced as raw text, so their contents stay
byte-identical and only gridPos and id are recomputed. Verified per dashboard:
panel count unchanged, no panel lost, every targets block byte-identical, ids
still 1..N, and no row section left with a stat below a chart.
These dashboards were hand-authored over time and had drifted apart: panel
heights spanned ten different values, nine dashboards had no row grouping,
line-chart styling was inconsistent, and no panel carried an id, so Grafana
assigned them positionally at load and every panelId deep link was only as
stable as the panel order.
Per panel, in document order:
- id written as 1..N so panelId links address a specific panel
- gridPos quantized to at most two panels across: charts h=8,
stats/gauges h=4, tables/logs h=12 full width. Panels are
paired only with an equal-height neighbour, so no row is left
with a ragged half-empty cell.
- line charts lineWidth=1, fillOpacity=0, pointSize=5, gradientMode=none
- repeat xrpl_network_type (horizontal, maxPerRow=2) with a
[$xrpl_network_type] title suffix, on the panel types where
overlaying two networks in one panel reads as noise
(stat/gauge/bargauge/table/state-timeline). Line charts keep
their networks as separate series, which is the point of a
line chart.
- decimals 0 where the value counts discrete things (threads, peers,
queue depths); a fractional thread count is meaningless.
- rows category rows added where a dashboard had none
Panels whose legend sits on the right stay full width: a side legend needs the
horizontal room, and squeezing it to half width clips the series names.
Edits were made as raw-text replacements, not a json.dump round-trip, so
formatting and escaping of untouched lines are byte-identical. Verified per
dashboard: panel count unchanged, every targets block byte-identical, all
descriptions unchanged, ids exactly 1..N, and no two panels overlapping on the
grid. The repo dashboard lint and the OTel naming check both pass.
The Transaction Overview panel "Queue Bypass Ratio (Direct Apply vs
Enqueue)" reported a confident 0.50 on every node while the true bypass
rate was zero. The two spans it divided are not disjoint alternatives:
txq.apply_direct is a child of txq.enqueue. TxQ.cpp creates the
apply_direct span as the first statement of tryDirectApply(), ahead of
the account, sequence and fee-level guards, and tryDirectApply() is
called from inside the live enqueue scope. The span therefore counts
attempts, so the denominator direct + enqueue counts each transaction
twice and pins the ratio to one half algebraically.
Measured on a four-node fleet: 6082443 direct against 6082877 enqueue
over the same population, panel output 0.5000170 on three nodes and
0.5000000 on the fourth. Grouping txq.enqueue by txq_status over seven
days returns only "rejected" -- no transaction has ever taken the
direct-apply path.
Remove the panel rather than repoint it. A correct expression using
txq_status as the disjoint discriminator would render permanently
empty on this fleet, which reads no better than a wrong number.
Widen the band partner "TxQ Enqueue Rate by Transaction Type" from 12
to 24 columns so the y=48 band still fills the grid. Every band in all
ten dashboards sums to 24 columns; leaving a half-width hole would be
the only exception. Panel order and every other panel's position,
width and height are unchanged.
The runbook already listed txq.apply_direct as available but not
paneled, so that row becomes accurate. Rows describing the span itself
are untouched -- the span and its metric are unchanged.
Panels 21 (NuDB Read Latency) and 23 (NuDB Read Found Ratio) on the
Ledger Data & Sync dashboard guarded their divisor with
clamp_min(<denominator>, 1). clamp_min raises the value, not just the
zero case, so any node reading fewer than 1 block per second was
divided by a fabricated 1 instead of by its real read count.
Replace the clamp with the filter (<denominator> > 0). A comparison
without the bool modifier drops the sample rather than rewriting it, so
these panels now show no data instead of a wrong number.
Measured over 7 days: five nodes fall below 1 read/s. On validator-0 the
clamp reported 2.726 us/read against a true 5.493, and on nonUNLmalloc-tc
it reported 0 us/read, which cannot occur. The error is largest exactly
when panel 21 is used as the bottleneck discriminator during a stall,
because that is when the read rate collapses toward zero.
Matches the existing idiom on the same nodestore_state metric family in
the NodeStore Write vs Read Latency panel.
Moves off the v2.53 LTS line, which this branch pinned, onto the current
v3 release, and corrects the accompanying comment that named the old line.
Verified against the new image rather than assumed, since this crosses a
major version:
- prometheus.yml passes `promtool check config` under v3.13.2
- all 391 unique dashboard PromQL queries parse under the v3 parser
- all 13 alert-rule expressions parse under the v3 parser
The config uses none of the surfaces v3 changed: no remote_write, no
holt_winters, no offset modifiers.
Bumps the three images this branch pins:
otel-collector-contrib 0.121.0 -> 0.158.0
tempo 2.7.2 -> 2.9.4
grafana 11.5.2 -> 13.1.2
Verified locally against the new images: the collector config passes
`validate` under 0.158.0 and tempo.yaml passes `-config.verify=true` under
2.9.4, both unchanged. The collector's log path uses the generic otlphttp
exporter (otlphttp/loki), not the dedicated loki exporter removed upstream in
v0.147.0, so the pipeline is unaffected by that removal.
Grafana crosses two majors. Operator-visible consequences are handled on the
branches that own the affected files: Grafana 13 enables the renderAuthJWT
feature toggle by default, so the image renderer now requires a matching
renderer_token on both the server and the renderer container.
