The dashboard's sections and panels were rearranged directly in Grafana. The
in-repo copy still carried the old arrangement, so the next replace-upload would
have undone that work.
Panel and row positions (y, and x for non-repeating panels), plus panel ids, now
come from the live copy. Ids matter because the uploader keys on them: matching
them makes an upload an in-place update rather than a reshuffle.
Repeating panels stay stored at w=12, x=0 rather than adopting the live w=24. For
a horizontal repeat Grafana ignores the stored width and lays each copy out at
24/min(N, maxPerRow), so the two are indistinguishable on screen - verified by
rendering a single-network view, where a w=12 repeat still fills the row. w=12 is
the honest description of one panel at half width.
Verified: panel and row order identical to live, rendered geometry identical at
two networks (what the live stack runs), no panel rendering with an empty column
band to its left at one through four networks, and every query, description and
panel option untouched.
Panels were appearing on the right of a row with nothing to their left. A
repeating panel expands into one copy per network at view time and, with
maxPerRow=2, claims the whole row; the non-repeating panel paired beside it was
pushed to the next line but kept its stored x=12, so it rendered on the right
against an empty gap. Thirteen rows here were affected - more than every other
dashboard combined, which is why it showed up on this one first.
Two changes:
- a repeating panel is given a row to itself when the layout is planned. It
keeps w=12, so its copies still tile two across inside that row.
- single-value panels are grouped to the top of each row section, so the
charts that follow can pair with each other instead of being split up by an
interleaved repeat. Without this the gaps become wasted half-rows.
Result: no mixed rows, no left-hand gaps, 13 paired rows and only 3 panels
sitting alone, from sections with an odd number of charts.
Verified: 54 panels before and after, no panel lost, every targets block and
description byte-identical, ids 1..N, no overlaps.
Brings this dashboard in line with the other fifteen, which were normalized
together:
- id written as 1..N so panelId deep links address a specific panel
rather than whichever one happens to sit in that position
- gridPos at most two panels across; every visualisation h=10. Three
panels sat at w=8, the only three-across row in the set.
- line charts lineWidth=1, fillOpacity=0, pointSize=5, gradientMode=none
- repeat xrpl_network_type (horizontal, maxPerRow=2) with a
[$xrpl_network_type] title suffix on the stat and bargauge
panels, where two networks overlaid in one panel reads as
noise. Charts keep their networks as separate series.
The nine existing row groups are kept as they are. The heatmap already carried
the full plugin option set, so it is untouched.
Raw-text edits, so untouched lines keep their formatting byte for byte.
Verified: 54 panels before and after, no panel lost, every targets block and
description byte-identical, ids 1..N, and no two panels overlapping.
The Mainnet telemetry config carried ssl_verify=0, inherited when the file
was copied from the Devnet config. Mainnet fetches its validator lists from
public HTTPS publishers (vl.ripple.com, unl.xrplf.org) that present valid CA
certificates, so there is no self-signed certificate to work around.
With ssl_verify=0 both halves of TLS peer authentication are skipped on those
fetches: the certificate chain check and the RFC 6125 hostname match. The list
blob's own signature is still verified against the pinned publisher keys, so
this restores defense in depth rather than fixing a validator-injection path.
Restores the secure default and documents why it stays that way.
The stat block was relaid two-per-row in the previous commit, but the reflow
shifted subsequent panels by a delta computed against the block's old height.
The block shrank, so the delta left a 12-grid-line dead band between the stat
row and "Node Operating State Transitions".
Recompute every panel's y by walking the list in order and pairing half-width
panels, rather than patching offsets incrementally. The layout is now derived
from panel order and heights, so this class of drift cannot recur.
Verified: 192 grid lines, fully covered, no empty band, no overlaps, and no grid
line occupied by more than two panels.
Fixes found while validating the Log-Derived Insights dashboard against live
Loki and Tempo. Every change was verified by executing the query, not by
inspection.
loki.yaml — the ConsensusLedgerHash derived field never matched anything:
- The TraceQL query named `.xrpl.consensus.ledger_id`. The attribute is
`consensus_ledger_id` (ConsensusSpanNames.h); live Tempo lists no dotted or
xrpl-prefixed variant among its span attributes.
- The bare `.` intrinsic scope cannot match a span attribute regardless of
name; TraceQL requires `span.`.
Corrected to `{span.consensus_ledger_id="..."}`, which returns the expected
consensus.round trace for a real ledger hash.
log-derived-insights.json:
- Stat units `suffix: <noun>` rendered raw integers, so large values printed
in full. `si:<noun>` is unsafe because Grafana parses the leading letter as
an SI prefix (`si:fee` rendered "29.0 Mee"). Switched to `short`, the
convention already used by 42 panels; the noun is in each panel title.
- Stat panels showed a single fleet-wide tile labelled "Value #A". Grafana's
Loki backend does not name binary-operation frames from legendFormat, so the
ratio panel could not resolve per-node labels. Replaced it with a plain sum
of duplicate fetches; the ratio remains available as a timeseries. Added an
explicit legendFormat to all eight stat panels.
- The ledger-fetch regex required `good:N dupe:N` together, but
SHAMapAddNode::get() omits any zero counter. That dropped 32% of lines --
disproportionately the efficient fetches -- biasing the ratio upward. Each
field now matches independently.
- state-timeline used `spanNulls: true`, which spans gaps indefinitely and
would render an outage as continuous. Set to the 30-minute threshold.
- Stat panels relaid two-per-row and given a tooltip, per dashboard guidelines.
telemetry-runbook.md:
- LogQL examples used line filters where structured metadata exists.
`|= "ERR"` also matches the literal in a message body: measured 423 ERR
lines plus 4 DBG lines per 6h. Replaced with severity, partition, and
trace_id field filters.
- Documented the new dashboard, its debug-log requirement, and the LogQL traps
these fixes exposed.
- Corrected the dashboard count, which was already stale at eleven against
fifteen on disk, and named the six with no reference section.
These changes were developed on the phase-10 branch but belong to content this
branch and its upstreams introduced. Carrying them on phase-10 made its PR diff
report churn in files phase-10 does not own, and left each PR claiming a scope
that did not match its contents.
Moved here from phase-10 (identical content, no functional change):
- Dashboards: all 14 existing boards plus the new log-derived-insights board.
- Docs: telemetry-runbook.md (minus the workload/benchmark sections, which
describe phase-10 tooling) and the new telemetry-glossary.md.
- Grafana Cloud + Alloy export path: collector config, compose override, the
two .env examples and alloy/config.alloy.
- Local stack: otel-collector-config.yaml gains sub-millisecond and
second-scale spanmetrics buckets, pins unit=ms, and promotes
close_time_correct; integration-test.sh and TESTING.md follow.
- Node configs: exported_instance -> service_instance_id in comments; the
mainnet sample now logs at warning to bound log volume.
- Metrics code: Telemetry.cpp builds the metrics pipeline in the constructor
via initMetrics() so the global MeterProvider is published before any
subsystem creates a beast::insight instrument, and the histogram view keeps
each instrument's own name instead of collapsing them under one series.
MetricsRegistry gains a last_close_time gauge and skips negative job-queue
durations. OTelCollector drops an unused accessor.
- Naming CI: xrpl_work_item joins EXTERNAL_INFRA_LABELS and Rule E accepts the
dotted perf-iac resource-attribute form. This must travel with the
dashboards and runbook that reference those labels, or the rules fail.
- Doxygen input glob no longer recurses dot-directories.
Sections describing phase-10 tooling stay on phase-10 and keep their
"Future Enhancement" / "Planned, not yet implemented" markers here; phase-10
removes those markers when it lands the tooling.
ValidatedLedgerStale fired on every node, healthy included.
LedgerMaster::getValidatedLedgerAge() returns weeks{2} (1209600s) as a
SENTINEL when no validated ledger exists, not as a measurement. The rule
read that as "14 days stale". Measured over six days it produced sustained
firing on all nine nodes. Excluding the exact sentinel value drops that to
zero while still tracking real staleness.
ManifestFloodInbound at 50 kB/s was routine paging: ~41 sustained 5-minute
samples across six healthy nodes in six days. Healthy p99 is 1.0-1.8 kB/s
and real storms peak at 2.7 MB/s, so 512 kB/s sits ~280x above normal and
~5x below the peaks, cutting sustained samples to 2.
Both thresholds were previously justified from a 24-hour window, which was
too short to expose either problem.
Forty-eight panels spread over seven dashboards used the standard
description template but stopped after Function, leaving out Keywords,
Computation boundary and References. ledger-data-sync was the worst at
33 of 35 panels. They read as half-written next to their complete
neighbours, and the Keywords section is what defines the domain terms a
reader needs.
Fill in the three sections for each, in the order the already-complete
panels use. Keyword text is chosen from the panel's own metric family
rather than written per panel: job_* panels get the job-type and
deferred-job definitions, getobject_* and nudb_* get GetObject and
NodeStore, peer_finder_* and the traffic counters get peer and overlay,
and span-derived panels state that the value comes from the SpanMetrics
connector rather than a native metric.
Every panel now carries all ten sections. Panel count, layout, units and
queries are untouched.
Thirty-nine panels on this dashboard carry the full ten-section
description; twelve did not, so they read as half-written next to their
neighbours.
Eleven of them used the template but stopped after Function, missing
Keywords, Computation boundary and References. Fill those in, placed in
the same order the complete panels use. Content comes from each panel's
own metric: the job-queue panels get the job-type and deferred-job
definitions, I/O Latency and Database Sizes get the scheduler and
NodeStore ones, and each states whether its value is a native
beast::insight metric or computed through the OpenTelemetry SDK.
Server State had no template at all, only a one-line summary. Rewrite it
in full, documenting the 0-4 operating-mode encoding and its colour
mapping, and noting that it is an instantaneous state rather than a rate.
Panel count, layout, units and queries are untouched.
Three dashboards had layout faults that Grafana renders as visible
artifacts.
On Node Health, two row headers carried the same title as the panel
directly beneath them -- "Cache Hit Rates" and "Object Instance Counts"
each appeared twice, once as a row bar and once as a panel. Rename the
rows to "Caches" and "Object Instances"; the panels keep their names.
Worse on the same dashboard, seven bands summed to 48 columns on a
24-column grid, because the Extended Metrics block reused the y-space
already occupied by NodeStore I/O. Grafana cannot lay that out
deterministically, which is what produced the artifacts.
RPC Performance and Transaction Overview each had blank bands -- one
row and eight rows respectively -- left behind by earlier panel moves.
Repack all three grids so every band fits 24 columns, each band starts
where the previous one ends, and rows stay full-width single-height
anchors. Panel order, panel count and every panel's own width and
height are unchanged; only position moves.
Sixteen bargauges on ledger-sync-health are titled "... (Count ...)" and their
descriptions promise a count over the selected time range, but the queries
used last_over_time(counter[$__range]). For a monotonically increasing counter
that returns the newest cumulative sample -- the lifetime total -- so every
panel read the same number no matter how the time picker was set. Verified
live: dns_resolve_total over [15m] and [24h] both returned 12/5.
Switches the 17 affected targets to increase(), which is the range-scoped
delta the titles and descriptions already describe.
Panels: DNS Resolve Outcomes, Outbound Dial Outcomes, Handshake Negotiation
Failures, UNL Fetch Outcomes, Server Stall Episodes, Acquire Stalls, Acquire
Source, Inbound Peer Accept Outcomes, Peer Disconnects, Ledger/Object Serve
Refusals, Byzantine Ledger Jumps, Replay Fallback, Replay Outcomes, Pre-Accept
Quorum Shortfalls, Sweep Heap-Trim Faults, Rotation Node Re-Stores.
Two live-verified dashboard defects found while auditing the devnet nodes.
Sub-millisecond spanmetrics buckets. The ladder started at 1ms, but most
xrpld spans are far below that: tx.preflight means ~0.012ms and tx.preclaim
~0.15ms. On the live nodes the first bucket held 13,761,009 of 13,761,419
tx.preflight samples (99.997%), so histogram_quantile pinned every p95/p99
at a constant 0.95ms and ~10 latency panels reported a boundary artifact
rather than a measurement. Adds 0.01/0.05/0.1/0.25/0.5ms ahead of 1ms in all
three bucket lists, which must stay identical or the connector misbuckets.
consensus_mode filter on spans that never carry the label. Only
consensus.round and consensus.ledger_close are stamped with consensus_mode
(verified against live series), so panels 1, 3, 12 and 14 -- which query
consensus.validation.send, consensus.proposal.send and
consensus.accept.apply -- blanked as soon as an operator picked any value
from the Consensus Mode dropdown. Drops the selector from those four; ids 2
and 11 keep it because their spans do carry it.
Note: existing histogram series retain the old le boundaries, so
span-latency panels show a discontinuity at collector restart.
Grafana renders `unit: "ops"` as the literal string "ops/s" and `cps`
as "counts/s", so seven rate panels on this dashboard read as
"operations per second" regardless of what they counted -- nodes added
to a SHAMap, dial attempts, ledger-data requests served.
Replace them with custom-suffix units naming the quantity, following
the `suffix:/hr` and `si:drops` precedent already in this repo. Each
noun was checked against the emitting code rather than the panel title:
sync_addnode_total counts SHAMap nodes, peer.dial spans are connect
attempts, ledger.serve spans are peer requests.
Also spell out "Ops / Sec" as "Operations / Sec" on the NodeStore panel,
since abbreviating the noun defeats the point of naming it.
Queries are unchanged; the values were already correct.
The two key-job latency panels on node-health declared milliseconds
while querying `job_running_us` / `job_queued_us`, which record
microseconds. Every reading was a thousand times too large: the p95 for
acceptLedger, 241495us, rendered as "241 sec" instead of 241ms.
job-queue.json already read the same metrics as microseconds, so the
two dashboards disagreed by 1000x on identical data.
Also replace the generic `ops`, `cps` and `cpm` units, which Grafana
renders as the literal "ops/s", "counts/s" and "counts/min", with
custom-suffix units naming what each panel counts -- jobs, ledgers,
validations, lookups. The per-minute panels already scale correctly in
their queries; only the noun was missing.
Normalise the micro sign to U+00B5 throughout. Four axis labels used
the visually identical Greek mu, U+03BC, while every unit field used
the micro sign.
Ledger Close Rate plots ledgers closed beside ledger fetches, which one
unit cannot describe; give the fetches series its own unit on a
right-hand axis. Drop two field overrides on NodeStore Read Latency
that restated the panel unit as a custom suffix -- a suffix is appended
verbatim, so it would have suppressed magnitude scaling and left large
values unreadable, the same defect fixed above.
Queries are unchanged apart from the transaction path piechart, which
moves from rate() to increase(): a per-slice "per second" reading is
not a share of a total.
Alongside, widen the Complete Ledger Ranges table to full width, hoist
the stat panels above the fold, and bring the touched panels up to the
tooltip and null-spanning guidelines.
The four job-latency panels on node-health declared milliseconds while
querying `job_running_us` / `job_queued_us`, which record microseconds
(MetricsRegistry records the raw value, and the instrument description
says microseconds). Every reading was therefore a thousand times too
large: the p95 for acceptLedger, 241495us, rendered as "241 sec"
instead of 241ms. job-queue.json already read these same metrics as
microseconds, so the two dashboards disagreed by 1000x on identical
data. Switch node-health to microseconds to match.
Also replace the generic `ops` and `cps` units, which Grafana renders
as the literal "ops/s" and "counts/s", with custom-suffix units naming
what each panel counts -- messages, fetches, calls, mismatches.
Two panels plot more than one quantity on a single axis, which no
single unit can describe. Give each series its own unit through field
overrides: reads per second beside two queue depths on NuDB Read
Pressure, and ledgers beside fetches on Ledger Close Rate, the latter
on a right-hand axis.
State Duration Rate plots a seconds-per-second time share, which can
exceed 1.0 and so is not a percentage; label it as the ratio it is.
The normalised share already exists as its own panel.
Queries are unchanged; the values were already correct.
Alongside, bring the touched panels up to the dashboard guidelines and
hoist the stat panels above the fold.
Grafana renders `unit: "ops"` as the literal string "ops/s", so every
rate panel read as "operations per second" regardless of what it
actually counted. `Ledger Build Rate` showed "0.258 ops/s" where the
value is one ledger every 3.9s -- the number was right, the unit was
meaningless.
Replace the generic units with Grafana custom-suffix units naming the
quantity, following the existing `suffix:/hr` and `si:drops` precedent
in this repo. Nine of these are `stat` panels with no axis, so the unit
string was the only text a reader ever saw.
Also switch the two trusted/untrusted piecharts and the transaction
path piechart from rate() to increase(): a per-slice "per second"
reading is not a share of a total, counts in the window are.
Queries are unchanged apart from those three; the values were already
correct.
Alongside, bring the touched panels up to the dashboard guidelines:
tooltip mode/sort/max-height, 30-minute null spanning, and axis labels
in title case. Hoist the stat panels above the fold on
ledger-operations and rpc-performance.
Panels that a later branch in this chain removes are deliberately left
alone -- fixing them would only add merge conflicts.
Someone filling in .env.grafanacloud-alloy has no reason to know that
deployment.environment and xrpl.network.type are not env vars: they are
literals inside the tier transform, because OTTL statements do not expand
sys.env(). Left alone they ship as prod/mainnet and every dashboard
environment and network filter reads the wrong tier. The example now says
so, and notes service_instance_id needs nothing here.
Alloy declared 24 spanmetrics dimensions where both collector configs
declare 28. The four missing ones -- timed_out, object_type,
validation_status and accept_gated -- are emitted by the code and queried
by ledger-sync-health panels "Ledger Serve Rate by Object Type" and
"Trusted Validation Accept Rate by Status", so on an Alloy deployment
those panels collapsed to one undifferentiated series.
deploy-run-otel-xrpld.md already requires the dimension sets to stay in
sync; the config had drifted from it. Alloy's other differences from the
collectors (no filelog, no tail_sampling) are deliberate -- it is
cloud-only by design.
Two gaps meant the last failure produced no evidence of its cause.
The node-log upload was gated on `if: failure()`, but the validation step
sets continue-on-error, so the job is not failing at that point and the
condition never fired. Every failed run silently skipped the one artifact
that records why a node did not reach consensus. It now keys on the
validation step's own outcome, and also collects the harness logs.
Transaction failures were logged at DEBUG, which CI does not enable, so a
run where all 3052 submissions failed on a refused connection reported
nothing about it. The first occurrence of each distinct failure kind is now
a warning and repeats stay at DEBUG, so one refused connection says so once
instead of 3052 times.
