The stat block was relaid two-per-row in the previous commit, but the reflow
shifted subsequent panels by a delta computed against the block's old height.
The block shrank, so the delta left a 12-grid-line dead band between the stat
row and "Node Operating State Transitions".
Recompute every panel's y by walking the list in order and pairing half-width
panels, rather than patching offsets incrementally. The layout is now derived
from panel order and heights, so this class of drift cannot recur.
Verified: 192 grid lines, fully covered, no empty band, no overlaps, and no grid
line occupied by more than two panels.
Fixes found while validating the Log-Derived Insights dashboard against live
Loki and Tempo. Every change was verified by executing the query, not by
inspection.
loki.yaml — the ConsensusLedgerHash derived field never matched anything:
- The TraceQL query named `.xrpl.consensus.ledger_id`. The attribute is
`consensus_ledger_id` (ConsensusSpanNames.h); live Tempo lists no dotted or
xrpl-prefixed variant among its span attributes.
- The bare `.` intrinsic scope cannot match a span attribute regardless of
name; TraceQL requires `span.`.
Corrected to `{span.consensus_ledger_id="..."}`, which returns the expected
consensus.round trace for a real ledger hash.
log-derived-insights.json:
- Stat units `suffix: <noun>` rendered raw integers, so large values printed
in full. `si:<noun>` is unsafe because Grafana parses the leading letter as
an SI prefix (`si:fee` rendered "29.0 Mee"). Switched to `short`, the
convention already used by 42 panels; the noun is in each panel title.
- Stat panels showed a single fleet-wide tile labelled "Value #A". Grafana's
Loki backend does not name binary-operation frames from legendFormat, so the
ratio panel could not resolve per-node labels. Replaced it with a plain sum
of duplicate fetches; the ratio remains available as a timeseries. Added an
explicit legendFormat to all eight stat panels.
- The ledger-fetch regex required `good:N dupe:N` together, but
SHAMapAddNode::get() omits any zero counter. That dropped 32% of lines --
disproportionately the efficient fetches -- biasing the ratio upward. Each
field now matches independently.
- state-timeline used `spanNulls: true`, which spans gaps indefinitely and
would render an outage as continuous. Set to the 30-minute threshold.
- Stat panels relaid two-per-row and given a tooltip, per dashboard guidelines.
telemetry-runbook.md:
- LogQL examples used line filters where structured metadata exists.
`|= "ERR"` also matches the literal in a message body: measured 423 ERR
lines plus 4 DBG lines per 6h. Replaced with severity, partition, and
trace_id field filters.
- Documented the new dashboard, its debug-log requirement, and the LogQL traps
these fixes exposed.
- Corrected the dashboard count, which was already stale at eleven against
fifteen on disk, and named the six with no reference section.
The Phase 7 section existed twice in this file; the copies disagreed on this
one label. Consolidating on the single copy kept the older wording, but Jaeger
was removed from the project earlier in this chain, so Tempo is the only trace
backend the diagram should name.
These changes were developed on the phase-10 branch but belong to content this
branch and its upstreams introduced. Carrying them on phase-10 made its PR diff
report churn in files phase-10 does not own, and left each PR claiming a scope
that did not match its contents.
Moved here from phase-10 (identical content, no functional change):
- Dashboards: all 14 existing boards plus the new log-derived-insights board.
- Docs: telemetry-runbook.md (minus the workload/benchmark sections, which
describe phase-10 tooling) and the new telemetry-glossary.md.
- Grafana Cloud + Alloy export path: collector config, compose override, the
two .env examples and alloy/config.alloy.
- Local stack: otel-collector-config.yaml gains sub-millisecond and
second-scale spanmetrics buckets, pins unit=ms, and promotes
close_time_correct; integration-test.sh and TESTING.md follow.
- Node configs: exported_instance -> service_instance_id in comments; the
mainnet sample now logs at warning to bound log volume.
- Metrics code: Telemetry.cpp builds the metrics pipeline in the constructor
via initMetrics() so the global MeterProvider is published before any
subsystem creates a beast::insight instrument, and the histogram view keeps
each instrument's own name instead of collapsing them under one series.
MetricsRegistry gains a last_close_time gauge and skips negative job-queue
durations. OTelCollector drops an unused accessor.
- Naming CI: xrpl_work_item joins EXTERNAL_INFRA_LABELS and Rule E accepts the
dotted perf-iac resource-attribute form. This must travel with the
dashboards and runbook that reference those labels, or the rules fail.
- Doxygen input glob no longer recurses dot-directories.
Sections describing phase-10 tooling stay on phase-10 and keep their
"Future Enhancement" / "Planned, not yet implemented" markers here; phase-10
removes those markers when it lands the tooling.
Update the catalogue table and prose for the two thresholds corrected by the
7-day backtest, and document why ValidatedLedgerStale must exclude the
1209600s sentinel that getValidatedLedgerAge() returns when no validated
ledger exists — that clause looks redundant and would otherwise be removed
as a simplification.
Also record the tuning lesson: justify thresholds from a 7-day sample, not
24 hours.
ValidatedLedgerStale fired on every node, healthy included.
LedgerMaster::getValidatedLedgerAge() returns weeks{2} (1209600s) as a
SENTINEL when no validated ledger exists, not as a measurement. The rule
read that as "14 days stale". Measured over six days it produced sustained
firing on all nine nodes. Excluding the exact sentinel value drops that to
zero while still tracking real staleness.
ManifestFloodInbound at 50 kB/s was routine paging: ~41 sustained 5-minute
samples across six healthy nodes in six days. Healthy p99 is 1.0-1.8 kB/s
and real storms peak at 2.7 MB/s, so 512 kB/s sits ~280x above normal and
~5x below the peaks, cutting sustained samples to 2.
Both thresholds were previously justified from a 24-hour window, which was
too short to expose either problem.
Three dashboards had layout faults that Grafana renders as visible
artifacts.
On Node Health, two row headers carried the same title as the panel
directly beneath them -- "Cache Hit Rates" and "Object Instance Counts"
each appeared twice, once as a row bar and once as a panel. Rename the
rows to "Caches" and "Object Instances"; the panels keep their names.
Worse on the same dashboard, seven bands summed to 48 columns on a
24-column grid, because the Extended Metrics block reused the y-space
already occupied by NodeStore I/O. Grafana cannot lay that out
deterministically, which is what produced the artifacts.
RPC Performance and Transaction Overview each had blank bands -- one
row and eight rows respectively -- left behind by earlier panel moves.
Repack all three grids so every band fits 24 columns, each band starts
where the previous one ends, and rows stay full-width single-height
anchors. Panel order, panel count and every panel's own width and
height are unchanged; only position moves.
Resolves the telemetry-startup conflict between the two branches. Both
sides move the telemetry start earlier in setup(); they disagree only on
how far the pipeline had been split at that point.
phase-1b (arriving) moved nodeIdentity_, setServiceInstanceId() and the
telemetry start up to just after the wallet DB is proven usable. phase-9
had split the metrics pipeline in two and left its copy of that block at
the old, later position.
Kept both intentions: the block stays at phase-1b's early position, and
metricsRegistry_ construction moves up with it so it precedes
startTelemetry() -- the metrics half is guarded on the registry existing,
so leaving the construction behind would have started tracing while
silently skipping metrics. phase-9's later copy is dropped as the stale
duplicate. The two-phase split is preserved: startTelemetryGauges() still
runs after overlay_ is constructed, because the observable callbacks read
it and getOverlay() asserts.
Net effect is that the metrics provider now starts earlier than on either
branch, and still before beginConsensus() emits the first spans and the
only operating-mode transition.
base_fee_xrp was observed from LoadFeeTrack::getLocalFee(), which is the
local load-scaled fee escalation, not the ledger's base fee. The panel
built on it therefore tracked this node's load state rather than the
network's cost of a reference transaction, and read as a flat line
whenever the node was unloaded.
Read it from the validated ledger's fee settings instead, alongside the
reserve values already taken from there. The observation now only
reports when a validated ledger is available, which is correct: before
that there is no network fee to report.
MetricsRegistry::start() fused three steps with different prerequisites:
the exporter/provider and the synchronous instruments need only config
strings, while the observable gauges read live Application services. The
whole pipeline therefore waited on the latest prerequisite and ran near
the end of setup() -- after beginConsensus() had already recorded the
process's only operating-mode transition. state_changes_total was
emitted into a pipeline that did not exist yet, so the series never
appeared at all.
Split the two halves. start() keeps the provider and the synchronous
instruments and now runs as soon as the registry is constructed; the new
startAsyncGauges() registers the observable callbacks and runs once
overlay_ exists, still before the first consensus round. Application
gains a matching startTelemetryGauges() so each phase has its own call
site and its own precondition.
Move the jq_trans_overflow_total observable out of initSyncInstruments()
into the gauge phase. Its callback reads getOverlay(), which asserts
overlay_ is non-null, so creating it in the early phase armed the reader
thread against a half-built application -- an assert is not caught by
the callback's catch-all. The instrument is an observable counter rather
than a gauge, which is how it was mistaken for a push-only instrument.
Both start log lines are kept, one per phase, because that timeline is
what made the original ordering bug diagnosable.
Comments and preconditions are corrected to state the rule rather than
the current arrangement: start() may only create instruments whose
values are pushed, and any observable whose callback reads a service
belongs in the gauge phase. The gauge precondition now lists the
services the callbacks actually read.
telemetry_->start() ran at the end of ApplicationImp::start(), after
overlay_->start(). Spans are emitted well before that, during setup():
beginConsensus() runs the first consensus round there. SpanGuard drops a
span whenever the global Telemetry instance is not yet live, so that
round's spans were never recorded.
Move the start into setup(), behind a new startTelemetry() seam, right
after the node identity is known. getNodeIdentity() needs only the
cmdline, the config, or the wallet DB, and initRelationalDatabase()
has already created the latter -- the adjacent peerReservations_ load
proves it is usable -- so the identity block moves up with it.
The new position is bounded on both sides: after initRelationalDatabase()
because the identity needs the wallet DB and a DB failure aborts setup(),
and before beginConsensus() because that emits the first spans.
The two key-job latency panels on node-health declared milliseconds
while querying `job_running_us` / `job_queued_us`, which record
microseconds. Every reading was a thousand times too large: the p95 for
acceptLedger, 241495us, rendered as "241 sec" instead of 241ms.
job-queue.json already read the same metrics as microseconds, so the
two dashboards disagreed by 1000x on identical data.
Also replace the generic `ops`, `cps` and `cpm` units, which Grafana
renders as the literal "ops/s", "counts/s" and "counts/min", with
custom-suffix units naming what each panel counts -- jobs, ledgers,
validations, lookups. The per-minute panels already scale correctly in
their queries; only the noun was missing.
Normalise the micro sign to U+00B5 throughout. Four axis labels used
the visually identical Greek mu, U+03BC, while every unit field used
the micro sign.
Ledger Close Rate plots ledgers closed beside ledger fetches, which one
unit cannot describe; give the fetches series its own unit on a
right-hand axis. Drop two field overrides on NodeStore Read Latency
that restated the panel unit as a custom suffix -- a suffix is appended
verbatim, so it would have suppressed magnitude scaling and left large
values unreadable, the same defect fixed above.
Queries are unchanged apart from the transaction path piechart, which
moves from rate() to increase(): a per-slice "per second" reading is
not a share of a total.
Alongside, widen the Complete Ledger Ranges table to full width, hoist
the stat panels above the fold, and bring the touched panels up to the
tooltip and null-spanning guidelines.
