Two conflicts, both additive-vs-additive; each resolution keeps both sides.
check_otel_naming.py -- phase-10 taught the L6 label extractor to match the
label MAP first and to resolve a key hoisted into a `k...Label` constant,
scanning headers as well as sources. Our side had added the two-regex
first/subsequent literal scan and the `metric_constants(root)[1]` union that
covers the `namespace label` header style.
Kept phase-10's mechanism whole: METRIC_LABEL_MAP + the `(?:^|\{)` key regex
already subsumes what METRIC_LABEL_NEXT did, since matching inside the map body
makes every pair after the first open with a single `{`. So METRIC_LABEL_NEXT is
dropped as genuinely redundant rather than kept as a duplicate scan, and the
reason it existed is folded into METRIC_LABEL's comment. Re-added our
`metric_constants(root)[1]` union on top: LABEL_CONST_DEF only matches
`k`-prefixed identifiers, so it cannot see MetricNames.h's `label::jobType`
style, and without that union Rule D would reject dashboards querying labels
Rule I forced into constants. The two derivations are complementary and both
are now documented as such.
MetricsRegistry.cpp -- both sides added a new sibling view-registration helper
next to addMicrosecondHistogramView, and both added a registration call in
initExporterAndProvider(). Kept all four helpers
(addHistogramView/Microsecond/RoundDuration/SubMillisecond) and every
registration: phase-10's addSubMillisecondHistogramView + kNodeStoreReadUs
alongside our addRoundDurationHistogramView, sweepMallocTrimUs and the two
millisecond dial/resolve ladders.
phase-10's nodestore_read_us histogram does not duplicate our work. The
nodestore_latency gauge that would have overlapped it was retired in c4e434d520
before this merge, and the surviving nodestore_state gauge is complementary
rather than duplicative: both read the same fetch measurement, but the gauge
publishes only a since-boot mean via scaledMean() and cannot yield a
percentile -- the consequence observeNodeStoreTotals' own docs state plainly --
while the histogram buckets each fetch and can. The histogram also splits by
fetch_type and found, which the gauge cannot. phase-10 registered its
explicit-bucket View, so it does not inherit the SDK default ladder.
Each file keeps its own existing naming style: phase-10's k-prefixed constants
are left as-is, ours stay namespaced.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Second pass on the 54-panel audit. The first commit handled the count panels and
the description drift; these are the query and threshold defects.
Panels 4 and 6 (DNS Resolve / Outbound Dial p95) read NaN for the whole run.
Two faults compounded: both hard-coded [5m] instead of $__rate_interval, so they
ignored the dashboard time range entirely, and both wrapped the histogram buckets
in rate() even though DNS resolution and outbound dialling only happen during
startup -- a windowed rate over a series that stopped moving is 0/0. Dropping
rate() and reading the cumulative buckets gives the real distribution: p95 900 ms
for DNS, 1750 ms for dialling. Panel 52 kept its rate() (consensus rounds are
ongoing) but its hard-coded [5m] became $__rate_interval.
Panel 15 plotted 105,892,534 "ledgers behind" during the flagship window. The
underlying cause is in NetworkOPs.cpp -- getLedgersBehindNetwork() subtracts the
validated sequence from a networkTarget of 0 before any peer has reported -- and
that still needs a code fix. Meanwhile one sentinel spike flattened the real
0-20 backlog for the rest of the window, so the query now clamps at 1e6, far
above any true backlog. Reads 4 where it read 105 million.
Panel 17's sum by (from, to) dropped node identity, so with All nodes selected
every node's transitions summed into one bar. It now carries service_instance_id,
xrpl_branch and xrpl_work_item like every other panel.
Panel 38 divided by clamp_min(op_rate, 1), which turns "no operations in this
interval" into "one operation", reporting the whole duration total as if a single
op had consumed it. Replaced with a `> 0` gate so an idle interval draws a gap
instead of a fabricated latency.
Panels 13 and 45 had inverted thresholds: green began at 1 second, so every
sub-second time-to-full and time-to-first-validated rendered red -- the healthy
case was the alarming colour. Now green by default, yellow past 10 minutes, red
past 30.
Panel 48's p95 had no outcome filter, mixing abandoned and timed-out spans (which
sit at the retry ceiling by construction) into what reads as completion latency.
Restricted to outcome="complete".
Verified against Grafana Cloud: 66 queries, 0 parse errors, 56 with data, 9
legitimately empty (fault-only counters plus the write-timing series that
pre-dates both recorded nodes). The single remaining NaN is on panel 47 and is an
artifact of my verification harness forcing a hard 5m window; the panel's own
$__rate_interval returns 492 ms, so it needs no change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A 54-panel audit across 11 dimensions, each finding adversarially re-verified
against live data, returned 38 confirmed defects. This fixes them. Most were
introduced by the recent rate-to-count conversion itself.
1. increase() was the wrong function for these counters. A counter that only
moves at startup is born at its final value inside the window and never
rises, so increase() reports 0. Measured: dns_resolve_total reads 4 but
round(increase(...[$__range])) returned 0 -- the panel lost the signal
entirely. increase() also drops whatever accrued before the window opened,
which under-reported the rest (overlay_connect_total 110 against a true 130,
unl_fetch_total 10 against 14, peer_disconnect_total 7 against 8).
All 18 count panels now use last_over_time(...[$__range]), which on a
fresh-node board is the cumulative total since the process started -- exactly
what "count" means here.
EXCEPT panel 49. span_calls_total comes from the collector's spanmetrics
connector, which is collector-side state and does NOT reset when xrpld
restarts, so last_over_time would report the collector's lifetime across every
run: it read 2624 header completions where run C actually had 297. That panel
keeps round(increase(...)) and now reports 297/278/212/7, matching the
analysis. The distinction is process-level counter vs collector-side counter,
and it decides which function is correct.
2. Descriptions still described rates after the conversion, over three passes of
wording (Reading it / Healthy range / Watch for blocks, "Rate of", "per
second", "/s", "a rising rate"). 11 panels corrected; the two surviving uses
of "rate" are legitimate (a cache hit-rate reference, and panel 49 explaining
why a count reads better than a rate).
3. Ten descriptions pointed at panel titles that no longer exist, because the
conversion renamed the panels they cross-referenced. Two others named panels
that never existed on this board at all: "Total Jobs Queued" (now Worker Pool
Capacity & Total Backlog, panel 27) and "Fetch-Pack Peer Starvation" (now
Peers Able to Serve Needed Sequence, panel 28).
4. Panels 18 and 23 applied $acquire_metric on top of a hard-coded metric
selector, so the two ANDed: any selection outside the panel's own values gave
an empty graph and All was the only usable state. The redundant template
selector is gone; the panel's metric pair is its identity.
5. Panel 23 drew two series with different ranges (received_data_depth 0-20,
in_flight 13-49) under one yellow threshold at 16, so in_flight was
permanently yellow. The threshold is now scoped to received_data_depth.
6. The "Spans & traces" row sat at y=248, the same y as panels 38/39, so Grafana
folded the Back-fill panels into the wrong row. Moved to y=296, below the last
back-fill panel. Rows are now strictly ascending with no collision.
Verified against Grafana Cloud Prometheus: all 66 panel queries parse, 0 errors,
57 returning data (up from 56 -- panel 3 was one of the ones increase() had
silenced). validate_dashboards and check_otel_naming both pass.
Not verified: no PNG renders this round; the local Grafana and Prometheus are
down, so every check ran against Cloud data via the datasource proxy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A stalled ledger sync has two distinct causes that look identical from the
job queue: in both, the ledgerData lane sits at its concurrency limit of 3
with work waiting. Lane occupancy therefore discriminates nothing, and the
dashboard had no panel that separated the two.
Adds a Sync Bottleneck Discrimination row to Ledger Data & Sync with seven
panels built on the nodestore_state write-path and acquire counters:
- NodeStore Read Latency, lifetime and windowed, on a log axis with
threshold lines at 10 and 100 us. This is the discriminator: fast reads
mean the cost is on the write side, slow reads with a high hit ratio
mean every access is paying disk latency.
- NuDB Writer Queue Depth, the confirming half. NuDB takes one global
mutex per insert, so depth above 1 is queueing at that mutex.
- NuDB Insert Time, mean against true max, which quantifies how much of
each insert is wait rather than work.
- Acquire Deferrals vs Timeouts on one panel, because the livelock
fingerprint is deferrals rising while timeouts stay flat, and neither
counter shows that alone.
- Acquisition Progress and Discarded Acquire Work, the outcome side: no
completions while the lane is busy means acquisitions resolve neither
way.
- NodeStore Read Latency Distribution split by fetch_type and found, from
the nodestore_read_us histogram, since a mean cannot tell a uniformly
warm store from a mostly-warm one with a cold tail.
Every expression carries the existing service_instance_id and tier filters.
The histogram introduces two new label dimensions, so fetch_type and found
template variables are added to match.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
nodestore_latency published six values that nodestore_state already
publishes from the same Database accessors, so the two gauges were
duplicate readings of the same atomics:
write_count -> node_writes getStoreCount()
read_count -> node_reads_total getFetchTotalCount()
write_duration_us -> node_writes_duration_us getStoreDurationUs()
read_duration_us -> node_reads_duration_us getFetchDurationUs()
write_mean_us -> write_mean_us store duration / count
read_mean_us -> read_mean_us fetch duration / count
nodestore_state is kept because its means go through scaledMean(), which
saturates at INT64_MAX instead of wrapping and omits a mean when the
denominator is zero rather than reporting a misleading 0 us.
Removes registerNodeStoreLatencyGauge, its instrument member, the
metric::nodestoreLatency constant and the lval::nodestore_latency label
namespace. The gauge-over-histogram rationale and the "p99 is not
obtainable" consequence are folded into observeNodeStoreTotals' docs.
Retargets the gauge-contract test onto nodestore_state rather than
deleting it: the scaledMean arithmetic is covered by the static_asserts
in tests/libxrpl/telemetry/MetricsRegistry.cpp, but nothing else asserts
that these named series multiplex onto one instrument keyed by `metric`.
The test now calls the production scaledMean instead of a copy of the
division, and its sub-microsecond case asserts scaledMean's actual
behaviour (a genuine mean of 0 on a zero numerator with a non-zero
count), which differs from the retired gauge's extra numerator guard.
Rewrites both ledger-sync-health copies' panel 38/39 queries and drops
the obsolete claim that the write numerator was never written: all three
concrete store paths call recordStoreDuration, so write_mean_us is live
on an ordinary node. The same stale [import_db] caveat is removed from
the runbook, the 09 reference row and the workload validator's note.
Two panels were left out of the previous sweep because a blanket rate-to-count
rewrite would have been wrong for them. Handled per target here.
Panel 65 (Sweep Heap-Trim) carries two cumulative counters, minor faults and
reclaimed KB. Both are count-shaped, so it becomes a bargauge over
round(increase(...[$__range])) like the rest, and the legends drop their
"/ Sec" suffix now that the values are totals rather than rates.
Panel 66 (Online-Delete Rotation) stays a timeseries. Its two targets are not
the same kind of thing: target A reads rotation_state{in_flight}, a 0/1 flag
whose whole value is seeing when it is high and for how long, and target B
rates rotation_state{copy_forward}, a cumulative write total. A count bargauge
would destroy the flag's time dimension. Instead the shared "cps" unit -- wrong
for a flag -- is replaced by per-target overrides: the flag pinned to a 0..1
left axis, the write rate on a right axis in cps. This matches how the metric
is documented to be read (MetricNames.h:648): copy_forward climbing while
in_flight is 1 is expected, climbing while it is 0 means the flag leaked.
Verified against Grafana Cloud Prometheus: 0 parse errors on both panels.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seventeen panels on ledger-sync-health drew low-frequency discrete counters as
timeseries with an ops/s unit. The rate is arithmetically right but unreadable:
"0.1394 ops/s" is 278 abandoned tree phases over 33 minutes, and no reader can
recover the 278. rate() also extrapolates, so whole events rendered as
fractions -- peer disconnects showed 8.008 for 8 actual disconnects.
Measured every candidate against live Prometheus over a 33-minute mainnet sync
before converting; none exceeded 0.48 events/s, so all are count-shaped:
serve_refused_total 955 events 0.4785/s
ledger_quorum_shortfall_total 670 0.3354/s
sync_acquire_source_total 511 0.2559/s
overlay_connect_total 130 0.0652/s
sync_acquire_no_progress_total 109 0.0545/s
unl_fetch_total 12 0.0061/s
peer_disconnect_total 8 0.0040/s
Each becomes a bargauge over round(increase(...[$__range])) with unit short and
decimals 0, matching the existing Mode Transitions panel. Legends follow
instructions.md OTel rule 7 -- "MetricName [labels]" -- and keep node identity
via xrpl_ident, so a nine-node view no longer collapses to one bar. Panel 49
also gained node identity in its aggregation, which it was missing entirely.
Label values stay as emitted (upgrade_fail, not "Upgrade Fail"). They are wire
identifiers: rewriting them in the legend would hide what the metric reports and
break silently when a new value appears. The Title Case sits in the metric name.
The five zero-valued panels were checked rather than assumed dead --
ledger_replay_*, sweep_malloc_trim_* and rotation_copy_node_restore_total all
have real emit sites and are feature-gated off in this configuration.
Verified against Grafana Cloud Prometheus: 0 parse errors across all panel
queries, whole-number results throughout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three defects, all found by rendering the panels and running their queries
against a live mainnet node.
1. Ledger Acquire Phase Outcomes had an invalid PromQL escape.
The saved JSON held "ledger\\.acquire\\.(.*)", which decodes to
`ledger\.acquire\.(.*)`. In a PromQL double-quoted string `\.` is not a
legal escape, so Prometheus rejected the whole query:
parse error: unknown escape sequence U+002E '.'
The panel therefore rendered an error badge and "No data". A PromQL string
needs two characters, so the JSON must carry four backslashes.
2. The same panel never grouped by outcome, despite its title.
`sum by (span_name, timed_out, ...)` omitted `outcome`, so complete and
abandoned collapsed into one line. Measured at 16:45 UTC that hid a 29x
difference: astree complete=2085 against abandoned=71, all drawn as a
single indistinguishable series -- and every phase then showed the same
0.2596/s value, which is what made the panel look meaningless.
Now grouped by outcome, giving four real series (verified live):
header complete 0.1439/s, header abandoned 0.0772/s,
astree abandoned 0.1404/s, txtree abandoned 0.1404/s
The selector moves from timed_out (always "false" here, so it carried no
information and its filter var was redundant) to the declared
$span_outcome. Legend becomes "<phase> <outcome>"; axis label reads
"Phases / sec" to match the ops unit.
3. Seven stat panels and one heatmap dumped the raw label set as the legend.
With no fieldConfig.defaults.displayName but textMode "value_and_name",
Grafana has no name to show and falls back to printing every label:
{deployment_environment="local", exported_instance="xrpld-mainnet",
exported_job="xrpld", instance="otel-collector:8889", ...}
Rendered PNGs of panels 10 and 26 confirmed it. Fixed on ids 10, 12, 13,
14, 26, 36, 45 and 52 with the board convention already used by 24 sibling
stat panels: "${__field.labels.series} ${__field.labels.xrpl_ident}".
Verified afterwards by executing all 66 panel queries on this board against
live Prometheus: 0 parse errors, 56 returning data. The 10 empty ones are
counters a healthy node never increments plus panel 29, which is gated to stay
blank until a ledger is validated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Peer Ledger Supply Window panel drew supply_min_seq, supply_max_seq and
nothing else on one linear axis. Measured on a mainnet node, those sit around
105,890,000 and roughly 300,000 apart, so the 588-ledger tip movement that
shows whether sync is progressing was 0.0006% of the axis and read as a flat
line. unit "none" also printed the sequences unabbreviated and clipped the
legend.
The panel's own "Watch for" text asked the reader to compare supply_min_seq
against this node's validated sequence, but that line was not on the panel at
all, so the comparison meant switching dashboards.
Plot the two distances instead, which is what the panel was always asking
about:
History Headroom = validated_ledger_seq - supply_min_seq
Tip Gap = supply_max_seq - validated_ledger_seq
Zero is now the boundary in both directions: negative headroom is exactly the
"every peer pruned what I still need" case the description warns about, and it
becomes a zero crossing rather than a line-order comparison. Tip Gap gets the
right-hand axis because the two ranges differ by orders of magnitude
(measured: 299999..300001 against -1..1).
Both operands are gated `> 0`. Ungated, differencing the documented
"unknown window" sentinel of 0 yields the whole sequence space: measured
-105854935 for headroom and 105890295 for tip gap during the first ticks,
which destroys the axis for the rest of the window. Gated, the panel stays
blank until the node has a validated ledger and a peer has advertised a
range, which is the honest reading for that state.
Both queries verified against a live mainnet node through the full template
substitution: refId A = 300001 legend "History Headroom [xrpld-mainnet]",
refId B = -1 legend "Tip Gap [xrpld-mainnet]".
Runbook branch-C table, step 11 walkthrough and the 09 reference row follow
the rename and the new reading.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The panel built its legend with label_replace from the `from` label alone, so
every edge leaving a mode collapsed onto one series: connected->syncing and
connected->full both drew as "connected". The whole point of the from/to pair
is to tell a healthy climb from flapping, and that was exactly what the
legend hid.
label_replace cannot concatenate two labels. label_join can, which is the
pattern the consensus board already uses for its multi-label legends, so the
series now reads "from -> to".
Also regenerates the Grafana Cloud copy of the board.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two suspects from the 3.3.0 slowdown investigation had no signal. Both were
already computing the numbers and throwing them away, so this exposes them
rather than adding measurement.
Per-sweep heap trim. The trim runs after every cache sweep, and its cost
scales with resident heap, so it is the leading explanation for a node with
a populated database syncing slower than a fresh one. The report already
carried duration, fault deltas and reclaimed pages, but the whole
measurement sat behind a debug-journal check, so an ordinary node measured
nothing, and the call site discarded the result. The measurement now always
runs and only the log line stays gated. Records trim duration, minor faults
and reclaimed kilobytes. Measured cost of the always-on path is about six
microseconds per sweep against a trim costing milliseconds, at a cadence of
ten to a hundred and twenty seconds.
Honest limit, stated in the runbook: the fault delta spans only the trim
call, so it shows the trim itself faulting but not the faults that follow as
caches refill. The duration is the signal to correlate against sweep-job
queueing.
Rotation writes. Rotation copies archive-served reads forward and re-stores
nodes missing from both backends, both of which compete with sync I/O and
only happen on a populated online_delete database. The copy-forward count
existed but was reset by the rotation's own log line, so a metric reading it
would drop to zero on every swap; a never-reset total sits beside it now.
The re-store count was not measured at all. Rotation duration is
deliberately not recorded: the health throttle sleeps at eight points inside
the sequence and dominates exactly when the node is unhealthy, so the number
would conflate work with waiting.
Nothing added for the other two suspects. Get-object serving is already
covered by the handler label, the lookup histogram and the deferred and
saturation gauges; peer churn by the disconnect-reason counter.
Also replaces nine per-file cspell ignores with one ignoreRegExpList entry
for the telemetry macro names, and picks up the levelization baseline for the
consensus span-name test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by reviewing what each metric actually measures, with attention to the
derived and bucketed ones. All five could report healthy while the node was
not, or the reverse.
- The nodestore latency panel took rate() of a mean. The gauge already
divides duration by count in code, so rating it produced a figure with no
unit, and Prometheus discards a gauge's decreases, so a heavy back-fill
read as roughly zero microseconds per operation. The cumulative duration
totals are now exported alongside the means, and the panel divides the
rate of the total by the rate of the count, which is the latency over the
panel's own window rather than a since-boot average that flattens with
uptime.
- The DNS-resolve and outbound-dial histograms had no explicit buckets, so
they inherited a ladder that stops at ten seconds while the dial timer is
fifteen. Every timed-out dial fell in the overflow bucket and p95 read
exactly ten seconds however bad it got. Both now have a ladder reaching
thirty seconds with fifteen on its own boundary, so a timeout is
distinguishable from merely slow.
- The missing-node counts only cleared when a tree completed, so a
timed-out or failed acquire left its last count latched. Since the gauge
reports the maximum across everything still in the collection, and
eviction waits on a grace period plus the sweep interval, a finished node
reported as stuck for minutes. That inverts the one signal that separates
stuck from slow. Cleared unconditionally on the terminal path instead.
- A disabled quorum published a sentinel so large that, on a timeseries
axis shared with the trusted-key count, it flattened the key line to the
baseline and hid the outage it was meant to mark. The series is now
omitted and a quorum_disabled flag carries the state.
- Two panel descriptions claimed a one-second export cycle. The reader is
configured for ten.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase-10 independently instrumented the peer object-fetch path while this
branch instrumented fresh-node sync, so the two overlapped in three places.
Resolved by keeping each side's stronger implementation rather than shipping
both.
Per-job-type waiting/running/deferred existed twice. Phase-10's version
survives: it publishes per-type gauges from JobQueue::collect(), which
snapshots under the queue lock and publishes after releasing it, a
deliberate lock-order fix against the collector's own lock. This branch's
jobq_backlog gauge and the JobQueue::getJobTypeCounts() accessor that fed it
are removed, along with their panels, assertions and reference rows.
jobq_saturation stays: it reports the whole worker pool, which phase-10 has
no equivalent for.
The histogram view helper also existed twice with identical bodies under two
names; one survives, and the microsecond ladder is now the named array
rather than boundaries repeated inline. The job_type label was declared
twice, once as a file-local constant invisible to the naming check; both it
and handler now come from the constants header.
Two things phase-10 adds are complementary, not duplicates, and are kept as
they are: the handler label, which separates the two request kinds that both
report as the same job type, and getobject_rejected_total, which counts
malformed requests where this branch's serve_refused_total counts requests
this node declined to serve.
Also fixes two naming-check failures that pre-date this merge on phase-10.
The check derived label keys only from namespaced constants, so it could not
see the per-subsystem headers' flat k-prefixed style and rejected dashboards
querying labels the code really emits. It now reads both styles, with the
enforcement rules unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The board and runbook had grown by append across eight work packages, so
they read in the order the work was done rather than the order a node
progresses. This is the coherence pass; it adds no new instrumentation.
- Dashboard: 52 panels regrouped from two rows into nine that follow the
fresh-start sequence — bootstrap, peer supply, sync state, acquire and
SHAMap fetch, job queue, quorum and publish, terminal blockers, then
back-fill and spans collapsed since they answer conditional questions.
Layout only: no title, query or description changed.
- Runbook: the flat step list becomes a decision tree branching on the
observed symptom, with the amendment-block check first because it is
terminal. Each branch names the panels, what healthy and unhealthy look
like, and what to conclude. The existing steps are kept as the detail
bodies.
- Reference table: every signal name re-checked against the code and every
named panel against the board; four stale panel references fixed.
- Validation: every signal is now either asserted or covered by a note
explaining why a five-node local cluster cannot produce it.
Also fixes the write-latency signal, which was inert on a real node: the
store duration was only recorded on the database-import path, while the two
production store implementations did not time themselves, so an ordinary
node reported a write count with no latency. Both now time the backend
write, which is the disk work this signal exists to expose. Without it the
"existing database syncs slower than a fresh one" diagnosis had no primary
signal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A slow fresh-sync ledger produced spans scattered across threads with no
way to relate them. They now share a trace id derived from the ledger's own
hash, the one value every participating site already holds, so nothing new
is plumbed across threads. This is the pattern the transaction pipeline
already uses for its tx id.
Joined: ledger.validate, ledger.store, and a new
consensus.validation.accept recorded when a trusted validation arrives. In
Tempo, searching one ledger hash returns them together, so an operator can
tell whether the ledger was slow to arrive, slow to be accepted, or slow to
be stored. They are siblings rather than a chain because the accept gate is
entered from three different threads, so no fixed parent order exists.
consensus.validation.accept also records why an arriving validation did or
did not advance the gate, which makes "validations arrive but are all
rejected" visible for the first time.
consensus_round_duration_ms turns the existing round-time span attribute
into a histogram, so a fleet trend needs a metric query rather than raw
trace inspection. An explicit bucket view is required, not optional: the
SDK default tops out at ten seconds while consensus abandons a round at two
minutes, so slow rounds would all fall in one bucket and every quantile
would read exactly ten seconds. Cost is one record per round.
Record layer: the histogram is native and needs no collector change. The
two new bounded attributes are added as span-metric dimensions to both
collector configs. The ledger hash stays out of them, since a per-ledger
dimension mints a series per ledger; it is indexed in Tempo as the join key.
The ledger.acquire span is not joined yet, because that file was being
changed concurrently. It is registered as an optional member of the join
group so nothing fails, and switching it is a one-line follow-up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The quorum and publish gauges were emitted but never surfaced: no panel, no
harness assertion, no reference entry. Completes those layers.
- Four panels: trusted validations against the quorum target on one axis so
a tally climbing toward quorum is visually distinct from one flat below
it; publish lag; pre-accept shortfall rate; and time to first validated
ledger.
- Both signals are asserted by the workload validator. The shortfall
counter does fire on a healthy cluster, because this node validates and
then immediately re-enters the accept gate before its peers' validations
arrive, so the first evaluation of every round tallies short. The panel
and note say so, and give the fault signature instead: the shortfall rate
outpacing the ledger-close rate while the tally stays flat and nothing
ever reaches first-validated.
- The quorum target is deliberately drawn as its own line rather than as a
headroom stat, so the disabled-quorum sentinel reads as an unreachable
target instead of an unreadable negative number.
Also removes three reference rows that were appended twice when two agents
each documented the same back-fill signals.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Registers the new gauges, renders them, asserts them and documents them, so
each signal reaches an operator rather than stopping at the emit site:
- MetricsRegistry: gauge registration for ledger_quorum_publish,
nodestore_latency, peer_ledger_supply, peerfinder_slot_census and
amendment_block, each guarded by the detached-callbacks check and
tolerant of services that are not ready yet.
- Ledger Sync Health dashboard: panels for the new signals, filtered by
the node template variable like every other board.
- Workload validation: the new series are asserted, so a signal that
regresses to absent fails CI. Signals the local cluster structurally
cannot produce, such as a replay fallback or an amendment block, are
noted rather than asserted, which would fail red on a healthy run.
- Reference, runbook and glossary entries, including the diagnosis order
for a node that has peers and validators but never validates.
- Regenerated levelization baseline: three new one-way edges from the
telemetry and test modules, no new cycles.
Also drops an unused cstddef include from the macro tests, which the
include checker rejects.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The span only recorded an outcome on the normal completion path. An
acquire that stalled and was later swept ended with no outcome at all, and
its duration stretched to the sweep interval rather than the real fetch
time. So the one case these signals exist to catch, a fetch that never
finishes, was the one case that could not be traced, and aggregate outcome
and timeout rates read low exactly when nodes are stuck.
- Adds an abandoned outcome value for the swept-while-fetching case.
- Routes every exit through one idempotent finalizer, so a span is
finalized exactly once whether it completes, fails, short-circuits on
local data, or is destroyed mid-fetch. The destructor path cannot throw.
- Adds the ledger hash to the span and backfills the sequence once known,
since by-hash acquires start without one and could not otherwise be tied
to a specific ledger.
- Record layer: outcome stays a span-metrics dimension in both collector
configs, which drift apart if only one is edited. The ledger hash is
indexed in Tempo for trace search instead, because a per-ledger value as
a metric dimension would mint a new series every ledger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sync-critical job types run at very low concurrency limits (ledgerRequest
and ledgerData allow 3 each), so a node can stall simply because those
jobs are held back behind other work. Nothing exposed that until now:
the existing job metrics are rates and quantiles of jobs that already
moved, or a single queue-wide depth.
- jobq_backlog{metric,job_type}: instantaneous waiting, running and
deferred counts per job type. Deferred is the starvation signal and had
no exposure anywhere; it is set when a type is at its concurrency limit.
- jobq_saturation{metric}: running tasks, worker-thread count and total
waiting, so a slowdown spanning several subsystems can be attributed to
worker-pool exhaustion instead of being diagnosed once per victim.
Both read through two new const accessors on JobQueue that take the
existing mutex once and copy integers, so a single reading is internally
consistent and no per-job cost is added. The job_type label reuses the
same JobTypes name helper the existing job counters use, so the two label
sets join.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Post-deploy verification against Grafana Cloud found two problems on the
job-queue dashboard.
Every aggregating panel's displayName names four labels, but the queries
aggregate with `sum by (...)` lists that omit `xrpl_work_item`, so that
part of the legend rendered as an unresolved reference. Pre-existing, but
splitting the wait and execution panels by job type made the legend
load-bearing, so it is fixed here: `xrpl_work_item` is added to the ten
affected sum-by lists.
The two panels split by job type in the previous commit also kept their
seven-section descriptions. They now carry the same ten sections as the
rest, and their glossary links point at the real `job-queue-job-type`
anchor rather than a `job-queue` anchor that does not exist.
Verified no dashboard links a glossary anchor that is absent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The eight panels added for the GetObject and job-queue metrics did not
follow the conventions the rest of the dashboards use.
- Descriptions go from seven sections to the ten used by the other 159
panels, adding Keywords, Computation boundary and References. The
existing diagnostic guidance is kept; only the format changed.
- Six glossary entries added for the terms those References link to
(concurrency limit, deferred job, handler label, NodeStore lookup
hit/miss, resource charge), so no link is dead.
- displayName becomes `${series} ${xrpl_ident}`, the form 122 of 143
panels use. This needs the label_join wrapper that builds xrpl_ident,
which these queries lacked, so it is added to ten targets; without it
the legend would render an unresolved label.
- Legend blocks now match each dashboard's own convention rather than
being split three-with and four-without across the new panels.
Job Queue Wait Time and Job Execution Time dropped job_type from their
aggregation, so every queue collapsed into one line and the legend could
not say which queue was slow. Both now split by type, capped with topk
to stay readable, matching the per-type panel already on that dashboard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolved five files. In each case both sides had content worth keeping,
so nothing was taken wholesale:
- MetricsRegistry.cpp: kept the incoming `handler` label on the job
instruments and re-applied this branch's `queuedDurUs >= 0` guard,
which the incoming side does not have.
- telemetry-runbook.md: took the incoming gauge table, which adds the
three per-job-type rows, and re-applied this branch's corrected
`jobq_job_count` name.
- 09-data-collection-reference.md: kept this branch's validation
inventory (newer counts, extra Config File column) and inserted the
incoming call-site and per-job-type gauge rows plus their explanation.
- node-health.json: merged structurally rather than by text. This
branch's panels are authoritative; only the two incoming job-queue
panels were appended, below the existing layout. The
`Validated Ledger Seq` panels added directly in Grafana are preserved.
- job-queue.json, ledger-data-sync.json: panel sets were identical, so
took the incoming side for its `$handler` variable, the handler filter
on existing queries, and the new panels. Verified no panel was lost.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Slowness on the peer object-fetch path could be observed but not
attributed. Job duration metrics carry only `job_type`, and both
`RcvGetLedger` and `RcvGetObjByHash` report as `ledgerRequest`, so a
queue-wait spike could not be traced to a handler. Nothing measured
NodeStore cost, request size, or the differential charge.
Latency now decomposes into three additive parts, each separately
measurable:
end-to-end = queue wait + NodeStore lookup + everything else
- `handler` label on job_queued_total/_started_total/_finished_total and
job_queued_us/job_running_us. The value is sanitised: a name passes
through only if non-empty and all ASCII letters, else "other". Two job
names embed a ledger sequence, so a raw label would mint one series
per ledger; the rule bounds the domain at 43 names plus "other".
- getobject_lookup_us, _request_objects, _lookups_total{result},
_rejected_total{reason} and _charge, recorded at their call sites.
All three histograms get explicit bucket views: the SDK default stops
at 10,000, which every one of them exceeds.
- Per-job-type waiting/running/deferred gauges for the 35 non-special
job types. `deferred` is the leading indicator, since addJob never
rejects -- it defers, so backpressure otherwise shows up only as
latency after the fact.
`JobQueue::collect()` snapshots the counters under the queue lock and
publishes gauges after releasing it. Writing them while holding the lock
would invert a lock order against the collector's own lock, which the
collector's flush thread already holds when it calls this hook.
Tests assert exact values, including that the charge is priced on the
requested count rather than the capped iteration count.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signals that separate a sync that is merely slow from one that will never
finish:
- sync_acquire{missing_state_nodes_max, missing_tx_nodes_max, in_flight,
received_data_depth}: how many SHAMap nodes each in-flight acquire is
still waiting for. getMissingNodes already computed this and the callers
discarded it after a trace log. A count that stays flat means the
acquire is wedged; a shrinking count means it is progressing. Recorded
once per sweep, never inside the per-node walk, and reset when a tree
completes so a finished acquire does not read as stuck forever.
- shamap_cache_hit_rate{treenode}: hit rate of the in-memory tree-node
cache, which sits above the node store, so it is distinct from the
existing NuDB ratio. A cold cache on a fresh node sends every traversal
step to disk.
- sync_acquire_no_progress_total: timer ticks where an acquire made no
progress, previously only logged.
- sync_addnode_total{good,duplicate,invalid}: whether arriving nodes are
useful, duplicated or rejected, so wasted fetch work is visible.
- sync_acquire_source_total{local,network}: whether a ledger was served
from the local store or had to be fetched.
Adds getBad()/getDuplicate() to SHAMapAddNode and an acquireProgress()
accessor on InboundLedgers so the xrpld gauge can read these without
libxrpl depending on telemetry.
ledger_seq is deliberately not a metric label: it is unbounded. Per-ledger
identity stays on the ledger.acquire span; the metrics expose bounded
aggregates instead.
The full-below cache hit rate is not exported: KeyCache updates different
counters than getHitRate() reads, so it would always report zero. That
libxrpl bug is documented rather than papered over.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five signals that explain why a node is not advancing toward full, none of
which were observable before:
- state_changes_total now carries {from,to} mode labels, emitted at
setMode using the existing strOperatingMode helper. A bare count could
not distinguish a healthy climb from a node flapping between tracking
and connected. Removes the now-unused incrementStateChanges wrapper.
- sync_state{initial_full_duration_us}: time to first reach full, which
StateAccounting already computed but exposed only in server_info.
- sync_state{network_ledger_gate}: whether the node is still refusing to
build ledgers because it has no network ledger.
- sync_state{server_stall_seconds} and server_stall_events_total: how
long the main thread has been unresponsive. LoadManager computed this
and only logged it, so a stall was invisible until the fatal threshold.
The episode rule is a pure function so it can be tested without adding
a test-only mutator to LoadManager.
- sync_state{ledgers_behind}: how far our validated sequence trails the
best sequence any peer advertises, read from already-cached peer ranges
so no extra network traffic is added.
Also fixes the naming checker: it derived only the first label of a
multi-label instrument, so a dashboard querying the second label was
wrongly rejected.
Note: the clang-tidy hook cannot run in this worktree (no build
directory); the remaining pre-commit hooks, the naming check, dashboard
schema and harness syntax all pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A freshly started node most often stalls before it ever peers or reaches
quorum, and that whole chain had no telemetry. Adds the six signals that
make it observable:
- dns_resolve_total / dns_resolve_latency_ms: configured-peer hostname
resolution, emitted from OverlayImpl so libxrpl stays independent.
- overlay_connect_total / overlay_dial_latency_ms: outbound dial outcome
by terminal reason, plus dial duration.
- handshake_negotiation_fail_total: protocol and network-id negotiation
rejections, labelled by reason, so a misconfigured network is no longer
indistinguishable from unreachable peers.
- unl_fetch_total and the unl_quorum gauge: validator-list fetch outcome
per site and trusted key count against the required quorum. Without
these a bad validators.txt leaves the node syncing forever with no
signal.
- clock_close_offset_seconds: network close-time offset, which server_info
hides below 60s but which stalls consensus participation.
Panels land in the Bootstrap row of the Ledger Sync Health dashboard, the
metrics are asserted by the workload validator, and both the reference and
the runbook flow describe them.
Levelization baseline regenerated: overlay now includes MetricMacros.h, so
the overlay/telemetry pair is reported one-way instead of bidirectional.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Match the panel height the operator settled on in Grafana (18 rows) so
all 20 ranked bars render without an inner scrollbar, and reflow the
Sync Diagnostics row and the panels below it accordingly.
Adds the anchors the sync-diagnostics signals attach to, with no signals
emitted yet:
- New "Ledger Sync Health" dashboard (uid ledger-sync-health) with the
standard template-variable block copied from an existing board, plus
empty "Bootstrap (Domain 0)" and "Sync pipeline" rows.
- Signal index section in the data-collection reference, an operator-flow
stub in the telemetry runbook, and a glossary anchor.
- A sync_diagnostics group in expected_metrics.json and a matching
assertion helper in validate_telemetry.py so CI fails when a signal
regresses to absent.
Also registers the new dashboard uid with the harness so the board is
covered by validation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bar labels on 'Overlay Traffic Heatmap (All Categories, Bytes In)'
were raw metric names carrying a redundant '_bytes_in' suffix, and the
half-width 8-row panel truncated both the category and the node
identity.
- Strip the '_bytes_in' suffix from the derived series label; the panel
title already states the metric is inbound bytes.
- Widen the panel to full width and grow it to 12 rows so 20 bars render
with their full category and node labels.
- Shift the Sync Diagnostics row and the panels below it down by 4 to
keep the layout gap-free.
- Note the label derivation in the panel description.
The panel plotted only Full and Tracking, so the states a node actually
passes through while catching up were invisible. On the dev box's cold
sync the node spent ~19 minutes in Connected and ~3 seconds in Syncing,
while Tracking totalled 2 microseconds -- the one non-Full line the
panel did draw was the least informative of the five.
Add Syncing, Connected and Disconnected series (state-ladder order,
matching the Operating Mode Transitions panel) and rename the panel to
"State Duration Rate (All States)". Because the node is always in
exactly one state, the five rates sum to ~1.0, so the panel now reads as
"which state is time going into right now", and a handover between two
lines marks a state change with its width showing the dwell time.
Description expanded to the full section set (keywords, computation
boundary, references) used by the other panels on this dashboard.
Verified live: rate(state_accounting_connected_duration)/1e6 reaches
1.0 then 0.387 across the catch-up window and syncing reaches 0.0103 --
both previously undrawable.
The dashboard showed 54 individual signals but no single answer to
"is this node healthy and doing its job?". A reader had to correlate
server state, ledger age and peer counts by eye.
Add a stat panel at the top that reduces the two conditions that define
a healthy observer node to one green/red verdict:
server_state == 4 (Full) AND validated_ledger_age < 30s
Both terms use `== bool` so each yields 0/1, joined with
`* on(service_instance_id)` so the verdict is per node. Value mappings
render 1 as "Healthy" (green) and 0 as "Not Healthy" (red) with
background colouring, so the state is readable at a glance.
Placed at y=0 per the dashboard convention that gauges/stats lead;
existing panels shift down by 4 rows with no other change.
Verified live against Grafana Cloud: returns 1 for aws-dev-xrpl-1
(Full, validated seq 105,824,596).
The four aggregated Sync Diagnostics panels grouped by xrpl_work_item,
which no layer in this repo emits (it is injected by the perf-iac
harness). That tripped Rule D of the telemetry naming check:
D ledger-data-sync.json xrpl_work_item
must exist in L1, a metric label, or be a builtin
Align with the convention used by every other aggregated panel in the
dashboard set: group by (service_instance_id, xrpl_branch,
xrpl_node_role) and leave the xrpl_ident label_join untouched. The
legend is unaffected -- label_join over a label dropped by the
aggregation contributes an empty segment, which the trailing
label_replace already strips.
Verified live: both the NuDB ratio and the job-queue p95 queries still
return per-node series rendering as "[aws-dev-xrpl-1]".
The 0-4 values now render as named states via value mappings, so the numeric
axis label is misleading. Rename to "Server State".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
