Three template variables (xrpl_work_item, xrpl_branch, xrpl_node_role)
select their datasource via ${DS_PROMETHEUS}, but the variable itself was
absent from templating.list, so those label_values() lookups resolved to
no datasource and the filters stayed empty.
Add the datasource variable as the first templating entry, matching the
other dashboards in this folder.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dashboard's template variables and panel targets select their
datasource via ${DS_PROMETHEUS}, but the variable itself was absent from
templating.list. An unresolvable datasource variable leaves those queries
without a datasource, so the panels render empty.
Add the datasource variable as the first templating entry, matching the
other dashboards in this folder.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three measurement fixes landed with no doc or dashboard change, leaving text
that is now false and one fix unusable from a dashboard.
Deferrals and timeouts are recorded in TimeoutCounter, a base shared by five
subclasses, so the all-lane pair could show the documented livelock
fingerprint while ledger acquisition was healthy. The runbook procedure and
the reference doc now name acquire_ledger_deferrals and
acquire_ledger_timeouts and say why the all-lane pair misleads; a new panel
plots the ledger-scoped pair as rates on one axis, since the divergence is
the signal. The existing panel is retitled All Lanes and points at it.
Writer mean depth is depthSum over depthSamples, not over insertCount, and
the measured 1.60 came from the biased estimator, so it and the 37% queueing
share derived from it are lower bounds rather than values. The reference
table now marks them as such, and the decision rule is shown to survive the
correction rather than depending on the exact figures.
Completions were never counted for acquisitions satisfied from the local
store, so the run that read zero across 510 seconds had in fact reached
full. Every place that treated a zero as a symptom now says it only means
something on a build that has the fix.
Also corrects the sync-diagnosis label-value count from 13 to 15 and a stale
source line range; the instrument count stays 35, because both new values
multiplex onto the existing nodestore_state gauge.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drops nodestore_read_us and everything added to reach it. read_mean_us already
carries microsecond precision and separated the two sync failure modes cleanly
in live testing -- 8.8 us on a clean store against a 223 us cold-store peak --
so the distribution added no signal that changed a diagnosis.
The cost of getting it was disproportionate. NodeStoreScheduler had no path to
the metrics registry, so its production constructor grew a ServiceRegistry
parameter: a metric addition changing a production signature. That in turn
forced an edit to a pre-existing test, src/test/app/SHAMapStore_test.cpp, whose
only stake in this is that it constructs a scheduler. Worse, the scheduler is
built in Application's member initializer list, long before metricsRegistry_
exists, so the registry could not be captured once and had to be re-resolved on
every fetch -- a lookup on a path that runs millions of times per sync.
The constructor returns to taking JobQueue& alone and SHAMapStore_test.cpp
returns to the single-argument call, leaving that file differing from its
pre-change form only by the NodeStore:: to node_store:: rename it picked up from
develop.
FetchReport::elapsed stays microseconds and onFetch keeps its explicit
duration_cast to milliseconds for addLoadEvents, which takes milliseconds. That
widening was a separate fix and is what makes read latency measurable at all.
kSubMillisecondBoundaries loses its only consumer and regains [[maybe_unused]],
which is the state the commit that introduced it left it in; without the
attribute an unused constant is an error under wextra with werr.
Also removes the ledger-data-sync panel that charted the histogram and the
fetch_type and found template variables, which filtered on labels no metric
emits any more, plus the runbook and reference-doc sections and the two
instrument and view counts that named it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Sync State panel only ever showed Connected and Full. Two causes:
1. state_tracking{metric="state_value"} is emitted with values 0-6, where
5 is FULL+validating and 6 is FULL+proposing, but the panel declared
max: 4 with value mappings for 0-4 only. Values above 4 were pinned to
the axis ceiling and rendered unmapped. Most nodes sit at 6, so the
majority of series were clipped.
2. The gauge samples the instantaneous mode on a 10s export tick, so
states shorter than one tick fall between samples. A real sync showed
SYNCING for a single scrape and skipped TRACKING entirely. This is the
sampling hazard already noted on StateAccounting in NetworkOPs.h.
Extend the panel domain to 0-6 with Validating and Proposing mappings and
matching threshold steps, and add a colour-coded state-timeline panel where
each band's width is the time spent in that state, so brief states appear
as thin slivers rather than disappearing. The timeline reads
server_info{metric="server_state"} (raw OperatingMode 0-4) rather than
state_value, which folds 5 and 6 onto FULL and would split one Full band
into three colours.
Panel layout below the insertion point shifts down by 6 rows.
Note this makes short states legible, not lossless: exporting
StateAccounting's per-state duration accumulators is the sampling-immune
fix and is left as follow-up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The nudb_bytes label value on the storage_detail gauge named something the
code never measured. It observes Database::getStoreSize(), which returns the
storeSz_ accumulator: the cumulative payload bytes of objects this process
handed to the NodeStore. That is not a NuDB file size. It excludes NuDB's
keys, bucket padding and log, and it resets with the process while the files
on disk do not.
The name caused two concrete errors. It invited sizing the store on disk from
a number that cannot do it, and it invited a write-amplification ratio against
node_written_bytes -- which reads the same accessor at MetricsRegistry.cpp:836,
so that ratio is a constant 1.0 and measures nothing.
The nudb_ prefix was wrong too. storeSz_ is written only by
Database::storeStats(), called from DatabaseNodeImp, DatabaseRotatingImp and
Database itself. No backend code touches it, so the value reads the same on
RocksDB. That distinguishes it from the real nudb_* family
(nudb_writers_in_flight and friends), which come from getWriteStats() and are
absent entirely on a non-NuDB backend.
stored_object_bytes says what the value is and claims nothing about the
filesystem. Docs already described the value correctly; they keep that
explanation and now also record the old name, so a query pinned to it can be
traced. Neither Backend nor Database exposes an on-disk size accessor and none
was added -- no metric reports the store's on-disk size today.
Updates the node-health panel title, description and PromQL, and the four docs
that name the label value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brings forward the node_reads_hit and nudb_bytes label corrections from phases 7
and 9.
Conflicts resolved keeping both sides:
- 06-implementation-phases.md: kept phase-10's unprefixed `storage_detail` gauge
name with phase-7's corrected getStoreSize() description.
- node-health.json: kept phase-10's rewritten panel structure and re-applied the
panel renames (NodeStore Read Found Ratio, NuDB Stored Bytes), legends and axis
labels on top of it.
node_reads_hit is a found count, not a cache hit. fetchHitCount_ increments
whenever a fetch returned an object, whatever served it, and a node with
online_delete uses DatabaseRotatingImp which has no NodeObject cache at all. The
ratio therefore reads near 100% while every fetch goes to disk, which made the
cold-read failure mode look impossible on the board.
- Phase9_taskList: node_reads_hit is a found count, not cache-served reads.
- node-health: "NodeStore Cache Hit Ratio" -> "NodeStore Read Found Ratio",
legends "Cache Hit Ratio" -> "Found Ratio" and "Reads Hit (cache)" ->
"Reads Found"; corrected the ratio panel's axis label, which read
"Operations / Sec" on a percentunit panel.
- node-health: "NuDB Storage Size" -> "NuDB Stored Bytes". nudb_bytes observes
getStoreSize(), the accumulator node_written_bytes also reads, so it is
cumulative object-payload bytes and not a filesystem measurement.
- ledger-data-sync: point the read-latency discriminator at the renamed panel.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
nudb_bytes was documented as a NuDB file size, one place even claiming a
filesystem stat. It observes Database::getStoreSize(), which sums the object
payloads this process has written. It excludes NuDB's keys, bucket padding and
log, and resets with the process. node_written_bytes calls the same accessor, so
the two series are equal by construction and a write-amplification ratio built
from them is a constant 1.0. Neither Backend nor Database exposes a file-size
accessor, so nothing reports on-disk size today.
The Ledger Data & Sync panel plotting node_reads_hit / node_reads_total was
titled "NuDB Cache Hit Ratio" and described as reads served from cache.
fetchHitCount_ increments whenever a fetch returned an object, whatever served
it, so the ratio is a found rate. It reads near 100% while every fetch goes to
disk, which made the cold-read failure mode look impossible. Renamed to
"NuDB Read Found Ratio" and rewrote the guidance to pair it with read latency.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A stalled ledger sync has two distinct causes that look identical from the
job queue: in both, the ledgerData lane sits at its concurrency limit of 3
with work waiting. Lane occupancy therefore discriminates nothing, and the
dashboard had no panel that separated the two.
Adds a Sync Bottleneck Discrimination row to Ledger Data & Sync with seven
panels built on the nodestore_state write-path and acquire counters:
- NodeStore Read Latency, lifetime and windowed, on a log axis with
threshold lines at 10 and 100 us. This is the discriminator: fast reads
mean the cost is on the write side, slow reads with a high hit ratio
mean every access is paying disk latency.
- NuDB Writer Queue Depth, the confirming half. NuDB takes one global
mutex per insert, so depth above 1 is queueing at that mutex.
- NuDB Insert Time, mean against true max, which quantifies how much of
each insert is wait rather than work.
- Acquire Deferrals vs Timeouts on one panel, because the livelock
fingerprint is deferrals rising while timeouts stay flat, and neither
counter shows that alone.
- Acquisition Progress and Discarded Acquire Work, the outcome side: no
completions while the lane is busy means acquisitions resolve neither
way.
- NodeStore Read Latency Distribution split by fetch_type and found, from
the nodestore_read_us histogram, since a mean cannot tell a uniformly
warm store from a mostly-warm one with a cold tail.
Every expression carries the existing service_instance_id and tier filters.
The histogram introduces two new label dimensions, so fetch_type and found
template variables are added to match.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Post-deploy verification against Grafana Cloud found two problems on the
job-queue dashboard.
Every aggregating panel's displayName names four labels, but the queries
aggregate with `sum by (...)` lists that omit `xrpl_work_item`, so that
part of the legend rendered as an unresolved reference. Pre-existing, but
splitting the wait and execution panels by job type made the legend
load-bearing, so it is fixed here: `xrpl_work_item` is added to the ten
affected sum-by lists.
The two panels split by job type in the previous commit also kept their
seven-section descriptions. They now carry the same ten sections as the
rest, and their glossary links point at the real `job-queue-job-type`
anchor rather than a `job-queue` anchor that does not exist.
Verified no dashboard links a glossary anchor that is absent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The eight panels added for the GetObject and job-queue metrics did not
follow the conventions the rest of the dashboards use.
- Descriptions go from seven sections to the ten used by the other 159
panels, adding Keywords, Computation boundary and References. The
existing diagnostic guidance is kept; only the format changed.
- Six glossary entries added for the terms those References link to
(concurrency limit, deferred job, handler label, NodeStore lookup
hit/miss, resource charge), so no link is dead.
- displayName becomes `${series} ${xrpl_ident}`, the form 122 of 143
panels use. This needs the label_join wrapper that builds xrpl_ident,
which these queries lacked, so it is added to ten targets; without it
the legend would render an unresolved label.
- Legend blocks now match each dashboard's own convention rather than
being split three-with and four-without across the new panels.
Job Queue Wait Time and Job Execution Time dropped job_type from their
aggregation, so every queue collapsed into one line and the legend could
not say which queue was slow. Both now split by type, capped with topk
to stay readable, matching the per-type panel already on that dashboard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolved five files. In each case both sides had content worth keeping,
so nothing was taken wholesale:
- MetricsRegistry.cpp: kept the incoming `handler` label on the job
instruments and re-applied this branch's `queuedDurUs >= 0` guard,
which the incoming side does not have.
- telemetry-runbook.md: took the incoming gauge table, which adds the
three per-job-type rows, and re-applied this branch's corrected
`jobq_job_count` name.
- 09-data-collection-reference.md: kept this branch's validation
inventory (newer counts, extra Config File column) and inserted the
incoming call-site and per-job-type gauge rows plus their explanation.
- node-health.json: merged structurally rather than by text. This
branch's panels are authoritative; only the two incoming job-queue
panels were appended, below the existing layout. The
`Validated Ledger Seq` panels added directly in Grafana are preserved.
- job-queue.json, ledger-data-sync.json: panel sets were identical, so
took the incoming side for its `$handler` variable, the handler filter
on existing queries, and the new panels. Verified no panel was lost.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Slowness on the peer object-fetch path could be observed but not
attributed. Job duration metrics carry only `job_type`, and both
`RcvGetLedger` and `RcvGetObjByHash` report as `ledgerRequest`, so a
queue-wait spike could not be traced to a handler. Nothing measured
NodeStore cost, request size, or the differential charge.
Latency now decomposes into three additive parts, each separately
measurable:
end-to-end = queue wait + NodeStore lookup + everything else
- `handler` label on job_queued_total/_started_total/_finished_total and
job_queued_us/job_running_us. The value is sanitised: a name passes
through only if non-empty and all ASCII letters, else "other". Two job
names embed a ledger sequence, so a raw label would mint one series
per ledger; the rule bounds the domain at 43 names plus "other".
- getobject_lookup_us, _request_objects, _lookups_total{result},
_rejected_total{reason} and _charge, recorded at their call sites.
All three histograms get explicit bucket views: the SDK default stops
at 10,000, which every one of them exceeds.
- Per-job-type waiting/running/deferred gauges for the 35 non-special
job types. `deferred` is the leading indicator, since addJob never
rejects -- it defers, so backpressure otherwise shows up only as
latency after the fact.
`JobQueue::collect()` snapshots the counters under the queue lock and
publishes gauges after releasing it. Writing them while holding the lock
would invert a lock order against the collector's own lock, which the
collector's flush thread already holds when it calls this hook.
Tests assert exact values, including that the charge is priced on the
requested count rather than the capped iteration count.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Match the panel height the operator settled on in Grafana (18 rows) so
all 20 ranked bars render without an inner scrollbar, and reflow the
Sync Diagnostics row and the panels below it accordingly.
The bar labels on 'Overlay Traffic Heatmap (All Categories, Bytes In)'
were raw metric names carrying a redundant '_bytes_in' suffix, and the
half-width 8-row panel truncated both the category and the node
identity.
- Strip the '_bytes_in' suffix from the derived series label; the panel
title already states the metric is inbound bytes.
- Widen the panel to full width and grow it to 12 rows so 20 bars render
with their full category and node labels.
- Shift the Sync Diagnostics row and the panels below it down by 4 to
keep the layout gap-free.
- Note the label derivation in the panel description.
The panel plotted only Full and Tracking, so the states a node actually
passes through while catching up were invisible. On the dev box's cold
sync the node spent ~19 minutes in Connected and ~3 seconds in Syncing,
while Tracking totalled 2 microseconds -- the one non-Full line the
panel did draw was the least informative of the five.
Add Syncing, Connected and Disconnected series (state-ladder order,
matching the Operating Mode Transitions panel) and rename the panel to
"State Duration Rate (All States)". Because the node is always in
exactly one state, the five rates sum to ~1.0, so the panel now reads as
"which state is time going into right now", and a handover between two
lines marks a state change with its width showing the dwell time.
Description expanded to the full section set (keywords, computation
boundary, references) used by the other panels on this dashboard.
Verified live: rate(state_accounting_connected_duration)/1e6 reaches
1.0 then 0.387 across the catch-up window and syncing reaches 0.0103 --
both previously undrawable.
The dashboard showed 54 individual signals but no single answer to
"is this node healthy and doing its job?". A reader had to correlate
server state, ledger age and peer counts by eye.
Add a stat panel at the top that reduces the two conditions that define
a healthy observer node to one green/red verdict:
server_state == 4 (Full) AND validated_ledger_age < 30s
Both terms use `== bool` so each yields 0/1, joined with
`* on(service_instance_id)` so the verdict is per node. Value mappings
render 1 as "Healthy" (green) and 0 as "Not Healthy" (red) with
background colouring, so the state is readable at a glance.
Placed at y=0 per the dashboard convention that gauges/stats lead;
existing panels shift down by 4 rows with no other change.
Verified live against Grafana Cloud: returns 1 for aws-dev-xrpl-1
(Full, validated seq 105,824,596).
The four aggregated Sync Diagnostics panels grouped by xrpl_work_item,
which no layer in this repo emits (it is injected by the perf-iac
harness). That tripped Rule D of the telemetry naming check:
D ledger-data-sync.json xrpl_work_item
must exist in L1, a metric label, or be a builtin
Align with the convention used by every other aggregated panel in the
dashboard set: group by (service_instance_id, xrpl_branch,
xrpl_node_role) and leave the xrpl_ident label_join untouched. The
legend is unaffected -- label_join over a label dropped by the
aggregation contributes an empty segment, which the trailing
label_replace already strips.
Verified live: both the NuDB ratio and the job-queue p95 queries still
return per-node series rendering as "[aws-dev-xrpl-1]".
The 0-4 values now render as named states via value mappings, so the numeric
axis label is misleading. Rename to "Server State".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add value mappings (0 Disconnected, 1 Connected, 2 Syncing, 3 Tracking,
4 Full) and matching red->green thresholds to the Sync State panel, so the
line, tooltip, and legend render state names and colors instead of bare 0-4.
Keeps the minimal custom-key convention; only the Sync State panel changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
