Resolves src/tests/libxrpl/CMakeLists.txt test_modules by keeping both
sides: develop's beast/nodestore/protocol additions (nodestore moved into
alphabetical position) and this branch's ledger module.
Two gaps meant the last failure produced no evidence of its cause.
The node-log upload was gated on `if: failure()`, but the validation step
sets continue-on-error, so the job is not failing at that point and the
condition never fired. Every failed run silently skipped the one artifact
that records why a node did not reach consensus. It now keys on the
validation step's own outcome, and also collects the harness logs.
Transaction failures were logged at DEBUG, which CI does not enable, so a
run where all 3052 submissions failed on a refused connection reported
nothing about it. The first occurrence of each distinct failure kind is now
a warning and repeats stay at DEBUG, so one refused connection says so once
instead of 3052 times.
Three template variables (xrpl_work_item, xrpl_branch, xrpl_node_role)
select their datasource via ${DS_PROMETHEUS}, but the variable itself was
absent from templating.list, so those label_values() lookups resolved to
no datasource and the filters stayed empty.
Add the datasource variable as the first templating entry, matching the
other dashboards in this folder.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dashboard's template variables and panel targets select their
datasource via ${DS_PROMETHEUS}, but the variable itself was absent from
templating.list. An unresolvable datasource variable leaves those queries
without a datasource, so the panels render empty.
Add the datasource variable as the first templating entry, matching the
other dashboards in this folder.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lane-attribution change added acquire_ledger_deferrals and
acquire_ledger_timeouts to the gauge but left this test expecting seven
labels, so it failed on every platform of both PRs: nine emitted against
seven expected. The exporters and the counters went in as separate
commits, which is how the test escaped the update.
The two new labels are asserted with values that differ from the all-lane
totals they are a subset of, so a counter wired to the wrong lane, or one
that ignored the flag and counted every lane, lands on the totals instead
and fails.
The round built each thread's batch inside the thread, before arriving at
the latch, so a throw there left the remaining threads waiting on an
arrival that never came -- the test hung instead of failing. A spawn loop
that ended early did the same.
Batches are now built before any thread starts, so nothing between spawn
and arrival can throw, and a guard counts down the shortfall for threads
that were never spawned before joining the ones that were.
The depth accounting having moved to insert entry, depthSamples is now
the denominator of the mean depth, so it gets its own assertions: equal
to insertCount once every thread has been joined, and moving with the
duplicate-key round. The overlap assertions are unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The counters that attribute deferrals and timeouts to ledger acquisition
compared a string literal in TimeoutCounter against a separate literal in
InboundLedger. Renaming the job would have silently sent both metrics to
zero: no build error, no failing test, and documentation that quietly
became wrong. Both now use one shared constant.
Three measurement fixes landed with no doc or dashboard change, leaving text
that is now false and one fix unusable from a dashboard.
Deferrals and timeouts are recorded in TimeoutCounter, a base shared by five
subclasses, so the all-lane pair could show the documented livelock
fingerprint while ledger acquisition was healthy. The runbook procedure and
the reference doc now name acquire_ledger_deferrals and
acquire_ledger_timeouts and say why the all-lane pair misleads; a new panel
plots the ledger-scoped pair as rates on one axis, since the divergence is
the signal. The existing panel is retitled All Lanes and points at it.
Writer mean depth is depthSum over depthSamples, not over insertCount, and
the measured 1.60 came from the biased estimator, so it and the 37% queueing
share derived from it are lower bounds rather than values. The reference
table now marks them as such, and the decision rule is shown to survive the
correction rather than depending on the exact figures.
Completions were never counted for acquisitions satisfied from the local
store, so the run that read zero across 510 seconds had in fact reached
full. Every place that treated a zero as a symptom now says it only means
something on a build that has the fix.
Also corrects the sync-diagnosis label-value count from 13 to 15 and a stale
source line range; the instrument count stays 35, because both new values
multiplex onto the existing nodestore_state gauge.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both counters are recorded in TimeoutCounter, a base shared by five
subclasses with different job limits, so they pooled every lane
together. The documented fingerprint for a stalled acquisition is
deferrals rising while timeouts stay flat -- but a saturated replay lane
reproduces that shape while ledger acquisition is healthy, and the two
counters being compared were not drawn from the same population.
Adds ledger-acquisition-scoped counts alongside the totals, discriminated
by the job name already held at both call sites, so no new state and no
signature change. The all-lane counters stay for callers that want them.
Their exporters went in with the preceding commit, which shared a file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Depth was observed at insert entry but its sample was only counted at
insert exit, so an insert still in flight contributed nothing to the
mean while completed inserts -- disproportionately the fast, shallow
ones -- all did. The reported mean understated queueing exactly when
queueing was worst: with every writer inside its first insert the gauge
was omitted entirely, while writers-in-flight correctly showed them all.
Depth and its sample count are now both folded in at entry, so the mean
is taken over one population. Mean depth is the L in Little's Law, so a
biased L understated the derived queueing share of each insert.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The workflow pinned its container image and compiler in a second place:
a hardcoded gcc-13 tag alongside the one in the build matrix. That copy
went stale when the rest of CI moved on, so this job kept building on
gcc 13 and failed on code the main CI compiled cleanly.
The tag now comes from the strategy matrix, so there is one source of
truth. The nix image carries several toolchains, so the compiler is also
selected explicitly for Conan to detect the intended one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
recordCompletion() was reached only from done(), but init() can satisfy an
acquisition entirely from the local store and return without ever calling
done(). Those completions went uncounted, so acquire_completions read zero
for a whole class of outcome. On a live clean-database sync it read 0 for
510 seconds while the node reached full, which was misread as the node
completing nothing.
Counted in init() rather than by calling done(), because done() drives the
state machine: it stores the ledger, dispatches AcqDone, updates the
fetch-rate average, and sets signaled_, which would disarm a later genuine
done(). A counter must not change behaviour. A completionCounted_ latch,
separate from signaled_, makes the count idempotent across the two
independent exits so a completion is counted exactly once however it was
reached.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
BEAST_EXPECT(getFetchDurationUs() > 0) failed on the macOS arm64 runner at
DatabaseConfig_test.cpp:879. It is an assertion about the host, not the
code: Database::fetchNodeObject() truncates each fetch to whole
microseconds, and a read served from NuDB's in-memory buckets can
genuinely measure under one microsecond. On a host fast enough for every
read to truncate, the total legitimately stays at zero.
Both fetch-duration checks now compare the accumulator against the total
the scheduler's fetch reports carried, via a CountingScheduler. Each
fetch is measured once and that one value feeds both, so the equality
holds at any host speed while being strictly stronger than '> 0': a
dropped fetch, a double-count, or the write member being returned all
break it. The same reasoning is already recorded in the GTest at
src/tests/libxrpl/nodestore/Database.cpp.
The store-duration '> 0' checks are left alone: those writes reach disk
and were not what failed.
All three were rejected by clang-tidy as errors:
- readability-math-missing-parentheses on 2 * kNumStored + kNumMissing
- misc-const-correctness on the never-mutated AcquireStats 'quiet'
- bugprone-unchecked-optional-access on the read_threads_running
dereferences: branching on BEAST_EXPECT's return value hides the
has_value() check from the analyser, so the guard is now a plain if and
the macro asserts separately.
The FeeCase table used positional aggregate initialisation, which
modernize-use-designated-initializers rejects for a five-field aggregate;
the clang-tidy job failed once per row. Naming the fields also makes the
requested/found/derived/literal ordering readable at each row instead of
relying on the reader tracking four ints positionally.
The header was added alongside sanitiseHandler(), which calls
std::ranges::all_of -- but that algorithm lives in <algorithm>, which the
same commit also added and which is still needed. <ranges> itself is never
used, so misc-include-cleaner rejects it and the clang-tidy job fails on
MetricsRegistry.h:146.
Drops nodestore_read_us and everything added to reach it. read_mean_us already
carries microsecond precision and separated the two sync failure modes cleanly
in live testing -- 8.8 us on a clean store against a 223 us cold-store peak --
so the distribution added no signal that changed a diagnosis.
The cost of getting it was disproportionate. NodeStoreScheduler had no path to
the metrics registry, so its production constructor grew a ServiceRegistry
parameter: a metric addition changing a production signature. That in turn
forced an edit to a pre-existing test, src/test/app/SHAMapStore_test.cpp, whose
only stake in this is that it constructs a scheduler. Worse, the scheduler is
built in Application's member initializer list, long before metricsRegistry_
exists, so the registry could not be captured once and had to be re-resolved on
every fetch -- a lookup on a path that runs millions of times per sync.
The constructor returns to taking JobQueue& alone and SHAMapStore_test.cpp
returns to the single-argument call, leaving that file differing from its
pre-change form only by the NodeStore:: to node_store:: rename it picked up from
develop.
FetchReport::elapsed stays microseconds and onFetch keeps its explicit
duration_cast to milliseconds for addLoadEvents, which takes milliseconds. That
widening was a separate fix and is what makes read latency measurable at all.
kSubMillisecondBoundaries loses its only consumer and regains [[maybe_unused]],
which is the state the commit that introduced it left it in; without the
attribute an unused constant is an error under wextra with werr.
Also removes the ledger-data-sync panel that charted the histogram and the
fetch_type and found template variables, which filtered on labels no metric
emits any more, plus the runbook and reference-doc sections and the two
instrument and view counts that named it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Sync State panel only ever showed Connected and Full. Two causes:
1. state_tracking{metric="state_value"} is emitted with values 0-6, where
5 is FULL+validating and 6 is FULL+proposing, but the panel declared
max: 4 with value mappings for 0-4 only. Values above 4 were pinned to
the axis ceiling and rendered unmapped. Most nodes sit at 6, so the
majority of series were clipped.
2. The gauge samples the instantaneous mode on a 10s export tick, so
states shorter than one tick fall between samples. A real sync showed
SYNCING for a single scrape and skipped TRACKING entirely. This is the
sampling hazard already noted on StateAccounting in NetworkOPs.h.
Extend the panel domain to 0-6 with Validating and Proposing mappings and
matching threshold steps, and add a colour-coded state-timeline panel where
each band's width is the time spent in that state, so brief states appear
as thin slivers rather than disappearing. The timeline reads
server_info{metric="server_state"} (raw OperatingMode 0-4) rather than
state_value, which folds 5 and 6 onto FULL and would split one Full band
into three colours.
Panel layout below the insertion point shifts down by 6 rows.
Note this makes short states legible, not lossless: exporting
StateAccounting's per-state duration accumulators is the sampling-immune
fix and is left as follow-up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The 22 metric label values on the nodestore_state gauge were asserted
nowhere, and the depthSum assertions could not tell the real
fetch_add(depth) from a fetch_add(1) that would silently zero every
derived queueing time.
Make the four observe* helpers and their ObserveFn sink public rather
than private, so the seam the header already claimed is actually
reachable. All four are static and read only their arguments, so this
exposes no object state; a friend declaration would have granted access
to every private member instead. The assertions live in the existing
Beast nodestore suite because src/test compiles into xrpld, which
contains MetricsRegistry.cpp, while xrpl_tests deliberately does not
when telemetry is enabled.
Each helper now has its exact emitted label set asserted, so a typo in
any literal fails instead of silently producing a disjoint series, and
each derived mean is asserted ABSENT on a fresh store -- a refactor to
value_or(0) would draw a believable flat zero on a latency axis and
otherwise pass everything.
Replace the concurrent write-stats test with one that forces genuine
overlap through a latch. NuDB holds one global mutex for the whole
insert and doInsert reads the depth before entering it, so a blocked
thread records a depth of at least 2; asserting depthSum strictly
exceeds insertCount therefore cannot be satisfied by a constant 1. The
old bounds admitted that bug at their floor.
Also: cover the std::nullopt branch on the two backends that exist in
every build, bound the store-duration accumulator by the wall clock,
drop four assertions that cannot fail, and correct two comments that
claimed coverage the tests do not have -- the duplicate-key test is not
the throwing path, because nudb reports key_exists without throwing,
and no test drives NodeStoreScheduler::onFetch.
The nudb_bytes label value on the storage_detail gauge named something the
code never measured. It observes Database::getStoreSize(), which returns the
storeSz_ accumulator: the cumulative payload bytes of objects this process
handed to the NodeStore. That is not a NuDB file size. It excludes NuDB's
keys, bucket padding and log, and it resets with the process while the files
on disk do not.
The name caused two concrete errors. It invited sizing the store on disk from
a number that cannot do it, and it invited a write-amplification ratio against
node_written_bytes -- which reads the same accessor at MetricsRegistry.cpp:836,
so that ratio is a constant 1.0 and measures nothing.
The nudb_ prefix was wrong too. storeSz_ is written only by
Database::storeStats(), called from DatabaseNodeImp, DatabaseRotatingImp and
Database itself. No backend code touches it, so the value reads the same on
RocksDB. That distinguishes it from the real nudb_* family
(nudb_writers_in_flight and friends), which come from getWriteStats() and are
absent entirely on a non-NuDB backend.
stored_object_bytes says what the value is and claims nothing about the
filesystem. Docs already described the value correctly; they keep that
explanation and now also record the old name, so a query pinned to it can be
traced. Neither Backend nor Database exposes an on-disk size accessor and none
was added -- no metric reports the store's on-disk size today.
Updates the node-health panel title, description and PromQL, and the four docs
that name the label value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The panel was renamed from NuDB Cache Hit Ratio to NuDB Read Found Ratio,
which left the runbook naming a title that no longer exists and carrying a
paragraph saying the rename had not happened yet. Both are corrected, and
the explanation now says why the ratio is a found rate: the counter
increments whenever a fetch returned an object, and a node with
online_delete has no object cache at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brings forward the node_reads_hit and nudb_bytes label corrections from phases 7
and 9.
Conflicts resolved keeping both sides:
- 06-implementation-phases.md: kept phase-10's unprefixed `storage_detail` gauge
name with phase-7's corrected getStoreSize() description.
- node-health.json: kept phase-10's rewritten panel structure and re-applied the
panel renames (NodeStore Read Found Ratio, NuDB Stored Bytes), legends and axis
labels on top of it.
node_reads_hit is a found count, not a cache hit. fetchHitCount_ increments
whenever a fetch returned an object, whatever served it, and a node with
online_delete uses DatabaseRotatingImp which has no NodeObject cache at all. The
ratio therefore reads near 100% while every fetch goes to disk, which made the
cold-read failure mode look impossible on the board.
- Phase9_taskList: node_reads_hit is a found count, not cache-served reads.
- node-health: "NodeStore Cache Hit Ratio" -> "NodeStore Read Found Ratio",
legends "Cache Hit Ratio" -> "Found Ratio" and "Reads Hit (cache)" ->
"Reads Found"; corrected the ratio panel's axis label, which read
"Operations / Sec" on a percentunit panel.
- node-health: "NuDB Storage Size" -> "NuDB Stored Bytes". nudb_bytes observes
getStoreSize(), the accumulator node_written_bytes also reads, so it is
cumulative object-payload bytes and not a filesystem measurement.
- ledger-data-sync: point the read-latency discriminator at the renamed panel.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
nudb_bytes was documented as a NuDB file size, one place even claiming a
filesystem stat. It observes Database::getStoreSize(), which sums the object
payloads this process has written. It excludes NuDB's keys, bucket padding and
log, and resets with the process. node_written_bytes calls the same accessor, so
the two series are equal by construction and a write-amplification ratio built
from them is a constant 1.0. Neither Backend nor Database exposes a file-size
accessor, so nothing reports on-disk size today.
The Ledger Data & Sync panel plotting node_reads_hit / node_reads_total was
titled "NuDB Cache Hit Ratio" and described as reads served from cache.
fetchHitCount_ increments whenever a fetch returned an object, whatever served
it, so the ratio is a found rate. It reads near 100% while every fetch goes to
disk, which made the cold-read failure mode look impossible. Renamed to
"NuDB Read Found Ratio" and rewrote the guidance to pair it with read latency.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
