Two metric-level defects the sync analysis identified, fixed at the source rather
than worked around on the dashboard.
1. InboundLedgers::sweep() destroys any acquire idle for more than a minute, and
that destruction is what telemetry reports as outcome=abandoned. But
lastAction_ was only refreshed by the constructor, update() and done() --
never by the receive path. With JtLedgerData capped at 3 concurrent jobs and
33 acquires in flight, an acquire whose peers were answering normally could
wait past the cutoff for its turn to apply data and be deleted for looking
idle. Measured on a fresh mainnet sync: 490 abandoned acquires against ZERO
expired retry budgets, so every one was a sweep, not a give-up.
gotData() now calls touch(). The sweeper's idle test measures real inactivity
instead of queue wait.
lastAction_ had to become atomic to allow this. It was a plain
clock_type::time_point written by the acquiring thread and read by sweep() on
the timer thread; adding a third writer on peer threads would have been a data
race. It is now std::atomic<clock_type::duration::rep> with relaxed ordering on
both sides -- the sweeper compares against a 60-second threshold, so a value
one tick stale cannot change its decision.
2. getLedgersBehindNetwork() returned the entire ledger sequence space on a fresh
node. The existing floor only guarded being ahead of every peer; it did not
guard having validated nothing at all, so validated=0 against a live tip gave
105,892,534 -- an accurate subtraction of a meaningless quantity. It
auto-scaled every consumer's axis and would trip any threshold. Distance to
tip is undefined before the first validated ledger, so it now reports 0 until
there is one, and the sync-state signals carry the initial-acquire progress.
The clamp_max(1e6) added to the Ledgers Behind Network panel as a stopgap is
removed: the metric is correct now, and leaving the clamp would hide a real
large backlog.
Verified: clang-tidy over the full compile database reports no finding on any
changed line in the three files (the pre-existing misc-include-cleaner and
misc-const-correctness findings elsewhere in InboundLedger.cpp are untouched by
this change). pre-commit passes including clang-format and the Doxygen style
check; validate_dashboards passes.
Not verified: not compiled -- per instructions.md the build needs approval, so CI
is the first real compile of the atomic change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two conflicts, both additive-vs-additive; each resolution keeps both sides.
check_otel_naming.py -- phase-10 taught the L6 label extractor to match the
label MAP first and to resolve a key hoisted into a `k...Label` constant,
scanning headers as well as sources. Our side had added the two-regex
first/subsequent literal scan and the `metric_constants(root)[1]` union that
covers the `namespace label` header style.
Kept phase-10's mechanism whole: METRIC_LABEL_MAP + the `(?:^|\{)` key regex
already subsumes what METRIC_LABEL_NEXT did, since matching inside the map body
makes every pair after the first open with a single `{`. So METRIC_LABEL_NEXT is
dropped as genuinely redundant rather than kept as a duplicate scan, and the
reason it existed is folded into METRIC_LABEL's comment. Re-added our
`metric_constants(root)[1]` union on top: LABEL_CONST_DEF only matches
`k`-prefixed identifiers, so it cannot see MetricNames.h's `label::jobType`
style, and without that union Rule D would reject dashboards querying labels
Rule I forced into constants. The two derivations are complementary and both
are now documented as such.
MetricsRegistry.cpp -- both sides added a new sibling view-registration helper
next to addMicrosecondHistogramView, and both added a registration call in
initExporterAndProvider(). Kept all four helpers
(addHistogramView/Microsecond/RoundDuration/SubMillisecond) and every
registration: phase-10's addSubMillisecondHistogramView + kNodeStoreReadUs
alongside our addRoundDurationHistogramView, sweepMallocTrimUs and the two
millisecond dial/resolve ladders.
phase-10's nodestore_read_us histogram does not duplicate our work. The
nodestore_latency gauge that would have overlapped it was retired in c4e434d520
before this merge, and the surviving nodestore_state gauge is complementary
rather than duplicative: both read the same fetch measurement, but the gauge
publishes only a since-boot mean via scaledMean() and cannot yield a
percentile -- the consequence observeNodeStoreTotals' own docs state plainly --
while the histogram buckets each fetch and can. The histogram also splits by
fetch_type and found, which the gauge cannot. phase-10 registered its
explicit-bucket View, so it does not inherit the SDK default ladder.
Each file keeps its own existing naming style: phase-10's k-prefixed constants
are left as-is, ours stay namespaced.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- JobQueue_test: drop the unused <functional>, and make the two
read-only GaugeFixture instances const. The other two stay mutable
because they submit jobs through fixture.queue.
- DatabaseConfig_test: drop the unused SystemParameters.h and include
ByteUtilities.h for megabytes(), which was reached only transitively.
Regenerate ordering.txt for the resulting edges: test.unit_test arrives
with SuiteJournal.h, and xrpl.protocol leaves with SystemParameters.h.
kSubMillisecondBoundaries existed but nothing used it, so per-fetch read
latency never reached Grafana -- only the coarse read_mean_us gauge did,
which cannot separate "every read took 9us" from "most took 2 and a few
took 900".
Add a nodestore_read_us histogram, register its view against the sub-
millisecond ladder rather than kMicrosecondBoundaries (whose first edge
is 100us, above the entire range a warm read occupies), and record into
it from NodeStoreScheduler::onFetch using FetchReport::elapsed, which a
previous change widened to microseconds for exactly this purpose.
The name and its labels live in a new include/xrpl/telemetry header
because the view registration (xrpld.telemetry) and the record site
(xrpld.app) sit in different levelization modules; a copy-pasted literal
would let them drift and silently drop the bucket override. Same reason
and same placement as GetObjectMetricNames.h. No new levelization edge:
xrpld.app > xrpl.telemetry already exists.
NodeStoreScheduler had no registry access, so it now takes a
ServiceRegistry and resolves the registry per call. It is constructed in
Application's initializer list, long before metricsRegistry_ is assigned
in setup() and started in startTelemetry(), so capturing a pointer at
construction would capture nullptr forever; the metric macros null-check
the registry, the meter and the instrument, so early fetches are simply
not recorded.
Labels are fetch_type and found, both already carried on the report --
4 series, fixed at compile time. A slow async read delays prefetch while
a slow sync read blocks a caller, and a miss can cost a read of every
backend, so neither dimension can be collapsed.
Negative elapsed times are skipped: the SDK rejects them and logs a
warning on every call, which on a per-fetch path is a log flood. Zero is
still recorded, since a page-cache-served read genuinely rounds to it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
nodestore_latency published six values that nodestore_state already
publishes from the same Database accessors, so the two gauges were
duplicate readings of the same atomics:
write_count -> node_writes getStoreCount()
read_count -> node_reads_total getFetchTotalCount()
write_duration_us -> node_writes_duration_us getStoreDurationUs()
read_duration_us -> node_reads_duration_us getFetchDurationUs()
write_mean_us -> write_mean_us store duration / count
read_mean_us -> read_mean_us fetch duration / count
nodestore_state is kept because its means go through scaledMean(), which
saturates at INT64_MAX instead of wrapping and omits a mean when the
denominator is zero rather than reporting a misleading 0 us.
Removes registerNodeStoreLatencyGauge, its instrument member, the
metric::nodestoreLatency constant and the lval::nodestore_latency label
namespace. The gauge-over-histogram rationale and the "p99 is not
obtainable" consequence are folded into observeNodeStoreTotals' docs.
Retargets the gauge-contract test onto nodestore_state rather than
deleting it: the scaledMean arithmetic is covered by the static_asserts
in tests/libxrpl/telemetry/MetricsRegistry.cpp, but nothing else asserts
that these named series multiplex onto one instrument keyed by `metric`.
The test now calls the production scaledMean instead of a copy of the
division, and its sub-microsecond case asserts scaledMean's actual
behaviour (a genuine mean of 0 on a zero numerator with a non-zero
count), which differs from the retired gauge's extra numerator guard.
Rewrites both ledger-sync-health copies' panel 38/39 queries and drops
the obsolete claim that the write numerator was never written: all three
concrete store paths call recordStoreDuration, so write_mean_us is live
on an ordinary node. The same stale [import_db] caveat is removed from
the runbook, the 09 reference row and the workload validator's note.
beast::Journal has no default constructor, so holding one as a plain
member deleted the suite's own default constructor and the Beast
registration macro could not instantiate it. SuiteJournal takes the
suite, converts implicitly where a journal is expected, and routes log
output into the test report.
Also removes a leftover unused local in run(), which is a hard error
under the warnings-as-errors build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit e3c2f8279a)
The nodestore suites moved from Beast to GTest upstream, which deleted
src/test/nodestore/TestBase.h. DatabaseConfig_test stayed on Beast and
still derived its journal and batch helpers from that base, so once both
sides met in a merge it referenced three symbols that no longer existed.
It now carries its own copies, matching the current API: node object
types are NodeObjectType::Ledger rather than the old hotLEDGER spelling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 05d01de81a)
The phase-10 merge brought a relocation: ConsensusSpanNames.h moved from
src/xrpld/consensus/ to include/xrpl/consensus/. Three of its four consumers were
updated during conflict resolution, but this test still included the old path,
which no longer exists.
Also corrects the file header, which explained that the test is guarded on
XRPL_ENABLE_TELEMETRY "because that is the configuration in which this test target
has src/ on its include path". That reason no longer holds: the header is lib-side
now, so a libxrpl test can include it directly without reaching into src/. The
guard remains because it is what builds the telemetry test target.
Verified statically: every in-tree xrpld/ and xrpl/ include across src/ and
include/ now resolves to a file that exists (only the three generated protobuf
headers are absent, as expected before a build). check_otel_naming.py exits 0 and
its 134 unit tests pass; the one remaining reference to the old path is a
synthetic fixture path inside those tests, built in a temp dir, so it is correct
as written.
Not verified: this was not compiled. The clangd diagnostics that surfaced the
problem come from a compile database generated ~19 hours before the merge, so it
still indexes the pre-relocation layout and reports errors for headers that are
present; those specific errors are an artifact, but a real build is still the only
proof the merge compiles.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase-10 brought in the upstream nodestore/peerfinder/consensus reorganisation
along with its own write-path telemetry, which collided with the sync-diagnostic
signals on this branch. Twelve files conflicted; every resolution keeps both
intents rather than picking a side.
The nodestore write timing was implemented twice, independently. Both sides
added getStoreDurationUs()/getFetchDurationUs() to Database and both timed the
backend call in each concrete store(). Keeping both would have added twice to
storeDurationUs_ per store while storeStats() still counted one, so the mean
write latency would have read double on every dashboard -- silently, since no
test on either side asserts an exact microsecond figure. Resolved to one
accumulator API: recordStoreDuration(), which takes a duration, clamps a
sub-microsecond sample to zero and uses a relaxed atomic add. Phase-10's
storeDurationStats() is gone and its two call sites now use the survivor, so
all three store paths -- both store() overrides and importInternal() -- add
exactly once.
SlotCensus and its pure virtual moved from src/xrpld/peerfinder/ to
include/xrpl/peerfinder/PeerfinderManager.h, following the Manager interface
upstream relocated. The xrpld header is now phase-10's makeConfig shim, and
Overlay.h, MetricMacros.cpp and the getSlotCensus() override chain point at the
new location. ConsensusSpanNames.h and peerfinder Slot.h/Config.h include paths
followed their headers into libxrpl the same way.
InboundLedger gained phase-10's AcquireStats counters next to this branch's
span activations in both the destructor abort path and done(); neither
displaces the other. nodestore_state keeps the constant-based name this branch
requires of it and phase-10's fuller description.
Upstream #7292 deleted src/test/nodestore/Database_test.cpp, which held this
branch's testDurationAccessors. Phase-10 restored the per-store half of that
coverage in DatabaseConfig_test, but nothing covered importInternal -- it writes
through storeBatch() and never through store(), so it is a third store path that
has to time itself. That half is ported to a GTest in
src/tests/libxrpl/nodestore/Database.cpp, keeping the exact zero-before and
accumulate-after assertions and the per-instance negative check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
beast::Journal has no default constructor, so holding one as a plain
member deleted the suite's own default constructor and the Beast
registration macro could not instantiate it. SuiteJournal takes the
suite, converts implicitly where a journal is expected, and routes log
output into the test report.
Also removes a leftover unused local in run(), which is a hard error
under the warnings-as-errors build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The nodestore suites moved from Beast to GTest upstream, which deleted
src/test/nodestore/TestBase.h. DatabaseConfig_test stayed on Beast and
still derived its journal and batch helpers from that base, so once both
sides met in a merge it referenced three symbols that no longer existed.
It now carries its own copies, matching the current API: node object
types are NodeObjectType::Ledger rather than the old hotLEDGER spelling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
FetchReport::elapsed was milliseconds, so every nodestore read rounded to
zero: a warm store answers in single-digit microseconds and a cold one in
low hundreds, and both became 0 ms. That difference is the whole signal
separating a cold-read stall from a healthy node, and it was being
discarded at the type. Database::fetchNodeObject now measures once and
uses that one value for both the cumulative counter and the report, so
the two can never disagree. The job-queue call still takes milliseconds
and now casts explicitly.
BatchWriteReport::elapsed stays milliseconds and is documented as such:
a batch write covers many objects and reaches the disk, so it belongs in
that range.
Also adds a sub-millisecond histogram ladder, because the existing bucket
edges start at 100 microseconds and put the entire warm range in bucket
0. It is not wired to a view yet: no sub-millisecond instrument exists to
name, so the edges wait for the instrument that records read latency.
The new test captures what the nodestore reports and asserts the reported
total equals the internal microsecond accumulator exactly, plus that at
least one report is not a whole number of milliseconds -- which a
millisecond-typed field can never satisfy on any hardware.
The nodestore namespace became xrpl::node_store when develop was merged
in, but one mock override still named the old spelling, so it did not
match the ServiceRegistry signature it overrides. The sibling mock in
TestServiceRegistry.h was already correct.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the derived read and write means, the NuDB writer depth and insert
timings, and the seven acquisition counters to the existing
nodestore_state gauge. Every value multiplexes onto that one instrument
through its `metric` label, so no new instrument is created.
Means are omitted rather than reported as zero when their denominator is
zero, so a dashboard shows a gap instead of a plausible wrong number. All
four go through one new scaledMean() helper so the guard cannot be
forgotten at a future call site; it also saturates instead of wrapping,
because a wrapped gauge reads as a healthy-looking dip. A zero total over
real samples still reports zero, since a store fast enough to truncate
every sample must not look dead.
The NuDB write-path block is skipped entirely when getWriteStats() is
nullopt, which is every backend but NuDB, so absent labels distinguish
"not measured" from "measured, and idle". Writer depth is scaled by 100
and named accordingly, because it sits just above 1.0 and an integral
gauge would truncate the whole signal away.
The gauge callback body is split into four static helpers to stay inside
the per-function line budget and to make each domain testable with a
recording sink.
Also corrects nudb_bytes, which called getStoreSize() exactly as
node_written_bytes does, so the obvious write-amplification ratio was a
constant 1.0 and the old "on-disk size" comment was wrong. No file-size
accessor exists on Backend or Database, so the value is unchanged and the
comment now states what it really is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
clang-tidy had been skipped on this branch while CMake configure was
failing, so these findings in makeStageSpan surfaced only now:
- brace the three single-statement if bodies (readability-braces-around-statements)
- compare the pointer parameter explicitly against nullptr
(readability-implicit-bool-conversion)
- include xrpl/protocol/Protocol.h for LedgerIndex (misc-include-cleaner)
A saturated ledgerData lane makes TimeoutCounter re-arm its timer
without running the timer body, so timeouts_ never advances and the
six-timeout give-up can never fire. Acquisitions then neither finish
nor fail until the one-minute sweep destroys their partial maps, and
the work restarts. Every step of that chain was debug-log-only, so a
node at warning level could not be diagnosed after the fact.
The counters are separate on purpose: deferrals rising while timeouts
stay flat is the signature, and no single counter shows it.
Completions are recorded in done() rather than at the "Done: complete"
log line, because that line also fires for failures and misses the
checkLocal and receiveNode paths; done() is the one funnel every
outcome passes through and its signaled_ guard makes it idempotent.
AcquireStats is only forward-declared in ServiceRegistry so libxrpl
still includes nothing from xrpld. The src/ include path for the test
binary moves out of the telemetry guard, since a header-only type
under src/xrpld/ is testable in every build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The consensus headers moved into the isolated xrpl/consensus module and
took a dependency on xrpl/telemetry for the tracing span constants, but
two things were left behind:
- Four includes still pointed at the old src/xrpld/consensus/ location,
which no longer exists, so the build failed with
"fatal error: 'xrpld/consensus/ConsensusParms.h' file not found".
- xrpl.libxrpl.consensus never linked xrpl.libxrpl.telemetry. add_module
isolates each module's headers, so xrpl/telemetry/SpanNames.h was not
on the include path even once the include was repointed.
Repoint the stale includes at xrpl/consensus/, and declare the telemetry
module before consensus so consensus can link it. Regenerate ordering.txt
for the resulting edge.
NuDB serializes every insert behind one global mutex held for the whole
call, so a caller cannot see how long it waited. Record instead the
writer depth joined at and the wall time spent; with mean depth L and
mean insert time W, Little's Law gives service time W/L and queueing
W - W/L. That distinguishes a serialized write path from a saturated
disk: measured on a dev box the device sat 89 percent idle while
throughput stayed flat at 42k inserts per second.
The accounting runs from a ScopeExit guard because the insert can
allocate and therefore throw; leaking the depth would strand the gauge
above zero for the life of the process.
getWriteLoad also stops returning a hardcoded zero. It now reports
writer depth, which is bounded by the writing-thread count and so stays
far below the kMaxWriteLoadAcquire cutoff that gates history
acquisition, where returning bytes or microseconds would have silently
suppressed it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The isValidJson2 call site in RPCCall.cpp already forces instantiation
of std::all_of over json::ValueConstIterator, so a regression that
removed the iterator traits would fail the real build. A dedicated
static_assert test is redundant; remove it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
json::ValueConstIterator and ValueIterator declared difference_type,
reference and pointer but not value_type or iterator_category. Under
C++23, std::iterator_traits then classifies them as output iterators,
so std::all_of over a Value's members (isValidJson2 in RPCCall.cpp)
fails to instantiate on GCC 13/14 with:
cannot convert 'output_iterator_tag' to 'std::input_iterator_tag'
GCC 15 masks this via LWG-3798/P2609, but the perf CI image ships
GCC 13, so the source needs the traits regardless. The iterators wrap
a std::map iterator (++/-- only), so the category is bidirectional.
Add value_type + iterator_category to both iterators, include <iterator>,
and add a regression test asserting the traits and that std::all_of /
std::count_if compile and run over Value members.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The gcc debug-coverage job rejected an unbraced `if` whose body is a GTest
assertion: EXPECT_EQ expands to an if/else, so the outer `if` leaves an else
that could bind either way, and -Werror=dangling-else refuses it. clang does
not warn, which is why only that one job failed.
Braced the span-names case that failed, then swept every test file this
branch touches for the same shape and braced the two others found, so the
next gcc run does not fail on the next one down the list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gcc debug-coverage job failed where clang, macos and windows passed:
gcc's -Werror=type-limits rejects comparing an unsigned expression against
zero with >=, because it is always true.
The assertion was vacuous anyway. Both operands are unsigned, so the check
proved nothing, while the comment beside it says the intent was a non-zero
microsecond figure. Now asserts strictly positive, which is what it meant.
The same job also logs a CMake LTO capability probe failing on a missing
compiler-ar and prints the code-generation guard's echo line; neither is
related to this branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
All eleven were reproduced locally against the same checks before fixing.
Six unused includes, left behind when the merge unioned two sets of includes
and later edits removed their only users: algorithm, functional, numeric and
thread in the job-queue test, and cstdint in the sync-state test and the load
manager. Each verified unused by grepping for every symbol the header
provides, so none is a still-needed include being dropped.
The telemetry registry header included ranges for a std::ranges::all_of call,
but that algorithm comes from algorithm, which the header already included.
Two consteval handler-name loops became std::ranges::all_of, which reads as
the predicate it is, and the two flagged fixtures are const.
Also picks up the levelization baseline the check asked for: the consensus
span-name test adds one edge from the libxrpl tests to xrpld.consensus, which
is the exact line CI's diff requested.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two suspects from the 3.3.0 slowdown investigation had no signal. Both were
already computing the numbers and throwing them away, so this exposes them
rather than adding measurement.
Per-sweep heap trim. The trim runs after every cache sweep, and its cost
scales with resident heap, so it is the leading explanation for a node with
a populated database syncing slower than a fresh one. The report already
carried duration, fault deltas and reclaimed pages, but the whole
measurement sat behind a debug-journal check, so an ordinary node measured
nothing, and the call site discarded the result. The measurement now always
runs and only the log line stays gated. Records trim duration, minor faults
and reclaimed kilobytes. Measured cost of the always-on path is about six
microseconds per sweep against a trim costing milliseconds, at a cadence of
ten to a hundred and twenty seconds.
Honest limit, stated in the runbook: the fault delta spans only the trim
call, so it shows the trim itself faulting but not the faults that follow as
caches refill. The duration is the signal to correlate against sweep-job
queueing.
Rotation writes. Rotation copies archive-served reads forward and re-stores
nodes missing from both backends, both of which compete with sync I/O and
only happen on a populated online_delete database. The copy-forward count
existed but was reset by the rotation's own log line, so a metric reading it
would drop to zero on every swap; a never-reset total sits beside it now.
The re-store count was not measured at all. Rotation duration is
deliberately not recorded: the health throttle sleeps at eight points inside
the sequence and dominates exactly when the node is unhealthy, so the number
would conflate work with waiting.
Nothing added for the other two suspects. Get-object serving is already
covered by the handler label, the lookup histogram and the deferred and
saturation gauges; peer churn by the disconnect-reason counter.
Also replaces nine per-file cspell ignores with one ignoreRegExpList entry
for the telemetry macro names, and picks up the levelization baseline for the
consensus span-name test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
fetchDurationUs_ had no getter, so telemetry reached it by building a JSON
object, stringifying a uint64 and parsing it back with stoll on every
collect tick. storeDurationUs_ was declared and never written or read at
all, along with the jss::node_writes_duration_us key.
Both now have accessors, both production store paths time their backend
call, and the registry reads them without the round trip. get_counts also
reports the write duration, so the RPC and the metric agree. Mean read
latency is the signal that separates a cold store from a warm one: warm
reads are single-digit microseconds, cold ones low hundreds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both were uint32_t while every sibling counter was uint64_t. On a
multi-day node node_read_bytes read 1,481,244,491 for 1,894,924,394
reads, i.e. 0.8 bytes per read, which is impossible: the counter had
wrapped about 350 times. Read hit rate is used to tell a cold-read
stall from a write-lock ceiling, so a wrapping numerator makes that
diagnosis wrong rather than merely imprecise.
getFetchTotalCount() was already backed by a uint64_t member, so its
32-bit return type truncated a correct value at the accessor. All
three getters now return uint64_t.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Found by reviewing what each metric actually measures, with attention to the
derived and bucketed ones. All five could report healthy while the node was
not, or the reverse.
- The nodestore latency panel took rate() of a mean. The gauge already
divides duration by count in code, so rating it produced a figure with no
unit, and Prometheus discards a gauge's decreases, so a heavy back-fill
read as roughly zero microseconds per operation. The cumulative duration
totals are now exported alongside the means, and the panel divides the
rate of the total by the rate of the count, which is the latency over the
panel's own window rather than a since-boot average that flattens with
uptime.
- The DNS-resolve and outbound-dial histograms had no explicit buckets, so
they inherited a ladder that stops at ten seconds while the dial timer is
fifteen. Every timed-out dial fell in the overflow bucket and p95 read
exactly ten seconds however bad it got. Both now have a ladder reaching
thirty seconds with fifteen on its own boundary, so a timeout is
distinguishable from merely slow.
- The missing-node counts only cleared when a tree completed, so a
timed-out or failed acquire left its last count latched. Since the gauge
reports the maximum across everything still in the collection, and
eviction waits on a grace period plus the sweep interval, a finished node
reported as stuck for minutes. That inverts the one signal that separates
stuck from slow. Cleared unconditionally on the terminal path instead.
- A disabled quorum published a sentinel so large that, on a timeseries
axis shared with the trusted-key count, it flattened the key line to the
baseline and hid the outage it was meant to mark. The series is now
omitted and a quorum_disabled flag carries the state.
- Two panel descriptions claimed a one-second export cycle. The reader is
configured for ten.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The write-duration test encoded the behaviour from before the store path was
timed. It populated the source database through store(), then asserted that
database's write duration was still zero, which was only true while
importDatabase was the sole timed path. Both concrete store overrides now
time their backend write, so an ordinary store run accumulates a duration
and the assertion failed.
Turned that stale assertion into a positive one: an ordinary storeBatch must
produce a non-zero duration, which is the case a real node actually
exercises. The negative half of the test, proving the accumulation is
per-database rather than a shared global, now checks that the import leaves
the source's store COUNT unchanged, since the source's duration is
legitimately non-zero from its own writes.
80552 tests across the six affected suites pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
