The phase-10 merge relocated PeerFinder's Manager interface from
src/xrpld/peerfinder/ to include/xrpl/peerfinder/, and SlotCensus moved with it
so libxrpl could see it. Logic.h uses SlotCensus as the return type of
getSlotCensus() but was never given the include, so every build failed:
build/modules/xrpl.libxrpl.peerfinder/xrpl/peerfinder/detail/Logic.h:192:5:
error: unknown type name 'SlotCensus'
...:197:16: error: use of undeclared identifier 'SlotCensus'; did you mean
'getSlotCensus'?
That single missing declaration was the whole failure. The 34 further errors in
the log were cascade: PeerFinderTest could not compile, so every TEST() in
src/tests/libxrpl/peerfinder/PeerFinder.cpp failed to instantiate against
gtest-internal.h. All four platforms (ubuntu-clang, ubuntu-gcc, macos-arm64,
windows-amd64) and clang-tidy reported the same root cause.
No include cycle: PeerfinderManager.h does not include detail/Logic.h, directly
or transitively. Levelization is unchanged because both headers are already in
xrpl.libxrpl.peerfinder, so generate.py produces no diff.
This is the risk called out when the merge landed -- moving SlotCensus into the
public header was the one resolution a static check could not confirm, and only
a compile would prove it. CI is that compile, and it found this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
nodestore_latency published six values that nodestore_state already
publishes from the same Database accessors, so the two gauges were
duplicate readings of the same atomics:
write_count -> node_writes getStoreCount()
read_count -> node_reads_total getFetchTotalCount()
write_duration_us -> node_writes_duration_us getStoreDurationUs()
read_duration_us -> node_reads_duration_us getFetchDurationUs()
write_mean_us -> write_mean_us store duration / count
read_mean_us -> read_mean_us fetch duration / count
nodestore_state is kept because its means go through scaledMean(), which
saturates at INT64_MAX instead of wrapping and omits a mean when the
denominator is zero rather than reporting a misleading 0 us.
Removes registerNodeStoreLatencyGauge, its instrument member, the
metric::nodestoreLatency constant and the lval::nodestore_latency label
namespace. The gauge-over-histogram rationale and the "p99 is not
obtainable" consequence are folded into observeNodeStoreTotals' docs.
Retargets the gauge-contract test onto nodestore_state rather than
deleting it: the scaledMean arithmetic is covered by the static_asserts
in tests/libxrpl/telemetry/MetricsRegistry.cpp, but nothing else asserts
that these named series multiplex onto one instrument keyed by `metric`.
The test now calls the production scaledMean instead of a copy of the
division, and its sub-microsecond case asserts scaledMean's actual
behaviour (a genuine mean of 0 on a zero numerator with a non-zero
count), which differs from the retired gauge's extra numerator guard.
Rewrites both ledger-sync-health copies' panel 38/39 queries and drops
the obsolete claim that the write numerator was never written: all three
concrete store paths call recordStoreDuration, so write_mean_us is live
on an ordinary node. The same stale [import_db] caveat is removed from
the runbook, the 09 reference row and the workload validator's note.
The phase-10 merge brought a relocation: ConsensusSpanNames.h moved from
src/xrpld/consensus/ to include/xrpl/consensus/. Three of its four consumers were
updated during conflict resolution, but this test still included the old path,
which no longer exists.
Also corrects the file header, which explained that the test is guarded on
XRPL_ENABLE_TELEMETRY "because that is the configuration in which this test target
has src/ on its include path". That reason no longer holds: the header is lib-side
now, so a libxrpl test can include it directly without reaching into src/. The
guard remains because it is what builds the telemetry test target.
Verified statically: every in-tree xrpld/ and xrpl/ include across src/ and
include/ now resolves to a file that exists (only the three generated protobuf
headers are absent, as expected before a build). check_otel_naming.py exits 0 and
its 134 unit tests pass; the one remaining reference to the old path is a
synthetic fixture path inside those tests, built in a temp dir, so it is correct
as written.
Not verified: this was not compiled. The clangd diagnostics that surfaced the
problem come from a compile database generated ~19 hours before the merge, so it
still indexes the pre-relocation layout and reports errors for headers that are
present; those specific errors are an artifact, but a real build is still the only
proof the merge compiles.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Phase-10 brought in the upstream nodestore/peerfinder/consensus reorganisation
along with its own write-path telemetry, which collided with the sync-diagnostic
signals on this branch. Twelve files conflicted; every resolution keeps both
intents rather than picking a side.
The nodestore write timing was implemented twice, independently. Both sides
added getStoreDurationUs()/getFetchDurationUs() to Database and both timed the
backend call in each concrete store(). Keeping both would have added twice to
storeDurationUs_ per store while storeStats() still counted one, so the mean
write latency would have read double on every dashboard -- silently, since no
test on either side asserts an exact microsecond figure. Resolved to one
accumulator API: recordStoreDuration(), which takes a duration, clamps a
sub-microsecond sample to zero and uses a relaxed atomic add. Phase-10's
storeDurationStats() is gone and its two call sites now use the survivor, so
all three store paths -- both store() overrides and importInternal() -- add
exactly once.
SlotCensus and its pure virtual moved from src/xrpld/peerfinder/ to
include/xrpl/peerfinder/PeerfinderManager.h, following the Manager interface
upstream relocated. The xrpld header is now phase-10's makeConfig shim, and
Overlay.h, MetricMacros.cpp and the getSlotCensus() override chain point at the
new location. ConsensusSpanNames.h and peerfinder Slot.h/Config.h include paths
followed their headers into libxrpl the same way.
InboundLedger gained phase-10's AcquireStats counters next to this branch's
span activations in both the destructor abort path and done(); neither
displaces the other. nodestore_state keeps the constant-based name this branch
requires of it and phase-10's fuller description.
Upstream #7292 deleted src/test/nodestore/Database_test.cpp, which held this
branch's testDurationAccessors. Phase-10 restored the per-store half of that
coverage in DatabaseConfig_test, but nothing covered importInternal -- it writes
through storeBatch() and never through store(), so it is a third store path that
has to time itself. That half is ported to a GTest in
src/tests/libxrpl/nodestore/Database.cpp, keeping the exact zero-before and
accumulate-after assertions and the per-instance negative check.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two panels were left out of the previous sweep because a blanket rate-to-count
rewrite would have been wrong for them. Handled per target here.
Panel 65 (Sweep Heap-Trim) carries two cumulative counters, minor faults and
reclaimed KB. Both are count-shaped, so it becomes a bargauge over
round(increase(...[$__range])) like the rest, and the legends drop their
"/ Sec" suffix now that the values are totals rather than rates.
Panel 66 (Online-Delete Rotation) stays a timeseries. Its two targets are not
the same kind of thing: target A reads rotation_state{in_flight}, a 0/1 flag
whose whole value is seeing when it is high and for how long, and target B
rates rotation_state{copy_forward}, a cumulative write total. A count bargauge
would destroy the flag's time dimension. Instead the shared "cps" unit -- wrong
for a flag -- is replaced by per-target overrides: the flag pinned to a 0..1
left axis, the write rate on a right axis in cps. This matches how the metric
is documented to be read (MetricNames.h:648): copy_forward climbing while
in_flight is 1 is expected, climbing while it is 0 means the flag leaked.
Verified against Grafana Cloud Prometheus: 0 parse errors on both panels.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seventeen panels on ledger-sync-health drew low-frequency discrete counters as
timeseries with an ops/s unit. The rate is arithmetically right but unreadable:
"0.1394 ops/s" is 278 abandoned tree phases over 33 minutes, and no reader can
recover the 278. rate() also extrapolates, so whole events rendered as
fractions -- peer disconnects showed 8.008 for 8 actual disconnects.
Measured every candidate against live Prometheus over a 33-minute mainnet sync
before converting; none exceeded 0.48 events/s, so all are count-shaped:
serve_refused_total 955 events 0.4785/s
ledger_quorum_shortfall_total 670 0.3354/s
sync_acquire_source_total 511 0.2559/s
overlay_connect_total 130 0.0652/s
sync_acquire_no_progress_total 109 0.0545/s
unl_fetch_total 12 0.0061/s
peer_disconnect_total 8 0.0040/s
Each becomes a bargauge over round(increase(...[$__range])) with unit short and
decimals 0, matching the existing Mode Transitions panel. Legends follow
instructions.md OTel rule 7 -- "MetricName [labels]" -- and keep node identity
via xrpl_ident, so a nine-node view no longer collapses to one bar. Panel 49
also gained node identity in its aggregation, which it was missing entirely.
Label values stay as emitted (upgrade_fail, not "Upgrade Fail"). They are wire
identifiers: rewriting them in the legend would hide what the metric reports and
break silently when a new value appears. The Title Case sits in the metric name.
The five zero-valued panels were checked rather than assumed dead --
ledger_replay_*, sweep_malloc_trim_* and rotation_copy_node_restore_total all
have real emit sites and are feature-gated off in this configuration.
Verified against Grafana Cloud Prometheus: 0 parse errors across all panel
queries, whole-number results throughout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three defects, all found by rendering the panels and running their queries
against a live mainnet node.
1. Ledger Acquire Phase Outcomes had an invalid PromQL escape.
The saved JSON held "ledger\\.acquire\\.(.*)", which decodes to
`ledger\.acquire\.(.*)`. In a PromQL double-quoted string `\.` is not a
legal escape, so Prometheus rejected the whole query:
parse error: unknown escape sequence U+002E '.'
The panel therefore rendered an error badge and "No data". A PromQL string
needs two characters, so the JSON must carry four backslashes.
2. The same panel never grouped by outcome, despite its title.
`sum by (span_name, timed_out, ...)` omitted `outcome`, so complete and
abandoned collapsed into one line. Measured at 16:45 UTC that hid a 29x
difference: astree complete=2085 against abandoned=71, all drawn as a
single indistinguishable series -- and every phase then showed the same
0.2596/s value, which is what made the panel look meaningless.
Now grouped by outcome, giving four real series (verified live):
header complete 0.1439/s, header abandoned 0.0772/s,
astree abandoned 0.1404/s, txtree abandoned 0.1404/s
The selector moves from timed_out (always "false" here, so it carried no
information and its filter var was redundant) to the declared
$span_outcome. Legend becomes "<phase> <outcome>"; axis label reads
"Phases / sec" to match the ops unit.
3. Seven stat panels and one heatmap dumped the raw label set as the legend.
With no fieldConfig.defaults.displayName but textMode "value_and_name",
Grafana has no name to show and falls back to printing every label:
{deployment_environment="local", exported_instance="xrpld-mainnet",
exported_job="xrpld", instance="otel-collector:8889", ...}
Rendered PNGs of panels 10 and 26 confirmed it. Fixed on ids 10, 12, 13,
14, 26, 36, 45 and 52 with the board convention already used by 24 sibling
stat panels: "${__field.labels.series} ${__field.labels.xrpl_ident}".
Verified afterwards by executing all 66 panel queries on this board against
live Prometheus: 0 parse errors, 56 returning data. The 10 empty ones are
counters a healthy node never increments plus panel 29, which is gated to stay
blank until a ledger is validated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Peer Ledger Supply Window panel drew supply_min_seq, supply_max_seq and
nothing else on one linear axis. Measured on a mainnet node, those sit around
105,890,000 and roughly 300,000 apart, so the 588-ledger tip movement that
shows whether sync is progressing was 0.0006% of the axis and read as a flat
line. unit "none" also printed the sequences unabbreviated and clipped the
legend.
The panel's own "Watch for" text asked the reader to compare supply_min_seq
against this node's validated sequence, but that line was not on the panel at
all, so the comparison meant switching dashboards.
Plot the two distances instead, which is what the panel was always asking
about:
History Headroom = validated_ledger_seq - supply_min_seq
Tip Gap = supply_max_seq - validated_ledger_seq
Zero is now the boundary in both directions: negative headroom is exactly the
"every peer pruned what I still need" case the description warns about, and it
becomes a zero crossing rather than a line-order comparison. Tip Gap gets the
right-hand axis because the two ranges differ by orders of magnitude
(measured: 299999..300001 against -1..1).
Both operands are gated `> 0`. Ungated, differencing the documented
"unknown window" sentinel of 0 yields the whole sequence space: measured
-105854935 for headroom and 105890295 for tip gap during the first ticks,
which destroys the axis for the rest of the window. Gated, the panel stays
blank until the node has a validated ledger and a peer has advertised a
range, which is the honest reading for that state.
Both queries verified against a live mainnet node through the full template
substitution: refId A = 300001 legend "History Headroom [xrpld-mainnet]",
refId B = -1 legend "Tip Gap [xrpld-mainnet]".
Runbook branch-C table, step 11 walkthrough and the 09 reference row follow
the rename and the new reading.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The panel built its legend with label_replace from the `from` label alone, so
every edge leaving a mode collapsed onto one series: connected->syncing and
connected->full both drew as "connected". The whole point of the from/to pair
is to tell a healthy climb from flapping, and that was exactly what the
legend hid.
label_replace cannot concatenate two labels. label_join can, which is the
pattern the consensus board already uses for its multi-label legends, so the
series now reads "from -> to".
Also regenerates the Grafana Cloud copy of the board.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
beast::Journal has no default constructor, so holding one as a plain
member deleted the suite's own default constructor and the Beast
registration macro could not instantiate it. SuiteJournal takes the
suite, converts implicitly where a journal is expected, and routes log
output into the test report.
Also removes a leftover unused local in run(), which is a hard error
under the warnings-as-errors build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The nodestore suites moved from Beast to GTest upstream, which deleted
src/test/nodestore/TestBase.h. DatabaseConfig_test stayed on Beast and
still derived its journal and batch helpers from that base, so once both
sides met in a merge it referenced three symbols that no longer existed.
It now carries its own copies, matching the current API: node object
types are NodeObjectType::Ledger rather than the old hotLEDGER spelling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
FetchReport::elapsed was milliseconds, so every nodestore read rounded to
zero: a warm store answers in single-digit microseconds and a cold one in
low hundreds, and both became 0 ms. That difference is the whole signal
separating a cold-read stall from a healthy node, and it was being
discarded at the type. Database::fetchNodeObject now measures once and
uses that one value for both the cumulative counter and the report, so
the two can never disagree. The job-queue call still takes milliseconds
and now casts explicitly.
BatchWriteReport::elapsed stays milliseconds and is documented as such:
a batch write covers many objects and reaches the disk, so it belongs in
that range.
Also adds a sub-millisecond histogram ladder, because the existing bucket
edges start at 100 microseconds and put the entire warm range in bucket
0. It is not wired to a view yet: no sub-millisecond instrument exists to
name, so the edges wait for the instrument that records read latency.
The new test captures what the nodestore reports and asserts the reported
total equals the internal microsecond accumulator exactly, plus that at
least one report is not a whole number of milliseconds -- which a
millisecond-typed field can never satisfy on any hardware.
The nodestore namespace became xrpl::node_store when develop was merged
in, but one mock override still named the old spelling, so it did not
match the ServiceRegistry signature it overrides. The sibling mock in
TestServiceRegistry.h was already correct.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the derived read and write means, the NuDB writer depth and insert
timings, and the seven acquisition counters to the existing
nodestore_state gauge. Every value multiplexes onto that one instrument
through its `metric` label, so no new instrument is created.
Means are omitted rather than reported as zero when their denominator is
zero, so a dashboard shows a gap instead of a plausible wrong number. All
four go through one new scaledMean() helper so the guard cannot be
forgotten at a future call site; it also saturates instead of wrapping,
because a wrapped gauge reads as a healthy-looking dip. A zero total over
real samples still reports zero, since a store fast enough to truncate
every sample must not look dead.
The NuDB write-path block is skipped entirely when getWriteStats() is
nullopt, which is every backend but NuDB, so absent labels distinguish
"not measured" from "measured, and idle". Writer depth is scaled by 100
and named accordingly, because it sits just above 1.0 and an integral
gauge would truncate the whole signal away.
The gauge callback body is split into four static helpers to stay inside
the per-function line budget and to make each domain testable with a
recording sink.
Also corrects nudb_bytes, which called getStoreSize() exactly as
node_written_bytes does, so the obvious write-amplification ratio was a
constant 1.0 and the old "on-disk size" comment was wrong. No file-size
accessor exists on Backend or Database, so the value is unchanged and the
comment now states what it really is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
clang-tidy had been skipped on this branch while CMake configure was
failing, so these findings in makeStageSpan surfaced only now:
- brace the three single-statement if bodies (readability-braces-around-statements)
- compare the pointer parameter explicitly against nullptr
(readability-implicit-bool-conversion)
- include xrpl/protocol/Protocol.h for LedgerIndex (misc-include-cleaner)
A saturated ledgerData lane makes TimeoutCounter re-arm its timer
without running the timer body, so timeouts_ never advances and the
six-timeout give-up can never fire. Acquisitions then neither finish
nor fail until the one-minute sweep destroys their partial maps, and
the work restarts. Every step of that chain was debug-log-only, so a
node at warning level could not be diagnosed after the fact.
The counters are separate on purpose: deferrals rising while timeouts
stay flat is the signature, and no single counter shows it.
Completions are recorded in done() rather than at the "Done: complete"
log line, because that line also fires for failures and misses the
checkLocal and receiveNode paths; done() is the one funnel every
outcome passes through and its signaled_ guard makes it idempotent.
AcquireStats is only forward-declared in ServiceRegistry so libxrpl
still includes nothing from xrpld. The src/ include path for the test
binary moves out of the telemetry guard, since a header-only type
under src/xrpld/ is testable in every build.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The consensus headers moved into the isolated xrpl/consensus module and
took a dependency on xrpl/telemetry for the tracing span constants, but
two things were left behind:
- Four includes still pointed at the old src/xrpld/consensus/ location,
which no longer exists, so the build failed with
"fatal error: 'xrpld/consensus/ConsensusParms.h' file not found".
- xrpl.libxrpl.consensus never linked xrpl.libxrpl.telemetry. add_module
isolates each module's headers, so xrpl/telemetry/SpanNames.h was not
on the include path even once the include was repointed.
Repoint the stale includes at xrpl/consensus/, and declare the telemetry
module before consensus so consensus can link it. Regenerate ordering.txt
for the resulting edge.
NuDB serializes every insert behind one global mutex held for the whole
call, so a caller cannot see how long it waited. Record instead the
writer depth joined at and the wall time spent; with mean depth L and
mean insert time W, Little's Law gives service time W/L and queueing
W - W/L. That distinguishes a serialized write path from a saturated
disk: measured on a dev box the device sat 89 percent idle while
throughput stayed flat at 42k inserts per second.
The accounting runs from a ScopeExit guard because the insert can
allocate and therefore throw; leaking the depth would strand the gauge
above zero for the life of the process.
getWriteLoad also stops returning a hardcoded zero. It now reports
writer depth, which is bounded by the writing-thread count and so stays
far below the kMaxWriteLoadAcquire cutoff that gates history
acquisition, where returning bytes or microseconds would have silently
suppressed it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The phase-2 merge re-added the pre-telemetry `add_module(xrpl tx)` block
without removing it, leaving the module declared twice. CMake's
add_library rejects a repeated target name, so configure failed before
any compilation:
add_library cannot create target "xrpl.libxrpl.tx" because another
target with the same name already exists.
Drop the stale pre-telemetry block and keep the one that follows
add_module(xrpl telemetry), which links both ledger and telemetry.
libxrpl/tx needs the telemetry link for the tx.transactor span, and
levelization already records `libxrpl.tx > xrpl.telemetry`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The isValidJson2 call site in RPCCall.cpp already forces instantiation
of std::all_of over json::ValueConstIterator, so a regression that
removed the iterator traits would fail the real build. A dedicated
static_assert test is redundant; remove it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
json::ValueConstIterator and ValueIterator declared difference_type,
reference and pointer but not value_type or iterator_category. Under
C++23, std::iterator_traits then classifies them as output iterators,
so std::all_of over a Value's members (isValidJson2 in RPCCall.cpp)
fails to instantiate on GCC 13/14 with:
cannot convert 'output_iterator_tag' to 'std::input_iterator_tag'
GCC 15 masks this via LWG-3798/P2609, but the perf CI image ships
GCC 13, so the source needs the traits regardless. The iterators wrap
a std::map iterator (++/-- only), so the category is bidirectional.
Add value_type + iterator_category to both iterators, include <iterator>,
and add a regression test asserting the traits and that std::all_of /
std::count_if compile and run over Value members.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The gcc debug-coverage job rejected an unbraced `if` whose body is a GTest
assertion: EXPECT_EQ expands to an if/else, so the outer `if` leaves an else
that could bind either way, and -Werror=dangling-else refuses it. clang does
not warn, which is why only that one job failed.
Braced the span-names case that failed, then swept every test file this
branch touches for the same shape and braced the two others found, so the
next gcc run does not fail on the next one down the list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gcc debug-coverage job failed where clang, macos and windows passed:
gcc's -Werror=type-limits rejects comparing an unsigned expression against
zero with >=, because it is always true.
The assertion was vacuous anyway. Both operands are unsigned, so the check
proved nothing, while the comment beside it says the intent was a non-zero
microsecond figure. Now asserts strictly positive, which is what it meant.
The same job also logs a CMake LTO capability probe failing on a missing
compiler-ar and prints the code-generation guard's echo line; neither is
related to this branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
