Review follow-up on the freshen lock-hold fix:
- Drop host names, dates and one-site figures from the new comments,
harness notes and docs; explain the mechanism in general terms.
- RotationPhase stores its stage and cache labels as owned std::string,
not std::string_view: the ctor still takes views so the label
constants pass without a copy, but a member view would dangle if a
caller ever passed a temporary. freshenCache/recordFreshen take the
cache name by std::string_view (read-only, call-scoped).
- The new DatabaseRotating test called fetchNodeObject through the
derived type, whose private override hides the public base method;
call it through Database& instead. This was the dev-box build break.
- freshenCache reports the exact fetched count when a health abort cuts
it short, and stops labelling the per-partition hold 'getKeys'.
- Remove a [[maybe_unused]] that silenced no warning (the build sets
-Wno-unused-parameter and disables misc-unused-parameters).
The reference doc, span-harness notes and histogram-bucket comments
named the internal AWS dev box and dates while explaining why the
rotation phases are timed. Reword to the general mechanism (a
multi-second freeze at the copy-walk to freshen boundary on a populated
node); the specific hosts, dates and trace ids stay in the task notes.
The runbook provenance paragraph named the internal AWS dev box and a
build hash and dates. State what was measured (one mainnet node, same
host and binary, differing only in store state) without the deployment
detail, which belongs in an internal runbook, not the public repo.
clang-tidy's bugprone-unchecked-optional-access does not model GTest's
ASSERT_TRUE(x.has_value()), so it flags every deref that follows one.
The reads are guarded; mark them NOLINT, matching the same suppression
in src/tests/libxrpl/consensus/LedgerTrie.cpp. .value() does not help --
the checker treats it as an unchecked access too.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The online-delete rotation's cache freshen called TaggedCache::getKeys(),
which held the cache mutex while copying every key. On the dev box's 26
million entry tree-node cache that hold lasted 5-6 s, froze every job
that touches the cache, and dropped the RocksDB node out of sync once per
rotation: each "getKeys held the lock" warning was followed within 1-5 s
by "View of consensus changed" (5 of 5 rotations on 2026-09-15).
Copy the keys one map partition at a time instead. TaggedCache gains
forEachKeyPartition(), which holds the mutex only while one partition's
keys are copied and runs the callback with the mutex released, so the
longest hold shrinks by the partition count (8 on the dev box). The
freshen.keys rotation phase no longer exists as one step, so its span,
stage value, harness entries and docs are removed; the per-partition hold
still shows on the cache lock-hold peak gauge.
Measure what the freshen achieves, which no existing signal did.
DatabaseRotating gains duplicateCopyForwardTotal(), counting archive
copies made on duplicate fetches (the rotation's own copy walk and
freshen); copyForwardTotal() deliberately excludes those. The freshen
phase records rotation_freshen_keys_total{cache,outcome} and stamps
key_count, cache and keys_copied on its span; the copy phase stamps
nodes_copied. A warn log line per freshen reports the same numbers, and
the ledger-sync-health dashboard gets a Rotation Freshen Yield panel.
Log the "STATE->" operating-mode change at warn instead of info. It is
the only record of a mode change with an exact timestamp; the
state_changes_total counter is scraped once a minute and cannot order a
flap against a multi-second event.
Tests: five GTests for forEachKeyPartition (every key once, empty cache,
mutex free during the callback, concurrent insert, lock-hold peak), three
for duplicateCopyForwardTotal over two memory backends, one for the new
counter's series, and the new name literals.
OTelCollectorHooks.cpp named nothing from Hook.h, took ResourceMetrics
without a direct include, and left the overriding
GetAggregationTemporality() unmarked. NodeIdentity.h carried <optional>
and <string> it no longer uses. All fail CI under warnings-as-errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SecretKey deletes operator==, so EXPECT_EQ on the pair's second member
does not compile; compare the byte ranges instead. Application.cpp no
longer calls logicError(), so clang-tidy's include-cleaner rejects
<xrpl/basics/contract.h> there.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Static constants take the k prefix (readability-identifier-naming), and
SpanGuardScope.cpp no longer names anything from <opentelemetry/metrics/noop.h>
since it calls noopMeter(). Both fail CI under warnings-as-errors.
The helper's docstring also claimed NoopMeterProvider hides the base
two-argument GetMeter; it declares that overload itself, so the only
detail worth sharing is the version.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
clang-tidy's include-cleaner flags Counter.h and Gauge.h here: the test
names neither type, so the includes fail CI under warnings-as-errors.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Resolutions:
- MetricsRegistry.cpp: keep both <exception> and <limits>; drop
incrementStateChanges(), which this branch removed on purpose (the
labelled state_changes_total call site in NetworkOPsImp::setMode
replaces it, and a compile-time test guards that).
- tests/MetricsRegistry.cpp: constructor-built pipeline wording from
phase-10, this branch's test list and gauge paragraphs kept; the two
lifecycle tests now call startAsyncGauges() and pass kTestOptions.
- tests/MetricMacros.cpp: comments name the recording() gate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The no-op-meter branches asserted provider.collect() was empty, but
drive() never wires the CollectingProvider to the no-op meter, so that
held by construction. Keep only the meter()-read-once check, say in the
header what the no-op branches prove and what they cannot, name the one
(enable, meter) tuple production cannot produce, and compare doubles
with EXPECT_DOUBLE_EQ.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
stop() destroys the MeterProvider, and with it every View's
AggregationConfig. The SDK's SyncMetricStorage keeps a raw pointer to
that config, and the call-site statics keep the storage alive, so a
histogram record with a first-seen attribute set during the shutdown
drain would dereference freed memory. Application::run() stops the
registry before the job queue and server handler, so that window is
real.
phase_ is now atomic and stop() stores Stopped before tearing down.
Every XRPL_METRIC_* macro and every record*/increment* method checks
recording() (enabled and not stopped) instead of isEnabled(). meter_ is
never written after construction, so record threads read it without a
lock.
Also: an empty [telemetry] service_instance_id now falls back to the
node key on both the trace and the metrics side, so one node reports one
identity; disablePipeline() uses telemetry::noopMeter(); comments
corrected.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Lift the seed parsing and the stored-vs-mint choice into two libxrpl
helpers, parseNodeIdentitySeed() and selectNodeIdentity(), so xrpl_tests
can drive each branch without an xrpld Config. resolveNodeIdentity() now
marshals Config and the cmdline into them; behaviour is unchanged.
Also pin that storeNodeIdentity() appends (row count, not SQLite row
order), fix the test header that described getNodeIdentity()'s property
as the store's, and route NullTelemetry::getMeter() through noopMeter().
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Drive every XRPL_METRIC_* synchronous macro through the three registry states
a call site can meet on this branch's phase-9 base: disabled, enabled with a
real meter, and enabled with the no-op meter the registry falls back to when
its pipeline failed to build. 12 cases, each at its own call site, plus a
real-meter/no-op-meter pair per instrument kind. Enabled with a real meter
records every call and reads meter() exactly once; the no-op meter absorbs
every call and crashes nowhere; disabled never reads meter() at all.
The macro invariant these pin — meter() never empty while enabled — is what
lets the macro body be one function-local static and one record call, with no
null check on the hot path.
CI's clang-tidy leg flagged eight include-cleaner errors and three
misc-const-correctness / readability-convert-member-functions-to-static /
modernize-use-designated-initializers issues, all inside WP-B6's own code.
Fixed as follows:
- `MetricsRegistry.h`: `#include <opentelemetry/metrics/observer_result.h>`
for ObserverResult; `observeCacheLockHoldPeaks` is now `static` because it
touches neither instance state nor telemetry members.
- `SHAMapStoreImp.h`: adds direct includes for `<cstddef>`, `<string_view>`
and `<xrpl/telemetry/SpanNames.h>` (the StaticStr provider). `seconds` in
`RotationPhase::~RotationPhase` is `[[maybe_unused]]` so a
`-DXRPL_ENABLE_TELEMETRY=0` build under `-Werror` keeps compiling.
- `SHAMapStoreImp.cpp`: direct includes for `SHAMapStoreSpanNames.h`,
`SpanGuard.h`, `SpanNames.h`; `RotationPhase` locals that never call
`setAttribute` are declared `const`; `RotationOutcome` uses designated
initialisers.
Final-review findings (WP-B6-rotation-stall-tracing.md, "What to check
when reviewing"):
- Panels 74 and 75 on `ledger-sync-health.json` still carried panel 41's
description, axisLabel, Source and Keywords copy; rewritten to describe
rotation phase duration and cache lock hold respectively.
- `consensus_view_change_total` and the `view.change` round-span event
were emitted but not registered with the harness. Added the counter to
`not_asserted.metrics_excluded` (workload-gated) and annotated the
`consensus.round` span note with the event and its two attribute keys.
Not fixed (parked, see progress ledger):
- The reviewer's second Important finding — a plan/code contradiction on
the consensus counter — was based on a misread of the plan; the plan's
"Rejected alternatives" table lists a new `TraceCategory::Nodestore` and
the getKeys() fix, not the consensus counter. No action.
- The Minor note about `sweep()`'s peak including lock-acquire time and
`getKeys()`'s not: `sweep()` acquires and releases the lock via a
`scoped_lock`, so `noteLockHold` still runs after the release and the
numbers are comparable. No action.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Merges pratik/otel-phase9-metric-gap-fill into
pratik/otel-phase10-workload-validation.
Auto-merged. Carries the weak_ptr gauges_ change into OTelCollector, the
Test 1 standalone-store fix in TESTING.md, and the collection-lifecycle
calls in the StatsD test that phase-7 requires.
Merges pratik/otel-phase8-log-correlation into pratik/otel-phase9-metric-gap-fill.
Auto-merged. TESTING.md picked up phase-8's Test 4 rewrite alongside this
branch's Step 2 standalone-store fix. OTelCollector.cpp gained the weak_ptr
gauges_ list; the StatsD test gained its onCollectionReady() calls.
Merges pratik/otel-phase7-native-metrics into pratik/otel-phase8-log-correlation.
Conflict was one TESTING.md hunk under "Nodes not reaching proposing state":
this branch renamed the node directories to Node-N in integration-test.sh,
phase-7 kept nodeN and expanded the [peer_private] explanation. Resolution
keeps this branch's Node-1 path (its own script uses that naming) and
phase-7's fuller prose citing peerfinder/Config.cpp.
Non-conflicting phase-7 changes come through: OTelCollector's gauges_ list
becomes weak_ptr, matching the earlier hooks_ change; the phase-6 revert of
the StatsD-test onCollectionReady() calls resolved against phase-7's version
that keeps them.
Merges pratik/otel-phase6-statsd into pratik/otel-phase7-native-metrics.
Phase-6 dropped the three onCollectionReady() calls that had been added to
its StatsD test, because that method is only declared here on phase-7.
This branch's own copy of the file was unchanged from the merge base, so
the default merge would have silently deleted the calls from here too —
where they are needed, because this branch gates polling behind
onCollectionReady() in OTelCollectorImp::onTimer.
Resolution keeps both sides: phase-6's two new include lines
(Counter.h, Gauge.h) and phase-7's three onCollectionReady() calls plus
their doxygen and inline explanations. The merged file is exactly
phase-7's tip plus those two includes.
TESTING.md auto-merged cleanly; both sides added text under Test 1 in
different regions.
The counter and event landed in the same commit as Task 6, so the runbook
recipe's step 5 stands on its own now.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `consensus_view_change_total{consensus_mode}` and a `view.change` event
on the round span, both fired from `RCLConsensus::Adaptor::getPrevLedger`
on the same transition-into-WrongLedger edge that already calls
`consensusViewChange()`. The counter is the exact detector for
"consensus disagreed with this node's view this minute"; the event lands the
disagreement on the same trace that carries the round.
`net_ledger_prefix` (16 hex chars) joins `prev_ledger_prefix` on the event,
so a Tempo view of one flap shows both ledger identities on a single line.
`consensus_mode` labels the mode being left (never WrongLedger itself).
Together with the rotation-phase spans, this closes the proof chain a
rotation-driven `full`->`syncing` flap needs — the runbook's step 5
resolves now that the counter and event exist.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three new panels on `ledger-sync-health.json`, cloned from existing panels
(id 41 for timeseries, id 27 for bargauge) so template filters, tooltip
mode, `xrpl_ident` legend idiom and `spanNulls` all match the surrounding
dashboard exactly. Every existing panel is untouched.
- id 73 bargauge Job queue row x=12 y=226 Job Stalls >=1 s (Count By Job Type)
- id 74 timeseries Back-fill row x=0 y=429 Rotation Phase Duration (p95 by stage)
- id 75 timeseries Back-fill row x=0 y=439 Cache Lock Hold Peak (us)
Count panels use `increase(...[$__range])` per memory
promql-counting-events-interval-not-rate-interval. Units set explicitly
(short / s / us).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `nodestore.rotate` and its eight phase children to expected_spans.json,
all `optional: true` because the 5-node localhost harness cluster never reaches
`online_delete`. Their parent-child relationships are asserted but skip-marked
so a run without a rotation stays green.
Adds `cache_metrics{metric="treenode_lock_hold_peak_us"|"fullbelow_lock_hold_peak_us"}`
to the asserted sync_diagnostics group -- both are observable and always emit,
even at zero. Puts `rotation_phase_duration_seconds` and `jobq_stall_total` in
`not_asserted.metrics_excluded`; both are workload-gated.
On the Cloud collector, adds an `ottl_condition` policy that keeps any trace
carrying a span whose name matches `^nodestore\.rotate`, so the 0.5% probabilistic
tail sampler cannot drop a rotation trace. Sampler is OR'd across policies.
Emits `nodestore.rotate` as a fresh trace root at the start of every rotation
in `SHAMapStoreImp::run`, with one child span per phase: `clear_prior`,
`copy`, `freshen.keys`, `freshen.fetch`, `new_backend`, `clear_caches`,
`swap`, and `health_wait`. `RotationPhase`'s destructor also records the
phase's wall-clock into `rotation_phase_duration_seconds{stage}`, so a
sampled trace and an exact histogram both reach Grafana.
The root carries `ledger_seq` and `last_rotated`; every phase carries a
count attribute the phase already computed (`node_count`, `key_count`,
`copy_forwards`) so nothing extra runs to satisfy telemetry. Outcome is one
of `complete|expired|stopping|missing_node`, stamped by a `RotationOutcome`
RAII helper on whichever exit runs first, and asserted in its destructor to
catch a new return path that forgot to name one.
`freshenCache` is split so `getKeys()` is timed apart from the fetch loop —
that split is what makes `freshen.keys` overlap the frozen receive handlers
in a Tempo view. The `health_wait` child opens only inside a running
rotation, so the pre-rotation `healthWait()` at the top of `run()` never
mints an orphan root.
nodeIdentity_ is no longer a std::optional, so setNodeId() must read it
directly. The merge could not flag this: phase-8 changed the member's type and
this line lives only on phase-9, so neither side of the merge touched the same
file region.
