Quorum and publish (A5):
- ledger_quorum_publish gauge: the trusted-validation tally against the
quorum the candidate ledger must reach, plus the gap between the
validated and published sequences. The published sequence was never
exported, so a publish pipeline falling behind healthy validation was
invisible.
- ledger_quorum_shortfall_total: counts the pre-accept early return where
a node has peers and validators yet still declines to declare a ledger
validated. That path was log-only, and it is the difference between
accumulating toward quorum and never reaching it.
Back-fill and persistence (A6):
- nodestore_latency: read and write service time. storeDurationUs_ was
declared but never written and had no accessor, so there was no write
latency signal at all. This is the direct fingerprint of a node with an
existing database syncing slower than a fresh one, where the node cache
is cold and every tree step reaches disk. Distinct from the existing
NuDB read panels, which show volume and hit ratio rather than service
time.
- ledger_replay_fallback_total and ledger_replay_outcome_total: the replay
path silently falls back to plain acquisition on timeout or failure, so
a defeated optimisation looked like ordinary slow back-fill.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Whether the network can even serve this node, and whether the node is
about to be shut out of validation, were both invisible:
- peer_ledger_supply: how many connected peers advertise a range covering
the sequence being fetched. Peers each track a range from status
changes, but nothing aggregated them, so "nobody has what I need" looked
identical to "peers are slow".
- peerfinder_slot_census: outbound active against capacity, connection
attempts, inbound, fixed configured against active, and the bootcache
and livecache sizes. All were computed already; only two were exported,
read at unrelated instants, so they could not be compared.
- peer_disconnect_total{reason,direction} and peer_accept_total{outcome}:
every disconnect previously collapsed into one number, so our own
backpressure could not be told from topology or network faults. Reasons
are a fixed set of literals recorded on the peer and emitted once at
close, never data supplied by the remote end.
- serve_refused_total{request,reason}: the other half of the sync
exchange, when this node declines to serve a peer.
- amendment_block: whether an unsupported amendment is expected and how
long until it activates. Amendment-blocked is terminal for validation,
so the countdown is the only leading indicator. The amendment id is
deliberately not a label, since the network can vote an id this build
has never heard of; it is already logged.
- ledger_jump_total: repeated last-closed-ledger switches, which mean the
node is thrashing between chains.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sync-critical job types run at very low concurrency limits (ledgerRequest
and ledgerData allow 3 each), so a node can stall simply because those
jobs are held back behind other work. Nothing exposed that until now:
the existing job metrics are rates and quantiles of jobs that already
moved, or a single queue-wide depth.
- jobq_backlog{metric,job_type}: instantaneous waiting, running and
deferred counts per job type. Deferred is the starvation signal and had
no exposure anywhere; it is set when a type is at its concurrency limit.
- jobq_saturation{metric}: running tasks, worker-thread count and total
waiting, so a slowdown spanning several subsystems can be attributed to
worker-pool exhaustion instead of being diagnosed once per victim.
Both read through two new const accessors on JobQueue that take the
existing mutex once and copy integers, so a single reading is internally
consistent and no per-job cost is added. The job_type label reuses the
same JobTypes name helper the existing job counters use, so the two label
sets join.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signals that separate a sync that is merely slow from one that will never
finish:
- sync_acquire{missing_state_nodes_max, missing_tx_nodes_max, in_flight,
received_data_depth}: how many SHAMap nodes each in-flight acquire is
still waiting for. getMissingNodes already computed this and the callers
discarded it after a trace log. A count that stays flat means the
acquire is wedged; a shrinking count means it is progressing. Recorded
once per sweep, never inside the per-node walk, and reset when a tree
completes so a finished acquire does not read as stuck forever.
- shamap_cache_hit_rate{treenode}: hit rate of the in-memory tree-node
cache, which sits above the node store, so it is distinct from the
existing NuDB ratio. A cold cache on a fresh node sends every traversal
step to disk.
- sync_acquire_no_progress_total: timer ticks where an acquire made no
progress, previously only logged.
- sync_addnode_total{good,duplicate,invalid}: whether arriving nodes are
useful, duplicated or rejected, so wasted fetch work is visible.
- sync_acquire_source_total{local,network}: whether a ledger was served
from the local store or had to be fetched.
Adds getBad()/getDuplicate() to SHAMapAddNode and an acquireProgress()
accessor on InboundLedgers so the xrpld gauge can read these without
libxrpl depending on telemetry.
ledger_seq is deliberately not a metric label: it is unbounded. Per-ledger
identity stays on the ledger.acquire span; the metrics expose bounded
aggregates instead.
The full-below cache hit rate is not exported: KeyCache updates different
counters than getHitRate() reads, so it would always report zero. That
libxrpl bug is documented rather than papered over.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five signals that explain why a node is not advancing toward full, none of
which were observable before:
- state_changes_total now carries {from,to} mode labels, emitted at
setMode using the existing strOperatingMode helper. A bare count could
not distinguish a healthy climb from a node flapping between tracking
and connected. Removes the now-unused incrementStateChanges wrapper.
- sync_state{initial_full_duration_us}: time to first reach full, which
StateAccounting already computed but exposed only in server_info.
- sync_state{network_ledger_gate}: whether the node is still refusing to
build ledgers because it has no network ledger.
- sync_state{server_stall_seconds} and server_stall_events_total: how
long the main thread has been unresponsive. LoadManager computed this
and only logged it, so a stall was invisible until the fatal threshold.
The episode rule is a pure function so it can be tested without adding
a test-only mutator to LoadManager.
- sync_state{ledgers_behind}: how far our validated sequence trails the
best sequence any peer advertises, read from already-cached peer ranges
so no extra network traffic is added.
Also fixes the naming checker: it derived only the first label of a
multi-label instrument, so a dashboard querying the second label was
wrongly rejected.
Note: the clang-tidy hook cannot run in this worktree (no build
directory); the remaining pre-commit hooks, the naming check, dashboard
schema and harness syntax all pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three defects found reviewing the WP-A1 commit:
- Handshake.cpp moved a std::string into std::runtime_error, which has no
rvalue constructor. The move never happened and clang-tidy rejects it
under performance-move-const-arg, so CI would fail even though the
local hook only runs clang-tidy with TIDY=1. Takes the message by const
reference instead, and drops the <utility> include that existed only
for that move.
- ValidatorList disables quorum by returning SIZE_MAX. Casting that to
int64_t wrapped it to -1, so the headroom panel computed
0 - (-1) = +1 and coloured yellow on a node that can never validate:
the sign inverted in exactly the bootstrap failure these signals exist
to catch. Reports the disabled state as int64 max so headroom goes
strongly negative instead.
- The runbook claimed an expired list loads no keys. Expired counts as
accepted, so its keys are loaded and then dropped by the expiry sweep,
which calls for a different fix than replacing validators.txt. Pending
is likewise a future-dated refresh, not a rejection. Documents both,
plus how the quorum-disabled state now reads.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A freshly started node most often stalls before it ever peers or reaches
quorum, and that whole chain had no telemetry. Adds the six signals that
make it observable:
- dns_resolve_total / dns_resolve_latency_ms: configured-peer hostname
resolution, emitted from OverlayImpl so libxrpl stays independent.
- overlay_connect_total / overlay_dial_latency_ms: outbound dial outcome
by terminal reason, plus dial duration.
- handshake_negotiation_fail_total: protocol and network-id negotiation
rejections, labelled by reason, so a misconfigured network is no longer
indistinguishable from unreachable peers.
- unl_fetch_total and the unl_quorum gauge: validator-list fetch outcome
per site and trusted key count against the required quorum. Without
these a bad validators.txt leaves the node syncing forever with no
signal.
- clock_close_offset_seconds: network close-time offset, which server_info
hides below 60s but which stalls consensus participation.
Panels land in the Bootstrap row of the Ledger Sync Health dashboard, the
metrics are asserted by the workload validator, and both the reference and
the runbook flow describe them.
Levelization baseline regenerated: overlay now includes MetricMacros.h, so
the overlay/telemetry pair is reported one-way instead of bidirectional.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add shared current_ledger_seq / current_ledger_hash span attributes so a
transaction's work can be joined to the ledger trace that produced it, and
fix discrepancy D1 (txq.enqueue was a detached trace root).
- Define current_ledger_seq / current_ledger_hash once in SpanNames.h and
re-export via `using` from TxQ/TxApply/Tx span-name headers. These name the
ledger being worked on (open/tentative apply or in-flight consensus build),
distinct from ledger_seq (the built/validated ledger on ledger.build /
consensus.round). Named after the RPC field ledger_current_index.
- txq.enqueue: set current_ledger_seq/hash from the view, and parent the span
to the caller's tx.process span via an explicit captured SpanContext (new
trailing TxQ::apply param) instead of a detached root. The parent is
explicit, not ambient-inherited, and the ScopedSpanGuard scope is RAII-bound
to the synchronous apply, so it cannot leak onto a reused worker (D1 fix).
On the open-ledger rebuild path no tx.process context exists, so it stays a
root and the attribute provides the correlation.
- tx.preclaim / tx.transactor: set both attributes from their ledger view.
tx.preflight is stateless (no view) and is the documented exception.
- tx.process / tx.receive: set current_ledger_seq from the current open ledger
index at submit/receive time (no hash: not yet applied to a ledger).
- Contract test pins the two new attribute key strings.
Neither key is a spanmetrics dimension, so there is no metric-cardinality
impact. Dashboards/collector/docs land on the later phases per the chain split.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the raw "trace_id="/" span_id=" prefix literals and the bare 32/16
hex-width magic numbers in the log trace-context injection with named
constexpr constants (kTraceIdPrefix, kSpanIdPrefix, kTraceIdHexLen,
kSpanIdHexLen), documenting that the widths are the W3C 16-byte trace_id /
8-byte span_id rendered as lowercase hex. No behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Carries the review fixes forward to phase-10: no-alloc GetCurrent hot path,
activateIfLive helper deduplicating the four worker-body activation sites,
non-copyable CoroAwareContextStorage, and the clarified drop-counter doc.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
