Sync-critical job types run at very low concurrency limits (ledgerRequest
and ledgerData allow 3 each), so a node can stall simply because those
jobs are held back behind other work. Nothing exposed that until now:
the existing job metrics are rates and quantiles of jobs that already
moved, or a single queue-wide depth.
- jobq_backlog{metric,job_type}: instantaneous waiting, running and
deferred counts per job type. Deferred is the starvation signal and had
no exposure anywhere; it is set when a type is at its concurrency limit.
- jobq_saturation{metric}: running tasks, worker-thread count and total
waiting, so a slowdown spanning several subsystems can be attributed to
worker-pool exhaustion instead of being diagnosed once per victim.
Both read through two new const accessors on JobQueue that take the
existing mutex once and copy integers, so a single reading is internally
consistent and no per-job cost is added. The job_type label reuses the
same JobTypes name helper the existing job counters use, so the two label
sets join.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signals that separate a sync that is merely slow from one that will never
finish:
- sync_acquire{missing_state_nodes_max, missing_tx_nodes_max, in_flight,
received_data_depth}: how many SHAMap nodes each in-flight acquire is
still waiting for. getMissingNodes already computed this and the callers
discarded it after a trace log. A count that stays flat means the
acquire is wedged; a shrinking count means it is progressing. Recorded
once per sweep, never inside the per-node walk, and reset when a tree
completes so a finished acquire does not read as stuck forever.
- shamap_cache_hit_rate{treenode}: hit rate of the in-memory tree-node
cache, which sits above the node store, so it is distinct from the
existing NuDB ratio. A cold cache on a fresh node sends every traversal
step to disk.
- sync_acquire_no_progress_total: timer ticks where an acquire made no
progress, previously only logged.
- sync_addnode_total{good,duplicate,invalid}: whether arriving nodes are
useful, duplicated or rejected, so wasted fetch work is visible.
- sync_acquire_source_total{local,network}: whether a ledger was served
from the local store or had to be fetched.
Adds getBad()/getDuplicate() to SHAMapAddNode and an acquireProgress()
accessor on InboundLedgers so the xrpld gauge can read these without
libxrpl depending on telemetry.
ledger_seq is deliberately not a metric label: it is unbounded. Per-ledger
identity stays on the ledger.acquire span; the metrics expose bounded
aggregates instead.
The full-below cache hit rate is not exported: KeyCache updates different
counters than getHitRate() reads, so it would always report zero. That
libxrpl bug is documented rather than papered over.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five signals that explain why a node is not advancing toward full, none of
which were observable before:
- state_changes_total now carries {from,to} mode labels, emitted at
setMode using the existing strOperatingMode helper. A bare count could
not distinguish a healthy climb from a node flapping between tracking
and connected. Removes the now-unused incrementStateChanges wrapper.
- sync_state{initial_full_duration_us}: time to first reach full, which
StateAccounting already computed but exposed only in server_info.
- sync_state{network_ledger_gate}: whether the node is still refusing to
build ledgers because it has no network ledger.
- sync_state{server_stall_seconds} and server_stall_events_total: how
long the main thread has been unresponsive. LoadManager computed this
and only logged it, so a stall was invisible until the fatal threshold.
The episode rule is a pure function so it can be tested without adding
a test-only mutator to LoadManager.
- sync_state{ledgers_behind}: how far our validated sequence trails the
best sequence any peer advertises, read from already-cached peer ranges
so no extra network traffic is added.
Also fixes the naming checker: it derived only the first label of a
multi-label instrument, so a dashboard querying the second label was
wrongly rejected.
Note: the clang-tidy hook cannot run in this worktree (no build
directory); the remaining pre-commit hooks, the naming check, dashboard
schema and harness syntax all pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three defects found reviewing the WP-A1 commit:
- Handshake.cpp moved a std::string into std::runtime_error, which has no
rvalue constructor. The move never happened and clang-tidy rejects it
under performance-move-const-arg, so CI would fail even though the
local hook only runs clang-tidy with TIDY=1. Takes the message by const
reference instead, and drops the <utility> include that existed only
for that move.
- ValidatorList disables quorum by returning SIZE_MAX. Casting that to
int64_t wrapped it to -1, so the headroom panel computed
0 - (-1) = +1 and coloured yellow on a node that can never validate:
the sign inverted in exactly the bootstrap failure these signals exist
to catch. Reports the disabled state as int64 max so headroom goes
strongly negative instead.
- The runbook claimed an expired list loads no keys. Expired counts as
accepted, so its keys are loaded and then dropped by the expiry sweep,
which calls for a different fix than replacing validators.txt. Pending
is likewise a future-dated refresh, not a rejection. Documents both,
plus how the quorum-disabled state now reads.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A freshly started node most often stalls before it ever peers or reaches
quorum, and that whole chain had no telemetry. Adds the six signals that
make it observable:
- dns_resolve_total / dns_resolve_latency_ms: configured-peer hostname
resolution, emitted from OverlayImpl so libxrpl stays independent.
- overlay_connect_total / overlay_dial_latency_ms: outbound dial outcome
by terminal reason, plus dial duration.
- handshake_negotiation_fail_total: protocol and network-id negotiation
rejections, labelled by reason, so a misconfigured network is no longer
indistinguishable from unreachable peers.
- unl_fetch_total and the unl_quorum gauge: validator-list fetch outcome
per site and trusted key count against the required quorum. Without
these a bad validators.txt leaves the node syncing forever with no
signal.
- clock_close_offset_seconds: network close-time offset, which server_info
hides below 60s but which stalls consensus participation.
Panels land in the Bootstrap row of the Ledger Sync Health dashboard, the
metrics are asserted by the workload validator, and both the reference and
the runbook flow describe them.
Levelization baseline regenerated: overlay now includes MetricMacros.h, so
the overlay/telemetry pair is reported one-way instead of bidirectional.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Adds the anchors the sync-diagnostics signals attach to, with no signals
emitted yet:
- New "Ledger Sync Health" dashboard (uid ledger-sync-health) with the
standard template-variable block copied from an existing board, plus
empty "Bootstrap (Domain 0)" and "Sync pipeline" rows.
- Signal index section in the data-collection reference, an operator-flow
stub in the telemetry runbook, and a glossary anchor.
- A sync_diagnostics group in expected_metrics.json and a matching
assertion helper in validate_telemetry.py so CI fails when a signal
regresses to absent.
Also registers the new dashboard uid with the harness so the board is
covered by validation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add current_ledger_seq / current_ledger_hash to the tx.process, tx.receive,
and txq.enqueue span-reference rows, correct the txq.enqueue parent note
(parents to tx.process on the submission path via explicit context; a root on
the open-ledger rebuild path), and add a "Correlating a transaction to the
ledger it was worked on" recipe joining the txID-keyed tx/txq spans to the
ledger trace via current_ledger_seq.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a "Protocol Span Flow" section to the telemetry operator runbook: 8
Mermaid diagrams that map every OTel span onto the real xrpld control flow
and XRPL protocol order (verified against code and docs/consensus.md), for
use as the canonical key when linking the span hierarchy.
- Master overview, client/peer ingress, shared apply pipeline, consensus
round, accept/build/finalize, and pathfinding/ledger-acquire side flows.
- Every node/branch is labelled with the span that represents that state or
transition (or explicit "(no span)"); drops/abandons are marked terminal.
- Shows real loops, retries, recovery, and drop branches: multi-round
consensus settling (avalanche threshold rounds + MovedOn/Expired retry with
wrong-ledger recovery), 3-pass tx apply retry, TxQ cross-ledger retry,
quorum-gated async validation with abandoned-ledger, ingress backpressure
drops, and cross-node context propagation.
- Adds a divergence table noting where OTel span parenting does not match the
real protocol flow (deterministic/hash trace-id roots, JtAccept/JtUpdatePf
job hand-offs, sequential peer->consensus receive stages).
Replace the stale planning-era diagram in OpenTelemetryPlan/08-appendix.md
(which named spans that were never built) with a pointer to the runbook.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The check-rename CI gate requires all prose/comment references to use the
new naming. Fix the four remaining occurrences ("Ripple epoch",
"rippled's doAccept") introduced by this branch's dashboard/glossary
commits. Comment- and doc-only; no behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
P95 of second-scale spans was a meaningless interpolation. The spanmetrics
histogram topped out at [.. 1s, 5s], so consensus.round (~3.9s) and
consensus.establish (~1.9s) all fell into one 1s-5s bucket and
histogram_quantile interpolated linearly across that 4s-wide gap — the
"Build vs Close" / "Ledger Close Duration" panels' P95 read ~4800ms purely
as an artifact (verified: sum/count avg = 3824ms). ledger.acquire was worse:
~17% of samples exceeded the 5s ceiling, so its p95/p99 were unmeasurable.
Add 2s, 3s, 4s (resolve the 1-5s pile-up) and 10s, 30s (give the
ledger.acquire catch-up tail a measurable home). All ten existing boundaries
are preserved and the list stays strictly ascending (the connector
binary-searches buckets and silently misbuckets otherwise). Pin unit=ms so a
future collector default-unit flip can't rename the metric to _seconds.
Buckets chosen from the live mainnet distribution, not guessed. Native
beast::insight histograms (ms-scale RPC/IO timers in Telemetry.cpp) are 100%
under 5s, so they keep the original buckets — this is collector-only.
Applies on collector restart (cumulative series reset once, handled by
rate()). Runbook and regression-threshold bucket notes updated to match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a scope tag to every panel keyword — (per node), (network-wide),
(network event), or (cluster-wide) — so a reader can tell whether a term
describes this server's own state, a protocol-shared fact, or a
network-wide consensus process the node participates in. Add a matching
'Scope:' line to each glossary entry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a Keywords definition-list and a References link line to 178 Grafana
panel descriptions across all 14 dashboards, defining the XRPL/rippled
domain terms each panel uses (tiers 1-9: ledger, consensus, transaction
pipeline, fees/queue, node state, peer/overlay, storage, validator,
RPC/pathfinding). Cross-cutting chart terms and job-queue internals are
intentionally excluded.
Add docs/telemetry-glossary.md (86 terms, 9 categories) as the deeper
reference, linked from each panel's References line and from the runbook.
Keywords are injected only where a term appears in that panel's prose
(matched over description text, not source-file citations).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "Ledger Close Duration" panel measured the consensus.ledger_close span
duration, which only wraps the sub-millisecond onClose() prologue — not the
ledger close. Repoint it to the consensus.round span (full round, open to
accept) so it reflects actual close time (~3-5s on mainnet). The consensus_mode
filter is preserved (consensus.round carries that attribute, verified live).
The sibling rate panels (Consensus Mode Over Time, Accept vs Close Rate,
Validation vs Close Rate) keep using consensus.ledger_close: a rate of that
span is a valid per-close event counter, only its duration was wrong.
Runbook updated to match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "ledger close time" was mis-derived and partly un-queryable:
- Build vs Close Duration derived close time from the consensus.ledger_close
span, which only wraps the onClose() prologue (~0.8ms live) — not the close.
Repoint the close series to consensus.round (full round, live P95 ~4.8s).
- The network close-time value (close_time) lived only as a span attribute,
un-queryable in Prometheus and unfit as a spanmetrics label (monotonic
timestamp -> unbounded cardinality). Expose it as last_close_time on the
existing server_info observable gauge (native gauge, no new instrument).
- Add a "Ledger Close Interval & Age" panel to ledger-operations and
node-health: interval = 1/rate(ledgers_closed_total) (counter-based,
scrape-independent); age = time() - (last_close_time + epoch_offset).
A gauge delta is deliberately NOT used for the interval — a timestamp gauge's
delta aliases to the scrape period, not the close cadence (verified live).
Guardrail comments in both collector configs record why close_time must never
become a spanmetrics dimension. Docs (09-reference) and the operator runbook
updated to match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
These inbound peer-message entry points (kConsumer) used span(), which
inherits whatever span is active on the peer thread — including a leaked
tx.receive scope — so validations/proposals were wrongly nested under
unrelated transaction traces. rootSpan() starts a fresh trace root.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
detached() strips the thread-local OTel Scope so a guard can be safely
moved to and destroyed on another thread; it pops the Scope on the origin
thread and moves the span into a scope-less guard. rootSpan() starts a span
as a fresh trace root (kIsRootSpanKey) so inbound entry points never inherit
an ambient span left active on the thread.
Impl now holds an optional<Scope> (nullopt for detached guards). Updated the
SpanGuard class docs and docs/build/telemetry.md with the cross-thread rules.
The unit test lands on phase2 where the telemetry test module exists.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Document the three perf-iac filter template variables (xrpl_work_item,
xrpl_branch, xrpl_node_role) in the telemetry runbook and the data-collection
reference: what they filter, their example values, and that perf-iac stamps
them as resource attributes from its own alloy pipeline (absent outside
perf comparison runs, so the filters default to All). Also record perf as a
network value.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dashboard audit against the live datasource surfaced two broken panel queries
and applied UX polish across all 14 dashboards.
- node-health "Job Queue Depth": job_count -> jobq_job_count. The JobQueue
collector is wrapped in group("jobq") (Application.cpp), so the registered
job_count gauge is emitted with the jobq_ prefix; the panel queried the
unprefixed name and returned nothing.
- network-traffic "Overlay Traffic by Category" + "All Traffic Categories":
topk(N, rate({__name__=~".*_bytes_in"}[...])) errors on Mimir ("vector
cannot contain metrics with the same labelset") because rate() drops
__name__ and the many counters collapse. Replaced with an enumerated
label_replace form that re-attaches __name__ per metric, preserving the
{{__name__}} legend and per-series display-name overrides.
- All 14 dashboards: refresh set to 10s.
- peer-quality: each panel full screen width.
- validator-health: at most two panels per row (row headers preserved).
- docs: telemetry-runbook and 06-implementation-phases updated for the
jobq_ prefix and the network-traffic query pattern.
Verified end-to-end against a local mainnet xrpld node feeding the local
stack: jobq_job_count returns data (old job_count empty), both network-traffic
exprs execute (old form reproduces the labelset error), and panels render
through the Grafana proxy. All 14 pass validate_dashboards.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the TxQ Accept Status piechart on the Transaction Overview
dashboard with a state-timeline showing each node's applied fraction of
TxQ accepts (applied / applied+failed) over time, colored by threshold
(green >=0.9, yellow >=0.7, red below). Remove the now-orphaned
txq_status template variable (the piechart was its only consumer) and
document the panel in the telemetry runbook.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
