Three measurement fixes landed with no doc or dashboard change, leaving text
that is now false and one fix unusable from a dashboard.
Deferrals and timeouts are recorded in TimeoutCounter, a base shared by five
subclasses, so the all-lane pair could show the documented livelock
fingerprint while ledger acquisition was healthy. The runbook procedure and
the reference doc now name acquire_ledger_deferrals and
acquire_ledger_timeouts and say why the all-lane pair misleads; a new panel
plots the ledger-scoped pair as rates on one axis, since the divergence is
the signal. The existing panel is retitled All Lanes and points at it.
Writer mean depth is depthSum over depthSamples, not over insertCount, and
the measured 1.60 came from the biased estimator, so it and the 37% queueing
share derived from it are lower bounds rather than values. The reference
table now marks them as such, and the decision rule is shown to survive the
correction rather than depending on the exact figures.
Completions were never counted for acquisitions satisfied from the local
store, so the run that read zero across 510 seconds had in fact reached
full. Every place that treated a zero as a symptom now says it only means
something on a build that has the fix.
Also corrects the sync-diagnosis label-value count from 13 to 15 and a stale
source line range; the instrument count stays 35, because both new values
multiplex onto the existing nodestore_state gauge.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drops nodestore_read_us and everything added to reach it. read_mean_us already
carries microsecond precision and separated the two sync failure modes cleanly
in live testing -- 8.8 us on a clean store against a 223 us cold-store peak --
so the distribution added no signal that changed a diagnosis.
The cost of getting it was disproportionate. NodeStoreScheduler had no path to
the metrics registry, so its production constructor grew a ServiceRegistry
parameter: a metric addition changing a production signature. That in turn
forced an edit to a pre-existing test, src/test/app/SHAMapStore_test.cpp, whose
only stake in this is that it constructs a scheduler. Worse, the scheduler is
built in Application's member initializer list, long before metricsRegistry_
exists, so the registry could not be captured once and had to be re-resolved on
every fetch -- a lookup on a path that runs millions of times per sync.
The constructor returns to taking JobQueue& alone and SHAMapStore_test.cpp
returns to the single-argument call, leaving that file differing from its
pre-change form only by the NodeStore:: to node_store:: rename it picked up from
develop.
FetchReport::elapsed stays microseconds and onFetch keeps its explicit
duration_cast to milliseconds for addLoadEvents, which takes milliseconds. That
widening was a separate fix and is what makes read latency measurable at all.
kSubMillisecondBoundaries loses its only consumer and regains [[maybe_unused]],
which is the state the commit that introduced it left it in; without the
attribute an unused constant is an error under wextra with werr.
Also removes the ledger-data-sync panel that charted the histogram and the
fetch_type and found template variables, which filtered on labels no metric
emits any more, plus the runbook and reference-doc sections and the two
instrument and view counts that named it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The nudb_bytes label value on the storage_detail gauge named something the
code never measured. It observes Database::getStoreSize(), which returns the
storeSz_ accumulator: the cumulative payload bytes of objects this process
handed to the NodeStore. That is not a NuDB file size. It excludes NuDB's
keys, bucket padding and log, and it resets with the process while the files
on disk do not.
The name caused two concrete errors. It invited sizing the store on disk from
a number that cannot do it, and it invited a write-amplification ratio against
node_written_bytes -- which reads the same accessor at MetricsRegistry.cpp:836,
so that ratio is a constant 1.0 and measures nothing.
The nudb_ prefix was wrong too. storeSz_ is written only by
Database::storeStats(), called from DatabaseNodeImp, DatabaseRotatingImp and
Database itself. No backend code touches it, so the value reads the same on
RocksDB. That distinguishes it from the real nudb_* family
(nudb_writers_in_flight and friends), which come from getWriteStats() and are
absent entirely on a non-NuDB backend.
stored_object_bytes says what the value is and claims nothing about the
filesystem. Docs already described the value correctly; they keep that
explanation and now also record the old name, so a query pinned to it can be
traced. Neither Backend nor Database exposes an on-disk size accessor and none
was added -- no metric reports the store's on-disk size today.
Updates the node-health panel title, description and PromQL, and the four docs
that name the label value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Brings forward the node_reads_hit and nudb_bytes label corrections from phases 7
and 9.
Conflicts resolved keeping both sides:
- 06-implementation-phases.md: kept phase-10's unprefixed `storage_detail` gauge
name with phase-7's corrected getStoreSize() description.
- node-health.json: kept phase-10's rewritten panel structure and re-applied the
panel renames (NodeStore Read Found Ratio, NuDB Stored Bytes), legends and axis
labels on top of it.
node_reads_hit is a found count, not a cache hit. fetchHitCount_ increments
whenever a fetch returned an object, whatever served it, and a node with
online_delete uses DatabaseRotatingImp which has no NodeObject cache at all. The
ratio therefore reads near 100% while every fetch goes to disk, which made the
cold-read failure mode look impossible on the board.
- Phase9_taskList: node_reads_hit is a found count, not cache-served reads.
- node-health: "NodeStore Cache Hit Ratio" -> "NodeStore Read Found Ratio",
legends "Cache Hit Ratio" -> "Found Ratio" and "Reads Hit (cache)" ->
"Reads Found"; corrected the ratio panel's axis label, which read
"Operations / Sec" on a percentunit panel.
- node-health: "NuDB Storage Size" -> "NuDB Stored Bytes". nudb_bytes observes
getStoreSize(), the accumulator node_written_bytes also reads, so it is
cumulative object-payload bytes and not a filesystem measurement.
- ledger-data-sync: point the read-latency discriminator at the renamed panel.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
nudb_bytes was documented as a NuDB file size, one place even claiming a
filesystem stat. It observes Database::getStoreSize(), which sums the object
payloads this process has written. It excludes NuDB's keys, bucket padding and
log, and resets with the process. node_written_bytes calls the same accessor, so
the two series are equal by construction and a write-amplification ratio built
from them is a constant 1.0. Neither Backend nor Database exposes a file-size
accessor, so nothing reports on-disk size today.
The Ledger Data & Sync panel plotting node_reads_hit / node_reads_total was
titled "NuDB Cache Hit Ratio" and described as reads served from cache.
fetchHitCount_ increments whenever a fetch returned an object, whatever served
it, so the ratio is a found rate. It reads near 100% while every fetch goes to
disk, which made the cold-read failure mode look impossible. Renamed to
"NuDB Read Found Ratio" and rewrote the guidance to pair it with read latency.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The decision rule for "slow to reach full" keyed on an absolute read-time
threshold and a found-rate threshold that misclassified the very run they
were written to explain: a populated-store run reading 31.8 us at 88.3%
found fell through both cuts and came out as "disk-bound" rather than the
cold-read case it is.
Replace it with two ordered questions -- is the read cost several times a
warm read, and is the write path queueing -- and demote the found rate to a
splitter that only applies once reads are known to be expensive. A high
found rate on its own is the normal state of a populated store, so it can
never be a trigger. The rule now classifies all four reference datasets
correctly, and the runbook shows the rule applied to each so the "confirm
against the reference points" step agrees with the table.
Also in the runbook:
- name the source of the devnet incident figures at the point of use, and
point forward to the caveat from the same paragraph
- state the provenance of the measured columns, and split the incident
figures into their own table marked as not our measurement
- say plainly that the compounding-factor explanation is an unconfirmed
hypothesis
- correct the deferral gate: it fires at the acquisition's own job limit of
5, not at the ledgerData lane cap of 3
- note that no read-max gauge exists, so the tie-break uses max_over_time
of the mean or the read histogram's p99
- split a PromQL block that put two expressions on adjacent lines, which
parses as one invalid expression
In the data-collection reference:
- node_reads_hit counts fetches that found an object, not cache hits
- nudb_bytes is cumulative payload bytes from the same accessor as
node_written_bytes, not on-disk size, so their ratio is a constant 1.0
- write_load and nudb_writers_in_flight are the same atomic on NuDB;
document that and what write_load means on RocksDB
- correct the instrument count to 8 and the view count to 7 after
nodestore_read_us and its view were added
- scope the Phase 9 query examples to one node with the regex form
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two different bottlenecks both present as the ledgerData job lane pinned
at its concurrency cap of 3, so lane occupancy diagnoses neither. One is
write-serialized (NuDB takes one global mutex per insert, so inserts
queue), the other is cold-read-bound on a populated store. Telling them
apart needs the storage-side signals, not the lane.
Adds to docs/telemetry-runbook.md a "Slow to reach full" procedure: a
Mermaid diagram of the two modes, a decision table keyed on whether
acquisitions are completing, the measured reference values from both
runs, and the deferral/timeout pair that fingerprints the disarmed
give-up path. States plainly that node_reads_hit is a found count rather
than a cache-hit rate, which is why a ~100% "hit rate" at 113 us per
read is the cold-read signature and not a contradiction.
Records honestly that the populated-store run was twice as fast despite
slower reads, so cold reads alone do not explain the long incident.
Adds reference rows for the 13 new nodestore_state label values and the
nodestore_read_us histogram to
OpenTelemetryPlan/09-data-collection-reference.md, in the authoritative
Phase 9 OTel SDK section alongside the existing NodeStore I/O table.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolved five files. In each case both sides had content worth keeping,
so nothing was taken wholesale:
- MetricsRegistry.cpp: kept the incoming `handler` label on the job
instruments and re-applied this branch's `queuedDurUs >= 0` guard,
which the incoming side does not have.
- telemetry-runbook.md: took the incoming gauge table, which adds the
three per-job-type rows, and re-applied this branch's corrected
`jobq_job_count` name.
- 09-data-collection-reference.md: kept this branch's validation
inventory (newer counts, extra Config File column) and inserted the
incoming call-site and per-job-type gauge rows plus their explanation.
- node-health.json: merged structurally rather than by text. This
branch's panels are authoritative; only the two incoming job-queue
panels were appended, below the existing layout. The
`Validated Ledger Seq` panels added directly in Grafana are preserved.
- job-queue.json, ledger-data-sync.json: panel sets were identical, so
took the incoming side for its `$handler` variable, the handler filter
on existing queries, and the new panels. Verified no panel was lost.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolved OpenTelemetryPlan/09-data-collection-reference.md: both sides
document the same new metrics, reached by different routes. The incoming
sections came from the phase-6 edit merged forward; this branch already
documents the same surface in its own layout, with the job lifecycle
metrics and the GetObject request path under the MetricsRegistry
section rather than a separate 2a.
Kept this branch's sections to avoid duplicate entries for every metric.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Resolved OpenTelemetryPlan/09-data-collection-reference.md: this branch
replaced the StatsD-shaped Section 2 with OTel-native naming, so the
incoming StatsD tables and the StatsD per-job timer section do not apply
here. Kept this branch's names throughout.
The incoming per-job-type saturation gauge section is new content and is
retained, renumbered to 2.5 and rewritten for the OTel naming: bare
lowercase `jobq_<jobtype>_{waiting,running,deferred}` derived through
OTelCollector rather than the case-preserving `xrpld_`-prefixed StatsD
form.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Slowness on the peer object-fetch path could be observed but not
attributed. Job duration metrics carry only `job_type`, and both
`RcvGetLedger` and `RcvGetObjByHash` report as `ledgerRequest`, so a
queue-wait spike could not be traced to a handler. Nothing measured
NodeStore cost, request size, or the differential charge.
Latency now decomposes into three additive parts, each separately
measurable:
end-to-end = queue wait + NodeStore lookup + everything else
- `handler` label on job_queued_total/_started_total/_finished_total and
job_queued_us/job_running_us. The value is sanitised: a name passes
through only if non-empty and all ASCII letters, else "other". Two job
names embed a ledger sequence, so a raw label would mint one series
per ledger; the rule bounds the domain at 43 names plus "other".
- getobject_lookup_us, _request_objects, _lookups_total{result},
_rejected_total{reason} and _charge, recorded at their call sites.
All three histograms get explicit bucket views: the SDK default stops
at 10,000, which every one of them exceeds.
- Per-job-type waiting/running/deferred gauges for the 35 non-special
job types. `deferred` is the leading indicator, since addJob never
rejects -- it defers, so backpressure otherwise shows up only as
latency after the fact.
`JobQueue::collect()` snapshots the counters under the queue lock and
publishes gauges after releasing it. Writing them while holding the lock
would invert a lock order against the collector's own lock, which the
collector's flush thread already holds when it calls this hook.
Tests assert exact values, including that the charge is priced on the
requested count rather than the capped iteration count.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add reference entries for the observability surface introduced on
phase-9: the `handler` label on the job instruments, the five
`getobject_*` request metrics, and the per-job-type queue saturation
gauges.
Names here follow this branch's StatsD pipeline, which preserves case
and carries the `xrpld_` prefix, so they differ from the lowercased
OTel-native names used from phase-7 onward. The sections state where
the implementing code lives, since it is introduced downstream.
Also correct pre-existing entries: `job_count` exports as
`jobq_job_count` via the collector group prefix, the non-special job
type count is 35 (not 36), and `JtLedgerData` has five producers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Document the two new ledger-correlation attributes in the data-collection
reference: add them to the transaction and TxQ attribute tables (with per-span
coverage), update the attribute count, and correct the txq.enqueue parent note
(parents to tx.process on the submission path via explicit context; a root on
the open-ledger rebuild path, correlated via current_ledger_seq).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a "Protocol Span Flow" section to the telemetry operator runbook: 8
Mermaid diagrams that map every OTel span onto the real xrpld control flow
and XRPL protocol order (verified against code and docs/consensus.md), for
use as the canonical key when linking the span hierarchy.
- Master overview, client/peer ingress, shared apply pipeline, consensus
round, accept/build/finalize, and pathfinding/ledger-acquire side flows.
- Every node/branch is labelled with the span that represents that state or
transition (or explicit "(no span)"); drops/abandons are marked terminal.
- Shows real loops, retries, recovery, and drop branches: multi-round
consensus settling (avalanche threshold rounds + MovedOn/Expired retry with
wrong-ledger recovery), 3-pass tx apply retry, TxQ cross-ledger retry,
quorum-gated async validation with abandoned-ledger, ingress backpressure
drops, and cross-node context propagation.
- Adds a divergence table noting where OTel span parenting does not match the
real protocol flow (deterministic/hash trace-id roots, JtAccept/JtUpdatePf
job hand-offs, sequential peer->consensus receive stages).
Replace the stale planning-era diagram in OpenTelemetryPlan/08-appendix.md
(which named spans that were never built) with a pointer to the runbook.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Brings coroutine-aware context storage + tx/consensus worker-body activation.
Resolved: Telemetry.cpp keeps both meterProvider_ (phase-7) and contextStorage_
(coro-aware); doc-09 keeps phase-7 structure and applies the pathfind.request →
rpc.command.<name> correction to phase-7's own PathFind section.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The check-rename CI gate requires all prose/comment references to use the
new naming. Fix the four remaining occurrences ("Ripple epoch",
"rippled's doAccept") introduced by this branch's dashboard/glossary
commits. Comment- and doc-only; no behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reflects the C1 fix: rpc.command.* stays unscoped (its dispatch wraps
doRipplePathFind which yields), so pathfind.request nests under rpc.process.
The request -> compute -> discover sub-tree nests correctly via ScopedSpanGuard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "ledger close time" was mis-derived and partly un-queryable:
- Build vs Close Duration derived close time from the consensus.ledger_close
span, which only wraps the onClose() prologue (~0.8ms live) — not the close.
Repoint the close series to consensus.round (full round, live P95 ~4.8s).
- The network close-time value (close_time) lived only as a span attribute,
un-queryable in Prometheus and unfit as a spanmetrics label (monotonic
timestamp -> unbounded cardinality). Expose it as last_close_time on the
existing server_info observable gauge (native gauge, no new instrument).
- Add a "Ledger Close Interval & Age" panel to ledger-operations and
node-health: interval = 1/rate(ledgers_closed_total) (counter-based,
scrape-independent); age = time() - (last_close_time + epoch_offset).
A gauge delta is deliberately NOT used for the interval — a timestamp gauge's
delta aliases to the scrape period, not the close cadence (verified live).
Guardrail comments in both collector configs record why close_time must never
become a spanmetrics dimension. Docs (09-reference) and the operator runbook
updated to match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Document that deterministic-trace_id spans (tx.* apply pipeline, tx.process,
tx.receive, consensus.round) are now genuine trace roots with empty
parent_span_id via the custom DeterministicIdGenerator, superseding the old
synthetic-parent behavior that showed 'root span not yet received' in Tempo.
Also update the fresh-root note: peer entry spans use ScopedSpanGuard::freshRoot().
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
