kSubMillisecondBoundaries existed but nothing used it, so per-fetch read
latency never reached Grafana -- only the coarse read_mean_us gauge did,
which cannot separate "every read took 9us" from "most took 2 and a few
took 900".
Add a nodestore_read_us histogram, register its view against the sub-
millisecond ladder rather than kMicrosecondBoundaries (whose first edge
is 100us, above the entire range a warm read occupies), and record into
it from NodeStoreScheduler::onFetch using FetchReport::elapsed, which a
previous change widened to microseconds for exactly this purpose.
The name and its labels live in a new include/xrpl/telemetry header
because the view registration (xrpld.telemetry) and the record site
(xrpld.app) sit in different levelization modules; a copy-pasted literal
would let them drift and silently drop the bucket override. Same reason
and same placement as GetObjectMetricNames.h. No new levelization edge:
xrpld.app > xrpl.telemetry already exists.
NodeStoreScheduler had no registry access, so it now takes a
ServiceRegistry and resolves the registry per call. It is constructed in
Application's initializer list, long before metricsRegistry_ is assigned
in setup() and started in startTelemetry(), so capturing a pointer at
construction would capture nullptr forever; the metric macros null-check
the registry, the meter and the instrument, so early fetches are simply
not recorded.
Labels are fetch_type and found, both already carried on the report --
4 series, fixed at compile time. A slow async read delays prefetch while
a slow sync read blocks a caller, and a miss can cost a read of every
backend, so neither dimension can be collapsed.
Negative elapsed times are skipped: the SDK rejects them and logs a
warning on every call, which on a per-fetch path is a log flood. Zero is
still recorded, since a page-cache-served read genuinely rounds to it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Slowness on the peer object-fetch path could be observed but not
attributed. Job duration metrics carry only `job_type`, and both
`RcvGetLedger` and `RcvGetObjByHash` report as `ledgerRequest`, so a
queue-wait spike could not be traced to a handler. Nothing measured
NodeStore cost, request size, or the differential charge.
Latency now decomposes into three additive parts, each separately
measurable:
end-to-end = queue wait + NodeStore lookup + everything else
- `handler` label on job_queued_total/_started_total/_finished_total and
job_queued_us/job_running_us. The value is sanitised: a name passes
through only if non-empty and all ASCII letters, else "other". Two job
names embed a ledger sequence, so a raw label would mint one series
per ledger; the rule bounds the domain at 43 names plus "other".
- getobject_lookup_us, _request_objects, _lookups_total{result},
_rejected_total{reason} and _charge, recorded at their call sites.
All three histograms get explicit bucket views: the SDK default stops
at 10,000, which every one of them exceeds.
- Per-job-type waiting/running/deferred gauges for the 35 non-special
job types. `deferred` is the leading indicator, since addJob never
rejects -- it defers, so backpressure otherwise shows up only as
latency after the fact.
`JobQueue::collect()` snapshots the counters under the queue lock and
publishes gauges after releasing it. Writing them while holding the lock
would invert a lock order against the collector's own lock, which the
collector's flush thread already holds when it calls this hook.
Tests assert exact values, including that the charge is priced on the
requested count rather than the capped iteration count.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add shared current_ledger_seq / current_ledger_hash span attributes so a
transaction's work can be joined to the ledger trace that produced it, and
fix discrepancy D1 (txq.enqueue was a detached trace root).
- Define current_ledger_seq / current_ledger_hash once in SpanNames.h and
re-export via `using` from TxQ/TxApply/Tx span-name headers. These name the
ledger being worked on (open/tentative apply or in-flight consensus build),
distinct from ledger_seq (the built/validated ledger on ledger.build /
consensus.round). Named after the RPC field ledger_current_index.
- txq.enqueue: set current_ledger_seq/hash from the view, and parent the span
to the caller's tx.process span via an explicit captured SpanContext (new
trailing TxQ::apply param) instead of a detached root. The parent is
explicit, not ambient-inherited, and the ScopedSpanGuard scope is RAII-bound
to the synchronous apply, so it cannot leak onto a reused worker (D1 fix).
On the open-ledger rebuild path no tx.process context exists, so it stays a
root and the attribute provides the correlation.
- tx.preclaim / tx.transactor: set both attributes from their ledger view.
tx.preflight is stateless (no view) and is the documented exception.
- tx.process / tx.receive: set current_ledger_seq from the current open ledger
index at submit/receive time (no hash: not yet applied to a ledger).
- Contract test pins the two new attribute key strings.
Neither key is a spanmetrics dimension, so there is no metric-cardinality
impact. Dashboards/collector/docs land on the later phases per the chain split.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
M1: ~PendingTraceId now increments a process-wide atomic counter when a
pinned deterministic trace_id is destroyed unconsumed, so the silent drop
stays observable in release builds where the existing XRPL_ASSERT is a
no-op. Exposed via unconsumedDeterministicIdDrops() for a future metric.
The deterministic trace_id bytes, the GenerateTraceId() consume logic, and
the unconditional reset() are all unchanged.
M2: tighten the Doxygen on SpanGuard::childSpan(std::string_view) to state
it parents to the current ambient context of this store (meaningful only
when a scope/ScopedSpanGuard/ScopedActivation is active) and to point
callers at childSpan(name, ctx) for explicit cross-store parenting.
Doc-only; no behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Brings coroutine-aware context storage + tx/consensus worker-body activation.
Resolved: Telemetry.cpp keeps both meterProvider_ (phase-7) and contextStorage_
(coro-aware); doc-09 keeps phase-7 structure and applies the pathfind.request →
rpc.command.<name> correction to phase-7's own PathFind section.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Brings coroutine-aware context storage, scoped rpc.command, coro-store-swap
tests, and the scoped pathfind.request forward.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Brings coroutine-aware OTel context storage forward: CoroAwareContextStorage,
its install in Telemetry, ScopedSpanGuard same-store assertion, and the
non-owning ScopedActivation helper.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add SpanGuard::activate() returning a ScopedActivation RAII helper that
activates an already-owned span (from a thread-free SpanGuard) as the
current context WITHOUT taking ownership. The activation pushes the span
onto the current LocalValue context store on construction and pops it on
destruction; it never ends the span (its owning SpanGuard does).
This lets a job-handoff span be made ambient for the duration of a
synchronous, non-yielding worker body so log lines there carry the
span's trace_id. Non-copyable and non-movable, mirroring ScopedSpanGuard.
owner is captured after the Scope push via declaration-order member
initialization (scope declared before owner), matching the A3
capture-after-materialization invariant. A #else no-op stub keeps the
API zero-overhead when telemetry is compiled out.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Coroutine-aware storage lets a scope resume on another worker within the same
coroutine store; store-identity is the correct pop-safety invariant. Same-store
equals same-thread for synchronous code, so no safety is lost.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backs the OTel active-context stack with xrpl::LocalValue so the ambient
context follows a JobQueue::Coro across yield/resume. Not yet installed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Rename file-local thread_local globals to the .clang-tidy convention
(GlobalVariablePrefix "g" + CamelCase), keeping the Tls marker:
tlsPendingTraceId -> gTlsPendingTraceId, tlsPendingConsumed -> gTlsPendingConsumed.
- Add direct includes for opentelemetry trace_id.h / span_id.h (header uses
TraceId/SpanId in signatures) and sdk/trace/id_generator.h (.cpp references
IdGenerator directly) to satisfy misc-include-cleaner.
Both verified clean with clang-tidy against a telemetry-enabled compile DB.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The OTel naming check (Rule F) scans @code doc-comment examples and fails
on string-literal span names; Rule H warns on undefined SpanNames
constants. Replace the literal "subtask" and the undefined
rpc_span::op::dispatch with the defined rpc_span::op::process constant so
the examples model correct API usage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
