Commit Graph

17370 Commits

Author SHA1 Message Date
Pratik Mankawde
2bfd1af6a0 fix(telemetry): accept.apply has one lawful parent after the fallback arm went
The accept.apply factory no longer falls back to the round context, so the
harness contract lists consensus.accept alone. Note rewritten to describe the
scoped child as it is now.
2026-09-23 13:58:45 +01:00
Pratik Mankawde
b205bb3449 Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-23 13:57:48 +01:00
Pratik Mankawde
769325d162 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-09-23 13:57:48 +01:00
Pratik Mankawde
3328bd2323 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-23 13:57:48 +01:00
Pratik Mankawde
360cde04ef Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-09-23 13:57:48 +01:00
Pratik Mankawde
3a9eae3998 Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd 2026-09-23 13:57:47 +01:00
Pratik Mankawde
f7f8aa79c3 Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-09-23 13:57:47 +01:00
Pratik Mankawde
d93749f752 feat(telemetry): assert span parents and consensus-round shape in the harness
The contract's parent field was documentation only, so no check could fail a
span for being parented when it should be a root. A live sweep found three
spans nested one level off while every harness check passed. The field becomes
allowed_parents, a list derived per span from its creation factory and every
call path that reaches it, and is asserted per span.

A parent on another node and a parent absent from the trace are both
inconclusive rather than violations: the receive spans are parented cross-node
by design, and a rotation in flight has not exported its root yet. Spans
reached by two call paths list every lawful parent.

A second gate reads a round as a unit: every required phase child under the
round span on the same node, their start times in protocol order, and no
mode_change recording mode_old == mode_new. It selects traces that already hold
the accept span, since that span always ends after the round span and can land
in a later export batch.

num_cleared leaves txq.batch_clear's required attributes: the code writes it
only after the batch actually clears.
2026-09-23 13:50:13 +01:00
Pratik Mankawde
6fadd0e2ee fix(telemetry): emit consensus.mode_change only on a real transition
MonitoredMode::set calls onModeChange on every round start, so the span was
created whether or not the mode moved. A node with a steady mode therefore
emitted one mode_change per round carrying mode_old == mode_new, which a live
sweep confirmed on every round of both instrumented builds. The round span's
own consensus_mode attribute is still written on every call, since that is
where the round learns the mode it is running in.
2026-09-23 13:49:54 +01:00
Pratik Mankawde
59bae37688 fix(telemetry): nest the accept work under consensus.accept.apply
accept.apply was a plain SpanGuard, so it never became the ambient span of
doAccept. The spans the function goes on to create inherited the activated
accept span instead and came out as accept.apply's siblings, while running
inside its own time window. Every guard was scoped before the SpanGuard split,
so this restores the hierarchy that design had.

Scoped now, so the hierarchy follows the call flow. Drops the parent-context
fallback arm with it: the accept context is captured only while the accept span
is live, and that span is a child of the round context, so an invalid accept
context implies an invalid round context and both arms returned an empty guard.
2026-09-23 13:49:42 +01:00
Pratik Mankawde
7d21baf558 feat(telemetry): add event-with-attributes overload to ScopedSpanGuard
Only SpanGuard carried addEvent(name, attrs), so a call site holding a scoped
guard could not record an event attribute. Forwarding overload, with the no-op
twin in the telemetry-disabled stub, so a span can be converted between scoped
and unscoped without dropping the attributes on its events.
2026-09-23 13:49:03 +01:00
Pratik Mankawde
407f5f1224 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-09-23 12:03:08 +01:00
Pratik Mankawde
e06fdd266e Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-23 12:03:08 +01:00
Pratik Mankawde
5e54b0f36e Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-23 12:03:08 +01:00
Pratik Mankawde
cf7889c008 test(telemetry): Parenthesise the daysUntil test arithmetic
clang-tidy's readability-math-missing-parentheses flags seven mixed
precedence expressions in this file, and warnings are errors in CI.

The parentheses match the precedence the compiler already applied, so every
static_assert value and the loop bound are unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-23 12:02:54 +01:00
Pratik Mankawde
964b0d64bf docs(telemetry): drop the example ticket id from the work-item filter
The $xrpl_work_item variable's description carried a real ticket id as its
example. The filter needs no example, so the id is gone and the wording stays.
2026-09-23 12:00:37 +01:00
Pratik Mankawde
bcc44dedf1 docs(telemetry): drop the example ticket id from the work-item filter
The $xrpl_work_item variable's description carried a real ticket id as its
example. The filter needs no example, so the id is gone and the wording stays.
2026-09-23 12:00:29 +01:00
Pratik Mankawde
0cd46de18d Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd 2026-09-22 21:30:34 +01:00
Pratik Mankawde
c9a97f9223 Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-09-22 21:30:34 +01:00
Pratik Mankawde
0b0534af5e Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-09-22 21:30:34 +01:00
Pratik Mankawde
d39053907e Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-22 21:30:34 +01:00
Pratik Mankawde
01fdd05225 Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-22 21:30:34 +01:00
Pratik Mankawde
aaee73ea1d Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-09-22 21:30:34 +01:00
Pratik Mankawde
8c90f7fed0 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-09-22 21:30:34 +01:00
Pratik Mankawde
c23646c279 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing 2026-09-22 21:30:34 +01:00
Pratik Mankawde
7272ea078c test(telemetry): Assert the cross-node span relationships
The harness checked parentage only within a node. Every declared
relationship named two spans from one process, so a receive span that
lost its cross-node parent still satisfied the contract: it is created
either way, with all its normal attributes, and only its parent differs.

Declare the three pairs that cross a node boundary, so a send side that
stops propagating context fails the run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:30:23 +01:00
Pratik Mankawde
a7b3a0df6e fix(tests): Locate the in-memory exporter by build config
The find_library hint was pinned to the _RELEASE variable CMakeDeps
generates, so in any other configuration it expanded to nothing. The
archive was then found only via CMAKE_PREFIX_PATH, which can hand a Debug
build the Release archive instead of failing. Derive the suffix from
CMAKE_BUILD_TYPE and ask for the package's lib directory directly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:30:22 +01:00
Pratik Mankawde
fcdc4f9f66 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-22 21:17:58 +01:00
Pratik Mankawde
aeac95b1d7 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-09-22 21:17:58 +01:00
Pratik Mankawde
c381d05bb8 Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-22 21:17:58 +01:00
Pratik Mankawde
e48b0286c2 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
StatsDCollector test kept both sides: this branch's onCollectionReady() call,
which enables polling and only exists from here on, followed by the upstream
branch's control assertion that reads the resulting datagram.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:17:48 +01:00
Pratik Mankawde
e9d9cffe3b Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd
Both doc indexes kept this branch's 09-data-collection-reference.md rows,
which only exist here, and dropped every secure-OTel.md reference because
the upstream branch removed that file. No dangling link remains.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:16:44 +01:00
Pratik Mankawde
a1d3ebbb92 Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-09-22 21:15:16 +01:00
Pratik Mankawde
aea4f56505 Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing
SpanGuardScope.cpp kept both includes: each side added one and both symbols
are used in the merged test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:15:05 +01:00
Pratik Mankawde
bba92767a6 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing
tempo.yaml kept both sides' filter blocks: phase-2's six path-finding
filters ahead of this branch's three transaction filters, matching chain
order, and both header comment lines.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:14:20 +01:00
Pratik Mankawde
d69bd5d19c Merge branch 'pratik/otel-phase1c-rpc-integration' into pratik/otel-phase2-rpc-tracing
RPCHandler.cpp composed both sides: phase-1c's null-guard and reply-aware
status logic, keeping this branch's load_type attribute inside that guard
because its argument allocates.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:13:17 +01:00
Pratik Mankawde
a9541a7300 fix(telemetry): Install the OTel context storage before any thread starts
SetRuntimeContextStorage() writes a process-global shared_ptr that every log
line reads through RuntimeContext::GetCurrent(). Neither side is atomic, and
assigning the wrapper destroys it and placement-news a replacement over the
same buffer, so a concurrent reader could make a virtual call through an
indeterminate vptr. Install it in main() while the process is still
single-threaded instead, and drop the member that held it.

Gated on the telemetry section being enabled. Only that one key is read
here, because parsing the whole section can throw on a contradictory TLS
combination and that error belongs where it already reports.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:11:58 +01:00
Pratik Mankawde
2d82bce5d4 fix(telemetry): Apply decided validation events oldest minute first
pending_ is a hash map, so one reconcile pass handed its entries over in no
useful order. The windows only move forward, so an event applied after a
later one landed in a bucket the grid had already rolled past and was
counted into the wrong window. Collect the decisions, sort by minute, then
apply.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:11:49 +01:00
Pratik Mankawde
daa82cea33 fix(telemetry): Inject the send span's own context into consensus messages
propose() and validate() both injected the ambient span context, but no
span is ever activated on the threads that reach them: the round span is
deliberately non-ambient and the validation span is parented through a
stored context. So no TraceContext was written, every receiving peer took
its standalone-span fallback, and the consensus receive spans arrived as
orphan trace roots.

Inject from the span in hand instead, which is the form the propagation
helper documents and the transaction relay path already uses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:11:38 +01:00
Pratik Mankawde
a293fb66e5 fix(telemetry): Report pathfind span status from the reply
Both pathfind handlers returned on many paths without recording a status,
so a failed request produced a span that read as success. Route every exit
through one helper that reads the rpc error token off the reply, which also
covers the replies built further down the call chain.

The token set is fixed by the error registry, so it is safe as a span
label; raw request text would not be.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:11:30 +01:00
Pratik Mankawde
35db8610cc fix(telemetry): Set rpc_status on the invalid-JSON websocket span
rpc_status is a span-metrics dimension, so leaving it unset on this path
emitted a series with a blank label. Any query selecting on error missed
the failure entirely.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:11:21 +01:00
Pratik Mankawde
87560c5157 test(telemetry): Restore the thread's LocalValue store from a scope guard
The store swap was undone by trailing statements, which a fatal assertion skips.
The thread store is a function-local static, so it outlives every test rather
than being reset between them: a skipped restore left it owning a stack object
from a dead frame, and the crash then landed in whichever test ran next.

The onCoro flag did not protect it either, because the cleanup function reads
that flag out of the freed object to decide not to delete it.

The guard is declared after both stack stores so it is destroyed before them.
2026-09-22 20:31:15 +01:00
Pratik Mankawde
3cfe8d139e feat(telemetry): Add Tempo search filters for the path-finding attributes
This branch emits nine pathfind attributes and the datasource offered a dropdown
for none of them, so the signal was there but not searchable in Explore. The
file's own header states that each phase adds filters for what it introduces.

Six filters, for the attributes that select a request. The three counts are
measurements read off a span rather than things an operator searches by, so they
get none. ledger_index is dynamic because it takes a new value every ledger.
2026-09-22 20:31:14 +01:00
Pratik Mankawde
68a805563f fix(telemetry): Reject a JSON reply or a weights argument that is not an object
Both workload scripts decoded JSON and went straight to .get() or .items().
An array or a scalar decodes fine and then raises AttributeError, which is not a
ValueError, so the handler around the weights parsing could not catch it and the
operator saw a traceback naming neither the command nor what arrived.

Four sites across the two files, all four guarded, and the weights handler now
also catches TypeError for a non-numeric weight value.

Negative weights are left alone: they do not raise, the type is dropped from the
mix silently, and changing that changes accepted input.
2026-09-22 20:30:53 +01:00
Pratik Mankawde
331a453c9d fix(telemetry): Read RPC success from the absence of an error, not from Ok
This board's own copy of the Success series selected status_code=STATUS_CODE_OK.
Successful rpc.command spans no longer carry Ok: the specification reserves it
for an operator asserting verified success and warns a tool may read it as
suppressing errors, so instrumentation leaves a successful span Unset.

The blob differs from phase-9's, so merging forward would not have corrected it.
2026-09-22 20:30:51 +01:00
Pratik Mankawde
cae3f3c477 test(telemetry): Assert what the consensus-attribute case was only executing
The case had no assertions at all, so it could not fail. Its comment also said
the attribute constants live in an xrpld-level header a libxrpl test cannot
include. ConsensusSpanNames.h is a libxrpl header, two sibling tests already
include it, and a comment fifty lines above says so correctly.

So the seven literal keys were never forced. They now come from the same
constants the emitter uses, and the case asserts the property it was written
for: the factory decides its verdict once at creation, and writing either
close-time outcome leaves the guard inert and publishing no propagation bytes.

The premise is asserted too, so a failure names which exit produced the null
guard the rest of the case rests on.
2026-09-22 20:30:15 +01:00
Pratik Mankawde
89fc3ba450 fix(telemetry): Name the reason an RPC failed in the span status
The status description was the fixed string error, so a trace recorded that a
request failed but not why.

It now carries the error token from the reply, falling back to the status's own
error code and then to the old string. Every source is a compile-time literal
from the error registry, so no request text reaches it. The rpc_status attribute
stays at success and error: that one is a span-metrics dimension, and widening
it would mint a series per token per command per node.

The work is gated on the span being live, so a build with telemetry compiled out
or a disabled guard pays nothing, matching how the neighbouring helper is
handled. asCString() is null-checked as well as type-checked, because it returns
the raw pointer where asString() guards it.

The comment above resolveCommandSpanName now states the invariant rather than
how it could be abused.
2026-09-22 20:29:58 +01:00
Pratik Mankawde
a2ee20b88f fix(telemetry): Read the RPC span status from the reply, not the Status
callMethod decided the rpc.command span's status from the Status the handler
returned. Handler.cpp registers 70 of its 72 methods through byRef(), which
returns a default Status whatever happened, because an old-style handler
reports its error in the reply body instead. So a failed account_info or a
refused path_find came out with rpc_status=success and span status Ok.

That is the opposite of what the comment above the code claimed it did, and it
left a {status.code=error} query blind to every non-throwing RPC error.

The status now comes from the reply as well as the Status, which covers both
handler styles. byRef() is unchanged and identical to develop: it behaves
correctly for its own purpose, and no RPC reply changes. Only telemetry was
reading the wrong signal.

setOk() is dropped rather than moved. The specification reserves Ok for an
operator asserting verified success and warns that a tool may treat it as
suppressing errors, so a successful call now leaves the status Unset.

No test accompanies this: the Beast tree has no telemetry fixture and the
in-memory span exporter is linked only into xrpl_tests, which cannot reach
daemon code. Asserting the exported status needs that infrastructure first.
2026-09-22 20:29:57 +01:00
Pratik Mankawde
5cecfc7d0b fix(insight): Read the StatsD polling gate under the lock it pairs with
onTimer read polling_ before taking metricsLock_. A tick that read the flag as
set could be preempted before acquiring the lock, letting onCollectionStopping()
clear the flag, take the uncontended lock and return. The tick then resumed and
ran every hook handler, which read the ledger master, the network operations,
the peer finder, the job queue and the overlay after shutdown had been told
polling stopped.

Collector::onCollectionStopping() promises polling has stopped by the time it
returns, and the shutdown ordering in ApplicationImp depends on that.

Reading the gate inside the lock makes seeing it set imply holding the lock, so
either the tick holds it across the handlers and the stop waits, or the stop
wins and the tick polls nothing. The old comment assumed that invariant rather
than establishing it.
2026-09-22 20:29:19 +01:00
Pratik Mankawde
e8bb4f4657 fix(telemetry): Catch a non-std exception from the metrics pipeline setup
The startup path caught std::exception only. initMetrics() reaches the OTel SDK,
which can throw something outside that hierarchy, and escaping a member
initializer would stop the node starting.

The section states the rule it must not break: a telemetry failure never stops
the node, the global provider stays a no-op and every instrument call remains
valid. The new clause drops the half-built provider and logs, exactly as the
std::exception clause does.

~ApplicationImp() already carries the same pairing for the same reason.
2026-09-22 20:29:17 +01:00