Commit Graph

17596 Commits

Author SHA1 Message Date
Pratik Mankawde
f23b66ca4d Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics
expected_spans.json: composed both sides. Kept this branch's 16 extra spans,
relationships and the per_ledger join group; took phase-10's allowed_parents
gate for the shared entries and derived allowed_parents for the 16 from their
factories. ledger.store, ledger.validate and ledger.acquire are hashSpan roots
on this branch, so they list ROOT alone rather than consensus.accept.apply.
2026-09-23 15:18:25 +01:00
Pratik Mankawde
fd29e92a0e Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-23 15:15:28 +01:00
Pratik Mankawde
9e635516da docs(telemetry): the accept log pair correlates to consensus.accept.apply
doAccept opens accept.apply as a scoped guard, so the CNF Val / CNF buildLCL
lines carry that span's id, not the accept span's. Same trace either way.
2026-09-23 15:15:01 +01:00
Pratik Mankawde
0764aa01aa docs(telemetry): correct the span-parenting claims the code has outgrown
tx.apply carries its own ledger_seq. On the consensus path ledger.build,
ledger.store, ledger.validate and the queue spans nest under
consensus.accept.apply, which doAccept opens as a scoped guard, so the passage
that said no ambient span exists there is replaced by the edge it now has. The
per-stage failure-rate comment no longer claims the stages leave status unset
on a failing result; all three set an error status.
2026-09-23 15:14:57 +01:00
Pratik Mankawde
628882f686 fix(telemetry): replace the alert notification body with the rule's own prose
Grafana's default notification body appends every expression node's value and
every label, so an alert arrived as `Value: A=0, B=0, C=1` over a five-line
`key = value` dump. The refIds mean nothing to a reader and the labels repeat
the title.

Add templates.yaml and point both Slack receivers and the email receiver at it,
so the body is the rule's own description plus its remediation line, and any
panel_* annotation renders as a link.

Rewrite all 13 descriptions to be status-neutral, since the same annotation is
rendered when the alert resolves: a firing-only wording made a resolved
notification claim the node had stopped closing ledgers while reporting a
healthy rate. The reason to care moves to a new `action` annotation, which the
template prints only while firing. Each description now formats its value with
printf and states its threshold, rather than emitting a bare float.

Also fix the Slack title, which referenced a `rulename` label that no rule sets
and so rendered blank; `alertname` is the label Grafana always provides.
2026-09-23 14:32:16 +01:00
Pratik Mankawde
2bfd1af6a0 fix(telemetry): accept.apply has one lawful parent after the fallback arm went
The accept.apply factory no longer falls back to the round context, so the
harness contract lists consensus.accept alone. Note rewritten to describe the
scoped child as it is now.
2026-09-23 13:58:45 +01:00
Pratik Mankawde
769325d162 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-09-23 13:57:48 +01:00
Pratik Mankawde
b205bb3449 Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-23 13:57:48 +01:00
Pratik Mankawde
3328bd2323 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-23 13:57:48 +01:00
Pratik Mankawde
360cde04ef Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-09-23 13:57:48 +01:00
Pratik Mankawde
3a9eae3998 Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd 2026-09-23 13:57:47 +01:00
Pratik Mankawde
f7f8aa79c3 Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-09-23 13:57:47 +01:00
Pratik Mankawde
d93749f752 feat(telemetry): assert span parents and consensus-round shape in the harness
The contract's parent field was documentation only, so no check could fail a
span for being parented when it should be a root. A live sweep found three
spans nested one level off while every harness check passed. The field becomes
allowed_parents, a list derived per span from its creation factory and every
call path that reaches it, and is asserted per span.

A parent on another node and a parent absent from the trace are both
inconclusive rather than violations: the receive spans are parented cross-node
by design, and a rotation in flight has not exported its root yet. Spans
reached by two call paths list every lawful parent.

A second gate reads a round as a unit: every required phase child under the
round span on the same node, their start times in protocol order, and no
mode_change recording mode_old == mode_new. It selects traces that already hold
the accept span, since that span always ends after the round span and can land
in a later export batch.

num_cleared leaves txq.batch_clear's required attributes: the code writes it
only after the batch actually clears.
2026-09-23 13:50:13 +01:00
Pratik Mankawde
6fadd0e2ee fix(telemetry): emit consensus.mode_change only on a real transition
MonitoredMode::set calls onModeChange on every round start, so the span was
created whether or not the mode moved. A node with a steady mode therefore
emitted one mode_change per round carrying mode_old == mode_new, which a live
sweep confirmed on every round of both instrumented builds. The round span's
own consensus_mode attribute is still written on every call, since that is
where the round learns the mode it is running in.
2026-09-23 13:49:54 +01:00
Pratik Mankawde
59bae37688 fix(telemetry): nest the accept work under consensus.accept.apply
accept.apply was a plain SpanGuard, so it never became the ambient span of
doAccept. The spans the function goes on to create inherited the activated
accept span instead and came out as accept.apply's siblings, while running
inside its own time window. Every guard was scoped before the SpanGuard split,
so this restores the hierarchy that design had.

Scoped now, so the hierarchy follows the call flow. Drops the parent-context
fallback arm with it: the accept context is captured only while the accept span
is live, and that span is a child of the round context, so an invalid accept
context implies an invalid round context and both arms returned an empty guard.
2026-09-23 13:49:42 +01:00
Pratik Mankawde
7d21baf558 feat(telemetry): add event-with-attributes overload to ScopedSpanGuard
Only SpanGuard carried addEvent(name, attrs), so a call site holding a scoped
guard could not record an event attribute. Forwarding overload, with the no-op
twin in the telemetry-disabled stub, so a span can be converted between scoped
and unscoped without dropping the attributes on its events.
2026-09-23 13:49:03 +01:00
Pratik Mankawde
992e600929 Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics 2026-09-23 12:03:08 +01:00
Pratik Mankawde
407f5f1224 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-09-23 12:03:08 +01:00
Pratik Mankawde
e06fdd266e Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-23 12:03:08 +01:00
Pratik Mankawde
5e54b0f36e Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-23 12:03:08 +01:00
Pratik Mankawde
cf7889c008 test(telemetry): Parenthesise the daysUntil test arithmetic
clang-tidy's readability-math-missing-parentheses flags seven mixed
precedence expressions in this file, and warnings are errors in CI.

The parentheses match the precedence the compiler already applied, so every
static_assert value and the loop bound are unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-23 12:02:54 +01:00
Pratik Mankawde
35da2952b0 docs(telemetry): drop the example ticket id from the work-item filter
The $xrpl_work_item variable's description carried a real ticket id as its
example. The filter needs no example, so the id is gone and the wording stays.
2026-09-23 12:00:51 +01:00
Pratik Mankawde
964b0d64bf docs(telemetry): drop the example ticket id from the work-item filter
The $xrpl_work_item variable's description carried a real ticket id as its
example. The filter needs no example, so the id is gone and the wording stays.
2026-09-23 12:00:37 +01:00
Pratik Mankawde
bcc44dedf1 docs(telemetry): drop the example ticket id from the work-item filter
The $xrpl_work_item variable's description carried a real ticket id as its
example. The filter needs no example, so the id is gone and the wording stays.
2026-09-23 12:00:29 +01:00
Pratik Mankawde
c8a4634cb4 Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics
expected_spans.json kept both sides' relationship entries: this branch's
eight nodestore.rotate phase children and the upstream branch's three
cross-node pairs. Both sides appended at the same list position.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:31:09 +01:00
Pratik Mankawde
0cd46de18d Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd 2026-09-22 21:30:34 +01:00
Pratik Mankawde
c9a97f9223 Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-09-22 21:30:34 +01:00
Pratik Mankawde
0b0534af5e Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-09-22 21:30:34 +01:00
Pratik Mankawde
d39053907e Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-22 21:30:34 +01:00
Pratik Mankawde
01fdd05225 Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-22 21:30:34 +01:00
Pratik Mankawde
aaee73ea1d Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-09-22 21:30:34 +01:00
Pratik Mankawde
8c90f7fed0 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-09-22 21:30:34 +01:00
Pratik Mankawde
c23646c279 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing 2026-09-22 21:30:34 +01:00
Pratik Mankawde
7272ea078c test(telemetry): Assert the cross-node span relationships
The harness checked parentage only within a node. Every declared
relationship named two spans from one process, so a receive span that
lost its cross-node parent still satisfied the contract: it is created
either way, with all its normal attributes, and only its parent differs.

Declare the three pairs that cross a node boundary, so a send side that
stops propagating context fails the run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:30:23 +01:00
Pratik Mankawde
a7b3a0df6e fix(tests): Locate the in-memory exporter by build config
The find_library hint was pinned to the _RELEASE variable CMakeDeps
generates, so in any other configuration it expanded to nothing. The
archive was then found only via CMAKE_PREFIX_PATH, which can hand a Debug
build the Release archive instead of failing. Derive the suffix from
CMAKE_BUILD_TYPE and ask for the package's lib directory directly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:30:22 +01:00
Pratik Mankawde
881faf5620 Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics
The runbook's nodestore_state table kept AppMetricGauges.cpp as the source
column, which is where those gauges are actually registered on every branch
in the chain; the incoming side named MetricsRegistry.cpp, which registers
none of them. Took the incoming side's clearer wording for the sweep row.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:19:08 +01:00
Pratik Mankawde
fcdc4f9f66 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-22 21:17:58 +01:00
Pratik Mankawde
aeac95b1d7 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-09-22 21:17:58 +01:00
Pratik Mankawde
c381d05bb8 Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-22 21:17:58 +01:00
Pratik Mankawde
e48b0286c2 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
StatsDCollector test kept both sides: this branch's onCollectionReady() call,
which enables polling and only exists from here on, followed by the upstream
branch's control assertion that reads the resulting datagram.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:17:48 +01:00
Pratik Mankawde
e9d9cffe3b Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd
Both doc indexes kept this branch's 09-data-collection-reference.md rows,
which only exist here, and dropped every secure-OTel.md reference because
the upstream branch removed that file. No dangling link remains.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:16:44 +01:00
Pratik Mankawde
a1d3ebbb92 Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-09-22 21:15:16 +01:00
Pratik Mankawde
aea4f56505 Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing
SpanGuardScope.cpp kept both includes: each side added one and both symbols
are used in the merged test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:15:05 +01:00
Pratik Mankawde
bba92767a6 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing
tempo.yaml kept both sides' filter blocks: phase-2's six path-finding
filters ahead of this branch's three transaction filters, matching chain
order, and both header comment lines.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:14:20 +01:00
Pratik Mankawde
d69bd5d19c Merge branch 'pratik/otel-phase1c-rpc-integration' into pratik/otel-phase2-rpc-tracing
RPCHandler.cpp composed both sides: phase-1c's null-guard and reply-aware
status logic, keeping this branch's load_type attribute inside that guard
because its argument allocates.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:13:17 +01:00
Pratik Mankawde
a9541a7300 fix(telemetry): Install the OTel context storage before any thread starts
SetRuntimeContextStorage() writes a process-global shared_ptr that every log
line reads through RuntimeContext::GetCurrent(). Neither side is atomic, and
assigning the wrapper destroys it and placement-news a replacement over the
same buffer, so a concurrent reader could make a virtual call through an
indeterminate vptr. Install it in main() while the process is still
single-threaded instead, and drop the member that held it.

Gated on the telemetry section being enabled. Only that one key is read
here, because parsing the whole section can throw on a contradictory TLS
combination and that error belongs where it already reports.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:11:58 +01:00
Pratik Mankawde
2d82bce5d4 fix(telemetry): Apply decided validation events oldest minute first
pending_ is a hash map, so one reconcile pass handed its entries over in no
useful order. The windows only move forward, so an event applied after a
later one landed in a bucket the grid had already rolled past and was
counted into the wrong window. Collect the decisions, sort by minute, then
apply.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:11:49 +01:00
Pratik Mankawde
daa82cea33 fix(telemetry): Inject the send span's own context into consensus messages
propose() and validate() both injected the ambient span context, but no
span is ever activated on the threads that reach them: the round span is
deliberately non-ambient and the validation span is parented through a
stored context. So no TraceContext was written, every receiving peer took
its standalone-span fallback, and the consensus receive spans arrived as
orphan trace roots.

Inject from the span in hand instead, which is the form the propagation
helper documents and the transaction relay path already uses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:11:38 +01:00
Pratik Mankawde
a293fb66e5 fix(telemetry): Report pathfind span status from the reply
Both pathfind handlers returned on many paths without recording a status,
so a failed request produced a span that read as success. Route every exit
through one helper that reads the rpc error token off the reply, which also
covers the replies built further down the call chain.

The token set is fixed by the error registry, so it is safe as a span
label; raw request text would not be.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:11:30 +01:00
Pratik Mankawde
35db8610cc fix(telemetry): Set rpc_status on the invalid-JSON websocket span
rpc_status is a span-metrics dimension, so leaving it unset on this path
emitted a series with a blank label. Any query selecting on error missed
the failure entirely.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:11:21 +01:00