Commit Graph

17386 Commits

Author SHA1 Message Date
Pratik Mankawde
d0e4d13136 Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics 2026-09-18 10:03:14 +01:00
Pratik Mankawde
f07beac0bb Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-18 10:03:14 +01:00
Pratik Mankawde
5945dd5f28 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-09-18 10:03:14 +01:00
Pratik Mankawde
95cc50d43f Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-18 10:03:14 +01:00
Pratik Mankawde
4e2b8ce536 merge: bring the plan-doc suppressed cleanup forward from phase-6
The attribute table conflicted because this branch had added the fee and
sequence rows and widened tx_type's set-on list. Kept those, and dropped the
suppressed row the incoming side removed.
2026-09-18 10:02:54 +01:00
Pratik Mankawde
2844277759 Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-18 10:00:52 +01:00
Pratik Mankawde
6e4d566ba6 Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd 2026-09-18 10:00:51 +01:00
Pratik Mankawde
84ff06ecec Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-09-18 10:00:51 +01:00
Pratik Mankawde
b2b5879e8e Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-09-18 10:00:51 +01:00
Pratik Mankawde
f1df5de8dd feat(telemetry): count relayed transactions this node did not process
Moving the tx.receive span past the duplicate check removed the only record of
why a relayed copy was dropped. A labelled counter restores it, and covers the
inner-batch reject the span never usefully reached.

The count itself was never span-derived: transactions_duplicate already totals
the dropped copies and predates the telemetry work. What was missing is the
split between ordinary relay overlap and a peer sending traffic it should not.

A counter also survives sampling and runs with tracing off, which the span
attribute did not.
2026-09-18 10:00:31 +01:00
Pratik Mankawde
536c5a09c4 docs(telemetry): correct the tx.receive note for where the span now starts
The note said the suppressed attribute is set on both outcomes. That attribute
is gone, and a duplicate produces no span, so the note now says where the
duplicate count lives instead.
2026-09-18 09:57:33 +01:00
Pratik Mankawde
ff44c3c851 docs(telemetry): drop suppressed from the attribute and label references
tx.receive no longer sets it, and it is no longer a spanmetrics dimension, so
the attribute tables, the span-scope list and the Prometheus label list all
named something that is not emitted. The reduce-relay suppressed_peers gauge is
a different thing and stays.
2026-09-18 09:56:57 +01:00
Pratik Mankawde
073ff1a93f docs(telemetry): drop the suppressed row from the attribute reference
tx.receive does not set it. The span starts after the duplicate check, so a
dropped copy produces no span for the attribute to sit on.
2026-09-18 09:55:45 +01:00
Pratik Mankawde
a0a23f0b61 docs(telemetry): match the task list to where the receive span now starts
The steps described setting a suppressed attribute on a dropped duplicate. The
span is now created after the duplicate check, so there is no span on that path.
2026-09-18 09:55:14 +01:00
Pratik Mankawde
d029ed9b59 Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics 2026-09-18 09:34:50 +01:00
Pratik Mankawde
60f348e28d Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-18 09:34:50 +01:00
Pratik Mankawde
41140ec6d1 merge: bring the runbook attribute-row fix forward from phase-8
The table conflicted because both sides edited it: this branch had dropped
ledger_seq from the tx.apply row, and the incoming side updated the tx.receive
row. Kept both.
2026-09-18 09:34:19 +01:00
Pratik Mankawde
673e12e7b3 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-18 09:33:24 +01:00
Pratik Mankawde
795c9ad39c Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-09-18 09:33:24 +01:00
Pratik Mankawde
92865edcea Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics 2026-09-18 09:33:24 +01:00
Pratik Mankawde
4ca5a21430 Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-18 09:33:24 +01:00
Pratik Mankawde
b541092016 test(telemetry): drop suppressed from the mirrored dimension list
The list mirrors the collector's spanmetrics dimensions, and that dimension is
gone. The reduce-relay and squelch counters keep their own names, which are a
different thing.
2026-09-18 09:33:04 +01:00
Pratik Mankawde
3b60a190de fix(telemetry): drop the remaining suppressed dimensions and diagram values
Alloy and the Grafana Cloud collector still declared a spanmetrics dimension
for an attribute tx.receive no longer sets, which only widens the label set.
The runbook's drop-path diagram still listed two tx_status values that no
longer exist, because the paths that set them now run before the span starts.
2026-09-18 09:32:29 +01:00
Pratik Mankawde
675dffdc71 docs(telemetry): drop suppressed from the tx.receive attribute list
The span is created after the duplicate check, so it never carries the
attribute, and the row now says which transactions it covers.
2026-09-18 09:31:23 +01:00
Pratik Mankawde
fa85bdb982 Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics 2026-09-18 09:29:55 +01:00
Pratik Mankawde
43d1fba458 Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-18 09:29:54 +01:00
Pratik Mankawde
8ae1d4c6a5 merge: bring the suppressed-reference removal forward from phase-8
transaction-overview.json conflicted only because this branch reordered every
panel, so the diff could not line the two sides up. The incoming side changed
one panel and nothing else, so this keeps every panel and rewrite this branch
made and re-applies that single change in this branch's own style: the receive
panel drops the suppressed grouping, names its series like the other
single-series panels, and says where the duplicate count now lives.
2026-09-18 09:29:35 +01:00
Pratik Mankawde
f666427570 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-18 09:26:33 +01:00
Pratik Mankawde
b972682955 merge: bring the suppressed-reference removal forward from phase-6
Resolved transaction-overview.json by composing both sides: this branch's
renamed span_calls_total metric and its 15s interval, with phase-6's removal of
the suppressed grouping and its new legend.
2026-09-18 09:26:17 +01:00
Pratik Mankawde
e87432fe3c fix(telemetry): drop the suppressed references the span no longer emits
tx.receive is created after the duplicate check, so it never carries a
suppressed attribute. The Tempo tag filter and the dashboard panel still
referenced it, and check_otel_naming rules C and D fail on a reference with no
constant behind it.

The panel keeps its remaining signal as a plain receive rate. How many relayed
copies were dropped is the transactions_duplicate traffic category, which does
not depend on a span.

The spanmetrics dimension goes too; a dimension for an attribute nothing sets
only widens the label set.
2026-09-18 09:24:56 +01:00
Pratik Mankawde
39c14e71f7 Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics 2026-09-17 23:03:42 +01:00
Pratik Mankawde
d5c2964fea test(telemetry): drop the suppressed requirement from tx.receive
tx.receive no longer carries a suppressed attribute: the span is created only
once the node has decided to process the transaction, so there is no dropped
copy for the attribute to describe.

The validator fails a span that is missing a required attribute, so leaving it
listed turns the telemetry-validation leg red.
2026-09-17 23:03:28 +01:00
Pratik Mankawde
94e9a79021 Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics 2026-09-17 23:02:12 +01:00
Pratik Mankawde
73af31c27f Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-09-17 23:01:58 +01:00
Pratik Mankawde
caa3eb5984 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-17 23:01:58 +01:00
Pratik Mankawde
04786c4c4d Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-17 23:01:58 +01:00
Pratik Mankawde
6024954459 Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-09-17 23:01:47 +01:00
Pratik Mankawde
67c9303c0c Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-09-17 23:01:47 +01:00
Pratik Mankawde
8c10cd710d Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd 2026-09-17 23:01:47 +01:00
Pratik Mankawde
c0c9478093 Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-09-17 23:01:35 +01:00
Pratik Mankawde
fe4a778fed perf(telemetry): trace only the transactions this node will process
A peer relays every transaction it hears, so most inbound copies are ones
handleTransaction() drops. The tx.receive span was created before those checks
ran, so every dropped copy paid for a span, a hex transaction id, and an
open-ledger index read that takes the mutex the apply path needs.

Measured over a four-hour payment run: 83.2M of 101.1M tx.receive spans
described a copy the node dropped. That is 82% of this span and 23% of every
span the node emitted.

Move the span and its attributes below the duplicate check. How many copies
were dropped is already reported as the transactions_duplicate traffic
category, which costs no span. Why a copy was dropped is no longer recorded;
a labelled counter restores it on the branch that carries the metric registry.

The suppressed attribute and the suppressed and rejected_inner_batch status
values go with it. This function was their only user.
2026-09-17 23:00:59 +01:00
Pratik Mankawde
37bee9a61c Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics 2026-09-17 17:02:47 +01:00
Pratik Mankawde
e0520b7e46 Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-17 17:02:27 +01:00
Pratik Mankawde
9abbdc0a2f merge: bring the single telemetry switch forward from phase-8
Both sides had already dropped option(telemetry) and describe the same behaviour;
the conflicts were wording. Kept this branch's text, which spells out that
opentelemetry-cpp is not needed at all when telemetry is off and keeps the note
about -DXRPL_ENABLE_TELEMETRY.
2026-09-17 17:02:16 +01:00
Pratik Mankawde
85fc1c5399 fix(telemetry): keep the workload cluster on [ips], not [ips_fixed]
The previous commit switched the generated node config from [ips] to
[ips_fixed] on the grounds that the variable, the comment and the sibling cfg
template all named ips_fixed, and that ips_fixed is the section whose
documented meaning fits a private cluster. Both of those are still true. The
switch is reverted anyway, because it is a workload change rather than a
naming fix.

Measured on CI, parent commit against this branch's previous tip, one
functional config line apart:

  span.consensus.ledger_close.p95    0.57 ms -> 6.43 ms   (tripped the gate)
  span.consensus.ledger_close.p99    0.94 ms -> 9.50 ms
  span.consensus.accept.p50          0.97 ms -> 2.63 ms
  span.tx.process.p50                0.36 ms -> 0.18 ms   (faster)
  job.acceptLedger.running.p95      21157 us -> 10938 us  (faster)

Every consensus-path span rose and every transaction-path metric fell, which
is the shape a denser always-connected mesh produces and not the shape of
run-to-run variance. [ips_fixed] holds connections open to all four peers
instead of treating the list as a discovery hint, so each node processes
proposals and validations from the full mesh every round. Nothing else in that
commit touches the consensus path: the emitted config differed in exactly
three lines, of which one is a die message and one expands to an identical
string.

The committed baseline describes the [ips] topology. Adopting [ips_fixed]
therefore needs a refreshed baseline and re-derived bounds, which is the
process baselines/README.md already documents for a workload change. Left as
its own work item rather than smuggled in behind a section rename, and the
reason is now recorded beside the line so it is not repeated.

This also falsified a claim the previous commit had written into
baselines/README.md and regression-thresholds.json: that none of the six
weakly-guarded keys fires on any observed run. Corrected in both, and the
measurement above is cited in place of the absolute.
2026-09-17 17:01:29 +01:00
Pratik Mankawde
d0baef8c92 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-17 17:01:21 +01:00
Pratik Mankawde
c1086e63a2 Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-09-17 17:01:20 +01:00
Pratik Mankawde
6220c211fe Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-09-17 17:01:20 +01:00
Pratik Mankawde
aea4431807 Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd 2026-09-17 17:01:20 +01:00
Pratik Mankawde
de3e376944 Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-09-17 17:01:20 +01:00