Commit Graph

796 Commits

Author SHA1 Message Date
Pratik Mankawde
521f00a484 merge: bring the lock-free ValidationTracker forward from phase10-workload-validation
Only the workload README conflicted; the tracker, config and test files merged
clean, which closes the chain from phase-7.
2026-09-09 17:10:15 +01:00
Pratik Mankawde
7f829a5929 fix(telemetry): fail the regression gate on a unit change, and report what it gated
compare_to_baseline took the unit from the baseline entry and dropped the current
run's, and nothing compared the two, so a us -> ms change was scored as a numeric
delta: four keys rewritten to the same physical durations reported 99.9%
improvements and the gate exited 0. prom_queries.py says the baseline preserves
the unit "so the comparator can sanity-check unit drift"; it never did. A unit
mismatch now fails and names both units.

The workflow's step summary printed total, regressions and improvements. total is
every key in the report -- the union of baseline and current -- so it was neither
the baseline count nor what was gated, and missing_in_current was computed and
never printed. A run that gated 16 of 20 keys read as a full comparison. The
comparator now reports a real "compared" count and the summary prints it beside
the not-captured count, with a warning when any key was missed. The table also
refused nothing on a truncated report; existence is not readability.

check_regression_bounds told the operator to add max_abs_increase while reading
max_abs_increase_ms / _us, so following the message added a key nothing reads and
the gate kept failing with no explanation. The committed thresholds use only the
suffixed spelling, so the message was the defect. Its three JSON inputs were also
unchecked: a top-level null, list or number parsed and then died on the first
.get, and a string "metrics" survived the placeholder test and reported its own
characters as gated keys -- wrong advice rather than a crash.

Four tests cover these; all four fail against the previous checker.
2026-09-09 13:15:46 +01:00
Pratik Mankawde
92b529cfe7 merge: bring the CI fixes forward from phase10-workload-validation 2026-09-08 17:22:30 +01:00
Pratik Mankawde
738b5905fd merge: bring the CI fixes forward from phase9-metric-gap-fill 2026-09-08 17:22:19 +01:00
Pratik Mankawde
bae31955db merge: bring the CI fixes forward from phase8-log-correlation
One conflicted file, docs/telemetry-runbook.md, with three spots:

- Build section: both sides added different text at one point. Kept both,
  incoming sentence first, then this branch's "Run against a live network".
- Disabling section, first spot: this branch's wording names the config
  section and says no rebuild is needed, so it already covers the incoming
  sentence.
- Disabling section, second spot: kept this branch's paragraph and folded in
  the one point it lacked, that both flags have to be passed.
2026-09-08 17:22:04 +01:00
Pratik Mankawde
ff1a905ede merge: bring the CI fixes forward from phase6-statsd
Three conflicts, all resolved by keeping this branch's rewrite and
re-applying the incoming change onto it:

- 09-data-collection-reference.md: phase-7 rewrote both attribute tables,
  so the incoming table would have reverted them. Kept phase-7's and
  re-applied the two "XRPL epoch" spellings.
- integration-test.sh: phase-7 moved these checks from StatsD to OTel and
  no longer defines check_statsd_metric, so only this side compiles.
- TelemetryConfig.cpp: the incoming side carried networkTypeFromId(), which
  this branch already has. Kept one definition and took the incoming
  doc wording, which the auto-merged body below it already matches.
2026-09-08 17:17:50 +01:00
Pratik Mankawde
dc2a8f121e Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-09-08 16:03:00 +01:00
Pratik Mankawde
0fd7ce9f8b Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing 2026-09-08 16:02:54 +01:00
Pratik Mankawde
96c11264b0 Merge branch 'pratik/otel-phase1c-rpc-integration' into pratik/otel-phase2-rpc-tracing 2026-09-08 16:02:50 +01:00
Pratik Mankawde
31ace1a357 Merge branch 'pratik/otel-phase1b-telemetry-infra' into pratik/otel-phase1c-rpc-integration 2026-09-08 16:02:42 +01:00
Pratik Mankawde
9e72c1a8bf Merge branch 'pratik/otel-phase1a-plan-docs' into pratik/otel-phase1b-telemetry-infra 2026-09-08 16:02:36 +01:00
dependabot[bot]
3e4bdf2782 ci: [DEPENDABOT] bump actions/deploy-pages from 5.0.0 to 5.0.1 in the github-actions group across 1 directory (#8180)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-07 16:07:56 +00:00
Pratik Mankawde
3c56697930 merge: bring the review fixes forward from otel-phase10-workload-validation 2026-09-07 15:47:20 +01:00
Pratik Mankawde
5a567841f1 merge: bring the review fixes forward from otel-phase9-metric-gap-fill 2026-09-07 15:41:03 +01:00
Pratik Mankawde
efdd9758f2 merge: bring the review fixes forward from otel-phase8-log-correlation 2026-09-07 15:39:59 +01:00
Pratik Mankawde
4ad84cae14 merge: bring the review fixes forward from otel-phase6-statsd 2026-09-07 15:23:22 +01:00
Pratik Mankawde
3d9ea4b9da merge: bring the review fixes forward from phase3-tx-tracing
One conflict, in 06-implementation-phases.md: this branch had rewritten the
phase-4 task table with a Status column, a descoping note and a Spans Produced
section, while upstream corrected the class name in the old plain table. This
branch's section is kept and the name correction re-applied to its 4.1 row.
2026-09-07 15:04:39 +01:00
Pratik Mankawde
08df97f431 merge: bring the review fixes forward from phase2-rpc-tracing
One conflict, in SpanGuard.h: this branch added struct TraceBytes and upstream
added enum SpanRole at the same position after TraceCategory. Unrelated
declarations, so both are kept.
2026-09-07 15:03:44 +01:00
Pratik Mankawde
caa704de10 merge: bring the review fixes forward from phase1c-rpc-integration
Two conflicts, both resolved by composing the sides rather than taking one.

cfg/xrpld-example.cfg: this branch had moved the batch-processor keys under
their own heading while upstream edited them in place, so a merge-both would
have documented them twice. Upstream's range sentences are applied to the
relocated block and the head-sampling note keeps its position.

02-design-decisions.md: the summary table changed on both sides for different
reasons. Upstream renamed ledger_index to current_ledger_seq and ledger_seq;
this branch had corrected the PathFinding row to the keys it actually emits.
Both are kept.
2026-09-07 15:00:53 +01:00
Pratik Mankawde
623e3d8ad3 feat(telemetry): add batch-size and discovered-path histograms
Review feedback asked for a Histogram rather than a span attribute at these two
places. Both, not either: the attribute answers how big one sampled request was,
which an aggregate cannot, and the histogram answers the distribution across all
requests, which an unsampled trace never reveals. Both attributes stay.

The metrics land here rather than with the attributes because neither
HistogramBuckets.h nor the metric macro exists on the branch that added them.

Both use kObjectCountBuckets. The argument is the floor, not the ceiling: the SDK
default edges start 0,5,10,25, so an ordinary batch of one to five falls in a
single bucket and every quantile becomes an interpolation on one edge. The object
ladder puts five edges over the mass of both distributions. Path count is bounded
at 352 by kMaxPaths times kMaxAutoSrcCur and cannot saturate. Batch size can, at
roughly 333k, but no measured traffic goes near it, so the ladder is not widened
for a range nothing occupies; the runbook records the overflow query and a test
asserts it stays readable.
2026-09-07 13:51:44 +01:00
Pratik Mankawde
f8e0a19b9f fix(telemetry): correct the childSpan doc examples and make Rule D tests real
Review feedback on the RPC integration PR.

The childSpan examples could not work as written. childSpan() takes its parent
from the ambient context and uses impl_ only as a liveness gate, so an unscoped
SpanGuard parent produced two siblings rather than a parent and child. The parent
is now a ScopedSpanGuard, the child no longer reuses the parent's name, and the
examples pass a full dotted constant because childSpan() takes the name verbatim.

Five of the ten Rule D tests could not fail. Four passed an empty L1 key set,
which makes the rule skip validation altogether; the fifth asserted an empty
result against an escaped-quote selector that extracted no labels at all. Each
now passes a nonempty L1 set and carries a known-bad label in the same
expression, so it asserts both that the intended labels are accepted and that
Rule D ran. Verified by disabling the rule: the old tests stay green, the new
ones all fail.

Span kind is not fixed here. categoryToSpanKind and the span factories belong to
the telemetry library, so the role parameter is routed to that branch, and the
two call sites here follow once it exists.
2026-09-07 13:25:22 +01:00
Ayaz Salikhov
3e54e7d00b fix: Don't use github.workspace as it slow downs gcovr (#8173) 2026-09-03 19:20:38 +00:00
Ayaz Salikhov
986065c16f build: Verify glibc version was determined in debian package (#8170) 2026-09-03 16:54:54 +00:00
Pratik Mankawde
5f3340520c merge: bring develop forward from phase10-workload-validation
NetworkOPs.cpp include block: kept develop's rpc/detail/SyntheticFields.h
alongside this branch's MetricMacros.h and the guarded MetricNames.h.
2026-09-03 17:40:39 +01:00
Pratik Mankawde
79d2be2352 merge: bring develop forward from phase9-metric-gap-fill 2026-09-03 17:40:07 +01:00
Pratik Mankawde
54d6e80e2e merge: bring develop forward from phase8-log-correlation
Two conflicts, both additions at the same spot:
NetworkOPs.cpp include block kept develop's rpc/detail/SyntheticFields.h
alongside this branch's telemetry/MetricsRegistry.h; .cspell.config.yaml
kept both new words.
2026-09-03 17:39:21 +01:00
Pratik Mankawde
5e33ee9b58 merge: bring develop forward from phase6-statsd 2026-09-03 17:34:06 +01:00
Pratik Mankawde
ec2fcdf4f2 merge: bring develop forward from phase3-tx-tracing 2026-09-03 17:33:37 +01:00
Pratik Mankawde
406f9f486c merge: bring develop forward from phase2-rpc-tracing
NetworkOPs.cpp include block: develop replaced DeliveredAmount.h,
MPTokenIssuanceID.h and NFTSyntheticSerializer.h with the single
rpc/detail/SyntheticFields.h. Kept that and this branch's two telemetry
includes.
2026-09-03 17:33:20 +01:00
Pratik Mankawde
d26e096a80 merge: bring develop forward from phase1c-rpc-integration 2026-09-03 17:32:17 +01:00
Pratik Mankawde
5eb8fa9f96 merge: bring develop forward from phase1b-telemetry-infra 2026-09-03 17:31:23 +01:00
Pratik Mankawde
46d13d928b Merge branch 'pratik/otel-phase1a-plan-docs' into pratik/otel-phase1b-telemetry-infra 2026-09-03 17:24:07 +01:00
Pratik Mankawde
06c5d36f03 Merge branch 'pratik/otel-phase1a-plan-docs' into pratik/otel-phase1b-telemetry-infra 2026-09-03 16:50:41 +01:00
Pratik Mankawde
3e4b5c71ff merge: bring the traces_endpoint rename forward from phase-10
Three conflicts, all between this branch's own sync-diagnostics work and
phase-10's older versions. Resolved to this branch in each case, since it
owns the newer content:

- InboundLedger.h keeps the missing-node and receive-depth gauges and the
  fuller acquire-span contract.
- MetricsRegistry.cpp keeps the namespaced label:: constants.
- LedgerMaster.cpp keeps makeLedgerTraceSpan(), which joins the store and
  validate spans into one per-ledger trace by hash.

LedgerMaster.cpp needed a second pass. The automatic merge had kept both
sides outside the conflict markers, nesting phase-10's older promotion
block inside this branch's `if (!pubLedger_)` — so setValidated,
setFull and setValidLedger would each have run twice. Taking this
branch's file wholesale removes the duplicate; brace balance and a single
"Advancing accepted ledger" confirm it.

That resolution drops two things phase-10 was carrying into this file:
the storeSpan/validateSpan guard names, and the explicit scope that keeps
the one-in-256 flag-ledger check outside the ledger.validate measurement.
Both are re-applied on this branch in the next commit; the scope needs a
variable-lifetime check that does not belong in a merge.
2026-09-03 16:12:07 +01:00
Pratik Mankawde
3e930d9d37 merge: bring the traces_endpoint rename forward from phase-9 2026-09-03 16:07:14 +01:00
Bart
2ddf6ee148 ci: Update pre-commit SHA (#8169)
Co-authored-by: Bart <11445373+bthomee@users.noreply.github.com>
2026-09-03 14:52:42 +00:00
Pratik Mankawde
8ce4f71427 docs: Use consistent heading levels in PR template (#8155) 2026-09-03 14:48:32 +00:00
Bart
6e1eb88e6e ci: Update prepare-runner SHA (#8168)
Co-authored-by: Bart <11445373+bthomee@users.noreply.github.com>
2026-09-03 11:52:08 +00:00
Ayaz Salikhov
49cdc105de fix: Correct and simplify Linux packaging (#8165) 2026-09-02 21:18:13 +00:00
Pratik Mankawde
3a63a17548 docs(telemetry): stop the harness contract narrating its own revisions
Notes across the workload contract described earlier versions of themselves, or
cited commits that only exist inside this chain. A squash merge publishes none of
it, so each reference resolves nowhere.

Notes that described their own earlier text:

- expected_spans.json: 'this note previously concluded', 'this note previously
  said', 'Un-skipped 2026-08-26', 'the reason had simply gone stale for two
  weeks' and 'the claim this entry carried' are replaced by the standing reason
  each entry holds. The wildcard pairs now say the validator globs the child via
  _span_name_matches(), and state the literal-collapse failure as what a
  different validator WOULD do rather than as history.
- regression-thresholds.json: 'an earlier version of this note wrongly claimed',
  'the earlier version oversold it' and 'an earlier note called that' become the
  cautions themselves -- do not reason from 'every ladder step is at least 2x',
  do not oversell the backstop, do not read a false fire as a missing override.
- test_check_regression_bounds.py: the docstring gives the reason a literal is
  wrong here, not the story of two tests that once hard-coded one.

Baseline-refresh history rewritten as measurement:

- README.md, baselines/README.md, telemetry-runbook.md and regression-metrics.json
  no longer attribute threshold moves to 'the 2026-08-26 refresh'. The evidence
  is kept as measurement -- span.tx.apply.p50 has read 0.7917 ms and 0.00597 ms
  on the same workload, 132x apart; job.acceptLedger.running.p95 has measured a
  5.74x floor on one baseline and 16.28x on another -- which is what supports the
  claim that a single-run baseline cannot bound these keys.

Two chain-only commit ids removed, d059f21bf3 and 3860c93db2. Neither is
reachable from develop, so both cease to exist on merge; the second is chain
bookkeeping. The facts they were cited for (the validator globs wildcards; the
span ladder's floor is 0.01 ms) are stated directly instead.

Capture provenance is deliberately kept: baseline-timings.json 'captured_at',
the 2026-08-26 baseline heading, and the 2026-08-24 figures cited as data.

Documentation, JSON note strings and one docstring only, no behaviour change.
2026-09-02 20:32:13 +01:00
Pratik Mankawde
9b1cd87d2e docs(telemetry): remove pre-squash references from the gap-fill comments
These comments dated themselves against this branch or against an earlier
revision of the same change, neither of which survives a squash merge.

- 'as of this branch' in the pricing-case doc and the pass-through static_assert
  becomes a statement about what the constants currently produce.
- 'Extracted from processGetObjectByHash()' and 'Split from start()' describe
  edits internal to this change; both now say why the method stands alone.
- Recording.h: the mock-abstract mismatch is a standing consequence of a member
  set that differs between builds, not something that 'has previously' happened.
- MetricsRegistry.cpp: describe the loops.txt entry as recording two cycles
  rather than as what ordering.txt 'previously had'.
- InboundLedger.h: the acquire span is the only signal for back-fill cost; it
  did not 'previously emit' nothing.
- check_bucket_parity.py: replace the eleven-phase drift story with the reason
  the check exists, and point the failure message at HistogramBuckets.h and the
  collector config instead of OpenTelemetryPlan/, which does not reach develop.

Comments and one error message only, no behaviour change.
2026-09-02 19:47:35 +01:00
Pratik Mankawde
e32a2bda75 Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-08-27 16:05:19 +01:00
Pratik Mankawde
6f0fd251ec Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics 2026-08-27 16:05:19 +01:00
Pratik Mankawde
279ccd84d7 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-08-27 16:04:58 +01:00
Pratik Mankawde
099109357f Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-08-27 16:04:36 +01:00
Pratik Mankawde
b060c76a76 Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-08-27 16:01:01 +01:00
Pratik Mankawde
cc24101629 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing 2026-08-27 16:01:01 +01:00
Pratik Mankawde
5dfee8564b Merge branch 'pratik/otel-phase1b-telemetry-infra' into pratik/otel-phase1c-rpc-integration 2026-08-27 16:00:26 +01:00
Pratik Mankawde
189755bbb2 Merge branch 'pratik/otel-phase1c-rpc-integration' into pratik/otel-phase2-rpc-tracing 2026-08-27 16:00:26 +01:00
Pratik Mankawde
fff4124d9b Merge branch 'pratik/otel-phase1a-plan-docs' into pratik/otel-phase1b-telemetry-infra 2026-08-27 16:00:25 +01:00