Commit Graph

15859 Commits

Author SHA1 Message Date
Pratik Mankawde
addd7f8d39 Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-07-22 18:48:25 +01:00
Pratik Mankawde
b68f7dc1f4 fix(telemetry): guard MetricMacros.h against WinSock1/Boost.Asio clash
On Windows the telemetry unit test src/tests/libxrpl/telemetry/MetricMacros.cpp
fails to compile with boost/asio/detail/socket_types.hpp C1189
"WinSock.h has already been included". Root cause traced with clang -E -H:
MetricMacros.h pulls MetricsRegistry.h -> OTel spin_lock_mutex.h, which
defines _WINSOCKAPI_ and includes <windows.h> (WinSock 1); it then pulls
ServiceRegistry.h -> <boost/asio.hpp>, whose socket_types.hpp requires
winsock2.h first. Production consumers (RCLConsensus.cpp, PerfLogImp.cpp)
happen to include Boost.Asio via other xrpld headers first, so only the test
TU -- which includes MetricMacros.h first -- hit it.

Pre-include boost/asio/detail/socket_types.hpp under _MSC_VER at the top of
the header so winsock2.h wins for every consumer. Mirrors the existing guard
in MetricsRegistry.cpp.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 18:48:13 +01:00
Pratik Mankawde
903a89daf8 Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-07-22 17:00:04 +01:00
Pratik Mankawde
3c86cea742 fix(telemetry): mark macro-only includes IWYU keep in MetricMacros.h
misc-include-cleaner does not expand macros, so it flags MetricsRegistry.h,
ServiceRegistry.h, <functional> and <mutex> as unused even though every
XRPL_METRIC_* macro body uses them ((app).getMetricsRegistry(),
std::once_flag/std::call_once, std::function). Add IWYU pragma: keep, the
same suppression already used across xrpld/ for macro/transitive includes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 16:59:49 +01:00
Pratik Mankawde
4263aad047 Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-07-22 16:20:57 +01:00
Pratik Mankawde
3dbeab845f Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-07-22 16:20:57 +01:00
Pratik Mankawde
608976d6ec Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-22 16:20:57 +01:00
Pratik Mankawde
5ce3061519 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-22 16:20:56 +01:00
Pratik Mankawde
4045e28ce2 Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd 2026-07-22 16:20:56 +01:00
Pratik Mankawde
e20f2c410c Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-07-22 16:20:56 +01:00
Pratik Mankawde
f9579b7356 fix(telemetry): drop redundant SpanNames.h include from Consensus.h
Consensus.h includes ConsensusSpanNames.h, which already includes
SpanNames.h (it is built on StaticStr/join()). Consensus.h uses no base
SpanNames.h symbols directly, so misc-include-cleaner flags the direct
include as unused. Remove it; SpanNames.h stays reachable transitively.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 16:20:44 +01:00
Pratik Mankawde
efdc2426a9 Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-07-22 16:20:30 +01:00
Pratik Mankawde
0de8aac1dc Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing 2026-07-22 16:20:29 +01:00
Pratik Mankawde
a0f8c87f56 Merge branch 'pratik/otel-phase1c-rpc-integration' into pratik/otel-phase2-rpc-tracing 2026-07-22 16:20:29 +01:00
Pratik Mankawde
7fec5d75d5 Merge branch 'pratik/otel-phase1b-telemetry-infra' into pratik/otel-phase1c-rpc-integration 2026-07-22 16:20:28 +01:00
Pratik Mankawde
d865a266b5 fix(telemetry): clang-tidy naming + include-cleaner in DeterministicIdGenerator
- Rename file-local thread_local globals to the .clang-tidy convention
  (GlobalVariablePrefix "g" + CamelCase), keeping the Tls marker:
  tlsPendingTraceId -> gTlsPendingTraceId, tlsPendingConsumed -> gTlsPendingConsumed.
- Add direct includes for opentelemetry trace_id.h / span_id.h (header uses
  TraceId/SpanId in signatures) and sdk/trace/id_generator.h (.cpp references
  IdGenerator directly) to satisfy misc-include-cleaner.

Both verified clean with clang-tidy against a telemetry-enabled compile DB.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 16:20:16 +01:00
Pratik Mankawde
1fc65db139 docs(telemetry): rename Ripple->XRPL / rippled->xrpld in telemetry docs
The check-rename CI gate requires all prose/comment references to use the
new naming. Fix the four remaining occurrences ("Ripple epoch",
"rippled's doAccept") introduced by this branch's dashboard/glossary
commits. Comment- and doc-only; no behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 15:05:29 +01:00
Pratik Mankawde
056c863466 delete throwaway scripts
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-07-22 14:52:37 +01:00
Pratik Mankawde
3ad291eb75 Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-07-22 14:44:21 +01:00
Pratik Mankawde
3a9dab6d7c Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-07-22 14:44:15 +01:00
Pratik Mankawde
644cf3a0a9 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-22 14:44:15 +01:00
Pratik Mankawde
531802ed81 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-22 14:44:15 +01:00
Pratik Mankawde
12034f1a7f Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd 2026-07-22 14:44:15 +01:00
Pratik Mankawde
c608131e26 Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-07-22 14:44:15 +01:00
Pratik Mankawde
032aa10d63 Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-07-22 14:44:15 +01:00
Pratik Mankawde
735e195005 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing 2026-07-22 14:44:15 +01:00
Pratik Mankawde
6f868e36e5 Merge branch 'pratik/otel-phase1c-rpc-integration' into pratik/otel-phase2-rpc-tracing 2026-07-22 14:44:15 +01:00
Pratik Mankawde
a25f18521e Merge branch 'pratik/otel-phase1b-telemetry-infra' into pratik/otel-phase1c-rpc-integration 2026-07-22 14:44:15 +01:00
Pratik Mankawde
64cea3ded4 docs(telemetry): use rpc_span::op::process constant in @code examples
The OTel naming check (Rule F) scans @code doc-comment examples and fails
on string-literal span names; Rule H warns on undefined SpanNames
constants. Replace the literal "subtask" and the undefined
rpc_span::op::dispatch with the defined rpc_span::op::process constant so
the examples model correct API usage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 14:43:59 +01:00
Pratik Mankawde
4535daa10d Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation
Brings in the telemetry/metrics init-order fix (start before [rpc_startup]).

Conflict in Application.cpp: phase10 had reordered start() to run
telemetry+metrics first; phase9 relocated that block into a new
startTelemetry() called from setup() before the [rpc_startup] loop.
Resolved by taking the fix — the block lives only in startTelemetry()
(called once from setup(), after overlay_ is built), removed from start().
telemetry_->start() and metricsRegistry_->start() each appear exactly once.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 14:39:39 +01:00
Pratik Mankawde
93e95bdc03 fix(telemetry): start telemetry + metrics before [rpc_startup]
Metric-emitting code that first runs during ApplicationImp::setup() (the
[rpc_startup] loop) executed while the OTel meter was still empty, because
telemetry_ and metricsRegistry_ were started later in start(). A call-site
metric macro caches its instrument on first use via std::call_once, so that
first pre-meter use latched a null instrument for the process lifetime and the
metric silently never recorded (observed: rpc_in_flight_requests, emitted from
PerfLogImp::rpcStart, never appeared).

Extract telemetry_->start() and the metricsRegistry_->start() block into a new
private ApplicationImp::startTelemetry() and call it from setup() just before
the [rpc_startup] loop, so the meter is live before any metric-emitting code
runs. The call sits after overlay_ (and the other subsystems the observable-
gauge callbacks read) is constructed, since starting the metrics reader thread
earlier would let its callbacks call getOverlay() before overlay_ exists.
perfLog_->start() stays in start() — the macro call sites do not depend on it.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 14:35:55 +01:00
Pratik Mankawde
7ef1f22ebe docs(telemetry): add Computation boundary to each panel description
Add a 'Computation boundary' section to every keyworded panel stating (a)
whether the result is per-node or aggregated across nodes, derived from
the query's grouping, and (b) where the value is computed along the
pipeline: xrpld code (MetricsRegistry OTel SDK, or beast::insight native),
the collector's SpanMetrics connector, or a Tempo TraceQL query over raw
spans. Classification is derived per panel from its actual queries and the
MetricsRegistry source, not guessed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 14:34:15 +01:00
Pratik Mankawde
8dd64d4dcd fix(telemetry): add second-scale spanmetrics histogram buckets
P95 of second-scale spans was a meaningless interpolation. The spanmetrics
histogram topped out at [.. 1s, 5s], so consensus.round (~3.9s) and
consensus.establish (~1.9s) all fell into one 1s-5s bucket and
histogram_quantile interpolated linearly across that 4s-wide gap — the
"Build vs Close" / "Ledger Close Duration" panels' P95 read ~4800ms purely
as an artifact (verified: sum/count avg = 3824ms). ledger.acquire was worse:
~17% of samples exceeded the 5s ceiling, so its p95/p99 were unmeasurable.

Add 2s, 3s, 4s (resolve the 1-5s pile-up) and 10s, 30s (give the
ledger.acquire catch-up tail a measurable home). All ten existing boundaries
are preserved and the list stays strictly ascending (the connector
binary-searches buckets and silently misbuckets otherwise). Pin unit=ms so a
future collector default-unit flip can't rename the metric to _seconds.

Buckets chosen from the live mainnet distribution, not guessed. Native
beast::insight histograms (ms-scale RPC/IO timers in Telemetry.cpp) are 100%
under 5s, so they keep the original buckets — this is collector-only.

Applies on collector restart (cumulative series reset once, handled by
rate()). Runbook and regression-threshold bucket notes updated to match.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 14:27:06 +01:00
Pratik Mankawde
03d04ea7b7 docs(telemetry): tag each keyword with per-node vs network-wide scope
Add a scope tag to every panel keyword — (per node), (network-wide),
(network event), or (cluster-wide) — so a reader can tell whether a term
describes this server's own state, a protocol-shared fact, or a
network-wide consensus process the node participates in. Add a matching
'Scope:' line to each glossary entry.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 14:21:34 +01:00
Pratik Mankawde
13a48cbbf2 refresh interval set to 30s
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-07-22 14:15:33 +01:00
Pratik Mankawde
62371de2a6 docs(telemetry): add panel Keywords + References and a telemetry glossary
Add a Keywords definition-list and a References link line to 178 Grafana
panel descriptions across all 14 dashboards, defining the XRPL/rippled
domain terms each panel uses (tiers 1-9: ledger, consensus, transaction
pipeline, fees/queue, node state, peer/overlay, storage, validator,
RPC/pathfinding). Cross-cutting chart terms and job-queue internals are
intentionally excluded.

Add docs/telemetry-glossary.md (86 terms, 9 categories) as the deeper
reference, linked from each panel's References line and from the runbook.
Keywords are injected only where a term appears in that panel's prose
(matched over description text, not source-file citations).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 14:14:12 +01:00
Pratik Mankawde
0e02a67e81 Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-07-22 13:46:45 +01:00
Pratik Mankawde
4858f65be4 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-22 13:43:24 +01:00
Pratik Mankawde
091d3b2bb2 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-22 13:43:24 +01:00
Pratik Mankawde
476963f1db Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd 2026-07-22 13:43:24 +01:00
Pratik Mankawde
23a0d70d49 Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-07-22 13:43:24 +01:00
Pratik Mankawde
f59e34192a Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-07-22 13:43:24 +01:00
Pratik Mankawde
d64b744540 docs(telemetry): InboundLedger acquireSpan_ is thread-free, not detached
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 13:43:12 +01:00
Pratik Mankawde
b0b174b94d docs(telemetry): RCLConsensus roundSpan_ is thread-free, not detached
The roundSpan_ / roundSpanContext_ comments still described the old
'detached, Scope stripped' model. roundSpan_ is now a thread-free SpanGuard
(no Scope); children link via the captured roundSpanContext_.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 13:43:07 +01:00
Pratik Mankawde
5150159ca2 refactor(telemetry): ledger.acquire uses thread-free SpanGuard (no detach)
acquireSpan_ is emplaced on the acquiring thread and reset() on a JtLedgerData
worker. SpanGuard is now thread-free (owns no thread-local Scope), so it can be
created here and destroyed on the worker with no scope to strip — dropping the
.detached() call the old scoped SpanGuard required.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 13:30:50 +01:00
Pratik Mankawde
dd96d8ad34 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-07-22 13:28:44 +01:00
Pratik Mankawde
dad2dc2acb Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-22 13:28:24 +01:00
Pratik Mankawde
cef7a8b16c Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
# Conflicts:
#	OpenTelemetryPlan/09-data-collection-reference.md
2026-07-22 13:28:15 +01:00
Pratik Mankawde
ba1b693177 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing 2026-07-22 13:13:24 +01:00
Pratik Mankawde
30398ce2fe Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd 2026-07-22 13:13:24 +01:00