Commit Graph

125 Commits

Author SHA1 Message Date
Pratik Mankawde
d17c1c5c7f Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase7-native-metrics 2026-07-08 18:44:16 +01:00
Pratik Mankawde
2321b4985d Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-07-08 18:44:04 +01:00
Pratik Mankawde
04f1775e10 Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-07-08 18:42:58 +01:00
Pratik Mankawde
ec7d99228a fix(telemetry): add missing OTel headers for TraceContextPropagator (misc-include-cleaner) 2026-07-08 18:42:31 +01:00
Pratik Mankawde
09e083a718 Merge branch 'pratik/otel-phase1b-telemetry-infra' into pratik/otel-phase3-tx-tracing 2026-07-08 18:42:10 +01:00
Pratik Mankawde
e7ba376839 fix(telemetry): add missing span_metadata.h for SpanKind (misc-include-cleaner) 2026-07-08 18:41:59 +01:00
Pratik Mankawde
087e6e5b63 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-08 15:41:21 +01:00
Pratik Mankawde
2ac68427fc Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-07-08 15:41:21 +01:00
Pratik Mankawde
5ba27cc90f Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-07-08 15:41:07 +01:00
Pratik Mankawde
f4549b68c7 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing
# Conflicts:
#	include/xrpl/telemetry/SpanGuard.h
2026-07-08 15:41:04 +01:00
Pratik Mankawde
8800160bb5 Merge branch 'pratik/otel-phase1c-rpc-integration' into pratik/otel-phase2-rpc-tracing 2026-07-08 15:40:50 +01:00
Pratik Mankawde
7848574a38 Merge branch 'pratik/otel-phase1b-telemetry-infra' into pratik/otel-phase1c-rpc-integration 2026-07-08 15:40:50 +01:00
Pratik Mankawde
4e9df8b5c3 fix(telemetry): add missing <cstddef> include for std::size_t in SpanGuard.h
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-08 15:40:38 +01:00
Pratik Mankawde
a396fe073f Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-08 15:38:26 +01:00
Pratik Mankawde
b1091a482c Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment
# Conflicts:
#	src/libxrpl/telemetry/Telemetry.cpp
2026-07-08 15:23:03 +01:00
Pratik Mankawde
87c003230c Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-07-08 15:20:07 +01:00
Pratik Mankawde
6938614468 fix(telemetry): remove unused includes (misc-include-cleaner)
Remove unused SpanNames.h from Transactor.cpp (transitively included
via TxApplySpanNames.h). Remove unused <ranges> from
TraceContextValidation.h (std::ranges::any_of lives in <algorithm>).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-08 15:20:01 +01:00
Pratik Mankawde
8108caffe3 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing 2026-07-08 15:19:01 +01:00
Pratik Mankawde
01c9f6cf08 Merge branch 'pratik/otel-phase1c-rpc-integration' into pratik/otel-phase2-rpc-tracing 2026-07-08 15:18:11 +01:00
Pratik Mankawde
ba8785dc16 Merge branch 'pratik/otel-phase1b-telemetry-infra' into pratik/otel-phase1c-rpc-integration 2026-07-08 15:18:06 +01:00
Pratik Mankawde
c6be7cf83a fix(telemetry): resolve clang-tidy CI errors (include-cleaner, braces)
Add missing <cstdint> include for std::uint32_t in Telemetry.h.
Add braces around single-line if bodies in Telemetry.cpp.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-08 15:17:58 +01:00
Pratik Mankawde
ab0d8249fd feat(telemetry): expose direct OTel metrics API via getMeter()
Give developers direct access to the full OpenTelemetry metrics API,
symmetric with getTracer(), so all seven OTel instrument types
(including UpDownCounter, sync Gauge, and the observable variants) are
reachable — not just the four beast::insight models.

- Telemetry: build and own the metrics pipeline (OTLP HTTP metric
  exporter + PeriodicExportingMetricReader + SpanMetrics histogram view)
  alongside the tracer, sharing the same resource attributes and TLS
  config. Register it globally via metrics::Provider::SetMeterProvider
  and expose Telemetry::getMeter(). Metrics enable with [telemetry];
  the metrics endpoint is derived from the trace endpoint.
- beast OTelCollector: no longer owns a pipeline. It fetches the global
  Meter, becoming a thin shim over the shared provider (legacy path
  during beast deprecation). This also resolves the review note that the
  metric exporter ignored [telemetry] use_tls — TLS now comes from the
  shared telemetry pipeline.
- Add a libxrpl unit test covering getMeter() on the enabled (global
  provider) and disabled (noop) paths, exercising an UpDownCounter.

Design: docs/superpowers/specs/2026-07-07-direct-otel-metrics-api-design.md

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 16:52:45 +01:00
Pratik Mankawde
363c2caf94 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
# Conflicts:
#	OpenTelemetryPlan/08-appendix.md
#	OpenTelemetryPlan/09-data-collection-reference.md
#	docker/telemetry/docker-compose.yml
#	docker/telemetry/grafana/dashboards/statsd-ledger-data-sync.json
#	docker/telemetry/grafana/dashboards/statsd-network-traffic.json
#	docker/telemetry/grafana/dashboards/statsd-node-health.json
#	docker/telemetry/grafana/dashboards/statsd-rpc-pathfinding.json
2026-07-06 22:01:22 +01:00
Pratik Mankawde
1938b9e377 Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-07-06 21:05:06 +01:00
Pratik Mankawde
04d4bba780 Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-07-06 21:04:57 +01:00
Pratik Mankawde
5287893d82 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing 2026-07-06 21:04:50 +01:00
Pratik Mankawde
89b5210e5b docs(telemetry): clarify redactAccount is obfuscation, not secrecy
Soften the "non-reversible" wording in Redaction.h. XRP account
addresses are a public, enumerable set, so an unsalted hash is
obfuscation (keeps plaintext out of traces/dashboards), not a
secrecy guarantee. Document why a salt is intentionally omitted:
it would break cross-node/restart correlation, the reason for
hashing rather than dropping the attribute.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 20:54:09 +01:00
Pratik Mankawde
ea9c639ee9 Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-07-06 20:50:24 +01:00
Pratik Mankawde
e02b7968ad Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-07-06 20:41:14 +01:00
Pratik Mankawde
7a6ff619e5 code review changes
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-07-06 20:37:08 +01:00
Pratik Mankawde
f55f8c59fc feat(telemetry): stamp service.name and network on native metrics
The native OTel metrics path hard-coded service.name="xrpld" and stamped
no network attribute, while traces stamped a configurable service.name
and xrpl.network.type. Metrics therefore could not be filtered by service
or network. Align the two paths:

- OTelCollector::New / OTelCollectorImp gain serviceName + networkType
  params. service.name uses the configured value (default "xrpld" when
  unset, preserving today's behavior); xrpl.network.type is stamped when
  provided. The key is a string literal because beast/insight sits below
  the telemetry module and cannot include its SpanNames const.
- CollectorManager reads service_name from [insight], falling back to the
  [telemetry] value, and receives the network type from the caller.
- Application derives the network type once via the shared
  telemetry::networkTypeFromId, now declared in Telemetry.h and moved out
  of an anonymous namespace so the trace and metric paths reuse a single
  0/1/2 -> mainnet/testnet/devnet mapping (no duplication).

Dashboards (5 system-* files): add $service_name, $deployment_environment,
$xrpl_network_type template variables and wire them into every panel query
that filters by $node.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 19:40:59 +01:00
Pratik Mankawde
3463a85943 Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-06-26 14:46:10 +01:00
Pratik Mankawde
bc7cff230c Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-06-26 14:45:43 +01:00
Pratik Mankawde
54d92ff973 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-06-26 14:42:55 +01:00
Pratik Mankawde
f9092e7948 Merge branch 'pratik/otel-phase1b-telemetry-infra' into pratik/otel-phase1c-rpc-integration
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-06-15 20:39:08 +01:00
Pratik Mankawde
f0b834a913 minor naming change
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-06-15 20:02:48 +01:00
Pratik Mankawde
0e9a71798c make samping ratio static constexpr
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-06-15 19:44:55 +01:00
Pratik Mankawde
ba85aecf00 hide gTlDiscardCurrentSpan better
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-06-15 17:21:35 +01:00
Pratik Mankawde
37a4c9deb8 core review comments
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-06-15 17:10:23 +01:00
Pratik Mankawde
e1045a84aa Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-06-11 23:16:54 +01:00
Pratik Mankawde
19a6c2a306 refactor(telemetry): consistent consensus span-attribute names
Make consensus span-attribute keys consistent with the peer spans that record
the same concepts, using ONE shared name per concept (told apart by span name),
not an emitter prefix:

- Add the shared ledger_hash and full_validation keys to the base SpanNames.h
  (a ledger-object property and an is-full-validation flag, both shared with the
  peer validation spans); consensus re-exports them via `using` instead of
  defining local copies.
- trusted (bare) -> proposal_trusted on consensus.proposal.receive and
  validation_trusted on consensus.validation.receive — the same message-type
  qualification the peer.{proposal,validation}.receive spans use.

Also collapse a same-emitter duplicate: establishCounter_ was recorded under
both establish_counter and establish_count on two consensus spans; both now use
establish_count (the establishCounter span-name constant is removed; the
establishCounter_ member variable is unchanged).
2026-06-11 23:08:39 +01:00
Pratik Mankawde
448aaa4718 Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment
Bring phase-4 forward into phase 5 (docs & deployment). Phase 5 owns the
operator runbook (docs/telemetry-runbook.md).

Conflict resolution:
- 05-configuration-reference.md: took phase-4's code-block-free prose for the
  config-parser section; the mTLS options (tls_client_cert/tls_client_key)
  remain documented in the §5.1.2 options table.

Phase-5-owned naming fix:
- docs/telemetry-runbook.md: converted 20 dotted xrpl.<domain>. attribute keys
  in the Span Reference tables to the underscore convention (tx_hash, peer_id,
  ledger_seq, consensus_mode/round/round_id/ledger_id, tx_id). Span NAMES stay
  dotted (rule 5). `trusted` left as-is — verified it matches the code constant
  ConsensusSpanNames.h (the proposal_trusted/validation_trusted split is a
  separate, not-yet-applied code change).

End-to-end reconciliation: every span-attribute token in the runbook (L5) now
matches an L1 *SpanNames.h constant. The xrpl_<domain>_<field> tokens are the
spanmetrics-derived Prometheus labels (documented mapping), not span attrs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 17:43:57 +01:00
Pratik Mankawde
294c276e29 Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing
Bring phase-3 forward into phase 4 (consensus tracing). Phase 4 introduces
ConsensusSpanNames.h (and already did the trusted→proposal/validation rename
and underscore-attr conversion in ffc197b914).

Conflict resolution (reviewed by a code-review agent before commit):
- 02-design-decisions.md §2.4 Consensus Attributes: merge-both — kept phase-3's
  underscore table form AND folded in phase-4's richer "Phase 4a" attribute set
  (round_id, ledger_id, trace_strategy, converge/establish/disputes counts,
  agree/disagree counts, threshold_percent, consensus_result, mode_old/new),
  each mapped to its authoritative ConsensusSpanNames.h constant. Dropped the
  planned-but-unimplemented proposers_agreed/proposers_total (no code constant;
  agree_count/disagree_count serve that role).
- SpanGuardFactory.cpp: kept phase-4's explanatory comment about why a libxrpl
  test uses literal keys, plus the converted command/rpc_status keys.

Naming check passes (100 keys across 8 *SpanNames.h headers, including consensus).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 17:04:06 +01:00
Pratik Mankawde
7cb08307a7 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing
Bring phase-2 forward into phase 3 (transaction tracing). Phase 3 introduces
TxSpanNames.h, TxQSpanNames.h, and TxApplySpanNames.h.

Conflict resolution:
- TxQ.cpp: kept phase-3's txq_span-based instrumentation (phase-2 had none).
  Dropped the orphaned `NumberSO{... fixUniversalNumber}` line — develop's
  #5962 (Retire fixUniversalNumber) removed that symbol repo-wide; the
  conflict block had carried one stale copy that would not compile.
- 05/08/OpenTelemetryPlan.md: dropped the deleted 04-code-samples / POC_taskList
  references (carried from phase-2), kept phase-3's new secure-OTel.md doc rows,
  section, and Mermaid node/edge/style. Config code block -> prose; merged the
  secure-OTel hardening pointer with the authoritative-config prose.
- Phase3_taskList.md: removed the "dotted keys for readability" note that came
  from phase-2 — phase 3 already uses the underscore keys.

Reviewed by code-review agents: telemetry instrumentation intact, naming check
green (47 keys across 7 *SpanNames.h headers), no conflict markers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 16:45:34 +01:00
Pratik Mankawde
4d044e6254 ci: Harden OTel naming check — unconditional Rule F, test exemption, Rule H
Three robustness fixes to check_otel_naming.py, all on phase-1c where the
script lives:

- Rule F now runs UNCONDITIONALLY. It is a purely syntactic check on the
  call-sites and does not need the L1 key set, so code that calls
  SpanGuard::span/setAttribute directly without ever defining a *SpanNames.h
  is still caught (previously it was silently skipped when no header existed).
- Exempt test files from Rule F (tests pass arbitrary literal keys to exercise
  the API). The call-site matcher now requires a SpanGuard/`.`/`->` receiver,
  so std::span and bare declarations no longer false-positive.
- Add Rule H (warning, non-fatal): a namespace-qualified constant used at a
  telemetry call-site but not defined in any *SpanNames.h is flagged, catching
  constants defined in-place instead of in the proper header. Bare locals and
  std:: names are not warned to avoid noise.

SpanGuard.h / Telemetry.h @code examples updated to reference constants that
exist on this branch. README documents the new behavior.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 16:10:08 +01:00
Pratik Mankawde
134a24d5bc ci: Add OpenTelemetry span-attribute naming check (phase 1c)
Add check_otel_naming.py and wire it into on-pr.yml so every PR validates
that span-attribute names stay consistent across the code, collector, Tempo,
dashboards, and docs.

- The valid key set is derived dynamically from the *SpanNames.h constants and
  the resource attributes the code registers in Telemetry.cpp — no hardcoded
  allowlist to drift.
- Each rule is presence-gated: it runs only when the file it needs is in the
  tree, so the check is correct whether telemetry changes land in one PR or
  several (the collector/Tempo/dashboard/runbook layers arrive in later phases).
- Rule A flags dotted span-attribute keys; Rule F flags string-literal
  attribute keys and span-name arguments (values may be runtime data).
- stdlib-only, mirroring the levelization check (bare `python`, no pip step).
- Telemetry.h / SpanGuard.h @code examples now use *SpanNames.h constants so
  the strict literal check passes.
- CONTRIBUTING.md documents the check and how to run it locally.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 15:26:38 +01:00
Pratik Mankawde
e9cb9421ef Merge branch 'pratik/otel-phase1a-plan-docs' into pratik/otel-phase1b-telemetry-infra
Bring the span attribute naming convention (phase 1a) into phase 1b.

Conflict resolution kept phase-1b's SpanGuard-based workflow and applied
the underscore naming convention to all non-code-sample text:
- Converted prose, tables, Mermaid labels, and TraceQL/PromQL query
  references across the plan docs to the underscore form.
- Converted the two @code attribute-key examples in Telemetry.h
  (command, tx_type).
- Left the code-sample files (04-code-samples.md, POC_taskList.md) and
  03-implementation-strategy.md code blocks at the phase-1b version; the
  code-sample docs are slated for removal on phase-1a.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 14:10:52 +01:00
Pratik Mankawde
53ebd7a8c2 Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-06-10 17:47:27 +01:00
Pratik Mankawde
4a0994209e Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-06-10 16:26:23 +01:00
Pratik Mankawde
f37589b1f5 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing 2026-06-10 16:05:29 +01:00