Commit Graph

14974 Commits

Author SHA1 Message Date
Pratik Mankawde
a396fe073f Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-08 15:38:26 +01:00
Pratik Mankawde
0bfb338666 Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd 2026-07-08 15:38:21 +01:00
Pratik Mankawde
0a373f631c fix(telemetry): resolve otel-naming and UBSAN CI failures
- Add job_type to otel-naming builtins (standard Prometheus label for
  job-queue metrics used in node-health dashboard)
- Add OpenTelemetry SDK to UBSAN suppressions (intentional unsigned
  integer overflow in attributemap_hash.h hash computation)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-08 15:25:59 +01:00
Pratik Mankawde
b1091a482c Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment
# Conflicts:
#	src/libxrpl/telemetry/Telemetry.cpp
2026-07-08 15:23:03 +01:00
Pratik Mankawde
a1a9d45bff Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-08 15:22:52 +01:00
Pratik Mankawde
4826038068 fix(telemetry): run rename scripts for CI compliance
Rename rippled → xrpld in statsd config prefix.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-08 15:22:45 +01:00
Pratik Mankawde
749535a3ca fix(telemetry): resolve clang-tidy CI errors in consensus tracing
Add missing <string_view> include in Consensus.h and RCLConsensus.h.
Remove unused SHAMap.h include from RCLConsensus.h.
Add braces around single-line if/else bodies in RCLConsensus.cpp.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-08 15:22:24 +01:00
Pratik Mankawde
87c003230c Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-07-08 15:20:07 +01:00
Pratik Mankawde
6938614468 fix(telemetry): remove unused includes (misc-include-cleaner)
Remove unused SpanNames.h from Transactor.cpp (transitively included
via TxApplySpanNames.h). Remove unused <ranges> from
TraceContextValidation.h (std::ranges::any_of lives in <algorithm>).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-08 15:20:01 +01:00
Pratik Mankawde
8108caffe3 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing 2026-07-08 15:19:01 +01:00
Pratik Mankawde
db38ba268f fix(telemetry): use std::ranges algorithms (modernize-use-ranges)
Replace std::transform with std::ranges::transform in Redaction.cpp.
Replace std::all_of with std::ranges::all_of in Redaction test.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-08 15:18:55 +01:00
Pratik Mankawde
01c9f6cf08 Merge branch 'pratik/otel-phase1c-rpc-integration' into pratik/otel-phase2-rpc-tracing 2026-07-08 15:18:11 +01:00
Pratik Mankawde
ba8785dc16 Merge branch 'pratik/otel-phase1b-telemetry-infra' into pratik/otel-phase1c-rpc-integration 2026-07-08 15:18:06 +01:00
Pratik Mankawde
c6be7cf83a fix(telemetry): resolve clang-tidy CI errors (include-cleaner, braces)
Add missing <cstdint> include for std::uint32_t in Telemetry.h.
Add braces around single-line if bodies in Telemetry.cpp.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-08 15:17:58 +01:00
Pratik Mankawde
a2a58a5d85 docs(telemetry): update plan docs to use service_instance_id
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-08 14:57:29 +01:00
Pratik Mankawde
87f370a64d fix(telemetry): use service_instance_id label instead of exported_instance
exported_instance is a Prometheus scrape artifact that only exists when
Prometheus scrapes a target with a conflicting instance label. Users
running push-based pipelines (remote_write, OTLP) never get this label.
service_instance_id is always present via resource_to_telemetry_conversion
regardless of ingestion path.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-08 14:51:48 +01:00
Pratik Mankawde
ff92c3727c Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-08 14:51:36 +01:00
Pratik Mankawde
1eb44689d0 fix(telemetry): use service_instance_id label instead of exported_instance
exported_instance is a Prometheus scrape artifact that only exists when
Prometheus scrapes a target with a conflicting instance label. Users
running push-based pipelines (remote_write, OTLP) never get this label.
service_instance_id is always present via resource_to_telemetry_conversion
regardless of ingestion path.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-08 14:51:26 +01:00
Pratik Mankawde
9a769b9729 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-07 19:58:08 +01:00
Pratik Mankawde
afe653e4b6 Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd 2026-07-07 19:57:31 +01:00
Pratik Mankawde
fca98d1003 Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-07-07 19:57:22 +01:00
Pratik Mankawde
35148b57f8 Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing
# Conflicts:
#	docker/telemetry/grafana/provisioning/datasources/tempo.yaml
2026-07-07 19:57:14 +01:00
Pratik Mankawde
5e77968ca2 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing
# Conflicts:
#	docker/telemetry/grafana/provisioning/datasources/tempo.yaml
2026-07-07 19:56:33 +01:00
Pratik Mankawde
e7bdf659fb Merge branch 'pratik/otel-phase1c-rpc-integration' into pratik/otel-phase2-rpc-tracing
# Conflicts:
#	docker/telemetry/grafana/provisioning/datasources/tempo.yaml
2026-07-07 19:48:02 +01:00
Pratik Mankawde
ba987bcb39 Merge branch 'pratik/otel-phase1b-telemetry-infra' into pratik/otel-phase1c-rpc-integration 2026-07-07 19:46:16 +01:00
Pratik Mankawde
40b457669e docs(telemetry): remove Phase reference from tempo.yaml additional-filter comment
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 3/4" token, keep the description.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 19:44:44 +01:00
Pratik Mankawde
b54e0e50c0 docs(telemetry): remove Phase references from tempo.yaml consensus filter comments
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 4" tokens, keep descriptions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 19:44:00 +01:00
Pratik Mankawde
36c08fdc66 docs(telemetry): remove Phase references from tempo.yaml TX filter comments
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 3" tokens, keep descriptions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 19:43:59 +01:00
Pratik Mankawde
9879780dd8 docs(telemetry): remove Phase references from tempo.yaml RPC filter comments
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 2" tokens, keep descriptions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 19:43:53 +01:00
Pratik Mankawde
b1af0a9511 docs(telemetry): remove Phase reference from tempo.yaml base-filter comment
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 1b" token, keep the filter
description.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 19:43:40 +01:00
Pratik Mankawde
a8ca3d50c3 fix(telemetry): node-health detail panels + OTelCollector hook deadlock
- node-health.json: repoint the two "All Jobs" detail panels from the
  __name__ regex over old xrpld_<job>_milliseconds_bucket names to the
  native xrpld_job_running/queued_duration_us_bucket histograms, grouping
  by the job_type label and legending on {{job_type}}.
- OTelCollector::callHooks: copy the hook list under mutex_ and invoke
  handlers outside the lock. A handler can drop the last reference to an
  OTelHookImpl, whose destructor calls removeHook() and re-acquires the
  non-recursive mutex_ — invoking handlers under the lock could deadlock.
- ~OTelGaugeImpl: document that the SDK ObservableRegistry serializes
  RemoveCallback against the Observe() callback pass with one mutex, so
  callback removal is synchronous and gaugeCallback cannot run on a
  dangling pointer after the destructor.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 19:15:11 +01:00
Pratik Mankawde
bc140d9e96 fix(telemetry): repoint node-health job panels to native job-duration metrics
The Key Jobs execution/dequeue panels queried StatsD-era per-job metric
names (xrpld_<job>_milliseconds_bucket, xrpld_<job>_q_milliseconds_bucket)
that the native OTel path no longer emits. Phase 9's MetricsRegistry emits
job timings as two label-dimensioned histograms instead:
  xrpld_job_running_duration_us_bucket{job_type="<job>"}
  xrpld_job_queued_duration_us_bucket{job_type="<job>"}

Rewrite the 22 affected panel queries (11 job types x running/queued) to
the label-dimensioned form, preserving the histogram_quantile / rate /
sum-by structure and all template-variable filters.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 18:54:01 +01:00
Pratik Mankawde
04f351f3b7 fix(telemetry): more phase-7 review fixes
- ValidationTracker: remove dead hard-trim Pass 1. The preceding
  eviction loop already erases every reconciled entry older than the
  late-repair cutoff using the same condition, so Pass 1 could never
  match. The "drop any reconciled entry" fallback is retained.
- integration-test.sh: probe the StatsD port with a UDP-aware check
  (ss -ulnp) instead of curl. StatsD is UDP-only on 8125, so the TCP
  curl probe always reported "refused" — a false negative. Guarded by
  a command -v ss check that logs a skip when ss is unavailable.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 16:55:19 +01:00
Pratik Mankawde
ab0d8249fd feat(telemetry): expose direct OTel metrics API via getMeter()
Give developers direct access to the full OpenTelemetry metrics API,
symmetric with getTracer(), so all seven OTel instrument types
(including UpDownCounter, sync Gauge, and the observable variants) are
reachable — not just the four beast::insight models.

- Telemetry: build and own the metrics pipeline (OTLP HTTP metric
  exporter + PeriodicExportingMetricReader + SpanMetrics histogram view)
  alongside the tracer, sharing the same resource attributes and TLS
  config. Register it globally via metrics::Provider::SetMeterProvider
  and expose Telemetry::getMeter(). Metrics enable with [telemetry];
  the metrics endpoint is derived from the trace endpoint.
- beast OTelCollector: no longer owns a pipeline. It fetches the global
  Meter, becoming a thin shim over the shared provider (legacy path
  during beast deprecation). This also resolves the review note that the
  metric exporter ignored [telemetry] use_tls — TLS now comes from the
  shared telemetry pipeline.
- Add a libxrpl unit test covering getMeter() on the enabled (global
  provider) and disabled (noop) paths, exercising an UpDownCounter.

Design: docs/superpowers/specs/2026-07-07-direct-otel-metrics-api-design.md

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 16:52:45 +01:00
Pratik Mankawde
cc4457b5a2 fix(telemetry): address phase-7 review comments
- cmake: keep the opentelemetry-cpp umbrella target for the beast metrics
  link and document why. The reviewer suggested linking individual
  component targets to avoid over-linking, but the OTel Conan package
  under-declares inter-component dependencies (the OTLP client references
  sdk::common symbols without a declared edge), so naming components
  directly reorders the static link into an unresolvable state. Verified
  by building xrpl_tests both ways.
- OTelCollector.h: add usage examples, thread-safety and limitations
  @note blocks to the class doc.
- OTelCollector.cpp: correct the @param name docs on the instrument
  Impl constructors to describe the already-formatName()'d value.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 16:50:38 +01:00
Pratik Mankawde
134b419895 docs(telemetry): fold external-parity spec into implementation-phases doc
Move the standalone External Dashboard Parity design spec out of
docs/superpowers/ (which the project guidelines say not to create) and into
an appendix of OpenTelemetryPlan/06-implementation-phases.md, so the phase
plan is self-contained. Repoint the 10 "Source" links in the phase 3/4/7
task lists to the new appendix anchor.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-07 13:08:04 +01:00
Pratik Mankawde
363c2caf94 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
# Conflicts:
#	OpenTelemetryPlan/08-appendix.md
#	OpenTelemetryPlan/09-data-collection-reference.md
#	docker/telemetry/docker-compose.yml
#	docker/telemetry/grafana/dashboards/statsd-ledger-data-sync.json
#	docker/telemetry/grafana/dashboards/statsd-network-traffic.json
#	docker/telemetry/grafana/dashboards/statsd-node-health.json
#	docker/telemetry/grafana/dashboards/statsd-rpc-pathfinding.json
2026-07-06 22:01:22 +01:00
Pratik Mankawde
f4981a1907 Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd
# Conflicts:
#	OpenTelemetryPlan/08-appendix.md
#	docker/telemetry/grafana/provisioning/datasources/tempo.yaml
#	docker/telemetry/otel-collector-config.yaml
#	docs/telemetry-runbook.md
2026-07-06 21:47:17 +01:00
Pratik Mankawde
7cc963f494 fix(telemetry): phase-6 statsd dashboards, config, and doc corrections
- Fix invalid PromQL in statsd dashboards (rate/topk/nested-__name__
  label selectors); add missing $node filter on a binary-op operand
  in statsd-node-health "Ledger Publish Gap".
- Wrap cumulative counters in rate(); correct Bytes_In metric.
- Pin prometheus image; add [insight] server=statsd stanza (prefix
  rippled matches the rippled_* StatsD metric names the dashboards
  query); bind statsd port to loopback; fix Jaeger->Grafana comment.
- Correct dotted attribute keys to underscore form in
  09-data-collection-reference (peer_id, tx_hash, consensus_mode, ...).
- Reconcile span count (11) and proposers attribute in the
  integration-test task list.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 21:36:59 +01:00
Pratik Mankawde
1938b9e377 Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-07-06 21:05:06 +01:00
Pratik Mankawde
04d4bba780 Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-07-06 21:04:57 +01:00
Pratik Mankawde
5287893d82 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing 2026-07-06 21:04:50 +01:00
Pratik Mankawde
b7d0b700c6 fix(telemetry): correct runbook pathfind spans + mTLS validation
- Add PathFinding Spans section to the runbook Span Reference and
  remove the incorrect "not emitted / later-phase" annotations:
  pathfind.request/compute/discover/update_all are emitted since
  Phase 2 (upstream of Phase 5).
- Runbook label names use the bare spanmetrics dimensions
  (command, rpc_status, local, consensus_mode); fix stale docs.
- 05-configuration-reference: note tls_client_cert/key require use_tls=1.
- TelemetryConfig: reject mTLS cert paths when use_tls=0 so telemetry
  is not silently sent in plaintext.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 21:04:07 +01:00
Pratik Mankawde
89b5210e5b docs(telemetry): clarify redactAccount is obfuscation, not secrecy
Soften the "non-reversible" wording in Redaction.h. XRP account
addresses are a public, enumerable set, so an unsalted hash is
obfuscation (keeps plaintext out of traces/dashboards), not a
secrecy guarantee. Document why a salt is intentionally omitted:
it would break cross-node/restart correlation, the reason for
hashing rather than dropping the attribute.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 20:54:09 +01:00
Pratik Mankawde
ea9c639ee9 Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-07-06 20:50:24 +01:00
Pratik Mankawde
7998d01dc9 code review comments.
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-07-06 20:49:51 +01:00
Pratik Mankawde
e02b7968ad Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-07-06 20:41:14 +01:00
Pratik Mankawde
38f6afbf4c code review comments
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-07-06 20:40:12 +01:00
Pratik Mankawde
092afb7aae Merge branch 'pratik/otel-phase1c-rpc-integration' into pratik/otel-phase2-rpc-tracing
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-07-06 20:39:27 +01:00
Pratik Mankawde
7a6ff619e5 code review changes
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-07-06 20:37:08 +01:00