Commit Graph

155 Commits

Author SHA1 Message Date
Pratik Mankawde
363c2caf94 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
# Conflicts:
#	OpenTelemetryPlan/08-appendix.md
#	OpenTelemetryPlan/09-data-collection-reference.md
#	docker/telemetry/docker-compose.yml
#	docker/telemetry/grafana/dashboards/statsd-ledger-data-sync.json
#	docker/telemetry/grafana/dashboards/statsd-network-traffic.json
#	docker/telemetry/grafana/dashboards/statsd-node-health.json
#	docker/telemetry/grafana/dashboards/statsd-rpc-pathfinding.json
2026-07-06 22:01:22 +01:00
Pratik Mankawde
f4981a1907 Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd
# Conflicts:
#	OpenTelemetryPlan/08-appendix.md
#	docker/telemetry/grafana/provisioning/datasources/tempo.yaml
#	docker/telemetry/otel-collector-config.yaml
#	docs/telemetry-runbook.md
2026-07-06 21:47:17 +01:00
Pratik Mankawde
7cc963f494 fix(telemetry): phase-6 statsd dashboards, config, and doc corrections
- Fix invalid PromQL in statsd dashboards (rate/topk/nested-__name__
  label selectors); add missing $node filter on a binary-op operand
  in statsd-node-health "Ledger Publish Gap".
- Wrap cumulative counters in rate(); correct Bytes_In metric.
- Pin prometheus image; add [insight] server=statsd stanza (prefix
  rippled matches the rippled_* StatsD metric names the dashboards
  query); bind statsd port to loopback; fix Jaeger->Grafana comment.
- Correct dotted attribute keys to underscore form in
  09-data-collection-reference (peer_id, tx_hash, consensus_mode, ...).
- Reconcile span count (11) and proposers attribute in the
  integration-test task list.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 21:36:59 +01:00
Pratik Mankawde
ea9c639ee9 Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-07-06 20:50:24 +01:00
Pratik Mankawde
7998d01dc9 code review comments.
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-07-06 20:49:51 +01:00
Pratik Mankawde
e02b7968ad Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-07-06 20:41:14 +01:00
Pratik Mankawde
092afb7aae Merge branch 'pratik/otel-phase1c-rpc-integration' into pratik/otel-phase2-rpc-tracing
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-07-06 20:39:27 +01:00
Pratik Mankawde
7a6ff619e5 code review changes
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-07-06 20:37:08 +01:00
Pratik Mankawde
0505ca35aa fix(telemetry): make Tempo search filters dynamic
Change the six remaining `type: static` search filters in tempo.yaml to
`type: dynamic` so Grafana queries Tempo for available tag values instead
of requiring manual free-text entry. Aligns service-name, node-id,
network-type, span-name, span-status, and span-duration with the existing
node-version and network-id filters.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 17:41:38 +01:00
Pratik Mankawde
7881b707d2 docs(telemetry): drop personal name from dashboard service.name example
Same cleanup for the five system-metric dashboards introduced in this phase:
replace the "pratik-xrpld" $service_name example with "xrpld-validator".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 16:54:14 +01:00
Pratik Mankawde
424be46662 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-06 16:54:03 +01:00
Pratik Mankawde
2da3e73b63 docs(telemetry): drop personal name from dashboard service.name example
The $service_name template variable's description used a personal-name example
value. Replace "pratik-xrpld" with the generic "xrpld-validator" in the five
dashboards that introduced it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-06 16:53:56 +01:00
Pratik Mankawde
f600660364 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
# Conflicts:
#	docker/telemetry/otel-collector-config.yaml
2026-07-04 03:24:45 +01:00
Pratik Mankawde
5cf22a9758 Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd
# Conflicts:
#	docker/telemetry/otel-collector-config.yaml
2026-07-04 03:24:02 +01:00
Pratik Mankawde
4e21aefe74 Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-07-04 03:22:59 +01:00
Pratik Mankawde
bb0df16c90 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing 2026-07-04 03:22:59 +01:00
Pratik Mankawde
5031ab2b4e Merge branch 'pratik/otel-phase1c-rpc-integration' into pratik/otel-phase2-rpc-tracing 2026-07-04 03:22:59 +01:00
Pratik Mankawde
80d33662e4 feat(telemetry): strip SDK-injected resource attributes at the collector
Add a resource/stripsdk processor that deletes telemetry.sdk.language,
telemetry.sdk.name, and telemetry.sdk.version (auto-added by the OTel SDK
to every Resource). Wire it into the traces pipeline; the metrics and
logs pipelines pick it up on the branches that introduce them.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 03:22:47 +01:00
Pratik Mankawde
1ea7bb5252 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
# Conflicts:
#	OpenTelemetryPlan/09-data-collection-reference.md
2026-07-04 02:47:42 +01:00
Pratik Mankawde
2c590a47c5 refactor(telemetry): rename system-* dashboards to bare names, drop redundant prefix
Rename the 5 system-* dashboard files to bare domain names (uid ==
filename stem) and update all doc references. The system- prefix was
redundant now that titles no longer carry a recorder-pipeline suffix.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 02:45:39 +01:00
Pratik Mankawde
d5d9d1d8da refactor(telemetry): drop redundant xrpld- prefix from dashboard uids
Make dashboard uids bare domain names (uid == filename stem) and update
doc references. Grafana links/bookmarks to the old uids will need
refreshing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 02:44:54 +01:00
Pratik Mankawde
78fb08ef38 docs(telemetry): dashboard-level descriptions to plain text
Grafana renders the dashboard-level description as plain text (not
markdown), so the h6 headings/italics showed as literal characters.
Convert to plain 'What this shows:' / 'Use it to:' labeled lines. Panel
descriptions (which DO render markdown) are unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 02:32:23 +01:00
Pratik Mankawde
9f8a4c9561 docs(telemetry): reformat system dashboard descriptions to structured markdown; fix malformed traffic-category query
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-04 02:17:28 +01:00
Pratik Mankawde
b0d46af0ca Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-03 21:57:27 +01:00
Pratik Mankawde
50377df271 docs(telemetry): rewrite dashboard descriptions and tags to describe data not pipeline
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 21:56:08 +01:00
Pratik Mankawde
7befb16705 docs(telemetry): drop (System Metrics) suffix, retag, rewrite descriptions to describe data not pipeline
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 21:54:07 +01:00
Pratik Mankawde
9a4a97cdec docs(telemetry): add dashboard-level descriptions for the info button
Consensus Health, Ledger Operations, and Transaction Overview had no
top-level description, so Grafana showed no info button. Add a one-line
summary of what each dashboard shows and its data source.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 21:24:36 +01:00
Pratik Mankawde
400f84aafe Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-03 21:18:38 +01:00
Pratik Mankawde
b59b3c1bb4 fix(telemetry): correct service.name to xrpld and restore rename README
The integration-test.sh trace queries used resource.service.name="rippled",
but the OTel resource attribute is "xrpld" (TelemetryConfig.cpp, MetricsRegistry.cpp),
so every span check and the service-registration check returned no results.

Also restore .github/scripts/rename/README.md, which an overzealous
find-and-replace had corrupted into circular no-ops (e.g. rename "xrpld" to
"xrpld"). The file documents the rename process and must retain the source
names (rippled/ripple); restored to match develop.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 21:18:22 +01:00
Pratik Mankawde
5ade9431ef Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
# Conflicts:
#	docker/telemetry/grafana/dashboards/statsd-ledger-data-sync.json
#	docker/telemetry/grafana/dashboards/statsd-rpc-pathfinding.json
2026-07-03 20:42:07 +01:00
Pratik Mankawde
c669c20bc0 chore(telemetry): drop redundant rippled dashboard tag
Every dashboard already lives in the rippled/xrpld dashboard folder and
carries domain-specific tags (consensus, rpc, statsd, etc.), so the
leading "rippled" tag added no filtering value and only cluttered the
tag list.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 20:41:27 +01:00
Pratik Mankawde
fdfa651006 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
# Conflicts:
#	OpenTelemetryPlan/09-data-collection-reference.md
2026-07-03 20:01:44 +01:00
Pratik Mankawde
f81919af05 docs(telemetry): document tier template variables and label checks
- 09-data-collection-reference: add $service_name, $deployment_environment,
  $xrpl_network_type to the template-variable table; note they appear on all
  dashboards and point to the runbook's Deployment Tiers section.
- TESTING.md: add a Prometheus check verifying the tier labels
  (deployment_environment, xrpl_network_type, service_name) are present on
  metric series.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 19:56:15 +01:00
Pratik Mankawde
f55f8c59fc feat(telemetry): stamp service.name and network on native metrics
The native OTel metrics path hard-coded service.name="xrpld" and stamped
no network attribute, while traces stamped a configurable service.name
and xrpl.network.type. Metrics therefore could not be filtered by service
or network. Align the two paths:

- OTelCollector::New / OTelCollectorImp gain serviceName + networkType
  params. service.name uses the configured value (default "xrpld" when
  unset, preserving today's behavior); xrpl.network.type is stamped when
  provided. The key is a string literal because beast/insight sits below
  the telemetry module and cannot include its SpanNames const.
- CollectorManager reads service_name from [insight], falling back to the
  [telemetry] value, and receives the network type from the caller.
- Application derives the network type once via the shared
  telemetry::networkTypeFromId, now declared in Telemetry.h and moved out
  of an anonymous namespace so the trace and metric paths reuse a single
  0/1/2 -> mainnet/testnet/devnet mapping (no duplication).

Dashboards (5 system-* files): add $service_name, $deployment_environment,
$xrpl_network_type template variables and wire them into every panel query
that filters by $node.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 19:40:59 +01:00
Pratik Mankawde
3c0ac8f257 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
# Conflicts:
#	docker/telemetry/otel-collector-config.yaml
2026-07-03 19:31:40 +01:00
Pratik Mankawde
3d4119a3ba feat(telemetry): filter metrics by deployment tier across dashboards
Extend the deployment-tier tagging to the metrics path and expose it in
the dashboards so a single Grafana stack can be sliced by environment,
network, and service:

Collector (otel-collector-config.yaml):
- prometheus exporter: enable resource_to_telemetry_conversion so
  resource attributes (deployment.environment, xrpl.network.type,
  service.name, service.instance.id) become Prometheus labels.
- metrics pipeline: run resource/tier so statsd metrics are tagged too.
- spanmetrics: add the tier keys to the resource-metrics grouping key
  so per-tier series stay grouped separately.

Dashboards (5 files):
- add $service_name, $deployment_environment, $xrpl_network_type
  template variables, named to match their Prometheus labels.
- inject the matching label filters into every panel query that
  already filters by $node.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 19:30:53 +01:00
Pratik Mankawde
50c7d672d4 Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd
# Conflicts:
#	docker/telemetry/otel-collector-config.yaml
2026-07-03 19:12:11 +01:00
Pratik Mankawde
04df7fd92c Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-07-03 19:11:30 +01:00
Pratik Mankawde
d79a41dde6 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing 2026-07-03 19:11:30 +01:00
Pratik Mankawde
78f3595bc1 Merge branch 'pratik/otel-phase1c-rpc-integration' into pratik/otel-phase2-rpc-tracing 2026-07-03 19:11:30 +01:00
Pratik Mankawde
ebc355a465 feat(telemetry): tag traces with deployment tier at the collector
Add a resource/tier processor so each collector stamps the deployment
tier onto the signals it forwards, enabling a single Grafana stack to
hold data from many collectors and filter by tier:

- deployment.environment (upsert): the collector is authoritative for
  its environment (local/ci/test/prod).
- xrpl.network.type (insert): the xrpld node already stamps its own
  chain, so the collector only fills this when absent, never
  overwriting a node's real network.

Wired into the traces pipeline (the only pipeline on this branch);
later phases extend it to metrics and logs as those pipelines appear.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-03 19:10:56 +01:00
Pratik Mankawde
f08c098049 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-06-11 23:28:41 +01:00
Pratik Mankawde
bef40b37e7 fix(telemetry): use bare consensus_ledger_id tempo filter tag
The Tempo datasource search filter used the stale dotted xrpl.consensus.ledger_id
tag, but the consensus.round span emits the bare consensus_ledger_id key
(ConsensusSpanNames.h). The dotted tag matched no spans and failed the Rule C
naming check. Align it with L1.
2026-06-11 23:28:29 +01:00
Pratik Mankawde
2efc932165 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
Bring the hardened OTel naming check (Rule E fix + 71 tests) and the phase 1-6
convention work forward into phase 7 (native metrics).

Conflict resolution:
- 05-configuration-reference.md: kept phase-7's native-OTLP metrics story
  (server=otel, /v1/metrics) over phase-6's superseded StatsD scrape job, in
  prose form (no code block).
- ordering.txt/loops.txt: regenerated via generate.py (not hand-edited).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 18:32:21 +01:00
Pratik Mankawde
5705c4dc02 Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-06-11 17:49:55 +01:00
Pratik Mankawde
ffc197b914 fix(telemetry): use underscore attr names and SpanNames constants in Phase 4
- tempo.yaml: align consensus filter tags with emitted keys
  (consensus_mode, consensus_round, ledger_seq) instead of dotted form
- haveConsensus(): set span attributes before early-return paths so the
  consensus.check span carries diagnostics even when consensus is not reached
- replace hardcoded consensus phase/result/vote literals with
  ConsensusSpanNames.h val constants; add val::phaseOpen/Establish/Accepted
- ConsensusReceiveTracing.h: use canonical consensus::span constants instead
  of duplicate inline detail:: names
- SpanGuardFactory test: use rpc_span / consensus::span constants now that
  levelization permits the dependency

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-10 17:07:10 +01:00
Pratik Mankawde
4a0994209e Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-06-10 16:26:23 +01:00
Pratik Mankawde
f37589b1f5 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing 2026-06-10 16:05:29 +01:00
Pratik Mankawde
4ed409f35a addressed code review comments
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-06-10 16:05:04 +01:00
Pratik Mankawde
a7c3cd8a79 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-06-10 10:20:03 +01:00