Commit Graph

373 Commits

Author SHA1 Message Date
Pratik Mankawde
e0979a2080 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill
# Conflicts:
#	docker/telemetry/grafana/dashboards/transaction-overview.json
2026-07-10 18:42:46 +01:00
Pratik Mankawde
8ca7036a8b Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-10 18:36:48 +01:00
Pratik Mankawde
5f8ca9d84c Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
# Conflicts:
#	docker/telemetry/grafana/dashboards/transaction-overview.json
#	docs/telemetry-runbook.md
2026-07-10 18:36:39 +01:00
Pratik Mankawde
24542c46c5 feat(telemetry): TxQ accept applied-ratio state timeline
Replace the TxQ Accept Status piechart on the Transaction Overview
dashboard with a state-timeline showing each node's applied fraction of
TxQ accepts (applied / applied+failed) over time, colored by threshold
(green >=0.9, yellow >=0.7, red below). Remove the now-orphaned
txq_status template variable (the piechart was its only consumer) and
document the panel in the telemetry runbook.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:33:22 +01:00
Pratik Mankawde
b218763c8d fix(telemetry): repoint apply-failed KPI to tx.transactor (lost in merge)
The phase8->phase9 merge resolution took phase9's side of transaction-overview,
which reverted the Transaction Apply Failed Rate panel to the old
tx.apply/STATUS_CODE_ERROR query (constant zero). Re-apply the tx.transactor
stage=apply / ter_result!=tesSUCCESS repoint and raised thresholds.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:31:42 +01:00
Pratik Mankawde
efe1386440 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-07-10 18:25:11 +01:00
Pratik Mankawde
0c9001d058 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-10 18:25:05 +01:00
Pratik Mankawde
e357cf743d Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
# Conflicts:
#	docker/telemetry/grafana/dashboards/consensus-health.json
#	docker/telemetry/grafana/dashboards/ledger-operations.json
#	docker/telemetry/grafana/dashboards/peer-network.json
#	docker/telemetry/grafana/dashboards/rpc-performance.json
#	docker/telemetry/grafana/dashboards/transaction-overview.json
2026-07-10 18:24:53 +01:00
Pratik Mankawde
251b214f7b fix(telemetry): seed $node dashboard filter from target_info
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:22:02 +01:00
Pratik Mankawde
086c85fb2c fix(telemetry): seed $node dashboard filter from target_info
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:21:59 +01:00
Pratik Mankawde
a998b80f7b fix(telemetry): seed $node dashboard filter from target_info
Seed the $node template variable from target_info instead of a
signal-specific metric. Signal metrics only list nodes that emit that
specific signal, so nodes (e.g. Alloy-collected ones) were missing from
the dropdown and every panel. target_info is emitted for every node, so
all nodes are always selectable. Panels already filter on
service_instance_id=~"$node", so no panel changes are needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 18:21:56 +01:00
Pratik Mankawde
eb502d6eea Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill
# Conflicts:
#	docker/telemetry/grafana/dashboards/consensus-health.json
#	docker/telemetry/grafana/dashboards/ledger-operations.json
#	docker/telemetry/grafana/dashboards/node-health.json
#	docker/telemetry/grafana/dashboards/peer-network.json
#	docker/telemetry/grafana/dashboards/rpc-pathfinding.json
#	docker/telemetry/grafana/dashboards/rpc-performance.json
#	docker/telemetry/grafana/dashboards/transaction-overview.json
#	docs/telemetry-runbook.md
2026-07-10 17:28:57 +01:00
Pratik Mankawde
4f2b7cfb45 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-10 17:19:01 +01:00
Pratik Mankawde
851bc75dfc feat(telemetry): wire jq_trans_overflow_total as observable counter
Observe the overflow total from Overlay::getJqTransOverflow() via an
ObservableCounter instead of the unused push helper (which had no call
site, so the metric read flat zero). Keeps the increment path in the
overlay-owned code and removes the dead incrementJqTransOverflow() helper.
Update the job-queue panel to reflect the now-live metric, and refresh
stale xrpld_-prefixed metric names in the header comments.

NOTE: committed without a local build per instruction; needs compile +
clang-tidy verification before push.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 17:16:35 +01:00
Pratik Mankawde
3143216ad1 docs(telemetry): flag RPC Response Size instrument mismatch pending byte-histogram fix
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 17:00:13 +01:00
Pratik Mankawde
535a95363f fix(telemetry): use $__rate_interval on phase9 rate panels
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:59:19 +01:00
Pratik Mankawde
9820ae8e38 fix(telemetry): use $__rate_interval + min step on all rate panels
Replace hardcoded [5m] rate windows with [$__rate_interval] inside
rate()/increase() on plain-counter panels so ops/s panels resample
correctly on zoom-out instead of under-sampling or flat-lining on
short windows. Histogram _bucket windows keep their fixed [5m] span.
Set Min step ("interval": "15s") on every rate target to match the
Prometheus 15s scrape interval.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:51:53 +01:00
Pratik Mankawde
d7ecdacbdf docs(telemetry): flag uninstrumented tx-overflow panel; use rate-interval
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:39:26 +01:00
Pratik Mankawde
0fbe7f4e4f fix(telemetry): remove misleading stacked load-factor attribution panel
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:39:24 +01:00
Pratik Mankawde
40d49e3969 fix(telemetry): rate() nodestore panels + cache-hit ratio, dual-axis last-close, ledger-age trend, operating-mode state timeline
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:39:22 +01:00
Pratik Mankawde
ba9ac50559 fix(telemetry): drop duplicate peer-tx panel, repoint apply-failed KPI to tx.transactor
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:22:47 +01:00
Pratik Mankawde
b0f4370397 fix(telemetry): show operating mode as % time-in-mode on phase7 node-health
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:18:06 +01:00
Pratik Mankawde
a90d51c808 fix(telemetry): render phase7 traffic dashboards as per-second rates
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 16:06:28 +01:00
Pratik Mankawde
989c420a7a Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill
# Conflicts:
#	OpenTelemetryPlan/09-data-collection-reference.md
#	docker/telemetry/grafana/dashboards/consensus-health.json
#	docker/telemetry/grafana/dashboards/ledger-operations.json
#	docker/telemetry/grafana/dashboards/node-health.json
#	docker/telemetry/grafana/dashboards/rpc-pathfinding.json
#	docker/telemetry/grafana/dashboards/rpc-performance.json
#	docker/telemetry/grafana/dashboards/transaction-overview.json
#	docs/telemetry-runbook.md
2026-07-10 15:35:34 +01:00
Pratik Mankawde
982e3cf9ef Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-10 14:31:07 +01:00
Pratik Mankawde
86bbe8f55b fix(telemetry): rename spanmetrics to span_ namespace on phase7 (dashboards + collector config)
The spanmetrics connector had no namespace, so it emitted traces_span_metrics_*
metric names by default. The span dashboards and docs are renamed to query
span_* names; this is only correct if the connector emits them too, so add
namespace: "span" to the spanmetrics connector. Both sides change together:
renaming the dashboards without the namespace (or vice versa) would break the
pipeline. Matches the phase9 collector config.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 14:30:01 +01:00
Pratik Mankawde
34a237fab6 fix(telemetry): strip xrpld_ prefix and snake_case native dashboard/doc metric names
The phase7 OTelCollector::formatName lowercases and strips names, emitting
snake_case metrics with no xrpld_ prefix. The native Grafana dashboards and
the telemetry docs still queried the old xrpld_CamelCase names, so they were
broken against their own pipeline. Rename every metric name to match what the
code emits: drop the xrpld_ prefix and lowercase the remainder. The two job
histograms also drop the redundant 'duration' word (job_queued_us,
job_running_us) to match the phase9 forms. Add havetxset to the cspell
dictionary since the lowercased metric name no longer word-splits.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 14:07:29 +01:00
Pratik Mankawde
d174d4a7d8 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-10 13:40:42 +01:00
Pratik Mankawde
6ae6e623a7 test(telemetry): add dashboard lint for raw-cumulative panels
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-10 13:21:52 +01:00
Pratik Mankawde
74a4a4ab7a refactor(telemetry): default the $service_name dashboard filter to 'xrpld'
Completes the default change applied on phase-6-statsd and
phase-7-native-metrics. The four dashboards introduced on phase-9
(validator-health, peer-quality, fee-market, job-queue) now also
pre-select 'xrpld' rather than 'All' for the service_name filter.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-09 17:22:56 +01:00
Pratik Mankawde
6530e7d76c Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-09 17:22:12 +01:00
Pratik Mankawde
e5b2ce1696 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-07-09 17:22:12 +01:00
Pratik Mankawde
dae3f6e4d1 refactor(telemetry): default the $service_name dashboard filter to 'xrpld'
Continuation of the same default change applied on phase-6-statsd
(50d7047abe): the 5 dashboards introduced on phase-7 (formerly
system-*, renamed to bare names on this branch) also get their
service_name filter pre-selected to 'xrpld' instead of 'All'.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-09 17:22:01 +01:00
Pratik Mankawde
f2d1ac5c78 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-09 17:21:40 +01:00
Pratik Mankawde
50d7047abe refactor(telemetry): default the $service_name dashboard filter to 'xrpld'
Change the pre-selected value of the service_name template variable from
'All' to 'xrpld'. On first load, dashboards now filter to service.name=xrpld
directly rather than showing every service the collector sees. The 'All'
option is still selectable — this only changes the default the widget
starts on.

Reduces the chance of a fresh dashboard link showing unrelated services
(e.g. non-xrpld exporters sharing the same Grafana Cloud stack) before
the user narrows the filter.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-09 17:21:24 +01:00
Pratik Mankawde
a835fbb297 refactor(telemetry): rename dashboard template variable label from 'Service' to 'Service Name'
Completes the rename applied on phase-6-statsd and phase-7-native-metrics.
These four dashboards were introduced on phase-9: peer-quality and
validator-health via b92354715d, fee-market and job-queue via 25868f2740.

Same reasoning: the underlying Prometheus label is `service_name`,
so the UI display label should say 'Service Name' rather than the
ambiguous 'Service'.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-09 17:04:23 +01:00
Pratik Mankawde
b2d3b31169 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-07-09 17:04:06 +01:00
Pratik Mankawde
4ed9bd595d Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-09 17:03:56 +01:00
Pratik Mankawde
bab0c84793 refactor(telemetry): rename dashboard template variable label from 'Service' to 'Service Name'
Continuation of the same rename applied on phase-6-statsd (commit
91c6582bd5): 5 more dashboards introduced on phase-7 (formerly the
system-* set, renamed to bare names by 2c590a47c5) still carried the
older 'Service' UI label. Aligning them with the rest.

Same reasoning as the phase-6 commit: the underlying Prometheus label
is `service_name`, and the UI label 'Service' misled users into typing
`service=xrpld` in Grafana Metrics Drilldown.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-09 17:03:47 +01:00
Pratik Mankawde
46cff27b15 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-09 17:03:35 +01:00
Pratik Mankawde
91c6582bd5 refactor(telemetry): rename dashboard template variable label from 'Service' to 'Service Name'
The template variable's Prometheus label name is `service_name` but the
UI display label read 'Service', which mismatched what users typed into
Grafana's Metrics Drilldown UI (which defaults to filter on a bare
`service` label). Rename the display label to 'Service Name' so it
unambiguously points at the underlying `service_name` Prometheus label.

Behaviour unchanged; template-variable name (`service_name`) and PromQL
label are the same. Only the widget label shown at the top of the
dashboard changes from 'Service' to 'Service Name'.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-07-09 17:03:15 +01:00
Pratik Mankawde
dd9635684a fix(telemetry): strip xrpld_ prefix from MetricsRegistry, rename spanmetrics namespace
Remove redundant xrpld_ prefix from all 34 MetricsRegistry instrument
names — service identity comes from the OTel resource, not the metric
name. Duration histograms retain their unit suffix (job_queued_duration_us,
job_running_duration_us, rpc_method_duration_us).

Set spanmetrics connector namespace to "span" so derived metrics become
span_calls_total and span_duration_milliseconds_bucket instead of the
verbose traces_span_metrics_ prefix.

Update all 14 dashboard JSONs: strip xrpld_ prefix, lowercase
beast::insight metric names, replace traces_span_metrics_ with span_.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-08 17:49:29 +01:00
Pratik Mankawde
a681eea6c4 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-08 15:26:06 +01:00
Pratik Mankawde
a1a9d45bff Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-08 15:22:52 +01:00
Pratik Mankawde
4826038068 fix(telemetry): run rename scripts for CI compliance
Rename rippled → xrpld in statsd config prefix.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-08 15:22:45 +01:00
Pratik Mankawde
21acd060c8 fix(telemetry): use service_instance_id label instead of exported_instance
Merge forward from phase6/7 and apply the same fix to phase9's own
dashboards and alerting rules. exported_instance is a Prometheus scrape
artifact absent in push-based pipelines; service_instance_id is always
present via resource_to_telemetry_conversion.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-08 14:52:55 +01:00
Pratik Mankawde
a3053a34a7 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-07-08 14:51:55 +01:00
Pratik Mankawde
87f370a64d fix(telemetry): use service_instance_id label instead of exported_instance
exported_instance is a Prometheus scrape artifact that only exists when
Prometheus scrapes a target with a conflicting instance label. Users
running push-based pipelines (remote_write, OTLP) never get this label.
service_instance_id is always present via resource_to_telemetry_conversion
regardless of ingestion path.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-08 14:51:48 +01:00
Pratik Mankawde
ff92c3727c Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-07-08 14:51:36 +01:00
Pratik Mankawde
1eb44689d0 fix(telemetry): use service_instance_id label instead of exported_instance
exported_instance is a Prometheus scrape artifact that only exists when
Prometheus scrapes a target with a conflicting instance label. Users
running push-based pipelines (remote_write, OTLP) never get this label.
service_instance_id is always present via resource_to_telemetry_conversion
regardless of ingestion path.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-08 14:51:26 +01:00