Grafana Cloud's OTLP ingest already promotes the `service.name` resource
attribute to a `service_name` metric label. The `transform/cloudlabels`
processor was ALSO setting `service_name` as a datapoint attribute,
which Mimir treats as a duplicate label and concatenates as
`service_name="xrpld;xrpld"` on every series. Dashboards filtering on
`service_name=xrpld` then returned no data.
Remove the redundant `set(attributes["service_name"], ...)` line and
add a comment documenting the doubling. `service.instance.id` does not
exhibit the same behaviour, so that copy is retained for the $node
filter to keep resolving.
Reintroduces the pre-e76eed5296 shape (which was correct on this label)
without reverting that commit's other cloud-label fixes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Completes the rename applied on phase-6-statsd and phase-7-native-metrics.
These four dashboards were introduced on phase-9: peer-quality and
validator-health via b92354715d, fee-market and job-queue via 25868f2740.
Same reasoning: the underlying Prometheus label is `service_name`,
so the UI display label should say 'Service Name' rather than the
ambiguous 'Service'.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Continuation of the same rename applied on phase-6-statsd (commit
91c6582bd5): 5 more dashboards introduced on phase-7 (formerly the
system-* set, renamed to bare names by 2c590a47c5) still carried the
older 'Service' UI label. Aligning them with the rest.
Same reasoning as the phase-6 commit: the underlying Prometheus label
is `service_name`, and the UI label 'Service' misled users into typing
`service=xrpld` in Grafana Metrics Drilldown.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The template variable's Prometheus label name is `service_name` but the
UI display label read 'Service', which mismatched what users typed into
Grafana's Metrics Drilldown UI (which defaults to filter on a bare
`service` label). Rename the display label to 'Service Name' so it
unambiguously points at the underlying `service_name` Prometheus label.
Behaviour unchanged; template-variable name (`service_name`) and PromQL
label are the same. Only the widget label shown at the top of the
dashboard changes from 'Service' to 'Service Name'.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Strip xrpld_ prefix, lowercase beast::insight names, and replace
traces_span_metrics_ with span_ in all remaining tracked files:
alert rules, integration tests, workload validation, TESTING.md,
OpenTelemetryPlan docs, code comments, and config templates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove redundant xrpld_ prefix from all 34 MetricsRegistry instrument
names — service identity comes from the OTel resource, not the metric
name. Duration histograms retain their unit suffix (job_queued_duration_us,
job_running_duration_us, rpc_method_duration_us).
Set spanmetrics connector namespace to "span" so derived metrics become
span_calls_total and span_duration_milliseconds_bucket instead of the
verbose traces_span_metrics_ prefix.
Update all 14 dashboard JSONs: strip xrpld_ prefix, lowercase
beast::insight metric names, replace traces_span_metrics_ with span_.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add service_name to transform/cloudlabels so Grafana Cloud dashboards
can filter by service.name (fixes empty panels for native metrics).
Set spanmetrics namespace to "span" in the cloud collector config and
update dashboards + runbook: traces_span_metrics_ → span_.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Raise TX error rate threshold from 50% to 95%. Short-lived CI test
environments lack pre-funded accounts for complex transactions
(AMMCreate, EscrowFinish, NFTokenCreateOffer, etc.), causing expected
failures that do not indicate an instrumentation problem.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update the transform/cloudlabels processor to set service_instance_id
instead of exported_instance, matching the local Prometheus path where
resource_to_telemetry_conversion promotes service.instance.id with that
exact label name. Also update source comments and documentation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merge forward from phase6/7 and apply the same fix to phase9's own
dashboards and alerting rules. exported_instance is a Prometheus scrape
artifact absent in push-based pipelines; service_instance_id is always
present via resource_to_telemetry_conversion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
exported_instance is a Prometheus scrape artifact that only exists when
Prometheus scrapes a target with a conflicting instance label. Users
running push-based pipelines (remote_write, OTLP) never get this label.
service_instance_id is always present via resource_to_telemetry_conversion
regardless of ingestion path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
exported_instance is a Prometheus scrape artifact that only exists when
Prometheus scrapes a target with a conflicting instance label. Users
running push-based pipelines (remote_write, OTLP) never get this label.
service_instance_id is always present via resource_to_telemetry_conversion
regardless of ingestion path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The workload validator queried each expected metric once, immediately after
a fixed post-workload propagation wait. Several beast::insight metrics
(ledger-age and peer-finder gauges, overlay-traffic and rpc-request counters)
only populate after the node validates ledgers and sustains peer traffic,
then travel a 1s OTLP export + 15s Prometheus scrape before they are
queryable. On a slower CI runner that pipeline can settle after the wait
ends, so the single query raced and reported "0 series", failing 12 checks
that pass locally with the same config and binary.
Poll each metric on the /api/v1/series endpoint until it appears or a 45s
window (two scrape cycles) elapses. Present metrics still return on the first
query with no added delay; a genuinely-absent metric still fails after the
timeout. Makes the check robust to runner speed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
xrpld resolves a relative [debug_logfile] against the config file's own
directory (docker/telemetry), not the working directory. The value
`docker/telemetry/data/logs/devnet/debug.log` therefore doubled the
prefix to docker/telemetry/docker/telemetry/... — the collector's mount
never saw the log, so no lines reached Loki.
Use the config-dir-relative form `data/logs/devnet/debug.log`, which
resolves to docker/telemetry/data/logs/devnet/debug.log — exactly the
dir the compose stack bind-mounts as /var/log/xrpld.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extend the log-mount consolidation to the phase-10 workload-validation
assets, which still used the old `rippled` container path:
- docker-compose.workload.yaml: mount target renamed to /var/log/xrpld,
source made overridable via XRPLD_LOG_DIR (default /tmp/xrpld-validation)
- otel-collector-config.grafanacloud.yaml: single glob /var/log/xrpld/*/debug.log
- run-full-validation.sh sets XRPLD_LOG_DIR to its workdir
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The collector's log ingestion used a container path named after the old
`rippled` binary and defaulted its host mount to a location that either
needed root or diverged from where the telemetry configs actually write.
Consolidate on one accessible log root:
- container mount target renamed /var/log/rippled -> /var/log/xrpld
- filelog glob -> /var/log/xrpld/*/debug.log
- compose mount source defaults to the repo-relative ./data/logs
(user-owned, no root), overridable via XRPLD_LOG_DIR
- devnet telemetry cfg writes to data/logs/devnet/debug.log so it lands
one subdir below the mount root and matches the glob
- integration-test.sh sets XRPLD_LOG_DIR to its own workdir
- docs/runbook/task-list updated to match
The default xrpld config (cfg/xrpld-example.cfg) is intentionally left
untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Correct log-timestamp examples to real Logs::format() output
(2024-Jan-15 10:30:45.123456 UTC) in docs, TESTING.md, and reference.
- Update Loki to v3.4.2 and switch the collector exporter to
otlphttp/loki on the native /otlp endpoint (docs + task list).
- Make the collector log-mount path configurable via XRPLD_LOG_DIR.
- Remove implementation-phase references from shipped config/script
comments, keeping the functional descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the phase-8-owned 'Phase 8' tokens from
loki.yaml and tempo.yaml, keeping the functional descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 3/4" token, keep the description.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 4" tokens, keep descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 3" tokens, keep descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 2" tokens, keep descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 1b" token, keep the filter
description.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
