Two live-verified dashboard defects found while auditing the devnet nodes.
Sub-millisecond spanmetrics buckets. The ladder started at 1ms, but most
xrpld spans are far below that: tx.preflight means ~0.012ms and tx.preclaim
~0.15ms. On the live nodes the first bucket held 13,761,009 of 13,761,419
tx.preflight samples (99.997%), so histogram_quantile pinned every p95/p99
at a constant 0.95ms and ~10 latency panels reported a boundary artifact
rather than a measurement. Adds 0.01/0.05/0.1/0.25/0.5ms ahead of 1ms in all
three bucket lists, which must stay identical or the connector misbuckets.
consensus_mode filter on spans that never carry the label. Only
consensus.round and consensus.ledger_close are stamped with consensus_mode
(verified against live series), so panels 1, 3, 12 and 14 -- which query
consensus.validation.send, consensus.proposal.send and
consensus.accept.apply -- blanked as soon as an operator picked any value
from the Consensus Mode dropdown. Drops the selector from those four; ids 2
and 11 keep it because their spans do carry it.
Note: existing histogram series retain the old le boundaries, so
span-latency panels show a discontinuity at collector restart.
Alloy declared 24 spanmetrics dimensions where both collector configs
declare 28. The four missing ones -- timed_out, object_type,
validation_status and accept_gated -- are emitted by the code and queried
by ledger-sync-health panels "Ledger Serve Rate by Object Type" and
"Trusted Validation Accept Rate by Status", so on an Alloy deployment
those panels collapsed to one undifferentiated series.
deploy-run-otel-xrpld.md already requires the dimension sets to stay in
sync; the config had drifted from it. Alloy's other differences from the
collectors (no filelog, no tail_sampling) are deliberate -- it is
cloud-only by design.
The Alloy spanmetrics connector (perf-iac nodes run Alloy instead of the
collector-contrib image) carried the same old [.. 1s, 5s] bucket list, whose
buckets are documented as "copied from" the collector config. Sync it to the
15-bucket layout (adds 2s, 3s, 4s, 10s, 30s) so Alloy-fed nodes get the same
accurate second-scale consensus and ledger.acquire percentiles as
collector-fed nodes. unit was already "ms". Kept strictly ascending.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The legend now uses raw labels in the dashboard Display name (see the paired
dashboard commit), so the collector no longer needs to build resource_group /
iac_group. Those derived labels also could not survive the dashboards' sum by()
aggregation, which is why the grouped approach was dropped.
Revert transform/legendgroups and its transform/cloudlabels datapoint copies
from the Grafana Cloud collector config and the in-repo Alloy config; both files
are now identical to their pre-change state.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update the config.alloy metric_statements comment to describe the current
bracket-baked, ", "-joined group-label values ("[node, mainnet]"), matching the
implementation. Comment-only; no behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror the local collector change on the Grafana Cloud collector config and the
in-repo Alloy config: build the group-label VALUES with brackets and comma
separators (resource_group="[node, mainnet]",
iac_group="[branch, role, work-item]") in both trace and metric resource
statements, so span-derived and native metrics carry the same bracketed values.
This lets the dashboards use a bracket-free Display name and avoids the empty
"[]" that a templated bracket would leave when a group is absent (Grafana's
renameByRegex runs pre-interpolation and cannot strip it). Separator changes
from "-" to ", ".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In the Alloy config the spanmetrics connector derives span_calls_total /
span_duration_* from the tagged TRACES and sends them straight to the batch
processor, bypassing the metric resource statements. The grouped legend labels
were built only in metric_statements, so span-derived RED metrics carried no
resource_group / iac_group and span-based dashboard panels rendered empty group
legends.
Build the same labels in trace_statements too (mirroring how the tier attrs are
already set in both), so spans carry them before spanmetrics derives from them.
The reference otel collector is unaffected: there spanmetrics is a connector
feeding the metrics pipeline, which runs transform/legendgroups on all inputs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The docker/telemetry/alloy/config.alloy node config feeds Grafana Cloud via the
OTLP gateway (which promotes resource attributes to labels server-side). Add the
resource_group / iac_group build to its metric resource statements so nodes
using this Alloy path get the same grouped legend labels the dashboards expect.
This is a generic node config (no perf-iac attrs), so iac_group's guard never
fires here and only resource_group is built; the iac_group statement is kept for
parity and activates automatically on a perf-comparison node. Guards check != ""
as well as != nil since OTTL Concat emits a stray separator for an empty element.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add an Alloy-based collector config as an alternative to the reference
OpenTelemetry Collector. It forwards xrpld OTLP traces and native metrics
to the Grafana Cloud OTLP gateway and, critically, runs a spanmetrics
connector so span-derived RED metrics (span_calls_total,
span_duration_milliseconds_*) are produced. Those metrics are not emitted
by xrpld; they are derived from spans, so an Alloy node without this
connector is missing from every span-based dashboard.
All credentials, endpoints, usernames and the per-node host label are read
from the environment (no secrets in the file). Adds a matching
.env.grafanacloud-alloy.example and gitignore exception, and the otelcol
term to the cspell dictionary.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
