Add $service_name, $deployment_environment, $xrpl_network_type template
variables to the metric-gap-fill dashboards and wire them into every
panel query that filters by $node, so these dashboards can be sliced by
tier like the rest of the set.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add resource/tier to the logs pipeline so log records carry the same
deployment.environment and xrpl.network.type attributes as traces and
metrics. This lets a single Grafana stack filter logs by tier alongside
the other signals. resource/logs still runs first to set service.name
and the job label; resource/tier then adds the tier attributes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The native OTel metrics path hard-coded service.name="xrpld" and stamped
no network attribute, while traces stamped a configurable service.name
and xrpl.network.type. Metrics therefore could not be filtered by service
or network. Align the two paths:
- OTelCollector::New / OTelCollectorImp gain serviceName + networkType
params. service.name uses the configured value (default "xrpld" when
unset, preserving today's behavior); xrpl.network.type is stamped when
provided. The key is a string literal because beast/insight sits below
the telemetry module and cannot include its SpanNames const.
- CollectorManager reads service_name from [insight], falling back to the
[telemetry] value, and receives the network type from the caller.
- Application derives the network type once via the shared
telemetry::networkTypeFromId, now declared in Telemetry.h and moved out
of an anonymous namespace so the trace and metric paths reuse a single
0/1/2 -> mainnet/testnet/devnet mapping (no duplication).
Dashboards (5 system-* files): add $service_name, $deployment_environment,
$xrpl_network_type template variables and wire them into every panel query
that filters by $node.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extend the deployment-tier tagging to the metrics path and expose it in
the dashboards so a single Grafana stack can be sliced by environment,
network, and service:
Collector (otel-collector-config.yaml):
- prometheus exporter: enable resource_to_telemetry_conversion so
resource attributes (deployment.environment, xrpl.network.type,
service.name, service.instance.id) become Prometheus labels.
- metrics pipeline: run resource/tier so statsd metrics are tagged too.
- spanmetrics: add the tier keys to the resource-metrics grouping key
so per-tier series stay grouped separately.
Dashboards (5 files):
- add $service_name, $deployment_environment, $xrpl_network_type
template variables, named to match their Prometheus labels.
- inject the matching label filters into every panel query that
already filters by $node.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a resource/tier processor so each collector stamps the deployment
tier onto the signals it forwards, enabling a single Grafana stack to
hold data from many collectors and filter by tier:
- deployment.environment (upsert): the collector is authoritative for
its environment (local/ci/test/prod).
- xrpl.network.type (insert): the xrpld node already stamps its own
chain, so the collector only fills this when absent, never
overwriting a node's real network.
Wired into the traces pipeline (the only pipeline on this branch);
later phases extend it to metrics and logs as those pipelines appear.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Apply the same width rule from the earlier transaction-overview fix to the
job-queue, rpc-perf-otel, and rpc-performance dashboards. Timeseries panels
whose legend is a right-side table need the horizontal room, so they go to
full width (w=24); each stacks onto its own row and panels below shift down.
Pie charts, heatmaps, and bottom-legend panels stay half width.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Tempo datasource search filter used the stale dotted xrpl.consensus.ledger_id
tag, but the consensus.round span emits the bare consensus_ledger_id key
(ConsensusSpanNames.h). The dotted tag matched no spans and failed the Rule C
naming check. Align it with L1.
Bring phase-8 (log correlation) and the naming-check + BasicConfig fixes forward
into phase 9 (native-metrics gap fill). Phase 9 adds MetricsRegistry and
LedgerSpanNames.h.
Conflict resolution:
- presentation.md: took phase-9's restructured deck — sampling now lives in
Slide 8 (Head vs Tail) and the Phase 10/11/12 future-phase slides; phase-8's
older standalone sampling section is superseded (no content lost).
- ordering.txt/loops.txt: regenerated via generate.py.
Naming check: A/B/E/F/G green. Rule D reports 6 findings that are all check
gaps, not data errors (verified every dashboard label traces to code):
- 3 are TraceQL `span.<attr>` references (bare attr is in L1); Rule D should
strip the span./resource. scope prefix.
- 2 (`reason`, `job_type`) are native OTel metric labels emitted by
MetricsRegistry.cpp — a new source of truth the check does not yet model.
The Rule D enhancement lands on phase-1c and merges forward.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bring native-metrics (phase 7) and the naming-check work forward into phase 8
(log correlation).
Conflict resolution — tempo.yaml datasource provisioning (search.filters):
- Took phase-7's complete 30-filter list as the base (it is the comprehensive
set), preserved phase-8's tracesToLogs Loki-correlation block, and added
phase-8's two unique filters (tx-type, ledger-hash). Verified the result is
the full union (32 filters) with no filter dropped and no duplicate ids.
- Fixed a pre-existing dotted tag carried in from phase-7: the
consensus-ledger-id filter used `xrpl.consensus.ledger_id`; the code emits
the underscore key `consensus_ledger_id` (ConsensusSpanNames.h), so the
dotted form was a dead filter. Now corrected.
Every span-scope filter tag verified to exist in the L1 *SpanNames.h key set.
Naming check green; 72 tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bring the hardened OTel naming check (Rule E fix + 71 tests) and the phase 1-6
convention work forward into phase 7 (native metrics).
Conflict resolution:
- 05-configuration-reference.md: kept phase-7's native-OTLP metrics story
(server=otel, /v1/metrics) over phase-6's superseded StatsD scrape job, in
prose form (no code block).
- ordering.txt/loops.txt: regenerated via generate.py (not hand-edited).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- tempo.yaml: align consensus filter tags with emitted keys
(consensus_mode, consensus_round, ledger_seq) instead of dotted form
- haveConsensus(): set span attributes before early-return paths so the
consensus.check span carries diagnostics even when consensus is not reached
- replace hardcoded consensus phase/result/vote literals with
ConsensusSpanNames.h val constants; add val::phaseOpen/Establish/Accepted
- ConsensusReceiveTracing.h: use canonical consensus::span constants instead
of duplicate inline detail:: names
- SpanGuardFactory test: use rpc_span / consensus::span constants now that
levelization permits the dependency
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Head sampling is fixed at 1.0 and not configurable. Remove the
sampling_ratio line and its stale tuning comment; point volume reduction
at collector-side tail sampling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Head sampling is fixed at 1.0 and not configurable. Drop sampling_ratio
from the docker sample/test configs and the data-collection reference;
rewrite the span-metric sampling caveat and volume guidance to point at
collector-side tail sampling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update the docker/telemetry sample config and testing guide to set
trace_peer=1, matching the new on-by-default behavior and the other
trace categories already enabled in these all-categories sample profiles.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Several close-time panels showed raw codes/booleans and unreadable
time-bar-charts. Relabel and re-visualize them so each reads clearly:
- Close-Time Proposal Spread (was "Close Time Bin Distribution"):
corrected a wrong description (it claimed per-node proposals; the
metric is the number of DISTINCT close-time positions peers proposed
per round, rawCloseTimes.peers.size()). Converted the time-barchart
(unreadable timestamp axis) to a horizontal bar gauge summing rounds
per distinct-position count.
- Consensus Outcome Distribution: renameByRegex maps the raw
consensus_state codes to human labels (yes->Agreed,
moved_on->Moved On (partial), expired->Expired (timeout), no->No
Consensus); value mappings alone do not relabel pie legends.
- Close-Time Agreement Rate (was "Close Time Agreement"): legend
relabelled from "close_time_correct=true/false" to Agreed/Disagreed.
- Close-Time Resolution Change (was "Close Time Resolution Direction"):
converted to a bar gauge; increased/decreased/unchanged relabelled to
Coarser (more disagreement) / Finer (better agreement) / Steady.
All four verified by rendering the panels to PNG via the Grafana
image renderer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
"Close Time: Raw Proposals" and "Close Time: Effective / Quantized"
plotted the absolute close time (Ripple-epoch seconds), which is an
ever-rising line with no analytical value. There is no clean way to make
them useful: the values are Ripple-epoch seconds (not Unix ms, so date
units misrender), TraceQL metrics cannot do the epoch offset or an
inter-ledger-gap subtraction in-query, and Grafana calculateField
transforms break on these grouped Tempo metric frames (verified by
render: "No data").
Remove both. The useful consensus-timing signals are already covered:
time-to-consensus (round_time_ms), rounds per ledger (establish_count),
previous round time, and the close-time vote-bins / resolution-direction
/ bin-distribution panels remain. Re-tile and re-id (22 panels).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Number of rounds is an integer, but avg_over_time(establish_count)
produced fractional values (2.1). Switch the Rounds panel to
count_over_time() by (span.establish_count): one integer series per
round count (1/2/3...), showing how many ledgers needed that many
establish rounds — the meaningful distribution, inherently integer
(decimals=0).
Apply dashboard rule 9: panels with a right-side table legend take full
width. Widen "Consensus Rounds per Ledger" and "Consensus Outcome
Distribution" to w=24 and re-tile the dashboard.
Verified via the Grafana proxy: rounds=2 dominates (~11-12 ledgers),
rounds=3 occasional.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 9 introduces the ledger.acquire span (InboundLedger fetch) that phases 7-8
do not have, so the forward-merged 09-data-collection-reference inventory is
extended here:
- §1.1: add ledger.acquire to the Ledger span table.
- §1.2: add its attributes (acquire_reason, timeouts, peer_count, outcome) and
note it also sets ledger_seq; bump the span count.
Also fix two stale StatsD metric references in the Peer Quality dashboard
(xrpld-peer-quality.json): rippled_Peer_Finder_Active_{Inbound,Outbound}_Peers
-> xrpld_Peer_Finder_* to match the xrpld_ metric prefix the rest of the stack uses.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The consensus duration panels plotted span wall-clock
(traces_span_metrics_duration_milliseconds), which is ~3-8 ms of
instrumentation overhead, not the real consensus time (~3000 ms). And
the close-time value panels plotted an ever-rising absolute epoch line.
Rework them to answer the actual operational questions, all from
attributes that already exist on the consensus spans:
- Time to Reach Consensus (p50/p95) and Average Time to Reach Consensus:
round_time_ms on consensus.accept — the wall-clock to agree a ledger.
- Consensus Rounds per Ledger (Establish Count): avg and max of
establish_count on consensus.establish — how many proposal rounds it
took to converge (1 = first proposal).
- Previous Round Time per Ledger: previous_round_time_ms on
consensus.round.
Reorder the dashboard into an investigation flow: health/throughput ->
time-to-consensus and rounds -> ledger close/apply timing -> close-time
detail -> failures/mode/mismatch. Assign stable sequential panel ids.
Verified each query returns data via the Grafana datasource proxy
(p95 ~4096 ms, avg ~2825 ms, rounds ~2).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Apply the dashboard guidelines to the five close-time panels:
- Axis labels (Title Case) on every panel: "Close Time (Ripple
Seconds)" for the value panels, "Count / Milliseconds" for vote
bins/resolution, "Rounds in Window" for the count panels.
- Human-readable legends with the dimension in brackets per the legend
convention: "Raw Close Time [{{resource.service.instance.id}}]",
"Effective Close Time [...]", "Resolution Direction
[{{span.resolution_direction}}]", "{{span.close_time_vote_bins}} Vote
Bins" — replacing the bare label tokens.
- Unit "none" (plain number): the close-time values are Ripple-epoch
seconds and TraceQL metrics cannot offset them to a wall-clock unit,
and the others are counts/ms on a shared axis.
Verified rendered values against raw spans: close times ~833,998,8xx,
resolution 10000 ms, vote bins 1/2/3 — all correct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TraceQL metrics queries default to a 3h max range
(query_frontend.metrics.max_duration), so a dashboard set to a longer
window failed with "range ... exceeds 3h0m0s". Add a query_frontend
block raising it to 168h, matching the search max_duration, so the
consensus close-time panels work at 6h/12h/24h ranges.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Raw Proposals and Effective/Quantized panels rendered nameless
series: their legendFormat used {{service.instance.id}}, but the
TraceQL metrics query groups by resource.service.instance.id and Tempo
returns that full key as the series label. The legend token did not
match any label, so each series showed blank. Use the matching
{{resource.service.instance.id}} token.
Verified via the Grafana datasource proxy that all six close-time panels
now return correctly-labelled series.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Raw Proposals and Effective/Quantized panels showed wrong values
(e.g. 759M, 852M, even 0) against a true value of ~834M. Cause:
quantile_over_time bucketizes into an exponential histogram tuned for
duration distributions, so it cannot represent large absolute integers
(Ripple-epoch seconds) accurately.
Switch both panels to avg_over_time, which returns the correct value
(verified ~833,996,7xx matching the raw span attribute). Average is also
the semantically right aggregation here: close time is a single agreed
value per consensus round, not a latency distribution, so a median was
never meaningful.
Set the unit to none rather than seconds: the value is Ripple-epoch
seconds (Unix = value + 946684800) and TraceQL metrics cannot do the
offset arithmetic in-query, so a duration unit would misrender it.
Clarify in the description that the absolute level tracks wall-clock and
the useful signal is per-node spread / raw-vs-effective gap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The five Close Time panels still rendered "No Data" after the metrics
rewrite. Root cause: each query carried
`span.close_time_correct=~"$close_time_correct"`, but close_time_correct
is a boolean span attribute and TraceQL's regex match (=~) against a
bool matches nothing in a metrics query, so every panel returned an
empty series set (HTTP 200, {"series":[]}).
Remove that filter clause. The panels do not break down by
close_time_correct, so dropping it restores data without losing any
dimension. The $node filter (a string attribute) is unaffected and
stays.
Verified via the Grafana datasource proxy that all six targets now
return series.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The five Close Time panels (Raw Proposals, Effective/Quantized, Vote
Bins & Resolution, Resolution Direction, Bin Distribution) rendered
empty: they used TraceQL `| select(attr)`, which returns a trace list
that a timeseries/barchart panel cannot plot.
Enable TraceQL metrics in Tempo and rewrite the panels to use it:
- tempo.yaml: add the local-blocks processor to the metrics generator so
recent blocks are queryable via /api/metrics/query_range. Set
filter_server_spans=false because the consensus spans are
SPAN_KIND_INTERNAL (the default keeps only server spans, so attribute
aggregations over internal spans returned nothing), and
flush_to_storage=true with a traces_storage path so query_range can
read the flushed blocks.
- consensus-health.json: replace each panel's select() with a metrics
query — quantile_over_time on the integer close-time attributes,
avg_over_time for vote bins / resolution, and count_over_time by the
resolution_direction and vote-bin dimensions. Set the raw/effective
panels' unit to seconds (the values are Ripple-epoch seconds, which
dateTimeFromNow rendered with the wrong epoch).
Verified the query forms compile and return series against live internal
spans; the close-time series populate once the node reaches full sync.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Current Job Latency gauge sat at the bottom of the Job Queue
Analysis dashboard; per the dashboard guideline gauges belong at the
top. Move it to the first row and reflow the remaining panels below it.
Also assign explicit sequential panel ids so deep links stay stable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
