Flip the tracePeer Setup default and the trace_peer config parser
default from off to on, and update the example config and build doc to
match. Peer spans record only peer_id (a node-local numeric connection
id) plus trust/ledger metadata — no IP addresses or public keys — so
the privacy concern behind disabling it does not apply. The high-volume
characteristic is retained in the docs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Flip trace_peer default false->true across the Phase-1a plan docs and
correct the rationale: peer spans record only peer_id (numeric local
connection id) plus trust/ledger metadata, never IP addresses or public
keys, so the 'includes addresses' caveat was inaccurate. The high-volume
note is retained.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Several close-time panels showed raw codes/booleans and unreadable
time-bar-charts. Relabel and re-visualize them so each reads clearly:
- Close-Time Proposal Spread (was "Close Time Bin Distribution"):
corrected a wrong description (it claimed per-node proposals; the
metric is the number of DISTINCT close-time positions peers proposed
per round, rawCloseTimes.peers.size()). Converted the time-barchart
(unreadable timestamp axis) to a horizontal bar gauge summing rounds
per distinct-position count.
- Consensus Outcome Distribution: renameByRegex maps the raw
consensus_state codes to human labels (yes->Agreed,
moved_on->Moved On (partial), expired->Expired (timeout), no->No
Consensus); value mappings alone do not relabel pie legends.
- Close-Time Agreement Rate (was "Close Time Agreement"): legend
relabelled from "close_time_correct=true/false" to Agreed/Disagreed.
- Close-Time Resolution Change (was "Close Time Resolution Direction"):
converted to a bar gauge; increased/decreased/unchanged relabelled to
Coarser (more disagreement) / Finer (better agreement) / Steady.
All four verified by rendering the panels to PNG via the Grafana
image renderer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
"Close Time: Raw Proposals" and "Close Time: Effective / Quantized"
plotted the absolute close time (Ripple-epoch seconds), which is an
ever-rising line with no analytical value. There is no clean way to make
them useful: the values are Ripple-epoch seconds (not Unix ms, so date
units misrender), TraceQL metrics cannot do the epoch offset or an
inter-ledger-gap subtraction in-query, and Grafana calculateField
transforms break on these grouped Tempo metric frames (verified by
render: "No data").
Remove both. The useful consensus-timing signals are already covered:
time-to-consensus (round_time_ms), rounds per ledger (establish_count),
previous round time, and the close-time vote-bins / resolution-direction
/ bin-distribution panels remain. Re-tile and re-id (22 panels).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Number of rounds is an integer, but avg_over_time(establish_count)
produced fractional values (2.1). Switch the Rounds panel to
count_over_time() by (span.establish_count): one integer series per
round count (1/2/3...), showing how many ledgers needed that many
establish rounds — the meaningful distribution, inherently integer
(decimals=0).
Apply dashboard rule 9: panels with a right-side table legend take full
width. Widen "Consensus Rounds per Ledger" and "Consensus Outcome
Distribution" to w=24 and re-tile the dashboard.
Verified via the Grafana proxy: rounds=2 dominates (~11-12 ledgers),
rounds=3 occasional.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 9 introduces the ledger.acquire span (InboundLedger fetch) that phases 7-8
do not have, so the forward-merged 09-data-collection-reference inventory is
extended here:
- §1.1: add ledger.acquire to the Ledger span table.
- §1.2: add its attributes (acquire_reason, timeouts, peer_count, outcome) and
note it also sets ledger_seq; bump the span count.
Also fix two stale StatsD metric references in the Peer Quality dashboard
(xrpld-peer-quality.json): rippled_Peer_Finder_Active_{Inbound,Outbound}_Peers
-> xrpld_Peer_Finder_* to match the xrpld_ metric prefix the rest of the stack uses.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The consensus duration panels plotted span wall-clock
(traces_span_metrics_duration_milliseconds), which is ~3-8 ms of
instrumentation overhead, not the real consensus time (~3000 ms). And
the close-time value panels plotted an ever-rising absolute epoch line.
Rework them to answer the actual operational questions, all from
attributes that already exist on the consensus spans:
- Time to Reach Consensus (p50/p95) and Average Time to Reach Consensus:
round_time_ms on consensus.accept — the wall-clock to agree a ledger.
- Consensus Rounds per Ledger (Establish Count): avg and max of
establish_count on consensus.establish — how many proposal rounds it
took to converge (1 = first proposal).
- Previous Round Time per Ledger: previous_round_time_ms on
consensus.round.
Reorder the dashboard into an investigation flow: health/throughput ->
time-to-consensus and rounds -> ledger close/apply timing -> close-time
detail -> failures/mode/mismatch. Assign stable sequential panel ids.
Verified each query returns data via the Grafana datasource proxy
(p95 ~4096 ms, avg ~2825 ms, rounds ~2).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The §1 span and attribute inventory had regressed to an older 16-span snapshot
that uses the pre-2026-05-13 dotted attribute keys, while phase-7's code emits
~36 spans with bare/underscore attribute keys. The §Data Flow Overview and §2
System Metrics sections (native OTLP transport — phase-7's migration) were already
correct and are left unchanged.
- §1.1: expand the span inventory to the full surface — add gRPC (grpc.<MethodName>),
TxQ (txq.*), PathFind (pathfind.*), and the full consensus set (round/phase.open/
establish/update_positions/check/mode_change/proposal.receive/validation.receive).
Fix the phantom rpc.request -> rpc.http_request, add rpc.ws_upgrade. No grpc.request,
no pathfind.rank, no ledger.acquire (the latter is added in phase-9, not yet present here).
- §1.2: convert every span-attribute key from dotted xrpl.<domain>.<field> to the
bare/underscore form. The sole span-attr dotted exception is xrpl.ledger.hash on
peer.validation.receive (shared constant); consensus.validation.send uses bare
ledger_hash. Resource attrs xrpl.network.id/type stay dotted. Fix tx_count/tx_failed
placement (on tx.apply, not ledger.build). Add attribute tables for the new families.
- §1.3: list the full set of spanmetrics dimension labels (bare keys, from the collector
config) instead of the stale xrpl_rpc_command-style names.
- §4/§5: convert Tempo TraceQL and PromQL examples to the bare attribute/label forms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Apply the dashboard guidelines to the five close-time panels:
- Axis labels (Title Case) on every panel: "Close Time (Ripple
Seconds)" for the value panels, "Count / Milliseconds" for vote
bins/resolution, "Rounds in Window" for the count panels.
- Human-readable legends with the dimension in brackets per the legend
convention: "Raw Close Time [{{resource.service.instance.id}}]",
"Effective Close Time [...]", "Resolution Direction
[{{span.resolution_direction}}]", "{{span.close_time_vote_bins}} Vote
Bins" — replacing the bare label tokens.
- Unit "none" (plain number): the close-time values are Ripple-epoch
seconds and TraceQL metrics cannot offset them to a wall-clock unit,
and the others are counts/ms on a shared axis.
Verified rendered values against raw spans: close times ~833,998,8xx,
resolution 10000 ms, vote bins 1/2/3 — all correct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TraceQL metrics queries default to a 3h max range
(query_frontend.metrics.max_duration), so a dashboard set to a longer
window failed with "range ... exceeds 3h0m0s". Add a query_frontend
block raising it to 168h, matching the search max_duration, so the
consensus close-time panels work at 6h/12h/24h ranges.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Raw Proposals and Effective/Quantized panels rendered nameless
series: their legendFormat used {{service.instance.id}}, but the
TraceQL metrics query groups by resource.service.instance.id and Tempo
returns that full key as the series label. The legend token did not
match any label, so each series showed blank. Use the matching
{{resource.service.instance.id}} token.
Verified via the Grafana datasource proxy that all six close-time panels
now return correctly-labelled series.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Raw Proposals and Effective/Quantized panels showed wrong values
(e.g. 759M, 852M, even 0) against a true value of ~834M. Cause:
quantile_over_time bucketizes into an exponential histogram tuned for
duration distributions, so it cannot represent large absolute integers
(Ripple-epoch seconds) accurately.
Switch both panels to avg_over_time, which returns the correct value
(verified ~833,996,7xx matching the raw span attribute). Average is also
the semantically right aggregation here: close time is a single agreed
value per consensus round, not a latency distribution, so a median was
never meaningful.
Set the unit to none rather than seconds: the value is Ripple-epoch
seconds (Unix = value + 946684800) and TraceQL metrics cannot do the
offset arithmetic in-query, so a duration unit would misrender it.
Clarify in the description that the absolute level tracks wall-clock and
the useful signal is per-node spread / raw-vs-effective gap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The five Close Time panels still rendered "No Data" after the metrics
rewrite. Root cause: each query carried
`span.close_time_correct=~"$close_time_correct"`, but close_time_correct
is a boolean span attribute and TraceQL's regex match (=~) against a
bool matches nothing in a metrics query, so every panel returned an
empty series set (HTTP 200, {"series":[]}).
Remove that filter clause. The panels do not break down by
close_time_correct, so dropping it restores data without losing any
dimension. The $node filter (a string attribute) is unaffected and
stays.
Verified via the Grafana datasource proxy that all six targets now
return series.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire the two previously-registered-but-never-incremented validation
counters to ValidationTracker's gross lifetime tallies, exported as
monotonic ObservableCounters. New gross atomics count each ledger once at
first classification and are never adjusted on late repair, keeping the
_total counters monotonic and additive (agreements_total + missed_total ==
ledgers reconciled); the repair-aware windowed view stays on the existing
xrpld_validation_agreement gauge. The validator-health dashboard panels
that already query these names now render data instead of "No data".
Also de-stale 09-data-collection-reference.md: §5b documented flat metric
names (xrpld_cache_SLE_hit_rate, ...) that the code never emits — it emits
labeled gauges (xrpld_cache_metrics{metric="SLE_hit_rate"}). Replace the
stale flat-name tables with a pointer to the canonical labeled section,
reconcile the contradictory headline counts, and correct xrpld_job_count
to its real exported name xrpld_jobq_job_count.
Adds two GTests asserting gross tallies stay frozen on repair while net
totals move, plus the additive invariant.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The five Close Time panels (Raw Proposals, Effective/Quantized, Vote
Bins & Resolution, Resolution Direction, Bin Distribution) rendered
empty: they used TraceQL `| select(attr)`, which returns a trace list
that a timeseries/barchart panel cannot plot.
Enable TraceQL metrics in Tempo and rewrite the panels to use it:
- tempo.yaml: add the local-blocks processor to the metrics generator so
recent blocks are queryable via /api/metrics/query_range. Set
filter_server_spans=false because the consensus spans are
SPAN_KIND_INTERNAL (the default keeps only server spans, so attribute
aggregations over internal spans returned nothing), and
flush_to_storage=true with a traces_storage path so query_range can
read the flushed blocks.
- consensus-health.json: replace each panel's select() with a metrics
query — quantile_over_time on the integer close-time attributes,
avg_over_time for vote bins / resolution, and count_over_time by the
resolution_direction and vote-bin dimensions. Set the raw/effective
panels' unit to seconds (the values are Ripple-epoch seconds, which
dateTimeFromNow rendered with the wrong epoch).
Verified the query forms compile and return series against live internal
spans; the close-time series populate once the node reaches full sync.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Current Job Latency gauge sat at the bottom of the Job Queue
Analysis dashboard; per the dashboard guideline gauges belong at the
top. Move it to the first row and reflow the remaining panels below it.
Also assign explicit sequential panel ids so deep links stay stable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Correct the width rule from the previous layout commit. Full width
(w=24) is now applied ONLY to timeseries panels whose legend is a
right-side table, since those legends need the horizontal room. Panels
with default/bottom legends, pie charts, and the heatmap return to half
width. This narrows "Transaction Receive vs Suppressed" and "TxQ
Enqueue Rate by Transaction Type", which were wrongly widened.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make transaction-overview deep-links stable and improve readability:
- Assign explicit sequential panel ids (1..20) so viewPanel=panel-N
URLs stay pinned to the same chart across edits. Previously ids were
unset and Grafana auto-assigned them by array position, so any
reorder silently repointed bookmarks.
- Move the single-value stat panel (Transaction Apply Failed Rate) to
the top row.
- Lay out in three topic sections (Processing, Apply Pipeline, Queue).
Within each, timeseries with a breakdown dimension (tx_type, stage,
ter_result, suppressed) take full width so their right-side table
legends are readable; single-series panels, pie charts, and the
heatmap stay half-width and pair up.
All six template variables already default to All (includeAll + multi);
no change needed there.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Apply the dashboard legend convention across all panels now that the
P50 series have been removed (P95-only):
- Drop the redundant "P95 " / "P50 " prefix; the panel title already
states the percentile.
- Put every filter/dimension value inside [] comma-separated, ending
with exported_instance, e.g. "AMMDeposit [Preclaim, xrpld-mainnet]".
- Add exported_instance to the by() clause and legend of the three
panels that filtered on $node but omitted it (Transaction Rate by
Type, Transaction Results by Type, TxQ Accept Status), so per-node
series are produced.
- Title-case the stage value for display via label_replace in the four
apply-pipeline panels; the span attribute stays lowercase
(preflight/preclaim/apply) since legendFormat cannot change case.
- Cap tooltip maxHeight at 500 on every panel.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The existing apply-pipeline panels show latency by stage (all types
combined) or by type (single span). Neither answers "for a given
transaction type, which stage dominates its latency". Add a p95 panel
grouped by both tx_type and stage, filterable via the $tx_type and
$stage variables. Both dimensions already exist in spanmetrics, so no
collector change is needed. Reflow the section so the full-width
failure panel sits below the new full-width panel.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire the apply-pipeline stage spans (tx.preflight, tx.preclaim,
tx.transactor) added on phase-3 through the observability stack so the
spanmetrics connector produces per-stage RED metrics without any native
instruments.
- collector: add the `stage` dimension to the spanmetrics connector so
the three stages split into separate metric series (3 bounded values).
- dashboard: add a "Tx Apply Pipeline" section to transaction-overview
with rate, p95 latency, and failure-rate panels grouped by stage, plus
a `stage` template variable. Panels follow the existing config (node
filter, exported_instance legends, Title Case, axis labels).
- The failure panel filters ter_result != tesSUCCESS rather than span
status, because a failing ter code completes the span normally — only
thrown exceptions set an error status. This matches the existing
"Transaction Results by Type" panel convention.
- docs: document the spans, attributes, and stage dimension in the data
collection reference and runbook, including the sampling caveat that
span-derived metrics inherit tracer head-sampling and undercount at
sampling_ratio < 1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
