Match the panel height the operator settled on in Grafana (18 rows) so
all 20 ranked bars render without an inner scrollbar, and reflow the
Sync Diagnostics row and the panels below it accordingly.
The bar labels on 'Overlay Traffic Heatmap (All Categories, Bytes In)'
were raw metric names carrying a redundant '_bytes_in' suffix, and the
half-width 8-row panel truncated both the category and the node
identity.
- Strip the '_bytes_in' suffix from the derived series label; the panel
title already states the metric is inbound bytes.
- Widen the panel to full width and grow it to 12 rows so 20 bars render
with their full category and node labels.
- Shift the Sync Diagnostics row and the panels below it down by 4 to
keep the layout gap-free.
- Note the label derivation in the panel description.
The four aggregated Sync Diagnostics panels grouped by xrpl_work_item,
which no layer in this repo emits (it is injected by the perf-iac
harness). That tripped Rule D of the telemetry naming check:
D ledger-data-sync.json xrpl_work_item
must exist in L1, a metric label, or be a builtin
Align with the convention used by every other aggregated panel in the
dashboard set: group by (service_instance_id, xrpl_branch,
xrpl_node_role) and leave the xrpl_ident label_join untouched. The
legend is unaffected -- label_join over a label dropped by the
aggregation contributes an empty segment, which the trailing
label_replace already strips.
Verified live: both the NuDB ratio and the job-queue p95 queries still
return per-node series rendering as "[aws-dev-xrpl-1]".
The 0-4 values now render as named states via value mappings, so the numeric
axis label is misleading. Rename to "Server State".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add value mappings (0 Disconnected, 1 Connected, 2 Syncing, 3 Tracking,
4 Full) and matching red->green thresholds to the Sync State panel, so the
line, tooltip, and legend render state names and colors instead of bare 0-4.
Keeps the minimal custom-key convention; only the Sync State panel changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rework the 10 Sync Diagnostics panels' legends to match the format used by the
existing panels: wrap each query in label_join + label_replace to build the
xrpl_ident label ([service_instance_id, xrpl_branch, xrpl_work_item] with empty
values stripped) and set displayName to "${series} ${xrpl_ident}". Verified
against a live node: renders as "Series [aws-dev-xrpl-1]" with no empty-label
gaps, identical mechanism to the original 15 panels. Only the new panels change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a 10-panel "Sync Diagnostics" row that shows ledger-sync slowdown and its
causes, laid out top-to-bottom as symptom -> latency -> root cause:
- Red flags: Sync State, Validated Ledger Age, Ledger Close Rate
- Latency: Job Queue Wait p95 by type, NuDB Read Latency, I/O Scheduler p95
- Root cause: NuDB Cache Hit Ratio, NuDB Read Pressure, Job Queue Depth,
Load Factor & Peers
All panels use native beast::insight metrics introduced on this branch
(nodestore_state, ledgermaster_*, jobq_*_q, ios_latency, load_factor_metrics,
peer_finder_*) plus the consensus.mode_change span metric. Queries filter by
the dashboard's $node and tier variables and were verified against a live
node. Nodestore ratio panels use sum-by(identity) matching to bridge the
metric= sub-label.
Format matches the dashboard convention: minimal custom keys, ${DS_PROMETHEUS}
datasource uid, and the standard What/How/Reading/Healthy/Watch/Source/Function
description structure. Pure-append; existing panels untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Document the two new ledger-correlation attributes in the data-collection
reference: add them to the transaction and TxQ attribute tables (with per-span
coverage), update the attribute count, and correct the txq.enqueue parent note
(parents to tx.process on the submission path via explicit context; a root on
the open-ledger rebuild path, correlated via current_ledger_seq).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add current_ledger_seq / current_ledger_hash to the tx.process, tx.receive,
and txq.enqueue span-reference rows, correct the txq.enqueue parent note
(parents to tx.process on the submission path via explicit context; a root on
the open-ledger rebuild path), and add a "Correlating a transaction to the
ledger it was worked on" recipe joining the txID-keyed tx/txq spans to the
ledger trace via current_ledger_seq.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add shared current_ledger_seq / current_ledger_hash span attributes so a
transaction's work can be joined to the ledger trace that produced it, and
fix discrepancy D1 (txq.enqueue was a detached trace root).
- Define current_ledger_seq / current_ledger_hash once in SpanNames.h and
re-export via `using` from TxQ/TxApply/Tx span-name headers. These name the
ledger being worked on (open/tentative apply or in-flight consensus build),
distinct from ledger_seq (the built/validated ledger on ledger.build /
consensus.round). Named after the RPC field ledger_current_index.
- txq.enqueue: set current_ledger_seq/hash from the view, and parent the span
to the caller's tx.process span via an explicit captured SpanContext (new
trailing TxQ::apply param) instead of a detached root. The parent is
explicit, not ambient-inherited, and the ScopedSpanGuard scope is RAII-bound
to the synchronous apply, so it cannot leak onto a reused worker (D1 fix).
On the open-ledger rebuild path no tx.process context exists, so it stays a
root and the attribute provides the correlation.
- tx.preclaim / tx.transactor: set both attributes from their ledger view.
tx.preflight is stateless (no view) and is the documented exception.
- tx.process / tx.receive: set current_ledger_seq from the current open ledger
index at submit/receive time (no hash: not yet applied to a ledger).
- Contract test pins the two new attribute key strings.
Neither key is a spanmetrics dimension, so there is no metric-cardinality
impact. Dashboards/collector/docs land on the later phases per the chain split.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The colocated span-name headers (RpcSpanNames.h, ConsensusSpanNames.h,
PeerSpanNames.h, etc.) hold only compile-time constant strings and live
next to their subsystem rather than under a telemetry/ directory, so the
existing dir-prefix ignores miss them. Add a **/*SpanNames.h glob so they
do not count against patch coverage; like the rest of the telemetry
surface they are not exercised in the coverage build.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The existing ignore list only covered src/xrpld/telemetry/, but the same
rationale — telemetry code is conditionally compiled behind
XRPL_ENABLE_TELEMETRY, which the coverage build does not enable — applies to
src/libxrpl/telemetry/ (SpanGuard.cpp, Telemetry.cpp, etc.) and the
include/xrpl/telemetry/ headers. Their lines are never executed in coverage
builds, so they were dragging patch coverage down on the telemetry PRs.
Extend the ignore list to match.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
