The alerting example env file, the contact-point provisioning header and one
runbook line still pointed at a gitignored helper script and at a rollout
phase number, neither of which ships. The contact-point header now states the
policy-tree warning inline rather than deferring to a file the reader cannot
open.
Four files conflicted.
xrpl.proto and TraceContextPropagator.h: phase-9's note is a semantic
superset of phase-8's and already carries no plan-folder pointer, so
phase-9's text is kept and nothing phase-8 said is lost.
node-health.json: the only change phase-8 makes to this file is renaming a
row that phase-9 had already deleted, so phase-9's file stands. Panel count
holds at 59 and both Validated Ledger Seq panels survive.
TESTING.md: resolved per hunk rather than by side. Phase-9's replacement of
the drifted span table and its new Grafana Cloud section are kept, and
phase-8's de-numbered "Test 3: Log-Trace Correlation" heading is taken --
keeping phase-9 wholesale would have reinstated the phase number that
phase-8 removed.
The integration-test sections, alert provisioning headers and a naming-check
test comment were indexed by rollout phase and task numbers defined only in
planning documents outside the shipped tree. Each now names what it covers.
The mainnet and devnet telemetry configs are host processes sharing one
network namespace and working directory, and they agreed on every port and
both database paths. Running them together left the second unable to bind,
and running them in sequence pointed a mainnet node at a store holding the
other network's ledgers, silently and with no error.
Offset the mainnet ports by ten and name both database paths after the
network, matching what debug_logfile already did.
Also narrow the attack surface this config carried. [port_ws_public] has
no admin key, so every caller on it is a guest, and it was bound to all
interfaces on a node following Mainnet; it is now loopback, and nothing in
the repo connects to that port. [signing_support] is removed: it only
affects non-admin callers, the admin ports already grant signing to local
callers, and upstream deprecates the commands it exposes. Verified against
a running node: signing still succeeds on the admin port and is refused as
notSupported on the public one.
Every aggregation that filtered on xrpl_work_item left it out of its
sum by() grouping, so PromQL dropped the label and the xrpl_ident legend
these panels build from it rendered without the work item. 217 clauses
across the touched boards.
Perf-iac gives each work item its own set of nodes, so service_instance_id
already separates the runs and the visible effect is the legend rather
than merged series. Four clauses are the exception and aggregate across
nodes, where the grouping does real work: Convergence, Lag Behind Network
Tip and Build Version now measure spread, lag and version distribution
within one work item instead of smearing two independent clusters
together. Lag Behind Network Tip also needed its on() join key extended to
match, or two coexisting work items would fail the query outright.
The five heatmap inner by (le) clauses are deliberately untouched: an
extra grouping label would superimpose several distributions.
Also in this change:
- transaction-overview gains per-type-per-stage rate and failure-rate
panels, so all three requested dimensions exist rather than latency
alone, and its stage-rate panel picks up the filters and legend its
siblings already had.
- The acquire-duration panel splits by outcome. An aborted acquisition is
open until the sweep reaps it, so mixing it into one percentile
conflated fetch latency with time-to-abandon.
- Ledger Acquire Rate by Outcome gets the same Title Case value mapping as
its neighbour, and node-health gains the $outcome variable the two
panels now need.
LedgerReq Wait by Handler and NodeStore Read Latency (Bottleneck
Discriminator) filter on xrpl_work_item but left it out of their sum by()
grouping, so PromQL dropped the label and the xrpl_ident legend lost that
segment.
These two targets took their current form on this branch, so they are
fixed here. The other nine in this file predate it and were fixed on
pratik/otel-phase7-native-metrics, which merges forward.
The discriminator panel divides two aggregations; both sides carry the
label so their label sets stay equal for vector matching.
Nine targets across four panels filtered on xrpl_work_item while leaving
it out of their sum by() grouping. PromQL keeps only the labels listed in
by(), so the label was dropped from the result and the xrpl_ident legend
these panels build from it lost that segment.
Perf-iac runs stamp xrpl_work_item and give each work item its own set of
nodes, so service_instance_id already separates the runs; the visible
effect is the legend rather than merged series. Adding the label makes the
work item identifiable, which is the comparison these panels exist for.
NuDB Read Latency and NuDB Read Found Ratio each divide two aggregations.
Both sides get the label so their label sets stay equal and vector
matching still works.
Two comment-only notes.
MetricsRegistry.cpp: record why the app and overlay includes exist and
what they cost. They are what makes loops.txt carry
xrpld.app <-> xrpld.telemetry and xrpld.overlay <-> xrpld.telemetry, where
ordering.txt previously had telemetry strictly below both. The observable
gauges are pull-model, so their callbacks need the concrete types to
sample live state. The cycle is confined to this translation unit: no
telemetry header includes app or overlay, and all of src/xrpld builds
into one target, so there is no header or link cycle. Inverting it needs
a metrics-source interface below overlay, which is left as follow-up.
Also note loops.txt is generated and must never be hand-edited.
docker-compose.yml: state the Compose >= 2.24.0 floor. The grafana
service uses the long-form env_file mapping, which older Compose cannot
parse, and it fails for the whole file rather than that one service. The
long form is required because .env.alerting is gitignored and absent in a
fresh clone, and the short form treats a missing env file as an error.
The row was titled after the rollout phase its metrics came from, a number
defined only in a planning folder outside the shipped tree. Grafana shows
this title to operators, so it named something no reader could look up.
Two panel descriptions pointed at a rollout phase and task number defined
only in a planning folder outside the shipped tree. The note still names
the file and the change it is waiting on, which is the part a reader can
act on.
The catalog tagged every span with a rollout phase number defined only in
a planning folder outside the shipped tree, so the column meant nothing
to a reader of the repository. The span name and source file identify
each entry.
These comments and headings identified the feature by a rollout phase
number defined only in a planning folder outside the shipped tree, so the
label meant nothing to a reader of the repository.
The section labels carried rollout phase numbers that are defined only in
a planning folder outside the shipped tree, so they meant nothing to a
reader of the repository. The descriptive half of each label already
identifies the section.
The label carried a rollout phase number defined only in a planning
folder outside the shipped tree. The descriptive half already identifies
the section.
Every telemetry build instruction told the reader to run
`cmake --preset default`. No `CMakePresets.json` is tracked in the repo, and the
only preset that exists is `conan-release`, generated into the gitignored
`CMakeUserPresets.json` by `conan install`. So the documented first build step
fails before it starts, in the runbook's quick-start, its disable procedure, and
the telemetry stack's testing guide.
Replaced with the flow BUILD.md actually documents -- `conan install ..
--output-folder .` followed by an explicit `conan_toolchain.cmake` invocation --
with `-o telemetry=True` / `-Dtelemetry=ON` as the telemetry delta, and a note
that `--preset conan-release` works as a shorthand.
The reference docs had drifted from the code in ways that break the reader
rather than merely misinform: PromQL examples that return no data, a rollback
flag that is a no-op, a sampling knob that does not exist, and two span parents
that moved. Code is treated as the truth throughout; where the code is the
defective side, the doc now records it as a known issue instead of describing
the bug as intent.
Renames the docs missed: histogram names gain the exporter's unit suffix
(ios_latency_milliseconds_bucket and four siblings), ledger_history_mismatch
gains _total, the StatsD-era quantile label gives way to le buckets,
rpc.request becomes rpc.http_request, traces_spanmetrics_calls_total becomes
span_calls_total, and the nine dotted xrpl.* span attributes are recorded as
renamed rather than left as live keys.
Re-parenting: consensus.update_positions and consensus.check are children of
consensus.establish, not of consensus.round.
Units and labels: state_accounting_*_duration is microseconds, not seconds;
cache_metrics label values are case-sensitive; object_count carries demangled
C++ type names. Nodestore read and write latency stays microseconds -- the
nanosecond accumulator change did not move the exported unit.
Adds what shipped but was undocumented: the ledger.acquire span, seven
consensus.round events, twelve span attributes, node_writes_duration_us, the
7-day validation-agreement window, the TxQ admission and reduce-relay metric
families, metrics_endpoint, and the phase-10 validation workflow.
Corrects claims that never held: 10% head sampling (it is fixed at 100%),
configurable redaction (it is unconditional), -DXRPL_ENABLE_TELEMETRY=OFF
(the flag is -Dtelemetry=OFF, default ON), FindOpenTelemetry.cmake and the
xrpl_telemetry target (neither exists), Promtail and a StatsD exporter in the
pipeline (neither exists), and Loki stream selection on job= (only
service_name is a stream label).
Phase 9 is marked complete, its provisioned alerting is attributed to the
branch that shipped it, and Phase 11 stays at zero except the one prerequisite
its code closes. Counts are reconciled repo-wide: 41 emitted span families,
15 dashboards on disk with 14 asserted, 13 alert rules in 5 groups.
Hardens the gate that let this drift through: Rule E of the naming check now
covers the reference docs, its allow-dotted marker is key-scoped and warns on
stale or empty use, a missing checked file is reported instead of silently
skipped, the test suite runs in CI, and doc paths trigger the check.
C++ and CMake changes are comment-only: three MetricsRegistry instrument names,
eight OTelCollector claims of a metric-name prefix that formatName never adds,
and the telemetry option's inverted default.
The Peer Disconnect Rate By Reason panel anchored its LogQL capture on
"\] ", which only matches a reason logged immediately after the [NNN]
peer-id prefix. PeerImp does not log that way: PeerImp::fail emits
"[NNN] <name> failed: <reason>" and the clean teardown emits
"close: Closed". Only ConnectAttempt::fail, which logs the bare reason,
ever matched. The panel's Timeout series was therefore connect-attempt
timeouts only, Ping Timeout was invisible, and PeerImp's own Closed was
uncounted.
Match all three prefixes and separate Ping Timeout from Connect Timeout.
Recorded as LogQL trap 11 in the runbook, alongside the other silent
failures this dashboard exposed.
Also document six overlay observability gaps found while auditing what
ping/pong and gossip traffic is actually tracked. All are pre-existing
and none is fixed here: the code fixes belong in develop-owned overlay
files (TrafficCount, OverlayImpl, PeerImp, PeerfinderManager), not on a
telemetry branch, and one of them needs a public signature change.
- 09 section 6: six known issues, each marked NOT IMPLEMENTED with
file:line evidence -- mtCLUSTER counted as unknown (overhead_cluster_*
always zero, 8 panels flatline), squelch_ignored byte counts always
zero, inbound/outbound byte-basis asymmetry plus a stale Total header
comment, ping/endpoints instrumentation absent, peer span coverage,
and PeerFinder exporting 2 of ~17 available readings.
- 02 section 2.3.2: add a Status column to the span catalog. Of 36
catalogued spans, 16 are live, 15 were never built, and 5 shipped
under different names (consensus.phase.establish became
consensus.establish, ledger.close became consensus.ledger_close,
rpc.request split into rpc.http_request/rpc.ws_message, txq.apply
became txq.apply_direct/txq.accept_tx). The catalog is a design
inventory; 09 section 1.1 remains authoritative for what emits.
- Phase9_taskList: tasks 9.14-9.17 tracking the deferred work, with
exit criteria checked only for what is actually done.
- Glossary: new Ping / pong keepalive term distinguishing ping timeout
from connect timeout. Correct the Cluster and Squelch entries, which
described behaviour the metrics cannot show.
The glossary header pointed at tasks/telemetry_terms.py as its
generator. That file is in no branch and nowhere on disk -- tasks/ is
gitignored one directory up -- so the header now states the file is
hand-maintained and gives the entry format.
Gates: check_otel_naming.py passes all 9 rules (Rule D over 555
dashboard queries, Rule E over the runbook); 19 doc anchors verified;
dashboard JSON valid with a one-line diff. No C++ changes.
The debug-log notice on Log-Derived Insights was two grid rows taller than its
content needs, pushing the first data row further down than necessary. Rendered
the panel to confirm the markdown still fits with no clipping.
It is the first panel in the dashboard, so no other panel's stored y needed to
change - Grafana's vertical compaction closes the gap.
Carries the layout fix to the dashboards this branch owns. A repeating panel
claims the whole row once it expands per network, so the non-repeating panel
paired beside it was pushed down while keeping its stored x=12 - rendering on
the right with an empty gap on its left.
Repeating panels now get a row to themselves, keeping w=12 so their copies still
tile two across. Single-value panels are grouped to the top of each row section
so the charts that follow pair with each other rather than being separated by an
interleaved repeat.
Verified against origin/phase9: no panel lost, every targets block unchanged,
ids 1..N, no overlaps, no rows with a left-hand gap.
A repeating panel expands into one copy per network at view time and, with
maxPerRow=2, claims the whole row. The non-repeating panel paired beside it was
pushed to the next line but kept its stored x=12, so it rendered on the right
against an empty gap.
Two changes to how the layout is planned:
- a repeating panel gets a row to itself. It keeps w=12, so its copies still
tile two across inside that row.
- single-value panels are grouped to the top of each row section, so the
charts that follow pair with each other instead of being split up by an
interleaved repeat. Without this the gaps just become wasted half-rows.
Verified per dashboard: no panel lost, every targets block byte-identical, ids
1..N, no overlaps, and no row left with a gap on its left.
The GetObject Request Size Distribution heatmap, added on this branch, was
missing calculate, color and cellGap. Grafana's heatmap plugin needs them, and
without them the whole dashboard opens with "An error occurred within the
plugin" instead of rendering.
Same values as the other four heatmaps fixed upstream, taken from the one that
renders correctly on Grafana Cloud. The panel keeps its own axis label and unit.
Every heatmap carried only `tooltip` and `yAxis`, missing `calculate`, `color`
and `cellGap`. Grafana's heatmap plugin treats those as required, and without
them the panel fails to initialise: the dashboard opens with "An error occurred
within the plugin" rather than a chart.
The option values are taken from the one heatmap in this stack that does render
on Grafana Cloud (ledger-sync-health): calculate=false since the queries already
return histogram buckets, the Turbo 64-step scheme, and cellGap=1. Each panel
keeps its own yAxis label, unit and tooltip settings.
This is a long-standing defect rather than fallout from the recent layout work -
the same options are absent in origin/phase9 and in the cloud copies that were
already live.
Carries the sizing fix through to the dashboards this branch owns, after the
live Cloud copies showed charts clipping their legends at h=8 and stats too
cramped at h=4.
Every visualisation is h=10; tables and logs h=12; the log-derived-insights
instruction banner h=12 for its prose. A single height per visualisation also
means any two panels pair cleanly side by side.
Repeat stays on single-value panels only. The five state-timeline and heatmap
panels that had picked it up in an earlier pass are cleared, since repeating a
timeline broke ledger-data-sync on Cloud.
Verified against origin/phase9: no panel lost, every targets block unchanged,
ids 1..N, no grid overlaps.
Two problems showed up once these dashboards were live on Grafana Cloud.
Panels were too short. Charts at h=8 clipped their legends mid-row, and stats
at h=4 were cramped. Every visualisation is now h=10, with tables and logs at
h=12; the log-derived-insights instruction banner keeps h=12 for its prose.
Uniform height also means any two panels can pair side by side.
Repeat on a state-timeline broke the dashboard outright: ledger-data-sync
failed to open on Cloud at v45 and had to be restored to v44. Repeat is now
limited to single-value panels (stat, gauge, bargauge, table). Charts show
their networks as separate series instead, which is what a chart is for.
Repeat was also sticky: normalization only ever added the keys, so a panel that
picked them up in an earlier pass kept them even after its type stopped being
eligible. The keys and the title suffix are now removed from ineligible panels,
which is what actually cleared the two timeline panels here.
Verified per dashboard: no panel lost, every targets block byte-identical, ids
1..N, no grid overlaps, and repeat present only on stat/gauge/bargauge.
Applies the guideline-8 ordering to the two dashboards introduced on this
branch, and clears the last two labelling gaps found while auditing the set:
- peer-quality, validator-health: stats and bar gauges now lead their row
section instead of trailing the charts, matching the other dashboards.
- node-health "Peer Disconnects (Resources)": had an empty axisLabel, the
only chart in the set without one. Now reads "Disconnects".
- rpc-performance row "Aggregate RPC (all commands)" -> "(All Commands)",
the one title that was not in Title Case.
Reordering stays inside each row section so no panel changes category, and the
panel objects are cut and re-spliced as raw text, so their contents are
byte-identical. Verified against origin/phase9: no panel lost, every targets
block unchanged, ids 1..N, and every row section now leads with its
single-value panels.
Guideline 8 asks for gauges and stats at the top. Seven dashboards had them
scattered below charts, so the reader met a wall of time series before the
at-a-glance numbers that give those series context.
Stats, gauges and bar gauges now come first within each row section. The move
is deliberately scoped to inside a section: shifting a panel across a row
boundary would change which category it belongs to. Panels keep their relative
order otherwise, so the reading sequence within each group is unchanged.
Whole panel objects are cut and re-spliced as raw text, so their contents stay
byte-identical and only gridPos and id are recomputed. Verified per dashboard:
panel count unchanged, no panel lost, every targets block byte-identical, ids
still 1..N, and no row section left with a stat below a chart.
Applies the same normalization the phase-7-owned dashboards received, to the
five introduced on this branch: fee-market, job-queue, log-derived-insights,
peer-quality and validator-health.
- id written as 1..N so panelId deep links address a specific panel
rather than whichever panel happens to sit in that position
- gridPos at most two panels across; charts h=8, stats/gauges h=4,
tables/logs h=12 full width. Panels pair only with an
equal-height neighbour, so no row keeps a ragged empty cell.
validator-health and peer-quality were stacked single-file at
full width, which is why they were the longest to scroll.
- line charts lineWidth=1, fillOpacity=0, pointSize=5, gradientMode=none
- repeat xrpl_network_type (horizontal, maxPerRow=2) with a
[$xrpl_network_type] title suffix on stat/gauge/bargauge/
table/state-timeline panels, where two networks overlaid in
one panel reads as noise. Line charts keep their networks as
separate series.
- decimals 0 on panels counting discrete things (peers, quorum,
connection mix) - a fractional peer count is meaningless.
- rows category rows added to fee-market, job-queue and peer-quality
Panels with a right-hand legend stay full width; a side legend needs the room.
Raw-text edits, so untouched lines keep their formatting and escaping byte for
byte. Verified per dashboard against origin/phase9: panel count unchanged, no
panel lost, every targets block byte-identical, descriptions unchanged, ids
exactly 1..N, and no two panels overlapping on the grid.
These dashboards were hand-authored over time and had drifted apart: panel
heights spanned ten different values, nine dashboards had no row grouping,
line-chart styling was inconsistent, and no panel carried an id, so Grafana
assigned them positionally at load and every panelId deep link was only as
stable as the panel order.
Per panel, in document order:
- id written as 1..N so panelId links address a specific panel
- gridPos quantized to at most two panels across: charts h=8,
stats/gauges h=4, tables/logs h=12 full width. Panels are
paired only with an equal-height neighbour, so no row is left
with a ragged half-empty cell.
- line charts lineWidth=1, fillOpacity=0, pointSize=5, gradientMode=none
- repeat xrpl_network_type (horizontal, maxPerRow=2) with a
[$xrpl_network_type] title suffix, on the panel types where
overlaying two networks in one panel reads as noise
(stat/gauge/bargauge/table/state-timeline). Line charts keep
their networks as separate series, which is the point of a
line chart.
- decimals 0 where the value counts discrete things (threads, peers,
queue depths); a fractional thread count is meaningless.
- rows category rows added where a dashboard had none
Panels whose legend sits on the right stay full width: a side legend needs the
horizontal room, and squeezing it to half width clips the series names.
Edits were made as raw-text replacements, not a json.dump round-trip, so
formatting and escaping of untouched lines are byte-identical. Verified per
dashboard: panel count unchanged, every targets block byte-identical, all
descriptions unchanged, ids exactly 1..N, and no two panels overlapping on the
grid. The repo dashboard lint and the OTel naming check both pass.
Four defects found by a query-correctness audit of all 16 dashboards, all
scoped to panels this branch owns.
job-queue "Current Job Latency (p99 Gauge)": the histogram by-clause dropped
service_instance_id, collapsing every node into one fleet-wide p99 and hiding
a slow node. Measured: gauge read 1325us while the worst node was 1868us. The
panel's displayName already referenced service_instance_id, so it was also
rendering an empty label. Now groups by service_instance_id and xrpl_work_item,
matching sibling panels 5, 6 and 7.
ledger-data-sync "NodeStore Read Latency (Bottleneck Discriminator)": replaced
clamp_min(<denominator>, 1) with (<denominator> > 0). clamp_min clamps the
value, not just the zero case, so any node reading below 1/s got a fabricated
denominator. Demonstrated with a zero-rate denominator: clamp_min invents
40.2/40.3/11.2/9.2 where the > 0 guard correctly returns no data. This panel is
the bottleneck discriminator, read during a stall, which is exactly when the
read rate collapses and the clamp is most wrong. Panels 21 and 23 carry the
same defect but originate on phase-7 and are fixed there.
node-health thresholds: percentunit fields are compared against the raw value,
so a step of 80 needed 8000% and could never fire. Rescaled panels 74, 81 and
85 to 0.8. Panel 81 is a found-ratio where high is healthy, so its bands were
also inverted. Note these three panels use palette-classic with thresholdsStyle
off, so the steps are currently dormant rather than visibly wrong.
node-health panels 81 and 85 descriptions: both described the multi-series
panels they were split from. Panel 81 carried a byte-identical copy of panel
80's text, promising three plotted rate lines where it draws a single bounded
ratio; panel 85's text described read-thread gauges absent from its expression.
Rewritten to match the actual queries.
The Transaction Overview panel "Queue Bypass Ratio (Direct Apply vs
Enqueue)" reported a confident 0.50 on every node while the true bypass
rate was zero. The two spans it divided are not disjoint alternatives:
txq.apply_direct is a child of txq.enqueue. TxQ.cpp creates the
apply_direct span as the first statement of tryDirectApply(), ahead of
the account, sequence and fee-level guards, and tryDirectApply() is
called from inside the live enqueue scope. The span therefore counts
attempts, so the denominator direct + enqueue counts each transaction
twice and pins the ratio to one half algebraically.
Measured on a four-node fleet: 6082443 direct against 6082877 enqueue
over the same population, panel output 0.5000170 on three nodes and
0.5000000 on the fourth. Grouping txq.enqueue by txq_status over seven
days returns only "rejected" -- no transaction has ever taken the
direct-apply path.
Remove the panel rather than repoint it. A correct expression using
txq_status as the disjoint discriminator would render permanently
empty on this fleet, which reads no better than a wrong number.
Widen the band partner "TxQ Enqueue Rate by Transaction Type" from 12
to 24 columns so the y=48 band still fills the grid. Every band in all
ten dashboards sums to 24 columns; leaving a half-width hole would be
the only exception. Panel order and every other panel's position,
width and height are unchanged.
The runbook already listed txq.apply_direct as available but not
paneled, so that row becomes accurate. Rows describing the span itself
are untouched -- the span and its metric are unchanged.
Panels 21 (NuDB Read Latency) and 23 (NuDB Read Found Ratio) on the
Ledger Data & Sync dashboard guarded their divisor with
clamp_min(<denominator>, 1). clamp_min raises the value, not just the
zero case, so any node reading fewer than 1 block per second was
divided by a fabricated 1 instead of by its real read count.
Replace the clamp with the filter (<denominator> > 0). A comparison
without the bool modifier drops the sample rather than rewriting it, so
these panels now show no data instead of a wrong number.
Measured over 7 days: five nodes fall below 1 read/s. On validator-0 the
clamp reported 2.726 us/read against a true 5.493, and on nonUNLmalloc-tc
it reported 0 us/read, which cannot occur. The error is largest exactly
when panel 21 is used as the bottleneck discriminator during a stall,
because that is when the read rate collapses toward zero.
Matches the existing idiom on the same nodestore_state metric family in
the NodeStore Write vs Read Latency panel.
