Node identity reached the OTel resource only as service.instance.id, which is
config-overridable and carries a deployment-chosen label rather than the node's
own identity. Add xrpl.node.id, set unconditionally from the node public key
(base58, TokenType::NodePublic), so traces and metrics share a stable per-node
key independent of [telemetry] service_instance_id.
Set on the tracer resource via Telemetry::setNodeId(), called from
ApplicationImp::setup() once nodeIdentity_ is known, and on the MetricsRegistry
resource via an added start() parameter. The beast::insight meter provider is
built in TelemetryImpl's constructor, before the wallet DB exists, so its
resource cannot carry the value; that path is left for later and the attribute
is omitted rather than stamped blank.
Also drops the transform/spanidentity collector processor added in
4a361a496d: per-node identity belongs on the resource, not copied onto every
span.
Consensus spans share one deterministic, ledger-derived trace_id, so a
single trace holds spans from every node and the resource-level node id is
not a reliable per-span discriminator in stored traces.
Add transform/spanidentity to both collector configs, copying
service.instance.id onto every span as service_instance_id so TraceQL can
filter per node with the same value the $node dashboard variable already
uses on the metrics side. Wired into the traces pipeline locally and into
traces/store (after tail_sampling) on the Grafana Cloud variant.
The transitions panel used increase(...[$__rate_interval]). $__rate_interval is
defined as max($__interval + scrape, 4 * scrape), i.e. deliberately one scrape
longer than the step so rate() windows overlap and lose no counter increase.
That overlap is harmless for rate(), but this panel reads the value as a count
of discrete events, and the overlap counts each event in more than one bucket.
Measured against a log-derived ground truth of 106 syncing transitions on
devnet-otel-usw2-01 over 2026-08-11T11:05Z..2026-08-12T23:04Z, the old query
reported 111.3 at a 300s step and 133.7 at a 60s step -- the error grew to +26%
as you zoomed in, because the overlap is a larger fraction of a smaller step.
Switch to $__interval so the buckets tile exactly, and wrap in round() because
increase() extrapolates to the window edges and so reports fractional counts for
an integer counter. The same measurement now gives 106 at 300s, 105 at 60s and
107 at 900s. Every state and both nodes land within a few counts of truth at any
zoom, and the legend Total is now a meaningful figure.
Pin Min step to 1m: the real scrape interval is 60s while the datasource
declares 15s, so without a floor $__interval can fall below one sample.
Draw as bars with 0 decimals -- the value is a discrete count per bucket, and a
line implies interpolation between counts that does not exist.
Nine conflicts, resolved as follows.
src/xrpld/app/ledger/detail/InboundLedger.cpp -- kept this branch's version.
phase10 sets the span's outcome/timeouts/peer_count attributes inline at each
exit; this branch replaced that with the idempotent finalizeAcquireSpan(), called
on all four exits (init, done, give-up, destructor). Taking phase10's blocks
would have set the outcome twice against a helper documented as not overwriting
what the real exit recorded. phase10's comment explains why peer_count must not
be read in a destructor; the helper solves that structurally by taking
std::optional<std::size_t> and being passed std::nullopt from there.
src/xrpld/telemetry/MetricsRegistry.cpp -- kept metric::ledgerEconomy over
phase10's "ledger_economy" literal. This branch added the naming check that
requires constants for converted families, so the literal would regress it. Took
phase10's comment cleanup.
src/xrpld/telemetry/MetricsRegistry.h -- kept registerRotationStateGauge(), which
only exists here, and took phase10's removal of the stale task-number comment.
validate_telemetry.py -- combined both. phase10 replaced serial metric polling
with a concurrent fan-out on one shared deadline, because 58 metrics x 45 s of
additive timeout overran the CI budget; that is kept. Its target list filters on
SKIPPED_METRIC_GROUPS rather than the two literals it hardcoded, so the
sync_diagnostics group stays owned by assert_sync_diagnostics_metrics() instead
of being polled and reported twice. Both SYNC_DIAGNOSTICS_GROUP and
METRIC_POLL_CONCURRENCY are needed and both are kept.
check_otel_naming.py -- both sides extend the rule docstring. Took phase10's
fuller Rule E text (doc discovery, allow-dotted markers) and re-appended rules
I/J/K/L, which exist only here.
expected_metrics.json -- the two sides add disjoint sibling groups, so both are
kept: sync_diagnostics alongside node_health_gauges, overlay_reduce_relay,
overlay_overflow, validation_lifetime_counters and not_asserted. Both dashboard
uids are kept, giving 16 asserted uids against 16 dashboards on disk.
expected_spans.json -- kept this branch's span set, a superset that adds the
acquire phase spans, ledger.serve, txset.acquire and peer.dial, and expands
ledger.acquire's required attributes. Took phase10's description, which documents
what the totals mean, and its note on how the RPC wildcard span is created.
total_span_types and total_unique_attributes are recomputed for the union: 48 and
74, since each side's figure counted only its own spans.
Docs: took phase10's more accurate wording on what the dashboard check actually
covers, and corrected the dashboard count from 15 to 16 where the merge made it
stale.
Verified: no conflict markers remain, both JSON contracts parse, both Python
files compile, asserted dashboard uids match the dashboards on disk exactly, and
the OTel naming check reports all layers consistent.
The Operating Mode Transitions panel queried state_accounting_*_transitions
directly. Those are monotonic counters, so the panel drew a slowly rising line
and a few transitions per hour were invisible against a total in the hundreds.
It also fell off a cliff whenever xrpld restarted and the counters reset to 0,
which reads as missing data rather than a restart.
Wrap each target in increase(...[$__rate_interval]) so each point is the number
of transitions in that bucket and the series survives a counter reset. This is
what the sibling panels on the same row (Operating Mode (Time Share), State
Duration Rate) already do.
Verified against devnet-otel-usw2-01/02 over 2026-08-11T11:01Z..2026-08-12T16:23Z:
the fixed expression reports 107 and 123 syncing transitions, matching the
counter deltas, and stays continuous across the 12:07 restart where the raw
counter dropped 630 -> 1.
Brief mode flaps remain invisible on Operating Mode (State Timeline) because a
~2 s dwell cannot be captured by a 60 s scrape; this panel is the place to read
them.
This branch had already made the same corrections independently, and in
richer form, so the resolution keeps this branch's version nearly throughout:
- 09-data-collection-reference.md: this branch already documents the
state-accounting gauges as cumulative **microseconds** with an explanatory
callout, and already names `jobq_job_count` with its `jobq` group. Kept.
- telemetry-runbook.md: already carries `jobq_job_count` in both tables. Kept,
along with this branch's larger additions.
- OpenTelemetryPlan.md: kept this branch's rewritten section 9 blurb, which
describes the inventory without hardcoding counts and so cannot drift.
- consensus-health.json: kept this branch's rewrite. It deliberately removed
the four TraceQL close-time detail panels and renamed the agreement panel;
the incoming side would have resurrected them. Panel count unchanged at 26.
- integration-test.sh: this branch's unprefixed native metric names were kept,
but it still asserted `job_count`, so the `jobq_job_count` correction was
carried over. That check would otherwise always fail.
Conflict resolution kept this branch's evolution and re-applied phase-6's
fixes on top of it, rather than taking either side wholesale:
- consensus-health.json: kept the native `span_calls_total` metric name and
the `interval: 15s` and point styling from this branch; added phase-6's
`close_time_correct` PromQL filter and the NetClock axis labels. The
TraceQL boolean-regex filter stays removed and the `byRegexp` overrides
carry over. Panel count unchanged at 27.
- 09-data-collection-reference.md: kept this branch's headings, its more
detailed consensus attribute table (which already types
`consensus_round_id` as int64) and its section numbering, including the
deliberate removal of the SpanNames inventory. Carried over only the
correction that the state-accounting duration gauges are cumulative
microseconds, not seconds.
- telemetry-runbook.md: kept this branch's native metric names
(`span_calls_total`, `span_duration_milliseconds_bucket`); carried the
`rpc.request` -> `rpc.http_request` span-name fix and the `jobq_` segment
on the job-queue depth metric.
- integration-test.sh: kept this branch's `check_otel_metric` form and
carried the `jobq_job_count` correction.
The integration test asserted `rippled_job_count`, which never reports any
series, so that check always failed. `JobQueue` registers the gauge as
`makeGauge("job_count")`, but `Application.cpp` passes it
`collectorManager_->group("jobq")`, so the emitted StatsD name is
`jobq.job_count` and the exported Prometheus name is
`<prefix>_jobq_job_count`.
Corrected the same name in two runbook tables that also dropped the `jobq`
segment. `09-data-collection-reference.md` already had it right, which is why
the two documents disagreed.
Routed here rather than to the phase-10 PR where it was reported: the wrong
name is present in `integration-test.sh` on every branch from phase 6
onward, and this is the branch that introduces the file.
Left alone deliberately:
- `statsd-node-health.json` still queries the old name, but that dashboard is
deleted at phase 7 in favour of `node-health.json`
- `06-implementation-phases.md` names `job_count`, which is accurate as the
code-level makeGauge argument rather than the exported metric name
The file_storage extension was added to otel-collector-config.yaml, which
every stack mounts. That made the extension mandatory: the collector image
runs as 10001:10001 and ships no writable directory, so any stack without a
prepared volume would fail to start rather than merely lose offsets. The
workload-validation stack mounts this same config and has no such volume.
Offset persistence is only useful where logs outlive a restart. The workload
harness creates a fresh log directory per run, so it has nothing to resume
from. Move the extension, the receiver's storage reference and the extended
service.extensions list into otel-collector-filestorage.yaml, layered as a
second --config by the developer stack alone. The base config keeps
start_at: beginning, which is what actually fixes the reported defect, and
stays self-sufficient for every other stack.
Verified against the pinned collector image: the base config validates and
runs on its own with no volume mounted and still ingests a line written
before startup; base plus overlay validates, preserves the base receiver's
operators through the merge, and re-ingests that line zero times on a second
run against the same volume.
Conflict resolutions:
- docker/telemetry/xrpld-telemetry.cfg: relocation conflict. phase-9 had
already moved [insight] to the end of the file with server=otel, so the
incoming block was dropped rather than inserted. Keeping both would have
produced two [insight] sections, which merge last-wins into a single
effective section, silently reviving the bug this branch just fixed.
phase-9's per-branch service_instance_id=xrpld-devnet is preserved.
- OpenTelemetryPlan/06-implementation-phases.md: kept both corrections.
phase-9's "Tempo" is right (no Jaeger anywhere in the stack) and
phase-8's "active, sampled span" is right: Log.cpp:328 injects only
when spanCtx.IsValid() && spanCtx.IsSampled().
- OpenTelemetryPlan/09-data-collection-reference.md and
docs/telemetry-runbook.md: kept phase-9's structured-metadata LogQL.
The collector's filelog regex_parser already extracts partition,
severity, trace_id and span_id, so phase-8's inline regexp forms are
redundant, and a line filter matches the literal text in a message body.
The generated node config carried two [insight] blocks. Duplicate ini
sections do not replace one another: parseIniFile emplaces the section
name (a no-op when it already exists) and appends the lines to the same
vector, then Section::append writes each key with insert_or_assign. The
effective section was therefore server=statsd with the first block's
endpoint and service_instance_id surviving but unused.
CollectorManager selects StatsDCollector for that value, so the nodes
emitted beast::insight metrics over UDP to 8125, which has no receiver
in the collector pipeline and no published port. The script's own check
asserts that 8125 is not listening, and its insight metric assertions
fail on zero series.
Keep only the server=otel block so the config matches what the script
verifies.
This branch removes the collector's StatsD receiver and un-publishes
8125/udp, but xrpld-telemetry.cfg still selected server=statsd, so the
sample config sent beast::insight metrics over UDP to a port nothing
listens on. Phase7_taskList.md:132 lists this switch as required work.
Select server=otel and replace address= with the OTLP metrics endpoint.
Document that endpoint and prefix are informational only, since
OTelCollector records on the global MeterProvider that [telemetry]
configures and formatName() applies no prefix, and note that beast
instruments are not exported yet because the collector is constructed
before the MeterProvider is registered.
Six findings from the review of #6494 survived independent verification.
Each was checked against the branch tip, and where behaviour was in
question, against a live collector and Loki rather than from the
reviewer's claim or from documentation alone.
Plan-doc section numbering. 06-implementation-phases.md used "## 6.9"
twice: for the new Phase 8 section and for the pre-existing Risk
Assessment. Three references already pointed at 6.8.1 and none at 6.9,
and the later phases are numbered 6.8.2 through 6.8.4, so Phase 8
becomes 6.8.1 and the sequence is monotonic. Renumbering to 6.10, as
suggested on the PR, would have collided with Success Metrics.
filelog read position. The receiver relied on the upstream default
start_at=end, which skips everything a node wrote before the first poll
and reads nothing at all from a log that has stopped being written to.
Read from the beginning instead, paired with a file_storage extension so
a restart resumes at the last offset rather than re-ingesting the file.
The collector image runs as 10001:10001 and ships no writable directory,
and a fresh named volume is root-owned, so a one-shot init service
prepares the volume first. It reuses an image the stack already pulls,
adding no new dependency.
Loki log stream label. The job resource attribute did not become a Loki
index label, so the documented {job="xrpld"} queries matched nothing.
Verified against grafana/loki:3.4.2 with its default config: only
service_name and deployment_environment are indexed, and job arrives as
structured metadata, which a stream selector cannot match. Dropped the
attribute and moved the twelve queries this branch introduced to
{service_name="xrpld"}. Three further occurrences in
07-observability-backends.md originate on the phase-1a branch and are
left for a commit there.
Trace ids on unsampled spans. Logs::format emitted trace_id and span_id
whenever the span context was valid. A span dropped by the
ParentBasedSampler still carries its parent's ids, so log lines
advertised traces that were never exported and the log-to-trace link
resolved to nothing. Require the sampled flag as well, and correct the
task list and the documentation that promised the fields unconditionally.
The remaining two findings were refuted. The reported risk of signing
material reaching Loki does not hold: Logs::format already scrubs seven
sensitive fields, and there is a single write path to the log file, so
every JLOG site is covered. The suggestion to add internalLink to the
Loki derived field is not applicable, because that key is not part of
Grafana's schema.
All five TraceQL panels on this dashboard returned nothing, and did so
without any visible error: they filtered on
span.close_time_correct=~"$close_time_correct", but close_time_correct is
a boolean attribute (RCLConsensus.cpp:601 passes a raw bool), and Tempo
restricts the regex operator to string operands, so the spanset resolved
to false. With the variable defaulting to All the clause rendered as
=~".*", so the panels were empty out of the box and looked exactly like a
node with no consensus activity.
Note this is the opposite of PromQL, where an absent or empty label does
match ".*" — which is why the 17 Prometheus panels on this same board were
unaffected and the dashboard appeared healthy.
Dropped the clause from all six queries, matching phases 9 and 10 where it
is already gone. The $close_time_correct variable now filters the
Prometheus "Close Time Agreement" panel instead, which already grouped by
that label but never filtered on it, so the control stays useful rather
than becoming dead UI.
Two defects were masked behind the empty panels and are fixed too:
- "Close Time: Raw Proposals" and "Close Time: Effective / Quantized"
carried unit dateTimeFromNow over close_time_self/close_time, which are
NetClock seconds (Ripple epoch), while Grafana's dateTime formatters
expect a millisecond Unix epoch — every point would have rendered as
roughly 1970. They now plot as plain numbers with the axis labelled
"NetClock Seconds (Ripple Epoch)", and the descriptions give the
946684800 offset for converting to Unix time.
- "Close Time Vote Bins & Resolution" matched its unit and axis overrides
byName against "Vote Bins" and "Resolution", which are not field names;
TraceQL select() yields close_time_vote_bins and close_resolution_ms, so
neither override applied. Switched to byRegexp so the match holds
whichever scope prefix Grafana emits.
No panel was added or removed: the (type, title) multiset is unchanged at
22. resolution_direction keeps its regex filter, which is correct there —
it is set from a std::string whose values are exactly the variable's
increased/decreased/unchanged.
The integration test's span assertions never actually ran. check_span()
built a Tempo /api/search call with --data-urlencode but no -G, so curl
POSTed the params as a body; Tempo answers 200 and ignores the query, so
every span name looked present. Verified against a live Tempo 2.9.4: the
buggy form returns the store's total trace count for any name, including
"zzz.does.not.exist"; with -G a real name returns 1 and a bogus one 0.
Fixed alongside it: the RPC check asserted "rpc.request", which is never
emitted (ServerHandler.cpp builds "rpc.http_request"). These two had to
change together, since -G turns the bogus name from a silent pass into a
hard failure.
Also in the script: a consensus timeout logged two failures and counted
two, because a post-loop else re-reported what the timeout branch had
already reported; and three unguarded curl calls aborted the whole script
under set -euo pipefail, making the ACCOUNT_ZERO fallback dead code with
no cleanup. Guarded the curls and wired an EXIT trap to the existing
cleanup(). The trap deliberately fires only before the summary, so a
completed run still leaves the stack up as the header documents.
Docs corrections, all re-derived from code:
- span inventory heading 35 -> 38, attribute heading 83 -> 89 rows
(78 unique keys), and the section 6 header table now carries the
missing TxApplySpanNames.h row so its columns sum to the same figures
- two stale paths: ConsensusSpanNames.h is under include/xrpl/consensus/,
TxSpanNames.h under src/xrpld/telemetry/
- consensus_round_id is int64, not string (RCLConsensus.cpp sets
prevLgr.seq() + 1); the runbook's TraceQL examples now use a numeric
literal instead of an unparseable bare <round_id>
- state-accounting duration gauges are cumulative MICROSECONDS, not
seconds (NetworkOPs.cpp declares std::chrono::microseconds and
publishes dur.count() raw)
- sampling_ratio is not a config key; head sampling is fixed at 1.0 and
the shipped collector has no tail sampling, so the caveat was rewritten
- the plan blurb referenced Jaeger; this stack is Tempo
The setting was flagged in review as unauthenticated admin access. It is
deliberate, and it matches the sibling stack in docker-compose.yml, which
carries the same two variables and the same published port with its intent
in comments. This copy had none, so the reasoning lived only in a review
thread and was rediscovered as a finding each time the file was looked at.
Viewer would break the harness rather than harden it: the validation suite
drives the Grafana API against this instance to confirm each dashboard
provisions and loads, and the dashboards and datasources come from the
read-only mounts on the same service.
Two defects reported against the harness, both confirmed.
The TPS field was computed with `bc` at scale=2, and bc omits the leading
zero: it prints ".25", not "0.25". A bare ".25" is not valid JSON, and this
was the normal case rather than an edge case — ledgers close every few
seconds, so ledger-advance over elapsed-seconds is well under 1 for any
realistic window. It survived earlier checks because those piped the file
through jq, which accepts the malformed form; Python's json rejects the whole
file. awk's %.2f always pads, so the field is now produced with awk. Audited
the other numeric fields at the same time: CPU average and memory peak
already used awk, and the p99, sample count and consensus mean are integers,
so TPS was the only one affected.
Separately, a failing attribute fetch was reported under the span's own check
name, which had already recorded the trace as found. That produced two
entries for one name, one passing and one failing, inflating the check total
and blaming the trace-existence check for a failure in a later network call.
The fetch now carries its own error handling and reports under
`span.attrs.<span>`, matching where its successful counterpart reports. It
moved into a helper rather than growing `validate_spans`, which was already
well over the line limit.
Three consecutive validation runs timed out at Step 3 with nodes stuck at
"unreachable", and the reason was not recoverable from the logs. The node
logs showed the failing nodes stopping at an identical point, immediately
after JobQueue initialisation and before the debug log is opened, with no
error text at all. The harness knew each node's pid and never used it, so a
crashed node was indistinguishable from a slow one.
The readiness loop now checks whether each node process is still alive and
fails as soon as one is not, instead of waiting out the remaining window and
burying the cause under two minutes of progress output. Liveness is not a
bare `kill -0`: an exited-but-unreaped child keeps its pid, so a zombie
answers `kill -0` and reads as alive for the whole window, which is exactly
how a crashed node came to look like a slow one.
On failure each stopped node reports its wait status and the tail of its
stdout. The status is the discriminator that was missing: 137 for a SIGKILL,
139 for a segfault, 134 for an abort, anything below 128 for a deliberate
exit. stdout is printed inline rather than left to the artifact upload,
because a node that dies before its debug log opens writes nothing else and
a cancelled run uploads nothing at all.
This is instrumentation, not a fix. The failure is not attributable to the
recent changes on this branch: the first red run touched only the two Python
files used at Steps 4 and 5, both of which run after this gate, and the same
harness passed 5/5 twice before that.
Run with no arguments the script iterated an empty list, found no
violations and printed "OK: 0 dashboard(s) passed" with exit 0 -- a clean
bill of health for reading no files, indistinguishable from a real pass.
A bare run now defaults to every dashboard beside the script, and a run
that still ends up with nothing to check exits 2 rather than reporting
success. Passing paths explicitly behaves as before.
Single-value panels carry the filtered network type in their title because
they have no legend to put it in; multi-series panels carry it in
legendFormat instead. This gauge was the only single-value panel across the
fifteen dashboards without it, so its reading was ambiguous once more than
one network type was in scope.
A validation run timed out at Step 3 with only 4 of 5 nodes proposing, and
the reason was unrecoverable afterwards. Two gaps caused that.
The node-log artifact collected `node*/debug.log` but not `node*/stdout.log`.
A node that dies before its log sink opens never writes a debug.log at all,
so stdout is the only place its reason survives — and that file is written by
the harness and read by nothing, so it went to the runner and was discarded.
The failing node's log was simply absent from the artifact.
The readiness loop also fetched each node's `server_state` and threw it away,
reporting only a count. "4/5 nodes proposing" says a node is missing but not
which one, so there is nothing to grep for even once the logs are kept. The
timeout now names each node that is not proposing along with the state it
last reported, distinguishing a node that answered with a non-proposing
state from one whose RPC port did not answer at all.
Neither change affects a healthy run: the accumulator resets each attempt and
stays empty while every node is proposing.
The alerting example env file, the contact-point provisioning header and one
runbook line still pointed at a gitignored helper script and at a rollout
phase number, neither of which ships. The contact-point header now states the
policy-tree warning inline rather than deferring to a file the reader cannot
open.
Four files conflicted.
xrpl.proto and TraceContextPropagator.h: phase-9's note is a semantic
superset of phase-8's and already carries no plan-folder pointer, so
phase-9's text is kept and nothing phase-8 said is lost.
node-health.json: the only change phase-8 makes to this file is renaming a
row that phase-9 had already deleted, so phase-9's file stands. Panel count
holds at 59 and both Validated Ledger Seq panels survive.
TESTING.md: resolved per hunk rather than by side. Phase-9's replacement of
the drifted span table and its new Grafana Cloud section are kept, and
phase-8's de-numbered "Test 3: Log-Trace Correlation" heading is taken --
keeping phase-9 wholesale would have reinstated the phase number that
phase-8 removed.
The integration-test sections, alert provisioning headers and a naming-check
test comment were indexed by rollout phase and task numbers defined only in
planning documents outside the shipped tree. Each now names what it covers.
Two defects reported against the validation harness. Both premises were
correct, but neither suggested fix was, so the remedies differ.
Dashboard panel count: `len(dashboard["panels"])` treated Grafana row
objects as panels and skipped the panels nested inside collapsed rows, so
every dashboard was over-reported by between 1 and 10 (`log-derived-insights`
read 41 against a true 31). The check also passed unconditionally on HTTP
200, so a dashboard that renders nothing would still pass. `_leaf_panel_count`
now walks row children and the result gates the verdict. Gating on the old
top-level length, as suggested, would not have caught the case it was aimed
at: a dashboard made only of collapsed rows counts its rows and reports a
positive number while rendering nothing.
RPC latency percentiles: `LoadStats.record` appended a latency for every
outcome, including requests that never got a reply, where the value is a
time-to-failure rather than a round trip. A timeout contributed the full
receive timeout, and at the error rate a real run shows this reported p95 and
p99 of 10000 ms where the true figure was 5 ms. `record` now takes an
optional latency and the timeout path passes none. The suggestion to append
only on success was not adopted: a reply carrying `status: error` is a
completed, timely round trip whose latency is a genuine measurement, and
discarding it would throw away real data. `per_command` is now keyed off the
request counts rather than the latency map, so a command whose every request
timed out still appears in the report instead of vanishing from it, and each
entry carries a `latency_samples` count.
The mainnet and devnet telemetry configs are host processes sharing one
network namespace and working directory, and they agreed on every port and
both database paths. Running them together left the second unable to bind,
and running them in sequence pointed a mainnet node at a store holding the
other network's ledgers, silently and with no error.
Offset the mainnet ports by ten and name both database paths after the
network, matching what debug_logfile already did.
Also narrow the attack surface this config carried. [port_ws_public] has
no admin key, so every caller on it is a guest, and it was bound to all
interfaces on a node following Mainnet; it is now loopback, and nothing in
the repo connects to that port. [signing_support] is removed: it only
affects non-admin callers, the admin ports already grant signing to local
callers, and upstream deprecates the commands it exposes. Verified against
a running node: signing still succeeds on the admin port and is refused as
notSupported on the public one.
Every aggregation that filtered on xrpl_work_item left it out of its
sum by() grouping, so PromQL dropped the label and the xrpl_ident legend
these panels build from it rendered without the work item. 217 clauses
across the touched boards.
Perf-iac gives each work item its own set of nodes, so service_instance_id
already separates the runs and the visible effect is the legend rather
than merged series. Four clauses are the exception and aggregate across
nodes, where the grouping does real work: Convergence, Lag Behind Network
Tip and Build Version now measure spread, lag and version distribution
within one work item instead of smearing two independent clusters
together. Lag Behind Network Tip also needed its on() join key extended to
match, or two coexisting work items would fail the query outright.
The five heatmap inner by (le) clauses are deliberately untouched: an
extra grouping label would superimpose several distributions.
Also in this change:
- transaction-overview gains per-type-per-stage rate and failure-rate
panels, so all three requested dimensions exist rather than latency
alone, and its stage-rate panel picks up the filters and legend its
siblings already had.
- The acquire-duration panel splits by outcome. An aborted acquisition is
open until the sweep reaps it, so mixing it into one percentile
conflated fetch latency with time-to-abandon.
- Ledger Acquire Rate by Outcome gets the same Title Case value mapping as
its neighbour, and node-health gains the $outcome variable the two
panels now need.
LedgerReq Wait by Handler and NodeStore Read Latency (Bottleneck
Discriminator) filter on xrpl_work_item but left it out of their sum by()
grouping, so PromQL dropped the label and the xrpl_ident legend lost that
segment.
These two targets took their current form on this branch, so they are
fixed here. The other nine in this file predate it and were fixed on
pratik/otel-phase7-native-metrics, which merges forward.
The discriminator panel divides two aggregations; both sides carry the
label so their label sets stay equal for vector matching.
Nine targets across four panels filtered on xrpl_work_item while leaving
it out of their sum by() grouping. PromQL keeps only the labels listed in
by(), so the label was dropped from the result and the xrpl_ident legend
these panels build from it lost that segment.
Perf-iac runs stamp xrpl_work_item and give each work item its own set of
nodes, so service_instance_id already separates the runs; the visible
effect is the legend rather than merged series. Adding the label makes the
work item identifiable, which is the comparison these panels exist for.
NuDB Read Latency and NuDB Read Found Ratio each divide two aggregations.
Both sides get the label so their label sets stay equal and vector
matching still works.
Two comment-only notes.
MetricsRegistry.cpp: record why the app and overlay includes exist and
what they cost. They are what makes loops.txt carry
xrpld.app <-> xrpld.telemetry and xrpld.overlay <-> xrpld.telemetry, where
ordering.txt previously had telemetry strictly below both. The observable
gauges are pull-model, so their callbacks need the concrete types to
sample live state. The cycle is confined to this translation unit: no
telemetry header includes app or overlay, and all of src/xrpld builds
into one target, so there is no header or link cycle. Inverting it needs
a metrics-source interface below overlay, which is left as follow-up.
Also note loops.txt is generated and must never be hand-edited.
docker-compose.yml: state the Compose >= 2.24.0 floor. The grafana
service uses the long-form env_file mapping, which older Compose cannot
parse, and it fails for the whole file rather than that one service. The
long form is required because .env.alerting is gitignored and absent in a
fresh clone, and the short form treats a missing env file as an error.
