This branch had already made the same corrections independently, and in
richer form, so the resolution keeps this branch's version nearly throughout:
- 09-data-collection-reference.md: this branch already documents the
state-accounting gauges as cumulative **microseconds** with an explanatory
callout, and already names `jobq_job_count` with its `jobq` group. Kept.
- telemetry-runbook.md: already carries `jobq_job_count` in both tables. Kept,
along with this branch's larger additions.
- OpenTelemetryPlan.md: kept this branch's rewritten section 9 blurb, which
describes the inventory without hardcoding counts and so cannot drift.
- consensus-health.json: kept this branch's rewrite. It deliberately removed
the four TraceQL close-time detail panels and renamed the agreement panel;
the incoming side would have resurrected them. Panel count unchanged at 26.
- integration-test.sh: this branch's unprefixed native metric names were kept,
but it still asserted `job_count`, so the `jobq_job_count` correction was
carried over. That check would otherwise always fail.
The troubleshooting step queried `job_count`, which returns no series. The
gauge is registered as `makeGauge("job_count")` but `Application.cpp` passes
`collectorManager_->group("jobq")`, so the exported name carries the `jobq`
segment. The two metric tables in this file were corrected when phase-6
merged forward; this example was missed because it sits outside the tables.
Conflict resolution kept this branch's evolution and re-applied phase-6's
fixes on top of it, rather than taking either side wholesale:
- consensus-health.json: kept the native `span_calls_total` metric name and
the `interval: 15s` and point styling from this branch; added phase-6's
`close_time_correct` PromQL filter and the NetClock axis labels. The
TraceQL boolean-regex filter stays removed and the `byRegexp` overrides
carry over. Panel count unchanged at 27.
- 09-data-collection-reference.md: kept this branch's headings, its more
detailed consensus attribute table (which already types
`consensus_round_id` as int64) and its section numbering, including the
deliberate removal of the SpanNames inventory. Carried over only the
correction that the state-accounting duration gauges are cumulative
microseconds, not seconds.
- telemetry-runbook.md: kept this branch's native metric names
(`span_calls_total`, `span_duration_milliseconds_bucket`); carried the
`rpc.request` -> `rpc.http_request` span-name fix and the `jobq_` segment
on the job-queue depth metric.
- integration-test.sh: kept this branch's `check_otel_metric` form and
carried the `jobq_job_count` correction.
The integration test asserted `rippled_job_count`, which never reports any
series, so that check always failed. `JobQueue` registers the gauge as
`makeGauge("job_count")`, but `Application.cpp` passes it
`collectorManager_->group("jobq")`, so the emitted StatsD name is
`jobq.job_count` and the exported Prometheus name is
`<prefix>_jobq_job_count`.
Corrected the same name in two runbook tables that also dropped the `jobq`
segment. `09-data-collection-reference.md` already had it right, which is why
the two documents disagreed.
Routed here rather than to the phase-10 PR where it was reported: the wrong
name is present in `integration-test.sh` on every branch from phase 6
onward, and this is the branch that introduces the file.
Left alone deliberately:
- `statsd-node-health.json` still queries the old name, but that dashboard is
deleted at phase 7 in favour of `node-health.json`
- `06-implementation-phases.md` names `job_count`, which is accurate as the
code-level makeGauge argument rather than the exported metric name
The file_storage extension was added to otel-collector-config.yaml, which
every stack mounts. That made the extension mandatory: the collector image
runs as 10001:10001 and ships no writable directory, so any stack without a
prepared volume would fail to start rather than merely lose offsets. The
workload-validation stack mounts this same config and has no such volume.
Offset persistence is only useful where logs outlive a restart. The workload
harness creates a fresh log directory per run, so it has nothing to resume
from. Move the extension, the receiver's storage reference and the extended
service.extensions list into otel-collector-filestorage.yaml, layered as a
second --config by the developer stack alone. The base config keeps
start_at: beginning, which is what actually fixes the reported defect, and
stays self-sufficient for every other stack.
Verified against the pinned collector image: the base config validates and
runs on its own with no volume mounted and still ingests a line written
before startup; base plus overlay validates, preserves the base receiver's
operators through the merge, and re-ingests that line zero times on a second
run against the same volume.
Conflict resolution, OpenTelemetryPlan/09-data-collection-reference.md
Known Issues table: kept phase-9's expanded 17-row table rather than the
incoming 4-row version, and re-applied the incoming correction to the one
row it changed. The other three incoming rows are already present in
phase-9's table, and phase-9's version is more current (jobq_job_count,
not job_count).
The rpc_requests row now says [insight] server=otel; server=statsd
reaches a collector with no statsd receiver on this branch. The section 8
Quick Reference and the fallback caveat auto-merged cleanly.
09-data-collection-reference.md contradicted itself. Section 2 presents
server=otel as the recommended transport with StatsD as a fallback, but
the Known Issues table and both Configuration Quick Reference examples
still prescribed server=statsd, which on this branch reaches a collector
with no statsd receiver and an unpublished 8125/udp.
Switch the Known Issues row and the Minimal and Production examples to
server=otel with the OTLP metrics endpoint. Keep the labelled fallback
block, and state what it actually requires: re-adding the statsd
receiver and republishing the port. Also record that StatsDCollector
applies prefix to metric names while OTelCollector does not, so the two
transports do not produce the same series.
The phase-6 copy is left alone; server=statsd is correct there.
check-otel-naming fails Rule D on this branch: ledger-data-sync.json
aggregates by xrpl_work_item, which nothing in-tree emits because
perf-iac's alloy pipeline stamps it, so it cannot be derived from
*SpanNames.h. The sibling perf-iac identities xrpl_branch and
xrpl_node_role are already allowlisted; this one was missed when the
by() clauses were reintroduced.
Add it to EXTERNAL_INFRA_LABELS alongside them. Verified by removing
the entry again, which restores the failure.
Conflict resolutions:
- docker/telemetry/xrpld-telemetry.cfg: relocation conflict. phase-9 had
already moved [insight] to the end of the file with server=otel, so the
incoming block was dropped rather than inserted. Keeping both would have
produced two [insight] sections, which merge last-wins into a single
effective section, silently reviving the bug this branch just fixed.
phase-9's per-branch service_instance_id=xrpld-devnet is preserved.
- OpenTelemetryPlan/06-implementation-phases.md: kept both corrections.
phase-9's "Tempo" is right (no Jaeger anywhere in the stack) and
phase-8's "active, sampled span" is right: Log.cpp:328 injects only
when spanCtx.IsValid() && spanCtx.IsSampled().
- OpenTelemetryPlan/09-data-collection-reference.md and
docs/telemetry-runbook.md: kept phase-9's structured-metadata LogQL.
The collector's filelog regex_parser already extracts partition,
severity, trace_id and span_id, so phase-8's inline regexp forms are
redundant, and a line filter matches the literal text in a message body.
The generated node config carried two [insight] blocks. Duplicate ini
sections do not replace one another: parseIniFile emplaces the section
name (a no-op when it already exists) and appends the lines to the same
vector, then Section::append writes each key with insert_or_assign. The
effective section was therefore server=statsd with the first block's
endpoint and service_instance_id surviving but unused.
CollectorManager selects StatsDCollector for that value, so the nodes
emitted beast::insight metrics over UDP to 8125, which has no receiver
in the collector pipeline and no published port. The script's own check
asserts that 8125 is not listening, and its insight metric assertions
fail on zero series.
Keep only the server=otel block so the config matches what the script
verifies.
This branch removes the collector's StatsD receiver and un-publishes
8125/udp, but xrpld-telemetry.cfg still selected server=statsd, so the
sample config sent beast::insight metrics over UDP to a port nothing
listens on. Phase7_taskList.md:132 lists this switch as required work.
Select server=otel and replace address= with the OTLP metrics endpoint.
Document that endpoint and prefix are informational only, since
OTelCollector records on the global MeterProvider that [telemetry]
configures and formatName() applies no prefix, and note that beast
instruments are not exported yet because the collector is constructed
before the MeterProvider is registered.
Six findings from the review of #6494 survived independent verification.
Each was checked against the branch tip, and where behaviour was in
question, against a live collector and Loki rather than from the
reviewer's claim or from documentation alone.
Plan-doc section numbering. 06-implementation-phases.md used "## 6.9"
twice: for the new Phase 8 section and for the pre-existing Risk
Assessment. Three references already pointed at 6.8.1 and none at 6.9,
and the later phases are numbered 6.8.2 through 6.8.4, so Phase 8
becomes 6.8.1 and the sequence is monotonic. Renumbering to 6.10, as
suggested on the PR, would have collided with Success Metrics.
filelog read position. The receiver relied on the upstream default
start_at=end, which skips everything a node wrote before the first poll
and reads nothing at all from a log that has stopped being written to.
Read from the beginning instead, paired with a file_storage extension so
a restart resumes at the last offset rather than re-ingesting the file.
The collector image runs as 10001:10001 and ships no writable directory,
and a fresh named volume is root-owned, so a one-shot init service
prepares the volume first. It reuses an image the stack already pulls,
adding no new dependency.
Loki log stream label. The job resource attribute did not become a Loki
index label, so the documented {job="xrpld"} queries matched nothing.
Verified against grafana/loki:3.4.2 with its default config: only
service_name and deployment_environment are indexed, and job arrives as
structured metadata, which a stream selector cannot match. Dropped the
attribute and moved the twelve queries this branch introduced to
{service_name="xrpld"}. Three further occurrences in
07-observability-backends.md originate on the phase-1a branch and are
left for a commit there.
Trace ids on unsampled spans. Logs::format emitted trace_id and span_id
whenever the span context was valid. A span dropped by the
ParentBasedSampler still carries its parent's ids, so log lines
advertised traces that were never exported and the log-to-trace link
resolved to nothing. Require the sampled flag as well, and correct the
task list and the documentation that promised the fields unconditionally.
The remaining two findings were refuted. The reported risk of signing
material reaching Loki does not hold: Logs::format already scrubs seven
sensitive fields, and there is a single write path to the log file, so
every JLOG site is covered. The suggestion to add internalLink to the
Loki derived field is not applicable, because that key is not part of
Grafana's schema.
All five TraceQL panels on this dashboard returned nothing, and did so
without any visible error: they filtered on
span.close_time_correct=~"$close_time_correct", but close_time_correct is
a boolean attribute (RCLConsensus.cpp:601 passes a raw bool), and Tempo
restricts the regex operator to string operands, so the spanset resolved
to false. With the variable defaulting to All the clause rendered as
=~".*", so the panels were empty out of the box and looked exactly like a
node with no consensus activity.
Note this is the opposite of PromQL, where an absent or empty label does
match ".*" — which is why the 17 Prometheus panels on this same board were
unaffected and the dashboard appeared healthy.
Dropped the clause from all six queries, matching phases 9 and 10 where it
is already gone. The $close_time_correct variable now filters the
Prometheus "Close Time Agreement" panel instead, which already grouped by
that label but never filtered on it, so the control stays useful rather
than becoming dead UI.
Two defects were masked behind the empty panels and are fixed too:
- "Close Time: Raw Proposals" and "Close Time: Effective / Quantized"
carried unit dateTimeFromNow over close_time_self/close_time, which are
NetClock seconds (Ripple epoch), while Grafana's dateTime formatters
expect a millisecond Unix epoch — every point would have rendered as
roughly 1970. They now plot as plain numbers with the axis labelled
"NetClock Seconds (Ripple Epoch)", and the descriptions give the
946684800 offset for converting to Unix time.
- "Close Time Vote Bins & Resolution" matched its unit and axis overrides
byName against "Vote Bins" and "Resolution", which are not field names;
TraceQL select() yields close_time_vote_bins and close_resolution_ms, so
neither override applied. Switched to byRegexp so the match holds
whichever scope prefix Grafana emits.
No panel was added or removed: the (type, title) multiset is unchanged at
22. resolution_direction keeps its regex filter, which is correct there —
it is set from a std::string whose values are exactly the variable's
increased/decreased/unchanged.
The integration test's span assertions never actually ran. check_span()
built a Tempo /api/search call with --data-urlencode but no -G, so curl
POSTed the params as a body; Tempo answers 200 and ignores the query, so
every span name looked present. Verified against a live Tempo 2.9.4: the
buggy form returns the store's total trace count for any name, including
"zzz.does.not.exist"; with -G a real name returns 1 and a bogus one 0.
Fixed alongside it: the RPC check asserted "rpc.request", which is never
emitted (ServerHandler.cpp builds "rpc.http_request"). These two had to
change together, since -G turns the bogus name from a silent pass into a
hard failure.
Also in the script: a consensus timeout logged two failures and counted
two, because a post-loop else re-reported what the timeout branch had
already reported; and three unguarded curl calls aborted the whole script
under set -euo pipefail, making the ACCOUNT_ZERO fallback dead code with
no cleanup. Guarded the curls and wired an EXIT trap to the existing
cleanup(). The trap deliberately fires only before the summary, so a
completed run still leaves the stack up as the header documents.
Docs corrections, all re-derived from code:
- span inventory heading 35 -> 38, attribute heading 83 -> 89 rows
(78 unique keys), and the section 6 header table now carries the
missing TxApplySpanNames.h row so its columns sum to the same figures
- two stale paths: ConsensusSpanNames.h is under include/xrpl/consensus/,
TxSpanNames.h under src/xrpld/telemetry/
- consensus_round_id is int64, not string (RCLConsensus.cpp sets
prevLgr.seq() + 1); the runbook's TraceQL examples now use a numeric
literal instead of an unparseable bare <round_id>
- state-accounting duration gauges are cumulative MICROSECONDS, not
seconds (NetworkOPs.cpp declares std::chrono::microseconds and
publishes dur.count() raw)
- sampling_ratio is not a config key; head sampling is fixed at 1.0 and
the shipped collector has no tail sampling, so the caveat was rewritten
- the plan blurb referenced Jaeger; this stack is Tempo
Run with no arguments the script iterated an empty list, found no
violations and printed "OK: 0 dashboard(s) passed" with exit 0 -- a clean
bill of health for reading no files, indistinguishable from a real pass.
A bare run now defaults to every dashboard beside the script, and a run
that still ends up with nothing to check exits 2 rather than reporting
success. Passing paths explicitly behaves as before.
Single-value panels carry the filtered network type in their title because
they have no legend to put it in; multi-series panels carry it in
legendFormat instead. This gauge was the only single-value panel across the
fifteen dashboards without it, so its reading was ambiguous once more than
one network type was in scope.
The alerting example env file, the contact-point provisioning header and one
runbook line still pointed at a gitignored helper script and at a rollout
phase number, neither of which ships. The contact-point header now states the
policy-tree warning inline rather than deferring to a file the reader cannot
open.
Four files conflicted.
xrpl.proto and TraceContextPropagator.h: phase-9's note is a semantic
superset of phase-8's and already carries no plan-folder pointer, so
phase-9's text is kept and nothing phase-8 said is lost.
node-health.json: the only change phase-8 makes to this file is renaming a
row that phase-9 had already deleted, so phase-9's file stands. Panel count
holds at 59 and both Validated Ledger Seq panels survive.
TESTING.md: resolved per hunk rather than by side. Phase-9's replacement of
the drifted span table and its new Grafana Cloud section are kept, and
phase-8's de-numbered "Test 3: Log-Trace Correlation" heading is taken --
keeping phase-9 wholesale would have reinstated the phase number that
phase-8 removed.
The runbook pointed at a planning document and a helper script that are
outside the shipped tree, and at a dashboard script that no longer exists,
so an operator following it hit three dead ends. The Cloud alert workflow
is now described by what it does to the tracked rules.yaml.
The integration-test sections, alert provisioning headers and a naming-check
test comment were indexed by rollout phase and task numbers defined only in
planning documents outside the shipped tree. Each now names what it covers.
These comments were indexed by task, use-case and limitation numbers that
are defined only in planning documents outside the shipped tree. Nothing
in the repository defined them, so the cross-references resolved nowhere.
Each comment now states what the code does.
Two unrelated edits had crept in. The added `.claude/` line duplicates
`/.claude`, which the same file has already carried since well before this
branch, so it never changed behaviour. Removing the root `.env.*` rule did
change behaviour: contributors lost repo-level cover for any env file
outside docker/telemetry, and the local ignore that masks this on one
machine is not part of the repo.
Restore the rule and drop the duplicate, which leaves the root gitignore
byte-identical to the parent branch.
The scoped rules added under docker/telemetry stay: they keep the three
tracked .env.*.example files visible and stop the generated Grafana Cloud
dashboards being committed, which is telemetry work this branch owns.
The runbook told operators to read a missing outcome as "never went to the
network", which stopped being true once the abort path started setting it,
and the glossary still described outcome as a two-value split.
Document all three values, where each is written, and why peer_count is
absent on the abort path. Several claims were wrong and are corrected:
- Give-up is reached at about 18s, not 21s. There is no setTimer call, so
the first timer runs immediately and the old derivation counted a wait
that does not happen.
- A live aborted rate does not imply stalled acquisitions. A clean
shutdown clears every in-flight acquire, the admin fetch_info clear does
the same, and a full job lane stops timeouts advancing so give-up cannot
fire. The runbook already said the last of these elsewhere.
- A missing outcome does not mean exactly one thing. tryDB can set failed_
and return before done() runs, exporting a span with no outcome at all.
- failed covers unusable ledger data as well as exhausting the retry
limit, so a failed span can carry timeouts=0.
- The aborted lower bound of one minute holds only on the sweep path.
- The sweep measures time since anything last asked for the ledger, not
since data last arrived.
Also fixes the mainnet verification command, which still used the old RPC
port, and drops an inaccurate claim from the config comment about which
ports the workload scripts use.
The mainnet and devnet telemetry configs are host processes sharing one
network namespace and working directory, and they agreed on every port and
both database paths. Running them together left the second unable to bind,
and running them in sequence pointed a mainnet node at a store holding the
other network's ledgers, silently and with no error.
Offset the mainnet ports by ten and name both database paths after the
network, matching what debug_logfile already did.
Also narrow the attack surface this config carried. [port_ws_public] has
no admin key, so every caller on it is a guest, and it was bound to all
interfaces on a node following Mainnet; it is now loopback, and nothing in
the repo connects to that port. [signing_support] is removed: it only
affects non-admin callers, the admin ports already grant signing to local
callers, and upstream deprecates the commands it exposes. Verified against
a running node: signing still succeeds on the admin port and is refused as
notSupported on the public one.
Every aggregation that filtered on xrpl_work_item left it out of its
sum by() grouping, so PromQL dropped the label and the xrpl_ident legend
these panels build from it rendered without the work item. 217 clauses
across the touched boards.
Perf-iac gives each work item its own set of nodes, so service_instance_id
already separates the runs and the visible effect is the legend rather
than merged series. Four clauses are the exception and aggregate across
nodes, where the grouping does real work: Convergence, Lag Behind Network
Tip and Build Version now measure spread, lag and version distribution
within one work item instead of smearing two independent clusters
together. Lag Behind Network Tip also needed its on() join key extended to
match, or two coexisting work items would fail the query outright.
The five heatmap inner by (le) clauses are deliberately untouched: an
extra grouping label would superimpose several distributions.
Also in this change:
- transaction-overview gains per-type-per-stage rate and failure-rate
panels, so all three requested dimensions exist rather than latency
alone, and its stage-rate panel picks up the filters and legend its
siblings already had.
- The acquire-duration panel splits by outcome. An aborted acquisition is
open until the sweep reaps it, so mixing it into one percentile
conflated fetch latency with time-to-abandon.
- Ledger Acquire Rate by Outcome gets the same Title Case value mapping as
its neighbour, and node-health gains the $outcome variable the two
panels now need.
LedgerReq Wait by Handler and NodeStore Read Latency (Bottleneck
Discriminator) filter on xrpl_work_item but left it out of their sum by()
grouping, so PromQL dropped the label and the xrpl_ident legend lost that
segment.
These two targets took their current form on this branch, so they are
fixed here. The other nine in this file predate it and were fixed on
pratik/otel-phase7-native-metrics, which merges forward.
The discriminator panel divides two aggregations; both sides carry the
label so their label sets stay equal for vector matching.
Nine targets across four panels filtered on xrpl_work_item while leaving
it out of their sum by() grouping. PromQL keeps only the labels listed in
by(), so the label was dropped from the result and the xrpl_ident legend
these panels build from it lost that segment.
Perf-iac runs stamp xrpl_work_item and give each work item its own set of
nodes, so service_instance_id already separates the runs; the visible
effect is the legend rather than merged series. Adding the label makes the
work item identifiable, which is the comparison these panels exist for.
NuDB Read Latency and NuDB Read Found Ratio each divide two aggregations.
Both sides get the label so their label sets stay equal and vector
matching still works.
Two comment-only notes.
MetricsRegistry.cpp: record why the app and overlay includes exist and
what they cost. They are what makes loops.txt carry
xrpld.app <-> xrpld.telemetry and xrpld.overlay <-> xrpld.telemetry, where
ordering.txt previously had telemetry strictly below both. The observable
gauges are pull-model, so their callbacks need the concrete types to
sample live state. The cycle is confined to this translation unit: no
telemetry header includes app or overlay, and all of src/xrpld builds
into one target, so there is no header or link cycle. Inverting it needs
a metrics-source interface below overlay, which is left as follow-up.
Also note loops.txt is generated and must never be hand-edited.
docker-compose.yml: state the Compose >= 2.24.0 floor. The grafana
service uses the long-form env_file mapping, which older Compose cannot
parse, and it fails for the whole file rather than that one service. The
long form is required because .env.alerting is gitignored and absent in a
fresh clone, and the short form treats a missing env file as an error.
The test carried both <xrpl/proto/xrpl.pb.h> and the bare <xrpl.pb.h>.
Both resolve to the same generated header, because the proto helper puts
the generated tree and its prefixed subdirectory on the target, so the
second include expanded to nothing behind the header guard.
The bare form arrived from merging two same-day clang-tidy commits that
added the include with different spellings. Keep the prefixed spelling,
which is what the telemetry headers and the upstream phase branches use.
An InboundLedger destroyed while !isDone() recorded recordAbort() in the
metrics but left ledger.acquire carrying only the attributes set at
construction, so an abandoned acquisition was indistinguishable from one
still in flight.
Set outcome=aborted plus timeouts on that path, and add the val::aborted
constant. peer_count is deliberately omitted: reading it goes through
Overlay, and a destructor must not depend on Overlay still existing.
The status stays Unset for an abort, because InboundLedgers::stop()
clears every in-flight acquisition, so a clean shutdown would otherwise
report errors. done() now sets Error when failed_, which is an
unambiguous failure of the operation. Success is left Unset rather than
Ok, per the OpenTelemetry guidance that instrumentation should not
assert Ok.
rpcStart, jobQueue, jobStart and jobFinish each acquired a lock without
braces, so it stayed held to the end of the function and covered the OTel
recording calls this branch added. counters_.jobsMutex and
counters_.methodsMutex are process-wide, so every worker thread starting
or finishing a job serialised on the SDK's work.
That work is not a bare atomic add: each record builds a map-backed
attribute set and takes a spin lock inside the SDK, whose backoff reaches
a millisecond-scale sleep under contention.
Brace the lock plus the state it guards, then record after it releases.
None of the metric calls read lock-protected state, so this is
semantics-preserving, and rpcEnd() in the same file already had this
shape.
The row was titled after the rollout phase its metrics came from, a number
defined only in a planning folder outside the shipped tree. Grafana shows
this title to operators, so it named something no reader could look up.
Two panel descriptions pointed at a rollout phase and task number defined
only in a planning folder outside the shipped tree. The note still names
the file and the change it is waiting on, which is the part a reader can
act on.
The catalog tagged every span with a rollout phase number defined only in
a planning folder outside the shipped tree, so the column meant nothing
to a reader of the repository. The span name and source file identify
each entry.
These comments and headings identified the feature by a rollout phase
number defined only in a planning folder outside the shipped tree, so the
label meant nothing to a reader of the repository.
