makeTelemetrySetup() rejects a contradictory [telemetry] mutual-TLS
setup by throwing, but it is called from ApplicationImp's
member-initializer list. A try/catch in the constructor body cannot
reach a throw from there, and nothing further up the stack caught it
either, so a config mistake reached std::terminate: the default handler
printed a terminate dump and raised SIGABRT, leaving a core file
instead of a startup error.
Catch std::exception around makeApplication() in run(), report the
reason on stderr and return -1, so the failure is a clean non-zero exit
with a message an operator can act on. Only the construction is
wrapped. setup() starts subsystems whose shutdown order is delicate and
is left outside deliberately, because unwinding a half-started
Application would skip the normal stop sequence.
Gate both validation guards on enabled. A node with telemetry switched
off previously refused to start over certificate paths that nothing
would read.
Document both throws on makeTelemetrySetup(), state in
cfg/xrpld-example.cfg and the configuration reference that a partial
mutual-TLS setup is fatal and that the checks apply only when
enabled=1, and add a runbook troubleshooting entry keyed on the two
error messages.
Tests cover both guards with the message asserted so the two are told
apart, both enabled=0 paths, and the default plaintext configuration.
The integration test asserted `rippled_job_count`, which never reports any
series, so that check always failed. `JobQueue` registers the gauge as
`makeGauge("job_count")`, but `Application.cpp` passes it
`collectorManager_->group("jobq")`, so the emitted StatsD name is
`jobq.job_count` and the exported Prometheus name is
`<prefix>_jobq_job_count`.
Corrected the same name in two runbook tables that also dropped the `jobq`
segment. `09-data-collection-reference.md` already had it right, which is why
the two documents disagreed.
Routed here rather than to the phase-10 PR where it was reported: the wrong
name is present in `integration-test.sh` on every branch from phase 6
onward, and this is the branch that introduces the file.
Left alone deliberately:
- `statsd-node-health.json` still queries the old name, but that dashboard is
deleted at phase 7 in favour of `node-health.json`
- `06-implementation-phases.md` names `job_count`, which is accurate as the
code-level makeGauge argument rather than the exported metric name
All five TraceQL panels on this dashboard returned nothing, and did so
without any visible error: they filtered on
span.close_time_correct=~"$close_time_correct", but close_time_correct is
a boolean attribute (RCLConsensus.cpp:601 passes a raw bool), and Tempo
restricts the regex operator to string operands, so the spanset resolved
to false. With the variable defaulting to All the clause rendered as
=~".*", so the panels were empty out of the box and looked exactly like a
node with no consensus activity.
Note this is the opposite of PromQL, where an absent or empty label does
match ".*" — which is why the 17 Prometheus panels on this same board were
unaffected and the dashboard appeared healthy.
Dropped the clause from all six queries, matching phases 9 and 10 where it
is already gone. The $close_time_correct variable now filters the
Prometheus "Close Time Agreement" panel instead, which already grouped by
that label but never filtered on it, so the control stays useful rather
than becoming dead UI.
Two defects were masked behind the empty panels and are fixed too:
- "Close Time: Raw Proposals" and "Close Time: Effective / Quantized"
carried unit dateTimeFromNow over close_time_self/close_time, which are
NetClock seconds (Ripple epoch), while Grafana's dateTime formatters
expect a millisecond Unix epoch — every point would have rendered as
roughly 1970. They now plot as plain numbers with the axis labelled
"NetClock Seconds (Ripple Epoch)", and the descriptions give the
946684800 offset for converting to Unix time.
- "Close Time Vote Bins & Resolution" matched its unit and axis overrides
byName against "Vote Bins" and "Resolution", which are not field names;
TraceQL select() yields close_time_vote_bins and close_resolution_ms, so
neither override applied. Switched to byRegexp so the match holds
whichever scope prefix Grafana emits.
No panel was added or removed: the (type, title) multiset is unchanged at
22. resolution_direction keeps its regex filter, which is correct there —
it is set from a std::string whose values are exactly the variable's
increased/decreased/unchanged.
The integration test's span assertions never actually ran. check_span()
built a Tempo /api/search call with --data-urlencode but no -G, so curl
POSTed the params as a body; Tempo answers 200 and ignores the query, so
every span name looked present. Verified against a live Tempo 2.9.4: the
buggy form returns the store's total trace count for any name, including
"zzz.does.not.exist"; with -G a real name returns 1 and a bogus one 0.
Fixed alongside it: the RPC check asserted "rpc.request", which is never
emitted (ServerHandler.cpp builds "rpc.http_request"). These two had to
change together, since -G turns the bogus name from a silent pass into a
hard failure.
Also in the script: a consensus timeout logged two failures and counted
two, because a post-loop else re-reported what the timeout branch had
already reported; and three unguarded curl calls aborted the whole script
under set -euo pipefail, making the ACCOUNT_ZERO fallback dead code with
no cleanup. Guarded the curls and wired an EXIT trap to the existing
cleanup(). The trap deliberately fires only before the summary, so a
completed run still leaves the stack up as the header documents.
Docs corrections, all re-derived from code:
- span inventory heading 35 -> 38, attribute heading 83 -> 89 rows
(78 unique keys), and the section 6 header table now carries the
missing TxApplySpanNames.h row so its columns sum to the same figures
- two stale paths: ConsensusSpanNames.h is under include/xrpl/consensus/,
TxSpanNames.h under src/xrpld/telemetry/
- consensus_round_id is int64, not string (RCLConsensus.cpp sets
prevLgr.seq() + 1); the runbook's TraceQL examples now use a numeric
literal instead of an unparseable bare <round_id>
- state-accounting duration gauges are cumulative MICROSECONDS, not
seconds (NetworkOPs.cpp declares std::chrono::microseconds and
publishes dur.count() raw)
- sampling_ratio is not a config key; head sampling is fixed at 1.0 and
the shipped collector has no tail sampling, so the caveat was rewritten
- the plan blurb referenced Jaeger; this stack is Tempo
Run with no arguments the script iterated an empty list, found no
violations and printed "OK: 0 dashboard(s) passed" with exit 0 -- a clean
bill of health for reading no files, indistinguishable from a real pass.
A bare run now defaults to every dashboard beside the script, and a run
that still ends up with nothing to check exits 2 rather than reporting
success. Passing paths explicitly behaves as before.
Two panel descriptions pointed at a rollout phase and task number defined
only in a planning folder outside the shipped tree. The note still names
the file and the change it is waiting on, which is the part a reader can
act on.
The catalog tagged every span with a rollout phase number defined only in
a planning folder outside the shipped tree, so the column meant nothing
to a reader of the repository. The span name and source file identify
each entry.
The section labels carried rollout phase numbers that are defined only in
a planning folder outside the shipped tree, so they meant nothing to a
reader of the repository. The descriptive half of each label already
identifies the section.
These comments pointed at a planning folder and at its rollout phase
numbering, neither of which is part of the shipped tree, so the
references would dangle for any reader of the repository. Each comment
now states the fact it was pointing at.
The trace_state comment pointed at a planning document that is not part
of the shipped tree, so the reference would dangle for any reader of the
repository. State the reserved-and-inert fact on its own.
