The three tracker files conflicted because both sides had rewritten them. Took
the incoming lock-free class, then put this branch's two monotonic accessors
back on top of it: totalAgreementsEver() and totalMissedEver(), backed by a
gross pair incremented at first classification and left alone by the repair
branch. MetricsRegistry reads both, so dropping them would not compile.
The test file kept the incoming suite, which renames every case, and gained
this branch's two gross-counter cases adapted to the injected clock.
Three CI failures, one cause each.
macOS could not compile the tracker: Apple's libc++ has no std::atomic for a
shared_ptr, so the primary template's trivially-copyable assert fired. Publish
through boost::atomic_shared_ptr instead, which every standard library the
matrix covers can build. boost/smart_ptr is already used in this tree. The
header's note no longer claims libstdc++ as the assumption.
Four tests that predate the metrics_endpoint scheme guard set tls_client_cert
and only put traces_endpoint on https, so the new guard threw before the check
each one asserts. They now set both endpoints. Nine tests in the two files set a
client cert; the other five stay correct because the pairing, use_tls and traces
checks all run ahead of the metrics one.
Eleven clang-tidy findings: redundant member initialisers, two aggregate
initialisations that wanted designated form, two unbraced bodies, three
unparenthesised multiplications, and a reserve before a loop that emplaces.
Dropping std::make_shared also left <memory> unused in both files.
The collector reads the per-node directory off the log file path and stamps it
as the Loki label service_instance_id, so the directory name has to equal the
node's own [telemetry] service_instance_id or log lines carry a node name that
no trace or metric shares and nothing joins.
Both harness scripts disagreed with themselves: run-full-validation.sh wrote to
node$i while setting validator-${i}, and benchmark.sh wrote to node$i while
setting bench-node-${i}. Rename the directories to match the ids rather than
the reverse, so no existing trace or metric label value moves and no harness
expectation has to be re-checked. Only path references are renamed; the
human-readable "node$i" in log and error messages is left as prose.
The config template is not rendered by any script, so its DATA_DIR
documentation gains a note about the same constraint instead.
Also rename the deprecated otlphttp/filelog collector component names in the
harness scripts and docs.
Alloy carried no log pipeline at all: no loki.source, loki.write or
loki.process anywhere in the config, against a working metrics and OTLP path.
Any deployment fed through Alloy rather than the reference collector therefore
sent traces and metrics but no logs, so log-to-trace correlation was
unavailable there even though both ends of the link were configured.
Logs now leave through the same otlphttp exporter as the other two signals, so
all three carry one resource identity. Alloy's own filelog receiver would have
been the closest match to the reference collector, but it is still
public-preview and refuses to load unless the service is started with
--stability.level=public-preview, which would mean editing the unit on a box
reachable only by RunCommand. The Loki source components are generally
available, so they are used and bridged into OTLP by otelcol.receiver.loki;
the service needs no extra flag.
That bridge hands over an empty resource and puts everything on the log
record, so the transform sets the resource attributes in log context.
service.instance.id is concatenated in from XRPLD_HOST_LABEL because OTTL has
no env() converter, and it is a resource attribute rather than a record one
because only resource attributes are promoted to indexed Loki labels. It must
equal the node's own service_instance_id or the logs join nothing. devnet
writes one flat file rather than a per-node directory, so identity cannot be
read off the path the way the docker collector does it; XRPLD_LOG_GLOB
overrides the path for other layouts.
The line is parsed for its own timestamp, severity and trace context, and
trace_id/span_id are set on the first-class OTLP record fields so Grafana
links a log to its trace without re-parsing the body. Lines emitted outside a
sampled span keep an empty trace id rather than an invalid one.
Also carry the node-identity operators into the Grafana Cloud collector
variant, align this config's log directory with its service_instance_id, and
rename the deprecated otlphttp/filelog collector component names. Alloy's
otelcol.exporter.otlphttp is that product's own name and is unchanged.
Addresses the open review findings on this branch.
The log root was never delivered at all. Docker creates a missing bind-mount
source as root, Config::getDebugLogFile() only warns when it cannot create the
network subdirectory inside it, and Application carries on. The node therefore
looked healthy while writing no debug.log, and Loki stayed empty with no error
at any layer. docker/telemetry/data/logs has in fact been root-owned in a
working checkout since it was first created. A one-shot xrpld-logdir-init
service now creates the directory and hands it to XRPLD_UID/XRPLD_GID,
following the pattern the storage-init service already uses.
Ingested logs carried no node identity, so a multi-node stack collapsed into
one indistinguishable stream while every dashboard filters on
service_instance_id. The receiver now sets include_file_path and lifts the
per-node directory onto the resource attribute service.instance.id, which is
on the allow-list Loki promotes to an indexed stream label. A record attribute
would only become structured metadata and could not be used in a selector.
For that to join anything the directory name has to equal the emitter's
service_instance_id, so the node directories are renamed to match: node$i
becomes Node-$i, and the standalone config writes to logs/xrpld-standalone.
The integration test aborted before reporting. Under set -o pipefail the
grep | head -1 pipeline is killed by SIGPIPE once the log exceeds the pipe
buffer, so the run exited 141 somewhere past a few hundred matching lines and
read as a flaky test. grep -m1 stops on its own. The test also verified the
local file and Tempo but never that a line reached Loki, which is the one hop
this branch adds, so a bounded Loki assertion is added alongside a readiness
wait.
Documentation fixes: the Tempo cross-check counted .data, but Tempo returns
OTLP shape so the array is batches and one trace can span several; the Loki
step used the instant /query endpoint, which rejects a bare log selector with
HTTP 400 and a text/plain body, so jq could never parse it and the step never
printed a number even when ingestion worked. The filelog comment claimed six
fractional digits where the node always emits nine. The two flowcharts used
<br/>, carried no legend, and advertised GetSpan(), which Log.cpp deliberately
avoids in favour of reading the thread-local context directly.
Finally, rename the deprecated collector component names: the pinned
collector warns on every start that otlphttp and filelog are aliases for
otlp_http and file_log. Alloy's otelcol.exporter.otlphttp and
otelcol.receiver.filelog are that product's own component names and are not
deprecated, so they are left alone.
The health_check extension was present on the previous branch and dropped
here with no replacement, while this branch's own TESTING.md still polls
http://localhost:13133/ to decide the collector is ready. That check has had
no listener since, so the documented readiness step cannot pass.
Also add batch to the metrics pipeline. Without it the OTLP metric path
exports one request per instrument; the added delay is bounded by the batch
timeout, well under the Prometheus scrape interval.
Both belong here rather than downstream: this branch owns the metrics
pipeline and is the one that regressed the extension.
Two files conflicted and both were composed rather than taken from one side.
integration-test.sh: kept this branch's spanmetrics names, since the collector
sets namespace: "span" here and traces_span_metrics_* matches nothing, and took
phase-6's --max-time on every probe. The Tempo time bound needed restoring by
hand: RUN_START, the check_span guard and the start/end parameters are on
phase-6 and absent here, so a plain resolution kept phase-6's comment about
bounding the search while shipping no bound. All four pieces are back.
TESTING.md: kept server=otel with the metrics endpoint. Phase-6's template sets
server=statsd and documents prefix, which this branch's OTel path ignores.
The two writer paths run on the consensus and ledger-publish threads, and both
took a single mutex that the nine window getters also held while scanning their
deques. At a week of 4-second ledgers those deques hold about 151,000 records,
the same record stored three times, so a read walked half a million entries and
the writers waited behind it.
Each writer now owns a 128-slot ring and shares nothing with the other, so a
record is one timestamp read and one store. A full ring drops the event and
bumps droppedEvents() rather than blocking a consensus thread; at 4-second
ledgers that needs about eight and a half minutes with no drain, against a
worst normal drain gap of one export interval plus one export timeout.
The windows become a grid of one-minute buckets with a running agreed and total
per span, so a getter reads a published snapshot instead of counting. Missed is
derived from the two, which leaves a late repair touching only the agreed side.
Steady-state footprint drops from about 8 MB to about 91 KB, and window edges
quantise to a minute.
Whichever thread is inside reconcile() owns the decision state; a second caller
returns instead of waiting. Readers take a shared_ptr to the snapshot, so the
values they read cannot be overwritten underneath them.
The clock is now injected, which is what makes bucket expiry reachable in a
test. The suite covers pairing, single-sided misses, the grace boundary, late
repair across a bucket edge and past the window, each window edge exactly, two
grid lengths of steady traffic, ring overflow, and three real-thread cases.
It no longer sleeps: nine sleep_for calls totalling 81 seconds are gone.
Every public method keeps its name and signature.
MetricsRegistry::start() takes a StartOptions aggregate. The two telemetry-off
tests this branch adds still passed a string literal, which cannot bind to
StartOptions const&, so the #ifndef XRPL_ENABLE_TELEMETRY block did not
compile. Nothing in CI compiles it: telemetry defaults ON, and the block is
skipped whenever the macro is defined.
Add kTestStartOptions carrying just the endpoint, and pass it at both sites.
Its text is identical to the constant phase-9 introduces, so merging phase-9
forward is an identical addition rather than a second constant with the same
meaning. The six call sites phase-9 owns are left for that merge.
MetricsRegistry::start() takes a StartOptions aggregate. Six call sites in
the #ifndef XRPL_ENABLE_TELEMETRY block still passed a std::string, so the
block did not compile. Nothing in CI compiles it: telemetry defaults ON, and
the block is skipped whenever the macro is defined.
Add one shared kTestStartOptions carrying just the endpoint -- the other
fields are never read on the no-op path -- and correct two comments that
still described the old signature and a #else stub that does not exist.
The workload README contradicted itself on --skip-loki: one bullet said CI always
passes it and so the two log-correlation checks are never exercised, another said
the workflow no longer passes it. The workflow mentions the flag nowhere, so the
first was the stale half.
Other claims checked against the tree and corrected:
- both the README and the plan doc described the push trigger as filtered on
branch names. The workflow has no branches filter, deliberately, because
GitHub ANDs branches with paths
- the plan doc printed 6 of the workflow's 12 paths globs, and claimed the
workflow was 367 lines against an actual 451. The glob block is now generated
from the workflow, and the line count dropped rather than restated
- rpcNOT_SUPPORTED does not exist anywhere in the tree. The symbol is
RpcNotSupported, and the refusal sites are RipplePathFind.cpp:59-60 and
PathFind.cpp:50-51, not :48-49 and :39
- RCLConsensus.cpp:666 and :663 are not log or event lines; the tx.included event
is at :720 and the per-transaction debug log at :715
- LedgerMaster.cpp:463 is fixIndex, not the ledger.store span, which is at :470
- ServerHandler.cpp:705 is inside makeJsonError; processRequest is at :718
- file counts: docker/telemetry/workload/ is 25 files, include/xrpl/telemetry/ 13
- the optional-span bullet named five causes covering 10 of 16 entries, omitting
the txq.* family and the WebSocket handshake
- the /api/v1/series choice was attributed to stale StatsD gauges; this harness
runs no StatsD
A line number in run-full-validation.sh was cited in five places and drifts on
every edit to that file, so those now name the file only. The keygen helper's
header records what production does instead -- validator-keys-tool create_keys
then create_token, keeping the master key off the node -- and why a disposable
cluster does not.
The four external-parity bounds checks each ran a single Prometheus instant query
and failed on an empty result. The metric checks that run earlier poll
/api/v1/series, which returns a series regardless of staleness, but a bounds
check needs the sample value and so cannot use that endpoint. This file's own
docstring records the consequence: a beast::insight gauge that stops changing can
fall out of an instant query while /api/v1/series still returns it, so one
attempt is not enough to call the series absent.
Poll to the same deadline the metric checks use. A Prometheus error is raised
rather than retried, because a rejected query never becomes valid and retrying it
only burns the full timeout.
compare_to_baseline took the unit from the baseline entry and dropped the current
run's, and nothing compared the two, so a us -> ms change was scored as a numeric
delta: four keys rewritten to the same physical durations reported 99.9%
improvements and the gate exited 0. prom_queries.py says the baseline preserves
the unit "so the comparator can sanity-check unit drift"; it never did. A unit
mismatch now fails and names both units.
The workflow's step summary printed total, regressions and improvements. total is
every key in the report -- the union of baseline and current -- so it was neither
the baseline count nor what was gated, and missing_in_current was computed and
never printed. A run that gated 16 of 20 keys read as a full comparison. The
comparator now reports a real "compared" count and the summary prints it beside
the not-captured count, with a warning when any key was missed. The table also
refused nothing on a truncated report; existence is not readability.
check_regression_bounds told the operator to add max_abs_increase while reading
max_abs_increase_ms / _us, so following the message added a key nothing reads and
the gate kept failing with no explanation. The committed thresholds use only the
suffixed spelling, so the message was the defect. Its three JSON inputs were also
unchecked: a top-level null, list or number parsed and then died on the first
.get, and a string "metrics" survived the placeholder test and reported its own
characters as gated keys -- wrong advice rather than a crash.
Four tests cover these; all four fail against the previous checker.
The overhead benchmark generated no workload. Each arm was start_cluster ->
collect_metrics -> stop_cluster, and collect_metrics only ran the sampler, so the
only client traffic was the sampler's own server_info probes at under
1 request/sec. The hottest instrumented paths -- tx.*, txq.*, the transactor
stage spans, every rpc.command.* other than server_info -- were never entered,
which is where per-operation span cost appears. Both arms now drive
rpc_load_generator and tx_submitter at one fixed rate for the whole window, over
a [port_ws] listener present in both arms so the listener is not part of the
delta. A flat rate rather than a workload profile, because both arms must issue
the same work and a profile's phase shaping only adds variance.
The sampler also selected xrpld host-wide. run-full-validation.sh leaves its five
validation nodes running while the benchmark's three start, so both arms averaged
eight processes -- diluting the CPU delta and making memory_rss_mb_peak report a
validation node either way. It now takes an optional pid list, and the benchmark
passes its own nodes' pids and refuses to measure if it cannot collect them all.
consensus_round_mean_ms counted distinct ledger sequences seen by a loop that
sampled every 5 s, so it read back 5000 ms for every close time from 2 s to 5 s
and a 10% regression measured 0%. Sampling at 2 s -- the close-time floor from
ConsensusParms.h:93 -- resolves a 10% regression as at least 9.3%. It also
divided by the requested DURATION rather than the measured ELAPSED, which the
TPS calculation in the same file already used.
Key generation, the workdir setup and the seed read exited 1 under errexit, the
code this script reserves for a measured threshold breach, so an infrastructure
failure was reported as "telemetry is too expensive". They map to cannot_measure
now. No guard is added after the config heredoc: a guard there is read as the
heredoc's first line, lands in the generated config and never runs.
curl probes across the harness had no --max-time, so a server that accepts the
connection and then stops answering blocks forever and the loops' attempt counts
stop bounding anything.
Three ways a run could produce no traffic and still report success:
- tx_submitter logged a funding shortfall and returned an empty stats object;
main() then printed the summary and exited 0, so the failure only surfaced
later as "spans missing", which points nowhere. It now records setup_failed in
the summary and exits 1 after the report is written.
- --weights was checked for valid JSON but not for a positive sum. An all-zero
mapping reached random.choices, which raises ValueError from inside the
dispatch loop where only CancelledError is caught. Rejected at parse time now,
in both generators.
- a profile phase declaring neither rpc nor tx logged a warning and returned no
error. Both error rates short-circuit to 0.0 when nothing was sent, so a
mistyped key produced zero traffic and still passed the exit gate. That phase
is now an error.
Every citation below was checked against the file it names:
- LedgerMaster.cpp:463 is fixIndex, not the ledger.store span; that guard is at
:470 and the insert it wraps at :476
- LedgerMaster.cpp:987 is the tvc assignment, which sits BEFORE the tvc < minVal
return at :988; the ledger.validate span opens at :1003
- ServerHandler.cpp:705 is inside makeJsonError; processRequest is at :718
- docker-compose.yml:71 and :75 are comments in the collector's volume block;
the loki service is at :112 and its config command at :116
Two claims were also wrong rather than merely stale. Log-trace correlation is
gated in CI, because the workflow passes no --skip-loki, and the separate check
in integration-test.sh is run by no workflow at all. The Loki label note
described the Grafana Cloud collector config rather than the local one: only the
cloud variant sets job=xrpld, and the local config's own comment says to select
on service_name. The dashboards carry 35 Loki queries, not 38.
This branch switches integration-test.sh to [insight] server=otel and adds an
assertion that no StatsD listener is needed, but TESTING.md still described the
metrics it verifies as StatsD-derived and its manual node-config template had no
[insight] stanza at all.
CollectorManagerImp falls through to NullCollector when server is neither statsd
nor otel, so a reader building configs from that template got zero
beast::insight metrics. The template also omitted service_instance_id and
metrics_endpoint, which the script writes; without the former every node is
indistinguishable in the $node dashboard filter.
curl applies no overall timeout of its own, so a server that accepts the
connection and then stops answering parks a poll loop for the rest of the run
and the loop's attempt count stops bounding anything. Add a CURL_MAX_TIME
ceiling and apply it to all 18 executable probes in integration-test.sh.
TESTING.md's manual node-config template also disagreed with what the script
writes, so a reader following it could not reproduce the automated path:
- no [insight] stanza, so no beast::insight metric leaves the node at all and
Step 10b's ten rippled_* assertions cannot pass
- [ips_fixed] listed all six peer ports including the node's own
The log level is deliberately untouched: the template and the script agree on
warning here.
With tls_client_cert set, only traces_endpoint was checked for an https
scheme. Telemetry::makeMetricExporter() attaches the client certificate and
key to the metric exporter whenever use_tls=1, and metrics_endpoint defaults
to a plain http URL, so an operator who set up mTLS and overrode only
traces_endpoint exported every metric in the clear with the configured client
identity unused.
Check both endpoints, and state the requirement under metrics_endpoint and
tls_client_cert in the example config. Four config tests cover an explicit
http metrics endpoint, the omitted-key default, both endpoints on https, and
a one-way-TLS control that must stay accepted.
A StatsDCollector polls its metrics only after onCollectionReady(), so
neither test flushed anything: the gauge test timed out and the counter test
passed for the wrong reason. Call it in both, and drop the two includes
whose symbols the file never names.
One conflicted file, docs/telemetry-runbook.md, with three spots:
- Build section: both sides added different text at one point. Kept both,
incoming sentence first, then this branch's "Run against a live network".
- Disabling section, first spot: this branch's wording names the config
section and says no rebuild is needed, so it already covers the incoming
sentence.
- Disabling section, second spot: kept this branch's paragraph and folded in
the one point it lacked, that both flags have to be passed.
Three conflicts, all resolved by keeping this branch's rewrite and
re-applying the incoming change onto it:
- 09-data-collection-reference.md: phase-7 rewrote both attribute tables,
so the incoming table would have reverted them. Kept phase-7's and
re-applied the two "XRPL epoch" spellings.
- integration-test.sh: phase-7 moved these checks from StatsD to OTel and
no longer defines check_statsd_metric, so only this side compiles.
- TelemetryConfig.cpp: the incoming side carried networkTypeFromId(), which
this branch already has. Kept one definition and took the incoming
doc wording, which the auto-merged body below it already matches.
getConsensusTraceStrategy() returns ConsensusTraceStrategy, so comparing
it to a string literal does not compile.
No coverage is lost: strategyName()'s spelling has its own assertions in
the TelemetryConfig test, in this same binary.
readability-identifier-naming wants lower_case for a namespace, so
clang-tidy failed on this file under warnings-as-errors. All seven use
sites move with the declaration.
The rename script rewrites "Ripple epoch" to "XRPL epoch", so the old
spelling in a tracked .md makes the check-rename job fail on a dirty tree.
The attribute keys are left alone: the script's pattern needs a space, and
those keys are a cross-layer contract.
The rename script rewrites "Ripple epoch" to "XRPL epoch", so the old
spelling in a tracked .md makes the check-rename job fail on a dirty tree.
The attribute keys are left alone: the script's pattern needs a space, and
those keys are a cross-layer contract.
