Static constants take the k prefix (readability-identifier-naming), and
SpanGuardScope.cpp no longer names anything from <opentelemetry/metrics/noop.h>
since it calls noopMeter(). Both fail CI under warnings-as-errors.
The helper's docstring also claimed NoopMeterProvider hides the base
two-argument GetMeter; it declares that overload itself, so the only
detail worth sharing is the version.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
clang-tidy's include-cleaner flags Counter.h and Gauge.h here: the test
names neither type, so the includes fail CI under warnings-as-errors.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
stop() destroys the MeterProvider, and with it every View's
AggregationConfig. The SDK's SyncMetricStorage keeps a raw pointer to
that config, and the call-site statics keep the storage alive, so a
histogram record with a first-seen attribute set during the shutdown
drain would dereference freed memory. Application::run() stops the
registry before the job queue and server handler, so that window is
real.
phase_ is now atomic and stop() stores Stopped before tearing down.
Every XRPL_METRIC_* macro and every record*/increment* method checks
recording() (enabled and not stopped) instead of isEnabled(). meter_ is
never written after construction, so record threads read it without a
lock.
Also: an empty [telemetry] service_instance_id now falls back to the
node key on both the trace and the metrics side, so one node reports one
identity; disablePipeline() uses telemetry::noopMeter(); comments
corrected.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Lift the seed parsing and the stored-vs-mint choice into two libxrpl
helpers, parseNodeIdentitySeed() and selectNodeIdentity(), so xrpl_tests
can drive each branch without an xrpld Config. resolveNodeIdentity() now
marshals Config and the cmdline into them; behaviour is unchanged.
Also pin that storeNodeIdentity() appends (row count, not SQLite row
order), fix the test header that described getNodeIdentity()'s property
as the store's, and route NullTelemetry::getMeter() through noopMeter().
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Merges pratik/otel-phase8-log-correlation into pratik/otel-phase9-metric-gap-fill.
Auto-merged. TESTING.md picked up phase-8's Test 4 rewrite alongside this
branch's Step 2 standalone-store fix. OTelCollector.cpp gained the weak_ptr
gauges_ list; the StatsD test gained its onCollectionReady() calls.
Merges pratik/otel-phase7-native-metrics into pratik/otel-phase8-log-correlation.
Conflict was one TESTING.md hunk under "Nodes not reaching proposing state":
this branch renamed the node directories to Node-N in integration-test.sh,
phase-7 kept nodeN and expanded the [peer_private] explanation. Resolution
keeps this branch's Node-1 path (its own script uses that naming) and
phase-7's fuller prose citing peerfinder/Config.cpp.
Non-conflicting phase-7 changes come through: OTelCollector's gauges_ list
becomes weak_ptr, matching the earlier hooks_ change; the phase-6 revert of
the StatsD-test onCollectionReady() calls resolved against phase-7's version
that keeps them.
Merges pratik/otel-phase6-statsd into pratik/otel-phase7-native-metrics.
Phase-6 dropped the three onCollectionReady() calls that had been added to
its StatsD test, because that method is only declared here on phase-7.
This branch's own copy of the file was unchanged from the merge base, so
the default merge would have silently deleted the calls from here too —
where they are needed, because this branch gates polling behind
onCollectionReady() in OTelCollectorImp::onTimer.
Resolution keeps both sides: phase-6's two new include lines
(Counter.h, Gauge.h) and phase-7's three onCollectionReady() calls plus
their doxygen and inline explanations. The merged file is exactly
phase-7's tip plus those two includes.
TESTING.md auto-merged cleanly; both sides added text under Test 1 in
different regions.
nodeIdentity_ is no longer a std::optional, so setNodeId() must read it
directly. The merge could not flag this: phase-8 changed the member's type and
this line lives only on phase-9, so neither side of the merge touched the same
file region.
Three conflicts, all from both branches editing the same passage:
- Main.cpp: kept phase-9's wording. The metrics registry only exists here, so
"unwinding destroys little: the metrics registry, whose destructor joins its
export thread" is the true statement on this branch.
- TESTING.md: kept both paragraphs. They document different things (the
private [network_id], and the log path plus log_level).
- 05-configuration-reference.md: composed both. The identity is now resolved
before construction and never empty, so every producer stamps the node key
from the start; the only divergence left is a wallet that already holds a
different key, which corrects the tracer alone. Rewrote the earlier
"three producers" blockquote too: its "no fallback", "first boot ... left
off" and "Known issue" claims are what this change removes.
One silent break the merge could not flag: makeMetricsRegistryOptions() took
the std::optional<std::string> node key that used to be a constructor
parameter, and that parameter is now the resolved keypair. It takes the base58
string directly, and the constructor derives it from nodeIdentity_, which is
declared before both telemetry_ and metricsRegistry_.
gauges_ was still a vector of raw pointers after the hook list moved to
weak references. onCollectionReady() and onCollectionStopping() copy that
list under mutex_ and then call arm()/disarm() with the lock released,
because both enter the SDK's observable registry lock. ~OTelGaugeImpl only
re-acquires mutex_ to erase its own entry, so nothing stopped a gauge from
being destroyed between the snapshot and the dereference.
Store weak references, for the reason the hook list already does: locking
an entry keeps that gauge alive for exactly its own arm or disarm call, and
one destroyed since the snapshot locks to null and is skipped. Registration
moves from the OTelGaugeImpl constructor to makeGauge(), because no
weak_ptr to the object exists until the owning shared_ptr does, and the
destructor now prunes by expiry rather than by address.
The startup log line counts the gauges that were still live rather than the
snapshot length, so a skipped entry is not reported as a failed
registration.
The SDK callback path is unchanged and does not need this: gaugeCallback
takes a void* and is guarded by RemoveCallback being synchronous, as
~OTelGaugeImpl already documents.
resolveNodePublicKey() returned std::nullopt in three real cases: a first boot
with no wallet database, a standalone run (its wallet is a private temporary
database), and --newnodeid. Telemetry's resources are built during
ApplicationImp's member-init list and are immutable, so on those runs the node
reported an empty service.instance.id and no xrpl.node.id for the whole run,
while setup() minted a key moments later and patched only the tracer.
Replace it with resolveNodeIdentity(), which always returns a keypair: derived
from a configured seed, else read from an existing wallet database, else
minted. Main.cpp passes that pair to makeApplication(), ApplicationImp stores
it in nodeIdentity_ -- now declared before telemetry_ and no longer an optional,
because it is always set -- and builds the telemetry resource from it.
setup() calls getNodeIdentity(), which now persists rather than mints: it
stores the resolved pair when the wallet holds no identity, adopts the stored
one when it does, and clears first for --newnodeid. The write stays in setup()
because that is where the database exists; a standalone run has no persistent
wallet to write to, which is why the pair has to be decided before
construction rather than read back afterwards. Wallet gains storeNodeIdentity()
for that write, and getNodeIdentity(session) now uses it instead of repeating
the insert.
The three-argument makeApplication() mints a keypair, so jtx::Env and any other
test Application behave as a standalone run always did.
Also fold the three hand-rolled "meter from a NoopMeterProvider" copies into
telemetry::noopMeter(): the base-pointer call and the kMeterVersion argument are
both easy to get wrong alone, and the meter identity has to match the one the
histogram views select on.
The new gtest covers the wallet half: store-then-read, store not replacing an
existing identity, clear-then-store, and that the mint path persists. It adds
the tests.libxrpl > xrpl.rdb levelization edge, regenerated here.
Test 1 pointed standalone at docker/telemetry/xrpld-telemetry.cfg, whose
[node_db], [database_path] and [debug_logfile] all resolve under
docker/telemetry/data. Standalone builds its own private chain, so that
left one NuDB holding two unrelated chains. This file already states the
rule for the key generation node in Test 2, and the sibling mainnet config
keeps its store under data/mainnet/ for the same reason.
Derive a standalone config with the three paths redirected under
data/standalone/, and run from that.
Note why the flag is not the problem: --start selects StartUpType::Fresh,
but the default Normal reaches startGenesisLedger() through the same branch
chain in ApplicationImp::setup, so any standalone run writes a genesis
ledger into whichever store the config names.
MetricsRegistry created its provider and synchronous instruments in start(),
called from setup() after the node identity was read. Every XRPL_METRIC_*
call site creates its instrument on first use, so any site that ran before
that point found no meter and never recorded again. The start was moved three
times to chase the newest early caller; nothing guaranteed the order.
Build the pipeline in the constructor instead. ApplicationImp declares
metricsRegistry_ right after telemetry_ and before every subsystem, so
declaration order now guarantees the instruments exist before any producer.
start() is gone and its config parsing moves to makeMetricsRegistryOptions().
The registry now guarantees a meter whenever it is enabled: the real one, or
the OTel no-op meter if the pipeline failed to build. disablePipeline() owns
that fallback and its one error log, and the constructor routes both
std::exception and a non-std throw through it, because the SDK is third-party
code. So the macros shrink to one function-local static built from meter()
plus the record call: no once-flag, no null check, and no path for a call
that arrives before the meter, because that state no longer exists. The three
observable macros drop the same now-dead meter check.
The lifecycle is three explicit phases with a Phase enum: Ready at
construction, GaugesArmed by startAsyncGauges() once overlay_ exists, and
Stopped by stop(). startAsyncGauges() checks the phase before the pipeline,
so a second call and a call after stop() are each reported as what they are.
run() stops both observers (insight collector, registry) before any service,
and ~ApplicationImp repeats the stop for the setup() failure paths that never
reach run(). stop() already detaches the callbacks, so it is the only call.
Meter name and version come from kMeterName/kMeterVersion, and the endpoint
default from Telemetry::Setup, so the two metric pipelines share one source
for both.
0eb4291688 added collector->onCollectionReady() to both StatsD tests. That
method does not exist on this branch: it is introduced later, alongside the
polling gate it belongs to, so all four build legs and clang-tidy failed.
Nothing gates polling here. The StatsDCollectorImp constructor starts its
thread, run() calls setTimer() unconditionally, and onTimer polls metrics_
and drains the buffers every second. The gauge test's expectation already
holds without any lifecycle call, because StatsDGaugeImpl starts dirty so a
untouched gauge emits its zero on the first flush.
Restores the two includes that commit also dropped.
The collector readiness note claimed docker-compose.yml publishes only 4317,
4318 and 8889 and that 13133 comes from a workload stack. It publishes 13133,
and that stack is not part of this branch. Probe health_check on 13133 and drop
the note; the troubleshooting entry now points at the same check instead of
carrying a second, weaker copy.
Stop restating BUILD.md. The hardcoded conan and cmake lines had drifted from
it, -Dtelemetry=ON is redundant because the Conan toolchain carries it, and the
conan-release preset resolves only from the repo root, builds into
.build/build/Release rather than .build, and sets no -Dxrpld=ON. Defer to
BUILD.md and docs/build/telemetry.md.
Test 2's keygen step reused the Devnet config with -a --start, which wrote a
genesis chain into the Devnet store, took RPC port 5005 from node 1, and was
followed by an rm -rf that also destroyed the mainnet node's store and every
log. Give it its own config under the test's temp root, as the script does.
The manual path also needs XRPLD_LOG_DIR, or the collector tails the wrong root
and Test 3 finds nothing without erroring.
Neither the template nor the script set [network_id], so a local cluster
stamped xrpl.network.type=mainnet and shared dashboard series with real mainnet
data. Set a private id in both, and say which label it produces. Also drop a
duplicate metrics_endpoint from the generated config.
Split the consensus trigger row: six families fire on a standalone
ledger_accept, and the remaining seven need the establish phase, a validator
key, or a peer. ledger.validate needs peers too, because checkAccept is
unreachable in standalone. Correct the trace-id note to 16 bytes, and name the
strategy it depends on.
The pathfinding bullet said raw account values reach Grafana Cloud. Both
accounts are already tokens when they leave the node; what differs is that the
base config hashes them a second time, so one account carries two tokens across
configs and traces must not be joined across them.
Also: the Loki allow-list is a fixed 18 keys on the pinned image with k8s and
cloud enumerated rather than wildcarded, the runbook documents 9 of 15
dashboards, and the spanmetrics block now uses one spelling with a note that
the cloud config uses the other.
The config template wrote each node's log to a lowercase node{N} directory
while setting service_instance_id=Node-{N}. The collector takes the node name
from the log file's parent directory and stamps it as the Loki
service_instance_id label, so the logs carried a name no trace or metric
shared and nothing joined. Use Node-{N} and state the rule.
The template also set log_level to warning. Nothing in the pipeline filters on
severity; the constraint is that a log line carries trace context only when it
is emitted inside an active span. At warning the only such statements in the
consensus accept span are a catch path a healthy round never takes and a
periodic censorship warning. At info the CNF Val / CNF buildLCL pair writes one
line per accepted ledger, which is what makes this test's Step 1 findable.
Grafana 13 offers the link per span, labelled "Logs for this span", in the
span's Links row — not per trace. Fix the step and the expected-results row.
The example log line quoted a message that does not exist. The real in-span
RPC statement logs at debug, so the severity code is DBG; say which line to
look for under each test, since Test 2 now logs at info.
Drop the reference to workload/validate_telemetry.py: that file is not part of
this branch, and its instant-endpoint call uses seconds, so the nanoseconds
claim applied only to query_range.
On the OTel path only [insight] server is load-bearing. CollectorManager reads
endpoint and hands it to OTelCollector, which logs it at startup and routes
nothing with it; the real export endpoint is [telemetry] metrics_endpoint,
which the template already sets. service_instance_id and service_name in that
section are read and discarded.
Leaving the line invited an operator to reconcile a mismatch that has no
effect. integration-test.sh already emits only server=otel with the same
explanation, so the two now agree.
The Payment destination was not a valid XRPL address — its base58 checksum
does not match — so Test 1 Step 4 and Test 2 Step 7 could never have returned
the tesSUCCESS they claim. Use a valid one and note that the destination does
not need to exist.
The Tempo search loop had no -G, so curl posted the parameters as a body,
Tempo answered 200 while ignoring the query, and every span name came back
non-zero. It also had no time bound, and Tempo keeps blocks for an hour, so a
re-run was answered by the previous run's traces. Add -G, RUN_START, and
start/end, matching what integration-test.sh already does.
Split the query list in two: 35 names that should be present, and 8 that need
a trigger neither test performs, where zero is the expected answer. Previously
two of the latter sat in the pass/fail list and read as failures.
Correct the standalone span table. consensus.mode_change fires once per round
start whether or not the mode changes, ledger.validate cannot fire because
checkAccept is unreachable in standalone, and the apply-stage, TxQ and ledger
families were missing rows. Give each "No" row the reason that actually
applies: the establish phase, a missing validator key, or no peers.
Also: ledger_accept is not required before submit, the teardown pgrep matched
more than this node, [peer_private] also disables the inbound listener, and
the 15-second wait covers Tempo but not Prometheus.
The `service_name`, not `job` note pointed at three file:line locations.
All three had drifted, because the cited files move on every merge forward
and nothing checks the references. Name the `resource/logs` processor and
the `loki` service instead — those survive line drift and a rename breaks
a grep loudly.
Grafana keys a dashboard by uid, so the Phase 9 and Phase 11 rows both
claiming `validator-health` meant one would silently overwrite the other.
Phase11_taskList.md already requires `validator-health-external`; the
reference table now agrees with it, and says why.
Grafana reaches the image renderer over the compose network at
http://renderer:8081, so the host publish gave nothing the stack needs.
AUTH_TOKEN is the only guard on the endpoint and its default is a fixed
string in this file.
Update the service table in the configuration reference to match.
callHooks() copied the hook list into a vector of raw pointers, released
mutex_, then dereferenced them. It has to release the lock: a handler may
drop the last reference to a hook, and ~OTelHookImpl re-acquires mutex_,
so invoking handlers under the non-recursive lock would deadlock. That
left a window in which an entry could be freed before it was used.
The window is not reachable today. Every hook belongs to a long-lived
ApplicationImp member, and onCollectionStopping() runs before those
members are destroyed, both from stop() and from the destructor body.
That call disarms each gauge via RemoveCallback, which blocks until an
in-flight callback finishes, because the SDK holds its registry mutex
across the callback. Safety therefore rests on four separate facts, none
of them enforced by a test, one of them internal to a vendored library.
Store weak references instead, so the code is correct by construction:
locking an entry keeps that hook alive for exactly its own handler call,
and a hook destroyed since the snapshot locks to null and is skipped.
Registration moves from the OTelHookImpl constructor to makeHook(),
because no weak_ptr to the object exists until the owning shared_ptr
does, and the destructor now prunes by expiry rather than by address.
Add three GTests over the real collector. A test-local MetricReader
drives one synchronous collection pass, since the SDK ships only a
threaded periodic reader. They assert a live hook runs, a destroyed hook
is skipped, and destroying one hook leaves its siblings registered --
the last pairing both directions so neither can pass vacuously.
Not yet run: verifying them needs a telemetry-enabled build of
xrpl_tests. Compile, clang-tidy and the pre-commit gates are clean.
Four doc conflicts, all where this branch had rewritten a passage that upstream
had only renamed. This branch's text is kept in seven of the eight hunks and
the spanmetrics -> span_metrics and otlp/tempo -> otlp_grpc/tempo spellings
carried into it, so the rewrite is not lost and the names stay current.
The exception is the TESTING.md span-call-count comment, where the incoming
side is the fuller text: it explains that the span_ prefix comes from the
connector's namespace setting. That side is taken.
Metric names are untouched — span_calls_total and traces_span_metrics_* are
produced by the connector's namespace, not by its component name.
Two conflicts, both where this branch's log-pipeline additions sat next to the
upstream spanmetrics -> span_metrics rename: the config header comment, which
this branch extended with a logs line, and the integration test, where the
log-correlation step precedes the span-metrics step. This branch's content is
kept in both and the rename carried into it.
Four conflicts, all where this branch's replacement of the StatsD path with
native OTLP met the upstream spanmetrics -> span_metrics rename. This branch's
design wins in every case; the rename is carried into its text rather than
reverting it, so the connector, its pipeline references, the header comment,
the TESTING.md summary and the runbook all use span_metrics while keeping the
native-OTLP wording.
One addition beyond a straight take-a-side: publish the collector's health
check port. This branch restored the health_check extension and its own
TESTING.md polls http://localhost:13133/ to decide the collector is ready, but
the port was never published on this side of docker-compose.yml, so that check
could not pass from the host. Verified the merged config loads with no
deprecation warnings and that 13133 is published exactly once.
No metric name changed: traces_span_metrics_* already read that way before the
rename, which only ever touched component names and prose.
Conflict in docker/telemetry/otel-collector-config.yaml, in the service
pipelines: this branch renamed the deprecated spanmetrics connector to
span_metrics and adds the statsd metrics pipeline, while upstream renamed the
deprecated otlp exporter to otlp_grpc. Both kept.
With both renames present the collector now starts with no deprecation
warnings at all, which was the point of the pair.
Conflict in docker/telemetry/otel-collector-config.yaml, in the traces
pipeline: this branch added the attributes/hash processor while upstream
renamed the deprecated otlp exporter to otlp_grpc. Both sides kept — the
processor list keeps attributes/hash and the exporter list takes the new name.
Checked that the exporter definition key was renamed to match the reference,
and that the collector still loads the merged config.
`docker compose down` keeps the `tempo-data` named volume, so a previous
run's traces stay in Tempo and can satisfy this run's span searches. Pass
-v in cleanup(), and tear the stack down before starting it: a run that
reaches the summary deliberately leaves the stack up, so nothing else
clears it.
The three tracker files conflicted because both sides had rewritten them. Took
the incoming lock-free class, then put this branch's two monotonic accessors
back on top of it: totalAgreementsEver() and totalMissedEver(), backed by a
gross pair incremented at first classification and left alone by the repair
branch. MetricsRegistry reads both, so dropping them would not compile.
The test file kept the incoming suite, which renames every case, and gained
this branch's two gross-counter cases adapted to the injected clock.
Three CI failures, one cause each.
macOS could not compile the tracker: Apple's libc++ has no std::atomic for a
shared_ptr, so the primary template's trivially-copyable assert fired. Publish
through boost::atomic_shared_ptr instead, which every standard library the
matrix covers can build. boost/smart_ptr is already used in this tree. The
header's note no longer claims libstdc++ as the assumption.
Four tests that predate the metrics_endpoint scheme guard set tls_client_cert
and only put traces_endpoint on https, so the new guard threw before the check
each one asserts. They now set both endpoints. Nine tests in the two files set a
client cert; the other five stay correct because the pairing, use_tls and traces
checks all run ahead of the metrics one.
Eleven clang-tidy findings: redundant member initialisers, two aggregate
initialisations that wanted designated form, two unbraced bodies, three
unparenthesised multiplications, and a reserve before a loop that emplaces.
Dropping std::make_shared also left <memory> unused in both files.
The pinned collector warns on every start that "spanmetrics" is a deprecated
alias for "span_metrics". Rename the connector, its pipeline references and
the prose that names it.
The derived metric names are untouched. They come from the connector's
`namespace` setting rather than its component name, so occurrences inside a
metric name such as traces_spanmetrics_calls_total are deliberately left as
they are; renaming those would break every span panel. The rename is applied
only to the bare word, never where it is joined to a metric name by
underscores.
This branch introduces the connector, so the change belongs here.
The pinned collector warns on every start that "otlp" is a deprecated alias
for "otlp_grpc". Rename the trace exporter to otlp_grpc/tempo.
Only the exporter is affected. The otlp RECEIVER keeps its name: it serves
both gRPC and HTTP under one component and is not deprecated, verified by
renaming the exporter alone and seeing the warning stop.
This belongs on this branch because it introduces the exporter, and it is the
last of four deprecated aliases in the collector config; the other three are
owned by later branches in the chain.
Alloy carried no log pipeline at all: no loki.source, loki.write or
loki.process anywhere in the config, against a working metrics and OTLP path.
Any deployment fed through Alloy rather than the reference collector therefore
sent traces and metrics but no logs, so log-to-trace correlation was
unavailable there even though both ends of the link were configured.
Logs now leave through the same otlphttp exporter as the other two signals, so
all three carry one resource identity. Alloy's own filelog receiver would have
been the closest match to the reference collector, but it is still
public-preview and refuses to load unless the service is started with
--stability.level=public-preview, which would mean editing the unit on a box
reachable only by RunCommand. The Loki source components are generally
available, so they are used and bridged into OTLP by otelcol.receiver.loki;
the service needs no extra flag.
That bridge hands over an empty resource and puts everything on the log
record, so the transform sets the resource attributes in log context.
service.instance.id is concatenated in from XRPLD_HOST_LABEL because OTTL has
no env() converter, and it is a resource attribute rather than a record one
because only resource attributes are promoted to indexed Loki labels. It must
equal the node's own service_instance_id or the logs join nothing. devnet
writes one flat file rather than a per-node directory, so identity cannot be
read off the path the way the docker collector does it; XRPLD_LOG_GLOB
overrides the path for other layouts.
The line is parsed for its own timestamp, severity and trace context, and
trace_id/span_id are set on the first-class OTLP record fields so Grafana
links a log to its trace without re-parsing the body. Lines emitted outside a
sampled span keep an empty trace id rather than an invalid one.
Also carry the node-identity operators into the Grafana Cloud collector
variant, align this config's log directory with its service_instance_id, and
rename the deprecated otlphttp/filelog collector component names. Alloy's
otelcol.exporter.otlphttp is that product's own name and is unchanged.
Addresses the open review findings on this branch.
The log root was never delivered at all. Docker creates a missing bind-mount
source as root, Config::getDebugLogFile() only warns when it cannot create the
network subdirectory inside it, and Application carries on. The node therefore
looked healthy while writing no debug.log, and Loki stayed empty with no error
at any layer. docker/telemetry/data/logs has in fact been root-owned in a
working checkout since it was first created. A one-shot xrpld-logdir-init
service now creates the directory and hands it to XRPLD_UID/XRPLD_GID,
following the pattern the storage-init service already uses.
Ingested logs carried no node identity, so a multi-node stack collapsed into
one indistinguishable stream while every dashboard filters on
service_instance_id. The receiver now sets include_file_path and lifts the
per-node directory onto the resource attribute service.instance.id, which is
on the allow-list Loki promotes to an indexed stream label. A record attribute
would only become structured metadata and could not be used in a selector.
For that to join anything the directory name has to equal the emitter's
service_instance_id, so the node directories are renamed to match: node$i
becomes Node-$i, and the standalone config writes to logs/xrpld-standalone.
The integration test aborted before reporting. Under set -o pipefail the
grep | head -1 pipeline is killed by SIGPIPE once the log exceeds the pipe
buffer, so the run exited 141 somewhere past a few hundred matching lines and
read as a flaky test. grep -m1 stops on its own. The test also verified the
local file and Tempo but never that a line reached Loki, which is the one hop
this branch adds, so a bounded Loki assertion is added alongside a readiness
wait.
Documentation fixes: the Tempo cross-check counted .data, but Tempo returns
OTLP shape so the array is batches and one trace can span several; the Loki
step used the instant /query endpoint, which rejects a bare log selector with
HTTP 400 and a text/plain body, so jq could never parse it and the step never
printed a number even when ingestion worked. The filelog comment claimed six
fractional digits where the node always emits nine. The two flowcharts used
<br/>, carried no legend, and advertised GetSpan(), which Log.cpp deliberately
avoids in favour of reading the thread-local context directly.
Finally, rename the deprecated collector component names: the pinned
collector warns on every start that otlphttp and filelog are aliases for
otlp_http and file_log. Alloy's otelcol.exporter.otlphttp and
otelcol.receiver.filelog are that product's own component names and are not
deprecated, so they are left alone.
The health_check extension was present on the previous branch and dropped
here with no replacement, while this branch's own TESTING.md still polls
http://localhost:13133/ to decide the collector is ready. That check has had
no listener since, so the documented readiness step cannot pass.
Also add batch to the metrics pipeline. Without it the OTLP metric path
exports one request per instrument; the added delay is bounded by the batch
timeout, well under the Prometheus scrape interval.
Both belong here rather than downstream: this branch owns the metrics
pipeline and is the one that regressed the extension.
Two files conflicted and both were composed rather than taken from one side.
integration-test.sh: kept this branch's spanmetrics names, since the collector
sets namespace: "span" here and traces_span_metrics_* matches nothing, and took
phase-6's --max-time on every probe. The Tempo time bound needed restoring by
hand: RUN_START, the check_span guard and the start/end parameters are on
phase-6 and absent here, so a plain resolution kept phase-6's comment about
bounding the search while shipping no bound. All four pieces are back.
TESTING.md: kept server=otel with the metrics endpoint. Phase-6's template sets
server=statsd and documents prefix, which this branch's OTel path ignores.
The two writer paths run on the consensus and ledger-publish threads, and both
took a single mutex that the nine window getters also held while scanning their
deques. At a week of 4-second ledgers those deques hold about 151,000 records,
the same record stored three times, so a read walked half a million entries and
the writers waited behind it.
Each writer now owns a 128-slot ring and shares nothing with the other, so a
record is one timestamp read and one store. A full ring drops the event and
bumps droppedEvents() rather than blocking a consensus thread; at 4-second
ledgers that needs about eight and a half minutes with no drain, against a
worst normal drain gap of one export interval plus one export timeout.
The windows become a grid of one-minute buckets with a running agreed and total
per span, so a getter reads a published snapshot instead of counting. Missed is
derived from the two, which leaves a late repair touching only the agreed side.
Steady-state footprint drops from about 8 MB to about 91 KB, and window edges
quantise to a minute.
Whichever thread is inside reconcile() owns the decision state; a second caller
returns instead of waiting. Readers take a shared_ptr to the snapshot, so the
values they read cannot be overwritten underneath them.
The clock is now injected, which is what makes bucket expiry reachable in a
test. The suite covers pairing, single-sided misses, the grace boundary, late
repair across a bucket edge and past the window, each window edge exactly, two
grid lengths of steady traffic, ring overflow, and three real-thread cases.
It no longer sleeps: nine sleep_for calls totalling 81 seconds are gone.
Every public method keeps its name and signature.
MetricsRegistry::start() takes a StartOptions aggregate. Six call sites in
the #ifndef XRPL_ENABLE_TELEMETRY block still passed a std::string, so the
block did not compile. Nothing in CI compiles it: telemetry defaults ON, and
the block is skipped whenever the macro is defined.
Add one shared kTestStartOptions carrying just the endpoint -- the other
fields are never read on the no-op path -- and correct two comments that
still described the old signature and a #else stub that does not exist.
Every citation below was checked against the file it names:
- LedgerMaster.cpp:463 is fixIndex, not the ledger.store span; that guard is at
:470 and the insert it wraps at :476
- LedgerMaster.cpp:987 is the tvc assignment, which sits BEFORE the tvc < minVal
return at :988; the ledger.validate span opens at :1003
- ServerHandler.cpp:705 is inside makeJsonError; processRequest is at :718
- docker-compose.yml:71 and :75 are comments in the collector's volume block;
the loki service is at :112 and its config command at :116
Two claims were also wrong rather than merely stale. Log-trace correlation is
gated in CI, because the workflow passes no --skip-loki, and the separate check
in integration-test.sh is run by no workflow at all. The Loki label note
described the Grafana Cloud collector config rather than the local one: only the
cloud variant sets job=xrpld, and the local config's own comment says to select
on service_name. The dashboards carry 35 Loki queries, not 38.
