Every timeseries and state-timeline panel sets
options.annotations.multiLane. The state-timeline panels set
options.perPage to 5.
99 panels across 13 dashboards. Seven of those already carried the option from
this branch's node-health export, so they flip here and not upstream.
Alloy and the Grafana Cloud collector still declared a spanmetrics dimension
for an attribute tx.receive no longer sets, which only widens the label set.
The runbook's drop-path diagram still listed two tx_status values that no
longer exist, because the paths that set them now run before the span starts.
transaction-overview.json conflicted only because this branch reordered every
panel, so the diff could not line the two sides up. The incoming side changed
one panel and nothing else, so this keeps every panel and rewrite this branch
made and re-applies that single change in this branch's own style: the receive
panel drops the suppressed grouping, names its series like the other
single-series panels, and says where the duplicate count now lives.
Resolved transaction-overview.json by composing both sides: this branch's
renamed span_calls_total metric and its 15s interval, with phase-6's removal of
the suppressed grouping and its new legend.
tx.receive is created after the duplicate check, so it never carries a
suppressed attribute. The Tempo tag filter and the dashboard panel still
referenced it, and check_otel_naming rules C and D fail on a reference with no
constant behind it.
The panel keeps its remaining signal as a plain receive rate. How many relayed
copies were dropped is the transactions_duplicate traffic category, which does
not depend on a span.
The spanmetrics dimension goes too; a dimension for an attribute nothing sets
only widens the label set.
MetricsRegistry did two jobs. It owned the OTel metrics pipeline, and it
registered the observable gauges whose callbacks read live application
services. The second job is what made the whole class xrpld-tier, so the
pipeline's lifecycle -- the recording() gate and the stop() teardown that
closes a use-after-free window -- could not be unit-tested in xrpl_tests.
Split it in two:
- xrpl::telemetry::MetricsRegistry (libxrpl) owns the exporter, provider,
meter, the 16 synchronous instruments, recording(), stop(), and the
record*/increment* methods.
- xrpl::telemetry::AppMetricGauges (xrpld) owns the 19 observable gauges
and their callbacks, holding a reference to the core and to the
ServiceRegistry.
MetricMacros.h and ValidationTracker move with the core. The macros need
only recording() and meter(), both core members; the core holds a tracker
by value, and a libxrpl header cannot include one from src/.
ApplicationImp owns both objects and sequences them. The core is built in
the member-init list, so every synchronous instrument exists before any
subsystem can record one. The gauges are armed once overlay_ exists, the
last service their callbacks read. Shutdown detaches the gauge callbacks
before the core drops the provider, and each shutdown step is isolated so
a failure in one cannot skip the others.
That detach call is new. detachCallbacks() had no callers, and the flag it
sets is read by the gauge callbacks but can no longer be written by the
core, so the caller now has to make the ordering explicit.
The telemetry module links xrpl.libxrpl.core and xrpl.libxrpl.protocol
PUBLIC: ValidationTracker.h takes a LedgerIndex and MetricMacros.h takes a
ServiceRegistry, both in interfaces a consumer compiles against.
Adds a MetricsRegistry gtest that drives an enabled core with telemetry on
and pins the recording() gate, stop() leaving the registry inert, and
stop() being idempotent. The libxrpl test tree no longer depends on
xrpld.telemetry at all, and the two CMake workarounds that compiled xrpld
sources into xrpl_tests are gone.
Documentation and dashboard source links follow the code to their new
paths, split between the two classes by which one now defines each metric.
The rule watched state_accounting_full_transitions > 3 per hour, so a node
that flaps once (one full -> syncing -> full round, e.g. per online-delete
rotation) never tripped it. Lower the threshold to > 0 so a single re-entry
into FULL, past the one-hour uptime gate, alerts.
Keep the state_accounting_full_transitions metric: it is a cumulative gauge
every node always reports, so increase() yields a real series (0 when
healthy) and the rule never evaluates to NoData. A sparse counter would
raise a false DatasourceNoData on a healthy node. Set noDataState: OK so a
scrape gap cannot page either.
Merges pratik/otel-phase8-log-correlation into pratik/otel-phase9-metric-gap-fill.
Auto-merged. TESTING.md picked up phase-8's Test 4 rewrite alongside this
branch's Step 2 standalone-store fix. OTelCollector.cpp gained the weak_ptr
gauges_ list; the StatsD test gained its onCollectionReady() calls.
Merges pratik/otel-phase7-native-metrics into pratik/otel-phase8-log-correlation.
Conflict was one TESTING.md hunk under "Nodes not reaching proposing state":
this branch renamed the node directories to Node-N in integration-test.sh,
phase-7 kept nodeN and expanded the [peer_private] explanation. Resolution
keeps this branch's Node-1 path (its own script uses that naming) and
phase-7's fuller prose citing peerfinder/Config.cpp.
Non-conflicting phase-7 changes come through: OTelCollector's gauges_ list
becomes weak_ptr, matching the earlier hooks_ change; the phase-6 revert of
the StatsD-test onCollectionReady() calls resolved against phase-7's version
that keeps them.
Merges pratik/otel-phase6-statsd into pratik/otel-phase7-native-metrics.
Phase-6 dropped the three onCollectionReady() calls that had been added to
its StatsD test, because that method is only declared here on phase-7.
This branch's own copy of the file was unchanged from the merge base, so
the default merge would have silently deleted the calls from here too —
where they are needed, because this branch gates polling behind
onCollectionReady() in OTelCollectorImp::onTimer.
Resolution keeps both sides: phase-6's two new include lines
(Counter.h, Gauge.h) and phase-7's three onCollectionReady() calls plus
their doxygen and inline explanations. The merged file is exactly
phase-7's tip plus those two includes.
TESTING.md auto-merged cleanly; both sides added text under Test 1 in
different regions.
Three conflicts, all from both branches editing the same passage:
- Main.cpp: kept phase-9's wording. The metrics registry only exists here, so
"unwinding destroys little: the metrics registry, whose destructor joins its
export thread" is the true statement on this branch.
- TESTING.md: kept both paragraphs. They document different things (the
private [network_id], and the log path plus log_level).
- 05-configuration-reference.md: composed both. The identity is now resolved
before construction and never empty, so every producer stamps the node key
from the start; the only divergence left is a wallet that already holds a
different key, which corrects the tracer alone. Rewrote the earlier
"three producers" blockquote too: its "no fallback", "first boot ... left
off" and "Known issue" claims are what this change removes.
One silent break the merge could not flag: makeMetricsRegistryOptions() took
the std::optional<std::string> node key that used to be a constructor
parameter, and that parameter is now the resolved keypair. It takes the base58
string directly, and the constructor derives it from nodeIdentity_, which is
declared before both telemetry_ and metricsRegistry_.
Test 1 pointed standalone at docker/telemetry/xrpld-telemetry.cfg, whose
[node_db], [database_path] and [debug_logfile] all resolve under
docker/telemetry/data. Standalone builds its own private chain, so that
left one NuDB holding two unrelated chains. This file already states the
rule for the key generation node in Test 2, and the sibling mainnet config
keeps its store under data/mainnet/ for the same reason.
Derive a standalone config with the three paths redirected under
data/standalone/, and run from that.
Note why the flag is not the problem: --start selects StartUpType::Fresh,
but the default Normal reaches startGenesisLedger() through the same branch
chain in ApplicationImp::setup, so any standalone run writes a genesis
ledger into whichever store the config names.
The collector readiness note claimed docker-compose.yml publishes only 4317,
4318 and 8889 and that 13133 comes from a workload stack. It publishes 13133,
and that stack is not part of this branch. Probe health_check on 13133 and drop
the note; the troubleshooting entry now points at the same check instead of
carrying a second, weaker copy.
Stop restating BUILD.md. The hardcoded conan and cmake lines had drifted from
it, -Dtelemetry=ON is redundant because the Conan toolchain carries it, and the
conan-release preset resolves only from the repo root, builds into
.build/build/Release rather than .build, and sets no -Dxrpld=ON. Defer to
BUILD.md and docs/build/telemetry.md.
Test 2's keygen step reused the Devnet config with -a --start, which wrote a
genesis chain into the Devnet store, took RPC port 5005 from node 1, and was
followed by an rm -rf that also destroyed the mainnet node's store and every
log. Give it its own config under the test's temp root, as the script does.
The manual path also needs XRPLD_LOG_DIR, or the collector tails the wrong root
and Test 3 finds nothing without erroring.
Neither the template nor the script set [network_id], so a local cluster
stamped xrpl.network.type=mainnet and shared dashboard series with real mainnet
data. Set a private id in both, and say which label it produces. Also drop a
duplicate metrics_endpoint from the generated config.
Split the consensus trigger row: six families fire on a standalone
ledger_accept, and the remaining seven need the establish phase, a validator
key, or a peer. ledger.validate needs peers too, because checkAccept is
unreachable in standalone. Correct the trace-id note to 16 bytes, and name the
strategy it depends on.
The pathfinding bullet said raw account values reach Grafana Cloud. Both
accounts are already tokens when they leave the node; what differs is that the
base config hashes them a second time, so one account carries two tokens across
configs and traces must not be joined across them.
Also: the Loki allow-list is a fixed 18 keys on the pinned image with k8s and
cloud enumerated rather than wildcarded, the runbook documents 9 of 15
dashboards, and the spanmetrics block now uses one spelling with a note that
the cloud config uses the other.
The config template wrote each node's log to a lowercase node{N} directory
while setting service_instance_id=Node-{N}. The collector takes the node name
from the log file's parent directory and stamps it as the Loki
service_instance_id label, so the logs carried a name no trace or metric
shared and nothing joined. Use Node-{N} and state the rule.
The template also set log_level to warning. Nothing in the pipeline filters on
severity; the constraint is that a log line carries trace context only when it
is emitted inside an active span. At warning the only such statements in the
consensus accept span are a catch path a healthy round never takes and a
periodic censorship warning. At info the CNF Val / CNF buildLCL pair writes one
line per accepted ledger, which is what makes this test's Step 1 findable.
Grafana 13 offers the link per span, labelled "Logs for this span", in the
span's Links row — not per trace. Fix the step and the expected-results row.
The example log line quoted a message that does not exist. The real in-span
RPC statement logs at debug, so the severity code is DBG; say which line to
look for under each test, since Test 2 now logs at info.
Drop the reference to workload/validate_telemetry.py: that file is not part of
this branch, and its instant-endpoint call uses seconds, so the nanoseconds
claim applied only to query_range.
On the OTel path only [insight] server is load-bearing. CollectorManager reads
endpoint and hands it to OTelCollector, which logs it at startup and routes
nothing with it; the real export endpoint is [telemetry] metrics_endpoint,
which the template already sets. service_instance_id and service_name in that
section are read and discarded.
Leaving the line invited an operator to reconcile a mismatch that has no
effect. integration-test.sh already emits only server=otel with the same
explanation, so the two now agree.
The Payment destination was not a valid XRPL address — its base58 checksum
does not match — so Test 1 Step 4 and Test 2 Step 7 could never have returned
the tesSUCCESS they claim. Use a valid one and note that the destination does
not need to exist.
The Tempo search loop had no -G, so curl posted the parameters as a body,
Tempo answered 200 while ignoring the query, and every span name came back
non-zero. It also had no time bound, and Tempo keeps blocks for an hour, so a
re-run was answered by the previous run's traces. Add -G, RUN_START, and
start/end, matching what integration-test.sh already does.
Split the query list in two: 35 names that should be present, and 8 that need
a trigger neither test performs, where zero is the expected answer. Previously
two of the latter sat in the pass/fail list and read as failures.
Correct the standalone span table. consensus.mode_change fires once per round
start whether or not the mode changes, ledger.validate cannot fire because
checkAccept is unreachable in standalone, and the apply-stage, TxQ and ledger
families were missing rows. Give each "No" row the reason that actually
applies: the establish phase, a missing validator key, or no peers.
Also: ledger_accept is not required before submit, the teardown pgrep matched
more than this node, [peer_private] also disables the inbound listener, and
the 15-second wait covers Tempo but not Prometheus.
The `service_name`, not `job` note pointed at three file:line locations.
All three had drifted, because the cited files move on every merge forward
and nothing checks the references. Name the `resource/logs` processor and
the `loki` service instead — those survive line drift and a rename breaks
a grep loudly.
Grafana reaches the image renderer over the compose network at
http://renderer:8081, so the host publish gave nothing the stack needs.
AUTH_TOKEN is the only guard on the endpoint and its default is a fixed
string in this file.
Update the service table in the configuration reference to match.
Four doc conflicts, all where this branch had rewritten a passage that upstream
had only renamed. This branch's text is kept in seven of the eight hunks and
the spanmetrics -> span_metrics and otlp/tempo -> otlp_grpc/tempo spellings
carried into it, so the rewrite is not lost and the names stay current.
The exception is the TESTING.md span-call-count comment, where the incoming
side is the fuller text: it explains that the span_ prefix comes from the
connector's namespace setting. That side is taken.
Metric names are untouched — span_calls_total and traces_span_metrics_* are
produced by the connector's namespace, not by its component name.
Two conflicts, both where this branch's log-pipeline additions sat next to the
upstream spanmetrics -> span_metrics rename: the config header comment, which
this branch extended with a logs line, and the integration test, where the
log-correlation step precedes the span-metrics step. This branch's content is
kept in both and the rename carried into it.
Four conflicts, all where this branch's replacement of the StatsD path with
native OTLP met the upstream spanmetrics -> span_metrics rename. This branch's
design wins in every case; the rename is carried into its text rather than
reverting it, so the connector, its pipeline references, the header comment,
the TESTING.md summary and the runbook all use span_metrics while keeping the
native-OTLP wording.
One addition beyond a straight take-a-side: publish the collector's health
check port. This branch restored the health_check extension and its own
TESTING.md polls http://localhost:13133/ to decide the collector is ready, but
the port was never published on this side of docker-compose.yml, so that check
could not pass from the host. Verified the merged config loads with no
deprecation warnings and that 13133 is published exactly once.
No metric name changed: traces_span_metrics_* already read that way before the
rename, which only ever touched component names and prose.
Conflict in docker/telemetry/otel-collector-config.yaml, in the service
pipelines: this branch renamed the deprecated spanmetrics connector to
span_metrics and adds the statsd metrics pipeline, while upstream renamed the
deprecated otlp exporter to otlp_grpc. Both kept.
With both renames present the collector now starts with no deprecation
warnings at all, which was the point of the pair.
Conflict in docker/telemetry/otel-collector-config.yaml, in the traces
pipeline: this branch added the attributes/hash processor while upstream
renamed the deprecated otlp exporter to otlp_grpc. Both sides kept — the
processor list keeps attributes/hash and the exporter list takes the new name.
Checked that the exporter definition key was renamed to match the reference,
and that the collector still loads the merged config.
`docker compose down` keeps the `tempo-data` named volume, so a previous
run's traces stay in Tempo and can satisfy this run's span searches. Pass
-v in cleanup(), and tear the stack down before starting it: a run that
reaches the summary deliberately leaves the stack up, so nothing else
clears it.
The three tracker files conflicted because both sides had rewritten them. Took
the incoming lock-free class, then put this branch's two monotonic accessors
back on top of it: totalAgreementsEver() and totalMissedEver(), backed by a
gross pair incremented at first classification and left alone by the repair
branch. MetricsRegistry reads both, so dropping them would not compile.
The test file kept the incoming suite, which renames every case, and gained
this branch's two gross-counter cases adapted to the injected clock.
The pinned collector warns on every start that "spanmetrics" is a deprecated
alias for "span_metrics". Rename the connector, its pipeline references and
the prose that names it.
The derived metric names are untouched. They come from the connector's
`namespace` setting rather than its component name, so occurrences inside a
metric name such as traces_spanmetrics_calls_total are deliberately left as
they are; renaming those would break every span panel. The rename is applied
only to the bare word, never where it is joined to a metric name by
underscores.
This branch introduces the connector, so the change belongs here.
The pinned collector warns on every start that "otlp" is a deprecated alias
for "otlp_grpc". Rename the trace exporter to otlp_grpc/tempo.
Only the exporter is affected. The otlp RECEIVER keeps its name: it serves
both gRPC and HTTP under one component and is not deprecated, verified by
renaming the exporter alone and seeing the warning stop.
This belongs on this branch because it introduces the exporter, and it is the
last of four deprecated aliases in the collector config; the other three are
owned by later branches in the chain.
Alloy carried no log pipeline at all: no loki.source, loki.write or
loki.process anywhere in the config, against a working metrics and OTLP path.
Any deployment fed through Alloy rather than the reference collector therefore
sent traces and metrics but no logs, so log-to-trace correlation was
unavailable there even though both ends of the link were configured.
Logs now leave through the same otlphttp exporter as the other two signals, so
all three carry one resource identity. Alloy's own filelog receiver would have
been the closest match to the reference collector, but it is still
public-preview and refuses to load unless the service is started with
--stability.level=public-preview, which would mean editing the unit on a box
reachable only by RunCommand. The Loki source components are generally
available, so they are used and bridged into OTLP by otelcol.receiver.loki;
the service needs no extra flag.
That bridge hands over an empty resource and puts everything on the log
record, so the transform sets the resource attributes in log context.
service.instance.id is concatenated in from XRPLD_HOST_LABEL because OTTL has
no env() converter, and it is a resource attribute rather than a record one
because only resource attributes are promoted to indexed Loki labels. It must
equal the node's own service_instance_id or the logs join nothing. devnet
writes one flat file rather than a per-node directory, so identity cannot be
read off the path the way the docker collector does it; XRPLD_LOG_GLOB
overrides the path for other layouts.
The line is parsed for its own timestamp, severity and trace context, and
trace_id/span_id are set on the first-class OTLP record fields so Grafana
links a log to its trace without re-parsing the body. Lines emitted outside a
sampled span keep an empty trace id rather than an invalid one.
Also carry the node-identity operators into the Grafana Cloud collector
variant, align this config's log directory with its service_instance_id, and
rename the deprecated otlphttp/filelog collector component names. Alloy's
otelcol.exporter.otlphttp is that product's own name and is unchanged.
Addresses the open review findings on this branch.
The log root was never delivered at all. Docker creates a missing bind-mount
source as root, Config::getDebugLogFile() only warns when it cannot create the
network subdirectory inside it, and Application carries on. The node therefore
looked healthy while writing no debug.log, and Loki stayed empty with no error
at any layer. docker/telemetry/data/logs has in fact been root-owned in a
working checkout since it was first created. A one-shot xrpld-logdir-init
service now creates the directory and hands it to XRPLD_UID/XRPLD_GID,
following the pattern the storage-init service already uses.
Ingested logs carried no node identity, so a multi-node stack collapsed into
one indistinguishable stream while every dashboard filters on
service_instance_id. The receiver now sets include_file_path and lifts the
per-node directory onto the resource attribute service.instance.id, which is
on the allow-list Loki promotes to an indexed stream label. A record attribute
would only become structured metadata and could not be used in a selector.
For that to join anything the directory name has to equal the emitter's
service_instance_id, so the node directories are renamed to match: node$i
becomes Node-$i, and the standalone config writes to logs/xrpld-standalone.
The integration test aborted before reporting. Under set -o pipefail the
grep | head -1 pipeline is killed by SIGPIPE once the log exceeds the pipe
buffer, so the run exited 141 somewhere past a few hundred matching lines and
read as a flaky test. grep -m1 stops on its own. The test also verified the
local file and Tempo but never that a line reached Loki, which is the one hop
this branch adds, so a bounded Loki assertion is added alongside a readiness
wait.
Documentation fixes: the Tempo cross-check counted .data, but Tempo returns
OTLP shape so the array is batches and one trace can span several; the Loki
step used the instant /query endpoint, which rejects a bare log selector with
HTTP 400 and a text/plain body, so jq could never parse it and the step never
printed a number even when ingestion worked. The filelog comment claimed six
fractional digits where the node always emits nine. The two flowcharts used
<br/>, carried no legend, and advertised GetSpan(), which Log.cpp deliberately
avoids in favour of reading the thread-local context directly.
Finally, rename the deprecated collector component names: the pinned
collector warns on every start that otlphttp and filelog are aliases for
otlp_http and file_log. Alloy's otelcol.exporter.otlphttp and
otelcol.receiver.filelog are that product's own component names and are not
deprecated, so they are left alone.
The health_check extension was present on the previous branch and dropped
here with no replacement, while this branch's own TESTING.md still polls
http://localhost:13133/ to decide the collector is ready. That check has had
no listener since, so the documented readiness step cannot pass.
Also add batch to the metrics pipeline. Without it the OTLP metric path
exports one request per instrument; the added delay is bounded by the batch
timeout, well under the Prometheus scrape interval.
Both belong here rather than downstream: this branch owns the metrics
pipeline and is the one that regressed the extension.
Two files conflicted and both were composed rather than taken from one side.
integration-test.sh: kept this branch's spanmetrics names, since the collector
sets namespace: "span" here and traces_span_metrics_* matches nothing, and took
phase-6's --max-time on every probe. The Tempo time bound needed restoring by
hand: RUN_START, the check_span guard and the start/end parameters are on
phase-6 and absent here, so a plain resolution kept phase-6's comment about
bounding the search while shipping no bound. All four pieces are back.
TESTING.md: kept server=otel with the metrics endpoint. Phase-6's template sets
server=statsd and documents prefix, which this branch's OTel path ignores.
Every citation below was checked against the file it names:
- LedgerMaster.cpp:463 is fixIndex, not the ledger.store span; that guard is at
:470 and the insert it wraps at :476
- LedgerMaster.cpp:987 is the tvc assignment, which sits BEFORE the tvc < minVal
return at :988; the ledger.validate span opens at :1003
- ServerHandler.cpp:705 is inside makeJsonError; processRequest is at :718
- docker-compose.yml:71 and :75 are comments in the collector's volume block;
the loki service is at :112 and its config command at :116
Two claims were also wrong rather than merely stale. Log-trace correlation is
gated in CI, because the workflow passes no --skip-loki, and the separate check
in integration-test.sh is run by no workflow at all. The Loki label note
described the Grafana Cloud collector config rather than the local one: only the
cloud variant sets job=xrpld, and the local config's own comment says to select
on service_name. The dashboards carry 35 Loki queries, not 38.
This branch switches integration-test.sh to [insight] server=otel and adds an
assertion that no StatsD listener is needed, but TESTING.md still described the
metrics it verifies as StatsD-derived and its manual node-config template had no
[insight] stanza at all.
CollectorManagerImp falls through to NullCollector when server is neither statsd
nor otel, so a reader building configs from that template got zero
beast::insight metrics. The template also omitted service_instance_id and
metrics_endpoint, which the script writes; without the former every node is
indistinguishable in the $node dashboard filter.
curl applies no overall timeout of its own, so a server that accepts the
connection and then stops answering parks a poll loop for the rest of the run
and the loop's attempt count stops bounding anything. Add a CURL_MAX_TIME
ceiling and apply it to all 18 executable probes in integration-test.sh.
TESTING.md's manual node-config template also disagreed with what the script
writes, so a reader following it could not reproduce the automated path:
- no [insight] stanza, so no beast::insight metric leaves the node at all and
Step 10b's ten rippled_* assertions cannot pass
- [ips_fixed] listed all six peer ports including the node's own
The log level is deliberately untouched: the template and the script agree on
warning here.
check_statsd_metric queried rippled_rpc_requests, which no pipeline
produces: the collector's statsd receiver runs with is_monotonic_counter,
so the Prometheus exporter appends _total. A wrong name returns zero
series rather than an error, so the assertion could not be told apart
from a broken pipeline. All eight assertions were re-derived from how
each metric is created in code; this was the only counter.
Tempo searches carried no start/end, and tempo-data is a named volume
that `docker compose down` preserves under a one-hour block retention, so
the 17 span assertions could pass on an earlier local run's traces. Bound
every search to this run, and tear the stack down with -v before starting
so no earlier data is present to match. The service-name check now
matches a whole line, because the tag-values endpoint ignores start/end.
Add a gtest for the StatsD gauge that publishes its initial zero and for
the counter that must publish nothing. Assert two metrics the harness
never checked: a traffic-category gauge no message reaches, and
io_context latency.
Update the consensus and ledger key inventories and the spanmetrics
dimension comment in both collector configs, so they name the emitted
keys. The comment's last_close_time reference is a server_info gauge, not
the span attribute, and is left as it is.
Addresses review findings on the native-metrics work.
StatsDCollector::onTimer drained the send buffer inside the polling_ gate. That
gate holds back hook handlers until the application's services are built, but
sendBuffers() is socket I/O. StatsDEventImpl derives only from EventImpl, so it
never enters metrics_ and posts straight to the buffer; its |ms timings piled up
before onCollectionReady and were dropped after onCollectionStopping. The drain
now runs every tick, and outside metricsLock_, so onCollectionStopping no longer
waits on a UDP flush.
TelemetryImpl's constructor left meterProvider_ set when initMetrics() threw.
initMetrics publishes globally as its last step, so a throw left getMeter()
callers holding a provider nothing else could reach. Reset it in the catch.
~ApplicationImp caught only std::exception around telemetry shutdown while the
callees reach third-party SDK code, so a foreign exception would have terminated
the process. Added a logging catch-all.
ValidationTracker's hard trim evicted by unordered_map bucket order. It now
evicts oldest-first, so the entry dropped under pressure is the one least likely
to still reconcile.
The GetMeter test restored the global meter provider only on the success path,
and ASSERT_TRUE early-returns past it. Uses xrpl::ScopeExit instead.
The hook debounce window is a named constant rather than a bare 500 in a
comparison, and the metric export cadence becomes operator-configurable through
metric_export_interval_ms and metric_export_timeout_ms. Both are range-checked:
the SDK warns and silently substitutes its own 60s/30s defaults when the timeout
is not below the interval, so an unchecked value would slow export rather than
speed it up. Parsing uses a signed representation because lexical_cast<uint32_t>
accepts a leading minus and wraps it.
Naming corrections: CollectorManager documented exported_instance, which no OTel
dashboard uses; node-health queried job_count where the exported name is
jobq_job_count; network-traffic and overlay-traffic-detail referenced an
undeclared DS_PROMETHEUS variable; the counter table omitted the _total suffix
the Prometheus exporter appends; the plan docs and task list carried an xrpld_
prefix formatName never applies; and OTelCollector::New()'s contract promised its
instanceId, serviceName and networkType arguments were read, contradicting the
definition that marks them unused.
Telemetry.cpp conflicted. Phase-9 rewrote the metrics pipeline into
makeTracerResource()/makeMetricsResource()/initMetrics() further down the
class, so its side of the region is empty and phase-8's private helper
block does not apply. Resolved to phase-9's structure; phase-8's own
hunks outside the region (the deleted kTracesPath/kMetricsPath, the
verbatim traces URL, the two-endpoint startup log) merged in.
Phase-9's initMetrics() still derives the metrics URL by suffix-swap.
That is fixed in the next commit, not here.
One [telemetry] key served both OTLP signals, and the metrics URL was
derived from it by suffix-swap: strip a trailing slash, strip a known
signal path if present, append the wanted one. Anything not ending
/v1/traces therefore posted metrics to the traces path, and the OTLP
version was pinned in code where an operator could not reach it.
Adds metrics_endpoint alongside traces_endpoint. Both are full URLs used
verbatim, so traces and metrics can go to different collectors, or to one
whose OTLP paths are not the defaults. signalEndpoint(), kTracesPath and
kMetricsPath are gone; nothing derives an endpoint from another.
The startup log names both URLs, since with two independent endpoints
there was otherwise no way to see where metrics were going.
Also drops exporter=otlp_http from the shipped config and the test
fixture. No branch in the chain reads an `exporter` key: it was a real
Setup member in the first phase-1b implementation, removed when only
OTLP/HTTP was wired up, and already deleted from TESTING.md once on the
same grounds.
Eighteen conflict regions across nine files. Resolved by asking, per
region, which side is the better final state rather than by taking a
branch wholesale.
Telemetry.cpp keeps phase-9's two resource builders. phase-8 offered a
single makeResource() with no node identity; phase-9 splits it into
makeTracerResource() and makeMetricsResource() because the metrics
provider is built in the constructor, before setNodeId() runs, so
xrpl.node.id can only be stamped unconditionally on the tracer side.
Collapsing them would have dropped that attribute, which is what keeps
per-node traces from folding into one identity.
Telemetry.h and the config test compose both sides: phase-9's nodeId
member and its assertion, plus the renamed endpoint.
xrpld-telemetry.cfg keeps phase-9's devnet identity and its
metrics_endpoint, renames the traces key, and drops exporter=otlp_http.
Nothing reads an `exporter` key on any branch in the chain: it was a real
Setup member in the first phase-1b implementation, removed when only
OTLP/HTTP was wired up, and already deleted from TESTING.md once on the
same grounds. The cfg line was the last carrier.
The docs keep phase-9's versions, which are both fuller and more
accurate: the incoming runbook listed the consensus strategy values as
"random" where the code compares against "attribute".
OTelCollector.cpp had five comment-only regions in a file phase-7 owns,
so those take the upstream side.
MetricsRegistry.h's usage example named a member that no longer exists
and the wrong arity; it now matches the real three-argument call and says
where the endpoint comes from.
Ten conflict regions in four files, none of them caused by the rename.
phase-8's own commit had rewritten comments in files phase-7 owns
(Unit.h, HistogramBuckets.h, OTelCollector.cpp) and in Telemetry.cpp,
while the same sweep ran independently on phase-7.
Resolved every region to phase-7's side on ownership grounds: those files
belong to phase-7 or earlier, so a downstream branch should not carry
divergent copies. phase-8 changed comments only in all four, verified
against the merge base, so no code was dropped.
The one structural region: phase-7 had refactored addUnitView from an
inline lambda into a member function, and phase-8 still held the lambda.
Keeping phase-8's would have shadowed the member.
Wording phase-8 had that is worth restoring on phase-7 -- the "legacy"
qualifier on the prefix parameter, and the rejected-alternative note on
the bucket ladders -- is recorded outside the tree for a follow-up.
Three conflicts, all composed rather than resolved by taking a side:
- TelemetryConfig.cpp: phase-6 kept networkTypeFromId file-local with
[[nodiscard]]; phase-7 had relocated it to public scope for
Application.cpp. Kept phase-7's relocation, so one definition remains.
The [[nodiscard]] survives on the declaration in Telemetry.h.
- Telemetry.cpp x2: phase-7 added getMeter overrides, phase-6 added
[[nodiscard]] to the startSpan below them. Kept both, and put
[[nodiscard]] on getMeter too.
- TESTING.md: phase-7 had the right metric name (span_calls_total, which
the spanmetrics namespace produces) but the wrong label. Its
xrpl.rpc.command appears nowhere else in the branch; the attribute is
bare `command`, which is what the dashboards query. Took phase-7's
metric with the correct label.
Both signalEndpoint call sites follow the renamed member. signalEndpoint
itself is left in place: removing it and adding metrics_endpoint is a
design change, not part of propagating a rename.
The rename arrived from phase-1b by merge. Four files still wrote the old
key, which the parser no longer reads, so each would have silently
fallen back to the default collector URL.
integration-test.sh is the load-bearing one: it generates the node config
the test harness starts, so the stale key would have pointed the node at
localhost regardless of the compose network. xrpld-telemetry.cfg is the
standalone node config; the other two document the key.
Note this cfg has a second, divergent variant on the devnet branches that
needs the same fix there.
