Commit Graph

339 Commits

Author SHA1 Message Date
Pratik Mankawde
95c3abb0c1 Merge branch 'pratik/otel-sync-diagnostics' into pratik/otel-sync-diagnostics-freshen-fix 2026-09-18 10:03:32 +01:00
Pratik Mankawde
f1df5de8dd feat(telemetry): count relayed transactions this node did not process
Moving the tx.receive span past the duplicate check removed the only record of
why a relayed copy was dropped. A labelled counter restores it, and covers the
inner-batch reject the span never usefully reached.

The count itself was never span-derived: transactions_duplicate already totals
the dropped copies and predates the telemetry work. What was missing is the
split between ordinary relay overlap and a peer sending traffic it should not.

A counter also survives sampling and runs with tracing off, which the span
attribute did not.
2026-09-18 10:00:31 +01:00
Pratik Mankawde
195e4e5d53 Merge branch 'pratik/otel-sync-diagnostics' into pratik/otel-sync-diagnostics-freshen-fix 2026-09-16 18:07:29 +01:00
Pratik Mankawde
f7037c8dc3 Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics
Brings the MetricsRegistry split onto this branch. The pipeline half is now
xrpl::telemetry::MetricsRegistry in libxrpl; the observable gauges are
xrpl::telemetry::AppMetricGauges in xrpld.

This branch had added its own instrumentation to the pre-split class, so the
merge had to route each addition to the correct half:

- The thirteen gauges added here -- amendment block, cache hit-rate detail,
  clock skew, job-queue saturation, ledger quorum publish, peer ledger supply,
  rotation state, slot census, stall events, sync acquire, sync state, UNL
  quorum, and the cache lock-hold observer -- all land on AppMetricGauges,
  reading the core's meter and validation tracker through it.
- The pipeline additions stay in libxrpl: the consensus round-duration and
  rotation-phase histogram views, the malloc-trim and dns/dial latency bucket
  ladders, the job-stall counter, and the switch from literal metric names to
  the MetricNames.h constants.

Git detected the pre-split MetricsRegistry.cpp and .h as renames of the gauge
files, so both sides' pipeline changes initially landed in the gauge half. They
were moved back, and the result was audited by inventory: every method
definition, instrument creation, view registration, and emitted string from
either side is present, with identical multiplicity.

MetricNames.h moves to include/xrpl/telemetry/ alongside the core. It has no
includes of its own and its two sibling name headers already live there, so
keeping it under src/ would leave an xrpld path in libxrpl's dependency
surface. Nineteen files follow it.

incrementStateChanges() stays removed. The labelled state_changes_total{from,to}
counter this branch introduced replaces it, and the test asserting the method is
absent is kept -- an unlabelled instrument alongside the labelled one would give
Prometheus two conflicting versions of one metric name.

Two tests that drove startAsyncGauges() against a mock ServiceRegistry are
dropped: xrpl_tests links only xrpl.libxrpl and cannot reach the gauge class.

Levelization regenerated. Both xrpld.telemetry loops become bidirectional
rather than one-way; neither is new.
2026-09-16 18:05:48 +01:00
Pratik Mankawde
ec0bfe521d refactor(telemetry): move the metrics pipeline core into libxrpl
MetricsRegistry did two jobs. It owned the OTel metrics pipeline, and it
registered the observable gauges whose callbacks read live application
services. The second job is what made the whole class xrpld-tier, so the
pipeline's lifecycle -- the recording() gate and the stop() teardown that
closes a use-after-free window -- could not be unit-tested in xrpl_tests.

Split it in two:

- xrpl::telemetry::MetricsRegistry (libxrpl) owns the exporter, provider,
  meter, the 16 synchronous instruments, recording(), stop(), and the
  record*/increment* methods.
- xrpl::telemetry::AppMetricGauges (xrpld) owns the 19 observable gauges
  and their callbacks, holding a reference to the core and to the
  ServiceRegistry.

MetricMacros.h and ValidationTracker move with the core. The macros need
only recording() and meter(), both core members; the core holds a tracker
by value, and a libxrpl header cannot include one from src/.

ApplicationImp owns both objects and sequences them. The core is built in
the member-init list, so every synchronous instrument exists before any
subsystem can record one. The gauges are armed once overlay_ exists, the
last service their callbacks read. Shutdown detaches the gauge callbacks
before the core drops the provider, and each shutdown step is isolated so
a failure in one cannot skip the others.

That detach call is new. detachCallbacks() had no callers, and the flag it
sets is read by the gauge callbacks but can no longer be written by the
core, so the caller now has to make the ordering explicit.

The telemetry module links xrpl.libxrpl.core and xrpl.libxrpl.protocol
PUBLIC: ValidationTracker.h takes a LedgerIndex and MetricMacros.h takes a
ServiceRegistry, both in interfaces a consumer compiles against.

Adds a MetricsRegistry gtest that drives an enabled core with telemetry on
and pins the recording() gate, stop() leaving the registry inert, and
stop() being idempotent. The libxrpl test tree no longer depends on
xrpld.telemetry at all, and the two CMake workarounds that compiled xrpld
sources into xrpl_tests are gone.

Documentation and dashboard source links follow the code to their new
paths, split between the two classes by which one now defines each metric.
2026-09-16 13:45:52 +01:00
Pratik Mankawde
44d1994d6a refactor(nodestore): scrub site details, own the phase label strings, fix the test overload
Review follow-up on the freshen lock-hold fix:

- Drop host names, dates and one-site figures from the new comments,
  harness notes and docs; explain the mechanism in general terms.
- RotationPhase stores its stage and cache labels as owned std::string,
  not std::string_view: the ctor still takes views so the label
  constants pass without a copy, but a member view would dangle if a
  caller ever passed a temporary. freshenCache/recordFreshen take the
  cache name by std::string_view (read-only, call-scoped).
- The new DatabaseRotating test called fetchNodeObject through the
  derived type, whose private override hides the public base method;
  call it through Database& instead. This was the dev-box build break.
- freshenCache reports the exact fetched count when a health abort cuts
  it short, and stops labelling the per-partition hold 'getKeys'.
- Remove a [[maybe_unused]] that silenced no warning (the build sets
  -Wno-unused-parameter and disables misc-unused-parameters).
2026-09-15 14:26:46 +01:00
Pratik Mankawde
b1345fff8d docs(telemetry): describe the rotation stall without internal host names
The reference doc, span-harness notes and histogram-bucket comments
named the internal AWS dev box and dates while explaining why the
rotation phases are timed. Reword to the general mechanism (a
multi-second freeze at the copy-walk to freshen boundary on a populated
node); the specific hosts, dates and trace ids stay in the task notes.
2026-09-15 14:26:25 +01:00
Pratik Mankawde
614c1a39ad fix(nodestore): bound the rotation freshen's cache lock hold and measure its yield
The online-delete rotation's cache freshen called TaggedCache::getKeys(),
which held the cache mutex while copying every key. On the dev box's 26
million entry tree-node cache that hold lasted 5-6 s, froze every job
that touches the cache, and dropped the RocksDB node out of sync once per
rotation: each "getKeys held the lock" warning was followed within 1-5 s
by "View of consensus changed" (5 of 5 rotations on 2026-09-15).

Copy the keys one map partition at a time instead. TaggedCache gains
forEachKeyPartition(), which holds the mutex only while one partition's
keys are copied and runs the callback with the mutex released, so the
longest hold shrinks by the partition count (8 on the dev box). The
freshen.keys rotation phase no longer exists as one step, so its span,
stage value, harness entries and docs are removed; the per-partition hold
still shows on the cache lock-hold peak gauge.

Measure what the freshen achieves, which no existing signal did.
DatabaseRotating gains duplicateCopyForwardTotal(), counting archive
copies made on duplicate fetches (the rotation's own copy walk and
freshen); copyForwardTotal() deliberately excludes those. The freshen
phase records rotation_freshen_keys_total{cache,outcome} and stamps
key_count, cache and keys_copied on its span; the copy phase stamps
nodes_copied. A warn log line per freshen reports the same numbers, and
the ledger-sync-health dashboard gets a Rotation Freshen Yield panel.

Log the "STATE->" operating-mode change at warn instead of info. It is
the only record of a mode change with an exact timestamp; the
state_changes_total counter is scraped once a minute and cannot order a
flap against a multi-second event.

Tests: five GTests for forEachKeyPartition (every key once, empty cache,
mutex free during the callback, concurrent insert, lock-hold peak), three
for duplicateCopyForwardTotal over two memory backends, one for the new
counter's series, and the new name literals.
2026-09-15 13:49:26 +01:00
Pratik Mankawde
545ec26eed merge: bring the noopMeter() clang-tidy fixes forward from phase10-workload-validation 2026-09-14 23:48:42 +01:00
Pratik Mankawde
5df8f71c7a merge: bring the noopMeter() clang-tidy fixes forward from phase8-log-correlation 2026-09-14 23:48:23 +01:00
Pratik Mankawde
7a4806c00a fix(telemetry): satisfy clang-tidy on the noopMeter() helper
Static constants take the k prefix (readability-identifier-naming), and
SpanGuardScope.cpp no longer names anything from <opentelemetry/metrics/noop.h>
since it calls noopMeter(). Both fail CI under warnings-as-errors.

The helper's docstring also claimed NoopMeterProvider hides the base
two-argument GetMeter; it declares that overload itself, so the only
detail worth sharing is the version.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-14 23:47:55 +01:00
Pratik Mankawde
1880c9a498 merge: bring phase10-workload-validation forward into sync-diagnostics
Resolutions:
- MetricsRegistry.cpp: keep both <exception> and <limits>; drop
  incrementStateChanges(), which this branch removed on purpose (the
  labelled state_changes_total call site in NetworkOPsImp::setMode
  replaces it, and a compile-time test guards that).
- tests/MetricsRegistry.cpp: constructor-built pipeline wording from
  phase-10, this branch's test list and gauge paragraphs kept; the two
  lifecycle tests now call startAsyncGauges() and pass kTestOptions.
- tests/MetricMacros.cpp: comments name the recording() gate.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-14 23:45:32 +01:00
Pratik Mankawde
b10fe32657 merge: bring the node-identity change forward from phase-8
Three conflicts, all from both branches editing the same passage:

- Main.cpp: kept phase-9's wording. The metrics registry only exists here, so
  "unwinding destroys little: the metrics registry, whose destructor joins its
  export thread" is the true statement on this branch.
- TESTING.md: kept both paragraphs. They document different things (the
  private [network_id], and the log path plus log_level).
- 05-configuration-reference.md: composed both. The identity is now resolved
  before construction and never empty, so every producer stamps the node key
  from the start; the only divergence left is a wallet that already holds a
  different key, which corrects the tracer alone. Rewrote the earlier
  "three producers" blockquote too: its "no fallback", "first boot ... left
  off" and "Known issue" claims are what this change removes.

One silent break the merge could not flag: makeMetricsRegistryOptions() took
the std::optional<std::string> node key that used to be a constructor
parameter, and that parameter is now the resolved keypair. It takes the base58
string directly, and the constructor derives it from nodeIdentity_, which is
declared before both telemetry_ and metricsRegistry_.
2026-09-14 20:39:18 +01:00
Pratik Mankawde
bec9e1c8a9 fix(telemetry): resolve the node identity before the Application is built
resolveNodePublicKey() returned std::nullopt in three real cases: a first boot
with no wallet database, a standalone run (its wallet is a private temporary
database), and --newnodeid. Telemetry's resources are built during
ApplicationImp's member-init list and are immutable, so on those runs the node
reported an empty service.instance.id and no xrpl.node.id for the whole run,
while setup() minted a key moments later and patched only the tracer.

Replace it with resolveNodeIdentity(), which always returns a keypair: derived
from a configured seed, else read from an existing wallet database, else
minted. Main.cpp passes that pair to makeApplication(), ApplicationImp stores
it in nodeIdentity_ -- now declared before telemetry_ and no longer an optional,
because it is always set -- and builds the telemetry resource from it.

setup() calls getNodeIdentity(), which now persists rather than mints: it
stores the resolved pair when the wallet holds no identity, adopts the stored
one when it does, and clears first for --newnodeid. The write stays in setup()
because that is where the database exists; a standalone run has no persistent
wallet to write to, which is why the pair has to be decided before
construction rather than read back afterwards. Wallet gains storeNodeIdentity()
for that write, and getNodeIdentity(session) now uses it instead of repeating
the insert.

The three-argument makeApplication() mints a keypair, so jtx::Env and any other
test Application behave as a standalone run always did.

Also fold the three hand-rolled "meter from a NoopMeterProvider" copies into
telemetry::noopMeter(): the base-pointer call and the kMeterVersion argument are
both easy to get wrong alone, and the meter identity has to match the one the
histogram views select on.

The new gtest covers the wallet half: store-then-read, store not replacing an
existing identity, clear-then-store, and that the mint path persists. It adds
the tests.libxrpl > xrpl.rdb levelization edge, regenerated here.
2026-09-14 20:34:31 +01:00
Pratik Mankawde
b52b3c7b8e feat(telemetry): name the rotation-stall metrics and give phase durations a seconds ladder
Adds the metric and label constants the rotation-stall proof chain uses:
- `rotation_phase_duration_seconds` — per-phase wall-clock, labelled by `stage`,
  with 8 values in `lval::rotation_phase` matching the child span suffixes.
- `jobq_stall_total` — a counter for jobs whose run time reached LoadMonitor's
  1 s warn threshold.
- `lval::cache_metrics::treenode_lock_hold_peak_us` / `fullbelow_lock_hold_peak_us`
  as `metric` values on the existing `cache_metrics` gauge, for the TaggedCache
  lock-hold peaks the next commits expose.

`kRotationPhaseSecondsBuckets` in HistogramBuckets.h covers 1 s to 1 h, wired
through `addRotationPhaseHistogramView` at instrument-registration time so
`histogram_quantile` never has to interpolate inside the +Inf bucket. Ladder
test and a labelled histogram macro test pin the values.
2026-09-14 20:33:42 +01:00
Pratik Mankawde
98cdf18d60 feat(telemetry): name the online-delete rotation spans and their outcome rule
Groundwork for WP-B6 (rotation stall tracing). Introduces the constant set the
rest of the work references without emitting anything yet: `seg::nodestore`,
`nodestore_span::rotateFull` and its eight phase suffixes, the attribute keys
and outcome values, and `rotationOutcome()` as a constexpr total function over
the `RotationExit` enum so no exit of `SHAMapStoreImp::run`'s rotation block
can leave the root span without an outcome.

Tests pin the literal names the collector keep-policy, expected_spans.json and
the dashboards match on, and static_assert `rotationOutcome()` over its whole
input domain.
2026-09-14 20:30:19 +01:00
Pratik Mankawde
0fab743c6d merge: bring the review fixes forward from otel-phase10-workload-validation 2026-09-08 15:55:05 +01:00
Pratik Mankawde
2fdfbe1b91 merge: bring the review fixes forward from otel-phase8-log-correlation 2026-09-08 15:51:07 +01:00
Pratik Mankawde
7d679ce596 merge: bring the review fixes forward from otel-phase6-statsd 2026-09-08 15:45:22 +01:00
Pratik Mankawde
868edce7e5 merge: bring the review fixes forward from phase4-consensus-tracing
Two conflicts, both additive.

TelemetryConfig.cpp: this branch added requireHttpsEndpoint next to
requireReadableFile; upstream added readConsensusTraceStrategy at the same spot.
Both kept.

05-configuration-reference.md: this branch added the two client-certificate rows
while upstream corrected the consensus strategy value from attribute to random.
Both kept. Also drops the stale "not yet implemented" row for
consensus_trace_strategy, which the merged table now contradicts twice over: the
option is parsed, and its value is no longer spelled attribute.
2026-09-08 15:31:40 +01:00
Pratik Mankawde
039c2768ba fix(telemetry): require an https endpoint when a client certificate is set
The OTLP/HTTP exporter selects TLS from the endpoint URL scheme alone
(HttpSslOptions in the pinned SDK matches "https:" exactly), so a client
certificate handed to it alongside an http:// traces_endpoint is loaded and
never presented. The parser checked the cert/key pairing, use_tls and file
readability, but never the scheme, and the default traces_endpoint is plain
HTTP. makeTelemetrySetup() now requires traces_endpoint to start with
"https://" whenever tls_client_cert is set, including when the key is left
at its default.

Nothing asserted the client options reaching the exporter, so a swapped
certificate and key would have passed every test. Move the options mapping
into makeTraceExporterOptions() and assert it at that boundary with
distinct certificate and key paths, plus a one-way-TLS control and a
use_tls=0 control. One case runs the whole path from a [telemetry] section.

Runbook and example-config fixes:

- tx.included is emitted per transaction of the agreed consensus set,
  before buildLCL() applies anything, so it is a superset of the accepted
  ledger rather than proof of inclusion.
- the dispute.resolve query used the descendant operator, but the event is
  on the consensus.update_positions span itself, so it matched nothing.
- the exhausted-retries query asked for txq_status="retried" with
  retries_remaining=0, which cannot occur: the attribute is stamped before
  the attempt and the retried branch only runs while retries are left.
  Exhaustion is txq_status="failed" with a zero count.
- consensus_round_id is an int64, so the two queries comparing it to a
  quoted string matched nothing.
- note that consensus_trace_strategy=random is experimental and not used.
- note that a trailing "| attr = value" is rejected by current Tempo;
  attribute filters belong inside the braces.
2026-09-08 14:41:39 +01:00
Pratik Mankawde
fb827dc0f1 fix(telemetry): make the consensus trace strategy an enum
consensus_trace_strategy was read as a std::string and compared against the
literal "attribute" in startRoundTracing(), while the runbook documented
"deterministic" and "random". The documented value "random" therefore fell
through to the default and did nothing.

Parse the setting once into ConsensusTraceStrategy, so the consensus code
branches on a type. The accepted spellings are now "deterministic" and
"random"; anything else fails at startup instead of silently defaulting.
The behaviour behind the old "attribute" name is unchanged and is now
reached by "random".

Document consensus_trace_strategy in xrpld-example.cfg, stating that
"random" is experimental and not used: it gives each node its own trace id,
so one round arrives as one trace per node.

Also state on the tx.included event that it covers the agreed consensus set
before the ledger is built, so it is a superset of the accepted ledger.
2026-09-08 14:39:05 +01:00
Pratik Mankawde
3c56697930 merge: bring the review fixes forward from otel-phase10-workload-validation 2026-09-07 15:47:20 +01:00
Pratik Mankawde
efdd9758f2 merge: bring the review fixes forward from otel-phase8-log-correlation 2026-09-07 15:39:59 +01:00
Pratik Mankawde
4ad84cae14 merge: bring the review fixes forward from otel-phase6-statsd 2026-09-07 15:23:22 +01:00
Pratik Mankawde
de895c6d1e merge: bring the review fixes forward from phase4-consensus-tracing
Two conflicts, both additive on each side.

TelemetryConfig.cpp: include blocks only. This branch added FileUtilities.h for
the certificate readability checks; upstream added <limits> and <optional> for
the bounds parser. Both kept.

The TelemetryConfig test: this branch's mutual-TLS cases and upstream's
batch-bounds cases were added at the same positions, so the file is rebuilt from
both stages and carries all 32 tests. Two shared cases were each edited by one
side only, so the edited side wins in each: upstream asserts the batch defaults
in parse_empty_section, and this branch's parse_full_section writes a real
certificate file, which is now required since the parser opens it.
2026-09-07 15:14:52 +01:00
Pratik Mankawde
3d9ea4b9da merge: bring the review fixes forward from phase3-tx-tracing
One conflict, in 06-implementation-phases.md: this branch had rewritten the
phase-4 task table with a Status column, a descoping note and a Spans Produced
section, while upstream corrected the class name in the old plain table. This
branch's section is kept and the name correction re-applied to its 4.1 row.
2026-09-07 15:04:39 +01:00
Pratik Mankawde
08df97f431 merge: bring the review fixes forward from phase2-rpc-tracing
One conflict, in SpanGuard.h: this branch added struct TraceBytes and upstream
added enum SpanRole at the same position after TraceCategory. Unrelated
declarations, so both are kept.
2026-09-07 15:03:44 +01:00
Pratik Mankawde
caa704de10 merge: bring the review fixes forward from phase1c-rpc-integration
Two conflicts, both resolved by composing the sides rather than taking one.

cfg/xrpld-example.cfg: this branch had moved the batch-processor keys under
their own heading while upstream edited them in place, so a merge-both would
have documented them twice. Upstream's range sentences are applied to the
relocated block and the head-sampling note keeps its position.

02-design-decisions.md: the summary table changed on both sides for different
reasons. Upstream renamed ledger_index to current_ledger_seq and ledger_seq;
this branch had corrected the PathFinding row to the keys it actually emits.
Both are kept.
2026-09-07 15:00:53 +01:00
Pratik Mankawde
a8d678f354 merge: bring the childSpan pseudocode fix forward from phase1b-telemetry-infra 2026-09-07 14:56:32 +01:00
Pratik Mankawde
0f49aecbf0 docs(telemetry): pass a full dotted constant in the childSpan pseudocode
childSpan() takes the span name verbatim, so a bare op:: suffix names the span
"process" rather than "rpc.process". The same defect was corrected in the
SpanGuard and Telemetry examples; this is the last copy.
2026-09-07 14:56:29 +01:00
Pratik Mankawde
a6c24848ba merge: bring the review fixes forward from phase1b-telemetry-infra 2026-09-07 14:55:22 +01:00
Pratik Mankawde
623e3d8ad3 feat(telemetry): add batch-size and discovered-path histograms
Review feedback asked for a Histogram rather than a span attribute at these two
places. Both, not either: the attribute answers how big one sampled request was,
which an aggregate cannot, and the histogram answers the distribution across all
requests, which an unsampled trace never reveals. Both attributes stay.

The metrics land here rather than with the attributes because neither
HistogramBuckets.h nor the metric macro exists on the branch that added them.

Both use kObjectCountBuckets. The argument is the floor, not the ceiling: the SDK
default edges start 0,5,10,25, so an ordinary batch of one to five falls in a
single bucket and every quantile becomes an interpolation on one edge. The object
ladder puts five edges over the mass of both distributions. Path count is bounded
at 352 by kMaxPaths times kMaxAutoSrcCur and cannot saturate. Batch size can, at
roughly 333k, but no measured traffic goes near it, so the ladder is not widened
for a range nothing occupies; the runbook records the overflow query and a test
asserts it stays readable.
2026-09-07 13:51:44 +01:00
Pratik Mankawde
18abd100b5 feat(telemetry): let a call site choose a span's role, not just its category
Span kind was derived from TraceCategory alone, so every Rpc-category span was
kServer. A category cannot tell an inbound handler from the internal work under
it, and trace backends pair kServer with kClient, so internal spans left as
kServer become unpaired edges in a service graph and read as extra inbound
requests.

SpanRole is a new xrpl-owned enum, orthogonal to TraceCategory: the category
names the subsystem and gates the span on config, the role says whether the span
handles a remote call. It is a defaulted fourth parameter on span(), freshRoot()
and the ScopedSpanGuard equivalents, defaulting to SpanRole::FromCategory, so no
existing call site changes. resolveSpanKind() applies an explicit role and falls
back to the category map, which keeps its single responsibility. The
telemetry-disabled stubs mirror all four signatures.

No call site passes a role yet. The two that need it are on a later branch.

Also fixes a ScopedSpanGuard example that passed a bare op:: suffix to
childSpan(), which takes the name verbatim. Naming the child rpc.command made it
a child of rpc.command.<cmd>, inverting the hierarchy, so the example's parent is
now rpc.process and the command attribute moved onto the command span.
2026-09-07 13:39:57 +01:00
Pratik Mankawde
f8e0a19b9f fix(telemetry): correct the childSpan doc examples and make Rule D tests real
Review feedback on the RPC integration PR.

The childSpan examples could not work as written. childSpan() takes its parent
from the ambient context and uses impl_ only as a liveness gate, so an unscoped
SpanGuard parent produced two siblings rather than a parent and child. The parent
is now a ScopedSpanGuard, the child no longer reuses the parent's name, and the
examples pass a full dotted constant because childSpan() takes the name verbatim.

Five of the ten Rule D tests could not fail. Four passed an empty L1 key set,
which makes the rule skip validation altogether; the fifth asserted an empty
result against an escaped-quote selector that extracted no labels at all. Each
now passes a nonempty L1 set and carries a known-bad label in the same
expression, so it asserts both that the intended labels are accepted and that
Rule D ran. Verified by disabling the rule: the old tests stay green, the new
ones all fail.

Span kind is not fixed here. categoryToSpanKind and the span factories belong to
the telemetry library, so the role parameter is routed to that branch, and the
two call sites here follow once it exists.
2026-09-07 13:25:22 +01:00
Pratik Mankawde
9c4937926f merge: bring telemetry config and doc fixes forward from phase10-workload-validation 2026-09-03 20:37:40 +01:00
Pratik Mankawde
637f9ae3d0 merge: bring telemetry config and doc fixes forward from phase8-log-correlation
Nine files conflicted. Resolutions, and why:

Telemetry.cpp - upstream carried its own initMetrics(), makeResource() and
makeMetricExporter(); this branch already has an initMetrics() that builds the
exporter inline and uses makeMetricsResource(), which stamps xrpl.node.id only
when it is already known. Keeping both would have defined initMetrics() twice.
Kept this branch's, then pointed its reader at setup_.metricExportInterval and
setup_.metricExportTimeout: upstream turned those constants into [telemetry]
keys and removed the old ones, so the previous spelling no longer resolves.

OTelCollector.h - kept upstream's parameter docs. This branch's text promised
instanceId, serviceName and networkType become resource attributes; the
constructor marks all three [[maybe_unused]] and the .cpp already says they are
not read.

CollectorManager.cpp - kept upstream's comment for the same reason.

node-health.json - kept this branch's 60 panels. Upstream's only change to the
file was job_count to jobq_job_count, which this branch already had.

cfg/xrpld-example.cfg - composed. Kept this branch's warning that
service_instance_id must be set explicitly for the metrics pipeline, took
upstream's traces_endpoint rename, and removed a duplicate metrics_endpoint
entry along with the claim that metrics derive from the traces URL by rewriting
the signal path. Nothing derives it; both metric exporters read
metrics_endpoint. 17 keys, one entry each.

05-configuration-reference.md - both sides misdescribed the parser. Kept this
branch's fuller text, corrected the endpoint default to traces_endpoint, and
replaced the "resolve their URL differently" table with what the code does now.

09-data-collection-reference.md, 06-implementation-phases.md,
Phase7_taskList.md - kept this branch's versions, which drop a metric that was
never implemented, correct the state encoding to 0-6, and rename nudb_bytes to
stored_object_bytes. Re-applied upstream's rpc_requests_total fix, which taking
this side had reverted.
2026-09-03 20:37:19 +01:00
Pratik Mankawde
f180b70640 fix(telemetry): correct drain, init and naming defects; add export cadence keys
Addresses review findings on the native-metrics work.

StatsDCollector::onTimer drained the send buffer inside the polling_ gate. That
gate holds back hook handlers until the application's services are built, but
sendBuffers() is socket I/O. StatsDEventImpl derives only from EventImpl, so it
never enters metrics_ and posts straight to the buffer; its |ms timings piled up
before onCollectionReady and were dropped after onCollectionStopping. The drain
now runs every tick, and outside metricsLock_, so onCollectionStopping no longer
waits on a UDP flush.

TelemetryImpl's constructor left meterProvider_ set when initMetrics() threw.
initMetrics publishes globally as its last step, so a throw left getMeter()
callers holding a provider nothing else could reach. Reset it in the catch.

~ApplicationImp caught only std::exception around telemetry shutdown while the
callees reach third-party SDK code, so a foreign exception would have terminated
the process. Added a logging catch-all.

ValidationTracker's hard trim evicted by unordered_map bucket order. It now
evicts oldest-first, so the entry dropped under pressure is the one least likely
to still reconcile.

The GetMeter test restored the global meter provider only on the success path,
and ASSERT_TRUE early-returns past it. Uses xrpl::ScopeExit instead.

The hook debounce window is a named constant rather than a bare 500 in a
comparison, and the metric export cadence becomes operator-configurable through
metric_export_interval_ms and metric_export_timeout_ms. Both are range-checked:
the SDK warns and silently substitutes its own 60s/30s defaults when the timeout
is not below the interval, so an unchecked value would slow export rather than
speed it up. Parsing uses a signed representation because lexical_cast<uint32_t>
accepts a leading minus and wraps it.

Naming corrections: CollectorManager documented exported_instance, which no OTel
dashboard uses; node-health queried job_count where the exported name is
jobq_job_count; network-traffic and overlay-traffic-detail referenced an
undeclared DS_PROMETHEUS variable; the counter table omitted the _total suffix
the Prometheus exporter appends; the plan docs and task list carried an xrpld_
prefix formatName never applies; and OTelCollector::New()'s contract promised its
instanceId, serviceName and networkType arguments were read, contradicting the
definition that marks them unused.
2026-09-03 20:20:13 +01:00
Pratik Mankawde
d806398810 merge: bring the metrics_endpoint key forward from phase-10 2026-09-03 16:48:10 +01:00
Pratik Mankawde
a5d5d0911f merge: bring the metrics_endpoint key forward from phase-8
Telemetry.cpp conflicted. Phase-9 rewrote the metrics pipeline into
makeTracerResource()/makeMetricsResource()/initMetrics() further down the
class, so its side of the region is empty and phase-8's private helper
block does not apply. Resolved to phase-9's structure; phase-8's own
hunks outside the region (the deleted kTracesPath/kMetricsPath, the
verbatim traces URL, the two-endpoint startup log) merged in.

Phase-9's initMetrics() still derives the metrics URL by suffix-swap.
That is fixed in the next commit, not here.
2026-09-03 16:45:09 +01:00
Pratik Mankawde
7b845392d4 feat(telemetry): give metrics its own endpoint key
One [telemetry] key served both OTLP signals, and the metrics URL was
derived from it by suffix-swap: strip a trailing slash, strip a known
signal path if present, append the wanted one. Anything not ending
/v1/traces therefore posted metrics to the traces path, and the OTLP
version was pinned in code where an operator could not reach it.

Adds metrics_endpoint alongside traces_endpoint. Both are full URLs used
verbatim, so traces and metrics can go to different collectors, or to one
whose OTLP paths are not the defaults. signalEndpoint(), kTracesPath and
kMetricsPath are gone; nothing derives an endpoint from another.

The startup log names both URLs, since with two independent endpoints
there was otherwise no way to see where metrics were going.

Also drops exporter=otlp_http from the shipped config and the test
fixture. No branch in the chain reads an `exporter` key: it was a real
Setup member in the first phase-1b implementation, removed when only
OTLP/HTTP was wired up, and already deleted from TESTING.md once on the
same grounds.
2026-09-03 16:39:04 +01:00
Pratik Mankawde
3e4b5c71ff merge: bring the traces_endpoint rename forward from phase-10
Three conflicts, all between this branch's own sync-diagnostics work and
phase-10's older versions. Resolved to this branch in each case, since it
owns the newer content:

- InboundLedger.h keeps the missing-node and receive-depth gauges and the
  fuller acquire-span contract.
- MetricsRegistry.cpp keeps the namespaced label:: constants.
- LedgerMaster.cpp keeps makeLedgerTraceSpan(), which joins the store and
  validate spans into one per-ledger trace by hash.

LedgerMaster.cpp needed a second pass. The automatic merge had kept both
sides outside the conflict markers, nesting phase-10's older promotion
block inside this branch's `if (!pubLedger_)` — so setValidated,
setFull and setValidLedger would each have run twice. Taking this
branch's file wholesale removes the duplicate; brace balance and a single
"Advancing accepted ledger" confirm it.

That resolution drops two things phase-10 was carrying into this file:
the storeSpan/validateSpan guard names, and the explicit scope that keeps
the one-in-256 flag-ledger check outside the ledger.validate measurement.
Both are re-applied on this branch in the next commit; the scope needs a
variable-lifetime check that does not belong in a merge.
2026-09-03 16:12:07 +01:00
Pratik Mankawde
3aff6d4d64 merge: bring the traces_endpoint rename forward from phase-8
Eighteen conflict regions across nine files. Resolved by asking, per
region, which side is the better final state rather than by taking a
branch wholesale.

Telemetry.cpp keeps phase-9's two resource builders. phase-8 offered a
single makeResource() with no node identity; phase-9 splits it into
makeTracerResource() and makeMetricsResource() because the metrics
provider is built in the constructor, before setNodeId() runs, so
xrpl.node.id can only be stamped unconditionally on the tracer side.
Collapsing them would have dropped that attribute, which is what keeps
per-node traces from folding into one identity.

Telemetry.h and the config test compose both sides: phase-9's nodeId
member and its assertion, plus the renamed endpoint.

xrpld-telemetry.cfg keeps phase-9's devnet identity and its
metrics_endpoint, renames the traces key, and drops exporter=otlp_http.
Nothing reads an `exporter` key on any branch in the chain: it was a real
Setup member in the first phase-1b implementation, removed when only
OTLP/HTTP was wired up, and already deleted from TESTING.md once on the
same grounds. The cfg line was the last carrier.

The docs keep phase-9's versions, which are both fuller and more
accurate: the incoming runbook listed the consensus strategy values as
"random" where the code compares against "attribute".

OTelCollector.cpp had five comment-only regions in a file phase-7 owns,
so those take the upstream side.

MetricsRegistry.h's usage example named a member that no longer exists
and the wrong arity; it now matches the real three-argument call and says
where the endpoint comes from.
2026-09-03 16:06:38 +01:00
Pratik Mankawde
8d1eacc5fc merge: bring the traces_endpoint rename forward from phase-6
Three conflicts, all composed rather than resolved by taking a side:

- TelemetryConfig.cpp: phase-6 kept networkTypeFromId file-local with
  [[nodiscard]]; phase-7 had relocated it to public scope for
  Application.cpp. Kept phase-7's relocation, so one definition remains.
  The [[nodiscard]] survives on the declaration in Telemetry.h.
- Telemetry.cpp x2: phase-7 added getMeter overrides, phase-6 added
  [[nodiscard]] to the startSpan below them. Kept both, and put
  [[nodiscard]] on getMeter too.
- TESTING.md: phase-7 had the right metric name (span_calls_total, which
  the spanmetrics namespace produces) but the wrong label. Its
  xrpl.rpc.command appears nowhere else in the branch; the attribute is
  bare `command`, which is what the dashboards query. Took phase-7's
  metric with the correct label.

Both signalEndpoint call sites follow the renamed member. signalEndpoint
itself is left in place: removing it and adding metrics_endpoint is a
design change, not part of propagating a rename.
2026-09-03 15:38:47 +01:00
Pratik Mankawde
5f76f1bdbc merge: bring the traces_endpoint rename forward from phase-4 2026-09-03 15:15:43 +01:00
Pratik Mankawde
e57d3c2f4b merge: bring the traces_endpoint rename forward from phase3-tx-tracing 2026-09-03 15:15:33 +01:00
Pratik Mankawde
65a679b7cd merge: bring the traces_endpoint rename forward from phase2-rpc-tracing 2026-09-03 15:15:33 +01:00
Pratik Mankawde
4baac61dd9 merge: bring the traces_endpoint rename forward from phase-1c 2026-09-03 15:14:20 +01:00
Pratik Mankawde
499984681c merge: bring the traces_endpoint rename forward from phase-1b 2026-09-03 15:13:38 +01:00
Pratik Mankawde
594bb15fe6 Merge branch 'pratik/otel-phase4-consensus-tracing' into pratik/otel-phase5-docs-deployment 2026-09-03 15:09:53 +01:00