Commit Graph

976 Commits

Author SHA1 Message Date
Pratik Mankawde
b0cea67aed fix(telemetry): address final-review + CI clang-tidy findings
CI's clang-tidy leg flagged eight include-cleaner errors and three
misc-const-correctness / readability-convert-member-functions-to-static /
modernize-use-designated-initializers issues, all inside WP-B6's own code.
Fixed as follows:

- `MetricsRegistry.h`: `#include <opentelemetry/metrics/observer_result.h>`
  for ObserverResult; `observeCacheLockHoldPeaks` is now `static` because it
  touches neither instance state nor telemetry members.
- `SHAMapStoreImp.h`: adds direct includes for `<cstddef>`, `<string_view>`
  and `<xrpl/telemetry/SpanNames.h>` (the StaticStr provider). `seconds` in
  `RotationPhase::~RotationPhase` is `[[maybe_unused]]` so a
  `-DXRPL_ENABLE_TELEMETRY=0` build under `-Werror` keeps compiling.
- `SHAMapStoreImp.cpp`: direct includes for `SHAMapStoreSpanNames.h`,
  `SpanGuard.h`, `SpanNames.h`; `RotationPhase` locals that never call
  `setAttribute` are declared `const`; `RotationOutcome` uses designated
  initialisers.

Final-review findings (WP-B6-rotation-stall-tracing.md, "What to check
when reviewing"):

- Panels 74 and 75 on `ledger-sync-health.json` still carried panel 41's
  description, axisLabel, Source and Keywords copy; rewritten to describe
  rotation phase duration and cache lock hold respectively.
- `consensus_view_change_total` and the `view.change` round-span event
  were emitted but not registered with the harness. Added the counter to
  `not_asserted.metrics_excluded` (workload-gated) and annotated the
  `consensus.round` span note with the event and its two attribute keys.

Not fixed (parked, see progress ledger):
- The reviewer's second Important finding — a plan/code contradiction on
  the consensus counter — was based on a misread of the plan; the plan's
  "Rejected alternatives" table lists a new `TraceCategory::Nodestore` and
  the getKeys() fix, not the consensus counter. No action.
- The Minor note about `sweep()`'s peak including lock-acquire time and
  `getKeys()`'s not: `sweep()` acquires and releases the lock via a
  `scoped_lock`, so `noteLockHold` still runs after the release and the
  numbers are comparable. No action.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-09-14 21:23:28 +01:00
Pratik Mankawde
8c3fd205ef feat(telemetry): chart rotation phases, lock holds and job stalls on Ledger Sync Health
Three new panels on `ledger-sync-health.json`, cloned from existing panels
(id 41 for timeseries, id 27 for bargauge) so template filters, tooltip
mode, `xrpl_ident` legend idiom and `spanNulls` all match the surrounding
dashboard exactly. Every existing panel is untouched.

- id 73  bargauge   Job queue row  x=12 y=226  Job Stalls >=1 s (Count By Job Type)
- id 74  timeseries Back-fill row  x=0  y=429  Rotation Phase Duration (p95 by stage)
- id 75  timeseries Back-fill row  x=0  y=439  Cache Lock Hold Peak (us)

Count panels use `increase(...[$__range])` per memory
promql-counting-events-interval-not-rate-interval. Units set explicitly
(short / s / us).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-09-14 21:00:42 +01:00
Pratik Mankawde
9adb6a255d test(telemetry): register the rotation spans and stall metrics with the harness
Adds `nodestore.rotate` and its eight phase children to expected_spans.json,
all `optional: true` because the 5-node localhost harness cluster never reaches
`online_delete`. Their parent-child relationships are asserted but skip-marked
so a run without a rotation stays green.

Adds `cache_metrics{metric="treenode_lock_hold_peak_us"|"fullbelow_lock_hold_peak_us"}`
to the asserted sync_diagnostics group -- both are observable and always emit,
even at zero. Puts `rotation_phase_duration_seconds` and `jobq_stall_total` in
`not_asserted.metrics_excluded`; both are workload-gated.

On the Cloud collector, adds an `ottl_condition` policy that keeps any trace
carrying a span whose name matches `^nodestore\.rotate`, so the 0.5% probabilistic
tail sampler cannot drop a rotation trace. Sampler is OR'd across policies.
2026-09-14 20:44:20 +01:00
Pratik Mankawde
3ad5bb1952 Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics 2026-09-09 19:23:00 +01:00
Pratik Mankawde
a75d277a22 Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-09 19:22:49 +01:00
Pratik Mankawde
2b6a5733e7 merge: bring the component renames forward from phase8-log-correlation
Four doc conflicts, all where this branch had rewritten a passage that upstream
had only renamed. This branch's text is kept in seven of the eight hunks and
the spanmetrics -> span_metrics and otlp/tempo -> otlp_grpc/tempo spellings
carried into it, so the rewrite is not lost and the names stay current.

The exception is the TESTING.md span-call-count comment, where the incoming
side is the fuller text: it explains that the span_ prefix comes from the
connector's namespace setting. That side is taken.

Metric names are untouched — span_calls_total and traces_span_metrics_* are
produced by the connector's namespace, not by its component name.
2026-09-09 19:22:16 +01:00
Pratik Mankawde
6f28d46682 merge: bring the component renames forward from phase7-native-metrics
Two conflicts, both where this branch's log-pipeline additions sat next to the
upstream spanmetrics -> span_metrics rename: the config header comment, which
this branch extended with a logs line, and the integration test, where the
log-correlation step precedes the span-metrics step. This branch's content is
kept in both and the rename carried into it.
2026-09-09 19:20:45 +01:00
Pratik Mankawde
366bcaa328 merge: bring the component renames forward from phase6-statsd
Four conflicts, all where this branch's replacement of the StatsD path with
native OTLP met the upstream spanmetrics -> span_metrics rename. This branch's
design wins in every case; the rename is carried into its text rather than
reverting it, so the connector, its pipeline references, the header comment,
the TESTING.md summary and the runbook all use span_metrics while keeping the
native-OTLP wording.

One addition beyond a straight take-a-side: publish the collector's health
check port. This branch restored the health_check extension and its own
TESTING.md polls http://localhost:13133/ to decide the collector is ready, but
the port was never published on this side of docker-compose.yml, so that check
could not pass from the host. Verified the merged config loads with no
deprecation warnings and that 13133 is published exactly once.

No metric name changed: traces_span_metrics_* already read that way before the
rename, which only ever touched component names and prose.
2026-09-09 19:19:22 +01:00
Pratik Mankawde
988015dc65 merge: bring the OTLP gRPC exporter rename forward from phase5-docs-deployment
Conflict in docker/telemetry/otel-collector-config.yaml, in the service
pipelines: this branch renamed the deprecated spanmetrics connector to
span_metrics and adds the statsd metrics pipeline, while upstream renamed the
deprecated otlp exporter to otlp_grpc. Both kept.

With both renames present the collector now starts with no deprecation
warnings at all, which was the point of the pair.
2026-09-09 19:17:07 +01:00
Pratik Mankawde
c0024c7c57 Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing 2026-09-09 19:16:08 +01:00
Pratik Mankawde
b98a9d3d13 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing 2026-09-09 19:16:08 +01:00
Pratik Mankawde
4dc706c606 merge: bring the OTLP gRPC exporter rename forward from phase1c-rpc-integration
Conflict in docker/telemetry/otel-collector-config.yaml, in the traces
pipeline: this branch added the attributes/hash processor while upstream
renamed the deprecated otlp exporter to otlp_grpc. Both sides kept — the
processor list keeps attributes/hash and the exporter list takes the new name.
Checked that the exporter definition key was renamed to match the reference,
and that the collector still loads the merged config.
2026-09-09 19:15:52 +01:00
Pratik Mankawde
800662a268 corrections
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
2026-09-09 19:12:37 +01:00
Pratik Mankawde
521f00a484 merge: bring the lock-free ValidationTracker forward from phase10-workload-validation
Only the workload README conflicted; the tracker, config and test files merged
clean, which closes the chain from phase-7.
2026-09-09 17:10:15 +01:00
Pratik Mankawde
8b1e577931 fix(telemetry): tear down the docker volumes on every integration-test run
`docker compose down` keeps the `tempo-data` named volume, so a previous
run's traces stay in Tempo and can satisfy this run's span searches. Pass
-v in cleanup(), and tear the stack down before starting it: a run that
reaches the summary deliberately leaves the stack up, so nothing else
clears it.
2026-09-09 16:10:46 +01:00
Pratik Mankawde
2ee24d4ab8 merge: bring the lock-free ValidationTracker forward from phase9-metric-gap-fill
Only the phase-10 task list conflicted, in its checklist section; the tracker,
config and test files merged clean.
2026-09-09 16:10:01 +01:00
Pratik Mankawde
d141405ff3 merge: bring the lock-free ValidationTracker forward from phase8-log-correlation
The three tracker files conflicted because both sides had rewritten them. Took
the incoming lock-free class, then put this branch's two monotonic accessors
back on top of it: totalAgreementsEver() and totalMissedEver(), backed by a
gross pair incremented at first classification and left alone by the repair
branch. MetricsRegistry reads both, so dropping them would not compile.

The test file kept the incoming suite, which renames every case, and gained
this branch's two gross-counter cases adapted to the injected clock.
2026-09-09 15:59:55 +01:00
Pratik Mankawde
5cb829a755 merge: bring the lock-free ValidationTracker forward from phase7-native-metrics
No conflicts. Carries the lock-free tracker, the metrics_endpoint scheme guard
with its repaired test set, and phase-6's Tempo time bounds.
2026-09-09 15:49:43 +01:00
Pratik Mankawde
c622da5a76 refactor(telemetry): use the current name for the span metrics connector
The pinned collector warns on every start that "spanmetrics" is a deprecated
alias for "span_metrics". Rename the connector, its pipeline references and
the prose that names it.

The derived metric names are untouched. They come from the connector's
`namespace` setting rather than its component name, so occurrences inside a
metric name such as traces_spanmetrics_calls_total are deliberately left as
they are; renaming those would break every span panel. The rename is applied
only to the bare word, never where it is joined to a metric name by
underscores.

This branch introduces the connector, so the change belongs here.
2026-09-09 15:40:11 +01:00
Pratik Mankawde
87457446be refactor(telemetry): use the current name for the OTLP gRPC exporter
The pinned collector warns on every start that "otlp" is a deprecated alias
for "otlp_grpc". Rename the trace exporter to otlp_grpc/tempo.

Only the exporter is affected. The otlp RECEIVER keeps its name: it serves
both gRPC and HTTP under one component and is not deprecated, verified by
renaming the exporter alone and seeing the warning stop.

This belongs on this branch because it introduces the exporter, and it is the
last of four deprecated aliases in the collector config; the other three are
owned by later branches in the chain.
2026-09-09 15:40:09 +01:00
Pratik Mankawde
ac07e1345f fix(telemetry): name the harness log directories after their instance ids
The collector reads the per-node directory off the log file path and stamps it
as the Loki label service_instance_id, so the directory name has to equal the
node's own [telemetry] service_instance_id or log lines carry a node name that
no trace or metric shares and nothing joins.

Both harness scripts disagreed with themselves: run-full-validation.sh wrote to
node$i while setting validator-${i}, and benchmark.sh wrote to node$i while
setting bench-node-${i}. Rename the directories to match the ids rather than
the reverse, so no existing trace or metric label value moves and no harness
expectation has to be re-checked. Only path references are renamed; the
human-readable "node$i" in log and error messages is left as prose.

The config template is not rendered by any script, so its DATA_DIR
documentation gains a note about the same constraint instead.

Also rename the deprecated otlphttp/filelog collector component names in the
harness scripts and docs.
2026-09-09 15:12:36 +01:00
Pratik Mankawde
7f8f5b4be3 feat(telemetry): ship logs through Alloy, alongside traces and metrics
Alloy carried no log pipeline at all: no loki.source, loki.write or
loki.process anywhere in the config, against a working metrics and OTLP path.
Any deployment fed through Alloy rather than the reference collector therefore
sent traces and metrics but no logs, so log-to-trace correlation was
unavailable there even though both ends of the link were configured.

Logs now leave through the same otlphttp exporter as the other two signals, so
all three carry one resource identity. Alloy's own filelog receiver would have
been the closest match to the reference collector, but it is still
public-preview and refuses to load unless the service is started with
--stability.level=public-preview, which would mean editing the unit on a box
reachable only by RunCommand. The Loki source components are generally
available, so they are used and bridged into OTLP by otelcol.receiver.loki;
the service needs no extra flag.

That bridge hands over an empty resource and puts everything on the log
record, so the transform sets the resource attributes in log context.
service.instance.id is concatenated in from XRPLD_HOST_LABEL because OTTL has
no env() converter, and it is a resource attribute rather than a record one
because only resource attributes are promoted to indexed Loki labels. It must
equal the node's own service_instance_id or the logs join nothing. devnet
writes one flat file rather than a per-node directory, so identity cannot be
read off the path the way the docker collector does it; XRPLD_LOG_GLOB
overrides the path for other layouts.

The line is parsed for its own timestamp, severity and trace context, and
trace_id/span_id are set on the first-class OTLP record fields so Grafana
links a log to its trace without re-parsing the body. Lines emitted outside a
sampled span keep an empty trace id rather than an invalid one.

Also carry the node-identity operators into the Grafana Cloud collector
variant, align this config's log directory with its service_instance_id, and
rename the deprecated otlphttp/filelog collector component names. Alloy's
otelcol.exporter.otlphttp is that product's own name and is unchanged.
2026-09-09 15:12:18 +01:00
Pratik Mankawde
cbb8581997 fix(telemetry): make the log pipeline actually deliver, and fix its docs
Addresses the open review findings on this branch.

The log root was never delivered at all. Docker creates a missing bind-mount
source as root, Config::getDebugLogFile() only warns when it cannot create the
network subdirectory inside it, and Application carries on. The node therefore
looked healthy while writing no debug.log, and Loki stayed empty with no error
at any layer. docker/telemetry/data/logs has in fact been root-owned in a
working checkout since it was first created. A one-shot xrpld-logdir-init
service now creates the directory and hands it to XRPLD_UID/XRPLD_GID,
following the pattern the storage-init service already uses.

Ingested logs carried no node identity, so a multi-node stack collapsed into
one indistinguishable stream while every dashboard filters on
service_instance_id. The receiver now sets include_file_path and lifts the
per-node directory onto the resource attribute service.instance.id, which is
on the allow-list Loki promotes to an indexed stream label. A record attribute
would only become structured metadata and could not be used in a selector.
For that to join anything the directory name has to equal the emitter's
service_instance_id, so the node directories are renamed to match: node$i
becomes Node-$i, and the standalone config writes to logs/xrpld-standalone.

The integration test aborted before reporting. Under set -o pipefail the
grep | head -1 pipeline is killed by SIGPIPE once the log exceeds the pipe
buffer, so the run exited 141 somewhere past a few hundred matching lines and
read as a flaky test. grep -m1 stops on its own. The test also verified the
local file and Tempo but never that a line reached Loki, which is the one hop
this branch adds, so a bounded Loki assertion is added alongside a readiness
wait.

Documentation fixes: the Tempo cross-check counted .data, but Tempo returns
OTLP shape so the array is batches and one trace can span several; the Loki
step used the instant /query endpoint, which rejects a bare log selector with
HTTP 400 and a text/plain body, so jq could never parse it and the step never
printed a number even when ingestion worked. The filelog comment claimed six
fractional digits where the node always emits nine. The two flowcharts used
<br/>, carried no legend, and advertised GetSpan(), which Log.cpp deliberately
avoids in favour of reading the thread-local context directly.

Finally, rename the deprecated collector component names: the pinned
collector warns on every start that otlphttp and filelog are aliases for
otlp_http and file_log. Alloy's otelcol.exporter.otlphttp and
otelcol.receiver.filelog are that product's own component names and are not
deprecated, so they are left alone.
2026-09-09 15:11:49 +01:00
Pratik Mankawde
ea4f7fd0d4 fix(telemetry): restore health_check and batch the metrics pipeline
The health_check extension was present on the previous branch and dropped
here with no replacement, while this branch's own TESTING.md still polls
http://localhost:13133/ to decide the collector is ready. That check has had
no listener since, so the documented readiness step cannot pass.

Also add batch to the metrics pipeline. Without it the OTLP metric path
exports one request per instrument; the added delay is bounded by the batch
timeout, well under the Prometheus scrape interval.

Both belong here rather than downstream: this branch owns the metrics
pipeline and is the one that regressed the extension.
2026-09-09 15:11:25 +01:00
Pratik Mankawde
cbd6cdbbdd merge: bring the integration-test bounds forward from phase6-statsd
Two files conflicted and both were composed rather than taken from one side.

integration-test.sh: kept this branch's spanmetrics names, since the collector
sets namespace: "span" here and traces_span_metrics_* matches nothing, and took
phase-6's --max-time on every probe. The Tempo time bound needed restoring by
hand: RUN_START, the check_span guard and the start/end parameters are on
phase-6 and absent here, so a plain resolution kept phase-6's comment about
bounding the search while shipping no bound. All four pieces are back.

TESTING.md: kept server=otel with the metrics endpoint. Phase-6's template sets
server=statsd and documents prefix, which this branch's OTel path ignores.
2026-09-09 15:03:08 +01:00
Pratik Mankawde
478b3e4b07 docs(telemetry): correct stale claims and citations in the harness docs
The workload README contradicted itself on --skip-loki: one bullet said CI always
passes it and so the two log-correlation checks are never exercised, another said
the workflow no longer passes it. The workflow mentions the flag nowhere, so the
first was the stale half.

Other claims checked against the tree and corrected:

- both the README and the plan doc described the push trigger as filtered on
  branch names. The workflow has no branches filter, deliberately, because
  GitHub ANDs branches with paths
- the plan doc printed 6 of the workflow's 12 paths globs, and claimed the
  workflow was 367 lines against an actual 451. The glob block is now generated
  from the workflow, and the line count dropped rather than restated
- rpcNOT_SUPPORTED does not exist anywhere in the tree. The symbol is
  RpcNotSupported, and the refusal sites are RipplePathFind.cpp:59-60 and
  PathFind.cpp:50-51, not :48-49 and :39
- RCLConsensus.cpp:666 and :663 are not log or event lines; the tx.included event
  is at :720 and the per-transaction debug log at :715
- LedgerMaster.cpp:463 is fixIndex, not the ledger.store span, which is at :470
- ServerHandler.cpp:705 is inside makeJsonError; processRequest is at :718
- file counts: docker/telemetry/workload/ is 25 files, include/xrpl/telemetry/ 13
- the optional-span bullet named five causes covering 10 of 16 entries, omitting
  the txq.* family and the WebSocket handshake
- the /api/v1/series choice was attributed to stale StatsD gauges; this harness
  runs no StatsD

A line number in run-full-validation.sh was cited in five places and drifts on
every edit to that file, so those now name the file only. The keygen helper's
header records what production does instead -- validator-keys-tool create_keys
then create_token, keeping the master key off the node -- and why a disposable
cluster does not.
2026-09-09 13:16:13 +01:00
Pratik Mankawde
ec308c6b00 fix(telemetry): poll the parity queries instead of racing one instant query
The four external-parity bounds checks each ran a single Prometheus instant query
and failed on an empty result. The metric checks that run earlier poll
/api/v1/series, which returns a series regardless of staleness, but a bounds
check needs the sample value and so cannot use that endpoint. This file's own
docstring records the consequence: a beast::insight gauge that stops changing can
fall out of an instant query while /api/v1/series still returns it, so one
attempt is not enough to call the series absent.

Poll to the same deadline the metric checks use. A Prometheus error is raised
rather than retried, because a rejected query never becomes valid and retrying it
only burns the full timeout.
2026-09-09 13:15:50 +01:00
Pratik Mankawde
7f829a5929 fix(telemetry): fail the regression gate on a unit change, and report what it gated
compare_to_baseline took the unit from the baseline entry and dropped the current
run's, and nothing compared the two, so a us -> ms change was scored as a numeric
delta: four keys rewritten to the same physical durations reported 99.9%
improvements and the gate exited 0. prom_queries.py says the baseline preserves
the unit "so the comparator can sanity-check unit drift"; it never did. A unit
mismatch now fails and names both units.

The workflow's step summary printed total, regressions and improvements. total is
every key in the report -- the union of baseline and current -- so it was neither
the baseline count nor what was gated, and missing_in_current was computed and
never printed. A run that gated 16 of 20 keys read as a full comparison. The
comparator now reports a real "compared" count and the summary prints it beside
the not-captured count, with a warning when any key was missed. The table also
refused nothing on a truncated report; existence is not readability.

check_regression_bounds told the operator to add max_abs_increase while reading
max_abs_increase_ms / _us, so following the message added a key nothing reads and
the gate kept failing with no explanation. The committed thresholds use only the
suffixed spelling, so the message was the defect. Its three JSON inputs were also
unchecked: a top-level null, list or number parsed and then died on the first
.get, and a string "metrics" survived the placeholder test and reported its own
characters as gated keys -- wrong advice rather than a crash.

Four tests cover these; all four fail against the previous checker.
2026-09-09 13:15:46 +01:00
Pratik Mankawde
c8d9d88113 fix(telemetry): measure telemetry overhead under load, on this cluster only
The overhead benchmark generated no workload. Each arm was start_cluster ->
collect_metrics -> stop_cluster, and collect_metrics only ran the sampler, so the
only client traffic was the sampler's own server_info probes at under
1 request/sec. The hottest instrumented paths -- tx.*, txq.*, the transactor
stage spans, every rpc.command.* other than server_info -- were never entered,
which is where per-operation span cost appears. Both arms now drive
rpc_load_generator and tx_submitter at one fixed rate for the whole window, over
a [port_ws] listener present in both arms so the listener is not part of the
delta. A flat rate rather than a workload profile, because both arms must issue
the same work and a profile's phase shaping only adds variance.

The sampler also selected xrpld host-wide. run-full-validation.sh leaves its five
validation nodes running while the benchmark's three start, so both arms averaged
eight processes -- diluting the CPU delta and making memory_rss_mb_peak report a
validation node either way. It now takes an optional pid list, and the benchmark
passes its own nodes' pids and refuses to measure if it cannot collect them all.

consensus_round_mean_ms counted distinct ledger sequences seen by a loop that
sampled every 5 s, so it read back 5000 ms for every close time from 2 s to 5 s
and a 10% regression measured 0%. Sampling at 2 s -- the close-time floor from
ConsensusParms.h:93 -- resolves a 10% regression as at least 9.3%. It also
divided by the requested DURATION rather than the measured ELAPSED, which the
TPS calculation in the same file already used.

Key generation, the workdir setup and the seed read exited 1 under errexit, the
code this script reserves for a measured threshold breach, so an infrastructure
failure was reported as "telemetry is too expensive". They map to cannot_measure
now. No guard is added after the config heredoc: a guard there is read as the
heredoc's first line, lands in the generated config and never runs.

curl probes across the harness had no --max-time, so a server that accepts the
connection and then stops answering blocks forever and the loops' attempt counts
stop bounding anything.
2026-09-09 13:15:23 +01:00
Pratik Mankawde
d2cefa05d9 fix(telemetry): make the load generators fail loudly instead of exiting 0
Three ways a run could produce no traffic and still report success:

- tx_submitter logged a funding shortfall and returned an empty stats object;
  main() then printed the summary and exited 0, so the failure only surfaced
  later as "spans missing", which points nowhere. It now records setup_failed in
  the summary and exits 1 after the report is written.
- --weights was checked for valid JSON but not for a positive sum. An all-zero
  mapping reached random.choices, which raises ValueError from inside the
  dispatch loop where only CancelledError is caught. Rejected at parse time now,
  in both generators.
- a profile phase declaring neither rpc nor tx logged a warning and returned no
  error. Both error rates short-circuit to 0.0 when nothing was sent, so a
  mistyped key produced zero traffic and still passed the exit gate. That phase
  is now an error.
2026-09-09 13:14:44 +01:00
Pratik Mankawde
8d2e2d15af docs(telemetry): correct stale code citations and the log-correlation claim
Every citation below was checked against the file it names:

- LedgerMaster.cpp:463 is fixIndex, not the ledger.store span; that guard is at
  :470 and the insert it wraps at :476
- LedgerMaster.cpp:987 is the tvc assignment, which sits BEFORE the tvc < minVal
  return at :988; the ledger.validate span opens at :1003
- ServerHandler.cpp:705 is inside makeJsonError; processRequest is at :718
- docker-compose.yml:71 and :75 are comments in the collector's volume block;
  the loki service is at :112 and its config command at :116

Two claims were also wrong rather than merely stale. Log-trace correlation is
gated in CI, because the workflow passes no --skip-loki, and the separate check
in integration-test.sh is run by no workflow at all. The Loki label note
described the Grafana Cloud collector config rather than the local one: only the
cloud variant sets job=xrpld, and the local config's own comment says to select
on service_name. The dashboards carry 35 Loki queries, not 38.
2026-09-09 13:14:21 +01:00
Pratik Mankawde
d972772747 docs(telemetry): match the manual test template to the OTel insight path
This branch switches integration-test.sh to [insight] server=otel and adds an
assertion that no StatsD listener is needed, but TESTING.md still described the
metrics it verifies as StatsD-derived and its manual node-config template had no
[insight] stanza at all.

CollectorManagerImp falls through to NullCollector when server is neither statsd
nor otel, so a reader building configs from that template got zero
beast::insight metrics. The template also omitted service_instance_id and
metrics_endpoint, which the script writes; without the former every node is
indistinguishable in the $node dashboard filter.
2026-09-09 13:14:15 +01:00
Pratik Mankawde
6976539274 fix(telemetry): bound the integration-test probes and align its config template
curl applies no overall timeout of its own, so a server that accepts the
connection and then stops answering parks a poll loop for the rest of the run
and the loop's attempt count stops bounding anything. Add a CURL_MAX_TIME
ceiling and apply it to all 18 executable probes in integration-test.sh.

TESTING.md's manual node-config template also disagreed with what the script
writes, so a reader following it could not reproduce the automated path:

- no [insight] stanza, so no beast::insight metric leaves the node at all and
  Step 10b's ten rippled_* assertions cannot pass
- [ips_fixed] listed all six peer ports including the node's own

The log level is deliberately untouched: the template and the script agree on
warning here.
2026-09-09 13:13:41 +01:00
Pratik Mankawde
30d165da43 fix(telemetry): bound integration-test assertions to the run under test
check_statsd_metric queried rippled_rpc_requests, which no pipeline
produces: the collector's statsd receiver runs with is_monotonic_counter,
so the Prometheus exporter appends _total. A wrong name returns zero
series rather than an error, so the assertion could not be told apart
from a broken pipeline. All eight assertions were re-derived from how
each metric is created in code; this was the only counter.

Tempo searches carried no start/end, and tempo-data is a named volume
that `docker compose down` preserves under a one-hour block retention, so
the 17 span assertions could pass on an earlier local run's traces. Bound
every search to this run, and tear the stack down with -v before starting
so no earlier data is present to match. The service-name check now
matches a whole line, because the tag-values endpoint ignores start/end.

Add a gtest for the StatsD gauge that publishes its initial zero and for
the counter that must publish nothing. Assert two metrics the harness
never checked: a traffic-category gauge no message reaches, and
io_context latency.
2026-09-08 16:43:41 +01:00
Pratik Mankawde
2c7f94bd72 merge: bring the close-time attr doc fixes forward from phase10-workload-validation
The ledger span table conflicted: this branch had already added ledger_hash
to the validate, store and acquire rows. Keep this branch's table and apply
the close-time rename to the ledger.build row.
2026-09-04 12:44:26 +01:00
Pratik Mankawde
8c124ab14c merge: bring the close-time attr doc fixes forward from phase9-metric-gap-fill 2026-09-04 12:43:52 +01:00
Pratik Mankawde
f3b0af8527 docs(telemetry): name the renamed close-time attr in the acquire comment
The cardinality note points at the close-time dimension rule above it, so
it needs the emitted key name.
2026-09-04 12:39:53 +01:00
Pratik Mankawde
0ee97a7951 docs(telemetry): follow the close-time attr rename in the key lists
Update the consensus and ledger key inventories and the spanmetrics
dimension comment in both collector configs, so they name the emitted
keys. The comment's last_close_time reference is a server_info gauge, not
the span attribute, and is left as it is.
2026-09-04 12:39:05 +01:00
Pratik Mankawde
198207eee4 merge: bring the close-time attr harness fix forward from phase10-workload-validation 2026-09-04 11:48:11 +01:00
Pratik Mankawde
f80faea85d fix(telemetry): match the renamed close-time attrs in expected_spans.json
The close-time span attributes name their unit and epoch:
close_time_ripple_epoch_s, parent_close_time_ripple_epoch_s and
close_time_self_ripple_epoch_s. The harness inventory still required the
unsuffixed keys, so the attribute checks for consensus.accept.apply and
ledger.build failed on every validation run while the spans themselves
were correct.

Rename the four required_attributes entries to the keys the code emits.
2026-09-04 11:47:33 +01:00
Pratik Mankawde
771ea277a5 merge: bring telemetry config and doc fixes forward from phase7-native-metrics 2026-09-03 20:30:33 +01:00
Pratik Mankawde
f180b70640 fix(telemetry): correct drain, init and naming defects; add export cadence keys
Addresses review findings on the native-metrics work.

StatsDCollector::onTimer drained the send buffer inside the polling_ gate. That
gate holds back hook handlers until the application's services are built, but
sendBuffers() is socket I/O. StatsDEventImpl derives only from EventImpl, so it
never enters metrics_ and posts straight to the buffer; its |ms timings piled up
before onCollectionReady and were dropped after onCollectionStopping. The drain
now runs every tick, and outside metricsLock_, so onCollectionStopping no longer
waits on a UDP flush.

TelemetryImpl's constructor left meterProvider_ set when initMetrics() threw.
initMetrics publishes globally as its last step, so a throw left getMeter()
callers holding a provider nothing else could reach. Reset it in the catch.

~ApplicationImp caught only std::exception around telemetry shutdown while the
callees reach third-party SDK code, so a foreign exception would have terminated
the process. Added a logging catch-all.

ValidationTracker's hard trim evicted by unordered_map bucket order. It now
evicts oldest-first, so the entry dropped under pressure is the one least likely
to still reconcile.

The GetMeter test restored the global meter provider only on the success path,
and ASSERT_TRUE early-returns past it. Uses xrpl::ScopeExit instead.

The hook debounce window is a named constant rather than a bare 500 in a
comparison, and the metric export cadence becomes operator-configurable through
metric_export_interval_ms and metric_export_timeout_ms. Both are range-checked:
the SDK warns and silently substitutes its own 60s/30s defaults when the timeout
is not below the interval, so an unchecked value would slow export rather than
speed it up. Parsing uses a signed representation because lexical_cast<uint32_t>
accepts a leading minus and wraps it.

Naming corrections: CollectorManager documented exported_instance, which no OTel
dashboard uses; node-health queried job_count where the exported name is
jobq_job_count; network-traffic and overlay-traffic-detail referenced an
undeclared DS_PROMETHEUS variable; the counter table omitted the _total suffix
the Prometheus exporter appends; the plan docs and task list carried an xrpld_
prefix formatName never applies; and OTelCollector::New()'s contract promised its
instanceId, serviceName and networkType arguments were read, contradicting the
definition that marks them unused.
2026-09-03 20:20:13 +01:00
Pratik Mankawde
d806398810 merge: bring the metrics_endpoint key forward from phase-10 2026-09-03 16:48:10 +01:00
Pratik Mankawde
f52e654c53 merge: bring the metrics_endpoint key forward from phase-9 2026-09-03 16:47:45 +01:00
Pratik Mankawde
a5d5d0911f merge: bring the metrics_endpoint key forward from phase-8
Telemetry.cpp conflicted. Phase-9 rewrote the metrics pipeline into
makeTracerResource()/makeMetricsResource()/initMetrics() further down the
class, so its side of the region is empty and phase-8's private helper
block does not apply. Resolved to phase-9's structure; phase-8's own
hunks outside the region (the deleted kTracesPath/kMetricsPath, the
verbatim traces URL, the two-endpoint startup log) merged in.

Phase-9's initMetrics() still derives the metrics URL by suffix-swap.
That is fixed in the next commit, not here.
2026-09-03 16:45:09 +01:00
Pratik Mankawde
4587986a93 merge: bring the metrics_endpoint key forward from phase-7 2026-09-03 16:40:59 +01:00
Pratik Mankawde
7b845392d4 feat(telemetry): give metrics its own endpoint key
One [telemetry] key served both OTLP signals, and the metrics URL was
derived from it by suffix-swap: strip a trailing slash, strip a known
signal path if present, append the wanted one. Anything not ending
/v1/traces therefore posted metrics to the traces path, and the OTLP
version was pinned in code where an operator could not reach it.

Adds metrics_endpoint alongside traces_endpoint. Both are full URLs used
verbatim, so traces and metrics can go to different collectors, or to one
whose OTLP paths are not the defaults. signalEndpoint(), kTracesPath and
kMetricsPath are gone; nothing derives an endpoint from another.

The startup log names both URLs, since with two independent endpoints
there was otherwise no way to see where metrics were going.

Also drops exporter=otlp_http from the shipped config and the test
fixture. No branch in the chain reads an `exporter` key: it was a real
Setup member in the first phase-1b implementation, removed when only
OTLP/HTTP was wired up, and already deleted from TESTING.md once on the
same grounds.
2026-09-03 16:39:04 +01:00
Pratik Mankawde
3e4b5c71ff merge: bring the traces_endpoint rename forward from phase-10
Three conflicts, all between this branch's own sync-diagnostics work and
phase-10's older versions. Resolved to this branch in each case, since it
owns the newer content:

- InboundLedger.h keeps the missing-node and receive-depth gauges and the
  fuller acquire-span contract.
- MetricsRegistry.cpp keeps the namespaced label:: constants.
- LedgerMaster.cpp keeps makeLedgerTraceSpan(), which joins the store and
  validate spans into one per-ledger trace by hash.

LedgerMaster.cpp needed a second pass. The automatic merge had kept both
sides outside the conflict markers, nesting phase-10's older promotion
block inside this branch's `if (!pubLedger_)` — so setValidated,
setFull and setValidLedger would each have run twice. Taking this
branch's file wholesale removes the duplicate; brace balance and a single
"Advancing accepted ledger" confirm it.

That resolution drops two things phase-10 was carrying into this file:
the storeSpan/validateSpan guard names, and the explicit scope that keeps
the one-in-256 flag-ledger check outside the ledger.validate measurement.
Both are re-applied on this branch in the next commit; the scope needs a
variable-lifetime check that does not belong in a merge.
2026-09-03 16:12:07 +01:00
Pratik Mankawde
3e930d9d37 merge: bring the traces_endpoint rename forward from phase-9 2026-09-03 16:07:14 +01:00
Pratik Mankawde
3aff6d4d64 merge: bring the traces_endpoint rename forward from phase-8
Eighteen conflict regions across nine files. Resolved by asking, per
region, which side is the better final state rather than by taking a
branch wholesale.

Telemetry.cpp keeps phase-9's two resource builders. phase-8 offered a
single makeResource() with no node identity; phase-9 splits it into
makeTracerResource() and makeMetricsResource() because the metrics
provider is built in the constructor, before setNodeId() runs, so
xrpl.node.id can only be stamped unconditionally on the tracer side.
Collapsing them would have dropped that attribute, which is what keeps
per-node traces from folding into one identity.

Telemetry.h and the config test compose both sides: phase-9's nodeId
member and its assertion, plus the renamed endpoint.

xrpld-telemetry.cfg keeps phase-9's devnet identity and its
metrics_endpoint, renames the traces key, and drops exporter=otlp_http.
Nothing reads an `exporter` key on any branch in the chain: it was a real
Setup member in the first phase-1b implementation, removed when only
OTLP/HTTP was wired up, and already deleted from TESTING.md once on the
same grounds. The cfg line was the last carrier.

The docs keep phase-9's versions, which are both fuller and more
accurate: the incoming runbook listed the consensus strategy values as
"random" where the code compares against "attribute".

OTelCollector.cpp had five comment-only regions in a file phase-7 owns,
so those take the upstream side.

MetricsRegistry.h's usage example named a member that no longer exists
and the wrong arity; it now matches the real three-argument call and says
where the endpoint comes from.
2026-09-03 16:06:38 +01:00