Commit Graph

365 Commits

Author SHA1 Message Date
Pratik Mankawde
9fd5414fe6 merge: bring the traces_endpoint rename forward from phase-7
Ten conflict regions in four files, none of them caused by the rename.
phase-8's own commit had rewritten comments in files phase-7 owns
(Unit.h, HistogramBuckets.h, OTelCollector.cpp) and in Telemetry.cpp,
while the same sweep ran independently on phase-7.

Resolved every region to phase-7's side on ownership grounds: those files
belong to phase-7 or earlier, so a downstream branch should not carry
divergent copies. phase-8 changed comments only in all four, verified
against the merge base, so no code was dropped.

The one structural region: phase-7 had refactored addUnitView from an
inline lambda into a member function, and phase-8 still held the lambda.
Keeping phase-8's would have shadowed the member.

Wording phase-8 had that is worth restoring on phase-7 -- the "legacy"
qualifier on the prefix parameter, and the rejected-alternative note on
the bucket ladders -- is recorded outside the tree for a follow-up.
2026-09-03 15:43:24 +01:00
Pratik Mankawde
8d1eacc5fc merge: bring the traces_endpoint rename forward from phase-6
Three conflicts, all composed rather than resolved by taking a side:

- TelemetryConfig.cpp: phase-6 kept networkTypeFromId file-local with
  [[nodiscard]]; phase-7 had relocated it to public scope for
  Application.cpp. Kept phase-7's relocation, so one definition remains.
  The [[nodiscard]] survives on the declaration in Telemetry.h.
- Telemetry.cpp x2: phase-7 added getMeter overrides, phase-6 added
  [[nodiscard]] to the startSpan below them. Kept both, and put
  [[nodiscard]] on getMeter too.
- TESTING.md: phase-7 had the right metric name (span_calls_total, which
  the spanmetrics namespace produces) but the wrong label. Its
  xrpl.rpc.command appears nowhere else in the branch; the attribute is
  bare `command`, which is what the dashboards query. Took phase-7's
  metric with the correct label.

Both signalEndpoint call sites follow the renamed member. signalEndpoint
itself is left in place: removing it and adding metrics_endpoint is a
design change, not part of propagating a rename.
2026-09-03 15:38:47 +01:00
Pratik Mankawde
8c695b69a5 fix(telemetry): follow the traces_endpoint rename in the node configs
The rename arrived from phase-1b by merge. Four files still wrote the old
key, which the parser no longer reads, so each would have silently
fallen back to the default collector URL.

integration-test.sh is the load-bearing one: it generates the node config
the test harness starts, so the stale key would have pointed the node at
localhost regardless of the compose network. xrpld-telemetry.cfg is the
standalone node config; the other two document the key.

Note this cfg has a second, divergent variant on the devnet branches that
needs the same fix there.
2026-09-03 15:21:01 +01:00
Pratik Mankawde
91c7137373 merge: bring the traces_endpoint rename forward from phase-5 2026-09-03 15:20:30 +01:00
Pratik Mankawde
e57d3c2f4b merge: bring the traces_endpoint rename forward from phase3-tx-tracing 2026-09-03 15:15:33 +01:00
Pratik Mankawde
65a679b7cd merge: bring the traces_endpoint rename forward from phase2-rpc-tracing 2026-09-03 15:15:33 +01:00
Pratik Mankawde
4baac61dd9 merge: bring the traces_endpoint rename forward from phase-1c 2026-09-03 15:14:20 +01:00
Pratik Mankawde
f68cf0d009 refactor(telemetry): name the traces endpoint for its signal
[telemetry] endpoint carried one OTLP signal while its name implied it
covered every signal. That asymmetry is what let the metrics URL be
guessed later by rewriting this one's path suffix, so anything not
ending /v1/traces silently posted metrics to the traces path.

Renames the key to traces_endpoint and Setup::exporterEndpoint to
tracesEndpoint. The default value is unchanged and the URL is still used
verbatim, with no path derived from it. The startup log line and the
compose-file example name the new key, the latter being where an
operator copies it from.

No metrics_endpoint is added here: this branch has no metrics pipeline,
so the key would parse into a member nothing reads.
2026-09-03 15:09:19 +01:00
Pratik Mankawde
d687903c3f fix(telemetry): correct the metric names, guard the arithmetic, tidy the collector
Harness and docs:

- integration-test.sh queried traces_span_metrics_* for spanmetrics, but this
  branch sets the connector namespace to "span", so those two checks matched
  nothing and failed. The dashboards and runbook had moved; the script had not.
- The same script queried eight native metric names with a product prefix and
  capitals that formatName() cannot produce: it lowercases, maps '.' and ' ' to
  '_', and prepends nothing. Corrected against the runbook tables.
- TESTING.md carried the same stale spanmetrics names and a jq example reading
  a Prometheus label that does not exist.
- The runbook now records where each part of a derived metric name comes from,
  since only the namespace is ours to choose.

Collector:

- OTelCounterImpl::increment silently dropped a negative amount. An OTel
  counter takes unsigned deltas, so assert and let a release build under-count
  rather than wrap.
- OTelGaugeImpl::increment computed current + amount in int64, which is
  undefined on overflow, and the clamp ran afterwards so it could not help.
  Check the headroom first. set() now clamps rather than casting a uint64 above
  INT64_MAX to a negative, which is what made underflow reachable.
- The meter scope was two bare literals. They are constants now, and
  Telemetry.cpp static_asserts them equal to kMeterName and kMeterVersion:
  beast cannot include the telemetry header, so a build failure is the only way
  to catch the copies drifting.
- formatName uses views::transform and ranges::to, as Backend.cpp already does.
- Unused constructor parameters take [[maybe_unused]] instead of (void) casts.
- The destructor logged "shutting down" and "stopped" with nothing between.

initMetrics was 79 lines doing four jobs. The exporter and the histogram views
are separate functions now, addUnitView is a member rather than a lambda
capturing this, and the export interval and timeout are named. It also derived
the metrics URL from the traces URL by suffix swap, which sent metrics to the
traces path whenever the configured URL had any other shape; both URLs now come
from one rule that handles a bare host, a trailing slash and either signal path.
2026-09-03 10:52:42 +01:00
Pratik Mankawde
5f53745fbf docs(telemetry): correct the TESTING guide and the span attribute tables
Review feedback on the testing guide:

- rm -rf targeted data/, but this config writes under docker/telemetry/data/,
  so teardown did nothing and a second run reused the old NuDB and SQLite
  state. Corrected at both sites, including the Test 2 keygen node, which
  launches with the same config.
- The standalone span table said consensus.* does not fire. It does:
  ledger_accept drives a simulated round, so consensus.round, .phase.open,
  .ledger_close, .accept and .accept.apply all appear. Only .establish,
  .update_positions, .check, .proposal.* , .validation.receive and
  .mode_change cannot. The test intro claimed the same thing and now agrees
  with the table.
- Three blocks duplicated content the file already had. Test 1 now points at
  the shared Verification Queries section as Test 2 already did, and the
  Test 2 submit block checks engine_result like Test 1 does.
- The numbered step list was a copy of the script's own Step N headers and had
  drifted by four entries, so it now points at those headers instead.

Also corrects the runbook's ledger and peer span tables against the code:
ledger.build was credited with tx_count and tx_failed, which tx.apply sets,
and was missing its three close-time attributes; peer.validation.receive was
missing ledger_hash and full_validation. The five source line numbers in those
two tables were stale, so they now name the file only, as the other nineteen
rows do.
2026-09-03 10:52:04 +01:00
Pratik Mankawde
4f4c9e8e3e fix(telemetry): correct span scope, span-guard names and the span docs
Review feedback, plus a sweep of the branch for the same defects elsewhere.

Scope: the ledger.validate guard was a plain local, so it stayed alive until
checkAccept returned and the flag-ledger upgrade check ran inside the measured
span. That check reads every trusted validation of the parent, so one span in
256 became a duration outlier for work unrelated to promoting a ledger. The
span is now scoped to the promotion. tryAdvance stays inside it because it only
sets a flag and posts a job.

Attributes: tx.apply now carries ledger_seq, which the runbook already
documented. The parent ledger.build span has it, but a child cannot be selected
by its parent's attributes, so the span could not be found by ledger.

Guard names: each span guard is now named after the span it holds, so
proposalReceiveSpan, validationReceiveSpan, storeSpan and validateSpan. The
name "span" previously meant the trace root in one inbound-message handler and
the job-queue handle in its sibling, which taught a reader the opposite of the
truth in the next function.

Comments: the StatsD gauge rationale now sits with the initialiser it explains
rather than in the constructor. The peer span header described its trust flags
as shared when they are in fact re-declared to match the consensus keys; the
duplication is intentional and the wording was not.

Docs: the ledger and peer span tables disagreed with the code, crediting
ledger.build with attributes that are set on tx.apply and omitting several that
it does set, and all five source-file line numbers in them were stale. The
testing guide listed attribute keys that exist nowhere in the code, so its
catalog now points at the runbook instead of keeping a second copy that drifts.
2026-09-02 15:30:30 +01:00
Pratik Mankawde
0fcd690cf9 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-08-27 12:59:13 +01:00
Pratik Mankawde
95fb21b5d5 docs(telemetry): drop the inert insight prefix from the OTel examples
OTelCollector routes every instrument name through a static formatName()
that only lowercases the name and maps '.' and space to '_'. The sole read
of prefix_ is the startup log line at OTelCollector.cpp:802, and all four
instrument factories go through formatName(), so no prefix can ever reach
an exported name. StatsDCollector does prepend it, so the StatsD example
keeps the key and now states why.

Covers the three server=otel blocks in the 09 reference and the config
integration-test.sh generates. This branch introduces OTelCollector, so it
is where the inert examples first appear; phase-6's examples are all
server=statsd and stay as they are.
2026-08-27 12:58:45 +01:00
Pratik Mankawde
4f056496f8 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-08-27 12:32:54 +01:00
Pratik Mankawde
cb92d59f11 fix(telemetry): drop the inert insight prefix from the OTel path
formatName() never reads prefix, so setting it here does nothing and the
exported names are bare and lowercase. Leaving it invites queries written
against xrpld_jobq_job_count, which match no series.

The StatsD examples keep it, because that path does apply it to the name.
2026-08-27 12:12:31 +01:00
Pratik Mankawde
520e8e5895 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-08-25 15:07:36 +01:00
Pratik Mankawde
cc5e098928 fix(telemetry): rate the overlay traffic bargauge instead of ranking totals
The panel described itself as showing which categories consume the most
receive bandwidth, and told operators to watch for a category dominating
during ongoing sync, but it ranked raw cumulative counters. That ranks by
lifetime total, so a sync burst days earlier still dominates the bars and
the stated guidance cannot work.

Wrapping the existing selector in rate() does not work: rate() drops
__name__, so the many *_bytes_in series collapse to one labelset and the
query fails with "vector cannot contain metrics with the same labelset".
Grouping with sum by (__name__) does not help either, because Prometheus
strips __name__ from aggregation output regardless of the grouping. Both
were tried against a local Prometheus with synthetic multi-category
counters and both error.

Instead recover the category into a normal `series` label before rating,
and use a subquery so the renamed series can be rated. Verified against
the same synthetic data: rates match a plain single-metric rate control
to within subquery resolution, ranking order is correct, and both the
category and node labels survive.

Also set unit to Bps, matching the ten sibling rate panels in this
dashboard (this was the only byte panel still on decbytes and the only
one plotting raw), and drop "Heatmap" from the title since the panel is
a bargauge.

Note the subquery evaluates the selector at several steps, so it costs
more than the previous instant query across the traffic categories.
2026-08-24 21:10:28 +01:00
Pratik Mankawde
8169894ada Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-08-21 12:30:50 +01:00
Pratik Mankawde
24094e427b fix(telemetry): give each histogram unit its own bucket ladder
This is the change that actually lifts the 5 s ceiling. Until now the
millisecond ladder and the Unit type existed but nothing consumed them.

Telemetry.cpp registered ONE histogram view: instrument name pattern "*",
unit exactly "ms", boundaries {1, 5, ..., 1000, 5000}. Verified against the
installed SDK, "*" matches every name and "ms" matches exactly, so that view
governed every beast::insight Event -- all 54 of them, whatever they measure.
Measured on devnet: 24.9% of rpc_size samples and 100% of jobq_updatepaths
samples fell above 5000. A quantile landing in the `+Inf` bucket reads back
as the second-highest edge, so those p95s reported a flat 5000 rather than a
measurement, and the 1 s to 5 s span was a single four-second-wide bucket
that any quantile inside it had to interpolate across.

Replaces it with one view per unit, keyed on the unit an instrument declares:

- `ms` gets kMillisecondBuckets: every representable edge of the collector's
  spanmetrics ladder, plus 60 s and 120 s. The extensions are deliberate --
  jobq_updatepaths was measured averaging 59,956 ms, which no span
  approaches, so parity alone would still censor it.
- `By` gets kByteBuckets, placed from the measured response distribution
  (mean 2131 B, half under 1 kB, tail mean bounded at 7538 B).

OTelEventImpl now derives its declared unit AND its description from unit()
instead of hardcoding "Duration in ms"/"ms", so rpc_size exports as
rpc_size_bytes on the byte ladder. rpc-pathfinding's "RPC Response Size"
panel follows the rename; its unit was already decbytes and is now truthful.

Also corrects Phase7_taskList.md, which still specified the 5000 ladder as
"matching SpanMetrics". That was true when written and became false when the
collector ladder was extended on its own -- implementing the plan as written
reproduced the bug, so the spec is where the defect had come to live. The
edges now have exactly one owner and the plan points at it.
2026-08-21 12:30:38 +01:00
Pratik Mankawde
8fac7fc08c Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-08-20 16:38:03 +01:00
Pratik Mankawde
4278014ab0 fix(telemetry): order the metrics pipeline by instrument kind
beast::insight instruments are created during ApplicationImp's member-init
list, and opentelemetry-cpp 1.28 never rebinds an already-vended Meter, so an
instrument created before the MeterProvider is published records nothing for
the rest of the process. Observable instruments carry the opposite constraint:
registering one arms the SDK reader thread, and its callbacks run hook handlers
that read services which do not exist that early.

Publish the provider in Telemetry's constructor, ahead of every producer, and
defer only the observables. Collector gains onCollectionReady() and
onCollectionStopping(); OTelCollector arms and disarms its gauges in response.
StatsDCollector starts its polling thread in its own constructor and had the
same hazard, so it uses the pair to gate that thread.

The metrics resource carries service.instance.id and is immutable once built,
so the node public key is resolved in Main.cpp, where a config error can still
be reported, and passed to makeApplication(). getNodeIdentity() remains
authoritative; both paths now share readNodeIdentity(), so telemetry cannot
report a key the node has abandoned.

An explicit ~ApplicationImp stops observing and stops telemetry, covering the
setup() failure paths that never reach run(). Telemetry::stop() is once-only
and no longer clears another instance's global pointer. The histogram view's
meter selector now matches the meter actually in use, so its bucket boundaries
apply for the first time.
2026-08-20 16:36:41 +01:00
Pratik Mankawde
bbc57202f6 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-08-17 19:36:57 +01:00
Pratik Mankawde
ac71480a62 fix(telemetry): count mode transitions with tiling buckets, not overlapping ones
The transitions panel used increase(...[$__rate_interval]). $__rate_interval is
defined as max($__interval + scrape, 4 * scrape), i.e. deliberately one scrape
longer than the step so rate() windows overlap and lose no counter increase.
That overlap is harmless for rate(), but this panel reads the value as a count
of discrete events, and the overlap counts each event in more than one bucket.

Measured against a log-derived ground truth of 106 syncing transitions on
devnet-otel-usw2-01 over 2026-08-11T11:05Z..2026-08-12T23:04Z, the old query
reported 111.3 at a 300s step and 133.7 at a 60s step -- the error grew to +26%
as you zoomed in, because the overlap is a larger fraction of a smaller step.

Switch to $__interval so the buckets tile exactly, and wrap in round() because
increase() extrapolates to the window edges and so reports fractional counts for
an integer counter. The same measurement now gives 106 at 300s, 105 at 60s and
107 at 900s. Every state and both nodes land within a few counts of truth at any
zoom, and the legend Total is now a meaningful figure.

Pin Min step to 1m: the real scrape interval is 60s while the datasource
declares 15s, so without a floor $__interval can fall below one sample.

Draw as bars with 0 decimals -- the value is a discrete count per bucket, and a
line implies interpolation between counts that does not exist.
2026-08-17 19:36:41 +01:00
Pratik Mankawde
b19e429636 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-08-17 18:02:59 +01:00
Pratik Mankawde
c8f9ab9597 fix(telemetry): plot operating-mode transitions per interval, not raw counters
The Operating Mode Transitions panel queried state_accounting_*_transitions
directly. Those are monotonic counters, so the panel drew a slowly rising line
and a few transitions per hour were invisible against a total in the hundreds.
It also fell off a cliff whenever xrpld restarted and the counters reset to 0,
which reads as missing data rather than a restart.

Wrap each target in increase(...[$__rate_interval]) so each point is the number
of transitions in that bucket and the series survives a counter reset. This is
what the sibling panels on the same row (Operating Mode (Time Share), State
Duration Rate) already do.

Verified against devnet-otel-usw2-01/02 over 2026-08-11T11:01Z..2026-08-12T16:23Z:
the fixed expression reports 107 and 123 syncing transitions, matching the
counter deltas, and stays continuous across the 12:07 restart where the raw
counter dropped 630 -> 1.

Brief mode flaps remain invisible on Operating Mode (State Timeline) because a
~2 s dwell cannot be captured by a 60 s scrape; this panel is the place to read
them.
2026-08-17 18:02:13 +01:00
Pratik Mankawde
f3681ad920 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation
Carries phase-6's integration-test and telemetry-doc fixes forward. Merged
cleanly with no conflicts.
2026-08-17 12:06:12 +01:00
Pratik Mankawde
b77f516f95 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
Conflict resolution kept this branch's evolution and re-applied phase-6's
fixes on top of it, rather than taking either side wholesale:

- consensus-health.json: kept the native `span_calls_total` metric name and
  the `interval: 15s` and point styling from this branch; added phase-6's
  `close_time_correct` PromQL filter and the NetClock axis labels. The
  TraceQL boolean-regex filter stays removed and the `byRegexp` overrides
  carry over. Panel count unchanged at 27.
- 09-data-collection-reference.md: kept this branch's headings, its more
  detailed consensus attribute table (which already types
  `consensus_round_id` as int64) and its section numbering, including the
  deliberate removal of the SpanNames inventory. Carried over only the
  correction that the state-accounting duration gauges are cumulative
  microseconds, not seconds.
- telemetry-runbook.md: kept this branch's native metric names
  (`span_calls_total`, `span_duration_milliseconds_bucket`); carried the
  `rpc.request` -> `rpc.http_request` span-name fix and the `jobq_` segment
  on the job-queue depth metric.
- integration-test.sh: kept this branch's `check_otel_metric` form and
  carried the `jobq_job_count` correction.
2026-08-17 12:05:10 +01:00
Pratik Mankawde
3a3ae1c3ae fix(telemetry): name the job-queue depth metric jobq_job_count
The integration test asserted `rippled_job_count`, which never reports any
series, so that check always failed. `JobQueue` registers the gauge as
`makeGauge("job_count")`, but `Application.cpp` passes it
`collectorManager_->group("jobq")`, so the emitted StatsD name is
`jobq.job_count` and the exported Prometheus name is
`<prefix>_jobq_job_count`.

Corrected the same name in two runbook tables that also dropped the `jobq`
segment. `09-data-collection-reference.md` already had it right, which is why
the two documents disagreed.

Routed here rather than to the phase-10 PR where it was reported: the wrong
name is present in `integration-test.sh` on every branch from phase 6
onward, and this is the branch that introduces the file.

Left alone deliberately:
- `statsd-node-health.json` still queries the old name, but that dashboard is
  deleted at phase 7 in favour of `node-health.json`
- `06-implementation-phases.md` names `job_count`, which is accurate as the
  code-level makeGauge argument rather than the exported metric name
2026-08-17 12:02:46 +01:00
Pratik Mankawde
301149d131 fix(telemetry): keep filelog offset storage out of the shared collector config
The file_storage extension was added to otel-collector-config.yaml, which
every stack mounts. That made the extension mandatory: the collector image
runs as 10001:10001 and ships no writable directory, so any stack without a
prepared volume would fail to start rather than merely lose offsets. The
workload-validation stack mounts this same config and has no such volume.

Offset persistence is only useful where logs outlive a restart. The workload
harness creates a fresh log directory per run, so it has nothing to resume
from. Move the extension, the receiver's storage reference and the extended
service.extensions list into otel-collector-filestorage.yaml, layered as a
second --config by the developer stack alone. The base config keeps
start_at: beginning, which is what actually fixes the reported defect, and
stays self-sufficient for every other stack.

Verified against the pinned collector image: the base config validates and
runs on its own with no volume mounted and still ingests a line written
before startup; base plus overlay validates, preserves the base receiver's
operators through the merge, and re-ingests that line zero times on a second
run against the same volume.
2026-08-17 11:44:30 +01:00
Pratik Mankawde
a09416eddc Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-08-15 17:49:18 +01:00
Pratik Mankawde
a61c349712 fix(telemetry): switch sample config to server=otel
This branch removes the collector's StatsD receiver and un-publishes
8125/udp, but xrpld-telemetry.cfg still selected server=statsd, so the
sample config sent beast::insight metrics over UDP to a port nothing
listens on. Phase7_taskList.md:132 lists this switch as required work.

Select server=otel and replace address= with the OTLP metrics endpoint.
Document that endpoint and prefix are informational only, since
OTelCollector records on the global MeterProvider that [telemetry]
configures and formatName() applies no prefix, and note that beast
instruments are not exported yet because the collector is constructed
before the MeterProvider is registered.
2026-08-15 17:48:44 +01:00
Pratik Mankawde
d4282cc36e fix(telemetry): correct log-correlation defects found in PR review
Six findings from the review of #6494 survived independent verification.
Each was checked against the branch tip, and where behaviour was in
question, against a live collector and Loki rather than from the
reviewer's claim or from documentation alone.

Plan-doc section numbering. 06-implementation-phases.md used "## 6.9"
twice: for the new Phase 8 section and for the pre-existing Risk
Assessment. Three references already pointed at 6.8.1 and none at 6.9,
and the later phases are numbered 6.8.2 through 6.8.4, so Phase 8
becomes 6.8.1 and the sequence is monotonic. Renumbering to 6.10, as
suggested on the PR, would have collided with Success Metrics.

filelog read position. The receiver relied on the upstream default
start_at=end, which skips everything a node wrote before the first poll
and reads nothing at all from a log that has stopped being written to.
Read from the beginning instead, paired with a file_storage extension so
a restart resumes at the last offset rather than re-ingesting the file.
The collector image runs as 10001:10001 and ships no writable directory,
and a fresh named volume is root-owned, so a one-shot init service
prepares the volume first. It reuses an image the stack already pulls,
adding no new dependency.

Loki log stream label. The job resource attribute did not become a Loki
index label, so the documented {job="xrpld"} queries matched nothing.
Verified against grafana/loki:3.4.2 with its default config: only
service_name and deployment_environment are indexed, and job arrives as
structured metadata, which a stream selector cannot match. Dropped the
attribute and moved the twelve queries this branch introduced to
{service_name="xrpld"}. Three further occurrences in
07-observability-backends.md originate on the phase-1a branch and are
left for a commit there.

Trace ids on unsampled spans. Logs::format emitted trace_id and span_id
whenever the span context was valid. A span dropped by the
ParentBasedSampler still carries its parent's ids, so log lines
advertised traces that were never exported and the log-to-trace link
resolved to nothing. Require the sampled flag as well, and correct the
task list and the documentation that promised the fields unconditionally.

The remaining two findings were refuted. The reported risk of signing
material reaching Loki does not hold: Logs::format already scrubs seven
sensitive fields, and there is a single write path to the log file, so
every JLOG site is covered. The suggestion to add internalLink to the
Loki derived field is not applicable, because that key is not part of
Grafana's schema.
2026-08-15 17:32:57 +01:00
Pratik Mankawde
2a1860951a fix(telemetry): repair the five silently-empty TraceQL panels on consensus-health
All five TraceQL panels on this dashboard returned nothing, and did so
without any visible error: they filtered on
span.close_time_correct=~"$close_time_correct", but close_time_correct is
a boolean attribute (RCLConsensus.cpp:601 passes a raw bool), and Tempo
restricts the regex operator to string operands, so the spanset resolved
to false. With the variable defaulting to All the clause rendered as
=~".*", so the panels were empty out of the box and looked exactly like a
node with no consensus activity.

Note this is the opposite of PromQL, where an absent or empty label does
match ".*" — which is why the 17 Prometheus panels on this same board were
unaffected and the dashboard appeared healthy.

Dropped the clause from all six queries, matching phases 9 and 10 where it
is already gone. The $close_time_correct variable now filters the
Prometheus "Close Time Agreement" panel instead, which already grouped by
that label but never filtered on it, so the control stays useful rather
than becoming dead UI.

Two defects were masked behind the empty panels and are fixed too:

- "Close Time: Raw Proposals" and "Close Time: Effective / Quantized"
  carried unit dateTimeFromNow over close_time_self/close_time, which are
  NetClock seconds (Ripple epoch), while Grafana's dateTime formatters
  expect a millisecond Unix epoch — every point would have rendered as
  roughly 1970. They now plot as plain numbers with the axis labelled
  "NetClock Seconds (Ripple Epoch)", and the descriptions give the
  946684800 offset for converting to Unix time.
- "Close Time Vote Bins & Resolution" matched its unit and axis overrides
  byName against "Vote Bins" and "Resolution", which are not field names;
  TraceQL select() yields close_time_vote_bins and close_resolution_ms, so
  neither override applied. Switched to byRegexp so the match holds
  whichever scope prefix Grafana emits.

No panel was added or removed: the (type, title) multiset is unchanged at
22. resolution_direction keeps its regex filter, which is correct there —
it is set from a std::string whose values are exactly the variable's
increased/decreased/unchanged.
2026-08-15 17:30:40 +01:00
Pratik Mankawde
18a40f86fb fix(telemetry): correct integration-test span checks and telemetry docs
The integration test's span assertions never actually ran. check_span()
built a Tempo /api/search call with --data-urlencode but no -G, so curl
POSTed the params as a body; Tempo answers 200 and ignores the query, so
every span name looked present. Verified against a live Tempo 2.9.4: the
buggy form returns the store's total trace count for any name, including
"zzz.does.not.exist"; with -G a real name returns 1 and a bogus one 0.

Fixed alongside it: the RPC check asserted "rpc.request", which is never
emitted (ServerHandler.cpp builds "rpc.http_request"). These two had to
change together, since -G turns the bogus name from a silent pass into a
hard failure.

Also in the script: a consensus timeout logged two failures and counted
two, because a post-loop else re-reported what the timeout branch had
already reported; and three unguarded curl calls aborted the whole script
under set -euo pipefail, making the ACCOUNT_ZERO fallback dead code with
no cleanup. Guarded the curls and wired an EXIT trap to the existing
cleanup(). The trap deliberately fires only before the summary, so a
completed run still leaves the stack up as the header documents.

Docs corrections, all re-derived from code:
- span inventory heading 35 -> 38, attribute heading 83 -> 89 rows
  (78 unique keys), and the section 6 header table now carries the
  missing TxApplySpanNames.h row so its columns sum to the same figures
- two stale paths: ConsensusSpanNames.h is under include/xrpl/consensus/,
  TxSpanNames.h under src/xrpld/telemetry/
- consensus_round_id is int64, not string (RCLConsensus.cpp sets
  prevLgr.seq() + 1); the runbook's TraceQL examples now use a numeric
  literal instead of an unparseable bare <round_id>
- state-accounting duration gauges are cumulative MICROSECONDS, not
  seconds (NetworkOPs.cpp declares std::chrono::microseconds and
  publishes dur.count() raw)
- sampling_ratio is not a config key; head sampling is fixed at 1.0 and
  the shipped collector has no tail sampling, so the caveat was rewritten
- the plan blurb referenced Jaeger; this stack is Tempo
2026-08-15 17:24:14 +01:00
Pratik Mankawde
87fc9eaee6 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics 2026-08-14 22:59:12 +01:00
Pratik Mankawde
efd51f50d4 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-08-14 22:59:12 +01:00
Pratik Mankawde
773a5cc0fb fix(telemetry): stop the dashboard lint passing when it checked nothing
Run with no arguments the script iterated an empty list, found no
violations and printed "OK: 0 dashboard(s) passed" with exit 0 -- a clean
bill of health for reading no files, indistinguishable from a real pass.

A bare run now defaults to every dashboard beside the script, and a run
that still ends up with nothing to check exits 2 rather than reporting
success. Passing paths explicitly behaves as before.
2026-08-14 22:58:58 +01:00
Pratik Mankawde
292aee922a Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-08-14 21:47:42 +01:00
Pratik Mankawde
004b8e92ab fix(telemetry): keep the work item in ledger-data-sync aggregations
Nine targets across four panels filtered on xrpl_work_item while leaving
it out of their sum by() grouping. PromQL keeps only the labels listed in
by(), so the label was dropped from the result and the xrpl_ident legend
these panels build from it lost that segment.

Perf-iac runs stamp xrpl_work_item and give each work item its own set of
nodes, so service_instance_id already separates the runs; the visible
effect is the legend rather than merged series. Adding the label makes the
work item identifiable, which is the comparison these panels exist for.

NuDB Read Latency and NuDB Read Found Ratio each divide two aggregations.
Both sides get the label so their label sets stay equal and vector
matching still works.
2026-08-14 21:47:20 +01:00
Pratik Mankawde
46ed205790 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-08-14 21:21:28 +01:00
Pratik Mankawde
be8f987b39 docs(telemetry): drop the phase number from a node-health row title
The row was titled after the rollout phase its metrics came from, a number
defined only in a planning folder outside the shipped tree. Grafana shows
this title to operators, so it named something no reader could look up.
2026-08-14 21:21:25 +01:00
Pratik Mankawde
3a91a7414e Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
# Conflicts:
#	docker/telemetry/grafana/dashboards/statsd-rpc-pathfinding.json
#	docker/telemetry/integration-test.sh
2026-08-14 21:20:39 +01:00
Pratik Mankawde
974835589a docs(telemetry): drop the plan task reference from a StatsD panel note
Two panel descriptions pointed at a rollout phase and task number defined
only in a planning folder outside the shipped tree. The note still names
the file and the change it is waiting on, which is the part a reader can
act on.
2026-08-14 21:19:49 +01:00
Pratik Mankawde
8670e31d6c docs(telemetry): drop rollout phase numbers from the span catalog
The catalog tagged every span with a rollout phase number defined only in
a planning folder outside the shipped tree, so the column meant nothing
to a reader of the repository. The span name and source file identify
each entry.
2026-08-14 21:17:02 +01:00
Pratik Mankawde
68b94e625c docs(telemetry): name log-trace correlation instead of its phase number
These comments and headings identified the feature by a rollout phase
number defined only in a planning folder outside the shipped tree, so the
label meant nothing to a reader of the repository.
2026-08-14 21:13:59 +01:00
Pratik Mankawde
2bac88e67c test(telemetry): label integration-test sections by what they verify
The section labels carried rollout phase numbers that are defined only in
a planning folder outside the shipped tree, so they meant nothing to a
reader of the repository. The descriptive half of each label already
identifies the section.
2026-08-14 21:11:54 +01:00
Pratik Mankawde
c4888dd066 test(telemetry): label the native-metrics section by what it verifies
The label carried a rollout phase number defined only in a planning
folder outside the shipped tree. The descriptive half already identifies
the section.
2026-08-14 21:11:14 +01:00
Pratik Mankawde
e9ce861add Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-08-07 12:54:35 +01:00
Pratik Mankawde
97089a3b6f fix(telemetry): stop repeating panels stranding their neighbours
A repeating panel expands into one copy per network at view time and, with
maxPerRow=2, claims the whole row. The non-repeating panel paired beside it was
pushed to the next line but kept its stored x=12, so it rendered on the right
against an empty gap.

Two changes to how the layout is planned:
  - a repeating panel gets a row to itself. It keeps w=12, so its copies still
    tile two across inside that row.
  - single-value panels are grouped to the top of each row section, so the
    charts that follow pair with each other instead of being split up by an
    interleaved repeat. Without this the gaps just become wasted half-rows.

Verified per dashboard: no panel lost, every targets block byte-identical, ids
1..N, no overlaps, and no row left with a gap on its left.
2026-08-07 12:54:25 +01:00
Pratik Mankawde
76d440aa40 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-08-07 12:38:46 +01:00