Commit Graph

1132 Commits

Author SHA1 Message Date
Pratik Mankawde
1748178cfa Merge branch 'pratik/otel-sync-diagnostics' into pratik/otel-sync-diagnostics-freshen-fix 2026-09-23 18:51:01 +01:00
Pratik Mankawde
dc696d260e Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics 2026-09-23 18:10:45 +01:00
Pratik Mankawde
67ec4071a2 docs(telemetry): Note the per-role account attributes on tx.process in the harness 2026-09-23 18:10:16 +01:00
Pratik Mankawde
64760545eb Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-23 18:09:26 +01:00
Pratik Mankawde
18181f3c31 docs(telemetry): Drop the remaining account-hashing claims 2026-09-23 18:08:58 +01:00
Pratik Mankawde
2969965d8c Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill
Conflicts, all resolved by keeping phase-9's rewritten text and applying
the upstream change to it:
- PathRequest.cpp: drop the Redaction.h include, keep MetricMacros.h.
- XrplCore.cmake: keep the PUBLIC-link comment, note the SField table use.
- OpenTelemetryPlan.md, 02-design-decisions.md, 05-configuration-reference.md,
  telemetry-runbook.md: account addresses are emitted raw; no hashing at
  any layer; tx.process lists the per-role account attributes.
2026-09-23 18:08:07 +01:00
Pratik Mankawde
1381b8f684 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-23 18:06:19 +01:00
Pratik Mankawde
331a26cb4c Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
Conflicts in src/libxrpl/telemetry/TelemetryConfig.cpp and its test:
phase-5's "require an https endpoint whenever use_tls=1" met phase-7's
version of the same guard, which already covers both the traces and the
metrics endpoint. Kept phase-7's side.
2026-09-23 18:05:31 +01:00
Pratik Mankawde
bf0670991a Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd
Conflicts: docker/telemetry/otel-collector-config.yaml (kept phase-6's
config, removed the attributes/hash processor as phase-2 did) and
docs/telemetry-runbook.md (kept phase-6's span table, applied phase-5's
tx.process attribute additions to its row).
2026-09-23 18:04:35 +01:00
Pratik Mankawde
52a7d52acd Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing
Conflict in cmake/XrplCore.cmake: kept phase-4's comment block and applied
phase-3's two-line note about the SField table.
2026-09-23 18:02:45 +01:00
Pratik Mankawde
3761dee234 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing 2026-09-23 18:01:54 +01:00
Pratik Mankawde
91b440820a fix(telemetry): Emit path-finding accounts as raw r-addresses
An XRP account address is a public ledger identifier drawn from an
enumerable set. An unsalted hash of it is reversible by lookup, so it
protected nothing and only broke the join against explorers, RPC
responses and logs that show the same address.

- pathfind_source_account and pathfind_dest_account carry the request's
  r-address. A value that does not parse as an r-address is not emitted,
  so a malformed or mistaken request value never reaches a span. Both
  handlers share setAccountAttribute() in PathFindSpanAttributes.h.
- pathfind_dest_currency is to_string(Asset): "XRP", "<issuer>/<CUR>",
  or the MPT issuance id.
- The collector's attributes/hash processor is removed. No layer hashes.
- redactAccount() stays available; its header no longer claims to sit in
  the emit path.
2026-09-23 18:00:32 +01:00
Pratik Mankawde
fc98dac0f1 fix(telemetry): name each rule's query steps, and let email keep its own layout
The alert notification ended with `Values: A=0 B=0 C=1`, the internal ids of the
three query steps, which mean nothing to a reader. The previous attempt removed
that line by replacing the whole email body with a custom template. That worked,
but it threw away everything else Grafana's default email gives: bold section
headings, a styled label table, annotations rendered as clickable links, and the
View and Silence buttons. The result read as a flat block of text and raw URLs.

Fix the line instead of the body. Rename all 13 rules' steps from A/B/C to names
that say what they hold -- close_rate_5m, close_rate_now, is_stalled -- so the
default body's value line reads close_rate_5m=0 close_rate_now=0 is_stalled=1.
The threshold step is named is_* so a 0/1 verdict is obviously a verdict and not
a rate.

Drop the email subject/message override so the default layout returns, and drop
the now-unused xrpld.email.body template. Slack keeps its own body: Grafana
escapes HTML in an email message, so a custom email body can only ever be plain
text, while Slack needs mrkdwn and <url|label> links. One body cannot serve both,
and only Slack benefits from having one at all.

Verified by provisioning the real files into Grafana: all 13 rules load with the
new step names and evaluate health=ok, and the Slack template renders with no
parse error.
2026-09-23 17:40:10 +01:00
Pratik Mankawde
bca04bc2c3 fix(telemetry): give email its own notification body, not the Slack one
Sharing one body between the Slack and email receivers made the email
unreadable. Email cannot render Slack markup, so the bold asterisks, the
backticks and the `🚨` shortcodes all arrived as literal
characters, and a `<url|label>` link could not become a link at all -- it dumped
the whole dashboard URL inline. Four panel links then buried the prose.

Split the body in two. The Slack body keeps mrkdwn. The email body is plain
text, one fact per line, with each dashboard link on its own labelled line.
Both still share the title and the node-identity fallback.

Measured while fixing this, and now recorded in templates.yaml: email escapes
any HTML in the message, so `<br>` arrives as `&lt;br&gt;` and no tag or anchor
is possible; but a newline in the template does become a real `<br>`, so line
breaks are the only layout tool email has.

Also correct two comments that were wrong. A missing template define does not
ship raw template text: it logs one warning and silently delivers Grafana's
default body, the value dump this file exists to remove, while the rule still
reports health=ok. And the Cloud contact point is not email-only; it holds a
Slack receiver and an email receiver, on an instance shared with other teams.
2026-09-23 15:19:51 +01:00
Pratik Mankawde
f23b66ca4d Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics
expected_spans.json: composed both sides. Kept this branch's 16 extra spans,
relationships and the per_ledger join group; took phase-10's allowed_parents
gate for the shared entries and derived allowed_parents for the 16 from their
factories. ledger.store, ledger.validate and ledger.acquire are hashSpan roots
on this branch, so they list ROOT alone rather than consensus.accept.apply.
2026-09-23 15:18:25 +01:00
Pratik Mankawde
fd29e92a0e Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-23 15:15:28 +01:00
Pratik Mankawde
628882f686 fix(telemetry): replace the alert notification body with the rule's own prose
Grafana's default notification body appends every expression node's value and
every label, so an alert arrived as `Value: A=0, B=0, C=1` over a five-line
`key = value` dump. The refIds mean nothing to a reader and the labels repeat
the title.

Add templates.yaml and point both Slack receivers and the email receiver at it,
so the body is the rule's own description plus its remediation line, and any
panel_* annotation renders as a link.

Rewrite all 13 descriptions to be status-neutral, since the same annotation is
rendered when the alert resolves: a firing-only wording made a resolved
notification claim the node had stopped closing ledgers while reporting a
healthy rate. The reason to care moves to a new `action` annotation, which the
template prints only while firing. Each description now formats its value with
printf and states its threshold, rather than emitting a bare float.

Also fix the Slack title, which referenced a `rulename` label that no rule sets
and so rendered blank; `alertname` is the label Grafana always provides.
2026-09-23 14:32:16 +01:00
Pratik Mankawde
2bfd1af6a0 fix(telemetry): accept.apply has one lawful parent after the fallback arm went
The accept.apply factory no longer falls back to the round context, so the
harness contract lists consensus.accept alone. Note rewritten to describe the
scoped child as it is now.
2026-09-23 13:58:45 +01:00
Pratik Mankawde
d93749f752 feat(telemetry): assert span parents and consensus-round shape in the harness
The contract's parent field was documentation only, so no check could fail a
span for being parented when it should be a root. A live sweep found three
spans nested one level off while every harness check passed. The field becomes
allowed_parents, a list derived per span from its creation factory and every
call path that reaches it, and is asserted per span.

A parent on another node and a parent absent from the trace are both
inconclusive rather than violations: the receive spans are parented cross-node
by design, and a rotation in flight has not exported its root yet. Spans
reached by two call paths list every lawful parent.

A second gate reads a round as a unit: every required phase child under the
round span on the same node, their start times in protocol order, and no
mode_change recording mode_old == mode_new. It selects traces that already hold
the accept span, since that span always ends after the round span and can land
in a later export batch.

num_cleared leaves txq.batch_clear's required attributes: the code writes it
only after the batch actually clears.
2026-09-23 13:50:13 +01:00
Pratik Mankawde
1d49c64a9d Merge branch 'pratik/otel-sync-diagnostics' into pratik/otel-sync-diagnostics-freshen-fix 2026-09-23 12:03:20 +01:00
Pratik Mankawde
992e600929 Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics 2026-09-23 12:03:08 +01:00
Pratik Mankawde
e06fdd266e Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-23 12:03:08 +01:00
Pratik Mankawde
407f5f1224 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-09-23 12:03:08 +01:00
Pratik Mankawde
5e54b0f36e Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-23 12:03:08 +01:00
Pratik Mankawde
35da2952b0 docs(telemetry): drop the example ticket id from the work-item filter
The $xrpl_work_item variable's description carried a real ticket id as its
example. The filter needs no example, so the id is gone and the wording stays.
2026-09-23 12:00:51 +01:00
Pratik Mankawde
964b0d64bf docs(telemetry): drop the example ticket id from the work-item filter
The $xrpl_work_item variable's description carried a real ticket id as its
example. The filter needs no example, so the id is gone and the wording stays.
2026-09-23 12:00:37 +01:00
Pratik Mankawde
bcc44dedf1 docs(telemetry): drop the example ticket id from the work-item filter
The $xrpl_work_item variable's description carried a real ticket id as its
example. The filter needs no example, so the id is gone and the wording stays.
2026-09-23 12:00:29 +01:00
Pratik Mankawde
f7e65949b8 Merge branch 'pratik/otel-sync-diagnostics' into pratik/otel-sync-diagnostics-freshen-fix 2026-09-22 21:31:17 +01:00
Pratik Mankawde
c8a4634cb4 Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics
expected_spans.json kept both sides' relationship entries: this branch's
eight nodestore.rotate phase children and the upstream branch's three
cross-node pairs. Both sides appended at the same list position.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:31:09 +01:00
Pratik Mankawde
7272ea078c test(telemetry): Assert the cross-node span relationships
The harness checked parentage only within a node. Every declared
relationship named two spans from one process, so a receive span that
lost its cross-node parent still satisfied the contract: it is created
either way, with all its normal attributes, and only its parent differs.

Declare the three pairs that cross a node boundary, so a send side that
stops propagating context fails the run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:30:23 +01:00
Pratik Mankawde
ec9af37796 Merge branch 'pratik/otel-sync-diagnostics' into pratik/otel-sync-diagnostics-freshen-fix 2026-09-22 21:22:37 +01:00
Pratik Mankawde
881faf5620 Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics
The runbook's nodestore_state table kept AppMetricGauges.cpp as the source
column, which is where those gauges are actually registered on every branch
in the chain; the incoming side named MetricsRegistry.cpp, which registers
none of them. Took the incoming side's clearer wording for the sweep row.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:19:08 +01:00
Pratik Mankawde
fcdc4f9f66 Merge branch 'pratik/otel-phase7-native-metrics' into pratik/otel-phase8-log-correlation 2026-09-22 21:17:58 +01:00
Pratik Mankawde
aeac95b1d7 Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill 2026-09-22 21:17:58 +01:00
Pratik Mankawde
c381d05bb8 Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-22 21:17:58 +01:00
Pratik Mankawde
e48b0286c2 Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics
StatsDCollector test kept both sides: this branch's onCollectionReady() call,
which enables polling and only exists from here on, followed by the upstream
branch's control assertion that reads the resulting datagram.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:17:48 +01:00
Pratik Mankawde
e9d9cffe3b Merge branch 'pratik/otel-phase5-docs-deployment' into pratik/otel-phase6-statsd
Both doc indexes kept this branch's 09-data-collection-reference.md rows,
which only exist here, and dropped every secure-OTel.md reference because
the upstream branch removed that file. No dangling link remains.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:16:44 +01:00
Pratik Mankawde
aea4f56505 Merge branch 'pratik/otel-phase3-tx-tracing' into pratik/otel-phase4-consensus-tracing
SpanGuardScope.cpp kept both includes: each side added one and both symbols
are used in the merged test.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:15:05 +01:00
Pratik Mankawde
bba92767a6 Merge branch 'pratik/otel-phase2-rpc-tracing' into pratik/otel-phase3-tx-tracing
tempo.yaml kept both sides' filter blocks: phase-2's six path-finding
filters ahead of this branch's three transaction filters, matching chain
order, and both header comment lines.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-22 21:14:20 +01:00
Pratik Mankawde
3cfe8d139e feat(telemetry): Add Tempo search filters for the path-finding attributes
This branch emits nine pathfind attributes and the datasource offered a dropdown
for none of them, so the signal was there but not searchable in Explore. The
file's own header states that each phase adds filters for what it introduces.

Six filters, for the attributes that select a request. The three counts are
measurements read off a span rather than things an operator searches by, so they
get none. ledger_index is dynamic because it takes a new value every ledger.
2026-09-22 20:31:14 +01:00
Pratik Mankawde
68a805563f fix(telemetry): Reject a JSON reply or a weights argument that is not an object
Both workload scripts decoded JSON and went straight to .get() or .items().
An array or a scalar decodes fine and then raises AttributeError, which is not a
ValueError, so the handler around the weights parsing could not catch it and the
operator saw a traceback naming neither the command nor what arrived.

Four sites across the two files, all four guarded, and the weights handler now
also catches TypeError for a non-numeric weight value.

Negative weights are left alone: they do not raise, the type is dropped from the
mix silently, and changing that changes accepted input.
2026-09-22 20:30:53 +01:00
Pratik Mankawde
331a453c9d fix(telemetry): Read RPC success from the absence of an error, not from Ok
This board's own copy of the Success series selected status_code=STATUS_CODE_OK.
Successful rpc.command spans no longer carry Ok: the specification reserves it
for an operator asserting verified success and warns a tool may read it as
suppressing errors, so instrumentation leaves a successful span Unset.

The blob differs from phase-9's, so merging forward would not have corrected it.
2026-09-22 20:30:51 +01:00
Pratik Mankawde
c3673f51cd fix(telemetry): Read RPC success from the absence of an error, not from Ok
The Success series selected status_code="STATUS_CODE_OK". Successful
rpc.command spans no longer carry Ok, because the specification reserves it for
an operator asserting verified success and warns a tool may read it as
suppressing errors, so instrumentation leaves a successful span Unset.

Selecting on the absence of an error keeps the panel correct either way, rather
than pinning it to whichever status a successful span happens to have.
2026-09-22 20:15:18 +01:00
Pratik Mankawde
35c7f9be4c docs(telemetry): use a placeholder zone in the cloud env example
The example endpoint named a concrete zone, which reads as the stack this repo
pushes to. The two sibling files already write <zone>, so match them.
2026-09-22 19:09:16 +01:00
Pratik Mankawde
2d3eb7a981 fix(telemetry): Stop unl_expiry_days wrapping, and count only unfinished sweep evictions
unl_expiry_days subtracted two NetClock time points, whose rep is uint32_t, so
the subtraction wrapped before the duration_cast ran. A list expired by one day
read about +49709 days. The panel is green above 30 while its own description
promises red at expiry, so an expired validator list rendered healthy.

daysUntil() widens both endpoints to int64_t first, which makes the wrap
impossible rather than checked for. It deliberately does not clamp at zero: a
negative reading is the signal that expiry has passed. A config-listed list,
which uses time_point::max(), now reports positive infinity, because any finite
sentinel could not be told apart from the wrap this removes. -1 keeps its
existing meaning of no published list fetched.

The sweep counter told a second story it could not support. It counted every
entry the 1-minute sweep evicted, including acquisitions that had already
completed or failed and were merely still in the map. Those were counted when
they ended, so the metric buried the wasteful case in ordinary cleanup while
the runbook, the reference doc and the panel description all described only the
unfinished population. It now counts what those three already claimed.

isComplete()/isFailed() are used rather than isDone(), which is protected on
TimeoutCounter and not callable here.
2026-09-22 19:07:41 +01:00
Pratik Mankawde
cbed889f4b fix(telemetry): Accept every legitimate unl_expiry_days reading in the validator
The sanity bound required the value to be strictly positive, which rejected
three of the four readings the gauge can legitimately produce: a negative
count once the validator list has expired, the -1 sentinel for "no published
list fetched", and +inf for a config-listed list that never expires.

The negative case is the one that matters. It is the signal that expiry has
already passed, so the gauge deliberately does not clamp at zero, and a gate
that rejects it would fail exactly when an operator most needs the reading.
The -1 sentinel already violated the bound and had simply never been hit,
because the validation cluster always fetches a published list.

The floor is now a century, which still catches a broken clock. Detecting the
unsigned wrap this bound used to hide moves to the MetricsRegistry::daysUntil
unit tests, which are deterministic and do not need a running cluster.
2026-09-22 19:04:48 +01:00
Pratik Mankawde
f81ef31b0e Merge branch 'pratik/otel-sync-diagnostics' into pratik/otel-sync-diagnostics-freshen-fix 2026-09-22 15:43:12 +01:00
Pratik Mankawde
6b1ced940e Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics 2026-09-22 15:43:12 +01:00
Pratik Mankawde
406e38275a Merge branch 'pratik/otel-phase9-metric-gap-fill' into pratik/otel-phase10-workload-validation 2026-09-22 15:42:55 +01:00
Pratik Mankawde
e53569e696 fix(telemetry): Split the job queue p99 bargauge into wait and exec
The panel fused queue-wait and execution p99 into a single bargauge, so
every job type drew two bars and neither measure could be read on its
own. It is now two panels side by side, one measure each, with its own
title and description.

The panel also ran an instant query over a fixed 5 minute rate window.
An instant query always evaluates at "now", so the dashboard time picker
had no effect and a network with no samples in the last few minutes
rendered empty at every range. The window is now $__range, which ties it
to the picker.

Repeat direction is vertical. Horizontal repeat with maxPerRow set makes
Grafana widen each copy to the full row, which would break the pair.
2026-09-22 15:42:21 +01:00