Conflict resolution kept this branch's evolution and re-applied phase-6's
fixes on top of it, rather than taking either side wholesale:
- consensus-health.json: kept the native `span_calls_total` metric name and
the `interval: 15s` and point styling from this branch; added phase-6's
`close_time_correct` PromQL filter and the NetClock axis labels. The
TraceQL boolean-regex filter stays removed and the `byRegexp` overrides
carry over. Panel count unchanged at 27.
- 09-data-collection-reference.md: kept this branch's headings, its more
detailed consensus attribute table (which already types
`consensus_round_id` as int64) and its section numbering, including the
deliberate removal of the SpanNames inventory. Carried over only the
correction that the state-accounting duration gauges are cumulative
microseconds, not seconds.
- telemetry-runbook.md: kept this branch's native metric names
(`span_calls_total`, `span_duration_milliseconds_bucket`); carried the
`rpc.request` -> `rpc.http_request` span-name fix and the `jobq_` segment
on the job-queue depth metric.
- integration-test.sh: kept this branch's `check_otel_metric` form and
carried the `jobq_job_count` correction.
09-data-collection-reference.md contradicted itself. Section 2 presents
server=otel as the recommended transport with StatsD as a fallback, but
the Known Issues table and both Configuration Quick Reference examples
still prescribed server=statsd, which on this branch reaches a collector
with no statsd receiver and an unpublished 8125/udp.
Switch the Known Issues row and the Minimal and Production examples to
server=otel with the OTLP metrics endpoint. Keep the labelled fallback
block, and state what it actually requires: re-adding the statsd
receiver and republishing the port. Also record that StatsDCollector
applies prefix to metric names while OTelCollector does not, so the two
transports do not produce the same series.
The phase-6 copy is left alone; server=statsd is correct there.
The integration test's span assertions never actually ran. check_span()
built a Tempo /api/search call with --data-urlencode but no -G, so curl
POSTed the params as a body; Tempo answers 200 and ignores the query, so
every span name looked present. Verified against a live Tempo 2.9.4: the
buggy form returns the store's total trace count for any name, including
"zzz.does.not.exist"; with -G a real name returns 1 and a bogus one 0.
Fixed alongside it: the RPC check asserted "rpc.request", which is never
emitted (ServerHandler.cpp builds "rpc.http_request"). These two had to
change together, since -G turns the bogus name from a silent pass into a
hard failure.
Also in the script: a consensus timeout logged two failures and counted
two, because a post-loop else re-reported what the timeout branch had
already reported; and three unguarded curl calls aborted the whole script
under set -euo pipefail, making the ACCOUNT_ZERO fallback dead code with
no cleanup. Guarded the curls and wired an EXIT trap to the existing
cleanup(). The trap deliberately fires only before the summary, so a
completed run still leaves the stack up as the header documents.
Docs corrections, all re-derived from code:
- span inventory heading 35 -> 38, attribute heading 83 -> 89 rows
(78 unique keys), and the section 6 header table now carries the
missing TxApplySpanNames.h row so its columns sum to the same figures
- two stale paths: ConsensusSpanNames.h is under include/xrpl/consensus/,
TxSpanNames.h under src/xrpld/telemetry/
- consensus_round_id is int64, not string (RCLConsensus.cpp sets
prevLgr.seq() + 1); the runbook's TraceQL examples now use a numeric
literal instead of an unparseable bare <round_id>
- state-accounting duration gauges are cumulative MICROSECONDS, not
seconds (NetworkOPs.cpp declares std::chrono::microseconds and
publishes dur.count() raw)
- sampling_ratio is not a config key; head sampling is fixed at 1.0 and
the shipped collector has no tail sampling, so the caveat was rewritten
- the plan blurb referenced Jaeger; this stack is Tempo
Resolved OpenTelemetryPlan/09-data-collection-reference.md: this branch
replaced the StatsD-shaped Section 2 with OTel-native naming, so the
incoming StatsD tables and the StatsD per-job timer section do not apply
here. Kept this branch's names throughout.
The incoming per-job-type saturation gauge section is new content and is
retained, renumbered to 2.5 and rewritten for the OTel naming: bare
lowercase `jobq_<jobtype>_{waiting,running,deferred}` derived through
OTelCollector rather than the case-preserving `xrpld_`-prefixed StatsD
form.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add reference entries for the observability surface introduced on
phase-9: the `handler` label on the job instruments, the five
`getobject_*` request metrics, and the per-job-type queue saturation
gauges.
Names here follow this branch's StatsD pipeline, which preserves case
and carries the `xrpld_` prefix, so they differ from the lowercased
OTel-native names used from phase-7 onward. The sections state where
the implementing code lives, since it is introduced downstream.
Also correct pre-existing entries: `job_count` exports as
`jobq_job_count` via the collector group prefix, the non-special job
type count is 35 (not 36), and `JtLedgerData` has five producers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Document the two new ledger-correlation attributes in the data-collection
reference: add them to the transaction and TxQ attribute tables (with per-span
coverage), update the attribute count, and correct the txq.enqueue parent note
(parents to tx.process on the submission path via explicit context; a root on
the open-ledger rebuild path, correlated via current_ledger_seq).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Brings coroutine-aware context storage + tx/consensus worker-body activation.
Resolved: Telemetry.cpp keeps both meterProvider_ (phase-7) and contextStorage_
(coro-aware); doc-09 keeps phase-7 structure and applies the pathfind.request →
rpc.command.<name> correction to phase-7's own PathFind section.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reflects the C1 fix: rpc.command.* stays unscoped (its dispatch wraps
doRipplePathFind which yields), so pathfind.request nests under rpc.process.
The request -> compute -> discover sub-tree nests correctly via ScopedSpanGuard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Document that deterministic-trace_id spans (tx.* apply pipeline, tx.process,
tx.receive, consensus.round) are now genuine trace roots with empty
parent_span_id via the custom DeterministicIdGenerator, superseding the old
synthetic-parent behavior that showed 'root span not yet received' in Tempo.
Also update the fresh-root note: peer entry spans use ScopedSpanGuard::freshRoot().
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
These inbound peer-message entry points (kConsumer) used span(), which
inherits whatever span is active on the peer thread — including a leaked
tx.receive scope — so validations/proposals were wrongly nested under
unrelated transaction traces. rootSpan() starts a fresh trace root.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The spanmetrics connector had no namespace, so it emitted traces_span_metrics_*
metric names by default. The span dashboards and docs are renamed to query
span_* names; this is only correct if the connector emits them too, so add
namespace: "span" to the spanmetrics connector. Both sides change together:
renaming the dashboards without the namespace (or vice versa) would break the
pipeline. Matches the phase9 collector config.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The phase7 OTelCollector::formatName lowercases and strips names, emitting
snake_case metrics with no xrpld_ prefix. The native Grafana dashboards and
the telemetry docs still queried the old xrpld_CamelCase names, so they were
broken against their own pipeline. Rename every metric name to match what the
code emits: drop the xrpld_ prefix and lowercase the remainder. The two job
histograms also drop the redundant 'duration' word (job_queued_us,
job_running_us) to match the phase9 forms. Add havetxset to the cspell
dictionary since the lowercased metric name no longer word-splits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rename the 5 system-* dashboard files to bare domain names (uid ==
filename stem) and update all doc references. The system- prefix was
redundant now that titles no longer carry a recorder-pipeline suffix.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make dashboard uids bare domain names (uid == filename stem) and update
doc references. Grafana links/bookmarks to the old uids will need
refreshing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Match the reader-facing dashboard catalog to the actual dashboard titles,
which no longer carry the recorder-pipeline suffix.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- 09-data-collection-reference: add $service_name, $deployment_environment,
$xrpl_network_type to the template-variable table; note they appear on all
dashboards and point to the runbook's Deployment Tiers section.
- TESTING.md: add a Prometheus check verifying the tier labels
(deployment_environment, xrpl_network_type, service_name) are present on
metric series.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The peer.validation.receive span now records the ledger hash and full-validation
flag under the shared bare keys ledger_hash / full_validation (same keys the
consensus validation spans use), so there is no longer a dotted span attribute.
Remove the "only dotted span attribute" note and update the peer attribute table.
Head sampling is fixed at 1.0 and not configurable. Drop sampling_ratio
from the docker sample/test configs and the data-collection reference;
rewrite the span-metric sampling caveat and volume guidance to point at
collector-side tail sampling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update the Peer Spans note, the known-gaps caveat, and the trace category
toggle table to show trace_peer enabled by default. The Production Setup
example keeps trace_peer=0 as a deliberate high-volume opt-out alongside
sampling_ratio=0.01.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The §1 span and attribute inventory had regressed to an older 16-span snapshot
that uses the pre-2026-05-13 dotted attribute keys, while phase-7's code emits
~36 spans with bare/underscore attribute keys. The §Data Flow Overview and §2
System Metrics sections (native OTLP transport — phase-7's migration) were already
correct and are left unchanged.
- §1.1: expand the span inventory to the full surface — add gRPC (grpc.<MethodName>),
TxQ (txq.*), PathFind (pathfind.*), and the full consensus set (round/phase.open/
establish/update_positions/check/mode_change/proposal.receive/validation.receive).
Fix the phantom rpc.request -> rpc.http_request, add rpc.ws_upgrade. No grpc.request,
no pathfind.rank, no ledger.acquire (the latter is added in phase-9, not yet present here).
- §1.2: convert every span-attribute key from dotted xrpl.<domain>.<field> to the
bare/underscore form. The sole span-attr dotted exception is xrpl.ledger.hash on
peer.validation.receive (shared constant); consensus.validation.send uses bare
ledger_hash. Resource attrs xrpl.network.id/type stay dotted. Fix tx_count/tx_failed
placement (on tx.apply, not ledger.build). Add attribute tables for the new families.
- §1.3: list the full set of spanmetrics dimension labels (bare keys, from the collector
config) instead of the stale xrpl_rpc_command-style names.
- §4/§5: convert Tempo TraceQL and PromQL examples to the bare attribute/label forms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the apply-pipeline stage spans (tx.preflight, tx.preclaim,
tx.transactor) added on phase-3 through the observability stack so the
spanmetrics connector produces per-stage RED metrics without any native
instruments.
- collector: add the `stage` dimension to the spanmetrics connector so
the three stages split into separate metric series (3 bounded values).
- dashboard: add a "Tx Apply Pipeline" section to transaction-overview
with rate, p95 latency, and failure-rate panels grouped by stage, plus
a `stage` template variable. Panels follow the existing config (node
filter, exported_instance legends, Title Case, axis labels).
- The failure panel filters ter_result != tesSUCCESS rather than span
status, because a failing ter code completes the span normally — only
thrown exceptions set an error status. This matches the existing
"Transaction Results by Type" panel convention.
- docs: document the spans, attributes, and stage dimension in the data
collection reference and runbook, including the sampling caveat that
span-derived metrics inherit tracer head-sampling and undercount at
sampling_ratio < 1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Build fixes in PeerImp.cpp:
- Rename duplicate `span` variable to `consSpan` in proposal and
validation handlers to avoid redefinition error
- Fix `->` on non-pointer SpanGuard (now correctly on shared_ptr)
- Fix move-only type copy in lambda capture
Clang-tidy fixes:
- Concatenate nested namespaces in LedgerSpanNames.h and PeerSpanNames.h
- Add missing SpanNames.h includes in BuildLedger.cpp, LedgerMaster.cpp,
PeerImp.cpp for direct seg:: symbol usage
- Add missing <chrono> and <cstdint> includes in BuildLedger.cpp
- Remove unused Feature.h include from BuildLedger.cpp
Rename check fix:
- Run docs.sh to rename rippled_ metric prefixes to xrpld_ in
09-data-collection-reference.md and telemetry-runbook.md
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Prettier formatting for markdown docs and OTelCollector header
- docs.sh rippled→xrpld renames in OTelCollector.cpp comments/strings
- Updated levelization ordering with new dependency edges
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The StatsD receiver config was lost during a branch rebase (--ours
conflict resolution dropped it). Re-add the statsd receiver to the
OTel Collector config and wire it into the metrics pipeline so
beast::insight UDP metrics flow to Prometheus.
Also fixes:
- Metric prefix mismatch: docs used xrpld_ but dashboards/tests use
rippled_ — align all documentation to match the runnable stack
- Remove phantom Peer_Disconnects_Charges from docs (plain atomic,
not a beast::insight gauge)
- Remove premature .codecov.yml exclusions for Phase 7 OTelCollector
files that don't exist on this branch
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace remaining rippled/Ripple references with xrpld/XRPL in
data collection reference, implementation phases, and runbook docs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Integrate the existing StatsD metrics pipeline (beast::insight) into
the OpenTelemetry observability stack and add new trace spans for
ledger build/store/validate and peer proposal/validation receive.
Phase 5b — Ledger, peer, and transaction spans:
- Add ledger.build span with close time attributes in BuildLedger.cpp
- Add tx.apply span with tx_count/tx_failed in BuildLedger.cpp
- Add ledger.store and ledger.validate spans in LedgerMaster.cpp
- Add peer.proposal.receive span with trusted attribute in PeerImp.cpp
- Add peer.validation.receive span with ledger_hash, full, trusted
attributes in PeerImp.cpp
- Add ledger-operations and peer-network Grafana dashboards
Phase 6 — StatsD metrics integration:
- Add StatsD UDP receiver (port 8125) to OTel Collector
- Add 5 StatsD Grafana dashboards: node health, network traffic,
overlay traffic detail, ledger data sync, RPC pathfinding
- Add 09-data-collection-reference.md cataloging all metrics/spans
- Update existing dashboards with new span panels
- Expand telemetry runbook and integration test script
- Add codecov exclusions for telemetry modules
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Bridge the existing beast::insight gauge for resource-limit peer
disconnects (peerDisconnectsCharges_) into the StatsD metric inventory.
Part of the external dashboard parity initiative.
See docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
