Completes the default change applied on phase-6-statsd and
phase-7-native-metrics. The four dashboards introduced on phase-9
(validator-health, peer-quality, fee-market, job-queue) now also
pre-select 'xrpld' rather than 'All' for the service_name filter.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Continuation of the same default change applied on phase-6-statsd
(50d7047abe): the 5 dashboards introduced on phase-7 (formerly
system-*, renamed to bare names on this branch) also get their
service_name filter pre-selected to 'xrpld' instead of 'All'.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Change the pre-selected value of the service_name template variable from
'All' to 'xrpld'. On first load, dashboards now filter to service.name=xrpld
directly rather than showing every service the collector sees. The 'All'
option is still selectable — this only changes the default the widget
starts on.
Reduces the chance of a fresh dashboard link showing unrelated services
(e.g. non-xrpld exporters sharing the same Grafana Cloud stack) before
the user narrows the filter.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Grafana Cloud's OTLP ingest already promotes the `service.name` resource
attribute to a `service_name` metric label. The `transform/cloudlabels`
processor was ALSO setting `service_name` as a datapoint attribute,
which Mimir treats as a duplicate label and concatenates as
`service_name="xrpld;xrpld"` on every series. Dashboards filtering on
`service_name=xrpld` then returned no data.
Remove the redundant `set(attributes["service_name"], ...)` line and
add a comment documenting the doubling. `service.instance.id` does not
exhibit the same behaviour, so that copy is retained for the $node
filter to keep resolving.
Reintroduces the pre-e76eed5296 shape (which was correct on this label)
without reverting that commit's other cloud-label fixes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Completes the rename applied on phase-6-statsd and phase-7-native-metrics.
These four dashboards were introduced on phase-9: peer-quality and
validator-health via b92354715d, fee-market and job-queue via 25868f2740.
Same reasoning: the underlying Prometheus label is `service_name`,
so the UI display label should say 'Service Name' rather than the
ambiguous 'Service'.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Continuation of the same rename applied on phase-6-statsd (commit
91c6582bd5): 5 more dashboards introduced on phase-7 (formerly the
system-* set, renamed to bare names by 2c590a47c5) still carried the
older 'Service' UI label. Aligning them with the rest.
Same reasoning as the phase-6 commit: the underlying Prometheus label
is `service_name`, and the UI label 'Service' misled users into typing
`service=xrpld` in Grafana Metrics Drilldown.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The template variable's Prometheus label name is `service_name` but the
UI display label read 'Service', which mismatched what users typed into
Grafana's Metrics Drilldown UI (which defaults to filter on a bare
`service` label). Rename the display label to 'Service Name' so it
unambiguously points at the underlying `service_name` Prometheus label.
Behaviour unchanged; template-variable name (`service_name`) and PromQL
label are the same. Only the widget label shown at the top of the
dashboard changes from 'Service' to 'Service Name'.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- OTelCollector::formatName made static (no member state); braces added;
<cctype> added for std::tolower.
- Telemetry.cpp getMeter(): braces around single-statement if.
- GetMeter.cpp: drop unused <opentelemetry/metrics/meter.h> and
sync_instruments.h; add <xrpl/beast/utility/Journal.h> and <memory>;
make sdkProvider const.
- ValidationTracker.h: add <cstddef> for std::size_t.
- detail/ValidationTracker.cpp: use std::views::reverse range-based loop;
add <ranges>.
- test ValidationTracker.cpp: use auto for cast result.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Strip xrpld_ prefix, lowercase beast::insight names, and replace
traces_span_metrics_ with span_ in all remaining tracked files:
alert rules, integration tests, workload validation, TESTING.md,
OpenTelemetryPlan docs, code comments, and config templates.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove redundant xrpld_ prefix from all 34 MetricsRegistry instrument
names — service identity comes from the OTel resource, not the metric
name. Duration histograms retain their unit suffix (job_queued_duration_us,
job_running_duration_us, rpc_method_duration_us).
Set spanmetrics connector namespace to "span" so derived metrics become
span_calls_total and span_duration_milliseconds_bucket instead of the
verbose traces_span_metrics_ prefix.
Update all 14 dashboard JSONs: strip xrpld_ prefix, lowercase
beast::insight metric names, replace traces_span_metrics_ with span_.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add service_name to transform/cloudlabels so Grafana Cloud dashboards
can filter by service.name (fixes empty panels for native metrics).
Set spanmetrics namespace to "span" in the cloud collector config and
update dashboards + runbook: traces_span_metrics_ → span_.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Drop the xrpld_ prefix from beast::insight metric names — the OTel
resource (service.name=xrpld) already identifies the service. formatName()
now lowercases and sanitizes spaces/dots to underscores for
Prometheus-compatible names.
Also fix 4 source strings containing spaces that produced invalid OTel
instrument names (silently rejected by the SDK).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Raise TX error rate threshold from 50% to 95%. Short-lived CI test
environments lack pre-funded accounts for complex transactions
(AMMCreate, EscrowFinish, NFTokenCreateOffer, etc.), causing expected
failures that do not indicate an instrumentation problem.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace unused observer_result.h with nostd/shared_ptr.h and
nostd/unique_ptr.h in MetricsRegistry.h (misc-include-cleaner)
- Use auto for cast initializations in MetricsRegistry.cpp and
ValidationTracker test (modernize-use-auto)
- Run rename scripts: rippled -> xrpld in comments and docs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add job_type to otel-naming builtins (standard Prometheus label for
job-queue metrics used in node-health dashboard)
- Add OpenTelemetry SDK to UBSAN suppressions (intentional unsigned
integer overflow in attributemap_hash.h hash computation)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add missing <string_view> include in Consensus.h and RCLConsensus.h.
Remove unused SHAMap.h include from RCLConsensus.h.
Add braces around single-line if/else bodies in RCLConsensus.cpp.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove unused SpanNames.h from Transactor.cpp (transitively included
via TxApplySpanNames.h). Remove unused <ranges> from
TraceContextValidation.h (std::ranges::any_of lives in <algorithm>).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace std::transform with std::ranges::transform in Redaction.cpp.
Replace std::all_of with std::ranges::all_of in Redaction test.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add missing <cstdint> include for std::uint32_t in Telemetry.h.
Add braces around single-line if bodies in Telemetry.cpp.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update the transform/cloudlabels processor to set service_instance_id
instead of exported_instance, matching the local Prometheus path where
resource_to_telemetry_conversion promotes service.instance.id with that
exact label name. Also update source comments and documentation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Merge forward from phase6/7 and apply the same fix to phase9's own
dashboards and alerting rules. exported_instance is a Prometheus scrape
artifact absent in push-based pipelines; service_instance_id is always
present via resource_to_telemetry_conversion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
exported_instance is a Prometheus scrape artifact that only exists when
Prometheus scrapes a target with a conflicting instance label. Users
running push-based pipelines (remote_write, OTLP) never get this label.
service_instance_id is always present via resource_to_telemetry_conversion
regardless of ingestion path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
exported_instance is a Prometheus scrape artifact that only exists when
Prometheus scrapes a target with a conflicting instance label. Users
running push-based pipelines (remote_write, OTLP) never get this label.
service_instance_id is always present via resource_to_telemetry_conversion
regardless of ingestion path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The workload validator queried each expected metric once, immediately after
a fixed post-workload propagation wait. Several beast::insight metrics
(ledger-age and peer-finder gauges, overlay-traffic and rpc-request counters)
only populate after the node validates ledgers and sustains peer traffic,
then travel a 1s OTLP export + 15s Prometheus scrape before they are
queryable. On a slower CI runner that pipeline can settle after the wait
ends, so the single query raced and reported "0 series", failing 12 checks
that pass locally with the same config and binary.
Poll each metric on the /api/v1/series endpoint until it appears or a 45s
window (two scrape cycles) elapses. Present metrics still return on the first
query with no added delay; a genuinely-absent metric still fails after the
timeout. Makes the check robust to runner speed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
xrpld resolves a relative [debug_logfile] against the config file's own
directory (docker/telemetry), not the working directory. The value
`docker/telemetry/data/logs/devnet/debug.log` therefore doubled the
prefix to docker/telemetry/docker/telemetry/... — the collector's mount
never saw the log, so no lines reached Loki.
Use the config-dir-relative form `data/logs/devnet/debug.log`, which
resolves to docker/telemetry/data/logs/devnet/debug.log — exactly the
dir the compose stack bind-mounts as /var/log/xrpld.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extend the log-mount consolidation to the phase-10 workload-validation
assets, which still used the old `rippled` container path:
- docker-compose.workload.yaml: mount target renamed to /var/log/xrpld,
source made overridable via XRPLD_LOG_DIR (default /tmp/xrpld-validation)
- otel-collector-config.grafanacloud.yaml: single glob /var/log/xrpld/*/debug.log
- run-full-validation.sh sets XRPLD_LOG_DIR to its workdir
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The collector's log ingestion used a container path named after the old
`rippled` binary and defaulted its host mount to a location that either
needed root or diverged from where the telemetry configs actually write.
Consolidate on one accessible log root:
- container mount target renamed /var/log/rippled -> /var/log/xrpld
- filelog glob -> /var/log/xrpld/*/debug.log
- compose mount source defaults to the repo-relative ./data/logs
(user-owned, no root), overridable via XRPLD_LOG_DIR
- devnet telemetry cfg writes to data/logs/devnet/debug.log so it lands
one subdir below the mount root and matches the glob
- integration-test.sh sets XRPLD_LOG_DIR to its own workdir
- docs/runbook/task-list updated to match
The default xrpld config (cfg/xrpld-example.cfg) is intentionally left
untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- PerfLogImp::rpcEnd(): return after the requestId-not-found UNREACHABLE
so a stale (now - epoch) duration is no longer recorded to the counter
and histogram in release builds.
- MetricsRegistry peer-version gauge: compare versions numerically via
BuildInfo::encodeSoftwareVersion() instead of a lexicographic string
compare, stripping the non-digit prefix so peer 'rippled-X.Y.Z' lines
up with our bare 'X.Y.Z'. Fixes every peer counting as higher-version.
- MetricsRegistry::stop(): call Shutdown() before ForceFlush() before
reset() so the reader thread stops before teardown and no gauge
callback fires during shutdown.
- MetricsRegistry::start(): extract initExporterAndProvider() and
initSyncInstruments() helpers to keep each function under the line
limit; no behavior change.
- time_in_current_state_seconds: read NetworkOPs::getServerStateDurationUs()
(a lightweight accessor over StateAccounting) and convert microseconds
to seconds, replacing the hardcoded 0.0.
- Correct log-timestamp examples to real Logs::format() output
(2024-Jan-15 10:30:45.123456 UTC) in docs, TESTING.md, and reference.
- Update Loki to v3.4.2 and switch the collector exporter to
otlphttp/loki on the native /otlp endpoint (docs + task list).
- Make the collector log-mount path configurable via XRPLD_LOG_DIR.
- Remove implementation-phase references from shipped config/script
comments, keeping the functional descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the phase-8-owned 'Phase 8' tokens from
loki.yaml and tempo.yaml, keeping the functional descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The getYays()/getNays() accessors already exist on the phase-7 branch.
A cleanup+restore pair on this branch moved them, producing a no-op
diff against phase-7 that mixed consensus churn into the log-correlation
PR. Restore DisputedTx.h to the phase-7 state so this branch carries no
unrelated consensus change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 3/4" token, keep the description.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 4" tokens, keep descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 3" tokens, keep descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 2" tokens, keep descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 1b" token, keep the filter
description.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- node-health.json: repoint the two "All Jobs" detail panels from the
__name__ regex over old xrpld_<job>_milliseconds_bucket names to the
native xrpld_job_running/queued_duration_us_bucket histograms, grouping
by the job_type label and legending on {{job_type}}.
- OTelCollector::callHooks: copy the hook list under mutex_ and invoke
handlers outside the lock. A handler can drop the last reference to an
OTelHookImpl, whose destructor calls removeHook() and re-acquires the
non-recursive mutex_ — invoking handlers under the lock could deadlock.
- ~OTelGaugeImpl: document that the SDK ObservableRegistry serializes
RemoveCallback against the Observe() callback pass with one mutex, so
callback removal is synchronous and gaugeCallback cannot run on a
dangling pointer after the destructor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Key Jobs execution/dequeue panels queried StatsD-era per-job metric
names (xrpld_<job>_milliseconds_bucket, xrpld_<job>_q_milliseconds_bucket)
that the native OTel path no longer emits. Phase 9's MetricsRegistry emits
job timings as two label-dimensioned histograms instead:
xrpld_job_running_duration_us_bucket{job_type="<job>"}
xrpld_job_queued_duration_us_bucket{job_type="<job>"}
Rewrite the 22 affected panel queries (11 job types x running/queued) to
the label-dimensioned form, preserving the histogram_quantile / rate /
sum-by structure and all template-variable filters.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- MetricsRegistry: service.name was recorded as boolean true because the
string literal "xrpld" bound to the OTel AttributeValue variant's
char-const* -> bool overload. Assign std::string so it selects the
string alternative; Prometheus now shows service_name=xrpld (and
exported_job round-trips correctly) on every MetricsRegistry series.
- Grafana alerting contact points: the Slack url / email addresses used
${ENV_VAR} references that expand to empty when unset, and Grafana's
provisioning validator (which lacks ${VAR:-default} support) then
crashes the whole stack on startup. Use non-empty disabled placeholders
(unroutable webhook host, .invalid email) so the stack boots with zero
configuration; delivery stays off until a real destination is supplied.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- ValidationTracker: remove dead hard-trim Pass 1. The preceding
eviction loop already erases every reconciled entry older than the
late-repair cutoff using the same condition, so Pass 1 could never
match. The "drop any reconciled entry" fallback is retained.
- integration-test.sh: probe the StatsD port with a UDP-aware check
(ss -ulnp) instead of curl. StatsD is UDP-only on 8125, so the TCP
curl probe always reported "refused" — a false negative. Guarded by
a command -v ss check that logs a skip when ss is unavailable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Give developers direct access to the full OpenTelemetry metrics API,
symmetric with getTracer(), so all seven OTel instrument types
(including UpDownCounter, sync Gauge, and the observable variants) are
reachable — not just the four beast::insight models.
- Telemetry: build and own the metrics pipeline (OTLP HTTP metric
exporter + PeriodicExportingMetricReader + SpanMetrics histogram view)
alongside the tracer, sharing the same resource attributes and TLS
config. Register it globally via metrics::Provider::SetMeterProvider
and expose Telemetry::getMeter(). Metrics enable with [telemetry];
the metrics endpoint is derived from the trace endpoint.
- beast OTelCollector: no longer owns a pipeline. It fetches the global
Meter, becoming a thin shim over the shared provider (legacy path
during beast deprecation). This also resolves the review note that the
metric exporter ignored [telemetry] use_tls — TLS now comes from the
shared telemetry pipeline.
- Add a libxrpl unit test covering getMeter() on the enabled (global
provider) and disabled (noop) paths, exercising an UpDownCounter.
Design: docs/superpowers/specs/2026-07-07-direct-otel-metrics-api-design.md
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- cmake: keep the opentelemetry-cpp umbrella target for the beast metrics
link and document why. The reviewer suggested linking individual
component targets to avoid over-linking, but the OTel Conan package
under-declares inter-component dependencies (the OTLP client references
sdk::common symbols without a declared edge), so naming components
directly reorders the static link into an unresolvable state. Verified
by building xrpl_tests both ways.
- OTelCollector.h: add usage examples, thread-safety and limitations
@note blocks to the class doc.
- OTelCollector.cpp: correct the @param name docs on the instrument
Impl constructors to describe the already-formatName()'d value.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Phase 11 task list still linked the moved External Dashboard Parity spec
at its old docs/superpowers/ path. Point them at the appendix in
06-implementation-phases.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 09-data-collection-reference and Phase9 task list still linked the moved
External Dashboard Parity spec at its old docs/superpowers/ path. Point them
at the appendix in 06-implementation-phases.md where the content now lives.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the phase-9 alerting design spec out of docs/superpowers/ (which the
project guidelines say not to create) into an appendix of
OpenTelemetryPlan/Phase9_taskList.md, keeping the phase plan self-contained.
Drops the stale one-off branch/PR-restructure section from the spec.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the standalone External Dashboard Parity design spec out of
docs/superpowers/ (which the project guidelines say not to create) and into
an appendix of OpenTelemetryPlan/06-implementation-phases.md, so the phase
plan is self-contained. Repoint the 10 "Source" links in the phase 3/4/7
task lists to the new appendix anchor.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Update the phase-9 task list and the alerting design spec to reference the
Alerting section of docs/telemetry-runbook.md, now that the standalone
ALERTING.md has been folded into the runbook.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the placeholder webhook contact point with two env-driven tiers:
- xrpld-default (Slack) receives warning-severity alerts
- xrpld-critical (Slack + email) receives critical-severity alerts, via a
severity=critical child route in the notification policy
Secrets stay out of git: contactpoints.yaml references ${SLACK_WEBHOOK_URL},
${SLACK_CHANNEL}, and ${ALERT_EMAIL_TO}, which Grafana expands from a
gitignored .env.alerting (loaded via an optional env_file on the grafana
service). .env.alerting.example documents the variables; email additionally
needs GF_SMTP_* configured.
Fold the ALERTING.md runbook content into the Alerting section of
docs/telemetry-runbook.md (operator docs live in one place) and delete
ALERTING.md. Move the .env.* ignore rule into docker/telemetry/.gitignore.
Verified live: Grafana 11.5.2 provisioned both contact points, the
severity-routed policy, and all 6 rules with no errors.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add PathFinding Spans section to the runbook Span Reference and
remove the incorrect "not emitted / later-phase" annotations:
pathfind.request/compute/discover/update_all are emitted since
Phase 2 (upstream of Phase 5).
- Runbook label names use the bare spanmetrics dimensions
(command, rpc_status, local, consensus_mode); fix stale docs.
- 05-configuration-reference: note tls_client_cert/key require use_tls=1.
- TelemetryConfig: reject mTLS cert paths when use_tls=0 so telemetry
is not silently sent in plaintext.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Soften the "non-reversible" wording in Redaction.h. XRP account
addresses are a public, enumerable set, so an unsalted hash is
obfuscation (keeps plaintext out of traces/dashboards), not a
secrecy guarantee. Document why a salt is intentionally omitted:
it would break cross-node/restart correlation, the reason for
hashing rather than dropping the attribute.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve the doCommand error-path span suffix and command attribute against
the handler registry instead of using raw request input. A recognized command
yields its canonical handler name; anything else (unregistered command, or
neither command/method present) collapses to the literal "unknown". This bounds
telemetry cardinality to the finite set of handler names plus "unknown" while
preserving per-command error attribution for known-but-rejected commands
(e.g. a submit that hit rpcTOO_BUSY stays rpc.command.submit). The command
attribute is promoted to a Prometheus label by the spanmetrics connector, so
attacker-controlled input previously enabled unbounded label cardinality.
Also add the missing LCOV_EXCL_STOP in the first SpanGuard::linkedSpan overload
so its coverage exclusion no longer bleeds into the second overload's guards.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Change the six remaining `type: static` search filters in tempo.yaml to
`type: dynamic` so Grafana queries Tempo for available tag values instead
of requiring manual free-text entry. Aligns service-name, node-id,
network-type, span-name, span-status, and span-duration with the existing
node-version and network-id filters.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Final four dashboards introduced in this phase: replace the "pratik-xrpld"
$service_name example with "xrpld-validator".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Same cleanup for the five system-metric dashboards introduced in this phase:
replace the "pratik-xrpld" $service_name example with "xrpld-validator".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The $service_name template variable's description used a personal-name example
value. Replace "pratik-xrpld" with the generic "xrpld-validator" in the five
dashboards that introduced it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow the config change that replaced the personal home directory: the
runbook prose now points at /var/log/xrpld/mainnet/debug.log, matching the
mainnet config and the collector's filelog receiver.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the hardcoded /home/pratik/xrpld-logs path with the conventional
/var/log/xrpld in the mainnet config's [debug_logfile], the docker-compose
log volume mount, and the collector-config comment. Removes a personal home
directory from shipped config.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add an operator-facing Alerting section to the runbook covering the six
provisioned Grafana rules and how to point them at a real receiver.
- Strip the "(Phase 9)" suffix from the MetricsRegistry Gauges heading.
- ALERTING.md: drop the internal dev-phase reference from the intro.
- Phase9_taskList.md: expand Task 9.9 to describe the provisioned alerting
(rules, contact point, policy, ALERTING.md runbook).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Operator-facing cleanup: strip the "(Phase 8)" suffix from the Log-Trace
Correlation section heading in the runbook.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Same operator-facing cleanup as the RPC/transaction/consensus headings: the
runbook documents the running system, not the development phase that added
each span. Strip the "(Phase 6)" suffixes from the Ledger and Peer span
reference headings.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The operator runbook is for people running OpenTelemetry against a node, not
a record of the internal development phases. Strip the "(Phase N)" suffixes
from the span-reference headings and replace the personal-name example
service.name value with a generic one.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add operator-facing alerting on the health-critical metrics phase 9 emits,
completing the "alerting rules" deliverable already listed in the phase-9
task list and Jira story.
Provisioned YAML under grafana/provisioning/alerting/ (auto-loaded by the
Grafana container; no docker-compose change needed):
- rules.yaml: 6 alerts in 3 groups — consensus/ledger health, validator
health, job queue. Each is a query(A, 5m rate/quantile by exported_instance)
-> reduce(B, last) -> threshold(C) on the Prometheus datasource.
- contactpoints.yaml: xrpld-default webhook receiver (local-dev placeholder).
- policies.yaml: flat routing to xrpld-default, grouped by
alertname + exported_instance.
Also adds docker/telemetry/ALERTING.md runbook (per-alert meaning, causes,
threshold tuning, how to point alerts at a real receiver) and the design spec.
Verified live: Grafana 11.5.2 provisioned all 6 rules with no rule in error
health; contact point loaded.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The test Peer adaptor's getTelemetry() returns a shared function-local
NullTelemetry instance and does not read any per-peer state, so
clang-tidy's readability-convert-member-functions-to-static flags it as
an error under -warnings-as-errors. Mark it static; it has no callers
that depend on it being a non-static member.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Filter dropdowns were empty on Grafana Cloud: template query-variables
referenced uid ${DS_PROMETHEUS} (a datasource variable that had no value
and was hidden), while panels used the bare default datasource. So panels
rendered but Service/Environment/Network/Node variables resolved against
nothing.
Wire DS_PROMETHEUS (and DS_TEMPO where used) through every panel, target,
annotation, and query-variable, and make the pickers visible. Empty
current value means each instance auto-selects its default datasource of
that type, so one dashboard set works on both the local stack and Grafana
Cloud without per-instance uids.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Grafana Cloud ingests metrics via OTLP with no Prometheus scrape, so the
tier/instance resource attributes never became series labels the way the
local prometheus exporter promotes them. Dashboard $node,
$deployment_environment, and $xrpl_network_type filters therefore matched
nothing on Cloud (only service_name existed).
Split the metrics pipeline: metrics/local keeps the Prometheus exporter
(unchanged label promotion), metrics/cloud adds a transform that copies
service.instance.id -> exported_instance, deployment.environment ->
deployment_environment, and xrpl.network.type -> xrpl_network_type onto
datapoint labels so the same dashboards filter correctly on both backends.
Also strip telemetry.sdk.* on all pipelines. Validated against
otel-collector-contrib 0.121.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a resource/stripsdk processor that deletes telemetry.sdk.language,
telemetry.sdk.name, and telemetry.sdk.version (auto-added by the OTel SDK
to every Resource). Wire it into the traces pipeline; the metrics and
logs pipelines pick it up on the branches that introduce them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Point the workload validation uid list and implementation-phases doc at
the renamed bare dashboard uids.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename the 4 xrpld-* dashboard files to bare names and set every
dashboard uid to its bare filename stem; update doc/plan references.
Removes the redundant xrpld-/system- prefixes now that titles are clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename the 5 system-* dashboard files to bare domain names (uid ==
filename stem) and update all doc references. The system- prefix was
redundant now that titles no longer carry a recorder-pipeline suffix.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make dashboard uids bare domain names (uid == filename stem) and update
doc references. Grafana links/bookmarks to the old uids will need
refreshing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Grafana renders the dashboard-level description as plain text (not
markdown), so the h6 headings/italics showed as literal characters.
Convert to plain 'What this shows:' / 'Use it to:' labeled lines. Panel
descriptions (which DO render markdown) are unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Grafana renders the dashboard-level description as plain text (not
markdown), so the h6 headings/italics showed as literal characters.
Convert to plain 'What this shows:' / 'Use it to:' labeled lines. Panel
descriptions (which DO render markdown) are unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The separate xrpld-rpc-perf-otel dashboard was merged into xrpld-rpc-perf,
so the validation harness must no longer expect it to exist.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Match the reader-facing dashboard catalog to the actual dashboard titles,
which no longer carry the recorder-pipeline suffix.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The separate RPC Performance (OTel) dashboard was folded into RPC
Performance as a per-method section, so describe it as an addition to the
existing dashboard rather than a new one.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fold the 7 panels from the separate RPC Performance (OTel) dashboard into
RPC Performance as a labelled 'Per-Method Detail' section, and retire the
duplicate dashboard. The two showed the same subject (RPC) from two data
sources; one dashboard with two sections is clearer and loses no panel.
Adds the $method template variable for per-method filtering.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Consensus Health, Ledger Operations, and Transaction Overview had no
top-level description, so Grafana showed no info button. Add a one-line
summary of what each dashboard shows and its data source.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The integration-test.sh trace queries used resource.service.name="rippled",
but the OTel resource attribute is "xrpld" (TelemetryConfig.cpp, MetricsRegistry.cpp),
so every span check and the service-registration check returned no results.
Also restore .github/scripts/rename/README.md, which an overzealous
find-and-replace had corrupted into circular no-ops (e.g. rename "xrpld" to
"xrpld"). The file documents the rename process and must retain the source
names (rippled/ripple); restored to match develop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every dashboard already lives in the rippled/xrpld dashboard folder and
carries domain-specific tags (consensus, rpc, statsd, etc.), so the
leading "rippled" tag added no filtering value and only cluttered the
tag list.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every dashboard already lives in the xrpld folder and carries a
domain-specific tag (rpc, consensus, statsd, etc.), so the leading
"xrpld" tag added no filtering value and only cluttered the tag list.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Repoint the Grafana Cloud collector config comment at the "Deployment
Tiers" section of docs/telemetry-runbook.md now that the standalone tier
doc has been folded into the runbook.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remove the standalone docker/telemetry/DEPLOYMENT_TIERS.md; its content now
lives in the "Deployment Tiers" section of docs/telemetry-runbook.md, the
established operator doc. Repoint the Grafana Cloud collector config comment
at that section.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- 09-data-collection-reference: add $service_name, $deployment_environment,
$xrpl_network_type to the template-variable table; note they appear on all
dashboards and point to the runbook's Deployment Tiers section.
- TESTING.md: add a Prometheus check verifying the tier labels
(deployment_environment, xrpl_network_type, service_name) are present on
metric series.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a Deployment Tiers section covering the four filtering dimensions
(node, service, network, environment), who owns each attribute, the
collector's upsert-vs-insert rule, per-tier collector values, and how the
tier labels reach metrics. This is the operator-facing home for the
deployment-tier feature.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- 05-configuration-reference: service_name now applies to metrics as well
as traces (service.name resource attribute).
- 07-observability-backends: document that collector enrichment includes
deployment-tier tagging (deployment.environment + fallback
xrpl.network.type) for filtering one Grafana stack by tier.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Apply the deployment-tier tagging to the Grafana Cloud dual-export config
so cloud data is filterable by tier just like the local stack:
- add the resource/tier processor (deployment.environment upsert,
xrpl.network.type insert) and run it on all three pipelines.
- enable resource_to_telemetry_conversion on the prometheus exporter and
add the tier keys to spanmetrics for the local scrape surface.
Add docker/telemetry/DEPLOYMENT_TIERS.md documenting the four filtering
dimensions, the upsert-vs-insert ownership rule, per-tier collector
values, and how the labels reach each signal.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add $service_name, $deployment_environment, $xrpl_network_type template
variables to the metric-gap-fill dashboards and wire them into every
panel query that filters by $node, so these dashboards can be sliced by
tier like the rest of the set.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add resource/tier to the logs pipeline so log records carry the same
deployment.environment and xrpl.network.type attributes as traces and
metrics. This lets a single Grafana stack filter logs by tier alongside
the other signals. resource/logs still runs first to set service.name
and the job label; resource/tier then adds the tier attributes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The native OTel metrics path hard-coded service.name="xrpld" and stamped
no network attribute, while traces stamped a configurable service.name
and xrpl.network.type. Metrics therefore could not be filtered by service
or network. Align the two paths:
- OTelCollector::New / OTelCollectorImp gain serviceName + networkType
params. service.name uses the configured value (default "xrpld" when
unset, preserving today's behavior); xrpl.network.type is stamped when
provided. The key is a string literal because beast/insight sits below
the telemetry module and cannot include its SpanNames const.
- CollectorManager reads service_name from [insight], falling back to the
[telemetry] value, and receives the network type from the caller.
- Application derives the network type once via the shared
telemetry::networkTypeFromId, now declared in Telemetry.h and moved out
of an anonymous namespace so the trace and metric paths reuse a single
0/1/2 -> mainnet/testnet/devnet mapping (no duplication).
Dashboards (5 system-* files): add $service_name, $deployment_environment,
$xrpl_network_type template variables and wire them into every panel query
that filters by $node.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extend the deployment-tier tagging to the metrics path and expose it in
the dashboards so a single Grafana stack can be sliced by environment,
network, and service:
Collector (otel-collector-config.yaml):
- prometheus exporter: enable resource_to_telemetry_conversion so
resource attributes (deployment.environment, xrpl.network.type,
service.name, service.instance.id) become Prometheus labels.
- metrics pipeline: run resource/tier so statsd metrics are tagged too.
- spanmetrics: add the tier keys to the resource-metrics grouping key
so per-tier series stay grouped separately.
Dashboards (5 files):
- add $service_name, $deployment_environment, $xrpl_network_type
template variables, named to match their Prometheus labels.
- inject the matching label filters into every panel query that
already filters by $node.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a resource/tier processor so each collector stamps the deployment
tier onto the signals it forwards, enabling a single Grafana stack to
hold data from many collectors and filter by tier:
- deployment.environment (upsert): the collector is authoritative for
its environment (local/ci/test/prod).
- xrpl.network.type (insert): the xrpld node already stamps its own
chain, so the collector only fills this when absent, never
overwriting a node's real network.
Wired into the traces pipeline (the only pipeline on this branch);
later phases extend it to metrics and logs as those pipelines appear.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Apply the same width rule from the earlier transaction-overview fix to the
job-queue, rpc-perf-otel, and rpc-performance dashboards. Timeseries panels
whose legend is a right-side table need the horizontal room, so they go to
full width (w=24); each stacks onto its own row and panels below shift down.
Pie charts, heatmaps, and bottom-legend panels stay half width.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CI enabled more clang-tidy checks via the upstream merge:
- DiscardScope discardScope -> const (misc-const-correctness)
- drop the (::max)() Windows macro-guard parens; NOMINMAX is defined
project-wide so the bare form is correct (readability-redundant-parentheses)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an opt-in collector config + compose override that forward traces,
metrics, and logs to a Grafana Cloud OTLP gateway alongside the local
backends. Enabling it is a runtime choice (layer the override compose
file) — no xrpld rebuild, base stack unchanged. Credentials come from a
gitignored .env file. Documented in the telemetry runbook and TESTING.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 5 added 4 more unit-test cases calling the old setupTelemetry name.
Align them with the renamed makeTelemetrySetup free function so xrpl_tests
builds.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The free function was renamed makeTelemetrySetup but this unit test still
called the old name, breaking the xrpl_tests build on arm64.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The followsFrom linkedSpan overload referenced attr::linkType /
attr_val::followsFrom, which don't exist in SpanNames.h — the other two
link sites already use the local kLinkTypeKey / kLinkTypeFollowsFrom
constants. Unify the third site. Fixes arm64 build (-Werror).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The network-type label is derived from [network_id] in TelemetryConfig.cpp;
unmapped/unset IDs fall through to "unknown", not "standalone". Align the
design-doc resource-attribute table with the code and cfg example.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor: Update RPM version scheme; remove auto-update script; service hardening
- **RPM version scheme**: pre-releases now use `~` in the `Version` field instead of the `0.<release>.<suffix>` `Release`-field hack. Matches Debian's `~` convention, so RPM and DEB version strings are symmetric. Requires rpm ≥ 4.10 (RHEL 9 ships 4.17).
Before/after for a pre-release build:
```
# before
xrpld-3.2.0-0.1.rc3+202606011647.d4cb68d5.el9.x86_64.rpm
# after (symmetric with DEB)
xrpld-3.2.0~rc2+202606010139.7679a310-1.el9.x86_64.rpm
xrpld_3.2.0~rc2+202606010139.7679a310-1_amd64.deb
```
- **Auto-update removed**: `update-xrpld`, `update-xrpld.service`, and `update-xrpld.timer` deleted. The `50-xrpld.preset` `disable` line for the timer is dropped too.
- **Service hardening** (two new `[Service]` directives in `xrpld.service`):
- `CapabilityBoundingSet=CAP_NET_BIND_SERVICE` — drops every Linux capability except `CAP_NET_BIND_SERVICE`, capping the privilege ceiling to least-privilege while still letting operators bind ports <1024 (e.g. WS/HTTPS on 443).
- `SystemCallArchitectures=native` — restricts the service to the native syscall ABI, blocking alternate-ABI (32-bit/x32) syscalls used to evade seccomp filtering.
- [ ] Build RPM from a pre-release version (e.g. `3.2.0-b1`) and confirm `rpm -qi` shows `Version: 3.2.0~b1`, `Release: 1`
- [ ] Confirm `3.2.0~b1` sorts before `3.2.0` via `rpmvercmp`
- [ ] Install package and confirm no `update-xrpld*` units appear in `systemctl list-unit-files`
- [ ] Confirm `systemctl show xrpld` reflects the new `CapabilityBoundingSet` and `SystemCallArchitectures`
* fix: Track tmpfiles-created directories in RPM %files as %ghost
The follows-from link attribute used the dotted xrpl.link.type form, which
is reserved for resource attributes. Span attributes use the <domain>_<field>
underscore form, so the key becomes link_type.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Tempo datasource search filter used the stale dotted xrpl.consensus.ledger_id
tag, but the consensus.round span emits the bare consensus_ledger_id key
(ConsensusSpanNames.h). The dotted tag matched no spans and failed the Rule C
naming check. Align it with L1.
The TxQSpanNames.h span-hierarchy comment and the Phase 3 task-list attribute
table showed txq.accept.tx, but the constant emits txq.accept_tx
(op::acceptTx = "accept_tx"). Correct both.
- peer.validation.receive now asserts the shared bare ledger_hash /
full_validation keys (was the dotted xrpl.ledger.hash and validation_full);
PARITY_SPAN_ATTRS checks both on the peer span too.
- Fix a span-name drift: the per-transaction accept span is txq.accept_tx
(op::acceptTx = "accept_tx"), not txq.accept.tx — the old assertion never
matched and was silently skipped as optional.
- Drop the "intentionally dotted" notes; there is no dotted span attribute.
The peer.validation.receive span now records the ledger hash and full-validation
flag under the shared bare keys ledger_hash / full_validation (same keys the
consensus validation spans use), so there is no longer a dotted span attribute.
Remove the "only dotted span attribute" note and update the peer attribute table.
The peer.validation.receive span recorded the ledger hash under the dotted
xrpl.ledger.hash form — the only dotted span attribute, and inconsistent with
the bare ledger_hash the consensus validation spans use for the same value.
Make the base SpanNames.h ledgerHash bare (ledger_hash) and add the shared
fullValidation key, so the peer and consensus validation spans share one key
per concept (told apart by span name, not an emitter prefix). PeerSpanNames now
re-exports both from the base; the peer validation_full key folds into the
shared full_validation. The peer trust attrs (proposal_trusted /
validation_trusted) already follow the convention and are unchanged.
Make consensus span-attribute keys consistent with the peer spans that record
the same concepts, using ONE shared name per concept (told apart by span name),
not an emitter prefix:
- Add the shared ledger_hash and full_validation keys to the base SpanNames.h
(a ledger-object property and an is-full-validation flag, both shared with the
peer validation spans); consensus re-exports them via `using` instead of
defining local copies.
- trusted (bare) -> proposal_trusted on consensus.proposal.receive and
validation_trusted on consensus.validation.receive — the same message-type
qualification the peer.{proposal,validation}.receive spans use.
Also collapse a same-emitter duplicate: establishCounter_ was recorded under
both establish_counter and establish_count on two consensus spans; both now use
establish_count (the establishCounter span-name constant is removed; the
establishCounter_ member variable is unchanged).
The consensus.proposal.receive and consensus.validation.receive spans record
the trust flag under proposal_trusted / validation_trusted (shared with the
peer spans of the same message type), not a bare `trusted`. Update the span
reference table to match.
State the rules so they stay consistent across code, collector, Tempo,
dashboards, and docs:
- Per-span-unique field -> bare name (the span name carries the domain).
- Same concept on more than one span -> ONE shared key, reused verbatim and
distinguished by span name, never tagged with the emitting workflow
(e.g. ledger_hash, full_validation, proposal_trusted/validation_trusted).
Defined once in the base SpanNames.h and re-exported by each domain header.
- Collision qualifier <domain>_<field> only to separate DIFFERENT concepts that
share a word, or the OTel-reserved status key (rpc_status, consensus_state).
- Dotted xrpl.<...> is reserved for resource attributes (xrpl.network.*).
Updates CONTRIBUTING.md (permanent home) and OpenTelemetryPlan §2.3.3.
Rule A silently missed a dotted span attribute (xrpl.ledger.hash) because of
two interacting bugs:
1. attr_keys_from_header resolved each constant via a flat global symbol table
keyed by bare name, so a later header defining a same-named constant (e.g.
consensus attr::ledgerHash = "ledger_hash") clobbered the base header's
attr::ledgerHash = "xrpl.ledger.hash", erasing the real dotted key from L1.
Now each constant is resolved against its own header (the global table only
seeds seg::/join() cross-file references); using-re-exports still resolve
globally.
2. derive_dotted_resource_keys allowlisted any dotted key declared in the base
SpanNames.h. Now it allowlists only the keys actually passed to
Resource::Create() in Telemetry.cpp (semconv service.* + the attr:: constants
set there, e.g. xrpl.network.*). A dotted key declared in a header but never
set as a resource attr is a Rule-A violation.
Adds 4 regression tests (collision, using-re-export, allowlist scope, brace
matching). No allowlist exception is added — the check now catches the
violation so the offending code can be fixed.
Forward-merge the OTel naming-check Rule C/D fixes and the phase 2-9 chain.
Conflict resolutions:
- 02-design-decisions.md TxQ Attributes: take the convention-correct markdown
table (txq_* keys) over the stale planned-attrs code block.
- SpanGuardFactory.cpp: keep the explanatory comment for the literal attribute
keys in the libxrpl test.
Rule C was reading docker/telemetry/tempo.yaml (the Tempo server config), which
has no filter tags, so it always SKIPped — L3 was silently unenforced. The
trace-search filter tags actually live in the Grafana datasource provisioning
file (docker/telemetry/grafana/provisioning/datasources/tempo.yaml) as
search.filters[].{tag,scope}. Point Rule C there (server file as fallback),
pair each tag with its scope, validate only span-scope tags against L1 (resource/
intrinsic tags like service.*/name/status/duration are exempt), and strip the
TraceQL span. prefix.
On phase-9 this turns "SKIP: C" into "OK: C: 24 tempo span-filter tags all in
L1" — L3 is now genuinely guarded. Adds a RuleCTempo test class (4 cases:
span-tag-not-in-L1 flagged, span-tags-pass, resource/intrinsic ignored, skip
when datasource absent). 83 tests total.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 9 surfaced two Rule D gaps (false positives, not data errors):
- TraceQL `span.<attr>` / `resource.<attr>` references: the bare attribute is
in L1, but the scope-prefixed form was flagged. Now strip the
span./resource./event./link/instrumentation_scope. prefix before the L1
lookup.
- Native OTel metric labels (e.g. `job_type`, `reason`) emitted by
MetricsRegistry are valid dashboard labels but are not span attributes. Add
an L6 source: parse `Add(.., {{"label", ...}})` instrument calls and accept
those label keys alongside L1 and builtins.
Verified against phase-9's real dashboards: 6 prior false positives -> 0.
79 tests (7 new for span-prefix stripping and metric-label extraction).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bring phase-8 (log correlation) and the naming-check + BasicConfig fixes forward
into phase 9 (native-metrics gap fill). Phase 9 adds MetricsRegistry and
LedgerSpanNames.h.
Conflict resolution:
- presentation.md: took phase-9's restructured deck — sampling now lives in
Slide 8 (Head vs Tail) and the Phase 10/11/12 future-phase slides; phase-8's
older standalone sampling section is superseded (no content lost).
- ordering.txt/loops.txt: regenerated via generate.py.
Naming check: A/B/E/F/G green. Rule D reports 6 findings that are all check
gaps, not data errors (verified every dashboard label traces to code):
- 3 are TraceQL `span.<attr>` references (bare attr is in L1); Rule D should
strip the span./resource. scope prefix.
- 2 (`reason`, `job_type`) are native OTel metric labels emitted by
MetricsRegistry.cpp — a new source of truth the check does not yet model.
The Rule D enhancement lands on phase-1c and merges forward.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Forward-merge the phase-2/3 BasicConfig.h include-path fix to keep the chain
consistent. (phases 4+ already had the correct include; no file change here.)
Forward-merge the phase-2/3 BasicConfig.h include-path fix to keep the chain
consistent. (phases 4+ already had the correct include; no file change here.)
Forward-merge the BasicConfig.h include-path fix (basics -> config) so the
phase-3 build-test job compiles. phase-3 had the same broken include as phase-2.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
src/tests/libxrpl/telemetry/TelemetryConfig.cpp included
<xrpl/basics/BasicConfig.h>, but the header lives at
<xrpl/config/BasicConfig.h> (the phase-1b layout). This broke the build-test
job on phase-2 (and phase-3) across all platforms with a "file not found"
fatal error. The fix already exists on phase-4+; backport it to the branch
that introduced the test so each PR builds on its own.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bring native-metrics (phase 7) and the naming-check work forward into phase 8
(log correlation).
Conflict resolution — tempo.yaml datasource provisioning (search.filters):
- Took phase-7's complete 30-filter list as the base (it is the comprehensive
set), preserved phase-8's tracesToLogs Loki-correlation block, and added
phase-8's two unique filters (tx-type, ledger-hash). Verified the result is
the full union (32 filters) with no filter dropped and no duplicate ids.
- Fixed a pre-existing dotted tag carried in from phase-7: the
consensus-ledger-id filter used `xrpl.consensus.ledger_id`; the code emits
the underscore key `consensus_ledger_id` (ConsensusSpanNames.h), so the
dotted form was a dead filter. Now corrected.
Every span-scope filter tag verified to exist in the L1 *SpanNames.h key set.
Naming check green; 72 tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rule D (dashboard PromQL labels must exist in L1) flagged `__name__` once the
phase-7 system-*.json dashboards started using `sum by (le, __name__)`.
`__name__` is the Prometheus reserved label for the metric name itself — a
builtin, not a span attribute. Add it to the builtin allowlist and cover it
with a test. (Earlier dashboards only used `__name__` inside `{__name__=~...}`
matchers, which the label regex did not extract, so this surfaced only now.)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bring the hardened OTel naming check (Rule E fix + 71 tests) and the phase 1-6
convention work forward into phase 7 (native metrics).
Conflict resolution:
- 05-configuration-reference.md: kept phase-7's native-OTLP metrics story
(server=otel, /v1/metrics) over phase-6's superseded StatsD scrape job, in
prose form (no code block).
- ordering.txt/loops.txt: regenerated via generate.py (not hand-edited).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bring phase-4 forward into phase 5 (docs & deployment). Phase 5 owns the
operator runbook (docs/telemetry-runbook.md).
Conflict resolution:
- 05-configuration-reference.md: took phase-4's code-block-free prose for the
config-parser section; the mTLS options (tls_client_cert/tls_client_key)
remain documented in the §5.1.2 options table.
Phase-5-owned naming fix:
- docs/telemetry-runbook.md: converted 20 dotted xrpl.<domain>. attribute keys
in the Span Reference tables to the underscore convention (tx_hash, peer_id,
ledger_seq, consensus_mode/round/round_id/ledger_id, tx_id). Span NAMES stay
dotted (rule 5). `trusted` left as-is — verified it matches the code constant
ConsensusSpanNames.h (the proposal_trusted/validation_trusted split is a
separate, not-yet-applied code change).
End-to-end reconciliation: every span-attribute token in the runbook (L5) now
matches an L1 *SpanNames.h constant. The xrpl_<domain>_<field> tokens are the
spanmetrics-derived Prometheus labels (documented mapping), not span attrs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three entries in Phase4_taskList.md annotated dotted attribute keys
(`xrpl.consensus.round`, `round_id`, `ledger_id`) as "(kept — rule 5)". That
is wrong: rule 5 keeps SPAN NAMES dotted, not attribute keys — the code emits
the underscore form (consensus_round, consensus_round_id, consensus_ledger_id)
per ConsensusSpanNames.h. Remove the false justification; the dotted form in
this task list remains as the illustrative-readability form documented in the
note at the top of the file.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The two "Spans Produced" tables in 06-implementation-phases.md (added by the
Phase 4 consensus work) listed span ATTRIBUTE keys in the dotted
xrpl.consensus.* / ledger.seq / mode.old form. Convert them to the underscore
convention, matching the authoritative ConsensusSpanNames.h constants
(consensus_round, ledger_seq, consensus_mode, consensus_round_id,
consensus_ledger_id, consensus_result, mode_old/new, agree_count/disagree_count,
etc.). Span NAMES in column 1 stay dotted (rule 5). Every key verified to exist
as a constant in ConsensusSpanNames.h / SpanNames.h.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bring phase-3 forward into phase 4 (consensus tracing). Phase 4 introduces
ConsensusSpanNames.h (and already did the trusted→proposal/validation rename
and underscore-attr conversion in ffc197b914).
Conflict resolution (reviewed by a code-review agent before commit):
- 02-design-decisions.md §2.4 Consensus Attributes: merge-both — kept phase-3's
underscore table form AND folded in phase-4's richer "Phase 4a" attribute set
(round_id, ledger_id, trace_strategy, converge/establish/disputes counts,
agree/disagree counts, threshold_percent, consensus_result, mode_old/new),
each mapped to its authoritative ConsensusSpanNames.h constant. Dropped the
planned-but-unimplemented proposers_agreed/proposers_total (no code constant;
agree_count/disagree_count serve that role).
- SpanGuardFactory.cpp: kept phase-4's explanatory comment about why a libxrpl
test uses literal keys, plus the converted command/rpc_status keys.
Naming check passes (100 keys across 8 *SpanNames.h headers, including consensus).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bring phase-2 forward into phase 3 (transaction tracing). Phase 3 introduces
TxSpanNames.h, TxQSpanNames.h, and TxApplySpanNames.h.
Conflict resolution:
- TxQ.cpp: kept phase-3's txq_span-based instrumentation (phase-2 had none).
Dropped the orphaned `NumberSO{... fixUniversalNumber}` line — develop's
#5962 (Retire fixUniversalNumber) removed that symbol repo-wide; the
conflict block had carried one stale copy that would not compile.
- 05/08/OpenTelemetryPlan.md: dropped the deleted 04-code-samples / POC_taskList
references (carried from phase-2), kept phase-3's new secure-OTel.md doc rows,
section, and Mermaid node/edge/style. Config code block -> prose; merged the
secure-OTel hardening pointer with the authoritative-config prose.
- Phase3_taskList.md: removed the "dotted keys for readability" note that came
from phase-2 — phase 3 already uses the underscore keys.
Reviewed by code-review agents: telemetry instrumentation intact, naming check
green (47 keys across 7 *SpanNames.h headers), no conflict markers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Phase3/4/5 task lists show attribute keys in the older dotted
`xrpl.<domain>.<field>` form because that mirrors how a fully qualified
attribute reads in a Tempo trace view, which keeps the planning prose readable.
Add a note to each clarifying that the implemented keys follow the underscore
convention in CONTRIBUTING.md and that the *SpanNames.h constants (enforced by
the CI naming check) are the single source of truth — so the dotted form here
is illustrative, not the literal key.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SpanGuardFactory.cpp set dotted "xrpl.rpc.command"/"xrpl.rpc.status" attribute
keys. Test files are exempt from the naming check's Rule F (they pass arbitrary
literals to exercise the API), so this slipped through, but the keys should
still illustrate the underscore convention. Use "command"/"rpc_status".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bring the hardened OTel naming check forward from phase-1c: unconditional
Rule F, test-file exemption, and the Rule H in-place-constant warning. The
check passes clean on phase 2 (24 keys across 4 *SpanNames.h headers including
PathFind; the SpanGuardFactory.cpp test is correctly exempt from Rule F).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three robustness fixes to check_otel_naming.py, all on phase-1c where the
script lives:
- Rule F now runs UNCONDITIONALLY. It is a purely syntactic check on the
call-sites and does not need the L1 key set, so code that calls
SpanGuard::span/setAttribute directly without ever defining a *SpanNames.h
is still caught (previously it was silently skipped when no header existed).
- Exempt test files from Rule F (tests pass arbitrary literal keys to exercise
the API). The call-site matcher now requires a SpanGuard/`.`/`->` receiver,
so std::span and bare declarations no longer false-positive.
- Add Rule H (warning, non-fatal): a namespace-qualified constant used at a
telemetry call-site but not defined in any *SpanNames.h is flagged, catching
constants defined in-place instead of in the proper header. Bare locals and
std:: names are not warned to avoid noise.
SpanGuard.h / Telemetry.h @code examples updated to reference constants that
exist on this branch. README documents the new behavior.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The script and its README live on phase-1c (where check_otel_naming.py was
introduced). The test-file Rule-F exemption was mistakenly applied here on
phase-2; revert to phase-1c's version verbatim. The exemption and further
script improvements will land on phase-1c and merge forward, keeping the
script's logic on the branch that owns it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bring the naming convention, code-sample cleanup, and CI naming check into
phase 2 (RPC tracing). Phase 2 introduces PathFindSpanNames.h.
Conflict resolution:
- 04-code-samples.md, POC_taskList.md: deletion wins.
- 02-design-decisions.md: took the convention-applied tables, but kept phase-2's
accurate PathFinding summary row (pathfind_fast/search_level/num_paths/...,
matching the implemented PathFindSpanNames.h).
- 05/08: took the code-block-free prose; kept phase-2's Phase2-5_taskList.md
index rows (dropping only the deleted POC row). Fixed stale setup_Telemetry/
make_Telemetry doc references to the code-correct setupTelemetry/makeTelemetry.
- Telemetry.h auto-merged to the constant-based @code examples.
check_otel_naming.py change: exempt test files from Rule F (tests pass
arbitrary literal keys to exercise the API). The check passes clean on the
merged tree (24 keys across 4 *SpanNames.h headers, including PathFind).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add Rule G to check_otel_naming.py: every span-attribute key must be
lower_snake_case (^[a-z][a-z0-9_]*$ per dot-separated segment). This catches
camelCase, UPPERCASE, and spaces in keys, which the structural (dotted) and
source (literal) rules did not. Document it in the script README and
CONTRIBUTING.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add check_otel_naming.py and wire it into on-pr.yml so every PR validates
that span-attribute names stay consistent across the code, collector, Tempo,
dashboards, and docs.
- The valid key set is derived dynamically from the *SpanNames.h constants and
the resource attributes the code registers in Telemetry.cpp — no hardcoded
allowlist to drift.
- Each rule is presence-gated: it runs only when the file it needs is in the
tree, so the check is correct whether telemetry changes land in one PR or
several (the collector/Tempo/dashboard/runbook layers arrive in later phases).
- Rule A flags dotted span-attribute keys; Rule F flags string-literal
attribute keys and span-name arguments (values may be runtime data).
- stdlib-only, mirroring the levelization check (bare `python`, no pip step).
- Telemetry.h / SpanGuard.h @code examples now use *SpanNames.h constants so
the strict literal check passes.
- CONTRIBUTING.md documents the check and how to run it locally.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bring the phase-1a/1b naming-convention and code-sample cleanup into 1c.
Conflict resolution:
- 04-code-samples.md, POC_taskList.md: deletion wins.
- OpenTelemetryPlan docs (01/02/03/05): took the convention-applied,
code-block-free versions; verified no attribute category, table row, or
section header was lost (the differences were dotted->underscore renames).
- Telemetry.h: kept 1c's RpcSpanNames.h constant-based example
(rpc_span::attr::command) over the string literal.
- 31 non-telemetry files are clean develop carry-forward (identical to 1b).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bring the phase-1a code-sample cleanup into phase-1b.
Phase 1a deleted the two code-sample docs (04-code-samples.md,
POC_taskList.md) and stripped C++/config code blocks from the remaining
plan docs, replacing them with prose summaries and pointers to the real
source/config files.
Conflict resolution:
- 04-code-samples.md, POC_taskList.md: deletion wins (phase-1b's SpanGuard
rewrites of these files are intentionally dropped).
- 03/05/OpenTelemetryPlan.md: took phase-1a's code-block-free prose, but
kept phase-1b's accurate descriptions of the real telemetry code
(DiscardFlag.h, FilteringSpanProcessor, SpanGuard factory methods, and
the corrected file-count/file-list tables).
- presentation.md left untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bring the span attribute naming convention (phase 1a) into phase 1b.
Conflict resolution kept phase-1b's SpanGuard-based workflow and applied
the underscore naming convention to all non-code-sample text:
- Converted prose, tables, Mermaid labels, and TraceQL/PromQL query
references across the plan docs to the underscore form.
- Converted the two @code attribute-key examples in Telemetry.h
(command, tx_type).
- Left the code-sample files (04-code-samples.md, POC_taskList.md) and
03-implementation-strategy.md code blocks at the phase-1b version; the
code-sample docs are slated for removal on phase-1a.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Establish the single, authoritative naming convention for OpenTelemetry
span attribute keys so the code, collector, Tempo, dashboards, and docs
stay in sync.
- CONTRIBUTING.md: new "Telemetry span attribute naming" section under
the Style guide as the permanent, canonical home for the rules.
- OpenTelemetryPlan/02-design-decisions.md: new section 2.3.3 stating the
decided convention as design, and section 2.4 attribute schema realigned
to the underscore form (exact key spelling defers to the *SpanNames.h
constants).
- Sweep the remaining plan docs: convert dotted xrpl.<domain>.<field> span
attribute keys to the underscore form; leave span names and the
OTel-standard service.*/http.* and xrpl.network.* resource keys dotted.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The xrpl_tests target links only xrpl.libxrpl and cannot include the
xrpld-level ConsensusSpanNames.h / RpcSpanNames.h (CI build failure on
ubuntu-gcc-debug-arm64). Revert the test to underscore-form attribute
literals, restore the SpanNames.h include for seg::consensus, and drop
the now-unused tests.libxrpl -> xrpld.* levelization entries.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- tempo.yaml: align consensus filter tags with emitted keys
(consensus_mode, consensus_round, ledger_seq) instead of dotted form
- haveConsensus(): set span attributes before early-return paths so the
consensus.check span carries diagnostics even when consensus is not reached
- replace hardcoded consensus phase/result/vote literals with
ConsensusSpanNames.h val constants; add val::phaseOpen/Establish/Accepted
- ConsensusReceiveTracing.h: use canonical consensus::span constants instead
of duplicate inline detail:: names
- SpanGuardFactory test: use rpc_span / consensus::span constants now that
levelization permits the dependency
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
BasicConfig.h moved from xrpl/basics/ to xrpl/config/ on develop
(PR #7095 / reorg). Phase 1b's telemetry headers still referenced the
old path, breaking a fresh compile with "BasicConfig.h: No such file or
directory". Point both telemetry includes at the new location.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Head sampling is fixed at 1.0 and not configurable; remove the
sampling_ratio line from the benchmark, full-validation, and validator
config-template sample blocks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Head sampling is fixed at 1.0 and not configurable. Remove the
sampling_ratio line and its stale tuning comment; point volume reduction
at collector-side tail sampling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Head sampling is fixed at 1.0 and not configurable. Drop sampling_ratio
from the docker sample/test configs and the data-collection reference;
rewrite the span-metric sampling caveat and volume guidance to point at
collector-side tail sampling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Head sampling is now fixed at 1.0 and not configurable. Drop the
sampling_ratio config row and redirect volume-reduction guidance to
collector-side tail sampling. trace_peer=0 stays as the high-volume opt-out.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The telemetry test module compiles ValidationTracker.cpp (from src/xrpld/)
into xrpl_tests and includes <xrpld/telemetry/ValidationTracker.h>. The
single-binary test scheme dropped the src/ include directory that the old
per-module target carried, breaking the build. Restore it on the telemetry
branch of the test CMakeLists.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
sampling_ratio is no longer parsed (head sampling fixed at 1.0), so remove
the parse/clamp assertions and the sampling_ratio_clamped test. Update the
default-tracePeer expectations to TRUE to match the enabled-by-default change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Document that head sampling is intentionally fixed at 100% and no longer
exposes a sampling_ratio config knob. A per-node ratio let nodes make
divergent keep/drop decisions for the same distributed trace, producing
broken/partial traces; pinning at 1.0 with a ParentBased sampler keeps
decisions coherent across the network. Volume reduction is delegated to
collector-side tail sampling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update the docker/telemetry sample config and testing guide to set
trace_peer=1, matching the new on-by-default behavior and the other
trace categories already enabled in these all-categories sample profiles.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Update the Peer Spans note, the known-gaps caveat, and the trace category
toggle table to show trace_peer enabled by default. The Production Setup
example keeps trace_peer=0 as a deliberate high-volume opt-out alongside
sampling_ratio=0.01.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reflect the new on-by-default behavior in the config defaults table. The
volume-reduction guidance (High memory usage, Production mainnet tuning)
intentionally keeps trace_peer=0 as an opt-out recommendation for
high-traffic deployments.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Flip the tracePeer Setup default and the trace_peer config parser
default from off to on, and update the example config and build doc to
match. Peer spans record only peer_id (a node-local numeric connection
id) plus trust/ledger metadata — no IP addresses or public keys — so
the privacy concern behind disabling it does not apply. The high-volume
characteristic is retained in the docs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Flip trace_peer default false->true across the Phase-1a plan docs and
correct the rationale: peer spans record only peer_id (numeric local
connection id) plus trust/ledger metadata, never IP addresses or public
keys, so the 'includes addresses' caveat was inaccurate. The high-volume
note is retained.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The txq.enqueue span sets tx_hash, tx_type, and txq_status on every
code path (TxQ.cpp:746/748/751), but fee_level_paid and
required_fee_level are set only on the fee-evaluated path
(TxQ.cpp:895-898), which is reached after the rejected and
applied_direct early exits. They are therefore not guaranteed on every
txq.enqueue span, so requiring them caused the validation harness to
fail whenever a txq.enqueue span took an early-exit path.
Remove the two conditional attributes from required_attributes and
document why in the span note.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Several close-time panels showed raw codes/booleans and unreadable
time-bar-charts. Relabel and re-visualize them so each reads clearly:
- Close-Time Proposal Spread (was "Close Time Bin Distribution"):
corrected a wrong description (it claimed per-node proposals; the
metric is the number of DISTINCT close-time positions peers proposed
per round, rawCloseTimes.peers.size()). Converted the time-barchart
(unreadable timestamp axis) to a horizontal bar gauge summing rounds
per distinct-position count.
- Consensus Outcome Distribution: renameByRegex maps the raw
consensus_state codes to human labels (yes->Agreed,
moved_on->Moved On (partial), expired->Expired (timeout), no->No
Consensus); value mappings alone do not relabel pie legends.
- Close-Time Agreement Rate (was "Close Time Agreement"): legend
relabelled from "close_time_correct=true/false" to Agreed/Disagreed.
- Close-Time Resolution Change (was "Close Time Resolution Direction"):
converted to a bar gauge; increased/decreased/unchanged relabelled to
Coarser (more disagreement) / Finer (better agreement) / Steady.
All four verified by rendering the panels to PNG via the Grafana
image renderer.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
"Close Time: Raw Proposals" and "Close Time: Effective / Quantized"
plotted the absolute close time (Ripple-epoch seconds), which is an
ever-rising line with no analytical value. There is no clean way to make
them useful: the values are Ripple-epoch seconds (not Unix ms, so date
units misrender), TraceQL metrics cannot do the epoch offset or an
inter-ledger-gap subtraction in-query, and Grafana calculateField
transforms break on these grouped Tempo metric frames (verified by
render: "No data").
Remove both. The useful consensus-timing signals are already covered:
time-to-consensus (round_time_ms), rounds per ledger (establish_count),
previous round time, and the close-time vote-bins / resolution-direction
/ bin-distribution panels remain. Re-tile and re-id (22 panels).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With validation now passing 133/133, the only remaining job failure was the
regression gate flagging 4 timing "regressions". Two compounding causes:
1. Stale baseline: the committed baseline was captured (2026-04-24) under the
old, lighter workload — before the new txq-burst phase (60 TPS) existed. The
heavier per-ledger work genuinely raises ledger.build / tx.apply /
ledger.validate / acceptLedger timings, so every run regressed against it.
Refreshed the baseline from the latest CI-measured timings (same workload).
2. Histogram quantization: SpanMetrics latency buckets are
[1,5,10,25,...]ms, so a sub-millisecond quantile near a low-end boundary can
jump a full bucket (1ms->5ms) between runs with no real change. The old
absolute bounds (2-5ms) were narrower than one bucket width, so that jitter
tripped the gate. Widened the default span bounds to 10-15ms (~2 low-end
buckets) and pct to 50%, and the job_queue running bound to 20ms, to tolerate
quantization noise while still catching genuine multi-bucket regressions. The
consensus.* overrides (tight pct, large abs) are unchanged.
The refreshed baseline also picks up real rpc.ws_message timings (previously null
under the phantom rpc.request key).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Number of rounds is an integer, but avg_over_time(establish_count)
produced fractional values (2.1). Switch the Rounds panel to
count_over_time() by (span.establish_count): one integer series per
round count (1/2/3...), showing how many ledgers needed that many
establish rounds — the meaningful distribution, inherently integer
(decimals=0).
Apply dashboard rule 9: panels with a right-side table legend take full
width. Widen "Consensus Rounds per Ledger" and "Consensus Outcome
Distribution" to w=24 and re-tile the dashboard.
Verified via the Grafana proxy: rounds=2 dominates (~11-12 ledgers),
rounds=3 occasional.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Phase 9 introduces the ledger.acquire span (InboundLedger fetch) that phases 7-8
do not have, so the forward-merged 09-data-collection-reference inventory is
extended here:
- §1.1: add ledger.acquire to the Ledger span table.
- §1.2: add its attributes (acquire_reason, timeouts, peer_count, outcome) and
note it also sets ledger_seq; bump the span count.
Also fix two stale StatsD metric references in the Peer Quality dashboard
(xrpld-peer-quality.json): rippled_Peer_Finder_Active_{Inbound,Outbound}_Peers
-> xrpld_Peer_Finder_* to match the xrpld_ metric prefix the rest of the stack uses.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The consensus duration panels plotted span wall-clock
(traces_span_metrics_duration_milliseconds), which is ~3-8 ms of
instrumentation overhead, not the real consensus time (~3000 ms). And
the close-time value panels plotted an ever-rising absolute epoch line.
Rework them to answer the actual operational questions, all from
attributes that already exist on the consensus spans:
- Time to Reach Consensus (p50/p95) and Average Time to Reach Consensus:
round_time_ms on consensus.accept — the wall-clock to agree a ledger.
- Consensus Rounds per Ledger (Establish Count): avg and max of
establish_count on consensus.establish — how many proposal rounds it
took to converge (1 = first proposal).
- Previous Round Time per Ledger: previous_round_time_ms on
consensus.round.
Reorder the dashboard into an investigation flow: health/throughput ->
time-to-consensus and rounds -> ledger close/apply timing -> close-time
detail -> failures/mode/mismatch. Assign stable sequential panel ids.
Verified each query returns data via the Grafana datasource proxy
(p95 ~4096 ms, avg ~2825 ms, rounds ~2).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The §1 span and attribute inventory had regressed to an older 16-span snapshot
that uses the pre-2026-05-13 dotted attribute keys, while phase-7's code emits
~36 spans with bare/underscore attribute keys. The §Data Flow Overview and §2
System Metrics sections (native OTLP transport — phase-7's migration) were already
correct and are left unchanged.
- §1.1: expand the span inventory to the full surface — add gRPC (grpc.<MethodName>),
TxQ (txq.*), PathFind (pathfind.*), and the full consensus set (round/phase.open/
establish/update_positions/check/mode_change/proposal.receive/validation.receive).
Fix the phantom rpc.request -> rpc.http_request, add rpc.ws_upgrade. No grpc.request,
no pathfind.rank, no ledger.acquire (the latter is added in phase-9, not yet present here).
- §1.2: convert every span-attribute key from dotted xrpl.<domain>.<field> to the
bare/underscore form. The sole span-attr dotted exception is xrpl.ledger.hash on
peer.validation.receive (shared constant); consensus.validation.send uses bare
ledger_hash. Resource attrs xrpl.network.id/type stay dotted. Fix tx_count/tx_failed
placement (on tx.apply, not ledger.build). Add attribute tables for the new families.
- §1.3: list the full set of spanmetrics dimension labels (bare keys, from the collector
config) instead of the stale xrpl_rpc_command-style names.
- §4/§5: convert Tempo TraceQL and PromQL examples to the bare attribute/label forms.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The print-env CI fix let the Telemetry Stack Validation job build and run the
workload harness end-to-end for the first time. It reported 129/136 checks
passing; this commit fixes the 7 real failures plus a latent regression-gate bug.
Validation-suite fixes (verified against the CI run's actual emission + live node):
- expected_metrics.json: the beast::insight job-depth gauge is `xrpld_jobq_job_count`,
not `xrpld_job_count` (the latter is a Phase 9 OTel counter). Reverted the prior
rename. Removed the statsd_histograms block (`xrpld_rpc_time`/`xrpld_rpc_size`):
these RPC timers do not emit under the WS workload (0 series in CI).
- expected_spans.json: `tx_status` is only set on suppressed/known-bad receives, so
it is no longer a required attribute of every `tx.receive`. Marked `pathfind.compute`
and `pathfind.discover` optional and the `pathfind.request -> pathfind.compute`
hierarchy as skip — the self-to-self XRP probe returns before computing paths in a
fresh cluster with no liquidity, so only `pathfind.request` fires.
Regression-gate bug (telemetry-validation.yml "Print regression summary"):
- `jq -e` exits non-zero when its filter result is boolean false — the normal case
for a populated (non-placeholder) baseline — which was misreported as
"Failed to parse baseline JSON" and failed the job. Dropped `-e` (kept `-r`) so a
non-zero exit genuinely means malformed JSON.
The optional-span handling and regression comparison both worked correctly in the
CI run (txq.* / pathfind.update_all skipped-when-absent, 0 regressions detected).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Apply the dashboard guidelines to the five close-time panels:
- Axis labels (Title Case) on every panel: "Close Time (Ripple
Seconds)" for the value panels, "Count / Milliseconds" for vote
bins/resolution, "Rounds in Window" for the count panels.
- Human-readable legends with the dimension in brackets per the legend
convention: "Raw Close Time [{{resource.service.instance.id}}]",
"Effective Close Time [...]", "Resolution Direction
[{{span.resolution_direction}}]", "{{span.close_time_vote_bins}} Vote
Bins" — replacing the bare label tokens.
- Unit "none" (plain number): the close-time values are Ripple-epoch
seconds and TraceQL metrics cannot offset them to a wall-clock unit,
and the others are counts/ms on a shared axis.
Verified rendered values against raw spans: close times ~833,998,8xx,
resolution 10000 ms, vote bins 1/2/3 — all correct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
TraceQL metrics queries default to a 3h max range
(query_frontend.metrics.max_duration), so a dashboard set to a longer
window failed with "range ... exceeds 3h0m0s". Add a query_frontend
block raising it to 168h, matching the search max_duration, so the
consensus close-time panels work at 6h/12h/24h ranges.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Raw Proposals and Effective/Quantized panels rendered nameless
series: their legendFormat used {{service.instance.id}}, but the
TraceQL metrics query groups by resource.service.instance.id and Tempo
returns that full key as the series label. The legend token did not
match any label, so each series showed blank. Use the matching
{{resource.service.instance.id}} token.
Verified via the Grafana datasource proxy that all six close-time panels
now return correctly-labelled series.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Raw Proposals and Effective/Quantized panels showed wrong values
(e.g. 759M, 852M, even 0) against a true value of ~834M. Cause:
quantile_over_time bucketizes into an exponential histogram tuned for
duration distributions, so it cannot represent large absolute integers
(Ripple-epoch seconds) accurately.
Switch both panels to avg_over_time, which returns the correct value
(verified ~833,996,7xx matching the raw span attribute). Average is also
the semantically right aggregation here: close time is a single agreed
value per consensus round, not a latency distribution, so a median was
never meaningful.
Set the unit to none rather than seconds: the value is Ripple-epoch
seconds (Unix = value + 946684800) and TraceQL metrics cannot do the
offset arithmetic in-query, so a duration unit would misrender it.
Clarify in the description that the absolute level tracks wall-clock and
the useful signal is per-node spread / raw-vs-effective gap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The five Close Time panels still rendered "No Data" after the metrics
rewrite. Root cause: each query carried
`span.close_time_correct=~"$close_time_correct"`, but close_time_correct
is a boolean span attribute and TraceQL's regex match (=~) against a
bool matches nothing in a metrics query, so every panel returned an
empty series set (HTTP 200, {"series":[]}).
Remove that filter clause. The panels do not break down by
close_time_correct, so dropping it restores data without losing any
dimension. The $node filter (a string attribute) is unaffected and
stays.
Verified via the Grafana datasource proxy that all six targets now
return series.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire the two previously-registered-but-never-incremented validation
counters to ValidationTracker's gross lifetime tallies, exported as
monotonic ObservableCounters. New gross atomics count each ledger once at
first classification and are never adjusted on late repair, keeping the
_total counters monotonic and additive (agreements_total + missed_total ==
ledgers reconciled); the repair-aware windowed view stays on the existing
xrpld_validation_agreement gauge. The validator-health dashboard panels
that already query these names now render data instead of "No data".
Also de-stale 09-data-collection-reference.md: §5b documented flat metric
names (xrpld_cache_SLE_hit_rate, ...) that the code never emits — it emits
labeled gauges (xrpld_cache_metrics{metric="SLE_hit_rate"}). Replace the
stale flat-name tables with a pointer to the canonical labeled section,
reconcile the contradictory headline counts, and correct xrpld_job_count
to its real exported name xrpld_jobq_job_count.
Adds two GTests asserting gross tallies stay frozen on repair while net
totals move, plus the additive invariant.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Phase 10 validation harness had drifted from the code's recording surface
and the telemetry-validation CI job was failing before it could build.
CI fix (telemetry-validation.yml):
- Replace nonexistent local action ./.github/actions/print-env with the remote
XRPLF/actions/print-build-env (the build-xrpld job failed in 56s on this).
- Sync prepare-runner and upload-artifact action SHAs to the canonical workflow.
Recording-surface reconciliation (docker/telemetry/workload/):
- Migrate span attributes from dotted xrpl.<domain>.<field> to the bare/underscore
form introduced by the 2026-05-13 span-attr naming redesign (tx_hash, peer_id,
ledger_seq, consensus_mode, consensus_round, full_validation, quorum, ...).
Dotted xrpl.ledger.hash is retained only on peer.validation.receive (shared
constant), while consensus.validation.send uses bare ledger_hash.
- Fix attribute placement: tx.apply carries tx_count/tx_failed (not ledger_seq);
ledger.build carries ledger_seq/close_* (not tx_count/tx_failed).
- Replace the phantom rpc.request span with the real WS root rpc.ws_message; drop
the never-emitted duration_ms; rebuild the parent-child map accordingly.
- Add the new spans the code emits: apply-pipeline stage spans
(tx.preflight/preclaim/transactor with stage/tx_type/ter_result), txq.*,
consensus sub-spans (round/establish/update_positions/check/phase.open),
ledger.acquire, grpc.*, pathfind.*. Conditional spans are marked optional so
they are skipped (not failed) when the workload does not exercise them.
- validate_telemetry.py: service.name and Loki job label rippled -> xrpld; fix
PARITY_SPAN_ATTRS (rename the 4 real attrs, drop the 3 that are metrics not span
attrs); add optional-span handling that skips missing optional spans while still
validating attributes when present.
- expected_metrics.json: rippled_ -> xrpld_ on all beast::insight/overlay metrics,
xrpld_job_count, the 15 on-disk xrpld-* dashboard UIDs, and the real bare
spanmetrics dimension labels.
- regression-metrics.json + baseline-timings.json: rpc.request -> rpc.ws_message.
Metrics pipeline fix:
- Switch node [insight] config from server=statsd/prefix=rippled to server=otel +
/v1/metrics endpoint + prefix=xrpld across run-full-validation.sh,
xrpld-validator.cfg.template, benchmark.sh and the workload compose. The
collector has no StatsD receiver, so system metrics only reach Prometheus over
OTLP.
Synthetic load for new spans:
- Add ripple_path_find to the RPC load generator (drives pathfind.* spans).
- Add a high-TPS txq-burst workload phase to force fee escalation (drives txq.*).
All facts verified against the *SpanNames.h headers and a live xrpld node +
collector (Tempo service.name=xrpld, tx.preflight attrs [stage,ter_result,tx_type],
279 xrpld_ Prometheus metrics and zero rippled_).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The five Close Time panels (Raw Proposals, Effective/Quantized, Vote
Bins & Resolution, Resolution Direction, Bin Distribution) rendered
empty: they used TraceQL `| select(attr)`, which returns a trace list
that a timeseries/barchart panel cannot plot.
Enable TraceQL metrics in Tempo and rewrite the panels to use it:
- tempo.yaml: add the local-blocks processor to the metrics generator so
recent blocks are queryable via /api/metrics/query_range. Set
filter_server_spans=false because the consensus spans are
SPAN_KIND_INTERNAL (the default keeps only server spans, so attribute
aggregations over internal spans returned nothing), and
flush_to_storage=true with a traces_storage path so query_range can
read the flushed blocks.
- consensus-health.json: replace each panel's select() with a metrics
query — quantile_over_time on the integer close-time attributes,
avg_over_time for vote bins / resolution, and count_over_time by the
resolution_direction and vote-bin dimensions. Set the raw/effective
panels' unit to seconds (the values are Ripple-epoch seconds, which
dateTimeFromNow rendered with the wrong epoch).
Verified the query forms compile and return series against live internal
spans; the close-time series populate once the node reaches full sync.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Current Job Latency gauge sat at the bottom of the Job Queue
Analysis dashboard; per the dashboard guideline gauges belong at the
top. Move it to the first row and reflow the remaining panels below it.
Also assign explicit sequential panel ids so deep links stay stable.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Correct the width rule from the previous layout commit. Full width
(w=24) is now applied ONLY to timeseries panels whose legend is a
right-side table, since those legends need the horizontal room. Panels
with default/bottom legends, pie charts, and the heatmap return to half
width. This narrows "Transaction Receive vs Suppressed" and "TxQ
Enqueue Rate by Transaction Type", which were wrongly widened.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make transaction-overview deep-links stable and improve readability:
- Assign explicit sequential panel ids (1..20) so viewPanel=panel-N
URLs stay pinned to the same chart across edits. Previously ids were
unset and Grafana auto-assigned them by array position, so any
reorder silently repointed bookmarks.
- Move the single-value stat panel (Transaction Apply Failed Rate) to
the top row.
- Lay out in three topic sections (Processing, Apply Pipeline, Queue).
Within each, timeseries with a breakdown dimension (tx_type, stage,
ter_result, suppressed) take full width so their right-side table
legends are readable; single-series panels, pie charts, and the
heatmap stay half-width and pair up.
All six template variables already default to All (includeAll + multi);
no change needed there.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Apply the dashboard legend convention across all panels now that the
P50 series have been removed (P95-only):
- Drop the redundant "P95 " / "P50 " prefix; the panel title already
states the percentile.
- Put every filter/dimension value inside [] comma-separated, ending
with exported_instance, e.g. "AMMDeposit [Preclaim, xrpld-mainnet]".
- Add exported_instance to the by() clause and legend of the three
panels that filtered on $node but omitted it (Transaction Rate by
Type, Transaction Results by Type, TxQ Accept Status), so per-node
series are produced.
- Title-case the stage value for display via label_replace in the four
apply-pipeline panels; the span attribute stays lowercase
(preflight/preclaim/apply) since legendFormat cannot change case.
- Cap tooltip maxHeight at 500 on every panel.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The existing apply-pipeline panels show latency by stage (all types
combined) or by type (single span). Neither answers "for a given
transaction type, which stage dominates its latency". Add a p95 panel
grouped by both tx_type and stage, filterable via the $tx_type and
$stage variables. Both dimensions already exist in spanmetrics, so no
collector change is needed. Reflow the section so the full-width
failure panel sits below the new full-width panel.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire the apply-pipeline stage spans (tx.preflight, tx.preclaim,
tx.transactor) added on phase-3 through the observability stack so the
spanmetrics connector produces per-stage RED metrics without any native
instruments.
- collector: add the `stage` dimension to the spanmetrics connector so
the three stages split into separate metric series (3 bounded values).
- dashboard: add a "Tx Apply Pipeline" section to transaction-overview
with rate, p95 latency, and failure-rate panels grouped by stage, plus
a `stage` template variable. Panels follow the existing config (node
filter, exported_instance legends, Title Case, axis labels).
- The failure panel filters ter_result != tesSUCCESS rather than span
status, because a failing ter code completes the span normally — only
thrown exceptions set an error status. This matches the existing
"Transaction Results by Type" panel convention.
- docs: document the spans, attributes, and stage dimension in the data
collection reference and runbook, including the sampling caveat that
span-derived metrics inherit tracer head-sampling and undercount at
sampling_ratio < 1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The tx.transactor span covered only the apply stage; preflight and
preclaim had no telemetry, so a transaction that hard-failed those
stages produced no apply-pipeline span and per-stage latency/failure
was invisible.
Add tx.preflight and tx.preclaim spans in applySteps.cpp via a
makeStageSpan() helper using SpanGuard::hashSpan, so all three stages
share a deterministic trace_id derived from txID[0:16] even though they
run sequentially and often cross-thread. Each span carries stage,
tx_type, and ter_result; exceptions are recorded as tefEXCEPTION before
the public wrappers map them. The type lookup is guarded behind the
span-active check so it costs nothing when tracing is off.
Add a stage="apply" attribute to the tx.transactor span and move its
three hardcoded attribute strings to a new library-safe header
include/xrpl/tx/detail/TxApplySpanNames.h, which mirrors the daemon-side
TxSpanNames.h strings so the collector spanmetrics connector aggregates
both span sets under one dimension set.
A constants-contract test pins the span-name, attribute-key, and
stage-value strings; span content stays covered by the docker
integration test, as the rest of the telemetry suite is.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reduced from 30 to 7 filters: service.instance.id, name, status, command,
tx_hash, tx_type, ledger_hash. Full attribute inventory is in
OpenTelemetryPlan/09-data-collection-reference.md §4; TraceQL autocomplete
covers the rest.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
From the code-review pass:
- transaction-overview.json: the tx.process and tx.transactor latency-by-type
panels used lowercase legends (p95/p50) without the per-node dimension. Use
Title Case (P95/P50), add exported_instance to the by() clause, and include
[{{exported_instance}}] in the legend, per the dashboard legend convention.
- consensus-health.json: panel descriptions still referenced the old dotted
attribute names (xrpl.consensus.mode, xrpl.ledger.seq) after the A1 rename;
update them to the bare emitted names (consensus_mode, ledger_seq).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
handleMismatch() recorded the "consensus_txset" reason but then fell through
to the transaction-level comparison, which also recorded a reason
("same_txset_diff_result" / "different_txset"). A single mismatch with
disagreeing consensus tx-set hashes therefore incremented
xrpld_ledger_history_mismatch_total twice across two reason labels, so the
sum over reason exceeded the real mismatch count.
The consensus tx-set hash disagreement is the root cause; return after
recording it so each mismatch contributes exactly one reason.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The A1 fix (xrpl_consensus_mode -> consensus_mode) was applied on phase-6, but
the phase-8->phase-9 merge conflict resolution for consensus-health.json took
phase-9's pre-fix panel base, silently reintroducing all 11 stale
xrpl_consensus_mode label references (the spanmetrics label that is never
populated — see the original A1 commit).
Re-apply the label fix on phase-9: xrpl_consensus_mode -> consensus_mode in
every panel expr, legendFormat, and the $consensus_mode template variable's
label_values() query. The Grafana variable name $consensus_mode is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Visualise the metrics added in this series:
- consensus-health: "Ledger History Mismatch Rate by Reason"
(xrpld_ledger_history_mismatch_total by reason — fork diagnostics)
- fee-market: "Queue Abandonment Rate (Expired)" and "Queue Admission
Rejections (Dropped)" (xrpld_txq_expired_total / dropped_total)
- peer-network: "Reduce-Relay Peer Selection" and "Reduce-Relay Missing-Tx
Frequency" (xrpld_reduce_relay_metrics)
- system-node-health: "Ledger Acquire Duration" and "Ledger Acquire Rate by
Outcome" (ledger.acquire span)
otel-collector-config.yaml: add outcome and acquire_reason spanmetrics
dimensions so the ledger.acquire outcome breakdown populates.
All panels follow the existing template: $node filter, exported_instance in
legends, Title Case, axis labels.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the new synchronous counters (ledger_history_mismatch_total{reason},
txq_expired_total, txq_dropped_total{reason}) and the reduce-relay observable
gauge to the ASCII ownership diagram in the MetricsRegistry header so the
documented instrument inventory matches the code.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The transaction reduce-relay subsystem (selected vs suppressed peers,
feature-disabled peers, missing-tx frequency) was computed in OverlayImpl's
TxMetrics but only surfaced via the get_counts JSON RPC — invisible to
Prometheus/Grafana, despite being the central efficiency KPI for the feature.
Add an observable gauge xrpld_reduce_relay_metrics{metric} that reads
Overlay::txMetrics() and parses its rolling-average fields:
- selected_peers (txr_selected_cnt)
- suppressed_peers (txr_suppressed_cnt)
- not_enabled_peers (txr_not_enabled_cnt)
- missing_tx_freq (txr_missing_tx_freq)
The JSON values are decimal strings (std::to_string), parsed via std::stoll —
the same JSON-reading pattern as registerNodeStoreGauge. No new Overlay
accessor or core-interface change required.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
InboundLedger drives ledger back-fill and fork recovery with timeout/retry
logic (kLedgerTimeoutRetriesMax = 6), but emitted only a global ledger_fetches
counter — sync/recovery cost was a telemetry blind spot.
Add a ledger.acquire span that wraps the acquisition lifecycle:
- Started in InboundLedger::init() with ledger_seq and acquire_reason
(history / consensus / generic, mirroring InboundLedger::Reason).
- Finalized in InboundLedger::done() with outcome (complete / failed),
timeouts, and peer_count, then reset so the span duration is exported.
Held as a std::optional<SpanGuard> member (same pattern as RCLConsensus
roundSpan_). New op/attr/val constants added to LedgerSpanNames.h. Compiles to
a no-op when telemetry is disabled via the SpanGuard fallback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The transaction queue had no metric for demand that leaves or never enters the
queue, so fee-underpayment abandonment and admission-control rejection were
invisible (distinct from jq_trans_overflow, which is the job queue).
Add two synchronous counters via MetricsRegistry:
- xrpld_txq_expired_total — incremented in TxQ::processClosedLedger() for each
queued transaction removed because its LastLedgerSequence passed (submitters
who under-bid the escalating fee and were never included).
- xrpld_txq_dropped_total{reason} — incremented in TxQ::apply() at the
queue-full admission-control returns (reason="queue_full").
Both reach MetricsRegistry via the Application& parameter already passed to
these methods; calls are null-guarded so they no-op when telemetry is disabled.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
LedgerHistory::handleMismatch() already classifies a built-vs-validated ledger
mismatch (prior ledger, close time, consensus tx set, same/different tx set),
but only bumped a single untyped beast::insight counter — the reason was
dropped. Fork diagnosis was therefore a log-grep exercise.
Add a labeled OTel counter so the mismatch reason is a queryable time series:
- MetricsRegistry: new ledgerHistoryMismatchCounter_ + incrementLedgerHistoryMismatch(reason)
- LedgerHistory: record one reason per classification branch (unknown,
prior_ledger, close_time, consensus_txset, same_txset_diff_result,
different_txset). Reaches MetricsRegistry via the existing app_ reference.
The existing beast::insight mismatchCounter_ is left intact.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The xrpld_peer_quality{metric="peers_insane_count"} gauge was hardcoded to 0.0
with a TODO, leaving the "Insane/Diverged Peers" panel permanently empty.
PeerImp::json() already exposes the peer's tracking state via the "track"
field (set to "diverged" when tracking_ == Tracking::Diverged). The peer-quality
callback already iterates peer->json() for latency and version, so count peers
whose "track" field equals "diverged" in the same loop — no change to the
abstract Peer interface required.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A phase-8->phase-9 merge (a675897aaf) duplicated the "Consensus Outcome
Distribution" and "Consensus Failures Over Time" panels: both appeared twice
with byte-identical queries (verified ignoring gridPos). The pair existed once
on phase-6/7/8 and became two on phase-9 only, so the duplication originated
in phase-9's own merge history.
Remove the second (lower) copy of each and re-stack panel y-positions with no
gaps. The single retained copy keeps the original y=64 row.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both metrics are already emitted and live in Prometheus but were not fully
visualised.
- Fee Market (xrpld-fee-market.json): "Load Factor Attribution (Stacked
Components)" — stacks load_factor_fee_escalation / fee_queue / local / net /
cluster so an operator can see which component drives the effective fee. The
existing panels showed the aggregate only.
- Validator Health (xrpld-validator-health.json): "Agreement % (7d)" and
"Agreements vs Missed (7d)" — the xrpld_validation_agreement gauge already
observes agreement_pct_7d / agreements_7d / missed_7d, but the dashboard only
plotted 1h and 24h windows.
Panels follow the existing template: $node filter, exported_instance in legends,
Title Case, axis labels.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The grpc.{Method} spans (GRPCServer.cpp) and pathfind.* spans (PathRequest.cpp)
are emitted but had no dashboard coverage. The existing RPC & Pathfinding
dashboard only plotted StatsD timers. Add span-derived rows:
- gRPC Request Rate by Method (grpc.* by method)
- gRPC Latency P95 by Method
- gRPC Error Rate by Status (by grpc_status)
- Pathfinding Compute Duration (pathfind.compute p95/p50)
- Pathfinding Request & Discovery Rate (pathfind.request / pathfind.discover)
otel-collector-config.yaml: add method, grpc_role, grpc_status spanmetrics
dimensions (bounded value sets). Add a $grpc_method template variable so the
gRPC panels can be filtered by method, consistent with the dashboard filter
conventions.
Note: these spans populate only when the node serves gRPC / pathfinding
traffic; they are correct but not exercised by the current health-check
workload (they will be covered by the Phase 10 workload generator).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The consensus state-machine and TxQ lifecycle spans are emitted by the code
and present in Prometheus, but no panel visualised them. Add panels keyed on
those span_names (verified live) plus the low-cardinality dimensions needed to
break them down.
Consensus Health (consensus-health.json) — new rows:
- Consensus Round Duration (full round, p95/p50, mode-filterable)
- Consensus Phase Duration (open vs establish breakdown)
- Position Update Duration (update_positions p95/p50)
- Consensus Stall Rate (consensus.check by consensus_stalled)
- Consensus Mode-Change Rate by Target Mode (mode_change by mode_new)
Transaction Overview (transaction-overview.json) — new rows:
- TxQ Enqueue Rate by Transaction Type (txq.enqueue by tx_type)
- Queue Bypass Ratio (txq.apply_direct vs txq.enqueue)
- Queue Accept (Drain) Duration per Ledger (txq.accept p95/p50)
- Queue Cleanup Rate (txq.cleanup expired entries)
otel-collector-config.yaml — add spanmetrics dimensions for the lifecycle
breakdowns: mode_new, consensus_stalled, consensus_phase, consensus_result
(all bounded value sets, safe as Prometheus labels).
All new panels follow the existing dashboard template: $node filter,
exported_instance in every legend, Title Case, axis labels, row layout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The spanmetrics connector dimension was `xrpl.consensus.mode`, but the code
emits the span attribute under the bare key `consensus_mode` (matching every
other dimension after the Phase 6 rename). The mismatch left the
`xrpl_consensus_mode` Prometheus label empty, so the Consensus Health
"Consensus Mode Over Time" panel and the `$consensus_mode` template variable
(which filters every panel) matched no live series.
- otel-collector-config.yaml: dimension `xrpl.consensus.mode` -> `consensus_mode`
- consensus-health.json: 11 label refs `xrpl_consensus_mode` -> `consensus_mode`
(the `$consensus_mode` Grafana variable name is unchanged)
- telemetry-runbook.md: refresh the stale spanmetrics label table to the bare
names actually emitted (command/rpc_status/consensus_mode/local/
proposal_trusted/validation_trusted), fix dotted->bare attribute names in
span tables and TraceQL examples (tx_hash, ledger_seq, consensus_round_id,
consensus_ledger_id, consensus_round, tx_id event attr), correct the
consensus_round_id query to int (not quoted string), and fix the
load_type value query ("exception_rpc" -> "exceptioned RPC").
Verified against the live stack: Tempo span tags confirm bare attribute keys
(consensus_mode, ledger_seq, tx_hash, ...); the populated xrpl_consensus_mode
series in Prometheus is stale retained data from an older build.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
beast::insight metrics exported via OTLP carried no exported_instance
label because [insight] omitted service_instance_id (only [telemetry]
set it). Every system-* dashboard filters insight metrics with
exported_instance=~"$node", and the $node template variable is sourced
from label_values(..., exported_instance) — so with the label absent,
$node was empty and all insight-backed panels showed no data.
Add service_instance_id to [insight] in both telemetry configs, matching
the [telemetry] value (xrpld-mainnet / xrpld-devnet). CollectorManager
already reads this key and passes it to OTelCollector, which sets the
service.instance.id resource attribute.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The phase-9 NodeStore I/O totals, write-load/read-queue, read-threads,
and object instance-count panels rendered large cumulative values with
unit "none". Switch to "short" for readable abbreviation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Count and message-volume panels (operating-mode transitions, job queue
depth, network/overlay message totals, getobject message counts) used
unit "none", rendering large values as raw unscaled numbers. Switch to
"short" so Grafana abbreviates (e.g. 1.5 Mil) for readability.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The system-* dashboards queried the legacy StatsD rippled_ prefix, but the
node now emits beast::insight metrics via native OTLP under the xrpld_
prefix (config: [insight] server=otel, prefix=xrpld). All queries returned
no data.
Migration (names derived from C++ beast::insight registrations, not live
Prometheus, since a syncing node does not emit every metric yet):
- rippled_ -> xrpld_ prefix across all panel queries and template variables
(including the $node variable query, which broke the whole dashboard filter)
- Histogram Event instruments export with unit ms, so bare _bucket becomes
_milliseconds_bucket: ios_latency, rpc_time, rpc_size, pathfind_fast/full
- Job-type metrics were StatsD summaries (label quantile="$quantile"); on the
OTLP path they are histograms. Converted those queries to
histogram_quantile($quantile, rate(xrpld_<job>_milliseconds_bucket[5m]))
and added the previously-undefined $quantile template variable
- Per-job-type detail panels: __name__ regex now matches _milliseconds_bucket
No panels removed. Panels for metrics not yet emitted (e.g. warn/drop, or
job types the syncing node has not run) show no data until the path executes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
P100 from a histogram is degenerate — it always returns the upper bound of
the highest populated bucket (a single slow outlier pins it to the top
boundary), producing a flat line. Revert to meaningful quantiles:
- Job Queue Wait Time / Job Execution Time: p75 (typical) + p99 (tail)
- Per-Job-Type / Per-Method: p99
- Added gauge panels showing current p99 with green/yellow/red thresholds
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The three duration heatmaps (transaction, consensus accept, RPC latency)
had an axisLabel of "Duration (ms)" but no unit code, so y-axis tick
values rendered unscaled. Set unit=ms on both the yAxis options and
panel defaults so buckets display as proper millisecond values.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Grafana does not recognize 'us' as a unit code, so microsecond values
rendered as raw numbers with a plain 'us' suffix (no scaling). The
correct code is 'µs'. Affects job-queue and OTel RPC latency panels
backed by *_duration_us histograms.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds template variables $tx_type, $ter_result, $txq_status to the
Transaction Overview dashboard. All relevant panels now respect these
filters, enabling operators to drill into specific transaction types
or result codes.
Changes:
- Panel 2 renamed to "Transaction Processing Latency by Type" (now
shows p95/p50 per tx_type instead of aggregate)
- Panels 1,3,4,5,7,9,12 filter by $tx_type
- Panel 10 filters by $tx_type and $ter_result
- Panel 11 filters by $txq_status
- Removed redundant "TX Processing Latency by Type (p95)" panel
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds template variables $tx_type, $ter_result, $txq_status to the
Transaction Overview dashboard. All relevant panels now respect these
filters, enabling operators to drill into specific transaction types
or result codes.
Changes:
- Panel 2 renamed to "Transaction Processing Latency by Type" (now
shows p95/p50 per tx_type instead of aggregate)
- Panels 1,3,4,5,7,9,12 filter by $tx_type
- Panel 10 filters by $tx_type and $ter_result
- Panel 11 filters by $txq_status
- Removed redundant "TX Processing Latency by Type (p95)" panel
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Shows p95 latency of tx.process span broken down by tx_type. Works for
both received and locally-processed transactions, unlike the tx.transactor
panel which requires the node to be synced and applying.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The OTel C++ SDK's SetAttribute appends rather than overwrites on
in-flight spans. Setting suppressed=false as a default then overriding
to true resulted in both values appearing in the exported span.
Fix: remove the default-false set, place suppressed=false once after
the HashRouter check passes (non-suppressed path), and suppressed=true
remains only in the suppressed path.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Resolve consensus dashboard conflict and remove duplicate
consensus_state dimension in collector config.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Collector config: add tx_type, ter_result, txq_status, consensus_state,
load_type, is_batch as spanmetrics dimensions so they appear as
Prometheus labels for dashboard queries.
New dashboard panels:
- Transaction Overview: Rate by Type, Results by Type, TxQ Status (pie),
Transactor Duration p95 by Type
- Consensus Health: Outcome Distribution (pie), Failures Over Time
- RPC Performance: Resource Cost by Command, Batch vs Single
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Resolve XrplCore.cmake conflict: keep telemetry module before tx module
(correct ordering from the cascade).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wraps Transactor::operator() with a span that captures tx_type,
ter_result, and applied. This is the universal dispatch point — every
transaction flows through it, giving per-type latency breakdown.
Adds libxrpl.tx > xrpl.telemetry levelization dependency.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wraps Transactor::operator() with a span that captures tx_type,
ter_result, and applied. This is the universal dispatch point — every
transaction flows through it, giving per-type latency breakdown.
Adds libxrpl.tx > xrpl.telemetry levelization dependency.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Resolve runbook conflict: keep both phase 6 ledger/peer span tables
AND new insights/sample queries section from the enrichment work.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds comprehensive "Insights and Sample Queries" section showing operators
what questions they can answer with the newly-added span attributes:
- Transaction workflow analysis (filter by tx_type, fee, ter_result)
- TxQ health (txq_status, ledger_changed)
- RPC debugging (is_batch, request_payload_size, load_type)
- PathFinding performance (dest_currency, num_source_assets)
- Consensus health (consensus_state, is_bow_out, disputes_count)
- Cross-subsystem correlation examples
Also updates all span reference tables with the new attributes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds workflow-critical attributes to consensus spans:
- consensus.proposal.send: is_bow_out (identifies resignation proposals)
- consensus.accept: consensus_state (yes/moved_on/expired), disputes_count
- consensus.validation.send: ledger_hash (correlates validation to ledger)
Enables answering: "Did we reach consensus or time out?", "How many
disputes existed at acceptance?", "Which ledger did we validate?"
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The apply() function doesn't have a `using namespace telemetry` directive
(unlike processTransaction), so tx_span attrs need explicit qualification.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Enriches the tx.process span with final outcome after batch application:
- ter_result: the TER code string (e.g., "tesSUCCESS", "tecPATH_DRY")
- applied: boolean whether the transaction was included in the ledger
These attributes complete the tx.process span lifecycle — it now captures
identity (tx_type, tx_hash), intent (fee, sequence), and outcome
(ter_result, applied) for full workflow traceability.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire up span attributes that enable filtering/grouping traces by request
characteristics: batch detection, payload size, resource cost category,
command name on WS spans, and pathfinding search parameters (destination
amount/currency, source asset count).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The check-rename CI job requires all rename scripts to have been run.
The telemetry config files had 'prefix=rippled' which should be 'xrpld'.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The project-wide rename check changed the factory function name but
missed this call site in the consensus simulation test framework.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ServiceRegistry gained the pure virtual getMetricsRegistry() in phase 7
but TestServiceRegistry was never updated. Returns nullptr since tests
don't need a real MetricsRegistry.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
clang-tidy misc-include-cleaner requires each symbol to be provided by
a directly-included header. Replace the convenience trace/context.h
(which only provides GetSpan/SetSpan) with the specific headers for
kSpanKey, holds_alternative, get, shared_ptr, Span, and span.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two stray "rippled" tokens introduced by 43258e8d ("docs(telemetry):
add secure-OTel pipeline analysis…") were caught by check-rename in
CI. Re-run docs.sh to convert them to xrpld so the rename check
passes on PR #6425 (and downstream PR #6426 once merged up).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings in pratik/otel-phase3-tx-tracing's two commits that move all
consensus-tracing content off phase-3:
- c9521b97fe refactor(telemetry): pull consensus-tracing scope-leak
out of phase-3
- d6b101069e refactor(telemetry): remove consensus tracing from
phase-3
Phase-4 already owns this content (with the consensus_span -> cons_span
namespace rename, round/accept/proposalSend/validationSend span
construction, and the relocated ConsensusSpanNames.h under
src/xrpld/consensus/), plus its own evolution on top — so the merge is
resolved as a tree-identity merge:
- src/xrpld/app/consensus/ConsensusSpanNames.h: keep phase-4's
renamed/expanded version (modify/delete conflict, resolved with
--ours).
- src/xrpld/app/consensus/RCLConsensus.cpp: keep phase-4's version
with cons_span attribute calls, the trace_context inject blocks
on broadcastPropose/validate, the secure-OTel TODO, and the
full validation/round span instrumentation (content conflict,
resolved with --ours).
- src/xrpld/overlay/detail/PeerImp.cpp: keep phase-4's version with
the proposalReceiveSpan/validationReceiveSpan calls, lambda span
captures, and cons_span::attr::* setAttribute calls (content
conflict, resolved with --ours).
- src/xrpld/telemetry/ConsensusReceiveTracing.h: phase-3 deleted it,
phase-4 still uses it. Restored from phase-4 HEAD (silent
auto-deletion otherwise).
- include/xrpl/telemetry/TraceContextPropagator.h: phase-3 stripped
consensus references and the secure-OTel TODO; phase-4 still has
both. Restored from phase-4 HEAD.
- src/xrpld/telemetry/PropagationHelpers.h: phase-3 swapped the
@see ConsensusReceiveTracing.h cross-reference for TxTracing.h;
phase-4 still wants the consensus reference. Restored from
phase-4 HEAD.
Net tree change on phase-4: zero. Verified via 'git diff <pre-merge-sha>
HEAD' returning empty.
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Phase-3 (PR #6425) is scoped to transaction tracing only; consensus
tracing belongs to phase-4 (PR #6426). The previous commit on this
branch removed the namespace/attribute scaffolding c6c019ed8b leaked
into phase-3, but phase-3 still carried the consensus span construction
and trace-context propagation introduced in earlier commits
(61cb1faf8f, 93bed03d8d). Move that out too so phase-3 creates and
propagates no consensus spans of any kind.
Removed:
- src/xrpld/telemetry/ConsensusReceiveTracing.h (deleted; phase-4
owns it).
- PeerImp.cpp: remove the std::make_shared<SpanGuard>(
proposalReceiveSpan(...))/validationReceiveSpan(...) constructions
in onMessage(TMProposeSet)/onMessage(TMValidation), drop the
sp = std::move(span) lambda captures, and drop the
#include <xrpld/telemetry/ConsensusReceiveTracing.h>.
- RCLConsensus.cpp: drop the two telemetry::injectToProtobuf() blocks
that injected the active trace context into TMProposeSet (in
Adaptor::propose, after addSuppression) and TMValidation (in
Adaptor::validate, around the broadcast call). Drop the now-unused
#include of TraceContextPropagator.h and the
XRPL_ENABLE_TELEMETRY-gated include of
opentelemetry/context/runtime_context.h.
- TraceContextPropagator.h: update file-level @see comment to drop
the ConsensusReceiveTracing.h reference and to scope the
"wired into the P2P message flow via PropagationHelpers.h"
sentence to TMTransaction only.
- PropagationHelpers.h: replace the
@see ConsensusReceiveTracing.h cross-reference with
@see TxTracing.h.
Inert consensus metadata (TraceCategory::Consensus enum value,
seg::consensus constant, isCategoryEnabled/categoryToSpanKind switch
arms, the SpanGuard.h doc-comment example) is intentionally preserved
on phase-3: nothing references it after this commit, but phase-4
needs it and removing it would widen the phase-3 -> phase-4 merge
surface for no benefit.
Verified via git grep: no remaining phase-3 references to
proposalReceiveSpan, validationReceiveSpan, ConsensusReceiveTracing,
consensus_span::, consensus.proposal, or consensus.validation.
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Commit c6c019ed8b ("addressed code review comments") bundled tx-tracing
review fixes with consensus-tracing scaffolding that belongs on
pratik/otel-phase4-consensus-tracing (PR #6426). This commit lifts the
consensus-only parts back out of phase-3 so PR #6425 stays scoped to
transaction tracing. Phase-4 already carries the same content (via prior
phase-3 -> phase-4 merges) plus its own evolution on top, so nothing is
moved across — only removed here.
Removed:
- src/xrpld/app/consensus/ConsensusSpanNames.h (deleted; phase-4 owns
it).
- PeerImp.cpp: revert onMessage(TMProposeSet)/onMessage(TMValidation)
consensus-attr setAttribute calls to the hardcoded
"xrpl.consensus.{trusted,round,ledger.seq}" strings used before
c6c019ed8b. Drop the now-unused
#include <xrpld/app/consensus/ConsensusSpanNames.h>.
- RCLConsensus::Adaptor::validate(): remove the
TODO(observability/secure-OTel) block on validation trace_context.
- TraceContextPropagator.h: remove the TODO(observability/secure-OTel)
block on injectToProtobuf().
Tx-tracing parts of c6c019ed8b are intentionally untouched.
No phase-3 caller of telemetry::consensus_span:: remains; verified via
git grep. No test on phase-3 references the removed header.
Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Document the threat model and chosen hardening approach for the OTel
pipeline: mTLS to the collector as primary defense (across-network
deployment), NetworkPolicy as defense-in-depth, and source-side
validation plus per-peer rate limiting for protocol::TraceContext on
peer messages. Skips Basic Auth (wrong shape for multi-operator
fleet) and HTTP-gateway header stripping (rippled is P2P).
Wires the new doc into the master plan ToC, mermaid diagram, and
body section, plus cross-refs from the privacy section in
02-design-decisions.md and the collector config in
05-configuration-reference.md so readers reach it from natural
in-context entry points. Adds a backlink at the top of secure-OTel.md
to the master plan.
Adds 'exfiltration' and 'htpasswd' to cspell dictionary.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Drop xrpl.node.amendment_blocked / xrpl.node.server_state from telemetry
surface (constants in SpanNames.h, two filters in tempo.yaml). Operators
read the same data via server_info / server_state RPC; OTel SDK 1.18.0
cannot refresh resource attrs at runtime so resource-level emission was
not viable either.
- Namespace all pathfind span attributes under pathfind_* (underscore form
per Phase 1c rule 5). Renames in PathFindSpanNames.h and call sites in
PathRequest.cpp, PathRequestManager.cpp, plus the rule-5 retention
xrpl.pathfind.ledger_index -> pathfind_ledger_index.
- Wire pathfind_source_account / pathfind_dest_account on pathfind.request
in doPathFind / doRipplePathFind handlers (only when present + string).
- Collapse per-asset pathfind.discover / pathfind.rank spans into one
pathfind.discover hoisted around the per-source-asset loop in
PathRequest::findPaths. Span count goes from 2N to 1 per RPC call;
per-asset breakdown traded for bounded storage and cardinality. Trade-off
documented inline.
- Fix pathfind_num_paths semantics: now sums getBestPaths().size() across
the loop (paths actually returned) instead of the maxPaths input cap.
- PathRequestManager::updateAll: move span creation after the locked
requests_ snapshot, early-return when no active subscriptions exist
(avoids empty span on every ledger close), set pathfind_num_requests
= requests.size().
- Update Phase2_taskList.md and 02-design-decisions.md to match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per PR #6438 review thread r3250432621: known-command errors
(rpcTOO_BUSY, rpcNO_PERMISSION, etc.) were collapsing into a
single rpc.command.unknown span name, hiding per-command error
rates in dashboards. Same anti-pattern existed for gRPC, where
every method was bucketed under grpc.request with the method
relegated to an attribute.
- RPCHandler.cpp: doCommand error path uses cmdName as the span
suffix; the rpc_span::val::unknownCommand fallback only applies
when the request truly omits both command and method fields.
- GRPCServer.cpp: gRPC span name is now grpc.<MethodName>
(e.g. grpc.GetLedger). Method also retained as an attribute.
- GrpcSpanNames.h: drop the unused op::request constant; update
the span-hierarchy comment.
- RpcSpanNames.h: update the gRPC span diagram to match.
Dashboards on downstream phases will benefit from per-command
breakdowns without needing TraceQL attribute filters.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: semgrep-companion-app[bot] <218312740+semgrep-companion-app[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: xrplf-ai-reviewer[bot] <266832837+xrplf-ai-reviewer[bot]@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Three interrelated fixes in otel-collector-config.yaml; without them the
Phase 8 log-trace correlation pipeline is silently broken.
1. `resource/logs` processor now upserts `job: xrpld` alongside
`service.name: xrpld`. Loki 3.x OTLP ingestion renames
`service.name` to the label `service_name`, so the runbook /
integration-test queries (`{job="xrpld"} |= "trace_id="`) returned
empty. Upserting the `job` resource attribute at the collector lets
the canonical Loki label flow through unchanged.
2. `filelog` regex makes the `partition:` capture non-capturing-optional.
`Logs::format()` omits the `partition:` prefix when partition is
empty (common for framework-level log lines); the old regex required
it and silently dropped those records.
3. Timestamp parser now matches the real log format. `Logs::format()`
writes microsecond-precision timestamps like
`2026-04-15 10:30:45.123456 UTC`. The layout was
`%Y-%b-%d %H:%M:%S` — missing fractional seconds and timezone —
which failed strptime and dropped timestamps. New layout is
`%Y-%b-%d %H:%M:%S.%f` with `location: UTC`.
Also adds a block-comment documenting the real log format so the
next person to touch this doesn't re-introduce the same gaps.
- `src/test/telemetry/MetricsRegistry_test.cpp` (Beast `unit_test::suite`
format under `src/test/`) duplicates the GTest version already
maintained at `src/tests/libxrpl/telemetry/MetricsRegistry.cpp`.
Project rule (`tasks/lessons.md` §Test Format): all new tests use
GTest under `src/tests/libxrpl/`. The GTest version exercises the
same four cases (disabled construction, start/stop lifecycle, recording
no-op, destructor-calls-stop). Deleting the Beast duplicate eliminates
drift and keeps the test authoritative in one place.
- Drop the matching `test.telemetry > xrpl.basics/xrpl.core/xrpld.telemetry`
entries from `.github/scripts/levelization/results/ordering.txt`
because `xrpl.test.telemetry` (the GTest binary) retains its own
entries; the removed ones belonged to the deleted Beast suite.
- `.claude/instructions.md` was committed as a symlink to an
author-local absolute path (`/home/pratik/sourceCode/personal/Rippled/
instructions.md`) that does not exist for any other contributor or in
CI. Remove the symlink from git tracking and add `.claude/` to
`.gitignore` so future agent commits do not re-add per-developer
settings.
MetricsRegistry observable-gauge callbacks run on the OTel reader thread
and read live state from nodeStore_, overlay_, networkOPs_, ledgerMaster,
inboundLedgers, loadManager, and others. The old shutdown sequence called
metricsRegistry_->stop() AFTER all those services were already stopped,
which left a race window between each service's stop() and the final
provider_->ForceFlush() during which a callback could dereference
already-stopped service state. The try/catch guards in each callback
mitigated crashes but not reads from freed members.
- Add MetricsRegistry::detachCallbacks() that sets an atomic<bool>
callbacksDetached_ with release ordering. Idempotent.
- Guard every ObservableGauge callback entry with an acquire-load of the
same flag and return early if it is set. Covers all 15 registered
callbacks (cacheHitRate, txq, objectCount, loadFactor, nodeStore,
serverInfo, buildInfo, completeLedgers, dbMetrics, validatorHealth,
peerQuality, ledgerEconomy, stateTracking, storageDetail,
validationAgreement).
- Application::run() shutdown sequence now calls
metricsRegistry_->detachCallbacks() right after m_loadManager->stop()
and BEFORE m_shaMapStore, m_jobQueue, overlay_, grpcServer_,
m_networkOPs, serverHandler_, m_ledgerReplayer, m_inboundTransactions,
m_inboundLedgers, ledgerCleaner_, m_nodeStore, perfLog_ are stopped.
The acquire/release pair guarantees subsequent reader-thread ticks see
the detach before they dereference stopped services.
- metricsRegistry_->stop() keeps setting the flag as a belt-and-suspenders
defense in case a future caller forgets to detach first.
- Drop the misleading "No explicit RemoveCallback is needed" comment
from stop(); provider destruction alone does not beat the reader
thread to already-freed state.
The objectCountGauge callback previously discarded its state pointer
via `void* /* state */`; restore the state argument so it can access
self->callbacksDetached_ too.
File renames to match the post-docs.sh project-wide rename + the UID
rename applied in the previous commit. Five phase-9 dashboards are
affected:
- rippled-fee-market.json -> xrpld-fee-market.json
- rippled-job-queue.json -> xrpld-job-queue.json
- rippled-peer-quality.json -> xrpld-peer-quality.json
- rippled-rpc-perf.json -> xrpld-rpc-perf-otel.json
- rippled-validator-health.json-> xrpld-validator-health.json
`rippled-rpc-perf.json` is renamed to `xrpld-rpc-perf-otel.json` (rather
than `xrpld-rpc-perf.json`) to avoid colliding with the
phase-6 `rpc-performance.json` dashboard which also uses the
`xrpld-rpc-perf` UID. The new filename matches its now-unique
`xrpld-rpc-perf-otel` UID that was set in the merge commit.
Phase 4 added a span catalog in `06-implementation-phases.md` listing the
source location for each consensus span. Line numbers `Consensus.h:707`,
`RCLConsensus.cpp:232/341/492/541/900` drift on every refactor and would
become stale PR after PR. Filename alone is enough for operators to
grep — the RCLConsensus.cpp spans are already unambiguous from the span
name itself.
Follow-up to the phase-6 dashboard cleanup. The three dashboards
introduced by commit f6105ece98 (consensus-health, rpc-performance,
transaction-overview) were missed in the initial UID rename and still
carried `rippled-*` UIDs plus line-number refs in panel descriptions.
- UIDs: `rippled-consensus` -> `xrpld-consensus`,
`rippled-rpc-perf` -> `xrpld-rpc-perf`,
`rippled-transactions` -> `xrpld-transactions`, matching the
post-`docs.sh`-rename runbook and the other dashboards in this PR.
- Strip `:<line>` suffixes from `ServerHandler.cpp`, `RCLConsensus.cpp`,
`NetworkOPs.cpp`, etc. references in panel descriptions. Line numbers
drift on every refactor; the filename is enough to grep.
- Fix the Overall RPC Throughput panel: two targets filtered on
`span_name="rpc.request"` (never emitted) instead of
`span_name="rpc.http_request"` (the real emitted name). The panel
would have shown zero data until this fix.
Follow-up to the dashboard cleanup on this branch. Caught additional sites
in TESTING.md that still reference the never-emitted `rpc.request` span:
- TraceQL query examples in Step 5 "Verify traces in Tempo" now filter on
`name="rpc.http_request"` (the real emitted name).
- Expected-spans table replaces `rpc.request` with `rpc.http_request`.
- Query loop under the Prometheus verification section now iterates over
the full set of emitted RPC entry-point names
(`rpc.http_request`, `rpc.ws_upgrade`, `rpc.ws_message`, `rpc.process`).
Also drop `exporter=otlp_http` from the sample telemetry config block.
`TelemetryConfig.cpp` does not parse an `exporter` key in any phase through
Phase 8; only OTLP/HTTP is wired up, so the line is either a silently
ignored no-op or misleading documentation.
Phase-6 introduces ledger-operations, peer-network, and the five StatsD
dashboards. Align them with the rest of the chain:
- Rename dashboard UIDs from `rippled-*` to `xrpld-*` so the provisioned
UIDs match the post-rename-script documentation (`docs.sh` rewrites
.md but not .json, so the two drifted). Runbook references
`xrpld-rpc-perf`, `xrpld-transactions`, etc., now the JSON matches.
- Add the `$node` template variable + `exported_instance=~"$node"` filter
to every target in the five `statsd-*` dashboards. Mirrors the pattern
already used by consensus-health, ledger-operations, and peer-network
per the project rule that every dashboard must support per-node
filtering.
- Strip `:<line>` (and `:NN-NN` range) suffixes from C++ file references
in every dashboard panel description and in docker/telemetry/TESTING.md.
Line numbers drift on every refactor; the filename alone is enough to
grep.
- Replace stale `rpc.request` entries with the real emitted span names
(`rpc.http_request`, `rpc.ws_upgrade`, `rpc.ws_message`, `rpc.process`)
in TESTING.md so operators can copy-paste the filters and hit real
traces.
- Also drop the `:706` line ref from the `StatsDCollector.cpp` callout
in `06-implementation-phases.md`.
- RPC Spans table: `rpc.request` was documented but the code actually emits
`rpc.http_request`. Listed the actual emitted names
(`rpc.http_request`, `rpc.ws_upgrade`, `rpc.ws_message`, `rpc.process`)
and their parent/child relationship.
- Drop `:<line>` suffixes from Source File columns in both RPC and
Transaction span tables. Line numbers drift with every refactor; the
filename is enough for operators to grep.
- Summary table: replace the never-emitted `rpc.request` row with the real
entry points so `span_name=` filters in PromQL / TraceQL match.
Phase-1a plan documents advertised OTLP/gRPC on port 4317 as the default
exporter, four unparsed [telemetry] config keys, and "Phase 4a Complete"
status with exit-criteria checkboxes marked done. Every downstream branch
through Phase 5 ships only OTLP/HTTP on port 4318 via OtlpHttpExporterFactory,
never parses the advertised keys, and the Phase 4 work is not yet delivered.
Fixes:
- 02-design-decisions.md: flip §2.1.1 SDK dependency recommendations to
OTLP/HTTP (shipped) with OTLP/gRPC marked Future. Update §2.2 architecture
diagram and text from OTLP/gRPC:4317 to OTLP/HTTP:4318. Rewrite §2.2.1 as
"OTLP/HTTP (Shipped)" and §2.2.2 as "OTLP/gRPC (Future Work — Planned
Upgrade)" with a concrete checklist (Conan dep, config parsing, factory
branch, runbook/dashboard updates) for landing the gRPC transport later.
- 05-configuration-reference.md: drop the fabricated exporter/otlp_grpc key
and the :4317 default from the sample config block and the options-summary
table. Move trace_pathfind, trace_txq, trace_validator, trace_amendment
into a new "Planned (not yet implemented)" table citing the phase that will
add each one. Keep the example config minimal so copy-paste does not produce
a silently-ignored stanza.
- 06-implementation-phases.md: reset Phase 4 Exit Criteria checkboxes from
[x] to [ ] (Phase 4 is not shipped at Phase-1a time). Rename "Phase 4a
Complete" to "Phase 4a Plan" and describe the work as future. Replace the
broken forward link to Phase4_taskList.md (introduced in the Phase 2 PR)
with a sentence pointing readers to where that spec will land. Renumber
the final section 6.12 to 6.11 so it sits directly after 6.10; section 6.11
("Effort Summary") was intentionally removed in earlier edits.
SpanGuard::span() hardcoded SpanKind::kInternal for every span. Tempo's
service-graph and spanmetrics RED calculations rely on kServer /
kConsumer / kClient / kProducer to classify inbound vs outbound vs
internal operations. With kInternal everywhere, the service graph
collapses to a single self-loop and RED metrics attribute all latency
to internal work.
Add categoryToSpanKind() mapping:
- Rpc -> kServer (inbound synchronous request)
- Peer -> kConsumer (inbound async peer message)
- Transactions -> kInternal
- Consensus -> kInternal
- Ledger -> kInternal
Only the single-argument overload is affected; childSpan / linkedSpan
continue to default to kInternal because they represent in-process
continuations of an already-kinded parent.
Spanmetrics dimensions used xrpl.rpc.command etc. but C++ emits bare
"command". Tempo tags for phase6-added consensus/tx/peer filters used
qualified names but C++ uses bare names. Dashboard panel referenced
xrpl_tx_suppressed (never populated) instead of suppressed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Consensus span attributes use bare names (close_time_correct,
consensus_state, close_resolution_ms) and shared canonical attrs
(xrpl.ledger.seq) per SpanNames.h. xrpl.consensus.mode and
xrpl.consensus.round are correct (domain-qualified to avoid collision).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Transaction span attributes use bare names (local, tx_status) per
SpanNames.h convention, not xrpl.tx.* qualified names. xrpl.tx.hash
is correct (shared canonical attr defined in SpanNames.h).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
RPC span attributes use bare names (command, rpc_status, rpc_role) per
the naming convention in SpanNames.h, not xrpl.rpc.* qualified names.
Node health attributes (amendment_blocked, server_state) are resource
attributes set at Tracer init, not span attributes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Panels 8-15 from statsd-node-health.json and panels 8-9 from
statsd-network-traffic.json were lost when Phase 7 renamed these files
to system-*. The merge (5cd71ed107) took Phase 7's smaller version
without the extra panels added by commit b933e8ae00 on Phase 6.
Recovered panels (system-node-health.json):
- Key Jobs Execution Time (11 job types)
- Key Jobs Dequeue Wait Time (11 job types)
- FullBelowCache Size
- FullBelowCache Hit Rate
- Ledger Publish Gap (validated - published age delta)
- State Duration Rate (Full vs Tracking)
- All Jobs Execution Time Detail (34 job types)
- All Jobs Dequeue Wait Detail (34 job types)
Recovered panels (system-network-traffic.json):
- Duplicate Traffic (Wasted Bandwidth)
- All Traffic Categories Detail (topk 15 by byte rate)
All recovered panels updated to include exported_instance=~"$node"
filter per project dashboard guidelines.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- MetricsRegistry.cpp: concatenate nested namespaces, add missing
direct includes (Journal.h, string, string_view, cstdint), suppress
readability-convert-member-functions-to-static in #else stubs by
referencing enabled_ member, void unused instanceId parameter.
- MetricsRegistry test: add missing direct includes (Log.h, Journal.h,
uint256.h, io_context.hpp, optional, stdexcept, string), make
throwUnimplemented() static, add [[nodiscard]] to getOpenLedger/
isStopping/getTrapTxID overrides, make const-eligible registry const.
- PerfLogImp.cpp: add braces around if/else body per
readability-braces-around-statements.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- MetricsRegistry.cpp: concatenate nested namespaces, add missing
direct includes (Journal.h, string, string_view, cstdint), suppress
readability-convert-member-functions-to-static in #else stubs by
referencing enabled_ member, void unused instanceId parameter.
- MetricsRegistry test: add missing direct includes (Log.h, Journal.h,
uint256.h, io_context.hpp, optional, stdexcept, string), make
throwUnimplemented() static, add [[nodiscard]] to getOpenLedger/
isStopping/getTrapTxID overrides, make const-eligible registry const.
- PerfLogImp.cpp: add braces around if/else body per
readability-braces-around-statements.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
NetworkOPs.h and SpanNames.h were only needed for per-span
nodeAmendmentBlocked/nodeServerState calls, which were removed
in the attr naming simplification. Fixes clang-tidy CI failure.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- otel-collector-config.yaml: spanmetrics dimensions use new bare names.
- tempo.yaml: TraceQL filter tags use new bare names.
- 02-design-decisions.md: strip xrpl.txq.* prefix from planned attrs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Phase2_taskList: update attr refs to bare names, note node-health
attrs moved to resource level.
- 02-design-decisions: strip xrpl.pathfind.* prefix from planned attrs.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Update OpenTelemetryPlan docs and Telemetry.h doc example to reflect
the renamed per-span attributes: xrpl.rpc.command -> command,
xrpl.rpc.status -> rpc_status, xrpl.grpc.method -> method, etc.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Update 31 attribute references in telemetry-runbook.md to match the
simplified naming: drop xrpl.<domain>. prefix on per-span attrs, use
domain-qualified names for collisions (rpc_status, consensus_state,
etc.), and unify cross-domain refs (xrpl.ledger.seq, xrpl.tx.hash).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Drop xrpl.pathfind.* prefix from per-span attrs (source_account,
dest_account, fast, search_level, num_complete_paths, num_paths,
num_requests).
- Keep xrpl.pathfind.ledger_index qualified (rule 5: distinct from
xrpl.ledger.seq).
- Remove per-span nodeAmendmentBlocked/nodeServerState calls from
RPCHandler — promoted to resource-level attrs.
- Mark node-health attrs in SpanNames.h as RESOURCE-ONLY with doc.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 10's workload validation configs (expected_metrics.json,
regression-metrics.json, validate_telemetry.py) queried the
MetricsRegistry metrics under the rippled_ prefix, but MetricsRegistry
emits them as xrpld_ (see MetricsRegistry.cpp). On a live run the
workload validator reported every MetricsRegistry metric as missing,
masking genuine regressions.
Rename the following to xrpld_ across the workload validator,
expected-metrics manifest, and regression-metrics template:
- nodestore_state, cache_metrics, txq_metrics, load_factor_metrics,
object_count
- rpc_method_started_total / _finished_total / _errored_total /
_duration_us
- job_queued_total / _started_total / _finished_total /
_queued_duration_us_bucket / _running_duration_us_bucket
- peer_quality, server_info, validator_health, ledger_economy,
db_metrics, complete_ledgers, build_info, state_tracking,
storage_detail
- ledgers_closed_total, validations_sent_total,
validations_checked_total, state_changes_total
- validation_agreement, validation_agreements_total,
validation_missed_total
Mirrors the phase-9 fix in commit 5601615952.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
MetricsRegistry emits OTel SDK metrics with the xrpld_ prefix
(MetricsRegistry.cpp defines "xrpld_nodestore_state",
"xrpld_cache_metrics", etc.), but the Phase 9 dashboards and the
Step 10c integration-test assertions introduced in 892fee638a
queried the rippled_ prefix. Every Phase 9 panel and assertion
therefore rendered "No data" or failed on a live run, even though
the underlying series were being exported correctly.
Rename the rippled_ prefix to xrpld_ for every MetricsRegistry
metric in dashboards and the integration test:
- nodestore_state, cache_metrics, txq_metrics, load_factor_metrics,
object_count
- rpc_method_started_total / _finished_total / _errored_total /
_duration_us_bucket
- job_queued_total / _started_total / _finished_total /
_queued_duration_us_bucket / _running_duration_us_bucket
- peer_quality, server_info, validator_health, ledger_economy,
db_metrics, complete_ledgers, build_info, state_tracking
- ledgers_closed_total, validations_sent_total,
validations_checked_total, state_changes_total
- validation_agreement (ValidationTracker 1h/24h/7d windows)
Also add ValidationTracker window-gauge assertions to Step 10c of
integration-test.sh so the 1h/24h/7d agreement and miss counts are
checked alongside the other Phase 9 gauges.
The rippled_ prefix is preserved for beast::insight metrics
(rippled_LedgerMaster_*, rippled_Peer_Finder_*, rippled_total_*,
rippled_Overlay_*, rippled_State_Accounting_*, rippled_transactions_*,
rippled_proposals_*, rippled_validations_Messages_*) because those
flow through the StatsD-style OTelCollector configured with
`[insight] prefix=rippled` and remain on that prefix by design.
Verified against a live 6-node consensus network: all 22 Phase 9 +
ValidationTracker assertions now report 6+ series per metric.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two fixes so gauges register in Prometheus (via StatsD) even when their
initial/steady-state value is 0:
1. StatsDGaugeImpl m_dirty: default-init to true so the initial value
(0) is emitted on the first flush. Previously, gauges whose value
never changed from 0 were never flushed and never appeared
downstream.
2. io_latency_sampler firstSample_: new atomic<bool>, init true.
m_event.notify now fires when either firstSample_ is true (exchanged
to false) or lastSample >= 10 ms. This guarantees the io_latency
metric is registered on startup; subsequent sub-10 ms samples are
still suppressed to avoid flooding.
Set defaults for tx_span::attr::suppressed (false) and
tx_span::attr::status ("new") immediately after creating the txReceive
span. Without defaults, spans whose suppressed/status attributes would
only be set in the HashRouter-suppressed branch lacked these attributes
entirely, producing incomplete span data in downstream stores.
The suppressed branch still overrides these when the transaction has
already been seen via HashRouter.
Two build failures surfaced by CI on the Phase 9 branch:
1. NetworkOPsImp stores the ServiceRegistry as
std::reference_wrapper<ServiceRegistry> registry_, so calls must go
through registry_.get().<method>(). The MetricsRegistry hooks added
in setMode() and recvValidation() dereferenced the wrapper directly,
which compiles against a pre-existing accessor on the wrapper type
on some toolchains but fails on clang 16/17/20 and gcc 13/15 with
"no member named 'getMetricsRegistry' in
std::reference_wrapper<xrpl::ServiceRegistry>".
2. MetricsRegistry::app_ and MetricsRegistry::journal_ are only used
inside XRPL_ENABLE_TELEMETRY-guarded code paths (gauge callbacks
and JLOG). When telemetry is disabled, clang's
-Werror=-Wunused-private-field tripped. Move the two fields under
the same #ifdef and guard the constructor initialisers with
[[maybe_unused]] so the no-op build continues to compile cleanly.
Addresses code review findings on PR #6513:
1. registerAsyncGauges() was ~730 lines, violating the CLAUDE.md
rule "No function longer than 80 lines." Split into fifteen
per-domain helpers (cache, TxQ, object count, load factor,
NodeStore, server info, build info, complete ledgers, DB,
validator health, peer quality, ledger economy, state tracking,
storage detail, validation agreement) dispatched from a thin
shell. Each helper now stays at or below the 80-line limit.
2. PerfLogImp::rpcEnd() only updated the in-memory counter and
never advanced the OTel xrpld_rpc_method_finished_total,
xrpld_rpc_method_errored_total, or xrpld_rpc_method_duration_us
instruments. rpcStart() was already wired up, so the finished
and errored counters stayed at zero for every RPC call.
rpcEnd() now computes the duration once, records it under the
existing mutex, and forwards finish/error events to
MetricsRegistry::recordRpcFinished / recordRpcErrored outside
the counter mutex to avoid lock nesting with the OTel SDK.
3. Added class-level Doxygen for MetricsRegistry with an ASCII
collaborator diagram and explicit @note tags covering
thread-safety, lifetime, and extension guidance.
Tempo /api/traces/{id} returns OTLP-shaped JSON with a top-level
"batches" key, not "data". The cross-check in check_log_correlation
was querying jq '.data | length' which always returned null, causing
the Log-Tempo cross-check to fail even when the trace existed.
MockServiceRegistry in MetricsRegistry.cpp still used the old method
names (timeKeeper, cachedSLEs, validators, overlay, cluster, app, etc.)
while ServiceRegistry has been standardized on getXxx()/isXxx() forms.
Windows CI caught this as C3668 "did not override any base class methods"
errors and C2259 "cannot instantiate abstract class".
Rename all 13 mismatched overrides to match the current interface:
timeKeeper -> getTimeKeeper
cachedSLEs -> getCachedSLEs
validators -> getValidators
validatorSites -> getValidatorSites
validatorManifests -> getValidatorManifests
publisherManifests -> getPublisherManifests
overlay -> getOverlay
cluster -> getCluster
peerReservations -> getPeerReservations
pendingSaves -> getPendingSaves
openLedger (x2) -> getOpenLedger
getPathRequests -> getPathRequestManager (type rename too)
journal -> getJournal
logs -> getLogs
trapTxID -> getTrapTxID
app -> getApp
Also regenerate levelization ordering.txt to reflect the new
tests.libxrpl -> xrpl.core edge introduced by ServiceRegistry.h include.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Consensus.h (Phase 4 tracing) depends on DisputedTx::getYays()/getNays()
to build disputeResolve span events. Both accessors were removed by
earlier 'duplicate accessor' cleanup commits on this branch, leaving
Consensus.h referencing non-existent members. CI caught this on
macOS/clang-17/gcc-13/Windows builds.
Restore the accessors on the branch where they were dropped so downstream
phase branches inherit a compiling DisputedTx.h via merge.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Clang-tidy fixes:
- Concatenate nested namespaces (modernize-concat-nested-namespaces)
in OTelCollector.h, OTelCollector.cpp, ValidationTracker.h/.cpp
- Add missing direct includes (misc-include-cleaner) in
ValidationTracker.cpp, test, CollectorManager.cpp, OTelCollector.cpp
- Make lock_guard variables const (misc-const-correctness)
- Add braces around single-line if/else (readability-braces-around-statements)
- Use designated initializer for WindowEvent (modernize-use-designated-initializers)
- Initialize LedgerEvent::seq field (cppcoreguidelines-pro-type-member-init)
Linker fix:
- Add ValidationTracker.cpp as source to xrpl.test.telemetry target
(it lives in src/xrpld/ but the test links against libxrpl only)
Levelization fix:
- Remove stale dependency edges from ordering.txt that were introduced
by the erroneous develop-merge commit
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
These were already added in an earlier phase branch. The duplicate
with slightly different Doxygen wording was introduced by the
erroneous merge/revert cycle.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace GetSpan() with direct context value check in Logs::format()
to avoid heap allocation (new DefaultSpan) on the no-span path
- Restore Phase 7 documentation accidentally deleted during merge
- Fix undefined $JAEGER variable → use $TEMPO in integration test
- Remove useless LCOV_EXCL markers around #ifdef block
- Fix indentation inconsistencies in Log.cpp injection block
- Remove incorrect url field from loki.yaml derivedFields
- Update stale code sample in Phase8_taskList.md to match implementation
- Correct "<10ns" performance claims to accurate ~15-20ns (no-span)
and ~50ns (active-span) measurements across all docs
- Replace Jaeger references with Tempo in TESTING.md (port 16686→3200)
- Improve error handling in check_log_correlation(): track files_scanned,
detect missing log files, fix silent grep error masking
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Fix use-after-free: extract gauge callback to static function and call
RemoveCallback in ~OTelGaugeImpl() before unregistering from collector
- Use memory_order_acq_rel on callHooks() debounce CAS for proper
happens-before relationship between hook invocations
- Add explicit 2s timeout to ForceFlush() in destructor to prevent
blocking indefinitely when OTLP endpoint is unreachable at shutdown
- Add OTLP receiver to metrics pipeline so native OTel metrics from
xrpld are actually received by the collector
- Remove stale health check port from docker-compose (extension was
removed from collector config)
- Clarify fallback docs: StatsD path requires re-enabling receiver/port
- Fix comments: Counter uses uint64_t not int64_t, gauge clamps to
[0, INT64_MAX] not [0, UINT64_MAX]
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The blanket revert of f4555c80fe also un-reverted some files that had
been correctly matched to phase-6 (nodestore Backend API refactor,
Vault_test changes). Restore those to the base branch state so the
phase-7 PR only contains telemetry changes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Build fixes in PeerImp.cpp:
- Rename duplicate `span` variable to `consSpan` in proposal and
validation handlers to avoid redefinition error
- Fix `->` on non-pointer SpanGuard (now correctly on shared_ptr)
- Fix move-only type copy in lambda capture
Clang-tidy fixes:
- Concatenate nested namespaces in LedgerSpanNames.h and PeerSpanNames.h
- Add missing SpanNames.h includes in BuildLedger.cpp, LedgerMaster.cpp,
PeerImp.cpp for direct seg:: symbol usage
- Add missing <chrono> and <cstdint> includes in BuildLedger.cpp
- Remove unused Feature.h include from BuildLedger.cpp
Rename check fix:
- Run docs.sh to rename rippled_ metric prefixes to xrpld_ in
09-data-collection-reference.md and telemetry-runbook.md
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- protocol/README.md: restore historical GitHub URL path (src/ripple/)
- Config.cpp: restore configLegacyName as "rippled.cfg" (legacy name
must remain as-is for backward compatibility)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Reverts 259 files that carried unrelated upstream changes through the
phase-6 merge: enum class removals (cppcoreguidelines-use-enum-class),
scoped_lock→lock_guard conversions (modernize-use-scoped-lock),
nodestore Backend API changes (void const* key), .clang-tidy config,
test infrastructure deletions, and miscellaneous develop changes.
These changes belong on develop, not in the telemetry PR chain.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Revert reusable-build-test-config.yml to develop (action SHA update
and "Show test failure summary" step removal don't belong here)
- Revert upload-conan-deps.yml to develop (action SHA update)
- Revert features.macro: BatchInnerSigs and Batch back to Supported::no
(these feature flag changes are unrelated to telemetry)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Separate local declarations from assignments to avoid hiding errors,
and use [[ instead of [ for non-POSIX comparisons.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Prettier formatting for markdown docs and OTelCollector header
- docs.sh rippled→xrpld renames in OTelCollector.cpp comments/strings
- Updated levelization ordering with new dependency edges
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add [[nodiscard]] to getConsensusTraceStrategy, getYays, getNays
- Add missing <string>, SpanGuard.h, SpanNames.h includes
- Fix widening cast placement (cast before arithmetic, not after)
- Replace nested ternary with lambda for const dir variable
- Add braces to if/else-if chains in Consensus.h
- Concatenate nested namespaces in ConsensusSpanNames.h
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Concatenate nested namespaces in SpanNames.h, RpcSpanNames.h, GrpcSpanNames.h
- Remove unused InfoSub.h and NetworkOPs.h includes from RPCHandler.cpp
- Add missing <string_view> includes in RPCHandler.cpp and GRPCServer.cpp
- Replace nested ternary with if/else-if in RPCHandler.cpp
- Add IWYU pragma keep for json_body.h in ServerHandler.cpp
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Clang-tidy misc-include-cleaner requires direct includes for all used
symbols. Application.cpp calls toBase58(TokenType::NodePublic, ...) at
line 1359 but did not directly include PublicKey.h.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Document the propagation infrastructure: send-side injection in
NetworkOPs/RCLConsensus, receive-side extraction in PeerImp via
PropagationHelpers.h and ConsensusReceiveTracing.h. Update
consensus receive span descriptions to reflect parent extraction.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add [[maybe_unused]] to RAII spans in TxQ.cpp
- Include Telemetry.h in RCLConsensus.cpp for complete type
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix quorum attribute to use actual validator quorum instead of proposer
count, add missing ConsensusState::Expired handling in haveConsensus()
span, move ConsensusSpanNames.h to xrpld/consensus/ to resolve
levelization cycle, remove unused constants, enrich proposal receive
span with sequence, and correct stale documentation references.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add cons_span::event namespace with disputeResolve and txIncluded
constants; replace hardcoded strings in Consensus.h and RCLConsensus.cpp
- Move proposal.receive and validation.receive spans in PeerImp into
shared_ptr captured by job lambdas so they measure checkPropose and
checkValidation timing, not just message parsing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update Phase4_taskList.md and 06-implementation-phases.md to reflect
completed implementation of all remaining Phase 4/4a tasks (4.2-4.6,
4a.5, 4a.6, 4a.8). Update exit criteria and summary tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implement remaining Phase 4/4a consensus tracing tasks:
- Add consensus.phase.open span (open → closeLedger lifecycle)
- Add consensus.proposal.receive span in PeerImp with trusted attr
- Add consensus.validation.receive span in PeerImp with trusted/seq attrs
- Add tx_count attr on accept.apply, disputes_count on update_positions
- Add tx.included events with txId in doAccept transaction loop
- Enhance dispute.resolve event with yays/nays fields
- Add avalanche_threshold attr on update_positions span
- Reparent accept/accept.apply as children of round span via childSpan()
Also adds compile-time constants in ConsensusSpanNames.h and updates
the span hierarchy diagram.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The 4-arg hashSpan overload was duplicated during a prior rebase
cascade — it appeared at both line 240 and line 305 in SpanGuard.cpp.
This would cause a linker error (multiple definition).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Record the close time voting threshold and consensus state on
consensus.update_positions and consensus.check spans:
- xrpl.consensus.close_time_threshold: the avCT_CONSENSUS_PCT (75%)
threshold required for close time agreement
- xrpl.consensus.have_close_time_consensus: whether validators
reached close time consensus in this iteration
These attributes enable dashboards to show how the close time
voting process converges (or stalls) across consensus iterations.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove the span-replacement logic in startRoundTracing() that was
discarding the hash-derived round span and replacing it with a linked
span (which gets a random trace_id). The deterministic trace_id from
the ledger hash is the key feature enabling cross-node correlation —
replacing it broke correlation on all rounds after the first.
Also: use thread_local mt19937 for hashSpan() span IDs (same fix as
phase-3 txSpan), add Doxygen to establish tracing method declarations
in Consensus.h, and update SpanGuard.h diagram with hashSpan/addEvent.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Instrument the consensus subsystem with OpenTelemetry spans covering
the full round lifecycle: round start, establish phase, proposal send,
ledger close, position updates, consensus check, accept, validation
send, and mode changes.
Key design choices adapted from the original Phase 4 implementation
to the new SpanGuard factory pattern introduced in Phase 3:
- Add SpanGuard::hashSpan() for category-gated hash-derived trace IDs
(consensus round spans share trace_id across validators via ledger hash)
- Add SpanGuard::addEvent() overload with key-value attribute pairs
(used for dispute.resolve events during position updates)
- Add ConsensusSpanNames.h with compile-time span name constants
following the colocated *SpanNames.h pattern from Phase 3
- Add consensusTraceStrategy config option ("deterministic"/"attribute")
for cross-node trace correlation strategy selection
- Use SpanGuard::linkedSpan() for follows-from relationships between
consecutive rounds and cross-thread validation spans
- Use SpanGuard::captureContext() for thread-safe context propagation
from consensus thread to jtACCEPT worker thread
Spans produced: consensus.round, consensus.proposal.send,
consensus.ledger_close, consensus.establish, consensus.update_positions,
consensus.check, consensus.accept, consensus.accept.apply,
consensus.validation.send, consensus.mode_change
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wire trace context into P2P message flow so distributed traces
link across nodes. TX relay injects SpanGuard context via
PropagationHelpers.h; consensus propose/validate injects via
TraceContextPropagator.h. Receive-side extraction in PeerImp
creates child spans for proposals and validations.
- Add TraceBytes struct and SpanGuard::getTraceBytes() for
extracting raw trace context without OTel type dependencies
- Add PropagationHelpers.h: injectSpanContext(SpanGuard, proto)
- Add ConsensusReceiveTracing.h: proposalReceiveSpan(),
validationReceiveSpan() with parent context extraction
- NetworkOPs::apply(): inject tx.process context before relay
- RCLConsensus::propose()/validate(): inject active span context
- PeerImp: create receive spans for proposals and validations
with sender's trace context as parent
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move TxQSpanNames.h include to correct alphabetical position, update
levelization results for new xrpld.telemetry module dependencies,
and apply rename script to docs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- tx.receive span in PeerImp: convert to shared_ptr, capture in
checkTransaction lambda so it measures actual processing, not just
message parsing
- tx.process span in NetworkOPs: convert to shared_ptr, store in
TransactionStatus so it lives until the batch job processes the entry;
sync path unchanged (span destructs on function return)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace SpanGuard::txSpan(prefix, name, hash) with the generic
SpanGuard::hashSpan(TraceCategory, name, hash) that accepts a
TraceCategory parameter instead of hardcoding Transactions. This
enables reuse for consensus round spans (Phase 4) and any future
subsystem needing deterministic cross-node trace correlation via
hash-derived trace IDs.
Both overloads are replaced:
- hashSpan(cat, name, hash, size) — standalone with random span_id
- hashSpan(cat, name, hash, size, parentSpanId, parentSize, flags)
— with remote parent from protobuf context propagation
Add full span name constants (tx_span::receive, tx_span::process)
to TxSpanNames.h following the ConsensusSpanNames.h pattern.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mark local variables in extractFromProtobuf() and injectToProtobuf()
as const since they are not modified after initialization: traceId,
spanId, flags, spanCtx, and span.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace thread_local mt19937 with xrpl::default_prng() for span ID
generation — uses the project's existing thread-local xor-shift engine.
One call yields a uint64_t (8 bytes), filling the span ID in a single
memcpy without loops.
Fix compilation failure when XRPL_ENABLE_TELEMETRY is not defined:
move xrpl.pb.h include outside the #ifdef guard in TxTracing.h since
protocol::TMTransaction is used unconditionally in the function
signature.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace per-call std::random_device with thread_local std::mt19937 in
txSpan() for span ID generation. random_device is ~423x slower due to
/dev/urandom syscalls on each construction; mt19937 is seeded once per
thread and reused for all subsequent span IDs.
Update the SpanGuard class ASCII diagram to include txSpan factory
methods that were added in the hash-derived trace ID commit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move TxSpanNames.h and TxQSpanNames.h from src/xrpld/telemetry/ to sit
next to the classes they instrument, matching the PathFindSpanNames.h
convention.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Derive trace_id from txHash[0:16] so all nodes handling the same
transaction produce spans under the same trace. Protobuf span_id
propagation provides parent-child relay ordering when available.
- Add SpanGuard::txSpan() factory methods (hash-derived trace ID)
- Add TxTracing.h helpers: txReceiveSpan(), txProcessSpan()
- Update PeerImp and NetworkOPs to use the new helpers
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add trace_id = txHash[0:16] strategy so all nodes handling the same
transaction independently produce spans under the same trace_id,
combined with protobuf span_id propagation for parent-child ordering.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move scattered string literals from PeerImp.cpp and NetworkOPs.cpp into
compile-time constants in src/xrpld/telemetry/TxSpanNames.h. Follows
the same StaticStr/join() pattern established in Phase 1c for RPC spans.
Constants cover: span prefixes (tx), operations (receive, process),
attribute keys (hash, local, path, suppressed, status, peerId,
peerVersion), and values (sync, async, knownBad).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace references to old XRPL_TRACE_TX/CONSENSUS macros with
SpanGuard::span(TraceCategory, ...) factory calls introduced in Phase 1c.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds xrpl.peer.version attribute to tx.receive spans for version-mismatch
correlation during network upgrades.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add "hicpp" to cspell dictionary for NOLINT annotations
- Concatenate nested namespaces in RpcSpanNames.h
- Fix include hygiene and nested ternary in RPCHandler.cpp
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Document the propagation infrastructure: send-side injection in
NetworkOPs/RCLConsensus, receive-side extraction in PeerImp via
PropagationHelpers.h and ConsensusReceiveTracing.h. Update
consensus receive span descriptions to reflect parent extraction.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix quorum attribute to use actual validator quorum instead of proposer
count, add missing ConsensusState::Expired handling in haveConsensus()
span, move ConsensusSpanNames.h to xrpld/consensus/ to resolve
levelization cycle, remove unused constants, enrich proposal receive
span with sequence, and correct stale documentation references.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add cons_span::event namespace with disputeResolve and txIncluded
constants; replace hardcoded strings in Consensus.h and RCLConsensus.cpp
- Move proposal.receive and validation.receive spans in PeerImp into
shared_ptr captured by job lambdas so they measure checkPropose and
checkValidation timing, not just message parsing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update Phase4_taskList.md and 06-implementation-phases.md to reflect
completed implementation of all remaining Phase 4/4a tasks (4.2-4.6,
4a.5, 4a.6, 4a.8). Update exit criteria and summary tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implement remaining Phase 4/4a consensus tracing tasks:
- Add consensus.phase.open span (open → closeLedger lifecycle)
- Add consensus.proposal.receive span in PeerImp with trusted attr
- Add consensus.validation.receive span in PeerImp with trusted/seq attrs
- Add tx_count attr on accept.apply, disputes_count on update_positions
- Add tx.included events with txId in doAccept transaction loop
- Enhance dispute.resolve event with yays/nays fields
- Add avalanche_threshold attr on update_positions span
- Reparent accept/accept.apply as children of round span via childSpan()
Also adds compile-time constants in ConsensusSpanNames.h and updates
the span hierarchy diagram.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The 4-arg hashSpan overload was duplicated during a prior rebase
cascade — it appeared at both line 240 and line 305 in SpanGuard.cpp.
This would cause a linker error (multiple definition).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Record the close time voting threshold and consensus state on
consensus.update_positions and consensus.check spans:
- xrpl.consensus.close_time_threshold: the avCT_CONSENSUS_PCT (75%)
threshold required for close time agreement
- xrpl.consensus.have_close_time_consensus: whether validators
reached close time consensus in this iteration
These attributes enable dashboards to show how the close time
voting process converges (or stalls) across consensus iterations.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove the span-replacement logic in startRoundTracing() that was
discarding the hash-derived round span and replacing it with a linked
span (which gets a random trace_id). The deterministic trace_id from
the ledger hash is the key feature enabling cross-node correlation —
replacing it broke correlation on all rounds after the first.
Also: use thread_local mt19937 for hashSpan() span IDs (same fix as
phase-3 txSpan), add Doxygen to establish tracing method declarations
in Consensus.h, and update SpanGuard.h diagram with hashSpan/addEvent.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Instrument the consensus subsystem with OpenTelemetry spans covering
the full round lifecycle: round start, establish phase, proposal send,
ledger close, position updates, consensus check, accept, validation
send, and mode changes.
Key design choices adapted from the original Phase 4 implementation
to the new SpanGuard factory pattern introduced in Phase 3:
- Add SpanGuard::hashSpan() for category-gated hash-derived trace IDs
(consensus round spans share trace_id across validators via ledger hash)
- Add SpanGuard::addEvent() overload with key-value attribute pairs
(used for dispute.resolve events during position updates)
- Add ConsensusSpanNames.h with compile-time span name constants
following the colocated *SpanNames.h pattern from Phase 3
- Add consensusTraceStrategy config option ("deterministic"/"attribute")
for cross-node trace correlation strategy selection
- Use SpanGuard::linkedSpan() for follows-from relationships between
consecutive rounds and cross-thread validation spans
- Use SpanGuard::captureContext() for thread-safe context propagation
from consensus thread to jtACCEPT worker thread
Spans produced: consensus.round, consensus.proposal.send,
consensus.ledger_close, consensus.establish, consensus.update_positions,
consensus.check, consensus.accept, consensus.accept.apply,
consensus.validation.send, consensus.mode_change
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wire trace context into P2P message flow so distributed traces
link across nodes. TX relay injects SpanGuard context via
PropagationHelpers.h; consensus propose/validate injects via
TraceContextPropagator.h. Receive-side extraction in PeerImp
creates child spans for proposals and validations.
- Add TraceBytes struct and SpanGuard::getTraceBytes() for
extracting raw trace context without OTel type dependencies
- Add PropagationHelpers.h: injectSpanContext(SpanGuard, proto)
- Add ConsensusReceiveTracing.h: proposalReceiveSpan(),
validationReceiveSpan() with parent context extraction
- NetworkOPs::apply(): inject tx.process context before relay
- RCLConsensus::propose()/validate(): inject active span context
- PeerImp: create receive spans for proposals and validations
with sender's trace context as parent
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move TxQSpanNames.h include to correct alphabetical position, update
levelization results for new xrpld.telemetry module dependencies,
and apply rename script to docs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- tx.receive span in PeerImp: convert to shared_ptr, capture in
checkTransaction lambda so it measures actual processing, not just
message parsing
- tx.process span in NetworkOPs: convert to shared_ptr, store in
TransactionStatus so it lives until the batch job processes the entry;
sync path unchanged (span destructs on function return)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace SpanGuard::txSpan(prefix, name, hash) with the generic
SpanGuard::hashSpan(TraceCategory, name, hash) that accepts a
TraceCategory parameter instead of hardcoding Transactions. This
enables reuse for consensus round spans (Phase 4) and any future
subsystem needing deterministic cross-node trace correlation via
hash-derived trace IDs.
Both overloads are replaced:
- hashSpan(cat, name, hash, size) — standalone with random span_id
- hashSpan(cat, name, hash, size, parentSpanId, parentSize, flags)
— with remote parent from protobuf context propagation
Add full span name constants (tx_span::receive, tx_span::process)
to TxSpanNames.h following the ConsensusSpanNames.h pattern.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mark local variables in extractFromProtobuf() and injectToProtobuf()
as const since they are not modified after initialization: traceId,
spanId, flags, spanCtx, and span.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace thread_local mt19937 with xrpl::default_prng() for span ID
generation — uses the project's existing thread-local xor-shift engine.
One call yields a uint64_t (8 bytes), filling the span ID in a single
memcpy without loops.
Fix compilation failure when XRPL_ENABLE_TELEMETRY is not defined:
move xrpl.pb.h include outside the #ifdef guard in TxTracing.h since
protocol::TMTransaction is used unconditionally in the function
signature.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace per-call std::random_device with thread_local std::mt19937 in
txSpan() for span ID generation. random_device is ~423x slower due to
/dev/urandom syscalls on each construction; mt19937 is seeded once per
thread and reused for all subsequent span IDs.
Update the SpanGuard class ASCII diagram to include txSpan factory
methods that were added in the hash-derived trace ID commit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move TxSpanNames.h and TxQSpanNames.h from src/xrpld/telemetry/ to sit
next to the classes they instrument, matching the PathFindSpanNames.h
convention.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Derive trace_id from txHash[0:16] so all nodes handling the same
transaction produce spans under the same trace. Protobuf span_id
propagation provides parent-child relay ordering when available.
- Add SpanGuard::txSpan() factory methods (hash-derived trace ID)
- Add TxTracing.h helpers: txReceiveSpan(), txProcessSpan()
- Update PeerImp and NetworkOPs to use the new helpers
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add trace_id = txHash[0:16] strategy so all nodes handling the same
transaction independently produce spans under the same trace_id,
combined with protobuf span_id propagation for parent-child ordering.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move scattered string literals from PeerImp.cpp and NetworkOPs.cpp into
compile-time constants in src/xrpld/telemetry/TxSpanNames.h. Follows
the same StaticStr/join() pattern established in Phase 1c for RPC spans.
Constants cover: span prefixes (tx), operations (receive, process),
attribute keys (hash, local, path, suppressed, status, peerId,
peerVersion), and values (sync, async, knownBad).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace references to old XRPL_TRACE_TX/CONSENSUS macros with
SpanGuard::span(TraceCategory, ...) factory calls introduced in Phase 1c.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds xrpl.peer.version attribute to tx.receive spans for version-mismatch
correlation during network upgrades.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add [[maybe_unused]] to RAII span variables in PathFind/RipplePathFind
handlers (Clang -Wunused-variable with -Werror)
- Restore over-renamed values: rippledb, rippled.cfg, historical GitHub URL
- Concatenate nested namespaces in SpanNames.h and PathFindSpanNames.h
(modernize-concat-nested-namespaces)
- Add missing includes and const qualifiers in test files
- Suppress intentional use-after-move in SpanGuardFactory move test
- Remove unused NetworkOPs.h include from PathRequest.cpp
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Document the propagation infrastructure: send-side injection in
NetworkOPs/RCLConsensus, receive-side extraction in PeerImp via
PropagationHelpers.h and ConsensusReceiveTracing.h. Update
consensus receive span descriptions to reflect parent extraction.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Wire trace context into P2P message flow so distributed traces
link across nodes. TX relay injects SpanGuard context via
PropagationHelpers.h; consensus propose/validate injects via
TraceContextPropagator.h. Receive-side extraction in PeerImp
creates child spans for proposals and validations.
- Add TraceBytes struct and SpanGuard::getTraceBytes() for
extracting raw trace context without OTel type dependencies
- Add PropagationHelpers.h: injectSpanContext(SpanGuard, proto)
- Add ConsensusReceiveTracing.h: proposalReceiveSpan(),
validationReceiveSpan() with parent context extraction
- NetworkOPs::apply(): inject tx.process context before relay
- RCLConsensus::propose()/validate(): inject active span context
- PeerImp: create receive spans for proposals and validations
with sender's trace context as parent
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The StatsD receiver config was lost during a branch rebase (--ours
conflict resolution dropped it). Re-add the statsd receiver to the
OTel Collector config and wire it into the metrics pipeline so
beast::insight UDP metrics flow to Prometheus.
Also fixes:
- Metric prefix mismatch: docs used xrpld_ but dashboards/tests use
rippled_ — align all documentation to match the runnable stack
- Remove phantom Peer_Disconnects_Charges from docs (plain atomic,
not a beast::insight gauge)
- Remove premature .codecov.yml exclusions for Phase 7 OTelCollector
files that don't exist on this branch
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add missing xrpl.consensus.quorum attribute to consensus.accept in runbook
- Fix dashboard legend formats: add exported_instance, use Title Case
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add 14 missing spans to runbook (6 TxQ + 8 consensus)
- Fix tx.receive attributes and config table in runbook
- Document dispute.resolve and tx.included span events
- Add spanmetrics dimensions for close_time_correct and tx.suppressed
- Fix Close Time Agreement and TX Receive vs Suppressed panel PromQL
- Wire $consensus_mode template variable to all consensus panels
- Add 10 Tempo search filters for operational attributes
- Apply rename script artifacts
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Document service_instance_id, use_tls, tls_ca_cert, batch_size,
batch_delay_ms, and max_queue_size in the [telemetry] section.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace remaining rippled/Ripple references with xrpld/XRPL in
data collection reference, implementation phases, and runbook docs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Resolve merge conflicts taking phase 4 consensus span improvements,
fix bashate indentation in integration test script, and apply rename
script to Phase5 integration test docs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move TxQSpanNames.h include to correct alphabetical position, update
levelization results for new xrpld.telemetry module dependencies,
and apply rename script to docs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add unknownCommand and wsUpgrade span name constants to RpcSpanNames.h,
fix SpanGuardFactory tests to use the 3-argument SpanGuard::span() API,
update levelization results, and apply rename script to docs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Use C++17 concatenated namespaces, add [[nodiscard]] to query methods,
add missing direct includes, and use pass-by-value + std::move in
NullTelemetry constructor.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix quorum attribute to use actual validator quorum instead of proposer
count, add missing ConsensusState::Expired handling in haveConsensus()
span, move ConsensusSpanNames.h to xrpld/consensus/ to resolve
levelization cycle, remove unused constants, enrich proposal receive
span with sequence, and correct stale documentation references.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- tx.receive span in PeerImp: convert to shared_ptr, capture in
checkTransaction lambda so it measures actual processing, not just
message parsing
- tx.process span in NetworkOPs: convert to shared_ptr, store in
TransactionStatus so it lives until the batch job processes the entry;
sync path unchanged (span destructs on function return)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add cons_span::event namespace with disputeResolve and txIncluded
constants; replace hardcoded strings in Consensus.h and RCLConsensus.cpp
- Move proposal.receive and validation.receive spans in PeerImp into
shared_ptr captured by job lambdas so they measure checkPropose and
checkValidation timing, not just message parsing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update Phase4_taskList.md and 06-implementation-phases.md to reflect
completed implementation of all remaining Phase 4/4a tasks (4.2-4.6,
4a.5, 4a.6, 4a.8). Update exit criteria and summary tables.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implement remaining Phase 4/4a consensus tracing tasks:
- Add consensus.phase.open span (open → closeLedger lifecycle)
- Add consensus.proposal.receive span in PeerImp with trusted attr
- Add consensus.validation.receive span in PeerImp with trusted/seq attrs
- Add tx_count attr on accept.apply, disputes_count on update_positions
- Add tx.included events with txId in doAccept transaction loop
- Enhance dispute.resolve event with yays/nays fields
- Add avalanche_threshold attr on update_positions span
- Reparent accept/accept.apply as children of round span via childSpan()
Also adds compile-time constants in ConsensusSpanNames.h and updates
the span hierarchy diagram.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sort PathFindSpanNames.h after AssetCache.h alphabetically in
PathRequestManager.cpp and Pathfinder.cpp to satisfy the project's
include-order convention enforced by pre-commit hooks.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move SpanGuard.h (associated header) to first include position,
separated by blank line from other project includes, per the
project's include-order convention enforced by pre-commit hooks.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move Telemetry.h (associated header) to first include position in
Telemetry.cpp per the project's include-order convention. Trim
trailing whitespace from POC_taskList.md markdown table columns.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 5 new panels to the consensus-health Grafana dashboard using Tempo
TraceQL queries against consensus.accept.apply span attributes:
- Close Time: Raw Proposals (Per Node) — each node's unrounded
wall-clock close_time_self, reveals clock drift across validators
- Close Time: Effective / Quantized — the consensus-agreed close_time
after rounding to resolution bins, written to ledger header
- Close Time Vote Bins & Resolution — number of distinct vote bins
(close_time_vote_bins) and bin size (close_resolution_ms) on dual axes
- Close Time Resolution Direction — whether resolution increased
(coarser), decreased (finer), or stayed unchanged
- Close Time Bin Distribution — bar chart showing how raw proposals
distribute across quantized bins per round
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Integrate the existing StatsD metrics pipeline (beast::insight) into
the OpenTelemetry observability stack and add new trace spans for
ledger build/store/validate and peer proposal/validation receive.
Phase 5b — Ledger, peer, and transaction spans:
- Add ledger.build span with close time attributes in BuildLedger.cpp
- Add tx.apply span with tx_count/tx_failed in BuildLedger.cpp
- Add ledger.store and ledger.validate spans in LedgerMaster.cpp
- Add peer.proposal.receive span with trusted attribute in PeerImp.cpp
- Add peer.validation.receive span with ledger_hash, full, trusted
attributes in PeerImp.cpp
- Add ledger-operations and peer-network Grafana dashboards
Phase 6 — StatsD metrics integration:
- Add StatsD UDP receiver (port 8125) to OTel Collector
- Add 5 StatsD Grafana dashboards: node health, network traffic,
overlay traffic detail, ledger data sync, RPC pathfinding
- Add 09-data-collection-reference.md cataloging all metrics/spans
- Update existing dashboards with new span panels
- Expand telemetry runbook and integration test script
- Add codecov exclusions for telemetry modules
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Run .github/scripts/rename/docs.sh to replace rippled → xrpld
references in TESTING.md, xrpld-telemetry.cfg, and
telemetry-runbook.md, fixing the check-rename CI failure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Mark Tasks 5.3 (alert definitions) and 5.6 (training materials) as
"Deferred — post-MVP" in the implementation phases document to
accurately reflect current delivery scope. Add status column to the
Phase 5 task table.
Also fix stale reference to XRPL_TRACE_* macros in Phase 4a section —
the implementation uses SpanGuard factory methods.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The 4-arg hashSpan overload was duplicated during a prior rebase
cascade — it appeared at both line 240 and line 305 in SpanGuard.cpp.
This would cause a linker error (multiple definition).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Record the close time voting threshold and consensus state on
consensus.update_positions and consensus.check spans:
- xrpl.consensus.close_time_threshold: the avCT_CONSENSUS_PCT (75%)
threshold required for close time agreement
- xrpl.consensus.have_close_time_consensus: whether validators
reached close time consensus in this iteration
These attributes enable dashboards to show how the close time
voting process converges (or stalls) across consensus iterations.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove the span-replacement logic in startRoundTracing() that was
discarding the hash-derived round span and replacing it with a linked
span (which gets a random trace_id). The deterministic trace_id from
the ledger hash is the key feature enabling cross-node correlation —
replacing it broke correlation on all rounds after the first.
Also: use thread_local mt19937 for hashSpan() span IDs (same fix as
phase-3 txSpan), add Doxygen to establish tracing method declarations
in Consensus.h, and update SpanGuard.h diagram with hashSpan/addEvent.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Instrument the consensus subsystem with OpenTelemetry spans covering
the full round lifecycle: round start, establish phase, proposal send,
ledger close, position updates, consensus check, accept, validation
send, and mode changes.
Key design choices adapted from the original Phase 4 implementation
to the new SpanGuard factory pattern introduced in Phase 3:
- Add SpanGuard::hashSpan() for category-gated hash-derived trace IDs
(consensus round spans share trace_id across validators via ledger hash)
- Add SpanGuard::addEvent() overload with key-value attribute pairs
(used for dispute.resolve events during position updates)
- Add ConsensusSpanNames.h with compile-time span name constants
following the colocated *SpanNames.h pattern from Phase 3
- Add consensusTraceStrategy config option ("deterministic"/"attribute")
for cross-node trace correlation strategy selection
- Use SpanGuard::linkedSpan() for follows-from relationships between
consecutive rounds and cross-thread validation spans
- Use SpanGuard::captureContext() for thread-safe context propagation
from consensus thread to jtACCEPT worker thread
Spans produced: consensus.round, consensus.proposal.send,
consensus.ledger_close, consensus.establish, consensus.update_positions,
consensus.check, consensus.accept, consensus.accept.apply,
consensus.validation.send, consensus.mode_change
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace SpanGuard::txSpan(prefix, name, hash) with the generic
SpanGuard::hashSpan(TraceCategory, name, hash) that accepts a
TraceCategory parameter instead of hardcoding Transactions. This
enables reuse for consensus round spans (Phase 4) and any future
subsystem needing deterministic cross-node trace correlation via
hash-derived trace IDs.
Both overloads are replaced:
- hashSpan(cat, name, hash, size) — standalone with random span_id
- hashSpan(cat, name, hash, size, parentSpanId, parentSize, flags)
— with remote parent from protobuf context propagation
Add full span name constants (tx_span::receive, tx_span::process)
to TxSpanNames.h following the ConsensusSpanNames.h pattern.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mark local variables in extractFromProtobuf() and injectToProtobuf()
as const since they are not modified after initialization: traceId,
spanId, flags, spanCtx, and span.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace thread_local mt19937 with xrpl::default_prng() for span ID
generation — uses the project's existing thread-local xor-shift engine.
One call yields a uint64_t (8 bytes), filling the span ID in a single
memcpy without loops.
Fix compilation failure when XRPL_ENABLE_TELEMETRY is not defined:
move xrpl.pb.h include outside the #ifdef guard in TxTracing.h since
protocol::TMTransaction is used unconditionally in the function
signature.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace per-call std::random_device with thread_local std::mt19937 in
txSpan() for span ID generation. random_device is ~423x slower due to
/dev/urandom syscalls on each construction; mt19937 is seeded once per
thread and reused for all subsequent span IDs.
Update the SpanGuard class ASCII diagram to include txSpan factory
methods that were added in the hash-derived trace ID commit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move TxSpanNames.h and TxQSpanNames.h from src/xrpld/telemetry/ to sit
next to the classes they instrument, matching the PathFindSpanNames.h
convention.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Derive trace_id from txHash[0:16] so all nodes handling the same
transaction produce spans under the same trace. Protobuf span_id
propagation provides parent-child relay ordering when available.
- Add SpanGuard::txSpan() factory methods (hash-derived trace ID)
- Add TxTracing.h helpers: txReceiveSpan(), txProcessSpan()
- Update PeerImp and NetworkOPs to use the new helpers
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add trace_id = txHash[0:16] strategy so all nodes handling the same
transaction independently produce spans under the same trace_id,
combined with protobuf span_id propagation for parent-child ordering.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move scattered string literals from PeerImp.cpp and NetworkOPs.cpp into
compile-time constants in src/xrpld/telemetry/TxSpanNames.h. Follows
the same StaticStr/join() pattern established in Phase 1c for RPC spans.
Constants cover: span prefixes (tx), operations (receive, process),
attribute keys (hash, local, path, suppressed, status, peerId,
peerVersion), and values (sync, async, knownBad).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace references to old XRPL_TRACE_TX/CONSENSUS macros with
SpanGuard::span(TraceCategory, ...) factory calls introduced in Phase 1c.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds xrpl.peer.version attribute to tx.receive spans for version-mismatch
correlation during network upgrades.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Instrument the path finding subsystem with full span coverage:
- pathfind.request: wraps doPathFind() and doRipplePathFind() RPC handlers
- pathfind.compute: wraps PathRequest::doUpdate() with fast/normal attr
- pathfind.update_all: wraps PathRequestManager::updateAll() on ledger
close with ledger_index attr
- pathfind.discover: wraps Pathfinder::findPaths() graph exploration
with search_level attr
- pathfind.rank: wraps Pathfinder::computePathRanks() liquidity
validation with num_paths attr
New file: PathFindSpanNames.h with compile-time constants following
the StaticStr/join() pattern from Phase 1c.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Move node health attribute strings to compile-time constants in
SpanNames.h (attr::nodeAmendmentBlocked, attr::nodeServerState)
- Add Tempo search filters for node health attributes
- Remove unnecessary .c_str() on strOperatingMode() return
- Add samplingRatio clamping test (values > 1.0 and < 0.0)
- Fix Task 2.3 status: delivered in Phase 1c, not Phase 2
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- serviceName default is "xrpld" not "rippled"
- Remove references to nonexistent exporterType field
- Pass networkId (4th param) to setup_Telemetry()
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Mark deferred tasks (2.1→Phase 3, 2.5→low priority) with rationale.
Mark superseded tasks (2.2→Phase 1c SpanGuard factory). Add Task 2.7
for Grafana search filters. Update summary table with status column.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Grafana Tempo datasource: add rpc-command, rpc-status, rpc-role
search filters for the Explore UI
- Unit tests: TelemetryConfig (config parsing defaults and sections),
SpanGuardFactory (null guard safety, move semantics, discard, all
factory methods)
- Test CMake registration with optional OTel linking
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add amendment_blocked and server_state span attributes to every
rpc.command.* span so operators can correlate RPC behavior with node state.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Introduces task list documents for Phases 2 through 5, with Tempo
references (replacing Jaeger) and Task 2.8 dashboard parity spec.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Centralise scattered string literals into compile-time constants using
StaticStr<N> and join() for dot-separated composition. Shared primitives
live in SpanNames.h; RPC-specific names in RpcSpanNames.h. Future modules
(consensus, peer, ledger) add their own *SpanNames.h without bloating
the central header.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add [[maybe_unused]] to the RAII span in processSession() — the
variable is not read but its lifetime scopes the active OTel context
for child spans created in processRequest()
- Regenerate levelization: remove premature xrpld.telemetry entries
that reference a module not yet present on this branch
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Use grpc_span::val::resourceExhausted constant instead of raw
"resource_exhausted" string in GRPCServer.cpp
- Fix unbounded span name cardinality in RPCHandler.cpp error path:
use fixed rpc_span::val::unknownCommand as span name instead of
user-supplied cmdName (attacker-controlled input). The actual
command is still captured in the xrpl.rpc.command attribute.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Without this, cloned CallData instances (created for the next incoming
gRPC request) would have an empty name_, making subsequent span attrs
blank.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Follow project convention (PerfLog.h, SpanGuard.h) for documentation
diagrams. Show HTTP single, HTTP batch, and WebSocket span nesting.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Document the span hierarchy, covered paths, and known instrumentation
gaps directly in the header that developers reference when adding spans.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Centralise scattered string literals into compile-time constants using
StaticStr<N> and join() for dot-separated composition. Shared primitives
live in SpanNames.h; RPC-specific names in RpcSpanNames.h. Future modules
(consensus, peer, ledger) add their own *SpanNames.h without bloating
the central header.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace rpcSpan(fullName) calls with span(TraceCategory::Rpc, prefix,
name). Add 'using namespace telemetry' to both RPC files so call sites
read cleanly without repeated namespace qualifiers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Delete TracingInstrumentation.h and replace all XRPL_TRACE_* macro
invocations with direct SpanGuard::rpcSpan() calls. SpanGuard's pimpl
design and global Telemetry accessor eliminate the need for macro
wrappers and explicit Telemetry instance passing at call sites.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TracingInstrumentation.h:
- Unify all span-creation macros to use std::optional<SpanGuard>
(fixes type mismatch between XRPL_TRACE_SPAN and SET_ATTR)
- Wrap XRPL_TRACE_SET_ATTR/EXCEPTION in do-while(0) (dangling-else)
- Move macros outside namespace blocks (macros are global)
- Cache telemetry reference to avoid double-evaluation
- Remove leaked _xrpl_span_ intermediate variable
- Add @note tags for thread safety, scope, and usage constraints
- Add 3 usage examples per CLAUDE.md requirements
ServerHandler.cpp:
- Remove misleading rpc.request span from onRequest() (span ended
before coroutine runs, producing orphan spans)
- Add rpc.http_request span to HTTP processSession() (runs inside
the coroutine, correct parent for rpc.process/rpc.command spans)
- Add XRPL_TRACE_EXCEPTION and error status in both catch blocks
(WS processSession and processRequest)
SpanGuard.h:
- Add null guards to all mutating methods (setOk, setStatus,
setAttribute, addEvent, recordException) for safety after discard()
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Concatenate nested namespaces (modernize-concat-nested-namespaces)
- Add [[nodiscard]] to factory and accessor methods
- NOLINT no-op stub instance methods that must stay non-static for API
parity with the real implementation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove the single-arg span(name) factory that creates unconditional
spans without category gating. All call sites use the 3-arg
span(TraceCategory, prefix, name) variant which checks whether the
category is enabled in config before creating a span. The 1-arg form
was dead code with no callers.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace references to non-existent TracingInstrumentation.h with
SpanGuard.cpp pimpl implementation that actually exists on this branch.
Update conditional compilation section to describe the pimpl approach.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replace rpcSpan(), txSpan(), consensusSpan(), peerSpan(), ledgerSpan()
with a single span(TraceCategory, prefix, name) factory method. Adding
a new traceable subsystem now requires only a new enum value and one
switch case — no new methods or header changes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- SpanContext::isValid(): add inline no-op when XRPL_ENABLE_TELEMETRY
is not defined, preventing a linker error if called in that path
- linkedSpan(): set kIsRootSpanKey on the StartSpanOptions parent
context so linked spans start a genuinely independent sub-tree
instead of silently becoming children of the current active span
- Telemetry::instance_: use std::atomic with acquire/release ordering
to avoid a data race between start()/stop() and factory methods
called from worker threads
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Redesign SpanGuard with pimpl idiom to hide all OpenTelemetry types
from public headers. Add global Telemetry accessor so SpanGuard factory
methods work without explicit Telemetry references. Add child/linked
span creation and cross-thread context propagation. Update plan docs
to reflect macro removal in favor of SpanGuard factory pattern.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add DiscardFlag.h and FilteringSpanProcessor references to the file
tree, key files table, and implementation summary in OpenTelemetryPlan.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add span discard mechanism that drops unwanted spans before they enter
the batch export queue, saving both network bandwidth and storage.
FilteringSpanProcessor is a custom SpanProcessor decorator that wraps
BatchSpanProcessor. SpanGuard::discard() sets a thread-local flag
(tl_discardCurrentSpan) before calling Span::End(). The OTel SDK calls
OnEnd() synchronously on the same thread, where the flag is checked and
cleared to drop the span.
New file: DiscardFlag.h — zero-dependency header for the thread-local
flag, avoiding transitive include bloat from Telemetry.h.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove duplicate otlp/tempo exporter block, duplicate tempo service
definition, and jaeger dependency from docker-compose example.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tempo is now the sole trace backend. Remove Jaeger all-in-one service
from docker-compose, otlp/jaeger exporter from OTel Collector config,
and Jaeger Grafana datasource provisioning file.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Run .github/scripts/rename/docs.sh to replace rippled → xrpld
references in all plan documentation files, fixing the check-rename
CI failure.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Sub-task 7.10a: Per-Validator Validation Count (Flag Ledger Window)
to the Phase 7 task list. This metric tracks how many of the last 256
ledgers each UNL validator has validated — the key participation metric
for UNL health monitoring.
Implementation plan:
- Observable gauge rippled_validator_participation with validator label
- Data from RCLValidations::getTrustedForLedger() over 256-ledger window
- Emitted at flag ledger boundaries (~15 min interval)
- Grafana table panel with threshold coloring (green/yellow/red)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Populate baselines/baseline-timings.json from the green CI run
(24906110133, commit f11ebc1253). 25/31 metrics have non-null values;
6 span.rpc.* are null due to sparse data in the 3m window.
Remove the rpc_methods section from regression-metrics.json and its
thresholds. rippled_rpc_method_duration_us_bucket is never populated
because PerfLogImp::rpcEnd never calls MetricsRegistry::recordRpcFinished
— only recordRpcStarted is wired up (Phase 9 instrumentation gap).
The span-based rpc.request/rpc.process metrics via spanmetrics already
cover RPC latency.
Two CI failures traced to root cause:
1. rippled_jobq_job_count: 0 series — StatsDGaugeImpl declared
m_dirty{false} despite the constructor comment saying "start dirty".
Gauges whose value starts and stays at 0 never emitted, so Prometheus
never scraped them. Fix: m_dirty{true} on the member initializer.
2. TX error rate 82.8% — the submitter tracked account sequences
locally, but in a multi-node consensus network other nodes' txns
advance sequences independently. After a few ledger closes the
locally-tracked sequence fell behind the ledger, producing
tefPAST_SEQ for every subsequent submission. Fix: refresh account
sequences from account_info every 10 s during the submission loop.
- capture_timings.py: fail when captured/total ratio < 50%
(--min-capture-ratio). Prevents silent pass on unreachable Prometheus.
- run-full-validation.sh: set REGRESSION_EXIT=2 on capture failure so
the final exit code reflects it. Update exit code docs in header.
- compare_to_baseline.py: extract _skip_delta helper to bring
compute_delta under 80 lines. Fix 0.0-as-falsy bug in abs_bound
resolution (use explicit None check instead of `or`). Remove dead
variable override_prefix_key.
- prom_queries.py: extract _build_simple_entries and _build_job_entries
to bring build_query_plan under 80 lines. Fix module docstring return
type example. Use aiohttp.ClientTimeout instead of bare int.
- telemetry-validation.yml: add set -euo pipefail to regression summary
step; guard jq calls with -e flag and fallback; fail on missing
baseline file; emit ::warning annotation when timings.json missing.
- baselines/README.md: document the placeholder field.
Captures per-span / per-RPC / per-job timings from Prometheus after the
workload run and diffs them against a committed baseline. Regression
requires breaching both a percentage and an absolute bound, tolerating
small-value noise. When the baseline is a placeholder, the comparator
emits the captured JSON in the exact schema for one-time paste into
baselines/baseline-timings.json, and the CI Step Summary surfaces that
block for the reviewer.
Scope: gate only — automated baseline persistence, benchmark.sh
PromQL migration, and the historical trend dashboard remain follow-ups.
Plan documents referenced Application.h and app_ for getTelemetry()
but the codebase now uses ServiceRegistry as the interface. Updated:
- 05-configuration-reference.md: getTelemetry() on ServiceRegistry,
deferred serviceInstanceId pattern in ApplicationImp
- POC_taskList.md Task 4: target ServiceRegistry.h not Application.h,
correct config file path and constructor pattern
- 04-code-samples.md: fix overlay() -> getOverlay(), rewrite JobQueue
sample to reflect actual architecture (no app_ member)
- 03-implementation-strategy.md: fix file impact table path
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Strip effort/risk columns from task tables and remove the §6.9 Effort
Summary section with its pie chart and resource requirements table.
Renumber §6.10 Quick Wins → §6.9.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split document index into Plan Documents and Task Lists sections.
These files were introduced in this branch but missing from the index.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The Jaeger-removal rebase used --ours conflict resolution which
dropped content added by intermediate phases (6-8): StatsD receiver,
filelog receiver, Loki service/exporter, health_check extension,
and OTLP metrics pipeline. Restore from pre-rebase origin minus
Jaeger references.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fix remaining Jaeger references that accumulated across intermediate
branches in the stacked PR chain. These were in files modified by
multiple phases where the per-branch fixes didn't cover all additions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Migrate validate_telemetry.py to Tempo TraceQL search API, remove
Jaeger service from workload docker-compose, update readiness checks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Pre-include boost/asio/detail/socket_types.hpp on Windows before OTel
SDK headers to ensure WinSock2.h is included before WinSock.h.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add ValidationTracker.cpp to xrpl.test.telemetry target sources
(implementation lives in src/xrpld/ but has no OTel SDK dependency)
- Change BEAST_DEFINE_TESTSUITE namespace from ripple to xrpl
- Replace recursive *.md glob with non-recursive GLOB in XrplDocs.cmake
to avoid picking up .claude/instructions.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Update MockServiceRegistry to match current ServiceRegistry interface
(17 method renames: get* prefix, PathRequests→PathRequestManager)
- Make throwUnimplemented() static to satisfy clang-tidy
- Regenerate levelization ordering.txt and loops.txt
- Remove 'rippled' prefix from 3 StatsD dashboard titles
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix validations_checked_total recording site (NetworkOPs, not LedgerMaster)
- Add Slide 11 to presentation: External Dashboard Parity overview with
Mermaid diagrams for new metric categories, ValidationTracker sequence,
and new dashboard summary
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove duplicate 'system-node-health' UID from expected_metrics.json
(already covered by 'rippled-system-node-health')
- Add parity span attributes to expected_spans.json: node health on
rpc.command.*, validation hash/full on consensus.validation.send,
quorum/proposers on consensus.accept, validation hash/full on
peer.validation.receive
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add agreement_pct_7d, agreements_7d, missed_7d labels to the
rippled_validation_agreement observable gauge, matching the external
xrpl-validator-dashboard's 7-day tracking.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add ValidationTracker member to MetricsRegistry with a public accessor,
register a rippled_validation_agreement observable gauge that calls
reconcile() and reports 1h/24h agreement percentages and counts, and
hook recordOurValidation/recordNetworkValidation into RCLConsensus
validate() and LedgerMaster setValidLedger() respectively.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Wire incrementValidationsChecked() into NetworkOPs::recvValidation() so
each received network validation increments the counter.
Note: incrementJqTransOverflow() hook is deferred — JobQueue has no
explicit overflow event path; the counter is reserved for future use.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add rippled_validation_agreements_total and rippled_validation_missed_total
counter declarations and creation (wiring to ValidationTracker pending rebase)
- Fix peer-quality dashboard: query rippled_server_info{metric="peer_disconnects_resources"}
instead of non-existent rippled_Overlay_Peer_Disconnects_Charges
- Remove dead getCountsJson() call in storageDetail callback
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Rename fee labels to match spec: base_fee_drops -> base_fee_xrp,
reserve_base_drops -> reserve_base_xrp, reserve_inc_drops -> reserve_inc_xrp
- Add peers_insane_count (stub with TODO for PeerImp::tracking_ exposure)
- Add transaction_rate to ledger economy gauge
- Replace node_store_writes/node_written_bytes with nudb_bytes per spec
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add validator health, peer quality, ledger economy, state tracking, and
storage detail observable gauges plus 5 synchronous counters with recording
hooks for ledger close, validation send, state change, and overflow events.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add window7d_ deque, agreementPct7d(), agreements7d(), missed7d() to
match the external xrpl-validator-dashboard's 7-day agreement tracking.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cover normal agreement, missed validation, late repair, empty window,
grace period boundary, max pending trimming, mixed results, duplicate
recording, and only-we-validated scenarios.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Use >= instead of > for grace period comparison to reconcile at exactly
8 seconds rather than skipping the boundary
- Two-pass hard trim: first remove entries past late-repair window, then
any reconciled entry, to avoid sabotaging late repairs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Standalone class that tracks whether this validator's validations agree
with network consensus, maintaining rolling 1h/24h windows and lifetime
totals with a late-repair mechanism for out-of-order arrivals.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds ValidationTracker (agreement computation with 8s grace period),
validator health, peer quality, ledger economy, state tracking,
storage detail gauges, 7 synchronous counters, and agreement gauge.
29 new metrics covering validation agreement, peer quality, UNL health,
ledger economy, state tracking, and upgrade awareness.
Part of the external dashboard parity initiative across phases 2-11.
See docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bridge the existing beast::insight gauge for resource-limit peer
disconnects (peerDisconnectsCharges_) into the StatsD metric inventory.
Part of the external dashboard parity initiative.
See docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The xrpld/app/misc/CanonicalTXSet.h header doesn't exist — it was
incorrectly added during a rebase conflict resolution. The correct
include xrpl/ledger/CanonicalTXSet.h is already present.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add ledger hash and full-validation flag to peer.validation.receive
spans for trace-level agreement analysis across validators.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move xrpld data paths from ./data/ to docker/telemetry/data/ so runtime
files stay within the docker telemetry directory. Add .gitignore to
exclude the data directory from version control.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move validation_quorum and proposers_validated attributes from
consensus.accept.apply to consensus.accept span to match the design
spec. Both values are available in onAccept() scope.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add validation ledger hash and full-validation flag to
consensus.validation.send spans, plus quorum and proposer count to
consensus.accept spans for trace-level agreement analysis.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds ledger_hash, validation.full to validation send/receive spans,
and validation_quorum, proposers_validated to consensus.accept spans.
Foundation for Phase 7 ValidationTracker agreement computation.
Part of the external dashboard parity initiative across phases 2-11.
See docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tag transaction receive spans with the relaying peer's rippled version
to enable version-mismatch correlation during network upgrades.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds xrpl.peer.version attribute to tx.receive spans for version-mismatch
correlation during network upgrades.
Part of the external dashboard parity initiative across phases 2-11.
See docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add amendment_blocked and server_state span attributes to every
rpc.command.* span so operators can correlate RPC behavior with node state.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds node health context (amendment_blocked, server_state) to rpc.command.*
spans, inspired by the community xrpl-validator-dashboard.
Part of the external dashboard parity initiative across phases 2-11.
See docs/superpowers/specs/2026-03-30-external-dashboard-parity-design.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Tempo is now the sole trace backend. Remove Jaeger all-in-one service
from docker-compose, otlp/jaeger exporter from OTel Collector config,
and Jaeger Grafana datasource provisioning file.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Strip effort/risk columns from task tables and remove the §6.9 Effort
Summary section with its pie chart and resource requirements table.
Renumber §6.10 Quick Wins → §6.9.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split document index into Plan Documents and Task Lists sections.
These files were introduced in this branch but missing from the index.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This change:
* Removes a set of unnecessary brackets in the initialization of an `std::uint32_t`.
* Fixes a couple of incorrect flags (same value, just wrong variables - so no amendment needed).
This change replaces all instances of `<variable> != tesSUCCESS` with `!isTesSuccess(<variable>)` and `<variable> == tesSUCCESS` with `isTesSuccess(<variable>)`.
This change fixes delegation:
* If the Delegate object is not present, we should disallow empty permission list in DelegateSet preclaim.
* Empty permission list is only allowed to delete the existing Delegate object.
* In `doApply`, permission list being empty returns `tecINTERNAL`, which should not happen.
This change:
* Makes `addSLE` in `DIDSet` a static function, instead of a free function.
* Renames `Attestation` to `Data` everywhere (an artifact of a previous name for the field).
* Actually runs a set of tests that were not included in the `run` function of `DID_test`.
This change:
* Introduces a new helper function on `STTx`, `getFeePayer`.
* Removes the usage of `mSourceBalance` and replaces it with SLE balance lookups.
* Renames `mPriorBalance` to `preFeeBalance_`
This simplifies some of the code in the transactors and makes it a lot more readable.
Throwing exceptions from code sometime confuses ASAN, as it cannot keep track of stack frames. This change therefore adds a macro to skip instrumentation around the `Throw` function.
This change moves the sanitizer runtime options out to dedicated files, such that they can be used in multiple places (CI, local runs) without any need to rewrite them.
Per [XLS-0095](https://xls.xrpl.org/xls/XLS-0095-rename-rippled-to-xrpld.html), we are taking steps to rename ripple(d) to xrpl(d). This change modifies the system name from `rippled` to `xrpld`.
The system name is used in limited places:
* When no explicit config file is passed via the `--config` flag, then the system name is used to construct the path where the config file and database may be stored, via the `$XDG_CONFIG_HOME` and `$XDG_DATA_HOME` directories, respectively.
* It is used in the metadata and user-agent as part of RPC calls.
* It is newly used in the full version string.
ASAN wasn't able to keep track of `boost::coroutine` context switches, and would lead to many false positives being detected. By switching to `boost::coroutine2` and `ucontext`, ASAN is able to know about the context switches advertised by the `boost::fiber` class, which in turn leads to more cleaner ASAN analysis.
The `HTTPClient` class initializes a global SSL context via `initializeSSLContext()`. However, it had no way to release it, which caused memory leaks flagged by the LeakSanitizer. Multiple LSAN suppressions in the sanitizers' suppressions file were masking these leaks. Our test code also manually called `initializeSSLContext()` in each test without guaranteed cleanup on failure paths.
This change fixes these memory leaks by adding a `cleanupSSLContext()` method to properly release the global SSL context, and removes the corresponding LSAN suppressions. The change further refactors the `HTTPClient` tests to use a Google Test fixture (`HTTPClientTest`) that manages the SSL context lifecycle via RAII (SetUp/TearDown), making it impossible for tests to leak the context.
This change deletes the `SecretKey` equality/inequality operators from the public library header and moves the comparison logic into test-only code.
Specifically, the `operator==` and `operator!=` free functions on `SecretKey` have been removed from `include/xrpl/protocol/SecretKey.h` and have been replaced with explicitly deleted member functions to prevent accidental use in production code. A named `test::equal()` helper has also been added in `src/test/unit_test/utils.h` for test assertions that need to compare secret keys.
This change removes an unnecessary newline in a logging statement. Namely, `std::endl` is unneeded in `JLOG`, since it automatically places a newline at the end of the string.
This change enforces a maximum length of 63 characters on feature names, as well as not permitting an exactly 32 character long feature name to avoid confusion with those that use a `uint256` hex representation, as that is an alternative way to specify a feature. This change further prevents the use of Unicode characters in feature names, because some can be confused with regular ASCII characters despite being valid in identifiers.
The `setFieldU32` call is currently used to set the credential's expiry using a `getFieldU32` call to obtain the expiration time. However, that value was already obtained previously and can thus be reused.
DID, Escrows, PaymentChannels, and Credentials previously contained multiple unrelated transactor classes in a single header/implementation pair. This change splits each into one class per file, following the same pattern established by the rest of the codebase.
Now that prefixes in PR titles are being validated as part of CI, the "Type of Change" section in the PR template is no longer needed. The prefixes and descriptions in the `CONTRIBUTING.md` file have been updated to reflect the currently supported list.
Subscribe tests have a problem that there is no way to synchronize application running in background threads and test threads. Threads are communicating via websocket messages. When the code is compiled in debug mode with code coverage enabled it executes quite slow, so receiving websocket messages by the client in subscribe tests may time out.
This change does 2 things to fix the problem:
* Increases timeout for receiving a websocket message.
* Decreases the number of tests running in parallel.
While testing the fix for subscribe test another flaky test in ledger replay was found, which has also been addressed.
The refs as previously used pointed to the source branch, not the target branch. However, determining the target branch is different depending on the GitHub event. The pull request logic was incorrect and needed to be fixed, and the logic inside the workflow could be simplified. Both modifications have been made in this commit.
This change reorganizes the `tx/transactors` directory for consistency and discoverability. There are no behavioral changes, this is a pure refactor. Underscores were chosen as the way to separate multi-words as this is the more popular option in C++ projects.
Specific changes:
- Rename all subdirectories to lowercase/snake_case (`AMM` → `amm`, `Check` → `check`, `NFT` → `nft`, `PermissionedDomain` → `permissioned_domain`, etc.)
- Merge `AMM/` and `Offer/` into `dex/`, including `PermissionedDEXHelpers`
- Rename `MPT/` → `token/`, absorbing `SetTrust` and `Clawback`
- Move top-level transactors into named groups: `account/`, `bridge/`, `credentials/`, `did/`, `escrow/`, `oracle/`, `payment/`, `payment_channel/`, `system/`
- Update all include paths across the codebase and `transactions.macro`
The existing code added the git commit info (`GIT_COMMIT_HASH` and `GIT_BRANCH`) to every file, which was a problem for leveraging `ccache` to cache build objects. This change adds a separate C++ file from where these compile-time variables are propagated to wherever they are needed. A new CMake file is added to set the commit info if the `git` binary is available.
When `gateway_balances` gets called on an account that is involved in the `EscrowCreate` transaction (with MPT being escrowed), the method returns internal error. This change fixes this case by excluding the MPT type when totaling escrow amount.
The `Subscribe` tests were flaky, because each test performs some operations (e.g. sends transactions) and waits for messages to appear in subscription with a 100ms timeout. If tests are slow (e.g. compiled in debug mode or a slow machine) then some of them could fail. This change adds an attempt to synchronize the background Env's thread and the test's thread by ensuring that all the scheduled operations are started before the test's thread starts to wait for a websocket message. This is done by limiting I/O threads of the app inside Env to 1 and adding a synchronization barrier after closing the ledger.
The invariant check system had grown into a single monolithic file pair containing 24 invariant checker classes. The large `InvariantCheck.cpp` file was a frequent source of merge conflicts and difficult to navigate. This refactoring improves maintainability and readability with zero behavioral changes.
In particular, this change:
- Splits `InvariantCheck.h` and `InvariantCheck.cpp` into 10 focused header/source pairs organized by domain under a new `invariants/` subdirectory.
- Extracts the shared `Privilege` enum and `hasPrivilege()` function into a dedicated `InvariantCheckPrivilege.h` header, so domain-specific files can reference them independently.
This change enables all clang-tidy checks that are already passing. It also modifies the clang-tidy CI job, so it runs against all files if .clang-tidy changed.
This change replaces `void const*` by `uint256 const&` for database fetches.
Object hashes are expressed using the `uint256` data type, and are converted to `void *` when calling the `fetch` or `fetchBatch` functions. However, in these fetch functions they are converted back to `uint256`, making the conversion process unnecessary. In a few cases the underlying pointer is needed, but that can then be easy obtained via `[hash variable].data()`.
Updates XRPLF/actions prepare-runner to version 2cbf48101 which fixes
pip upgrade failures on macOS runners with Homebrew-managed Python.
* This commit was cherry-picked from "release-3.1", but ended up empty
because the changes are already present. It is included only for
accounting - to indicate that all changes/commits from the previous
release will be in the next one.
For coverage builds, we try to link against the `gcov` library (specific to the environment). But as macOS doesn't have this library and thus doesn't have the coverage tools to generate reports, the coverage builds on that platform were failing on linking.
We actually don't need to explicitly force this linking, as the `CodeCoverage` file already has correct detection logic (currently on lines 177-193), which is invoked when the `--coverage` flag is provided:
* AppleClang: Uses `xcrun -f llvm-cov` to set `GCOV_TOOL="llvm-cov gcov"`.
* Clang: Finds `llvm-cov` to set `GCOV_TOOL="llvm-cov gcov"`.
* GCC: Finds `gcov` to set `GCOV_TOOL="gcov"`.
The `GCOV_TOOL` is then passed to `gcovr` on line 416, so the correct tool is used for processing coverage data.
This change therefore removes the `gcov` suffix from lines 473 and 475 in the `CodeCoverage.cmake` file.
The rdb module was not properly designed, which is fixed in this change. The module had three classes:
1) The abstract class `RelationalDB`.
2) The abstract class `SQLiteDatabase`, which inherited from `RelationalDB` and added some pure virtual methods.
3) The concrete class `SQLiteDatabaseImp`, which inherited from `SQLiteDatabase` and implemented all methods.
The updated code simplifies this as follows:
* The `SQLiteDatabaseImp` has become `SQLiteDatabase`, and
* The former `SQLiteDatabase `has merged with `RelationalDatabase`.
This change modularizes the `WalletDB` and `Manifest`. Note that the wallet db has nothing to do with account wallets and it stores node configuration, which is why it depends on the manifest code.
This change removes the cache in `DatabaseNodeImp` and simplifies the caching logic in `SHAMapStoreImp`. As NuDB and RocksDB internally already use caches, additional caches in the code are not very valuable or may even be unnecessary, as also confirmed during preliminary performance analyses.
In certain cases, such as when modifying headers used by many compilation units, performing a unity build is slower than when performing a regular build with `ccache` enabled. There is also a benefit to a unity build in that it can detect things such as macro redefinitions within the group of files that are compiled together as a unit. This change therefore restores the ability to perform unity builds. However, instead of running every configuration with and without unity enabled, it is now only enabled for a single configuration to maintain lower computational use.
As part of restoring the code, it became clear that currently two configurations have coverage enabled, since the check doesn't focus specifically on Debian Bookworm so it also applies to Debian Trixie. This has been fixed too in this change.
The `ManifestCache::applyManifest` function was returning early without incrementing `seq_`. `OverlayImpl `uses this sequence to identify/invalidate a cached `TMManifests` message, which is exchanged with peers on connection. Depending on network size, startup sequencing, and topology, this can cause syncing issues. This change therefore increments `seq_` when a new manifest is accepted.
Unity builds were intended to speed up builds, by bundling multiple files into compilation units. However, now that ccache is available on all platforms, there is no need for unity builds anymore, as ccache stores compiled individual build objects for reuse. This change therefore removes the ability to make unity builds.
Currently we're passing the `Application` object around, whereby the `Application` class acts more like a service registry that gives other classes access to other services. In order to allow modularization, we should replace `Application` with a service registry class so that modules depending on `Application` for other services can be moved easily. This change adds the `ServiceRegistry` class.
This change introduces the `fixExpiredNFTokenOfferRemoval` amendment that allows expired offers to pass through `preclaim()` and be deleted in `doApply()`, following the same pattern used for expired credentials.
This change adds the project configuration directory to `.gitignore` for the `zed` editor.
As per the [documentation](https://zed.dev/docs/remote-development?highlight=.zed#zed-settings), the project configuration files are stored in the `.zed` directory at the project root dir.
This change cleans up the `API-CHANGELOG.md` file. It moves the version-specific documentation to other files and fleshes out the changelog with all the API-related changes in each version.
When support was added for `xrpld.cfg` in addition to `rippled.cfg` in https://github.com/XRPLF/rippled/pull/6098, as part of an effort to rename occurrences of ripple(d) to xrpl(d), the clearing and creation of the data directory were modified for what, at the time, seemed to result in an equivalent code flow. This has turned out to not be true, which is why this change restores two modifications to `Config.cpp` that currently break running the binary in standalone mode.
This change adds `cmake-format` as. a pre-commit hook. The style file closely matches that in Clio, and they will be made to be equivalent over time. For now, some files have been excluded, as those need some manual adjustments, which will be done in future changes.
This change makes the `read` function call in `handleConnection` async, adds a new class `TestSink` to help debugging, and adds a new target `xrpl.tests.helpers` to put the helper class in.
To allow developers to consume the latest unstable and (near-)stable versions of our `xrpl` Conan recipe, we should export and upload it whenever a push occurs to the corresponding branch or a release tag has been created. This way, developers do not have to figure out themselves what the most recent shortened commit hash was to determine the latest unstable recipe version (e.g. `3.2.0-b0+a1b2c3d`) or what the most recent release (candidate) was to determine the latest (near-)stable recipe version (e.g. `3.1.0-rc2`).
Now, pushes to the `develop` branch will produce the `develop` recipe version, pushes to the `release` branch will produce the `rc` recipe version, and creation of versioned tags will produce the `release` recipe version.
This change replaces the mutex `stoppingMutex_`, the `atomic_bool` variable `isTimeToStop`, and the conditional variable `stoppingCondition_` with an `atomic_flag` variable.
When `xrpld` is running the embedded tests as a child process, it has a control thread (the app bundle thread) that starts the application, and an application thread (the thread that executes `app_->run()`). Due to the relaxed memory ordering on ARM, it's not guaranteed that the application thread can see the change of the value resulting from the `isTimeToStop.exchange(true)` call before it is notified by `stoppingCondition_.notify_all()`, even though they do happen in the right order in the app bundle thread in `ApplicationImp::signalStop`. We therefore often get into the situation where `isTimeToStop` is `true`, but the application thread is waiting for `stoppingCondition_` to notify, because the app bundle thread may have already notified before the application thread actually starts waiting.
Switching to a single `atomic_flag` variable makes sure that there's only one synchronisation object and then the memory order guarantee provided by c++ can make sure that `notify_all` gets synchronised after `test_and_set` does.
Fixing this issue will stop the unit tests hanging forever and then we should see less (or hopefully no) time out errors in daily github action runs
Since the minimum Clang version we support is 16, the checks for version < 15 are no longer necessary. This change therefore removes the macros checking if the clang version is < 15 and simplifies uses of `std::source_location`.
The `upload-conan-deps` workflow that's triggered on push is supposed to upload the Conan dependencies to our remote, so future PR commits can pull those dependencies from the remote. However, as the `sanitize` argument is missing, it was building different dependencies than what the PRs are building for the asan/tsan/ubsan job, so the latter would not find anything in the remote that they could use. This change sets the missing `sanitizers` input variable when running the `build-deps` action.
Separately, the `setup-conan` action showed the default profile, while we are using the `ci` profile. To ensure the profile is correctly printed when sanitizers are enabled, the environment variable the profile uses is set before calling the action.
The export and upload steps were initially in a separate action, where GitHub Actions does not support the `secrets` keyword, but only `inputs` for the credentials. After they were moved to a reusable workflow, only part of the references to the credentials were updated. This change correctly references to the Conan credentials via `secrets` instead of `inputs`.
By default the Conan recipe extracts the version from `BuildInfo.cpp`, but in some of the cases we want to upload a recipe with a suffix derived from the commit hash. This currently then results in the uploading to fail, since there is a version mismatch.
Here we explicitly set the version, and then simplify the steps in the upload workflow since we now need the recipe name (embedded within the conanfile.py but also needed when uploading), the recipe version, and the recipe ref (name/version).
Conan recipes use semantic versioning, and since our version already contains a hyphen the second hyphen causes Conan to ignore it. The plus sign is a valid separator we can use instead, so this change uses a `+` to separate a version suffix (commit hash) instead of a `-`.
There were a few uninitialized variables in CMake files. This change will make sure we always check if a variable has been initialized before using them, or in come cases initialize them by default. This change will raise an error on CI if a developer introduced an uninitialized variable in CMake files.
This change continues the thread naming work from #5691 and #5758, which enables more useful lock contention profiling by ensuring threads/jobs have short, stable, human-readable names (rather than being truncated/failing due to OS limits). This changes diagnostic naming only (thread names and job/load-event labels), not behavior.
Specific modifications are:
* Shortens all thread/job names used with `beast::setCurrentThreadName`, so the effective Linux thread name stays within the 15-character limit.
* Removes per-ledger sequence numbers from job/thread names to avoid long labels. This improves aggregation in lock contention profiling for short-lived job executions.
The Ripple Bug Bounty program recently changed the public keys that security researchers can use to encrypt vulnerabilities and messages for submission to the program. This information was updated on https://ripple.com/legal/bug-bounty/ and this PR updates the `SECURITY.md` to align.
During several iterations of development of https://github.com/XRPLF/rippled/pull/6235, the commit hash was supposed to be moved into the `run:` statement, but it slipped through the cracks and did not get added. This change adds the commit hash as suffix to the Conan recipe version.
The `Number.h` header file now has `std::reference_wrapper` from `<functional>`, but the include is missing, causing downstream build problems. This change adds the header.
This change uploads the `libxrpl` library as a Conan recipe to our remote when (i) merging into the `develop` branch, (ii) committing to a PR that targets a `release*` branch, and (iii) a versioned tag is applied. Clio is only notified in the second case. The user and channel are no longer used when uploading the recipe.
Specific changes are:
* A `generate-version` action is added, which extracts the build version from `BuildInfo.cpp` and appends the short 7-character commit hash to it for merges into the `develop` branch and for commits to a PR that targets a `release*` branch. When a tag is applied, however, the tag itself is used as the version. This functionality has been turned into a separate action as we will use the same versioning logic for creating .rpm and .deb packages, as well as Docker images.
* An `upload-recipe` action is added, which calls the `generate-version` action and further handles the uploading of the recipe to Conan.
* This action is called by both the `on-pr` and `on-trigger` workflows, and a new `on-tag` workflow.
The reason for this change is that we have downstream uses for the `libxrpl` library, but currently only upload the recipe to check for compatibility with Clio when making commits to a PR that targets the release branch.
`PeerImp` processes `TMGetObjectByHash` queries with an unbounded per-request loop, which performs a `NodeStore` fetch and then appends retrieved data to the reply for each queried object without a local count cap or reply-byte budget. However, the `Nodestore` fetches are expensive when high in numbers, which might slow down the process overall. Hence this code change adds an upper cap on the response size.
This change removes the `master` branch as a trigger for the CI pipelines, and updates comments accordingly. It also fixes the pre-commit workflow, so it will run on all release branches.
These "fixed location" objects can be found in multiple ways:
1. The lookup parameters use the same format as other ledger objects, but the only valid value is true or the valid index of the object:
- Amendments: "amendments" : true
- FeeSettings: "fee" : true
- NegativeUNL: "nunl" : true
- LedgerHashes: "hashes" : true (For the "short" list. See below.)
2. With RPC API >= 3, using special case values to "index", such as "index" : "amendments". Uses the same names as above. Note that for "hashes", this option will only return the recent ledger hashes / "short" skip list.
3. LedgerHashes has two types: "short", which stores recent ledger hashes, and "long", which stores the flag ledger hashes for a particular ledger range.
- To find a "long" LedgerHashes object, request '"hashes" : <ledger sequence>'. <ledger sequence> must be a number that evaluates to an unsigned integer.
- To find the "short" LedgerHashes object, request "hashes": true as with the other fixed objects.
The following queries are all functionally equivalent:
- "amendments" : true
- "index" : "amendments" (API >=3 only)
- "amendments" : "7DB0788C020F02780A673DC74757F23823FA3014C1866E72CC4CD8B226CD6EF4"
- "index" : "7DB0788C020F02780A673DC74757F23823FA3014C1866E72CC4CD8B226CD6EF4"
Finally, whether the object is found or not, if a valid index is computed, that index will be returned. This can be used to confirm the query was valid, or to save the index for future use.
This change adds support for sanitizer build options in CI builds workflow. Currently `asan+ubsan` is enabled, while `tsan+ubsan` is left disabled as more changes are required.
The word `failed` in the test case makes it hard to search through the test logs when an actual test failure occurs, so this change renames the word to just `fail` instead.
- Refactor Number internals away from int64 to uint64 & a sign flag
- ctors and accessors use `rep`. Very few things expose
`internalrep`.
- An exception is "unchecked" and the new "normalized", which explicitly
take an internalrep. But with those special control flags, it's easier
to distinguish and control when they are used.
- For now, skip the larger mantissas in AMM transactions and tests
- Remove trailing zeros from scientific notation Number strings
- Update tests. This has the happy side effect of making some of the string
representations _more_ consistent between the small and large
mantissa ranges.
- Add semi-automatic rounding of STNumbers based on Asset types
- Create a new SField metadata enum, sMD_NeedsAsset, which indicates
the field should be associated with an Asset so it can be rounded.
- Add a new STTakesAsset intermediate class to handle the Asset
association to a derived ST class. Currently only used in STNumber,
but could be used by other types in the future.
- Add "associateAsset" which takes an SLE and an Asset, finds the
sMD_NeedsAsset fields, and associates the Asset to them. In the case
of STNumber, that both stores the Asset, and rounds the value
immediately.
- Transactors only need to add a call to associateAsset _after_ all of
the STNumbers have been set. Unfortunately, the inner workings of
STObject do not do the association correctly with uninitialized
fields.
- When serializing an STNumber that has an Asset, round it before
serializing.
- Add an override of roundToAsset, which rounds a Number value in place
to an Asset, but without any additional scale.
- Update and fix a bunch of Loan-related tests to accommodate the
expanded Number class.
---------
Co-authored-by: Vito <5780819+Tapanito@users.noreply.github.com>
- Spec: XLS-66
Fix overpayment asserts (#6084)
MPTTester::operator() parameter should be std::int64_t
- Originally defined as uint64_t, but the testIssuerLoan() test called
it with a negative number, causing an overflow to a very large number
that in some circumstances could be silently cast back to an int64_t,
but might not be. I believe this is UB, and we don't want to rely on
that.
Review feedback from @Tapanito: overpayment value change
- In overpayment results, the management fee was being calculated twice:
once as part of the value change, and as part of the fees paid.
Exclude it from the value change.
Fix Overpayment Calculation (#6087)
- Adds additional unit tests to cover math calculations.
- Removes unused methods.
Review feedback from @shawnxie999: even more rounding
- Round the initial total value computation upward, unless there is
0-interest.
- Rename getVaultScale to getAssetsTotalScale, and convert one incorrect
computation to use it.
- Use adjustImpreciseNumber for LossUnrealized.
- Add some logging to computeLoanProperties.
Fix LoanBrokerSet debtMaximum limits (#6116)
Fix some minor bugs in Lending Protocol (#6101)
- add nodiscard to unimpairLoan, and check result in LoanPay
- add a check to verify that issuer exists
- improve LoanManage error code for dust amounts
Check permissions in LoanSet and LoanPay (#6108)
Disallow pseudo accounts to be Destination for LoanBrokerCoverWithdraw (#6106)
Ensure vault asset cap is not exceeded (#6124)
Fix Overpayment ValueChange calculation in Lending Protocol (#6114)
- Adds loan state to LoanProperties.
- Cleans up computeLoanProperties.
- Fixes missing management fee from overpayment.
fix: Enable LP Deposits when the broker is the asset issuer (#6119)
* Replace accountHolds with accountSpendable when checking
for account funds in VaultDeposit and LoanBrokerCoverDeposit
Add a few minor changes (#6158)
- Updates or fixes a couple of things I noticed while reviewing changes
to the spec.
- Rename sfPreviousPaymentDate to sfPreviousPaymentDueDate.
- Make the vault asset cap check added in #6124 a little more robust:
1. Check in preflight if the vault is _already_ over the limit.
2. Prevent overflow when checking with the loan value. (Subtract
instead of adding, in case the values are near maxint. Both return
the same result. Also add a unit test so each case is covered.
Add minimum grace period validation (#6133)
Fix bugs: frozen pseudo-account, and FLC cutoff (#6170)
refactor: Rename raw state to theoretical state (#6187)
Check if a withdrawal amount exceeds any applicable receiving limit. (#6117)
Fix overpayment result calculation (#6195)
Address review feedback from Lending Protocol re-review (#6161)
---------
Co-authored-by: Gregory Tsipenyuk <gregtatcam@users.noreply.github.com>
Co-authored-by: Bronek Kozicki <brok@incorrekt.com>
Co-authored-by: Vito Tumas <5780819+Tapanito@users.noreply.github.com>
Co-authored-by: Shawn Xie <35279399+shawnxie999@users.noreply.github.com>
Co-authored-by: Jingchen <a1q123456@users.noreply.github.com>
This change removes unnecessary version numbers in the OpenSSL and Boost `find_package` CMake statements. An unnecessary OpenSSL definition is removed, while Conan options for SSL are updated to disable insecure ciphers. Moreover, the statements are now ordered alphabetically and more logically.
- Introduces amendment `fixBatchInnerSigs`
- Update Batch unit tests
- Fix all the Env instantiations to _use_ the "features" parameter.
- testInnerSubmitRPC runs with Batch enabled and disabled.
- Add a test to testInnerSubmitRPC for a correctly signed tx incorrectly
using the tfInnerBatchTxn flag.
- Generalize the submitAndValidate lambda in testInnerSubmitRPC.
- With the fix amendment, a transaction never reaches the transaction
engine (Transactor and derived classes.)
- Test submitting a pseudo-transaction. Stopped before reaching the
transaction engine, but with different errors.
- The tests verify that without the amendment, a transaction with
tfInnerBatchTxn is immediately rejected. Without the amendment, things
are safe. The amendment just makes things safer and more future-proof.
- Adds a mechanism for the vault owner to burn user shares when the vault is stuck. If the Vault has 0 AssetsAvailable and Total, the owner may submit a VaultClawback to reclaim the worthless fees, and thus allow the Vault to be deleted. The Amount must be left off (unless the owner is the asset issuer), specified as 0 Shares, or specified as the number of Shares held.
This change:
* Truncates thread names if more than 15 chars with `snprintf`.
* Adds warnings for truncated thread names if `-DTRUNCATED_THREAD_NAME_LOGS=ON`.
* Add a static assert for string literals to stop compiling if > 15 chars.
* Shortens `Resource::Manager` to `Resource::Mngr` to fix the static assert failure.
* Updates `CurrentThreadName_test` unit test specifically for Linux to verify truncation.
This change updates the XRPLF pre-commit workflow and prepare-runner action to their latest versions. For naming consistency the prepare-runner action changed the disable_ccache variable into enable_ccache, which matches our naming.
This change fixes the last of the spelling issues, and enables the pre-commit (and CI) check for spelling. There are no functionality changes, but it does rename some enum values.
Right now, each pipeline invocation builds the source code from scratch. Although compiled Conan dependencies are cached in a remote server, the source build objects are not. We are able to further speed up our builds by leveraging `ccache`. This change enables caching of build objects using `ccache` on Linux, macOS, and Windows.
This change adds some basic tests for all the `ledger_entry` helper functions, so each ledger entry type is covered. There are further some minor refactors in `parseAMM` to provide better error messages. Finally, to improve readability, alphabetization was applied in the helper functions.
This change renames all occurrences of `rippled.cfg` to `xrpld.cfg`. It also provides a script to allow developers to replicate the changes in their local branch or fork to avoid conflicts. For the time being it maintains support for `rippled.cfg` as config file, if `xrpld.cfg` does not exist.
This change modifies the build directory structure from `build/build/xxx` or `.build/build/xxx` to just `build/xxx`. Namely, the `conanfile.py` has the CMake generators build directory hardcoded to `build/generators`. We may as well leverage the top-level build directory without introducing another layer of directory nesting.
`Json::Object` and related objects are not used at all, so this change removes `include/xrpl/json/Object.h` and all downstream files. There are a number of minor downstream changes as well.
Full list of deleted classes and functions:
* `Json::Collections`
* `Json::Object`
* `Json::Array`
* `Json::WriterObject`
* `Json::setArray`
* `Json::addObject`
* `Json::appendArray`
* `Json::appendObject`
The last helper function, `copyFrom`, seemed a bit more complex and was actually used in a few places, so it was moved to `LedgerToJson.h` instead of deleting it.
The latest update to `cleanup-workspace`, `get-nproc`, and `prepare-runner` moved the action to the repository root directory, and also includes some ccache changes. In response, this change updates the various shared actions to the latest commit hash.
# This file is sorted in reverse chronological order, with the most recent commits at the top.
# The commits listed here are ignored by git blame, which is useful for formatting-only commits that would otherwise obscure the history of changes to a file.
| A | No stray dotted span-attribute key (only the derived resource keys may be dotted). |
| G | Attribute keys are `lower_snake_case` (`^[a-z][a-z0-9_]*$` per dot-segment) — no camelCase, UPPERCASE, or spaces. |
| F | No string literals as attribute keys or span-name arguments in `setAttribute`/`addEvent`/`span`/`childSpan`. Attribute _values_ are exempt (runtime data); `*SpanNames.h` definitions and test files are exempt. |
| B | Every collector `spanmetrics.dimensions` name exists in the L1 key set. |
| C | Every Tempo span-filter tag exists in the L1 key set. |
| D | Every dashboard label resolves to an L1 span attribute, a native-metric label (L6, emitted by MetricsRegistry), or a Prometheus/Grafana builtin. TraceQL scope prefixes (`span.`/`resource.`/…) are stripped before the L1 lookup. |
| E | No dotted `xrpl.<domain>.<field>` attribute key in the runbook (only the L1 resource attrs `xrpl.network.*` may be dotted). Span names, filenames, OTel-standard keys, and metric labels are not flagged. |
Rule F runs **unconditionally** (it is a purely syntactic check on the
call-sites and needs no `*SpanNames.h`), so a code path that calls
`SpanGuard::span`/`setAttribute` directly without ever defining a header is
| H | A namespace-qualified constant (e.g. `foo::bar::myKey`) used at a telemetry call-site is not defined in any `*SpanNames.h`. The constant should live in the proper header; defining it in-place bypasses rules A/G/F. Warns rather than fails — the argument may be a legitimately dynamic value, and the header may live on a later branch. Bare locals and `std::` names are not warned. |
## Presence-gated
Every rule runs **only when the source files it needs are present** in the tree
and is otherwise skipped (printed as `SKIP: <rule> — <reason>`), never failed.
This keeps the check correct no matter how telemetry work is split across PRs —
a stacked chain, one large PR, or independent per-stage PRs where (for example)
the collector config lands before the dashboards. The collector/Tempo/dashboard/
runbook layers are introduced in later phases; on a branch without them, only
# Restore copyright notices that were removed from specific files, without
# restoring the verbiage that is already present in LICENSE.md. Ensure that if
# the script is run multiple times, duplicate notices are not added.
if ! grep -q 'Raw Material Software' include/xrpl/beast/core/CurrentThreadName.h;then
echo -e "// Portions of this file are from JUCE (http://www.juce.com).\n// Copyright (c) 2013 - Raw Material Software Ltd.\n// Please visit http://www.juce.com\n\n$(cat include/xrpl/beast/core/CurrentThreadName.h)" >include/xrpl/beast/core/CurrentThreadName.h
echo -e "// Portions of this file are from JUCE (http://www.juce.com).\n// Copyright (c) 2013 - Raw Material Software Ltd.\n// Please visit http://www.juce.com\n\n$(cat include/xrpl/beast/core/CurrentThreadName.h)" >include/xrpl/beast/core/CurrentThreadName.h
fi
if ! grep -q 'Dev Null' src/test/app/NetworkID_test.cpp;then
help="Set to generate all configurations (generally used when merging a PR) or leave unset to generate a subset of configurations (generally used when committing to a PR).",
action="store_true",
parser=argparse.ArgumentParser(
description="Generate a CI strategy matrix for all platforms or a specific one."
)
parser.add_argument(
"-c",
"--config",
help="Path to the JSON file containing the strategy matrix configurations.",
required=False,
type=Path,
help="Platform to generate for ('linux', 'macos', or 'windows'). Defaults to all platforms.",
choices=["linux","macos","windows"],
default=None,
)
parser.add_argument(
"-p",
"--packaging",
help="Emit the Linux packaging matrix instead of the build/test matrix.",
action="store_true",
)
parser.add_argument(
"-e",
"--event",
help="The GitHub event name that triggered the workflow (e.g. 'push', "
"'pull_request'). Configs are filtered by their 'event_type'. If "
@@ -4,92 +4,116 @@ This changelog is intended to list all updates to the [public API methods](https
For info about how [API versioning](https://xrpl.org/request-formatting.html#api-versioning) works, including examples, please view the [XLS-22d spec](https://github.com/XRPLF/XRPL-Standards/discussions/54). For details about the implementation of API versioning, view the [implementation PR](https://github.com/XRPLF/rippled/pull/3155). API versioning ensures existing integrations and users continue to receive existing behavior, while those that request a higher API version will experience new behavior.
The API version controls the API behavior you see. This includes what properties you see in responses, what parameters you're permitted to send in requests, and so on. You specify the API version in each of your requests. When a breaking change is introduced to the `rippled` API, a new version is released. To avoid breaking your code, you should set (or increase) your version when you're ready to upgrade.
For a log of breaking changes, see the **API Version [number]** headings. In general, breaking changes are associated with a particular API Version number. For non-breaking changes, scroll to the **XRP Ledger version [x.y.z]** headings. Non-breaking changes are associated with a particular XRP Ledger (`rippled`) release.
## API Version 2
API version 2 is available in `rippled` version 2.0.0 and later. To use this API, clients specify `"api_version" : 2` in each request.
#### Removed methods
In API version 2, the following deprecated methods are no longer available: (https://github.com/XRPLF/rippled/pull/4759)
-`tx_history` - Instead, use other methods such as `account_tx` or `ledger` with the `transactions` field set to `true`.
-`ledger_header` - Instead, use the `ledger` method.
#### Modifications to JSON transaction element in V2
In API version 2, JSON elements for transaction output have been changed and made consistent for all methods which output transactions. (https://github.com/XRPLF/rippled/pull/4775)
This helps to unify the JSON serialization format of transactions. (https://github.com/XRPLF/clio/issues/722, https://github.com/XRPLF/rippled/issues/4727)
- JSON transaction element is named `tx_json`
- Binary transaction element is named `tx_blob`
- JSON transaction metadata element is named `meta`
- Binary transaction metadata element is named `meta_blob`
Additionally, these elements are now consistently available next to `tx_json` (i.e. sibling elements), where possible:
-`hash` - Transaction ID. This data was stored inside transaction output in API version 1, but in API version 2 is a sibling element.
-`ledger_index` - Ledger index (only set on validated ledgers)
-`ledger_hash` - Ledger hash (only set on closed or validated ledgers)
-`close_time_iso` - Ledger close time expressed in ISO 8601 time format (only set on validated ledgers)
-`validated` - Bool element set to `true` if the transaction is in a validated ledger, otherwise `false`
This change affects the following methods:
-`tx` - Transaction data moved into element `tx_json` (was inline inside `result`) or, if binary output was requested, moved from `tx` to `tx_blob`. Renamed binary transaction metadata element (if it was requested) from `meta` to `meta_blob`. Changed location of `hash` and added new elements
-`account_tx` - Renamed transaction element from `tx` to `tx_json`. Renamed binary transaction metadata element (if it was requested) from `meta` to `meta_blob`. Changed location of `hash` and added new elements
-`transaction_entry` - Renamed transaction metadata element from `metadata` to `meta`. Changed location of `hash` and added new elements
-`subscribe` - Renamed transaction element from `transaction` to `tx_json`. Changed location of `hash` and added new elements
-`sign`, `sign_for`, `submit` and `submit_multisigned` - Changed location of `hash` element.
#### Modification to `Payment` transaction JSON schema
When reading Payments, the `Amount` field should generally **not** be used. Instead, use [delivered_amount](https://xrpl.org/partial-payments.html#the-delivered_amount-field) to see the amount that the Payment delivered. To clarify its meaning, the `Amount` field is being renamed to `DeliverMax`. (https://github.com/XRPLF/rippled/pull/4733)
- In `Payment` transaction type, JSON RPC field `Amount` is renamed to `DeliverMax`. To enable smooth client transition, `Amount` is still handled, as described below: (https://github.com/XRPLF/rippled/pull/4733)
- On JSON RPC input (e.g. `submit_multisigned` etc. methods), `Amount` is recognized as an alias to `DeliverMax` for both API version 1 and version 2 clients.
- On JSON RPC input, submitting both `Amount` and `DeliverMax` fields is allowed _only_ if they are identical; otherwise such input is rejected with `rpcINVALID_PARAMS` error.
- On JSON RPC output (e.g. `subscribe`, `account_tx` etc. methods), `DeliverMax` is present in both API version 1 and version 2.
- On JSON RPC output, `Amount` is only present in API version 1 and _not_ in version 2.
#### Modifications to account_info response
-`signer_lists` is returned in the root of the response. (In API version 1, it was nested under `account_data`.) (https://github.com/XRPLF/rippled/pull/3770)
- When using an invalid `signer_lists` value, the API now returns an "invalidParams" error. (https://github.com/XRPLF/rippled/pull/4585)
- (`signer_lists` must be a boolean. In API version 1, strings were accepted and may return a normal response - i.e. as if `signer_lists` were `true`.)
#### Modifications to [account_tx](https://xrpl.org/account_tx.html#account_tx) response
- Using `ledger_index_min`, `ledger_index_max`, and `ledger_index` returns `invalidParams` because if you use `ledger_index_min` or `ledger_index_max`, then it does not make sense to also specify `ledger_index`. In API version 1, no error was returned. (https://github.com/XRPLF/rippled/pull/4571)
- The same applies for `ledger_index_min`, `ledger_index_max`, and `ledger_hash`. (https://github.com/XRPLF/rippled/issues/4545#issuecomment-1565065579)
- Using a `ledger_index_min` or `ledger_index_max` beyond the range of ledgers that the server has:
- returns `lgrIdxMalformed` in API version 2. Previously, in API version 1, no error was returned. (https://github.com/XRPLF/rippled/issues/4288)
- Attempting to use a non-boolean value (such as a string) for the `binary` or `forward` parameters returns `invalidParams` (`rpcINVALID_PARAMS`). Previously, in API version 1, no error was returned. (https://github.com/XRPLF/rippled/pull/4620)
#### Modifications to [noripple_check](https://xrpl.org/noripple_check.html#noripple_check) response
- Attempting to use a non-boolean value (such as a string) for the `transactions` parameter returns `invalidParams` (`rpcINVALID_PARAMS`). Previously, in API version 1, no error was returned. (https://github.com/XRPLF/rippled/pull/4620)
## API Version 1
This version is supported by all `rippled` versions. For WebSocket and HTTP JSON-RPC requests, it is currently the default API version used when no `api_version` is specified.
The API version controls the API behavior you see. This includes what properties you see in responses, what parameters you're permitted to send in requests, and so on. You specify the API version in each of your requests. When a breaking change is introduced to the `xrpld` API, a new version is released. To avoid breaking your code, you should set (or increase) your version when you're ready to upgrade.
The [commandline](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/request-formatting/#commandline-format) always uses the latest API version. The command line is intended for ad-hoc usage by humans, not programs or automated scripts. The command line is not meant for use in production code.
### Inconsistency: server_info - network_id
For a log of breaking changes, see the **API Version [number]** headings. In general, breaking changes are associated with a particular API Version number. For non-breaking changes, scroll to the **XRP Ledgerversion [x.y.z]** headings. Non-breaking changes are associated with a particular XRP Ledger (`xrpld`) release.
The `network_id` field was added in the `server_info` response in version 1.5.0 (2019), but it is not returned in [reporting mode](https://xrpl.org/rippled-server-modes.html#reporting-mode). However, use of reporting mode is now discouraged, in favor of using [Clio](https://github.com/XRPLF/clio) instead.
## API Version 3 (Beta)
API version 3 is currently a beta API. It requires enabling `[beta_rpc_api]` in the xrpld configuration to use. See [API-VERSION-3.md](API-VERSION-3.md) for the full list of changes in API version 3.
## API Version 2
API version 2 is available in `xrpld` version 2.0.0 and later. See [API-VERSION-2.md](API-VERSION-2.md) for the full list of changes in API version 2.
## API Version 1
This version is supported by all `xrpld` versions. For WebSocket and HTTP JSON-RPC requests, it is currently the default API version used when no `api_version` is specified.
## Unreleased
This section contains changes targeting a future version.
### Additions
-`ledger_entry`, `account_objects`: The `Delegate` ledger entry now includes an optional `DestinationNode` field, which stores the index into the authorized account's owner directory. This field is present on entries created after bidirectional directory tracking was introduced and may appear in RPC responses for those entries. ([#6681](https://github.com/XRPLF/rippled/pull/6681))
-`server_definitions`: Added the following new sections to the response ([#6321](https://github.com/XRPLF/rippled/pull/6321)):
-`TRANSACTION_FORMATS`: Describes the fields and their optionality for each transaction type, including common fields shared across all transactions.
-`LEDGER_ENTRY_FORMATS`: Describes the fields and their optionality for each ledger entry type, including common fields shared across all ledger entries.
-`TRANSACTION_FLAGS`: Maps transaction type names to their supported flags and flag values.
-`LEDGER_ENTRY_FLAGS`: Maps ledger entry type names to their flags and flag values.
-`ACCOUNT_SET_FLAGS`: Maps AccountSet flag names (asf flags) to their numeric values.
### Bugfixes
- Peer Crawler: The `port` field in `overlay.active[]` now consistently returns an integer instead of a string for outbound peers. [#6318](https://github.com/XRPLF/rippled/pull/6318)
-`ping`: The `ip` field is no longer returned as an empty string for proxied connections without a forwarded-for header. It is now omitted, consistent with the behavior for identified connections. [#6730](https://github.com/XRPLF/rippled/pull/6730)
- gRPC `GetLedgerDiff`: Fixed error message that incorrectly said "base ledger not validated" when the desired ledger was not validated. [#6730](https://github.com/XRPLF/rippled/pull/6730)
-`account_channels`: The `destination_account` field now returns an error if the value is not a string. [#6529](https://github.com/XRPLF/rippled/pull/6529)
-`subscribe`: The `taker` field in the `books` array now returns an error if the value is not a string. [#6529](https://github.com/XRPLF/rippled/pull/6529)
-`account_info`: The `urlgravatar` field now uses HTTPS instead of HTTP. [#6529](https://github.com/XRPLF/rippled/pull/6529)
-`ledger`: The `full`, `accounts`, `transactions`, `expand`, `binary`, `owner_funds`, and `queue` fields now return an error if the value is not a boolean. [#6529](https://github.com/XRPLF/rippled/pull/6529)
-`ledger_data`: The `binary` field now returns an error if the value is not a boolean. [#6529](https://github.com/XRPLF/rippled/pull/6529)
-`submit`: The `fail_hard` field now returns an error if the value is not a boolean. [#6529](https://github.com/XRPLF/rippled/pull/6529)
-`subscribe`: The `taker` field in the `books` array now returns `actMalformed` instead of `badIssuer` if the value is not a valid account. [#6529](https://github.com/XRPLF/rippled/pull/6529)
- Fixed a bug in `Forwarded` HTTP header parsing where the extracted IP address could be incorrect when no comma or semicolon delimiter follows the address. This could cause the server to misidentify a client's IP address when operating behind a reverse proxy. [#6529](https://github.com/XRPLF/rippled/pull/6529)
## XRP Ledger server version 3.1.0
[Version 3.1.0](https://github.com/XRPLF/rippled/releases/tag/3.1.0) was released on Jan 27, 2026.
### Additions in 3.1.0
-`vault_info`: New RPC method to retrieve information about a specific vault (part of XLS-66 Lending Protocol). ([#6156](https://github.com/XRPLF/rippled/pull/6156))
## XRP Ledger server version 3.0.0
[Version 3.0.0](https://github.com/XRPLF/rippled/releases/tag/3.0.0) was released on Dec 9, 2025.
### Additions in 3.0.0
-`ledger_entry`: Supports all ledger entry types with dedicated parsers. ([#5237](https://github.com/XRPLF/rippled/pull/5237))
-`ledger_entry`: New error codes `entryNotFound` and `unexpectedLedgerType` for more specific error handling. ([#5237](https://github.com/XRPLF/rippled/pull/5237))
-`ledger_entry`: Improved error messages with more context (e.g., specifying which field is invalid or missing). ([#5237](https://github.com/XRPLF/rippled/pull/5237))
-`ledger_entry`: Assorted bug fixes in RPC processing. ([#5237](https://github.com/XRPLF/rippled/pull/5237))
-`simulate`: Supports additional metadata in the response. ([#5754](https://github.com/XRPLF/rippled/pull/5754))
## XRP Ledger server version 2.6.2
[Version 2.6.2](https://github.com/XRPLF/rippled/releases/tag/2.6.2) was released on Nov 19, 2025.
This release contains bug fixes only and no API changes.
## XRP Ledger server version 2.6.1
[Version 2.6.1](https://github.com/XRPLF/rippled/releases/tag/2.6.1) was released on Sep 30, 2025.
This release contains bug fixes only and no API changes.
## XRP Ledger server version 2.6.0
[Version 2.6.0](https://github.com/XRPLF/rippled/releases/tag/2.6.0) was released on Aug 27, 2025.
### Additions in 2.6.0
-`account_info`: Added `allowTrustLineLocking` flag in response. ([#5525](https://github.com/XRPLF/rippled/pull/5525))
-`ledger`: Removed the type filter from the RPC command. ([#4934](https://github.com/XRPLF/rippled/pull/4934))
-`subscribe` (`validations` stream): `network_id` is now included. ([#5579](https://github.com/XRPLF/rippled/pull/5579))
-`subscribe` (`transactions` stream): `nftoken_id`, `nftoken_ids`, and `offer_id` are now included in transaction metadata. ([#5230](https://github.com/XRPLF/rippled/pull/5230))
## XRP Ledger server version 2.5.1
[Version 2.5.1](https://github.com/XRPLF/rippled/releases/tag/2.5.1) was released on Sep 17, 2025.
This release contains bug fixes only and no API changes.
## XRP Ledger server version 2.5.0
As of 2025-04-04, version 2.5.0 is in development. You can use a pre-release version by building from source or [using the `nightly` package](https://xrpl.org/docs/infrastructure/installation/install-rippled-on-ubuntu).
[Version 2.5.0](https://github.com/XRPLF/rippled/releases/tag/2.5.0) was released on Jun 24, 2025.
### Additions and bugfixes in 2.5.0
-`channel_authorize`: If `signing_support` is not enabled in the config, the RPC is disabled.
-`tx`: Added `ctid` field to the response and improved error handling. ([#4738](https://github.com/XRPLF/rippled/pull/4738))
-`ledger_entry`: Improved error messages in `permissioned_domain`. ([#5344](https://github.com/XRPLF/rippled/pull/5344))
-`channel_authorize`: If `signing_support` is not enabled in the config, the RPC is disabled. ([#5385](https://github.com/XRPLF/rippled/pull/5385))
-`subscribe` (admin): Removed webhook queue limit to prevent dropping notifications; reduced HTTP timeout from 10 minutes to 30 seconds. ([#5163](https://github.com/XRPLF/rippled/pull/5163))
-`ledger_data` (gRPC): Fixed crashing issue with some invalid markers. ([#5137](https://github.com/XRPLF/rippled/pull/5137))
-`account_lines`: Fixed error with `no_ripple` and `no_ripple_peer` sometimes showing up incorrectly. ([#5345](https://github.com/XRPLF/rippled/pull/5345))
-`account_tx`: Fixed issue with incorrect CTIDs. ([#5408](https://github.com/XRPLF/rippled/pull/5408))
## XRP Ledger server version 2.4.0
@@ -97,11 +121,19 @@ As of 2025-04-04, version 2.5.0 is in development. You can use a pre-release ver
### Additions and bugfixes in 2.4.0
-`ledger_entry`: `state` is added an alias for `ripple_state`.
-`ledger_entry`: Enables case-insensitive filtering by canonical name in addition to case-sensitive filtering by RPC name.
-`validators`: Added new field `validator_list_threshold` in response.
-`simulate`: A new RPC that executes a [dry run of a transaction submission](https://github.com/XRPLF/XRPL-Standards/tree/master/XLS-0069d-simulate#2-rpc-simulate)
-Signing methods autofill fees better and properly handle transactions that don't have a base fee, and will also autofill the `NetworkID` field.
-`simulate`: A new RPC that executes a [dry run of a transaction submission](https://github.com/XRPLF/XRPL-Standards/tree/master/XLS-0069d-simulate#2-rpc-simulate). ([#5069](https://github.com/XRPLF/rippled/pull/5069))
-Signing methods (`sign`, `sign_for`, `submit`): Autofill fees better, properly handle transactions without a base fee, and autofill the `NetworkID` field. ([#5069](https://github.com/XRPLF/rippled/pull/5069))
-`ledger_entry`: `state` is added as an alias for `ripple_state`. ([#5199](https://github.com/XRPLF/rippled/pull/5199))
-`ledger`, `ledger_data`, `account_objects`: Support filtering ledger entry types by their canonical names (case-insensitive). ([#5271](https://github.com/XRPLF/rippled/pull/5271))
-`validators`: Added new field `validator_list_threshold` in response. ([#5112](https://github.com/XRPLF/rippled/pull/5112))
-`server_info`: Added git commit hash info on admin connection. ([#5225](https://github.com/XRPLF/rippled/pull/5225))
-`server_definitions`: Changed larger `UInt` serialized types to `Hash`. ([#5231](https://github.com/XRPLF/rippled/pull/5231))
## XRP Ledger server version 2.3.1
[Version 2.3.1](https://github.com/XRPLF/rippled/releases/tag/2.3.1) was released on Jan 29, 2025.
This release contains bug fixes only and no API changes.
## XRP Ledger server version 2.3.0
@@ -109,19 +141,30 @@ As of 2025-04-04, version 2.5.0 is in development. You can use a pre-release ver
### Breaking changes in 2.3.0
-`book_changes`: If the requested ledger version is not available on this node, a `ledgerNotFound` error is returned and the node does not attempt to acquire the ledger from the p2p network (as with other non-admin RPCs).
Admins can still attempt to retrieve old ledgers with the `ledger_request` RPC.
-`book_changes`: If the requested ledger version is not available on this node, a `ledgerNotFound` error is returned and the node does not attempt to acquire the ledger from the p2p network (as with other non-admin RPCs). Admins can still attempt to retrieve old ledgers with the `ledger_request` RPC.
### Additions and bugfixes in 2.3.0
-`book_changes`: Returns a `validated` field in its response, which was missing in prior versions.
-`book_changes`: Returns a `validated` field in its response. ([#5096](https://github.com/XRPLF/rippled/pull/5096))
-`book_changes`: Accepts shortcut strings (`current`, `closed`, `validated`) for the `ledger_index` parameter. ([#5096](https://github.com/XRPLF/rippled/pull/5096))
-`server_definitions`: Include `index` in response. ([#5190](https://github.com/XRPLF/rippled/pull/5190))
-`account_nfts`: Fix issue where unassociated marker would return incorrect results. ([#5045](https://github.com/XRPLF/rippled/pull/5045))
-`account_objects`: Fix issue where invalid marker would not return an error. ([#5046](https://github.com/XRPLF/rippled/pull/5046))
-`account_objects`: Disallow filtering by ledger entry types that an account cannot hold. ([#5056](https://github.com/XRPLF/rippled/pull/5056))
-`feature`: Better error handling for invalid values of `feature`. ([#5063](https://github.com/XRPLF/rippled/pull/5063))
## XRP Ledger server version 2.2.0
[Version 2.2.0](https://github.com/XRPLF/rippled/releases/tag/2.2.0) was released on Jun 5, 2024. The following additions are non-breaking (because they are purely additive):
- The`feature` method now has a non-admin mode for users. (It was previously only available to admin connections.) The method returns an updated list of amendments, including their names and other information. ([#4781](https://github.com/XRPLF/rippled/pull/4781))
-`feature`: Add a non-admin mode for users. (It was previously only available to admin connections.) The method returns an updated list of amendments, including their names and other information. ([#4781](https://github.com/XRPLF/rippled/pull/4781))
## XRP Ledger server version 2.0.1
[Version 2.0.1](https://github.com/XRPLF/rippled/releases/tag/2.0.1) was released on Jan 29, 2024. The following additions are non-breaking:
@@ -129,24 +172,18 @@ Admins can still attempt to retrieve old ledgers with the `ledger_request` RPC.
-`server_definitions`: A new RPC that generates a `definitions.json`-like output that can be used in XRPL libraries.
- In `Payment` transactions, `DeliverMax` has been added. This is a replacement for the `Amount` field, which should not be used. Typically, the `delivered_amount` (in transaction metadata) should be used. To ease the transition, `DeliverMax` is present regardless of API version, since adding a field is non-breaking.
- API version 2 has been moved from beta to supported, meaning that it is generally available (regardless of the `beta_rpc_api` setting).
## XRP Ledger server version 2.2.0
The following is a non-breaking addition to the API.
- The `feature` method now has a non-admin mode for users. (It was previously only available to admin connections.) The method returns an updated list of amendments, including their names and other information. ([#4781](https://github.com/XRPLF/rippled/pull/4781))
- API version 2 has been moved from beta to supported, meaning that it is generally available (regardless of the `beta_rpc_api` setting). The full list of changes is in [API-VERSION-2.md](API-VERSION-2.md).
## XRP Ledger server version 1.12.0
[Version 1.12.0](https://github.com/XRPLF/rippled/releases/tag/1.12.0) was released on Sep 6, 2023. The following additions are non-breaking (because they are purely additive).
[Version 1.12.0](https://github.com/XRPLF/rippled/releases/tag/1.12.0) was released on Sep 6, 2023. The following additions are non-breaking (because they are purely additive):
-`server_info`: Added `ports`, an array which advertises the RPC and WebSocket ports. This information is also included in the `/crawl` endpoint (which calls `server_info` internally). `grpc` and `peer` ports are also included. (https://github.com/XRPLF/rippled/pull/4427)
-`server_info`: Added `ports`, an array which advertises the RPC and WebSocket ports. This information is also included in the `/crawl` endpoint (which calls `server_info` internally). `grpc` and `peer` ports are also included. ([#4427](https://github.com/XRPLF/rippled/pull/4427))
-`ports` contains objects, each containing a `port` for the listening port (a number string), and a `protocol` array listing the supported protocols on that port.
- This allows crawlers to build a more detailed topology without needing to port-scan nodes.
- (For peers and other non-admin clients, the info about admin ports is excluded.)
- Clawback: The following additions are gated by the Clawback amendment (`featureClawback`). (https://github.com/XRPLF/rippled/pull/4553)
- Adds an [AccountRoot flag](https://xrpl.org/accountroot.html#accountroot-flags) called `lsfAllowTrustLineClawback` (https://github.com/XRPLF/rippled/pull/4617)
- Clawback: The following additions are gated by the Clawback amendment (`featureClawback`). ([#4553](https://github.com/XRPLF/rippled/pull/4553))
- Adds an [AccountRoot flag](https://xrpl.org/accountroot.html#accountroot-flags) called `lsfAllowTrustLineClawback`. ([#4617](https://github.com/XRPLF/rippled/pull/4617))
- Adds the corresponding `asfAllowTrustLineClawback` [AccountSet Flag](https://xrpl.org/accountset.html#accountset-flags) as well.
- Clawback is disabled by default, so if an issuer desires the ability to claw back funds, they must use an `AccountSet` transaction to set the AllowTrustLineClawback flag. They must do this before creating any trust lines, offers, escrows, payment channels, or checks.
- Adds the [Clawback transaction type](https://github.com/XRPLF/XRPL-Standards/blob/master/XLS-39d-clawback/README.md#331-clawback-transaction), containing these fields:
@@ -181,16 +218,16 @@ The following is a non-breaking addition to the API.
### Breaking changes in 1.11
- Added the ability to mark amendments as obsolete. For the `feature` admin API, there is a new possible value for the `vetoed` field. (https://github.com/XRPLF/rippled/pull/4291)
- Added the ability to mark amendments as obsolete. For the `feature` admin API, there is a new possible value for the `vetoed` field. ([#4291](https://github.com/XRPLF/rippled/pull/4291))
- The value of `vetoed` can now be `true`, `false`, or `"Obsolete"`.
- Removed the acceptance of seeds or public keys in place of account addresses. (https://github.com/XRPLF/rippled/pull/4404)
- Removed the acceptance of seeds or public keys in place of account addresses. ([#4404](https://github.com/XRPLF/rippled/pull/4404))
- This simplifies the API and encourages better security practices (i.e. seeds should never be sent over the network).
- For the `ledger_data` method, when all entries are filtered out, the `state` field of the response is now an empty list (in other words, an empty array, `[]`). (Previously, it would return `null`.) While this is technically a breaking change, the new behavior is consistent with the documentation, so this is considered only a bug fix. (https://github.com/XRPLF/rippled/pull/4398)
- For the `ledger_data` method, when all entries are filtered out, the `state` field of the response is now an empty list (in other words, an empty array, `[]`). (Previously, it would return `null`.) While this is technically a breaking change, the new behavior is consistent with the documentation, so this is considered only a bug fix. ([#4398](https://github.com/XRPLF/rippled/pull/4398))
- If and when the `fixNFTokenRemint` amendment activates, there will be a new AccountRoot field, `FirstNFTSequence`. This field is set to the current account sequence when the account issues their first NFT. If an account has not issued any NFTs, then the field is not set. ([#4406](https://github.com/XRPLF/rippled/pull/4406))
- There is a new account deletion restriction: an account can only be deleted if `FirstNFTSequence` + `MintedNFTokens` + `256` is less than the current ledger sequence.
- This is potentially a breaking change if clients have logic for determining whether an account can be deleted.
- NetworkID
- For sidechains and networks with a network ID greater than 1024, there is a new [transaction common field](https://xrpl.org/transaction-common-fields.html), `NetworkID`. (https://github.com/XRPLF/rippled/pull/4370)
- For sidechains and networks with a network ID greater than 1024, there is a new [transaction common field](https://xrpl.org/transaction-common-fields.html), `NetworkID`. ([#4370](https://github.com/XRPLF/rippled/pull/4370))
- This field helps to prevent replay attacks and is now required for chains whose network ID is 1025 or higher.
- The field must be omitted for Mainnet, so there is no change for Mainnet users.
- There are three new local error codes:
@@ -200,10 +237,10 @@ The following is a non-breaking addition to the API.
### Additions and bug fixes in 1.11
- Added `nftoken_id`, `nftoken_ids` and `offer_id` meta fields into NFT `tx` and `account_tx` responses. (https://github.com/XRPLF/rippled/pull/4447)
- Added an `account_flags` object to the `account_info` method response. (https://github.com/XRPLF/rippled/pull/4459)
- Added `NFTokenPages` to the `account_objects` RPC. (https://github.com/XRPLF/rippled/pull/4352)
- Fixed: `marker` returned from the `account_lines` command would not work on subsequent commands. (https://github.com/XRPLF/rippled/pull/4361)
- Added `nftoken_id`, `nftoken_ids` and `offer_id` meta fields into NFT `tx` and `account_tx` responses. ([#4447](https://github.com/XRPLF/rippled/pull/4447))
- Added an `account_flags` object to the `account_info` method response. ([#4459](https://github.com/XRPLF/rippled/pull/4459))
- Added `NFTokenPages` to the `account_objects` RPC. ([#4352](https://github.com/XRPLF/rippled/pull/4352))
- Fixed: `marker` returned from the `account_lines` command would not work on subsequent commands. ([#4361](https://github.com/XRPLF/rippled/pull/4361))
API version 2 is available in `xrpld` version 2.0.0 and later. To use this API, clients specify `"api_version" : 2` in each request.
For info about how [API versioning](https://xrpl.org/request-formatting.html#api-versioning) works, including examples, please view the [XLS-22d spec](https://github.com/XRPLF/XRPL-Standards/discussions/54). For details about the implementation of API versioning, view the [implementation PR](https://github.com/XRPLF/rippled/pull/3155). API versioning ensures existing integrations and users continue to receive existing behavior, while those that request a higher API version will experience new behavior.
## Removed methods
In API version 2, the following deprecated methods are no longer available: ([#4759](https://github.com/XRPLF/rippled/pull/4759))
-`tx_history` - Instead, use other methods such as `account_tx` or `ledger` with the `transactions` field set to `true`.
-`ledger_header` - Instead, use the `ledger` method.
## Modifications to JSON transaction element in API version 2
In API version 2, JSON elements for transaction output have been changed and made consistent for all methods which output transactions. ([#4775](https://github.com/XRPLF/rippled/pull/4775))
This helps to unify the JSON serialization format of transactions. ([clio#722](https://github.com/XRPLF/clio/issues/722), [#4727](https://github.com/XRPLF/rippled/issues/4727))
- JSON transaction element is named `tx_json`
- Binary transaction element is named `tx_blob`
- JSON transaction metadata element is named `meta`
- Binary transaction metadata element is named `meta_blob`
Additionally, these elements are now consistently available next to `tx_json` (i.e. sibling elements), where possible:
-`hash` - Transaction ID. This data was stored inside transaction output in API version 1, but in API version 2 is a sibling element.
-`ledger_index` - Ledger index (only set on validated ledgers)
-`ledger_hash` - Ledger hash (only set on closed or validated ledgers)
-`close_time_iso` - Ledger close time expressed in ISO 8601 time format (only set on validated ledgers)
-`validated` - Bool element set to `true` if the transaction is in a validated ledger, otherwise `false`
This change affects the following methods:
-`tx` - Transaction data moved into element `tx_json` (was inline inside `result`) or, if binary output was requested, moved from `tx` to `tx_blob`. Renamed binary transaction metadata element (if it was requested) from `meta` to `meta_blob`. Changed location of `hash` and added new elements
-`account_tx` - Renamed transaction element from `tx` to `tx_json`. Renamed binary transaction metadata element (if it was requested) from `meta` to `meta_blob`. Changed location of `hash` and added new elements
-`transaction_entry` - Renamed transaction metadata element from `metadata` to `meta`. Changed location of `hash` and added new elements
-`subscribe` - Renamed transaction element from `transaction` to `tx_json`. Changed location of `hash` and added new elements
-`sign`, `sign_for`, `submit` and `submit_multisigned` - Changed location of `hash` element.
## Modifications to `Payment` transaction JSON schema
When reading Payments, the `Amount` field should generally **not** be used. Instead, use [delivered_amount](https://xrpl.org/partial-payments.html#the-delivered_amount-field) to see the amount that the Payment delivered. To clarify its meaning, the `Amount` field is being renamed to `DeliverMax`. ([#4733](https://github.com/XRPLF/rippled/pull/4733))
- In `Payment` transaction type, JSON RPC field `Amount` is renamed to `DeliverMax`. To enable smooth client transition, `Amount` is still handled, as described below: ([#4733](https://github.com/XRPLF/rippled/pull/4733))
- On JSON RPC input (e.g. `submit_multisigned` etc. methods), `Amount` is recognized as an alias to `DeliverMax` for both API version 1 and version 2 clients.
- On JSON RPC input, submitting both `Amount` and `DeliverMax` fields is allowed _only_ if they are identical; otherwise such input is rejected with `rpcINVALID_PARAMS` error.
- On JSON RPC output (e.g. `subscribe`, `account_tx` etc. methods), `DeliverMax` is present in both API version 1 and version 2.
- On JSON RPC output, `Amount` is only present in API version 1 and _not_ in version 2.
## Modifications to account_info response
-`signer_lists` is returned in the root of the response. (In API version 1, it was nested under `account_data`.) ([#3770](https://github.com/XRPLF/rippled/pull/3770))
- When using an invalid `signer_lists` value, the API now returns an "invalidParams" error. ([#4585](https://github.com/XRPLF/rippled/pull/4585))
- (`signer_lists` must be a boolean. In API version 1, strings were accepted and may return a normal response - i.e. as if `signer_lists` were `true`.)
## Modifications to [account_tx](https://xrpl.org/account_tx.html#account_tx) response
- Using `ledger_index_min`, `ledger_index_max`, and `ledger_index` returns `invalidParams` because if you use `ledger_index_min` or `ledger_index_max`, then it does not make sense to also specify `ledger_index`. In API version 1, no error was returned. ([#4571](https://github.com/XRPLF/rippled/pull/4571))
- The same applies for `ledger_index_min`, `ledger_index_max`, and `ledger_hash`. ([#4545](https://github.com/XRPLF/rippled/issues/4545#issuecomment-1565065579))
- Using a `ledger_index_min` or `ledger_index_max` beyond the range of ledgers that the server has:
- returns `lgrIdxMalformed` in API version 2. Previously, in API version 1, no error was returned. ([#4288](https://github.com/XRPLF/rippled/issues/4288))
- Attempting to use a non-boolean value (such as a string) for the `binary` or `forward` parameters returns `invalidParams` (`rpcINVALID_PARAMS`). Previously, in API version 1, no error was returned. ([#4620](https://github.com/XRPLF/rippled/pull/4620))
## Modifications to [noripple_check](https://xrpl.org/noripple_check.html#noripple_check) response
- Attempting to use a non-boolean value (such as a string) for the `transactions` parameter returns `invalidParams` (`rpcINVALID_PARAMS`). Previously, in API version 1, no error was returned. ([#4620](https://github.com/XRPLF/rippled/pull/4620))
API version 3 is currently a **beta API**. It requires enabling `[beta_rpc_api]` in the xrpld configuration to use. To use this API, clients specify `"api_version" : 3` in each request.
For info about how [API versioning](https://xrpl.org/request-formatting.html#api-versioning) works, including examples, please view the [XLS-22d spec](https://github.com/XRPLF/XRPL-Standards/discussions/54). For details about the implementation of API versioning, view the [implementation PR](https://github.com/XRPLF/rippled/pull/3155). API versioning ensures existing integrations and users continue to receive existing behavior, while those that request a higher API version will experience new behavior.
## Breaking Changes
### Modifications to `amm_info`
The order of error checks has been changed to provide more specific error messages. ([#4924](https://github.com/XRPLF/rippled/pull/4924))
- **Before (API v2)**: When sending an invalid account or asset to `amm_info` while other parameters are not set as expected, the method returns a generic `rpcINVALID_PARAMS` error.
- **After (API v3)**: The same scenario returns a more specific error: `rpcISSUE_MALFORMED` for malformed assets or `rpcACT_MALFORMED` for malformed accounts.
### Modifications to `ledger_entry`
Added support for string shortcuts to look up fixed-location ledger entries using the `"index"` parameter. ([#5644](https://github.com/XRPLF/rippled/pull/5644))
In API version 3, the following string values can be used with the `"index"` parameter:
-`"index": "amendments"` - Returns the `Amendments` ledger entry
-`"index": "fee"` - Returns the `FeeSettings` ledger entry
-`"index": "nunl"` - Returns the `NegativeUNL` ledger entry
| These instructions assume you have a C++ development environment ready with Git, Python, Conan, CMake, and a C++ compiler. For help setting one up on Linux, macOS, or Windows, [see this guide](./docs/build/environment.md). |
| These instructions assume you have a C++ development environment ready with Git, Python, Conan, CMake, and a C++ compiler. For help setting one up on Linux, macOS, or Windows, [see this guide](./docs/build/environment.md).<br><br>These instructions also assume a basic familiarity with Conan and CMake. If you are unfamiliar with Conan, you can read our [crash course](./docs/build/conan.md) or the official [Getting Started][conan-getting-started] walkthrough. |
> These instructions also assume a basic familiarity with Conan and CMake.
> If you are unfamiliar with Conan, you can read our
> [crash course](./docs/build/conan.md) or the official [Getting Started][3]
> walkthrough.
## Minimum Requirements
## Branches
See [System Requirements](https://xrpl.org/system-requirements.html).
For a stable release, choose the `master` branch or one of the [tagged
Building xrpld generally requires Git, Python, Conan, CMake, and a C++
compiler.
- [Python](https://www.python.org/downloads/)
- [Conan](https://conan.io/downloads.html)
- [CMake](https://cmake.org/download/)
You can verify that the required tools are installed and runnable with:
```bash
git checkout master
./bin/check-tools.sh
```
For the latest release candidate, choose the `release` branch.
`xrpld` is written in the C++23 dialect. The [tested compiler versions][cpp23-support] are:
```bash
git checkout release
```
| Compiler | Version |
| ----------- | --------------- |
| GCC | 15.2 |
| Clang | 22 |
| Apple Clang | 21 |
| MSVC | 19.44[^windows] |
## Operating Systems
Please see the [environment setup guide](./docs/build/environment.md) for detailed instructions for all platforms.
### Linux
The Ubuntu Linux distribution has received the highest level of quality
assurance, testing, and support. We also support Red Hat and use Debian
internally.
Our Linux CI tooling is distro-independent and uses a Nix-based environment, so it should be possible to build on other Linux distributions as well, although we have not tested them.
### macOS
Many `xrpld` engineers use macOS for development.
### Windows
Windows is used by some engineers for development only.
[^windows]: Windows is not recommended for production use.
## Steps
### Branches
For the latest set of untested features, or to contribute, choose the `develop`
branch.
@@ -29,55 +60,15 @@ branch.
git checkout develop
```
## Minimum Requirements
For a release candidate, choose the relevant release branch, e.g.
`release/3.2.x`.
See [System Requirements](https://xrpl.org/system-requirements.html).
```bash
git checkout release/3.2.x
```
Building xrpld generally requires git, Python, Conan, CMake, and a C++
compiler. Some guidance on setting up such a [C++ development environment can be
found here](./docs/build/environment.md).
- [Python 3.11](https://www.python.org/downloads/), or higher
- [Conan 2.17](https://conan.io/downloads.html)[^1], or higher
- [CMake 3.22](https://cmake.org/download/), or higher
[^1]:
It is possible to build with Conan 1.60+, but the instructions are
significantly different, which is why we are not recommending it.
`xrpld` is written in the C++20 dialect and includes the `<concepts>` header.
The [minimum compiler versions][2] required are:
| Compiler | Version |
| ----------- | --------- |
| GCC | 12 |
| Clang | 16 |
| Apple Clang | 16 |
| MSVC | 19.44[^3] |
### Linux
The Ubuntu Linux distribution has received the highest level of quality
assurance, testing, and support. We also support Red Hat and use Debian
internally.
Here are [sample instructions for setting up a C++ development environment on
Linux](./docs/build/environment.md#linux).
### Mac
Many xrpld engineers use macOS for development.
Here are [sample instructions for setting up a C++ development environment on
macOS](./docs/build/environment.md#macos).
### Windows
Windows is used by some engineers for development only.
[^3]: Windows is not recommended for production use.
You can manage multiple Conan profiles in the directory
`$(conan config home)/profiles`, for example renaming `default` to a different
name and then creating a new `default` profile for a different compiler.
#### Select language
The default profile created by Conan will typically select different C++ dialect
than C++20 used by this project. You should set `20` in the profile line
starting with `compiler.cppstd=`. For example:
```bash
sed -i.bak -e 's|^compiler\.cppstd=.*$|compiler.cppstd=20|'$(conan config home)/profiles/default
```
#### Select standard library in Linux
**Linux** developers will commonly have a default Conan [profile][] that
compiles with GCC and links with libstdc++. If you are linking with libstdc++
(see profile setting `compiler.libcxx`), then you will need to choose the
`libstdc++11` ABI:
```bash
sed -i.bak -e 's|^compiler\.libcxx=.*$|compiler.libcxx=libstdc++11|'$(conan config home)/profiles/default
```
#### Select architecture and runtime in Windows
**Windows** developers may need to use the x64 native build tools. An easy way
to do that is to run the shortcut "x64 Native Tools Command Prompt" for the
version of Visual Studio that you have installed.
Windows developers must also build `xrpld` and its dependencies for the x64
architecture:
```bash
sed -i.bak -e 's|^arch=.*$|arch=x86_64|'$(conan config home)/profiles/default
```
**Windows** developers also must select static runtime:
```bash
sed -i.bak -e 's|^compiler\.runtime=.*$|compiler.runtime=static|'$(conan config home)/profiles/default
```
#### Clang workaround for grpc
If your compiler is clang, version 19 or later, or apple-clang, version 17 or
later, you may encounter a compilation error while building the `grpc`
dependency:
```text
In file included from .../lib/promise/try_seq.h:26:
.../lib/promise/detail/basic_seq.h:499:38: error: a template argument list is expected after a name prefixed by the template keyword [-Wmissing-template-arg-list-after-template-kw]
If your compiler is gcc, version 12, and you have enabled `werr` option, you may
encounter a compilation error such as:
```text
/usr/include/c++/12/bits/char_traits.h:435:56: error: 'void* __builtin_memcpy(void*, const void*, long unsigned int)' accessing 9223372036854775810 or more bytes at offsets [2, 9223372036854775807] and 1 may overlap up to 9223372036854775813 bytes at offset -3 [-Werror=restrict]
You can then build and test as usual, with the generated `xrpld` binary containing the sanitizer instrumentation. When you run it, it will report any sanitizer errors it detects in the console output.
See [Sanitizers docs](./docs/build/sanitizers.md) for more details.
If you're seeing an error like `ERROR: Package 'snappy/1.1.10' not resolved: Unable to find 'snappy/1.1.10#968fef506ff261592ec30c574d4a7809%1756234314.246' in remotes.`,
please add `xrplf` remote or re-run `conan export` for [patched recipes](#patched-recipes).
please [add `xrplf` remote](#add-xrplf-remote) or re-run `conan export` for [patched recipes](./docs/build/advanced_conan.md#patched-recipes).
### `protobuf/port_def.inc` file not found
@@ -584,28 +400,9 @@ For example, if you want to build Debug:
1. For conan install, pass `--settings build_type=Debug`
2. For cmake, pass `-DCMAKE_BUILD_TYPE=Debug`
## Add a Dependency
If you want to experiment with a new package, follow these steps:
1. Search for the package on [Conan Center](https://conan.io/center/).
2. Modify [`conanfile.py`](./conanfile.py):
- Add a version of the package to the `requires` property.
- Change any default options for the package by adding them to the
@@ -14,9 +14,9 @@ The following branches exist in the main project repository:
-`develop`: The latest set of unreleased features, and the most common
starting point for contributions.
-`release`: The latest beta release or release candidate.
-`master`: The latest stable release.
-`gh-pages`: The documentation for this project, built by Doxygen.
-`release/*` (e.g. `release/3.2.x`): Release branches, one per release line,
holding the latest release candidate, or stable release for that line.
Stable releases are published as [tagged releases](https://github.com/XRPLF/rippled/releases).
The tip of each branch must be signed. In order for GitHub to sign a
squashed commit that it builds from your pull request, GitHub must know
@@ -127,34 +127,12 @@ tl;dr
> 6. Wrap the body at 72 characters.
> 7. Use the body to explain what and why vs. how.
In addition to those guidelines, please add one of the following
prefixes to the subject line if appropriate.
-`fix:` - The primary purpose is to fix an existing bug.
-`perf:` - The primary purpose is performance improvements.
-`refactor:` - The changes refactor code without affecting
functionality.
-`test:` - The changes _only_ affect unit tests.
-`docs:` - The changes _only_ affect documentation. This can
include code comments in addition to `.md` files like this one.
-`build:` - The changes _only_ affect the build process,
including CMake and/or Conan settings.
-`chore:` - Other tasks that don't affect the binary, but don't fit
any of the other cases. e.g. formatting, git settings, updating
Github Actions jobs.
Whenever possible, when updating commits after the PR is open, please
add the PR number to the end of the subject line. e.g. `test: Add
unit tests for Feature X (#1234)`.
## Pull requests
In general, pull requests use `develop` as the base branch.
The exceptions are
- Fixes and improvements to a release candidate use `release` as the
base.
- Hotfixes use `master` as the base.
The exceptions are fixes, improvements, and hotfixes for an existing release,
which use that release's branch (e.g. `release/3.2.x`) as the base.
If your changes are not quite ready, but you want to make it easily available
for preliminary examination or review, you can create a "Draft" pull request.
@@ -180,6 +158,23 @@ credibility of the existing approvals is insufficient.
Pull requests must be merged by [squash-and-merge][squash]
to preserve a linear history for the `develop` branch.
### Type of Change
In addition to those guidelines, please start your PR title with one of the following:
-`build:` - The changes _only_ affect the build process, including CMake and/or Conan settings.
-`feat`: New feature (change which adds functionality).
-`fix:` - The primary purpose is to fix an existing bug.
-`docs:` - The changes _only_ affect documentation.
-`test:` - The changes _only_ affect unit tests.
-`ci`: Continuous Integration (changes to our CI configuration files and scripts).
-`style`: Code style (formatting).
-`refactor:` - The changes refactor code without affecting functionality.
-`perf:` - The primary purpose is performance improvements.
-`chore:` - Other tasks that don't affect the binary, but don't fit any of the other cases. e.g. `git` settings, `clang-tidy`, removing dead code, dropping support for older tooling.
First letter after the type prefix should be capitalized, and the type prefix should be followed by a colon and a space. e.g. `feat: Add support for Borrowing Protocol`.
### "Ready to merge"
A pull request should only have the "Ready to merge" label added when it
@@ -219,7 +214,7 @@ coherent rather than a set of _thou shalt not_ commandments.
## Formatting
All code must conform to `clang-format` version 18,
All code must conform to `clang-format` version 22,
according to the settings in [`.clang-format`](./.clang-format),
unless the result would be unreasonably difficult to read or maintain.
To demarcate lines that should be left as-is, surround them with comments like
@@ -251,6 +246,118 @@ pip3 install pre-commit
pre-commit install
```
## Clang-tidy
All code must pass `clang-tidy` checks according to the settings in [`.clang-tidy`](./.clang-tidy).
There is a Continuous Integration job that runs clang-tidy on pull requests. The CI will check:
- All changed C++ files (`.cpp`, `.h`, `.ipp`) when only code files are modified
- **All files in the repository** when the `.clang-tidy` configuration file is changed
This ensures that configuration changes don't introduce new warnings across the codebase.
### Installing clang-tidy
See the [environment setup guide](./docs/build/environment.md#clang-tidy) for how to get clang-tidy.
### Running clang-tidy locally
Before running clang-tidy, you must build the project to generate required files (particularly protobuf headers). Refer to [`BUILD.md`](./BUILD.md) for build instructions.
#### Via pre-commit (recommended)
If you have already installed the pre-commit hooks (see above), you can run clang-tidy on your staged files using:
```
TIDY=1 pre-commit run clang-tidy
```
This runs clang-tidy locally with the same configuration/flags as CI, scoped to your staged C++ files. The `TIDY=1` environment variable is required to opt in — without it the hook is skipped.
You can also have clang-tidy run automatically on every `git commit` by setting `TIDY=1` in your shell environment:
```
export TIDY=1
```
With this set, the hook will run as part of `git commit` alongside the other pre-commit checks.
Distributed tracing is a method for tracking data objects as they flow through distributed systems. In a network like XRP Ledger, a single transaction touches multiple independent nodes—each with no shared memory or logging. Distributed tracing connects these dots.
**Without tracing:** You see isolated logs on each node with no way to correlate them.
**With tracing:** You see the complete journey of a transaction or an event across all nodes it touched.
D --> E["ledger.update<br/>span_id: 005<br/>20ms"]
end
style A fill:#0d47a1,stroke:#082f6a,color:#ffffff
style B fill:#1b5e20,stroke:#0d3d14,color:#ffffff
style C fill:#1b5e20,stroke:#0d3d14,color:#ffffff
style D fill:#1b5e20,stroke:#0d3d14,color:#ffffff
style E fill:#bf360c,stroke:#8c2809,color:#ffffff
```
**Reading the diagram:**
- **tx.submit (blue, root)**: The top-level span representing the entire transaction submission; all other spans are its descendants.
- **tx.validate, tx.relay, tx.apply (green)**: Direct children of tx.submit, representing the three main stages -- validation, relay to peers, and application to the ledger.
- **ledger.update (red)**: A grandchild span nested under tx.apply, representing the actual ledger state mutation triggered by applying the transaction.
- **Arrows (parent to child)**: Each arrow indicates a parent-child span relationship where the parent's completion depends on the child finishing.
The same trace visualized as a **timeline (Gantt chart)**:
Spans don't always form simple parent-child trees. Distributed tracing defines several relationship types to capture different causal patterns:
### 1. Parent-Child (ChildOf)
The default relationship. The parent span **depends on** or **contains** the child span. The child runs within the scope of the parent.
```
tx.submit (parent)
├── tx.validate (child) ← parent waits for this
├── tx.relay (child) ← parent waits for this
└── tx.apply (child) ← parent waits for this
```
**When to use:** Synchronous calls, nested operations, any case where the parent's completion depends on the child.
### 2. Follows-From
A causal relationship where the first span **triggers** the second, but does **not wait** for it. The originator fires and moves on.
```
Time →
tx.receive [=======]
↓ triggers (follows-from)
tx.relay [===========] ← runs independently
```
**When to use:** Asynchronous jobs, queued work, fire-and-forget patterns. For example, a node receives a transaction and queues it for relay — the relay span _follows from_ the receive span but the receiver doesn't wait for relaying to complete.
> **OpenTracing** defined `FollowsFrom` as a first-class reference type alongside `ChildOf`.
> **OpenTelemetry** represents this using **Span Links** with descriptive attributes instead (see below).
### 3. Span Links (Cross-Trace and Non-Hierarchical)
Links connect spans that are **causally related but not in a parent-child hierarchy**. Unlike parent-child, links can cross trace boundaries.
| **Follows-From** | Usually | Causal but no dependency | Link + attributes |
| **Span Link** | Either | Correlation, no dependency | Link + attributes |
---
## Trace ID Generation
A `trace_id` is a 128-bit (16-byte) identifier that groups all spans belonging to one logical operation. How it's generated determines how easily you can find and correlate traces later.
### General Approaches
#### 1. Random (W3C Default)
Generate a random 128-bit ID when a trace starts. Standard approach for most services.
| Prefix search: "find all traces for TX abc" | Must propagate to maintain full trace_id |
| Unique per processing instance | More complex generation logic |
| Retries get distinct trace_ids | Partial correlation only (prefix match) |
### XRPL Workflow Analysis
XRPL has a unique advantage: its core workflows produce **globally unique 256-bit hashes** that are known on every node. This makes deterministic trace_id generation practical in ways most systems can't achieve.
#### Natural Identifiers by Workflow
| Workflow | Natural Identifier | Size | Known at Start? | Same on All Nodes? |
The string prefix (`"tx"`, `"cons"`, `"fetch"`) prevents collisions between workflows that might share underlying hashes.
**Why this works for XRPL:**
1.**Propagation-resilient** — Even if a P2P message drops trace context, every node independently computes the same trace_id from the same tx_hash or ledger_hash. Spans still correlate.
2.**Zero-cost search** — "Show me the trace for transaction ABC" becomes a direct lookup: compute `SHA-256("tx" || ABC)[0:16]` and query. No secondary index needed.
3.**Cross-workflow linking via Span Links** — A consensus trace links to individual transaction traces. A validation span links to the consensus trace. This connects the full picture without forcing everything into one giant trace.
### Cross-Workflow Correlation
Each workflow gets its own trace. Span Links tie them together:
- **Transaction Trace (blue)**: An independent trace whose `trace_id` is deterministically derived from the transaction hash. Contains receive, validate, and relay spans.
- **Consensus Trace (green)**: An independent trace whose `trace_id` is derived from the previous ledger hash and sequence number. Covers the open, propose, and accept phases.
- **Validation (red)**: Validation spans live within the consensus trace (not a separate trace). They are created after the accept phase completes.
- **Catch-Up Trace (purple)**: An independent trace for ledger acquisition, derived from the target ledger hash. Used when a node is behind and fetching missing ledgers.
- **Dotted arrows (span links)**: Cross-trace correlations. Consensus links to transaction traces it included; catch-up links to the consensus trace that produced the target ledger.
- **Solid arrow (C3 to V1)**: A parent-child relationship -- validation spans are direct children of the consensus accept span within the same trace.
**How a query flows:**
```
"Why was TX abc slow?"
1. Compute trace_id = SHA-256("tx" || abc)[0:16]
2. Find transaction trace → see it was included in consensus round N
3. Follow span link → consensus trace for round N
4. See which phase was slow (propose? accept?)
5. If a node was catching up, follow link → catch-up trace
| **Retries get same trace_id** | Add `attempt` attribute to root span; spans have unique span_ids and timestamps |
| **256→128 bit truncation** | Birthday-bound collision at ~2⁶⁴ operations — negligible for XRPL's throughput |
| **Non-standard generation** | OTel spec allows any 16-byte non-zero value; tooling works on the hex string |
| **Hash computation cost** | SHA-256 is ~0.3μs per call; XRPL already computes these hashes for other purposes |
| **Late-binding identifiers** | Ledger hash isn't known until after consensus — validation spans use ledger_seq as fallback, then link to the consensus trace |
---
## Distributed Traces Across Nodes
In distributed systems like xrpld, traces span **multiple independent nodes**. The trace context must be propagated in network messages:
```mermaid
sequenceDiagram
participant Client
participant NodeA as Node A
participant NodeB as Node B
participant NodeC as Node C
Client->>NodeA: Submit TX<br/>(no trace context)
Note over NodeA: Creates new trace<br/>trace_id: abc123<br/>span: tx.receive
Note over NodeC: Creates child span<br/>span: tx.relay<br/>parent_span_id: 001
Note over NodeA,NodeC: All spans share trace_id: abc123<br/>enabling correlation across nodes
```
**Reading the diagram:**
- **Client**: The external entity that submits a transaction. It does not carry trace context -- the trace originates at the first node.
- **Node A**: The entry point that creates a new trace (trace_id: abc123) and the root span `tx.receive`. It relays the transaction to peers with trace context attached.
- **Node B and Node C**: Peer nodes that receive the relayed transaction along with the propagated trace context. Each creates a child span under Node A's span, preserving the same `trace_id`.
- **Arrows with trace context**: The relay messages carry `trace_id` and `parent_span_id`, allowing each downstream node to link its spans back to the originating span on Node A.
---
## Context Propagation
For traces to work across nodes, **trace context must be propagated** in messages.
xrpld P2P messages such as `TMTransaction` carry the trace context in two added byte fields alongside the existing payload: `trace_parent` holds the W3C traceparent (`trace_id`, `span_id`, and `trace_flags`), and `trace_state` holds the optional W3C tracestate. Together they propagate the trace across the P2P boundary so a receiving node can attach its spans to the sender's span.
---
## Sampling
Not every trace needs to be recorded. **Sampling** reduces overhead:
### Head Sampling (at trace start)
```
Request arrives → Random N% chance → Record or skip entire trace
```
- ✅ Low overhead
- ❌ May miss interesting traces
> **xrpld note**: xrpld intentionally fixes head sampling at 100% (sample
> everything) and does not expose a configurable ratio. A per-node ratio
> would let different nodes make divergent keep/drop decisions for the same
> distributed trace, producing broken/partial traces. xrpld uses a
> `ParentBased` sampler so spans with a remote parent honor the upstream
> decision. Volume reduction is delegated to collector-side tail sampling.
- **Core Services (blue)**: The entry points into xrpld -- RPC Server handles client requests, Overlay manages peer-to-peer networking, Consensus drives agreement, and ValidatorList manages trusted validators.
- **JobQueue (center)**: The asynchronous thread pool that decouples Core Services from the Processing and Application layers. All work flows through it.
- **Application Services (purple)**: Higher-level features -- PathFinding computes payment routes, TxQ manages fee-based queuing, and LoadManager tracks server load.
- **Existing Observability (orange)**: The current monitoring stack (PerfLog, Insight, Journal logging) that OpenTelemetry will complement, not replace.
- **Arrows (Services to JobQueue to layers)**: Work originates at Core Services, is enqueued onto the JobQueue, and dispatched to Processing or Application layers for execution.
| **SHAMap** | `src/xrpld/shamap/` | State hash tree | Sync speed, missing nodes |
---
## 1.3 Transaction Flow Diagram
Transaction flow spans multiple nodes in the network. Each node creates linked spans to form a distributed trace:
```mermaid
sequenceDiagram
participant Client
participant PeerA as Peer A (Receive)
participant PeerB as Peer B (Relay)
participant PeerC as Peer C (Validate)
Client->>PeerA: 1. Submit TX
rect rgb(230, 245, 255)
Note over PeerA: tx.receive SPAN START
PeerA->>PeerA: HashRouter Deduplication
PeerA->>PeerA: tx.validate (child span)
end
PeerA->>PeerB: 2. Relay TX (with trace ctx)
rect rgb(230, 245, 255)
Note over PeerB: tx.receive (linked span)
end
PeerB->>PeerC: 3. Relay TX
rect rgb(230, 245, 255)
Note over PeerC: tx.receive (linked span)
PeerC->>PeerC: tx.process
end
Note over Client,PeerC: DISTRIBUTED TRACE (same trace_id: abc123)
```
**Reading the diagram:**
- **Client**: The external entity that submits a transaction to Peer A. It has no trace context -- the trace starts at the first node.
- **Peer A (Receive)**: The entry node that creates the root span `tx.receive`, runs HashRouter deduplication to avoid processing duplicates, and creates a child `tx.validate` span.
- **Peer A to Peer B arrow**: The relay message carries trace context (trace_id + parent span_id), enabling Peer B to create a linked span under the same trace.
- **Peer B (Relay)**: Receives the transaction and trace context, creates a `tx.receive` span linked to Peer A's trace, then relays onward.
- **Peer C (Validate)**: Final hop in this example. Creates a linked `tx.receive` span and runs `tx.process` to fully process the transaction.
- **Blue rectangles**: Highlight the span boundaries on each node, showing where instrumentation creates and closes spans.
### Trace Structure
```
trace_id: abc123
├── span: tx.receive (Peer A)
│ ├── span: tx.validate
│ └── span: tx.relay
├── span: tx.receive (Peer B) [parent: Peer A]
│ └── span: tx.relay
└── span: tx.receive (Peer C) [parent: Peer B]
└── span: tx.process
```
---
## 1.4 Consensus Round Flow
Consensus rounds are multi-phase operations that benefit significantly from tracing:
- **consensus.round (orange, root span)**: The top-level span encompassing the entire consensus round, with attributes like ledger sequence, mode, and proposer count.
- **consensus.phase.open (blue)**: The first phase where the node waits (~3s) to collect incoming transactions before proposing.
- **consensus.phase.establish (green)**: The negotiation phase where validators exchange proposals, resolve disputes, and converge on a transaction set. Child spans track each proposal received/sent and each dispute resolved.
- **consensus.phase.accept (pink)**: The final phase where the agreed transaction set is applied, a new ledger is built, and the ledger is validated. Child spans cover `ledger.build` and `ledger.validate`.
- **Arrows (open to establish to accept)**: The sequential flow through the three consensus phases. Each phase must complete before the next begins.
---
## 1.5 RPC Request Flow
> **WS** = WebSocket
RPC requests support W3C Trace Context headers for distributed tracing across services:
```mermaid
flowchart TB
subgraph request["rpc.request (root span)"]
http["HTTP Request — POST /<br/>traceparent:<br/>00-abc123...-def456...-01"]
- **rpc.request (green, root span)**: The outermost span representing the full RPC request lifecycle, from HTTP receipt to response. Carries the W3C `traceparent` header for distributed tracing.
- **HTTP Request node**: Shows the incoming POST request with its `traceparent` header and extracted attributes (method, peer IP, command name).
- **jobqueue.enqueue (blue)**: The span covering the asynchronous handoff from the RPC thread to the JobQueue worker thread. The trace context is preserved across this async boundary.
- **rpc.command.submit (orange)**: The span for the actual command execution, with child spans for deserialization, local validation, and network submission.
- **Response node**: The final output with HTTP status and total duration, marking the end of the root span.
- **Arrows (top to bottom)**: The sequential processing pipeline -- receive request, extract attributes, enqueue job, execute command, return response.
---
## 1.6 Key Trace Points
> **TxQ** = Transaction Queue
The following table identifies priority instrumentation points across the codebase:
| **Memory Spike** | Large batch of spans correlating with memory increase | Tune batch_size or sampling |
| **Network Partition** | Traces missing cross-node links for specific peer | Check peer connectivity |
| **Path Computation Slow** | pathfind.compute span shows high latency; cache miss rate in attributes | Warm the RippleLineCache, check order book depth |
- **xrpld Nodes (blue)**: The source of telemetry data. Each xrpld node exports spans via OTLP/HTTP on port 4318 (the only exporter shipped in Phase 1b).
- **OpenTelemetry Collector (red)**: The central aggregation point that receives spans from all nodes. Can run as a sidecar (per-node) or standalone (shared). Handles batching, filtering, and routing.
- **Observability Backends (green)**: The storage and visualization destinations. Tempo is the recommended backend for both development and production, and Elastic APM is an alternative. The Collector routes to one or more backends.
- **Arrows (nodes to collector to backends)**: The data pipeline -- spans flow from nodes to the Collector over HTTP, then the Collector fans out to the configured backends.
### 2.2.1 OTLP/HTTP (Shipped in Phase 1b)
OTLP/HTTP is the only exporter wired up in Phase 1b. It is configured via
`OtlpHttpExporterOptions` with the collector traces endpoint
(`http://localhost:4318/v1/traces` by default) and a JSON content type
(binary protobuf is also available).
### 2.2.2 OTLP/gRPC (Future Work — Planned Upgrade)
OTLP/gRPC is planned as a future upgrade from the HTTP exporter. The gRPC
transport offers lower per-span overhead and tighter back-pressure semantics
than HTTP/JSON, making it attractive for production deployments once the HTTP
path is validated in earlier phases.
Required to land this upgrade:
1. Add `opentelemetry-cpp::otlp_grpc_exporter` to the Conan recipe (the
dependency already exists but is not linked in Phase 1b builds).
2. Extend `TelemetryConfig.cpp` to parse an `exporter` key (`otlp_http`
default, `otlp_grpc` opt-in) and a gRPC endpoint override.
3. In `Telemetry::start()` branch on the parsed exporter type and construct
either `OtlpHttpExporterFactory::Create(httpOpts)` or
| **Account Hashing** | Account addresses are hashed both SDK-side (`pathfind_source_account`, `pathfind_dest_account` — always hashed before emission) and again at the collector level, so raw addresses never reach storage |
| **Configurable Redaction** | Sensitive fields can be excluded via `[telemetry]` config section |
| **Collector Tail Sampling** | xrpld head sampling is fixed at 1.0 (every span emitted); the collector retains ~10% of non-error traces, reducing stored data exposure |
| **Sampling** | Only 10% of traces recorded by default, reducing data exposure |
| **Local Control** | Node operators have full control over what gets exported |
| **No Raw Payloads** | Transaction content is never recorded, only metadata (hash, type, result) |
| **Collector-Level Filtering** | Additional redaction/hashing can be configured at OTel Collector |
#### Account Address Hashing
Account addresses are **always** hashed before they reach the telemetry
backend — there is no opt-out flag and therefore no insecure-by-default
failure mode. Protection is applied in two independent layers:
1.**SDK-side** (this node): the path-finding RPC handlers call
`redactAccount()` (`xrpl::telemetry`, `Redaction.h`) before setting the
`pathfind_source_account` / `pathfind_dest_account` span attributes. The
helper emits the first 16 characters of `sha512Half(address)` as
lowercase hex — deterministic (spans for one account still correlate)
but non-reversible.
2.**Collector-side** (defense-in-depth): an `attributes/hash` processor in
the OpenTelemetry Collector re-hashes those same attributes, so any node
that emitted a raw value is still redacted before storage.
#### Collector-Level Data Protection
The OpenTelemetry Collector can be configured (via an `attributes` processor)
to hash or redact sensitive attributes before export — for example, hashing
(`trace_transactions`, `trace_consensus`, `trace_rpc`, `trace_peer` — the last
often disabled due to high volume) select which spans are emitted. Account
address hashing is not configurable: addresses are hashed unconditionally by
the SDK helper described above, with collector-level hashing as a second
layer.
> **Key Principle**: Telemetry collects **operational metadata** (timing, counts, hashes) — never **sensitive content** (keys, balances, amounts, raw payloads).
> **See also**: [Securing the OTel Pipeline](./secure-OTel.md) covers transport-level protection for telemetry leaving the node — mTLS to the collector and validation of incoming peer trace context. Privacy controls in this section keep sensitive data out of spans; the security doc keeps the spans themselves out of untrusted hands.
---
## 2.5 Context Propagation Design
> **WS** = WebSocket
### 2.5.0 Deterministic Trace ID Strategy
Both transaction and consensus tracing use **deterministic trace IDs** derived from
a globally known hash, so all nodes handling the same workflow independently produce
spans under the same `trace_id`. This is combined with protobuf `span_id` propagation
for parent-child relay ordering when available.
#### Transactions — `trace_id = txHash[0:16]`
Every node that handles a transaction knows its `txID` (the `uint256` transaction
hash). The first 16 bytes of this hash are used as the OTel `trace_id`:
- **HTTP/WebSocket - RPC (blue)**: For client-facing RPC requests, trace context is propagated using the W3C `traceparent` header. This is the standard approach and works with any OTel-compatible client.
- **Protocol Buffers - P2P (green)**: For peer-to-peer messages between xrpld nodes, trace context is embedded as a protobuf `TraceContext` message carrying trace_id, span_id, flags, and optional trace_state.
- **JobQueue - Internal Async (red)**: For asynchronous work within a single node, the OTel context is captured when a job is created and restored when the job executes on a worker thread. This bridges the async gap so spans remain linked.
| "Why was this specific TX slow?" | ⚠️ partial | ❌ | ✅ |
| "Which node delayed consensus?" | ❌ | ❌ | ✅ |
| "What happened on node X at time T?" | ✅ | ❌ | ✅ |
| "Show me the TX journey across 5 nodes" | ❌ | ❌ | ✅ |
### 2.6.4 Coexistence Strategy
> **Note**: Phase 7 replaces the StatsD bridge with native OTel Metrics SDK export. The diagram below shows the Phase 6 intermediate state. See [Phase7_taskList.md](./Phase7_taskList.md) for the migration design where Beast Insight emits via OTLP instead of StatsD.
- **xrpld Process (dark gray)**: The single xrpld node running all three observability frameworks side by side. Each framework operates independently with no interference.
- **PerfLog to perf.log**: PerfLog writes JSON-formatted event logs to a local file. Grafana can ingest these via Loki or a file-based datasource.
- **Beast Insight to StatsD Server**: Insight sends aggregated metrics (counters, gauges) over UDP to a StatsD server. Grafana reads from StatsD-compatible backends like Graphite or Prometheus (via StatsD exporter).
- **OpenTelemetry to OTLP Collector**: OTel exports spans over OTLP/HTTP to a Collector, which then forwards to a trace backend (Tempo). (OTLP/gRPC is future work — §2.2.2.)
- **Grafana (red, unified UI)**: All three data streams converge in Grafana, enabling operators to correlate logs, metrics, and traces in a single dashboard.
**Phase 7 target state**: Beast Insight routes to `OTelCollector` (new `Collector` implementation) which exports via OTLP/HTTP to the same collector endpoint as traces. StatsD UDP path becomes a deprecated fallback (`[insight] server=statsd`). See [06-implementation-phases.md §6.8](./06-implementation-phases.md) and [Phase7_taskList.md](./Phase7_taskList.md) for details.
### 2.6.5 Correlation with PerfLog
Trace IDs can be correlated with existing PerfLog entries for comprehensive
debugging. The design is for `RPCHandler.cpp` to start an `rpc.command.<method>`
span alongside the existing PerfLog `rpcStart`/`rpcFinish`/`rpcError` calls,
extract the span's `trace_id` (when valid), and eventually stamp it onto the
PerfLog entry (a planned `setTraceId` hook) so logs and traces share a key. The
span status is set to OK on success or to error (recording the exception) on
Instrumentation is gated on two levels. A compile-time feature flag (`XRPL_ENABLE_TELEMETRY`) reduces the trace macros to no-ops when telemetry is built out, so disabled builds carry zero cost. At runtime, per-component guards (e.g. `shouldTracePeer()`) skip span creation for components whose tracing is turned off, incurring no overhead beyond a single boolean check.
---
## 3.8 Links to Detailed Documentation
- **[Configuration Reference](./05-configuration-reference.md)**: Configuration options and collector setup
- **[Implementation Phases](./06-implementation-phases.md)**: Detailed timeline and milestones
---
## 3.9 Code Intrusiveness Assessment
> **TxQ** = Transaction Queue
This section provides a detailed assessment of how intrusive the OpenTelemetry integration is to the existing xrpld codebase.
| **Runtime** | ✅ Full | `enabled=0` produces zero overhead |
| **API** | ✅ Full | No changes to public RPC or P2P APIs |
### 3.9.6 Rollback Strategy
If issues are discovered after deployment:
1.**Immediate**: Set `enabled=0` in config and restart (zero code change)
2.**Quick**: Rebuild with `XRPL_ENABLE_TELEMETRY=OFF`
3.**Complete**: Revert telemetry commits (clean separation makes this easy)
### 3.9.7 Code Change Examples
**Minimal RPC Instrumentation (Low Intrusiveness):** Instrumenting an RPC handler adds roughly 3-4 lines: one macro to start the span and one or two `setAttribute` calls (command name, status). The span ends automatically via RAII, so the existing control flow — process the request, send the result — is untouched.
**Consensus Instrumentation (Medium Intrusiveness):** Consensus is slightly more intrusive because child spans in later phase transitions need the round's context. Beyond the span-start and attribute macros, this requires storing the active context in a new member variable (`currentRoundContext_`) at round start. The existing round logic itself remains unchanged.
The authoritative `[telemetry]` example lives in `cfg/xrpld-example.cfg`. Telemetry is disabled by default (`enabled=0`); enabling it turns on distributed tracing for transaction flow, consensus, and RPC calls, with traces exported to an OpenTelemetry Collector over OTLP. Head sampling is intentionally fixed at 1.0 (sample everything) and is not configurable — per-node head-sampling would produce broken/partial distributed traces, so volume reduction is delegated to the collector's tail sampling (see Section 7.4.2). The full option reference follows.
| `consensus_trace_strategy` | Phase 4 | Trace ID strategy for consensus rounds (`deterministic` \| `attribute`) |
---
## 5.2 Configuration Parser
> **TxQ** = Transaction Queue
The parser `setupTelemetry()` in `src/libxrpl/telemetry/TelemetryConfig.cpp` reads the `[telemetry]``Section` and populates a `Telemetry::Setup` struct, applying the defaults listed in Section 5.1.2 via `section.value_or(...)`. It derives `serviceInstanceId` from the node public key when not overridden, selects the exporter endpoint default by exporter type, and leaves the sampling ratio at its fixed 1.0 default (not read from config — see Section 7.4.2).
---
## 5.3 Application Integration
### 5.3.1 ApplicationImp Changes
> **Deferred identity**: The node public key (`nodeIdentity_`) is not
> available during `ApplicationImp`'s member initializer list — it is
> resolved later in `setup()`. The `Telemetry` object is therefore
> constructed with an empty `serviceInstanceId` and patched via
> `setServiceInstanceId()` once `setup()` has called `getNodeIdentity()`.
`ApplicationImp` (in `src/xrpld/app/main/Application.cpp`) owns a `std::unique_ptr<telemetry::Telemetry> telemetry_`. It is built in the member initializer list via `makeTelemetry(setupTelemetry(...))` with an empty `serviceInstanceId`, then patched in `setup()` by calling `setServiceInstanceId()` with the Base58 node public key (unless the user supplied a custom `service_instance_id`). `start()` and `run()` forward to `telemetry_->start()` / `telemetry_->stop()`, and `getTelemetry()` returns the owned instance.
### 5.3.2 ServiceRegistry Interface Addition
`include/xrpl/core/ServiceRegistry.h` gains a pure-virtual `telemetry::Telemetry& getTelemetry()` (with a forward declaration of `telemetry::Telemetry`), giving every component a uniform accessor for the tracing subsystem.
> **Note:** `Application` extends `ServiceRegistry`, so `getTelemetry()` is
> available on both. Components that hold a `ServiceRegistry&` (e.g.
> `NetworkOPsImp`) call `registry_.get().getTelemetry()`. Components that
> still hold an `Application&` (e.g. `ServerHandler`, `PeerImp`,
A `cmake/FindOpenTelemetry.cmake` module locates the OpenTelemetry C++ SDK. It first tries `find_package(opentelemetry-cpp CONFIG)`, aliasing the imported targets `OpenTelemetry::api`, `OpenTelemetry::sdk`, and `OpenTelemetry::otlp_grpc_exporter`, and falls back to `pkg-config` when no CMake config package is present.
### 5.4.2 CMakeLists.txt Changes
The top-level `CMakeLists.txt` adds an `XRPL_ENABLE_TELEMETRY` option (default `OFF`). When enabled, it runs `find_package(OpenTelemetry REQUIRED)`, defines the `XRPL_ENABLE_TELEMETRY` compile flag, and builds the `xrpl_telemetry` library from the real telemetry sources linked against the OpenTelemetry targets; when disabled, it builds the same target from a no-op `NullTelemetry.cpp` so call sites compile unchanged.
> **Production hardening**: The configurations in this section are starting points. For production deployments where xrpld ships telemetry across a network to a centrally-hosted collector, see [Securing the OTel Pipeline](./secure-OTel.md) for the required mTLS receiver config, NetworkPolicy, and peer trace-context validation.
The authoritative collector config lives in the repo at `docker/telemetry/otel-collector-config.yaml` (with Tempo backend config in `docker/telemetry/tempo.yaml`). The sections below summarize the development and production shapes of that pipeline.
### 5.5.1 Development Configuration
The development collector enables an OTLP receiver on both gRPC (`0.0.0.0:4317`) and HTTP (`0.0.0.0:4318`), a single `batch` processor (1s timeout, batch size 100), and two exporters: a `logging` exporter for console debugging and `otlp/tempo` (insecure) for trace visualization. The single `traces` pipeline wires receiver → batch → both exporters.
### 5.5.2 Production Configuration
The production collector adds TLS on the OTLP gRPC receiver and a richer processor chain: a `memory_limiter` (OOM guard), `batch` (5s timeout, size 512), `tail_sampling`, and an `attributes` processor that hashes sensitive fields (e.g. `tx_account`) and stamps `deployment.environment`. Tail sampling keeps all `ERROR` traces, slow consensus rounds (>5s) and slow RPC requests (>1s), and probabilistically samples the remainder at 10%. Exporters target Grafana Tempo (TLS) and Elastic APM; `health_check` and `zpages` extensions are enabled for operability.
---
## 5.6 Docker Compose Development Environment
> **OTLP** = OpenTelemetry Protocol
The authoritative development stack lives in the repo at `docker/telemetry/docker-compose.yml`. It brings up four services on a shared `xrpld-telemetry` network: an `otel-collector` (otel/opentelemetry-collector-contrib) exposing OTLP gRPC `4317`, OTLP HTTP `4318`, and health check `13133`; `tempo` for trace storage/visualization; `grafana` with provisioned datasources and dashboards (anonymous admin enabled); and an optional `prometheus` for metric correlation.
---
## 5.7 Configuration Architecture
> **OTLP** = OpenTelemetry Protocol
```mermaid
flowchart TB
subgraph config["Configuration Sources"]
cfgFile["xrpld.cfg<br/>[telemetry] section"]
cmake["CMake<br/>XRPL_ENABLE_TELEMETRY"]
end
subgraph init["Initialization"]
parse["setupTelemetry()"]
factory["makeTelemetry()"]
end
subgraph runtime["Runtime Components"]
tracer["TracerProvider"]
exporter["OTLP Exporter"]
processor["BatchProcessor"]
end
subgraph collector["Collector Pipeline"]
recv["Receivers"]
proc["Processors"]
exp["Exporters"]
end
cfgFile --> parse
cmake -->|"compile flag"| parse
parse --> factory
factory --> tracer
tracer --> processor
processor --> exporter
exporter -->|"OTLP"| recv
recv --> proc
proc --> exp
style config fill:#e3f2fd,stroke:#1976d2
style runtime fill:#e8f5e9,stroke:#388e3c
style collector fill:#fff3e0,stroke:#ff9800
```
**Reading the diagram:**
- **Configuration Sources**: `xrpld.cfg` provides runtime settings (endpoint, per-component trace toggles) while the CMake flag controls whether telemetry is compiled in at all. Head sampling is fixed at 1.0 and is not a config option; volume reduction happens via tail sampling in the collector.
- **Initialization**: `setupTelemetry()` parses config values, then `makeTelemetry()` constructs the provider, processor, and exporter objects.
- **Runtime Components**: The `TracerProvider` creates spans, the `BatchProcessor` buffers them, and the `OTLP Exporter` serializes and sends them over the wire.
- **OTLP arrow to Collector**: Trace data leaves the xrpld process via OTLP/HTTP and enters the external Collector pipeline. (OTLP/gRPC is future work — see design decisions §2.2.2.)
- **Collector Pipeline**: `Receivers` ingest OTLP data, `Processors` apply sampling/filtering/enrichment, and `Exporters` forward traces to storage backends (Tempo, etc.).
---
## 5.8 Grafana Integration
> **APM** = Application Performance Monitoring
Step-by-step instructions for integrating xrpld traces with Grafana.
### 5.8.1 Data Source Configuration
#### Tempo (Recommended)
A Tempo datasource (`grafana/provisioning/datasources/tempo.yaml`, provisioned from `docker/telemetry/grafana/`) points at `http://tempo:3200` and enables `tracesToLogs` (linking to Loki on `service.name`/`tx_hash` and mapping `trace_id` → `traceID`), `serviceMap` against Prometheus, the node graph, and Loki search.
#### Elastic APM
Alternatively, an Elasticsearch datasource (`grafana/provisioning/datasources/elastic-apm.yaml`) of type `elasticsearch` points at `http://elasticsearch:9200` against the `apm-*` index, using `@timestamp` as the time field and mapping the log message/level fields.
### 5.8.2 Dashboard Provisioning
A dashboard provider (`grafana/provisioning/dashboards/dashboards.yaml`) loads the `xrpld` dashboard folder from disk (`/var/lib/grafana/dashboards/rippled`), polling for changes every 30s with deletion disabled.
### 5.8.3 Example Dashboard: RPC Performance
An example `xrpld RPC Performance` dashboard (uid `xrpld-rpc-performance`) sourced from Tempo via TraceQL provides four panels: RPC latency by command (heatmap), RPC error rate by command (timeseries), the top 10 slowest RPC commands by average duration (table), and a recent-traces table.
### 5.8.4 Example Dashboard: Transaction Tracing
An example `xrpld Transaction Tracing` dashboard (uid `xrpld-tx-tracing`) over Tempo provides three panels: transaction throughput (`tx.receive` rate, stat), cross-node relay count (average `span.relay_count` on `tx.relay`, timeseries), and a table of transaction validation errors (`tx.validate` with `status.code=error`).
{resource.service.name="xrpld" && name="rpc.command.account_info"} | avg(duration) by (resource.service.instance.id)
```
### 5.8.6 Correlation with PerfLog
To correlate OpenTelemetry traces with existing PerfLog data:
**Step 1: Configure Loki to ingest PerfLog**
Configure a Promtail scrape job (`promtail-config.yaml`) that tails `/var/log/rippled/perf*.log`, parses each JSON line, and promotes `trace_id`, `ledger_seq`, and `tx_hash` to Loki labels.
**Step 2: Add trace_id to PerfLog entries**
Modify PerfLog so its JSON output includes a `trace_id` field whenever a valid span is active: fetch the current span from the OpenTelemetry runtime context, and if its context is valid, render the trace ID as a 32-character lowercase hex string into the log entry.
**Step 3: Configure Grafana trace-to-logs link**
In the Tempo datasource, set the `tracesToLogs` derived field to link to Loki on the `trace_id` and `tx_hash` tags, with `filterByTraceID: true`.
### 5.8.7 Correlation with Insight/OTel System Metrics
To correlate traces with Beast Insight system metrics:
**Step 1: Export Insight metrics to Prometheus**
Beast Insight metrics are exported natively via OTLP to the OTel Collector,
which exposes them on the Prometheus endpoint alongside spanmetrics. Configure
the `[insight]` section of `xrpld.cfg` with `server=otel`,
`endpoint=http://localhost:4318/v1/metrics`, and `prefix=xrpld`; no separate
StatsD exporter or Prometheus scrape job is needed when using `server=otel`.
**Step 2: Add exemplars to metrics**
The OpenTelemetry SDK automatically adds exemplars (trace IDs) to metrics when using the Prometheus exporter, linking metric spikes to specific traces.
- **Budget Constraints? (Yes)**: Leads to open-source options. If you already run Grafana or Elastic, pick the matching backend; otherwise default to Grafana Tempo.
- **Budget Constraints? (No) → Prefer SaaS?**: If you want a managed service, choose between Datadog (enterprise support) and Honeycomb (developer-focused). If not, fall back to open-source.
- **Terminal nodes (Tempo / Elastic / Honeycomb / Datadog)**: Each represents a concrete backend choice, all of which feed into the same final step.
- **Configure Collector**: Regardless of backend, you always finish by configuring the OTel Collector to export to your chosen destination.
- **Validator / Stock Nodes**: All xrpld nodes emit trace data via OTLP. Validators and stock nodes are grouped separately because they may reside in different network zones.
- **Collector Cluster (DC1, DC2)**: Regional collectors receive OTLP from nodes in their datacenter, apply processing (sampling, enrichment), and fan out to multiple backends. Enrichment includes deployment-tier tagging: each collector stamps `deployment.environment` and (as a fallback) `xrpl.network.type` so one Grafana stack can filter data from many collectors by tier.
- **Storage Backends**: Tempo and Elastic provide queryable trace storage; S3/GCS Archive provides long-term cold storage for compliance or post-incident analysis.
- **Grafana Dashboards**: The single visualization layer that queries both Tempo and Elastic, giving operators a unified view of all traces.
> **Note**: Production deployments should use multiple collector instances behind a load balancer for high availability. The diagram shows a simplified single-collector topology for clarity.
| **Gateway** | Central collector(s) | Centralized processing | Single point of failure |
**Recommendation**: Use **Gateway** pattern with regional collectors for xrpld networks:
- One collector cluster per datacenter/region
- Tail-based sampling at collector level
- Multiple export destinations for redundancy
### 7.4.2 Sampling Strategy
```mermaid
flowchart LR
subgraph head["Head Sampling (Node)"]
hs[Node-level head sampling<br/>fixed at 100%<br/>not configurable]
end
subgraph tail["Tail Sampling (Collector)"]
ts1[Keep all errors]
ts2[Keep slow >5s]
ts3[Keep 10% rest]
end
head --> tail
ts1 --> final[Final Traces]
ts2 --> final
ts3 --> final
style head fill:#0d47a1,stroke:#082f6a,color:#fff
style tail fill:#1b5e20,stroke:#0d3d14,color:#fff
style hs fill:#0d47a1,stroke:#082f6a,color:#fff
style ts1 fill:#1b5e20,stroke:#0d3d14,color:#fff
style ts2 fill:#1b5e20,stroke:#0d3d14,color:#fff
style ts3 fill:#1b5e20,stroke:#0d3d14,color:#fff
style final fill:#bf360c,stroke:#8c2809,color:#fff
```
**Reading the diagram:**
- **Head Sampling (Node)**: xrpld pins head sampling at 100% (sample everything) and does not expose a configurable ratio. This is intentional: a per-node ratio would let different nodes make divergent keep/drop decisions for the same distributed trace, producing broken/partial traces. xrpld uses a `ParentBased` sampler so spans inheriting a remote parent honor the upstream decision. Volume reduction is delegated to the collector's tail sampling.
- **Tail Sampling (Collector)**: The second filter -- the collector inspects completed traces and applies rules: keep all errors, keep anything slower than 5 seconds, and keep 10% of the remainder.
- **Arrow head → tail**: All head-sampled traces flow to the collector, where tail sampling further reduces volume while preserving the most valuable data.
- **Final Traces**: The output after both sampling stages; this is what gets stored and queried. The two-stage approach balances cost with debuggability.
- **RPC Error Rate Spike** (critical, `for: 2m`): fires when the error rate across `rpc.command.*` spans exceeds 5%. Error _rate_ is a ratio, so it must divide the error-span rate by the total-span rate — a single TraceQL `rate()` returns spans/second, not a percentage, and would fire on traffic volume alone. This uses span metrics emitted by the collector's `spanmetrics` connector (Prometheus datasource), not a TraceQL query:
> **Note**: The Consensus Round Slow and Transaction Throughput Drop rules use TraceQL aggregates (`avg(duration)`, `rate()`), which require Tempo 2.3+ with TraceQL metrics enabled. Verify aggregate query support in your Tempo version before provisioning. The RPC Error Rate Spike rule instead queries Prometheus span metrics (collector `spanmetrics` connector), so it needs that connector enabled in the collector pipeline.
---
## 7.7 PerfLog and Insight Correlation
> **OTLP** = OpenTelemetry Protocol
How to correlate OpenTelemetry traces with existing xrpld observability.
- **xrpld Node (three sources)**: A single node emits three independent data streams -- OpenTelemetry spans, PerfLog JSON logs, and Beast Insight StatsD metrics.
- **Data Collection layer**: Each stream has its own collector -- OTel Collector for spans, Promtail/Fluentd for logs, and a StatsD exporter for metrics. They operate independently.
- **Storage layer (Tempo, Loki, Prometheus)**: Each data type lands in a purpose-built store optimized for its query patterns (trace search, log grep, metric aggregation).
- **Grafana Correlation Panel**: The key integration point -- Grafana queries all three stores and links them via shared fields (`trace_id`, `tx_hash`, `ledger_seq`), enabling a single-pane debugging experience.
**Step 4: Check Insight metrics for the time window**
```
# In Grafana with Prometheus
rate(xrpld_tx_applied_total[1m])
@ timestamp_from_trace
```
### 7.7.4 Unified Dashboard Example
A single dashboard (uid `xrpld-unified`) that ties traces, metrics, and logs together across the Tempo, Prometheus, and Loki datasources:
- **Transaction Latency (Traces)** (timeseries, Tempo): `histogram_over_time(duration)` of `tx.receive` spans.
- **Transaction Rate (Metrics)** (timeseries, Prometheus): `rate(xrpld_tx_received_total[5m])` per instance, with a data link that opens the matching `tx.receive` traces in Tempo.
- **Trace Search** (table, Tempo): all `xrpld` traces, with per-row data links on `traceID` that jump to the trace in Tempo and to the correlated logs in Loki (`{job="xrpld"} |= "<traceID>"`).
The cross-datasource data links are what make this a single-pane debugging view; the correlation fields they rely on are listed in section 7.7.2.
---
_Previous: [Implementation Phases](./06-implementation-phases.md)_ | _Next: [Appendix](./08-appendix.md)_ | _Back to: [Overview](./OpenTelemetryPlan.md)_
- **rpc.request (blue, top)**: The entry point — every traced transaction starts as an RPC call; this root span is the parent of all downstream work.
- **tx.validate and pathfind.request (green/teal, first fork)**: The RPC request fans out into transaction validation and, for cross-currency payments, a PathFinding branch (`pathfind.request` -> `pathfind.compute`).
- **tx.relay -> Peer Spans (teal, middle)**: After validation, the transaction is relayed to peers A, B, and C in parallel; each `peer.send` is a sibling child span showing fan-out across the network.
- **context propagation (dashed arrow)**: The dotted line from `peer.send Peer A` to `consensus.round` represents the trace context crossing a node boundary — the receiving validator picks up the same `trace_id` and continues the trace.
- **consensus.round -> tx.apply -> TxQ Spans (green, lower)**: Once consensus accepts the transaction, it is applied to the ledger; the TxQ spans (`txq.enqueue`, `txq.apply`, `fee.escalate`) capture queue depth and fee escalation behavior.
- **Validator Spans (purple, detached)**: `validator.list.fetch` and `validator.manifest` are independent workflows for UNL management — they run on their own traces and are linked to consensus via Span Links, not parent-child relationships.
---
## 8.3 References
> **OTLP** = OpenTelemetry Protocol
### OpenTelemetry Resources
1. [OpenTelemetry C++ SDK](https://github.com/open-telemetry/opentelemetry-cpp)
| [Phase11_taskList.md](./Phase11_taskList.md) | Third-party data collection pipelines (future) |
> **Note**: Phases 1 and 6 do not have separate task list files. Phase 1 tasks are documented in [06-implementation-phases.md §6.2](./06-implementation-phases.md). Phase 6 tasks are documented in [06-implementation-phases.md §6.7](./06-implementation-phases.md).
---
## 8.6 Phase 9–11 Cross-Reference Guide
This guide maps Phase 9–11 content to its location across the documentation.
### Phase 9: Internal Metric Instrumentation Gap Fill
# [OpenTelemetry](00-tracing-fundamentals.md) Distributed Tracing Implementation Plan for xrpld
## Executive Summary
> **OTLP** = OpenTelemetry Protocol
This document provides a comprehensive implementation plan for integrating OpenTelemetry distributed tracing into the xrpld XRP Ledger node software. The plan addresses the unique challenges of a decentralized peer-to-peer system where trace context must propagate across network boundaries between independent nodes.
### Key Benefits
- **End-to-end transaction visibility**: Track transactions from submission through consensus to ledger inclusion
- **Consensus round analysis**: Understand timing and behavior of consensus phases across validators
- **RPC performance insights**: Identify slow handlers and optimize response times
- **Network topology understanding**: Visualize message propagation patterns between peers
- **Incident debugging**: Correlate events across distributed nodes during issues
| **7** | [Observability Backends](./07-observability-backends.md) | Backend selection guide and production architecture |
| **8** | [Appendix](./08-appendix.md) | Glossary, references, version history |
| **9** | [Data Collection Reference](./09-data-collection-reference.md) | Complete inventory of spans, attributes, metrics, and dashboards |
| **Sec** | [Securing the OTel Pipeline](./secure-OTel.md) | Threat model and hardening (mTLS, peer trace-context validation) |
---
## 0. Tracing Fundamentals
This document introduces distributed tracing concepts for readers unfamiliar with the domain. It covers what traces and spans are, how parent-child and follows-from relationships model causality, how context propagates across service boundaries, and how sampling controls data volume. It also maps these concepts to xrpld-specific scenarios like transaction relay and consensus.
The xrpld node consists of several key components that require instrumentation for comprehensive distributed tracing. The main areas include the RPC server (HTTP/WebSocket), Overlay P2P network, Consensus mechanism (RCLConsensus), JobQueue for async task execution, PathFinding, Transaction Queue (TxQ), fee escalation (LoadManager), ledger acquisition, validator management, and existing observability infrastructure (PerfLog, Insight/StatsD, Journal logging).
Key trace points span across transaction submission via RPC, peer-to-peer message propagation, consensus round execution, ledger building, path computation, transaction queue behavior, fee escalation, and validator health. The implementation prioritizes high-value, low-risk components first: RPC handlers provide immediate value with minimal risk, while consensus tracing requires careful implementation to avoid timing impacts.
➡️ **[Read full Architecture Analysis](./01-architecture-analysis.md)**
The OpenTelemetry C++ SDK is selected for its CNCF backing, active development, and native performance characteristics. Traces are exported via OTLP/HTTP to an OpenTelemetry Collector, which provides flexible routing and sampling. OTLP/gRPC is planned future work (see design decisions §2.2.2).
Span naming follows a hierarchical `<component>.<operation>` convention (e.g., `rpc.submit`, `tx.relay`, `consensus.round`). Context propagation uses W3C Trace Context headers for HTTP and embedded Protocol Buffer fields for P2P messages. The implementation coexists with existing PerfLog and Insight observability systems through correlation IDs.
**Data Collection & Privacy**: Telemetry collects only operational metadata (timing, counts, hashes) — never sensitive content (private keys, balances, amounts, raw payloads). Privacy protection includes account hashing, configurable redaction, sampling, and collector-level filtering. Node operators retain full control over telemetry configuration.
➡️ **[Read full Design Decisions](./02-design-decisions.md)**
---
## 3. Implementation Strategy
The telemetry code is organized under `include/xrpl/telemetry/` for headers and `src/libxrpl/telemetry/` for implementation. Key principles include RAII-based span management via `SpanGuard` (with `discard()` for dropping unwanted spans), a `FilteringSpanProcessor` that intercepts `OnEnd()` to prevent discarded spans from entering the export pipeline, conditional compilation with `XRPL_ENABLE_TELEMETRY`, and minimal runtime overhead through batch processing and efficient sampling.
Performance optimization strategies include head sampling fixed at 100% (intentionally not configurable, so trace keep/drop decisions stay coherent across nodes), tail-based sampling at the collector for errors and slow traces to reduce volume, batch export to reduce network overhead, and conditional instrumentation that compiles to no-ops when disabled.
➡️ **[Read full Implementation Strategy](./03-implementation-strategy.md)**
Configuration is handled through the `[telemetry]` section in `xrpld.cfg` with options for enabling/disabling, exporter selection, endpoint configuration, and component-level filtering. Head sampling is fixed at 1.0 (not operator-configurable); volume reduction is done by tail sampling in the collector. CMake integration includes a `XRPL_ENABLE_TELEMETRY` option for compile-time control.
OpenTelemetry Collector configurations are provided for development and production (with tail-based sampling, Tempo, and Elastic APM). Docker Compose examples enable quick local development environment setup.
➡️ **[View full Configuration Reference](./05-configuration-reference.md)**
---
## 6. Implementation Phases
The implementation spans 13 weeks across 8 phases:
Grafana Tempo is recommended for all environments due to its cost-effectiveness and Grafana integration, while Elastic APM is ideal for organizations with existing Elastic infrastructure.
The recommended production architecture uses a gateway collector pattern with regional collectors performing tail-based sampling, routing traces to multiple backends (Tempo for primary storage, Elastic for log correlation, S3/GCS for long-term archive).
The appendix contains a glossary of OpenTelemetry and xrpld-specific terms, references to external documentation and specifications, version history for this implementation plan, and a complete document index.
➡️ **[View Appendix](./08-appendix.md)**
---
## 9. Data Collection Reference
A single-source-of-truth reference documenting every piece of telemetry data collected by xrpld. Covers all 16 OpenTelemetry spans with their 22 attributes, all StatsD metrics (gauges, counters, histograms, overlay traffic), SpanMetrics-derived Prometheus metrics, and all 10 Grafana dashboards. Includes Tempo search guides and Prometheus query examples.
➡️ **[View Data Collection Reference](./09-data-collection-reference.md)**
---
## Securing the OTel Pipeline
Threat model and hardening guidance for production deployments where xrpld nodes ship telemetry to a centrally-hosted collector across an untrusted network. Covers the two attack surfaces (collector ingress and peer trace-context spoofing) and the chosen defenses: mTLS as primary collector auth, NetworkPolicy as defense-in-depth, and source-side validation plus per-peer rate limiting for the `protocol::TraceContext` field on peer messages.
➡️ **[View Securing the OTel Pipeline](./secure-OTel.md)**
---
_This document provides a comprehensive implementation plan for integrating OpenTelemetry distributed tracing into the xrpld XRP Ledger node software. For detailed information on any section, follow the links to the corresponding sub-documents._
> **Goal**: Build tools that generate realistic XRPL traffic to validate the full Phases 1-9 telemetry stack end-to-end — all spans, attributes, metrics, dashboards, and log-trace correlation — under controlled load.
# Phase 11: Third-Party Data Collection Pipelines — Task List
> **Status**: Future Enhancement
>
> **Goal**: Build a custom OTel Collector receiver that periodically polls xrpld's admin RPCs and exports structured metrics for external consumers — making all XRPL health, validator, peer, fee, and DEX data available as Prometheus/OTLP metrics without xrpld code changes.
| [09-data-collection-reference.md](./09-data-collection-reference.md) | Defines full metric inventory including third-party metrics |
| [Phase10_taskList.md](./Phase10_taskList.md) | Prerequisite — validation harness for testing |
### Third-Party Consumer Gap Analysis
This phase addresses the cross-cutting gap identified during research: **xrpld has no native Prometheus/OTLP metrics export for data accessible only via RPC**. Every consumer (exchanges, payment processors, analytics providers, validators, researchers, compliance firms, custodians) must build custom JSON-RPC polling and conversion. This receiver centralizes that work.
| Consumer Category | Data Unlocked by This Phase |
- `xrpl_object_count{type="<name>"}` for each counted object type (Transaction, Ledger, NodeObject, STTx, STLedgerEntry, InboundLedger, Pathfinder, etc.)
**Cardinality note**: Per-peer metrics use truncated keys. For large peer sets (50+), the aggregate distribution gauges are preferred over per-peer labels.
---
## Task 11.5: Validator & Amendment Collector
**Objective**: Poll `validators` and `feature` to export validator health and amendment voting status.
**What to do**:
- Implement `validatorCollector`:
**From `validators` RPC:**
- `xrpl_trusted_validators_count`
- `xrpl_validator_signing` (0 or 1 — whether local validator is signing)
**From `feature` RPC:**
- `xrpl_amendment_enabled_count` — total enabled amendments
- `xrpl_amendment_majority_count` — amendments with majority but not yet enabled
- Add alerting playbook (what to do for each Tier 1/Tier 2 alert)
---
## Task 11.12: Alert Rules for External Dashboard Parity Metrics
> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity) — 18 alert rules ported from the community [xrpl-validator-dashboard](https://github.com/realgrapedrop/xrpl-validator-dashboard).
**Objective**: Document the external dashboard's "fast path" pattern as a future optimization for real-time panels.
**Pattern**: A lightweight Prometheus scrape endpoint (separate from OTLP pipeline) that polls critical metrics every 2-5s, bypassing the 10s OTLP metric reader interval and Prometheus scrape interval.
**Use case**: Real-time state panels (server state, ledger age, peer count) where 10-15s latency is too slow for operational dashboards.
**Decision**: Document as a future option, not implement now. The current 10s interval is acceptable for v1. The external dashboard achieves 2-5s freshness by polling RPC directly, which is what the Phase 11 receiver already does. Adding a separate scrape endpoint to xrpld would only be needed if sub-second metric freshness is required from the internal metrics pipeline.
**What to document**:
- Architecture comparison: OTLP pipeline (10-15s) vs. direct scrape (2-5s) vs. push gateway
- When to consider: operator feedback indicating 10s is insufficient for alerting SLOs
- How to implement if needed: add `/metrics` HTTP endpoint to xrpld with Prometheus client library
- Trade-offs: additional port, additional dependency, duplication with OTLP metrics
> **Goal**: Complete RPC tracing coverage with unit tests, Grafana search filters, PathFind instrumentation, and config hardening. Build on the Phase 1c SpanGuard factory foundation to achieve production-quality RPC observability.
>
> **Scope**: Unit tests for core telemetry, Grafana Tempo search filters, PathFind RPC tracing, config validation (`std::clamp`).
**Reason**: W3C context propagation (`traceparent`/`tracestate` headers) requires a consumer — in Phase 2, RPC spans are entirely local to the node. Phase 3 introduces cross-node transaction tracing via protobuf context propagation, which is the first use case for extracted trace context. Implementing it here without a consumer would be dead code.
**Status**: COMPLETE (superseded by Phase 1c design)
**Original plan**: Add `XRPL_TRACE_PEER` and `XRPL_TRACE_LEDGER` macros.
**Actual implementation**: Phase 1c replaced all tracing macros with the `SpanGuard::span(TraceCategory, prefix, name)` factory pattern. The `TraceCategory` enum (`Rpc`, `Transactions`, `Consensus`, `Peer`, `Ledger`) serves the same conditional-creation purpose without macros. No separate task needed — the factory already supports all categories.
---
## Task 2.3: Add shouldTraceLedger() to Telemetry Interface
**Objective**: The `Setup` struct has a `traceLedger` field but there's no corresponding virtual method. Add it for interface completeness.
-`src/tests/libxrpl/CMakeLists.txt` — test target with optional OTel linking
---
## Task 2.5: Enhance RPC Span Attributes
**Status**: DEFERRED (low priority)
**Reason**: The high-value attributes (`command`, `version`, `role`, `status`) are already set by Phase 1c. The remaining HTTP transport-level attributes (`http.method`, `net.peer.ip`, `http.status_code`) provide limited additional insight since:
-`http.method` is always POST for JSON-RPC
-`net.peer.ip` is debug-level info available in logs
-`duration_ms` is redundant with span duration (OTel captures start/end time natively)
These can be added later if dashboard queries specifically need them. The node health attributes (Task 2.8) provide far more operational value and were prioritized instead.
---
## Task 2.6: Build Verification and Performance Baseline
**Objective**: Verify the build succeeds with and without telemetry, and establish a performance baseline.
**What to do**:
1. Build with `telemetry=ON` and verify no compilation errors
2. Build with `telemetry=OFF` and verify no regressions
Node health (`amendment_blocked`, `server_state`) is not part of the telemetry surface. Operators consume the same data via the existing `server_info` / `server_state` RPC commands, so duplicating it on traces adds storage and cardinality cost without new value. The OTel C++ SDK 1.18.0 also does not support runtime updates to the resource, ruling out resource-level emission of these dynamic-by-nature flags.
---
## Task 2.9: PathFind RPC Instrumentation
**Status**: COMPLETE
**Objective**: Trace the path_find and ripple_path_find RPC handlers to capture request latency and computation cost.
**Spans added**:
-`pathfind.request` — wraps `doPathFind()` and `doRipplePathFind()` RPC handlers
-`pathfind.update_all` — wraps `PathRequestManager::updateAll()` on ledger close (`pathfind_ledger_index`, `pathfind_num_requests` attrs; emitted only when active subscriptions exist)
-`pathfind.discover` — wraps the entire per-source-asset loop in `PathRequest::findPaths()` (`pathfind_search_level`, `pathfind_num_paths` attrs). One span per RPC call instead of N (one per source asset). Trade-off: per-asset breakdown is lost; storage and cardinality bounded.
**Attribute namespacing**: All pathfind attributes use the `pathfind_*` underscore form per the Phase 1c naming-spec rule 5.
## Task 2.10: RPC and PathFind Span Attribute Gap Fill
**Status**: COMPLETE
**Objective**: Wire up workflow-identifying attributes that enable filtering and grouping traces by request characteristics without drilling into child spans.
> **Goal**: Trace the full transaction lifecycle from RPC submission through peer relay, including cross-node context propagation via Protocol Buffer extensions. This is the WALK phase that demonstrates true distributed tracing.
## Task 3.5: Instrument HashRouter for Dedup Visibility
**Objective**: Make transaction deduplication visible in traces by recording HashRouter decisions as span attributes/events.
**What to do**:
- Edit `src/xrpld/overlay/detail/PeerImp.cpp` (in handleTransaction):
- After calling `HashRouter::shouldProcess()` or `addSuppressionPeer()`:
- Record `suppressed` attribute (true/false)
- Record `tx_flags` showing current HashRouter state (SAVED, TRUSTED, etc.)
- Add `tx.first_seen` or `tx.duplicate` event
- This is NOT a modification to HashRouter itself — just recording its decisions as span attributes in the existing PeerImp instrumentation from Task 3.3.
**Key modified files**:
- `src/xrpld/overlay/detail/PeerImp.cpp` (same changes as 3.3, logically grouped)
---
## Task 3.6: Context Propagation in Transaction Relay
**Objective**: Verify all Phase 3 changes compile and work correctly.
**What to do**:
1. Build with `telemetry=ON` — verify no compilation errors
2. Build with `telemetry=OFF` — verify no regressions
3. Run existing unit tests
4. Verify protobuf regeneration produces correct C++ code
5. Document any issues encountered
**Verification Checklist**:
- [ ] Protobuf changes generate valid C++
- [ ] Build succeeds with telemetry ON
- [ ] Build succeeds with telemetry OFF
- [ ] Existing tests pass
- [ ] No undefined symbols from new telemetry calls
---
## Task 3.8: Transaction Span Peer Version Attribute
> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity) — adds peer version context inspired by the community [xrpl-validator-dashboard](https://github.com/realgrapedrop/xrpl-validator-dashboard).
>
> **Upstream**: Phase 2 (RPC span infrastructure must exist).
> **Downstream**: Phase 10 (validation checks for this attribute).
**Objective**: Add the relaying peer's xrpld version to `tx.receive` spans so operators can correlate transaction issues with peer version mismatches during network upgrades.
**What to do**:
- Edit `src/xrpld/overlay/detail/PeerImp.cpp`:
- In the `tx.receive` span block (after existing `peer_id` setAttribute call):
- Add `peer_version` (string) — from `this->getVersion()`
- Only set if `getVersion()` returns a non-empty string (avoid empty-string attributes)
**Rationale**: Transaction relay is where version mismatches cause subtle serialization or validation bugs. Tracing "this tx came from a v2.3.0 peer" helps diagnose compatibility issues. The community dashboard tracks peer versions externally; this brings version awareness into the trace itself.
**Key modified files**:
- `src/xrpld/overlay/detail/PeerImp.cpp`
**Exit Criteria**:
- [ ] `tx.receive` spans carry `peer_version` attribute with a non-empty version string
- [ ] Attribute is omitted (not set to empty string) when `getVersion()` returns empty
- [ ] Attribute visible in Tempo trace detail view
**Objective**: Add workflow-identifying attributes to transaction spans so operators can filter by transaction type and see outcomes without off-chain correlation.
| 3.11 | TX/TxQ span attribute gap fill | 0 | 5 | 3.3, 3.10 |
**Parallel work**: Tasks 3.1 and 3.4 can start in parallel. Task 3.2 depends on 3.1. Tasks 3.3 and 3.5 depend on 3.2. Task 3.6 depends on 3.3 and 3.5. Task 3.8 depends on 3.3 (span must exist). Task 3.9 depends on 3.2 and 3.3. Task 3.10 depends on 3.4 (tx.process span must exist).
> **Goal**: Full observability into consensus rounds — track round lifecycle, phase transitions, proposal handling, and validation. This is the RUN phase that completes the distributed tracing story.
>
> **Scope**: RCLConsensus instrumentation for round starts, phase transitions (open/establish/accept), proposal send/receive, validation handling, and correlation with transaction traces from Phase 3.
**Objective**: Trace validation send and receive to show ledger validation flow.
**Status**: DONE. Both send and receive paths are instrumented.
**What was done**:
- In `Adaptor::validate()` (called from `doAccept()`):
- Creates `consensus.validation.send` span via `Adaptor::createValidationSpan()`
- Uses `SpanGuard::linkedSpan()` to create a follows-from link to the round span
- Thread-safe: uses `roundSpanContext_` snapshot (captured on consensus thread,
read on jtACCEPT thread)
- Sets `xrpl.ledger.seq` and `proposing` attributes
- In `PeerImp::onMessage(TMValidation)`:
- Creates `consensus.validation.receive` span
- Sets `trusted` attribute (bool)
- Sets `xrpl.ledger.seq` attribute
**Not implemented** (deferred to Phase 4b — cross-node propagation):
- Validated ledger hash, signing time attributes on send span (see Task 4.8)
**Key modified files**:
-`src/xrpld/app/consensus/RCLConsensus.cpp`
---
## Task 4.5: Add Consensus-Specific Attributes ✅
**Objective**: Enrich consensus spans with detailed attributes for debugging and analysis.
**Status**: DONE. All core attributes are set across various spans, including the previously missing `tx_count` and `disputes_count`.
**Implemented attributes** (across various spans):
-`xrpl.ledger.seq` — on `consensus.round`, `consensus.accept.apply`
-`xrpl.consensus.round` — on `consensus.proposal.send`
-`xrpl.consensus.mode` — on `consensus.round`, `consensus.ledger_close`
-`proposers` — on `consensus.accept`, `consensus.establish`, `consensus.update_positions`
-`converge_percent` — on `consensus.establish`, `consensus.update_positions`, `consensus.check`
-`tx_count` — on `consensus.accept.apply` span (in `doAccept()`)
-`disputes_count` — on `consensus.update_positions` span (in `updateOurPositions()`)
**Design notes**:
-`phase` — phases distinguished by span names instead
-`phase_duration_ms` — span duration captures this
**Key modified files**:
-`src/xrpld/app/consensus/RCLConsensus.cpp`
-`src/xrpld/consensus/Consensus.h`
---
## Task 4.6: Correlate Transaction and Consensus Traces ✅
**Objective**: Link transaction traces from Phase 3 with consensus traces so you can follow a transaction from submission through consensus into the ledger.
**Status**: DONE. Transaction-consensus correlation implemented via `tx.included` events in `doAccept()`.
**What was done**:
- In `doAccept()` (RCLConsensus.cpp):
- Records `tx.included` events on the `consensus.accept.apply` span for each transaction in the accepted set
- Each event includes `xrpl.tx.id` attribute with the transaction hash
- This links consensus traces to individual transactions
**Key modified files**:
-`src/xrpld/app/consensus/RCLConsensus.cpp`
---
## Task 4.7: Build Verification and Testing ✅
**Objective**: Verify all Phase 4 changes compile and don't affect consensus timing.
**What to do**:
1. Build with `telemetry=ON` — verify no compilation errors
2. Build with `telemetry=OFF` — verify no regressions (critical for consensus code)
3. Run existing consensus-related unit tests
4. Verify that `SpanGuard` factory methods compile to no-ops when disabled
5. Check that no consensus-critical code paths are affected by instrumentation overhead
**Verification Checklist**:
- [x] Build succeeds with telemetry ON
- [x] Build succeeds with telemetry OFF
- [x] Existing consensus tests pass
- [x]`SpanGuard` no-op implementation prevents overhead when telemetry is OFF
- [x] Phase timing instrumentation doesn't use blocking operations
---
## Task 4.8: Consensus Validation Span Enrichment — NOT DONE
> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity) — adds validation agreement context inspired by the community [xrpl-validator-dashboard](https://github.com/realgrapedrop/xrpl-validator-dashboard).
>
> **Upstream**: Phase 4 tasks 4.1-4.4 (span creation must exist).
**Objective**: Add ledger hash, validation type, and quorum data to consensus validation spans on both send and receive paths. This enables trace-level validation agreement analysis — filter by ledger hash to see which validators agreed for a given ledger.
**Status**: Not implemented. None of the enrichment attributes are set. The `consensus.validation.send` span only has `ledger.seq` and `proposing`. The `consensus.accept` span has `quorum` set to `result.proposers` (not the actual validator quorum from `app_.validators().quorum()`). No `PeerImp.cpp` changes were made.
**Rationale**: The external dashboard's most valuable feature is validation agreement tracking. By recording the ledger hash on both outgoing and incoming validation spans, we create the raw data for agreement analysis at the trace level. Example Tempo query:
Phase 7's `ValidationTracker` builds metric-level aggregation (1h/24h agreement %) on top of this data.
**Key modified files (not yet modified)**:
-`src/xrpld/app/consensus/RCLConsensus.cpp`
-`src/xrpld/overlay/detail/PeerImp.cpp`
**Exit Criteria**:
- [x]`consensus.validation.send` spans carry `ledger_hash` and `full_validation`
- [ ]`peer.validation.receive` spans carry `xrpl.peer.validation.ledger_hash` and `xrpl.peer.validation.full`
- [ ]`consensus.accept` spans carry `validation_quorum` and `proposers_validated`
- [x] Ledger hash attributes match between send and receive for the same ledger
- [ ] No impact on consensus performance
---
## Task 4.9: Consensus Span Attribute Gap Fill
**Status**: COMPLETE
**Objective**: Add workflow-critical attributes to consensus spans that enable operators to understand consensus outcomes, identify bow-out proposals, and correlate validations to specific ledgers.
| 4.9 | Consensus span attribute gap fill | ✅ Done | 0 | 2 | 4.1-4.5 |
**Parallel work**: Tasks 4.2, 4.3, and 4.4 can run in parallel after 4.1 is complete. Task 4.5 depends on all three. Task 4.6 depends on 4.2 and Phase 3. Task 4.8 depends on 4.4 (validation spans must exist).
#### Close Time Attributes (consensus.accept.apply)
The `consensus.accept.apply` span captures ledger close time agreement details
driven by `avCT_CONSENSUS_PCT` (75% validator agreement threshold):
- **`close_time`** — Agreed-upon ledger close time (epoch seconds). When validators disagree (`consensusCloseTime == epoch`), this is synthetically set to `prevCloseTime + 1s`.
- **`close_time_correct`** — `true` if validators reached agreement, `false` if they "agreed to disagree" (close time forced to prev+1s).
- **`close_resolution_ms`** — Rounding granularity for close time (starts at 30s, decreases as ledger interval stabilizes).
- **`consensus_state`** — `"finished"` (normal) or `"moved_on"` (consensus failed, adopted best available).
- **`proposing`** — Whether this node was proposing.
- **`round_time_ms`** — Total consensus round duration.
- **`parent_close_time`** — Previous ledger's close time (epoch seconds). Enables computing close-time deltas across consecutive rounds without correlating separate spans.
- **`close_time_self`** — This node's own proposed close time before consensus voting.
- **`close_time_vote_bins`** — Number of distinct close-time vote bins from peer proposals. Higher values indicate less agreement among validators.
- **`resolution_direction`** — Whether close-time resolution `"increased"` (coarser), `"decreased"` (finer), or stayed `"unchanged"` relative to the previous ledger.
> **Goal**: Production readiness — Grafana dashboards, spanmetrics pipeline, operator runbook, alert definitions, and final integration testing. This phase ensures the telemetry system is useful and maintainable in production.
- **OTelGaugeImpl**: Uses `opentelemetry::metrics::ObservableGauge<uint64_t>` with an async callback. `set(value)` stores value atomically; callback reads it during collection.
- **OTelMeterImpl**: Wraps `opentelemetry::metrics::Counter<uint64_t>`. `increment(amount)` calls `counter->Add(amount)`. Semantically identical to Counter but unsigned.
- The prefix + group + name concatenation must produce the same string as `StatsDCollector`'s format
- Use underscores as separators (matching StatsD convention)
- Verify in integration test that key Prometheus queries still return data:
- `xrpld_LedgerMaster_Validated_Ledger_Age`
- `xrpld_Peer_Finder_Active_Inbound_Peers`
- `xrpld_rpc_requests`
**Key consideration**: OTel Prometheus exporter may normalize metric names differently than StatsD receiver. Test this early (Task 7.2) and adjust naming strategy if needed. The OTel SDK's Prometheus exporter adds `_total` suffix to counters and converts dots to underscores — match existing conventions.
---
## Task 7.6: Update Grafana Dashboards
**Objective**: Update the 3 StatsD dashboards if any metric names change due to OTLP export format differences.
**What to do**:
- If Task 7.5 confirms metric names are preserved exactly, no dashboard changes needed.
- If OTLP export produces different names (e.g., `_total` suffix on counters), update:
> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity) — the most valuable metric from the community [xrpl-validator-dashboard](https://github.com/realgrapedrop/xrpl-validator-dashboard).
**Objective**: Implement a stateful class that tracks whether our validator's validations agree with network consensus, maintaining rolling 1h and 24h windows with an 8-second grace period and 5-minute late repair window.
**Parallel work**: Tasks 7.4 and 7.5 can run in parallel after 7.2/7.3 complete. Task 7.6 depends on 7.5's findings. Tasks 7.7 and 7.8 can run in parallel after 7.6. Tasks 7.10-7.14 can all run in parallel after 7.2. Task 7.15 depends on 7.2. Task 7.16 depends on 7.9. Task 7.9 depends on 7.2 and Phase 4 Task 4.8.
# Phase 8: Log-Trace Correlation and Centralized Log Ingestion — Task List
> **Goal**: Inject trace context (trace_id, span_id) into xrpld's Journal log output for log-trace correlation, and add OTel Collector filelog receiver to ingest logs into Grafana Loki for unified observability.
>
> **Scope**: Two independent sub-phases — 8a (code change: trace_id in logs) and 8b (infra only: filelog receiver to Loki). No changes to the `beast::Journal` public API.
**Objective**: Add OTel trace context to every log line that is emitted within an active span.
**What to do**:
- Edit `src/libxrpl/basics/Log.cpp`:
- In `Logs::format()` (around line 346), after severity is appended, check for active OTel span. The implementation checks the context value directly to avoid the heap allocation that `GetSpan()` performs on the no-span path:
```cpp
#ifdef XRPL_ENABLE_TELEMETRY
{
auto context = opentelemetry::context::RuntimeContext::GetCurrent();
auto spanValue = context.GetValue(opentelemetry::trace::kSpanKey);
- Add `#include` for OTel context headers, guarded by `#ifdef XRPL_ENABLE_TELEMETRY`
- Edit `include/xrpl/basics/Log.h`:
- No changes needed — format() signature unchanged
**Key modified files**:
- `src/libxrpl/basics/Log.cpp`
**Performance note**: The implementation checks the thread-local context value directly (avoiding the heap allocation that `GetSpan()` performs on the no-span path). On threads without an active span (~99% of log lines), the cost is a thread-local read + variant type check (~15-20ns). On the active-span path, an additional shared_ptr copy + `GetContext()` + `IsValid()` adds ~50ns total. Overhead is negligible at typical logging rates.
---
## Task 8.2: Add Loki to Docker Compose Stack
**Objective**: Add Grafana Loki as a log storage backend in the development observability stack.
**What to do**:
- Edit `docker/telemetry/docker-compose.yml`:
- Add Loki service:
```yaml
loki:
image: grafana/loki:3.4.2
ports:
- "3100:3100"
command: -config.file=/etc/loki/local-config.yaml
```
- Add Loki as a Grafana data source in provisioning
# Phase 9: Internal Metric Instrumentation Gap Fill — Task List
> **Status**: Future Enhancement
>
> **Goal**: Instrument xrpld to emit ~50+ metrics that exist in `get_counts`/`server_info`/TxQ/PerfLog but currently lack time-series export via the OTel or beast::insight pipelines.
>
> **Scope**: Hybrid approach — extend `beast::insight` for metrics near existing registrations, use OTel Metrics SDK `ObservableGauge` callbacks for new categories (TxQ, PerfLog, CountedObjects).
- The callback should read from the same sources as `GetCounts.cpp` handler (line ~43).
- Create a centralized `MetricsRegistry` class that holds all OTel async gauge registrations, polled at 10-second intervals by the `PeriodicMetricReader`.
**Objective**: Fill the remaining metric gaps between the external `push_metrics.py` script (in `ripplex-ansible`) and the internal OTel `MetricsRegistry` observable gauges. After this task, all metrics collected by `push_metrics.py` that CAN be collected internally are covered.
**What was done**:
- Extended existing `cacheHitRateGauge_` callback with `AL_size` (AcceptedLedger cache size)
- Extended existing `nodeStoreGauge_` callback with 4 new metrics from `getCountsJson()`:
- Existing integration test script (extend assertions)
---
## Task 9.11: Validator Health Dashboard (External Dashboard Parity)
> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity) — dashboards for Phase 7 metrics inspired by the community [xrpl-validator-dashboard](https://github.com/realgrapedrop/xrpl-validator-dashboard).
>
> **Upstream**: Phase 7 Tasks 7.9-7.16 (metrics must be emitting).
|Consensus|LedgerHistoryMismatch|`sum by (service_instance_id)(rate(xrpld_ledger_history_mismatch_total[5m]))`|`> 0`|5m|critical|
|Consensus|LedgerCloseStalled|`sum by (service_instance_id)(rate(xrpld_ledgers_closed_total[5m]))`|`< 0.001`(≈0)|3m|critical|
|Validator|ValidationsMissed|`sum by (service_instance_id)(rate(xrpld_validation_missed_total[5m]))`|`> 0`|5m|warning|
|Validator|ValidationsNotChecked|`sum by (service_instance_id)(rate(xrpld_validations_checked_total[5m]))`|`< 0.001`(≈0)|5m|warning|
|Jobqueue|JobQueueTxOverflow|`sum by (service_instance_id)(rate(xrpld_jq_trans_overflow_total[5m]))`|`> 0`|5m|warning|
|Jobqueue|JobQueueLatencyHigh|`histogram_quantile(0.99, sum by (le, service_instance_id)(rate(xrpld_job_queued_duration_us_bucket[5m])))`|`> 1000000`(µs =1s)|5m|warning|
# Securing OpenTelemetry Against Trace Context Spoofing
> **Part of**: [OpenTelemetry Implementation Plan](./OpenTelemetryPlan.md) — see also [Design Decisions § Privacy](./02-design-decisions.md#244-privacy--sensitive-data-policy) (what we don't collect) and [Configuration Reference § 5.5](./05-configuration-reference.md#55-opentelemetry-collector-configuration) (collector base config).
Trace context spoofing (or poisoning) occurs when untrusted actors inject tampered or stale trace IDs into your system. If these requests are processed, the spans are appended to historical trace buckets, stretching trace durations, ruining p99 latency metrics, and breaking Grafana dashboards.
This guide outlines two categories of defense: mitigating tampered contexts and locking down the OpenTelemetry (OTel) Collector to trusted clients only.
---
## Part 1: Mitigating Tampered Trace Contexts
### 1. Perimeter Defense: Strip Headers at the API Gateway
The most effective way to prevent spoofing from external sources is to treat your API Gateway (Envoy, NGINX, AWS ALB) as a hard boundary. Strip incoming W3C tracing headers (`traceparent`, `tracestate`) from public traffic so the gateway is forced to generate a fresh, legitimate `trace_id`.
**NGINX Example (Stripping Headers):**
```nginx
server{
listen80;
location/{
# Clear out untrusted incoming trace headers
proxy_set_headertraceparent"";
proxy_set_headertracestate"";
proxy_passhttp://backend_service;
}
}
```
### **2. Timestamp-Anchored Trace IDs and OTTL Filtering**
If you use a custom trace ID generator that embeds a timestamp in the first few bytes (like AWS X-Ray or UUIDv7), you can use the OTel Collector's OpenTelemetry Transform Language (OTTL) to detect anomalies.
Ensure your Collector is not exposed to the public internet. If running in Kubernetes, use a NetworkPolicy to restrict ingress traffic to specific namespaces.
**Kubernetes NetworkPolicy Example:**
```yaml
apiVersion:networking.k8s.io/v1
kind:NetworkPolicy
metadata:
name:allow-internal-otel
namespace:observability
spec:
podSelector:
matchLabels:
app:opentelemetry-collector
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
environment:production
ports:
- protocol:TCP
port:4317# gRPC
- protocol:TCP
port:4318# HTTP
```
### **Approach B: Transport Layer Security (Mutual TLS / mTLS)**
Require clients to present a valid cryptographic certificate to connect to the Collector.
**Collector Configuration (mTLS):**
```yaml
receivers:
otlp:
protocols:
grpc:
endpoint:0.0.0.0:4317
tls:
# Setting client_ca_file makes the collector require and verify a
# client cert, rejecting connections without a trusted one.
client_ca_file:/certs/client_ca.pem# CA that signs trusted client certs
Available routes to build on top of: https://github.com/XRPLF/rippled/pull/6425#discussion_r3234751995
---
# Analysis: Applying the Guide to xrpld
The guide above is written for HTTP-fronted web services. xrpld is a P2P node daemon, so the threat model and the applicable defenses differ. This section captures how each approach maps to xrpld and the chosen direction.
## Threat Model
xrpld has **two distinct attack surfaces**, not one. The original guide conflates them under "trace context spoofing"; for xrpld they need separate defenses.
| **Collector ingress** (xrpld → collector) | Anyone who can reach `4317`/`4318` on the collector host | Forged OTLP traffic, telemetry exfiltration, DoS on collector | mTLS + network policy |
| **Peer trace context** (peer → xrpld) | Malicious peer in the XRPL overlay | Crafted `protocol::TraceContext` field inside peer protobuf messages (TMTransaction, consensus, etc.) — used to forge `trace_id`/`span_id`, pollute p99, attach spans to historical traces | Validate + rate-limit at the receive boundary |
**Deployment context:** Across-network. xrpld nodes (potentially run by external operators or in different DCs) ship telemetry to a centrally-hosted collector across an untrusted network. The collector is NOT on the same host or private VPC as every node.
## Part 1 Applicability — Peer Trace-Context Validation
The guide's NGINX header stripping and OTTL stale-span filtering target HTTP gateways and post-hoc cleanup. Neither fits xrpld directly:
- **NGINX header stripping** — N/A. There is no HTTP gateway between peers and xrpld; trace context arrives inside protobuf peer messages (`protocol::TraceContext`), not as W3C `traceparent` headers. See [src/xrpld/telemetry/PropagationHelpers.h](../src/xrpld/telemetry/PropagationHelpers.h).
- **OTTL stale-span filtering** — Weak fit. Post-hoc cleanup at the collector loses peer identity (you can't tell _which_ peer poisoned the trace). Validation at the receive site is stronger.
**xrpld-specific Part 1 mitigations:**
1.**Validate extracted context at the boundary** in [src/xrpld/telemetry/ConsensusReceiveTracing.h](../src/xrpld/telemetry/ConsensusReceiveTracing.h) and any other peer-message receive site. Reject if `trace_id` is all-zero, wrong length, or fails W3C format checks. Treat invalid context as "no propagated context" — start a fresh span — rather than dropping the message.
2.**Per-peer sample rate limiting** so a hostile peer cannot flood the collector with spans bearing a fabricated `trace_id`. Use probabilistic sampling on the receive path keyed by peer identity.
## Part 2 — Comparison of Collector Hardening Approaches
Evaluated for the across-network deployment shape:
| Approach | Across-network fit | Cost | Verdict |
| **A. NetworkPolicy / firewall** | Necessary baseline (don't expose `4317`/`4318` to the internet), but insufficient on its own when traffic genuinely crosses networks — you cannot NetworkPolicy the public internet. | Cheap. | **Defense-in-depth, not primary.** |
| **B. mTLS** | Strongest fit. Every xrpld node holds a client cert; the collector verifies it via `client_ca_file` in the receiver's `tls` block. Encrypts in transit (raw OTLP over the internet leaks transaction patterns and validator identity). Compromised node = revoke one cert, no shared secret to rotate everywhere. | Cert issuance + rotation pipeline. | **Primary.** |
| **C. Basic Auth** | Worst shape for this topology. Single shared password across all xrpld nodes — one leaked node config compromises the whole fleet. Doesn't encrypt; you'd need TLS underneath anyway, at which point you're 80% of the way to mTLS. | Cheap to set up, expensive to operate (rotation across N operators). | **Skip.** |
## Decision
**Primary defense:** mTLS (Approach B) on the collector's OTLP receivers. The collector requires and verifies each client certificate when `client_ca_file` is set in the receiver's `tls` block (there is no `auth_type` field — setting `client_ca_file` is what enforces client-cert verification).
**Defense-in-depth:** NetworkPolicy / firewall rules (Approach A) so `4317`/`4318` are never reachable from outside the expected operator subnets even if mTLS were misconfigured.
**Skipped:** Basic Auth (Approach C) — wrong shape for an across-network, multi-operator topology.
**Plus xrpld-specific Part 1 work:** trace-context validation and per-peer rate limiting at peer-message receive sites.
| Cert source for mTLS | **Reuse XRPL node identity key** | One identity per node, no separate PKI to operate. Fits XRPL's existing trust model; requires small CA tooling step to derive/sign the OTel client cert from the node key. |
| Part 1 scope | **Include in this spec** | Collector hardening and peer trace-context validation share one threat model. Coherent design doc; can still be split into multiple PRs at implementation. |
| Dev impact | **Production-only** | Local `docker/telemetry/docker-compose.yml` keeps `insecure: true` and no auth for fast iteration. Only production deployment manifests gain mTLS. Accepted risk: minor dev/prod drift, mitigated by integration tests against a TLS-enabled collector in CI. |
## Out of Scope
- NGINX/Envoy header stripping (no HTTP gateway in front of xrpld-to-collector traffic).
- OTTL stale-span filtering at the collector (weaker than source validation; loses peer identity).
- Local development docker-compose hardening.
- Telemetry backend (Tempo) hardening — separate concern, downstream of the collector.
## Next Step
Write this up as a design doc with full sections covering:
1. Threat model & architecture (this section, expanded)
@@ -8,11 +8,11 @@ The [XRP Ledger](https://xrpl.org/) is a decentralized cryptographic ledger powe
[XRP](https://xrpl.org/xrp.html) is a public, counterparty-free crypto-asset native to the XRP Ledger, and is designed as a gas token for network services and to bridge different currencies. XRP is traded on the open-market and is available for anyone to access. The XRP Ledger was created in 2012 with a finite supply of 100 billion units of XRP.
## rippled
## xrpld
The server software that powers the XRP Ledger is called `rippled` and is available in this repository under the permissive [ISC open-source license](LICENSE.md). The `rippled` server software is written primarily in C++ and runs on a variety of platforms. The `rippled` server software can run in several modes depending on its [configuration](https://xrpl.org/rippled-server-modes.html).
The server software that powers the XRP Ledger is called `xrpld` and is available in this repository under the permissive [ISC open-source license](LICENSE.md). The `xrpld` server software is written primarily in C++ and runs on a variety of platforms. The `xrpld` server software can run in several modes depending on its [configuration](https://xrpl.org/rippled-server-modes.html).
If you are interested in running an **API Server** (including a **Full History Server**), take a look at [Clio](https://github.com/XRPLF/clio). (rippled Reporting Mode has been replaced by Clio.)
If you are interested in running an **API Server** (including a **Full History Server**), take a look at [Clio](https://github.com/XRPLF/clio). (xrpld Reporting Mode has been replaced by Clio.)
### Build from Source
@@ -41,19 +41,19 @@ If you are interested in running an **API Server** (including a **Full History S
Here are some good places to start learning the source code:
- Read the markdown files in the source tree: `src/ripple/**/*.md`.
- Read the markdown files in the source tree: `src/xrpld/**/*.md`.
- Read [the levelization document](.github/scripts/levelization) to get an idea of the internal dependency graph.
- In the big picture, the `main` function constructs an `ApplicationImp` object, which implements the `Application` virtual interface. Almost every component in the application takes an `Application&` parameter in its constructor, typically named `app` and stored as a member variable `app_`. This allows most components to depend on any other component.
@@ -6,7 +6,7 @@ For more details on operating an XRP Ledger server securely, please visit https:
## Supported Versions
Software constantly evolves. In order to focus resources, we only generally only accept vulnerability reports that affect recent and current versions of the software. We always accept reports for issues present in the **master**, **release** or **develop** branches, and with proposed, [open pull requests](https://github.com/ripple/rippled/pulls).
Software constantly evolves. In order to focus resources, we generally only accept vulnerability reports that affect recent and current versions of the software. We always accept reports for issues present in the **master**, **release** or **develop** branches, and with proposed, [open pull requests](https://github.com/XRPLF/rippled/pulls).
## Identifying and Reporting Vulnerabilities
@@ -22,128 +22,10 @@ Responsible investigation includes, but isn't limited to, the following:
- Not targeting physical security measures, or attempting to use social engineering, spam, distributed denial of service (DDOS) attacks, etc.
- Investigating bugs in a way that makes a reasonable, good faith effort not to be disruptive or harmful to the XRP Ledger and the broader ecosystem.
### Responsible Disclosure
If you discover a vulnerability or potential threat, or if you _think_
you have, please reach out by dropping an email using the contact
information below.
Your report should include the following:
- Your contact information (typically, an email address);
- The description of the vulnerability;
- The attack scenario (if any);
- The steps to reproduce the vulnerability;
- Any other relevant details or artifacts, including code, scripts or patches.
In your email, please describe the issue or potential threat. If possible, include a "repro" (code that can reproduce the issue) or describe the best way to reproduce and replicate the issue. Please make your report as detailed and comprehensive as possible.
For more information on responsible disclosure, please read this [Wikipedia article](https://en.wikipedia.org/wiki/Responsible_disclosure).
## Report Handling Process
Please report the bug directly to us and limit further disclosure. If you want to prove that you knew the bug as of a given time, consider using a cryptographic precommitment: hash the content of your report and publish the hash on a medium of your choice (e.g. on Twitter or as a memo in a transaction) as "proof" that you had written the text at a given point in time.
Once we receive a report, we:
1. Assign two people to independently evaluate the report;
2. Consider their recommendations;
3. If action is necessary, formulate a plan to address the issue;
4. Communicate privately with the reporter to explain our plan.
5. Prepare, test and release a version which fixes the issue; and
6. Announce the vulnerability publicly.
We will triage and respond to your disclosure within 24 hours. Beyond that, we will work to analyze the issue in more detail, formulate, develop and test a fix.
While we commit to responding with 24 hours of your initial report with our triage assessment, we cannot guarantee a response time for the remaining steps. We will communicate with you throughout this process, letting you know where we are and keeping you updated on the timeframe.
## Bug Bounty Program
[Ripple](https://ripple.com) is generously sponsoring a bug bounty program for vulnerabilities in [`rippled`](https://github.com/XRPLF/rippled) (and other related projects, like [`xrpl.js`](https://github.com/XRPLF/xrpl.js), [`xrpl-py`](https://github.com/XRPLF/xrpl-py), [`xrpl4j`](https://github.com/XRPLF/xrpl4j)).
[Ripple](https://ripple.com) is generously sponsoring a bug bounty program for vulnerabilities in [`xrpld`](https://github.com/XRPLF/rippled) (and other related projects, like [`Clio`](https://github.com/XRPLF/clio), [`xrpl.js`](https://github.com/XRPLF/xrpl.js), [`xrpl-py`](https://github.com/XRPLF/xrpl-py), [`xrpl4j`](https://github.com/XRPLF/xrpl4j)).
This program allows us to recognize and reward individuals or groups that identify and report bugs. In summary, in order to qualify for a bounty, the bug must be:
This program allows us to recognize and reward individuals or groups that identify and report bugs.
1.**In scope**. Only bugs in software under the scope of the program qualify. Currently, that means `rippled`, `xrpl.js`, `xrpl-py`, `xrpl4j`.
2.**Relevant**. A security issue, posing a danger to user funds, privacy, or the operation of the XRP Ledger.
3.**Original and previously unknown**. Bugs that are already known and discussed in public do not qualify. Previously reported bugs, even if publicly unknown, are not eligible.
4.**Specific**. We welcome general security advice or recommendations, but we cannot pay bounties for that.
5.**Fixable**. There has to be something we can do to permanently fix the problem. Note that bugs in other people’s software may still qualify in some cases. For example, if you find a bug in a library that we use which can compromise the security of software that is in scope and we can get it fixed, you may qualify for a bounty.
6.**Unused**. If you use the exploit to attack the XRP Ledger, you do not qualify for a bounty. If you report a vulnerability used in an ongoing or past attack and there is specific, concrete evidence that suggests you are the attacker we reserve the right not to pay a bounty.
The amount paid varies dramatically. Vulnerabilities that are harmless on their own, but could form part of a critical exploit will usually receive a bounty. Full-blown exploits can receive much higher bounties. Please don’t hold back partial vulnerabilities while trying to construct a full-blown exploit. We will pay a bounty to anyone who reports a complete chain of vulnerabilities even if they have reported each component of the exploit separately and those vulnerabilities have been fixed in the meantime. However, to qualify for a the full bounty, you must to have been the first to report each of the partial exploits.
### Contacting Us
To report a qualifying bug, please send a detailed report to:
We have partnered with Bugcrowd to manage this program. It is a private program, and security researchers can participate based on invitation. If you need access to the program, please email bugs@ripple.com with your Bugcrowd handle or Bugcrowd registered email, and we will get you added to the program. Once you have been added, please submit vulnerability reports through Bugcrowd, not by email. The detailed bug bounty policy is available on the Bugcrowd website.
This script will not push. Verify everything is correct, then push
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
