The collector's log ingestion used a container path named after the old
`rippled` binary and defaulted its host mount to a location that either
needed root or diverged from where the telemetry configs actually write.
Consolidate on one accessible log root:
- container mount target renamed /var/log/rippled -> /var/log/xrpld
- filelog glob -> /var/log/xrpld/*/debug.log
- compose mount source defaults to the repo-relative ./data/logs
(user-owned, no root), overridable via XRPLD_LOG_DIR
- devnet telemetry cfg writes to data/logs/devnet/debug.log so it lands
one subdir below the mount root and matches the glob
- integration-test.sh sets XRPLD_LOG_DIR to its own workdir
- docs/runbook/task-list updated to match
The default xrpld config (cfg/xrpld-example.cfg) is intentionally left
untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- PerfLogImp::rpcEnd(): return after the requestId-not-found UNREACHABLE
so a stale (now - epoch) duration is no longer recorded to the counter
and histogram in release builds.
- MetricsRegistry peer-version gauge: compare versions numerically via
BuildInfo::encodeSoftwareVersion() instead of a lexicographic string
compare, stripping the non-digit prefix so peer 'rippled-X.Y.Z' lines
up with our bare 'X.Y.Z'. Fixes every peer counting as higher-version.
- MetricsRegistry::stop(): call Shutdown() before ForceFlush() before
reset() so the reader thread stops before teardown and no gauge
callback fires during shutdown.
- MetricsRegistry::start(): extract initExporterAndProvider() and
initSyncInstruments() helpers to keep each function under the line
limit; no behavior change.
- time_in_current_state_seconds: read NetworkOPs::getServerStateDurationUs()
(a lightweight accessor over StateAccounting) and convert microseconds
to seconds, replacing the hardcoded 0.0.
- Correct log-timestamp examples to real Logs::format() output
(2024-Jan-15 10:30:45.123456 UTC) in docs, TESTING.md, and reference.
- Update Loki to v3.4.2 and switch the collector exporter to
otlphttp/loki on the native /otlp endpoint (docs + task list).
- Make the collector log-mount path configurable via XRPLD_LOG_DIR.
- Remove implementation-phase references from shipped config/script
comments, keeping the functional descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the phase-8-owned 'Phase 8' tokens from
loki.yaml and tempo.yaml, keeping the functional descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The getYays()/getNays() accessors already exist on the phase-7 branch.
A cleanup+restore pair on this branch moved them, producing a no-op
diff against phase-7 that mixed consensus churn into the log-correlation
PR. Restore DisputedTx.h to the phase-7 state so this branch carries no
unrelated consensus change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 3/4" token, keep the description.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 4" tokens, keep descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 3" tokens, keep descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 2" tokens, keep descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewer noted implementation-phase references should not appear in
shipped config comments. Drop the "Phase 1b" token, keep the filter
description.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- node-health.json: repoint the two "All Jobs" detail panels from the
__name__ regex over old xrpld_<job>_milliseconds_bucket names to the
native xrpld_job_running/queued_duration_us_bucket histograms, grouping
by the job_type label and legending on {{job_type}}.
- OTelCollector::callHooks: copy the hook list under mutex_ and invoke
handlers outside the lock. A handler can drop the last reference to an
OTelHookImpl, whose destructor calls removeHook() and re-acquires the
non-recursive mutex_ — invoking handlers under the lock could deadlock.
- ~OTelGaugeImpl: document that the SDK ObservableRegistry serializes
RemoveCallback against the Observe() callback pass with one mutex, so
callback removal is synchronous and gaugeCallback cannot run on a
dangling pointer after the destructor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Key Jobs execution/dequeue panels queried StatsD-era per-job metric
names (xrpld_<job>_milliseconds_bucket, xrpld_<job>_q_milliseconds_bucket)
that the native OTel path no longer emits. Phase 9's MetricsRegistry emits
job timings as two label-dimensioned histograms instead:
xrpld_job_running_duration_us_bucket{job_type="<job>"}
xrpld_job_queued_duration_us_bucket{job_type="<job>"}
Rewrite the 22 affected panel queries (11 job types x running/queued) to
the label-dimensioned form, preserving the histogram_quantile / rate /
sum-by structure and all template-variable filters.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- MetricsRegistry: service.name was recorded as boolean true because the
string literal "xrpld" bound to the OTel AttributeValue variant's
char-const* -> bool overload. Assign std::string so it selects the
string alternative; Prometheus now shows service_name=xrpld (and
exported_job round-trips correctly) on every MetricsRegistry series.
- Grafana alerting contact points: the Slack url / email addresses used
${ENV_VAR} references that expand to empty when unset, and Grafana's
provisioning validator (which lacks ${VAR:-default} support) then
crashes the whole stack on startup. Use non-empty disabled placeholders
(unroutable webhook host, .invalid email) so the stack boots with zero
configuration; delivery stays off until a real destination is supplied.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- ValidationTracker: remove dead hard-trim Pass 1. The preceding
eviction loop already erases every reconciled entry older than the
late-repair cutoff using the same condition, so Pass 1 could never
match. The "drop any reconciled entry" fallback is retained.
- integration-test.sh: probe the StatsD port with a UDP-aware check
(ss -ulnp) instead of curl. StatsD is UDP-only on 8125, so the TCP
curl probe always reported "refused" — a false negative. Guarded by
a command -v ss check that logs a skip when ss is unavailable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Give developers direct access to the full OpenTelemetry metrics API,
symmetric with getTracer(), so all seven OTel instrument types
(including UpDownCounter, sync Gauge, and the observable variants) are
reachable — not just the four beast::insight models.
- Telemetry: build and own the metrics pipeline (OTLP HTTP metric
exporter + PeriodicExportingMetricReader + SpanMetrics histogram view)
alongside the tracer, sharing the same resource attributes and TLS
config. Register it globally via metrics::Provider::SetMeterProvider
and expose Telemetry::getMeter(). Metrics enable with [telemetry];
the metrics endpoint is derived from the trace endpoint.
- beast OTelCollector: no longer owns a pipeline. It fetches the global
Meter, becoming a thin shim over the shared provider (legacy path
during beast deprecation). This also resolves the review note that the
metric exporter ignored [telemetry] use_tls — TLS now comes from the
shared telemetry pipeline.
- Add a libxrpl unit test covering getMeter() on the enabled (global
provider) and disabled (noop) paths, exercising an UpDownCounter.
Design: docs/superpowers/specs/2026-07-07-direct-otel-metrics-api-design.md
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- cmake: keep the opentelemetry-cpp umbrella target for the beast metrics
link and document why. The reviewer suggested linking individual
component targets to avoid over-linking, but the OTel Conan package
under-declares inter-component dependencies (the OTLP client references
sdk::common symbols without a declared edge), so naming components
directly reorders the static link into an unresolvable state. Verified
by building xrpl_tests both ways.
- OTelCollector.h: add usage examples, thread-safety and limitations
@note blocks to the class doc.
- OTelCollector.cpp: correct the @param name docs on the instrument
Impl constructors to describe the already-formatName()'d value.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Phase 11 task list still linked the moved External Dashboard Parity spec
at its old docs/superpowers/ path. Point them at the appendix in
06-implementation-phases.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 09-data-collection-reference and Phase9 task list still linked the moved
External Dashboard Parity spec at its old docs/superpowers/ path. Point them
at the appendix in 06-implementation-phases.md where the content now lives.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the phase-9 alerting design spec out of docs/superpowers/ (which the
project guidelines say not to create) into an appendix of
OpenTelemetryPlan/Phase9_taskList.md, keeping the phase plan self-contained.
Drops the stale one-off branch/PR-restructure section from the spec.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the standalone External Dashboard Parity design spec out of
docs/superpowers/ (which the project guidelines say not to create) and into
an appendix of OpenTelemetryPlan/06-implementation-phases.md, so the phase
plan is self-contained. Repoint the 10 "Source" links in the phase 3/4/7
task lists to the new appendix anchor.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Update the phase-9 task list and the alerting design spec to reference the
Alerting section of docs/telemetry-runbook.md, now that the standalone
ALERTING.md has been folded into the runbook.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the placeholder webhook contact point with two env-driven tiers:
- xrpld-default (Slack) receives warning-severity alerts
- xrpld-critical (Slack + email) receives critical-severity alerts, via a
severity=critical child route in the notification policy
Secrets stay out of git: contactpoints.yaml references ${SLACK_WEBHOOK_URL},
${SLACK_CHANNEL}, and ${ALERT_EMAIL_TO}, which Grafana expands from a
gitignored .env.alerting (loaded via an optional env_file on the grafana
service). .env.alerting.example documents the variables; email additionally
needs GF_SMTP_* configured.
Fold the ALERTING.md runbook content into the Alerting section of
docs/telemetry-runbook.md (operator docs live in one place) and delete
ALERTING.md. Move the .env.* ignore rule into docker/telemetry/.gitignore.
Verified live: Grafana 11.5.2 provisioned both contact points, the
severity-routed policy, and all 6 rules with no errors.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
