Document the three perf-iac filter template variables (xrpl_work_item,
xrpl_branch, xrpl_node_role) in the telemetry runbook and the data-collection
reference: what they filter, their example values, and that perf-iac stamps
them as resource attributes from its own alloy pipeline (absent outside
perf comparison runs, so the filters default to All). Also record perf as a
network value.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dashboard audit against the live datasource surfaced two broken panel queries
and applied UX polish across all 14 dashboards.
- node-health "Job Queue Depth": job_count -> jobq_job_count. The JobQueue
collector is wrapped in group("jobq") (Application.cpp), so the registered
job_count gauge is emitted with the jobq_ prefix; the panel queried the
unprefixed name and returned nothing.
- network-traffic "Overlay Traffic by Category" + "All Traffic Categories":
topk(N, rate({__name__=~".*_bytes_in"}[...])) errors on Mimir ("vector
cannot contain metrics with the same labelset") because rate() drops
__name__ and the many counters collapse. Replaced with an enumerated
label_replace form that re-attaches __name__ per metric, preserving the
{{__name__}} legend and per-series display-name overrides.
- All 14 dashboards: refresh set to 10s.
- peer-quality: each panel full screen width.
- validator-health: at most two panels per row (row headers preserved).
- docs: telemetry-runbook and 06-implementation-phases updated for the
jobq_ prefix and the network-traffic query pattern.
Verified end-to-end against a local mainnet xrpld node feeding the local
stack: jobq_job_count returns data (old job_count empty), both network-traffic
exprs execute (old form reproduces the labelset error), and panels render
through the Grafana proxy. All 14 pass validate_dashboards.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the TxQ Accept Status piechart on the Transaction Overview
dashboard with a state-timeline showing each node's applied fraction of
TxQ accepts (applied / applied+failed) over time, colored by threshold
(green >=0.9, yellow >=0.7, red below). Remove the now-orphaned
txq_status template variable (the piechart was its only consumer) and
document the panel in the telemetry runbook.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The spanmetrics connector had no namespace, so it emitted traces_span_metrics_*
metric names by default. The span dashboards and docs are renamed to query
span_* names; this is only correct if the connector emits them too, so add
namespace: "span" to the spanmetrics connector. Both sides change together:
renaming the dashboards without the namespace (or vice versa) would break the
pipeline. Matches the phase9 collector config.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The phase7 OTelCollector::formatName lowercases and strips names, emitting
snake_case metrics with no xrpld_ prefix. The native Grafana dashboards and
the telemetry docs still queried the old xrpld_CamelCase names, so they were
broken against their own pipeline. Rename every metric name to match what the
code emits: drop the xrpld_ prefix and lowercase the remainder. The two job
histograms also drop the redundant 'duration' word (job_queued_us,
job_running_us) to match the phase9 forms. Add havetxset to the cspell
dictionary since the lowercased metric name no longer word-splits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add service_name to transform/cloudlabels so Grafana Cloud dashboards
can filter by service.name (fixes empty panels for native metrics).
Set spanmetrics namespace to "span" in the cloud collector config and
update dashboards + runbook: traces_span_metrics_ → span_.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace unused observer_result.h with nostd/shared_ptr.h and
nostd/unique_ptr.h in MetricsRegistry.h (misc-include-cleaner)
- Use auto for cast initializations in MetricsRegistry.cpp and
ValidationTracker test (modernize-use-auto)
- Run rename scripts: rippled -> xrpld in comments and docs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update the transform/cloudlabels processor to set service_instance_id
instead of exported_instance, matching the local Prometheus path where
resource_to_telemetry_conversion promotes service.instance.id with that
exact label name. Also update source comments and documentation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The collector's log ingestion used a container path named after the old
`rippled` binary and defaulted its host mount to a location that either
needed root or diverged from where the telemetry configs actually write.
Consolidate on one accessible log root:
- container mount target renamed /var/log/rippled -> /var/log/xrpld
- filelog glob -> /var/log/xrpld/*/debug.log
- compose mount source defaults to the repo-relative ./data/logs
(user-owned, no root), overridable via XRPLD_LOG_DIR
- devnet telemetry cfg writes to data/logs/devnet/debug.log so it lands
one subdir below the mount root and matches the glob
- integration-test.sh sets XRPLD_LOG_DIR to its own workdir
- docs/runbook/task-list updated to match
The default xrpld config (cfg/xrpld-example.cfg) is intentionally left
untouched.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Correct log-timestamp examples to real Logs::format() output
(2024-Jan-15 10:30:45.123456 UTC) in docs, TESTING.md, and reference.
- Update Loki to v3.4.2 and switch the collector exporter to
otlphttp/loki on the native /otlp endpoint (docs + task list).
- Make the collector log-mount path configurable via XRPLD_LOG_DIR.
- Remove implementation-phase references from shipped config/script
comments, keeping the functional descriptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the phase-9 alerting design spec out of docs/superpowers/ (which the
project guidelines say not to create) into an appendix of
OpenTelemetryPlan/Phase9_taskList.md, keeping the phase plan self-contained.
Drops the stale one-off branch/PR-restructure section from the spec.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the standalone External Dashboard Parity design spec out of
docs/superpowers/ (which the project guidelines say not to create) and into
an appendix of OpenTelemetryPlan/06-implementation-phases.md, so the phase
plan is self-contained. Repoint the 10 "Source" links in the phase 3/4/7
task lists to the new appendix anchor.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Update the phase-9 task list and the alerting design spec to reference the
Alerting section of docs/telemetry-runbook.md, now that the standalone
ALERTING.md has been folded into the runbook.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the placeholder webhook contact point with two env-driven tiers:
- xrpld-default (Slack) receives warning-severity alerts
- xrpld-critical (Slack + email) receives critical-severity alerts, via a
severity=critical child route in the notification policy
Secrets stay out of git: contactpoints.yaml references ${SLACK_WEBHOOK_URL},
${SLACK_CHANNEL}, and ${ALERT_EMAIL_TO}, which Grafana expands from a
gitignored .env.alerting (loaded via an optional env_file on the grafana
service). .env.alerting.example documents the variables; email additionally
needs GF_SMTP_* configured.
Fold the ALERTING.md runbook content into the Alerting section of
docs/telemetry-runbook.md (operator docs live in one place) and delete
ALERTING.md. Move the .env.* ignore rule into docker/telemetry/.gitignore.
Verified live: Grafana 11.5.2 provisioned both contact points, the
severity-routed policy, and all 6 rules with no errors.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add PathFinding Spans section to the runbook Span Reference and
remove the incorrect "not emitted / later-phase" annotations:
pathfind.request/compute/discover/update_all are emitted since
Phase 2 (upstream of Phase 5).
- Runbook label names use the bare spanmetrics dimensions
(command, rpc_status, local, consensus_mode); fix stale docs.
- 05-configuration-reference: note tls_client_cert/key require use_tls=1.
- TelemetryConfig: reject mTLS cert paths when use_tls=0 so telemetry
is not silently sent in plaintext.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow the config change that replaced the personal home directory: the
runbook prose now points at /var/log/xrpld/mainnet/debug.log, matching the
mainnet config and the collector's filelog receiver.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add an operator-facing Alerting section to the runbook covering the six
provisioned Grafana rules and how to point them at a real receiver.
- Strip the "(Phase 9)" suffix from the MetricsRegistry Gauges heading.
- ALERTING.md: drop the internal dev-phase reference from the intro.
- Phase9_taskList.md: expand Task 9.9 to describe the provisioned alerting
(rules, contact point, policy, ALERTING.md runbook).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
