rippled/OpenTelemetryPlan/Phase7_taskList.md

Phase 7: Native OTel Metrics Migration — Task List

Goal: Replace StatsDCollector with a native OpenTelemetry Metrics SDK implementation behind the existing beast::insight::Collector interface, eliminating the StatsD UDP dependency.

Scope: New OTelCollectorImpl class, CollectorManager config change, OTel Collector pipeline update, Grafana dashboard metric name migration, integration tests.

Branch: pratik/otel-phase7-native-metrics (from pratik/otel-phase6-statsd)

Related Plan Documents

Document	Relevance
06-implementation-phases.md	Phase 7 plan: motivation, architecture, exit criteria (§6.8)
02-design-decisions.md	Collector interface design, beast::insight coexistence strategy
05-configuration-reference.md	[insight] and [telemetry] config sections
09-data-collection-reference.md	Complete metric inventory that must be preserved

---

Task 7.1: Add OTel Metrics SDK to Build Dependencies

Objective: Enable the OTel C++ Metrics SDK components in the build system.

What to do:

• Edit conanfile.py:

  • Add OTel metrics SDK components to the dependency list when telemetry=True
  • Components needed: opentelemetry-cpp::metrics, opentelemetry-cpp::otlp_http_metric_exporter

• Edit CMakeLists.txt (telemetry section):

  • Link opentelemetry::metrics and opentelemetry::otlp_http_metric_exporter targets

Key modified files:

• conanfile.py
• CMakeLists.txt (or the relevant telemetry cmake target)

Reference: 05-configuration-reference.md §5.3 — CMake integration

---

Task 7.2: Implement OTelCollector Class

Objective: Create the core OTelCollector implementation that maps beast::insight instruments to OTel Metrics SDK instruments.

What to do:

• Create include/xrpl/beast/insight/OTelCollector.h:

  • Public factory: static std::shared_ptr<OTelCollector> New(std::string const& endpoint, std::string const& prefix, beast::Journal journal)
  • Derives from StatsDCollector (or directly from Collector — TBD based on shared code)

• Create src/libxrpl/beast/insight/OTelCollector.cpp (~400-500 lines):

  • OTelCounterImpl: Wraps opentelemetry::metrics::Counter<int64_t>. increment(amount) calls counter->Add(amount).
  • OTelGaugeImpl: Uses opentelemetry::metrics::ObservableGauge<uint64_t> with an async callback. set(value) stores value atomically; callback reads it during collection.
  • OTelMeterImpl: Wraps opentelemetry::metrics::Counter<uint64_t>. increment(amount) calls counter->Add(amount). Semantically identical to Counter but unsigned.
  • OTelEventImpl: Wraps opentelemetry::metrics::Histogram<double>. notify(duration) calls histogram->Record(duration.count()). Uses explicit bucket boundaries matching SpanMetrics: [1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000] ms.
  • OTelHookImpl: Stores handler function. Called during periodic metric collection (same 1s pattern via PeriodicMetricReader).
  • OTelCollectorImp: Main class.
    • Creates MeterProvider with PeriodicMetricReader (1s export interval)
    • Creates OtlpHttpMetricExporter pointing to [telemetry] endpoint
    • Sets resource attributes (service.name, service.instance.id) matching trace exporter
    • Implements all make_*() factory methods
    • Prefixes metric names with [insight] prefix= value

• Guard all OTel SDK includes with #ifdef XRPL_ENABLE_TELEMETRY to compile to NullCollector equivalents when telemetry disabled.

Key new files:

• include/xrpl/beast/insight/OTelCollector.h
• src/libxrpl/beast/insight/OTelCollector.cpp

Key patterns to follow:

• Match StatsDCollector.cpp structure: private impl classes, intrusive list for metrics, strand-based thread safety
• Match existing telemetry code style from src/libxrpl/telemetry/Telemetry.cpp
• Use RAII for MeterProvider lifecycle (shutdown on destructor)

Reference: 04-code-samples.md — code style and patterns

---

Task 7.3: Update CollectorManager

Objective: Add server=otel config option to route metric creation to the new OTel backend.

What to do:

• Edit src/xrpld/app/main/CollectorManager.cpp:

  • In the constructor, add a third branch after server == "statsd":

    else if (server == "otel")
    {
        // Read endpoint from [telemetry] section
        auto const endpoint = get(telemetryParams, "endpoint",
            "http://localhost:4318/v1/metrics");
        std::string const& prefix(get(params, "prefix"));
        collector_ = beast::insight::OTelCollector::New(
            endpoint, prefix, journal);
    }
    

  • This requires access to the [telemetry] config section — may need to pass it as a parameter or read from Application config.

• Edit src/xrpld/app/main/CollectorManager.h:

  • Add #include <xrpl/beast/insight/OTelCollector.h>

Key modified files:

• src/xrpld/app/main/CollectorManager.cpp
• src/xrpld/app/main/CollectorManager.h

---

Task 7.4: Update OTel Collector Configuration

Objective: Add a metrics pipeline to the OTLP receiver and remove the StatsD receiver dependency.

What to do:

• Edit docker/telemetry/otel-collector-config.yaml:

  • Remove statsd receiver (no longer needed when server=otel)
  • Add metrics pipeline under service.pipelines:

    metrics:
      receivers: [otlp, spanmetrics]
      processors: [batch]
      exporters: [prometheus]
    

  • The OTLP receiver already listens on :4318 — it just needs to be added to the metrics pipeline receivers.
  • Keep spanmetrics connector in the metrics pipeline so span-derived RED metrics continue working.

• Edit docker/telemetry/docker-compose.yml:

  • Remove UDP :8125 port mapping from otel-collector service
  • Update xrpld service config: change [insight] server=statsd to server=otel

Key modified files:

• docker/telemetry/otel-collector-config.yaml
• docker/telemetry/docker-compose.yml

Note: Keep a commented-out statsd receiver block for operators who need backward compatibility.

---

Task 7.5: Preserve Metric Names in Prometheus

Objective: Ensure existing Grafana dashboards continue working with identical metric names.

What to do:

• In OTelCollector.cpp, construct OTel instrument names to match existing Prometheus metric names:

  • beast::insight make_gauge("LedgerMaster", "Validated_Ledger_Age") → OTel instrument name: xrpld_LedgerMaster_Validated_Ledger_Age
  • The prefix + group + name concatenation must produce the same string as StatsDCollector's format
  • Use underscores as separators (matching StatsD convention)

• Verify in integration test that key Prometheus queries still return data:

  • xrpld_LedgerMaster_Validated_Ledger_Age
  • xrpld_Peer_Finder_Active_Inbound_Peers
  • xrpld_rpc_requests

Key consideration: OTel Prometheus exporter may normalize metric names differently than StatsD receiver. Test this early (Task 7.2) and adjust naming strategy if needed. The OTel SDK's Prometheus exporter adds _total suffix to counters and converts dots to underscores — match existing conventions.

---

Task 7.6: Update Grafana Dashboards

Objective: Update the 3 StatsD dashboards if any metric names change due to OTLP export format differences.

What to do:

• If Task 7.5 confirms metric names are preserved exactly, no dashboard changes needed.
• If OTLP export produces different names (e.g., _total suffix on counters), update:
  • docker/telemetry/grafana/dashboards/statsd-node-health.json
  • docker/telemetry/grafana/dashboards/statsd-network-traffic.json
  • docker/telemetry/grafana/dashboards/statsd-rpc-pathfinding.json
• Rename dashboard titles from "StatsD" to "System Metrics" or similar (since they're no longer StatsD-sourced).

Key modified files:

• docker/telemetry/grafana/dashboards/statsd-*.json (3 files, conditionally)

---

Task 7.7: Update Integration Tests

Objective: Verify the full OTLP metrics pipeline end-to-end.

What to do:

• Edit docker/telemetry/integration-test.sh:
  • Update test config to use [insight] server=otel
  • Verify metrics arrive in Prometheus via OTLP (not StatsD)
  • Add check that StatsD receiver is no longer required
  • Preserve all existing metric presence checks

Key modified files:

• docker/telemetry/integration-test.sh

---

Task 7.8: Update Documentation

Objective: Update all plan docs, runbook, and reference docs to reflect the migration.

What to do:

• Edit docs/telemetry-runbook.md:

  • Update [insight] config examples to show server=otel
  • Update troubleshooting section (no more StatsD UDP debugging)

• Edit OpenTelemetryPlan/09-data-collection-reference.md:

  • Update Data Flow Overview diagram (remove StatsD receiver)
  • Update Section 2 header from "StatsD Metrics" to "System Metrics (OTel native)"
  • Update config examples

• Edit OpenTelemetryPlan/05-configuration-reference.md:

  • Add server=otel option to [insight] section docs

• Edit docker/telemetry/TESTING.md:

  • Update setup instructions to use server=otel

Key modified files:

• docs/telemetry-runbook.md
• OpenTelemetryPlan/09-data-collection-reference.md
• OpenTelemetryPlan/05-configuration-reference.md
• docker/telemetry/TESTING.md

---

Task 7.9: ValidationTracker — Validation Agreement Computation

Source: External Dashboard Parity — the most valuable metric from the community xrpl-validator-dashboard.

Upstream: Phase 4 Task 4.8 (validation span attributes provide ledger hash context). Downstream: Phase 9 (Validator Health dashboard), Phase 10 (validation checks), Phase 11 (agreement alert rules).

Objective: Implement a stateful class that tracks whether our validator's validations agree with network consensus, maintaining rolling 1h and 24h windows with an 8-second grace period and 5-minute late repair window.

Architecture:

consensus.validation.send ────> ValidationTracker ────> MetricsRegistry
(records our validation         (reconciles after       (exports agreement
 for ledger X)                   8s grace period)        gauges every 10s)

ledger.validate ──────────────> ValidationTracker
(records which ledger           (marks ledger X as
 network validated)              agreed or missed)

What to do:

• Create src/xrpld/telemetry/ValidationTracker.h:

  • recordOurValidation(ledgerHash, ledgerSeq) — called when we send a validation
  • recordNetworkValidation(ledgerHash, seq) — called when a ledger is fully validated
  • reconcile() — called periodically; reconciles pending ledger events after 8s grace period
  • Getters: agreementPct1h(), agreementPct24h(), agreements1h(), missed1h(), agreements24h(), missed24h(), totalAgreements(), totalMissed(), totalValidationsSent(), totalValidationsChecked()
  • Thread-safety: atomics for counters, mutex for window deques

• Create src/xrpld/telemetry/detail/ValidationTracker.cpp:

  • Reconciliation logic: after 8s grace period, check if weValidated && networkValidated && sameHash → agreement; else missed
  • Late repair: if a late validation arrives within 5 minutes, correct a false-positive miss
  • Sliding window: std::deque<WindowEvent> evicts entries older than 1h/24h on each reconciliation pass
  • Ring buffer of 1000 LedgerEvent structs for pending reconciliation

• Add recording hooks (modifying Phase 4 code from Phase 7 branch):

  • RCLConsensus.cpp validate(): call tracker.recordOurValidation()
  • LedgerMaster.cpp fully-validated path: call tracker.recordNetworkValidation()

Key data structures:

struct LedgerEvent {
    uint256 ledgerHash;
    LedgerIndex seq;
    TimePoint closeTime;
    bool weValidated = false;
    bool networkValidated = false;
    bool reconciled = false;
    bool agreed = false;
};

struct WindowEvent {
    TimePoint time;
    bool agreed;
};

Key new files:

• src/xrpld/telemetry/ValidationTracker.h
• src/xrpld/telemetry/detail/ValidationTracker.cpp

Key modified files:

• src/xrpld/telemetry/MetricsRegistry.h (add ValidationTracker member)
• src/xrpld/telemetry/MetricsRegistry.cpp (add gauge callback reading from tracker)
• src/xrpld/app/consensus/RCLConsensus.cpp (add recording hooks)
• src/xrpld/app/ledger/detail/LedgerMaster.cpp (add recording hook)

Exit Criteria:

• ValidationTracker correctly tracks agreement with 8s grace period
• 5-minute late repair corrects false-positive misses
• Thread-safe (atomics + mutex for window deques)
• Rolling windows correctly evict stale entries
• Unit tests: normal agreement, missed validation, late repair, window eviction

---

Task 7.10: Validator Health Observable Gauges

Source: External Dashboard Parity

Objective: Export amendment blocked, UNL health, and quorum data as a native OTel observable gauge.

What to do:

• In MetricsRegistry.cpp registerAsyncGauges(), add:

validatorHealthGauge_ = meter_->CreateDoubleObservableGauge(
    "xrpld_validator_health", "Validator health indicators");

Gauge label values:

Label metric=	Type	Source
amendment_blocked	int64	app_.getOPs().isAmendmentBlocked() → 0/1
unl_blocked	int64	app_.getOPs().isUNLBlocked() → 0/1
unl_expiry_days	double	app_.validators().expires() → days until expiry
validation_quorum	int64	app_.validators().quorum()

Sub-task 7.10a: Per-Validator Validation Count (Flag Ledger Window)

Objective: Track how many ledgers each UNL validator has validated over the last 256 consecutive ledgers (one flag ledger window). This is the key UNL participation metric — validators consistently below threshold may be candidates for removal from the UNL.

What to do:

• Add a new observable gauge:

validatorParticipationGauge_ = meter_->CreateInt64ObservableGauge(
    "xrpld_validator_participation",
    "Per-validator validation count over the last 256 ledgers");

• The callback queries app_.getValidations() to get the trusted validation set for each of the last 256 ledger hashes (from LedgerMaster::getValidatedLedger() walking backwards). For each validator public key in the UNL, count how many of those 256 ledgers have a matching validation.

• Label dimensions:

  • validator — base58-encoded validator master public key
  • service_instance_id — this node's identity (standard)

• Emission: every flag ledger (256 ledgers, ~15 minutes) or on a 10-second async gauge callback with cached results (recompute only at flag ledger boundaries).

• Data source: RCLValidations::getTrustedForLedger(hash, seq) returns std::vector<std::shared_ptr<STValidation>> with getSignerPublic() for each. The UNL list is from app_.getValidators().getTrustedMasterKeys().

• Dashboard panel: Add a table panel to the Validator Health dashboard showing xrpld_validator_participation grouped by validator label, with a threshold color (green >= 240, yellow >= 200, red < 200).

Key modified files: src/xrpld/telemetry/MetricsRegistry.h/.cpp

Exit Criteria:

• Gauge emits one time series per UNL validator
• Values range 0-256 and update at flag ledger boundaries
• Grafana table panel shows per-validator participation
• Validators below 75% participation are highlighted in red

---

Key modified files: src/xrpld/telemetry/MetricsRegistry.h/.cpp

Exit Criteria:

• All 4 base label values emitted every 10s
• unl_expiry_days is negative when expired, positive when active
• Per-validator participation gauge emits at flag ledger boundaries
• Values visible in Prometheus

---

Task 7.11: Peer Quality Observable Gauges

Source: External Dashboard Parity

Objective: Export peer health aggregates (latency P90, insane peers, version awareness) as a native OTel observable gauge.

What to do:

• In MetricsRegistry.cpp registerAsyncGauges(), add a callback that iterates app_.overlay().foreach(...) to:
  • Collect per-peer latency values, sort, compute P90
  • Count peers with tracking_ == diverged (insane)
  • Compare peer getVersion() to own version for upgrade awareness

Gauge label values:

Label metric=	Type	Source
peer_latency_p90_ms	double	P90 from sorted peer latencies
peers_insane_count	int64	Peers with diverged tracking status
peers_higher_version_pct	double	% of peers on newer xrpld version
upgrade_recommended	int64	1 if peers_higher_version_pct > 60%

Implementation note: The callback runs every 10s on the metrics reader thread. Iterating ~50-200 peers is acceptable overhead.

Key modified files: src/xrpld/telemetry/MetricsRegistry.h/.cpp

Exit Criteria:

• P90 latency computed correctly
• Insane count matches peers RPC output
• Version comparison handles format variations (e.g., "xrpld-2.4.0-rc1")

---

Task 7.12: Ledger Economy Observable Gauges

Source: External Dashboard Parity

Objective: Export fee, reserve, ledger age, and transaction rate as a native OTel observable gauge.

Gauge label values:

Label metric=	Type	Source
base_fee_xrp	double	Base fee from validated ledger fee settings (drops)
reserve_base_xrp	double	Account reserve from validated ledger (drops)
reserve_inc_xrp	double	Owner reserve increment (drops)
ledger_age_seconds	double	now - lastValidatedCloseTime
transaction_rate	double	Derived: tx count delta / time delta (smoothed)

Key modified files: src/xrpld/telemetry/MetricsRegistry.h/.cpp

Exit Criteria:

• Fee values match server_info RPC output
• ledger_age_seconds increases monotonically between ledger closes
• transaction_rate is smoothed (rolling average)

---

Task 7.13: State Tracking Observable Gauges

Source: External Dashboard Parity

Objective: Export extended state value (0-6 encoding combining OperatingMode + ConsensusMode) and time-in-current-state.

Gauge label values:

Label metric=	Type	Source
state_value	int64	0-6 encoding (see spec for mapping)
time_in_current_state_seconds	double	now - lastModeChangeTime from StateAccounting

State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full, 5=validating (full + validating), 6=proposing (full + proposing).

Key modified files: src/xrpld/telemetry/MetricsRegistry.h/.cpp

Exit Criteria:

• state_value correctly combines OperatingMode and ConsensusMode
• time_in_current_state_seconds resets on mode change

---

Task 7.14: Storage Detail and Sync Info Gauges

Source: External Dashboard Parity

Objective: Export NuDB-specific stored-bytes total and initial sync duration.

Gauge label values:

Gauge Name	Label metric=	Type	Source
xrpld_storage_detail	nudb_bytes	int64	Database::getStoreSize() — cumulative object bytes
xrpld_sync_info	initial_sync_duration_seconds	double	Time from start to first FULL

Despite the name, nudb_bytes is not a file size. getStoreSize() sums the object payloads this process has written, so it excludes NuDB's keys, bucket padding and log, and it resets when the process restarts while the files on disk do not. It is the same accessor node_written_bytes uses, so the two series are equal by construction and any write-amplification ratio built from the pair is a constant 1.0. Neither Backend nor Database exposes a file-size accessor, so no metric reports the store's on-disk size today.

Key modified files: src/xrpld/telemetry/MetricsRegistry.h/.cpp

Exit Criteria:

• Cumulative stored object bytes reported (0 if NuDB not configured)
• Sync duration captured once and remains stable after reaching FULL

---

Task 7.15: New Synchronous Counters

Source: External Dashboard Parity

Objective: Add 7 new event counters incremented at their respective instrumentation sites.

Counter Name	Increment Site	Source File
xrpld_ledgers_closed_total	onAccept() in consensus	RCLConsensus.cpp
xrpld_validations_sent_total	validate() in consensus	RCLConsensus.cpp
xrpld_validations_checked_total	Network validation received	LedgerMaster.cpp
xrpld_validation_agreements_total	ValidationTracker reconciliation	ValidationTracker.cpp
xrpld_validation_missed_total	ValidationTracker reconciliation	ValidationTracker.cpp
xrpld_state_changes_total	setMode() in NetworkOPs	NetworkOPs.cpp
xrpld_jq_trans_overflow_total	Job queue overflow path	JobQueue.cpp

Key modified files: src/xrpld/telemetry/MetricsRegistry.h/.cpp (declarations), plus recording sites in RCLConsensus.cpp, LedgerMaster.cpp, NetworkOPs.cpp, JobQueue.cpp

Exit Criteria:

• All 7 counters monotonically increase during normal operation
• Counter values match expected rates (e.g., ledgers_closed ≈ 1 per 3-5s)

---

Task 7.16: Validation Agreement Observable Gauge

Source: External Dashboard Parity

Objective: Export rolling window agreement stats from ValidationTracker (Task 7.9).

Gauge label values:

Gauge Name	Label metric=	Type	Source
xrpld_validation_agreement	agreement_pct_1h	double	tracker.agreementPct1h()
	agreements_1h	int64	tracker.agreements1h()
	missed_1h	int64	tracker.missed1h()
	agreement_pct_24h	double	tracker.agreementPct24h()
	agreements_24h	int64	tracker.agreements24h()
	missed_24h	int64	tracker.missed24h()

Key modified files: src/xrpld/telemetry/MetricsRegistry.cpp

Exit Criteria:

• Agreement percentages in range [0.0, 100.0]
• Window stats stabilize after 1h/24h of operation

---

Summary Table

Task	Description	New Files	Modified Files	Depends On
7.1	Add OTel Metrics SDK to build deps	0	2	—
7.2	Implement OTelCollector class	2	0	7.1
7.3	Update CollectorManager config routing	0	2	7.2
7.4	Update OTel Collector YAML and Docker	0	2	7.3
7.5	Preserve metric names in Prometheus	0	1	7.2
7.6	Update Grafana dashboards (if needed)	0	3	7.5
7.7	Update integration tests	0	1	7.4
7.8	Update documentation	0	4	7.6
7.9	ValidationTracker (agreement tracking)	2	4	7.2, P4.8
7.10	Validator health observable gauges	0	2	7.2
7.11	Peer quality observable gauges	0	2	7.2
7.12	Ledger economy observable gauges	0	2	7.2
7.13	State tracking observable gauges	0	2	7.2
7.14	Storage detail and sync info gauges	0	2	7.2
7.15	New synchronous counters	0	6	7.2
7.16	Validation agreement observable gauge	0	1	7.9

Parallel work: Tasks 7.4 and 7.5 can run in parallel after 7.2/7.3 complete. Task 7.6 depends on 7.5's findings. Tasks 7.7 and 7.8 can run in parallel after 7.6. Tasks 7.10-7.14 can all run in parallel after 7.2. Task 7.15 depends on 7.2. Task 7.16 depends on 7.9. Task 7.9 depends on 7.2 and Phase 4 Task 4.8.

Exit Criteria (from 06-implementation-phases.md §6.8):

• All 255+ metrics visible in Prometheus via OTLP pipeline (no StatsD receiver)
• server=otel is the default in development docker-compose
• server=statsd still works as a fallback
• Existing Grafana dashboards display data correctly
• Integration test passes with OTLP-only metrics pipeline
• No performance regression vs StatsD baseline (< 1% CPU overhead)
• Deferred Task 6.1 (|m wire format) no longer relevant — Meter mapped to OTel Counter
• ValidationTracker agreement % stabilizes after 1h under normal consensus
• All new gauges and counters visible in Prometheus with non-zero values