diff --git a/.github/scripts/levelization/results/loops.txt b/.github/scripts/levelization/results/loops.txt index f4666bf6c3..6cc0a78db5 100644 --- a/.github/scripts/levelization/results/loops.txt +++ b/.github/scripts/levelization/results/loops.txt @@ -8,11 +8,11 @@ Loop: xrpld.app xrpld.shamap xrpld.shamap > xrpld.app Loop: xrpld.app xrpld.telemetry - xrpld.app > xrpld.telemetry + xrpld.telemetry ~= xrpld.app Loop: xrpld.overlay xrpld.rpc xrpld.rpc ~= xrpld.overlay Loop: xrpld.overlay xrpld.telemetry - xrpld.overlay > xrpld.telemetry + xrpld.telemetry ~= xrpld.overlay diff --git a/.github/scripts/levelization/results/ordering.txt b/.github/scripts/levelization/results/ordering.txt index 200bc1d082..c5f4658b2a 100644 --- a/.github/scripts/levelization/results/ordering.txt +++ b/.github/scripts/levelization/results/ordering.txt @@ -195,7 +195,6 @@ tests.libxrpl > xrpl.core tests.libxrpl > xrpld.app tests.libxrpl > xrpld.overlay tests.libxrpl > xrpld.rpc -tests.libxrpl > xrpld.telemetry tests.libxrpl > xrpl.json tests.libxrpl > xrpl.ledger tests.libxrpl > xrpl.net @@ -254,6 +253,8 @@ xrpl.shamap > xrpl.nodestore xrpl.shamap > xrpl.protocol xrpl.telemetry > xrpl.basics xrpl.telemetry > xrpl.config +xrpl.telemetry > xrpl.core +xrpl.telemetry > xrpl.protocol xrpl.tx > xrpl.basics xrpl.tx > xrpl.core xrpl.tx > xrpl.ledger @@ -308,15 +309,14 @@ xrpld.perflog > xrpl.config xrpld.perflog > xrpl.core xrpld.perflog > xrpld.app xrpld.perflog > xrpld.rpc -xrpld.perflog > xrpld.telemetry xrpld.perflog > xrpl.json xrpld.perflog > xrpl.nodestore xrpld.perflog > xrpl.protocol +xrpld.perflog > xrpl.telemetry xrpld.rpc > xrpl.basics xrpld.rpc > xrpl.config xrpld.rpc > xrpl.core xrpld.rpc > xrpld.core -xrpld.rpc > xrpld.telemetry xrpld.rpc > xrpl.json xrpld.rpc > xrpl.ledger xrpld.rpc > xrpl.net diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 4dea32ae88..069c9c72e9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -511,7 +511,7 @@ Enforcement is by the same script as the span rules, whose metric rules are: ## Adding a new OTel metric -See `src/xrpld/telemetry/MetricMacros.h` for the call-site macros covering every +See `include/xrpl/telemetry/MetricMacros.h` for the call-site macros covering every OTel instrument kind (Counter, UpDownCounter, Histogram, Gauge, and their Observable/async counterparts), `src/xrpld/telemetry/MetricNames.h` for the name and label constants to reference (and the rules above), and the "Adding a New diff --git a/OpenTelemetryPlan/06-implementation-phases.md b/OpenTelemetryPlan/06-implementation-phases.md index 5f4f2ad95b..0d94e61d24 100644 --- a/OpenTelemetryPlan/06-implementation-phases.md +++ b/OpenTelemetryPlan/06-implementation-phases.md @@ -794,8 +794,10 @@ flowchart LR ## 6.8.2 Phase 9: Internal Metric Instrumentation Gap Fill (Weeks 14-15) > **Status**: Complete. Merged on `pratik/otel-phase9-metric-gap-fill`. Shipped -> artefacts: `src/xrpld/telemetry/MetricsRegistry.{h,cpp}` (~41 KB + ~71 KB), -> `src/xrpld/telemetry/MetricMacros.h`, `include/xrpl/nodestore/WriteStats.h`, +> artefacts: `include/xrpl/telemetry/MetricsRegistry.h` (~40 KB) with +> `src/libxrpl/telemetry/MetricsRegistry.cpp` (~28 KB), +> `src/xrpld/telemetry/AppMetricGauges.{h,cpp}` (~20 KB + ~56 KB), +> `include/xrpl/telemetry/MetricMacros.h`, `include/xrpl/nodestore/WriteStats.h`, > `src/xrpld/app/ledger/AcquireStats.h`, > `include/xrpl/telemetry/GetObjectMetricNames.h`, 10 GTest files under > `src/tests/libxrpl/telemetry/`, 4 new Grafana dashboards, provisioned Grafana @@ -1955,13 +1957,13 @@ class ValidationTracker **Key new files**: -- `src/xrpld/telemetry/ValidationTracker.h` -- `src/xrpld/telemetry/detail/ValidationTracker.cpp` +- `include/xrpl/telemetry/ValidationTracker.h` +- `src/libxrpl/telemetry/detail/ValidationTracker.cpp` **Key modified files**: -- `src/xrpld/telemetry/MetricsRegistry.h` (add ValidationTracker member) -- `src/xrpld/telemetry/MetricsRegistry.cpp` (add gauge callback reading from tracker) +- `include/xrpl/telemetry/MetricsRegistry.h` (add ValidationTracker member) +- `src/xrpld/telemetry/AppMetricGauges.cpp` (add gauge callback reading from tracker) - `src/xrpld/app/consensus/RCLConsensus.cpp` (add recording hooks) - `src/xrpld/app/ledger/detail/LedgerMaster.cpp` (add recording hook) @@ -1986,7 +1988,7 @@ New MetricsRegistry observable gauge for amendment, UNL, and quorum health. | | `unl_expiry_days` | double | `app_.validators().expires()` → days until expiry | | | `validation_quorum` | int64 | `app_.validators().quorum()` | -**File**: `src/xrpld/telemetry/MetricsRegistry.cpp` (new gauge callback in `registerAsyncGauges()`) +**File**: `src/xrpld/telemetry/AppMetricGauges.cpp` (new gauge callback in `registerAsyncGauges()`) **Exit Criteria**: @@ -2009,7 +2011,7 @@ New MetricsRegistry observable gauge for peer health aggregates. **Implementation note**: The callback iterates `app_.overlay().foreach(...)` to collect per-peer latency and version data. This runs every 10s on the metrics reader thread — acceptable overhead for ~50-200 peers. -**File**: `src/xrpld/telemetry/MetricsRegistry.cpp` +**File**: `src/xrpld/telemetry/AppMetricGauges.cpp` **Exit Criteria**: @@ -2032,7 +2034,7 @@ New MetricsRegistry observable gauge for fee and ledger metrics. | | `ledger_age_seconds` | double | `now - lastValidatedCloseTime` | | | `transaction_rate` | double | Derived: tx count delta / time delta | -**File**: `src/xrpld/telemetry/MetricsRegistry.cpp` +**File**: `src/xrpld/telemetry/AppMetricGauges.cpp` **Exit Criteria**: @@ -2067,7 +2069,7 @@ xrpld's `OperatingMode` enum maps 0-4 (DISCONNECTED through FULL). The external **Note**: Values 5-6 require checking both `OperatingMode` and `ConsensusMode`. The callback should derive these from `app_.getOPs().getOperatingMode()` combined with `mConsensus.mode()`. If operating mode is FULL and consensus is proposing → 6; if FULL and validating → 5; otherwise use the raw OperatingMode enum value. -**File**: `src/xrpld/telemetry/MetricsRegistry.cpp` +**File**: `src/xrpld/telemetry/AppMetricGauges.cpp` **Exit Criteria**: @@ -2094,7 +2096,7 @@ The label value was `nudb_bytes` through Phase 8 and was renamed in Phase 9: the value is read from `Database`, not from the NuDB backend, so a backend prefix misdescribed it and the old name implied an on-disk size it never reported. -**File**: `src/xrpld/telemetry/MetricsRegistry.cpp` +**File**: `src/xrpld/telemetry/AppMetricGauges.cpp` **Exit Criteria**: @@ -2119,7 +2121,10 @@ New counters incremented at event sites. Declared in MetricsRegistry, recording **Key modified files**: -- `src/xrpld/telemetry/MetricsRegistry.h/.cpp` (counter declarations) +- `include/xrpl/telemetry/MetricsRegistry.h` and + `src/libxrpl/telemetry/MetricsRegistry.cpp` (synchronous counter declarations) +- `src/xrpld/telemetry/AppMetricGauges.cpp` (the three observed as ObservableCounters: + `validation_agreements_total`, `validation_missed_total`, `jq_trans_overflow_total`) - `src/xrpld/app/consensus/RCLConsensus.cpp` (recording: ledgers_closed, validations_sent) - `src/xrpld/app/ledger/detail/LedgerMaster.cpp` (recording: validations_checked) - `src/xrpld/app/misc/NetworkOPs.cpp` (recording: state_changes) @@ -2145,7 +2150,7 @@ Reads from the `ValidationTracker` (Task 7.8) to export rolling window stats. | | `agreements_24h` | int64 | `tracker.agreements24h()` | | | `missed_24h` | int64 | `tracker.missed24h()` | -**File**: `src/xrpld/telemetry/MetricsRegistry.cpp` +**File**: `src/xrpld/telemetry/AppMetricGauges.cpp` **Exit Criteria**: @@ -2333,12 +2338,12 @@ Phase 9 additionally ships 9 rules with no external counterpart: | Peer Count Critical | `server_info{metric="peers"} < 5` | — | > **"Not Proposing" is unblocked.** The `state_tracking` gauge **is** -> implemented: `MetricsRegistry::registerStateTrackingGauge()` -> (`MetricsRegistry.cpp:1461-1510`) creates -> `CreateDoubleObservableGauge("state_tracking", …)` at `:1466` and observes -> `state_value` (`:1497`) and `time_in_current_state_seconds` (`:1502`). It is -> already consumed by `validator-health.json:765,971` and -> `ledger-data-sync.json:869`, and documented in +> implemented: `AppMetricGauges::registerStateTrackingGauge()` +> (`src/xrpld/telemetry/AppMetricGauges.cpp`) creates +> `CreateDoubleObservableGauge("state_tracking", …)` and observes `state_value` +> and `time_in_current_state_seconds`. It is +> already consumed by `validator-health.json` and +> `ledger-data-sync.json`, and documented in > [09-data-collection-reference.md](./09-data-collection-reference.md) § > "State Tracking". Only **3** of the 14 remaining rules are blocked on anything — > CPU High, Memory Critical and Disk Warning, all needing `node_exporter`. diff --git a/OpenTelemetryPlan/08-appendix.md b/OpenTelemetryPlan/08-appendix.md index 3cff77a23f..e059cfcf7f 100644 --- a/OpenTelemetryPlan/08-appendix.md +++ b/OpenTelemetryPlan/08-appendix.md @@ -163,15 +163,15 @@ This guide maps Phase 9–11 content to its location across the documentation. ### Phase 9: Internal Metric Instrumentation Gap Fill -| Content | Location | -| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | -| Plan & architecture | [06-implementation-phases.md §6.8.2](./06-implementation-phases.md) | -| Task list (18 entries, 9.1–9.17) | [Phase9_taskList.md](./Phase9_taskList.md) | -| Metric definitions | [09-data-collection-reference.md §5b](./09-data-collection-reference.md) | -| New class: `MetricsRegistry` | `src/xrpld/telemetry/MetricsRegistry.h/.cpp` — **shipped** | -| New dashboards (4) | `fee-market`, `job-queue`, `peer-quality`, `validator-health` — **shipped** | -| Updated dashboards (2) | `node-health`, `rpc-performance` | -| Provisioned alert rules | `docker/telemetry/grafana/provisioning/alerting/rules.yaml` — 13 rules in 5 groups ([07 §7.6.2](./07-observability-backends.md)) | +| Content | Location | +| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Plan & architecture | [06-implementation-phases.md §6.8.2](./06-implementation-phases.md) | +| Task list (18 entries, 9.1–9.17) | [Phase9_taskList.md](./Phase9_taskList.md) | +| Metric definitions | [09-data-collection-reference.md §5b](./09-data-collection-reference.md) | +| New classes: `MetricsRegistry`, `AppMetricGauges` | `include/xrpl/telemetry/MetricsRegistry.h` + `src/libxrpl/telemetry/MetricsRegistry.cpp` (pipeline, counters, histograms) and `src/xrpld/telemetry/AppMetricGauges.h/.cpp` (observable gauges) — **shipped** | +| New dashboards (4) | `fee-market`, `job-queue`, `peer-quality`, `validator-health` — **shipped** | +| Updated dashboards (2) | `node-health`, `rpc-performance` | +| Provisioned alert rules | `docker/telemetry/grafana/provisioning/alerting/rules.yaml` — 13 rules in 5 groups ([07 §7.6.2](./07-observability-backends.md)) | > **Task numbering**: `Phase9_taskList.md` carries 18 `## Task 9.x` headings — > 9.1 through 9.17 plus the inserted 9.7a (`push_metrics.py` parity). The "10 diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md index 5c96c6234b..1394a9a91e 100644 --- a/OpenTelemetryPlan/09-data-collection-reference.md +++ b/OpenTelemetryPlan/09-data-collection-reference.md @@ -801,7 +801,7 @@ reader, and OTLP/HTTP exporter, even though both request a meter named `xrpld` / `1.0.0`. `OTelCollector` takes its meter from the **global** provider, which `Telemetry` publishes and reads every 1000 ms; `MetricsRegistry` builds a private provider it does not publish, read every 10000 ms -(`src/xrpld/telemetry/MetricsRegistry.cpp`). So `jobq__*` and +(`src/libxrpl/telemetry/MetricsRegistry.cpp`). So `jobq__*` and `job_*_total` reach Prometheus on different cadences and should not be assumed sampled at the same instant. @@ -1081,8 +1081,8 @@ async callbacks for new categories. > **Label values are case-sensitive and three cache values are not lowercase.** > The `metric` label carries the string literal passed to `Observe()`, verbatim: > `SLE_hit_rate`, `AL_hit_rate` and `AL_size` are upper-case -> (`src/xrpld/telemetry/MetricsRegistry.cpp:666`, `:682`, `:708`), while -> `ledger_hit_rate` genuinely is lowercase (`:675`). A selector written as +> (all four `Observe()` calls are in `AppMetricGauges::registerCacheHitRateGauge()`), +> while `ledger_hit_rate` genuinely is lowercase. A selector written as > `cache_metrics{metric="sle_hit_rate"}` matches nothing. #### Server Info (via OTel MetricsRegistry) @@ -1263,7 +1263,7 @@ docker/telemetry/workload/benchmark.sh --xrpld .build/xrpld --duration 300 > (`nodestore_state`, `cache_metrics`, …) once. > > Note that `ledgers_closed_total` appears in **both** instrument rows: it is -> created as a `MetricsRegistry` member (`MetricsRegistry.cpp:386-387`, whose +> created as a `MetricsRegistry` member (in `MetricsRegistry::initSyncInstruments()`, whose > `incrementLedgersClosed()` has no callers) and separately incremented at its > call site via `XRPL_METRIC_COUNTER_INC` (`RCLConsensus.cpp:749`). The distinct > name count across the two rows is therefore 41, not 42. @@ -1361,9 +1361,13 @@ Phase 11 builds a custom OTel Collector receiver (Go) that polls xrpld's admin R ### Phase 9: OTel SDK-Exported Metrics (MetricsRegistry) -Phase 9 introduces the `MetricsRegistry` class (`src/xrpld/telemetry/MetricsRegistry.h/.cpp`) -which registers metrics directly with the OpenTelemetry Metrics SDK. These are exported -via OTLP/HTTP to the OTel Collector and scraped by Prometheus. +Phase 9 introduces the `MetricsRegistry` class (`include/xrpl/telemetry/MetricsRegistry.h`, +`src/libxrpl/telemetry/MetricsRegistry.cpp`) which registers metrics directly with the +OpenTelemetry Metrics SDK. The synchronous counters and histograms are created there. The +observable gauges in the tables below are registered by `AppMetricGauges` +(`src/xrpld/telemetry/AppMetricGauges.h`, `src/xrpld/telemetry/AppMetricGauges.cpp`), which +stays in `xrpld` because its callbacks read `Application`. Both are exported via OTLP/HTTP +to the OTel Collector and scraped by Prometheus. #### NodeStore I/O (Observable Gauge — `nodestore_state`) @@ -1403,9 +1407,9 @@ via OTLP/HTTP to the OTel Collector and scraped by Prometheus. Further label values on the same instrument, added to separate the two bottlenecks that both present as the `ledgerData` job lane pinned at its -concurrency cap. Observed in `MetricsRegistry::observeNodeStoreTotals()`, +concurrency cap. Observed in `AppMetricGauges::observeNodeStoreTotals()`, `observeWritePathDetail()`, and `observeAcquireStats()` -(`src/xrpld/telemetry/MetricsRegistry.cpp:871-942`). +(`src/xrpld/telemetry/AppMetricGauges.cpp`). | Prometheus Metric | Type | Labels | Description | | ---------------------------------------------------- | ----- | -------- | ------------------------------------------------------- | @@ -1497,7 +1501,7 @@ data as uninformative unless the build is known to include the fix. #### TxQ Admission and Ledger Mismatch (Synchronous Counters) Three monotonic counters created alongside the Phase 7+ parity counters -(`src/xrpld/telemetry/MetricsRegistry.cpp:394-399`). The gauges above answer +(in `MetricsRegistry::initSyncInstruments()`). The gauges above answer "how deep is the queue"; these answer "what did the queue refuse, and did the ledger we built match the one the network validated". @@ -1543,7 +1547,7 @@ Rejections (Dropped)", "Queue Abandonment Rate (Expired)"; _Consensus Health_ #### Reduce-Relay Efficiency (Observable Gauge — `reduce_relay_metrics`) Transaction reduce-relay effectiveness, read from `Overlay::txMetrics()` each -collection cycle (`src/xrpld/telemetry/MetricsRegistry.cpp:1370-1402`). A high +collection cycle (`AppMetricGauges::registerReduceRelayGauge()`). A high `suppressed_peers` : `selected_peers` ratio proves the feature is saving bandwidth; a high `not_enabled_peers` means stale peers are forcing full relay. @@ -1572,7 +1576,7 @@ Selection", "Reduce-Relay Missing-Tx Frequency". | `rpc_in_flight_requests` | UpDownCounter | (none) | RPC calls currently executing (+1 rpcStart, -1 rpcEnd) | `rpc_in_flight_requests` is emitted at its call site via the `XRPL_METRIC_UPDOWN_ADD` -macro (see `src/xrpld/telemetry/MetricMacros.h` and `PerfLogImp.cpp`), not through a +macro (see `include/xrpl/telemetry/MetricMacros.h` and `PerfLogImp.cpp`), not through a `MetricsRegistry` member. As an UpDownCounter it carries no `_total` suffix (that is reserved for monotonic counters). @@ -1582,7 +1586,7 @@ Two histograms describing how much work one request asks for. Names and descriptions are the `constexpr` constants in `include/xrpl/telemetry/RpcMetricNames.h`; both are recorded at their call sites via `XRPL_METRIC_*`, and both have an explicit-bucket view registered in -`src/xrpld/telemetry/MetricsRegistry.cpp`. +`src/libxrpl/telemetry/MetricsRegistry.cpp`. | Prometheus Metric | Type | Labels | Description | | --------------------------- | --------- | ------ | ------------------------------------------------------- | @@ -1702,8 +1706,8 @@ information the batch totals do not already carry. **All three histograms need an explicit bucket view.** The SDK's default histogram boundaries top out at 10000. Every one of these three exceeds that, so -without a view their top quantiles would all read as a flat 10000. Twelve views are -registered in `src/xrpld/telemetry/MetricsRegistry.cpp`, and three of the twelve are +without a view their top quantiles would all read as a flat 10000. Thirteen views are +registered in `src/libxrpl/telemetry/MetricsRegistry.cpp`, and three of the thirteen are for this family: | Instrument | View helper | Boundaries | @@ -1712,13 +1716,15 @@ for this family: | `getobject_request_objects` | `addHistogramView()`, own set | `1, 2, 4, 8, 16, 64, 256, 1024, 4096, 12288` | | `getobject_charge` | `addHistogramView()`, own set | `0, 100, 500, 1000, 5000, 10000, 25000, 50000, 100000` | -The other nine views are `addMicrosecondHistogramView()` on `job_queued_us`, +The other ten views are `addMicrosecondHistogramView()` on `job_queued_us`, `job_running_us`, `rpc_method_us` and `sweep_malloc_trim_us`; -`addRoundDurationHistogramView()` on `consensus_round_duration_ms`; and +`addRoundDurationHistogramView()` on `consensus_round_duration_ms`; +`addRotationPhaseHistogramView()` on `rotation_phase_duration_seconds`; and `addHistogramView()` with its own set on `dns_resolve_latency_ms`, `overlay_dial_latency_ms`, `rpc_batch_size` and `pathfind_discovered_paths`. That -is five µs-ladder views, one round-duration ladder, and six caller-supplied sets — -the two above, those two millisecond latencies, and two object counts. See +is five µs-ladder views, one round-duration ladder, one seconds ladder for the +rotation phases, and six caller-supplied sets — the two above, those two +millisecond latencies, and two object counts. See [RPC Request-Count Histograms](#rpc-request-count-histograms) for the last two. **Why the latter two do not use the µs ladder.** They are not durations. The µs @@ -1758,7 +1764,7 @@ not a lowercase word and not a friendly alias. The value is `beast::typeName()` (`include/xrpl/basics/CountedObject.h:115`), which demangles `typeid(T).name()` with `abi::__cxa_demangle` (`include/xrpl/beast/type_name.h:16-45`) and applies no stripping; the observer -copies it through verbatim (`src/xrpld/telemetry/MetricsRegistry.cpp:781-787`). +copies it through verbatim (`AppMetricGauges::registerObjectCountGauge()`). Values therefore keep their `xrpl::` namespace, nested `::`, and template arguments. @@ -1893,16 +1899,17 @@ These metrics fill gaps identified by comparing xrpld's internal observability w Data source: `ValidationTracker` class with 8s grace period and 5m late repair window. > **Every value on this instrument is a double.** The family is one -> `CreateDoubleObservableGauge` (`src/xrpld/telemetry/MetricsRegistry.cpp:1593`), +> `CreateDoubleObservableGauge` (in `AppMetricGauges::registerValidationAgreementGauge()`), > so the integral counts are cast to `double` before `Observe()` — there is no > Int64 sub-series to filter on. The same holds for `validator_health`, > `peer_quality` and `state_tracking` below; an earlier revision of these four > tables split the Type column between Int64 and Double, which the code does not > do. > -> The 7-day window is `ValidationTracker::kWindow7d` = 168 hours -> (`src/xrpld/telemetry/ValidationTracker.h:311`) and is observed alongside the 1h -> and 24h windows at `MetricsRegistry.cpp:1623-1626`. Panels exist on _Validator +> The 7-day window spans `ValidationTracker::kBuckets7d` = `7 * 24 * 60` one-minute +> buckets, i.e. 168 hours (`include/xrpl/telemetry/ValidationTracker.h`), and is +> observed alongside the 1h and 24h windows in +> `AppMetricGauges::registerValidationAgreementGauge()`. Panels exist on _Validator > Health_ (`validator-health`): "Agreement % (7d)" and "Agreements vs Missed > (7d)". @@ -1915,7 +1922,7 @@ Data source: `ValidationTracker` class with 8s grace period and 5m late repair w | `validator_health{metric="unl_expiry_days"}` | Double | `metric` | Days until UNL list expires | | `validator_health{metric="validation_quorum"}` | Double | `metric` | Validation quorum threshold | -Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1217`. +Single `CreateDoubleObservableGauge`, in `AppMetricGauges::registerValidatorHealthGauge()`. #### Peer Quality (Observable Gauge — `peer_quality`) @@ -1926,7 +1933,7 @@ Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1217`. | `peer_quality{metric="peers_higher_version_pct"}` | Double | `metric` | % of peers on newer xrpld version | | `peer_quality{metric="upgrade_recommended"}` | Double | `metric` | 1 if >60% of peers are newer version | -Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1266`. +Single `CreateDoubleObservableGauge`, in `AppMetricGauges::registerPeerQualityGauge()`. #### Ledger Economy (Observable Gauge — `ledger_economy`) @@ -1945,9 +1952,9 @@ Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1266`. | `state_tracking{metric="state_value"}` | Double | `metric` | Numeric state 0-6 (see encoding below) | | `state_tracking{metric="time_in_current_state_seconds"}` | Double | `metric` | Duration in current state | -Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1483`. +Single `CreateDoubleObservableGauge`, in `AppMetricGauges::registerStateTrackingGauge()`. -State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full, 5=validating (FULL + validating), 6=proposing (FULL + proposing). Values 0-4 are `OperatingMode` cast to double (`include/xrpl/server/NetworkOPs.h:60-66`); 5 and 6 are the FULL-only refinements at `MetricsRegistry.cpp:1500-1515`. **The range is 0-6, not 0-7** — there is no seventh state. +State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full, 5=validating (FULL + validating), 6=proposing (FULL + proposing). Values 0-4 are `OperatingMode` cast to double (`include/xrpl/server/NetworkOPs.h:60-66`); 5 and 6 are the FULL-only refinements in `AppMetricGauges::registerStateTrackingGauge()`. **The range is 0-6, not 0-7** — there is no seventh state. #### Storage Detail (Observable Gauge — `storage_detail`) @@ -1956,11 +1963,11 @@ State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full | `storage_detail{metric="stored_object_bytes"}` | Int64 | `metric` | Cumulative object-payload bytes written (not on-disk size) | > **`stored_object_bytes` is not a file size.** It observes `getStoreSize()` -> (`src/xrpld/telemetry/MetricsRegistry.cpp:1574`), which sums the object payloads +> (in `AppMetricGauges::registerStorageDetailGauge()`), which sums the object payloads > this process has written. It therefore excludes NuDB's keys, bucket padding and > log, and it resets when the process restarts while the files on disk do not. > `node_written_bytes` on the `nodestore_state` gauge calls the same accessor -> (`MetricsRegistry.cpp:877`), so the two series are equal by construction and any +> (in `AppMetricGauges::observeNodeStoreTotals()`), so the two series are equal by construction and any > write-amplification ratio built from the pair is a constant 1.0. To size the store > on disk, stat the backend's files; no metric reports it today. > @@ -1979,12 +1986,11 @@ State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full | `state_changes_total` | Counter | Operating mode transitions | NetworkOPs.cpp | > **Known issue — `ledgers_closed_total` has a dead second producer.** The -> instrument is created twice. `MetricsRegistry::registerCounters()` eagerly +> instrument is created twice. `MetricsRegistry::initSyncInstruments()` eagerly > creates it as the member `ledgersClosedCounter_` -> (`src/xrpld/telemetry/MetricsRegistry.cpp:386-387`), and its only mutator, -> `MetricsRegistry::incrementLedgersClosed()` -> (declared `MetricsRegistry.h:591`, defined `MetricsRegistry.cpp:1703`), has -> **zero callers** — the header says so itself at `MetricsRegistry.h:584-588`. +> (`src/libxrpl/telemetry/MetricsRegistry.cpp`), and its only mutator, +> `MetricsRegistry::incrementLedgersClosed()`, has **zero callers** — the `@note` +> on its declaration in `include/xrpl/telemetry/MetricsRegistry.h` says so itself. > The value operators actually see comes from the single live increment, > the `XRPL_METRIC_COUNTER_INC` call site in > `RCLConsensus::Adaptor::doAccept()` (`src/xrpld/app/consensus/RCLConsensus.cpp:749`). @@ -2031,15 +2037,15 @@ The dotted form was dropped by the 2026-05-13 naming redesign, in three commits: What the code emits today, and where it is documented: -| Old dotted key (never emitted) | Live equivalent | -| ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `xrpl.peer.version` | `peer_version` — see [§Transaction Attributes](#transaction-attributes) | -| `xrpl.validation.ledger_hash`, `xrpl.peer.validation.ledger_hash` | one bare `ledger_hash` on both `consensus.validation.send` and `peer.validation.receive` | -| `xrpl.validation.full`, `xrpl.peer.validation.full` | one bare `full_validation` on both of those spans | -| `xrpl.consensus.validation_quorum` | `quorum`, on `consensus.accept` only | -| `xrpl.node.amendment_blocked` | **not a span attribute at all** — only the metric `validator_health{metric="amendment_blocked"}` (`MetricsRegistry.cpp:1233`) | -| `xrpl.node.server_state` | **not a span attribute at all** — only the metric `server_info{metric="server_state"}` (`MetricsRegistry.cpp:1031`) | -| `xrpl.consensus.proposers_validated` | **never implemented** in any form | +| Old dotted key (never emitted) | Live equivalent | +| ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `xrpl.peer.version` | `peer_version` — see [§Transaction Attributes](#transaction-attributes) | +| `xrpl.validation.ledger_hash`, `xrpl.peer.validation.ledger_hash` | one bare `ledger_hash` on both `consensus.validation.send` and `peer.validation.receive` | +| `xrpl.validation.full`, `xrpl.peer.validation.full` | one bare `full_validation` on both of those spans | +| `xrpl.consensus.validation_quorum` | `quorum`, on `consensus.accept` only | +| `xrpl.node.amendment_blocked` | **not a span attribute at all** — only the metric `validator_health{metric="amendment_blocked"}` (`AppMetricGauges::registerValidatorHealthGauge()`) | +| `xrpl.node.server_state` | **not a span attribute at all** — only the metric `server_info{metric="server_state"}` (`AppMetricGauges::registerServerInfoGauge()`) | +| `xrpl.consensus.proposers_validated` | **never implemented** in any form | The identical nine-row list was deleted from `docker/telemetry/workload/expected_spans.json` by commit `cb9fce6890` for the diff --git a/cmake/XrplCore.cmake b/cmake/XrplCore.cmake index 2458d0ec81..aeb815d160 100644 --- a/cmake/XrplCore.cmake +++ b/cmake/XrplCore.cmake @@ -229,12 +229,17 @@ target_link_libraries( # each module's headers: a module can only include xrpl/telemetry/ headers if # it links this target, and the target must already exist at that point. # -# Links xrpl.libxrpl.protocol PRIVATELY for sha512Half (digest.h) +# Links xrpl.libxrpl.protocol and xrpl.libxrpl.core PUBLICLY: ValidationTracker.h +# takes LedgerIndex and MetricMacros.h takes ServiceRegistry, both in interfaces. add_module(xrpl telemetry) target_link_libraries( xrpl.libxrpl.telemetry - PUBLIC xrpl.libxrpl.basics xrpl.libxrpl.beast xrpl.libxrpl.config - PRIVATE xrpl.libxrpl.protocol + PUBLIC + xrpl.libxrpl.basics + xrpl.libxrpl.beast + xrpl.libxrpl.config + xrpl.libxrpl.core + xrpl.libxrpl.protocol ) if(telemetry) # Telemetry owns both the trace and (as of the direct-metrics API) the diff --git a/docker/telemetry/grafana/dashboards/consensus-health.json b/docker/telemetry/grafana/dashboards/consensus-health.json index fbadb7f1fb..f66d65a993 100644 --- a/docker/telemetry/grafana/dashboards/consensus-health.json +++ b/docker/telemetry/grafana/dashboards/consensus-health.json @@ -1187,7 +1187,7 @@ }, { "title": "Ledger History Mismatch Rate by Reason", - "description": "###### What this is:\n*Rate of built-versus-validated ledger mismatches, broken down by why they diverged.*\n\n###### How it's computed:\n*Per-second rate of mismatch events grouped by reason, per node, over a 5-minute window.*\n\n###### Reading it:\n*Flat at zero is healthy; the reason label tells you the nature of any divergence.*\n\n###### Healthy range:\n*Zero under normal operation.*\n\n###### Watch for:\n*Any sustained rate marks a fork; the reason distinguishes close-time disagreement, sync drift, and transaction-processing differences.*\n\n###### Keywords:\n- **Ledger history mismatch** *(per node)* \u2014 when a locally built ledger's hash does not match the network-validated hash.\n- **Fork** *(network-wide)* \u2014 when nodes validate divergent ledger chains instead of a single agreed history.\n- **Close time** *(network event)* \u2014 the timestamp validators agree to stamp on a ledger, rounded to a shared resolution.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementLedgerHistoryMismatch`\n\n###### References:\n[Fork](https://xrpl.org/docs/concepts/consensus-protocol) \u00b7 [Close time](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-history-mismatch)", + "description": "###### What this is:\n*Rate of built-versus-validated ledger mismatches, broken down by why they diverged.*\n\n###### How it's computed:\n*Per-second rate of mismatch events grouped by reason, per node, over a 5-minute window.*\n\n###### Reading it:\n*Flat at zero is healthy; the reason label tells you the nature of any divergence.*\n\n###### Healthy range:\n*Zero under normal operation.*\n\n###### Watch for:\n*Any sustained rate marks a fork; the reason distinguishes close-time disagreement, sync drift, and transaction-processing differences.*\n\n###### Keywords:\n- **Ledger history mismatch** *(per node)* \u2014 when a locally built ledger's hash does not match the network-validated hash.\n- **Fork** *(network-wide)* \u2014 when nodes validate divergent ledger chains instead of a single agreed history.\n- **Close time** *(network event)* \u2014 the timestamp validators agree to stamp on a ledger, rounded to a shared resolution.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementLedgerHistoryMismatch`\n\n###### References:\n[Fork](https://xrpl.org/docs/concepts/consensus-protocol) \u00b7 [Close time](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-history-mismatch)", "type": "timeseries", "gridPos": { "h": 10, diff --git a/docker/telemetry/grafana/dashboards/fee-market.json b/docker/telemetry/grafana/dashboards/fee-market.json index 8faf0acc7c..bc90c3e7a7 100644 --- a/docker/telemetry/grafana/dashboards/fee-market.json +++ b/docker/telemetry/grafana/dashboards/fee-market.json @@ -71,7 +71,7 @@ }, { "title": "Transaction Queue Depth", - "description": "###### What this is:\n*Transactions currently waiting in the transaction queue versus the queue's maximum capacity.*\n\n###### How it's computed:\n*Instantaneous gauge readings of current queue count and configured max size.*\n\n###### Reading it:\n*Queue depth well below capacity is normal; depth approaching capacity means the node is saturating.*\n\n###### Healthy range:\n*Depth near 0 in quiet periods; workload-dependent under load.*\n\n###### Watch for:\n*Depth pinned at capacity for sustained periods, which signals demand exceeding throughput or a fee-spam burst.*\n\n###### Keywords:\n- **Transaction queue (TxQ)** *(per node)* \u2014 holds transactions that meet local cost but not the open-ledger cost, to include in a later ledger.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerTxqGauge`\n\n###### References:\n[Transaction queue (TxQ)](https://xrpl.org/docs/concepts/transactions/transaction-queue) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#transaction-queue-txq)", + "description": "###### What this is:\n*Transactions currently waiting in the transaction queue versus the queue's maximum capacity.*\n\n###### How it's computed:\n*Instantaneous gauge readings of current queue count and configured max size.*\n\n###### Reading it:\n*Queue depth well below capacity is normal; depth approaching capacity means the node is saturating.*\n\n###### Healthy range:\n*Depth near 0 in quiet periods; workload-dependent under load.*\n\n###### Watch for:\n*Depth pinned at capacity for sustained periods, which signals demand exceeding throughput or a fee-spam burst.*\n\n###### Keywords:\n- **Transaction queue (TxQ)** *(per node)* \u2014 holds transactions that meet local cost but not the open-ledger cost, to include in a later ledger.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerTxqGauge`\n\n###### References:\n[Transaction queue (TxQ)](https://xrpl.org/docs/concepts/transactions/transaction-queue) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#transaction-queue-txq)", "type": "timeseries", "gridPos": { "h": 10, @@ -126,7 +126,7 @@ }, { "title": "Transactions Per Ledger", - "description": "###### What this is:\n*Transactions already placed in the current open ledger versus the expected per-ledger target.*\n\n###### How it's computed:\n*Instantaneous gauge readings of in-ledger count and the target count that governs fee escalation.*\n\n###### Reading it:\n*Staying at or below the expected target is normal; exceeding it triggers open-ledger fee escalation.*\n\n###### Healthy range:\n*At or under the expected per-ledger target.*\n\n###### Watch for:\n*In-ledger count persistently above target, indicating sustained congestion pushing fees up.*\n\n###### Keywords:\n- **Open ledger** *(per node)* \u2014 the temporary workspace ledger where incoming transactions are provisionally applied before a close.\n- **In-ledger vs target count** *(per node)* \u2014 transactions already in the open ledger versus the soft target that triggers fee escalation.\n- **Fee escalation** *(per node)* \u2014 the exponential rise in the open-ledger cost once the ledger exceeds its soft transaction target.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerTxqGauge`\n\n###### References:\n[Open ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [In-ledger vs target count](https://xrpl.org/docs/concepts/transactions/transaction-cost#open-ledger-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#open-ledger)", + "description": "###### What this is:\n*Transactions already placed in the current open ledger versus the expected per-ledger target.*\n\n###### How it's computed:\n*Instantaneous gauge readings of in-ledger count and the target count that governs fee escalation.*\n\n###### Reading it:\n*Staying at or below the expected target is normal; exceeding it triggers open-ledger fee escalation.*\n\n###### Healthy range:\n*At or under the expected per-ledger target.*\n\n###### Watch for:\n*In-ledger count persistently above target, indicating sustained congestion pushing fees up.*\n\n###### Keywords:\n- **Open ledger** *(per node)* \u2014 the temporary workspace ledger where incoming transactions are provisionally applied before a close.\n- **In-ledger vs target count** *(per node)* \u2014 transactions already in the open ledger versus the soft target that triggers fee escalation.\n- **Fee escalation** *(per node)* \u2014 the exponential rise in the open-ledger cost once the ledger exceeds its soft transaction target.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerTxqGauge`\n\n###### References:\n[Open ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [In-ledger vs target count](https://xrpl.org/docs/concepts/transactions/transaction-cost#open-ledger-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#open-ledger)", "type": "timeseries", "gridPos": { "h": 10, @@ -193,7 +193,7 @@ }, { "title": "Fee Escalation Levels", - "description": "###### What this is:\n*The fee levels that govern queue admission: reference (baseline), minimum processing, median, and open-ledger levels.*\n\n###### How it's computed:\n*Instantaneous gauge readings of each fee level, shown on a log scale.*\n\n###### Reading it:\n*Open-ledger level near the reference level means cheap entry; a large gap above reference means escalation is active.*\n\n###### Healthy range:\n*Open-ledger level at or near reference during normal traffic.*\n\n###### Watch for:\n*Open-ledger level spiking far above reference, the hallmark of congestion or a fee-bidding war.*\n\n###### Keywords:\n- **Queue admission rejection** *(per node)* \u2014 a transaction refused entry to the queue, e.g. queue_full when the queue is at capacity.\n- **Fee levels** *(per node)* \u2014 cost thresholds governing queue admission: reference (baseline), minimum, median, and open-ledger.\n- **Open ledger** *(per node)* \u2014 the temporary workspace ledger where incoming transactions are provisionally applied before a close.\n- **Fee escalation** *(per node)* \u2014 the exponential rise in the open-ledger cost once the ledger exceeds its soft transaction target.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerTxqGauge`\n\n###### References:\n[Queue admission rejection](https://xrpl.org/docs/concepts/transactions/transaction-queue) \u00b7 [Fee levels](https://xrpl.org/docs/concepts/transactions/transaction-cost#fee-levels) \u00b7 [Open ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Fee escalation](https://xrpl.org/docs/concepts/transactions/transaction-cost#open-ledger-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#queue-admission-rejection)", + "description": "###### What this is:\n*The fee levels that govern queue admission: reference (baseline), minimum processing, median, and open-ledger levels.*\n\n###### How it's computed:\n*Instantaneous gauge readings of each fee level, shown on a log scale.*\n\n###### Reading it:\n*Open-ledger level near the reference level means cheap entry; a large gap above reference means escalation is active.*\n\n###### Healthy range:\n*Open-ledger level at or near reference during normal traffic.*\n\n###### Watch for:\n*Open-ledger level spiking far above reference, the hallmark of congestion or a fee-bidding war.*\n\n###### Keywords:\n- **Queue admission rejection** *(per node)* \u2014 a transaction refused entry to the queue, e.g. queue_full when the queue is at capacity.\n- **Fee levels** *(per node)* \u2014 cost thresholds governing queue admission: reference (baseline), minimum, median, and open-ledger.\n- **Open ledger** *(per node)* \u2014 the temporary workspace ledger where incoming transactions are provisionally applied before a close.\n- **Fee escalation** *(per node)* \u2014 the exponential rise in the open-ledger cost once the ledger exceeds its soft transaction target.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerTxqGauge`\n\n###### References:\n[Queue admission rejection](https://xrpl.org/docs/concepts/transactions/transaction-queue) \u00b7 [Fee levels](https://xrpl.org/docs/concepts/transactions/transaction-cost#fee-levels) \u00b7 [Open ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Fee escalation](https://xrpl.org/docs/concepts/transactions/transaction-cost#open-ledger-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#queue-admission-rejection)", "type": "timeseries", "gridPos": { "h": 10, @@ -265,7 +265,7 @@ }, { "title": "Load Factor Breakdown", - "description": "###### What this is:\n*The combined load factor and its server, fee-escalation, and fee-queue contributors as unitless fee multipliers (1.0 = no load).*\n\n###### How it's computed:\n*Instantaneous gauge readings of each load-factor component.*\n\n###### Reading it:\n*Values at 1.0 mean base fees; higher values raise the fee to transact.*\n\n###### Healthy range:\n*Around 1.0 under normal conditions.*\n\n###### Watch for:\n*Combined factor climbing well above 1.0, showing the node is charging premium fees due to congestion or overload.*\n\n###### Keywords:\n- **Load factor** *(per node)* \u2014 a unitless multiplier (1.0 = no load) that scales the base transaction cost as the node comes under load.\n- **Fee escalation** *(per node)* \u2014 the exponential rise in the open-ledger cost once the ledger exceeds its soft transaction target.\n- **Transaction cost** *(network-wide)* \u2014 the XRP a transaction destroys to be processed; scales up with load to deter spam.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerLoadFactorGauge`\n\n###### References:\n[Load factor](https://xrpl.org/docs/concepts/transactions/transaction-cost#local-load-cost) \u00b7 [Fee escalation](https://xrpl.org/docs/concepts/transactions/transaction-cost#open-ledger-cost) \u00b7 [Transaction cost](https://xrpl.org/docs/concepts/transactions/transaction-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#load-factor)", + "description": "###### What this is:\n*The combined load factor and its server, fee-escalation, and fee-queue contributors as unitless fee multipliers (1.0 = no load).*\n\n###### How it's computed:\n*Instantaneous gauge readings of each load-factor component.*\n\n###### Reading it:\n*Values at 1.0 mean base fees; higher values raise the fee to transact.*\n\n###### Healthy range:\n*Around 1.0 under normal conditions.*\n\n###### Watch for:\n*Combined factor climbing well above 1.0, showing the node is charging premium fees due to congestion or overload.*\n\n###### Keywords:\n- **Load factor** *(per node)* \u2014 a unitless multiplier (1.0 = no load) that scales the base transaction cost as the node comes under load.\n- **Fee escalation** *(per node)* \u2014 the exponential rise in the open-ledger cost once the ledger exceeds its soft transaction target.\n- **Transaction cost** *(network-wide)* \u2014 the XRP a transaction destroys to be processed; scales up with load to deter spam.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerLoadFactorGauge`\n\n###### References:\n[Load factor](https://xrpl.org/docs/concepts/transactions/transaction-cost#local-load-cost) \u00b7 [Fee escalation](https://xrpl.org/docs/concepts/transactions/transaction-cost#open-ledger-cost) \u00b7 [Transaction cost](https://xrpl.org/docs/concepts/transactions/transaction-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#load-factor)", "type": "timeseries", "gridPos": { "h": 10, @@ -349,7 +349,7 @@ }, { "title": "Load Factor Components", - "description": "###### What this is:\n*The individual load-factor inputs, local server load, network load, and cluster load, as unitless multipliers.*\n\n###### How it's computed:\n*Instantaneous gauge readings of each component.*\n\n###### Reading it:\n*All at 1.0 means no load pressure from any source; a raised component identifies where load originates.*\n\n###### Healthy range:\n*Around 1.0 for each component.*\n\n###### Watch for:\n*A single component rising sharply, which pinpoints whether the pressure is local, network-wide, or cluster-driven.*\n\n###### Keywords:\n- **Load factor** *(per node)* \u2014 a unitless multiplier (1.0 = no load) that scales the base transaction cost as the node comes under load.\n- **Cluster** *(cluster-wide)* \u2014 a group of trusted co-operated nodes that share load information and skip some verification.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerLoadFactorGauge`\n\n###### References:\n[Load factor](https://xrpl.org/docs/concepts/transactions/transaction-cost#local-load-cost) \u00b7 [Cluster](https://xrpl.org/docs/concepts/networks-and-servers/clustering) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#load-factor)", + "description": "###### What this is:\n*The individual load-factor inputs, local server load, network load, and cluster load, as unitless multipliers.*\n\n###### How it's computed:\n*Instantaneous gauge readings of each component.*\n\n###### Reading it:\n*All at 1.0 means no load pressure from any source; a raised component identifies where load originates.*\n\n###### Healthy range:\n*Around 1.0 for each component.*\n\n###### Watch for:\n*A single component rising sharply, which pinpoints whether the pressure is local, network-wide, or cluster-driven.*\n\n###### Keywords:\n- **Load factor** *(per node)* \u2014 a unitless multiplier (1.0 = no load) that scales the base transaction cost as the node comes under load.\n- **Cluster** *(cluster-wide)* \u2014 a group of trusted co-operated nodes that share load information and skip some verification.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerLoadFactorGauge`\n\n###### References:\n[Load factor](https://xrpl.org/docs/concepts/transactions/transaction-cost#local-load-cost) \u00b7 [Cluster](https://xrpl.org/docs/concepts/networks-and-servers/clustering) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#load-factor)", "type": "timeseries", "gridPos": { "h": 10, @@ -423,7 +423,7 @@ }, { "title": "Queue Abandonment Rate (Expired)", - "description": "###### What this is:\n*Transactions dropped from the queue because their last-ledger deadline passed before they could be included.*\n\n###### How it's computed:\n*Per-second rate of the cumulative expired-transaction counter over the dashboard's rate interval.*\n\n###### Reading it:\n*Near zero is healthy; a rising rate means submitters under-bid the escalating fee and their transactions timed out.*\n\n###### Healthy range:\n*Near 0 expirations per second.*\n\n###### Watch for:\n*Sustained expiry rate, a demand-frustration signal often coinciding with fee spikes or spam that crowds out honest traffic.*\n\n###### Keywords:\n- **Queue admission rejection** *(per node)* \u2014 a transaction refused entry to the queue, e.g. queue_full when the queue is at capacity.\n- **Queue expiry / abandonment** *(per node)* \u2014 removing a queued transaction whose LastLedgerSequence deadline passed before inclusion.\n- **Fee escalation** *(per node)* \u2014 the exponential rise in the open-ledger cost once the ledger exceeds its soft transaction target.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementTxqExpired (caller TxQ.cpp)`\n\n###### References:\n[Queue admission rejection](https://xrpl.org/docs/concepts/transactions/transaction-queue) \u00b7 [Queue expiry / abandonment](https://xrpl.org/docs/concepts/transactions/reliable-transaction-submission) \u00b7 [Fee escalation](https://xrpl.org/docs/concepts/transactions/transaction-cost#open-ledger-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#queue-admission-rejection)", + "description": "###### What this is:\n*Transactions dropped from the queue because their last-ledger deadline passed before they could be included.*\n\n###### How it's computed:\n*Per-second rate of the cumulative expired-transaction counter over the dashboard's rate interval.*\n\n###### Reading it:\n*Near zero is healthy; a rising rate means submitters under-bid the escalating fee and their transactions timed out.*\n\n###### Healthy range:\n*Near 0 expirations per second.*\n\n###### Watch for:\n*Sustained expiry rate, a demand-frustration signal often coinciding with fee spikes or spam that crowds out honest traffic.*\n\n###### Keywords:\n- **Queue admission rejection** *(per node)* \u2014 a transaction refused entry to the queue, e.g. queue_full when the queue is at capacity.\n- **Queue expiry / abandonment** *(per node)* \u2014 removing a queued transaction whose LastLedgerSequence deadline passed before inclusion.\n- **Fee escalation** *(per node)* \u2014 the exponential rise in the open-ledger cost once the ledger exceeds its soft transaction target.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementTxqExpired (caller TxQ.cpp)`\n\n###### References:\n[Queue admission rejection](https://xrpl.org/docs/concepts/transactions/transaction-queue) \u00b7 [Queue expiry / abandonment](https://xrpl.org/docs/concepts/transactions/reliable-transaction-submission) \u00b7 [Fee escalation](https://xrpl.org/docs/concepts/transactions/transaction-cost#open-ledger-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#queue-admission-rejection)", "type": "timeseries", "gridPos": { "h": 10, @@ -468,7 +468,7 @@ }, { "title": "Queue Admission Rejections (Dropped)", - "description": "###### What this is:\n*Transactions refused entry to the queue, broken down by reason such as queue_full.*\n\n###### How it's computed:\n*Per-second rate of the cumulative dropped-transaction counter over the dashboard's rate interval, split by reason.*\n\n###### Reading it:\n*Near zero is healthy; queue_full rejections mean the queue is at capacity and applying backpressure.*\n\n###### Healthy range:\n*Near 0 rejections per second.*\n\n###### Watch for:\n*A burst of queue_full drops, distinct from expiry, indicating the node is being flooded faster than it can drain.*\n\n###### Keywords:\n- **Queue admission rejection** *(per node)* \u2014 a transaction refused entry to the queue, e.g. queue_full when the queue is at capacity.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementTxqDropped (caller TxQ.cpp)`\n\n###### References:\n[Queue admission rejection](https://xrpl.org/docs/concepts/transactions/transaction-queue) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#queue-admission-rejection)", + "description": "###### What this is:\n*Transactions refused entry to the queue, broken down by reason such as queue_full.*\n\n###### How it's computed:\n*Per-second rate of the cumulative dropped-transaction counter over the dashboard's rate interval, split by reason.*\n\n###### Reading it:\n*Near zero is healthy; queue_full rejections mean the queue is at capacity and applying backpressure.*\n\n###### Healthy range:\n*Near 0 rejections per second.*\n\n###### Watch for:\n*A burst of queue_full drops, distinct from expiry, indicating the node is being flooded faster than it can drain.*\n\n###### Keywords:\n- **Queue admission rejection** *(per node)* \u2014 a transaction refused entry to the queue, e.g. queue_full when the queue is at capacity.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementTxqDropped (caller TxQ.cpp)`\n\n###### References:\n[Queue admission rejection](https://xrpl.org/docs/concepts/transactions/transaction-queue) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#queue-admission-rejection)", "type": "timeseries", "gridPos": { "h": 10, diff --git a/docker/telemetry/grafana/dashboards/job-queue.json b/docker/telemetry/grafana/dashboards/job-queue.json index d7baefa1b1..9273c3eb68 100644 --- a/docker/telemetry/grafana/dashboards/job-queue.json +++ b/docker/telemetry/grafana/dashboards/job-queue.json @@ -71,7 +71,7 @@ }, { "title": "Current Job Latency (p99 Gauge) [$xrpl_network_type]", - "description": "###### What this is:\n*At-a-glance p99 of how long jobs wait in the queue and how long they run once started.*\n\n###### How it's computed:\n*99th percentile derived from the job wait-time and run-time histograms over the last 5 minutes.*\n\n###### Reading it:\n*Lower is better; green under 100ms, yellow to 1s, red beyond 1s.*\n\n###### Healthy range:\n*Wait and exec p99 under 100ms.*\n\n###### Watch for:\n*p99 wait climbing into the red, meaning worker threads are saturated and jobs are backing up.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Deferred job** *(per node)* \u2014 a job held back because its type is already at its concurrency limit; the leading indicator of queue backpressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobStarted / recordJobFinished`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", + "description": "###### What this is:\n*At-a-glance p99 of how long jobs wait in the queue and how long they run once started.*\n\n###### How it's computed:\n*99th percentile derived from the job wait-time and run-time histograms over the last 5 minutes.*\n\n###### Reading it:\n*Lower is better; green under 100ms, yellow to 1s, red beyond 1s.*\n\n###### Healthy range:\n*Wait and exec p99 under 100ms.*\n\n###### Watch for:\n*p99 wait climbing into the red, meaning worker threads are saturated and jobs are backing up.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Deferred job** *(per node)* \u2014 a job held back because its type is already at its concurrency limit; the leading indicator of queue backpressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobStarted / recordJobFinished`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", "type": "gauge", "gridPos": { "h": 10, @@ -148,7 +148,7 @@ }, { "title": "Job Throughput Rate (Per Second)", - "description": "###### What this is:\n*Rate of jobs queued, started, and finished across all job types.*\n\n###### How it's computed:\n*Per-second rate of each cumulative job counter over a 5-minute window.*\n\n###### Reading it:\n*Queued, started, and finished tracking together means the queue keeps up.*\n\n###### Healthy range:\n*Workload-dependent; the three rates should stay roughly equal.*\n\n###### Watch for:\n*Queued rate persistently above finished rate, which indicates a growing backlog.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Deferred job** *(per node)* \u2014 a job held back because its type is already at its concurrency limit; the leading indicator of queue backpressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobQueued / recordJobStarted / recordJobFinished`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", + "description": "###### What this is:\n*Rate of jobs queued, started, and finished across all job types.*\n\n###### How it's computed:\n*Per-second rate of each cumulative job counter over a 5-minute window.*\n\n###### Reading it:\n*Queued, started, and finished tracking together means the queue keeps up.*\n\n###### Healthy range:\n*Workload-dependent; the three rates should stay roughly equal.*\n\n###### Watch for:\n*Queued rate persistently above finished rate, which indicates a growing backlog.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Deferred job** *(per node)* \u2014 a job held back because its type is already at its concurrency limit; the leading indicator of queue backpressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobQueued / recordJobStarted / recordJobFinished`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", "type": "timeseries", "gridPos": { "h": 10, @@ -208,7 +208,7 @@ }, { "title": "Per-Job-Type Queued Rate", - "description": "###### What this is:\n*Rate of jobs entering the queue, broken down by job type (top 10).*\n\n###### How it's computed:\n*Per-second rate of the queued-job counter per job_type over a 5-minute window.*\n\n###### Reading it:\n*Identifies which job types generate the most queue activity.*\n\n###### Healthy range:\n*Workload-dependent.*\n\n###### Watch for:\n*A single job type surging unexpectedly, which can point to a flood of a particular request or peer message.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Deferred job** *(per node)* \u2014 a job held back because its type is already at its concurrency limit; the leading indicator of queue backpressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobQueued`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", + "description": "###### What this is:\n*Rate of jobs entering the queue, broken down by job type (top 10).*\n\n###### How it's computed:\n*Per-second rate of the queued-job counter per job_type over a 5-minute window.*\n\n###### Reading it:\n*Identifies which job types generate the most queue activity.*\n\n###### Healthy range:\n*Workload-dependent.*\n\n###### Watch for:\n*A single job type surging unexpectedly, which can point to a flood of a particular request or peer message.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Deferred job** *(per node)* \u2014 a job held back because its type is already at its concurrency limit; the leading indicator of queue backpressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobQueued`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", "type": "timeseries", "gridPos": { "h": 10, @@ -261,7 +261,7 @@ }, { "title": "Per-Job-Type Finish Rate", - "description": "###### What this is:\n*Rate of jobs completing, broken down by job type (top 10).*\n\n###### How it's computed:\n*Per-second rate of the finished-job counter per job_type over a 5-minute window.*\n\n###### Reading it:\n*Compare against the queued rate per type to spot which types are falling behind.*\n\n###### Healthy range:\n*Workload-dependent; should match the per-type queued rate.*\n\n###### Watch for:\n*A type whose finish rate lags its queued rate, revealing where the backlog concentrates.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Deferred job** *(per node)* \u2014 a job held back because its type is already at its concurrency limit; the leading indicator of queue backpressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobFinished`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", + "description": "###### What this is:\n*Rate of jobs completing, broken down by job type (top 10).*\n\n###### How it's computed:\n*Per-second rate of the finished-job counter per job_type over a 5-minute window.*\n\n###### Reading it:\n*Compare against the queued rate per type to spot which types are falling behind.*\n\n###### Healthy range:\n*Workload-dependent; should match the per-type queued rate.*\n\n###### Watch for:\n*A type whose finish rate lags its queued rate, revealing where the backlog concentrates.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Deferred job** *(per node)* \u2014 a job held back because its type is already at its concurrency limit; the leading indicator of queue backpressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobFinished`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", "type": "timeseries", "gridPos": { "h": 10, @@ -327,7 +327,7 @@ }, { "title": "Job Queue Wait Time", - "description": "###### What this is:\n*How long jobs sit in the queue before a worker picks them up, split by job type (p75 typical, p99 tail).*\n\n###### How it's computed:\n*Percentiles derived from the job wait-time histogram over a 5-minute window, kept per job type. Limited to the ten types with the highest wait so the legend stays readable.*\n\n###### Reading it:\n*Lower is better. The legend names the job type, so a single slow queue is identifiable rather than hidden in an all-types average. A widening p75-to-p99 gap on one type signals occasional stalls there.*\n\n###### Healthy range:\n*Sub-millisecond to low-millisecond waits on an unloaded node.*\n\n###### Watch for:\n*Rising p99 wait on a capped type -- ledgerRequest, ledgerData and makeFetchPack have small concurrency limits, so they queue first. Cross-check the deferred gauge for that type.*\n\n###### Keywords:\n- **Job queue** *(per node)* \u2014 the worker pool that runs xrpld's background work; each unit of work is a job with a type.\n- **Dequeue wait** *(per node)* \u2014 time a job sits enqueued before a worker starts it, as distinct from how long it then runs.\n- **Concurrency limit** *(per node)* \u2014 the maximum number of jobs of one type allowed to run at once; work beyond it is deferred, not rejected.\n\n###### Computation boundary:\n*Result: Per node and job type \u2014 each series is one server's own value for one job type.*\n*Recorded in code as an OTel SDK histogram, then aggregated to percentiles by the Grafana query.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobStarted`\n\n###### References:\n[Job queue](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type) \u00b7 [Concurrency limit](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#concurrency-limit)\n", + "description": "###### What this is:\n*How long jobs sit in the queue before a worker picks them up, split by job type (p75 typical, p99 tail).*\n\n###### How it's computed:\n*Percentiles derived from the job wait-time histogram over a 5-minute window, kept per job type. Limited to the ten types with the highest wait so the legend stays readable.*\n\n###### Reading it:\n*Lower is better. The legend names the job type, so a single slow queue is identifiable rather than hidden in an all-types average. A widening p75-to-p99 gap on one type signals occasional stalls there.*\n\n###### Healthy range:\n*Sub-millisecond to low-millisecond waits on an unloaded node.*\n\n###### Watch for:\n*Rising p99 wait on a capped type -- ledgerRequest, ledgerData and makeFetchPack have small concurrency limits, so they queue first. Cross-check the deferred gauge for that type.*\n\n###### Keywords:\n- **Job queue** *(per node)* \u2014 the worker pool that runs xrpld's background work; each unit of work is a job with a type.\n- **Dequeue wait** *(per node)* \u2014 time a job sits enqueued before a worker starts it, as distinct from how long it then runs.\n- **Concurrency limit** *(per node)* \u2014 the maximum number of jobs of one type allowed to run at once; work beyond it is deferred, not rejected.\n\n###### Computation boundary:\n*Result: Per node and job type \u2014 each series is one server's own value for one job type.*\n*Recorded in code as an OTel SDK histogram, then aggregated to percentiles by the Grafana query.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobStarted`\n\n###### References:\n[Job queue](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type) \u00b7 [Concurrency limit](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#concurrency-limit)\n", "type": "timeseries", "gridPos": { "h": 10, @@ -381,7 +381,7 @@ }, { "title": "Job Execution Time", - "description": "###### What this is:\n*How long jobs run once started, split by job type (p75 typical, p99 tail).*\n\n###### How it's computed:\n*Percentiles derived from the job run-time histogram over a 5-minute window, kept per job type. Limited to the ten slowest types so the legend stays readable.*\n\n###### Reading it:\n*Lower is better. The legend names the job type, so an expensive job type is identifiable rather than averaged away. Stable p75 with a controlled p99 is healthy.*\n\n###### Healthy range:\n*Workload-dependent, but stable over time.*\n\n###### Watch for:\n*Growing execution times, which point to CPU pressure or expensive individual jobs.*\n\n###### Keywords:\n- **Job queue** *(per node)* \u2014 the worker pool that runs xrpld's background work; each unit of work is a job with a type.\n- **Execution time** *(per node)* \u2014 time a job spends running after a worker picks it up, excluding its queue wait.\n\n###### Computation boundary:\n*Result: Per node and job type \u2014 each series is one server's own value for one job type.*\n*Recorded in code as an OTel SDK histogram, then aggregated to percentiles by the Grafana query.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobFinished`\n\n###### References:\n[Job queue](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)\n", + "description": "###### What this is:\n*How long jobs run once started, split by job type (p75 typical, p99 tail).*\n\n###### How it's computed:\n*Percentiles derived from the job run-time histogram over a 5-minute window, kept per job type. Limited to the ten slowest types so the legend stays readable.*\n\n###### Reading it:\n*Lower is better. The legend names the job type, so an expensive job type is identifiable rather than averaged away. Stable p75 with a controlled p99 is healthy.*\n\n###### Healthy range:\n*Workload-dependent, but stable over time.*\n\n###### Watch for:\n*Growing execution times, which point to CPU pressure or expensive individual jobs.*\n\n###### Keywords:\n- **Job queue** *(per node)* \u2014 the worker pool that runs xrpld's background work; each unit of work is a job with a type.\n- **Execution time** *(per node)* \u2014 time a job spends running after a worker picks it up, excluding its queue wait.\n\n###### Computation boundary:\n*Result: Per node and job type \u2014 each series is one server's own value for one job type.*\n*Recorded in code as an OTel SDK histogram, then aggregated to percentiles by the Grafana query.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobFinished`\n\n###### References:\n[Job queue](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)\n", "type": "timeseries", "gridPos": { "h": 10, @@ -435,7 +435,7 @@ }, { "title": "Per-Job-Type Execution Time (p99)", - "description": "###### What this is:\n*The 10 slowest job types ranked by p99 execution time.*\n\n###### How it's computed:\n*p99 derived from the run-time histogram per job_type over a 5-minute window, top 10 selected.*\n\n###### Reading it:\n*Highlights which job types cost the most CPU time at the tail.*\n\n###### Healthy range:\n*Workload-dependent.*\n\n###### Watch for:\n*A job type whose p99 grows over time, indicating a slow or degrading operation.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Deferred job** *(per node)* \u2014 a job held back because its type is already at its concurrency limit; the leading indicator of queue backpressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobFinished`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", + "description": "###### What this is:\n*The 10 slowest job types ranked by p99 execution time.*\n\n###### How it's computed:\n*p99 derived from the run-time histogram per job_type over a 5-minute window, top 10 selected.*\n\n###### Reading it:\n*Highlights which job types cost the most CPU time at the tail.*\n\n###### Healthy range:\n*Workload-dependent.*\n\n###### Watch for:\n*A job type whose p99 grows over time, indicating a slow or degrading operation.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Deferred job** *(per node)* \u2014 a job held back because its type is already at its concurrency limit; the leading indicator of queue backpressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobFinished`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", "type": "timeseries", "gridPos": { "h": 10, @@ -501,7 +501,7 @@ }, { "title": "Transaction Overflow Rate", - "description": "###### What this is:\n*Rate at which transaction jobs are shed when the queue's transaction limit is exceeded.*\n\n###### How it's computed:\n*Per-second rate of the overflow counter over the dashboard's rate interval, scaled to per minute. The counter is observed from the overlay's cumulative overflow tally.*\n\n###### Reading it:\n*Near zero is healthy; a rising rate means the job queue is shedding transaction work under load.*\n\n###### Healthy range:\n*0 overflows per minute.*\n\n###### Watch for:\n*Any sustained non-zero rate \u2014 the node is dropping transaction jobs because the queue is saturated.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Transaction queue (TxQ)** *(per node)* \u2014 holds transactions that cannot enter the open ledger yet, ordered by fee level.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerParityCounters (observed from Overlay::getJqTransOverflow)`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", + "description": "###### What this is:\n*Rate at which transaction jobs are shed when the queue's transaction limit is exceeded.*\n\n###### How it's computed:\n*Per-second rate of the overflow counter over the dashboard's rate interval, scaled to per minute. The counter is observed from the overlay's cumulative overflow tally.*\n\n###### Reading it:\n*Near zero is healthy; a rising rate means the job queue is shedding transaction work under load.*\n\n###### Healthy range:\n*0 overflows per minute.*\n\n###### Watch for:\n*Any sustained non-zero rate \u2014 the node is dropping transaction jobs because the queue is saturated.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Transaction queue (TxQ)** *(per node)* \u2014 holds transactions that cannot enter the open ledger yet, ordered by fee level.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerParityCounters (observed from Overlay::getJqTransOverflow)`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", "type": "timeseries", "gridPos": { "h": 10, diff --git a/docker/telemetry/grafana/dashboards/ledger-data-sync.json b/docker/telemetry/grafana/dashboards/ledger-data-sync.json index 5b2505ecfe..d14a8b88f0 100644 --- a/docker/telemetry/grafana/dashboards/ledger-data-sync.json +++ b/docker/telemetry/grafana/dashboards/ledger-data-sync.json @@ -1729,7 +1729,7 @@ }, { "title": "LedgerReq Wait by Handler", - "description": "###### What this is:\n*Queue wait for the ledgerRequest job type, split by which handler enqueued the job. The type has a concurrency limit of 3 and two producers that compete for those slots: RcvGetLedger, which serves TMGetLedger to syncing peers, and RcvGetObjByHash, which serves TMGetObjectByHash. Both report the same job_type, so without the handler split a wait spike cannot be attributed to either.*\n\n###### How it's computed:\n*p99 of job_queued_us for job_type=\"ledgerRequest\", grouped by the handler label. JobQueue::processTask measures the wait, then PerfLog hands it to MetricsRegistry::recordJobStarted, which is where the histogram is recorded. The handler value is the addJob name passed through a sanitizer that keeps letters-only names and folds everything else to \"other\", which bounds the label domain to 43 names plus \"other\". Both producers here are letters-only, so both appear under their own names; \"other\" is a mixed bucket and never means one specific caller.*\n\n###### Reading it:\n*This is the panel that answers which producer is starving the 3-slot queue. Both lines high together means the queue is genuinely oversubscribed and both kinds of peer request are being delayed. One line high while the other is flat means that producer is arriving faster than 3 concurrent slots can absorb, and it is the one delaying the other. Wait is queue time only, so a high line here is contention, not slow work; the work itself is on the GetObject Handler Latency Breakdown panel.*\n\n###### Healthy range:\n*Single-digit to low-tens of milliseconds p99 for both handlers, matching the wider Job Queue Wait p95 By Type panel.*\n\n###### Watch for:\n*RcvGetObjByHash wait climbing: one in-bounds TMGetObjectByHash request can perform thousands of NodeStore lookups, so a few concurrent ones occupy every slot and delay TMGetLedger to peers that are themselves syncing. Cross-check Job Queue Backlog and Deferred by Type for jobq_ledgerrequest_deferred above zero to confirm the limit, not the work, is the binding constraint.*\n\n###### Keywords:\n- **Handler label** *(per node)* \u2014 the addJob call-site name attached to job metrics, so producers sharing one job type stay separable.\n- **Concurrency limit** *(per node)* \u2014 the cap on how many jobs of one type may run at once; a type at its cap cannot start more work.\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`MetricsRegistry::recordJobStarted`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#handler-label)", + "description": "###### What this is:\n*Queue wait for the ledgerRequest job type, split by which handler enqueued the job. The type has a concurrency limit of 3 and two producers that compete for those slots: RcvGetLedger, which serves TMGetLedger to syncing peers, and RcvGetObjByHash, which serves TMGetObjectByHash. Both report the same job_type, so without the handler split a wait spike cannot be attributed to either.*\n\n###### How it's computed:\n*p99 of job_queued_us for job_type=\"ledgerRequest\", grouped by the handler label. JobQueue::processTask measures the wait, then PerfLog hands it to MetricsRegistry::recordJobStarted, which is where the histogram is recorded. The handler value is the addJob name passed through a sanitizer that keeps letters-only names and folds everything else to \"other\", which bounds the label domain to 43 names plus \"other\". Both producers here are letters-only, so both appear under their own names; \"other\" is a mixed bucket and never means one specific caller.*\n\n###### Reading it:\n*This is the panel that answers which producer is starving the 3-slot queue. Both lines high together means the queue is genuinely oversubscribed and both kinds of peer request are being delayed. One line high while the other is flat means that producer is arriving faster than 3 concurrent slots can absorb, and it is the one delaying the other. Wait is queue time only, so a high line here is contention, not slow work; the work itself is on the GetObject Handler Latency Breakdown panel.*\n\n###### Healthy range:\n*Single-digit to low-tens of milliseconds p99 for both handlers, matching the wider Job Queue Wait p95 By Type panel.*\n\n###### Watch for:\n*RcvGetObjByHash wait climbing: one in-bounds TMGetObjectByHash request can perform thousands of NodeStore lookups, so a few concurrent ones occupy every slot and delay TMGetLedger to peers that are themselves syncing. Cross-check Job Queue Backlog and Deferred by Type for jobq_ledgerrequest_deferred above zero to confirm the limit, not the work, is the binding constraint.*\n\n###### Keywords:\n- **Handler label** *(per node)* \u2014 the addJob call-site name attached to job metrics, so producers sharing one job type stay separable.\n- **Concurrency limit** *(per node)* \u2014 the cap on how many jobs of one type may run at once; a type at its cap cannot start more work.\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`MetricsRegistry::recordJobStarted`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#handler-label)", "type": "timeseries", "gridPos": { "h": 10, @@ -1862,7 +1862,7 @@ }, { "title": "NuDB Writer Queue Depth", - "description": "###### What this is:\n*How many writers are queued at NuDB's insert mutex. NuDB takes one global lock per insert, so concurrent writers do not overlap -- they line up. This panel is the confirming half of the read-latency discriminator: when reads are fast and the lane is still full, the queueing is here.*\n\n###### How it's computed:\n*Mean Depth is nodestore_state{metric=\"nudb_writer_depth_x100\"} divided by 100. At the source it is depthSum over depthSamples, both accumulated when an insert enters the critical section, so an insert still in flight is part of the mean. The exported gauge is integral, so the mean is scaled by 100 to keep the fractional part; dividing it back is what makes 1.60 readable instead of 1. In Flight is the instantaneous nudb_writers_in_flight sample.*\n\n###### Reading it:\n*Depth is a queue length, so 1.0 is the floor and means every insert found the lock free. Anything meaningfully above 1.0 means inserts are waiting on each other and the write path, not the disk, is setting the pace. In Flight is a point sample from the scrape instant and will look spikier than the mean; read the mean for the trend and In Flight for the peak.*\n\n###### Healthy range:\n*Mean depth at or just above 1.0, In Flight low.*\n\n###### Watch for:\n*Mean depth above 1 while read latency stays in single-digit microseconds -- that pair is the write-lock ceiling, and no amount of read tuning will move it. A measured run of this mode held depth at 1.60 with reads flat at 8-9 microseconds and took 510 seconds to reach full. That 1.60 came from a build whose sample count advanced at insert exit rather than entry, which biased the mean down, so treat it as a lower bound. The same run appeared to complete nothing, which was a separate counting defect, not a stalled node. Depth at 1.0 with slow reads is the opposite mode; look at the read panel.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[telemetry/MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`MetricsRegistry::observeWritePathDetail`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", + "description": "###### What this is:\n*How many writers are queued at NuDB's insert mutex. NuDB takes one global lock per insert, so concurrent writers do not overlap -- they line up. This panel is the confirming half of the read-latency discriminator: when reads are fast and the lane is still full, the queueing is here.*\n\n###### How it's computed:\n*Mean Depth is nodestore_state{metric=\"nudb_writer_depth_x100\"} divided by 100. At the source it is depthSum over depthSamples, both accumulated when an insert enters the critical section, so an insert still in flight is part of the mean. The exported gauge is integral, so the mean is scaled by 100 to keep the fractional part; dividing it back is what makes 1.60 readable instead of 1. In Flight is the instantaneous nudb_writers_in_flight sample.*\n\n###### Reading it:\n*Depth is a queue length, so 1.0 is the floor and means every insert found the lock free. Anything meaningfully above 1.0 means inserts are waiting on each other and the write path, not the disk, is setting the pace. In Flight is a point sample from the scrape instant and will look spikier than the mean; read the mean for the trend and In Flight for the peak.*\n\n###### Healthy range:\n*Mean depth at or just above 1.0, In Flight low.*\n\n###### Watch for:\n*Mean depth above 1 while read latency stays in single-digit microseconds -- that pair is the write-lock ceiling, and no amount of read tuning will move it. A measured run of this mode held depth at 1.60 with reads flat at 8-9 microseconds and took 510 seconds to reach full. That 1.60 came from a build whose sample count advanced at insert exit rather than entry, which biased the mean down, so treat it as a lower bound. The same run appeared to complete nothing, which was a separate counting defect, not a stalled node. Depth at 1.0 with slow reads is the opposite mode; look at the read panel.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[telemetry/AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`AppMetricGauges::observeWritePathDetail`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", "type": "timeseries", "gridPos": { "h": 10, @@ -1915,7 +1915,7 @@ }, { "title": "NuDB Insert Time (Mean & Max)", - "description": "###### What this is:\n*Time spent inside a NuDB insert, mean and worst case, alongside the backend's overall mean write latency. Splits an insert's cost into the part that is real work and the part that is waiting for the global insert mutex.*\n\n###### How it's computed:\n*Insert Mean is nodestore_state{metric=\"nudb_insert_mean_us\"}, total insert microseconds over insert count. Insert Max is nudb_insert_max_us, a true running maximum rather than a quantile, so one bad insert is visible and never averaged away. Write Mean is the write_mean_us gauge for the whole backend write path.*\n\n###### Reading it:\n*Compare Insert Mean against the service time implied by Writer Queue Depth. Mean insert time above the unqueued service time is wait, and the gap is the fraction of every write spent queued rather than working. Insert Max far above Insert Mean means the distribution has a tail -- typically a bucket split or a commit -- which a mean alone hides.*\n\n###### Healthy range:\n*Insert Mean in the low tens of microseconds with Insert Max within roughly an order of magnitude of it.*\n\n###### Watch for:\n*A widening gap between Insert Mean and the service floor. In a measured write-lock-bound run the mean was 20 microseconds of which only 12.5 was service, leaving 7.5 -- at least 37 percent of every insert -- as pure queueing. It is a floor rather than an exact figure because it is derived from a mean depth that the build of the day biased downward, and a larger depth implies a larger queueing share. That is the cost the writer-depth panel predicts, quantified. Insert Max climbing on its own without the mean moving is a tail problem, not a saturation problem.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[telemetry/MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`MetricsRegistry::observeWritePathDetail`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", + "description": "###### What this is:\n*Time spent inside a NuDB insert, mean and worst case, alongside the backend's overall mean write latency. Splits an insert's cost into the part that is real work and the part that is waiting for the global insert mutex.*\n\n###### How it's computed:\n*Insert Mean is nodestore_state{metric=\"nudb_insert_mean_us\"}, total insert microseconds over insert count. Insert Max is nudb_insert_max_us, a true running maximum rather than a quantile, so one bad insert is visible and never averaged away. Write Mean is the write_mean_us gauge for the whole backend write path.*\n\n###### Reading it:\n*Compare Insert Mean against the service time implied by Writer Queue Depth. Mean insert time above the unqueued service time is wait, and the gap is the fraction of every write spent queued rather than working. Insert Max far above Insert Mean means the distribution has a tail -- typically a bucket split or a commit -- which a mean alone hides.*\n\n###### Healthy range:\n*Insert Mean in the low tens of microseconds with Insert Max within roughly an order of magnitude of it.*\n\n###### Watch for:\n*A widening gap between Insert Mean and the service floor. In a measured write-lock-bound run the mean was 20 microseconds of which only 12.5 was service, leaving 7.5 -- at least 37 percent of every insert -- as pure queueing. It is a floor rather than an exact figure because it is derived from a mean depth that the build of the day biased downward, and a larger depth implies a larger queueing share. That is the cost the writer-depth panel predicts, quantified. Insert Max climbing on its own without the mean moving is a tail problem, not a saturation problem.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[telemetry/AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`AppMetricGauges::observeWritePathDetail`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", "type": "timeseries", "gridPos": { "h": 10, diff --git a/docker/telemetry/grafana/dashboards/ledger-operations.json b/docker/telemetry/grafana/dashboards/ledger-operations.json index 7abc014089..8fefbda59b 100644 --- a/docker/telemetry/grafana/dashboards/ledger-operations.json +++ b/docker/telemetry/grafana/dashboards/ledger-operations.json @@ -446,7 +446,7 @@ }, { "title": "Ledger Close Interval & Age", - "description": "###### What this is:\n*The wall-clock time between consecutive ledger closes \u2014 the network close cadence.*\n\n###### How it's computed:\n*Close Interval: 1 / rate(ledgers_closed_total), the average seconds between closes from the monotonic close counter (scrape-independent, unlike a gauge delta which would alias to the scrape period). Last-Close Age: time() minus the last-close network time (server_info last_close_time gauge + Ripple-epoch offset), i.e. seconds since the last ledger closed.*\n\n###### Reading it:\n*A steady line near the network's target close interval.*\n\n###### Healthy range:\n*About 3-5s on mainnet; workload-dependent on test networks.*\n\n###### Watch for:\n*A rising interval (consensus slowing or the node lagging) or a flat line at zero (ledgers no longer closing).*\n\n###### Keywords:\n- **Ledger close** *(network event)* \u2014 the current open ledger is closed and a new closed ledger is built from the agreed transaction set.\n- **Ledger close interval** *(network-wide)* \u2014 the network's steady ledger rhythm \u2014 roughly one closed ledger every 3-5 seconds on Mainnet.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp) \u00b7 [RCLConsensus.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/consensus/RCLConsensus.cpp)\n\n###### Function:\n`registerServerInfoGauge (last_close_time) ; ledgers_closed_total`\n\n###### References:\n[Ledger close](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Ledger close interval](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-close)", + "description": "###### What this is:\n*The wall-clock time between consecutive ledger closes \u2014 the network close cadence.*\n\n###### How it's computed:\n*Close Interval: 1 / rate(ledgers_closed_total), the average seconds between closes from the monotonic close counter (scrape-independent, unlike a gauge delta which would alias to the scrape period). Last-Close Age: time() minus the last-close network time (server_info last_close_time gauge + Ripple-epoch offset), i.e. seconds since the last ledger closed.*\n\n###### Reading it:\n*A steady line near the network's target close interval.*\n\n###### Healthy range:\n*About 3-5s on mainnet; workload-dependent on test networks.*\n\n###### Watch for:\n*A rising interval (consensus slowing or the node lagging) or a flat line at zero (ledgers no longer closing).*\n\n###### Keywords:\n- **Ledger close** *(network event)* \u2014 the current open ledger is closed and a new closed ledger is built from the agreed transaction set.\n- **Ledger close interval** *(network-wide)* \u2014 the network's steady ledger rhythm \u2014 roughly one closed ledger every 3-5 seconds on Mainnet.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp) \u00b7 [MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp) \u00b7 [RCLConsensus.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/consensus/RCLConsensus.cpp)\n\n###### Function:\n`registerServerInfoGauge (last_close_time) ; ledgers_closed_total`\n\n###### References:\n[Ledger close](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Ledger close interval](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-close)", "type": "timeseries", "gridPos": { "h": 10, diff --git a/docker/telemetry/grafana/dashboards/node-health.json b/docker/telemetry/grafana/dashboards/node-health.json index 08a693c524..2416247d58 100644 --- a/docker/telemetry/grafana/dashboards/node-health.json +++ b/docker/telemetry/grafana/dashboards/node-health.json @@ -358,7 +358,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Rate at which a locally built ledger hash fails to match the network-validated hash.*\n\n###### How it's computed:\n*Per-second rate of history-mismatch events over a 5-minute window.*\n\n###### Reading it:\n*Flat at zero is the only healthy reading.*\n\n###### Healthy range:\n*Zero.*\n\n###### Watch for:\n*Any nonzero value indicates consensus divergence or database corruption and warrants immediate investigation.*\n\n###### Keywords:\n- **Ledger history mismatch** *(per node)* \u2014 when a locally built ledger's hash does not match the network-validated hash.\n- **Validated ledger** *(network-wide)* \u2014 a ledger confirmed final by the trusted validator quorum; its contents never change.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementLedgerHistoryMismatch`\n\n###### References:\n[Validated ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-history-mismatch)", + "description": "###### What this is:\n*Rate at which a locally built ledger hash fails to match the network-validated hash.*\n\n###### How it's computed:\n*Per-second rate of history-mismatch events over a 5-minute window.*\n\n###### Reading it:\n*Flat at zero is the only healthy reading.*\n\n###### Healthy range:\n*Zero.*\n\n###### Watch for:\n*Any nonzero value indicates consensus divergence or database corruption and warrants immediate investigation.*\n\n###### Keywords:\n- **Ledger history mismatch** *(per node)* \u2014 when a locally built ledger's hash does not match the network-validated hash.\n- **Validated ledger** *(network-wide)* \u2014 a ledger confirmed final by the trusted validator quorum; its contents never change.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementLedgerHistoryMismatch`\n\n###### References:\n[Validated ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-history-mismatch)", "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", @@ -1099,7 +1099,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*The node's operating mode over time as a colored timeline (Disconnected, Connected, Syncing, Tracking, Full).*\n\n###### How it's computed:\n*Current value of the server-state gauge mapped to a mode name; equal consecutive samples are merged into a single band.*\n\n###### Reading it:\n*A solid green Full band across the window is the goal; other colors mark periods the node was not fully synced.*\n\n###### Healthy range:\n*Continuously Full (green).*\n\n###### Watch for:\n*Bands of Syncing, Connected, or Disconnected, which pinpoint exactly when the node dropped out of Full.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", + "description": "###### What this is:\n*The node's operating mode over time as a colored timeline (Disconnected, Connected, Syncing, Tracking, Full).*\n\n###### How it's computed:\n*Current value of the server-state gauge mapped to a mode name; equal consecutive samples are merged into a single band.*\n\n###### Reading it:\n*A solid green Full band across the window is the goal; other colors mark periods the node was not fully synced.*\n\n###### Healthy range:\n*Continuously Full (green).*\n\n###### Watch for:\n*Bands of Syncing, Connected, or Disconnected, which pinpoint exactly when the node dropped out of Full.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", "fieldConfig": { "defaults": { "color": { @@ -1241,7 +1241,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Object-store read, found and write operation rates. The found series is `node_reads_hit`, which counts fetches that returned an object whatever served them, so it is not a cache-hit count.*\n\n###### How it's computed:\n*Per-second rates of the read, found and write counters, plotted as lines.*\n\n###### Reading it:\n*Steadily rising lines are normal; the slope reflects store activity.*\n\n###### Healthy range:\n*Reads and writes rising smoothly, with found tracking almost all reads on a node that has the data.*\n\n###### Watch for:\n*A sudden surge in reads or writes signals heavy back-end I/O, from sync, replay, or query load.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", + "description": "###### What this is:\n*Object-store read, found and write operation rates. The found series is `node_reads_hit`, which counts fetches that returned an object whatever served them, so it is not a cache-hit count.*\n\n###### How it's computed:\n*Per-second rates of the read, found and write counters, plotted as lines.*\n\n###### Reading it:\n*Steadily rising lines are normal; the slope reflects store activity.*\n\n###### Healthy range:\n*Reads and writes rising smoothly, with found tracking almost all reads on a node that has the data.*\n\n###### Watch for:\n*A sudden surge in reads or writes signals heavy back-end I/O, from sync, replay, or query load.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", "fieldConfig": { "defaults": { "color": { @@ -1360,7 +1360,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Fraction of object-store reads that returned an object. `node_reads_hit` counts fetches that found the object whatever served them, so this is not a cache-hit rate.*\n\n###### How it's computed:\n*Per-second rate of `node_reads_hit` divided by the per-second rate of `node_reads_total`, as a single ratio per node.*\n\n###### Reading it:\n*A value near 1.0 means almost every read finds its object; dips mean reads are missing.*\n\n###### Healthy range:\n*Close to 1.0 on a node that holds the data it is being asked for.*\n\n###### Watch for:\n*A sustained drop means the node is repeatedly asked for objects it does not have, which usually accompanies backfill or a gap in history.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", + "description": "###### What this is:\n*Fraction of object-store reads that returned an object. `node_reads_hit` counts fetches that found the object whatever served them, so this is not a cache-hit rate.*\n\n###### How it's computed:\n*Per-second rate of `node_reads_hit` divided by the per-second rate of `node_reads_total`, as a single ratio per node.*\n\n###### Reading it:\n*A value near 1.0 means almost every read finds its object; dips mean reads are missing.*\n\n###### Healthy range:\n*Close to 1.0 on a node that holds the data it is being asked for.*\n\n###### Watch for:\n*A sustained drop means the node is repeatedly asked for objects it does not have, which usually accompanies backfill or a gap in history.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", "fieldConfig": { "defaults": { "color": { @@ -1463,7 +1463,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Instantaneous write-load score and read-queue depth of the object store.*\n\n###### How it's computed:\n*Current values of the write-load and read-queue gauges, plotted over time.*\n\n###### Reading it:\n*Lower is better for both; short, flat lines are healthy.*\n\n###### Healthy range:\n*Write load near zero and read queue in low double digits or less.*\n\n###### Watch for:\n*High write load means back-end pressure; a high read queue means the prefetch threads are saturated.*\n\n###### Keywords:\n- **Transaction queue (TxQ)** *(per node)* \u2014 holds transactions that meet local cost but not the open-ledger cost, to include in a later ledger.\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n- **Read threads / read queue / write load** *(per node)* \u2014 NodeStore back-end I/O internals \u2014 worker threads reading, their queue depth, and write pressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Transaction queue (TxQ)](https://xrpl.org/docs/concepts/transactions/transaction-queue) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#transaction-queue-txq)", + "description": "###### What this is:\n*Instantaneous write-load score and read-queue depth of the object store.*\n\n###### How it's computed:\n*Current values of the write-load and read-queue gauges, plotted over time.*\n\n###### Reading it:\n*Lower is better for both; short, flat lines are healthy.*\n\n###### Healthy range:\n*Write load near zero and read queue in low double digits or less.*\n\n###### Watch for:\n*High write load means back-end pressure; a high read queue means the prefetch threads are saturated.*\n\n###### Keywords:\n- **Transaction queue (TxQ)** *(per node)* \u2014 holds transactions that meet local cost but not the open-ledger cost, to include in a later ledger.\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n- **Read threads / read queue / write load** *(per node)* \u2014 NodeStore back-end I/O internals \u2014 worker threads reading, their queue depth, and write pressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Transaction queue (TxQ)](https://xrpl.org/docs/concepts/transactions/transaction-queue) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#transaction-queue-txq)", "fieldConfig": { "defaults": { "color": { @@ -1578,7 +1578,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Cumulative bytes read from and written to the object-store back end.*\n\n###### How it's computed:\n*Current values of the bytes-read and bytes-written counters, plotted as lines.*\n\n###### Reading it:\n*Steadily rising lines are normal; the slope shows throughput.*\n\n###### Healthy range:\n*Smooth growth consistent with ledger and query activity.*\n\n###### Watch for:\n*A sharp acceleration indicates a heavy I/O phase such as sync, replay, or large queries.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", + "description": "###### What this is:\n*Cumulative bytes read from and written to the object-store back end.*\n\n###### How it's computed:\n*Current values of the bytes-read and bytes-written counters, plotted as lines.*\n\n###### Reading it:\n*Steadily rising lines are normal; the slope shows throughput.*\n\n###### Healthy range:\n*Smooth growth consistent with ledger and query activity.*\n\n###### Watch for:\n*A sharp acceleration indicates a heavy I/O phase such as sync, replay, or large queries.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", "fieldConfig": { "defaults": { "color": { @@ -1689,7 +1689,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Read-thread utilization, bundled read count, and cumulative read time for the object store.*\n\n###### How it's computed:\n*Current values of the running/total read-thread gauges, read-bundle gauge, and cumulative read-duration counter, plotted as lines.*\n\n###### Reading it:\n*Running threads well below total means spare capacity; a rising duration line reflects time spent in read I/O.*\n\n###### Healthy range:\n*Running threads below the total count most of the time.*\n\n###### Watch for:\n*Running threads pinned at the total for long periods means read I/O is saturated.*\n\n###### Keywords:\n- **Read threads / read queue / write load** *(per node)* \u2014 NodeStore back-end I/O internals \u2014 worker threads reading, their queue depth, and write pressure.\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#read-threads-read-queue-write-load)", + "description": "###### What this is:\n*Read-thread utilization, bundled read count, and cumulative read time for the object store.*\n\n###### How it's computed:\n*Current values of the running/total read-thread gauges, read-bundle gauge, and cumulative read-duration counter, plotted as lines.*\n\n###### Reading it:\n*Running threads well below total means spare capacity; a rising duration line reflects time spent in read I/O.*\n\n###### Healthy range:\n*Running threads below the total count most of the time.*\n\n###### Watch for:\n*Running threads pinned at the total for long periods means read I/O is saturated.*\n\n###### Keywords:\n- **Read threads / read queue / write load** *(per node)* \u2014 NodeStore back-end I/O internals \u2014 worker threads reading, their queue depth, and write pressure.\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#read-threads-read-queue-write-load)", "fieldConfig": { "defaults": { "color": { @@ -1809,7 +1809,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Share of wall-clock time the object store spent inside read I/O.*\n\n###### How it's computed:\n*Per-second rate of the cumulative `node_reads_duration_us` counter, converted from microseconds to seconds, as a single ratio per node.*\n\n###### Reading it:\n*1.0 means the store spent a full second per second in reads; well below 1.0 means spare read capacity.*\n\n###### Healthy range:\n*Below roughly 0.8 in steady state.*\n\n###### Watch for:\n*Sustained values at or above 1.0 mean read I/O is saturated and reads are queueing.*\n\n###### Keywords:\n- **Read threads / read queue / write load** *(per node)* \u2014 NodeStore back-end I/O internals \u2014 worker threads reading, their queue depth, and write pressure.\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#read-threads-read-queue-write-load)", + "description": "###### What this is:\n*Share of wall-clock time the object store spent inside read I/O.*\n\n###### How it's computed:\n*Per-second rate of the cumulative `node_reads_duration_us` counter, converted from microseconds to seconds, as a single ratio per node.*\n\n###### Reading it:\n*1.0 means the store spent a full second per second in reads; well below 1.0 means spare read capacity.*\n\n###### Healthy range:\n*Below roughly 0.8 in steady state.*\n\n###### Watch for:\n*Sustained values at or above 1.0 mean read I/O is saturated and reads are queueing.*\n\n###### Keywords:\n- **Read threads / read queue / write load** *(per node)* \u2014 NodeStore back-end I/O internals \u2014 worker threads reading, their queue depth, and write pressure.\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#read-threads-read-queue-write-load)", "fieldConfig": { "defaults": { "color": { @@ -2610,7 +2610,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Hit rates for the SLE, Ledger, and AcceptedLedger caches, from 0 to 1.*\n\n###### How it's computed:\n*Current values of the per-cache hit-rate gauges, plotted as lines.*\n\n###### Reading it:\n*Higher is better; each line is the fraction of lookups served from cache.*\n\n###### Healthy range:\n*Above roughly 0.8 in steady state.*\n\n###### Watch for:\n*Low or falling hit rates indicate cache thrashing and extra back-end reads.*\n\n###### Keywords:\n- **Caches (SLE / Ledger / TreeNode / FullBelow / AcceptedLedger)** *(per node)* \u2014 in-memory caches that avoid re-reading or re-computing ledger data; higher hit rates mean less back-end work.\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerCacheHitRateGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#caches)", + "description": "###### What this is:\n*Hit rates for the SLE, Ledger, and AcceptedLedger caches, from 0 to 1.*\n\n###### How it's computed:\n*Current values of the per-cache hit-rate gauges, plotted as lines.*\n\n###### Reading it:\n*Higher is better; each line is the fraction of lookups served from cache.*\n\n###### Healthy range:\n*Above roughly 0.8 in steady state.*\n\n###### Watch for:\n*Low or falling hit rates indicate cache thrashing and extra back-end reads.*\n\n###### Keywords:\n- **Caches (SLE / Ledger / TreeNode / FullBelow / AcceptedLedger)** *(per node)* \u2014 in-memory caches that avoid re-reading or re-computing ledger data; higher hit rates mean less back-end work.\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerCacheHitRateGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#caches)", "fieldConfig": { "defaults": { "color": { @@ -2731,7 +2731,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Entry counts for the TreeNode cache and track set, the FullBelow cache, and the AcceptedLedger cache.*\n\n###### How it's computed:\n*Current values of the per-cache size gauges, plotted as lines.*\n\n###### Reading it:\n*Stable lines are normal; sizes grow with working set and shrink after sweeps.*\n\n###### Healthy range:\n*Stable within configured limits.*\n\n###### Watch for:\n*Unbounded growth suggests memory pressure or a cache not being swept.*\n\n###### Keywords:\n- **Caches (SLE / Ledger / TreeNode / FullBelow / AcceptedLedger)** *(per node)* \u2014 in-memory caches that avoid re-reading or re-computing ledger data; higher hit rates mean less back-end work.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerCacheHitRateGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#caches)", + "description": "###### What this is:\n*Entry counts for the TreeNode cache and track set, the FullBelow cache, and the AcceptedLedger cache.*\n\n###### How it's computed:\n*Current values of the per-cache size gauges, plotted as lines.*\n\n###### Reading it:\n*Stable lines are normal; sizes grow with working set and shrink after sweeps.*\n\n###### Healthy range:\n*Stable within configured limits.*\n\n###### Watch for:\n*Unbounded growth suggests memory pressure or a cache not being swept.*\n\n###### Keywords:\n- **Caches (SLE / Ledger / TreeNode / FullBelow / AcceptedLedger)** *(per node)* \u2014 in-memory caches that avoid re-reading or re-computing ledger data; higher hit rates mean less back-end work.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerCacheHitRateGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#caches)", "fieldConfig": { "defaults": { "color": { @@ -3060,7 +3060,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Which sync state each node is in right now: Disconnected, Connected, Syncing, Tracking or Full. Only a Full node holds the current validated ledger and can answer authoritatively.*\n\n###### How it's computed:\n*The `server_state` gauge, the node's operating mode as an integer 0-4, value-mapped to a name and colour: 0 Disconnected (red), 1 Connected (yellow), 2 Syncing (orange), 3 Tracking (blue), 4 Full (green). No rate or aggregation \u2014 it is the instantaneous state.*\n\n###### Reading it:\n*One tile per node. Green FULL is the steady state; any other colour says the node is not yet serving the current ledger and how far along it is.*\n\n###### Healthy range:\n*FULL on every node.*\n\n###### Watch for:\n*A node leaving FULL and staying out, or flapping between TRACKING and FULL \u2014 that points at ledger acquisition falling behind rather than a connectivity fault. Cross-check Operating Mode (Time Share) and Validated Ledger Age.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", + "description": "###### What this is:\n*Which sync state each node is in right now: Disconnected, Connected, Syncing, Tracking or Full. Only a Full node holds the current validated ledger and can answer authoritatively.*\n\n###### How it's computed:\n*The `server_state` gauge, the node's operating mode as an integer 0-4, value-mapped to a name and colour: 0 Disconnected (red), 1 Connected (yellow), 2 Syncing (orange), 3 Tracking (blue), 4 Full (green). No rate or aggregation \u2014 it is the instantaneous state.*\n\n###### Reading it:\n*One tile per node. Green FULL is the steady state; any other colour says the node is not yet serving the current ledger and how far along it is.*\n\n###### Healthy range:\n*FULL on every node.*\n\n###### Watch for:\n*A node leaving FULL and staying out, or flapping between TRACKING and FULL \u2014 that points at ledger acquisition falling behind rather than a connectivity fault. Cross-check Operating Mode (Time Share) and Validated Ledger Age.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", "fieldConfig": { "defaults": { "color": { @@ -3236,7 +3236,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*How long the server process has been running, in seconds.*\n\n###### How it's computed:\n*Current value of the uptime gauge.*\n\n###### Reading it:\n*Higher is generally better; a reset to a small value means the process restarted.*\n\n###### Healthy range:\n*Continuously increasing.*\n\n###### Watch for:\n*An unexpected drop to near zero indicates a restart or crash.*\n\n###### Keywords:\n- **Uptime** *(per node)* \u2014 seconds since the server process started.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md)", + "description": "###### What this is:\n*How long the server process has been running, in seconds.*\n\n###### How it's computed:\n*Current value of the uptime gauge.*\n\n###### Reading it:\n*Higher is generally better; a reset to a small value means the process restarted.*\n\n###### Healthy range:\n*Continuously increasing.*\n\n###### Watch for:\n*An unexpected drop to near zero indicates a restart or crash.*\n\n###### Keywords:\n- **Uptime** *(per node)* \u2014 seconds since the server process started.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md)", "fieldConfig": { "defaults": { "color": { @@ -3301,7 +3301,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Total connected peers, inbound plus outbound.*\n\n###### How it's computed:\n*Current value of the peer-count gauge.*\n\n###### Reading it:\n*A stable count in the healthy range is good; too few limits connectivity.*\n\n###### Healthy range:\n*Workload- and config-dependent, typically 10 or more.*\n\n###### Watch for:\n*A sudden drop points to network or connectivity problems; an unusually high inbound count can indicate connection-flood pressure.*\n\n###### Keywords:\n- **Peer** *(per node)* \u2014 another server this node holds a protocol connection to.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md)", + "description": "###### What this is:\n*Total connected peers, inbound plus outbound.*\n\n###### How it's computed:\n*Current value of the peer-count gauge.*\n\n###### Reading it:\n*A stable count in the healthy range is good; too few limits connectivity.*\n\n###### Healthy range:\n*Workload- and config-dependent, typically 10 or more.*\n\n###### Watch for:\n*A sudden drop points to network or connectivity problems; an unusually high inbound count can indicate connection-flood pressure.*\n\n###### Keywords:\n- **Peer** *(per node)* \u2014 another server this node holds a protocol connection to.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md)", "fieldConfig": { "defaults": { "color": { @@ -3375,7 +3375,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Sequence number of the current open ledger.*\n\n###### How it's computed:\n*Current value of the open-ledger index gauge.*\n\n###### Reading it:\n*Should climb steadily; the gap above the validated sequence is the ledgers in flight.*\n\n###### Healthy range:\n*One or two ahead of the validated sequence.*\n\n###### Watch for:\n*A large or growing gap above the validated sequence means validation is lagging behind ledger creation.*\n\n###### Keywords:\n- **Ledger index** *(network-wide)* \u2014 the sequence number identifying a ledger version; increases by one each close.\n- **Open ledger** *(per node)* \u2014 the temporary workspace ledger where incoming transactions are provisionally applied before a close.\n- **Validated ledger** *(network-wide)* \u2014 a ledger confirmed final by the trusted validator quorum; its contents never change.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Ledger index](https://xrpl.org/docs/references/protocol/data-types/basic-data-types#ledger-index) \u00b7 [Open ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-index)", + "description": "###### What this is:\n*Sequence number of the current open ledger.*\n\n###### How it's computed:\n*Current value of the open-ledger index gauge.*\n\n###### Reading it:\n*Should climb steadily; the gap above the validated sequence is the ledgers in flight.*\n\n###### Healthy range:\n*One or two ahead of the validated sequence.*\n\n###### Watch for:\n*A large or growing gap above the validated sequence means validation is lagging behind ledger creation.*\n\n###### Keywords:\n- **Ledger index** *(network-wide)* \u2014 the sequence number identifying a ledger version; increases by one each close.\n- **Open ledger** *(per node)* \u2014 the temporary workspace ledger where incoming transactions are provisionally applied before a close.\n- **Validated ledger** *(network-wide)* \u2014 a ledger confirmed final by the trusted validator quorum; its contents never change.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Ledger index](https://xrpl.org/docs/references/protocol/data-types/basic-data-types#ledger-index) \u00b7 [Open ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-index)", "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", @@ -3440,7 +3440,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*The spread in validated ledger sequence across all selected nodes.*\n\n###### How it's computed:\n*Highest validated ledger sequence minus the lowest, among the selected nodes on the same network.*\n\n###### Reading it:\n*0 means every node agrees on the same validated ledger; larger means they diverge.*\n\n###### Healthy range:\n*0 to 1 ledger in steady state.*\n\n###### Watch for:\n*A sustained spread above a few ledgers means some nodes are lagging or the fleet is diverging.*\n\n###### Keywords:\n- **Validated ledger** *(network-wide)* \u2014 a ledger confirmed final by the trusted validator quorum; its contents never change.\n\n###### Computation boundary:\n*Result: Per network \u2014 the query aggregates the selected nodes into one series for each `xrpl_network_type`, so mainnet and devnet sequences are never subtracted from each other.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Validated ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validated-ledger)", + "description": "###### What this is:\n*The spread in validated ledger sequence across all selected nodes.*\n\n###### How it's computed:\n*Highest validated ledger sequence minus the lowest, among the selected nodes on the same network.*\n\n###### Reading it:\n*0 means every node agrees on the same validated ledger; larger means they diverge.*\n\n###### Healthy range:\n*0 to 1 ledger in steady state.*\n\n###### Watch for:\n*A sustained spread above a few ledgers means some nodes are lagging or the fleet is diverging.*\n\n###### Keywords:\n- **Validated ledger** *(network-wide)* \u2014 a ledger confirmed final by the trusted validator quorum; its contents never change.\n\n###### Computation boundary:\n*Result: Per network \u2014 the query aggregates the selected nodes into one series for each `xrpl_network_type`, so mainnet and devnet sequences are never subtracted from each other.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Validated ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validated-ledger)", "fieldConfig": { "defaults": { "color": { @@ -3542,7 +3542,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*How far each node's validated ledger lags behind the network tip, in ledgers.*\n\n###### How it's computed:\n*Highest validated ledger sequence within the node's own network, minus each node's own sequence.*\n\n###### Reading it:\n*0 means the node is at the tip; larger values mean it trails further behind.*\n\n###### Healthy range:\n*0 to 1 ledger on a synced node.*\n\n###### Watch for:\n*A node stuck at a growing value is falling behind and not keeping up with consensus.*\n\n###### Keywords:\n- **Validated ledger** *(network-wide)* \u2014 a ledger confirmed final by the trusted validator quorum; its contents never change.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each node is compared against the highest sequence on its own `xrpl_network_type`, so mainnet and devnet sequences are never subtracted from each other.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Validated ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validated-ledger)", + "description": "###### What this is:\n*How far each node's validated ledger lags behind the network tip, in ledgers.*\n\n###### How it's computed:\n*Highest validated ledger sequence within the node's own network, minus each node's own sequence.*\n\n###### Reading it:\n*0 means the node is at the tip; larger values mean it trails further behind.*\n\n###### Healthy range:\n*0 to 1 ledger on a synced node.*\n\n###### Watch for:\n*A node stuck at a growing value is falling behind and not keeping up with consensus.*\n\n###### Keywords:\n- **Validated ledger** *(network-wide)* \u2014 a ledger confirmed final by the trusted validator quorum; its contents never change.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each node is compared against the highest sequence on its own `xrpl_network_type`, so mainnet and devnet sequences are never subtracted from each other.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Validated ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validated-ledger)", "fieldConfig": { "defaults": { "color": { @@ -3611,7 +3611,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*The running server's build version string.*\n\n###### How it's computed:\n*Read from the version label of the build-info metric (its value is always 1).*\n\n###### Reading it:\n*Confirms which version each node is running.*\n\n###### Healthy range:\n*The expected release version across all nodes.*\n\n###### Watch for:\n*A node on an unexpected or mismatched version in a fleet.*\n\n###### Keywords:\n- **Build version** *(per node)* \u2014 the xrpld release the process is running.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerBuildInfoGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md)", + "description": "###### What this is:\n*The running server's build version string.*\n\n###### How it's computed:\n*Read from the version label of the build-info metric (its value is always 1).*\n\n###### Reading it:\n*Confirms which version each node is running.*\n\n###### Healthy range:\n*The expected release version across all nodes.*\n\n###### Watch for:\n*A node on an unexpected or mismatched version in a fleet.*\n\n###### Keywords:\n- **Build version** *(per node)* \u2014 the xrpld release the process is running.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerBuildInfoGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md)", "fieldConfig": { "defaults": { "thresholds": { @@ -3675,7 +3675,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Proposer count and convergence time from the last closed consensus round.*\n\n###### How it's computed:\n*Current values of the last-close proposer-count and convergence-time gauges, plotted as lines.*\n\n###### Reading it:\n*A healthy proposer count with a low convergence time is good.*\n\n###### Healthy range:\n*Convergence time of a few seconds with the expected number of proposers.*\n\n###### Watch for:\n*A falling proposer count or rising convergence time signals degrading consensus conditions.*\n\n###### Keywords:\n- **Proposers** *(network event)* \u2014 the count of validators whose proposals this node heard in the last closed round.\n- **Convergence time** *(network event)* \u2014 the wall-clock time the network took to agree a ledger in a round.\n- **Consensus round** *(network event)* \u2014 one propose-and-revise iteration of consensus; several may run before validators converge on a ledger.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Proposers](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#proposers)", + "description": "###### What this is:\n*Proposer count and convergence time from the last closed consensus round.*\n\n###### How it's computed:\n*Current values of the last-close proposer-count and convergence-time gauges, plotted as lines.*\n\n###### Reading it:\n*A healthy proposer count with a low convergence time is good.*\n\n###### Healthy range:\n*Convergence time of a few seconds with the expected number of proposers.*\n\n###### Watch for:\n*A falling proposer count or rising convergence time signals degrading consensus conditions.*\n\n###### Keywords:\n- **Proposers** *(network event)* \u2014 the count of validators whose proposals this node heard in the last closed round.\n- **Convergence time** *(network event)* \u2014 the wall-clock time the network took to agree a ledger in a round.\n- **Consensus round** *(network event)* \u2014 one propose-and-revise iteration of consensus; several may run before validators converge on a ledger.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Proposers](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#proposers)", "fieldConfig": { "defaults": { "color": { @@ -3774,7 +3774,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Proposer count and convergence time from the last closed consensus round.*\n\n###### How it's computed:\n*Current values of the last-close proposer-count and convergence-time gauges, plotted as lines.*\n\n###### Reading it:\n*A healthy proposer count with a low convergence time is good.*\n\n###### Healthy range:\n*Convergence time of a few seconds with the expected number of proposers.*\n\n###### Watch for:\n*A falling proposer count or rising convergence time signals degrading consensus conditions.*\n\n###### Keywords:\n- **Proposers** *(network event)* \u2014 the count of validators whose proposals this node heard in the last closed round.\n- **Convergence time** *(network event)* \u2014 the wall-clock time the network took to agree a ledger in a round.\n- **Consensus round** *(network event)* \u2014 one propose-and-revise iteration of consensus; several may run before validators converge on a ledger.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Proposers](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#proposers)", + "description": "###### What this is:\n*Proposer count and convergence time from the last closed consensus round.*\n\n###### How it's computed:\n*Current values of the last-close proposer-count and convergence-time gauges, plotted as lines.*\n\n###### Reading it:\n*A healthy proposer count with a low convergence time is good.*\n\n###### Healthy range:\n*Convergence time of a few seconds with the expected number of proposers.*\n\n###### Watch for:\n*A falling proposer count or rising convergence time signals degrading consensus conditions.*\n\n###### Keywords:\n- **Proposers** *(network event)* \u2014 the count of validators whose proposals this node heard in the last closed round.\n- **Convergence time** *(network event)* \u2014 the wall-clock time the network took to agree a ledger in a round.\n- **Consensus round** *(network event)* \u2014 one propose-and-revise iteration of consensus; several may run before validators converge on a ledger.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Proposers](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#proposers)", "fieldConfig": { "defaults": { "color": { @@ -3874,7 +3874,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*The wall-clock time between consecutive ledger closes \u2014 the network close cadence.*\n\n###### How it's computed:\n*Close Interval: 1 / rate(ledgers_closed_total), the average seconds between closes from the monotonic close counter (scrape-independent, unlike a gauge delta which would alias to the scrape period). Last-Close Age: time() minus the last-close network time (server_info last_close_time gauge + Ripple-epoch offset), i.e. seconds since the last ledger closed.*\n\n###### Reading it:\n*A steady line near the network's target close interval.*\n\n###### Healthy range:\n*About 3-5s on mainnet; workload-dependent on test networks.*\n\n###### Watch for:\n*A rising interval (consensus slowing or the node lagging) or a flat line at zero (ledgers no longer closing).*\n\n###### Keywords:\n- **Ledger close** *(network event)* \u2014 the current open ledger is closed and a new closed ledger is built from the agreed transaction set.\n- **Ledger close interval** *(network-wide)* \u2014 the network's steady ledger rhythm \u2014 roughly one closed ledger every 3-5 seconds on Mainnet.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp) \u00b7 [RCLConsensus.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/consensus/RCLConsensus.cpp)\n\n###### Function:\n`registerServerInfoGauge (last_close_time) ; ledgers_closed_total`\n\n###### References:\n[Ledger close](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Ledger close interval](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-close)", + "description": "###### What this is:\n*The wall-clock time between consecutive ledger closes \u2014 the network close cadence.*\n\n###### How it's computed:\n*Close Interval: 1 / rate(ledgers_closed_total), the average seconds between closes from the monotonic close counter (scrape-independent, unlike a gauge delta which would alias to the scrape period). Last-Close Age: time() minus the last-close network time (server_info last_close_time gauge + Ripple-epoch offset), i.e. seconds since the last ledger closed.*\n\n###### Reading it:\n*A steady line near the network's target close interval.*\n\n###### Healthy range:\n*About 3-5s on mainnet; workload-dependent on test networks.*\n\n###### Watch for:\n*A rising interval (consensus slowing or the node lagging) or a flat line at zero (ledgers no longer closing).*\n\n###### Keywords:\n- **Ledger close** *(network event)* \u2014 the current open ledger is closed and a new closed ledger is built from the agreed transaction set.\n- **Ledger close interval** *(network-wide)* \u2014 the network's steady ledger rhythm \u2014 roughly one closed ledger every 3-5 seconds on Mainnet.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp) \u00b7 [MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp) \u00b7 [RCLConsensus.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/consensus/RCLConsensus.cpp)\n\n###### Function:\n`registerServerInfoGauge (last_close_time) ; ledgers_closed_total`\n\n###### References:\n[Ledger close](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Ledger close interval](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-close)", "fieldConfig": { "defaults": { "color": { @@ -4019,7 +4019,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Cumulative object-payload bytes this process has written to the NodeStore back end. This is not the size of the store on disk.*\n\n###### How it's computed:\n*Current value of the stored_object_bytes gauge, plotted over time. It observes getStoreSize(), the same accessor node_written_bytes uses, so the two series are equal and their ratio is a constant 1.0 rather than a write-amplification measure.*\n\n###### Reading it:\n*A smoothly growing line is normal; the slope is the write rate. It excludes NuDB's keys, bucket padding and log, and it restarts from zero with the process while the files on disk do not.*\n\n###### Healthy range:\n*Gradual growth consistent with ledger data being stored.*\n\n###### Watch for:\n*A sudden jump in growth rate can indicate runaway storage or an unexpected back-fill. Do not use this panel to size the store on disk or to plan disk capacity; no metric reports on-disk size today, so check the filesystem directly.*\n\n###### Keywords:\n- **NuDB** *(per node)* \u2014 the append-only key-value database used as the default NodeStore backend.\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n- **Back-fill / catch-up** *(per node)* \u2014 fetching missing historical ledgers from peers to fill gaps or reach the network tip.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerStorageDetailGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nudb)", + "description": "###### What this is:\n*Cumulative object-payload bytes this process has written to the NodeStore back end. This is not the size of the store on disk.*\n\n###### How it's computed:\n*Current value of the stored_object_bytes gauge, plotted over time. It observes getStoreSize(), the same accessor node_written_bytes uses, so the two series are equal and their ratio is a constant 1.0 rather than a write-amplification measure.*\n\n###### Reading it:\n*A smoothly growing line is normal; the slope is the write rate. It excludes NuDB's keys, bucket padding and log, and it restarts from zero with the process while the files on disk do not.*\n\n###### Healthy range:\n*Gradual growth consistent with ledger data being stored.*\n\n###### Watch for:\n*A sudden jump in growth rate can indicate runaway storage or an unexpected back-fill. Do not use this panel to size the store on disk or to plan disk capacity; no metric reports on-disk size today, so check the filesystem directly.*\n\n###### Keywords:\n- **NuDB** *(per node)* \u2014 the append-only key-value database used as the default NodeStore backend.\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n- **Back-fill / catch-up** *(per node)* \u2014 fetching missing historical ledgers from peers to fill gaps or reach the network tip.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerStorageDetailGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nudb)", "fieldConfig": { "defaults": { "color": { @@ -4118,7 +4118,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Live instance counts for the busiest internal object types.*\n\n###### How it's computed:\n*Current per-type instance counts, showing the top 15 types over time.*\n\n###### Reading it:\n*Stable lines are healthy; each line is one object type's live count.*\n\n###### Healthy range:\n*Steady counts that rise and fall with load.*\n\n###### Watch for:\n*A single type climbing without bound suggests memory pressure or a leak.*\n\n###### Keywords:\n- **Object instance count** *(per node)* \u2014 live in-memory instances of a tracked C++ type, used to spot leaks.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerObjectCountGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md)", + "description": "###### What this is:\n*Live instance counts for the busiest internal object types.*\n\n###### How it's computed:\n*Current per-type instance counts, showing the top 15 types over time.*\n\n###### Reading it:\n*Stable lines are healthy; each line is one object type's live count.*\n\n###### Healthy range:\n*Steady counts that rise and fall with load.*\n\n###### Watch for:\n*A single type climbing without bound suggests memory pressure or a leak.*\n\n###### Keywords:\n- **Object instance count** *(per node)* \u2014 live in-memory instances of a tracked C++ type, used to spot leaks.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerObjectCountGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md)", "fieldConfig": { "defaults": { "color": { @@ -4234,7 +4234,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*How many historical ledgers the node is back-filling per minute.*\n\n###### How it's computed:\n*Current value of the historical-fetch-per-minute gauge.*\n\n###### Reading it:\n*Near zero once history is complete; elevated while back-filling.*\n\n###### Healthy range:\n*Close to zero in steady state.*\n\n###### Watch for:\n*A sustained high rate means the node is still filling gaps in its stored history.*\n\n###### Keywords:\n- **Historical fetch rate** *(per node)* \u2014 how many historical ledgers the node is back-filling per minute.\n- **Back-fill / catch-up** *(per node)* \u2014 fetching missing historical ledgers from peers to fill gaps or reach the network tip.\n- **Complete ledger ranges** *(per node)* \u2014 the contiguous spans of ledgers the node holds locally; one unbroken range is ideal.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerDbMetricsGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#historical-fetch-rate)", + "description": "###### What this is:\n*How many historical ledgers the node is back-filling per minute.*\n\n###### How it's computed:\n*Current value of the historical-fetch-per-minute gauge.*\n\n###### Reading it:\n*Near zero once history is complete; elevated while back-filling.*\n\n###### Healthy range:\n*Close to zero in steady state.*\n\n###### Watch for:\n*A sustained high rate means the node is still filling gaps in its stored history.*\n\n###### Keywords:\n- **Historical fetch rate** *(per node)* \u2014 how many historical ledgers the node is back-filling per minute.\n- **Back-fill / catch-up** *(per node)* \u2014 fetching missing historical ledgers from peers to fill gaps or reach the network tip.\n- **Complete ledger ranges** *(per node)* \u2014 the contiguous spans of ledgers the node holds locally; one unbroken range is ideal.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerDbMetricsGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#historical-fetch-rate)", "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", @@ -4303,7 +4303,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*The contiguous ranges of ledgers the node holds locally.*\n\n###### How it's computed:\n*Current start and end bounds of each complete range, listed as table rows.*\n\n###### Reading it:\n*Fewer ranges is better; one continuous range means an unbroken history.*\n\n###### Healthy range:\n*A single range covering the configured retention window.*\n\n###### Watch for:\n*Many fragmented ranges indicate gaps in stored history from missed or failed fetches.*\n\n###### Keywords:\n- **Complete ledger ranges** *(per node)* \u2014 the contiguous spans of ledgers the node holds locally; one unbroken range is ideal.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerCompleteLedgersGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#complete-ledger-ranges)", + "description": "###### What this is:\n*The contiguous ranges of ledgers the node holds locally.*\n\n###### How it's computed:\n*Current start and end bounds of each complete range, listed as table rows.*\n\n###### Reading it:\n*Fewer ranges is better; one continuous range means an unbroken history.*\n\n###### Healthy range:\n*A single range covering the configured retention window.*\n\n###### Watch for:\n*Many fragmented ranges indicate gaps in stored history from missed or failed fetches.*\n\n###### Keywords:\n- **Complete ledger ranges** *(per node)* \u2014 the contiguous spans of ledgers the node holds locally; one unbroken range is ideal.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerCompleteLedgersGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#complete-ledger-ranges)", "fieldConfig": { "defaults": { "custom": { @@ -4369,7 +4369,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Sizes of the relational databases in KB (total, ledger, transaction).*\n\n###### How it's computed:\n*Current values of the per-database size gauges, plotted as lines.*\n\n###### Reading it:\n*Smoothly growing lines are normal; the split shows where storage is used.*\n\n###### Healthy range:\n*Gradual growth consistent with retained history.*\n\n###### Watch for:\n*An abrupt change in growth rate can indicate storage pressure or a pruning issue.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerDbMetricsGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", + "description": "###### What this is:\n*Sizes of the relational databases in KB (total, ledger, transaction).*\n\n###### How it's computed:\n*Current values of the per-database size gauges, plotted as lines.*\n\n###### Reading it:\n*Smoothly growing lines are normal; the split shows where storage is used.*\n\n###### Healthy range:\n*Gradual growth consistent with retained history.*\n\n###### Watch for:\n*An abrupt change in growth rate can indicate storage pressure or a pruning issue.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerDbMetricsGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", "fieldConfig": { "defaults": { "color": { @@ -4488,7 +4488,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Cumulative count of peers disconnected for exceeding resource limits.*\n\n###### How it's computed:\n*Current value of the resource-disconnect gauge, plotted over time.*\n\n###### Reading it:\n*A flat line is healthy; steps up mean peers were dropped for overuse.*\n\n###### Healthy range:\n*Flat or very slowly rising.*\n\n###### Watch for:\n*A rising line indicates peers are being throttled off, consistent with abusive or misbehaving peers.*\n\n###### Keywords:\n- **Resource disconnect** *(per node)* \u2014 a peer dropped for exceeding resource/load limits \u2014 the node shedding abusive or overactive peers.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#resource-disconnect)", + "description": "###### What this is:\n*Cumulative count of peers disconnected for exceeding resource limits.*\n\n###### How it's computed:\n*Current value of the resource-disconnect gauge, plotted over time.*\n\n###### Reading it:\n*A flat line is healthy; steps up mean peers were dropped for overuse.*\n\n###### Healthy range:\n*Flat or very slowly rising.*\n\n###### Watch for:\n*A rising line indicates peers are being throttled off, consistent with abusive or misbehaving peers.*\n\n###### Keywords:\n- **Resource disconnect** *(per node)* \u2014 a peer dropped for exceeding resource/load limits \u2014 the node shedding abusive or overactive peers.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#resource-disconnect)", "fieldConfig": { "defaults": { "color": { @@ -4604,7 +4604,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*The node's local load-based fee factor that scales its minimum transaction cost; baseline 256 at idle.*\n\n###### How it's computed:\n*Current value of the local load-fee economy gauge.*\n\n###### Reading it:\n*Steady at the baseline (256) is normal; higher values mean the node is raising its fee in response to load.*\n\n###### Healthy range:\n*Around 256 (the normal baseline) when idle.*\n\n###### Watch for:\n*A climbing factor, which indicates the node is under transaction load pressure.*\n\n###### Keywords:\n- **Load factor** *(per node)* \u2014 a unitless multiplier (1.0 = no load) that scales the base transaction cost as the node comes under load.\n- **Base fee** *(network-wide)* \u2014 the baseline transaction cost for a reference transaction under minimum load, in drops.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerLedgerEconomyGauge`\n\n###### References:\n[Load factor](https://xrpl.org/docs/concepts/transactions/transaction-cost#local-load-cost) \u00b7 [Base fee](https://xrpl.org/docs/concepts/transactions/transaction-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#load-factor)", + "description": "###### What this is:\n*The node's local load-based fee factor that scales its minimum transaction cost; baseline 256 at idle.*\n\n###### How it's computed:\n*Current value of the local load-fee economy gauge.*\n\n###### Reading it:\n*Steady at the baseline (256) is normal; higher values mean the node is raising its fee in response to load.*\n\n###### Healthy range:\n*Around 256 (the normal baseline) when idle.*\n\n###### Watch for:\n*A climbing factor, which indicates the node is under transaction load pressure.*\n\n###### Keywords:\n- **Load factor** *(per node)* \u2014 a unitless multiplier (1.0 = no load) that scales the base transaction cost as the node comes under load.\n- **Base fee** *(network-wide)* \u2014 the baseline transaction cost for a reference transaction under minimum load, in drops.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerLedgerEconomyGauge`\n\n###### References:\n[Load factor](https://xrpl.org/docs/concepts/transactions/transaction-cost#local-load-cost) \u00b7 [Base fee](https://xrpl.org/docs/concepts/transactions/transaction-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#load-factor)", "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", @@ -4673,7 +4673,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*The minimum XRP balance required to keep an account on the ledger, in drops.*\n\n###### How it's computed:\n*Current value of the reserve-base economy gauge.*\n\n###### Reading it:\n*A stable value is expected; it changes only via network amendment or vote.*\n\n###### Healthy range:\n*The configured network reserve base.*\n\n###### Watch for:\n*An unexpected change outside a known amendment or fee vote.*\n\n###### Keywords:\n- **drops** *(network-wide)* \u2014 the smallest XRP unit \u2014 one drop is 0.000001 XRP (one millionth).\n- **Reserve (base & owner)** *(network-wide)* \u2014 the minimum XRP an account must hold \u2014 a base reserve plus an increment per owned ledger object.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerLedgerEconomyGauge`\n\n###### References:\n[drops](https://xrpl.org/docs/references/protocol/data-types/basic-data-types#specifying-currency-amounts) \u00b7 [Reserve (base & owner)](https://xrpl.org/docs/concepts/accounts/reserves) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#drops)", + "description": "###### What this is:\n*The minimum XRP balance required to keep an account on the ledger, in drops.*\n\n###### How it's computed:\n*Current value of the reserve-base economy gauge.*\n\n###### Reading it:\n*A stable value is expected; it changes only via network amendment or vote.*\n\n###### Healthy range:\n*The configured network reserve base.*\n\n###### Watch for:\n*An unexpected change outside a known amendment or fee vote.*\n\n###### Keywords:\n- **drops** *(network-wide)* \u2014 the smallest XRP unit \u2014 one drop is 0.000001 XRP (one millionth).\n- **Reserve (base & owner)** *(network-wide)* \u2014 the minimum XRP an account must hold \u2014 a base reserve plus an increment per owned ledger object.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerLedgerEconomyGauge`\n\n###### References:\n[drops](https://xrpl.org/docs/references/protocol/data-types/basic-data-types#specifying-currency-amounts) \u00b7 [Reserve (base & owner)](https://xrpl.org/docs/concepts/accounts/reserves) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#drops)", "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", @@ -4742,7 +4742,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*The additional XRP reserve required per owned ledger object, in drops.*\n\n###### How it's computed:\n*Current value of the reserve-increment economy gauge.*\n\n###### Reading it:\n*A stable value is expected; it changes only via network amendment or vote.*\n\n###### Healthy range:\n*The configured network owner reserve increment.*\n\n###### Watch for:\n*An unexpected change outside a known amendment or fee vote.*\n\n###### Keywords:\n- **Reserve (base & owner)** *(network-wide)* \u2014 the minimum XRP an account must hold \u2014 a base reserve plus an increment per owned ledger object.\n- **drops** *(network-wide)* \u2014 the smallest XRP unit \u2014 one drop is 0.000001 XRP (one millionth).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerLedgerEconomyGauge`\n\n###### References:\n[Reserve (base & owner)](https://xrpl.org/docs/concepts/accounts/reserves) \u00b7 [drops](https://xrpl.org/docs/references/protocol/data-types/basic-data-types#specifying-currency-amounts) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#reserve-base-owner)", + "description": "###### What this is:\n*The additional XRP reserve required per owned ledger object, in drops.*\n\n###### How it's computed:\n*Current value of the reserve-increment economy gauge.*\n\n###### Reading it:\n*A stable value is expected; it changes only via network amendment or vote.*\n\n###### Healthy range:\n*The configured network owner reserve increment.*\n\n###### Watch for:\n*An unexpected change outside a known amendment or fee vote.*\n\n###### Keywords:\n- **Reserve (base & owner)** *(network-wide)* \u2014 the minimum XRP an account must hold \u2014 a base reserve plus an increment per owned ledger object.\n- **drops** *(network-wide)* \u2014 the smallest XRP unit \u2014 one drop is 0.000001 XRP (one millionth).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerLedgerEconomyGauge`\n\n###### References:\n[Reserve (base & owner)](https://xrpl.org/docs/concepts/accounts/reserves) \u00b7 [drops](https://xrpl.org/docs/references/protocol/data-types/basic-data-types#specifying-currency-amounts) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#reserve-base-owner)", "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", @@ -4811,7 +4811,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Seconds since the last validated ledger closed, plotted over time.*\n\n###### How it's computed:\n*Current value of the ledger-age economy gauge, sampled each interval.*\n\n###### Reading it:\n*Lower is better; it should stay near the ledger close interval. Mirrors the Validated Ledger Age panel.*\n\n###### Healthy range:\n*Under about 10 seconds.*\n\n###### Watch for:\n*Growth beyond the expected close interval, meaning the node is not keeping up with validated ledgers.*\n\n###### Keywords:\n- **Validated ledger** *(network-wide)* \u2014 a ledger confirmed final by the trusted validator quorum; its contents never change.\n- **Ledger close interval** *(network-wide)* \u2014 the network's steady ledger rhythm \u2014 roughly one closed ledger every 3-5 seconds on Mainnet.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerLedgerEconomyGauge`\n\n###### References:\n[Validated ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Ledger close interval](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validated-ledger)", + "description": "###### What this is:\n*Seconds since the last validated ledger closed, plotted over time.*\n\n###### How it's computed:\n*Current value of the ledger-age economy gauge, sampled each interval.*\n\n###### Reading it:\n*Lower is better; it should stay near the ledger close interval. Mirrors the Validated Ledger Age panel.*\n\n###### Healthy range:\n*Under about 10 seconds.*\n\n###### Watch for:\n*Growth beyond the expected close interval, meaning the node is not keeping up with validated ledgers.*\n\n###### Keywords:\n- **Validated ledger** *(network-wide)* \u2014 a ledger confirmed final by the trusted validator quorum; its contents never change.\n- **Ledger close interval** *(network-wide)* \u2014 the network's steady ledger rhythm \u2014 roughly one closed ledger every 3-5 seconds on Mainnet.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerLedgerEconomyGauge`\n\n###### References:\n[Validated ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Ledger close interval](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validated-ledger)", "fieldConfig": { "defaults": { "color": { @@ -4918,7 +4918,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*The network transaction throughput reported by the ledger economy metrics.*\n\n###### How it's computed:\n*Current value of the transaction-rate economy gauge, plotted over time.*\n\n###### Reading it:\n*Reflects how many transactions are being processed; higher means busier.*\n\n###### Healthy range:\n*Workload-dependent.*\n\n###### Watch for:\n*A sudden sustained surge can indicate a transaction flood; a drop to zero can indicate the node stopped processing.*\n\n###### Keywords:\n- **Transaction type** *(network-wide)* \u2014 the kind of transaction (Payment, OfferCreate, TrustSet, AMM*, NFToken*, etc.), used as a breakdown dimension.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerLedgerEconomyGauge`\n\n###### References:\n[Transaction type](https://xrpl.org/docs/references/protocol/transactions/types) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#transaction-type)", + "description": "###### What this is:\n*The network transaction throughput reported by the ledger economy metrics.*\n\n###### How it's computed:\n*Current value of the transaction-rate economy gauge, plotted over time.*\n\n###### Reading it:\n*Reflects how many transactions are being processed; higher means busier.*\n\n###### Healthy range:\n*Workload-dependent.*\n\n###### Watch for:\n*A sudden sustained surge can indicate a transaction flood; a drop to zero can indicate the node stopped processing.*\n\n###### Keywords:\n- **Transaction type** *(network-wide)* \u2014 the kind of transaction (Payment, OfferCreate, TrustSet, AMM*, NFToken*, etc.), used as a breakdown dimension.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerLedgerEconomyGauge`\n\n###### References:\n[Transaction type](https://xrpl.org/docs/references/protocol/transactions/types) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#transaction-type)", "fieldConfig": { "defaults": { "color": { diff --git a/docker/telemetry/grafana/dashboards/peer-network.json b/docker/telemetry/grafana/dashboards/peer-network.json index fc2f98b6d4..3788de3be6 100644 --- a/docker/telemetry/grafana/dashboards/peer-network.json +++ b/docker/telemetry/grafana/dashboards/peer-network.json @@ -257,7 +257,7 @@ }, { "title": "Reduce-Relay Peer Selection", - "description": "###### What this is:\n*How transaction relay picks peers: chosen relay sources, suppressed peers, and peers with the feature off.*\n\n###### How it's computed:\n*Current peer counts in each category (selected, suppressed, not-enabled), per node.*\n\n###### Reading it:\n*A high suppressed-to-selected ratio means relay is saving bandwidth as intended.*\n\n###### Healthy range:\n*Workload-dependent; suppressed should exceed selected in a well-connected mesh.*\n\n###### Watch for:\n*A large not-enabled count (older peers forcing full relay) or selected climbing while suppressed falls.*\n\n###### Keywords:\n- **Reduce-relay** *(per node)* \u2014 an optimization that relays messages through selected peers only, suppressing redundant forwarding.\n- **Transaction suppression** *(per node)* \u2014 dropping a transaction already seen from another peer, so it is not reprocessed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerReduceRelayGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#reduce-relay)", + "description": "###### What this is:\n*How transaction relay picks peers: chosen relay sources, suppressed peers, and peers with the feature off.*\n\n###### How it's computed:\n*Current peer counts in each category (selected, suppressed, not-enabled), per node.*\n\n###### Reading it:\n*A high suppressed-to-selected ratio means relay is saving bandwidth as intended.*\n\n###### Healthy range:\n*Workload-dependent; suppressed should exceed selected in a well-connected mesh.*\n\n###### Watch for:\n*A large not-enabled count (older peers forcing full relay) or selected climbing while suppressed falls.*\n\n###### Keywords:\n- **Reduce-relay** *(per node)* \u2014 an optimization that relays messages through selected peers only, suppressing redundant forwarding.\n- **Transaction suppression** *(per node)* \u2014 dropping a transaction already seen from another peer, so it is not reprocessed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerReduceRelayGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#reduce-relay)", "type": "timeseries", "gridPos": { "h": 10, @@ -317,7 +317,7 @@ }, { "title": "Reduce-Relay Missing-Tx Frequency", - "description": "###### What this is:\n*How often a peer has to fetch a transaction it missed because relay suppressed it.*\n\n###### How it's computed:\n*The reported frequency of on-demand missing-transaction fetches, per node.*\n\n###### Reading it:\n*Lower is better; near-flat means suppression is well tuned.*\n\n###### Healthy range:\n*Workload-dependent; a low, stable value is expected.*\n\n###### Watch for:\n*A rising trend, meaning suppression is too aggressive and the on-demand fetch path is growing.*\n\n###### Keywords:\n- **Reduce-relay** *(per node)* \u2014 an optimization that relays messages through selected peers only, suppressing redundant forwarding.\n- **Transaction suppression** *(per node)* \u2014 dropping a transaction already seen from another peer, so it is not reprocessed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerReduceRelayGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#reduce-relay)", + "description": "###### What this is:\n*How often a peer has to fetch a transaction it missed because relay suppressed it.*\n\n###### How it's computed:\n*The reported frequency of on-demand missing-transaction fetches, per node.*\n\n###### Reading it:\n*Lower is better; near-flat means suppression is well tuned.*\n\n###### Healthy range:\n*Workload-dependent; a low, stable value is expected.*\n\n###### Watch for:\n*A rising trend, meaning suppression is too aggressive and the on-demand fetch path is growing.*\n\n###### Keywords:\n- **Reduce-relay** *(per node)* \u2014 an optimization that relays messages through selected peers only, suppressing redundant forwarding.\n- **Transaction suppression** *(per node)* \u2014 dropping a transaction already seen from another peer, so it is not reprocessed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerReduceRelayGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#reduce-relay)", "type": "timeseries", "gridPos": { "h": 10, diff --git a/docker/telemetry/grafana/dashboards/peer-quality.json b/docker/telemetry/grafana/dashboards/peer-quality.json index 6fa96c0547..ae4dfe73ec 100644 --- a/docker/telemetry/grafana/dashboards/peer-quality.json +++ b/docker/telemetry/grafana/dashboards/peer-quality.json @@ -71,7 +71,7 @@ }, { "title": "P90 Peer Latency", - "description": "###### What this is:\n*90th-percentile round-trip latency to connected peers, in milliseconds.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the p90 peer latency.*\n\n###### Reading it:\n*Lower is better; green under 200ms, yellow to 500ms, red above.*\n\n###### Healthy range:\n*Under 200ms.*\n\n###### Watch for:\n*Rising latency, which points to network congestion or geographically distant, poorly performing peers.*\n\n###### Keywords:\n- **Peer** *(per node)* \u2014 another server this node holds a protocol connection to.\n- **Overlay** *(network-wide)* \u2014 the peer-to-peer mesh xrpld nodes form to gossip transactions, proposals and validations.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerPeerQualityGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#peer)", + "description": "###### What this is:\n*90th-percentile round-trip latency to connected peers, in milliseconds.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the p90 peer latency.*\n\n###### Reading it:\n*Lower is better; green under 200ms, yellow to 500ms, red above.*\n\n###### Healthy range:\n*Under 200ms.*\n\n###### Watch for:\n*Rising latency, which points to network congestion or geographically distant, poorly performing peers.*\n\n###### Keywords:\n- **Peer** *(per node)* \u2014 another server this node holds a protocol connection to.\n- **Overlay** *(network-wide)* \u2014 the peer-to-peer mesh xrpld nodes form to gossip transactions, proposals and validations.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerPeerQualityGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#peer)", "type": "timeseries", "gridPos": { "h": 10, @@ -149,7 +149,7 @@ }, { "title": "Insane/Diverged Peers [$xrpl_network_type]", - "description": "###### What this is:\n*Count of connected peers whose ledger state has diverged from the network.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the diverged-peer count.*\n\n###### Reading it:\n*Zero is healthy; any count means those peers disagree on ledger state.*\n\n###### Healthy range:\n*0 diverged peers.*\n\n###### Watch for:\n*A persistent non-zero count, which can indicate peers on a fork or misbehaving peers.*\n\n###### Keywords:\n- **Insane / diverged peers** *(per node)* \u2014 connected peers whose ledger state disagrees with the network \u2014 possibly on a fork or misbehaving.\n- **Fork** *(network-wide)* \u2014 when nodes validate divergent ledger chains instead of a single agreed history.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerPeerQualityGauge`\n\n###### References:\n[Fork](https://xrpl.org/docs/concepts/consensus-protocol) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#insane-diverged-peers)", + "description": "###### What this is:\n*Count of connected peers whose ledger state has diverged from the network.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the diverged-peer count.*\n\n###### Reading it:\n*Zero is healthy; any count means those peers disagree on ledger state.*\n\n###### Healthy range:\n*0 diverged peers.*\n\n###### Watch for:\n*A persistent non-zero count, which can indicate peers on a fork or misbehaving peers.*\n\n###### Keywords:\n- **Insane / diverged peers** *(per node)* \u2014 connected peers whose ledger state disagrees with the network \u2014 possibly on a fork or misbehaving.\n- **Fork** *(network-wide)* \u2014 when nodes validate divergent ledger chains instead of a single agreed history.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerPeerQualityGauge`\n\n###### References:\n[Fork](https://xrpl.org/docs/concepts/consensus-protocol) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#insane-diverged-peers)", "type": "stat", "gridPos": { "h": 10, @@ -205,7 +205,7 @@ }, { "title": "Higher Version Peers % [$xrpl_network_type]", - "description": "###### What this is:\n*Percentage of connected peers running a newer rippled version than this node.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the higher-version peer percentage.*\n\n###### Reading it:\n*A high percentage suggests this node is behind and should be upgraded.*\n\n###### Healthy range:\n*Under 30%.*\n\n###### Watch for:\n*A majority of peers on a newer version, a strong upgrade signal.*\n\n###### Keywords:\n- **Peer** *(per node)* \u2014 another server this node holds a protocol connection to.\n- **Overlay** *(network-wide)* \u2014 the peer-to-peer mesh xrpld nodes form to gossip transactions, proposals and validations.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerPeerQualityGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#peer)", + "description": "###### What this is:\n*Percentage of connected peers running a newer rippled version than this node.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the higher-version peer percentage.*\n\n###### Reading it:\n*A high percentage suggests this node is behind and should be upgraded.*\n\n###### Healthy range:\n*Under 30%.*\n\n###### Watch for:\n*A majority of peers on a newer version, a strong upgrade signal.*\n\n###### Keywords:\n- **Peer** *(per node)* \u2014 another server this node holds a protocol connection to.\n- **Overlay** *(network-wide)* \u2014 the peer-to-peer mesh xrpld nodes form to gossip transactions, proposals and validations.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerPeerQualityGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#peer)", "type": "stat", "gridPos": { "h": 10, @@ -262,7 +262,7 @@ }, { "title": "Upgrade Recommended [$xrpl_network_type]", - "description": "###### What this is:\n*A flag indicating whether an upgrade is advised based on peer version analysis (Yes/No).*\n\n###### How it's computed:\n*Instantaneous gauge reading of the upgrade-recommended flag.*\n\n###### Reading it:\n*No is healthy; Yes means most peers run a newer version.*\n\n###### Healthy range:\n*No.*\n\n###### Watch for:\n*A Yes state, indicating the node risks falling out of step with the network.*\n\n###### Keywords:\n- **Peer** *(per node)* \u2014 another server this node holds a protocol connection to.\n- **Overlay** *(network-wide)* \u2014 the peer-to-peer mesh xrpld nodes form to gossip transactions, proposals and validations.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerPeerQualityGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#peer)", + "description": "###### What this is:\n*A flag indicating whether an upgrade is advised based on peer version analysis (Yes/No).*\n\n###### How it's computed:\n*Instantaneous gauge reading of the upgrade-recommended flag.*\n\n###### Reading it:\n*No is healthy; Yes means most peers run a newer version.*\n\n###### Healthy range:\n*No.*\n\n###### Watch for:\n*A Yes state, indicating the node risks falling out of step with the network.*\n\n###### Keywords:\n- **Peer** *(per node)* \u2014 another server this node holds a protocol connection to.\n- **Overlay** *(network-wide)* \u2014 the peer-to-peer mesh xrpld nodes form to gossip transactions, proposals and validations.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerPeerQualityGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#peer)", "type": "stat", "gridPos": { "h": 10, @@ -434,7 +434,7 @@ }, { "title": "Resource Disconnects", - "description": "###### What this is:\n*Cumulative count of peers dropped for exceeding resource (load) limits.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the resource-disconnect total over time.*\n\n###### Reading it:\n*A flat line is healthy; a rising line means peers are being dropped for overuse.*\n\n###### Healthy range:\n*Flat / near constant.*\n\n###### Watch for:\n*A steep climb, which flags aggressive or misbehaving peers being shed as backpressure.*\n\n###### Keywords:\n- **Resource disconnect** *(per node)* \u2014 a peer dropped for exceeding resource/load limits \u2014 the node shedding abusive or overactive peers.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#resource-disconnect)", + "description": "###### What this is:\n*Cumulative count of peers dropped for exceeding resource (load) limits.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the resource-disconnect total over time.*\n\n###### Reading it:\n*A flat line is healthy; a rising line means peers are being dropped for overuse.*\n\n###### Healthy range:\n*Flat / near constant.*\n\n###### Watch for:\n*A steep climb, which flags aggressive or misbehaving peers being shed as backpressure.*\n\n###### Keywords:\n- **Resource disconnect** *(per node)* \u2014 a peer dropped for exceeding resource/load limits \u2014 the node shedding abusive or overactive peers.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#resource-disconnect)", "type": "timeseries", "gridPos": { "h": 10, diff --git a/docker/telemetry/grafana/dashboards/rpc-performance.json b/docker/telemetry/grafana/dashboards/rpc-performance.json index 57f9d6b005..e5743792cc 100644 --- a/docker/telemetry/grafana/dashboards/rpc-performance.json +++ b/docker/telemetry/grafana/dashboards/rpc-performance.json @@ -558,7 +558,7 @@ }, { "title": "Current RPC Latency (p99 Gauge) [$xrpl_network_type]", - "description": "###### What this is:\n*Current tail latency (p99) of RPC handling across all methods, as a live gauge.*\n\n###### How it's computed:\n*p99 of the method-duration histogram over the recent window.*\n\n###### Reading it:\n*A single at-a-glance number for current RPC responsiveness.*\n\n###### Healthy range:\n*Low-millisecond under normal load.*\n\n###### Watch for:\n*Sustained elevation, indicating the node is under query pressure.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcFinished`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", + "description": "###### What this is:\n*Current tail latency (p99) of RPC handling across all methods, as a live gauge.*\n\n###### How it's computed:\n*p99 of the method-duration histogram over the recent window.*\n\n###### Reading it:\n*A single at-a-glance number for current RPC responsiveness.*\n\n###### Healthy range:\n*Low-millisecond under normal load.*\n\n###### Watch for:\n*Sustained elevation, indicating the node is under query pressure.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcFinished`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", "type": "gauge", "gridPos": { "h": 10, @@ -618,7 +618,7 @@ }, { "title": "RPC Call Rate (All Methods)", - "description": "###### What this is:\n*Overall rate of RPC method calls that started, finished, and errored, across all methods.*\n\n###### How it's computed:\n*Per-second rate of each counter over a 5-minute window, summed per node.*\n\n###### Reading it:\n*Started should closely track finished; errored should be a small fraction.*\n\n###### Healthy range:\n*Workload-dependent; started \u2248 finished, errored near zero.*\n\n###### Watch for:\n*A growing gap between started and finished (calls hanging), or an errored line that rises with load.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcStarted / recordRpcFinished / recordRpcErrored`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", + "description": "###### What this is:\n*Overall rate of RPC method calls that started, finished, and errored, across all methods.*\n\n###### How it's computed:\n*Per-second rate of each counter over a 5-minute window, summed per node.*\n\n###### Reading it:\n*Started should closely track finished; errored should be a small fraction.*\n\n###### Healthy range:\n*Workload-dependent; started \u2248 finished, errored near zero.*\n\n###### Watch for:\n*A growing gap between started and finished (calls hanging), or an errored line that rises with load.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcStarted / recordRpcFinished / recordRpcErrored`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", "type": "timeseries", "gridPos": { "h": 10, @@ -681,7 +681,7 @@ }, { "title": "Per-Method Call Rate (Top 10)", - "description": "###### What this is:\n*The ten busiest RPC methods by call rate.*\n\n###### How it's computed:\n*Per-second start rate over 5 minutes, per method, showing the top ten.*\n\n###### Reading it:\n*Identifies which methods dominate load; the mix shifts with client behaviour.*\n\n###### Healthy range:\n*Workload-dependent.*\n\n###### Watch for:\n*A single method suddenly dominating, which can signal a runaway client or abusive query pattern.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcStarted`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", + "description": "###### What this is:\n*The ten busiest RPC methods by call rate.*\n\n###### How it's computed:\n*Per-second start rate over 5 minutes, per method, showing the top ten.*\n\n###### Reading it:\n*Identifies which methods dominate load; the mix shifts with client behaviour.*\n\n###### Healthy range:\n*Workload-dependent.*\n\n###### Watch for:\n*A single method suddenly dominating, which can signal a runaway client or abusive query pattern.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcStarted`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", "type": "timeseries", "gridPos": { "h": 10, @@ -735,7 +735,7 @@ }, { "title": "Per-Method Error Rate (Top 10)", - "description": "###### What this is:\n*The ten RPC methods producing the most errors.*\n\n###### How it's computed:\n*Per-second error rate over 5 minutes, per method, top ten.*\n\n###### Reading it:\n*Normally near zero; persistent errors point to a specific failing method.*\n\n###### Healthy range:\n*Near zero for well-behaved traffic.*\n\n###### Watch for:\n*Sustained errors concentrated on one method \u2014 a broken client, a bad input, or probing.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcErrored`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", + "description": "###### What this is:\n*The ten RPC methods producing the most errors.*\n\n###### How it's computed:\n*Per-second error rate over 5 minutes, per method, top ten.*\n\n###### Reading it:\n*Normally near zero; persistent errors point to a specific failing method.*\n\n###### Healthy range:\n*Near zero for well-behaved traffic.*\n\n###### Watch for:\n*Sustained errors concentrated on one method \u2014 a broken client, a bad input, or probing.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcErrored`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", "type": "timeseries", "gridPos": { "h": 10, @@ -789,7 +789,7 @@ }, { "title": "RPC Latency - All Methods", - "description": "###### What this is:\n*Aggregate RPC handler latency across all methods (p75 and p99).*\n\n###### How it's computed:\n*Percentiles of the method-duration histogram over a 5-minute window.*\n\n###### Reading it:\n*p75 reflects typical responsiveness; p99 captures the slow tail.*\n\n###### Healthy range:\n*Sub-millisecond to low-millisecond for light commands; heavier commands run longer.*\n\n###### Watch for:\n*A rising p99 while p75 stays flat \u2014 a subset of calls degrading, often from expensive queries.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcFinished`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", + "description": "###### What this is:\n*Aggregate RPC handler latency across all methods (p75 and p99).*\n\n###### How it's computed:\n*Percentiles of the method-duration histogram over a 5-minute window.*\n\n###### Reading it:\n*p75 reflects typical responsiveness; p99 captures the slow tail.*\n\n###### Healthy range:\n*Sub-millisecond to low-millisecond for light commands; heavier commands run longer.*\n\n###### Watch for:\n*A rising p99 while p75 stays flat \u2014 a subset of calls degrading, often from expensive queries.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcFinished`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", "type": "timeseries", "gridPos": { "h": 10, @@ -845,7 +845,7 @@ }, { "title": "Per-Method Latency (p99, Top 10 Slowest)", - "description": "###### What this is:\n*The ten slowest RPC methods by tail latency.*\n\n###### How it's computed:\n*p99 of each method's duration histogram over 5 minutes, top ten.*\n\n###### Reading it:\n*Surfaces which specific methods are expensive.*\n\n###### Healthy range:\n*Method-dependent; ledger/account queries are heavier than status calls.*\n\n###### Watch for:\n*A method whose p99 climbs over time, or an unexpectedly cheap method appearing here.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcFinished`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", + "description": "###### What this is:\n*The ten slowest RPC methods by tail latency.*\n\n###### How it's computed:\n*p99 of each method's duration histogram over 5 minutes, top ten.*\n\n###### Reading it:\n*Surfaces which specific methods are expensive.*\n\n###### Healthy range:\n*Method-dependent; ledger/account queries are heavier than status calls.*\n\n###### Watch for:\n*A method whose p99 climbs over time, or an unexpectedly cheap method appearing here.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcFinished`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", "type": "timeseries", "gridPos": { "h": 10, @@ -899,7 +899,7 @@ }, { "title": "RPC Error Ratio by Method", - "description": "###### What this is:\n*The methods with the highest error rates, for spotting failure hotspots.*\n\n###### How it's computed:\n*Per-second error rate over 5 minutes, per method, top ten.*\n\n###### Reading it:\n*Highlights where failures concentrate.*\n\n###### Healthy range:\n*Near zero.*\n\n###### Watch for:\n*One method with a persistently high error rate \u2014 malformed requests or targeted probing.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcErrored`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", + "description": "###### What this is:\n*The methods with the highest error rates, for spotting failure hotspots.*\n\n###### How it's computed:\n*Per-second error rate over 5 minutes, per method, top ten.*\n\n###### Reading it:\n*Highlights where failures concentrate.*\n\n###### Healthy range:\n*Near zero.*\n\n###### Watch for:\n*One method with a persistently high error rate \u2014 malformed requests or targeted probing.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcErrored`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", "type": "timeseries", "gridPos": { "h": 10, diff --git a/docker/telemetry/grafana/dashboards/validator-health.json b/docker/telemetry/grafana/dashboards/validator-health.json index 603ef3cda9..5746a7567d 100644 --- a/docker/telemetry/grafana/dashboards/validator-health.json +++ b/docker/telemetry/grafana/dashboards/validator-health.json @@ -71,7 +71,7 @@ }, { "title": "Agreement % (1h) [$xrpl_network_type]", - "description": "###### What this is:\n*Share of ledgers over the last hour where this validator agreed with the network consensus.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the 1-hour agreement percentage.*\n\n###### Reading it:\n*Higher is better; green at 95%+, yellow from 80%, red below.*\n\n###### Healthy range:\n*95-100%.*\n\n###### Watch for:\n*Values below 80%, meaning the validator frequently disagrees with consensus.*\n\n###### Keywords:\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#consensus)", + "description": "###### What this is:\n*Share of ledgers over the last hour where this validator agreed with the network consensus.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the 1-hour agreement percentage.*\n\n###### Reading it:\n*Higher is better; green at 95%+, yellow from 80%, red below.*\n\n###### Healthy range:\n*95-100%.*\n\n###### Watch for:\n*Values below 80%, meaning the validator frequently disagrees with consensus.*\n\n###### Keywords:\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#consensus)", "type": "stat", "gridPos": { "h": 10, @@ -128,7 +128,7 @@ }, { "title": "Agreement % (24h) [$xrpl_network_type]", - "description": "###### What this is:\n*Share of ledgers over the last 24 hours where this validator agreed with consensus.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the 24-hour agreement percentage.*\n\n###### Reading it:\n*Higher is better; a smoother, longer-term view than the 1h stat.*\n\n###### Healthy range:\n*95-100%.*\n\n###### Watch for:\n*A sustained dip below 90%, which can indicate configuration drift or a network partition.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", + "description": "###### What this is:\n*Share of ledgers over the last 24 hours where this validator agreed with consensus.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the 24-hour agreement percentage.*\n\n###### Reading it:\n*Higher is better; a smoother, longer-term view than the 1h stat.*\n\n###### Healthy range:\n*95-100%.*\n\n###### Watch for:\n*A sustained dip below 90%, which can indicate configuration drift or a network partition.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", "type": "stat", "gridPos": { "h": 10, @@ -185,7 +185,7 @@ }, { "title": "Agreements vs Missed (1h) [$xrpl_network_type]", - "description": "###### What this is:\n*Counts of agreed versus missed validations over the last hour.*\n\n###### How it's computed:\n*Instantaneous gauge readings of the 1-hour agreed and missed counts.*\n\n###### Reading it:\n*Agreements should dominate; missed should be small.*\n\n###### Healthy range:\n*Missed near 0.*\n\n###### Watch for:\n*A high missed count, meaning the validator is skipping consensus rounds.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n- **Consensus round** *(network event)* \u2014 one propose-and-revise iteration of consensus; several may run before validators converge on a ledger.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Consensus round](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", + "description": "###### What this is:\n*Counts of agreed versus missed validations over the last hour.*\n\n###### How it's computed:\n*Instantaneous gauge readings of the 1-hour agreed and missed counts.*\n\n###### Reading it:\n*Agreements should dominate; missed should be small.*\n\n###### Healthy range:\n*Missed near 0.*\n\n###### Watch for:\n*A high missed count, meaning the validator is skipping consensus rounds.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n- **Consensus round** *(network event)* \u2014 one propose-and-revise iteration of consensus; several may run before validators converge on a ledger.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Consensus round](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", "type": "bargauge", "gridPos": { "h": 10, @@ -257,7 +257,7 @@ }, { "title": "Agreements vs Missed (24h) [$xrpl_network_type]", - "description": "###### What this is:\n*Counts of agreed versus missed validations over the last 24 hours.*\n\n###### How it's computed:\n*Instantaneous gauge readings of the 24-hour agreed and missed counts.*\n\n###### Reading it:\n*Agreements should dominate over the full day.*\n\n###### Healthy range:\n*Missed a small fraction of agreements.*\n\n###### Watch for:\n*A growing missed share, signalling longer-term reliability problems.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", + "description": "###### What this is:\n*Counts of agreed versus missed validations over the last 24 hours.*\n\n###### How it's computed:\n*Instantaneous gauge readings of the 24-hour agreed and missed counts.*\n\n###### Reading it:\n*Agreements should dominate over the full day.*\n\n###### Healthy range:\n*Missed a small fraction of agreements.*\n\n###### Watch for:\n*A growing missed share, signalling longer-term reliability problems.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", "type": "bargauge", "gridPos": { "h": 10, @@ -342,7 +342,7 @@ }, { "title": "Validation Rate [$xrpl_network_type]", - "description": "###### What this is:\n*Validations this node sends per minute.*\n\n###### How it's computed:\n*Per-second rate of the sent-validations counter over 5 minutes, scaled to per minute.*\n\n###### Reading it:\n*Should track the ledger close cadence; roughly one validation per closed ledger.*\n\n###### Healthy range:\n*About 12-20 per minute (one per closed ledger, ~3-5s close).*\n\n###### Watch for:\n*A drop toward zero, meaning the validator has stopped participating.*\n\n###### Keywords:\n- **Validations checked vs sent** *(per node)* \u2014 validations this node received from peers and checked, versus validations it issued itself.\n- **Ledger close interval** *(network-wide)* \u2014 the network's steady ledger rhythm \u2014 roughly one closed ledger every 3-5 seconds on Mainnet.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementValidationsSent (caller RCLConsensus.cpp)`\n\n###### References:\n[Validations checked vs sent](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Ledger close interval](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validations-checked-vs-sent)", + "description": "###### What this is:\n*Validations this node sends per minute.*\n\n###### How it's computed:\n*Per-second rate of the sent-validations counter over 5 minutes, scaled to per minute.*\n\n###### Reading it:\n*Should track the ledger close cadence; roughly one validation per closed ledger.*\n\n###### Healthy range:\n*About 12-20 per minute (one per closed ledger, ~3-5s close).*\n\n###### Watch for:\n*A drop toward zero, meaning the validator has stopped participating.*\n\n###### Keywords:\n- **Validations checked vs sent** *(per node)* \u2014 validations this node received from peers and checked, versus validations it issued itself.\n- **Ledger close interval** *(network-wide)* \u2014 the network's steady ledger rhythm \u2014 roughly one closed ledger every 3-5 seconds on Mainnet.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementValidationsSent (caller RCLConsensus.cpp)`\n\n###### References:\n[Validations checked vs sent](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Ledger close interval](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validations-checked-vs-sent)", "type": "stat", "gridPos": { "h": 10, @@ -397,7 +397,7 @@ }, { "title": "Validations Checked Rate [$xrpl_network_type]", - "description": "###### What this is:\n*Validations received from peers and checked per minute.*\n\n###### How it's computed:\n*Per-second rate of the checked-validations counter over 5 minutes, scaled to per minute.*\n\n###### Reading it:\n*Reflects how much validation traffic the network is delivering to this node.*\n\n###### Healthy range:\n*Workload-dependent; scales with trusted validator count.*\n\n###### Watch for:\n*A sudden collapse, which suggests peer connectivity loss or network isolation.*\n\n###### Keywords:\n- **Validations checked vs sent** *(per node)* \u2014 validations this node received from peers and checked, versus validations it issued itself.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementValidationsChecked (caller NetworkOPs.cpp)`\n\n###### References:\n[Validations checked vs sent](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validations-checked-vs-sent)", + "description": "###### What this is:\n*Validations received from peers and checked per minute.*\n\n###### How it's computed:\n*Per-second rate of the checked-validations counter over 5 minutes, scaled to per minute.*\n\n###### Reading it:\n*Reflects how much validation traffic the network is delivering to this node.*\n\n###### Healthy range:\n*Workload-dependent; scales with trusted validator count.*\n\n###### Watch for:\n*A sudden collapse, which suggests peer connectivity loss or network isolation.*\n\n###### Keywords:\n- **Validations checked vs sent** *(per node)* \u2014 validations this node received from peers and checked, versus validations it issued itself.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementValidationsChecked (caller NetworkOPs.cpp)`\n\n###### References:\n[Validations checked vs sent](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validations-checked-vs-sent)", "type": "stat", "gridPos": { "h": 10, @@ -436,7 +436,7 @@ }, { "title": "Amendment Blocked [$xrpl_network_type]", - "description": "###### What this is:\n*Whether the node is amendment-blocked (OK/BLOCKED).*\n\n###### How it's computed:\n*Instantaneous gauge reading of the amendment-blocked flag.*\n\n###### Reading it:\n*OK is healthy; BLOCKED means an enabled amendment is unsupported by this build.*\n\n###### Healthy range:\n*OK.*\n\n###### Watch for:\n*A BLOCKED state, which halts validation and requires a software upgrade.*\n\n###### Keywords:\n- **Amendment blocked** *(per node)* \u2014 the node has halted because the network enabled an amendment its software version does not support.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidatorHealthGauge`\n\n###### References:\n[Amendment blocked](https://xrpl.org/docs/concepts/networks-and-servers/amendments) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#amendment-blocked)", + "description": "###### What this is:\n*Whether the node is amendment-blocked (OK/BLOCKED).*\n\n###### How it's computed:\n*Instantaneous gauge reading of the amendment-blocked flag.*\n\n###### Reading it:\n*OK is healthy; BLOCKED means an enabled amendment is unsupported by this build.*\n\n###### Healthy range:\n*OK.*\n\n###### Watch for:\n*A BLOCKED state, which halts validation and requires a software upgrade.*\n\n###### Keywords:\n- **Amendment blocked** *(per node)* \u2014 the node has halted because the network enabled an amendment its software version does not support.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidatorHealthGauge`\n\n###### References:\n[Amendment blocked](https://xrpl.org/docs/concepts/networks-and-servers/amendments) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#amendment-blocked)", "type": "stat", "gridPos": { "h": 10, @@ -507,7 +507,7 @@ }, { "title": "UNL Expiry (days) [$xrpl_network_type]", - "description": "###### What this is:\n*Days remaining until the current UNL (trusted validator list) expires.*\n\n###### How it's computed:\n*Instantaneous gauge reading of days-to-expiry.*\n\n###### Reading it:\n*Higher is safer; green at 30+, yellow under 7, red at expiry.*\n\n###### Healthy range:\n*30+ days.*\n\n###### Watch for:\n*Fewer than 7 days, after which the node loses its trusted validator set if not renewed.*\n\n###### Keywords:\n- **UNL (Unique Node List)** *(per node)* \u2014 the list of validators a node trusts not to collude; the basis for its consensus and quorum.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidatorHealthGauge`\n\n###### References:\n[UNL (Unique Node List)](https://xrpl.org/docs/concepts/consensus-protocol/unl) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#unl-unique-node-list)", + "description": "###### What this is:\n*Days remaining until the current UNL (trusted validator list) expires.*\n\n###### How it's computed:\n*Instantaneous gauge reading of days-to-expiry.*\n\n###### Reading it:\n*Higher is safer; green at 30+, yellow under 7, red at expiry.*\n\n###### Healthy range:\n*30+ days.*\n\n###### Watch for:\n*Fewer than 7 days, after which the node loses its trusted validator set if not renewed.*\n\n###### Keywords:\n- **UNL (Unique Node List)** *(per node)* \u2014 the list of validators a node trusts not to collude; the basis for its consensus and quorum.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidatorHealthGauge`\n\n###### References:\n[UNL (Unique Node List)](https://xrpl.org/docs/concepts/consensus-protocol/unl) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#unl-unique-node-list)", "type": "stat", "gridPos": { "h": 10, @@ -562,7 +562,7 @@ }, { "title": "UNL Blocked [$xrpl_network_type]", - "description": "###### What this is:\n*Whether the node's UNL is blocked (OK/BLOCKED).*\n\n###### How it's computed:\n*Instantaneous gauge reading of the UNL-blocked flag.*\n\n###### Reading it:\n*OK is healthy; BLOCKED means validator trust cannot be established.*\n\n###### Healthy range:\n*OK.*\n\n###### Watch for:\n*A BLOCKED state, which can stop the node participating in consensus.*\n\n###### Keywords:\n- **UNL (Unique Node List)** *(per node)* \u2014 the list of validators a node trusts not to collude; the basis for its consensus and quorum.\n- **UNL blocked** *(per node)* \u2014 the node cannot establish a usable trusted validator list, so it cannot safely validate.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidatorHealthGauge`\n\n###### References:\n[UNL (Unique Node List)](https://xrpl.org/docs/concepts/consensus-protocol/unl) \u00b7 [UNL blocked](https://xrpl.org/docs/concepts/consensus-protocol) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#unl-unique-node-list)", + "description": "###### What this is:\n*Whether the node's UNL is blocked (OK/BLOCKED).*\n\n###### How it's computed:\n*Instantaneous gauge reading of the UNL-blocked flag.*\n\n###### Reading it:\n*OK is healthy; BLOCKED means validator trust cannot be established.*\n\n###### Healthy range:\n*OK.*\n\n###### Watch for:\n*A BLOCKED state, which can stop the node participating in consensus.*\n\n###### Keywords:\n- **UNL (Unique Node List)** *(per node)* \u2014 the list of validators a node trusts not to collude; the basis for its consensus and quorum.\n- **UNL blocked** *(per node)* \u2014 the node cannot establish a usable trusted validator list, so it cannot safely validate.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidatorHealthGauge`\n\n###### References:\n[UNL (Unique Node List)](https://xrpl.org/docs/concepts/consensus-protocol/unl) \u00b7 [UNL blocked](https://xrpl.org/docs/concepts/consensus-protocol) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#unl-unique-node-list)", "type": "stat", "gridPos": { "h": 10, @@ -633,7 +633,7 @@ }, { "title": "Agreement/Missed Counters (Rate)", - "description": "###### What this is:\n*Rate of cumulative agreement and missed-validation counters per minute.*\n\n###### How it's computed:\n*Per-second rate of each monotonic counter over 5 minutes, scaled to per minute.*\n\n###### Reading it:\n*Agreements should dominate; the missed line should stay low.*\n\n###### Healthy range:\n*Missed rate near 0.*\n\n###### Watch for:\n*A rising missed rate, complementing the windowed agreement percentages above.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidationTotalsCounters`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", + "description": "###### What this is:\n*Rate of cumulative agreement and missed-validation counters per minute.*\n\n###### How it's computed:\n*Per-second rate of each monotonic counter over 5 minutes, scaled to per minute.*\n\n###### Reading it:\n*Agreements should dominate; the missed line should stay low.*\n\n###### Healthy range:\n*Missed rate near 0.*\n\n###### Watch for:\n*A rising missed rate, complementing the windowed agreement percentages above.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidationTotalsCounters`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", "type": "timeseries", "gridPos": { "h": 10, @@ -718,7 +718,7 @@ }, { "title": "Validation Quorum [$xrpl_network_type]", - "description": "###### What this is:\n*Minimum number of trusted validations required to declare a ledger fully validated.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the current quorum requirement.*\n\n###### Reading it:\n*Tracks the quorum derived from the active validator list; changes when the list changes.*\n\n###### Healthy range:\n*Stable at the network-appropriate value.*\n\n###### Watch for:\n*An unexpected drop, which can weaken consensus safety guarantees.*\n\n###### Keywords:\n- **Validation quorum** *(network-wide)* \u2014 the minimum number of agreeing trusted validations needed to declare a ledger fully validated.\n- **Validator list** *(network-wide)* \u2014 signed lists of recommended validators (UNLs) that peers distribute to each other.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidatorHealthGauge`\n\n###### References:\n[Validation quorum](https://xrpl.org/docs/concepts/consensus-protocol/negative-unl) \u00b7 [Validator list](https://xrpl.org/docs/concepts/consensus-protocol/unl) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-quorum)", + "description": "###### What this is:\n*Minimum number of trusted validations required to declare a ledger fully validated.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the current quorum requirement.*\n\n###### Reading it:\n*Tracks the quorum derived from the active validator list; changes when the list changes.*\n\n###### Healthy range:\n*Stable at the network-appropriate value.*\n\n###### Watch for:\n*An unexpected drop, which can weaken consensus safety guarantees.*\n\n###### Keywords:\n- **Validation quorum** *(network-wide)* \u2014 the minimum number of agreeing trusted validations needed to declare a ledger fully validated.\n- **Validator list** *(network-wide)* \u2014 signed lists of recommended validators (UNLs) that peers distribute to each other.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidatorHealthGauge`\n\n###### References:\n[Validation quorum](https://xrpl.org/docs/concepts/consensus-protocol/negative-unl) \u00b7 [Validator list](https://xrpl.org/docs/concepts/consensus-protocol/unl) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-quorum)", "type": "stat", "gridPos": { "h": 10, @@ -758,7 +758,7 @@ }, { "title": "Time in Current State [$xrpl_network_type]", - "description": "###### What this is:\n*How long the server has held its current operating state, in seconds.*\n\n###### How it's computed:\n*Current value of the time-in-state gauge.*\n\n###### Reading it:\n*Not yet wired in the code; the value currently always reads 0.*\n\n###### Healthy range:\n*Not applicable; the value is always 0 today.*\n\n###### Watch for:\n*n/a until implemented.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerStateTrackingGauge`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", + "description": "###### What this is:\n*How long the server has held its current operating state, in seconds.*\n\n###### How it's computed:\n*Current value of the time-in-state gauge.*\n\n###### Reading it:\n*Not yet wired in the code; the value currently always reads 0.*\n\n###### Healthy range:\n*Not applicable; the value is always 0 today.*\n\n###### Watch for:\n*n/a until implemented.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerStateTrackingGauge`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", "type": "stat", "gridPos": { "h": 10, @@ -797,7 +797,7 @@ }, { "title": "State Changes Rate [$xrpl_network_type]", - "description": "###### What this is:\n*Rate of server operating-state changes per hour.*\n\n###### How it's computed:\n*Per-hour rate of the state-change counter, averaged over a 1-hour window.*\n\n###### Reading it:\n*Near zero is healthy; each increment is one state transition.*\n\n###### Healthy range:\n*Near 0 changes per hour.*\n\n###### Watch for:\n*Frequent transitions, which point to network instability or configuration problems.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementStateChanges (caller NetworkOPs.cpp)`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", + "description": "###### What this is:\n*Rate of server operating-state changes per hour.*\n\n###### How it's computed:\n*Per-hour rate of the state-change counter, averaged over a 1-hour window.*\n\n###### Reading it:\n*Near zero is healthy; each increment is one state transition.*\n\n###### Healthy range:\n*Near 0 changes per hour.*\n\n###### Watch for:\n*Frequent transitions, which point to network instability or configuration problems.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementStateChanges (caller NetworkOPs.cpp)`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", "type": "stat", "gridPos": { "h": 10, @@ -852,7 +852,7 @@ }, { "title": "Ledgers Closed Rate [$xrpl_network_type]", - "description": "###### What this is:\n*Ledgers closed per minute by this node.*\n\n###### How it's computed:\n*Per-second rate of the ledgers-closed counter over 5 minutes, scaled to per minute.*\n\n###### Reading it:\n*Should match the network's steady close cadence.*\n\n###### Healthy range:\n*About 12-20 per minute (one per closed ledger, ~3-5s close).*\n\n###### Watch for:\n*Deviation from the expected cadence, which indicates consensus timing trouble or the node falling behind.*\n\n###### Keywords:\n- **Ledgers closed rate** *(per node)* \u2014 how many ledgers this node closed per minute; should match the network close cadence.\n- **Ledger close interval** *(network-wide)* \u2014 the network's steady ledger rhythm \u2014 roughly one closed ledger every 3-5 seconds on Mainnet.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementLedgersClosed (caller RCLConsensus.cpp)`\n\n###### References:\n[Ledgers closed rate](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledgers-closed-rate)", + "description": "###### What this is:\n*Ledgers closed per minute by this node.*\n\n###### How it's computed:\n*Per-second rate of the ledgers-closed counter over 5 minutes, scaled to per minute.*\n\n###### Reading it:\n*Should match the network's steady close cadence.*\n\n###### Healthy range:\n*About 12-20 per minute (one per closed ledger, ~3-5s close).*\n\n###### Watch for:\n*Deviation from the expected cadence, which indicates consensus timing trouble or the node falling behind.*\n\n###### Keywords:\n- **Ledgers closed rate** *(per node)* \u2014 how many ledgers this node closed per minute; should match the network close cadence.\n- **Ledger close interval** *(network-wide)* \u2014 the network's steady ledger rhythm \u2014 roughly one closed ledger every 3-5 seconds on Mainnet.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementLedgersClosed (caller RCLConsensus.cpp)`\n\n###### References:\n[Ledgers closed rate](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledgers-closed-rate)", "type": "stat", "gridPos": { "h": 10, @@ -907,7 +907,7 @@ }, { "title": "Agreement % (7d) [$xrpl_network_type]", - "description": "###### What this is:\n*Share of ledgers over the trailing 7 days where this validator agreed with consensus.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the 7-day agreement percentage.*\n\n###### Reading it:\n*The long-term reliability window; higher is better.*\n\n###### Healthy range:\n*95-100%.*\n\n###### Watch for:\n*A gradual decline, which reflects chronic rather than transient disagreement.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", + "description": "###### What this is:\n*Share of ledgers over the trailing 7 days where this validator agreed with consensus.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the 7-day agreement percentage.*\n\n###### Reading it:\n*The long-term reliability window; higher is better.*\n\n###### Healthy range:\n*95-100%.*\n\n###### Watch for:\n*A gradual decline, which reflects chronic rather than transient disagreement.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", "type": "stat", "gridPos": { "h": 10, @@ -964,7 +964,7 @@ }, { "title": "State Value Timeline", - "description": "###### What this is:\n*Numeric encoding of the server operating state (disconnected, connected, syncing, tracking, full, validating, proposing) over time.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the encoded state value.*\n\n###### Reading it:\n*A flat line at the full-operation state is healthy; steps show transitions.*\n\n###### Healthy range:\n*Steady at the highest (full) state.*\n\n###### Watch for:\n*Frequent transitions, useful for correlating state flapping with other metrics.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n- **Consensus mode** *(per node)* \u2014 the node's role/health in the current round: Proposing, Observing, Wrong Ledger, or Switched Ledger.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerStateTrackingGauge`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Consensus mode](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", + "description": "###### What this is:\n*Numeric encoding of the server operating state (disconnected, connected, syncing, tracking, full, validating, proposing) over time.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the encoded state value.*\n\n###### Reading it:\n*A flat line at the full-operation state is healthy; steps show transitions.*\n\n###### Healthy range:\n*Steady at the highest (full) state.*\n\n###### Watch for:\n*Frequent transitions, useful for correlating state flapping with other metrics.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n- **Consensus mode** *(per node)* \u2014 the node's role/health in the current round: Proposing, Observing, Wrong Ledger, or Switched Ledger.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerStateTrackingGauge`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Consensus mode](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", "type": "timeseries", "gridPos": { "h": 10, @@ -1013,7 +1013,7 @@ }, { "title": "Agreements vs Missed (7d)", - "description": "###### What this is:\n*Agreed versus missed validation counts over the trailing 7 days.*\n\n###### How it's computed:\n*Instantaneous gauge readings of the 7-day agreed and missed counts.*\n\n###### Reading it:\n*Agreements should dominate across the week.*\n\n###### Healthy range:\n*Missed a small fraction of agreements.*\n\n###### Watch for:\n*A rising missed trend, signalling sustained validator unreliability.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", + "description": "###### What this is:\n*Agreed versus missed validation counts over the trailing 7 days.*\n\n###### How it's computed:\n*Instantaneous gauge readings of the 7-day agreed and missed counts.*\n\n###### Reading it:\n*Agreements should dominate across the week.*\n\n###### Healthy range:\n*Missed a small fraction of agreements.*\n\n###### Watch for:\n*A rising missed trend, signalling sustained validator unreliability.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", "type": "timeseries", "gridPos": { "h": 10, diff --git a/docker/telemetry/grafana/provisioning/alerting/rules.yaml b/docker/telemetry/grafana/provisioning/alerting/rules.yaml index 7d909074b2..0f0f8223a3 100644 --- a/docker/telemetry/grafana/provisioning/alerting/rules.yaml +++ b/docker/telemetry/grafana/provisioning/alerting/rules.yaml @@ -629,25 +629,25 @@ groups: # Node state flapping: full -> syncing/tracking -> full, repeatedly. # # state_accounting_full_transitions counts transitions INTO full - # (NetworkOPs.cpp StateAccounting::mode) and is exported as a cumulative - # gauge, so increase() is correct — and its counter-reset correction - # turns a process restart into a small positive delta rather than a - # false spike. + # (NetworkOPs.cpp StateAccounting::mode), a cumulative gauge that EVERY + # node always reports, so increase() returns a real series (0 when + # healthy) and this rule never evaluates to NoData. A sparse counter such + # as state_changes_total{from,to} has no series until the edge occurs, so + # it would raise a false DatasourceNoData on a healthy node -- do not + # switch to it here. # - # state_changes_total cannot be used here: it carries no from/to labels, - # so it cannot distinguish a flap from a normal startup walk. + # Threshold >0: one full -> syncing -> full round is a single re-entry, + # which is exactly the online-delete rotation flap to catch. # # The uptime gate is load-bearing. Every node walks # disconnected -> connected -> syncing -> tracking -> full once at boot; - # without the gate every restart pages. Measured: flapping nodes re-enter - # full 4-6 times per hour sustained, healthy nodes 0-1, so >3 separates - # the populations with a 3x margin. + # the gate suppresses that first hour so a restart does not page. - uid: xrpld-node-state-flapping title: NodeStateFlapping condition: C for: 15m isPaused: true - noDataState: NoData + noDataState: OK execErrState: Error labels: severity: warning @@ -656,9 +656,10 @@ groups: summary: "Node state flapping on {{ $labels.service_instance_id }}" description: >- Node {{ $labels.service_instance_id }} re-entered the FULL state - {{ $values.B.Value }} times in the last hour (>3). It is oscillating - between full and syncing/connected rather than holding sync. Check - node-store IO latency, peer connectivity, and clock sync. + {{ $values.B.Value }} time(s) in the last hour past its first hour + of uptime. It is flapping out of sync rather than holding FULL. + Likely the online-delete rotation cache-freshen; check the rotation + spans and cache lock-hold peak. data: - refId: A relativeTimeRange: @@ -700,7 +701,7 @@ groups: conditions: - evaluator: type: gt - params: [3] + params: [0] datasource: type: __expr__ uid: __expr__ diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index 9e4bf2001a..0ce3ae2876 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -1753,27 +1753,27 @@ These gauges are exported via the OTel Metrics SDK `PeriodicMetricReader` (10s i | Prometheus Metric | Source | Description | | --------------------------------------------------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `server_info{metric="server_state"}` | MetricsRegistry.cpp | Operating mode (0=DISCONNECTED .. 4=FULL) | -| `server_info{metric="uptime"}` | MetricsRegistry.cpp | Seconds since server start | -| `server_info{metric="peers"}` | MetricsRegistry.cpp | Total connected peers | -| `server_info{metric="validated_ledger_seq"}` | MetricsRegistry.cpp | Validated ledger sequence number | -| `server_info{metric="ledger_current_index"}` | MetricsRegistry.cpp | Current open ledger sequence | -| `server_info{metric="peer_disconnects_resources"}` | MetricsRegistry.cpp | Cumulative resource-related peer disconnects | -| `server_info{metric="last_close_proposers"}` | MetricsRegistry.cpp | Proposers in last closed round | -| `server_info{metric="last_close_converge_time_ms"}` | MetricsRegistry.cpp | Last close convergence time (ms) | -| `server_info{metric="last_close_time"}` | MetricsRegistry.cpp | Network close time of last closed ledger (NetClock secs since XRPL epoch). Age = `time() - (value + 946684800)`; close interval = `1/rate(ledgers_closed_total)`, not a gauge delta | -| `build_info{version=""}` | MetricsRegistry.cpp | Info-style metric (always 1) | -| `complete_ledgers{bound="start\|end",index=""}` | MetricsRegistry.cpp | Complete ledger range start/end pairs | -| `db_metrics{metric="db_kb_total"}` | MetricsRegistry.cpp | Total database size (KB) | -| `db_metrics{metric="db_kb_ledger"}` | MetricsRegistry.cpp | Ledger database size (KB) | -| `db_metrics{metric="db_kb_transaction"}` | MetricsRegistry.cpp | Transaction database size (KB) | -| `db_metrics{metric="historical_perminute"}` | MetricsRegistry.cpp | Historical ledger fetches per minute | -| `cache_metrics{metric="AL_size"}` | MetricsRegistry.cpp | AcceptedLedger cache size | -| `nodestore_state{metric="node_reads_duration_us"}` | MetricsRegistry.cpp | Cumulative read time (microseconds) | -| `nodestore_state{metric="node_writes_duration_us"}` | MetricsRegistry.cpp | Cumulative write time (microseconds) | -| `nodestore_state{metric="read_request_bundle"}` | MetricsRegistry.cpp | Read request bundle count | -| `nodestore_state{metric="read_threads_running"}` | MetricsRegistry.cpp | Active read threads | -| `nodestore_state{metric="read_threads_total"}` | MetricsRegistry.cpp | Total read threads configured | +| `server_info{metric="server_state"}` | AppMetricGauges.cpp | Operating mode (0=DISCONNECTED .. 4=FULL) | +| `server_info{metric="uptime"}` | AppMetricGauges.cpp | Seconds since server start | +| `server_info{metric="peers"}` | AppMetricGauges.cpp | Total connected peers | +| `server_info{metric="validated_ledger_seq"}` | AppMetricGauges.cpp | Validated ledger sequence number | +| `server_info{metric="ledger_current_index"}` | AppMetricGauges.cpp | Current open ledger sequence | +| `server_info{metric="peer_disconnects_resources"}` | AppMetricGauges.cpp | Cumulative resource-related peer disconnects | +| `server_info{metric="last_close_proposers"}` | AppMetricGauges.cpp | Proposers in last closed round | +| `server_info{metric="last_close_converge_time_ms"}` | AppMetricGauges.cpp | Last close convergence time (ms) | +| `server_info{metric="last_close_time"}` | AppMetricGauges.cpp | Network close time of last closed ledger (NetClock secs since XRPL epoch). Age = `time() - (value + 946684800)`; close interval = `1/rate(ledgers_closed_total)`, not a gauge delta | +| `build_info{version=""}` | AppMetricGauges.cpp | Info-style metric (always 1) | +| `complete_ledgers{bound="start\|end",index=""}` | AppMetricGauges.cpp | Complete ledger range start/end pairs | +| `db_metrics{metric="db_kb_total"}` | AppMetricGauges.cpp | Total database size (KB) | +| `db_metrics{metric="db_kb_ledger"}` | AppMetricGauges.cpp | Ledger database size (KB) | +| `db_metrics{metric="db_kb_transaction"}` | AppMetricGauges.cpp | Transaction database size (KB) | +| `db_metrics{metric="historical_perminute"}` | AppMetricGauges.cpp | Historical ledger fetches per minute | +| `cache_metrics{metric="AL_size"}` | AppMetricGauges.cpp | AcceptedLedger cache size | +| `nodestore_state{metric="node_reads_duration_us"}` | AppMetricGauges.cpp | Cumulative read time (microseconds) | +| `nodestore_state{metric="node_writes_duration_us"}` | AppMetricGauges.cpp | Cumulative write time (microseconds) | +| `nodestore_state{metric="read_request_bundle"}` | AppMetricGauges.cpp | Read request bundle count | +| `nodestore_state{metric="read_threads_running"}` | AppMetricGauges.cpp | Active read threads | +| `nodestore_state{metric="read_threads_total"}` | AppMetricGauges.cpp | Total read threads configured | | `rpc_in_flight_requests` | PerfLogImp.cpp | RPC requests currently executing (UpDownCounter) | #### Sync Diagnosis Signals @@ -1786,21 +1786,21 @@ label values rather than reporting them as zero. | Prometheus Metric | Source | Description | | ---------------------------------------------------- | ------------------- | ----------------------------------------------------------- | -| `nodestore_state{metric="read_mean_us"}` | MetricsRegistry.cpp | Mean time per backend read (microseconds) | -| `nodestore_state{metric="write_mean_us"}` | MetricsRegistry.cpp | Mean time per backend write (microseconds) | -| `nodestore_state{metric="nudb_writers_in_flight"}` | MetricsRegistry.cpp | Threads inside a NuDB insert right now | -| `nodestore_state{metric="nudb_writer_depth_x100"}` | MetricsRegistry.cpp | Mean queue depth at the NuDB insert mutex, ×100 | -| `nodestore_state{metric="nudb_insert_mean_us"}` | MetricsRegistry.cpp | Mean NuDB insert time, queueing included (microseconds) | -| `nodestore_state{metric="nudb_insert_max_us"}` | MetricsRegistry.cpp | Slowest single NuDB insert seen (microseconds) | -| `nodestore_state{metric="acquire_deferrals"}` | MetricsRegistry.cpp | Timer jobs skipped because the lane was full, **all lanes** | -| `nodestore_state{metric="acquire_timeouts"}` | MetricsRegistry.cpp | Timer bodies that ran and advanced retry, **all lanes** | -| `nodestore_state{metric="acquire_ledger_deferrals"}` | MetricsRegistry.cpp | Deferrals from ledger acquisition alone | -| `nodestore_state{metric="acquire_ledger_timeouts"}` | MetricsRegistry.cpp | Timeouts from ledger acquisition alone | -| `nodestore_state{metric="acquire_give_ups"}` | MetricsRegistry.cpp | Acquisitions that exhausted their retry budget | -| `nodestore_state{metric="acquire_aborts"}` | MetricsRegistry.cpp | Acquisitions destroyed before finishing | -| `nodestore_state{metric="acquire_aborts_partial"}` | MetricsRegistry.cpp | Subset of aborts that discarded partly built maps | -| `nodestore_state{metric="acquire_completions"}` | MetricsRegistry.cpp | Acquisitions that finished successfully | -| `nodestore_state{metric="acquire_sweep_evictions"}` | MetricsRegistry.cpp | Acquisitions evicted by the 1-minute sweep | +| `nodestore_state{metric="read_mean_us"}` | AppMetricGauges.cpp | Mean time per backend read (microseconds) | +| `nodestore_state{metric="write_mean_us"}` | AppMetricGauges.cpp | Mean time per backend write (microseconds) | +| `nodestore_state{metric="nudb_writers_in_flight"}` | AppMetricGauges.cpp | Threads inside a NuDB insert right now | +| `nodestore_state{metric="nudb_writer_depth_x100"}` | AppMetricGauges.cpp | Mean queue depth at the NuDB insert mutex, ×100 | +| `nodestore_state{metric="nudb_insert_mean_us"}` | AppMetricGauges.cpp | Mean NuDB insert time, queueing included (microseconds) | +| `nodestore_state{metric="nudb_insert_max_us"}` | AppMetricGauges.cpp | Slowest single NuDB insert seen (microseconds) | +| `nodestore_state{metric="acquire_deferrals"}` | AppMetricGauges.cpp | Timer jobs skipped because the lane was full, **all lanes** | +| `nodestore_state{metric="acquire_timeouts"}` | AppMetricGauges.cpp | Timer bodies that ran and advanced retry, **all lanes** | +| `nodestore_state{metric="acquire_ledger_deferrals"}` | AppMetricGauges.cpp | Deferrals from ledger acquisition alone | +| `nodestore_state{metric="acquire_ledger_timeouts"}` | AppMetricGauges.cpp | Timeouts from ledger acquisition alone | +| `nodestore_state{metric="acquire_give_ups"}` | AppMetricGauges.cpp | Acquisitions that exhausted their retry budget | +| `nodestore_state{metric="acquire_aborts"}` | AppMetricGauges.cpp | Acquisitions destroyed before finishing | +| `nodestore_state{metric="acquire_aborts_partial"}` | AppMetricGauges.cpp | Subset of aborts that discarded partly built maps | +| `nodestore_state{metric="acquire_completions"}` | AppMetricGauges.cpp | Acquisitions that finished successfully | +| `nodestore_state{metric="acquire_sweep_evictions"}` | AppMetricGauges.cpp | Acquisitions evicted by the 1-minute sweep | `nudb_writer_depth_x100` is fixed-point: divide by 100 to read it. The depth sits just above 1.0 even under load, so an integer gauge would truncate the whole @@ -1839,8 +1839,10 @@ ledger acquisition deferring". Use `acquire_ledger_deferrals` and These five come from the `PerfLog` job hooks, not from beast::insight, so they are exported by the `MetricsRegistry` meter. `job_queued_us` and `job_running_us` have explicit microsecond bucket views registered -(`addMicrosecondHistogramView()` calls at MetricsRegistry.cpp:310-311; the helper -itself is at `:197`) spanning 100 µs to 60 s; without those the SDK default +(`addMicrosecondHistogramView()`, called from +`MetricsRegistry::initExporterAndProvider()` — both live in +`src/libxrpl/telemetry/MetricsRegistry.cpp`) spanning 100 µs to 60 s; without +those the SDK default buckets stop at 10 ms and every quantile saturates. | Prometheus Metric | Kind | Labels | Description | @@ -1872,7 +1874,7 @@ two production job names embed a ledger sequence number: A raw label would mint a new Prometheus series for every ledger — unbounded growth at ~1 series every 3-5 s, forever. `MetricsRegistry::sanitiseHandler()` (declared inline in -`src/xrpld/telemetry/MetricsRegistry.h`) therefore applies one rule: +`include/xrpl/telemetry/MetricsRegistry.h`) therefore applies one rule: - Keep the name when it is **non-empty and every character is an ASCII letter**. - Otherwise return the constant `"other"`. An empty name, a digit, a hyphen, or @@ -1999,7 +2001,7 @@ rpc_batch_size_count - rpc_batch_size_bucket{le="12288"} -Use the call-site macros in `src/xrpld/telemetry/MetricMacros.h` -- no +Use the call-site macros in `include/xrpl/telemetry/MetricMacros.h` -- no `MetricsRegistry.h`/`.cpp` edit is needed for any of these: | Need | Macro | @@ -2010,12 +2012,12 @@ Use the call-site macros in `src/xrpld/telemetry/MetricMacros.h` -- no | Last-value snapshot (not a distribution) | `XRPL_METRIC_GAUGE_RECORD` [+ `_LABELED`] -- requires an ABI v2 opentelemetry-cpp build; this repo currently builds ABI v1, so use the observable-gauge row below instead | | Value your own code already tracks, sampled on a timer | `XRPL_METRIC_OBSERVABLE_GAUGE_REGISTER` / `_COUNTER_REGISTER` / `_UPDOWN_REGISTER` | -First declare the name in `src/xrpld/telemetry/MetricNames.h` -- the emit site +First declare the name in `include/xrpl/telemetry/MetricNames.h` -- the emit site must reference a constant, never a string literal, and CI Rule I enforces that for any metric family that already has constants: ```cpp -// in src/xrpld/telemetry/MetricNames.h, namespace metric: +// in include/xrpl/telemetry/MetricNames.h, namespace metric: inline constexpr char myNewThingTotal[] = "my_new_thing_total"; inline constexpr char myInFlightRequests[] = "my_in_flight_requests"; inline constexpr char myThingSize[] = "my_thing_size"; @@ -2024,8 +2026,8 @@ inline constexpr char myThingSize[] = "my_thing_size"; Then emit against it: ```cpp -#include -#include +#include +#include // Monotonic counter: XRPL_METRIC_COUNTER_INC( @@ -2053,8 +2055,9 @@ Naming rules (counter `_total`, duration `_us`/`_ms`/`_seconds`, no `xrpld_` prefix, bounded label cardinality) are listed in CONTRIBUTING.md -> "Telemetry metric naming" and enforced by CI Rules I/J/K. A histogram whose values can exceed ~10,000 units (e.g. a microsecond duration beyond 10ms) still needs one -line added to `addMicrosecondHistogramView()` in `MetricsRegistry.cpp` -- the -only case that still touches a central file. There is no way to read a metric's +line added to `addMicrosecondHistogramView()` in +`src/libxrpl/telemetry/MetricsRegistry.cpp` -- the only case that still touches a +central file. There is no way to read a metric's current value back from application code -- OTel's API is write-only by design; keep your own state if your logic needs to both record and read a running value (see the Doxygen header in `MetricMacros.h` for the full explanation). @@ -2324,10 +2327,12 @@ Requires `trace_peer=1` in the `[telemetry]` config section. > `{quantile="$quantile"}` matches nothing and reports no error. The job queue > exposes two parallel families: `job_running_us` / `job_queued_us` > (`MetricsRegistry` instruments, labelled by `job_type` and `handler`, -> microseconds — what these panels use; -> [MetricsRegistry.cpp:94-95](../src/xrpld/telemetry/MetricsRegistry.cpp#L94), -> [363-366](../src/xrpld/telemetry/MetricsRegistry.cpp#L363), recorded from the -> `PerfLog` job hooks at +> microseconds — what these panels use; the two names come from the +> `kJobQueuedDurationUs` / `kJobRunningDurationUs` constants and the microsecond +> buckets from `addMicrosecondHistogramView()` in +> `MetricsRegistry::initExporterAndProvider()`, all in +> [MetricsRegistry.cpp](../src/libxrpl/telemetry/MetricsRegistry.cpp), recorded +> from the `PerfLog` job hooks at > [PerfLogImp.cpp:432](../src/xrpld/perflog/detail/PerfLogImp.cpp#L432)) and > `jobq_[_q]_milliseconds` > (beast::insight, one instrument per job type, milliseconds — diff --git a/src/xrpld/telemetry/MetricMacros.h b/include/xrpl/telemetry/MetricMacros.h similarity index 99% rename from src/xrpld/telemetry/MetricMacros.h rename to include/xrpl/telemetry/MetricMacros.h index ae6c820468..39c8fd8b85 100644 --- a/src/xrpld/telemetry/MetricMacros.h +++ b/include/xrpl/telemetry/MetricMacros.h @@ -128,9 +128,8 @@ #include #endif -#include // IWYU pragma: keep - -#include // IWYU pragma: keep +#include // IWYU pragma: keep +#include // IWYU pragma: keep #ifdef XRPL_ENABLE_TELEMETRY diff --git a/src/xrpld/telemetry/MetricNames.h b/include/xrpl/telemetry/MetricNames.h similarity index 100% rename from src/xrpld/telemetry/MetricNames.h rename to include/xrpl/telemetry/MetricNames.h diff --git a/include/xrpl/telemetry/MetricsRegistry.h b/include/xrpl/telemetry/MetricsRegistry.h new file mode 100644 index 0000000000..faa5a94692 --- /dev/null +++ b/include/xrpl/telemetry/MetricsRegistry.h @@ -0,0 +1,943 @@ +#pragma once + +/** + * Central OTel metrics registry: the export pipeline and the instruments that + * app code pushes values into. + * + * Owns the OpenTelemetry MeterProvider, the OTLP/HTTP exporter, the periodic + * reader and every SYNCHRONOUS instrument (counters and histograms) that is + * not already covered by the beast::insight StatsD pipeline. The instruments + * are created once at startup and drained by the OTel + * PeriodicExportingMetricReader at a fixed interval (10 s). + * + * When XRPL_ENABLE_TELEMETRY is **not** defined, this class compiles to a + * lightweight no-op: every public method is an empty inline. + * + * Every caller reaches it through ServiceRegistry::getMetricsRegistry(), and + * the XRPL_METRIC_* macros then create their own instruments from meter(). + * + * Dependency / ownership diagram (ASCII): + * + * MetricsRegistry + * | + * +-- OTel MeterProvider (owns reader + exporter) + * | | + * | +-- PeriodicExportingMetricReader + * | +-- OtlpHttpMetricExporter + * | + * +-- Counters / Histograms (synchronous instruments) + * | +-- rpc_method_started_total + * | +-- rpc_method_finished_total + * | +-- rpc_method_errored_total + * | +-- rpc_method_us (Histogram) + * | +-- job_queued_total{job_type,handler} + * | +-- job_started_total{job_type,handler} + * | +-- job_finished_total{job_type,handler} + * | +-- job_queued_us{job_type,handler} (Histogram) + * | +-- job_running_us{job_type,handler} (Histogram) + * | +-- ledgers_closed_total + * | +-- validations_sent_total + * | +-- validations_checked_total + * | +-- ledger_history_mismatch_total{reason} + * | +-- txq_expired_total + * | +-- txq_dropped_total{reason} + * | + * +-- ValidationTracker (rolling validation-agreement windows) + * + * Control-flow for synchronous instruments: + * + * PerfLogImp::rpcStart/rpcEnd/jobQueue/jobStart/jobFinish + * | + * v + * MetricsRegistry::recordRpc*(method, ...) / recordJob*(type, ...) + * | + * v + * OTel Counter::Add() or Histogram::Record() + * | + * v + * Periodically flushed by the MetricReader + * + * Example usage: + * + * @code + * // In ApplicationImp's member-init list, right after telemetry_ and before + * // every subsystem. The constructor builds the pipeline and every + * // synchronous instrument, so no producer can exist before they do. The + * // endpoint, the TLS settings and the resource identity come from + * // [telemetry] and [network_id], read by Application.cpp rather than + * // through Telemetry::Setup. + * metricsRegistry_(std::make_unique( + * telemetry_->isEnabled(), journal, options)) + * + * // In PerfLogImp::rpcStart(): + * if (auto* mr = app_.getMetricsRegistry()) + * mr->recordRpcStarted("server_info"); + * + * // In PerfLogImp::rpcEnd(): + * if (auto* mr = app_.getMetricsRegistry()) + * { + * mr->recordRpcFinished("server_info", durationUs); + * // or: mr->recordRpcErrored("server_info", durationUs); + * } + * + * // In PerfLogImp::jobQueue(). The second argument is the addJob name; + * // it is sanitised internally into the bounded `handler` label. + * if (auto* mr = app_.getMetricsRegistry()) + * mr->recordJobQueued("ledgerData", "ProcessLData"); + * + * // Shutdown, before any observer of live server state is torn down. + * // Idempotent, so run() and ~ApplicationImp both call it: + * metricsRegistry_->stop(); + * @endcode + * + * Caveats: + * - The MetricsRegistry must be created AFTER the Telemetry object because + * it reads isEnabled() to decide whether to initialize the OTel SDK, and + * BEFORE every subsystem that records a metric. Declaration order in + * ApplicationImp is the guarantee; keep the member where it is. + * - Adding a new synchronous instrument requires updating both the header + * and the .cpp, then calling the new record*() method from the + * instrumentation site. Prefer the XRPL_METRIC_* macros, which need + * neither. + */ + +#ifdef XRPL_ENABLE_TELEMETRY +// The tracker is held and exposed only in this configuration, where the +// observable-gauge callbacks that drain it exist. +#include +#endif + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#ifdef XRPL_ENABLE_TELEMETRY +#include +#include +#include +#include +#include + +// These two serve only the telemetry-only members below, so they are guarded +// like their uses: std::atomic by phase_, std::shared_ptr by provider_. +#include +#include +#endif + +namespace xrpl::telemetry { + +/** + * Run time at which a finished job counts as a stall, in microseconds. + * Equal to LoadMonitor's 1 s warn threshold (LoadMonitor.cpp + * addLoadSample) so this counter and the "Job: ... run:" log line + * describe the same event. + */ +inline constexpr std::int64_t kJobStallThresholdUs = 1'000'000; + +/** + * Central OpenTelemetry metric registry. + * + * Owns the metrics export pipeline and every push-model instrument that the + * beast::insight StatsD pipeline does not already cover. See the file-level + * header comment above for the instrument inventory and usage examples. + * + * Class / collaborator diagram (ASCII): + * + * MetricsRegistry + * | + * +-- creates/owns --> MeterProvider (SDK) + * | | + * | v + * | reader thread (~10 s) -> OTLP/HTTP export + * | + * +-- creates/owns --> Counter and Histogram instruments + * | + * +-- holds ----------> ValidationTracker (rolling windows) + * + * @note Thread safety: + * - The recordRpc, recordJob, and increment methods are invoked + * from hot paths. OTel Counter::Add() and Histogram::Record() + * are documented thread-safe, and null-guard checks protect + * uninitialized instruments. + * - recording() is a single acquire load and is read on every + * XRPL_METRIC_* call site, from any thread. + * - meter() may be called from any thread. The constructor is the + * last writer of the handle it returns; stop() leaves it alone. + * - ValidationTracker protects its rolling windows internally. + * - The constructor, hasPipeline() and stop() are NOT thread-safe + * with each other. All three read or write provider_, a plain + * shared_ptr that stop() resets, so all three belong on the + * single server lifecycle thread, in that order. + * + * @note Lifetime, in two phases (see Phase): + * - Ready: the constructor built the pipeline and the synchronous + * instruments. Runs in ApplicationImp's member-init list, so it precedes + * every subsystem that could record. + * - Stopped: stop() joined the reader thread. Runs before any observed + * service stops, from run() and again from ~ApplicationImp for the + * paths that never reach run(). + * + * @note Extending: + * - Adding a new SYNCHRONOUS instrument (counter/histogram): prefer the + * XRPL_METRIC_* call-site macros in MetricMacros.h -- no header/cpp + * edit needed. Fall back to a dedicated member + init line + record + * method (the pattern below) only when the metric needs to be read + * back by other code (e.g. ValidationTracker-style accumulation) or + * needs a custom histogram bucket View (see the histogram note in + * MetricMacros.h). + * - An OBSERVABLE instrument does not belong here. Its callback reads live + * server state, so it must be registered only once that state exists, + * which is later than this object is built. Register it from the layer + * that owns those callbacks. + */ +class MetricsRegistry +{ +public: + /** + * Everything the constructor needs from config: where to export, how to + * secure the connection, and the process identity stamped on the OTel + * resource. + * + * The values come from the `[telemetry]` section plus `[network_id]`, read + * by `makeMetricsRegistryOptions()` in `Application.cpp`. They must match + * what `makeTelemetrySetup()` gives the trace pipeline, or one node reports + * two identities and a dashboard filter shows half its series. + * + * A struct rather than ten positional parameters: seven of them are + * strings, so a swapped pair would compile and silently stamp the wrong + * label. Designated initializers name every value at the call site. + * + * @code + * MetricsRegistry::Options opts{ + * .endpoint = "http://localhost:4318/v1/metrics", + * .serviceName = "xrpld", + * .serviceVersion = build_info::getVersionString(), + * .serviceInstanceId = nodePublicKey, + * .nodeId = nodePublicKey, + * .networkId = 2}; + * MetricsRegistry registry(enabled, journal, opts); + * + * // Edge case: mutual TLS to a collector that requires it. + * opts.useTls = true; + * opts.tlsCaCertPath = "/etc/xrpld/otel-ca.pem"; + * opts.tlsClientCertPath = "/etc/xrpld/node.pem"; + * opts.tlsClientKeyPath = "/etc/xrpld/node.key"; + * MetricsRegistry secure(enabled, journal, opts); + * @endcode + * + * @note Plain aggregate, no invariants enforced. `networkType` is not a + * field: it is derived from @ref networkId inside the constructor so + * the two can never disagree. + */ + struct Options + { + /** + * OTLP/HTTP endpoint URL for metric export, from + * `[telemetry] metrics_endpoint`. + */ + std::string endpoint; + + /** + * service.name resource attribute, from `[telemetry] service_name`. + * Stamped unconditionally, so an empty value here yields an empty + * label rather than the SDK's `unknown_service` default. The caller + * seeds it with `systemName()`. + */ + std::string serviceName; + + /** + * service.version resource attribute — the build's version string. + * Left off the resource when empty. + */ + std::string serviceVersion; + + /** + * service.instance.id resource attribute, from + * `[telemetry] service_instance_id` or the node's base58 public key. + * Left off the resource when empty. + */ + std::string serviceInstanceId; + + /** + * xrpl.node.id resource attribute — the node's base58 public key, + * which config cannot override. Left off the resource when empty. + */ + std::string nodeId; + + /** + * Network identifier from `[network_id]`. Stamped as xrpl.network.id, + * and mapped to the xrpl.network.type label by `networkTypeFromId()`. + */ + std::uint32_t networkId{0}; + + /** + * Whether the exporter connects to the collector over TLS. The three + * paths below apply only when this is true. + */ + bool useTls{false}; + + /** + * CA bundle used to verify the collector. Empty selects the system + * CA store. + */ + std::string tlsCaCertPath; + + /** + * This node's client certificate, presented for mutual TLS. Empty + * means one-way TLS. + */ + std::string tlsClientCertPath; + + /** + * Private key for @ref tlsClientCertPath. + */ + std::string tlsClientKeyPath; + }; + + /** + * Construct the registry and, when enabled, build the whole metrics + * pipeline: OTLP exporter, periodic reader, MeterProvider and every + * SYNCHRONOUS instrument (counters and histograms). + * + * Doing this in the constructor is what fixes the init order. The + * Application declares its registry before every subsystem, so no + * producer can exist before the instruments do. A failure to build the + * pipeline is logged and leaves the registry a no-op; it never stops the + * node. + * + * @note Invariant for future changes: the constructor may create only + * instruments with NO callback of their own. Push-model counters + * and histograms qualify; app code records into them when it is + * ready. An instrument registered here is live immediately, and + * the reader thread may invoke its callback before the rest of + * the server is built, so any observable whose callback reads + * live server state must be registered later, by the layer that + * owns those callbacks. This applies to observable COUNTERS as + * well as gauges. + * + * @param enabled False makes every method a no-op (telemetry disabled). + * @param journal Log output. + * @param options Endpoint, TLS settings and resource identity, all read + * from config by the caller. See @ref Options. + */ + MetricsRegistry(bool enabled, beast::Journal journal, Options const& options); + + /** + * Stops the pipeline if run() or ~ApplicationImp did not already. + */ + ~MetricsRegistry(); + + /** + * Non-copyable, non-movable. + */ + MetricsRegistry(MetricsRegistry const&) = delete; + MetricsRegistry& + operator=(MetricsRegistry const&) = delete; + + /** + * Flush pending metrics and shut down the pipeline. + * + * Stores `Phase::Stopped` first so `recording()` reads false on every + * later record call, then destroys the SDK provider. meter_ is not + * touched: record threads may still be running, and the gate is what + * keeps them off the dying pipeline. Idempotent. + * + * @pre Anything that observes live server state on the reader thread has + * already been disarmed. Shutting the provider down joins that + * thread, so a caller that has not disarmed its observers leaves a + * narrow race between the final tick and the teardown of what those + * observers read. + */ + void + stop(); + + /** + * @return true if the registry is actively exporting metrics. + */ + [[nodiscard]] bool + isEnabled() const noexcept + { + return enabled_; + } + + /** + * @return true when a record call is safe to run. + * + * False when the registry is disabled, or after stop() has torn down the + * export pipeline. After stop() the SDK's SyncMetricStorage still holds a + * raw pointer to an AggregationConfig owned by a destroyed View, so a + * record with a first-seen attribute set would fire the factory lambda + * and deref that dangling pointer. Every XRPL_METRIC_* macro reads this + * once before touching an instrument. + * + * One acquire atomic load in the hot path. + */ + [[nodiscard]] bool + recording() const noexcept + { +#ifdef XRPL_ENABLE_TELEMETRY + return enabled_ && phase_.load(std::memory_order_acquire) != Phase::Stopped; +#else + return enabled_; +#endif + } + + /** + * @return true when a real exporting pipeline exists, as opposed to the + * no-op meter installed when the pipeline is disabled. + * + * A meter() check cannot answer this. The registry always hands out a + * meter, so registering instruments on a no-op one would report success + * and export nothing. Ask this before registering an observable + * instrument. + * + * @note Not thread-safe against stop(), which drops the provider this + * reads. Call it from the server lifecycle thread, like the constructor + * and stop(). + */ + [[nodiscard]] bool + hasPipeline() const noexcept; + + // ----------------------------------------------------------------- + // Synchronous instrument recording (called from PerfLog hot paths) + // ----------------------------------------------------------------- + + /** + * Record an RPC method call start. + * @param method The RPC method name (e.g. "server_info"). + */ + void + recordRpcStarted(std::string_view method); + + /** + * Record an RPC method call completion. + * @param method The RPC method name. + * @param durationUs Execution time in microseconds. + */ + void + recordRpcFinished(std::string_view method, std::int64_t durationUs); + + /** + * Record an RPC method call error. + * @param method The RPC method name. + * @param durationUs Execution time in microseconds. + */ + void + recordRpcErrored(std::string_view method, std::int64_t durationUs); + + /** + * The `handler` label value used for any job name that fails the + * sanitiser's all-ASCII-letters rule. + * + * Public because both sanitiseHandler() and its unit tests must agree + * on the exact fallback token; a test asserting against its own copy + * of the string would not catch a change made here. + * + * Declared as std::string_view rather than the `constexpr char k[]` + * form used for instrument names in MetricsRegistry.cpp: this value is + * *returned* by sanitiseHandler(), whose return type is + * std::string_view, and is compared against std::string_view in tests. + * Matching the type avoids array-to-pointer decay and a needless + * strlen at each use. + */ + static constexpr std::string_view kHandlerOther{"other"}; + + /** + * Reduce a job name to a bounded-cardinality `handler` label value. + * + * A job type can have several producers — both `RcvGetLedger` and + * `RcvGetObjByHash` run as `JtLedgerReq` — so `job_type` alone cannot + * attribute a latency spike to one of them. The job name can, but it + * cannot be used raw: two names embed a ledger sequence number + * (`"Pub" + std::to_string(seq)` in LedgerPersistence.cpp and + * `"OB" + std::to_string(...)` in OrderBookDBImpl.cpp), which would + * mint a fresh Prometheus series for every ledger. + * + * The rule is therefore: keep the name only when it is non-empty and + * every character is an ASCII letter; otherwise return `"other"`. + * Both dynamic names always contain digits, so they always fold to + * `"other"`, while every all-letter name is a compile-time literal. + * The label domain is thus a function of the literals present in the + * source — 43 names plus `"other"` at the time of writing — and + * cannot grow at runtime. A name added later that does not satisfy + * the rule degrades to `"other"` rather than becoming unbounded, + * which is a stronger guarantee than an allowlist that would have to + * be maintained by hand. + * + * Defined inline so it is available in a build without telemetry and + * usable in a constant expression. + * + * @param name The job name as passed to JobQueue::addJob. + * @return @p name when it is non-empty and all ASCII letters, else + * kHandlerOther. + * + * @note Pure and reentrant: holds no state, performs no I/O, and is + * safe to call concurrently from any thread. + * @note The letter test is an explicit ASCII range check rather than + * std::isalpha, which classifies by the current C locale. A + * locale-dependent test could admit non-ASCII bytes and so + * weaken the cardinality bound this function exists to provide. + * @note When the name is kept, the returned view aliases @p name, so + * it must not outlive the caller's buffer. The kHandlerOther case + * returns a view of a static constant and is always valid. + * + * Example: + * @code + * sanitiseHandler("RcvGetObjByHash"); // "RcvGetObjByHash" + * sanitiseHandler("Pub94512331"); // kHandlerOther (digits) + * sanitiseHandler(""); // kHandlerOther (empty) + * @endcode + */ + [[nodiscard]] static constexpr std::string_view + sanitiseHandler(std::string_view name) noexcept + { + auto const isAsciiLetter = [](char const c) { + return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z'); + }; + + if (name.empty() || !std::ranges::all_of(name, isAsciiLetter)) + return kHandlerOther; + + return name; + } + + /** + * Divide a cumulative total by its count, optionally scaled, reporting + * absence rather than zero when the count is zero. + * + * Every cumulative counter this registry publishes has a companion mean + * that is only defined once the counter has moved. Reporting such a mean + * as `0` is worse than not reporting it: `0` is a plausible reading, so a + * dashboard draws a flat line at the bottom of the axis and an operator + * concludes "reads are instant" when the truth is "nothing has been + * read". Returning std::nullopt makes the caller skip the observation, so + * the series has a genuine gap instead. + * + * @p scale exists because the gauge these feed is integral. A mean writer + * depth of 1.4 truncates to 1, which is indistinguishable from a healthy + * 1.0, so the caller scales by 100 and says so in the metric name. + * + * The arithmetic divides before scaling and scales the remainder + * separately, so a long-lived node cannot overflow the product. Should + * the result still exceed the gauge's range it saturates at + * INT64_MAX rather than wrapping, because a wrapped gauge reads as a + * sudden healthy-looking dip. + * + * Defined inline for the same reason as sanitiseHandler(); constexpr so + * the cases below are checked at compile time. + * + * @param total Cumulative numerator (e.g. summed microseconds). + * @param count Number of samples in @p total. + * @param scale Fixed-point multiplier applied to the quotient. Must be + * at least 1; 0 is meaningless and yields std::nullopt. + * @return The scaled mean, or std::nullopt when @p count is 0 (mean + * undefined) or @p scale is 0. + * + * @note Pure and reentrant: holds no state and performs no I/O. + * @note Truncates toward zero, like integer division. A mean of 9.9 us + * reads as 9 at @p scale 1 and as 990 at @p scale 100. + * + * Example: + * @code + * scaledMean(500, 4); // 125 -- mean microseconds + * scaledMean(7, 5, 100); // 140 -- mean 1.4, scaled by 100 + * scaledMean(500, 0); // nullopt -- no samples, so no mean + * @endcode + */ + [[nodiscard]] static constexpr std::optional + scaledMean(std::uint64_t total, std::uint64_t count, std::uint64_t scale = 1) noexcept + { + if (count == 0 || scale == 0) + return std::nullopt; + + constexpr auto kInt64Max = + static_cast(std::numeric_limits::max()); + + auto const whole = total / count; + if (whole > kInt64Max / scale) + return static_cast(kInt64Max); + + // Scale the remainder too, so `scale` recovers the fractional digits + // it exists for. Skipped when the product itself would overflow, at + // which point it is worth less than one part in 2^63 of the result. + auto const remainder = total % count; + std::uint64_t fraction = 0; + if (remainder <= std::numeric_limits::max() / scale) + fraction = remainder * scale / count; + + auto const scaled = whole * scale; + if (scaled > kInt64Max - fraction) + return static_cast(kInt64Max); + + return static_cast(scaled + fraction); + } + + /** + * Read one comma-separated segment of a complete-ledger range string. + * + * The producer is xrpl::to_string(RangeSet), documented in + * xrpl/basics/RangeSet.h. It renders an interval as `first-last`, and an + * interval whose first equals its last as a bare sequence number. A segment + * with no dash is therefore a range of one ledger, not a malformed one. + * + * Defined inline for the same reason as sanitiseHandler(). + * + * @param segment One segment, already split on ','. Leading or trailing + * whitespace is rejected, because the producer emits none. + * @return The inclusive first and last sequence of the range. The two are + * equal for a single-ledger range. std::nullopt when @p segment is not + * something this producer can emit. + * + * @note Pure and reentrant: holds no state, performs no I/O, and is safe to + * call concurrently from any thread. + * @note Reports malformed input instead of throwing, so one unreadable + * segment costs its own range and not every range after it. + * @note A reversed range such as "9-4" is returned as given. RangeSet + * cannot emit one. + * + * Example: + * @code + * parseLedgerRange("32570-50000"); // {32570, 50000} + * parseLedgerRange("5000"); // {5000, 5000} -- one ledger + * parseLedgerRange("5-"); // nullopt + * @endcode + */ + [[nodiscard]] static std::optional> + parseLedgerRange(std::string_view segment) noexcept + { + auto const parseSeq = [](std::string_view text) -> std::optional { + std::uint32_t value = 0; + auto const* const begin = text.data(); + auto const* const end = begin + text.size(); + auto const [ptr, ec] = std::from_chars(begin, end, value); + + // from_chars stops at the first character it cannot use, so the + // whole segment counts as read only when it consumed all of it. + if (ec != std::errc{} || ptr != end) + return std::nullopt; + + return value; + }; + + auto const dash = segment.find('-'); + if (dash == std::string_view::npos) + { + auto const only = parseSeq(segment); + if (!only) + return std::nullopt; + + return std::pair{*only, *only}; + } + + auto const first = parseSeq(segment.substr(0, dash)); + auto const last = parseSeq(segment.substr(dash + 1)); + if (!first || !last) + return std::nullopt; + + return std::pair{*first, *last}; + } + + /** + * Record a job enqueued event. + * @param jobType The job type name (e.g. "ledgerData"). + * @param jobName The addJob name, reduced to a bounded `handler` + * label by sanitiseHandler(). Distinguishes producers + * that share a job type. + */ + void + recordJobQueued(std::string_view jobType, std::string_view jobName); + + /** + * Record a job start event. + * @param jobType The job type name. + * @param jobName The addJob name; see recordJobQueued(). + * @param queuedDurUs Time the job spent waiting in the queue (us). + */ + void + recordJobStarted(std::string_view jobType, std::string_view jobName, std::int64_t queuedDurUs); + + /** + * Record a job finish event. + * @param jobType The job type name. + * @param jobName The addJob name; see recordJobQueued(). + * @param runningDurUs Execution time in microseconds. + */ + void + recordJobFinished( + std::string_view jobType, + std::string_view jobName, + std::int64_t runningDurUs); + + // ----------------------------------------------------------------- + // External dashboard parity counters + // ----------------------------------------------------------------- + + /** + * Increment the ledgers_closed_total counter. + * + * @note Currently has no callers: the ledgers_closed_total counter is + * incremented at its consensus call site via the XRPL_METRIC_COUNTER_INC + * macro (see MetricMacros.h). This method and its eagerly-created + * counter are retained as a fallback and are slated for removal in a + * separate cleanup once the macro path has proven out. + */ + void + incrementLedgersClosed(); + + /** + * Increment the validations_sent_total counter. + * Called from RCLConsensus::Adaptor::validate() when a validation + * is produced and broadcast. + */ + void + incrementValidationsSent(); + + /** + * Increment the validations_checked_total counter. + * Called from NetworkOPs::recvValidation() when a network validation + * is received and checked. + */ + void + incrementValidationsChecked(); + + /** + * Increment the ledger_history_mismatch_total counter for a reason. + * Called from LedgerHistory::handleMismatch() once the mismatch has + * been classified. The reason label turns fork diagnosis from a + * log-grep into a queryable time series. + * @param reason Classified mismatch cause (e.g. "prior_ledger", + * "close_time", "consensus_txset", "same_txset_diff_result", + * "unknown"). + */ + void + incrementLedgerHistoryMismatch(std::string_view reason); + + /** + * Increment the txq_expired_total counter. + * Called from TxQ::processClosedLedger() for each queued transaction + * removed because its LastLedgerSequence has passed — submitters who + * under-bid the escalating fee and were never included. + */ + void + incrementTxqExpired(); + + /** + * Increment the txq_dropped_total{reason} counter. + * Called from TxQ::apply() when a transaction is refused admission to + * the queue (e.g. the queue is full). Distinct from expiry (already + * queued) and from jq_trans_overflow (job queue, not TxQ). + * @param reason Admission-control rejection cause (e.g. "queue_full"). + */ + void + incrementTxqDropped(std::string_view reason); + +#ifdef XRPL_ENABLE_TELEMETRY + /** + * Access the validation agreement tracker. + * Used by consensus and ledger hooks to record our validations and + * network validations so the tracker can compute agreement percentages. + * + * Guarded, along with the tracker itself, because only the observable-gauge + * callbacks read it and those exist only in this configuration. Recording + * into it is not free: each call takes its lock and inserts an entry. + * @return Reference to the internal ValidationTracker instance. + */ + [[nodiscard]] ValidationTracker& + getValidationTracker() + { + return validationTracker_; + } + + /** + * Access the shared OTel Meter for call-site instrument creation. + * Used by the XRPL_METRIC_* macros (MetricMacros.h) so new synchronous + * counters/histograms can be declared at their call site instead of as + * MetricsRegistry members. + * + * Invariant: never empty while recording() is true. The constructor sets + * it to the real meter, or to a no-op meter when the pipeline failed to + * build, and never writes it again, so reads need no lock. After stop() + * the meter's SDK context is gone; the macros gate on recording() first, + * so no caller reaches it then. + * + * @return The shared Meter. + */ + [[nodiscard]] opentelemetry::nostd::shared_ptr + meter() const noexcept + { + return meter_; + } +#endif + +private: + /** + * Master enable flag; when false all methods are no-ops. + */ + bool const enabled_; + +#ifdef XRPL_ENABLE_TELEMETRY + /** + * Tracks validation agreement between this node and the network. + * + * Guarded because reconcile() -- which resolves and then prunes recorded + * events -- runs only from the observable-gauge callbacks. Recording + * without it accumulates one entry per validated ledger, so the tracker + * exists only where something drains it. + */ + ValidationTracker validationTracker_; + + /** + * Journal for logging. + */ + beast::Journal const journal_; + + /** + * Where the registry is in its life. Construction ends in `Ready`; + * stop() moves to `Stopped`. + * + * After `Stopped` the SDK pipeline is gone. recording() reads false, so + * no macro touches meter_ or a cached instrument. + */ + enum class Phase { Ready, Stopped }; + + /** + * Current phase. Written from the server lifecycle thread with release + * ordering; read from record threads via `recording()` with acquire + * ordering, so no record starts once stop() has stored `Stopped`. + */ + std::atomic phase_{Phase::Ready}; + + /** + * The SDK MeterProvider that owns the export pipeline. + */ + std::shared_ptr provider_; + + /** + * The Meter used to create all instruments. + */ + opentelemetry::nostd::shared_ptr meter_; + + // --- Synchronous instruments (RPC) --- + /** + * Counter: rpc_method_started_total{method=""} + */ + opentelemetry::nostd::unique_ptr> rpcStartedCounter_; + /** + * Counter: rpc_method_finished_total{method=""} + */ + opentelemetry::nostd::unique_ptr> rpcFinishedCounter_; + /** + * Counter: rpc_method_errored_total{method=""} + */ + opentelemetry::nostd::unique_ptr> rpcErroredCounter_; + /** + * Histogram: rpc_method_us{method=""} + */ + opentelemetry::nostd::unique_ptr> + rpcDurationHistogram_; + + // --- Synchronous instruments (Job Queue) --- + // All five carry handler="" in addition to + // job_type, so producers that share a job type stay distinguishable. + /** + * Counter: job_queued_total{job_type="",handler=""} + */ + opentelemetry::nostd::unique_ptr> jobQueuedCounter_; + /** + * Counter: job_started_total{job_type="",handler=""} + */ + opentelemetry::nostd::unique_ptr> jobStartedCounter_; + /** + * Counter: job_finished_total{job_type="",handler=""} + */ + opentelemetry::nostd::unique_ptr> jobFinishedCounter_; + /** + * Counter: jobq_stall_total{job_type=""} — one per finished job + * whose run time reached kJobStallThresholdUs. + */ + opentelemetry::nostd::unique_ptr> jobStallCounter_; + /** + * Histogram: job_queued_us{job_type="",handler=""} + */ + opentelemetry::nostd::unique_ptr> + jobQueuedDurationHistogram_; + /** + * Histogram: job_running_us{job_type="",handler=""} + */ + opentelemetry::nostd::unique_ptr> + jobRunningDurationHistogram_; + + // --- External dashboard parity counters --- + /** + * Counter: ledgers_closed_total — incremented each consensus round. + */ + opentelemetry::nostd::unique_ptr> + ledgersClosedCounter_; + /** + * Counter: validations_sent_total — incremented when this node sends a validation. + */ + opentelemetry::nostd::unique_ptr> + validationsSentCounter_; + /** + * Counter: validations_checked_total — incremented for each network validation + * received. + */ + opentelemetry::nostd::unique_ptr> + validationsCheckedCounter_; + /** + * Counter: ledger_history_mismatch_total{reason} — incremented per classified + * built-vs-validated ledger mismatch. + */ + opentelemetry::nostd::unique_ptr> + ledgerHistoryMismatchCounter_; + /** + * Counter: txq_expired_total — incremented per transaction expired out of the + * transaction queue. + */ + opentelemetry::nostd::unique_ptr> txqExpiredCounter_; + /** + * Counter: txq_dropped_total{reason} — incremented when a transaction is refused + * admission to the queue. + */ + opentelemetry::nostd::unique_ptr> txqDroppedCounter_; + + /** + * Build the OTLP/HTTP exporter, periodic reader, resource attributes and + * histogram views, then create the MeterProvider and meter. Extracted + * from the constructor to keep each function under the 80-line limit. + * + * @param options Endpoint, TLS settings and resource identity, forwarded + * unchanged from the constructor. See @ref Options. + */ + void + initExporterAndProvider(Options const& options); + + /** + * Create the synchronous instruments (RPC and job-queue counters and + * histograms, plus the external dashboard parity counters). Extracted + * from the constructor to keep each function under the 80-line limit. + */ + void + initSyncInstruments(); + + /** + * Give up the pipeline after a build failure: drop the provider, hand + * out a no-op meter so every call site still gets an instrument, and log + * why. The registry stays enabled and inert for the process. + * + * @param reason What failed, for the log line. + */ + void + disablePipeline(std::string_view reason); +#endif // XRPL_ENABLE_TELEMETRY +}; + +} // namespace xrpl::telemetry diff --git a/src/xrpld/telemetry/ValidationTracker.h b/include/xrpl/telemetry/ValidationTracker.h similarity index 100% rename from src/xrpld/telemetry/ValidationTracker.h rename to include/xrpl/telemetry/ValidationTracker.h diff --git a/src/libxrpl/telemetry/MetricsRegistry.cpp b/src/libxrpl/telemetry/MetricsRegistry.cpp new file mode 100644 index 0000000000..6c16c649b1 --- /dev/null +++ b/src/libxrpl/telemetry/MetricsRegistry.cpp @@ -0,0 +1,734 @@ +/** + * MetricsRegistry implementation — the OpenTelemetry metrics pipeline. + * + * This file contains: + * - Construction / destruction logic for the OTel MeterProvider pipeline. + * - Synchronous instrument creation (counters, histograms) for RPC, job + * queue and the external dashboard parity counters. + * - The record / increment methods app code pushes values through. + * - No-op stubs when XRPL_ENABLE_TELEMETRY is not defined. + */ + +// On Windows, OTel's spin_lock_mutex.h (transitively included from +// MetricsRegistry.h) defines _WINSOCKAPI_ and includes . +// This poisons the include state for boost/asio/detail/socket_types.hpp, +// which requires winsock2.h to be included first. Pre-including the +// boost/asio socket types header gets winsock2.h in before the OTel +// headers can interfere. +#ifdef _MSC_VER +#include +#endif + +#include + +// Unguarded because the constructor's `beast::Journal journal` parameter is +// declared in both configurations; only the member it initialises is guarded. +#include + +#ifdef XRPL_ENABLE_TELEMETRY + +#include +#include +#include +#include +#include +#include +// For networkTypeFromId(), the one xrpl.network.type mapping both export +// paths use, plus noopMeter() and the shared meter name and version. +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace metric_sdk = opentelemetry::sdk::metrics; +namespace otlp_http = opentelemetry::exporter::otlp; +// Not `resource`: that would collide with xrpl::resource (the resource-accounting +// namespace), which encloses every use site below. Inner-scope lookup would find +// that namespace instead of this file-scope alias. +namespace otel_resource = opentelemetry::sdk::resource; + +namespace { + +// Microsecond-valued duration histogram instrument names. Each is +// referenced twice — once to register the explicit-bucket view and once +// to create the instrument — so they are named constants to keep the two +// sites in sync (a mismatch would silently drop the bucket override). +constexpr char kJobQueuedDurationUs[] = "job_queued_us"; +constexpr char kJobRunningDurationUs[] = "job_running_us"; +constexpr char kRpcMethodDurationUs[] = "rpc_method_us"; + +// Millisecond-valued duration histogram instrument names. Same +// register-then-create pairing as the microsecond names above, so the same +// reason applies for naming them: the view and the record site must agree. +// +// consensus_round_duration_ms is recorded from RCLConsensus at the call site +// (via XRPL_METRIC_HISTOGRAM_RECORD, which creates the instrument lazily +// there), not created here. Only the VIEW is registered here, because a view +// matches by instrument name and must exist before the instrument is first +// used — the MeterProvider is built with the view registry, and the round +// histogram is not created until the first consensus round completes, well +// after start(). +constexpr char kConsensusRoundDurationMs[] = "consensus_round_duration_ms"; + +/** + * Register an explicit-bucket histogram view. + * + * The SDK's default boundaries top out at 10,000, so any instrument whose + * values exceed that saturates and every quantile reads as the ceiling. The + * floor matters just as much and is easier to miss: a ladder whose first edge + * sits above the mass of the distribution makes every low quantile an + * interpolation inside bucket 0 -- a number derived from the bucket edge + * rather than from any sample. Both ends are chosen from measured + * distributions in HistogramBuckets.h. + * + * @param views The registry to add the view to. + * @param name Instrument name to match (e.g. "job_running_us"). + * @param boundaries Bucket upper bounds, ascending. + */ +void +addHistogramView( + metric_sdk::ViewRegistry& views, + std::string const& name, + std::vector boundaries) +{ + auto config = std::make_shared(); + config->boundaries_ = std::move(boundaries); + + auto selector = metric_sdk::InstrumentSelectorFactory::Create( + metric_sdk::InstrumentType::kHistogram, name, ""); + auto meterSelector = metric_sdk::MeterSelectorFactory::Create( + std::string(xrpl::telemetry::kMeterName), std::string(xrpl::telemetry::kMeterVersion), ""); + auto view = + metric_sdk::ViewFactory::Create(name, "", metric_sdk::AggregationType::kHistogram, config); + + views.AddView(std::move(selector), std::move(meterSelector), std::move(view)); +} + +/** + * Register the microsecond-ladder view for a duration instrument. + * + * Job wait/run times and RPC latencies routinely exceed the SDK default + * ceiling, so they all share `buckets::kMicrosecondBuckets`. + * + * @param views The registry to add the view to. + * @param name Instrument name to match (e.g. "job_running_us"). + */ +void +addMicrosecondHistogramView(metric_sdk::ViewRegistry& views, std::string const& name) +{ + addHistogramView( + views, + name, + xrpl::telemetry::buckets::toVector(xrpl::telemetry::buckets::kMicrosecondBuckets)); +} + +/** + * Register the explicit-bucket view for a consensus-round duration in + * MILLISECONDS. + * + * The round histogram needs its own boundaries for two reasons. The SDK + * default tops out at 10,000 ms, and a recovering or stalled node routinely + * rounds slower than that — the consensus parameters themselves allow up to + * `ledgerAbandonConsensus` = 120 s — so the default would collapse exactly the + * slow rounds this signal exists to show into one saturated top bucket. And a + * healthy round is about 3-4 s, which the default's coarse spacing near that + * value cannot resolve, so a round drifting from 3 s to 5 s would not move any + * quantile. + * + * Boundaries: 500ms, 1s, 2s, 3s, 4s, 5s, 7.5s, 10s, 15s, 20s, 30s, 60s, 120s. + * Dense across the healthy 2-5 s band, then widening to the 120 s abandon + * limit so a stalled round still lands in a real bucket. + * + * @param views The registry to add the view to. + * @param name Instrument name to match ("consensus_round_duration_ms"). + */ +void +addRoundDurationHistogramView(metric_sdk::ViewRegistry& views, std::string const& name) +{ + addHistogramView( + views, + name, + {500.0, + 1'000.0, + 2'000.0, + 3'000.0, + 4'000.0, + 5'000.0, + 7'500.0, + 10'000.0, + 15'000.0, + 20'000.0, + 30'000.0, + 60'000.0, + 120'000.0}); +} + +/** + * Register the seconds-ladder view for an online-delete rotation phase. + * + * Rotation phases run from seconds to many minutes, far past the SDK default + * ceiling, so they share `buckets::kRotationPhaseSecondsBuckets`. + * + * @param views The registry to add the view to. + * @param name Instrument name to match ("rotation_phase_duration_seconds"). + */ +void +addRotationPhaseHistogramView(metric_sdk::ViewRegistry& views, std::string const& name) +{ + addHistogramView( + views, + name, + xrpl::telemetry::buckets::toVector(xrpl::telemetry::buckets::kRotationPhaseSecondsBuckets)); +} + +} // namespace + +#endif // XRPL_ENABLE_TELEMETRY + +namespace xrpl::telemetry { + +MetricsRegistry::MetricsRegistry( + [[maybe_unused]] bool enabled, + [[maybe_unused]] beast::Journal journal, + [[maybe_unused]] Options const& options) + : enabled_(enabled) +#ifdef XRPL_ENABLE_TELEMETRY + , journal_(journal) +#endif +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (!enabled_) + return; + + // useTls is logged because a collector that requires TLS rejects a + // plaintext exporter with no local error. The paths are left out. + JLOG(journal_.info()) << "MetricsRegistry: starting, endpoint=" << options.endpoint + << ", serviceName=" << options.serviceName + << ", serviceVersion=" << options.serviceVersion + << ", instanceId=" << options.serviceInstanceId + << ", nodeId=" << options.nodeId << ", networkId=" << options.networkId + << ", useTls=" << options.useTls; + + // A broken pipeline must not stop the node. The SDK is third-party code, + // so the catch-all is deliberate, as in ~ApplicationImp. + try + { + initExporterAndProvider(options); + + // Rule for anything added below: the constructor may create only + // instruments whose recording is PUSHED from app code -- counters and + // histograms. An instrument registered here is live immediately, and + // the reader thread may invoke a registered callback before the rest + // of the server is built, so any observable whose callback reads live + // server state belongs in the layer that owns those callbacks, not + // here. That includes observable COUNTERS, not just gauges: + // jq_trans_overflow_total was created here and its callback read + // getOverlay(), which asserts overlay_ is non-null. + initSyncInstruments(); + } + catch (std::exception const& e) + { + disablePipeline(e.what()); + return; + } + catch (...) + { + disablePipeline("unknown exception"); + return; + } + + JLOG(journal_.info()) << "MetricsRegistry: provider and instruments ready"; +#endif // XRPL_ENABLE_TELEMETRY +} + +#ifdef XRPL_ENABLE_TELEMETRY +void +MetricsRegistry::disablePipeline(std::string_view reason) +{ + provider_.reset(); + // A no-op meter keeps the invariant the XRPL_METRIC_* macros rely on: an + // enabled registry always has a meter, so every call site gets an inert + // instrument here with no check of its own. + meter_ = noopMeter(kMeterName); + JLOG(journal_.error()) << "MetricsRegistry: metrics pipeline failed to initialise, " + "continuing without native metrics: " + << reason; +} +#endif // XRPL_ENABLE_TELEMETRY + +MetricsRegistry::~MetricsRegistry() +{ + stop(); +} + +#ifdef XRPL_ENABLE_TELEMETRY +void +MetricsRegistry::initExporterAndProvider(Options const& options) +{ + // Configure OTLP/HTTP metric exporter. The TLS settings come from the one + // [telemetry] block that also drives the trace exporter in Telemetry.cpp, + // so both exporters reach the collector on the same terms. + otlp_http::OtlpHttpMetricExporterOptions exporterOpts; + exporterOpts.url = options.endpoint; + if (options.useTls) + { + exporterOpts.ssl_ca_cert_path = options.tlsCaCertPath; + exporterOpts.ssl_client_cert_path = options.tlsClientCertPath; + exporterOpts.ssl_client_key_path = options.tlsClientKeyPath; + } + + auto exporter = otlp_http::OtlpHttpMetricExporterFactory::Create(exporterOpts); + + // Configure periodic reader with 10-second export interval. + metric_sdk::PeriodicExportingMetricReaderOptions readerOpts; + readerOpts.export_interval_millis = std::chrono::milliseconds(10000); + readerOpts.export_timeout_millis = std::chrono::milliseconds(5000); + auto reader = + metric_sdk::PeriodicExportingMetricReaderFactory::Create(std::move(exporter), readerOpts); + + // Stamp the same resource Telemetry::makeMetricsResource() builds for the + // trace pipeline. Both must agree: a node whose service.name or + // xrpl.network.type differs between the two pipelines splits its own + // series, and a dashboard filtering on either label shows only half. + // + // Use std::string, never a string literal: ResourceAttributes stores an + // OTel AttributeValue variant whose char-const* overload binds to bool, + // so a literal would be recorded as the boolean true. + otel_resource::ResourceAttributes attrs; + attrs[opentelemetry::semconv::service::kServiceName] = options.serviceName; + // int64_t, matching the trace resource. The same key with two types would + // give the two pipelines incompatible attribute values. + attrs[std::string(attr::networkId)] = static_cast(options.networkId); + // Derived here rather than passed in, so the id and the type label cannot + // disagree. Same helper the trace path uses. + attrs[std::string(attr::networkType)] = networkTypeFromId(options.networkId); + + // The three below are left off when empty rather than stamped blank. An + // absent label reads as "not reported"; an empty one looks like a value. + if (!options.serviceVersion.empty()) + attrs[opentelemetry::semconv::service::kServiceVersion] = options.serviceVersion; + if (!options.serviceInstanceId.empty()) + attrs[opentelemetry::semconv::service::kServiceInstanceId] = options.serviceInstanceId; + // xrpl.node.id: the same per-node key the trace resource carries, so + // metrics and traces resolve to one node. + if (!options.nodeId.empty()) + attrs[std::string(attr::nodeId)] = options.nodeId; + auto resourceAttrs = otel_resource::Resource::Create(attrs); + + // Build a view registry with explicit buckets for the duration + // histograms. Without this they use the SDK default buckets (max 10,000), + // which saturates every quantile at 10 ms for the µs instruments and at + // 10 s for the round histogram. + auto views = std::make_unique(); + addMicrosecondHistogramView(*views, kJobQueuedDurationUs); + addMicrosecondHistogramView(*views, kJobRunningDurationUs); + addMicrosecondHistogramView(*views, kRpcMethodDurationUs); + // Millisecond-scale: recorded at the RCLConsensus call site, so only the + // view is declared here (see the constant's comment). + addRoundDurationHistogramView(*views, kConsensusRoundDurationMs); + + // Recorded at its SHAMapStoreImp RotationPhase destructor, only the view + // lives here. Seconds ladder from HistogramBuckets.h. + addRotationPhaseHistogramView(*views, metric::rotationPhaseDurationSeconds); + + // Recorded at its PeerImp.cpp call site, not created here, so the name + // comes from the shared constant both sites use. + addMicrosecondHistogramView(*views, kGetObjectLookupUs); + + // Sweep malloc_trim duration. Shares the microsecond ladder rather than + // getting a bespoke one, and the ladder is what makes it readable: a trim on + // a small heap lands in the tens-of-microseconds buckets, while a trim on a + // multi-gigabyte resident heap runs well past 10 ms -- which is exactly the + // large-existing-database case this signal exists to catch. With the SDK + // default ceiling of 10,000 every one of those would collapse into the + // overflow bucket and p95 would read exactly 10 ms however bad it got. The + // shared ladder's upper reaches (25 ms, 50 ms, 100 ms, 250 ms, 500 ms, 1 s + // and beyond) resolve those, and its lower reaches (100 us, 500 us) resolve + // the healthy fresh-node case, so a per-instrument ladder would add a second + // thing to maintain for no extra resolution. + addMicrosecondHistogramView(*views, metric::sweepMallocTrimUs); + + // Millisecond dial/resolve latencies. Both exceed the SDK default ceiling + // of 10,000: the dial timer is 15 s, so without an explicit ladder every + // timed-out dial lands in the overflow bucket and p95 reads exactly 10 s + // however bad it gets. The 15 s boundary sits on its own so a timeout is + // distinguishable from merely slow. + addHistogramView( + *views, + metric::dnsResolveLatencyMs, + {1.0, + 5.0, + 10.0, + 25.0, + 50.0, + 100.0, + 250.0, + 500.0, + 1'000.0, + 2'500.0, + 5'000.0, + 10'000.0, + 15'000.0, + 20'000.0, + 30'000.0}); + addHistogramView( + *views, + metric::overlayDialLatencyMs, + {1.0, + 5.0, + 10.0, + 25.0, + 50.0, + 100.0, + 250.0, + 500.0, + 1'000.0, + 2'500.0, + 5'000.0, + 10'000.0, + 15'000.0, + 20'000.0, + 30'000.0}); + + // The remaining two GetObject histograms are not durations, so the + // microsecond ladder above does not fit them. Both still need explicit + // boundaries: the SDK default stops at 10,000 and both ranges exceed it. + // + // Object counts run 1..kHardMaxReplyNodes (12288). The honest sync path + // asks for at most 8, so the low buckets are fine-grained and the upper + // ones follow the charge size bands (64, 1024) up to the hard cap. + addHistogramView( + *views, kGetObjectRequestObjects, buckets::toVector(buckets::kObjectCountBuckets)); + + // Charge values span 0 (free tier) to ~99k for a full-size all-miss + // request. Boundaries bracket the resource thresholds that decide a + // peer's fate -- kWarningThreshold (5000) and kDropThreshold (25000) -- + // so a dashboard can show how close charges run to each. + addHistogramView(*views, kGetObjectCharge, buckets::toVector(buckets::kChargeBuckets)); + + // The two RPC request-count histograms are recorded at their ServerHandler + // and PathRequest call sites, so the names come from the shared constants + // all three sites use. Both are small counts, and the reason they need a + // view is the FLOOR rather than the ceiling: the SDK default edges start + // 0, 5, 10, 25, so a batch of one to five sub-requests -- the normal case -- + // would land in a single bucket and every quantile over it would be an + // interpolation inside that bucket rather than a measurement. + // + // The object-count ladder is the fit: its 1, 2, 4, 8, 16 edges sit exactly + // where both distributions have their mass. Path counts are hard-bounded at + // kMaxPaths * kMaxAutoSrcCur = 352, well under its 12288 top. Batch sizes + // have no such cap; see the ceiling note in RpcMetricNames.h. + addHistogramView(*views, kRpcBatchSize, buckets::toVector(buckets::kObjectCountBuckets)); + addHistogramView( + *views, kPathfindDiscoveredPaths, buckets::toVector(buckets::kObjectCountBuckets)); + + // Create MeterProvider with resource, then attach the metric reader. + provider_ = metric_sdk::MeterProviderFactory::Create(std::move(views), resourceAttrs); + provider_->AddMetricReader(std::move(reader)); + + // Get a meter for all xrpld instruments. + meter_ = provider_->GetMeter(std::string(kMeterName), std::string(kMeterVersion)); +} + +void +MetricsRegistry::initSyncInstruments() +{ + // RPC per-method counters and histogram. + rpcStartedCounter_ = + meter_->CreateUInt64Counter("rpc_method_started_total", "Total RPC method calls started"); + rpcFinishedCounter_ = meter_->CreateUInt64Counter( + "rpc_method_finished_total", "Total RPC method calls completed successfully"); + rpcErroredCounter_ = meter_->CreateUInt64Counter( + "rpc_method_errored_total", "Total RPC method calls that errored"); + rpcDurationHistogram_ = meter_->CreateDoubleHistogram( + kRpcMethodDurationUs, "RPC method execution time in microseconds"); + + // Job queue per-type counters and histograms. + jobQueuedCounter_ = meter_->CreateUInt64Counter("job_queued_total", "Total jobs enqueued"); + jobStartedCounter_ = meter_->CreateUInt64Counter("job_started_total", "Total jobs started"); + jobFinishedCounter_ = meter_->CreateUInt64Counter("job_finished_total", "Total jobs completed"); + jobStallCounter_ = meter_->CreateUInt64Counter( + metric::jobqStallTotal, "Jobs whose run time reached the 1 s stall threshold"); + jobQueuedDurationHistogram_ = meter_->CreateDoubleHistogram( + kJobQueuedDurationUs, "Time jobs spent waiting in the queue (microseconds)"); + jobRunningDurationHistogram_ = + meter_->CreateDoubleHistogram(kJobRunningDurationUs, "Job execution time in microseconds"); + + // --- External dashboard parity counters --- + ledgersClosedCounter_ = + meter_->CreateUInt64Counter("ledgers_closed_total", "Total ledgers closed by consensus"); + validationsSentCounter_ = meter_->CreateUInt64Counter( + "validations_sent_total", "Total validations sent by this node"); + validationsCheckedCounter_ = meter_->CreateUInt64Counter( + "validations_checked_total", "Total network validations received and checked"); + // state_changes_total is NOT created here. It is emitted at its call site + // (NetworkOPsImp::setMode) through XRPL_METRIC_COUNTER_INC_LABELED so it + // can carry the {from,to} transition labels; a registry-owned instrument + // would only give an unlabelled total. + ledgerHistoryMismatchCounter_ = meter_->CreateUInt64Counter( + metric::ledgerHistoryMismatchTotal, "Total built-vs-validated ledger mismatches by reason"); + txqExpiredCounter_ = meter_->CreateUInt64Counter( + "txq_expired_total", "Total transactions expired out of the transaction queue"); + txqDroppedCounter_ = meter_->CreateUInt64Counter( + "txq_dropped_total", "Total transactions refused admission to the queue by reason"); + // Note: validation_agreements_total / validation_missed_total are monotonic + // ObservableCounters owned by the observable-gauge layer. +} +#endif // XRPL_ENABLE_TELEMETRY + +void +MetricsRegistry::stop() +{ +#ifdef XRPL_ENABLE_TELEMETRY + // Store Stopped with release ordering BEFORE the pipeline goes away. + // Every recording thread reads phase_ through recording() with acquire + // ordering, so any record that has not yet passed the gate will see + // Stopped and skip. Idempotent: destructor calls this after run() or + // ~ApplicationImp already did. + phase_.store(Phase::Stopped, std::memory_order_release); + if (!provider_) + return; + + JLOG(journal_.info()) << "MetricsRegistry: stopping"; + + // meter_ is left alone on purpose. Job threads are still running here and + // may be inside a macro, so writing meter_ would race with their read. + // The recording() gate is what keeps them off the dying pipeline: only the + // macros read meter_, and none of them does so once phase_ is Stopped. + // + // SDK teardown order: Shutdown() stops the PeriodicExportingMetricReader + // thread (so no further gauge callbacks fire) and performs the final + // collect-and-export drain itself. The trailing ForceFlush() is a + // redundant safety net (a no-op once the reader is shut down), then + // reset() destroys the provider. + // + // provider_.reset() destroys MeterProvider -> MeterContext -> ViewRegistry + // -> each View -> its shared_ptr. Live SDK + // SyncMetricStorage instances cached in call-site statics still hold a + // raw AggregationConfig pointer; a Record with a NEW attribute set after + // this point would fire the factory lambda and deref that dangling + // pointer, and a late meter()->CreateXxx would return null. + provider_->Shutdown(); + provider_->ForceFlush(); + provider_.reset(); + + JLOG(journal_.info()) << "MetricsRegistry: stopped"; +#endif // XRPL_ENABLE_TELEMETRY +} + +bool +MetricsRegistry::hasPipeline() const noexcept +{ +#ifdef XRPL_ENABLE_TELEMETRY + return provider_ != nullptr; +#else + return false; +#endif +} + +// ----------------------------------------------------------------- +// Synchronous instrument recording — RPC metrics +// ----------------------------------------------------------------- + +void +MetricsRegistry::recordRpcStarted([[maybe_unused]] std::string_view method) +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (!recording() || !rpcStartedCounter_) + return; + rpcStartedCounter_->Add(1, {{"method", std::string(method)}}); +#endif +} + +void +MetricsRegistry::recordRpcFinished( + [[maybe_unused]] std::string_view method, + [[maybe_unused]] std::int64_t durationUs) +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (!recording() || !rpcFinishedCounter_) + return; + rpcFinishedCounter_->Add(1, {{"method", std::string(method)}}); + if (rpcDurationHistogram_) + { + rpcDurationHistogram_->Record( + static_cast(durationUs), + {{"method", std::string(method)}}, + opentelemetry::context::Context{}); + } +#endif +} + +void +MetricsRegistry::recordRpcErrored( + [[maybe_unused]] std::string_view method, + [[maybe_unused]] std::int64_t durationUs) +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (!recording() || !rpcErroredCounter_) + return; + rpcErroredCounter_->Add(1, {{"method", std::string(method)}}); + if (rpcDurationHistogram_) + { + rpcDurationHistogram_->Record( + static_cast(durationUs), + {{"method", std::string(method)}}, + opentelemetry::context::Context{}); + } +#endif +} + +// ----------------------------------------------------------------- +// Synchronous instrument recording — Job Queue metrics +// ----------------------------------------------------------------- + +void +MetricsRegistry::recordJobQueued( + [[maybe_unused]] std::string_view jobType, + [[maybe_unused]] std::string_view jobName) +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (!recording() || !jobQueuedCounter_) + return; + jobQueuedCounter_->Add( + 1, + {{label::jobType, std::string(jobType)}, + {label::handler, std::string(sanitiseHandler(jobName))}}); +#endif +} + +void +MetricsRegistry::recordJobStarted( + [[maybe_unused]] std::string_view jobType, + [[maybe_unused]] std::string_view jobName, + [[maybe_unused]] std::int64_t queuedDurUs) +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (!recording() || !jobStartedCounter_) + return; + // Build the attribute pair once: both the counter and the histogram + // must carry the identical label set or they cannot be joined. + std::string const handler(sanitiseHandler(jobName)); + jobStartedCounter_->Add(1, {{label::jobType, std::string(jobType)}, {label::handler, handler}}); + if (jobQueuedDurationHistogram_ && queuedDurUs >= 0) + { + // Guard against negative queued durations: the caller derives this + // from a steady-clock delta that can go slightly negative under clock + // skew or reordering. The OTel SDK rejects negative histogram values + // (logging a warning per call), so skip them rather than spam. + jobQueuedDurationHistogram_->Record( + static_cast(queuedDurUs), + {{label::jobType, std::string(jobType)}, {label::handler, handler}}, + opentelemetry::context::Context{}); + } +#endif +} + +void +MetricsRegistry::recordJobFinished( + [[maybe_unused]] std::string_view jobType, + [[maybe_unused]] std::string_view jobName, + [[maybe_unused]] std::int64_t runningDurUs) +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (!recording() || !jobFinishedCounter_) + return; + std::string const handler(sanitiseHandler(jobName)); + jobFinishedCounter_->Add( + 1, {{label::jobType, std::string(jobType)}, {label::handler, handler}}); + if (jobRunningDurationHistogram_) + { + jobRunningDurationHistogram_->Record( + static_cast(runningDurUs), + {{label::jobType, std::string(jobType)}, {label::handler, handler}}, + opentelemetry::context::Context{}); + } + // One compare per job finish. A process-wide freeze shows up here as + // several job types crossing the bar in the same second. + if (runningDurUs >= kJobStallThresholdUs && jobStallCounter_) + jobStallCounter_->Add(1, {{label::jobType, std::string(jobType)}}); +#endif +} + +// ----------------------------------------------------------------- +// External dashboard parity counter increments +// ----------------------------------------------------------------- + +void +MetricsRegistry::incrementLedgersClosed() +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (recording() && ledgersClosedCounter_) + ledgersClosedCounter_->Add(1); +#endif +} + +void +MetricsRegistry::incrementValidationsSent() +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (recording() && validationsSentCounter_) + validationsSentCounter_->Add(1); +#endif +} + +void +MetricsRegistry::incrementValidationsChecked() +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (recording() && validationsCheckedCounter_) + validationsCheckedCounter_->Add(1); +#endif +} + +void +MetricsRegistry::incrementLedgerHistoryMismatch(std::string_view reason) +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (recording() && ledgerHistoryMismatchCounter_) + ledgerHistoryMismatchCounter_->Add(1, {{"reason", std::string(reason)}}); +#endif +} + +void +MetricsRegistry::incrementTxqExpired() +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (recording() && txqExpiredCounter_) + txqExpiredCounter_->Add(1); +#endif +} + +void +MetricsRegistry::incrementTxqDropped(std::string_view reason) +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (recording() && txqDroppedCounter_) + txqDroppedCounter_->Add(1, {{"reason", std::string(reason)}}); +#endif +} + +} // namespace xrpl::telemetry diff --git a/src/xrpld/telemetry/detail/ValidationTracker.cpp b/src/libxrpl/telemetry/detail/ValidationTracker.cpp similarity index 99% rename from src/xrpld/telemetry/detail/ValidationTracker.cpp rename to src/libxrpl/telemetry/detail/ValidationTracker.cpp index d37c97bee8..9f334a4fcb 100644 --- a/src/xrpld/telemetry/detail/ValidationTracker.cpp +++ b/src/libxrpl/telemetry/detail/ValidationTracker.cpp @@ -3,7 +3,7 @@ * Implementation of the ValidationTracker class. */ -#include +#include #include #include diff --git a/src/test/nodestore/DatabaseConfig_test.cpp b/src/test/nodestore/DatabaseConfig_test.cpp index 04669bc6be..ca3f354650 100644 --- a/src/test/nodestore/DatabaseConfig_test.cpp +++ b/src/test/nodestore/DatabaseConfig_test.cpp @@ -9,7 +9,7 @@ // of them reads. Both live in xrpld and are only declared in a // telemetry-enabled build, so the include is guarded like its uses below. #include -#include +#include #endif #include @@ -988,7 +988,7 @@ public: /** * Return a sink callable that appends into @ref emitted. */ - telemetry::MetricsRegistry::ObserveFn + telemetry::AppMetricGauges::ObserveFn fn() { return @@ -1082,7 +1082,7 @@ public: // Fresh store: the eight unconditional labels are published, each at // exactly zero, and NEITHER mean appears. MetricSink fresh; - telemetry::MetricsRegistry::observeNodeStoreTotals(*db, fresh.fn()); + telemetry::AppMetricGauges::observeNodeStoreTotals(*db, fresh.fn()); std::vector const kFreshLabels{ "node_read_bytes", @@ -1139,7 +1139,7 @@ public: BEAST_EXPECT(db.fetchNodeObject(object->getHash(), 0) == nullptr); MetricSink busy; - telemetry::MetricsRegistry::observeNodeStoreTotals(db, busy.fn()); + telemetry::AppMetricGauges::observeNodeStoreTotals(db, busy.fn()); // Ten labels now: the eight above plus both means. BEAST_EXPECT(busy.emitted.size() == 10); @@ -1201,7 +1201,7 @@ public: storeBatch(*mem, batch); MetricSink sink; - telemetry::MetricsRegistry::observeWritePathDetail(*mem, sink.fn()); + telemetry::AppMetricGauges::observeWritePathDetail(*mem, sink.fn()); // Cause as well as state: the store really was written to, so the // emptiness is the std::nullopt branch and not an idle database. BEAST_EXPECT(sink.emitted.empty()); @@ -1223,7 +1223,7 @@ public: // pins the deliberate asymmetry -- zero is meaningful for a gauge and // meaningless for a mean. MetricSink fresh; - telemetry::MetricsRegistry::observeWritePathDetail(*db, fresh.fn()); + telemetry::AppMetricGauges::observeWritePathDetail(*db, fresh.fn()); std::vector const kFreshLabels{"nudb_insert_max_us", "nudb_writers_in_flight"}; BEAST_EXPECT(fresh.names() == kFreshLabels); BEAST_EXPECT(fresh.value("nudb_writers_in_flight") == std::int64_t{0}); @@ -1236,7 +1236,7 @@ public: storeBatch(*db, stored); MetricSink busy; - telemetry::MetricsRegistry::observeWritePathDetail(*db, busy.fn()); + telemetry::AppMetricGauges::observeWritePathDetail(*db, busy.fn()); std::vector const kBusyLabels{ "nudb_insert_max_us", "nudb_insert_mean_us", @@ -1285,7 +1285,7 @@ public: // these would lose the ability to see that nothing happened. AcquireStats const quiet; MetricSink fresh; - telemetry::MetricsRegistry::observeAcquireStats(quiet, fresh.fn()); + telemetry::AppMetricGauges::observeAcquireStats(quiet, fresh.fn()); BEAST_EXPECT(fresh.names() == kLabels); BEAST_EXPECT(fresh.emitted.size() == kLabels.size()); for (auto const& label : kLabels) @@ -1313,7 +1313,7 @@ public: busy.recordSweepEviction(); MetricSink sink; - telemetry::MetricsRegistry::observeAcquireStats(busy, sink.fn()); + telemetry::AppMetricGauges::observeAcquireStats(busy, sink.fn()); BEAST_EXPECT(sink.names() == kLabels); BEAST_EXPECT(sink.value("acquire_deferrals") == std::int64_t{3}); BEAST_EXPECT(sink.value("acquire_timeouts") == std::int64_t{7}); @@ -1350,7 +1350,7 @@ public: return; MetricSink sink; - telemetry::MetricsRegistry::observeReadQueue(*db, sink.fn()); + telemetry::AppMetricGauges::observeReadQueue(*db, sink.fn()); std::vector const kLabels{ "read_queue", "read_request_bundle", "read_threads_running", "read_threads_total"}; diff --git a/src/tests/libxrpl/CMakeLists.txt b/src/tests/libxrpl/CMakeLists.txt index 9d9bc64691..2114975ea6 100644 --- a/src/tests/libxrpl/CMakeLists.txt +++ b/src/tests/libxrpl/CMakeLists.txt @@ -21,10 +21,11 @@ set_target_properties( ) # Lets test sources include the shared helpers as . target_include_directories(xrpl_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) -# Some headers under test live in src/xrpld/ rather than in libxrpl (for -# example app/ledger/AcquireStats.h and telemetry/ValidationTracker.h), so put -# src/ on the include path to reach them as . This is unconditional -# because header-only ones are testable in every build, telemetry or not. +# Two tests reach a header under src/xrpld/ rather than in libxrpl: +# ledger/AcquireStats.cpp includes and +# telemetry/RpcMetricNames.cpp includes . Put src/ on +# the include path so both resolve as . This is unconditional because +# those headers are header-only and testable in every build, telemetry or not. target_include_directories(xrpl_tests PRIVATE ${CMAKE_SOURCE_DIR}/src) target_link_libraries(xrpl_tests PRIVATE GTest::gtest GTest::gmock xrpl.libxrpl) @@ -116,29 +117,6 @@ if(telemetry) "${OTEL_IN_MEMORY_EXPORTER_LIB}" opentelemetry-cpp::opentelemetry-cpp ) -else() - # MetricsRegistry lives in xrpld; compile its .cpp directly into the test - # target so the no-op path can be tested without linking all of xrpld. - # When telemetry=ON, XRPL_ENABLE_TELEMETRY is globally defined and the - # .cpp pulls in xrpld symbols we cannot satisfy here. - target_sources( - xrpl_tests - PRIVATE ${CMAKE_SOURCE_DIR}/src/xrpld/telemetry/MetricsRegistry.cpp - ) endif() -# ValidationTracker lives in src/xrpld/ (not libxrpl), so we compile its -# implementation directly into the test binary and put src/ on the include path -# so its tests can reach headers. -# -# Both are unconditional: the class carries no telemetry guards, so its tests -# compile and run in every build. Gating them would leave the test file (which -# is likewise unguarded) without the header it includes and without the -# definitions it calls. -target_include_directories(xrpl_tests PRIVATE ${CMAKE_SOURCE_DIR}/src) -target_sources( - xrpl_tests - PRIVATE ${CMAKE_SOURCE_DIR}/src/xrpld/telemetry/detail/ValidationTracker.cpp -) - gtest_discover_tests(xrpl_tests DISCOVERY_TIMEOUT 60) diff --git a/src/tests/libxrpl/beast/insight/OTelCollectorHooks.cpp b/src/tests/libxrpl/beast/insight/OTelCollectorHooks.cpp index 2964b3ebe9..5ab29698e1 100644 --- a/src/tests/libxrpl/beast/insight/OTelCollectorHooks.cpp +++ b/src/tests/libxrpl/beast/insight/OTelCollectorHooks.cpp @@ -6,17 +6,13 @@ #include #include +#include #include #include -#include #include -#include -#include #include #include -#include -#include #include #include #include @@ -27,54 +23,10 @@ namespace beast::insight { namespace metrics_api = opentelemetry::metrics; namespace metrics_sdk = opentelemetry::sdk::metrics; -/** - * A MetricReader that collects only when the test asks it to. - * - * The SDK ships only PeriodicExportingMetricReader, whose background thread - * would make these tests depend on timing. MetricReader::Collect() is public - * and synchronous, so a minimal subclass lets a test drive one collection pass - * on the calling thread. That pass is what invokes an observable gauge's - * callback, which is the only path that reaches the collector's hooks. - * - * @code - * auto reader = std::make_shared(); - * provider->AddMetricReader(reader); - * reader->collectOnce(); // runs every registered observable callback - * @endcode - */ -class ManualMetricReader : public metrics_sdk::MetricReader -{ -public: - /** - * @brief Run exactly one collection pass, discarding the metric data. - * - * The tests assert on hook side effects, not on exported points, so the - * callback returns true without inspecting what it was handed. - */ - void - collectOnce() - { - Collect([](metrics_sdk::ResourceMetrics&) { return true; }); - } - - [[nodiscard]] metrics_sdk::AggregationTemporality - GetAggregationTemporality(metrics_sdk::InstrumentType) const noexcept override - { - return metrics_sdk::AggregationTemporality::kCumulative; - } - - bool - OnForceFlush(std::chrono::microseconds) noexcept override - { - return true; - } - - bool - OnShutDown(std::chrono::microseconds) noexcept override - { - return true; - } -}; +// The reader is not specific to this suite -- any test that needs a real SDK +// provider without a background export thread wants it -- so its one +// definition lives in the test helpers. +using xrpl::test::ManualMetricReader; /** * Installs a real SDK MeterProvider so observable gauges actually fire. diff --git a/src/tests/libxrpl/helpers/ManualMetricReader.h b/src/tests/libxrpl/helpers/ManualMetricReader.h new file mode 100644 index 0000000000..9f528e4db0 --- /dev/null +++ b/src/tests/libxrpl/helpers/ManualMetricReader.h @@ -0,0 +1,80 @@ +#pragma once + +/** + * @file ManualMetricReader.h + * A metric reader that collects on demand, for tests that need a real SDK + * MeterProvider without a background export thread. + * + * Guarded as a whole: every type it names comes from the OpenTelemetry metrics + * SDK, which is on the link line only when XRPL_ENABLE_TELEMETRY is defined. + */ + +#ifdef XRPL_ENABLE_TELEMETRY + +#include +#include +#include +#include + +#include + +namespace xrpl::test { + +/** + * A MetricReader that collects only when the test asks it to. + * + * The SDK ships only PeriodicExportingMetricReader, whose background thread + * would make a test depend on timing. MetricReader::Collect() is public and + * synchronous, so a minimal subclass lets a test drive one collection pass on + * the calling thread. That pass is what invokes an observable instrument's + * callback. + * + * The SDK types are spelled in full rather than through a namespace alias. An + * alias here would be a member of xrpl::test, and a translation unit that + * pulled that namespace in wholesale could then find two spellings of the same + * short name. + * + * @code + * auto reader = std::make_shared(); + * provider->AddMetricReader(reader); + * reader->collectOnce(); // runs every registered observable callback + * @endcode + */ +class ManualMetricReader : public opentelemetry::sdk::metrics::MetricReader +{ +public: + /** + * @brief Run exactly one collection pass, discarding the metric data. + * + * A caller asserting on callback side effects does not need the exported + * points, so the callback returns true without inspecting what it was + * handed. + */ + void + collectOnce() + { + Collect([](opentelemetry::sdk::metrics::ResourceMetrics&) { return true; }); + } + + [[nodiscard]] opentelemetry::sdk::metrics::AggregationTemporality + GetAggregationTemporality(opentelemetry::sdk::metrics::InstrumentType) const noexcept override + { + return opentelemetry::sdk::metrics::AggregationTemporality::kCumulative; + } + + bool + OnForceFlush(std::chrono::microseconds) noexcept override + { + return true; + } + + bool + OnShutDown(std::chrono::microseconds) noexcept override + { + return true; + } +}; + +} // namespace xrpl::test + +#endif // XRPL_ENABLE_TELEMETRY diff --git a/src/tests/libxrpl/nodestore/Backend.cpp b/src/tests/libxrpl/nodestore/Backend.cpp index 0b37815843..6935409aa4 100644 --- a/src/tests/libxrpl/nodestore/Backend.cpp +++ b/src/tests/libxrpl/nodestore/Backend.cpp @@ -217,7 +217,7 @@ INSTANTIATE_TEST_SUITE_P( // parameterized suite above only reaches when XRPL_ROCKSDB_AVAILABLE. // // Why absence and not zeros: the exporter skips the whole nudb_* label group -// when getWriteStats() is empty (MetricsRegistry.cpp observeWritePathDetail +// when getWriteStats() is empty (AppMetricGauges.cpp observeWritePathDetail // returns early). If the base class returned a default-constructed WriteStats // instead, every non-NuDB node would publish nudb_writers_in_flight=0 and // nudb_insert_max_us=0 -- a perfectly idle write path, on a node whose write diff --git a/src/tests/libxrpl/telemetry/MetricMacros.cpp b/src/tests/libxrpl/telemetry/MetricMacros.cpp index 86f943928a..8e654f0e52 100644 --- a/src/tests/libxrpl/telemetry/MetricMacros.cpp +++ b/src/tests/libxrpl/telemetry/MetricMacros.cpp @@ -19,15 +19,15 @@ #ifdef XRPL_ENABLE_TELEMETRY -#include +#include #include -#include -#include #include #include #include +#include +#include #include #include diff --git a/src/tests/libxrpl/telemetry/MetricsRegistry.cpp b/src/tests/libxrpl/telemetry/MetricsRegistry.cpp index 2938692590..351d64bea9 100644 --- a/src/tests/libxrpl/telemetry/MetricsRegistry.cpp +++ b/src/tests/libxrpl/telemetry/MetricsRegistry.cpp @@ -1,70 +1,50 @@ /** * GTest unit tests for MetricsRegistry. * - * Four independent groups, split by what they can link: + * Four groups. The first three drive the pure static helpers, which are + * constexpr inline in the header and so need nothing on the link line. The + * fourth drives a real registry object. * - * 1. sanitiseHandler() — the `handler` label sanitiser. Runs in **both** - * builds. sanitiseHandler() is a public static constexpr defined inline - * in the header, so it needs no part of MetricsRegistry.cpp on the link - * line. These tests therefore sit outside the guard below; putting them - * inside it would silently compile them out of the telemetry-enabled - * build, which is the build that actually exports the label. + * 1. sanitiseHandler() — the `handler` label sanitiser. * * 2. scaledMean() — the guarded-division helper behind every derived mean - * on the nodestore_state gauge. Also a public static constexpr inline, - * so it runs in both builds for the same reason. + * on the nodestore_state gauge. * * 3. parseLedgerRange() — reads one segment of the complete-ledger range - * string the complete_ledgers gauge publishes. A public static inline, so - * it runs in both builds for the same reason. The last case drives the + * string the complete_ledgers gauge publishes. The last case drives the * real producer, xrpl::to_string(RangeSet), rather than restating its * format. * - * 4. The no-op / telemetry-disabled path — construction (which is where the - * pipeline and the synchronous instruments are built), startAsyncGauges(), - * stop(), and the synchronous record*() methods. Guarded, because when - * XRPL_ENABLE_TELEMETRY is - * defined MetricsRegistry.cpp is not compiled into this binary (see - * src/tests/libxrpl/CMakeLists.txt) and its out-of-line symbols are - * unresolvable here. + * 4. The registry lifecycle — construction, stop(), and the record and + * increment methods. Every test here runs in **both** builds: the core + * is compiled into xrpl.libxrpl, which this binary links either way, so + * with telemetry on these tests drive a real OTel pipeline and with it + * off they drive the no-op stubs. An assertion that holds in only one + * build carries its own #ifdef and says which build it pins. * - * Tests cover: - * - Construction with telemetry disabled (no-op behavior). - * - The startAsyncGauges() / stop() lifecycle when disabled. - * - Synchronous instrument recording methods do not crash when disabled. - * - Double stop() is safe. - * - Destructor handles cleanup without crash. - * - Compile-time-disabled proof for the sync-diagnostics gauges: the whole - * async-gauge registration surface is compiled away, and a full disabled - * lifecycle never touches any ServiceRegistry service -- including the - * Overlay and AmendmentTable the peer and amendment gauges would read. + * What group 4 pins about stop(), and what it does not: * - * NOTE: These tests only exercise the no-op path (telemetry disabled). - * When XRPL_ENABLE_TELEMETRY is defined, MetricsRegistry.cpp pulls in - * xrpld symbols that cannot be linked into this standalone test binary, - * so the tests are compiled out. + * stop() stores Phase::Stopped before it destroys the SDK provider, and every + * record method reads that phase through recording() first. Without the store, + * a record carrying a first-seen attribute set would reach an + * AggregationConfig that the destroyed View owned. The tests below assert that + * the gate is shut after stop() and that a record past it is inert. That pins + * the gate. It does not prove the memory is safe: with no sanitizer, a read of + * freed memory can still pass. A sanitizer build running these same tests is + * what would catch a regression in the memory itself. * - * CONSEQUENCE for the sync-diagnostics gauges (`unl_quorum`, - * `clock_close_offset_seconds`, `sync_state`, - * `server_stall_events_total`, `sync_acquire`, `shamap_cache_hit_rate`, - * `jobq_saturation`, `peer_ledger_supply`, - * `peerfinder_slot_census`, `amendment_block`, `nodestore_state`): - * this file CANNOT assert an observed gauge - * value, because on this build the gauges do not exist -- their registration - * methods and the OTel instrument members are inside - * `#ifdef XRPL_ENABLE_TELEMETRY`, and there is no MeterProvider at all. What - * is provable here, and what the tests below assert, is the complementary - * half: that nothing is registered and no service is consulted. The exact - * observed values (trusted_keys=5, quorum=4, offset=-3, the sync_state / - * stall-episode values, the acquire-progress / cache-hit-rate values, the - * per-type backlog / pool-saturation values, the peer-supply / - * slot-census / amendment-countdown values, and the nodestore - * read/write mean-latency values) are - * asserted in MetricMacros.cpp, which is the file compiled when telemetry IS - * enabled. + * Two tests in group 4 assert on the class surface rather than on behaviour: + * `state_changes_total` has no registry-owned increment method, and `meter()` + * is not a member in a telemetry-off build. Both read the surface with a + * `requires` expression, so the compiler decides the property and the test + * reports it. + * + * The observable gauges are not part of this class, and this binary links + * xrpl.libxrpl only, so no gauge value can be observed here. Those values are + * asserted where the gauges live. */ -#include +#include #include @@ -620,609 +600,408 @@ TEST(MetricsRegistryParseLedgerRange, reads_back_what_the_real_producer_wrote) EXPECT_EQ(recovered.size(), ledgers.iterative_size()); } -// When telemetry is globally enabled, MetricsRegistry.cpp requires xrpld -// link dependencies we cannot satisfy in a standalone GTest binary. -#ifndef XRPL_ENABLE_TELEMETRY +// --------------------------------------------------------------------------- +// 4. The registry lifecycle. +// +// The core is compiled into xrpl.libxrpl, which this binary links in both +// builds, so every test below runs in both. The headers here serve only this +// group, and two of them name types that exist in one build only, so they sit +// beside their uses rather than at the top of the file. +// --------------------------------------------------------------------------- -#include -#include #include -#include -#include - -#include -#include - -using namespace xrpl; +#ifdef XRPL_ENABLE_TELEMETRY +#include +#include +#include +#endif namespace { /** - * OTLP/HTTP endpoint given to every registry below. Nothing ever dials it - * -- these tests exercise the no-op path -- it just has to be a plausible URL. - * It reaches the constructor through @ref kTestOptions. + * OTLP/HTTP endpoint every registry below is given. + * + * Port 1 has no listener, so the one export attempt a test can provoke gets an + * immediate connection refusal. The reader's interval is 10 s, so no periodic + * export fires inside a test; stop() is what exports, because Shutdown() + * performs a final collect-and-export drain. A routable-but-dead address would + * make every test that calls stop() wait out the 5 s export timeout. */ -constexpr std::string_view kTestEndpoint{"http://localhost:4318/v1/metrics"}; +constexpr std::string_view kTestEndpoint{"http://127.0.0.1:1/v1/metrics"}; /** - * The only Options field these tests need. - * - * The constructor takes the Options aggregate, not a string. The other fields - * -- resource identity, network id, TLS paths -- are never read on the no-op - * path, and their defaults already mean "unset". One shared value keeps every - * construction on the same endpoint. + * Resource identity stamped on the test pipeline. Fixed values, so any + * difference a test sees between two registries comes from the objects and not + * from their config. */ -telemetry::MetricsRegistry::Options const kTestOptions{.endpoint = std::string{kTestEndpoint}}; +constexpr std::string_view kTestServiceName{"metrics-registry-test-service"}; +constexpr std::string_view kTestServiceVersion{"0.0.0-test"}; +constexpr std::string_view kTestInstanceId{"metrics-registry-test-instance"}; +constexpr std::string_view kTestNodeId{"metrics-registry-test-node"}; /** - * Minimal mock ServiceRegistry for MetricsRegistry testing. + * Options for every registry below. * - * Only the getMetricsRegistry() call is used in the tests; other methods - * are not invoked because the registry is disabled (enabled=false) so no - * gauge callbacks execute. + * A function rather than a namespace-scope constant: the fields are + * std::string, so a constant would need dynamic initialisation to run before + * the first test. * - * All pure virtual methods throw to catch accidental calls during tests. + * Default-constructed and then assigned, the shape makeMetricsRegistryOptions() + * in Application.cpp uses to build this same struct. Default construction + * leaves no member indeterminate: every std::string is empty, networkId is 0 + * and useTls is false. A designated-initializer list naming a subset would trip + * -Wmissing-designated-field-initializers, an error in this build, and would + * trip it again the next time a field is added to Options. + * + * networkId stays 0 and useTls false, so the three TLS paths stay empty: the + * exporter reads them only over TLS. No test asserts on a resource attribute, + * because nothing here reads exported points back. */ -class MockServiceRegistry : public ServiceRegistry +MetricsRegistry::Options +testOptions() { - [[noreturn]] static void - throwUnimplemented() - { - throw std::logic_error("MockServiceRegistry: method not implemented"); - } - -public: - // ServiceRegistry interface — stubs that should never be called. - CollectorManager& - getCollectorManager() override - { - throwUnimplemented(); - } - Family& - getNodeFamily() override - { - throwUnimplemented(); - } - TimeKeeper& - getTimeKeeper() override - { - throwUnimplemented(); - } - JobQueue& - getJobQueue() override - { - throwUnimplemented(); - } - NodeCache& - getTempNodeCache() override - { - throwUnimplemented(); - } - CachedSLEs& - getCachedSLEs() override - { - throwUnimplemented(); - } - NetworkIDService& - getNetworkIDService() override - { - throwUnimplemented(); - } - AmendmentTable& - getAmendmentTable() override - { - throwUnimplemented(); - } - HashRouter& - getHashRouter() override - { - throwUnimplemented(); - } - LoadFeeTrack& - getFeeTrack() override - { - throwUnimplemented(); - } - LoadManager& - getLoadManager() override - { - throwUnimplemented(); - } - RCLValidations& - getValidations() override - { - throwUnimplemented(); - } - ValidatorList& - getValidators() override - { - throwUnimplemented(); - } - ValidatorSite& - getValidatorSites() override - { - throwUnimplemented(); - } - ManifestCache& - getValidatorManifests() override - { - throwUnimplemented(); - } - ManifestCache& - getPublisherManifests() override - { - throwUnimplemented(); - } - Overlay& - getOverlay() override - { - throwUnimplemented(); - } - Cluster& - getCluster() override - { - throwUnimplemented(); - } - PeerReservationTable& - getPeerReservations() override - { - throwUnimplemented(); - } - resource::Manager& - getResourceManager() override - { - throwUnimplemented(); - } - node_store::Database& - getNodeStore() override - { - throwUnimplemented(); - } - SHAMapStore& - getSHAMapStore() override - { - throwUnimplemented(); - } - RelationalDatabase& - getRelationalDatabase() override - { - throwUnimplemented(); - } - InboundLedgers& - getInboundLedgers() override - { - throwUnimplemented(); - } - InboundTransactions& - getInboundTransactions() override - { - throwUnimplemented(); - } - TaggedCache& - getAcceptedLedgerCache() override - { - throwUnimplemented(); - } - LedgerMaster& - getLedgerMaster() override - { - throwUnimplemented(); - } - LedgerCleaner& - getLedgerCleaner() override - { - throwUnimplemented(); - } - LedgerReplayer& - getLedgerReplayer() override - { - throwUnimplemented(); - } - PendingSaves& - getPendingSaves() override - { - throwUnimplemented(); - } - // AcquireStats lives in src/xrpld/ and is only forward-declared here; a - // reference return to an incomplete type is fine because this throws. - AcquireStats& - getAcquireStats() override - { - throwUnimplemented(); - } - [[nodiscard]] OpenLedger& - getOpenLedger() override - { - throwUnimplemented(); - } - [[nodiscard]] OpenLedger const& - getOpenLedger() const override - { - throwUnimplemented(); - } - NetworkOPs& - getOPs() override - { - throwUnimplemented(); - } - OrderBookDB& - getOrderBookDB() override - { - throwUnimplemented(); - } - TransactionMaster& - getMasterTransaction() override - { - throwUnimplemented(); - } - TxQ& - getTxQ() override - { - throwUnimplemented(); - } - PathRequestManager& - getPathRequestManager() override - { - throwUnimplemented(); - } - ServerHandler& - getServerHandler() override - { - throwUnimplemented(); - } - perf::PerfLog& - getPerfLog() override - { - throwUnimplemented(); - } - telemetry::Telemetry& - getTelemetry() override - { - throwUnimplemented(); - } - telemetry::MetricsRegistry* - getMetricsRegistry() override - { - return nullptr; - } - [[nodiscard]] bool - isStopping() const override - { - return false; - } - beast::Journal - getJournal(std::string const&) override - { - return beast::Journal(beast::Journal::getNullSink()); - } - boost::asio::io_context& - getIOContext() override - { - throwUnimplemented(); - } - Logs& - getLogs() override - { - throwUnimplemented(); - } - [[nodiscard]] std::optional const& - getTrapTxID() const override - { - static std::optional const kEmpty; - return kEmpty; - } - DatabaseCon& - getWalletDB() override - { - throwUnimplemented(); - } - Application& - getApp() override - { - throwUnimplemented(); - } -}; + MetricsRegistry::Options options; + options.endpoint = std::string{kTestEndpoint}; + options.serviceName = std::string{kTestServiceName}; + options.serviceVersion = std::string{kTestServiceVersion}; + options.serviceInstanceId = std::string{kTestInstanceId}; + options.nodeId = std::string{kTestNodeId}; + return options; +} /** - * Test fixture that provides a MockServiceRegistry and null Journal. + * Call every record and increment method on @p registry once. + * + * All twelve are driven from one place, so a method added to the class + * without a line here reads as an uncovered method rather than as a passing + * test. + * + * @param registry The registry to drive. + * @param tag Folded into every attribute value, so one call's label sets + * are disjoint from another call's. A tag unused before + * stop() is what makes each set first-seen afterwards. + */ +void +recordEverything(MetricsRegistry& registry, std::string const& tag) +{ + registry.recordRpcStarted("started_" + tag); + registry.recordRpcFinished("finished_" + tag, 1000); + registry.recordRpcErrored("errored_" + tag, 500); + registry.recordJobQueued("queued_" + tag, "ProcessLData"); + registry.recordJobStarted("started_" + tag, "RcvGetLedger", 200); + registry.recordJobFinished("finished_" + tag, "RcvGetObjByHash", 3000); + registry.incrementLedgersClosed(); + registry.incrementValidationsSent(); + registry.incrementValidationsChecked(); + registry.incrementLedgerHistoryMismatch("mismatch_" + tag); + registry.incrementTxqExpired(); + registry.incrementTxqDropped("dropped_" + tag); +} + +/** + * Fixture for the lifecycle tests. + * + * Holds the journal only. Each test builds its own registry: the class is + * neither copyable nor movable, and each test needs its own enable flag or its + * own stop ordering. */ class MetricsRegistryTest : public ::testing::Test { protected: - MockServiceRegistry mockApp_; beast::Journal j_{beast::Journal::getNullSink()}; }; } // namespace +// --------------------------------------------------------------------------- +// The disabled path. enabled=false makes every method inert in both builds, so +// every assertion here holds unguarded. +// --------------------------------------------------------------------------- + TEST_F(MetricsRegistryTest, disabled_construction) { - // Construct with enabled=false; should be a no-op. - telemetry::MetricsRegistry const registry(false, mockApp_, j_, kTestOptions); - EXPECT_FALSE(registry.isEnabled()); + MetricsRegistry const registry(false, j_, testOptions()); + + EXPECT_EQ(registry.isEnabled(), false); + + // Mutation: drop the `enabled_ &&` term from recording(). A disabled + // registry would report itself recordable, and every call site would walk + // into an instrument that was never created. + EXPECT_EQ(registry.recording(), false); + + // Mutation: delete `if (!enabled_) return;` from the constructor. A node + // with telemetry off would open an OTLP exporter and start a reader + // thread. In a telemetry-off build the same value comes from the #else + // branch of hasPipeline(). + EXPECT_EQ(registry.hasPipeline(), false); } TEST_F(MetricsRegistryTest, disabled_construct_stop) { - telemetry::MetricsRegistry registry(false, mockApp_, j_, kTestOptions); + MetricsRegistry registry(false, j_, testOptions()); - // stop() should be a no-op when disabled. + registry.stop(); registry.stop(); - // Double stop should be safe. - registry.stop(); -} - -// --------------------------------------------------------------------------- -// The two startup phases: construction, then startAsyncGauges(). -// -// Why two phases: the constructor needs only config strings, so it can run in -// the Application's member-init list, before any subsystem that records a -// metric exists. The observable-instrument callbacks registered by -// startAsyncGauges() read live Application services (getOverlay() asserts -// overlay_ is non-null), so they wait until those services are built. -// -// SCOPE OF THESE TESTS -- read before adding to them. MetricsRegistry.cpp is -// compiled into this binary ONLY when telemetry is OFF -// (src/tests/libxrpl/CMakeLists.txt -- the `else()` branch; when it is ON the -// .cpp needs concrete xrpld types such as LedgerMaster, TxQ, NetworkOPs, -// Overlay and node_store::Database, which a standalone GTest binary cannot -// link). The constructor body and startAsyncGauges() sit inside -// #ifdef XRPL_ENABLE_TELEMETRY, so here they compile to empty bodies. So these -// tests pin the API SURFACE -- that the entry points exist, are callable in -// the documented order, and leave the object usable -- and NOT the gauge -// behaviour. Real coverage of "gauges observe values only after -// startAsyncGauges()" is unreachable from this target; it needs the enabled -// path plus an in-memory metric reader. -// --------------------------------------------------------------------------- - -TEST_F(MetricsRegistryTest, async_gauges_after_construction_is_safe) -{ - telemetry::MetricsRegistry registry(false, mockApp_, j_, kTestOptions); - - // The documented order: instruments at construction, gauges second. - registry.startAsyncGauges(); - - // State: the enable flag is untouched by either phase. Exact value, not - // merely "falsy" -- a phase that flipped it would be a real defect. + // Mutation, telemetry-on build: delete `if (!provider_) return;` from + // stop(). provider_ is null on this path because the constructor returned + // before building it, so the first call would dereference an empty + // shared_ptr. With telemetry off stop() has no body to break, and the three + // values below are what that build pins. EXPECT_EQ(registry.isEnabled(), false); - - // Synchronous recording must work off construction alone. Nothing here - // needs the gauges to be registered. - registry.recordRpcStarted("server_info"); - registry.recordRpcFinished("server_info", 1000); - - registry.stop(); - EXPECT_EQ(registry.isEnabled(), false); -} - -TEST_F(MetricsRegistryTest, async_gauges_twice_is_safe) -{ - telemetry::MetricsRegistry registry(false, mockApp_, j_, kTestOptions); - - // A second arm must be a no-op, not a second set of instruments. On the - // enabled path the Phase guard logs and returns; here the stub returns. - registry.startAsyncGauges(); - registry.startAsyncGauges(); - EXPECT_EQ(registry.isEnabled(), false); - - registry.recordJobQueued("ledgerData", "ProcessLData"); - registry.stop(); -} - -TEST_F(MetricsRegistryTest, async_gauges_respect_the_compile_time_guard) -{ - // Constructed with enabled=true, which on the enabled path would build the - // pipeline and register instruments for real. In this build - // XRPL_ENABLE_TELEMETRY is undefined, so both phases compile to the stub - // branch and neither touches the mock -- every MockServiceRegistry - // accessor throws, so a callback that actually ran would surface as a - // thrown exception here. - telemetry::MetricsRegistry registry(true, mockApp_, j_, kTestOptions); - - // Cause, not just state: the flag really is true, so the no-op below is - // attributable to the compile-time guard and not to an early enabled_ - // return. - EXPECT_EQ(registry.isEnabled(), true); - - EXPECT_NO_THROW(registry.startAsyncGauges()); - EXPECT_NO_THROW(registry.stop()); - - EXPECT_EQ(registry.isEnabled(), true); + EXPECT_EQ(registry.recording(), false); + EXPECT_EQ(registry.hasPipeline(), false); } TEST_F(MetricsRegistryTest, disabled_recording_methods) { - telemetry::MetricsRegistry registry(false, mockApp_, j_, kTestOptions); + MetricsRegistry registry(false, j_, testOptions()); - // All recording methods should be no-ops (not crash). - registry.recordRpcStarted("server_info"); - registry.recordRpcFinished("server_info", 1000); - registry.recordRpcErrored("ledger", 500); - registry.recordJobQueued("ledgerData", "ProcessLData"); - registry.recordJobStarted("ledgerData", "ProcessLData", 200); - registry.recordJobFinished("ledgerData", "ProcessLData", 3000); + // A crash canary rather than a guard with a single-line mutation: both the + // recording() test and the null-instrument test would have to go before a + // record method faulted here. It is the line a sanitizer build turns into + // real coverage. + EXPECT_NO_THROW(recordEverything(registry, "disabled")); + + // State after the sweep. enabled_ is `bool const`, so no mutation can turn + // the first two lines red on this path; hasPipeline() is the one that can -- + // a record path that assigned provider_ would fail it. + EXPECT_EQ(registry.isEnabled(), false); + EXPECT_EQ(registry.recording(), false); + EXPECT_EQ(registry.hasPipeline(), false); registry.stop(); + EXPECT_EQ(registry.isEnabled(), false); + EXPECT_EQ(registry.recording(), false); } -TEST_F(MetricsRegistryTest, destructor_calls_stop) +// --------------------------------------------------------------------------- +// The enabled path. In a telemetry-on build these drive a real OTLP exporter, +// MeterProvider and set of instruments; in a telemetry-off build they drive the +// stubs, where recording() is just the enable flag and no pipeline exists. +// --------------------------------------------------------------------------- + +TEST_F(MetricsRegistryTest, enabled_registry_records_from_construction) { - { - // Let the destructor handle cleanup. - telemetry::MetricsRegistry const registry(false, mockApp_, j_, kTestOptions); - } - // If we get here without crash, the destructor handled stop. + // The registry is usable the moment it exists, which is why Application + // can declare it ahead of every subsystem that records. Nothing stops it + // here either, so the scope exit also covers the enabled destructor path. + // const because every method this test calls is const. + MetricsRegistry const registry(true, j_, testOptions()); + + // Mutation: isEnabled() returning a literal false. The disabled test + // asserts the opposite value, so only the enabled tests catch this. Not + // "drop the enabled_(enabled) member init" -- enabled_ is `bool const` with + // no default, so a constructor omitting it does not compile. + EXPECT_EQ(registry.isEnabled(), true); + + // Mutation: seed phase_ with Phase::Stopped instead of Phase::Ready. + // Nothing in the class ever stores Ready, so the node would stay silent + // for its whole run. Also catches recording() testing phase_ == Stopped. + EXPECT_EQ(registry.recording(), true); + +#ifdef XRPL_ENABLE_TELEMETRY + // Telemetry-on only, and the property this refactoring exists to test: the + // constructor builds the pipeline, so no later start() call has to. + // Mutation: delete the initExporterAndProvider(options) call from the + // constructor's try block. + EXPECT_EQ(registry.hasPipeline(), true); +#else + // Telemetry-off: the pipeline compiles out, so hasPipeline() is a literal + // false. Mutation: return true from that #else branch, which would tell a + // caller it may register an observable that can never export. + EXPECT_EQ(registry.hasPipeline(), false); +#endif } -// ----------------------------------------------------------------- -// Sync-diagnostics gauges: compile-time-disabled proof. -// -// `unl_quorum` reads ValidatorList::trustedKeyCount() and quorum(); -// `clock_close_offset_seconds` reads TimeKeeper::closeOffset(); `sync_state` and -// `server_stall_events_total` read NetworkOPs and LoadManager; `sync_acquire` -// reads InboundLedgers::acquireProgress() and `shamap_cache_hit_rate` reads the -// node Family's tree-node cache; `jobq_saturation` reads -// JobQueue::getWorkerSaturation(); `peer_ledger_supply` and -// `peerfinder_slot_census` read Overlay::getPeerLedgerSupply() / -// getSlotCensus() and `amendment_block` reads -// AmendmentTable::firstUnsupportedExpected(). All are -// reached through the ServiceRegistry, and MockServiceRegistry::getValidators() -// / getTimeKeeper() / getOPs() / getLoadManager() / getInboundLedgers() / -// getNodeFamily() / getJobQueue() / getOverlay() / getAmendmentTable() THROW -// std::logic_error. So "no -// gauge callback ran" is directly observable here: had registerAsyncGauges() run -// and had a callback fired, one of those accessors would have thrown. -// -// Honest scope note: these tests do NOT assert an observed gauge value. On this -// build the gauges are not compiled at all (see the file header), so there is no -// value to read -- inventing one would be fiction. The value assertions live in -// MetricMacros.cpp. What is asserted here is the other half of the contract: -// registration is absent and no service is consulted. -// ----------------------------------------------------------------- - -// The observable-gauge registration surface is compiled OUT when telemetry is -// disabled: `meter()` -- the only accessor the gauges and the XRPL_METRIC_* -// macros use to reach the OTel SDK -- does not exist as a member at all. This is -// a compile-time assertion, so it fails the build (not the run) if the accessor -// ever escapes its #ifdef and drags the SDK into a telemetry-off build. -TEST_F(MetricsRegistryTest, disabled_build_exposes_no_meter_accessor) +TEST_F(MetricsRegistryTest, every_record_method_runs_while_recording) { - // Detects `registry.meter()` being callable. Under #ifndef - // XRPL_ENABLE_TELEMETRY it must not be, so the trait is false. - auto hasMeter = [](T* r) { return requires { r->meter(); }; }; - EXPECT_FALSE(hasMeter(static_cast(nullptr))); + MetricsRegistry registry(true, j_, testOptions()); + ASSERT_EQ(registry.recording(), true); - // The enable flag is still queryable and reports exactly false -- the class - // is a no-op, not an absent type. - telemetry::MetricsRegistry const registry(false, mockApp_, j_, kTestOptions); - EXPECT_FALSE(registry.isEnabled()); + // The one test that drives all twelve real entry points against a real + // SDK provider. No point can be read back -- the core owns its provider and + // exposes no reader -- so the sweep is a crash canary and the assertions + // below are the deterministic part. + EXPECT_NO_THROW(recordEverything(registry, "live")); + + // Mutation: a record method that stores Phase::Stopped or resets provider_ + // as a side effect. Either would silence the node after its first metric. + EXPECT_EQ(registry.isEnabled(), true); + EXPECT_EQ(registry.recording(), true); +#ifdef XRPL_ENABLE_TELEMETRY + EXPECT_EQ(registry.hasPipeline(), true); +#endif } -// A full disabled lifecycle registers no gauge and therefore consults NO -// ServiceRegistry service. Asserting the cause, not just the absence of a crash: -// every MockServiceRegistry accessor a sync-diagnostics gauge would need throws, -// so reaching the end without an exception proves no callback ran. -TEST_F(MetricsRegistryTest, disabled_lifecycle_never_consults_gauge_services) +TEST_F(MetricsRegistryTest, stop_closes_the_gate_and_leaves_enabled_true) { - telemetry::MetricsRegistry registry(false, mockApp_, j_, kTestOptions); + MetricsRegistry registry(true, j_, testOptions()); - // startAsyncGauges() is where registerAsyncGauges() -- and with it - // registerUnlQuorumGauge() / registerClockSkewGauge() / - // registerSyncStateGauge() / registerStallEventsCounter() / - // registerSyncAcquireGauge() / registerCacheHitRateDetailGauge() / - // registerJobQueueBacklogGauge() / registerJobQueueSaturationGauge() / - // registerPeerLedgerSupplyGauge() / registerSlotCensusGauge() / - // registerAmendmentBlockGauge() / registerNodeStoreGauge() -- - // would run. - EXPECT_NO_THROW(registry.startAsyncGauges()); + // Setup: the gate really is open, so a false reading below is attributable + // to stop() and not to construction. + ASSERT_EQ(registry.recording(), true); - // detachCallbacks() is the shutdown hook the real gauges honour. It must be - // safe and idempotent even though there is nothing to detach. - EXPECT_NO_THROW(registry.detachCallbacks()); - EXPECT_NO_THROW(registry.detachCallbacks()); + registry.stop(); - // Still disabled after startAsyncGauges(): it must not flip the flag. - EXPECT_FALSE(registry.isEnabled()); + // isEnabled() reports what config asked for; recording() reports whether a + // record call is safe. The value of this line is the PAIR it forms with the + // recording() assertion below -- true beside false -- which is what shows + // the gate is a phase and not the enable flag. + // + // Named honestly, because no single-line change makes this line fail in + // BOTH builds. enabled_ is `bool const`, so clearing it in stop() does not + // compile -- the type already forbids the defect. Rewriting isEnabled() as + // recording() is red only where stop() can move the phase, which is the + // telemetry-on build. + EXPECT_EQ(registry.isEnabled(), true); +#ifdef XRPL_ENABLE_TELEMETRY + // Mutation: delete the phase_.store(Phase::Stopped, release) line from + // stop(). Every XRPL_METRIC_* call site reads recording() before it + // touches an instrument, so that one store is the whole gate. + EXPECT_EQ(registry.recording(), false); + + // A separate observation from the gate, because a separate line does it: + // one stores the phase, another drops the provider. Mutation: delete + // provider_.reset() from stop(). + EXPECT_EQ(registry.hasPipeline(), false); +#else + // Telemetry-off: stop()'s body is entirely inside the guard, so it cannot + // move phase_, and recording() is the enable flag here. Pinned so that an + // #else branch which started gating shows up as a change. + EXPECT_EQ(registry.recording(), true); +#endif +} + +TEST_F(MetricsRegistryTest, records_after_stop_are_inert) +{ + MetricsRegistry registry(true, j_, testOptions()); + + // Label sets that already have SDK storage by the time stop() runs. + recordEverything(registry, "before_stop"); + + registry.stop(); + +#ifdef XRPL_ENABLE_TELEMETRY + ASSERT_EQ(registry.recording(), false); +#endif + + // The same twelve methods with a tag never used before stop(), so every + // attribute set here is first-seen -- including four histogram records + // across three instruments, which is the case that allocates through the + // AggregationConfig the destroyed View owned. + // + // Mutation: delete the `!recording()` test from any record method. That is + // the regression this file exists for, and it is reliably red only under a + // sanitizer: with none, a read of freed memory can still return and pass. + EXPECT_NO_THROW(recordEverything(registry, "after_stop")); + + // Deterministic part: no record path reopens the gate or rebuilds the + // pipeline. + EXPECT_EQ(registry.isEnabled(), true); +#ifdef XRPL_ENABLE_TELEMETRY + EXPECT_EQ(registry.recording(), false); + EXPECT_EQ(registry.hasPipeline(), false); +#endif +} + +TEST_F(MetricsRegistryTest, stop_twice_is_safe) +{ + MetricsRegistry registry(true, j_, testOptions()); + + registry.stop(); + + // run() and the Application destructor both call stop(), so a second call + // is the ordinary shutdown path. Mutation: delete `if (!provider_) return;` + // from stop(). provider_ is null by now, so this call would dereference an + // empty shared_ptr on every clean shutdown. The registry's own destructor + // then makes a third call. EXPECT_NO_THROW(registry.stop()); - // Positive control: the mock DOES throw when a gauge-backing service is - // actually requested. Without this, "nothing threw" would be vacuous -- it - // could mean the mock is permissive rather than that no callback ran. - EXPECT_THROW(mockApp_.getValidators(), std::logic_error); - EXPECT_THROW(mockApp_.getTimeKeeper(), std::logic_error); - // The two services the sync-state signals read. sync_state needs both - // (NetworkOPs for the gate/duration/ledgers-behind, LoadManager for stall - // seconds) and server_stall_events_total needs the second, so either one - // firing would have thrown above. - EXPECT_THROW(mockApp_.getOPs(), std::logic_error); - EXPECT_THROW(mockApp_.getLoadManager(), std::logic_error); - // The two services the acquire signals read: sync_acquire polls the - // in-flight acquire collection, shamap_cache_hit_rate polls the node - // Family's tree-node cache. Neither was consulted above. - EXPECT_THROW(mockApp_.getInboundLedgers(), std::logic_error); - EXPECT_THROW(mockApp_.getNodeFamily(), std::logic_error); - // The service the job-queue gauge reads: jobq_saturation polls - // getWorkerSaturation() on the JobQueue. Not consulted above, so the - // gauge never took the JobQueue mutex on a telemetry-off build. - EXPECT_THROW(mockApp_.getJobQueue(), std::logic_error); - // The service both peer gauges read: peer_ledger_supply polls - // getPeerLedgerSupply(), which walks the active-peer list, and - // peerfinder_slot_census polls getSlotCensus(), which takes the PeerFinder - // lock. Both go through the Overlay, so a single throw here proves neither - // gauge walked the peer list nor took the PeerFinder lock on a - // telemetry-off build. - EXPECT_THROW(mockApp_.getOverlay(), std::logic_error); - // The service the amendment countdown reads: amendment_block polls - // firstUnsupportedExpected() on the AmendmentTable, which takes that - // table's mutex. Not consulted above, so the countdown never ran. (Its - // `warned` half reads NetworkOPs, already covered by the getOPs() check.) - EXPECT_THROW(mockApp_.getAmendmentTable(), std::logic_error); - // The service the nodestore gauge reads: nodestore_state polls - // getStoreDurationUs()/getStoreCount() and - // getFetchDurationUs()/getFetchTotalCount() on the node-store Database, - // alongside its I/O totals and write-queue detail. Not consulted above, - // so the gauge never read those atomics on a telemetry-off build. - EXPECT_THROW(mockApp_.getNodeStore(), std::logic_error); + EXPECT_EQ(registry.isEnabled(), true); +#ifdef XRPL_ENABLE_TELEMETRY + EXPECT_EQ(registry.recording(), false); + EXPECT_EQ(registry.hasPipeline(), false); +#endif } -// Even asking for enabled=true registers no sync-diagnostics gauge on a -// telemetry-off build. isEnabled() faithfully echoes the constructor argument -// (the flag lives outside the #ifdef), so the flag alone does NOT prove the -// gauges are inert -- the mock does: a full startAsyncGauges()/stop() cycle with -// enabled=true still consults no service, so no callback was ever registered. -// Asserts the exact flag value on BOTH construction paths. -TEST_F(MetricsRegistryTest, enabled_flag_alone_registers_no_gauges_when_compiled_out) -{ - telemetry::MetricsRegistry enabledRequest(true, mockApp_, j_, kTestOptions); - - // The flag is echoed back exactly, true not false: it is a plain member, - // not gated on XRPL_ENABLE_TELEMETRY. - EXPECT_TRUE(enabledRequest.isEnabled()); - - // Yet the whole lifecycle stays inert. If registerAsyncGauges() had run and - // registered registerUnlQuorumGauge()/registerClockSkewGauge()/ - // registerSyncStateGauge()/registerStallEventsCounter()/ - // registerSyncAcquireGauge()/registerCacheHitRateDetailGauge()/ - // registerJobQueueBacklogGauge()/registerJobQueueSaturationGauge()/ - // registerPeerLedgerSupplyGauge()/registerSlotCensusGauge()/ - // registerAmendmentBlockGauge()/registerNodeStoreGauge(), a - // callback would reach getValidators()/getTimeKeeper()/getOPs()/ - // getLoadManager()/getInboundLedgers()/getNodeFamily()/getJobQueue()/ - // getOverlay()/getAmendmentTable()/getNodeStore() and - // throw std::logic_error. - EXPECT_NO_THROW(enabledRequest.startAsyncGauges()); - EXPECT_NO_THROW(enabledRequest.detachCallbacks()); - EXPECT_NO_THROW(enabledRequest.stop()); - - // Contrast: enabled=false reports exactly false. - telemetry::MetricsRegistry const disabledRequest(false, mockApp_, j_, kTestOptions); - EXPECT_FALSE(disabledRequest.isEnabled()); -} +// --------------------------------------------------------------------------- +// Class-surface assertions. These read what the class declares, not what a +// registry does, so the compiler decides them and the test reports the answer. +// --------------------------------------------------------------------------- // The `state_changes_total` counter has no registry-owned wrapper method by // design: it is emitted from a labelled call-site macro in -// NetworkOPsImp::setMode, which is the only place that knows {from,to}. This -// compile-time assertion is the guard -- if someone adds -// incrementStateChanges(), the unlabelled instrument would coexist with the -// labelled one and Prometheus would carry two conflicting versions of the same +// NetworkOPsImp::setMode, which is the only place that knows {from,to}. Adding +// incrementStateChanges() would put an unlabelled instrument beside the +// labelled one, so Prometheus would carry two conflicting versions of one // metric name. TEST_F(MetricsRegistryTest, state_changes_counter_has_no_registry_wrapper) { auto hasIncrementStateChanges = [](T* r) { return requires { r->incrementStateChanges(); }; }; - EXPECT_FALSE(hasIncrementStateChanges(static_cast(nullptr))); + EXPECT_FALSE(hasIncrementStateChanges(static_cast(nullptr))); - // Positive control: a sibling parity counter that WAS deliberately kept as - // a registry wrapper is still detectable, so the trait above is really - // probing for the method and not vacuously false. + // Positive control: a sibling parity counter that is a registry wrapper is + // still detectable, so the trait above is really probing for the method and + // not vacuously false. auto hasIncrementLedgersClosed = [](T* r) { return requires { r->incrementLedgersClosed(); }; }; - EXPECT_TRUE(hasIncrementLedgersClosed(static_cast(nullptr))); + EXPECT_TRUE(hasIncrementLedgersClosed(static_cast(nullptr))); +} + +#ifndef XRPL_ENABLE_TELEMETRY + +// meter() is the only accessor that reaches the OTel SDK, and in a +// telemetry-off build it is not a member at all. The check is red if the +// accessor escapes its #ifdef and drags the SDK into this build. +TEST_F(MetricsRegistryTest, telemetry_off_build_exposes_no_meter_accessor) +{ + auto hasMeter = [](T* r) { return requires { r->meter(); }; }; + EXPECT_FALSE(hasMeter(static_cast(nullptr))); + + // Positive control: recording() -- an accessor that exists in both builds + // -- is detectable on the same trait shape. + auto hasRecording = [](T* r) { return requires { r->recording(); }; }; + EXPECT_TRUE(hasRecording(static_cast(nullptr))); } #endif // !XRPL_ENABLE_TELEMETRY + +#ifdef XRPL_ENABLE_TELEMETRY + +// --------------------------------------------------------------------------- +// getValidationTracker() is declared in a telemetry-on build only, because only +// the observable-gauge callbacks drain the tracker. Two production call sites +// reach it this way, so the accessor has to hand back the live member. +// --------------------------------------------------------------------------- + +TEST_F(MetricsRegistryTest, validation_tracker_is_owned_per_registry) +{ + MetricsRegistry first(true, j_, testOptions()); + MetricsRegistry second(true, j_, testOptions()); + + // Setup: both trackers start empty, so a count below is attributable to + // the record call and not to fixture state. + ASSERT_EQ(first.getValidationTracker().totalValidationsSent(), 0u); + ASSERT_EQ(second.getValidationTracker().totalValidationsSent(), 0u); + + first.getValidationTracker().recordOurValidation( + xrpl::uint256{std::uint64_t{7}}, xrpl::LedgerIndex{7}); + + // Exact counts on both sides: the reference is live, so the write lands, + // and it lands on one registry only. + // + // Mutation: return a reference to a function-local static from + // getValidationTracker(). One shared tracker would put this count on + // `second` as well, so two registries in one process would report one + // merged agreement figure. + EXPECT_EQ(first.getValidationTracker().totalValidationsSent(), 1u); + EXPECT_EQ(second.getValidationTracker().totalValidationsSent(), 0u); +} + +#endif // XRPL_ENABLE_TELEMETRY diff --git a/src/tests/libxrpl/telemetry/ValidationTracker.cpp b/src/tests/libxrpl/telemetry/ValidationTracker.cpp index 6acbedbda0..5ba3d24c98 100644 --- a/src/tests/libxrpl/telemetry/ValidationTracker.cpp +++ b/src/tests/libxrpl/telemetry/ValidationTracker.cpp @@ -7,7 +7,7 @@ * period and a bucket boundary reachable without waiting for one. */ -#include +#include #include #include diff --git a/src/xrpld/app/consensus/RCLConsensus.cpp b/src/xrpld/app/consensus/RCLConsensus.cpp index f29fc290e1..bc437b5b97 100644 --- a/src/xrpld/app/consensus/RCLConsensus.cpp +++ b/src/xrpld/app/consensus/RCLConsensus.cpp @@ -18,13 +18,6 @@ #include #include #include -#include -#ifdef XRPL_ENABLE_TELEMETRY -// The metric-name constants are named only as macro arguments, which the -// macros drop when telemetry is compiled out. -#include -#endif -#include #include #include @@ -71,6 +64,13 @@ #include #include #include +#include +#ifdef XRPL_ENABLE_TELEMETRY +// The metric-name constants are named only as macro arguments, which the +// macros drop when telemetry is compiled out. +#include +#endif +#include #include #include diff --git a/src/xrpld/app/ledger/LedgerHistory.cpp b/src/xrpld/app/ledger/LedgerHistory.cpp index d94f114d33..71dcbfbec1 100644 --- a/src/xrpld/app/ledger/LedgerHistory.cpp +++ b/src/xrpld/app/ledger/LedgerHistory.cpp @@ -4,7 +4,6 @@ #include #include #include -#include #include #include @@ -22,6 +21,7 @@ #include #include #include +#include #include #include diff --git a/src/xrpld/app/ledger/detail/InboundLedger.cpp b/src/xrpld/app/ledger/detail/InboundLedger.cpp index 997925b9b9..d00d446ef8 100644 --- a/src/xrpld/app/ledger/detail/InboundLedger.cpp +++ b/src/xrpld/app/ledger/detail/InboundLedger.cpp @@ -12,12 +12,6 @@ #include #include #include -#include -#ifdef XRPL_ENABLE_TELEMETRY -// The metric-name constants are named only as macro arguments, which the -// macros drop when telemetry is compiled out. -#include -#endif #include #include @@ -40,6 +34,12 @@ #include #include #include +#include +#ifdef XRPL_ENABLE_TELEMETRY +// The metric-name constants are named only as macro arguments, which the +// macros drop when telemetry is compiled out. +#include +#endif #include #include diff --git a/src/xrpld/app/ledger/detail/LedgerDeltaAcquire.cpp b/src/xrpld/app/ledger/detail/LedgerDeltaAcquire.cpp index 3094d20a1f..ef5af96cc1 100644 --- a/src/xrpld/app/ledger/detail/LedgerDeltaAcquire.cpp +++ b/src/xrpld/app/ledger/detail/LedgerDeltaAcquire.cpp @@ -10,12 +10,6 @@ #include #include #include -#include -#ifdef XRPL_ENABLE_TELEMETRY -// The metric-name constants are named only as macro arguments, which the -// macros drop when telemetry is compiled out. -#include -#endif #include #include @@ -26,6 +20,12 @@ #include #include #include +#include +#ifdef XRPL_ENABLE_TELEMETRY +// The metric-name constants are named only as macro arguments, which the +// macros drop when telemetry is compiled out. +#include +#endif #include diff --git a/src/xrpld/app/ledger/detail/LedgerMaster.cpp b/src/xrpld/app/ledger/detail/LedgerMaster.cpp index c7b0aad789..daa2733482 100644 --- a/src/xrpld/app/ledger/detail/LedgerMaster.cpp +++ b/src/xrpld/app/ledger/detail/LedgerMaster.cpp @@ -18,13 +18,6 @@ #include #include #include -#include -#ifdef XRPL_ENABLE_TELEMETRY -// The metric-name constants are named only as macro arguments, which the -// macros drop when telemetry is compiled out. -#include -#endif -#include #include #include @@ -64,6 +57,12 @@ #include #include #include +#include +#ifdef XRPL_ENABLE_TELEMETRY +// The metric-name constants are named only as macro arguments, which the +// macros drop when telemetry is compiled out. +#include +#endif #include #include diff --git a/src/xrpld/app/ledger/detail/LedgerReplayTask.cpp b/src/xrpld/app/ledger/detail/LedgerReplayTask.cpp index e0f0fd1bb0..3521526078 100644 --- a/src/xrpld/app/ledger/detail/LedgerReplayTask.cpp +++ b/src/xrpld/app/ledger/detail/LedgerReplayTask.cpp @@ -8,13 +8,13 @@ #include #include #include -#include -#include #include #include #include #include +#include +#include #include #include diff --git a/src/xrpld/app/ledger/detail/SkipListAcquire.cpp b/src/xrpld/app/ledger/detail/SkipListAcquire.cpp index 5238f300bc..097ba0f40c 100644 --- a/src/xrpld/app/ledger/detail/SkipListAcquire.cpp +++ b/src/xrpld/app/ledger/detail/SkipListAcquire.cpp @@ -8,12 +8,6 @@ #include #include #include -#include -#ifdef XRPL_ENABLE_TELEMETRY -// The metric-name constants are named only as macro arguments, which the -// macros drop when telemetry is compiled out. -#include -#endif #include #include @@ -22,6 +16,12 @@ #include #include #include +#include +#ifdef XRPL_ENABLE_TELEMETRY +// The metric-name constants are named only as macro arguments, which the +// macros drop when telemetry is compiled out. +#include +#endif #include diff --git a/src/xrpld/app/main/Application.cpp b/src/xrpld/app/main/Application.cpp index 34f747dc45..a46146fb5b 100644 --- a/src/xrpld/app/main/Application.cpp +++ b/src/xrpld/app/main/Application.cpp @@ -39,13 +39,7 @@ #include #include #include -#include -#ifdef XRPL_ENABLE_TELEMETRY -// The metric-name constants are named only as macro arguments, which the -// macros drop when telemetry is compiled out. -#include -#endif -#include +#include #include #include @@ -110,6 +104,13 @@ #include #include #include +#include +#ifdef XRPL_ENABLE_TELEMETRY +// The metric-name constants are named only as macro arguments, which the +// macros drop when telemetry is compiled out. +#include +#endif +#include #include #include @@ -142,6 +143,7 @@ #include #include #include +#include #include #include #include @@ -297,13 +299,21 @@ public: std::pair nodeIdentity_; std::unique_ptr telemetry_; /** - * OTel metrics registry for gap-fill metrics (counters, histograms, - * observable gauges). Its constructor builds the pipeline and every - * synchronous instrument, so it must stay declared after telemetry_ and - * before every subsystem that records a metric. Declaration order is the - * whole guarantee. Gauges are armed later by startTelemetryGauges(). + * OTel metrics pipeline for gap-fill metrics. Its constructor builds the + * provider and every synchronous instrument (counters and histograms), so + * it must stay declared after telemetry_ and before every subsystem that + * records a metric. Declaration order is the whole guarantee. */ std::unique_ptr metricsRegistry_; + /** + * The observable gauges, whose callbacks read live application services. + * Armed later by startTelemetryGauges(). + * + * Declared after metricsRegistry_ because members are destroyed in reverse + * declaration order: the gauges hold a MetricsRegistry& and must be + * destroyed before it. + */ + std::unique_ptr metricGauges_; Application::MutexType masterMutex_; // Required by the SHAMapStore @@ -444,11 +454,18 @@ public: , metricsRegistry_( std::make_unique( telemetry_->isEnabled(), - *this, logs_->journal("MetricsRegistry"), makeMetricsRegistryOptions( *config_, toBase58(TokenType::NodePublic, nodeIdentity_.first)))) + // Registers nothing until startAsyncGauges(), so building it here + // costs nothing and keeps the member non-null for the whole lifetime. + // That is what lets the shutdown path call it unconditionally. + , metricGauges_( + std::make_unique( + *metricsRegistry_, + *this, + logs_->journal("MetricsRegistry"))) , txMaster_(*this) , collectorManager_(makeCollectorManager( @@ -640,24 +657,35 @@ public: */ ~ApplicationImp() override { + // Each step is isolated, so a throw from one still leaves the others to + // run. Skipping stopMetricsRegistry() would be the costly one: nothing + // would join the OTel reader thread until ~MetricsRegistry(), and + // metricGauges_ is destroyed before that, so the callbacks' instrument + // handles would go away while the thread was still sampling them. + // // A shutdown diagnostic must never terminate the process, and a - // destructor is implicitly noexcept. - try - { - collectorManager_->collector()->onCollectionStopping(); - stopMetricsRegistry(); - telemetry_->stop(); - } - catch (std::exception const& e) - { - JLOG(journal_.error()) << "Error stopping telemetry: " << e.what(); - } - catch (...) - { - // The callees reach third-party SDK code, which may throw something - // outside std::exception. Escaping here would terminate the process. - JLOG(journal_.error()) << "Error stopping telemetry: unknown exception"; - } + // destructor is implicitly noexcept, so nothing may escape. + auto const stopStep = [this](std::string_view name, auto&& step) noexcept { + try + { + step(); + } + catch (std::exception const& e) + { + JLOG(journal_.error()) << "Error stopping " << name << ": " << e.what(); + } + catch (...) + { + // The callees reach third-party SDK code, which may throw + // something outside std::exception. + JLOG(journal_.error()) << "Error stopping " << name << ": unknown exception"; + } + }; + + // Both observers stop before telemetry, which they export through. + stopStep("collector", [this] { collectorManager_->collector()->onCollectionStopping(); }); + stopStep("metrics registry", [this] { stopMetricsRegistry(); }); + stopStep("telemetry", [this] { telemetry_->stop(); }); } //-------------------------------------------------------------------------- @@ -1417,13 +1445,13 @@ private: * nodeStore_, nodeFamily_, validators_, acceptedLedgerCache_, * cachedSLEs_, acquireStats_, timeKeeper_, relationalDatabase_, * inboundLedgers_, feeTrack_) are built earlier in setup(). See - * MetricsRegistry::startAsyncGauges() for the full list. + * AppMetricGauges::startAsyncGauges() for the full list. */ void startTelemetryGauges() const; /** - * Stop the metrics registry: detach its gauge callbacks and join its + * Detach the gauge callbacks, then stop the metrics pipeline and join its * reader thread. Idempotent. Called from run() before any observed * service stops, and again from ~ApplicationImp for the paths that never * reach run(). @@ -1880,16 +1908,17 @@ ApplicationImp::startTelemetry() const void ApplicationImp::startTelemetryGauges() const { - metricsRegistry_->startAsyncGauges(); + metricGauges_->startAsyncGauges(); } void ApplicationImp::stopMetricsRegistry() const { - // stop() detaches the callbacks and then shuts the provider down, which - // joins the reader thread, so once it returns no callback is running or - // can start. The cost is that metrics recorded after this point are not - // exported. + // Detach first, then stop. Detaching guarantees no gauge callback reads an + // application service or the meter after this point; stop() then closes the + // recording gate and destroys the provider. Reversed, a callback already + // running on the OTel reader thread could touch a destroyed provider. + metricGauges_->detachCallbacks(); metricsRegistry_->stop(); } @@ -1967,7 +1996,7 @@ ApplicationImp::run() // Both observers stop before any service below is stopped. The collector's // gauge callbacks run hook handlers that read ledgerMaster_, networkOPs_, - // the peer finder, the job queue and overlay_; the registry's callbacks + // the peer finder, the job queue and overlay_; the metric gauges' callbacks // run on the OTel reader thread and read nodeStore_, overlay_, networkOPs_, // loadManager_, ledgerMaster, inboundLedgers and more. Each call returns // once no callback is running or can start. diff --git a/src/xrpld/app/misc/NetworkOPs.cpp b/src/xrpld/app/misc/NetworkOPs.cpp index a9c2b3abcd..4debc51099 100644 --- a/src/xrpld/app/misc/NetworkOPs.cpp +++ b/src/xrpld/app/misc/NetworkOPs.cpp @@ -30,13 +30,6 @@ #include #include #include -#include -#ifdef XRPL_ENABLE_TELEMETRY -// The metric-name constants are named only as macro arguments, which the -// macros drop when telemetry is compiled out. -#include -#endif -#include #include #include @@ -125,6 +118,12 @@ #include #include #include +#include +#ifdef XRPL_ENABLE_TELEMETRY +// The metric-name constants are named only as macro arguments, which the +// macros drop when telemetry is compiled out. +#include +#endif #include #include diff --git a/src/xrpld/app/misc/SHAMapStoreImp.cpp b/src/xrpld/app/misc/SHAMapStoreImp.cpp index 3dc1ef6383..6f454a1f04 100644 --- a/src/xrpld/app/misc/SHAMapStoreImp.cpp +++ b/src/xrpld/app/misc/SHAMapStoreImp.cpp @@ -6,12 +6,6 @@ #include #include #include -#include -#ifdef XRPL_ENABLE_TELEMETRY -// The metric-name constants are named only as macro arguments, which the -// macros drop when telemetry is compiled out. -#include -#endif #include #include @@ -35,6 +29,12 @@ #include #include #include +#include +#ifdef XRPL_ENABLE_TELEMETRY +// The metric-name constants are named only as macro arguments, which the +// macros drop when telemetry is compiled out. +#include +#endif #include #include diff --git a/src/xrpld/app/misc/SHAMapStoreImp.h b/src/xrpld/app/misc/SHAMapStoreImp.h index 726425147a..5d03b26ce6 100644 --- a/src/xrpld/app/misc/SHAMapStoreImp.h +++ b/src/xrpld/app/misc/SHAMapStoreImp.h @@ -5,8 +5,6 @@ #include #include #include -#include -#include #include #include @@ -21,6 +19,8 @@ #include #include #include +#include +#include #include #include diff --git a/src/xrpld/app/misc/detail/TxQ.cpp b/src/xrpld/app/misc/detail/TxQ.cpp index ae390aabce..33d9e22ea0 100644 --- a/src/xrpld/app/misc/detail/TxQ.cpp +++ b/src/xrpld/app/misc/detail/TxQ.cpp @@ -3,7 +3,6 @@ #include #include #include -#include #include #include @@ -33,6 +32,7 @@ #include #include #include +#include #include #include #include diff --git a/src/xrpld/app/misc/detail/ValidatorSite.cpp b/src/xrpld/app/misc/detail/ValidatorSite.cpp index bd6e8f2369..78bc341d9b 100644 --- a/src/xrpld/app/misc/detail/ValidatorSite.cpp +++ b/src/xrpld/app/misc/detail/ValidatorSite.cpp @@ -6,14 +6,6 @@ #include #include #include -#ifdef XRPL_ENABLE_TELEMETRY -// The metric macro is named only inside reportFetchOutcome(), whose body is -// compiled out with the counter it records. -#include -#endif -// Not gated: the outcome-label constants are named by the fetch handlers, which -// pass them whether or not the counter exists. -#include #include #include @@ -24,6 +16,14 @@ #include #include #include +#ifdef XRPL_ENABLE_TELEMETRY +// The metric macro is named only inside reportFetchOutcome(), whose body is +// compiled out with the counter it records. +#include +#endif +// Not gated: the outcome-label constants are named by the fetch handlers, which +// pass them whether or not the counter exists. +#include #include #include diff --git a/src/xrpld/overlay/detail/ConnectAttempt.cpp b/src/xrpld/overlay/detail/ConnectAttempt.cpp index 538b9dafd9..5fb557865a 100644 --- a/src/xrpld/overlay/detail/ConnectAttempt.cpp +++ b/src/xrpld/overlay/detail/ConnectAttempt.cpp @@ -8,12 +8,6 @@ #include #include #include -#ifdef XRPL_ENABLE_TELEMETRY -// The macros and the metric-name constants are named only inside -// reportOutcome(), whose body is compiled out with the metrics it records. -#include -#include -#endif #include #include @@ -33,6 +27,10 @@ #include #include #ifdef XRPL_ENABLE_TELEMETRY +// The macros and the metric-name constants are named only inside +// reportOutcome(), whose body is compiled out with the metrics it records. +#include +#include // Named only where the dial span is opened and ended, both compiled out below. // The span member itself is declared unconditionally, so ConnectAttempt.h keeps // its own SpanGuard.h include either way. diff --git a/src/xrpld/overlay/detail/Handshake.cpp b/src/xrpld/overlay/detail/Handshake.cpp index 52f26e1b09..f6d37f1c7f 100644 --- a/src/xrpld/overlay/detail/Handshake.cpp +++ b/src/xrpld/overlay/detail/Handshake.cpp @@ -3,8 +3,6 @@ #include #include #include -#include -#include #include #include @@ -23,6 +21,8 @@ #include #include #include +#include +#include #include #include diff --git a/src/xrpld/overlay/detail/OverlayImpl.cpp b/src/xrpld/overlay/detail/OverlayImpl.cpp index c6f6496e80..32b20127f4 100644 --- a/src/xrpld/overlay/detail/OverlayImpl.cpp +++ b/src/xrpld/overlay/detail/OverlayImpl.cpp @@ -14,8 +14,6 @@ #include #include #include -#include -#include #include #include @@ -56,6 +54,8 @@ #include #include #include +#include +#include #include #include diff --git a/src/xrpld/overlay/detail/PeerImp.cpp b/src/xrpld/overlay/detail/PeerImp.cpp index 0f2de57289..20484b2dc1 100644 --- a/src/xrpld/overlay/detail/PeerImp.cpp +++ b/src/xrpld/overlay/detail/PeerImp.cpp @@ -21,8 +21,6 @@ #include #include #include -#include -#include #include #include @@ -74,6 +72,8 @@ #include #include #include +#include +#include #include #include #include diff --git a/src/xrpld/overlay/detail/PeerImp.h b/src/xrpld/overlay/detail/PeerImp.h index 5073491f87..11b3407f3f 100644 --- a/src/xrpld/overlay/detail/PeerImp.h +++ b/src/xrpld/overlay/detail/PeerImp.h @@ -9,7 +9,6 @@ #include #include #include -#include #include #include @@ -35,6 +34,7 @@ #include #include #include +#include #include #include diff --git a/src/xrpld/perflog/detail/PerfLogImp.cpp b/src/xrpld/perflog/detail/PerfLogImp.cpp index 4173ea1a7c..42e0bf2061 100644 --- a/src/xrpld/perflog/detail/PerfLogImp.cpp +++ b/src/xrpld/perflog/detail/PerfLogImp.cpp @@ -1,12 +1,11 @@ #include #include -#include #ifdef XRPL_ENABLE_TELEMETRY // Only the recording calls below and the metric macros' expansion name the // registry, and neither survives with telemetry compiled out. -#include +#include #endif #include @@ -24,6 +23,7 @@ #include #include #include +#include #include #include diff --git a/src/xrpld/rpc/detail/PathRequest.cpp b/src/xrpld/rpc/detail/PathRequest.cpp index b61bf1edee..c891111978 100644 --- a/src/xrpld/rpc/detail/PathRequest.cpp +++ b/src/xrpld/rpc/detail/PathRequest.cpp @@ -12,7 +12,6 @@ #include #include #include -#include #include #include @@ -40,6 +39,7 @@ #include #include #include +#include #include #include #include diff --git a/src/xrpld/rpc/detail/ServerHandler.cpp b/src/xrpld/rpc/detail/ServerHandler.cpp index 9730e4519f..eefef37909 100644 --- a/src/xrpld/rpc/detail/ServerHandler.cpp +++ b/src/xrpld/rpc/detail/ServerHandler.cpp @@ -14,7 +14,6 @@ #include #include #include // IWYU pragma: keep -#include #include #include @@ -54,6 +53,7 @@ #include #include #include +#include #include #include diff --git a/src/xrpld/telemetry/MetricsRegistry.cpp b/src/xrpld/telemetry/AppMetricGauges.cpp similarity index 66% rename from src/xrpld/telemetry/MetricsRegistry.cpp rename to src/xrpld/telemetry/AppMetricGauges.cpp index 2d26d96768..2d6704d449 100644 --- a/src/xrpld/telemetry/MetricsRegistry.cpp +++ b/src/xrpld/telemetry/AppMetricGauges.cpp @@ -1,13 +1,15 @@ /** - * MetricsRegistry implementation — OpenTelemetry metric instruments for xrpld. + * AppMetricGauges implementation — the pull-model half of the OTel metric + * surface. * * This file contains: - * - Construction / destruction logic for the OTel MeterProvider pipeline. - * - Synchronous instrument creation (counters, histograms) for RPC, job - * queue, and NodeStore I/O metrics. - * - Observable gauge callback registration for cache hit rates, TxQ state, - * CountedObject instances, load factors, and NodeStore queue depth. - * - No-op stubs when XRPL_ENABLE_TELEMETRY is not defined. + * - Registration of every observable instrument whose callback samples live + * server state: cache hit rates, TxQ state, CountedObject instances, load + * factors, NodeStore I/O, server info, complete ledger ranges, validator + * health, peer quality, reduce-relay efficiency, ledger economy, state + * tracking, storage detail and validation agreement. + * - The nodestore_state helpers those callbacks publish values through. + * - The arm and disarm entry points for the whole set. */ // On Windows, OTel's spin_lock_mutex.h (transitively included from @@ -20,11 +22,7 @@ #include #endif -#include - -// Unguarded because the constructor's `beast::Journal journal` parameter is -// declared in both configurations; only the member it initialises is guarded. -#include +#include #ifdef XRPL_ENABLE_TELEMETRY @@ -38,9 +36,9 @@ // txMetrics(). // // The cycle is confined to this translation unit. No telemetry header includes -// app or overlay (MetricsRegistry.h forward-declares what it needs and takes a -// ServiceRegistry&), and all of src/xrpld builds into a single CMake target, so -// there is no header cycle and no link cycle to break. +// app or overlay -- the callbacks reach every service through the +// ServiceRegistry reference they are given -- and all of src/xrpld builds into a +// single CMake target, so there is no header cycle and no link cycle to break. // // Inverting it properly means declaring a metrics-source interface below overlay // and implementing it there, which is deliberately left as follow-up rather than @@ -55,11 +53,11 @@ #include #include #include -#include #include #include #include +#include #include #include #include @@ -71,39 +69,17 @@ #include #include #include -#include -#include -#include -#include -// For networkTypeFromId(), the one xrpl.network.type mapping both export -// paths use. Adds no levelization edge: xrpld.telemetry > xrpl.telemetry -// already holds via SpanNames.h above. -#include +#include -#include -#include -#include #include #include #include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include #include #include #include #include #include -#include #include #include #include @@ -112,251 +88,64 @@ #include #include -namespace metric_sdk = opentelemetry::sdk::metrics; -namespace otlp_http = opentelemetry::exporter::otlp; -// Not `resource`: that would collide with xrpl::resource (the resource-accounting -// namespace), which encloses every use site below. Inner-scope lookup would find -// that namespace instead of this file-scope alias. -namespace otel_resource = opentelemetry::sdk::resource; - -namespace { - -// Microsecond-valued duration histogram instrument names. Each is -// referenced twice — once to register the explicit-bucket view and once -// to create the instrument — so they are named constants to keep the two -// sites in sync (a mismatch would silently drop the bucket override). -constexpr char kJobQueuedDurationUs[] = "job_queued_us"; -constexpr char kJobRunningDurationUs[] = "job_running_us"; -constexpr char kRpcMethodDurationUs[] = "rpc_method_us"; - -// Millisecond-valued duration histogram instrument names. Same -// register-then-create pairing as the microsecond names above, so the same -// reason applies for naming them: the view and the record site must agree. -// -// consensus_round_duration_ms is recorded from RCLConsensus at the call site -// (via XRPL_METRIC_HISTOGRAM_RECORD, which creates the instrument lazily -// there), not created here. Only the VIEW is registered here, because a view -// matches by instrument name and must exist before the instrument is first -// used — the MeterProvider is built with the view registry, and the round -// histogram is not created until the first consensus round completes, well -// after start(). -constexpr char kConsensusRoundDurationMs[] = "consensus_round_duration_ms"; - -/** - * Register an explicit-bucket histogram view. - * - * The SDK's default boundaries top out at 10,000, so any instrument whose - * values exceed that saturates and every quantile reads as the ceiling. The - * floor matters just as much and is easier to miss: a ladder whose first edge - * sits above the mass of the distribution makes every low quantile an - * interpolation inside bucket 0 -- a number derived from the bucket edge - * rather than from any sample. Both ends are chosen from measured - * distributions in HistogramBuckets.h. - * - * @param views The registry to add the view to. - * @param name Instrument name to match (e.g. "job_running_us"). - * @param boundaries Bucket upper bounds, ascending. - */ -void -addHistogramView( - metric_sdk::ViewRegistry& views, - std::string const& name, - std::vector boundaries) -{ - auto config = std::make_shared(); - config->boundaries_ = std::move(boundaries); - - auto selector = metric_sdk::InstrumentSelectorFactory::Create( - metric_sdk::InstrumentType::kHistogram, name, ""); - auto meterSelector = metric_sdk::MeterSelectorFactory::Create( - std::string(xrpl::telemetry::kMeterName), std::string(xrpl::telemetry::kMeterVersion), ""); - auto view = - metric_sdk::ViewFactory::Create(name, "", metric_sdk::AggregationType::kHistogram, config); - - views.AddView(std::move(selector), std::move(meterSelector), std::move(view)); -} - -/** - * Register the microsecond-ladder view for a duration instrument. - * - * Job wait/run times and RPC latencies routinely exceed the SDK default - * ceiling, so they all share `buckets::kMicrosecondBuckets`. - * - * @param views The registry to add the view to. - * @param name Instrument name to match (e.g. "job_running_us"). - */ -void -addMicrosecondHistogramView(metric_sdk::ViewRegistry& views, std::string const& name) -{ - addHistogramView( - views, - name, - xrpl::telemetry::buckets::toVector(xrpl::telemetry::buckets::kMicrosecondBuckets)); -} - -/** - * Register the explicit-bucket view for a consensus-round duration in - * MILLISECONDS. - * - * The round histogram needs its own boundaries for two reasons. The SDK - * default tops out at 10,000 ms, and a recovering or stalled node routinely - * rounds slower than that — the consensus parameters themselves allow up to - * `ledgerAbandonConsensus` = 120 s — so the default would collapse exactly the - * slow rounds this signal exists to show into one saturated top bucket. And a - * healthy round is about 3-4 s, which the default's coarse spacing near that - * value cannot resolve, so a round drifting from 3 s to 5 s would not move any - * quantile. - * - * Boundaries: 500ms, 1s, 2s, 3s, 4s, 5s, 7.5s, 10s, 15s, 20s, 30s, 60s, 120s. - * Dense across the healthy 2-5 s band, then widening to the 120 s abandon - * limit so a stalled round still lands in a real bucket. - * - * @param views The registry to add the view to. - * @param name Instrument name to match ("consensus_round_duration_ms"). - */ -void -addRoundDurationHistogramView(metric_sdk::ViewRegistry& views, std::string const& name) -{ - addHistogramView( - views, - name, - {500.0, - 1'000.0, - 2'000.0, - 3'000.0, - 4'000.0, - 5'000.0, - 7'500.0, - 10'000.0, - 15'000.0, - 20'000.0, - 30'000.0, - 60'000.0, - 120'000.0}); -} - -void -addRotationPhaseHistogramView(metric_sdk::ViewRegistry& views, std::string const& name) -{ - addHistogramView( - views, - name, - xrpl::telemetry::buckets::toVector(xrpl::telemetry::buckets::kRotationPhaseSecondsBuckets)); -} - -} // namespace - #endif // XRPL_ENABLE_TELEMETRY namespace xrpl::telemetry { -MetricsRegistry::MetricsRegistry( - [[maybe_unused]] bool enabled, +AppMetricGauges::AppMetricGauges( + [[maybe_unused]] MetricsRegistry& core, [[maybe_unused]] ServiceRegistry& app, - [[maybe_unused]] beast::Journal journal, - [[maybe_unused]] Options const& options) - : enabled_(enabled) + [[maybe_unused]] beast::Journal journal) #ifdef XRPL_ENABLE_TELEMETRY + : core_(core) , app_(app) + // The core logs through the same partition, so one log-level setting + // covers the whole metric pipeline. , journal_(journal) #endif { -#ifdef XRPL_ENABLE_TELEMETRY - if (!enabled_) - return; - - // useTls is logged because a collector that requires TLS rejects a - // plaintext exporter with no local error. The paths are left out. - JLOG(journal_.info()) << "MetricsRegistry: starting, endpoint=" << options.endpoint - << ", serviceName=" << options.serviceName - << ", serviceVersion=" << options.serviceVersion - << ", instanceId=" << options.serviceInstanceId - << ", nodeId=" << options.nodeId << ", networkId=" << options.networkId - << ", useTls=" << options.useTls; - - // A broken pipeline must not stop the node. The SDK is third-party code, - // so the catch-all is deliberate, as in ~ApplicationImp. - try - { - initExporterAndProvider(options); - - // Rule for anything added below: the constructor may create only - // instruments whose recording is PUSHED from app code -- counters and - // histograms. An instrument registered here is live immediately, and - // the reader thread may invoke a registered callback before the rest - // of the Application is built, so any observable whose callback reads - // an Application service belongs in startAsyncGauges(), not here. - // That includes observable COUNTERS, not just gauges: - // jq_trans_overflow_total was created here and its callback read - // getOverlay(), which asserts overlay_ is non-null. - initSyncInstruments(); - } - catch (std::exception const& e) - { - disablePipeline(e.what()); - return; - } - catch (...) - { - disablePipeline("unknown exception"); - return; - } - - JLOG(journal_.info()) << "MetricsRegistry: provider and instruments ready"; -#endif // XRPL_ENABLE_TELEMETRY } -#ifdef XRPL_ENABLE_TELEMETRY -void -MetricsRegistry::disablePipeline(std::string_view reason) +AppMetricGauges::~AppMetricGauges() { - provider_.reset(); - // A no-op meter keeps the invariant the XRPL_METRIC_* macros rely on: an - // enabled registry always has a meter, so every call site gets an inert - // instrument here with no check of its own. - meter_ = noopMeter(kMeterName); - JLOG(journal_.error()) << "MetricsRegistry: metrics pipeline failed to initialise, " - "continuing without native metrics: " - << reason; -} -#endif // XRPL_ENABLE_TELEMETRY - -MetricsRegistry::~MetricsRegistry() -{ - stop(); + // A last resort, not the teardown path: the flag this sets lives here, so + // it cannot protect anything once this object is gone. The safe order is + // detachCallbacks(), then the core's stop() to join the reader thread, + // then destruction. + detachCallbacks(); } void -MetricsRegistry::startAsyncGauges() +AppMetricGauges::startAsyncGauges() { #ifdef XRPL_ENABLE_TELEMETRY - if (!enabled_) + if (!core_.isEnabled()) return; // One arm per life. A second call would create a second set of // same-named instruments, and a call after stop() would register on a // provider that is gone. Checked before the pipeline, so a call after - // stop() is reported as what it is and not as a build failure. - auto const currentPhase = phase_.load(std::memory_order_relaxed); - if (currentPhase != Phase::Ready) + // stop() is reported as what it is and not as a build failure. The core + // is enabled by here, so a false recording() means exactly stopped. + bool const stopped = !core_.recording(); + if (armed_ || stopped) { JLOG(journal_.warn()) << "MetricsRegistry: startAsyncGauges() called " - << (currentPhase == Phase::Stopped ? "after stop()" : "twice") - << "; ignored"; + << (stopped ? "after stop()" : "twice") << "; ignored"; return; } // The pipeline failed to build: the meter is a no-op, so registering - // gauges on it would only log a success that is not one. phase_ stays - // at Ready, so a second call lands here again and logs the same message. + // gauges on it would only log a success that is not one. armed_ stays + // false, so a second call lands here again and logs the same message. // Idempotent. - if (!provider_) + if (!core_.hasPipeline()) { JLOG(journal_.warn()) << "MetricsRegistry: startAsyncGauges() without a pipeline; " "no gauges registered"; return; } - phase_.store(Phase::GaugesArmed, std::memory_order_relaxed); + armed_ = true; registerAsyncGauges(); @@ -364,223 +153,8 @@ MetricsRegistry::startAsyncGauges() #endif // XRPL_ENABLE_TELEMETRY } -#ifdef XRPL_ENABLE_TELEMETRY void -MetricsRegistry::initExporterAndProvider(Options const& options) -{ - // Configure OTLP/HTTP metric exporter. The TLS settings come from the one - // [telemetry] block that also drives the trace exporter in Telemetry.cpp, - // so both exporters reach the collector on the same terms. - otlp_http::OtlpHttpMetricExporterOptions exporterOpts; - exporterOpts.url = options.endpoint; - if (options.useTls) - { - exporterOpts.ssl_ca_cert_path = options.tlsCaCertPath; - exporterOpts.ssl_client_cert_path = options.tlsClientCertPath; - exporterOpts.ssl_client_key_path = options.tlsClientKeyPath; - } - - auto exporter = otlp_http::OtlpHttpMetricExporterFactory::Create(exporterOpts); - - // Configure periodic reader with 10-second export interval. - metric_sdk::PeriodicExportingMetricReaderOptions readerOpts; - readerOpts.export_interval_millis = std::chrono::milliseconds(10000); - readerOpts.export_timeout_millis = std::chrono::milliseconds(5000); - auto reader = - metric_sdk::PeriodicExportingMetricReaderFactory::Create(std::move(exporter), readerOpts); - - // Stamp the same resource Telemetry::makeMetricsResource() builds for the - // trace pipeline. Both must agree: a node whose service.name or - // xrpl.network.type differs between the two pipelines splits its own - // series, and a dashboard filtering on either label shows only half. - // - // Use std::string, never a string literal: ResourceAttributes stores an - // OTel AttributeValue variant whose char-const* overload binds to bool, - // so a literal would be recorded as the boolean true. - otel_resource::ResourceAttributes attrs; - attrs[opentelemetry::semconv::service::kServiceName] = options.serviceName; - // int64_t, matching the trace resource. The same key with two types would - // give the two pipelines incompatible attribute values. - attrs[std::string(attr::networkId)] = static_cast(options.networkId); - // Derived here rather than passed in, so the id and the type label cannot - // disagree. Same helper the trace path uses. - attrs[std::string(attr::networkType)] = networkTypeFromId(options.networkId); - - // The three below are left off when empty rather than stamped blank. An - // absent label reads as "not reported"; an empty one looks like a value. - if (!options.serviceVersion.empty()) - attrs[opentelemetry::semconv::service::kServiceVersion] = options.serviceVersion; - if (!options.serviceInstanceId.empty()) - attrs[opentelemetry::semconv::service::kServiceInstanceId] = options.serviceInstanceId; - // xrpl.node.id: the same per-node key the trace resource carries, so - // metrics and traces resolve to one node. - if (!options.nodeId.empty()) - attrs[std::string(attr::nodeId)] = options.nodeId; - auto resourceAttrs = otel_resource::Resource::Create(attrs); - - // Build a view registry with explicit buckets for the duration - // histograms. Without this they use the SDK default buckets (max 10,000), - // which saturates every quantile at 10 ms for the µs instruments and at - // 10 s for the round histogram. - auto views = std::make_unique(); - addMicrosecondHistogramView(*views, kJobQueuedDurationUs); - addMicrosecondHistogramView(*views, kJobRunningDurationUs); - addMicrosecondHistogramView(*views, kRpcMethodDurationUs); - // Millisecond-scale: recorded at the RCLConsensus call site, so only the - // view is declared here (see the constant's comment). - addRoundDurationHistogramView(*views, kConsensusRoundDurationMs); - - // Recorded at its SHAMapStoreImp RotationPhase destructor, only the view - // lives here. Seconds ladder from HistogramBuckets.h. - addRotationPhaseHistogramView(*views, metric::rotationPhaseDurationSeconds); - - // Recorded at its PeerImp.cpp call site, not created here, so the name - // comes from the shared constant both sites use. - addMicrosecondHistogramView(*views, kGetObjectLookupUs); - - // Sweep malloc_trim duration. Shares the microsecond ladder rather than - // getting a bespoke one, and the ladder is what makes it readable: a trim on - // a small heap lands in the tens-of-microseconds buckets, while a trim on a - // multi-gigabyte resident heap runs well past 10 ms -- which is exactly the - // large-existing-database case this signal exists to catch. With the SDK - // default ceiling of 10,000 every one of those would collapse into the - // overflow bucket and p95 would read exactly 10 ms however bad it got. The - // shared ladder's upper reaches (25 ms, 50 ms, 100 ms, 250 ms, 500 ms, 1 s - // and beyond) resolve those, and its lower reaches (100 us, 500 us) resolve - // the healthy fresh-node case, so a per-instrument ladder would add a second - // thing to maintain for no extra resolution. - addMicrosecondHistogramView(*views, metric::sweepMallocTrimUs); - - // Millisecond dial/resolve latencies. Both exceed the SDK default ceiling - // of 10,000: the dial timer is 15 s, so without an explicit ladder every - // timed-out dial lands in the overflow bucket and p95 reads exactly 10 s - // however bad it gets. The 15 s boundary sits on its own so a timeout is - // distinguishable from merely slow. - addHistogramView( - *views, - metric::dnsResolveLatencyMs, - {1.0, - 5.0, - 10.0, - 25.0, - 50.0, - 100.0, - 250.0, - 500.0, - 1'000.0, - 2'500.0, - 5'000.0, - 10'000.0, - 15'000.0, - 20'000.0, - 30'000.0}); - addHistogramView( - *views, - metric::overlayDialLatencyMs, - {1.0, - 5.0, - 10.0, - 25.0, - 50.0, - 100.0, - 250.0, - 500.0, - 1'000.0, - 2'500.0, - 5'000.0, - 10'000.0, - 15'000.0, - 20'000.0, - 30'000.0}); - - // The remaining two GetObject histograms are not durations, so the - // microsecond ladder above does not fit them. Both still need explicit - // boundaries: the SDK default stops at 10,000 and both ranges exceed it. - // - // Object counts run 1..kHardMaxReplyNodes (12288). The honest sync path - // asks for at most 8, so the low buckets are fine-grained and the upper - // ones follow the charge size bands (64, 1024) up to the hard cap. - addHistogramView( - *views, kGetObjectRequestObjects, buckets::toVector(buckets::kObjectCountBuckets)); - - // Charge values span 0 (free tier) to ~99k for a full-size all-miss - // request. Boundaries bracket the resource thresholds that decide a - // peer's fate -- kWarningThreshold (5000) and kDropThreshold (25000) -- - // so a dashboard can show how close charges run to each. - addHistogramView(*views, kGetObjectCharge, buckets::toVector(buckets::kChargeBuckets)); - - // The two RPC request-count histograms are recorded at their ServerHandler - // and PathRequest call sites, so the names come from the shared constants - // all three sites use. Both are small counts, and the reason they need a - // view is the FLOOR rather than the ceiling: the SDK default edges start - // 0, 5, 10, 25, so a batch of one to five sub-requests -- the normal case -- - // would land in a single bucket and every quantile over it would be an - // interpolation inside that bucket rather than a measurement. - // - // The object-count ladder is the fit: its 1, 2, 4, 8, 16 edges sit exactly - // where both distributions have their mass. Path counts are hard-bounded at - // kMaxPaths * kMaxAutoSrcCur = 352, well under its 12288 top. Batch sizes - // have no such cap; see the ceiling note in RpcMetricNames.h. - addHistogramView(*views, kRpcBatchSize, buckets::toVector(buckets::kObjectCountBuckets)); - addHistogramView( - *views, kPathfindDiscoveredPaths, buckets::toVector(buckets::kObjectCountBuckets)); - - // Create MeterProvider with resource, then attach the metric reader. - provider_ = metric_sdk::MeterProviderFactory::Create(std::move(views), resourceAttrs); - provider_->AddMetricReader(std::move(reader)); - - // Get a meter for all xrpld instruments. - meter_ = provider_->GetMeter(std::string(kMeterName), std::string(kMeterVersion)); -} - -void -MetricsRegistry::initSyncInstruments() -{ - // RPC per-method counters and histogram. - rpcStartedCounter_ = - meter_->CreateUInt64Counter("rpc_method_started_total", "Total RPC method calls started"); - rpcFinishedCounter_ = meter_->CreateUInt64Counter( - "rpc_method_finished_total", "Total RPC method calls completed successfully"); - rpcErroredCounter_ = meter_->CreateUInt64Counter( - "rpc_method_errored_total", "Total RPC method calls that errored"); - rpcDurationHistogram_ = meter_->CreateDoubleHistogram( - kRpcMethodDurationUs, "RPC method execution time in microseconds"); - - // Job queue per-type counters and histograms. - jobQueuedCounter_ = meter_->CreateUInt64Counter("job_queued_total", "Total jobs enqueued"); - jobStartedCounter_ = meter_->CreateUInt64Counter("job_started_total", "Total jobs started"); - jobFinishedCounter_ = meter_->CreateUInt64Counter("job_finished_total", "Total jobs completed"); - jobStallCounter_ = meter_->CreateUInt64Counter( - metric::jobqStallTotal, "Jobs whose run time reached the 1 s stall threshold"); - jobQueuedDurationHistogram_ = meter_->CreateDoubleHistogram( - kJobQueuedDurationUs, "Time jobs spent waiting in the queue (microseconds)"); - jobRunningDurationHistogram_ = - meter_->CreateDoubleHistogram(kJobRunningDurationUs, "Job execution time in microseconds"); - - // --- External dashboard parity counters --- - ledgersClosedCounter_ = - meter_->CreateUInt64Counter("ledgers_closed_total", "Total ledgers closed by consensus"); - validationsSentCounter_ = meter_->CreateUInt64Counter( - "validations_sent_total", "Total validations sent by this node"); - validationsCheckedCounter_ = meter_->CreateUInt64Counter( - "validations_checked_total", "Total network validations received and checked"); - // state_changes_total is NOT created here. It is emitted at its call site - // (NetworkOPsImp::setMode) through XRPL_METRIC_COUNTER_INC_LABELED so it - // can carry the {from,to} transition labels; a registry-owned instrument - // would only give an unlabelled total. - ledgerHistoryMismatchCounter_ = meter_->CreateUInt64Counter( - metric::ledgerHistoryMismatchTotal, "Total built-vs-validated ledger mismatches by reason"); - txqExpiredCounter_ = meter_->CreateUInt64Counter( - "txq_expired_total", "Total transactions expired out of the transaction queue"); - txqDroppedCounter_ = meter_->CreateUInt64Counter( - "txq_dropped_total", "Total transactions refused admission to the queue by reason"); - // Note: validation_agreements_total / validation_missed_total are monotonic - // ObservableCounters created in registerValidationTotalsCounters() (below). -} -#endif // XRPL_ENABLE_TELEMETRY - -void -MetricsRegistry::detachCallbacks() noexcept +AppMetricGauges::detachCallbacks() noexcept { #ifdef XRPL_ENABLE_TELEMETRY // Release so every subsequent callback acquire-load sees true. @@ -588,177 +162,6 @@ MetricsRegistry::detachCallbacks() noexcept #endif // XRPL_ENABLE_TELEMETRY } -void -MetricsRegistry::stop() -{ -#ifdef XRPL_ENABLE_TELEMETRY - // Store Stopped with release ordering BEFORE the pipeline goes away. - // Every recording thread reads phase_ through recording() with acquire - // ordering, so any record that has not yet passed the gate will see - // Stopped and skip. Idempotent: destructor calls this after run() or - // ~ApplicationImp already did. - phase_.store(Phase::Stopped, std::memory_order_release); - if (!provider_) - return; - - JLOG(journal_.info()) << "MetricsRegistry: stopping"; - - // Belt-and-suspenders: detachCallbacks() should have already been - // called by Application shutdown before any service the callbacks - // observe was stopped. Setting the flag here is redundant for a - // correct caller but protects against a future caller that forgets - // to detach first. - callbacksDetached_.store(true, std::memory_order_release); - - // meter_ is left alone on purpose. Job threads are still running here and - // may be inside a macro, so writing meter_ would race with their read. - // The recording() gate is what keeps them off the dying pipeline: only the - // macros read meter_, and none of them does so once phase_ is Stopped. - // - // SDK teardown order: Shutdown() stops the PeriodicExportingMetricReader - // thread (so no further gauge callbacks fire) and performs the final - // collect-and-export drain itself. The trailing ForceFlush() is a - // redundant safety net (a no-op once the reader is shut down), then - // reset() destroys the provider. - // - // provider_.reset() destroys MeterProvider -> MeterContext -> ViewRegistry - // -> each View -> its shared_ptr. Live SDK - // SyncMetricStorage instances cached in call-site statics still hold a - // raw AggregationConfig pointer; a Record with a NEW attribute set after - // this point would fire the factory lambda and deref that dangling - // pointer, and a late meter()->CreateXxx would return null. - provider_->Shutdown(); - provider_->ForceFlush(); - provider_.reset(); - - JLOG(journal_.info()) << "MetricsRegistry: stopped"; -#endif // XRPL_ENABLE_TELEMETRY -} - -// ----------------------------------------------------------------- -// Synchronous instrument recording — RPC metrics -// ----------------------------------------------------------------- - -void -MetricsRegistry::recordRpcStarted([[maybe_unused]] std::string_view method) -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (!recording() || !rpcStartedCounter_) - return; - rpcStartedCounter_->Add(1, {{"method", std::string(method)}}); -#endif -} - -void -MetricsRegistry::recordRpcFinished( - [[maybe_unused]] std::string_view method, - [[maybe_unused]] std::int64_t durationUs) -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (!recording() || !rpcFinishedCounter_) - return; - rpcFinishedCounter_->Add(1, {{"method", std::string(method)}}); - if (rpcDurationHistogram_) - { - rpcDurationHistogram_->Record( - static_cast(durationUs), - {{"method", std::string(method)}}, - opentelemetry::context::Context{}); - } -#endif -} - -void -MetricsRegistry::recordRpcErrored( - [[maybe_unused]] std::string_view method, - [[maybe_unused]] std::int64_t durationUs) -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (!recording() || !rpcErroredCounter_) - return; - rpcErroredCounter_->Add(1, {{"method", std::string(method)}}); - if (rpcDurationHistogram_) - { - rpcDurationHistogram_->Record( - static_cast(durationUs), - {{"method", std::string(method)}}, - opentelemetry::context::Context{}); - } -#endif -} - -// ----------------------------------------------------------------- -// Synchronous instrument recording — Job Queue metrics -// ----------------------------------------------------------------- - -void -MetricsRegistry::recordJobQueued( - [[maybe_unused]] std::string_view jobType, - [[maybe_unused]] std::string_view jobName) -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (!recording() || !jobQueuedCounter_) - return; - jobQueuedCounter_->Add( - 1, - {{label::jobType, std::string(jobType)}, - {label::handler, std::string(sanitiseHandler(jobName))}}); -#endif -} - -void -MetricsRegistry::recordJobStarted( - [[maybe_unused]] std::string_view jobType, - [[maybe_unused]] std::string_view jobName, - [[maybe_unused]] std::int64_t queuedDurUs) -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (!recording() || !jobStartedCounter_) - return; - // Build the attribute pair once: both the counter and the histogram - // must carry the identical label set or they cannot be joined. - std::string const handler(sanitiseHandler(jobName)); - jobStartedCounter_->Add(1, {{label::jobType, std::string(jobType)}, {label::handler, handler}}); - if (jobQueuedDurationHistogram_ && queuedDurUs >= 0) - { - // Guard against negative queued durations: the caller derives this - // from a steady-clock delta that can go slightly negative under clock - // skew or reordering. The OTel SDK rejects negative histogram values - // (logging a warning per call), so skip them rather than spam. - jobQueuedDurationHistogram_->Record( - static_cast(queuedDurUs), - {{label::jobType, std::string(jobType)}, {label::handler, handler}}, - opentelemetry::context::Context{}); - } -#endif -} - -void -MetricsRegistry::recordJobFinished( - [[maybe_unused]] std::string_view jobType, - [[maybe_unused]] std::string_view jobName, - [[maybe_unused]] std::int64_t runningDurUs) -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (!recording() || !jobFinishedCounter_) - return; - std::string const handler(sanitiseHandler(jobName)); - jobFinishedCounter_->Add( - 1, {{label::jobType, std::string(jobType)}, {label::handler, handler}}); - if (jobRunningDurationHistogram_) - { - jobRunningDurationHistogram_->Record( - static_cast(runningDurUs), - {{label::jobType, std::string(jobType)}, {label::handler, handler}}, - opentelemetry::context::Context{}); - } - // One compare per job finish. A process-wide freeze shows up here as - // several job types crossing the bar in the same second. - if (runningDurUs >= kJobStallThresholdUs && jobStallCounter_) - jobStallCounter_->Add(1, {{label::jobType, std::string(jobType)}}); -#endif -} - // ----------------------------------------------------------------- // Observable gauge callbacks // ----------------------------------------------------------------- @@ -766,7 +169,7 @@ MetricsRegistry::recordJobFinished( #ifdef XRPL_ENABLE_TELEMETRY void -MetricsRegistry::registerAsyncGauges() +AppMetricGauges::registerAsyncGauges() { // Each helper creates one observable instrument and attaches one // callback. Keeping the registration bodies in separate methods @@ -804,7 +207,7 @@ MetricsRegistry::registerAsyncGauges() } void -MetricsRegistry::registerJqTransOverflowCounter() +AppMetricGauges::registerJqTransOverflowCounter() { // jq_trans_overflow_total is observed from Overlay's existing cumulative // atomic (Overlay::getJqTransOverflow()) rather than pushed. The overlay @@ -816,11 +219,11 @@ MetricsRegistry::registerJqTransOverflowCounter() // callback reads getOverlay(), which asserts overlay_ is non-null. Arming // it any earlier would let a reader tick fire before the overlay exists, // and an assert is not caught by the try block below. - jqTransOverflowObservable_ = meter_->CreateInt64ObservableCounter( + jqTransOverflowObservable_ = core_.meter()->CreateInt64ObservableCounter( "jq_trans_overflow_total", "Total job queue transaction overflows"); jqTransOverflowObservable_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; try @@ -838,14 +241,14 @@ MetricsRegistry::registerJqTransOverflowCounter() } void -MetricsRegistry::registerCacheHitRateGauge() +AppMetricGauges::registerCacheHitRateGauge() { // --- Cache hit rate and size gauges --- cacheHitRateGauge_ = - meter_->CreateDoubleObservableGauge("cache_metrics", "Cache hit rates and sizes"); + core_.meter()->CreateDoubleObservableGauge("cache_metrics", "Cache hit rates and sizes"); cacheHitRateGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -902,7 +305,7 @@ MetricsRegistry::registerCacheHitRateGauge() // Longest TaggedCache mutex hold since the last tick. // Split out to keep this callback under the 80-line limit. - MetricsRegistry::observeCacheLockHoldPeaks(result, app); + AppMetricGauges::observeCacheLockHoldPeaks(result, app); } catch (...) // NOLINT(bugprone-empty-catch) { @@ -913,7 +316,7 @@ MetricsRegistry::registerCacheHitRateGauge() } void -MetricsRegistry::observeCacheLockHoldPeaks( +AppMetricGauges::observeCacheLockHoldPeaks( opentelemetry::metrics::ObserverResult& result, ServiceRegistry& app) { @@ -935,13 +338,14 @@ MetricsRegistry::observeCacheLockHoldPeaks( } void -MetricsRegistry::registerTxqGauge() +AppMetricGauges::registerTxqGauge() { // --- TxQ metrics gauges --- - txqGauge_ = meter_->CreateDoubleObservableGauge("txq_metrics", "Transaction queue metrics"); + txqGauge_ = + core_.meter()->CreateDoubleObservableGauge("txq_metrics", "Transaction queue metrics"); txqGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -982,14 +386,14 @@ MetricsRegistry::registerTxqGauge() } void -MetricsRegistry::registerObjectCountGauge() +AppMetricGauges::registerObjectCountGauge() { // --- Counted object instance gauges --- - objectCountGauge_ = meter_->CreateInt64ObservableGauge( + objectCountGauge_ = core_.meter()->CreateInt64ObservableGauge( "object_count", "Live instance counts for key internal object types"); objectCountGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; try @@ -1014,14 +418,14 @@ MetricsRegistry::registerObjectCountGauge() } void -MetricsRegistry::registerLoadFactorGauge() +AppMetricGauges::registerLoadFactorGauge() { // --- Load factor breakdown gauges --- - loadFactorGauge_ = - meter_->CreateDoubleObservableGauge("load_factor_metrics", "Fee load factor breakdown"); + loadFactorGauge_ = core_.meter()->CreateDoubleObservableGauge( + "load_factor_metrics", "Fee load factor breakdown"); loadFactorGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1087,7 +491,7 @@ MetricsRegistry::registerLoadFactorGauge() } void -MetricsRegistry::observeNodeStoreTotals(node_store::Database& db, ObserveFn const& observe) +AppMetricGauges::observeNodeStoreTotals(node_store::Database& db, ObserveFn const& observe) { // Cumulative counters (monotonically increasing). observe("node_reads_total", static_cast(db.getFetchTotalCount())); @@ -1106,9 +510,10 @@ MetricsRegistry::observeNodeStoreTotals(node_store::Database& db, ObserveFn cons // latency differs. Each mean is omitted rather than reported as zero // when nothing has been read or written, so a dashboard shows a gap // instead of a plausible wrong number. - if (auto const mean = scaledMean(db.getFetchDurationUs(), db.getFetchTotalCount())) + if (auto const mean = + MetricsRegistry::scaledMean(db.getFetchDurationUs(), db.getFetchTotalCount())) observe("read_mean_us", *mean); - if (auto const mean = scaledMean(db.getStoreDurationUs(), db.getStoreCount())) + if (auto const mean = MetricsRegistry::scaledMean(db.getStoreDurationUs(), db.getStoreCount())) observe("write_mean_us", *mean); // Write load score (instantaneous). @@ -1116,7 +521,7 @@ MetricsRegistry::observeNodeStoreTotals(node_store::Database& db, ObserveFn cons } void -MetricsRegistry::observeWritePathDetail(node_store::Database const& db, ObserveFn const& observe) +AppMetricGauges::observeWritePathDetail(node_store::Database const& db, ObserveFn const& observe) { auto const ws = db.getWriteStats(); if (!ws) @@ -1125,7 +530,7 @@ MetricsRegistry::observeWritePathDetail(node_store::Database const& db, ObserveF observe("nudb_writers_in_flight", static_cast(ws->concurrentWriters)); observe("nudb_insert_max_us", static_cast(ws->insertMaxUs)); - if (auto const mean = scaledMean(ws->insertTotalUs, ws->insertCount)) + if (auto const mean = MetricsRegistry::scaledMean(ws->insertTotalUs, ws->insertCount)) observe("nudb_insert_mean_us", *mean); // Mean writer depth times 100. NuDB serializes inserts behind one @@ -1133,12 +538,12 @@ MetricsRegistry::observeWritePathDetail(node_store::Database const& db, ObserveF // above 1.0 even under load. An integral gauge would truncate that to 1 // and lose the whole signal, hence the fixed-point scale -- which the // name states, so nobody reads 140 as 140 writers. - if (auto const mean = scaledMean(ws->depthSum, ws->depthSamples, 100)) + if (auto const mean = MetricsRegistry::scaledMean(ws->depthSum, ws->depthSamples, 100)) observe("nudb_writer_depth_x100", *mean); } void -MetricsRegistry::observeAcquireStats(AcquireStats const& stats, ObserveFn const& observe) +AppMetricGauges::observeAcquireStats(AcquireStats const& stats, ObserveFn const& observe) { // Published unconditionally: for a counter, zero is the meaningful // "no such event yet" reading, unlike for a mean. The diagnostic value @@ -1161,7 +566,7 @@ MetricsRegistry::observeAcquireStats(AcquireStats const& stats, ObserveFn const& } void -MetricsRegistry::observeReadQueue(node_store::Database& db, ObserveFn const& observe) +AppMetricGauges::observeReadQueue(node_store::Database& db, ObserveFn const& observe) { json::Value obj(json::ValueType::Object); db.getCountsJson(obj); @@ -1178,24 +583,24 @@ MetricsRegistry::observeReadQueue(node_store::Database& db, ObserveFn const& obs } void -MetricsRegistry::registerNodeStoreGauge() +AppMetricGauges::registerNodeStoreGauge() { // --- NodeStore I/O gauges --- // The cumulative counters (reads, writes, bytes) are also exposed here // as observable gauges. This avoids adding an xrpld dependency into the - // libxrpl nodestore code — the MetricsRegistry reads the existing atomic + // libxrpl nodestore code — the callback reads the existing atomic // counters from Database via its public accessors. // // Every value multiplexes onto this one gauge through its `metric` // label, so a new value needs no new instrument. The body is split // across four helpers, one per domain, to stay inside the per-function // line budget and to keep each domain testable on its own. - nodeStoreGauge_ = meter_->CreateInt64ObservableGauge( + nodeStoreGauge_ = core_.meter()->CreateInt64ObservableGauge( metric::nodestoreState, "NodeStore I/O counters, latencies, write-queue depth and acquisition stalls"); nodeStoreGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1212,10 +617,10 @@ MetricsRegistry::registerNodeStoreGauge() // Qualified because the enclosing lambda captures nothing: // these are static members, and the explicit scope says so. - MetricsRegistry::observeNodeStoreTotals(db, observe); - MetricsRegistry::observeWritePathDetail(db, observe); - MetricsRegistry::observeAcquireStats(app.getAcquireStats(), observe); - MetricsRegistry::observeReadQueue(db, observe); + AppMetricGauges::observeNodeStoreTotals(db, observe); + AppMetricGauges::observeWritePathDetail(db, observe); + AppMetricGauges::observeAcquireStats(app.getAcquireStats(), observe); + AppMetricGauges::observeReadQueue(db, observe); } catch (...) // NOLINT(bugprone-empty-catch) { @@ -1226,7 +631,7 @@ MetricsRegistry::registerNodeStoreGauge() } void -MetricsRegistry::registerRotationStateGauge() +AppMetricGauges::registerRotationStateGauge() { // --- Sync diagnostics: what an online_delete rotation costs --- // A rotation performs writes an ordinary fetch would not: the archive @@ -1242,11 +647,11 @@ MetricsRegistry::registerRotationStateGauge() // registerNodeStoreGauge above: DatabaseRotatingImp lives in libxrpl and // cannot include xrpld/telemetry, so the counters are read through the // DatabaseRotating accessors on each collection tick instead. - rotationStateGauge_ = meter_->CreateInt64ObservableGauge( + rotationStateGauge_ = core_.meter()->CreateInt64ObservableGauge( metric::rotationState, "Online-delete rotation state and copy-forward write total"); rotationStateGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1290,14 +695,14 @@ MetricsRegistry::registerRotationStateGauge() } void -MetricsRegistry::registerServerInfoGauge() +AppMetricGauges::registerServerInfoGauge() { // --- Server info gauges --- - serverInfoGauge_ = - meter_->CreateInt64ObservableGauge(metric::serverInfo, "Server-level health metrics"); + serverInfoGauge_ = core_.meter()->CreateInt64ObservableGauge( + metric::serverInfo, "Server-level health metrics"); serverInfoGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1375,10 +780,11 @@ MetricsRegistry::registerServerInfoGauge() } void -MetricsRegistry::registerBuildInfoGauge() +AppMetricGauges::registerBuildInfoGauge() { // --- Build info gauge --- - buildInfoGauge_ = meter_->CreateInt64ObservableGauge("build_info", "Build version information"); + buildInfoGauge_ = + core_.meter()->CreateInt64ObservableGauge("build_info", "Build version information"); buildInfoGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* /* state */) { try @@ -1395,14 +801,14 @@ MetricsRegistry::registerBuildInfoGauge() } void -MetricsRegistry::registerCompleteLedgersGauge() +AppMetricGauges::registerCompleteLedgersGauge() { // --- Complete ledgers range gauge --- - completeLedgersGauge_ = meter_->CreateInt64ObservableGauge( + completeLedgersGauge_ = core_.meter()->CreateInt64ObservableGauge( "complete_ledgers", "Complete ledger range start/end pairs"); completeLedgersGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1452,14 +858,14 @@ MetricsRegistry::registerCompleteLedgersGauge() } void -MetricsRegistry::registerDbMetricsGauge() +AppMetricGauges::registerDbMetricsGauge() { // --- Database size and fetch rate gauges --- - dbMetricsGauge_ = - meter_->CreateInt64ObservableGauge("db_metrics", "Database storage sizes and fetch rates"); + dbMetricsGauge_ = core_.meter()->CreateInt64ObservableGauge( + "db_metrics", "Database storage sizes and fetch rates"); dbMetricsGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1491,14 +897,14 @@ MetricsRegistry::registerDbMetricsGauge() } void -MetricsRegistry::registerValidatorHealthGauge() +AppMetricGauges::registerValidatorHealthGauge() { // --- Validator health gauges --- - validatorHealthGauge_ = - meter_->CreateDoubleObservableGauge("validator_health", "Validator health indicators"); + validatorHealthGauge_ = core_.meter()->CreateDoubleObservableGauge( + "validator_health", "Validator health indicators"); validatorHealthGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1538,16 +944,16 @@ MetricsRegistry::registerValidatorHealthGauge() } void -MetricsRegistry::registerPeerQualityGauge() +AppMetricGauges::registerPeerQualityGauge() { // --- Peer quality gauges --- // Uses Peer::json() to read latency and version since those accessors // are not on the abstract Peer interface (they live on PeerImp). - peerQualityGauge_ = - meter_->CreateDoubleObservableGauge(metric::peerQuality, "Peer network quality metrics"); + peerQualityGauge_ = core_.meter()->CreateDoubleObservableGauge( + metric::peerQuality, "Peer network quality metrics"); peerQualityGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1641,18 +1047,18 @@ MetricsRegistry::registerPeerQualityGauge() } void -MetricsRegistry::registerReduceRelayGauge() +AppMetricGauges::registerReduceRelayGauge() { // Transaction reduce-relay efficiency. Overlay::txMetrics() exposes the // rolling averages as a JSON object with string values (std::to_string), // so parse each field. A high suppressed:selected ratio proves the // feature is saving bandwidth; a high not_enabled count means stale peers // force full relay. - reduceRelayGauge_ = meter_->CreateInt64ObservableGauge( + reduceRelayGauge_ = core_.meter()->CreateInt64ObservableGauge( "reduce_relay_metrics", "Transaction reduce-relay efficiency metrics"); reduceRelayGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1692,14 +1098,14 @@ MetricsRegistry::registerReduceRelayGauge() } void -MetricsRegistry::registerLedgerEconomyGauge() +AppMetricGauges::registerLedgerEconomyGauge() { // --- Ledger economy gauges --- - ledgerEconomyGauge_ = meter_->CreateDoubleObservableGauge( + ledgerEconomyGauge_ = core_.meter()->CreateDoubleObservableGauge( metric::ledgerEconomy, "Ledger fee and economy metrics"); ledgerEconomyGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1757,14 +1163,14 @@ MetricsRegistry::registerLedgerEconomyGauge() } void -MetricsRegistry::registerStateTrackingGauge() +AppMetricGauges::registerStateTrackingGauge() { // --- State tracking gauges --- - stateTrackingGauge_ = - meter_->CreateDoubleObservableGauge(metric::stateTracking, "Node state and mode tracking"); + stateTrackingGauge_ = core_.meter()->CreateDoubleObservableGauge( + metric::stateTracking, "Node state and mode tracking"); stateTrackingGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1811,7 +1217,7 @@ MetricsRegistry::registerStateTrackingGauge() } void -MetricsRegistry::registerStorageDetailGauge() +AppMetricGauges::registerStorageDetailGauge() { // --- Storage detail gauges --- // Reports the cumulative payload bytes handed to the NodeStore. See the @@ -1819,10 +1225,10 @@ MetricsRegistry::registerStorageDetailGauge() // on-disk file size, because no accessor for the latter exists. The label // value names it that way so it is not read as a filesystem measurement. storageDetailGauge_ = - meter_->CreateInt64ObservableGauge("storage_detail", "Storage detail metrics"); + core_.meter()->CreateInt64ObservableGauge("storage_detail", "Storage detail metrics"); storageDetailGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1863,7 +1269,7 @@ MetricsRegistry::registerStorageDetailGauge() } void -MetricsRegistry::registerValidationAgreementGauge() +AppMetricGauges::registerValidationAgreementGauge() { // --- Validation agreement gauges --- // Reports rolling-window agreement percentages and counts from @@ -1871,18 +1277,18 @@ MetricsRegistry::registerValidationAgreementGauge() // callback so that pending ledger events are resolved before the // window data is read (the callback fires every ~10 s from the // PeriodicExportingMetricReader thread). - validationAgreementGauge_ = meter_->CreateDoubleObservableGauge( + validationAgreementGauge_ = core_.meter()->CreateDoubleObservableGauge( "validation_agreement", "Validation agreement percentages and counts (1h/24h windows)"); validationAgreementGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; try { // Reconcile pending events before reading window data. - self->validationTracker_.reconcile(); + self->core_.getValidationTracker().reconcile(); auto observe = [&](char const* name, double value) { opentelemetry::nostd::getObserve(value, {{label::metric, name}}); }; - observe("agreement_pct_1h", self->validationTracker_.agreementPct1h()); - observe("agreement_pct_24h", self->validationTracker_.agreementPct24h()); + observe("agreement_pct_1h", self->core_.getValidationTracker().agreementPct1h()); + observe("agreement_pct_24h", self->core_.getValidationTracker().agreementPct24h()); observe( - "agreements_1h", static_cast(self->validationTracker_.agreements1h())); - observe("missed_1h", static_cast(self->validationTracker_.missed1h())); + "agreements_1h", + static_cast(self->core_.getValidationTracker().agreements1h())); + observe( + "missed_1h", + static_cast(self->core_.getValidationTracker().missed1h())); observe( "agreements_24h", - static_cast(self->validationTracker_.agreements24h())); - observe("missed_24h", static_cast(self->validationTracker_.missed24h())); + static_cast(self->core_.getValidationTracker().agreements24h())); + observe( + "missed_24h", + static_cast(self->core_.getValidationTracker().missed24h())); // 7-day window (matches external xrpl-validator-dashboard). - observe("agreement_pct_7d", self->validationTracker_.agreementPct7d()); + observe("agreement_pct_7d", self->core_.getValidationTracker().agreementPct7d()); observe( - "agreements_7d", static_cast(self->validationTracker_.agreements7d())); - observe("missed_7d", static_cast(self->validationTracker_.missed7d())); + "agreements_7d", + static_cast(self->core_.getValidationTracker().agreements7d())); + observe( + "missed_7d", + static_cast(self->core_.getValidationTracker().missed7d())); } catch (...) // NOLINT(bugprone-empty-catch) { @@ -1915,12 +1329,12 @@ MetricsRegistry::registerValidationAgreementGauge() } void -MetricsRegistry::registerValidationTotalsCounters() +AppMetricGauges::registerValidationTotalsCounters() { // Lifetime validation agreement/miss counters. // - // These are monotonic ObservableCounters (not the sync Counters they used - // to be): a Prometheus _total must never decrease, but ValidationTracker's + // These are monotonic ObservableCounters rather than synchronous Counters: + // a Prometheus _total must never decrease, but ValidationTracker's // NET totals are non-monotonic (a late repair decrements the net miss // count). We therefore observe the tracker's GROSS lifetime tallies, which // count each ledger once at first classification and are never adjusted on @@ -1930,20 +1344,22 @@ MetricsRegistry::registerValidationTotalsCounters() // reconcile() is called first so pending events are resolved before the // tallies are read; the callback fires every ~10 s from the // PeriodicExportingMetricReader thread. - validationAgreementsObservable_ = meter_->CreateInt64ObservableCounter( + validationAgreementsObservable_ = core_.meter()->CreateInt64ObservableCounter( "validation_agreements_total", "Lifetime validations that initially agreed with network consensus"); validationAgreementsObservable_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; try { - self->validationTracker_.reconcile(); + self->core_.getValidationTracker().reconcile(); opentelemetry::nostd::get>>(result) - ->Observe(static_cast(self->validationTracker_.totalAgreementsEver())); + ->Observe( + static_cast( + self->core_.getValidationTracker().totalAgreementsEver())); } catch (...) // NOLINT(bugprone-empty-catch) { @@ -1952,19 +1368,20 @@ MetricsRegistry::registerValidationTotalsCounters() }, this); - validationMissedObservable_ = meter_->CreateInt64ObservableCounter( + validationMissedObservable_ = core_.meter()->CreateInt64ObservableCounter( "validation_missed_total", "Lifetime validations that initially missed network consensus"); validationMissedObservable_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; try { - self->validationTracker_.reconcile(); + self->core_.getValidationTracker().reconcile(); opentelemetry::nostd::get>>(result) - ->Observe(static_cast(self->validationTracker_.totalMissedEver())); + ->Observe( + static_cast(self->core_.getValidationTracker().totalMissedEver())); } catch (...) // NOLINT(bugprone-empty-catch) { @@ -1975,17 +1392,17 @@ MetricsRegistry::registerValidationTotalsCounters() } void -MetricsRegistry::registerUnlQuorumGauge() +AppMetricGauges::registerUnlQuorumGauge() { // --- Sync diagnostics: trusted UNL size against required quorum --- // validator_health already exports the quorum on its own; pairing it // with the trusted-key count in one instrument is what makes the // "can this node ever validate?" comparison a single query. - unlQuorumGauge_ = meter_->CreateInt64ObservableGauge( + unlQuorumGauge_ = core_.meter()->CreateInt64ObservableGauge( metric::unlQuorum, "Trusted UNL key count vs required quorum"); unlQuorumGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -2034,18 +1451,18 @@ MetricsRegistry::registerUnlQuorumGauge() } void -MetricsRegistry::registerClockSkewGauge() +AppMetricGauges::registerClockSkewGauge() { // --- Sync diagnostics: network close-time offset --- // A persistent offset shows the local clock disagrees with the // network, which delays consensus participation. server_info hides // this below 60 s, so export it continuously instead. - clockSkewGauge_ = meter_->CreateInt64ObservableGauge( + clockSkewGauge_ = core_.meter()->CreateInt64ObservableGauge( metric::clockCloseOffsetSeconds, "Network close time offset from the local clock, in seconds"); clockSkewGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -2072,16 +1489,16 @@ MetricsRegistry::registerClockSkewGauge() } void -MetricsRegistry::registerSyncStateGauge() +AppMetricGauges::registerSyncStateGauge() { // --- Sync diagnostics: why a fresh node is not FULL yet --- // Four values otherwise visible only in a log line or in server_info // JSON. All four are cheap reads pulled on the ~10 s reader tick. - syncStateGauge_ = - meter_->CreateInt64ObservableGauge(metric::syncState, "Sync-pipeline health signals"); + syncStateGauge_ = core_.meter()->CreateInt64ObservableGauge( + metric::syncState, "Sync-pipeline health signals"); syncStateGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -2126,7 +1543,7 @@ MetricsRegistry::registerSyncStateGauge() } void -MetricsRegistry::registerStallEventsCounter() +AppMetricGauges::registerStallEventsCounter() { // --- Sync diagnostics: stall episode count --- // Observed rather than pushed: LoadManager's monitor thread already owns @@ -2134,11 +1551,11 @@ MetricsRegistry::registerStallEventsCounter() // cycle without threading a push path through the load-monitor loop. // Kept out of the sync_state gauge because a cumulative total needs // counter aggregation for rate() to be meaningful. - stallEventsObservable_ = meter_->CreateInt64ObservableCounter( + stallEventsObservable_ = core_.meter()->CreateInt64ObservableCounter( metric::serverStallEventsTotal, "Total server main-loop stall episodes"); stallEventsObservable_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; try @@ -2157,17 +1574,17 @@ MetricsRegistry::registerStallEventsCounter() } void -MetricsRegistry::registerSyncAcquireGauge() +AppMetricGauges::registerSyncAcquireGauge() { // --- Sync diagnostics: is ledger acquisition actually progressing? --- // Aggregated on purpose: a per-ledger label would add one series per ledger // acquired, which is unbounded. The per-ledger view lives on the // ledger.acquire span instead. - syncAcquireGauge_ = meter_->CreateInt64ObservableGauge( + syncAcquireGauge_ = core_.meter()->CreateInt64ObservableGauge( metric::syncAcquire, "Aggregate ledger-acquire progress across in-flight acquires"); syncAcquireGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -2211,16 +1628,16 @@ MetricsRegistry::registerSyncAcquireGauge() } void -MetricsRegistry::registerCacheHitRateDetailGauge() +AppMetricGauges::registerCacheHitRateDetailGauge() { // --- Sync diagnostics: SHAMap tree-node cache hit rate --- // The memory layer above the node store: a miss here is what causes a // node-store read, which the NuDB hit-ratio panel then measures. - shamapCacheHitRateGauge_ = meter_->CreateDoubleObservableGauge( + shamapCacheHitRateGauge_ = core_.meter()->CreateDoubleObservableGauge( metric::shamapCacheHitRate, "SHAMap tree-node cache hit rate (0.0-1.0), by cache"); shamapCacheHitRateGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -2245,18 +1662,18 @@ MetricsRegistry::registerCacheHitRateDetailGauge() } void -MetricsRegistry::registerJobQueueSaturationGauge() +AppMetricGauges::registerJobQueueSaturationGauge() { // --- Sync diagnostics: is the whole worker pool exhausted? --- // Attributes a broad multi-stage slowdown to the pool once, instead of // leaving it to look like an independent fault in every subsystem whose // jobs are queued behind it. - jobQueueSaturationGauge_ = meter_->CreateInt64ObservableGauge( + jobQueueSaturationGauge_ = core_.meter()->CreateInt64ObservableGauge( metric::jobqSaturation, "Worker-pool saturation: tasks in flight, worker threads, jobs queued"); jobQueueSaturationGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -2292,18 +1709,18 @@ MetricsRegistry::registerJobQueueSaturationGauge() } void -MetricsRegistry::registerPeerLedgerSupplyGauge() +AppMetricGauges::registerPeerLedgerSupplyGauge() { // --- Sync diagnostics: can the network even serve what I need? --- // Each peer advertises its ledger range and the connection caches it, but // nothing ever compared those ranges, so "no peer holds the sequence I // want" looked exactly like "my peers are slow" -- two faults with // completely different fixes. - peerLedgerSupplyGauge_ = meter_->CreateInt64ObservableGauge( + peerLedgerSupplyGauge_ = core_.meter()->CreateInt64ObservableGauge( metric::peerLedgerSupply, "Peer coverage of the ledger sequence this node needs"); peerLedgerSupplyGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -2344,17 +1761,17 @@ MetricsRegistry::registerPeerLedgerSupplyGauge() } void -MetricsRegistry::registerSlotCensusGauge() +AppMetricGauges::registerSlotCensusGauge() { // --- Sync diagnostics: why can this node not get peers? --- // All nine numbers already exist inside PeerFinder; only the two active // counts are exported today, which cannot distinguish "not dialling", // "dialling and failing" and "nothing to dial". - slotCensusGauge_ = meter_->CreateInt64ObservableGauge( + slotCensusGauge_ = core_.meter()->CreateInt64ObservableGauge( metric::peerfinderSlotCensus, "PeerFinder slots, connection attempts and address caches"); slotCensusGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -2398,17 +1815,17 @@ MetricsRegistry::registerSlotCensusGauge() } void -MetricsRegistry::registerAmendmentBlockGauge() +AppMetricGauges::registerAmendmentBlockGauge() { // --- Sync diagnostics: how long until this node stops validating? --- // The existing validator_health{metric="amendment_blocked"} reports the // terminal state, when nothing can be done. This is the window before it. - amendmentBlockGauge_ = meter_->CreateInt64ObservableGauge( + amendmentBlockGauge_ = core_.meter()->CreateInt64ObservableGauge( metric::amendmentBlock, "Amendment-block warning and seconds until the node stops validating"); amendmentBlockGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -2454,7 +1871,7 @@ MetricsRegistry::registerAmendmentBlockGauge() } void -MetricsRegistry::registerLedgerQuorumPublishGauge() +AppMetricGauges::registerLedgerQuorumPublishGauge() { // --- Sync diagnostics: the quorum gate and the publish pipeline --- // The last two stages of a fresh sync, and the two whose failures are @@ -2462,12 +1879,12 @@ MetricsRegistry::registerLedgerQuorumPublishGauge() // declare one validated (quorum short), or validate correctly and never // publish (pipeline behind). The quorum shortfall is otherwise only a // trace log line; the publish lag is not derivable from any other signal. - ledgerQuorumPublishGauge_ = meter_->CreateInt64ObservableGauge( + ledgerQuorumPublishGauge_ = core_.meter()->CreateInt64ObservableGauge( metric::ledgerQuorumPublish, "Pre-accept quorum gate and publish lag (tally vs quorum, first-validated, lag)"); ledgerQuorumPublishGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -2515,62 +1932,4 @@ MetricsRegistry::registerLedgerQuorumPublishGauge() #endif // XRPL_ENABLE_TELEMETRY -// ----------------------------------------------------------------- -// External dashboard parity counter increments -// ----------------------------------------------------------------- - -void -MetricsRegistry::incrementLedgersClosed() -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (recording() && ledgersClosedCounter_) - ledgersClosedCounter_->Add(1); -#endif -} - -void -MetricsRegistry::incrementValidationsSent() -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (recording() && validationsSentCounter_) - validationsSentCounter_->Add(1); -#endif -} - -void -MetricsRegistry::incrementValidationsChecked() -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (recording() && validationsCheckedCounter_) - validationsCheckedCounter_->Add(1); -#endif -} - -void -MetricsRegistry::incrementLedgerHistoryMismatch(std::string_view reason) -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (recording() && ledgerHistoryMismatchCounter_) - ledgerHistoryMismatchCounter_->Add(1, {{"reason", std::string(reason)}}); -#endif -} - -void -MetricsRegistry::incrementTxqExpired() -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (recording() && txqExpiredCounter_) - txqExpiredCounter_->Add(1); -#endif -} - -void -MetricsRegistry::incrementTxqDropped(std::string_view reason) -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (recording() && txqDroppedCounter_) - txqDroppedCounter_->Add(1, {{"reason", std::string(reason)}}); -#endif -} - } // namespace xrpl::telemetry diff --git a/src/xrpld/telemetry/AppMetricGauges.h b/src/xrpld/telemetry/AppMetricGauges.h new file mode 100644 index 0000000000..4b75568814 --- /dev/null +++ b/src/xrpld/telemetry/AppMetricGauges.h @@ -0,0 +1,924 @@ +#pragma once + +/** + * Observable-gauge layer for xrpld — the pull-model half of the OTel metric + * surface. + * + * Declares the class that registers every observable instrument whose callback + * samples live server state, and that owns the handles those registrations + * return. The export pipeline itself — provider, meter, exporter and the + * synchronous counters and histograms — belongs to MetricsRegistry, the + * sibling class in this namespace. This layer borrows that meter and adds the + * pull-model instruments on top of it. + */ + +// Unguarded because the constructor names beast::Journal and MetricsRegistry in +// both configurations. beast::Journal is taken by value, so it needs a complete +// type even when telemetry is off. +#include +#include + +#ifdef XRPL_ENABLE_TELEMETRY +// Guarded like the members that use them: std::atomic by callbacksDetached_, +// std::function and std::int64_t by the ObserveFn sink, the OTel instrument +// headers by the 31 instrument handles, and observer_result.h by the +// ObserverResult parameter of observeCacheLockHoldPeaks(). +#include +#include +#include + +#include +#include +#include +#endif + +namespace xrpl { + +class ServiceRegistry; + +// Defined in src/xrpld/app/ledger/AcquireStats.h. Forward-declared because +// only one helper signature names it, and pulling an xrpld/app header in here +// would widen the dependencies of every file that includes this one. +class AcquireStats; + +namespace node_store { +class Database; +} // namespace node_store + +} // namespace xrpl + +namespace xrpl::telemetry { + +/** + * Registers and owns the pull-model OTel instruments that sample live server + * state. + * + * Each registered callback runs on the OTel reader thread and reads services + * through the ServiceRegistry reference given at construction. Both the core + * registry and the ServiceRegistry are borrowed, so both must outlive this + * object. + * + * Collaborator diagram (ASCII): + * + * AppMetricGauges + * +-- MetricsRegistry (borrowed) + * | +-- meter() -- creates all 31 observable instruments + * | +-- getValidationTracker() -- read by the agreement instruments + * | +-- OTel MeterProvider + * | +-- PeriodicExportingMetricReader (~10 s tick, drives the callbacks) + * +-- ServiceRegistry (borrowed) -- every value the callbacks sample + * +-- 31 ObservableInstrument handles (owned) + * + * Callback flow, once startAsyncGauges() has run: + * + * Reader thread tick (~10 s) + * v + * SDK invokes each callback, passing this object as the state pointer + * v + * callbacksDetached_ true? -- yes --> return without observing anything + * v no + * read current values from the ServiceRegistry, Observe() each one + * v + * the core's pipeline exports them over OTLP/HTTP + * + * One instrument per metric domain: cache hit rates and sizes, TxQ state, + * CountedObject instances, load-factor breakdown, NodeStore I/O and + * acquisition stalls, online-delete rotation state, server info, build + * version, complete ledger ranges, database sizes, validator health, peer + * quality, reduce-relay efficiency, ledger economy, state tracking, storage + * detail, validation agreement, UNL quorum, clock close offset, sync state, + * ledger-acquire progress, SHAMap tree-node cache hit rate, worker-pool + * saturation, peer ledger supply, PeerFinder slot census, amendment block and + * the ledger quorum/publish gate. + * Most multiplex their values through a `metric` label, so a new value needs + * no new instrument; object counts use `type`, build info uses `version`, and + * complete ledgers uses `bound` and `index`. Twenty-seven are ObservableGauges + * and four are ObservableCounters, the latter where the value read is already + * cumulative and must never decrease. + * + * Teardown order is a caller contract, in this order: detachCallbacks(), then + * MetricsRegistry::stop(), then destroy this object. stop() joins the reader + * thread, so once it returns no callback can run again and destruction is + * safe. + * + * @code + * // Primary use. Construct after the core registry, and arm only once every + * // service the callbacks read exists. The overlay is built last, so it + * // fixes where this call can go. + * gauges_ = std::make_unique( + * *metricsRegistry_, *this, logs_->journal("MetricsRegistry")); + * gauges_->startAsyncGauges(); + * + * // Shutdown, in the required order. + * gauges_->detachCallbacks(); + * metricsRegistry_->stop(); + * + * // Edge case: arming without a working pipeline. The core hands out a + * // no-op meter when the pipeline fails to build, so this logs a warning + * // and registers nothing rather than reporting a success it cannot keep. + * // A second startAsyncGauges() behaves the same way. + * gauges_->startAsyncGauges(); + * @endcode + * + * @note Thread safety: + * - The callbacks run on the OTel reader thread, concurrently with the + * writers of the state they read. Each reads only lock-protected or + * atomic state and wraps its body in a catch-all try block, so a + * transient failure never brings down the reader thread. + * - startAsyncGauges() and the destructor are NOT thread-safe with each + * other and belong on the single server lifecycle thread. armed_ is a + * plain bool because that call is its only reader and writer. + * - detachCallbacks() may be called from any thread. It is one release + * store to an atomic that every callback acquire-loads. + * + * @note Limitations: + * - Arms once per object. A second startAsyncGauges() logs a warning and + * registers nothing, so the instruments are never duplicated. + * - detachCallbacks() is one-way. Calling it before startAsyncGauges() + * leaves every instrument registered but permanently silent. + * - Destroying this object while the core is still exporting is unsafe. + * The SDK holds this address as its callback state, and the flag the + * destructor sets dies with the object. Only stop() on the core closes + * that window, which is why it comes first. + * - The instrument set is fixed at registration. A pull-model instrument + * cannot be created lazily, so a new metric domain needs a new helper + * and a new handle here. + */ +class AppMetricGauges +{ +public: + /** + * Bind the layer to the core registry and to the services its callbacks + * will sample. Registers nothing; startAsyncGauges() does that. + * + * @param core Registry owning the meter these instruments are created on, + * and the validation tracker two of them read. Must outlive this object. + * @param app Services the callbacks sample. Must outlive this object. + * @param journal Log output. + */ + AppMetricGauges(MetricsRegistry& core, ServiceRegistry& app, beast::Journal journal); + + /** + * Disarms the callbacks, then releases the instrument handles. + * + * @note This is a last resort, not the teardown path. See the class note + * on destruction order. + */ + ~AppMetricGauges(); + + /** + * Non-copyable, non-movable. The registered callbacks hold this object's + * address, so it cannot move. + */ + AppMetricGauges(AppMetricGauges const&) = delete; + AppMetricGauges& + operator=(AppMetricGauges const&) = delete; + + /** + * Create and arm every pull-model instrument — mostly ObservableGauges, + * plus the ObservableCounters whose source value is already cumulative. + * + * Registering an observable also arms the reader thread to invoke its + * callback on the next tick, so this is an ordering decision and not just + * tidiness: it cannot run before the services those callbacks read exist. + * + * Does nothing but log a warning when the core is disabled, when it has no + * real pipeline, when this object is already armed, or when the core has + * already stopped. + * + * @pre Every service the callbacks read is constructed. The full set, from + * the `app.get*()` calls in the registration helpers, is: Overlay, OPs + * (NetworkOPs), LedgerMaster, OpenLedger, TxQ, NodeStore, NodeFamily, + * Validators, AcceptedLedgerCache, CachedSLEs, AcquireStats, TimeKeeper, + * RelationalDatabase, InboundLedgers, FeeTrack, LoadManager, JobQueue and + * AmendmentTable. + * Overlay is built last, so it fixes this call's position: + * `ServiceRegistry::getOverlay()` `XRPL_ASSERT`s that `overlay_` is + * non-null, and a reader-thread tick before the overlay exists aborts a + * Debug build. The callbacks' catch-all try block does not catch an + * assert. `getTxQ()` and `getRelationalDatabase()` assert likewise. + */ + void + startAsyncGauges(); + + /** + * Disarm every registered callback so it no-ops on the next reader-thread + * tick. + * + * Must be called BEFORE any service the callbacks read (nodeStore, + * overlay, networkOPs, ledgerMaster and the rest) is stopped. The flag is + * checked with acquire ordering at the top of every callback; together + * with the release store here that guarantees no callback starting after + * this returns will dereference an already-stopped service. + * + * Idempotent: the flag is one-way, only ever set to true, and nothing + * clears it. + * + * @note One-way means this is a shutdown-only call. Calling it before + * startAsyncGauges() does not "have no effect" — it permanently disarms + * every instrument that call registers, so they exist but never observe a + * value. + */ + void + detachCallbacks() noexcept; + +#ifdef XRPL_ENABLE_TELEMETRY + /** + * Sink handed to the nodestore_state helpers below. + * + * Every value they publish multiplexes onto the single `nodestore_state` + * instrument through its `metric` label, so the helpers need no access to + * the OTel observer result -- just somewhere to put a name and a number. + */ + using ObserveFn = std::function; + + // The four helpers below are public because each is a pure transform from + // a statistics object to a set of name-value pairs. They read only their + // arguments and need no AppMetricGauges instance, so a test can drive one + // directly with a recording sink and assert the exact `metric` label + // values it publishes. Exposing them widens no state. + + /** + * Observe the NodeStore I/O totals and the means derived from them. + * + * Publishes the four cumulative totals (`node_reads_total`, + * `node_writes`, `node_reads_duration_us`, `node_writes_duration_us`) + * unconditionally, plus `read_mean_us` and `write_mean_us` derived from + * them via MetricsRegistry::scaledMean(). `write_mean_us` is the signal + * for the "a node with a large existing database syncs slower than a fresh + * one" symptom: back-fill is write-bound, so no read-side reading can show + * it. All three concrete store paths time themselves through + * Database::recordStoreDuration(), so the write mean is live on an + * ordinary node. + * + * Gauge rather than histogram, deliberately. A histogram would give true + * percentiles, but it costs one Record() per node object on the + * store/fetch path, and one ledger write walks thousands of SHAMap + * nodes. This reads the existing atomics once per ~10 s tick and adds + * nothing to the hot path. Consequence, stated plainly: p99 is NOT + * obtainable from this signal. A histogram added later would also need an + * explicit-bucket View registered on the core registry's meter provider, + * because the SDK's default buckets top out at 10,000. + * + * @param db NodeStore to read the counters from. + * @param observe Sink for one `metric`-labelled value. + * + * @note The totals are monotonic and never reset, so a panel wanting + * current rather than since-boot latency divides the two rates. That is + * why the counts and duration totals are exported beside the means. + * @note A mean is omitted when its count is 0, so a dashboard shows a gap + * rather than a plausible-looking 0 us. + */ + static void + observeNodeStoreTotals(node_store::Database& db, ObserveFn const& observe); + + /** + * Observe the backend write-path detail, when the backend measures it. + * + * Publishes nothing for a backend whose getWriteStats() is std::nullopt, + * which is every backend except NuDB. Absent labels let a reader tell + * "not measured" from "measured, and idle"; zeros would read as a + * perfectly idle write path. + * + * @param db NodeStore whose writable backend is sampled. + * @param observe Sink for one `metric`-labelled value. + */ + static void + observeWritePathDetail(node_store::Database const& db, ObserveFn const& observe); + + /** + * Observe the ledger-acquisition progress and stall counters. + * + * @param stats Process-wide acquisition counters. + * @param observe Sink for one `metric`-labelled value. + */ + static void + observeAcquireStats(AcquireStats const& stats, ObserveFn const& observe); + + /** + * Observe the read queue depth and the read thread-pool counts. + * + * These four have no accessor on Database, so its JSON counters object + * is still the only way to reach them. + * + * @param db NodeStore to read the JSON counters from. + * @param observe Sink for one `metric`-labelled value. + */ + static void + observeReadQueue(node_store::Database& db, ObserveFn const& observe); + +private: + /** + * Registry owning the meter these instruments are created on, the + * pipeline they export through, and the validation tracker two of them + * read. Borrowed; it outlives this object. + */ + MetricsRegistry& core_; + + /** + * Services the callbacks sample. Borrowed; it outlives this object. + */ + ServiceRegistry& app_; + + /** + * Log output. Shares the `MetricsRegistry` journal partition, so one + * log-level setting covers the whole metric pipeline. + */ + beast::Journal const journal_; + + /** + * True once startAsyncGauges() has registered the instruments. Read and + * written only by that call, from the server lifecycle thread, so it needs + * no atomic. + */ + bool armed_{false}; + + /** + * Set by detachCallbacks() during shutdown so every callback returns early + * before reading services that may already be stopped. Checked with + * memory_order_acquire at the top of each callback to pair with the + * memory_order_release store in detachCallbacks(). + */ + std::atomic callbacksDetached_{false}; + + // --- Observable instrument handles --- + // Held so the callbacks stay registered for as long as this object lives. + /** + * Cache hit rates and sizes. + */ + opentelemetry::nostd::shared_ptr + cacheHitRateGauge_; + /** + * Transaction queue state. + */ + opentelemetry::nostd::shared_ptr txqGauge_; + /** + * Live instance counts for every CountedObject type. + */ + opentelemetry::nostd::shared_ptr + objectCountGauge_; + /** + * Fee load-factor breakdown. + */ + opentelemetry::nostd::shared_ptr loadFactorGauge_; + /** + * Every NodeStore value on one instrument, separated by its `metric` + * label: I/O totals, the read and write means derived from them, the NuDB + * write-queue detail, and the ledger-acquisition stall counters. + */ + opentelemetry::nostd::shared_ptr nodeStoreGauge_; + /** + * Online-delete rotation state and its copy-forward write total. Publishes + * nothing on a node without `online_delete`, where the node store is not a + * rotating one. + */ + opentelemetry::nostd::shared_ptr + rotationStateGauge_; + /** + * Server-level health: operating mode, uptime, peers, ledger sequences and + * the last consensus round. + */ + opentelemetry::nostd::shared_ptr serverInfoGauge_; + /** + * Trusted UNL key count against the required quorum. + */ + opentelemetry::nostd::shared_ptr unlQuorumGauge_; + /** + * Network close-time offset (local clock skew). + */ + opentelemetry::nostd::shared_ptr clockSkewGauge_; + /** + * Sync-pipeline state signals: time to first FULL, the network-ledger gate, + * the current server stall and how many ledgers behind the network. + */ + opentelemetry::nostd::shared_ptr syncStateGauge_; + /** + * ObservableCounter: server_stall_events_total — observed from + * LoadManager::getStallEventCount() (cumulative episode tally owned by the + * load-monitor thread). + */ + opentelemetry::nostd::shared_ptr + stallEventsObservable_; + /** + * Aggregate ledger-acquire progress: max missing state and tx nodes, + * received-data stash depth and the in-flight acquire count. + */ + opentelemetry::nostd::shared_ptr + syncAcquireGauge_; + /** + * SHAMap tree-node cache hit rate, the memory layer above the node store's + * own hit ratio. + */ + opentelemetry::nostd::shared_ptr + shamapCacheHitRateGauge_; + /** + * Global worker-pool saturation: tasks in flight, configured worker threads + * and total jobs queued. + */ + opentelemetry::nostd::shared_ptr + jobQueueSaturationGauge_; + /** + * How much of the needed ledger range the connected peer set can serve. + */ + opentelemetry::nostd::shared_ptr + peerLedgerSupplyGauge_; + /** + * PeerFinder slot occupancy, connection attempts, fixed peers and + * address-cache depth. + */ + opentelemetry::nostd::shared_ptr slotCensusGauge_; + /** + * Amendment-block warning flag and the countdown to the amendment + * activating. + */ + opentelemetry::nostd::shared_ptr + amendmentBlockGauge_; + /** + * Pre-accept quorum gate and the publish lag: the trusted-validation tally + * against the quorum it must reach, the time to the first fully-validated + * ledger, and how far publishing trails validation. + */ + opentelemetry::nostd::shared_ptr + ledgerQuorumPublishGauge_; + /** + * Build version, carried as a label with a constant value of 1. + */ + opentelemetry::nostd::shared_ptr buildInfoGauge_; + /** + * Complete ledger range start/end pairs. + */ + opentelemetry::nostd::shared_ptr + completeLedgersGauge_; + /** + * Database sizes and the historical fetch rate. + */ + opentelemetry::nostd::shared_ptr dbMetricsGauge_; + + // --- External dashboard parity instruments --- + /** + * Validator health: amendment blocked, UNL blocked, quorum, UNL expiry. + */ + opentelemetry::nostd::shared_ptr + validatorHealthGauge_; + /** + * Peer network quality: P90 latency, diverged peer count, version spread + * and the upgrade recommendation derived from it. + */ + opentelemetry::nostd::shared_ptr + peerQualityGauge_; + /** + * Transaction reduce-relay efficiency: selected against suppressed peers, + * feature-disabled peers, missing-tx frequency. + */ + opentelemetry::nostd::shared_ptr + reduceRelayGauge_; + /** + * Ledger economy: base fee, reserves, ledger age and transaction rate. + */ + opentelemetry::nostd::shared_ptr + ledgerEconomyGauge_; + /** + * Node state tracking: operating mode as a number, and time in that mode. + */ + opentelemetry::nostd::shared_ptr + stateTrackingGauge_; + /** + * Storage detail: the cumulative payload bytes handed to the NodeStore. + * Logical bytes stored, not on-disk file size. + */ + opentelemetry::nostd::shared_ptr + storageDetailGauge_; + /** + * Validation agreement percentages and counts over the 1h, 24h and 7d + * windows kept by ValidationTracker. + */ + opentelemetry::nostd::shared_ptr + validationAgreementGauge_; + /** + * ObservableCounter: jq_trans_overflow_total — observed from + * Overlay::getJqTransOverflow() (cumulative overflow tally owned by the + * overlay). + */ + opentelemetry::nostd::shared_ptr + jqTransOverflowObservable_; + /** + * ObservableCounter: validation_agreements_total — observed from + * ValidationTracker::totalAgreementsEver() (monotonic gross lifetime + * tally, initial-classification semantics). + */ + opentelemetry::nostd::shared_ptr + validationAgreementsObservable_; + /** + * ObservableCounter: validation_missed_total — observed from + * ValidationTracker::totalMissedEver() (monotonic gross lifetime tally, + * initial-classification semantics). + */ + opentelemetry::nostd::shared_ptr + validationMissedObservable_; + + /** + * Create and arm every instrument, one helper per metric domain so that + * each helper stays well under the 80-line-per-function limit. + * + * Called only from startAsyncGauges(), which owns the enable, arm-once, + * pipeline and service-readiness guards. + */ + void + registerAsyncGauges(); + + // Per-domain registration helpers. Each creates its instrument -- an + // ObservableGauge, or an ObservableCounter where the underlying value is + // cumulative -- and attaches a single callback that reads current values + // from the ServiceRegistry. The callbacks run on the OTel + // PeriodicExportingMetricReader background thread (~10 s tick). + void + registerJqTransOverflowCounter(); // gap-fill: overlay overflow total + void + registerCacheHitRateGauge(); + /** + * Observe the two TaggedCache lock-hold peaks onto the cache_metrics + * gauge. Split out to keep registerCacheHitRateGauge's callback under + * the 80-line limit. Static because it touches neither instance state + * nor telemetry members — it reads through the passed app reference. + * + * @param result Observer result the two peaks are published onto. + * @param app Services holding the caches whose peaks are read. + */ + static void + observeCacheLockHoldPeaks(opentelemetry::metrics::ObserverResult& result, ServiceRegistry& app); + void + registerTxqGauge(); + void + registerObjectCountGauge(); + void + registerLoadFactorGauge(); + void + registerNodeStoreGauge(); + void + registerRotationStateGauge(); // Sync diagnostics: online_delete rotation + void + registerServerInfoGauge(); + void + registerBuildInfoGauge(); + void + registerCompleteLedgersGauge(); + void + registerDbMetricsGauge(); + void + registerValidatorHealthGauge(); + void + registerPeerQualityGauge(); + void + registerReduceRelayGauge(); // Reduce-relay efficiency + void + registerLedgerEconomyGauge(); + void + registerStateTrackingGauge(); + void + registerStorageDetailGauge(); + void + registerValidationAgreementGauge(); + void + registerValidationTotalsCounters(); // gap-fill: lifetime agree/miss _total + + /** + * Register the `unl_quorum` gauge. + * + * Observes two series under the `metric` attribute: + * `trusted_keys` (ValidatorList::trustedKeyCount()) and `quorum` + * (ValidatorList::quorum()). Both are cheap accessors — one shared + * lock and one atomic load. + * + * `trusted_keys < quorum` means the node can never fully validate a + * ledger, so it will sit in `syncing` until the UNL is fixed. That + * makes this the first place to look when a node never leaves + * `syncing`. + * + * @note Pulled on the OTel reader thread (~10 s tick); does no work + * on any hot path. + */ + void + registerUnlQuorumGauge(); // sync diagnostics: UNL vs quorum + + /** + * Register the `clock_close_offset_seconds` gauge. + * + * Observes one series, `offset`, from + * TimeKeeper::closeOffset(): the seconds this node's notion of + * network close time is displaced from its own wall clock. + * + * The value MAY BE NEGATIVE, meaning the local clock runs ahead of + * the network. Whole-second resolution is all the signal carries, + * since that is the unit TimeKeeper stores. + * + * @note `server_info` only reports this field once |offset| >= 60 s + * (NetworkOPs), so this gauge is the first continuous export of it. + * Pulled on the OTel reader thread (~10 s tick); one atomic load. + */ + void + registerClockSkewGauge(); // sync diagnostics: close-time offset + + /** + * Register the `sync_state` gauge. + * + * One instrument fanning out four series under the `metric` attribute, + * each answering a different "why is this node not FULL yet?" question + * that is otherwise visible only in a log line or in server_info JSON: + * + * `initial_full_duration_us` — microseconds from process start to the + * first FULL transition (NetworkOPs::getInitialSyncDurationUs()). + * Stays 0 until FULL is reached, so a flat 0 IS the "never synced" + * signal; once set it never changes again. + * `network_ledger_gate` — 1 while the node is still waiting to see a + * full network ledger (NetworkOPs::isNeedNetworkLedger()), else 0. A + * persistent 1 blocks transaction submission and FULL. + * `server_stall_seconds` — current main-loop stall duration + * (LoadManager::getCurrentStallSeconds()), 0 when healthy. + * `ledgers_behind` — network tip minus our validated sequence + * (NetworkOPs::getLedgersBehindNetwork()). + * + * The monotonic stall-episode count is a separate instrument + * (`server_stall_events_total`) because a counter and a gauge cannot share + * one instrument: Prometheus would otherwise see a cumulative total under + * last-value aggregation and `rate()` would be meaningless. + * + * @note Pulled on the OTel reader thread (~10 s tick), never on a hot + * path. Three of the four reads are a lock or atomic load; `ledgers_behind` + * additionally walks the connected-peer list, reading each peer's already + * cached ledger range — bounded by peer count and issuing no network I/O. + */ + void + registerSyncStateGauge(); // sync diagnostics: gate, stall, ledgers behind + + /** + * Register the `server_stall_events_total` observable counter. + * + * Observes LoadManager::getStallEventCount(): how many distinct stall + * episodes the monitor thread has reported since process start. Separate + * from `sync_state` because it is cumulative and monotonic, so it needs + * counter (not last-value) aggregation for `rate()` to mean anything. + * + * Read together with `sync_state{metric="server_stall_seconds"}`: a rising + * event count means repeated fresh stalls, while a flat count with a large + * stall-seconds value means one long unresolved stall. + * + * @note Pulled on the OTel reader thread (~10 s tick); one atomic load. + */ + void + registerStallEventsCounter(); // sync diagnostics: stall episode count + + /** + * Register the `sync_acquire` gauge. + * + * One instrument fanning out four series under the `metric` attribute, all + * from a single InboundLedgers::acquireProgress() snapshot: + * + * `missing_state_nodes_max` — largest outstanding account-state node count + * of any in-flight acquire. THE headline stuck-sync signal: flat and + * non-zero across ticks means the acquire will never finish, shrinking + * means it is slow but alive. + * `missing_tx_nodes_max` — the same for the transaction tree. + * `received_data_depth` — peer packets stashed across all acquires waiting + * to be applied. Deep means processing, not peer supply, is the limit. + * `in_flight` — how many acquires are running, so the three values above + * can be read in context: all zero with `in_flight` zero is idle, not + * healthy. + * + * Deliberately aggregated rather than per-ledger. A `ledger_seq` label would + * mint a new time series for every ledger the node ever acquires, which is + * unbounded cardinality; the max/sum keeps the "is it stuck?" answer while + * the per-ledger identity stays on the `ledger.acquire` span, where + * high-cardinality identity belongs. + * + * @note Pulled on the OTel reader thread (~10 s tick), never on a hot path. + * The snapshot takes the acquire-collection lock only to copy shared_ptrs, + * then reads relaxed atomics; the emit sites that feed those atomics all sit + * outside the per-tree-node loops. + */ + void + registerSyncAcquireGauge(); // sync diagnostics: acquire progress + + /** + * Register the `shamap_cache_hit_rate` gauge. + * + * Observes one series, `treenode`, from TreeNodeCache::getHitRate(): the + * percentage of SHAMap tree-node lookups served from memory instead of the + * node store. During a fresh sync a low rate means the node re-reads the + * same subtrees from disk, so sync is disk-bound rather than peer-bound. + * + * Distinct from the `NuDB Cache Hit Ratio` panel on the ledger-data-sync + * dashboard: that one is derived from `nodestore_state` and measures the + * node-store layer (`node_reads_hit / node_reads_total`). This gauge + * measures the in-memory tree-node cache that sits ABOVE it, so a request + * missing here is what produces a node-store read there. + * + * The full-below cache is deliberately NOT reported. It is a KeyCache, whose + * only lookup path is TaggedCache::touchIfExists(), and that method + * increments `stats_.hits`/`stats_.misses` while `getHitRate()` reads the + * separate `hits_`/`misses_` members. Its hit rate is therefore hard-wired + * to 0 regardless of behaviour, so exporting it would ship a permanently + * empty panel; fixing that accounting belongs in a libxrpl change of its own. + * + * @note Pulled on the OTel reader thread (~10 s tick). Takes the cache's + * mutex for two integer reads and a divide; no hot-path cost. + */ + void + registerCacheHitRateDetailGauge(); // sync diagnostics: treenode cache + + /** + * Register the `jobq_saturation` gauge. + * + * Three series under the `metric` attribute, from one + * JobQueue::getWorkerSaturation() reading: + * + * `running_tasks` — worker threads currently executing a job. + * `worker_threads` — threads the pool is configured to run, the + * denominator that makes `running_tasks` legible. Exported rather + * than hardcoded in the dashboard because it is derived at startup + * from `[workers]`, node size and hardware concurrency. + * `total_waiting` — jobs queued across all types. + * + * The reason this is separate from the per-job-type gauges JobQueue + * itself publishes (`jobq__waiting` / `_running` / `_deferred`): + * when the pool itself is exhausted, every subsystem waiting behind it + * looks independently slow, and each per-type panel invites the wrong + * conclusion. A `running_tasks / worker_threads` ratio at 1.0 with a + * non-zero `total_waiting` attributes the whole slowdown to pool + * exhaustion once. Those per-type gauges carry no capacity term at all, + * so no reading there can say whether the pool is the cause. + * + * @note Pulled on the OTel reader thread (~10 s tick). One atomic load, + * one plain int read, and one pass over the per-type counters under the + * JobQueue mutex. + */ + void + registerJobQueueSaturationGauge(); // sync diagnostics: pool saturation + + /** + * Register the `peer_ledger_supply` gauge. + * + * Five series under the `metric` attribute, from one + * Overlay::getPeerLedgerSupply() pass over the active peers: + * + * `peers_reporting` — peers that have advertised a ledger range at all. + * The denominator that makes the rest readable. + * `peers_serving_validated` — peers whose range covers this node's + * validated sequence. + * `peers_serving_next` — **the signal this gauge exists for.** Peers + * whose range covers validated + 1, the next ledger this node must + * acquire. Zero here with a non-zero `peers_reporting` means no + * connected peer holds what this node needs, so no amount of waiting + * will finish the sync; the peer set has to change. + * `supply_min_seq` / `supply_max_seq` — the sequence window the peer set + * covers, so an operator can see whether the node is asking for + * history nobody kept or for a tip nobody has reached. + * + * Each peer already caches the range it advertises in mtSTATUS_CHANGE + * (`PeerImp::minLedger_` / `maxLedger_`, read via `Peer::ledgerRange()`), + * but those ranges were never compared against each other, so "no peer has + * what I need" was indistinguishable from "my peers are slow" — the two + * faults with completely different fixes. + * + * Distinct from what already exists. `server_info{metric="peers"}` is a + * bare connection count with no notion of what those peers hold. + * `sync_state{metric="ledgers_behind"}` uses the same per-peer maxima but + * collapses them to a single distance-to-tip number, which cannot say how + * many peers can serve that distance or whether the range has a hole. + * `peer_quality{metric="peers_insane_count"}` counts peers on a different + * chain, which is a correctness signal, not an availability one. + * + * Peers advertising [0, 0] have not reported yet and are excluded from + * every field, so they cannot make a healthy peer set appear to serve from + * genesis. When nothing has reported, both window fields read 0, which is + * why `peers_reporting` must be read alongside them. + * + * @note Pulled on the OTel reader thread (~10 s tick), never on a message + * path. O(peers): `getActivePeers()` copies the peer list under the overlay + * lock and releases it, then each peer's cached range is read under that + * peer's own short-lived lock. + */ + void + registerPeerLedgerSupplyGauge(); // sync diagnostics: peer range coverage + + /** + * Register the `peerfinder_slot_census` gauge. + * + * Nine series under the `metric` attribute, from one + * Overlay::getSlotCensus() snapshot: `out_active`, `out_max`, `in_active`, + * `in_max`, `connecting`, `fixed_configured`, `fixed_active`, `bootcache` + * and `livecache`. + * + * All nine are already computed inside PeerFinder (`Counts`, `Bootcache`, + * `Livecache`, the fixed-peer map) and only two of them are exported + * today, as the legacy beast::insight gauges + * `peer_finder_active_inbound_peers` and + * `peer_finder_active_outbound_peers`. Those two carry no capacity, + * attempt or cache term, which leaves the three most common bootstrap + * failures invisible: + * + * - `connecting` non-zero while `out_active` stays below `out_max` — + * dials are being started and never completing. Without the attempt + * count this looks the same as a node that is not dialling at all. + * - `bootcache` at 0 — no seed addresses to dial in the first place. + * - `fixed_active` below `fixed_configured` — a peer named in the + * configuration is unreachable. + * + * The nine fields come from a single acquire of the PeerFinder lock, so + * they are mutually consistent and share one label set. The two legacy + * gauges are read at unrelated instants and cannot be joined with each + * other, let alone with a capacity term. + * + * @note Pulled on the OTel reader thread (~10 s tick). One lock acquire, + * then integer and container-size reads. + */ + void + registerSlotCensusGauge(); // sync diagnostics: peerfinder slot census + + /** + * Register the `amendment_block` gauge. + * + * Two series under the `metric` attribute: + * + * `warned` — 1 once an unsupported amendment has reached majority, from + * NetworkOPs::isAmendmentWarned(). + * `seconds_to_block` — **the leading indicator.** Seconds until that + * amendment activates, derived from + * `AmendmentTable::firstUnsupportedExpected()` against the network + * close time. `-1` when nothing is pending, matching the sentinel + * `validator_health{metric="unl_expiry_days"}` already uses, so the + * healthy state is a distinct value rather than a missing series. + * Clamped at 0 rather than going negative, because past-due means the + * block is imminent, not overdue by some amount. + * + * Amendment-blocked is a terminal sync blocker: the node stops validating + * and never resumes without a software upgrade. The existing + * `validator_health{metric="amendment_blocked"}` reports that state after + * it has happened, when nothing can be done about it. This gauge is the + * window before it, which is the only actionable part. + * + * The blocking amendment's identity is deliberately NOT a label. The + * network can vote on an arbitrary 256-bit amendment id — the set is not + * drawn from this build's known features — so an id label would be + * unbounded cardinality and would mint a permanent new series per + * amendment. The id is already logged by + * `AmendmentTableImpl::doValidatedLedger` ("Unsupported amendment + * reached majority at ..."), so it is available through logs, correlated + * to this series by node and time. + * + * @note Pulled on the OTel reader thread (~10 s tick). One mutex acquire + * inside the amendment table plus one clock read. + * @note The subtraction is done in `std::int64_t`, not in NetClock's + * unsigned representation, so a past-due activation cannot wrap to a huge + * positive count. + */ + void + registerAmendmentBlockGauge(); // sync diagnostics: amendment countdown + + /** + * Register the `ledger_quorum_publish` gauge. + * + * Four series under the `metric` attribute, read from LedgerMaster: + * + * `trusted_validation_tally` — trusted validations counted at the last + * pre-accept gate in `LedgerMaster::checkAccept`. + * `quorum_target` — validations that gate required. **The pair is the + * signal.** The tally alone cannot separate a node accumulating + * validations toward quorum (slow, will finish) from one whose tally + * plateaus below the target (stuck, never will); with the target + * beside it, the two shapes are unmistakable. + * `time_to_first_validated_us` — how long the node took to get its + * first ledger through that gate. One-shot, like the + * `sync_state{initial_full_duration_us}` milestone: a value means it + * happened and this is how long it took, 0 means it never has. + * `publish_lag` — validated sequence minus published sequence. Non-zero + * and growing means validation is fine and the publish pipeline is + * behind, which no other signal distinguishes. + * + * All four are grouped under one instrument because they answer one + * question in sequence — did enough validations arrive, did the gate pass, + * how long did that take, and did the result reach clients — so an + * operator reads them from a single consistent poll. + * + * Distinct from what already exists. `unl_quorum{quorum}` is the quorum + * the validator list *configures*, a static property of the trusted set; + * `quorum_target` is what an actual gate evaluation *required*, and the + * tally beside it is the live count that must reach it — neither existed + * anywhere before. `server_info{validated_ledger_seq}` publishes the + * validated sequence but nothing published the pubLedgerSeq_ counterpart, + * so the lag between them was not derivable at all. + * + * @note `quorum_target` reports int64 max when the validator list has + * switched quorum off (`ValidatorList::quorum()` returns SIZE_MAX). The + * clamp lives in `LedgerMaster::checkAccept`, so the wrap to -1 that would + * invert a tally-versus-target panel cannot happen here. + * @note Pulled on the OTel reader thread (~10 s tick). Five relaxed atomic + * loads through lock-free LedgerMaster accessors: no lock is taken, which + * is what keeps an OTel callback from ever contending with, or inverting + * lock order against, the LedgerMaster mutex held by the emit path. + */ + void + registerLedgerQuorumPublishGauge(); // sync diagnostics: quorum + publish +#endif // XRPL_ENABLE_TELEMETRY +}; + +} // namespace xrpl::telemetry diff --git a/src/xrpld/telemetry/MetricsRegistry.h b/src/xrpld/telemetry/MetricsRegistry.h deleted file mode 100644 index fb1641ca06..0000000000 --- a/src/xrpld/telemetry/MetricsRegistry.h +++ /dev/null @@ -1,1750 +0,0 @@ -#pragma once - -/** - * Central OTel Metrics Registry for xrpld. - * - * Owns all OpenTelemetry metric instruments (counters, histograms, - * observable gauges) that are NOT already covered by the beast::insight - * StatsD pipeline. The instruments are created once at startup and polled - * by the OTel PeriodicExportingMetricReader at a configurable interval - * (default 10 s). - * - * When XRPL_ENABLE_TELEMETRY is **not** defined, this class compiles to a - * lightweight no-op: every public method is an empty inline. - * - * Dependency / ownership diagram (ASCII): - * - * Application - * | - * +-- MetricsRegistry (unique_ptr, created in setup(), started/stopped with telemetry) - * | - * +-- OTel MeterProvider (owns reader + exporter) - * | | - * | +-- PeriodicExportingMetricReader - * | +-- OtlpHttpMetricExporter - * | - * +-- Counters / Histograms (synchronous instruments) - * | +-- rpc_method_started_total - * | +-- rpc_method_finished_total - * | +-- rpc_method_errored_total - * | +-- rpc_method_us (Histogram) - * | +-- job_queued_total{job_type,handler} - * | +-- job_started_total{job_type,handler} - * | +-- job_finished_total{job_type,handler} - * | +-- job_queued_us{job_type,handler} (Histogram) - * | +-- job_running_us{job_type,handler} (Histogram) - * | +-- ledgers_closed_total - * | +-- validations_sent_total - * | +-- validations_checked_total - * | +-- ledger_history_mismatch_total{reason} - * | +-- txq_expired_total - * | +-- txq_dropped_total{reason} - * | - * +-- ValidationTracker (validation agreement tracker) - * | - * +-- Observable Gauges (async callbacks, polled by reader) - * +-- Cache hit rates (SLE, ledger, AL) - * +-- TreeNode / FullBelow sizes - * +-- TxQ metrics - * +-- CountedObject counts - * +-- Load factor breakdown - * +-- NodeStore I/O gauges (totals, derived means, NuDB write queue, - * ledger-acquisition stall counters) - * +-- Server info (state, uptime, peers, consensus) - * +-- Build info (version label) - * +-- Complete ledger ranges (start/end pairs) - * +-- DB metrics (storage KB, fetch rate) - * +-- Validator health (amend blocked, UNL, quorum) - * +-- Peer quality (P90 latency, version spread) - * +-- Reduce-relay efficiency (selected/suppressed peers) - * +-- Ledger economy (fees, reserves, age) - * +-- State tracking (mode value, time in state) - * +-- Storage detail (NuDB sizes) - * +-- Validation agreement (1h/24h pct, counts) - * +-- UNL quorum (trusted keys vs required quorum) - * +-- Clock close offset (local clock skew) - * +-- Sync state (time to first FULL, network-ledger gate, - * | server stall seconds, ledgers behind network) - * +-- JobQueue saturation (running tasks vs worker threads vs backlog) - * +-- Peer ledger supply (how many peers can serve the needed sequence) - * +-- PeerFinder slot census (slots, attempts, fixed peers, address caches) - * +-- Amendment block (warned flag + seconds until the node stops validating) - * +-- Ledger quorum + publish (validation tally vs quorum target, - * | time to first validated, publish lag) - * +-- jq_trans_overflow_total (observed from Overlay) - * +-- server_stall_events_total (observed from LoadManager) - * - * Control-flow for async gauges: - * - * PeriodicExportingMetricReader (background thread, 10 s tick) - * | - * v - * OTel SDK invokes registered ObservableGauge callbacks - * | - * v - * Each callback reads current value from Application services - * (e.g. app.getTxQ().getMetrics(), app.getFeeTrack().getLoadFactor()) - * | - * v - * Result set is exported via OTLP/HTTP to the collector - * - * Control-flow for synchronous instruments: - * - * PerfLogImp::rpcStart/rpcEnd/jobQueue/jobStart/jobFinish - * | - * v - * MetricsRegistry::recordRpc*(method, ...) / recordJob*(type, ...) - * | - * v - * OTel Counter::Add() or Histogram::Record() - * | - * v - * Periodically flushed by the MetricReader - * - * Example usage: - * - * @code - * // In ApplicationImp's member-init list, right after telemetry_ and before - * // every subsystem. The constructor builds the pipeline and every - * // synchronous instrument, so no producer can exist before they do. The - * // endpoint, the TLS settings and the resource identity come from - * // [telemetry] and [network_id], read by Application.cpp rather than - * // through Telemetry::Setup. - * metricsRegistry_(std::make_unique( - * telemetry_->isEnabled(), *this, journal, options)) - * - * // Later, in setup(), once overlay_ exists (the last of the services the - * // callbacks read). Phase 2 registers the observable instruments: - * metricsRegistry_->startAsyncGauges(); - * - * // In PerfLogImp::rpcStart(): - * if (auto* mr = app_.getMetricsRegistry()) - * mr->recordRpcStarted("server_info"); - * - * // In PerfLogImp::rpcEnd(): - * if (auto* mr = app_.getMetricsRegistry()) - * { - * mr->recordRpcFinished("server_info", durationUs); - * // or: mr->recordRpcErrored("server_info", durationUs); - * } - * - * // In PerfLogImp::jobQueue(). The second argument is the addJob name; - * // it is sanitised internally into the bounded `handler` label. - * if (auto* mr = app_.getMetricsRegistry()) - * mr->recordJobQueued("ledgerData", "ProcessLData"); - * - * // Shutdown, before any service the callbacks read is stopped. Idempotent, - * // so run() and ~ApplicationImp both call it: - * metricsRegistry_->stop(); - * @endcode - * - * Caveats: - * - The MetricsRegistry must be created AFTER the Telemetry object because - * it reads isEnabled() to decide whether to initialize the OTel SDK, and - * BEFORE every subsystem that records a metric. Declaration order in - * ApplicationImp is the guarantee; keep the member where it is. - * - Observable gauge callbacks capture a reference to the Application; the - * Application must outlive the MetricsRegistry (guaranteed because - * MetricsRegistry is stopped before Application teardown). - * - If a new CountedObject type is added, it will NOT appear automatically - * in the object_count gauge; the callback iterates a fixed list. - * - Adding a new synchronous instrument requires updating both the header - * and the .cpp, then calling the new record*() method from the - * instrumentation site. - */ - -#ifdef XRPL_ENABLE_TELEMETRY -// The tracker is held and exposed only in this configuration, where the gauge -// callbacks that drain it exist. -#include -#endif - -#include - -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#ifdef XRPL_ENABLE_TELEMETRY -#include -#include -#include -#include -#include -#include - -// These three serve only the telemetry-only members below, so they are guarded -// like their uses: std::atomic by callbacksDetached_, std::function by the -// ObserveFn sink, std::shared_ptr by provider_. -#include -#include -#include -#endif - -namespace xrpl { - -class ServiceRegistry; - -// Defined in src/xrpld/app/ledger/AcquireStats.h. Forward-declared because -// only the gauge helpers in the .cpp touch it, and pulling an xrpld/app -// header in here would widen the dependencies of every file that includes -// this one. -class AcquireStats; - -namespace node_store { -class Database; -} // namespace node_store - -namespace telemetry { - -/** - * Central OpenTelemetry metric registry. - * - * Owns all OTel instruments (counters, histograms, observable gauges) - * that are not covered by the beast::insight StatsD pipeline. See the - * file-level header comment above for the full dependency diagram, - * gauge domain list, and usage examples. - * - * Class / collaborator diagram (ASCII): - * - * +-----------------+ +-------------------+ - * | Application |------->| MetricsRegistry | - * +-----------------+ +-------------------+ - * | | | - * creates/owns v v v - * +-----------+ +---------+ +-------------------+ - * | Meter | | Counter | | ValidationTracker | - * | Provider | | /Hist. | | (rolling windows) | - * +-----------+ +---------+ +-------------------+ - * | - * v - * Periodic reader thread (~10 s) - * -> ObservableGauge callbacks - * -> OTLP/HTTP export - * - * @note Thread safety: - * - The recordRpc, recordJob, and increment methods are invoked - * from xrpld hot paths. OTel Counter::Add() and - * Histogram::Record() are documented thread-safe, and - * null-guard checks protect uninitialized instruments. - * - ObservableGauge callbacks run on the OTel SDK background - * reader thread (~10 s tick), concurrently with writers. - * Each callback reads only lock-protected or atomic state - * from Application services and wraps the body in a - * catch-all try block so a transient failure never crashes - * the reader thread. - * - ValidationTracker protects its rolling windows internally. - * - The constructor, startAsyncGauges() and stop() are NOT thread-safe - * with each other and must all be called, in that order, from - * the single Application lifecycle thread. - * - * @note Lifetime, in three phases (see Phase): - * - Ready: the constructor built the pipeline and the synchronous - * instruments. Runs in ApplicationImp's member-init list, so it precedes - * every subsystem that could record. - * - GaugesArmed: startAsyncGauges() registered the observable callbacks. - * Runs once overlay_ exists, the last service those callbacks read. - * - Stopped: stop() joined the reader thread. Runs before any observed - * service stops, from run() and again from ~ApplicationImp for the - * paths that never reach run(). - * - * @note Extending: - * - Adding a new CountedObject type is auto-picked up by the - * object_count gauge via iteration. - * - Adding a new SYNCHRONOUS instrument (counter/histogram): prefer the - * XRPL_METRIC_* call-site macros in MetricMacros.h -- no header/cpp - * edit needed. Fall back to a dedicated member + init line + record - * method (the pattern below) only when the metric needs to be read - * back by other code (e.g. ValidationTracker-style accumulation) or - * needs a custom histogram bucket View (see the histogram note in - * MetricMacros.h). - * - Adding a new OBSERVABLE gauge still requires eager central - * registration -- pull-model instruments cannot be lazily created. - */ -/** - * Run time at which a finished job counts as a stall, in microseconds. - * Equal to LoadMonitor's 1 s warn threshold (LoadMonitor.cpp - * addLoadSample) so this counter and the "Job: ... run:" log line - * describe the same event. - */ -inline constexpr std::int64_t kJobStallThresholdUs = 1'000'000; - -class MetricsRegistry -{ -public: - /** - * Everything the constructor needs from config: where to export, how to - * secure the connection, and the process identity stamped on the OTel - * resource. - * - * The values come from the `[telemetry]` section plus `[network_id]`, read - * by `makeMetricsRegistryOptions()` in `Application.cpp`. They must match - * what `makeTelemetrySetup()` gives the trace pipeline, or one node reports - * two identities and a dashboard filter shows half its series. - * - * A struct rather than ten positional parameters: seven of them are - * strings, so a swapped pair would compile and silently stamp the wrong - * label. Designated initializers name every value at the call site. - * - * @code - * MetricsRegistry::Options opts{ - * .endpoint = "http://localhost:4318/v1/metrics", - * .serviceName = "xrpld", - * .serviceVersion = build_info::getVersionString(), - * .serviceInstanceId = nodePublicKey, - * .nodeId = nodePublicKey, - * .networkId = 2}; - * MetricsRegistry registry(enabled, app, journal, opts); - * - * // Edge case: mutual TLS to a collector that requires it. - * opts.useTls = true; - * opts.tlsCaCertPath = "/etc/xrpld/otel-ca.pem"; - * opts.tlsClientCertPath = "/etc/xrpld/node.pem"; - * opts.tlsClientKeyPath = "/etc/xrpld/node.key"; - * MetricsRegistry secure(enabled, app, journal, opts); - * @endcode - * - * @note Plain aggregate, no invariants enforced. `networkType` is not a - * field: it is derived from @ref networkId inside the constructor so - * the two can never disagree. - */ - struct Options - { - /** - * OTLP/HTTP endpoint URL for metric export, from - * `[telemetry] metrics_endpoint`. - */ - std::string endpoint; - - /** - * service.name resource attribute, from `[telemetry] service_name`. - * Stamped unconditionally, so an empty value here yields an empty - * label rather than the SDK's `unknown_service` default. The caller - * seeds it with `systemName()`. - */ - std::string serviceName; - - /** - * service.version resource attribute — the build's version string. - * Left off the resource when empty. - */ - std::string serviceVersion; - - /** - * service.instance.id resource attribute, from - * `[telemetry] service_instance_id` or the node's base58 public key. - * Left off the resource when empty. - */ - std::string serviceInstanceId; - - /** - * xrpl.node.id resource attribute — the node's base58 public key, - * which config cannot override. Left off the resource when empty. - */ - std::string nodeId; - - /** - * Network identifier from `[network_id]`. Stamped as xrpl.network.id, - * and mapped to the xrpl.network.type label by `networkTypeFromId()`. - */ - std::uint32_t networkId{0}; - - /** - * Whether the exporter connects to the collector over TLS. The three - * paths below apply only when this is true. - */ - bool useTls{false}; - - /** - * CA bundle used to verify the collector. Empty selects the system - * CA store. - */ - std::string tlsCaCertPath; - - /** - * This node's client certificate, presented for mutual TLS. Empty - * means one-way TLS. - */ - std::string tlsClientCertPath; - - /** - * Private key for @ref tlsClientCertPath. - */ - std::string tlsClientKeyPath; - }; - - /** - * Construct the registry and, when enabled, build the whole metrics - * pipeline: OTLP exporter, periodic reader, MeterProvider and every - * SYNCHRONOUS instrument (counters and histograms). - * - * Doing this in the constructor is what fixes the init order. The - * Application declares its registry before every subsystem, so no - * producer can exist before the instruments do. A failure to build the - * pipeline is logged and leaves the registry a no-op; it never stops the - * node. - * - * @note Invariant for future changes: the constructor may create only - * instruments with NO Application-reading callback. Push-model - * counters and histograms qualify; app code records into them - * when it is ready. Any observable instrument whose callback - * reads an Application service belongs in `startAsyncGauges()`, - * because registering it here arms the reader thread to invoke - * that callback against a half-built Application. This applies - * to observable COUNTERS as well as gauges. - * - * @param enabled False makes every method a no-op (telemetry disabled). - * @param app Services the observable-gauge callbacks read. - * @param journal Log output. - * @param options Endpoint, TLS settings and resource identity, all read - * from config by the caller. See @ref Options. - */ - MetricsRegistry( - bool enabled, - ServiceRegistry& app, - beast::Journal journal, - Options const& options); - - /** - * Stops the pipeline if run() or ~ApplicationImp did not already. - */ - ~MetricsRegistry(); - - /** - * Non-copyable, non-movable. - */ - MetricsRegistry(MetricsRegistry const&) = delete; - MetricsRegistry& - operator=(MetricsRegistry const&) = delete; - - /** - * Register the pull-model observable instruments — the second startup - * phase. Mostly ObservableGauges, plus the ObservableCounters whose - * source value is already cumulative. - * - * A separate entry point from the constructor because the two halves have - * different prerequisites. The constructor needs only config strings; - * these callbacks read live Application services, so this half must run - * later. Registering an observable also arms the reader thread to invoke - * its callback on the next tick, which is why the separation is about - * ordering and not just tidiness. - * - * Calling it twice, or after stop(), logs a warning and does nothing. - * - * @pre Every service the callbacks read is constructed. The full set, - * from the `app.get*()` calls in the registration helpers, is: - * Overlay, OPs (NetworkOPs), LedgerMaster, OpenLedger, TxQ, - * NodeStore, NodeFamily, Validators, AcceptedLedgerCache, - * CachedSLEs, AcquireStats, TimeKeeper, RelationalDatabase, - * InboundLedgers and FeeTrack. - * Overlay is built last, so it fixes this call's position: - * `ServiceRegistry::getOverlay()` `XRPL_ASSERT`s that - * `overlay_` is non-null, and a reader-thread tick before the - * overlay exists aborts a Debug build. The callbacks' catch-all - * try block does not catch an assert. `getTxQ()` and - * `getRelationalDatabase()` assert likewise. - */ - void - startAsyncGauges(); - - /** - * Detach all ObservableGauge callbacks so they no-op on the next - * reader-thread tick. - * - * Must be called BEFORE any Application service that the callbacks - * read (nodeStore, overlay, networkOPs, ledgerMaster, etc.) is - * stopped. The flag is checked with acquire ordering at the top of - * every callback; together with the release store here it - * guarantees that once `detachCallbacks()` returns, no subsequent - * callback invocation will dereference an already-stopped service. - * - * Idempotent, and safe to call multiple times: the flag is one-way, - * only ever set to true, and nothing clears it. The actual - * SDK-level provider shutdown still happens in `stop()`. - * - * @note One-way means this is a shutdown-only call. Calling it before - * `startAsyncGauges()` does not "have no effect" — it - * permanently disarms every gauge the later call registers, so - * the instruments exist but never observe a value. Only call it - * once the process is shutting down. - */ - void - detachCallbacks() noexcept; - - /** - * Flush pending metrics and shut down the pipeline. - * - * Stores `Phase::Stopped` first so `recording()` reads false on every - * later record call, then destroys the SDK provider. meter_ is not - * touched: record threads may still be running, and the gate is what - * keeps them off the dying pipeline. Idempotent. - * - * @pre `detachCallbacks()` should have been called earlier in the - * shutdown sequence; otherwise there is a narrow race between - * the final reader-thread tick and the destruction of - * Application services that the gauge callbacks read from. - */ - void - stop(); - - /** - * @return true if the registry is actively exporting metrics. - */ - [[nodiscard]] bool - isEnabled() const noexcept - { - return enabled_; - } - - /** - * @return true when a record call is safe to run. - * - * False when the registry is disabled, or after stop() has torn down the - * export pipeline. After stop() the SDK's SyncMetricStorage still holds a - * raw pointer to an AggregationConfig owned by a destroyed View, so a - * record with a first-seen attribute set would fire the factory lambda - * and deref that dangling pointer. Every XRPL_METRIC_* macro reads this - * once before touching an instrument. - * - * One acquire atomic load in the hot path. - */ - [[nodiscard]] bool - recording() const noexcept - { -#ifdef XRPL_ENABLE_TELEMETRY - return enabled_ && phase_.load(std::memory_order_acquire) != Phase::Stopped; -#else - return enabled_; -#endif - } - - // ----------------------------------------------------------------- - // Synchronous instrument recording (called from PerfLog hot paths) - // ----------------------------------------------------------------- - - /** - * Record an RPC method call start. - * @param method The RPC method name (e.g. "server_info"). - */ - void - recordRpcStarted(std::string_view method); - - /** - * Record an RPC method call completion. - * @param method The RPC method name. - * @param durationUs Execution time in microseconds. - */ - void - recordRpcFinished(std::string_view method, std::int64_t durationUs); - - /** - * Record an RPC method call error. - * @param method The RPC method name. - * @param durationUs Execution time in microseconds. - */ - void - recordRpcErrored(std::string_view method, std::int64_t durationUs); - - /** - * The `handler` label value used for any job name that fails the - * sanitiser's all-ASCII-letters rule. - * - * Public because both sanitiseHandler() and its unit tests must agree - * on the exact fallback token; a test asserting against its own copy - * of the string would not catch a change made here. - * - * Declared as std::string_view rather than the `constexpr char k[]` - * form used for instrument names in MetricsRegistry.cpp: this value is - * *returned* by sanitiseHandler(), whose return type is - * std::string_view, and is compared against std::string_view in tests. - * Matching the type avoids array-to-pointer decay and a needless - * strlen at each use. - */ - static constexpr std::string_view kHandlerOther{"other"}; - - /** - * Reduce a job name to a bounded-cardinality `handler` label value. - * - * A job type can have several producers — both `RcvGetLedger` and - * `RcvGetObjByHash` run as `JtLedgerReq` — so `job_type` alone cannot - * attribute a latency spike to one of them. The job name can, but it - * cannot be used raw: two names embed a ledger sequence number - * (`"Pub" + std::to_string(seq)` in LedgerPersistence.cpp and - * `"OB" + std::to_string(...)` in OrderBookDBImpl.cpp), which would - * mint a fresh Prometheus series for every ledger. - * - * The rule is therefore: keep the name only when it is non-empty and - * every character is an ASCII letter; otherwise return `"other"`. - * Both dynamic names always contain digits, so they always fold to - * `"other"`, while every all-letter name is a compile-time literal. - * The label domain is thus a function of the literals present in the - * source — 43 names plus `"other"` at the time of writing — and - * cannot grow at runtime. A name added later that does not satisfy - * the rule degrades to `"other"` rather than becoming unbounded, - * which is a stronger guarantee than an allowlist that would have to - * be maintained by hand. - * - * Defined inline so unit tests can call it without linking the rest - * of the registry: in a telemetry-enabled build MetricsRegistry.cpp - * is not compiled into the test binary, so an out-of-line definition - * would be unreachable from tests. Being inline also makes it usable - * regardless of XRPL_ENABLE_TELEMETRY. - * - * @param name The job name as passed to JobQueue::addJob. - * @return @p name when it is non-empty and all ASCII letters, else - * kHandlerOther. - * - * @note Pure and reentrant: holds no state, performs no I/O, and is - * safe to call concurrently from any thread. - * @note The letter test is an explicit ASCII range check rather than - * std::isalpha, which classifies by the current C locale. A - * locale-dependent test could admit non-ASCII bytes and so - * weaken the cardinality bound this function exists to provide. - * @note When the name is kept, the returned view aliases @p name, so - * it must not outlive the caller's buffer. The kHandlerOther case - * returns a view of a static constant and is always valid. - * - * Example: - * @code - * sanitiseHandler("RcvGetObjByHash"); // "RcvGetObjByHash" - * sanitiseHandler("Pub94512331"); // kHandlerOther (digits) - * sanitiseHandler(""); // kHandlerOther (empty) - * @endcode - */ - [[nodiscard]] static constexpr std::string_view - sanitiseHandler(std::string_view name) noexcept - { - auto const isAsciiLetter = [](char const c) { - return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z'); - }; - - if (name.empty() || !std::ranges::all_of(name, isAsciiLetter)) - return kHandlerOther; - - return name; - } - - /** - * Divide a cumulative total by its count, optionally scaled, reporting - * absence rather than zero when the count is zero. - * - * Every cumulative counter this registry publishes has a companion mean - * that is only defined once the counter has moved. Reporting such a mean - * as `0` is worse than not reporting it: `0` is a plausible reading, so a - * dashboard draws a flat line at the bottom of the axis and an operator - * concludes "reads are instant" when the truth is "nothing has been - * read". Returning std::nullopt makes the caller skip the observation, so - * the series has a genuine gap instead. - * - * @p scale exists because the gauge these feed is integral. A mean writer - * depth of 1.4 truncates to 1, which is indistinguishable from a healthy - * 1.0, so the caller scales by 100 and says so in the metric name. - * - * The arithmetic divides before scaling and scales the remainder - * separately, so a long-lived node cannot overflow the product. Should - * the result still exceed the gauge's range it saturates at - * INT64_MAX rather than wrapping, because a wrapped gauge reads as a - * sudden healthy-looking dip. - * - * Defined inline for the same reason as sanitiseHandler(): in a - * telemetry-enabled build MetricsRegistry.cpp is not compiled into the - * unit-test binary, so an out-of-line definition would be untestable. - * constexpr so the cases below are checked at compile time. - * - * @param total Cumulative numerator (e.g. summed microseconds). - * @param count Number of samples in @p total. - * @param scale Fixed-point multiplier applied to the quotient. Must be - * at least 1; 0 is meaningless and yields std::nullopt. - * @return The scaled mean, or std::nullopt when @p count is 0 (mean - * undefined) or @p scale is 0. - * - * @note Pure and reentrant: holds no state and performs no I/O. - * @note Truncates toward zero, like integer division. A mean of 9.9 us - * reads as 9 at @p scale 1 and as 990 at @p scale 100. - * - * Example: - * @code - * scaledMean(500, 4); // 125 -- mean microseconds - * scaledMean(7, 5, 100); // 140 -- mean 1.4, scaled by 100 - * scaledMean(500, 0); // nullopt -- no samples, so no mean - * @endcode - */ - [[nodiscard]] static constexpr std::optional - scaledMean(std::uint64_t total, std::uint64_t count, std::uint64_t scale = 1) noexcept - { - if (count == 0 || scale == 0) - return std::nullopt; - - constexpr auto kInt64Max = - static_cast(std::numeric_limits::max()); - - auto const whole = total / count; - if (whole > kInt64Max / scale) - return static_cast(kInt64Max); - - // Scale the remainder too, so `scale` recovers the fractional digits - // it exists for. Skipped when the product itself would overflow, at - // which point it is worth less than one part in 2^63 of the result. - auto const remainder = total % count; - std::uint64_t fraction = 0; - if (remainder <= std::numeric_limits::max() / scale) - fraction = remainder * scale / count; - - auto const scaled = whole * scale; - if (scaled > kInt64Max - fraction) - return static_cast(kInt64Max); - - return static_cast(scaled + fraction); - } - - /** - * Read one comma-separated segment of a complete-ledger range string. - * - * The producer is xrpl::to_string(RangeSet), documented in - * xrpl/basics/RangeSet.h. It renders an interval as `first-last`, and an - * interval whose first equals its last as a bare sequence number. A segment - * with no dash is therefore a range of one ledger, not a malformed one. - * - * Defined inline for the same reason as sanitiseHandler(): in a - * telemetry-enabled build MetricsRegistry.cpp is not compiled into the - * unit-test binary, so an out-of-line definition would be untestable. - * - * @param segment One segment, already split on ','. Leading or trailing - * whitespace is rejected, because the producer emits none. - * @return The inclusive first and last sequence of the range. The two are - * equal for a single-ledger range. std::nullopt when @p segment is not - * something this producer can emit. - * - * @note Pure and reentrant: holds no state, performs no I/O, and is safe to - * call concurrently from any thread. - * @note Reports malformed input instead of throwing, so one unreadable - * segment costs its own range and not every range after it. - * @note A reversed range such as "9-4" is returned as given. RangeSet - * cannot emit one. - * - * Example: - * @code - * parseLedgerRange("32570-50000"); // {32570, 50000} - * parseLedgerRange("5000"); // {5000, 5000} -- one ledger - * parseLedgerRange("5-"); // nullopt - * @endcode - */ - [[nodiscard]] static std::optional> - parseLedgerRange(std::string_view segment) noexcept - { - auto const parseSeq = [](std::string_view text) -> std::optional { - std::uint32_t value = 0; - auto const* const begin = text.data(); - auto const* const end = begin + text.size(); - auto const [ptr, ec] = std::from_chars(begin, end, value); - - // from_chars stops at the first character it cannot use, so the - // whole segment counts as read only when it consumed all of it. - if (ec != std::errc{} || ptr != end) - return std::nullopt; - - return value; - }; - - auto const dash = segment.find('-'); - if (dash == std::string_view::npos) - { - auto const only = parseSeq(segment); - if (!only) - return std::nullopt; - - return std::pair{*only, *only}; - } - - auto const first = parseSeq(segment.substr(0, dash)); - auto const last = parseSeq(segment.substr(dash + 1)); - if (!first || !last) - return std::nullopt; - - return std::pair{*first, *last}; - } - - /** - * Record a job enqueued event. - * @param jobType The job type name (e.g. "ledgerData"). - * @param jobName The addJob name, reduced to a bounded `handler` - * label by sanitiseHandler(). Distinguishes producers - * that share a job type. - */ - void - recordJobQueued(std::string_view jobType, std::string_view jobName); - - /** - * Record a job start event. - * @param jobType The job type name. - * @param jobName The addJob name; see recordJobQueued(). - * @param queuedDurUs Time the job spent waiting in the queue (us). - */ - void - recordJobStarted(std::string_view jobType, std::string_view jobName, std::int64_t queuedDurUs); - - /** - * Record a job finish event. - * @param jobType The job type name. - * @param jobName The addJob name; see recordJobQueued(). - * @param runningDurUs Execution time in microseconds. - */ - void - recordJobFinished( - std::string_view jobType, - std::string_view jobName, - std::int64_t runningDurUs); - - // ----------------------------------------------------------------- - // External dashboard parity counters - // ----------------------------------------------------------------- - - /** - * Increment the ledgers_closed_total counter. - * - * @note Currently has no callers: the ledgers_closed_total counter is - * incremented at its consensus call site via the XRPL_METRIC_COUNTER_INC - * macro (see MetricMacros.h). This method and its eagerly-created - * counter are retained as a fallback and are slated for removal in a - * separate cleanup once the macro path has proven out. - */ - void - incrementLedgersClosed(); - - /** - * Increment the validations_sent_total counter. - * Called from RCLConsensus::Adaptor::validate() when a validation - * is produced and broadcast. - */ - void - incrementValidationsSent(); - - /** - * Increment the validations_checked_total counter. - * Called from NetworkOPs::recvValidation() when a network validation - * is received and checked. - */ - void - incrementValidationsChecked(); - - /** - * Increment the ledger_history_mismatch_total counter for a reason. - * Called from LedgerHistory::handleMismatch() once the mismatch has - * been classified. The reason label turns fork diagnosis from a - * log-grep into a queryable time series. - * @param reason Classified mismatch cause (e.g. "prior_ledger", - * "close_time", "consensus_txset", "same_txset_diff_result", - * "unknown"). - */ - void - incrementLedgerHistoryMismatch(std::string_view reason); - - /** - * Increment the txq_expired_total counter. - * Called from TxQ::processClosedLedger() for each queued transaction - * removed because its LastLedgerSequence has passed — submitters who - * under-bid the escalating fee and were never included. - */ - void - incrementTxqExpired(); - - /** - * Increment the txq_dropped_total{reason} counter. - * Called from TxQ::apply() when a transaction is refused admission to - * the queue (e.g. the queue is full). Distinct from expiry (already - * queued) and from jq_trans_overflow (job queue, not TxQ). - * @param reason Admission-control rejection cause (e.g. "queue_full"). - */ - void - incrementTxqDropped(std::string_view reason); - -#ifdef XRPL_ENABLE_TELEMETRY - /** - * Access the validation agreement tracker. - * Used by consensus and ledger hooks to record our validations and - * network validations so the tracker can compute agreement percentages. - * - * Guarded, along with the tracker itself, because only the observable-gauge - * callbacks read it and those exist only in this configuration. Recording - * into it is not free: each call takes its lock and inserts an entry. - * @return Reference to the internal ValidationTracker instance. - */ - [[nodiscard]] ValidationTracker& - getValidationTracker() - { - return validationTracker_; - } - - /** - * Access the shared OTel Meter for call-site instrument creation. - * Used by the XRPL_METRIC_* macros (MetricMacros.h) so new synchronous - * counters/histograms can be declared at their call site instead of as - * MetricsRegistry members. - * - * Invariant: never empty while recording() is true. The constructor sets - * it to the real meter, or to a no-op meter when the pipeline failed to - * build, and never writes it again, so reads need no lock. After stop() - * the meter's SDK context is gone; the macros gate on recording() first, - * so no caller reaches it then. - * - * @return The shared Meter. - */ - [[nodiscard]] opentelemetry::nostd::shared_ptr - meter() const noexcept - { - return meter_; - } - - /** - * Sink handed to the nodestore_state gauge helpers below. - * - * Every value they publish multiplexes onto the single `nodestore_state` - * gauge through its `metric` label, so the helpers need no access to the - * OTel observer result -- just somewhere to put a name and a number. - */ - using ObserveFn = std::function; - - /** - * Observe the NodeStore I/O totals and the means derived from them. - * - * Publishes the four cumulative totals (`node_reads_total`, - * `node_writes`, `node_reads_duration_us`, `node_writes_duration_us`) - * unconditionally, plus `read_mean_us` and `write_mean_us` derived from - * them via scaledMean(). `write_mean_us` is the signal for the "a node - * with a large existing database syncs slower than a fresh one" symptom: - * back-fill is write-bound, so no read-side reading can show it. All - * three concrete store paths time themselves through - * Database::recordStoreDuration(), so the write mean is live on an - * ordinary node. - * - * Gauge rather than histogram, deliberately. A histogram would give true - * percentiles, but it costs one Record() per node object on the - * store/fetch path, and one ledger write walks thousands of SHAMap - * nodes. This reads the existing atomics once per ~10 s tick and adds - * nothing to the hot path. Consequence, stated plainly: p99 is NOT - * obtainable from this signal. A histogram added later would also need an - * explicit-bucket View registered via addMicrosecondHistogramView(), - * because the SDK's default buckets top out at 10,000. - * - * @param db NodeStore to read the counters from. - * @param observe Sink for one `metric`-labelled value. - * - * @note The totals are monotonic and never reset, so a panel wanting - * current rather than since-boot latency divides the two rates. That is - * why the counts and duration totals are exported beside the means. - * @note A mean is omitted when its count is 0, so a dashboard shows a gap - * rather than a plausible-looking 0 us. - */ - static void - observeNodeStoreTotals(node_store::Database& db, ObserveFn const& observe); - - /** - * Observe the backend write-path detail, when the backend measures it. - * - * Publishes nothing for a backend whose getWriteStats() is std::nullopt, - * which is every backend except NuDB. Absent labels let a reader tell - * "not measured" from "measured, and idle"; zeros would read as a - * perfectly idle write path. - * - * @param db NodeStore whose writable backend is sampled. - * @param observe Sink for one `metric`-labelled value. - */ - static void - observeWritePathDetail(node_store::Database const& db, ObserveFn const& observe); - - /** - * Observe the ledger-acquisition progress and stall counters. - * - * @param stats Process-wide acquisition counters. - * @param observe Sink for one `metric`-labelled value. - */ - static void - observeAcquireStats(AcquireStats const& stats, ObserveFn const& observe); - - /** - * Observe the read queue depth and the read thread-pool counts. - * - * These four have no accessor on Database, so its JSON counters object - * is still the only way to reach them. - * - * @param db NodeStore to read the JSON counters from. - * @param observe Sink for one `metric`-labelled value. - */ - static void - observeReadQueue(node_store::Database& db, ObserveFn const& observe); -#endif - -private: - /** - * Master enable flag; when false all methods are no-ops. - */ - bool const enabled_; - -#ifdef XRPL_ENABLE_TELEMETRY - /** - * Tracks validation agreement between this node and the network. - * - * Guarded because reconcile() -- which resolves and then prunes recorded - * events -- runs only from the observable-gauge callbacks. Recording - * without it accumulates one entry per validated ledger, so the tracker - * exists only where something drains it. - */ - ValidationTracker validationTracker_; - - /** - * Reference to Application services for gauge callbacks. - * Only needed when OTel is compiled in, since observable gauge - * callbacks live entirely inside the XRPL_ENABLE_TELEMETRY guard. - */ - ServiceRegistry& app_; - - /** - * Journal for logging. - */ - beast::Journal const journal_; - - /** - * Where the registry is in its life. Construction ends in `Ready`; - * startAsyncGauges() moves to `GaugesArmed`; stop() to `Stopped`. A call - * that does not fit the current phase logs a warning and does nothing. - * - * After `Stopped` the SDK pipeline is gone. recording() reads false, so - * no macro touches meter_ or a cached instrument. - */ - enum class Phase { Ready, GaugesArmed, Stopped }; - - /** - * Current phase. Written from the Application lifecycle thread with - * release ordering; read from record threads via `recording()` with - * acquire ordering, so no record starts once stop() has stored `Stopped`. - */ - std::atomic phase_{Phase::Ready}; - - /** - * Set by detachCallbacks() during shutdown so every ObservableGauge - * callback returns early before reading Application services that - * may already be stopped. Checked with memory_order_acquire at the - * top of each callback to pair with the memory_order_release store - * in detachCallbacks(). - */ - std::atomic callbacksDetached_{false}; - - /** - * The SDK MeterProvider that owns the export pipeline. - */ - std::shared_ptr provider_; - - /** - * The Meter used to create all instruments. - */ - opentelemetry::nostd::shared_ptr meter_; - - // --- Synchronous instruments (RPC) --- - /** - * Counter: rpc_method_started_total{method=""} - */ - opentelemetry::nostd::unique_ptr> rpcStartedCounter_; - /** - * Counter: rpc_method_finished_total{method=""} - */ - opentelemetry::nostd::unique_ptr> rpcFinishedCounter_; - /** - * Counter: rpc_method_errored_total{method=""} - */ - opentelemetry::nostd::unique_ptr> rpcErroredCounter_; - /** - * Histogram: rpc_method_us{method=""} - */ - opentelemetry::nostd::unique_ptr> - rpcDurationHistogram_; - - // --- Synchronous instruments (Job Queue) --- - // All five carry handler="" in addition to - // job_type, so producers that share a job type stay distinguishable. - /** - * Counter: job_queued_total{job_type="",handler=""} - */ - opentelemetry::nostd::unique_ptr> jobQueuedCounter_; - /** - * Counter: job_started_total{job_type="",handler=""} - */ - opentelemetry::nostd::unique_ptr> jobStartedCounter_; - /** - * Counter: job_finished_total{job_type="",handler=""} - */ - opentelemetry::nostd::unique_ptr> jobFinishedCounter_; - /** - * Counter: jobq_stall_total{job_type=""} — one per finished job - * whose run time reached kJobStallThresholdUs. - */ - opentelemetry::nostd::unique_ptr> jobStallCounter_; - /** - * Histogram: job_queued_us{job_type="",handler=""} - */ - opentelemetry::nostd::unique_ptr> - jobQueuedDurationHistogram_; - /** - * Histogram: job_running_us{job_type="",handler=""} - */ - opentelemetry::nostd::unique_ptr> - jobRunningDurationHistogram_; - - // --- Observable gauges (registered via callbacks) --- - // Handles are stored so we can remove callbacks on shutdown. - /** - * Observable gauges for cache hit rates and sizes. - */ - opentelemetry::nostd::shared_ptr - cacheHitRateGauge_; - /** - * Observable gauges for TxQ metrics. - */ - opentelemetry::nostd::shared_ptr txqGauge_; - /** - * Observable gauges for counted object instances. - */ - opentelemetry::nostd::shared_ptr - objectCountGauge_; - /** - * Observable gauges for load factor breakdown. - */ - opentelemetry::nostd::shared_ptr loadFactorGauge_; - /** - * Observable gauge multiplexing every NodeStore value onto one - * instrument via its `metric` label: I/O totals, the read and write - * means derived from them, the NuDB write-queue detail, and the - * ledger-acquisition stall counters. - */ - opentelemetry::nostd::shared_ptr nodeStoreGauge_; - /** - * Observable gauge for online-delete rotation state and its copy-forward - * write total. Publishes nothing on a node without `online_delete`, where - * the node store is not a rotating one. - */ - opentelemetry::nostd::shared_ptr - rotationStateGauge_; - /** - * Observable gauge for server-level health metrics (state, uptime, peers, etc.). - */ - opentelemetry::nostd::shared_ptr serverInfoGauge_; - /** - * Observable gauge for trusted UNL key count against the required quorum. - */ - opentelemetry::nostd::shared_ptr unlQuorumGauge_; - /** - * Observable gauge for the network close-time offset (local clock skew). - */ - opentelemetry::nostd::shared_ptr clockSkewGauge_; - /** - * Observable gauge for the sync-pipeline state signals (time to first - * FULL, network-ledger gate, server stall, ledgers behind the network). - */ - opentelemetry::nostd::shared_ptr syncStateGauge_; - /** - * ObservableCounter: server_stall_events_total — observed from - * LoadManager::getStallEventCount() (cumulative episode tally owned by the - * load-monitor thread). - */ - opentelemetry::nostd::shared_ptr - stallEventsObservable_; - /** - * Observable gauge for aggregate ledger-acquire progress (max missing state - * and tx nodes, received-data stash depth, in-flight acquire count). - */ - opentelemetry::nostd::shared_ptr - syncAcquireGauge_; - /** - * Observable gauge for the SHAMap tree-node cache hit rate, which is the - * memory layer above the node store's own hit ratio. - */ - opentelemetry::nostd::shared_ptr - shamapCacheHitRateGauge_; - /** - * Observable gauge for global worker-pool saturation: tasks in flight, - * configured worker threads, and total jobs queued. - */ - opentelemetry::nostd::shared_ptr - jobQueueSaturationGauge_; - /** - * Observable gauge for how much of the needed ledger range the connected - * peer set can actually serve. - */ - opentelemetry::nostd::shared_ptr - peerLedgerSupplyGauge_; - /** - * Observable gauge for PeerFinder slot occupancy, connection attempts, - * fixed peers and address-cache depth. - */ - opentelemetry::nostd::shared_ptr slotCensusGauge_; - /** - * Observable gauge for the amendment-block warning flag and the countdown - * to the amendment activating. - */ - opentelemetry::nostd::shared_ptr - amendmentBlockGauge_; - /** - * Observable gauge for the pre-accept quorum gate and the publish lag: - * the trusted-validation tally against the quorum it must reach, the - * time to the first fully-validated ledger, and how far publishing - * trails validation. - */ - opentelemetry::nostd::shared_ptr - ledgerQuorumPublishGauge_; - /** - * Observable gauge for build version info (label-based, value=1). - */ - opentelemetry::nostd::shared_ptr buildInfoGauge_; - /** - * Observable gauge for complete ledger range start/end pairs. - */ - opentelemetry::nostd::shared_ptr - completeLedgersGauge_; - /** - * Observable gauge for database sizes and historical fetch rate. - */ - opentelemetry::nostd::shared_ptr dbMetricsGauge_; - - // --- External dashboard parity gauges --- - /** - * Observable gauge for validator health indicators (amendment blocked, - * UNL blocked, quorum, UNL expiry). - */ - opentelemetry::nostd::shared_ptr - validatorHealthGauge_; - /** - * Observable gauge for peer network quality metrics (P90 latency, - * insane peer count, version spread, upgrade recommendation). - */ - opentelemetry::nostd::shared_ptr - peerQualityGauge_; - /** - * Observable gauge for transaction reduce-relay efficiency (selected vs - * suppressed peers, feature-disabled peers, missing-tx frequency). - */ - opentelemetry::nostd::shared_ptr - reduceRelayGauge_; - /** - * Observable gauge for ledger economy metrics (base fee, reserve, - * reserve increment, ledger age). - */ - opentelemetry::nostd::shared_ptr - ledgerEconomyGauge_; - /** - * Observable gauge for node state tracking (operating mode value, - * time in current state). - */ - opentelemetry::nostd::shared_ptr - stateTrackingGauge_; - /** - * Observable gauge for storage detail metrics (NuDB on-disk size). - */ - opentelemetry::nostd::shared_ptr - storageDetailGauge_; - /** - * Observable gauge for validation agreement metrics (1h/24h percentages - * and counts from ValidationTracker). - */ - opentelemetry::nostd::shared_ptr - validationAgreementGauge_; - - // --- External dashboard parity counters --- - /** - * Counter: ledgers_closed_total — incremented each consensus round. - */ - opentelemetry::nostd::unique_ptr> - ledgersClosedCounter_; - /** - * Counter: validations_sent_total — incremented when this node sends a validation. - */ - opentelemetry::nostd::unique_ptr> - validationsSentCounter_; - /** - * Counter: validations_checked_total — incremented for each network validation - * received. - */ - opentelemetry::nostd::unique_ptr> - validationsCheckedCounter_; - /** - * ObservableCounter: jq_trans_overflow_total — observed from - * Overlay::getJqTransOverflow() (cumulative overflow tally owned by the overlay). - */ - opentelemetry::nostd::shared_ptr - jqTransOverflowObservable_; - /** - * Counter: ledger_history_mismatch_total{reason} — incremented per classified - * built-vs-validated ledger mismatch. - */ - opentelemetry::nostd::unique_ptr> - ledgerHistoryMismatchCounter_; - /** - * Counter: txq_expired_total — incremented per transaction expired out of the - * transaction queue. - */ - opentelemetry::nostd::unique_ptr> txqExpiredCounter_; - /** - * Counter: txq_dropped_total{reason} — incremented when a transaction is refused - * admission to the queue. - */ - opentelemetry::nostd::unique_ptr> txqDroppedCounter_; - /** - * ObservableCounter: validation_agreements_total — observed from - * ValidationTracker::totalAgreementsEver() (monotonic gross lifetime - * tally, initial-classification semantics). - */ - opentelemetry::nostd::shared_ptr - validationAgreementsObservable_; - /** - * ObservableCounter: validation_missed_total — observed from - * ValidationTracker::totalMissedEver() (monotonic gross lifetime tally, - * initial-classification semantics). - */ - opentelemetry::nostd::shared_ptr - validationMissedObservable_; - - /** - * Build the OTLP/HTTP exporter, periodic reader, resource attributes and - * histogram views, then create the MeterProvider and meter. Extracted - * from the constructor to keep each function under the 80-line limit. - * - * @param options Endpoint, TLS settings and resource identity, forwarded - * unchanged from the constructor. See @ref Options. - */ - void - initExporterAndProvider(Options const& options); - - /** - * Create the synchronous instruments (RPC and job-queue counters and - * histograms, plus the external dashboard parity counters). Extracted - * from the constructor to keep each function under the 80-line limit. - */ - void - initSyncInstruments(); - - /** - * Give up the pipeline after a build failure: drop the provider, hand - * out a no-op meter so every call site still gets an instrument, and log - * why. The registry stays enabled and inert for the process. - * - * @param reason What failed, for the log line. - */ - void - disablePipeline(std::string_view reason); - - /** - * Register all observable gauge callbacks with the OTel SDK. - * Dispatches to one helper per metric domain so that each helper - * stays well under the 80-line-per-function limit. - * - * Called only from `startAsyncGauges()`, which owns the enabled_, - * phase_ and provider_ guards and the Application-state precondition. - */ - void - registerAsyncGauges(); - - // Per-domain registration helpers for the async (pull-model) phase. - // Each creates its instrument -- an ObservableGauge, or an - // ObservableCounter where the underlying value is cumulative -- and - // attaches a single callback that reads current values from Application - // services. The callbacks run on the OTel - // PeriodicExportingMetricReader background thread (~10 s tick). - void - registerJqTransOverflowCounter(); // gap-fill: overlay overflow total - void - registerCacheHitRateGauge(); - /** - * Observe the two TaggedCache lock-hold peaks onto the cache_metrics - * gauge. Split out to keep registerCacheHitRateGauge's callback under - * the 80-line limit. Static because it touches neither instance state - * nor telemetry members — it reads through the passed app reference. - */ - static void - observeCacheLockHoldPeaks(opentelemetry::metrics::ObserverResult& result, ServiceRegistry& app); - void - registerTxqGauge(); - void - registerObjectCountGauge(); - void - registerLoadFactorGauge(); - void - registerNodeStoreGauge(); - - // The four nodestore_state helpers and their ObserveFn sink are public - // (above), so a test can drive each one with a recording sink and assert - // the exact `metric` label values it publishes. They read only their - // arguments, so exposing them widens no state. - - void - registerRotationStateGauge(); // Sync diagnostics: online_delete rotation - void - registerServerInfoGauge(); - void - registerBuildInfoGauge(); - void - registerCompleteLedgersGauge(); - void - registerDbMetricsGauge(); - void - registerValidatorHealthGauge(); - void - registerPeerQualityGauge(); - void - registerReduceRelayGauge(); // Reduce-relay efficiency - void - registerLedgerEconomyGauge(); - void - registerStateTrackingGauge(); - void - registerStorageDetailGauge(); - void - registerValidationAgreementGauge(); - void - registerValidationTotalsCounters(); // gap-fill: lifetime agree/miss _total - - /** - * Register the `unl_quorum` gauge. - * - * Observes two series under the `metric` attribute: - * `trusted_keys` (ValidatorList::trustedKeyCount()) and `quorum` - * (ValidatorList::quorum()). Both are cheap accessors — one shared - * lock and one atomic load. - * - * `trusted_keys < quorum` means the node can never fully validate a - * ledger, so it will sit in `syncing` until the UNL is fixed. That - * makes this the first place to look when a node never leaves - * `syncing`. - * - * @note Pulled on the OTel reader thread (~10 s tick); does no work - * on any hot path. - */ - void - registerUnlQuorumGauge(); // sync diagnostics: UNL vs quorum - - /** - * Register the `clock_close_offset_seconds` gauge. - * - * Observes one series, `offset`, from - * TimeKeeper::closeOffset(): the seconds this node's notion of - * network close time is displaced from its own wall clock. - * - * The value MAY BE NEGATIVE, meaning the local clock runs ahead of - * the network. Whole-second resolution is all the signal carries, - * since that is the unit TimeKeeper stores. - * - * @note `server_info` only reports this field once |offset| >= 60 s - * (NetworkOPs), so this gauge is the first continuous export of it. - * Pulled on the OTel reader thread (~10 s tick); one atomic load. - */ - void - registerClockSkewGauge(); // sync diagnostics: close-time offset - - /** - * Register the `sync_state` gauge. - * - * One instrument fanning out four series under the `metric` attribute, - * each answering a different "why is this node not FULL yet?" question - * that is otherwise visible only in a log line or in server_info JSON: - * - * `initial_full_duration_us` — microseconds from process start to the - * first FULL transition (NetworkOPs::getInitialSyncDurationUs()). - * Stays 0 until FULL is reached, so a flat 0 IS the "never synced" - * signal; once set it never changes again. - * `network_ledger_gate` — 1 while the node is still waiting to see a - * full network ledger (NetworkOPs::isNeedNetworkLedger()), else 0. A - * persistent 1 blocks transaction submission and FULL. - * `server_stall_seconds` — current main-loop stall duration - * (LoadManager::getCurrentStallSeconds()), 0 when healthy. - * `ledgers_behind` — network tip minus our validated sequence - * (NetworkOPs::getLedgersBehindNetwork()). - * - * The monotonic stall-episode count is a separate instrument - * (`server_stall_events_total`) because a counter and a gauge cannot share - * one instrument: Prometheus would otherwise see a cumulative total under - * last-value aggregation and `rate()` would be meaningless. - * - * @note Pulled on the OTel reader thread (~10 s tick), never on a hot - * path. Three of the four reads are a lock or atomic load; `ledgers_behind` - * additionally walks the connected-peer list, reading each peer's already - * cached ledger range — bounded by peer count and issuing no network I/O. - */ - void - registerSyncStateGauge(); // sync diagnostics: gate, stall, ledgers behind - - /** - * Register the `server_stall_events_total` observable counter. - * - * Observes LoadManager::getStallEventCount(): how many distinct stall - * episodes the monitor thread has reported since process start. Separate - * from `sync_state` because it is cumulative and monotonic, so it needs - * counter (not last-value) aggregation for `rate()` to mean anything. - * - * Read together with `sync_state{metric="server_stall_seconds"}`: a rising - * event count means repeated fresh stalls, while a flat count with a large - * stall-seconds value means one long unresolved stall. - * - * @note Pulled on the OTel reader thread (~10 s tick); one atomic load. - */ - void - registerStallEventsCounter(); // sync diagnostics: stall episode count - - /** - * Register the `sync_acquire` gauge. - * - * One instrument fanning out four series under the `metric` attribute, all - * from a single InboundLedgers::acquireProgress() snapshot: - * - * `missing_state_nodes_max` — largest outstanding account-state node count - * of any in-flight acquire. THE headline stuck-sync signal: flat and - * non-zero across ticks means the acquire will never finish, shrinking - * means it is slow but alive. - * `missing_tx_nodes_max` — the same for the transaction tree. - * `received_data_depth` — peer packets stashed across all acquires waiting - * to be applied. Deep means processing, not peer supply, is the limit. - * `in_flight` — how many acquires are running, so the three values above - * can be read in context: all zero with `in_flight` zero is idle, not - * healthy. - * - * Deliberately aggregated rather than per-ledger. A `ledger_seq` label would - * mint a new time series for every ledger the node ever acquires, which is - * unbounded cardinality; the max/sum keeps the "is it stuck?" answer while - * the per-ledger identity stays on the `ledger.acquire` span, where - * high-cardinality identity belongs. - * - * @note Pulled on the OTel reader thread (~10 s tick), never on a hot path. - * The snapshot takes the acquire-collection lock only to copy shared_ptrs, - * then reads relaxed atomics; the emit sites that feed those atomics all sit - * outside the per-tree-node loops. - */ - void - registerSyncAcquireGauge(); // sync diagnostics: acquire progress - - /** - * Register the `shamap_cache_hit_rate` gauge. - * - * Observes one series, `treenode`, from TreeNodeCache::getHitRate(): the - * percentage of SHAMap tree-node lookups served from memory instead of the - * node store. During a fresh sync a low rate means the node re-reads the - * same subtrees from disk, so sync is disk-bound rather than peer-bound. - * - * Distinct from the `NuDB Cache Hit Ratio` panel on the ledger-data-sync - * dashboard: that one is derived from `nodestore_state` and measures the - * node-store layer (`node_reads_hit / node_reads_total`). This gauge - * measures the in-memory tree-node cache that sits ABOVE it, so a request - * missing here is what produces a node-store read there. - * - * The full-below cache is deliberately NOT reported. It is a KeyCache, whose - * only lookup path is TaggedCache::touchIfExists(), and that method - * increments `stats_.hits`/`stats_.misses` while `getHitRate()` reads the - * separate `hits_`/`misses_` members. Its hit rate is therefore hard-wired - * to 0 regardless of behaviour, so exporting it would ship a permanently - * empty panel; fixing that accounting belongs in a libxrpl change of its own. - * - * @note Pulled on the OTel reader thread (~10 s tick). Takes the cache's - * mutex for two integer reads and a divide; no hot-path cost. - */ - void - registerCacheHitRateDetailGauge(); // sync diagnostics: treenode cache - - /** - * Register the `jobq_saturation` gauge. - * - * Three series under the `metric` attribute, from one - * JobQueue::getWorkerSaturation() reading: - * - * `running_tasks` — worker threads currently executing a job. - * `worker_threads` — threads the pool is configured to run, the - * denominator that makes `running_tasks` legible. Exported rather - * than hardcoded in the dashboard because it is derived at startup - * from `[workers]`, node size and hardware concurrency. - * `total_waiting` — jobs queued across all types. - * - * The reason this is separate from the per-job-type gauges JobQueue - * itself publishes (`jobq__waiting` / `_running` / `_deferred`): - * when the pool itself is exhausted, every subsystem waiting behind it - * looks independently slow, and each per-type panel invites the wrong - * conclusion. A `running_tasks / worker_threads` ratio at 1.0 with a - * non-zero `total_waiting` attributes the whole slowdown to pool - * exhaustion once. Those per-type gauges carry no capacity term at all, - * so no reading there can say whether the pool is the cause. - * - * @note Pulled on the OTel reader thread (~10 s tick). One atomic load, - * one plain int read, and one pass over the per-type counters under the - * JobQueue mutex. - */ - void - registerJobQueueSaturationGauge(); // sync diagnostics: pool saturation - - /** - * Register the `peer_ledger_supply` gauge. - * - * Five series under the `metric` attribute, from one - * Overlay::getPeerLedgerSupply() pass over the active peers: - * - * `peers_reporting` — peers that have advertised a ledger range at all. - * The denominator that makes the rest readable. - * `peers_serving_validated` — peers whose range covers this node's - * validated sequence. - * `peers_serving_next` — **the signal this gauge exists for.** Peers - * whose range covers validated + 1, the next ledger this node must - * acquire. Zero here with a non-zero `peers_reporting` means no - * connected peer holds what this node needs, so no amount of waiting - * will finish the sync; the peer set has to change. - * `supply_min_seq` / `supply_max_seq` — the sequence window the peer set - * covers, so an operator can see whether the node is asking for - * history nobody kept or for a tip nobody has reached. - * - * Each peer already caches the range it advertises in mtSTATUS_CHANGE - * (`PeerImp::minLedger_` / `maxLedger_`, read via `Peer::ledgerRange()`), - * but those ranges were never compared against each other, so "no peer has - * what I need" was indistinguishable from "my peers are slow" — the two - * faults with completely different fixes. - * - * Distinct from what already exists. `server_info{metric="peers"}` is a - * bare connection count with no notion of what those peers hold. - * `sync_state{metric="ledgers_behind"}` uses the same per-peer maxima but - * collapses them to a single distance-to-tip number, which cannot say how - * many peers can serve that distance or whether the range has a hole. - * `peer_quality{metric="peers_insane_count"}` counts peers on a different - * chain, which is a correctness signal, not an availability one. - * - * Peers advertising [0, 0] have not reported yet and are excluded from - * every field, so they cannot make a healthy peer set appear to serve from - * genesis. When nothing has reported, both window fields read 0, which is - * why `peers_reporting` must be read alongside them. - * - * @note Pulled on the OTel reader thread (~10 s tick), never on a message - * path. O(peers): `getActivePeers()` copies the peer list under the overlay - * lock and releases it, then each peer's cached range is read under that - * peer's own short-lived lock. - */ - void - registerPeerLedgerSupplyGauge(); // sync diagnostics: peer range coverage - - /** - * Register the `peerfinder_slot_census` gauge. - * - * Nine series under the `metric` attribute, from one - * Overlay::getSlotCensus() snapshot: `out_active`, `out_max`, `in_active`, - * `in_max`, `connecting`, `fixed_configured`, `fixed_active`, `bootcache` - * and `livecache`. - * - * All nine are already computed inside PeerFinder (`Counts`, `Bootcache`, - * `Livecache`, the fixed-peer map) and only two of them are exported - * today, as the legacy beast::insight gauges - * `peer_finder_active_inbound_peers` and - * `peer_finder_active_outbound_peers`. Those two carry no capacity, - * attempt or cache term, which leaves the three most common bootstrap - * failures invisible: - * - * - `connecting` non-zero while `out_active` stays below `out_max` — - * dials are being started and never completing. Without the attempt - * count this looks the same as a node that is not dialling at all. - * - `bootcache` at 0 — no seed addresses to dial in the first place. - * - `fixed_active` below `fixed_configured` — a peer named in the - * configuration is unreachable. - * - * The nine fields come from a single acquire of the PeerFinder lock, so - * they are mutually consistent and share one label set. The two legacy - * gauges are read at unrelated instants and cannot be joined with each - * other, let alone with a capacity term. - * - * @note Pulled on the OTel reader thread (~10 s tick). One lock acquire, - * then integer and container-size reads. - */ - void - registerSlotCensusGauge(); // sync diagnostics: peerfinder slot census - - /** - * Register the `amendment_block` gauge. - * - * Two series under the `metric` attribute: - * - * `warned` — 1 once an unsupported amendment has reached majority, from - * NetworkOPs::isAmendmentWarned(). - * `seconds_to_block` — **the leading indicator.** Seconds until that - * amendment activates, derived from - * `AmendmentTable::firstUnsupportedExpected()` against the network - * close time. `-1` when nothing is pending, matching the sentinel - * `validator_health{metric="unl_expiry_days"}` already uses, so the - * healthy state is a distinct value rather than a missing series. - * Clamped at 0 rather than going negative, because past-due means the - * block is imminent, not overdue by some amount. - * - * Amendment-blocked is a terminal sync blocker: the node stops validating - * and never resumes without a software upgrade. The existing - * `validator_health{metric="amendment_blocked"}` reports that state after - * it has happened, when nothing can be done about it. This gauge is the - * window before it, which is the only actionable part. - * - * The blocking amendment's identity is deliberately NOT a label. The - * network can vote on an arbitrary 256-bit amendment id — the set is not - * drawn from this build's known features — so an id label would be - * unbounded cardinality and would mint a permanent new series per - * amendment. The id is already logged by - * `AmendmentTableImpl::doValidatedLedger` ("Unsupported amendment - * reached majority at ..."), so it is available through logs, correlated - * to this series by node and time. - * - * @note Pulled on the OTel reader thread (~10 s tick). One mutex acquire - * inside the amendment table plus one clock read. - * @note The subtraction is done in `std::int64_t`, not in NetClock's - * unsigned representation, so a past-due activation cannot wrap to a huge - * positive count. - */ - void - registerAmendmentBlockGauge(); // sync diagnostics: amendment countdown - - /** - * Register the `ledger_quorum_publish` gauge. - * - * Four series under the `metric` attribute, read from LedgerMaster: - * - * `trusted_validation_tally` — trusted validations counted at the last - * pre-accept gate in `LedgerMaster::checkAccept`. - * `quorum_target` — validations that gate required. **The pair is the - * signal.** The tally alone cannot separate a node accumulating - * validations toward quorum (slow, will finish) from one whose tally - * plateaus below the target (stuck, never will); with the target - * beside it, the two shapes are unmistakable. - * `time_to_first_validated_us` — how long the node took to get its - * first ledger through that gate. One-shot, like the - * `sync_state{initial_full_duration_us}` milestone: a value means it - * happened and this is how long it took, 0 means it never has. - * `publish_lag` — validated sequence minus published sequence. Non-zero - * and growing means validation is fine and the publish pipeline is - * behind, which no other signal distinguishes. - * - * All four are grouped under one instrument because they answer one - * question in sequence — did enough validations arrive, did the gate pass, - * how long did that take, and did the result reach clients — so an - * operator reads them from a single consistent poll. - * - * Distinct from what already exists. `unl_quorum{quorum}` is the quorum - * the validator list *configures*, a static property of the trusted set; - * `quorum_target` is what an actual gate evaluation *required*, and the - * tally beside it is the live count that must reach it — neither existed - * anywhere before. `server_info{validated_ledger_seq}` publishes the - * validated sequence but nothing published the pubLedgerSeq_ counterpart, - * so the lag between them was not derivable at all. - * - * @note `quorum_target` reports int64 max when the validator list has - * switched quorum off (`ValidatorList::quorum()` returns SIZE_MAX). The - * clamp lives in `LedgerMaster::checkAccept`, so the wrap to -1 that would - * invert a tally-versus-target panel cannot happen here. - * @note Pulled on the OTel reader thread (~10 s tick). Five relaxed atomic - * loads through lock-free LedgerMaster accessors: no lock is taken, which - * is what keeps an OTel callback from ever contending with, or inverting - * lock order against, the LedgerMaster mutex held by the emit path. - */ - void - registerLedgerQuorumPublishGauge(); // sync diagnostics: quorum + publish -#endif // XRPL_ENABLE_TELEMETRY -}; - -} // namespace telemetry -} // namespace xrpl