From d29e392c0b48fc4fe8199f86c34e2d843c8c7d29 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Tue, 15 Sep 2026 16:57:52 +0100 Subject: [PATCH 1/2] alert(NodeStateFlapping): fire on a single flap, keep the always-present metric The rule watched state_accounting_full_transitions > 3 per hour, so a node that flaps once (one full -> syncing -> full round, e.g. per online-delete rotation) never tripped it. Lower the threshold to > 0 so a single re-entry into FULL, past the one-hour uptime gate, alerts. Keep the state_accounting_full_transitions metric: it is a cumulative gauge every node always reports, so increase() yields a real series (0 when healthy) and the rule never evaluates to NoData. A sparse counter would raise a false DatasourceNoData on a healthy node. Set noDataState: OK so a scrape gap cannot page either. --- .../grafana/provisioning/alerting/rules.yaml | 29 ++++++++++--------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/docker/telemetry/grafana/provisioning/alerting/rules.yaml b/docker/telemetry/grafana/provisioning/alerting/rules.yaml index 7d909074b2..0f0f8223a3 100644 --- a/docker/telemetry/grafana/provisioning/alerting/rules.yaml +++ b/docker/telemetry/grafana/provisioning/alerting/rules.yaml @@ -629,25 +629,25 @@ groups: # Node state flapping: full -> syncing/tracking -> full, repeatedly. # # state_accounting_full_transitions counts transitions INTO full - # (NetworkOPs.cpp StateAccounting::mode) and is exported as a cumulative - # gauge, so increase() is correct — and its counter-reset correction - # turns a process restart into a small positive delta rather than a - # false spike. + # (NetworkOPs.cpp StateAccounting::mode), a cumulative gauge that EVERY + # node always reports, so increase() returns a real series (0 when + # healthy) and this rule never evaluates to NoData. A sparse counter such + # as state_changes_total{from,to} has no series until the edge occurs, so + # it would raise a false DatasourceNoData on a healthy node -- do not + # switch to it here. # - # state_changes_total cannot be used here: it carries no from/to labels, - # so it cannot distinguish a flap from a normal startup walk. + # Threshold >0: one full -> syncing -> full round is a single re-entry, + # which is exactly the online-delete rotation flap to catch. # # The uptime gate is load-bearing. Every node walks # disconnected -> connected -> syncing -> tracking -> full once at boot; - # without the gate every restart pages. Measured: flapping nodes re-enter - # full 4-6 times per hour sustained, healthy nodes 0-1, so >3 separates - # the populations with a 3x margin. + # the gate suppresses that first hour so a restart does not page. - uid: xrpld-node-state-flapping title: NodeStateFlapping condition: C for: 15m isPaused: true - noDataState: NoData + noDataState: OK execErrState: Error labels: severity: warning @@ -656,9 +656,10 @@ groups: summary: "Node state flapping on {{ $labels.service_instance_id }}" description: >- Node {{ $labels.service_instance_id }} re-entered the FULL state - {{ $values.B.Value }} times in the last hour (>3). It is oscillating - between full and syncing/connected rather than holding sync. Check - node-store IO latency, peer connectivity, and clock sync. + {{ $values.B.Value }} time(s) in the last hour past its first hour + of uptime. It is flapping out of sync rather than holding FULL. + Likely the online-delete rotation cache-freshen; check the rotation + spans and cache lock-hold peak. data: - refId: A relativeTimeRange: @@ -700,7 +701,7 @@ groups: conditions: - evaluator: type: gt - params: [3] + params: [0] datasource: type: __expr__ uid: __expr__ From ec0bfe521d4616e61fd70ecd4876a3f88113607a Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Wed, 16 Sep 2026 13:45:52 +0100 Subject: [PATCH 2/2] refactor(telemetry): move the metrics pipeline core into libxrpl MetricsRegistry did two jobs. It owned the OTel metrics pipeline, and it registered the observable gauges whose callbacks read live application services. The second job is what made the whole class xrpld-tier, so the pipeline's lifecycle -- the recording() gate and the stop() teardown that closes a use-after-free window -- could not be unit-tested in xrpl_tests. Split it in two: - xrpl::telemetry::MetricsRegistry (libxrpl) owns the exporter, provider, meter, the 16 synchronous instruments, recording(), stop(), and the record*/increment* methods. - xrpl::telemetry::AppMetricGauges (xrpld) owns the 19 observable gauges and their callbacks, holding a reference to the core and to the ServiceRegistry. MetricMacros.h and ValidationTracker move with the core. The macros need only recording() and meter(), both core members; the core holds a tracker by value, and a libxrpl header cannot include one from src/. ApplicationImp owns both objects and sequences them. The core is built in the member-init list, so every synchronous instrument exists before any subsystem can record one. The gauges are armed once overlay_ exists, the last service their callbacks read. Shutdown detaches the gauge callbacks before the core drops the provider, and each shutdown step is isolated so a failure in one cannot skip the others. That detach call is new. detachCallbacks() had no callers, and the flag it sets is read by the gauge callbacks but can no longer be written by the core, so the caller now has to make the ordering explicit. The telemetry module links xrpl.libxrpl.core and xrpl.libxrpl.protocol PUBLIC: ValidationTracker.h takes a LedgerIndex and MetricMacros.h takes a ServiceRegistry, both in interfaces a consumer compiles against. Adds a MetricsRegistry gtest that drives an enabled core with telemetry on and pins the recording() gate, stop() leaving the registry inert, and stop() being idempotent. The libxrpl test tree no longer depends on xrpld.telemetry at all, and the two CMake workarounds that compiled xrpld sources into xrpl_tests are gone. Documentation and dashboard source links follow the code to their new paths, split between the two classes by which one now defines each metric. --- .../scripts/levelization/results/loops.txt | 2 +- .../scripts/levelization/results/ordering.txt | 6 +- CONTRIBUTING.md | 2 +- OpenTelemetryPlan/06-implementation-phases.md | 43 +- OpenTelemetryPlan/08-appendix.md | 18 +- .../09-data-collection-reference.md | 82 +- cmake/XrplCore.cmake | 11 +- .../grafana/dashboards/consensus-health.json | 2 +- .../grafana/dashboards/fee-market.json | 14 +- .../grafana/dashboards/job-queue.json | 16 +- .../grafana/dashboards/ledger-data-sync.json | 6 +- .../grafana/dashboards/ledger-operations.json | 2 +- .../grafana/dashboards/node-health.json | 62 +- .../grafana/dashboards/peer-network.json | 4 +- .../grafana/dashboards/peer-quality.json | 10 +- .../grafana/dashboards/rpc-performance.json | 14 +- .../grafana/dashboards/validator-health.json | 34 +- docs/telemetry-runbook.md | 27 +- .../xrpl}/telemetry/MetricMacros.h | 5 +- .../xrpl}/telemetry/MetricsRegistry.h | 542 +++--------- .../xrpl}/telemetry/ValidationTracker.h | 0 src/libxrpl/telemetry/MetricsRegistry.cpp | 603 +++++++++++++ .../telemetry/detail/ValidationTracker.cpp | 2 +- src/test/nodestore/DatabaseConfig_test.cpp | 20 +- src/tests/libxrpl/CMakeLists.txt | 32 +- .../beast/insight/OTelCollectorHooks.cpp | 58 +- .../libxrpl/helpers/ManualMetricReader.h | 80 ++ src/tests/libxrpl/nodestore/Backend.cpp | 2 +- src/tests/libxrpl/telemetry/MetricMacros.cpp | 2 +- .../libxrpl/telemetry/MetricsRegistry.cpp | 741 ++++++++-------- .../libxrpl/telemetry/ValidationTracker.cpp | 2 +- src/xrpld/app/consensus/RCLConsensus.cpp | 4 +- src/xrpld/app/ledger/LedgerHistory.cpp | 2 +- src/xrpld/app/ledger/detail/LedgerMaster.cpp | 2 +- src/xrpld/app/main/Application.cpp | 93 +- src/xrpld/app/misc/NetworkOPs.cpp | 2 +- src/xrpld/app/misc/detail/TxQ.cpp | 2 +- src/xrpld/overlay/detail/PeerImp.cpp | 2 +- src/xrpld/perflog/detail/PerfLogImp.cpp | 4 +- src/xrpld/rpc/detail/PathRequest.cpp | 2 +- src/xrpld/rpc/detail/ServerHandler.cpp | 2 +- ...etricsRegistry.cpp => AppMetricGauges.cpp} | 804 ++++-------------- src/xrpld/telemetry/AppMetricGauges.h | 473 +++++++++++ 43 files changed, 2013 insertions(+), 1823 deletions(-) rename {src/xrpld => include/xrpl}/telemetry/MetricMacros.h (99%) rename {src/xrpld => include/xrpl}/telemetry/MetricsRegistry.h (62%) rename {src/xrpld => include/xrpl}/telemetry/ValidationTracker.h (100%) create mode 100644 src/libxrpl/telemetry/MetricsRegistry.cpp rename src/{xrpld => libxrpl}/telemetry/detail/ValidationTracker.cpp (99%) create mode 100644 src/tests/libxrpl/helpers/ManualMetricReader.h rename src/xrpld/telemetry/{MetricsRegistry.cpp => AppMetricGauges.cpp} (60%) create mode 100644 src/xrpld/telemetry/AppMetricGauges.h diff --git a/.github/scripts/levelization/results/loops.txt b/.github/scripts/levelization/results/loops.txt index 649f1c7b95..6cc0a78db5 100644 --- a/.github/scripts/levelization/results/loops.txt +++ b/.github/scripts/levelization/results/loops.txt @@ -8,7 +8,7 @@ Loop: xrpld.app xrpld.shamap xrpld.shamap > xrpld.app Loop: xrpld.app xrpld.telemetry - xrpld.app > xrpld.telemetry + xrpld.telemetry ~= xrpld.app Loop: xrpld.overlay xrpld.rpc xrpld.rpc ~= xrpld.overlay diff --git a/.github/scripts/levelization/results/ordering.txt b/.github/scripts/levelization/results/ordering.txt index dc5e0a9774..c1287c7ac3 100644 --- a/.github/scripts/levelization/results/ordering.txt +++ b/.github/scripts/levelization/results/ordering.txt @@ -194,7 +194,6 @@ tests.libxrpl > xrpl.consensus tests.libxrpl > xrpl.core tests.libxrpl > xrpld.app tests.libxrpl > xrpld.rpc -tests.libxrpl > xrpld.telemetry tests.libxrpl > xrpl.json tests.libxrpl > xrpl.ledger tests.libxrpl > xrpl.net @@ -253,6 +252,8 @@ xrpl.shamap > xrpl.nodestore xrpl.shamap > xrpl.protocol xrpl.telemetry > xrpl.basics xrpl.telemetry > xrpl.config +xrpl.telemetry > xrpl.core +xrpl.telemetry > xrpl.protocol xrpl.tx > xrpl.basics xrpl.tx > xrpl.core xrpl.tx > xrpl.ledger @@ -307,15 +308,14 @@ xrpld.perflog > xrpl.config xrpld.perflog > xrpl.core xrpld.perflog > xrpld.app xrpld.perflog > xrpld.rpc -xrpld.perflog > xrpld.telemetry xrpld.perflog > xrpl.json xrpld.perflog > xrpl.nodestore xrpld.perflog > xrpl.protocol +xrpld.perflog > xrpl.telemetry xrpld.rpc > xrpl.basics xrpld.rpc > xrpl.config xrpld.rpc > xrpl.core xrpld.rpc > xrpld.core -xrpld.rpc > xrpld.telemetry xrpld.rpc > xrpl.json xrpld.rpc > xrpl.ledger xrpld.rpc > xrpl.net diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 41f4745a7f..d8c3b2372d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -458,7 +458,7 @@ for the full rule list. ## Adding a new OTel metric -See `src/xrpld/telemetry/MetricMacros.h` for the call-site macros covering every +See `include/xrpl/telemetry/MetricMacros.h` for the call-site macros covering every OTel instrument kind (Counter, UpDownCounter, Histogram, Gauge, and their Observable/async counterparts) and the "Adding a New Metric" section in [docs/telemetry-runbook.md](docs/telemetry-runbook.md) for the walkthrough and a diff --git a/OpenTelemetryPlan/06-implementation-phases.md b/OpenTelemetryPlan/06-implementation-phases.md index bf8f07bf46..3a3f096c3a 100644 --- a/OpenTelemetryPlan/06-implementation-phases.md +++ b/OpenTelemetryPlan/06-implementation-phases.md @@ -794,8 +794,10 @@ flowchart LR ## 6.8.2 Phase 9: Internal Metric Instrumentation Gap Fill (Weeks 14-15) > **Status**: Complete. Merged on `pratik/otel-phase9-metric-gap-fill`. Shipped -> artefacts: `src/xrpld/telemetry/MetricsRegistry.{h,cpp}` (~41 KB + ~71 KB), -> `src/xrpld/telemetry/MetricMacros.h`, `include/xrpl/nodestore/WriteStats.h`, +> artefacts: `include/xrpl/telemetry/MetricsRegistry.h` (~40 KB) with +> `src/libxrpl/telemetry/MetricsRegistry.cpp` (~28 KB), +> `src/xrpld/telemetry/AppMetricGauges.{h,cpp}` (~20 KB + ~56 KB), +> `include/xrpl/telemetry/MetricMacros.h`, `include/xrpl/nodestore/WriteStats.h`, > `src/xrpld/app/ledger/AcquireStats.h`, > `include/xrpl/telemetry/GetObjectMetricNames.h`, 10 GTest files under > `src/tests/libxrpl/telemetry/`, 4 new Grafana dashboards, provisioned Grafana @@ -1838,13 +1840,13 @@ class ValidationTracker **Key new files**: -- `src/xrpld/telemetry/ValidationTracker.h` -- `src/xrpld/telemetry/detail/ValidationTracker.cpp` +- `include/xrpl/telemetry/ValidationTracker.h` +- `src/libxrpl/telemetry/detail/ValidationTracker.cpp` **Key modified files**: -- `src/xrpld/telemetry/MetricsRegistry.h` (add ValidationTracker member) -- `src/xrpld/telemetry/MetricsRegistry.cpp` (add gauge callback reading from tracker) +- `include/xrpl/telemetry/MetricsRegistry.h` (add ValidationTracker member) +- `src/xrpld/telemetry/AppMetricGauges.cpp` (add gauge callback reading from tracker) - `src/xrpld/app/consensus/RCLConsensus.cpp` (add recording hooks) - `src/xrpld/app/ledger/detail/LedgerMaster.cpp` (add recording hook) @@ -1869,7 +1871,7 @@ New MetricsRegistry observable gauge for amendment, UNL, and quorum health. | | `unl_expiry_days` | double | `app_.validators().expires()` → days until expiry | | | `validation_quorum` | int64 | `app_.validators().quorum()` | -**File**: `src/xrpld/telemetry/MetricsRegistry.cpp` (new gauge callback in `registerAsyncGauges()`) +**File**: `src/xrpld/telemetry/AppMetricGauges.cpp` (new gauge callback in `registerAsyncGauges()`) **Exit Criteria**: @@ -1892,7 +1894,7 @@ New MetricsRegistry observable gauge for peer health aggregates. **Implementation note**: The callback iterates `app_.overlay().foreach(...)` to collect per-peer latency and version data. This runs every 10s on the metrics reader thread — acceptable overhead for ~50-200 peers. -**File**: `src/xrpld/telemetry/MetricsRegistry.cpp` +**File**: `src/xrpld/telemetry/AppMetricGauges.cpp` **Exit Criteria**: @@ -1915,7 +1917,7 @@ New MetricsRegistry observable gauge for fee and ledger metrics. | | `ledger_age_seconds` | double | `now - lastValidatedCloseTime` | | | `transaction_rate` | double | Derived: tx count delta / time delta | -**File**: `src/xrpld/telemetry/MetricsRegistry.cpp` +**File**: `src/xrpld/telemetry/AppMetricGauges.cpp` **Exit Criteria**: @@ -1950,7 +1952,7 @@ xrpld's `OperatingMode` enum maps 0-4 (DISCONNECTED through FULL). The external **Note**: Values 5-6 require checking both `OperatingMode` and `ConsensusMode`. The callback should derive these from `app_.getOPs().getOperatingMode()` combined with `mConsensus.mode()`. If operating mode is FULL and consensus is proposing → 6; if FULL and validating → 5; otherwise use the raw OperatingMode enum value. -**File**: `src/xrpld/telemetry/MetricsRegistry.cpp` +**File**: `src/xrpld/telemetry/AppMetricGauges.cpp` **Exit Criteria**: @@ -1977,7 +1979,7 @@ The label value was `nudb_bytes` through Phase 8 and was renamed in Phase 9: the value is read from `Database`, not from the NuDB backend, so a backend prefix misdescribed it and the old name implied an on-disk size it never reported. -**File**: `src/xrpld/telemetry/MetricsRegistry.cpp` +**File**: `src/xrpld/telemetry/AppMetricGauges.cpp` **Exit Criteria**: @@ -2002,7 +2004,10 @@ New counters incremented at event sites. Declared in MetricsRegistry, recording **Key modified files**: -- `src/xrpld/telemetry/MetricsRegistry.h/.cpp` (counter declarations) +- `include/xrpl/telemetry/MetricsRegistry.h` and + `src/libxrpl/telemetry/MetricsRegistry.cpp` (synchronous counter declarations) +- `src/xrpld/telemetry/AppMetricGauges.cpp` (the three observed as ObservableCounters: + `validation_agreements_total`, `validation_missed_total`, `jq_trans_overflow_total`) - `src/xrpld/app/consensus/RCLConsensus.cpp` (recording: ledgers_closed, validations_sent) - `src/xrpld/app/ledger/detail/LedgerMaster.cpp` (recording: validations_checked) - `src/xrpld/app/misc/NetworkOPs.cpp` (recording: state_changes) @@ -2028,7 +2033,7 @@ Reads from the `ValidationTracker` (Task 7.8) to export rolling window stats. | | `agreements_24h` | int64 | `tracker.agreements24h()` | | | `missed_24h` | int64 | `tracker.missed24h()` | -**File**: `src/xrpld/telemetry/MetricsRegistry.cpp` +**File**: `src/xrpld/telemetry/AppMetricGauges.cpp` **Exit Criteria**: @@ -2216,12 +2221,12 @@ Phase 9 additionally ships 9 rules with no external counterpart: | Peer Count Critical | `server_info{metric="peers"} < 5` | — | > **"Not Proposing" is unblocked.** The `state_tracking` gauge **is** -> implemented: `MetricsRegistry::registerStateTrackingGauge()` -> (`MetricsRegistry.cpp:1461-1510`) creates -> `CreateDoubleObservableGauge("state_tracking", …)` at `:1466` and observes -> `state_value` (`:1497`) and `time_in_current_state_seconds` (`:1502`). It is -> already consumed by `validator-health.json:765,971` and -> `ledger-data-sync.json:869`, and documented in +> implemented: `AppMetricGauges::registerStateTrackingGauge()` +> (`src/xrpld/telemetry/AppMetricGauges.cpp`) creates +> `CreateDoubleObservableGauge("state_tracking", …)` and observes `state_value` +> and `time_in_current_state_seconds`. It is +> already consumed by `validator-health.json` and +> `ledger-data-sync.json`, and documented in > [09-data-collection-reference.md](./09-data-collection-reference.md) § > "State Tracking". Only **3** of the 14 remaining rules are blocked on anything — > CPU High, Memory Critical and Disk Warning, all needing `node_exporter`. diff --git a/OpenTelemetryPlan/08-appendix.md b/OpenTelemetryPlan/08-appendix.md index 3cff77a23f..e059cfcf7f 100644 --- a/OpenTelemetryPlan/08-appendix.md +++ b/OpenTelemetryPlan/08-appendix.md @@ -163,15 +163,15 @@ This guide maps Phase 9–11 content to its location across the documentation. ### Phase 9: Internal Metric Instrumentation Gap Fill -| Content | Location | -| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | -| Plan & architecture | [06-implementation-phases.md §6.8.2](./06-implementation-phases.md) | -| Task list (18 entries, 9.1–9.17) | [Phase9_taskList.md](./Phase9_taskList.md) | -| Metric definitions | [09-data-collection-reference.md §5b](./09-data-collection-reference.md) | -| New class: `MetricsRegistry` | `src/xrpld/telemetry/MetricsRegistry.h/.cpp` — **shipped** | -| New dashboards (4) | `fee-market`, `job-queue`, `peer-quality`, `validator-health` — **shipped** | -| Updated dashboards (2) | `node-health`, `rpc-performance` | -| Provisioned alert rules | `docker/telemetry/grafana/provisioning/alerting/rules.yaml` — 13 rules in 5 groups ([07 §7.6.2](./07-observability-backends.md)) | +| Content | Location | +| ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Plan & architecture | [06-implementation-phases.md §6.8.2](./06-implementation-phases.md) | +| Task list (18 entries, 9.1–9.17) | [Phase9_taskList.md](./Phase9_taskList.md) | +| Metric definitions | [09-data-collection-reference.md §5b](./09-data-collection-reference.md) | +| New classes: `MetricsRegistry`, `AppMetricGauges` | `include/xrpl/telemetry/MetricsRegistry.h` + `src/libxrpl/telemetry/MetricsRegistry.cpp` (pipeline, counters, histograms) and `src/xrpld/telemetry/AppMetricGauges.h/.cpp` (observable gauges) — **shipped** | +| New dashboards (4) | `fee-market`, `job-queue`, `peer-quality`, `validator-health` — **shipped** | +| Updated dashboards (2) | `node-health`, `rpc-performance` | +| Provisioned alert rules | `docker/telemetry/grafana/provisioning/alerting/rules.yaml` — 13 rules in 5 groups ([07 §7.6.2](./07-observability-backends.md)) | > **Task numbering**: `Phase9_taskList.md` carries 18 `## Task 9.x` headings — > 9.1 through 9.17 plus the inserted 9.7a (`push_metrics.py` parity). The "10 diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md index ed69c1d5ad..33367cfece 100644 --- a/OpenTelemetryPlan/09-data-collection-reference.md +++ b/OpenTelemetryPlan/09-data-collection-reference.md @@ -805,7 +805,7 @@ reader, and OTLP/HTTP exporter, even though both request a meter named `xrpld` / `1.0.0`. `OTelCollector` takes its meter from the **global** provider, which `Telemetry` publishes and reads every 1000 ms; `MetricsRegistry` builds a private provider it does not publish, read every 10000 ms -(`src/xrpld/telemetry/MetricsRegistry.cpp`). So `jobq__*` and +(`src/libxrpl/telemetry/MetricsRegistry.cpp`). So `jobq__*` and `job_*_total` reach Prometheus on different cadences and should not be assumed sampled at the same instant. @@ -1066,8 +1066,8 @@ async callbacks for new categories. > **Label values are case-sensitive and three cache values are not lowercase.** > The `metric` label carries the string literal passed to `Observe()`, verbatim: > `SLE_hit_rate`, `AL_hit_rate` and `AL_size` are upper-case -> (`src/xrpld/telemetry/MetricsRegistry.cpp:666`, `:682`, `:708`), while -> `ledger_hit_rate` genuinely is lowercase (`:675`). A selector written as +> (all four `Observe()` calls are in `AppMetricGauges::registerCacheHitRateGauge()`), +> while `ledger_hit_rate` genuinely is lowercase. A selector written as > `cache_metrics{metric="sle_hit_rate"}` matches nothing. #### Server Info (via OTel MetricsRegistry) @@ -1219,7 +1219,7 @@ Phase 10 builds a 5-node validator docker-compose harness with RPC load generato > (`nodestore_state`, `cache_metrics`, …) once. > > Note that `ledgers_closed_total` appears in **both** instrument rows: it is -> created as a `MetricsRegistry` member (`MetricsRegistry.cpp:386-387`, whose +> created as a `MetricsRegistry` member (in `MetricsRegistry::initSyncInstruments()`, whose > `incrementLedgersClosed()` has no callers) and separately incremented at its > call site via `XRPL_METRIC_COUNTER_INC` (`RCLConsensus.cpp:749`). The distinct > name count across the two rows is therefore 41, not 42. @@ -1307,9 +1307,13 @@ Phase 11 builds a custom OTel Collector receiver (Go) that polls xrpld's admin R ### Phase 9: OTel SDK-Exported Metrics (MetricsRegistry) -Phase 9 introduces the `MetricsRegistry` class (`src/xrpld/telemetry/MetricsRegistry.h/.cpp`) -which registers metrics directly with the OpenTelemetry Metrics SDK. These are exported -via OTLP/HTTP to the OTel Collector and scraped by Prometheus. +Phase 9 introduces the `MetricsRegistry` class (`include/xrpl/telemetry/MetricsRegistry.h`, +`src/libxrpl/telemetry/MetricsRegistry.cpp`) which registers metrics directly with the +OpenTelemetry Metrics SDK. The synchronous counters and histograms are created there. The +observable gauges in the tables below are registered by `AppMetricGauges` +(`src/xrpld/telemetry/AppMetricGauges.h`, `src/xrpld/telemetry/AppMetricGauges.cpp`), which +stays in `xrpld` because its callbacks read `Application`. Both are exported via OTLP/HTTP +to the OTel Collector and scraped by Prometheus. #### NodeStore I/O (Observable Gauge — `nodestore_state`) @@ -1349,9 +1353,9 @@ via OTLP/HTTP to the OTel Collector and scraped by Prometheus. Further label values on the same instrument, added to separate the two bottlenecks that both present as the `ledgerData` job lane pinned at its -concurrency cap. Observed in `MetricsRegistry::observeNodeStoreTotals()`, +concurrency cap. Observed in `AppMetricGauges::observeNodeStoreTotals()`, `observeWritePathDetail()`, and `observeAcquireStats()` -(`src/xrpld/telemetry/MetricsRegistry.cpp:871-942`). +(`src/xrpld/telemetry/AppMetricGauges.cpp`). | Prometheus Metric | Type | Labels | Description | | ---------------------------------------------------- | ----- | -------- | ------------------------------------------------------- | @@ -1443,7 +1447,7 @@ data as uninformative unless the build is known to include the fix. #### TxQ Admission and Ledger Mismatch (Synchronous Counters) Three monotonic counters created alongside the Phase 7+ parity counters -(`src/xrpld/telemetry/MetricsRegistry.cpp:394-399`). The gauges above answer +(in `MetricsRegistry::initSyncInstruments()`). The gauges above answer "how deep is the queue"; these answer "what did the queue refuse, and did the ledger we built match the one the network validated". @@ -1489,7 +1493,7 @@ Rejections (Dropped)", "Queue Abandonment Rate (Expired)"; _Consensus Health_ #### Reduce-Relay Efficiency (Observable Gauge — `reduce_relay_metrics`) Transaction reduce-relay effectiveness, read from `Overlay::txMetrics()` each -collection cycle (`src/xrpld/telemetry/MetricsRegistry.cpp:1370-1402`). A high +collection cycle (`AppMetricGauges::registerReduceRelayGauge()`). A high `suppressed_peers` : `selected_peers` ratio proves the feature is saving bandwidth; a high `not_enabled_peers` means stale peers are forcing full relay. @@ -1518,7 +1522,7 @@ Selection", "Reduce-Relay Missing-Tx Frequency". | `rpc_in_flight_requests` | UpDownCounter | (none) | RPC calls currently executing (+1 rpcStart, -1 rpcEnd) | `rpc_in_flight_requests` is emitted at its call site via the `XRPL_METRIC_UPDOWN_ADD` -macro (see `src/xrpld/telemetry/MetricMacros.h` and `PerfLogImp.cpp`), not through a +macro (see `include/xrpl/telemetry/MetricMacros.h` and `PerfLogImp.cpp`), not through a `MetricsRegistry` member. As an UpDownCounter it carries no `_total` suffix (that is reserved for monotonic counters). @@ -1528,7 +1532,7 @@ Two histograms describing how much work one request asks for. Names and descriptions are the `constexpr` constants in `include/xrpl/telemetry/RpcMetricNames.h`; both are recorded at their call sites via `XRPL_METRIC_*`, and both have an explicit-bucket view registered in -`src/xrpld/telemetry/MetricsRegistry.cpp`. +`src/libxrpl/telemetry/MetricsRegistry.cpp`. | Prometheus Metric | Type | Labels | Description | | --------------------------- | --------- | ------ | ------------------------------------------------------- | @@ -1649,7 +1653,7 @@ information the batch totals do not already carry. **All three histograms need an explicit bucket view.** The SDK's default histogram boundaries top out at 10000. Every one of these three exceeds that, so without a view their top quantiles would all read as a flat 10000. Eight views are -registered in `src/xrpld/telemetry/MetricsRegistry.cpp`, and three of the eight are +registered in `src/libxrpl/telemetry/MetricsRegistry.cpp`, and three of the eight are for this family: | Instrument | View helper | Boundaries | @@ -1701,7 +1705,7 @@ not a lowercase word and not a friendly alias. The value is `beast::typeName()` (`include/xrpl/basics/CountedObject.h:115`), which demangles `typeid(T).name()` with `abi::__cxa_demangle` (`include/xrpl/beast/type_name.h:16-45`) and applies no stripping; the observer -copies it through verbatim (`src/xrpld/telemetry/MetricsRegistry.cpp:781-787`). +copies it through verbatim (`AppMetricGauges::registerObjectCountGauge()`). Values therefore keep their `xrpl::` namespace, nested `::`, and template arguments. @@ -1836,16 +1840,17 @@ These metrics fill gaps identified by comparing xrpld's internal observability w Data source: `ValidationTracker` class with 8s grace period and 5m late repair window. > **Every value on this instrument is a double.** The family is one -> `CreateDoubleObservableGauge` (`src/xrpld/telemetry/MetricsRegistry.cpp:1593`), +> `CreateDoubleObservableGauge` (in `AppMetricGauges::registerValidationAgreementGauge()`), > so the integral counts are cast to `double` before `Observe()` — there is no > Int64 sub-series to filter on. The same holds for `validator_health`, > `peer_quality` and `state_tracking` below; an earlier revision of these four > tables split the Type column between Int64 and Double, which the code does not > do. > -> The 7-day window is `ValidationTracker::kWindow7d` = 168 hours -> (`src/xrpld/telemetry/ValidationTracker.h:311`) and is observed alongside the 1h -> and 24h windows at `MetricsRegistry.cpp:1623-1626`. Panels exist on _Validator +> The 7-day window spans `ValidationTracker::kBuckets7d` = `7 * 24 * 60` one-minute +> buckets, i.e. 168 hours (`include/xrpl/telemetry/ValidationTracker.h`), and is +> observed alongside the 1h and 24h windows in +> `AppMetricGauges::registerValidationAgreementGauge()`. Panels exist on _Validator > Health_ (`validator-health`): "Agreement % (7d)" and "Agreements vs Missed > (7d)". @@ -1858,7 +1863,7 @@ Data source: `ValidationTracker` class with 8s grace period and 5m late repair w | `validator_health{metric="unl_expiry_days"}` | Double | `metric` | Days until UNL list expires | | `validator_health{metric="validation_quorum"}` | Double | `metric` | Validation quorum threshold | -Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1217`. +Single `CreateDoubleObservableGauge`, in `AppMetricGauges::registerValidatorHealthGauge()`. #### Peer Quality (Observable Gauge — `peer_quality`) @@ -1869,7 +1874,7 @@ Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1217`. | `peer_quality{metric="peers_higher_version_pct"}` | Double | `metric` | % of peers on newer xrpld version | | `peer_quality{metric="upgrade_recommended"}` | Double | `metric` | 1 if >60% of peers are newer version | -Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1266`. +Single `CreateDoubleObservableGauge`, in `AppMetricGauges::registerPeerQualityGauge()`. #### Ledger Economy (Observable Gauge — `ledger_economy`) @@ -1888,9 +1893,9 @@ Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1266`. | `state_tracking{metric="state_value"}` | Double | `metric` | Numeric state 0-6 (see encoding below) | | `state_tracking{metric="time_in_current_state_seconds"}` | Double | `metric` | Duration in current state | -Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1483`. +Single `CreateDoubleObservableGauge`, in `AppMetricGauges::registerStateTrackingGauge()`. -State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full, 5=validating (FULL + validating), 6=proposing (FULL + proposing). Values 0-4 are `OperatingMode` cast to double (`include/xrpl/server/NetworkOPs.h:60-66`); 5 and 6 are the FULL-only refinements at `MetricsRegistry.cpp:1500-1515`. **The range is 0-6, not 0-7** — there is no seventh state. +State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full, 5=validating (FULL + validating), 6=proposing (FULL + proposing). Values 0-4 are `OperatingMode` cast to double (`include/xrpl/server/NetworkOPs.h:60-66`); 5 and 6 are the FULL-only refinements in `AppMetricGauges::registerStateTrackingGauge()`. **The range is 0-6, not 0-7** — there is no seventh state. #### Storage Detail (Observable Gauge — `storage_detail`) @@ -1899,11 +1904,11 @@ State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full | `storage_detail{metric="stored_object_bytes"}` | Int64 | `metric` | Cumulative object-payload bytes written (not on-disk size) | > **`stored_object_bytes` is not a file size.** It observes `getStoreSize()` -> (`src/xrpld/telemetry/MetricsRegistry.cpp:1574`), which sums the object payloads +> (in `AppMetricGauges::registerStorageDetailGauge()`), which sums the object payloads > this process has written. It therefore excludes NuDB's keys, bucket padding and > log, and it resets when the process restarts while the files on disk do not. > `node_written_bytes` on the `nodestore_state` gauge calls the same accessor -> (`MetricsRegistry.cpp:877`), so the two series are equal by construction and any +> (in `AppMetricGauges::observeNodeStoreTotals()`), so the two series are equal by construction and any > write-amplification ratio built from the pair is a constant 1.0. To size the store > on disk, stat the backend's files; no metric reports it today. > @@ -1922,12 +1927,11 @@ State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full | `state_changes_total` | Counter | Operating mode transitions | NetworkOPs.cpp | > **Known issue — `ledgers_closed_total` has a dead second producer.** The -> instrument is created twice. `MetricsRegistry::registerCounters()` eagerly +> instrument is created twice. `MetricsRegistry::initSyncInstruments()` eagerly > creates it as the member `ledgersClosedCounter_` -> (`src/xrpld/telemetry/MetricsRegistry.cpp:386-387`), and its only mutator, -> `MetricsRegistry::incrementLedgersClosed()` -> (declared `MetricsRegistry.h:591`, defined `MetricsRegistry.cpp:1703`), has -> **zero callers** — the header says so itself at `MetricsRegistry.h:584-588`. +> (`src/libxrpl/telemetry/MetricsRegistry.cpp`), and its only mutator, +> `MetricsRegistry::incrementLedgersClosed()`, has **zero callers** — the `@note` +> on its declaration in `include/xrpl/telemetry/MetricsRegistry.h` says so itself. > The value operators actually see comes from the single live increment, > the `XRPL_METRIC_COUNTER_INC` call site in > `RCLConsensus::Adaptor::doAccept()` (`src/xrpld/app/consensus/RCLConsensus.cpp:749`). @@ -1974,15 +1978,15 @@ The dotted form was dropped by the 2026-05-13 naming redesign, in three commits: What the code emits today, and where it is documented: -| Old dotted key (never emitted) | Live equivalent | -| ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `xrpl.peer.version` | `peer_version` — see [§Transaction Attributes](#transaction-attributes) | -| `xrpl.validation.ledger_hash`, `xrpl.peer.validation.ledger_hash` | one bare `ledger_hash` on both `consensus.validation.send` and `peer.validation.receive` | -| `xrpl.validation.full`, `xrpl.peer.validation.full` | one bare `full_validation` on both of those spans | -| `xrpl.consensus.validation_quorum` | `quorum`, on `consensus.accept` only | -| `xrpl.node.amendment_blocked` | **not a span attribute at all** — only the metric `validator_health{metric="amendment_blocked"}` (`MetricsRegistry.cpp:1233`) | -| `xrpl.node.server_state` | **not a span attribute at all** — only the metric `server_info{metric="server_state"}` (`MetricsRegistry.cpp:1031`) | -| `xrpl.consensus.proposers_validated` | **never implemented** in any form | +| Old dotted key (never emitted) | Live equivalent | +| ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `xrpl.peer.version` | `peer_version` — see [§Transaction Attributes](#transaction-attributes) | +| `xrpl.validation.ledger_hash`, `xrpl.peer.validation.ledger_hash` | one bare `ledger_hash` on both `consensus.validation.send` and `peer.validation.receive` | +| `xrpl.validation.full`, `xrpl.peer.validation.full` | one bare `full_validation` on both of those spans | +| `xrpl.consensus.validation_quorum` | `quorum`, on `consensus.accept` only | +| `xrpl.node.amendment_blocked` | **not a span attribute at all** — only the metric `validator_health{metric="amendment_blocked"}` (`AppMetricGauges::registerValidatorHealthGauge()`) | +| `xrpl.node.server_state` | **not a span attribute at all** — only the metric `server_info{metric="server_state"}` (`AppMetricGauges::registerServerInfoGauge()`) | +| `xrpl.consensus.proposers_validated` | **never implemented** in any form | The identical nine-row list was deleted from `docker/telemetry/workload/expected_spans.json` by commit `cb9fce6890` for the diff --git a/cmake/XrplCore.cmake b/cmake/XrplCore.cmake index 2458d0ec81..aeb815d160 100644 --- a/cmake/XrplCore.cmake +++ b/cmake/XrplCore.cmake @@ -229,12 +229,17 @@ target_link_libraries( # each module's headers: a module can only include xrpl/telemetry/ headers if # it links this target, and the target must already exist at that point. # -# Links xrpl.libxrpl.protocol PRIVATELY for sha512Half (digest.h) +# Links xrpl.libxrpl.protocol and xrpl.libxrpl.core PUBLICLY: ValidationTracker.h +# takes LedgerIndex and MetricMacros.h takes ServiceRegistry, both in interfaces. add_module(xrpl telemetry) target_link_libraries( xrpl.libxrpl.telemetry - PUBLIC xrpl.libxrpl.basics xrpl.libxrpl.beast xrpl.libxrpl.config - PRIVATE xrpl.libxrpl.protocol + PUBLIC + xrpl.libxrpl.basics + xrpl.libxrpl.beast + xrpl.libxrpl.config + xrpl.libxrpl.core + xrpl.libxrpl.protocol ) if(telemetry) # Telemetry owns both the trace and (as of the direct-metrics API) the diff --git a/docker/telemetry/grafana/dashboards/consensus-health.json b/docker/telemetry/grafana/dashboards/consensus-health.json index fbadb7f1fb..f66d65a993 100644 --- a/docker/telemetry/grafana/dashboards/consensus-health.json +++ b/docker/telemetry/grafana/dashboards/consensus-health.json @@ -1187,7 +1187,7 @@ }, { "title": "Ledger History Mismatch Rate by Reason", - "description": "###### What this is:\n*Rate of built-versus-validated ledger mismatches, broken down by why they diverged.*\n\n###### How it's computed:\n*Per-second rate of mismatch events grouped by reason, per node, over a 5-minute window.*\n\n###### Reading it:\n*Flat at zero is healthy; the reason label tells you the nature of any divergence.*\n\n###### Healthy range:\n*Zero under normal operation.*\n\n###### Watch for:\n*Any sustained rate marks a fork; the reason distinguishes close-time disagreement, sync drift, and transaction-processing differences.*\n\n###### Keywords:\n- **Ledger history mismatch** *(per node)* \u2014 when a locally built ledger's hash does not match the network-validated hash.\n- **Fork** *(network-wide)* \u2014 when nodes validate divergent ledger chains instead of a single agreed history.\n- **Close time** *(network event)* \u2014 the timestamp validators agree to stamp on a ledger, rounded to a shared resolution.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementLedgerHistoryMismatch`\n\n###### References:\n[Fork](https://xrpl.org/docs/concepts/consensus-protocol) \u00b7 [Close time](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-history-mismatch)", + "description": "###### What this is:\n*Rate of built-versus-validated ledger mismatches, broken down by why they diverged.*\n\n###### How it's computed:\n*Per-second rate of mismatch events grouped by reason, per node, over a 5-minute window.*\n\n###### Reading it:\n*Flat at zero is healthy; the reason label tells you the nature of any divergence.*\n\n###### Healthy range:\n*Zero under normal operation.*\n\n###### Watch for:\n*Any sustained rate marks a fork; the reason distinguishes close-time disagreement, sync drift, and transaction-processing differences.*\n\n###### Keywords:\n- **Ledger history mismatch** *(per node)* \u2014 when a locally built ledger's hash does not match the network-validated hash.\n- **Fork** *(network-wide)* \u2014 when nodes validate divergent ledger chains instead of a single agreed history.\n- **Close time** *(network event)* \u2014 the timestamp validators agree to stamp on a ledger, rounded to a shared resolution.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementLedgerHistoryMismatch`\n\n###### References:\n[Fork](https://xrpl.org/docs/concepts/consensus-protocol) \u00b7 [Close time](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-history-mismatch)", "type": "timeseries", "gridPos": { "h": 10, diff --git a/docker/telemetry/grafana/dashboards/fee-market.json b/docker/telemetry/grafana/dashboards/fee-market.json index 8faf0acc7c..bc90c3e7a7 100644 --- a/docker/telemetry/grafana/dashboards/fee-market.json +++ b/docker/telemetry/grafana/dashboards/fee-market.json @@ -71,7 +71,7 @@ }, { "title": "Transaction Queue Depth", - "description": "###### What this is:\n*Transactions currently waiting in the transaction queue versus the queue's maximum capacity.*\n\n###### How it's computed:\n*Instantaneous gauge readings of current queue count and configured max size.*\n\n###### Reading it:\n*Queue depth well below capacity is normal; depth approaching capacity means the node is saturating.*\n\n###### Healthy range:\n*Depth near 0 in quiet periods; workload-dependent under load.*\n\n###### Watch for:\n*Depth pinned at capacity for sustained periods, which signals demand exceeding throughput or a fee-spam burst.*\n\n###### Keywords:\n- **Transaction queue (TxQ)** *(per node)* \u2014 holds transactions that meet local cost but not the open-ledger cost, to include in a later ledger.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerTxqGauge`\n\n###### References:\n[Transaction queue (TxQ)](https://xrpl.org/docs/concepts/transactions/transaction-queue) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#transaction-queue-txq)", + "description": "###### What this is:\n*Transactions currently waiting in the transaction queue versus the queue's maximum capacity.*\n\n###### How it's computed:\n*Instantaneous gauge readings of current queue count and configured max size.*\n\n###### Reading it:\n*Queue depth well below capacity is normal; depth approaching capacity means the node is saturating.*\n\n###### Healthy range:\n*Depth near 0 in quiet periods; workload-dependent under load.*\n\n###### Watch for:\n*Depth pinned at capacity for sustained periods, which signals demand exceeding throughput or a fee-spam burst.*\n\n###### Keywords:\n- **Transaction queue (TxQ)** *(per node)* \u2014 holds transactions that meet local cost but not the open-ledger cost, to include in a later ledger.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerTxqGauge`\n\n###### References:\n[Transaction queue (TxQ)](https://xrpl.org/docs/concepts/transactions/transaction-queue) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#transaction-queue-txq)", "type": "timeseries", "gridPos": { "h": 10, @@ -126,7 +126,7 @@ }, { "title": "Transactions Per Ledger", - "description": "###### What this is:\n*Transactions already placed in the current open ledger versus the expected per-ledger target.*\n\n###### How it's computed:\n*Instantaneous gauge readings of in-ledger count and the target count that governs fee escalation.*\n\n###### Reading it:\n*Staying at or below the expected target is normal; exceeding it triggers open-ledger fee escalation.*\n\n###### Healthy range:\n*At or under the expected per-ledger target.*\n\n###### Watch for:\n*In-ledger count persistently above target, indicating sustained congestion pushing fees up.*\n\n###### Keywords:\n- **Open ledger** *(per node)* \u2014 the temporary workspace ledger where incoming transactions are provisionally applied before a close.\n- **In-ledger vs target count** *(per node)* \u2014 transactions already in the open ledger versus the soft target that triggers fee escalation.\n- **Fee escalation** *(per node)* \u2014 the exponential rise in the open-ledger cost once the ledger exceeds its soft transaction target.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerTxqGauge`\n\n###### References:\n[Open ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [In-ledger vs target count](https://xrpl.org/docs/concepts/transactions/transaction-cost#open-ledger-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#open-ledger)", + "description": "###### What this is:\n*Transactions already placed in the current open ledger versus the expected per-ledger target.*\n\n###### How it's computed:\n*Instantaneous gauge readings of in-ledger count and the target count that governs fee escalation.*\n\n###### Reading it:\n*Staying at or below the expected target is normal; exceeding it triggers open-ledger fee escalation.*\n\n###### Healthy range:\n*At or under the expected per-ledger target.*\n\n###### Watch for:\n*In-ledger count persistently above target, indicating sustained congestion pushing fees up.*\n\n###### Keywords:\n- **Open ledger** *(per node)* \u2014 the temporary workspace ledger where incoming transactions are provisionally applied before a close.\n- **In-ledger vs target count** *(per node)* \u2014 transactions already in the open ledger versus the soft target that triggers fee escalation.\n- **Fee escalation** *(per node)* \u2014 the exponential rise in the open-ledger cost once the ledger exceeds its soft transaction target.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerTxqGauge`\n\n###### References:\n[Open ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [In-ledger vs target count](https://xrpl.org/docs/concepts/transactions/transaction-cost#open-ledger-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#open-ledger)", "type": "timeseries", "gridPos": { "h": 10, @@ -193,7 +193,7 @@ }, { "title": "Fee Escalation Levels", - "description": "###### What this is:\n*The fee levels that govern queue admission: reference (baseline), minimum processing, median, and open-ledger levels.*\n\n###### How it's computed:\n*Instantaneous gauge readings of each fee level, shown on a log scale.*\n\n###### Reading it:\n*Open-ledger level near the reference level means cheap entry; a large gap above reference means escalation is active.*\n\n###### Healthy range:\n*Open-ledger level at or near reference during normal traffic.*\n\n###### Watch for:\n*Open-ledger level spiking far above reference, the hallmark of congestion or a fee-bidding war.*\n\n###### Keywords:\n- **Queue admission rejection** *(per node)* \u2014 a transaction refused entry to the queue, e.g. queue_full when the queue is at capacity.\n- **Fee levels** *(per node)* \u2014 cost thresholds governing queue admission: reference (baseline), minimum, median, and open-ledger.\n- **Open ledger** *(per node)* \u2014 the temporary workspace ledger where incoming transactions are provisionally applied before a close.\n- **Fee escalation** *(per node)* \u2014 the exponential rise in the open-ledger cost once the ledger exceeds its soft transaction target.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerTxqGauge`\n\n###### References:\n[Queue admission rejection](https://xrpl.org/docs/concepts/transactions/transaction-queue) \u00b7 [Fee levels](https://xrpl.org/docs/concepts/transactions/transaction-cost#fee-levels) \u00b7 [Open ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Fee escalation](https://xrpl.org/docs/concepts/transactions/transaction-cost#open-ledger-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#queue-admission-rejection)", + "description": "###### What this is:\n*The fee levels that govern queue admission: reference (baseline), minimum processing, median, and open-ledger levels.*\n\n###### How it's computed:\n*Instantaneous gauge readings of each fee level, shown on a log scale.*\n\n###### Reading it:\n*Open-ledger level near the reference level means cheap entry; a large gap above reference means escalation is active.*\n\n###### Healthy range:\n*Open-ledger level at or near reference during normal traffic.*\n\n###### Watch for:\n*Open-ledger level spiking far above reference, the hallmark of congestion or a fee-bidding war.*\n\n###### Keywords:\n- **Queue admission rejection** *(per node)* \u2014 a transaction refused entry to the queue, e.g. queue_full when the queue is at capacity.\n- **Fee levels** *(per node)* \u2014 cost thresholds governing queue admission: reference (baseline), minimum, median, and open-ledger.\n- **Open ledger** *(per node)* \u2014 the temporary workspace ledger where incoming transactions are provisionally applied before a close.\n- **Fee escalation** *(per node)* \u2014 the exponential rise in the open-ledger cost once the ledger exceeds its soft transaction target.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerTxqGauge`\n\n###### References:\n[Queue admission rejection](https://xrpl.org/docs/concepts/transactions/transaction-queue) \u00b7 [Fee levels](https://xrpl.org/docs/concepts/transactions/transaction-cost#fee-levels) \u00b7 [Open ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Fee escalation](https://xrpl.org/docs/concepts/transactions/transaction-cost#open-ledger-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#queue-admission-rejection)", "type": "timeseries", "gridPos": { "h": 10, @@ -265,7 +265,7 @@ }, { "title": "Load Factor Breakdown", - "description": "###### What this is:\n*The combined load factor and its server, fee-escalation, and fee-queue contributors as unitless fee multipliers (1.0 = no load).*\n\n###### How it's computed:\n*Instantaneous gauge readings of each load-factor component.*\n\n###### Reading it:\n*Values at 1.0 mean base fees; higher values raise the fee to transact.*\n\n###### Healthy range:\n*Around 1.0 under normal conditions.*\n\n###### Watch for:\n*Combined factor climbing well above 1.0, showing the node is charging premium fees due to congestion or overload.*\n\n###### Keywords:\n- **Load factor** *(per node)* \u2014 a unitless multiplier (1.0 = no load) that scales the base transaction cost as the node comes under load.\n- **Fee escalation** *(per node)* \u2014 the exponential rise in the open-ledger cost once the ledger exceeds its soft transaction target.\n- **Transaction cost** *(network-wide)* \u2014 the XRP a transaction destroys to be processed; scales up with load to deter spam.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerLoadFactorGauge`\n\n###### References:\n[Load factor](https://xrpl.org/docs/concepts/transactions/transaction-cost#local-load-cost) \u00b7 [Fee escalation](https://xrpl.org/docs/concepts/transactions/transaction-cost#open-ledger-cost) \u00b7 [Transaction cost](https://xrpl.org/docs/concepts/transactions/transaction-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#load-factor)", + "description": "###### What this is:\n*The combined load factor and its server, fee-escalation, and fee-queue contributors as unitless fee multipliers (1.0 = no load).*\n\n###### How it's computed:\n*Instantaneous gauge readings of each load-factor component.*\n\n###### Reading it:\n*Values at 1.0 mean base fees; higher values raise the fee to transact.*\n\n###### Healthy range:\n*Around 1.0 under normal conditions.*\n\n###### Watch for:\n*Combined factor climbing well above 1.0, showing the node is charging premium fees due to congestion or overload.*\n\n###### Keywords:\n- **Load factor** *(per node)* \u2014 a unitless multiplier (1.0 = no load) that scales the base transaction cost as the node comes under load.\n- **Fee escalation** *(per node)* \u2014 the exponential rise in the open-ledger cost once the ledger exceeds its soft transaction target.\n- **Transaction cost** *(network-wide)* \u2014 the XRP a transaction destroys to be processed; scales up with load to deter spam.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerLoadFactorGauge`\n\n###### References:\n[Load factor](https://xrpl.org/docs/concepts/transactions/transaction-cost#local-load-cost) \u00b7 [Fee escalation](https://xrpl.org/docs/concepts/transactions/transaction-cost#open-ledger-cost) \u00b7 [Transaction cost](https://xrpl.org/docs/concepts/transactions/transaction-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#load-factor)", "type": "timeseries", "gridPos": { "h": 10, @@ -349,7 +349,7 @@ }, { "title": "Load Factor Components", - "description": "###### What this is:\n*The individual load-factor inputs, local server load, network load, and cluster load, as unitless multipliers.*\n\n###### How it's computed:\n*Instantaneous gauge readings of each component.*\n\n###### Reading it:\n*All at 1.0 means no load pressure from any source; a raised component identifies where load originates.*\n\n###### Healthy range:\n*Around 1.0 for each component.*\n\n###### Watch for:\n*A single component rising sharply, which pinpoints whether the pressure is local, network-wide, or cluster-driven.*\n\n###### Keywords:\n- **Load factor** *(per node)* \u2014 a unitless multiplier (1.0 = no load) that scales the base transaction cost as the node comes under load.\n- **Cluster** *(cluster-wide)* \u2014 a group of trusted co-operated nodes that share load information and skip some verification.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerLoadFactorGauge`\n\n###### References:\n[Load factor](https://xrpl.org/docs/concepts/transactions/transaction-cost#local-load-cost) \u00b7 [Cluster](https://xrpl.org/docs/concepts/networks-and-servers/clustering) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#load-factor)", + "description": "###### What this is:\n*The individual load-factor inputs, local server load, network load, and cluster load, as unitless multipliers.*\n\n###### How it's computed:\n*Instantaneous gauge readings of each component.*\n\n###### Reading it:\n*All at 1.0 means no load pressure from any source; a raised component identifies where load originates.*\n\n###### Healthy range:\n*Around 1.0 for each component.*\n\n###### Watch for:\n*A single component rising sharply, which pinpoints whether the pressure is local, network-wide, or cluster-driven.*\n\n###### Keywords:\n- **Load factor** *(per node)* \u2014 a unitless multiplier (1.0 = no load) that scales the base transaction cost as the node comes under load.\n- **Cluster** *(cluster-wide)* \u2014 a group of trusted co-operated nodes that share load information and skip some verification.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerLoadFactorGauge`\n\n###### References:\n[Load factor](https://xrpl.org/docs/concepts/transactions/transaction-cost#local-load-cost) \u00b7 [Cluster](https://xrpl.org/docs/concepts/networks-and-servers/clustering) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#load-factor)", "type": "timeseries", "gridPos": { "h": 10, @@ -423,7 +423,7 @@ }, { "title": "Queue Abandonment Rate (Expired)", - "description": "###### What this is:\n*Transactions dropped from the queue because their last-ledger deadline passed before they could be included.*\n\n###### How it's computed:\n*Per-second rate of the cumulative expired-transaction counter over the dashboard's rate interval.*\n\n###### Reading it:\n*Near zero is healthy; a rising rate means submitters under-bid the escalating fee and their transactions timed out.*\n\n###### Healthy range:\n*Near 0 expirations per second.*\n\n###### Watch for:\n*Sustained expiry rate, a demand-frustration signal often coinciding with fee spikes or spam that crowds out honest traffic.*\n\n###### Keywords:\n- **Queue admission rejection** *(per node)* \u2014 a transaction refused entry to the queue, e.g. queue_full when the queue is at capacity.\n- **Queue expiry / abandonment** *(per node)* \u2014 removing a queued transaction whose LastLedgerSequence deadline passed before inclusion.\n- **Fee escalation** *(per node)* \u2014 the exponential rise in the open-ledger cost once the ledger exceeds its soft transaction target.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementTxqExpired (caller TxQ.cpp)`\n\n###### References:\n[Queue admission rejection](https://xrpl.org/docs/concepts/transactions/transaction-queue) \u00b7 [Queue expiry / abandonment](https://xrpl.org/docs/concepts/transactions/reliable-transaction-submission) \u00b7 [Fee escalation](https://xrpl.org/docs/concepts/transactions/transaction-cost#open-ledger-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#queue-admission-rejection)", + "description": "###### What this is:\n*Transactions dropped from the queue because their last-ledger deadline passed before they could be included.*\n\n###### How it's computed:\n*Per-second rate of the cumulative expired-transaction counter over the dashboard's rate interval.*\n\n###### Reading it:\n*Near zero is healthy; a rising rate means submitters under-bid the escalating fee and their transactions timed out.*\n\n###### Healthy range:\n*Near 0 expirations per second.*\n\n###### Watch for:\n*Sustained expiry rate, a demand-frustration signal often coinciding with fee spikes or spam that crowds out honest traffic.*\n\n###### Keywords:\n- **Queue admission rejection** *(per node)* \u2014 a transaction refused entry to the queue, e.g. queue_full when the queue is at capacity.\n- **Queue expiry / abandonment** *(per node)* \u2014 removing a queued transaction whose LastLedgerSequence deadline passed before inclusion.\n- **Fee escalation** *(per node)* \u2014 the exponential rise in the open-ledger cost once the ledger exceeds its soft transaction target.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementTxqExpired (caller TxQ.cpp)`\n\n###### References:\n[Queue admission rejection](https://xrpl.org/docs/concepts/transactions/transaction-queue) \u00b7 [Queue expiry / abandonment](https://xrpl.org/docs/concepts/transactions/reliable-transaction-submission) \u00b7 [Fee escalation](https://xrpl.org/docs/concepts/transactions/transaction-cost#open-ledger-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#queue-admission-rejection)", "type": "timeseries", "gridPos": { "h": 10, @@ -468,7 +468,7 @@ }, { "title": "Queue Admission Rejections (Dropped)", - "description": "###### What this is:\n*Transactions refused entry to the queue, broken down by reason such as queue_full.*\n\n###### How it's computed:\n*Per-second rate of the cumulative dropped-transaction counter over the dashboard's rate interval, split by reason.*\n\n###### Reading it:\n*Near zero is healthy; queue_full rejections mean the queue is at capacity and applying backpressure.*\n\n###### Healthy range:\n*Near 0 rejections per second.*\n\n###### Watch for:\n*A burst of queue_full drops, distinct from expiry, indicating the node is being flooded faster than it can drain.*\n\n###### Keywords:\n- **Queue admission rejection** *(per node)* \u2014 a transaction refused entry to the queue, e.g. queue_full when the queue is at capacity.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementTxqDropped (caller TxQ.cpp)`\n\n###### References:\n[Queue admission rejection](https://xrpl.org/docs/concepts/transactions/transaction-queue) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#queue-admission-rejection)", + "description": "###### What this is:\n*Transactions refused entry to the queue, broken down by reason such as queue_full.*\n\n###### How it's computed:\n*Per-second rate of the cumulative dropped-transaction counter over the dashboard's rate interval, split by reason.*\n\n###### Reading it:\n*Near zero is healthy; queue_full rejections mean the queue is at capacity and applying backpressure.*\n\n###### Healthy range:\n*Near 0 rejections per second.*\n\n###### Watch for:\n*A burst of queue_full drops, distinct from expiry, indicating the node is being flooded faster than it can drain.*\n\n###### Keywords:\n- **Queue admission rejection** *(per node)* \u2014 a transaction refused entry to the queue, e.g. queue_full when the queue is at capacity.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementTxqDropped (caller TxQ.cpp)`\n\n###### References:\n[Queue admission rejection](https://xrpl.org/docs/concepts/transactions/transaction-queue) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#queue-admission-rejection)", "type": "timeseries", "gridPos": { "h": 10, diff --git a/docker/telemetry/grafana/dashboards/job-queue.json b/docker/telemetry/grafana/dashboards/job-queue.json index d7baefa1b1..9273c3eb68 100644 --- a/docker/telemetry/grafana/dashboards/job-queue.json +++ b/docker/telemetry/grafana/dashboards/job-queue.json @@ -71,7 +71,7 @@ }, { "title": "Current Job Latency (p99 Gauge) [$xrpl_network_type]", - "description": "###### What this is:\n*At-a-glance p99 of how long jobs wait in the queue and how long they run once started.*\n\n###### How it's computed:\n*99th percentile derived from the job wait-time and run-time histograms over the last 5 minutes.*\n\n###### Reading it:\n*Lower is better; green under 100ms, yellow to 1s, red beyond 1s.*\n\n###### Healthy range:\n*Wait and exec p99 under 100ms.*\n\n###### Watch for:\n*p99 wait climbing into the red, meaning worker threads are saturated and jobs are backing up.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Deferred job** *(per node)* \u2014 a job held back because its type is already at its concurrency limit; the leading indicator of queue backpressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobStarted / recordJobFinished`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", + "description": "###### What this is:\n*At-a-glance p99 of how long jobs wait in the queue and how long they run once started.*\n\n###### How it's computed:\n*99th percentile derived from the job wait-time and run-time histograms over the last 5 minutes.*\n\n###### Reading it:\n*Lower is better; green under 100ms, yellow to 1s, red beyond 1s.*\n\n###### Healthy range:\n*Wait and exec p99 under 100ms.*\n\n###### Watch for:\n*p99 wait climbing into the red, meaning worker threads are saturated and jobs are backing up.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Deferred job** *(per node)* \u2014 a job held back because its type is already at its concurrency limit; the leading indicator of queue backpressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobStarted / recordJobFinished`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", "type": "gauge", "gridPos": { "h": 10, @@ -148,7 +148,7 @@ }, { "title": "Job Throughput Rate (Per Second)", - "description": "###### What this is:\n*Rate of jobs queued, started, and finished across all job types.*\n\n###### How it's computed:\n*Per-second rate of each cumulative job counter over a 5-minute window.*\n\n###### Reading it:\n*Queued, started, and finished tracking together means the queue keeps up.*\n\n###### Healthy range:\n*Workload-dependent; the three rates should stay roughly equal.*\n\n###### Watch for:\n*Queued rate persistently above finished rate, which indicates a growing backlog.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Deferred job** *(per node)* \u2014 a job held back because its type is already at its concurrency limit; the leading indicator of queue backpressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobQueued / recordJobStarted / recordJobFinished`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", + "description": "###### What this is:\n*Rate of jobs queued, started, and finished across all job types.*\n\n###### How it's computed:\n*Per-second rate of each cumulative job counter over a 5-minute window.*\n\n###### Reading it:\n*Queued, started, and finished tracking together means the queue keeps up.*\n\n###### Healthy range:\n*Workload-dependent; the three rates should stay roughly equal.*\n\n###### Watch for:\n*Queued rate persistently above finished rate, which indicates a growing backlog.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Deferred job** *(per node)* \u2014 a job held back because its type is already at its concurrency limit; the leading indicator of queue backpressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobQueued / recordJobStarted / recordJobFinished`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", "type": "timeseries", "gridPos": { "h": 10, @@ -208,7 +208,7 @@ }, { "title": "Per-Job-Type Queued Rate", - "description": "###### What this is:\n*Rate of jobs entering the queue, broken down by job type (top 10).*\n\n###### How it's computed:\n*Per-second rate of the queued-job counter per job_type over a 5-minute window.*\n\n###### Reading it:\n*Identifies which job types generate the most queue activity.*\n\n###### Healthy range:\n*Workload-dependent.*\n\n###### Watch for:\n*A single job type surging unexpectedly, which can point to a flood of a particular request or peer message.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Deferred job** *(per node)* \u2014 a job held back because its type is already at its concurrency limit; the leading indicator of queue backpressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobQueued`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", + "description": "###### What this is:\n*Rate of jobs entering the queue, broken down by job type (top 10).*\n\n###### How it's computed:\n*Per-second rate of the queued-job counter per job_type over a 5-minute window.*\n\n###### Reading it:\n*Identifies which job types generate the most queue activity.*\n\n###### Healthy range:\n*Workload-dependent.*\n\n###### Watch for:\n*A single job type surging unexpectedly, which can point to a flood of a particular request or peer message.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Deferred job** *(per node)* \u2014 a job held back because its type is already at its concurrency limit; the leading indicator of queue backpressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobQueued`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", "type": "timeseries", "gridPos": { "h": 10, @@ -261,7 +261,7 @@ }, { "title": "Per-Job-Type Finish Rate", - "description": "###### What this is:\n*Rate of jobs completing, broken down by job type (top 10).*\n\n###### How it's computed:\n*Per-second rate of the finished-job counter per job_type over a 5-minute window.*\n\n###### Reading it:\n*Compare against the queued rate per type to spot which types are falling behind.*\n\n###### Healthy range:\n*Workload-dependent; should match the per-type queued rate.*\n\n###### Watch for:\n*A type whose finish rate lags its queued rate, revealing where the backlog concentrates.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Deferred job** *(per node)* \u2014 a job held back because its type is already at its concurrency limit; the leading indicator of queue backpressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobFinished`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", + "description": "###### What this is:\n*Rate of jobs completing, broken down by job type (top 10).*\n\n###### How it's computed:\n*Per-second rate of the finished-job counter per job_type over a 5-minute window.*\n\n###### Reading it:\n*Compare against the queued rate per type to spot which types are falling behind.*\n\n###### Healthy range:\n*Workload-dependent; should match the per-type queued rate.*\n\n###### Watch for:\n*A type whose finish rate lags its queued rate, revealing where the backlog concentrates.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Deferred job** *(per node)* \u2014 a job held back because its type is already at its concurrency limit; the leading indicator of queue backpressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobFinished`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", "type": "timeseries", "gridPos": { "h": 10, @@ -327,7 +327,7 @@ }, { "title": "Job Queue Wait Time", - "description": "###### What this is:\n*How long jobs sit in the queue before a worker picks them up, split by job type (p75 typical, p99 tail).*\n\n###### How it's computed:\n*Percentiles derived from the job wait-time histogram over a 5-minute window, kept per job type. Limited to the ten types with the highest wait so the legend stays readable.*\n\n###### Reading it:\n*Lower is better. The legend names the job type, so a single slow queue is identifiable rather than hidden in an all-types average. A widening p75-to-p99 gap on one type signals occasional stalls there.*\n\n###### Healthy range:\n*Sub-millisecond to low-millisecond waits on an unloaded node.*\n\n###### Watch for:\n*Rising p99 wait on a capped type -- ledgerRequest, ledgerData and makeFetchPack have small concurrency limits, so they queue first. Cross-check the deferred gauge for that type.*\n\n###### Keywords:\n- **Job queue** *(per node)* \u2014 the worker pool that runs xrpld's background work; each unit of work is a job with a type.\n- **Dequeue wait** *(per node)* \u2014 time a job sits enqueued before a worker starts it, as distinct from how long it then runs.\n- **Concurrency limit** *(per node)* \u2014 the maximum number of jobs of one type allowed to run at once; work beyond it is deferred, not rejected.\n\n###### Computation boundary:\n*Result: Per node and job type \u2014 each series is one server's own value for one job type.*\n*Recorded in code as an OTel SDK histogram, then aggregated to percentiles by the Grafana query.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobStarted`\n\n###### References:\n[Job queue](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type) \u00b7 [Concurrency limit](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#concurrency-limit)\n", + "description": "###### What this is:\n*How long jobs sit in the queue before a worker picks them up, split by job type (p75 typical, p99 tail).*\n\n###### How it's computed:\n*Percentiles derived from the job wait-time histogram over a 5-minute window, kept per job type. Limited to the ten types with the highest wait so the legend stays readable.*\n\n###### Reading it:\n*Lower is better. The legend names the job type, so a single slow queue is identifiable rather than hidden in an all-types average. A widening p75-to-p99 gap on one type signals occasional stalls there.*\n\n###### Healthy range:\n*Sub-millisecond to low-millisecond waits on an unloaded node.*\n\n###### Watch for:\n*Rising p99 wait on a capped type -- ledgerRequest, ledgerData and makeFetchPack have small concurrency limits, so they queue first. Cross-check the deferred gauge for that type.*\n\n###### Keywords:\n- **Job queue** *(per node)* \u2014 the worker pool that runs xrpld's background work; each unit of work is a job with a type.\n- **Dequeue wait** *(per node)* \u2014 time a job sits enqueued before a worker starts it, as distinct from how long it then runs.\n- **Concurrency limit** *(per node)* \u2014 the maximum number of jobs of one type allowed to run at once; work beyond it is deferred, not rejected.\n\n###### Computation boundary:\n*Result: Per node and job type \u2014 each series is one server's own value for one job type.*\n*Recorded in code as an OTel SDK histogram, then aggregated to percentiles by the Grafana query.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobStarted`\n\n###### References:\n[Job queue](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type) \u00b7 [Concurrency limit](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#concurrency-limit)\n", "type": "timeseries", "gridPos": { "h": 10, @@ -381,7 +381,7 @@ }, { "title": "Job Execution Time", - "description": "###### What this is:\n*How long jobs run once started, split by job type (p75 typical, p99 tail).*\n\n###### How it's computed:\n*Percentiles derived from the job run-time histogram over a 5-minute window, kept per job type. Limited to the ten slowest types so the legend stays readable.*\n\n###### Reading it:\n*Lower is better. The legend names the job type, so an expensive job type is identifiable rather than averaged away. Stable p75 with a controlled p99 is healthy.*\n\n###### Healthy range:\n*Workload-dependent, but stable over time.*\n\n###### Watch for:\n*Growing execution times, which point to CPU pressure or expensive individual jobs.*\n\n###### Keywords:\n- **Job queue** *(per node)* \u2014 the worker pool that runs xrpld's background work; each unit of work is a job with a type.\n- **Execution time** *(per node)* \u2014 time a job spends running after a worker picks it up, excluding its queue wait.\n\n###### Computation boundary:\n*Result: Per node and job type \u2014 each series is one server's own value for one job type.*\n*Recorded in code as an OTel SDK histogram, then aggregated to percentiles by the Grafana query.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobFinished`\n\n###### References:\n[Job queue](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)\n", + "description": "###### What this is:\n*How long jobs run once started, split by job type (p75 typical, p99 tail).*\n\n###### How it's computed:\n*Percentiles derived from the job run-time histogram over a 5-minute window, kept per job type. Limited to the ten slowest types so the legend stays readable.*\n\n###### Reading it:\n*Lower is better. The legend names the job type, so an expensive job type is identifiable rather than averaged away. Stable p75 with a controlled p99 is healthy.*\n\n###### Healthy range:\n*Workload-dependent, but stable over time.*\n\n###### Watch for:\n*Growing execution times, which point to CPU pressure or expensive individual jobs.*\n\n###### Keywords:\n- **Job queue** *(per node)* \u2014 the worker pool that runs xrpld's background work; each unit of work is a job with a type.\n- **Execution time** *(per node)* \u2014 time a job spends running after a worker picks it up, excluding its queue wait.\n\n###### Computation boundary:\n*Result: Per node and job type \u2014 each series is one server's own value for one job type.*\n*Recorded in code as an OTel SDK histogram, then aggregated to percentiles by the Grafana query.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobFinished`\n\n###### References:\n[Job queue](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)\n", "type": "timeseries", "gridPos": { "h": 10, @@ -435,7 +435,7 @@ }, { "title": "Per-Job-Type Execution Time (p99)", - "description": "###### What this is:\n*The 10 slowest job types ranked by p99 execution time.*\n\n###### How it's computed:\n*p99 derived from the run-time histogram per job_type over a 5-minute window, top 10 selected.*\n\n###### Reading it:\n*Highlights which job types cost the most CPU time at the tail.*\n\n###### Healthy range:\n*Workload-dependent.*\n\n###### Watch for:\n*A job type whose p99 grows over time, indicating a slow or degrading operation.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Deferred job** *(per node)* \u2014 a job held back because its type is already at its concurrency limit; the leading indicator of queue backpressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobFinished`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", + "description": "###### What this is:\n*The 10 slowest job types ranked by p99 execution time.*\n\n###### How it's computed:\n*p99 derived from the run-time histogram per job_type over a 5-minute window, top 10 selected.*\n\n###### Reading it:\n*Highlights which job types cost the most CPU time at the tail.*\n\n###### Healthy range:\n*Workload-dependent.*\n\n###### Watch for:\n*A job type whose p99 grows over time, indicating a slow or degrading operation.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Deferred job** *(per node)* \u2014 a job held back because its type is already at its concurrency limit; the leading indicator of queue backpressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordJobFinished`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", "type": "timeseries", "gridPos": { "h": 10, @@ -501,7 +501,7 @@ }, { "title": "Transaction Overflow Rate", - "description": "###### What this is:\n*Rate at which transaction jobs are shed when the queue's transaction limit is exceeded.*\n\n###### How it's computed:\n*Per-second rate of the overflow counter over the dashboard's rate interval, scaled to per minute. The counter is observed from the overlay's cumulative overflow tally.*\n\n###### Reading it:\n*Near zero is healthy; a rising rate means the job queue is shedding transaction work under load.*\n\n###### Healthy range:\n*0 overflows per minute.*\n\n###### Watch for:\n*Any sustained non-zero rate \u2014 the node is dropping transaction jobs because the queue is saturated.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Transaction queue (TxQ)** *(per node)* \u2014 holds transactions that cannot enter the open ledger yet, ordered by fee level.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerParityCounters (observed from Overlay::getJqTransOverflow)`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", + "description": "###### What this is:\n*Rate at which transaction jobs are shed when the queue's transaction limit is exceeded.*\n\n###### How it's computed:\n*Per-second rate of the overflow counter over the dashboard's rate interval, scaled to per minute. The counter is observed from the overlay's cumulative overflow tally.*\n\n###### Reading it:\n*Near zero is healthy; a rising rate means the job queue is shedding transaction work under load.*\n\n###### Healthy range:\n*0 overflows per minute.*\n\n###### Watch for:\n*Any sustained non-zero rate \u2014 the node is dropping transaction jobs because the queue is saturated.*\n\n###### Keywords:\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n- **Transaction queue (TxQ)** *(per node)* \u2014 holds transactions that cannot enter the open ledger yet, ordered by fee level.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerParityCounters (observed from Overlay::getJqTransOverflow)`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#job-queue-job-type)", "type": "timeseries", "gridPos": { "h": 10, diff --git a/docker/telemetry/grafana/dashboards/ledger-data-sync.json b/docker/telemetry/grafana/dashboards/ledger-data-sync.json index 5b2505ecfe..d14a8b88f0 100644 --- a/docker/telemetry/grafana/dashboards/ledger-data-sync.json +++ b/docker/telemetry/grafana/dashboards/ledger-data-sync.json @@ -1729,7 +1729,7 @@ }, { "title": "LedgerReq Wait by Handler", - "description": "###### What this is:\n*Queue wait for the ledgerRequest job type, split by which handler enqueued the job. The type has a concurrency limit of 3 and two producers that compete for those slots: RcvGetLedger, which serves TMGetLedger to syncing peers, and RcvGetObjByHash, which serves TMGetObjectByHash. Both report the same job_type, so without the handler split a wait spike cannot be attributed to either.*\n\n###### How it's computed:\n*p99 of job_queued_us for job_type=\"ledgerRequest\", grouped by the handler label. JobQueue::processTask measures the wait, then PerfLog hands it to MetricsRegistry::recordJobStarted, which is where the histogram is recorded. The handler value is the addJob name passed through a sanitizer that keeps letters-only names and folds everything else to \"other\", which bounds the label domain to 43 names plus \"other\". Both producers here are letters-only, so both appear under their own names; \"other\" is a mixed bucket and never means one specific caller.*\n\n###### Reading it:\n*This is the panel that answers which producer is starving the 3-slot queue. Both lines high together means the queue is genuinely oversubscribed and both kinds of peer request are being delayed. One line high while the other is flat means that producer is arriving faster than 3 concurrent slots can absorb, and it is the one delaying the other. Wait is queue time only, so a high line here is contention, not slow work; the work itself is on the GetObject Handler Latency Breakdown panel.*\n\n###### Healthy range:\n*Single-digit to low-tens of milliseconds p99 for both handlers, matching the wider Job Queue Wait p95 By Type panel.*\n\n###### Watch for:\n*RcvGetObjByHash wait climbing: one in-bounds TMGetObjectByHash request can perform thousands of NodeStore lookups, so a few concurrent ones occupy every slot and delay TMGetLedger to peers that are themselves syncing. Cross-check Job Queue Backlog and Deferred by Type for jobq_ledgerrequest_deferred above zero to confirm the limit, not the work, is the binding constraint.*\n\n###### Keywords:\n- **Handler label** *(per node)* \u2014 the addJob call-site name attached to job metrics, so producers sharing one job type stay separable.\n- **Concurrency limit** *(per node)* \u2014 the cap on how many jobs of one type may run at once; a type at its cap cannot start more work.\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`MetricsRegistry::recordJobStarted`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#handler-label)", + "description": "###### What this is:\n*Queue wait for the ledgerRequest job type, split by which handler enqueued the job. The type has a concurrency limit of 3 and two producers that compete for those slots: RcvGetLedger, which serves TMGetLedger to syncing peers, and RcvGetObjByHash, which serves TMGetObjectByHash. Both report the same job_type, so without the handler split a wait spike cannot be attributed to either.*\n\n###### How it's computed:\n*p99 of job_queued_us for job_type=\"ledgerRequest\", grouped by the handler label. JobQueue::processTask measures the wait, then PerfLog hands it to MetricsRegistry::recordJobStarted, which is where the histogram is recorded. The handler value is the addJob name passed through a sanitizer that keeps letters-only names and folds everything else to \"other\", which bounds the label domain to 43 names plus \"other\". Both producers here are letters-only, so both appear under their own names; \"other\" is a mixed bucket and never means one specific caller.*\n\n###### Reading it:\n*This is the panel that answers which producer is starving the 3-slot queue. Both lines high together means the queue is genuinely oversubscribed and both kinds of peer request are being delayed. One line high while the other is flat means that producer is arriving faster than 3 concurrent slots can absorb, and it is the one delaying the other. Wait is queue time only, so a high line here is contention, not slow work; the work itself is on the GetObject Handler Latency Breakdown panel.*\n\n###### Healthy range:\n*Single-digit to low-tens of milliseconds p99 for both handlers, matching the wider Job Queue Wait p95 By Type panel.*\n\n###### Watch for:\n*RcvGetObjByHash wait climbing: one in-bounds TMGetObjectByHash request can perform thousands of NodeStore lookups, so a few concurrent ones occupy every slot and delay TMGetLedger to peers that are themselves syncing. Cross-check Job Queue Backlog and Deferred by Type for jobq_ledgerrequest_deferred above zero to confirm the limit, not the work, is the binding constraint.*\n\n###### Keywords:\n- **Handler label** *(per node)* \u2014 the addJob call-site name attached to job metrics, so producers sharing one job type stay separable.\n- **Concurrency limit** *(per node)* \u2014 the cap on how many jobs of one type may run at once; a type at its cap cannot start more work.\n- **Job queue / job type** *(per node)* \u2014 xrpld's worker-thread pool; every unit of background work is enqueued under a named job type.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`MetricsRegistry::recordJobStarted`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#handler-label)", "type": "timeseries", "gridPos": { "h": 10, @@ -1862,7 +1862,7 @@ }, { "title": "NuDB Writer Queue Depth", - "description": "###### What this is:\n*How many writers are queued at NuDB's insert mutex. NuDB takes one global lock per insert, so concurrent writers do not overlap -- they line up. This panel is the confirming half of the read-latency discriminator: when reads are fast and the lane is still full, the queueing is here.*\n\n###### How it's computed:\n*Mean Depth is nodestore_state{metric=\"nudb_writer_depth_x100\"} divided by 100. At the source it is depthSum over depthSamples, both accumulated when an insert enters the critical section, so an insert still in flight is part of the mean. The exported gauge is integral, so the mean is scaled by 100 to keep the fractional part; dividing it back is what makes 1.60 readable instead of 1. In Flight is the instantaneous nudb_writers_in_flight sample.*\n\n###### Reading it:\n*Depth is a queue length, so 1.0 is the floor and means every insert found the lock free. Anything meaningfully above 1.0 means inserts are waiting on each other and the write path, not the disk, is setting the pace. In Flight is a point sample from the scrape instant and will look spikier than the mean; read the mean for the trend and In Flight for the peak.*\n\n###### Healthy range:\n*Mean depth at or just above 1.0, In Flight low.*\n\n###### Watch for:\n*Mean depth above 1 while read latency stays in single-digit microseconds -- that pair is the write-lock ceiling, and no amount of read tuning will move it. A measured run of this mode held depth at 1.60 with reads flat at 8-9 microseconds and took 510 seconds to reach full. That 1.60 came from a build whose sample count advanced at insert exit rather than entry, which biased the mean down, so treat it as a lower bound. The same run appeared to complete nothing, which was a separate counting defect, not a stalled node. Depth at 1.0 with slow reads is the opposite mode; look at the read panel.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[telemetry/MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`MetricsRegistry::observeWritePathDetail`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", + "description": "###### What this is:\n*How many writers are queued at NuDB's insert mutex. NuDB takes one global lock per insert, so concurrent writers do not overlap -- they line up. This panel is the confirming half of the read-latency discriminator: when reads are fast and the lane is still full, the queueing is here.*\n\n###### How it's computed:\n*Mean Depth is nodestore_state{metric=\"nudb_writer_depth_x100\"} divided by 100. At the source it is depthSum over depthSamples, both accumulated when an insert enters the critical section, so an insert still in flight is part of the mean. The exported gauge is integral, so the mean is scaled by 100 to keep the fractional part; dividing it back is what makes 1.60 readable instead of 1. In Flight is the instantaneous nudb_writers_in_flight sample.*\n\n###### Reading it:\n*Depth is a queue length, so 1.0 is the floor and means every insert found the lock free. Anything meaningfully above 1.0 means inserts are waiting on each other and the write path, not the disk, is setting the pace. In Flight is a point sample from the scrape instant and will look spikier than the mean; read the mean for the trend and In Flight for the peak.*\n\n###### Healthy range:\n*Mean depth at or just above 1.0, In Flight low.*\n\n###### Watch for:\n*Mean depth above 1 while read latency stays in single-digit microseconds -- that pair is the write-lock ceiling, and no amount of read tuning will move it. A measured run of this mode held depth at 1.60 with reads flat at 8-9 microseconds and took 510 seconds to reach full. That 1.60 came from a build whose sample count advanced at insert exit rather than entry, which biased the mean down, so treat it as a lower bound. The same run appeared to complete nothing, which was a separate counting defect, not a stalled node. Depth at 1.0 with slow reads is the opposite mode; look at the read panel.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[telemetry/AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`AppMetricGauges::observeWritePathDetail`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", "type": "timeseries", "gridPos": { "h": 10, @@ -1915,7 +1915,7 @@ }, { "title": "NuDB Insert Time (Mean & Max)", - "description": "###### What this is:\n*Time spent inside a NuDB insert, mean and worst case, alongside the backend's overall mean write latency. Splits an insert's cost into the part that is real work and the part that is waiting for the global insert mutex.*\n\n###### How it's computed:\n*Insert Mean is nodestore_state{metric=\"nudb_insert_mean_us\"}, total insert microseconds over insert count. Insert Max is nudb_insert_max_us, a true running maximum rather than a quantile, so one bad insert is visible and never averaged away. Write Mean is the write_mean_us gauge for the whole backend write path.*\n\n###### Reading it:\n*Compare Insert Mean against the service time implied by Writer Queue Depth. Mean insert time above the unqueued service time is wait, and the gap is the fraction of every write spent queued rather than working. Insert Max far above Insert Mean means the distribution has a tail -- typically a bucket split or a commit -- which a mean alone hides.*\n\n###### Healthy range:\n*Insert Mean in the low tens of microseconds with Insert Max within roughly an order of magnitude of it.*\n\n###### Watch for:\n*A widening gap between Insert Mean and the service floor. In a measured write-lock-bound run the mean was 20 microseconds of which only 12.5 was service, leaving 7.5 -- at least 37 percent of every insert -- as pure queueing. It is a floor rather than an exact figure because it is derived from a mean depth that the build of the day biased downward, and a larger depth implies a larger queueing share. That is the cost the writer-depth panel predicts, quantified. Insert Max climbing on its own without the mean moving is a tail problem, not a saturation problem.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[telemetry/MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`MetricsRegistry::observeWritePathDetail`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", + "description": "###### What this is:\n*Time spent inside a NuDB insert, mean and worst case, alongside the backend's overall mean write latency. Splits an insert's cost into the part that is real work and the part that is waiting for the global insert mutex.*\n\n###### How it's computed:\n*Insert Mean is nodestore_state{metric=\"nudb_insert_mean_us\"}, total insert microseconds over insert count. Insert Max is nudb_insert_max_us, a true running maximum rather than a quantile, so one bad insert is visible and never averaged away. Write Mean is the write_mean_us gauge for the whole backend write path.*\n\n###### Reading it:\n*Compare Insert Mean against the service time implied by Writer Queue Depth. Mean insert time above the unqueued service time is wait, and the gap is the fraction of every write spent queued rather than working. Insert Max far above Insert Mean means the distribution has a tail -- typically a bucket split or a commit -- which a mean alone hides.*\n\n###### Healthy range:\n*Insert Mean in the low tens of microseconds with Insert Max within roughly an order of magnitude of it.*\n\n###### Watch for:\n*A widening gap between Insert Mean and the service floor. In a measured write-lock-bound run the mean was 20 microseconds of which only 12.5 was service, leaving 7.5 -- at least 37 percent of every insert -- as pure queueing. It is a floor rather than an exact figure because it is derived from a mean depth that the build of the day biased downward, and a larger depth implies a larger queueing share. That is the cost the writer-depth panel predicts, quantified. Insert Max climbing on its own without the mean moving is a tail problem, not a saturation problem.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[telemetry/AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`AppMetricGauges::observeWritePathDetail`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", "type": "timeseries", "gridPos": { "h": 10, diff --git a/docker/telemetry/grafana/dashboards/ledger-operations.json b/docker/telemetry/grafana/dashboards/ledger-operations.json index 7abc014089..8fefbda59b 100644 --- a/docker/telemetry/grafana/dashboards/ledger-operations.json +++ b/docker/telemetry/grafana/dashboards/ledger-operations.json @@ -446,7 +446,7 @@ }, { "title": "Ledger Close Interval & Age", - "description": "###### What this is:\n*The wall-clock time between consecutive ledger closes \u2014 the network close cadence.*\n\n###### How it's computed:\n*Close Interval: 1 / rate(ledgers_closed_total), the average seconds between closes from the monotonic close counter (scrape-independent, unlike a gauge delta which would alias to the scrape period). Last-Close Age: time() minus the last-close network time (server_info last_close_time gauge + Ripple-epoch offset), i.e. seconds since the last ledger closed.*\n\n###### Reading it:\n*A steady line near the network's target close interval.*\n\n###### Healthy range:\n*About 3-5s on mainnet; workload-dependent on test networks.*\n\n###### Watch for:\n*A rising interval (consensus slowing or the node lagging) or a flat line at zero (ledgers no longer closing).*\n\n###### Keywords:\n- **Ledger close** *(network event)* \u2014 the current open ledger is closed and a new closed ledger is built from the agreed transaction set.\n- **Ledger close interval** *(network-wide)* \u2014 the network's steady ledger rhythm \u2014 roughly one closed ledger every 3-5 seconds on Mainnet.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp) \u00b7 [RCLConsensus.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/consensus/RCLConsensus.cpp)\n\n###### Function:\n`registerServerInfoGauge (last_close_time) ; ledgers_closed_total`\n\n###### References:\n[Ledger close](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Ledger close interval](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-close)", + "description": "###### What this is:\n*The wall-clock time between consecutive ledger closes \u2014 the network close cadence.*\n\n###### How it's computed:\n*Close Interval: 1 / rate(ledgers_closed_total), the average seconds between closes from the monotonic close counter (scrape-independent, unlike a gauge delta which would alias to the scrape period). Last-Close Age: time() minus the last-close network time (server_info last_close_time gauge + Ripple-epoch offset), i.e. seconds since the last ledger closed.*\n\n###### Reading it:\n*A steady line near the network's target close interval.*\n\n###### Healthy range:\n*About 3-5s on mainnet; workload-dependent on test networks.*\n\n###### Watch for:\n*A rising interval (consensus slowing or the node lagging) or a flat line at zero (ledgers no longer closing).*\n\n###### Keywords:\n- **Ledger close** *(network event)* \u2014 the current open ledger is closed and a new closed ledger is built from the agreed transaction set.\n- **Ledger close interval** *(network-wide)* \u2014 the network's steady ledger rhythm \u2014 roughly one closed ledger every 3-5 seconds on Mainnet.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp) \u00b7 [MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp) \u00b7 [RCLConsensus.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/consensus/RCLConsensus.cpp)\n\n###### Function:\n`registerServerInfoGauge (last_close_time) ; ledgers_closed_total`\n\n###### References:\n[Ledger close](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Ledger close interval](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-close)", "type": "timeseries", "gridPos": { "h": 10, diff --git a/docker/telemetry/grafana/dashboards/node-health.json b/docker/telemetry/grafana/dashboards/node-health.json index 7260a6f85a..6d6cc69f86 100644 --- a/docker/telemetry/grafana/dashboards/node-health.json +++ b/docker/telemetry/grafana/dashboards/node-health.json @@ -358,7 +358,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Rate at which a locally built ledger hash fails to match the network-validated hash.*\n\n###### How it's computed:\n*Per-second rate of history-mismatch events over a 5-minute window.*\n\n###### Reading it:\n*Flat at zero is the only healthy reading.*\n\n###### Healthy range:\n*Zero.*\n\n###### Watch for:\n*Any nonzero value indicates consensus divergence or database corruption and warrants immediate investigation.*\n\n###### Keywords:\n- **Ledger history mismatch** *(per node)* \u2014 when a locally built ledger's hash does not match the network-validated hash.\n- **Validated ledger** *(network-wide)* \u2014 a ledger confirmed final by the trusted validator quorum; its contents never change.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementLedgerHistoryMismatch`\n\n###### References:\n[Validated ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-history-mismatch)", + "description": "###### What this is:\n*Rate at which a locally built ledger hash fails to match the network-validated hash.*\n\n###### How it's computed:\n*Per-second rate of history-mismatch events over a 5-minute window.*\n\n###### Reading it:\n*Flat at zero is the only healthy reading.*\n\n###### Healthy range:\n*Zero.*\n\n###### Watch for:\n*Any nonzero value indicates consensus divergence or database corruption and warrants immediate investigation.*\n\n###### Keywords:\n- **Ledger history mismatch** *(per node)* \u2014 when a locally built ledger's hash does not match the network-validated hash.\n- **Validated ledger** *(network-wide)* \u2014 a ledger confirmed final by the trusted validator quorum; its contents never change.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementLedgerHistoryMismatch`\n\n###### References:\n[Validated ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-history-mismatch)", "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", @@ -1099,7 +1099,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*The node's operating mode over time as a colored timeline (Disconnected, Connected, Syncing, Tracking, Full).*\n\n###### How it's computed:\n*Current value of the server-state gauge mapped to a mode name; equal consecutive samples are merged into a single band.*\n\n###### Reading it:\n*A solid green Full band across the window is the goal; other colors mark periods the node was not fully synced.*\n\n###### Healthy range:\n*Continuously Full (green).*\n\n###### Watch for:\n*Bands of Syncing, Connected, or Disconnected, which pinpoint exactly when the node dropped out of Full.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", + "description": "###### What this is:\n*The node's operating mode over time as a colored timeline (Disconnected, Connected, Syncing, Tracking, Full).*\n\n###### How it's computed:\n*Current value of the server-state gauge mapped to a mode name; equal consecutive samples are merged into a single band.*\n\n###### Reading it:\n*A solid green Full band across the window is the goal; other colors mark periods the node was not fully synced.*\n\n###### Healthy range:\n*Continuously Full (green).*\n\n###### Watch for:\n*Bands of Syncing, Connected, or Disconnected, which pinpoint exactly when the node dropped out of Full.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", "fieldConfig": { "defaults": { "color": { @@ -1241,7 +1241,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Object-store read, found and write operation rates. The found series is `node_reads_hit`, which counts fetches that returned an object whatever served them, so it is not a cache-hit count.*\n\n###### How it's computed:\n*Per-second rates of the read, found and write counters, plotted as lines.*\n\n###### Reading it:\n*Steadily rising lines are normal; the slope reflects store activity.*\n\n###### Healthy range:\n*Reads and writes rising smoothly, with found tracking almost all reads on a node that has the data.*\n\n###### Watch for:\n*A sudden surge in reads or writes signals heavy back-end I/O, from sync, replay, or query load.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", + "description": "###### What this is:\n*Object-store read, found and write operation rates. The found series is `node_reads_hit`, which counts fetches that returned an object whatever served them, so it is not a cache-hit count.*\n\n###### How it's computed:\n*Per-second rates of the read, found and write counters, plotted as lines.*\n\n###### Reading it:\n*Steadily rising lines are normal; the slope reflects store activity.*\n\n###### Healthy range:\n*Reads and writes rising smoothly, with found tracking almost all reads on a node that has the data.*\n\n###### Watch for:\n*A sudden surge in reads or writes signals heavy back-end I/O, from sync, replay, or query load.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", "fieldConfig": { "defaults": { "color": { @@ -1360,7 +1360,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Fraction of object-store reads that returned an object. `node_reads_hit` counts fetches that found the object whatever served them, so this is not a cache-hit rate.*\n\n###### How it's computed:\n*Per-second rate of `node_reads_hit` divided by the per-second rate of `node_reads_total`, as a single ratio per node.*\n\n###### Reading it:\n*A value near 1.0 means almost every read finds its object; dips mean reads are missing.*\n\n###### Healthy range:\n*Close to 1.0 on a node that holds the data it is being asked for.*\n\n###### Watch for:\n*A sustained drop means the node is repeatedly asked for objects it does not have, which usually accompanies backfill or a gap in history.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", + "description": "###### What this is:\n*Fraction of object-store reads that returned an object. `node_reads_hit` counts fetches that found the object whatever served them, so this is not a cache-hit rate.*\n\n###### How it's computed:\n*Per-second rate of `node_reads_hit` divided by the per-second rate of `node_reads_total`, as a single ratio per node.*\n\n###### Reading it:\n*A value near 1.0 means almost every read finds its object; dips mean reads are missing.*\n\n###### Healthy range:\n*Close to 1.0 on a node that holds the data it is being asked for.*\n\n###### Watch for:\n*A sustained drop means the node is repeatedly asked for objects it does not have, which usually accompanies backfill or a gap in history.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", "fieldConfig": { "defaults": { "color": { @@ -1463,7 +1463,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Instantaneous write-load score and read-queue depth of the object store.*\n\n###### How it's computed:\n*Current values of the write-load and read-queue gauges, plotted over time.*\n\n###### Reading it:\n*Lower is better for both; short, flat lines are healthy.*\n\n###### Healthy range:\n*Write load near zero and read queue in low double digits or less.*\n\n###### Watch for:\n*High write load means back-end pressure; a high read queue means the prefetch threads are saturated.*\n\n###### Keywords:\n- **Transaction queue (TxQ)** *(per node)* \u2014 holds transactions that meet local cost but not the open-ledger cost, to include in a later ledger.\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n- **Read threads / read queue / write load** *(per node)* \u2014 NodeStore back-end I/O internals \u2014 worker threads reading, their queue depth, and write pressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Transaction queue (TxQ)](https://xrpl.org/docs/concepts/transactions/transaction-queue) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#transaction-queue-txq)", + "description": "###### What this is:\n*Instantaneous write-load score and read-queue depth of the object store.*\n\n###### How it's computed:\n*Current values of the write-load and read-queue gauges, plotted over time.*\n\n###### Reading it:\n*Lower is better for both; short, flat lines are healthy.*\n\n###### Healthy range:\n*Write load near zero and read queue in low double digits or less.*\n\n###### Watch for:\n*High write load means back-end pressure; a high read queue means the prefetch threads are saturated.*\n\n###### Keywords:\n- **Transaction queue (TxQ)** *(per node)* \u2014 holds transactions that meet local cost but not the open-ledger cost, to include in a later ledger.\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n- **Read threads / read queue / write load** *(per node)* \u2014 NodeStore back-end I/O internals \u2014 worker threads reading, their queue depth, and write pressure.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Transaction queue (TxQ)](https://xrpl.org/docs/concepts/transactions/transaction-queue) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#transaction-queue-txq)", "fieldConfig": { "defaults": { "color": { @@ -1578,7 +1578,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Cumulative bytes read from and written to the object-store back end.*\n\n###### How it's computed:\n*Current values of the bytes-read and bytes-written counters, plotted as lines.*\n\n###### Reading it:\n*Steadily rising lines are normal; the slope shows throughput.*\n\n###### Healthy range:\n*Smooth growth consistent with ledger and query activity.*\n\n###### Watch for:\n*A sharp acceleration indicates a heavy I/O phase such as sync, replay, or large queries.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", + "description": "###### What this is:\n*Cumulative bytes read from and written to the object-store back end.*\n\n###### How it's computed:\n*Current values of the bytes-read and bytes-written counters, plotted as lines.*\n\n###### Reading it:\n*Steadily rising lines are normal; the slope shows throughput.*\n\n###### Healthy range:\n*Smooth growth consistent with ledger and query activity.*\n\n###### Watch for:\n*A sharp acceleration indicates a heavy I/O phase such as sync, replay, or large queries.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", "fieldConfig": { "defaults": { "color": { @@ -1689,7 +1689,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Read-thread utilization, bundled read count, and cumulative read time for the object store.*\n\n###### How it's computed:\n*Current values of the running/total read-thread gauges, read-bundle gauge, and cumulative read-duration counter, plotted as lines.*\n\n###### Reading it:\n*Running threads well below total means spare capacity; a rising duration line reflects time spent in read I/O.*\n\n###### Healthy range:\n*Running threads below the total count most of the time.*\n\n###### Watch for:\n*Running threads pinned at the total for long periods means read I/O is saturated.*\n\n###### Keywords:\n- **Read threads / read queue / write load** *(per node)* \u2014 NodeStore back-end I/O internals \u2014 worker threads reading, their queue depth, and write pressure.\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#read-threads-read-queue-write-load)", + "description": "###### What this is:\n*Read-thread utilization, bundled read count, and cumulative read time for the object store.*\n\n###### How it's computed:\n*Current values of the running/total read-thread gauges, read-bundle gauge, and cumulative read-duration counter, plotted as lines.*\n\n###### Reading it:\n*Running threads well below total means spare capacity; a rising duration line reflects time spent in read I/O.*\n\n###### Healthy range:\n*Running threads below the total count most of the time.*\n\n###### Watch for:\n*Running threads pinned at the total for long periods means read I/O is saturated.*\n\n###### Keywords:\n- **Read threads / read queue / write load** *(per node)* \u2014 NodeStore back-end I/O internals \u2014 worker threads reading, their queue depth, and write pressure.\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#read-threads-read-queue-write-load)", "fieldConfig": { "defaults": { "color": { @@ -1809,7 +1809,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Share of wall-clock time the object store spent inside read I/O.*\n\n###### How it's computed:\n*Per-second rate of the cumulative `node_reads_duration_us` counter, converted from microseconds to seconds, as a single ratio per node.*\n\n###### Reading it:\n*1.0 means the store spent a full second per second in reads; well below 1.0 means spare read capacity.*\n\n###### Healthy range:\n*Below roughly 0.8 in steady state.*\n\n###### Watch for:\n*Sustained values at or above 1.0 mean read I/O is saturated and reads are queueing.*\n\n###### Keywords:\n- **Read threads / read queue / write load** *(per node)* \u2014 NodeStore back-end I/O internals \u2014 worker threads reading, their queue depth, and write pressure.\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#read-threads-read-queue-write-load)", + "description": "###### What this is:\n*Share of wall-clock time the object store spent inside read I/O.*\n\n###### How it's computed:\n*Per-second rate of the cumulative `node_reads_duration_us` counter, converted from microseconds to seconds, as a single ratio per node.*\n\n###### Reading it:\n*1.0 means the store spent a full second per second in reads; well below 1.0 means spare read capacity.*\n\n###### Healthy range:\n*Below roughly 0.8 in steady state.*\n\n###### Watch for:\n*Sustained values at or above 1.0 mean read I/O is saturated and reads are queueing.*\n\n###### Keywords:\n- **Read threads / read queue / write load** *(per node)* \u2014 NodeStore back-end I/O internals \u2014 worker threads reading, their queue depth, and write pressure.\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#read-threads-read-queue-write-load)", "fieldConfig": { "defaults": { "color": { @@ -2610,7 +2610,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Hit rates for the SLE, Ledger, and AcceptedLedger caches, from 0 to 1.*\n\n###### How it's computed:\n*Current values of the per-cache hit-rate gauges, plotted as lines.*\n\n###### Reading it:\n*Higher is better; each line is the fraction of lookups served from cache.*\n\n###### Healthy range:\n*Above roughly 0.8 in steady state.*\n\n###### Watch for:\n*Low or falling hit rates indicate cache thrashing and extra back-end reads.*\n\n###### Keywords:\n- **Caches (SLE / Ledger / TreeNode / FullBelow / AcceptedLedger)** *(per node)* \u2014 in-memory caches that avoid re-reading or re-computing ledger data; higher hit rates mean less back-end work.\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerCacheHitRateGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#caches)", + "description": "###### What this is:\n*Hit rates for the SLE, Ledger, and AcceptedLedger caches, from 0 to 1.*\n\n###### How it's computed:\n*Current values of the per-cache hit-rate gauges, plotted as lines.*\n\n###### Reading it:\n*Higher is better; each line is the fraction of lookups served from cache.*\n\n###### Healthy range:\n*Above roughly 0.8 in steady state.*\n\n###### Watch for:\n*Low or falling hit rates indicate cache thrashing and extra back-end reads.*\n\n###### Keywords:\n- **Caches (SLE / Ledger / TreeNode / FullBelow / AcceptedLedger)** *(per node)* \u2014 in-memory caches that avoid re-reading or re-computing ledger data; higher hit rates mean less back-end work.\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerCacheHitRateGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#caches)", "fieldConfig": { "defaults": { "color": { @@ -2731,7 +2731,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Entry counts for the TreeNode cache and track set, the FullBelow cache, and the AcceptedLedger cache.*\n\n###### How it's computed:\n*Current values of the per-cache size gauges, plotted as lines.*\n\n###### Reading it:\n*Stable lines are normal; sizes grow with working set and shrink after sweeps.*\n\n###### Healthy range:\n*Stable within configured limits.*\n\n###### Watch for:\n*Unbounded growth suggests memory pressure or a cache not being swept.*\n\n###### Keywords:\n- **Caches (SLE / Ledger / TreeNode / FullBelow / AcceptedLedger)** *(per node)* \u2014 in-memory caches that avoid re-reading or re-computing ledger data; higher hit rates mean less back-end work.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerCacheHitRateGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#caches)", + "description": "###### What this is:\n*Entry counts for the TreeNode cache and track set, the FullBelow cache, and the AcceptedLedger cache.*\n\n###### How it's computed:\n*Current values of the per-cache size gauges, plotted as lines.*\n\n###### Reading it:\n*Stable lines are normal; sizes grow with working set and shrink after sweeps.*\n\n###### Healthy range:\n*Stable within configured limits.*\n\n###### Watch for:\n*Unbounded growth suggests memory pressure or a cache not being swept.*\n\n###### Keywords:\n- **Caches (SLE / Ledger / TreeNode / FullBelow / AcceptedLedger)** *(per node)* \u2014 in-memory caches that avoid re-reading or re-computing ledger data; higher hit rates mean less back-end work.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerCacheHitRateGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#caches)", "fieldConfig": { "defaults": { "color": { @@ -3060,7 +3060,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Which sync state each node is in right now: Disconnected, Connected, Syncing, Tracking or Full. Only a Full node holds the current validated ledger and can answer authoritatively.*\n\n###### How it's computed:\n*The `server_state` gauge, the node's operating mode as an integer 0-4, value-mapped to a name and colour: 0 Disconnected (red), 1 Connected (yellow), 2 Syncing (orange), 3 Tracking (blue), 4 Full (green). No rate or aggregation \u2014 it is the instantaneous state.*\n\n###### Reading it:\n*One tile per node. Green FULL is the steady state; any other colour says the node is not yet serving the current ledger and how far along it is.*\n\n###### Healthy range:\n*FULL on every node.*\n\n###### Watch for:\n*A node leaving FULL and staying out, or flapping between TRACKING and FULL \u2014 that points at ledger acquisition falling behind rather than a connectivity fault. Cross-check Operating Mode (Time Share) and Validated Ledger Age.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", + "description": "###### What this is:\n*Which sync state each node is in right now: Disconnected, Connected, Syncing, Tracking or Full. Only a Full node holds the current validated ledger and can answer authoritatively.*\n\n###### How it's computed:\n*The `server_state` gauge, the node's operating mode as an integer 0-4, value-mapped to a name and colour: 0 Disconnected (red), 1 Connected (yellow), 2 Syncing (orange), 3 Tracking (blue), 4 Full (green). No rate or aggregation \u2014 it is the instantaneous state.*\n\n###### Reading it:\n*One tile per node. Green FULL is the steady state; any other colour says the node is not yet serving the current ledger and how far along it is.*\n\n###### Healthy range:\n*FULL on every node.*\n\n###### Watch for:\n*A node leaving FULL and staying out, or flapping between TRACKING and FULL \u2014 that points at ledger acquisition falling behind rather than a connectivity fault. Cross-check Operating Mode (Time Share) and Validated Ledger Age.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", "fieldConfig": { "defaults": { "color": { @@ -3236,7 +3236,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*How long the server process has been running, in seconds.*\n\n###### How it's computed:\n*Current value of the uptime gauge.*\n\n###### Reading it:\n*Higher is generally better; a reset to a small value means the process restarted.*\n\n###### Healthy range:\n*Continuously increasing.*\n\n###### Watch for:\n*An unexpected drop to near zero indicates a restart or crash.*\n\n###### Keywords:\n- **Uptime** *(per node)* \u2014 seconds since the server process started.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md)", + "description": "###### What this is:\n*How long the server process has been running, in seconds.*\n\n###### How it's computed:\n*Current value of the uptime gauge.*\n\n###### Reading it:\n*Higher is generally better; a reset to a small value means the process restarted.*\n\n###### Healthy range:\n*Continuously increasing.*\n\n###### Watch for:\n*An unexpected drop to near zero indicates a restart or crash.*\n\n###### Keywords:\n- **Uptime** *(per node)* \u2014 seconds since the server process started.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md)", "fieldConfig": { "defaults": { "color": { @@ -3301,7 +3301,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Total connected peers, inbound plus outbound.*\n\n###### How it's computed:\n*Current value of the peer-count gauge.*\n\n###### Reading it:\n*A stable count in the healthy range is good; too few limits connectivity.*\n\n###### Healthy range:\n*Workload- and config-dependent, typically 10 or more.*\n\n###### Watch for:\n*A sudden drop points to network or connectivity problems; an unusually high inbound count can indicate connection-flood pressure.*\n\n###### Keywords:\n- **Peer** *(per node)* \u2014 another server this node holds a protocol connection to.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md)", + "description": "###### What this is:\n*Total connected peers, inbound plus outbound.*\n\n###### How it's computed:\n*Current value of the peer-count gauge.*\n\n###### Reading it:\n*A stable count in the healthy range is good; too few limits connectivity.*\n\n###### Healthy range:\n*Workload- and config-dependent, typically 10 or more.*\n\n###### Watch for:\n*A sudden drop points to network or connectivity problems; an unusually high inbound count can indicate connection-flood pressure.*\n\n###### Keywords:\n- **Peer** *(per node)* \u2014 another server this node holds a protocol connection to.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md)", "fieldConfig": { "defaults": { "color": { @@ -3375,7 +3375,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Sequence number of the current open ledger.*\n\n###### How it's computed:\n*Current value of the open-ledger index gauge.*\n\n###### Reading it:\n*Should climb steadily; the gap above the validated sequence is the ledgers in flight.*\n\n###### Healthy range:\n*One or two ahead of the validated sequence.*\n\n###### Watch for:\n*A large or growing gap above the validated sequence means validation is lagging behind ledger creation.*\n\n###### Keywords:\n- **Ledger index** *(network-wide)* \u2014 the sequence number identifying a ledger version; increases by one each close.\n- **Open ledger** *(per node)* \u2014 the temporary workspace ledger where incoming transactions are provisionally applied before a close.\n- **Validated ledger** *(network-wide)* \u2014 a ledger confirmed final by the trusted validator quorum; its contents never change.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Ledger index](https://xrpl.org/docs/references/protocol/data-types/basic-data-types#ledger-index) \u00b7 [Open ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-index)", + "description": "###### What this is:\n*Sequence number of the current open ledger.*\n\n###### How it's computed:\n*Current value of the open-ledger index gauge.*\n\n###### Reading it:\n*Should climb steadily; the gap above the validated sequence is the ledgers in flight.*\n\n###### Healthy range:\n*One or two ahead of the validated sequence.*\n\n###### Watch for:\n*A large or growing gap above the validated sequence means validation is lagging behind ledger creation.*\n\n###### Keywords:\n- **Ledger index** *(network-wide)* \u2014 the sequence number identifying a ledger version; increases by one each close.\n- **Open ledger** *(per node)* \u2014 the temporary workspace ledger where incoming transactions are provisionally applied before a close.\n- **Validated ledger** *(network-wide)* \u2014 a ledger confirmed final by the trusted validator quorum; its contents never change.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Ledger index](https://xrpl.org/docs/references/protocol/data-types/basic-data-types#ledger-index) \u00b7 [Open ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-index)", "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", @@ -3440,7 +3440,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*The spread in validated ledger sequence across all selected nodes.*\n\n###### How it's computed:\n*Highest validated ledger sequence minus the lowest, among the selected nodes on the same network.*\n\n###### Reading it:\n*0 means every node agrees on the same validated ledger; larger means they diverge.*\n\n###### Healthy range:\n*0 to 1 ledger in steady state.*\n\n###### Watch for:\n*A sustained spread above a few ledgers means some nodes are lagging or the fleet is diverging.*\n\n###### Keywords:\n- **Validated ledger** *(network-wide)* \u2014 a ledger confirmed final by the trusted validator quorum; its contents never change.\n\n###### Computation boundary:\n*Result: Per network \u2014 the query aggregates the selected nodes into one series for each `xrpl_network_type`, so mainnet and devnet sequences are never subtracted from each other.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Validated ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validated-ledger)", + "description": "###### What this is:\n*The spread in validated ledger sequence across all selected nodes.*\n\n###### How it's computed:\n*Highest validated ledger sequence minus the lowest, among the selected nodes on the same network.*\n\n###### Reading it:\n*0 means every node agrees on the same validated ledger; larger means they diverge.*\n\n###### Healthy range:\n*0 to 1 ledger in steady state.*\n\n###### Watch for:\n*A sustained spread above a few ledgers means some nodes are lagging or the fleet is diverging.*\n\n###### Keywords:\n- **Validated ledger** *(network-wide)* \u2014 a ledger confirmed final by the trusted validator quorum; its contents never change.\n\n###### Computation boundary:\n*Result: Per network \u2014 the query aggregates the selected nodes into one series for each `xrpl_network_type`, so mainnet and devnet sequences are never subtracted from each other.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Validated ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validated-ledger)", "fieldConfig": { "defaults": { "color": { @@ -3541,7 +3541,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*How far each node's validated ledger lags behind the network tip, in ledgers.*\n\n###### How it's computed:\n*Highest validated ledger sequence within the node's own network, minus each node's own sequence.*\n\n###### Reading it:\n*0 means the node is at the tip; larger values mean it trails further behind.*\n\n###### Healthy range:\n*0 to 1 ledger on a synced node.*\n\n###### Watch for:\n*A node stuck at a growing value is falling behind and not keeping up with consensus.*\n\n###### Keywords:\n- **Validated ledger** *(network-wide)* \u2014 a ledger confirmed final by the trusted validator quorum; its contents never change.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each node is compared against the highest sequence on its own `xrpl_network_type`, so mainnet and devnet sequences are never subtracted from each other.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Validated ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validated-ledger)", + "description": "###### What this is:\n*How far each node's validated ledger lags behind the network tip, in ledgers.*\n\n###### How it's computed:\n*Highest validated ledger sequence within the node's own network, minus each node's own sequence.*\n\n###### Reading it:\n*0 means the node is at the tip; larger values mean it trails further behind.*\n\n###### Healthy range:\n*0 to 1 ledger on a synced node.*\n\n###### Watch for:\n*A node stuck at a growing value is falling behind and not keeping up with consensus.*\n\n###### Keywords:\n- **Validated ledger** *(network-wide)* \u2014 a ledger confirmed final by the trusted validator quorum; its contents never change.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each node is compared against the highest sequence on its own `xrpl_network_type`, so mainnet and devnet sequences are never subtracted from each other.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Validated ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validated-ledger)", "fieldConfig": { "defaults": { "color": { @@ -3610,7 +3610,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*The running server's build version string.*\n\n###### How it's computed:\n*Read from the version label of the build-info metric (its value is always 1).*\n\n###### Reading it:\n*Confirms which version each node is running.*\n\n###### Healthy range:\n*The expected release version across all nodes.*\n\n###### Watch for:\n*A node on an unexpected or mismatched version in a fleet.*\n\n###### Keywords:\n- **Build version** *(per node)* \u2014 the xrpld release the process is running.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerBuildInfoGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md)", + "description": "###### What this is:\n*The running server's build version string.*\n\n###### How it's computed:\n*Read from the version label of the build-info metric (its value is always 1).*\n\n###### Reading it:\n*Confirms which version each node is running.*\n\n###### Healthy range:\n*The expected release version across all nodes.*\n\n###### Watch for:\n*A node on an unexpected or mismatched version in a fleet.*\n\n###### Keywords:\n- **Build version** *(per node)* \u2014 the xrpld release the process is running.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerBuildInfoGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md)", "fieldConfig": { "defaults": { "thresholds": { @@ -3674,7 +3674,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Proposer count and convergence time from the last closed consensus round.*\n\n###### How it's computed:\n*Current values of the last-close proposer-count and convergence-time gauges, plotted as lines.*\n\n###### Reading it:\n*A healthy proposer count with a low convergence time is good.*\n\n###### Healthy range:\n*Convergence time of a few seconds with the expected number of proposers.*\n\n###### Watch for:\n*A falling proposer count or rising convergence time signals degrading consensus conditions.*\n\n###### Keywords:\n- **Proposers** *(network event)* \u2014 the count of validators whose proposals this node heard in the last closed round.\n- **Convergence time** *(network event)* \u2014 the wall-clock time the network took to agree a ledger in a round.\n- **Consensus round** *(network event)* \u2014 one propose-and-revise iteration of consensus; several may run before validators converge on a ledger.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Proposers](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#proposers)", + "description": "###### What this is:\n*Proposer count and convergence time from the last closed consensus round.*\n\n###### How it's computed:\n*Current values of the last-close proposer-count and convergence-time gauges, plotted as lines.*\n\n###### Reading it:\n*A healthy proposer count with a low convergence time is good.*\n\n###### Healthy range:\n*Convergence time of a few seconds with the expected number of proposers.*\n\n###### Watch for:\n*A falling proposer count or rising convergence time signals degrading consensus conditions.*\n\n###### Keywords:\n- **Proposers** *(network event)* \u2014 the count of validators whose proposals this node heard in the last closed round.\n- **Convergence time** *(network event)* \u2014 the wall-clock time the network took to agree a ledger in a round.\n- **Consensus round** *(network event)* \u2014 one propose-and-revise iteration of consensus; several may run before validators converge on a ledger.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Proposers](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#proposers)", "fieldConfig": { "defaults": { "color": { @@ -3773,7 +3773,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Proposer count and convergence time from the last closed consensus round.*\n\n###### How it's computed:\n*Current values of the last-close proposer-count and convergence-time gauges, plotted as lines.*\n\n###### Reading it:\n*A healthy proposer count with a low convergence time is good.*\n\n###### Healthy range:\n*Convergence time of a few seconds with the expected number of proposers.*\n\n###### Watch for:\n*A falling proposer count or rising convergence time signals degrading consensus conditions.*\n\n###### Keywords:\n- **Proposers** *(network event)* \u2014 the count of validators whose proposals this node heard in the last closed round.\n- **Convergence time** *(network event)* \u2014 the wall-clock time the network took to agree a ledger in a round.\n- **Consensus round** *(network event)* \u2014 one propose-and-revise iteration of consensus; several may run before validators converge on a ledger.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Proposers](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#proposers)", + "description": "###### What this is:\n*Proposer count and convergence time from the last closed consensus round.*\n\n###### How it's computed:\n*Current values of the last-close proposer-count and convergence-time gauges, plotted as lines.*\n\n###### Reading it:\n*A healthy proposer count with a low convergence time is good.*\n\n###### Healthy range:\n*Convergence time of a few seconds with the expected number of proposers.*\n\n###### Watch for:\n*A falling proposer count or rising convergence time signals degrading consensus conditions.*\n\n###### Keywords:\n- **Proposers** *(network event)* \u2014 the count of validators whose proposals this node heard in the last closed round.\n- **Convergence time** *(network event)* \u2014 the wall-clock time the network took to agree a ledger in a round.\n- **Consensus round** *(network event)* \u2014 one propose-and-revise iteration of consensus; several may run before validators converge on a ledger.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Proposers](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#proposers)", "fieldConfig": { "defaults": { "color": { @@ -3873,7 +3873,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*The wall-clock time between consecutive ledger closes \u2014 the network close cadence.*\n\n###### How it's computed:\n*Close Interval: 1 / rate(ledgers_closed_total), the average seconds between closes from the monotonic close counter (scrape-independent, unlike a gauge delta which would alias to the scrape period). Last-Close Age: time() minus the last-close network time (server_info last_close_time gauge + Ripple-epoch offset), i.e. seconds since the last ledger closed.*\n\n###### Reading it:\n*A steady line near the network's target close interval.*\n\n###### Healthy range:\n*About 3-5s on mainnet; workload-dependent on test networks.*\n\n###### Watch for:\n*A rising interval (consensus slowing or the node lagging) or a flat line at zero (ledgers no longer closing).*\n\n###### Keywords:\n- **Ledger close** *(network event)* \u2014 the current open ledger is closed and a new closed ledger is built from the agreed transaction set.\n- **Ledger close interval** *(network-wide)* \u2014 the network's steady ledger rhythm \u2014 roughly one closed ledger every 3-5 seconds on Mainnet.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp) \u00b7 [RCLConsensus.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/consensus/RCLConsensus.cpp)\n\n###### Function:\n`registerServerInfoGauge (last_close_time) ; ledgers_closed_total`\n\n###### References:\n[Ledger close](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Ledger close interval](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-close)", + "description": "###### What this is:\n*The wall-clock time between consecutive ledger closes \u2014 the network close cadence.*\n\n###### How it's computed:\n*Close Interval: 1 / rate(ledgers_closed_total), the average seconds between closes from the monotonic close counter (scrape-independent, unlike a gauge delta which would alias to the scrape period). Last-Close Age: time() minus the last-close network time (server_info last_close_time gauge + Ripple-epoch offset), i.e. seconds since the last ledger closed.*\n\n###### Reading it:\n*A steady line near the network's target close interval.*\n\n###### Healthy range:\n*About 3-5s on mainnet; workload-dependent on test networks.*\n\n###### Watch for:\n*A rising interval (consensus slowing or the node lagging) or a flat line at zero (ledgers no longer closing).*\n\n###### Keywords:\n- **Ledger close** *(network event)* \u2014 the current open ledger is closed and a new closed ledger is built from the agreed transaction set.\n- **Ledger close interval** *(network-wide)* \u2014 the network's steady ledger rhythm \u2014 roughly one closed ledger every 3-5 seconds on Mainnet.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp) \u00b7 [MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp) \u00b7 [RCLConsensus.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/consensus/RCLConsensus.cpp)\n\n###### Function:\n`registerServerInfoGauge (last_close_time) ; ledgers_closed_total`\n\n###### References:\n[Ledger close](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Ledger close interval](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledger-close)", "fieldConfig": { "defaults": { "color": { @@ -4018,7 +4018,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Cumulative object-payload bytes this process has written to the NodeStore back end. This is not the size of the store on disk.*\n\n###### How it's computed:\n*Current value of the stored_object_bytes gauge, plotted over time. It observes getStoreSize(), the same accessor node_written_bytes uses, so the two series are equal and their ratio is a constant 1.0 rather than a write-amplification measure.*\n\n###### Reading it:\n*A smoothly growing line is normal; the slope is the write rate. It excludes NuDB's keys, bucket padding and log, and it restarts from zero with the process while the files on disk do not.*\n\n###### Healthy range:\n*Gradual growth consistent with ledger data being stored.*\n\n###### Watch for:\n*A sudden jump in growth rate can indicate runaway storage or an unexpected back-fill. Do not use this panel to size the store on disk or to plan disk capacity; no metric reports on-disk size today, so check the filesystem directly.*\n\n###### Keywords:\n- **NuDB** *(per node)* \u2014 the append-only key-value database used as the default NodeStore backend.\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n- **Back-fill / catch-up** *(per node)* \u2014 fetching missing historical ledgers from peers to fill gaps or reach the network tip.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerStorageDetailGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nudb)", + "description": "###### What this is:\n*Cumulative object-payload bytes this process has written to the NodeStore back end. This is not the size of the store on disk.*\n\n###### How it's computed:\n*Current value of the stored_object_bytes gauge, plotted over time. It observes getStoreSize(), the same accessor node_written_bytes uses, so the two series are equal and their ratio is a constant 1.0 rather than a write-amplification measure.*\n\n###### Reading it:\n*A smoothly growing line is normal; the slope is the write rate. It excludes NuDB's keys, bucket padding and log, and it restarts from zero with the process while the files on disk do not.*\n\n###### Healthy range:\n*Gradual growth consistent with ledger data being stored.*\n\n###### Watch for:\n*A sudden jump in growth rate can indicate runaway storage or an unexpected back-fill. Do not use this panel to size the store on disk or to plan disk capacity; no metric reports on-disk size today, so check the filesystem directly.*\n\n###### Keywords:\n- **NuDB** *(per node)* \u2014 the append-only key-value database used as the default NodeStore backend.\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n- **Back-fill / catch-up** *(per node)* \u2014 fetching missing historical ledgers from peers to fill gaps or reach the network tip.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerStorageDetailGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nudb)", "fieldConfig": { "defaults": { "color": { @@ -4117,7 +4117,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Live instance counts for the busiest internal object types.*\n\n###### How it's computed:\n*Current per-type instance counts, showing the top 15 types over time.*\n\n###### Reading it:\n*Stable lines are healthy; each line is one object type's live count.*\n\n###### Healthy range:\n*Steady counts that rise and fall with load.*\n\n###### Watch for:\n*A single type climbing without bound suggests memory pressure or a leak.*\n\n###### Keywords:\n- **Object instance count** *(per node)* \u2014 live in-memory instances of a tracked C++ type, used to spot leaks.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerObjectCountGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md)", + "description": "###### What this is:\n*Live instance counts for the busiest internal object types.*\n\n###### How it's computed:\n*Current per-type instance counts, showing the top 15 types over time.*\n\n###### Reading it:\n*Stable lines are healthy; each line is one object type's live count.*\n\n###### Healthy range:\n*Steady counts that rise and fall with load.*\n\n###### Watch for:\n*A single type climbing without bound suggests memory pressure or a leak.*\n\n###### Keywords:\n- **Object instance count** *(per node)* \u2014 live in-memory instances of a tracked C++ type, used to spot leaks.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerObjectCountGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md)", "fieldConfig": { "defaults": { "color": { @@ -4233,7 +4233,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*How many historical ledgers the node is back-filling per minute.*\n\n###### How it's computed:\n*Current value of the historical-fetch-per-minute gauge.*\n\n###### Reading it:\n*Near zero once history is complete; elevated while back-filling.*\n\n###### Healthy range:\n*Close to zero in steady state.*\n\n###### Watch for:\n*A sustained high rate means the node is still filling gaps in its stored history.*\n\n###### Keywords:\n- **Historical fetch rate** *(per node)* \u2014 how many historical ledgers the node is back-filling per minute.\n- **Back-fill / catch-up** *(per node)* \u2014 fetching missing historical ledgers from peers to fill gaps or reach the network tip.\n- **Complete ledger ranges** *(per node)* \u2014 the contiguous spans of ledgers the node holds locally; one unbroken range is ideal.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerDbMetricsGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#historical-fetch-rate)", + "description": "###### What this is:\n*How many historical ledgers the node is back-filling per minute.*\n\n###### How it's computed:\n*Current value of the historical-fetch-per-minute gauge.*\n\n###### Reading it:\n*Near zero once history is complete; elevated while back-filling.*\n\n###### Healthy range:\n*Close to zero in steady state.*\n\n###### Watch for:\n*A sustained high rate means the node is still filling gaps in its stored history.*\n\n###### Keywords:\n- **Historical fetch rate** *(per node)* \u2014 how many historical ledgers the node is back-filling per minute.\n- **Back-fill / catch-up** *(per node)* \u2014 fetching missing historical ledgers from peers to fill gaps or reach the network tip.\n- **Complete ledger ranges** *(per node)* \u2014 the contiguous spans of ledgers the node holds locally; one unbroken range is ideal.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerDbMetricsGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#historical-fetch-rate)", "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", @@ -4302,7 +4302,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*The contiguous ranges of ledgers the node holds locally.*\n\n###### How it's computed:\n*Current start and end bounds of each complete range, listed as table rows.*\n\n###### Reading it:\n*Fewer ranges is better; one continuous range means an unbroken history.*\n\n###### Healthy range:\n*A single range covering the configured retention window.*\n\n###### Watch for:\n*Many fragmented ranges indicate gaps in stored history from missed or failed fetches.*\n\n###### Keywords:\n- **Complete ledger ranges** *(per node)* \u2014 the contiguous spans of ledgers the node holds locally; one unbroken range is ideal.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerCompleteLedgersGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#complete-ledger-ranges)", + "description": "###### What this is:\n*The contiguous ranges of ledgers the node holds locally.*\n\n###### How it's computed:\n*Current start and end bounds of each complete range, listed as table rows.*\n\n###### Reading it:\n*Fewer ranges is better; one continuous range means an unbroken history.*\n\n###### Healthy range:\n*A single range covering the configured retention window.*\n\n###### Watch for:\n*Many fragmented ranges indicate gaps in stored history from missed or failed fetches.*\n\n###### Keywords:\n- **Complete ledger ranges** *(per node)* \u2014 the contiguous spans of ledgers the node holds locally; one unbroken range is ideal.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerCompleteLedgersGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#complete-ledger-ranges)", "fieldConfig": { "defaults": { "custom": { @@ -4368,7 +4368,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Sizes of the relational databases in KB (total, ledger, transaction).*\n\n###### How it's computed:\n*Current values of the per-database size gauges, plotted as lines.*\n\n###### Reading it:\n*Smoothly growing lines are normal; the split shows where storage is used.*\n\n###### Healthy range:\n*Gradual growth consistent with retained history.*\n\n###### Watch for:\n*An abrupt change in growth rate can indicate storage pressure or a pruning issue.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerDbMetricsGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", + "description": "###### What this is:\n*Sizes of the relational databases in KB (total, ledger, transaction).*\n\n###### How it's computed:\n*Current values of the per-database size gauges, plotted as lines.*\n\n###### Reading it:\n*Smoothly growing lines are normal; the split shows where storage is used.*\n\n###### Healthy range:\n*Gradual growth consistent with retained history.*\n\n###### Watch for:\n*An abrupt change in growth rate can indicate storage pressure or a pruning issue.*\n\n###### Keywords:\n- **NodeStore** *(per node)* \u2014 the key-value object store holding ledger data (tree nodes), backed by NuDB.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerDbMetricsGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nodestore)", "fieldConfig": { "defaults": { "color": { @@ -4487,7 +4487,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Cumulative count of peers disconnected for exceeding resource limits.*\n\n###### How it's computed:\n*Current value of the resource-disconnect gauge, plotted over time.*\n\n###### Reading it:\n*A flat line is healthy; steps up mean peers were dropped for overuse.*\n\n###### Healthy range:\n*Flat or very slowly rising.*\n\n###### Watch for:\n*A rising line indicates peers are being throttled off, consistent with abusive or misbehaving peers.*\n\n###### Keywords:\n- **Resource disconnect** *(per node)* \u2014 a peer dropped for exceeding resource/load limits \u2014 the node shedding abusive or overactive peers.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#resource-disconnect)", + "description": "###### What this is:\n*Cumulative count of peers disconnected for exceeding resource limits.*\n\n###### How it's computed:\n*Current value of the resource-disconnect gauge, plotted over time.*\n\n###### Reading it:\n*A flat line is healthy; steps up mean peers were dropped for overuse.*\n\n###### Healthy range:\n*Flat or very slowly rising.*\n\n###### Watch for:\n*A rising line indicates peers are being throttled off, consistent with abusive or misbehaving peers.*\n\n###### Keywords:\n- **Resource disconnect** *(per node)* \u2014 a peer dropped for exceeding resource/load limits \u2014 the node shedding abusive or overactive peers.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#resource-disconnect)", "fieldConfig": { "defaults": { "color": { @@ -4603,7 +4603,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*The node's local load-based fee factor that scales its minimum transaction cost; baseline 256 at idle.*\n\n###### How it's computed:\n*Current value of the local load-fee economy gauge.*\n\n###### Reading it:\n*Steady at the baseline (256) is normal; higher values mean the node is raising its fee in response to load.*\n\n###### Healthy range:\n*Around 256 (the normal baseline) when idle.*\n\n###### Watch for:\n*A climbing factor, which indicates the node is under transaction load pressure.*\n\n###### Keywords:\n- **Load factor** *(per node)* \u2014 a unitless multiplier (1.0 = no load) that scales the base transaction cost as the node comes under load.\n- **Base fee** *(network-wide)* \u2014 the baseline transaction cost for a reference transaction under minimum load, in drops.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerLedgerEconomyGauge`\n\n###### References:\n[Load factor](https://xrpl.org/docs/concepts/transactions/transaction-cost#local-load-cost) \u00b7 [Base fee](https://xrpl.org/docs/concepts/transactions/transaction-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#load-factor)", + "description": "###### What this is:\n*The node's local load-based fee factor that scales its minimum transaction cost; baseline 256 at idle.*\n\n###### How it's computed:\n*Current value of the local load-fee economy gauge.*\n\n###### Reading it:\n*Steady at the baseline (256) is normal; higher values mean the node is raising its fee in response to load.*\n\n###### Healthy range:\n*Around 256 (the normal baseline) when idle.*\n\n###### Watch for:\n*A climbing factor, which indicates the node is under transaction load pressure.*\n\n###### Keywords:\n- **Load factor** *(per node)* \u2014 a unitless multiplier (1.0 = no load) that scales the base transaction cost as the node comes under load.\n- **Base fee** *(network-wide)* \u2014 the baseline transaction cost for a reference transaction under minimum load, in drops.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerLedgerEconomyGauge`\n\n###### References:\n[Load factor](https://xrpl.org/docs/concepts/transactions/transaction-cost#local-load-cost) \u00b7 [Base fee](https://xrpl.org/docs/concepts/transactions/transaction-cost) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#load-factor)", "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", @@ -4672,7 +4672,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*The minimum XRP balance required to keep an account on the ledger, in drops.*\n\n###### How it's computed:\n*Current value of the reserve-base economy gauge.*\n\n###### Reading it:\n*A stable value is expected; it changes only via network amendment or vote.*\n\n###### Healthy range:\n*The configured network reserve base.*\n\n###### Watch for:\n*An unexpected change outside a known amendment or fee vote.*\n\n###### Keywords:\n- **drops** *(network-wide)* \u2014 the smallest XRP unit \u2014 one drop is 0.000001 XRP (one millionth).\n- **Reserve (base & owner)** *(network-wide)* \u2014 the minimum XRP an account must hold \u2014 a base reserve plus an increment per owned ledger object.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerLedgerEconomyGauge`\n\n###### References:\n[drops](https://xrpl.org/docs/references/protocol/data-types/basic-data-types#specifying-currency-amounts) \u00b7 [Reserve (base & owner)](https://xrpl.org/docs/concepts/accounts/reserves) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#drops)", + "description": "###### What this is:\n*The minimum XRP balance required to keep an account on the ledger, in drops.*\n\n###### How it's computed:\n*Current value of the reserve-base economy gauge.*\n\n###### Reading it:\n*A stable value is expected; it changes only via network amendment or vote.*\n\n###### Healthy range:\n*The configured network reserve base.*\n\n###### Watch for:\n*An unexpected change outside a known amendment or fee vote.*\n\n###### Keywords:\n- **drops** *(network-wide)* \u2014 the smallest XRP unit \u2014 one drop is 0.000001 XRP (one millionth).\n- **Reserve (base & owner)** *(network-wide)* \u2014 the minimum XRP an account must hold \u2014 a base reserve plus an increment per owned ledger object.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerLedgerEconomyGauge`\n\n###### References:\n[drops](https://xrpl.org/docs/references/protocol/data-types/basic-data-types#specifying-currency-amounts) \u00b7 [Reserve (base & owner)](https://xrpl.org/docs/concepts/accounts/reserves) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#drops)", "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", @@ -4741,7 +4741,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*The additional XRP reserve required per owned ledger object, in drops.*\n\n###### How it's computed:\n*Current value of the reserve-increment economy gauge.*\n\n###### Reading it:\n*A stable value is expected; it changes only via network amendment or vote.*\n\n###### Healthy range:\n*The configured network owner reserve increment.*\n\n###### Watch for:\n*An unexpected change outside a known amendment or fee vote.*\n\n###### Keywords:\n- **Reserve (base & owner)** *(network-wide)* \u2014 the minimum XRP an account must hold \u2014 a base reserve plus an increment per owned ledger object.\n- **drops** *(network-wide)* \u2014 the smallest XRP unit \u2014 one drop is 0.000001 XRP (one millionth).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerLedgerEconomyGauge`\n\n###### References:\n[Reserve (base & owner)](https://xrpl.org/docs/concepts/accounts/reserves) \u00b7 [drops](https://xrpl.org/docs/references/protocol/data-types/basic-data-types#specifying-currency-amounts) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#reserve-base-owner)", + "description": "###### What this is:\n*The additional XRP reserve required per owned ledger object, in drops.*\n\n###### How it's computed:\n*Current value of the reserve-increment economy gauge.*\n\n###### Reading it:\n*A stable value is expected; it changes only via network amendment or vote.*\n\n###### Healthy range:\n*The configured network owner reserve increment.*\n\n###### Watch for:\n*An unexpected change outside a known amendment or fee vote.*\n\n###### Keywords:\n- **Reserve (base & owner)** *(network-wide)* \u2014 the minimum XRP an account must hold \u2014 a base reserve plus an increment per owned ledger object.\n- **drops** *(network-wide)* \u2014 the smallest XRP unit \u2014 one drop is 0.000001 XRP (one millionth).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerLedgerEconomyGauge`\n\n###### References:\n[Reserve (base & owner)](https://xrpl.org/docs/concepts/accounts/reserves) \u00b7 [drops](https://xrpl.org/docs/references/protocol/data-types/basic-data-types#specifying-currency-amounts) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#reserve-base-owner)", "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", @@ -4810,7 +4810,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Seconds since the last validated ledger closed, plotted over time.*\n\n###### How it's computed:\n*Current value of the ledger-age economy gauge, sampled each interval.*\n\n###### Reading it:\n*Lower is better; it should stay near the ledger close interval. Mirrors the Validated Ledger Age panel.*\n\n###### Healthy range:\n*Under about 10 seconds.*\n\n###### Watch for:\n*Growth beyond the expected close interval, meaning the node is not keeping up with validated ledgers.*\n\n###### Keywords:\n- **Validated ledger** *(network-wide)* \u2014 a ledger confirmed final by the trusted validator quorum; its contents never change.\n- **Ledger close interval** *(network-wide)* \u2014 the network's steady ledger rhythm \u2014 roughly one closed ledger every 3-5 seconds on Mainnet.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerLedgerEconomyGauge`\n\n###### References:\n[Validated ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Ledger close interval](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validated-ledger)", + "description": "###### What this is:\n*Seconds since the last validated ledger closed, plotted over time.*\n\n###### How it's computed:\n*Current value of the ledger-age economy gauge, sampled each interval.*\n\n###### Reading it:\n*Lower is better; it should stay near the ledger close interval. Mirrors the Validated Ledger Age panel.*\n\n###### Healthy range:\n*Under about 10 seconds.*\n\n###### Watch for:\n*Growth beyond the expected close interval, meaning the node is not keeping up with validated ledgers.*\n\n###### Keywords:\n- **Validated ledger** *(network-wide)* \u2014 a ledger confirmed final by the trusted validator quorum; its contents never change.\n- **Ledger close interval** *(network-wide)* \u2014 the network's steady ledger rhythm \u2014 roughly one closed ledger every 3-5 seconds on Mainnet.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerLedgerEconomyGauge`\n\n###### References:\n[Validated ledger](https://xrpl.org/docs/concepts/ledgers/open-closed-validated-ledgers) \u00b7 [Ledger close interval](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validated-ledger)", "fieldConfig": { "defaults": { "color": { @@ -4917,7 +4917,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*The network transaction throughput reported by the ledger economy metrics.*\n\n###### How it's computed:\n*Current value of the transaction-rate economy gauge, plotted over time.*\n\n###### Reading it:\n*Reflects how many transactions are being processed; higher means busier.*\n\n###### Healthy range:\n*Workload-dependent.*\n\n###### Watch for:\n*A sudden sustained surge can indicate a transaction flood; a drop to zero can indicate the node stopped processing.*\n\n###### Keywords:\n- **Transaction type** *(network-wide)* \u2014 the kind of transaction (Payment, OfferCreate, TrustSet, AMM*, NFToken*, etc.), used as a breakdown dimension.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerLedgerEconomyGauge`\n\n###### References:\n[Transaction type](https://xrpl.org/docs/references/protocol/transactions/types) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#transaction-type)", + "description": "###### What this is:\n*The network transaction throughput reported by the ledger economy metrics.*\n\n###### How it's computed:\n*Current value of the transaction-rate economy gauge, plotted over time.*\n\n###### Reading it:\n*Reflects how many transactions are being processed; higher means busier.*\n\n###### Healthy range:\n*Workload-dependent.*\n\n###### Watch for:\n*A sudden sustained surge can indicate a transaction flood; a drop to zero can indicate the node stopped processing.*\n\n###### Keywords:\n- **Transaction type** *(network-wide)* \u2014 the kind of transaction (Payment, OfferCreate, TrustSet, AMM*, NFToken*, etc.), used as a breakdown dimension.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerLedgerEconomyGauge`\n\n###### References:\n[Transaction type](https://xrpl.org/docs/references/protocol/transactions/types) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#transaction-type)", "fieldConfig": { "defaults": { "color": { diff --git a/docker/telemetry/grafana/dashboards/peer-network.json b/docker/telemetry/grafana/dashboards/peer-network.json index fc2f98b6d4..3788de3be6 100644 --- a/docker/telemetry/grafana/dashboards/peer-network.json +++ b/docker/telemetry/grafana/dashboards/peer-network.json @@ -257,7 +257,7 @@ }, { "title": "Reduce-Relay Peer Selection", - "description": "###### What this is:\n*How transaction relay picks peers: chosen relay sources, suppressed peers, and peers with the feature off.*\n\n###### How it's computed:\n*Current peer counts in each category (selected, suppressed, not-enabled), per node.*\n\n###### Reading it:\n*A high suppressed-to-selected ratio means relay is saving bandwidth as intended.*\n\n###### Healthy range:\n*Workload-dependent; suppressed should exceed selected in a well-connected mesh.*\n\n###### Watch for:\n*A large not-enabled count (older peers forcing full relay) or selected climbing while suppressed falls.*\n\n###### Keywords:\n- **Reduce-relay** *(per node)* \u2014 an optimization that relays messages through selected peers only, suppressing redundant forwarding.\n- **Transaction suppression** *(per node)* \u2014 dropping a transaction already seen from another peer, so it is not reprocessed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerReduceRelayGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#reduce-relay)", + "description": "###### What this is:\n*How transaction relay picks peers: chosen relay sources, suppressed peers, and peers with the feature off.*\n\n###### How it's computed:\n*Current peer counts in each category (selected, suppressed, not-enabled), per node.*\n\n###### Reading it:\n*A high suppressed-to-selected ratio means relay is saving bandwidth as intended.*\n\n###### Healthy range:\n*Workload-dependent; suppressed should exceed selected in a well-connected mesh.*\n\n###### Watch for:\n*A large not-enabled count (older peers forcing full relay) or selected climbing while suppressed falls.*\n\n###### Keywords:\n- **Reduce-relay** *(per node)* \u2014 an optimization that relays messages through selected peers only, suppressing redundant forwarding.\n- **Transaction suppression** *(per node)* \u2014 dropping a transaction already seen from another peer, so it is not reprocessed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerReduceRelayGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#reduce-relay)", "type": "timeseries", "gridPos": { "h": 10, @@ -317,7 +317,7 @@ }, { "title": "Reduce-Relay Missing-Tx Frequency", - "description": "###### What this is:\n*How often a peer has to fetch a transaction it missed because relay suppressed it.*\n\n###### How it's computed:\n*The reported frequency of on-demand missing-transaction fetches, per node.*\n\n###### Reading it:\n*Lower is better; near-flat means suppression is well tuned.*\n\n###### Healthy range:\n*Workload-dependent; a low, stable value is expected.*\n\n###### Watch for:\n*A rising trend, meaning suppression is too aggressive and the on-demand fetch path is growing.*\n\n###### Keywords:\n- **Reduce-relay** *(per node)* \u2014 an optimization that relays messages through selected peers only, suppressing redundant forwarding.\n- **Transaction suppression** *(per node)* \u2014 dropping a transaction already seen from another peer, so it is not reprocessed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerReduceRelayGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#reduce-relay)", + "description": "###### What this is:\n*How often a peer has to fetch a transaction it missed because relay suppressed it.*\n\n###### How it's computed:\n*The reported frequency of on-demand missing-transaction fetches, per node.*\n\n###### Reading it:\n*Lower is better; near-flat means suppression is well tuned.*\n\n###### Healthy range:\n*Workload-dependent; a low, stable value is expected.*\n\n###### Watch for:\n*A rising trend, meaning suppression is too aggressive and the on-demand fetch path is growing.*\n\n###### Keywords:\n- **Reduce-relay** *(per node)* \u2014 an optimization that relays messages through selected peers only, suppressing redundant forwarding.\n- **Transaction suppression** *(per node)* \u2014 dropping a transaction already seen from another peer, so it is not reprocessed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerReduceRelayGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#reduce-relay)", "type": "timeseries", "gridPos": { "h": 10, diff --git a/docker/telemetry/grafana/dashboards/peer-quality.json b/docker/telemetry/grafana/dashboards/peer-quality.json index 6339794def..fb66e33b20 100644 --- a/docker/telemetry/grafana/dashboards/peer-quality.json +++ b/docker/telemetry/grafana/dashboards/peer-quality.json @@ -71,7 +71,7 @@ }, { "title": "P90 Peer Latency", - "description": "###### What this is:\n*90th-percentile round-trip latency to connected peers, in milliseconds.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the p90 peer latency.*\n\n###### Reading it:\n*Lower is better; green under 200ms, yellow to 500ms, red above.*\n\n###### Healthy range:\n*Under 200ms.*\n\n###### Watch for:\n*Rising latency, which points to network congestion or geographically distant, poorly performing peers.*\n\n###### Keywords:\n- **Peer** *(per node)* \u2014 another server this node holds a protocol connection to.\n- **Overlay** *(network-wide)* \u2014 the peer-to-peer mesh xrpld nodes form to gossip transactions, proposals and validations.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerPeerQualityGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#peer)", + "description": "###### What this is:\n*90th-percentile round-trip latency to connected peers, in milliseconds.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the p90 peer latency.*\n\n###### Reading it:\n*Lower is better; green under 200ms, yellow to 500ms, red above.*\n\n###### Healthy range:\n*Under 200ms.*\n\n###### Watch for:\n*Rising latency, which points to network congestion or geographically distant, poorly performing peers.*\n\n###### Keywords:\n- **Peer** *(per node)* \u2014 another server this node holds a protocol connection to.\n- **Overlay** *(network-wide)* \u2014 the peer-to-peer mesh xrpld nodes form to gossip transactions, proposals and validations.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerPeerQualityGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#peer)", "type": "timeseries", "gridPos": { "h": 10, @@ -149,7 +149,7 @@ }, { "title": "Insane/Diverged Peers [$xrpl_network_type]", - "description": "###### What this is:\n*Count of connected peers whose ledger state has diverged from the network.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the diverged-peer count.*\n\n###### Reading it:\n*Zero is healthy; any count means those peers disagree on ledger state.*\n\n###### Healthy range:\n*0 diverged peers.*\n\n###### Watch for:\n*A persistent non-zero count, which can indicate peers on a fork or misbehaving peers.*\n\n###### Keywords:\n- **Insane / diverged peers** *(per node)* \u2014 connected peers whose ledger state disagrees with the network \u2014 possibly on a fork or misbehaving.\n- **Fork** *(network-wide)* \u2014 when nodes validate divergent ledger chains instead of a single agreed history.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerPeerQualityGauge`\n\n###### References:\n[Fork](https://xrpl.org/docs/concepts/consensus-protocol) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#insane-diverged-peers)", + "description": "###### What this is:\n*Count of connected peers whose ledger state has diverged from the network.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the diverged-peer count.*\n\n###### Reading it:\n*Zero is healthy; any count means those peers disagree on ledger state.*\n\n###### Healthy range:\n*0 diverged peers.*\n\n###### Watch for:\n*A persistent non-zero count, which can indicate peers on a fork or misbehaving peers.*\n\n###### Keywords:\n- **Insane / diverged peers** *(per node)* \u2014 connected peers whose ledger state disagrees with the network \u2014 possibly on a fork or misbehaving.\n- **Fork** *(network-wide)* \u2014 when nodes validate divergent ledger chains instead of a single agreed history.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerPeerQualityGauge`\n\n###### References:\n[Fork](https://xrpl.org/docs/concepts/consensus-protocol) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#insane-diverged-peers)", "type": "stat", "gridPos": { "h": 10, @@ -205,7 +205,7 @@ }, { "title": "Higher Version Peers % [$xrpl_network_type]", - "description": "###### What this is:\n*Percentage of connected peers running a newer rippled version than this node.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the higher-version peer percentage.*\n\n###### Reading it:\n*A high percentage suggests this node is behind and should be upgraded.*\n\n###### Healthy range:\n*Under 30%.*\n\n###### Watch for:\n*A majority of peers on a newer version, a strong upgrade signal.*\n\n###### Keywords:\n- **Peer** *(per node)* \u2014 another server this node holds a protocol connection to.\n- **Overlay** *(network-wide)* \u2014 the peer-to-peer mesh xrpld nodes form to gossip transactions, proposals and validations.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerPeerQualityGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#peer)", + "description": "###### What this is:\n*Percentage of connected peers running a newer rippled version than this node.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the higher-version peer percentage.*\n\n###### Reading it:\n*A high percentage suggests this node is behind and should be upgraded.*\n\n###### Healthy range:\n*Under 30%.*\n\n###### Watch for:\n*A majority of peers on a newer version, a strong upgrade signal.*\n\n###### Keywords:\n- **Peer** *(per node)* \u2014 another server this node holds a protocol connection to.\n- **Overlay** *(network-wide)* \u2014 the peer-to-peer mesh xrpld nodes form to gossip transactions, proposals and validations.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerPeerQualityGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#peer)", "type": "stat", "gridPos": { "h": 10, @@ -262,7 +262,7 @@ }, { "title": "Upgrade Recommended [$xrpl_network_type]", - "description": "###### What this is:\n*A flag indicating whether an upgrade is advised based on peer version analysis (Yes/No).*\n\n###### How it's computed:\n*Instantaneous gauge reading of the upgrade-recommended flag.*\n\n###### Reading it:\n*No is healthy; Yes means most peers run a newer version.*\n\n###### Healthy range:\n*No.*\n\n###### Watch for:\n*A Yes state, indicating the node risks falling out of step with the network.*\n\n###### Keywords:\n- **Peer** *(per node)* \u2014 another server this node holds a protocol connection to.\n- **Overlay** *(network-wide)* \u2014 the peer-to-peer mesh xrpld nodes form to gossip transactions, proposals and validations.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerPeerQualityGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#peer)", + "description": "###### What this is:\n*A flag indicating whether an upgrade is advised based on peer version analysis (Yes/No).*\n\n###### How it's computed:\n*Instantaneous gauge reading of the upgrade-recommended flag.*\n\n###### Reading it:\n*No is healthy; Yes means most peers run a newer version.*\n\n###### Healthy range:\n*No.*\n\n###### Watch for:\n*A Yes state, indicating the node risks falling out of step with the network.*\n\n###### Keywords:\n- **Peer** *(per node)* \u2014 another server this node holds a protocol connection to.\n- **Overlay** *(network-wide)* \u2014 the peer-to-peer mesh xrpld nodes form to gossip transactions, proposals and validations.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerPeerQualityGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#peer)", "type": "stat", "gridPos": { "h": 10, @@ -434,7 +434,7 @@ }, { "title": "Resource Disconnects", - "description": "###### What this is:\n*Cumulative count of peers dropped for exceeding resource (load) limits.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the resource-disconnect total over time.*\n\n###### Reading it:\n*A flat line is healthy; a rising line means peers are being dropped for overuse.*\n\n###### Healthy range:\n*Flat / near constant.*\n\n###### Watch for:\n*A steep climb, which flags aggressive or misbehaving peers being shed as backpressure.*\n\n###### Keywords:\n- **Resource disconnect** *(per node)* \u2014 a peer dropped for exceeding resource/load limits \u2014 the node shedding abusive or overactive peers.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#resource-disconnect)", + "description": "###### What this is:\n*Cumulative count of peers dropped for exceeding resource (load) limits.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the resource-disconnect total over time.*\n\n###### Reading it:\n*A flat line is healthy; a rising line means peers are being dropped for overuse.*\n\n###### Healthy range:\n*Flat / near constant.*\n\n###### Watch for:\n*A steep climb, which flags aggressive or misbehaving peers being shed as backpressure.*\n\n###### Keywords:\n- **Resource disconnect** *(per node)* \u2014 a peer dropped for exceeding resource/load limits \u2014 the node shedding abusive or overactive peers.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerServerInfoGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#resource-disconnect)", "type": "timeseries", "gridPos": { "h": 10, diff --git a/docker/telemetry/grafana/dashboards/rpc-performance.json b/docker/telemetry/grafana/dashboards/rpc-performance.json index 57f9d6b005..e5743792cc 100644 --- a/docker/telemetry/grafana/dashboards/rpc-performance.json +++ b/docker/telemetry/grafana/dashboards/rpc-performance.json @@ -558,7 +558,7 @@ }, { "title": "Current RPC Latency (p99 Gauge) [$xrpl_network_type]", - "description": "###### What this is:\n*Current tail latency (p99) of RPC handling across all methods, as a live gauge.*\n\n###### How it's computed:\n*p99 of the method-duration histogram over the recent window.*\n\n###### Reading it:\n*A single at-a-glance number for current RPC responsiveness.*\n\n###### Healthy range:\n*Low-millisecond under normal load.*\n\n###### Watch for:\n*Sustained elevation, indicating the node is under query pressure.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcFinished`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", + "description": "###### What this is:\n*Current tail latency (p99) of RPC handling across all methods, as a live gauge.*\n\n###### How it's computed:\n*p99 of the method-duration histogram over the recent window.*\n\n###### Reading it:\n*A single at-a-glance number for current RPC responsiveness.*\n\n###### Healthy range:\n*Low-millisecond under normal load.*\n\n###### Watch for:\n*Sustained elevation, indicating the node is under query pressure.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcFinished`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", "type": "gauge", "gridPos": { "h": 10, @@ -618,7 +618,7 @@ }, { "title": "RPC Call Rate (All Methods)", - "description": "###### What this is:\n*Overall rate of RPC method calls that started, finished, and errored, across all methods.*\n\n###### How it's computed:\n*Per-second rate of each counter over a 5-minute window, summed per node.*\n\n###### Reading it:\n*Started should closely track finished; errored should be a small fraction.*\n\n###### Healthy range:\n*Workload-dependent; started \u2248 finished, errored near zero.*\n\n###### Watch for:\n*A growing gap between started and finished (calls hanging), or an errored line that rises with load.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcStarted / recordRpcFinished / recordRpcErrored`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", + "description": "###### What this is:\n*Overall rate of RPC method calls that started, finished, and errored, across all methods.*\n\n###### How it's computed:\n*Per-second rate of each counter over a 5-minute window, summed per node.*\n\n###### Reading it:\n*Started should closely track finished; errored should be a small fraction.*\n\n###### Healthy range:\n*Workload-dependent; started \u2248 finished, errored near zero.*\n\n###### Watch for:\n*A growing gap between started and finished (calls hanging), or an errored line that rises with load.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcStarted / recordRpcFinished / recordRpcErrored`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", "type": "timeseries", "gridPos": { "h": 10, @@ -681,7 +681,7 @@ }, { "title": "Per-Method Call Rate (Top 10)", - "description": "###### What this is:\n*The ten busiest RPC methods by call rate.*\n\n###### How it's computed:\n*Per-second start rate over 5 minutes, per method, showing the top ten.*\n\n###### Reading it:\n*Identifies which methods dominate load; the mix shifts with client behaviour.*\n\n###### Healthy range:\n*Workload-dependent.*\n\n###### Watch for:\n*A single method suddenly dominating, which can signal a runaway client or abusive query pattern.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcStarted`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", + "description": "###### What this is:\n*The ten busiest RPC methods by call rate.*\n\n###### How it's computed:\n*Per-second start rate over 5 minutes, per method, showing the top ten.*\n\n###### Reading it:\n*Identifies which methods dominate load; the mix shifts with client behaviour.*\n\n###### Healthy range:\n*Workload-dependent.*\n\n###### Watch for:\n*A single method suddenly dominating, which can signal a runaway client or abusive query pattern.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcStarted`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", "type": "timeseries", "gridPos": { "h": 10, @@ -735,7 +735,7 @@ }, { "title": "Per-Method Error Rate (Top 10)", - "description": "###### What this is:\n*The ten RPC methods producing the most errors.*\n\n###### How it's computed:\n*Per-second error rate over 5 minutes, per method, top ten.*\n\n###### Reading it:\n*Normally near zero; persistent errors point to a specific failing method.*\n\n###### Healthy range:\n*Near zero for well-behaved traffic.*\n\n###### Watch for:\n*Sustained errors concentrated on one method \u2014 a broken client, a bad input, or probing.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcErrored`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", + "description": "###### What this is:\n*The ten RPC methods producing the most errors.*\n\n###### How it's computed:\n*Per-second error rate over 5 minutes, per method, top ten.*\n\n###### Reading it:\n*Normally near zero; persistent errors point to a specific failing method.*\n\n###### Healthy range:\n*Near zero for well-behaved traffic.*\n\n###### Watch for:\n*Sustained errors concentrated on one method \u2014 a broken client, a bad input, or probing.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcErrored`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", "type": "timeseries", "gridPos": { "h": 10, @@ -789,7 +789,7 @@ }, { "title": "RPC Latency - All Methods", - "description": "###### What this is:\n*Aggregate RPC handler latency across all methods (p75 and p99).*\n\n###### How it's computed:\n*Percentiles of the method-duration histogram over a 5-minute window.*\n\n###### Reading it:\n*p75 reflects typical responsiveness; p99 captures the slow tail.*\n\n###### Healthy range:\n*Sub-millisecond to low-millisecond for light commands; heavier commands run longer.*\n\n###### Watch for:\n*A rising p99 while p75 stays flat \u2014 a subset of calls degrading, often from expensive queries.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcFinished`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", + "description": "###### What this is:\n*Aggregate RPC handler latency across all methods (p75 and p99).*\n\n###### How it's computed:\n*Percentiles of the method-duration histogram over a 5-minute window.*\n\n###### Reading it:\n*p75 reflects typical responsiveness; p99 captures the slow tail.*\n\n###### Healthy range:\n*Sub-millisecond to low-millisecond for light commands; heavier commands run longer.*\n\n###### Watch for:\n*A rising p99 while p75 stays flat \u2014 a subset of calls degrading, often from expensive queries.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcFinished`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", "type": "timeseries", "gridPos": { "h": 10, @@ -845,7 +845,7 @@ }, { "title": "Per-Method Latency (p99, Top 10 Slowest)", - "description": "###### What this is:\n*The ten slowest RPC methods by tail latency.*\n\n###### How it's computed:\n*p99 of each method's duration histogram over 5 minutes, top ten.*\n\n###### Reading it:\n*Surfaces which specific methods are expensive.*\n\n###### Healthy range:\n*Method-dependent; ledger/account queries are heavier than status calls.*\n\n###### Watch for:\n*A method whose p99 climbs over time, or an unexpectedly cheap method appearing here.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcFinished`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", + "description": "###### What this is:\n*The ten slowest RPC methods by tail latency.*\n\n###### How it's computed:\n*p99 of each method's duration histogram over 5 minutes, top ten.*\n\n###### Reading it:\n*Surfaces which specific methods are expensive.*\n\n###### Healthy range:\n*Method-dependent; ledger/account queries are heavier than status calls.*\n\n###### Watch for:\n*A method whose p99 climbs over time, or an unexpectedly cheap method appearing here.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcFinished`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", "type": "timeseries", "gridPos": { "h": 10, @@ -899,7 +899,7 @@ }, { "title": "RPC Error Ratio by Method", - "description": "###### What this is:\n*The methods with the highest error rates, for spotting failure hotspots.*\n\n###### How it's computed:\n*Per-second error rate over 5 minutes, per method, top ten.*\n\n###### Reading it:\n*Highlights where failures concentrate.*\n\n###### Healthy range:\n*Near zero.*\n\n###### Watch for:\n*One method with a persistently high error rate \u2014 malformed requests or targeted probing.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcErrored`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", + "description": "###### What this is:\n*The methods with the highest error rates, for spotting failure hotspots.*\n\n###### How it's computed:\n*Per-second error rate over 5 minutes, per method, top ten.*\n\n###### Reading it:\n*Highlights where failures concentrate.*\n\n###### Healthy range:\n*Near zero.*\n\n###### Watch for:\n*One method with a persistently high error rate \u2014 malformed requests or targeted probing.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`recordRpcErrored`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)", "type": "timeseries", "gridPos": { "h": 10, diff --git a/docker/telemetry/grafana/dashboards/validator-health.json b/docker/telemetry/grafana/dashboards/validator-health.json index 603ef3cda9..5746a7567d 100644 --- a/docker/telemetry/grafana/dashboards/validator-health.json +++ b/docker/telemetry/grafana/dashboards/validator-health.json @@ -71,7 +71,7 @@ }, { "title": "Agreement % (1h) [$xrpl_network_type]", - "description": "###### What this is:\n*Share of ledgers over the last hour where this validator agreed with the network consensus.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the 1-hour agreement percentage.*\n\n###### Reading it:\n*Higher is better; green at 95%+, yellow from 80%, red below.*\n\n###### Healthy range:\n*95-100%.*\n\n###### Watch for:\n*Values below 80%, meaning the validator frequently disagrees with consensus.*\n\n###### Keywords:\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#consensus)", + "description": "###### What this is:\n*Share of ledgers over the last hour where this validator agreed with the network consensus.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the 1-hour agreement percentage.*\n\n###### Reading it:\n*Higher is better; green at 95%+, yellow from 80%, red below.*\n\n###### Healthy range:\n*95-100%.*\n\n###### Watch for:\n*Values below 80%, meaning the validator frequently disagrees with consensus.*\n\n###### Keywords:\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#consensus)", "type": "stat", "gridPos": { "h": 10, @@ -128,7 +128,7 @@ }, { "title": "Agreement % (24h) [$xrpl_network_type]", - "description": "###### What this is:\n*Share of ledgers over the last 24 hours where this validator agreed with consensus.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the 24-hour agreement percentage.*\n\n###### Reading it:\n*Higher is better; a smoother, longer-term view than the 1h stat.*\n\n###### Healthy range:\n*95-100%.*\n\n###### Watch for:\n*A sustained dip below 90%, which can indicate configuration drift or a network partition.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", + "description": "###### What this is:\n*Share of ledgers over the last 24 hours where this validator agreed with consensus.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the 24-hour agreement percentage.*\n\n###### Reading it:\n*Higher is better; a smoother, longer-term view than the 1h stat.*\n\n###### Healthy range:\n*95-100%.*\n\n###### Watch for:\n*A sustained dip below 90%, which can indicate configuration drift or a network partition.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", "type": "stat", "gridPos": { "h": 10, @@ -185,7 +185,7 @@ }, { "title": "Agreements vs Missed (1h) [$xrpl_network_type]", - "description": "###### What this is:\n*Counts of agreed versus missed validations over the last hour.*\n\n###### How it's computed:\n*Instantaneous gauge readings of the 1-hour agreed and missed counts.*\n\n###### Reading it:\n*Agreements should dominate; missed should be small.*\n\n###### Healthy range:\n*Missed near 0.*\n\n###### Watch for:\n*A high missed count, meaning the validator is skipping consensus rounds.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n- **Consensus round** *(network event)* \u2014 one propose-and-revise iteration of consensus; several may run before validators converge on a ledger.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Consensus round](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", + "description": "###### What this is:\n*Counts of agreed versus missed validations over the last hour.*\n\n###### How it's computed:\n*Instantaneous gauge readings of the 1-hour agreed and missed counts.*\n\n###### Reading it:\n*Agreements should dominate; missed should be small.*\n\n###### Healthy range:\n*Missed near 0.*\n\n###### Watch for:\n*A high missed count, meaning the validator is skipping consensus rounds.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n- **Consensus round** *(network event)* \u2014 one propose-and-revise iteration of consensus; several may run before validators converge on a ledger.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Consensus round](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", "type": "bargauge", "gridPos": { "h": 10, @@ -257,7 +257,7 @@ }, { "title": "Agreements vs Missed (24h) [$xrpl_network_type]", - "description": "###### What this is:\n*Counts of agreed versus missed validations over the last 24 hours.*\n\n###### How it's computed:\n*Instantaneous gauge readings of the 24-hour agreed and missed counts.*\n\n###### Reading it:\n*Agreements should dominate over the full day.*\n\n###### Healthy range:\n*Missed a small fraction of agreements.*\n\n###### Watch for:\n*A growing missed share, signalling longer-term reliability problems.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", + "description": "###### What this is:\n*Counts of agreed versus missed validations over the last 24 hours.*\n\n###### How it's computed:\n*Instantaneous gauge readings of the 24-hour agreed and missed counts.*\n\n###### Reading it:\n*Agreements should dominate over the full day.*\n\n###### Healthy range:\n*Missed a small fraction of agreements.*\n\n###### Watch for:\n*A growing missed share, signalling longer-term reliability problems.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", "type": "bargauge", "gridPos": { "h": 10, @@ -342,7 +342,7 @@ }, { "title": "Validation Rate [$xrpl_network_type]", - "description": "###### What this is:\n*Validations this node sends per minute.*\n\n###### How it's computed:\n*Per-second rate of the sent-validations counter over 5 minutes, scaled to per minute.*\n\n###### Reading it:\n*Should track the ledger close cadence; roughly one validation per closed ledger.*\n\n###### Healthy range:\n*About 12-20 per minute (one per closed ledger, ~3-5s close).*\n\n###### Watch for:\n*A drop toward zero, meaning the validator has stopped participating.*\n\n###### Keywords:\n- **Validations checked vs sent** *(per node)* \u2014 validations this node received from peers and checked, versus validations it issued itself.\n- **Ledger close interval** *(network-wide)* \u2014 the network's steady ledger rhythm \u2014 roughly one closed ledger every 3-5 seconds on Mainnet.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementValidationsSent (caller RCLConsensus.cpp)`\n\n###### References:\n[Validations checked vs sent](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Ledger close interval](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validations-checked-vs-sent)", + "description": "###### What this is:\n*Validations this node sends per minute.*\n\n###### How it's computed:\n*Per-second rate of the sent-validations counter over 5 minutes, scaled to per minute.*\n\n###### Reading it:\n*Should track the ledger close cadence; roughly one validation per closed ledger.*\n\n###### Healthy range:\n*About 12-20 per minute (one per closed ledger, ~3-5s close).*\n\n###### Watch for:\n*A drop toward zero, meaning the validator has stopped participating.*\n\n###### Keywords:\n- **Validations checked vs sent** *(per node)* \u2014 validations this node received from peers and checked, versus validations it issued itself.\n- **Ledger close interval** *(network-wide)* \u2014 the network's steady ledger rhythm \u2014 roughly one closed ledger every 3-5 seconds on Mainnet.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementValidationsSent (caller RCLConsensus.cpp)`\n\n###### References:\n[Validations checked vs sent](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Ledger close interval](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validations-checked-vs-sent)", "type": "stat", "gridPos": { "h": 10, @@ -397,7 +397,7 @@ }, { "title": "Validations Checked Rate [$xrpl_network_type]", - "description": "###### What this is:\n*Validations received from peers and checked per minute.*\n\n###### How it's computed:\n*Per-second rate of the checked-validations counter over 5 minutes, scaled to per minute.*\n\n###### Reading it:\n*Reflects how much validation traffic the network is delivering to this node.*\n\n###### Healthy range:\n*Workload-dependent; scales with trusted validator count.*\n\n###### Watch for:\n*A sudden collapse, which suggests peer connectivity loss or network isolation.*\n\n###### Keywords:\n- **Validations checked vs sent** *(per node)* \u2014 validations this node received from peers and checked, versus validations it issued itself.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementValidationsChecked (caller NetworkOPs.cpp)`\n\n###### References:\n[Validations checked vs sent](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validations-checked-vs-sent)", + "description": "###### What this is:\n*Validations received from peers and checked per minute.*\n\n###### How it's computed:\n*Per-second rate of the checked-validations counter over 5 minutes, scaled to per minute.*\n\n###### Reading it:\n*Reflects how much validation traffic the network is delivering to this node.*\n\n###### Healthy range:\n*Workload-dependent; scales with trusted validator count.*\n\n###### Watch for:\n*A sudden collapse, which suggests peer connectivity loss or network isolation.*\n\n###### Keywords:\n- **Validations checked vs sent** *(per node)* \u2014 validations this node received from peers and checked, versus validations it issued itself.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementValidationsChecked (caller NetworkOPs.cpp)`\n\n###### References:\n[Validations checked vs sent](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validations-checked-vs-sent)", "type": "stat", "gridPos": { "h": 10, @@ -436,7 +436,7 @@ }, { "title": "Amendment Blocked [$xrpl_network_type]", - "description": "###### What this is:\n*Whether the node is amendment-blocked (OK/BLOCKED).*\n\n###### How it's computed:\n*Instantaneous gauge reading of the amendment-blocked flag.*\n\n###### Reading it:\n*OK is healthy; BLOCKED means an enabled amendment is unsupported by this build.*\n\n###### Healthy range:\n*OK.*\n\n###### Watch for:\n*A BLOCKED state, which halts validation and requires a software upgrade.*\n\n###### Keywords:\n- **Amendment blocked** *(per node)* \u2014 the node has halted because the network enabled an amendment its software version does not support.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidatorHealthGauge`\n\n###### References:\n[Amendment blocked](https://xrpl.org/docs/concepts/networks-and-servers/amendments) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#amendment-blocked)", + "description": "###### What this is:\n*Whether the node is amendment-blocked (OK/BLOCKED).*\n\n###### How it's computed:\n*Instantaneous gauge reading of the amendment-blocked flag.*\n\n###### Reading it:\n*OK is healthy; BLOCKED means an enabled amendment is unsupported by this build.*\n\n###### Healthy range:\n*OK.*\n\n###### Watch for:\n*A BLOCKED state, which halts validation and requires a software upgrade.*\n\n###### Keywords:\n- **Amendment blocked** *(per node)* \u2014 the node has halted because the network enabled an amendment its software version does not support.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidatorHealthGauge`\n\n###### References:\n[Amendment blocked](https://xrpl.org/docs/concepts/networks-and-servers/amendments) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#amendment-blocked)", "type": "stat", "gridPos": { "h": 10, @@ -507,7 +507,7 @@ }, { "title": "UNL Expiry (days) [$xrpl_network_type]", - "description": "###### What this is:\n*Days remaining until the current UNL (trusted validator list) expires.*\n\n###### How it's computed:\n*Instantaneous gauge reading of days-to-expiry.*\n\n###### Reading it:\n*Higher is safer; green at 30+, yellow under 7, red at expiry.*\n\n###### Healthy range:\n*30+ days.*\n\n###### Watch for:\n*Fewer than 7 days, after which the node loses its trusted validator set if not renewed.*\n\n###### Keywords:\n- **UNL (Unique Node List)** *(per node)* \u2014 the list of validators a node trusts not to collude; the basis for its consensus and quorum.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidatorHealthGauge`\n\n###### References:\n[UNL (Unique Node List)](https://xrpl.org/docs/concepts/consensus-protocol/unl) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#unl-unique-node-list)", + "description": "###### What this is:\n*Days remaining until the current UNL (trusted validator list) expires.*\n\n###### How it's computed:\n*Instantaneous gauge reading of days-to-expiry.*\n\n###### Reading it:\n*Higher is safer; green at 30+, yellow under 7, red at expiry.*\n\n###### Healthy range:\n*30+ days.*\n\n###### Watch for:\n*Fewer than 7 days, after which the node loses its trusted validator set if not renewed.*\n\n###### Keywords:\n- **UNL (Unique Node List)** *(per node)* \u2014 the list of validators a node trusts not to collude; the basis for its consensus and quorum.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidatorHealthGauge`\n\n###### References:\n[UNL (Unique Node List)](https://xrpl.org/docs/concepts/consensus-protocol/unl) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#unl-unique-node-list)", "type": "stat", "gridPos": { "h": 10, @@ -562,7 +562,7 @@ }, { "title": "UNL Blocked [$xrpl_network_type]", - "description": "###### What this is:\n*Whether the node's UNL is blocked (OK/BLOCKED).*\n\n###### How it's computed:\n*Instantaneous gauge reading of the UNL-blocked flag.*\n\n###### Reading it:\n*OK is healthy; BLOCKED means validator trust cannot be established.*\n\n###### Healthy range:\n*OK.*\n\n###### Watch for:\n*A BLOCKED state, which can stop the node participating in consensus.*\n\n###### Keywords:\n- **UNL (Unique Node List)** *(per node)* \u2014 the list of validators a node trusts not to collude; the basis for its consensus and quorum.\n- **UNL blocked** *(per node)* \u2014 the node cannot establish a usable trusted validator list, so it cannot safely validate.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidatorHealthGauge`\n\n###### References:\n[UNL (Unique Node List)](https://xrpl.org/docs/concepts/consensus-protocol/unl) \u00b7 [UNL blocked](https://xrpl.org/docs/concepts/consensus-protocol) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#unl-unique-node-list)", + "description": "###### What this is:\n*Whether the node's UNL is blocked (OK/BLOCKED).*\n\n###### How it's computed:\n*Instantaneous gauge reading of the UNL-blocked flag.*\n\n###### Reading it:\n*OK is healthy; BLOCKED means validator trust cannot be established.*\n\n###### Healthy range:\n*OK.*\n\n###### Watch for:\n*A BLOCKED state, which can stop the node participating in consensus.*\n\n###### Keywords:\n- **UNL (Unique Node List)** *(per node)* \u2014 the list of validators a node trusts not to collude; the basis for its consensus and quorum.\n- **UNL blocked** *(per node)* \u2014 the node cannot establish a usable trusted validator list, so it cannot safely validate.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidatorHealthGauge`\n\n###### References:\n[UNL (Unique Node List)](https://xrpl.org/docs/concepts/consensus-protocol/unl) \u00b7 [UNL blocked](https://xrpl.org/docs/concepts/consensus-protocol) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#unl-unique-node-list)", "type": "stat", "gridPos": { "h": 10, @@ -633,7 +633,7 @@ }, { "title": "Agreement/Missed Counters (Rate)", - "description": "###### What this is:\n*Rate of cumulative agreement and missed-validation counters per minute.*\n\n###### How it's computed:\n*Per-second rate of each monotonic counter over 5 minutes, scaled to per minute.*\n\n###### Reading it:\n*Agreements should dominate; the missed line should stay low.*\n\n###### Healthy range:\n*Missed rate near 0.*\n\n###### Watch for:\n*A rising missed rate, complementing the windowed agreement percentages above.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidationTotalsCounters`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", + "description": "###### What this is:\n*Rate of cumulative agreement and missed-validation counters per minute.*\n\n###### How it's computed:\n*Per-second rate of each monotonic counter over 5 minutes, scaled to per minute.*\n\n###### Reading it:\n*Agreements should dominate; the missed line should stay low.*\n\n###### Healthy range:\n*Missed rate near 0.*\n\n###### Watch for:\n*A rising missed rate, complementing the windowed agreement percentages above.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidationTotalsCounters`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", "type": "timeseries", "gridPos": { "h": 10, @@ -718,7 +718,7 @@ }, { "title": "Validation Quorum [$xrpl_network_type]", - "description": "###### What this is:\n*Minimum number of trusted validations required to declare a ledger fully validated.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the current quorum requirement.*\n\n###### Reading it:\n*Tracks the quorum derived from the active validator list; changes when the list changes.*\n\n###### Healthy range:\n*Stable at the network-appropriate value.*\n\n###### Watch for:\n*An unexpected drop, which can weaken consensus safety guarantees.*\n\n###### Keywords:\n- **Validation quorum** *(network-wide)* \u2014 the minimum number of agreeing trusted validations needed to declare a ledger fully validated.\n- **Validator list** *(network-wide)* \u2014 signed lists of recommended validators (UNLs) that peers distribute to each other.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidatorHealthGauge`\n\n###### References:\n[Validation quorum](https://xrpl.org/docs/concepts/consensus-protocol/negative-unl) \u00b7 [Validator list](https://xrpl.org/docs/concepts/consensus-protocol/unl) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-quorum)", + "description": "###### What this is:\n*Minimum number of trusted validations required to declare a ledger fully validated.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the current quorum requirement.*\n\n###### Reading it:\n*Tracks the quorum derived from the active validator list; changes when the list changes.*\n\n###### Healthy range:\n*Stable at the network-appropriate value.*\n\n###### Watch for:\n*An unexpected drop, which can weaken consensus safety guarantees.*\n\n###### Keywords:\n- **Validation quorum** *(network-wide)* \u2014 the minimum number of agreeing trusted validations needed to declare a ledger fully validated.\n- **Validator list** *(network-wide)* \u2014 signed lists of recommended validators (UNLs) that peers distribute to each other.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidatorHealthGauge`\n\n###### References:\n[Validation quorum](https://xrpl.org/docs/concepts/consensus-protocol/negative-unl) \u00b7 [Validator list](https://xrpl.org/docs/concepts/consensus-protocol/unl) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-quorum)", "type": "stat", "gridPos": { "h": 10, @@ -758,7 +758,7 @@ }, { "title": "Time in Current State [$xrpl_network_type]", - "description": "###### What this is:\n*How long the server has held its current operating state, in seconds.*\n\n###### How it's computed:\n*Current value of the time-in-state gauge.*\n\n###### Reading it:\n*Not yet wired in the code; the value currently always reads 0.*\n\n###### Healthy range:\n*Not applicable; the value is always 0 today.*\n\n###### Watch for:\n*n/a until implemented.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerStateTrackingGauge`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", + "description": "###### What this is:\n*How long the server has held its current operating state, in seconds.*\n\n###### How it's computed:\n*Current value of the time-in-state gauge.*\n\n###### Reading it:\n*Not yet wired in the code; the value currently always reads 0.*\n\n###### Healthy range:\n*Not applicable; the value is always 0 today.*\n\n###### Watch for:\n*n/a until implemented.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerStateTrackingGauge`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", "type": "stat", "gridPos": { "h": 10, @@ -797,7 +797,7 @@ }, { "title": "State Changes Rate [$xrpl_network_type]", - "description": "###### What this is:\n*Rate of server operating-state changes per hour.*\n\n###### How it's computed:\n*Per-hour rate of the state-change counter, averaged over a 1-hour window.*\n\n###### Reading it:\n*Near zero is healthy; each increment is one state transition.*\n\n###### Healthy range:\n*Near 0 changes per hour.*\n\n###### Watch for:\n*Frequent transitions, which point to network instability or configuration problems.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementStateChanges (caller NetworkOPs.cpp)`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", + "description": "###### What this is:\n*Rate of server operating-state changes per hour.*\n\n###### How it's computed:\n*Per-hour rate of the state-change counter, averaged over a 1-hour window.*\n\n###### Reading it:\n*Near zero is healthy; each increment is one state transition.*\n\n###### Healthy range:\n*Near 0 changes per hour.*\n\n###### Watch for:\n*Frequent transitions, which point to network instability or configuration problems.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementStateChanges (caller NetworkOPs.cpp)`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", "type": "stat", "gridPos": { "h": 10, @@ -852,7 +852,7 @@ }, { "title": "Ledgers Closed Rate [$xrpl_network_type]", - "description": "###### What this is:\n*Ledgers closed per minute by this node.*\n\n###### How it's computed:\n*Per-second rate of the ledgers-closed counter over 5 minutes, scaled to per minute.*\n\n###### Reading it:\n*Should match the network's steady close cadence.*\n\n###### Healthy range:\n*About 12-20 per minute (one per closed ledger, ~3-5s close).*\n\n###### Watch for:\n*Deviation from the expected cadence, which indicates consensus timing trouble or the node falling behind.*\n\n###### Keywords:\n- **Ledgers closed rate** *(per node)* \u2014 how many ledgers this node closed per minute; should match the network close cadence.\n- **Ledger close interval** *(network-wide)* \u2014 the network's steady ledger rhythm \u2014 roughly one closed ledger every 3-5 seconds on Mainnet.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementLedgersClosed (caller RCLConsensus.cpp)`\n\n###### References:\n[Ledgers closed rate](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledgers-closed-rate)", + "description": "###### What this is:\n*Ledgers closed per minute by this node.*\n\n###### How it's computed:\n*Per-second rate of the ledgers-closed counter over 5 minutes, scaled to per minute.*\n\n###### Reading it:\n*Should match the network's steady close cadence.*\n\n###### Healthy range:\n*About 12-20 per minute (one per closed ledger, ~3-5s close).*\n\n###### Watch for:\n*Deviation from the expected cadence, which indicates consensus timing trouble or the node falling behind.*\n\n###### Keywords:\n- **Ledgers closed rate** *(per node)* \u2014 how many ledgers this node closed per minute; should match the network close cadence.\n- **Ledger close interval** *(network-wide)* \u2014 the network's steady ledger rhythm \u2014 roughly one closed ledger every 3-5 seconds on Mainnet.\n- **Consensus** *(network event)* \u2014 the protocol by which validators agree on the next ledger's transaction set and close time.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`incrementLedgersClosed (caller RCLConsensus.cpp)`\n\n###### References:\n[Ledgers closed rate](https://xrpl.org/docs/concepts/ledgers/ledger-close-times) \u00b7 [Consensus](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#ledgers-closed-rate)", "type": "stat", "gridPos": { "h": 10, @@ -907,7 +907,7 @@ }, { "title": "Agreement % (7d) [$xrpl_network_type]", - "description": "###### What this is:\n*Share of ledgers over the trailing 7 days where this validator agreed with consensus.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the 7-day agreement percentage.*\n\n###### Reading it:\n*The long-term reliability window; higher is better.*\n\n###### Healthy range:\n*95-100%.*\n\n###### Watch for:\n*A gradual decline, which reflects chronic rather than transient disagreement.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", + "description": "###### What this is:\n*Share of ledgers over the trailing 7 days where this validator agreed with consensus.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the 7-day agreement percentage.*\n\n###### Reading it:\n*The long-term reliability window; higher is better.*\n\n###### Healthy range:\n*95-100%.*\n\n###### Watch for:\n*A gradual decline, which reflects chronic rather than transient disagreement.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", "type": "stat", "gridPos": { "h": 10, @@ -964,7 +964,7 @@ }, { "title": "State Value Timeline", - "description": "###### What this is:\n*Numeric encoding of the server operating state (disconnected, connected, syncing, tracking, full, validating, proposing) over time.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the encoded state value.*\n\n###### Reading it:\n*A flat line at the full-operation state is healthy; steps show transitions.*\n\n###### Healthy range:\n*Steady at the highest (full) state.*\n\n###### Watch for:\n*Frequent transitions, useful for correlating state flapping with other metrics.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n- **Consensus mode** *(per node)* \u2014 the node's role/health in the current round: Proposing, Observing, Wrong Ledger, or Switched Ledger.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerStateTrackingGauge`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Consensus mode](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", + "description": "###### What this is:\n*Numeric encoding of the server operating state (disconnected, connected, syncing, tracking, full, validating, proposing) over time.*\n\n###### How it's computed:\n*Instantaneous gauge reading of the encoded state value.*\n\n###### Reading it:\n*A flat line at the full-operation state is healthy; steps show transitions.*\n\n###### Healthy range:\n*Steady at the highest (full) state.*\n\n###### Watch for:\n*Frequent transitions, useful for correlating state flapping with other metrics.*\n\n###### Keywords:\n- **Operating mode / server state** *(per node)* \u2014 the node's sync level: Disconnected, Connected, Syncing, Tracking, Full (and Validating/Proposing).\n- **Consensus mode** *(per node)* \u2014 the node's role/health in the current round: Proposing, Observing, Wrong Ledger, or Switched Ledger.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerStateTrackingGauge`\n\n###### References:\n[Operating mode / server state](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states) \u00b7 [Consensus mode](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#operating-mode-server-state)", "type": "timeseries", "gridPos": { "h": 10, @@ -1013,7 +1013,7 @@ }, { "title": "Agreements vs Missed (7d)", - "description": "###### What this is:\n*Agreed versus missed validation counts over the trailing 7 days.*\n\n###### How it's computed:\n*Instantaneous gauge readings of the 7-day agreed and missed counts.*\n\n###### Reading it:\n*Agreements should dominate across the week.*\n\n###### Healthy range:\n*Missed a small fraction of agreements.*\n\n###### Watch for:\n*A rising missed trend, signalling sustained validator unreliability.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", + "description": "###### What this is:\n*Agreed versus missed validation counts over the trailing 7 days.*\n\n###### How it's computed:\n*Instantaneous gauge readings of the 7-day agreed and missed counts.*\n\n###### Reading it:\n*Agreements should dominate across the week.*\n\n###### Healthy range:\n*Missed a small fraction of agreements.*\n\n###### Watch for:\n*A rising missed trend, signalling sustained validator unreliability.*\n\n###### Keywords:\n- **Validation agreement** *(per node)* \u2014 the share of ledgers where this validator's validation matched network consensus versus was missed.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[AppMetricGauges.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/AppMetricGauges.cpp)\n\n###### Function:\n`registerValidationAgreementGauge`\n\n###### References:\n[Validation agreement](https://xrpl.org/docs/concepts/consensus-protocol/consensus-structure#validation) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#validation-agreement)", "type": "timeseries", "gridPos": { "h": 10, diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index 1274ee51df..ea56ec3d5f 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -1829,8 +1829,10 @@ ledger acquisition deferring". Use `acquire_ledger_deferrals` and These five come from the `PerfLog` job hooks, not from beast::insight, so they are exported by the `MetricsRegistry` meter. `job_queued_us` and `job_running_us` have explicit microsecond bucket views registered -(`addMicrosecondHistogramView()` calls at MetricsRegistry.cpp:310-311; the helper -itself is at `:197`) spanning 100 µs to 60 s; without those the SDK default +(`addMicrosecondHistogramView()`, called from +`MetricsRegistry::initExporterAndProvider()` — both live in +`src/libxrpl/telemetry/MetricsRegistry.cpp`) spanning 100 µs to 60 s; without +those the SDK default buckets stop at 10 ms and every quantile saturates. | Prometheus Metric | Kind | Labels | Description | @@ -1862,7 +1864,7 @@ two production job names embed a ledger sequence number: A raw label would mint a new Prometheus series for every ledger — unbounded growth at ~1 series every 3-5 s, forever. `MetricsRegistry::sanitiseHandler()` (declared inline in -`src/xrpld/telemetry/MetricsRegistry.h`) therefore applies one rule: +`include/xrpl/telemetry/MetricsRegistry.h`) therefore applies one rule: - Keep the name when it is **non-empty and every character is an ASCII letter**. - Otherwise return the constant `"other"`. An empty name, a digit, a hyphen, or @@ -1989,7 +1991,7 @@ rpc_batch_size_count - rpc_batch_size_bucket{le="12288"} -Use the call-site macros in `src/xrpld/telemetry/MetricMacros.h` -- no +Use the call-site macros in `include/xrpl/telemetry/MetricMacros.h` -- no `MetricsRegistry.h`/`.cpp` edit is needed for any of these: | Need | Macro | @@ -2001,7 +2003,7 @@ Use the call-site macros in `src/xrpld/telemetry/MetricMacros.h` -- no | Value your own code already tracks, sampled on a timer | `XRPL_METRIC_OBSERVABLE_GAUGE_REGISTER` / `_COUNTER_REGISTER` / `_UPDOWN_REGISTER` | ```cpp -#include +#include // Monotonic counter: XRPL_METRIC_COUNTER_INC(app_, "my_new_thing_total", "Description of what this counts"); @@ -2018,8 +2020,9 @@ XRPL_METRIC_OBSERVABLE_GAUGE_REGISTER(app_, "my_thing_size", "Current size", Counters use a `_total` suffix by convention. A histogram whose values can exceed ~10,000 units (e.g. a microsecond duration beyond 10ms) still needs one -line added to `addMicrosecondHistogramView()` in `MetricsRegistry.cpp` -- the -only case that still touches a central file. There is no way to read a metric's +line added to `addMicrosecondHistogramView()` in +`src/libxrpl/telemetry/MetricsRegistry.cpp` -- the only case that still touches a +central file. There is no way to read a metric's current value back from application code -- OTel's API is write-only by design; keep your own state if your logic needs to both record and read a running value (see the Doxygen header in `MetricMacros.h` for the full explanation). @@ -2289,10 +2292,12 @@ Requires `trace_peer=1` in the `[telemetry]` config section. > `{quantile="$quantile"}` matches nothing and reports no error. The job queue > exposes two parallel families: `job_running_us` / `job_queued_us` > (`MetricsRegistry` instruments, labelled by `job_type` and `handler`, -> microseconds — what these panels use; -> [MetricsRegistry.cpp:94-95](../src/xrpld/telemetry/MetricsRegistry.cpp#L94), -> [363-366](../src/xrpld/telemetry/MetricsRegistry.cpp#L363), recorded from the -> `PerfLog` job hooks at +> microseconds — what these panels use; the two names come from the +> `kJobQueuedDurationUs` / `kJobRunningDurationUs` constants and the microsecond +> buckets from `addMicrosecondHistogramView()` in +> `MetricsRegistry::initExporterAndProvider()`, all in +> [MetricsRegistry.cpp](../src/libxrpl/telemetry/MetricsRegistry.cpp), recorded +> from the `PerfLog` job hooks at > [PerfLogImp.cpp:432](../src/xrpld/perflog/detail/PerfLogImp.cpp#L432)) and > `jobq_[_q]_milliseconds` > (beast::insight, one instrument per job type, milliseconds — diff --git a/src/xrpld/telemetry/MetricMacros.h b/include/xrpl/telemetry/MetricMacros.h similarity index 99% rename from src/xrpld/telemetry/MetricMacros.h rename to include/xrpl/telemetry/MetricMacros.h index 0dbf642222..31ef5db4c6 100644 --- a/src/xrpld/telemetry/MetricMacros.h +++ b/include/xrpl/telemetry/MetricMacros.h @@ -132,9 +132,8 @@ #include #endif -#include // IWYU pragma: keep - -#include // IWYU pragma: keep +#include // IWYU pragma: keep +#include // IWYU pragma: keep #ifdef XRPL_ENABLE_TELEMETRY diff --git a/src/xrpld/telemetry/MetricsRegistry.h b/include/xrpl/telemetry/MetricsRegistry.h similarity index 62% rename from src/xrpld/telemetry/MetricsRegistry.h rename to include/xrpl/telemetry/MetricsRegistry.h index c9c4f5b757..a9624632d9 100644 --- a/src/xrpld/telemetry/MetricsRegistry.h +++ b/include/xrpl/telemetry/MetricsRegistry.h @@ -1,22 +1,24 @@ #pragma once /** - * Central OTel Metrics Registry for xrpld. + * Central OTel metrics registry: the export pipeline and the instruments that + * app code pushes values into. * - * Owns all OpenTelemetry metric instruments (counters, histograms, - * observable gauges) that are NOT already covered by the beast::insight - * StatsD pipeline. The instruments are created once at startup and polled - * by the OTel PeriodicExportingMetricReader at a configurable interval - * (default 10 s). + * Owns the OpenTelemetry MeterProvider, the OTLP/HTTP exporter, the periodic + * reader and every SYNCHRONOUS instrument (counters and histograms) that is + * not already covered by the beast::insight StatsD pipeline. The instruments + * are created once at startup and drained by the OTel + * PeriodicExportingMetricReader at a fixed interval (10 s). * * When XRPL_ENABLE_TELEMETRY is **not** defined, this class compiles to a * lightweight no-op: every public method is an empty inline. * + * Every caller reaches it through ServiceRegistry::getMetricsRegistry(), and + * the XRPL_METRIC_* macros then create their own instruments from meter(). + * * Dependency / ownership diagram (ASCII): * - * Application - * | - * +-- MetricsRegistry (unique_ptr, created in setup(), started/stopped with telemetry) + * MetricsRegistry * | * +-- OTel MeterProvider (owns reader + exporter) * | | @@ -41,42 +43,7 @@ * | +-- txq_expired_total * | +-- txq_dropped_total{reason} * | - * +-- ValidationTracker (validation agreement tracker) - * | - * +-- Observable Gauges (async callbacks, polled by reader) - * +-- Cache hit rates (SLE, ledger, AL) - * +-- TreeNode / FullBelow sizes - * +-- TxQ metrics - * +-- CountedObject counts - * +-- Load factor breakdown - * +-- NodeStore I/O gauges (totals, derived means, NuDB write queue, - * ledger-acquisition stall counters) - * +-- Server info (state, uptime, peers, consensus) - * +-- Build info (version label) - * +-- Complete ledger ranges (start/end pairs) - * +-- DB metrics (storage KB, fetch rate) - * +-- Validator health (amend blocked, UNL, quorum) - * +-- Peer quality (P90 latency, version spread) - * +-- Reduce-relay efficiency (selected/suppressed peers) - * +-- Ledger economy (fees, reserves, age) - * +-- State tracking (mode value, time in state) - * +-- Storage detail (NuDB sizes) - * +-- Validation agreement (1h/24h pct, counts) - * +-- jq_trans_overflow_total (observed from Overlay) - * - * Control-flow for async gauges: - * - * PeriodicExportingMetricReader (background thread, 10 s tick) - * | - * v - * OTel SDK invokes registered ObservableGauge callbacks - * | - * v - * Each callback reads current value from Application services - * (e.g. app.getTxQ().getMetrics(), app.getFeeTrack().getLoadFactor()) - * | - * v - * Result set is exported via OTLP/HTTP to the collector + * +-- ValidationTracker (rolling validation-agreement windows) * * Control-flow for synchronous instruments: * @@ -101,11 +68,7 @@ * // [telemetry] and [network_id], read by Application.cpp rather than * // through Telemetry::Setup. * metricsRegistry_(std::make_unique( - * telemetry_->isEnabled(), *this, journal, options)) - * - * // Later, in setup(), once overlay_ exists (the last of the services the - * // callbacks read). Phase 2 registers the observable instruments: - * metricsRegistry_->startAsyncGauges(); + * telemetry_->isEnabled(), journal, options)) * * // In PerfLogImp::rpcStart(): * if (auto* mr = app_.getMetricsRegistry()) @@ -123,8 +86,8 @@ * if (auto* mr = app_.getMetricsRegistry()) * mr->recordJobQueued("ledgerData", "ProcessLData"); * - * // Shutdown, before any service the callbacks read is stopped. Idempotent, - * // so run() and ~ApplicationImp both call it: + * // Shutdown, before any observer of live server state is torn down. + * // Idempotent, so run() and ~ApplicationImp both call it: * metricsRegistry_->stop(); * @endcode * @@ -133,20 +96,16 @@ * it reads isEnabled() to decide whether to initialize the OTel SDK, and * BEFORE every subsystem that records a metric. Declaration order in * ApplicationImp is the guarantee; keep the member where it is. - * - Observable gauge callbacks capture a reference to the Application; the - * Application must outlive the MetricsRegistry (guaranteed because - * MetricsRegistry is stopped before Application teardown). - * - If a new CountedObject type is added, it will NOT appear automatically - * in the object_count gauge; the callback iterates a fixed list. * - Adding a new synchronous instrument requires updating both the header * and the .cpp, then calling the new record*() method from the - * instrumentation site. + * instrumentation site. Prefer the XRPL_METRIC_* macros, which need + * neither. */ #ifdef XRPL_ENABLE_TELEMETRY -// The tracker is held and exposed only in this configuration, where the gauge -// callbacks that drain it exist. -#include +// The tracker is held and exposed only in this configuration, where the +// observable-gauge callbacks that drain it exist. +#include #endif #include @@ -168,84 +127,58 @@ #include #include -// These three serve only the telemetry-only members below, so they are guarded -// like their uses: std::atomic by callbacksDetached_, std::function by the -// ObserveFn sink, std::shared_ptr by provider_. +// These two serve only the telemetry-only members below, so they are guarded +// like their uses: std::atomic by phase_, std::shared_ptr by provider_. #include -#include #include #endif -namespace xrpl { - -class ServiceRegistry; - -// Defined in src/xrpld/app/ledger/AcquireStats.h. Forward-declared because -// only the gauge helpers in the .cpp touch it, and pulling an xrpld/app -// header in here would widen the dependencies of every file that includes -// this one. -class AcquireStats; - -namespace node_store { -class Database; -} // namespace node_store - -namespace telemetry { +namespace xrpl::telemetry { /** * Central OpenTelemetry metric registry. * - * Owns all OTel instruments (counters, histograms, observable gauges) - * that are not covered by the beast::insight StatsD pipeline. See the - * file-level header comment above for the full dependency diagram, - * gauge domain list, and usage examples. + * Owns the metrics export pipeline and every push-model instrument that the + * beast::insight StatsD pipeline does not already cover. See the file-level + * header comment above for the instrument inventory and usage examples. * * Class / collaborator diagram (ASCII): * - * +-----------------+ +-------------------+ - * | Application |------->| MetricsRegistry | - * +-----------------+ +-------------------+ - * | | | - * creates/owns v v v - * +-----------+ +---------+ +-------------------+ - * | Meter | | Counter | | ValidationTracker | - * | Provider | | /Hist. | | (rolling windows) | - * +-----------+ +---------+ +-------------------+ + * MetricsRegistry * | - * v - * Periodic reader thread (~10 s) - * -> ObservableGauge callbacks - * -> OTLP/HTTP export + * +-- creates/owns --> MeterProvider (SDK) + * | | + * | v + * | reader thread (~10 s) -> OTLP/HTTP export + * | + * +-- creates/owns --> Counter and Histogram instruments + * | + * +-- holds ----------> ValidationTracker (rolling windows) * * @note Thread safety: * - The recordRpc, recordJob, and increment methods are invoked - * from xrpld hot paths. OTel Counter::Add() and - * Histogram::Record() are documented thread-safe, and - * null-guard checks protect uninitialized instruments. - * - ObservableGauge callbacks run on the OTel SDK background - * reader thread (~10 s tick), concurrently with writers. - * Each callback reads only lock-protected or atomic state - * from Application services and wraps the body in a - * catch-all try block so a transient failure never crashes - * the reader thread. + * from hot paths. OTel Counter::Add() and Histogram::Record() + * are documented thread-safe, and null-guard checks protect + * uninitialized instruments. + * - recording() is a single acquire load and is read on every + * XRPL_METRIC_* call site, from any thread. + * - meter() may be called from any thread. The constructor is the + * last writer of the handle it returns; stop() leaves it alone. * - ValidationTracker protects its rolling windows internally. - * - The constructor, startAsyncGauges() and stop() are NOT thread-safe - * with each other and must all be called, in that order, from - * the single Application lifecycle thread. + * - The constructor, hasPipeline() and stop() are NOT thread-safe + * with each other. All three read or write provider_, a plain + * shared_ptr that stop() resets, so all three belong on the + * single server lifecycle thread, in that order. * - * @note Lifetime, in three phases (see Phase): + * @note Lifetime, in two phases (see Phase): * - Ready: the constructor built the pipeline and the synchronous * instruments. Runs in ApplicationImp's member-init list, so it precedes * every subsystem that could record. - * - GaugesArmed: startAsyncGauges() registered the observable callbacks. - * Runs once overlay_ exists, the last service those callbacks read. * - Stopped: stop() joined the reader thread. Runs before any observed * service stops, from run() and again from ~ApplicationImp for the * paths that never reach run(). * * @note Extending: - * - Adding a new CountedObject type is auto-picked up by the - * object_count gauge via iteration. * - Adding a new SYNCHRONOUS instrument (counter/histogram): prefer the * XRPL_METRIC_* call-site macros in MetricMacros.h -- no header/cpp * edit needed. Fall back to a dedicated member + init line + record @@ -253,8 +186,10 @@ namespace telemetry { * back by other code (e.g. ValidationTracker-style accumulation) or * needs a custom histogram bucket View (see the histogram note in * MetricMacros.h). - * - Adding a new OBSERVABLE gauge still requires eager central - * registration -- pull-model instruments cannot be lazily created. + * - An OBSERVABLE instrument does not belong here. Its callback reads live + * server state, so it must be registered only once that state exists, + * which is later than this object is built. Register it from the layer + * that owns those callbacks. */ class MetricsRegistry { @@ -281,14 +216,14 @@ public: * .serviceInstanceId = nodePublicKey, * .nodeId = nodePublicKey, * .networkId = 2}; - * MetricsRegistry registry(enabled, app, journal, opts); + * MetricsRegistry registry(enabled, journal, opts); * * // Edge case: mutual TLS to a collector that requires it. * opts.useTls = true; * opts.tlsCaCertPath = "/etc/xrpld/otel-ca.pem"; * opts.tlsClientCertPath = "/etc/xrpld/node.pem"; * opts.tlsClientKeyPath = "/etc/xrpld/node.key"; - * MetricsRegistry secure(enabled, app, journal, opts); + * MetricsRegistry secure(enabled, journal, opts); * @endcode * * @note Plain aggregate, no invariants enforced. `networkType` is not a @@ -372,25 +307,21 @@ public: * node. * * @note Invariant for future changes: the constructor may create only - * instruments with NO Application-reading callback. Push-model - * counters and histograms qualify; app code records into them - * when it is ready. Any observable instrument whose callback - * reads an Application service belongs in `startAsyncGauges()`, - * because registering it here arms the reader thread to invoke - * that callback against a half-built Application. This applies - * to observable COUNTERS as well as gauges. + * instruments with NO callback of their own. Push-model counters + * and histograms qualify; app code records into them when it is + * ready. An instrument registered here is live immediately, and + * the reader thread may invoke its callback before the rest of + * the server is built, so any observable whose callback reads + * live server state must be registered later, by the layer that + * owns those callbacks. This applies to observable COUNTERS as + * well as gauges. * * @param enabled False makes every method a no-op (telemetry disabled). - * @param app Services the observable-gauge callbacks read. * @param journal Log output. * @param options Endpoint, TLS settings and resource identity, all read * from config by the caller. See @ref Options. */ - MetricsRegistry( - bool enabled, - ServiceRegistry& app, - beast::Journal journal, - Options const& options); + MetricsRegistry(bool enabled, beast::Journal journal, Options const& options); /** * Stops the pipeline if run() or ~ApplicationImp did not already. @@ -404,60 +335,6 @@ public: MetricsRegistry& operator=(MetricsRegistry const&) = delete; - /** - * Register the pull-model observable instruments — the second startup - * phase. Mostly ObservableGauges, plus the ObservableCounters whose - * source value is already cumulative. - * - * A separate entry point from the constructor because the two halves have - * different prerequisites. The constructor needs only config strings; - * these callbacks read live Application services, so this half must run - * later. Registering an observable also arms the reader thread to invoke - * its callback on the next tick, which is why the separation is about - * ordering and not just tidiness. - * - * Calling it twice, or after stop(), logs a warning and does nothing. - * - * @pre Every service the callbacks read is constructed. The full set, - * from the `app.get*()` calls in the registration helpers, is: - * Overlay, OPs (NetworkOPs), LedgerMaster, OpenLedger, TxQ, - * NodeStore, NodeFamily, Validators, AcceptedLedgerCache, - * CachedSLEs, AcquireStats, TimeKeeper, RelationalDatabase, - * InboundLedgers and FeeTrack. - * Overlay is built last, so it fixes this call's position: - * `ServiceRegistry::getOverlay()` `XRPL_ASSERT`s that - * `overlay_` is non-null, and a reader-thread tick before the - * overlay exists aborts a Debug build. The callbacks' catch-all - * try block does not catch an assert. `getTxQ()` and - * `getRelationalDatabase()` assert likewise. - */ - void - startAsyncGauges(); - - /** - * Detach all ObservableGauge callbacks so they no-op on the next - * reader-thread tick. - * - * Must be called BEFORE any Application service that the callbacks - * read (nodeStore, overlay, networkOPs, ledgerMaster, etc.) is - * stopped. The flag is checked with acquire ordering at the top of - * every callback; together with the release store here it - * guarantees that once `detachCallbacks()` returns, no subsequent - * callback invocation will dereference an already-stopped service. - * - * Idempotent, and safe to call multiple times: the flag is one-way, - * only ever set to true, and nothing clears it. The actual - * SDK-level provider shutdown still happens in `stop()`. - * - * @note One-way means this is a shutdown-only call. Calling it before - * `startAsyncGauges()` does not "have no effect" — it - * permanently disarms every gauge the later call registers, so - * the instruments exist but never observe a value. Only call it - * once the process is shutting down. - */ - void - detachCallbacks() noexcept; - /** * Flush pending metrics and shut down the pipeline. * @@ -466,10 +343,11 @@ public: * touched: record threads may still be running, and the gate is what * keeps them off the dying pipeline. Idempotent. * - * @pre `detachCallbacks()` should have been called earlier in the - * shutdown sequence; otherwise there is a narrow race between - * the final reader-thread tick and the destruction of - * Application services that the gauge callbacks read from. + * @pre Anything that observes live server state on the reader thread has + * already been disarmed. Shutting the provider down joins that + * thread, so a caller that has not disarmed its observers leaves a + * narrow race between the final tick and the teardown of what those + * observers read. */ void stop(); @@ -505,6 +383,22 @@ public: #endif } + /** + * @return true when a real exporting pipeline exists, as opposed to the + * no-op meter installed when the pipeline is disabled. + * + * A meter() check cannot answer this. The registry always hands out a + * meter, so registering instruments on a no-op one would report success + * and export nothing. Ask this before registering an observable + * instrument. + * + * @note Not thread-safe against stop(), which drops the provider this + * reads. Call it from the server lifecycle thread, like the constructor + * and stop(). + */ + [[nodiscard]] bool + hasPipeline() const noexcept; + // ----------------------------------------------------------------- // Synchronous instrument recording (called from PerfLog hot paths) // ----------------------------------------------------------------- @@ -571,11 +465,8 @@ public: * which is a stronger guarantee than an allowlist that would have to * be maintained by hand. * - * Defined inline so unit tests can call it without linking the rest - * of the registry: in a telemetry-enabled build MetricsRegistry.cpp - * is not compiled into the test binary, so an out-of-line definition - * would be unreachable from tests. Being inline also makes it usable - * regardless of XRPL_ENABLE_TELEMETRY. + * Defined inline so it is available in a build without telemetry and + * usable in a constant expression. * * @param name The job name as passed to JobQueue::addJob. * @return @p name when it is non-empty and all ASCII letters, else @@ -633,10 +524,8 @@ public: * INT64_MAX rather than wrapping, because a wrapped gauge reads as a * sudden healthy-looking dip. * - * Defined inline for the same reason as sanitiseHandler(): in a - * telemetry-enabled build MetricsRegistry.cpp is not compiled into the - * unit-test binary, so an out-of-line definition would be untestable. - * constexpr so the cases below are checked at compile time. + * Defined inline for the same reason as sanitiseHandler(); constexpr so + * the cases below are checked at compile time. * * @param total Cumulative numerator (e.g. summed microseconds). * @param count Number of samples in @p total. @@ -692,9 +581,7 @@ public: * interval whose first equals its last as a bare sequence number. A segment * with no dash is therefore a range of one ledger, not a malformed one. * - * Defined inline for the same reason as sanitiseHandler(): in a - * telemetry-enabled build MetricsRegistry.cpp is not compiled into the - * unit-test binary, so an out-of-line definition would be untestable. + * Defined inline for the same reason as sanitiseHandler(). * * @param segment One segment, already split on ','. Leading or trailing * whitespace is rejected, because the producer emits none. @@ -889,59 +776,6 @@ public: { return meter_; } - - /** - * Sink handed to the nodestore_state gauge helpers below. - * - * Every value they publish multiplexes onto the single `nodestore_state` - * gauge through its `metric` label, so the helpers need no access to the - * OTel observer result -- just somewhere to put a name and a number. - */ - using ObserveFn = std::function; - - /** - * Observe the NodeStore I/O totals and the means derived from them. - * - * @param db NodeStore to read the counters from. - * @param observe Sink for one `metric`-labelled value. - */ - static void - observeNodeStoreTotals(node_store::Database& db, ObserveFn const& observe); - - /** - * Observe the backend write-path detail, when the backend measures it. - * - * Publishes nothing for a backend whose getWriteStats() is std::nullopt, - * which is every backend except NuDB. Absent labels let a reader tell - * "not measured" from "measured, and idle"; zeros would read as a - * perfectly idle write path. - * - * @param db NodeStore whose writable backend is sampled. - * @param observe Sink for one `metric`-labelled value. - */ - static void - observeWritePathDetail(node_store::Database const& db, ObserveFn const& observe); - - /** - * Observe the ledger-acquisition progress and stall counters. - * - * @param stats Process-wide acquisition counters. - * @param observe Sink for one `metric`-labelled value. - */ - static void - observeAcquireStats(AcquireStats const& stats, ObserveFn const& observe); - - /** - * Observe the read queue depth and the read thread-pool counts. - * - * These four have no accessor on Database, so its JSON counters object - * is still the only way to reach them. - * - * @param db NodeStore to read the JSON counters from. - * @param observe Sink for one `metric`-labelled value. - */ - static void - observeReadQueue(node_store::Database& db, ObserveFn const& observe); #endif private: @@ -961,13 +795,6 @@ private: */ ValidationTracker validationTracker_; - /** - * Reference to Application services for gauge callbacks. - * Only needed when OTel is compiled in, since observable gauge - * callbacks live entirely inside the XRPL_ENABLE_TELEMETRY guard. - */ - ServiceRegistry& app_; - /** * Journal for logging. */ @@ -975,30 +802,20 @@ private: /** * Where the registry is in its life. Construction ends in `Ready`; - * startAsyncGauges() moves to `GaugesArmed`; stop() to `Stopped`. A call - * that does not fit the current phase logs a warning and does nothing. + * stop() moves to `Stopped`. * * After `Stopped` the SDK pipeline is gone. recording() reads false, so * no macro touches meter_ or a cached instrument. */ - enum class Phase { Ready, GaugesArmed, Stopped }; + enum class Phase { Ready, Stopped }; /** - * Current phase. Written from the Application lifecycle thread with - * release ordering; read from record threads via `recording()` with - * acquire ordering, so no record starts once stop() has stored `Stopped`. + * Current phase. Written from the server lifecycle thread with release + * ordering; read from record threads via `recording()` with acquire + * ordering, so no record starts once stop() has stored `Stopped`. */ std::atomic phase_{Phase::Ready}; - /** - * Set by detachCallbacks() during shutdown so every ObservableGauge - * callback returns early before reading Application services that - * may already be stopped. Checked with memory_order_acquire at the - * top of each callback to pair with the memory_order_release store - * in detachCallbacks(). - */ - std::atomic callbacksDetached_{false}; - /** * The SDK MeterProvider that owns the export pipeline. */ @@ -1054,94 +871,6 @@ private: opentelemetry::nostd::unique_ptr> jobRunningDurationHistogram_; - // --- Observable gauges (registered via callbacks) --- - // Handles are stored so we can remove callbacks on shutdown. - /** - * Observable gauges for cache hit rates and sizes. - */ - opentelemetry::nostd::shared_ptr - cacheHitRateGauge_; - /** - * Observable gauges for TxQ metrics. - */ - opentelemetry::nostd::shared_ptr txqGauge_; - /** - * Observable gauges for counted object instances. - */ - opentelemetry::nostd::shared_ptr - objectCountGauge_; - /** - * Observable gauges for load factor breakdown. - */ - opentelemetry::nostd::shared_ptr loadFactorGauge_; - /** - * Observable gauge multiplexing every NodeStore value onto one - * instrument via its `metric` label: I/O totals, the read and write - * means derived from them, the NuDB write-queue detail, and the - * ledger-acquisition stall counters. - */ - opentelemetry::nostd::shared_ptr nodeStoreGauge_; - /** - * Observable gauge for server-level health metrics (state, uptime, peers, etc.). - */ - opentelemetry::nostd::shared_ptr serverInfoGauge_; - /** - * Observable gauge for build version info (label-based, value=1). - */ - opentelemetry::nostd::shared_ptr buildInfoGauge_; - /** - * Observable gauge for complete ledger range start/end pairs. - */ - opentelemetry::nostd::shared_ptr - completeLedgersGauge_; - /** - * Observable gauge for database sizes and historical fetch rate. - */ - opentelemetry::nostd::shared_ptr dbMetricsGauge_; - - // --- External dashboard parity gauges --- - /** - * Observable gauge for validator health indicators (amendment blocked, - * UNL blocked, quorum, UNL expiry). - */ - opentelemetry::nostd::shared_ptr - validatorHealthGauge_; - /** - * Observable gauge for peer network quality metrics (P90 latency, - * insane peer count, version spread, upgrade recommendation). - */ - opentelemetry::nostd::shared_ptr - peerQualityGauge_; - /** - * Observable gauge for transaction reduce-relay efficiency (selected vs - * suppressed peers, feature-disabled peers, missing-tx frequency). - */ - opentelemetry::nostd::shared_ptr - reduceRelayGauge_; - /** - * Observable gauge for ledger economy metrics (base fee, reserve, - * reserve increment, ledger age). - */ - opentelemetry::nostd::shared_ptr - ledgerEconomyGauge_; - /** - * Observable gauge for node state tracking (operating mode value, - * time in current state). - */ - opentelemetry::nostd::shared_ptr - stateTrackingGauge_; - /** - * Observable gauge for storage detail metrics (NuDB on-disk size). - */ - opentelemetry::nostd::shared_ptr - storageDetailGauge_; - /** - * Observable gauge for validation agreement metrics (1h/24h percentages - * and counts from ValidationTracker). - */ - opentelemetry::nostd::shared_ptr - validationAgreementGauge_; - // --- External dashboard parity counters --- /** * Counter: ledgers_closed_total — incremented each consensus round. @@ -1164,12 +893,6 @@ private: */ opentelemetry::nostd::unique_ptr> stateChangesCounter_; - /** - * ObservableCounter: jq_trans_overflow_total — observed from - * Overlay::getJqTransOverflow() (cumulative overflow tally owned by the overlay). - */ - opentelemetry::nostd::shared_ptr - jqTransOverflowObservable_; /** * Counter: ledger_history_mismatch_total{reason} — incremented per classified * built-vs-validated ledger mismatch. @@ -1186,20 +909,6 @@ private: * admission to the queue. */ opentelemetry::nostd::unique_ptr> txqDroppedCounter_; - /** - * ObservableCounter: validation_agreements_total — observed from - * ValidationTracker::totalAgreementsEver() (monotonic gross lifetime - * tally, initial-classification semantics). - */ - opentelemetry::nostd::shared_ptr - validationAgreementsObservable_; - /** - * ObservableCounter: validation_missed_total — observed from - * ValidationTracker::totalMissedEver() (monotonic gross lifetime tally, - * initial-classification semantics). - */ - opentelemetry::nostd::shared_ptr - validationMissedObservable_; /** * Build the OTLP/HTTP exporter, periodic reader, resource attributes and @@ -1229,68 +938,7 @@ private: */ void disablePipeline(std::string_view reason); - - /** - * Register all observable gauge callbacks with the OTel SDK. - * Dispatches to one helper per metric domain so that each helper - * stays well under the 80-line-per-function limit. - * - * Called only from `startAsyncGauges()`, which owns the enabled_, - * phase_ and provider_ guards and the Application-state precondition. - */ - void - registerAsyncGauges(); - - // Per-domain registration helpers for the async (pull-model) phase. - // Each creates its instrument -- an ObservableGauge, or an - // ObservableCounter where the underlying value is cumulative -- and - // attaches a single callback that reads current values from Application - // services. The callbacks run on the OTel - // PeriodicExportingMetricReader background thread (~10 s tick). - void - registerJqTransOverflowCounter(); // gap-fill: overlay overflow total - void - registerCacheHitRateGauge(); - void - registerTxqGauge(); - void - registerObjectCountGauge(); - void - registerLoadFactorGauge(); - void - registerNodeStoreGauge(); - - // The four nodestore_state helpers and their ObserveFn sink are public - // (above), so a test can drive each one with a recording sink and assert - // the exact `metric` label values it publishes. They read only their - // arguments, so exposing them widens no state. - - void - registerServerInfoGauge(); - void - registerBuildInfoGauge(); - void - registerCompleteLedgersGauge(); - void - registerDbMetricsGauge(); - void - registerValidatorHealthGauge(); - void - registerPeerQualityGauge(); - void - registerReduceRelayGauge(); // Reduce-relay efficiency - void - registerLedgerEconomyGauge(); - void - registerStateTrackingGauge(); - void - registerStorageDetailGauge(); - void - registerValidationAgreementGauge(); - void - registerValidationTotalsCounters(); // gap-fill: lifetime agree/miss _total -#endif // XRPL_ENABLE_TELEMETRY +#endif // XRPL_ENABLE_TELEMETRY }; -} // namespace telemetry -} // namespace xrpl +} // namespace xrpl::telemetry diff --git a/src/xrpld/telemetry/ValidationTracker.h b/include/xrpl/telemetry/ValidationTracker.h similarity index 100% rename from src/xrpld/telemetry/ValidationTracker.h rename to include/xrpl/telemetry/ValidationTracker.h diff --git a/src/libxrpl/telemetry/MetricsRegistry.cpp b/src/libxrpl/telemetry/MetricsRegistry.cpp new file mode 100644 index 0000000000..177cedf8b0 --- /dev/null +++ b/src/libxrpl/telemetry/MetricsRegistry.cpp @@ -0,0 +1,603 @@ +/** + * MetricsRegistry implementation — the OpenTelemetry metrics pipeline. + * + * This file contains: + * - Construction / destruction logic for the OTel MeterProvider pipeline. + * - Synchronous instrument creation (counters, histograms) for RPC, job + * queue and the external dashboard parity counters. + * - The record / increment methods app code pushes values through. + * - No-op stubs when XRPL_ENABLE_TELEMETRY is not defined. + */ + +// On Windows, OTel's spin_lock_mutex.h (transitively included from +// MetricsRegistry.h) defines _WINSOCKAPI_ and includes . +// This poisons the include state for boost/asio/detail/socket_types.hpp, +// which requires winsock2.h to be included first. Pre-including the +// boost/asio socket types header gets winsock2.h in before the OTel +// headers can interfere. +#ifdef _MSC_VER +#include +#endif + +#include + +// Unguarded because the constructor's `beast::Journal journal` parameter is +// declared in both configurations; only the member it initialises is guarded. +#include + +#ifdef XRPL_ENABLE_TELEMETRY + +#include +#include +#include +#include +#include +// For networkTypeFromId(), the one xrpl.network.type mapping both export +// paths use, plus noopMeter() and the shared meter name and version. +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace metric_sdk = opentelemetry::sdk::metrics; +namespace otlp_http = opentelemetry::exporter::otlp; +// Not `resource`: that would collide with xrpl::resource (the resource-accounting +// namespace), which encloses every use site below. Inner-scope lookup would find +// that namespace instead of this file-scope alias. +namespace otel_resource = opentelemetry::sdk::resource; + +namespace { + +// Microsecond-valued duration histogram instrument names. Each is +// referenced twice — once to register the explicit-bucket view and once +// to create the instrument — so they are named constants to keep the two +// sites in sync (a mismatch would silently drop the bucket override). +constexpr char kJobQueuedDurationUs[] = "job_queued_us"; +constexpr char kJobRunningDurationUs[] = "job_running_us"; +constexpr char kRpcMethodDurationUs[] = "rpc_method_us"; + +// Attribute (label) keys for the job instruments. Each is referenced from +// several record sites, and a counter and its histogram must carry exactly +// the same key spelling or the two series cannot be joined in a query. +constexpr char kJobTypeLabel[] = "job_type"; +constexpr char kHandlerLabel[] = "handler"; + +/** + * Register an explicit-bucket histogram view. + * + * The SDK's default boundaries top out at 10,000, so any instrument whose + * values exceed that saturates and every quantile reads as the ceiling. The + * floor matters just as much and is easier to miss: a ladder whose first edge + * sits above the mass of the distribution makes every low quantile an + * interpolation inside bucket 0 -- a number derived from the bucket edge + * rather than from any sample. Both ends are chosen from measured + * distributions in HistogramBuckets.h. + * + * @param views The registry to add the view to. + * @param name Instrument name to match (e.g. "job_running_us"). + * @param boundaries Bucket upper bounds, ascending. + */ +void +addHistogramView( + metric_sdk::ViewRegistry& views, + std::string const& name, + std::vector boundaries) +{ + auto config = std::make_shared(); + config->boundaries_ = std::move(boundaries); + + auto selector = metric_sdk::InstrumentSelectorFactory::Create( + metric_sdk::InstrumentType::kHistogram, name, ""); + auto meterSelector = metric_sdk::MeterSelectorFactory::Create( + std::string(xrpl::telemetry::kMeterName), std::string(xrpl::telemetry::kMeterVersion), ""); + auto view = + metric_sdk::ViewFactory::Create(name, "", metric_sdk::AggregationType::kHistogram, config); + + views.AddView(std::move(selector), std::move(meterSelector), std::move(view)); +} + +/** + * Register the microsecond-ladder view for a duration instrument. + * + * Job wait/run times and RPC latencies routinely exceed the SDK default + * ceiling, so they all share `buckets::kMicrosecondBuckets`. + * + * @param views The registry to add the view to. + * @param name Instrument name to match. + */ +void +addMicrosecondHistogramView(metric_sdk::ViewRegistry& views, std::string const& name) +{ + addHistogramView( + views, + name, + xrpl::telemetry::buckets::toVector(xrpl::telemetry::buckets::kMicrosecondBuckets)); +} + +} // namespace + +#endif // XRPL_ENABLE_TELEMETRY + +namespace xrpl::telemetry { + +MetricsRegistry::MetricsRegistry( + [[maybe_unused]] bool enabled, + [[maybe_unused]] beast::Journal journal, + [[maybe_unused]] Options const& options) + : enabled_(enabled) +#ifdef XRPL_ENABLE_TELEMETRY + , journal_(journal) +#endif +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (!enabled_) + return; + + // useTls is logged because a collector that requires TLS rejects a + // plaintext exporter with no local error. The paths are left out. + JLOG(journal_.info()) << "MetricsRegistry: starting, endpoint=" << options.endpoint + << ", serviceName=" << options.serviceName + << ", serviceVersion=" << options.serviceVersion + << ", instanceId=" << options.serviceInstanceId + << ", nodeId=" << options.nodeId << ", networkId=" << options.networkId + << ", useTls=" << options.useTls; + + // A broken pipeline must not stop the node. The SDK is third-party code, + // so the catch-all is deliberate, as in ~ApplicationImp. + try + { + initExporterAndProvider(options); + + // Rule for anything added below: the constructor may create only + // instruments whose recording is PUSHED from app code -- counters and + // histograms. An instrument registered here is live immediately, and + // the reader thread may invoke a registered callback before the rest + // of the server is built, so any observable whose callback reads live + // server state belongs in the layer that owns those callbacks, not + // here. That includes observable COUNTERS, not just gauges: + // jq_trans_overflow_total was created here and its callback read + // getOverlay(), which asserts overlay_ is non-null. + initSyncInstruments(); + } + catch (std::exception const& e) + { + disablePipeline(e.what()); + return; + } + catch (...) + { + disablePipeline("unknown exception"); + return; + } + + JLOG(journal_.info()) << "MetricsRegistry: provider and instruments ready"; +#endif // XRPL_ENABLE_TELEMETRY +} + +#ifdef XRPL_ENABLE_TELEMETRY +void +MetricsRegistry::disablePipeline(std::string_view reason) +{ + provider_.reset(); + // A no-op meter keeps the invariant the XRPL_METRIC_* macros rely on: an + // enabled registry always has a meter, so every call site gets an inert + // instrument here with no check of its own. + meter_ = noopMeter(kMeterName); + JLOG(journal_.error()) << "MetricsRegistry: metrics pipeline failed to initialise, " + "continuing without native metrics: " + << reason; +} +#endif // XRPL_ENABLE_TELEMETRY + +MetricsRegistry::~MetricsRegistry() +{ + stop(); +} + +#ifdef XRPL_ENABLE_TELEMETRY +void +MetricsRegistry::initExporterAndProvider(Options const& options) +{ + // Configure OTLP/HTTP metric exporter. The TLS settings come from the one + // [telemetry] block that also drives the trace exporter in Telemetry.cpp, + // so both exporters reach the collector on the same terms. + otlp_http::OtlpHttpMetricExporterOptions exporterOpts; + exporterOpts.url = options.endpoint; + if (options.useTls) + { + exporterOpts.ssl_ca_cert_path = options.tlsCaCertPath; + exporterOpts.ssl_client_cert_path = options.tlsClientCertPath; + exporterOpts.ssl_client_key_path = options.tlsClientKeyPath; + } + + auto exporter = otlp_http::OtlpHttpMetricExporterFactory::Create(exporterOpts); + + // Configure periodic reader with 10-second export interval. + metric_sdk::PeriodicExportingMetricReaderOptions readerOpts; + readerOpts.export_interval_millis = std::chrono::milliseconds(10000); + readerOpts.export_timeout_millis = std::chrono::milliseconds(5000); + auto reader = + metric_sdk::PeriodicExportingMetricReaderFactory::Create(std::move(exporter), readerOpts); + + // Stamp the same resource Telemetry::makeMetricsResource() builds for the + // trace pipeline. Both must agree: a node whose service.name or + // xrpl.network.type differs between the two pipelines splits its own + // series, and a dashboard filtering on either label shows only half. + // + // Use std::string, never a string literal: ResourceAttributes stores an + // OTel AttributeValue variant whose char-const* overload binds to bool, + // so a literal would be recorded as the boolean true. + otel_resource::ResourceAttributes attrs; + attrs[opentelemetry::semconv::service::kServiceName] = options.serviceName; + // int64_t, matching the trace resource. The same key with two types would + // give the two pipelines incompatible attribute values. + attrs[std::string(attr::networkId)] = static_cast(options.networkId); + // Derived here rather than passed in, so the id and the type label cannot + // disagree. Same helper the trace path uses. + attrs[std::string(attr::networkType)] = networkTypeFromId(options.networkId); + + // The three below are left off when empty rather than stamped blank. An + // absent label reads as "not reported"; an empty one looks like a value. + if (!options.serviceVersion.empty()) + attrs[opentelemetry::semconv::service::kServiceVersion] = options.serviceVersion; + if (!options.serviceInstanceId.empty()) + attrs[opentelemetry::semconv::service::kServiceInstanceId] = options.serviceInstanceId; + // xrpl.node.id: the same per-node key the trace resource carries, so + // metrics and traces resolve to one node. + if (!options.nodeId.empty()) + attrs[std::string(attr::nodeId)] = options.nodeId; + auto resourceAttrs = otel_resource::Resource::Create(attrs); + + // Build a view registry with explicit microsecond buckets for the + // duration histograms. Without this they use the SDK default buckets + // (max 10,000 = 10 ms), saturating every quantile at 10 ms. + auto views = std::make_unique(); + addMicrosecondHistogramView(*views, kJobQueuedDurationUs); + addMicrosecondHistogramView(*views, kJobRunningDurationUs); + addMicrosecondHistogramView(*views, kRpcMethodDurationUs); + // Recorded at its PeerImp.cpp call site, not created here, so the name + // comes from the shared constant both sites use. + addMicrosecondHistogramView(*views, kGetObjectLookupUs); + + // The remaining two GetObject histograms are not durations, so the + // microsecond ladder above does not fit them. Both still need explicit + // boundaries: the SDK default stops at 10,000 and both ranges exceed it. + // + // Object counts run 1..kHardMaxReplyNodes (12288). The honest sync path + // asks for at most 8, so the low buckets are fine-grained and the upper + // ones follow the charge size bands (64, 1024) up to the hard cap. + addHistogramView( + *views, kGetObjectRequestObjects, buckets::toVector(buckets::kObjectCountBuckets)); + + // Charge values span 0 (free tier) to ~99k for a full-size all-miss + // request. Boundaries bracket the resource thresholds that decide a + // peer's fate -- kWarningThreshold (5000) and kDropThreshold (25000) -- + // so a dashboard can show how close charges run to each. + addHistogramView(*views, kGetObjectCharge, buckets::toVector(buckets::kChargeBuckets)); + + // The two RPC request-count histograms are recorded at their ServerHandler + // and PathRequest call sites, so the names come from the shared constants + // all three sites use. Both are small counts, and the reason they need a + // view is the FLOOR rather than the ceiling: the SDK default edges start + // 0, 5, 10, 25, so a batch of one to five sub-requests -- the normal case -- + // would land in a single bucket and every quantile over it would be an + // interpolation inside that bucket rather than a measurement. + // + // The object-count ladder is the fit: its 1, 2, 4, 8, 16 edges sit exactly + // where both distributions have their mass. Path counts are hard-bounded at + // kMaxPaths * kMaxAutoSrcCur = 352, well under its 12288 top. Batch sizes + // have no such cap; see the ceiling note in RpcMetricNames.h. + addHistogramView(*views, kRpcBatchSize, buckets::toVector(buckets::kObjectCountBuckets)); + addHistogramView( + *views, kPathfindDiscoveredPaths, buckets::toVector(buckets::kObjectCountBuckets)); + + // Create MeterProvider with resource, then attach the metric reader. + provider_ = metric_sdk::MeterProviderFactory::Create(std::move(views), resourceAttrs); + provider_->AddMetricReader(std::move(reader)); + + // Get a meter for all xrpld instruments. + meter_ = provider_->GetMeter(std::string(kMeterName), std::string(kMeterVersion)); +} + +void +MetricsRegistry::initSyncInstruments() +{ + // RPC per-method counters and histogram. + rpcStartedCounter_ = + meter_->CreateUInt64Counter("rpc_method_started_total", "Total RPC method calls started"); + rpcFinishedCounter_ = meter_->CreateUInt64Counter( + "rpc_method_finished_total", "Total RPC method calls completed successfully"); + rpcErroredCounter_ = meter_->CreateUInt64Counter( + "rpc_method_errored_total", "Total RPC method calls that errored"); + rpcDurationHistogram_ = meter_->CreateDoubleHistogram( + kRpcMethodDurationUs, "RPC method execution time in microseconds"); + + // Job queue per-type counters and histograms. + jobQueuedCounter_ = meter_->CreateUInt64Counter("job_queued_total", "Total jobs enqueued"); + jobStartedCounter_ = meter_->CreateUInt64Counter("job_started_total", "Total jobs started"); + jobFinishedCounter_ = meter_->CreateUInt64Counter("job_finished_total", "Total jobs completed"); + jobQueuedDurationHistogram_ = meter_->CreateDoubleHistogram( + kJobQueuedDurationUs, "Time jobs spent waiting in the queue (microseconds)"); + jobRunningDurationHistogram_ = + meter_->CreateDoubleHistogram(kJobRunningDurationUs, "Job execution time in microseconds"); + + // --- External dashboard parity counters --- + ledgersClosedCounter_ = + meter_->CreateUInt64Counter("ledgers_closed_total", "Total ledgers closed by consensus"); + validationsSentCounter_ = meter_->CreateUInt64Counter( + "validations_sent_total", "Total validations sent by this node"); + validationsCheckedCounter_ = meter_->CreateUInt64Counter( + "validations_checked_total", "Total network validations received and checked"); + stateChangesCounter_ = + meter_->CreateUInt64Counter("state_changes_total", "Total operating mode changes"); + ledgerHistoryMismatchCounter_ = meter_->CreateUInt64Counter( + "ledger_history_mismatch_total", "Total built-vs-validated ledger mismatches by reason"); + txqExpiredCounter_ = meter_->CreateUInt64Counter( + "txq_expired_total", "Total transactions expired out of the transaction queue"); + txqDroppedCounter_ = meter_->CreateUInt64Counter( + "txq_dropped_total", "Total transactions refused admission to the queue by reason"); + // Note: validation_agreements_total / validation_missed_total are monotonic + // ObservableCounters owned by the observable-gauge layer. +} +#endif // XRPL_ENABLE_TELEMETRY + +void +MetricsRegistry::stop() +{ +#ifdef XRPL_ENABLE_TELEMETRY + // Store Stopped with release ordering BEFORE the pipeline goes away. + // Every recording thread reads phase_ through recording() with acquire + // ordering, so any record that has not yet passed the gate will see + // Stopped and skip. Idempotent: destructor calls this after run() or + // ~ApplicationImp already did. + phase_.store(Phase::Stopped, std::memory_order_release); + if (!provider_) + return; + + JLOG(journal_.info()) << "MetricsRegistry: stopping"; + + // meter_ is left alone on purpose. Job threads are still running here and + // may be inside a macro, so writing meter_ would race with their read. + // The recording() gate is what keeps them off the dying pipeline: only the + // macros read meter_, and none of them does so once phase_ is Stopped. + // + // SDK teardown order: Shutdown() stops the PeriodicExportingMetricReader + // thread (so no further gauge callbacks fire) and performs the final + // collect-and-export drain itself. The trailing ForceFlush() is a + // redundant safety net (a no-op once the reader is shut down), then + // reset() destroys the provider. + // + // provider_.reset() destroys MeterProvider -> MeterContext -> ViewRegistry + // -> each View -> its shared_ptr. Live SDK + // SyncMetricStorage instances cached in call-site statics still hold a + // raw AggregationConfig pointer; a Record with a NEW attribute set after + // this point would fire the factory lambda and deref that dangling + // pointer, and a late meter()->CreateXxx would return null. + provider_->Shutdown(); + provider_->ForceFlush(); + provider_.reset(); + + JLOG(journal_.info()) << "MetricsRegistry: stopped"; +#endif // XRPL_ENABLE_TELEMETRY +} + +bool +MetricsRegistry::hasPipeline() const noexcept +{ +#ifdef XRPL_ENABLE_TELEMETRY + return provider_ != nullptr; +#else + return false; +#endif +} + +// ----------------------------------------------------------------- +// Synchronous instrument recording — RPC metrics +// ----------------------------------------------------------------- + +void +MetricsRegistry::recordRpcStarted([[maybe_unused]] std::string_view method) +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (!recording() || !rpcStartedCounter_) + return; + rpcStartedCounter_->Add(1, {{"method", std::string(method)}}); +#endif +} + +void +MetricsRegistry::recordRpcFinished( + [[maybe_unused]] std::string_view method, + [[maybe_unused]] std::int64_t durationUs) +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (!recording() || !rpcFinishedCounter_) + return; + rpcFinishedCounter_->Add(1, {{"method", std::string(method)}}); + if (rpcDurationHistogram_) + { + rpcDurationHistogram_->Record( + static_cast(durationUs), + {{"method", std::string(method)}}, + opentelemetry::context::Context{}); + } +#endif +} + +void +MetricsRegistry::recordRpcErrored( + [[maybe_unused]] std::string_view method, + [[maybe_unused]] std::int64_t durationUs) +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (!recording() || !rpcErroredCounter_) + return; + rpcErroredCounter_->Add(1, {{"method", std::string(method)}}); + if (rpcDurationHistogram_) + { + rpcDurationHistogram_->Record( + static_cast(durationUs), + {{"method", std::string(method)}}, + opentelemetry::context::Context{}); + } +#endif +} + +// ----------------------------------------------------------------- +// Synchronous instrument recording — Job Queue metrics +// ----------------------------------------------------------------- + +void +MetricsRegistry::recordJobQueued( + [[maybe_unused]] std::string_view jobType, + [[maybe_unused]] std::string_view jobName) +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (!recording() || !jobQueuedCounter_) + return; + jobQueuedCounter_->Add( + 1, + {{kJobTypeLabel, std::string(jobType)}, + {kHandlerLabel, std::string(sanitiseHandler(jobName))}}); +#endif +} + +void +MetricsRegistry::recordJobStarted( + [[maybe_unused]] std::string_view jobType, + [[maybe_unused]] std::string_view jobName, + [[maybe_unused]] std::int64_t queuedDurUs) +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (!recording() || !jobStartedCounter_) + return; + // Build the attribute pair once: both the counter and the histogram + // must carry the identical label set or they cannot be joined. + std::string const handler(sanitiseHandler(jobName)); + jobStartedCounter_->Add(1, {{kJobTypeLabel, std::string(jobType)}, {kHandlerLabel, handler}}); + if (jobQueuedDurationHistogram_ && queuedDurUs >= 0) + { + // Guard against negative queued durations: the caller derives this + // from a steady-clock delta that can go slightly negative under clock + // skew or reordering. The OTel SDK rejects negative histogram values + // (logging a warning per call), so skip them rather than spam. + jobQueuedDurationHistogram_->Record( + static_cast(queuedDurUs), + {{kJobTypeLabel, std::string(jobType)}, {kHandlerLabel, handler}}, + opentelemetry::context::Context{}); + } +#endif +} + +void +MetricsRegistry::recordJobFinished( + [[maybe_unused]] std::string_view jobType, + [[maybe_unused]] std::string_view jobName, + [[maybe_unused]] std::int64_t runningDurUs) +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (!recording() || !jobFinishedCounter_) + return; + std::string const handler(sanitiseHandler(jobName)); + jobFinishedCounter_->Add(1, {{kJobTypeLabel, std::string(jobType)}, {kHandlerLabel, handler}}); + if (jobRunningDurationHistogram_) + { + jobRunningDurationHistogram_->Record( + static_cast(runningDurUs), + {{kJobTypeLabel, std::string(jobType)}, {kHandlerLabel, handler}}, + opentelemetry::context::Context{}); + } +#endif +} + +// ----------------------------------------------------------------- +// External dashboard parity counter increments +// ----------------------------------------------------------------- + +void +MetricsRegistry::incrementLedgersClosed() +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (recording() && ledgersClosedCounter_) + ledgersClosedCounter_->Add(1); +#endif +} + +void +MetricsRegistry::incrementValidationsSent() +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (recording() && validationsSentCounter_) + validationsSentCounter_->Add(1); +#endif +} + +void +MetricsRegistry::incrementValidationsChecked() +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (recording() && validationsCheckedCounter_) + validationsCheckedCounter_->Add(1); +#endif +} + +void +MetricsRegistry::incrementStateChanges() +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (recording() && stateChangesCounter_) + stateChangesCounter_->Add(1); +#endif +} + +void +MetricsRegistry::incrementLedgerHistoryMismatch(std::string_view reason) +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (recording() && ledgerHistoryMismatchCounter_) + ledgerHistoryMismatchCounter_->Add(1, {{"reason", std::string(reason)}}); +#endif +} + +void +MetricsRegistry::incrementTxqExpired() +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (recording() && txqExpiredCounter_) + txqExpiredCounter_->Add(1); +#endif +} + +void +MetricsRegistry::incrementTxqDropped(std::string_view reason) +{ +#ifdef XRPL_ENABLE_TELEMETRY + if (recording() && txqDroppedCounter_) + txqDroppedCounter_->Add(1, {{"reason", std::string(reason)}}); +#endif +} + +} // namespace xrpl::telemetry diff --git a/src/xrpld/telemetry/detail/ValidationTracker.cpp b/src/libxrpl/telemetry/detail/ValidationTracker.cpp similarity index 99% rename from src/xrpld/telemetry/detail/ValidationTracker.cpp rename to src/libxrpl/telemetry/detail/ValidationTracker.cpp index d37c97bee8..9f334a4fcb 100644 --- a/src/xrpld/telemetry/detail/ValidationTracker.cpp +++ b/src/libxrpl/telemetry/detail/ValidationTracker.cpp @@ -3,7 +3,7 @@ * Implementation of the ValidationTracker class. */ -#include +#include #include #include diff --git a/src/test/nodestore/DatabaseConfig_test.cpp b/src/test/nodestore/DatabaseConfig_test.cpp index 04669bc6be..ca3f354650 100644 --- a/src/test/nodestore/DatabaseConfig_test.cpp +++ b/src/test/nodestore/DatabaseConfig_test.cpp @@ -9,7 +9,7 @@ // of them reads. Both live in xrpld and are only declared in a // telemetry-enabled build, so the include is guarded like its uses below. #include -#include +#include #endif #include @@ -988,7 +988,7 @@ public: /** * Return a sink callable that appends into @ref emitted. */ - telemetry::MetricsRegistry::ObserveFn + telemetry::AppMetricGauges::ObserveFn fn() { return @@ -1082,7 +1082,7 @@ public: // Fresh store: the eight unconditional labels are published, each at // exactly zero, and NEITHER mean appears. MetricSink fresh; - telemetry::MetricsRegistry::observeNodeStoreTotals(*db, fresh.fn()); + telemetry::AppMetricGauges::observeNodeStoreTotals(*db, fresh.fn()); std::vector const kFreshLabels{ "node_read_bytes", @@ -1139,7 +1139,7 @@ public: BEAST_EXPECT(db.fetchNodeObject(object->getHash(), 0) == nullptr); MetricSink busy; - telemetry::MetricsRegistry::observeNodeStoreTotals(db, busy.fn()); + telemetry::AppMetricGauges::observeNodeStoreTotals(db, busy.fn()); // Ten labels now: the eight above plus both means. BEAST_EXPECT(busy.emitted.size() == 10); @@ -1201,7 +1201,7 @@ public: storeBatch(*mem, batch); MetricSink sink; - telemetry::MetricsRegistry::observeWritePathDetail(*mem, sink.fn()); + telemetry::AppMetricGauges::observeWritePathDetail(*mem, sink.fn()); // Cause as well as state: the store really was written to, so the // emptiness is the std::nullopt branch and not an idle database. BEAST_EXPECT(sink.emitted.empty()); @@ -1223,7 +1223,7 @@ public: // pins the deliberate asymmetry -- zero is meaningful for a gauge and // meaningless for a mean. MetricSink fresh; - telemetry::MetricsRegistry::observeWritePathDetail(*db, fresh.fn()); + telemetry::AppMetricGauges::observeWritePathDetail(*db, fresh.fn()); std::vector const kFreshLabels{"nudb_insert_max_us", "nudb_writers_in_flight"}; BEAST_EXPECT(fresh.names() == kFreshLabels); BEAST_EXPECT(fresh.value("nudb_writers_in_flight") == std::int64_t{0}); @@ -1236,7 +1236,7 @@ public: storeBatch(*db, stored); MetricSink busy; - telemetry::MetricsRegistry::observeWritePathDetail(*db, busy.fn()); + telemetry::AppMetricGauges::observeWritePathDetail(*db, busy.fn()); std::vector const kBusyLabels{ "nudb_insert_max_us", "nudb_insert_mean_us", @@ -1285,7 +1285,7 @@ public: // these would lose the ability to see that nothing happened. AcquireStats const quiet; MetricSink fresh; - telemetry::MetricsRegistry::observeAcquireStats(quiet, fresh.fn()); + telemetry::AppMetricGauges::observeAcquireStats(quiet, fresh.fn()); BEAST_EXPECT(fresh.names() == kLabels); BEAST_EXPECT(fresh.emitted.size() == kLabels.size()); for (auto const& label : kLabels) @@ -1313,7 +1313,7 @@ public: busy.recordSweepEviction(); MetricSink sink; - telemetry::MetricsRegistry::observeAcquireStats(busy, sink.fn()); + telemetry::AppMetricGauges::observeAcquireStats(busy, sink.fn()); BEAST_EXPECT(sink.names() == kLabels); BEAST_EXPECT(sink.value("acquire_deferrals") == std::int64_t{3}); BEAST_EXPECT(sink.value("acquire_timeouts") == std::int64_t{7}); @@ -1350,7 +1350,7 @@ public: return; MetricSink sink; - telemetry::MetricsRegistry::observeReadQueue(*db, sink.fn()); + telemetry::AppMetricGauges::observeReadQueue(*db, sink.fn()); std::vector const kLabels{ "read_queue", "read_request_bundle", "read_threads_running", "read_threads_total"}; diff --git a/src/tests/libxrpl/CMakeLists.txt b/src/tests/libxrpl/CMakeLists.txt index 9d9bc64691..2114975ea6 100644 --- a/src/tests/libxrpl/CMakeLists.txt +++ b/src/tests/libxrpl/CMakeLists.txt @@ -21,10 +21,11 @@ set_target_properties( ) # Lets test sources include the shared helpers as . target_include_directories(xrpl_tests PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}) -# Some headers under test live in src/xrpld/ rather than in libxrpl (for -# example app/ledger/AcquireStats.h and telemetry/ValidationTracker.h), so put -# src/ on the include path to reach them as . This is unconditional -# because header-only ones are testable in every build, telemetry or not. +# Two tests reach a header under src/xrpld/ rather than in libxrpl: +# ledger/AcquireStats.cpp includes and +# telemetry/RpcMetricNames.cpp includes . Put src/ on +# the include path so both resolve as . This is unconditional because +# those headers are header-only and testable in every build, telemetry or not. target_include_directories(xrpl_tests PRIVATE ${CMAKE_SOURCE_DIR}/src) target_link_libraries(xrpl_tests PRIVATE GTest::gtest GTest::gmock xrpl.libxrpl) @@ -116,29 +117,6 @@ if(telemetry) "${OTEL_IN_MEMORY_EXPORTER_LIB}" opentelemetry-cpp::opentelemetry-cpp ) -else() - # MetricsRegistry lives in xrpld; compile its .cpp directly into the test - # target so the no-op path can be tested without linking all of xrpld. - # When telemetry=ON, XRPL_ENABLE_TELEMETRY is globally defined and the - # .cpp pulls in xrpld symbols we cannot satisfy here. - target_sources( - xrpl_tests - PRIVATE ${CMAKE_SOURCE_DIR}/src/xrpld/telemetry/MetricsRegistry.cpp - ) endif() -# ValidationTracker lives in src/xrpld/ (not libxrpl), so we compile its -# implementation directly into the test binary and put src/ on the include path -# so its tests can reach headers. -# -# Both are unconditional: the class carries no telemetry guards, so its tests -# compile and run in every build. Gating them would leave the test file (which -# is likewise unguarded) without the header it includes and without the -# definitions it calls. -target_include_directories(xrpl_tests PRIVATE ${CMAKE_SOURCE_DIR}/src) -target_sources( - xrpl_tests - PRIVATE ${CMAKE_SOURCE_DIR}/src/xrpld/telemetry/detail/ValidationTracker.cpp -) - gtest_discover_tests(xrpl_tests DISCOVERY_TIMEOUT 60) diff --git a/src/tests/libxrpl/beast/insight/OTelCollectorHooks.cpp b/src/tests/libxrpl/beast/insight/OTelCollectorHooks.cpp index 2964b3ebe9..5ab29698e1 100644 --- a/src/tests/libxrpl/beast/insight/OTelCollectorHooks.cpp +++ b/src/tests/libxrpl/beast/insight/OTelCollectorHooks.cpp @@ -6,17 +6,13 @@ #include #include +#include #include #include -#include #include -#include -#include #include #include -#include -#include #include #include #include @@ -27,54 +23,10 @@ namespace beast::insight { namespace metrics_api = opentelemetry::metrics; namespace metrics_sdk = opentelemetry::sdk::metrics; -/** - * A MetricReader that collects only when the test asks it to. - * - * The SDK ships only PeriodicExportingMetricReader, whose background thread - * would make these tests depend on timing. MetricReader::Collect() is public - * and synchronous, so a minimal subclass lets a test drive one collection pass - * on the calling thread. That pass is what invokes an observable gauge's - * callback, which is the only path that reaches the collector's hooks. - * - * @code - * auto reader = std::make_shared(); - * provider->AddMetricReader(reader); - * reader->collectOnce(); // runs every registered observable callback - * @endcode - */ -class ManualMetricReader : public metrics_sdk::MetricReader -{ -public: - /** - * @brief Run exactly one collection pass, discarding the metric data. - * - * The tests assert on hook side effects, not on exported points, so the - * callback returns true without inspecting what it was handed. - */ - void - collectOnce() - { - Collect([](metrics_sdk::ResourceMetrics&) { return true; }); - } - - [[nodiscard]] metrics_sdk::AggregationTemporality - GetAggregationTemporality(metrics_sdk::InstrumentType) const noexcept override - { - return metrics_sdk::AggregationTemporality::kCumulative; - } - - bool - OnForceFlush(std::chrono::microseconds) noexcept override - { - return true; - } - - bool - OnShutDown(std::chrono::microseconds) noexcept override - { - return true; - } -}; +// The reader is not specific to this suite -- any test that needs a real SDK +// provider without a background export thread wants it -- so its one +// definition lives in the test helpers. +using xrpl::test::ManualMetricReader; /** * Installs a real SDK MeterProvider so observable gauges actually fire. diff --git a/src/tests/libxrpl/helpers/ManualMetricReader.h b/src/tests/libxrpl/helpers/ManualMetricReader.h new file mode 100644 index 0000000000..9f528e4db0 --- /dev/null +++ b/src/tests/libxrpl/helpers/ManualMetricReader.h @@ -0,0 +1,80 @@ +#pragma once + +/** + * @file ManualMetricReader.h + * A metric reader that collects on demand, for tests that need a real SDK + * MeterProvider without a background export thread. + * + * Guarded as a whole: every type it names comes from the OpenTelemetry metrics + * SDK, which is on the link line only when XRPL_ENABLE_TELEMETRY is defined. + */ + +#ifdef XRPL_ENABLE_TELEMETRY + +#include +#include +#include +#include + +#include + +namespace xrpl::test { + +/** + * A MetricReader that collects only when the test asks it to. + * + * The SDK ships only PeriodicExportingMetricReader, whose background thread + * would make a test depend on timing. MetricReader::Collect() is public and + * synchronous, so a minimal subclass lets a test drive one collection pass on + * the calling thread. That pass is what invokes an observable instrument's + * callback. + * + * The SDK types are spelled in full rather than through a namespace alias. An + * alias here would be a member of xrpl::test, and a translation unit that + * pulled that namespace in wholesale could then find two spellings of the same + * short name. + * + * @code + * auto reader = std::make_shared(); + * provider->AddMetricReader(reader); + * reader->collectOnce(); // runs every registered observable callback + * @endcode + */ +class ManualMetricReader : public opentelemetry::sdk::metrics::MetricReader +{ +public: + /** + * @brief Run exactly one collection pass, discarding the metric data. + * + * A caller asserting on callback side effects does not need the exported + * points, so the callback returns true without inspecting what it was + * handed. + */ + void + collectOnce() + { + Collect([](opentelemetry::sdk::metrics::ResourceMetrics&) { return true; }); + } + + [[nodiscard]] opentelemetry::sdk::metrics::AggregationTemporality + GetAggregationTemporality(opentelemetry::sdk::metrics::InstrumentType) const noexcept override + { + return opentelemetry::sdk::metrics::AggregationTemporality::kCumulative; + } + + bool + OnForceFlush(std::chrono::microseconds) noexcept override + { + return true; + } + + bool + OnShutDown(std::chrono::microseconds) noexcept override + { + return true; + } +}; + +} // namespace xrpl::test + +#endif // XRPL_ENABLE_TELEMETRY diff --git a/src/tests/libxrpl/nodestore/Backend.cpp b/src/tests/libxrpl/nodestore/Backend.cpp index 0b37815843..6935409aa4 100644 --- a/src/tests/libxrpl/nodestore/Backend.cpp +++ b/src/tests/libxrpl/nodestore/Backend.cpp @@ -217,7 +217,7 @@ INSTANTIATE_TEST_SUITE_P( // parameterized suite above only reaches when XRPL_ROCKSDB_AVAILABLE. // // Why absence and not zeros: the exporter skips the whole nudb_* label group -// when getWriteStats() is empty (MetricsRegistry.cpp observeWritePathDetail +// when getWriteStats() is empty (AppMetricGauges.cpp observeWritePathDetail // returns early). If the base class returned a default-constructed WriteStats // instead, every non-NuDB node would publish nudb_writers_in_flight=0 and // nudb_insert_max_us=0 -- a perfectly idle write path, on a node whose write diff --git a/src/tests/libxrpl/telemetry/MetricMacros.cpp b/src/tests/libxrpl/telemetry/MetricMacros.cpp index 2cbb202cdb..e93917ac5c 100644 --- a/src/tests/libxrpl/telemetry/MetricMacros.cpp +++ b/src/tests/libxrpl/telemetry/MetricMacros.cpp @@ -23,7 +23,7 @@ #ifdef XRPL_ENABLE_TELEMETRY -#include +#include #include #include diff --git a/src/tests/libxrpl/telemetry/MetricsRegistry.cpp b/src/tests/libxrpl/telemetry/MetricsRegistry.cpp index ba5d62e268..2f3727457f 100644 --- a/src/tests/libxrpl/telemetry/MetricsRegistry.cpp +++ b/src/tests/libxrpl/telemetry/MetricsRegistry.cpp @@ -1,35 +1,40 @@ /** * GTest unit tests for MetricsRegistry. * - * Four independent groups, split by what they can link: + * Four groups. The first three drive the pure static helpers, which are + * constexpr inline in the header and so need nothing on the link line. The + * fourth drives a real registry object. * - * 1. sanitiseHandler() — the `handler` label sanitiser. Runs in **both** - * builds. sanitiseHandler() is a public static constexpr defined inline - * in the header, so it needs no part of MetricsRegistry.cpp on the link - * line. These tests therefore sit outside the guard below; putting them - * inside it would silently compile them out of the telemetry-enabled - * build, which is the build that actually exports the label. + * 1. sanitiseHandler() — the `handler` label sanitiser. * * 2. scaledMean() — the guarded-division helper behind every derived mean - * on the nodestore_state gauge. Also a public static constexpr inline, - * so it runs in both builds for the same reason. + * on the nodestore_state gauge. * * 3. parseLedgerRange() — reads one segment of the complete-ledger range - * string the complete_ledgers gauge publishes. A public static inline, so - * it runs in both builds for the same reason. The last case drives the + * string the complete_ledgers gauge publishes. The last case drives the * real producer, xrpl::to_string(RangeSet), rather than restating its * format. * - * 4. The no-op / telemetry-disabled path — construction (which is where the - * pipeline and the synchronous instruments are built), startAsyncGauges(), - * stop(), and the synchronous record*() methods. Guarded, because when - * XRPL_ENABLE_TELEMETRY is - * defined MetricsRegistry.cpp is not compiled into this binary (see - * src/tests/libxrpl/CMakeLists.txt) and its out-of-line symbols are - * unresolvable here. + * 4. The registry lifecycle — construction, stop(), and the record and + * increment methods. Every test here runs in **both** builds: the core + * is compiled into xrpl.libxrpl, which this binary links either way, so + * with telemetry on these tests drive a real OTel pipeline and with it + * off they drive the no-op stubs. An assertion that holds in only one + * build carries its own #ifdef and says which build it pins. + * + * What group 4 pins about stop(), and what it does not: + * + * stop() stores Phase::Stopped before it destroys the SDK provider, and every + * record method reads that phase through recording() first. Without the store, + * a record carrying a first-seen attribute set would reach an + * AggregationConfig that the destroyed View owned. The tests below assert that + * the gate is shut after stop() and that a record past it is inert. That pins + * the gate. It does not prove the memory is safe: with no sanitizer, a read of + * freed memory can still pass. A sanitizer build running these same tests is + * what would catch a regression in the memory itself. */ -#include +#include #include @@ -585,440 +590,364 @@ TEST(MetricsRegistryParseLedgerRange, reads_back_what_the_real_producer_wrote) EXPECT_EQ(recovered.size(), ledgers.iterative_size()); } -// When telemetry is globally enabled, MetricsRegistry.cpp requires xrpld -// link dependencies we cannot satisfy in a standalone GTest binary. -#ifndef XRPL_ENABLE_TELEMETRY +// --------------------------------------------------------------------------- +// 4. The registry lifecycle. +// +// The core is compiled into xrpl.libxrpl, which this binary links in both +// builds, so every test below runs in both. The headers here serve only this +// group, and two of them name types that exist in one build only, so they sit +// beside their uses rather than at the top of the file. +// --------------------------------------------------------------------------- -#include -#include #include -#include -#include - -#include -#include - -using namespace xrpl; +#ifdef XRPL_ENABLE_TELEMETRY +#include +#include +#include +#endif namespace { /** - * OTLP/HTTP endpoint given to every registry below. Nothing ever dials it - * -- these tests exercise the no-op path -- it just has to be a plausible URL. - * It reaches the constructor through @ref kTestOptions. + * OTLP/HTTP endpoint every registry below is given. + * + * Port 1 has no listener, so the one export attempt a test can provoke gets an + * immediate connection refusal. The reader's interval is 10 s, so no periodic + * export fires inside a test; stop() is what exports, because Shutdown() + * performs a final collect-and-export drain. A routable-but-dead address would + * make every test that calls stop() wait out the 5 s export timeout. */ -constexpr std::string_view kTestEndpoint{"http://localhost:4318/v1/metrics"}; +constexpr std::string_view kTestEndpoint{"http://127.0.0.1:1/v1/metrics"}; /** - * The only Options field these tests need. - * - * The constructor takes the Options aggregate, not a string. The other fields - * -- resource identity, network id, TLS paths -- are never read on the no-op - * path, and their defaults already mean "unset". One shared value keeps every - * construction on the same endpoint. + * Resource identity stamped on the test pipeline. Fixed values, so any + * difference a test sees between two registries comes from the objects and not + * from their config. */ -telemetry::MetricsRegistry::Options const kTestOptions{.endpoint = std::string{kTestEndpoint}}; +constexpr std::string_view kTestServiceName{"metrics-registry-test-service"}; +constexpr std::string_view kTestServiceVersion{"0.0.0-test"}; +constexpr std::string_view kTestInstanceId{"metrics-registry-test-instance"}; +constexpr std::string_view kTestNodeId{"metrics-registry-test-node"}; /** - * Minimal mock ServiceRegistry for MetricsRegistry testing. + * Options for every registry below. * - * Only the getMetricsRegistry() call is used in the tests; other methods - * are not invoked because the registry is disabled (enabled=false) so no - * gauge callbacks execute. + * A function rather than a namespace-scope constant: the fields are + * std::string, so a constant would need dynamic initialisation to run before + * the first test. * - * All pure virtual methods throw to catch accidental calls during tests. + * Default-constructed and then assigned, the shape makeMetricsRegistryOptions() + * in Application.cpp uses to build this same struct. Default construction + * leaves no member indeterminate: every std::string is empty, networkId is 0 + * and useTls is false. A designated-initializer list naming a subset would trip + * -Wmissing-designated-field-initializers, an error in this build, and would + * trip it again the next time a field is added to Options. + * + * networkId stays 0 and useTls false, so the three TLS paths stay empty: the + * exporter reads them only over TLS. No test asserts on a resource attribute, + * because nothing here reads exported points back. */ -class MockServiceRegistry : public ServiceRegistry +MetricsRegistry::Options +testOptions() { - [[noreturn]] static void - throwUnimplemented() - { - throw std::logic_error("MockServiceRegistry: method not implemented"); - } - -public: - // ServiceRegistry interface — stubs that should never be called. - CollectorManager& - getCollectorManager() override - { - throwUnimplemented(); - } - Family& - getNodeFamily() override - { - throwUnimplemented(); - } - TimeKeeper& - getTimeKeeper() override - { - throwUnimplemented(); - } - JobQueue& - getJobQueue() override - { - throwUnimplemented(); - } - NodeCache& - getTempNodeCache() override - { - throwUnimplemented(); - } - CachedSLEs& - getCachedSLEs() override - { - throwUnimplemented(); - } - NetworkIDService& - getNetworkIDService() override - { - throwUnimplemented(); - } - AmendmentTable& - getAmendmentTable() override - { - throwUnimplemented(); - } - HashRouter& - getHashRouter() override - { - throwUnimplemented(); - } - LoadFeeTrack& - getFeeTrack() override - { - throwUnimplemented(); - } - LoadManager& - getLoadManager() override - { - throwUnimplemented(); - } - RCLValidations& - getValidations() override - { - throwUnimplemented(); - } - ValidatorList& - getValidators() override - { - throwUnimplemented(); - } - ValidatorSite& - getValidatorSites() override - { - throwUnimplemented(); - } - ManifestCache& - getValidatorManifests() override - { - throwUnimplemented(); - } - ManifestCache& - getPublisherManifests() override - { - throwUnimplemented(); - } - Overlay& - getOverlay() override - { - throwUnimplemented(); - } - Cluster& - getCluster() override - { - throwUnimplemented(); - } - PeerReservationTable& - getPeerReservations() override - { - throwUnimplemented(); - } - resource::Manager& - getResourceManager() override - { - throwUnimplemented(); - } - node_store::Database& - getNodeStore() override - { - throwUnimplemented(); - } - SHAMapStore& - getSHAMapStore() override - { - throwUnimplemented(); - } - RelationalDatabase& - getRelationalDatabase() override - { - throwUnimplemented(); - } - InboundLedgers& - getInboundLedgers() override - { - throwUnimplemented(); - } - InboundTransactions& - getInboundTransactions() override - { - throwUnimplemented(); - } - TaggedCache& - getAcceptedLedgerCache() override - { - throwUnimplemented(); - } - LedgerMaster& - getLedgerMaster() override - { - throwUnimplemented(); - } - LedgerCleaner& - getLedgerCleaner() override - { - throwUnimplemented(); - } - LedgerReplayer& - getLedgerReplayer() override - { - throwUnimplemented(); - } - PendingSaves& - getPendingSaves() override - { - throwUnimplemented(); - } - // AcquireStats lives in src/xrpld/ and is only forward-declared here; a - // reference return to an incomplete type is fine because this throws. - AcquireStats& - getAcquireStats() override - { - throwUnimplemented(); - } - [[nodiscard]] OpenLedger& - getOpenLedger() override - { - throwUnimplemented(); - } - [[nodiscard]] OpenLedger const& - getOpenLedger() const override - { - throwUnimplemented(); - } - NetworkOPs& - getOPs() override - { - throwUnimplemented(); - } - OrderBookDB& - getOrderBookDB() override - { - throwUnimplemented(); - } - TransactionMaster& - getMasterTransaction() override - { - throwUnimplemented(); - } - TxQ& - getTxQ() override - { - throwUnimplemented(); - } - PathRequestManager& - getPathRequestManager() override - { - throwUnimplemented(); - } - ServerHandler& - getServerHandler() override - { - throwUnimplemented(); - } - perf::PerfLog& - getPerfLog() override - { - throwUnimplemented(); - } - telemetry::Telemetry& - getTelemetry() override - { - throwUnimplemented(); - } - telemetry::MetricsRegistry* - getMetricsRegistry() override - { - return nullptr; - } - [[nodiscard]] bool - isStopping() const override - { - return false; - } - beast::Journal - getJournal(std::string const&) override - { - return beast::Journal(beast::Journal::getNullSink()); - } - boost::asio::io_context& - getIOContext() override - { - throwUnimplemented(); - } - Logs& - getLogs() override - { - throwUnimplemented(); - } - [[nodiscard]] std::optional const& - getTrapTxID() const override - { - static std::optional const kEmpty; - return kEmpty; - } - DatabaseCon& - getWalletDB() override - { - throwUnimplemented(); - } - Application& - getApp() override - { - throwUnimplemented(); - } -}; + MetricsRegistry::Options options; + options.endpoint = std::string{kTestEndpoint}; + options.serviceName = std::string{kTestServiceName}; + options.serviceVersion = std::string{kTestServiceVersion}; + options.serviceInstanceId = std::string{kTestInstanceId}; + options.nodeId = std::string{kTestNodeId}; + return options; +} /** - * Test fixture that provides a MockServiceRegistry and null Journal. + * Call every record and increment method on @p registry once. + * + * All thirteen are driven from one place, so a method added to the class + * without a line here reads as an uncovered method rather than as a passing + * test. + * + * @param registry The registry to drive. + * @param tag Folded into every attribute value, so one call's label sets + * are disjoint from another call's. A tag unused before + * stop() is what makes each set first-seen afterwards. + */ +void +recordEverything(MetricsRegistry& registry, std::string const& tag) +{ + registry.recordRpcStarted("started_" + tag); + registry.recordRpcFinished("finished_" + tag, 1000); + registry.recordRpcErrored("errored_" + tag, 500); + registry.recordJobQueued("queued_" + tag, "ProcessLData"); + registry.recordJobStarted("started_" + tag, "RcvGetLedger", 200); + registry.recordJobFinished("finished_" + tag, "RcvGetObjByHash", 3000); + registry.incrementLedgersClosed(); + registry.incrementValidationsSent(); + registry.incrementValidationsChecked(); + registry.incrementStateChanges(); + registry.incrementLedgerHistoryMismatch("mismatch_" + tag); + registry.incrementTxqExpired(); + registry.incrementTxqDropped("dropped_" + tag); +} + +/** + * Fixture for the lifecycle tests. + * + * Holds the journal only. Each test builds its own registry: the class is + * neither copyable nor movable, and each test needs its own enable flag or its + * own stop ordering. */ class MetricsRegistryTest : public ::testing::Test { protected: - MockServiceRegistry mockApp_; beast::Journal j_{beast::Journal::getNullSink()}; }; } // namespace +// --------------------------------------------------------------------------- +// The disabled path. enabled=false makes every method inert in both builds, so +// every assertion here holds unguarded. +// --------------------------------------------------------------------------- + TEST_F(MetricsRegistryTest, disabled_construction) { - // Construct with enabled=false; should be a no-op. - telemetry::MetricsRegistry const registry(false, mockApp_, j_, kTestOptions); - EXPECT_FALSE(registry.isEnabled()); + MetricsRegistry const registry(false, j_, testOptions()); + + EXPECT_EQ(registry.isEnabled(), false); + + // Mutation: drop the `enabled_ &&` term from recording(). A disabled + // registry would report itself recordable, and every call site would walk + // into an instrument that was never created. + EXPECT_EQ(registry.recording(), false); + + // Mutation: delete `if (!enabled_) return;` from the constructor. A node + // with telemetry off would open an OTLP exporter and start a reader + // thread. In a telemetry-off build the same value comes from the #else + // branch of hasPipeline(). + EXPECT_EQ(registry.hasPipeline(), false); } TEST_F(MetricsRegistryTest, disabled_construct_stop) { - telemetry::MetricsRegistry registry(false, mockApp_, j_, kTestOptions); + MetricsRegistry registry(false, j_, testOptions()); - // stop() should be a no-op when disabled. + registry.stop(); registry.stop(); - // Double stop should be safe. - registry.stop(); -} - -// --------------------------------------------------------------------------- -// The two startup phases: construction, then startAsyncGauges(). -// -// Why two phases: the constructor needs only config strings, so it can run in -// the Application's member-init list, before any subsystem that records a -// metric exists. The observable-instrument callbacks registered by -// startAsyncGauges() read live Application services (getOverlay() asserts -// overlay_ is non-null), so they wait until those services are built. -// -// SCOPE OF THESE TESTS -- read before adding to them. MetricsRegistry.cpp is -// compiled into this binary ONLY when telemetry is OFF -// (src/tests/libxrpl/CMakeLists.txt -- the `else()` branch; when it is ON the -// .cpp needs concrete xrpld types such as LedgerMaster, TxQ, NetworkOPs, -// Overlay and node_store::Database, which a standalone GTest binary cannot -// link). The constructor body and startAsyncGauges() sit inside -// #ifdef XRPL_ENABLE_TELEMETRY, so here they compile to empty bodies. So these -// tests pin the API SURFACE -- that the entry points exist, are callable in -// the documented order, and leave the object usable -- and NOT the gauge -// behaviour. Real coverage of "gauges observe values only after -// startAsyncGauges()" is unreachable from this target; it needs the enabled -// path plus an in-memory metric reader. -// --------------------------------------------------------------------------- - -TEST_F(MetricsRegistryTest, async_gauges_after_construction_is_safe) -{ - telemetry::MetricsRegistry registry(false, mockApp_, j_, kTestOptions); - - // The documented order: instruments at construction, gauges second. - registry.startAsyncGauges(); - - // State: the enable flag is untouched by either phase. Exact value, not - // merely "falsy" -- a phase that flipped it would be a real defect. + // Mutation, telemetry-on build: delete `if (!provider_) return;` from + // stop(). provider_ is null on this path because the constructor returned + // before building it, so the first call would dereference an empty + // shared_ptr. With telemetry off stop() has no body to break, and the three + // values below are what that build pins. EXPECT_EQ(registry.isEnabled(), false); - - // Synchronous recording must work off construction alone. Nothing here - // needs the gauges to be registered. - registry.recordRpcStarted("server_info"); - registry.recordRpcFinished("server_info", 1000); - - registry.stop(); - EXPECT_EQ(registry.isEnabled(), false); -} - -TEST_F(MetricsRegistryTest, async_gauges_twice_is_safe) -{ - telemetry::MetricsRegistry registry(false, mockApp_, j_, kTestOptions); - - // A second arm must be a no-op, not a second set of instruments. On the - // enabled path the Phase guard logs and returns; here the stub returns. - registry.startAsyncGauges(); - registry.startAsyncGauges(); - EXPECT_EQ(registry.isEnabled(), false); - - registry.recordJobQueued("ledgerData", "ProcessLData"); - registry.stop(); -} - -TEST_F(MetricsRegistryTest, async_gauges_respect_the_compile_time_guard) -{ - // Constructed with enabled=true, which on the enabled path would build the - // pipeline and register instruments for real. In this build - // XRPL_ENABLE_TELEMETRY is undefined, so both phases compile to the stub - // branch and neither touches the mock -- every MockServiceRegistry - // accessor throws, so a callback that actually ran would surface as a - // thrown exception here. - telemetry::MetricsRegistry registry(true, mockApp_, j_, kTestOptions); - - // Cause, not just state: the flag really is true, so the no-op below is - // attributable to the compile-time guard and not to an early enabled_ - // return. - EXPECT_EQ(registry.isEnabled(), true); - - EXPECT_NO_THROW(registry.startAsyncGauges()); - EXPECT_NO_THROW(registry.stop()); - - EXPECT_EQ(registry.isEnabled(), true); + EXPECT_EQ(registry.recording(), false); + EXPECT_EQ(registry.hasPipeline(), false); } TEST_F(MetricsRegistryTest, disabled_recording_methods) { - telemetry::MetricsRegistry registry(false, mockApp_, j_, kTestOptions); + MetricsRegistry registry(false, j_, testOptions()); - // All recording methods should be no-ops (not crash). - registry.recordRpcStarted("server_info"); - registry.recordRpcFinished("server_info", 1000); - registry.recordRpcErrored("ledger", 500); - registry.recordJobQueued("ledgerData", "ProcessLData"); - registry.recordJobStarted("ledgerData", "ProcessLData", 200); - registry.recordJobFinished("ledgerData", "ProcessLData", 3000); + // A crash canary rather than a guard with a single-line mutation: both the + // recording() test and the null-instrument test would have to go before a + // record method faulted here. It is the line a sanitizer build turns into + // real coverage. + EXPECT_NO_THROW(recordEverything(registry, "disabled")); + + // State after the sweep. enabled_ is `bool const`, so no mutation can turn + // the first two lines red on this path; hasPipeline() is the one that can -- + // a record path that assigned provider_ would fail it. + EXPECT_EQ(registry.isEnabled(), false); + EXPECT_EQ(registry.recording(), false); + EXPECT_EQ(registry.hasPipeline(), false); registry.stop(); + EXPECT_EQ(registry.isEnabled(), false); + EXPECT_EQ(registry.recording(), false); } -TEST_F(MetricsRegistryTest, destructor_calls_stop) +// --------------------------------------------------------------------------- +// The enabled path. In a telemetry-on build these drive a real OTLP exporter, +// MeterProvider and set of instruments; in a telemetry-off build they drive the +// stubs, where recording() is just the enable flag and no pipeline exists. +// --------------------------------------------------------------------------- + +TEST_F(MetricsRegistryTest, enabled_registry_records_from_construction) { - { - // Let the destructor handle cleanup. - telemetry::MetricsRegistry const registry(false, mockApp_, j_, kTestOptions); - } - // If we get here without crash, the destructor handled stop. + // The registry is usable the moment it exists, which is why Application + // can declare it ahead of every subsystem that records. Nothing stops it + // here either, so the scope exit also covers the enabled destructor path. + // const because every method this test calls is const. + MetricsRegistry const registry(true, j_, testOptions()); + + // Mutation: isEnabled() returning a literal false. The disabled test + // asserts the opposite value, so only the enabled tests catch this. Not + // "drop the enabled_(enabled) member init" -- enabled_ is `bool const` with + // no default, so a constructor omitting it does not compile. + EXPECT_EQ(registry.isEnabled(), true); + + // Mutation: seed phase_ with Phase::Stopped instead of Phase::Ready. + // Nothing in the class ever stores Ready, so the node would stay silent + // for its whole run. Also catches recording() testing phase_ == Stopped. + EXPECT_EQ(registry.recording(), true); + +#ifdef XRPL_ENABLE_TELEMETRY + // Telemetry-on only, and the property this refactoring exists to test: the + // constructor builds the pipeline, so no later start() call has to. + // Mutation: delete the initExporterAndProvider(options) call from the + // constructor's try block. + EXPECT_EQ(registry.hasPipeline(), true); +#else + // Telemetry-off: the pipeline compiles out, so hasPipeline() is a literal + // false. Mutation: return true from that #else branch, which would tell a + // caller it may register an observable that can never export. + EXPECT_EQ(registry.hasPipeline(), false); +#endif } -#endif // !XRPL_ENABLE_TELEMETRY +TEST_F(MetricsRegistryTest, every_record_method_runs_while_recording) +{ + MetricsRegistry registry(true, j_, testOptions()); + ASSERT_EQ(registry.recording(), true); + + // The one test that drives all thirteen real entry points against a real + // SDK provider. No point can be read back -- the core owns its provider and + // exposes no reader -- so the sweep is a crash canary and the assertions + // below are the deterministic part. + EXPECT_NO_THROW(recordEverything(registry, "live")); + + // Mutation: a record method that stores Phase::Stopped or resets provider_ + // as a side effect. Either would silence the node after its first metric. + EXPECT_EQ(registry.isEnabled(), true); + EXPECT_EQ(registry.recording(), true); +#ifdef XRPL_ENABLE_TELEMETRY + EXPECT_EQ(registry.hasPipeline(), true); +#endif +} + +TEST_F(MetricsRegistryTest, stop_closes_the_gate_and_leaves_enabled_true) +{ + MetricsRegistry registry(true, j_, testOptions()); + + // Setup: the gate really is open, so a false reading below is attributable + // to stop() and not to construction. + ASSERT_EQ(registry.recording(), true); + + registry.stop(); + + // isEnabled() reports what config asked for; recording() reports whether a + // record call is safe. The value of this line is the PAIR it forms with the + // recording() assertion below -- true beside false -- which is what shows + // the gate is a phase and not the enable flag. + // + // Named honestly, because no single-line change makes this line fail in + // BOTH builds. enabled_ is `bool const`, so clearing it in stop() does not + // compile -- the type already forbids the defect. Rewriting isEnabled() as + // recording() is red only where stop() can move the phase, which is the + // telemetry-on build. + EXPECT_EQ(registry.isEnabled(), true); + +#ifdef XRPL_ENABLE_TELEMETRY + // Mutation: delete the phase_.store(Phase::Stopped, release) line from + // stop(). Every XRPL_METRIC_* call site reads recording() before it + // touches an instrument, so that one store is the whole gate. + EXPECT_EQ(registry.recording(), false); + + // A separate observation from the gate, because a separate line does it: + // one stores the phase, another drops the provider. Mutation: delete + // provider_.reset() from stop(). + EXPECT_EQ(registry.hasPipeline(), false); +#else + // Telemetry-off: stop()'s body is entirely inside the guard, so it cannot + // move phase_, and recording() is the enable flag here. Pinned so that an + // #else branch which started gating shows up as a change. + EXPECT_EQ(registry.recording(), true); +#endif +} + +TEST_F(MetricsRegistryTest, records_after_stop_are_inert) +{ + MetricsRegistry registry(true, j_, testOptions()); + + // Label sets that already have SDK storage by the time stop() runs. + recordEverything(registry, "before_stop"); + + registry.stop(); + +#ifdef XRPL_ENABLE_TELEMETRY + ASSERT_EQ(registry.recording(), false); +#endif + + // The same thirteen methods with a tag never used before stop(), so every + // attribute set here is first-seen -- including four histogram records + // across three instruments, which is the case that allocates through the + // AggregationConfig the destroyed View owned. + // + // Mutation: delete the `!recording()` test from any record method. That is + // the regression this file exists for, and it is reliably red only under a + // sanitizer: with none, a read of freed memory can still return and pass. + EXPECT_NO_THROW(recordEverything(registry, "after_stop")); + + // Deterministic part: no record path reopens the gate or rebuilds the + // pipeline. + EXPECT_EQ(registry.isEnabled(), true); +#ifdef XRPL_ENABLE_TELEMETRY + EXPECT_EQ(registry.recording(), false); + EXPECT_EQ(registry.hasPipeline(), false); +#endif +} + +TEST_F(MetricsRegistryTest, stop_twice_is_safe) +{ + MetricsRegistry registry(true, j_, testOptions()); + + registry.stop(); + + // run() and the Application destructor both call stop(), so a second call + // is the ordinary shutdown path. Mutation: delete `if (!provider_) return;` + // from stop(). provider_ is null by now, so this call would dereference an + // empty shared_ptr on every clean shutdown. The registry's own destructor + // then makes a third call. + EXPECT_NO_THROW(registry.stop()); + + EXPECT_EQ(registry.isEnabled(), true); +#ifdef XRPL_ENABLE_TELEMETRY + EXPECT_EQ(registry.recording(), false); + EXPECT_EQ(registry.hasPipeline(), false); +#endif +} + +#ifdef XRPL_ENABLE_TELEMETRY + +// --------------------------------------------------------------------------- +// getValidationTracker() is declared in a telemetry-on build only, because only +// the observable-gauge callbacks drain the tracker. Two production call sites +// reach it this way, so the accessor has to hand back the live member. +// --------------------------------------------------------------------------- + +TEST_F(MetricsRegistryTest, validation_tracker_is_owned_per_registry) +{ + MetricsRegistry first(true, j_, testOptions()); + MetricsRegistry second(true, j_, testOptions()); + + // Setup: both trackers start empty, so a count below is attributable to + // the record call and not to fixture state. + ASSERT_EQ(first.getValidationTracker().totalValidationsSent(), 0u); + ASSERT_EQ(second.getValidationTracker().totalValidationsSent(), 0u); + + first.getValidationTracker().recordOurValidation( + xrpl::uint256{std::uint64_t{7}}, xrpl::LedgerIndex{7}); + + // Exact counts on both sides: the reference is live, so the write lands, + // and it lands on one registry only. + // + // Mutation: return a reference to a function-local static from + // getValidationTracker(). One shared tracker would put this count on + // `second` as well, so two registries in one process would report one + // merged agreement figure. + EXPECT_EQ(first.getValidationTracker().totalValidationsSent(), 1u); + EXPECT_EQ(second.getValidationTracker().totalValidationsSent(), 0u); +} + +#endif // XRPL_ENABLE_TELEMETRY diff --git a/src/tests/libxrpl/telemetry/ValidationTracker.cpp b/src/tests/libxrpl/telemetry/ValidationTracker.cpp index 6acbedbda0..5ba3d24c98 100644 --- a/src/tests/libxrpl/telemetry/ValidationTracker.cpp +++ b/src/tests/libxrpl/telemetry/ValidationTracker.cpp @@ -7,7 +7,7 @@ * period and a bucket boundary reachable without waiting for one. */ -#include +#include #include #include diff --git a/src/xrpld/app/consensus/RCLConsensus.cpp b/src/xrpld/app/consensus/RCLConsensus.cpp index 3768525b24..08c323e711 100644 --- a/src/xrpld/app/consensus/RCLConsensus.cpp +++ b/src/xrpld/app/consensus/RCLConsensus.cpp @@ -18,8 +18,6 @@ #include #include #include -#include -#include #include #include @@ -66,6 +64,8 @@ #include #include #include +#include +#include #include #include diff --git a/src/xrpld/app/ledger/LedgerHistory.cpp b/src/xrpld/app/ledger/LedgerHistory.cpp index d94f114d33..71dcbfbec1 100644 --- a/src/xrpld/app/ledger/LedgerHistory.cpp +++ b/src/xrpld/app/ledger/LedgerHistory.cpp @@ -4,7 +4,6 @@ #include #include #include -#include #include #include @@ -22,6 +21,7 @@ #include #include #include +#include #include #include diff --git a/src/xrpld/app/ledger/detail/LedgerMaster.cpp b/src/xrpld/app/ledger/detail/LedgerMaster.cpp index 435615c129..d97291e5f5 100644 --- a/src/xrpld/app/ledger/detail/LedgerMaster.cpp +++ b/src/xrpld/app/ledger/detail/LedgerMaster.cpp @@ -18,7 +18,6 @@ #include #include #include -#include #include #include @@ -58,6 +57,7 @@ #include #include #include +#include #include #include diff --git a/src/xrpld/app/main/Application.cpp b/src/xrpld/app/main/Application.cpp index c0e99695a6..fd68962267 100644 --- a/src/xrpld/app/main/Application.cpp +++ b/src/xrpld/app/main/Application.cpp @@ -39,7 +39,7 @@ #include #include #include -#include +#include #include #include @@ -104,6 +104,7 @@ #include #include #include +#include #include #include @@ -136,6 +137,7 @@ #include #include #include +#include #include #include #include @@ -291,13 +293,21 @@ public: std::pair nodeIdentity_; std::unique_ptr telemetry_; /** - * OTel metrics registry for gap-fill metrics (counters, histograms, - * observable gauges). Its constructor builds the pipeline and every - * synchronous instrument, so it must stay declared after telemetry_ and - * before every subsystem that records a metric. Declaration order is the - * whole guarantee. Gauges are armed later by startTelemetryGauges(). + * OTel metrics pipeline for gap-fill metrics. Its constructor builds the + * provider and every synchronous instrument (counters and histograms), so + * it must stay declared after telemetry_ and before every subsystem that + * records a metric. Declaration order is the whole guarantee. */ std::unique_ptr metricsRegistry_; + /** + * The observable gauges, whose callbacks read live application services. + * Armed later by startTelemetryGauges(). + * + * Declared after metricsRegistry_ because members are destroyed in reverse + * declaration order: the gauges hold a MetricsRegistry& and must be + * destroyed before it. + */ + std::unique_ptr metricGauges_; Application::MutexType masterMutex_; // Required by the SHAMapStore @@ -438,11 +448,18 @@ public: , metricsRegistry_( std::make_unique( telemetry_->isEnabled(), - *this, logs_->journal("MetricsRegistry"), makeMetricsRegistryOptions( *config_, toBase58(TokenType::NodePublic, nodeIdentity_.first)))) + // Registers nothing until startAsyncGauges(), so building it here + // costs nothing and keeps the member non-null for the whole lifetime. + // That is what lets the shutdown path call it unconditionally. + , metricGauges_( + std::make_unique( + *metricsRegistry_, + *this, + logs_->journal("MetricsRegistry"))) , txMaster_(*this) , collectorManager_(makeCollectorManager( @@ -634,24 +651,35 @@ public: */ ~ApplicationImp() override { + // Each step is isolated, so a throw from one still leaves the others to + // run. Skipping stopMetricsRegistry() would be the costly one: nothing + // would join the OTel reader thread until ~MetricsRegistry(), and + // metricGauges_ is destroyed before that, so the callbacks' instrument + // handles would go away while the thread was still sampling them. + // // A shutdown diagnostic must never terminate the process, and a - // destructor is implicitly noexcept. - try - { - collectorManager_->collector()->onCollectionStopping(); - stopMetricsRegistry(); - telemetry_->stop(); - } - catch (std::exception const& e) - { - JLOG(journal_.error()) << "Error stopping telemetry: " << e.what(); - } - catch (...) - { - // The callees reach third-party SDK code, which may throw something - // outside std::exception. Escaping here would terminate the process. - JLOG(journal_.error()) << "Error stopping telemetry: unknown exception"; - } + // destructor is implicitly noexcept, so nothing may escape. + auto const stopStep = [this](std::string_view name, auto&& step) noexcept { + try + { + step(); + } + catch (std::exception const& e) + { + JLOG(journal_.error()) << "Error stopping " << name << ": " << e.what(); + } + catch (...) + { + // The callees reach third-party SDK code, which may throw + // something outside std::exception. + JLOG(journal_.error()) << "Error stopping " << name << ": unknown exception"; + } + }; + + // Both observers stop before telemetry, which they export through. + stopStep("collector", [this] { collectorManager_->collector()->onCollectionStopping(); }); + stopStep("metrics registry", [this] { stopMetricsRegistry(); }); + stopStep("telemetry", [this] { telemetry_->stop(); }); } //-------------------------------------------------------------------------- @@ -1337,13 +1365,13 @@ private: * nodeStore_, nodeFamily_, validators_, acceptedLedgerCache_, * cachedSLEs_, acquireStats_, timeKeeper_, relationalDatabase_, * inboundLedgers_, feeTrack_) are built earlier in setup(). See - * MetricsRegistry::startAsyncGauges() for the full list. + * AppMetricGauges::startAsyncGauges() for the full list. */ void startTelemetryGauges() const; /** - * Stop the metrics registry: detach its gauge callbacks and join its + * Detach the gauge callbacks, then stop the metrics pipeline and join its * reader thread. Idempotent. Called from run() before any observed * service stops, and again from ~ApplicationImp for the paths that never * reach run(). @@ -1800,16 +1828,17 @@ ApplicationImp::startTelemetry() const void ApplicationImp::startTelemetryGauges() const { - metricsRegistry_->startAsyncGauges(); + metricGauges_->startAsyncGauges(); } void ApplicationImp::stopMetricsRegistry() const { - // stop() detaches the callbacks and then shuts the provider down, which - // joins the reader thread, so once it returns no callback is running or - // can start. The cost is that metrics recorded after this point are not - // exported. + // Detach first, then stop. Detaching guarantees no gauge callback reads an + // application service or the meter after this point; stop() then closes the + // recording gate and destroys the provider. Reversed, a callback already + // running on the OTel reader thread could touch a destroyed provider. + metricGauges_->detachCallbacks(); metricsRegistry_->stop(); } @@ -1887,7 +1916,7 @@ ApplicationImp::run() // Both observers stop before any service below is stopped. The collector's // gauge callbacks run hook handlers that read ledgerMaster_, networkOPs_, - // the peer finder, the job queue and overlay_; the registry's callbacks + // the peer finder, the job queue and overlay_; the metric gauges' callbacks // run on the OTel reader thread and read nodeStore_, overlay_, networkOPs_, // loadManager_, ledgerMaster, inboundLedgers and more. Each call returns // once no callback is running or can start. diff --git a/src/xrpld/app/misc/NetworkOPs.cpp b/src/xrpld/app/misc/NetworkOPs.cpp index 55462d0a6b..efa3435cbd 100644 --- a/src/xrpld/app/misc/NetworkOPs.cpp +++ b/src/xrpld/app/misc/NetworkOPs.cpp @@ -30,7 +30,6 @@ #include #include #include -#include #include #include @@ -119,6 +118,7 @@ #include #include #include +#include #include #include diff --git a/src/xrpld/app/misc/detail/TxQ.cpp b/src/xrpld/app/misc/detail/TxQ.cpp index ae390aabce..33d9e22ea0 100644 --- a/src/xrpld/app/misc/detail/TxQ.cpp +++ b/src/xrpld/app/misc/detail/TxQ.cpp @@ -3,7 +3,6 @@ #include #include #include -#include #include #include @@ -33,6 +32,7 @@ #include #include #include +#include #include #include #include diff --git a/src/xrpld/overlay/detail/PeerImp.cpp b/src/xrpld/overlay/detail/PeerImp.cpp index d575bca584..e2d40b0a7f 100644 --- a/src/xrpld/overlay/detail/PeerImp.cpp +++ b/src/xrpld/overlay/detail/PeerImp.cpp @@ -24,7 +24,6 @@ #include #include #include -#include #include #include @@ -75,6 +74,7 @@ #include #include #include +#include #include #include #include diff --git a/src/xrpld/perflog/detail/PerfLogImp.cpp b/src/xrpld/perflog/detail/PerfLogImp.cpp index 4173ea1a7c..42e0bf2061 100644 --- a/src/xrpld/perflog/detail/PerfLogImp.cpp +++ b/src/xrpld/perflog/detail/PerfLogImp.cpp @@ -1,12 +1,11 @@ #include #include -#include #ifdef XRPL_ENABLE_TELEMETRY // Only the recording calls below and the metric macros' expansion name the // registry, and neither survives with telemetry compiled out. -#include +#include #endif #include @@ -24,6 +23,7 @@ #include #include #include +#include #include #include diff --git a/src/xrpld/rpc/detail/PathRequest.cpp b/src/xrpld/rpc/detail/PathRequest.cpp index b61bf1edee..c891111978 100644 --- a/src/xrpld/rpc/detail/PathRequest.cpp +++ b/src/xrpld/rpc/detail/PathRequest.cpp @@ -12,7 +12,6 @@ #include #include #include -#include #include #include @@ -40,6 +39,7 @@ #include #include #include +#include #include #include #include diff --git a/src/xrpld/rpc/detail/ServerHandler.cpp b/src/xrpld/rpc/detail/ServerHandler.cpp index 9730e4519f..eefef37909 100644 --- a/src/xrpld/rpc/detail/ServerHandler.cpp +++ b/src/xrpld/rpc/detail/ServerHandler.cpp @@ -14,7 +14,6 @@ #include #include #include // IWYU pragma: keep -#include #include #include @@ -54,6 +53,7 @@ #include #include #include +#include #include #include diff --git a/src/xrpld/telemetry/MetricsRegistry.cpp b/src/xrpld/telemetry/AppMetricGauges.cpp similarity index 60% rename from src/xrpld/telemetry/MetricsRegistry.cpp rename to src/xrpld/telemetry/AppMetricGauges.cpp index 0c04d16e34..655bb7195f 100644 --- a/src/xrpld/telemetry/MetricsRegistry.cpp +++ b/src/xrpld/telemetry/AppMetricGauges.cpp @@ -1,13 +1,15 @@ /** - * MetricsRegistry implementation — OpenTelemetry metric instruments for xrpld. + * AppMetricGauges implementation — the pull-model half of the OTel metric + * surface. * * This file contains: - * - Construction / destruction logic for the OTel MeterProvider pipeline. - * - Synchronous instrument creation (counters, histograms) for RPC, job - * queue, and NodeStore I/O metrics. - * - Observable gauge callback registration for cache hit rates, TxQ state, - * CountedObject instances, load factors, and NodeStore queue depth. - * - No-op stubs when XRPL_ENABLE_TELEMETRY is not defined. + * - Registration of every observable instrument whose callback samples live + * server state: cache hit rates, TxQ state, CountedObject instances, load + * factors, NodeStore I/O, server info, complete ledger ranges, validator + * health, peer quality, reduce-relay efficiency, ledger economy, state + * tracking, storage detail and validation agreement. + * - The nodestore_state helpers those callbacks publish values through. + * - The arm and disarm entry points for the whole set. */ // On Windows, OTel's spin_lock_mutex.h (transitively included from @@ -20,11 +22,7 @@ #include #endif -#include - -// Unguarded because the constructor's `beast::Journal journal` parameter is -// declared in both configurations; only the member it initialises is guarded. -#include +#include #ifdef XRPL_ENABLE_TELEMETRY @@ -38,9 +36,9 @@ // txMetrics(). // // The cycle is confined to this translation unit. No telemetry header includes -// app or overlay (MetricsRegistry.h forward-declares what it needs and takes a -// ServiceRegistry&), and all of src/xrpld builds into a single CMake target, so -// there is no header cycle and no link cycle to break. +// app or overlay -- the callbacks reach every service through the +// ServiceRegistry reference they are given -- and all of src/xrpld builds into a +// single CMake target, so there is no header cycle and no link cycle to break. // // Inverting it properly means declaring a metrics-source interface below overlay // and implementing it there, which is deliberately left as follow-up rather than @@ -58,6 +56,7 @@ #include #include #include +#include #include #include #include @@ -66,39 +65,16 @@ #include #include #include -#include -#include -#include -#include -// For networkTypeFromId(), the one xrpl.network.type mapping both export -// paths use. Adds no levelization edge: xrpld.telemetry > xrpl.telemetry -// already holds via SpanNames.h above. -#include -#include -#include -#include #include #include #include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include #include #include #include #include #include -#include #include #include #include @@ -106,194 +82,64 @@ #include #include -namespace metric_sdk = opentelemetry::sdk::metrics; -namespace otlp_http = opentelemetry::exporter::otlp; -// Not `resource`: that would collide with xrpl::resource (the resource-accounting -// namespace), which encloses every use site below. Inner-scope lookup would find -// that namespace instead of this file-scope alias. -namespace otel_resource = opentelemetry::sdk::resource; - -namespace { - -// Microsecond-valued duration histogram instrument names. Each is -// referenced twice — once to register the explicit-bucket view and once -// to create the instrument — so they are named constants to keep the two -// sites in sync (a mismatch would silently drop the bucket override). -constexpr char kJobQueuedDurationUs[] = "job_queued_us"; -constexpr char kJobRunningDurationUs[] = "job_running_us"; -constexpr char kRpcMethodDurationUs[] = "rpc_method_us"; - -// Attribute (label) keys for the job instruments. Each is referenced from -// several record sites, and a counter and its histogram must carry exactly -// the same key spelling or the two series cannot be joined in a query. -constexpr char kJobTypeLabel[] = "job_type"; -constexpr char kHandlerLabel[] = "handler"; - -/** - * Register an explicit-bucket histogram view. - * - * The SDK's default boundaries top out at 10,000, so any instrument whose - * values exceed that saturates and every quantile reads as the ceiling. The - * floor matters just as much and is easier to miss: a ladder whose first edge - * sits above the mass of the distribution makes every low quantile an - * interpolation inside bucket 0 -- a number derived from the bucket edge - * rather than from any sample. Both ends are chosen from measured - * distributions in HistogramBuckets.h. - * - * @param views The registry to add the view to. - * @param name Instrument name to match (e.g. "job_running_us"). - * @param boundaries Bucket upper bounds, ascending. - */ -void -addHistogramView( - metric_sdk::ViewRegistry& views, - std::string const& name, - std::vector boundaries) -{ - auto config = std::make_shared(); - config->boundaries_ = std::move(boundaries); - - auto selector = metric_sdk::InstrumentSelectorFactory::Create( - metric_sdk::InstrumentType::kHistogram, name, ""); - auto meterSelector = metric_sdk::MeterSelectorFactory::Create( - std::string(xrpl::telemetry::kMeterName), std::string(xrpl::telemetry::kMeterVersion), ""); - auto view = - metric_sdk::ViewFactory::Create(name, "", metric_sdk::AggregationType::kHistogram, config); - - views.AddView(std::move(selector), std::move(meterSelector), std::move(view)); -} - -/** - * Register the microsecond-ladder view for a duration instrument. - * - * Job wait/run times and RPC latencies routinely exceed the SDK default - * ceiling, so they all share `buckets::kMicrosecondBuckets`. - * - * @param views The registry to add the view to. - * @param name Instrument name to match. - */ -void -addMicrosecondHistogramView(metric_sdk::ViewRegistry& views, std::string const& name) -{ - addHistogramView( - views, - name, - xrpl::telemetry::buckets::toVector(xrpl::telemetry::buckets::kMicrosecondBuckets)); -} - -} // namespace - #endif // XRPL_ENABLE_TELEMETRY namespace xrpl::telemetry { -MetricsRegistry::MetricsRegistry( - [[maybe_unused]] bool enabled, +AppMetricGauges::AppMetricGauges( + [[maybe_unused]] MetricsRegistry& core, [[maybe_unused]] ServiceRegistry& app, - [[maybe_unused]] beast::Journal journal, - [[maybe_unused]] Options const& options) - : enabled_(enabled) + [[maybe_unused]] beast::Journal journal) #ifdef XRPL_ENABLE_TELEMETRY + : core_(core) , app_(app) + // The core logs through the same partition, so one log-level setting + // covers the whole metric pipeline. , journal_(journal) #endif { -#ifdef XRPL_ENABLE_TELEMETRY - if (!enabled_) - return; - - // useTls is logged because a collector that requires TLS rejects a - // plaintext exporter with no local error. The paths are left out. - JLOG(journal_.info()) << "MetricsRegistry: starting, endpoint=" << options.endpoint - << ", serviceName=" << options.serviceName - << ", serviceVersion=" << options.serviceVersion - << ", instanceId=" << options.serviceInstanceId - << ", nodeId=" << options.nodeId << ", networkId=" << options.networkId - << ", useTls=" << options.useTls; - - // A broken pipeline must not stop the node. The SDK is third-party code, - // so the catch-all is deliberate, as in ~ApplicationImp. - try - { - initExporterAndProvider(options); - - // Rule for anything added below: the constructor may create only - // instruments whose recording is PUSHED from app code -- counters and - // histograms. An instrument registered here is live immediately, and - // the reader thread may invoke a registered callback before the rest - // of the Application is built, so any observable whose callback reads - // an Application service belongs in startAsyncGauges(), not here. - // That includes observable COUNTERS, not just gauges: - // jq_trans_overflow_total was created here and its callback read - // getOverlay(), which asserts overlay_ is non-null. - initSyncInstruments(); - } - catch (std::exception const& e) - { - disablePipeline(e.what()); - return; - } - catch (...) - { - disablePipeline("unknown exception"); - return; - } - - JLOG(journal_.info()) << "MetricsRegistry: provider and instruments ready"; -#endif // XRPL_ENABLE_TELEMETRY } -#ifdef XRPL_ENABLE_TELEMETRY -void -MetricsRegistry::disablePipeline(std::string_view reason) +AppMetricGauges::~AppMetricGauges() { - provider_.reset(); - // A no-op meter keeps the invariant the XRPL_METRIC_* macros rely on: an - // enabled registry always has a meter, so every call site gets an inert - // instrument here with no check of its own. - meter_ = noopMeter(kMeterName); - JLOG(journal_.error()) << "MetricsRegistry: metrics pipeline failed to initialise, " - "continuing without native metrics: " - << reason; -} -#endif // XRPL_ENABLE_TELEMETRY - -MetricsRegistry::~MetricsRegistry() -{ - stop(); + // A last resort, not the teardown path: the flag this sets lives here, so + // it cannot protect anything once this object is gone. The safe order is + // detachCallbacks(), then the core's stop() to join the reader thread, + // then destruction. + detachCallbacks(); } void -MetricsRegistry::startAsyncGauges() +AppMetricGauges::startAsyncGauges() { #ifdef XRPL_ENABLE_TELEMETRY - if (!enabled_) + if (!core_.isEnabled()) return; // One arm per life. A second call would create a second set of // same-named instruments, and a call after stop() would register on a // provider that is gone. Checked before the pipeline, so a call after - // stop() is reported as what it is and not as a build failure. - auto const currentPhase = phase_.load(std::memory_order_relaxed); - if (currentPhase != Phase::Ready) + // stop() is reported as what it is and not as a build failure. The core + // is enabled by here, so a false recording() means exactly stopped. + bool const stopped = !core_.recording(); + if (armed_ || stopped) { JLOG(journal_.warn()) << "MetricsRegistry: startAsyncGauges() called " - << (currentPhase == Phase::Stopped ? "after stop()" : "twice") - << "; ignored"; + << (stopped ? "after stop()" : "twice") << "; ignored"; return; } // The pipeline failed to build: the meter is a no-op, so registering - // gauges on it would only log a success that is not one. phase_ stays - // at Ready, so a second call lands here again and logs the same message. + // gauges on it would only log a success that is not one. armed_ stays + // false, so a second call lands here again and logs the same message. // Idempotent. - if (!provider_) + if (!core_.hasPipeline()) { JLOG(journal_.warn()) << "MetricsRegistry: startAsyncGauges() without a pipeline; " "no gauges registered"; return; } - phase_.store(Phase::GaugesArmed, std::memory_order_relaxed); + armed_ = true; registerAsyncGauges(); @@ -301,155 +147,8 @@ MetricsRegistry::startAsyncGauges() #endif // XRPL_ENABLE_TELEMETRY } -#ifdef XRPL_ENABLE_TELEMETRY void -MetricsRegistry::initExporterAndProvider(Options const& options) -{ - // Configure OTLP/HTTP metric exporter. The TLS settings come from the one - // [telemetry] block that also drives the trace exporter in Telemetry.cpp, - // so both exporters reach the collector on the same terms. - otlp_http::OtlpHttpMetricExporterOptions exporterOpts; - exporterOpts.url = options.endpoint; - if (options.useTls) - { - exporterOpts.ssl_ca_cert_path = options.tlsCaCertPath; - exporterOpts.ssl_client_cert_path = options.tlsClientCertPath; - exporterOpts.ssl_client_key_path = options.tlsClientKeyPath; - } - - auto exporter = otlp_http::OtlpHttpMetricExporterFactory::Create(exporterOpts); - - // Configure periodic reader with 10-second export interval. - metric_sdk::PeriodicExportingMetricReaderOptions readerOpts; - readerOpts.export_interval_millis = std::chrono::milliseconds(10000); - readerOpts.export_timeout_millis = std::chrono::milliseconds(5000); - auto reader = - metric_sdk::PeriodicExportingMetricReaderFactory::Create(std::move(exporter), readerOpts); - - // Stamp the same resource Telemetry::makeMetricsResource() builds for the - // trace pipeline. Both must agree: a node whose service.name or - // xrpl.network.type differs between the two pipelines splits its own - // series, and a dashboard filtering on either label shows only half. - // - // Use std::string, never a string literal: ResourceAttributes stores an - // OTel AttributeValue variant whose char-const* overload binds to bool, - // so a literal would be recorded as the boolean true. - otel_resource::ResourceAttributes attrs; - attrs[opentelemetry::semconv::service::kServiceName] = options.serviceName; - // int64_t, matching the trace resource. The same key with two types would - // give the two pipelines incompatible attribute values. - attrs[std::string(attr::networkId)] = static_cast(options.networkId); - // Derived here rather than passed in, so the id and the type label cannot - // disagree. Same helper the trace path uses. - attrs[std::string(attr::networkType)] = networkTypeFromId(options.networkId); - - // The three below are left off when empty rather than stamped blank. An - // absent label reads as "not reported"; an empty one looks like a value. - if (!options.serviceVersion.empty()) - attrs[opentelemetry::semconv::service::kServiceVersion] = options.serviceVersion; - if (!options.serviceInstanceId.empty()) - attrs[opentelemetry::semconv::service::kServiceInstanceId] = options.serviceInstanceId; - // xrpl.node.id: the same per-node key the trace resource carries, so - // metrics and traces resolve to one node. - if (!options.nodeId.empty()) - attrs[std::string(attr::nodeId)] = options.nodeId; - auto resourceAttrs = otel_resource::Resource::Create(attrs); - - // Build a view registry with explicit microsecond buckets for the - // duration histograms. Without this they use the SDK default buckets - // (max 10,000 = 10 ms), saturating every quantile at 10 ms. - auto views = std::make_unique(); - addMicrosecondHistogramView(*views, kJobQueuedDurationUs); - addMicrosecondHistogramView(*views, kJobRunningDurationUs); - addMicrosecondHistogramView(*views, kRpcMethodDurationUs); - // Recorded at its PeerImp.cpp call site, not created here, so the name - // comes from the shared constant both sites use. - addMicrosecondHistogramView(*views, kGetObjectLookupUs); - - // The remaining two GetObject histograms are not durations, so the - // microsecond ladder above does not fit them. Both still need explicit - // boundaries: the SDK default stops at 10,000 and both ranges exceed it. - // - // Object counts run 1..kHardMaxReplyNodes (12288). The honest sync path - // asks for at most 8, so the low buckets are fine-grained and the upper - // ones follow the charge size bands (64, 1024) up to the hard cap. - addHistogramView( - *views, kGetObjectRequestObjects, buckets::toVector(buckets::kObjectCountBuckets)); - - // Charge values span 0 (free tier) to ~99k for a full-size all-miss - // request. Boundaries bracket the resource thresholds that decide a - // peer's fate -- kWarningThreshold (5000) and kDropThreshold (25000) -- - // so a dashboard can show how close charges run to each. - addHistogramView(*views, kGetObjectCharge, buckets::toVector(buckets::kChargeBuckets)); - - // The two RPC request-count histograms are recorded at their ServerHandler - // and PathRequest call sites, so the names come from the shared constants - // all three sites use. Both are small counts, and the reason they need a - // view is the FLOOR rather than the ceiling: the SDK default edges start - // 0, 5, 10, 25, so a batch of one to five sub-requests -- the normal case -- - // would land in a single bucket and every quantile over it would be an - // interpolation inside that bucket rather than a measurement. - // - // The object-count ladder is the fit: its 1, 2, 4, 8, 16 edges sit exactly - // where both distributions have their mass. Path counts are hard-bounded at - // kMaxPaths * kMaxAutoSrcCur = 352, well under its 12288 top. Batch sizes - // have no such cap; see the ceiling note in RpcMetricNames.h. - addHistogramView(*views, kRpcBatchSize, buckets::toVector(buckets::kObjectCountBuckets)); - addHistogramView( - *views, kPathfindDiscoveredPaths, buckets::toVector(buckets::kObjectCountBuckets)); - - // Create MeterProvider with resource, then attach the metric reader. - provider_ = metric_sdk::MeterProviderFactory::Create(std::move(views), resourceAttrs); - provider_->AddMetricReader(std::move(reader)); - - // Get a meter for all xrpld instruments. - meter_ = provider_->GetMeter(std::string(kMeterName), std::string(kMeterVersion)); -} - -void -MetricsRegistry::initSyncInstruments() -{ - // RPC per-method counters and histogram. - rpcStartedCounter_ = - meter_->CreateUInt64Counter("rpc_method_started_total", "Total RPC method calls started"); - rpcFinishedCounter_ = meter_->CreateUInt64Counter( - "rpc_method_finished_total", "Total RPC method calls completed successfully"); - rpcErroredCounter_ = meter_->CreateUInt64Counter( - "rpc_method_errored_total", "Total RPC method calls that errored"); - rpcDurationHistogram_ = meter_->CreateDoubleHistogram( - kRpcMethodDurationUs, "RPC method execution time in microseconds"); - - // Job queue per-type counters and histograms. - jobQueuedCounter_ = meter_->CreateUInt64Counter("job_queued_total", "Total jobs enqueued"); - jobStartedCounter_ = meter_->CreateUInt64Counter("job_started_total", "Total jobs started"); - jobFinishedCounter_ = meter_->CreateUInt64Counter("job_finished_total", "Total jobs completed"); - jobQueuedDurationHistogram_ = meter_->CreateDoubleHistogram( - kJobQueuedDurationUs, "Time jobs spent waiting in the queue (microseconds)"); - jobRunningDurationHistogram_ = - meter_->CreateDoubleHistogram(kJobRunningDurationUs, "Job execution time in microseconds"); - - // --- External dashboard parity counters --- - ledgersClosedCounter_ = - meter_->CreateUInt64Counter("ledgers_closed_total", "Total ledgers closed by consensus"); - validationsSentCounter_ = meter_->CreateUInt64Counter( - "validations_sent_total", "Total validations sent by this node"); - validationsCheckedCounter_ = meter_->CreateUInt64Counter( - "validations_checked_total", "Total network validations received and checked"); - stateChangesCounter_ = - meter_->CreateUInt64Counter("state_changes_total", "Total operating mode changes"); - ledgerHistoryMismatchCounter_ = meter_->CreateUInt64Counter( - "ledger_history_mismatch_total", "Total built-vs-validated ledger mismatches by reason"); - txqExpiredCounter_ = meter_->CreateUInt64Counter( - "txq_expired_total", "Total transactions expired out of the transaction queue"); - txqDroppedCounter_ = meter_->CreateUInt64Counter( - "txq_dropped_total", "Total transactions refused admission to the queue by reason"); - // Note: validation_agreements_total / validation_missed_total are monotonic - // ObservableCounters created in registerValidationTotalsCounters() (below). -} -#endif // XRPL_ENABLE_TELEMETRY - -void -MetricsRegistry::detachCallbacks() noexcept +AppMetricGauges::detachCallbacks() noexcept { #ifdef XRPL_ENABLE_TELEMETRY // Release so every subsequent callback acquire-load sees true. @@ -457,172 +156,6 @@ MetricsRegistry::detachCallbacks() noexcept #endif // XRPL_ENABLE_TELEMETRY } -void -MetricsRegistry::stop() -{ -#ifdef XRPL_ENABLE_TELEMETRY - // Store Stopped with release ordering BEFORE the pipeline goes away. - // Every recording thread reads phase_ through recording() with acquire - // ordering, so any record that has not yet passed the gate will see - // Stopped and skip. Idempotent: destructor calls this after run() or - // ~ApplicationImp already did. - phase_.store(Phase::Stopped, std::memory_order_release); - if (!provider_) - return; - - JLOG(journal_.info()) << "MetricsRegistry: stopping"; - - // Belt-and-suspenders: detachCallbacks() should have already been - // called by Application shutdown before any service the callbacks - // observe was stopped. Setting the flag here is redundant for a - // correct caller but protects against a future caller that forgets - // to detach first. - callbacksDetached_.store(true, std::memory_order_release); - - // meter_ is left alone on purpose. Job threads are still running here and - // may be inside a macro, so writing meter_ would race with their read. - // The recording() gate is what keeps them off the dying pipeline: only the - // macros read meter_, and none of them does so once phase_ is Stopped. - // - // SDK teardown order: Shutdown() stops the PeriodicExportingMetricReader - // thread (so no further gauge callbacks fire) and performs the final - // collect-and-export drain itself. The trailing ForceFlush() is a - // redundant safety net (a no-op once the reader is shut down), then - // reset() destroys the provider. - // - // provider_.reset() destroys MeterProvider -> MeterContext -> ViewRegistry - // -> each View -> its shared_ptr. Live SDK - // SyncMetricStorage instances cached in call-site statics still hold a - // raw AggregationConfig pointer; a Record with a NEW attribute set after - // this point would fire the factory lambda and deref that dangling - // pointer, and a late meter()->CreateXxx would return null. - provider_->Shutdown(); - provider_->ForceFlush(); - provider_.reset(); - - JLOG(journal_.info()) << "MetricsRegistry: stopped"; -#endif // XRPL_ENABLE_TELEMETRY -} - -// ----------------------------------------------------------------- -// Synchronous instrument recording — RPC metrics -// ----------------------------------------------------------------- - -void -MetricsRegistry::recordRpcStarted([[maybe_unused]] std::string_view method) -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (!recording() || !rpcStartedCounter_) - return; - rpcStartedCounter_->Add(1, {{"method", std::string(method)}}); -#endif -} - -void -MetricsRegistry::recordRpcFinished( - [[maybe_unused]] std::string_view method, - [[maybe_unused]] std::int64_t durationUs) -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (!recording() || !rpcFinishedCounter_) - return; - rpcFinishedCounter_->Add(1, {{"method", std::string(method)}}); - if (rpcDurationHistogram_) - { - rpcDurationHistogram_->Record( - static_cast(durationUs), - {{"method", std::string(method)}}, - opentelemetry::context::Context{}); - } -#endif -} - -void -MetricsRegistry::recordRpcErrored( - [[maybe_unused]] std::string_view method, - [[maybe_unused]] std::int64_t durationUs) -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (!recording() || !rpcErroredCounter_) - return; - rpcErroredCounter_->Add(1, {{"method", std::string(method)}}); - if (rpcDurationHistogram_) - { - rpcDurationHistogram_->Record( - static_cast(durationUs), - {{"method", std::string(method)}}, - opentelemetry::context::Context{}); - } -#endif -} - -// ----------------------------------------------------------------- -// Synchronous instrument recording — Job Queue metrics -// ----------------------------------------------------------------- - -void -MetricsRegistry::recordJobQueued( - [[maybe_unused]] std::string_view jobType, - [[maybe_unused]] std::string_view jobName) -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (!recording() || !jobQueuedCounter_) - return; - jobQueuedCounter_->Add( - 1, - {{kJobTypeLabel, std::string(jobType)}, - {kHandlerLabel, std::string(sanitiseHandler(jobName))}}); -#endif -} - -void -MetricsRegistry::recordJobStarted( - [[maybe_unused]] std::string_view jobType, - [[maybe_unused]] std::string_view jobName, - [[maybe_unused]] std::int64_t queuedDurUs) -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (!recording() || !jobStartedCounter_) - return; - // Build the attribute pair once: both the counter and the histogram - // must carry the identical label set or they cannot be joined. - std::string const handler(sanitiseHandler(jobName)); - jobStartedCounter_->Add(1, {{kJobTypeLabel, std::string(jobType)}, {kHandlerLabel, handler}}); - if (jobQueuedDurationHistogram_ && queuedDurUs >= 0) - { - // Guard against negative queued durations: the caller derives this - // from a steady-clock delta that can go slightly negative under clock - // skew or reordering. The OTel SDK rejects negative histogram values - // (logging a warning per call), so skip them rather than spam. - jobQueuedDurationHistogram_->Record( - static_cast(queuedDurUs), - {{kJobTypeLabel, std::string(jobType)}, {kHandlerLabel, handler}}, - opentelemetry::context::Context{}); - } -#endif -} - -void -MetricsRegistry::recordJobFinished( - [[maybe_unused]] std::string_view jobType, - [[maybe_unused]] std::string_view jobName, - [[maybe_unused]] std::int64_t runningDurUs) -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (!recording() || !jobFinishedCounter_) - return; - std::string const handler(sanitiseHandler(jobName)); - jobFinishedCounter_->Add(1, {{kJobTypeLabel, std::string(jobType)}, {kHandlerLabel, handler}}); - if (jobRunningDurationHistogram_) - { - jobRunningDurationHistogram_->Record( - static_cast(runningDurUs), - {{kJobTypeLabel, std::string(jobType)}, {kHandlerLabel, handler}}, - opentelemetry::context::Context{}); - } -#endif -} - // ----------------------------------------------------------------- // Observable gauge callbacks // ----------------------------------------------------------------- @@ -630,7 +163,7 @@ MetricsRegistry::recordJobFinished( #ifdef XRPL_ENABLE_TELEMETRY void -MetricsRegistry::registerAsyncGauges() +AppMetricGauges::registerAsyncGauges() { // Each helper creates one observable instrument and attaches one // callback. Keeping the registration bodies in separate methods @@ -656,7 +189,7 @@ MetricsRegistry::registerAsyncGauges() } void -MetricsRegistry::registerJqTransOverflowCounter() +AppMetricGauges::registerJqTransOverflowCounter() { // jq_trans_overflow_total is observed from Overlay's existing cumulative // atomic (Overlay::getJqTransOverflow()) rather than pushed. The overlay @@ -668,11 +201,11 @@ MetricsRegistry::registerJqTransOverflowCounter() // callback reads getOverlay(), which asserts overlay_ is non-null. Arming // it any earlier would let a reader tick fire before the overlay exists, // and an assert is not caught by the try block below. - jqTransOverflowObservable_ = meter_->CreateInt64ObservableCounter( + jqTransOverflowObservable_ = core_.meter()->CreateInt64ObservableCounter( "jq_trans_overflow_total", "Total job queue transaction overflows"); jqTransOverflowObservable_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; try @@ -690,14 +223,14 @@ MetricsRegistry::registerJqTransOverflowCounter() } void -MetricsRegistry::registerCacheHitRateGauge() +AppMetricGauges::registerCacheHitRateGauge() { // --- Cache hit rate and size gauges --- cacheHitRateGauge_ = - meter_->CreateDoubleObservableGauge("cache_metrics", "Cache hit rates and sizes"); + core_.meter()->CreateDoubleObservableGauge("cache_metrics", "Cache hit rates and sizes"); cacheHitRateGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -761,13 +294,14 @@ MetricsRegistry::registerCacheHitRateGauge() } void -MetricsRegistry::registerTxqGauge() +AppMetricGauges::registerTxqGauge() { // --- TxQ metrics gauges --- - txqGauge_ = meter_->CreateDoubleObservableGauge("txq_metrics", "Transaction queue metrics"); + txqGauge_ = + core_.meter()->CreateDoubleObservableGauge("txq_metrics", "Transaction queue metrics"); txqGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -808,14 +342,14 @@ MetricsRegistry::registerTxqGauge() } void -MetricsRegistry::registerObjectCountGauge() +AppMetricGauges::registerObjectCountGauge() { // --- Counted object instance gauges --- - objectCountGauge_ = meter_->CreateInt64ObservableGauge( + objectCountGauge_ = core_.meter()->CreateInt64ObservableGauge( "object_count", "Live instance counts for key internal object types"); objectCountGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; try @@ -840,14 +374,14 @@ MetricsRegistry::registerObjectCountGauge() } void -MetricsRegistry::registerLoadFactorGauge() +AppMetricGauges::registerLoadFactorGauge() { // --- Load factor breakdown gauges --- - loadFactorGauge_ = - meter_->CreateDoubleObservableGauge("load_factor_metrics", "Fee load factor breakdown"); + loadFactorGauge_ = core_.meter()->CreateDoubleObservableGauge( + "load_factor_metrics", "Fee load factor breakdown"); loadFactorGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -913,7 +447,7 @@ MetricsRegistry::registerLoadFactorGauge() } void -MetricsRegistry::observeNodeStoreTotals(node_store::Database& db, ObserveFn const& observe) +AppMetricGauges::observeNodeStoreTotals(node_store::Database& db, ObserveFn const& observe) { // Cumulative counters (monotonically increasing). observe("node_reads_total", static_cast(db.getFetchTotalCount())); @@ -932,9 +466,10 @@ MetricsRegistry::observeNodeStoreTotals(node_store::Database& db, ObserveFn cons // latency differs. Each mean is omitted rather than reported as zero // when nothing has been read or written, so a dashboard shows a gap // instead of a plausible wrong number. - if (auto const mean = scaledMean(db.getFetchDurationUs(), db.getFetchTotalCount())) + if (auto const mean = + MetricsRegistry::scaledMean(db.getFetchDurationUs(), db.getFetchTotalCount())) observe("read_mean_us", *mean); - if (auto const mean = scaledMean(db.getStoreDurationUs(), db.getStoreCount())) + if (auto const mean = MetricsRegistry::scaledMean(db.getStoreDurationUs(), db.getStoreCount())) observe("write_mean_us", *mean); // Write load score (instantaneous). @@ -942,7 +477,7 @@ MetricsRegistry::observeNodeStoreTotals(node_store::Database& db, ObserveFn cons } void -MetricsRegistry::observeWritePathDetail(node_store::Database const& db, ObserveFn const& observe) +AppMetricGauges::observeWritePathDetail(node_store::Database const& db, ObserveFn const& observe) { auto const ws = db.getWriteStats(); if (!ws) @@ -951,7 +486,7 @@ MetricsRegistry::observeWritePathDetail(node_store::Database const& db, ObserveF observe("nudb_writers_in_flight", static_cast(ws->concurrentWriters)); observe("nudb_insert_max_us", static_cast(ws->insertMaxUs)); - if (auto const mean = scaledMean(ws->insertTotalUs, ws->insertCount)) + if (auto const mean = MetricsRegistry::scaledMean(ws->insertTotalUs, ws->insertCount)) observe("nudb_insert_mean_us", *mean); // Mean writer depth times 100. NuDB serializes inserts behind one @@ -959,12 +494,12 @@ MetricsRegistry::observeWritePathDetail(node_store::Database const& db, ObserveF // above 1.0 even under load. An integral gauge would truncate that to 1 // and lose the whole signal, hence the fixed-point scale -- which the // name states, so nobody reads 140 as 140 writers. - if (auto const mean = scaledMean(ws->depthSum, ws->depthSamples, 100)) + if (auto const mean = MetricsRegistry::scaledMean(ws->depthSum, ws->depthSamples, 100)) observe("nudb_writer_depth_x100", *mean); } void -MetricsRegistry::observeAcquireStats(AcquireStats const& stats, ObserveFn const& observe) +AppMetricGauges::observeAcquireStats(AcquireStats const& stats, ObserveFn const& observe) { // Published unconditionally: for a counter, zero is the meaningful // "no such event yet" reading, unlike for a mean. The diagnostic value @@ -987,7 +522,7 @@ MetricsRegistry::observeAcquireStats(AcquireStats const& stats, ObserveFn const& } void -MetricsRegistry::observeReadQueue(node_store::Database& db, ObserveFn const& observe) +AppMetricGauges::observeReadQueue(node_store::Database& db, ObserveFn const& observe) { json::Value obj(json::ValueType::Object); db.getCountsJson(obj); @@ -1004,24 +539,24 @@ MetricsRegistry::observeReadQueue(node_store::Database& db, ObserveFn const& obs } void -MetricsRegistry::registerNodeStoreGauge() +AppMetricGauges::registerNodeStoreGauge() { // --- NodeStore I/O gauges --- // The cumulative counters (reads, writes, bytes) are also exposed here // as observable gauges. This avoids adding an xrpld dependency into the - // libxrpl nodestore code — the MetricsRegistry reads the existing atomic + // libxrpl nodestore code — the callback reads the existing atomic // counters from Database via its public accessors. // // Every value multiplexes onto this one gauge through its `metric` // label, so a new value needs no new instrument. The body is split // across four helpers, one per domain, to stay inside the per-function // line budget and to keep each domain testable on its own. - nodeStoreGauge_ = meter_->CreateInt64ObservableGauge( + nodeStoreGauge_ = core_.meter()->CreateInt64ObservableGauge( "nodestore_state", "NodeStore I/O counters, latencies, write-queue depth and acquisition stalls"); nodeStoreGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1038,10 +573,10 @@ MetricsRegistry::registerNodeStoreGauge() // Qualified because the enclosing lambda captures nothing: // these are static members, and the explicit scope says so. - MetricsRegistry::observeNodeStoreTotals(db, observe); - MetricsRegistry::observeWritePathDetail(db, observe); - MetricsRegistry::observeAcquireStats(app.getAcquireStats(), observe); - MetricsRegistry::observeReadQueue(db, observe); + AppMetricGauges::observeNodeStoreTotals(db, observe); + AppMetricGauges::observeWritePathDetail(db, observe); + AppMetricGauges::observeAcquireStats(app.getAcquireStats(), observe); + AppMetricGauges::observeReadQueue(db, observe); } catch (...) // NOLINT(bugprone-empty-catch) { @@ -1052,14 +587,14 @@ MetricsRegistry::registerNodeStoreGauge() } void -MetricsRegistry::registerServerInfoGauge() +AppMetricGauges::registerServerInfoGauge() { // --- Server info gauges --- serverInfoGauge_ = - meter_->CreateInt64ObservableGauge("server_info", "Server-level health metrics"); + core_.meter()->CreateInt64ObservableGauge("server_info", "Server-level health metrics"); serverInfoGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1137,10 +672,11 @@ MetricsRegistry::registerServerInfoGauge() } void -MetricsRegistry::registerBuildInfoGauge() +AppMetricGauges::registerBuildInfoGauge() { // --- Build info gauge --- - buildInfoGauge_ = meter_->CreateInt64ObservableGauge("build_info", "Build version information"); + buildInfoGauge_ = + core_.meter()->CreateInt64ObservableGauge("build_info", "Build version information"); buildInfoGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* /* state */) { try @@ -1157,14 +693,14 @@ MetricsRegistry::registerBuildInfoGauge() } void -MetricsRegistry::registerCompleteLedgersGauge() +AppMetricGauges::registerCompleteLedgersGauge() { // --- Complete ledgers range gauge --- - completeLedgersGauge_ = meter_->CreateInt64ObservableGauge( + completeLedgersGauge_ = core_.meter()->CreateInt64ObservableGauge( "complete_ledgers", "Complete ledger range start/end pairs"); completeLedgersGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1214,14 +750,14 @@ MetricsRegistry::registerCompleteLedgersGauge() } void -MetricsRegistry::registerDbMetricsGauge() +AppMetricGauges::registerDbMetricsGauge() { // --- Database size and fetch rate gauges --- - dbMetricsGauge_ = - meter_->CreateInt64ObservableGauge("db_metrics", "Database storage sizes and fetch rates"); + dbMetricsGauge_ = core_.meter()->CreateInt64ObservableGauge( + "db_metrics", "Database storage sizes and fetch rates"); dbMetricsGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1253,14 +789,14 @@ MetricsRegistry::registerDbMetricsGauge() } void -MetricsRegistry::registerValidatorHealthGauge() +AppMetricGauges::registerValidatorHealthGauge() { // --- Validator health gauges --- - validatorHealthGauge_ = - meter_->CreateDoubleObservableGauge("validator_health", "Validator health indicators"); + validatorHealthGauge_ = core_.meter()->CreateDoubleObservableGauge( + "validator_health", "Validator health indicators"); validatorHealthGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1300,16 +836,16 @@ MetricsRegistry::registerValidatorHealthGauge() } void -MetricsRegistry::registerPeerQualityGauge() +AppMetricGauges::registerPeerQualityGauge() { // --- Peer quality gauges --- // Uses Peer::json() to read latency and version since those accessors // are not on the abstract Peer interface (they live on PeerImp). peerQualityGauge_ = - meter_->CreateDoubleObservableGauge("peer_quality", "Peer network quality metrics"); + core_.meter()->CreateDoubleObservableGauge("peer_quality", "Peer network quality metrics"); peerQualityGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1403,18 +939,18 @@ MetricsRegistry::registerPeerQualityGauge() } void -MetricsRegistry::registerReduceRelayGauge() +AppMetricGauges::registerReduceRelayGauge() { // Transaction reduce-relay efficiency. Overlay::txMetrics() exposes the // rolling averages as a JSON object with string values (std::to_string), // so parse each field. A high suppressed:selected ratio proves the // feature is saving bandwidth; a high not_enabled count means stale peers // force full relay. - reduceRelayGauge_ = meter_->CreateInt64ObservableGauge( + reduceRelayGauge_ = core_.meter()->CreateInt64ObservableGauge( "reduce_relay_metrics", "Transaction reduce-relay efficiency metrics"); reduceRelayGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1454,14 +990,14 @@ MetricsRegistry::registerReduceRelayGauge() } void -MetricsRegistry::registerLedgerEconomyGauge() +AppMetricGauges::registerLedgerEconomyGauge() { // --- Ledger economy gauges --- - ledgerEconomyGauge_ = - meter_->CreateDoubleObservableGauge("ledger_economy", "Ledger fee and economy metrics"); + ledgerEconomyGauge_ = core_.meter()->CreateDoubleObservableGauge( + "ledger_economy", "Ledger fee and economy metrics"); ledgerEconomyGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1519,14 +1055,14 @@ MetricsRegistry::registerLedgerEconomyGauge() } void -MetricsRegistry::registerStateTrackingGauge() +AppMetricGauges::registerStateTrackingGauge() { // --- State tracking gauges --- - stateTrackingGauge_ = - meter_->CreateDoubleObservableGauge("state_tracking", "Node state and mode tracking"); + stateTrackingGauge_ = core_.meter()->CreateDoubleObservableGauge( + "state_tracking", "Node state and mode tracking"); stateTrackingGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1573,7 +1109,7 @@ MetricsRegistry::registerStateTrackingGauge() } void -MetricsRegistry::registerStorageDetailGauge() +AppMetricGauges::registerStorageDetailGauge() { // --- Storage detail gauges --- // Reports the cumulative payload bytes handed to the NodeStore. See the @@ -1581,10 +1117,10 @@ MetricsRegistry::registerStorageDetailGauge() // on-disk file size, because no accessor for the latter exists. The label // value names it that way so it is not read as a filesystem measurement. storageDetailGauge_ = - meter_->CreateInt64ObservableGauge("storage_detail", "Storage detail metrics"); + core_.meter()->CreateInt64ObservableGauge("storage_detail", "Storage detail metrics"); storageDetailGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; auto& app = self->app_; @@ -1625,7 +1161,7 @@ MetricsRegistry::registerStorageDetailGauge() } void -MetricsRegistry::registerValidationAgreementGauge() +AppMetricGauges::registerValidationAgreementGauge() { // --- Validation agreement gauges --- // Reports rolling-window agreement percentages and counts from @@ -1633,18 +1169,18 @@ MetricsRegistry::registerValidationAgreementGauge() // callback so that pending ledger events are resolved before the // window data is read (the callback fires every ~10 s from the // PeriodicExportingMetricReader thread). - validationAgreementGauge_ = meter_->CreateDoubleObservableGauge( + validationAgreementGauge_ = core_.meter()->CreateDoubleObservableGauge( "validation_agreement", "Validation agreement percentages and counts (1h/24h windows)"); validationAgreementGauge_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; try { // Reconcile pending events before reading window data. - self->validationTracker_.reconcile(); + self->core_.getValidationTracker().reconcile(); auto observe = [&](char const* name, double value) { opentelemetry::nostd::getObserve(value, {{"metric", name}}); }; - observe("agreement_pct_1h", self->validationTracker_.agreementPct1h()); - observe("agreement_pct_24h", self->validationTracker_.agreementPct24h()); + observe("agreement_pct_1h", self->core_.getValidationTracker().agreementPct1h()); + observe("agreement_pct_24h", self->core_.getValidationTracker().agreementPct24h()); observe( - "agreements_1h", static_cast(self->validationTracker_.agreements1h())); - observe("missed_1h", static_cast(self->validationTracker_.missed1h())); + "agreements_1h", + static_cast(self->core_.getValidationTracker().agreements1h())); + observe( + "missed_1h", + static_cast(self->core_.getValidationTracker().missed1h())); observe( "agreements_24h", - static_cast(self->validationTracker_.agreements24h())); - observe("missed_24h", static_cast(self->validationTracker_.missed24h())); + static_cast(self->core_.getValidationTracker().agreements24h())); + observe( + "missed_24h", + static_cast(self->core_.getValidationTracker().missed24h())); // 7-day window (matches external xrpl-validator-dashboard). - observe("agreement_pct_7d", self->validationTracker_.agreementPct7d()); + observe("agreement_pct_7d", self->core_.getValidationTracker().agreementPct7d()); observe( - "agreements_7d", static_cast(self->validationTracker_.agreements7d())); - observe("missed_7d", static_cast(self->validationTracker_.missed7d())); + "agreements_7d", + static_cast(self->core_.getValidationTracker().agreements7d())); + observe( + "missed_7d", + static_cast(self->core_.getValidationTracker().missed7d())); } catch (...) // NOLINT(bugprone-empty-catch) { @@ -1677,12 +1221,12 @@ MetricsRegistry::registerValidationAgreementGauge() } void -MetricsRegistry::registerValidationTotalsCounters() +AppMetricGauges::registerValidationTotalsCounters() { // Lifetime validation agreement/miss counters. // - // These are monotonic ObservableCounters (not the sync Counters they used - // to be): a Prometheus _total must never decrease, but ValidationTracker's + // These are monotonic ObservableCounters rather than synchronous Counters: + // a Prometheus _total must never decrease, but ValidationTracker's // NET totals are non-monotonic (a late repair decrements the net miss // count). We therefore observe the tracker's GROSS lifetime tallies, which // count each ledger once at first classification and are never adjusted on @@ -1692,20 +1236,22 @@ MetricsRegistry::registerValidationTotalsCounters() // reconcile() is called first so pending events are resolved before the // tallies are read; the callback fires every ~10 s from the // PeriodicExportingMetricReader thread. - validationAgreementsObservable_ = meter_->CreateInt64ObservableCounter( + validationAgreementsObservable_ = core_.meter()->CreateInt64ObservableCounter( "validation_agreements_total", "Lifetime validations that initially agreed with network consensus"); validationAgreementsObservable_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; try { - self->validationTracker_.reconcile(); + self->core_.getValidationTracker().reconcile(); opentelemetry::nostd::get>>(result) - ->Observe(static_cast(self->validationTracker_.totalAgreementsEver())); + ->Observe( + static_cast( + self->core_.getValidationTracker().totalAgreementsEver())); } catch (...) // NOLINT(bugprone-empty-catch) { @@ -1714,19 +1260,20 @@ MetricsRegistry::registerValidationTotalsCounters() }, this); - validationMissedObservable_ = meter_->CreateInt64ObservableCounter( + validationMissedObservable_ = core_.meter()->CreateInt64ObservableCounter( "validation_missed_total", "Lifetime validations that initially missed network consensus"); validationMissedObservable_->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); + auto* self = static_cast(state); if (self->callbacksDetached_.load(std::memory_order_acquire)) return; try { - self->validationTracker_.reconcile(); + self->core_.getValidationTracker().reconcile(); opentelemetry::nostd::get>>(result) - ->Observe(static_cast(self->validationTracker_.totalMissedEver())); + ->Observe( + static_cast(self->core_.getValidationTracker().totalMissedEver())); } catch (...) // NOLINT(bugprone-empty-catch) { @@ -1738,71 +1285,4 @@ MetricsRegistry::registerValidationTotalsCounters() #endif // XRPL_ENABLE_TELEMETRY -// ----------------------------------------------------------------- -// External dashboard parity counter increments -// ----------------------------------------------------------------- - -void -MetricsRegistry::incrementLedgersClosed() -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (recording() && ledgersClosedCounter_) - ledgersClosedCounter_->Add(1); -#endif -} - -void -MetricsRegistry::incrementValidationsSent() -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (recording() && validationsSentCounter_) - validationsSentCounter_->Add(1); -#endif -} - -void -MetricsRegistry::incrementValidationsChecked() -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (recording() && validationsCheckedCounter_) - validationsCheckedCounter_->Add(1); -#endif -} - -void -MetricsRegistry::incrementStateChanges() -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (recording() && stateChangesCounter_) - stateChangesCounter_->Add(1); -#endif -} - -void -MetricsRegistry::incrementLedgerHistoryMismatch(std::string_view reason) -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (recording() && ledgerHistoryMismatchCounter_) - ledgerHistoryMismatchCounter_->Add(1, {{"reason", std::string(reason)}}); -#endif -} - -void -MetricsRegistry::incrementTxqExpired() -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (recording() && txqExpiredCounter_) - txqExpiredCounter_->Add(1); -#endif -} - -void -MetricsRegistry::incrementTxqDropped(std::string_view reason) -{ -#ifdef XRPL_ENABLE_TELEMETRY - if (recording() && txqDroppedCounter_) - txqDroppedCounter_->Add(1, {{"reason", std::string(reason)}}); -#endif -} - } // namespace xrpl::telemetry diff --git a/src/xrpld/telemetry/AppMetricGauges.h b/src/xrpld/telemetry/AppMetricGauges.h new file mode 100644 index 0000000000..15eb67395a --- /dev/null +++ b/src/xrpld/telemetry/AppMetricGauges.h @@ -0,0 +1,473 @@ +#pragma once + +/** + * Observable-gauge layer for xrpld — the pull-model half of the OTel metric + * surface. + * + * Declares the class that registers every observable instrument whose callback + * samples live server state, and that owns the handles those registrations + * return. The export pipeline itself — provider, meter, exporter and the + * synchronous counters and histograms — belongs to MetricsRegistry, the + * sibling class in this namespace. This layer borrows that meter and adds the + * pull-model instruments on top of it. + */ + +// Unguarded because the constructor names beast::Journal and MetricsRegistry in +// both configurations. beast::Journal is taken by value, so it needs a complete +// type even when telemetry is off. +#include +#include + +#ifdef XRPL_ENABLE_TELEMETRY +// Guarded like the members that use them: std::atomic by callbacksDetached_, +// std::function and std::int64_t by the ObserveFn sink, and the two OTel +// headers by the 19 instrument handles. +#include +#include + +#include +#include +#include +#endif + +namespace xrpl { + +class ServiceRegistry; + +// Defined in src/xrpld/app/ledger/AcquireStats.h. Forward-declared because +// only one helper signature names it, and pulling an xrpld/app header in here +// would widen the dependencies of every file that includes this one. +class AcquireStats; + +namespace node_store { +class Database; +} // namespace node_store + +} // namespace xrpl + +namespace xrpl::telemetry { + +/** + * Registers and owns the pull-model OTel instruments that sample live server + * state. + * + * Each registered callback runs on the OTel reader thread and reads services + * through the ServiceRegistry reference given at construction. Both the core + * registry and the ServiceRegistry are borrowed, so both must outlive this + * object. + * + * Collaborator diagram (ASCII): + * + * AppMetricGauges + * +-- MetricsRegistry (borrowed) + * | +-- meter() -- creates all 19 observable instruments + * | +-- getValidationTracker() -- read by the agreement instruments + * | +-- OTel MeterProvider + * | +-- PeriodicExportingMetricReader (~10 s tick, drives the callbacks) + * +-- ServiceRegistry (borrowed) -- every value the callbacks sample + * +-- 19 ObservableInstrument handles (owned) + * + * Callback flow, once startAsyncGauges() has run: + * + * Reader thread tick (~10 s) + * v + * SDK invokes each callback, passing this object as the state pointer + * v + * callbacksDetached_ true? -- yes --> return without observing anything + * v no + * read current values from the ServiceRegistry, Observe() each one + * v + * the core's pipeline exports them over OTLP/HTTP + * + * One instrument per metric domain: cache hit rates and sizes, TxQ state, + * CountedObject instances, load-factor breakdown, NodeStore I/O and + * acquisition stalls, server info, build version, complete ledger ranges, + * database sizes, validator health, peer quality, reduce-relay efficiency, + * ledger economy, state tracking, storage detail and validation agreement. + * Most multiplex their values through a `metric` label, so a new value needs + * no new instrument; object counts use `type`, build info uses `version`, and + * complete ledgers uses `bound` and `index`. Sixteen are ObservableGauges and + * three are ObservableCounters, the latter where the value read is already + * cumulative and must never decrease. + * + * Teardown order is a caller contract, in this order: detachCallbacks(), then + * MetricsRegistry::stop(), then destroy this object. stop() joins the reader + * thread, so once it returns no callback can run again and destruction is + * safe. + * + * @code + * // Primary use. Construct after the core registry, and arm only once every + * // service the callbacks read exists. The overlay is built last, so it + * // fixes where this call can go. + * gauges_ = std::make_unique( + * *metricsRegistry_, *this, logs_->journal("MetricsRegistry")); + * gauges_->startAsyncGauges(); + * + * // Shutdown, in the required order. + * gauges_->detachCallbacks(); + * metricsRegistry_->stop(); + * + * // Edge case: arming without a working pipeline. The core hands out a + * // no-op meter when the pipeline fails to build, so this logs a warning + * // and registers nothing rather than reporting a success it cannot keep. + * // A second startAsyncGauges() behaves the same way. + * gauges_->startAsyncGauges(); + * @endcode + * + * @note Thread safety: + * - The callbacks run on the OTel reader thread, concurrently with the + * writers of the state they read. Each reads only lock-protected or + * atomic state and wraps its body in a catch-all try block, so a + * transient failure never brings down the reader thread. + * - startAsyncGauges() and the destructor are NOT thread-safe with each + * other and belong on the single server lifecycle thread. armed_ is a + * plain bool because that call is its only reader and writer. + * - detachCallbacks() may be called from any thread. It is one release + * store to an atomic that every callback acquire-loads. + * + * @note Limitations: + * - Arms once per object. A second startAsyncGauges() logs a warning and + * registers nothing, so the instruments are never duplicated. + * - detachCallbacks() is one-way. Calling it before startAsyncGauges() + * leaves every instrument registered but permanently silent. + * - Destroying this object while the core is still exporting is unsafe. + * The SDK holds this address as its callback state, and the flag the + * destructor sets dies with the object. Only stop() on the core closes + * that window, which is why it comes first. + * - The instrument set is fixed at registration. A pull-model instrument + * cannot be created lazily, so a new metric domain needs a new helper + * and a new handle here. + */ +class AppMetricGauges +{ +public: + /** + * Bind the layer to the core registry and to the services its callbacks + * will sample. Registers nothing; startAsyncGauges() does that. + * + * @param core Registry owning the meter these instruments are created on, + * and the validation tracker two of them read. Must outlive this object. + * @param app Services the callbacks sample. Must outlive this object. + * @param journal Log output. + */ + AppMetricGauges(MetricsRegistry& core, ServiceRegistry& app, beast::Journal journal); + + /** + * Disarms the callbacks, then releases the instrument handles. + * + * @note This is a last resort, not the teardown path. See the class note + * on destruction order. + */ + ~AppMetricGauges(); + + /** + * Non-copyable, non-movable. The registered callbacks hold this object's + * address, so it cannot move. + */ + AppMetricGauges(AppMetricGauges const&) = delete; + AppMetricGauges& + operator=(AppMetricGauges const&) = delete; + + /** + * Create and arm every pull-model instrument — mostly ObservableGauges, + * plus the ObservableCounters whose source value is already cumulative. + * + * Registering an observable also arms the reader thread to invoke its + * callback on the next tick, so this is an ordering decision and not just + * tidiness: it cannot run before the services those callbacks read exist. + * + * Does nothing but log a warning when the core is disabled, when it has no + * real pipeline, when this object is already armed, or when the core has + * already stopped. + * + * @pre Every service the callbacks read is constructed. The full set, from + * the `app.get*()` calls in the registration helpers, is: Overlay, OPs + * (NetworkOPs), LedgerMaster, OpenLedger, TxQ, NodeStore, NodeFamily, + * Validators, AcceptedLedgerCache, CachedSLEs, AcquireStats, TimeKeeper, + * RelationalDatabase, InboundLedgers and FeeTrack. + * Overlay is built last, so it fixes this call's position: + * `ServiceRegistry::getOverlay()` `XRPL_ASSERT`s that `overlay_` is + * non-null, and a reader-thread tick before the overlay exists aborts a + * Debug build. The callbacks' catch-all try block does not catch an + * assert. `getTxQ()` and `getRelationalDatabase()` assert likewise. + */ + void + startAsyncGauges(); + + /** + * Disarm every registered callback so it no-ops on the next reader-thread + * tick. + * + * Must be called BEFORE any service the callbacks read (nodeStore, + * overlay, networkOPs, ledgerMaster and the rest) is stopped. The flag is + * checked with acquire ordering at the top of every callback; together + * with the release store here that guarantees no callback starting after + * this returns will dereference an already-stopped service. + * + * Idempotent: the flag is one-way, only ever set to true, and nothing + * clears it. + * + * @note One-way means this is a shutdown-only call. Calling it before + * startAsyncGauges() does not "have no effect" — it permanently disarms + * every instrument that call registers, so they exist but never observe a + * value. + */ + void + detachCallbacks() noexcept; + +#ifdef XRPL_ENABLE_TELEMETRY + /** + * Sink handed to the nodestore_state helpers below. + * + * Every value they publish multiplexes onto the single `nodestore_state` + * instrument through its `metric` label, so the helpers need no access to + * the OTel observer result -- just somewhere to put a name and a number. + */ + using ObserveFn = std::function; + + // The four helpers below are public because each is a pure transform from + // a statistics object to a set of name-value pairs. They read only their + // arguments and need no AppMetricGauges instance, so a test can drive one + // directly with a recording sink and assert the exact `metric` label + // values it publishes. Exposing them widens no state. + + /** + * Observe the NodeStore I/O totals and the means derived from them. + * + * @param db NodeStore to read the counters from. + * @param observe Sink for one `metric`-labelled value. + */ + static void + observeNodeStoreTotals(node_store::Database& db, ObserveFn const& observe); + + /** + * Observe the backend write-path detail, when the backend measures it. + * + * Publishes nothing for a backend whose getWriteStats() is std::nullopt, + * which is every backend except NuDB. Absent labels let a reader tell + * "not measured" from "measured, and idle"; zeros would read as a + * perfectly idle write path. + * + * @param db NodeStore whose writable backend is sampled. + * @param observe Sink for one `metric`-labelled value. + */ + static void + observeWritePathDetail(node_store::Database const& db, ObserveFn const& observe); + + /** + * Observe the ledger-acquisition progress and stall counters. + * + * @param stats Process-wide acquisition counters. + * @param observe Sink for one `metric`-labelled value. + */ + static void + observeAcquireStats(AcquireStats const& stats, ObserveFn const& observe); + + /** + * Observe the read queue depth and the read thread-pool counts. + * + * These four have no accessor on Database, so its JSON counters object + * is still the only way to reach them. + * + * @param db NodeStore to read the JSON counters from. + * @param observe Sink for one `metric`-labelled value. + */ + static void + observeReadQueue(node_store::Database& db, ObserveFn const& observe); + +private: + /** + * Registry owning the meter these instruments are created on, the + * pipeline they export through, and the validation tracker two of them + * read. Borrowed; it outlives this object. + */ + MetricsRegistry& core_; + + /** + * Services the callbacks sample. Borrowed; it outlives this object. + */ + ServiceRegistry& app_; + + /** + * Log output. Shares the `MetricsRegistry` journal partition, so one + * log-level setting covers the whole metric pipeline. + */ + beast::Journal const journal_; + + /** + * True once startAsyncGauges() has registered the instruments. Read and + * written only by that call, from the server lifecycle thread, so it needs + * no atomic. + */ + bool armed_{false}; + + /** + * Set by detachCallbacks() during shutdown so every callback returns early + * before reading services that may already be stopped. Checked with + * memory_order_acquire at the top of each callback to pair with the + * memory_order_release store in detachCallbacks(). + */ + std::atomic callbacksDetached_{false}; + + // --- Observable instrument handles --- + // Held so the callbacks stay registered for as long as this object lives. + /** + * Cache hit rates and sizes. + */ + opentelemetry::nostd::shared_ptr + cacheHitRateGauge_; + /** + * Transaction queue state. + */ + opentelemetry::nostd::shared_ptr txqGauge_; + /** + * Live instance counts for every CountedObject type. + */ + opentelemetry::nostd::shared_ptr + objectCountGauge_; + /** + * Fee load-factor breakdown. + */ + opentelemetry::nostd::shared_ptr loadFactorGauge_; + /** + * Every NodeStore value on one instrument, separated by its `metric` + * label: I/O totals, the read and write means derived from them, the NuDB + * write-queue detail, and the ledger-acquisition stall counters. + */ + opentelemetry::nostd::shared_ptr nodeStoreGauge_; + /** + * Server-level health: operating mode, uptime, peers, ledger sequences and + * the last consensus round. + */ + opentelemetry::nostd::shared_ptr serverInfoGauge_; + /** + * Build version, carried as a label with a constant value of 1. + */ + opentelemetry::nostd::shared_ptr buildInfoGauge_; + /** + * Complete ledger range start/end pairs. + */ + opentelemetry::nostd::shared_ptr + completeLedgersGauge_; + /** + * Database sizes and the historical fetch rate. + */ + opentelemetry::nostd::shared_ptr dbMetricsGauge_; + + // --- External dashboard parity instruments --- + /** + * Validator health: amendment blocked, UNL blocked, quorum, UNL expiry. + */ + opentelemetry::nostd::shared_ptr + validatorHealthGauge_; + /** + * Peer network quality: P90 latency, diverged peer count, version spread + * and the upgrade recommendation derived from it. + */ + opentelemetry::nostd::shared_ptr + peerQualityGauge_; + /** + * Transaction reduce-relay efficiency: selected against suppressed peers, + * feature-disabled peers, missing-tx frequency. + */ + opentelemetry::nostd::shared_ptr + reduceRelayGauge_; + /** + * Ledger economy: base fee, reserves, ledger age and transaction rate. + */ + opentelemetry::nostd::shared_ptr + ledgerEconomyGauge_; + /** + * Node state tracking: operating mode as a number, and time in that mode. + */ + opentelemetry::nostd::shared_ptr + stateTrackingGauge_; + /** + * Storage detail: the cumulative payload bytes handed to the NodeStore. + * Logical bytes stored, not on-disk file size. + */ + opentelemetry::nostd::shared_ptr + storageDetailGauge_; + /** + * Validation agreement percentages and counts over the 1h, 24h and 7d + * windows kept by ValidationTracker. + */ + opentelemetry::nostd::shared_ptr + validationAgreementGauge_; + /** + * ObservableCounter: jq_trans_overflow_total — observed from + * Overlay::getJqTransOverflow() (cumulative overflow tally owned by the + * overlay). + */ + opentelemetry::nostd::shared_ptr + jqTransOverflowObservable_; + /** + * ObservableCounter: validation_agreements_total — observed from + * ValidationTracker::totalAgreementsEver() (monotonic gross lifetime + * tally, initial-classification semantics). + */ + opentelemetry::nostd::shared_ptr + validationAgreementsObservable_; + /** + * ObservableCounter: validation_missed_total — observed from + * ValidationTracker::totalMissedEver() (monotonic gross lifetime tally, + * initial-classification semantics). + */ + opentelemetry::nostd::shared_ptr + validationMissedObservable_; + + /** + * Create and arm every instrument, one helper per metric domain so that + * each helper stays well under the 80-line-per-function limit. + * + * Called only from startAsyncGauges(), which owns the enable, arm-once, + * pipeline and service-readiness guards. + */ + void + registerAsyncGauges(); + + // Per-domain registration helpers. Each creates its instrument -- an + // ObservableGauge, or an ObservableCounter where the underlying value is + // cumulative -- and attaches a single callback that reads current values + // from the ServiceRegistry. The callbacks run on the OTel + // PeriodicExportingMetricReader background thread (~10 s tick). + void + registerJqTransOverflowCounter(); // gap-fill: overlay overflow total + void + registerCacheHitRateGauge(); + void + registerTxqGauge(); + void + registerObjectCountGauge(); + void + registerLoadFactorGauge(); + void + registerNodeStoreGauge(); + void + registerServerInfoGauge(); + void + registerBuildInfoGauge(); + void + registerCompleteLedgersGauge(); + void + registerDbMetricsGauge(); + void + registerValidatorHealthGauge(); + void + registerPeerQualityGauge(); + void + registerReduceRelayGauge(); // Reduce-relay efficiency + void + registerLedgerEconomyGauge(); + void + registerStateTrackingGauge(); + void + registerStorageDetailGauge(); + void + registerValidationAgreementGauge(); + void + registerValidationTotalsCounters(); // gap-fill: lifetime agree/miss _total +#endif // XRPL_ENABLE_TELEMETRY +}; + +} // namespace xrpl::telemetry