diff --git a/OpenTelemetryPlan/06-implementation-phases.md b/OpenTelemetryPlan/06-implementation-phases.md index 536f906a47..ef311643f7 100644 --- a/OpenTelemetryPlan/06-implementation-phases.md +++ b/OpenTelemetryPlan/06-implementation-phases.md @@ -1680,18 +1680,22 @@ xrpld's `OperatingMode` enum maps 0-4 (DISCONNECTED through FULL). The external **Task 7.13: Storage Detail Observable Gauge** -| Gauge Name | Label `metric=` | Type | Source | -| ---------------------- | --------------- | ----- | ---------------------------------------------------- | -| `xrpld_storage_detail` | `nudb_bytes` | int64 | `Database::getStoreSize()` — cumulative object bytes | +| Gauge Name | Label `metric=` | Type | Source | +| ---------------------- | --------------------- | ----- | ---------------------------------------------------- | +| `xrpld_storage_detail` | `stored_object_bytes` | int64 | `Database::getStoreSize()` — cumulative object bytes | -Despite the name, this is not a filesystem measurement. `getStoreSize()` sums the -object payloads this process has written, so it excludes NuDB's keys, bucket padding -and log, and it resets with the process while the files on disk do not. It is the -same accessor `node_written_bytes` uses, so the two series are equal by construction -and any write-amplification ratio built from the pair is a constant 1.0. There is no +This is not a filesystem measurement. `getStoreSize()` sums the object payloads this +process has written, so it excludes NuDB's keys, bucket padding and log, and it +resets with the process while the files on disk do not. It is the same accessor +`node_written_bytes` uses, so the two series are equal by construction and any +write-amplification ratio built from the pair is a constant 1.0. There is no file-size accessor on `Backend` or `Database`, so no metric reports the store's on-disk size today. +The label value was `nudb_bytes` through Phase 8 and was renamed in Phase 9: the +value is read from `Database`, not from the NuDB backend, so a backend prefix +misdescribed it and the old name implied an on-disk size it never reported. + **File**: `src/xrpld/telemetry/MetricsRegistry.cpp` **Exit Criteria**: @@ -1847,7 +1851,7 @@ Add checks to `validate_telemetry.py` for all new span attributes and metrics. | `xrpld_ledgers_closed_total` | | `xrpld_validations_sent_total` | | `xrpld_state_changes_total` | -| `xrpld_storage_detail{metric="nudb_bytes"}` | +| `xrpld_storage_detail{metric="stored_object_bytes"}` | **New dashboard load checks (~3)**: diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md index b24e4a5a82..3a05e4cf35 100644 --- a/OpenTelemetryPlan/09-data-collection-reference.md +++ b/OpenTelemetryPlan/09-data-collection-reference.md @@ -1565,18 +1565,23 @@ State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full #### Storage Detail (Observable Gauge — `storage_detail`) -| Prometheus Metric | Type | Labels | Description | -| ------------------------------------- | ----- | -------- | ---------------------------------------------------------- | -| `storage_detail{metric="nudb_bytes"}` | Int64 | `metric` | Cumulative object-payload bytes written (not on-disk size) | +| Prometheus Metric | Type | Labels | Description | +| ---------------------------------------------- | ----- | -------- | ---------------------------------------------------------- | +| `storage_detail{metric="stored_object_bytes"}` | Int64 | `metric` | Cumulative object-payload bytes written (not on-disk size) | -> **`nudb_bytes` is not a file size, despite the name.** It observes -> `getStoreSize()` (`src/xrpld/telemetry/MetricsRegistry.cpp:1507`), which sums the -> object payloads this process has written. It therefore excludes NuDB's keys, -> bucket padding and log, and it resets when the process restarts while the files -> on disk do not. `node_written_bytes` on the `nodestore_state` gauge calls the same -> accessor (`MetricsRegistry.cpp:836`), so the two series are equal by construction -> and any write-amplification ratio built from the pair is a constant 1.0. To size -> the store on disk, stat the backend's files; no metric reports it today. +> **`stored_object_bytes` is not a file size.** It observes `getStoreSize()` +> (`src/xrpld/telemetry/MetricsRegistry.cpp:1511`), which sums the object payloads +> this process has written. It therefore excludes NuDB's keys, bucket padding and +> log, and it resets when the process restarts while the files on disk do not. +> `node_written_bytes` on the `nodestore_state` gauge calls the same accessor +> (`MetricsRegistry.cpp:836`), so the two series are equal by construction and any +> write-amplification ratio built from the pair is a constant 1.0. To size the store +> on disk, stat the backend's files; no metric reports it today. +> +> This label value was called `nudb_bytes` before Phase 9. The value comes from +> `node_store::Database`, not from the NuDB backend, so it reads the same on +> RocksDB and carries no backend prefix. Queries and dashboards pinned to the old +> name return no data. #### Synchronous Counters (Phase 7+) diff --git a/OpenTelemetryPlan/Phase7_taskList.md b/OpenTelemetryPlan/Phase7_taskList.md index d7ce34b7b8..e22c4b37c9 100644 --- a/OpenTelemetryPlan/Phase7_taskList.md +++ b/OpenTelemetryPlan/Phase7_taskList.md @@ -482,22 +482,25 @@ validatorParticipationGauge_ = meter_->CreateInt64ObservableGauge( > **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity) -**Objective**: Export NuDB-specific stored-bytes total and initial sync duration. +**Objective**: Export the NodeStore stored-bytes total and initial sync duration. **Gauge label values**: | Gauge Name | Label `metric=` | Type | Source | | ---------------------- | ------------------------------- | ------ | ---------------------------------------------------- | -| `xrpld_storage_detail` | `nudb_bytes` | int64 | `Database::getStoreSize()` — cumulative object bytes | +| `xrpld_storage_detail` | `stored_object_bytes` | int64 | `Database::getStoreSize()` — cumulative object bytes | | `xrpld_sync_info` | `initial_sync_duration_seconds` | double | Time from start to first FULL | -Despite the name, `nudb_bytes` is not a file size. `getStoreSize()` sums the object -payloads this process has written, so it excludes NuDB's keys, bucket padding and -log, and it resets when the process restarts while the files on disk do not. It is -the same accessor `node_written_bytes` uses, so the two series are equal by -construction and any write-amplification ratio built from the pair is a constant 1.0. -Neither `Backend` nor `Database` exposes a file-size accessor, so no metric reports -the store's on-disk size today. +`stored_object_bytes` is not a file size. `getStoreSize()` sums the object payloads +this process has written, so it excludes NuDB's keys, bucket padding and log, and it +resets when the process restarts while the files on disk do not. It is the same +accessor `node_written_bytes` uses, so the two series are equal by construction and +any write-amplification ratio built from the pair is a constant 1.0. Neither +`Backend` nor `Database` exposes a file-size accessor, so no metric reports the +store's on-disk size today. + +This label value was `nudb_bytes` when Phase 7 shipped it and was renamed in Phase 9, +because the value comes from `Database` rather than the NuDB backend. **Key modified files**: `src/xrpld/telemetry/MetricsRegistry.h/.cpp` diff --git a/docker/telemetry/grafana/dashboards/node-health.json b/docker/telemetry/grafana/dashboards/node-health.json index 61379e639e..327f222543 100644 --- a/docker/telemetry/grafana/dashboards/node-health.json +++ b/docker/telemetry/grafana/dashboards/node-health.json @@ -1347,8 +1347,8 @@ } }, { - "title": "NuDB Stored Bytes", - "description": "###### What this is:\n*Cumulative object-payload bytes this process has written to the NuDB object-store back end. Despite the metric name, this is not the size of the store on disk.*\n\n###### How it's computed:\n*Current value of the nudb_bytes gauge, plotted over time. It observes getStoreSize(), the same accessor node_written_bytes uses, so the two series are equal and their ratio is a constant 1.0 rather than a write-amplification measure.*\n\n###### Reading it:\n*A smoothly growing line is normal; the slope is the write rate. It excludes NuDB's keys, bucket padding and log, and it restarts from zero with the process while the files on disk do not.*\n\n###### Healthy range:\n*Gradual growth consistent with ledger data being stored.*\n\n###### Watch for:\n*A sudden jump in growth rate can indicate runaway storage or an unexpected back-fill. Do not use this panel to size the store on disk or to plan disk capacity; no metric reports on-disk size today, so check the filesystem directly.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerStorageDetailGauge`", + "title": "Stored Object Bytes", + "description": "###### What this is:\n*Cumulative object-payload bytes this process has written to the NodeStore object-store back end. This is not the size of the store on disk.*\n\n###### How it's computed:\n*Current value of the stored_object_bytes gauge, plotted over time. It observes getStoreSize(), the same accessor node_written_bytes uses, so the two series are equal and their ratio is a constant 1.0 rather than a write-amplification measure.*\n\n###### Reading it:\n*A smoothly growing line is normal; the slope is the write rate. It excludes NuDB's keys, bucket padding and log, and it restarts from zero with the process while the files on disk do not.*\n\n###### Healthy range:\n*Gradual growth consistent with ledger data being stored.*\n\n###### Watch for:\n*A sudden jump in growth rate can indicate runaway storage or an unexpected back-fill. Do not use this panel to size the store on disk or to plan disk capacity; no metric reports on-disk size today, so check the filesystem directly.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerStorageDetailGauge`", "type": "timeseries", "gridPos": { "h": 8, @@ -1368,7 +1368,7 @@ "datasource": { "type": "prometheus" }, - "expr": "label_replace(storage_detail{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"nudb_bytes\"}, \"series\", \"NuDB Stored Bytes\", \"\", \"\")" + "expr": "label_replace(storage_detail{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"stored_object_bytes\"}, \"series\", \"Stored Object Bytes\", \"\", \"\")" } ], "fieldConfig": { diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index 6bf79a81dc..a97fda2a68 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -1803,11 +1803,13 @@ Two more pairs from the same family: quantity, the larger of the recorded load and the pending batch size (`src/libxrpl/nodestore/BatchWriter.cpp:47-53`), so it is a batch-queue length rather than a thread count. -- **`nudb_bytes` is not the size of the store on disk**, despite the name. It reports - the cumulative object-payload bytes this process has written — the same value as +- **`stored_object_bytes` is not the size of the store on disk.** It reports the + cumulative object-payload bytes this process has written — the same value as `node_written_bytes`, from the same accessor — so it excludes keys, padding and the log, and it resets with the process. A ratio of the two is a constant 1.0 and - measures nothing. + measures nothing. This label value was called `nudb_bytes` before Phase 9; it + comes from `node_store::Database` rather than the NuDB backend, so it is not part + of the `nudb_*` family above and reads the same on RocksDB. - These gauges are sampled on the `MetricsRegistry` reader's 10 s cadence, while the `jobq_*` lane gauges are sampled at 1 s by a different provider. Widen the window when correlating them rather than reading a single scrape; see the caveat diff --git a/src/xrpld/telemetry/MetricsRegistry.cpp b/src/xrpld/telemetry/MetricsRegistry.cpp index fbdd6edcbd..bef7ec8f2d 100644 --- a/src/xrpld/telemetry/MetricsRegistry.cpp +++ b/src/xrpld/telemetry/MetricsRegistry.cpp @@ -1472,7 +1472,8 @@ MetricsRegistry::registerStorageDetailGauge() // --- Task 7.13: Storage detail gauges --- // Reports the cumulative payload bytes handed to the NodeStore. See the // note at the observe() call below: this is logical bytes stored, not - // on-disk file size, because no accessor for the latter exists. + // on-disk file size, because no accessor for the latter exists. The label + // value names it that way so it is not read as a filesystem measurement. storageDetailGauge_ = meter_->CreateInt64ObservableGauge("storage_detail", "Storage detail metrics"); storageDetailGauge_->AddCallback( @@ -1490,11 +1491,13 @@ MetricsRegistry::registerStorageDetailGauge() ->Observe(value, {{"metric", name}}); }; - // Cumulative payload bytes handed to the NodeStore -- NOT - // on-disk file size, despite the name. getStoreSize() sums - // the object payloads this process has written, so it - // excludes NuDB's keys, bucket padding and log, and it - // resets with the process while the files do not. + // Cumulative payload bytes handed to the NodeStore. This is + // not an on-disk file size: getStoreSize() sums the object + // payloads this process has written, so it excludes NuDB's + // keys, bucket padding and log, and it resets with the + // process while the files do not. The value comes from + // Database, not from any backend, so it carries no nudb_ + // prefix -- it reads the same on RocksDB. // // This is the same call node_written_bytes makes on the // nodestore_state gauge, so the two series are equal by @@ -1504,7 +1507,8 @@ MetricsRegistry::registerStorageDetailGauge() // computing one would mean stat()ing the backend's files // from the reader thread, which needs a new Backend method // rather than a change at this call site. - observe("nudb_bytes", static_cast(app.getNodeStore().getStoreSize())); + observe( + "stored_object_bytes", static_cast(app.getNodeStore().getStoreSize())); } catch (...) // NOLINT(bugprone-empty-catch) {