ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-07-30 02:20:39 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(telemetry): correct nudb_bytes and NuDB found-ratio descriptions

Browse Source 
nudb_bytes was documented as a NuDB file size, one place even claiming a
filesystem stat. It observes Database::getStoreSize(), which sums the object
payloads this process has written. It excludes NuDB's keys, bucket padding and
log, and resets with the process. node_written_bytes calls the same accessor, so
the two series are equal by construction and a write-amplification ratio built
from them is a constant 1.0. Neither Backend nor Database exposes a file-size
accessor, so nothing reports on-disk size today.

The Ledger Data & Sync panel plotting node_reads_hit / node_reads_total was
titled "NuDB Cache Hit Ratio" and described as reads served from cache.
fetchHitCount_ increments whenever a fetch returned an object, whatever served
it, so the ratio is a found rate. It reads near 100% while every fetch goes to
disk, which made the cold-read failure mode look impossible. Renamed to
"NuDB Read Found Ratio" and rewrote the guidance to pair it with read latency.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pratik Mankawde
2026-07-28 15:20:02 +01:00
parent b9f1c19851
commit 7da5ac5992
3 changed files with 31 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
OpenTelemetryPlan/06-implementation-phases.md
View File

@@ -1289,15 +1289,23 @@ xrpld's `OperatingMode` enum maps 0-4 (DISCONNECTED through FULL). The external
**Task 7.13: Storage Detail Observable Gauge**
| Gauge Name | Label `metric=` | Type | Source |
| ---------------------- | --------------- | ----- | ---------------------------------------- |
| `xrpld_storage_detail` | `nudb_bytes` | int64 | NuDB backend file size (filesystem stat) |
| Gauge Name | Label `metric=` | Type | Source |
| ---------------------- | --------------- | ----- | ---------------------------------------------------- |
| `xrpld_storage_detail` | `nudb_bytes` | int64 | `Database::getStoreSize()` — cumulative object bytes |
Despite the name, this is not a filesystem measurement. `getStoreSize()` sums the
object payloads this process has written, so it excludes NuDB's keys, bucket padding
and log, and it resets with the process while the files on disk do not. It is the
same accessor `node_written_bytes` uses, so the two series are equal by construction
and any write-amplification ratio built from the pair is a constant 1.0. There is no
file-size accessor on `Backend` or `Database`, so no metric reports the store's
on-disk size today.
**File**: `src/xrpld/telemetry/MetricsRegistry.cpp`
**Exit Criteria**:
- [ ] NuDB file size reported in bytes
- [ ] Cumulative stored object bytes reported
- [ ] Gracefully returns 0 if NuDB not configured
---

20
OpenTelemetryPlan/Phase7_taskList.md
View File

@@ -482,20 +482,28 @@ validatorParticipationGauge_ = meter_->CreateInt64ObservableGauge(
> **Source**: [External Dashboard Parity](./06-implementation-phases.md#appendix-external-dashboard-parity)
**Objective**: Export NuDB-specific storage size and initial sync duration.
**Objective**: Export NuDB-specific stored-bytes total and initial sync duration.
**Gauge label values**:
| Gauge Name | Label `metric=` | Type | Source |
| ---------------------- | ------------------------------- | ------ | ----------------------------- |
| `xrpld_storage_detail` | `nudb_bytes` | int64 | NuDB backend file size |
| `xrpld_sync_info` | `initial_sync_duration_seconds` | double | Time from start to first FULL |
| Gauge Name | Label `metric=` | Type | Source |
| ---------------------- | ------------------------------- | ------ | ---------------------------------------------------- |
| `xrpld_storage_detail` | `nudb_bytes` | int64 | `Database::getStoreSize()` — cumulative object bytes |
| `xrpld_sync_info` | `initial_sync_duration_seconds` | double | Time from start to first FULL |
Despite the name, `nudb_bytes` is not a file size. `getStoreSize()` sums the object
payloads this process has written, so it excludes NuDB's keys, bucket padding and
log, and it resets when the process restarts while the files on disk do not. It is
the same accessor `node_written_bytes` uses, so the two series are equal by
construction and any write-amplification ratio built from the pair is a constant 1.0.
Neither `Backend` nor `Database` exposes a file-size accessor, so no metric reports
the store's on-disk size today.
**Key modified files**: `src/xrpld/telemetry/MetricsRegistry.h/.cpp`
**Exit Criteria**:
- [ ] NuDB file size reported in bytes (0 if NuDB not configured)
- [ ] Cumulative stored object bytes reported (0 if NuDB not configured)
- [ ] Sync duration captured once and remains stable after reaching FULL
---

10
docker/telemetry/grafana/dashboards/ledger-data-sync.json
View File

@@ -1079,7 +1079,7 @@
},
{
"title": "NuDB Read Latency",
"description": "###### What this is:\n*Average nodestore (NuDB) read latency: how long each on-disk object read takes. The direct disk-layer cost that backs up the job queue.*\n\n###### How it's computed:\n*rate(nodestore_state{metric=\"node_reads_duration_us\"}[$__rate_interval]) / rate(nodestore_state{metric=\"node_reads_total\"}[$__rate_interval]), in us.*\n\n###### Reading it:\n*Low single-digit us when the OS page cache is warm; tens-to-hundreds of us when reads hit the disk. Rises sharply during cold-cache catch-up.*\n\n###### Healthy range:\n*< ~10us/read warm; higher is expected briefly after a wipe/restart.*\n\n###### Watch for:\n*Sustained high us/read together with low cache hit % (below) = disk-bound reads are the bottleneck; check EBS IOPS / io scheduler latency.*\n\n###### Source:\n[nodestore/backend/NuDBFactory.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/nodestore/backend/NuDBFactory.cpp)\n\n###### Function:\n`NuDB backend read path`",
"description": "###### What this is:\n*Average nodestore (NuDB) read latency: how long each on-disk object read takes. The direct disk-layer cost that backs up the job queue.*\n\n###### How it's computed:\n*rate(nodestore_state{metric=\"node_reads_duration_us\"}[$__rate_interval]) / rate(nodestore_state{metric=\"node_reads_total\"}[$__rate_interval]), in us.*\n\n###### Reading it:\n*Low single-digit us when the OS page cache is warm; tens-to-hundreds of us when reads hit the disk. Rises sharply during cold-cache catch-up.*\n\n###### Healthy range:\n*< ~10us/read warm; higher is expected briefly after a wipe/restart.*\n\n###### Watch for:\n*Sustained high us/read is the disk-bound signal on its own; the found ratio below stays near 100% even then, so do not wait for it to drop. Check EBS IOPS / io scheduler latency.*\n\n###### Source:\n[nodestore/backend/NuDBFactory.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/nodestore/backend/NuDBFactory.cpp)\n\n###### Function:\n`NuDB backend read path`",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -1162,8 +1162,8 @@
"id": 22
},
{
"title": "NuDB Cache Hit Ratio",
"description": "###### What this is:\n*Fraction of nodestore reads served from cache rather than disk. A cold cache (fresh NuDB after a wipe) forces almost every read to disk - the root cause of high read latency during catch-up.*\n\n###### How it's computed:\n*rate(nodestore_state{metric=\"node_reads_hit\"}[$__rate_interval]) / rate(nodestore_state{metric=\"node_reads_total\"}[$__rate_interval]).*\n\n###### Reading it:\n*Near 1.0 (100%) on a warm node. Near 0 right after a wipe/restart, climbing as the working set warms.*\n\n###### Healthy range:\n*> ~0.9 warm; near 0 briefly after a cold start.*\n\n###### Watch for:\n*Persistently low hit ratio on a node that should be warm = working set does not fit cache, or continuous re-acquisition churning the cache.*\n\n###### Source:\n[nodestore/backend/NuDBFactory.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/nodestore/backend/NuDBFactory.cpp)\n\n###### Function:\n`TaggedCache hit accounting`",
"title": "NuDB Read Found Ratio",
"description": "###### What this is:\n*Fraction of nodestore fetches that found the object they asked for. This is not a cache hit ratio: `node_reads_hit` counts every fetch that returned an object, whatever served it, so a fetch that went all the way to disk still counts here.*\n\n###### How it's computed:\n*rate(nodestore_state{metric=\"node_reads_hit\"}[$__rate_interval]) / rate(nodestore_state{metric=\"node_reads_total\"}[$__rate_interval]).*\n\n###### Reading it:\n*Normally sits near 1.0 (100%) on any node that has the data, warm or cold, because a synced node almost always finds what it asks for. It does not fall when the page cache goes cold.*\n\n###### Healthy range:\n*Near 1.0. A ratio well below 1.0 means fetches are missing, which points at a gap in local history rather than at cache pressure.*\n\n###### Watch for:\n*Never read this panel on its own. A ~100% found ratio at over 100 microseconds per read is the cold-read signature, not a healthy cache: the data is found every time and paid for every time. Always pair it with NuDB Read Latency.*\n\n###### Source:\n[nodestore/Database.cpp](https://github.com/XRPLF/rippled/blob/develop/src/libxrpl/nodestore/Database.cpp)\n\n###### Function:\n`Database::fetchNodeObject`",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -1184,7 +1184,7 @@
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_reads_hit\"}[$__rate_interval])) / clamp_min(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_reads_total\"}[$__rate_interval])), 1), \"series\", \"NuDB Hit Ratio\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")"
"expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_reads_hit\"}[$__rate_interval])) / clamp_min(sum by (service_instance_id, xrpl_branch, xrpl_node_role) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_reads_total\"}[$__rate_interval])), 1), \"series\", \"NuDB Found Ratio\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")"
}
],
"fieldConfig": {
@@ -1192,7 +1192,7 @@
"displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}",
"unit": "percentunit",
"custom": {
"axisLabel": "Hit Ratio",
"axisLabel": "Found Ratio",
"spanNulls": 1800000,
"insertNulls": false,
"showPoints": "auto",