ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-21 14:20:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-sync-diagnostics

Browse Source 
Brings in the phase-10 revert of the nodestore read-latency histogram plus the
nudb_bytes -> stored_object_bytes rename.

Conflicts in MetricsRegistry.{h,cpp} resolved keeping both intents:

- MetricsRegistry.cpp: dropped everything that existed only to serve the
  reverted nodestore_read_us histogram -- the addSubMillisecondHistogramView()
  helper, its call site, the kSubMillisecondBoundaries array and the
  NodeStoreMetricNames.h include. Kept every view this branch registers
  (consensus round duration, sweep_malloc_trim_us, dns_resolve_latency_ms,
  overlay_dial_latency_ms) and the shared addHistogramView() base helper.
  Took the rename at the storage_detail observe() call site.

- MetricsRegistry.h: took phase-10's move of the four nodestore_state observe
  helpers and their ObserveFn sink from private to public, while keeping this
  branch's enriched Doxygen on observeNodeStoreTotals().

Also corrected the registered-view count in the 09 reference doc: neither side's
arithmetic survives the merge, since this branch adds four views phase-10 never
saw and the revert removes one. Ten views are registered now, not six or seven.
This commit is contained in:
Pratik Mankawde
2026-07-28 16:53:13 +01:00
parent da58baef99 356e0af1fd
commit 0ddb4e2686
22 changed files with 1079 additions and 1359 deletions
Download Patch File Download Diff File Expand all files Collapse all files

372
docker/telemetry/grafana/dashboards/ledger-data-sync.json
View File

@@ -800,7 +800,7 @@
},
{
"title": "Sync State",
"description": "###### What this is:\n*Current server operating state as a numeric code: 0 disconnected, 1 connected, 2 syncing, 3 tracking, 4 full. A healthy synced node sits at 4.*\n\n###### How it's computed:\n*state_tracking{metric=\"state_value\"} (gauge). Companion time_in_current_state_seconds shows how long it has been stuck there.*\n\n###### Reading it:\n*Flat at 4 = full/healthy. Dropping to 1-2 and staying = the node fell out of sync and is re-acquiring (the primary red flag this row explains).*\n\n###### Healthy range:\n*4 (full), briefly 2-3 right after restart.*\n\n###### Watch for:\n*A node stuck below 4 for more than a few minutes, or oscillating - read the lower panels for the cause.*\n\n###### Source:\n[app/misc/NetworkOPs.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/misc/NetworkOPs.cpp)\n\n###### Function:\n`NetworkOPsImp::setMode`",
"description": "###### What this is:\n*Current server operating state as a numeric code: 0 disconnected, 1 connected, 2 syncing, 3 tracking, 4 full, 5 validating, 6 proposing. A healthy validator sits at 6, a healthy non-validating node at 4.*\n\n###### How it's computed:\n*state_tracking{metric=\"state_value\"} (gauge). Companion time_in_current_state_seconds shows how long it has been stuck there.*\n\n###### Reading it:\n*Flat at 4-6 = full/healthy. Dropping to 1-2 and staying = the node fell out of sync and is re-acquiring (the primary red flag this row explains).*\n\n###### Healthy range:\n*4-6 steady, briefly 2-3 right after restart.*\n\n###### Watch for:\n*A node stuck below 4 for more than a few minutes, or oscillating - read the lower panels for the cause.*\n\n###### Source:\n[app/misc/NetworkOPs.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/misc/NetworkOPs.cpp)\n\n###### Function:\n`NetworkOPsImp::setMode`",
"type": "timeseries",
"gridPos": {
"h": 8,
@@ -835,6 +835,185 @@
"showPoints": "auto",
"pointSize": 3
},
"mappings": [
{
"type": "value",
"options": {
"0": {
"text": "Disconnected",
"color": "red",
"index": 0
},
"1": {
"text": "Connected",
"color": "orange",
"index": 1
},
"2": {
"text": "Syncing",
"color": "yellow",
"index": 2
},
"3": {
"text": "Tracking",
"color": "blue",
"index": 3
},
"4": {
"text": "Full",
"color": "green",
"index": 4
},
"5": {
"text": "Validating",
"color": "super-light-green",
"index": 5
},
"6": {
"text": "Proposing",
"color": "dark-green",
"index": 6
}
}
}
],
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "red",
"value": null
},
{
"color": "orange",
"value": 1
},
{
"color": "yellow",
"value": 2
},
{
"color": "blue",
"value": 3
},
{
"color": "green",
"value": 4
},
{
"color": "super-light-green",
"value": 5
},
{
"color": "dark-green",
"value": 6
}
]
},
"color": {
"mode": "thresholds"
},
"min": 0,
"max": 6
},
"overrides": []
},
"id": 17
},
{
"title": "Validated Ledger Age",
"description": "###### What this is:\n*Seconds since this node last had a freshly validated ledger. The single clearest 'am I keeping up with the network' signal.*\n\n###### How it's computed:\n*ledgermaster_validated_ledger_age gauge (seconds), per node.*\n\n###### Reading it:\n*Should hover at the network close interval (~3-5s). A rising sawtooth or a high plateau means the node is falling behind or not validating.*\n\n###### Healthy range:\n*<= ~6s on a synced node.*\n\n###### Watch for:\n*Sustained climb above ~15s, or a monotonic ramp = the node is not keeping up; correlate with job-queue wait and NuDB read latency below.*\n\n###### Source:\n[app/ledger/LedgerMaster.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/LedgerMaster.cpp)\n\n###### Function:\n`LedgerMaster::getValidatedLedgerAge`",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 75
},
"options": {
"tooltip": {
"maxHeight": 600,
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "label_replace(label_join(label_replace(ledgermaster_validated_ledger_age{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}, \"series\", \"Validated Age\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "label_replace(label_join(label_replace(ledgermaster_published_ledger_age{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}, \"series\", \"Published Age\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")"
}
],
"fieldConfig": {
"defaults": {
"displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}",
"unit": "s",
"custom": {
"axisLabel": "Age (seconds)",
"spanNulls": 1800000,
"insertNulls": false,
"showPoints": "auto",
"pointSize": 3
}
},
"overrides": []
},
"id": 18
},
{
"title": "Time Spent Per State",
"description": "###### What this is:\n*Operating mode as a colour-coded timeline. Each band's width is the time spent in that state, so short-lived states show as thin slivers instead of vanishing.*\n\n###### How it's computed:\n*server_info{metric=\"server_state\"} (gauge), the raw OperatingMode 0-4. Uses server_state rather than state_value because state_value folds 5 and 6 onto FULL, which would split one Full band into three colours.*\n\n###### Reading it:\n*One green band across the window = healthy. Red/orange/yellow bands show when and for how long the node was degraded.*\n\n###### Healthy range:\n*Continuously green (Full), brief orange/yellow/blue only after a restart.*\n\n###### Watch for:\n*Repeated thin bands = the node is oscillating. This is sampled every 10s, so a state shorter than one sample can still be missed.*\n\n###### Source:\n[app/misc/NetworkOPs.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/misc/NetworkOPs.cpp)\n\n###### Function:\n`NetworkOPsImp::setMode`",
"type": "state-timeline",
"gridPos": {
"h": 6,
"w": 24,
"x": 0,
"y": 83
},
"options": {
"mergeValues": true,
"showValue": "auto",
"alignValue": "left",
"rowHeight": 0.9,
"tooltip": {
"mode": "single",
"sort": "none"
},
"legend": {
"displayMode": "list",
"placement": "bottom",
"showLegend": true
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "label_replace(label_join(server_info{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"server_state\"}, \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"displayName": "State ${__field.labels.xrpl_ident}",
"unit": "none",
"custom": {
"fillOpacity": 80,
"lineWidth": 0,
"insertNulls": false,
"spanNulls": false
},
"mappings": [
{
"type": "value",
@@ -900,56 +1079,7 @@
},
"overrides": []
},
"id": 17
},
{
"title": "Validated Ledger Age",
"description": "###### What this is:\n*Seconds since this node last had a freshly validated ledger. The single clearest 'am I keeping up with the network' signal.*\n\n###### How it's computed:\n*ledgermaster_validated_ledger_age gauge (seconds), per node.*\n\n###### Reading it:\n*Should hover at the network close interval (~3-5s). A rising sawtooth or a high plateau means the node is falling behind or not validating.*\n\n###### Healthy range:\n*<= ~6s on a synced node.*\n\n###### Watch for:\n*Sustained climb above ~15s, or a monotonic ramp = the node is not keeping up; correlate with job-queue wait and NuDB read latency below.*\n\n###### Source:\n[app/ledger/LedgerMaster.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/LedgerMaster.cpp)\n\n###### Function:\n`LedgerMaster::getValidatedLedgerAge`",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 12,
"x": 12,
"y": 75
},
"options": {
"tooltip": {
"maxHeight": 600,
"mode": "multi",
"sort": "desc"
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "label_replace(label_join(label_replace(ledgermaster_validated_ledger_age{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}, \"series\", \"Validated Age\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "label_replace(label_join(label_replace(ledgermaster_published_ledger_age{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}, \"series\", \"Published Age\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")"
}
],
"fieldConfig": {
"defaults": {
"displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}",
"unit": "s",
"custom": {
"axisLabel": "Age (seconds)",
"spanNulls": 1800000,
"insertNulls": false,
"showPoints": "auto",
"pointSize": 3
}
},
"overrides": []
},
"id": 18
"id": 27
},
{
"title": "Ledger Close Rate",
@@ -959,7 +1089,7 @@
"h": 8,
"w": 12,
"x": 0,
"y": 83
"y": 89
},
"options": {
"tooltip": {
@@ -1008,7 +1138,7 @@
"h": 8,
"w": 12,
"x": 12,
"y": 83
"y": 89
},
"options": {
"tooltip": {
@@ -1085,7 +1215,7 @@
"h": 8,
"w": 12,
"x": 0,
"y": 91
"y": 97
},
"options": {
"tooltip": {
@@ -1127,7 +1257,7 @@
"h": 8,
"w": 12,
"x": 12,
"y": 91
"y": 97
},
"options": {
"tooltip": {
@@ -1169,7 +1299,7 @@
"h": 8,
"w": 12,
"x": 0,
"y": 99
"y": 105
},
"options": {
"tooltip": {
@@ -1211,7 +1341,7 @@
"h": 8,
"w": 12,
"x": 12,
"y": 99
"y": 105
},
"options": {
"tooltip": {
@@ -1267,7 +1397,7 @@
"h": 8,
"w": 12,
"x": 0,
"y": 107
"y": 113
},
"options": {
"tooltip": {
@@ -1309,7 +1439,7 @@
"h": 8,
"w": 12,
"x": 12,
"y": 107
"y": 113
},
"options": {
"tooltip": {
@@ -1808,90 +1938,6 @@
"overrides": []
},
"id": 36
},
{
"title": "NodeStore Read Latency Distribution By Fetch Type & Found",
"description": "###### What this is:\n*Per-fetch nodestore read latency as a distribution rather than a mean, split by whether the fetch was a read-ahead or a blocking one and by whether the object was found. A mean cannot tell \"every read took 9 microseconds\" apart from \"most took 2 and a few took 900\", and the second is the cold-store shape. The split matters because a slow async fetch only delays prefetch while a slow sync fetch blocks a caller outright, and because a miss can cost a read of every backend.*\n\n###### How it's computed:\n*p50, p95 and p99 from histogram_quantile over rate(nodestore_read_us_bucket[$__rate_interval]), grouped by fetch_type and found. The instrument carries explicit sub-millisecond bucket boundaries starting at 1 microsecond, because the default ladder's lowest edge sits above the entire warm-read range and would make every warm read land in bucket zero.*\n\n###### Reading it:\n*Use the Fetch Type and Found template filters at the top of the dashboard to isolate a dimension. sync + found is the series that maps most directly to caller-visible stall time. A p99 that pulls away from p50 while p50 stays low is the cold-read signature the mean-based panels can only hint at: most reads are cached and a minority are paying full disk latency. found=false rising in absolute terms points at working-set misses rather than at cold cache.*\n\n###### Healthy range:\n*p99 within a small multiple of p50, both in the single-digit-to-low-tens of microseconds warm.*\n\n###### Watch for:\n*p99 above roughly 100 microseconds on the sync + found series while the hit ratio stays high -- objects are present but every access reaches disk. Note this instrument is separate from the read_mean_us gauge on the first panel of this row; if the histogram is empty but the gauge is populated, the exporter on that build predates the histogram and only the mean-based panels apply.*\n\n###### Source:\n[app/main/NodeStoreScheduler.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/main/NodeStoreScheduler.cpp)\n\n###### Function:\n`NodeStoreScheduler::onFetch`",
"type": "timeseries",
"gridPos": {
"h": 8,
"w": 24,
"x": 0,
"y": 149
},
"options": {
"tooltip": {
"maxHeight": 600,
"mode": "multi",
"sort": "desc"
},
"legend": {
"displayMode": "table",
"placement": "right",
"calcs": ["mean", "max"]
}
},
"targets": [
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "label_replace(label_join(label_replace(label_join(histogram_quantile(0.5, sum by (le, fetch_type, found, service_instance_id, xrpl_branch, xrpl_node_role) (rate(nodestore_read_us_bucket{fetch_type=~\"$fetch_type\", found=~\"$found\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"read_dims\", \"-\", \"fetch_type\", \"found\"), \"series\", \"Read p50 [$1]\", \"read_dims\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "label_replace(label_join(label_replace(label_join(histogram_quantile(0.95, sum by (le, fetch_type, found, service_instance_id, xrpl_branch, xrpl_node_role) (rate(nodestore_read_us_bucket{fetch_type=~\"$fetch_type\", found=~\"$found\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"read_dims\", \"-\", \"fetch_type\", \"found\"), \"series\", \"Read p95 [$1]\", \"read_dims\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")"
},
{
"datasource": {
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "label_replace(label_join(label_replace(label_join(histogram_quantile(0.99, sum by (le, fetch_type, found, service_instance_id, xrpl_branch, xrpl_node_role) (rate(nodestore_read_us_bucket{fetch_type=~\"$fetch_type\", found=~\"$found\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\"}[$__rate_interval]))), \"read_dims\", \"-\", \"fetch_type\", \"found\"), \"series\", \"Read p99 [$1]\", \"read_dims\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")"
}
],
"fieldConfig": {
"defaults": {
"displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}",
"unit": "\u00b5s",
"thresholds": {
"steps": [
{
"color": "green",
"value": null
},
{
"color": "yellow",
"value": 10
},
{
"color": "red",
"value": 100
}
]
},
"custom": {
"axisLabel": "Read Latency (\u00b5s)",
"scaleDistribution": {
"type": "log",
"log": 10
},
"thresholdsStyle": {
"mode": "line"
},
"spanNulls": 1800000,
"insertNulls": false,
"showPoints": "auto",
"pointSize": 3
}
},
"overrides": []
},
"id": 37
}
],
"schemaVersion": 39,
@@ -2057,46 +2103,6 @@
"multi": true,
"refresh": 2,
"sort": 1
},
{
"name": "fetch_type",
"label": "Fetch Type",
"description": "Filter nodestore read latency by fetch kind (async read-ahead / sync blocking)",
"type": "query",
"query": "label_values(nodestore_read_us_bucket, fetch_type)",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"includeAll": true,
"allValue": ".*",
"current": {
"text": "All",
"value": "$__all"
},
"multi": true,
"refresh": 2,
"sort": 1
},
{
"name": "found",
"label": "Found",
"description": "Filter nodestore read latency by whether the fetch found the object",
"type": "query",
"query": "label_values(nodestore_read_us_bucket, found)",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"includeAll": true,
"allValue": ".*",
"current": {
"text": "All",
"value": "$__all"
},
"multi": true,
"refresh": 2,
"sort": 1
}
]
},

6
docker/telemetry/grafana/dashboards/node-health.json
View File

@@ -3908,7 +3908,7 @@
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"description": "###### What this is:\n*Cumulative object-payload bytes this process has written to the NuDB object-store back end. Despite the metric name, this is not the size of the store on disk.*\n\n###### How it's computed:\n*Current value of the nudb_bytes gauge, plotted over time. It observes getStoreSize(), the same accessor node_written_bytes uses, so the two series are equal and their ratio is a constant 1.0 rather than a write-amplification measure.*\n\n###### Reading it:\n*A smoothly growing line is normal; the slope is the write rate. It excludes NuDB's keys, bucket padding and log, and it restarts from zero with the process while the files on disk do not.*\n\n###### Healthy range:\n*Gradual growth consistent with ledger data being stored.*\n\n###### Watch for:\n*A sudden jump in growth rate can indicate runaway storage or an unexpected back-fill. Do not use this panel to size the store on disk or to plan disk capacity; no metric reports on-disk size today, so check the filesystem directly.*\n\n###### Keywords:\n- **NuDB** *(per node)* — the append-only key-value database used as the default NodeStore backend.\n- **NodeStore** *(per node)* — the key-value object store holding ledger data (tree nodes), backed by NuDB.\n- **Back-fill / catch-up** *(per node)* — fetching missing historical ledgers from peers to fill gaps or reach the network tip.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerStorageDetailGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nudb)",
"description": "###### What this is:\n*Cumulative object-payload bytes this process has written to the NodeStore back end. This is not the size of the store on disk.*\n\n###### How it's computed:\n*Current value of the stored_object_bytes gauge, plotted over time. It observes getStoreSize(), the same accessor node_written_bytes uses, so the two series are equal and their ratio is a constant 1.0 rather than a write-amplification measure.*\n\n###### Reading it:\n*A smoothly growing line is normal; the slope is the write rate. It excludes NuDB's keys, bucket padding and log, and it restarts from zero with the process while the files on disk do not.*\n\n###### Healthy range:\n*Gradual growth consistent with ledger data being stored.*\n\n###### Watch for:\n*A sudden jump in growth rate can indicate runaway storage or an unexpected back-fill. Do not use this panel to size the store on disk or to plan disk capacity; no metric reports on-disk size today, so check the filesystem directly.*\n\n###### Keywords:\n- **NuDB** *(per node)* — the append-only key-value database used as the default NodeStore backend.\n- **NodeStore** *(per node)* — the key-value object store holding ledger data (tree nodes), backed by NuDB.\n- **Back-fill / catch-up** *(per node)* — fetching missing historical ledgers from peers to fill gaps or reach the network tip.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerStorageDetailGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#nudb)",
"fieldConfig": {
"defaults": {
"color": {
@@ -3995,11 +3995,11 @@
"type": "prometheus",
"uid": "${DS_PROMETHEUS}"
},
"expr": "label_replace(label_join(label_replace(storage_detail{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"nudb_bytes\"}, \"series\", \"NuDB Stored Bytes\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")",
"expr": "label_replace(label_join(label_replace(storage_detail{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"stored_object_bytes\"}, \"series\", \"NuDB Stored Bytes\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")",
"refId": "A"
}
],
"title": "NuDB Stored Bytes",
"title": "Stored Object Bytes",
"type": "timeseries"
},
{

2
docker/telemetry/workload/expected_metrics.json
View File

@@ -125,7 +125,7 @@
},
"parity_storage": {
"description": "External dashboard parity: storage detail metrics (MetricsRegistry).",
"metrics": ["storage_detail{metric=\"nudb_bytes\"}"]
"metrics": ["storage_detail{metric=\"stored_object_bytes\"}"]
},
"sync_diagnostics": {
"description": "Fresh-node sync diagnostics (native metrics). Bootstrap (Domain 0) and acquire-pipeline signals rendered by the ledger-sync-health dashboard. Names are appended one per signal. Histograms are listed by their Prometheus _bucket series (the bare instrument name is not a series). The two observable gauges carry an inline metric= selector so the specific sub-series is asserted, matching the parity_* groups.",