mirror of
https://github.com/XRPLF/rippled.git
synced 2026-08-21 22:30:57 +00:00
feat(telemetry): render, assert and document the quorum signals (WP-A5)
The quorum and publish gauges were emitted but never surfaced: no panel, no harness assertion, no reference entry. Completes those layers. - Four panels: trusted validations against the quorum target on one axis so a tally climbing toward quorum is visually distinct from one flat below it; publish lag; pre-accept shortfall rate; and time to first validated ledger. - Both signals are asserted by the workload validator. The shortfall counter does fire on a healthy cluster, because this node validates and then immediately re-enters the accept gate before its peers' validations arrive, so the first evaluation of every round tallies short. The panel and note say so, and give the fault signature instead: the shortfall rate outpacing the ledger-close rate while the tally stays flat and nothing ever reaches first-validated. - The quorum target is deliberately drawn as its own line rather than as a headroom stat, so the disabled-quorum sentinel reads as an unreachable target instead of an unreadable negative number. Also removes three reference rows that were appended twice when two agents each documented the same back-fill signals. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pratik Mankawde
@@ -1440,6 +1440,7 @@ panel that renders it.
|
||||
| `nodestore_latency` (`metric` = `write_mean_us` \| `read_mean_us` \| `write_count` \| `read_count`) | observable gauge | `MetricsRegistry.cpp` — `registerNodeStoreLatencyGauge` | NodeStore Write vs Read Latency (us/op); NodeStore Operation Rate | Mean microseconds per node-store store and per fetch, with both operation counts so a panel can divide the two rates and read _interval_ latency instead of the since-boot average. **The write side is the new signal.** `storeDurationUs_` was declared in `Database.h` and never written, and no accessor existed, so no write-path latency was observable anywhere; the read total was already exposed as `nodestore_state{metric="node_reads_duration_us"}`. This is the fingerprint of the "a node with a large existing DB syncs slower than a fresh one" symptom, which is write-bound and therefore invisible in every read-side metric. Chosen as a gauge over a histogram deliberately: a histogram gives true percentiles but costs one `Record()` per node object on the store/fetch path, and a single ledger write walks thousands of SHAMap nodes — this gauge instead reads four existing atomics once per ~10 s tick and adds nothing to the hot path. Consequence: **p99 is not obtainable from this signal**, and a histogram added later would also need an explicit-bucket View (`addMicrosecondHistogramView`) because the SDK default buckets top out at 10,000. Distinct from the Ledger Data Sync dashboard's NuDB Read Latency panel, which divides two `nodestore_state` fields in PromQL: that panel has no write-duration input to divide, because the quantity did not exist. **Known gap:** `write_mean_us` is emitted only when the store-duration total is non-zero, and that total is fed by `Database::recordStoreDuration`, today called only from `Database::importInternal` (the `[import_db]` admin path). `Database::store()` is pure virtual and neither `DatabaseNodeImp::store` nor `DatabaseRotatingImp::store` times itself yet, so an ordinary node reports `write_count` with no `write_mean_us`. The mean is omitted rather than reported as 0 so the gap stays visible instead of reading as "writes are instantaneous". |
|
||||
| `ledger_replay_fallback_total` (`stage` = `skiplist` \| `delta`) | counter | `SkipListAcquire.cpp` / `LedgerDeltaAcquire.cpp` — `trigger` | Replay Fallback to Full Acquire (by stage) | A ledger-replay sub-task abandoning its shortcut and acquiring the whole ledger through `InboundLedger` instead, because too few connected peers support the `LedgerReplay` protocol feature. Both branches were debug-log-only, so a silently defeated replay optimisation left no metric at all — back-fill simply ran on the slower path with nothing to show why. Emitted once, on the transition into fallback, not at the acquire call, which re-runs on every later trigger. The `stage` label separates the skip-list acquire (which fetches the list of historical ledger hashes) from the per-ledger delta acquire, because they fail independently. |
|
||||
| `ledger_replay_outcome_total` (`outcome` = `success` \| `timeout` \| `build_failed` \| `parameter_failed`) | counter | `LedgerReplayTask.cpp` — `LedgerReplayTask::recordOutcome` | Replay Outcomes (by terminal state) | Terminal state of every ledger-replay task, one emit per task. Every terminal path previously only set an internal `complete_`/`failed_` flag and wrote a log line, so a replay that never succeeded was indistinguishable from one that was never attempted. The outcome names the layer at fault: `timeout` means the deltas never arrived (a peer-supply problem), `build_failed` means a delta would not apply to its parent, and `parameter_failed` means a peer served a skip list inconsistent with what the task asked for — the latter two are data faults, not slowness. Read with `ledger_replay_fallback_total`: fallbacks rising while successes stay flat is replay-based catch-up degrading to full-ledger acquisition. |
|
||||
| `nodestore_latency` (`metric` = `write_mean_us` \| `read_mean_us` \| `write_count` \| `read_count`) | observable gauge | `MetricsRegistry.cpp` — `registerNodeStoreLatencyGauge` | NodeStore Write vs Read Latency (us/op); NodeStore Operation Rate | Mean microseconds per node-store store and per fetch, with both operation counts so a panel can divide the two rates and read _interval_ latency instead of the since-boot average. **The write side is the new signal.** `storeDurationUs_` was declared in `Database.h` and never written, and no accessor existed, so no write-path latency was observable anywhere; the read total was already exposed as `nodestore_state{metric="node_reads_duration_us"}`. This is the fingerprint of the "a node with a large existing DB syncs slower than a fresh one" symptom, which is write-bound and therefore invisible in every read-side metric. Chosen as a gauge over a histogram deliberately: a histogram gives true percentiles but costs one `Record()` per node object on the store/fetch path, and a single ledger write walks thousands of SHAMap nodes — this gauge instead reads four existing atomics once per ~10 s tick and adds nothing to the hot path. Consequence: **p99 is not obtainable from this signal**, and a histogram added later would also need an explicit-bucket View (`addMicrosecondHistogramView`) because the SDK default buckets top out at 10,000. Distinct from the Ledger Data Sync dashboard's NuDB Read Latency panel, which divides two `nodestore_state` fields in PromQL: that panel has no write-duration input to divide, because the quantity did not exist. **Known gap:** `write_mean_us` is emitted only when the store-duration total is non-zero, and that total is fed by `Database::recordStoreDuration`, today called only from `Database::importInternal` (the `[import_db]` admin path). `Database::store()` is pure virtual and neither `DatabaseNodeImp::store` nor `DatabaseRotatingImp::store` times itself yet, so an ordinary node reports `write_count` with no `write_mean_us`. The mean is omitted rather than reported as 0 so the gap stays visible instead of reading as "writes are instantaneous". |
|
||||
| `ledger_replay_fallback_total` (`stage` = `skiplist` \| `delta`) | counter | `SkipListAcquire.cpp` / `LedgerDeltaAcquire.cpp` — `trigger` | Replay Fallback to Full Acquire (by stage) | A ledger-replay sub-task abandoning its shortcut and acquiring the whole ledger through `InboundLedger` instead, because too few connected peers support the `LedgerReplay` protocol feature. Both branches were debug-log-only, so a silently defeated replay optimisation left no metric at all — back-fill simply ran on the slower path with nothing to show why. Emitted once, on the transition into fallback, not at the acquire call, which re-runs on every later trigger. The `stage` label separates the skip-list acquire (which fetches the list of historical ledger hashes) from the per-ledger delta acquire, because they fail independently. |
|
||||
| `ledger_replay_outcome_total` (`outcome` = `success` \| `timeout` \| `build_failed` \| `parameter_failed`) | counter | `LedgerReplayTask.cpp` — `LedgerReplayTask::recordOutcome` | Replay Outcomes (by terminal state) | Terminal state of every ledger-replay task, one emit per task. Every terminal path previously only set an internal `complete_`/`failed_` flag and wrote a log line, so a replay that never succeeded was indistinguishable from one that was never attempted. The outcome names the layer at fault: `timeout` means the deltas never arrived (a peer-supply problem), `build_failed` means a delta would not apply to its parent, and `parameter_failed` means a peer served a skip list inconsistent with what the task asked for — the latter two are data faults, not slowness. Read with `ledger_replay_fallback_total`: fallbacks rising while successes stay flat is replay-based catch-up degrading to full-ledger acquisition. |
|
||||
| `ledger_quorum_publish` (`metric` = `trusted_validation_tally` \| `quorum_target`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Trusted Validations vs Quorum Target | Trusted validations counted at the most recent pre-accept gate, beside the number that gate required. Snapshotted in `LedgerMaster::checkAccept` before the shortfall check, so a node that keeps failing the gate still reports both numbers — which is the whole point: the tally alone cannot say whether validations are accumulating toward quorum (slow, will finish) or plateaued below it (stuck). Read the sustained floor of the tally, not a single sample: each series is a snapshot of the last evaluation, and the first evaluation of each round runs before peer validations arrive, so a healthy node sawtooths. `quorum_target` is what the gate actually demanded, as opposed to `unl_quorum{metric="quorum"}` which is what the trusted list configures. When the trusted list disables quorum entirely (`getNeededValidations` returns `SIZE_MAX`) the target is reported as int64 max rather than wrapping to -1, so it reads far above any tally instead of inverting the comparison — the same sentinel handling as the `unl_quorum` gauge. |
|
||||
| `ledger_quorum_publish` (`metric` = `publish_lag`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Publish Lag (validated minus published) | Ledgers fully validated but not yet published to clients and subscribers: the validated sequence minus the published sequence, floored at zero. `pubLedgerSeq_` was never exported, so this gap was not derivable from any other series. Publishing trails validation by design and a small lag drains each round; a lag that stays positive or grows means validation is healthy and the publish pipeline is not, which is a distinct fault from anything the quorum or acquire signals can show. The two sequences are read as independent relaxed loads, so a sample taken mid-update may be off by one ledger for one poll — immaterial for a lag trend, and the price of not taking the LedgerMaster mutex on the metrics poll thread. |
|
||||
| `ledger_quorum_publish` (`metric` = `time_to_first_validated_us`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Time to First Validated Ledger | Microseconds from process start until the first ledger passed the pre-accept quorum gate. A one-shot measurement like `sync_state{metric="initial_full_duration_us"}`: written once under `mutex_` and never changed, so it has no trend. Exactly two readings are meaningful — a duration, meaning the node reached its first fully-validated ledger and this is how long that took, or 0, meaning it never has. Clamped to a minimum of 1 so a genuine sub-microsecond reading can never be confused with the never-reached zero. A value here alongside a zero on time-to-first-FULL, or the reverse, separates "reached the full server state" from "fully validated a ledger". |
|
||||
| `ledger_quorum_shortfall_total` (`stage` = `pre_accept`) | counter | `LedgerMaster.cpp` — `LedgerMaster::checkAccept` | Pre-Accept Quorum Shortfall Rate | One increment per pre-accept gate evaluation rejected because the trusted validation tally was below quorum. Previously trace-log-only, which made a node that peers and receives validations yet never validates indistinguishable from an idle one. A non-zero rate is **not** by itself a fault: `doAccept` issues this node's own validation and calls `consensusBuilt` → `checkAccept` immediately, before peer validations for that ledger arrive, so the first evaluation of every round tallies short and is retried as validations come in — a healthy cluster emits this counter every round. The fault signature is the rate climbing well above the ledger-close rate while the tally stays flat below its target and `time_to_first_validated_us` stays at 0. Emitted while `mutex_` is held, which is safe against the metrics poll because every accessor the sync gauges read is a lock-free atomic load, so no OTel callback ever acquires `mutex_`. |
|
||||
|
||||
@@ -3809,6 +3809,386 @@
|
||||
],
|
||||
"title": "Replay Outcomes (by terminal state)",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"description": "###### What this is:\n*Trusted validations counted at the most recent pre-accept gate, plotted against the number that gate required.*\n\n###### How it's computed:\n*Two series from the same gauge: trusted_validation_tally (agreeing trusted validations seen for the candidate ledger, after the negative-UNL filter) and quorum_target (what the gate demanded). Both are snapshotted on every gate evaluation, whether it passed or failed, so a node that keeps failing still reports both numbers.*\n\n###### Reading it:\n*Read the shape of the tally, not any single value. A tally climbing toward the target is a slow sync that will finish, so keep waiting. A tally flat below the target is stuck: it will never reach quorum on its own, and nothing in the acquire pipeline can fix it. Expect a sawtooth on a healthy node: each series is a snapshot of the most recent gate evaluation, and the first evaluation of every round runs before peer validations arrive, so a sampled low reading between higher ones is normal. Judge it over minutes, and only the sustained floor of the tally against the target carries the signal.*\n\n###### Healthy range:\n*Tally at or above Target, both flat, on a node that is validating.*\n\n###### Watch for:\n*A tally pinned below the target — too few trusted validators are reachable, or the UNL / negative-UNL configuration excludes the ones that are. Also watch the target jumping to about 9.2e18 (signed 64-bit maximum): that is the explicit quorum-disabled sentinel, meaning too many publishers are unavailable and the trusted list switched quorum off entirely, so the node can never validate however far the tally climbs. It is reported as that maximum rather than wrapping negative precisely so it cannot be misread as a tally that already exceeds its target. Both series flat at 0 means the gate has never been evaluated — nothing has been offered for validation yet, which sends you back to the Bootstrap row.*\n\n###### Keywords:\n- **Quorum shortfall** *(per node)* — trusted validations for a candidate ledger falling short of the quorum needed to declare it validated, so the node holds the ledger and still cannot call it validated.\n- **Validation quorum** *(per node)* — the number of agreeing trusted validations a ledger needs before this node treats it as validated.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerLedgerQuorumPublishGauge`\n\n###### References:\n[Negative UNL and validation quorum on xrpl.org](https://xrpl.org/docs/concepts/consensus-protocol/negative-unl) · [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#quorum-shortfall)",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "Validations",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"barWidthFactor": 0.6,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"insertNulls": false,
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 2,
|
||||
"pointSize": 3,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "auto",
|
||||
"showValues": false,
|
||||
"spanNulls": 1800000,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "off"
|
||||
}
|
||||
},
|
||||
"displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}",
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
}
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 12,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 242
|
||||
},
|
||||
"id": 42,
|
||||
"options": {
|
||||
"annotations": {
|
||||
"clustering": -1,
|
||||
"multiLane": false
|
||||
},
|
||||
"legend": {
|
||||
"calcs": [],
|
||||
"displayMode": "list",
|
||||
"enableFacetedFilter": false,
|
||||
"overflow": "ellipsis",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"hideZeros": false,
|
||||
"maxHeight": 600,
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "13.2.0-28926505616",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "label_replace(label_join(label_replace(ledger_quorum_publish{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"trusted_validation_tally\"}, \"series\", \"Trusted Tally\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "label_replace(label_join(label_replace(ledger_quorum_publish{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"quorum_target\"}, \"series\", \"Quorum Target\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")",
|
||||
"refId": "B"
|
||||
}
|
||||
],
|
||||
"title": "Trusted Validations vs Quorum Target",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"description": "###### What this is:\n*How many ledgers this node has fully validated but not yet published to its clients and subscribers.*\n\n###### How it's computed:\n*ledger_quorum_publish series publish_lag: the validated ledger sequence minus the published ledger sequence, floored at zero. The published sequence was never exported before, so this gap was not derivable from any other series.*\n\n###### Reading it:\n*Publishing trails validation by design, so a small lag that drains each round is normal. A lag that stays positive, or grows, means validation is healthy and the publish pipeline is not — a different fault from anything the quorum or acquire panels can show.*\n\n###### Healthy range:\n*0 to 1 ledger.*\n\n###### Watch for:\n*A monotonic climb: the publish loop is falling behind a chain tip the node already holds, so clients and subscriptions see stale data while the node itself is current. Read it with Worker Pool Saturation and Deferred Jobs by Type (starvation) — a starved job queue is the usual cause. A flat 0 is only healthy on a node that is validating: on one that never has, the 0 means nothing has been validated to publish, so read Trusted Validations vs Quorum Target first.*\n\n###### Keywords:\n- **Publish lag** *(per node)* — validated ledgers not yet published to clients and subscribers, i.e. the gap between the validated and the published sequence.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerLedgerQuorumPublishGauge`\n\n###### References:\n[Ledger close and publication on xrpl.org](https://xrpl.org/docs/concepts/consensus-protocol) · [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#publish-lag)",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "Ledgers",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"barWidthFactor": 0.6,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"insertNulls": false,
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 2,
|
||||
"pointSize": 3,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "auto",
|
||||
"showValues": false,
|
||||
"spanNulls": 1800000,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "line"
|
||||
}
|
||||
},
|
||||
"displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}",
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "yellow",
|
||||
"value": 2
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 10
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "short"
|
||||
}
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 12,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 242
|
||||
},
|
||||
"id": 43,
|
||||
"options": {
|
||||
"annotations": {
|
||||
"clustering": -1,
|
||||
"multiLane": false
|
||||
},
|
||||
"legend": {
|
||||
"calcs": [],
|
||||
"displayMode": "list",
|
||||
"enableFacetedFilter": false,
|
||||
"overflow": "ellipsis",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"hideZeros": false,
|
||||
"maxHeight": 600,
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "13.2.0-28926505616",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "label_replace(label_join(label_replace(ledger_quorum_publish{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"publish_lag\"}, \"series\", \"Publish Lag\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Publish Lag (validated minus published)",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"description": "###### What this is:\n*Rate at which the pre-accept gate refused to declare a candidate ledger validated because its trusted validations were below quorum.*\n\n###### How it's computed:\n*Rate of ledger_quorum_shortfall_total by stage. One increment per rejected gate evaluation, emitted from the early return that was trace-log-only before, so a node that peers and receives validations yet never validates is no longer indistinguishable from an idle one. The gate is re-entered on every fresh trusted validation for the candidate ledger, so one ledger that eventually validates can contribute several increments on its way there.*\n\n###### Reading it:\n*A steady low rate is NORMAL and is not a fault. The gate is evaluated the instant this node finishes building a ledger, before its peers' validations for that ledger have arrived, so the first evaluation of each round routinely tallies short and is retried as validations come in. Read this against the rate of ledger closes and against Trusted Validations vs Quorum Target — it is the ratio and the accompanying tally that carry the signal, never the bare presence of a rate.*\n\n###### Healthy range:\n*A low steady rate on the order of one per ledger close or less, on a node that is validating.*\n\n###### Watch for:\n*A rate that climbs well above the ledger-close rate while Publish Lag grows and Time to First Validated Ledger stays at zero — that combination is the retry loop never converging, so the tally is not merely early, it never reaches the target. Confirm on Trusted Validations vs Quorum Target: a climbing tally is slow and will finish, a flat tally below the target is stuck (too few trusted validators reachable, or a UNL / negative-UNL misconfiguration). Check UNL Quorum Headroom in the Bootstrap row before anything else in this row, because a trusted list that cannot satisfy quorum makes every panel below it look starved.*\n\n###### Keywords:\n- **Quorum shortfall** *(per node)* — trusted validations for a candidate ledger falling short of the quorum needed to declare it validated, so the node holds the ledger and still cannot call it validated.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (call-site metric macro, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[LedgerMaster.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/app/ledger/detail/LedgerMaster.cpp)\n\n###### Function:\n`LedgerMaster::checkAccept`\n\n###### References:\n[Negative UNL and validation quorum on xrpl.org](https://xrpl.org/docs/concepts/consensus-protocol/negative-unl) · [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#quorum-shortfall)",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "palette-classic"
|
||||
},
|
||||
"custom": {
|
||||
"axisBorderShow": false,
|
||||
"axisCenteredZero": false,
|
||||
"axisColorMode": "text",
|
||||
"axisLabel": "Shortfalls / Sec",
|
||||
"axisPlacement": "auto",
|
||||
"barAlignment": 0,
|
||||
"barWidthFactor": 0.6,
|
||||
"drawStyle": "line",
|
||||
"fillOpacity": 10,
|
||||
"gradientMode": "none",
|
||||
"hideFrom": {
|
||||
"legend": false,
|
||||
"tooltip": false,
|
||||
"viz": false
|
||||
},
|
||||
"insertNulls": false,
|
||||
"lineInterpolation": "linear",
|
||||
"lineWidth": 2,
|
||||
"pointSize": 3,
|
||||
"scaleDistribution": {
|
||||
"type": "linear"
|
||||
},
|
||||
"showPoints": "auto",
|
||||
"showValues": false,
|
||||
"spanNulls": 1800000,
|
||||
"stacking": {
|
||||
"group": "A",
|
||||
"mode": "none"
|
||||
},
|
||||
"thresholdsStyle": {
|
||||
"mode": "line"
|
||||
}
|
||||
},
|
||||
"displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}",
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "green",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "red",
|
||||
"value": 0.1
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "ops"
|
||||
}
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 12,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 254
|
||||
},
|
||||
"id": 44,
|
||||
"options": {
|
||||
"annotations": {
|
||||
"clustering": -1,
|
||||
"multiLane": false
|
||||
},
|
||||
"legend": {
|
||||
"calcs": [],
|
||||
"displayMode": "list",
|
||||
"enableFacetedFilter": false,
|
||||
"overflow": "ellipsis",
|
||||
"placement": "bottom",
|
||||
"showLegend": true
|
||||
},
|
||||
"tooltip": {
|
||||
"hideZeros": false,
|
||||
"maxHeight": 600,
|
||||
"mode": "multi",
|
||||
"sort": "desc"
|
||||
}
|
||||
},
|
||||
"pluginVersion": "13.2.0-28926505616",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "label_replace(label_join(label_replace(sum by (stage, service_instance_id, xrpl_branch, xrpl_work_item) (rate(ledger_quorum_shortfall_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", stage=~\"$shortfall_stage\"}[$__rate_interval])), \"series\", \"$1\", \"stage\", \"(.*)\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Pre-Accept Quorum Shortfall Rate",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"description": "###### What this is:\n*How long the node took, from process start, to pass the pre-accept quorum gate for the first time.*\n\n###### How it's computed:\n*ledger_quorum_publish series time_to_first_validated_us, converted from microseconds to seconds.*\n\n###### Reading it:\n*A one-shot measurement: it fills in the moment the node first fully validates a ledger and never changes again, so it has no trend to read. Exactly two readings matter — a duration, meaning the node got there and this is how long it took, or zero (red), meaning it never has.*\n\n###### Healthy range:\n*Seconds to a few minutes; longer on a fresh node that must acquire history first.*\n\n###### Watch for:\n*A flat zero while Time to First FULL shows a value: the node reached the full server state but has still never fully validated a ledger, which points at the quorum gate rather than at acquire. The measurement is clamped to a minimum of 1 microsecond so a genuine reading can never be confused with the never-reached zero.*\n\n###### Keywords:\n- **Time to first validated ledger** *(per node)* — elapsed time from process start until the node first declared a ledger fully validated.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerLedgerQuorumPublishGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#time-to-first-validated-ledger)",
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": {
|
||||
"mode": "thresholds"
|
||||
},
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{
|
||||
"color": "red",
|
||||
"value": null
|
||||
},
|
||||
{
|
||||
"color": "green",
|
||||
"value": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
"unit": "s"
|
||||
}
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 12,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 254
|
||||
},
|
||||
"id": 45,
|
||||
"options": {
|
||||
"colorMode": "value",
|
||||
"graphMode": "none",
|
||||
"justifyMode": "center",
|
||||
"orientation": "auto",
|
||||
"percentChangeColorMode": "standard",
|
||||
"reduceOptions": {
|
||||
"calcs": ["lastNotNull"],
|
||||
"fields": "",
|
||||
"values": false
|
||||
},
|
||||
"showPercentChange": false,
|
||||
"textMode": "value_and_name",
|
||||
"wideLayout": true
|
||||
},
|
||||
"pluginVersion": "13.2.0-28926505616",
|
||||
"targets": [
|
||||
{
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"expr": "label_replace(label_join(label_replace(ledger_quorum_publish{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"time_to_first_validated_us\"} / 1e6, \"series\", \"Time to First Validated\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Time to First Validated Ledger",
|
||||
"type": "stat"
|
||||
}
|
||||
],
|
||||
"schemaVersion": 39,
|
||||
@@ -4327,6 +4707,26 @@
|
||||
"multi": true,
|
||||
"refresh": 2,
|
||||
"sort": 1
|
||||
},
|
||||
{
|
||||
"name": "shortfall_stage",
|
||||
"label": "Shortfall Stage",
|
||||
"description": "Filter quorum-shortfall rejections by gate stage [pre_accept]",
|
||||
"type": "query",
|
||||
"query": "label_values(ledger_quorum_shortfall_total, stage)",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "${DS_PROMETHEUS}"
|
||||
},
|
||||
"includeAll": true,
|
||||
"allValue": ".*",
|
||||
"current": {
|
||||
"text": "All",
|
||||
"value": "$__all"
|
||||
},
|
||||
"multi": true,
|
||||
"refresh": 2,
|
||||
"sort": 1
|
||||
}
|
||||
]
|
||||
},
|
||||
|
||||
@@ -176,14 +176,20 @@
|
||||
"peer_accept_total",
|
||||
"nodestore_latency{metric=\"write_count\"}",
|
||||
"nodestore_latency{metric=\"read_count\"}",
|
||||
"nodestore_latency{metric=\"read_mean_us\"}"
|
||||
"nodestore_latency{metric=\"read_mean_us\"}",
|
||||
"ledger_quorum_publish{metric=\"trusted_validation_tally\"}",
|
||||
"ledger_quorum_publish{metric=\"quorum_target\"}",
|
||||
"ledger_quorum_publish{metric=\"time_to_first_validated_us\"}",
|
||||
"ledger_quorum_publish{metric=\"publish_lag\"}",
|
||||
"ledger_quorum_shortfall_total{stage=\"pre_accept\"}"
|
||||
],
|
||||
"_acquire_note": "The four sync_acquire sub-series and shamap_cache_hit_rate are unconditional: both are observable gauges whose callbacks observe every series on each collection tick, so each is present even when the value is 0 (an idle node reports in_flight=0 and missing_state_nodes_max=0, and a cold cache reports a 0.0 hit rate). Absence, not a zero, is the regression. The three WP-A3 counters (sync_acquire_source_total, sync_addnode_total, sync_acquire_no_progress_total) are deliberately NOT asserted here: all three are emitted only from InboundLedger, which runs only when a node must fetch a ledger it lacks. expected_spans.json already marks the ledger.acquire span optional for exactly this reason (\"A healthy local cluster rarely back-fills history\"), and the metric validator has no per-metric optional flag, so listing them would fail the harness red on a healthy run. They are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp and by the ledger-sync-health panels; add them here only alongside a harness step that forces a real acquire (e.g. starting a node against an existing ledger history).",
|
||||
"_jobq_note": "The jobq_backlog and jobq_saturation series are unconditional: both are observable gauges whose callbacks iterate EVERY registered JobType (jobData_ is populated from JobTypes at JobQueue construction) and observe all three fields on each collection tick, so a series exists even when the value is 0. That is why an idle-but-registered type like ledgerData is safe to assert by name here — a fresh harness node that never defers a single job still reports jobq_backlog{metric=\"deferred\",job_type=\"ledgerData\"} = 0, and absence, not the zero, is the regression. Two job_type values are asserted (ledgerData and ledgerRequest) because they are the sync-critical types capped at concurrency 3 in JobTypes.h, so they are the ones whose deferred series must never silently vanish. Only deferred is asserted for ledgerRequest to keep the list short: the three-field fan-out is already proven by ledgerData. worker_threads is asserted because it is the denominator of the dashboard saturation ratio, and it is always at least 1 (the JobQueue ctor gives standalone mode exactly one worker), so a zero or missing reading there means the accessor regressed rather than the node being idle.",
|
||||
"_conditional_note": "handshake_negotiation_fail_total and unl_fetch_total are conditional under the local harness: the first only exists once a handshake is rejected, and the second needs a [validator_list_sites] entry (run-full-validation.sh generates a static [validators] file instead). The validator has no per-metric optional flag, so if either reports 0 series in a harness run, move it out of this group rather than weakening the check.",
|
||||
"_sync_state_note": "The four sync_state sub-series are unconditional: the gauge observes all four on every collection tick, so each is present as a series even when its value is 0 (a node that never reached FULL reports initial_full_duration_us=0, and a healthy node reports server_stall_seconds=0). The check asserts series presence, not a non-zero value, which is exactly right here — a zero is a meaningful reading for these signals, and absence is the regression. server_stall_events_total is likewise always present because the observable counter reports the tally (0 or more) every tick. state_changes_total is asserted here with a from!=\"\",to!=\"\" selector rather than bare (parity_counters already asserts the bare name): the selector is what proves the WP-A2 {from,to} label dimension actually reached Prometheus, so a regression to the old unlabelled counter fails this check instead of silently passing on the bare name. It needs at least one real mode transition, which any node reaching connected/syncing produces during startup.",
|
||||
"_a7_note": "WP-A7 adds three observable gauges and four counters. The 16 gauge sub-series (peer_ledger_supply, peerfinder_slot_census, amendment_block) are unconditional and asserted individually: each callback in MetricsRegistry.cpp calls observe() for every field on every collection tick with no early return between them, so the series exists whatever the value. That includes the two sentinel readings — a node whose peers have advertised nothing reports peer_ledger_supply{metric=\"supply_min_seq\"} = 0 meaning unknown, and a node with no pending amendment reports amendment_block{metric=\"seconds_to_block\"} = -1 meaning healthy. Absence, not the sentinel, is the regression. Of the four counters only peer_accept_total is asserted: run-full-validation.sh gives every node a [port_peer] on 0.0.0.0 and lists the other four nodes in [ips], so all 5 nodes dial each other and each one is also dialled, which means OverlayImpl::onHandoff runs and reports outcome=accepted (or slot_refused/no_slot on the duplicate half of each mutual dial) on every node. It is asserted bare rather than with an outcome= selector because which outcome a given node records depends on dial ordering, which the harness does not control. The other three counters are deliberately NOT asserted. peer_disconnect_total is emitted only from PeerImp::close, and a healthy 5-node localhost cluster holds its 4 fixed peers for the whole run: the timer-driven reasons need maxUnknownTime (600 s) or maxDivergedTime (300 s) to elapse (Config.h) while the full-validation profile totals well under that, and the shutdown reasons only fire during teardown, which happens in run-full-validation.sh after Step 5 has already scraped. serve_refused_total needs a peer to ask this node for a ledger, tx set or object it cannot serve — on a cluster where every node has the same complete history from genesis, getLedger()/getTxSet() succeed and the send queues never approach Tuning::kDropSendQueue. ledger_jump_total needs NetworkOPsImp::switchLastClosedLedger, reached only when consensus reports an LCL this node did not build on; a healthy 5-node cluster agrees every round, so it never jumps. The metric validator has no per-metric optional flag, so listing any of the three would fail the harness red on a healthy run — the same reasoning _acquire_note applies to the WP-A3 InboundLedger counters. All four counters are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp and rendered by the ledger-sync-health panels Peer Disconnects by Reason, Ledger/Object Serve Refusals and Byzantine Ledger Jumps. To make them assertable the harness would need a fault-injection step: kill one node mid-run and re-scrape before teardown (peer_disconnect_total, reason=read_error/graceful), request a ledger sequence outside the cluster's history or drive a node past its send-queue limit (serve_refused_total), and start a node on a divergent chain tip or partition the cluster and heal it (ledger_jump_total).",
|
||||
"_a6_note": "WP-A6 adds one observable gauge (nodestore_latency) and two counters (ledger_replay_fallback_total, ledger_replay_outcome_total). Only three of the four gauge sub-series are asserted. write_count and read_count are unconditional: the callback observes both on every collection tick with no early return before them, so a series exists whatever the value, and a node that has written nothing reports write_count=0 rather than dropping the series. read_mean_us is safe because any node that has opened a ledger has already fetched objects, so the fetch duration total is non-zero. write_mean_us is deliberately NOT asserted: the mean is emitted only when the store-duration total is non-zero, and that total is fed by Database::recordStoreDuration(), which today is called only from Database::importInternal -- the [import_db] admin path. Database::store() is pure virtual and the two concrete runtime overrides (DatabaseNodeImp::store, DatabaseRotatingImp::store) do not time themselves yet, so an ordinary harness node produces write_count but no write_mean_us. Asserting it would ship a permanently red CI check for a known, documented gap; the omission is the honest encoding of that gap. The two replay counters are likewise NOT asserted, for the same reason _acquire_note gives for the WP-A3 InboundLedger counters: both are emitted only from the ledger-replay path, which requires the [ledger_replay] config stanza AND a real historical back-fill against peers that support the LedgerReplay protocol feature. run-full-validation.sh starts a fresh local cluster with no history to back-fill, so no replay task is ever created and neither counter can produce a series. All three unasserted signals are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp and rendered by the ledger-sync-health panels NodeStore Write vs Read Latency, Replay Fallback to Full Acquire and Replay Outcomes. To make them assertable the harness would need to enable [ledger_replay] and start a node against an existing ledger history so it back-fills through the replay path, and to time the two concrete store overrides."
|
||||
"_a6_note": "WP-A6 adds one observable gauge (nodestore_latency) and two counters (ledger_replay_fallback_total, ledger_replay_outcome_total). Only three of the four gauge sub-series are asserted. write_count and read_count are unconditional: the callback observes both on every collection tick with no early return before them, so a series exists whatever the value, and a node that has written nothing reports write_count=0 rather than dropping the series. read_mean_us is safe because any node that has opened a ledger has already fetched objects, so the fetch duration total is non-zero. write_mean_us is deliberately NOT asserted: the mean is emitted only when the store-duration total is non-zero, and that total is fed by Database::recordStoreDuration(), which today is called only from Database::importInternal -- the [import_db] admin path. Database::store() is pure virtual and the two concrete runtime overrides (DatabaseNodeImp::store, DatabaseRotatingImp::store) do not time themselves yet, so an ordinary harness node produces write_count but no write_mean_us. Asserting it would ship a permanently red CI check for a known, documented gap; the omission is the honest encoding of that gap. The two replay counters are likewise NOT asserted, for the same reason _acquire_note gives for the WP-A3 InboundLedger counters: both are emitted only from the ledger-replay path, which requires the [ledger_replay] config stanza AND a real historical back-fill against peers that support the LedgerReplay protocol feature. run-full-validation.sh starts a fresh local cluster with no history to back-fill, so no replay task is ever created and neither counter can produce a series. All three unasserted signals are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp and rendered by the ledger-sync-health panels NodeStore Write vs Read Latency, Replay Fallback to Full Acquire and Replay Outcomes. To make them assertable the harness would need to enable [ledger_replay] and start a node against an existing ledger history so it back-fills through the replay path, and to time the two concrete store overrides.",
|
||||
"_a5_note": "WP-A5 adds one observable gauge (ledger_quorum_publish) and one counter (ledger_quorum_shortfall_total). All four gauge sub-series are asserted and are unconditional: registerLedgerQuorumPublishGauge's callback in MetricsRegistry.cpp calls observe() for every field on every collection tick with no early return between them, and each accessor is a plain relaxed atomic load that always returns a value, so the series exists whatever the reading. That deliberately includes the three diagnostic zeros: a node that has never had a gate evaluated reports trusted_validation_tally=0 and quorum_target=0, one that has never fully validated reports time_to_first_validated_us=0, and one that is caught up reports publish_lag=0. Absence, not the zero, is the regression -- the same reasoning _sync_state_note gives for initial_full_duration_us. Note the sentinel: when the trusted list disables quorum entirely, getNeededValidations() returns SIZE_MAX and LedgerMaster reports quorum_target as int64 max rather than letting the cast wrap to -1, so the target reads far above any tally instead of inverting the comparison (the same fix as the unl_quorum gauge). ledger_quorum_shortfall_total IS asserted, which differs from the WP-A3/A6/A7 counters, and the reason is that this counter does not need a fault to fire. RCLConsensus::Adaptor::doAccept issues this node's own validation and then calls ledgerMaster_.consensusBuilt immediately (RCLConsensus.cpp), which calls checkAccept on the freshly built ledger (LedgerMaster.cpp) BEFORE the peers' validations for that same ledger have arrived. With the harness's 5 validators the quorum is max(ceil(5*0.8), ceil(5*0.6)) = 4 (ValidatorList::calculateQuorum), so that first evaluation of each round tallies short of 4 and takes the shortfall early return; the gate is then re-entered from RCLValidations handleNewValidation as each trusted validation arrives and eventually passes. A HEALTHY 5-node cluster therefore emits this counter every round, which is why it is safe to assert on a clean run -- unlike peer_disconnect_total or ledger_replay_fallback_total, it needs no fault injection, no [ledger_replay] stanza and no historical back-fill. The stage=\"pre_accept\" selector is asserted rather than the bare name so that the label dimension is proven to have reached Prometheus, matching the state_changes_total{from,to} pattern. Consequence for readers of the panels: a non-zero rate on Pre-Accept Quorum Shortfall Rate is NOT by itself a fault, and the panel description says so; the fault signature is that rate climbing well above the ledger-close rate while the tally on Trusted Validations vs Quorum Target stays flat below its target. If a future harness change makes the cluster single-node or standalone this assertion must move to a note: standalone_ short-circuits consensusBuilt before checkAccept, and getNeededValidations() returns 0 in standalone mode, so the gate can never report a shortfall."
|
||||
},
|
||||
"grafana_dashboards": {
|
||||
"description": "All Grafana dashboards that must render data (UIDs as provisioned on disk under docker/telemetry/grafana/dashboards/).",
|
||||
|
||||
@@ -795,6 +795,36 @@ The share of the job-queue worker threads currently executing a job. The thread
|
||||
|
||||
**See also:** [Deferred job](#deferred-job) · [Job queue occupancy](#job-queue-occupancy) · [Server stall](#server-stall)
|
||||
|
||||
<a id="quorum-shortfall"></a>
|
||||
|
||||
### Quorum shortfall
|
||||
|
||||
A candidate ledger being refused the status of validated because the agreeing trusted validations counted for it fell short of the quorum it needed. It is the last gate of the sync pipeline and the one that can fail while everything upstream looks healthy: the node can hold every ledger it needs, apply them all, and still never declare one validated. Two shapes mean opposite things. A tally accumulating toward the quorum is slow and will get there, so the shortfall is transient. A tally that plateaus below the quorum is stuck, and the causes are upstream of ledger acquisition entirely — too few trusted validators reachable, or a validator-list or negative-UNL configuration that excludes the ones that are — so nothing in acquisition can fix it. A shortfall is also expected briefly on every healthy round, because the gate is first evaluated the moment this node finishes building a ledger, before its peers' validations for that ledger have arrived, and is retried as they come in. That makes the bare occurrence of a shortfall uninformative; only its persistence alongside a flat tally is a fault.
|
||||
|
||||
**Scope:** per node — measured on and specific to this individual server.
|
||||
|
||||
**See also:** [Validation quorum](#validation-quorum) · [UNL quorum headroom](#unl-quorum-headroom) · [Validated ledger](#validated-ledger) · [Time to first validated ledger](#time-to-first-validated-ledger) · [Negative UNL on xrpl.org](https://xrpl.org/docs/concepts/consensus-protocol/negative-unl)
|
||||
|
||||
<a id="publish-lag"></a>
|
||||
|
||||
### Publish lag
|
||||
|
||||
The number of ledgers a node has fully validated but not yet published to its clients and subscribers. Publication trails validation by design, so a small lag that drains each round is the normal state; the diagnostic reading is a lag that stays positive or grows. That is a distinct fault from anything the quorum or acquisition signals describe: validation is working, the node itself is current, and only the pipeline that hands finished ledgers to subscribers is behind — so the visible symptom is stale data for API clients on a server that is not itself behind the network. Because it is local processing rather than peer supply that falls behind, the causes sit in job-queue starvation and main-loop stalls. One reading trap: a lag of zero is only healthy on a node that is validating. On a node that never has, the zero means there is nothing validated to publish at all, and the quorum gate is the thing to read instead.
|
||||
|
||||
**Scope:** per node — measured on and specific to this individual server.
|
||||
|
||||
**See also:** [Published ledger](#published-ledger) · [Validated ledger](#validated-ledger) · [Quorum shortfall](#quorum-shortfall) · [Deferred job](#deferred-job) · [Worker-pool saturation](#worker-pool-saturation)
|
||||
|
||||
<a id="time-to-first-validated-ledger"></a>
|
||||
|
||||
### Time to first validated ledger
|
||||
|
||||
The elapsed time from process start until the node first declared a ledger fully validated. Like the time to first reaching the full server state, it is a one-shot measurement: it is set the first time the quorum gate passes and never changes afterwards, so it has no trend to read. That leaves exactly two meaningful readings — a duration, meaning the node got there and this is how long it took, or zero, meaning it never has. The zero is the diagnostic signal rather than absent data. Its value is in the pairing: a duration for reaching the full server state beside a zero here means the node reached that state but has still never fully validated a ledger, which places the fault at the quorum gate rather than in ledger acquisition.
|
||||
|
||||
**Scope:** per node — measured on and specific to this individual server.
|
||||
|
||||
**See also:** [Time to first FULL](#time-to-first-full) · [Quorum shortfall](#quorum-shortfall) · [Validated ledger](#validated-ledger) · [Operating mode / server state](#operating-mode-server-state)
|
||||
|
||||
<a id="cat-peer-overlay-networking"></a>
|
||||
|
||||
## Peer & Overlay Networking
|
||||
|
||||
@@ -2500,6 +2500,86 @@ each step gates the next: stop at the first one that is wrong.
|
||||
not a regression; it means no replay task was ever created. For the same
|
||||
reason neither counter is asserted by the local validation harness.
|
||||
|
||||
16. **The node has peers and a UNL, yet never declares a ledger validated —
|
||||
is it slow, stuck, or is only publishing behind?**
|
||||
This is the last stage of the pipeline and the one every step above
|
||||
assumes away. Steps 1 to 15 all ask whether ledger data arrives and gets
|
||||
applied; this step asks whether the ledgers the node already holds ever
|
||||
pass the **quorum gate**, and whether the ones that pass ever reach
|
||||
clients. It is the step for the specific symptom **"peers are connected,
|
||||
the trusted list is loaded, acquisition looks healthy, and
|
||||
`server_state` still never becomes `full`."**
|
||||
- **Slow or stuck?** Panel _Trusted Validations vs Quorum Target_
|
||||
(`ledger_quorum_publish`, `metric=trusted_validation_tally` against
|
||||
`quorum_target`). Both are snapshots of the most recent gate
|
||||
evaluation, recorded whether it passed or failed, so a node that keeps
|
||||
failing still reports both numbers — which is exactly what separates
|
||||
the two cases:
|
||||
- **Tally climbing toward the target** — slow, not stuck. Validations
|
||||
are accumulating and the gate will pass. Keep waiting; nothing here
|
||||
needs fixing.
|
||||
- **Tally flat below the target** — **stuck.** Validations arrive and
|
||||
never reach quorum, so this node can hold every ledger it needs and
|
||||
still never declare one validated. Two causes: too few trusted
|
||||
validators are reachable, or the UNL / negative-UNL configuration
|
||||
excludes the ones that are. Go to Bootstrap step 4 (_UNL Trusted Keys
|
||||
vs Quorum_ and _UNL Quorum Headroom_) — a trusted list that cannot
|
||||
satisfy quorum makes every panel in this row look starved as a
|
||||
consequence, so do not chase them.
|
||||
- **Target reading about 9.2e18** (signed 64-bit maximum) — the
|
||||
explicit **quorum-disabled** sentinel. Too many list publishers are
|
||||
unavailable, so the trusted list switched quorum off entirely rather
|
||||
than merely setting it high, and the node can never validate however
|
||||
far the tally climbs. The value is reported as that maximum rather
|
||||
than being allowed to wrap to -1, precisely so it cannot be misread
|
||||
as a target the tally already exceeds. Fix publisher reachability
|
||||
first; the key count is irrelevant until quorum is enabled again.
|
||||
- **Both series flat at 0** — the gate has never been evaluated at all,
|
||||
so nothing has yet been offered for validation. That is an upstream
|
||||
problem, not a quorum one: go back to the Bootstrap row.
|
||||
One reading caveat that matters more here than anywhere else in this
|
||||
row: judge the tally by its **sustained floor over minutes**, never by
|
||||
a single sample. Each series is a snapshot of the last evaluation, and
|
||||
the first evaluation of every round runs before peer validations for
|
||||
that ledger arrive, so a healthy node sawtooths.
|
||||
- **Is the gate actually rejecting?** Panel _Pre-Accept Quorum Shortfall
|
||||
Rate_ (`ledger_quorum_shortfall_total`, `stage=pre_accept`). Read this
|
||||
one carefully, because **a non-zero rate is not by itself a fault.**
|
||||
Consensus issues this node's own validation and evaluates the gate
|
||||
immediately, before its peers' validations for that same ledger have
|
||||
arrived, so the first evaluation of each round tallies short and is
|
||||
retried as validations come in. A healthy cluster emits this counter
|
||||
every round. The fault signature is the **combination**: this rate
|
||||
climbing well above the ledger-close rate while the tally above stays
|
||||
flat below its target and _Time to First Validated Ledger_ stays at
|
||||
zero. That is the retry loop never converging, rather than merely
|
||||
running early.
|
||||
- **Did it ever validate at all?** Panel _Time to First Validated Ledger_
|
||||
(`ledger_quorum_publish`, `metric=time_to_first_validated_us`, shown in
|
||||
seconds). A one-shot measurement, read exactly like _Time to First
|
||||
FULL_ in step 1: a value means the node got there and this is how long
|
||||
it took, a flat zero means it never has. Reading the two together is
|
||||
what pins down the fault — a value on _Time to First FULL_ beside a
|
||||
zero here means the node reached the `full` server state but has still
|
||||
never fully validated a ledger, which points at the quorum gate rather
|
||||
than at acquisition.
|
||||
- **Validation fine, publishing behind?** Panel _Publish Lag (validated
|
||||
minus published)_ (`ledger_quorum_publish`, `metric=publish_lag`). This
|
||||
is the separate question, and the one no other panel can answer: the
|
||||
published sequence was never exported before, so this gap was not
|
||||
derivable from any other series. Publishing trails validation by
|
||||
design, so a small lag that drains each round is normal.
|
||||
- **Lag flat at 0 or 1** — healthy.
|
||||
- **Lag positive and growing** — validation is healthy and the
|
||||
**publish pipeline is not.** The node itself is current while its
|
||||
clients and subscriptions see stale data. This is a local processing
|
||||
fault, not a peer or quorum one, so go back to steps 3, 9 and 10:
|
||||
a starved job queue or a stalling main loop is the usual cause.
|
||||
- **Lag flat at 0 on a node that has never validated** — not healthy,
|
||||
merely empty. There is nothing validated to publish, so read
|
||||
_Trusted Validations vs Quorum Target_ first and ignore this panel
|
||||
until the gate passes.
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
| Scenario | Recommendation |
|
||||
|
||||
Reference in New Issue
Block a user