diff --git a/.github/scripts/otel-naming/README.md b/.github/scripts/otel-naming/README.md index 50051e8e31..a3aa8a5e12 100644 --- a/.github/scripts/otel-naming/README.md +++ b/.github/scripts/otel-naming/README.md @@ -56,7 +56,7 @@ hardcoded allowlist: | D | Every dashboard label resolves to an L1 span attribute, a native-metric label (L6, emitted by MetricsRegistry), or a Prometheus/Grafana builtin. TraceQL scope prefixes (`span.`/`resource.`/…) are stripped before the L1 lookup. | | E | No dotted `xrpl..` attribute key in the runbook (only the L1 resource attrs `xrpl.network.*` may be dotted). Span names, filenames, OTel-standard keys, and metric labels are not flagged. | | I | No string literals as **metric** instrument names or label keys — the mirror of Rule F. Applies to the name passed to an `XRPL_METRIC_*` macro or a `meter->Create*` factory and to the label _keys_ in its label set. Label _values_, descriptions, `*MetricNames.h`, `MetricMacros.h` and test files are exempt. Scoped by metric **family** (first underscore segment): declaring a constant opts that family in, so the metric surface can be converted subsystem by subsystem. Unconverted families warn as Rule L. | -| J | Metric instrument names follow the suffix conventions: `lower_snake_case`, no `xrpld_`/`xrpl_` prefix (the exporter adds it), a counter ends `_total`, a histogram ends `_us`/`_ms`/`_seconds`, a gauge does not end `_total`. The instrument **kind** is read from the emit site, never guessed from words in the name — so a multi-series gauge carrying units in its label values (e.g. `nodestore_latency` observing `write_mean_us`) is not a violation. | +| J | Metric instrument names follow the suffix conventions: `lower_snake_case`, no `xrpld_`/`xrpl_` prefix (the exporter adds it), a counter ends `_total`, a histogram ends `_us`/`_ms`/`_seconds`, a gauge does not end `_total`. The instrument **kind** is read from the emit site, never guessed from words in the name — so a multi-series gauge carrying units in its label values (e.g. `nodestore_state` observing `write_mean_us`) is not a violation. | | K | Every metric named in `docker/telemetry/workload/expected_metrics.json` resolves to a declared constant, so a rename in code cannot leave the workload validator asserting a name nothing emits. PromQL selectors (`m{label="v"}`) and exporter-appended histogram suffixes (`_bucket`/`_count`/`_sum`) are normalized away first; groups fed by another emit path (`statsd_gauges`, `statsd_counters`, `spanmetrics`) are out of scope by design. | Rule F runs **unconditionally** (it is a purely syntactic check on the diff --git a/.github/scripts/otel-naming/check_otel_naming.py b/.github/scripts/otel-naming/check_otel_naming.py index 685d7489c4..05442a89a0 100644 --- a/.github/scripts/otel-naming/check_otel_naming.py +++ b/.github/scripts/otel-naming/check_otel_naming.py @@ -1263,9 +1263,9 @@ def instrument_kinds(root: Path, wire_by_symbol: Dict[str, str]) -> Dict[str, st The kind is what decides which suffix is correct, so it must be read from the emit site rather than guessed from the name -- guessing from words like - "latency" mislabels a multi-series gauge whose units live in its label - VALUES (e.g. `nodestore_latency` observing `write_mean_us`), which is a - legitimate shape, not a violation. + "latency" or "us" mislabels a multi-series gauge whose units live in its + label VALUES (e.g. `nodestore_state` observing `write_mean_us`), which is + a legitimate shape, not a violation. Returns one of `counter`, `histogram`, `gauge`, `updown` per wire name. A name whose emit site is not found is absent from the result, so Rule J diff --git a/.github/scripts/otel-naming/test_check_otel_naming.py b/.github/scripts/otel-naming/test_check_otel_naming.py index 6a61ae2c1e..6a1dec5fe4 100644 --- a/.github/scripts/otel-naming/test_check_otel_naming.py +++ b/.github/scripts/otel-naming/test_check_otel_naming.py @@ -1184,14 +1184,14 @@ class RuleJMetricSuffixes(unittest.TestCase): [], ) - def test_gauge_named_latency_is_not_flagged(self): + def test_gauge_with_unit_bearing_label_values_is_not_flagged(self): # The regression this rule's kind-awareness exists for: a multi-series - # GAUGE whose units live in its label VALUES (nodestore_latency + # GAUGE whose units live in its label VALUES (nodestore_state # observing write_mean_us) must not be read as a mis-suffixed duration. self.assertEqual( self._run( - _mc("nodestoreLatency", "nodestore_latency"), - 'meter_->CreateInt64ObservableGauge(metric::nodestoreLatency, "d");\n', + _mc("nodestoreState", "nodestore_state"), + 'meter_->CreateInt64ObservableGauge(metric::nodestoreState, "d");\n', ), [], ) diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md index f9e53c9a08..10ff0cbfc3 100644 --- a/OpenTelemetryPlan/09-data-collection-reference.md +++ b/OpenTelemetryPlan/09-data-collection-reference.md @@ -1674,59 +1674,59 @@ no panel (it is read in Tempo instead). /proc/self/statm kernel path the heap-trim RSS readings are taken from, so it likewise cannot be respelled. --> -| Signal | Type | Emit site | Panel | Meaning | -| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `dns_resolve_total` (`outcome` = `resolved` \| `empty`) | counter | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Outcome Rate | Peer hostname resolutions. `empty` means a configured bootstrap or `[ips_fixed]` name returned no address, so that peer is never dialled. | -| `dns_resolve_latency_ms` | histogram | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Latency (p95) | Time to resolve a configured peer hostname. Seconds-scale values mean the resolver is timing out ahead of every dial. | -| `overlay_connect_total` (`outcome` = `connected` \| `tcp_fail` \| `tls_fail` \| `upgrade_fail` \| `timeout`) | counter | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Outcome Rate | Outbound peer connection attempts by terminal outcome. The outcome names the stage that broke: TCP, TLS, HTTP upgrade, or no terminal state in time. | -| `overlay_dial_latency_ms` | histogram | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Latency (p95) | Time from starting an outbound dial to its terminal outcome, successes and failures together. A p95 near the dial timeout means peers accept TCP but never finish the handshake. | -| `handshake_negotiation_fail_total` (`reason`, 14 values incl. `wrong_network`, `invalid_network_id`, `clock_skew`, `self_connection`, `session_verify_failed`) | counter | `Handshake.cpp` — `throwNegotiationFailure` (from `verifyHandshake`) | Handshake Negotiation Failures by Reason | Peer handshakes rejected after TLS while checking network id, clock, keys and addresses. `reason` names the failing check. | -| `unl_fetch_total` (`site` = configured UNL URI; `outcome` = the 9 `ListDisposition` strings `accepted` \| `expired` \| `same_sequence` \| `pending` \| `known_sequence` \| `unsupported_version` \| `untrusted` \| `stale` \| `invalid`, plus `fetch_error` \| `bad_status` \| `parse_error`) | counter | `ValidatorSite.cpp` — `ValidatorSite::reportFetchOutcome` | UNL Fetch Rate by Site & Outcome | Validator-list fetches per site. `accepted` is the only success; `same_sequence` and `known_sequence` are normal no-op refreshes; the three literals are transport or content faults. | -| `unl_quorum` (`metric` = `trusted_keys` \| `quorum` \| `quorum_disabled`) | observable gauge | `MetricsRegistry.cpp` — `registerUnlQuorumGauge` | UNL Trusted Keys vs Quorum; UNL Quorum Headroom | Trusted UNL key count against the validations a ledger needs. `trusted_keys` at or below `quorum` means the node can never declare a ledger validated. | -| `clock_close_offset_seconds` (`metric` = `offset`) | observable gauge | `MetricsRegistry.cpp` — `registerClockSkewGauge` | Clock Close Offset | Network close time offset from the local clock. Negative means the local clock runs ahead. `server_info` only surfaces `close_time_offset` at 60 s or more, so this gauge sees skew far earlier. | -| `state_changes_total` (`from`, `to` = `disconnected` \| `connected` \| `syncing` \| `tracking` \| `full`) | counter | `NetworkOPs.cpp` — `NetworkOPsImp::setMode` | Mode Transitions by Edge | Operating-mode transitions keyed on the (`from`, `to`) edge. The edge is what separates a clean `disconnected`→`connected`→`syncing`→`tracking`→`full` climb from `full`→`connected` flapping; an unlabelled total cannot tell them apart. | -| `sync_state` (`metric` = `initial_full_duration_us`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Time to First FULL | Microseconds from process start to the first `full` transition, sourced from `NetworkOPs::getInitialSyncDurationUs()`. Stays 0 until `full` is reached, so a flat 0 is itself the "never synced" signal; once set it never changes. | -| `sync_state` (`metric` = `network_ledger_gate`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Network Ledger Gate | 1 while the node is still waiting to see a full network ledger (`NetworkOPs::isNeedNetworkLedger()`), else 0. A persistent 1 blocks transaction submission and `full`, whatever the rest of the pipeline shows. | -| `sync_state` (`metric` = `server_stall_seconds`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Server Stall | Current main-loop stall duration from `LoadManager::getCurrentStallSeconds()`, 0 when healthy. Same duration the load monitor logs as "Server stalled for N seconds", which previously existed only in that log line. | -| `sync_state` (`metric` = `ledgers_behind`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Ledgers Behind Network | Peer-reported network tip minus our validated sequence, floored at 0 (`NetworkOPs::getLedgersBehindNetwork()`). Reads each peer's already cached ledger range, so no new network round trip. | -| `server_stall_events_total` | observable counter | `MetricsRegistry.cpp` — `registerStallEventsCounter` | Server Stall Event Rate | Distinct stall episodes since process start, counted once per episode rather than per stalled second. A rising rate is repeated fresh stalls; a flat rate with a large `server_stall_seconds` is one long stall. | -| `sync_acquire` (`metric` = `missing_state_nodes_max` \| `missing_tx_nodes_max`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Missing SHAMap Nodes per Acquire (state/tx) | Largest outstanding SHAMap node count across in-flight acquires, split by tree, from the count `getMissingNodes()` already produces during its sweep (`InboundLedger.cpp` — `InboundLedger::trigger`). **The headline stuck-sync signal:** flat and non-zero across ticks means the acquire will never finish; shrinking means slow but alive. Aggregated as a max rather than labelled per ledger, because a `ledger_seq` label would mint one series per ledger acquired — per-ledger identity stays on the `ledger.acquire` span. | -| `sync_acquire` (`metric` = `received_data_depth`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Peer packets stashed across all in-flight acquires waiting to be applied, summed because it measures one shared processing backlog. A growing depth means arriving node data outpaces processing, so the limit is the job queue or disk rather than peer supply. | -| `sync_acquire` (`metric` = `in_flight`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Number of ledger acquires currently running. Exported so the three values above can be read in context: all zero with `in_flight` zero is an idle node, not a healthy one. | -| `shamap_cache_hit_rate` (`metric` = `treenode`) | observable gauge | `MetricsRegistry.cpp` — `registerCacheHitRateDetailGauge` | SHAMap TreeNode Cache Hit Rate | Share of SHAMap tree-node lookups served from memory, from the previously-uncalled `TaggedCache::getHitRate()`, normalized from 0-100 to 0.0-1.0. Distinct from `nodestore_state`-derived NuDB Cache Hit Ratio on the Ledger Data Sync dashboard: this is the in-memory layer **above** the node store, so a miss here is what causes a read there. The full-below cache is not reported — it is a `KeyCache` whose only lookup path increments `stats_.hits`/`stats_.misses` while `getHitRate()` reads the separate `hits_`/`misses_` members, so its rate is hard-wired to 0 until that accounting is fixed. | -| `sync_acquire_no_progress_total` | counter | `InboundLedger.cpp` — `InboundLedger::onTimer` | Acquire Stall Rate (no progress) | Acquire timeouts where not one new node arrived since the previous timeout, from the `progress_` flag that was previously log-only. Fires on the 3 s acquire timer, never per node. A sustained rate together with a flat missing-node count is the definitive "stuck, not slow" signature. | -| `sync_addnode_total` (`outcome` = `good` \| `duplicate` \| `invalid`) | counter | `InboundLedger.cpp` — `InboundLedger::recordBatchOutcome` | Add-Node Outcomes | SHAMap nodes received during acquire, split by result. Emitted once per received packet from the aggregated batch tally the trace log already printed — never inside the per-node `receiveNode()` loop. Separates real progress (`good`) from wasted bandwidth (`duplicate`) and a misbehaving peer (`invalid`), all three of which look like healthy throughput in traffic metrics. | -| `sync_acquire_source_total` (`source` = `local` \| `network`) | counter | `InboundLedger.cpp` — `InboundLedger::init` | Acquire Source (local vs network) | Whether an acquire was satisfied entirely from the local node store or needed peers, emitted once per new acquire after the first local lookup. Sustained `network` on a node that should already hold the range means sync is disk-bound rather than peer-bound. | -| `jobq_saturation` (`metric` = `running_tasks` \| `worker_threads` \| `total_waiting`) | observable gauge | `MetricsRegistry.cpp` — `registerJobQueueSaturationGauge` | Worker Pool Saturation; Worker Pool Capacity & Total Backlog | Global worker-pool saturation from `JobQueue::getWorkerSaturation()`: tasks in flight, threads the pool is configured to run, and jobs queued across all types, all from one reading so the ratio and the backlog describe the same instant. `worker_threads` is exported rather than hardcoded in the dashboard because it is derived at startup from `[workers]`, node size and hardware concurrency. Exists separately from the per-job-type gauges `JobQueue::collect()` publishes (`jobq__waiting` / `_running` / `_deferred`) because a pool-wide slowdown otherwise appears as an independent fault in every subsystem queued behind it; a `running_tasks / worker_threads` ratio at 1.0 **with** a non-zero `total_waiting` attributes it to pool exhaustion once. | -| `peer_ledger_supply` (`metric` = `peers_reporting` \| `peers_serving_validated` \| `peers_serving_next` \| `supply_min_seq` \| `supply_max_seq`) | observable gauge | `MetricsRegistry.cpp` — `registerPeerLedgerSupplyGauge` (aggregating `OverlayImpl::getPeerLedgerSupply`) | Peers Able to Serve Needed Sequence; Peer Supply Window Margin (history headroom vs tip gap) | How much of the sequence range this node needs its connected peer set can actually serve, from one pass over the active peers reading the range each already advertised in `mtSTATUS_CHANGE`. **`peers_serving_next` is the signal this exists for:** zero there with a non-zero `peers_reporting` means no connected peer holds validated + 1, so the peer set must change and waiting cannot finish the sync. `peers_reporting` is the denominator that makes the rest readable — peers advertising `[0, 0]` have not reported yet and are excluded from every field, so they cannot make a healthy peer set appear to serve from genesis; when nothing has reported, both window fields read 0 meaning **unknown**, not genesis. `supply_min_seq` / `supply_max_seq` separate "asking for history nobody kept" from "asking for a tip nobody reached". The _Peer Supply Window Margin_ panel renders both as distances from `server_info{metric="validated_ledger_seq"}` rather than as absolute sequences, because the raw values sit around 1.05e8 and roughly 3e5 apart, so one linear axis flattens the tip movement that shows whether sync is progressing; the subtraction also makes zero the boundary in both directions. Both operands are gated `> 0` in PromQL so the unknown-window sentinel cannot turn into a whole-sequence-space spike when subtracted. Distinct from `server_info{metric="peers"}`, a bare connection count with no notion of what those peers hold; from `sync_state{metric="ledgers_behind"}`, which uses the same per-peer maxima but collapses them to a single distance-to-tip number that cannot say how many peers can serve that distance or whether the range has a hole; and from `peer_quality{metric="peers_insane_count"}`, which counts peers on a different chain and is therefore a correctness signal, not an availability one. | -| `peer_disconnect_total` (`reason` = `graceful` \| `shutdown` \| `stopping` \| `read_error` \| `write_error` \| `timer_error` \| `ping_timeout` \| `not_useful` \| `large_sendq` \| `charge_resources` \| `malformed_handshake` \| `shared_value` \| `unknown`; `direction` = `inbound` \| `outbound`) | counter | `PeerImp.cpp` — `PeerImp::close` | Peer Disconnects by Reason | Peer teardowns split by cause and by which side opened the connection. Emitted once per teardown at `close()`, the single funnel every disconnect path passes through, and `close()` already self-guards on the socket being open, so a repeated close cannot double-count and the total matches the existing unlabelled tally. `reason` is set by whichever site decided to disconnect, first writer wins, so a later generic reason never masks the real one; the value is always one of a fixed set of literals in `PeerImp.cpp`, never peer-supplied data, so cardinality is bounded by the code. The split is the whole point: it separates our-fault backpressure (`large_sendq`, `charge_resources`) from topology and network faults (`not_useful`, `ping_timeout`, `read_error`), and normal churn (`graceful`) from either. Distinct from the existing `server_info{metric="peer_disconnects_resources"}`, which counts only the resource-charge subset and carries no labels, and from the StatsD `overlay_peer_disconnects`, which is the unlabelled grand total in which every reason above collapses into one number. | -| `peer_accept_total` (`outcome` = `accepted` \| `local_endpoint_fail` \| `resource_limit` \| `no_slot` \| `not_peer_request` \| `protocol_mismatch` \| `bad_cookie` \| `slot_refused` \| `handshake_error`) | counter | `OverlayImpl.cpp` — `OverlayImpl::onHandoff` via `reportAcceptOutcome` | Inbound Peer Accept Outcomes | Terminal outcome of every inbound connection this node is offered, one emit per handoff. `accepted` is reported only after `run()`, so anything that threw on the way lands on `handshake_error` instead; the two early returns that are not peer attempts at all (a handled HTTP request, and a request that never asked to upgrade) are deliberately not counted. The `outcome` names the stage that refused: no local endpoint, the resource manager, PeerFinder having no slot or seeing a duplicate, a non-peer upgrade request, protocol version disagreement, a bad security cookie, or activation being refused. This is the **inbound twin** of the existing `overlay_connect_total{outcome}`, which covers outbound dials only; without it a node refusing every inbound connection is indistinguishable from one nobody dials, and reading the two together gives the full in/out split. | -| `peerfinder_slot_census` (`metric` = `out_active` \| `out_max` \| `in_active` \| `in_max` \| `connecting` \| `fixed_configured` \| `fixed_active` \| `bootcache` \| `livecache`) | observable gauge | `MetricsRegistry.cpp` — `registerSlotCensusGauge` (from `Logic::getSlotCensus`) | PeerFinder Slot Census; PeerFinder Address Caches & Fixed Peers | Slot occupancy against capacity, outbound dials in flight, configured-versus-connected fixed peers, and the depth of both address caches. All nine come from a single acquire of the PeerFinder lock, so they are mutually consistent, share one label set and can be compared against each other. That is what makes the three most common bootstrap failures visible: `connecting` non-zero while `out_active` stays below `out_max` (dials starting and never completing), `bootcache` and `livecache` both at 0 (nothing to dial at all), and `fixed_active` below `fixed_configured` (a peer named in the configuration is unreachable). `fixed_configured` is the count of peers named in the config, so the pair an operator reads is "how many did I ask for" against "how many do I have" — the same comparison `autoconnect()` makes. All nine values already existed inside PeerFinder; only two of them were exported, as the legacy beast::insight gauges `peer_finder_active_inbound_peers` and `peer_finder_active_outbound_peers`. Those two carry no capacity, attempt or cache term, are read at unrelated instants, and so cannot be joined with each other let alone with a capacity term — leaving all three failures above indistinguishable from a node that is simply not dialling. | -| `serve_refused_total` (`request` = `ledger` \| `txset` \| `object` \| `fetchpack`; `reason` = `sendq_full` \| `load_shed` \| `not_found` \| `no_map` \| `bad_type` \| `empty_reply`) | counter | `PeerImp.cpp` — `processLedgerRequest`, `onMessage(TMGetObjectByHash)`, `doFetchPack` | Ledger/Object Serve Refusals | Peer data requests this node declined to answer, split by what was asked for and why. This is the **supply side** of the sync exchange — what this node refuses to serve OTHERS — and nothing equivalent existed before, so a node shedding every ledger request looked identical to one being asked for nothing. `sendq_full` and `load_shed` are self-inflicted backpressure (the send queue at `Tuning::kDropSendQueue`, or the local fee track loaded, or too many pack jobs queued), while `not_found` is a genuine history gap and `no_map` / `bad_type` / `empty_reply` mean the request was answerable in principle but produced nothing to send. `fetchpack` is counted apart from `ledger` because a fetch pack is how a syncing peer catches up in bulk and its shed threshold is a different one. Emitted at most once per request — `empty_reply` is reported after the node loop, never inside it — and both labels are code literals, so cardinality is bounded at compile time. | -| `amendment_block` (`metric` = `warned` \| `seconds_to_block`) | observable gauge | `MetricsRegistry.cpp` — `registerAmendmentBlockGauge` | Amendment Block Countdown; Amendment Warned | `warned` is 1 once an unsupported amendment has reached majority (`NetworkOPs::isAmendmentWarned()`, previously only an admin-only `server_info` warning). **`seconds_to_block` is the leading indicator:** seconds until that amendment activates, from `AmendmentTable::firstUnsupportedExpected()` against the network close time. It reads `-1` when nothing is pending — a distinct healthy value rather than a missing series, matching the sentinel `validator_health{metric="unl_expiry_days"}` already uses — and is clamped at 0 rather than going negative, because past-due means the block is imminent, not overdue by some amount worth charting; the subtraction is done in `std::int64_t` so a past-due activation cannot wrap. Amendment-blocked is a terminal sync blocker: the node stops validating and never resumes without a software upgrade. The existing `validator_health{metric="amendment_blocked"}` reports that state after the fact, when nothing can be done about it; this gauge is the window before it, which is the only actionable part. The blocking amendment's identity is deliberately **not** a label — the network can vote on an arbitrary 256-bit amendment id, not drawn from this build's known features, so an id label would be unbounded cardinality and would mint a permanent new series per amendment. The id is available in logs from `AmendmentTableImpl::doValidatedLedger` ("Unsupported amendment \ reached majority at ..."), correlated to this series by node and time. | -| `ledger_jump_total` | counter | `NetworkOPs.cpp` — `NetworkOPsImp::switchLastClosedLedger` | Byzantine Ledger Jumps | Forced jumps of the last closed ledger onto a divergent chain: the node was told the network's LCL is not the one it built on and discarded its own chain tip to follow. Nothing equivalent existed — this was log-only ("JUMP last closed ledger to ..."), so a node repeatedly thrashing between chains left no time series to correlate against the rest of the sync pipeline. Any non-zero rate is abnormal by construction; repeated jumps are wrong-chain thrash, which points at the peer set and the configured network id rather than anywhere in the acquire pipeline. Deliberately unlabelled: the ledger hash and sequence would both be unbounded as label values, and the log line beside the emit already carries them. | -| `nodestore_latency` (`metric` = `write_mean_us` \| `read_mean_us` \| `write_count` \| `read_count` \| `write_duration_us` \| `read_duration_us`) | observable gauge | `MetricsRegistry.cpp` — `registerNodeStoreLatencyGauge` | NodeStore Write vs Read Latency (us/op); NodeStore Operation Rate (writes vs reads) | Mean microseconds per node-store store and per fetch, with both operation counts so a panel can divide the two rates and read _interval_ latency instead of the since-boot average. **The write side is the new signal.** `storeDurationUs_` was declared in `Database.h` and never written, and no accessor existed, so no write-path latency was observable anywhere; the read total was already exposed as `nodestore_state{metric="node_reads_duration_us"}`. This is the fingerprint of the "a node with a large existing DB syncs slower than a fresh one" symptom, which is write-bound and therefore invisible in every read-side metric. Chosen as a gauge over a histogram deliberately: a histogram gives true percentiles but costs one `Record()` per node object on the store/fetch path, and a single ledger write walks thousands of SHAMap nodes — this gauge instead reads four existing atomics once per ~10 s tick and adds nothing to the hot path. Consequence: **p99 is not obtainable from this signal**, and a histogram added later would also need an explicit-bucket View (`addMicrosecondHistogramView`) because the SDK default buckets top out at 10,000. Distinct from the Ledger Data Sync dashboard's NuDB Read Latency panel, which divides two `nodestore_state` fields in PromQL: that panel has no write-duration input to divide, because the quantity did not exist. **Known gap:** `write_mean_us` is emitted only when the store-duration total is non-zero, and that total is fed by `Database::recordStoreDuration`, today called only from `Database::importInternal` (the `[import_db]` admin path). `Database::store()` is pure virtual and neither `DatabaseNodeImp::store` nor `DatabaseRotatingImp::store` times itself yet, so an ordinary node reports `write_count` with no `write_mean_us`. The mean is omitted rather than reported as 0 so the gap stays visible instead of reading as "writes are instantaneous". | -| `sweep_malloc_trim_us` | histogram | `Application.cpp` — `ApplicationImp::trimHeapAndRecord` | Sweep Heap-Trim Duration (p50/p95) | Wall-clock duration of the `malloc_trim` call that ends every cache sweep. **This is the leading explanation for "a node with a large existing DB syncs slower than a fresh one" on glibc:** the trim runs after EVERY sweep, its cost scales with the resident heap, and the pages it hands back must be re-faulted as the caches refill. The numbers all already existed on `MallocTrimReport`, but were unreachable twice over — the whole measurement block sat inside `if (journal.debug())` in `MallocTrim.cpp`, so an ordinary node at default log level measured nothing, and the return value was then discarded at the call site. The gate now covers only the `JLOG`; measuring costs about 6 µs (two `/proc/self/statm` reads at ~2.8 µs and two `getrusage` calls at ~0.17 µs) against a trim that costs milliseconds on a large heap, at a cadence of `SizedItem::SweepInterval` (10-120 s by node size) — a duty cycle below 1e-6 %, so keeping the RSS read debug-only would only have preserved the blind spot. Needs an explicit-bucket View (`addMicrosecondHistogramView`) because a trim on a multi-gigabyte heap runs well past the SDK default ceiling of 10,000, which would collapse exactly the slow trims this signal exists to catch into one saturated bucket. | -| `sweep_malloc_trim_minor_faults_total` | counter | `Application.cpp` — `ApplicationImp::trimHeapAndRecord` | Sweep Heap-Trim Faults & Reclaim Rate | Minor page faults taken _inside_ the trim call, from the `getrusage(RUSAGE_THREAD)` delta the report already carried. **Honest limitation, and it must not be over-claimed:** the delta is scoped to the trim call only, so it proves the trim itself faults — it does NOT prove the trim causes the faults taken later, as the caches refill and touch the pages the trim returned. That later re-fault cost is the actual mechanism the hypothesis proposes and it is NOT measured by this counter. Read the duration against sweep-job queueing rather than treating this number as the total cost of trimming. Emitted only when the delta is above zero: a trim that faulted nothing publishes no series, because a zero would read as "measured, and free" when the honest statement is that there was nothing to fault on. | -| `sweep_malloc_trim_reclaimed_kb_total` | counter | `Application.cpp` — `ApplicationImp::trimHeapAndRecord` | Sweep Heap-Trim Faults & Reclaim Rate | Resident kilobytes the trim actually returned to the kernel, so the cost above can be judged against what it bought. Cumulative and sign-corrected: `MallocTrimReport::deltaKB()` is after-minus-before, so a successful trim is NEGATIVE and the emit site publishes its magnitude. A sweep across which RSS GREW — another thread allocating faster than the trim released — is dropped rather than negated, because a counter cannot decrease and there is no reclaim of a negative size. Zero reclaim beside a non-zero duration is the worst reading: the trim is walking the heap and freeing nothing, which is pure cost. | -| `rotation_state` (`metric` = `in_flight` \| `copy_forward`) | observable gauge | `MetricsRegistry.cpp` — `registerRotationStateGauge` | Online-Delete Rotation Window & Copy-Forward Writes | The online-delete rotation window, and the running total of the extra writes it forces. A rotation rewrites into the new backend any node body the doomed archive serves, which is I/O an ordinary fetch would never perform and which scales with the archive — so it appears only on a populated, already-rotated database, which is precisely why it never shows on a fresh node. `copy_forward` comes from `DatabaseRotatingImp::copyForwardCount_`, which existed but was log-only AND reset by `rotate()` on every swap; a series that drops to zero per rotation cannot be rated, so a second never-reset total was added beside it and this gauge reads that. `in_flight` is exposed because the extra writes only happen inside that window, so a panel needs to know when to expect the total to move; the same total climbing while the flag reads 0 would mean the flag leaked, not that rotation is cheap. Polled rather than pushed because `DatabaseRotatingImp` is libxrpl and cannot include `xrpld/telemetry` — the two readings are taken through new `DatabaseRotating` accessors from the same collection tick pattern `registerNodeStoreGauge` already uses. **Publishes NO series when `online_delete` is not configured** (the `dynamic_cast` to `DatabaseRotating` fails and the callback returns early), deliberately: an absent series means "rotation is not configured", which a zero would misreport as "rotation is free". Rotation _duration_ is deliberately not recorded — see the note below the table. | -| `rotation_copy_node_restore_total` | counter | `SHAMapStoreImp.cpp` — `SHAMapStoreImp::copyNode` | Rotation Node Re-Store Rate | Nodes the rotation had to rescue because they were present in NEITHER backend, re-stored from the in-memory state map. The genuinely unmeasured rotation write: each is an extra store on top of the whole-state-map walk the rotation already performs, and the branch was warn-log-only, so the volume was invisible unless someone was reading logs. A non-zero rate says more than cost — it says an earlier rotation removed the only on-disk copy of a clean node (`cowid == 0`, so `flushDirty` skips it) that the current validated state map still reaches, and without the rescue it would later surface as an unresolvable `SHAMapMissingNode`. The node hash is deliberately not a label: it is unbounded runtime data that would mint one series per rescued node. Correlate a spike with the `copyNode` warning line in Loki, by node and time. | -| `ledger_replay_fallback_total` (`stage` = `skiplist` \| `delta`) | counter | `SkipListAcquire.cpp` / `LedgerDeltaAcquire.cpp` — `trigger` | Replay Fallback to Full Acquire (by stage) | A ledger-replay sub-task abandoning its shortcut and acquiring the whole ledger through `InboundLedger` instead, because too few connected peers support the `LedgerReplay` protocol feature. Both branches were debug-log-only, so a silently defeated replay optimisation left no metric at all — back-fill simply ran on the slower path with nothing to show why. Emitted once, on the transition into fallback, not at the acquire call, which re-runs on every later trigger. The `stage` label separates the skip-list acquire (which fetches the list of historical ledger hashes) from the per-ledger delta acquire, because they fail independently. | -| `ledger_replay_outcome_total` (`outcome` = `success` \| `timeout` \| `build_failed` \| `parameter_failed`) | counter | `LedgerReplayTask.cpp` — `LedgerReplayTask::recordOutcome` | Replay Outcomes (by terminal state) | Terminal state of every ledger-replay task, one emit per task. Every terminal path previously only set an internal `complete_`/`failed_` flag and wrote a log line, so a replay that never succeeded was indistinguishable from one that was never attempted. The outcome names the layer at fault: `timeout` means the deltas never arrived (a peer-supply problem), `build_failed` means a delta would not apply to its parent, and `parameter_failed` means a peer served a skip list inconsistent with what the task asked for — the latter two are data faults, not slowness. Read with `ledger_replay_fallback_total`: fallbacks rising while successes stay flat is replay-based catch-up degrading to full-ledger acquisition. | -| `ledger_quorum_publish` (`metric` = `trusted_validation_tally` \| `quorum_target`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Trusted Validations vs Quorum Target | Trusted validations counted at the most recent pre-accept gate, beside the number that gate required. Snapshotted in `LedgerMaster::checkAccept` before the shortfall check, so a node that keeps failing the gate still reports both numbers — which is the whole point: the tally alone cannot say whether validations are accumulating toward quorum (slow, will finish) or plateaued below it (stuck). Read the sustained floor of the tally, not a single sample: each series is a snapshot of the last evaluation, and the first evaluation of each round runs before peer validations arrive, so a healthy node sawtooths. `quorum_target` is what the gate actually demanded, as opposed to `unl_quorum{metric="quorum"}` which is what the trusted list configures. When the trusted list disables quorum entirely (`getNeededValidations` returns `SIZE_MAX`) the target is reported as int64 max rather than wrapping to -1, so it reads far above any tally instead of inverting the comparison — the same sentinel handling as the `unl_quorum` gauge. | -| `ledger_quorum_publish` (`metric` = `publish_lag`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Publish Lag (validated minus published) | Ledgers fully validated but not yet published to clients and subscribers: the validated sequence minus the published sequence, floored at zero. `pubLedgerSeq_` was never exported, so this gap was not derivable from any other series. Publishing trails validation by design and a small lag drains each round; a lag that stays positive or grows means validation is healthy and the publish pipeline is not, which is a distinct fault from anything the quorum or acquire signals can show. The two sequences are read as independent relaxed loads, so a sample taken mid-update may be off by one ledger for one poll — immaterial for a lag trend, and the price of not taking the LedgerMaster mutex on the metrics poll thread. | -| `ledger_quorum_publish` (`metric` = `time_to_first_validated_us`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Time to First Validated Ledger | Microseconds from process start until the first ledger passed the pre-accept quorum gate. A one-shot measurement like `sync_state{metric="initial_full_duration_us"}`: written once under `mutex_` and never changed, so it has no trend. Exactly two readings are meaningful — a duration, meaning the node reached its first fully-validated ledger and this is how long that took, or 0, meaning it never has. Clamped to a minimum of 1 so a genuine sub-microsecond reading can never be confused with the never-reached zero. A value here alongside a zero on time-to-first-FULL, or the reverse, separates "reached the full server state" from "fully validated a ledger". | -| `ledger_quorum_shortfall_total` (`stage` = `pre_accept`) | counter | `LedgerMaster.cpp` — `LedgerMaster::checkAccept` | Pre-Accept Quorum Shortfall Rate | One increment per pre-accept gate evaluation rejected because the trusted validation tally was below quorum. Previously trace-log-only, which made a node that peers and receives validations yet never validates indistinguishable from an idle one. A non-zero rate is **not** by itself a fault: `doAccept` issues this node's own validation and calls `consensusBuilt` → `checkAccept` immediately, before peer validations for that ledger arrive, so the first evaluation of every round tallies short and is retried as validations come in — a healthy cluster emits this counter every round. The fault signature is the rate climbing well above the ledger-close rate while the tally stays flat below its target and `time_to_first_validated_us` stays at 0. Emitted while `mutex_` is held, which is safe against the metrics poll because every accessor the sync gauges read is a lock-free atomic load, so no OTel callback ever acquires `mutex_`. | -| `consensus_round_duration_ms` | histogram | `RCLConsensus.cpp` — `RCLConsensus::Adaptor::makeAcceptSpan` | Consensus Round Duration Distribution; Consensus Round Duration (p50/p95) | Wall-clock duration of a completed consensus round, in milliseconds. Promotes the long-standing `round_time_ms` span attribute into a native instrument: the attribute answers "how long did THIS round take" inside a trace, next to the proposers and disputes that explain it, while the histogram gives the distribution over time, which is what an alert or SLO panel needs and what a raw trace query cannot cheaply produce fleet-wide. Being native it is also **never sampled**, so it stays complete when tracing is head-sampled down. Recorded at exactly one site — `makeAcceptSpan` is the single function both the synchronous (`onForceAccept`) and asynchronous (`onAccept`) accept paths call once per round — so it can neither double-count nor be skipped, and it adds no per-peer, per-proposal or per-transaction work. **Explicit buckets** are registered for it in `MetricsRegistry::initExporterAndProvider` (`addRoundDurationHistogramView`, boundaries 500 ms → 120 s): the SDK default tops out at 10,000 ms, which would collapse every slow round into one saturated bucket and read every quantile as 10 s, and the consensus parameters themselves allow a round up to `ledgerAbandonConsensus` = 120 s. Needs **no collector change** — a native metric rides the existing OTLP → Prometheus path. | -| `consensus.validation.accept` (`validation_status`, `accept_gated`, `ledger_hash`, `ledger_seq`, `full_validation`) | span + span attr | `RCLValidations.cpp` — `handleNewValidation` | Trusted Validation Accept Rate by Status | One span per **trusted** validation as it reaches the ledger-acceptance gate, so its rate is bounded by the UNL size per ledger close (untrusted validations cannot move acceptance and get no span). Its trace id is derived from the **validated ledger's** hash, so it joins that ledger's trace rather than the round trace — see the per-ledger trace join below. `validation_status` is one of the six `ValStatus` values and only `current` continues to the gate, which is the difference between a node whose arriving validations are counting and one whose validations are all rejected; from the outside both look like a node that receives validations and never validates. `accept_gated` is true when another thread was already accepting the same ledger, which is why a trace can show a validation with no `ledger.validate` after it. Both are spanmetrics dimensions in **both** collector configs (6 and 2 values, bounded); `ledger_hash` / `ledger_seq` stay span-only and Tempo-indexed, since a per-ledger metric dimension would mint one series per ledger. | -| Per-ledger trace join (`ledger_hash` as trace-id seed) | trace scheme | `LedgerMaster.h/.cpp` — `LedgerMaster::makeLedgerTraceSpan` | n/a — read in Tempo, `{span.ledger_hash="LEDGER_HASH"}` | Makes one slow ledger readable as **one connected trace** instead of a set of orphan spans on different threads. `ledger.validate` (`LedgerMaster::checkAccept`), `ledger.store` (`LedgerMaster::storeLedger`) and `consensus.validation.accept` (`handleNewValidation`) each derive their trace id from the **same 32-byte ledger hash** via `SpanGuard::hashSpan`, which seeds the trace id from `hash[0:16]`. Nothing is propagated between the threads: every one of those sites already holds the ledger hash, which is the whole reason the key was chosen — the same pattern the apply pipeline uses to join `tx.preflight` / `tx.preclaim` / `tx.transactor` on the transaction id (`libxrpl/tx/applySteps.cpp`). Each span is a **true root** (deterministic trace id, empty parent), so the ledger's spans are siblings in one trace rather than a parent/child chain, which is the honest shape: none causes another directly and their order varies with the sync path (`checkAccept` is entered from a peer thread via `handleNewValidation`, from the acquire-completion job, and from the consensus thread via `switchLCL`). The full hash is also recorded as the `ledger_hash` attribute — it is what an operator searches by, and it is how a reader confirms two spans are genuinely the same ledger rather than a trace-id coincidence, since the trace id is only the leading 16 bytes. Asserted end-to-end by the `trace_join_groups` block in `expected_spans.json` (`assert_trace_join_groups` in `validate_telemetry.py`), which fails CI if the members stop sharing a trace. | -| `ledger.acquire` span (`outcome` = `complete` \| `failed` \| `abandoned`; `acquire_reason`, `timeouts`, `peer_count`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::init` / `InboundLedger::finalizeAcquireSpan` | Ledger Acquire Phase Duration (p95 by phase) (its three phase children); Ledger Acquire Duration (Inbound Fetch) and Ledger Acquire Rate by Outcome, both on the **node-health** board | Parent of the three phase spans: one whole fetch of one missing ledger, from the first request to the terminal state. Pre-existing since Phase 6, extended here with `ledger_hash` (set at `init()`, so a fetch that never finishes is still findable in a trace search, and it is the trace-id seed that joins this span to the `ledger.validate`, `ledger.store` and `consensus.validation.accept` spans for the same ledger) and with the fourth `outcome` value `abandoned`, recorded when the acquire is destroyed by a sweep or shutdown before reaching a result. Without `abandoned` a stuck-then-swept fetch left the span with no `outcome` at all, so it vanished from every outcome rate — the exact failure a stalled fresh sync produces. `ledger_seq` is re-stamped at the end because a by-hash acquire starts with `seq_ == 0` and learns the sequence only when the header arrives. | -| `ledger.acquire.header` span (`outcome`, `timed_out`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::syncPhaseSpans` / `endPhaseSpan` | Ledger Acquire Phase Duration (p95 by phase); Ledger Acquire Phase Outcomes (by phase & timeout) | Child of `ledger.acquire` covering the wait for the ledger header, which gates both tree phases — until it arrives the account-state and transaction root hashes are unknown, so nothing else can even be requested. The parent span is flat and its duration is dominated by the state tree, so a node stuck waiting to be TOLD what to fetch was indistinguishable from one stuck fetching it. No `missing_nodes`: a header is a single object, not a tree. Opened and closed by one idempotent state sync over the `have*_` flags rather than by open/close calls scattered through the fetch code, so the span boundary cannot drift out of step with the real phase boundary. | -| `ledger.acquire.astree` span (`outcome`, `timed_out`, `missing_nodes`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::syncPhaseSpans` / `endPhaseSpan` | Ledger Acquire Phase Duration (p95 by phase); Ledger Acquire Phase Outcomes (by phase & timeout) | Child of `ledger.acquire` covering the account-state SHAMap fetch — **nearly all of the work in a real fresh sync**, and the reason the phase split exists: the flat parent span could not separate it from the small transaction tree. `missing_nodes` is read from the count `getMissingNodes()` already produced during its sweep, never recomputed, so no second tree walk is added. `outcome=timeout` together with a non-zero `missing_nodes` is the "peers are not serving this tree" signature; `timed_out` is a separate dimension from `outcome` because a phase can time out and still be retried by its parent acquire. | -| `ledger.acquire.txtree` span (`outcome`, `timed_out`, `missing_nodes`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::syncPhaseSpans` / `endPhaseSpan` | Ledger Acquire Phase Duration (p95 by phase); Ledger Acquire Phase Outcomes (by phase & timeout) | Child of `ledger.acquire` covering the transaction SHAMap fetch. Usually completes long before the account-state phase, and that asymmetry is the point of separating them: the parent span's duration is the state tree's, not this one's, so a transaction tree that is genuinely slow is invisible inside it. Closed the moment its own tree completes (from `receiveNode`, `trigger` or `takeHeader`), so its duration is the real fetch time rather than stretching to the next trigger. | -| `txset.acquire` span (`outcome`, `txset_hash`, `duration_ms`, `timeouts`, `peer_count`) | span | `TransactionAcquire.cpp` — `TransactionAcquire::finalizeAcquireSpan` | Tx-Set Acquire Outcomes; Tx-Set Acquire Duration (p95) | One attempt to fetch the transaction set a consensus proposal referenced but this node did not hold. `TransactionAcquire` had **zero** telemetry of any kind before this, so a consensus round stalled waiting on a set was indistinguishable from an idle one. The sibling of `ledger.acquire`: same `TimeoutCounter` base, same trigger/onTimer/takeNodes shape, and the same `trace_ledger` flag so the two halves of a stuck sync cannot be enabled apart. `outcome` is `complete` \| `failed` \| `timeout` \| `abandoned`, stamped on both exits (`done()`, and the destructor when the round sweep in `InboundTransactions::newRound` drops a set that never arrived) by one idempotent finalizer. `timeout` is distinct from `failed` because the exhausted-budget path sets the terminal `failed_` flag too — that flag is how the timer loop stops — so the outcome rule checks the timeout first or every timeout would read as a data fault. `txset_hash` identifies WHICH set stalled and stays span-only: one metric series per consensus round would be unbounded. | -| `ledger.serve` span (`object_type`, `outcome`, `served_nodes`, `peer_id`, `ledger_seq`) | span | `PeerImp.cpp` — `PeerImp::processLedgerRequest` (the `JtLedgerReq` worker) | Ledger Serve Rate by Object Type | This node answering a peer's `TMGetLedger` request — the **supply side** of the sync exchange, and the trace-level companion to the existing `serve_refused_total` counter. The whole serve path had no span, so how long this node takes to answer, and whether it answered at all, was unobservable. A fresh trace root, because the request arrives from the wire on a shared worker whose ambient span is unrelated. `object_type` (`header` \| `tx` \| `as` \| `txset`) and `outcome` (`complete` \| `partial` \| `refused`) are both derived by shared rules in `LedgerSpanNames.h` rather than named per branch, which is what stops the eight exits of `processLedgerRequest` disagreeing about one request. `outcome` is derived from the reply itself — `served_nodes` is the reply's own node count and is 0 on all seven refusal paths — so nothing is accumulated and no work is added to the per-node assembly loop. `partial` means the reply hit `Tuning::kSoftMaxReplyNodes`, so the peer must make another round trip. | -| `peer.dial` span (`outcome`, `remote_endpoint`, `duration_ms`) | span | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Outcomes (span-derived, per attempt) | One outbound connect attempt, as a per-attempt timeline rather than a rate. The trace-level companion to `overlay_connect_total` / `overlay_dial_latency_ms`: it carries the same five `outcome` values, set from the same `reportOutcome` funnel, so span and counter cannot disagree, and the funnel's existing first-call-wins guard makes the span exactly-once for free. What it adds is `remote_endpoint` — WHICH peer — which the counter deliberately cannot carry, because one series per peer address would be unbounded cardinality; it is a dedicated Tempo span column instead. A fresh trace root: a dial is the first thing a starting node does, so there is nothing to parent it to. An attempt torn down mid-dial by shutdown ends its span in the destructor with no `outcome`, which is the honest record of "never concluded" rather than a dropped span. | +| Signal | Type | Emit site | Panel | Meaning | +| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `dns_resolve_total` (`outcome` = `resolved` \| `empty`) | counter | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Outcome Rate | Peer hostname resolutions. `empty` means a configured bootstrap or `[ips_fixed]` name returned no address, so that peer is never dialled. | +| `dns_resolve_latency_ms` | histogram | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Latency (p95) | Time to resolve a configured peer hostname. Seconds-scale values mean the resolver is timing out ahead of every dial. | +| `overlay_connect_total` (`outcome` = `connected` \| `tcp_fail` \| `tls_fail` \| `upgrade_fail` \| `timeout`) | counter | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Outcome Rate | Outbound peer connection attempts by terminal outcome. The outcome names the stage that broke: TCP, TLS, HTTP upgrade, or no terminal state in time. | +| `overlay_dial_latency_ms` | histogram | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Latency (p95) | Time from starting an outbound dial to its terminal outcome, successes and failures together. A p95 near the dial timeout means peers accept TCP but never finish the handshake. | +| `handshake_negotiation_fail_total` (`reason`, 14 values incl. `wrong_network`, `invalid_network_id`, `clock_skew`, `self_connection`, `session_verify_failed`) | counter | `Handshake.cpp` — `throwNegotiationFailure` (from `verifyHandshake`) | Handshake Negotiation Failures by Reason | Peer handshakes rejected after TLS while checking network id, clock, keys and addresses. `reason` names the failing check. | +| `unl_fetch_total` (`site` = configured UNL URI; `outcome` = the 9 `ListDisposition` strings `accepted` \| `expired` \| `same_sequence` \| `pending` \| `known_sequence` \| `unsupported_version` \| `untrusted` \| `stale` \| `invalid`, plus `fetch_error` \| `bad_status` \| `parse_error`) | counter | `ValidatorSite.cpp` — `ValidatorSite::reportFetchOutcome` | UNL Fetch Rate by Site & Outcome | Validator-list fetches per site. `accepted` is the only success; `same_sequence` and `known_sequence` are normal no-op refreshes; the three literals are transport or content faults. | +| `unl_quorum` (`metric` = `trusted_keys` \| `quorum` \| `quorum_disabled`) | observable gauge | `MetricsRegistry.cpp` — `registerUnlQuorumGauge` | UNL Trusted Keys vs Quorum; UNL Quorum Headroom | Trusted UNL key count against the validations a ledger needs. `trusted_keys` at or below `quorum` means the node can never declare a ledger validated. | +| `clock_close_offset_seconds` (`metric` = `offset`) | observable gauge | `MetricsRegistry.cpp` — `registerClockSkewGauge` | Clock Close Offset | Network close time offset from the local clock. Negative means the local clock runs ahead. `server_info` only surfaces `close_time_offset` at 60 s or more, so this gauge sees skew far earlier. | +| `state_changes_total` (`from`, `to` = `disconnected` \| `connected` \| `syncing` \| `tracking` \| `full`) | counter | `NetworkOPs.cpp` — `NetworkOPsImp::setMode` | Mode Transitions by Edge | Operating-mode transitions keyed on the (`from`, `to`) edge. The edge is what separates a clean `disconnected`→`connected`→`syncing`→`tracking`→`full` climb from `full`→`connected` flapping; an unlabelled total cannot tell them apart. | +| `sync_state` (`metric` = `initial_full_duration_us`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Time to First FULL | Microseconds from process start to the first `full` transition, sourced from `NetworkOPs::getInitialSyncDurationUs()`. Stays 0 until `full` is reached, so a flat 0 is itself the "never synced" signal; once set it never changes. | +| `sync_state` (`metric` = `network_ledger_gate`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Network Ledger Gate | 1 while the node is still waiting to see a full network ledger (`NetworkOPs::isNeedNetworkLedger()`), else 0. A persistent 1 blocks transaction submission and `full`, whatever the rest of the pipeline shows. | +| `sync_state` (`metric` = `server_stall_seconds`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Server Stall | Current main-loop stall duration from `LoadManager::getCurrentStallSeconds()`, 0 when healthy. Same duration the load monitor logs as "Server stalled for N seconds", which previously existed only in that log line. | +| `sync_state` (`metric` = `ledgers_behind`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Ledgers Behind Network | Peer-reported network tip minus our validated sequence, floored at 0 (`NetworkOPs::getLedgersBehindNetwork()`). Reads each peer's already cached ledger range, so no new network round trip. | +| `server_stall_events_total` | observable counter | `MetricsRegistry.cpp` — `registerStallEventsCounter` | Server Stall Event Rate | Distinct stall episodes since process start, counted once per episode rather than per stalled second. A rising rate is repeated fresh stalls; a flat rate with a large `server_stall_seconds` is one long stall. | +| `sync_acquire` (`metric` = `missing_state_nodes_max` \| `missing_tx_nodes_max`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Missing SHAMap Nodes per Acquire (state/tx) | Largest outstanding SHAMap node count across in-flight acquires, split by tree, from the count `getMissingNodes()` already produces during its sweep (`InboundLedger.cpp` — `InboundLedger::trigger`). **The headline stuck-sync signal:** flat and non-zero across ticks means the acquire will never finish; shrinking means slow but alive. Aggregated as a max rather than labelled per ledger, because a `ledger_seq` label would mint one series per ledger acquired — per-ledger identity stays on the `ledger.acquire` span. | +| `sync_acquire` (`metric` = `received_data_depth`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Peer packets stashed across all in-flight acquires waiting to be applied, summed because it measures one shared processing backlog. A growing depth means arriving node data outpaces processing, so the limit is the job queue or disk rather than peer supply. | +| `sync_acquire` (`metric` = `in_flight`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Number of ledger acquires currently running. Exported so the three values above can be read in context: all zero with `in_flight` zero is an idle node, not a healthy one. | +| `shamap_cache_hit_rate` (`metric` = `treenode`) | observable gauge | `MetricsRegistry.cpp` — `registerCacheHitRateDetailGauge` | SHAMap TreeNode Cache Hit Rate | Share of SHAMap tree-node lookups served from memory, from the previously-uncalled `TaggedCache::getHitRate()`, normalized from 0-100 to 0.0-1.0. Distinct from `nodestore_state`-derived NuDB Cache Hit Ratio on the Ledger Data Sync dashboard: this is the in-memory layer **above** the node store, so a miss here is what causes a read there. The full-below cache is not reported — it is a `KeyCache` whose only lookup path increments `stats_.hits`/`stats_.misses` while `getHitRate()` reads the separate `hits_`/`misses_` members, so its rate is hard-wired to 0 until that accounting is fixed. | +| `sync_acquire_no_progress_total` | counter | `InboundLedger.cpp` — `InboundLedger::onTimer` | Acquire Stall Rate (no progress) | Acquire timeouts where not one new node arrived since the previous timeout, from the `progress_` flag that was previously log-only. Fires on the 3 s acquire timer, never per node. A sustained rate together with a flat missing-node count is the definitive "stuck, not slow" signature. | +| `sync_addnode_total` (`outcome` = `good` \| `duplicate` \| `invalid`) | counter | `InboundLedger.cpp` — `InboundLedger::recordBatchOutcome` | Add-Node Outcomes | SHAMap nodes received during acquire, split by result. Emitted once per received packet from the aggregated batch tally the trace log already printed — never inside the per-node `receiveNode()` loop. Separates real progress (`good`) from wasted bandwidth (`duplicate`) and a misbehaving peer (`invalid`), all three of which look like healthy throughput in traffic metrics. | +| `sync_acquire_source_total` (`source` = `local` \| `network`) | counter | `InboundLedger.cpp` — `InboundLedger::init` | Acquire Source (local vs network) | Whether an acquire was satisfied entirely from the local node store or needed peers, emitted once per new acquire after the first local lookup. Sustained `network` on a node that should already hold the range means sync is disk-bound rather than peer-bound. | +| `jobq_saturation` (`metric` = `running_tasks` \| `worker_threads` \| `total_waiting`) | observable gauge | `MetricsRegistry.cpp` — `registerJobQueueSaturationGauge` | Worker Pool Saturation; Worker Pool Capacity & Total Backlog | Global worker-pool saturation from `JobQueue::getWorkerSaturation()`: tasks in flight, threads the pool is configured to run, and jobs queued across all types, all from one reading so the ratio and the backlog describe the same instant. `worker_threads` is exported rather than hardcoded in the dashboard because it is derived at startup from `[workers]`, node size and hardware concurrency. Exists separately from the per-job-type gauges `JobQueue::collect()` publishes (`jobq__waiting` / `_running` / `_deferred`) because a pool-wide slowdown otherwise appears as an independent fault in every subsystem queued behind it; a `running_tasks / worker_threads` ratio at 1.0 **with** a non-zero `total_waiting` attributes it to pool exhaustion once. | +| `peer_ledger_supply` (`metric` = `peers_reporting` \| `peers_serving_validated` \| `peers_serving_next` \| `supply_min_seq` \| `supply_max_seq`) | observable gauge | `MetricsRegistry.cpp` — `registerPeerLedgerSupplyGauge` (aggregating `OverlayImpl::getPeerLedgerSupply`) | Peers Able to Serve Needed Sequence; Peer Supply Window Margin (history headroom vs tip gap) | How much of the sequence range this node needs its connected peer set can actually serve, from one pass over the active peers reading the range each already advertised in `mtSTATUS_CHANGE`. **`peers_serving_next` is the signal this exists for:** zero there with a non-zero `peers_reporting` means no connected peer holds validated + 1, so the peer set must change and waiting cannot finish the sync. `peers_reporting` is the denominator that makes the rest readable — peers advertising `[0, 0]` have not reported yet and are excluded from every field, so they cannot make a healthy peer set appear to serve from genesis; when nothing has reported, both window fields read 0 meaning **unknown**, not genesis. `supply_min_seq` / `supply_max_seq` separate "asking for history nobody kept" from "asking for a tip nobody reached". The _Peer Supply Window Margin_ panel renders both as distances from `server_info{metric="validated_ledger_seq"}` rather than as absolute sequences, because the raw values sit around 1.05e8 and roughly 3e5 apart, so one linear axis flattens the tip movement that shows whether sync is progressing; the subtraction also makes zero the boundary in both directions. Both operands are gated `> 0` in PromQL so the unknown-window sentinel cannot turn into a whole-sequence-space spike when subtracted. Distinct from `server_info{metric="peers"}`, a bare connection count with no notion of what those peers hold; from `sync_state{metric="ledgers_behind"}`, which uses the same per-peer maxima but collapses them to a single distance-to-tip number that cannot say how many peers can serve that distance or whether the range has a hole; and from `peer_quality{metric="peers_insane_count"}`, which counts peers on a different chain and is therefore a correctness signal, not an availability one. | +| `peer_disconnect_total` (`reason` = `graceful` \| `shutdown` \| `stopping` \| `read_error` \| `write_error` \| `timer_error` \| `ping_timeout` \| `not_useful` \| `large_sendq` \| `charge_resources` \| `malformed_handshake` \| `shared_value` \| `unknown`; `direction` = `inbound` \| `outbound`) | counter | `PeerImp.cpp` — `PeerImp::close` | Peer Disconnects by Reason | Peer teardowns split by cause and by which side opened the connection. Emitted once per teardown at `close()`, the single funnel every disconnect path passes through, and `close()` already self-guards on the socket being open, so a repeated close cannot double-count and the total matches the existing unlabelled tally. `reason` is set by whichever site decided to disconnect, first writer wins, so a later generic reason never masks the real one; the value is always one of a fixed set of literals in `PeerImp.cpp`, never peer-supplied data, so cardinality is bounded by the code. The split is the whole point: it separates our-fault backpressure (`large_sendq`, `charge_resources`) from topology and network faults (`not_useful`, `ping_timeout`, `read_error`), and normal churn (`graceful`) from either. Distinct from the existing `server_info{metric="peer_disconnects_resources"}`, which counts only the resource-charge subset and carries no labels, and from the StatsD `overlay_peer_disconnects`, which is the unlabelled grand total in which every reason above collapses into one number. | +| `peer_accept_total` (`outcome` = `accepted` \| `local_endpoint_fail` \| `resource_limit` \| `no_slot` \| `not_peer_request` \| `protocol_mismatch` \| `bad_cookie` \| `slot_refused` \| `handshake_error`) | counter | `OverlayImpl.cpp` — `OverlayImpl::onHandoff` via `reportAcceptOutcome` | Inbound Peer Accept Outcomes | Terminal outcome of every inbound connection this node is offered, one emit per handoff. `accepted` is reported only after `run()`, so anything that threw on the way lands on `handshake_error` instead; the two early returns that are not peer attempts at all (a handled HTTP request, and a request that never asked to upgrade) are deliberately not counted. The `outcome` names the stage that refused: no local endpoint, the resource manager, PeerFinder having no slot or seeing a duplicate, a non-peer upgrade request, protocol version disagreement, a bad security cookie, or activation being refused. This is the **inbound twin** of the existing `overlay_connect_total{outcome}`, which covers outbound dials only; without it a node refusing every inbound connection is indistinguishable from one nobody dials, and reading the two together gives the full in/out split. | +| `peerfinder_slot_census` (`metric` = `out_active` \| `out_max` \| `in_active` \| `in_max` \| `connecting` \| `fixed_configured` \| `fixed_active` \| `bootcache` \| `livecache`) | observable gauge | `MetricsRegistry.cpp` — `registerSlotCensusGauge` (from `Logic::getSlotCensus`) | PeerFinder Slot Census; PeerFinder Address Caches & Fixed Peers | Slot occupancy against capacity, outbound dials in flight, configured-versus-connected fixed peers, and the depth of both address caches. All nine come from a single acquire of the PeerFinder lock, so they are mutually consistent, share one label set and can be compared against each other. That is what makes the three most common bootstrap failures visible: `connecting` non-zero while `out_active` stays below `out_max` (dials starting and never completing), `bootcache` and `livecache` both at 0 (nothing to dial at all), and `fixed_active` below `fixed_configured` (a peer named in the configuration is unreachable). `fixed_configured` is the count of peers named in the config, so the pair an operator reads is "how many did I ask for" against "how many do I have" — the same comparison `autoconnect()` makes. All nine values already existed inside PeerFinder; only two of them were exported, as the legacy beast::insight gauges `peer_finder_active_inbound_peers` and `peer_finder_active_outbound_peers`. Those two carry no capacity, attempt or cache term, are read at unrelated instants, and so cannot be joined with each other let alone with a capacity term — leaving all three failures above indistinguishable from a node that is simply not dialling. | +| `serve_refused_total` (`request` = `ledger` \| `txset` \| `object` \| `fetchpack`; `reason` = `sendq_full` \| `load_shed` \| `not_found` \| `no_map` \| `bad_type` \| `empty_reply`) | counter | `PeerImp.cpp` — `processLedgerRequest`, `onMessage(TMGetObjectByHash)`, `doFetchPack` | Ledger/Object Serve Refusals | Peer data requests this node declined to answer, split by what was asked for and why. This is the **supply side** of the sync exchange — what this node refuses to serve OTHERS — and nothing equivalent existed before, so a node shedding every ledger request looked identical to one being asked for nothing. `sendq_full` and `load_shed` are self-inflicted backpressure (the send queue at `Tuning::kDropSendQueue`, or the local fee track loaded, or too many pack jobs queued), while `not_found` is a genuine history gap and `no_map` / `bad_type` / `empty_reply` mean the request was answerable in principle but produced nothing to send. `fetchpack` is counted apart from `ledger` because a fetch pack is how a syncing peer catches up in bulk and its shed threshold is a different one. Emitted at most once per request — `empty_reply` is reported after the node loop, never inside it — and both labels are code literals, so cardinality is bounded at compile time. | +| `amendment_block` (`metric` = `warned` \| `seconds_to_block`) | observable gauge | `MetricsRegistry.cpp` — `registerAmendmentBlockGauge` | Amendment Block Countdown; Amendment Warned | `warned` is 1 once an unsupported amendment has reached majority (`NetworkOPs::isAmendmentWarned()`, previously only an admin-only `server_info` warning). **`seconds_to_block` is the leading indicator:** seconds until that amendment activates, from `AmendmentTable::firstUnsupportedExpected()` against the network close time. It reads `-1` when nothing is pending — a distinct healthy value rather than a missing series, matching the sentinel `validator_health{metric="unl_expiry_days"}` already uses — and is clamped at 0 rather than going negative, because past-due means the block is imminent, not overdue by some amount worth charting; the subtraction is done in `std::int64_t` so a past-due activation cannot wrap. Amendment-blocked is a terminal sync blocker: the node stops validating and never resumes without a software upgrade. The existing `validator_health{metric="amendment_blocked"}` reports that state after the fact, when nothing can be done about it; this gauge is the window before it, which is the only actionable part. The blocking amendment's identity is deliberately **not** a label — the network can vote on an arbitrary 256-bit amendment id, not drawn from this build's known features, so an id label would be unbounded cardinality and would mint a permanent new series per amendment. The id is available in logs from `AmendmentTableImpl::doValidatedLedger` ("Unsupported amendment \ reached majority at ..."), correlated to this series by node and time. | +| `ledger_jump_total` | counter | `NetworkOPs.cpp` — `NetworkOPsImp::switchLastClosedLedger` | Byzantine Ledger Jumps | Forced jumps of the last closed ledger onto a divergent chain: the node was told the network's LCL is not the one it built on and discarded its own chain tip to follow. Nothing equivalent existed — this was log-only ("JUMP last closed ledger to ..."), so a node repeatedly thrashing between chains left no time series to correlate against the rest of the sync pipeline. Any non-zero rate is abnormal by construction; repeated jumps are wrong-chain thrash, which points at the peer set and the configured network id rather than anywhere in the acquire pipeline. Deliberately unlabelled: the ledger hash and sequence would both be unbounded as label values, and the log line beside the emit already carries them. | +| `nodestore_state` (`metric` = `write_mean_us` \| `read_mean_us` \| `node_writes` \| `node_reads_total` \| `node_writes_duration_us` \| `node_reads_duration_us`) | observable gauge | `MetricsRegistry.cpp` — `registerNodeStoreGauge` (`observeNodeStoreTotals`) | NodeStore Write vs Read Latency (us/op); NodeStore Operation Rate (writes vs reads) | The store/fetch latency half of the pre-existing `nodestore_state` gauge (its I/O counters and queue depth are tabled under NodeStore I/O above). Mean microseconds per node-store store and per fetch, with both operation counts and both cumulative duration totals so a panel can divide the two _rates_ and read interval latency instead of the since-boot average. **The write side is the signal.** `storeDurationUs_` was declared in `Database.h` and never written, and no accessor existed, so no write-path latency was observable anywhere; the read total was already exposed as `nodestore_state{metric="node_reads_duration_us"}`. This is the fingerprint of the "a node with a large existing DB syncs slower than a fresh one" symptom, which is write-bound and therefore invisible in every read-side metric. All three concrete store paths now time themselves through `Database::recordStoreDuration` (`DatabaseNodeImp::store`, `DatabaseRotatingImp::store` and `Database::importInternal`), so `write_mean_us` is live on an ordinary node rather than only on the `[import_db]` admin path. Chosen as a gauge over a histogram deliberately: a histogram gives true percentiles but costs one `Record()` per node object on the store/fetch path, and a single ledger write walks thousands of SHAMap nodes — this gauge instead reads existing atomics once per ~10 s tick and adds nothing to the hot path. Consequence: **p99 is not obtainable from this signal**, and a histogram added later would also need an explicit-bucket View (`addMicrosecondHistogramView`) because the SDK default buckets top out at 10,000. Related to the Ledger Data Sync dashboard's NuDB Read Latency panel, which divides the two read-side fields in PromQL; that panel predates the write numerator and has no write-side equivalent. Each mean is computed by `MetricsRegistry::scaledMean`, which saturates at `INT64_MAX` rather than wrapping and returns no value when its count is 0 — so the series is omitted instead of reporting a misleading 0 µs, while the four totals are always observed. A short-lived duplicate `nodestore_latency` gauge published the same six values from the same accessors and has been retired. | +| `sweep_malloc_trim_us` | histogram | `Application.cpp` — `ApplicationImp::trimHeapAndRecord` | Sweep Heap-Trim Duration (p50/p95) | Wall-clock duration of the `malloc_trim` call that ends every cache sweep. **This is the leading explanation for "a node with a large existing DB syncs slower than a fresh one" on glibc:** the trim runs after EVERY sweep, its cost scales with the resident heap, and the pages it hands back must be re-faulted as the caches refill. The numbers all already existed on `MallocTrimReport`, but were unreachable twice over — the whole measurement block sat inside `if (journal.debug())` in `MallocTrim.cpp`, so an ordinary node at default log level measured nothing, and the return value was then discarded at the call site. The gate now covers only the `JLOG`; measuring costs about 6 µs (two `/proc/self/statm` reads at ~2.8 µs and two `getrusage` calls at ~0.17 µs) against a trim that costs milliseconds on a large heap, at a cadence of `SizedItem::SweepInterval` (10-120 s by node size) — a duty cycle below 1e-6 %, so keeping the RSS read debug-only would only have preserved the blind spot. Needs an explicit-bucket View (`addMicrosecondHistogramView`) because a trim on a multi-gigabyte heap runs well past the SDK default ceiling of 10,000, which would collapse exactly the slow trims this signal exists to catch into one saturated bucket. | +| `sweep_malloc_trim_minor_faults_total` | counter | `Application.cpp` — `ApplicationImp::trimHeapAndRecord` | Sweep Heap-Trim Faults & Reclaim Rate | Minor page faults taken _inside_ the trim call, from the `getrusage(RUSAGE_THREAD)` delta the report already carried. **Honest limitation, and it must not be over-claimed:** the delta is scoped to the trim call only, so it proves the trim itself faults — it does NOT prove the trim causes the faults taken later, as the caches refill and touch the pages the trim returned. That later re-fault cost is the actual mechanism the hypothesis proposes and it is NOT measured by this counter. Read the duration against sweep-job queueing rather than treating this number as the total cost of trimming. Emitted only when the delta is above zero: a trim that faulted nothing publishes no series, because a zero would read as "measured, and free" when the honest statement is that there was nothing to fault on. | +| `sweep_malloc_trim_reclaimed_kb_total` | counter | `Application.cpp` — `ApplicationImp::trimHeapAndRecord` | Sweep Heap-Trim Faults & Reclaim Rate | Resident kilobytes the trim actually returned to the kernel, so the cost above can be judged against what it bought. Cumulative and sign-corrected: `MallocTrimReport::deltaKB()` is after-minus-before, so a successful trim is NEGATIVE and the emit site publishes its magnitude. A sweep across which RSS GREW — another thread allocating faster than the trim released — is dropped rather than negated, because a counter cannot decrease and there is no reclaim of a negative size. Zero reclaim beside a non-zero duration is the worst reading: the trim is walking the heap and freeing nothing, which is pure cost. | +| `rotation_state` (`metric` = `in_flight` \| `copy_forward`) | observable gauge | `MetricsRegistry.cpp` — `registerRotationStateGauge` | Online-Delete Rotation Window & Copy-Forward Writes | The online-delete rotation window, and the running total of the extra writes it forces. A rotation rewrites into the new backend any node body the doomed archive serves, which is I/O an ordinary fetch would never perform and which scales with the archive — so it appears only on a populated, already-rotated database, which is precisely why it never shows on a fresh node. `copy_forward` comes from `DatabaseRotatingImp::copyForwardCount_`, which existed but was log-only AND reset by `rotate()` on every swap; a series that drops to zero per rotation cannot be rated, so a second never-reset total was added beside it and this gauge reads that. `in_flight` is exposed because the extra writes only happen inside that window, so a panel needs to know when to expect the total to move; the same total climbing while the flag reads 0 would mean the flag leaked, not that rotation is cheap. Polled rather than pushed because `DatabaseRotatingImp` is libxrpl and cannot include `xrpld/telemetry` — the two readings are taken through new `DatabaseRotating` accessors from the same collection tick pattern `registerNodeStoreGauge` already uses. **Publishes NO series when `online_delete` is not configured** (the `dynamic_cast` to `DatabaseRotating` fails and the callback returns early), deliberately: an absent series means "rotation is not configured", which a zero would misreport as "rotation is free". Rotation _duration_ is deliberately not recorded — see the note below the table. | +| `rotation_copy_node_restore_total` | counter | `SHAMapStoreImp.cpp` — `SHAMapStoreImp::copyNode` | Rotation Node Re-Store Rate | Nodes the rotation had to rescue because they were present in NEITHER backend, re-stored from the in-memory state map. The genuinely unmeasured rotation write: each is an extra store on top of the whole-state-map walk the rotation already performs, and the branch was warn-log-only, so the volume was invisible unless someone was reading logs. A non-zero rate says more than cost — it says an earlier rotation removed the only on-disk copy of a clean node (`cowid == 0`, so `flushDirty` skips it) that the current validated state map still reaches, and without the rescue it would later surface as an unresolvable `SHAMapMissingNode`. The node hash is deliberately not a label: it is unbounded runtime data that would mint one series per rescued node. Correlate a spike with the `copyNode` warning line in Loki, by node and time. | +| `ledger_replay_fallback_total` (`stage` = `skiplist` \| `delta`) | counter | `SkipListAcquire.cpp` / `LedgerDeltaAcquire.cpp` — `trigger` | Replay Fallback to Full Acquire (by stage) | A ledger-replay sub-task abandoning its shortcut and acquiring the whole ledger through `InboundLedger` instead, because too few connected peers support the `LedgerReplay` protocol feature. Both branches were debug-log-only, so a silently defeated replay optimisation left no metric at all — back-fill simply ran on the slower path with nothing to show why. Emitted once, on the transition into fallback, not at the acquire call, which re-runs on every later trigger. The `stage` label separates the skip-list acquire (which fetches the list of historical ledger hashes) from the per-ledger delta acquire, because they fail independently. | +| `ledger_replay_outcome_total` (`outcome` = `success` \| `timeout` \| `build_failed` \| `parameter_failed`) | counter | `LedgerReplayTask.cpp` — `LedgerReplayTask::recordOutcome` | Replay Outcomes (by terminal state) | Terminal state of every ledger-replay task, one emit per task. Every terminal path previously only set an internal `complete_`/`failed_` flag and wrote a log line, so a replay that never succeeded was indistinguishable from one that was never attempted. The outcome names the layer at fault: `timeout` means the deltas never arrived (a peer-supply problem), `build_failed` means a delta would not apply to its parent, and `parameter_failed` means a peer served a skip list inconsistent with what the task asked for — the latter two are data faults, not slowness. Read with `ledger_replay_fallback_total`: fallbacks rising while successes stay flat is replay-based catch-up degrading to full-ledger acquisition. | +| `ledger_quorum_publish` (`metric` = `trusted_validation_tally` \| `quorum_target`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Trusted Validations vs Quorum Target | Trusted validations counted at the most recent pre-accept gate, beside the number that gate required. Snapshotted in `LedgerMaster::checkAccept` before the shortfall check, so a node that keeps failing the gate still reports both numbers — which is the whole point: the tally alone cannot say whether validations are accumulating toward quorum (slow, will finish) or plateaued below it (stuck). Read the sustained floor of the tally, not a single sample: each series is a snapshot of the last evaluation, and the first evaluation of each round runs before peer validations arrive, so a healthy node sawtooths. `quorum_target` is what the gate actually demanded, as opposed to `unl_quorum{metric="quorum"}` which is what the trusted list configures. When the trusted list disables quorum entirely (`getNeededValidations` returns `SIZE_MAX`) the target is reported as int64 max rather than wrapping to -1, so it reads far above any tally instead of inverting the comparison — the same sentinel handling as the `unl_quorum` gauge. | +| `ledger_quorum_publish` (`metric` = `publish_lag`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Publish Lag (validated minus published) | Ledgers fully validated but not yet published to clients and subscribers: the validated sequence minus the published sequence, floored at zero. `pubLedgerSeq_` was never exported, so this gap was not derivable from any other series. Publishing trails validation by design and a small lag drains each round; a lag that stays positive or grows means validation is healthy and the publish pipeline is not, which is a distinct fault from anything the quorum or acquire signals can show. The two sequences are read as independent relaxed loads, so a sample taken mid-update may be off by one ledger for one poll — immaterial for a lag trend, and the price of not taking the LedgerMaster mutex on the metrics poll thread. | +| `ledger_quorum_publish` (`metric` = `time_to_first_validated_us`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Time to First Validated Ledger | Microseconds from process start until the first ledger passed the pre-accept quorum gate. A one-shot measurement like `sync_state{metric="initial_full_duration_us"}`: written once under `mutex_` and never changed, so it has no trend. Exactly two readings are meaningful — a duration, meaning the node reached its first fully-validated ledger and this is how long that took, or 0, meaning it never has. Clamped to a minimum of 1 so a genuine sub-microsecond reading can never be confused with the never-reached zero. A value here alongside a zero on time-to-first-FULL, or the reverse, separates "reached the full server state" from "fully validated a ledger". | +| `ledger_quorum_shortfall_total` (`stage` = `pre_accept`) | counter | `LedgerMaster.cpp` — `LedgerMaster::checkAccept` | Pre-Accept Quorum Shortfall Rate | One increment per pre-accept gate evaluation rejected because the trusted validation tally was below quorum. Previously trace-log-only, which made a node that peers and receives validations yet never validates indistinguishable from an idle one. A non-zero rate is **not** by itself a fault: `doAccept` issues this node's own validation and calls `consensusBuilt` → `checkAccept` immediately, before peer validations for that ledger arrive, so the first evaluation of every round tallies short and is retried as validations come in — a healthy cluster emits this counter every round. The fault signature is the rate climbing well above the ledger-close rate while the tally stays flat below its target and `time_to_first_validated_us` stays at 0. Emitted while `mutex_` is held, which is safe against the metrics poll because every accessor the sync gauges read is a lock-free atomic load, so no OTel callback ever acquires `mutex_`. | +| `consensus_round_duration_ms` | histogram | `RCLConsensus.cpp` — `RCLConsensus::Adaptor::makeAcceptSpan` | Consensus Round Duration Distribution; Consensus Round Duration (p50/p95) | Wall-clock duration of a completed consensus round, in milliseconds. Promotes the long-standing `round_time_ms` span attribute into a native instrument: the attribute answers "how long did THIS round take" inside a trace, next to the proposers and disputes that explain it, while the histogram gives the distribution over time, which is what an alert or SLO panel needs and what a raw trace query cannot cheaply produce fleet-wide. Being native it is also **never sampled**, so it stays complete when tracing is head-sampled down. Recorded at exactly one site — `makeAcceptSpan` is the single function both the synchronous (`onForceAccept`) and asynchronous (`onAccept`) accept paths call once per round — so it can neither double-count nor be skipped, and it adds no per-peer, per-proposal or per-transaction work. **Explicit buckets** are registered for it in `MetricsRegistry::initExporterAndProvider` (`addRoundDurationHistogramView`, boundaries 500 ms → 120 s): the SDK default tops out at 10,000 ms, which would collapse every slow round into one saturated bucket and read every quantile as 10 s, and the consensus parameters themselves allow a round up to `ledgerAbandonConsensus` = 120 s. Needs **no collector change** — a native metric rides the existing OTLP → Prometheus path. | +| `consensus.validation.accept` (`validation_status`, `accept_gated`, `ledger_hash`, `ledger_seq`, `full_validation`) | span + span attr | `RCLValidations.cpp` — `handleNewValidation` | Trusted Validation Accept Rate by Status | One span per **trusted** validation as it reaches the ledger-acceptance gate, so its rate is bounded by the UNL size per ledger close (untrusted validations cannot move acceptance and get no span). Its trace id is derived from the **validated ledger's** hash, so it joins that ledger's trace rather than the round trace — see the per-ledger trace join below. `validation_status` is one of the six `ValStatus` values and only `current` continues to the gate, which is the difference between a node whose arriving validations are counting and one whose validations are all rejected; from the outside both look like a node that receives validations and never validates. `accept_gated` is true when another thread was already accepting the same ledger, which is why a trace can show a validation with no `ledger.validate` after it. Both are spanmetrics dimensions in **both** collector configs (6 and 2 values, bounded); `ledger_hash` / `ledger_seq` stay span-only and Tempo-indexed, since a per-ledger metric dimension would mint one series per ledger. | +| Per-ledger trace join (`ledger_hash` as trace-id seed) | trace scheme | `LedgerMaster.h/.cpp` — `LedgerMaster::makeLedgerTraceSpan` | n/a — read in Tempo, `{span.ledger_hash="LEDGER_HASH"}` | Makes one slow ledger readable as **one connected trace** instead of a set of orphan spans on different threads. `ledger.validate` (`LedgerMaster::checkAccept`), `ledger.store` (`LedgerMaster::storeLedger`) and `consensus.validation.accept` (`handleNewValidation`) each derive their trace id from the **same 32-byte ledger hash** via `SpanGuard::hashSpan`, which seeds the trace id from `hash[0:16]`. Nothing is propagated between the threads: every one of those sites already holds the ledger hash, which is the whole reason the key was chosen — the same pattern the apply pipeline uses to join `tx.preflight` / `tx.preclaim` / `tx.transactor` on the transaction id (`libxrpl/tx/applySteps.cpp`). Each span is a **true root** (deterministic trace id, empty parent), so the ledger's spans are siblings in one trace rather than a parent/child chain, which is the honest shape: none causes another directly and their order varies with the sync path (`checkAccept` is entered from a peer thread via `handleNewValidation`, from the acquire-completion job, and from the consensus thread via `switchLCL`). The full hash is also recorded as the `ledger_hash` attribute — it is what an operator searches by, and it is how a reader confirms two spans are genuinely the same ledger rather than a trace-id coincidence, since the trace id is only the leading 16 bytes. Asserted end-to-end by the `trace_join_groups` block in `expected_spans.json` (`assert_trace_join_groups` in `validate_telemetry.py`), which fails CI if the members stop sharing a trace. | +| `ledger.acquire` span (`outcome` = `complete` \| `failed` \| `abandoned`; `acquire_reason`, `timeouts`, `peer_count`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::init` / `InboundLedger::finalizeAcquireSpan` | Ledger Acquire Phase Duration (p95 by phase) (its three phase children); Ledger Acquire Duration (Inbound Fetch) and Ledger Acquire Rate by Outcome, both on the **node-health** board | Parent of the three phase spans: one whole fetch of one missing ledger, from the first request to the terminal state. Pre-existing since Phase 6, extended here with `ledger_hash` (set at `init()`, so a fetch that never finishes is still findable in a trace search, and it is the trace-id seed that joins this span to the `ledger.validate`, `ledger.store` and `consensus.validation.accept` spans for the same ledger) and with the fourth `outcome` value `abandoned`, recorded when the acquire is destroyed by a sweep or shutdown before reaching a result. Without `abandoned` a stuck-then-swept fetch left the span with no `outcome` at all, so it vanished from every outcome rate — the exact failure a stalled fresh sync produces. `ledger_seq` is re-stamped at the end because a by-hash acquire starts with `seq_ == 0` and learns the sequence only when the header arrives. | +| `ledger.acquire.header` span (`outcome`, `timed_out`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::syncPhaseSpans` / `endPhaseSpan` | Ledger Acquire Phase Duration (p95 by phase); Ledger Acquire Phase Outcomes (by phase & timeout) | Child of `ledger.acquire` covering the wait for the ledger header, which gates both tree phases — until it arrives the account-state and transaction root hashes are unknown, so nothing else can even be requested. The parent span is flat and its duration is dominated by the state tree, so a node stuck waiting to be TOLD what to fetch was indistinguishable from one stuck fetching it. No `missing_nodes`: a header is a single object, not a tree. Opened and closed by one idempotent state sync over the `have*_` flags rather than by open/close calls scattered through the fetch code, so the span boundary cannot drift out of step with the real phase boundary. | +| `ledger.acquire.astree` span (`outcome`, `timed_out`, `missing_nodes`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::syncPhaseSpans` / `endPhaseSpan` | Ledger Acquire Phase Duration (p95 by phase); Ledger Acquire Phase Outcomes (by phase & timeout) | Child of `ledger.acquire` covering the account-state SHAMap fetch — **nearly all of the work in a real fresh sync**, and the reason the phase split exists: the flat parent span could not separate it from the small transaction tree. `missing_nodes` is read from the count `getMissingNodes()` already produced during its sweep, never recomputed, so no second tree walk is added. `outcome=timeout` together with a non-zero `missing_nodes` is the "peers are not serving this tree" signature; `timed_out` is a separate dimension from `outcome` because a phase can time out and still be retried by its parent acquire. | +| `ledger.acquire.txtree` span (`outcome`, `timed_out`, `missing_nodes`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::syncPhaseSpans` / `endPhaseSpan` | Ledger Acquire Phase Duration (p95 by phase); Ledger Acquire Phase Outcomes (by phase & timeout) | Child of `ledger.acquire` covering the transaction SHAMap fetch. Usually completes long before the account-state phase, and that asymmetry is the point of separating them: the parent span's duration is the state tree's, not this one's, so a transaction tree that is genuinely slow is invisible inside it. Closed the moment its own tree completes (from `receiveNode`, `trigger` or `takeHeader`), so its duration is the real fetch time rather than stretching to the next trigger. | +| `txset.acquire` span (`outcome`, `txset_hash`, `duration_ms`, `timeouts`, `peer_count`) | span | `TransactionAcquire.cpp` — `TransactionAcquire::finalizeAcquireSpan` | Tx-Set Acquire Outcomes; Tx-Set Acquire Duration (p95) | One attempt to fetch the transaction set a consensus proposal referenced but this node did not hold. `TransactionAcquire` had **zero** telemetry of any kind before this, so a consensus round stalled waiting on a set was indistinguishable from an idle one. The sibling of `ledger.acquire`: same `TimeoutCounter` base, same trigger/onTimer/takeNodes shape, and the same `trace_ledger` flag so the two halves of a stuck sync cannot be enabled apart. `outcome` is `complete` \| `failed` \| `timeout` \| `abandoned`, stamped on both exits (`done()`, and the destructor when the round sweep in `InboundTransactions::newRound` drops a set that never arrived) by one idempotent finalizer. `timeout` is distinct from `failed` because the exhausted-budget path sets the terminal `failed_` flag too — that flag is how the timer loop stops — so the outcome rule checks the timeout first or every timeout would read as a data fault. `txset_hash` identifies WHICH set stalled and stays span-only: one metric series per consensus round would be unbounded. | +| `ledger.serve` span (`object_type`, `outcome`, `served_nodes`, `peer_id`, `ledger_seq`) | span | `PeerImp.cpp` — `PeerImp::processLedgerRequest` (the `JtLedgerReq` worker) | Ledger Serve Rate by Object Type | This node answering a peer's `TMGetLedger` request — the **supply side** of the sync exchange, and the trace-level companion to the existing `serve_refused_total` counter. The whole serve path had no span, so how long this node takes to answer, and whether it answered at all, was unobservable. A fresh trace root, because the request arrives from the wire on a shared worker whose ambient span is unrelated. `object_type` (`header` \| `tx` \| `as` \| `txset`) and `outcome` (`complete` \| `partial` \| `refused`) are both derived by shared rules in `LedgerSpanNames.h` rather than named per branch, which is what stops the eight exits of `processLedgerRequest` disagreeing about one request. `outcome` is derived from the reply itself — `served_nodes` is the reply's own node count and is 0 on all seven refusal paths — so nothing is accumulated and no work is added to the per-node assembly loop. `partial` means the reply hit `Tuning::kSoftMaxReplyNodes`, so the peer must make another round trip. | +| `peer.dial` span (`outcome`, `remote_endpoint`, `duration_ms`) | span | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Outcomes (span-derived, per attempt) | One outbound connect attempt, as a per-attempt timeline rather than a rate. The trace-level companion to `overlay_connect_total` / `overlay_dial_latency_ms`: it carries the same five `outcome` values, set from the same `reportOutcome` funnel, so span and counter cannot disagree, and the funnel's existing first-call-wins guard makes the span exactly-once for free. What it adds is `remote_endpoint` — WHICH peer — which the counter deliberately cannot carry, because one series per peer address would be unbounded cardinality; it is a dedicated Tempo span column instead. A fresh trace root: a dial is the first thing a starting node does, so there is nothing to parent it to. An attempt torn down mid-dial by shutdown ends its span in the destructor with no `outcome`, which is the honest record of "never concluded" rather than a dropped span. | ### Why rotation duration is not recorded diff --git a/docker/telemetry/grafana/dashboards/ledger-sync-health.json b/docker/telemetry/grafana/dashboards/ledger-sync-health.json index 71f244a992..d6dec81f37 100644 --- a/docker/telemetry/grafana/dashboards/ledger-sync-health.json +++ b/docker/telemetry/grafana/dashboards/ledger-sync-health.json @@ -3509,7 +3509,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Node-store write latency next to read latency, in microseconds per operation. The write side is the signal: a node with a large existing database back-fills slower than a fresh one, and back-fill is write-bound, so no read-side metric can show it.*\n\n###### How it's computed:\n*nodestore_latency series write_mean_us and read_mean_us, each divided by its own count series so the reading is the latency during the selected interval rather than the average since boot. The write numerator comes from a store-duration total that was declared but never written before this signal existed.*\n\n###### Reading it:\n*Compare the two lines. Reads far above writes points at the read path or a cold cache; writes far above reads points at backend write pressure, which is the large-existing-database case.*\n\n###### Healthy range:\n*Both well under a few hundred microseconds on healthy local storage.*\n\n###### Watch for:\n*A rising write line during history back-fill: the backend cannot absorb writes fast enough and sync will stay slow no matter how many peers are available. Read with Fetch-Pack Peer Starvation to tell a data-supply problem from a disk problem. This is a mean, not a percentile — a tail that matters will move it, but p99 is not available from this signal.*\n\n###### Keywords:\n- **Node-store write latency** *(per node)* — how long the node store takes to persist one object.\n- **Node-store read latency** *(per node)* — how long the node store takes to retrieve one object.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreLatencyGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#node-store-write-latency)", + "description": "###### What this is:\n*Node-store write latency next to read latency, in microseconds per operation. The write side is the signal: a node with a large existing database back-fills slower than a fresh one, and back-fill is write-bound, so no read-side metric can show it.*\n\n###### How it's computed:\n*nodestore_state series node_writes_duration_us and node_reads_duration_us, each divided by its own count series (node_writes, node_reads_total) so the reading is the latency during the selected interval rather than the average since boot. All three concrete store paths time themselves, so the write side is live on an ordinary node.*\n\n###### Reading it:\n*Compare the two lines. Reads far above writes points at the read path or a cold cache; writes far above reads points at backend write pressure, which is the large-existing-database case.*\n\n###### Healthy range:\n*Both well under a few hundred microseconds on healthy local storage.*\n\n###### Watch for:\n*A rising write line during history back-fill: the backend cannot absorb writes fast enough and sync will stay slow no matter how many peers are available. Read with Fetch-Pack Peer Starvation to tell a data-supply problem from a disk problem. This is a mean, not a percentile — a tail that matters will move it, but p99 is not available from this signal.*\n\n###### Keywords:\n- **Node-store write latency** *(per node)* — how long the node store takes to persist one object.\n- **Node-store read latency** *(per node)* — how long the node store takes to retrieve one object.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#node-store-write-latency)", "fieldConfig": { "defaults": { "color": { @@ -3596,7 +3596,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"write_duration_us\"}[$__rate_interval])) / clamp_min(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"write_count\"}[$__rate_interval])), 1), \"series\", \"Write us/op (interval)\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_writes_duration_us\"}[$__rate_interval])) / clamp_min(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_writes\"}[$__rate_interval])), 1), \"series\", \"Write us/op (interval)\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "A" }, { @@ -3604,7 +3604,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"read_duration_us\"}[$__rate_interval])) / clamp_min(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"read_count\"}[$__rate_interval])), 1), \"series\", \"Read us/op (interval)\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_reads_duration_us\"}[$__rate_interval])) / clamp_min(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_reads_total\"}[$__rate_interval])), 1), \"series\", \"Read us/op (interval)\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "B" } ], @@ -3616,7 +3616,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Node-store write and read operation rates — the denominators behind the latency panel.*\n\n###### How it's computed:\n*Rate of the nodestore_latency write_count and read_count series.*\n\n###### Reading it:\n*Writes climb while a node is back-filling history and fall to near the ledger-close rate once it is caught up.*\n\n###### Healthy range:\n*Non-zero writes whenever the node is ingesting ledgers.*\n\n###### Watch for:\n*Write rate at zero while the node is still behind the network: nothing is being persisted, so the stall is upstream of the node store — check peer supply and the acquire panels rather than storage. A flat latency with a collapsing operation rate also means the latency figure above has gone stale rather than good.*\n\n###### Keywords:\n- **Node-store operation rate** *(per node)* — stores and fetches per second against the node store.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreLatencyGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#node-store-operation-rate)", + "description": "###### What this is:\n*Node-store write and read operation rates — the denominators behind the latency panel.*\n\n###### How it's computed:\n*Rate of the nodestore_state node_writes and node_reads_total series.*\n\n###### Reading it:\n*Writes climb while a node is back-filling history and fall to near the ledger-close rate once it is caught up.*\n\n###### Healthy range:\n*Non-zero writes whenever the node is ingesting ledgers.*\n\n###### Watch for:\n*Write rate at zero while the node is still behind the network: nothing is being persisted, so the stall is upstream of the node store — check peer supply and the acquire panels rather than storage. A flat latency with a collapsing operation rate also means the latency figure above has gone stale rather than good.*\n\n###### Keywords:\n- **Node-store operation rate** *(per node)* — stores and fetches per second against the node store.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[MetricsRegistry.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/telemetry/MetricsRegistry.cpp)\n\n###### Function:\n`registerNodeStoreGauge`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#node-store-operation-rate)", "fieldConfig": { "defaults": { "color": { @@ -3703,7 +3703,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"write_count\"}[$__rate_interval])), \"series\", \"Writes/s\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_writes\"}[$__rate_interval])), \"series\", \"Writes/s\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "A" }, { @@ -3711,7 +3711,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_latency{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"read_count\"}[$__rate_interval])), \"series\", \"Reads/s\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", + "expr": "label_replace(label_join(label_replace(sum by (service_instance_id, xrpl_branch, xrpl_work_item) (rate(nodestore_state{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", metric=\"node_reads_total\"}[$__rate_interval])), \"series\", \"Reads/s\", \"\", \"\"), \"xrpl_ident\", \", \", \"service_instance_id\", \"xrpl_branch\", \"xrpl_work_item\"), \"xrpl_ident\", \"[$1]\", \"xrpl_ident\", \"(?:, )*(.*[^, ])(?:, )*\")", "refId": "B" } ], diff --git a/docker/telemetry/workload/expected_metrics.json b/docker/telemetry/workload/expected_metrics.json index 8ace992723..61130976de 100644 --- a/docker/telemetry/workload/expected_metrics.json +++ b/docker/telemetry/workload/expected_metrics.json @@ -170,9 +170,9 @@ "amendment_block{metric=\"warned\"}", "amendment_block{metric=\"seconds_to_block\"}", "peer_accept_total", - "nodestore_latency{metric=\"write_count\"}", - "nodestore_latency{metric=\"read_count\"}", - "nodestore_latency{metric=\"read_mean_us\"}", + "nodestore_state{metric=\"node_writes\"}", + "nodestore_state{metric=\"node_reads_total\"}", + "nodestore_state{metric=\"read_mean_us\"}", "ledger_quorum_publish{metric=\"trusted_validation_tally\"}", "ledger_quorum_publish{metric=\"quorum_target\"}", "ledger_quorum_publish{metric=\"time_to_first_validated_us\"}", @@ -180,8 +180,8 @@ "ledger_quorum_shortfall_total{stage=\"pre_accept\"}", "consensus_round_duration_ms_bucket", "consensus_round_duration_ms_count", - "nodestore_latency{metric=\"write_duration_us\"}", - "nodestore_latency{metric=\"read_duration_us\"}", + "nodestore_state{metric=\"node_writes_duration_us\"}", + "nodestore_state{metric=\"node_reads_duration_us\"}", "unl_quorum{metric=\"quorum_disabled\"}", "sweep_malloc_trim_us_bucket", "sweep_malloc_trim_us_count" @@ -191,7 +191,7 @@ "_conditional_note": "handshake_negotiation_fail_total and unl_fetch_total are conditional under the local harness: the first only exists once a handshake is rejected, and the second needs a [validator_list_sites] entry (run-full-validation.sh generates a static [validators] file instead). The validator has no per-metric optional flag, so if either reports 0 series in a harness run, move it out of this group rather than weakening the check.", "_sync_state_note": "The four sync_state sub-series are unconditional: the gauge observes all four on every collection tick, so each is present as a series even when its value is 0 (a node that never reached FULL reports initial_full_duration_us=0, and a healthy node reports server_stall_seconds=0). The check asserts series presence, not a non-zero value, which is exactly right here \u2014 a zero is a meaningful reading for these signals, and absence is the regression. server_stall_events_total is likewise always present because the observable counter reports the tally (0 or more) every tick. state_changes_total is asserted here with a from!=\"\",to!=\"\" selector rather than bare (parity_counters already asserts the bare name): the selector is what proves the WP-A2 {from,to} label dimension actually reached Prometheus, so a regression to the old unlabelled counter fails this check instead of silently passing on the bare name. It needs at least one real mode transition, which any node reaching connected/syncing produces during startup.", "_a7_note": "WP-A7 adds three observable gauges and four counters. The 16 gauge sub-series (peer_ledger_supply, peerfinder_slot_census, amendment_block) are unconditional and asserted individually: each callback in MetricsRegistry.cpp calls observe() for every field on every collection tick with no early return between them, so the series exists whatever the value. That includes the two sentinel readings \u2014 a node whose peers have advertised nothing reports peer_ledger_supply{metric=\"supply_min_seq\"} = 0 meaning unknown, and a node with no pending amendment reports amendment_block{metric=\"seconds_to_block\"} = -1 meaning healthy. Absence, not the sentinel, is the regression. Of the four counters only peer_accept_total is asserted: run-full-validation.sh gives every node a [port_peer] on 0.0.0.0 and lists the other four nodes in [ips], so all 5 nodes dial each other and each one is also dialled, which means OverlayImpl::onHandoff runs and reports outcome=accepted (or slot_refused/no_slot on the duplicate half of each mutual dial) on every node. It is asserted bare rather than with an outcome= selector because which outcome a given node records depends on dial ordering, which the harness does not control. The other three counters are deliberately NOT asserted. peer_disconnect_total is emitted only from PeerImp::close, and a healthy 5-node localhost cluster holds its 4 fixed peers for the whole run: the timer-driven reasons need maxUnknownTime (600 s) or maxDivergedTime (300 s) to elapse (Config.h) while the full-validation profile totals well under that, and the shutdown reasons only fire during teardown, which happens in run-full-validation.sh after Step 5 has already scraped. serve_refused_total needs a peer to ask this node for a ledger, tx set or object it cannot serve \u2014 on a cluster where every node has the same complete history from genesis, getLedger()/getTxSet() succeed and the send queues never approach Tuning::kDropSendQueue. ledger_jump_total needs NetworkOPsImp::switchLastClosedLedger, reached only when consensus reports an LCL this node did not build on; a healthy 5-node cluster agrees every round, so it never jumps. The metric validator has no per-metric optional flag, so listing any of the three would fail the harness red on a healthy run \u2014 the same reasoning _acquire_note applies to the WP-A3 InboundLedger counters. All four counters are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp and rendered by the ledger-sync-health panels Peer Disconnects by Reason, Ledger/Object Serve Refusals and Byzantine Ledger Jumps. To make them assertable the harness would need a fault-injection step: kill one node mid-run and re-scrape before teardown (peer_disconnect_total, reason=read_error/graceful), request a ledger sequence outside the cluster's history or drive a node past its send-queue limit (serve_refused_total), and start a node on a divergent chain tip or partition the cluster and heal it (ledger_jump_total).", - "_a6_note": "WP-A6 adds one observable gauge (nodestore_latency) and two counters (ledger_replay_fallback_total, ledger_replay_outcome_total). Only three of the four gauge sub-series are asserted. write_count and read_count are unconditional: the callback observes both on every collection tick with no early return before them, so a series exists whatever the value, and a node that has written nothing reports write_count=0 rather than dropping the series. read_mean_us is safe because any node that has opened a ledger has already fetched objects, so the fetch duration total is non-zero. write_mean_us is deliberately NOT asserted: the mean is emitted only when the store-duration total is non-zero, and that total is fed by Database::recordStoreDuration(), which today is called only from Database::importInternal -- the [import_db] admin path. Database::store() is pure virtual and the two concrete runtime overrides (DatabaseNodeImp::store, DatabaseRotatingImp::store) do not time themselves yet, so an ordinary harness node produces write_count but no write_mean_us. Asserting it would ship a permanently red CI check for a known, documented gap; the omission is the honest encoding of that gap. The two replay counters are likewise NOT asserted, for the same reason _acquire_note gives for the WP-A3 InboundLedger counters: both are emitted only from the ledger-replay path, which requires the [ledger_replay] config stanza AND a real historical back-fill against peers that support the LedgerReplay protocol feature. run-full-validation.sh starts a fresh local cluster with no history to back-fill, so no replay task is ever created and neither counter can produce a series. All three unasserted signals are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp and rendered by the ledger-sync-health panels NodeStore Write vs Read Latency, Replay Fallback to Full Acquire and Replay Outcomes. To make them assertable the harness would need to enable [ledger_replay] and start a node against an existing ledger history so it back-fills through the replay path, and to time the two concrete store overrides.", + "_a6_note": "WP-A6 originally added a separate nodestore_latency gauge; it was retired because every one of its sub-metrics had an exact counterpart on nodestore_state computed from the same Database accessor, so the five entries above are the surviving equivalents (write_count -> node_writes, read_count -> node_reads_total, write_duration_us -> node_writes_duration_us, read_duration_us -> node_reads_duration_us, read_mean_us unchanged). All four totals are unconditional: MetricsRegistry::observeNodeStoreTotals observes them on every collection tick with no early return before them, so a series exists whatever the value and a node that has written nothing reports node_writes=0 rather than dropping the series. read_mean_us is safe because any node that has opened a ledger has already fetched objects, so the fetch count is non-zero. write_mean_us is NOT asserted, but only because the two means are the sub-series that scaledMean() omits when their denominator is zero, and this validator has no per-metric optional flag -- not because the numerator is missing. That older caveat is gone: all three concrete store paths now time themselves through Database::recordStoreDuration() (DatabaseNodeImp::store, DatabaseRotatingImp::store and Database::importInternal), so write_mean_us is live on an ordinary node and only a node that has performed literally zero stores would lack it. It can be promoted to an assertion once a harness run confirms it present. WP-A6's two replay counters (ledger_replay_fallback_total, ledger_replay_outcome_total) are likewise NOT asserted, for the same reason _acquire_note gives for the WP-A3 InboundLedger counters: both are emitted only from the ledger-replay path, which requires the [ledger_replay] config stanza AND a real historical back-fill against peers that support the LedgerReplay protocol feature. run-full-validation.sh starts a fresh local cluster with no history to back-fill, so no replay task is ever created and neither counter can produce a series. Both are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp and rendered by the ledger-sync-health panels Replay Fallback to Full Acquire and Replay Outcomes. To make them assertable the harness would need to enable [ledger_replay] and start a node against an existing ledger history so it back-fills through the replay path.", "_a5_note": "WP-A5 adds one observable gauge (ledger_quorum_publish) and one counter (ledger_quorum_shortfall_total). All four gauge sub-series are asserted and are unconditional: registerLedgerQuorumPublishGauge's callback in MetricsRegistry.cpp calls observe() for every field on every collection tick with no early return between them, and each accessor is a plain relaxed atomic load that always returns a value, so the series exists whatever the reading. That deliberately includes the three diagnostic zeros: a node that has never had a gate evaluated reports trusted_validation_tally=0 and quorum_target=0, one that has never fully validated reports time_to_first_validated_us=0, and one that is caught up reports publish_lag=0. Absence, not the zero, is the regression -- the same reasoning _sync_state_note gives for initial_full_duration_us. Note the sentinel: when the trusted list disables quorum entirely, getNeededValidations() returns SIZE_MAX and LedgerMaster reports quorum_target as int64 max rather than letting the cast wrap to -1, so the target reads far above any tally instead of inverting the comparison (the same fix as the unl_quorum gauge). ledger_quorum_shortfall_total IS asserted, which differs from the WP-A3/A6/A7 counters, and the reason is that this counter does not need a fault to fire. RCLConsensus::Adaptor::doAccept issues this node's own validation and then calls ledgerMaster_.consensusBuilt immediately (RCLConsensus.cpp), which calls checkAccept on the freshly built ledger (LedgerMaster.cpp) BEFORE the peers' validations for that same ledger have arrived. With the harness's 5 validators the quorum is max(ceil(5*0.8), ceil(5*0.6)) = 4 (ValidatorList::calculateQuorum), so that first evaluation of each round tallies short of 4 and takes the shortfall early return; the gate is then re-entered from RCLValidations handleNewValidation as each trusted validation arrives and eventually passes. A HEALTHY 5-node cluster therefore emits this counter every round, which is why it is safe to assert on a clean run -- unlike peer_disconnect_total or ledger_replay_fallback_total, it needs no fault injection, no [ledger_replay] stanza and no historical back-fill. The stage=\"pre_accept\" selector is asserted rather than the bare name so that the label dimension is proven to have reached Prometheus, matching the state_changes_total{from,to} pattern. Consequence for readers of the panels: a non-zero rate on Pre-Accept Quorum Shortfall Rate is NOT by itself a fault, and the panel description says so; the fault signature is that rate climbing well above the ledger-close rate while the tally on Trusted Validations vs Quorum Target stays flat below its target. If a future harness change makes the cluster single-node or standalone this assertion must move to a note: standalone_ short-circuits consensusBuilt before checkAccept, and getNeededValidations() returns 0 in standalone mode, so the gate can never report a shortfall.", "_b5_sweep_note": "WP-B5 Suspect 3 (per-sweep malloc_trim) adds one histogram and two counters. Only the histogram is asserted, by its Prometheus _bucket and _count series because the bare instrument name is not a series. It is unconditional on any running node: ApplicationImp::start arms the sweep timer before the workload begins, the interval is SizedItem::SweepInterval (Config.cpp: 10 s at nodeSize 0 through 120 s at nodeSize 4), and the full-validation profile runs 270 s of workload before Step 5 scrapes -- so at least two sweeps complete even in the slowest case, and each one records exactly one sample. The measurement itself is now unconditional too: it used to sit inside `if (journal.debug())` in MallocTrim.cpp, so a node at ordinary log level measured nothing; that gate now covers only the JLOG. The two counters are deliberately NOT asserted. sweep_malloc_trim_minor_faults_total is emitted only when the trim's minor-fault delta is greater than zero, and a trim on the small heap of a fresh localhost node routinely faults zero times -- the emit site publishes nothing rather than a zero, because a zero-valued series would claim the trim was measured as free when the honest statement is that there was nothing to fault on. sweep_malloc_trim_reclaimed_kb_total is emitted only when resident memory actually FELL across the trim, which on a node whose caches are still filling frequently does not happen (glibc has nothing above the top of the heap to release, and mmap-backed allocations are returned on free regardless of trimming). The validator has no per-metric optional flag, so asserting either would ship a permanently red CI check for a healthy run. Both are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp -- including the skip paths -- and rendered by the ledger-sync-health panel Sweep Heap-Trim Faults & Reclaim Rate. To make them assertable the harness would need a node with a large enough resident heap for a trim to reclaim, e.g. starting against an existing populated database rather than from genesis.", "_b5_rotation_note": "WP-B5 Suspect 4 (online_delete rotation extra writes) adds one observable gauge (rotation_state, sub-series in_flight and copy_forward) and one counter (rotation_copy_node_restore_total). NONE is asserted, because the 5-node localhost harness structurally CANNOT produce any of them -- this is a documented note rather than a check that would fail CI red. Two independent reasons. First, no rotation ever runs: xrpld-validator.cfg.template sets online_delete=256 and does not set advisory_delete, so SHAMapStoreImp's gate is validatedSeq >= lastRotated + 256 (SHAMapStoreImp.cpp), which needs 256 validated ledgers; at the network's several-seconds-per-ledger close rate that is on the order of 15-20 minutes, while the full-validation profile totals 270 s of workload before Step 5 scrapes. Second, even the in_flight flag needs a rotation to have started, and copy_forward additionally needs an ARCHIVE holding data that a fetch actually reads during the rotation window -- which requires a populated, already-rotated database, exactly the condition the hypothesis says is why this slowdown never appears on a fresh node. rotation_copy_node_restore_total is narrower still: it fires only for a clean tree node reachable from the validated state map whose sole on-disk copy was removed by an EARLIER rotation, so it needs at least two rotations plus real prior data loss. Note that rotation_state publishes no series at all when online_delete is not configured, by design: MetricsRegistry::registerRotationStateGauge dynamic_casts the node store to DatabaseRotating and returns early on failure, so an absent series means 'rotation is not configured' rather than the false 'rotation is free' a zero would report. All four signals are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp (including the not-configured and between-rotations cases) and rendered by the ledger-sync-health panels Online-Delete Rotation Window & Copy-Forward Writes and Rotation Node Re-Store Rate. To make them assertable the harness would need a step that starts a node against a pre-populated database that has already rotated at least once, or that lowers online_delete and advisory_delete far enough to force a rotation inside the run window and then re-scrapes before teardown.", diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index a68f8e8081..6cb75a0086 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -2534,11 +2534,9 @@ whose extra writes need an archive to read from. Both are absent by construction on a fresh node, which is what makes them candidate explanations for this branch's symptom rather than general slowness. -Three limits to respect here. The node-store numbers are **means, not -percentiles**, and `write_mean_us` is currently emitted only for the -`[import_db]` admin import path — on an ordinary node `write_count` climbs with -no `write_mean_us` line, which is a known instrumentation gap, not a healthy -zero. And the trim's fault counter is scoped to the **trim call only**: it shows +Two limits to respect here. The node-store numbers are **means, not +percentiles**, so a tail that matters will move them but there is no p99. And +the trim's fault counter is scoped to the **trim call only**: it shows that the trim itself faults, and it cannot show the faults paid later as the caches refill and touch the pages the trim returned. That later re-fault cost is the actual mechanism by which a trim would slow a sync, and no metric here @@ -2974,9 +2972,10 @@ panel it reads. write-bound, so no read-side panel can show it; check this step whenever a node with existing history is the slow one. Both panels live in the collapsed **Back-fill & persistence** row — expand it. - Panel _NodeStore Write vs Read Latency (us/op)_ (`nodestore_latency`, - `metric=write_mean_us` and `read_mean_us`) with _NodeStore Operation Rate - (writes vs reads)_ (`metric=write_count` / `read_count`) beside it: + Panel _NodeStore Write vs Read Latency (us/op)_ (`nodestore_state`, + `metric=node_writes_duration_us` / `node_reads_duration_us` rated against + their counts) with _NodeStore Operation Rate (writes vs reads)_ + (`metric=node_writes` / `node_reads_total`) beside it: - **Write line rising during history back-fill** — the backend cannot absorb writes fast enough. Sync will stay slow however many peers are available, so adding peers will not help. Check storage IOPS, the @@ -2999,12 +2998,10 @@ panel it reads. a tail that matters will move it, but there is no p99 here. That is a deliberate cost trade — a histogram would need one `Record()` per node object, and a single ledger write walks thousands of SHAMap nodes. - Second, `write_mean_us` is currently emitted only for store paths that - record their duration, which today is the `[import_db]` admin import. - On an ordinary node you will see `write_count` climbing with **no** - `write_mean_us` line: that is a known instrumentation gap, not a healthy - zero, and the mean is deliberately omitted rather than drawn as 0 so it - cannot be misread as "writes are instantaneous". + Second, a mean is **omitted rather than drawn as 0** when nothing has + been stored or fetched yet, so an absent line means "no samples", not + "instantaneous". All three concrete store paths time themselves, so + `write_mean_us` is present on any node that has written at all. 15. **Is replay-based back-fill silently falling back to the slow path?** Only relevant when `[ledger_replay]` is enabled. Panels _Replay Fallback to diff --git a/src/tests/libxrpl/telemetry/MetricMacros.cpp b/src/tests/libxrpl/telemetry/MetricMacros.cpp index d4cf3e56ba..7c93ed82c6 100644 --- a/src/tests/libxrpl/telemetry/MetricMacros.cpp +++ b/src/tests/libxrpl/telemetry/MetricMacros.cpp @@ -23,6 +23,7 @@ #include #include +#include #include #include @@ -2719,17 +2720,20 @@ TEST(MetricMacros, ledger_replay_counters_emit_nothing_when_disabled) EXPECT_EQ(app.registry().meterCalls(), 0); } -// The nodestore_latency gauge derives a mean from two cumulative totals the -// node store already keeps. This mirrors the production callback in -// MetricsRegistry::registerNodeStoreLatencyGauge, whose enabled path cannot be -// linked into this binary, so the derivation is asserted here against the same -// four inputs. -TEST(MetricMacros, nodestore_latency_gauge_observes_exact_derived_means) +// The nodestore_state gauge derives its two mean latencies from cumulative +// totals the node store already keeps. This mirrors the production callback in +// MetricsRegistry::observeNodeStoreTotals, whose enabled path cannot be linked +// into this binary (MetricsRegistry.cpp is excluded from xrpl_tests when +// telemetry is ON -- see src/tests/libxrpl/CMakeLists.txt), so the derivation +// is asserted here against the same inputs. The division itself is the real +// MetricsRegistry::scaledMean, a public constexpr inline that IS linkable, so +// this test exercises production arithmetic rather than a copy of it. +TEST(MetricMacros, nodestore_state_gauge_observes_exact_derived_means) { // Each scenario gets a FRESH provider. The reader reports cumulative // temporality (see CollectOnDemandReader), so a series observed by one // scenario would still be present in the next collect() -- which would - // defeat the two assertions below that a mean is ABSENT when its + // defeat the assertions below that a mean is ABSENT when its // denominator is zero. // The four totals the production callback reads, chosen so each mean @@ -2754,8 +2758,8 @@ TEST(MetricMacros, nodestore_latency_gauge_observes_exact_derived_means) auto collectWith = [](NodeStoreTotals& state) { CollectingProvider const provider; auto gauge = provider.meter()->CreateInt64ObservableGauge( - telemetry::metric::nodestoreLatency, - "NodeStore mean store/fetch latency in microseconds, with counts"); + telemetry::metric::nodestoreState, + "NodeStore I/O counters, latencies, write-queue depth and acquisition stalls"); gauge->AddCallback( [](opentelemetry::metrics::ObserverResult result, void* state) { auto const* self = static_cast(state); @@ -2764,19 +2768,27 @@ TEST(MetricMacros, nodestore_latency_gauge_observes_exact_derived_means) opentelemetry::metrics::ObserverResultT>>(result) ->Observe(value, {{telemetry::label::metric, field}}); }; - observe("write_count", static_cast(self->storeCount)); - observe("read_count", static_cast(self->fetchCount)); - if (self->storeCount > 0 && self->storeDurationUs > 0) + // The four cumulative totals, observed unconditionally: for a + // total, zero is the meaningful "nothing yet" reading. + observe("node_writes", static_cast(self->storeCount)); + observe("node_reads_total", static_cast(self->fetchCount)); + observe( + "node_writes_duration_us", static_cast(self->storeDurationUs)); + observe("node_reads_duration_us", static_cast(self->fetchDurationUs)); + + // The two derived means, through the production helper. It + // returns nullopt when the denominator is 0, and the series is + // then omitted rather than observed as 0: a reported 0 us would + // claim the operation is instantaneous, which is worse than a + // visible gap. + using Registry = telemetry::MetricsRegistry; + if (auto const mean = Registry::scaledMean(self->fetchDurationUs, self->fetchCount)) { - observe( - "write_mean_us", - static_cast(self->storeDurationUs / self->storeCount)); + observe("read_mean_us", *mean); } - if (self->fetchCount > 0 && self->fetchDurationUs > 0) + if (auto const mean = Registry::scaledMean(self->storeDurationUs, self->storeCount)) { - observe( - "read_mean_us", - static_cast(self->fetchDurationUs / self->fetchCount)); + observe("write_mean_us", *mean); } }, &state); @@ -2785,37 +2797,43 @@ TEST(MetricMacros, nodestore_latency_gauge_observes_exact_derived_means) auto const busy = collectWith(totals); - // Exactly four series: two means and the two denominators that let a - // dashboard recover interval latency from these cumulative totals. - ASSERT_EQ(busy.at("nodestore_latency").size(), 4u); - EXPECT_EQ(gaugeValue(busy, "nodestore_latency", attrs("metric", "write_mean_us")), 4000); - EXPECT_EQ(gaugeValue(busy, "nodestore_latency", attrs("metric", "read_mean_us")), 1000); - EXPECT_EQ(gaugeValue(busy, "nodestore_latency", attrs("metric", "write_count")), 500); - EXPECT_EQ(gaugeValue(busy, "nodestore_latency", attrs("metric", "read_count")), 1000); + // Exactly six series: the two means and the four cumulative totals that let + // a dashboard recover interval latency from them. + ASSERT_EQ(busy.at("nodestore_state").size(), 6u); + EXPECT_EQ(gaugeValue(busy, "nodestore_state", attrs("metric", "write_mean_us")), 4000); + EXPECT_EQ(gaugeValue(busy, "nodestore_state", attrs("metric", "read_mean_us")), 1000); + EXPECT_EQ(gaugeValue(busy, "nodestore_state", attrs("metric", "node_writes")), 500); + EXPECT_EQ(gaugeValue(busy, "nodestore_state", attrs("metric", "node_reads_total")), 1000); + EXPECT_EQ( + gaugeValue(busy, "nodestore_state", attrs("metric", "node_writes_duration_us")), 2'000'000); + EXPECT_EQ( + gaugeValue(busy, "nodestore_state", attrs("metric", "node_reads_duration_us")), 1'000'000); - // The write mean is the new signal, and it must be legible next to the - // read mean rather than merely present. + // The write mean is the signal this pair exists for, and it must be legible + // next to the read mean rather than merely present. EXPECT_GT( - gaugeValue(busy, "nodestore_latency", attrs("metric", "write_mean_us")), - gaugeValue(busy, "nodestore_latency", attrs("metric", "read_mean_us"))); + gaugeValue(busy, "nodestore_state", attrs("metric", "write_mean_us")), + gaugeValue(busy, "nodestore_state", attrs("metric", "read_mean_us"))); // Single fixed-cardinality label group, keyed exactly `metric`. - auto const& firstKey = busy.at("nodestore_latency").begin()->first; + auto const& firstKey = busy.at("nodestore_state").begin()->first; ASSERT_EQ(firstKey.size(), 1u); EXPECT_EQ(firstKey.begin()->first, "metric"); // EDGE CASE: a node that has never written. The zero denominator must skip - // the mean rather than divide by zero, while the count is still reported -- + // the mean rather than divide by zero, while the total is still reported -- // that is what distinguishes "nothing written yet" from "writes are // instant". The read side is unaffected and still reports both. totals = NodeStoreTotals{ .storeCount = 0, .storeDurationUs = 0, .fetchCount = 4, .fetchDurationUs = 800}; auto const idle = collectWith(totals); - EXPECT_EQ(idle.at("nodestore_latency").count(attrs("metric", "write_mean_us")), 0u); - EXPECT_EQ(gaugeValue(idle, "nodestore_latency", attrs("metric", "write_count")), 0); - EXPECT_EQ(gaugeValue(idle, "nodestore_latency", attrs("metric", "read_mean_us")), 200); - EXPECT_EQ(gaugeValue(idle, "nodestore_latency", attrs("metric", "read_count")), 4); + EXPECT_EQ(idle.at("nodestore_state").count(attrs("metric", "write_mean_us")), 0u); + EXPECT_EQ(gaugeValue(idle, "nodestore_state", attrs("metric", "node_writes")), 0); + EXPECT_EQ(gaugeValue(idle, "nodestore_state", attrs("metric", "read_mean_us")), 200); + EXPECT_EQ(gaugeValue(idle, "nodestore_state", attrs("metric", "node_reads_total")), 4); + // Five series, not six: every total plus the one mean that is derivable. + EXPECT_EQ(idle.at("nodestore_state").size(), 5u); // EDGE CASE: integer division truncates rather than rounding. 7 stores // over 100 us is 14.28 us, reported as 14 -- asserted so a future change @@ -2824,30 +2842,35 @@ TEST(MetricMacros, nodestore_latency_gauge_observes_exact_derived_means) .storeCount = 7, .storeDurationUs = 100, .fetchCount = 0, .fetchDurationUs = 0}; auto const truncating = collectWith(totals); - EXPECT_EQ(gaugeValue(truncating, "nodestore_latency", attrs("metric", "write_mean_us")), 14); + EXPECT_EQ(gaugeValue(truncating, "nodestore_state", attrs("metric", "write_mean_us")), 14); // The read side now has the zero denominator, so its mean drops out too. - EXPECT_EQ(truncating.at("nodestore_latency").count(attrs("metric", "read_mean_us")), 0u); - EXPECT_EQ(gaugeValue(truncating, "nodestore_latency", attrs("metric", "read_count")), 0); + EXPECT_EQ(truncating.at("nodestore_state").count(attrs("metric", "read_mean_us")), 0u); + EXPECT_EQ(gaugeValue(truncating, "nodestore_state", attrs("metric", "node_reads_total")), 0); - // EDGE CASE, and the one that matters most on a real node: stores were - // counted but never TIMED. Database::store() is pure virtual and only the - // paths calling recordStoreDuration() contribute a numerator, so a node - // whose concrete store override does not time itself has a non-zero count - // with a zero duration. The mean must be OMITTED, not reported as 0 -- - // a 0 would read as "writes are instantaneous", which is worse than a - // visible gap. This assertion is the guard on that choice. + // EDGE CASE: stores counted, but every one measured below a microsecond. + // recordStoreDuration() only adds when the cast to microseconds is > 0, so + // sub-microsecond stores leave the duration total at 0 while the count + // climbs. scaledMean divides on the COUNT alone, so it reports a genuine + // mean of 0 here rather than omitting the series. That is the correct + // reading: the stores really did complete + // in under a microsecond each, and the total beside it proves they + // happened. Absence is reserved for "no samples at all". totals = NodeStoreTotals{ .storeCount = 9000, .storeDurationUs = 0, .fetchCount = 10, .fetchDurationUs = 50}; - auto const untimed = collectWith(totals); + auto const subMicrosecond = collectWith(totals); - EXPECT_EQ(untimed.at("nodestore_latency").count(attrs("metric", "write_mean_us")), 0u); - // The count is still published, so the gap is visible rather than silent: - // a panel shows real write throughput with no latency line beside it. - EXPECT_EQ(gaugeValue(untimed, "nodestore_latency", attrs("metric", "write_count")), 9000); - // The read side is independent and unaffected by the write-side gap. - EXPECT_EQ(gaugeValue(untimed, "nodestore_latency", attrs("metric", "read_mean_us")), 5); - // Exactly three series: both counts plus the one mean that is derivable. - EXPECT_EQ(untimed.at("nodestore_latency").size(), 3u); + EXPECT_EQ(subMicrosecond.at("nodestore_state").count(attrs("metric", "write_mean_us")), 1u); + EXPECT_EQ(gaugeValue(subMicrosecond, "nodestore_state", attrs("metric", "write_mean_us")), 0); + // The count is published beside it, so the reading is interpretable: real + // write throughput with a sub-microsecond per-operation cost. + EXPECT_EQ(gaugeValue(subMicrosecond, "nodestore_state", attrs("metric", "node_writes")), 9000); + EXPECT_EQ( + gaugeValue(subMicrosecond, "nodestore_state", attrs("metric", "node_writes_duration_us")), + 0); + // The read side is independent and unaffected by the write-side reading. + EXPECT_EQ(gaugeValue(subMicrosecond, "nodestore_state", attrs("metric", "read_mean_us")), 5); + // All six series present: both means are derivable here. + EXPECT_EQ(subMicrosecond.at("nodestore_state").size(), 6u); } // consensus_round_duration_ms: exact recorded values, not "greater than zero". @@ -3123,7 +3146,7 @@ TEST(MetricMacros, sweep_malloc_trim_skips_reclaim_when_rss_grew) // rotation_state is polled from the node store, so the production callback in // MetricsRegistry::registerRotationStateGauge cannot be linked into this // binary. The derivation it performs is asserted here against the same two -// inputs, mirroring how nodestore_latency is tested above. +// inputs, mirroring how nodestore_state is tested above. TEST(MetricMacros, rotation_state_gauge_observes_in_flight_window_and_copy_forward_total) { // Each scenario gets a FRESH provider: the reader is cumulative, so a diff --git a/src/tests/libxrpl/telemetry/MetricsRegistry.cpp b/src/tests/libxrpl/telemetry/MetricsRegistry.cpp index f2d7199de8..6116600ff8 100644 --- a/src/tests/libxrpl/telemetry/MetricsRegistry.cpp +++ b/src/tests/libxrpl/telemetry/MetricsRegistry.cpp @@ -40,7 +40,7 @@ * `clock_close_offset_seconds`, `sync_state`, * `server_stall_events_total`, `sync_acquire`, `shamap_cache_hit_rate`, * `jobq_saturation`, `peer_ledger_supply`, - * `peerfinder_slot_census`, `amendment_block`, `nodestore_latency`): + * `peerfinder_slot_census`, `amendment_block`, `nodestore_state`): * this file CANNOT assert an observed gauge * value, because on this build the gauges do not exist -- their registration * methods and the OTel instrument members are inside @@ -854,7 +854,7 @@ TEST_F(MetricsRegistryTest, disabled_lifecycle_never_consults_gauge_services) // registerSyncAcquireGauge() / registerCacheHitRateDetailGauge() / // registerJobQueueBacklogGauge() / registerJobQueueSaturationGauge() / // registerPeerLedgerSupplyGauge() / registerSlotCensusGauge() / - // registerAmendmentBlockGauge() / registerNodeStoreLatencyGauge() -- + // registerAmendmentBlockGauge() / registerNodeStoreGauge() -- // would run. EXPECT_NO_THROW(registry.start("http://localhost:4318/v1/metrics")); @@ -900,12 +900,11 @@ TEST_F(MetricsRegistryTest, disabled_lifecycle_never_consults_gauge_services) // table's mutex. Not consulted above, so the countdown never ran. (Its // `warned` half reads NetworkOPs, already covered by the getOPs() check.) EXPECT_THROW(mockApp_.getAmendmentTable(), std::logic_error); - // The service the WP-A6 nodestore latency gauge reads: nodestore_latency - // polls getStoreDurationUs()/getStoreCount() and - // getFetchDurationUs()/getFetchTotalCount() on the node-store Database. - // Not consulted above, so the latency gauge never read those atomics on a - // telemetry-off build. (The existing nodestore_state gauge reads the same - // service, so this single throw covers both.) + // The service the nodestore gauge reads: nodestore_state polls + // getStoreDurationUs()/getStoreCount() and + // getFetchDurationUs()/getFetchTotalCount() on the node-store Database, + // alongside its I/O totals and write-queue detail. Not consulted above, + // so the gauge never read those atomics on a telemetry-off build. EXPECT_THROW(mockApp_.getNodeStore(), std::logic_error); } @@ -929,7 +928,7 @@ TEST_F(MetricsRegistryTest, enabled_flag_alone_registers_no_gauges_when_compiled // registerSyncAcquireGauge()/registerCacheHitRateDetailGauge()/ // registerJobQueueBacklogGauge()/registerJobQueueSaturationGauge()/ // registerPeerLedgerSupplyGauge()/registerSlotCensusGauge()/ - // registerAmendmentBlockGauge()/registerNodeStoreLatencyGauge(), a + // registerAmendmentBlockGauge()/registerNodeStoreGauge(), a // callback would reach getValidators()/getTimeKeeper()/getOPs()/ // getLoadManager()/getInboundLedgers()/getNodeFamily()/getJobQueue()/ // getOverlay()/getAmendmentTable()/getNodeStore() and diff --git a/src/xrpld/telemetry/MetricNames.h b/src/xrpld/telemetry/MetricNames.h index 331a18a19b..ff04649d39 100644 --- a/src/xrpld/telemetry/MetricNames.h +++ b/src/xrpld/telemetry/MetricNames.h @@ -256,10 +256,6 @@ inline constexpr char peerfinderSlotCensus[] = "peerfinder_slot_census"; * Amendment-block warning and the countdown to this node ceasing to validate. */ inline constexpr char amendmentBlock[] = "amendment_block"; -/** - * NodeStore mean store/fetch latency, with the operation counts. - */ -inline constexpr char nodestoreLatency[] = "nodestore_latency"; // ===== Consensus ============================================================= @@ -658,23 +654,6 @@ inline constexpr char inFlight[] = "in_flight"; inline constexpr char copyForward[] = "copy_forward"; } // namespace rotation_state -/** - * `nodestore_latency` sub-metrics: mean latency per direction, with counts. - */ -namespace nodestore_latency { -inline constexpr char writeCount[] = "write_count"; -inline constexpr char readCount[] = "read_count"; -inline constexpr char writeMeanUs[] = "write_mean_us"; -inline constexpr char readMeanUs[] = "read_mean_us"; -// Cumulative microsecond totals. The means above are convenient to read at a -// glance but cannot be rated: they are already a ratio, and a gauge of a ratio -// has no meaningful derivative. Dividing the rate of these totals by the rate of -// the matching count yields the latency over the panel's own window, which is -// what a dashboard actually wants. -inline constexpr char writeDurationUs[] = "write_duration_us"; -inline constexpr char readDurationUs[] = "read_duration_us"; -} // namespace nodestore_latency - /** * `ledger_quorum_publish` sub-metrics: the gate, and how late publish is. */ diff --git a/src/xrpld/telemetry/MetricsRegistry.cpp b/src/xrpld/telemetry/MetricsRegistry.cpp index 279780073a..868302f055 100644 --- a/src/xrpld/telemetry/MetricsRegistry.cpp +++ b/src/xrpld/telemetry/MetricsRegistry.cpp @@ -726,7 +726,6 @@ MetricsRegistry::registerAsyncGauges() registerPeerLedgerSupplyGauge(); registerSlotCensusGauge(); registerAmendmentBlockGauge(); - registerNodeStoreLatencyGauge(); registerLedgerQuorumPublishGauge(); } @@ -2313,95 +2312,6 @@ MetricsRegistry::registerAmendmentBlockGauge() this); } -void -MetricsRegistry::registerNodeStoreLatencyGauge() -{ - // --- Sync diagnostics: is the node store slow, and on which side? --- - // The write mean is the new signal. storeDurationUs_ was declared and - // never written, so no write latency existed anywhere; only the read side - // had a duration total. A node with a large existing DB back-fills slower - // than a fresh one, and back-fill is write-bound, so the read-side - // metrics cannot show it. Exporting both means from one reading also makes - // the two sides directly comparable. - // - // Gauge rather than histogram: a histogram would cost one Record() per - // node object on the store/fetch path, which runs thousands of times per - // ledger write. This reads four atomics per ~10 s tick instead. The - // trade-off is that percentiles are unavailable -- see the header comment. - nodeStoreLatencyGauge_ = meter_->CreateInt64ObservableGauge( - metric::nodestoreLatency, - "NodeStore mean store/fetch latency in microseconds, with counts"); - nodeStoreLatencyGauge_->AddCallback( - [](opentelemetry::metrics::ObserverResult result, void* state) { - auto* self = static_cast(state); - if (self->callbacksDetached_.load(std::memory_order_acquire)) - return; - auto& app = self->app_; - - try - { - auto observe = [&](char const* field, int64_t value) { - opentelemetry::nostd::get>>(result) - ->Observe(value, {{"metric", field}}); - }; - - auto& db = app.getNodeStore(); - - // One reading of each pair, so a mean and its own denominator - // describe the same instant. - auto const storeCount = db.getStoreCount(); - auto const storeDurationUs = db.getStoreDurationUs(); - auto const fetchCount = db.getFetchTotalCount(); - auto const fetchDurationUs = db.getFetchDurationUs(); - - // Counts are always observed, including zero: that is what - // separates "nothing written yet" from "writes are instant". - observe(lval::nodestore_latency::writeCount, static_cast(storeCount)); - observe(lval::nodestore_latency::readCount, static_cast(fetchCount)); - - // A mean needs a non-zero denominator, and it needs a - // numerator that was actually measured. Both are required, and - // the series is omitted rather than observed as 0 when either - // is missing: a reported 0 us would claim writes are - // instantaneous, which is worse than no reading at all. - // - // The numerator guard covers the pre-first-write window only. - // Both concrete databases time their backend write, so the - // total advances on any ordinary node; before the first write - // it is still 0, and omitting the mean then is better than - // publishing a false "writes take 0 us". - // The cumulative totals are observed unconditionally, so a - // panel can divide rate(duration) by rate(count) and read the - // latency over its own window rather than a since-boot average - // that flattens as uptime grows. - observe( - lval::nodestore_latency::writeDurationUs, - static_cast(storeDurationUs)); - observe( - lval::nodestore_latency::readDurationUs, static_cast(fetchDurationUs)); - - if (storeCount > 0 && storeDurationUs > 0) - { - observe( - lval::nodestore_latency::writeMeanUs, - static_cast(storeDurationUs / storeCount)); - } - if (fetchCount > 0 && fetchDurationUs > 0) - { - observe( - lval::nodestore_latency::readMeanUs, - static_cast(fetchDurationUs / fetchCount)); - } - } - catch (...) // NOLINT(bugprone-empty-catch) - { - // Silently skip if services are not yet ready. - } - }, - this); -} - void MetricsRegistry::registerLedgerQuorumPublishGauge() { diff --git a/src/xrpld/telemetry/MetricsRegistry.h b/src/xrpld/telemetry/MetricsRegistry.h index 356a28cc15..0379dc25f1 100644 --- a/src/xrpld/telemetry/MetricsRegistry.h +++ b/src/xrpld/telemetry/MetricsRegistry.h @@ -69,7 +69,6 @@ * +-- Peer ledger supply (how many peers can serve the needed sequence) * +-- PeerFinder slot census (slots, attempts, fixed peers, address caches) * +-- Amendment block (warned flag + seconds until the node stops validating) - * +-- NodeStore latency (mean us per store and per fetch, with counts) * +-- Ledger quorum + publish (validation tally vs quorum target, * | time to first validated, publish lag) * +-- jq_trans_overflow_total (observed from Overlay) @@ -801,13 +800,6 @@ private: */ opentelemetry::nostd::shared_ptr amendmentBlockGauge_; - /** - * Observable gauge for node-store read and write latency, as mean - * microseconds per operation derived from the cumulative duration and - * operation-count totals the node store already keeps. - */ - opentelemetry::nostd::shared_ptr - nodeStoreLatencyGauge_; /** * Observable gauge for the pre-accept quorum gate and the publish lag: * the trusted-validation tally against the quorum it must reach, the @@ -983,8 +975,33 @@ private: /** * Observe the NodeStore I/O totals and the means derived from them. * + * Publishes the four cumulative totals (`node_reads_total`, + * `node_writes`, `node_reads_duration_us`, `node_writes_duration_us`) + * unconditionally, plus `read_mean_us` and `write_mean_us` derived from + * them via scaledMean(). `write_mean_us` is the signal for the "a node + * with a large existing database syncs slower than a fresh one" symptom: + * back-fill is write-bound, so no read-side reading can show it. All + * three concrete store paths time themselves through + * Database::recordStoreDuration(), so the write mean is live on an + * ordinary node. + * + * Gauge rather than histogram, deliberately. A histogram would give true + * percentiles, but it costs one Record() per node object on the + * store/fetch path, and one ledger write walks thousands of SHAMap + * nodes. This reads the existing atomics once per ~10 s tick and adds + * nothing to the hot path. Consequence, stated plainly: p99 is NOT + * obtainable from this signal. A histogram added later would also need an + * explicit-bucket View registered via addMicrosecondHistogramView(), + * because the SDK's default buckets top out at 10,000. + * * @param db NodeStore to read the counters from. * @param observe Sink for one `metric`-labelled value. + * + * @note The totals are monotonic and never reset, so a panel wanting + * current rather than since-boot latency divides the two rates. That is + * why the counts and duration totals are exported beside the means. + * @note A mean is omitted when its count is 0, so a dashboard shows a gap + * rather than a plausible-looking 0 us. */ static void observeNodeStoreTotals(node_store::Database& db, ObserveFn const& observe); @@ -1343,78 +1360,6 @@ private: void registerAmendmentBlockGauge(); // sync diagnostics: amendment countdown - /** - * Register the `nodestore_latency` gauge. - * - * Four series under the `metric` attribute, from the node store's own - * cumulative totals: - * - * `write_mean_us` — **the signal this gauge exists for.** Mean - * microseconds per store, `getStoreDurationUs() / getStoreCount()`. - * No write-side latency existed anywhere before this: - * `storeDurationUs_` was declared in Database.h and never written, and - * there was no accessor for it. This is the fingerprint of the - * "a node with a large existing DB syncs slower than a fresh one" - * symptom, which is write-bound and therefore invisible in every - * read-side metric. - * `read_mean_us` — mean microseconds per fetch, - * `getFetchDurationUs() / getFetchTotalCount()`, so the write mean has - * a same-instant, same-derivation counterpart to be compared against. - * `write_count` / `read_count` — the denominators, exported so a - * dashboard can recover *interval* latency as - * `rate(duration) / rate(count)`. Without them the means above are - * since-boot averages, which on a long-running node move so slowly - * that a current stall is invisible. - * - * Gauge, not a histogram — deliberate. A histogram would give true - * percentiles, which a mean cannot, but it costs one `Record()` per - * operation on a path that runs per node object: a single ledger write - * walks thousands of SHAMap nodes, and fetches are more frequent still. - * That is a per-object synchronous instrument call plus bucket search on - * the hot store/fetch path. This gauge instead reads four already-existing - * atomics once per ~10 s collection tick, adding nothing whatsoever to the - * hot path — the store side pays only the one clock-sample pair per store - * that the read side has always paid per fetch. For the question this work - * package answers ("is the write path slow, and slower than the read - * path?") a rate-derived mean is sufficient, and a tail latency that - * matters will move the mean. Consequence, stated plainly: p99 is NOT - * obtainable from this signal. Adding a histogram later would also require - * an explicit-bucket View registered in initExporterAndProvider() via - * addMicrosecondHistogramView(), because the SDK's default buckets top out - * at 10,000 and every microsecond duration above 10 ms would saturate. - * - * Distinct from `nodestore_state`, which already carries the raw - * cumulative `node_reads_duration_us`, `node_reads_total` and - * `node_writes` fields, and from the Ledger Data Sync dashboard's "NuDB - * Read Latency" panel that divides the first two in PromQL. Neither has - * any write-duration input to divide — that quantity did not exist. This - * gauge adds the missing write numerator and publishes both means from one - * reading so the two sides are directly comparable. - * - * @note Pulled on the OTel reader thread (~10 s tick). Four relaxed atomic - * loads and two integer divisions; no lock, no allocation, no hot-path - * cost. - * @note A mean is observed only when both its count and its duration total - * are non-zero; otherwise the series is omitted rather than reported as 0, - * because a 0 would claim the operation is instantaneous. The counts are - * always observed, so `write_count` still distinguishes "nothing written - * yet" from "writes are instant". - * @warning `write_mean_us` is currently produced only by store paths that - * call `Database::recordStoreDuration()`, which today is - * `Database::importInternal` (the `[import_db]` admin path). `store()` is - * pure virtual, and neither `DatabaseNodeImp::store` nor - * `DatabaseRotatingImp::store` calls it yet, so on an ordinary node - * `write_count` climbs while `write_mean_us` is absent. That is a - * deliberate, visible gap: closing it means adding one clock-sample pair to - * those two concrete store overrides, which live outside this work - * package's file scope. - * @note Both totals are monotonic and never reset. A panel wanting current - * rather than since-boot latency must divide the two rates, which is why - * the counts are exported alongside the means. - */ - void - registerNodeStoreLatencyGauge(); // sync diagnostics: store/fetch latency - /** * Register the `ledger_quorum_publish` gauge. *