diff --git a/.github/scripts/otel-naming/README.md b/.github/scripts/otel-naming/README.md index a3aa8a5e12..d0a6b9f603 100644 --- a/.github/scripts/otel-naming/README.md +++ b/.github/scripts/otel-naming/README.md @@ -46,18 +46,18 @@ hardcoded allowlist: ### Rules (each fails the build, when its inputs are present) -| Rule | Check | -| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| A | No stray dotted span-attribute key (only the derived resource keys may be dotted). | -| G | Attribute keys are `lower_snake_case` (`^[a-z][a-z0-9_]*$` per dot-segment) — no camelCase, UPPERCASE, or spaces. | -| F | No string literals as attribute keys or span-name arguments in `setAttribute`/`addEvent`/`span`/`rootSpan`/`childSpan` (`rootSpan` shares `span`'s `(cat, prefix, name)` signature). Attribute _values_ are exempt (runtime data); `*SpanNames.h` definitions and test files are exempt. | -| B | Every collector `spanmetrics.dimensions` name exists in the L1 key set. | -| C | Every Tempo span-filter tag exists in the L1 key set. | -| D | Every dashboard label resolves to an L1 span attribute, a native-metric label (L6, emitted by MetricsRegistry), or a Prometheus/Grafana builtin. TraceQL scope prefixes (`span.`/`resource.`/…) are stripped before the L1 lookup. | -| E | No dotted `xrpl..` attribute key in the runbook (only the L1 resource attrs `xrpl.network.*` may be dotted). Span names, filenames, OTel-standard keys, and metric labels are not flagged. | -| I | No string literals as **metric** instrument names or label keys — the mirror of Rule F. Applies to the name passed to an `XRPL_METRIC_*` macro or a `meter->Create*` factory and to the label _keys_ in its label set. Label _values_, descriptions, `*MetricNames.h`, `MetricMacros.h` and test files are exempt. Scoped by metric **family** (first underscore segment): declaring a constant opts that family in, so the metric surface can be converted subsystem by subsystem. Unconverted families warn as Rule L. | -| J | Metric instrument names follow the suffix conventions: `lower_snake_case`, no `xrpld_`/`xrpl_` prefix (the exporter adds it), a counter ends `_total`, a histogram ends `_us`/`_ms`/`_seconds`, a gauge does not end `_total`. The instrument **kind** is read from the emit site, never guessed from words in the name — so a multi-series gauge carrying units in its label values (e.g. `nodestore_state` observing `write_mean_us`) is not a violation. | -| K | Every metric named in `docker/telemetry/workload/expected_metrics.json` resolves to a declared constant, so a rename in code cannot leave the workload validator asserting a name nothing emits. PromQL selectors (`m{label="v"}`) and exporter-appended histogram suffixes (`_bucket`/`_count`/`_sum`) are normalized away first; groups fed by another emit path (`statsd_gauges`, `statsd_counters`, `spanmetrics`) are out of scope by design. | +| Rule | Check | +| ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| A | No stray dotted span-attribute key (only the derived resource keys may be dotted). | +| G | Attribute keys are `lower_snake_case` (`^[a-z][a-z0-9_]*$` per dot-segment) — no camelCase, UPPERCASE, or spaces. | +| F | No string literals as attribute keys or span-name arguments in `setAttribute`/`addEvent`/`span`/`rootSpan`/`childSpan` (`rootSpan` shares `span`'s `(cat, prefix, name)` signature). Attribute _values_ are exempt (runtime data); `*SpanNames.h` definitions and test files are exempt. | +| B | Every collector `spanmetrics.dimensions` name exists in the L1 key set. | +| C | Every Tempo span-filter tag exists in the L1 key set. | +| D | Every dashboard label resolves to an L1 span attribute, a native-metric label (L6, emitted by MetricsRegistry), or a Prometheus/Grafana builtin. TraceQL scope prefixes (`span.`/`resource.`/…) are stripped before the L1 lookup. | +| E | No dotted `xrpl..` attribute key in the runbook (only the L1 resource attrs `xrpl.network.*` may be dotted). Span names, filenames, OTel-standard keys, and metric labels are not flagged. | +| I | No string literals as **metric** instrument names or label keys — the mirror of Rule F. Applies to the name passed to an `XRPL_METRIC_*` macro or a `meter->Create*` factory and to the label _keys_ in its label set. Label _values_, descriptions, `*MetricNames.h`, `MetricMacros.h` and test files are exempt. Scoped by metric **family** (first underscore segment): declaring a constant opts that family in, so the metric surface can be converted subsystem by subsystem. Unconverted families warn as Rule L. | +| J | Metric instrument names follow the suffix conventions: `lower_snake_case`, no `xrpld_`/`xrpl_` prefix (the exporter adds it), a counter ends `_total`, a histogram ends `_us`/`_ms`/`_seconds`, a gauge does not end `_total`. The instrument **kind** is read from the emit site, never guessed from words in the name — so a multi-series gauge carrying units in its label values (e.g. `nodestore_state` observing `write_mean_us`) is not a violation. A name created through two different factories is itself reported as a kind conflict, since no suffix can be correct for both. | +| K | Every metric named in `docker/telemetry/workload/expected_metrics.json` resolves to a declared constant, so a rename in code cannot leave the workload validator asserting a name nothing emits. PromQL selectors (`m{label="v"}`) and exporter-appended histogram suffixes (`_bucket`/`_count`/`_sum`) are normalized away first; groups fed by another emit path (`statsd_gauges`, `statsd_counters`, `spanmetrics`) are out of scope by design. | Rule F runs **unconditionally** (it is a purely syntactic check on the call-sites and needs no `*SpanNames.h`), so a code path that calls diff --git a/.github/scripts/otel-naming/check_otel_naming.py b/.github/scripts/otel-naming/check_otel_naming.py index fa56475306..f05849593c 100644 --- a/.github/scripts/otel-naming/check_otel_naming.py +++ b/.github/scripts/otel-naming/check_otel_naming.py @@ -1335,7 +1335,15 @@ def instrument_kinds(root: Path, wire_by_symbol: Dict[str, str]) -> Dict[str, Se ) if wire is None: continue - kinds.setdefault(wire, set()).add(classify_instrument_kind(kind)) + classified = classify_instrument_kind(kind) + # `other` is the sentinel for a macro that is not an instrument + # factory. Adding it would let a future non-instrument + # `XRPL_METRIC_*` macro turn a correctly named metric into + # "created as counter and other", a conflict message that reads as + # nonsense. Dropping it keeps the sentinel doing exactly what it did + # before: matching no shape rule. + if classified != "other": + kinds.setdefault(wire, set()).add(classified) return kinds diff --git a/.github/scripts/otel-naming/test_check_otel_naming.py b/.github/scripts/otel-naming/test_check_otel_naming.py index 5d4205751f..5569e6716a 100644 --- a/.github/scripts/otel-naming/test_check_otel_naming.py +++ b/.github/scripts/otel-naming/test_check_otel_naming.py @@ -1281,6 +1281,21 @@ class RuleJMetricSuffixes(unittest.TestCase): self.assertIn("counter", violations[0][-1]) self.assertIn("gauge", violations[0][-1]) + def test_two_kinds_where_the_last_one_alone_looks_clean(self): + # The sharper case for the same bug. Here the LAST emit site visited is a + # histogram and the name carries a duration suffix, so under last-wins + # semantics Rule J saw a well-formed histogram and reported nothing at + # all -- the conflict was not merely mislabelled, it was invisible. With + # a set per name the mismatch surfaces regardless of walk order. + violations = self._run( + _mc("dualShape", "dual_shape_us"), + 'meter_->CreateInt64ObservableGauge(metric::dualShape, "d");\n' + 'meter_->CreateUInt64Histogram(metric::dualShape, "d");\n', + ) + self.assertEqual(len(violations), 1, violations) + self.assertIn("gauge", violations[0][-1]) + self.assertIn("histogram", violations[0][-1]) + def test_skip_when_no_header(self): d = Path(tempfile.mkdtemp()) try: diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e88183a285..b937a664a0 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -457,7 +457,9 @@ Enforcement is by the same script as the span rules, whose metric rules are: non-fatal **L** warning, keeping the remaining work visible. - **J** — the suffix conventions above. The instrument _kind_ is read from the emit site, not guessed from the name, so a multi-series gauge whose units live - in its label values is not mistaken for a mis-suffixed duration. + in its label values is not mistaken for a mis-suffixed duration. A name created + through two different factories is reported as a kind conflict rather than a + suffix complaint, because no suffix can be correct for both. - **K** — every metric named in `docker/telemetry/workload/expected_metrics.json` resolves to a declared constant. This is the check that catches a metric renamed in code while the workload validator still asserts the old name. diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md index 84d7eae673..77575e5cb5 100644 --- a/OpenTelemetryPlan/09-data-collection-reference.md +++ b/OpenTelemetryPlan/09-data-collection-reference.md @@ -1787,59 +1787,59 @@ no panel (it is read in Tempo instead). /proc/self/statm kernel path the heap-trim RSS readings are taken from, so it likewise cannot be respelled. --> -| Signal | Type | Emit site | Panel | Meaning | -| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| `dns_resolve_total` (`outcome` = `resolved` \| `empty`) | counter | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Outcome Rate | Peer hostname resolutions. `empty` means a configured bootstrap or `[ips_fixed]` name returned no address, so that peer is never dialled. | -| `dns_resolve_latency_ms` | histogram | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Latency (p95) | Time to resolve a configured peer hostname. Seconds-scale values mean the resolver is timing out ahead of every dial. | -| `overlay_connect_total` (`outcome` = `connected` \| `tcp_fail` \| `tls_fail` \| `upgrade_fail` \| `timeout`) | counter | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Outcome Rate | Outbound peer connection attempts by terminal outcome. The outcome names the stage that broke: TCP, TLS, HTTP upgrade, or no terminal state in time. | -| `overlay_dial_latency_ms` | histogram | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Latency (p95) | Time from starting an outbound dial to its terminal outcome, successes and failures together. A p95 near the dial timeout means peers accept TCP but never finish the handshake. | -| `handshake_negotiation_fail_total` (`reason`, 14 values incl. `wrong_network`, `invalid_network_id`, `clock_skew`, `self_connection`, `session_verify_failed`) | counter | `Handshake.cpp` — `throwNegotiationFailure` (from `verifyHandshake`) | Handshake Negotiation Failures by Reason | Peer handshakes rejected after TLS while checking network id, clock, keys and addresses. `reason` names the failing check. | -| `unl_fetch_total` (`site` = configured UNL URI; `outcome` = the 9 `ListDisposition` strings `accepted` \| `expired` \| `same_sequence` \| `pending` \| `known_sequence` \| `unsupported_version` \| `untrusted` \| `stale` \| `invalid`, plus `fetch_error` \| `bad_status` \| `parse_error`) | counter | `ValidatorSite.cpp` — `ValidatorSite::reportFetchOutcome` | UNL Fetch Rate by Site & Outcome | Validator-list fetches per site. `accepted` is the only success; `same_sequence` and `known_sequence` are normal no-op refreshes; the three literals are transport or content faults. | -| `unl_quorum` (`metric` = `trusted_keys` \| `quorum` \| `quorum_disabled`) | observable gauge | `MetricsRegistry.cpp` — `registerUnlQuorumGauge` | UNL Trusted Keys vs Quorum; UNL Quorum Headroom | Trusted UNL key count against the validations a ledger needs. `trusted_keys` at or below `quorum` means the node can never declare a ledger validated. | -| `clock_close_offset_seconds` (`metric` = `offset`) | observable gauge | `MetricsRegistry.cpp` — `registerClockSkewGauge` | Clock Close Offset | Network close time offset from the local clock. Negative means the local clock runs ahead. `server_info` only surfaces `close_time_offset` at 60 s or more, so this gauge sees skew far earlier. | -| `state_changes_total` (`from`, `to` = `disconnected` \| `connected` \| `syncing` \| `tracking` \| `full`) | counter | `NetworkOPs.cpp` — `NetworkOPsImp::setMode` | Mode Transitions by Edge | Operating-mode transitions keyed on the (`from`, `to`) edge. The edge is what separates a clean `disconnected`→`connected`→`syncing`→`tracking`→`full` climb from `full`→`connected` flapping; an unlabelled total cannot tell them apart. | -| `sync_state` (`metric` = `initial_full_duration_us`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Time to First FULL | Microseconds from process start to the first `full` transition, sourced from `NetworkOPs::getInitialSyncDurationUs()`. Stays 0 until `full` is reached, so a flat 0 is itself the "never synced" signal; once set it never changes. | -| `sync_state` (`metric` = `network_ledger_gate`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Network Ledger Gate | 1 while the node is still waiting to see a full network ledger (`NetworkOPs::isNeedNetworkLedger()`), else 0. A persistent 1 blocks transaction submission and `full`, whatever the rest of the pipeline shows. | -| `sync_state` (`metric` = `server_stall_seconds`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Server Stall | Current main-loop stall duration from `LoadManager::getCurrentStallSeconds()`, 0 when healthy. Same duration the load monitor logs as "Server stalled for N seconds", which previously existed only in that log line. | -| `sync_state` (`metric` = `ledgers_behind`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Ledgers Behind Network | Peer-reported network tip minus our validated sequence, floored at 0 (`NetworkOPs::getLedgersBehindNetwork()`). Reads each peer's already cached ledger range, so no new network round trip. | -| `server_stall_events_total` | observable counter | `MetricsRegistry.cpp` — `registerStallEventsCounter` | Server Stall Event Rate | Distinct stall episodes since process start, counted once per episode rather than per stalled second. A rising rate is repeated fresh stalls; a flat rate with a large `server_stall_seconds` is one long stall. | -| `sync_acquire` (`metric` = `missing_state_nodes_max` \| `missing_tx_nodes_max`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Missing SHAMap Nodes per Acquire (state/tx) | Largest outstanding SHAMap node count across in-flight acquires, split by tree, from the count `getMissingNodes()` already produces during its sweep (`InboundLedger.cpp` — `InboundLedger::trigger`). **The headline stuck-sync signal:** flat and non-zero across ticks means the acquire will never finish; shrinking means slow but alive. Aggregated as a max rather than labelled per ledger, because a `ledger_seq` label would mint one series per ledger acquired — per-ledger identity stays on the `ledger.acquire` span. | -| `sync_acquire` (`metric` = `received_data_depth`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Peer packets stashed across all in-flight acquires waiting to be applied, summed because it measures one shared processing backlog. A growing depth means arriving node data outpaces processing, so the limit is the job queue or disk rather than peer supply. | -| `sync_acquire` (`metric` = `in_flight`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Number of ledger acquires currently running. Exported so the three values above can be read in context: all zero with `in_flight` zero is an idle node, not a healthy one. | -| `shamap_cache_hit_rate` (`metric` = `treenode`) | observable gauge | `MetricsRegistry.cpp` — `registerCacheHitRateDetailGauge` | SHAMap TreeNode Cache Hit Rate | Share of SHAMap tree-node lookups served from memory, from the previously-uncalled `TaggedCache::getHitRate()`, normalized from 0-100 to 0.0-1.0. Distinct from `nodestore_state`-derived NuDB Cache Hit Ratio on the Ledger Data Sync dashboard: this is the in-memory layer **above** the node store, so a miss here is what causes a read there. The full-below cache is not reported — it is a `KeyCache` whose only lookup path increments `stats_.hits`/`stats_.misses` while `getHitRate()` reads the separate `hits_`/`misses_` members, so its rate is hard-wired to 0 until that accounting is fixed. | -| `sync_acquire_no_progress_total` | counter | `InboundLedger.cpp` — `InboundLedger::onTimer` | Acquire Stall Rate (no progress) | Acquire timeouts where not one new node arrived since the previous timeout, from the `progress_` flag that was previously log-only. Fires on the 3 s acquire timer, never per node. A sustained rate together with a flat missing-node count is the definitive "stuck, not slow" signature. | -| `sync_addnode_total` (`outcome` = `good` \| `duplicate` \| `invalid`) | counter | `InboundLedger.cpp` — `InboundLedger::recordBatchOutcome` | Add-Node Outcomes | SHAMap nodes received during acquire, split by result. Emitted once per received packet from the aggregated batch tally the trace log already printed — never inside the per-node `receiveNode()` loop. Separates real progress (`good`) from wasted bandwidth (`duplicate`) and a misbehaving peer (`invalid`), all three of which look like healthy throughput in traffic metrics. | -| `sync_acquire_source_total` (`source` = `local` \| `network`) | counter | `InboundLedger.cpp` — `InboundLedger::init` | Acquire Source (local vs network) | Whether an acquire was satisfied entirely from the local node store or needed peers, emitted once per new acquire after the first local lookup. Sustained `network` on a node that should already hold the range means sync is disk-bound rather than peer-bound. | -| `jobq_saturation` (`metric` = `running_tasks` \| `worker_threads` \| `total_waiting`) | observable gauge | `MetricsRegistry.cpp` — `registerJobQueueSaturationGauge` | Worker Pool Saturation; Worker Pool Capacity & Total Backlog | Global worker-pool saturation from `JobQueue::getWorkerSaturation()`: tasks in flight, threads the pool is configured to run, and jobs queued across all types, all from one reading so the ratio and the backlog describe the same instant. `worker_threads` is exported rather than hardcoded in the dashboard because it is derived at startup from `[workers]`, node size and hardware concurrency. Exists separately from the per-job-type gauges `JobQueue::collect()` publishes (`jobq__waiting` / `_running` / `_deferred`) because a pool-wide slowdown otherwise appears as an independent fault in every subsystem queued behind it; a `running_tasks / worker_threads` ratio at 1.0 **with** a non-zero `total_waiting` attributes it to pool exhaustion once. | -| `peer_ledger_supply` (`metric` = `peers_reporting` \| `peers_serving_validated` \| `peers_serving_next` \| `supply_min_seq` \| `supply_max_seq`) | observable gauge | `MetricsRegistry.cpp` — `registerPeerLedgerSupplyGauge` (aggregating `OverlayImpl::getPeerLedgerSupply`) | Peers Able to Serve Needed Sequence; Peer Supply Window Margin (history headroom vs tip gap) | How much of the sequence range this node needs its connected peer set can actually serve, from one pass over the active peers reading the range each already advertised in `mtSTATUS_CHANGE`. **`peers_serving_next` is the signal this exists for:** zero there with a non-zero `peers_reporting` means no connected peer holds validated + 1, so the peer set must change and waiting cannot finish the sync. `peers_reporting` is the denominator that makes the rest readable — peers advertising `[0, 0]` have not reported yet and are excluded from every field, so they cannot make a healthy peer set appear to serve from genesis; when nothing has reported, both window fields read 0 meaning **unknown**, not genesis. `supply_min_seq` / `supply_max_seq` separate "asking for history nobody kept" from "asking for a tip nobody reached". The _Peer Supply Window Margin_ panel renders both as distances from `server_info{metric="validated_ledger_seq"}` rather than as absolute sequences, because the raw values sit around 1.05e8 and roughly 3e5 apart, so one linear axis flattens the tip movement that shows whether sync is progressing; the subtraction also makes zero the boundary in both directions. Both operands are gated `> 0` in PromQL so the unknown-window sentinel cannot turn into a whole-sequence-space spike when subtracted. Distinct from `server_info{metric="peers"}`, a bare connection count with no notion of what those peers hold; from `sync_state{metric="ledgers_behind"}`, which uses the same per-peer maxima but collapses them to a single distance-to-tip number that cannot say how many peers can serve that distance or whether the range has a hole; and from `peer_quality{metric="peers_insane_count"}`, which counts peers on a different chain and is therefore a correctness signal, not an availability one. | -| `peer_disconnect_total` (`reason` = `graceful` \| `shutdown` \| `stopping` \| `read_error` \| `write_error` \| `timer_error` \| `ping_timeout` \| `not_useful` \| `large_sendq` \| `charge_resources` \| `malformed_handshake` \| `shared_value` \| `unknown`; `direction` = `inbound` \| `outbound`) | counter | `PeerImp.cpp` — `PeerImp::close` | Peer Disconnects by Reason | Peer teardowns split by cause and by which side opened the connection. Emitted once per teardown at `close()`, the single funnel every disconnect path passes through, and `close()` already self-guards on the socket being open, so a repeated close cannot double-count and the total matches the existing unlabelled tally. `reason` is set by whichever site decided to disconnect, first writer wins, so a later generic reason never masks the real one; the value is always one of a fixed set of literals in `PeerImp.cpp`, never peer-supplied data, so cardinality is bounded by the code. The split is the whole point: it separates our-fault backpressure (`large_sendq`, `charge_resources`) from topology and network faults (`not_useful`, `ping_timeout`, `read_error`), and normal churn (`graceful`) from either. Distinct from the existing `server_info{metric="peer_disconnects_resources"}`, which counts only the resource-charge subset and carries no labels, and from the StatsD `overlay_peer_disconnects`, which is the unlabelled grand total in which every reason above collapses into one number. | -| `peer_accept_total` (`outcome` = `accepted` \| `local_endpoint_fail` \| `resource_limit` \| `no_slot` \| `not_peer_request` \| `protocol_mismatch` \| `bad_cookie` \| `slot_refused` \| `handshake_error`) | counter | `OverlayImpl.cpp` — `OverlayImpl::onHandoff` via `reportAcceptOutcome` | Inbound Peer Accept Outcomes | Terminal outcome of every inbound connection this node is offered, one emit per handoff. `accepted` is reported only after `run()`, so anything that threw on the way lands on `handshake_error` instead; the two early returns that are not peer attempts at all (a handled HTTP request, and a request that never asked to upgrade) are deliberately not counted. The `outcome` names the stage that refused: no local endpoint, the resource manager, PeerFinder having no slot or seeing a duplicate, a non-peer upgrade request, protocol version disagreement, a bad security cookie, or activation being refused. This is the **inbound twin** of the existing `overlay_connect_total{outcome}`, which covers outbound dials only; without it a node refusing every inbound connection is indistinguishable from one nobody dials, and reading the two together gives the full in/out split. | -| `peerfinder_slot_census` (`metric` = `out_active` \| `out_max` \| `in_active` \| `in_max` \| `connecting` \| `fixed_configured` \| `fixed_active` \| `bootcache` \| `livecache`) | observable gauge | `MetricsRegistry.cpp` — `registerSlotCensusGauge` (from `Logic::getSlotCensus`) | PeerFinder Slot Census; PeerFinder Address Caches & Fixed Peers | Slot occupancy against capacity, outbound dials in flight, configured-versus-connected fixed peers, and the depth of both address caches. All nine come from a single acquire of the PeerFinder lock, so they are mutually consistent, share one label set and can be compared against each other. That is what makes the three most common bootstrap failures visible: `connecting` non-zero while `out_active` stays below `out_max` (dials starting and never completing), `bootcache` and `livecache` both at 0 (nothing to dial at all), and `fixed_active` below `fixed_configured` (a peer named in the configuration is unreachable). `fixed_configured` is the count of peers named in the config, so the pair an operator reads is "how many did I ask for" against "how many do I have" — the same comparison `autoconnect()` makes. All nine values already existed inside PeerFinder; only two of them were exported, as the legacy beast::insight gauges `peer_finder_active_inbound_peers` and `peer_finder_active_outbound_peers`. Those two carry no capacity, attempt or cache term, are read at unrelated instants, and so cannot be joined with each other let alone with a capacity term — leaving all three failures above indistinguishable from a node that is simply not dialling. | -| `serve_refused_total` (`request` = `ledger` \| `txset` \| `object` \| `fetchpack`; `reason` = `sendq_full` \| `load_shed` \| `not_found` \| `no_map` \| `bad_type` \| `empty_reply`) | counter | `PeerImp.cpp` — `processLedgerRequest`, `onMessage(TMGetObjectByHash)`, `doFetchPack` | Ledger/Object Serve Refusals | Peer data requests this node declined to answer, split by what was asked for and why. This is the **supply side** of the sync exchange — what this node refuses to serve OTHERS — and nothing equivalent existed before, so a node shedding every ledger request looked identical to one being asked for nothing. `sendq_full` and `load_shed` are self-inflicted backpressure (the send queue at `Tuning::kDropSendQueue`, or the local fee track loaded, or too many pack jobs queued), while `not_found` is a genuine history gap and `no_map` / `bad_type` / `empty_reply` mean the request was answerable in principle but produced nothing to send. `fetchpack` is counted apart from `ledger` because a fetch pack is how a syncing peer catches up in bulk and its shed threshold is a different one. Emitted at most once per request — `empty_reply` is reported after the node loop, never inside it — and both labels are code literals, so cardinality is bounded at compile time. | -| `amendment_block` (`metric` = `warned` \| `seconds_to_block`) | observable gauge | `MetricsRegistry.cpp` — `registerAmendmentBlockGauge` | Amendment Block Countdown; Amendment Warned | `warned` is 1 once an unsupported amendment has reached majority (`NetworkOPs::isAmendmentWarned()`, previously only an admin-only `server_info` warning). **`seconds_to_block` is the leading indicator:** seconds until that amendment activates, from `AmendmentTable::firstUnsupportedExpected()` against the network close time. It reads `-1` when nothing is pending — a distinct healthy value rather than a missing series, matching the sentinel `validator_health{metric="unl_expiry_days"}` already uses — and is clamped at 0 rather than going negative, because past-due means the block is imminent, not overdue by some amount worth charting; the subtraction is done in `std::int64_t` so a past-due activation cannot wrap. Amendment-blocked is a terminal sync blocker: the node stops validating and never resumes without a software upgrade. The existing `validator_health{metric="amendment_blocked"}` reports that state after the fact, when nothing can be done about it; this gauge is the window before it, which is the only actionable part. The blocking amendment's identity is deliberately **not** a label — the network can vote on an arbitrary 256-bit amendment id, not drawn from this build's known features, so an id label would be unbounded cardinality and would mint a permanent new series per amendment. The id is available in logs from `AmendmentTableImpl::doValidatedLedger` ("Unsupported amendment \ reached majority at ..."), correlated to this series by node and time. | -| `ledger_jump_total` | counter | `NetworkOPs.cpp` — `NetworkOPsImp::switchLastClosedLedger` | Byzantine Ledger Jumps | Forced jumps of the last closed ledger onto a divergent chain: the node was told the network's LCL is not the one it built on and discarded its own chain tip to follow. Nothing equivalent existed — this was log-only ("JUMP last closed ledger to ..."), so a node repeatedly thrashing between chains left no time series to correlate against the rest of the sync pipeline. Any non-zero rate is abnormal by construction; repeated jumps are wrong-chain thrash, which points at the peer set and the configured network id rather than anywhere in the acquire pipeline. Deliberately unlabelled: the ledger hash and sequence would both be unbounded as label values, and the log line beside the emit already carries them. | -| `nodestore_state` (`metric` = `write_mean_us` \| `read_mean_us` \| `node_writes` \| `node_reads_total` \| `node_writes_duration_us` \| `node_reads_duration_us`) | observable gauge | `MetricsRegistry.cpp` — `registerNodeStoreGauge` (`observeNodeStoreTotals`) | NodeStore Write vs Read Latency (us/op); NodeStore Operation Rate (writes vs reads) | The store/fetch latency half of the pre-existing `nodestore_state` gauge (its I/O counters and queue depth are tabled under NodeStore I/O above). Mean microseconds per node-store store and per fetch, with both operation counts and both cumulative duration totals so a panel can divide the two _rates_ and read interval latency instead of the since-boot average. **The write side is the signal.** `storeDurationUs_` was declared in `Database.h` and never written, and no accessor existed, so no write-path latency was observable anywhere; the read total was already exposed as `nodestore_state{metric="node_reads_duration_us"}`. This is the fingerprint of the "a node with a large existing DB syncs slower than a fresh one" symptom, which is write-bound and therefore invisible in every read-side metric. All three concrete store paths now time themselves through `Database::recordStoreDuration` (`DatabaseNodeImp::store`, `DatabaseRotatingImp::store` and `Database::importInternal`), so `write_mean_us` is live on an ordinary node rather than only on the `[import_db]` admin path. Chosen as a gauge over a histogram deliberately: a histogram gives true percentiles but costs one `Record()` per node object on the store/fetch path, and a single ledger write walks thousands of SHAMap nodes — this gauge instead reads existing atomics once per ~10 s tick and adds nothing to the hot path. Consequence: **p99 is not obtainable from this signal**, and a histogram added later would also need an explicit-bucket View (`addMicrosecondHistogramView`) because the SDK default buckets top out at 10,000. Related to the Ledger Data Sync dashboard's NuDB Read Latency panel, which divides the two read-side fields in PromQL; that panel predates the write numerator and has no write-side equivalent. Each mean is computed by `MetricsRegistry::scaledMean`, which saturates at `INT64_MAX` rather than wrapping and returns no value when its count is 0 — so the series is omitted instead of reporting a misleading 0 µs, while the four totals are always observed. A short-lived duplicate `nodestore_latency` gauge published the same six values from the same accessors and has been retired. | -| `sweep_malloc_trim_us` | histogram | `Application.cpp` — `ApplicationImp::trimHeapAndRecord` | Sweep Heap-Trim Duration (p50/p95) | Wall-clock duration of the `malloc_trim` call that ends every cache sweep. **This is the leading explanation for "a node with a large existing DB syncs slower than a fresh one" on glibc:** the trim runs after EVERY sweep, its cost scales with the resident heap, and the pages it hands back must be re-faulted as the caches refill. The numbers all already existed on `MallocTrimReport`, but were unreachable twice over — the whole measurement block sat inside `if (journal.debug())` in `MallocTrim.cpp`, so an ordinary node at default log level measured nothing, and the return value was then discarded at the call site. The gate now covers only the `JLOG`; measuring costs about 6 µs (two `/proc/self/statm` reads at ~2.8 µs and two `getrusage` calls at ~0.17 µs) against a trim that costs milliseconds on a large heap, at a cadence of `SizedItem::SweepInterval` (10-120 s by node size) — a duty cycle below 1e-6 %, so keeping the RSS read debug-only would only have preserved the blind spot. Needs an explicit-bucket View (`addMicrosecondHistogramView`) because a trim on a multi-gigabyte heap runs well past the SDK default ceiling of 10,000, which would collapse exactly the slow trims this signal exists to catch into one saturated bucket. | -| `sweep_malloc_trim_minor_faults_total` | counter | `Application.cpp` — `ApplicationImp::trimHeapAndRecord` | Sweep Heap-Trim Faults & Reclaim Rate | Minor page faults taken _inside_ the trim call, from the `getrusage(RUSAGE_THREAD)` delta the report already carried. **Honest limitation, and it must not be over-claimed:** the delta is scoped to the trim call only, so it proves the trim itself faults — it does NOT prove the trim causes the faults taken later, as the caches refill and touch the pages the trim returned. That later re-fault cost is the actual mechanism the hypothesis proposes and it is NOT measured by this counter. Read the duration against sweep-job queueing rather than treating this number as the total cost of trimming. Emitted only when the delta is above zero: a trim that faulted nothing publishes no series, because a zero would read as "measured, and free" when the honest statement is that there was nothing to fault on. | -| `sweep_malloc_trim_reclaimed_kb_total` | counter | `Application.cpp` — `ApplicationImp::trimHeapAndRecord` | Sweep Heap-Trim Faults & Reclaim Rate | Resident kilobytes the trim actually returned to the kernel, so the cost above can be judged against what it bought. Cumulative and sign-corrected: `MallocTrimReport::deltaKB()` is after-minus-before, so a successful trim is NEGATIVE and the emit site publishes its magnitude. A sweep across which RSS GREW — another thread allocating faster than the trim released — is dropped rather than negated, because a counter cannot decrease and there is no reclaim of a negative size. Zero reclaim beside a non-zero duration is the worst reading: the trim is walking the heap and freeing nothing, which is pure cost. | -| `rotation_state` (`metric` = `in_flight` \| `copy_forward`) | observable gauge | `MetricsRegistry.cpp` — `registerRotationStateGauge` | Online-Delete Rotation Window & Copy-Forward Writes | The online-delete rotation window, and the running total of the extra writes it forces. A rotation rewrites into the new backend any node body the doomed archive serves, which is I/O an ordinary fetch would never perform and which scales with the archive — so it appears only on a populated, already-rotated database, which is precisely why it never shows on a fresh node. `copy_forward` comes from `DatabaseRotatingImp::copyForwardCount_`, which existed but was log-only AND reset by `rotate()` on every swap; a series that drops to zero per rotation cannot be rated, so a second never-reset total was added beside it and this gauge reads that. `in_flight` is exposed because the extra writes only happen inside that window, so a panel needs to know when to expect the total to move; the same total climbing while the flag reads 0 would mean the flag leaked, not that rotation is cheap. Polled rather than pushed because `DatabaseRotatingImp` is libxrpl and cannot include `xrpld/telemetry` — the two readings are taken through new `DatabaseRotating` accessors from the same collection tick pattern `registerNodeStoreGauge` already uses. **Publishes NO series when `online_delete` is not configured** (the `dynamic_cast` to `DatabaseRotating` fails and the callback returns early), deliberately: an absent series means "rotation is not configured", which a zero would misreport as "rotation is free". Rotation _duration_ is deliberately not recorded — see the note below the table. | -| `rotation_copy_node_restore_total` | counter | `SHAMapStoreImp.cpp` — `SHAMapStoreImp::copyNode` | Rotation Node Re-Store Rate | Nodes the rotation had to rescue because they were present in NEITHER backend, re-stored from the in-memory state map. The genuinely unmeasured rotation write: each is an extra store on top of the whole-state-map walk the rotation already performs, and the branch was warn-log-only, so the volume was invisible unless someone was reading logs. A non-zero rate says more than cost — it says an earlier rotation removed the only on-disk copy of a clean node (`cowid == 0`, so `flushDirty` skips it) that the current validated state map still reaches, and without the rescue it would later surface as an unresolvable `SHAMapMissingNode`. The node hash is deliberately not a label: it is unbounded runtime data that would mint one series per rescued node. Correlate a spike with the `copyNode` warning line in Loki, by node and time. | -| `ledger_replay_fallback_total` (`stage` = `skiplist` \| `delta`) | counter | `SkipListAcquire.cpp` / `LedgerDeltaAcquire.cpp` — `trigger` | Replay Fallback to Full Acquire (by stage) | A ledger-replay sub-task abandoning its shortcut and acquiring the whole ledger through `InboundLedger` instead, because too few connected peers support the `LedgerReplay` protocol feature. Both branches were debug-log-only, so a silently defeated replay optimisation left no metric at all — back-fill simply ran on the slower path with nothing to show why. Emitted once, on the transition into fallback, not at the acquire call, which re-runs on every later trigger. The `stage` label separates the skip-list acquire (which fetches the list of historical ledger hashes) from the per-ledger delta acquire, because they fail independently. | -| `ledger_replay_outcome_total` (`outcome` = `success` \| `timeout` \| `build_failed` \| `parameter_failed`) | counter | `LedgerReplayTask.cpp` — `LedgerReplayTask::recordOutcome` | Replay Outcomes (by terminal state) | Terminal state of every ledger-replay task, one emit per task. Every terminal path previously only set an internal `complete_`/`failed_` flag and wrote a log line, so a replay that never succeeded was indistinguishable from one that was never attempted. The outcome names the layer at fault: `timeout` means the deltas never arrived (a peer-supply problem), `build_failed` means a delta would not apply to its parent, and `parameter_failed` means a peer served a skip list inconsistent with what the task asked for — the latter two are data faults, not slowness. Read with `ledger_replay_fallback_total`: fallbacks rising while successes stay flat is replay-based catch-up degrading to full-ledger acquisition. | -| `ledger_quorum_publish` (`metric` = `trusted_validation_tally` \| `quorum_target`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Trusted Validations vs Quorum Target | Trusted validations counted at the most recent pre-accept gate, beside the number that gate required. Snapshotted in `LedgerMaster::checkAccept` before the shortfall check, so a node that keeps failing the gate still reports both numbers — which is the whole point: the tally alone cannot say whether validations are accumulating toward quorum (slow, will finish) or plateaued below it (stuck). Read the sustained floor of the tally, not a single sample: each series is a snapshot of the last evaluation, and the first evaluation of each round runs before peer validations arrive, so a healthy node sawtooths. `quorum_target` is what the gate actually demanded, as opposed to `unl_quorum{metric="quorum"}` which is what the trusted list configures. When the trusted list disables quorum entirely (`getNeededValidations` returns `SIZE_MAX`) the target is reported as int64 max rather than wrapping to -1, so it reads far above any tally instead of inverting the comparison — the same sentinel handling as the `unl_quorum` gauge. | -| `ledger_quorum_publish` (`metric` = `publish_lag`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Publish Lag (validated minus published) | Ledgers fully validated but not yet published to clients and subscribers: the validated sequence minus the published sequence, floored at zero. `pubLedgerSeq_` was never exported, so this gap was not derivable from any other series. Publishing trails validation by design and a small lag drains each round; a lag that stays positive or grows means validation is healthy and the publish pipeline is not, which is a distinct fault from anything the quorum or acquire signals can show. The two sequences are read as independent relaxed loads, so a sample taken mid-update may be off by one ledger for one poll — immaterial for a lag trend, and the price of not taking the LedgerMaster mutex on the metrics poll thread. | -| `ledger_quorum_publish` (`metric` = `time_to_first_validated_us`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Time to First Validated Ledger | Microseconds from process start until the first ledger passed the pre-accept quorum gate. A one-shot measurement like `sync_state{metric="initial_full_duration_us"}`: written once under `mutex_` and never changed, so it has no trend. Exactly two readings are meaningful — a duration, meaning the node reached its first fully-validated ledger and this is how long that took, or 0, meaning it never has. Clamped to a minimum of 1 so a genuine sub-microsecond reading can never be confused with the never-reached zero. A value here alongside a zero on time-to-first-FULL, or the reverse, separates "reached the full server state" from "fully validated a ledger". | -| `ledger_quorum_shortfall_total` (`stage` = `pre_accept`) | counter | `LedgerMaster.cpp` — `LedgerMaster::checkAccept` | Pre-Accept Quorum Shortfall Rate | One increment per pre-accept gate evaluation rejected because the trusted validation tally was below quorum. Previously trace-log-only, which made a node that peers and receives validations yet never validates indistinguishable from an idle one. A non-zero rate is **not** by itself a fault: `doAccept` issues this node's own validation and calls `consensusBuilt` → `checkAccept` immediately, before peer validations for that ledger arrive, so the first evaluation of every round tallies short and is retried as validations come in — a healthy cluster emits this counter every round. The fault signature is the rate climbing well above the ledger-close rate while the tally stays flat below its target and `time_to_first_validated_us` stays at 0. Emitted while `mutex_` is held, which is safe against the metrics poll because every accessor the sync gauges read is a lock-free atomic load, so no OTel callback ever acquires `mutex_`. | -| `consensus_round_duration_ms` | histogram | `RCLConsensus.cpp` — `RCLConsensus::Adaptor::makeAcceptSpan` | Consensus Round Duration Distribution; Consensus Round Duration (p50/p95) | Wall-clock duration of a completed consensus round, in milliseconds. Promotes the long-standing `round_time_ms` span attribute into a native instrument: the attribute answers "how long did THIS round take" inside a trace, next to the proposers and disputes that explain it, while the histogram gives the distribution over time, which is what an alert or SLO panel needs and what a raw trace query cannot cheaply produce fleet-wide. Being native it is also **never sampled**, so it stays complete when tracing is head-sampled down. Recorded at exactly one site — `makeAcceptSpan` is the single function both the synchronous (`onForceAccept`) and asynchronous (`onAccept`) accept paths call once per round — so it can neither double-count nor be skipped, and it adds no per-peer, per-proposal or per-transaction work. **Explicit buckets** are registered for it in `MetricsRegistry::initExporterAndProvider` (`addRoundDurationHistogramView`, boundaries 500 ms → 120 s): the SDK default tops out at 10,000 ms, which would collapse every slow round into one saturated bucket and read every quantile as 10 s, and the consensus parameters themselves allow a round up to `ledgerAbandonConsensus` = 120 s. Needs **no collector change** — a native metric rides the existing OTLP → Prometheus path. | -| `consensus.validation.accept` (`validation_status`, `accept_gated`, `ledger_hash`, `ledger_seq`, `full_validation`) | span + span attr | `RCLValidations.cpp` — `handleNewValidation` | Trusted Validation Accept Rate by Status | One span per **trusted** validation as it reaches the ledger-acceptance gate, so its rate is bounded by the UNL size per ledger close (untrusted validations cannot move acceptance and get no span). Its trace id is derived from the **validated ledger's** hash, so it joins that ledger's trace rather than the round trace — see the per-ledger trace join below. `validation_status` is one of the six `ValStatus` values and only `current` continues to the gate, which is the difference between a node whose arriving validations are counting and one whose validations are all rejected; from the outside both look like a node that receives validations and never validates. `accept_gated` is true when another thread was already accepting the same ledger, which is why a trace can show a validation with no `ledger.validate` after it. Both are spanmetrics dimensions in **both** collector configs (6 and 2 values, bounded); `ledger_hash` / `ledger_seq` stay span-only and Tempo-indexed, since a per-ledger metric dimension would mint one series per ledger. | -| Per-ledger trace join (`ledger_hash` as trace-id seed) | trace scheme | `LedgerMaster.h/.cpp` — `LedgerMaster::makeLedgerTraceSpan` | n/a — read in Tempo, `{span.ledger_hash="LEDGER_HASH"}` | Makes one slow ledger readable as **one connected trace** instead of a set of orphan spans on different threads. `ledger.validate` (`LedgerMaster::checkAccept`), `ledger.store` (`LedgerMaster::storeLedger`) and `consensus.validation.accept` (`handleNewValidation`) each derive their trace id from the **same 32-byte ledger hash** via `SpanGuard::hashSpan`, which seeds the trace id from `hash[0:16]`. Nothing is propagated between the threads: every one of those sites already holds the ledger hash, which is the whole reason the key was chosen — the same pattern the apply pipeline uses to join `tx.preflight` / `tx.preclaim` / `tx.transactor` on the transaction id (`libxrpl/tx/applySteps.cpp`). Each span is a **true root** (deterministic trace id, empty parent), so the ledger's spans are siblings in one trace rather than a parent/child chain, which is the honest shape: none causes another directly and their order varies with the sync path (`checkAccept` is entered from a peer thread via `handleNewValidation`, from the acquire-completion job, and from the consensus thread via `switchLCL`). The full hash is also recorded as the `ledger_hash` attribute — it is what an operator searches by, and it is how a reader confirms two spans are genuinely the same ledger rather than a trace-id coincidence, since the trace id is only the leading 16 bytes. Asserted end-to-end by the `trace_join_groups` block in `expected_spans.json` (`assert_trace_join_groups` in `validate_telemetry.py`), which fails CI if the members stop sharing a trace. | -| `ledger.acquire` span (`outcome` = `complete` \| `failed` \| `abandoned`; `acquire_reason`, `timeouts`, `peer_count`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::init` / `InboundLedger::finalizeAcquireSpan` | Ledger Acquire Phase Duration (p95 by phase) (its three phase children); Ledger Acquire Duration (Inbound Fetch) and Ledger Acquire Rate by Outcome, both on the **node-health** board | Parent of the three phase spans: one whole fetch of one missing ledger, from the first request to the terminal state. Pre-existing since Phase 6, extended here with `ledger_hash` (set at `init()`, so a fetch that never finishes is still findable in a trace search, and it is the trace-id seed that joins this span to the `ledger.validate`, `ledger.store` and `consensus.validation.accept` spans for the same ledger) and with the fourth `outcome` value `abandoned`, recorded when the acquire is destroyed by a sweep or shutdown before reaching a result. Without `abandoned` a stuck-then-swept fetch left the span with no `outcome` at all, so it vanished from every outcome rate — the exact failure a stalled fresh sync produces. `ledger_seq` is re-stamped at the end because a by-hash acquire starts with `seq_ == 0` and learns the sequence only when the header arrives. | -| `ledger.acquire.header` span (`outcome`, `timed_out`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::syncPhaseSpans` / `endPhaseSpan` | Ledger Acquire Phase Duration (p95 by phase); Ledger Acquire Phase Outcomes (by phase & timeout) | Child of `ledger.acquire` covering the wait for the ledger header, which gates both tree phases — until it arrives the account-state and transaction root hashes are unknown, so nothing else can even be requested. The parent span is flat and its duration is dominated by the state tree, so a node stuck waiting to be TOLD what to fetch was indistinguishable from one stuck fetching it. No `missing_nodes`: a header is a single object, not a tree. Opened and closed by one idempotent state sync over the `have*_` flags rather than by open/close calls scattered through the fetch code, so the span boundary cannot drift out of step with the real phase boundary. | -| `ledger.acquire.astree` span (`outcome`, `timed_out`, `missing_nodes`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::syncPhaseSpans` / `endPhaseSpan` | Ledger Acquire Phase Duration (p95 by phase); Ledger Acquire Phase Outcomes (by phase & timeout) | Child of `ledger.acquire` covering the account-state SHAMap fetch — **nearly all of the work in a real fresh sync**, and the reason the phase split exists: the flat parent span could not separate it from the small transaction tree. `missing_nodes` is read from the count `getMissingNodes()` already produced during its sweep, never recomputed, so no second tree walk is added. `outcome=timeout` together with a non-zero `missing_nodes` is the "peers are not serving this tree" signature; `timed_out` is a separate dimension from `outcome` because a phase can time out and still be retried by its parent acquire. | -| `ledger.acquire.txtree` span (`outcome`, `timed_out`, `missing_nodes`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::syncPhaseSpans` / `endPhaseSpan` | Ledger Acquire Phase Duration (p95 by phase); Ledger Acquire Phase Outcomes (by phase & timeout) | Child of `ledger.acquire` covering the transaction SHAMap fetch. Usually completes long before the account-state phase, and that asymmetry is the point of separating them: the parent span's duration is the state tree's, not this one's, so a transaction tree that is genuinely slow is invisible inside it. Closed the moment its own tree completes (from `receiveNode`, `trigger` or `takeHeader`), so its duration is the real fetch time rather than stretching to the next trigger. | -| `txset.acquire` span (`outcome`, `txset_hash`, `duration_ms`, `timeouts`, `peer_count`) | span | `TransactionAcquire.cpp` — `TransactionAcquire::finalizeAcquireSpan` | Tx-Set Acquire Outcomes; Tx-Set Acquire Duration (p95) | One attempt to fetch the transaction set a consensus proposal referenced but this node did not hold. `TransactionAcquire` had **zero** telemetry of any kind before this, so a consensus round stalled waiting on a set was indistinguishable from an idle one. The sibling of `ledger.acquire`: same `TimeoutCounter` base, same trigger/onTimer/takeNodes shape, and the same `trace_ledger` flag so the two halves of a stuck sync cannot be enabled apart. `outcome` is `complete` \| `failed` \| `timeout` \| `abandoned`, stamped on both exits (`done()`, and the destructor when the round sweep in `InboundTransactions::newRound` drops a set that never arrived) by one idempotent finalizer. `timeout` is distinct from `failed` because the exhausted-budget path sets the terminal `failed_` flag too — that flag is how the timer loop stops — so the outcome rule checks the timeout first or every timeout would read as a data fault. `txset_hash` identifies WHICH set stalled and stays span-only: one metric series per consensus round would be unbounded. | -| `ledger.serve` span (`object_type`, `outcome`, `served_nodes`, `peer_id`, `ledger_seq`) | span | `PeerImp.cpp` — `PeerImp::processLedgerRequest` (the `JtLedgerReq` worker) | Ledger Serve Rate by Object Type | This node answering a peer's `TMGetLedger` request — the **supply side** of the sync exchange, and the trace-level companion to the existing `serve_refused_total` counter. The whole serve path had no span, so how long this node takes to answer, and whether it answered at all, was unobservable. A fresh trace root, because the request arrives from the wire on a shared worker whose ambient span is unrelated. `object_type` (`header` \| `tx` \| `as` \| `txset`) and `outcome` (`complete` \| `partial` \| `refused`) are both derived by shared rules in `LedgerSpanNames.h` rather than named per branch, which is what stops the eight exits of `processLedgerRequest` disagreeing about one request. `outcome` is derived from the reply itself — `served_nodes` is the reply's own node count and is 0 on all seven refusal paths — so nothing is accumulated and no work is added to the per-node assembly loop. `partial` means the reply hit `Tuning::kSoftMaxReplyNodes`, so the peer must make another round trip. | -| `peer.dial` span (`outcome`, `remote_endpoint`, `duration_ms`) | span | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Outcomes (span-derived, per attempt) | One outbound connect attempt, as a per-attempt timeline rather than a rate. The trace-level companion to `overlay_connect_total` / `overlay_dial_latency_ms`: it carries the same five `outcome` values, set from the same `reportOutcome` funnel, so span and counter cannot disagree, and the funnel's existing first-call-wins guard makes the span exactly-once for free. What it adds is `remote_endpoint` — WHICH peer — which the counter deliberately cannot carry, because one series per peer address would be unbounded cardinality; it is a dedicated Tempo span column instead. A fresh trace root: a dial is the first thing a starting node does, so there is nothing to parent it to. An attempt torn down mid-dial by shutdown ends its span in the destructor with no `outcome`, which is the honest record of "never concluded" rather than a dropped span. | +| Signal | Type | Emit site | Panel | Meaning | +| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | -------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `dns_resolve_total` (`outcome` = `resolved` \| `empty`) | counter | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Outcome Rate | Peer hostname resolutions. `empty` means a configured bootstrap or `[ips_fixed]` name returned no address, so that peer is never dialled. | +| `dns_resolve_latency_ms` | histogram | `OverlayImpl.cpp` — `OverlayImpl::reportDnsResolve` | DNS Resolve Latency (p95) | Time to resolve a configured peer hostname. Seconds-scale values mean the resolver is timing out ahead of every dial. | +| `overlay_connect_total` (`outcome` = `connected` \| `tcp_fail` \| `tls_fail` \| `self_connection` \| `upgrade_fail` \| `timeout`) | counter | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Outcome Rate | Outbound peer connection attempts by terminal outcome. The outcome names the stage that broke: TCP, TLS, HTTP upgrade, or no terminal state in time. | +| `overlay_dial_latency_ms` | histogram | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Latency (p95) | Time from starting an outbound dial to its terminal outcome, successes and failures together. A p95 near the dial timeout means peers accept TCP but never finish the handshake. | +| `handshake_negotiation_fail_total` (`reason`, 14 values incl. `wrong_network`, `invalid_network_id`, `clock_skew`, `self_connection`, `session_verify_failed`) | counter | `Handshake.cpp` — `throwNegotiationFailure` (from `verifyHandshake`) | Handshake Negotiation Failures by Reason | Peer handshakes rejected after TLS while checking network id, clock, keys and addresses. `reason` names the failing check. | +| `unl_fetch_total` (`site` = UNL site as scheme://host/path, userinfo and query stripped; `outcome` = the 9 `ListDisposition` strings `accepted` \| `expired` \| `same_sequence` \| `pending` \| `known_sequence` \| `unsupported_version` \| `untrusted` \| `stale` \| `invalid`, plus `fetch_error` \| `bad_status` \| `parse_error`) | counter | `ValidatorSite.cpp` — `ValidatorSite::reportFetchOutcome` | UNL Fetch Rate by Site & Outcome | Validator-list fetches per site. `accepted` is the only success; `same_sequence` and `known_sequence` are normal no-op refreshes; the three literals are transport or content faults. | +| `unl_quorum` (`metric` = `trusted_keys` \| `quorum` \| `quorum_disabled`) | observable gauge | `MetricsRegistry.cpp` — `registerUnlQuorumGauge` | UNL Trusted Keys vs Quorum; UNL Quorum Headroom | Trusted UNL key count against the validations a ledger needs. `trusted_keys` at or below `quorum` means the node can never declare a ledger validated. | +| `clock_close_offset_seconds` (`metric` = `offset`) | observable gauge | `MetricsRegistry.cpp` — `registerClockSkewGauge` | Clock Close Offset | Network close time offset from the local clock. Negative means the local clock runs ahead. `server_info` only surfaces `close_time_offset` at 60 s or more, so this gauge sees skew far earlier. | +| `state_changes_total` (`from`, `to` = `disconnected` \| `connected` \| `syncing` \| `tracking` \| `full`) | counter | `NetworkOPs.cpp` — `NetworkOPsImp::setMode` | Mode Transitions by Edge | Operating-mode transitions keyed on the (`from`, `to`) edge. The edge is what separates a clean `disconnected`→`connected`→`syncing`→`tracking`→`full` climb from `full`→`connected` flapping; an unlabelled total cannot tell them apart. | +| `sync_state` (`metric` = `initial_full_duration_us`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Time to First FULL | Microseconds from process start to the first `full` transition, sourced from `NetworkOPs::getInitialSyncDurationUs()`. Stays 0 until `full` is reached, so a flat 0 is itself the "never synced" signal; once set it never changes. | +| `sync_state` (`metric` = `network_ledger_gate`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Network Ledger Gate | 1 while the node is still waiting to see a full network ledger (`NetworkOPs::isNeedNetworkLedger()`), else 0. A persistent 1 blocks transaction submission and `full`, whatever the rest of the pipeline shows. | +| `sync_state` (`metric` = `server_stall_seconds`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Server Stall | Current main-loop stall duration from `LoadManager::getCurrentStallSeconds()`, 0 when healthy. Same duration the load monitor logs as "Server stalled for N seconds", which previously existed only in that log line. | +| `sync_state` (`metric` = `ledgers_behind`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncStateGauge` | Ledgers Behind Network | Peer-reported network tip minus our validated sequence, floored at 0 (`NetworkOPs::getLedgersBehindNetwork()`). Reads each peer's already cached ledger range, so no new network round trip. | +| `server_stall_events_total` | observable counter | `MetricsRegistry.cpp` — `registerStallEventsCounter` | Server Stall Event Rate | Distinct stall episodes since process start, counted once per episode rather than per stalled second. A rising rate is repeated fresh stalls; a flat rate with a large `server_stall_seconds` is one long stall. | +| `sync_acquire` (`metric` = `missing_state_nodes_max` \| `missing_tx_nodes_max`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Missing SHAMap Nodes per Acquire (state/tx) | Largest outstanding SHAMap node count across in-flight acquires, split by tree, from the count `getMissingNodes()` already produces during its sweep (`InboundLedger.cpp` — `InboundLedger::trigger`). **The headline stuck-sync signal:** flat and non-zero across ticks means the acquire will never finish; shrinking means slow but alive. Aggregated as a max rather than labelled per ledger, because a `ledger_seq` label would mint one series per ledger acquired — per-ledger identity stays on the `ledger.acquire` span. | +| `sync_acquire` (`metric` = `received_data_depth`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Peer packets stashed across all in-flight acquires waiting to be applied, summed because it measures one shared processing backlog. A growing depth means arriving node data outpaces processing, so the limit is the job queue or disk rather than peer supply. | +| `sync_acquire` (`metric` = `in_flight`) | observable gauge | `MetricsRegistry.cpp` — `registerSyncAcquireGauge` | Received-Data Stash Depth & In-Flight Acquires | Number of ledger acquires currently running. Exported so the three values above can be read in context: all zero with `in_flight` zero is an idle node, not a healthy one. | +| `shamap_cache_hit_rate` (`metric` = `treenode`) | observable gauge | `MetricsRegistry.cpp` — `registerCacheHitRateDetailGauge` | SHAMap TreeNode Cache Hit Rate | Share of SHAMap tree-node lookups served from memory, from the previously-uncalled `TaggedCache::getHitRate()`, normalized from 0-100 to 0.0-1.0. Distinct from `nodestore_state`-derived NuDB Cache Hit Ratio on the Ledger Data Sync dashboard: this is the in-memory layer **above** the node store, so a miss here is what causes a read there. The full-below cache is not reported — it is a `KeyCache` whose only lookup path increments `stats_.hits`/`stats_.misses` while `getHitRate()` reads the separate `hits_`/`misses_` members, so its rate is hard-wired to 0 until that accounting is fixed. | +| `sync_acquire_no_progress_total` | counter | `InboundLedger.cpp` — `InboundLedger::onTimer` | Acquire Stall Rate (no progress) | Acquire timeouts where not one new node arrived since the previous timeout, from the `progress_` flag that was previously log-only. Fires on the 3 s acquire timer, never per node. A sustained rate together with a flat missing-node count is the definitive "stuck, not slow" signature. | +| `sync_addnode_total` (`outcome` = `good` \| `duplicate` \| `invalid`) | counter | `InboundLedger.cpp` — `InboundLedger::recordBatchOutcome` | Add-Node Outcomes | SHAMap nodes received during acquire, split by result. Emitted once per received packet from the aggregated batch tally the trace log already printed — never inside the per-node `receiveNode()` loop. Separates real progress (`good`) from wasted bandwidth (`duplicate`) and a misbehaving peer (`invalid`), all three of which look like healthy throughput in traffic metrics. | +| `sync_acquire_source_total` (`source` = `local` \| `network`) | counter | `InboundLedger.cpp` — `InboundLedger::init` | Acquire Source (local vs network) | Whether an acquire was satisfied entirely from the local node store or needed peers, emitted once per new acquire after the first local lookup. Sustained `network` on a node that should already hold the range means sync is disk-bound rather than peer-bound. | +| `jobq_saturation` (`metric` = `running_tasks` \| `worker_threads` \| `total_waiting`) | observable gauge | `MetricsRegistry.cpp` — `registerJobQueueSaturationGauge` | Worker Pool Saturation; Worker Pool Capacity & Total Backlog | Global worker-pool saturation from `JobQueue::getWorkerSaturation()`: tasks in flight, threads the pool is configured to run, and jobs queued across all types, all from one reading so the ratio and the backlog describe the same instant. `worker_threads` is exported rather than hardcoded in the dashboard because it is derived at startup from `[workers]`, node size and hardware concurrency. Exists separately from the per-job-type gauges `JobQueue::collect()` publishes (`jobq__waiting` / `_running` / `_deferred`) because a pool-wide slowdown otherwise appears as an independent fault in every subsystem queued behind it; a `running_tasks / worker_threads` ratio at 1.0 **with** a non-zero `total_waiting` attributes it to pool exhaustion once. | +| `peer_ledger_supply` (`metric` = `peers_reporting` \| `peers_serving_validated` \| `peers_serving_next` \| `supply_min_seq` \| `supply_max_seq`) | observable gauge | `MetricsRegistry.cpp` — `registerPeerLedgerSupplyGauge` (aggregating `OverlayImpl::getPeerLedgerSupply`) | Peers Able to Serve Needed Sequence; Peer Supply Window Margin (history headroom vs tip gap) | How much of the sequence range this node needs its connected peer set can actually serve, from one pass over the active peers reading the range each already advertised in `mtSTATUS_CHANGE`. **`peers_serving_next` is the signal this exists for:** zero there with a non-zero `peers_reporting` means no connected peer holds validated + 1, so the peer set must change and waiting cannot finish the sync. `peers_reporting` is the denominator that makes the rest readable — peers advertising `[0, 0]` have not reported yet and are excluded from every field, so they cannot make a healthy peer set appear to serve from genesis; when nothing has reported, both window fields read 0 meaning **unknown**, not genesis. `supply_min_seq` / `supply_max_seq` separate "asking for history nobody kept" from "asking for a tip nobody reached". The _Peer Supply Window Margin_ panel renders both as distances from `server_info{metric="validated_ledger_seq"}` rather than as absolute sequences, because the raw values sit around 1.05e8 and roughly 3e5 apart, so one linear axis flattens the tip movement that shows whether sync is progressing; the subtraction also makes zero the boundary in both directions. Both operands are gated `> 0` in PromQL so the unknown-window sentinel cannot turn into a whole-sequence-space spike when subtracted. Distinct from `server_info{metric="peers"}`, a bare connection count with no notion of what those peers hold; from `sync_state{metric="ledgers_behind"}`, which uses the same per-peer maxima but collapses them to a single distance-to-tip number that cannot say how many peers can serve that distance or whether the range has a hole; and from `peer_quality{metric="peers_insane_count"}`, which counts peers on a different chain and is therefore a correctness signal, not an availability one. | +| `peer_disconnect_total` (`reason` = `graceful` \| `shutdown` \| `stopping` \| `read_error` \| `write_error` \| `timer_error` \| `ping_timeout` \| `not_useful` \| `large_sendq` \| `charge_resources` \| `malformed_handshake` \| `shared_value` \| `unknown`; `direction` = `inbound` \| `outbound`) | counter | `PeerImp.cpp` — `PeerImp::close` | Peer Disconnects by Reason | Peer teardowns split by cause and by which side opened the connection. Emitted once per teardown at `close()`, the single funnel every disconnect path passes through, and `close()` already self-guards on the socket being open, so a repeated close cannot double-count and the total matches the existing unlabelled tally. `reason` is set by whichever site decided to disconnect, first writer wins, so a later generic reason never masks the real one; the value is always one of a fixed set of literals in `PeerImp.cpp`, never peer-supplied data, so cardinality is bounded by the code. The split is the whole point: it separates our-fault backpressure (`large_sendq`, `charge_resources`) from topology and network faults (`not_useful`, `ping_timeout`, `read_error`), and normal churn (`graceful`) from either. Distinct from the existing `server_info{metric="peer_disconnects_resources"}`, which counts only the resource-charge subset and carries no labels, and from the StatsD `overlay_peer_disconnects`, which is the unlabelled grand total in which every reason above collapses into one number. | +| `peer_accept_total` (`outcome` = `accepted` \| `local_endpoint_fail` \| `resource_limit` \| `no_slot` \| `not_peer_request` \| `protocol_mismatch` \| `bad_cookie` \| `slot_refused` \| `handshake_error`) | counter | `OverlayImpl.cpp` — `OverlayImpl::onHandoff` via `reportAcceptOutcome` | Inbound Peer Accept Outcomes | Terminal outcome of every inbound connection this node is offered, one emit per handoff. `accepted` is reported only after `run()`, so anything that threw on the way lands on `handshake_error` instead; the two early returns that are not peer attempts at all (a handled HTTP request, and a request that never asked to upgrade) are deliberately not counted. The `outcome` names the stage that refused: no local endpoint, the resource manager, PeerFinder having no slot or seeing a duplicate, a non-peer upgrade request, protocol version disagreement, a bad security cookie, or activation being refused. This is the **inbound twin** of the existing `overlay_connect_total{outcome}`, which covers outbound dials only; without it a node refusing every inbound connection is indistinguishable from one nobody dials, and reading the two together gives the full in/out split. | +| `peerfinder_slot_census` (`metric` = `out_active` \| `out_max` \| `in_active` \| `in_max` \| `connecting` \| `fixed_configured` \| `fixed_active` \| `bootcache` \| `livecache`) | observable gauge | `MetricsRegistry.cpp` — `registerSlotCensusGauge` (from `Logic::getSlotCensus`) | PeerFinder Slot Census; PeerFinder Address Caches & Fixed Peers | Slot occupancy against capacity, outbound dials in flight, configured-versus-connected fixed peers, and the depth of both address caches. All nine come from a single acquire of the PeerFinder lock, so they are mutually consistent, share one label set and can be compared against each other. That is what makes the three most common bootstrap failures visible: `connecting` non-zero while `out_active` stays below `out_max` (dials starting and never completing), `bootcache` and `livecache` both at 0 (nothing to dial at all), and `fixed_active` below `fixed_configured` (a peer named in the configuration is unreachable). `fixed_configured` is the count of peers named in the config, so the pair an operator reads is "how many did I ask for" against "how many do I have" — the same comparison `autoconnect()` makes. All nine values already existed inside PeerFinder; only two of them were exported, as the legacy beast::insight gauges `peer_finder_active_inbound_peers` and `peer_finder_active_outbound_peers`. Those two carry no capacity, attempt or cache term, are read at unrelated instants, and so cannot be joined with each other let alone with a capacity term — leaving all three failures above indistinguishable from a node that is simply not dialling. | +| `serve_refused_total` (`request` = `ledger` \| `txset` \| `object` \| `fetchpack`; `reason` = `sendq_full` \| `load_shed` \| `not_found` \| `no_map` \| `bad_type` \| `empty_reply`) | counter | `PeerImp.cpp` — `processLedgerRequest`, `onMessage(TMGetObjectByHash)`, `doFetchPack` | Ledger/Object Serve Refusals | Peer data requests this node declined to answer, split by what was asked for and why. This is the **supply side** of the sync exchange — what this node refuses to serve OTHERS — and nothing equivalent existed before, so a node shedding every ledger request looked identical to one being asked for nothing. `sendq_full` and `load_shed` are self-inflicted backpressure (the send queue at `Tuning::kDropSendQueue`, or the local fee track loaded, or too many pack jobs queued), while `not_found` is a genuine history gap and `no_map` / `bad_type` / `empty_reply` mean the request was answerable in principle but produced nothing to send. `fetchpack` is counted apart from `ledger` because a fetch pack is how a syncing peer catches up in bulk and its shed threshold is a different one. Emitted at most once per request — `empty_reply` is reported after the node loop, never inside it — and both labels are code literals, so cardinality is bounded at compile time. | +| `amendment_block` (`metric` = `warned` \| `seconds_to_block`) | observable gauge | `MetricsRegistry.cpp` — `registerAmendmentBlockGauge` | Amendment Block Countdown; Amendment Warned | `warned` is 1 once an unsupported amendment has reached majority (`NetworkOPs::isAmendmentWarned()`, previously only an admin-only `server_info` warning). **`seconds_to_block` is the leading indicator:** seconds until that amendment activates, from `AmendmentTable::firstUnsupportedExpected()` against the network close time. It reads `-1` when nothing is pending — a distinct healthy value rather than a missing series, matching the sentinel `validator_health{metric="unl_expiry_days"}` already uses — and is clamped at 0 rather than going negative, because past-due means the block is imminent, not overdue by some amount worth charting; the subtraction is done in `std::int64_t` so a past-due activation cannot wrap. Amendment-blocked is a terminal sync blocker: the node stops validating and never resumes without a software upgrade. The existing `validator_health{metric="amendment_blocked"}` reports that state after the fact, when nothing can be done about it; this gauge is the window before it, which is the only actionable part. The blocking amendment's identity is deliberately **not** a label — the network can vote on an arbitrary 256-bit amendment id, not drawn from this build's known features, so an id label would be unbounded cardinality and would mint a permanent new series per amendment. The id is available in logs from `AmendmentTableImpl::doValidatedLedger` ("Unsupported amendment \ reached majority at ..."), correlated to this series by node and time. | +| `ledger_jump_total` | counter | `NetworkOPs.cpp` — `NetworkOPsImp::switchLastClosedLedger` | Byzantine Ledger Jumps | Forced jumps of the last closed ledger onto a divergent chain: the node was told the network's LCL is not the one it built on and discarded its own chain tip to follow. Nothing equivalent existed — this was log-only ("JUMP last closed ledger to ..."), so a node repeatedly thrashing between chains left no time series to correlate against the rest of the sync pipeline. Any non-zero rate is abnormal by construction; repeated jumps are wrong-chain thrash, which points at the peer set and the configured network id rather than anywhere in the acquire pipeline. Deliberately unlabelled: the ledger hash and sequence would both be unbounded as label values, and the log line beside the emit already carries them. | +| `nodestore_state` (`metric` = `write_mean_us` \| `read_mean_us` \| `node_writes` \| `node_reads_total` \| `node_writes_duration_us` \| `node_reads_duration_us`) | observable gauge | `MetricsRegistry.cpp` — `registerNodeStoreGauge` (`observeNodeStoreTotals`) | NodeStore Write vs Read Latency (us/op); NodeStore Operation Rate (writes vs reads) | The store/fetch latency half of the pre-existing `nodestore_state` gauge (its I/O counters and queue depth are tabled under NodeStore I/O above). Mean microseconds per node-store store and per fetch, with both operation counts and both cumulative duration totals so a panel can divide the two _rates_ and read interval latency instead of the since-boot average. **The write side is the signal.** `storeDurationUs_` was declared in `Database.h` and never written, and no accessor existed, so no write-path latency was observable anywhere; the read total was already exposed as `nodestore_state{metric="node_reads_duration_us"}`. This is the fingerprint of the "a node with a large existing DB syncs slower than a fresh one" symptom, which is write-bound and therefore invisible in every read-side metric. All three concrete store paths now time themselves through `Database::recordStoreDuration` (`DatabaseNodeImp::store`, `DatabaseRotatingImp::store` and `Database::importInternal`), so `write_mean_us` is live on an ordinary node rather than only on the `[import_db]` admin path. Chosen as a gauge over a histogram deliberately: a histogram gives true percentiles but costs one `Record()` per node object on the store/fetch path, and a single ledger write walks thousands of SHAMap nodes — this gauge instead reads existing atomics once per ~10 s tick and adds nothing to the hot path. Consequence: **p99 is not obtainable from this signal**, and a histogram added later would also need an explicit-bucket View (`addMicrosecondHistogramView`) because the SDK default buckets top out at 10,000. Related to the Ledger Data Sync dashboard's NuDB Read Latency panel, which divides the two read-side fields in PromQL; that panel predates the write numerator and has no write-side equivalent. Each mean is computed by `MetricsRegistry::scaledMean`, which saturates at `INT64_MAX` rather than wrapping and returns no value when its count is 0 — so the series is omitted instead of reporting a misleading 0 µs, while the four totals are always observed. A short-lived duplicate `nodestore_latency` gauge published the same six values from the same accessors and has been retired. | +| `sweep_malloc_trim_us` | histogram | `Application.cpp` — `ApplicationImp::trimHeapAndRecord` | Sweep Heap-Trim Duration (p50/p95) | Wall-clock duration of the `malloc_trim` call that ends every cache sweep. **This is the leading explanation for "a node with a large existing DB syncs slower than a fresh one" on glibc:** the trim runs after EVERY sweep, its cost scales with the resident heap, and the pages it hands back must be re-faulted as the caches refill. The numbers all already existed on `MallocTrimReport`, but were unreachable twice over — the whole measurement block sat inside `if (journal.debug())` in `MallocTrim.cpp`, so an ordinary node at default log level measured nothing, and the return value was then discarded at the call site. The gate now covers only the `JLOG`; measuring costs about 6 µs (two `/proc/self/statm` reads at ~2.8 µs and two `getrusage` calls at ~0.17 µs) against a trim that costs milliseconds on a large heap, at a cadence of `SizedItem::SweepInterval` (10-120 s by node size) — a duty cycle below 1e-6 %, so keeping the RSS read debug-only would only have preserved the blind spot. Needs an explicit-bucket View (`addMicrosecondHistogramView`) because a trim on a multi-gigabyte heap runs well past the SDK default ceiling of 10,000, which would collapse exactly the slow trims this signal exists to catch into one saturated bucket. | +| `sweep_malloc_trim_minor_faults_total` | counter | `Application.cpp` — `ApplicationImp::trimHeapAndRecord` | Sweep Heap-Trim Faults & Reclaim Rate | Minor page faults taken _inside_ the trim call, from the `getrusage(RUSAGE_THREAD)` delta the report already carried. **Honest limitation, and it must not be over-claimed:** the delta is scoped to the trim call only, so it proves the trim itself faults — it does NOT prove the trim causes the faults taken later, as the caches refill and touch the pages the trim returned. That later re-fault cost is the actual mechanism the hypothesis proposes and it is NOT measured by this counter. Read the duration against sweep-job queueing rather than treating this number as the total cost of trimming. Emitted only when the delta is above zero: a trim that faulted nothing publishes no series, because a zero would read as "measured, and free" when the honest statement is that there was nothing to fault on. | +| `sweep_malloc_trim_reclaimed_kb_total` | counter | `Application.cpp` — `ApplicationImp::trimHeapAndRecord` | Sweep Heap-Trim Faults & Reclaim Rate | Resident kilobytes the trim actually returned to the kernel, so the cost above can be judged against what it bought. Cumulative and sign-corrected: `MallocTrimReport::deltaKB()` is after-minus-before, so a successful trim is NEGATIVE and the emit site publishes its magnitude. A sweep across which RSS GREW — another thread allocating faster than the trim released — is dropped rather than negated, because a counter cannot decrease and there is no reclaim of a negative size. Zero reclaim beside a non-zero duration is the worst reading: the trim is walking the heap and freeing nothing, which is pure cost. | +| `rotation_state` (`metric` = `in_flight` \| `copy_forward`) | observable gauge | `MetricsRegistry.cpp` — `registerRotationStateGauge` | Online-Delete Rotation Window & Copy-Forward Writes | The online-delete rotation window, and the running total of the extra writes it forces. A rotation rewrites into the new backend any node body the doomed archive serves, which is I/O an ordinary fetch would never perform and which scales with the archive — so it appears only on a populated, already-rotated database, which is precisely why it never shows on a fresh node. `copy_forward` comes from `DatabaseRotatingImp::copyForwardCount_`, which existed but was log-only AND reset by `rotate()` on every swap; a series that drops to zero per rotation cannot be rated, so a second never-reset total was added beside it and this gauge reads that. `in_flight` is exposed because the extra writes only happen inside that window, so a panel needs to know when to expect the total to move; the same total climbing while the flag reads 0 would mean the flag leaked, not that rotation is cheap. Polled rather than pushed because `DatabaseRotatingImp` is libxrpl and cannot include `xrpld/telemetry` — the two readings are taken through new `DatabaseRotating` accessors from the same collection tick pattern `registerNodeStoreGauge` already uses. **Publishes NO series when `online_delete` is not configured** (the `dynamic_cast` to `DatabaseRotating` fails and the callback returns early), deliberately: an absent series means "rotation is not configured", which a zero would misreport as "rotation is free". Rotation _duration_ is deliberately not recorded — see the note below the table. | +| `rotation_copy_node_restore_total` | counter | `SHAMapStoreImp.cpp` — `SHAMapStoreImp::copyNode` | Rotation Node Re-Store Rate | Nodes the rotation had to rescue because they were present in NEITHER backend, re-stored from the in-memory state map. The genuinely unmeasured rotation write: each is an extra store on top of the whole-state-map walk the rotation already performs, and the branch was warn-log-only, so the volume was invisible unless someone was reading logs. A non-zero rate says more than cost — it says an earlier rotation removed the only on-disk copy of a clean node (`cowid == 0`, so `flushDirty` skips it) that the current validated state map still reaches, and without the rescue it would later surface as an unresolvable `SHAMapMissingNode`. The node hash is deliberately not a label: it is unbounded runtime data that would mint one series per rescued node. Correlate a spike with the `copyNode` warning line in Loki, by node and time. | +| `ledger_replay_fallback_total` (`stage` = `skiplist` \| `delta`) | counter | `SkipListAcquire.cpp` / `LedgerDeltaAcquire.cpp` — `trigger` | Replay Fallback to Full Acquire (by stage) | A ledger-replay sub-task abandoning its shortcut and acquiring the whole ledger through `InboundLedger` instead, because too few connected peers support the `LedgerReplay` protocol feature. Both branches were debug-log-only, so a silently defeated replay optimisation left no metric at all — back-fill simply ran on the slower path with nothing to show why. Emitted once, on the transition into fallback, not at the acquire call, which re-runs on every later trigger. The `stage` label separates the skip-list acquire (which fetches the list of historical ledger hashes) from the per-ledger delta acquire, because they fail independently. | +| `ledger_replay_outcome_total` (`outcome` = `success` \| `timeout` \| `build_failed` \| `parameter_failed`) | counter | `LedgerReplayTask.cpp` — `LedgerReplayTask::recordOutcome` | Replay Outcomes (by terminal state) | Terminal state of every ledger-replay task, one emit per task. Every terminal path previously only set an internal `complete_`/`failed_` flag and wrote a log line, so a replay that never succeeded was indistinguishable from one that was never attempted. The outcome names the layer at fault: `timeout` means the deltas never arrived (a peer-supply problem), `build_failed` means a delta would not apply to its parent, and `parameter_failed` means a peer served a skip list inconsistent with what the task asked for — the latter two are data faults, not slowness. Read with `ledger_replay_fallback_total`: fallbacks rising while successes stay flat is replay-based catch-up degrading to full-ledger acquisition. | +| `ledger_quorum_publish` (`metric` = `trusted_validation_tally` \| `quorum_target`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Trusted Validations vs Quorum Target | Trusted validations counted at the most recent pre-accept gate, beside the number that gate required. Snapshotted in `LedgerMaster::checkAccept` before the shortfall check, so a node that keeps failing the gate still reports both numbers — which is the whole point: the tally alone cannot say whether validations are accumulating toward quorum (slow, will finish) or plateaued below it (stuck). Read the sustained floor of the tally, not a single sample: each series is a snapshot of the last evaluation, and the first evaluation of each round runs before peer validations arrive, so a healthy node sawtooths. `quorum_target` is what the gate actually demanded, as opposed to `unl_quorum{metric="quorum"}` which is what the trusted list configures. When the trusted list disables quorum entirely (`getNeededValidations` returns `SIZE_MAX`) the target is reported as int64 max rather than wrapping to -1, so it reads far above any tally instead of inverting the comparison — the same sentinel handling as the `unl_quorum` gauge. | +| `ledger_quorum_publish` (`metric` = `publish_lag`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Publish Lag (validated minus published) | Ledgers fully validated but not yet published to clients and subscribers: the validated sequence minus the published sequence, floored at zero. `pubLedgerSeq_` was never exported, so this gap was not derivable from any other series. Publishing trails validation by design and a small lag drains each round; a lag that stays positive or grows means validation is healthy and the publish pipeline is not, which is a distinct fault from anything the quorum or acquire signals can show. The two sequences are read as independent relaxed loads, so a sample taken mid-update may be off by one ledger for one poll — immaterial for a lag trend, and the price of not taking the LedgerMaster mutex on the metrics poll thread. | +| `ledger_quorum_publish` (`metric` = `time_to_first_validated_us`) | observable gauge | `MetricsRegistry.cpp` — `registerLedgerQuorumPublishGauge` | Time to First Validated Ledger | Microseconds from process start until the first ledger passed the pre-accept quorum gate. A one-shot measurement like `sync_state{metric="initial_full_duration_us"}`: written once under `mutex_` and never changed, so it has no trend. Exactly two readings are meaningful — a duration, meaning the node reached its first fully-validated ledger and this is how long that took, or 0, meaning it never has. Clamped to a minimum of 1 so a genuine sub-microsecond reading can never be confused with the never-reached zero. A value here alongside a zero on time-to-first-FULL, or the reverse, separates "reached the full server state" from "fully validated a ledger". | +| `ledger_quorum_shortfall_total` (`stage` = `pre_accept`) | counter | `LedgerMaster.cpp` — `LedgerMaster::checkAccept` | Pre-Accept Quorum Shortfall Rate | One increment per pre-accept gate evaluation rejected because the trusted validation tally was below quorum. Previously trace-log-only, which made a node that peers and receives validations yet never validates indistinguishable from an idle one. A non-zero rate is **not** by itself a fault: `doAccept` issues this node's own validation and calls `consensusBuilt` → `checkAccept` immediately, before peer validations for that ledger arrive, so the first evaluation of every round tallies short and is retried as validations come in — a healthy cluster emits this counter every round. The fault signature is the rate climbing well above the ledger-close rate while the tally stays flat below its target and `time_to_first_validated_us` stays at 0. Emitted while `mutex_` is held, which is safe against the metrics poll because every accessor the sync gauges read is a lock-free atomic load, so no OTel callback ever acquires `mutex_`. | +| `consensus_round_duration_ms` | histogram | `RCLConsensus.cpp` — `RCLConsensus::Adaptor::makeAcceptSpan` | Consensus Round Duration Distribution; Consensus Round Duration (p50/p95) | Wall-clock duration of a completed consensus round, in milliseconds. Promotes the long-standing `round_time_ms` span attribute into a native instrument: the attribute answers "how long did THIS round take" inside a trace, next to the proposers and disputes that explain it, while the histogram gives the distribution over time, which is what an alert or SLO panel needs and what a raw trace query cannot cheaply produce fleet-wide. Being native it is also **never sampled**, so it stays complete when tracing is head-sampled down. Recorded at exactly one site — `makeAcceptSpan` is the single function both the synchronous (`onForceAccept`) and asynchronous (`onAccept`) accept paths call once per round — so it can neither double-count nor be skipped, and it adds no per-peer, per-proposal or per-transaction work. **Explicit buckets** are registered for it in `MetricsRegistry::initExporterAndProvider` (`addRoundDurationHistogramView`, boundaries 500 ms → 120 s): the SDK default tops out at 10,000 ms, which would collapse every slow round into one saturated bucket and read every quantile as 10 s, and the consensus parameters themselves allow a round up to `ledgerAbandonConsensus` = 120 s. Needs **no collector change** — a native metric rides the existing OTLP → Prometheus path. | +| `consensus.validation.accept` (`validation_status`, `accept_gated`, `ledger_hash`, `ledger_seq`, `full_validation`) | span + span attr | `RCLValidations.cpp` — `handleNewValidation` | Trusted Validation Accept Rate by Status | One span per **trusted** validation as it reaches the ledger-acceptance gate, so its rate is bounded by the UNL size per ledger close (untrusted validations cannot move acceptance and get no span). Its trace id is derived from the **validated ledger's** hash, so it joins that ledger's trace rather than the round trace — see the per-ledger trace join below. `validation_status` is one of the six `ValStatus` values and only `current` continues to the gate, which is the difference between a node whose arriving validations are counting and one whose validations are all rejected; from the outside both look like a node that receives validations and never validates. `accept_gated` is true when another thread was already accepting the same ledger, which is why a trace can show a validation with no `ledger.validate` after it. Both are spanmetrics dimensions in **both** collector configs (6 and 2 values, bounded); `ledger_hash` / `ledger_seq` stay span-only and Tempo-indexed, since a per-ledger metric dimension would mint one series per ledger. | +| Per-ledger trace join (`ledger_hash` as trace-id seed) | trace scheme | `LedgerMaster.h/.cpp` — `LedgerMaster::makeLedgerTraceSpan` | n/a — read in Tempo, `{span.ledger_hash="LEDGER_HASH"}` | Makes one slow ledger readable as **one connected trace** instead of a set of orphan spans on different threads. `ledger.validate` (`LedgerMaster::checkAccept`), `ledger.store` (`LedgerMaster::storeLedger`) and `consensus.validation.accept` (`handleNewValidation`) each derive their trace id from the **same 32-byte ledger hash** via `SpanGuard::hashSpan`, which seeds the trace id from `hash[0:16]`. Nothing is propagated between the threads: every one of those sites already holds the ledger hash, which is the whole reason the key was chosen — the same pattern the apply pipeline uses to join `tx.preflight` / `tx.preclaim` / `tx.transactor` on the transaction id (`libxrpl/tx/applySteps.cpp`). Each span is a **true root** (deterministic trace id, empty parent), so the ledger's spans are siblings in one trace rather than a parent/child chain, which is the honest shape: none causes another directly and their order varies with the sync path (`checkAccept` is entered from a peer thread via `handleNewValidation`, from the acquire-completion job, and from the consensus thread via `switchLCL`). The full hash is also recorded as the `ledger_hash` attribute — it is what an operator searches by, and it is how a reader confirms two spans are genuinely the same ledger rather than a trace-id coincidence, since the trace id is only the leading 16 bytes. Asserted end-to-end by the `trace_join_groups` block in `expected_spans.json` (`assert_trace_join_groups` in `validate_telemetry.py`), which fails CI if the members stop sharing a trace. | +| `ledger.acquire` span (`outcome` = `complete` \| `failed` \| `abandoned`; `acquire_reason`, `timeouts`, `peer_count`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::init` / `InboundLedger::finalizeAcquireSpan` | Ledger Acquire Phase Duration (p95 by phase) (its three phase children); Ledger Acquire Duration (Inbound Fetch) and Ledger Acquire Rate by Outcome, both on the **node-health** board | Parent of the three phase spans: one whole fetch of one missing ledger, from the first request to the terminal state. Pre-existing since Phase 6, extended here with `ledger_hash` (set at `init()`, so a fetch that never finishes is still findable in a trace search, and it is the trace-id seed that joins this span to the `ledger.validate`, `ledger.store` and `consensus.validation.accept` spans for the same ledger) and with the fourth `outcome` value `abandoned`, recorded when the acquire is destroyed by a sweep or shutdown before reaching a result. Without `abandoned` a stuck-then-swept fetch left the span with no `outcome` at all, so it vanished from every outcome rate — the exact failure a stalled fresh sync produces. `ledger_seq` is re-stamped at the end because a by-hash acquire starts with `seq_ == 0` and learns the sequence only when the header arrives. | +| `ledger.acquire.header` span (`outcome`, `timed_out`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::syncPhaseSpans` / `endPhaseSpan` | Ledger Acquire Phase Duration (p95 by phase); Ledger Acquire Phase Outcomes (by phase & timeout) | Child of `ledger.acquire` covering the wait for the ledger header, which gates both tree phases — until it arrives the account-state and transaction root hashes are unknown, so nothing else can even be requested. The parent span is flat and its duration is dominated by the state tree, so a node stuck waiting to be TOLD what to fetch was indistinguishable from one stuck fetching it. No `missing_nodes`: a header is a single object, not a tree. Opened and closed by one idempotent state sync over the `have*_` flags rather than by open/close calls scattered through the fetch code, so the span boundary cannot drift out of step with the real phase boundary. | +| `ledger.acquire.astree` span (`outcome`, `timed_out`, `missing_nodes`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::syncPhaseSpans` / `endPhaseSpan` | Ledger Acquire Phase Duration (p95 by phase); Ledger Acquire Phase Outcomes (by phase & timeout) | Child of `ledger.acquire` covering the account-state SHAMap fetch — **nearly all of the work in a real fresh sync**, and the reason the phase split exists: the flat parent span could not separate it from the small transaction tree. `missing_nodes` is read from the count `getMissingNodes()` already produced during its sweep, never recomputed, so no second tree walk is added. `outcome=timeout` together with a non-zero `missing_nodes` is the "peers are not serving this tree" signature; `timed_out` is a separate dimension from `outcome` because a phase can time out and still be retried by its parent acquire. | +| `ledger.acquire.txtree` span (`outcome`, `timed_out`, `missing_nodes`, `ledger_hash`, `ledger_seq`) | span | `InboundLedger.cpp` — `InboundLedger::syncPhaseSpans` / `endPhaseSpan` | Ledger Acquire Phase Duration (p95 by phase); Ledger Acquire Phase Outcomes (by phase & timeout) | Child of `ledger.acquire` covering the transaction SHAMap fetch. Usually completes long before the account-state phase, and that asymmetry is the point of separating them: the parent span's duration is the state tree's, not this one's, so a transaction tree that is genuinely slow is invisible inside it. Closed the moment its own tree completes (from `receiveNode`, `trigger` or `takeHeader`), so its duration is the real fetch time rather than stretching to the next trigger. | +| `txset.acquire` span (`outcome`, `txset_hash`, `duration_ms`, `timeouts`, `peer_count`) | span | `TransactionAcquire.cpp` — `TransactionAcquire::finalizeAcquireSpan` | Tx-Set Acquire Outcomes; Tx-Set Acquire Duration (p95) | One attempt to fetch the transaction set a consensus proposal referenced but this node did not hold. `TransactionAcquire` had **zero** telemetry of any kind before this, so a consensus round stalled waiting on a set was indistinguishable from an idle one. The sibling of `ledger.acquire`: same `TimeoutCounter` base, same trigger/onTimer/takeNodes shape, and the same `trace_ledger` flag so the two halves of a stuck sync cannot be enabled apart. `outcome` is `complete` \| `failed` \| `timeout` \| `abandoned`, stamped on both exits (`done()`, and the destructor when the round sweep in `InboundTransactions::newRound` drops a set that never arrived) by one idempotent finalizer. `timeout` is distinct from `failed` because the exhausted-budget path sets the terminal `failed_` flag too — that flag is how the timer loop stops — so the outcome rule checks the timeout first or every timeout would read as a data fault. `txset_hash` identifies WHICH set stalled and stays span-only: one metric series per consensus round would be unbounded. | +| `ledger.serve` span (`object_type`, `outcome`, `served_nodes`, `peer_id`, `ledger_seq`) | span | `PeerImp.cpp` — `PeerImp::processLedgerRequest` (the `JtLedgerReq` worker) | Ledger Serve Rate by Object Type | This node answering a peer's `TMGetLedger` request — the **supply side** of the sync exchange, and the trace-level companion to the existing `serve_refused_total` counter. The whole serve path had no span, so how long this node takes to answer, and whether it answered at all, was unobservable. A fresh trace root, because the request arrives from the wire on a shared worker whose ambient span is unrelated. `object_type` (`header` \| `tx` \| `as` \| `txset`) and `outcome` (`complete` \| `partial` \| `refused`) are both derived by shared rules in `LedgerSpanNames.h` rather than named per branch, which is what stops the eight exits of `processLedgerRequest` disagreeing about one request. `outcome` is derived from the reply itself — `served_nodes` is the reply's own node count and is 0 on all seven refusal paths — so nothing is accumulated and no work is added to the per-node assembly loop. `partial` means the reply hit `Tuning::kSoftMaxReplyNodes`, so the peer must make another round trip. | +| `peer.dial` span (`outcome`, `remote_endpoint`, `duration_ms`) | span | `ConnectAttempt.cpp` — `ConnectAttempt::reportOutcome` | Outbound Dial Outcomes (span-derived, per attempt) | One outbound connect attempt, as a per-attempt timeline rather than a rate. The trace-level companion to `overlay_connect_total` / `overlay_dial_latency_ms`: it carries the same six `outcome` values, set from the same `reportOutcome` funnel, so span and counter cannot disagree, and the funnel's existing first-call-wins guard makes the span exactly-once for free. What it adds is `remote_endpoint` — WHICH peer — which the counter deliberately cannot carry, because one series per peer address would be unbounded cardinality; it is a dedicated Tempo span column instead. A fresh trace root: a dial is the first thing a starting node does, so there is nothing to parent it to. An attempt torn down mid-dial by shutdown ends its span in the destructor with no `outcome`, which is the honest record of "never concluded" rather than a dropped span. | ### Why rotation duration is not recorded diff --git a/docker/telemetry/grafana/dashboards/ledger-sync-health.json b/docker/telemetry/grafana/dashboards/ledger-sync-health.json index d7e39cc006..2f31e4f0db 100644 --- a/docker/telemetry/grafana/dashboards/ledger-sync-health.json +++ b/docker/telemetry/grafana/dashboards/ledger-sync-health.json @@ -230,7 +230,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Count of outbound peer connection attempts, split by terminal outcome.*\n\n###### How it's computed:\n*Count over the selected range of finished dials grouped by outcome, per node. Filter the outcome set with the Dial Outcome variable.*\n\n###### Reading it:\n*Connected should dominate. The failure lines name the stage that broke: tcp_fail (no route or refused), tls_fail (TLS handshake or shared-value exchange), duplicate (already connected to that address, not a fault), upgrade_fail (HTTP upgrade or protocol negotiation), timeout (no terminal state in time).*\n\n###### Healthy range:\n*Connected rising to the configured peer count, then flat with failures near zero.*\n\n###### Watch for:\n*All attempts landing on one failure outcome and no connected line — the node has no outbound peers and can never sync.*\n\n###### Keywords:\n- **Outbound dial latency** *(per node)* — an outbound peer connection attempt from TCP connect through TLS to protocol upgrade; each attempt ends in exactly one outcome.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[ConnectAttempt.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/ConnectAttempt.cpp)\n\n###### Function:\n`ConnectAttempt::reportOutcome`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#outbound-dial-latency)", + "description": "###### What this is:\n*Count of outbound peer connection attempts, split by terminal outcome.*\n\n###### How it's computed:\n*Count over the selected range of finished dials grouped by outcome, per node. Filter the outcome set with the Dial Outcome variable.*\n\n###### Reading it:\n*Connected should dominate. The failure lines name the stage that broke: tcp_fail (no route or refused), tls_fail (TLS handshake or shared-value exchange), self_connection (we dialled our own address -- a local misconfiguration), upgrade_fail (HTTP upgrade or protocol negotiation), timeout (no terminal state in time).*\n\n###### Healthy range:\n*Connected rising to the configured peer count, then flat with failures near zero.*\n\n###### Watch for:\n*All attempts landing on one failure outcome and no connected line — the node has no outbound peers and can never sync.*\n\n###### Keywords:\n- **Outbound dial latency** *(per node)* — an outbound peer connection attempt from TCP connect through TLS to protocol upgrade; each attempt ends in exactly one outcome.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Computed in xrpld code (MetricsRegistry, OpenTelemetry SDK) and exported as a metric; the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[ConnectAttempt.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/ConnectAttempt.cpp)\n\n###### Function:\n`ConnectAttempt::reportOutcome`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#outbound-dial-latency)", "fieldConfig": { "defaults": { "color": { @@ -4468,7 +4468,7 @@ "type": "prometheus", "uid": "${DS_PROMETHEUS}" }, - "description": "###### What this is:\n*Rate of outbound peer dials reaching each terminal outcome, derived from the per-attempt peer.dial span.*\n\n###### How it's computed:\n*Rate of span-derived call counts for the peer.dial span, split by outcome. The same five outcome values the overlay_connect_total counter carries, set from the same funnel in the dial state machine so the two cannot disagree.*\n\n###### Reading it:\n*Read alongside Outbound Dial Outcomes (Count By Outcome) in the Bootstrap row, which is the native counter for the same events. This panel exists for what the counter cannot do: each point here is backed by traces, so clicking through gives the individual attempt and the peer address it was dialling, which is never a metric label because one series per peer address would be unbounded.*\n\n###### Healthy range:\n*The connected series non-zero, failure series at or near zero.*\n\n###### Watch for:\n*A failure series dominating while connected stays at zero means the node has no outbound peers and cannot sync at all. Use the trace drill-down to find which endpoint keeps failing — the aggregate rate cannot tell one bad peer from a broken local network.*\n\n###### Keywords:\n- **Outbound dial** *(per node)* — one attempt by this node to open a peer connection, spanning the TCP connect, the TLS handshake and the protocol upgrade.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Derived by the OTel Collector's spanmetrics connector from spans emitted by xrpld code; the collector counts the spans and their durations, and the Grafana query selects and aggregates the resulting series.*\n\n###### Source:\n[ConnectAttempt.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/ConnectAttempt.cpp)\n\n###### Function:\n`ConnectAttempt::reportOutcome`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#outbound-dial-latency)", + "description": "###### What this is:\n*Rate of outbound peer dials reaching each terminal outcome, derived from the per-attempt peer.dial span.*\n\n###### How it's computed:\n*Rate of span-derived call counts for the peer.dial span, split by outcome. The same six outcome values the overlay_connect_total counter carries, set from the same funnel in the dial state machine so the two cannot disagree.*\n\n###### Reading it:\n*Read alongside Outbound Dial Outcomes (Count By Outcome) in the Bootstrap row, which is the native counter for the same events. This panel exists for what the counter cannot do: each point here is backed by traces, so clicking through gives the individual attempt and the peer address it was dialling, which is never a metric label because one series per peer address would be unbounded.*\n\n###### Healthy range:\n*The connected series non-zero, failure series at or near zero.*\n\n###### Watch for:\n*A failure series dominating while connected stays at zero means the node has no outbound peers and cannot sync at all. Use the trace drill-down to find which endpoint keeps failing — the aggregate rate cannot tell one bad peer from a broken local network.*\n\n###### Keywords:\n- **Outbound dial** *(per node)* — one attempt by this node to open a peer connection, spanning the TCP connect, the TLS handshake and the protocol upgrade.\n\n###### Computation boundary:\n*Result: Per node — each series is one server's own value.*\n*Derived by the OTel Collector's spanmetrics connector from spans emitted by xrpld code; the collector counts the spans and their durations, and the Grafana query selects and aggregates the resulting series.*\n\n###### Source:\n[ConnectAttempt.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/ConnectAttempt.cpp)\n\n###### Function:\n`ConnectAttempt::reportOutcome`\n\n###### References:\n[Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#outbound-dial-latency)", "fieldConfig": { "defaults": { "color": { @@ -5158,7 +5158,7 @@ { "name": "dial_outcome", "label": "Dial Outcome", - "description": "Filter outbound dial attempts by terminal outcome [connected / tcp_fail / tls_fail / duplicate / upgrade_fail / timeout]", + "description": "Filter outbound dial attempts by terminal outcome [connected / tcp_fail / tls_fail / self_connection / upgrade_fail / timeout]", "type": "query", "query": "label_values(overlay_connect_total, outcome)", "datasource": { @@ -5518,7 +5518,7 @@ { "name": "span_outcome", "label": "Span Outcome", - "description": "Filter the span-derived sync panels by terminal outcome [complete / failed / timeout / abandoned / partial / refused / connected / tcp_fail / tls_fail / duplicate / upgrade_fail]", + "description": "Filter the span-derived sync panels by terminal outcome [complete / failed / timeout / abandoned / partial / refused / connected / tcp_fail / tls_fail / self_connection / upgrade_fail]", "type": "query", "query": "label_values(span_calls_total, outcome)", "datasource": { diff --git a/docker/telemetry/workload/expected_spans.json b/docker/telemetry/workload/expected_spans.json index 790b7d7968..ec6cf28935 100644 --- a/docker/telemetry/workload/expected_spans.json +++ b/docker/telemetry/workload/expected_spans.json @@ -395,7 +395,7 @@ "parent": null, "required_attributes": ["remote_endpoint", "outcome", "duration_ms"], "config_flag": "trace_peer", - "note": "One outbound connect attempt (ConnectAttempt), a fresh trace root because a dial is the first thing a starting node does and there is nothing to parent it to. Required: run-full-validation.sh lists the other four nodes in each node's [ips], so every node dials and the span always fires. Telemetry is live in time to catch it -- ApplicationImp::setup() calls startTelemetry() before start() calls overlay_->start(). outcome carries the same six values as the overlay_connect_total counter (connected|tcp_fail|tls_fail|duplicate|upgrade_fail|timeout) and is set from the same reportOutcome() funnel, so span and counter cannot disagree. remote_endpoint is the span-only dimension the counter cannot carry, since one series per peer address would be unbounded cardinality." + "note": "One outbound connect attempt (ConnectAttempt), a fresh trace root because a dial is the first thing a starting node does and there is nothing to parent it to. Required: run-full-validation.sh lists the other four nodes in each node's [ips], so every node dials and the span always fires. Telemetry is live in time to catch it -- ApplicationImp::setup() calls startTelemetry() before start() calls overlay_->start(). outcome carries the same six values as the overlay_connect_total counter (connected|tcp_fail|tls_fail|self_connection|upgrade_fail|timeout) and is set from the same reportOutcome() funnel, so span and counter cannot disagree. remote_endpoint is the span-only dimension the counter cannot carry, since one series per peer address would be unbounded cardinality." }, { "name": "peer.proposal.receive", diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index d2878c01a5..d839a88ccb 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -2752,14 +2752,14 @@ outranks every other symptom regardless of what the mode machine says. Peer count flat at zero; _Mode Transitions by Edge_ shows the node never leaving `disconnected`, or churning straight back to it. -| Look at | Healthy | Unhealthy | Conclude | -| ------------------------------------------ | -------------------------------------- | ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| _DNS Resolve Outcome Rate_ | all rate on `outcome=resolved` | any rate on `empty`, or both flat at zero | a name in `[ips]`/`[ips_fixed]` returns no address, or the list is empty — fix the hostname or use an IP | -| _DNS Resolve Latency (p95)_ | milliseconds | seconds-scale | the resolver is timing out and delaying every dial behind it | -| _Outbound Dial Outcome Rate_ | `connected` non-zero | all attempts on one failure outcome | `tcp_fail` = route/firewall/closed port · `tls_fail` = TLS · `duplicate` = already connected, not a fault · `upgrade_fail` = negotiation, go to the next row · `timeout` = never terminal | -| _Outbound Dial Latency (p95)_ | well under the dial timeout | pinned near it | peers accept TCP but never finish the handshake | -| _Handshake Negotiation Failures by Reason_ | flat, or a low background rate | any sustained `reason` | `wrong_network`/`invalid_network_id` is the most common fresh-node fault — the node is on a different network and can never reach quorum; `clock_skew` sends you to branch B | -| _PeerFinder Slot Census_ | `out_active` climbing toward `out_max` | `connecting` non-zero with `out_active` low; or `bootcache` and `livecache` both 0 | dials never complete; or there is nothing to dial at all | +| Look at | Healthy | Unhealthy | Conclude | +| ------------------------------------------ | -------------------------------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| _DNS Resolve Outcome Rate_ | all rate on `outcome=resolved` | any rate on `empty`, or both flat at zero | a name in `[ips]`/`[ips_fixed]` returns no address, or the list is empty — fix the hostname or use an IP | +| _DNS Resolve Latency (p95)_ | milliseconds | seconds-scale | the resolver is timing out and delaying every dial behind it | +| _Outbound Dial Outcome Rate_ | `connected` non-zero | all attempts on one failure outcome | `tcp_fail` = route/firewall/closed port · `tls_fail` = TLS · `self_connection` = we dialled our own address · `upgrade_fail` = negotiation, go to the next row · `timeout` = never terminal | +| _Outbound Dial Latency (p95)_ | well under the dial timeout | pinned near it | peers accept TCP but never finish the handshake | +| _Handshake Negotiation Failures by Reason_ | flat, or a low background rate | any sustained `reason` | `wrong_network`/`invalid_network_id` is the most common fresh-node fault — the node is on a different network and can never reach quorum; `clock_skew` sends you to branch B | +| _PeerFinder Slot Census_ | `out_active` climbing toward `out_max` | `connecting` non-zero with `out_active` low; or `bootcache` and `livecache` both 0 | dials never complete; or there is nothing to dial at all | **Conclusion:** the node has no usable overlay. Nothing downstream can be diagnosed until `connected` on _Outbound Dial Outcome Rate_ is non-zero. Detail: @@ -2943,9 +2943,11 @@ first one that is wrong and fix it before reading further panels. - `connected` — success; this is the line that must be non-zero. - `tcp_fail` — no route, refused, or the peer port is closed or firewalled. - `tls_fail` — the TLS handshake failed. - - `duplicate` — TLS succeeded but PeerFinder already holds a slot for - that address. Ordinary churn on a healthy node, not a failure; it is - reported separately so a rising `tls_fail` cannot be confused with it. + - `self_connection` — TLS succeeded and PeerFinder then recognised the + remote address as one of this node's own, so it had dialled itself. A + local misconfiguration (own address in `[ips_fixed]`, or behind the + advertised endpoint), not an unreachable peer; reported separately so a + rising `tls_fail` is not confused with it. - `upgrade_fail` — TLS succeeded but the HTTP upgrade or protocol negotiation was rejected. This is the outcome that pairs with step 3. - `timeout` — the attempt never reached a terminal state. @@ -3527,7 +3529,7 @@ panel it reads. - **All series flat at zero** — normal. It means this node already held every proposed set locally and never had to fetch one. - **Which peer is the dial failing against?** Panel _Outbound Dial - Outcomes (span-derived, per attempt)_ (`peer.dial`). The same five + Outcomes (span-derived, per attempt)_ (`peer.dial`). The same six outcomes as `overlay_connect_total` in Bootstrap step 2, set from the same code path so the two cannot disagree — read the counter first for the rate, then come here for the **identity**. The span carries diff --git a/src/tests/libxrpl/telemetry/LedgerSpanNames.cpp b/src/tests/libxrpl/telemetry/LedgerSpanNames.cpp index 3329af5073..ce19c7e8f6 100644 --- a/src/tests/libxrpl/telemetry/LedgerSpanNames.cpp +++ b/src/tests/libxrpl/telemetry/LedgerSpanNames.cpp @@ -584,7 +584,7 @@ TEST(LedgerSpanNames, peer_dial_attribute_keys_are_bare_underscore) TEST(LedgerSpanNames, peer_dial_outcome_values_match_the_counter_label_set) { - // These five ARE the values ConnectAttempt::reportOutcome passes to the + // These six ARE the values ConnectAttempt::reportOutcome passes to the // overlay_connect_total counter -- the span and the counter read the same // constants from the same funnel, which is what stops them drifting apart. // Pinned literally because the Bootstrap-row dial panel and the runbook @@ -594,6 +594,10 @@ TEST(LedgerSpanNames, peer_dial_outcome_values_match_the_counter_label_set) EXPECT_EQ(std::string_view(peer_span::val::tlsFail), "tls_fail"); EXPECT_EQ(std::string_view(peer_span::val::upgradeFail), "upgrade_fail"); EXPECT_EQ(std::string_view(peer_span::val::timeout), "timeout"); + + // Reuses the slug handshake_negotiation_fail_total already publishes for the + // same fault, so one misconfiguration reads identically on both signals. + EXPECT_EQ(std::string_view(peer_span::val::selfConnection), "self_connection"); } TEST(LedgerSpanNames, peer_dial_outcome_values_are_mutually_distinct) @@ -601,10 +605,11 @@ TEST(LedgerSpanNames, peer_dial_outcome_values_are_mutually_distinct) // The dial panel splits by this attribute, so two outcomes sharing a value // would merge two different failure stages into one line -- and the stage // is the whole diagnostic content of the dial signal. - std::array const values{ + std::array const values{ peer_span::val::connected, peer_span::val::tcpFail, peer_span::val::tlsFail, + peer_span::val::selfConnection, peer_span::val::upgradeFail, peer_span::val::timeout}; for (std::size_t i = 0; i < values.size(); ++i) diff --git a/src/xrpld/app/misc/detail/ValidatorSite.cpp b/src/xrpld/app/misc/detail/ValidatorSite.cpp index 5af583d1f0..d9411c4709 100644 --- a/src/xrpld/app/misc/detail/ValidatorSite.cpp +++ b/src/xrpld/app/misc/detail/ValidatorSite.cpp @@ -395,16 +395,26 @@ ValidatorSite::reportFetchOutcome( // the time series stable when a site redirects. // // The label is rebuilt from the parsed parts rather than using the raw - // configured URI: [validator_list_sites] accepts credentials in the URI, - // and ParsedUrl keeps them in username/password. Emitting the raw string - // would copy them into a metric label, from which they would reach the - // collector, Prometheus and every dashboard. Scheme, host, port and path - // are all a reader needs to tell one site from another. + // configured URI. [validator_list_sites] accepts userinfo in the URI and + // ParsedUrl retains it in username/password even though the fetch itself + // never sends it, so the label was the one place a configured + // `https://user:pass@host` could surface -- and from there it would reach + // the collector, Prometheus and every dashboard. + // + // Scheme, host and path only, and the path truncated at the first '?' or + // '#'. Two reasons: + // - The port is omitted because the Resource constructor defaults it to + // 443/https and 80/http when the config omits one. Including it would + // rewrite the existing `https://vl.ripple.com` series as + // `https://vl.ripple.com:443/` and break continuity for every deployment + // already scraping this metric. + // - parseUrl's path group is `(/.*)?`, which is greedy to end of string, so + // a query or fragment lands inside `path`. A list URL authenticated by + // `?token=...` would otherwise leak through the label the same way + // userinfo would. auto const& url = sites_[siteIdx].loadedResource->pUrl; std::string siteLabel = url.scheme + "://" + url.domain; - if (url.port) - siteLabel += ":" + std::to_string(*url.port); - siteLabel += url.path; + siteLabel += url.path.substr(0, url.path.find_first_of("?#")); XRPL_METRIC_COUNTER_INC_LABELED( app_, diff --git a/src/xrpld/overlay/detail/ConnectAttempt.cpp b/src/xrpld/overlay/detail/ConnectAttempt.cpp index d8debe4090..a87e5e38f7 100644 --- a/src/xrpld/overlay/detail/ConnectAttempt.cpp +++ b/src/xrpld/overlay/detail/ConnectAttempt.cpp @@ -354,11 +354,13 @@ ConnectAttempt::onHandshake(error_code ec) if (!overlay_.peerFinder().onConnected( slot_, beast::IPAddressConversion::fromAsio(localEndpoint))) { - // Not a TLS failure: the handshake succeeded and PeerFinder simply - // already holds a slot for this address. Reporting it as tls_fail - // conflated ordinary dial churn with peers we cannot speak to. - reportOutcome(telemetry::peer_span::val::duplicate); - fail("Duplicate connection"); + // Not a TLS failure: the handshake succeeded and PeerFinder then + // recognised the remote address as our own. Logic::onConnected has + // exactly one false-returning path and it is the self-connect check + // ("Logic dropping as self connect"), so this branch means we dialled + // ourselves -- a local misconfiguration, not an unreachable peer. + reportOutcome(telemetry::peer_span::val::selfConnection); + fail("Self connection"); return; } diff --git a/src/xrpld/overlay/detail/ConnectAttempt.h b/src/xrpld/overlay/detail/ConnectAttempt.h index 1467ad8b34..3945ac33f2 100644 --- a/src/xrpld/overlay/detail/ConnectAttempt.h +++ b/src/xrpld/overlay/detail/ConnectAttempt.h @@ -158,18 +158,25 @@ private: * | * +-- onTimer ................................. "timeout" * +-- onConnect (connect / local_endpoint) . "tcp_fail" - * +-- onHandshake (TLS / slot / shared value) "tls_fail" + * +-- onHandshake + * | +-- TLS handshake / shared value ....... "tls_fail" + * | +-- PeerFinder rejects our own address . "self_connection" * +-- onWrite / onRead / onShutdown ............. "upgrade_fail" * +-- processResponse * +-- bad status / protocol / activate ... "upgrade_fail" * +-- PeerImp created + addActive ........ "connected" * + * The slot branch is drawn separately from the TLS one because + * `Logic::onConnected` fails for exactly one reason -- the remote address is + * ours -- and that is a local misconfiguration rather than an unreachable + * peer. + * * @param outcome One of the `peer_span::val` dial-outcome constants: - * `connected`, `tcpFail`, `tlsFail`, `upgradeFail`, `timeout`. Taken - * as a string_view over a compile-time constant, so no allocation - * happens on the caller side. The constants are the single source - * for both the counter label and the span attribute, so the two - * cannot drift apart. + * `connected`, `tcpFail`, `tlsFail`, `selfConnection`, `upgradeFail`, + * `timeout`. Taken as a string_view over a compile-time constant, so + * no allocation happens on the caller side. The constants are the + * single source for both the counter label and the span attribute, so + * the two cannot drift apart. * * @note Per-connection path: one dial per outbound peer, so this is not * a hot loop. diff --git a/src/xrpld/overlay/detail/PeerSpanNames.h b/src/xrpld/overlay/detail/PeerSpanNames.h index f9904fdb87..535424ff45 100644 --- a/src/xrpld/overlay/detail/PeerSpanNames.h +++ b/src/xrpld/overlay/detail/PeerSpanNames.h @@ -85,26 +85,29 @@ namespace val { * cannot drift apart: the dial state machine names its outcome once and both * signals receive that same value. * - * - connected: the peer was activated and added to the overlay. - * - tcp_fail: the TCP connect or local-endpoint read failed. - * - tls_fail: the TLS handshake or the shared-value exchange failed. - * - duplicate: TLS succeeded but PeerFinder already holds a slot for this - * address, so the attempt was redundant rather than faulty. - * - upgrade_fail: TLS succeeded but the HTTP upgrade, protocol negotiation - * or activation was rejected. - * - timeout: the attempt never reached any terminal state in time. + * - connected: the peer was activated and added to the overlay. + * - tcp_fail: the TCP connect or local-endpoint read failed. + * - tls_fail: the TLS handshake, or the shared-value read taken before + * the HTTP upgrade, failed. + * - self_connection: TLS succeeded and then PeerFinder recognised the remote + * address as one of our own, so we had dialled ourselves. + * - upgrade_fail: TLS succeeded but the HTTP upgrade, protocol negotiation + * or activation was rejected. + * - timeout: the attempt never reached any terminal state in time. * - * `duplicate` is separate from `tls_fail` on purpose. Dialling an address we - * are already connected to is normal churn on a healthy node, while a TLS - * failure means the peer could not be spoken to at all. Reporting both as - * `tls_fail` made a rising TLS-failure count unreadable: it could equally mean - * broken peers or merely a busy PeerFinder, and the two need opposite - * responses. + * `self_connection` is separate from `tls_fail` because it is a local + * misconfiguration, not an unreachable peer: the node has its own address in + * `[ips_fixed]` or behind its advertised endpoint, and every dial to it is + * wasted. Counting it as a TLS failure made a rising `tls_fail` unreadable -- + * broken peers and a self-dial loop need completely different responses. The + * slug matches `handshake_fail::selfConnection` on + * `handshake_negotiation_fail_total`, so the same fault reads the same way + * whichever signal surfaces it. */ inline constexpr auto connected = makeStr("connected"); inline constexpr auto tcpFail = makeStr("tcp_fail"); inline constexpr auto tlsFail = makeStr("tls_fail"); -inline constexpr auto duplicate = makeStr("duplicate"); +inline constexpr auto selfConnection = makeStr("self_connection"); inline constexpr auto upgradeFail = makeStr("upgrade_fail"); inline constexpr auto timeout = makeStr("timeout"); } // namespace val diff --git a/src/xrpld/telemetry/MetricNames.h b/src/xrpld/telemetry/MetricNames.h index ff04649d39..6eb64bc7a9 100644 --- a/src/xrpld/telemetry/MetricNames.h +++ b/src/xrpld/telemetry/MetricNames.h @@ -88,12 +88,16 @@ * * Example usage -- edge case: a value that must NOT be a constant. The site * URI is runtime data, so only the KEY is named here; declaring the value - * would imply a bounded set that does not exist: + * would imply a bounded set that does not exist. Note the value is built from + * the PARSED url, not the configured string: the config accepts userinfo and a + * query, and either would otherwise ride the label into Prometheus. * @code + * auto const& url = sites_[siteIdx].loadedResource->pUrl; + * std::string site = url.scheme + "://" + url.domain; + * site += url.path.substr(0, url.path.find_first_of("?#")); * XRPL_METRIC_COUNTER_INC_LABELED( * app_, metric::unlFetchTotal, "...", - * {{label::site, std::string(sites_[siteIdx].loadedResource->uri)}, - * {label::outcome, std::string(outcome)}}); + * {{label::site, site}, {label::outcome, std::string(outcome)}}); * @endcode * * @note Header-only and dependency-free: nothing here includes an OTel or an