mirror of
https://github.com/XRPLF/rippled.git
synced 2026-08-21 22:30:57 +00:00
docs(telemetry): correct the ledger.acquire outcome guidance
The runbook told operators to read a missing outcome as "never went to the network", which stopped being true once the abort path started setting it, and the glossary still described outcome as a two-value split. Document all three values, where each is written, and why peer_count is absent on the abort path. Several claims were wrong and are corrected: - Give-up is reached at about 18s, not 21s. There is no setTimer call, so the first timer runs immediately and the old derivation counted a wait that does not happen. - A live aborted rate does not imply stalled acquisitions. A clean shutdown clears every in-flight acquire, the admin fetch_info clear does the same, and a full job lane stops timeouts advancing so give-up cannot fire. The runbook already said the last of these elsewhere. - A missing outcome does not mean exactly one thing. tryDB can set failed_ and return before done() runs, exporting a span with no outcome at all. - failed covers unusable ledger data as well as exhausting the retry limit, so a failed span can carry timeouts=0. - The aborted lower bound of one minute holds only on the sweep path. - The sweep measures time since anything last asked for the ledger, not since data last arrived. Also fixes the mainnet verification command, which still used the old RPC port, and drops an inaccurate claim from the config comment about which ports the workload scripts use.
This commit is contained in:
Pratik Mankawde
@@ -447,8 +447,8 @@ Join a transaction's work to its ledger with `{span.current_ledger_seq=<N>}`.
|
||||
|
||||
> **`consensus.check` carries nine attributes, all set before the early
|
||||
> returns.** `Consensus<Adaptor>::haveConsensus()` sets them at
|
||||
> `include/xrpl/consensus/Consensus.h:1899-1912` and `consensus_result` at
|
||||
> `:1925`, deliberately ahead of the `No` / `Expired` branches, so the span is
|
||||
> `include/xrpl/consensus/Consensus.h:1899-1911` and `consensus_result` at
|
||||
> `:1926`, deliberately ahead of the `No` / `Expired` branches, so the span is
|
||||
> fully populated even on rounds that never reach consensus. In set order:
|
||||
> `agree_count`, `disagree_count`, `converge_percent`,
|
||||
> `have_close_time_consensus`, `threshold_percent`, `proposers_finished`,
|
||||
@@ -477,15 +477,24 @@ Join a transaction's work to its ledger with `{span.current_ledger_seq=<N>}`.
|
||||
| `tx_count` | int64 | `tx.apply` | Transactions applied to the ledger |
|
||||
| `tx_failed` | int64 | `tx.apply` | Failed transactions in the apply set |
|
||||
| `validations` | int64 | `ledger.validate` | Number of validations received for this ledger |
|
||||
| `acquire_reason` | string | `ledger.acquire` | Why the ledger fetch was triggered |
|
||||
| `acquire_reason` | string | `ledger.acquire` | Fetch trigger (`history`/`consensus`/`generic`) |
|
||||
| `timeouts` | int64 | `ledger.acquire` | Number of fetch timeouts |
|
||||
| `peer_count` | int64 | `ledger.acquire` | Peers queried during the fetch |
|
||||
| `outcome` | string | `ledger.acquire` | Fetch outcome |
|
||||
| `outcome` | string | `ledger.acquire` | Fetch outcome (`complete`/`failed`/`aborted`) |
|
||||
|
||||
The apply-step span `tx.apply` (child of `ledger.build`) carries `tx_count`/`tx_failed`;
|
||||
the parent `ledger.build` carries `ledger_seq` and the close-time attributes.
|
||||
`ledger.acquire` (InboundLedger) also sets `ledger_seq`.
|
||||
|
||||
`outcome` takes one of **three** values, not two. `complete` and `failed` are both set in
|
||||
`done()` (`InboundLedger.cpp:530-532`), where `failed` covers both giving up after the
|
||||
retry limit and hitting unusable ledger data, so a `failed` span can carry `timeouts=0`. `aborted` is set in `~InboundLedger()` when the object is destroyed while
|
||||
`!isDone()` (`InboundLedger.cpp:242-246`) — the acquisition was **abandoned** before it
|
||||
finished, rather than having run to its retry limit. The abort path records `timeouts` but
|
||||
deliberately **not** `peer_count`, because reading the peer count goes through `Overlay`, which
|
||||
a destructor must not depend on still existing. A query that only groups by
|
||||
`complete`/`failed` therefore silently loses every abandoned fetch.
|
||||
|
||||
**Tempo query**: `{span.ledger_seq=12345}` to find all spans for a specific ledger.
|
||||
|
||||
#### Peer Attributes
|
||||
@@ -626,7 +635,7 @@ prefix=xrpld
|
||||
> (`src/xrpld/app/misc/NetworkOPs.cpp:4884-4897`). Divide by `1e6` for seconds.
|
||||
> The `node-health` "State Duration Rate (All States)" panel already does
|
||||
> (`/ 1000000` on each `rate(...)`), and
|
||||
> `docker/telemetry/grafana/dashboards/validate_dashboards.py:43` lints the
|
||||
> `docker/telemetry/grafana/dashboards/validate_dashboards.py:44-45` lints the
|
||||
> family as "cumulative µs". Reading the raw value as seconds overstates time
|
||||
> in state by a factor of one million.
|
||||
|
||||
@@ -755,7 +764,7 @@ types where this bites are the ones with a low concurrency limit
|
||||
> **Sampling caveat.** These are sampled, not integrated. The values are read
|
||||
> when the SDK's periodic reader invokes the observable callbacks, which run the
|
||||
> collector hooks; the export interval is 1000 ms
|
||||
> (`export_interval_millis` in `src/libxrpl/telemetry/Telemetry.cpp:441`) and
|
||||
> (`export_interval_millis` in `src/libxrpl/telemetry/Telemetry.cpp:476`) and
|
||||
> hook invocation is debounced to at most once per 500 ms. A spike shorter than
|
||||
> the interval can be missed entirely, so read these as pressure indicators
|
||||
> rather than as exact peak depths.
|
||||
@@ -1030,8 +1039,8 @@ async callbacks for new categories.
|
||||
> **Label values are case-sensitive and three cache values are not lowercase.**
|
||||
> The `metric` label carries the string literal passed to `Observe()`, verbatim:
|
||||
> `SLE_hit_rate`, `AL_hit_rate` and `AL_size` are upper-case
|
||||
> (`src/xrpld/telemetry/MetricsRegistry.cpp:649`, `:665`, `:691`), while
|
||||
> `ledger_hit_rate` genuinely is lowercase (`:658`). A selector written as
|
||||
> (`src/xrpld/telemetry/MetricsRegistry.cpp:666`, `:682`, `:708`), while
|
||||
> `ledger_hit_rate` genuinely is lowercase (`:675`). A selector written as
|
||||
> `cache_metrics{metric="sle_hit_rate"}` matches nothing.
|
||||
|
||||
#### Server Info (via OTel MetricsRegistry)
|
||||
@@ -1131,7 +1140,7 @@ Phase-7 parity set, see
|
||||
| Dashboard | UID | Data Source | Key Panels |
|
||||
| ------------------ | ------------ | ----------- | ----------------------------------------------------------------- |
|
||||
| Fee Market & TxQ | `fee-market` | Prometheus | TxQ depth/capacity, fee levels, load factor breakdown, escalation |
|
||||
| Job Queue Analysis | `job-queue` | Prometheus | Per-job rates, queue wait times, execution times, queue depth |
|
||||
| Job Queue Analysis | `job-queue` | Prometheus | Per-job rates, queue wait times, execution times, overflow rate |
|
||||
|
||||
---
|
||||
|
||||
@@ -1179,7 +1188,7 @@ Phase 10 builds a 5-node validator docker-compose harness with RPC load generato
|
||||
> (`nodestore_state`, `cache_metrics`, …) once.
|
||||
>
|
||||
> Note that `ledgers_closed_total` appears in **both** instrument rows: it is
|
||||
> created as a `MetricsRegistry` member (`MetricsRegistry.cpp:369-370`, whose
|
||||
> created as a `MetricsRegistry` member (`MetricsRegistry.cpp:386-387`, whose
|
||||
> `incrementLedgersClosed()` has no callers) and separately incremented at its
|
||||
> call site via `XRPL_METRIC_COUNTER_INC` (`RCLConsensus.cpp:749`). The distinct
|
||||
> name count across the two rows is therefore 41, not 42.
|
||||
@@ -1294,7 +1303,7 @@ via OTLP/HTTP to the OTel Collector and scraped by Prometheus.
|
||||
> **On NuDB, `write_load` and `nudb_writers_in_flight` are the same number.**
|
||||
> Both read the same atomic. `NuDBBackend::getWriteLoad()` returns
|
||||
> `concurrentWriters.load()`
|
||||
> (`src/libxrpl/nodestore/backend/NuDBFactory.cpp:355-361`), and
|
||||
> (`src/libxrpl/nodestore/backend/NuDBFactory.cpp:375-381`), and
|
||||
> `WriteStats::concurrentWriters` is that same counter. So the two series track
|
||||
> each other exactly, sampled microseconds apart in one callback. Do not read
|
||||
> their agreement as two signals confirming each other — it is one signal twice.
|
||||
@@ -1311,7 +1320,7 @@ Further label values on the same instrument, added to separate the two
|
||||
bottlenecks that both present as the `ledgerData` job lane pinned at its
|
||||
concurrency cap. Observed in `MetricsRegistry::observeNodeStoreTotals()`,
|
||||
`observeWritePathDetail()`, and `observeAcquireStats()`
|
||||
(`src/xrpld/telemetry/MetricsRegistry.cpp:854-925`).
|
||||
(`src/xrpld/telemetry/MetricsRegistry.cpp:871-942`).
|
||||
|
||||
| Prometheus Metric | Type | Labels | Description |
|
||||
| ---------------------------------------------------- | ----- | -------- | ------------------------------------------------------- |
|
||||
@@ -1403,7 +1412,7 @@ data as uninformative unless the build is known to include the fix.
|
||||
#### TxQ Admission and Ledger Mismatch (Synchronous Counters)
|
||||
|
||||
Three monotonic counters created alongside the Phase 7+ parity counters
|
||||
(`src/xrpld/telemetry/MetricsRegistry.cpp:377-382`). The gauges above answer
|
||||
(`src/xrpld/telemetry/MetricsRegistry.cpp:394-399`). The gauges above answer
|
||||
"how deep is the queue"; these answer "what did the queue refuse, and did the
|
||||
ledger we built match the one the network validated".
|
||||
|
||||
@@ -1449,7 +1458,7 @@ Rejections (Dropped)", "Queue Abandonment Rate (Expired)"; _Consensus Health_
|
||||
#### Reduce-Relay Efficiency (Observable Gauge — `reduce_relay_metrics`)
|
||||
|
||||
Transaction reduce-relay effectiveness, read from `Overlay::txMetrics()` each
|
||||
collection cycle (`src/xrpld/telemetry/MetricsRegistry.cpp:1353-1385`). A high
|
||||
collection cycle (`src/xrpld/telemetry/MetricsRegistry.cpp:1370-1402`). A high
|
||||
`suppressed_peers` : `selected_peers` ratio proves the feature is saving
|
||||
bandwidth; a high `not_enabled_peers` means stale peers are forcing full relay.
|
||||
|
||||
@@ -1610,7 +1619,7 @@ not a lowercase word and not a friendly alias. The value is
|
||||
`beast::typeName<Object>()` (`include/xrpl/basics/CountedObject.h:115`), which
|
||||
demangles `typeid(T).name()` with `abi::__cxa_demangle`
|
||||
(`include/xrpl/beast/type_name.h:16-45`) and applies no stripping; the observer
|
||||
copies it through verbatim (`src/xrpld/telemetry/MetricsRegistry.cpp:764-770`).
|
||||
copies it through verbatim (`src/xrpld/telemetry/MetricsRegistry.cpp:781-787`).
|
||||
Values therefore keep their `xrpl::` namespace, nested `::`, and template
|
||||
arguments.
|
||||
|
||||
@@ -1745,7 +1754,7 @@ These metrics fill gaps identified by comparing xrpld's internal observability w
|
||||
Data source: `ValidationTracker` class with 8s grace period and 5m late repair window.
|
||||
|
||||
> **Every value on this instrument is a double.** The family is one
|
||||
> `CreateDoubleObservableGauge` (`src/xrpld/telemetry/MetricsRegistry.cpp:1576`),
|
||||
> `CreateDoubleObservableGauge` (`src/xrpld/telemetry/MetricsRegistry.cpp:1593`),
|
||||
> so the integral counts are cast to `double` before `Observe()` — there is no
|
||||
> Int64 sub-series to filter on. The same holds for `validator_health`,
|
||||
> `peer_quality` and `state_tracking` below; an earlier revision of these four
|
||||
@@ -1754,7 +1763,7 @@ Data source: `ValidationTracker` class with 8s grace period and 5m late repair w
|
||||
>
|
||||
> The 7-day window is `ValidationTracker::kWindow7d` = 168 hours
|
||||
> (`src/xrpld/telemetry/ValidationTracker.h:311`) and is observed alongside the 1h
|
||||
> and 24h windows at `MetricsRegistry.cpp:1606-1609`. Panels exist on _Validator
|
||||
> and 24h windows at `MetricsRegistry.cpp:1623-1626`. Panels exist on _Validator
|
||||
> Health_ (`validator-health`): "Agreement % (7d)" and "Agreements vs Missed
|
||||
> (7d)".
|
||||
|
||||
@@ -1767,7 +1776,7 @@ Data source: `ValidationTracker` class with 8s grace period and 5m late repair w
|
||||
| `validator_health{metric="unl_expiry_days"}` | Double | `metric` | Days until UNL list expires |
|
||||
| `validator_health{metric="validation_quorum"}` | Double | `metric` | Validation quorum threshold |
|
||||
|
||||
Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1200`.
|
||||
Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1217`.
|
||||
|
||||
#### Peer Quality (Observable Gauge — `peer_quality`)
|
||||
|
||||
@@ -1778,7 +1787,7 @@ Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1200`.
|
||||
| `peer_quality{metric="peers_higher_version_pct"}` | Double | `metric` | % of peers on newer xrpld version |
|
||||
| `peer_quality{metric="upgrade_recommended"}` | Double | `metric` | 1 if >60% of peers are newer version |
|
||||
|
||||
Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1249`.
|
||||
Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1266`.
|
||||
|
||||
#### Ledger Economy (Observable Gauge — `ledger_economy`)
|
||||
|
||||
@@ -1797,9 +1806,9 @@ Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1249`.
|
||||
| `state_tracking{metric="state_value"}` | Double | `metric` | Numeric state 0-6 (see encoding below) |
|
||||
| `state_tracking{metric="time_in_current_state_seconds"}` | Double | `metric` | Duration in current state |
|
||||
|
||||
Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1466`.
|
||||
Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1483`.
|
||||
|
||||
State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full, 5=validating (FULL + validating), 6=proposing (FULL + proposing). Values 0-4 are `OperatingMode` cast to double (`include/xrpl/server/NetworkOPs.h:60-66`); 5 and 6 are the FULL-only refinements at `MetricsRegistry.cpp:1483-1498`. **The range is 0-6, not 0-7** — there is no seventh state.
|
||||
State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full, 5=validating (FULL + validating), 6=proposing (FULL + proposing). Values 0-4 are `OperatingMode` cast to double (`include/xrpl/server/NetworkOPs.h:60-66`); 5 and 6 are the FULL-only refinements at `MetricsRegistry.cpp:1500-1515`. **The range is 0-6, not 0-7** — there is no seventh state.
|
||||
|
||||
#### Storage Detail (Observable Gauge — `storage_detail`)
|
||||
|
||||
@@ -1808,11 +1817,11 @@ State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full
|
||||
| `storage_detail{metric="stored_object_bytes"}` | Int64 | `metric` | Cumulative object-payload bytes written (not on-disk size) |
|
||||
|
||||
> **`stored_object_bytes` is not a file size.** It observes `getStoreSize()`
|
||||
> (`src/xrpld/telemetry/MetricsRegistry.cpp:1557`), which sums the object payloads
|
||||
> (`src/xrpld/telemetry/MetricsRegistry.cpp:1574`), which sums the object payloads
|
||||
> this process has written. It therefore excludes NuDB's keys, bucket padding and
|
||||
> log, and it resets when the process restarts while the files on disk do not.
|
||||
> `node_written_bytes` on the `nodestore_state` gauge calls the same accessor
|
||||
> (`MetricsRegistry.cpp:860`), so the two series are equal by construction and any
|
||||
> (`MetricsRegistry.cpp:877`), so the two series are equal by construction and any
|
||||
> write-amplification ratio built from the pair is a constant 1.0. To size the store
|
||||
> on disk, stat the backend's files; no metric reports it today.
|
||||
>
|
||||
@@ -1833,9 +1842,9 @@ State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full
|
||||
> **Known issue — `ledgers_closed_total` has a dead second producer.** The
|
||||
> instrument is created twice. `MetricsRegistry::registerCounters()` eagerly
|
||||
> creates it as the member `ledgersClosedCounter_`
|
||||
> (`src/xrpld/telemetry/MetricsRegistry.cpp:369-370`), and its only mutator,
|
||||
> (`src/xrpld/telemetry/MetricsRegistry.cpp:386-387`), and its only mutator,
|
||||
> `MetricsRegistry::incrementLedgersClosed()`
|
||||
> (declared `MetricsRegistry.h:591`, defined `MetricsRegistry.cpp:1686`), has
|
||||
> (declared `MetricsRegistry.h:591`, defined `MetricsRegistry.cpp:1703`), has
|
||||
> **zero callers** — the header says so itself at `MetricsRegistry.h:584-588`.
|
||||
> The value operators actually see comes from the single live increment,
|
||||
> the `XRPL_METRIC_COUNTER_INC` call site in
|
||||
@@ -1889,8 +1898,8 @@ What the code emits today, and where it is documented:
|
||||
| `xrpl.validation.ledger_hash`, `xrpl.peer.validation.ledger_hash` | one bare `ledger_hash` on both `consensus.validation.send` and `peer.validation.receive` <!-- otel-naming:allow-dotted: xrpl.validation.ledger_hash, xrpl.peer.validation.ledger_hash --> |
|
||||
| `xrpl.validation.full`, `xrpl.peer.validation.full` | one bare `full_validation` on both of those spans <!-- otel-naming:allow-dotted: xrpl.validation.full, xrpl.peer.validation.full --> |
|
||||
| `xrpl.consensus.validation_quorum` | `quorum`, on `consensus.accept` only <!-- otel-naming:allow-dotted: xrpl.consensus.validation_quorum --> |
|
||||
| `xrpl.node.amendment_blocked` | **not a span attribute at all** — only the metric `validator_health{metric="amendment_blocked"}` (`MetricsRegistry.cpp:1216`) <!-- otel-naming:allow-dotted: xrpl.node.amendment_blocked --> |
|
||||
| `xrpl.node.server_state` | **not a span attribute at all** — only the metric `server_info{metric="server_state"}` (`MetricsRegistry.cpp:1014`) <!-- otel-naming:allow-dotted: xrpl.node.server_state --> |
|
||||
| `xrpl.node.amendment_blocked` | **not a span attribute at all** — only the metric `validator_health{metric="amendment_blocked"}` (`MetricsRegistry.cpp:1233`) <!-- otel-naming:allow-dotted: xrpl.node.amendment_blocked --> |
|
||||
| `xrpl.node.server_state` | **not a span attribute at all** — only the metric `server_info{metric="server_state"}` (`MetricsRegistry.cpp:1031`) <!-- otel-naming:allow-dotted: xrpl.node.server_state --> |
|
||||
| `xrpl.consensus.proposers_validated` | **never implemented** in any form <!-- otel-naming:allow-dotted: xrpl.consensus.proposers_validated --> |
|
||||
|
||||
The identical nine-row list was deleted from
|
||||
@@ -1910,10 +1919,9 @@ query, an alert — matches nothing and should be pointed at the live keys above
|
||||
|
||||
### Updated Grafana Dashboards (Phase 9)
|
||||
|
||||
| Dashboard | UID | New Panels Added |
|
||||
| -------------------- | -------------------------- | -------------------------------------------------------------------- |
|
||||
| Node Health (StatsD) | `xrpld-statsd-node-health` | NodeStore I/O, cache hit rates, object instance counts |
|
||||
| System Node Health | `node-health` | Ledger economy row: base fee, reserves, ledger age, transaction rate |
|
||||
| Dashboard | UID | New Panels Added |
|
||||
| ----------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Node Health | `node-health` | NodeStore I/O row, cache hit rates, object instance counts; Ledger Economy row: base fee, reserves, ledger age, transaction rate |
|
||||
|
||||
### New Grafana Dashboards (Phase 11)
|
||||
|
||||
@@ -2079,7 +2087,7 @@ discovery cache has any instrument at all.
|
||||
|
||||
| Reading | Source | Why it matters |
|
||||
| --------------------------------------------------- | ---------------------- | ---------------------------------------------------------------------------------------------------------------------- |
|
||||
| `attempts()`, `attemptsNeeded()` | `Counts.h:68,79` | Outbound connection churn; distinguishes "not trying" from "trying and failing" |
|
||||
| `attemptsNeeded()`, `attempts()` | `Counts.h:68,79` | Outbound connection churn; distinguishes "not trying" from "trying and failing" |
|
||||
| `outMax()`, `outActive()`, `outboundSlotsFree()` | `Counts.h:88,98,205` | Outbound slot saturation |
|
||||
| `inMax()`, `inboundActive()`, `inboundSlotsFree()` | `Counts.h:165,174,193` | Inbound slot saturation — the two exported gauges give the actives but not the caps, so utilization cannot be computed |
|
||||
| `acceptCount()`, `connectCount()`, `closingCount()` | `Counts.h:138,147,156` | Handshake pipeline depth; `closingCount()` rising is teardown backpressure |
|
||||
|
||||
@@ -553,7 +553,7 @@ The rate at which the node fetches older ledgers to extend or repair its stored
|
||||
|
||||
### Ledger acquire (inbound fetch)
|
||||
|
||||
Acquiring a ledger means requesting it and its contents from peers when the node lacks it. Acquire outcomes split into complete and failed; a rising failed rate means the node cannot fetch needed ledgers from its peers.
|
||||
Acquiring a ledger means requesting it and its contents from peers when the node lacks it. Acquire outcomes split three ways: complete, failed (the acquisition ended on its own without the ledger, having run out of retries or hit unusable data), and aborted (it was abandoned before finishing, either swept away as stale or discarded wholesale at shutdown). A rising failed rate means the node cannot fetch needed ledgers from its peers.
|
||||
|
||||
**Scope:** per node — measured on and specific to this individual server.
|
||||
|
||||
|
||||
@@ -114,7 +114,7 @@ Metrics begin flowing as soon as the node connects to peers (`server_state`
|
||||
(`server_state` = `full`). Check progress with:
|
||||
|
||||
```bash
|
||||
curl -s http://localhost:5005 -d '{"method":"server_info"}' |
|
||||
curl -s http://localhost:5015 -d '{"method":"server_info"}' |
|
||||
jq '.result.info | {server_state, peers, complete_ledgers}'
|
||||
```
|
||||
|
||||
@@ -393,10 +393,35 @@ Span attributes are filtered with `span.<attr>` inside `{}`. Combine conditions
|
||||
| `ledger.acquire` | InboundLedger.cpp | `ledger_seq`, `acquire_reason`, `timeouts`, `peer_count`, `outcome` | Fetch a missing ledger from peers (parent varies — see [known issues](#where-telemetry-parenting-differs-from-protocol-flow)) |
|
||||
|
||||
`ledger.acquire` sets only `ledger_seq` and `acquire_reason` when the span opens
|
||||
in `init()`; `outcome`, `timeouts` and `peer_count` are written on the `done()`
|
||||
path. All three are therefore **absent** when `init()` satisfies the ledger
|
||||
straight from the local store and the acquire never runs — treat a missing
|
||||
`outcome` as "never went to the network", not as a lost span.
|
||||
in `init()`. `outcome` has three values, written on two different paths:
|
||||
|
||||
| `outcome` | Written where | Meaning |
|
||||
| ---------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `complete` | `done()` | The ledger was fetched. |
|
||||
| `failed` | `done()` | The acquisition ended on its own without the ledger. Usually it gave up after `timeouts_ > kLedgerTimeoutRetriesMax` (= 6), but `trigger()` also fails immediately on an unusable state or transaction map, so a `failed` span can carry `timeouts=0`. Carries span status `Error`. |
|
||||
| `aborted` | destructor | The acquisition was abandoned before finishing — the sweep evicted it a minute after anything last asked for it, or `ledgers_` was cleared wholesale by shutdown or by `clearFailures()`. Status is left `Unset`, because the shutdown case is benign. |
|
||||
|
||||
`peer_count` is written only on the `done()` path, so it is absent on `aborted`
|
||||
spans: reading it would go through `Overlay`, which a destructor running at
|
||||
teardown cannot depend on still existing. `timeouts` is written on both paths.
|
||||
|
||||
A missing `outcome` has two causes, and neither is a lost span. The common one is
|
||||
that `init()` satisfied the ledger straight from the local store, so the acquire
|
||||
never went to the network. The other is a hard failure inside `tryDB()`: a stored
|
||||
header that cannot be this ledger, or a zero account hash, sets `failed_` and
|
||||
`init()` returns without ever calling `done()`, so no outcome is written. The
|
||||
destructor does not fill the gap either — its `if (!isDone())` guard is already
|
||||
false once `failed_` is set, because `isDone()` is `complete_ || failed_`. Such a
|
||||
span carries `ledger_seq` and `acquire_reason` only. Since `aborted` exists, a
|
||||
missing `outcome` is no longer how an abandoned acquisition presents.
|
||||
|
||||
When reading acquire **duration**, exclude or split out `outcome="aborted"`.
|
||||
Those spans stay open from `init()` until the object is destroyed, so they measure
|
||||
how long the acquisition stayed outstanding rather than fetch latency, and will
|
||||
skew a percentile that mixes them with `complete`. Only on the sweep path is that
|
||||
duration bounded below by the one-minute threshold. The shutdown and
|
||||
`clearFailures()` paths abort at whatever age the acquisition happened to have, so
|
||||
an `aborted` span can also be arbitrarily short.
|
||||
|
||||
`ledger.build` does **not** carry `tx_count` / `tx_failed`. Those two live on its
|
||||
child `tx.apply` span, which is where the set is actually applied
|
||||
@@ -1032,6 +1057,34 @@ Side-flow evidence:
|
||||
- **Acquire outcome fork**: `timeouts_ > kLedgerTimeoutRetriesMax` (= 6) sets
|
||||
`failed_` → terminal `logFailure`, no store/checkAccept
|
||||
([InboundLedger.cpp:402](../src/xrpld/app/ledger/detail/InboundLedger.cpp#L402)).
|
||||
A third path never reaches `done()` at all: the destructor marks any acquisition
|
||||
that is still neither `complete_` nor `failed_` as `outcome=aborted`
|
||||
([InboundLedgers.cpp:393](../src/xrpld/app/ledger/detail/InboundLedgers.cpp#L393)
|
||||
sweep eviction; [InboundLedger.cpp:224](../src/xrpld/app/ledger/detail/InboundLedger.cpp#L224)
|
||||
abort branch). Give-up fires at roughly **18s**, not 21s: `init()` enters the
|
||||
retry loop through `queueJob()` with no preceding `setTimer()`, so the first
|
||||
`invokeOnTimer()` runs immediately with `progress_` still `false` and takes
|
||||
`timeouts_` to 1 at t≈0. The test needs `timeouts_ > 6` — the seventh invocation
|
||||
— and only six 3s intervals separate the seventh from the first, so 6 x 3s = 18s.
|
||||
A live `aborted` rate does **not** by itself mean acquisitions are stalling.
|
||||
Three unrelated paths produce it:
|
||||
- **Sweep eviction** — the only cause that implies staleness, and it fires a
|
||||
minute after anything last _asked for_ this ledger, not a minute after the
|
||||
last byte arrived.
|
||||
- **Shutdown** — `InboundLedgers::stop()` clears `ledgers_` wholesale, so every
|
||||
clean stop aborts every acquisition still in flight.
|
||||
- **`clearFailures()`** — also clears `ledgers_`, and is reachable at runtime
|
||||
from the `fetch_info` admin RPC (`clear: true` →
|
||||
`NetworkOPsImp::clearLedgerFetch()`), so an operator can produce aborts on a
|
||||
perfectly healthy node.
|
||||
|
||||
The 18s-vs-60s gap does not settle it either: while the acquisition lane sits at
|
||||
its job limit the timer body never runs, so `timeouts_` cannot advance and the
|
||||
give-up path is disarmed exactly when aborts are likeliest — see
|
||||
[The deferral/timeout pair](#the-deferraltimeout-pair). Rule out shutdown and
|
||||
`clearFailures()` first, then read a sustained `aborted` rate against
|
||||
`acquire_sweep_evictions`.
|
||||
|
||||
- **done() reason branch (store side only)**: `HISTORY` → `onLedgerFetched`, **no**
|
||||
`storeLedger`; else → `storeLedger`. But `checkAccept` + `tryAdvance` run for
|
||||
**any** `complete_ && !failed_` acquire regardless of reason
|
||||
|
||||
Reference in New Issue
Block a user