ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-22 23:00:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(telemetry): align runbook and plan docs with the shipped phase-9/10 code

Browse Source 
The reference docs had drifted from the code in ways that break the reader
rather than merely misinform: PromQL examples that return no data, a rollback
flag that is a no-op, a sampling knob that does not exist, and two span parents
that moved. Code is treated as the truth throughout; where the code is the
defective side, the doc now records it as a known issue instead of describing
the bug as intent.

Renames the docs missed: histogram names gain the exporter's unit suffix
(ios_latency_milliseconds_bucket and four siblings), ledger_history_mismatch
gains _total, the StatsD-era quantile label gives way to le buckets,
rpc.request becomes rpc.http_request, traces_spanmetrics_calls_total becomes
span_calls_total, and the nine dotted xrpl.* span attributes are recorded as
renamed rather than left as live keys.

Re-parenting: consensus.update_positions and consensus.check are children of
consensus.establish, not of consensus.round.

Units and labels: state_accounting_*_duration is microseconds, not seconds;
cache_metrics label values are case-sensitive; object_count carries demangled
C++ type names. Nodestore read and write latency stays microseconds -- the
nanosecond accumulator change did not move the exported unit.

Adds what shipped but was undocumented: the ledger.acquire span, seven
consensus.round events, twelve span attributes, node_writes_duration_us, the
7-day validation-agreement window, the TxQ admission and reduce-relay metric
families, metrics_endpoint, and the phase-10 validation workflow.

Corrects claims that never held: 10% head sampling (it is fixed at 100%),
configurable redaction (it is unconditional), -DXRPL_ENABLE_TELEMETRY=OFF
(the flag is -Dtelemetry=OFF, default ON), FindOpenTelemetry.cmake and the
xrpl_telemetry target (neither exists), Promtail and a StatsD exporter in the
pipeline (neither exists), and Loki stream selection on job= (only
service_name is a stream label).

Phase 9 is marked complete, its provisioned alerting is attributed to the
branch that shipped it, and Phase 11 stays at zero except the one prerequisite
its code closes. Counts are reconciled repo-wide: 41 emitted span families,
15 dashboards on disk with 14 asserted, 13 alert rules in 5 groups.

Hardens the gate that let this drift through: Rule E of the naming check now
covers the reference docs, its allow-dotted marker is key-scoped and warns on
stale or empty use, a missing checked file is reported instead of silently
skipped, the test suite runs in CI, and doc paths trigger the check.

C++ and CMake changes are comment-only: three MetricsRegistry instrument names,
eight OTelCollector claims of a metric-name prefix that formatName never adds,
and the telemetry option's inverted default.
This commit is contained in:
Pratik Mankawde
2026-08-13 16:18:47 +01:00
parent 733af97ce3
commit 3153f3ef56
35 changed files with 4598 additions and 1390 deletions
Download Patch File Download Diff File Expand all files Collapse all files

394
OpenTelemetryPlan/02-design-decisions.md
View File

@@ -125,54 +125,79 @@ path in Phase 1b through Phase 5.
> **Status column.** This catalog is the design inventory; it is not a
> statement of what currently emits. `Live` means the span is present in the
> implemented inventory ([09-data-collection-reference.md §1.1](./09-data-collection-reference.md#11-complete-span-inventory-37-spans)),
> implemented inventory ([09-data-collection-reference.md §1.1](./09-data-collection-reference.md#11-complete-span-inventory-41-spans)),
> which is the authoritative list. `Renamed`/`Split` means the concept shipped
> under a different name than planned here. **Not built** means no span is
> emitted for it today.
>
> **"Not built" is not one thing.** All 14 such entries fall into three cases, and the
> fourth column says which — filing them all as oversights would be wrong:
>
> - **Superseded by metrics or logs (7)** — a deliberate trade-off: the signal is already
> carried by a metric or by a log-derived panel, and a span would add per-event volume
> without adding information. `tx.relay`, `fee.escalate`, `validator.list.fetch`,
> `validator.manifest`, `shamap.sync`, `job.enqueue`, `job.execute`.
> - **Gap (6)** — nothing was decided; they were simply never instrumented. The four
> `peer.*` entries, plus `ledger.replay` and `ledger.delta` — and those last two are the
> sharpest, because they have **no metric substitute at all**.
> - **Deferred (1)** — scheduled work: `amendment.vote` (Phase 11).
>
> The four `peer.*` entries are the peer-span coverage gap: only
> `peer.proposal.receive` and `peer.validation.receive` exist, so protocol
> message send/receive and connection lifecycle are untraced. See
> [09 §6.4](./09-data-collection-reference.md#64-peer-span-coverage-gap-not-implemented).
>
> `tx.validate` did ship, but renamed and split three ways: the apply pipeline
> traces `tx.preflight` (stateless checks), `tx.preclaim` (ledger-state checks)
> and `tx.transactor` (application), each stamped with a `stage` attribute.
> Names come from `TxApplySpanNames.h:90,94,99`. The spans are created in two
> different files, not one: `tx.preflight` and `tx.preclaim` come from
> `applySteps.cpp` (`invokePreflight()` at `:211-212`, `invokePreclaim()` at
> `:258-261`, both via the shared `makeStageSpan()` helper at `:89-126`), while
> `tx.transactor` is created in `Transactor::operator()()`
> (`Transactor.cpp:1601-1605`). Query them with
> `name=~"tx\.(preflight|preclaim|transactor)"` — a **single** backslash; RE2
> reads `\\.` as a literal backslash followed by any character, which matches
> nothing here — never `name="tx.validate"`.
| Span name | Description | Status |
| ------------------------------ | --------------------------------------- | ------------------------------------------------ |
| `tx.receive` | Transaction received from network | Live |
| `tx.validate` | Transaction signature/format validation | **Not built** |
| `tx.process` | Full transaction processing | Live |
| `tx.relay` | Transaction relay to peers | **Not built** |
| `tx.apply` | Apply transaction to ledger | Live |
| `consensus.round` | Complete consensus round | Live |
| `consensus.phase.open` | Open phase - collecting transactions | Live |
| `consensus.phase.establish` | Establish phase - reaching agreement | Renamed `consensus.establish` |
| `consensus.phase.accept` | Accept phase - applying consensus | Renamed `consensus.accept` |
| `consensus.proposal.receive` | Receive peer proposal | Live |
| `consensus.proposal.send` | Send our proposal | Live |
| `consensus.validation.receive` | Receive peer validation | Live |
| `consensus.validation.send` | Send our validation | Live |
| `rpc.request` | HTTP/WebSocket request handling | Split into `rpc.http_request` / `rpc.ws_message` |
| `rpc.command.*` | Specific RPC command (dynamic) | Live |
| `peer.connect` | Peer connection establishment | **Not built** |
| `peer.disconnect` | Peer disconnection | **Not built** |
| `peer.message.send` | Send protocol message | **Not built** |
| `peer.message.receive` | Receive protocol message | **Not built** |
| `ledger.acquire` | Ledger acquisition from network | Live |
| `ledger.build` | Build new ledger | Live |
| `ledger.validate` | Ledger validation | Live |
| `ledger.close` | Close ledger | Renamed `consensus.ledger_close` |
| `ledger.replay` | Ledger replay executed | **Not built** |
| `ledger.delta` | Delta-based ledger acquired | **Not built** |
| `pathfind.request` | Path request initiated | Live |
| `pathfind.compute` | Path computation executed | Live |
| `txq.enqueue` | Transaction queued | Live |
| `txq.apply` | Queued transaction applied | Renamed `txq.apply_direct` / `txq.accept_tx` |
| `fee.escalate` | Fee escalation triggered | **Not built** |
| `validator.list.fetch` | UNL list fetched | **Not built** |
| `validator.manifest` | Manifest update processed | **Not built** |
| `amendment.vote` | Amendment voting executed | **Not built** |
| `shamap.sync` | State tree synchronization | **Not built** |
| `job.enqueue` | Job added to queue | **Not built** |
| `job.execute` | Job execution | **Not built** |
| Span name | Description | Status | Why not built / where the signal lives instead |
| ------------------------------ | --------------------------------------- | ---------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `tx.receive` | Transaction received from network | Live | — |
| `tx.validate` | Transaction signature/format validation | Renamed + split → `tx.preflight`, `tx.preclaim`, `tx.transactor` | — |
| `tx.process` | Full transaction processing | Live | — |
| `tx.relay` | Transaction relay to peers | **Not built** | **Superseded by metrics.** Relay volume is carried by the overlay traffic counters (`total_bytes_in/out`, `total_messages_in/out`, per-`TrafficCount` category). Relay is also per-peer fan-out, so one span per relay multiplies by peer count for data the counters already aggregate. |
| `tx.apply` | Apply transaction to ledger | Live | — |
| `consensus.round` | Complete consensus round | Live | — |
| `consensus.phase.open` | Open phase - collecting transactions | Live | — |
| `consensus.phase.establish` | Establish phase - reaching agreement | Renamed `consensus.establish` | — |
| `consensus.phase.accept` | Accept phase - applying consensus | Renamed `consensus.accept` | — |
| `consensus.proposal.receive` | Receive peer proposal | Live | — |
| `consensus.proposal.send` | Send our proposal | Live | — |
| `consensus.validation.receive` | Receive peer validation | Live | — |
| `consensus.validation.send` | Send our validation | Live | — |
| `rpc.request` | HTTP/WebSocket request handling | Split into `rpc.http_request` / `rpc.ws_message` | — |
| `rpc.command.*` | Specific RPC command (dynamic) | Live | — |
| `peer.connect` | Peer connection establishment | **Not built** | **Gap, scoped as its own change** — see [09 §6.4](./09-data-collection-reference.md#64-peer-span-coverage-gap-not-implemented). Adding these changes the 41-family span count and the 40 catalogued in `expected_spans.json`. |
| `peer.disconnect` | Peer disconnection | **Not built** | **Gap.** Partially observable: the aggregate count via the `Overlay.Peer_Disconnects` insight gauge and resource-charge drops via `server_info{metric="peer_disconnects_resources"}`, but not per-reason. Disconnect reasons are only recoverable from `debug.log` (the `log-derived-insights` dashboard). |
| `peer.message.send` | Send protocol message | **Not built** | **Gap.** Of the 13 protocol message families only `mtGET_OBJECTS` has native instrumentation (`getobject_*`); byte/message volume is aggregated by `TrafficCount` category, not traced per message. |
| `peer.message.receive` | Receive protocol message | **Not built** | **Gap.** Same as `peer.message.send`. |
| `ledger.acquire` | Ledger acquisition from network | Live | — |
| `ledger.build` | Build new ledger | Live | — |
| `ledger.validate` | Ledger validation | Live | — |
| `ledger.close` | Close ledger | Renamed `consensus.ledger_close` | — |
| `ledger.replay` | Ledger replay executed | **Not built** | **Gap, no substitute.** `LedgerReplayer.cpp` and `LedgerReplayTask.cpp` contain zero `SpanGuard` uses and no metric covers the replay path. A real hole, not a trade-off. |
| `ledger.delta` | Delta-based ledger acquired | **Not built** | **Gap, no substitute.** `LedgerDeltaAcquire.cpp` contains zero `SpanGuard` uses. The `acquire_*` stats cover whole-ledger acquisition, not the delta path. |
| `pathfind.request` | Path request initiated | Live | — |
| `pathfind.compute` | Path computation executed | Live | — |
| `txq.enqueue` | Transaction queued | Live | — |
| `txq.apply` | Queued transaction applied | Renamed `txq.apply_direct` / `txq.accept_tx` | — |
| `fee.escalate` | Fee escalation triggered | **Not built** | **Superseded by metrics + existing spans.** Escalation state is `txq_metrics{metric=…}` and `load_factor_metrics{metric=…}`; the queueing path that triggers it is already traced by the six `txq.*` spans. An event span would restate a gauge. |
| `validator.list.fetch` | UNL list fetched | **Not built** | **Superseded by metrics.** `validator_health{metric="unl_expiry_days"}`, `{metric="unl_blocked"}` and `{metric="validation_quorum"}` carry the outcome. A fetch span would fire on a slow timer and tell an operator nothing the gauges do not. |
| `validator.manifest` | Manifest update processed | **Not built** | **Superseded by logs.** Per-master-key manifest dispositions are on the `log-derived-insights` dashboard (`ManifestCache` partition, requires `log_level ManifestCache debug`). |
| `amendment.vote` | Amendment voting executed | **Not built** | **Deferred to Phase 11.** `validator_health{metric="amendment_blocked"}` covers the blocked state in the meantime. |
| `shamap.sync` | State tree synchronization | **Not built** | **Superseded by metrics.** Covered by the nine `acquire_*` stats, `nodestore_state{metric=…}` and the five `getobject_*` families. Per-node-fetch spans would be prohibitive volume. |
| `job.enqueue` | Job added to queue | **Not built** | **Superseded by metrics.** `job_queued_total` and `job_queued_us{job_type}` plus the 105 per-job-type `jobq_*` gauges. A span per enqueue is one span per unit of daemon work, for latency the histogram already records exactly. |
| `job.execute` | Job execution | **Not built** | **Superseded by metrics.** `job_started_total`, `job_finished_total`, `job_running_us{job_type}`. Same volume argument as `job.enqueue`. |
### 2.3.3 Attribute Naming Conventions
@@ -227,24 +252,36 @@ Resource attributes identify the process and are set once at startup. They use
the standard OpenTelemetry semantic conventions plus custom dotted `xrpl.*`
keys (the dotted form is reserved for resource scope per §2.3.3).
| Key | Type / value | Description |
| --------------------- | ------------------------------------------------------- | ------------------------------ |
| `service.name` | `"xrpld"` | Standard `SERVICE_NAME` |
| `service.version` | `build_info::getVersionString()` | Standard `SERVICE_VERSION` |
| `service.instance.id` | node public key (base58) | Standard `SERVICE_INSTANCE_ID` |
| `xrpl.network.id` | network id (e.g. 0 for mainnet) | Network identifier |
| `xrpl.network.type` | `"mainnet"` \| `"testnet"` \| `"devnet"` \| `"unknown"` | Network kind |
| `xrpl.node.type` | `"validator"` \| `"stock"` \| `"reporting"` | Node role |
| `xrpl.node.cluster` | cluster name | Cluster name, if clustered |
Five are set, by `Telemetry.cpp:380-387` (tracer resource) and the matching
block in `initMetrics()` (metrics resource); the custom key constants are
`SpanNames.h:117-118`.
| Key | Type / value | Description | Status |
| --------------------- | -------------------------------------------------------------- | ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `service.name` | `"xrpld"` | Standard `SERVICE_NAME` | Set |
| `service.version` | `build_info::getVersionString()` | Standard `SERVICE_VERSION` | Set |
| `service.instance.id` | node public key (base58), or `[telemetry] service_instance_id` | Standard `SERVICE_INSTANCE_ID` | Set — but the node-key fallback reaches traces only; see [05 §5.1.1](./05-configuration-reference.md) |
| `xrpl.network.id` | network id (e.g. 0 for mainnet) | Network identifier | Set |
| `xrpl.network.type` | `"mainnet"` \| `"testnet"` \| `"devnet"` \| `"unknown"` | Network kind | Set |
| `xrpl.node.type` | `"validator"` \| `"stock"` \| `"reporting"` | Node role | **Not implemented** — no constant, no set-site. Node role is therefore not queryable from a trace. (Dashboards do offer an `$xrpl_node_role` filter, but it matches a Prometheus label stamped by the external perf-iac deployment — `check_otel_naming.py:872` — not by anything in this repo) |
| `xrpl.node.cluster` | cluster name | Cluster name, if clustered | **Not implemented** — no constant, no set-site |
The collector adds two more resource attributes of its own (`deployment.environment`
and, when the node did not stamp it, `xrpl.network.type`) via the
`resource/tier` processor, and deletes the SDK-injected `telemetry.sdk.*` trio
via `resource/stripsdk`. See [05 §5.5.1](./05-configuration-reference.md).
### 2.4.2 Span Attributes by Category
> Span attribute keys use the underscore form from §2.3.3 (shared/qualified
> keys are `<domain>_<field>`; per-span unique keys are bare). The dotted form
> is reserved for the resource attributes in §2.4.1 above. This catalog lists
> the planned attribute set by category; the exact emitted key for each
> implemented span is defined by the `*SpanNames.h` constants, which are the
> single source of truth where the two differ.
> the planned attribute set by category; the exact emitted key **and its type**
> for each implemented span is defined by the `*SpanNames.h` constants and their
> set-sites, which win where the two differ. The types in the tables below are
> the ones originally planned and are **not** all what shipped — `peer_id` is
> the notable case (planned as a base58 string, shipped as an int64). §2.4.3
> is the implemented view.
#### Transaction Attributes
@@ -303,15 +340,15 @@ Establish-phase gap fill and cross-node correlation attributes (Phase 4a):
#### Peer & Message Attributes
| Key | Type | Description |
| -------------------- | ------- | -------------------------- |
| `peer_id` | string | Peer public key (base58) |
| `peer_address` | string | IP:port |
| `peer_latency_ms` | float64 | Measured latency |
| `peer_cluster` | string | Cluster name if clustered |
| `message_type` | string | Protocol message type name |
| `message_size_bytes` | int64 | Message size |
| `message_compressed` | bool | Whether compressed |
| Key | Type | Description |
| -------------------- | ------- | ------------------------------------------------------------------------- |
| `peer_id` | string | Peer public key (base58) — **planned only; shipped as int64, see §2.4.3** |
| `peer_address` | string | IP:port |
| `peer_latency_ms` | float64 | Measured latency |
| `peer_cluster` | string | Cluster name if clustered |
| `message_type` | string | Protocol message type name |
| `message_size_bytes` | int64 | Message size |
| `message_compressed` | bool | Whether compressed |
#### Ledger & Job Attributes
@@ -373,22 +410,72 @@ Establish-phase gap fill and cross-node correlation attributes (Phase 4a):
### 2.4.3 Data Collection Summary
The following table summarizes what data is collected by category:
§2.4.2 above is the _planned_ catalogue; this table is the **implemented** one.
Its left column lists the keys of the `attr` namespaces of the `*SpanNames.h`
headers; every key shown has at least one live `attr::` set-site in
non-test code. The right column lists keys this document once claimed were
collected but which have no constant and no set-site at all.
| Category | Attributes Collected | Purpose |
| --------------- | ---------------------------------------------------------------------------------------------------------------- | ---------------------------- |
| **Transaction** | `tx_hash`, `tx_type`, `tx_result`, `tx_fee`, `ledger_index` | Trace transaction lifecycle |
| **Consensus** | `consensus_round`, `consensus_phase`, `consensus_mode`, `proposers`, `round_time_ms` | Analyze consensus timing |
| **RPC** | `command`, `version`, `rpc_status`, `duration_ms` | Monitor RPC performance |
| **Peer** | `peer_id` (public key), `peer_latency_ms`, `message_type`, `message_size_bytes` | Network topology analysis |
| **Ledger** | `ledger_hash`, `ledger_index`, `close_time`, `ledger_tx_count` | Ledger progression tracking |
| **Job** | `job_type`, `job_queue_ms`, `job_worker` | JobQueue performance |
| **PathFinding** | `pathfind_fast`, `pathfind_search_level`, `pathfind_num_paths`, `pathfind_ledger_index`, `pathfind_num_requests` | Payment path analysis |
| **TxQ** | `txq_queue_depth`, `txq_fee_level`, `txq_eviction_reason` | Queue depth and fee tracking |
| **Fee** | `fee_load_factor`, `fee_escalation_level` | Fee escalation monitoring |
| **Validator** | `validator_list_size`, `validator_list_age_sec` | UNL health monitoring |
| **Amendment** | `amendment_name`, `amendment_status` | Protocol upgrade tracking |
| **SHAMap** | `shamap_type`, `shamap_missing_nodes`, `shamap_duration_ms` | State tree sync performance |
**This table is a category-level roll-up, not the authority.** The
authoritative per-span breakdown — which span carries which attribute — is
[09-data-collection-reference.md §1.2](./09-data-collection-reference.md#12-complete-attribute-inventory-bareunderscore-keys),
and the exact key _spelling_ is owned by the `*SpanNames.h` constants. Where
this table disagrees with either, they win.
> **Known divergence (documented, not resolved here).** 09 §1.2's Consensus
> subsection lists 47 keys; `include/xrpl/consensus/ConsensusSpanNames.h`
> defines 54 in its `attr` namespace (48 own `makeStr` constants plus 6
> `using` re-exports of the shared keys in `SpanNames.h`), all 54 with
> set-sites. Five of the difference — `open_duration_ms`,
> `peer_positions_at_close`, `position_hash_prefix`, `prev_ledger_prefix`,
> `disputes_resolved_count` — are emitted but absent from 09 §1.2's consensus
> table; the other two, `proposal_trusted` and `validation_trusted`, are
> documented in 09 §1.2's Peer subsection instead (they are shared keys set on
> both the `peer.*` and the `consensus.*` receive spans — `PeerImp.cpp:1953`
> and `:2027` for the proposal pair, `:2591` and `:2635` for the validation
> pair). Fixing 09 is tracked separately; the Consensus row below lists all 54.
| Category | Attributes emitted (from `*SpanNames.h`) | Named here but NOT emitted | Purpose |
| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- |
| **Transaction** | `tx_hash`, `tx_type`, `ter_result`, `fee`, `sequence`, `current_ledger_seq`, `current_ledger_hash`, `local`, `path`, `suppressed`, `tx_status`, `peer_version`, `peer_id`, `stage`, `applied` | `tx_result` (renamed → `ter_result`), `tx_fee` (→ `fee`), `ledger_index` (→ `current_ledger_seq`), `relay_count`. **`ledger_seq` is not a `tx.*` key**: no `tx.*` span sets it — the receive and apply-stage spans stamp `current_ledger_seq` (`NetworkOPs.cpp:1422`, `PeerImp.cpp:1337`, `Transactor.cpp:1613`, `applySteps.cpp:115`) and, where a view exists, `current_ledger_hash` (`Transactor.cpp:1615`, `applySteps.cpp:121`) | Trace transaction lifecycle |
| **Consensus** | All 54 keys in `ConsensusSpanNames.h`'s `attr` namespace (48 own constants + 6 `using` re-exports), each with a set-site: `consensus_ledger_id`, `consensus_round`, `consensus_round_id`, `consensus_phase`, `consensus_mode`, `consensus_state`, `consensus_result`, `consensus_stalled`, `proposers`, `proposers_finished`, `previous_proposers`, `previous_ledger_seq`, `previous_round_time_ms`, `round_time_ms`, `open_duration_ms`, `quorum`, `proposing`, `is_bow_out`, `trace_strategy`, `converge_percent`, `establish_count`, `tx_count`, `tx_count_open`, `tx_id`, `disputes_count`, `disputes_resolved_count`, `dispute_our_vote`, `dispute_yays`, `dispute_nays`, `agree_count`, `disagree_count`, `threshold_percent`, `avalanche_threshold`, `close_time_threshold`, `have_close_time_consensus`, `close_time_resolution_ms`, `close_time_self`, `close_time_vote_bins`, `resolution_direction`, `parent_close_time`, `peer_positions_at_close`, `prev_ledger_prefix`, `position_hash_prefix`, `mode_old`, `mode_new`, `validation_sign_time`, `proposal_trusted`, `validation_trusted`; re-exported shared keys `ledger_seq`, `ledger_hash`, `full_validation`, `close_time`, `close_time_correct`, `close_resolution_ms` | — | Analyze consensus timing |
| **RPC** | `command`, `version`, `rpc_role`, `rpc_status`, `request_payload_size`, `is_batch`, `batch_size`, `load_type` | `duration_ms` (span duration is a TraceQL intrinsic — query `duration`), `params` | Monitor RPC performance |
| **Peer** | `peer_id` (**int64**, the process-local `Peer::id_` slot number — not a key of any kind; also set on `tx.receive`), `proposal_trusted`, `validation_trusted`, `ledger_hash`, `full_validation`. (`peer_version` is **not** a peer-span key: the constant lives in `TxSpanNames.h:79` and its only set-site is `PeerImp.cpp:1342` on the `tx.receive` span — see the Transaction row) | `peer_address`, `peer_latency_ms`, `peer_cluster`, `message_type`, `message_size_bytes`, `message_compressed` — the peer-span coverage gap (§2.3.2) | Network topology analysis |
| **Ledger** | `ledger_seq`, `tx_count`, `tx_failed`, `validations`, `acquire_reason`, `timeouts`, `peer_count`, `outcome`, `close_time`, `close_time_correct`, `close_resolution_ms` | `ledger_index` (→ `ledger_seq`), `ledger_tx_count` (→ `tx_count`). `ledger_hash` is a live key, but **no `ledger.*` span sets it** — only `consensus.validation.send` (`RCLConsensus.cpp:977`; that span is the one returned by `createValidationSpan()`, which names `cs::validationSend` at `RCLConsensus.cpp:1365,1373`) and `peer.validation.receive` (`PeerImp.cpp:2573`) do. The `LedgerSpanNames.h:41` `using` alias has zero uses. `consensus.ledger_close` sets **no** hash: its four attributes are `ledger_seq`, `consensus_mode`, `tx_count_open` and `close_time_resolution_ms` (`RCLConsensus.cpp:354-361`) | Ledger progression tracking |
| **gRPC** | `method`, `grpc_role`, `grpc_status` | — | gRPC surface monitoring |
| **Job** | — (no job spans exist) | `job_type`, `job_queue_ms`, `job_worker`. JobQueue is observed via **metrics**, not spans — but by **two disjoint families**, and only one of them has a `job_type` label. See the note below the table | JobQueue performance |
| **PathFinding** | `pathfind_fast`, `pathfind_search_level`, `pathfind_num_paths`, `pathfind_ledger_index`, `pathfind_num_requests`, `pathfind_num_source_assets`, `pathfind_dest_currency`, `pathfind_source_account` (hashed), `pathfind_dest_account` (hashed) | `pathfind_source_currency`, `pathfind_path_count`, `pathfind_cache_hit` | Payment path analysis |
| **TxQ** | `txq_status`, `fee_level_paid`, `required_fee_level`, `queue_size`, `ledger_changed`, `expired_count`, `ter_code`, `retries_remaining`, `num_cleared`, `tx_type`, plus the re-exported shared keys `tx_hash`, `ledger_seq`, `current_ledger_seq`, `current_ledger_hash` | `txq_queue_depth` (→ `queue_size`), `txq_fee_level` (→ `fee_level_paid`), `txq_eviction_reason` | Queue depth and fee tracking |
| **Fee** | — (no `fee.escalate` span, §2.3.2) | `fee_load_factor`, `fee_escalation_level`. Fee escalation is dashboarded from metrics (`fee-market`), not spans | Fee escalation monitoring |
| **Validator** | — (no `validator.*` span, §2.3.2) | `validator_list_size`, `validator_list_age_sec`. UNL health is dashboarded from metrics (`validator-health`) | UNL health monitoring |
| **Amendment** | — (no `amendment.vote` span, §2.3.2) | `amendment_name`, `amendment_status` | Protocol upgrade tracking |
| **SHAMap** | — (no `shamap.sync` span, §2.3.2) | `shamap_type`, `shamap_missing_nodes`, `shamap_duration_ms` | State tree sync performance |
The right-hand column is the honest gap list: every key in it appears in the
§2.4.2 design catalogue but has **zero set-sites** in the code. Where a rename
happened the live name is given in parentheses; where the concept shipped as a
metric rather than a span that is stated. Do not build a dashboard panel, an
alert rule, or a TraceQL query against anything in that column — the query will
return empty, and (per the PromQL/TraceQL asymmetry) a `=~".*"` matcher on an
absent attribute silently blanks a TraceQL panel while quietly passing in
PromQL.
> **JobQueue metrics: two families, one label.** The Job row above has no span
> attributes, and the metrics that replace them do **not** all carry a
> `job_type` label. Getting this wrong produces a panel that renders but is
> wrong, so treat the two families as separate query surfaces:
>
> | Family | Where the job type lives | Source |
> | ------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- |
> | Native `XRPL_METRIC_*`: `job_queued_total`, `job_started_total`, `job_finished_total`, `job_queued_us`, `job_running_us` | In a **`job_type` label** | `MetricsRegistry.cpp:360-362` (counters), `:94-95` (histogram names), `:101` (label key) |
> | `beast::insight` `jobq` group: `jobq_<jobtype>_waiting` / `_running` / `_deferred` / `_q` | In the **metric name itself** — there is **no** `job_type` label at all | `JobTypeData.h:29-32` (naming contract), `:35-38` (suffixes), `Application.cpp:392` (group) |
>
> **The trap:** `sum by (job_type)(jobq_…)` collapses every job type into a
> single series with an empty `job_type`, because an absent PromQL label is
> equivalent to `""` — the query returns a plausible-looking number rather than
> an error. Aggregate the `jobq_*` family with a name matcher
> (`{__name__=~"jobq_.*_waiting"}`) and reserve `by (job_type)` for the
> `job_*_total` / `job_*_us` family.
### 2.4.4 Privacy & Sensitive Data Policy
@@ -400,26 +487,47 @@ OpenTelemetry instrumentation is designed to collect **operational metadata only
The following data is explicitly **excluded** from telemetry collection:
| Excluded Data | Reason |
| ----------------------- | ----------------------------------------- |
| **Private Keys** | Never exposed; not relevant to tracing |
| **Account Balances** | Financial data; privacy sensitive |
| **Transaction Amounts** | Financial data; privacy sensitive |
| **Raw TX Payloads** | May contain sensitive memo/data fields |
| **Personal Data** | No PII collected |
| **IP Addresses** | Configurable; excluded by default in prod |
| Excluded Data | Reason |
| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Private Keys** | Never exposed; not relevant to tracing |
| **Account Balances** | Financial data; privacy sensitive |
| **Transaction Amounts** | Financial data; privacy sensitive |
| **Raw TX Payloads** | May contain sensitive memo/data fields |
| **Personal Data** | No PII collected |
| **IP Addresses** | **Never in spans** — no span sets an address attribute (`peer_address` has zero set-sites); peer spans identify peers by `peer_id`, an int64 process-local slot number. **But the log pipeline is a different story** — see the note below this table |
> **Peer IPs DO leave the node — via the log pipeline, not via spans.** The
> "IP Addresses" row above is scoped to spans, and only to spans. This same
> document describes a log pipeline (§2.6.5) that carries peer addresses:
>
> 1. `PeerImp`'s constructor logs the peer's `remoteAddress_` — an `IP:port` —
> at `info` severity (`PeerImp.h:837-842`), and other overlay call sites log
> addresses too. These land in the ordinary `debug.log` stream.
> 2. The collector's `filelog` receiver tails exactly that file
> (`otel-collector-config.yaml:38-47`, `include: [/var/log/xrpld/*/debug.log]`)
> and the `logs` pipeline exports it to Loki (`:236-239`).
>
> So a deployment running the shipped stack **does** ship peer IPs off-box, as
> log bodies. There is no attribute to drop and no span-level switch to flip,
> because the IPs are inside free-text log messages rather than in structured
> fields — a `delete` action on an attribute key would not touch them.
>
> **The control points are therefore log-side, not trace-side:** Loki
> retention and access control on the log store; the `filelog` receiver's
> `include` list (dropping it disables log↔trace correlation entirely); or a
> collector-side transform on the log body. Do not describe the telemetry
> pipeline as IP-free without qualifying it to traces.
#### Privacy Protection Mechanisms
| Mechanism | Description |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Account Hashing** | Account addresses are hashed both SDK-side (`pathfind_source_account`, `pathfind_dest_account` — always hashed before emission) and again at the collector level, so raw addresses never reach storage |
| **Configurable Redaction** | Sensitive fields can be excluded via `[telemetry]` config section |
| **Collector Tail Sampling** | xrpld head sampling is fixed at 1.0 (every span emitted); the collector retains ~10% of non-error traces, reducing stored data exposure |
| **Sampling** | Only 10% of traces recorded by default, reducing data exposure |
| **Local Control** | Node operators have full control over what gets exported |
| **No Raw Payloads** | Transaction content is never recorded, only metadata (hash, type, result) |
| **Collector-Level Filtering** | Additional redaction/hashing can be configured at OTel Collector |
| Mechanism | Description |
| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Account Hashing** | Account addresses are hashed both SDK-side (`pathfind_source_account`, `pathfind_dest_account` — always hashed before emission) and again at the collector level, so raw addresses never reach storage |
| **Unconditional Redaction** | Account redaction is **not** configurable and cannot be turned off: `redactAccount()` (`Redaction.cpp:14-29`) hashes every **non-empty** address handed to it, with no flag and no bypass (an empty input returns empty — `Redaction.cpp:18-19` — so there is no raw value to leak either way). That is a stronger guarantee than a config switch: there is no insecure-by-default state to misconfigure |
| **Collector Tail Sampling** | **Optional, and OFF in the base stack.** xrpld head sampling is fixed at 1.0 (`Telemetry.h:234` `static constexpr double samplingRatio = 1.0;`), so 100% of traces leave the node. `docker/telemetry/otel-collector-config.yaml` has **no** `tail_sampling` processor either, so the local stack stores 100%. The only shipped policy is in the Grafana Cloud overlay (`otel-collector-config.grafanacloud.yaml:60-67`, wired at `:261`): one `probabilistic` policy at **0.5%**, on the trace-storage branch only so spanmetrics still see every span. Treat sampling as a cost control you opt into — not as a privacy control |
| **Local Control** | Node operators have full control over what gets exported |
| **No Raw Payloads** | Transaction content is never recorded, only metadata (hash, type, result) |
| **Collector-Level Filtering** | Additional redaction/hashing can be configured at OTel Collector |
#### Account Address Hashing
@@ -429,20 +537,40 @@ failure mode. Protection is applied in two independent layers:
1. **SDK-side** (this node): the path-finding RPC handlers call
`redactAccount()` (`xrpl::telemetry`, `Redaction.h`) before setting the
`pathfind_source_account` / `pathfind_dest_account` span attributes. The
helper emits the first 16 characters of `sha512Half(address)` as
lowercase hex — deterministic (spans for one account still correlate)
but non-reversible.
`pathfind_source_account` / `pathfind_dest_account` span attributes. For a
non-empty address the helper emits the first 16 characters of
`sha512Half(address)` as lowercase hex — deterministic (spans for one
account still correlate) but non-reversible. An empty address returns empty
rather than the hash of the empty string (`Redaction.cpp:18-19`).
2. **Collector-side** (defense-in-depth): an `attributes/hash` processor in
the OpenTelemetry Collector re-hashes those same attributes, so any node
that emitted a raw value is still redacted before storage.
#### Collector-Level Data Protection
The OpenTelemetry Collector can be configured (via an `attributes` processor)
to hash or redact sensitive attributes before export — for example, hashing
`pathfind_source_account` / `pathfind_dest_account`, deleting `peer_address`
to drop IP addresses, and deleting `params` to redact request parameters.
The shipped base config does exactly one thing here, and it is the
defense-in-depth layer described above: an `attributes/hash` processor
(`otel-collector-config.yaml:105-110`) hashing `pathfind_source_account` and
`pathfind_dest_account`.
**No `peer_address` or `params` scrubbing rule is needed on the trace pipeline,
and none is shipped.** Earlier drafts prescribed `delete` actions for both.
Neither attribute is ever emitted: `peer_address` has zero set-sites in the code
(peer spans carry `peer_id`, an int64 process-local slot number — not an IP and
not a key), and no span sets a `params` attribute — RPC spans carry `command`,
`version`, `rpc_role`, `rpc_status`, `request_payload_size`, `is_batch`,
`batch_size` and `load_type`, never the request body. Adding delete rules for
absent keys would be harmless but misleading: it would imply the node emits IPs
and request parameters in spans when it does not.
This says nothing about the **log** pipeline, which is where peer IPs actually
do leave the node (see the note under "Data NOT Collected" above). An
`attributes` processor cannot help there — the addresses are inside free-text
log bodies, not in structured attributes.
If a future span _does_ introduce an IP-bearing or payload-bearing attribute,
the `attributes` processor is the right place to strip it — and the attribute
should be added to the §2.4 catalogue in the same change.
#### Configuration Options for Privacy
@@ -555,8 +683,8 @@ flowchart TB
proto["message TraceContext {<br/> bytes trace_id = 1; // 16 bytes<br/> bytes span_id = 2; // 8 bytes<br/> uint32 trace_flags = 3;<br/> string trace_state = 4;<br/>}"]
end
subgraph jobqueue["JobQueue (Internal Async)"]
job["Context captured at job creation,<br/>restored at execution<br/><br/>class Job {<br/> otel::context::Context<br/> traceContext_;<br/>};"]
subgraph jobqueue["JobQueue / Coroutines (Internal Async)"]
job["CoroAwareContextStorage<br/>(RuntimeContextStorage override)<br/><br/>Per-coroutine context stack,<br/>installed globally at startup.<br/>Job itself carries no context."]
end
style http fill:#0d47a1,stroke:#082f6a,color:#ffffff
@@ -568,7 +696,7 @@ flowchart TB
- **HTTP/WebSocket - RPC (blue)**: For client-facing RPC requests, trace context is propagated using the W3C `traceparent` header. This is the standard approach and works with any OTel-compatible client.
- **Protocol Buffers - P2P (green)**: For peer-to-peer messages between xrpld nodes, trace context is embedded as a protobuf `TraceContext` message carrying trace_id, span_id, flags, and optional trace_state.
- **JobQueue - Internal Async (red)**: For asynchronous work within a single node, the OTel context is captured when a job is created and restored when the job executes on a worker thread. This bridges the async gap so spans remain linked.
- **JobQueue / Coroutines - Internal Async (red)**: For asynchronous work within a single node, the ambient OTel context follows the coroutine rather than being carried on the work item. `include/xrpl/core/Job.h` has **no** telemetry include and no `traceContext_` member — an earlier draft of this diagram showed one, and that was never built. Instead `xrpl::telemetry::CoroAwareContextStorage` (`include/xrpl/telemetry/CoroAwareContextStorage.h:84`) overrides the SDK's `RuntimeContextStorage` with a per-coroutine context stack, and is installed as the global storage in `Telemetry::start()` (`Telemetry.cpp:416-419`) before the tracer provider and before the first span. That fixes the wrong-thread scope pop across coroutine yield/resume and keeps log↔trace correlation intact. The storage is never reset — tearing it down while spans may still exist is undefined behaviour in the SDK — so it lives for the process lifetime.
---
@@ -651,7 +779,10 @@ parent through the active context.
### 2.6.4 Coexistence Strategy
> **Note**: Phase 7 replaces the StatsD bridge with native OTel Metrics SDK export. The diagram below shows the Phase 6 intermediate state. See [Phase7_taskList.md](./Phase7_taskList.md) for the migration design where Beast Insight emits via OTLP instead of StatsD.
> **Note**: Phase 7 **added** a native OTel Metrics export path alongside the
> StatsD bridge; it did not replace it. The diagram below shows the Phase 6
> state, which is still reachable today via `[insight] server=statsd`. See
> [Phase7_taskList.md](./Phase7_taskList.md) for the design.
```mermaid
flowchart TB
@@ -681,17 +812,54 @@ flowchart TB
- **OpenTelemetry to OTLP Collector**: OTel exports spans over OTLP/HTTP to a Collector, which then forwards to a trace backend (Tempo). (OTLP/gRPC is future work — §2.2.2.)
- **Grafana (red, unified UI)**: All three data streams converge in Grafana, enabling operators to correlate logs, metrics, and traces in a single dashboard.
**Phase 7 target state**: Beast Insight routes to `OTelCollector` (new `Collector` implementation) which exports via OTLP/HTTP to the same collector endpoint as traces. StatsD UDP path becomes a deprecated fallback (`[insight] server=statsd`). See [06-implementation-phases.md §6.8](./06-implementation-phases.md) and [Phase7_taskList.md](./Phase7_taskList.md) for details.
**Phase 7 outcome (as shipped)**: Beast Insight gained an `OTelCollector`
`Collector` implementation that rides the global MeterProvider and exports via
OTLP/HTTP to the same collector as traces. It is selected with
`[insight] server=otel`.
### 2.6.5 Correlation with PerfLog
The three back ends are **co-equal branches of one `if/else` chain** in
`makeCollectorManager()` (`CollectorManager.cpp:37-75`), not a migration path:
Trace IDs can be correlated with existing PerfLog entries for comprehensive
debugging. The design is for `RPCHandler.cpp` to start an `rpc.command.<method>`
span alongside the existing PerfLog `rpcStart`/`rpcFinish`/`rpcError` calls,
extract the span's `trace_id` (when valid), and eventually stamp it onto the
PerfLog entry (a planned `setTraceId` hook) so logs and traces share a key. The
span status is set to OK on success or to error (recording the exception) on
failure.
| `[insight] server=` | Collector | Status |
| ---------------------- | ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `otel` | `OTelCollector` | OTLP/HTTP to the OTel Collector — the recommended setting |
| `statsd` | `StatsDCollector` | Unchanged from before Phase 7. **Not deprecated**: no warning is logged, no removal is scheduled, and the code path is not marked legacy |
| absent / anything else | `NullCollector` | **The default.** A node with no `[insight]` section emits no metrics at all |
Two corrections to earlier drafts, both of which matter operationally: StatsD
is not a "deprecated fallback", and `otel` is not the default — you must set it
explicitly. See [06-implementation-phases.md §6.8](./06-implementation-phases.md),
[Phase7_taskList.md](./Phase7_taskList.md), and
[05 §5.8.6](./05-configuration-reference.md) for which `[insight]` keys are live
under `server=otel` (most are inert).
### 2.6.5 Correlation with Logs
**Shipped in Phase 8 — and not the way this section originally planned it.**
The design here was a `setTraceId` hook on PerfLog, fed from the
`rpc.command.<method>` span in `RPCHandler.cpp`. That hook was never built:
`setTraceId` has zero occurrences in **source** — the only hits in the tree are
in these plan documents, describing the design that was dropped — and PerfLog's
JSON output carries no trace ID.
What shipped instead is broader and needs no per-call-site wiring: the **journal
sink** stamps the IDs onto _every_ log line written while a span is active.
`Logs::format()` (`src/libxrpl/basics/Log.cpp:304-338`, inside
`#ifdef XRPL_ENABLE_TELEMETRY`) reads the thread-local OTel context, and when
the active span context is valid it prefixes the message with
`trace_id=<32 hex> span_id=<16 hex>`. It inspects the context value directly
rather than calling `GetSpan()`, so the common no-span path costs no heap
allocation.
Because the IDs land in the ordinary `debug.log` stream, correlation is
end-to-end without touching PerfLog: the collector's `filelog` receiver parses
`trace_id`/`span_id` as optional capture groups and ships the lines to Loki, and
Grafana links both directions (Tempo `tracesToLogs` → Loki, Loki derived fields
→ Tempo). Details in [05 §5.8.5](./05-configuration-reference.md).
RPC spans still exist and still set status (OK on success, error with the
recorded exception on failure) — that part of the original design is intact.
Only the PerfLog-stamping mechanism was replaced.
---

248
OpenTelemetryPlan/03-implementation-strategy.md
View File

@@ -7,26 +7,54 @@
## 3.1 Directory Structure
The telemetry implementation follows xrpld's existing code organization pattern:
The telemetry implementation follows xrpld's existing code organization
pattern. The tree below is the current on-disk contents of the three telemetry
directories, and it has three differences from the original design sketch worth
calling out: `TelemetryConfig.h`, `TraceContext.h`, `SpanAttributes.h` and
`TraceContext.cpp` were never created (config structs live inside
`Telemetry.h`, propagation lives in `TraceContextPropagator.h`, and attribute
constants live in the `*SpanNames.h` headers next to their owning class); the
metrics work of Phase 7/9 added a whole second module under
`src/xrpld/telemetry/`, which the sketch predated.
```
include/xrpl/
├── telemetry/
│ ├── Telemetry.h # Main telemetry interface (global singleton)
│ ├── TelemetryConfig.h # Configuration structures
│ ├── TraceContext.h # Context propagation utilities
│ ├── SpanGuard.h # RAII span management with factory methods + discard()
│ ├── DiscardFlag.h # Thread-local discard flag
│ └── SpanAttributes.h # Attribute helper functions
include/xrpl/telemetry/ # libxrpl layer: tracing SDK wrapper
├── Telemetry.h # Interface + Setup config struct + factories
├── SpanGuard.h # RAII span management, factory methods, discard()
├── SpanNames.h # StaticStr/join() + shared span & attr constants
├── DiscardFlag.h # Thread-local discard flag
├── CoroAwareContextStorage.h # RuntimeContextStorage override for coroutines
├── DeterministicIdGenerator.h # trace_id from txHash / prevLedgerHash
├── TraceContextPropagator.h # protobuf TraceContext inject/extract (P2P)
├── TraceContextValidation.h # Validation of peer-supplied trace context
├── Redaction.h # redactAccount() — unconditional address hashing
└── GetObjectMetricNames.h # getobject_* metric name constants
src/libxrpl/
├── telemetry/
│ ├── Telemetry.cpp # Implementation + FilteringSpanProcessor
│ ├── TelemetryConfig.cpp # Config parsing
│ ├── TraceContext.cpp # Context serialization
│ └── NullTelemetry.cpp # No-op implementation
src/libxrpl/telemetry/
├── Telemetry.cpp # TelemetryImpl + FilteringSpanProcessor + initMetrics()
├── TelemetryConfig.cpp # [telemetry] section parsing (makeTelemetrySetup)
├── SpanGuard.cpp # Span/scope guard implementation
├── CoroAwareContextStorage.cpp
├── DeterministicIdGenerator.cpp
├── Redaction.cpp
└── NullTelemetry.cpp # No-op impl — ALWAYS compiled (in-source #ifdef)
src/xrpld/telemetry/ # xrpld layer: native metrics + tx tracing helpers
├── MetricsRegistry.h / .cpp # Owns the XRPL_METRIC_* instruments + MeterProvider
├── MetricMacros.h # XRPL_METRIC_COUNTER_ADD / _HISTOGRAM_RECORD / ...
├── ValidationTracker.h # Validation-agreement tracking (impl in detail/)
├── detail/ValidationTracker.cpp
├── ConsensusReceiveTracing.h # Peer proposal/validation receive spans
├── PropagationHelpers.h # Context inject/extract call-site helpers
├── TxSpanNames.h # tx.* span + attribute constants
└── TxTracing.h # Transaction span helpers
```
Per-class span-name headers deliberately live next to their owning class rather
than in `telemetry/` — see `ConsensusSpanNames.h`, `TxApplySpanNames.h`,
`LedgerSpanNames.h`, `RpcSpanNames.h`, `PathFindSpanNames.h`,
`PeerSpanNames.h`, `TxQSpanNames.h`, `GrpcSpanNames.h`.
---
## 3.2 Implementation Approach
@@ -100,13 +128,22 @@ flowchart TB
| --------------------- | --------- | ---------------------- | ---------- |
| Span creation | 500-1000 | Every traced operation | Low |
| Span end | 100-200 | Every traced operation | Low |
| SetAttribute (string) | 80-120 | 3-5 per span | Low |
| SetAttribute (int) | 40-60 | 2-3 per span | Negligible |
| SetAttribute (string) | 80-120 | 3-5 per span (typical) | Low |
| SetAttribute (int) | 40-60 | 2-3 per span (typical) | Negligible |
| AddEvent | 100-200 | 0-2 per span | Low |
| Context injection | 150-250 | Per outgoing message | Low |
| Context extraction | 100-180 | Per incoming message | Low |
| GetCurrent context | 10-20 | Thread-local access | Negligible |
> **"3-5 attributes per span" is a typical case, not a bound.** The frequency
> column above describes the median span (`tx.receive`, `rpc.command.*`). A few
> spans are deliberately attribute-rich: `consensus.accept.apply` sets **13**
> attributes (`RCLConsensus.cpp:600-674`), and `consensus.round` /
> `consensus.establish` are of the same order. Use ~15 as the worst case when
> sizing per-span attribute cost and memory; the consensus spans that hit it fire
> once per ~3-second round, so their absolute cost stays in the noise
> (see §3.4.3).
**Source**: Span creation based on OTel C++ SDK `BM_SpanCreation` benchmark (AlwaysOnSampler +
SimpleSpanProcessor + InMemoryExporter), median ~1,000 ns on CI hardware. AddEvent includes
timestamp read + string copy + vector push + mutex acquisition. Context injection/extraction
@@ -120,8 +157,8 @@ confirmed by `BM_SpanCreationWithScope` benchmark delta (~160 ns).
%%{init: {'pie': {'textPosition': 0.75}}}%%
pie showData
"tx.receive (1400ns)" : 1400
"tx.validate (1200ns)" : 1200
"tx.relay (1200ns)" : 1200
"tx.process (1200ns)" : 1200
"tx.apply (1200ns)" : 1200
"Context inject (200ns)" : 200
```
@@ -131,9 +168,17 @@ pie showData
**Overhead percentage**: 4.0 μs / 200 μs (avg tx processing) = **~2.0%**
> **Breakdown**: Each span (tx.receive, tx.validate, tx.relay) costs ~1,000 ns for creation plus
> **Breakdown**: Each span (tx.receive, tx.process, tx.apply) costs ~1,000 ns for creation plus
> ~200-400 ns for 3-5 attribute sets. Context injection is ~200 ns (confirmed by benchmarks).
> On production hardware, expect ~2.6 μs total (~1.3% overhead) due to faster span creation (~500-600 ns).
>
> This three-span model predates the apply-pipeline instrumentation. The shipped
> transaction path also emits `tx.preflight`, `tx.preclaim` and `tx.transactor`
> (the spans planned here as `tx.validate`), and never emits `tx.relay`. Scale
> the estimate by span count for a current figure: ~6 spans ≈ 7-8 μs on CI
> hardware, ~4-5 μs on server hardware. The measured end-to-end cost is in
> §3.5.3 (~3-4% throughput at head sampling 1.0), which supersedes this
> bottom-up estimate.
### 3.4.3 Consensus Round Overhead
@@ -148,18 +193,26 @@ pie showData
> **Why higher**: Each span costs ~1,000 ns creation + ~100-200 ns for 1-2 attributes, totaling ~1,100-1,200 ns.
> Context operations remain ~200 ns (confirmed by benchmarks). On production hardware, expect ~24 μs total.
>
> The "1-2 attributes" figure understates the shipped consensus spans, which are
> the attribute-rich ones: `consensus.accept.apply` alone sets 13
> (`RCLConsensus.cpp:600-674`). Adding ~1 μs per such span still leaves the
> round total under ~40 μs against a ~3 s round, so the conclusion below is
> unaffected. Note also that the `consensus.phase` row covers the shipped names
> `consensus.phase.open`, `consensus.establish` and `consensus.accept` — see
> [02 §2.3.2](./02-design-decisions.md).
**Overhead percentage**: 36 μs / 3s (typical round) = **~0.001%** (negligible)
### 3.4.4 RPC Request Overhead
| Operation | Cost (ns) |
| ---------------- | ------------ |
| rpc.request span | ~1200 |
| rpc.command span | ~1100 |
| Context extract | ~250 |
| Context inject | ~200 |
| **TOTAL** | **~2.75 μs** |
| Operation | Cost (ns) |
| ------------------------------------------ | ------------ |
| `rpc.http_request` / `rpc.ws_message` span | ~1200 |
| `rpc.command.*` span | ~1100 |
| Context extract | ~250 |
| Context inject | ~200 |
| **TOTAL** | **~2.75 μs** |
> **Why higher**: Each span costs ~1,000 ns creation + ~100-200 ns for attributes (command name,
> version, role). Context extract/inject costs are confirmed by OTel C++ benchmarks.
@@ -263,12 +316,20 @@ The overhead estimates in Sections 3.3-3.5 are derived from the following source
> compression (~60-70% of raw) and batching (amortized headers), ~350 bytes/span is more realistic.
> The table uses the conservative estimate for capacity planning.
| Sampling Rate | Spans/sec | Bandwidth | Notes |
| ------------- | --------- | --------- | ---------------- |
| 100% | ~500 | ~250 KB/s | Development only |
| 10% | ~50 | ~25 KB/s | Staging |
| 1% | ~5 | ~2.5 KB/s | Production |
| Error-only | ~1 | ~0.5 KB/s | Minimal overhead |
**Node → collector bandwidth is always the 100% row.** Head sampling is a
`static constexpr` 1.0 (`Telemetry.h:234`) with no config key, so every node
exports every span and the export bandwidth is not tunable from `xrpld.cfg`.
| Sampling Rate | Spans/sec | Bandwidth | Where it applies |
| --------------------- | --------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| 100% | ~500 | ~250 KB/s | **The only reachable node→collector figure.** Plan capacity against this row |
| 0.5% | ~2.5 | ~1.25 KB/s | Collector→backend only, and only with the Grafana Cloud overlay's `tail_sampling` (`otel-collector-config.grafanacloud.yaml:60-67`) |
| 10% / 1% / error-only | — | — | **Not implemented.** No shipped config produces these ratios; treat them as illustrative of what a tail-sampling policy could do |
The rows below 100% therefore reduce _storage_ cost at the backend, never the
node's egress. Note also that the shipped 0.5% policy is applied to the
trace-storage branch only, so the spanmetrics-derived RED metrics still see
100% of spans and stay exact.
### 3.6.2 Trace Context Propagation
@@ -285,7 +346,26 @@ The overhead estimates in Sections 3.3-3.5 are derived from the following source
### 3.7.1 Sampling Strategies
#### Tail Sampling
#### Head Sampling (node) — fixed, not a decision point
There is no sampling decision on the node. `samplingRatio` is a
`static constexpr double = 1.0` (`Telemetry.h:234`) and `TelemetryConfig.cpp:139`
records why nothing is parsed: a per-node ratio would let two nodes make
opposite keep/drop decisions for the same distributed trace, yielding partial
traces. The ratio sampler is wrapped in a `ParentBasedSampler` so a span with a
remote parent honours the upstream flag. The only node-local way to drop a span
is the explicit, per-call-site `SpanGuard::discard()`, enforced downstream by
`FilteringSpanProcessor`.
#### Tail Sampling (collector) — aspirational shape
The flowchart below is a **design sketch of a multi-policy tail sampler. It is
not what ships.** The base collector config has no `tail_sampling` processor at
all; the Grafana Cloud overlay has exactly one `probabilistic` policy at 0.5%
with no error or latency carve-outs. Read it as a template for a policy you
might write, not as a description of this repo — and note that adding
error/latency policies would need `decision_wait` tuning, since a policy can
only see spans that arrived within that window.
```mermaid
flowchart TD
@@ -299,13 +379,18 @@ flowchart TD
consensus -->|No| slow{"Is Slow?"}
slow -->|Yes| sample
slow -->|No| prob{"Random < 10%?"}
slow -->|No| prob{"Probabilistic keep?<br/>(shipped policy: 0.5%)"}
prob -->|Yes| sample
prob -->|No| drop["DROP"]
style sample fill:#4caf50,stroke:#388e3c,color:#fff
style drop fill:#f44336,stroke:#c62828,color:#fff
style sample fill:#1b5e20,stroke:#0d3d14,color:#fff
style drop fill:#b71c1c,stroke:#7f1d1d,color:#fff
style trace fill:#0d47a1,stroke:#082f6a,color:#fff
style errors fill:#334155,stroke:#1e293b,color:#fff
style consensus fill:#334155,stroke:#1e293b,color:#fff
style slow fill:#334155,stroke:#1e293b,color:#fff
style prob fill:#334155,stroke:#1e293b,color:#fff
```
### 3.7.2 Batch Tuning Recommendations
@@ -318,7 +403,17 @@ flowchart TD
### 3.7.3 Conditional Instrumentation
Instrumentation is gated on two levels. A compile-time feature flag (`XRPL_ENABLE_TELEMETRY`) reduces the trace macros to no-ops when telemetry is built out, so disabled builds carry zero cost. At runtime, per-component guards (e.g. `shouldTracePeer()`) skip span creation for components whose tracing is turned off, incurring no overhead beyond a single boolean check.
Instrumentation is gated on two levels. A compile-time feature flag reduces the trace macros to no-ops when telemetry is built out, so disabled builds carry zero cost. At runtime, per-component guards (e.g. `shouldTracePeer()`) skip span creation for components whose tracing is turned off, incurring no overhead beyond a single boolean check.
> The compile-time gate is the macro `XRPL_ENABLE_TELEMETRY`, but that macro is
> **not** the switch you flip. It is a compile definition added by
> `CMakeLists.txt` (`add_compile_definitions(XRPL_ENABLE_TELEMETRY)`) when the CMake option `telemetry` is ON.
> That option is declared ON today (`option(telemetry "Enable OpenTelemetry tracing" ON)`)
> only so that CI compiles the instrumented build while the telemetry branches are
> in review; **OFF is the intended default once merged**, flipped in a separate
> change. Select the value explicitly instead of relying on the default:
> `-Dtelemetry=ON|OFF` (CMake) or `-o telemetry=True|False` (Conan). See
> [05 §5.4.2](./05-configuration-reference.md).
---
@@ -372,39 +467,78 @@ quadrantChart
### 3.9.4 Architectural Impact Assessment
| Aspect | Impact | Justification |
| -------------------- | ------- | -------------------------------------------------------------------------------- |
| **Data Flow** | Minimal | Read-only instrumentation; no modification to consensus or transaction data flow |
| **Threading Model** | Minimal | Context propagation uses thread-local storage (standard OTel pattern) |
| **Memory Model** | Low | Bounded queues prevent unbounded growth; RAII ensures cleanup |
| **Network Protocol** | Low | Optional fields in protobuf (high field numbers); backward compatible |
| **Configuration** | None | New config section; existing configs unaffected |
| **Build System** | Low | Optional CMake flag; builds work without OpenTelemetry |
| **Dependencies** | Low | OpenTelemetry SDK is optional; null implementation when disabled |
| Aspect | Impact | Justification |
| -------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Data Flow** | Minimal | Read-only instrumentation; no modification to consensus or transaction data flow |
| **Threading Model** | Minimal | Context propagation uses thread-local storage (standard OTel pattern) |
| **Memory Model** | Low | Bounded queues prevent unbounded growth; RAII ensures cleanup |
| **Network Protocol** | Low | Optional fields in protobuf (high field numbers); backward compatible |
| **Configuration** | None | New config section; existing configs unaffected |
| **Build System** | Low | A single CMake option (`telemetry`) selects the whole feature in or out, and builds work either way (`-Dtelemetry=ON` / `-Dtelemetry=OFF`). It is declared ON today only so CI compiles the instrumented paths; **OFF is the intended default once merged**, so the shipped build is opt-in |
| **Dependencies** | Medium | `opentelemetry-cpp/1.28.0` is a **conditional** requirement, never a hard one: `conanfile.py:152-153` adds it only `if self.options.telemetry`, and `:238-239` adds the matching `libxrpl` component requirement the same way. The option's declared default is `True` today (`conanfile.py:59`), so a default `conan install` does resolve it; with `-o telemetry=False` it never enters the graph and the null implementation supplies the factory |
### 3.9.5 Backward Compatibility
| Compatibility | Status | Notes |
| --------------- | ------- | ----------------------------------------------------- |
| **Config File** | ✅ Full | New `[telemetry]` section is optional |
| **Protocol** | ✅ Full | Optional protobuf fields with high field numbers |
| **Build** | ✅ Full | `XRPL_ENABLE_TELEMETRY=OFF` produces identical binary |
| **Runtime** | ✅ Full | `enabled=0` produces zero overhead |
| **API** | ✅ Full | No changes to public RPC or P2P APIs |
| Compatibility | Status | Notes |
| --------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Config File** | ✅ Full | New `[telemetry]` section is optional |
| **Protocol** | ✅ Full | Optional protobuf fields with high field numbers |
| **Build** | ✅ Full | `-Dtelemetry=OFF` (or `-o telemetry=False`) produces a binary with all tracing compiled out, whatever the option's declared default happens to be. **Not** `-DXRPL_ENABLE_TELEMETRY=OFF`, which does not disable anything — it is not a CMake option, only a compile definition that `CMakeLists.txt:152` adds inside the `if(telemetry)` block. CMake does flag it (`Manually-specified variables were not used by the project`) at the end of configuration, so it is not literally silent — but the warning is easy to scroll past and the resulting binary still has telemetry compiled in. See [05 §5.4.2](./05-configuration-reference.md) |
| **Runtime** | ✅ Full | `enabled=0` produces zero overhead |
| **API** | ✅ Full | No changes to public RPC or P2P APIs |
### 3.9.6 Rollback Strategy
If issues are discovered after deployment:
1. **Immediate**: Set `enabled=0` in config and restart (zero code change)
2. **Quick**: Rebuild with `XRPL_ENABLE_TELEMETRY=OFF`
1. **Immediate**: Set `enabled=0` in `[telemetry]` and restart (zero code change).
Also set `[insight] server=` to something other than `otel` if metrics must
stop too — `enabled=0` governs tracing, and the metrics pipeline is selected
separately ([02 §2.6.4](./02-design-decisions.md)).
2. **Quick**: Rebuild with `-Dtelemetry=OFF` (CMake) or `-o telemetry=False`
(Conan). Pass the flag explicitly — an omitted flag resolves to the option's
declared default, which is ON today and OFF once the feature is merged; a
build that already has telemetry off needs no rebuild at all.
**Do not use `-DXRPL_ENABLE_TELEMETRY=OFF`** — it is not a CMake option, so
it is ignored (CMake reports it under `Manually-specified variables were not
used by the project`) and the rebuilt binary still has telemetry compiled in.
This step also drops the `opentelemetry-cpp` dependency, so expect a full
rebuild rather than an incremental one.
3. **Complete**: Revert telemetry commits (clean separation makes this easy)
### 3.9.7 Code Change Examples
**Minimal RPC Instrumentation (Low Intrusiveness):** Instrumenting an RPC handler adds roughly 3-4 lines: one macro to start the span and one or two `setAttribute` calls (command name, status). The span ends automatically via RAII, so the existing control flow — process the request, send the result — is untouched.
**Consensus Instrumentation (Medium Intrusiveness):** Consensus is slightly more intrusive because child spans in later phase transitions need the round's context. Beyond the span-start and attribute macros, this requires storing the active context in a new member variable (`currentRoundContext_`) at round start. The existing round logic itself remains unchanged.
**Consensus Instrumentation (Medium Intrusiveness):** Consensus is slightly more intrusive because child spans in later phase transitions need the round's context. Beyond the span-start and attribute macros, this requires **four** new member variables on the adaptor rather than the single `currentRoundContext_` this section originally sketched (`RCLConsensus.h:103,113,123,143`):
- `std::optional<telemetry::SpanGuard> roundSpan_` (`:103`) — the round span
itself. It is **created and ended in one place**, `startRoundTracing()`: the
previous round's guard is released at `RCLConsensus.cpp:1288-1289`
(`if (roundSpan_) roundSpan_.reset();`) and the new one is emplaced a few
lines later — at `:1306` or `:1310` on the `"attribute"` strategy, at `:1319`
on the default `"deterministic"` one. `preStartRound()` does not create it; it
calls `startRoundTracing()` at `:1229`. There is no `reset()` method — the
span simply lives until the next round begins. A `SpanGuard` owns no
thread-local scope, so emplacing and resetting on different job workers is
safe.
- `telemetry::SpanContext roundSpanContext_` (`:113`) — a lightweight value-type
snapshot, captured at the end of `startRoundTracing()` (`:1350`). Child spans
link through this, not through an ambient parent, so code running on another
worker (e.g. `createValidationSpan()` on `jtACCEPT`) never touches
`roundSpan_` cross-thread.
- `telemetry::SpanContext prevRoundSpanContext_` (`:123`) — the prior round's
context, saved at `:1282` **before** the new span overwrites
`roundSpanContext_`, so the new round span can carry a follows-from link and
consecutive rounds stay navigable.
- `telemetry::SpanContext acceptSpanContext_` (`:143`) — the current round's
accept-span context, set at `:544` and cleared at `:1286` on each new round.
`createValidationSpan()` prefers it as the parent and falls back to
`roundSpanContext_` (`:1363-1373`), so a stale value must not survive into the
next round.
The split is the point: the guard is owned by one thread, the contexts are
copied freely. The existing round logic itself remains unchanged.
---

578
OpenTelemetryPlan/05-configuration-reference.md
View File

@@ -11,43 +11,115 @@
### 5.1.1 Configuration File Section
The authoritative `[telemetry]` example lives in `cfg/xrpld-example.cfg`. Telemetry is disabled by default (`enabled=0`); enabling it turns on distributed tracing for transaction flow, consensus, and RPC calls, with traces exported to an OpenTelemetry Collector over OTLP. Head sampling is intentionally fixed at 1.0 (sample everything) and is not configurable — per-node head-sampling would produce broken/partial distributed traces, so volume reduction is delegated to the collector's tail sampling (see Section 7.4.2). The full option reference follows.
The authoritative `[telemetry]` example lives in `cfg/xrpld-example.cfg`. Telemetry is disabled by default (`enabled=0`); enabling it turns on distributed tracing for transaction flow, consensus, and RPC calls, with traces exported to an OpenTelemetry Collector over OTLP. Head sampling is intentionally fixed at 1.0 (sample everything) and is not configurable — per-node head-sampling would produce broken/partial distributed traces, so volume reduction is delegated to the collector's tail sampling (see Section 7.4.2). Transaction trace IDs are always deterministic (`trace_id = txHash[0:16]`); there is no strategy switch for the transaction path. The full option reference follows.
> **`service_instance_id` is effectively required for `beast::insight`
> metrics — and only for those.** Three producers resolve the instance id
> independently, and exactly one of them lacks a node-key fallback:
>
> | Producer | Resource built by | Unset `service_instance_id` yields |
> | ------------------------------------------- | -------------------------------------------- | ---------------------------------------------- |
> | Traces (and therefore all `span_*` metrics) | `Telemetry::start()` | Base58 node public key |
> | Native `XRPL_METRIC_*` (`MetricsRegistry`) | `MetricsRegistry::initExporterAndProvider()` | Base58 node public key |
> | `beast::insight` (`[insight] server=otel`) | `TelemetryImpl` **constructor** | **`service.instance.id` absent** — no fallback |
>
> - **Traces**: the tracer resource is built in `Telemetry::start()`
> (`Telemetry.cpp:380-387`), which runs after `ApplicationImp::setup()` has
> called `setServiceInstanceId()` (`Application.cpp:1323`) with the Base58
> node public key. An unset key therefore still yields the node key. The
> `spanmetrics` connector derives `span_calls_total` /
> `span_duration_milliseconds_*` from those spans, so span metrics inherit
> the correct id too.
> - **Native `XRPL_METRIC_*` metrics** build their **own** MeterProvider
> resource in `MetricsRegistry::initExporterAndProvider()`
> (`MetricsRegistry.cpp:280`, `:296-304`, provider created at `:339`), and
> `ApplicationImp::startTelemetry()` supplies the id with an explicit node-key
> fallback (`Application.cpp:1674-1679`: read the config key, and
> `if (instanceId.empty() && nodeIdentity_)` substitute
> `toBase58(TokenType::NodePublic, …)`). By then `setup()` has resolved
> `nodeIdentity_` (`Application.cpp:1315`), so these metrics carry the node
> key even with the config key unset.
> - **`beast::insight` metrics** are the exception. They use the **global**
> MeterProvider, whose resource is built in the `TelemetryImpl`
> **constructor** (`Telemetry.cpp:321-338`, `initMetrics()` at `:447`),
> because insight instruments are created eagerly in subsystem constructors
> and would otherwise bind to the noop provider forever. At that point
> `serviceInstanceId` is still `""` (`Application.cpp:348` passes an empty
> node key), and the code comment at `Telemetry.cpp:333-336` states plainly
> that the later setter "cannot change this immutable resource". Worse,
> `initMetrics()` sets the attribute **unconditionally**
> (`Telemetry.cpp:488`), so the resource carries `service.instance.id=""`
> rather than omitting it — whereas `MetricsRegistry` guards the same write
> with `if (!instanceId.empty())` (`MetricsRegistry.cpp:302-303`).
>
> Result: with `service_instance_id` unset, `beast::insight` metrics — and only
> those — export with an empty `service.instance.id`. Every shipped Grafana
> dashboard filters on `service_instance_id=~"$node"`, so **insight-backed
> panels** lose their per-node dimension; span-metric and `XRPL_METRIC_*`
> panels are unaffected. Set the key explicitly on any node whose insight
> metrics are dashboarded.
>
> **Known issue.** The asymmetry is a defect, not a design: `MetricsRegistry`
> already demonstrates the node-key fallback that the global provider needs.
> A fix would have to resolve the node identity before `TelemetryImpl` is
> constructed, or make the insight metrics use a late-built provider.
### 5.1.2 Configuration Options Summary
| Option | Type | Default | Description |
| -------------------------- | ------ | --------------------------------- | ---------------------------------------------------------------------------------------------------------- |
| `enabled` | bool | `false` | Enable/disable telemetry |
| `endpoint` | string | `http://localhost:4318/v1/traces` | OTLP/HTTP collector endpoint |
| `use_tls` | bool | `false` | Enable TLS for exporter connection |
| `tls_ca_cert` | string | `""` | Path to CA certificate file |
| `tls_client_cert` | string | `""` | Path to node's client certificate (PEM) for mutual TLS; requires `use_tls=1`; empty = one-way TLS |
| `tls_client_key` | string | `""` | Path to private key (PEM) for `tls_client_cert`; requires `use_tls=1`; required when the cert is set |
| `batch_size` | uint | `512` | Spans per export batch |
| `batch_delay_ms` | uint | `5000` | Max delay before sending batch (ms) |
| `max_queue_size` | uint | `2048` | Maximum queued spans |
| `trace_transactions` | bool | `true` | Enable transaction tracing |
| `trace_consensus` | bool | `true` | Enable consensus tracing |
| `trace_rpc` | bool | `true` | Enable RPC tracing |
| `trace_peer` | bool | `true` | Enable peer message tracing (high volume) |
| `trace_ledger` | bool | `true` | Enable ledger tracing |
| `tx_trace_strategy` | string | `"deterministic"` | TX trace ID strategy: `"deterministic"` (trace_id = txHash[0:16]) or `"attribute"` (random) |
| `consensus_trace_strategy` | string | `"deterministic"` | Consensus trace ID strategy: `"deterministic"` (trace_id = prevLedgerHash[0:16]) or `"attribute"` (random) |
| `service_name` | string | `"xrpld"` | Service name (`service.name`) for traces and metrics |
| `service_instance_id` | string | `<node_pubkey>` | Instance identifier |
| Option | Type | Default | Description |
| -------------------------- | ------ | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `enabled` | bool | `false` | Enable/disable telemetry |
| `endpoint` | string | `http://localhost:4318/v1/traces` | OTLP/HTTP collector endpoint for **traces** |
| `metrics_endpoint` | string | `http://localhost:4318/v1/metrics` | OTLP/HTTP collector endpoint for the native metrics pipeline (`MetricsRegistry`). Read in `Application.cpp:1670` |
| `use_tls` | bool | `false` | Enable TLS for exporter connection |
| `tls_ca_cert` | string | `""` | Path to CA certificate file |
| `tls_client_cert` | string | `""` | Path to node's client certificate (PEM) for mutual TLS; requires `use_tls=1`; empty = one-way TLS |
| `tls_client_key` | string | `""` | Path to private key (PEM) for `tls_client_cert`; requires `use_tls=1`; required when the cert is set |
| `batch_size` | uint | `512` | Spans per export batch |
| `batch_delay_ms` | uint | `5000` | Max delay before sending batch (ms) |
| `max_queue_size` | uint | `2048` | Maximum queued spans |
| `trace_transactions` | bool | `true` | Enable transaction tracing |
| `trace_consensus` | bool | `true` | Enable consensus tracing |
| `trace_rpc` | bool | `true` | Enable RPC tracing |
| `trace_peer` | bool | `true` | Enable peer message tracing (high volume) |
| `trace_ledger` | bool | `true` | Enable ledger tracing |
| `consensus_trace_strategy` | string | `"deterministic"` | Consensus trace ID strategy: `"deterministic"` (trace_id = prevLedgerHash[0:16]) or `"attribute"` (random). Parsed at `TelemetryConfig.cpp:155-156`, consumed at `RCLConsensus.cpp:1291,1296`. **Not validated** — see the note below |
| `service_name` | string | `"xrpld"` | Service name (`service.name`) for traces and metrics |
| `service_instance_id` | string | node public key (base58) | Instance identifier (`service.instance.id`). Traces, span metrics and native `XRPL_METRIC_*` metrics all fall back to the node key; **`beast::insight` metrics do not** — see the note in §5.1.1 |
**`consensus_trace_strategy` is not validated.** `TelemetryConfig.cpp:155-156`
copies the raw string into `Setup::consensusTraceStrategy` without checking it
against an allowed set, and the only comparison in the code is
`strategy == "attribute"` (`RCLConsensus.cpp:1296`). Any unrecognised value —
including a typo — silently takes the deterministic branch with no log warning.
The two accepted values are documented at `include/xrpl/telemetry/Telemetry.h:287-292`.
**Not a config key — deterministic transaction trace IDs are unconditional.**
Earlier drafts of this document listed a `tx_trace_strategy` option
(`"deterministic"` \| `"attribute"`). No such key exists: `TelemetryConfig.cpp`
parses no transaction-strategy key, and the transaction trace ID is always
derived from the transaction hash. Only the **consensus** path has a
switchable strategy.
**Planned (not yet implemented)**: the following options appear in the design
documents but are not parsed by `TelemetryConfig.cpp` in Phase 1b and later
phases. They will be added as the corresponding subsystems are instrumented:
documents but are not parsed by `TelemetryConfig.cpp`. They will be added as
the corresponding subsystems are instrumented:
| Option | Planned Phase | Purpose |
| -------------------------- | ------------- | ----------------------------------------------------------------------- |
| `exporter` | Future | Select between OTLP/HTTP and OTLP/gRPC |
| `trace_pathfind` | Phase 2 | Path computation tracing toggle |
| `trace_txq` | Phase 3 | Transaction queue tracing toggle |
| `trace_validator` | Future | Validator list / manifest update tracing |
| `trace_amendment` | Future | Amendment voting tracing |
| `consensus_trace_strategy` | Phase 4 | Trace ID strategy for consensus rounds (`deterministic` \| `attribute`) |
| Option | Planned Phase | Purpose |
| ----------------- | ------------- | ---------------------------------------- |
| `exporter` | Future | Select between OTLP/HTTP and OTLP/gRPC |
| `trace_pathfind` | Phase 2 | Path computation tracing toggle |
| `trace_txq` | Phase 3 | Transaction queue tracing toggle |
| `trace_validator` | Future | Validator list / manifest update tracing |
| `trace_amendment` | Future | Amendment voting tracing |
> **`exporter` is not read, so do not set it.** Both shipped sample configs
> (`docker/telemetry/xrpld-telemetry.cfg`,
> `docker/telemetry/xrpld-telemetry-mainnet.cfg`) used to carry
> `exporter=otlp_http`; the line had no effect and has since been replaced with
> a comment saying so. OTLP/HTTP is the only transport that exists (§2.2.1), and
> `endpoint` / `metrics_endpoint` are the only transport knobs, until the §2.2.2
> gRPC work lands.
---
@@ -55,7 +127,16 @@ phases. They will be added as the corresponding subsystems are instrumented:
> **TxQ** = Transaction Queue
The parser `makeTelemetrySetup()` in `src/libxrpl/telemetry/TelemetryConfig.cpp` reads the `[telemetry]` `Section` and populates a `Telemetry::Setup` struct, applying the defaults listed in Section 5.1.2 via `section.value_or(...)`. It derives `serviceInstanceId` from the node public key when not overridden, selects the exporter endpoint default by exporter type, and leaves the sampling ratio at its fixed 1.0 default (not read from config — see Section 7.4.2).
The parser `makeTelemetrySetup()` in `src/libxrpl/telemetry/TelemetryConfig.cpp` reads the `[telemetry]` `Section` and populates a `Telemetry::Setup` struct, applying the defaults listed in Section 5.1.2 via `section.valueOr(...)`. It takes `serviceInstanceId` from the `nodePublicKey` argument when the key is absent, applies one unconditional `endpoint` default (`dflt::endpoint`, `TelemetryConfig.cpp:61`, used at `:108`) — the parser has no notion of exporter type — and leaves the sampling ratio at its fixed 1.0 default (a `static constexpr` member, so there is nothing to parse; `TelemetryConfig.cpp:139`, `Telemetry.h:234`). It also rejects two contradictory mTLS configurations outright (`tls_client_cert` without `tls_client_key`, and either without `use_tls=1`) rather than failing open at handshake time.
`metrics_endpoint` is deliberately **not** handled here: it is read separately in `ApplicationImp::startTelemetry()` (`Application.cpp:1670`) and passed to `MetricsRegistry::start()`. Note the consequence — the two metric exporters resolve their URL differently:
| Metric source | Exporter built by | URL comes from |
| ------------------------------------------ | -------------------------------------------- | -------------------------------------------------------------------- |
| `beast::insight` (`[insight] server=otel`) | `Telemetry::initMetrics()` (global provider) | `endpoint` with a trailing `/v1/traces` rewritten to `/v1/metrics` |
| Native `XRPL_METRIC_*` (`MetricsRegistry`) | `MetricsRegistry::initExporterAndProvider()` | `metrics_endpoint`, defaulting to `http://localhost:4318/v1/metrics` |
Setting a non-default `endpoint` therefore moves the insight metrics with it, but leaves the native metrics on localhost unless `metrics_endpoint` is set too.
---
@@ -68,6 +149,13 @@ The parser `makeTelemetrySetup()` in `src/libxrpl/telemetry/TelemetryConfig.cpp`
> resolved later in `setup()`. The `Telemetry` object is therefore
> constructed with an empty `serviceInstanceId` and patched via
> `setServiceInstanceId()` once `setup()` has called `getNodeIdentity()`.
> **This patch reaches traces only.** The **global** MeterProvider resource —
> the one `beast::insight` metrics use — is already frozen by then (§5.1.1), so
> those metrics keep whatever `service_instance_id` the config supplied (`""`
> if it supplied none). Native `XRPL_METRIC_*` metrics do not go through this
> patch at all: `startTelemetry()` re-reads the config key and applies its own
> node-key fallback when building `MetricsRegistry`'s separate resource
> (`Application.cpp:1674-1679`).
`ApplicationImp` (in `src/xrpld/app/main/Application.cpp`) owns a `std::unique_ptr<telemetry::Telemetry> telemetry_`. It is built in the member initializer list via `makeTelemetry(makeTelemetrySetup(...))` with an empty `serviceInstanceId`, then patched in `setup()` by calling `setServiceInstanceId()` with the Base58 node public key (unless the user supplied a custom `service_instance_id`). `start()` and `run()` forward to `telemetry_->start()` / `telemetry_->stop()`, and `getTelemetry()` returns the owned instance.
@@ -87,13 +175,72 @@ The parser `makeTelemetrySetup()` in `src/libxrpl/telemetry/TelemetryConfig.cpp`
> **OTLP** = OpenTelemetry Protocol
### 5.4.1 Find OpenTelemetry Module
### 5.4.1 Locating the OpenTelemetry SDK
A `cmake/FindOpenTelemetry.cmake` module locates the OpenTelemetry C++ SDK. It first tries `find_package(opentelemetry-cpp CONFIG)`, aliasing the imported targets `OpenTelemetry::api`, `OpenTelemetry::sdk`, and `OpenTelemetry::otlp_grpc_exporter`, and falls back to `pkg-config` when no CMake config package is present.
> **Superseded design.** Earlier drafts described a hand-written
> `cmake/FindOpenTelemetry.cmake` module that aliased `OpenTelemetry::api`,
> `OpenTelemetry::sdk` and `OpenTelemetry::otlp_grpc_exporter` with a
> `pkg-config` fallback. That module was never written — it exists in no
> commit — and the aliasing approach it described does not work with the
> package the build actually consumes.
The SDK is located by the Conan-generated CMake config package, nothing else:
- `CMakeLists.txt` — `find_package(opentelemetry-cpp CONFIG REQUIRED)`,
guarded by the `telemetry` option (§5.4.2). The dependency itself is
declared in `conanfile.py:153` (`opentelemetry-cpp/1.28.0`), also guarded —
`requirements()` adds it only `if self.options.telemetry` (`:152`), so with
the option off the package never enters the dependency graph.
- Linking goes through the **umbrella** target
`opentelemetry-cpp::opentelemetry-cpp`, never the per-component targets.
`cmake/XrplCore.cmake:221-225` and `:83-91` record why: the Conan package
under-declares its inter-component dependencies, so naming `::api` / `::sdk`
individually produces the wrong static-link order and fails at executable
link time. The umbrella target supplies both the trace and metrics
components with the correct ordering.
### 5.4.2 CMakeLists.txt Changes
The top-level `CMakeLists.txt` adds an `XRPL_ENABLE_TELEMETRY` option (default `OFF`). When enabled, it runs `find_package(OpenTelemetry REQUIRED)`, defines the `XRPL_ENABLE_TELEMETRY` compile flag, and builds the `xrpl_telemetry` library from the real telemetry sources linked against the OpenTelemetry targets; when disabled, it builds the same target from a no-op `NullTelemetry.cpp` so call sites compile unchanged.
The build flag is `telemetry`:
```
option(telemetry "Enable OpenTelemetry tracing" ON) # top-level CMakeLists.txt
```
The declared value is ON **temporarily**, so that CI compiles the telemetry code
paths while the feature branches are in review. **OFF is the intended default
once merged**, and the flip is a separate change. Set the value explicitly
rather than relying on the default:
| To … | Use (CMake) | Use (Conan) |
| ------------------------- | ----------------- | -------------------- |
| Build telemetry in | `-Dtelemetry=ON` | `-o telemetry=True` |
| Build it out (all no-ops) | `-Dtelemetry=OFF` | `-o telemetry=False` |
When the option is ON, the guarded block below it runs
`find_package(opentelemetry-cpp CONFIG REQUIRED)` and adds the
**compile definition** `XRPL_ENABLE_TELEMETRY`.
> **`XRPL_ENABLE_TELEMETRY` is not a CMake option.** It is only ever _added_
> as a compile definition by `add_compile_definitions(XRPL_ENABLE_TELEMETRY)` in that same block. Passing
> `-DXRPL_ENABLE_TELEMETRY=OFF` on the CMake command line disables **nothing** —
> it defines an unused cache variable and telemetry stays compiled in. CMake does
> report it, at the end of configuration under `Manually-specified variables were
not used by the project`, so it is not literally silent — but that line is easy
> to scroll past. Any procedure that relies on it (including the rollback path in
> [§3.9.6](./03-implementation-strategy.md)) must use `-Dtelemetry=OFF`.
The target is `xrpl.libxrpl.telemetry`, created by `add_module(xrpl telemetry)`
at `cmake/XrplCore.cmake:231` from `include/xrpl/telemetry/` +
`src/libxrpl/telemetry/`. There is no `xrpl_telemetry` target.
Selection between the real and the no-op implementation is an **in-source
`#ifdef`, not a source swap**: `NullTelemetry.cpp` is compiled into the target
unconditionally (see its header comment, `NullTelemetry.cpp:1-12`). It provides
the `makeTelemetry()` factory when `XRPL_ENABLE_TELEMETRY` is undefined; when
the macro is defined, `Telemetry.cpp` provides the factory instead and
`NullTelemetry`'s virtuals only serve as noop tracer/span fallbacks. Call sites
compile unchanged either way.
---
@@ -105,13 +252,101 @@ The top-level `CMakeLists.txt` adds an `XRPL_ENABLE_TELEMETRY` option (default `
The authoritative collector config lives in the repo at `docker/telemetry/otel-collector-config.yaml` (with Tempo backend config in `docker/telemetry/tempo.yaml`). The sections below summarize the development and production shapes of that pipeline.
### 5.5.1 Development Configuration
### 5.5.1 Development / Base Configuration
The development collector enables an OTLP receiver on both gRPC (`0.0.0.0:4317`) and HTTP (`0.0.0.0:4318`), a single `batch` processor (1s timeout, batch size 100), and two exporters: a `logging` exporter for console debugging and `otlp/tempo` (insecure) for trace visualization. The single `traces` pipeline wires receiver → batch → both exporters.
`docker/telemetry/otel-collector-config.yaml` is the base config used by the
local stack and by CI. It carries **three** pipelines, not one:
| Pipeline | Receivers | Processors | Exporters |
| --------- | --------------------- | ---------------------------------------------------------------- | ------------------------------------ |
| `traces` | `otlp` | `resource/tier`, `resource/stripsdk`, `attributes/hash`, `batch` | `debug`, `otlp/tempo`, `spanmetrics` |
| `metrics` | `otlp`, `spanmetrics` | `resource/tier`, `resource/stripsdk`, `batch` | `prometheus` |
| `logs` | `filelog` | `resource/logs`, `resource/tier`, `resource/stripsdk`, `batch` | `otlphttp/loki` |
Component detail:
- **Receivers.** `otlp` on gRPC `0.0.0.0:4317` and HTTP `0.0.0.0:4318` (both
traces and native metrics arrive on 4318). `filelog` tails
`/var/log/xrpld/*/debug.log` and runs a `regex_parser` that lifts
`timestamp`, `partition`, `severity` and the optional `trace_id`/`span_id`
emitted by the journal sink (§5.8.5).
- **Processors.** `batch` (1s timeout, `send_batch_size: 100`);
`resource/tier` (`action: upsert` on `deployment.environment`, `action: insert` on
`xrpl.network.type` only when absent); `resource/stripsdk` (drops the
`telemetry.sdk.*` attributes); `resource/logs` (`action: upsert` on
`service.name` and `job` — only the former becomes a Loki stream label, see
the known issue in §5.8.5); `attributes/hash` (hashes
`pathfind_source_account` and `pathfind_dest_account`).
- **Connector.** `spanmetrics` with `namespace: "span"`
(`otel-collector-config.yaml:114`) — this is why the derived RED metrics are
`span_calls_total` / `span_duration_milliseconds_*`. The connector's own
default namespace is **empty**, so without this setting the names would be
the bare `calls_total` / `duration_milliseconds_*`. The
`traces_spanmetrics_*` family is **not** the connector's default and is not
produced here at all — it comes from a different producer, Tempo's
`metrics_generator` `span-metrics` processor (`tempo.yaml:75`), whose
`remote_write` is commented out in this repo (see §5.8.6). Histogram
`unit: ms`
with sub-millisecond buckets from `0.01ms`, plus explicit `2s`–`30s`
boundaries for consensus and `ledger.acquire`. ~25 low-cardinality
dimensions are promoted to labels (`command`, `rpc_status`, `tx_type`,
`ter_result`, `stage`, `consensus_mode`, `outcome`, …).
- **Exporters.** `debug` (console, `verbosity: detailed`), `otlp/tempo`
(`tempo:4317`, `tls.insecure: true`), `otlphttp/loki`
(`http://loki:3100/otlp` — Loki 3.x native OTLP; the old `loki` exporter was
removed in collector-contrib v0.147.0), and `prometheus` on
`0.0.0.0:8889` with `resource_to_telemetry_conversion.enabled: true` so the
tier and instance resource attributes become Prometheus labels.
- **Extensions.** `health_check` on `0.0.0.0:13133` only. There is **no**
`zpages` extension.
Deliberately absent from the base config — do not document them as present:
no `memory_limiter`, no `tail_sampling`, no Elastic APM exporter, and no
`tx_account` attribute rule (the hashed keys are the two `pathfind_*_account`
ones).
### 5.5.2 Production Configuration
The production collector adds TLS on the OTLP gRPC receiver and a richer processor chain: a `memory_limiter` (OOM guard), `batch` (5s timeout, size 512), `tail_sampling`, and an `attributes` processor that hashes sensitive fields (e.g. `tx_account`) and stamps `deployment.environment`. Tail sampling keeps all `ERROR` traces, slow consensus rounds (>5s) and slow RPC requests (>1s), and probabilistically samples the remainder at 10%. Exporters target Grafana Tempo (TLS) and Elastic APM; `health_check` and `zpages` extensions are enabled for operability.
There is no separate "production" collector config in this repo. The one
overlay that exists is `docker/telemetry/otel-collector-config.grafanacloud.yaml`.
It is **not** the base config plus one processor — it restructures the service
graph. The full delta:
| Added by the overlay | Where | Purpose |
| ------------------------ | ------ | ------------------------------------------------------------------------- |
| `basicauth/grafanacloud` | `:29` | Extension; instance id / API token from the container environment |
| `tail_sampling` | `:60` | One `probabilistic` policy at **0.5%**, `decision_wait: 10s` |
| `transform/cloudlabels` | `:119` | Copies three resource attrs onto datapoint labels for Cloud (OTLP) ingest |
| `otlphttp/grafanacloud` | `:236` | Single OTLP/HTTP exporter fanning all three signals to Grafana Cloud |
| `metrics_flush_interval` | `:136` | `spanmetrics` flushes every 15s instead of the 60s default |
| Removed by the overlay | Consequence |
| ---------------------- | ---------------------------------------------------------------------------- |
| `attributes/hash` | **Pathfinding account attributes are not hashed on this config** — see below |
| `debug` | No console span dump; collector logs alone when diagnosing ingest |
Pipelines go from **three** (`traces`, `metrics`, `logs`) to **five**
(`:253-280`): `traces/metrics`, `traces/store`, `metrics/local`,
`metrics/cloud`, `logs`. `tail_sampling` is applied in **`traces/store`**
(`:259-261`) — the branch feeding Tempo and Grafana Cloud — not in a pipeline
named `traces`, which does not exist in the overlay. The `traces/metrics`
branch feeds `spanmetrics` unsampled, so the derived RED metrics stay exact
while stored traces are ~1/200 of ingested ones.
> **Known issue — the cloud path does not hash pathfinding accounts.** The base
> config runs `attributes/hash` on its `traces` pipeline
> (`otel-collector-config.yaml:105-110`), hashing `pathfind_source_account` and
> `pathfind_dest_account` as defense in depth behind the node-side hashing. The
> overlay declares no such processor and lists none on any of its five
> pipelines, so on the Grafana Cloud config those two attributes reach **both**
> Grafana Cloud and the local Tempo with whatever value the node sent. Any node
> that emits raw addresses loses its second line of defense. Adding
> `attributes/hash` to `traces/store` and `traces/metrics` would close the gap.
Hardening a collector for a real deployment (TLS/mTLS on the receiver,
NetworkPolicy, peer trace-context validation) is covered in
[Securing the OTel Pipeline](./secure-OTel.md) — not by any config file in
`docker/telemetry/`.
---
@@ -119,7 +354,31 @@ The production collector adds TLS on the OTLP gRPC receiver and a richer process
> **OTLP** = OpenTelemetry Protocol
The authoritative development stack lives in the repo at `docker/telemetry/docker-compose.yml`. It brings up four services on a shared `xrpld-telemetry` network: an `otel-collector` (otel/opentelemetry-collector-contrib) exposing OTLP gRPC `4317`, OTLP HTTP `4318`, and health check `13133`; `tempo` for trace storage/visualization; `grafana` with provisioned datasources and dashboards (anonymous admin enabled); and an optional `prometheus` for metric correlation.
The authoritative development stack lives in the repo at `docker/telemetry/docker-compose.yml`. It brings up **six** services on a shared `xrpld-telemetry` bridge network. All images are pinned to exact tags.
| Service | Image | Published ports | Role |
| ---------------- | ---------------------------------------------- | ---------------------- | ---------------------------------------------------------------- |
| `otel-collector` | `otel/opentelemetry-collector-contrib:0.158.0` | `4317`, `4318`, `8889` | OTLP ingest, spanmetrics, filelog tail, Prometheus scrape target |
| `tempo` | `grafana/tempo:2.9.4` | `3200` | Trace storage and TraceQL |
| `loki` | `grafana/loki:3.7.6` | `3100` | Log storage for log↔trace correlation |
| `prometheus` | `prom/prometheus:v3.13.2` | `9090` | Scrapes the collector's `:8889` |
| `grafana` | `grafana/grafana:13.1.2` | `3000` | Dashboards + provisioned datasources/alerts, anonymous admin |
| `renderer` | `grafana/grafana-image-renderer:v5.12.0` | `8081` | Panel→PNG rendering for image export and alert screenshots |
Two corrections to earlier drafts:
- **`prometheus` is not optional.** `grafana` lists it in `depends_on` (along
with `tempo`, `loki` and `renderer`), and 7 of the 15 dashboards query
`span_calls_total` from it. Removing it blanks most panels.
- **Port `13133` is not published.** The collector's `health_check` extension
listens on `13133` inside the container, but the base compose file publishes
only `4317`, `4318` and `8889`. Health checks from the host must either add a
port mapping or run `docker compose exec`.
The collector also bind-mounts the xrpld log root read-only
(`${XRPLD_LOG_DIR:-./data/logs}` → `/var/log/xrpld`) for the `filelog`
receiver, and the `grafana` service reads Slack/email alert secrets from an
optional gitignored `.env.alerting`.
---
@@ -131,7 +390,7 @@ The authoritative development stack lives in the repo at `docker/telemetry/docke
flowchart TB
subgraph config["Configuration Sources"]
cfgFile["xrpld.cfg<br/>[telemetry] section"]
cmake["CMake<br/>XRPL_ENABLE_TELEMETRY"]
cmake["CMake option: telemetry<br/>ON today for CI, OFF once merged<br/>when ON, defines XRPL_ENABLE_TELEMETRY"]
end
subgraph init["Initialization"]
@@ -168,7 +427,7 @@ flowchart TB
**Reading the diagram:**
- **Configuration Sources**: `xrpld.cfg` provides runtime settings (endpoint, per-component trace toggles) while the CMake flag controls whether telemetry is compiled in at all. Head sampling is fixed at 1.0 and is not a config option; volume reduction happens via tail sampling in the collector.
- **Configuration Sources**: `xrpld.cfg` provides runtime settings (endpoint, per-component trace toggles) while the CMake `telemetry` option controls whether telemetry is compiled in at all. That option is declared ON today only so CI compiles the instrumented paths; OFF is the intended default once merged, so treat the build gate as something to pass explicitly, and the runtime gate is opt-in either way (`enabled=0` by default). Head sampling is fixed at 1.0 and is not a config option; volume reduction happens via tail sampling in the collector.
- **Initialization**: `makeTelemetrySetup()` parses config values, then `makeTelemetry()` constructs the provider, processor, and exporter objects.
- **Runtime Components**: The `TracerProvider` creates spans, the `BatchProcessor` buffers them, and the `OTLP Exporter` serializes and sends them over the wire.
- **OTLP arrow to Collector**: Trace data leaves the xrpld process via OTLP/HTTP and enters the external Collector pipeline. (OTLP/gRPC is future work — see design decisions §2.2.2.)
@@ -184,29 +443,88 @@ Step-by-step instructions for integrating xrpld traces with Grafana.
### 5.8.1 Data Source Configuration
#### Tempo (Recommended)
Three datasources are provisioned from `docker/telemetry/grafana/provisioning/datasources/`. There is **no** Elastic APM datasource — `elastic-apm.yaml` was described in an earlier draft but never existed. Elastic remains a _possible_ backend (§7.2); nothing in this repo provisions it.
A Tempo datasource (`grafana/provisioning/datasources/tempo.yaml`, provisioned from `docker/telemetry/grafana/`) points at `http://tempo:3200` and enables `tracesToLogs` (linking to Loki on `service.name`/`tx_hash` and mapping `trace_id` → `traceID`), `serviceMap` against Prometheus, the node graph, and Loki search.
| File | Type | URL | uid | Notes |
| ----------------- | ------------ | ------------------------ | ------------ | -------------------------------------------------------------------------------------------------------------------- |
| `tempo.yaml` | `tempo` | `http://tempo:3200` | `tempo` | `nodeGraph`, `serviceMap`/`tracesToMetrics` → `prometheus`, `tracesToLogs` → `loki`, plus ~30 Explore search filters |
| `prometheus.yaml` | `prometheus` | `http://prometheus:9090` | `prometheus` | Backs every span-metric and native-metric panel |
| `loki.yaml` | `loki` | `http://loki:3100` | `loki` | Backs `log-derived-insights`; derived fields jump back to Tempo |
#### Elastic APM
The Tempo `tracesToLogs` block is configured as `filterByTraceID: true`,
`filterBySpanID: false`, **`tags: []`**. The empty tag list is deliberate: the
correlation is by trace ID alone, so no span attribute needs to exist on both
sides. Earlier drafts claimed `trace_id` + `tx_hash` tags — that is not what
ships, and adding a tag Tempo cannot resolve blanks the link.
Alternatively, an Elasticsearch datasource (`grafana/provisioning/datasources/elastic-apm.yaml`) of type `elasticsearch` points at `http://elasticsearch:9200` against the `apm-*` index, using `@timestamp` as the time field and mapping the log message/level fields.
The search-filter list is the practical index of queryable span attributes:
resource scope (`service.name`, `service.instance.id`, `service.version`,
`xrpl.network.id`, `xrpl.network.type`), intrinsics (`name`, `status`,
`duration`), and span scope (`command`, `rpc_status`, `rpc_role`, `tx_hash`,
`tx_type`, `tx_status`, `local`, `path`, `suppressed`, `peer_version`,
`consensus_*`, `ledger_seq`, `ledger_hash`, `close_time_correct`,
`close_resolution_ms`, `proposers`, `mode_old`, `mode_new`, `txq_status`,
`ter_code`).
### 5.8.2 Dashboard Provisioning
A dashboard provider (`grafana/provisioning/dashboards/dashboards.yaml`) loads the `xrpld` dashboard folder from disk (`/var/lib/grafana/dashboards/rippled`), polling for changes every 30s with deletion disabled.
`grafana/provisioning/dashboards/dashboards.yaml` declares a single `file`
provider named `xrpld-telemetry`, `orgId: 1`, targeting Grafana folder `xrpld`
from path `/var/lib/grafana/dashboards` (no `/rippled` suffix), with
`disableDeletion: false`, `editable: true`, `foldersFromFilesStructure: false`.
It sets **no** poll interval — Grafana's `updateIntervalSeconds` default
applies; the "every 30s" figure in earlier drafts was invented.
### 5.8.3 Example Dashboard: RPC Performance
`docker-compose.yml` mounts `./grafana/dashboards` read-only at that path, so
the 15 JSON files in `docker/telemetry/grafana/dashboards/` are what gets
provisioned.
An example `xrpld RPC Performance` dashboard (uid `xrpld-rpc-performance`) sourced from Tempo via TraceQL provides four panels: RPC latency by command (heatmap), RPC error rate by command (timeseries), the top 10 slowest RPC commands by average duration (table), and a recent-traces table.
### 5.8.3 Shipped Dashboards
### 5.8.4 Example Dashboard: Transaction Tracing
The dashboards are Prometheus-first, not TraceQL-first, and their uids are
bare (no `xrpld-` prefix). The full inventory and per-panel query reference is
[09-data-collection-reference.md](./09-data-collection-reference.md); the uids
are:
An example `xrpld Transaction Tracing` dashboard (uid `xrpld-tx-tracing`) over Tempo provides three panels: transaction throughput (`tx.receive` rate, stat), cross-node relay count (average `span.relay_count` on `tx.relay`, timeseries), and a table of transaction validation errors (`tx.validate` with `status.code=error`).
`consensus-health`, `fee-market`, `job-queue`, `ledger-data-sync`,
`ledger-operations`, `log-derived-insights`, `network-traffic`, `node-health`,
`overlay-traffic-detail`, `peer-network`, `peer-quality`, `rpc-pathfinding`,
`rpc-performance`, `transaction-overview`, `validator-health`.
### 5.8.5 TraceQL Query Examples
> **Panel-count convention used in these docs**: counts are of **data panels
> only** — `type: "row"` collapsible headers are excluded, because a row is a
> layout element with no query. A board's raw `panels` array is therefore longer
> than its stated count (e.g. `rpc-performance` has 19 array entries: 2 rows +
> 17 data panels).
Common queries for xrpld traces:
Two examples described in earlier drafts do not exist and should not be looked
for: `xrpld-rpc-performance` (the real board is `rpc-performance`, **17** data
panels in 2 rows, mostly Prometheus span metrics) and `xrpld-tx-tracing` (the
transaction board is `transaction-overview`, **18** data panels in 3 rows; its
error panel filters `span_calls_total{span_name="tx.process",
ter_result!~"tesSUCCESS|"}`, since no `tx.validate` span was ever built — see
[02 §2.3.2](./02-design-decisions.md)).
> **Why `!~"tesSUCCESS|"` and not `!="tesSUCCESS"`.** An absent Prometheus label
> compares equal to the empty string, and `tx.process` can end **without** a
> `ter_result` attribute: `processTransaction()` returns early when
> `preProcessTransaction()` rejects the transaction
> (`NetworkOPs.cpp:1437-1438`) and `doTransactionAsync()` returns early when the
> transaction is already applying (`:1461-1462`); the only setter runs later, at
> `:1674`. Those series carry `ter_result=""`, which `!="tesSUCCESS"` counts as
> an error. The regex form excludes the empty value explicitly (the trailing
> `|` alternative), which is the form `docs/telemetry-runbook.md:1198` and two
> of the three `transaction-overview.json` failure panels already use.
Every dashboard exposes a `$node` template variable bound to
`service_instance_id`; see the §5.1.1 note on why `service_instance_id` must be
set for metric panels to split per node.
### 5.8.4 TraceQL Query Examples
Common queries for xrpld traces. Every span name and attribute below is one
that the code actually emits — check against the `*SpanNames.h` constants
before adding more.
```
# Find all traces for a specific transaction hash
@@ -218,57 +536,163 @@ Common queries for xrpld traces:
# Find consensus rounds taking >5 seconds
{resource.service.name="xrpld" && name="consensus.round"} | duration > 5s
# Find failed transactions with error details
{resource.service.name="xrpld" && name="tx.validate" && status.code=error}
# Find failed transaction processing
{resource.service.name="xrpld" && name="tx.process" && span.ter_result!="tesSUCCESS"}
# Find transactions relayed to many peers
{resource.service.name="xrpld" && name="tx.relay"} | span.relay_count > 10
# Find failed apply-pipeline stages (preflight / preclaim / transactor)
{resource.service.name="xrpld" && name=~"tx\\.(preflight|preclaim|transactor)" && status=error}
# Find transactions that arrived from a peer rather than a local client.
# The `local` attribute lives on tx.process, NOT on tx.receive (see the note
# below).
{resource.service.name="xrpld" && name="tx.process" && span.local=false}
# Compare latency across nodes
{resource.service.name="xrpld" && name="rpc.command.account_info"} | avg(duration) by (resource.service.instance.id)
```
### 5.8.6 Correlation with PerfLog
> Queries in earlier drafts used `tx.validate`, `tx.relay` and
> `span.relay_count`. None of the three exists: signature/format validation
> ships as `tx.preflight`/`tx.preclaim`, and no relay span or relay-count
> attribute was ever built. See [02 §2.3.2](./02-design-decisions.md).
To correlate OpenTelemetry traces with existing PerfLog data:
> **TraceQL silently returns nothing for an absent attribute.** Unlike PromQL,
> where a missing label compares equal to `""`, a TraceQL attribute predicate
> matches only spans that actually carry the attribute — including negated
> forms such as `!=` and `=~".*"`. So filtering on the wrong span name yields
> zero rows with no error. `local` has exactly one set-site,
> `NetworkOPs.cpp:1417`, and it is on **`tx.process`**: an earlier draft paired
> it with `name="tx.receive"`, which can never match. Check the attribute's
> owning span in
> [09 §1.2](./09-data-collection-reference.md) before combining a `name=` and a
> `span.` predicate.
**Step 1: Configure Loki to ingest PerfLog**
### 5.8.5 Correlation with Logs
Configure a Promtail scrape job (`promtail-config.yaml`) that tails `/var/log/rippled/perf*.log`, parses each JSON line, and promotes `trace_id`, `ledger_seq`, and `tx_hash` to Loki labels.
Log↔trace correlation is **implemented** (Phase 8) and needs no Promtail,
Fluentd or PerfLog change. Two pieces:
**Step 2: Add trace_id to PerfLog entries**
1. **The node stamps the IDs.** The journal sink `Logs::format()`
(`src/libxrpl/basics/Log.cpp:304-338`, guarded by `XRPL_ENABLE_TELEMETRY`)
reads the thread-local OTel context and, when a valid span is active,
prefixes the message with `trace_id=<32 hex> span_id=<16 hex>`. It reads
the context value directly rather than calling `GetSpan()` to avoid a heap
allocation on the (common) no-span path. This is the ordinary `debug.log`
stream — PerfLog is not involved, and the `setTraceId` hook described in
earlier drafts was never built.
2. **The collector ingests them.** The `filelog` receiver tails
`/var/log/xrpld/*/debug.log` and its `regex_parser` lifts `trace_id` and
`span_id` as optional capture groups (§5.5.1). `resource/logs` applies an
`upsert` of `service.name=xrpld`, which Loki promotes to the stream label
`service_name`, so the canonical selector is **`{service_name="xrpld"}`**.
Logs land in Loki via `otlphttp/loki`.
Modify PerfLog so its JSON output includes a `trace_id` field whenever a valid span is active: fetch the current span from the OpenTelemetry runtime context, and if its context is valid, render the trace ID as a 32-character lowercase hex string into the log entry.
> **Known issue — the collector's `job` upsert is ineffective for stream
> selection.** `resource/logs` also applies an `upsert` of a `job=xrpld` attribute
> (`otel-collector-config.yaml:62-70`) with the stated intent that operators
> could paste `{job="xrpld"}`. That does not work. On OTLP ingest Loki promotes
> only an **allow-listed** set of resource attributes to indexed stream labels
> (`service.name`, `service.namespace`, `service.instance.id`,
> `deployment.environment`, the `k8s.*`/`cloud.*` keys); `job` is not on that
> list, and this repo ships no Loki config override — `docker-compose.yml:75`
> starts Loki with the image's built-in `/etc/loki/local-config.yaml`. `job`
> therefore lands in **structured metadata**, which cannot appear in a stream
> selector, so `{job="xrpld"}` returns an empty result rather than an error.
> Corroboration in-repo: `docs/telemetry-runbook.md:2533` states the same
> ("`service_name="xrpld"` (not `job="xrpld"`)"), and **all 38 Loki queries** in
> the shipped dashboards (35 panel targets + 3 Loki-backed template variables)
> select on `service_name` — **zero** use `job`. Either drop the `job`
> upsert or add `job` to Loki's `distributor.otlp_config.resource_attributes`
> allow-list via a mounted Loki config; until then, use `service_name`.
**Step 3: Configure Grafana trace-to-logs link**
Grafana then links the two directions: the Tempo datasource's `tracesToLogs`
(`filterByTraceID: true`, `tags: []`) jumps trace → logs, and `loki.yaml`'s
derived fields jump log → trace.
In the Tempo datasource, set the `tracesToLogs` derived field to link to Loki on the `trace_id` and `tx_hash` tags, with `filterByTraceID: true`.
### 5.8.7 Correlation with Insight/OTel System Metrics
### 5.8.6 Correlation with Insight/OTel System Metrics
To correlate traces with Beast Insight system metrics:
**Step 1: Export Insight metrics to Prometheus**
Beast Insight metrics are exported natively via OTLP to the OTel Collector,
which exposes them on the Prometheus endpoint alongside spanmetrics. Configure
the `[insight]` section of `xrpld.cfg` with `server=otel`,
`endpoint=http://localhost:4318/v1/metrics`, and `prefix=xrpld`; no separate
StatsD exporter or Prometheus scrape job is needed when using `server=otel`.
which exposes them on its Prometheus endpoint (`:8889`) alongside spanmetrics.
Set `server=otel` in the `[insight]` section of `xrpld.cfg`; no separate StatsD
exporter or Prometheus scrape job is needed.
**Step 2: Add exemplars to metrics**
`makeCollectorManager()` (`src/xrpld/app/main/CollectorManager.cpp`) reads these
`[insight]` keys:
The OpenTelemetry SDK automatically adds exemplars (trace IDs) to metrics when using the Prometheus exporter, linking metric spikes to specific traces.
| Key | Read at | Effect when `server=otel` |
| --------------------- | ------------ | ---------------------------------------------------------------------------------------------------------------------------------- |
| `server` | `:35` | **Live.** `statsd` \| `otel` \| anything else. Selects the collector implementation. |
| `address` | `:39` | StatsD only — the UDP endpoint. |
| `prefix` | `:41`, `:53` | **Inert.** Stored on the OTel collector but `formatName()` prepends nothing (`OTelCollector.cpp:855-866`); only StatsD applies it. |
| `endpoint` | `:50` | **Inert.** Logged for diagnostics (`OTelCollector.cpp:730`), then unused. |
| `service_instance_id` | `:58` | **Inert.** `(void)`-discarded (`OTelCollector.cpp:722`). |
| `service_name` | `:64` | **Inert.** `(void)`-discarded (`OTelCollector.cpp:723`). |
**Step 3: Configure Grafana metric-to-trace link**
> **Where the identity and endpoint actually come from.** `OTelCollector`
> deliberately does **not** own a pipeline: it fetches the Meter from the
> **global** MeterProvider that `Telemetry::initMetrics()` published
> (`OTelCollector.cpp:726-745`). So the resource attributes — including
> `service.instance.id`, which every dashboard filters on — and the exporter
> URL both come from the **`[telemetry]`** section, not `[insight]`. The four
> inert keys above are back-compat leftovers from the StatsD-era signature;
> setting them has no effect. Set `[telemetry] service_instance_id` instead
> (§5.1.1).
In the Prometheus datasource, set `exemplarTraceIdDestinations` to map the `trace_id` exemplar to the Tempo datasource.
> **`server=otel` is not the default.** `CollectorManager.cpp:72-75` falls through
> to `NullCollector` for any unrecognised or absent `server` value, so a node
> with no `[insight]` section emits no metrics at all.
**Step 4: Dashboard panel with exemplars**
**Step 2: Correlate metrics to traces**
Add a timeseries panel over Prometheus (e.g. `histogram_quantile(0.99, rate(rpc_duration_seconds_bucket[5m]))`) with `exemplar: true` enabled.
Today this is a **time-range** correlation, not a click-through one: note the
window from the metric panel, then search Tempo over the same window filtered
by `service.instance.id`.
This allows clicking on metric data points to jump directly to the related trace.
> **Exemplars are NOT implemented.** Earlier drafts of this section instructed
> operators to rely on automatic exemplars, set
> `exemplarTraceIdDestinations` on the Prometheus datasource, and enable
> `exemplar: true` on panels. None of that is wired up: the string `exemplar`
> appears **nowhere** in `src/libxrpl/telemetry/`, `src/xrpld/telemetry/`, or
> `docker/telemetry/`. Concretely, three things are missing —
>
> 1. the SDK's exemplar filter is left at its default and no reservoir is
> configured in `Telemetry::initMetrics()` or `MetricsRegistry`;
> 2. the collector's `prometheus` exporter has no exemplar settings;
> 3. `grafana/provisioning/datasources/prometheus.yaml` has no
> `exemplarTraceIdDestinations` block.
>
> Note also that the query used as an example, `rpc_duration_seconds_bucket`,
> does not exist — RPC latency histograms are `span_duration_milliseconds_bucket`
> (spanmetrics, `unit: ms`) and `rpc_method_us` (native). Wiring exemplars end
> to end is genuine open work; until it lands, do not document a click-through
> that operators cannot perform.
**Step 3: Jump the other way instead**
Trace → metrics is available now: the Tempo datasource sets
`tracesToMetrics.datasourceUid: prometheus` with a ±1h time shift, so the
span-metric queries it builds resolve against the `span_*` families the
collector's `spanmetrics` connector produces. Trace → logs and log → trace are
both live (§5.8.5).
> **Known gap — Service Map is configured but inactive.** The Tempo datasource
> declares `serviceMap.datasourceUid: prometheus`, and `tempo.yaml:70-76`
> enables the `service-graphs` metrics-generator processor, but the generator
> has nowhere to write: its `remote_write` block is **commented out**
> (`tempo.yaml:53-56`), and `prometheus.yml:6-9` defines a single scrape job
> against `otel-collector:8889` — it never scrapes or accepts writes from
> Tempo. `traces_service_graph_request_total` and its siblings are therefore
> never stored, so the Service Map / Node Graph tab renders empty. The same gap
> means Tempo's `span-metrics` processor never lands
> `traces_spanmetrics_*` either (§5.5.1) — every span metric the dashboards use
> comes from the collector's connector instead. Closing it needs both halves:
> uncomment `remote_write` in `tempo.yaml` **and** enable
> `--web.enable-remote-write-receiver` on the Prometheus service (or add a
> scrape job for Tempo).
---

739
OpenTelemetryPlan/06-implementation-phases.md
View File

File diff suppressed because it is too large Load Diff

427
OpenTelemetryPlan/07-observability-backends.md
View File

@@ -17,14 +17,33 @@
### Quick Start with Tempo
```bash
# Start Tempo with OTLP support
# Start Tempo with OTLP support.
# Version pinned to match docker/telemetry/docker-compose.yml:55 — keep the
# two in step, since Tempo config keys change between minor releases.
#
# Only 4317 (OTLP/gRPC) is published: docker/telemetry/tempo.yaml:28-33
# declares a single distributor receiver, `otlp.protocols.grpc` on
# 0.0.0.0:4317. There is no `http` protocol block, so nothing listens on 4318
# and publishing it would give you a port that silently refuses connections.
# 3200 is Tempo's HTTP API/query port (tempo.yaml:17-18), not an ingest port.
docker run -d --name tempo \
-p 3200:3200 \
-p 4317:4317 \
-p 4318:4318 \
grafana/tempo:2.6.1
grafana/tempo:2.9.4
```
> Note that xrpld itself exports OTLP/**HTTP** only (§2.2.1), so it cannot send
> to this container directly — the collector is what bridges HTTP ingest to
> Tempo's gRPC receiver (`otlp/tempo` → `tempo:4317`). A bare Tempo container is
> useful for replaying traces from another OTLP/gRPC producer, not as an xrpld
> endpoint.
> In practice, prefer the full stack —
> `docker compose -f docker/telemetry/docker-compose.yml up -d` — over a bare
> Tempo container. Most shipped dashboards query Prometheus span metrics, which
> need the collector and Prometheus services too. See
> [05 §5.6](./05-configuration-reference.md).
---
## 7.2 Production Backends
@@ -168,47 +187,100 @@ flowchart TB
### 7.4.2 Sampling Strategy
An earlier version of this section described a three-policy tail sampler (keep
all errors / keep anything >5s / keep 10% of the rest). **No such sampler
exists in this repo.** What ships is below.
```mermaid
flowchart LR
subgraph head["Head Sampling (Node)"]
hs[Node-level head sampling<br/>fixed at 100%<br/>not configurable]
subgraph head["Head Sampling (Node) — fixed"]
hs["ParentBased(TraceIdRatio 1.0)<br/>samplingRatio is static constexpr<br/>no config key exists<br/>100% of spans exported"]
end
subgraph tail["Tail Sampling (Collector)"]
ts1[Keep all errors]
ts2[Keep slow >5s]
ts3[Keep 10% rest]
subgraph tail["Tail Sampling (Collector) — opt-in"]
base["Base config:<br/>NO tail_sampling processor<br/>100% of traces stored"]
cloud["grafanacloud overlay only:<br/>one probabilistic policy<br/>sampling_percentage: 0.5"]
end
head --> tail
base --> final["Stored Traces"]
cloud --> final
ts1 --> final[Final Traces]
ts2 --> final
ts3 --> final
style head fill:#0d47a1,stroke:#082f6a,color:#fff
style tail fill:#1b5e20,stroke:#0d3d14,color:#fff
style hs fill:#0d47a1,stroke:#082f6a,color:#fff
style ts1 fill:#1b5e20,stroke:#0d3d14,color:#fff
style ts2 fill:#1b5e20,stroke:#0d3d14,color:#fff
style ts3 fill:#1b5e20,stroke:#0d3d14,color:#fff
style final fill:#bf360c,stroke:#8c2809,color:#fff
style head fill:#0d47a1,stroke:#082f6a,color:#ffffff
style tail fill:#1b5e20,stroke:#0d3d14,color:#ffffff
style hs fill:#0d47a1,stroke:#082f6a,color:#ffffff
style base fill:#1b5e20,stroke:#0d3d14,color:#ffffff
style cloud fill:#1b5e20,stroke:#0d3d14,color:#ffffff
style final fill:#bf360c,stroke:#8c2809,color:#ffffff
```
**Reading the diagram:**
- **Head Sampling (Node)**: xrpld pins head sampling at 100% (sample everything) and does not expose a configurable ratio. This is intentional: a per-node ratio would let different nodes make divergent keep/drop decisions for the same distributed trace, producing broken/partial traces. xrpld uses a `ParentBased` sampler so spans inheriting a remote parent honor the upstream decision. Volume reduction is delegated to the collector's tail sampling.
- **Tail Sampling (Collector)**: The second filter -- the collector inspects completed traces and applies rules: keep all errors, keep anything slower than 5 seconds, and keep 10% of the remainder.
- **Arrow head → tail**: All head-sampled traces flow to the collector, where tail sampling further reduces volume while preserving the most valuable data.
- **Final Traces**: The output after both sampling stages; this is what gets stored and queried. The two-stage approach balances cost with debuggability.
- **Head Sampling (Node)** — fixed at 100% and genuinely not configurable:
`Telemetry.h:234` declares `static constexpr double samplingRatio = 1.0;` and
`TelemetryConfig.cpp:139` records that there is nothing to parse. This is
intentional: a per-node ratio would let different nodes make divergent
keep/drop decisions for the same distributed trace, producing broken/partial
traces. The ratio sampler is wrapped in a `ParentBased` sampler so spans
inheriting a remote parent honour the upstream decision.
- **Tail Sampling (Collector)** — the base config
(`docker/telemetry/otel-collector-config.yaml`) has **no** `tail_sampling`
processor, so the local and CI stacks keep 100% of traces. The only shipped
policy lives in `otel-collector-config.grafanacloud.yaml:60-67`, wired into
the **`traces/store`** pipeline (`:259-261`) — the overlay has no pipeline
named `traces`; it splits the trace stream into `traces/metrics` (unsampled,
feeds `spanmetrics`) and `traces/store` (sampled, feeds Tempo and Grafana
Cloud). See [05 §5.5.2](./05-configuration-reference.md) for the full overlay
delta. The policy is a single `probabilistic` at **0.5%**,
`decision_wait: 10s`, `num_traces: 50000`. There are no error or latency
carve-outs.
- **Why 0.5% does not damage the dashboards**: the policy is applied on the
trace-storage branch only. The `spanmetrics` connector runs on a separate
branch that still sees every span, so `span_calls_total` and
`span_duration_milliseconds_*` remain exact. Sampling costs you individual
example traces in Tempo, not metric accuracy.
- **If you want the error/latency policies**: they are a reasonable thing to
add, but they must be written — and `decision_wait` sized so a trace's spans
have all arrived before the policy evaluates it.
#### Companion guard: `memory_limiter` (recommended, not configured)
Tail sampling bounds what the collector **stores**; it does not bound what the
collector **buffers**. `tail_sampling` is the opposite of cheap here — it holds
up to `num_traces` (50 000) traces in memory for `decision_wait` before
deciding — and the `spanmetrics` connector keeps a live series cache on top of
that. A production gateway collector should therefore also run a
[`memory_limiter`](https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiterprocessor/README.md)
processor as an OOM guard: it applies backpressure (refusing new data with a
retryable error, which the node's `sending_queue` will retry) instead of letting
the process be killed and losing every buffered trace.
> **Not currently configured anywhere in this repo.** Neither
> `otel-collector-config.yaml` nor
> `otel-collector-config.grafanacloud.yaml` declares a `memory_limiter`, and
> neither compose file sets a container memory limit — so today a traffic spike
> is bounded only by host RAM. This is a recommendation for real deployments,
> recorded here because [05 §5.5.1](./05-configuration-reference.md) lists
> `memory_limiter` among the processors deliberately **absent** from the shipped
> config and that must not be read as "not needed". Placement rules if you add
> it: it must be the **first** processor in every pipeline (ahead of `batch`),
> and `limit_mib` must sit below the container/cgroup limit with headroom for
> the sampling and spanmetrics caches.
### 7.4.3 Data Retention
| Environment | Hot Storage | Warm Storage | Cold Archive |
| ----------- | ----------- | ------------ | ------------ |
| Development | 24 hours | N/A | N/A |
| Staging | 7 days | N/A | N/A |
| Production | 7 days | 30 days | many years |
| Environment | Hot Storage | Warm Storage | Cold Archive | Source |
| --------------------------- | ----------- | ------------ | ------------ | ------------------------------------------------------------ |
| Development (local stack) | **1 hour** | N/A | N/A | `tempo.yaml:40` — `compactor.compaction.block_retention: 1h` |
| Staging (recommendation) | 7 days | N/A | N/A | Not configured in this repo |
| Production (recommendation) | 7 days | 30 days | many years | Not configured in this repo |
> **The local stack keeps traces for 1 hour, not 24.** `block_retention: 1h`
> is deliberate — it bounds disk for a long-running dev node — but it means a
> trace you found this morning is gone by lunchtime. Raise
> `block_retention` in `docker/telemetry/tempo.yaml` before starting any
> investigation that needs to span a working day. The staging and production
> rows are recommendations only; nothing in this repo provisions them.
---
@@ -224,56 +296,106 @@ flowchart LR
---
## 7.6 Grafana Dashboard Examples
## 7.6 Grafana Dashboards and Alerts
Pre-built dashboards for xrpld observability.
> **Superseded.** This section was written in Phase 1a, before any dashboard
> shipped, and described three hypothetical boards (`xrpld-consensus-health`,
> `xrpld-node-overview`, `xrpld-unified`) and three TraceQL alert rules in a
> group called `xrpld-tracing-alerts`. **None of those uids or rule names exist
> anywhere in the repo.** What actually ships is 15 dashboards and 13 alert
> rules, and both are Prometheus-first rather than TraceQL-first. The
> authoritative references are:
>
> | For | See |
> | ------------------------------------------------ | --------------------------------------------------------------------------------------------------------- |
> | Dashboard and panel inventory, per-panel queries | [09-data-collection-reference.md](./09-data-collection-reference.md) |
> | Alert catalogue, thresholds and response steps | `docs/telemetry-runbook.md` |
> | Files on disk | `docker/telemetry/grafana/dashboards/*.json`, `docker/telemetry/grafana/provisioning/alerting/rules.yaml` |
>
> The rest of this section records only the facts a reader needs so as not to
> chase the removed names.
### 7.6.1 Consensus Health Dashboard
### 7.6.1 Shipped Dashboards
A Tempo-backed dashboard (uid `xrpld-consensus-health`) with four panels, all driven by TraceQL:
15 JSON dashboards are provisioned into Grafana folder `xrpld`. The uids are
bare — there is no `xrpld-` prefix:
- **Consensus Round Duration** (timeseries, ms): average `consensus.round` span duration per node instance, with yellow/red thresholds at 4s/5s.
- **Phase Duration Breakdown** (barchart): average duration of `consensus.phase.*` spans grouped by span name.
- **Proposers per Round** (stat): average of the `span.proposers` attribute on `consensus.round` spans.
- **Recent Slow Rounds (>5s)** (table): `consensus.round` spans filtered to `duration > 5s`.
`consensus-health`, `fee-market`, `job-queue`, `ledger-data-sync`,
`ledger-operations`, `log-derived-insights`, `network-traffic`, `node-health`,
`overlay-traffic-detail`, `peer-network`, `peer-quality`, `rpc-pathfinding`,
`rpc-performance`, `transaction-overview`, `validator-health`.
Each panel's TraceQL query is described inline in its bullet above.
> **Panel-count convention** (shared with [05 §5.8.3](./05-configuration-reference.md)):
> counts are of **data panels only**. `type: "row"` collapsible headers are
> excluded because a row carries no query, so a board's raw `panels` array is
> longer than its stated count.
### 7.6.2 Node Overview Dashboard
`consensus-health.json` is a useful calibration for how far this section drifted:
where the removed text described "four TraceQL panels", the real board carries **22
data panels** in 4 rows (26 `panels` array entries) — 19 Prometheus targets
against `${DS_PROMETHEUS}` and 9 TraceQL targets against `${DS_TEMPO}`. Tempo is
used for trace _drill-down_; the time series come from span metrics.
A Tempo-backed dashboard (uid `xrpld-node-overview`) with four panels:
### 7.6.2 Shipped Alert Rules
- **Active Nodes** (stat): count of distinct `resource.service.instance.id` values seen for the `xrpld` service.
- **Total Transactions (1h)** (stat): count of `tx.receive` spans.
- **Error Rate** (gauge, percent): ratio of `status.code=error` spans to all spans, with yellow/red thresholds at 1%/5%.
- **Service Map** (nodeGraph): Tempo-generated service dependency graph.
`docker/telemetry/grafana/provisioning/alerting/rules.yaml` provisions **13
rules in 5 groups**, all in folder `xrpld`, all `interval: 1m`, and all
**PromQL** — there are zero TraceQL alert rules.
### 7.6.3 Alert Rules
| Group | Rules |
| ------------------ | --------------------------------------------------------------------------- |
| `xrpld-consensus` | `LedgerHistoryMismatch`, `LedgerCloseStalled`, `ValidatedLedgerStale` |
| `xrpld-validator` | `ValidationsMissed`, `ValidationsNotChecked` |
| `xrpld-jobqueue` | `JobQueueTxOverflow`, `JobQueueLatencyHigh`, `NodeStoreIOLatencyHigh` |
| `xrpld-node-state` | `NodeStateFlapping`, `NodeNotFull` |
| `xrpld-overlay` | `ManifestJobQueueConvoy`, `ManifestFloodInbound`, `PeerResourceDisconnects` |
Grafana provisions three TraceQL-based alert rules (group `xrpld-tracing-alerts`, evaluated every 1m) against the Tempo datasource:
> Two placements are worth noting because they are not what the rule name
> suggests. `ValidatedLedgerStale` is grouped under `xrpld-consensus`, not
> `xrpld-validator` — it fires on any node whose validated-ledger sequence stops
> advancing, which is a chain-progress symptom rather than a validator-identity
> one. `NodeStoreIOLatencyHigh` is grouped under `xrpld-jobqueue`, not
> `xrpld-node-state` — slow NodeStore I/O manifests first as job-queue backlog,
> so grouping it there keeps the cause and its effect in one notification.
- **Consensus Round Slow** (warning, `for: 5m`): fires when average `consensus.round` duration exceeds 5s.
Thresholds, measured baselines and response procedures are in the runbook's
alert catalogue, not here.
```
{resource.service.name="xrpld" && name="consensus.round"} | avg(duration) > 5s
```
### 7.6.3 Writing New Rules: the metric name
- **RPC Error Rate Spike** (critical, `for: 2m`): fires when the error rate across `rpc.command.*` spans exceeds 5%. Error _rate_ is a ratio, so it must divide the error-span rate by the total-span rate — a single TraceQL `rate()` returns spans/second, not a percentage, and would fire on traffic volume alone. This uses span metrics emitted by the collector's `spanmetrics` connector (Prometheus datasource), not a TraceQL query:
If you add a span-metric alert, the metric is **`span_calls_total`**. This stack
sets the `spanmetrics` connector's `namespace: "span"`
(`otel-collector-config.yaml:114`); the connector's own default namespace is
**empty**, so without that setting the names would be the bare `calls_total` /
`duration_milliseconds_*`. 7 of the 15 dashboards already query the `span_`
names. Durations are likewise `span_duration_milliseconds_bucket`.
```
sum(rate(traces_spanmetrics_calls_total{service_name="xrpld", span_name=~"rpc.command.*", status_code="STATUS_CODE_ERROR"}[5m]))
/
sum(rate(traces_spanmetrics_calls_total{service_name="xrpld", span_name=~"rpc.command.*"}[5m]))
> 0.05
```
> **`traces_spanmetrics_*` is a different producer, not the connector's
> default.** That family is emitted by **Tempo's** `metrics_generator`
> `span-metrics` processor (`tempo.yaml:70-76`), which is a separate
> implementation from the collector connector. It does not exist in this stack
> either: the generator's `remote_write` is commented out (`tempo.yaml:53-56`)
> and `prometheus.yml:6-9` scrapes only `otel-collector:8889`, so nothing stores
> what Tempo generates. Do not write a rule against `traces_spanmetrics_*` and
> do not describe `namespace: "span"` as overriding it.
- **Transaction Throughput Drop** (warning, `for: 10m`): fires when the `tx.receive` span rate falls below 10/s.
An RPC error-rate rule, written against the real metric name, looks like this.
Note that error _rate_ is a ratio, so it must divide the error-span rate by the
total-span rate — a bare rate returns calls/second and would fire on traffic
volume alone:
```
{resource.service.name="xrpld" && name="tx.receive"} | rate() < 10
```
```
sum(rate(span_calls_total{service_name="xrpld", span_name=~"rpc.command.*", status_code="STATUS_CODE_ERROR"}[5m]))
/
sum(rate(span_calls_total{service_name="xrpld", span_name=~"rpc.command.*"}[5m]))
> 0.05
```
> **Note**: The Consensus Round Slow and Transaction Throughput Drop rules use TraceQL aggregates (`avg(duration)`, `rate()`), which require Tempo 2.3+ with TraceQL metrics enabled. Verify aggregate query support in your Tempo version before provisioning. The RPC Error Rate Spike rule instead queries Prometheus span metrics (collector `spanmetrics` connector), so it needs that connector enabled in the collector pipeline.
> **Prefer PromQL over TraceQL for alerting.** TraceQL aggregates
> (`avg(duration)`, `rate()`) need Tempo 2.3+ with TraceQL metrics enabled, are
> slower, and are distorted by any tail sampling in the path (§7.4.2). Span
> metrics are computed pre-sampling and cost nothing extra to query. That is
> why all 13 shipped rules are PromQL.
---
@@ -285,81 +407,73 @@ How to correlate OpenTelemetry traces with existing xrpld observability.
### 7.7.1 Correlation Architecture
There is **one** collection agent, not three. Earlier drafts of this diagram
routed logs through "Promtail/Fluentd" and metrics through a "StatsD Exporter";
neither exists in this stack. Logs are read by the OTel Collector's own
`filelog` receiver, and `beast::insight` metrics arrive at the same collector
over OTLP (`[insight] server=otel`). The single-agent shape is the point: one
process, one config file, one place to add redaction or tier tagging.
```mermaid
flowchart TB
subgraph xrpld["xrpld Node"]
otel[OpenTelemetry<br/>Spans]
perflog[PerfLog<br/>JSON Logs]
insight[Beast Insight<br/>StatsD Metrics]
otel["OpenTelemetry Spans"]
journal["Journal debug.log<br/>trace_id= span_id= prefix<br/>(Log.cpp:304-338)"]
insight["Beast Insight + XRPL_METRIC_*<br/>native OTLP metrics"]
end
subgraph collectors["Data Collection"]
otelc[OTel Collector]
promtail[Promtail/Fluentd]
statsd[StatsD Exporter]
end
otelc["OTel Collector<br/>receivers: otlp, filelog<br/>connector: spanmetrics<br/>3 pipelines"]
subgraph storage["Storage"]
tempo[(Tempo)]
loki[(Loki)]
prom[(Prometheus)]
tempo[("Tempo")]
loki[("Loki")]
prom[("Prometheus")]
end
subgraph grafana["Grafana"]
traces[Trace View]
logs[Log View]
metrics[Metrics View]
corr[Correlation<br/>Panel]
end
dashboards["Grafana<br/>Tempo to Loki via tracesToLogs<br/>Loki to Tempo via derived fields"]
otel -->|OTLP| otelc --> tempo
perflog -->|JSON| promtail --> loki
insight -->|StatsD| statsd --> prom
otel -->|"OTLP/HTTP :4318"| otelc
journal -->|"filelog tails<br/>/var/log/xrpld"| otelc
insight -->|"OTLP/HTTP :4318"| otelc
tempo --> traces
loki --> logs
prom --> metrics
otelc -->|"otlp/tempo"| tempo
otelc -->|"otlphttp/loki"| loki
otelc -->|"prometheus :8889"| prom
traces --> corr
logs --> corr
metrics --> corr
tempo --> dashboards
loki --> dashboards
prom --> dashboards
style xrpld fill:#0d47a1,stroke:#082f6a,color:#fff
style collectors fill:#bf360c,stroke:#8c2809,color:#fff
style storage fill:#1b5e20,stroke:#0d3d14,color:#fff
style grafana fill:#4a148c,stroke:#2e0d57,color:#fff
style otel fill:#0d47a1,stroke:#082f6a,color:#fff
style perflog fill:#0d47a1,stroke:#082f6a,color:#fff
style insight fill:#0d47a1,stroke:#082f6a,color:#fff
style otelc fill:#bf360c,stroke:#8c2809,color:#fff
style promtail fill:#bf360c,stroke:#8c2809,color:#fff
style statsd fill:#bf360c,stroke:#8c2809,color:#fff
style tempo fill:#1b5e20,stroke:#0d3d14,color:#fff
style loki fill:#1b5e20,stroke:#0d3d14,color:#fff
style prom fill:#1b5e20,stroke:#0d3d14,color:#fff
style traces fill:#4a148c,stroke:#2e0d57,color:#fff
style logs fill:#4a148c,stroke:#2e0d57,color:#fff
style metrics fill:#4a148c,stroke:#2e0d57,color:#fff
style corr fill:#4a148c,stroke:#2e0d57,color:#fff
style xrpld fill:#0d47a1,stroke:#082f6a,color:#ffffff
style storage fill:#1b5e20,stroke:#0d3d14,color:#ffffff
style otel fill:#0d47a1,stroke:#082f6a,color:#ffffff
style journal fill:#0d47a1,stroke:#082f6a,color:#ffffff
style insight fill:#0d47a1,stroke:#082f6a,color:#ffffff
style otelc fill:#bf360c,stroke:#8c2809,color:#ffffff
style tempo fill:#1b5e20,stroke:#0d3d14,color:#ffffff
style loki fill:#1b5e20,stroke:#0d3d14,color:#ffffff
style prom fill:#1b5e20,stroke:#0d3d14,color:#ffffff
style dashboards fill:#4a148c,stroke:#2e0d57,color:#ffffff
```
**Reading the diagram:**
- **xrpld Node (three sources)**: A single node emits three independent data streams -- OpenTelemetry spans, PerfLog JSON logs, and Beast Insight StatsD metrics.
- **Data Collection layer**: Each stream has its own collector -- OTel Collector for spans, Promtail/Fluentd for logs, and a StatsD exporter for metrics. They operate independently.
- **Storage layer (Tempo, Loki, Prometheus)**: Each data type lands in a purpose-built store optimized for its query patterns (trace search, log grep, metric aggregation).
- **Grafana Correlation Panel**: The key integration point -- Grafana queries all three stores and links them via shared fields (`trace_id`, `tx_hash`, `ledger_seq`), enabling a single-pane debugging experience.
- **xrpld Node (three signals, one transport)**: spans and metrics both leave over OTLP/HTTP on port 4318. Logs do not leave the node at all — the node just writes `debug.log`, and the journal sink prefixes `trace_id=`/`span_id=` whenever a span is active (`Log.cpp:304-338`).
- **OTel Collector (single agent)**: an `otlp` receiver takes spans and metrics; a `filelog` receiver tails `/var/log/xrpld/*/debug.log` and regex-parses the trace/span IDs out of each line. A `spanmetrics` connector derives RED metrics from the trace stream and feeds them into the metrics pipeline. Three pipelines, three exporters — see [05 §5.5.1](./05-configuration-reference.md).
- **PerfLog is not in this picture.** It still writes `perf.log`, but nothing collects it and it carries no trace ID; the `setTraceId` hook once planned for it was never built ([02 §2.6.5](./02-design-decisions.md)).
- **StatsD is not in this picture either.** It remains a supported `[insight] server=` choice, but selecting it takes metrics _out_ of this pipeline and requires a StatsD receiver you would have to add yourself — the compose file's StatsD port mapping is commented out.
- **Grafana**: correlation is bidirectional and configured in the datasources, not in a bespoke panel — Tempo's `tracesToLogs` (`filterByTraceID: true`) jumps trace → logs, and `loki.yaml`'s derived fields jump log → trace.
### 7.7.2 Correlation Fields
| Source | Field | Link To | Purpose |
| ----------- | ------------------- | ------------- | -------------------------- |
| **Trace** | `trace_id` | Logs | Find log entries for trace |
| **Trace** | `tx_hash` | Logs, Metrics | Find TX-related data |
| **Trace** | `ledger_seq` | Logs | Find ledger-related logs |
| **PerfLog** | `trace_id` (new) | Traces | Jump to trace from log |
| **PerfLog** | `ledger_seq` | Traces | Find consensus trace |
| **Insight** | `exemplar.trace_id` | Traces | Jump from metric spike |
| Source | Field | Link To | Status |
| --------------- | --------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Trace** | `trace_id` | Logs | **Live.** Tempo `tracesToLogs`, `filterByTraceID: true` |
| **Trace** | `tx_hash` | — | Live as a span attribute for search; **not** used as a cross-signal join key (`tags: []`) |
| **Trace** | `ledger_seq` | — | Live as a span attribute; not a join key |
| **Journal log** | `trace_id`, `span_id` | Traces | **Live.** Emitted by `Log.cpp:304-338` into `debug.log`, parsed by the collector's `filelog` receiver, jumped via `loki.yaml` derived fields |
| **PerfLog** | `trace_id` | Traces | **Not implemented.** PerfLog output has no trace ID; the planned `setTraceId` hook was never built. Use the journal log instead |
| **Insight** | `exemplar.trace_id` | Traces | **Not implemented.** No exemplar configuration exists anywhere in the code or collector config — no `exemplar_filter` on the SDK side, no `exemplarTraceIdDestinations` on the Prometheus datasource. Metric spike → trace jumps must be done by time range today |
### 7.7.3 Example: Debugging a Slow Transaction
@@ -376,31 +490,82 @@ flowchart TB
Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736
```
**Step 3: Find related PerfLog entries**
**Step 3: Find related log lines**
```
# In Grafana Explore with Loki
{job="xrpld"} |= "4bf92f3577b34da6a3ce929d0e0e4736"
# In Grafana Explore with Loki. `service_name` is the promoted stream label;
# do NOT use {job="xrpld"} — see the note below.
{service_name="xrpld"} |= "4bf92f3577b34da6a3ce929d0e0e4736"
```
**Step 4: Check Insight metrics for the time window**
These are journal (`debug.log`) lines, not PerfLog lines — see §7.7.2.
> **Known issue — `{job="xrpld"}` does not select anything.** The collector's
> `resource/logs` processor does upsert a `job=xrpld` resource attribute
> (`otel-collector-config.yaml:62-70`), explicitly so that operators could paste
> `{job="xrpld"}`. Loki does not cooperate: on OTLP ingest it promotes only an
> **allow-listed** set of resource attributes to indexed stream labels
> (`service.name`, `service.namespace`, `service.instance.id`,
> `deployment.environment`, `k8s.*`, `cloud.*`), and `job` is not on it. This
> repo mounts no Loki config override (`docker-compose.yml:75` uses the image's
> built-in `local-config.yaml`), so `job` lands in **structured metadata** —
> queryable only with a `|` filter after a selector, never as the selector
> itself. A `{job="xrpld"}` query returns empty with no error, which is why this
> is easy to miss. `docs/telemetry-runbook.md:2533` says the same, and all 38
> Loki queries in the shipped dashboards (35 panel targets + 3 template
> variables) select on `service_name` — zero use `job`. Fix options:
> drop the ineffective `job` upsert, or mount a Loki config adding `job` to
> `distributor.otlp_config.resource_attributes`.
**Step 4: Check metrics for the time window**
```
# In Grafana with Prometheus
rate(xrpld_tx_applied_total[1m])
@ timestamp_from_trace
# In Grafana with Prometheus. Span-derived RED metrics for the transaction
# pipeline (namespace "span" — see 7.6.3):
sum(rate(span_calls_total{span_name="tx.process"}[1m])) by (service_instance_id)
# Error share of the same pipeline. Note !~"tesSUCCESS|" — NOT
# !="tesSUCCESS" — so spans that carry no ter_result are excluded:
sum(rate(span_calls_total{span_name="tx.process", ter_result!~"tesSUCCESS|"}[5m]))
/
sum(rate(span_calls_total{span_name="tx.process"}[5m]))
```
### 7.7.4 Unified Dashboard Example
> **Why the regex form.** An absent Prometheus label is indistinguishable from
> the empty string, and `tx.process` can end **without** a `ter_result`: the span
> is opened at `NetworkOPs.cpp:1416`, but `processTransaction()` returns early
> when `preProcessTransaction()` rejects the transaction (`:1437-1438`), and
> `doTransactionAsync()` returns early when the transaction is already applying
> (`:1461-1462`) — both before the only setter, at `:1674`. Those series arrive
> with `ter_result=""`, which `!="tesSUCCESS"` happily counts as a failure and
> inflates the ratio. `!~"tesSUCCESS|"` excludes the empty value via the trailing
> `|` alternative. This is the form `docs/telemetry-runbook.md:1198` and the
> `transaction-overview.json` stage-failure panels already use; apply it to any
> new `ter_result` predicate.
A single dashboard (uid `xrpld-unified`) that ties traces, metrics, and logs together across the Tempo, Prometheus, and Loki datasources:
> Earlier drafts used `rate(xrpld_tx_applied_total[1m])` and
> `rate(xrpld_tx_received_total[5m])`. **Neither metric exists** — there is no
> `xrpld_`-prefixed metric family at all, because `OTelCollector::formatName()`
> deliberately prepends no prefix (`OTelCollector.cpp:855-866`); the OTel
> resource `service.name` identifies the service instead. Use the `span_*`
> families above (verified in `transaction-overview.json` and
> `rpc-performance.json`) or the native `XRPL_METRIC_*` instrument names listed
> in [09-data-collection-reference.md](./09-data-collection-reference.md).
- **Transaction Latency (Traces)** (timeseries, Tempo): `histogram_over_time(duration)` of `tx.receive` spans.
- **Transaction Rate (Metrics)** (timeseries, Prometheus): `rate(xrpld_tx_received_total[5m])` per instance, with a data link that opens the matching `tx.receive` traces in Tempo.
- **Recent Logs** (logs, Loki): `{job="xrpld"} | json`.
- **Trace Search** (table, Tempo): all `xrpld` traces, with per-row data links on `traceID` that jump to the trace in Tempo and to the correlated logs in Loki (`{job="xrpld"} |= "<traceID>"`).
### 7.7.4 Unified Dashboard
The cross-datasource data links are what make this a single-pane debugging view; the correlation fields they rely on are listed in section 7.7.2.
> **Superseded.** No `xrpld-unified` dashboard exists. The single-pane view it
> described is instead delivered by two things that did ship: the
> **`log-derived-insights`** dashboard (31 data panels in 10 rows, all
> Loki-backed — 41 `panels` array entries; see the counting convention in
> §7.6.1) plus the
> bidirectional datasource links (Tempo `tracesToLogs` → Loki, `loki.yaml`
> derived fields → Tempo), which let you cross signals from _any_ board rather
> than only from one dedicated dashboard.
>
> The correlation fields those links rely on — and which of them are actually
> implemented — are in §7.7.2. For the full board inventory see
> [09-data-collection-reference.md](./09-data-collection-reference.md).
---

83
OpenTelemetryPlan/08-appendix.md
View File

@@ -133,18 +133,25 @@ The full span inventory (names, attributes, parents as instrumented) is in
### Task Lists
| Document | Description |
| -------------------------------------------------------------------------- | --------------------------------------------------- |
| [Phase2_taskList.md](./Phase2_taskList.md) | RPC layer trace instrumentation |
| [Phase3_taskList.md](./Phase3_taskList.md) | Peer overlay & consensus tracing |
| [Phase4_taskList.md](./Phase4_taskList.md) | Transaction lifecycle tracing |
| [Phase5_taskList.md](./Phase5_taskList.md) | Ledger processing & advanced tracing |
| [Phase5_IntegrationTest_taskList.md](./Phase5_IntegrationTest_taskList.md) | Observability stack integration tests |
| [Phase7_taskList.md](./Phase7_taskList.md) | Native OTel metrics migration |
| [Phase8_taskList.md](./Phase8_taskList.md) | Log-trace correlation |
| [Phase9_taskList.md](./Phase9_taskList.md) | Internal metric instrumentation gap fill (future) |
| [Phase10_taskList.md](./Phase10_taskList.md) | Synthetic workload generation & validation (future) |
| [Phase11_taskList.md](./Phase11_taskList.md) | Third-party data collection pipelines (future) |
| Document | Description |
| -------------------------------------------------------------------------- | ---------------------------------------------- |
| [Phase2_taskList.md](./Phase2_taskList.md) | RPC layer trace instrumentation |
| [Phase3_taskList.md](./Phase3_taskList.md) | Peer overlay & consensus tracing |
| [Phase4_taskList.md](./Phase4_taskList.md) | Transaction lifecycle tracing |
| [Phase5_taskList.md](./Phase5_taskList.md) | Ledger processing & advanced tracing |
| [Phase5_IntegrationTest_taskList.md](./Phase5_IntegrationTest_taskList.md) | Observability stack integration tests |
| [Phase7_taskList.md](./Phase7_taskList.md) | Native OTel metrics migration |
| [Phase8_taskList.md](./Phase8_taskList.md) | Log-trace correlation |
| [Phase9_taskList.md](./Phase9_taskList.md) | Internal metric instrumentation gap fill |
| [Phase10_taskList.md](./Phase10_taskList.md) | Synthetic workload generation & validation |
| [Phase11_taskList.md](./Phase11_taskList.md) | Third-party data collection pipelines (future) |
> **Only Phase 11 is still "future".** Phase 9 ships on
> `pratik/otel-phase9-metric-gap-fill` (18 task entries, 9.1–9.17 plus 9.7a) and
> Phase 10 on `pratik/otel-phase10-workload-validation` (7 tasks). Their task
> lists are present on every branch from those points forward, so a reader on a
> later branch sees plans that are already implemented, not proposals. Phase 11
> (13 tasks) has no implementation branch.
> **Note**: Phases 1 and 6 do not have separate task list files. Phase 1 tasks are documented in [06-implementation-phases.md §6.2](./06-implementation-phases.md). Phase 6 tasks are documented in [06-implementation-phases.md §6.7](./06-implementation-phases.md).
@@ -156,13 +163,21 @@ This guide maps Phase 9–11 content to its location across the documentation.
### Phase 9: Internal Metric Instrumentation Gap Fill
| Content | Location |
| ------------------------------- | ------------------------------------------------------------------------ |
| Plan & architecture | [06-implementation-phases.md §6.8.2](./06-implementation-phases.md) |
| Task list (10 tasks) | [Phase9_taskList.md](./Phase9_taskList.md) |
| Future metric definitions (~50) | [09-data-collection-reference.md §5b](./09-data-collection-reference.md) |
| New class: `MetricsRegistry` | `src/xrpld/telemetry/MetricsRegistry.h/.cpp` (planned) |
| New dashboards | `fee-market`, `job-queue` (planned) |
| Content | Location |
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| Plan & architecture | [06-implementation-phases.md §6.8.2](./06-implementation-phases.md) |
| Task list (18 entries, 9.1–9.17) | [Phase9_taskList.md](./Phase9_taskList.md) |
| Metric definitions | [09-data-collection-reference.md §5b](./09-data-collection-reference.md) |
| New class: `MetricsRegistry` | `src/xrpld/telemetry/MetricsRegistry.h/.cpp` — **shipped** |
| New dashboards (4) | `fee-market`, `job-queue`, `peer-quality`, `validator-health` — **shipped** |
| Updated dashboards (2) | `node-health`, `rpc-performance` |
| Provisioned alert rules | `docker/telemetry/grafana/provisioning/alerting/rules.yaml` — 13 rules in 5 groups ([07 §7.6.2](./07-observability-backends.md)) |
> **Task numbering**: `Phase9_taskList.md` carries 18 `## Task 9.x` headings —
> 9.1 through 9.17 plus the inserted 9.7a (`push_metrics.py` parity). The "10
> tasks" figure in earlier revisions predates 9.7a and 9.11–9.17. Tasks 9.8 and
> 9.11–9.13 together produce the four new dashboards; Task 9.17 (peer span
> coverage) is explicitly **deferred to Phase 11**.
**Metric categories**: NodeStore I/O, Cache Hit Rates, TxQ, PerfLog Per-RPC, PerfLog Per-Job, Counted Objects, Fee Escalation & Load Factors.
@@ -172,23 +187,45 @@ This guide maps Phase 9–11 content to its location across the documentation.
| -------------------- | ------------------------------------------------------------------------ |
| Plan & architecture | [06-implementation-phases.md §6.8.3](./06-implementation-phases.md) |
| Task list (7 tasks) | [Phase10_taskList.md](./Phase10_taskList.md) |
| Branch | `pratik/otel-phase10-workload-validation` |
| Validation inventory | [09-data-collection-reference.md §5c](./09-data-collection-reference.md) |
| Test harness | `docker/telemetry/docker-compose.workload.yaml` (planned) |
| CI workflow | `.github/workflows/telemetry-validation.yml` (planned) |
| Test harness | `docker/telemetry/docker-compose.workload.yaml` (phase-10 branch) |
| CI workflow | `.github/workflows/telemetry-validation.yml` (phase-10 branch) |
**Validates**: 16 spans, 22 attributes, 300+ metrics, 10 dashboards, log-trace correlation.
**Validates** (Phase-10 harness inventory): **40** span types, **67** unique
required span attributes, **36** metric entries, **14** dashboards, log-trace
correlation.
> **These are the harness manifests' counts, and two of them lag the code.** The
> manifests (`docker/telemetry/workload/expected_spans.json`,
> `expected_metrics.json`) live only on the phase-10 branch. `expected_spans.json`
> holds 40 span entries against the **41** span-name families the code emits
> (`rpc.ws_upgrade` has no entry), and its own `total_unique_attributes: 58` field
> is stale against the 67 attributes its per-span `required_attributes` lists
> actually name. `expected_metrics.json` asserts 14 dashboard uids against the
> **15** dashboard JSONs in `docker/telemetry/grafana/dashboards/`;
> `log-derived-insights` is the unasserted one. The full emitted inventory is in
> [09-data-collection-reference.md §1.1](./09-data-collection-reference.md#11-complete-span-inventory-41-spans)
> and [§5c](./09-data-collection-reference.md#validated-telemetry-inventory).
### Phase 11: Third-Party Data Collection Pipelines
| Content | Location |
| --------------------------------- | ------------------------------------------------------------------------ |
| Plan & architecture | [06-implementation-phases.md §6.8.4](./06-implementation-phases.md) |
| Task list (11 tasks) | [Phase11_taskList.md](./Phase11_taskList.md) |
| Task list (13 tasks) | [Phase11_taskList.md](./Phase11_taskList.md) |
| External metric definitions (~30) | [09-data-collection-reference.md §5d](./09-data-collection-reference.md) |
| Custom OTel Collector receiver | `docker/telemetry/otel-rippled-receiver/` (planned) |
| Prometheus alerting rules (11) | [09-data-collection-reference.md §5d](./09-data-collection-reference.md) |
| New dashboards (4) | Validator Health, Network Topology, Fee Market (External), DEX & AMM |
> **Two of those names now collide with shipped Phase-9 boards.** Phase 9
> already ships `validator-health` and `fee-market`, both built from the node's
> **own** telemetry. The Phase-11 entries are the third-party-data variants
> (network-wide validator agreement, external fee/DEX feeds via the custom
> receiver). They need distinct uids, or they will overwrite the Phase-9 boards
> on provisioning.
**Consumer categories**: Exchanges, Payment Processors, DeFi/AMM, NFT Marketplaces, Analytics Providers, Wallets, Compliance, Academic Researchers, Institutional Custody, CBDC Bridge Operators.
---

641
OpenTelemetryPlan/09-data-collection-reference.md
View File

@@ -2,7 +2,7 @@
> **Audience**: Developers and operators. This is the single source of truth for all telemetry data collected by xrpld's observability stack.
>
> **Related docs**: [docs/telemetry-runbook.md](../docs/telemetry-runbook.md) (operator runbook with alerting and troubleshooting) | [03-implementation-strategy.md](./03-implementation-strategy.md) (code structure and performance optimization) | [04-code-samples.md](./04-code-samples.md) (C++ instrumentation examples)
> **Related docs**: [docs/telemetry-runbook.md](../docs/telemetry-runbook.md) (operator runbook with alerting and troubleshooting) | [03-implementation-strategy.md](./03-implementation-strategy.md) (code structure and performance optimization) | [docs/telemetry-runbook.md § Protocol Span Flow](../docs/telemetry-runbook.md#protocol-span-flow) (authoritative span-flow reference; replaces the deleted `04-code-samples.md`)
## Data Flow Overview
@@ -33,7 +33,7 @@ graph LR
end
subgraph viz["Visualization"]
F["Grafana :3000<br/>13 dashboards"]
F["Grafana :3000<br/>15 dashboards"]
end
A -->|"OTLP/HTTP :4318<br/>(traces + attributes)"| R1
@@ -72,9 +72,13 @@ There are three independent telemetry pipelines entering a single **OTel Collect
A third, narrower metrics path exists for instruments created at their call site through the
`XRPL_METRIC_*` macros. These use the OTel Metrics SDK directly and reach the collector's OTLP
receiver rather than the StatsD receiver, so their names carry no `xrpld_` prefix. See
[§2a](#2a-call-site-otel-metrics-metricsregistry). Code in `libxrpl` cannot use these macros and
always goes through `beast::insight` instead.
receiver rather than the StatsD receiver, so their names carry no `xrpld_` prefix. The seven
call-site instruments are documented with the families they belong to:
`rpc_in_flight_requests` in
[§Per-RPC Method Metrics](#per-rpc-method-metrics-synchronous-countershistogram), the five
`getobject_*` in [§GetObject Request Path](#getobject-request-path-synchronous-countershistograms),
and `ledgers_closed_total` in [§Synchronous Counters (Phase 7+)](#synchronous-counters-phase-7).
Code in `libxrpl` cannot use these macros and always goes through `beast::insight` instead.
**Trace backend** — The collector exports traces via OTLP/gRPC to:
@@ -86,13 +90,22 @@ always goes through `beast::insight` instead.
## 1. OpenTelemetry Spans
### 1.1 Complete Span Inventory (~37 spans)
### 1.1 Complete Span Inventory (41 spans)
> **See also**: [02-design-decisions.md §2.3](./02-design-decisions.md#23-span-naming-conventions) for naming conventions and the full span catalog with rationale. [04-code-samples.md §4.6](./04-code-samples.md#46-span-flow-visualization) for span flow diagrams.
> **41 emitted span-name families.** The count is derived from the `*SpanNames.h`
> headers and their call sites, one family per distinct span name
> (`rpc.command.<name>` and `grpc.<MethodName>` each count once, since the
> command / method name is a parameter of a single family). The tables below list
> all 41: RPC 5, gRPC 1, transaction 6, TxQ 6, consensus 13, ledger 4, peer 2,
> pathfind 4. The Phase-10 validation harness
> (`docker/telemetry/workload/expected_spans.json`) catalogues **40** of them —
> `rpc.ws_upgrade` has no entry.
> **See also**: [02-design-decisions.md §2.3](./02-design-decisions.md#23-span-naming-conventions) for naming conventions and the full span catalog with rationale. [docs/telemetry-runbook.md § Protocol Span Flow](../docs/telemetry-runbook.md#protocol-span-flow) for the span flow diagrams (the former `04-code-samples.md` §4.6 was deleted).
> **Span names vs. attribute keys**: span names use dotted `subsystem.operation`
> form (e.g. `rpc.http_request`). Span _attribute_ keys use the bare/underscore
> form from the 2026-05-13 naming redesign (e.g. `tx_hash`, not `xrpl.tx.hash`).
> form from the 2026-05-13 naming redesign (e.g. `tx_hash`, not `xrpl.tx.hash`). <!-- otel-naming:allow-dotted: xrpl.tx.hash -->
> The dotted `xrpl.*` form is reserved for OTel **resource** attributes set once
> at startup. See §1.2 for the full attribute inventory.
@@ -190,26 +203,35 @@ Controlled by `trace_transactions=1` in `[telemetry]` config.
Controlled by `trace_consensus=1` in `[telemetry]` config.
| Span Name | Parent | Source File | Description |
| ------------------------------ | ------------------ | ---------------- | ------------------------------------------------------------------- |
| `consensus.round` | — (root) | RCLConsensus.cpp | Root span for one consensus round (deterministic trace per round) |
| `consensus.phase.open` | `consensus.round` | Consensus.h | Open phase — collecting transactions before close |
| `consensus.proposal.send` | `consensus.round` | RCLConsensus.cpp | Node broadcasts its transaction set proposal |
| `consensus.ledger_close` | `consensus.round` | RCLConsensus.cpp | Ledger close event triggered by consensus |
| `consensus.establish` | `consensus.round` | Consensus.h | Establish phase — converging on the transaction set |
| `consensus.update_positions` | `consensus.round` | Consensus.h | Position update with per-dispute vote details |
| `consensus.check` | `consensus.round` | Consensus.h | Consensus threshold check (agree/disagree tally) |
| `consensus.accept` | `consensus.round` | RCLConsensus.cpp | Consensus accepts a ledger (round complete) |
| `consensus.accept.apply` | `consensus.accept` | RCLConsensus.cpp | Ledger application with close-time details (jtACCEPT thread) |
| `consensus.validation.send` | `consensus.round` | RCLConsensus.cpp | Validation message sent after ledger accepted (follows-from link) |
| `consensus.mode_change` | `consensus.round` | RCLConsensus.cpp | Operating-mode transition during the round |
| `consensus.proposal.receive` | (context) | PeerImp.cpp | Proposal received from a peer (context-propagated into the round) |
| `consensus.validation.receive` | (context) | PeerImp.cpp | Validation received from a peer (context-propagated into the round) |
| Span Name | Parent | Source File | Description |
| ------------------------------ | --------------------- | ---------------- | ------------------------------------------------------------------- |
| `consensus.round` | — (root) | RCLConsensus.cpp | Root span for one consensus round (deterministic trace per round) |
| `consensus.phase.open` | `consensus.round` | Consensus.h | Open phase — collecting transactions before close |
| `consensus.proposal.send` | `consensus.round` | RCLConsensus.cpp | Node broadcasts its transaction set proposal |
| `consensus.ledger_close` | `consensus.round` | RCLConsensus.cpp | Ledger close event triggered by consensus |
| `consensus.establish` | `consensus.round` | Consensus.h | Establish phase — converging on the transaction set |
| `consensus.update_positions` | `consensus.establish` | Consensus.h | Position update with per-dispute vote details |
| `consensus.check` | `consensus.establish` | Consensus.h | Consensus threshold check (agree/disagree tally) |
| `consensus.accept` | `consensus.round` | RCLConsensus.cpp | Consensus accepts a ledger (round complete) |
| `consensus.accept.apply` | `consensus.accept` | RCLConsensus.cpp | Ledger application with close-time details (jtACCEPT thread) |
| `consensus.validation.send` | `consensus.round` | RCLConsensus.cpp | Validation message sent after ledger accepted (follows-from link) |
| `consensus.mode_change` | `consensus.round` | RCLConsensus.cpp | Operating-mode transition during the round |
| `consensus.proposal.receive` | (context) | PeerImp.cpp | Proposal received from a peer (context-propagated into the round) |
| `consensus.validation.receive` | (context) | PeerImp.cpp | Validation received from a peer (context-propagated into the round) |
The `.receive` spans are created per-message in the overlay and joined to the
round trace via context propagation rather than direct parenting. The
`consensus.validation.send` span uses a follows-from link off the round.
> **`update_positions` and `check` sit one level below `establish`, not below
> the round.** Both are created with
> `SpanGuard::childSpan(..., establishSpanContext_)`
> (`include/xrpl/consensus/Consensus.h:1628` and `:1837`), and
> `consensus.establish` is itself parented to `roundSpanContext_`
> (`Consensus.h:2099-2101`). An earlier revision of this table showed them as
> direct children of `consensus.round`; queries or trace-shape assertions built
> on that tree are wrong by one level.
**Where to find**: Tempo → TraceQL: `{resource.service.name="xrpld" && name=~"consensus.*"}`
**Grafana dashboard**: _Consensus Health_ (`consensus-health`)
@@ -251,12 +273,12 @@ under an unrelated transaction's trace.
Controlled by `trace_rpc=1` in `[telemetry]` config.
| Span Name | Parent | Source File | Description |
| --------------------- | -------------------- | --------------- | ---------------------------------------------------------- |
| `pathfind.request` | `rpc.command.<name>` | PathFind.cpp | `path_find` RPC entry (`doPathFind`) |
| `pathfind.compute` | `pathfind.request` | PathRequest.cpp | Path computation for one request (`PathRequest::doUpdate`) |
| `pathfind.discover` | `pathfind.compute` | Pathfinder.cpp | Graph exploration (one per RPC call) |
| `pathfind.update_all` | — | PathRequest.cpp | Async recomputation of all active requests at ledger close |
| Span Name | Parent | Source File | Description |
| --------------------- | -------------------- | -------------------------------------- | ---------------------------------------------------------- |
| `pathfind.request` | `rpc.command.<name>` | PathFind.cpp:27, RipplePathFind.cpp:36 | `path_find` / `ripple_path_find` RPC entry |
| `pathfind.compute` | `pathfind.request` | PathRequest.cpp:750 | Path computation for one request (`PathRequest::doUpdate`) |
| `pathfind.discover` | `pathfind.compute` | PathRequest.cpp:599-600 | Graph exploration (one per RPC call) |
| `pathfind.update_all` | — | PathRequestManager.cpp:88-92 | Async recomputation of all active requests at ledger close |
> **Note**: `pathfind.request` nests under the active `rpc.command.<name>` span.
> Because OTel context storage is coroutine-aware (backed by `LocalValue`), the
@@ -367,50 +389,78 @@ Join a transaction's work to its ledger with `{span.current_ledger_seq=<N>}`.
#### Consensus Attributes
| Attribute | Type | Set On | Description |
| -------------------------- | ------- | -------------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
| `consensus_ledger_id` | string | `consensus.round` | Previous-ledger id anchoring the round |
| `ledger_seq` | int64 | `consensus.round`, `consensus.ledger_close`, `consensus.accept.apply`, `consensus.validation.send` | Ledger sequence number |
| `consensus_mode` | string | `consensus.round`, `consensus.ledger_close` | Node mode: `"Proposing"`, `"Observing"`, `"Wrong"`, etc. |
| `consensus_round_id` | int64 | `consensus.round` | Round identifier |
| `consensus_phase` | string | `consensus.round` | Current phase name (updated on each transition) |
| `trace_strategy` | string | `consensus.round` | Trace-id strategy (`deterministic` / `random`) |
| `previous_ledger_seq` | int64 | `consensus.round` | Sequence of the previous ledger |
| `previous_proposers` | int64 | `consensus.round` | Proposer count in the previous round |
| `previous_round_time_ms` | int64 | `consensus.round` | Duration of the previous round |
| `consensus_round` | int64 | `consensus.proposal.send` | Proposal sequence number for the broadcast proposal |
| `is_bow_out` | boolean | `consensus.proposal.send` | Whether the proposal is a bow-out (resigning the round) |
| `tx_count_open` | int64 | `consensus.ledger_close` | Transactions in the open ledger at close |
| `close_time_resolution_ms` | int64 | `consensus.ledger_close` | Close-time rounding granularity |
| `converge_percent` | int64 | `consensus.establish`, `consensus.update_positions` | Convergence percentage |
| `establish_count` | int64 | `consensus.establish` | Establish-phase iteration count |
| `proposers` | int64 | `consensus.establish`, `consensus.update_positions`, `consensus.accept` | Number of proposers |
| `disputes_count` | int64 | `consensus.establish`, `consensus.update_positions` | Number of disputed transactions |
| `tx_id` | string | `consensus.update_positions` | Disputed transaction id (per-dispute event) |
| `dispute_our_vote` | boolean | `consensus.update_positions` | Our vote on the disputed tx |
| `dispute_yays` | int64 | `consensus.update_positions` | Yes votes on the disputed tx |
| `dispute_nays` | int64 | `consensus.update_positions` | No votes on the disputed tx |
| `agree_count` | int64 | `consensus.check` | Agreeing proposer count |
| `disagree_count` | int64 | `consensus.check` | Disagreeing proposer count |
| `threshold_percent` | int64 | `consensus.check` | Agreement threshold percentage |
| `consensus_result` | string | `consensus.check` | Check outcome |
| `quorum` | int64 | `consensus.check`, `consensus.accept` | Quorum required |
| `round_time_ms` | int64 | `consensus.accept`, `consensus.accept.apply` | Total consensus round duration in milliseconds |
| `consensus_state` | string | `consensus.accept.apply` | Consensus outcome: `"finished"` or `"moved_on"` |
| `close_time` | int64 | `consensus.accept.apply` | Agreed-upon ledger close time (epoch seconds) |
| `close_time_correct` | boolean | `consensus.accept.apply` | Whether validators agreed on close time |
| `close_resolution_ms` | int64 | `consensus.accept.apply` | Close-time rounding granularity in milliseconds |
| `proposing` | boolean | `consensus.accept.apply`, `consensus.validation.send` | Whether this node was a proposer |
| `parent_close_time` | int64 | `consensus.accept.apply` | Parent ledger close time |
| `close_time_self` | int64 | `consensus.accept.apply` | This node's close-time vote |
| `close_time_vote_bins` | string | `consensus.accept.apply` | Distribution of close-time votes |
| `resolution_direction` | string | `consensus.accept.apply` | Whether close resolution increased/decreased/unchanged |
| `tx_count` | int64 | `consensus.accept.apply` | Transactions in the accepted set |
| `ledger_hash` | string | `consensus.validation.send` | Full hash of the validated ledger (shared with peer) |
| `full_validation` | boolean | `consensus.validation.send` | Whether this is a full validation |
| `validation_sign_time` | int64 | `consensus.validation.send` | Validation signing time |
| `mode_old` | string | `consensus.mode_change` | Operating mode before the transition |
| `mode_new` | string | `consensus.mode_change` | Operating mode after the transition |
| Attribute | Type | Set On | Description |
| --------------------------- | ------- | -------------------------------------------------------------------------------------------------- | -------------------------------------------------------- |
| `consensus_ledger_id` | string | `consensus.round` | Previous-ledger id anchoring the round |
| `ledger_seq` | int64 | `consensus.round`, `consensus.ledger_close`, `consensus.accept.apply`, `consensus.validation.send` | Ledger sequence number |
| `consensus_mode` | string | `consensus.round`, `consensus.ledger_close` | Node mode: `"Proposing"`, `"Observing"`, `"Wrong"`, etc. |
| `consensus_round_id` | int64 | `consensus.round` | Round identifier |
| `consensus_phase` | string | `consensus.round` | Current phase name (updated on each transition) |
| `trace_strategy` | string | `consensus.round` | Trace-id strategy (`deterministic` / `attribute`) |
| `previous_ledger_seq` | int64 | `consensus.round` | Sequence of the previous ledger |
| `previous_proposers` | int64 | `consensus.round` | Proposer count in the previous round |
| `previous_round_time_ms` | int64 | `consensus.round` | Duration of the previous round |
| `consensus_round` | int64 | `consensus.proposal.send` | Proposal sequence number for the broadcast proposal |
| `is_bow_out` | boolean | `consensus.proposal.send` | Whether the proposal is a bow-out (resigning the round) |
| `tx_count_open` | int64 | `consensus.ledger_close` | Transactions in the open ledger at close |
| `close_time_resolution_ms` | int64 | `consensus.ledger_close` | Close-time rounding granularity |
| `converge_percent` | int64 | `consensus.establish`, `consensus.update_positions`, `consensus.check` | Convergence percentage |
| `establish_count` | int64 | `consensus.establish`, `consensus.check` | Establish-phase iteration count |
| `proposers` | int64 | `consensus.establish`, `consensus.update_positions`, `consensus.accept` | Number of proposers |
| `disputes_count` | int64 | `consensus.establish`, `consensus.update_positions` | Number of disputed transactions |
| `tx_id` | string | `consensus.update_positions` | Disputed transaction id (per-dispute event) |
| `dispute_our_vote` | boolean | `consensus.update_positions` | Our vote on the disputed tx |
| `dispute_yays` | int64 | `consensus.update_positions` | Yes votes on the disputed tx |
| `dispute_nays` | int64 | `consensus.update_positions` | No votes on the disputed tx |
| `avalanche_threshold` | int64 | `consensus.update_positions` | Escalated weight needed to change our vote |
| `close_time_threshold` | int64 | `consensus.update_positions` | Close-time agreement threshold percentage |
| `agree_count` | int64 | `consensus.check` | Agreeing proposer count |
| `disagree_count` | int64 | `consensus.check` | Disagreeing proposer count |
| `threshold_percent` | int64 | `consensus.check` | Agreement threshold percentage |
| `have_close_time_consensus` | boolean | `consensus.update_positions`, `consensus.check` | Whether the close time reached consensus |
| `proposers_finished` | int64 | `consensus.check` | Proposers that have already validated the next ledger |
| `consensus_stalled` | boolean | `consensus.check` | Whether `checkConsensus` reported a stall |
| `consensus_result` | string | `consensus.check` | Check outcome |
| `quorum` | int64 | `consensus.accept` | Quorum required |
| `round_time_ms` | int64 | `consensus.accept`, `consensus.accept.apply` | Total consensus round duration in milliseconds |
| `consensus_state` | string | `consensus.accept.apply` | Consensus outcome: `"finished"` or `"moved_on"` |
| `close_time` | int64 | `consensus.accept.apply` | Agreed-upon ledger close time (epoch seconds) |
| `close_time_correct` | boolean | `consensus.accept.apply` | Whether validators agreed on close time |
| `close_resolution_ms` | int64 | `consensus.accept.apply` | Close-time rounding granularity in milliseconds |
| `proposing` | boolean | `consensus.accept.apply`, `consensus.validation.send` | Whether this node was a proposer |
| `parent_close_time` | int64 | `consensus.accept.apply` | Parent ledger close time |
| `close_time_self` | int64 | `consensus.accept.apply` | This node's close-time vote |
| `close_time_vote_bins` | string | `consensus.accept.apply` | Distribution of close-time votes |
| `resolution_direction` | string | `consensus.accept.apply` | Whether close resolution increased/decreased/unchanged |
| `tx_count` | int64 | `consensus.accept.apply` | Transactions in the accepted set |
| `ledger_hash` | string | `consensus.validation.send` | Full hash of the validated ledger (shared with peer) |
| `full_validation` | boolean | `consensus.validation.send` | Whether this is a full validation |
| `validation_sign_time` | int64 | `consensus.validation.send` | Validation signing time |
| `mode_old` | string | `consensus.mode_change` | Operating mode before the transition |
| `mode_new` | string | `consensus.mode_change` | Operating mode after the transition |
> **`quorum` is on `consensus.accept` only.** Its single set site is
> `RCLConsensus::Adaptor::makeAcceptSpan()`
> (`src/xrpld/app/consensus/RCLConsensus.cpp:516`). `consensus.check`
> (`include/xrpl/consensus/Consensus.h:1899-1926`) never sets it, so
> `{name="consensus.check" && span.quorum>0}` matches nothing.
> **`consensus.check` carries nine attributes, all set before the early
> returns.** `Consensus<Adaptor>::haveConsensus()` sets them at
> `include/xrpl/consensus/Consensus.h:1899-1912` and `consensus_result` at
> `:1925`, deliberately ahead of the `No` / `Expired` branches, so the span is
> fully populated even on rounds that never reach consensus. In set order:
> `agree_count`, `disagree_count`, `converge_percent`,
> `have_close_time_consensus`, `threshold_percent`, `proposers_finished`,
> `consensus_stalled`, `establish_count`, `consensus_result`.
>
> Three of these are shared with sibling spans and were previously scoped too
> narrowly in the table above: `converge_percent` and `establish_count` are set on
> `consensus.check` as well as `consensus.establish` /
> `consensus.update_positions`, and `have_close_time_consensus` is set on both
> `consensus.update_positions` (`Consensus.h:1779`) and `consensus.check`
> (`:1903`). `close_time_threshold` (`:1781`) and `avalanche_threshold` (`:1730`)
> stay `consensus.update_positions`-only.
**Tempo query**: `{span.consensus_mode="Proposing"}` to find rounds where the node was proposing.
@@ -551,25 +601,44 @@ prefix=xrpld
### 2.1 Gauges
| Prometheus Metric | Source File | Description | Typical Range |
| ------------------------------------------- | --------------------- | ----------------------------------------- | ------------------------------- |
| `ledgermaster_validated_ledger_age` | LedgerMaster.h | Seconds since last validated ledger | 0–10 (healthy), >30 (stale) |
| `ledgermaster_published_ledger_age` | LedgerMaster.h | Seconds since last published ledger | 0–10 (healthy) |
| `state_accounting_disconnected_duration` | NetworkOPs.cpp | Cumulative seconds in Disconnected state | Monotonic |
| `state_accounting_connected_duration` | NetworkOPs.cpp | Cumulative seconds in Connected state | Monotonic |
| `state_accounting_syncing_duration` | NetworkOPs.cpp | Cumulative seconds in Syncing state | Monotonic |
| `state_accounting_tracking_duration` | NetworkOPs.cpp | Cumulative seconds in Tracking state | Monotonic |
| `state_accounting_full_duration` | NetworkOPs.cpp | Cumulative seconds in Full state | Monotonic (should dominate) |
| `state_accounting_disconnected_transitions` | NetworkOPs.cpp | Count of transitions to Disconnected | Low |
| `state_accounting_connected_transitions` | NetworkOPs.cpp | Count of transitions to Connected | Low |
| `state_accounting_syncing_transitions` | NetworkOPs.cpp | Count of transitions to Syncing | Low |
| `state_accounting_tracking_transitions` | NetworkOPs.cpp | Count of transitions to Tracking | Low |
| `state_accounting_full_transitions` | NetworkOPs.cpp | Count of transitions to Full | Low (should be 1 after startup) |
| `peer_finder_active_inbound_peers` | PeerfinderManager.cpp | Active inbound peer connections | 0–85 |
| `peer_finder_active_outbound_peers` | PeerfinderManager.cpp | Active outbound peer connections | 10–21 |
| `overlay_peer_disconnects` | OverlayImpl.cpp | Cumulative peer disconnection count | Low growth |
| `overlay_peer_disconnects_charges` | OverlayImpl.cpp | Disconnects due to resource limit charges | Low growth (subset of above) |
| `jobq_job_count` | JobQueue.cpp | Current job queue depth (group `jobq`) | 0–100 (healthy) |
| Prometheus Metric | Source File | Description | Typical Range |
| ------------------------------------------- | --------------------- | ------------------------------------------------- | ------------------------------- |
| `ledgermaster_validated_ledger_age` | LedgerMaster.h | Seconds since last validated ledger | 0–10 (healthy), >30 (stale) |
| `ledgermaster_published_ledger_age` | LedgerMaster.h | Seconds since last published ledger | 0–10 (healthy) |
| `state_accounting_disconnected_duration` | NetworkOPs.cpp | Cumulative **microseconds** in Disconnected state | Monotonic |
| `state_accounting_connected_duration` | NetworkOPs.cpp | Cumulative **microseconds** in Connected state | Monotonic |
| `state_accounting_syncing_duration` | NetworkOPs.cpp | Cumulative **microseconds** in Syncing state | Monotonic |
| `state_accounting_tracking_duration` | NetworkOPs.cpp | Cumulative **microseconds** in Tracking state | Monotonic |
| `state_accounting_full_duration` | NetworkOPs.cpp | Cumulative **microseconds** in Full state | Monotonic (should dominate) |
| `state_accounting_disconnected_transitions` | NetworkOPs.cpp | Count of transitions to Disconnected | Low |
| `state_accounting_connected_transitions` | NetworkOPs.cpp | Count of transitions to Connected | Low |
| `state_accounting_syncing_transitions` | NetworkOPs.cpp | Count of transitions to Syncing | Low |
| `state_accounting_tracking_transitions` | NetworkOPs.cpp | Count of transitions to Tracking | Low |
| `state_accounting_full_transitions` | NetworkOPs.cpp | Count of transitions to Full | Low (should be 1 after startup) |
| `peer_finder_active_inbound_peers` | PeerfinderManager.cpp | Active inbound peer connections | 0–85 |
| `peer_finder_active_outbound_peers` | PeerfinderManager.cpp | Active outbound peer connections | 10–21 |
| `overlay_peer_disconnects` | OverlayImpl.cpp | Cumulative peer disconnection count | Low growth |
| `jobq_job_count` | JobQueue.cpp | Current job queue depth (group `jobq`) | 0–100 (healthy) |
> **`state_accounting_*_duration` is microseconds, not seconds.**
> `NetworkOPsImp::collectMetrics()` does
> `duration_cast<std::chrono::microseconds>(...)` and publishes `.count()`
> (`src/xrpld/app/misc/NetworkOPs.cpp:4884-4897`). Divide by `1e6` for seconds.
> The `node-health` "State Duration Rate (All States)" panel already does
> (`/ 1000000` on each `rate(...)`), and
> `docker/telemetry/grafana/dashboards/validate_dashboards.py:43` lints the
> family as "cumulative µs". Reading the raw value as seconds overstates time
> in state by a factor of one million.
> **`overlay_peer_disconnects_charges` was never implemented: NOT IMPLEMENTED.**
> No instrument of that name exists anywhere in `src/`, `include/` or `docker/`.
> The resource-charge disconnect count is exported from the OTel
> `MetricsRegistry` instead, as
> `server_info{metric="peer_disconnects_resources"}` — see
> [§Server Info](#server-info-via-otel-metricsregistry). Use that selector;
> the previously documented `overlay_peer_disconnects_charges` matches nothing.
> `06-implementation-phases.md` still names the old metric in its Phase 6/7
> task text and panel table.
**Grafana dashboard**: _Node Health_ (`node-health`)
@@ -758,13 +827,14 @@ for how the tier attributes are set and reach metrics.
1. Open Grafana at **http://localhost:3000**
2. Navigate to **Dashboards → xrpld** folder
3. All 10 dashboards are auto-provisioned from `docker/telemetry/grafana/dashboards/`
3. All 15 dashboards are auto-provisioned from `docker/telemetry/grafana/dashboards/`
(the Phase-10 harness asserts 14 of them — `log-derived-insights` is unasserted)
---
## 4. Tempo Trace Search Guide
> **See also**: [08-appendix.md](./08-appendix.md) §8.2 for span hierarchy visualizations. [05-configuration-reference.md](./05-configuration-reference.md) §5.8.5 for TraceQL query examples.
> **See also**: [08-appendix.md](./08-appendix.md) §8.2 for span hierarchy visualizations. [05-configuration-reference.md](./05-configuration-reference.md) §5.8.4 for TraceQL query examples.
### Finding Traces by Type
@@ -796,16 +866,16 @@ A consensus round groups its lifecycle spans under a single root
(`consensus.round`); the build/ledger spans run as their own trees:
```
consensus.round (root — one per round)
├── consensus.phase.open (open phase)
├── consensus.proposal.send (broadcast proposal)
├── consensus.ledger_close (close event)
├── consensus.establish (establish phase)
├── consensus.update_positions (position updates)
├── consensus.check (threshold check)
├── consensus.accept (accept result)
│ └── consensus.accept.apply (apply, jtACCEPT thread)
└── consensus.validation.send (send validation, follows-from link)
consensus.round (root — one per round)
├── consensus.phase.open (open phase)
├── consensus.proposal.send (broadcast proposal)
├── consensus.ledger_close (close event)
├── consensus.establish (establish phase)
│ ├── consensus.update_positions (position updates)
│ └── consensus.check (threshold check)
├── consensus.accept (accept result)
│ └── consensus.accept.apply (apply, jtACCEPT thread)
└── consensus.validation.send (send validation, follows-from link)
ledger.build (build new ledger)
└── tx.apply (apply transaction set)
@@ -817,7 +887,7 @@ ledger.store (persist to DB)
## 5. Prometheus Query Examples
> **See also**: [05-configuration-reference.md](./05-configuration-reference.md) §5.8.7 for correlating Prometheus system metrics with trace-derived metrics.
> **See also**: [05-configuration-reference.md](./05-configuration-reference.md) §5.8.6 for correlating Prometheus system metrics with trace-derived metrics.
### Span-Derived Metrics
@@ -916,18 +986,26 @@ Grafana Loki (v3.7.6) serves as the log storage backend. It receives log entries
### LogQL Query Examples
The stream selector is `{service_name="xrpld"}`, **not** `{job="xrpld"}`. Loki's
OTLP ingestion promotes only a small set of resource attributes to stream labels
(`service_name`, `service_instance_id`, `deployment_environment`); everything else
— including the `job` attribute the collector sets — lands in structured
metadata and must be filtered with `|` after the selector. A `{job="xrpld"}`
selector returns zero rows and no error. All shipped queries and the
`log-derived-insights` dashboard use the `service_name` form.
```logql
# Find all logs for a specific trace
{job="xrpld"} |= "trace_id=abc123def456789012345678abcdef01"
{service_name="xrpld"} |= "trace_id=abc123def456789012345678abcdef01"
# Error logs with trace context
{job="xrpld"} |= "ERR" |= "trace_id="
{service_name="xrpld"} |= "ERR" |= "trace_id="
# Logs from a specific partition with trace context
{job="xrpld"} |= "LedgerMaster" | regexp `trace_id=(?P<trace_id>[a-f0-9]+)` | trace_id != ""
{service_name="xrpld"} | partition = `LedgerMaster` | trace_id != ""
# Count traced log lines over time
count_over_time({job="xrpld"} |= "trace_id=" [5m])
count_over_time({service_name="xrpld"} |= "trace_id=" [5m])
```
---
@@ -944,10 +1022,17 @@ async callbacks for new categories.
> **Authoritative metric names live in [§ Phase 9: OTel SDK-Exported Metrics](#phase-9-otel-sdk-exported-metrics-metricsregistry) below.**
> Most internal metrics are emitted as **labeled** gauges — one instrument carrying many logical
> values via a `metric` label (e.g. `cache_metrics{metric="sle_hit_rate"}`,
> values via a `metric` label (e.g. `cache_metrics{metric="SLE_hit_rate"}`,
> `txq_metrics{metric="txq_count"}`, `load_factor_metrics{metric="load_factor"}`,
> `nodestore_state{metric="node_reads_total"}`) — not the flat per-name form. Query the
> labeled names; the flat names (`cache_sle_hit_rate`, `txq_count`, …) are **not** emitted.
> labeled names; the flat names (`cache_SLE_hit_rate`, `txq_count`, …) are **not** emitted.
>
> **Label values are case-sensitive and three cache values are not lowercase.**
> The `metric` label carries the string literal passed to `Observe()`, verbatim:
> `SLE_hit_rate`, `AL_hit_rate` and `AL_size` are upper-case
> (`src/xrpld/telemetry/MetricsRegistry.cpp:649`, `:665`, `:691`), while
> `ledger_hit_rate` genuinely is lowercase (`:658`). A selector written as
> `cache_metrics{metric="sle_hit_rate"}` matches nothing.
#### Server Info (via OTel MetricsRegistry)
@@ -989,16 +1074,33 @@ async callbacks for new categories.
| Prometheus Metric | Type | Labels | Description |
| --------------------------------- | ----- | -------- | ------------------------- |
| `cache_metrics{metric="al_size"}` | Gauge | `metric` | AcceptedLedger cache size |
| `cache_metrics{metric="AL_size"}` | Gauge | `metric` | AcceptedLedger cache size |
#### Extended NodeStore Metrics (additions to existing nodestore_state)
| Prometheus Metric | Type | Labels | Description |
| -------------------------------------------------- | ----- | -------- | ----------------------------------- |
| `nodestore_state{metric="node_reads_duration_us"}` | Gauge | `metric` | Cumulative read time (microseconds) |
| `nodestore_state{metric="read_request_bundle"}` | Gauge | `metric` | Read request bundle count |
| `nodestore_state{metric="read_threads_running"}` | Gauge | `metric` | Active read threads |
| `nodestore_state{metric="read_threads_total"}` | Gauge | `metric` | Total read threads configured |
| Prometheus Metric | Type | Labels | Description |
| --------------------------------------------------- | ----- | -------- | ------------------------------------ |
| `nodestore_state{metric="node_reads_duration_us"}` | Gauge | `metric` | Cumulative read time (microseconds) |
| `nodestore_state{metric="node_writes_duration_us"}` | Gauge | `metric` | Cumulative write time (microseconds) |
| `nodestore_state{metric="read_request_bundle"}` | Gauge | `metric` | Read request bundle count |
| `nodestore_state{metric="read_threads_running"}` | Gauge | `metric` | Active read threads |
| `nodestore_state{metric="read_threads_total"}` | Gauge | `metric` | Total read threads configured |
> **The cumulative duration pair truncates to whole microseconds.** Both values
> are accumulated in nanoseconds internally and divided on read —
> `getFetchDurationUs()` returns `fetchDurationNs_ / 1000` and
> `getStoreDurationUs()` returns `storeDurationNs_ / 1000`
> (`include/xrpl/nodestore/Database.h:232-254`). The exported unit is
> microseconds and every doc, metric and dashboard agrees on that — this is
> **not** a unit mismatch. The consequence is only at the low end: a handful of
> sub-microsecond reads on a warm store can leave the gauge reading `0` until
> their nanosecond total passes 1000. Read a flat `0` on a low-traffic node as
> "not yet a microsecond of I/O", not as "no I/O".
>
> `node_writes_duration_us` is covered by
> `validate_dashboards.py`'s `NODESTORE_CUMULATIVE` tuple, so the raw-counter
> lint would catch a misuse, but it has **no dashboard panel** yet — an open
> follow-up, unlike its `node_reads_duration_us` sibling.
#### Job Queue and GetObject Additions
@@ -1019,7 +1121,12 @@ repeated here:
write-serialized stall from a cold-read stall. See
[Sync Diagnosis Signals](#sync-diagnosis-signals-observable-gauge--nodestore_state).
### New Grafana Dashboards (Phase 9)
### New Grafana Dashboards for the Phase 9 Gap-Fill Metrics
These two boards were created specifically to surface the gap-fill metrics above.
For the full Phase-9 dashboard delivery record, including the boards added to the
Phase-7 parity set, see
[New Grafana Dashboards (Phase 9)](#new-grafana-dashboards-phase-9).
| Dashboard | UID | Data Source | Key Panels |
| ------------------ | ------------ | ----------- | ----------------------------------------------------------------- |
@@ -1048,16 +1155,35 @@ Phase 10 builds a 5-node validator docker-compose harness with RPC load generato
| Category | Expected Count | Validation Method |
| ------------------------- | ------------------- | -------------------------------- |
| Trace spans | 16 | Jaeger/Tempo API query |
| Span attributes | 22 | Per-span attribute assertion |
| Trace spans | 40 of 41 emitted | Jaeger/Tempo API query |
| Span attributes | 67 required | Per-span attribute assertion |
| Legacy `*` families | ~270 (≈224 traffic) | Prometheus `__name__` query |
| Native MetricsRegistry | 35 instruments | Prometheus query |
| Call-site `XRPL_METRIC_*` | 7 instruments | Prometheus query |
| Per-job-type gauges | 105 (35 types × 3) | Prometheus `__name__` query |
| SpanMetrics RED | 4 per span | Prometheus query |
| Grafana dashboards | 10 | Dashboard API "no data" check |
| Grafana dashboards | 14 of 15 on disk | Dashboard API "no data" check |
| Log-trace links | Present | Loki query + Tempo reverse check |
> **These are the harness's numbers, not the code's, and three of them differ.**
> `docker/telemetry/workload/expected_spans.json` carries 40 span entries against
> the **41** families the code emits ([§1.1](#11-complete-span-inventory-41-spans)) —
> `rpc.ws_upgrade` has no entry — and 67 distinct required attributes (the
> manifest's own `total_unique_attributes: 58` field is stale).
> `expected_metrics.json` asserts 36 metric entries and 14 dashboard uids against
> the **15** dashboard JSONs in `docker/telemetry/grafana/dashboards/`;
> `log-derived-insights` is the unasserted one. The 35 native instruments match
> the tables in
> [§Phase 9: OTel SDK-Exported Metrics](#phase-9-otel-sdk-exported-metrics-metricsregistry)
> and the Phase 7+ section exactly, counting each labeled gauge family
> (`nodestore_state`, `cache_metrics`, …) once.
>
> Note that `ledgers_closed_total` appears in **both** instrument rows: it is
> created as a `MetricsRegistry` member (`MetricsRegistry.cpp:369-370`, whose
> `incrementLedgersClosed()` has no callers) and separately incremented at its
> call site via `XRPL_METRIC_COUNTER_INC` (`RCLConsensus.cpp:749`). The distinct
> name count across the two rows is therefore 41, not 42.
The two added rows are the families that do not originate as `MetricsRegistry`
members. **Call-site** instruments are declared by the `XRPL_METRIC_*` macros
(7 distinct names: `rpc_in_flight_requests`, `ledgers_closed_total`, and the
@@ -1185,7 +1311,7 @@ Further label values on the same instrument, added to separate the two
bottlenecks that both present as the `ledgerData` job lane pinned at its
concurrency cap. Observed in `MetricsRegistry::observeNodeStoreTotals()`,
`observeWritePathDetail()`, and `observeAcquireStats()`
(`src/xrpld/telemetry/MetricsRegistry.cpp:805-877`).
(`src/xrpld/telemetry/MetricsRegistry.cpp:854-925`).
| Prometheus Metric | Type | Labels | Description |
| ---------------------------------------------------- | ----- | -------- | ------------------------------------------------------- |
@@ -1254,9 +1380,9 @@ data as uninformative unless the build is known to include the fix.
| Prometheus Metric | Type | Labels | Description |
| --------------------------------------------- | ----- | -------- | ----------------------------- |
| `cache_metrics{metric="sle_hit_rate"}` | Gauge | `metric` | SLE cache hit rate (0.0-1.0) |
| `cache_metrics{metric="SLE_hit_rate"}` | Gauge | `metric` | SLE cache hit rate (0.0-1.0) |
| `cache_metrics{metric="ledger_hit_rate"}` | Gauge | `metric` | Ledger cache hit rate |
| `cache_metrics{metric="al_hit_rate"}` | Gauge | `metric` | AcceptedLedger cache hit rate |
| `cache_metrics{metric="AL_hit_rate"}` | Gauge | `metric` | AcceptedLedger cache hit rate |
| `cache_metrics{metric="treenode_cache_size"}` | Gauge | `metric` | SHAMap TreeNode cache entries |
| `cache_metrics{metric="treenode_track_size"}` | Gauge | `metric` | Tracked tree nodes |
| `cache_metrics{metric="fullbelow_size"}` | Gauge | `metric` | FullBelow cache entries |
@@ -1274,6 +1400,73 @@ data as uninformative unless the build is known to include the fix.
| `txq_metrics{metric="txq_med_fee_level"}` | Gauge | `metric` | Median fee level in queue |
| `txq_metrics{metric="txq_open_ledger_fee_level"}` | Gauge | `metric` | Open ledger fee escalation level |
#### TxQ Admission and Ledger Mismatch (Synchronous Counters)
Three monotonic counters created alongside the Phase 7+ parity counters
(`src/xrpld/telemetry/MetricsRegistry.cpp:377-382`). The gauges above answer
"how deep is the queue"; these answer "what did the queue refuse, and did the
ledger we built match the one the network validated".
| Prometheus Metric | Type | Labels | Description | Increment Site |
| ------------------------------- | ------- | ----------------- | -------------------------------------------------- | --------------------- |
| `txq_dropped_total` | Counter | `reason="<name>"` | Transactions refused admission to the queue | TxQ.cpp:1302,1347 |
| `txq_expired_total` | Counter | (none) | Transactions abandoned out of the queue on expiry | TxQ.cpp:1428 |
| `ledger_history_mismatch_total` | Counter | `reason="<name>"` | Built-vs-validated ledger hash mismatches, by kind | LedgerHistory.cpp:332 |
Label domains, as emitted:
| Label | Values |
| --------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| `txq_dropped_total{reason}` | `queue_full` |
| `ledger_history_mismatch_total{reason}` | `prior_ledger`, `close_time`, `consensus_txset`, `different_txset`, `same_txset_diff_result`, `unknown` |
**Grafana dashboards**: _Fee Market & TxQ_ (`fee-market`) — "Queue Admission
Rejections (Dropped)", "Queue Abandonment Rate (Expired)"; _Consensus Health_
(`consensus-health`) — "Ledger History Mismatch Rate by Reason"; _Node Health_
(`node-health`) — "Ledger History Mismatches".
> **Known issue — `ledger_history_mismatch_total` has two producers, so a bare
> `sum()` double-counts.** `LedgerHistory::handleMismatch()` increments **both**
> a `beast::insight` counter registered as `ledger.history` / `mismatch`
> (`src/xrpld/app/ledger/LedgerHistory.cpp:323`, created at `:41`) **and** the
> OTel counter above (`:331-332`). The insight counter carries **no** `reason`
> label, and the Prometheus exporter appends `_total` to both, so the two land in
> one metric family: per-node series carrying a `reason` label, plus per-node
> series with `reason` absent that already total all of them. The dual-producer
> mechanism is verifiable from the code above; the exact series count in any given
> stack depends on how many nodes report and how many distinct reasons they have
> hit, so do not treat a fixed number as an invariant.
>
> Consequence: `sum(rate(ledger_history_mismatch_total[5m]))` counts every
> mismatch twice. Always group or filter by `reason`:
> `sum by (reason) (rate(ledger_history_mismatch_total{reason!=""}[5m]))` for the
> per-reason breakdown, or `reason=""` for the untyped total alone. This is a
> **code** defect, not a documentation one — the fix is to retire one producer;
> until then the shipped panels avoid the trap (`consensus-health` groups
> `by (reason)`, `node-health` plots the series unaggregated), and any new panel
> or alert must do the same.
#### Reduce-Relay Efficiency (Observable Gauge — `reduce_relay_metrics`)
Transaction reduce-relay effectiveness, read from `Overlay::txMetrics()` each
collection cycle (`src/xrpld/telemetry/MetricsRegistry.cpp:1353-1385`). A high
`suppressed_peers` : `selected_peers` ratio proves the feature is saving
bandwidth; a high `not_enabled_peers` means stale peers are forcing full relay.
| Prometheus Metric | Type | Labels | Description |
| -------------------------------------------------- | ----- | -------- | ------------------------------------------------------- |
| `reduce_relay_metrics{metric="selected_peers"}` | Gauge | `metric` | Peers selected to receive a relayed transaction |
| `reduce_relay_metrics{metric="suppressed_peers"}` | Gauge | `metric` | Peer sends suppressed by reduce-relay |
| `reduce_relay_metrics{metric="not_enabled_peers"}` | Gauge | `metric` | Peers without reduce-relay support, so relayed in full |
| `reduce_relay_metrics{metric="missing_tx_freq"}` | Gauge | `metric` | Frequency of transactions this node had to request back |
Each source field is a decimal **string** in the `txMetrics()` JSON, parsed with
`std::stoll`; a field that is absent or unparseable is skipped rather than
reported as zero, so absent is not zero here either.
**Grafana dashboard**: _Peer Network_ (`peer-network`) — "Reduce-Relay Peer
Selection", "Reduce-Relay Missing-Tx Frequency".
#### Per-RPC Method Metrics (Synchronous Counters/Histogram)
| Prometheus Metric | Type | Labels | Description |
@@ -1412,17 +1605,43 @@ spelling would silently drop the override.
#### Counted Object Instances (Observable Gauge — `object_count`)
| Prometheus Metric | Type | Labels | Description |
| -------------------------------------- | ----- | --------------- | ------------------------------ |
| `object_count{type="transaction"}` | Gauge | `type="<name>"` | Live Transaction objects |
| `object_count{type="ledger"}` | Gauge | `type="<name>"` | Live Ledger objects |
| `object_count{type="nodeobject"}` | Gauge | `type="<name>"` | Live NodeObject instances |
| `object_count{type="sttx"}` | Gauge | `type="<name>"` | Serialized transaction objects |
| `object_count{type="stledgerentry"}` | Gauge | `type="<name>"` | Serialized ledger entries |
| `object_count{type="inboundledger"}` | Gauge | `type="<name>"` | Ledgers being fetched |
| `object_count{type="pathfinder"}` | Gauge | `type="<name>"` | Active pathfinding operations |
| `object_count{type="pathrequest"}` | Gauge | `type="<name>"` | Active path requests |
| `object_count{type="hashrouterentry"}` | Gauge | `type="<name>"` | Hash router entries |
**The `type` label value is the demangled, fully-qualified C++ type name.** It is
not a lowercase word and not a friendly alias. The value is
`beast::typeName<Object>()` (`include/xrpl/basics/CountedObject.h:115`), which
demangles `typeid(T).name()` with `abi::__cxa_demangle`
(`include/xrpl/beast/type_name.h:16-45`) and applies no stripping; the observer
copies it through verbatim (`src/xrpld/telemetry/MetricsRegistry.cpp:764-770`).
Values therefore keep their `xrpl::` namespace, nested `::`, and template
arguments.
| Prometheus Metric | Type | Labels | Description |
| ---------------------------------------------- | ----- | --------------- | ------------------------------ |
| `object_count{type="xrpl::Transaction"}` | Gauge | `type="<name>"` | Live Transaction objects |
| `object_count{type="xrpl::Ledger"}` | Gauge | `type="<name>"` | Live Ledger objects |
| `object_count{type="xrpl::NodeObject"}` | Gauge | `type="<name>"` | Live NodeObject instances |
| `object_count{type="xrpl::STTx"}` | Gauge | `type="<name>"` | Serialized transaction objects |
| `object_count{type="xrpl::STLedgerEntry"}` | Gauge | `type="<name>"` | Serialized ledger entries |
| `object_count{type="xrpl::InboundLedger"}` | Gauge | `type="<name>"` | Ledgers being fetched |
| `object_count{type="xrpl::Pathfinder"}` | Gauge | `type="<name>"` | Active pathfinding operations |
| `object_count{type="xrpl::PathRequest"}` | Gauge | `type="<name>"` | Active path requests |
| `object_count{type="xrpl::HashRouter::Entry"}` | Gauge | `type="<name>"` | Hash router entries |
The list above is the subset most often queried, not the whole label domain. The
series set is whatever `CountedObject<T>` subclasses have been instantiated, so
it also includes `xrpl::SHAMapItem`, `xrpl::SHAMapInnerNode`,
`xrpl::AcceptedLedger`, `xrpl::Job`, template instantiations such as
`xrpl::STBitString<256>` and `xrpl::STInteger<unsigned int>`, and a few types
outside the `xrpl` namespace such as `CachedView::hit`. Enumerate it rather than
guess:
```promql
# Every type currently reporting on one node
count by (type) (object_count{service_instance_id=~"$node"})
```
Grafana's `$type` template variable on _Node Health_ is populated the same way
(`label_values(object_count, type)`), which is why that dashboard needs no
hardcoded list.
#### Load Factor Breakdown (Observable Gauge — `load_factor_metrics`)
@@ -1515,30 +1734,51 @@ These metrics fill gaps identified by comparing xrpld's internal observability w
| -------------------------------------------------- | ------ | -------- | --------------------------------------- |
| `validation_agreement{metric="agreement_pct_1h"}` | Double | `metric` | Rolling 1h agreement percentage (0-100) |
| `validation_agreement{metric="agreement_pct_24h"}` | Double | `metric` | Rolling 24h agreement percentage |
| `validation_agreement{metric="agreements_1h"}` | Int64 | `metric` | Agreed validations in 1h window |
| `validation_agreement{metric="missed_1h"}` | Int64 | `metric` | Missed validations in 1h window |
| `validation_agreement{metric="agreements_24h"}` | Int64 | `metric` | Agreed validations in 24h window |
| `validation_agreement{metric="missed_24h"}` | Int64 | `metric` | Missed validations in 24h window |
| `validation_agreement{metric="agreement_pct_7d"}` | Double | `metric` | Rolling 7-day agreement percentage |
| `validation_agreement{metric="agreements_1h"}` | Double | `metric` | Agreed validations in 1h window |
| `validation_agreement{metric="missed_1h"}` | Double | `metric` | Missed validations in 1h window |
| `validation_agreement{metric="agreements_24h"}` | Double | `metric` | Agreed validations in 24h window |
| `validation_agreement{metric="missed_24h"}` | Double | `metric` | Missed validations in 24h window |
| `validation_agreement{metric="agreements_7d"}` | Double | `metric` | Agreed validations in the 7-day window |
| `validation_agreement{metric="missed_7d"}` | Double | `metric` | Missed validations in the 7-day window |
Data source: `ValidationTracker` class with 8s grace period and 5m late repair window.
> **Every value on this instrument is a double.** The family is one
> `CreateDoubleObservableGauge` (`src/xrpld/telemetry/MetricsRegistry.cpp:1576`),
> so the integral counts are cast to `double` before `Observe()` — there is no
> Int64 sub-series to filter on. The same holds for `validator_health`,
> `peer_quality` and `state_tracking` below; an earlier revision of these four
> tables split the Type column between Int64 and Double, which the code does not
> do.
>
> The 7-day window is `ValidationTracker::kWindow7d` = 168 hours
> (`src/xrpld/telemetry/ValidationTracker.h:311`) and is observed alongside the 1h
> and 24h windows at `MetricsRegistry.cpp:1606-1609`. Panels exist on _Validator
> Health_ (`validator-health`): "Agreement % (7d)" and "Agreements vs Missed
> (7d)".
#### Validator Health (Observable Gauge — `validator_health`)
| Prometheus Metric | Type | Labels | Description |
| ---------------------------------------------- | ------ | -------- | ------------------------------ |
| `validator_health{metric="amendment_blocked"}` | Int64 | `metric` | 1 if amendment-blocked, else 0 |
| `validator_health{metric="unl_blocked"}` | Int64 | `metric` | 1 if UNL-blocked, else 0 |
| `validator_health{metric="amendment_blocked"}` | Double | `metric` | 1 if amendment-blocked, else 0 |
| `validator_health{metric="unl_blocked"}` | Double | `metric` | 1 if UNL-blocked, else 0 |
| `validator_health{metric="unl_expiry_days"}` | Double | `metric` | Days until UNL list expires |
| `validator_health{metric="validation_quorum"}` | Int64 | `metric` | Validation quorum threshold |
| `validator_health{metric="validation_quorum"}` | Double | `metric` | Validation quorum threshold |
Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1200`.
#### Peer Quality (Observable Gauge — `peer_quality`)
| Prometheus Metric | Type | Labels | Description |
| ------------------------------------------------- | ------ | -------- | ------------------------------------ |
| `peer_quality{metric="peer_latency_p90_ms"}` | Double | `metric` | P90 peer latency in milliseconds |
| `peer_quality{metric="peers_insane_count"}` | Int64 | `metric` | Peers with diverged tracking status |
| `peer_quality{metric="peers_insane_count"}` | Double | `metric` | Peers with diverged tracking status |
| `peer_quality{metric="peers_higher_version_pct"}` | Double | `metric` | % of peers on newer xrpld version |
| `peer_quality{metric="upgrade_recommended"}` | Int64 | `metric` | 1 if >60% of peers are newer version |
| `peer_quality{metric="upgrade_recommended"}` | Double | `metric` | 1 if >60% of peers are newer version |
Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1249`.
#### Ledger Economy (Observable Gauge — `ledger_economy`)
@@ -1554,10 +1794,12 @@ Data source: `ValidationTracker` class with 8s grace period and 5m late repair w
| Prometheus Metric | Type | Labels | Description |
| -------------------------------------------------------- | ------ | -------- | -------------------------------------- |
| `state_tracking{metric="state_value"}` | Int64 | `metric` | Numeric state 0-6 (see encoding below) |
| `state_tracking{metric="state_value"}` | Double | `metric` | Numeric state 0-6 (see encoding below) |
| `state_tracking{metric="time_in_current_state_seconds"}` | Double | `metric` | Duration in current state |
State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full, 5=validating (FULL + validating), 6=proposing (FULL + proposing).
Single `CreateDoubleObservableGauge` at `MetricsRegistry.cpp:1466`.
State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full, 5=validating (FULL + validating), 6=proposing (FULL + proposing). Values 0-4 are `OperatingMode` cast to double (`include/xrpl/server/NetworkOPs.h:60-66`); 5 and 6 are the FULL-only refinements at `MetricsRegistry.cpp:1483-1498`. **The range is 0-6, not 0-7** — there is no seventh state.
#### Storage Detail (Observable Gauge — `storage_detail`)
@@ -1566,11 +1808,11 @@ State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full
| `storage_detail{metric="stored_object_bytes"}` | Int64 | `metric` | Cumulative object-payload bytes written (not on-disk size) |
> **`stored_object_bytes` is not a file size.** It observes `getStoreSize()`
> (`src/xrpld/telemetry/MetricsRegistry.cpp:1511`), which sums the object payloads
> (`src/xrpld/telemetry/MetricsRegistry.cpp:1557`), which sums the object payloads
> this process has written. It therefore excludes NuDB's keys, bucket padding and
> log, and it resets when the process restarts while the files on disk do not.
> `node_written_bytes` on the `nodestore_state` gauge calls the same accessor
> (`MetricsRegistry.cpp:836`), so the two series are equal by construction and any
> (`MetricsRegistry.cpp:860`), so the two series are equal by construction and any
> write-amplification ratio built from the pair is a constant 1.0. To size the store
> on disk, stat the backend's files; no metric reports it today.
>
@@ -1588,6 +1830,24 @@ State value encoding: 0=disconnected, 1=connected, 2=syncing, 3=tracking, 4=full
| `validations_checked_total` | Counter | Network validations observed | LedgerMaster.cpp |
| `state_changes_total` | Counter | Operating mode transitions | NetworkOPs.cpp |
> **Known issue — `ledgers_closed_total` has a dead second producer.** The
> instrument is created twice. `MetricsRegistry::registerCounters()` eagerly
> creates it as the member `ledgersClosedCounter_`
> (`src/xrpld/telemetry/MetricsRegistry.cpp:369-370`), and its only mutator,
> `MetricsRegistry::incrementLedgersClosed()`
> (declared `MetricsRegistry.h:591`, defined `MetricsRegistry.cpp:1686`), has
> **zero callers** — the header says so itself at `MetricsRegistry.h:584-588`.
> The value operators actually see comes from the single live increment,
> the `XRPL_METRIC_COUNTER_INC` call site in
> `RCLConsensus::Adaptor::doAccept()` (`src/xrpld/app/consensus/RCLConsensus.cpp:749`).
>
> No metric is wrong and nothing double-counts: the dead member never adds to the
> series. The cost is a redundant eagerly-created instrument plus a misleading API
> that looks like the increment path. **Code follow-up**: delete
> `incrementLedgersClosed()` and `ledgersClosedCounter_` once the macro path is
> considered proven, per the header note. Tracked here rather than fixed in a doc
> pass — the doc is not reworded to imply the member is used.
Lifetime tallies exported as monotonic **ObservableCounters** (not synchronous
counters), observed from an existing cumulative source each collection cycle:
@@ -1603,19 +1863,40 @@ counters), observed from an existing cumulative source each collection cycle:
> decrease) and additive (`agreements_total + missed_total` = ledgers reconciled). The
> repair-aware, windowed view remains on `validation_agreement{metric="…"}`.
#### Span Attribute Enrichments (Phases 2-4)
#### Span Attribute Enrichments (Phases 2-4): REMOVED
| Span Name | New Attribute | Type | Source |
| --------------------------- | ------------------------------------ | ------ | ------------------------ |
| `rpc.command.*` | `xrpl.node.amendment_blocked` | bool | Phase 2 — RPCHandler.cpp |
| `rpc.command.*` | `xrpl.node.server_state` | string | Phase 2 — RPCHandler.cpp |
| `tx.receive` | `xrpl.peer.version` | string | Phase 3 — PeerImp.cpp |
| `consensus.validation.send` | `xrpl.validation.ledger_hash` | string | Phase 4 — RCLConsensus |
| `consensus.validation.send` | `xrpl.validation.full` | bool | Phase 4 — RCLConsensus |
| `peer.validation.receive` | `xrpl.peer.validation.ledger_hash` | string | Phase 4 — PeerImp.cpp |
| `peer.validation.receive` | `xrpl.peer.validation.full` | bool | Phase 4 — PeerImp.cpp |
| `consensus.accept` | `xrpl.consensus.validation_quorum` | int64 | Phase 4 — RCLConsensus |
| `consensus.accept` | `xrpl.consensus.proposers_validated` | int64 | Phase 4 — RCLConsensus |
This section used to list nine dotted `xrpl.node.*` / `xrpl.peer.*` /
`xrpl.validation.*` / `xrpl.consensus.*` **span** attributes. **None of them
exists.** A grep for `xrpl.node.`, `xrpl.peer.`, `xrpl.validation.` and <!-- otel-naming:allow-dotted: xrpl.node., xrpl.peer., xrpl.validation. -->
`xrpl.consensus.` across non-test `src/` and `include/` returns nothing, and the <!-- otel-naming:allow-dotted: xrpl.consensus. -->
table also contradicted this document's own rule in
[§1.2](#12-complete-attribute-inventory-bareunderscore-keys): dotted keys are
OTel **resource** attributes, never span attributes.
The dotted form was dropped by the 2026-05-13 naming redesign, in three commits:
| Commit | Scope |
| ------------ | ------------------------------------------------------------------------------------------------------------- |
| `e339ba1f6b` | tx / txq — dropped the `xrpl.<domain>.` prefix (phase-3) |
| `46d1012ad4` | consensus — dropped the `xrpl.consensus.` prefix (phase-4) <!-- otel-naming:allow-dotted: xrpl.consensus. --> |
| `9e27120a15` | ledger / peer — simplified the keys, updated dashboards (phase-6) |
What the code emits today, and where it is documented:
| Old dotted key (never emitted) | Live equivalent |
| ----------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `xrpl.peer.version` | `peer_version` — see [§Transaction Attributes](#transaction-attributes) <!-- otel-naming:allow-dotted: xrpl.peer.version --> |
| `xrpl.validation.ledger_hash`, `xrpl.peer.validation.ledger_hash` | one bare `ledger_hash` on both `consensus.validation.send` and `peer.validation.receive` <!-- otel-naming:allow-dotted: xrpl.validation.ledger_hash, xrpl.peer.validation.ledger_hash --> |
| `xrpl.validation.full`, `xrpl.peer.validation.full` | one bare `full_validation` on both of those spans <!-- otel-naming:allow-dotted: xrpl.validation.full, xrpl.peer.validation.full --> |
| `xrpl.consensus.validation_quorum` | `quorum`, on `consensus.accept` only <!-- otel-naming:allow-dotted: xrpl.consensus.validation_quorum --> |
| `xrpl.node.amendment_blocked` | **not a span attribute at all** — only the metric `validator_health{metric="amendment_blocked"}` (`MetricsRegistry.cpp:1216`) <!-- otel-naming:allow-dotted: xrpl.node.amendment_blocked --> |
| `xrpl.node.server_state` | **not a span attribute at all** — only the metric `server_info{metric="server_state"}` (`MetricsRegistry.cpp:1014`) <!-- otel-naming:allow-dotted: xrpl.node.server_state --> |
| `xrpl.consensus.proposers_validated` | **never implemented** in any form <!-- otel-naming:allow-dotted: xrpl.consensus.proposers_validated --> |
The identical nine-row list was deleted from
`docker/telemetry/workload/expected_spans.json` by commit `cb9fce6890` for the
same reason. Anything still asserting these keys — a dashboard filter, a TraceQL
query, an alert — matches nothing and should be pointed at the live keys above.
### New Grafana Dashboards (Phase 9)
@@ -1663,20 +1944,24 @@ counters), observed from an existing cumulative source each collection cycle:
## 6. Known Issues
| Issue | Impact | Status |
| --------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| `warn` and `drop` metrics use non-standard StatsD `\|m` meter type | Metrics silently dropped by OTel StatsD receiver | Phase 6 Task 6.1 — needs `\|m` → `\|c` change in StatsDCollector.cpp |
| `jobq_job_count` may not emit in standalone mode | Missing from Prometheus in some test configs | Requires active job queue activity |
| `rpc_requests` depends on `[insight]` config | Zero series if StatsD not configured | Requires `[insight] server=statsd` in xrpld.cfg |
| Peer tracing enabled by default | `peer.*` spans emit unless `trace_peer=0` | High volume — set `trace_peer=0` to opt out on busy mainnet nodes |
| `handler="other"` mixes several producers | Cannot separate `GetConsL1` from `GetConsL2` | By design — the cardinality bound; see [§Per-Job-Type Metrics](#per-job-type-metrics-synchronous-countershistogram) |
| `overhead_cluster_*` is always zero | 8 dashboard panel references are flatlines by construction; cluster traffic is counted as `unknown` | **NOT IMPLEMENTED** — see [§6.0](#60-mtcluster-is-counted-as-unknown-not-implemented) |
| `squelch_ignored_bytes_in/out` always read zero | Only the `_messages_*` pair carries signal for this category | **NOT IMPLEMENTED** — see [§6.1](#61-squelch_ignored-byte-counts-not-implemented) |
| `total_bytes_in` and `total_bytes_out` use different size bases | In/out byte totals are not directly comparable when compression is on | **NOT IMPLEMENTED** — see [§6.2](#62-inboundoutbound-byte-basis-asymmetry-not-implemented) |
| `overhead` conflates `mtPING` with `mtSTATUS_CHANGE` | Keepalive traffic cannot be isolated from status-change traffic | **NOT IMPLEMENTED** — needs a new category; see [§6.3](#63-peer-keepalive-and-discovery-traffic-gaps-not-implemented) |
| No metrics for ping RTT distribution, ping timeouts, or `mtENDPOINTS` | Peer keepalive and discovery health are not observable | **NOT IMPLEMENTED** — see [§6.3](#63-peer-keepalive-and-discovery-traffic-gaps-not-implemented) |
| 11 of 13 peer message families have no spans | `02` §2.3.2 catalogs `peer.message.*`, `peer.connect`, `peer.disconnect` that were never built | **NOT IMPLEMENTED** — see [§6.4](#64-peer-span-coverage-gap-not-implemented) |
| PeerFinder exports 2 of ~17 available slot/cache readings | Slot pressure, connection churn and discovery-cache health are not observable | **NOT IMPLEMENTED** — see [§6.5](#65-peerfinder-slot-and-cache-metrics-not-implemented) |
| Issue | Impact | Status |
| --------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `warn` and `drop` metrics use non-standard StatsD `\|m` meter type | Metrics silently dropped by OTel StatsD receiver | Phase 6 Task 6.1 — needs `\|m` → `\|c` change in StatsDCollector.cpp |
| `jobq_job_count` may not emit in standalone mode | Missing from Prometheus in some test configs | Requires active job queue activity |
| `rpc_requests` depends on `[insight]` config | Zero series if StatsD not configured | Requires `[insight] server=statsd` in xrpld.cfg |
| Peer tracing enabled by default | `peer.*` spans emit unless `trace_peer=0` | High volume — set `trace_peer=0` to opt out on busy mainnet nodes |
| `handler="other"` mixes several producers | Cannot separate `GetConsL1` from `GetConsL2` | By design — the cardinality bound; see [§Per-Job-Type Metrics](#per-job-type-metrics-synchronous-countershistogram) |
| `overhead_cluster_*` is always zero | 8 dashboard panel references are flatlines by construction; cluster traffic is counted as `unknown` | **NOT IMPLEMENTED** — see [§6.0](#60-mtcluster-is-counted-as-unknown-not-implemented) |
| `squelch_ignored_bytes_in/out` always read zero | Only the `_messages_*` pair carries signal for this category | **NOT IMPLEMENTED** — see [§6.1](#61-squelch_ignored-byte-counts-not-implemented) |
| `total_bytes_in` and `total_bytes_out` use different size bases | In/out byte totals are not directly comparable when compression is on | **NOT IMPLEMENTED** — see [§6.2](#62-inboundoutbound-byte-basis-asymmetry-not-implemented) |
| `overhead` conflates `mtPING` with `mtSTATUS_CHANGE` | Keepalive traffic cannot be isolated from status-change traffic | **NOT IMPLEMENTED** — needs a new category; see [§6.3](#63-peer-keepalive-and-discovery-traffic-gaps-not-implemented) |
| No metrics for ping RTT distribution, ping timeouts, or `mtENDPOINTS` | Peer keepalive and discovery health are not observable | **NOT IMPLEMENTED** — see [§6.3](#63-peer-keepalive-and-discovery-traffic-gaps-not-implemented) |
| 11 of 13 peer message families have no spans | `02` §2.3.2 catalogs `peer.message.*`, `peer.connect`, `peer.disconnect` that were never built | **NOT IMPLEMENTED** — see [§6.4](#64-peer-span-coverage-gap-not-implemented) |
| PeerFinder exports 2 of ~17 available slot/cache readings | Slot pressure, connection churn and discovery-cache health are not observable | **NOT IMPLEMENTED** — see [§6.5](#65-peerfinder-slot-and-cache-metrics-not-implemented) |
| `ledger_history_mismatch_total` has two producers in one family | A bare `sum()` double-counts every mismatch; one series carries no `reason` label | **CODE BUG** — retire one producer; group by `reason` meanwhile. See [§TxQ Admission and Ledger Mismatch](#txq-admission-and-ledger-mismatch-synchronous-counters) |
| `overlay_peer_disconnects_charges` never existed | The documented selector matches nothing; use `server_info{metric="peer_disconnects_resources"}` | **NOT IMPLEMENTED** — see [§2.1](#21-gauges) |
| Nine dotted `xrpl.<domain>.*` span attributes never shipped | TraceQL filters and harness assertions on the dotted keys match nothing | **NOT IMPLEMENTED** — renamed to bare keys; see [§Span Attribute Enrichments](#span-attribute-enrichments-phases-2-4-removed) |
| `node_writes_duration_us` has no dashboard panel | Cumulative write latency is exported and linted, but never charted | Open follow-up — see [§Extended NodeStore Metrics](#extended-nodestore-metrics-additions-to-existing-nodestore_state) |
### 6.0 `mtCLUSTER` is counted as `unknown`: NOT IMPLEMENTED
@@ -1778,9 +2063,9 @@ messages are traced.
**Status**: NOT IMPLEMENTED. The span catalog in `02` §2.3.2 is a design
inventory, not a statement of what emits; §2.3.2 now marks which entries are
live. Instrumenting the remaining families would change the "~37 spans" count
asserted in [§1.1](#11-complete-span-inventory-37-spans) and in
`docker/telemetry/workload/expected_spans.json`, so it is scoped as its own
live. Instrumenting the remaining families would change the **41 span families**
counted in [§1.1](#11-complete-span-inventory-41-spans) and the **40** catalogued
in `docker/telemetry/workload/expected_spans.json`, so it is scoped as its own
change rather than folded into a metric task.
### 6.5 PeerFinder slot and cache metrics: NOT IMPLEMENTED

69
OpenTelemetryPlan/OpenTelemetryPlan.md
View File

@@ -102,12 +102,16 @@ flowchart TB
| **2** | [Design Decisions](./02-design-decisions.md) | SDK selection, exporters, span naming, attributes, context propagation |
| **3** | [Implementation Strategy](./03-implementation-strategy.md) | Directory structure, key principles, performance optimization |
| **5** | [Configuration Reference](./05-configuration-reference.md) | xrpld config, CMake integration, Collector configurations |
| **6** | [Implementation Phases](./06-implementation-phases.md) | 5-phase timeline, tasks, risks, success metrics |
| **6** | [Implementation Phases](./06-implementation-phases.md) | 11-phase timeline, tasks, risks, success metrics |
| **7** | [Observability Backends](./07-observability-backends.md) | Backend selection guide and production architecture |
| **8** | [Appendix](./08-appendix.md) | Glossary, references, version history |
| **9** | [Data Collection Reference](./09-data-collection-reference.md) | Complete inventory of spans, attributes, metrics, and dashboards |
| **Sec** | [Securing the OTel Pipeline](./secure-OTel.md) | Threat model and hardening (mTLS, peer trace-context validation) |
> Note there is no document 4: `04-code-samples.md` was removed during the
> rollout, and the numbering was left as-is rather than renumbering every
> cross-reference in the chain.
---
## 0. Tracing Fundamentals
@@ -136,9 +140,9 @@ Key trace points span across transaction submission via RPC, peer-to-peer messag
The OpenTelemetry C++ SDK is selected for its CNCF backing, active development, and native performance characteristics. Traces are exported via OTLP/HTTP to an OpenTelemetry Collector, which provides flexible routing and sampling. OTLP/gRPC is planned future work (see design decisions §2.2.2).
Span naming follows a hierarchical `<component>.<operation>` convention (e.g., `rpc.submit`, `tx.relay`, `consensus.round`). Context propagation uses W3C Trace Context headers for HTTP and embedded Protocol Buffer fields for P2P messages. The implementation coexists with existing PerfLog and Insight observability systems through correlation IDs.
Span naming follows a hierarchical `<component>.<operation>` convention (e.g., `rpc.command.server_info`, `tx.process`, `consensus.round`). Context propagation uses W3C Trace Context headers for HTTP and embedded Protocol Buffer fields for P2P messages. The implementation coexists with existing PerfLog and Insight observability systems through correlation IDs.
**Data Collection & Privacy**: Telemetry collects only operational metadata (timing, counts, hashes) — never sensitive content (private keys, balances, amounts, raw payloads). Privacy protection includes account hashing, configurable redaction, sampling, and collector-level filtering. Node operators retain full control over telemetry configuration.
**Data Collection & Privacy**: Telemetry collects only operational metadata (timing, counts, hashes) — never sensitive content (private keys, balances, amounts, raw payloads). Account addresses are hashed **unconditionally** by the SDK helper and hashed again at the collector; there is no redaction config key and therefore no insecure-by-default state. Trace volume is _not_ reduced on the node (head sampling is fixed at 100%); reduction, where wanted, is a collector-side tail-sampling decision. Node operators control which subsystems are traced via the `[telemetry]` per-component toggles.
➡️ **[Read full Design Decisions](./02-design-decisions.md)**
@@ -146,9 +150,9 @@ Span naming follows a hierarchical `<component>.<operation>` convention (e.g., `
## 3. Implementation Strategy
The telemetry code is organized under `include/xrpl/telemetry/` for headers and `src/libxrpl/telemetry/` for implementation. Key principles include RAII-based span management via `SpanGuard` (with `discard()` for dropping unwanted spans), a `FilteringSpanProcessor` that intercepts `OnEnd()` to prevent discarded spans from entering the export pipeline, conditional compilation with `XRPL_ENABLE_TELEMETRY`, and minimal runtime overhead through batch processing and efficient sampling.
The telemetry code is organized under `include/xrpl/telemetry/` for headers, `src/libxrpl/telemetry/` for implementation, and `src/xrpld/telemetry/` for the native-metrics module added in Phases 7 and 9. Key principles include RAII-based span management via `SpanGuard` (with `discard()` for dropping unwanted spans), a `FilteringSpanProcessor` that intercepts `OnEnd()` to prevent discarded spans from entering the export pipeline, conditional compilation behind the `XRPL_ENABLE_TELEMETRY` compile definition (set by the CMake `telemetry` option, which defaults to **ON** — build it out with `-Dtelemetry=OFF`), and minimal runtime overhead through batch processing.
Performance optimization strategies include head sampling fixed at 100% (intentionally not configurable, so trace keep/drop decisions stay coherent across nodes), tail-based sampling at the collector for errors and slow traces to reduce volume, batch export to reduce network overhead, and conditional instrumentation that compiles to no-ops when disabled.
Performance optimization strategies include head sampling fixed at 100% (intentionally not configurable, so trace keep/drop decisions stay coherent across nodes), optional tail-based sampling at the collector to reduce stored volume (not enabled in the base stack — the only shipped policy is a 0.5% probabilistic one in the Grafana Cloud overlay), batch export to reduce network overhead, and conditional instrumentation that compiles to no-ops when disabled.
➡️ **[Read full Implementation Strategy](./03-implementation-strategy.md)**
@@ -158,9 +162,19 @@ Performance optimization strategies include head sampling fixed at 100% (intenti
> **OTLP** = OpenTelemetry Protocol | **APM** = Application Performance Monitoring
Configuration is handled through the `[telemetry]` section in `xrpld.cfg` with options for enabling/disabling, exporter selection, endpoint configuration, and component-level filtering. Head sampling is fixed at 1.0 (not operator-configurable); volume reduction is done by tail sampling in the collector. CMake integration includes a `XRPL_ENABLE_TELEMETRY` option for compile-time control.
Configuration is handled through the `[telemetry]` section in `xrpld.cfg` with options for enabling/disabling, TLS/mTLS, batch tuning, and component-level filtering. Exporter selection is _not_ configurable — OTLP/HTTP is the only transport. Head sampling is fixed at 1.0 (not operator-configurable); volume reduction is done by tail sampling in the collector. CMake integration uses the `telemetry` option (default **ON**) for compile-time control.
OpenTelemetry Collector configurations are provided for development and production (with tail-based sampling, Tempo, and Elastic APM). Docker Compose examples enable quick local development environment setup.
Endpoints are spread across **three** keys in two sections, not one "traces and metrics" pair:
| Signal | Key | Default | Source |
| ---------------------------------------------------- | ------------------------------ | ---------------------------------- | --------------------------- |
| Traces | `[telemetry] endpoint` | `http://localhost:4318/v1/traces` | `TelemetryConfig.cpp:36,61` |
| Native metrics (`XRPL_METRIC_*` / `MetricsRegistry`) | `[telemetry] metrics_endpoint` | `http://localhost:4318/v1/metrics` | `Application.cpp:1670` |
| `beast::insight` metrics (`server=otel`) | `[insight] endpoint` | `http://localhost:4318/v1/metrics` | `CollectorManager.cpp:50` |
`[telemetry]` itself has exactly **one** `endpoint` key, and it is traces-only.
The repo ships one collector config (`docker/telemetry/otel-collector-config.yaml`, three pipelines: traces, metrics, logs) plus a Grafana Cloud overlay that adds 0.5% tail sampling. A six-service Docker Compose stack — collector, Tempo, Loki, Prometheus, Grafana, renderer — gives a complete local environment.
➡️ **[View full Configuration Reference](./05-configuration-reference.md)**
@@ -168,20 +182,31 @@ OpenTelemetry Collector configurations are provided for development and producti
## 6. Implementation Phases
The implementation spans 13 weeks across 8 phases:
The plan was originally scoped at **13 weeks across 8 phases** — the table below
is that original scope. As delivered it grew to **11 phases through week 20**;
Phases 9-11 were added after the original plan was written. See
[06-implementation-phases.md §6.12.6](./06-implementation-phases.md) for the
authoritative per-phase status, and treat the eight rows below as the
originally-planned subset rather than the current timeline:
| Phase | Duration | Focus | Key Deliverables |
| ----- | ----------- | --------------------- | ----------------------------------------------------------- |
| 1 | Weeks 1-2 | Core Infrastructure | SDK integration, Telemetry interface, Configuration |
| 2 | Weeks 3-4 | RPC Tracing | HTTP context extraction, Handler instrumentation |
| 3 | Weeks 5-6 | Transaction Tracing | Protocol Buffer context, Relay propagation |
| 4 | Weeks 7-8 | Consensus Tracing | Round spans, Proposal/validation tracing |
| 5 | Week 9 | Documentation | Runbook, Dashboards, Training |
| 6 | Week 10 | StatsD Metrics Bridge | OTel Collector StatsD receiver, 3 Grafana dashboards |
| 7 | Weeks 11-12 | Native OTel Metrics | OTelCollector impl, OTLP metrics export, StatsD deprecation |
| 8 | Week 13 | Log-Trace Correlation | trace_id in logs, Loki ingestion, Tempo↔Loki linking |
| Phase | Duration | Focus | Key Deliverables |
| ----- | ----------- | --------------------- | --------------------------------------------------------- |
| 1 | Weeks 1-2 | Core Infrastructure | SDK integration, Telemetry interface, Configuration |
| 2 | Weeks 3-4 | RPC Tracing | HTTP context extraction, Handler instrumentation |
| 3 | Weeks 5-6 | Transaction Tracing | Protocol Buffer context, Relay propagation |
| 4 | Weeks 7-8 | Consensus Tracing | Round spans, Proposal/validation tracing |
| 5 | Week 9 | Documentation | Runbook, Dashboards, Training |
| 6 | Week 10 | StatsD Metrics Bridge | OTel Collector StatsD receiver, 3 Grafana dashboards |
| 7 | Weeks 11-12 | Native OTel Metrics | OTelCollector impl, OTLP metrics export (StatsD retained) |
| 8 | Week 13 | Log-Trace Correlation | trace_id in logs, Loki ingestion, Tempo↔Loki linking |
**Total Effort**: 65.1 developer-days with 2 developers
Delivered beyond the original scope: **Phase 9** (weeks 14-15, internal metric
instrumentation gap fill), **Phase 10** (weeks 16-17, synthetic workload
generation and telemetry validation) and **Phase 11** (weeks 18-20, third-party
data-collection pipelines).
**Total Effort**: 65.1 developer-days with 2 developers, for the eight
originally-planned phases only.
➡️ **[View full Implementation Phases](./06-implementation-phases.md)**
@@ -191,9 +216,9 @@ The implementation spans 13 weeks across 8 phases:
> **APM** = Application Performance Monitoring | **GCS** = Google Cloud Storage
Grafana Tempo is recommended for all environments due to its cost-effectiveness and Grafana integration, while Elastic APM is ideal for organizations with existing Elastic infrastructure.
Grafana Tempo is recommended for all environments due to its cost-effectiveness and Grafana integration, and it is the only backend this repo provisions. Elastic APM remains a reasonable choice for organizations with existing Elastic infrastructure, but nothing here configures it.
The recommended production architecture uses a gateway collector pattern with regional collectors performing tail-based sampling, routing traces to multiple backends (Tempo for primary storage, Elastic for log correlation, S3/GCS for long-term archive).
The recommended production architecture uses a gateway collector pattern with regional collectors performing tail-based sampling, routing traces to multiple backends (Tempo for primary storage, Elastic for log correlation, S3/GCS for long-term archive). Note that several subsections of doc 7 predate the shipped dashboards and alert rules and are marked superseded in place, pointing at [09-data-collection-reference.md](./09-data-collection-reference.md) and `docs/telemetry-runbook.md`.
➡️ **[View Observability Backend Recommendations](./07-observability-backends.md)**
@@ -209,7 +234,7 @@ The appendix contains a glossary of OpenTelemetry and xrpld-specific terms, refe
## 9. Data Collection Reference
A single-source-of-truth reference documenting every piece of telemetry data collected by xrpld. Covers all 16 OpenTelemetry spans with their 22 attributes, all StatsD metrics (gauges, counters, histograms, overlay traffic), SpanMetrics-derived Prometheus metrics, and all 10 Grafana dashboards. Includes Tempo search guides and Prometheus query examples.
A single-source-of-truth reference documenting every piece of telemetry data collected by xrpld: the OpenTelemetry span inventory with per-span attributes, the `beast::insight` and native `XRPL_METRIC_*` instruments (gauges, counters, histograms, overlay traffic), the SpanMetrics-derived Prometheus metrics, and the **15** Grafana dashboards. Includes Tempo search guides and Prometheus query examples. Consult that document rather than this index for any count — it tracks the code, this summary does not.
➡️ **[View Data Collection Reference](./09-data-collection-reference.md)**

94
OpenTelemetryPlan/Phase10_taskList.md
View File

@@ -22,10 +22,13 @@
Before Phases 1-9 can be considered production-ready, we need proof that:
1. All 16 spans fire with correct attributes under real transaction workloads
1. Every emitted span fires with its required attributes under real transaction
workloads (the "16 spans / 22 attributes" figures below are stale; the harness
derives both totals from `expected_spans.json`)
2. All 255+ StatsD metrics + ~50 Phase 9 metrics appear in Prometheus with non-zero values
3. Log-trace correlation (Phase 8) produces clickable trace_id links in Loki
4. All 10 Grafana dashboards render meaningful data (no empty panels)
4. The 14 harness-asserted Grafana dashboards render meaningful data (no empty
panels); 15 are on disk
5. Performance overhead stays within bounds (< 3% CPU, < 5MB memory)
6. The telemetry stack survives sustained load without data loss or queue backpressure
@@ -37,25 +40,42 @@ Before Phases 1-9 can be considered production-ready, we need proof that:
**What to do**:
- Create `docker/telemetry/docker-compose.workload.yaml`:
- 5 xrpld validator nodes with UNL configured for each other
- All telemetry enabled: `[telemetry] enabled=1`, `[insight] server=otel`
- Full OTel stack: Collector, Tempo, Prometheus, Loki, Grafana
- Shared network with service discovery
- Create `docker/telemetry/docker-compose.workload.yaml` — **as shipped this file
holds only the observability backend**: `otel-collector`, `tempo`,
`prometheus`, `loki`, `grafana`. It contains **no xrpld services**.
- Shared network (`workload-net`) with service discovery
- Each node should:
- Generate validator keys at startup
- Configure all 5 nodes in its UNL
- Enable all trace categories including `trace_peer=1`
- Write logs to a file tailed by the OTel Collector filelog receiver
- The 5 validators are **native `xrpld` processes**, not containers.
`docker/telemetry/workload/run-full-validation.sh` (`NUM_NODES=5`) generates
keys, writes a per-node `xrpld.cfg`, and launches each node on
`127.0.0.1` with sequential RPC / WS / peer ports. Each node:
- Gets its validator key from `generate-validator-keys.sh`
- Lists the other 4 nodes in `ips_fixed`
- Has all telemetry enabled: `[telemetry] enabled=1`, `[insight] server=otel`
- Enables all trace categories including `trace_peer=1`
- Writes logs to a file tailed by the OTel Collector filelog receiver
- Include a `Makefile` target: `make telemetry-workload-up` / `make telemetry-workload-down`
- ❌ **`make telemetry-workload-up` / `make telemetry-workload-down` were never
implemented.** There is no `Makefile` anywhere in the repository. The entry
point is `run-full-validation.sh` (with `--profile`, `--nodes`,
`--skip-loki`, `--skip-regression`, `--with-benchmark`). The node-count flag is
spelled `--nodes`, **not** `--num-nodes` — `run-full-validation.sh:80` (usage)
and `:100` (the `case` arm). `NUM_NODES` is the internal shell variable it
assigns to.
**Key files**:
- New: `docker/telemetry/docker-compose.workload.yaml`
- New: `docker/telemetry/docker-compose.workload.yaml` (backend only)
- New: `docker/telemetry/workload/generate-validator-keys.sh`
- New: `docker/telemetry/workload/xrpld-validator.cfg.template`
- New: `docker/telemetry/workload/run-full-validation.sh` — writes each node's
cfg **inline** via a heredoc at `run-full-validation.sh:242`
(`cat >"$NODE_DIR/xrpld.cfg" <<EOCFG`)
- New: `docker/telemetry/workload/xrpld-validator.cfg.template` (96 lines) — it
**was** created and is tracked on the Phase 10 branch, but it is **unused**:
nothing reads it, and its `{{NODE_INDEX}}` / `{{RPC_PORT}}` / `{{OTEL_ENDPOINT}}`
placeholders are never substituted, because the inline heredoc above supersedes
it. Either wire the script to the template or delete the template — keeping both
guarantees they drift.
---
@@ -122,15 +142,30 @@ Before Phases 1-9 can be considered production-ready, we need proof that:
- Create `docker/telemetry/workload/validate_telemetry.py`:
**Span validation** (queries Tempo API):
- Assert all 16 span names appear in traces
- Assert each span has its required attributes (22 total attributes across spans)
- Assert parent-child relationships are correct (`rpc.request` → `rpc.process` → `rpc.command.*`)
- Assert every span name in `expected_spans.json` appears in traces
- Assert each span has its required attributes
- Assert parent-child relationships are correct. `rpc.request` no longer
exists — it split into `rpc.http_request` (HTTP) and `rpc.ws_message`
(WebSocket) (`RpcSpanNames.h:135`, `:133`). The two live trees are:
- HTTP: `rpc.http_request` → `rpc.process` → `rpc.command.*`
- WebSocket: `rpc.ws_message` → `rpc.command.*` — **there is no
`rpc.process` on the WS path**. `rpc.process` is created only in
`ServerHandler::processRequest()` (`ServerHandler.cpp:705`), reached from
`processSession(Session, coro)`, i.e. HTTP only. Under WS-only load
`rpc.process` never appears, and `rpc.command.*` parents directly to
`rpc.ws_message`.
- Assert span durations are reasonable (> 0, < 60s)
**Metric validation** (queries Prometheus API):
- Assert all SpanMetrics-derived metrics are non-zero: `traces_span_metrics_calls_total`, `traces_span_metrics_duration_milliseconds_bucket`
- Assert all StatsD metrics are non-zero: `xrpld_LedgerMaster_Validated_Ledger_Age`, `xrpld_Peer_Finder_Active_*`, etc.
- Assert all Phase 9 metrics are non-zero: `xrpld_nodestore_*`, `xrpld_cache_*`, `xrpld_txq_*`, `xrpld_rpc_method_*`, `xrpld_object_count`, `xrpld_load_factor*`
- Assert all SpanMetrics-derived metrics are non-zero: `span_calls_total`,
`span_duration_milliseconds_bucket` (the connector's `namespace` is `span`,
not `traces_span_metrics` — `otel-collector-config.yaml:113-114`)
- Assert the insight-sourced metrics are non-zero: `ledgermaster_validated_ledger_age`,
`peer_finder_active_{inbound,outbound}_peers`, etc. — all lowercase, no
`xrpld_` prefix (`77f35c03db` removed the prefix and lowercased names)
- Assert all Phase 9 metrics are non-zero: `nodestore_state`, `cache_metrics`,
`txq_metrics`, `rpc_method_{started,finished,errored}_total`, `object_count`,
`load_factor_metrics`
- Assert metric label cardinality is within bounds
**Log-trace correlation validation** (queries Loki API):
@@ -139,7 +174,9 @@ Before Phases 1-9 can be considered production-ready, we need proof that:
- Assert Grafana derived field links are functional
**Dashboard validation**:
- For each of the 10 Grafana dashboards, query the dashboard API and assert no panels show "No data"
- For each dashboard, query the dashboard API and assert no panels show "No
data". There are **15 dashboards on disk**; the harness asserts **14** —
`log-derived-insights` is provisioned but unasserted.
- Output: JSON report with pass/fail per check, suitable for CI.
@@ -231,12 +268,17 @@ Before Phases 1-9 can be considered production-ready, we need proof that:
## Exit Criteria
- [ ] 5-node validator cluster starts and reaches consensus in docker-compose
- [ ] 5-node validator cluster starts and reaches consensus — as native `xrpld`
processes driven by `run-full-validation.sh`, not from docker-compose
- [ ] RPC load generator fires all traced RPC commands at configurable rates
- [ ] Transaction submitter generates 6+ transaction types at configurable TPS
- [ ] Validation suite confirms all 16 spans, 22 attributes, 300+ metrics are present
- [ ] Log-trace correlation validated end-to-end (Loki ↔ Tempo)
- [ ] All 10 Grafana dashboards render data (no empty panels)
- [ ] Validation suite confirms the full span / attribute / metric inventory
(totals computed dynamically from `expected_spans.json` /
`expected_metrics.json`, not the stale 16 / 22 figures)
- [ ] Log-trace correlation validated end-to-end (Loki ↔ Tempo) — implemented,
but CI runs with `--skip-loki`, so it is not gated
- [ ] All 14 harness-asserted Grafana dashboards render data (no empty panels);
15 on disk
- [ ] Benchmark shows < 3% CPU overhead, < 5MB memory overhead
- [ ] CI workflow runs validation on telemetry branch changes
- [ ] Validation report output is CI-parseable (JSON with exit codes)

172
OpenTelemetryPlan/Phase11_taskList.md
View File

@@ -1,6 +1,17 @@
# Phase 11: Third-Party Data Collection Pipelines — Task List
> **Status**: Future Enhancement
> **Status**: Not started — 0 of 13 tasks complete (`grep -c '^## Task 11\.'` = 13:
> Tasks 11.1 through 11.13). Verified against the tree:
> no `.go` files exist anywhere, `docker/telemetry/otel-rippled-receiver/` does
> not exist, `docker/telemetry/prometheus/` does not exist (so no
> `rippled-alerts.yml`), and no `network-topology` / `dex-amm` dashboards are
> present under `docker/telemetry/grafana/dashboards/`. **No Phase 11 work has
> been done, so no task box below may be ticked.**
>
> One **prerequisite** box is ticked, and only one: Task 11.12's
> "`state_tracking` gauge implemented (Task 7.12)". That is an upstream
> dependency satisfied by Phase 7/9 code, not Phase 11 work — see the citation
> there.
>
> **Goal**: Build a custom OTel Collector receiver that periodically polls xrpld's admin RPCs and exports structured metrics for external consumers — making all XRPL health, validator, peer, fee, and DEX data available as Prometheus/OTLP metrics without xrpld code changes.
>
@@ -287,7 +298,35 @@ This phase addresses the cross-cutting gap identified during research: **xrpld h
## Task 11.8: Prometheus Alerting Rules
**Objective**: Create production-ready alerting rules for the metrics exported by this receiver.
**Objective**: Create production-ready alerting rules for the `xrpl_*` metrics
exported by this receiver.
> **Scope note — do not duplicate Phase 9.** Phase 9 already ships provisioned
> **Grafana** alerting at
> `docker/telemetry/grafana/provisioning/alerting/{rules,contactpoints,policies}.yaml`
> — 13 rules in 5 groups, 2 contact points (`xrpld-default` Slack,
> `xrpld-critical` Slack + email), and a nested notification policy keyed on
> `severity = critical`. Four of the rules below overlap it:
>
> | Rule here | Addressed by (Phase 9) | Coverage |
> | ------------------- | ------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
> | `XRPLServerNotFull` | `NodeNotFull` (group `xrpld-node-state`) | Full |
> | `XRPLLedgerStale` | `ValidatedLedgerStale` (group `xrpld-consensus`) | **Partial** — Phase 9: `ledgermaster_validated_ledger_age > 60` for 5m; the external shape is `> 30` for 1m |
> | `XRPLHighIOLatency` | `NodeStoreIOLatencyHigh` (group `xrpld-jobqueue`) | **Partial** — Phase 9: p95 of `ios_latency_milliseconds_bucket` **> 1000 ms for 10m**; external: **> 50 for 1m** |
> | `XRPLStateFlapping` | `NodeStateFlapping` (group `xrpld-node-state`) | Full |
>
> The remaining 8 (`XRPLAmendmentBlocked`, `XRPLNoPeers`,
> `XRPLUnsupportedAmendmentMajority`, `XRPLLowPeerCount`, `XRPLHighLoadFactor`,
> `XRPLSlowConsensus`, `XRPLValidatorListExpiring`, `XRPLClockDrift`) are
> genuinely new. Note the two sets watch different metric surfaces — the Phase 9
> rules fire on xrpld's own OTLP metrics, these on the receiver's `xrpl_*`
> metrics — so if both are kept, dedupe the notification policy to avoid
> double-paging on the same underlying condition.
>
> `docker/telemetry/prometheus/` does not exist today. Prefer extending the
> Phase 9 Grafana provisioning tree over introducing a second, Prometheus-native
> alerting mechanism; if a `prometheus/` tree is added anyway, say explicitly in
> its header which alerts it owns.
**What to do**:
@@ -360,9 +399,22 @@ This phase addresses the cross-cutting gap identified during research: **xrpld h
**Objective**: Create 4 new dashboards for the data exported by the receiver.
> **UID COLLISION — pick a different uid.** Phase 9 already ships
> `docker/telemetry/grafana/dashboards/validator-health.json` with
> **uid `validator-health`** (17 panels, backed by xrpld's own
> `validation_agreement` / `validator_health` / `state_tracking` OTLP metrics).
> Provisioning a second dashboard with the same uid makes Grafana overwrite one
> with the other — whichever the provisioner loads last wins, silently. Use a
> distinct uid such as `validator-health-external` (and a distinct filename), the
> same way this task already disambiguates Fee Market as
> `xrpld-fee-market-external` against Phase 9's `fee-market`. Also check
> `peer-quality`, `fee-market`, `job-queue` and `node-health` before adding any
> further uid.
**What to do**:
- **Validator Health** (`validator-health`):
- **Validator Health** (`validator-health-external` — **not** `validator-health`,
see the collision note above):
- Server state timeline, state duration breakdown
- Proposer count trend, converge time trend, validation quorum
- Validator list expiration countdown
@@ -386,10 +438,16 @@ This phase addresses the cross-cutting gap identified during research: **xrpld h
**Key files**:
- New: `docker/telemetry/grafana/dashboards/rippled-validator-health.json`
- New: `docker/telemetry/grafana/dashboards/rippled-network-topology.json`
- New: `docker/telemetry/grafana/dashboards/rippled-fee-market-external.json`
- New: `docker/telemetry/grafana/dashboards/rippled-dex-amm.json`
- New: `docker/telemetry/grafana/dashboards/validator-health-external.json`
(**must not** reuse Phase 9's `validator-health.json` / uid `validator-health`)
- New: `docker/telemetry/grafana/dashboards/network-topology.json`
- New: `docker/telemetry/grafana/dashboards/fee-market-external.json`
(Phase 9 owns `fee-market.json` / uid `fee-market`)
- New: `docker/telemetry/grafana/dashboards/dex-amm.json`
> Filenames drop the `rippled-` prefix: `145b1469d6` and `25868f2740` renamed
> every dashboard to bare names with bare uids, so no `rippled-*.json` path
> exists in the tree.
---
@@ -446,20 +504,55 @@ This phase addresses the cross-cutting gap identified during research: **xrpld h
> **Upstream**: Phase 7 Tasks 7.9-7.16 (metrics), Phase 9 Tasks 9.11-9.13 (dashboards).
> **Downstream**: None — terminal task in the parity chain.
**Objective**: Add Grafana alerting rules for the Phase 7+ parity metrics (validation agreement, validator health, peer quality, state tracking, ledger economy). These complement Task 11.8's `xrpl_*` alerts by covering the `xrpld_*` internal metrics.
**Objective**: Add Grafana alerting rules for the Phase 7+ parity metrics (validation agreement, validator health, peer quality, state tracking, ledger economy). These complement Task 11.8's `xrpl_*` alerts by covering the internal metrics.
> **4 of the 18 are addressed by Phase 9** — 2 fully, 2 only partially. Extend,
> do not blindly re-create:
>
> | Rule here | Addressed by (Phase 9) | Coverage |
> | ------------------ | ------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
> | Unhealthy State | `NodeNotFull` (group `xrpld-node-state`) | Full |
> | High IO Latency | `NodeStoreIOLatencyHigh` (group `xrpld-jobqueue`, p95 of `ios_latency_milliseconds_bucket`) | **Partial** — Phase 9 fires at p95 **> 1000 ms for 10m**; the rule below wants **> 50 for 1m** (20× tighter) |
> | Job Queue Overflow | `JobQueueTxOverflow` (group `xrpld-jobqueue`, `jq_trans_overflow_total`) | Full |
> | Stale Ledger | `ValidatedLedgerStale` (group `xrpld-consensus`, `ledgermaster_validated_ledger_age`) | **Partial** — different metric: Phase 9 uses `ledgermaster_validated_ledger_age > 60` for 5m; the rule below uses `ledger_economy{metric="ledger_age_seconds"} > 30` for 1m |
>
> The two **Partial** rows are not closed. Either re-baseline the Phase 9
> thresholds or ship the tighter variants here — do not skip them as duplicates.
>
> Remaining open work is **14 rules**, of which **3** (CPU High, Memory Critical,
> Disk Warning) need `node_exporter`, which is not in the stack. Nothing else is
> blocked: "Not Proposing" used to be listed as blocked on an unimplemented
> `state_tracking` gauge, but that gauge **ships** — see the Exit Criteria note
> below.
>
> **Metric-name translation.** Names carry **no** `xrpld_` prefix
> (`77f35c03db`), so as a rule of thumb read every `xrpld_<name>` below as plain
> `<name>`. **Two shapes do not follow that rule:**
>
> - **Multiplexed observable gauges.** Many readings are a `metric` **label
> value** on a shared instrument, not a metric name. `xrpld_txq_count` is
> `txq_metrics{metric="txq_count"}`; likewise `load_factor_metrics{…}`,
> `nodestore_state{…}`, `cache_metrics{…}`. The rows below that already use the
> `<instrument>{metric="…"}` form (`state_tracking`, `validator_health`,
> `validation_agreement`, `server_info`, `peer_quality`, `load_factor_metrics`,
> `ledger_economy`) are correct; only drop the prefix on those.
> - **Unit-suffixed histograms** from `beast::insight`. `OTelCollectorImp` appends
> the unit, so `xrpld_ios_latency_bucket` is really
> `ios_latency_milliseconds_bucket` — the spelling used by
> `node-health.json:577` and `ledger-data-sync.json:1353`.
**Critical Group** (8 rules, eval interval 10s):
| Rule | Condition | For |
| ------------------- | ------------------------------------------------------------- | --- |
| Agreement Below 90% | `xrpld_validation_agreement{metric="agreement_pct_24h"} < 90` | 30s |
| Not Proposing | `xrpld_state_tracking{metric="state_value"} < 6` | 10s |
| Unhealthy State | `xrpld_state_tracking{metric="state_value"} < 4` | 10s |
| Amendment Blocked | `xrpld_validator_health{metric="amendment_blocked"} == 1` | 1m |
| UNL Expiring | `xrpld_validator_health{metric="unl_expiry_days"} < 14` | 1h |
| High IO Latency | `histogram_quantile(0.95, xrpld_ios_latency_bucket) > 50` | 1m |
| High Load Factor | `xrpld_load_factor_metrics{metric="load_factor"} > 1000` | 1m |
| Peer Count Critical | `xrpld_server_info{metric="peers"} < 5` | 1m |
| Rule | Condition | For |
| ------------------- | ---------------------------------------------------------------- | --- |
| Agreement Below 90% | `xrpld_validation_agreement{metric="agreement_pct_24h"} < 90` | 30s |
| Not Proposing | `xrpld_state_tracking{metric="state_value"} < 6` | 10s |
| Unhealthy State | `xrpld_state_tracking{metric="state_value"} < 4` | 10s |
| Amendment Blocked | `xrpld_validator_health{metric="amendment_blocked"} == 1` | 1m |
| UNL Expiring | `xrpld_validator_health{metric="unl_expiry_days"} < 14` | 1h |
| High IO Latency | `histogram_quantile(0.95, ios_latency_milliseconds_bucket) > 50` | 1m |
| High Load Factor | `xrpld_load_factor_metrics{metric="load_factor"} > 1000` | 1m |
| Peer Count Critical | `xrpld_server_info{metric="peers"} < 5` | 1m |
**Network Group** (3 rules, eval interval 10s):
@@ -481,19 +574,44 @@ This phase addresses the cross-cutting gap identified during research: **xrpld h
| TX Rate Drop | Transaction rate dropped > 50% in 5m window | 5m |
| Stale Ledger | `xrpld_ledger_economy{metric="ledger_age_seconds"} > 30` | 1m |
**Notification channel templates**: Email/SMTP, Discord, Slack, PagerDuty.
**Notification channel templates**: Slack and Email/SMTP already ship in Phase
9's `contactpoints.yaml` (`xrpld-default`, `xrpld-critical`). Discord and
PagerDuty templates remain open.
**Key files**:
**Key files** — extend the **Phase 9** provisioning tree. The
`docker/telemetry/grafana/alerting/` directory named in the original spec has
never existed in any commit; the real location is
`docker/telemetry/grafana/provisioning/alerting/`:
- New/extend: `docker/telemetry/grafana/alerting/alert-rules-parity.yaml`
- New: `docker/telemetry/grafana/alerting/contact-points.yaml` (template configs)
- New: `docker/telemetry/grafana/alerting/notification-policies.yaml`
- Extend: `docker/telemetry/grafana/provisioning/alerting/rules.yaml` (add groups
alongside the existing `xrpld-consensus`, `xrpld-validator`, `xrpld-jobqueue`,
`xrpld-node-state`, `xrpld-overlay`)
- Extend: `docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml`
(add Discord / PagerDuty receivers)
- Extend: `docker/telemetry/grafana/provisioning/alerting/policies.yaml`
(add routes; the root route and the `severity = critical` child already exist)
**Exit Criteria**:
- [ ] All 18 rules evaluate without errors in Grafana alerting UI
- [ ] The 14 not-yet-shipped rules evaluate without errors in Grafana alerting UI
- [ ] The 2 rules **fully** covered by Phase 9 (Unhealthy State, Job Queue
Overflow) are not duplicated; the 2 **partially** covered ones (High IO
Latency, Stale Ledger) are either re-baselined on the Phase 9 rule or shipped
as tighter variants — decision recorded either way
- [ ] Critical rules fire within expected timeframe when conditions are met
- [ ] Notification channel templates are documented (not hard-coded to any service)
- [ ] `node_exporter` decision recorded for the 3 host-level rules (CPU, memory, disk)
- [x] `state_tracking` gauge implemented (Task 7.12) before adding "Not Proposing"
— **prerequisite met upstream**, not Phase 11 work.
`MetricsRegistry::registerStateTrackingGauge()`
(`src/xrpld/telemetry/MetricsRegistry.cpp:1461-1510`) creates
`CreateDoubleObservableGauge("state_tracking", "Node state and mode tracking")`
at `:1466` and observes `state_value` (`:1497`) and
`time_in_current_state_seconds` (`:1502`). Already queried by
`validator-health.json:765,971` and `ledger-data-sync.json:869`, and
documented in
[09-data-collection-reference.md](./09-data-collection-reference.md)
§ State Tracking. "Not Proposing" can be written now.
---
@@ -533,12 +651,14 @@ This phase addresses the cross-cutting gap identified during research: **xrpld h
- [ ] Custom OTel Collector receiver builds and starts without errors
- [ ] All `xrpl_*` metrics from server_info, get_counts, peers, validators, fee appear in Prometheus
- [ ] Metrics update at configured poll interval (default 30s)
- [ ] 4 new Grafana dashboards operational with data
- [ ] 4 new Grafana dashboards operational with data, none reusing a Phase 9 uid
(`validator-health`, `peer-quality`, `fee-market`, `job-queue`, `node-health`)
- [ ] Prometheus alerting rules fire correctly for simulated failure conditions
- [ ] DEX/AMM collector works when configured (optional — not required for base exit criteria)
- [ ] Phase 10 validation suite passes with receiver metrics included
- [ ] Receiver handles xrpld restart/unavailability gracefully (no crash, logs warning, retries)
- [ ] Documentation complete: receiver README, metric reference, alerting playbook
- [ ] Go receiver has unit tests with >80% coverage
- [ ] 18 Grafana alert rules for Phase 7+ parity metrics evaluate correctly (Task 11.12)
- [ ] The 14 not-yet-shipped Grafana alert rules for Phase 7+ parity metrics
evaluate correctly (Task 11.12); the other 4 of the 18 already ship in Phase 9
- [ ] Dual-datasource architecture documented with trade-offs (Task 11.13)

10
OpenTelemetryPlan/Phase2_taskList.md
View File

@@ -8,11 +8,11 @@
### Related Plan Documents
| Document | Relevance |
| ------------------------------------------------------------ | ------------------------------------------------------------- |
| [04-code-samples.md](./04-code-samples.md) | TraceContextPropagator (§4.4.2), RPC instrumentation (§4.5.3) |
| [02-design-decisions.md](./02-design-decisions.md) | W3C Trace Context (§2.5), span attributes (§2.4.2) |
| [06-implementation-phases.md](./06-implementation-phases.md) | Phase 2 tasks (§6.3), definition of done (§6.11.2) |
| Document | Relevance |
| ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| [03-implementation-strategy.md](./03-implementation-strategy.md) | Code structure and instrumentation patterns (replaces the deleted `04-code-samples.md` §4.4.2 / §4.5.3, removed by `d6450631bf`) |
| [02-design-decisions.md](./02-design-decisions.md) | W3C Trace Context (§2.5), span attributes (§2.4.2) |
| [06-implementation-phases.md](./06-implementation-phases.md) | Phase 2 tasks (§6.3), definition of done (§6.11.2) |
---

33
OpenTelemetryPlan/Phase3_taskList.md
View File

@@ -8,12 +8,12 @@
### Related Plan Documents
| Document | Relevance |
| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------ |
| [04-code-samples.md](./04-code-samples.md) | TraceContext protobuf (§4.4.1), PeerImp instrumentation (§4.5.1), context serialization (§4.4.2) |
| [01-architecture-analysis.md](./01-architecture-analysis.md) | Transaction flow (§1.3), key trace points (§1.6) |
| [06-implementation-phases.md](./06-implementation-phases.md) | Phase 3 tasks (§6.4), definition of done (§6.11.3) |
| [02-design-decisions.md](./02-design-decisions.md) | Context propagation design (§2.5), attribute schema (§2.4.3) |
| Document | Relevance |
| ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| [docs/telemetry-runbook.md § Protocol Span Flow](../docs/telemetry-runbook.md#protocol-span-flow) | Authoritative protocol span-flow reference — replaces the deleted `04-code-samples.md` (TraceContext protobuf §4.4.1, PeerImp instrumentation §4.5.1, context serialization §4.4.2), removed by `d6450631bf` |
| [01-architecture-analysis.md](./01-architecture-analysis.md) | Transaction flow (§1.3), key trace points (§1.6) |
| [06-implementation-phases.md](./06-implementation-phases.md) | Phase 3 tasks (§6.4), definition of done (§6.11.3) |
| [02-design-decisions.md](./02-design-decisions.md) | Context propagation design (§2.5), attribute schema (§2.4.3) |
---
@@ -47,7 +47,9 @@
**Reference**:
- [04-code-samples.md §4.4.1](./04-code-samples.md) — TraceContext message definition
- `04-code-samples.md` §4.4.1 (TraceContext message definition) was deleted by
`d6450631bf`; the live definition is `include/xrpl/proto/xrpl.proto:101`
(`message TraceContext`), attached as field `1001` on the relevant messages
- [02-design-decisions.md §2.5.2](./02-design-decisions.md) — Protocol buffer context propagation design
---
@@ -75,7 +77,13 @@
**Reference**:
- [04-code-samples.md §4.4.2](./04-code-samples.md) — Full extract/inject implementation
- `04-code-samples.md` §4.4.2 (full extract/inject implementation) was deleted by
`d6450631bf`. As shipped there is **no**
`src/libxrpl/telemetry/TraceContextPropagator.cpp`; extract/inject live on
`SpanGuard` (`include/xrpl/telemetry/SpanGuard.h:467` extract, `:480-491`
`injectCurrentContextToProtobuf`, implemented in
`src/libxrpl/telemetry/SpanGuard.cpp`) with the protocol-layer wrappers in
`src/xrpld/telemetry/PropagationHelpers.h:52` (`injectSpanContext`)
---
@@ -110,7 +118,9 @@
**Reference**:
- [04-code-samples.md §4.5.1](./04-code-samples.md) — Full PeerImp instrumentation example
- [docs/telemetry-runbook.md § Protocol Span Flow](../docs/telemetry-runbook.md#protocol-span-flow)
— the authoritative `tx.receive` / relay span-flow reference; replaces
`04-code-samples.md` §4.5.1, deleted by `d6450631bf`
- [01-architecture-analysis.md §1.3](./01-architecture-analysis.md) — Transaction flow diagram
- [01-architecture-analysis.md §1.6](./01-architecture-analysis.md) — tx.receive trace point
@@ -231,7 +241,10 @@ design.
**Reference**:
- [02-design-decisions.md §2.5](./02-design-decisions.md) — Context propagation design
- [04-code-samples.md §4.5.1](./04-code-samples.md) — Relay context injection pattern
- Relay context injection pattern: `04-code-samples.md` §4.5.1 was deleted by
`d6450631bf`; the live pattern is `injectSpanContext()` in
`src/xrpld/telemetry/PropagationHelpers.h:52`, with the flow documented in
[docs/telemetry-runbook.md § Protocol Span Flow](../docs/telemetry-runbook.md#protocol-span-flow)
---

210
OpenTelemetryPlan/Phase4_taskList.md
View File

@@ -6,21 +6,53 @@
>
> **Branch**: `pratik/otel-phase4-consensus-tracing` (from `pratik/otel-phase3-tx-tracing`)
> **Note on attribute names**: the `xrpl.<domain>.<field>` keys shown below are
> written in the older dotted form for readability — it mirrors how the fully
> qualified attribute reads in a Tempo trace view. The implemented keys follow
> the convention in [CONTRIBUTING.md](../CONTRIBUTING.md#telemetry-span-attribute-naming)
> (underscore form, e.g. `consensus_round`, `consensus_mode`); the
> **Note on attribute names**: the `xrpl.<domain>.<field>` keys that earlier
> revisions of this task list used were **never emitted**. `9e27120a15` removed
> the dotted `xrpl.*` namespace from **span** attributes repo-wide. Falsifiable
> check: `grep -rn 'seg::xrpl' src/ include/` → exactly **2** hits, both
> `include/xrpl/telemetry/SpanNames.h:117-118` (`attr::networkId`,
> `attr::networkType`), and both are **resource** attributes
> (`xrpl.network.id` / `xrpl.network.type`) set on the OTel resource at startup —
> the one place the dotted form is still reserved. No span attribute uses it.
> (Do **not** cite `grep 'makeStr("xrpl\.' src/ include/` → 0 hits as evidence:
> these keys were always composed with `join(seg::…, …)`, never that literal, so
> the grep has returned 0 for the entire history of the file and cannot fail.)
> Those spellings have been corrected in place, so every attribute key below is
> the live one. The mapping that was applied:
> `xrpl.ledger.seq` → `ledger_seq`, `xrpl.consensus.mode` → `consensus_mode`,
> `xrpl.consensus.round` → `consensus_round`,
> `xrpl.consensus.round_id` → `consensus_round_id`,
> `xrpl.consensus.ledger_id` → `consensus_ledger_id`,
> `xrpl.tx.id` → `tx_id`,
> `xrpl.validation.ledger_hash` / `xrpl.peer.validation.ledger_hash` → `ledger_hash`,
> `xrpl.validation.full` / `xrpl.peer.validation.full` → `full_validation`,
> `xrpl.peer.version` → `peer_version`.
> Separately, `19a6c2a306` split the single `trusted` key into
> `proposal_trusted` (on `consensus.proposal.receive` and `peer.proposal.receive`)
> and `validation_trusted` (on `consensus.validation.receive` and
> `peer.validation.receive`). Naming follows
> [CONTRIBUTING.md](../CONTRIBUTING.md#telemetry-span-attribute-naming); the
> `*SpanNames.h` constants are the single source of truth.
>
> **Three names in this document are not span attributes at all**:
>
> - `amendment_blocked` — a **metric label value** only:
> `validator_health{metric="amendment_blocked"}` (`MetricsRegistry.cpp:1216`).
> No span carries it.
> - `server_state` — a **metric label value** only:
> `server_info{metric="server_state"}` (`MetricsRegistry.cpp:1014`). It is also
> an RPC method name. No span carries it.
> - `proposers_validated` — **never implemented** on any span. `proposersValidated`
> exists only as a C++ function/parameter name (`RCLConsensus.cpp:310`);
> `consensus.accept` carries `proposers` instead (see Task 4.8).
### Related Plan Documents
| Document | Relevance |
| ------------------------------------------------------------ | ----------------------------------------------------------- |
| [04-code-samples.md](./04-code-samples.md) | Consensus instrumentation (§4.5.2), consensus span patterns |
| [01-architecture-analysis.md](./01-architecture-analysis.md) | Consensus round flow (§1.4), key trace points (§1.6) |
| [06-implementation-phases.md](./06-implementation-phases.md) | Phase 4 tasks (§6.5), definition of done (§6.11.4) |
| [02-design-decisions.md](./02-design-decisions.md) | Consensus attribute schema (§2.4.4) |
| Document | Relevance |
| ------------------------------------------------------------ | -------------------------------------------------------------------- |
| [01-architecture-analysis.md](./01-architecture-analysis.md) | Consensus round flow (§1.4), key trace points (§1.6) |
| [06-implementation-phases.md](./06-implementation-phases.md) | Phase 4 tasks and exit criteria (§6.5), definition of done (§6.12.4) |
| [02-design-decisions.md](./02-design-decisions.md) | Consensus attribute schema (§2.4.2 → "Consensus Attributes") |
---
@@ -34,8 +66,8 @@
- `RCLConsensus::Adaptor::startRoundTracing()` creates `consensus.round` span
via `SpanGuard::hashSpan()` (deterministic) or `SpanGuard::span()` (attribute strategy)
- Attributes set: `xrpl.consensus.ledger_id`, `xrpl.ledger.seq`,
`xrpl.consensus.mode`, `trace_strategy`, `xrpl.consensus.round_id`
- Attributes set: `consensus_ledger_id`, `ledger_seq`,
`consensus_mode`, `trace_strategy`, `consensus_round_id`
- Round span stored as `roundSpan_` member in `RCLConsensus::Adaptor`
- `roundSpanContext_` snapshot captured for cross-thread span linking
@@ -46,7 +78,9 @@
**Reference**:
- [04-code-samples.md §4.5.2](./04-code-samples.md) — startRound instrumentation example
- `RCLConsensus::Adaptor::startRoundTracing()` — the live startRound
instrumentation (the former `04-code-samples.md` §4.5.2 was deleted by
`d6450631bf`; the code is the reference now)
- [01-architecture-analysis.md §1.4](./01-architecture-analysis.md) — Consensus round flow
---
@@ -75,7 +109,8 @@
**Reference**:
- [04-code-samples.md §4.5.2](./04-code-samples.md) — phaseTransition instrumentation
- `Consensus.h` — the live phase-transition instrumentation (`04-code-samples.md`
was deleted by `d6450631bf`)
---
@@ -89,11 +124,13 @@
- In `Adaptor::propose()`:
- Creates `consensus.proposal.send` span via `SpanGuard::span()`
- Sets `xrpl.consensus.round` attribute
- Sets `consensus_round` attribute
- In `PeerImp::onMessage(TMProposeSet)`:
- Creates `consensus.proposal.receive` span
- Sets `trusted` attribute (bool)
- Sets `proposal_trusted` attribute (bool) — `PeerSpanNames.h:41`,
`ConsensusSpanNames.h:244`; renamed from the original `trusted` by
`19a6c2a306`, and the dotted `xrpl.peer.*` form was dropped by `9e27120a15`
**Done here** (cross-node propagation, send + receive):
@@ -112,8 +149,11 @@
**Reference**:
- [04-code-samples.md §4.5.2](./04-code-samples.md) — peerProposal instrumentation
- [02-design-decisions.md §2.4.4](./02-design-decisions.md) — Consensus attribute schema
- `PeerImp::onMessage(TMProposeSet)` — the live peerProposal instrumentation
(`04-code-samples.md` was deleted by `d6450631bf`)
- [02-design-decisions.md §2.4.2](./02-design-decisions.md) — Consensus attribute
schema (the "Consensus Attributes" table under "Span Attributes by Category";
§2.4.4 is the Privacy & Sensitive Data Policy, not the schema)
---
@@ -130,12 +170,14 @@
- Uses `SpanGuard::linkedSpan()` to create a follows-from link to the round span
- Thread-safe: uses `roundSpanContext_` snapshot (captured on consensus thread,
read on jtACCEPT thread)
- Sets `xrpl.ledger.seq` and `proposing` attributes
- Sets `ledger_seq` and `proposing` attributes
- In `PeerImp::onMessage(TMValidation)`:
- Creates `consensus.validation.receive` span
- Sets `trusted` attribute (bool)
- Sets `xrpl.ledger.seq` attribute
- Sets `validation_trusted` attribute (bool) — `PeerSpanNames.h:42`,
`ConsensusSpanNames.h:245`; renamed from the original `trusted` by
`19a6c2a306`, and the dotted `xrpl.peer.*` form was dropped by `9e27120a15`
- Sets `ledger_seq` attribute
**Not implemented** (deferred to Phase 4b — cross-node propagation):
@@ -155,9 +197,9 @@
**Implemented attributes** (across various spans):
- `xrpl.ledger.seq` — on `consensus.round`, `consensus.accept.apply`
- `xrpl.consensus.round` — on `consensus.proposal.send`
- `xrpl.consensus.mode` — on `consensus.round`, `consensus.ledger_close`
- `ledger_seq` — on `consensus.round`, `consensus.accept.apply`
- `consensus_round` — on `consensus.proposal.send`
- `consensus_mode` — on `consensus.round`, `consensus.ledger_close`
- `proposers` — on `consensus.accept`, `consensus.establish`, `consensus.update_positions`
- `converge_percent` — on `consensus.establish`, `consensus.update_positions`, `consensus.check`
- `tx_count` — on `consensus.accept.apply` span (in `doAccept()`)
@@ -185,7 +227,7 @@
- In `doAccept()` (RCLConsensus.cpp):
- Records `tx.included` events on the `consensus.accept.apply` span for each transaction in the accepted set
- Each event includes `xrpl.tx.id` attribute with the transaction hash
- Each event includes `tx_id` attribute with the transaction hash
- This links consensus traces to individual transactions
**Key modified files**:
@@ -225,54 +267,73 @@
**Objective**: Add ledger hash, validation type, and quorum data to consensus validation spans on both send and receive paths. This enables trace-level validation agreement analysis — filter by ledger hash to see which validators agreed for a given ledger.
**Status**: Not implemented. None of the enrichment attributes are set. The `consensus.validation.send` span only has `ledger.seq` and `proposing`. The `consensus.accept` span has `quorum` set to `result.proposers` (not the actual validator quorum from `app_.validators().quorum()`). No `PeerImp.cpp` changes were made.
**Status**: Implemented, except `proposers_validated`.
- `consensus.validation.send` sets `ledger_seq`, `ledger_hash`, `proposing` and
`full_validation` (`RCLConsensus.cpp:975-981`).
- `peer.validation.receive` sets `ledger_hash` and `full_validation`
(`PeerImp.cpp:2573-2574`).
- `consensus.accept` sets `quorum` from `app_.getValidators().quorum()`
(`RCLConsensus.cpp:516`) — the earlier defect where `quorum` carried
`result.proposers` instead of the real validator quorum is **fixed**.
- Still open: `proposers_validated` on `consensus.accept` — never implemented.
`consensus.accept` already carries `proposers` (`RCLConsensus.cpp:513`), so a
second key for the same value was not added.
All attribute keys are bare/underscore; the dotted `xrpl.*` forms in the spec
below were never emitted as **span** attributes. Check:
`grep -rn 'seg::xrpl' src/ include/` → 2 hits, both `SpanNames.h:117-118`
resource attributes (`xrpl.network.{id,type}`). See the note at the top of this
document for why the old `makeStr("xrpl\.` grep proved nothing.
**What to do**:
- Edit `src/xrpld/app/consensus/RCLConsensus.cpp`:
- On the `consensus.validation.send` span (in `validate()` / `doAccept()`):
- Add `xrpl.validation.ledger_hash` (string) — the ledger hash being validated
- Add `xrpl.validation.full` (bool) — whether this is a full validation (not partial)
- Add `ledger_hash` (string) — the ledger hash being validated
- Add `full_validation` (bool) — whether this is a full validation (not partial)
- On the `consensus.accept` span (in `onAccept()`):
- Add `validation_quorum` (int64) — from `app_.validators().quorum()`
- Add `proposers_validated` (int64) — from `result.proposers`
- Add `quorum` (int64) — from `app_.getValidators().quorum()` ✅ shipped
- Add `proposers_validated` (int64) — from `result.proposers` ❌ never
implemented; `proposers` already carries this value
- Edit `src/xrpld/overlay/detail/PeerImp.cpp`:
- On the `peer.validation.receive` span:
- Add `xrpl.peer.validation.ledger_hash` (string) — from deserialized `STValidation` object
- Add `xrpl.peer.validation.full` (bool) — from `STValidation` flags
- Add `ledger_hash` (string) — from deserialized `STValidation` object
- Add `full_validation` (bool) — from `STValidation` flags
**New span attributes**:
| Span | Attribute | Type | Source |
| --------------------------- | ---------------------------------- | ------ | --------------------------------- |
| `consensus.validation.send` | `xrpl.validation.ledger_hash` | string | Ledger hash from validate() args |
| `consensus.validation.send` | `xrpl.validation.full` | bool | Full vs partial validation |
| `peer.validation.receive` | `xrpl.peer.validation.ledger_hash` | string | From STValidation deserialization |
| `peer.validation.receive` | `xrpl.peer.validation.full` | bool | From STValidation flags |
| `consensus.accept` | `validation_quorum` | int64 | `app_.validators().quorum()` |
| `consensus.accept` | `proposers_validated` | int64 | `result.proposers` |
| Span | Attribute (live name) | Type | Source | Status |
| --------------------------- | --------------------- | ------ | --------------------------------- | ------------------------- |
| `consensus.validation.send` | `ledger_hash` | string | Ledger hash from validate() args | ✅ `RCLConsensus.cpp:977` |
| `consensus.validation.send` | `full_validation` | bool | Full vs partial validation | ✅ `RCLConsensus.cpp:981` |
| `peer.validation.receive` | `ledger_hash` | string | From STValidation deserialization | ✅ `PeerImp.cpp:2573` |
| `peer.validation.receive` | `full_validation` | bool | From STValidation flags | ✅ `PeerImp.cpp:2574` |
| `consensus.accept` | `quorum` | int64 | `app_.getValidators().quorum()` | ✅ `RCLConsensus.cpp:516` |
| `consensus.accept` | `proposers_validated` | int64 | `result.proposers` | ❌ never implemented |
**Rationale**: The external dashboard's most valuable feature is validation agreement tracking. By recording the ledger hash on both outgoing and incoming validation spans, we create the raw data for agreement analysis at the trace level. Example Tempo query:
```
{name="consensus.validation.send"} | xrpl.validation.ledger_hash = "A1B2C3..."
{name="consensus.validation.send" && span.ledger_hash = "A1B2C3..."}
```
Phase 7's `ValidationTracker` builds metric-level aggregation (1h/24h agreement %) on top of this data.
**Key modified files (not yet modified)**:
**Key modified files**:
- `src/xrpld/app/consensus/RCLConsensus.cpp`
- `src/xrpld/overlay/detail/PeerImp.cpp`
- `src/xrpld/app/consensus/RCLConsensus.cpp` (`:516`, `:975-981`)
- `src/xrpld/overlay/detail/PeerImp.cpp` (`:2573-2574`)
**Exit Criteria**:
- [x] `consensus.validation.send` spans carry `ledger_hash` and `full_validation`
- [ ] `peer.validation.receive` spans carry `xrpl.peer.validation.ledger_hash` and `xrpl.peer.validation.full`
- [ ] `consensus.accept` spans carry `validation_quorum` and `proposers_validated`
- [x] `peer.validation.receive` spans carry `ledger_hash` and `full_validation` — `PeerImp.cpp:2573-2574`
- [x] `consensus.accept` spans carry `quorum` — `RCLConsensus.cpp:516`
- [ ] `consensus.accept` spans carry `proposers_validated` — **open**, never implemented
- [x] Ledger hash attributes match between send and receive for the same ledger
- [ ] No impact on consensus performance
- [ ] No impact on consensus performance — not measured
---
@@ -318,13 +379,13 @@ Phase 7's `ValidationTracker` builds metric-level aggregation (1h/24h agreement
### Implemented Spans
| Span Name | Method | Key Attributes |
| --------------------------- | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `consensus.proposal.send` | `Adaptor::propose` | `xrpl.consensus.round`, `is_bow_out` |
| `consensus.ledger_close` | `Adaptor::onClose` | `xrpl.ledger.seq`, `xrpl.consensus.mode` |
| `consensus.accept` | `Adaptor::onAccept` | `proposers`, `round_time_ms`, `quorum`, `disputes_count`, `consensus_state` |
| `consensus.accept.apply` | `Adaptor::doAccept` | `close_time`, `close_time_correct`, `close_resolution_ms`, `consensus_state`, `proposing`, `round_time_ms`, `xrpl.ledger.seq`, `parent_close_time`, `close_time_self`, `close_time_vote_bins`, `resolution_direction` |
| `consensus.validation.send` | `Adaptor::onAccept` (via validate) | `proposing`, `ledger_hash`, `ledger_seq`, `full_validation`, `validation_sign_time` |
| Span Name | Method | Key Attributes |
| --------------------------- | ---------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `consensus.proposal.send` | `Adaptor::propose` | `consensus_round`, `is_bow_out` |
| `consensus.ledger_close` | `Adaptor::onClose` | `ledger_seq`, `consensus_mode` |
| `consensus.accept` | `Adaptor::onAccept` | `proposers`, `round_time_ms`, `quorum`, `disputes_count`, `consensus_state` |
| `consensus.accept.apply` | `Adaptor::doAccept` | `close_time`, `close_time_correct`, `close_resolution_ms`, `consensus_state`, `proposing`, `round_time_ms`, `ledger_seq`, `parent_close_time`, `close_time_self`, `close_time_vote_bins`, `resolution_direction` |
| `consensus.validation.send` | `Adaptor::onAccept` (via validate) | `proposing`, `ledger_hash`, `ledger_seq`, `full_validation`, `validation_sign_time` |
#### Close Time Attributes (consensus.accept.apply)
@@ -342,13 +403,15 @@ driven by `avCT_CONSENSUS_PCT` (75% validator agreement threshold):
- **`close_time_vote_bins`** — Number of distinct close-time vote bins from peer proposals. Higher values indicate less agreement among validators.
- **`resolution_direction`** — Whether close-time resolution `"increased"` (coarser), `"decreased"` (finer), or stayed `"unchanged"` relative to the previous ledger.
**Exit Criteria** (from [06-implementation-phases.md §6.11.4](./06-implementation-phases.md)):
**Exit Criteria** (from [06-implementation-phases.md §6.5](./06-implementation-phases.md)
— §6.11.4 is the WALK phase, i.e. transaction tracing, and does not carry these
criteria; the Phase 4 definition of done is §6.12.4):
- [x] Complete consensus round traces
- [x] Phase transitions visible (open, establish, close, accept)
- [x] Proposals and validations traced — send and receive; relay deferred to Phase 4b
- [x] Close time agreement tracked (per `avCT_CONSENSUS_PCT`)
- [x] No impact on consensus timing
- [ ] No impact on consensus timing — **not measured**
- [x] Transaction-consensus correlation (Task 4.6) — `tx.included` events in doAccept
- [ ] Validation span enrichment (Task 4.8) — not implemented
@@ -386,7 +449,7 @@ consensus round share the same trace_id without P2P context propagation.
### Strategy B — Attribute-Based Correlation
Use normal random trace_id but attach `xrpl.consensus.ledger_id` as an attribute
Use normal random trace_id but attach `consensus_ledger_id` as an attribute
on every consensus span. Correlation happens at query time via Tempo/Grafana
`by attribute` queries.
@@ -423,10 +486,10 @@ In `RCLConsensus::Adaptor::startRound()`:
5. Call `startSpan("consensus.round", parentContext)` so the new span
inherits the deterministic trace_id.
- If `attribute`: start a normal `consensus.round` span, set
`xrpl.consensus.ledger_id = previousLedger.id()` as attribute.
`consensus_ledger_id = previousLedger.id()` as attribute.
Both strategies always set `xrpl.consensus.round_id` (round number) and
`xrpl.consensus.ledger_id` (previous ledger hash) as attributes.
Both strategies always set `consensus_round_id` (round number) and
`consensus_ledger_id` (previous ledger hash) as attributes.
---
@@ -542,7 +605,7 @@ spans in `Consensus.h`.
- Reads `consensus_trace_strategy` via `app_.getTelemetry().getConsensusTraceStrategy()`
- **Deterministic**: uses `SpanGuard::hashSpan()` with `prevLgr.id()` data
- **Attribute**: uses `SpanGuard::span(TraceCategory::Consensus, seg::consensus, "round")`
- Sets attributes: `xrpl.consensus.ledger_id`, `xrpl.ledger.seq`, `xrpl.consensus.mode`, `trace_strategy`, `xrpl.consensus.round_id`
- Sets attributes: `consensus_ledger_id`, `ledger_seq`, `consensus_mode`, `trace_strategy`, `consensus_round_id`
- Captures `roundSpanContext_` snapshot for cross-thread span linking
- Saves `prevRoundContext_` from previous round for follows-from links
@@ -811,7 +874,7 @@ and OFF, and don't affect consensus timing.
| Span Name | Location | Key Attributes (actually set) |
| ---------------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------- |
| `consensus.round` | `RCLConsensus.cpp` | `xrpl.consensus.round_id`, `xrpl.consensus.ledger_id`, `xrpl.ledger.seq`, `xrpl.consensus.mode`, `trace_strategy` |
| `consensus.round` | `RCLConsensus.cpp` | `consensus_round_id`, `consensus_ledger_id`, `ledger_seq`, `consensus_mode`, `trace_strategy` |
| `consensus.establish` | `Consensus.h` | `converge_percent`, `establish_count`, `proposers` |
| `consensus.update_positions` | `Consensus.h` | `converge_percent`, `proposers`, `have_close_time_consensus`, `close_time_threshold`, `disputes_count`, `avalanche_threshold` |
| `consensus.check` | `Consensus.h` | `agree_count`, `disagree_count`, `converge_percent`, `have_close_time_consensus`, `threshold_percent`, `consensus_result` |
@@ -819,17 +882,17 @@ and OFF, and don't affect consensus timing.
### New Events (Phase 4a)
| Event Name | Parent Span | Attributes (actually set) |
| ----------------- | ---------------------------- | ---------------------------------------------------------------- |
| `dispute.resolve` | `consensus.update_positions` | `xrpl.tx.id`, `dispute_our_vote`, `dispute_yays`, `dispute_nays` |
| `tx.included` | `consensus.accept.apply` | `xrpl.tx.id` |
| Event Name | Parent Span | Attributes (actually set) |
| ----------------- | ---------------------------- | ----------------------------------------------------------- |
| `dispute.resolve` | `consensus.update_positions` | `tx_id`, `dispute_our_vote`, `dispute_yays`, `dispute_nays` |
| `tx.included` | `consensus.accept.apply` | `tx_id` |
### New Attributes (Phase 4a)
```cpp
// Round-level (on consensus.round) — ALL IMPLEMENTED
"xrpl.consensus.round_id" = int64 // Consensus round number
"xrpl.consensus.ledger_id" = string // previousLedger.id() hash
"consensus_round_id" = int64 // Consensus round number
"consensus_ledger_id" = string // previousLedger.id() hash
"trace_strategy" = string // "deterministic" or "attribute"
// Establish-level — IMPLEMENTED
@@ -877,9 +940,12 @@ and OFF, and don't affect consensus timing.
- **No `getTelemetry()` adaptor method**: `SpanGuard::span()` is a static factory that
internally checks telemetry state, so `Consensus.h` doesn't need adaptor access
for span creation. Only `RCLConsensus::Adaptor` accesses `app_.getTelemetry()` directly.
- **Config validation**: `consensus_trace_strategy` is validated to be either
`"deterministic"` or `"attribute"`, falling back to `"deterministic"` for
unrecognised values.
- **No config validation**: `consensus_trace_strategy` is **not** validated.
`TelemetryConfig.cpp:155-156` copies the raw string through, and the only
comparison in the code is `strategy == "attribute"` (`RCLConsensus.cpp:1296`).
Any unrecognised value — including a typo — silently takes the deterministic
branch, with no log warning. The effective fallback is correct; the absence of
a diagnostic is a known gap.
- **Plan deviation**: `roundSpan_` is stored in `RCLConsensus::Adaptor` (not
`Consensus.h`) because the adaptor has access to telemetry config and can
implement the deterministic trace ID strategy. `establishSpan_` is correctly

3
OpenTelemetryPlan/Phase7_taskList.md
View File

@@ -75,7 +75,8 @@
- Match existing telemetry code style from `src/libxrpl/telemetry/Telemetry.cpp`
- Use RAII for MeterProvider lifecycle (shutdown on destructor)
**Reference**: [04-code-samples.md](./04-code-samples.md) — code style and patterns
**Reference**: [03-implementation-strategy.md](./03-implementation-strategy.md) —
code style and patterns (`04-code-samples.md` was deleted by `d6450631bf`)
---

565
OpenTelemetryPlan/Phase9_taskList.md
View File

@@ -1,6 +1,11 @@
<!-- cspell:ignore ISTOGRAM -->
<!-- The all-caps macro name XRPL_METRIC_HISTOGRAM_RECORD trips cspell's
compound-word splitter, which emits the subword "ISTOGRAM"; ignore it here. -->
# Phase 9: Internal Metric Instrumentation Gap Fill — Task List
> **Status**: Future Enhancement
> **Status**: Complete for Tasks 9.1-9.13. Tasks 9.14-9.17 remain open by design
> (see each task for the blocker).
>
> **Goal**: Instrument xrpld to emit ~50+ metrics that exist in `get_counts`/`server_info`/TxQ/PerfLog but currently lack time-series export via the OTel or beast::insight pipelines.
>
@@ -10,6 +15,36 @@
>
> **Depends on**: Phase 7 (native OTel metrics pipeline) and Phase 8 (log-trace correlation)
> **Note on metric names**: there is **no `xrpld_` prefix** on any emitted
> metric. `77f35c03db` removed it and lowercased names, and
> `OTelCollectorImp::formatName()`
> (`src/libxrpl/beast/insight/OTelCollector.cpp:855-874`) adds no prefix at all —
> it only lowercases the raw name and turns `.` and spaces into `_`. Earlier
> revisions of this task list spelled every metric `xrpld_<name>`; those spellings
> have been corrected in place to the emitted names, so the names below can be
> pasted into Prometheus as written. Instruments created in
> `src/xrpld/telemetry/MetricsRegistry.cpp` (35 of them) are the single source of
> truth. `MetricsRegistry.h`'s Doxygen used to disagree on three histogram names;
> those header comments were repaired in this change set (see Tasks 9.4 and 9.5),
> so header and `.cpp` now agree.
>
> **Two shapes do not simply lose the prefix**, so `xrpld_<name>` → `<name>` is
> not a blanket rule:
>
> - **Multiplexed observable gauges.** Most of the value names in these task
> descriptions are a **`metric` label value** on a shared instrument, not a
> standalone metric name — queue depth is `txq_metrics{metric="txq_count"}`, not
> `txq_count`. The same applies to `nodestore_state`, `cache_metrics`,
> `load_factor_metrics`, `server_info`, `db_metrics`, `validator_health`,
> `peer_quality`, `state_tracking` and `ledger_economy`. Each task below names
> its owning instrument.
> - **Unit-suffixed histograms** coming through `beast::insight`.
> `OTelCollectorImp` appends the unit to the name, so the `ios_latency`
> histogram is `ios_latency_milliseconds_bucket` in Prometheus — not
> `ios_latency_bucket`. Instruments created directly on `MetricsRegistry` keep
> their literal name (`job_queued_us_bucket`, `rpc_method_us_bucket`) because
> the unit is already in the instrument name.
### Related Plan Documents
| Document | Relevance |
@@ -40,7 +75,16 @@ These metrics serve multiple external consumer categories identified during rese
**What to do**:
- In `src/libxrpl/nodestore/Database.cpp`, extend existing `beast::insight` registrations to add:
> **As shipped, this did _not_ go through `beast::insight`.** `Database.cpp` has
> no insight members. The metrics are a single `nodestore_state`
> `Int64ObservableGauge` on `MetricsRegistry`
> (`src/xrpld/telemetry/MetricsRegistry.cpp:957-965`) whose callback reads
> `Database`'s public accessors (`getFetchTotalCount()`, `getFetchHitCount()`,
> `getStoreCount()`, `getFetchDurationUs()`, `getStoreDurationUs()`, …) and
> multiplexes every value onto the `metric` label. Write-queue depth comes from
> the new `include/xrpl/nodestore/WriteStats.h`.
- Export the following as `nodestore_state{metric="…"}` label values:
- Gauge: `node_reads_total` (cumulative read operations)
- Gauge: `node_reads_hit` (fetches that found an object — not a cache hit; `fetchHitCount_` increments whatever served the fetch)
- Gauge: `node_writes` (cumulative write operations)
@@ -50,14 +94,18 @@ These metrics serve multiple external consumer categories identified during rese
- Gauge: `write_load` (current write load score)
- Gauge: `read_queue` (items in read queue)
- These values are already computed in `Database::getCountsJson()` (line ~236). Wire the same counters to `beast::insight` hooks.
- These values are already computed in `Database::getCountsJson()`. The gauge
callback reads the same counters through `Database`'s public accessors.
**Key modified files**:
- `src/libxrpl/nodestore/Database.cpp`
- `src/libxrpl/nodestore/Database.h` (add insight members)
- `src/xrpld/telemetry/MetricsRegistry.cpp` (the `nodestore_state` gauge)
- `include/xrpl/nodestore/Database.h` (accessors; **not** `src/libxrpl/nodestore/Database.h`, which does not exist)
- `include/xrpl/nodestore/WriteStats.h` (new — write-queue depth snapshot)
**Derived Prometheus metrics**: `xrpld_nodestore_reads_total`, `xrpld_nodestore_reads_hit`, `xrpld_nodestore_write_load`, etc.
**Derived Prometheus metrics**: `nodestore_state{metric="node_reads_total"}`,
`nodestore_state{metric="node_reads_hit"}`, `nodestore_state{metric="write_load"}`,
etc. There is **no** `xrpld_` prefix — `OTelCollectorImp::formatName()` adds none.
**Grafana dashboard**: Add "NodeStore I/O" panel group to _Node Health_ dashboard.
@@ -77,17 +125,22 @@ These metrics serve multiple external consumer categories identified during rese
- `treenode_track_size` — Tracked tree nodes
- `fullbelow_size` — FullBelow cache size
- The callback should read from the same sources as `GetCounts.cpp` handler (line ~43).
- The callback reads from the same sources as the `GetCounts` handler
(`src/xrpld/rpc/handlers/admin/status/GetCounts.cpp` — **not**
`src/xrpld/rpc/handlers/GetCounts.cpp`).
- Create a centralized `MetricsRegistry` class that holds all OTel async gauge registrations, polled at 10-second intervals by the `PeriodicMetricReader`.
**Key modified files**:
- New: `src/xrpld/telemetry/MetricsRegistry.h` / `.cpp`
- `src/xrpld/rpc/handlers/GetCounts.cpp` (extract shared access methods)
- New: `src/xrpld/telemetry/MetricMacros.h` (the `XRPL_METRIC_*` call-site macros)
- `src/xrpld/rpc/handlers/admin/status/GetCounts.cpp` (extract shared access methods)
- `src/xrpld/app/main/Application.cpp` (register MetricsRegistry at startup)
**Derived Prometheus metrics**: `xrpld_cache_SLE_hit_rate`, `xrpld_cache_ledger_hit_rate`, `xrpld_cache_treenode_size`, etc.
**Derived Prometheus metrics**: `cache_metrics{metric="SLE_hit_rate"}`,
`cache_metrics{metric="ledger_hit_rate"}`, `cache_metrics{metric="treenode_cache_size"}`,
etc. Label values are **case-sensitive** (`SLE_hit_rate`, `AL_size`, `AL_hit_rate`).
---
@@ -97,7 +150,8 @@ These metrics serve multiple external consumer categories identified during rese
**What to do**:
- Register OTel `ObservableGauge` callbacks for TxQ state (from `TxQ.h` line ~143):
- Register OTel `ObservableGauge` callbacks for TxQ state (from
`src/xrpld/app/misc/TxQ.h` — **not** `src/xrpld/app/tx/detail/TxQ.h`):
- `txq_count` — Current transactions in queue
- `txq_max_size` — Maximum queue capacity
- `txq_in_ledger` — Transactions in current open ledger
@@ -112,9 +166,12 @@ These metrics serve multiple external consumer categories identified during rese
**Key modified files**:
- `src/xrpld/telemetry/MetricsRegistry.cpp` (add TxQ callbacks)
- `src/xrpld/app/tx/detail/TxQ.h` (expose metrics accessor if needed)
- `src/xrpld/app/misc/TxQ.h` (expose metrics accessor if needed)
**Derived Prometheus metrics**: `xrpld_txq_count`, `xrpld_txq_max_size`, `xrpld_txq_open_ledger_fee_level`, etc.
**Derived Prometheus metrics**: `txq_metrics{metric="txq_count"}`,
`txq_metrics{metric="txq_max_size"}`, `txq_metrics{metric="txq_open_ledger_fee_level"}`, etc.
There is one instrument, `txq_metrics` (`MetricsRegistry.cpp:705`); each value above
is a `metric` label value, not a metric name of its own.
**Grafana dashboard**: New _Fee Market & TxQ_ dashboard (`fee-market`).
@@ -126,13 +183,25 @@ These metrics serve multiple external consumer categories identified during rese
**What to do**:
- Register OTel instruments for PerfLog RPC counters (from `PerfLogImp.cpp` line ~63):
- Counter: `xrpld_rpc_method_started_total{method="<name>"}` — calls started
- Counter: `xrpld_rpc_method_finished_total{method="<name>"}` — calls completed
- Counter: `xrpld_rpc_method_errored_total{method="<name>"}` — calls errored
- Histogram: `xrpld_rpc_method_duration_us{method="<name>"}` — execution time distribution
- Register OTel instruments for PerfLog RPC counters (from `PerfLogImp.cpp`):
- Counter: `rpc_method_started_total{method="<name>"}` — calls started
- Counter: `rpc_method_finished_total{method="<name>"}` — calls completed
- Counter: `rpc_method_errored_total{method="<name>"}` — calls errored
- Histogram: `rpc_method_us{method="<name>"}` — execution time distribution
- Use OTel `Counter<int64_t>` and `Histogram<double>` instruments with `method` attribute label.
- Use OTel `Counter<uint64_t>` and `Histogram<double>` instruments with the
`method` attribute label. The RPC instruments carry **only** `method`
(`MetricsRegistry.cpp:436-475`) — the `handler` label belongs to the job
instruments (Task 9.5), not these.
> **Naming**: the instrument is `rpc_method_us` — declared as
> `kRpcMethodDurationUs` at `MetricsRegistry.cpp:96` and used both to register the
> explicit-bucket view and to create the instrument. `MetricsRegistry.h`'s Doxygen
> comment used to read `rpc_method_duration_us`; **that was fixed in this change**
> (`MetricsRegistry.h:789`), so header and `.cpp` now agree and there is no
> caveat left. The prefix `xrpld_` in the original spec is not emitted by anything.
>
> Same for the job histograms in Task 9.5: `job_queued_us` / `job_running_us`.
- Hook into the existing PerfLog callback mechanism rather than adding new instrumentation points.
@@ -141,7 +210,7 @@ These metrics serve multiple external consumer categories identified during rese
- `src/xrpld/perflog/detail/PerfLogImp.cpp` (add OTel instrument updates alongside existing JSON counters)
- `src/xrpld/telemetry/MetricsRegistry.cpp` (register instruments)
**Derived Prometheus metrics**: `xrpld_rpc_method_started_total{method="server_info"}`, `xrpld_rpc_method_duration_us_bucket{method="ledger"}`, etc.
**Derived Prometheus metrics**: `rpc_method_started_total{method="server_info"}`, `rpc_method_us_bucket{method="ledger"}`, etc.
**Grafana dashboard**: Add "Per-Method RPC Breakdown" panel group to _RPC Performance_ dashboard.
@@ -153,12 +222,24 @@ These metrics serve multiple external consumer categories identified during rese
**What to do**:
- Register OTel instruments for PerfLog job counters:
- Counter: `xrpld_job_queued_total{job_type="<name>"}` — jobs queued
- Counter: `xrpld_job_started_total{job_type="<name>"}` — jobs started
- Counter: `xrpld_job_finished_total{job_type="<name>"}` — jobs completed
- Histogram: `xrpld_job_queued_duration_us{job_type="<name>"}` — time spent waiting in queue
- Histogram: `xrpld_job_running_duration_us{job_type="<name>"}` — execution time distribution
- Register OTel instruments for PerfLog job counters. All five carry **two**
labels — `job_type` and `handler` — so producers sharing a job type stay
distinguishable (`MetricsRegistry.h:794-818`, recorded at
`MetricsRegistry.cpp:498,518,527,548,553`). `handler` is the sanitised
`addJob` name; `sanitiseHandler()` folds dynamic names into a bounded domain
of exactly 44 values, so cardinality stays fixed.
- Counter: `job_queued_total{job_type="<name>",handler="<name>"}` — jobs queued
- Counter: `job_started_total{job_type="<name>",handler="<name>"}` — jobs started
- Counter: `job_finished_total{job_type="<name>",handler="<name>"}` — jobs completed
- Histogram: `job_queued_us{job_type="<name>",handler="<name>"}` — time spent waiting in queue
- Histogram: `job_running_us{job_type="<name>",handler="<name>"}` — execution time distribution
> **Naming**: the instruments are `job_queued_us` / `job_running_us`
> (`kJobQueuedDurationUs` / `kJobRunningDurationUs`, `MetricsRegistry.cpp:94-95`).
> `MetricsRegistry.h`'s Doxygen comments used to read
> `job_queued_duration_us` / `job_running_duration_us`; **both were fixed in this
> change** (`MetricsRegistry.h:810,815`), so there is no header/`.cpp` divergence
> left to work around.
- Hook into PerfLog's existing job tracking alongside Task 9.4.
@@ -167,7 +248,7 @@ These metrics serve multiple external consumer categories identified during rese
- `src/xrpld/perflog/detail/PerfLogImp.cpp`
- `src/xrpld/telemetry/MetricsRegistry.cpp`
**Derived Prometheus metrics**: `xrpld_job_queued_total{job_type="ledgerData"}`, `xrpld_job_running_duration_us_bucket{job_type="transaction"}`, etc.
**Derived Prometheus metrics**: `job_queued_total{job_type="ledgerData",handler="ProcessLData"}`, `job_running_us_bucket{job_type="transaction",handler="…"}`, etc.
**Grafana dashboard**: New _Job Queue Analysis_ dashboard (`job-queue`).
@@ -180,15 +261,16 @@ These metrics serve multiple external consumer categories identified during rese
**What to do**:
- Register OTel `ObservableGauge` callbacks for `CountedObject<T>` instance counts:
- `xrpld_object_count{type="Transaction"}` — live Transaction objects
- `xrpld_object_count{type="Ledger"}` — live Ledger objects
- `xrpld_object_count{type="NodeObject"}` — live NodeObject instances
- `xrpld_object_count{type="STTx"}` — serialized transaction objects
- `xrpld_object_count{type="STLedgerEntry"}` — serialized ledger entries
- `xrpld_object_count{type="InboundLedger"}` — ledgers being fetched
- `xrpld_object_count{type="Pathfinder"}` — active pathfinding computations
- `xrpld_object_count{type="PathRequest"}` — active path requests
- `xrpld_object_count{type="HashRouterEntry"}` — hash router entries
- `object_count{type="xrpl::Transaction"}` — live Transaction objects
- `object_count{type="xrpl::Ledger"}` — live Ledger objects
- `object_count{type="xrpl::NodeObject"}` — live NodeObject instances
- `object_count{type="xrpl::STTx"}` — serialized transaction objects
- `object_count{type="xrpl::STLedgerEntry"}` — serialized ledger entries
- `object_count{type="xrpl::InboundLedger"}` — ledgers being fetched
- `object_count{type="xrpl::Pathfinder"}` — active pathfinding computations
- `object_count{type="xrpl::PathRequest"}` — active path requests
- `object_count{type="xrpl::HashRouter::Entry"}` — hash router entries (the type is
`HashRouter::Entry`; there is no `HashRouterEntry` type)
- The `CountedObject` template already tracks these via atomic counters. The callback just reads the current counts.
@@ -197,7 +279,9 @@ These metrics serve multiple external consumer categories identified during rese
- `src/xrpld/telemetry/MetricsRegistry.cpp` (add counted object callbacks)
- `include/xrpl/basics/CountedObject.h` (may need static accessor for iteration)
**Derived Prometheus metrics**: `xrpld_object_count{type="Transaction"}`, `xrpld_object_count{type="NodeObject"}`, etc.
**Derived Prometheus metrics**: `object_count{type="xrpl::Transaction"}`, `object_count{type="xrpl::NodeObject"}`, etc.
The `type` label value is `beast::typeName<Object>()` — the fully-qualified
demangled C++ type name (`CountedObject.h:109`), not a short word.
**Grafana dashboard**: Add "Object Instance Counts" panel to _Node Health_ dashboard.
@@ -225,7 +309,10 @@ These metrics serve multiple external consumer categories identified during rese
- `src/xrpld/telemetry/MetricsRegistry.cpp`
- `src/xrpld/app/misc/NetworkOPs.cpp` (expose load factor accessors if needed)
**Derived Prometheus metrics**: `xrpld_load_factor`, `xrpld_load_factor_fee_escalation`, etc.
**Derived Prometheus metrics**: `load_factor_metrics{metric="load_factor"}`,
`load_factor_metrics{metric="load_factor_fee_escalation"}`, etc. There is one
instrument, `load_factor_metrics` (`MetricsRegistry.cpp:785`); every value listed
above is a `metric` label value, not a metric name of its own.
**Grafana dashboard**: Add "Load Factor Breakdown" panel to _Fee Market & TxQ_ dashboard.
@@ -243,7 +330,7 @@ These metrics serve multiple external consumer categories identified during rese
- `read_request_bundle` (native JSON int)
- `read_threads_running` (native JSON int)
- `read_threads_total` (native JSON int)
- Added new `xrpld_server_info` Int64ObservableGauge with 8 metrics:
- Added new `server_info` Int64ObservableGauge with 8 metrics:
- `server_state` — operating mode as int (0=DISCONNECTED .. 4=FULL)
- `uptime` — seconds since server start
- `peers` — total peer count
@@ -252,9 +339,9 @@ These metrics serve multiple external consumer categories identified during rese
- `peer_disconnects_resources` — cumulative resource-related disconnects
- `last_close_proposers` — from `getConsensusInfo()["previous_proposers"]`
- `last_close_converge_time_ms` — from `getConsensusInfo()["previous_mseconds"]`
- Added new `xrpld_build_info` Int64ObservableGauge (info-style, value=1 with `version` label)
- Added new `xrpld_complete_ledgers` Int64ObservableGauge parsing comma-separated ranges into `{bound, index}` pairs
- Added new `xrpld_db_metrics` Int64ObservableGauge with 4 metrics:
- Added new `build_info` Int64ObservableGauge (info-style, value=1 with `version` label)
- Added new `complete_ledgers` Int64ObservableGauge parsing comma-separated ranges into `{bound, index}` pairs
- Added new `db_metrics` Int64ObservableGauge with 4 metrics:
- `db_kb_total`, `db_kb_ledger`, `db_kb_transaction` (SQLite stat queries)
- `historical_perminute` (historical ledger fetch rate)
@@ -267,7 +354,7 @@ These metrics serve multiple external consumer categories identified during rese
- `connection_count_51233/51234` — OS-level port connection counts from external shell script (`get_connection.sh`)
**Derived Prometheus metrics**: `xrpld_server_info{metric="server_state"}`, `xrpld_build_info{version="2.4.0"}`, `xrpld_complete_ledgers{bound="start",index="0"}`, `xrpld_db_metrics{metric="db_kb_total"}`, etc.
**Derived Prometheus metrics**: `server_info{metric="server_state"}`, `build_info{version="2.4.0"}`, `complete_ledgers{bound="start",index="0"}`, `db_metrics{metric="db_kb_total"}`, etc.
**Grafana dashboard**: New panels added to _Node Health_ dashboard (`node-health.json`).
@@ -284,15 +371,19 @@ These metrics serve multiple external consumer categories identified during rese
2. **Job Queue Analysis** (`job-queue`) — Per-job-type rates, queue wait times, execution times, job queue depth
- Update 2 existing dashboards:
1. **Node Health** (`xrpld-statsd-node-health`) — Add NodeStore I/O panels, cache hit rate panels, object instance counts
1. **Node Health** (`node-health`) — Add NodeStore I/O panels, cache hit rate panels, object instance counts
2. **RPC Performance** (`rpc-performance`) — Add per-method RPC breakdown panels
**Key modified files**:
> Tasks 9.11-9.13 add two more new dashboards (`validator-health`,
> `peer-quality`), so Phase 9's total is **4 new + 2 updated**.
- New: `docker/telemetry/grafana/dashboards/rippled-fee-market.json`
- New: `docker/telemetry/grafana/dashboards/rippled-job-queue.json`
- `docker/telemetry/grafana/dashboards/rippled-statsd-node-health.json`
- `docker/telemetry/grafana/dashboards/rippled-rpc-perf.json`
**Key modified files** (filenames and uids after the `rippled-*` → bare rename
in `145b1469d6` and `25868f2740` — the `rippled-*.json` paths no longer exist):
- New: `docker/telemetry/grafana/dashboards/fee-market.json` (uid `fee-market`)
- New: `docker/telemetry/grafana/dashboards/job-queue.json` (uid `job-queue`)
- `docker/telemetry/grafana/dashboards/node-health.json` (uid `node-health`)
- `docker/telemetry/grafana/dashboards/rpc-performance.json` (uid `rpc-performance`)
---
@@ -302,18 +393,37 @@ These metrics serve multiple external consumer categories identified during rese
**What to do**:
- Update `OpenTelemetryPlan/09-data-collection-reference.md`:
- Add new section for OTel SDK-exported metrics (NodeStore, cache, TxQ, PerfLog, CountedObjects, load factors)
- Update Grafana dashboard reference table (add 2 new dashboards)
- Update `OpenTelemetryPlan/09-data-collection-reference.md`: ✅ done
- Add new section for OTel SDK-exported metrics (NodeStore, cache, TxQ, PerfLog, CountedObjects, load factors) — §5b + "Phase 9: OTel SDK-Exported Metrics (MetricsRegistry)"
- Update Grafana dashboard reference table (add 4 new dashboards) — "New Grafana Dashboards (Phase 9)" / "Updated Grafana Dashboards (Phase 9)"
- Add Prometheus query examples for new metrics
- Update `docs/telemetry-runbook.md`:
- Add an Alerting section covering the provisioned rules and how to wire a receiver
- Add troubleshooting entries for new metric categories
- ✅ Alerting section covering the provisioned rules and how to wire a receiver
- ✅ Troubleshooting entries for new metric categories
- ❌ **Still open**: dashboard guides for **six** dashboards — `fee-market`,
`job-queue`, `ledger-data-sync`, `overlay-traffic-detail`, `peer-quality` and
`validator-health`. The runbook's dashboard reference records the gap
verbatim: "Nine dashboards have a reference section below. `fee-market`,
`job-queue`, `ledger-data-sync`, `overlay-traffic-detail`, `peer-quality`, and
`validator-health` are provisioned but not yet documented here — their panel
descriptions carry the same six-heading reference format, so open the panel
info icon in Grafana until a section is written." (15 provisioned − 6
undocumented = 9 documented.) Also still open: the Validation Agreement
explainer (8s grace / 5m late repair)
- Provision Grafana alert rules (`docker/telemetry/grafana/provisioning/alerting/`):
- 6 rules in 3 groups — consensus/ledger (`LedgerHistoryMismatch`, `LedgerCloseStalled`), validator (`ValidationsMissed`, `ValidationsNotChecked`), job queue (`JobQueueTxOverflow`, `JobQueueLatencyHigh`)
- `xrpld-default` webhook contact point + flat notification policy; auto-loaded via the existing `provisioning/` mount (no docker-compose change)
- Provision Grafana alert rules (`docker/telemetry/grafana/provisioning/alerting/`) — **as shipped**:
- **13 rules in 5 groups**: `xrpld-consensus` (`LedgerHistoryMismatch`,
`LedgerCloseStalled`, `ValidatedLedgerStale`), `xrpld-validator`
(`ValidationsMissed`, `ValidationsNotChecked`), `xrpld-jobqueue`
(`JobQueueTxOverflow`, `JobQueueLatencyHigh`, `NodeStoreIOLatencyHigh`),
`xrpld-node-state` (`NodeStateFlapping`, `NodeNotFull`), `xrpld-overlay`
(`ManifestJobQueueConvoy`, `ManifestFloodInbound`, `PeerResourceDisconnects`)
- **2 contact points** — `xrpld-default` (Slack) and `xrpld-critical`
(Slack + email) — and a **nested** notification policy: root →
`xrpld-default`, child route `severity = critical` → `xrpld-critical`.
Auto-loaded via the existing `provisioning/` mount (no docker-compose change)
- 3 rules are `severity: critical`, 10 are `severity: warning`
- Alerting operator docs (per-alert meaning, tuning, receiver wiring) now live in the Alerting section of `docs/telemetry-runbook.md`
**Key modified files**:
@@ -331,21 +441,35 @@ These metrics serve multiple external consumer categories identified during rese
**What to do**:
- Extend the existing telemetry integration test:
- Start xrpld with `[telemetry] enabled=1` and `[insight] server=otel`
- Submit a batch of RPC calls and transactions
- Query Prometheus for each new metric family
- Assert non-zero values for: NodeStore reads, cache hit rates, TxQ count, PerfLog RPC counters, object counts, load factors
- ❌ **Not done on this branch**: extend the telemetry integration test to
start xrpld with `[telemetry] enabled=1` / `[insight] server=otel`, drive RPC
and transaction load, query Prometheus for each new metric family and assert
non-zero values. The end-to-end metric assertions live in the **Phase 10**
harness (`docker/telemetry/workload/expected_metrics.json`), not here.
- Add unit tests for the `MetricsRegistry` class:
- Verify callback registration and deregistration
- Verify metric values match `get_counts` JSON output
- Verify graceful behavior when telemetry is disabled
- ✅ **Done**: unit tests for the `MetricsRegistry` class —
`src/tests/libxrpl/telemetry/MetricsRegistry.cpp` (**18** GTest cases —
`grep -cE '\bTEST(_F|_P)?\s*\(' src/tests/libxrpl/telemetry/MetricsRegistry.cpp`
= 18, and the four bullets below sum to 4 + 3 + 5 + 6 = 18):
- Callback registration / deregistration and shutdown ordering —
`async_gauges_start_after_start_is_safe`,
`async_gauges_before_start_does_not_break_start`,
`async_gauges_respect_the_compile_time_guard`, `destructor_calls_stop`
- Graceful behaviour when telemetry is disabled — `disabled_construction`,
`disabled_start_stop`, `disabled_recording_methods`
- Label sanitisation and mean scaling — `MetricsRegistrySanitiseHandler` (5
cases, incl. `output_domain_is_exactly_44_values`) and
`MetricsRegistryScaledMean` (6 cases)
- ❌ Not covered: asserting metric values match `get_counts` JSON output —
that needs a live `Application`, so it is left to the Phase 10 harness
**Key modified files**:
**Key files**:
- `src/test/telemetry/MetricsRegistry_test.cpp` (new)
- Existing integration test script (extend assertions)
- `src/tests/libxrpl/telemetry/MetricsRegistry.cpp` (new). The originally
planned `src/test/telemetry/MetricsRegistry_test.cpp` was **never created** —
Phase 9 tests are GTest under `src/tests/libxrpl/`, per project convention.
- `src/tests/libxrpl/telemetry/MetricMacros.cpp`, `GetMeter.cpp` (new — cover
the `XRPL_METRIC_*` macros and meter lookup)
---
@@ -360,31 +484,43 @@ These metrics serve multiple external consumer categories identified during rese
**Dashboard**: `validator-health.json`
| Panel | Type | PromQL |
| -------------------------- | ---------- | -------------------------------------------------------------- |
| Agreement % (1h) | stat | `xrpld_validation_agreement{metric="agreement_pct_1h"}` |
| Agreement % (24h) | stat | `xrpld_validation_agreement{metric="agreement_pct_24h"}` |
| Agreements vs Missed (1h) | bargauge | `agreements_1h` and `missed_1h` side by side |
| Agreements vs Missed (24h) | bargauge | `agreements_24h` and `missed_24h` side by side |
| Validation Rate | stat | `rate(xrpld_validations_sent_total[5m]) * 60` |
| Validations Checked Rate | stat | `rate(xrpld_validations_checked_total[5m]) * 60` |
| Amendment Blocked | stat | `xrpld_validator_health{metric="amendment_blocked"}` |
| UNL Expiry (days) | stat | `xrpld_validator_health{metric="unl_expiry_days"}` |
| Validation Quorum | stat | `xrpld_validator_health{metric="validation_quorum"}` |
| State Value Timeline | timeseries | `xrpld_state_tracking{metric="state_value"}` |
| Time in Current State | stat | `xrpld_state_tracking{metric="time_in_current_state_seconds"}` |
| State Changes Rate | stat | `rate(xrpld_state_changes_total[1h])` |
| Ledgers Closed Rate | stat | `rate(xrpld_ledgers_closed_total[5m]) * 60` |
| Panel | Type | PromQL |
| -------------------------- | ---------- | -------------------------------------------------------- |
| Agreement % (1h) | stat | `validation_agreement{metric="agreement_pct_1h"}` |
| Agreement % (24h) | stat | `validation_agreement{metric="agreement_pct_24h"}` |
| Agreements vs Missed (1h) | bargauge | `agreements_1h` and `missed_1h` side by side |
| Agreements vs Missed (24h) | bargauge | `agreements_24h` and `missed_24h` side by side |
| Validation Rate | stat | `rate(validations_sent_total[5m]) * 60` |
| Validations Checked Rate | stat | `rate(validations_checked_total[5m]) * 60` |
| Amendment Blocked | stat | `validator_health{metric="amendment_blocked"}` |
| UNL Expiry (days) | stat | `validator_health{metric="unl_expiry_days"}` |
| Validation Quorum | stat | `validator_health{metric="validation_quorum"}` |
| State Value Timeline | timeseries | `state_tracking{metric="state_value"}` |
| Time in Current State | stat | `state_tracking{metric="time_in_current_state_seconds"}` |
| State Changes Rate | stat | `rate(state_changes_total[1h])` |
| Ledgers Closed Rate | stat | `rate(ledgers_closed_total[5m]) * 60` |
**Dashboard conventions**: `$node` template variable for `service_instance_id` filtering, dark theme, matching existing panel sizes and color schemes.
**Key new files**: `docker/telemetry/grafana/dashboards/rippled-validator-health.json`
**Key new files**: `docker/telemetry/grafana/dashboards/validator-health.json`
(uid `validator-health`). The name reached its current form in **two** renames:
`rippled-validator-health.json` → `xrpld-validator-health.json` (`145b1469d6`,
the `rippled-` → `xrpld-` pass), then `xrpld-validator-health.json` →
`validator-health.json` (`25868f2740`, which dropped the `xrpld-` prefix).
**Exit Criteria**:
- [ ] All 13 panels render with non-zero data during normal operation
- [ ] `$node` filter works correctly for multi-node deployments
- [ ] Amendment blocked and UNL expiry panels use color thresholds (red=blocked/expiring)
- [x] Dashboard ships **17** panels (4 more than the 13 planned above) across 3
rows — Validation Agreement, Validation Rates, Server State & Consensus
- [ ] All panels render with non-zero data during normal operation — needs a live
stack; the Phase 10 harness asserts the dashboard _loads_, not that panels
are non-empty
- [x] `$node` filter works correctly for multi-node deployments — `node`
template variable present (filters on `service_instance_id`), alongside
`service_name`, `deployment_environment`, `xrpl_network_type`,
`xrpl_work_item`, `xrpl_branch`, `xrpl_node_role`
- [x] Amendment blocked and UNL expiry panels use color thresholds
(red=blocked/expiring) — 11 `thresholds` blocks in the dashboard JSON
---
@@ -396,22 +532,35 @@ These metrics serve multiple external consumer categories identified during rese
**Dashboard**: `peer-quality.json`
| Panel | Type | PromQL |
| ---------------------- | ---------- | -------------------------------------------------------------- |
| P90 Peer Latency | timeseries | `xrpld_peer_quality{metric="peer_latency_p90_ms"}` |
| Insane/Diverged Peers | stat | `xrpld_peer_quality{metric="peers_insane_count"}` |
| Higher Version Peers % | stat | `xrpld_peer_quality{metric="peers_higher_version_pct"}` |
| Upgrade Recommended | stat | `xrpld_peer_quality{metric="upgrade_recommended"}` |
| Resource Disconnects | timeseries | `xrpld_Overlay_Peer_Disconnects_Charges` |
| Inbound vs Outbound | bargauge | `xrpld_Peer_Finder_Active_Inbound_Peers`, `..._Outbound_Peers` |
| Panel | Type | PromQL |
| ---------------------- | ---------- | ----------------------------------------------------------------------- |
| P90 Peer Latency | timeseries | `peer_quality{metric="peer_latency_p90_ms"}` |
| Insane/Diverged Peers | stat | `peer_quality{metric="peers_insane_count"}` |
| Higher Version Peers % | stat | `peer_quality{metric="peers_higher_version_pct"}` |
| Upgrade Recommended | stat | `peer_quality{metric="upgrade_recommended"}` |
| Resource Disconnects | timeseries | `server_info{metric="peer_disconnects_resources"}` |
| Inbound vs Outbound | bargauge | `peer_finder_active_inbound_peers`, `peer_finder_active_outbound_peers` |
**Key new files**: `docker/telemetry/grafana/dashboards/rippled-peer-quality.json`
> `overlay_peer_disconnects_charges` (the name in the original spec) is **not a
> real instrument** — nothing registers it. The shipped panel reads
> `server_info{metric="peer_disconnects_resources"}` instead. Peer-finder gauge
> names are lowercase: `GroupImp::makeName()` + `OTelCollectorImp::formatName()`
> turn the `"Peer_Finder"` group into `peer_finder_<name>` with no prefix.
**Key new files**: `docker/telemetry/grafana/dashboards/peer-quality.json`
(uid `peer-quality`). Two renames, same as Task 9.11:
`rippled-peer-quality.json` → `xrpld-peer-quality.json` (`145b1469d6`), then
`xrpld-peer-quality.json` → `peer-quality.json` (`25868f2740`).
**Exit Criteria**:
- [ ] All 6 panels render correctly
- [ ] P90 latency panel shows trend over time
- [ ] Upgrade recommended panel uses color threshold (red=1, green=0)
- [x] All 6 panels present — P90 Peer Latency, Insane/Diverged Peers, Higher
Version Peers %, Upgrade Recommended, Inbound vs Outbound Peers, Resource
Disconnects — across 3 rows, with the `$node` template variable
- [ ] All 6 panels render with data — needs a live stack
- [x] P90 latency panel is a `timeseries` (shows trend over time)
- [x] Upgrade recommended panel uses color threshold (red=1, green=0) — 5
`thresholds` blocks in the dashboard JSON
---
@@ -421,21 +570,22 @@ These metrics serve multiple external consumer categories identified during rese
**Objective**: Add "Ledger Economy" row to the existing `node-health.json` dashboard.
| Panel | Type | PromQL |
| -------------------- | ---------- | --------------------------------------------------- |
| Base Fee (drops) | stat | `xrpld_ledger_economy{metric="base_fee_xrp"}` |
| Reserve Base (drops) | stat | `xrpld_ledger_economy{metric="reserve_base_xrp"}` |
| Reserve Inc (drops) | stat | `xrpld_ledger_economy{metric="reserve_inc_xrp"}` |
| Ledger Age | stat | `xrpld_ledger_economy{metric="ledger_age_seconds"}` |
| Transaction Rate | timeseries | `xrpld_ledger_economy{metric="transaction_rate"}` |
| Panel | Type | PromQL |
| -------------------- | ---------- | --------------------------------------------- |
| Base Fee (drops) | stat | `ledger_economy{metric="base_fee_xrp"}` |
| Reserve Base (drops) | stat | `ledger_economy{metric="reserve_base_xrp"}` |
| Reserve Inc (drops) | stat | `ledger_economy{metric="reserve_inc_xrp"}` |
| Ledger Age | stat | `ledger_economy{metric="ledger_age_seconds"}` |
| Transaction Rate | timeseries | `ledger_economy{metric="transaction_rate"}` |
**Key modified files**: `docker/telemetry/grafana/dashboards/node-health.json`
**Exit Criteria**:
- [ ] 5 new panels render correctly in existing dashboard
- [ ] Fee values match `server_info` RPC output
- [ ] Transaction rate shows smooth trend (not spiky)
- [x] 5 new panels present in the existing dashboard — a "Ledger Economy" row
with 5 `ledger_economy` queries is on `node-health.json`
- [ ] Fee values match `server_info` RPC output — needs a live comparison
- [ ] Transaction rate shows smooth trend (not spiky) — needs a live run
---
@@ -456,9 +606,22 @@ files, so **no code fix lands on this branch**.
**Why deferred**: Defect 3 requires widening the two
`OverlayImpl::updateSlotAndSquelch` overloads — a public signature change on
shared overlay code. Defects 1, 2 and 4 sit in `TrafficCount.{h,cpp}`, likewise
not telemetry-owned. Routing them through the telemetry chain would hide overlay
changes from overlay reviewers and couple them to a 12-PR merge timeline.
shared overlay code. Defects 1 and 4 need `TrafficCount.cpp` and `PeerImp.cpp`
edits that are not telemetry-owned. Routing them through the telemetry chain
would hide overlay changes from overlay reviewers and couple them to a 12-PR
merge timeline.
> **Constraint narrowed.** The blanket "no telemetry change may touch
> `TrafficCount.{h,cpp}`" no longer holds for the header: the telemetry chain
> already edits `TrafficCount.h` — Phase 6's `77f35c03db` fixed the
> `Category::GetFetchPack` label from `"getobject_Fetch Pack_get"` to
> `"getobject_Fetch_Pack_get"` at `TrafficCount.h:285`, the sole difference from
> `develop`. Defect 2 (the stale `Total` header comment, `TrafficCount.h:28-31`)
> is therefore **unblocked** and can land here. Defects **1, 3 and 4** stay
> blocked: defect 1 needs `TrafficCount.cpp`'s `kTypeLookup`, defect 3 needs the
> `OverlayImpl` signature change, and defect 4 needs `PeerImp.cpp:1079` vs `:313`
> to agree on a byte basis (compressed vs uncompressed) — a change to overlay
> accounting semantics, not telemetry.
**Key modified files**: `OpenTelemetryPlan/09-data-collection-reference.md` only.
@@ -466,16 +629,31 @@ changes from overlay reviewers and couple them to a 12-PR merge timeline.
- [x] Each defect documented with file:line evidence in `09` §6
- [x] `overhead_cluster_*` documented as "no data", not "no cluster traffic"
- [ ] Follow-up overlay-owned branch raised for the four code fixes
- [ ] Defect 2 (stale `Total` header comment, `TrafficCount.h:28-31`) fixed on
this branch — it is **unblocked** (the chain already edits
`TrafficCount.h`) but the comment is still uncorrected
- [ ] Follow-up overlay-owned branch raised for the three still-blocked code
fixes (defects 1, 3, 4)
- [ ] Re-baseline any threshold keyed on `unknown_bytes_in` when defect 1 lands
---
## Task 9.15: Peer Keepalive and Discovery Instrumentation
> **Status**: NOT IMPLEMENTED — awaiting a decision on whether `XRPL_METRIC_*`
> call sites may be added to `src/xrpld/overlay/detail/PeerImp.cpp` from this
> branch. Reference: [09 §6.3](./09-data-collection-reference.md#63-peer-keepalive-and-discovery-traffic-gaps-not-implemented)
> **Status**: NOT IMPLEMENTED. The instruments themselves are still to be
> written; the _permission_ question is settled. Reference:
> [09 §6.3](./09-data-collection-reference.md#63-peer-keepalive-and-discovery-traffic-gaps-not-implemented)
>
> **Blocker cleared.** This task used to be held "awaiting a decision on whether
> `XRPL_METRIC_*` call sites may be added to
> `src/xrpld/overlay/detail/PeerImp.cpp` from this branch". That decision is
> de facto **yes** — `PeerImp.cpp` already carries **7** such call sites on this
> branch (`:2723`, `:2741`, `:2925`, `:2928`, `:2931`, `:2947`, `:2954`, of which
> three are `XRPL_METRIC_HISTOGRAM_RECORD` — `:2925`, `:2928`, `:2931` — and four
> are labelled counters — `:2723`, `:2741`, `:2947`, `:2954`). Note that
> `grep -c XRPL_METRIC src/xrpld/overlay/detail/PeerImp.cpp` returns 8: the eighth
> hit is the `cspell:ignore` explanation comment at `PeerImp.cpp:2`, not a call
> site. What remains is the implementation work below, not an approval.
**Objective**: Make peer keepalive and peer-discovery health observable. Today
`mtPING`, `mtSTATUS_CHANGE` and `mtENDPOINTS` are byte counters only.
@@ -494,7 +672,8 @@ changes from overlay reviewers and couple them to a 12-PR merge timeline.
- `peer_id` as a label is unbounded cardinality — rejected. A bounded
`peer_role`-style label is the alternative if per-peer attribution is needed.
- Splitting `mtPING` out of `Category::Base` is a `TrafficCount.cpp` change and
therefore blocked with Task 9.14.
therefore still blocked with Task 9.14 defect 1. (The `.h` half of that
constraint no longer applies — see Task 9.14.)
- Per the runbook's "Adding a New Metric" contract, `_total` is reserved for
monotonic counters; a histogram takes no suffix.
@@ -505,7 +684,8 @@ changes from overlay reviewers and couple them to a 12-PR merge timeline.
**Exit Criteria**:
- [ ] Decision recorded on editing `PeerImp.cpp` from the telemetry chain
- [x] Decision recorded on editing `PeerImp.cpp` from the telemetry chain — yes;
7 `XRPL_METRIC_*` call sites already ship in `PeerImp.cpp`
- [ ] Three instruments emitting, with an explicit histogram bucket view
- [ ] Rows added to `09` §5b, runbook § Metric Reference, and `expected_metrics.json`
- [ ] Peer Quality dashboard panels follow the Task 9.12 conventions (`$node`, Title Case, legend dimensions)
@@ -557,10 +737,11 @@ actually emits. `peer.connect`, `peer.disconnect`, `peer.message.send` and
protocol message families have no spans.
**Scope warning**: This is larger than Tasks 9.14-9.16 combined and changes the
"~37 spans" figure asserted in `09` §1.1 and in
`docker/telemetry/workload/expected_spans.json`. `trace_peer` is also **on by
default** and already flagged as high-volume, so adding per-message spans has a
volume cost that needs measuring before commitment.
span-family inventory asserted in `09` §1.1 (**41** emitted families) and in
`docker/telemetry/workload/expected_spans.json` (**40** catalogued — `rpc.ws_upgrade`
has no entry). `trace_peer` is also **on by default** and already flagged as
high-volume, so adding per-message spans has a volume cost that needs measuring
before commitment.
**Exit Criteria**:
@@ -572,17 +753,41 @@ volume cost that needs measuring before commitment.
## Exit Criteria
- [ ] All ~50 new metrics visible in Prometheus via OTLP pipeline
- [ ] `MetricsRegistry` class registers/deregisters cleanly with OTel SDK
- [ ] Async gauge callbacks execute at 10s intervals without performance impact
- [ ] 2 new Grafana dashboards operational (Fee Market, Job Queue)
- [ ] 2 existing dashboards updated with new panel groups
- [ ] Integration test validates all new metric families are non-zero
- [ ] No performance regression (< 0.5% CPU overhead from new callbacks)
- [ ] Documentation updated with full new metric inventory
- [ ] Validator Health dashboard renders all 13 panels
- [ ] Peer Quality dashboard renders all 6 panels
- [ ] Ledger Economy panels added to node-health dashboard
- [ ] All ~50 new metrics visible in Prometheus via OTLP pipeline — every
instrument is registered in `MetricsRegistry.cpp`, but end-to-end
visibility is asserted only by the Phase 10 harness
- [x] `MetricsRegistry` class registers/deregisters cleanly with OTel SDK —
`src/tests/libxrpl/telemetry/MetricsRegistry.cpp`
(`async_gauges_start_after_start_is_safe`,
`async_gauges_before_start_does_not_break_start`,
`async_gauges_respect_the_compile_time_guard`, `destructor_calls_stop`)
- [x] Async gauge callbacks execute at 10s intervals —
`MetricsRegistry.cpp:289`, `readerOpts.export_interval_millis = 10000`.
(The "without performance impact" half is unmeasured — see below.)
- [x] 4 new Grafana dashboards operational (Fee Market, Job Queue, Validator
Health, Peer Quality) — all four JSONs are under
`docker/telemetry/grafana/dashboards/`
- [x] 2 existing dashboards updated with new panel groups — `node-health`
(NodeStore I/O, Caches, Server Info, Complete Ledgers & DB, Ledger
Economy, Job Queue Concurrency Limits rows) and `rpc-performance`
(per-method section)
- [ ] Integration test validates all new metric families are non-zero — not on
this branch; lives in the Phase 10 harness (`expected_metrics.json`)
- [ ] No performance regression (< 0.5% CPU overhead from new callbacks) — not
measured; needs the Phase 10 benchmark suite
- [x] Documentation updated with full new metric inventory —
`09-data-collection-reference.md` §5b + "Phase 9: OTel SDK-Exported
Metrics (MetricsRegistry)" + "Phase 7+: External Dashboard Parity Metrics"
- [x] Validator Health dashboard ships (17 panels, 4 more than the 13 planned)
- [x] Peer Quality dashboard ships (6 panels)
- [x] Ledger Economy panels added to node-health dashboard (5 panels in a
"Ledger Economy" row)
- [x] Provisioned Grafana alerting: 13 rules / 5 groups, 2 contact points,
nested notification policy
- [ ] Tasks 9.14-9.17 closed — **open by design**: 9.14 documented-not-fixed
(defects 1, 3 and 4 still blocked; defect 2 unblocked but not yet fixed),
9.15 and 9.16 not implemented, 9.17 deferred pending approval and volume
measurement
---
@@ -591,7 +796,7 @@ volume cost that needs measuring before commitment.
> Design for the provisioned Grafana alert rules (Task 9.9a). Previously a standalone spec; merged here so the phase plan is self-contained.
**Date:** 2026-07-06
**Branch:** `pratik/otel-phase9-metric-gap-fill` (PR #6513, Jira RIPD-5187)
**Branch:** `pratik/otel-phase9-metric-gap-fill` (PR #6513)
**Status:** Approved
### Purpose
@@ -599,8 +804,8 @@ volume cost that needs measuring before commitment.
Phase 9 exports ~68 internal xrpld metrics and ships Grafana dashboards for
them. This adds the missing operator-facing piece: **provisioned Grafana alert
rules** that fire on the health-critical metrics phase 9 introduces. The
phase-9 task list (line 311) and Jira story RIPD-5187 both already list
"alerting rules" as a phase-9 deliverable, so this closes that gap.
phase-9 task list already lists "alerting rules" as a phase-9 deliverable
(Task 9.9), so this closes that gap.
Scope is deliberately narrow — the three subsystems whose failure is
node-fatal: **consensus/ledger health, validator health, job queue**. RPC/API
@@ -609,9 +814,11 @@ health is explicitly out of scope.
### Why phase 9 (not phase 11)
Every metric these alerts fire on is _born_ in phase 9
(`xrpld_ledger_history_mismatch_total`, `xrpld_ledgers_closed_total`,
`xrpld_validation_missed_total`, `xrpld_validations_checked_total`,
`xrpld_jq_trans_overflow_total`, `xrpld_job_queued_duration_us_bucket`). Alerts
(`ledger_history_mismatch_total`, `ledgers_closed_total`,
`validation_missed_total`, `validations_checked_total`,
`jq_trans_overflow_total`, `job_queued_us_bucket` — the histogram instrument is
`job_queued_us` (`MetricsRegistry.cpp:94`), so the Prometheus bucket series is
`job_queued_us_bucket`, not `job_queued_duration_us_bucket`). Alerts
belong with the metrics they watch, and this is where the dependency lives.
### Delivery
@@ -623,11 +830,11 @@ Grafana auto-loads `provisioning/alerting/*.yaml`.
New files under `docker/telemetry/grafana/provisioning/alerting/`:
| File | Purpose |
| -------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `contactpoints.yaml` | One contact point `xrpld-default` (webhook to a documented placeholder; comments show how to swap for Slack/email). |
| `policies.yaml` | Default notification policy: route all alerts → `xrpld-default`, grouped by `alertname` + `service_instance_id`. |
| `rules.yaml` | 6 alert rules across 3 groups (below). |
| File | Purpose |
| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `contactpoints.yaml` | **Two** contact points: `xrpld-default` (Slack) and `xrpld-critical` (Slack + email). |
| `policies.yaml` | **Nested** notification policy: root route → `xrpld-default`; child route matching `severity = critical` → `xrpld-critical` (`repeat_interval: 1h` vs the root's `4h`). Both grouped by `alertname` + `service_instance_id`. |
| `rules.yaml` | **13** alert rules across **5** groups (below). |
Plus the Alerting section of `docs/telemetry-runbook.md` — operator runbook:
what each alert means, likely causes, and how to point the contact point at a
@@ -642,16 +849,44 @@ Grafana rule shape: query (A) → reduce (B, last value) → threshold (C). All
Alert rules run headless, so they cannot use the dashboards' `$node` template
variables — they match all series and group by `service_instance_id` instead.
| Group | Alert | Expression (5m window) | Fires | `for` | severity |
| --------- | --------------------- | ----------------------------------------------------------------------------------------------------------- | --------------------- | ----- | -------- |
| Consensus | LedgerHistoryMismatch | `sum by (service_instance_id)(rate(xrpld_ledger_history_mismatch_total[5m]))` | `> 0` | 5m | critical |
| Consensus | LedgerCloseStalled | `sum by (service_instance_id)(rate(xrpld_ledgers_closed_total[5m]))` | `< 0.001` (≈0) | 3m | critical |
| Validator | ValidationsMissed | `sum by (service_instance_id)(rate(xrpld_validation_missed_total[5m]))` | `> 0` | 5m | warning |
| Validator | ValidationsNotChecked | `sum by (service_instance_id)(rate(xrpld_validations_checked_total[5m]))` | `< 0.001` (≈0) | 5m | warning |
| Job queue | JobQueueTxOverflow | `sum by (service_instance_id)(rate(xrpld_jq_trans_overflow_total[5m]))` | `> 0` | 5m | warning |
| Job queue | JobQueueLatencyHigh | `histogram_quantile(0.99, sum by (le, service_instance_id)(rate(xrpld_job_queued_duration_us_bucket[5m])))` | `> 1000000` (µs = 1s) | 5m | warning |
All 5 groups evaluate at `interval: 1m`. Metric names carry **no** `xrpld_`
prefix — `OTelCollectorImp::formatName()` adds none.
Each rule carries labels `severity` and `category` (consensus/validator/jobqueue)
The **Threshold** column is the rule's refId `C` evaluator, read straight from
`rules.yaml` — it is the firing condition, so it is load-bearing, not decoration.
| Group | Alert | Expression (refId A) | Threshold (refId C) | `for` | severity |
| ------------------ | ----------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------------------------------- | ----- | -------- |
| `xrpld-consensus` | LedgerHistoryMismatch | `sum by (service_instance_id) (increase(ledger_history_mismatch_total[15m]))` | `gt [0]` | 2m | critical |
| `xrpld-consensus` | LedgerCloseStalled | `rate(ledgers_closed_total)` decayed to ≈0 | `lt [0.001]` | 3m | critical |
| `xrpld-consensus` | ValidatedLedgerStale | `max by (service_instance_id) (ledgermaster_validated_ledger_age < 1209600)` | `gt [60]` (seconds) | 5m | critical |
| `xrpld-validator` | ValidationsMissed | miss **ratio**, gated on send activity — see the expression below the table | `gt [0.1]` | 15m | warning |
| `xrpld-validator` | ValidationsNotChecked | `rate(validations_checked_total)` ≈0 | `lt [0.001]` | 5m | warning |
| `xrpld-jobqueue` | JobQueueTxOverflow | `sum by (service_instance_id) (increase(jq_trans_overflow_total[15m]))` | `gt [0]` | 2m | warning |
| `xrpld-jobqueue` | JobQueueLatencyHigh | `histogram_quantile(0.99, sum by (le, service_instance_id) (rate(job_queued_us_bucket[5m])))` | `gt [1000000]` (µs = 1s) | 5m | warning |
| `xrpld-jobqueue` | NodeStoreIOLatencyHigh | `histogram_quantile(0.95, sum by (le, service_instance_id) (rate(ios_latency_milliseconds_bucket[10m])))` | `gt [1000]` (ms) | 10m | warning |
| `xrpld-node-state` | NodeStateFlapping | state-transition rate over the node-state series | `gt [3]` (transitions) | 15m | warning |
| `xrpld-node-state` | NodeNotFull | operating mode below FULL | `lt [4]` (FULL = 4) | 15m | warning |
| `xrpld-overlay` | ManifestJobQueueConvoy | `sum by (service_instance_id) (jobq_manifest_waiting)` | `gt [3]` (waiting jobs) | 10m | warning |
| `xrpld-overlay` | ManifestFloodInbound | inbound manifest byte rate | `gt [524288]` (B/s = 512 **KiB**/s, not 512 kB/s) | 10m | warning |
| `xrpld-overlay` | PeerResourceDisconnects | `sum by (service_instance_id) (increase(server_info{metric="peer_disconnects_resources"}[30m]))` | `gt [5]` | 5m | warning |
**`ValidationsMissed` is a gated ratio, not `rate(...) > 0`.** The raw-rate shape
is the pre-fix version and it fires on **every non-validating node**:
`ValidationTracker` counts a miss whenever `weValidated && networkValidated` is
not both true, and a non-validator never sets `weValidated`, so its measured
ratio is exactly **1.0**. No threshold can separate "not a validator" from
"validator disagreeing", hence the `and on (...)` activity gate. The shipped
expression is:
- numerator: `sum by (service_instance_id) (rate(validation_missed_total[15m]))`
- denominator: `clamp_min(` that same numerator `+ sum by (service_instance_id) (rate(validation_agreements_total[15m])), 1e-9)`
- gate: `and on (service_instance_id) (sum by (service_instance_id) (rate(validations_sent_total[15m])) > 0)`
- evaluator: `gt [0.1]` — i.e. >10% disagreement among nodes that do validate
3 rules are `severity: critical`, 10 are `severity: warning`.
Each rule carries labels `severity` and `category`
and annotations `summary` + `description` (with `{{ $labels.service_instance_id }}`
and `{{ $values.B.Value }}` interpolation).
@@ -660,21 +895,31 @@ and `{{ $values.B.Value }}` interpolation).
- **LedgerCloseStalled `< 0.001` for 3m**: healthy nodes close a ledger every
~3-5s; a 5m rate decaying to ~0 means the node is stuck. The epsilon (not
exact `0`) avoids float rate-noise suppressing the alert.
- **JobQueueLatencyHigh 1s p99**: a default starting point, easy to tune — jobs
queued >1s at p99 indicate the node is saturated.
- Others are `> 0` on error/miss counters: any sustained nonzero rate is
actionable.
- **JobQueueLatencyHigh 1s p99**: `gt [1000000]` µs = 1s. A default starting
point, easy to tune — jobs queued >1s at p99 indicate the node is saturated.
- **ValidationsMissed `> 0.1` on a gated ratio**, not `> 0` on a raw rate: the
raw rate is permanently nonzero (ratio 1.0) on non-validators, so a `> 0` rule
pages on every non-validating node in the fleet. See the note above the
rationale list.
- **ManifestFloodInbound 524288 B/s**: an earlier 50 kB/s threshold produced ~41
sustained 5-minute samples on healthy nodes; 512 KiB/s clears normal
manifest-exchange peaks.
- Remaining `gt [0]` rules (`LedgerHistoryMismatch`, `JobQueueTxOverflow`) sit on
true error counters where any sustained nonzero rate is actionable.
### Non-goals / YAGNI
- No per-alert silencing schedules, no mute timings.
- No RPC/API, overlay, or fee-market alerts (dashboards cover those visually).
- Single contact point — multi-receiver routing is left to the operator.
- No RPC/API or fee-market alerts (dashboards cover those visually). Overlay
alerts _were_ added during implementation — the `xrpld-overlay` group carries
three (manifest convoy, manifest flood, peer resource disconnects).
- Two contact points and a two-level policy tree shipped; deeper routing
(Discord, PagerDuty, per-team splits) is left to the operator.
### Verification
1. `yamllint` (or `python -c yaml.safe_load`) on all three YAML files.
2. `docker compose -f docker/telemetry/docker-compose.yml config -q` still parses.
3. Optional live check: start stack, `GET /api/v1/provisioning/alert-rules`
returns the 6 rules; Grafana logs show no provisioning errors.
returns the 13 rules; Grafana logs show no provisioning errors.
4. Code-review pass (subagent) against phase conventions before commit.