From cb7ee2358dcbd538ed4db14557d1a41fd6236c8c Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Mon, 27 Apr 2026 23:12:27 +0100 Subject: [PATCH] docs(telemetry): update data collection reference with complete span/attribute inventory Update 09-data-collection-reference.md to reflect the full implementation across all phases: - Expand span inventory from 16 to 35 spans across 8 categories (RPC, PathFind, TX, TxQ, Consensus, Ledger, Peer, gRPC) - Add complete attribute inventory (81 attributes) - Add TxQ spans (6), PathFind spans (5), and all 10 consensus spans - Document LedgerSpanNames.h and PeerSpanNames.h in file inventory - Add close time analysis dashboard panels to dashboard reference - Add $close_time_correct and $resolution_direction template variables - Document toDisplayString(ConsensusMode) utility - Fix section numbering (duplicate section 8) Co-Authored-By: Claude Opus 4.6 (1M context) --- .../09-data-collection-reference.md | 302 ++++++++++++++---- 1 file changed, 236 insertions(+), 66 deletions(-) diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md index 475257b60a..3223f9a660 100644 --- a/OpenTelemetryPlan/09-data-collection-reference.md +++ b/OpenTelemetryPlan/09-data-collection-reference.md @@ -78,7 +78,7 @@ There are two independent telemetry pipelines entering a single **OTel Collector ## 1. OpenTelemetry Spans -### 1.1 Complete Span Inventory (16 spans) +### 1.1 Complete Span Inventory (35 spans) > **See also**: [02-design-decisions.md §2.3](./02-design-decisions.md#23-span-naming-conventions) for naming conventions and the full span catalog with rationale. [04-code-samples.md §4.6](./04-code-samples.md#46-span-flow-visualization) for span flow diagrams. @@ -86,14 +86,15 @@ There are two independent telemetry pipelines entering a single **OTel Collector Controlled by `trace_rpc=1` in `[telemetry]` config. -| Span Name | Parent | Source File | Description | -| -------------------- | ------------- | ----------------- | ------------------------------------------------------------------------ | -| `rpc.request` | — | ServerHandler.cpp | Top-level HTTP RPC request entry point | -| `rpc.process` | `rpc.request` | ServerHandler.cpp | RPC processing pipeline | -| `rpc.ws_message` | — | ServerHandler.cpp | WebSocket message handling | -| `rpc.command.` | `rpc.process` | RPCHandler.cpp | Per-command span (e.g., `rpc.command.server_info`, `rpc.command.ledger`) | +| Span Name | Parent | Source File | Description | +| -------------------- | ------------------ | ----------------- | ------------------------------------------------------------------------ | +| `rpc.http_request` | — | ServerHandler.cpp | Top-level HTTP RPC request entry point | +| `rpc.process` | `rpc.http_request` | ServerHandler.cpp | RPC processing pipeline | +| `rpc.ws_message` | — | ServerHandler.cpp | WebSocket message handling | +| `rpc.ws_upgrade` | — | ServerHandler.cpp | WebSocket upgrade handshake (error path) | +| `rpc.command.` | `rpc.process` | RPCHandler.cpp | Per-command span (e.g., `rpc.command.server_info`, `rpc.command.ledger`) | -**Where to find**: Tempo → TraceQL: `{resource.service.name="rippled" && name=~"rpc.request|rpc.command.*"}` +**Where to find**: Tempo → TraceQL: `{resource.service.name="rippled" && name=~"rpc.http_request|rpc.command.*"}` **Grafana dashboard**: _RPC Performance_ (`rippled-rpc-perf`) @@ -111,17 +112,67 @@ Controlled by `trace_transactions=1` in `[telemetry]` config. **Grafana dashboard**: _Transaction Overview_ (`rippled-transactions`) +#### PathFind Spans + +Controlled by `trace_rpc=1` in `[telemetry]` config (pathfinding spans fire within RPC request handling). + +| Span Name | Parent | Source File | Description | +| --------------------- | ------------------ | ---------------- | -------------------------------------------------------- | +| `pathfind.request` | `rpc.command.*` | PathRequests.cpp | RPC entry for path_find / ripple_path_find | +| `pathfind.compute` | `pathfind.request` | PathRequest.cpp | Single path computation (doUpdate) | +| `pathfind.update_all` | — | PathRequests.cpp | Async recomputation of all active path requests on close | +| `pathfind.discover` | `pathfind.compute` | Pathfinder.cpp | Graph exploration phase (Pathfinder::find) | +| `pathfind.rank` | `pathfind.compute` | Pathfinder.cpp | Path ranking and selection phase | + +**Where to find**: Tempo → TraceQL: `{resource.service.name="rippled" && name=~"pathfind.*"}` + +**Grafana dashboard**: _RPC & Pathfinding (StatsD)_ (`rippled-statsd-rpc`) for StatsD timers; span-derived metrics via _RPC Performance_ (`rippled-rpc-perf`) + +#### TxQ Spans + +Controlled by `trace_transactions=1` in `[telemetry]` config. + +| Span Name | Parent | Source File | Description | +| ------------------ | ------------- | ----------- | ---------------------------------------------------- | +| `txq.enqueue` | `tx.process` | TxQ.cpp | Queue admission decision (apply/queue/reject) | +| `txq.apply_direct` | `txq.enqueue` | TxQ.cpp | Direct application attempt (bypassing queue) | +| `txq.batch_clear` | `txq.enqueue` | TxQ.cpp | Batch clear of account's queued transactions | +| `txq.accept` | — | TxQ.cpp | Ledger-close accept loop (drain queued transactions) | +| `txq.accept.tx` | `txq.accept` | TxQ.cpp | Per-transaction apply within accept loop | +| `txq.cleanup` | — | TxQ.cpp | Post-close cleanup (expire old transactions) | + +**Where to find**: Tempo → TraceQL: `{resource.service.name="rippled" && name=~"txq.*"}` + +**Grafana dashboard**: _Transaction Overview_ (`rippled-transactions`) + +#### gRPC Spans + +Controlled by `trace_rpc=1` in `[telemetry]` config. + +| Span Name | Parent | Source File | Description | +| -------------- | ------ | -------------- | ----------------------------------------------------------------------------- | +| `grpc.request` | — | GRPCServer.cpp | Single gRPC request (GetLedger, GetLedgerData, GetLedgerDiff, GetLedgerEntry) | + +**Where to find**: Tempo → TraceQL: `{resource.service.name="rippled" && name="grpc.request"}` + #### Consensus Spans Controlled by `trace_consensus=1` in `[telemetry]` config. -| Span Name | Parent | Source File | Description | -| --------------------------- | ------ | ---------------- | --------------------------------------------- | -| `consensus.proposal.send` | — | RCLConsensus.cpp | Node broadcasts its transaction set proposal | -| `consensus.ledger_close` | — | RCLConsensus.cpp | Ledger close event triggered by consensus | -| `consensus.accept` | — | RCLConsensus.cpp | Consensus accepts a ledger (round complete) | -| `consensus.validation.send` | — | RCLConsensus.cpp | Validation message sent after ledger accepted | -| `consensus.accept.apply` | — | RCLConsensus.cpp | Ledger application with close time details | +| Span Name | Parent | Source File | Description | +| ---------------------------- | ----------------- | ---------------- | ----------------------------------------------------- | +| `consensus.round` | — | RCLConsensus.cpp | Top-level round span (deterministic trace ID) | +| `consensus.proposal.send` | `consensus.round` | RCLConsensus.cpp | Node broadcasts its transaction set proposal | +| `consensus.ledger_close` | `consensus.round` | RCLConsensus.cpp | Ledger close event triggered by consensus | +| `consensus.establish` | `consensus.round` | Consensus.h | Establish phase — convergence loop | +| `consensus.update_positions` | `consensus.round` | Consensus.h | Update positions during establish phase | +| `consensus.check` | `consensus.round` | Consensus.h | Check for consensus agreement | +| `consensus.accept` | `consensus.round` | RCLConsensus.cpp | Consensus accepts a ledger (round complete) | +| `consensus.accept.apply` | `consensus.round` | RCLConsensus.cpp | Ledger application with close time details | +| `consensus.validation.send` | `consensus.round` | RCLConsensus.cpp | Validation message sent after ledger accepted | +| `consensus.mode_change` | `consensus.round` | RCLConsensus.cpp | Consensus mode transition (e.g., tracking->proposing) | + +> **Note**: `toDisplayString(ConsensusMode)` (in `ConsensusTypes.h`) provides Title Case display names for mode attribute values: `"Proposing"`, `"Observing"`, `"Wrong Ledger"`, `"Switched Ledger"`. This is separate from `to_string()` which returns stable log-format strings. **Where to find**: Tempo → TraceQL: `{resource.service.name="rippled" && name=~"consensus.*"}` @@ -156,7 +207,7 @@ Controlled by `trace_peer=1` in `[telemetry]` config. **Disabled by default** (h --- -### 1.2 Complete Attribute Inventory (22 attributes) +### 1.2 Complete Attribute Inventory (81 attributes) > **See also**: [02-design-decisions.md §2.4.2](./02-design-decisions.md#242-span-attributes-by-category) for attribute design rationale and privacy considerations. @@ -164,14 +215,13 @@ Every span can carry key-value attributes that provide context for filtering and #### RPC Attributes -| Attribute | Type | Set On | Description | -| ------------------------ | ------ | --------------- | ------------------------------------------------ | -| `xrpl.rpc.command` | string | `rpc.command.*` | RPC command name (e.g., `server_info`, `ledger`) | -| `xrpl.rpc.version` | int64 | `rpc.command.*` | API version number | -| `xrpl.rpc.role` | string | `rpc.command.*` | Caller role: `"admin"` or `"user"` | -| `xrpl.rpc.status` | string | `rpc.command.*` | Result: `"success"` or `"error"` | -| `xrpl.rpc.duration_ms` | int64 | `rpc.command.*` | Command execution time in milliseconds | -| `xrpl.rpc.error_message` | string | `rpc.command.*` | Error details (only set on failure) | +| Attribute | Type | Set On | Description | +| ----------------------- | ------ | --------------- | ------------------------------------------------ | +| `xrpl.rpc.command` | string | `rpc.command.*` | RPC command name (e.g., `server_info`, `ledger`) | +| `xrpl.rpc.version` | int64 | `rpc.command.*` | API version number | +| `xrpl.rpc.role` | string | `rpc.command.*` | Caller role: `"admin"` or `"user"` | +| `xrpl.rpc.status` | string | `rpc.command.*` | Result: `"success"` or `"error"` | +| `xrpl.rpc.payload_size` | int64 | `rpc.command.*` | Request payload size in bytes | **Tempo query**: `{span.xrpl.rpc.command="server_info"}` to find all `server_info` calls. @@ -186,48 +236,123 @@ Every span can carry key-value attributes that provide context for filtering and | `xrpl.tx.path` | string | `tx.process` | Submission path: `"sync"` or `"async"` | | `xrpl.tx.suppressed` | boolean | `tx.receive` | `true` if transaction was suppressed (duplicate) | | `xrpl.tx.status` | string | `tx.receive` | Transaction status (e.g., `"known_bad"`) | +| `xrpl.peer.id` | int64 | `tx.receive` | Peer identifier (also set on peer spans) | +| `xrpl.peer.version` | string | `tx.receive` | Peer protocol version string | **Tempo query**: `{span.xrpl.tx.hash=""}` to trace a specific transaction across nodes. **Prometheus label**: `xrpl_tx_local` (used as SpanMetrics dimension). +#### PathFind Attributes + +| Attribute | Type | Set On | Description | +| ---------------------------------- | ------- | --------------------- | ----------------------------------------------- | +| `xrpl.pathfind.source_account` | string | `pathfind.request` | Source account address | +| `xrpl.pathfind.dest_account` | string | `pathfind.request` | Destination account address | +| `xrpl.pathfind.fast` | boolean | `pathfind.compute` | Whether this is a fast (non-full) pathfind | +| `xrpl.pathfind.search_level` | int64 | `pathfind.compute` | Search depth level | +| `xrpl.pathfind.num_complete_paths` | int64 | `pathfind.compute` | Number of complete paths found | +| `xrpl.pathfind.num_paths` | int64 | `pathfind.compute` | Total number of paths explored | +| `xrpl.pathfind.num_requests` | int64 | `pathfind.update_all` | Number of active path requests being recomputed | +| `xrpl.pathfind.ledger_index` | int64 | `pathfind.update_all` | Ledger index used for recomputation | + +**Tempo query**: `{span.xrpl.pathfind.source_account="rHb9..."}` to find pathfind requests from a specific account. + +#### TxQ Attributes + +| Attribute | Type | Set On | Description | +| ----------------------------- | ------- | ------------------------------ | ---------------------------------------------------------- | +| `xrpl.txq.tx_hash` | string | `txq.enqueue`, `txq.accept.tx` | Transaction hash in the queue | +| `xrpl.txq.status` | string | `txq.enqueue` | Queue result: `"queued"`, `"applied_direct"`, `"rejected"` | +| `xrpl.txq.fee_level_paid` | int64 | `txq.enqueue` | Fee level paid by the transaction | +| `xrpl.txq.required_fee_level` | int64 | `txq.enqueue` | Minimum fee level required for queue admission | +| `xrpl.txq.queue_size` | int64 | `txq.accept` | Queue depth at start of accept | +| `xrpl.txq.ledger_changed` | boolean | `txq.accept` | Whether the open ledger changed since last accept | +| `xrpl.txq.ledger_seq` | int64 | `txq.cleanup` | Ledger sequence for cleanup | +| `xrpl.txq.expired_count` | int64 | `txq.cleanup` | Number of expired transactions removed | +| `xrpl.txq.ter_code` | string | `txq.accept.tx` | Transaction engine result code | +| `xrpl.txq.retries_remaining` | int64 | `txq.accept.tx` | Remaining retry attempts for this transaction | +| `xrpl.txq.num_cleared` | int64 | `txq.batch_clear` | Number of transactions cleared in batch | + +**Tempo query**: `{span.xrpl.txq.status="rejected"}` to find rejected queue attempts. + +#### gRPC Attributes + +| Attribute | Type | Set On | Description | +| ------------------ | ------ | -------------- | ------------------------------------------------------------ | +| `xrpl.grpc.method` | string | `grpc.request` | gRPC method name (e.g., `GetLedger`, `GetLedgerData`) | +| `xrpl.grpc.role` | string | `grpc.request` | Caller role: `"admin"` or `"user"` | +| `xrpl.grpc.status` | string | `grpc.request` | Result: `"success"`, `"error"`, `"resource_exhausted"`, etc. | + +**Tempo query**: `{span.xrpl.grpc.method="GetLedger"}` to find gRPC ledger requests. + #### Consensus Attributes -| Attribute | Type | Set On | Description | -| ------------------------------------ | ------- | --------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | -| `xrpl.consensus.round` | int64 | `consensus.proposal.send` | Consensus round number | -| `xrpl.consensus.mode` | string | `consensus.proposal.send`, `consensus.ledger_close` | Node mode: `"syncing"`, `"tracking"`, `"full"`, `"proposing"` | -| `xrpl.consensus.proposers` | int64 | `consensus.proposal.send`, `consensus.accept` | Number of proposers in the round | -| `xrpl.consensus.proposing` | boolean | `consensus.validation.send` | Whether this node was a proposer | -| `xrpl.consensus.ledger.seq` | int64 | `consensus.ledger_close`, `consensus.accept`, `consensus.validation.send`, `consensus.accept.apply` | Ledger sequence number | -| `xrpl.consensus.close_time` | int64 | `consensus.accept.apply` | Agreed-upon ledger close time (epoch seconds) | -| `xrpl.consensus.close_time_correct` | boolean | `consensus.accept.apply` | Whether validators reached agreement on close time | -| `xrpl.consensus.close_resolution_ms` | int64 | `consensus.accept.apply` | Close time rounding granularity in milliseconds | -| `xrpl.consensus.state` | string | `consensus.accept.apply` | Consensus outcome: `"finished"` or `"moved_on"` | -| `xrpl.consensus.round_time_ms` | int64 | `consensus.accept.apply` | Total consensus round duration in milliseconds | +| Attribute | Type | Set On | Description | +| ------------------------------------------ | ------- | ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- | +| `xrpl.consensus.ledger_id` | string | `consensus.round` | Previous ledger hash (used for deterministic trace ID) | +| `xrpl.consensus.ledger.seq` | int64 | `consensus.round`, `consensus.ledger_close`, `consensus.accept`, `consensus.validation.send`, `consensus.accept.apply` | Ledger sequence number | +| `xrpl.consensus.mode` | string | `consensus.round`, `consensus.proposal.send`, `consensus.ledger_close` | Node mode via `toDisplayString()`: `"Proposing"`, `"Observing"`, etc. | +| `xrpl.consensus.round` | int64 | `consensus.proposal.send` | Consensus round number | +| `xrpl.consensus.proposers` | int64 | `consensus.proposal.send`, `consensus.accept` | Number of proposers in the round | +| `xrpl.consensus.round_time_ms` | int64 | `consensus.accept`, `consensus.accept.apply` | Total consensus round duration in milliseconds | +| `xrpl.consensus.proposing` | boolean | `consensus.validation.send` | Whether this node was a proposer | +| `xrpl.consensus.state` | string | `consensus.accept.apply` | Consensus outcome: `"finished"` or `"moved_on"` | +| `xrpl.consensus.close_time` | int64 | `consensus.accept.apply` | Agreed-upon ledger close time (epoch seconds) | +| `xrpl.consensus.close_time_correct` | boolean | `consensus.accept.apply` | Whether validators reached agreement on close time | +| `xrpl.consensus.close_resolution_ms` | int64 | `consensus.accept.apply` | Close time rounding granularity in milliseconds | +| `xrpl.consensus.parent_close_time` | int64 | `consensus.accept.apply` | Parent ledger's close time (epoch seconds) | +| `xrpl.consensus.close_time_self` | int64 | `consensus.accept.apply` | This node's proposed close time | +| `xrpl.consensus.close_time_vote_bins` | string | `consensus.accept.apply` | Histogram of close time votes from validators | +| `xrpl.consensus.resolution_direction` | string | `consensus.accept.apply` | Resolution change: `"increased"`, `"decreased"`, or `"unchanged"` | +| `xrpl.consensus.converge_percent` | int64 | `consensus.establish` | Convergence percentage threshold | +| `xrpl.consensus.establish_count` | int64 | `consensus.establish` | Number of establish iterations completed | +| `xrpl.consensus.proposers_agreed` | int64 | `consensus.establish` | Number of proposers that agreed on this round | +| `xrpl.consensus.avalanche_threshold` | int64 | `consensus.update_positions` | Avalanche threshold for dispute resolution | +| `xrpl.consensus.close_time_threshold` | int64 | `consensus.update_positions` | Close time agreement threshold | +| `xrpl.consensus.have_close_time_consensus` | boolean | `consensus.update_positions` | Whether close time consensus has been reached | +| `xrpl.consensus.agree_count` | int64 | `consensus.check` | Number of proposers that agree with our position | +| `xrpl.consensus.disagree_count` | int64 | `consensus.check` | Number of proposers that disagree with our position | +| `xrpl.consensus.threshold_percent` | int64 | `consensus.check` | Required agreement threshold percentage | +| `xrpl.consensus.result` | string | `consensus.check` | Check result: `"yes"`, `"no"`, or `"expired"` | +| `xrpl.consensus.quorum` | int64 | `consensus.check` | Required quorum for validation | +| `xrpl.consensus.validation_count` | int64 | `consensus.check` | Number of validations received | +| `xrpl.consensus.trace_strategy` | string | `consensus.round` | Trace sampling strategy used for this round | +| `xrpl.consensus.round_id` | string | `consensus.round` | Deterministic round identifier | +| `xrpl.consensus.mode.old` | string | `consensus.mode_change` | Previous consensus mode | +| `xrpl.consensus.mode.new` | string | `consensus.mode_change` | New consensus mode | +| `xrpl.tx.id` | string | `consensus.update_positions` | Disputed transaction ID | +| `xrpl.dispute.our_vote` | boolean | `consensus.update_positions` | Our vote on the disputed transaction | +| `xrpl.dispute.yays` | int64 | `consensus.update_positions` | Number of proposers voting to include | +| `xrpl.dispute.nays` | int64 | `consensus.update_positions` | Number of proposers voting to exclude | -**Tempo query**: `{span.xrpl.consensus.mode="proposing"}` to find rounds where node was proposing. +**Tempo query**: `{span.xrpl.consensus.mode="Proposing"}` to find rounds where node was proposing. **Prometheus label**: `xrpl_consensus_mode` (used as SpanMetrics dimension). #### Ledger Attributes -| Attribute | Type | Set On | Description | -| ------------------------- | ----- | ------------------------------------------------------------- | ---------------------------------------------- | -| `xrpl.ledger.seq` | int64 | `ledger.build`, `ledger.validate`, `ledger.store`, `tx.apply` | Ledger sequence number | -| `xrpl.ledger.validations` | int64 | `ledger.validate` | Number of validations received for this ledger | -| `xrpl.ledger.tx_count` | int64 | `ledger.build`, `tx.apply` | Transactions in the ledger | -| `xrpl.ledger.tx_failed` | int64 | `ledger.build`, `tx.apply` | Failed transactions in the ledger | +| Attribute | Type | Set On | Description | +| --------------------------------- | ------- | ------------------------------------------------------------- | ------------------------------------------------ | +| `xrpl.ledger.seq` | int64 | `ledger.build`, `ledger.validate`, `ledger.store`, `tx.apply` | Ledger sequence number | +| `xrpl.ledger.close_time` | int64 | `ledger.build` | Ledger close time (epoch seconds) | +| `xrpl.ledger.close_time_correct` | boolean | `ledger.build` | Whether close time was agreed upon by validators | +| `xrpl.ledger.close_resolution_ms` | int64 | `ledger.build` | Close time rounding granularity in milliseconds | +| `xrpl.ledger.tx_count` | int64 | `ledger.build`, `tx.apply` | Transactions in the ledger | +| `xrpl.ledger.tx_failed` | int64 | `ledger.build`, `tx.apply` | Failed transactions in the ledger | +| `xrpl.ledger.validations` | int64 | `ledger.validate` | Number of validations received for this ledger | **Tempo query**: `{span.xrpl.ledger.seq=12345}` to find all spans for a specific ledger. #### Peer Attributes -| Attribute | Type | Set On | Description | -| ------------------------------ | ------- | ---------------------------------------------------------------- | ---------------------------------------------------- | -| `xrpl.peer.id` | int64 | `tx.receive`, `peer.proposal.receive`, `peer.validation.receive` | Peer identifier | -| `xrpl.peer.proposal.trusted` | boolean | `peer.proposal.receive` | Whether the proposal came from a trusted validator | -| `xrpl.peer.validation.trusted` | boolean | `peer.validation.receive` | Whether the validation came from a trusted validator | +| Attribute | Type | Set On | Description | +| ---------------------------------- | ------- | ---------------------------------------------------------------- | ---------------------------------------------------- | +| `xrpl.peer.id` | int64 | `tx.receive`, `peer.proposal.receive`, `peer.validation.receive` | Peer identifier | +| `xrpl.peer.proposal.trusted` | boolean | `peer.proposal.receive` | Whether the proposal came from a trusted validator | +| `xrpl.peer.validation.ledger_hash` | string | `peer.validation.receive` | Ledger hash the validation refers to | +| `xrpl.peer.validation.full` | boolean | `peer.validation.receive` | Whether this is a full (not partial) validation | +| `xrpl.peer.validation.trusted` | boolean | `peer.validation.receive` | Whether the validation came from a trusted validator | **Prometheus labels**: `xrpl_peer_proposal_trusted`, `xrpl_peer_validation_trusted` (SpanMetrics dimensions). @@ -366,13 +491,13 @@ For each of the 45+ overlay traffic categories (defined in `TrafficCount.h`), fo ### 3.1 Span-Derived Dashboards (5) -| Dashboard | UID | Data Source | Key Panels | -| -------------------- | ---------------------- | ------------------------ | ---------------------------------------------------------------------------------- | -| RPC Performance | `rippled-rpc-perf` | Prometheus (SpanMetrics) | Request rate by command, p95 latency by command, error rate, heatmap, top commands | -| Transaction Overview | `rippled-transactions` | Prometheus (SpanMetrics) | Processing rate, latency p95/p50, local vs relay split, apply duration, heatmap | -| Consensus Health | `rippled-consensus` | Prometheus (SpanMetrics) | Round duration p95/p50, proposals rate, close duration, mode timeline, heatmap | -| Ledger Operations | `rippled-ledger-ops` | Prometheus (SpanMetrics) | Build rate, build duration, validation rate, store rate, build vs close comparison | -| Peer Network | `rippled-peer-net` | Prometheus (SpanMetrics) | Proposal receive rate, validation receive rate, trusted vs untrusted breakdown | +| Dashboard | UID | Data Source | Key Panels | +| -------------------- | ---------------------- | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| RPC Performance | `rippled-rpc-perf` | Prometheus (SpanMetrics) | Request rate by command, p95 latency by command, error rate, heatmap, top commands | +| Transaction Overview | `rippled-transactions` | Prometheus (SpanMetrics) | Processing rate, latency p95/p50, local vs relay split, apply duration, heatmap | +| Consensus Health | `rippled-consensus` | Prometheus (SpanMetrics) | Round duration p95/p50, proposals rate, close duration, mode timeline, heatmap, close time correctness, resolution direction, close time drift, resolution change timeline, close time vote distribution | +| Ledger Operations | `rippled-ledger-ops` | Prometheus (SpanMetrics) | Build rate, build duration, validation rate, store rate, build vs close comparison | +| Peer Network | `rippled-peer-net` | Prometheus (SpanMetrics) | Proposal receive rate, validation receive rate, trusted vs untrusted breakdown | ### 3.2 StatsD Dashboards (5) @@ -384,7 +509,27 @@ For each of the 45+ overlay traffic categories (defined in `TrafficCount.h`), fo | Overlay Traffic Detail | `rippled-statsd-overlay-detail` | Prometheus (StatsD) | Squelch, overhead, validator lists, set get/share, have/requested tx, proof paths | | Ledger Data & Sync | `rippled-statsd-ledger-sync` | Prometheus (StatsD) | Ledger data exchange, legacy ledger share/get, getobject by type, traffic heatmap | -### 3.3 Accessing the Dashboards +### 3.3 Consensus Close-Time Panels + +The Consensus Health dashboard includes 5 close-time panels added in Phase 4: + +| Panel | Metric / Attribute | Description | +| ---------------------------- | --------------------------------------------------------------- | ------------------------------------------------------------------------ | +| Close Time Correctness | `xrpl.consensus.close_time_correct` | Percentage of rounds with agreed-upon close time | +| Resolution Direction | `xrpl.consensus.resolution_direction` | Rate of resolution increases, decreases, and unchanged per time interval | +| Close Time Drift | `xrpl.consensus.close_time` vs `xrpl.consensus.close_time_self` | Difference between agreed close time and node's own proposed close time | +| Resolution Change Timeline | `xrpl.consensus.close_resolution_ms` | Close time resolution granularity over time | +| Close Time Vote Distribution | `xrpl.consensus.close_time_vote_bins` | Histogram of validator close time votes per round | + +**Template variables** (Consensus Health dashboard): + +| Variable | Source Attribute | Description | +| ----------------------- | ------------------------------------- | ------------------------------------------------------------------------ | +| `$node` | `exported_instance` | Filter by rippled node instance | +| `$close_time_correct` | `xrpl_consensus_close_time_correct` | Filter by close time correctness (`true` / `false`) | +| `$resolution_direction` | `xrpl_consensus_resolution_direction` | Filter by resolution direction (`increased` / `decreased` / `unchanged`) | + +### 3.4 Accessing the Dashboards 1. Open Grafana at **http://localhost:3000** 2. Navigate to **Dashboards → rippled** folder @@ -400,7 +545,7 @@ For each of the 45+ overlay traffic categories (defined in `TrafficCount.h`), fo | What to Find | Tempo TraceQL Query | | ------------------------ | -------------------------------------------------------------------------------- | -| All RPC calls | `{resource.service.name="rippled" && name="rpc.request"}` | +| All RPC calls | `{resource.service.name="rippled" && name="rpc.http_request"}` | | Specific RPC command | `{resource.service.name="rippled" && name="rpc.command.server_info"}` | | Slow RPC calls | `{resource.service.name="rippled" && name=~"rpc.command.*"} \| duration > 100ms` | | Failed RPC calls | `{span.xrpl.rpc.status="error"}` | @@ -416,20 +561,26 @@ For each of the 45+ overlay traffic categories (defined in `TrafficCount.h`), fo A typical RPC trace shows the span hierarchy: ``` -rpc.request (ServerHandler) +rpc.http_request (ServerHandler) └── rpc.process (ServerHandler) └── rpc.command.server_info (RPCHandler) ``` -A consensus round produces independent spans (not parent-child): +A consensus round groups child spans under a deterministic trace ID: ``` -consensus.ledger_close (close event) -consensus.proposal.send (broadcast proposal) +consensus.round (top-level, deterministic trace ID from ledger hash) + ├── consensus.ledger_close (close event) + ├── consensus.proposal.send (broadcast proposal) + ├── consensus.establish (convergence loop) + │ ├── consensus.update_positions (update disputes) + │ └── consensus.check (check agreement) + ├── consensus.accept (accept result) + ├── consensus.accept.apply (apply with close time details) + ├── consensus.validation.send (send validation) + └── consensus.mode_change (mode transition, if any) ledger.build (build new ledger) └── tx.apply (apply transaction set) -consensus.accept (accept result) -consensus.validation.send (send validation) ledger.validate (promote to validated) ledger.store (persist to DB) ``` @@ -480,7 +631,26 @@ rippled_State_Accounting_Full_duration --- -## 6. Known Issues +## 6. SpanNames Header File Inventory + +All span names and attributes are defined as compile-time constants in colocated `SpanNames.h` headers. Each header lives next to its subsystem's implementation. + +| Header File | Subsystem | Span Count | Attribute Count | Notes | +| ----------------------------------------------- | ------------- | ---------- | --------------- | ------------------------------------------- | +| `src/xrpld/rpc/detail/RpcSpanNames.h` | RPC (HTTP/WS) | 5 | 5 | Includes `rpc.ws_upgrade` error path | +| `src/xrpld/rpc/detail/PathFindSpanNames.h` | PathFind | 5 | 8 | Covers one-shot and subscription paths | +| `src/xrpld/app/main/GrpcSpanNames.h` | gRPC | 1 | 3 | Flat single-span structure per request | +| `src/xrpld/app/misc/TxSpanNames.h` | Transaction | 2 | 7 | Includes peer context attributes | +| `src/xrpld/app/misc/detail/TxQSpanNames.h` | TxQ | 6 | 11 | Queue lifecycle: enqueue through cleanup | +| `src/xrpld/app/consensus/ConsensusSpanNames.h` | Consensus | 10 | 35 | Deterministic trace IDs, close-time details | +| `src/xrpld/app/ledger/detail/LedgerSpanNames.h` | Ledger | 4 | 7 | Build, store, validate, tx.apply | +| `src/xrpld/overlay/detail/PeerSpanNames.h` | Peer Overlay | 2 | 5 | Proposal and validation receive | + +> **Design convention**: SpanNames headers are colocated with their subsystem classes rather than centralized in `telemetry/`. See [memory/feedback_span-names-colocation.md](../.claude/memory/feedback_span-names-colocation.md) for rationale. + +--- + +## 7. Known Issues | Issue | Impact | Status | | ------------------------------------------------------------------ | ------------------------------------------------ | -------------------------------------------------------------------- | @@ -491,7 +661,7 @@ rippled_State_Accounting_Full_duration --- -## 7. Privacy and Data Collection +## 8. Privacy and Data Collection The telemetry system is designed with privacy in mind: @@ -505,7 +675,7 @@ The telemetry system is designed with privacy in mind: --- -## 8. Configuration Quick Reference +## 9. Configuration Quick Reference > **Full reference**: [05-configuration-reference.md](./05-configuration-reference.md) §5.1 for all `[telemetry]` options with defaults, the config parser implementation, and collector YAML configurations (dev and production).