ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-07-30 02:20:39 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
249e407bb1e6d15d968a96fa4c5b1fc57f11953d
rippled/OpenTelemetryPlan/09-data-collection-reference.md
Pratik Mankawde 6768aa4c07 docs(telemetry): document handler label, GetObject and queue-saturation metrics 
Add reference entries for the observability surface introduced on
phase-9: the `handler` label on the job instruments, the five
`getobject_*` request metrics, and the per-job-type queue saturation
gauges.

Names here follow this branch's StatsD pipeline, which preserves case
and carries the `xrpld_` prefix, so they differ from the lowercased
OTel-native names used from phase-7 onward. The sections state where
the implementing code lives, since it is introduced downstream.

Also correct pre-existing entries: `job_count` exports as
`jobq_job_count` via the collector group prefix, the non-special job
type count is 35 (not 36), and `JtLedgerData` has five producers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 11:58:16 +01:00

973 lines
74 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Observability Data Collection Reference
> **Audience**: Developers and operators. This is the single source of truth for all telemetry data collected by xrpld's observability stack.
>
> **Related docs**: [docs/telemetry-runbook.md](../docs/telemetry-runbook.md) (operator runbook with alerting and troubleshooting) | [03-implementation-strategy.md](./03-implementation-strategy.md) (code structure and performance optimization) | [04-code-samples.md](./04-code-samples.md) (C++ instrumentation examples)
## Data Flow Overview
```mermaid
graph LR
subgraph xrpldNode["xrpld Node"]
A["Trace Macros<br/>XRPL_TRACE_SPAN<br/>(OTLP/HTTP exporter)"]
B["beast::insight<br/>StatsD metrics<br/>(UDP sender)"]
end
subgraph collector["OTel Collector :4317 / :4318 / :8125"]
direction TB
R1["OTLP Receiver<br/>:4317 gRPC | :4318 HTTP"]
R2["StatsD Receiver<br/>:8125 UDP"]
BP["Batch Processor<br/>timeout 1s, batch 100"]
SM["SpanMetrics Connector<br/>derives RED metrics<br/>from trace spans"]
R1 --> BP
BP --> SM
end
subgraph backends["Trace Backend"]
D["Grafana Tempo :3200<br/>TraceQL search &<br/>S3/GCS long-term storage"]
end
subgraph metrics["Metrics Stack"]
E["Prometheus :9090<br/>scrapes :8889<br/>span-derived + StatsD metrics"]
end
subgraph viz["Visualization"]
F["Grafana :3000<br/>10 dashboards"]
end
A -->|"OTLP/HTTP :4318<br/>(traces + attributes)"| R1
B -->|"UDP :8125<br/>(gauges, counters, timers)"| R2
BP -->|"OTLP/gRPC :4317"| D
SM -->|"span_calls_total<br/>span_duration_ms<br/>(6 dimension labels)"| E
R2 -->|"xrpld_* gauges<br/>xrpld_* counters<br/>xrpld_* summaries"| E
E -->|"Prometheus<br/>data source"| F
D -->|"Tempo<br/>data source"| F
style A fill:#4a90d9,color:#fff,stroke:#2a6db5
style B fill:#d9534f,color:#fff,stroke:#b52d2d
style R1 fill:#5cb85c,color:#fff,stroke:#3d8b3d
style R2 fill:#5cb85c,color:#fff,stroke:#3d8b3d
style BP fill:#449d44,color:#fff,stroke:#2d6e2d
style SM fill:#449d44,color:#fff,stroke:#2d6e2d
style D fill:#f0ad4e,color:#000,stroke:#c78c2e
style E fill:#f0ad4e,color:#000,stroke:#c78c2e
style F fill:#5bc0de,color:#000,stroke:#3aa8c1
style xrpldNode fill:#1a2633,color:#ccc,stroke:#4a90d9
style collector fill:#1a3320,color:#ccc,stroke:#5cb85c
style backends fill:#332a1a,color:#ccc,stroke:#f0ad4e
style metrics fill:#332a1a,color:#ccc,stroke:#f0ad4e
style viz fill:#1a2d33,color:#ccc,stroke:#5bc0de
```
There are two independent telemetry pipelines entering a single **OTel Collector**:
1. **OpenTelemetry Traces** — Distributed spans with attributes, exported via OTLP/HTTP (:4318) to the collector's **OTLP Receiver**. The **Batch Processor** groups spans (1s timeout, batch size 100) before forwarding to trace backends. The **SpanMetrics Connector** derives RED metrics (rate, errors, duration) from every span and feeds them into the metrics pipeline.
2. **beast::insight StatsD** — System-level gauges, counters, and timers emitted as StatsD UDP packets to port :8125, ingested by the collector's **StatsD Receiver**, and exported alongside span-derived metrics to Prometheus.
A third, narrower metrics path exists for instruments created at their call site through the
`XRPL_METRIC_*` macros. These use the OTel Metrics SDK directly and reach the collector's OTLP
receiver rather than the StatsD receiver, so their names carry no `xrpld_` prefix. See
[§2a](#2a-call-site-otel-metrics-metricsregistry). Code in `libxrpl` cannot use these macros and
always goes through `beast::insight` instead.
**Trace backend** — The collector exports traces via OTLP/gRPC to:
- **Grafana Tempo** — Preferred trace backend. Supports TraceQL queries at `:3200`, S3/GCS object storage for cost-effective long-term trace retention, and integrates natively with Grafana.
> **Further reading**: [00-tracing-fundamentals.md](./00-tracing-fundamentals.md) for core OpenTelemetry concepts (traces, spans, context propagation, sampling). [07-observability-backends.md](./07-observability-backends.md) for production backend selection, collector placement, and sampling strategies.
---
## 1. OpenTelemetry Spans
### 1.1 Complete Span Inventory (35 spans)
> **See also**: [02-design-decisions.md §2.3](./02-design-decisions.md#23-span-naming-conventions) for naming conventions and the full span catalog with rationale. [04-code-samples.md §4.6](./04-code-samples.md#46-span-flow-visualization) for span flow diagrams.
#### RPC Spans
Controlled by `trace_rpc=1` in `[telemetry]` config.
| Span Name | Parent | Source File | Description |
| -------------------- | ------------------ | ----------------- | ------------------------------------------------------------------------ |
| `rpc.http_request` | — | ServerHandler.cpp | Top-level HTTP RPC request entry point |
| `rpc.process` | `rpc.http_request` | ServerHandler.cpp | RPC processing pipeline |
| `rpc.ws_message` | — | ServerHandler.cpp | WebSocket message handling |
| `rpc.ws_upgrade` | — | ServerHandler.cpp | WebSocket upgrade handshake (error path) |
| `rpc.command.<name>` | `rpc.process` | RPCHandler.cpp | Per-command span (e.g., `rpc.command.server_info`, `rpc.command.ledger`) |
**Where to find**: Tempo → TraceQL: `{resource.service.name="xrpld" && name=~"rpc.http_request|rpc.command.*"}`
**Grafana dashboard**: _RPC Performance_ (`rpc-performance`)
#### Transaction Spans
Controlled by `trace_transactions=1` in `[telemetry]` config.
| Span Name | Parent | Source File | Description |
| --------------- | -------------- | --------------- | ----------------------------------------------------------------- |
| `tx.process` | — | NetworkOPs.cpp | Transaction submission entry point (local or peer-relayed) |
| `tx.receive` | — | PeerImp.cpp | Raw transaction received from peer overlay (before deduplication) |
| `tx.apply` | `ledger.build` | BuildLedger.cpp | Transaction set applied to new ledger during consensus |
| `tx.preflight` | — | applySteps.cpp | Stateless checks stage (`stage=preflight`) |
| `tx.preclaim` | — | applySteps.cpp | Ledger-aware checks stage before fee claim (`stage=preclaim`) |
| `tx.transactor` | — | Transactor.cpp | Apply stage — the transactor runs (`stage=apply`) |
The three apply-pipeline spans share a deterministic `trace_id` derived from
`txID[0:16]`, so preflight, preclaim, and transactor for one transaction group
under a single trace even though they run sequentially and often on different
threads. A transaction that hard-fails preflight or preclaim never reaches the
later spans — the `stage` attribute identifies where it stopped.
> **Deterministic roots are true roots.** Spans with a deterministic `trace_id`
> (the `tx.*` apply pipeline, `tx.process`, `tx.receive`, and `consensus.round`)
> are emitted as genuine trace roots with an empty `parent_span_id`. The chosen
> `trace_id` is injected through a custom `DeterministicIdGenerator` on the SDK's
> no-parent branch, so there is no synthetic placeholder parent — Tempo shows a
> clean root, not a "root span not yet received" warning. Cross-node correlation
> still works because every node derives the same `trace_id` from the shared hash.
> **Log-trace correlation is retained across coroutines.** OTel context storage
> is coroutine-aware (backed by `LocalValue`), so the active span travels with a
> coroutine across `yield()` and resumes on whatever thread the scheduler picks.
> RPC, consensus, and transaction spans therefore keep per-line log-trace
> correlation, and their scopes are safe across coroutine yields. Job-handoff
> spans — transaction apply and receive, and consensus accept — are activated
> inside their worker bodies rather than at enqueue, so each worker's log lines
> carry the span's trace context.
**Where to find**: Tempo → TraceQL: `{resource.service.name="xrpld" && name=~"tx.process|tx.receive"}`
or, for the apply pipeline: `{resource.service.name="xrpld" && name=~"tx.preflight|tx.preclaim|tx.transactor"}`
**Grafana dashboard**: _Transaction Overview_ (`transaction-overview`)
#### PathFind Spans
Controlled by `trace_rpc=1` in `[telemetry]` config (pathfinding spans fire within RPC request handling).
| Span Name | Parent | Source File | Description |
| --------------------- | -------------------- | ---------------- | -------------------------------------------------------- |
| `pathfind.request` | `rpc.command.<name>` | PathFind.cpp | RPC entry for path_find (doPathFind) |
| `pathfind.compute` | `pathfind.request` | PathRequest.cpp | Single path computation (doUpdate) |
| `pathfind.update_all` | — | PathRequests.cpp | Async recomputation of all active path requests on close |
| `pathfind.discover` | `pathfind.compute` | Pathfinder.cpp | Graph exploration phase (Pathfinder::find) |
| `pathfind.rank` | `pathfind.compute` | Pathfinder.cpp | Path ranking and selection phase |
> **Note**: `pathfind.request` nests under the active `rpc.command.<name>` span.
> Because OTel context storage is coroutine-aware (backed by `LocalValue`), the
> `rpc.command.*` scope stays correct even though its generic dispatch
> (`callMethod`) also wraps handlers such as `doRipplePathFind` whose span is held
> across a coroutine `yield()` — the ambient context travels with the coroutine
> when it resumes, so there is no wrong-thread scope pop. The
> `pathfind.request → compute → discover` sub-tree therefore parents to
> `rpc.command.<name>`, giving an exact request-to-pathfind nesting.
**Where to find**: Tempo → TraceQL: `{resource.service.name="xrpld" && name=~"pathfind.*"}`
**Grafana dashboard**: _RPC & Pathfinding (StatsD)_ (`xrpld-statsd-rpc`) for StatsD timers; span-derived metrics via _RPC Performance_ (`rpc-performance`)
#### TxQ Spans
Controlled by `trace_transactions=1` in `[telemetry]` config.
| Span Name | Parent | Source File | Description |
| ------------------ | ----------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `txq.enqueue` | `tx.process` (submission path; root on open-ledger rebuild) | TxQ.cpp | Queue admission decision (apply/queue/reject). Parents to `tx.process` via explicit context on submit; correlates via `current_ledger_seq` on all paths |
| `txq.apply_direct` | `txq.enqueue` | TxQ.cpp | Direct application attempt (bypassing queue) |
| `txq.batch_clear` | `txq.enqueue` | TxQ.cpp | Batch clear of account's queued transactions |
| `txq.accept` | — | TxQ.cpp | Ledger-close accept loop (drain queued transactions) |
| `txq.accept.tx` | `txq.accept` | TxQ.cpp | Per-transaction apply within accept loop |
| `txq.cleanup` | — | TxQ.cpp | Post-close cleanup (expire old transactions) |
**Where to find**: Tempo → TraceQL: `{resource.service.name="xrpld" && name=~"txq.*"}`
**Grafana dashboard**: _Transaction Overview_ (`transaction-overview`)
#### gRPC Spans
Controlled by `trace_rpc=1` in `[telemetry]` config.
| Span Name | Parent | Source File | Description |
| -------------- | ------ | -------------- | ----------------------------------------------------------------------------- |
| `grpc.request` | — | GRPCServer.cpp | Single gRPC request (GetLedger, GetLedgerData, GetLedgerDiff, GetLedgerEntry) |
**Where to find**: Tempo → TraceQL: `{resource.service.name="xrpld" && name="grpc.request"}`
#### Consensus Spans
Controlled by `trace_consensus=1` in `[telemetry]` config.
| Span Name | Parent | Source File | Description |
| ---------------------------- | ----------------- | ---------------- | ----------------------------------------------------- |
| `consensus.round` | — | RCLConsensus.cpp | Top-level round span (deterministic trace ID) |
| `consensus.proposal.send` | `consensus.round` | RCLConsensus.cpp | Node broadcasts its transaction set proposal |
| `consensus.ledger_close` | `consensus.round` | RCLConsensus.cpp | Ledger close event triggered by consensus |
| `consensus.establish` | `consensus.round` | Consensus.h | Establish phase — convergence loop |
| `consensus.update_positions` | `consensus.round` | Consensus.h | Update positions during establish phase |
| `consensus.check` | `consensus.round` | Consensus.h | Check for consensus agreement |
| `consensus.accept` | `consensus.round` | RCLConsensus.cpp | Consensus accepts a ledger (round complete) |
| `consensus.accept.apply` | `consensus.round` | RCLConsensus.cpp | Ledger application with close time details |
| `consensus.validation.send` | `consensus.round` | RCLConsensus.cpp | Validation message sent after ledger accepted |
| `consensus.mode_change` | `consensus.round` | RCLConsensus.cpp | Consensus mode transition (e.g., tracking->proposing) |
> **Note**: `toDisplayString(ConsensusMode)` (in `ConsensusTypes.h`) provides Title Case display names for mode attribute values: `"Proposing"`, `"Observing"`, `"Wrong Ledger"`, `"Switched Ledger"`. This is separate from `to_string()` which returns stable log-format strings.
**Where to find**: Tempo → TraceQL: `{resource.service.name="xrpld" && name=~"consensus.*"}`
**Grafana dashboard**: _Consensus Health_ (`consensus-health`)
#### Ledger Spans
Controlled by `trace_ledger=1` in `[telemetry]` config.
| Span Name | Parent | Source File | Description |
| ----------------- | ------ | ---------------- | ---------------------------------------------- |
| `ledger.build` | — | BuildLedger.cpp | Build new ledger from accepted transaction set |
| `ledger.validate` | — | LedgerMaster.cpp | Ledger promoted to validated status |
| `ledger.store` | — | LedgerMaster.cpp | Ledger stored to database/history |
**Where to find**: Tempo → TraceQL: `{resource.service.name="xrpld" && name=~"ledger.*"}`
**Grafana dashboard**: _Ledger Operations_ (`ledger-operations`)
#### Peer Spans
Controlled by `trace_peer` in `[telemetry]` config. **Enabled by default** (high volume).
| Span Name | Parent | Source File | Description |
| ------------------------- | ------ | ----------- | ------------------------------------- |
| `peer.proposal.receive` | — | PeerImp.cpp | Consensus proposal received from peer |
| `peer.validation.receive` | — | PeerImp.cpp | Validation message received from peer |
A `—` parent means the span is a fresh trace root (`kConsumer`): it is started
via `ScopedSpanGuard::freshRoot()` at the inbound-message entry point and never
inherits an ambient span left active on the peer thread, so it does not nest
under an unrelated transaction's trace.
**Where to find**: Tempo → TraceQL: `{resource.service.name="xrpld" && name=~"peer.*"}`
**Grafana dashboard**: _Peer Network_ (`peer-network`)
---
### 1.2 Complete Attribute Inventory (83 attributes)
> **See also**: [02-design-decisions.md §2.4.2](./02-design-decisions.md#242-span-attributes-by-category) for attribute design rationale and privacy considerations.
Every span can carry key-value attributes that provide context for filtering and aggregation.
#### RPC Attributes
| Attribute | Type | Set On | Description |
| ---------------------- | ------ | --------------- | ------------------------------------------------ |
| `command` | string | `rpc.command.*` | RPC command name (e.g., `server_info`, `ledger`) |
| `version` | int64 | `rpc.command.*` | API version number |
| `rpc_role` | string | `rpc.command.*` | Caller role: `"admin"` or `"user"` |
| `rpc_status` | string | `rpc.command.*` | Result: `"success"` or `"error"` |
| `request_payload_size` | int64 | `rpc.command.*` | Request payload size in bytes |
**Tempo query**: `{span.command="server_info"}` to find all `server_info` calls.
**Prometheus label**: `xrpl_rpc_command` (dots converted to underscores by SpanMetrics).
#### Transaction Attributes
| Attribute | Type | Set On | Description |
| --------------------- | ------- | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| `tx_hash` | string | `tx.process`, `tx.receive` | Transaction hash (hex-encoded) |
| `local` | boolean | `tx.process` | `true` if locally submitted, `false` if peer-relayed |
| `path` | string | `tx.process` | Submission path: `"sync"` or `"async"` |
| `suppressed` | boolean | `tx.receive` | `true` if transaction was suppressed (duplicate) |
| `tx_status` | string | `tx.receive` | Transaction status (e.g., `"known_bad"`) |
| `peer_id` | int64 | `tx.receive` | Peer identifier (also set on peer spans) |
| `peer_version` | string | `tx.receive` | Peer protocol version string |
| `stage` | string | `tx.preflight`, `tx.preclaim`, `tx.transactor` | Apply-pipeline stage: `preflight`, `preclaim`, or `apply` |
| `tx_type` | string | `tx.preflight`, `tx.preclaim`, `tx.transactor` | Transaction type name (e.g., `Payment`) |
| `ter_result` | string | `tx.preflight`, `tx.preclaim`, `tx.transactor` | Engine result token for that stage (e.g., `tesSUCCESS`, `terPRE_SEQ`) |
| `applied` | boolean | `tx.transactor` | `true` if the transaction was applied to the ledger |
| `current_ledger_seq` | int64 | `tx.process`, `tx.receive`, `tx.preclaim`, `tx.transactor` | Seq of the ledger being worked on (open/in-flight, not established) — joins the txID-keyed spans to the ledger trace |
| `current_ledger_hash` | string | `tx.preclaim`, `tx.transactor` | Parent hash of that ledger (= `consensus.round` trace-id seed on the build path). View-bearing stages only; `tx.preflight` omits both |
**Tempo query**: `{span.tx_hash="<hash>"}` to trace a specific transaction across nodes.
Join a transaction's work to its ledger with `{span.current_ledger_seq=<N>}`.
**Prometheus label**: `xrpl_tx_local` (used as SpanMetrics dimension).
#### PathFind Attributes
| Attribute | Type | Set On | Description |
| ----------------------- | ------- | --------------------- | ----------------------------------------------- |
| `source_account` | string | `pathfind.request` | Source account address |
| `dest_account` | string | `pathfind.request` | Destination account address |
| `fast` | boolean | `pathfind.compute` | Whether this is a fast (non-full) pathfind |
| `search_level` | int64 | `pathfind.compute` | Search depth level |
| `num_complete_paths` | int64 | `pathfind.compute` | Number of complete paths found |
| `num_paths` | int64 | `pathfind.compute` | Total number of paths explored |
| `num_requests` | int64 | `pathfind.update_all` | Number of active path requests being recomputed |
| `pathfind_ledger_index` | int64 | `pathfind.update_all` | Ledger index used for recomputation |
**Tempo query**: `{span.source_account="rHb9..."}` to find pathfind requests from a specific account.
#### TxQ Attributes
| Attribute | Type | Set On | Description |
| --------------------- | ------- | ------------------------------ | -------------------------------------------------------------------------------- |
| `tx_hash` | string | `txq.enqueue`, `txq.accept.tx` | Transaction hash in the queue |
| `current_ledger_seq` | int64 | `txq.enqueue` | Seq of the ledger being worked on — correlates the enqueue to the ledger trace |
| `current_ledger_hash` | string | `txq.enqueue` | Parent hash of that ledger (= `consensus.round` trace-id seed on the build path) |
| `txq_status` | string | `txq.enqueue` | Queue result: `"queued"`, `"applied_direct"`, `"rejected"` |
| `fee_level_paid` | int64 | `txq.enqueue` | Fee level paid by the transaction |
| `required_fee_level` | int64 | `txq.enqueue` | Minimum fee level required for queue admission |
| `queue_size` | int64 | `txq.accept` | Queue depth at start of accept |
| `ledger_changed` | boolean | `txq.accept` | Whether the open ledger changed since last accept |
| `ledger_seq` | int64 | `txq.cleanup` | Ledger sequence for cleanup |
| `expired_count` | int64 | `txq.cleanup` | Number of expired transactions removed |
| `ter_code` | string | `txq.accept.tx` | Transaction engine result code |
| `retries_remaining` | int64 | `txq.accept.tx` | Remaining retry attempts for this transaction |
| `num_cleared` | int64 | `txq.batch_clear` | Number of transactions cleared in batch |
**Tempo query**: `{span.txq_status="rejected"}` to find rejected queue attempts.
#### gRPC Attributes
| Attribute | Type | Set On | Description |
| ------------ | ------ | -------------- | ------------------------------------------------------------ |
| `method` | string | `grpc.request` | gRPC method name (e.g., `GetLedger`, `GetLedgerData`) |
| `rpc_role` | string | `grpc.request` | Caller role: `"admin"` or `"user"` |
| `rpc_status` | string | `grpc.request` | Result: `"success"`, `"error"`, `"resource_exhausted"`, etc. |
**Tempo query**: `{span.method="GetLedger"}` to find gRPC ledger requests.
#### Consensus Attributes
| Attribute | Type | Set On | Description |
| --------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
| `consensus_ledger_id` | string | `consensus.round` | Previous ledger hash (used for deterministic trace ID) |
| `ledger_seq` | int64 | `consensus.round`, `consensus.ledger_close`, `consensus.accept`, `consensus.validation.send`, `consensus.accept.apply` | Ledger sequence number |
| `consensus_mode` | string | `consensus.round`, `consensus.proposal.send`, `consensus.ledger_close` | Node mode via `toDisplayString()`: `"Proposing"`, `"Observing"`, etc. |
| `consensus_round` | int64 | `consensus.proposal.send` | Consensus round number |
| `proposers` | int64 | `consensus.proposal.send`, `consensus.accept` | Number of proposers in the round |
| `round_time_ms` | int64 | `consensus.accept`, `consensus.accept.apply` | Total consensus round duration in milliseconds |
| `proposing` | boolean | `consensus.validation.send` | Whether this node was a proposer |
| `consensus_state` | string | `consensus.accept.apply` | Consensus outcome: `"finished"` or `"moved_on"` |
| `close_time` | int64 | `consensus.accept.apply` | Agreed-upon ledger close time (epoch seconds) |
| `close_time_correct` | boolean | `consensus.accept.apply` | Whether validators reached agreement on close time |
| `close_resolution_ms` | int64 | `consensus.accept.apply` | Close time rounding granularity in milliseconds |
| `parent_close_time` | int64 | `consensus.accept.apply` | Parent ledger's close time (epoch seconds) |
| `close_time_self` | int64 | `consensus.accept.apply` | This node's proposed close time |
| `close_time_vote_bins` | string | `consensus.accept.apply` | Histogram of close time votes from validators |
| `resolution_direction` | string | `consensus.accept.apply` | Resolution change: `"increased"`, `"decreased"`, or `"unchanged"` |
| `converge_percent` | int64 | `consensus.establish` | Convergence percentage threshold |
| `establish_count` | int64 | `consensus.establish` | Number of establish iterations completed |
| `proposers_agreed` | int64 | `consensus.establish` | Number of proposers that agreed on this round |
| `avalanche_threshold` | int64 | `consensus.update_positions` | Avalanche threshold for dispute resolution |
| `close_time_threshold` | int64 | `consensus.update_positions` | Close time agreement threshold |
| `have_close_time_consensus` | boolean | `consensus.update_positions` | Whether close time consensus has been reached |
| `agree_count` | int64 | `consensus.check` | Number of proposers that agree with our position |
| `disagree_count` | int64 | `consensus.check` | Number of proposers that disagree with our position |
| `threshold_percent` | int64 | `consensus.check` | Required agreement threshold percentage |
| `consensus_result` | string | `consensus.check` | Check result: `"yes"`, `"no"`, or `"expired"` |
| `quorum` | int64 | `consensus.check` | Required quorum for validation |
| `validation_count` | int64 | `consensus.check` | Number of validations received |
| `trace_strategy` | string | `consensus.round` | Trace sampling strategy used for this round |
| `consensus_round_id` | string | `consensus.round` | Deterministic round identifier |
| `mode_old` | string | `consensus.mode_change` | Previous consensus mode |
| `mode_new` | string | `consensus.mode_change` | New consensus mode |
| `tx_id` | string | `consensus.update_positions` | Disputed transaction ID |
| `dispute_our_vote` | boolean | `consensus.update_positions` | Our vote on the disputed transaction |
| `dispute_yays` | int64 | `consensus.update_positions` | Number of proposers voting to include |
| `dispute_nays` | int64 | `consensus.update_positions` | Number of proposers voting to exclude |
**Tempo query**: `{span.consensus_mode="Proposing"}` to find rounds where node was proposing.
**Prometheus label**: `xrpl_consensus_mode` (used as SpanMetrics dimension).
#### Ledger Attributes
| Attribute | Type | Set On | Description |
| --------------------- | ------- | ------------------------------------------------------------- | ------------------------------------------------ |
| `ledger_seq` | int64 | `ledger.build`, `ledger.validate`, `ledger.store`, `tx.apply` | Ledger sequence number |
| `close_time` | int64 | `ledger.build` | Ledger close time (epoch seconds) |
| `close_time_correct` | boolean | `ledger.build` | Whether close time was agreed upon by validators |
| `close_resolution_ms` | int64 | `ledger.build` | Close time rounding granularity in milliseconds |
| `tx_count` | int64 | `ledger.build`, `tx.apply` | Transactions in the ledger |
| `tx_failed` | int64 | `ledger.build`, `tx.apply` | Failed transactions in the ledger |
| `validations` | int64 | `ledger.validate` | Number of validations received for this ledger |
**Tempo query**: `{span.ledger_seq=12345}` to find all spans for a specific ledger.
#### Peer Attributes
| Attribute | Type | Set On | Description |
| -------------------- | ------- | ---------------------------------------------------------------- | ---------------------------------------------------- |
| `peer_id` | int64 | `tx.receive`, `peer.proposal.receive`, `peer.validation.receive` | Peer identifier |
| `proposal_trusted` | boolean | `peer.proposal.receive` | Whether the proposal came from a trusted validator |
| `ledger_hash` | string | `peer.validation.receive` | Ledger hash the validation refers to |
| `validation_full` | boolean | `peer.validation.receive` | Whether this is a full (not partial) validation |
| `validation_trusted` | boolean | `peer.validation.receive` | Whether the validation came from a trusted validator |
**Prometheus labels**: `xrpl_peer_proposal_trusted`, `xrpl_peer_validation_trusted` (SpanMetrics dimensions).
---
### 1.3 SpanMetrics — Derived Prometheus Metrics
> **See also**: [01-architecture-analysis.md](./01-architecture-analysis.md) §1.8.2 for how span-derived metrics map to operational insights.
The OTel Collector's SpanMetrics connector automatically generates RED (Rate, Errors, Duration) metrics from every span. No custom metrics code in xrpld is needed.
| Prometheus Metric | Type | Description |
| -------------------------------------------------- | --------- | ------------------------------------------------------------------------------ |
| `traces_span_metrics_calls_total` | Counter | Total span invocations |
| `traces_span_metrics_duration_milliseconds_bucket` | Histogram | Latency distribution (buckets: 1, 5, 10, 25, 50, 100, 250, 500, 1000, 5000 ms) |
| `traces_span_metrics_duration_milliseconds_count` | Histogram | Observation count |
| `traces_span_metrics_duration_milliseconds_sum` | Histogram | Cumulative latency |
**Standard labels on every metric**: `span_name`, `status_code`, `service_name`, `span_kind`
**Additional dimension labels** (configured in `otel-collector-config.yaml`):
| Span Attribute | Prometheus Label | Applies To |
| -------------------- | ------------------------------ | ---------------------------------------------- |
| `command` | `xrpl_rpc_command` | `rpc.command.*` |
| `rpc_status` | `xrpl_rpc_status` | `rpc.command.*` |
| `consensus_mode` | `xrpl_consensus_mode` | `consensus.ledger_close` |
| `local` | `xrpl_tx_local` | `tx.process` |
| `proposal_trusted` | `xrpl_peer_proposal_trusted` | `peer.proposal.receive` |
| `validation_trusted` | `xrpl_peer_validation_trusted` | `peer.validation.receive` |
| `stage` | `stage` | `tx.preflight`, `tx.preclaim`, `tx.transactor` |
The `stage` dimension (3 values: `preflight`, `preclaim`, `apply`) turns the
apply-pipeline spans into per-stage RED metrics with no native instruments — the
_Transaction Overview_ dashboard charts rate, p95 latency, and failure rate by stage.
> **Sampling caveat**: xrpld head sampling is fixed at 1.0 (every trace is
> recorded), so span-derived metrics are not undercounted at the node. If the
> collector is configured with tail sampling, span-derived metrics reflect only
> the retained traces, whereas native StatsD/meter metrics do not sample.
> Account for any collector-side tail sampling when reading absolute stage rates.
**Where to query**: Prometheus → `traces_span_metrics_calls_total{span_name="rpc.command.server_info"}`
---
## 2. StatsD Metrics (beast::insight)
> **See also**: [02-design-decisions.md](./02-design-decisions.md) for the beast::insight coexistence design. [06-implementation-phases.md](./06-implementation-phases.md) for the Phase 6 metric inventory.
These are system-level metrics emitted by xrpld's `beast::insight` framework via StatsD UDP. They cover operational data that doesn't map to individual trace spans.
### Configuration
```ini
[insight]
server=statsd
address=127.0.0.1:8125
prefix=xrpld
```
> **Note**: The `prefix` value is user-configurable — all metric names in the tables below assume `prefix=xrpld` (matching the integration test and Grafana dashboards). If you change the prefix, replace `xrpld_` with `{your_prefix}_` in all PromQL queries.
### 2.1 Gauges
| Prometheus Metric | Source File | Description | Typical Range |
| ------------------------------------------------- | --------------------- | ---------------------------------------- | ------------------------------- |
| `xrpld_LedgerMaster_Validated_Ledger_Age` | LedgerMaster.h | Seconds since last validated ledger | 0–10 (healthy), >30 (stale) |
| `xrpld_LedgerMaster_Published_Ledger_Age` | LedgerMaster.h | Seconds since last published ledger | 0–10 (healthy) |
| `xrpld_State_Accounting_Disconnected_duration` | NetworkOPs.cpp | Cumulative seconds in Disconnected state | Monotonic |
| `xrpld_State_Accounting_Connected_duration` | NetworkOPs.cpp | Cumulative seconds in Connected state | Monotonic |
| `xrpld_State_Accounting_Syncing_duration` | NetworkOPs.cpp | Cumulative seconds in Syncing state | Monotonic |
| `xrpld_State_Accounting_Tracking_duration` | NetworkOPs.cpp | Cumulative seconds in Tracking state | Monotonic |
| `xrpld_State_Accounting_Full_duration` | NetworkOPs.cpp | Cumulative seconds in Full state | Monotonic (should dominate) |
| `xrpld_State_Accounting_Disconnected_transitions` | NetworkOPs.cpp | Count of transitions to Disconnected | Low |
| `xrpld_State_Accounting_Connected_transitions` | NetworkOPs.cpp | Count of transitions to Connected | Low |
| `xrpld_State_Accounting_Syncing_transitions` | NetworkOPs.cpp | Count of transitions to Syncing | Low |
| `xrpld_State_Accounting_Tracking_transitions` | NetworkOPs.cpp | Count of transitions to Tracking | Low |
| `xrpld_State_Accounting_Full_transitions` | NetworkOPs.cpp | Count of transitions to Full | Low (should be 1 after startup) |
| `xrpld_Peer_Finder_Active_Inbound_Peers` | PeerfinderManager.cpp | Active inbound peer connections | 0–85 |
| `xrpld_Peer_Finder_Active_Outbound_Peers` | PeerfinderManager.cpp | Active outbound peer connections | 10–21 |
| `xrpld_Overlay_Peer_Disconnects` | OverlayImpl.cpp | Cumulative peer disconnection count | Low growth |
| `xrpld_jobq_job_count` | JobQueue.cpp | Current job queue depth (all types) | 0–100 (healthy) |
| `xrpld_Node_family_full_below_cache_size` | TaggedCache.h | FullBelowCache entry count | Varies |
| `xrpld_Node_family_full_below_cache_hit_rate` | TaggedCache.h | FullBelowCache hit rate percentage | 0–100 |
**Grafana dashboard**: _Node Health (StatsD)_ (`xrpld-statsd-node-health`)
### 2.2 Counters
| Prometheus Metric | Source File | Description |
| ------------------------------- | ------------------ | --------------------------------------------- |
| `xrpld_rpc_requests` | ServerHandler.cpp | Total RPC requests received |
| `xrpld_ledger_fetches` | InboundLedgers.cpp | Inbound ledger fetch attempts |
| `xrpld_ledger_history_mismatch` | LedgerHistory.cpp | Ledger hash mismatches detected |
| `xrpld_warn` | Logic.h | Resource manager warnings issued |
| `xrpld_drop` | Logic.h | Resource manager drops (connections rejected) |
**Note**: `xrpld_warn` and `xrpld_drop` use non-standard StatsD meter type (`|m`). The OTel StatsD receiver only recognizes `|c`, `|g`, `|ms`, `|h`, `|s` — these metrics may be silently dropped. See Known Issues below.
**Grafana dashboard**: _RPC & Pathfinding (StatsD)_ (`xrpld-statsd-rpc`)
### 2.3 Histograms (from StatsD timers)
| Prometheus Metric | Source File | Unit | Description |
| --------------------- | ----------------- | ----- | ------------------------------ |
| `xrpld_rpc_time` | ServerHandler.cpp | ms | RPC response time distribution |
| `xrpld_rpc_size` | ServerHandler.cpp | bytes | RPC response size distribution |
| `xrpld_ios_latency` | Application.cpp | ms | I/O service loop latency |
| `xrpld_pathfind_fast` | PathRequests.h | ms | Fast pathfinding duration |
| `xrpld_pathfind_full` | PathRequests.h | ms | Full pathfinding duration |
Quantiles collected: 0th, 50th, 90th, 95th, 99th, 100th percentile.
**Grafana dashboards**: _Node Health_ (`ios_latency`), _RPC & Pathfinding_ (`rpc_time`, `rpc_size`, `pathfind_*`)
### 2.4 Overlay Traffic Metrics
For each of the 45+ overlay traffic categories (defined in `TrafficCount.h`), four gauges are emitted:
- `xrpld_{category}_Bytes_In`
- `xrpld_{category}_Bytes_Out`
- `xrpld_{category}_Messages_In`
- `xrpld_{category}_Messages_Out`
**Key categories**:
| Category | Description |
| ----------------------------------------------------------------- | -------------------------- |
| `total` | All traffic aggregated |
| `overhead` / `overhead_overlay` | Protocol overhead |
| `transactions` / `transactions_duplicate` | Transaction relay |
| `proposals` / `proposals_untrusted` / `proposals_duplicate` | Consensus proposals |
| `validations` / `validations_untrusted` / `validations_duplicate` | Consensus validations |
| `ledger_data_get` / `ledger_data_share` | Ledger data exchange |
| `ledger_data_Transaction_Node_get/share` | Transaction node data |
| `ledger_data_Account_State_Node_get/share` | Account state node data |
| `ledger_data_Transaction_Set_candidate_get/share` | Transaction set candidates |
| `getObject` / `haveTxSet` / `ledgerData` | Object requests |
| `ping` / `status` | Keepalive and status |
| `set_get` | Set requests |
**Grafana dashboards**: _Network Traffic_ (`xrpld-statsd-network`), _Overlay Traffic Detail_ (`xrpld-statsd-overlay-detail`), _Ledger Data & Sync_ (`xrpld-statsd-ledger-sync`)
### 2.5 Per-Job Timer Events
For each of the 35 non-special job types (defined in `JobTypes.h`), two StatsD timer events are emitted:
- `xrpld_jobq_{jobName}` — execution duration
- `xrpld_jobq_{jobName}_q` — dequeue wait time
The `jobq` segment is added at runtime by the collector group the JobQueue is constructed with —
see [§2.6](#26-per-job-type-saturation-gauges) for the derivation, which applies to all `jobq_*`
metrics including the queue-depth gauge in §2.1.
These produce summary metrics with quantiles (0th, 50th, 90th, 95th, 99th, 100th).
**Key job types** (most operationally relevant):
| Job Name | Source Enum | Description |
| ------------------- | ---------------- | ----------------------------- |
| `acceptLedger` | `jtACCEPT` | Consensus round acceptance |
| `advanceLedger` | `jtADVANCE` | Ledger advancement |
| `transaction` | `jtTRANSACTION` | Transaction processing |
| `writeObjects` | `jtWRITE` | Database object writes |
| `publishNewLedger` | `jtPUBLEDGER` | New ledger publication |
| `trustedValidation` | `jtVALIDATION_t` | Trusted validation processing |
| `trustedProposal` | `jtPROPOSAL_t` | Trusted proposal processing |
| `clientRPC` | `jtCLIENT_RPC` | Client RPC request handling |
| `heartbeat` | `jtNETOP_TIMER` | Network heartbeat timer |
| `sweep` | `jtSWEEP` | Cache sweep / cleanup |
| `ledgerData` | `jtLEDGER_DATA` | Ledger data processing |
Special job types (`limit=0`: `peerCommand`, `diskAccess`, `processTransaction`, `orderBookSetup`, `pathFind`, `nodeRead`, `nodeWrite`, `generic`, `SyncReadNode`, `AsyncReadNode`, `WriteNode`) do **not** emit timer events.
**Grafana dashboard**: _Node Health (StatsD)_ (`xrpld-statsd-node-health`) — Key Jobs and All Jobs panels
### 2.6 Per-Job-Type Saturation Gauges
`JobTypeData` has always tracked three per-type counters — `waiting`, `running`, and `deferred`
(`include/xrpl/core/JobTypeData.h`) — but only the process-wide queue depth was exported. These
three gauge families export them per job type, so queue pressure can be attributed to a type
instead of only being visible as a single total.
For each of the same 35 non-special job types, three gauges are created in the `JobTypeData`
constructor and assigned in `JobQueue::collect()`. Note that the gauge members do not exist on
this branch — they are introduced on phase-9, which is where the creation and assignment live.
| Metric | Source File | Description |
| ------------------------------- | ------------- | ----------------------------------------------- |
| `xrpld_jobq_{jobName}_waiting` | JobTypeData.h | Jobs enqueued for this type but not yet started |
| `xrpld_jobq_{jobName}_running` | JobTypeData.h | Jobs of this type currently executing |
| `xrpld_jobq_{jobName}_deferred` | JobTypeData.h | Jobs held back by this type's concurrency limit |
**Why `deferred` matters most.** `JobQueue::addJob()` never rejects — when a type is at its
concurrency limit it increments `deferred` and returns `true` (`JobQueue.cpp`,
`addRefCountedJob`). Backpressure on a capped type therefore shows up only as latency, after the
delay has already happened. A non-zero `deferred` reading precedes that, which makes it the
leading indicator for a saturating type. It is structurally always zero for special types, which
is why they are excluded — the same `!info.special()` guard that already gates the timer events.
**Name derivation.** These names are not literals anywhere in the source; the `jobq` segment is
added at runtime. `Application.cpp` constructs the JobQueue with
`collectorManager_->group("jobq")`, and `GroupImp::makeName()` (`Groups.cpp`) joins the group name
and the metric name with a `.`. The StatsD sender then prefixes the configured `prefix` the same
way, so `JtLedgerReq` (whose type name is `ledgerRequest`) is sent as
`xrpld.jobq.ledgerRequest_deferred` and arrives in Prometheus as
`xrpld_jobq_ledgerRequest_deferred`. The same `jobq.` prefix applies to the §2.5 timer events and
to the queue-depth gauge in §2.1.
> **Sampling granularity**: these are gauges read by the collector hook once per second, not
> counters incremented at the event. A `deferred` spike shorter than the sampling interval can be
> missed. Saturation long enough to delay ledger sync persists across a sample, which is the case
> these gauges are for.
> **Pipeline**: `JobQueue.cpp` is in `libxrpl` and reaches the exporter through
> `beast::insight::Collector`, like every other metric in this section. It cannot use the
> `xrpld`-only call-site metric macros used by
> [§2a](#2a-call-site-otel-metrics-metricsregistry).
---
## 2a. Call-Site OTel Metrics (MetricsRegistry)
> **See also**: [§2](#2-statsd-metrics-beastinsight) for the `beast::insight` metrics that use the
> StatsD transport instead.
Every metric in [§2](#2-statsd-metrics-beastinsight) goes through `beast::insight`. The metrics
below are created directly at their call site through the `XRPL_METRIC_*` macros
(`src/xrpld/telemetry/MetricMacros.h`) and exported by the OTel Metrics SDK rather than over
StatsD. The macros create each instrument once and expand to an empty statement when telemetry is
compiled out, so a call site needs no conditional compilation.
These names carry no `xrpld_` prefix: the prefix belongs to the StatsD sender, and this path
identifies the node through OTel resource attributes instead.
> **Note**: this family is emitted by `MetricsRegistry`, which arrives with the native-metrics
> work in a later phase than this document's own StatsD pipeline. It is catalogued here because
> this document is the single source of truth for the whole telemetry surface.
### 2a.1 Job Lifecycle Metrics
Recorded from the three `PerfLog` job hooks that `JobQueue` already calls.
| Metric | Type | Unit | Labels | Description |
| -------------------- | --------- | ---- | --------------------- | ---------------------------- |
| `job_queued_total` | Counter | — | `job_type`, `handler` | Jobs enqueued |
| `job_started_total` | Counter | — | `job_type`, `handler` | Jobs started |
| `job_finished_total` | Counter | — | `job_type`, `handler` | Jobs completed |
| `job_queued_us` | Histogram | µs | `job_type`, `handler` | Queue wait time distribution |
| `job_running_us` | Histogram | µs | `job_type`, `handler` | Execution time distribution |
`job_queued_us` and `job_running_us` register explicit microsecond buckets spanning 100 µs to
60 s. Without that view the SDK default buckets stop at 10 ms and every quantile reads as 10 ms.
#### The `handler` Label
`job_type` names the queue a job ran on, not who produced it. Several producers share one type —
`RcvGetLedger` and `RcvGetObjByHash` both run as `JtLedgerReq`, and `JtLedgerData` has five
(`ProcessLData` and `GotStaleData` in `InboundLedgers.cpp`, `AcqDone` and the `InboundLedger`
`TimeoutCounter` job name in `InboundLedger.cpp`, and `GotFetchPack` in `LedgerMaster.cpp`) — so a
queue-wait spike cannot be pinned to one of them by `job_type` alone. The `handler` label carries
the name passed to `addJob`, which does distinguish them.
The value is not the raw name. It passes through `MetricsRegistry::sanitiseHandler`, which keeps
the name only when it is non-empty and every character is an ASCII letter, and otherwise returns
the literal `other`. The letter test is an explicit ASCII range check rather than a
locale-dependent one, so it cannot admit non-ASCII bytes.
**Why reduce the name.** Two `addJob` names embed a ledger sequence number:
`LedgerPersistence.cpp` builds `"Pub" + std::to_string(ledger->seq())`, and
`OrderBookDBImpl.cpp` builds `"OB" + std::to_string(ledger->seq() % 1000000000)`. Used raw,
either would mint a new time series for every ledger. Both always contain digits, so both fold to
`other`.
Every other production job name is a compile-time literal, which makes the label domain a
function of the source rather than of traffic — it cannot grow at runtime. Names that are literals
but still fail the rule also fold to `other`: `GetConsL1` and `GetConsL2` contain digits, and the
coroutine names `RPC-Client`, `WS-Client`, and `gRPC-Client` contain a hyphen. A job name added
later that fails the rule degrades to `other` rather than becoming unbounded, which is a stronger
guarantee than a hand-maintained allowlist.
### 2a.2 GetObject Request Metrics
Emitted from the `TMGetObjectByHash` handler in `src/xrpld/overlay/detail/PeerImp.cpp`. Together
with `job_queued_us` and `job_running_us` filtered to `handler="RcvGetObjByHash"`, they split the
handler's end-to-end time into queue wait, NodeStore lookup, and everything else — so slowness is
attributable to a stage rather than merely observed.
| Metric | Type | Unit | Labels | Description |
| --------------------------- | --------- | ---- | -------- | ---------------------------------------------- |
| `getobject_lookup_us` | Histogram | µs | — | Time in the NodeStore fetch loop |
| `getobject_request_objects` | Histogram | — | — | Objects requested per message |
| `getobject_lookups_total` | Counter | — | `result` | NodeStore lookup outcomes |
| `getobject_rejected_total` | Counter | — | `reason` | Requests refused before any NodeStore access |
| `getobject_charge` | Histogram | — | — | Dynamic resource charge applied to the request |
| Label | Values | Meaning |
| -------- | ---------------------------------- | ------------------------------------------------------- |
| `result` | `hit`, `miss` | Whether the requested object was found in the NodeStore |
| `reason` | `oversize`, `malformed_ledgerhash` | Which admission check refused the request |
**Aggregation.** `getobject_lookup_us` times the fetch loop once per request, and
`getobject_lookups_total` is added once per request carrying the batch hit and miss totals. The
loop is bounded by `Tuning::kHardMaxReplyNodes` (12288), so timing or counting each iteration
would cost more than the lookups it measures. `getobject_request_objects` records the requested
count, which is what the charge bands price on. `getobject_charge` records only the dynamic
component returned by `computeGetObjectByHashFee()`; the admission-time base charge is a constant.
> **Bucket view required**: `getobject_lookup_us` needs an explicit microsecond-bucket view like
> the two job histograms above. Without one the SDK default buckets stop at 10 ms, and a
> lookup loop running toward its 12288 cap exceeds that routinely — so the metric would saturate
> exactly when it is needed. The view is registered alongside the other three in
> `MetricsRegistry.cpp`. Note that `MetricsRegistry` does not exist on this branch — it is
> introduced on phase-9, which is where the registration lives.
> **Expected to read zero**: `getobject_rejected_total` counts non-conforming requests, which a
> healthy peer does not send. A zero series does not prove the metric works — verify it with a
> deliberately oversized request.
---
## 3. Grafana Dashboard Reference
> **See also**: [05-configuration-reference.md](./05-configuration-reference.md) §5.8 for Grafana data source provisioning (Tempo, Prometheus) and TraceQL query examples.
### 3.1 Span-Derived Dashboards (5)
| Dashboard | UID | Data Source | Key Panels |
| -------------------- | ---------------------- | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| RPC Performance | `rpc-performance` | Prometheus (SpanMetrics) | Request rate by command, p95 latency by command, error rate, heatmap, top commands |
| Transaction Overview | `transaction-overview` | Prometheus (SpanMetrics) | Processing rate, latency p95/p50, local vs relay split, apply duration, heatmap |
| Consensus Health | `consensus-health` | Prometheus (SpanMetrics) | Round duration p95/p50, proposals rate, close duration, mode timeline, heatmap, close time correctness, resolution direction, close time drift, resolution change timeline, close time vote distribution |
| Ledger Operations | `ledger-operations` | Prometheus (SpanMetrics) | Build rate, build duration, validation rate, store rate, build vs close comparison |
| Peer Network | `peer-network` | Prometheus (SpanMetrics) | Proposal receive rate, validation receive rate, trusted vs untrusted breakdown |
### 3.2 StatsD Dashboards (5)
| Dashboard | UID | Data Source | Key Panels |
| ---------------------- | ----------------------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| Node Health | `xrpld-statsd-node-health` | Prometheus (StatsD) | Ledger age, operating mode, I/O latency, job queue, fetch rate, key/all jobs execution time, cache size/hit rate, publish gap, state duration rate |
| Network Traffic | `xrpld-statsd-network` | Prometheus (StatsD) | Active peers, disconnects, bytes in/out, messages in/out, traffic by category, duplicate traffic, all traffic categories detail |
| RPC & Pathfinding | `xrpld-statsd-rpc` | Prometheus (StatsD) | RPC rate, response time/size, pathfinding duration, resource warnings/drops |
| Overlay Traffic Detail | `xrpld-statsd-overlay-detail` | Prometheus (StatsD) | Squelch, overhead, validator lists, set get/share, have/requested tx, proof paths |
| Ledger Data & Sync | `xrpld-statsd-ledger-sync` | Prometheus (StatsD) | Ledger data exchange, legacy ledger share/get, getobject by type, traffic heatmap |
### 3.3 Consensus Close-Time Panels
The Consensus Health dashboard includes 5 close-time panels added in Phase 4:
| Panel | Metric / Attribute | Description |
| ---------------------------- | --------------------------------- | ------------------------------------------------------------------------ |
| Close Time Correctness | `close_time_correct` | Percentage of rounds with agreed-upon close time |
| Resolution Direction | `resolution_direction` | Rate of resolution increases, decreases, and unchanged per time interval |
| Close Time Drift | `close_time` vs `close_time_self` | Difference between agreed close time and node's own proposed close time |
| Resolution Change Timeline | `close_resolution_ms` | Close time resolution granularity over time |
| Close Time Vote Distribution | `close_time_vote_bins` | Histogram of validator close time votes per round |
**Template variables** (Consensus Health dashboard):
| Variable | Source Attribute | Description |
| ------------------------- | ------------------------------------- | ------------------------------------------------------------------------ |
| `$node` | `exported_instance` | Filter by xrpld node instance |
| `$service_name` | `service_name` | Filter by service (`service.name`, e.g. `xrpld`) |
| `$deployment_environment` | `deployment_environment` | Filter by deployment tier (`local` / `test` / `ci` / `prod`) |
| `$xrpl_network_type` | `xrpl_network_type` | Filter by network (`mainnet` / `testnet` / `devnet`) |
| `$close_time_correct` | `xrpl_consensus_close_time_correct` | Filter by close time correctness (`true` / `false`) |
| `$resolution_direction` | `xrpl_consensus_resolution_direction` | Filter by resolution direction (`increased` / `decreased` / `unchanged`) |
The `$service_name`, `$deployment_environment`, and `$xrpl_network_type`
variables are present on all dashboards; each variable name matches its
Prometheus label. See [telemetry-runbook.md](../docs/telemetry-runbook.md)
"Deployment Tiers" for how the tier attributes are set and reach metrics.
### 3.4 Accessing the Dashboards
1. Open Grafana at **http://localhost:3000**
2. Navigate to **Dashboards → xrpld** folder
3. All 10 dashboards are auto-provisioned from `docker/telemetry/grafana/dashboards/`
---
## 4. Tempo Trace Search Guide
> **See also**: [08-appendix.md](./08-appendix.md) §8.2 for span hierarchy visualizations. [05-configuration-reference.md](./05-configuration-reference.md) §5.8.5 for TraceQL query examples.
### Finding Traces by Type
| What to Find | Tempo TraceQL Query |
| ------------------------ | ------------------------------------------------------------------------------ |
| All RPC calls | `{resource.service.name="xrpld" && name="rpc.http_request"}` |
| Specific RPC command | `{resource.service.name="xrpld" && name="rpc.command.server_info"}` |
| Slow RPC calls | `{resource.service.name="xrpld" && name=~"rpc.command.*"} \| duration > 100ms` |
| Failed RPC calls | `{span.rpc_status="error"}` |
| Specific transaction | `{span.tx_hash="<hex_hash>"}` |
| Local transactions only | `{span.local=true}` |
| Consensus rounds | `{resource.service.name="xrpld" && name="consensus.accept"}` |
| Rounds by mode | `{span.consensus_mode="proposing"}` |
| Specific ledger | `{span.ledger_seq=12345}` |
| Peer proposals (trusted) | `{span.proposal_trusted=true}` |
### Trace Structure
A typical RPC trace shows the span hierarchy:
```
rpc.http_request (ServerHandler)
└── rpc.process (ServerHandler)
└── rpc.command.server_info (RPCHandler)
```
A consensus round groups child spans under a deterministic trace ID:
```
consensus.round (top-level, deterministic trace ID from ledger hash)
├── consensus.ledger_close (close event)
├── consensus.proposal.send (broadcast proposal)
├── consensus.establish (convergence loop)
│ ├── consensus.update_positions (update disputes)
│ └── consensus.check (check agreement)
├── consensus.accept (accept result)
├── consensus.accept.apply (apply with close time details)
├── consensus.validation.send (send validation)
└── consensus.mode_change (mode transition, if any)
ledger.build (build new ledger)
└── tx.apply (apply transaction set)
ledger.validate (promote to validated)
ledger.store (persist to DB)
```
---
## 5. Prometheus Query Examples
> **See also**: [05-configuration-reference.md](./05-configuration-reference.md) §5.8.7 for correlating Prometheus StatsD metrics with trace-derived metrics.
### Span-Derived Metrics
```promql
# RPC request rate by command (last 5 minutes)
sum by (xrpl_rpc_command) (rate(traces_span_metrics_calls_total{span_name=~"rpc.command.*"}[5m]))
# RPC p95 latency by command
histogram_quantile(0.95, sum by (le, xrpl_rpc_command) (rate(traces_span_metrics_duration_milliseconds_bucket{span_name=~"rpc.command.*"}[5m])))
# Consensus round duration p95
histogram_quantile(0.95, sum by (le) (rate(traces_span_metrics_duration_milliseconds_bucket{span_name="consensus.accept"}[5m])))
# Transaction processing rate (local vs relay)
sum by (xrpl_tx_local) (rate(traces_span_metrics_calls_total{span_name="tx.process"}[5m]))
# Trusted vs untrusted proposal rate
sum by (xrpl_peer_proposal_trusted) (rate(traces_span_metrics_calls_total{span_name="peer.proposal.receive"}[5m]))
```
### StatsD Metrics
```promql
# Validated ledger age (should be < 10s)
xrpld_LedgerMaster_Validated_Ledger_Age
# Active peer count
xrpld_Peer_Finder_Active_Inbound_Peers + xrpld_Peer_Finder_Active_Outbound_Peers
# RPC response time p95
histogram_quantile(0.95, xrpld_rpc_time_bucket)
# Total network bytes in (rate)
rate(xrpld_total_Bytes_In[5m])
# Operating mode (should be "Full" after startup)
xrpld_State_Accounting_Full_duration
```
---
## 6. SpanNames Header File Inventory
All span names and attributes are defined as compile-time constants in colocated `SpanNames.h` headers. Each header lives next to its subsystem's implementation.
| Header File | Subsystem | Span Count | Attribute Count | Notes |
| ----------------------------------------------- | ------------- | ---------- | --------------- | ------------------------------------------- |
| `src/xrpld/rpc/detail/RpcSpanNames.h` | RPC (HTTP/WS) | 5 | 5 | Includes `rpc.ws_upgrade` error path |
| `src/xrpld/rpc/detail/PathFindSpanNames.h` | PathFind | 5 | 8 | Covers one-shot and subscription paths |
| `src/xrpld/app/main/GrpcSpanNames.h` | gRPC | 1 | 3 | Flat single-span structure per request |
| `src/xrpld/app/misc/TxSpanNames.h` | Transaction | 2 | 7 | Includes peer context attributes |
| `src/xrpld/app/misc/detail/TxQSpanNames.h` | TxQ | 6 | 11 | Queue lifecycle: enqueue through cleanup |
| `src/xrpld/app/consensus/ConsensusSpanNames.h` | Consensus | 10 | 35 | Deterministic trace IDs, close-time details |
| `src/xrpld/app/ledger/detail/LedgerSpanNames.h` | Ledger | 4 | 7 | Build, store, validate, tx.apply |
| `src/xrpld/overlay/detail/PeerSpanNames.h` | Peer Overlay | 2 | 5 | Proposal and validation receive |
> **Design convention**: SpanNames headers are colocated with their subsystem classes rather than centralized in `telemetry/`. See [memory/feedback_span-names-colocation.md](../.claude/memory/feedback_span-names-colocation.md) for rationale.
---
## 7. Known Issues
| Issue | Impact | Status |
| ------------------------------------------------------------------ | ------------------------------------------------ | -------------------------------------------------------------------- |
| `warn` and `drop` metrics use non-standard StatsD `\|m` meter type | Metrics silently dropped by OTel StatsD receiver | Phase 6 Task 6.1 — needs `\|m` → `\|c` change in StatsDCollector.cpp |
| `xrpld_jobq_job_count` may not emit in standalone mode | Missing from Prometheus in some test configs | Requires active job queue activity |
| `xrpld_rpc_requests` depends on `[insight]` config | Zero series if StatsD not configured | Requires `[insight] server=statsd` in xrpld.cfg |
| Peer tracing enabled by default | `peer.*` spans emit unless `trace_peer=0` | High volume — set `trace_peer=0` to opt out on busy mainnet nodes |
---
## 8. Privacy and Data Collection
The telemetry system is designed with privacy in mind:
- **No private keys** are ever included in spans or metrics
- **No account balances** or financial data is traced
- **Transaction hashes** are included (public on-ledger data) but not transaction contents
- **Peer IDs** are internal identifiers, not IP addresses
- **All telemetry is opt-in** — disabled by default at build time (`-Dtelemetry=OFF`)
- **Sampling** — head sampling is fixed at 1.0 (sample everything); reduce data volume with collector-side tail sampling
- **Data stays local** — the default stack sends data to `localhost` only
---
## 9. Configuration Quick Reference
> **Full reference**: [05-configuration-reference.md](./05-configuration-reference.md) §5.1 for all `[telemetry]` options with defaults, the config parser implementation, and collector YAML configurations (dev and production).
### Minimal Setup (development)
```ini
[telemetry]
enabled=1
[insight]
server=statsd
address=127.0.0.1:8125
prefix=xrpld
```
### Production Setup
```ini
[telemetry]
enabled=1
endpoint=http://otel-collector:4318/v1/traces
trace_peer=0
batch_size=1024
max_queue_size=4096
[insight]
server=statsd
address=otel-collector:8125
prefix=xrpld
```
### Trace Category Toggle
| Config Key | Default | Controls |
| -------------------- | ------- | ---------------------------- |
| `trace_rpc` | `1` | `rpc.*` spans |
| `trace_transactions` | `1` | `tx.*` spans |
| `trace_consensus` | `1` | `consensus.*` spans |
| `trace_ledger` | `1` | `ledger.*` spans |
| `trace_peer` | `1` | `peer.*` spans (high volume) |
Reference in New Issue View Git Blame Copy Permalink