ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-09-27 23:38:08 +00:00
Code Issues Packages Projects Releases Wiki Activity

merge: bring the review fixes forward from otel-phase7-native-metrics

Browse Source
This commit is contained in:
Pratik Mankawde
2026-09-08 15:46:05 +01:00
parent efa6535427 7d679ce596
commit 547cc8bac5
16 changed files with 702 additions and 114 deletions
Download Patch File Download Diff File Expand all files Collapse all files

86
docs/telemetry-runbook.md
View File

@@ -44,27 +44,29 @@ cmake --build --preset default
## Configuration Reference
| Option | Default | Description |
| -------------------------- | --------------------------------- | --------------------------------------------------------- |
| `enabled` | `0` | Master switch for telemetry |
| `traces_endpoint` | `http://localhost:4318/v1/traces` | Full OTLP/HTTP URL for spans, used verbatim |
| `service_name` | `xrpld` | OpenTelemetry service name resource attribute |
| `service_instance_id` | node public key | OpenTelemetry service instance ID resource attribute |
| `trace_rpc` | `1` | Enable RPC request tracing |
| `trace_transactions` | `1` | Enable transaction tracing |
| `trace_consensus` | `1` | Enable consensus tracing |
| `trace_peer` | `1` | Enable peer message tracing (high volume) |
| `trace_ledger` | `1` | Enable ledger tracing |
| `consensus_trace_strategy` | `deterministic` | Consensus trace ID strategy (`deterministic` or `random`) |
| `batch_size` | `512` | Max spans per batch export |
| `batch_delay_ms` | `5000` | Delay between batch exports |
| `max_queue_size` | `2048` | Max spans queued before dropping |
| `use_tls` | `0` | Use TLS for exporter connection |
| `tls_ca_cert` | (empty) | Path to CA certificate bundle |
| `tls_client_cert` | (empty) | Client cert (PEM) for mTLS; empty = one-way. See note |
| `tls_client_key` | (empty) | Private key (PEM) for `tls_client_cert`. See note |
| Option | Default | Description |
| -------------------------- | --------------------------------- | ----------------------------------------------------------------------------------------------------- |
| `enabled` | `0` | Master switch for telemetry |
| `traces_endpoint` | `http://localhost:4318/v1/traces` | Full OTLP/HTTP URL for spans, used verbatim |
| `service_name` | `xrpld` | OpenTelemetry service name resource attribute |
| `service_instance_id` | node public key | OpenTelemetry service instance ID resource attribute |
| `trace_rpc` | `1` | Enable RPC request tracing |
| `trace_transactions` | `1` | Enable transaction tracing |
| `trace_consensus` | `1` | Enable consensus tracing |
| `trace_peer` | `1` | Enable peer message tracing (high volume) |
| `trace_ledger` | `1` | Enable ledger tracing |
| `consensus_trace_strategy` | `deterministic` | Consensus trace ID strategy. `deterministic` is the value to use; `random` is experimental — see note |
| `batch_size` | `512` | Max spans per batch export |
| `batch_delay_ms` | `5000` | Delay between batch exports |
| `max_queue_size` | `2048` | Max spans queued before dropping |
| `use_tls` | `0` | Use TLS for exporter connection |
| `tls_ca_cert` | (empty) | Path to CA certificate bundle |
| `tls_client_cert` | (empty) | Client cert (PEM) for mTLS; empty = one-way. See note |
| `tls_client_key` | (empty) | Private key (PEM) for `tls_client_cert`. See note |
> **mTLS (mutual TLS) note**: `tls_client_cert` and `tls_client_key` are optional — leaving both empty gives one-way (server-only) TLS. **If either one is set**, `enabled=1` requires both of them **and** `use_tls=1`, or the node exits at startup; see the Troubleshooting entry for `Unable to start ...: [telemetry] ...`. When `enabled=0` they are read but never validated.
> **mTLS (mutual TLS) note**: `tls_client_cert` and `tls_client_key` are optional — leaving both empty gives one-way (server-only) TLS. **If either one is set**, `enabled=1` requires both of them, `use_tls=1`, **and** a `traces_endpoint` starting with `https://` — the exporter decides encryption from the URL scheme, so the certificate is only ever presented on an `https://` endpoint. The default `traces_endpoint` is plain HTTP, so mTLS means setting that key too. Breaking any of these makes the node exit at startup; see the Troubleshooting entry for `Unable to start ...: [telemetry] ...`. When `enabled=0` they are read but never validated.
> **`consensus_trace_strategy` note**: only `deterministic` and `random` are accepted, and anything else makes the node exit at startup. Use `deterministic`: it seeds the round's trace ID from the previous ledger hash, so every validator of a round reports into one trace. `random` is experimental and not used — each node would invent its own trace ID, so a single round would arrive as one separate trace per node, joinable only by hand through `consensus_ledger_id`.
## Span Reference
@@ -144,13 +146,22 @@ hash); `tx.preflight` is stateless and omits both.
#### Consensus Span Events
| Parent Span | Event Name | Event Attributes | Description |
| ---------------------------- | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------- |
| `consensus.update_positions` | `dispute.resolve` | `tx_id`, `dispute_our_vote`, `dispute_yays`, `dispute_nays` | Emitted per dispute when votes are tallied |
| `consensus.accept.apply` | `tx.included` | `tx_id` | Emitted per transaction included in the accepted ledger |
| Parent Span | Event Name | Event Attributes | Description |
| ---------------------------- | ----------------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| `consensus.update_positions` | `dispute.resolve` | `tx_id`, `dispute_our_vote`, `dispute_yays`, `dispute_nays` | Emitted per dispute when votes are tallied |
| `consensus.accept.apply` | `tx.included` | `tx_id` | Emitted per transaction of the agreed consensus set, before the ledger is built — see note |
> **`tx.included` note**: the event is recorded while the canonical transaction set is being assembled, which happens before `buildLCL()` applies anything. So a transaction that fails to apply, or is left over to retry in a later ledger, still has a `tx.included` event. Treat the events as the round's **input** set, not as proof a transaction reached the accepted ledger; `tx_count` on the same span counts the same set. A transaction whose bytes cannot be parsed gets no event, and nothing in the accepted ledger is missing one, so the events are always a superset of the ledger's contents. To confirm a transaction actually applied, read `ter_result` and `applied` on its `tx.transactor` span.
#### Close Time Queries (Tempo TraceQL)
> **TraceQL syntax**: an attribute filter belongs inside the braces, as
> `{name="x" && span.attr = value}`. The `{name="x"} | attr = value` form used
> by several examples in this document is rejected by current Tempo with a parse
> error, so convert an example to the braced form before running it. Numeric
> attributes such as `consensus_round_id` and `retries_remaining` must be
> compared unquoted.
```
# Find rounds where validators disagreed on close time
{name="consensus.accept.apply"} | close_time_correct = false
@@ -164,12 +175,14 @@ hash); `tx.preflight` is stateless and omits both.
# Find specific ledger's consensus details
{name="consensus.accept.apply"} | ledger_seq = 92345678
# Find all spans in a consensus round (deterministic trace strategy).
# consensus_round_id is an int64 — the previous ledger sequence plus one.
{name="consensus.round"} | consensus_round_id = 92345679
# Find a consensus round by its id. consensus_round_id is an int64 — the
# previous ledger sequence plus one — so it must not be quoted, and it is set
# only on consensus.round.
{name="consensus.round" && span.consensus_round_id = 92345678}
# Find dispute resolutions
{name="consensus.update_positions"} >> {event:name="dispute.resolve"}
# Find dispute resolutions. The event is recorded on the update_positions
# span itself, so it is a condition on that span, not on a descendant.
{name="consensus.update_positions" && event:name="dispute.resolve"}
```
### Ledger Spans
@@ -276,8 +289,11 @@ sum by (stage) (rate(span_calls_total{span_name=~"tx.preflight|tx.preclaim|tx.tr
# Find ledger closes that applied queued transactions
{name="txq.accept"} | ledger_changed = true
# Find transactions that exhausted retries
{name="txq.accept_tx"} | txq_status = "retried" && retries_remaining = 0
# Find transactions dropped because they had no retries left.
# retries_remaining is recorded before the attempt, and the "retried" branch
# is only reached while retries are left, so exhaustion always shows up as
# "failed" with a zero count.
{name="txq.accept_tx" && span.txq_status = "failed" && span.retries_remaining <= 0}
```
### RPC Debugging
@@ -460,8 +476,12 @@ all its normal attributes, it just lacks a cross-node parent link.
# Trace a transaction across the network by its hash
{name=~"tx\\..*"} | tx_hash = "<hash>"
# Find all spans in a cross-node consensus trace
{rootServiceName="xrpld"} | consensus_round_id = "<round_id>"
# Find a cross-node consensus trace by round id, then open the returned trace
# to see every node's spans. Under the deterministic strategy all validators of
# a round share one trace id, so one trace holds all of them. The value is an
# integer, so it must not be quoted, and it only matches the consensus.round
# span that carries it.
{name="consensus.round" && span.consensus_round_id = 92345678}
# Compare latency between sender and receiver for validations
{name="consensus.validation.send" || name="consensus.validation.receive"}