ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-18 21:10:02 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'pratik/otel-phase6-statsd' into pratik/otel-phase7-native-metrics

Browse Source 
Conflict resolution kept this branch's evolution and re-applied phase-6's
fixes on top of it, rather than taking either side wholesale:

- consensus-health.json: kept the native `span_calls_total` metric name and
  the `interval: 15s` and point styling from this branch; added phase-6's
  `close_time_correct` PromQL filter and the NetClock axis labels. The
  TraceQL boolean-regex filter stays removed and the `byRegexp` overrides
  carry over. Panel count unchanged at 27.
- 09-data-collection-reference.md: kept this branch's headings, its more
  detailed consensus attribute table (which already types
  `consensus_round_id` as int64) and its section numbering, including the
  deliberate removal of the SpanNames inventory. Carried over only the
  correction that the state-accounting duration gauges are cumulative
  microseconds, not seconds.
- telemetry-runbook.md: kept this branch's native metric names
  (`span_calls_total`, `span_duration_milliseconds_bucket`); carried the
  `rpc.request` -> `rpc.http_request` span-name fix and the `jobq_` segment
  on the job-queue depth metric.
- integration-test.sh: kept this branch's `check_otel_metric` form and
  carried the `jobq_job_count` correction.
This commit is contained in:
Pratik Mankawde
2026-08-17 12:05:10 +01:00
parent 77f338c0d0 3a3ae1c3ae
commit b77f516f95
5 changed files with 96 additions and 55 deletions
Download Patch File Download Diff File Expand all files Collapse all files

40
OpenTelemetryPlan/09-data-collection-reference.md
View File

@@ -285,6 +285,8 @@ aggregation. Per the 2026-05-13 naming redesign, span-attribute keys use the
> it (both `consensus.validation.send` and `peer.validation.receive`) — there
> is no dotted span attribute.
The tables below list one row per attribute per subsystem, so a key shared by two subsystems (for example `ledger_seq`) appears once in each. That is 89 rows over 78 distinct keys. The §6 per-header counts use the same row-based rule, so they sum to 89.
#### RPC Attributes
| Attribute | Type | Set On | Description |
@@ -546,25 +548,25 @@ prefix=xrpld
### 2.1 Gauges
| Prometheus Metric | Source File | Description | Typical Range |
| ------------------------------------------- | --------------------- | ----------------------------------------- | ------------------------------- |
| `ledgermaster_validated_ledger_age` | LedgerMaster.h | Seconds since last validated ledger | 0–10 (healthy), >30 (stale) |
| `ledgermaster_published_ledger_age` | LedgerMaster.h | Seconds since last published ledger | 0–10 (healthy) |
| `state_accounting_disconnected_duration` | NetworkOPs.cpp | Cumulative seconds in Disconnected state | Monotonic |
| `state_accounting_connected_duration` | NetworkOPs.cpp | Cumulative seconds in Connected state | Monotonic |
| `state_accounting_syncing_duration` | NetworkOPs.cpp | Cumulative seconds in Syncing state | Monotonic |
| `state_accounting_tracking_duration` | NetworkOPs.cpp | Cumulative seconds in Tracking state | Monotonic |
| `state_accounting_full_duration` | NetworkOPs.cpp | Cumulative seconds in Full state | Monotonic (should dominate) |
| `state_accounting_disconnected_transitions` | NetworkOPs.cpp | Count of transitions to Disconnected | Low |
| `state_accounting_connected_transitions` | NetworkOPs.cpp | Count of transitions to Connected | Low |
| `state_accounting_syncing_transitions` | NetworkOPs.cpp | Count of transitions to Syncing | Low |
| `state_accounting_tracking_transitions` | NetworkOPs.cpp | Count of transitions to Tracking | Low |
| `state_accounting_full_transitions` | NetworkOPs.cpp | Count of transitions to Full | Low (should be 1 after startup) |
| `peer_finder_active_inbound_peers` | PeerfinderManager.cpp | Active inbound peer connections | 0–85 |
| `peer_finder_active_outbound_peers` | PeerfinderManager.cpp | Active outbound peer connections | 10–21 |
| `overlay_peer_disconnects` | OverlayImpl.cpp | Cumulative peer disconnection count | Low growth |
| `overlay_peer_disconnects_charges` | OverlayImpl.cpp | Disconnects due to resource limit charges | Low growth (subset of above) |
| `job_count` | JobQueue.cpp | Current job queue depth | 0–100 (healthy) |
| Prometheus Metric | Source File | Description | Typical Range |
| ------------------------------------------- | --------------------- | --------------------------------------------- | ------------------------------- |
| `ledgermaster_validated_ledger_age` | LedgerMaster.h | Seconds since last validated ledger | 0–10 (healthy), >30 (stale) |
| `ledgermaster_published_ledger_age` | LedgerMaster.h | Seconds since last published ledger | 0–10 (healthy) |
| `state_accounting_disconnected_duration` | NetworkOPs.cpp | Cumulative microseconds in Disconnected state | Monotonic |
| `state_accounting_connected_duration` | NetworkOPs.cpp | Cumulative microseconds in Connected state | Monotonic |
| `state_accounting_syncing_duration` | NetworkOPs.cpp | Cumulative microseconds in Syncing state | Monotonic |
| `state_accounting_tracking_duration` | NetworkOPs.cpp | Cumulative microseconds in Tracking state | Monotonic |
| `state_accounting_full_duration` | NetworkOPs.cpp | Cumulative microseconds in Full state | Monotonic (should dominate) |
| `state_accounting_disconnected_transitions` | NetworkOPs.cpp | Count of transitions to Disconnected | Low |
| `state_accounting_connected_transitions` | NetworkOPs.cpp | Count of transitions to Connected | Low |
| `state_accounting_syncing_transitions` | NetworkOPs.cpp | Count of transitions to Syncing | Low |
| `state_accounting_tracking_transitions` | NetworkOPs.cpp | Count of transitions to Tracking | Low |
| `state_accounting_full_transitions` | NetworkOPs.cpp | Count of transitions to Full | Low (should be 1 after startup) |
| `peer_finder_active_inbound_peers` | PeerfinderManager.cpp | Active inbound peer connections | 0–85 |
| `peer_finder_active_outbound_peers` | PeerfinderManager.cpp | Active outbound peer connections | 10–21 |
| `overlay_peer_disconnects` | OverlayImpl.cpp | Cumulative peer disconnection count | Low growth |
| `overlay_peer_disconnects_charges` | OverlayImpl.cpp | Disconnects due to resource limit charges | Low growth (subset of above) |
| `job_count` | JobQueue.cpp | Current job queue depth | 0–100 (healthy) |
**Grafana dashboard**: _Node Health_ (`node-health`)

2
OpenTelemetryPlan/OpenTelemetryPlan.md
View File

@@ -208,7 +208,7 @@ The appendix contains a glossary of OpenTelemetry and xrpld-specific terms, refe
## 9. Data Collection Reference
A single-source-of-truth reference documenting every piece of telemetry data collected by xrpld. Covers all 16 OpenTelemetry spans with their 22 attributes, all StatsD metrics (gauges, counters, histograms, overlay traffic), SpanMetrics-derived Prometheus metrics, and all 10 Grafana dashboards. Includes Jaeger search guides and Prometheus query examples.
A single-source-of-truth reference documenting every piece of telemetry data collected by xrpld. Covers all 38 OpenTelemetry spans with their 89 attribute rows (78 unique keys), all StatsD metrics (gauges, counters, histograms, overlay traffic), SpanMetrics-derived Prometheus metrics, and all 10 Grafana dashboards. Includes Tempo trace search guides and Prometheus query examples.
➡️ **[View Data Collection Reference](./09-data-collection-reference.md)**

36
docker/telemetry/grafana/dashboards/consensus-health.json
View File

@@ -282,7 +282,7 @@
"datasource": {
"type": "prometheus"
},
"expr": "sum by (close_time_correct, service_instance_id) (rate(span_calls_total{span_name=\"consensus.accept.apply\", consensus_mode=~\"$consensus_mode\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval]))",
"expr": "sum by (close_time_correct, service_instance_id) (rate(span_calls_total{span_name=\"consensus.accept.apply\", close_time_correct=~\"$close_time_correct\", consensus_mode=~\"$consensus_mode\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval]))",
"interval": "15s",
"legendFormat": "Close Time Correct={{close_time_correct}} [{{service_instance_id}}]"
}
@@ -530,7 +530,7 @@
},
{
"title": "Close Time: Raw Proposals (Per Node)",
"description": "**What:** Each node's raw, unrounded proposed close time at the instant it closed its ledger.\n**How it's computed:** Latest raw close-time value per node, plotted per round.\n**Reading it:** Compare nodes at the same round; values should cluster tightly.\n**Healthy range:** All nodes within a few seconds of each other.\n**Watch for:** A node consistently offset from the pack indicates local clock drift.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::doAccept",
"description": "**What:** Each node's raw, unrounded proposed close time at the instant it closed its ledger.\n**How it's computed:** Latest raw close-time value per node, plotted per round. Values are NetClock seconds (Ripple epoch, 2000-01-01); add 946684800 for Unix time.\n**Reading it:** Compare nodes at the same round; values should cluster tightly.\n**Healthy range:** All nodes within a few seconds of each other.\n**Watch for:** A node consistently offset from the pack indicates local clock drift.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::doAccept",
"type": "timeseries",
"gridPos": {
"h": 10,
@@ -540,7 +540,7 @@
},
"fieldConfig": {
"defaults": {
"unit": "dateTimeFromNow",
"unit": "none",
"custom": {
"spanNulls": 1800000,
"drawStyle": "points",
@@ -548,7 +548,8 @@
"showPoints": "always",
"lineWidth": 1,
"fillOpacity": 0,
"gradientMode": "none"
"gradientMode": "none",
"axisLabel": "NetClock Seconds (Ripple Epoch)"
}
},
"overrides": []
@@ -571,7 +572,7 @@
"type": "tempo"
},
"queryType": "traceql",
"query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\" && span.close_time_correct=~\"$close_time_correct\"} | select(span.close_time_self)",
"query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\"} | select(span.close_time_self)",
"refId": "A"
}
],
@@ -579,7 +580,7 @@
},
{
"title": "Close Time: Effective / Quantized",
"description": "**What:** The consensus-agreed close time after rounding to the active resolution bin, i.e. the value written to the ledger header.\n**How it's computed:** Latest effective close-time value per node, plotted per round.\n**Reading it:** All in-agreement nodes should report the identical value each round.\n**Healthy range:** Identical across agreeing nodes.\n**Watch for:** Nodes reporting different effective values are not in close-time agreement for that round.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::doAccept",
"description": "**What:** The consensus-agreed close time after rounding to the active resolution bin, i.e. the value written to the ledger header.\n**How it's computed:** Latest effective close-time value per node, plotted per round. Values are NetClock seconds (Ripple epoch, 2000-01-01); add 946684800 for Unix time.\n**Reading it:** All in-agreement nodes should report the identical value each round.\n**Healthy range:** Identical across agreeing nodes.\n**Watch for:** Nodes reporting different effective values are not in close-time agreement for that round.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::doAccept",
"type": "timeseries",
"gridPos": {
"h": 10,
@@ -589,7 +590,7 @@
},
"fieldConfig": {
"defaults": {
"unit": "dateTimeFromNow",
"unit": "none",
"custom": {
"spanNulls": 1800000,
"drawStyle": "points",
@@ -597,7 +598,8 @@
"showPoints": "always",
"lineWidth": 1,
"fillOpacity": 0,
"gradientMode": "none"
"gradientMode": "none",
"axisLabel": "NetClock Seconds (Ripple Epoch)"
}
},
"overrides": []
@@ -620,7 +622,7 @@
"type": "tempo"
},
"queryType": "traceql",
"query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\" && span.close_time_correct=~\"$close_time_correct\"} | select(span.close_time)",
"query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\"} | select(span.close_time)",
"refId": "A"
}
],
@@ -652,8 +654,8 @@
"overrides": [
{
"matcher": {
"id": "byName",
"options": "Vote Bins"
"id": "byRegexp",
"options": ".*vote_bins.*"
},
"properties": [
{
@@ -668,8 +670,8 @@
},
{
"matcher": {
"id": "byName",
"options": "Resolution"
"id": "byRegexp",
"options": ".*close_resolution_ms.*"
},
"properties": [
{
@@ -702,7 +704,7 @@
"type": "tempo"
},
"queryType": "traceql",
"query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\" && span.close_time_correct=~\"$close_time_correct\"} | select(span.close_time_vote_bins)",
"query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\"} | select(span.close_time_vote_bins)",
"refId": "A"
},
{
@@ -710,7 +712,7 @@
"type": "tempo"
},
"queryType": "traceql",
"query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\" && span.close_time_correct=~\"$close_time_correct\"} | select(span.close_resolution_ms)",
"query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\"} | select(span.close_resolution_ms)",
"refId": "B"
}
],
@@ -758,7 +760,7 @@
"type": "tempo"
},
"queryType": "traceql",
"query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\" && span.close_time_correct=~\"$close_time_correct\" && span.resolution_direction=~\"$resolution_direction\"} | select(span.resolution_direction)",
"query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\" && span.resolution_direction=~\"$resolution_direction\"} | select(span.resolution_direction)",
"refId": "A"
}
],
@@ -805,7 +807,7 @@
"type": "tempo"
},
"queryType": "traceql",
"query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\" && span.close_time_correct=~\"$close_time_correct\"} | select(span.close_time, span.close_time_vote_bins)",
"query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\"} | select(span.close_time, span.close_time_vote_bins)",
"refId": "A"
}
],

48
docker/telemetry/integration-test.sh
View File

@@ -62,7 +62,10 @@ die() {
check_span() {
local op="$1"
local count
count=$(curl -sf "$TEMPO/api/search" \
# -G is required: it moves the urlencoded params into the query string.
# Without it curl POSTs them as a request body, and Tempo answers 200
# while ignoring the query — so every span name would look present.
count=$(curl -sfG "$TEMPO/api/search" \
--data-urlencode "q={resource.service.name=\"xrpld\" && name=\"$op\"}" \
--data-urlencode "limit=5" |
jq '.traces | length' 2>/dev/null || echo 0)
@@ -133,6 +136,23 @@ mkdir -p "$WORKDIR"
# ---------------------------------------------------------------------------
# Step 2: Start observability stack
# ---------------------------------------------------------------------------
# From here on the script owns the docker stack and the xrpld nodes, so an
# abort must tear them down instead of leaving them behind. A run that
# reaches the summary deliberately leaves everything up for inspection
# (see the header comment), so the trap only fires before that point.
RUN_COMPLETED=0
on_exit() {
local status=$?
if [ "$RUN_COMPLETED" -eq 0 ]; then
log "Aborted with exit status $status — tearing down."
cleanup
fi
}
trap on_exit EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
log "Starting observability stack..."
docker compose -f "$COMPOSE_FILE" up -d
@@ -358,12 +378,14 @@ log "Waiting for nodes to reach 'proposing' state (timeout: ${CONSENSUS_TIMEOUT}
start_time=$(date +%s)
nodes_ready=0
consensus_timed_out=0
while [ "$nodes_ready" -lt "$NUM_NODES" ]; do
elapsed=$(($(date +%s) - start_time))
if [ "$elapsed" -ge "$CONSENSUS_TIMEOUT" ]; then
fail "Consensus timeout after ${CONSENSUS_TIMEOUT}s ($nodes_ready/$NUM_NODES nodes ready)"
log "Continuing with partial consensus..."
consensus_timed_out=1
break
fi
@@ -386,7 +408,10 @@ echo ""
if [ "$nodes_ready" -eq "$NUM_NODES" ]; then
ok "All $NUM_NODES nodes reached 'proposing' state"
else
elif [ "$consensus_timed_out" -eq 0 ]; then
# The timeout branch above already called fail(), so reporting again here
# would count one timeout twice. Only reachable if the loop ever gains
# another early exit.
fail "Only $nodes_ready/$NUM_NODES nodes reached 'proposing' state"
fi
@@ -430,9 +455,11 @@ log "Submitting Payment transaction..."
# Generate a destination wallet
log " Generating destination wallet..."
# Guarded: under set -e an unguarded curl failure would abort the whole
# script, so the fallback below could never run.
wallet_result=$(curl -sf "http://localhost:$RPC_PORT_BASE" \
-d '{"method":"wallet_propose"}')
DEST_ACCOUNT=$(echo "$wallet_result" | jq -r '.result.account_id' 2>/dev/null)
-d '{"method":"wallet_propose"}') || wallet_result=""
DEST_ACCOUNT=$(echo "$wallet_result" | jq -r '.result.account_id' 2>/dev/null || echo "")
if [ -z "$DEST_ACCOUNT" ] || [ "$DEST_ACCOUNT" = "null" ]; then
fail "Could not generate destination wallet"
DEST_ACCOUNT="rrrrrrrrrrrrrrrrrrrrrhoLvTp" # ACCOUNT_ZERO fallback
@@ -441,13 +468,13 @@ log " Destination: $DEST_ACCOUNT"
# Get genesis account info
acct_result=$(curl -sf "http://localhost:$RPC_PORT_BASE" \
-d "{\"method\":\"account_info\",\"params\":[{\"account\":\"$GENESIS_ACCOUNT\"}]}")
-d "{\"method\":\"account_info\",\"params\":[{\"account\":\"$GENESIS_ACCOUNT\"}]}") || acct_result=""
seq_num=$(echo "$acct_result" | jq -r '.result.account_data.Sequence' 2>/dev/null || echo "unknown")
log " Genesis account sequence: $seq_num"
# Submit payment
submit_result=$(curl -sf "http://localhost:$RPC_PORT_BASE" \
-d "{\"method\":\"submit\",\"params\":[{\"secret\":\"$GENESIS_SEED\",\"tx_json\":{\"TransactionType\":\"Payment\",\"Account\":\"$GENESIS_ACCOUNT\",\"Destination\":\"$DEST_ACCOUNT\",\"Amount\":\"10000000\"}}]}")
-d "{\"method\":\"submit\",\"params\":[{\"secret\":\"$GENESIS_SEED\",\"tx_json\":{\"TransactionType\":\"Payment\",\"Account\":\"$GENESIS_ACCOUNT\",\"Destination\":\"$DEST_ACCOUNT\",\"Amount\":\"10000000\"}}]}") || submit_result=""
engine_result=$(echo "$submit_result" | jq -r '.result.engine_result' 2>/dev/null || echo "unknown")
tx_hash=$(echo "$submit_result" | jq -r '.result.tx_json.hash' 2>/dev/null || echo "unknown")
@@ -478,7 +505,7 @@ fi
log ""
log "--- RPC Spans ---"
check_span "rpc.request"
check_span "rpc.http_request"
check_span "rpc.process"
check_span "rpc.command.server_info"
check_span "rpc.command.server_state"
@@ -562,7 +589,7 @@ check_otel_metric() {
# Node health gauges (ObservableGauge — no _total suffix)
check_otel_metric "rippled_LedgerMaster_Validated_Ledger_Age"
check_otel_metric "rippled_LedgerMaster_Published_Ledger_Age"
check_otel_metric "rippled_job_count"
check_otel_metric "rippled_jobq_job_count"
# State accounting
check_otel_metric "rippled_State_Accounting_Full_duration"
@@ -594,6 +621,11 @@ fi
# ---------------------------------------------------------------------------
# Step 11: Summary
# ---------------------------------------------------------------------------
# All checks are done, so the run counts as complete: keep the stack and the
# nodes up for inspection even when some checks failed.
RUN_COMPLETED=1
echo ""
echo "==========================================================="
echo " INTEGRATION TEST RESULTS"

25
docs/telemetry-runbook.md
View File

@@ -160,8 +160,9 @@ hash); `tx.preflight` is stateless and omits both.
# Find specific ledger's consensus details
{name="consensus.accept.apply"} | ledger_seq = 92345678
# Find all spans in a consensus round (deterministic trace strategy)
{name="consensus.round"} | consensus_round_id = <round_id>
# Find all spans in a consensus round (deterministic trace strategy).
# consensus_round_id is an int64 — the previous ledger sequence plus one.
{name="consensus.round"} | consensus_round_id = 92345679
# Find dispute resolutions
{name="consensus.update_positions"} >> {event:name="dispute.resolve"}
@@ -249,10 +250,14 @@ sum by (stage) (rate(span_calls_total{span_name=~"tx.preflight|tx.preclaim|tx.tr
> a rising `tx.transactor` failure rate points to apply-time problems. Alert per
> stage rather than on a single aggregate so the failing stage is obvious.
> **Sampling caveat**: these stage metrics are span-derived and inherit the
> **tracer head-sampling** ratio (`sampling_ratio`). At `sampling_ratio < 1.0`
> they undercount proportionally — treat them as relative trends, not absolute
> transaction counts. Native StatsD metrics are unsampled.
> **Sampling caveat**: these stage metrics are span-derived, so they count only
> the spans the collector's spanmetrics connector sees. Head sampling at the node
> is fixed at 1.0 and is not configurable (`Telemetry.h`), and the shipped
> collector pipeline has no tail sampling, so today nothing is dropped and the
> counts are absolute. Volume reduction is delegated to the collector: adding a
> tail-sampling processor to the traces pipeline puts it ahead of the spanmetrics
> connector, and these metrics would then undercount proportionally — treat them
> as relative trends in that case. Native StatsD metrics are never sampled.
### Transaction Queue Health
@@ -452,7 +457,7 @@ all its normal attributes, it just lacks a cross-node parent link.
{name=~"tx\\..*"} | tx_hash = "<hash>"
# Find all spans in a cross-node consensus trace
{rootServiceName="xrpld"} | consensus_round_id = <round_id>
{rootServiceName="xrpld"} | consensus_round_id = 92345679
# Compare latency between sender and receiver for validations
{name="consensus.validation.send" || name="consensus.validation.receive"}
@@ -538,7 +543,7 @@ The `OTelCollector` implementation exports metrics via OTLP/HTTP to the same OTe
| `peer_finder_active_inbound_peers` | PeerfinderManager.cpp:214 | Active inbound peer connections |
| `peer_finder_active_outbound_peers` | PeerfinderManager.cpp:215 | Active outbound peer connections |
| `overlay_peer_disconnects` | OverlayImpl.h:557 | Peer disconnect count |
| `job_count` | JobQueue.cpp:26 | Current job queue depth |
| `jobq_job_count` | JobQueue.cpp:26 | Current job queue depth |
| `{category}_bytes_in/out` | OverlayImpl.h:535 | Overlay traffic bytes per category (57 categories) |
| `{category}_messages_in/out` | OverlayImpl.h:535 | Overlay traffic messages per category |
@@ -658,7 +663,7 @@ Ten dashboards are pre-provisioned in `docker/telemetry/grafana/dashboards/`:
| RPC Latency p95 by Command | timeseries | `histogram_quantile(0.95, sum by (le, command) (rate(span_duration_milliseconds_bucket{span_name=~"rpc.command.*"}[5m])))` | `command` |
| RPC Error Rate | bargauge | Error spans / total spans × 100, grouped by `command` | `command`, `status_code` |
| RPC Latency Heatmap | heatmap | `sum(increase(span_duration_milliseconds_bucket{span_name=~"rpc.command.*"}[5m])) by (le)` | `le` (bucket boundaries) |
| Overall RPC Throughput | timeseries | `rpc.request` + `rpc.process` rate | — |
| Overall RPC Throughput | timeseries | `rpc.http_request` + `rpc.process` rate | — |
| RPC Success vs Error | timeseries | by `status_code` (UNSET vs ERROR) | `status_code` |
| Top Commands by Volume | bargauge | `topk(10, ...)` by `command` | `command` |
| WebSocket Message Rate | stat | `rpc.ws_message` rate | — |
@@ -725,7 +730,7 @@ Requires `trace_peer=1` in the `[telemetry]` config section.
| Operating Mode (Time Share) | timeseries | `rate(state_accounting_X_duration) / sum(rate(all modes))` | — |
| Operating Mode Transitions | timeseries | `state_accounting_*_transitions` | — |
| I/O Latency | timeseries | `histogram_quantile(0.95, ios_latency_bucket)` | — |
| Job Queue Depth | timeseries | `job_count` | — |
| Job Queue Depth | timeseries | `jobq_job_count` | — |
| Ledger Fetch Rate | stat | `rate(ledger_fetches_total[$__rate_interval])` | — |
| Ledger History Mismatches | stat | `rate(ledger_history_mismatch_total[$__rate_interval])` | — |
| Key Jobs Execution Time | timeseries | `acceptledger{quantile="$quantile"}` (+ 10 more key jobs) | `quantile` |