From 18a40f86fb2e026d18432815808dfe673b908581 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Sat, 15 Aug 2026 17:24:14 +0100 Subject: [PATCH 1/3] fix(telemetry): correct integration-test span checks and telemetry docs The integration test's span assertions never actually ran. check_span() built a Tempo /api/search call with --data-urlencode but no -G, so curl POSTed the params as a body; Tempo answers 200 and ignores the query, so every span name looked present. Verified against a live Tempo 2.9.4: the buggy form returns the store's total trace count for any name, including "zzz.does.not.exist"; with -G a real name returns 1 and a bogus one 0. Fixed alongside it: the RPC check asserted "rpc.request", which is never emitted (ServerHandler.cpp builds "rpc.http_request"). These two had to change together, since -G turns the bogus name from a silent pass into a hard failure. Also in the script: a consensus timeout logged two failures and counted two, because a post-loop else re-reported what the timeout branch had already reported; and three unguarded curl calls aborted the whole script under set -euo pipefail, making the ACCOUNT_ZERO fallback dead code with no cleanup. Guarded the curls and wired an EXIT trap to the existing cleanup(). The trap deliberately fires only before the summary, so a completed run still leaves the stack up as the header documents. Docs corrections, all re-derived from code: - span inventory heading 35 -> 38, attribute heading 83 -> 89 rows (78 unique keys), and the section 6 header table now carries the missing TxApplySpanNames.h row so its columns sum to the same figures - two stale paths: ConsensusSpanNames.h is under include/xrpl/consensus/, TxSpanNames.h under src/xrpld/telemetry/ - consensus_round_id is int64, not string (RCLConsensus.cpp sets prevLgr.seq() + 1); the runbook's TraceQL examples now use a numeric literal instead of an unparseable bare - state-accounting duration gauges are cumulative MICROSECONDS, not seconds (NetworkOPs.cpp declares std::chrono::microseconds and publishes dur.count() raw) - sampling_ratio is not a config key; head sampling is fixed at 1.0 and the shipped collector has no tail sampling, so the caveat was rewritten - the plan blurb referenced Jaeger; this stack is Tempo --- .../09-data-collection-reference.md | 57 ++++++++++--------- OpenTelemetryPlan/OpenTelemetryPlan.md | 2 +- docker/telemetry/integration-test.sh | 46 ++++++++++++--- docs/telemetry-runbook.md | 21 ++++--- 4 files changed, 84 insertions(+), 42 deletions(-) diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md index 3447718567..8bce3eca19 100644 --- a/OpenTelemetryPlan/09-data-collection-reference.md +++ b/OpenTelemetryPlan/09-data-collection-reference.md @@ -84,7 +84,7 @@ always goes through `beast::insight` instead. ## 1. OpenTelemetry Spans -### 1.1 Complete Span Inventory (35 spans) +### 1.1 Complete Span Inventory (38 spans) > **See also**: [02-design-decisions.md §2.3](./02-design-decisions.md#23-span-naming-conventions) for naming conventions and the full span catalog with rationale. [04-code-samples.md §4.6](./04-code-samples.md#46-span-flow-visualization) for span flow diagrams. @@ -254,12 +254,14 @@ under an unrelated transaction's trace. --- -### 1.2 Complete Attribute Inventory (83 attributes) +### 1.2 Complete Attribute Inventory (89 attribute rows, 78 unique keys) > **See also**: [02-design-decisions.md §2.4.2](./02-design-decisions.md#242-span-attributes-by-category) for attribute design rationale and privacy considerations. Every span can carry key-value attributes that provide context for filtering and aggregation. +The tables below list one row per attribute per subsystem, so a key shared by two subsystems (for example `ledger_seq`) appears once in each. That is 89 rows over 78 distinct keys. The §6 per-header counts use the same row-based rule, so they sum to 89. + #### RPC Attributes | Attribute | Type | Set On | Description | @@ -374,7 +376,7 @@ Join a transaction's work to its ledger with `{span.current_ledger_seq=}`. | `quorum` | int64 | `consensus.check` | Required quorum for validation | | `validation_count` | int64 | `consensus.check` | Number of validations received | | `trace_strategy` | string | `consensus.round` | Trace sampling strategy used for this round | -| `consensus_round_id` | string | `consensus.round` | Deterministic round identifier | +| `consensus_round_id` | int64 | `consensus.round` | Deterministic round identifier (previous ledger seq + 1) | | `mode_old` | string | `consensus.mode_change` | Previous consensus mode | | `mode_new` | string | `consensus.mode_change` | New consensus mode | | `tx_id` | string | `consensus.update_positions` | Disputed transaction ID | @@ -474,26 +476,26 @@ prefix=xrpld ### 2.1 Gauges -| Prometheus Metric | Source File | Description | Typical Range | -| ------------------------------------------------- | --------------------- | ---------------------------------------- | ------------------------------- | -| `xrpld_LedgerMaster_Validated_Ledger_Age` | LedgerMaster.h | Seconds since last validated ledger | 0–10 (healthy), >30 (stale) | -| `xrpld_LedgerMaster_Published_Ledger_Age` | LedgerMaster.h | Seconds since last published ledger | 0–10 (healthy) | -| `xrpld_State_Accounting_Disconnected_duration` | NetworkOPs.cpp | Cumulative seconds in Disconnected state | Monotonic | -| `xrpld_State_Accounting_Connected_duration` | NetworkOPs.cpp | Cumulative seconds in Connected state | Monotonic | -| `xrpld_State_Accounting_Syncing_duration` | NetworkOPs.cpp | Cumulative seconds in Syncing state | Monotonic | -| `xrpld_State_Accounting_Tracking_duration` | NetworkOPs.cpp | Cumulative seconds in Tracking state | Monotonic | -| `xrpld_State_Accounting_Full_duration` | NetworkOPs.cpp | Cumulative seconds in Full state | Monotonic (should dominate) | -| `xrpld_State_Accounting_Disconnected_transitions` | NetworkOPs.cpp | Count of transitions to Disconnected | Low | -| `xrpld_State_Accounting_Connected_transitions` | NetworkOPs.cpp | Count of transitions to Connected | Low | -| `xrpld_State_Accounting_Syncing_transitions` | NetworkOPs.cpp | Count of transitions to Syncing | Low | -| `xrpld_State_Accounting_Tracking_transitions` | NetworkOPs.cpp | Count of transitions to Tracking | Low | -| `xrpld_State_Accounting_Full_transitions` | NetworkOPs.cpp | Count of transitions to Full | Low (should be 1 after startup) | -| `xrpld_Peer_Finder_Active_Inbound_Peers` | PeerfinderManager.cpp | Active inbound peer connections | 0–85 | -| `xrpld_Peer_Finder_Active_Outbound_Peers` | PeerfinderManager.cpp | Active outbound peer connections | 10–21 | -| `xrpld_Overlay_Peer_Disconnects` | OverlayImpl.cpp | Cumulative peer disconnection count | Low growth | -| `xrpld_jobq_job_count` | JobQueue.cpp | Current job queue depth (all types) | 0–100 (healthy) | -| `xrpld_Node_family_full_below_cache_size` | TaggedCache.h | FullBelowCache entry count | Varies | -| `xrpld_Node_family_full_below_cache_hit_rate` | TaggedCache.h | FullBelowCache hit rate percentage | 0–100 | +| Prometheus Metric | Source File | Description | Typical Range | +| ------------------------------------------------- | --------------------- | --------------------------------------------- | ------------------------------- | +| `xrpld_LedgerMaster_Validated_Ledger_Age` | LedgerMaster.h | Seconds since last validated ledger | 0–10 (healthy), >30 (stale) | +| `xrpld_LedgerMaster_Published_Ledger_Age` | LedgerMaster.h | Seconds since last published ledger | 0–10 (healthy) | +| `xrpld_State_Accounting_Disconnected_duration` | NetworkOPs.cpp | Cumulative microseconds in Disconnected state | Monotonic | +| `xrpld_State_Accounting_Connected_duration` | NetworkOPs.cpp | Cumulative microseconds in Connected state | Monotonic | +| `xrpld_State_Accounting_Syncing_duration` | NetworkOPs.cpp | Cumulative microseconds in Syncing state | Monotonic | +| `xrpld_State_Accounting_Tracking_duration` | NetworkOPs.cpp | Cumulative microseconds in Tracking state | Monotonic | +| `xrpld_State_Accounting_Full_duration` | NetworkOPs.cpp | Cumulative microseconds in Full state | Monotonic (should dominate) | +| `xrpld_State_Accounting_Disconnected_transitions` | NetworkOPs.cpp | Count of transitions to Disconnected | Low | +| `xrpld_State_Accounting_Connected_transitions` | NetworkOPs.cpp | Count of transitions to Connected | Low | +| `xrpld_State_Accounting_Syncing_transitions` | NetworkOPs.cpp | Count of transitions to Syncing | Low | +| `xrpld_State_Accounting_Tracking_transitions` | NetworkOPs.cpp | Count of transitions to Tracking | Low | +| `xrpld_State_Accounting_Full_transitions` | NetworkOPs.cpp | Count of transitions to Full | Low (should be 1 after startup) | +| `xrpld_Peer_Finder_Active_Inbound_Peers` | PeerfinderManager.cpp | Active inbound peer connections | 0–85 | +| `xrpld_Peer_Finder_Active_Outbound_Peers` | PeerfinderManager.cpp | Active outbound peer connections | 10–21 | +| `xrpld_Overlay_Peer_Disconnects` | OverlayImpl.cpp | Cumulative peer disconnection count | Low growth | +| `xrpld_jobq_job_count` | JobQueue.cpp | Current job queue depth (all types) | 0–100 (healthy) | +| `xrpld_Node_family_full_below_cache_size` | TaggedCache.h | FullBelowCache entry count | Varies | +| `xrpld_Node_family_full_below_cache_hit_rate` | TaggedCache.h | FullBelowCache hit rate percentage | 0–100 | **Grafana dashboard**: _Node Health (StatsD)_ (`xrpld-statsd-node-health`) @@ -894,12 +896,15 @@ All span names and attributes are defined as compile-time constants in colocated | `src/xrpld/rpc/detail/RpcSpanNames.h` | RPC (HTTP/WS) | 5 | 5 | Includes `rpc.ws_upgrade` error path | | `src/xrpld/rpc/detail/PathFindSpanNames.h` | PathFind | 5 | 8 | Covers one-shot and subscription paths | | `src/xrpld/app/main/GrpcSpanNames.h` | gRPC | 1 | 3 | Flat single-span structure per request | -| `src/xrpld/app/misc/TxSpanNames.h` | Transaction | 2 | 7 | Includes peer context attributes | -| `src/xrpld/app/misc/detail/TxQSpanNames.h` | TxQ | 6 | 11 | Queue lifecycle: enqueue through cleanup | -| `src/xrpld/app/consensus/ConsensusSpanNames.h` | Consensus | 10 | 35 | Deterministic trace IDs, close-time details | +| `src/xrpld/telemetry/TxSpanNames.h` | Transaction | 2 | 7 | Includes peer context attributes | +| `include/xrpl/tx/detail/TxApplySpanNames.h` | Tx Apply | 3 | 6 | Apply pipeline: preflight, preclaim, apply | +| `src/xrpld/app/misc/detail/TxQSpanNames.h` | TxQ | 6 | 13 | Queue lifecycle: enqueue through cleanup | +| `include/xrpl/consensus/ConsensusSpanNames.h` | Consensus | 10 | 35 | Deterministic trace IDs, close-time details | | `src/xrpld/app/ledger/detail/LedgerSpanNames.h` | Ledger | 4 | 7 | Build, store, validate, tx.apply | | `src/xrpld/overlay/detail/PeerSpanNames.h` | Peer Overlay | 2 | 5 | Proposal and validation receive | +Column totals: **38 spans** and **89 attribute rows**, matching §1.1 and §1.2. `tx.apply` is counted under `LedgerSpanNames.h`, which defines it; §1.1 lists it with the transaction spans. + > **Design convention**: SpanNames headers are colocated with their subsystem classes rather than centralized in `telemetry/`. See [memory/feedback_span-names-colocation.md](../.claude/memory/feedback_span-names-colocation.md) for rationale. --- diff --git a/OpenTelemetryPlan/OpenTelemetryPlan.md b/OpenTelemetryPlan/OpenTelemetryPlan.md index 50a24e3670..43d9bda802 100644 --- a/OpenTelemetryPlan/OpenTelemetryPlan.md +++ b/OpenTelemetryPlan/OpenTelemetryPlan.md @@ -206,7 +206,7 @@ The appendix contains a glossary of OpenTelemetry and xrpld-specific terms, refe ## 9. Data Collection Reference -A single-source-of-truth reference documenting every piece of telemetry data collected by xrpld. Covers all 16 OpenTelemetry spans with their 22 attributes, all StatsD metrics (gauges, counters, histograms, overlay traffic), SpanMetrics-derived Prometheus metrics, and all 10 Grafana dashboards. Includes Jaeger search guides and Prometheus query examples. +A single-source-of-truth reference documenting every piece of telemetry data collected by xrpld. Covers all 38 OpenTelemetry spans with their 89 attribute rows (78 unique keys), all StatsD metrics (gauges, counters, histograms, overlay traffic), SpanMetrics-derived Prometheus metrics, and all 10 Grafana dashboards. Includes Tempo trace search guides and Prometheus query examples. ➡️ **[View Data Collection Reference](./09-data-collection-reference.md)** diff --git a/docker/telemetry/integration-test.sh b/docker/telemetry/integration-test.sh index ba0974f2f4..0ba2327da7 100755 --- a/docker/telemetry/integration-test.sh +++ b/docker/telemetry/integration-test.sh @@ -62,7 +62,10 @@ die() { check_span() { local op="$1" local count - count=$(curl -sf "$TEMPO/api/search" \ + # -G is required: it moves the urlencoded params into the query string. + # Without it curl POSTs them as a request body, and Tempo answers 200 + # while ignoring the query — so every span name would look present. + count=$(curl -sfG "$TEMPO/api/search" \ --data-urlencode "q={resource.service.name=\"xrpld\" && name=\"$op\"}" \ --data-urlencode "limit=5" | jq '.traces | length' 2>/dev/null || echo 0) @@ -133,6 +136,23 @@ mkdir -p "$WORKDIR" # --------------------------------------------------------------------------- # Step 2: Start observability stack # --------------------------------------------------------------------------- + +# From here on the script owns the docker stack and the xrpld nodes, so an +# abort must tear them down instead of leaving them behind. A run that +# reaches the summary deliberately leaves everything up for inspection +# (see the header comment), so the trap only fires before that point. +RUN_COMPLETED=0 +on_exit() { + local status=$? + if [ "$RUN_COMPLETED" -eq 0 ]; then + log "Aborted with exit status $status — tearing down." + cleanup + fi +} +trap on_exit EXIT +trap 'exit 130' INT +trap 'exit 143' TERM + log "Starting observability stack..." docker compose -f "$COMPOSE_FILE" up -d @@ -358,12 +378,14 @@ log "Waiting for nodes to reach 'proposing' state (timeout: ${CONSENSUS_TIMEOUT} start_time=$(date +%s) nodes_ready=0 +consensus_timed_out=0 while [ "$nodes_ready" -lt "$NUM_NODES" ]; do elapsed=$(($(date +%s) - start_time)) if [ "$elapsed" -ge "$CONSENSUS_TIMEOUT" ]; then fail "Consensus timeout after ${CONSENSUS_TIMEOUT}s ($nodes_ready/$NUM_NODES nodes ready)" log "Continuing with partial consensus..." + consensus_timed_out=1 break fi @@ -386,7 +408,10 @@ echo "" if [ "$nodes_ready" -eq "$NUM_NODES" ]; then ok "All $NUM_NODES nodes reached 'proposing' state" -else +elif [ "$consensus_timed_out" -eq 0 ]; then + # The timeout branch above already called fail(), so reporting again here + # would count one timeout twice. Only reachable if the loop ever gains + # another early exit. fail "Only $nodes_ready/$NUM_NODES nodes reached 'proposing' state" fi @@ -430,9 +455,11 @@ log "Submitting Payment transaction..." # Generate a destination wallet log " Generating destination wallet..." +# Guarded: under set -e an unguarded curl failure would abort the whole +# script, so the fallback below could never run. wallet_result=$(curl -sf "http://localhost:$RPC_PORT_BASE" \ - -d '{"method":"wallet_propose"}') -DEST_ACCOUNT=$(echo "$wallet_result" | jq -r '.result.account_id' 2>/dev/null) + -d '{"method":"wallet_propose"}') || wallet_result="" +DEST_ACCOUNT=$(echo "$wallet_result" | jq -r '.result.account_id' 2>/dev/null || echo "") if [ -z "$DEST_ACCOUNT" ] || [ "$DEST_ACCOUNT" = "null" ]; then fail "Could not generate destination wallet" DEST_ACCOUNT="rrrrrrrrrrrrrrrrrrrrrhoLvTp" # ACCOUNT_ZERO fallback @@ -441,13 +468,13 @@ log " Destination: $DEST_ACCOUNT" # Get genesis account info acct_result=$(curl -sf "http://localhost:$RPC_PORT_BASE" \ - -d "{\"method\":\"account_info\",\"params\":[{\"account\":\"$GENESIS_ACCOUNT\"}]}") + -d "{\"method\":\"account_info\",\"params\":[{\"account\":\"$GENESIS_ACCOUNT\"}]}") || acct_result="" seq_num=$(echo "$acct_result" | jq -r '.result.account_data.Sequence' 2>/dev/null || echo "unknown") log " Genesis account sequence: $seq_num" # Submit payment submit_result=$(curl -sf "http://localhost:$RPC_PORT_BASE" \ - -d "{\"method\":\"submit\",\"params\":[{\"secret\":\"$GENESIS_SEED\",\"tx_json\":{\"TransactionType\":\"Payment\",\"Account\":\"$GENESIS_ACCOUNT\",\"Destination\":\"$DEST_ACCOUNT\",\"Amount\":\"10000000\"}}]}") + -d "{\"method\":\"submit\",\"params\":[{\"secret\":\"$GENESIS_SEED\",\"tx_json\":{\"TransactionType\":\"Payment\",\"Account\":\"$GENESIS_ACCOUNT\",\"Destination\":\"$DEST_ACCOUNT\",\"Amount\":\"10000000\"}}]}") || submit_result="" engine_result=$(echo "$submit_result" | jq -r '.result.engine_result' 2>/dev/null || echo "unknown") tx_hash=$(echo "$submit_result" | jq -r '.result.tx_json.hash' 2>/dev/null || echo "unknown") @@ -478,7 +505,7 @@ fi log "" log "--- RPC Spans ---" -check_span "rpc.request" +check_span "rpc.http_request" check_span "rpc.process" check_span "rpc.command.server_info" check_span "rpc.command.server_state" @@ -580,6 +607,11 @@ check_statsd_metric "rippled_total_Bytes_In" # --------------------------------------------------------------------------- # Step 11: Summary # --------------------------------------------------------------------------- + +# All checks are done, so the run counts as complete: keep the stack and the +# nodes up for inspection even when some checks failed. +RUN_COMPLETED=1 + echo "" echo "===========================================================" echo " INTEGRATION TEST RESULTS" diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index ef4d3f2dec..9c5337508b 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -160,8 +160,9 @@ hash); `tx.preflight` is stateless and omits both. # Find specific ledger's consensus details {name="consensus.accept.apply"} | ledger_seq = 92345678 -# Find all spans in a consensus round (deterministic trace strategy) -{name="consensus.round"} | consensus_round_id = +# Find all spans in a consensus round (deterministic trace strategy). +# consensus_round_id is an int64 — the previous ledger sequence plus one. +{name="consensus.round"} | consensus_round_id = 92345679 # Find dispute resolutions {name="consensus.update_positions"} >> {event:name="dispute.resolve"} @@ -249,10 +250,14 @@ sum by (stage) (rate(traces_span_metrics_calls_total{span_name=~"tx.preflight|tx > a rising `tx.transactor` failure rate points to apply-time problems. Alert per > stage rather than on a single aggregate so the failing stage is obvious. -> **Sampling caveat**: these stage metrics are span-derived and inherit the -> **tracer head-sampling** ratio (`sampling_ratio`). At `sampling_ratio < 1.0` -> they undercount proportionally — treat them as relative trends, not absolute -> transaction counts. Native StatsD metrics are unsampled. +> **Sampling caveat**: these stage metrics are span-derived, so they count only +> the spans the collector's spanmetrics connector sees. Head sampling at the node +> is fixed at 1.0 and is not configurable (`Telemetry.h`), and the shipped +> collector pipeline has no tail sampling, so today nothing is dropped and the +> counts are absolute. Volume reduction is delegated to the collector: adding a +> tail-sampling processor to the traces pipeline puts it ahead of the spanmetrics +> connector, and these metrics would then undercount proportionally — treat them +> as relative trends in that case. Native StatsD metrics are never sampled. ### Transaction Queue Health @@ -452,7 +457,7 @@ all its normal attributes, it just lacks a cross-node parent link. {name=~"tx\\..*"} | tx_hash = "" # Find all spans in a cross-node consensus trace -{rootServiceName="xrpld"} | consensus_round_id = +{rootServiceName="xrpld"} | consensus_round_id = 92345679 # Compare latency between sender and receiver for validations {name="consensus.validation.send" || name="consensus.validation.receive"} @@ -656,7 +661,7 @@ Ten dashboards are pre-provisioned in `docker/telemetry/grafana/dashboards/`: | RPC Latency p95 by Command | timeseries | `histogram_quantile(0.95, sum by (le, command) (rate(traces_span_metrics_duration_milliseconds_bucket{span_name=~"rpc.command.*"}[5m])))` | `command` | | RPC Error Rate | bargauge | Error spans / total spans × 100, grouped by `command` | `command`, `status_code` | | RPC Latency Heatmap | heatmap | `sum(increase(traces_span_metrics_duration_milliseconds_bucket{span_name=~"rpc.command.*"}[5m])) by (le)` | `le` (bucket boundaries) | -| Overall RPC Throughput | timeseries | `rpc.request` + `rpc.process` rate | — | +| Overall RPC Throughput | timeseries | `rpc.http_request` + `rpc.process` rate | — | | RPC Success vs Error | timeseries | by `status_code` (UNSET vs ERROR) | `status_code` | | Top Commands by Volume | bargauge | `topk(10, ...)` by `command` | `command` | | WebSocket Message Rate | stat | `rpc.ws_message` rate | — | From 2a1860951a9b42395cf64a4f3550b40217cd8655 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Sat, 15 Aug 2026 17:30:40 +0100 Subject: [PATCH 2/3] fix(telemetry): repair the five silently-empty TraceQL panels on consensus-health MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All five TraceQL panels on this dashboard returned nothing, and did so without any visible error: they filtered on span.close_time_correct=~"$close_time_correct", but close_time_correct is a boolean attribute (RCLConsensus.cpp:601 passes a raw bool), and Tempo restricts the regex operator to string operands, so the spanset resolved to false. With the variable defaulting to All the clause rendered as =~".*", so the panels were empty out of the box and looked exactly like a node with no consensus activity. Note this is the opposite of PromQL, where an absent or empty label does match ".*" — which is why the 17 Prometheus panels on this same board were unaffected and the dashboard appeared healthy. Dropped the clause from all six queries, matching phases 9 and 10 where it is already gone. The $close_time_correct variable now filters the Prometheus "Close Time Agreement" panel instead, which already grouped by that label but never filtered on it, so the control stays useful rather than becoming dead UI. Two defects were masked behind the empty panels and are fixed too: - "Close Time: Raw Proposals" and "Close Time: Effective / Quantized" carried unit dateTimeFromNow over close_time_self/close_time, which are NetClock seconds (Ripple epoch), while Grafana's dateTime formatters expect a millisecond Unix epoch — every point would have rendered as roughly 1970. They now plot as plain numbers with the axis labelled "NetClock Seconds (Ripple Epoch)", and the descriptions give the 946684800 offset for converting to Unix time. - "Close Time Vote Bins & Resolution" matched its unit and axis overrides byName against "Vote Bins" and "Resolution", which are not field names; TraceQL select() yields close_time_vote_bins and close_resolution_ms, so neither override applied. Switched to byRegexp so the match holds whichever scope prefix Grafana emits. No panel was added or removed: the (type, title) multiset is unchanged at 22. resolution_direction keeps its regex filter, which is correct there — it is set from a std::string whose values are exactly the variable's increased/decreased/unchanged. --- .../grafana/dashboards/consensus-health.json | 36 ++++++++++--------- 1 file changed, 19 insertions(+), 17 deletions(-) diff --git a/docker/telemetry/grafana/dashboards/consensus-health.json b/docker/telemetry/grafana/dashboards/consensus-health.json index 3f03f9bbd7..bc42930978 100644 --- a/docker/telemetry/grafana/dashboards/consensus-health.json +++ b/docker/telemetry/grafana/dashboards/consensus-health.json @@ -229,7 +229,7 @@ "datasource": { "type": "prometheus" }, - "expr": "sum by (close_time_correct, service_instance_id) (rate(traces_span_metrics_calls_total{span_name=\"consensus.accept.apply\", consensus_mode=~\"$consensus_mode\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval]))", + "expr": "sum by (close_time_correct, service_instance_id) (rate(traces_span_metrics_calls_total{span_name=\"consensus.accept.apply\", close_time_correct=~\"$close_time_correct\", consensus_mode=~\"$consensus_mode\", service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\"}[$__rate_interval]))", "legendFormat": "Close Time Correct={{close_time_correct}} [{{service_instance_id}}]" } ], @@ -430,7 +430,7 @@ }, { "title": "Close Time: Raw Proposals (Per Node)", - "description": "**What:** Each node's raw, unrounded proposed close time at the instant it closed its ledger.\n**How it's computed:** Latest raw close-time value per node, plotted per round.\n**Reading it:** Compare nodes at the same round; values should cluster tightly.\n**Healthy range:** All nodes within a few seconds of each other.\n**Watch for:** A node consistently offset from the pack indicates local clock drift.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::doAccept", + "description": "**What:** Each node's raw, unrounded proposed close time at the instant it closed its ledger.\n**How it's computed:** Latest raw close-time value per node, plotted per round. Values are NetClock seconds (Ripple epoch, 2000-01-01); add 946684800 for Unix time.\n**Reading it:** Compare nodes at the same round; values should cluster tightly.\n**Healthy range:** All nodes within a few seconds of each other.\n**Watch for:** A node consistently offset from the pack indicates local clock drift.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::doAccept", "type": "timeseries", "gridPos": { "h": 8, @@ -440,12 +440,13 @@ }, "fieldConfig": { "defaults": { - "unit": "dateTimeFromNow", + "unit": "none", "custom": { "spanNulls": 1800000, "drawStyle": "points", "pointSize": 6, - "showPoints": "always" + "showPoints": "always", + "axisLabel": "NetClock Seconds (Ripple Epoch)" } }, "overrides": [] @@ -468,14 +469,14 @@ "type": "tempo" }, "queryType": "traceql", - "query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\" && span.close_time_correct=~\"$close_time_correct\"} | select(span.close_time_self)", + "query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\"} | select(span.close_time_self)", "refId": "A" } ] }, { "title": "Close Time: Effective / Quantized", - "description": "**What:** The consensus-agreed close time after rounding to the active resolution bin, i.e. the value written to the ledger header.\n**How it's computed:** Latest effective close-time value per node, plotted per round.\n**Reading it:** All in-agreement nodes should report the identical value each round.\n**Healthy range:** Identical across agreeing nodes.\n**Watch for:** Nodes reporting different effective values are not in close-time agreement for that round.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::doAccept", + "description": "**What:** The consensus-agreed close time after rounding to the active resolution bin, i.e. the value written to the ledger header.\n**How it's computed:** Latest effective close-time value per node, plotted per round. Values are NetClock seconds (Ripple epoch, 2000-01-01); add 946684800 for Unix time.\n**Reading it:** All in-agreement nodes should report the identical value each round.\n**Healthy range:** Identical across agreeing nodes.\n**Watch for:** Nodes reporting different effective values are not in close-time agreement for that round.\n**Source:** src/xrpld/app/consensus/RCLConsensus.cpp:RCLConsensus::Adaptor::doAccept", "type": "timeseries", "gridPos": { "h": 8, @@ -485,12 +486,13 @@ }, "fieldConfig": { "defaults": { - "unit": "dateTimeFromNow", + "unit": "none", "custom": { "spanNulls": 1800000, "drawStyle": "points", "pointSize": 6, - "showPoints": "always" + "showPoints": "always", + "axisLabel": "NetClock Seconds (Ripple Epoch)" } }, "overrides": [] @@ -513,7 +515,7 @@ "type": "tempo" }, "queryType": "traceql", - "query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\" && span.close_time_correct=~\"$close_time_correct\"} | select(span.close_time)", + "query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\"} | select(span.close_time)", "refId": "A" } ] @@ -541,8 +543,8 @@ "overrides": [ { "matcher": { - "id": "byName", - "options": "Vote Bins" + "id": "byRegexp", + "options": ".*vote_bins.*" }, "properties": [ { @@ -557,8 +559,8 @@ }, { "matcher": { - "id": "byName", - "options": "Resolution" + "id": "byRegexp", + "options": ".*close_resolution_ms.*" }, "properties": [ { @@ -591,7 +593,7 @@ "type": "tempo" }, "queryType": "traceql", - "query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\" && span.close_time_correct=~\"$close_time_correct\"} | select(span.close_time_vote_bins)", + "query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\"} | select(span.close_time_vote_bins)", "refId": "A" }, { @@ -599,7 +601,7 @@ "type": "tempo" }, "queryType": "traceql", - "query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\" && span.close_time_correct=~\"$close_time_correct\"} | select(span.close_resolution_ms)", + "query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\"} | select(span.close_resolution_ms)", "refId": "B" } ] @@ -644,7 +646,7 @@ "type": "tempo" }, "queryType": "traceql", - "query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\" && span.close_time_correct=~\"$close_time_correct\" && span.resolution_direction=~\"$resolution_direction\"} | select(span.resolution_direction)", + "query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\" && span.resolution_direction=~\"$resolution_direction\"} | select(span.resolution_direction)", "refId": "A" } ] @@ -690,7 +692,7 @@ "type": "tempo" }, "queryType": "traceql", - "query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\" && span.close_time_correct=~\"$close_time_correct\"} | select(span.close_time, span.close_time_vote_bins)", + "query": "{name=\"consensus.accept.apply\" && resource.service.instance.id=~\"$node\"} | select(span.close_time, span.close_time_vote_bins)", "refId": "A" } ] From 3a3ae1c3aeba6528ffa01b53d9056691a4bf3556 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Mon, 17 Aug 2026 12:02:46 +0100 Subject: [PATCH 3/3] fix(telemetry): name the job-queue depth metric jobq_job_count The integration test asserted `rippled_job_count`, which never reports any series, so that check always failed. `JobQueue` registers the gauge as `makeGauge("job_count")`, but `Application.cpp` passes it `collectorManager_->group("jobq")`, so the emitted StatsD name is `jobq.job_count` and the exported Prometheus name is `_jobq_job_count`. Corrected the same name in two runbook tables that also dropped the `jobq` segment. `09-data-collection-reference.md` already had it right, which is why the two documents disagreed. Routed here rather than to the phase-10 PR where it was reported: the wrong name is present in `integration-test.sh` on every branch from phase 6 onward, and this is the branch that introduces the file. Left alone deliberately: - `statsd-node-health.json` still queries the old name, but that dashboard is deleted at phase 7 in favour of `node-health.json` - `06-implementation-phases.md` names `job_count`, which is accurate as the code-level makeGauge argument rather than the exported metric name --- docker/telemetry/integration-test.sh | 2 +- docs/telemetry-runbook.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docker/telemetry/integration-test.sh b/docker/telemetry/integration-test.sh index 0ba2327da7..fb9f4eda06 100755 --- a/docker/telemetry/integration-test.sh +++ b/docker/telemetry/integration-test.sh @@ -589,7 +589,7 @@ check_statsd_metric() { # Node health gauges check_statsd_metric "rippled_LedgerMaster_Validated_Ledger_Age" check_statsd_metric "rippled_LedgerMaster_Published_Ledger_Age" -check_statsd_metric "rippled_job_count" +check_statsd_metric "rippled_jobq_job_count" # State accounting check_statsd_metric "rippled_State_Accounting_Full_duration" diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index 9c5337508b..0c7a2ca68d 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -541,7 +541,7 @@ The OTel Collector receives these via a `statsd` receiver on UDP port 8125 and e | `xrpld_Peer_Finder_Active_Inbound_Peers` | PeerfinderManager.cpp:214 | Active inbound peer connections | | `xrpld_Peer_Finder_Active_Outbound_Peers` | PeerfinderManager.cpp:215 | Active outbound peer connections | | `xrpld_Overlay_Peer_Disconnects` | OverlayImpl.h:557 | Peer disconnect count | -| `xrpld_job_count` | JobQueue.cpp:26 | Current job queue depth | +| `xrpld_jobq_job_count` | JobQueue.cpp:26 | Current job queue depth | | `xrpld_{category}_Bytes_In/Out` | OverlayImpl.h:535 | Overlay traffic bytes per category (57 categories) | | `xrpld_{category}_Messages_In/Out` | OverlayImpl.h:535 | Overlay traffic messages per category | @@ -728,7 +728,7 @@ Requires `trace_peer=1` in the `[telemetry]` config section. | Operating Mode Duration | timeseries | `xrpld_State_Accounting_*_duration` | — | | Operating Mode Transitions | timeseries | `xrpld_State_Accounting_*_transitions` | — | | I/O Latency | timeseries | `histogram_quantile(0.95, xrpld_ios_latency_bucket)` | — | -| Job Queue Depth | timeseries | `xrpld_job_count` | — | +| Job Queue Depth | timeseries | `xrpld_jobq_job_count` | — | | Ledger Fetch Rate | stat | `rate(xrpld_ledger_fetches[5m])` | — | | Ledger History Mismatches | stat | `rate(xrpld_ledger_history_mismatch[5m])` | — | | Key Jobs Execution Time | timeseries | `xrpld_acceptLedger{quantile="$quantile"}` (+ 10 more key jobs) | `quantile` |