mirror of
https://github.com/XRPLF/rippled.git
synced 2026-09-27 23:38:08 +00:00
e5d7b2a4b0
c531ac569basserted txq.accept -> txq.accept_tx. Run 32990348089 failed it: "txq.accept_tx not found in txq.accept traces", the only failure in 278 checks. Skipped rather than left red. Not a missing span, and not an xrpld defect. Both ends emit on that same run, 5 traces each with all their attributes. The assertion was simply stronger than the check can evaluate, and the reason is a conditional child meeting newest-N sampling. The parent is created once per accept pass, so every ledger close (TxQ.cpp:1499). The child is created inside the loop over queued transactions and behind `if (feeLevelPaid >= requiredFeeLevel)` (TxQ.cpp:1530), so it exists only for a close where the queue actually held a transaction whose fee cleared the level. _validate_parent_child searches the parent with limit=3 (validate_telemetry.py:803). Queue pressure comes from workload phase 5 of 7, txq-burst, and mixed-peak (60s) then cooldown (30s) run after it -- so by the time validation queries, the three newest txq.accept traces are quiet closes with an empty queue and no child to find. That is the same shape as the rpc.command.* skips already in this file: sampling the newest traces of the parent is wrong whenever the child is conditional on load that has since stopped. Recorded in the reason, with the two real fixes in preference order -- prefer parent traces that contain the child via a TraceQL child filter instead of newest-N, or move txq-burst to the final workload phase. Raising the limit alone only shifts the odds, which would make the check flaky rather than correct, so it is named and rejected there. The other 13 assertions added inc531ac569ball PASS, including the three consensus.round children, the two consensus.establish children, rpc.http_request -> rpc.process and the three ledger.acquire phases. The regression gate is clean at 0 regressions now that phase-10 recaptured the baseline, and both reverse-coverage checks pass. Verification: JSON parses; 18 relationships, 13 asserted and 5 skipped; counters still 41 span types; churn 3/1, surgical; otel-naming exits 0; pre-commit clean.
492 lines
31 KiB
JSON
492 lines
31 KiB
JSON
{
|
|
"description": "Expected span inventory for xrpld telemetry validation. Attribute keys follow the 2026-05-13 span-attr naming redesign (bare/underscore form; dotted xrpl.* reserved for resource attributes). Sourced from the *SpanNames.h headers and verified against the emitting call sites. Spans marked \"optional\": true are conditional — they only fire under traffic the harness may not produce (e.g. gRPC client, path-finding RPC, missing-ledger fetch, mode transitions) and are not failed when absent. \"parent\" is documentation only (validate_telemetry.py asserts hierarchy from parent_child_relationships, not from this field) and records the parent as the code actually produces it: null means the span is a root or an explicit freshRoot. required_attributes lists only attributes set on EVERY code path that creates the span — attributes set after an early return are described in the span's note instead, because _validate_span_attributes_otlp samples a single trace and would fail on a legitimate short-circuit path. total_unique_attributes is the size of the union of all required_attributes; total_span_types is len(spans). Span EVENTS (consensus.round phase.*/outcome.*, consensus.update_positions dispute.resolve, consensus.accept.apply tx.included) are NOT represented: validate_telemetry.py reads only span name, attributes and timestamps from Tempo, so an \"events\" key would be silently ignored. They are documented in the relevant span notes until the validator gains event support.",
|
|
"spans": [
|
|
{
|
|
"name": "rpc.ws_message",
|
|
"category": "rpc",
|
|
"parent": null,
|
|
"required_attributes": ["command"],
|
|
"config_flag": "trace_rpc",
|
|
"note": "WebSocket RPC root span. The load generator uses WS, so this is the RPC entry span (not rpc.http_request, which needs an HTTP/JSON-RPC client)."
|
|
},
|
|
{
|
|
"name": "rpc.ws_upgrade",
|
|
"category": "rpc",
|
|
"parent": null,
|
|
"required_attributes": [],
|
|
"config_flag": "trace_rpc",
|
|
"optional": true,
|
|
"note": "WebSocket handshake span (ServerHandler::onHandoff, ServerHandler.cpp:272-273). A freshRoot with no attributes — only setOk() on success or recordException() on an upgrade failure. Fires once per WS connection, so the load generator produces only a handful of these at connect time; by the time validation runs after the propagation wait they may fall outside the Tempo search window. Optional for that reason, not because the code path is conditional."
|
|
},
|
|
{
|
|
"name": "rpc.process",
|
|
"category": "rpc",
|
|
"parent": "rpc.http_request",
|
|
"required_attributes": [],
|
|
"config_flag": "trace_rpc",
|
|
"optional": true,
|
|
"note": "HTTP-only. Created solely in ServerHandler::processRequest() (ServerHandler.cpp:705), which is reached only from processSession(Session, coro) (ServerHandler.cpp:646) — the HTTP/JSON-RPC path that roots rpc.http_request at ServerHandler.cpp:640-641. The WebSocket path (processSession(WSSession, coro, jv), ServerHandler.cpp:467) never calls processRequest, so this span never appears under a WebSocket request. It does still appear under this harness, 5 traces on a normal run, because run-full-validation.sh polls each node over HTTP with curl (:449, :502) and those requests take the HTTP path. Corrected 2026-08-26; this note previously concluded the span cannot appear at all, which sent a reader looking for a way to make the harness speak HTTP that it already speaks."
|
|
},
|
|
{
|
|
"name": "rpc.command.*",
|
|
"category": "rpc",
|
|
"parent": "rpc.ws_message",
|
|
"required_attributes": ["command", "version", "rpc_role", "rpc_status"],
|
|
"config_flag": "trace_rpc",
|
|
"note": "Wildcard — matches rpc.command.server_info, rpc.command.ledger, etc. Created as an ambient (scoped) child in rpc::doCommand / rpc::callMethod (RPCHandler.cpp:168, :271), so its parent is whichever transport span is active on the thread: rpc.ws_message on the WebSocket path (the harness workload) and rpc.process on the HTTP/JSON-RPC path."
|
|
},
|
|
{
|
|
"name": "rpc.http_request",
|
|
"category": "rpc",
|
|
"parent": null,
|
|
"required_attributes": ["request_payload_size"],
|
|
"config_flag": "trace_rpc",
|
|
"optional": true,
|
|
"note": "HTTP/JSON-RPC root span. It DOES fire under this harness, 5 traces on a normal run -- one per node -- even though the load generator is WebSocket-only, because run-full-validation.sh polls each node's HTTP port with curl for readiness and validated-ledger progress (:449, :502). Corrected 2026-08-26; this note previously said it does not fire. Kept optional rather than promoted to required because those polls are harness scaffolding rather than workload: a future change to how the script waits for a node could remove them without anything being wrong with the node."
|
|
},
|
|
{
|
|
"name": "tx.process",
|
|
"category": "transaction",
|
|
"parent": null,
|
|
"required_attributes": ["tx_hash", "local", "path"],
|
|
"config_flag": "trace_transactions"
|
|
},
|
|
{
|
|
"name": "tx.receive",
|
|
"category": "transaction",
|
|
"parent": null,
|
|
"required_attributes": ["tx_hash", "peer_id", "suppressed"],
|
|
"config_flag": "trace_transactions",
|
|
"note": "Cross-node span: parent context propagated from the sender's tx.process via protobuf. Also carries tx_type and peer_version. tx_status is only set when a tx is suppressed/known-bad, so it is not a required attribute on every tx.receive."
|
|
},
|
|
{
|
|
"name": "tx.apply",
|
|
"category": "transaction",
|
|
"parent": "ledger.build",
|
|
"required_attributes": ["tx_count", "tx_failed"],
|
|
"config_flag": "trace_transactions",
|
|
"note": "Apply-step span inside BuildLedger. Carries tx_count/tx_failed (ledger_seq lives on the parent ledger.build span)."
|
|
},
|
|
{
|
|
"name": "tx.preflight",
|
|
"category": "transaction",
|
|
"parent": null,
|
|
"required_attributes": ["stage", "tx_type", "ter_result"],
|
|
"config_flag": "trace_transactions",
|
|
"note": "Apply-pipeline stage span (stage=preflight). Shares a deterministic trace_id (txID[0:16]) with tx.preclaim/tx.transactor."
|
|
},
|
|
{
|
|
"name": "tx.preclaim",
|
|
"category": "transaction",
|
|
"parent": null,
|
|
"required_attributes": ["stage", "tx_type", "ter_result"],
|
|
"config_flag": "trace_transactions",
|
|
"note": "Apply-pipeline stage span (stage=preclaim)."
|
|
},
|
|
{
|
|
"name": "tx.transactor",
|
|
"category": "transaction",
|
|
"parent": null,
|
|
"required_attributes": ["stage", "tx_type"],
|
|
"config_flag": "trace_transactions",
|
|
"note": "Apply-pipeline stage span (stage=apply). Also carries applied."
|
|
},
|
|
{
|
|
"name": "txq.enqueue",
|
|
"category": "transaction",
|
|
"parent": "tx.process",
|
|
"required_attributes": ["tx_hash", "tx_type", "txq_status"],
|
|
"config_flag": "trace_transactions",
|
|
"optional": true,
|
|
"note": "Only fires when a tx is queued (fee below open-ledger level). Requires fee escalation — driven by the txq-burst workload phase. tx_hash/tx_type/txq_status are set on every code path; fee_level_paid/required_fee_level are conditional (TxQ.cpp ~895-898, after the rejected and applied_direct early exits), so they are NOT guaranteed on every txq.enqueue span and cannot be required."
|
|
},
|
|
{
|
|
"name": "txq.apply_direct",
|
|
"category": "transaction",
|
|
"parent": "txq.enqueue",
|
|
"required_attributes": [],
|
|
"config_flag": "trace_transactions",
|
|
"optional": true,
|
|
"note": "Child of txq.enqueue when the tx applies directly without queueing."
|
|
},
|
|
{
|
|
"name": "txq.batch_clear",
|
|
"category": "transaction",
|
|
"parent": "txq.enqueue",
|
|
"required_attributes": ["num_cleared"],
|
|
"config_flag": "trace_transactions",
|
|
"optional": true
|
|
},
|
|
{
|
|
"name": "txq.accept",
|
|
"category": "transaction",
|
|
"parent": null,
|
|
"required_attributes": ["queue_size", "ledger_changed"],
|
|
"config_flag": "trace_transactions",
|
|
"optional": true,
|
|
"note": "Ledger-close accept loop (TxQ::accept, TxQ.cpp:1499). Only meaningful when the queue is non-empty. Root on BOTH call paths, verified: the consensus path (RCLConsensus.cpp:823, inside doAccept) and the switchLastClosedLedger jump path (NetworkOPs.cpp:2150). The span is a ScopedSpanGuard, so it adopts whatever OTel context is ambient — but consensus.accept and consensus.accept.apply are unscoped thread-free SpanGuards and activate() is never called outside unit tests, so no consensus span is ever the ambient parent on the JtAccept worker. ledger.build's ScopedSpanGuard has already been destroyed by the time OpenLedger::accept runs."
|
|
},
|
|
{
|
|
"name": "txq.accept_tx",
|
|
"category": "transaction",
|
|
"parent": "txq.accept",
|
|
"required_attributes": [
|
|
"tx_hash",
|
|
"ter_code",
|
|
"retries_remaining",
|
|
"txq_status"
|
|
],
|
|
"config_flag": "trace_transactions",
|
|
"optional": true
|
|
},
|
|
{
|
|
"name": "txq.cleanup",
|
|
"category": "transaction",
|
|
"parent": null,
|
|
"required_attributes": ["ledger_seq", "expired_count"],
|
|
"config_flag": "trace_transactions",
|
|
"optional": true,
|
|
"note": "TxQ::processClosedLedger (TxQ.cpp:1403). Root on BOTH call paths for the same reason as txq.accept: the consensus path (RCLConsensus.cpp:950) and the switchLastClosedLedger jump path (NetworkOPs.cpp:2121) both run with no consensus span activated as ambient context."
|
|
},
|
|
{
|
|
"name": "consensus.round",
|
|
"category": "consensus",
|
|
"parent": null,
|
|
"required_attributes": [
|
|
"consensus_ledger_id",
|
|
"ledger_seq",
|
|
"consensus_mode",
|
|
"consensus_round_id",
|
|
"consensus_phase"
|
|
],
|
|
"config_flag": "trace_consensus",
|
|
"note": "Root consensus span created per round. Also carries trace_strategy, previous_ledger_seq, previous_proposers, previous_round_time_ms. Emits seven span EVENTS that this manifest cannot assert: phase.open, phase.recovery, phase.establish, phase.accepted, outcome.yes, outcome.moved_on, outcome.expired (declared ConsensusSpanNames.h:265-277; emitted RCLConsensus.cpp:1344 and via onPhaseEvent/onOutcomeEvent from Consensus.h:764, 793, 1047, 1517-1525, 1530, 1566). validate_telemetry.py reads only span name, attributes and start/end timestamps from the Tempo OTLP payload — it has no event assertion support — so adding an \"events\" key here would be silently ignored. Recorded as a note instead; asserting events needs validator support first."
|
|
},
|
|
{
|
|
"name": "consensus.phase.open",
|
|
"category": "consensus",
|
|
"parent": "consensus.round",
|
|
"required_attributes": [],
|
|
"config_flag": "trace_consensus"
|
|
},
|
|
{
|
|
"name": "consensus.proposal.send",
|
|
"category": "consensus",
|
|
"parent": "consensus.round",
|
|
"required_attributes": ["consensus_round"],
|
|
"config_flag": "trace_consensus",
|
|
"note": "Also carries is_bow_out."
|
|
},
|
|
{
|
|
"name": "consensus.ledger_close",
|
|
"category": "consensus",
|
|
"parent": "consensus.round",
|
|
"required_attributes": ["ledger_seq", "consensus_mode"],
|
|
"config_flag": "trace_consensus",
|
|
"note": "Also carries tx_count_open, close_time_resolution_ms."
|
|
},
|
|
{
|
|
"name": "consensus.establish",
|
|
"category": "consensus",
|
|
"parent": "consensus.round",
|
|
"required_attributes": [
|
|
"converge_percent",
|
|
"establish_count",
|
|
"proposers",
|
|
"disputes_count"
|
|
],
|
|
"config_flag": "trace_consensus"
|
|
},
|
|
{
|
|
"name": "consensus.update_positions",
|
|
"category": "consensus",
|
|
"parent": "consensus.establish",
|
|
"required_attributes": [
|
|
"converge_percent",
|
|
"proposers",
|
|
"disputes_count"
|
|
],
|
|
"config_flag": "trace_consensus",
|
|
"note": "childSpan of establishSpanContext_ (Consensus.h:1628), so the parent is consensus.establish — not consensus.round. Also emits a dispute.resolve span EVENT per resolved dispute (Consensus.h:1697-1698), which validate_telemetry.py cannot assert (no event support)."
|
|
},
|
|
{
|
|
"name": "consensus.check",
|
|
"category": "consensus",
|
|
"parent": "consensus.establish",
|
|
"required_attributes": [
|
|
"agree_count",
|
|
"disagree_count",
|
|
"threshold_percent",
|
|
"consensus_result"
|
|
],
|
|
"config_flag": "trace_consensus",
|
|
"note": "childSpan of establishSpanContext_ (Consensus.h:1837), so the parent is consensus.establish — not consensus.round."
|
|
},
|
|
{
|
|
"name": "consensus.accept",
|
|
"category": "consensus",
|
|
"parent": "consensus.round",
|
|
"required_attributes": ["proposers", "round_time_ms", "quorum"],
|
|
"config_flag": "trace_consensus"
|
|
},
|
|
{
|
|
"name": "consensus.accept.apply",
|
|
"category": "consensus",
|
|
"parent": "consensus.accept",
|
|
"required_attributes": [
|
|
"ledger_seq",
|
|
"close_time",
|
|
"parent_close_time",
|
|
"close_time_self",
|
|
"close_time_vote_bins",
|
|
"resolution_direction"
|
|
],
|
|
"config_flag": "trace_consensus",
|
|
"note": "Also carries close_time_correct, close_resolution_ms, consensus_state, proposing, round_time_ms, tx_count. Emits a tx.included span EVENT per transaction in the accepted set (RCLConsensus.cpp:666, with a tx_id attribute), which validate_telemetry.py cannot assert (no event support)."
|
|
},
|
|
{
|
|
"name": "consensus.validation.send",
|
|
"category": "consensus",
|
|
"parent": null,
|
|
"required_attributes": [
|
|
"ledger_seq",
|
|
"proposing",
|
|
"ledger_hash",
|
|
"full_validation"
|
|
],
|
|
"config_flag": "trace_consensus",
|
|
"note": "follows-from consensus.accept. ledger_hash and full_validation are shared with peer.validation.receive (same keys, told apart by span name). Also carries validation_sign_time."
|
|
},
|
|
{
|
|
"name": "consensus.proposal.receive",
|
|
"category": "consensus",
|
|
"parent": null,
|
|
"required_attributes": [],
|
|
"config_flag": "trace_consensus",
|
|
"note": "Context-propagated from the sending peer. No required local attributes."
|
|
},
|
|
{
|
|
"name": "consensus.validation.receive",
|
|
"category": "consensus",
|
|
"parent": null,
|
|
"required_attributes": [],
|
|
"config_flag": "trace_consensus",
|
|
"note": "Context-propagated from the sending peer. No required local attributes."
|
|
},
|
|
{
|
|
"name": "consensus.mode_change",
|
|
"category": "consensus",
|
|
"parent": "consensus.round",
|
|
"required_attributes": ["mode_old", "mode_new"],
|
|
"config_flag": "trace_consensus",
|
|
"optional": true,
|
|
"note": "childSpan of roundSpanContext_ (RCLConsensus.cpp:1101), so the parent is consensus.round. Only fires on an operating-mode transition; a steady cluster rarely changes mode after warmup. A mode change outside a round leaves roundSpanContext_ invalid, which yields a null (no-op) guard rather than a root span."
|
|
},
|
|
{
|
|
"name": "ledger.build",
|
|
"category": "ledger",
|
|
"parent": null,
|
|
"required_attributes": [
|
|
"ledger_seq",
|
|
"close_time",
|
|
"close_time_correct",
|
|
"close_resolution_ms"
|
|
],
|
|
"config_flag": "trace_ledger",
|
|
"note": "tx_count/tx_failed live on the child tx.apply span, not here."
|
|
},
|
|
{
|
|
"name": "ledger.validate",
|
|
"category": "ledger",
|
|
"parent": null,
|
|
"required_attributes": ["ledger_seq", "validations"],
|
|
"config_flag": "trace_ledger"
|
|
},
|
|
{
|
|
"name": "ledger.store",
|
|
"category": "ledger",
|
|
"parent": null,
|
|
"required_attributes": ["ledger_seq"],
|
|
"config_flag": "trace_ledger"
|
|
},
|
|
{
|
|
"name": "ledger.acquire",
|
|
"category": "ledger",
|
|
"parent": null,
|
|
"required_attributes": ["ledger_seq", "acquire_reason"],
|
|
"config_flag": "trace_ledger",
|
|
"optional": true,
|
|
"note": "Only fires when a node must fetch a missing ledger (InboundLedger). A healthy local cluster rarely back-fills history. ledger_seq and acquire_reason are set unconditionally in init() (InboundLedger.cpp:116, :131) and are therefore required. outcome, timeouts and peer_count are set only on the done() path (InboundLedger.cpp:508-515); init() can satisfy the acquisition from the local store or bail on failed_/!complete_ and return without reaching done(), so those three cannot be required."
|
|
},
|
|
{
|
|
"name": "peer.proposal.receive",
|
|
"category": "peer",
|
|
"parent": null,
|
|
"required_attributes": ["peer_id"],
|
|
"config_flag": "trace_peer",
|
|
"note": "peer_id is set immediately after the freshRoot (PeerImp.cpp:1925) and is the only unconditional attribute. proposal_trusted is set at PeerImp.cpp:1953, after several early returns (stale/duplicate/self-originated proposal checks), so a single rejected proposal in the sampled trace would fail the check — it is therefore not required."
|
|
},
|
|
{
|
|
"name": "peer.validation.receive",
|
|
"category": "peer",
|
|
"parent": null,
|
|
"required_attributes": ["peer_id", "ledger_hash", "full_validation"],
|
|
"config_flag": "trace_peer",
|
|
"note": "ledger_hash and full_validation are shared with consensus.validation.send (same keys, told apart by span name). Both are set at PeerImp.cpp:2573-2574, BEFORE the isCurrent() gate, so only a too-small or unparseable validation skips them — they stay required (and validate_telemetry.py's PARITY_SPAN_ATTRS already asserts them independently). validation_trusted is set at PeerImp.cpp:2591, after the isCurrent() early return at :2576-2584, so a single not-current validation in the sampled trace would fail the check — it is therefore not required."
|
|
},
|
|
{
|
|
"name": "pathfind.request",
|
|
"category": "pathfind",
|
|
"parent": "rpc.command.*",
|
|
"required_attributes": [
|
|
"pathfind_source_account",
|
|
"pathfind_dest_account"
|
|
],
|
|
"config_flag": "trace_rpc",
|
|
"optional": true,
|
|
"note": "Fires on ripple_path_find / path_find RPC. Optional because the harness issues neither: the ripple_path_find weight was removed from rpc_load_generator.py's DEFAULT_WEIGHTS on 2026-08-25, and no workload-profiles.json phase names a path-finding command in a weights override, so no such RPC reaches a node at all. Created as an ambient (scoped) child inside the RPC command handler (RipplePathFind.cpp:35-36, PathFind.cpp:26-27), so its parent is the enclosing rpc.command.* span — RipplePathFind.cpp:30 states this explicitly. Note the span does NOT need pathfinding to be ENABLED, only the RPC to be issued: the ScopedSpanGuard is constructed at RipplePathFind.cpp:35, above the 'if (pathSearchMax == 0) return rpcError(RpcNotSupported)' guard at :48-49, so even a refused call opens and closes it. That positional accident is why the load alone used to satisfy this entry while pathfind.compute and pathfind.discover below stayed unasserted — and why those refusals were also manufacturing a steady ~3% STATUS_CODE_ERROR floor in span_calls_total, which removing the load has now cleared. Restoring the load alone would make this span required again and bring that error floor back; covering the whole family needs pathfinding actually enabled (a [path_search_max] override in run-full-validation.sh — see the pathfind.compute entry below). The workload README section 'Pathfinding is not exercised' carries the full restore recipe."
|
|
},
|
|
{
|
|
"name": "pathfind.compute",
|
|
"category": "pathfind",
|
|
"parent": "pathfind.request",
|
|
"required_attributes": ["pathfind_fast"],
|
|
"config_flag": "trace_rpc",
|
|
"optional": true,
|
|
"note": "Created by PathRequest::doUpdate (PathRequest.cpp:749-750), which the harness never reaches: pathfinding is disabled on every harness node, so no PathRequest is ever constructed. Config.cpp:725-726 sets pathSearchMax to 0 when a [validation_seed] or [validator_token] section is present, run-full-validation.sh writes [validation_seed] for every node (:308) and has no [path_search*] override, and doRipplePathFind then returns rpcNOT_SUPPORTED at RipplePathFind.cpp:48-49 — above the request-construction branches and below the pathfind.request span guard at :35. Liquidity is not the reason and never was: the call is refused before any path search is attempted, so the outcome does not depend on what the ledger holds. Since 2026-08-25 there is a second, independent reason: the harness sends no path-finding RPC at all, the ripple_path_find weight having been removed from rpc_load_generator.py. Enabling this span therefore needs BOTH a [path_search_max] override (or a non-validator node) in run-full-validation.sh AND the load restored — see the workload README section 'Pathfinding is not exercised'."
|
|
},
|
|
{
|
|
"name": "pathfind.discover",
|
|
"category": "pathfind",
|
|
"parent": "pathfind.compute",
|
|
"required_attributes": ["pathfind_search_level", "pathfind_num_paths"],
|
|
"config_flag": "trace_rpc",
|
|
"optional": true,
|
|
"note": "Graph exploration; only fires under pathfind.compute, which the harness never reaches because pathfinding is disabled on every node (see the pathfind.compute entry above for the config chain). Liquidity is not the reason: the RPC is refused before any search runs."
|
|
},
|
|
{
|
|
"name": "pathfind.update_all",
|
|
"category": "pathfind",
|
|
"parent": null,
|
|
"required_attributes": ["pathfind_ledger_index", "pathfind_num_requests"],
|
|
"config_flag": "trace_rpc",
|
|
"optional": true,
|
|
"note": "Async recomputation at ledger close. PathRequestManager::updateAll emits the span only when requests_ is non-empty (PathRequestManager.cpp:88-95), so it needs a live path_find subscription. On the harness requests_ can never become non-empty at all: the only two insertPathRequest call sites are makePathRequest (:268) and makeLegacyPathRequest (:296), and both handlers return rpcNOT_SUPPORTED first because pathfinding is disabled on every node (PathFind.cpp:39, RipplePathFind.cpp:48; see the pathfind.compute entry above for the config chain)."
|
|
},
|
|
{
|
|
"name": "grpc.*",
|
|
"category": "grpc",
|
|
"parent": null,
|
|
"required_attributes": ["method", "grpc_role", "grpc_status"],
|
|
"config_flag": "trace_rpc",
|
|
"optional": true,
|
|
"note": "Wildcard — grpc.<MethodName>. The harness has no gRPC client, so these do not fire. Tracked for completeness."
|
|
}
|
|
],
|
|
"parent_child_relationships": [
|
|
{
|
|
"parent": "rpc.ws_message",
|
|
"child": "rpc.process",
|
|
"description": "WebSocket message contains processing span",
|
|
"skip": true,
|
|
"skip_reason": "This relationship does not exist in the code: rpc.process is created only in ServerHandler::processRequest() (ServerHandler.cpp:705), reached only from processSession(Session, coro) (ServerHandler.cpp:646) — the HTTP/JSON-RPC path. The WebSocket path (processSession(WSSession, coro, jv), ServerHandler.cpp:467) never calls processRequest, so rpc.process is never emitted at all under the WebSocket-only harness. The earlier diagnosis (cross-thread context loss needing a C++ fix) was wrong: rpc.ws_message is a deliberate freshRoot (ServerHandler.cpp:473-474) so each WS message is its own trace rather than nesting under a span leaked on a reused coroutine worker. Nothing to fix."
|
|
},
|
|
{
|
|
"parent": "rpc.ws_message",
|
|
"child": "rpc.command.*",
|
|
"description": "WebSocket message contains the per-command span — the real relationship on the harness WS path (rpc::doCommand at RPCHandler.cpp:271 creates an ambient child of the rpc.ws_message scope inside the same coroutine)",
|
|
"skip": true,
|
|
"skip_reason": "Code-verified real, but not assertable by the current validator. _validate_parent_child() collapses the wildcard to the single literal name via child_name.replace(\"*\", \"server_info\") and samples only the 3 most recent parent traces. Each rpc.ws_message trace carries exactly one command, and server_info is 25/100 of rpc_load_generator.py's DEFAULT_WEIGHTS, so roughly 42% of healthy runs would sample three non-server_info traces and fail. Asserting this needs the validator to accept a wildcard child as a prefix match (or to raise the trace sample size); until then the relationship is documented, not enforced."
|
|
},
|
|
{
|
|
"parent": "rpc.process",
|
|
"child": "rpc.command.*",
|
|
"description": "Processing span contains per-command span (HTTP/JSON-RPC path only)",
|
|
"skip": true,
|
|
"skip_reason": "Real relationship, skipped for a VALIDATOR limitation, not for absence: _validate_parent_child() resolves the wildcard child to one literal probe name, so it cannot assert a family whose members vary per request. Corrected 2026-08-26 -- the earlier reason claimed there are no rpc.process traces to check because that span exists only on the HTTP/JSON-RPC path while the load generator is WebSocket-only. The premise is right and the conclusion is wrong: rpc.process DOES appear, 5 traces on a normal run, because run-full-validation.sh polls each node's HTTP port with curl for readiness and validated-ledger progress (:449, :502). Those curl calls are what produce rpc.http_request and its rpc.process child, entirely independently of the load generator. Anyone acting on the old reason would have gone looking for a way to make the harness speak HTTP, which it already does. That same reason also said the WS-path equivalent is 'asserted above instead'; it is not, rpc.ws_message -> rpc.command.* is skipped for this identical wildcard limitation. To assert either, teach _validate_parent_child to accept a wildcard child by matching any span whose name has the declared prefix."
|
|
},
|
|
{
|
|
"parent": "ledger.build",
|
|
"child": "tx.apply",
|
|
"description": "Ledger build contains transaction application"
|
|
},
|
|
{
|
|
"parent": "consensus.round",
|
|
"child": "consensus.accept",
|
|
"description": "Consensus round contains the accept sub-span"
|
|
},
|
|
{
|
|
"parent": "consensus.accept",
|
|
"child": "consensus.accept.apply",
|
|
"description": "Accept contains the ledger-apply sub-span"
|
|
},
|
|
{
|
|
"parent": "pathfind.request",
|
|
"child": "pathfind.compute",
|
|
"description": "Pathfind request contains the compute sub-span",
|
|
"skip": true,
|
|
"skip_reason": "Real relationship (pathfind.compute is created inside PathRequest::doUpdate at PathRequest.cpp:749-750, under the pathfind.request scope), but the child never exists on the harness because pathfinding is disabled on every node: Config.cpp:725-726 zeroes pathSearchMax whenever a [validation_seed] or [validator_token] section is present, run-full-validation.sh writes [validation_seed] for all five nodes (:308) with no [path_search*] override, and doRipplePathFind returns rpcNOT_SUPPORTED at RipplePathFind.cpp:48-49 before constructing a PathRequest. The parent used to appear anyway, because its ScopedSpanGuard is created at RipplePathFind.cpp:35, above that guard; since 2026-08-25 not even the parent appears, the ripple_path_find load having been removed from rpc_load_generator.py, so both ends of this relationship are now absent. Liquidity has nothing to do with it — the earlier 'no liquidity, returns before computing' reason was wrong, because no path search is attempted at all. Asserting this relationship needs the load restored AND a [path_search_max] override (or a non-validator node) in run-full-validation.sh."
|
|
},
|
|
|
|
{
|
|
"parent": "rpc.http_request",
|
|
"child": "rpc.process",
|
|
"description": "The HTTP request span contains the request-processing span. ServerHandler::processRequest() opens rpc.process as a scoped child and its own comment names rpc.http_request as the parent; processRequest has exactly one caller. Both ends emit on every run: the harness polls each node's HTTP port with curl, so this is the one rpc hierarchy that is assertable without a wildcard."
|
|
},
|
|
{
|
|
"parent": "tx.process",
|
|
"child": "txq.enqueue",
|
|
"description": "Transaction processing contains the queue-admission span."
|
|
},
|
|
{
|
|
"parent": "txq.enqueue",
|
|
"child": "txq.apply_direct",
|
|
"description": "Queue admission contains the direct-apply path taken when the transaction is applied straight to the open ledger instead of being queued."
|
|
},
|
|
{
|
|
"parent": "txq.accept",
|
|
"child": "txq.accept_tx",
|
|
"description": "The queue's accept pass contains the per-transaction accept span.",
|
|
"skip": true,
|
|
"skip_reason": "Real relationship, but not assertable by this check as written -- a conditional child plus newest-N sampling. Asserted in c531ac569b and it failed on run 32990348089 with 'txq.accept_tx not found in txq.accept traces'; skipped rather than left red. Both ends DO emit, 5 traces each on that run, so this is not a missing span. The parent is created once per accept pass, every ledger close (TxQ.cpp:1499). The child is created inside the loop over queued transactions and behind `if (feeLevelPaid >= requiredFeeLevel)` (TxQ.cpp:1530), so it exists only for a close where the queue actually held a fee-clearing transaction. _validate_parent_child searches the parent with limit=3 (validate_telemetry.py:803), and queue pressure comes from workload phase 5 of 7, txq-burst, after which mixed-peak (60s) and cooldown (30s) run -- so the three newest txq.accept traces are from quiet closes with an empty queue and no child. Same shape as the rpc.command.* skips: the check samples the newest traces of the parent, which is wrong whenever the child is conditional on load that has since stopped. Two ways to un-skip, in preference order: make the search prefer parent traces that contain the child (a TraceQL child filter rather than newest-N), or move txq-burst to the final workload phase so the newest closes are the loaded ones. Raising the limit alone only shifts the odds and would make this flaky rather than fixed."
|
|
},
|
|
{
|
|
"parent": "consensus.round",
|
|
"child": "consensus.phase.open",
|
|
"description": "A consensus round contains its open phase."
|
|
},
|
|
{
|
|
"parent": "consensus.round",
|
|
"child": "consensus.establish",
|
|
"description": "A consensus round contains its establish phase."
|
|
},
|
|
{
|
|
"parent": "consensus.round",
|
|
"child": "consensus.proposal.send",
|
|
"description": "A consensus round contains the proposals this node sends during it."
|
|
},
|
|
{
|
|
"parent": "consensus.round",
|
|
"child": "consensus.ledger_close",
|
|
"description": "A consensus round contains the ledger close it performs."
|
|
},
|
|
{
|
|
"parent": "consensus.round",
|
|
"child": "consensus.mode_change",
|
|
"description": "A consensus round contains any mode transition that happens inside it."
|
|
},
|
|
{
|
|
"parent": "consensus.establish",
|
|
"child": "consensus.update_positions",
|
|
"description": "The establish phase contains each position-update pass."
|
|
},
|
|
{
|
|
"parent": "consensus.establish",
|
|
"child": "consensus.check",
|
|
"description": "The establish phase contains each consensus-reached check."
|
|
}
|
|
],
|
|
"total_span_types": 41,
|
|
"total_unique_attributes": 62
|
|
}
|