ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-09-27 07:26:51 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
efb4cc2edd94fb3976efeefbc405ae7ed498d228
rippled/docker/telemetry/workload/expected_spans.json

784 lines
61 KiB
JSON
Raw Blame History

{
"description": "Expected span inventory for xrpld telemetry validation. Attribute keys follow the 2026-05-13 span-attr naming redesign (bare/underscore form; dotted xrpl.* reserved for resource attributes). Sourced from the *SpanNames.h headers and verified against the emitting call sites. Spans marked \"optional\": true are conditional — they only fire under traffic the harness may not produce (e.g. gRPC client, path-finding RPC, missing-ledger fetch, mode transitions) and are not failed when absent. \"allowed_parents\" is asserted, by validate_span_parents in validate_telemetry.py: every emitted instance of the span must be parented to one of the names listed, so a span declared a root that is emitted as somebody's child now fails a check instead of passing silently. \"ROOT\" means \"no parent from this node\" — a genuine root, an explicit freshRoot, or a span whose parent was created on a different node, because a cross-node parent is the design for the receive spans and is never a violation. A span reachable by two call paths lists every parent either path can give it, and a listed parent may itself be a glob (pathfind.request is declared under rpc.command.*), matched the same way the span names in this file are. Each list is derived from the span's creation factory plus every call path that reaches it — README.md, section \"expected_spans.json Format\", has the rule — so a span note that lists fewer parents than the code can produce is a bug in this file, not a reason to widen the gate at the call site. Three spans list ROOT even though their factory inherits the ambient scope (ledger.acquire, pathfind.update_all, grpc.*): they are created at a boundary or outlive the creating scope, so an observed parent there is an ambient leak to fix in the C++, not an edge to add here. required_attributes lists only attributes set on EVERY code path that creates the span — attributes set after an early return are described in the span's note instead, because _validate_span_attributes_otlp samples a single trace and would fail on a legitimate short-circuit path. total_unique_attributes is the size of the union of all required_attributes; total_span_types is len(spans). Span EVENTS (consensus.round phase.*/outcome.*, consensus.update_positions dispute.resolve, consensus.accept.apply tx.included) are NOT represented: validate_telemetry.py reads only span name, attributes and timestamps from Tempo, so an \"events\" key would be silently ignored. They are documented in the relevant span notes until the validator gains event support.",
"spans": [
{
"name": "rpc.ws_message",
"category": "rpc",
"allowed_parents": ["ROOT"],
"required_attributes": ["command"],
"config_flag": "trace_rpc",
"note": "WebSocket RPC root span. The load generator uses WS, so this is the RPC entry span (not rpc.http_request, which needs an HTTP/JSON-RPC client)."
},
{
"name": "rpc.ws_upgrade",
"category": "rpc",
"allowed_parents": ["ROOT"],
"required_attributes": [],
"config_flag": "trace_rpc",
"optional": true,
"note": "WebSocket handshake span (ServerHandler::onHandoff, ServerHandler.cpp:272-273). A freshRoot with no attributes — only setOk() on success or recordException() on an upgrade failure. Fires once per WS connection, so the load generator produces only a handful of these at connect time; by the time validation runs after the propagation wait they may fall outside the Tempo search window. Optional for that reason, not because the code path is conditional."
},
{
"name": "rpc.process",
"category": "rpc",
"allowed_parents": ["rpc.http_request"],
"required_attributes": [],
"config_flag": "trace_rpc",
"optional": true,
"note": "HTTP-only. Created solely in ServerHandler::processRequest() (ServerHandler.cpp:718), which is reached only from processSession(Session, coro) (ServerHandler.cpp:646) — the HTTP/JSON-RPC path that roots rpc.http_request at ServerHandler.cpp:640-641. The WebSocket path (processSession(WSSession, coro, jv), ServerHandler.cpp:467) never calls processRequest, so this span never appears under a WebSocket request. It does still appear under this harness, 5 traces on a normal run, because run-full-validation.sh polls each node over HTTP with curl (:449, :502) and those requests take the HTTP path. Note that the harness does speak HTTP: a reader concluding this span is unreachable here would go looking for a way to add HTTP traffic that already exists."
},
{
"name": "rpc.command.*",
"category": "rpc",
"allowed_parents": ["rpc.ws_message", "rpc.process", "ROOT"],
"required_attributes": ["command", "version", "rpc_role", "rpc_status"],
"config_flag": "trace_rpc",
"note": "Wildcard — matches rpc.command.server_info, rpc.command.ledger, etc. Created as a plain ScopedSpanGuard in rpc::callMethod (RPCHandler.cpp:204-205) and, on the fillHandler-rejection path, in rpc::doCommand (:340-341), so its parent is whichever transport span is ambient on the thread. Three call paths, all three exercised by this harness: rpc::doCommand from the WebSocket handler inside the rpc.ws_message scope (ServerHandler.cpp:589 under the freshRoot at :495); from ServerHandler::processRequest inside the rpc.process scope (:1037 under the guard at :734); and from ApplicationImp::setup's [rpc_startup] loop (Application.cpp:1786) on the main thread with no ambient scope, which makes it a ROOT. The third is why ROOT is listed: startTelemetry() runs at Application.cpp:1511, well before that loop, so the startup command really does emit a rooted rpc.command span — run-full-validation.sh writes a [rpc_startup] log_level command for every node, so there are five of them per run."
},
{
"name": "rpc.http_request",
"category": "rpc",
"allowed_parents": ["ROOT"],
"required_attributes": ["request_payload_size"],
"config_flag": "trace_rpc",
"optional": true,
"note": "HTTP/JSON-RPC root span. It DOES fire under this harness, 5 traces on a normal run -- one per node -- even though the load generator is WebSocket-only, because run-full-validation.sh polls each node's HTTP port with curl for readiness and validated-ledger progress (:449, :502). Kept optional rather than required because those polls are harness scaffolding rather than workload: a future change to how the script waits for a node could remove them without anything being wrong with the node."
},
{
"name": "tx.process",
"category": "transaction",
"allowed_parents": ["ROOT"],
"required_attributes": ["tx_hash", "local", "path"],
"config_flag": "trace_transactions",
"note": "Also carries tx_type, fee, sequence, and one attribute per account-typed top-level field of the transaction: tx_account always, plus tx_destination, tx_owner, tx_issuer and so on when the transaction has them (keys in TxAccountSpanNames.h, raw r-addresses). These are set only when the STTx parses, so like tx_type they are not required attributes."
},
{
"name": "tx.receive",
"category": "transaction",
"allowed_parents": ["ROOT"],
"required_attributes": ["tx_hash", "peer_id"],
"config_flag": "trace_transactions",
"note": "Cross-node span: parent context propagated from the sender's tx.process via protobuf. Also carries tx_type and peer_version. The span is created only after the node decides to process the transaction, so a relayed duplicate produces no span at all; how many were dropped is the transactions_duplicate traffic category. tx_status is set only on the paths that drop a transaction after that point, so it is not a required attribute."
},
{
"name": "tx.apply",
"category": "transaction",
"allowed_parents": ["ledger.build"],
"required_attributes": ["tx_count", "tx_failed"],
"config_flag": "trace_transactions",
"note": "Apply-step span inside BuildLedger. Sets its own ledger_seq next to tx_count and tx_failed (BuildLedger.cpp:197-199), so do not re-add the earlier claim that ledger_seq lives on the parent ledger.build span — it does not, and ledger.build sets it independently."
},
{
"name": "tx.preflight",
"category": "transaction",
"allowed_parents": ["ROOT"],
"required_attributes": ["stage", "tx_type", "ter_result"],
"config_flag": "trace_transactions",
"note": "Apply-pipeline stage span (stage=preflight). Shares a deterministic trace_id (txID[0:16]) with tx.preclaim/tx.transactor."
},
{
"name": "tx.preclaim",
"category": "transaction",
"allowed_parents": ["ROOT"],
"required_attributes": ["stage", "tx_type", "ter_result"],
"config_flag": "trace_transactions",
"note": "Apply-pipeline stage span (stage=preclaim)."
},
{
"name": "tx.transactor",
"category": "transaction",
"allowed_parents": ["ROOT"],
"required_attributes": ["stage", "tx_type"],
"config_flag": "trace_transactions",
"note": "Apply-pipeline stage span (stage=apply). Also carries applied."
},
{
"name": "txq.enqueue",
"category": "transaction",
"allowed_parents": [
"tx.process",
"consensus.accept.apply",
"rpc.command.*",
"ROOT"
],
"required_attributes": ["tx_hash", "tx_type", "txq_status"],
"config_flag": "trace_transactions",
"optional": true,
"note": "Only fires when a tx is queued (fee below open-ledger level). Requires fee escalation — driven by the txq-burst workload phase. tx_hash/tx_type/txq_status are set on every code path; fee_level_paid/required_fee_level are conditional (TxQ.cpp ~895-898, after the rejected and applied_direct early exits), so they are NOT guaranteed on every txq.enqueue span and cannot be required. Four parents, because TxQ::apply (TxQ.cpp:770) takes an OPTIONAL parent context (:790-795): with a context it is an explicit childSpan, without one it is a plain ScopedSpanGuard that inherits whatever is ambient. tx.process comes from the submission path, which passes the captured context (NetworkOPs.cpp:1803-1810). consensus.accept.apply comes from the open-ledger rebuild, which passes none: OpenLedger.cpp:121 replays local transactions, and OpenLedger::accept is called from inside doAccept (RCLConsensus.cpp:871) where the consensus span is ambient. ROOT comes from that same rebuild reached by the switchLastClosedLedger jump path instead (NetworkOPs.cpp:2313), which has no ambient scope, and from the submission path when the tx.process context is invalid. rpc.command.* comes from the simulate RPC (Simulate.cpp:249), which passes no context and runs inside the command span's scope; the harness never issues simulate, so that parent is unreachable here but is still lawful."
},
{
"name": "txq.apply_direct",
"category": "transaction",
"allowed_parents": ["txq.enqueue"],
"required_attributes": [],
"config_flag": "trace_transactions",
"optional": true,
"note": "Child of txq.enqueue when the tx applies directly without queueing."
},
{
"name": "txq.batch_clear",
"category": "transaction",
"allowed_parents": ["txq.enqueue"],
"required_attributes": [],
"config_flag": "trace_transactions",
"optional": true,
"note": "TxQ::tryClearAccountQueueUpThruTx (TxQ.cpp:571-572). num_cleared is NOT required: it is written only once the batch has actually cleared, inside the `if (txResult.applied)` branch at TxQ.cpp:646. Three earlier returns leave the span without it — a fee-level overflow (:589-590), a batch that did not pay enough (:598-599), and a queued transaction that failed to apply (:632-635) — as does the final doApply failing, which falls past the branch. The span is opened before any of those decisions, so a required num_cleared would fail the check on a legitimate fall-back-to-normal-processing path rather than on a defect."
},
{
"name": "txq.accept",
"category": "transaction",
"allowed_parents": ["consensus.accept.apply", "ROOT"],
"required_attributes": ["queue_size", "ledger_changed"],
"config_flag": "trace_transactions",
"optional": true,
"note": "Ledger-close accept loop (TxQ::accept, TxQ.cpp:1499). Only meaningful when the queue is non-empty. Two call paths, and they give it two different parents. On the consensus path (RCLConsensus.cpp:823, inside doAccept) consensus.accept.apply is the ambient scope for the whole of doAccept, so this span is its child. On the switchLastClosedLedger jump path (NetworkOPs.cpp:2150) no consensus span is open on that thread, so the span is a root. The span is a ScopedSpanGuard and adopts whatever OTel context is ambient, which is what makes the parent a property of the caller rather than of this span."
},
{
"name": "txq.accept_tx",
"category": "transaction",
"allowed_parents": ["txq.accept"],
"required_attributes": [
"tx_hash",
"ter_code",
"retries_remaining",
"txq_status"
],
"config_flag": "trace_transactions",
"optional": true
},
{
"name": "txq.cleanup",
"category": "transaction",
"allowed_parents": ["consensus.accept.apply", "ROOT"],
"required_attributes": ["ledger_seq", "expired_count"],
"config_flag": "trace_transactions",
"optional": true,
"note": "TxQ::processClosedLedger (TxQ.cpp:1403). The same two call paths as txq.accept, with the same split: the consensus path (RCLConsensus.cpp:950) runs inside the consensus.accept.apply scope and nests under it, while the switchLastClosedLedger jump path (NetworkOPs.cpp:2121) has no consensus span open and leaves it a root."
},
{
"name": "consensus.round",
"category": "consensus",
"allowed_parents": ["ROOT"],
"required_attributes": [
"consensus_ledger_id",
"ledger_seq",
"consensus_mode",
"consensus_round_id",
"consensus_phase"
],
"config_flag": "trace_consensus",
"note": "Root consensus span created per round. Also carries trace_strategy, previous_ledger_seq, previous_proposers, previous_round_time_ms. Emits seven span EVENTS that this manifest cannot assert: phase.open, phase.recovery, phase.establish, phase.accepted, outcome.yes, outcome.moved_on, outcome.expired (declared ConsensusSpanNames.h:265-277; emitted RCLConsensus.cpp:1344 and via onPhaseEvent/onOutcomeEvent from Consensus.h:764, 793, 1047, 1517-1525, 1530, 1566), plus a `view.change` event emitted from RCLConsensus::Adaptor::getPrevLedger when the network's preferred ledger differs from this node's (carries prev_ledger_prefix and net_ledger_prefix). validate_telemetry.py reads only span name, attributes and start/end timestamps from the Tempo OTLP payload — it has no event assertion support — so adding an \"events\" key here would be silently ignored. Recorded as a note instead; asserting events needs validator support first."
},
{
"name": "consensus.phase.open",
"category": "consensus",
"allowed_parents": ["consensus.round"],
"required_attributes": [],
"config_flag": "trace_consensus"
},
{
"name": "consensus.proposal.send",
"category": "consensus",
"allowed_parents": ["consensus.round"],
"required_attributes": ["consensus_round"],
"config_flag": "trace_consensus",
"note": "Also carries is_bow_out."
},
{
"name": "consensus.ledger_close",
"category": "consensus",
"allowed_parents": ["consensus.round"],
"required_attributes": ["ledger_seq", "consensus_mode"],
"config_flag": "trace_consensus",
"note": "Also carries tx_count_open, close_time_resolution_ms."
},
{
"name": "consensus.establish",
"category": "consensus",
"allowed_parents": ["consensus.round"],
"required_attributes": [
"converge_percent",
"establish_count",
"proposers",
"disputes_count"
],
"config_flag": "trace_consensus"
},
{
"name": "consensus.update_positions",
"category": "consensus",
"allowed_parents": ["consensus.establish"],
"required_attributes": [
"converge_percent",
"proposers",
"disputes_count"
],
"config_flag": "trace_consensus",
"note": "childSpan of establishSpanContext_ (Consensus.h:1628), so the parent is consensus.establish — not consensus.round. Also emits a dispute.resolve span EVENT per resolved dispute (Consensus.h:1697-1698), which validate_telemetry.py cannot assert (no event support)."
},
{
"name": "consensus.check",
"category": "consensus",
"allowed_parents": ["consensus.establish"],
"required_attributes": [
"agree_count",
"disagree_count",
"threshold_percent",
"consensus_result"
],
"config_flag": "trace_consensus",
"note": "childSpan of establishSpanContext_ (Consensus.h:1837), so the parent is consensus.establish — not consensus.round."
},
{
"name": "consensus.accept",
"category": "consensus",
"allowed_parents": ["consensus.round"],
"required_attributes": ["proposers", "round_time_ms", "quorum"],
"config_flag": "trace_consensus"
},
{
"name": "consensus.accept.apply",
"category": "consensus",
"allowed_parents": ["consensus.accept"],
"required_attributes": [
"ledger_seq",
"close_time_ripple_epoch_s",
"parent_close_time_ripple_epoch_s",
"close_time_self_ripple_epoch_s",
"close_time_vote_bins",
"resolution_direction"
],
"config_flag": "trace_consensus",
"note": "Also carries close_time_correct, close_resolution_ms, consensus_state, proposing, round_time_ms, tx_count. Emits a tx.included span EVENT per transaction in the accepted set (RCLConsensus.cpp:720, with a tx_id attribute), which validate_telemetry.py cannot assert (no event support). One parent: a scoped childSpan of the captured accept context (RCLConsensus.cpp doAccept). The context is valid whenever the accept span is live; when it is not, the factory returns a null guard and no span is emitted at all. Scoped, so the txq spans doAccept creates after it are its children."
},
{
"name": "consensus.validation.send",
"category": "consensus",
"allowed_parents": ["ROOT"],
"required_attributes": [
"ledger_seq",
"proposing",
"ledger_hash",
"full_validation"
],
"config_flag": "trace_consensus",
"note": "follows-from consensus.accept. ledger_hash and full_validation are shared with peer.validation.receive (same keys, told apart by span name). Also carries validation_sign_time."
},
{
"name": "consensus.proposal.receive",
"category": "consensus",
"allowed_parents": ["ROOT"],
"required_attributes": [],
"config_flag": "trace_consensus",
"note": "Context-propagated from the sending peer. No required local attributes."
},
{
"name": "consensus.validation.receive",
"category": "consensus",
"allowed_parents": ["ROOT"],
"required_attributes": [],
"config_flag": "trace_consensus",
"note": "Context-propagated from the sending peer. No required local attributes."
},
{
"name": "consensus.validation.accept",
"category": "consensus",
"allowed_parents": ["ROOT"],
"required_attributes": [
"ledger_hash",
"ledger_seq",
"validation_status",
"accept_gated",
"full_validation"
],
"config_flag": "trace_ledger",
"note": "Emitted once per TRUSTED validation as it reaches the ledger-acceptance gate (handleNewValidation), so its rate is bounded by the UNL size per ledger close. Required, not optional: every node in the harness cluster validates and every other node receives those validations, so the span fires continuously. config_flag is trace_ledger, not trace_consensus, because the span is created through the ledger-hash trace join (TraceCategory::Ledger) -- it belongs to the ledger's trace, not the round's. Its trace id is derived from the VALIDATED ledger hash, the same key ledger.validate and ledger.store use, which is what makes one slow ledger read as one connected trace across the three threads that produce those spans. validation_status is one of the 6 ValStatus values and accept_gated says whether the validation actually reached the gate; both are spanmetrics dimensions. See trace_join_groups below."
},
{
"name": "consensus.mode_change",
"category": "consensus",
"allowed_parents": ["consensus.round"],
"required_attributes": ["mode_old", "mode_new"],
"config_flag": "trace_consensus",
"optional": true,
"note": "childSpan of roundSpanContext_ (RCLConsensus.cpp:1101), so the parent is consensus.round. Only fires on an operating-mode transition; a steady cluster rarely changes mode after warmup. Note the handler itself is called at the start of EVERY round, with the before and after mode equal on almost all of them, so the guard on before != after is the whole reason this span is conditional rather than per-round — without it the span records no change and span.mode_change.records_a_real_change fails. A mode change outside a round leaves roundSpanContext_ invalid, which yields a null (no-op) guard rather than a root span."
},
{
"name": "ledger.build",
"category": "ledger",
"allowed_parents": ["consensus.accept.apply", "ROOT"],
"required_attributes": [
"ledger_seq",
"close_time_ripple_epoch_s",
"close_time_correct",
"close_resolution_ms"
],
"config_flag": "trace_ledger",
"note": "tx_count/tx_failed live on the child tx.apply span, not here. Two call paths give it two parents. The consensus path (RCLConsensus.cpp:983-985, inside doAccept) builds the ledger inside the consensus.accept.apply scope, so the span nests under it. The LedgerDeltaAcquire replay path (LedgerDeltaAcquire.cpp:208) rebuilds a historical ledger with no consensus span open on that thread, so the span is a root there."
},
{
"name": "ledger.validate",
"category": "ledger",
"allowed_parents": ["ROOT"],
"required_attributes": ["ledger_hash", "ledger_seq", "validations"],
"config_flag": "trace_ledger",
"note": "Created in LedgerMaster::checkAccept (LedgerMaster.cpp:1002-1003) with the ambient-inheriting span() factory, so its parent is whatever scope is open on the calling thread. The consensus path reaches it inside doAccept — consensusBuilt (RCLConsensus.cpp:803) calls checkAccept at LedgerMaster.cpp:1149, and the node's own validation takes the same route through handleNewValidation (RCLConsensus.cpp:1115) — so there it nests under the consensus span. ROOT covers the two paths with no ambient scope: an inbound peer validation, which PeerImp defers to a JtValidationT/JtValidationUt job (PeerImp.cpp:2751-2757) carrying only a never-activated SpanGuard handle, and the InboundLedger completion path (InboundLedger.cpp:168, :583). allowed_parents is ROOT alone on this branch: the span is a hashSpan (a true root on the ledger-hash trace), so it never inherits the consensus.accept.apply scope the upstream branches document."
},
{
"name": "ledger.store",
"category": "ledger",
"allowed_parents": ["ROOT"],
"required_attributes": ["ledger_hash", "ledger_seq"],
"config_flag": "trace_ledger",
"note": "Created in LedgerMaster::storeLedger (LedgerMaster.cpp:470) with the ambient-inheriting span() factory. The consensus path reaches it from RCLConsensus.cpp:1001, inside doAccept, so there it nests under the consensus span. ROOT covers every other caller, none of which has an ambient scope: InboundLedger.cpp:164 and :574, LedgerDeltaAcquire.cpp:243, and the genesis/startup stores at Application.cpp:2028 and :2037. allowed_parents is ROOT alone on this branch: the span is a hashSpan (a true root on the ledger-hash trace), so it never inherits the consensus.accept.apply scope the upstream branches document."
},
{
"name": "ledger.acquire",
"category": "ledger",
"allowed_parents": ["ROOT"],
"required_attributes": [
"ledger_seq",
"acquire_reason",
"timeouts",
"outcome",
"ledger_hash"
],
"config_flag": "trace_ledger",
"optional": true,
"note": "Only fires when a node must fetch a missing ledger (InboundLedger). A healthy local cluster rarely back-fills history. outcome is one of complete|failed|abandoned and is stamped on every exit path, including the sweep/shutdown path where the fetch never finished (abandoned). ledger_hash identifies the target ledger from the first moment, since a by-hash acquire starts with ledger_seq 0. peer_count is not required: the destructor path deliberately skips the peer lookup to avoid taking the Overlay lock under the InboundLedgers collection lock. allowed_parents is ROOT alone on this branch: the span is a hashSpan (a true root on the ledger-hash trace), so it never inherits the consensus.accept.apply scope the upstream branches document."
},
{
"name": "ledger.acquire.header",
"category": "ledger",
"allowed_parents": ["ledger.acquire"],
"required_attributes": ["ledger_hash", "outcome", "timed_out"],
"config_flag": "trace_ledger",
"optional": true,
"note": "Child of ledger.acquire, so it is optional for the same reason the parent is: it exists only while a node is fetching a missing ledger, and a healthy 5-node cluster that has agreed from genesis rarely back-fills. Covers the wait for the ledger header, which gates both tree phases -- until it arrives the account-state and transaction root hashes are unknown, so nothing else can be requested. missing_nodes is deliberately absent: a header is a single object, not a tree."
},
{
"name": "ledger.acquire.astree",
"category": "ledger",
"allowed_parents": ["ledger.acquire"],
"required_attributes": [
"ledger_hash",
"outcome",
"timed_out",
"missing_nodes"
],
"config_flag": "trace_ledger",
"optional": true,
"note": "Child of ledger.acquire (optional for the same reason). The account-state SHAMap phase, which is nearly all of the work in a real fresh sync -- the flat parent span could not separate it from the small transaction tree. outcome=timeout with a non-zero missing_nodes is the 'peers are not serving this tree' signature. missing_nodes is read from the count getMissingNodes() already produced during its sweep; no extra tree walk."
},
{
"name": "ledger.acquire.txtree",
"category": "ledger",
"allowed_parents": ["ledger.acquire"],
"required_attributes": [
"ledger_hash",
"outcome",
"timed_out",
"missing_nodes"
],
"config_flag": "trace_ledger",
"optional": true,
"note": "Child of ledger.acquire (optional for the same reason). The transaction SHAMap phase. Usually completes long before the astree phase, and that asymmetry is the point of splitting them: the parent span's duration is the state tree's, not this one's."
},
{
"name": "ledger.serve",
"category": "ledger",
"allowed_parents": ["ROOT"],
"required_attributes": [
"object_type",
"peer_id",
"served_nodes",
"outcome"
],
"config_flag": "trace_ledger",
"optional": true,
"note": "The supply side: this node answering a peer's TMGetLedger request on the JtLedgerReq worker. In practice every node in the harness cluster is listed in the others' [ips], so they exchange ledger and tx-set requests continuously throughout the run -- but optional even so, because the supply side cannot be required while every demand-side span is optional. A TMGetLedger request is constructed in exactly two places, InboundLedger.cpp and TransactionAcquire.cpp, and their ledger.acquire and txset.acquire spans are both optional; requiring this one would rest a mandatory check on an optional cause. Promote it together with them if a future harness step forces a real fetch. A fresh trace root -- the request arrives from the wire on a shared worker, so it must not inherit an unrelated span active there. object_type is header|tx|as|txset and outcome is complete|partial|refused, both derived by shared rules in LedgerSpanNames.h so the eight exits of processLedgerRequest cannot disagree. ledger_seq is present only once getLedger() succeeded, so it is not required."
},
{
"name": "txset.acquire",
"category": "ledger",
"allowed_parents": ["ROOT"],
"required_attributes": [
"txset_hash",
"outcome",
"timeouts",
"duration_ms",
"peer_count"
],
"config_flag": "trace_ledger",
"optional": true,
"note": "One attempt to fetch a transaction set a consensus proposal referenced (TransactionAcquire, whose only telemetry is this span). Optional because it only fires when a node does NOT already hold a proposed set: in the harness cluster every node sees the same relayed transactions and builds the same set locally, so InboundTransactions::getSet finds it in its map and never constructs a TransactionAcquire. It is the sibling of ledger.acquire -- same TimeoutCounter base, same trigger/onTimer/takeNodes shape -- and shares the trace_ledger flag so the two halves of a stuck sync cannot be enabled apart. outcome is complete|failed|timeout|abandoned, stamped on all three exits (done(); abandonAcquireSpan(), when InboundTransactions stops pursuing the fetch -- giveSet, the newRound sweep, stop, or ~InboundTransactionsImp destroying the container; and cancel(), when the TimeoutCounter base marks the task failed without reaching done()); the destructor only asserts one of them already ran. A fetch dropped in the window before init() runs emits NO span at all rather than an abandoned one, so the outcome set describes the spans that exist, not every TransactionAcquire ever constructed. The requesting round(s) are carried by repeated round.request EVENTS (current_ledger_hash + current_ledger_seq), not by a parent or a link: one round starts many fetches and one fetch is wanted by many rounds, so no round owns the span -- which is why there is deliberately no parent_child_relationships entry for it. Events are not asserted by this validator; they are read in Tempo."
},
{
"name": "peer.dial",
"category": "peer",
"allowed_parents": ["ROOT"],
"required_attributes": ["remote_endpoint"],
"config_flag": "trace_peer",
"note": "One outbound connect attempt (ConnectAttempt), a fresh trace root because a dial is the first thing a starting node does and there is nothing to parent it to. Required: run-full-validation.sh lists the other four nodes in each node's [ips], so every node dials and the span always fires. Telemetry is live in time to catch it -- ApplicationImp::setup() calls startTelemetry() before start() calls overlay_->start(). outcome carries the same six values as the overlay_connect_total counter (connected|tcp_fail|tls_fail|self_connection|upgrade_fail|timeout) and is set from the same reportOutcome() funnel, so span and counter cannot disagree. remote_endpoint is the span-only dimension the counter cannot carry, since one series per peer address would be unbounded cardinality. Among its attributes only remote_endpoint is required. outcome and duration_ms are set on every terminal path, both inside reportOutcome(), but NOT on the teardown path: an attempt destroyed during overlay shutdown, or one whose connect was aborted, ends its span in ~ConnectAttempt with neither, deliberately -- the destructor's own comment records that a span ending with no outcome is the honest record of a dial that never concluded. Because peer.dial is a freshRoot, each dial is its own trace holding exactly one instance of the span, and the validator inspects only the most recent trace; a single newly-aborted dial would therefore fail CI on healthy behaviour. That single-trace sampling of a one-instance span is what makes requiring them unsafe -- not any doubt about the terminal paths setting them."
},
{
"name": "peer.proposal.receive",
"category": "peer",
"allowed_parents": ["ROOT"],
"required_attributes": ["peer_id"],
"config_flag": "trace_peer",
"note": "peer_id is set immediately after the freshRoot (PeerImp.cpp:1925) and is the only unconditional attribute. proposal_trusted is set at PeerImp.cpp:1953, after several early returns (stale/duplicate/self-originated proposal checks), so a single rejected proposal in the sampled trace would fail the check — it is therefore not required."
},
{
"name": "peer.validation.receive",
"category": "peer",
"allowed_parents": ["ROOT"],
"required_attributes": ["peer_id", "ledger_hash", "full_validation"],
"config_flag": "trace_peer",
"note": "ledger_hash and full_validation are shared with consensus.validation.send (same keys, told apart by span name). Both are set at PeerImp.cpp:2573-2574, BEFORE the isCurrent() gate, so only a too-small or unparseable validation skips them — they stay required (and validate_telemetry.py's PARITY_SPAN_ATTRS already asserts them independently). validation_trusted is set at PeerImp.cpp:2591, after the isCurrent() early return at :2576-2584, so a single not-current validation in the sampled trace would fail the check — it is therefore not required."
},
{
"name": "pathfind.request",
"category": "pathfind",
"allowed_parents": ["rpc.command.*"],
"required_attributes": [
"pathfind_source_account",
"pathfind_dest_account"
],
"config_flag": "trace_rpc",
"optional": true,
"note": "Fires on ripple_path_find / path_find RPC. Optional because the harness issues neither: rpc_load_generator.py's DEFAULT_WEIGHTS carries no ripple_path_find entry, and no workload-profiles.json phase names a path-finding command in a weights override, so no such RPC reaches a node at all. Created as an ambient (scoped) child inside the RPC command handler (RipplePathFind.cpp:35-36, PathFind.cpp:26-27), so its parent is the enclosing rpc.command.* span — RipplePathFind.cpp:30 states this explicitly. Note the span does NOT need pathfinding to be ENABLED, only the RPC to be issued: the ScopedSpanGuard is constructed at RipplePathFind.cpp:35, above the 'if (pathSearchMax == 0) return rpcError(RpcNotSupported)' guard at :48-49, so even a refused call opens and closes it. That positional accident means the load alone would satisfy this entry while pathfind.compute and pathfind.discover below stayed unasserted, and it is why such refusals would also drive a steady ~3% STATUS_CODE_ERROR floor in span_calls_total. Adding the load alone would make this span required and introduce that error floor; covering the whole family needs pathfinding actually enabled (a [path_search_max] override in run-full-validation.sh — see the pathfind.compute entry below). The workload README section 'Pathfinding is not exercised' carries the full recipe for enabling it."
},
{
"name": "pathfind.compute",
"category": "pathfind",
"allowed_parents": ["pathfind.request", "pathfind.update_all"],
"required_attributes": ["pathfind_fast"],
"config_flag": "trace_rpc",
"optional": true,
"note": "Created by PathRequest::doUpdate (PathRequest.cpp:749-750), which the harness never reaches: pathfinding is disabled on every harness node, so no PathRequest is ever constructed. Config.cpp:725-726 sets pathSearchMax to 0 when a [validation_seed] or [validator_token] section is present, run-full-validation.sh writes [validation_seed] for every node and has no [path_search*] override, and doRipplePathFind then returns RpcNotSupported at RipplePathFind.cpp:59-60 — above the request-construction branches and below the pathfind.request span guard at :35. Liquidity is not the reason and never was: the call is refused before any path search is attempted, so the outcome does not depend on what the ledger holds. There is a second, independent reason: the harness sends no path-finding RPC at all, since rpc_load_generator.py carries no ripple_path_find weight. Enabling this span therefore needs BOTH a [path_search_max] override (or a non-validator node) in run-full-validation.sh AND the load restored — see the workload README section 'Pathfinding is not exercised'. Two parents: doUpdate's plain ScopedSpanGuard (PathRequest.cpp:788-789) inherits the ambient scope, and doUpdate has callers under both of them — under pathfind.request via PathRequest.cpp:272 and PathRequestManager.cpp:334, and under pathfind.update_all via PathRequestManager.cpp:161 and :176, where updateAll's own scoped guard is open."
},
{
"name": "pathfind.discover",
"category": "pathfind",
"allowed_parents": ["pathfind.compute"],
"required_attributes": ["pathfind_search_level", "pathfind_num_paths"],
"config_flag": "trace_rpc",
"optional": true,
"note": "Graph exploration; only fires under pathfind.compute, which the harness never reaches because pathfinding is disabled on every node (see the pathfind.compute entry above for the config chain). Liquidity is not the reason: the RPC is refused before any search runs."
},
{
"name": "pathfind.update_all",
"category": "pathfind",
"allowed_parents": ["ROOT"],
"required_attributes": ["pathfind_ledger_index", "pathfind_num_requests"],
"config_flag": "trace_rpc",
"optional": true,
"note": "Async recomputation at ledger close. PathRequestManager::updateAll emits the span only when requests_ is non-empty (PathRequestManager.cpp:88-95), so it needs a live path_find subscription. On the harness requests_ can never become non-empty at all: the only two insertPathRequest call sites are makePathRequest (:268) and makeLegacyPathRequest (:296), and both handlers return RpcNotSupported first because pathfinding is disabled on every node (PathFind.cpp:50-51, RipplePathFind.cpp:59; see the pathfind.compute entry above for the config chain). ROOT despite the ambient-inheriting ScopedSpanGuard, because the one caller is LedgerMaster's advance thread (LedgerMaster.cpp:1491) with no scope open. An observed parent is an ambient leak into that thread, not a lawful edge."
},
{
"name": "grpc.*",
"category": "grpc",
"allowed_parents": ["ROOT"],
"required_attributes": ["method", "grpc_role", "grpc_status"],
"config_flag": "trace_rpc",
"optional": true,
"note": "Wildcard — grpc.<MethodName>. The harness has no gRPC client, so these do not fire. Tracked for completeness. ROOT despite the ambient-inheriting span() factory (GRPCServer.cpp:174), because the span is created at the inbound boundary on a gRPC handler thread that runs no other instrumented work. An observed parent is an ambient leak onto that thread, not a lawful edge."
},
{
"name": "nodestore.rotate",
"category": "ledger",
"allowed_parents": ["ROOT"],
"required_attributes": ["ledger_seq", "last_rotated", "outcome"],
"config_flag": "trace_ledger",
"optional": true,
"note": "One per online-delete rotation (SHAMapStoreImp::run). Optional: the harness cluster has no online_delete interval it can reach in a run, so no node rotates. A fresh trace root on the SHAMapStore thread; every nodestore.rotate.* span is its child through the thread's ambient scope. outcome is complete|expired|stopping|missing_node, stamped by a RotationOutcome finalizer on every exit of the rotation block. The root closes when the rotation ends (13-17 min on mainnet), so it reaches Tempo after its children."
},
{
"name": "nodestore.rotate.clear_prior",
"category": "ledger",
"allowed_parents": ["nodestore.rotate"],
"required_attributes": [],
"config_flag": "trace_ledger",
"optional": true,
"note": "SQL deletion of ledgers below lastRotated."
},
{
"name": "nodestore.rotate.copy",
"category": "ledger",
"allowed_parents": ["nodestore.rotate"],
"required_attributes": ["node_count", "nodes_copied"],
"config_flag": "trace_ledger",
"optional": true,
"note": "The whole-state-map walk; ten minutes or more on a populated node. nodes_copied is how many of the walked nodes were found only in the archive and had to be written to the writable backend (DatabaseRotating::duplicateCopyForwardTotal delta across the walk)."
},
{
"name": "nodestore.rotate.freshen.fetch",
"category": "ledger",
"allowed_parents": ["nodestore.rotate"],
"required_attributes": ["key_count", "cache", "keys_copied"],
"config_flag": "trace_ledger",
"optional": true,
"note": "The per-key re-fetch loop. Keys are copied one cache partition at a time (TaggedCache::forEachKeyPartition) so the tree-node cache mutex is never held across the whole cache; the per-partition hold shows up on cache_metrics{metric=\"treenode_lock_hold_peak_us\"}. A separate freshen.keys child once covered a single getKeys() hold that lasted seconds on a large cache and dropped the node out of sync on every rotation. Two instances per rotation, told apart by cache (treenode | master_tx). keys_copied is the freshen's yield: keys that were only in the archive (duplicateCopyForwardTotal delta)."
},
{
"name": "nodestore.rotate.new_backend",
"category": "ledger",
"allowed_parents": ["nodestore.rotate"],
"required_attributes": [],
"config_flag": "trace_ledger",
"optional": true,
"note": "Creating the new NuDB/RocksDB backend."
},
{
"name": "nodestore.rotate.clear_caches",
"category": "ledger",
"allowed_parents": ["nodestore.rotate"],
"required_attributes": [],
"config_flag": "trace_ledger",
"optional": true,
"note": "LedgerMaster::clearLedgerCachePrior + FullBelowCache::clear."
},
{
"name": "nodestore.rotate.swap",
"category": "ledger",
"allowed_parents": ["nodestore.rotate"],
"required_attributes": ["copy_forwards"],
"config_flag": "trace_ledger",
"optional": true,
"note": "DatabaseRotatingImp::rotate, including the second clearCaches inside its callback. copy_forwards is DatabaseRotating::copyForwardTotal() at swap time."
},
{
"name": "nodestore.rotate.health_wait",
"category": "ledger",
"allowed_parents": [
"nodestore.rotate",
"nodestore.rotate.clear_prior",
"nodestore.rotate.copy",
"nodestore.rotate.freshen.fetch",
"nodestore.rotate.new_backend",
"nodestore.rotate.clear_caches",
"nodestore.rotate.swap"
],
"required_attributes": ["server_mode", "missing_ledgers"],
"config_flag": "trace_ledger",
"optional": true,
"note": "Emitted only when healthWait() actually waits inside a running rotation; may repeat if the loop iterates. The pre-rotation healthWait() at the top of run() does not open this span."
}
],
"parent_child_relationships": [
{
"parent": "rpc.ws_message",
"child": "rpc.process",
"description": "WebSocket message contains processing span",
"skip": true,
"skip_reason": "This relationship does not exist in the code: rpc.process is created only in ServerHandler::processRequest() (ServerHandler.cpp:718), reached only from processSession(Session, coro) (ServerHandler.cpp:646) — the HTTP/JSON-RPC path. The WebSocket path (processSession(WSSession, coro, jv), ServerHandler.cpp:467) never calls processRequest, so rpc.process is never emitted at all under the WebSocket-only harness. The earlier diagnosis (cross-thread context loss needing a C++ fix) was wrong: rpc.ws_message is a deliberate freshRoot (ServerHandler.cpp:473-474) so each WS message is its own trace rather than nesting under a span leaked on a reused coroutine worker. Nothing to fix."
},
{
"parent": "rpc.ws_message",
"child": "rpc.command.*",
"description": "WebSocket message contains the per-command span — the real relationship on the harness WS path (rpc::doCommand at RPCHandler.cpp:271 creates an ambient child of the rpc.ws_message scope inside the same coroutine). Not skipped, because the validator globs the wildcard child: _span_name_matches() matches via fnmatch.fnmatchcase, so any rpc.command.<anything> under the parent satisfies the contract and the command mix does not matter. A validator that instead collapsed the wildcard to one literal name -- child_name.replace(\"*\", \"server_info\") -- would make this check depend on which command the sampled traces happened to carry: server_info is 25/100 of rpc_load_generator.py's DEFAULT_WEIGHTS, so a healthy run could sample three non-server_info traces and fail."
},
{
"parent": "rpc.process",
"child": "rpc.command.*",
"description": "Processing span contains the per-command span, on the HTTP/JSON-RPC path. Not skipped, for the same reason as the WebSocket pair above: the validator globs a wildcard child via _span_name_matches(), so no single literal command has to be sampled. Both ends emit: rpc.process reports 5 traces on a normal run, not from the WebSocket-only load generator but because run-full-validation.sh polls each node's HTTP port with curl for readiness and validated-ledger progress (:449, :502), and every such request runs a command."
},
{
"parent": "ledger.build",
"child": "tx.apply",
"description": "Ledger build contains transaction application"
},
{
"parent": "consensus.round",
"child": "consensus.accept",
"description": "Consensus round contains the accept sub-span"
},
{
"parent": "consensus.accept",
"child": "consensus.accept.apply",
"description": "Accept contains the ledger-apply sub-span"
},
{
"parent": "pathfind.request",
"child": "pathfind.compute",
"description": "Pathfind request contains the compute sub-span",
"skip": true,
"skip_reason": "Real relationship (pathfind.compute is created inside PathRequest::doUpdate at PathRequest.cpp:749-750, under the pathfind.request scope), but the child never exists on the harness because pathfinding is disabled on every node: Config.cpp:725-726 zeroes pathSearchMax whenever a [validation_seed] or [validator_token] section is present, run-full-validation.sh writes [validation_seed] for all five nodes with no [path_search*] override, and doRipplePathFind returns RpcNotSupported at RipplePathFind.cpp:59-60 before constructing a PathRequest. The parent would appear anyway if the RPC were issued, because its ScopedSpanGuard is created at RipplePathFind.cpp:35, above that guard; but rpc_load_generator.py carries no ripple_path_find weight, so not even the parent appears and both ends of this relationship are absent. Liquidity has nothing to do with it — the earlier 'no liquidity, returns before computing' reason was wrong, because no path search is attempted at all. Asserting this relationship needs the load restored AND a [path_search_max] override (or a non-validator node) in run-full-validation.sh."
},
{
"parent": "ledger.acquire",
"child": "ledger.acquire.astree",
"description": "Ledger acquire contains the account-state tree fetch phase. Un-skipped 2026-08-26. The old skip reason said the parent is optional because a healthy 5-node cluster agreeing from genesis rarely back-fills, so the check would fail on a parent with no traces. Run 32969481032 refutes the premise: ledger.acquire reported 5 traces and so did each of its three phase children, one per node. The parenting was never the doubtful part -- beginPhaseSpan() parents through the acquire span's own captured SpanContext rather than the ambient thread context, so it holds on whichever worker opens a phase."
},
{
"parent": "ledger.acquire",
"child": "ledger.acquire.header",
"description": "Ledger acquire contains the header fetch phase, which gates the two tree fetches. Added 2026-08-26: all three phases are opened identically through the parent's captured SpanContext and all three emitted 5 traces on run 32969481032, but only astree had ever been declared here, so two thirds of the phase hierarchy went unchecked."
},
{
"parent": "ledger.acquire",
"child": "ledger.acquire.txtree",
"description": "Ledger acquire contains the transaction-tree fetch phase. Un-skipped once the hierarchy check stopped sampling only the newest parent traces. This phase is the conditional one of the three: header opens on !haveHeader_ (InboundLedger.cpp:672) and astree in the else of haveState_ (:689), both of which hold on essentially every acquire, while txtree opens only in the else of haveTransactions_ (:698) -- and a node in this cluster usually already holds the transaction set, because every node sees the same relayed transactions and builds the same set. So the phase fires on a minority of acquires, which is precisely the case newest-N sampling of the parent got wrong; the child was always emitting, just not in the three most recent acquires."
},
{
"parent": "rpc.command.*",
"child": "pathfind.request",
"description": "The RPC command span contains the path-finding request span.",
"skip": true,
"skip_reason": "Real relationship, and the one skip here caused by a WILDCARD PARENT rather than by a missing span. _validate_parent_child builds its Tempo query as name=\"<parent>\" with the contract string inserted literally (validate_telemetry.py:801), so a parent of rpc.command.* searches for a span literally named that and finds nothing. Note the asymmetry: the ancestry check globs both sides through _span_name_matches, which is why rpc.ws_message -> rpc.command.* is asserted, but the Tempo query that selects the candidate traces is literal on the parent, so no trace is ever fetched to run it on. Asserting this needs the parent query to accept a glob -- a TraceQL name=~ regex, or resolving the glob to the concrete names Tempo reports first. Independently of that, both ends are absent today anyway: the harness issues no path-finding RPC, see the pathfind.compute entry above.",
"added": "Closes the declared-but-unlisted gap"
},
{
"parent": "pathfind.compute",
"child": "pathfind.discover",
"description": "The path computation contains the discovery pass.",
"skip": true,
"skip_reason": "Real relationship with BOTH ends absent, for the reason given in full on the pathfind.compute entry above: pathfinding is disabled on every harness node because Config.cpp:725-726 zeroes pathSearchMax whenever a [validation_seed] section is present, run-full-validation.sh writes one for all five nodes with no [path_search_max] override, so doRipplePathFind returns RpcNotSupported before any PathRequest is constructed -- and the harness sends no path-finding RPC at all either. Asserting this needs both blockers lifted, which is a workload and node-config change rather than a harness one. Listed here so that the pathfinding family is fully accounted for rather than partly silent.",
"added": "Closes the declared-but-unlisted gap"
},
{
"parent": "rpc.http_request",
"child": "rpc.process",
"description": "The HTTP request span contains the request-processing span. ServerHandler::processRequest() opens rpc.process as a scoped child and its own comment names rpc.http_request as the parent; processRequest has exactly one caller. Both ends emit on every run: the harness polls each node's HTTP port with curl, so this is the one rpc hierarchy that is assertable without a wildcard."
},
{
"parent": "tx.process",
"child": "txq.enqueue",
"description": "Transaction processing contains the queue-admission span."
},
{
"parent": "txq.enqueue",
"child": "txq.apply_direct",
"description": "Queue admission contains the direct-apply path taken when the transaction is applied straight to the open ledger instead of being queued."
},
{
"parent": "txq.enqueue",
"child": "txq.batch_clear",
"description": "Queue admission contains the batch-clear pass that drops an account's superseded queued transactions.",
"skip": true,
"skip_reason": "Real relationship, skipped for one reason: the child never fires at all under this workload -- the run reports \"span.txq.batch_clear: optional span not emitted under this workload\" -- because it is created in TxQ::tryClearAccountQueueUpThruTx (TxQ.cpp:550), which needs one account holding several queued transactions AND an arriving transaction that supersedes the whole batch. Nothing in txq-burst arranges that shape. Sampling is not a second reason here: the hierarchy check queries Tempo for traces containing both parent and child, so it would find the pair wherever it occurred. This is purely a workload gap, and un-skipping it needs the workload to produce a supersedable batch -- nothing further from the validator."
},
{
"parent": "txq.accept",
"child": "txq.accept_tx",
"description": "The queue's accept pass contains the per-transaction accept span. Not skipped, because the hierarchy check asks Tempo for traces containing both spans rather than sampling the newest parent traces. That matters here: the child is created inside the loop over queued transactions and behind `if (feeLevelPaid >= requiredFeeLevel)` (TxQ.cpp:1530), so it exists only for a close whose queue held a fee-clearing transaction, while the parent fires on every close (:1499) -- exactly the shape newest-N sampling gets wrong. Searching for co-occurrence finds it wherever it happened, not only in the most recent closes."
},
{
"parent": "consensus.round",
"child": "consensus.phase.open",
"description": "A consensus round contains its open phase."
},
{
"parent": "consensus.round",
"child": "consensus.establish",
"description": "A consensus round contains its establish phase."
},
{
"parent": "consensus.round",
"child": "consensus.proposal.send",
"description": "A consensus round contains the proposals this node sends during it."
},
{
"parent": "consensus.round",
"child": "consensus.ledger_close",
"description": "A consensus round contains the ledger close it performs."
},
{
"parent": "consensus.round",
"child": "consensus.mode_change",
"description": "A consensus round contains any mode transition that happens inside it."
},
{
"parent": "consensus.establish",
"child": "consensus.update_positions",
"description": "The establish phase contains each position-update pass."
},
{
"parent": "consensus.establish",
"child": "consensus.check",
"description": "The establish phase contains each consensus-reached check."
},
{
"parent": "nodestore.rotate",
"child": "nodestore.rotate.clear_prior",
"description": "phase child of the rotation root; both optional, so the check is a no-op until a harness step forces a rotation",
"skip": true,
"skip_reason": "nodestore.rotate never fires under the harness; see the span note."
},
{
"parent": "nodestore.rotate",
"child": "nodestore.rotate.copy",
"description": "phase child of the rotation root; both optional",
"skip": true,
"skip_reason": "nodestore.rotate never fires under the harness; see the span note."
},
{
"parent": "nodestore.rotate",
"child": "nodestore.rotate.freshen.fetch",
"description": "phase child of the rotation root; both optional",
"skip": true,
"skip_reason": "nodestore.rotate never fires under the harness; see the span note."
},
{
"parent": "nodestore.rotate",
"child": "nodestore.rotate.new_backend",
"description": "phase child of the rotation root; both optional",
"skip": true,
"skip_reason": "nodestore.rotate never fires under the harness; see the span note."
},
{
"parent": "nodestore.rotate",
"child": "nodestore.rotate.clear_caches",
"description": "phase child of the rotation root; both optional",
"skip": true,
"skip_reason": "nodestore.rotate never fires under the harness; see the span note."
},
{
"parent": "nodestore.rotate",
"child": "nodestore.rotate.swap",
"description": "phase child of the rotation root; both optional",
"skip": true,
"skip_reason": "nodestore.rotate never fires under the harness; see the span note."
},
{
"parent": "nodestore.rotate",
"child": "nodestore.rotate.health_wait",
"description": "phase child, emitted only when a rotation waits",
"skip": true,
"skip_reason": "nodestore.rotate never fires under the harness; see the span note."
},
{
"parent": "tx.process",
"child": "tx.receive",
"description": "CROSS-NODE. The submitting node's tx.process span is the parent of the tx.receive span on every peer the transaction is relayed to. NetworkOPs::apply() injects the tx.process span's own context into the relayed TMTransaction (NetworkOPs.cpp:1979, the span created at :1630 by txProcessSpan), and PeerImp extracts it into a child span. Both ends emit on every run: tx_submitter.py submits to one node and the cluster relays."
},
{
"parent": "consensus.proposal.send",
"child": "consensus.proposal.receive",
"description": "CROSS-NODE. RCLConsensus::Adaptor::propose() injects the send span's own context into the TMProposeSet, and the receiving peer parents consensus.proposal.receive to it (ConsensusReceiveTracing.h, the has_trace_context() branch). Asserted rather than joined on a hash, because this pair really is parent and child. Until the send side was fixed it injected the AMBIENT context, which is never set on that thread, so every receiver silently took its freshRoot fallback and this relationship held for nobody."
},
{
"parent": "consensus.validation.send",
"child": "consensus.validation.receive",
"description": "CROSS-NODE. The same pairing for validations, injected by RCLConsensus::Adaptor::validate(). Declared separately from the proposal pair because the two send sites are independent, and a regression in one says nothing about the other."
}
],
"trace_join_groups": {
"description": "Groups of spans that share ONE trace id without any parent/child link between them, because each derives its trace id deterministically from the same hash (SpanGuard::hashSpan). This is how spans produced on unrelated threads are joined: no context is propagated, so there is no parent to assert -- the assertion is that the spans co-occur in a single trace. Checked by assert_trace_join_groups() in validate_telemetry.py, which searches for the anchor span and requires at least one of its traces to also contain every member listed in required_members.",
"groups": [
{
"name": "per_ledger",
"join_key": "ledger_hash",
"anchor": "ledger.validate",
"required_members": ["ledger.store"],
"optional_members": ["consensus.validation.accept", "ledger.acquire"],
"note": "One ledger's spans, keyed on its own 32-byte hash: the acceptance decision (ledger.validate, LedgerMaster::checkAccept), the persist (ledger.store), the trusted validation that drove acceptance (consensus.validation.accept) and the network fetch (ledger.acquire). Each runs on a different thread, so before the join each was its own single-span trace and a slow ledger could not be read as one unit. ledger.validate is the anchor because it fires for every validated ledger. ledger.store is required with it: both run for every ledger the node accepts, and checkAccept reaches storeLedger on the acquire path while consensus reaches it via buildLCL, so at least one ordering always produces both. consensus.validation.accept is optional here only because a ledger this node built itself is accepted via switchLCL without a peer validation arriving first, so the two need not land in the same trace on every ledger. ledger.acquire is optional for the reason its own entry gives: a healthy cluster agreeing from genesis rarely back-fills."
}
]
},
"_conditional_attributes_note": "Five attributes documented in the 'Fresh-node sync diagnostics' table of docs/telemetry-glossary.md are deliberately absent from required_attributes above, because each is emitted only when its value is known and _validate_span_attributes_otlp() has no per-attribute optional flag -- listing one would fail CI red on a healthy run. ledger.acquire/peer_count is set only when finalizeAcquireSpan() is passed a peer count (InboundLedger.cpp), which the sweep and shutdown paths cannot supply. ledger_seq on the three ledger.acquire.header/.astree/.txtree phase spans is set only when seq_ != 0 (InboundLedger.cpp startPhaseSpan), and a by-hash acquire starts with seq_ == 0 and learns the sequence only when the header arrives -- so a phase that opens before the header legitimately carries no sequence. ledger_seq on ledger.serve is set only when the reply carries one (PeerImp.cpp), which an object-by-hash request does not. All five ARE indexed in the glossary table and rendered by the Ledger Sync Health board; the honest encoding is to document them here rather than assert a conditional attribute as required.",
"total_span_types": 56,
"total_unique_attributes": 81
}
Reference in New Issue View Git Blame Copy Permalink