mirror of
https://github.com/XRPLF/rippled.git
synced 2026-08-21 14:20:56 +00:00
22fd5e8601
The board and runbook had grown by append across eight work packages, so they read in the order the work was done rather than the order a node progresses. This is the coherence pass; it adds no new instrumentation. - Dashboard: 52 panels regrouped from two rows into nine that follow the fresh-start sequence — bootstrap, peer supply, sync state, acquire and SHAMap fetch, job queue, quorum and publish, terminal blockers, then back-fill and spans collapsed since they answer conditional questions. Layout only: no title, query or description changed. - Runbook: the flat step list becomes a decision tree branching on the observed symptom, with the amendment-block check first because it is terminal. Each branch names the panels, what healthy and unhealthy look like, and what to conclude. The existing steps are kept as the detail bodies. - Reference table: every signal name re-checked against the code and every named panel against the board; four stale panel references fixed. - Validation: every signal is now either asserted or covered by a note explaining why a five-node local cluster cannot produce it. Also fixes the write-latency signal, which was inert on a real node: the store duration was only recorded on the database-import path, while the two production store implementations did not time themselves, so an ordinary node reported a write count with no latency. Both now time the backend write, which is the disk work this signal exists to expose. Without it the "existing database syncs slower than a fresh one" diagnosis had no primary signal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
528 lines
25 KiB
JSON
528 lines
25 KiB
JSON
{
|
|
"description": "Expected span inventory for xrpld telemetry validation. Attribute keys follow the 2026-05-13 span-attr naming redesign (bare/underscore form; dotted xrpl.* reserved for resource attributes). Sourced from the *SpanNames.h headers. Spans marked \"optional\": true are conditional \u2014 they only fire under traffic the harness may not produce (e.g. gRPC client, missing-ledger fetch, mode transitions) and are not failed when absent.",
|
|
"spans": [
|
|
{
|
|
"name": "rpc.ws_message",
|
|
"category": "rpc",
|
|
"parent": null,
|
|
"required_attributes": ["command"],
|
|
"config_flag": "trace_rpc",
|
|
"note": "WebSocket RPC root span. The load generator uses WS, so this is the RPC entry span (not rpc.http_request, which needs an HTTP/JSON-RPC client)."
|
|
},
|
|
{
|
|
"name": "rpc.process",
|
|
"category": "rpc",
|
|
"parent": "rpc.ws_message",
|
|
"required_attributes": [],
|
|
"config_flag": "trace_rpc"
|
|
},
|
|
{
|
|
"name": "rpc.command.*",
|
|
"category": "rpc",
|
|
"parent": "rpc.process",
|
|
"required_attributes": ["command", "version", "rpc_role", "rpc_status"],
|
|
"config_flag": "trace_rpc",
|
|
"note": "Wildcard \u2014 matches rpc.command.server_info, rpc.command.ledger, etc."
|
|
},
|
|
{
|
|
"name": "rpc.http_request",
|
|
"category": "rpc",
|
|
"parent": null,
|
|
"required_attributes": ["request_payload_size"],
|
|
"config_flag": "trace_rpc",
|
|
"optional": true,
|
|
"note": "HTTP/JSON-RPC root span. The harness load generator is WebSocket-only, so this does not fire."
|
|
},
|
|
{
|
|
"name": "tx.process",
|
|
"category": "transaction",
|
|
"parent": null,
|
|
"required_attributes": ["tx_hash", "local", "path"],
|
|
"config_flag": "trace_transactions"
|
|
},
|
|
{
|
|
"name": "tx.receive",
|
|
"category": "transaction",
|
|
"parent": null,
|
|
"required_attributes": ["tx_hash", "peer_id", "suppressed"],
|
|
"config_flag": "trace_transactions",
|
|
"note": "Cross-node span: parent context propagated from the sender's tx.process via protobuf. Also carries tx_type and peer_version. tx_status is only set when a tx is suppressed/known-bad, so it is not a required attribute on every tx.receive."
|
|
},
|
|
{
|
|
"name": "tx.apply",
|
|
"category": "transaction",
|
|
"parent": "ledger.build",
|
|
"required_attributes": ["tx_count", "tx_failed"],
|
|
"config_flag": "trace_transactions",
|
|
"note": "Apply-step span inside BuildLedger. Carries tx_count/tx_failed (ledger_seq lives on the parent ledger.build span)."
|
|
},
|
|
{
|
|
"name": "tx.preflight",
|
|
"category": "transaction",
|
|
"parent": null,
|
|
"required_attributes": ["stage", "tx_type", "ter_result"],
|
|
"config_flag": "trace_transactions",
|
|
"note": "Apply-pipeline stage span (stage=preflight). Shares a deterministic trace_id (txID[0:16]) with tx.preclaim/tx.transactor."
|
|
},
|
|
{
|
|
"name": "tx.preclaim",
|
|
"category": "transaction",
|
|
"parent": null,
|
|
"required_attributes": ["stage", "tx_type", "ter_result"],
|
|
"config_flag": "trace_transactions",
|
|
"note": "Apply-pipeline stage span (stage=preclaim)."
|
|
},
|
|
{
|
|
"name": "tx.transactor",
|
|
"category": "transaction",
|
|
"parent": null,
|
|
"required_attributes": ["stage", "tx_type"],
|
|
"config_flag": "trace_transactions",
|
|
"note": "Apply-pipeline stage span (stage=apply). Also carries applied."
|
|
},
|
|
{
|
|
"name": "txq.enqueue",
|
|
"category": "transaction",
|
|
"parent": "tx.process",
|
|
"required_attributes": ["tx_hash", "tx_type", "txq_status"],
|
|
"config_flag": "trace_transactions",
|
|
"optional": true,
|
|
"note": "Only fires when a tx is queued (fee below open-ledger level). Requires fee escalation \u2014 driven by the txq-burst workload phase. tx_hash/tx_type/txq_status are set on every code path; fee_level_paid/required_fee_level are conditional (TxQ.cpp ~895-898, after the rejected and applied_direct early exits), so they are NOT guaranteed on every txq.enqueue span and cannot be required."
|
|
},
|
|
{
|
|
"name": "txq.apply_direct",
|
|
"category": "transaction",
|
|
"parent": "txq.enqueue",
|
|
"required_attributes": [],
|
|
"config_flag": "trace_transactions",
|
|
"optional": true,
|
|
"note": "Child of txq.enqueue when the tx applies directly without queueing."
|
|
},
|
|
{
|
|
"name": "txq.batch_clear",
|
|
"category": "transaction",
|
|
"parent": "txq.enqueue",
|
|
"required_attributes": ["num_cleared"],
|
|
"config_flag": "trace_transactions",
|
|
"optional": true
|
|
},
|
|
{
|
|
"name": "txq.accept",
|
|
"category": "transaction",
|
|
"parent": null,
|
|
"required_attributes": ["queue_size", "ledger_changed"],
|
|
"config_flag": "trace_transactions",
|
|
"optional": true,
|
|
"note": "Ledger-close accept loop. Fires on the consensus thread; only meaningful when the queue is non-empty."
|
|
},
|
|
{
|
|
"name": "txq.accept_tx",
|
|
"category": "transaction",
|
|
"parent": "txq.accept",
|
|
"required_attributes": [
|
|
"tx_hash",
|
|
"ter_code",
|
|
"retries_remaining",
|
|
"txq_status"
|
|
],
|
|
"config_flag": "trace_transactions",
|
|
"optional": true
|
|
},
|
|
{
|
|
"name": "txq.cleanup",
|
|
"category": "transaction",
|
|
"parent": null,
|
|
"required_attributes": ["ledger_seq", "expired_count"],
|
|
"config_flag": "trace_transactions",
|
|
"optional": true
|
|
},
|
|
{
|
|
"name": "consensus.round",
|
|
"category": "consensus",
|
|
"parent": null,
|
|
"required_attributes": [
|
|
"consensus_ledger_id",
|
|
"ledger_seq",
|
|
"consensus_mode",
|
|
"consensus_round_id",
|
|
"consensus_phase"
|
|
],
|
|
"config_flag": "trace_consensus",
|
|
"note": "Root consensus span created per round. Also carries trace_strategy, previous_ledger_seq, previous_proposers, previous_round_time_ms."
|
|
},
|
|
{
|
|
"name": "consensus.phase.open",
|
|
"category": "consensus",
|
|
"parent": "consensus.round",
|
|
"required_attributes": [],
|
|
"config_flag": "trace_consensus"
|
|
},
|
|
{
|
|
"name": "consensus.proposal.send",
|
|
"category": "consensus",
|
|
"parent": "consensus.round",
|
|
"required_attributes": ["consensus_round"],
|
|
"config_flag": "trace_consensus",
|
|
"note": "Also carries is_bow_out."
|
|
},
|
|
{
|
|
"name": "consensus.ledger_close",
|
|
"category": "consensus",
|
|
"parent": "consensus.round",
|
|
"required_attributes": ["ledger_seq", "consensus_mode"],
|
|
"config_flag": "trace_consensus",
|
|
"note": "Also carries tx_count_open, close_time_resolution_ms."
|
|
},
|
|
{
|
|
"name": "consensus.establish",
|
|
"category": "consensus",
|
|
"parent": "consensus.round",
|
|
"required_attributes": [
|
|
"converge_percent",
|
|
"establish_count",
|
|
"proposers",
|
|
"disputes_count"
|
|
],
|
|
"config_flag": "trace_consensus"
|
|
},
|
|
{
|
|
"name": "consensus.update_positions",
|
|
"category": "consensus",
|
|
"parent": "consensus.round",
|
|
"required_attributes": [
|
|
"converge_percent",
|
|
"proposers",
|
|
"disputes_count"
|
|
],
|
|
"config_flag": "trace_consensus"
|
|
},
|
|
{
|
|
"name": "consensus.check",
|
|
"category": "consensus",
|
|
"parent": "consensus.round",
|
|
"required_attributes": [
|
|
"agree_count",
|
|
"disagree_count",
|
|
"threshold_percent",
|
|
"consensus_result"
|
|
],
|
|
"config_flag": "trace_consensus"
|
|
},
|
|
{
|
|
"name": "consensus.accept",
|
|
"category": "consensus",
|
|
"parent": "consensus.round",
|
|
"required_attributes": ["proposers", "round_time_ms", "quorum"],
|
|
"config_flag": "trace_consensus"
|
|
},
|
|
{
|
|
"name": "consensus.accept.apply",
|
|
"category": "consensus",
|
|
"parent": "consensus.accept",
|
|
"required_attributes": [
|
|
"ledger_seq",
|
|
"close_time",
|
|
"parent_close_time",
|
|
"close_time_self",
|
|
"close_time_vote_bins",
|
|
"resolution_direction"
|
|
],
|
|
"config_flag": "trace_consensus",
|
|
"note": "Also carries close_time_correct, close_resolution_ms, consensus_state, proposing, round_time_ms, tx_count."
|
|
},
|
|
{
|
|
"name": "consensus.validation.send",
|
|
"category": "consensus",
|
|
"parent": null,
|
|
"required_attributes": [
|
|
"ledger_seq",
|
|
"proposing",
|
|
"ledger_hash",
|
|
"full_validation"
|
|
],
|
|
"config_flag": "trace_consensus",
|
|
"note": "follows-from consensus.accept. ledger_hash and full_validation are shared with peer.validation.receive (same keys, told apart by span name). Also carries validation_sign_time."
|
|
},
|
|
{
|
|
"name": "consensus.proposal.receive",
|
|
"category": "consensus",
|
|
"parent": null,
|
|
"required_attributes": [],
|
|
"config_flag": "trace_consensus",
|
|
"note": "Context-propagated from the sending peer. No required local attributes."
|
|
},
|
|
{
|
|
"name": "consensus.validation.receive",
|
|
"category": "consensus",
|
|
"parent": null,
|
|
"required_attributes": [],
|
|
"config_flag": "trace_consensus",
|
|
"note": "Context-propagated from the sending peer. No required local attributes."
|
|
},
|
|
{
|
|
"name": "consensus.validation.accept",
|
|
"category": "consensus",
|
|
"parent": null,
|
|
"required_attributes": [
|
|
"ledger_hash",
|
|
"ledger_seq",
|
|
"validation_status",
|
|
"accept_gated",
|
|
"full_validation"
|
|
],
|
|
"config_flag": "trace_ledger",
|
|
"note": "Emitted once per TRUSTED validation as it reaches the ledger-acceptance gate (handleNewValidation), so its rate is bounded by the UNL size per ledger close. Required, not optional: every node in the harness cluster validates and every other node receives those validations, so the span fires continuously. config_flag is trace_ledger, not trace_consensus, because the span is created through the ledger-hash trace join (TraceCategory::Ledger) -- it belongs to the ledger's trace, not the round's. Its trace id is derived from the VALIDATED ledger hash, the same key ledger.validate and ledger.store use, which is what makes one slow ledger read as one connected trace across the three threads that produce those spans. validation_status is one of the 6 ValStatus values and accept_gated says whether the validation actually reached the gate; both are spanmetrics dimensions. See trace_join_groups below."
|
|
},
|
|
{
|
|
"name": "consensus.mode_change",
|
|
"category": "consensus",
|
|
"parent": null,
|
|
"required_attributes": ["mode_old", "mode_new"],
|
|
"config_flag": "trace_consensus",
|
|
"optional": true,
|
|
"note": "Only fires on an operating-mode transition; a steady cluster rarely changes mode after warmup."
|
|
},
|
|
{
|
|
"name": "ledger.build",
|
|
"category": "ledger",
|
|
"parent": null,
|
|
"required_attributes": [
|
|
"ledger_seq",
|
|
"close_time",
|
|
"close_time_correct",
|
|
"close_resolution_ms"
|
|
],
|
|
"config_flag": "trace_ledger",
|
|
"note": "tx_count/tx_failed live on the child tx.apply span, not here."
|
|
},
|
|
{
|
|
"name": "ledger.validate",
|
|
"category": "ledger",
|
|
"parent": null,
|
|
"required_attributes": ["ledger_seq", "validations"],
|
|
"config_flag": "trace_ledger"
|
|
},
|
|
{
|
|
"name": "ledger.store",
|
|
"category": "ledger",
|
|
"parent": null,
|
|
"required_attributes": ["ledger_seq"],
|
|
"config_flag": "trace_ledger"
|
|
},
|
|
{
|
|
"name": "ledger.acquire",
|
|
"category": "ledger",
|
|
"parent": null,
|
|
"required_attributes": [
|
|
"ledger_seq",
|
|
"acquire_reason",
|
|
"timeouts",
|
|
"outcome",
|
|
"ledger_hash"
|
|
],
|
|
"config_flag": "trace_ledger",
|
|
"optional": true,
|
|
"note": "Only fires when a node must fetch a missing ledger (InboundLedger). A healthy local cluster rarely back-fills history. outcome is one of complete|failed|abandoned and is stamped on every exit path, including the sweep/shutdown path where the fetch never finished (abandoned). ledger_hash identifies the target ledger from the first moment, since a by-hash acquire starts with ledger_seq 0. peer_count is not required: the destructor path deliberately skips the peer lookup to avoid taking the Overlay lock under the InboundLedgers collection lock."
|
|
},
|
|
{
|
|
"name": "ledger.acquire.header",
|
|
"category": "ledger",
|
|
"parent": "ledger.acquire",
|
|
"required_attributes": ["ledger_hash", "outcome", "timed_out"],
|
|
"config_flag": "trace_ledger",
|
|
"optional": true,
|
|
"note": "Child of ledger.acquire, so it is optional for the same reason the parent is: it exists only while a node is fetching a missing ledger, and a healthy 5-node cluster that has agreed from genesis rarely back-fills. Covers the wait for the ledger header, which gates both tree phases -- until it arrives the account-state and transaction root hashes are unknown, so nothing else can be requested. missing_nodes is deliberately absent: a header is a single object, not a tree."
|
|
},
|
|
{
|
|
"name": "ledger.acquire.astree",
|
|
"category": "ledger",
|
|
"parent": "ledger.acquire",
|
|
"required_attributes": [
|
|
"ledger_hash",
|
|
"outcome",
|
|
"timed_out",
|
|
"missing_nodes"
|
|
],
|
|
"config_flag": "trace_ledger",
|
|
"optional": true,
|
|
"note": "Child of ledger.acquire (optional for the same reason). The account-state SHAMap phase, which is nearly all of the work in a real fresh sync -- the flat parent span could not separate it from the small transaction tree. outcome=timeout with a non-zero missing_nodes is the 'peers are not serving this tree' signature. missing_nodes is read from the count getMissingNodes() already produced during its sweep; no extra tree walk."
|
|
},
|
|
{
|
|
"name": "ledger.acquire.txtree",
|
|
"category": "ledger",
|
|
"parent": "ledger.acquire",
|
|
"required_attributes": [
|
|
"ledger_hash",
|
|
"outcome",
|
|
"timed_out",
|
|
"missing_nodes"
|
|
],
|
|
"config_flag": "trace_ledger",
|
|
"optional": true,
|
|
"note": "Child of ledger.acquire (optional for the same reason). The transaction SHAMap phase. Usually completes long before the astree phase, and that asymmetry is the point of splitting them: the parent span's duration is the state tree's, not this one's."
|
|
},
|
|
{
|
|
"name": "ledger.serve",
|
|
"category": "ledger",
|
|
"parent": null,
|
|
"required_attributes": [
|
|
"object_type",
|
|
"peer_id",
|
|
"served_nodes",
|
|
"outcome"
|
|
],
|
|
"config_flag": "trace_ledger",
|
|
"note": "The supply side: this node answering a peer's TMGetLedger request on the JtLedgerReq worker. Required (not optional) because every node in the harness cluster is listed in the others' [ips], so they exchange ledger and tx-set requests continuously throughout the run. A fresh trace root -- the request arrives from the wire on a shared worker, so it must not inherit an unrelated span active there. object_type is header|tx|as|txset and outcome is complete|partial|refused, both derived by shared rules in LedgerSpanNames.h so the eight exits of processLedgerRequest cannot disagree. ledger_seq is present only once getLedger() succeeded, so it is not required."
|
|
},
|
|
{
|
|
"name": "txset.acquire",
|
|
"category": "ledger",
|
|
"parent": null,
|
|
"required_attributes": [
|
|
"txset_hash",
|
|
"outcome",
|
|
"timeouts",
|
|
"duration_ms",
|
|
"peer_count"
|
|
],
|
|
"config_flag": "trace_ledger",
|
|
"optional": true,
|
|
"note": "One attempt to fetch a transaction set a consensus proposal referenced (TransactionAcquire, which had zero telemetry before WP-B2). Optional because it only fires when a node does NOT already hold a proposed set: in the harness cluster every node sees the same relayed transactions and builds the same set locally, so InboundTransactions::getSet finds it in its map and never constructs a TransactionAcquire. It is the sibling of ledger.acquire -- same TimeoutCounter base, same trigger/onTimer/takeNodes shape -- and shares the trace_ledger flag so the two halves of a stuck sync cannot be enabled apart. outcome is complete|failed|timeout|abandoned, stamped on both exits (done() and the destructor when the round sweep drops the set)."
|
|
},
|
|
{
|
|
"name": "peer.dial",
|
|
"category": "peer",
|
|
"parent": null,
|
|
"required_attributes": ["remote_endpoint", "outcome", "duration_ms"],
|
|
"config_flag": "trace_peer",
|
|
"note": "One outbound connect attempt (ConnectAttempt), a fresh trace root because a dial is the first thing a starting node does and there is nothing to parent it to. Required: run-full-validation.sh lists the other four nodes in each node's [ips], so every node dials and the span always fires. Telemetry is live in time to catch it -- ApplicationImp::setup() calls startTelemetry() before start() calls overlay_->start(). outcome carries the same five values as the overlay_connect_total counter (connected|tcp_fail|tls_fail|upgrade_fail|timeout) and is set from the same reportOutcome() funnel, so span and counter cannot disagree. remote_endpoint is the span-only dimension the counter cannot carry, since one series per peer address would be unbounded cardinality."
|
|
},
|
|
{
|
|
"name": "peer.proposal.receive",
|
|
"category": "peer",
|
|
"parent": null,
|
|
"required_attributes": ["peer_id", "proposal_trusted"],
|
|
"config_flag": "trace_peer"
|
|
},
|
|
{
|
|
"name": "peer.validation.receive",
|
|
"category": "peer",
|
|
"parent": null,
|
|
"required_attributes": [
|
|
"peer_id",
|
|
"validation_trusted",
|
|
"ledger_hash",
|
|
"full_validation"
|
|
],
|
|
"config_flag": "trace_peer",
|
|
"note": "ledger_hash and full_validation are shared with consensus.validation.send (same keys, told apart by span name)."
|
|
},
|
|
{
|
|
"name": "pathfind.request",
|
|
"category": "pathfind",
|
|
"parent": null,
|
|
"required_attributes": [
|
|
"pathfind_source_account",
|
|
"pathfind_dest_account"
|
|
],
|
|
"config_flag": "trace_rpc",
|
|
"note": "Fires on ripple_path_find / path_find RPC. Driven by the ripple_path_find load in rpc_load_generator.py."
|
|
},
|
|
{
|
|
"name": "pathfind.compute",
|
|
"category": "pathfind",
|
|
"parent": "pathfind.request",
|
|
"required_attributes": ["pathfind_fast"],
|
|
"config_flag": "trace_rpc",
|
|
"optional": true,
|
|
"note": "Only fires when PathRequest::doUpdate runs a computation; the self-to-self XRP probe from the load generator returns early without computing paths in a fresh cluster with no liquidity."
|
|
},
|
|
{
|
|
"name": "pathfind.discover",
|
|
"category": "pathfind",
|
|
"parent": "pathfind.compute",
|
|
"required_attributes": ["pathfind_search_level", "pathfind_num_paths"],
|
|
"config_flag": "trace_rpc",
|
|
"optional": true,
|
|
"note": "Graph exploration; only fires under pathfind.compute, which needs real path liquidity not present in the fresh test cluster."
|
|
},
|
|
{
|
|
"name": "pathfind.update_all",
|
|
"category": "pathfind",
|
|
"parent": null,
|
|
"required_attributes": ["pathfind_ledger_index", "pathfind_num_requests"],
|
|
"config_flag": "trace_rpc",
|
|
"optional": true,
|
|
"note": "Async recomputation at ledger close; only fires when there are active path_find subscriptions (the one-shot ripple_path_find load does not register one)."
|
|
},
|
|
{
|
|
"name": "grpc.*",
|
|
"category": "grpc",
|
|
"parent": null,
|
|
"required_attributes": ["method", "grpc_role", "grpc_status"],
|
|
"config_flag": "trace_rpc",
|
|
"optional": true,
|
|
"note": "Wildcard \u2014 grpc.<MethodName>. The harness has no gRPC client, so these do not fire. Tracked for completeness."
|
|
}
|
|
],
|
|
"parent_child_relationships": [
|
|
{
|
|
"parent": "rpc.ws_message",
|
|
"child": "rpc.process",
|
|
"description": "WebSocket message contains processing span",
|
|
"skip": true,
|
|
"skip_reason": "rpc.ws_message and rpc.process run on different threads (the WS handler posts a coroutine to JobQueue for processing). Span context is not propagated across the thread boundary. Requires a C++ fix to capture and forward the span context through the coroutine lambda."
|
|
},
|
|
{
|
|
"parent": "rpc.process",
|
|
"child": "rpc.command.*",
|
|
"description": "Processing span contains per-command span"
|
|
},
|
|
{
|
|
"parent": "ledger.build",
|
|
"child": "tx.apply",
|
|
"description": "Ledger build contains transaction application"
|
|
},
|
|
{
|
|
"parent": "consensus.round",
|
|
"child": "consensus.accept",
|
|
"description": "Consensus round contains the accept sub-span"
|
|
},
|
|
{
|
|
"parent": "consensus.accept",
|
|
"child": "consensus.accept.apply",
|
|
"description": "Accept contains the ledger-apply sub-span"
|
|
},
|
|
{
|
|
"parent": "pathfind.request",
|
|
"child": "pathfind.compute",
|
|
"description": "Pathfind request contains the compute sub-span",
|
|
"skip": true,
|
|
"skip_reason": "pathfind.compute only fires when a path computation actually runs; the self-to-self XRP probe in a fresh cluster with no liquidity returns before computing, so the child is not emitted under the harness workload."
|
|
},
|
|
{
|
|
"parent": "ledger.acquire",
|
|
"child": "ledger.acquire.astree",
|
|
"description": "Ledger acquire contains the account-state tree fetch phase",
|
|
"skip": true,
|
|
"skip_reason": "The parent ledger.acquire is itself optional: it only fires when a node must fetch a missing ledger, and a healthy 5-node cluster agreeing from genesis rarely back-fills history. The hierarchy check has no optional handling and fails when the parent produces no traces, so it is skipped rather than shipped red. The parenting itself is explicit and not thread-dependent -- beginPhaseSpan() parents through the acquire span's own captured SpanContext, not the ambient thread context -- so it holds on whichever worker opens a phase. Un-skip once the harness gains a step that forces a back-fill (start a sixth node with an empty database against the running cluster)."
|
|
}
|
|
],
|
|
"trace_join_groups": {
|
|
"description": "Groups of spans that share ONE trace id without any parent/child link between them, because each derives its trace id deterministically from the same hash (SpanGuard::hashSpan). This is how spans produced on unrelated threads are joined: no context is propagated, so there is no parent to assert -- the assertion is that the spans co-occur in a single trace. Checked by assert_trace_join_groups() in validate_telemetry.py, which searches for the anchor span and requires at least one of its traces to also contain every member listed in required_members.",
|
|
"groups": [
|
|
{
|
|
"name": "per_ledger",
|
|
"join_key": "ledger_hash",
|
|
"anchor": "ledger.validate",
|
|
"required_members": ["ledger.store"],
|
|
"optional_members": ["consensus.validation.accept", "ledger.acquire"],
|
|
"note": "One ledger's spans, keyed on its own 32-byte hash: the acceptance decision (ledger.validate, LedgerMaster::checkAccept), the persist (ledger.store), the trusted validation that drove acceptance (consensus.validation.accept) and the network fetch (ledger.acquire). Each runs on a different thread, so before the join each was its own single-span trace and a slow ledger could not be read as one unit. ledger.validate is the anchor because it fires for every validated ledger. ledger.store is required with it: both run for every ledger the node accepts, and checkAccept reaches storeLedger on the acquire path while consensus reaches it via buildLCL, so at least one ordering always produces both. consensus.validation.accept is optional here only because a ledger this node built itself is accepted via switchLCL without a peer validation arriving first, so the two need not land in the same trace on every ledger. ledger.acquire is optional for the reason its own entry gives: a healthy cluster agreeing from genesis rarely back-fills."
|
|
}
|
|
]
|
|
},
|
|
"_conditional_attributes_note": "Five attributes documented in the 'Fresh-node sync diagnostics' table of OpenTelemetryPlan/09-data-collection-reference.md are deliberately absent from required_attributes above, because each is emitted only when its value is known and _validate_span_attributes_otlp() has no per-attribute optional flag -- listing one would fail CI red on a healthy run. ledger.acquire/peer_count is set only when finalizeAcquireSpan() is passed a peer count (InboundLedger.cpp), which the sweep and shutdown paths cannot supply. ledger_seq on the three ledger.acquire.header/.astree/.txtree phase spans is set only when seq_ != 0 (InboundLedger.cpp startPhaseSpan), and a by-hash acquire starts with seq_ == 0 and learns the sequence only when the header arrives -- so a phase that opens before the header legitimately carries no sequence. ledger_seq on ledger.serve is set only when the reply carries one (PeerImp.cpp), which an object-by-hash request does not. All five ARE indexed in the 09-reference table and rendered by the Ledger Sync Health board; the honest encoding is to document them here rather than assert a conditional attribute as required.",
|
|
"total_span_types": 47,
|
|
"total_unique_attributes": 76
|
|
}
|