Files
rippled/include/xrpl/consensus/ConsensusSpanNames.h
Pratik Mankawde 1eb18a6a59 feat(telemetry): count consensus view changes and mark them on the round span
Adds `consensus_view_change_total{consensus_mode}` and a `view.change` event
on the round span, both fired from `RCLConsensus::Adaptor::getPrevLedger`
on the same transition-into-WrongLedger edge that already calls
`consensusViewChange()`. The counter is the exact detector for
"consensus disagreed with this node's view this minute"; the event lands the
disagreement on the same trace that carries the round.

`net_ledger_prefix` (16 hex chars) joins `prev_ledger_prefix` on the event,
so a Tempo view of one flap shows both ledger identities on a single line.
`consensus_mode` labels the mode being left (never WrongLedger itself).

Together with the rotation-phase spans, this closes the proof chain a
rotation-driven `full`->`syncing` flap needs — the runbook's step 5
resolves now that the counter and event exist.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-09-14 21:05:30 +01:00

510 lines
23 KiB
C++

#pragma once
/**
* Compile-time span name constants for consensus tracing.
*
* Used by RCLConsensus (app), Consensus.h (template), and PeerImp
* (overlay) for consensus lifecycle spans.
* Built on StaticStr/join() from SpanNames.h.
*
* ## Span Hierarchy
*
* Root span created in Adaptor::startRoundTracing(). In "deterministic"
* strategy the trace-id is derived from the previous ledger hash so all
* nodes tracing the same round share a trace.
*
* consensus.round [main thread, root]
* | Created: Adaptor::startRoundTracing()
* | Attrs: consensus_ledger_id, ledger_seq, trace_strategy,
* | consensus_round_id; consensus_mode from
* | Adaptor::onModeChange()
* |
* +-- consensus.phase.open [main thread, child]
* | Created: Consensus::startRoundInternal()
* | Ended: Consensus::closeLedger()
* | Attrs: start_reason, previous_close_agree, peer_positions_at_open,
* | early_close_triggered (at start); open_duration_ms,
* | peer_positions_at_close, tx_sets_acquired, close_reason,
* | proposers_validated (at close; absent if the round is
* | recovered or simulated, neither of which reaches
* | closeLedger())
* |
* +-- consensus.proposal.send [main thread]
* | Created: Adaptor::propose()
* | Attrs: consensus_round (proposeSeq)
* |
* +-- consensus.ledger_close [main thread]
* | Created: Adaptor::onClose()
* | Attrs: ledger_seq, consensus_mode
* |
* +-- consensus.establish [main thread, child]
* | Created: Consensus::startEstablishTracing()
* | Ended: Consensus::phaseEstablish() on accept
* | Attrs: converge_percent, establish_count, proposers,
* | disputes_count (overwritten each iteration);
* | close_time_avalanche_state (terminal, at end)
* |
* +-- consensus.update_positions [main thread]
* | Created: Consensus::updateOurPositions()
* | Attrs: converge_percent, proposers, disputes_count
* | Events: per-dispute vote details (tx_id, our_vote, yays, nays)
* |
* +-- consensus.check [main thread]
* | Created: Consensus::haveConsensus()
* | Attrs: agree/disagree counts, threshold_percent, result
* |
* +-- consensus.accept [main thread, child of round]
* | Created: Adaptor::makeAcceptSpan(), shared_ptr kept alive
* | until doAccept() completes on jtACCEPT thread
* | Attrs: proposers, round_time_ms, quorum
* | |
* | +-- consensus.accept.apply [jtACCEPT thread, child of accept]
* | Created: Adaptor::doAccept()
* | Attrs: ledger_seq, close_time_ripple_epoch_s, close_time_correct,
* | close_resolution_ms, consensus_state, proposing, round_time_ms,
* | parent_close_time_ripple_epoch_s, close_time_self_ripple_epoch_s,
* | close_time_vote_bins, resolution_direction, tx_count
* | Events: tx.included (per tx, attrs: tx_id)
* |
* +~~~ consensus.validation.send [jtACCEPT thread, linked]
* | Created: Adaptor::createValidationSpan() (follows-from link)
* | Attrs: ledger_seq, proposing
* |
* +-- consensus.mode_change [main thread]
* Created: Adaptor::onModeChange()
* Attrs: mode_old, mode_new
*
* Standalone spans (no parent, created per-message in overlay):
*
* consensus.proposal.receive [PeerImp I/O thread]
* Created: PeerImp::onMessage(TMProposeSet)
*
* consensus.validation.receive [PeerImp I/O thread]
* Created: PeerImp::onMessage(TMValidation)
*
* ## Per-ledger trace (separate from the per-round trace above)
*
* consensus.validation.accept sits in a DIFFERENT trace from the spans above.
* The round trace is keyed on the PREVIOUS ledger hash (the round's seed);
* this span is keyed on the hash of the ledger the arriving validation is FOR,
* the same key LedgerMaster uses for ledger.validate and ledger.store, so a
* reader following one slow ledger sees the validation that triggered its
* acceptance beside the acceptance itself:
*
* trace_id = validatedLedgerHash[0:16]
* +-- consensus.validation.accept [JtValidationT worker / consensus thread]
* +-- ledger.validate [LedgerMaster::checkAccept]
* +-- ledger.store [LedgerMaster::storeLedger]
*
* Legend:
* +-- child-of relationship (same trace)
* +~~~ follows-from link (separate sub-tree, causal link)
*/
#include <xrpl/telemetry/SpanNames.h>
#include <string_view>
namespace xrpl::telemetry::consensus::span {
// ===== Span name segments ====================================================
namespace part {
inline constexpr auto proposal = makeStr("proposal");
inline constexpr auto validation = makeStr("validation");
inline constexpr auto accept = makeStr("accept");
inline constexpr auto phase = makeStr("phase");
} // namespace part
namespace op {
inline constexpr auto round = makeStr("round");
inline constexpr auto proposalSend = join(part::proposal, makeStr("send"));
inline constexpr auto ledgerClose = makeStr("ledger_close");
inline constexpr auto establish = makeStr("establish");
inline constexpr auto updatePositions = makeStr("update_positions");
inline constexpr auto check = makeStr("check");
inline constexpr auto accept = makeStr("accept");
inline constexpr auto acceptApply = join(part::accept, makeStr("apply"));
inline constexpr auto validationSend = join(part::validation, makeStr("send"));
inline constexpr auto modeChange = makeStr("mode_change");
inline constexpr auto proposalReceive = join(part::proposal, makeStr("receive"));
inline constexpr auto validationReceive = join(part::validation, makeStr("receive"));
inline constexpr auto phaseOpen = join(part::phase, makeStr("open"));
/**
* "validation.accept" — an arriving trusted validation being handed to the
* acceptance gate. Distinct from `validationReceive`, which is the overlay
* decoding a validation message: this is the later, per-ledger step where the
* validation is counted toward accepting a specific ledger.
*/
inline constexpr auto validationAccept = join(part::validation, makeStr("accept"));
} // namespace op
// ===== Full span names (prefix.op) ===========================================
inline constexpr auto round = join(seg::consensus, op::round);
inline constexpr auto proposalSend = join(seg::consensus, op::proposalSend);
inline constexpr auto ledgerClose = join(seg::consensus, op::ledgerClose);
inline constexpr auto establish = join(seg::consensus, op::establish);
inline constexpr auto updatePositions = join(seg::consensus, op::updatePositions);
inline constexpr auto check = join(seg::consensus, op::check);
inline constexpr auto accept = join(seg::consensus, op::accept);
inline constexpr auto acceptApply = join(seg::consensus, op::acceptApply);
inline constexpr auto validationSend = join(seg::consensus, op::validationSend);
inline constexpr auto modeChange = join(seg::consensus, op::modeChange);
inline constexpr auto proposalReceive = join(seg::consensus, op::proposalReceive);
inline constexpr auto validationReceive = join(seg::consensus, op::validationReceive);
inline constexpr auto phaseOpen = join(seg::consensus, op::phaseOpen);
/**
* "consensus.validation.accept" — full name, passed to `SpanGuard::hashSpan()`
* (which takes one complete span name) so the span adopts the trace id derived
* from the VALIDATED ledger's hash and joins that ledger's trace.
*/
inline constexpr auto validationAccept = join(seg::consensus, op::validationAccept);
// ===== Attribute keys ========================================================
namespace attr {
/**
* Canonical shared constants (defined in SpanNames.h). `ledgerHash` and
* `fullValidation` are shared with the peer.validation.receive span — same
* concept, same key, distinguished by span name (not an emitter prefix).
*/
using ::xrpl::telemetry::attr::closeResolutionMs;
using ::xrpl::telemetry::attr::closeTimeCorrect;
using ::xrpl::telemetry::attr::closeTimeRippleEpochS;
using ::xrpl::telemetry::attr::fullValidation;
using ::xrpl::telemetry::attr::ledgerHash;
using ::xrpl::telemetry::attr::ledgerSeq;
/**
* Domain-qualified attrs (rule 5 — bare name ambiguous across domains).
* Use `<domain>_<field>` underscore form for TraceQL ergonomics.
*/
inline constexpr auto ledgerId = makeStr("consensus_ledger_id");
/**
* Consensus mode. On consensus.round it is written by onModeChange, the point
* at which the engine applies the mode; on consensus.ledger_close the engine
* passes the mode in.
*/
inline constexpr auto mode = makeStr("consensus_mode");
inline constexpr auto round = makeStr("consensus_round");
inline constexpr auto roundId = makeStr("consensus_round_id");
/**
* Current phase name attached to consensus.round; updated on each
* phase transition event (open/establish/accepted).
*/
inline constexpr auto consensusPhase = makeStr("consensus_phase");
/**
* Boolean flag set on consensus.check when checkConsensus reports stalled.
*/
inline constexpr auto consensusStalled = makeStr("consensus_stalled");
/**
* Domain-owned bare attrs.
*/
inline constexpr auto proposers = makeStr("proposers");
inline constexpr auto roundTimeMs = makeStr("round_time_ms");
inline constexpr auto proposing = makeStr("proposing");
/**
* Round continuity / context attrs (set on consensus.round at round start).
*/
inline constexpr auto previousProposers = makeStr("previous_proposers");
inline constexpr auto previousRoundTimeMs = makeStr("previous_round_time_ms");
inline constexpr auto previousLedgerSeq = makeStr("previous_ledger_seq");
inline constexpr auto closeTimeResolutionMs = makeStr("close_time_resolution_ms");
/**
* Open-phase start metadata (set on consensus.phase.open at creation).
*
* A handleWrongLedger recovery emits a SECOND phase.open span under the same
* round, so `start_reason` is what tells the two apart.
*/
inline constexpr auto startReason = makeStr("start_reason");
inline constexpr auto previousCloseAgree = makeStr("previous_close_agree");
inline constexpr auto peerPositionsAtOpen = makeStr("peer_positions_at_open");
inline constexpr auto earlyCloseTriggered = makeStr("early_close_triggered");
/**
* Open-phase end metadata (set on consensus.phase.open before reset).
*
* `proposers_validated` counts validators of the previous ledger, unlike
* `proposers_finished` below, which counts those already past it.
*/
inline constexpr auto openDurationMs = makeStr("open_duration_ms");
inline constexpr auto peerPositionsAtClose = makeStr("peer_positions_at_close");
inline constexpr auto txSetsAcquired = makeStr("tx_sets_acquired");
inline constexpr auto closeReason = makeStr("close_reason");
inline constexpr auto proposersValidated = makeStr("proposers_validated");
/**
* Ledger-close inputs.
*/
inline constexpr auto txCountOpen = makeStr("tx_count_open");
/**
* Establish/check additional state.
*/
inline constexpr auto proposersFinished = makeStr("proposers_finished");
/**
* Establish-phase end metadata.
*
* The terminal close-time regime, set once. Qualified because DisputedTx
* tracks a SECOND, per-transaction avalanche; this is not that one. The
* derived `avalanche_threshold` cannot be inverted back to it.
*/
inline constexpr auto closeTimeAvalancheState = makeStr("close_time_avalanche_state");
/**
* Accept/apply enrichment.
*/
inline constexpr auto disputesResolvedCount = makeStr("disputes_resolved_count");
/**
* Validation send/receive enrichment. (`full_validation` is shared — see the
* `using` re-export above.)
*/
inline constexpr auto validationSignTime = makeStr("validation_sign_time");
/**
* "validation_status" — what the validation store did with an arriving
* validation, set on consensus.validation.accept. One of the bounded
* `ValStatus` names (see `val::status*`), so it is safe to aggregate: it is the
* difference between "validations are arriving and counting" and "they arrive
* and are all rejected", which on a stuck node look identical from the outside.
*/
inline constexpr auto validationStatus = makeStr("validation_status");
/**
* "accept_gated" — true when this validation did NOT reach the acceptance gate
* because another thread was already accepting the same ledger (the
* bypass-accept path). Two values, so it is aggregatable; without it the
* absence of a following ledger.validate span in the trace is unexplained.
*/
inline constexpr auto acceptGated = makeStr("accept_gated");
/**
* Receive-side hash prefixes for cross-peer correlation.
*/
inline constexpr auto prevLedgerPrefix = makeStr("prev_ledger_prefix");
inline constexpr auto positionHashPrefix = makeStr("position_hash_prefix");
/**
* "net_ledger_prefix" — first 16 hex chars of the ledger the network
* preferred when this node's differed. Paired with prev_ledger_prefix on
* the view.change event.
*/
inline constexpr auto netLedgerPrefix = makeStr("net_ledger_prefix");
/**
* "consensus_state" — domain-qualified (collides with other domains' state).
*/
inline constexpr auto consensusState = makeStr("consensus_state");
/**
* Close-time instants, both NetClock readings in whole seconds since the XRP
* Ledger epoch (2000-01-01T00:00:00Z) — see `closeTimeRippleEpochS` in
* SpanNames.h for why the epoch is spelled into the key.
*
* `parentCloseTimeRippleEpochS` is the previous ledger's close time;
* `closeTimeSelfRippleEpochS` is this node's own close-time vote for the round,
* so the pair shows how far the node's position sat from the ledger it built on.
*
* `closeTimeVoteBins` is not a time: it holds the number of distinct close-time
* positions seen from peers this round.
*/
inline constexpr auto parentCloseTimeRippleEpochS = makeStr("parent_close_time_ripple_epoch_s");
inline constexpr auto closeTimeSelfRippleEpochS = makeStr("close_time_self_ripple_epoch_s");
inline constexpr auto closeTimeVoteBins = makeStr("close_time_vote_bins");
inline constexpr auto resolutionDirection = makeStr("resolution_direction");
inline constexpr auto convergePercent = makeStr("converge_percent");
inline constexpr auto establishCount = makeStr("establish_count");
inline constexpr auto avalancheThreshold = makeStr("avalanche_threshold");
inline constexpr auto closeTimeThreshold = makeStr("close_time_threshold");
inline constexpr auto haveCloseTimeConsensus = makeStr("have_close_time_consensus");
inline constexpr auto agreeCount = makeStr("agree_count");
inline constexpr auto disagreeCount = makeStr("disagree_count");
inline constexpr auto thresholdPercent = makeStr("threshold_percent");
/**
* "consensus_result" — domain-qualified (collides with generic result).
*/
inline constexpr auto consensusResult = makeStr("consensus_result");
inline constexpr auto quorum = makeStr("quorum");
inline constexpr auto traceStrategy = makeStr("trace_strategy");
inline constexpr auto modeOld = makeStr("mode_old");
inline constexpr auto modeNew = makeStr("mode_new");
/**
* "is_bow_out" — whether this proposal is a bow-out (resigning from round).
*/
inline constexpr auto isBowOut = makeStr("is_bow_out");
/**
* Transaction/dispute attrs used in consensus accept spans.
*/
inline constexpr auto txId = makeStr("tx_id");
inline constexpr auto disputeOurVote = makeStr("dispute_our_vote");
inline constexpr auto disputeYays = makeStr("dispute_yays");
inline constexpr auto disputeNays = makeStr("dispute_nays");
inline constexpr auto txCount = makeStr("tx_count");
inline constexpr auto disputesCount = makeStr("disputes_count");
/**
* Trust flag (is the message origin a trusted UNL validator). Qualified by
* message type, shared with the peer.{proposal,validation}.receive spans:
* consensus.proposal.receive uses `proposal_trusted`, consensus.validation.
* receive uses `validation_trusted`. Same concept on both emitters → same key.
*/
inline constexpr auto proposalTrusted = makeStr("proposal_trusted");
inline constexpr auto validationTrusted = makeStr("validation_trusted");
/**
* "validation_receive_status" — which exit the inbound validation took on
* consensus.validation.receive. Set once per exit, so a dropped validation
* (microseconds) is separable from a queued one (job wait plus
* checkValidation); without it the span reports two unrelated latency
* distributions and every quantile over it is meaningless.
*
* Deliberately NOT `validation_status`: that key belongs to
* consensus.validation.accept and carries what the validation store did
* (`ValStatus`). One key with two value domains would make any aggregation
* that does not also filter on span name meaningless.
*/
inline constexpr auto validationReceiveStatus = makeStr("validation_receive_status");
} // namespace attr
// ===== Event names ===========================================================
namespace event {
/**
* "dispute.resolve"
*/
inline constexpr auto disputeResolve = join(makeStr("dispute"), makeStr("resolve"));
/**
* "tx.included" — one per transaction of the agreed consensus set, recorded
* before the ledger is built. A transaction that then fails to apply still
* has an event, so this is a superset of the accepted ledger's contents.
*/
inline constexpr auto txIncluded = join(makeStr("tx"), makeStr("included"));
/**
* Phase transition events — fired on consensus.round at each transition
* so the round-level span carries a complete timeline of phase changes,
* including the handleWrongLedger recovery edge that re-enters Open.
*/
inline constexpr auto phaseOpen = join(makeStr("phase"), makeStr("open"));
inline constexpr auto phaseEstablish = join(makeStr("phase"), makeStr("establish"));
inline constexpr auto phaseAccepted = join(makeStr("phase"), makeStr("accepted"));
inline constexpr auto phaseRecovery = join(makeStr("phase"), makeStr("recovery"));
/**
* Outcome events — fired on consensus.round at the establish→accepted
* transition so the path that drove acceptance is queryable.
*/
inline constexpr auto outcomeYes = join(makeStr("outcome"), makeStr("yes"));
inline constexpr auto outcomeMovedOn = join(makeStr("outcome"), makeStr("moved_on"));
inline constexpr auto outcomeExpired = join(makeStr("outcome"), makeStr("expired"));
/**
* "view.change" — fired on consensus.round when getPrevLedger sees the
* network's preferred ledger differ from ours. Carries prev_ledger_prefix
* and net_ledger_prefix so a single trace shows which ledger this node
* was on and which one the network preferred.
*/
inline constexpr auto viewChange = join(makeStr("view"), makeStr("change"));
} // namespace event
// ===== Attribute values ======================================================
namespace val {
inline constexpr auto finished = makeStr("finished");
inline constexpr auto movedOn = makeStr("moved_on");
inline constexpr auto yes = makeStr("yes");
inline constexpr auto no = makeStr("no");
inline constexpr auto expired = makeStr("expired");
inline constexpr auto increased = makeStr("increased");
inline constexpr auto decreased = makeStr("decreased");
inline constexpr auto unchanged = makeStr("unchanged");
// consensus_phase attribute values (the phase the round is entering).
inline constexpr auto phaseOpen = makeStr("open");
inline constexpr auto phaseEstablish = makeStr("establish");
inline constexpr auto phaseAccepted = makeStr("accepted");
/**
* validation_status values — the five `ValStatus` outcomes of adding an
* arriving validation to the validation store, plus a sentinel.
*
* Only `current` continues to the acceptance gate; every other value means the
* validation was counted for nothing, which is exactly the state a node stuck
* below quorum is in. Spelled here rather than reusing `to_string(ValStatus)`
* because that function returns camelCase ("badSeq"), and attribute values on
* an aggregated dimension must stay lower_snake_case.
*/
inline constexpr auto statusCurrent = makeStr("current");
inline constexpr auto statusStale = makeStr("stale");
inline constexpr auto statusBadSeq = makeStr("bad_seq");
inline constexpr auto statusMultiple = makeStr("multiple");
inline constexpr auto statusConflicting = makeStr("conflicting");
inline constexpr auto statusUnknown = makeStr("unknown");
// start_reason values (how startRoundInternal was entered).
inline constexpr auto startInitial = makeStr("initial");
inline constexpr auto startRecovered = makeStr("recovered");
// close_time_avalanche_state values, one per AvalancheState enumerator.
inline constexpr auto avalancheInit = makeStr("init");
inline constexpr auto avalancheMid = makeStr("mid");
inline constexpr auto avalancheLate = makeStr("late");
inline constexpr auto avalancheStuck = makeStr("stuck");
// Sentinel for an unmapped enumerator, matching to_string(ConsensusPhase).
inline constexpr auto unknown = makeStr("unknown");
// close_reason values, one per LedgerCloseReason enumerator. keep_open is
// never emitted: the attribute is only set on the path that closes.
inline constexpr auto closeKeepOpen = makeStr("keep_open");
inline constexpr auto closeAnomaly = makeStr("anomaly");
inline constexpr auto closeOthersClosed = makeStr("others_closed");
inline constexpr auto closeIdle = makeStr("idle");
inline constexpr auto closeNormal = makeStr("normal");
// validation_receive_status values, one per exit of the receive path.
inline constexpr auto validationQueued = makeStr("queued");
inline constexpr auto validationDroppedDiverged = makeStr("dropped_diverged");
inline constexpr auto validationDroppedLoad = makeStr("dropped_load");
} // namespace val
// ===== Value rules ===========================================================
/**
* Map a `ValStatus` to its `validation_status` attribute value.
*
* Takes a plain int rather than the enum so this header stays free of
* `Validations.h` (which pulls in the whole validation-store template) and so
* the rule can be asserted directly from the lib-only test binary, which cannot
* link xrpld. The caller passes `static_cast<int>(status)`; the enumerator order
* is fixed by `ValStatus` and asserted by the paired unit test, which is what
* keeps this mapping honest if a value is ever inserted.
*
* @param valStatus `ValStatus` as an int: 0 Current, 1 Stale, 2 BadSeq,
* 3 Multiple, 4 Conflicting.
* @return The matching `val::status*` value, or `unknown` for anything else.
*
* Example — the two readings that matter on a stuck node:
* @code
* validationStatusValue(0); // "current" -- counted, gate will be tried
* validationStatusValue(2); // "bad_seq" -- rejected, counted for nothing
* @endcode
*
* Example — edge case: an out-of-range value still yields a usable label rather
* than an empty attribute, so the dimension never gains a blank series:
* @code
* validationStatusValue(99); // "unknown"
* @endcode
*
* @note Pure and side-effect free; safe to call from any thread.
*/
[[nodiscard]] constexpr std::string_view
validationStatusValue(int valStatus) noexcept
{
switch (valStatus)
{
case 0:
return val::statusCurrent;
case 1:
return val::statusStale;
case 2:
return val::statusBadSeq;
case 3:
return val::statusMultiple;
case 4:
return val::statusConflicting;
default:
return val::statusUnknown;
}
}
} // namespace xrpl::telemetry::consensus::span