rippled/docker/telemetry/workload/expected_metrics.json

{
  "description": "Expected metric inventory for xrpld telemetry validation. Metric names have no prefix (the xrpld_ prefix was removed). beast::insight metrics are lowercased by formatName. Every name here was verified against its declaration in MetricsRegistry.cpp or include/xrpl/telemetry/GetObjectMetricNames.h and against a panel query under docker/telemetry/grafana/dashboards/. IMPORTANT: validate_telemetry.py has no notion of an optional metric \u2014 validate_metrics() iterates every group that has a \"metrics\" key and hard-fails any name with 0 Prometheus series after a 45 s poll. A metric is therefore listed only when the harness workload guarantees it will appear: observable gauges/counters whose callbacks Observe unconditionally (series exist at value 0), or push counters/histograms on a path every run exercises. Workload-gated and defect-gated names are recorded in the \"not_asserted\" group, which intentionally has no \"metrics\" key so the validator skips it. Only series existence is checked, never a value, except for the four bounds checks hardcoded in PARITY_VALUE_SANITY. A group may additionally declare \"required_labels\": each label there becomes one check that at least one of the group's series carries it with a non-empty value, matched as <label>!=\"\" because Prometheus cannot tell an absent label from an empty one. The same guarantee rule applies \u2014 list a label only where the workload guarantees it. Two conventions apply to the consumer evidence cited in the group descriptions below. First, only tracked files count: dashboards under docker/telemetry/grafana/dashboards/ and rules under docker/telemetry/grafana/provisioning/alerting/. The docker/telemetry/grafanacloud/ tree is gitignored (sourceCode/.gitignore:27) and has zero tracked files, so a cloud copy cannot be cited as evidence on a clean checkout and is not counted here. Second, every rule in provisioning/alerting/rules.yaml is provisioned with isPaused: true (14 of 14), so a metric feeding a rule is a wiring dependency, not a live pager: losing it leaves the rule wired to a signal that no longer resolves, and it starts silent rather than becoming silent. Finally, the top-level \"accounted_patterns\" list is not a group and asserts nothing. It declares anchored regexes for the bulk families that cannot usefully be enumerated as literal names \u2014 the per-job-type gauges and histograms, the overlay per-category traffic cross product \u2014 plus the Prometheus scrape plumbing that is not xrpld telemetry at all. Its only consumer is the reverse coverage check in validate_telemetry.py, which walks the metric families Prometheus actually holds and names every one that neither a group's \"metrics\" entry, nor a \"metrics_excluded\" key, nor one of these patterns accounts for. That check warns and never fails: downstream branches legitimately add telemetry this contract has not seen yet, and a hard failure would redden all of them. Its value is visibility, not enforcement.",
  "spanmetrics": {
    "description": "SpanMetrics-derived RED metrics from the OTel Collector spanmetrics connector.",
    "metrics": [
      "span_calls_total",
      "span_duration_milliseconds_bucket",
      "span_duration_milliseconds_count",
      "span_duration_milliseconds_sum"
    ],
    "required_labels": [
      "span_name",
      "status_code",
      "service_name",
      "span_kind"
    ],
    "dimension_labels": [
      "command",
      "rpc_status",
      "consensus_mode",
      "local",
      "proposal_trusted",
      "validation_trusted",
      "tx_type",
      "ter_result",
      "stage",
      "txq_status",
      "close_time_correct",
      "consensus_state"
    ],
    "_dimension_labels_note": "Bare label names as configured in otel-collector-config.yaml spanmetrics dimensions. Informational only (not asserted by the validator)."
  },
  "statsd_gauges": {
    "description": "beast::insight gauges exported via OTLP/HTTP to the collector (server=otel). What guarantees these is the export mechanism, not the workload. On the OTel path a beast gauge is an Int64ObservableGauge (OTelCollector.cpp:703); every OTelGaugeImpl registers itself with the collector in its own constructor (:690); onCollectionReady() arms every registered gauge with no per-metric condition (:941-979); and the armed callback Observes currentValue() on each export cycle regardless of the value and regardless of whether set() was ever called (:719-727). Series therefore exist from the first export, at 0 if nothing has happened. The 'gauges only mark dirty on value changes' caveat in _poll_series_count is a property of the StatsD backend and does not apply here. The consequence is that a beast gauge whose object is constructed on an unconditional startup path is as safe to assert as an observable gauge from MetricsRegistry \u2014 provided the object is constructed before Application.cpp:1570, which is where onCollectionReady() runs. That precondition is load-bearing and is the one way a gauge on this path can still go missing. Registration and arming are separate steps: addGauge() (:927-931) only appends to gauges_ under the lock, and onCollectionReady() snapshots that vector and calls arm() on each entry, exactly once per process (Application.cpp:1570, on the setup path, with no second call site). A gauge whose owner is built after that line is registered, never armed, and never exported \u2014 no callback, no series, and no error. Every gauge asserted in this group clears the precondition: jobQueue_, nodeFamily_, ledgerMaster_ and networkOPs_ are all in ApplicationImp's member-init list (Application.cpp:377, :430, :439, :465), and overlay_ \u2014 which owns peerFinder_ in its own member-init list (OverlayImpl.cpp:185) and Stats::peerDisconnects (OverlayImpl.h:603) \u2014 is built at Application.cpp:1551, nineteen lines before the arming call. The comment at Application.cpp:1258-1272 states that ordering deliberately: overlay_ is the last service the callbacks read, which is why arming waits for it. This is also why the state_accounting family below is asserted whole rather than one member deep. The ten state_accounting_* names are one family with one guarantee. NetworkOPsImp::Stats creates all ten in the NetworkOPsImp constructor (NetworkOPs.cpp:1069-1080; makeGauge(\"State_Accounting\", \"Full_duration\") joins on '.' at Collector.h:151-155 and formatName lowercases it, giving state_accounting_full_duration), and NetworkOPsImp::collectMetrics() set()s all ten unconditionally in a single block (:5204-5231). No state has to be entered for its pair to exist: a node that never disconnects still publishes state_accounting_disconnected_duration and _transitions at 0. All ten appear on the node-health dashboard, and state_accounting_full_transitions is also the input to the NodeStateFlapping alert rule (provisioning/alerting/rules.yaml:645-649, uid xrpld-node-state-flapping \u2014 provisioned isPaused: true, so this is a wiring dependency and not a live pager). Asserting only full_duration left the other nine unchecked, the alert's own input among them, even though a regression could not plausibly drop one sibling without dropping full_duration too. node_family_full_below_cache_hit_rate and _size are the TaggedCache Stats pair (TaggedCache.h:282-283), created in the cache constructor (TaggedCache.ipp:63-66) and published by collectMetrics(), which reports a hit_rate of 0 rather than skipping the gauge when hits+misses is 0 (TaggedCache.ipp:751-765). The instance is NodeFamily's full-below cache, constructed with the real collector in NodeFamily's constructor (NodeFamily.cpp:28-35), which every node builds. Both are on node-health. overlay_peer_disconnects is the first member of OverlayImpl::Stats (OverlayImpl.h:603) and OverlayImpl::collectMetrics() assigns it getPeerDisconnect() unconditionally (:646), in the same hook that publishes the overlay_traffic group's gauges. Its only query consumer is network-traffic.json:148. It also appears once in log-derived-insights.json:1908, but that occurrence is inside a panel description which explicitly disclaims it ('overlay_peer_disconnects exists as a metric but carries no reason breakdown, which is what this panel adds') \u2014 that panel is Loki-derived and queries nothing from Prometheus, so it is not a consumer. The third occurrence, validate_dashboards.py:17, is a lint rule listing the name as a cumulative gauge that must be rate()-wrapped; also not a query consumer. It is listed here rather than in overlay_traffic because that group's contract is an explicit subset of the per-category traffic family, and a disconnect count is not one of those categories.",
    "metrics": [
      "ledgermaster_validated_ledger_age",
      "ledgermaster_published_ledger_age",
      "state_accounting_full_duration",
      "peer_finder_active_inbound_peers",
      "peer_finder_active_outbound_peers",
      "jobq_job_count",
      "state_accounting_connected_duration",
      "state_accounting_connected_transitions",
      "state_accounting_disconnected_duration",
      "state_accounting_disconnected_transitions",
      "state_accounting_full_transitions",
      "state_accounting_syncing_duration",
      "state_accounting_syncing_transitions",
      "state_accounting_tracking_duration",
      "state_accounting_tracking_transitions",
      "node_family_full_below_cache_hit_rate",
      "node_family_full_below_cache_size",
      "overlay_peer_disconnects"
    ]
  },
  "statsd_counters": {
    "description": "beast::insight counters exported via OTLP/HTTP. The OTel Prometheus exporter appends _total to monotonic counters.",
    "metrics": ["rpc_requests_total", "ledger_fetches_total"]
  },
  "io_latency": {
    "description": "io_context scheduling-latency histogram \u2014 a beast::insight Event exported over OTLP (Application.cpp:515, makeEvent('ios_latency') with the default millisecond unit). It is the one beast Event the node itself guarantees, which is why it is asserted while rpc_time_milliseconds and the jobq_* pairs sit in not_asserted: the sampler starts on the unconditional startup path (Application.cpp:1697, outside the 'if (withTimers)' guard), and its handler always emits the first sample whatever its value (Application.cpp:185-189, 'firstSample_.exchange(false) || lastSample >= 10ms', with a comment stating the point is to register the metric downstream). The OTel histogram is cumulative, so that one notify creates series that persist for the rest of the run. Named with the _bucket/_count/_sum suffixes the Prometheus exporter emits for a histogram \u2014 no bare ios_latency_milliseconds series exists, the same convention rpc_method_us and span_duration_milliseconds follow, and every consumer queries ios_latency_milliseconds_bucket: 2 distinct panels (ledger-data-sync 'I/O Scheduler Latency p95' and node-health 'I/O Latency') plus one alert rule (grafana/provisioning/alerting/rules.yaml:586). Only existence is asserted: past the first sample the handler reports only latencies >= 10 ms, so neither the sample count nor the value is predictable.",
    "metrics": [
      "ios_latency_milliseconds_bucket",
      "ios_latency_milliseconds_count",
      "ios_latency_milliseconds_sum"
    ]
  },
  "overlay_traffic": {
    "description": "Overlay traffic metrics (subset \u2014 full list has 45+ categories).",
    "metrics": [
      "total_bytes_in",
      "total_bytes_out",
      "total_messages_in",
      "total_messages_out"
    ]
  },
  "nodestore_io": {
    "description": "NodeStore I/O observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label distinguishing sub-metrics.",
    "metrics": ["nodestore_state"]
  },
  "cache_hit_rates": {
    "description": "Cache hit rate observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label.",
    "metrics": ["cache_metrics"]
  },
  "transaction_queue": {
    "description": "Transaction queue observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label.",
    "metrics": ["txq_metrics"]
  },
  "rpc_method_detail": {
    "description": "Per-RPC-method counters and duration histogram (MetricsRegistry.cpp:351-357). rpc_method_errored_total is deliberately absent \u2014 see not_asserted below. rpc_method_us is a Histogram, so the Prometheus exporter emits only the _bucket/_count/_sum triple and there is no bare rpc_method_us series to match \u2014 same convention as span_duration_milliseconds in the spanmetrics group above.",
    "metrics": [
      "rpc_method_started_total",
      "rpc_method_finished_total",
      "rpc_method_us_bucket",
      "rpc_method_us_count",
      "rpc_method_us_sum"
    ]
  },
  "job_queue": {
    "description": "Job-queue counters and latency histograms (MetricsRegistry.cpp:360-366). Every xrpld job passes through these, so they populate under any workload. Both histograms are recorded in the same function bodies as job_started_total / job_finished_total, under the same guard and with the same labels, so their presence is equally guaranteed. They are named with the _bucket/_count/_sum suffixes the Prometheus exporter emits: regression-metrics.json and the job-queue dashboard both query job_queued_us_bucket / job_running_us_bucket, and no bare series exists. This group also carries the xrpl_node_id required_labels assertion. 070d29b465 added the xrpl.node.id resource attribute so Grafana Cloud trace ingest would stop folding distinct nodes into one \u2014 ingest groups ResourceSpans but ignores service.instance.id, so before the attribute existed the boards reported one node's ledger.build five times over and saw 2 of 9 nodes. Nothing asserted it, so a regression would silently restore the folding. It is asserted here rather than on a statsd_* group because these metrics are MetricsRegistry-backed and guaranteed by any workload, whereas the beast::insight meter provider is constructed before the wallet DB exists and so has no node public key to stamp: those metrics legitimately lack the label and 070d29b465 omits it rather than writing it blank.",
    "required_labels": ["xrpl_node_id"],
    "metrics": [
      "job_queued_total",
      "job_started_total",
      "job_finished_total",
      "job_queued_us_bucket",
      "job_queued_us_count",
      "job_queued_us_sum",
      "job_running_us_bucket",
      "job_running_us_count",
      "job_running_us_sum"
    ]
  },
  "job_queue_per_type_gauges": {
    "description": "Per-job-type queue-depth gauges from the beast::insight \"jobq\" group, for the six job types the ledger-sync diagnostics single out. JobTypeData's constructor creates waiting/running/deferred for every non-special job type (JobTypeData.h:96-103), the JobQueue constructor eagerly constructs a JobTypeData for every type in JobTypes (JobQueue.cpp:45-53), and JobQueue::collect() assigns all three from one snapshot for every type on every hook cycle, with no per-type condition (JobQueue.cpp:104-106). The collector is the \"jobq\" group, so GroupImp::makeName() prefixes \"jobq.\" and formatName lowercases the job type: JtTxnData's \"fetchTxnData\" + \"_running\" exports as jobq_fetchtxndata_running (JobTypeData.h:29-37). Existence rests on the same ObservableGauge mechanism documented on statsd_gauges, so these series exist at 0 on an idle node. All six were present in the CI validation run's own emitted-metric list, which is the strongest evidence behind any entry in this file. All six job types are non-special: makeFetchPack (limit 1), manifest (maxLimit), ledgerRequest (3), ledgerData (3), updatePaths (1) and fetchTxnData (5) all declare a non-zero limit in JobTypes.h, so none is skipped by the special() guard in JobTypeData's constructor. The 11 special types (limit 0) get no gauges at all, which is why no jobq_pathfind_* or jobq_peercommand_* name can be asserted. Why these six and not all 105. The guarantee is identical for every one of the 35 non-special types x 3 states, so the discriminator here is consumer coverage rather than emission: these six are the only per-type names a dashboard or an alert queries literally, so losing one blanks a specific panel or leaves a rule wired to a signal that no longer resolves. jobq_manifest_waiting is the input to the ManifestJobQueueConvoy alert rule (provisioning/alerting/rules.yaml:796-800, uid xrpld-manifest-job-convoy \u2014 provisioned isPaused: true like every rule in that file, so the consequence is a broken wire rather than a silenced pager); the other five are per-type saturation panels on node-health.json:5267-5291. The rest of the family is reached only through two regex queries, both on ledger-data-sync: __name__=~\"jobq_.*_deferred\" at :1701 and __name__=~\"jobq_.*_waiting\" at :1708. Those are the only two jobq regexes anywhere in the tree \u2014 there is no jobq_.*_running regex \u2014 so the 30 unasserted _running gauges have zero consumers of any kind, and the unasserted _waiting and _deferred gauges are reached only by a topk that keeps working as long as the family exists and depends on no single job type. Consumer coverage is the whole argument and it stands alone. An earlier revision also cited poll bandwidth; that reason was wrong and has been removed, because METRIC_POLL_CONCURRENCY is 8 and the semaphore is released across each sleep, so 105 targets cost on the order of a second against the shared 45 s deadline. A bogus cost argument would be a bad reason to refuse a legitimate expansion later: if a panel or rule starts naming a seventh job type, add it.",
    "metrics": [
      "jobq_fetchtxndata_running",
      "jobq_ledgerdata_running",
      "jobq_ledgerrequest_running",
      "jobq_makefetchpack_running",
      "jobq_manifest_waiting",
      "jobq_updatepaths_running"
    ]
  },
  "rpc_in_flight": {
    "description": "In-flight RPC gauge via the XRPL_METRIC_UPDOWN_ADD call-site macro (PerfLogImp.cpp, +1 rpcStart / -1 rpcEnd). UpDownCounter: no _total suffix.",
    "metrics": ["rpc_in_flight_requests"]
  },
  "object_counts": {
    "description": "Counted object instances observable gauge (MetricsRegistry via OTLP).",
    "metrics": ["object_count"]
  },
  "load_factors": {
    "description": "Fee escalation and load factor observable gauge (MetricsRegistry via OTLP).",
    "metrics": ["load_factor_metrics"]
  },
  "parity_validation_agreement": {
    "description": "External dashboard parity: validation agreement percentages (MetricsRegistry).",
    "metrics": [
      "validation_agreement{metric=\"agreement_pct_1h\"}",
      "validation_agreement{metric=\"agreement_pct_24h\"}"
    ]
  },
  "parity_validator_health": {
    "description": "External dashboard parity: validator health indicators (MetricsRegistry).",
    "metrics": [
      "validator_health{metric=\"amendment_blocked\"}",
      "validator_health{metric=\"unl_expiry_days\"}"
    ]
  },
  "parity_peer_quality": {
    "description": "External dashboard parity: peer quality metrics (MetricsRegistry).",
    "metrics": [
      "peer_quality{metric=\"peer_latency_p90_ms\"}",
      "peer_quality{metric=\"peers_insane_count\"}"
    ]
  },
  "parity_ledger_economy": {
    "description": "External dashboard parity: ledger economy metrics (MetricsRegistry.cpp:1401). transaction_rate is observed on every export, in both branches of the ledger-age test (MetricsRegistry.cpp:1444-1451). base_fee_xrp is observed only inside the 'if (ledger)' guard on getValidatedLedger() (MetricsRegistry.cpp:1418-1423), and that returns validLedger_ (LedgerMaster.cpp:1569-1572), which stays null until a ledger validates \u2014 the same precondition complete_ledgers has. Both are asserted because run-full-validation.sh waits for a validated ledger before running the workload. base_fee_xrp absent while transaction_rate is present is the signature of a cluster that never validated, not of a missing metric.",
    "metrics": [
      "ledger_economy{metric=\"base_fee_xrp\"}",
      "ledger_economy{metric=\"transaction_rate\"}"
    ]
  },
  "parity_state_tracking": {
    "description": "External dashboard parity: server state tracking (MetricsRegistry).",
    "metrics": ["state_tracking{metric=\"state_value\"}"]
  },
  "parity_counters": {
    "description": "External dashboard parity: monotonic counters (MetricsRegistry). validations_checked_total is incremented unconditionally at the top of NetworkOPsImp::recvValidation (NetworkOPs.cpp:2681), and run-full-validation.sh brings up a 5-node validator cluster, so inbound validations are guaranteed.",
    "metrics": [
      "ledgers_closed_total",
      "validations_sent_total",
      "validations_checked_total",
      "state_changes_total"
    ]
  },
  "parity_storage": {
    "description": "External dashboard parity: storage detail metrics (MetricsRegistry).",
    "metrics": ["storage_detail{metric=\"stored_object_bytes\"}"]
  },
  "sync_diagnostics": {
    "description": "Fresh-node sync diagnostics (native metrics). Bootstrap (Domain 0) and acquire-pipeline signals rendered by the ledger-sync-health dashboard. Names are appended one per signal. Histograms are listed by all three of their Prometheus series -- _bucket, _count and _sum -- because the bare instrument name is not a series and an exporter regression can drop one without dropping the others; this matches the rpc_method_us / job_queued_us convention in the groups above. Observable gauges carry an inline metric= selector so the specific sub-series is asserted, matching the parity_* groups. Signals this workload cannot produce are NOT listed here: they belong in the not_asserted group, whose metrics_excluded map is the single machine-readable record of every emitted-but-unasserted name.",
    "metrics": [
      "dns_resolve_total",
      "dns_resolve_latency_ms_bucket",
      "dns_resolve_latency_ms_count",
      "dns_resolve_latency_ms_sum",
      "overlay_connect_total",
      "overlay_dial_latency_ms_bucket",
      "overlay_dial_latency_ms_count",
      "overlay_dial_latency_ms_sum",
      "unl_quorum{metric=\"trusted_keys\"}",
      "unl_quorum{metric=\"quorum\"}",
      "clock_close_offset_seconds{metric=\"offset\"}",
      "sync_state{metric=\"initial_full_duration_us\"}",
      "sync_state{metric=\"network_ledger_gate\"}",
      "sync_state{metric=\"server_stall_seconds\"}",
      "sync_state{metric=\"ledgers_behind\"}",
      "server_stall_events_total",
      "state_changes_total{from!=\"\",to!=\"\"}",
      "sync_acquire{metric=\"missing_state_nodes_max\"}",
      "sync_acquire{metric=\"missing_tx_nodes_max\"}",
      "sync_acquire{metric=\"received_data_depth\"}",
      "sync_acquire{metric=\"in_flight\"}",
      "shamap_cache_hit_rate{metric=\"treenode\"}",
      "jobq_saturation{metric=\"running_tasks\"}",
      "jobq_saturation{metric=\"worker_threads\"}",
      "jobq_saturation{metric=\"total_waiting\"}",
      "peer_ledger_supply{metric=\"peers_reporting\"}",
      "peer_ledger_supply{metric=\"peers_serving_validated\"}",
      "peer_ledger_supply{metric=\"peers_serving_next\"}",
      "peer_ledger_supply{metric=\"supply_min_seq\"}",
      "peer_ledger_supply{metric=\"supply_max_seq\"}",
      "peerfinder_slot_census{metric=\"out_active\"}",
      "peerfinder_slot_census{metric=\"out_max\"}",
      "peerfinder_slot_census{metric=\"in_active\"}",
      "peerfinder_slot_census{metric=\"in_max\"}",
      "peerfinder_slot_census{metric=\"connecting\"}",
      "peerfinder_slot_census{metric=\"fixed_configured\"}",
      "peerfinder_slot_census{metric=\"fixed_active\"}",
      "peerfinder_slot_census{metric=\"bootcache\"}",
      "peerfinder_slot_census{metric=\"livecache\"}",
      "amendment_block{metric=\"warned\"}",
      "amendment_block{metric=\"seconds_to_block\"}",
      "peer_accept_total",
      "nodestore_state{metric=\"node_writes\"}",
      "nodestore_state{metric=\"node_reads_total\"}",
      "nodestore_state{metric=\"read_mean_us\"}",
      "ledger_quorum_publish{metric=\"trusted_validation_tally\"}",
      "ledger_quorum_publish{metric=\"quorum_target\"}",
      "ledger_quorum_publish{metric=\"time_to_first_validated_us\"}",
      "ledger_quorum_publish{metric=\"publish_lag\"}",
      "ledger_quorum_shortfall_total{stage=\"pre_accept\"}",
      "consensus_round_duration_ms_bucket",
      "consensus_round_duration_ms_count",
      "consensus_round_duration_ms_sum",
      "nodestore_state{metric=\"node_writes_duration_us\"}",
      "nodestore_state{metric=\"node_reads_duration_us\"}",
      "unl_quorum{metric=\"quorum_disabled\"}",
      "sweep_malloc_trim_us_bucket",
      "sweep_malloc_trim_us_count",
      "sweep_malloc_trim_us_sum",
      "rotation_state{metric=\"in_flight\"}",
      "rotation_state{metric=\"copy_forward\"}",
      "cache_metrics{metric=\"treenode_lock_hold_peak_us\"}",
      "cache_metrics{metric=\"fullbelow_lock_hold_peak_us\"}"
    ],
    "_acquire_note": "The four sync_acquire sub-series and shamap_cache_hit_rate are unconditional: both are observable gauges whose callbacks observe every series on each collection tick, so each is present even when the value is 0 (an idle node reports in_flight=0 and missing_state_nodes_max=0, and a cold cache reports a 0.0 hit rate). Absence, not a zero, is the regression. The three InboundLedger acquire counters (sync_acquire_source_total, sync_addnode_total, sync_acquire_no_progress_total) are deliberately NOT asserted here: all three are emitted only from InboundLedger, which runs only when a node must fetch a ledger it lacks. expected_spans.json already marks the ledger.acquire span optional for exactly this reason (\"A healthy local cluster rarely back-fills history\"), and the metric validator has no per-metric optional flag, so listing them would fail the harness red on a healthy run. They are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp and by the ledger-sync-health panels; add them here only alongside a harness step that forces a real acquire (e.g. starting a node against an existing ledger history).",
    "_jobq_note": "The three jobq_saturation series are unconditional: it is an observable gauge whose callback observes all three fields on every collection tick, so each series exists even when the value is 0, and absence rather than a zero is the regression. worker_threads is asserted because it is the denominator of the dashboard saturation ratio, and it is always at least 1 (the JobQueue ctor gives standalone mode exactly one worker), so a zero or missing reading there means the accessor regressed rather than the node being idle. The per-job-type waiting/running/deferred counts are published separately by JobQueue::collect() as the beast::insight gauges jobq_<type>_waiting / _running / _deferred, which the collector translates; they are covered by the StatsD-derived groups, not here.",
    "_conditional_note": "handshake_negotiation_fail_total and unl_fetch_total used to be asserted here, with a note saying to move them out if a run ever reported 0 series. Both have now moved to not_asserted.metrics_excluded on CODE REACHABILITY rather than on any run: unl_fetch_total is emitted only from a site indexed out of ValidatorSite's sites_, which this harness leaves empty because it writes a static [validators] file and configures no list site at all. That argument does not depend on observing a run, which matters, because for a stretch of this branch's history no run could have observed anything here -- see _gate_history_note. Anything added to this group must be reachable under run-full-validation.sh's own cluster and workload, not merely emitted somewhere in the code.",
    "_gate_history_note": "Why this group's contents were never validated by a run, recorded because it explains why several notes here rest on code reading rather than on observed output, and because the same failure mode can recur. assert_sync_diagnostics_metrics arrived on 2026-07-24 (96914b9f40) calling _check_prometheus_metric with five positional arguments, which was CORRECT then: the function took (session, prometheus_url, metric_name, category, report), returned None and recorded the result itself. On 2026-08-14 phase-10 gave that function a shared deadline and a semaphore (d059f21bf3), updating its own caller; 96914b9f40 is not an ancestor of that commit, so it never saw this one. The merge that first contained both, 7c70e142e9 on 2026-08-21, was CLEAN -- the two edits sit in different regions of the file -- and produced a caller that no longer matches the signature it calls. From then until the fix, every invocation raised TypeError before the first query, so the whole group asserted nothing and took the validation phases ordered after it down with it. Nothing could have caught this: it is a semantic conflict, invisible to git because neither side's text overlaps, and Python has no compile step to reject the arity. The general lesson for this harness is that a merge reported as clean is not evidence that a cross-branch caller still matches its callee -- after any merge that touches this file, re-check the call sites, not just the diff.",
    "_sync_state_note": "The four sync_state sub-series are unconditional: the gauge observes all four on every collection tick, so each is present as a series even when its value is 0 (a node that never reached FULL reports initial_full_duration_us=0, and a healthy node reports server_stall_seconds=0). The check asserts series presence, not a non-zero value, which is exactly right here \u2014 a zero is a meaningful reading for these signals, and absence is the regression. server_stall_events_total is likewise always present because the observable counter reports the tally (0 or more) every tick. state_changes_total is asserted here with a from!=\"\",to!=\"\" selector rather than bare (parity_counters already asserts the bare name): the selector is what proves the {from,to} label dimension actually reached Prometheus, so an unlabelled counter fails this check instead of silently passing on the bare name. It needs at least one real mode transition, which any node reaching connected/syncing produces during startup.",
    "_a7_note": "The peer-supply, slot-census and amendment-countdown signals are three observable gauges and four counters. The 16 gauge sub-series (peer_ledger_supply, peerfinder_slot_census, amendment_block) are unconditional and asserted individually: each callback in MetricsRegistry.cpp calls observe() for every field on every collection tick with no early return between them, so the series exists whatever the value. That includes the two sentinel readings \u2014 a node whose peers have advertised nothing reports peer_ledger_supply{metric=\"supply_min_seq\"} = 0 meaning unknown, and a node with no pending amendment reports amendment_block{metric=\"seconds_to_block\"} = -1 meaning healthy. Absence, not the sentinel, is the regression. Of the four counters only peer_accept_total is asserted: run-full-validation.sh gives every node a [port_peer] on 0.0.0.0 and lists the other four nodes in [ips], so all 5 nodes dial each other and each one is also dialled, which means OverlayImpl::onHandoff runs and reports outcome=accepted (or slot_refused/no_slot on the duplicate half of each mutual dial) on every node. It is asserted bare rather than with an outcome= selector because which outcome a given node records depends on dial ordering, which the harness does not control. The other three counters are deliberately NOT asserted. peer_disconnect_total is emitted only from PeerImp::close, and a healthy 5-node localhost cluster holds its 4 fixed peers for the whole run: the timer-driven reasons need maxUnknownTime (600 s) or maxDivergedTime (300 s) to elapse (Config.h) while the full-validation profile totals well under that, and the shutdown reasons only fire during teardown, which happens in run-full-validation.sh after Step 5 has already scraped. serve_refused_total needs a peer to ask this node for a ledger, tx set or object it cannot serve \u2014 on a cluster where every node has the same complete history from genesis, getLedger()/getTxSet() succeed and the send queues never approach Tuning::kDropSendQueue. ledger_jump_total needs NetworkOPsImp::switchLastClosedLedger, reached only when consensus reports an LCL this node did not build on; a healthy 5-node cluster agrees every round, so it never jumps. The metric validator has no per-metric optional flag, so listing any of the three would fail the harness red on a healthy run \u2014 the same reasoning _acquire_note applies to the InboundLedger acquire counters. All four counters are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp and rendered by the ledger-sync-health panels Peer Disconnects (Count By Reason & Direction), Ledger/Object Serve Refusals and Byzantine Ledger Jumps, and by the peer-quality panels Peer Disconnect Rate and Peer Disconnects By Reason & Direction. To make them assertable the harness would need a fault-injection step: kill one node mid-run and re-scrape before teardown (peer_disconnect_total, reason=read_error/graceful), request a ledger sequence outside the cluster's history or drive a node past its send-queue limit (serve_refused_total), and start a node on a divergent chain tip or partition the cluster and heal it (ledger_jump_total).",
    "_a6_note": "There is deliberately no separate nodestore_latency gauge, because every one of its sub-metrics had an exact counterpart on nodestore_state computed from the same Database accessor, so the five entries above are the surviving equivalents (write_count -> node_writes, read_count -> node_reads_total, write_duration_us -> node_writes_duration_us, read_duration_us -> node_reads_duration_us, read_mean_us unchanged). All four totals are unconditional: MetricsRegistry::observeNodeStoreTotals observes them on every collection tick with no early return before them, so a series exists whatever the value and a node that has written nothing reports node_writes=0 rather than dropping the series. read_mean_us is safe because any node that has opened a ledger has already fetched objects, so the fetch count is non-zero. write_mean_us is NOT asserted, but only because the two means are the sub-series that scaledMean() omits when their denominator is zero, and this validator has no per-metric optional flag -- not because the numerator is missing. That older caveat is gone: all three concrete store paths now time themselves through Database::recordStoreDuration() (DatabaseNodeImp::store, DatabaseRotatingImp::store and Database::importInternal), so write_mean_us is live on an ordinary node and only a node that has performed literally zero stores would lack it. It can be promoted to an assertion once a harness run confirms it present. The two replay counters (ledger_replay_fallback_total, ledger_replay_outcome_total) are likewise NOT asserted, for the same reason _acquire_note gives for the InboundLedger acquire counters: both are emitted only from the ledger-replay path, which requires the [ledger_replay] config stanza AND a real historical back-fill against peers that support the LedgerReplay protocol feature. run-full-validation.sh starts a fresh local cluster with no history to back-fill, so no replay task is ever created and neither counter can produce a series. Both are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp and rendered by the ledger-sync-health panels Replay Fallback to Full Acquire and Replay Outcomes. To make them assertable the harness would need to enable [ledger_replay] and start a node against an existing ledger history so it back-fills through the replay path.",
    "_a5_note": "The quorum-and-publish signals are one observable gauge (ledger_quorum_publish) and one counter (ledger_quorum_shortfall_total). All four gauge sub-series are asserted and are unconditional: registerLedgerQuorumPublishGauge's callback in MetricsRegistry.cpp calls observe() for every field on every collection tick with no early return between them, and each accessor is a plain relaxed atomic load that always returns a value, so the series exists whatever the reading. That deliberately includes the three diagnostic zeros: a node that has never had a gate evaluated reports trusted_validation_tally=0 and quorum_target=0, one that has never fully validated reports time_to_first_validated_us=0, and one that is caught up reports publish_lag=0. Absence, not the zero, is the regression -- the same reasoning _sync_state_note gives for initial_full_duration_us. Note the sentinel: when the trusted list disables quorum entirely, getNeededValidations() returns SIZE_MAX and LedgerMaster reports quorum_target as int64 max rather than letting the cast wrap to -1, so the target reads far above any tally instead of inverting the comparison (the same fix as the unl_quorum gauge). ledger_quorum_shortfall_total IS asserted, which differs from the acquire, replay and peer counters, and the reason is that this counter does not need a fault to fire. RCLConsensus::Adaptor::doAccept issues this node's own validation and then calls ledgerMaster_.consensusBuilt immediately (RCLConsensus.cpp), which calls checkAccept on the freshly built ledger (LedgerMaster.cpp) BEFORE the peers' validations for that same ledger have arrived. With the harness's 5 validators the quorum is max(ceil(5*0.8), ceil(5*0.6)) = 4 (ValidatorList::calculateQuorum), so that first evaluation of each round tallies short of 4 and takes the shortfall early return; the gate is then re-entered from RCLValidations handleNewValidation as each trusted validation arrives and eventually passes. A HEALTHY 5-node cluster therefore emits this counter every round, which is why it is safe to assert on a clean run -- unlike peer_disconnect_total or ledger_replay_fallback_total, it needs no fault injection, no [ledger_replay] stanza and no historical back-fill. The stage=\"pre_accept\" selector is asserted rather than the bare name so that the label dimension is proven to have reached Prometheus, matching the state_changes_total{from,to} pattern. Consequence for readers of the panels: a non-zero rate on Pre-Accept Quorum Shortfall Rate is NOT by itself a fault, and the panel description says so; the fault signature is that rate climbing well above the ledger-close rate while the tally on Trusted Validations vs Quorum Target stays flat below its target. If a future harness change makes the cluster single-node or standalone this assertion must move to a note: standalone_ short-circuits consensusBuilt before checkAccept, and getNeededValidations() returns 0 in standalone mode, so the gate can never report a shortfall.",
    "_b5_sweep_note": "The per-sweep malloc_trim signals are one histogram and two counters. Only the histogram is asserted, by all three of its Prometheus series -- _bucket, _count and _sum -- because the bare instrument name is not a series. It is unconditional on any running node: ApplicationImp::start arms the sweep timer before the workload begins, the interval is SizedItem::SweepInterval (Config.cpp: 10 s at nodeSize 0 through 120 s at nodeSize 4), and the full-validation profile runs 270 s of workload before Step 5 scrapes -- so at least two sweeps complete even in the slowest case, and each one records exactly one sample. The measurement itself is now unconditional too: it used to sit inside `if (journal.debug())` in MallocTrim.cpp, so a node at ordinary log level measured nothing; that gate now covers only the JLOG. The two counters are deliberately NOT asserted. sweep_malloc_trim_minor_faults_total is emitted only when the trim's minor-fault delta is greater than zero, and a trim on the small heap of a fresh localhost node routinely faults zero times -- the emit site publishes nothing rather than a zero, because a zero-valued series would claim the trim was measured as free when the honest statement is that there was nothing to fault on. sweep_malloc_trim_reclaimed_kb_total is emitted only when resident memory actually FELL across the trim, which on a node whose caches are still filling frequently does not happen (glibc has nothing above the top of the heap to release, and mmap-backed allocations are returned on free regardless of trimming). The validator has no per-metric optional flag, so asserting either would ship a permanently red CI check for a healthy run. Both are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp -- including the skip paths -- and rendered by the ledger-sync-health panel Sweep Heap-Trim Faults & Reclaim Rate. To make them assertable the harness would need a node with a large enough resident heap for a trim to reclaim, e.g. starting against an existing populated database rather than from genesis.",
    "_b5_rotation_note": "The online_delete rotation-write signals are one observable gauge (rotation_state, sub-series in_flight and copy_forward) and one counter (rotation_copy_node_restore_total). BOTH gauge sub-series ARE asserted; only the counter is excluded. An earlier version of this note claimed none of them could be produced by the harness, which conflated 'no rotation runs' with 'no series is published'. Those are different facts: registerRotationStateGauge's callback gates ONLY on dynamic_cast<DatabaseRotating*>, and both xrpld-validator.cfg.template:48 and run-full-validation.sh:300 set online_delete=256, so SHAMapStoreImp::makeNodeStore takes its deleteInterval_ != 0 branch and builds a DatabaseRotatingImp -- the cast succeeds and both observe() calls then run on every collection tick with no early return between them. So each series exists at value 0 (in_flight=0 meaning no rotation in progress, copy_forward=0 meaning none has yet copied anything), and absence rather than the zero is the regression, exactly as for the sync_state and peer_ledger_supply gauges. What the harness genuinely cannot produce is a rotation ACTUALLY RUNNING; that bounds the values, not the series. Two independent reasons, and note that the mechanism sentence further down describes the CALLBACK's early return, not the instrument -- the instrument itself is created eagerly and unconditionally at MetricsRegistry.cpp:1116, so it always exists. First, no rotation ever runs: xrpld-validator.cfg.template sets online_delete=256 and does not set advisory_delete, so SHAMapStoreImp's gate is validatedSeq >= lastRotated + 256 (SHAMapStoreImp.cpp), which needs 256 validated ledgers; at the network's several-seconds-per-ledger close rate that is on the order of 15-20 minutes, while the full-validation profile totals 270 s of workload before Step 5 scrapes. Second, even the in_flight flag needs a rotation to have started, and copy_forward additionally needs an ARCHIVE holding data that a fetch actually reads during the rotation window -- which requires a populated, already-rotated database, exactly the condition the hypothesis says is why this slowdown never appears on a fresh node. rotation_copy_node_restore_total is narrower still: it fires only for a clean tree node reachable from the validated state map whose sole on-disk copy was removed by an EARLIER rotation, so it needs at least two rotations plus real prior data loss. Note that rotation_state publishes no series at all when online_delete is not configured, by design: the gauge's OBSERVE CALLBACK dynamic_casts the node store to DatabaseRotating and returns early on failure (MetricsRegistry.cpp:1132-1133), so an absent series means 'rotation is not configured' rather than the false 'rotation is free' a zero would report. The instrument is still created, so this is a per-tick decision not to observe, not a missing instrument. All three signals are covered by exact-value unit tests in src/tests/libxrpl/telemetry/MetricMacros.cpp (including the not-configured and between-rotations cases) and rendered by the ledger-sync-health panels Online-Delete Rotation Window & Copy-Forward Writes and Rotation Node Re-Store Rate. To make rotation_copy_node_restore_total assertable -- and to give the two asserted gauge sub-series a NON-ZERO reading rather than merely a present one -- the harness would need a step that starts a node against a pre-populated database that has already rotated at least once, or that lowers online_delete and advisory_delete far enough to force a rotation inside the run window and then re-scrapes before teardown.",
    "_round_histogram_note": "consensus_round_duration_ms is a native OTel histogram recorded once per consensus round in RCLConsensus, so it needs no collector configuration -- it rides the existing OTLP -> Prometheus path. It is asserted by all three of its Prometheus series -- _bucket, _count and _sum -- because the bare instrument name is not a series. All three are unconditional on any running cluster: every node closes ledgers continuously, so a round completes within the harness window and the histogram is populated. Absence means the record site or the explicit-bucket view regressed, not that the node was idle. The instrument carries NO labels, so exactly one series exists per node and per bucket boundary.",
    "_overlay_dial_series_note": "overlay_connect_total and the three overlay_dial_latency_ms series report 4 series per run where sibling families such as dns_resolve_* report 5, one per node. That asymmetry is expected and benign, and it is recorded here so nobody investigates it again. The mechanism is an early return: OverlayImpl::connect asks peerFinder().newOutboundSlot(remoteEndpoint) for a slot and returns immediately when it hands back a null one (OverlayImpl.cpp:464-470), which is BEFORE the ConnectAttempt that emits both signals is constructed (OverlayImpl.cpp:472). Every node in the harness is seeded to dial every other node -- run-full-validation.sh builds IPS_FIXED from all NUM_NODES-1 peers (run-full-validation.sh:324-331) and writes it into [ips] (:373-374) -- so all five nodes do attempt to dial. But in a full mesh each pair is dialled twice, once from each end, and the node whose peer got there first is refused an outbound slot for an address it is already connected to inbound: it constructs no ConnectAttempt and therefore emits neither the counter nor the latency histogram. Which node loses that race depends on startup and dial ordering, which the harness does not control, so the identity of the missing node is not stable across runs either. dns_resolve_* reports all 5 because reportDnsResolve fires from inside the resolver handler (OverlayImpl.cpp:603), which runs before any slot allocation and is unaffected by slot refusal. This is exactly why the four entries above assert series PRESENCE and not a series count. Asserting a count of 4 would bake today's mesh topology into the contract: it would break the moment NUM_NODES changes, and it would also break on a topology that is not a full mesh (a node seeded with no peers dials nothing, a node no one dials wins every slot), neither of which is a telemetry defect. If a future run reports fewer than 4, that IS worth investigating -- it would mean a node failed to dial at all rather than losing one race.",
    "_lock_hold_peak_note": "The two cache_metrics{metric=\"*_lock_hold_peak_us\"} series are unconditional: the cache_hit_rates callback observes both on every collection tick via TaggedCache::takeLockHoldPeak(), so each exists at value 0 on an idle node. A multi-second value in either is the mutex hold that froze every job, and is the exact signal WP-B6's rotation root cause exists to make legible. Neither is asserted for VALUE, only existence; the values are the whole point but they only appear on a node that actually rotates."
  },
  "node_health_gauges": {
    "description": "Node-health observable gauges (MetricsRegistry.cpp:997, :1081, :1102, :1161). server_info, build_info and db_metrics Observe unconditionally on every periodic export (build_info observes a literal 1; server_info and db_metrics read live services), so their series exist regardless of workload shape. complete_ledgers is the exception and is asserted on a narrower guarantee: its callback returns without observing when the range is empty (MetricsRegistry.cpp:1113-1114) and skips any segment that carries no '-' (:1122-1127), and a one-sequence range renders with no '-' (RangeSet.h:70-71), so it needs a complete range spanning at least two sequences. completeLedgers_ is filled by setFullLedger (LedgerMaster.cpp:862-863), which on a peered node is reached only from the publish path in doAdvance (LedgerMaster.cpp:1972) \u2014 closing a ledger is not enough, it has to validate. run-full-validation.sh waits for that before the workload starts, so on a healthy cluster the series always exists \u2014 a 5-node run yields 10 series, one start and one end per node. If this check ever fails, read the Step 3 output first: a run that logged 'No validated ledger' cannot produce this series and the cluster, not the exporter, is what broke.",
    "metrics": ["server_info", "build_info", "complete_ledgers", "db_metrics"]
  },
  "overlay_reduce_relay": {
    "description": "Transaction reduce-relay efficiency gauge (MetricsRegistry.cpp:1354, peer-network dashboard). Backed by Overlay::txMetrics(); TxMetrics::json() emits txr_selected_cnt / txr_suppressed_cnt / txr_not_enabled_cnt unconditionally (TxMetrics.cpp:121-127), so the gauge always reports at least the selected_peers series.",
    "metrics": ["reduce_relay_metrics"]
  },
  "overlay_overflow": {
    "description": "Job-queue transaction overflow total (MetricsRegistry.cpp:609, job-queue dashboard). An ObservableCounter that reads Overlay::getJqTransOverflow() and Observes unconditionally, so the series exists at value 0 even when no overflow occurs.",
    "metrics": ["jq_trans_overflow_total"]
  },
  "validation_lifetime_counters": {
    "description": "Lifetime validation agreement/miss ObservableCounters (MetricsRegistry.cpp:1636, :1658, validator-health dashboard). Both callbacks reconcile the tracker and Observe unconditionally, so the series exist even on a node that has not yet agreed or missed (value 0). Only existence is asserted, never the value \u2014 validation_missed_total legitimately dominates on a non-validating node.",
    "metrics": ["validation_agreements_total", "validation_missed_total"]
  },
  "not_asserted": {
    "description": "Emitted-and-dashboarded metrics deliberately left unasserted because they are workload-gated or defect-gated: the harness workload cannot guarantee they appear, and a check that fails on a healthy run is worse than no check. This group has no \"metrics\" key, so validate_telemetry.py skips it (validate_metrics iterates category_data.get(\"metrics\", [])). Promote an entry into an asserted group only after the workload is changed to guarantee it. This map is the ONLY machine-readable record of an emitted-but-unasserted name, so every such name belongs here rather than in a prose note inside an asserted group -- prose cannot be linted. Concretely, phase-10 adds an _unaccounted_metric_names pass that harvests this map's keys as accounted names, so a name recorded only in a prose note is reported as unaccounted once that lands. It is warning-only and cannot fail CI, and it accepts a third source as well (an accounted_patterns regex list), so this map is the right home but not the only possible one. Caveat when adding: check_otel_naming.py Rule K harvests names only from `metrics` LISTS and from metric/name keys, so these dict KEYS are never validated against the MetricNames.h constants -- a typo here is silent, and must be checked by eye against the emit site cited in its own reason string.",
    "metrics_excluded": {
      "rpc_method_errored_total": "MetricsRegistry.cpp:332-333, push counter. 'Errored' here means a thrown C++ exception, not an error status in the JSON reply: the only caller is PerfLogImp.cpp:409 under 'if (!finish)', reached only through PerfLogImp::rpcError (PerfLogImp.h:150-153), whose only call site is the catch (std::exception&) handler in RPCHandler.cpp:213. An RPC that returns an error status normally still takes the rpcFinish path at RPCHandler.cpp:190 and increments rpc_method_finished_total. That distinction mattered here while the generator still issued ripple_path_find: those calls were in fact refused \u2014 pathfinding is off on every harness node, so doRipplePathFind returns RpcNotSupported (see pathfind_fast_milliseconds below) \u2014 and it would have been easy to conclude from that alone that this counter must fire. It does not, because a refusal is a normal return, not a throw. The harness issues no path-finding command at all, so the question is moot here, but the distinction is kept on record because it is the one that decides this entry. Nothing in rpc_load_generator.py's remaining server_info / fee / account / ledger / tx / DEX mix is expected to throw either, so no series may ever be created.",
      "ledger_history_mismatch_total": "MetricsRegistry.cpp:377, incremented only from LedgerHistory.cpp:332 on a built-vs-validated ledger mismatch. On a healthy run it never fires \u2014 asserting it would mean asserting a defect.",
      "txq_expired_total": "MetricsRegistry.cpp:379, incremented only at TxQ.cpp:1428 when a queued tx expires past its LastLedgerSequence. CI does run a txq-burst phase (workload-profiles.json:41, 30 s of single-type Payment at 60 TPS), but that does not guarantee sustained fee escalation followed by expiry: a run in which every other check passed still exposed only txq_metrics and no txq_expired_total.",
      "txq_dropped_total": "MetricsRegistry.cpp:381, incremented only at TxQ.cpp:1302 / :1347 on queue-full admission refusal. Same reason as txq_expired_total.",
      "getobject_rejected_total": "GetObjectMetricNames.h:81, emitted from PeerImp.cpp:2725/:2743 only for a TMGetObjectByHash message refused as oversize or malformed_ledgerhash. A cooperating cluster never sends one.",
      "getobject_request_objects": "GetObjectMetricNames.h:86, emitted from PeerImp.cpp:2926 only while serving an inbound TMGetObjectByHash. The XRPL_METRIC_* macros create their instrument lazily on first use (MetricMacros.h:174-285), so no series exists until a peer actually requests objects by hash \u2014 which a 5-node cluster started at genesis and already in sync may never do.",
      "getobject_lookup_us": "GetObjectMetricNames.h:95, PeerImp.cpp:2929. Same lazy-creation and same inbound-request gate as getobject_request_objects.",
      "getobject_lookups_total": "GetObjectMetricNames.h:100, PeerImp.cpp:2949/:2956. Same gate.",
      "getobject_charge": "GetObjectMetricNames.h:105, PeerImp.cpp:2931. Same gate.",
      "unl_fetch_total": "ValidatorSite.cpp:435, emitted only from reportFetchOutcome, which indexes sites_[siteIdx]. sites_ is populated from [validator_list_sites]; run-full-validation.sh writes a static [validators] file and no [validator_list_sites] exists anywhere under docker/telemetry/workload/, so sites_ is empty and no fetch outcome is ever reported. Needs a harness that serves a UNL over HTTP.",
      "handshake_negotiation_fail_total": "Handshake.cpp:265, emitted only when a handshake is REJECTED. Every node in the cluster shares one network id and one build, dials 127.0.0.1 and has no clock skew, so no reject path was found to be reachable. Not proven unreachable -- promote it if a run ever shows the series.",
      "sync_acquire_source_total": "InboundLedger.cpp:165, emitted from InboundLedger::init(), the same function that opens the ledger.acquire span. expected_spans.json already marks that span optional (\"A healthy local cluster rarely back-fills history\"), so the counter inherits exactly that condition. Best first candidate to promote if the harness ever forces an acquire: unlike its siblings it fires even when the store was already complete, via source=local.",
      "sync_acquire_no_progress_total": "InboundLedger.cpp:536. Strictly narrower than sync_acquire_source_total: needs an acquire AND a timer tick that made no progress.",
      "sync_addnode_total": "InboundLedger.cpp:1636 (recordBatchOutcome). Needs an acquire that actually receives SHAMap nodes over the wire; the emit site returns early when the batch count is <= 0, so a zero tally publishes nothing rather than a zero.",
      "ledger_replay_fallback_total": "SkipListAcquire.cpp:113 / LedgerDeltaAcquire.cpp:117. Structurally impossible here: reachable only from a LedgerReplayTask, created only via LedgerReplayer::replay(), whose sole production caller is LedgerMaster.cpp:1459 behind `if (app_.config().ledgerReplay)`. Config::ledgerReplay defaults false and is set only from [ledger_replay], which the harness never writes.",
      "ledger_replay_outcome_total": "LedgerReplayTask.cpp:237. Same [ledger_replay] gate as ledger_replay_fallback_total, plus a real historical back-fill against peers advertising the LedgerReplay feature.",
      "ledger_jump_total": "NetworkOPs.cpp:2307, unconditional inside switchLastClosedLedger, which is reached from checkLastClosedLedger -- called every round at :2489. Excluded as NOT DELIBERATELY PROVOKED rather than as impossible: the harness injects no divergence, but a node that falls behind can be told the network's LCL is not the one it built on and follow it, and the rpc-burst / tx-flood / txq-burst phases exist precisely to create load spikes (the same group already asserts server_stall_events_total on that basis). So this may fire on some healthy runs and not others, which is a flaky check either way. Cheap decisive test before ever promoting it: grep the five nodes' debug.log for 'JUMP last closed ledger' after a run.",
      "peer_disconnect_total": "PeerImp.cpp:655, inside PeerImp::close(), immediately after the !socket_.is_open() early return that de-duplicates repeat closes -- so it does fire on the first genuine close, for any of its 13 reason values. Excluded because a healthy run produces no disconnect CAUSE, not because there is too little time: the cluster's 5 nodes hold each other for the whole run, the timer-driven reasons require the peer to actually be diverged or unknown rather than merely for maxDivergedTime (300 s) / maxUnknownTime (600 s) to pass, and the shutdown reasons fire during --cleanup teardown, after Step 5 has scraped. Do not rest this on the window length: the profile's 270 s of workload plus the 60 s propagation wait, node startup and up to 45 s of metric polling exceeds 300 s. What keeps it out of the gate is that a mutual-dial race CAN produce one non-deterministically -- asserting a signal that appears on some healthy runs and not others is a flaky check either way.",
      "serve_refused_total": "PeerImp.cpp:669 (reportServeRefusal), reached from the notFound paths at :3759 (tx set) and :3793 (ledger). Excluded because nothing in the harness ASKS: a refusal requires an inbound TMGetLedger, and that message is originated only from InboundLedger.cpp:904 and TransactionAcquire.cpp:304/340 -- the ledger.acquire and txset.acquire paths, both of which expected_spans.json marks optional because a healthy cluster does not back-fill. Note the reason is the absence of requests, NOT that a request would succeed: PeerSetImpl::addPeers uses hasItem(peer) only as a sort SCORE and then adds peers in score order up to its limit (PeerSet.cpp:77-95), so peers that lack the item are asked too and would be refused. It follows that this counter and the acquire spans stand or fall together, and if a harness change ever makes an acquire reliable this should be promoted alongside them.",
      "sweep_malloc_trim_minor_faults_total": "Application.cpp:1249, published only when the trim's minor-fault delta is > 0. A trim on the small heap of a fresh localhost node routinely faults zero times, and the site deliberately publishes nothing rather than a zero that would claim the trim was measured as free.",
      "sweep_malloc_trim_reclaimed_kb_total": "Application.cpp:1263, published only when resident memory actually FELL across the trim. On a node whose caches are still filling this frequently does not happen: glibc has nothing above the top of the heap to release, and mmap-backed allocations are returned on free regardless of trimming.",
      "rotation_copy_node_restore_total": "SHAMapStoreImp.cpp:283. Fires only for a clean tree node reachable from the validated state map whose sole on-disk copy an EARLIER rotation removed, so it needs at least two rotations (512 validated ledgers, ~15-20 min at the cluster's close rate) plus real prior data loss. The run window is 270 s. Note that the rotation_state gauge sub-series ARE asserted -- see sync_diagnostics._b5_rotation_note for why a series exists while a rotation never runs.",
      "rpc_size_bytes": "ServerHandler.cpp:191, group('rpc')->makeEvent('size', Unit::Bytes). The OTLP Prometheus exporter derives the metric-name suffix from the declared unit, so a byte unit yields rpc_size_bytes. The Unit::Bytes declaration itself landed earlier, in 76c9051203; what 24094e427b changed was the exporter finally consuming it, replacing a hardcoded CreateDoubleHistogram(name, 'Duration in ms', 'ms') with otelUnitDescription(unit)/otelUnitCode(unit), and that is what renamed the series off rpc_size_milliseconds and the millisecond bucket ladder. Neither name was ever recorded here, so the harness could confirm neither the rename nor a regression back onto that ladder. Notified from ServerHandler::processRequest:1133, the HTTP JSON-RPC path \u2014 it computes an HTTP status and appends a trailing newline \u2014 and the load generators are WebSocket-only, the same gate regression-metrics.json:4 records for rpc.process, so only the harness's handful of HTTP health polls reach it. Real coverage needs an HTTP JSON-RPC phase in rpc_load_generator.py; that is a workload change rather than a harness correction, and is deliberately out of scope here.",
      "rpc_time_milliseconds": "ServerHandler.cpp:192, group('rpc')->makeEvent('time') with the default millisecond unit. Notified from ServerHandler::processRequest:1129, the same HTTP JSON-RPC call site as rpc_size_bytes and behind the same WebSocket-only gate.",
      "pathfind_fast_milliseconds": "PathRequestManager.h:35, makeEvent('pathfind_fast') with the default millisecond unit, so the exported form is the pathfind_fast_milliseconds_bucket/_count/_sum triple and there is no bare series \u2014 the same convention io_latency and rpc_method_us follow, and rpc-pathfinding queries the _bucket. THE OPERATIVE BLOCKER IS THE CONFIG, NOT THE CALL GRAPH: pathfinding is disabled outright on every harness node, so no PathRequest is ever constructed and no pathfind_* histogram can exist. Config.cpp:725-726 sets pathSearchMax to 0 whenever a [validation_seed] or [validator_token] section is present ('By default, validators don't have pathfinding enabled'); run-full-validation.sh writes [validation_seed] into every generated node cfg and contains no [path_search], [path_search_fast] or [path_search_max] section to put it back (grep count 0 for path_search in that file \u2014 the only [path_search*] sections in docker/telemetry/ are in xrpld-telemetry.cfg and xrpld-telemetry-mainnet.cfg, neither of which the harness uses); and doRipplePathFind returns RpcNotSupported at RipplePathFind.cpp:59-60, before context.loadType is set and before any branch on the ledger parameter. That config gate alone would keep the metric absent even if the generator did issue the command, because every such call is refused at the front door; and the generator issues no path-finding RPC in the first place. Two independent reasons. Read that first: the structural argument below is correct and matters if pathfinding is ever enabled, but it is not why the metric is missing today. STRUCTURAL ARGUMENT (verified, applies once pathSearchMax is non-zero): reportFast's only caller is PathRequest.cpp:852, inside the 'if (fast && quickReply_ == {})' branch of PathRequest::doUpdate. The only doUpdate call that passes fast=true is in PathRequest::doCreate (PathRequest.cpp:259), guarded by '!hasCompletion()'. Both ripple_path_find entry points construct the PathRequest with a completion function, so hasCompletion() (:161-164) is true and the fast pass is skipped: with no ledger specified doRipplePathFind goes to makeLegacyPathRequest, which passes the coroutine-post lambda (RipplePathFind.cpp:140-160); with a ledger specified it goes to doLegacyPathRequest, which passes an empty-body but non-null lambda (PathRequestManager.cpp:317). Only the path_find streaming subscription reaches reportFast, because makePathRequest builds the request from a subscriber with no completion (PathRequestManager.cpp:261). The load generator has never used path_find: it fires one request per send and awaits one reply, which a streaming subscription does not fit. Covering this metric therefore needs both a [path_search_max] override (or a non-validator node) in run-full-validation.sh and a path_find subscription phase in the generator. Both are harness/workload changes and out of scope here. Grafana Cloud shows zero series in 180 days on the devnet nodes, consistent with those nodes receiving no pathfinding RPC.",
      "pathfind_full_milliseconds": "PathRequestManager.h:36, makeEvent('pathfind_full'); same histogram naming as pathfind_fast_milliseconds above. Notified from reportFull (:87-90) via PathRequest.cpp:857, the 'else if (!fast && fullReply_ == {})' branch. Blocked by exactly the same config gate as pathfind_fast_milliseconds, and the probability of emission under the harness workload is zero, not low: pathSearchMax is 0 on every harness node (Config.cpp:725-726 plus the [validation_seed] section run-full-validation.sh writes, with no [path_search*] override anywhere in that file), so doRipplePathFind returns RpcNotSupported at RipplePathFind.cpp:59-60 and no PathRequest object is ever constructed for reportFull to fire from. An earlier revision of this entry described the path as reachable-but-probabilistic \u2014 emitting one ledger close behind the request via PathRequestManager::updateAll's one-shot branch (PathRequestManager.cpp:160-166) \u2014 and prescribed sending an explicit ledger_index from the generator so doLegacyPathRequest would call doUpdate(cache, false) synchronously (PathRequestManager.cpp:321). Both halves were wrong. The probability is zero rather than merely unreliable, and the prescribed remedy cannot work at all, because the pathSearchMax guard fires before the ledger parameter is read: adding ledger_index changes nothing while pathfinding is off. The generator also issues no path-finding RPC, so covering this metric needs a [path_search_max] override (or a non-validator node) in run-full-validation.sh AND path-finding load added \u2014 a harness-topology plus workload change, out of scope here. The workload README section 'Pathfinding is not exercised' holds the recipe. Grafana Cloud shows zero series in 180 days on the devnet nodes, consistent with those nodes receiving no pathfinding RPC rather than with a broken exporter.",
      "warn_total": "include/xrpl/resource/detail/Logic.h:41, makeMeter('warn'). makeMeter maps to CreateUInt64Counter (OTelCollector.cpp:878-881 -> :773-777), so the Prometheus exporter appends _total; the meter is created on the bare collector with no group, hence the unprefixed name. Incremented only at Logic.h:481, inside the 'if (notify)' branch reached when a consumer's balance crosses kWarningThreshold. A cooperating 5-node cluster plus a rate-limited load generator never charges a consumer that far, and Grafana Cloud confirms zero series in 180 days. Recorded explicitly because this was briefly mis-diagnosed as a phantom metric: the rpc-pathfinding panel that queries it is correct, and renders empty only because the condition has not occurred.",
      "drop_total": "include/xrpl/resource/detail/Logic.h:42, makeMeter('drop'); same CreateUInt64Counter mapping and same _total suffix as warn_total. Incremented only at Logic.h:505, when a consumer's balance is at or above kDropThreshold and the connection is dropped. Grafana Cloud shows 2 live series, so unlike warn_total this one does fire in the wild \u2014 but only on a genuinely abusive consumer, which the harness deliberately does not create, so it is condition-gated all the same. Its rpc-pathfinding panel is likewise correct rather than phantom.",
      "jobq_*_milliseconds, jobq_*_q_milliseconds": "This key is a pattern rather than a literal metric name \u2014 unlike every other entry in this map it stands for a whole family, one pair per job type. Created per job type in JobTypeData.h:97-98 from info.name() and info.name() + kSuffixQueued ('_q'), so the exported names are jobq_<jobtype>_milliseconds and jobq_<jobtype>_q_milliseconds with the job type lowercased by formatName. Which job types appear depends on which jobs a run happens to schedule, so no individual name is guaranteed. They are also rounded up to a whole millisecond at source (Event.h:47-51 applies ceil to a millisecond value type), which is why 6e2b2da772 moved the ledger-data-sync q-wait panels off jobq_<jobtype>_q_milliseconds_bucket onto job_queued_us_bucket \u2014 they are poor assertion targets regardless.",
      "rotation_phase_duration_seconds": "Recorded only at the end of an online-delete rotation phase (SHAMapStoreImp::RotationPhase destructor). The harness never rotates; see the nodestore.rotate note in expected_spans.json.",
      "jobq_stall_total": "Incremented in MetricsRegistry::recordJobFinished only when a job ran >= kJobStallThresholdUs (1 s). A healthy 5-node localhost run has no such job, so the series may not exist.",
      "consensus_view_change_total": "Incremented only when RCLConsensus::Adaptor::getPrevLedger sees the network's preferred ledger differ from this node's and mode is not already WrongLedger. A healthy 5-node harness never diverges from the network view, so the series may not exist under run-full-validation.sh."
    }
  },
  "accounted_patterns": [
    {
      "pattern": "^jobq_[a-z]+_(waiting|running|deferred)$",
      "family": "Per-job-type queue-depth gauges: 35 non-special job types x 3 states = 105 families.",
      "reason": "Cannot be enumerated usefully. JobTypeData's constructor creates waiting/running/deferred for every non-special job type (JobTypeData.h:96-103) and JobQueue eagerly constructs a JobTypeData for every type in JobTypes (JobQueue.cpp:45-53), so the family's membership is derived mechanically from the JobTypes.h table rather than chosen. Six members are asserted by literal name in job_queue_per_type_gauges on consumer-coverage grounds; this pattern accounts for the rest so the reverse coverage check does not report 99 names that the contract has already reasoned about. Narrow by construction: every one of the 35 non-special job type names lowercases to ^[a-z]+$ with no digits and no underscores (verified mechanically against JobTypes.h:52-86), and the state suffix is a closed set of three, so nothing outside this family can match. A new job type therefore stays covered without an edit here, which is correct \u2014 adding a job type is not a telemetry design change. A renamed state, a job type containing a digit or an underscore, or any other new jobq gauge shape would surface as unaccounted."
    },
    {
      "pattern": "^jobq_[a-z]+(_q)?_milliseconds$",
      "family": "Per-job-type job-latency and queue-wait histogram families: 35 job types x 2 = 70 families, each exported as a _bucket/_count/_sum triple.",
      "reason": "The same mechanically derived family as the gauges above, in histogram form: JobTypeData.h:97-98 creates one Event from info.name() and one from info.name() + '_q'. Which job types actually record a sample depends on what a run schedules, which is why not_asserted.metrics_excluded carries these under a single pattern key rather than as literal names. The reverse coverage check matches this pattern against the base family name, after stripping the exporter's _bucket/_count/_sum suffix, so all three series of every triple are accounted for by this one entry. Same narrowness argument as above: the job type part is ^[a-z]+$ and the optional _q plus the _milliseconds unit suffix close the shape."
    },
    {
      "pattern": "^(getobject_account_state_node_get|getobject_account_state_node_share|getobject_cas_get|getobject_cas_share|getobject_fetch_pack_get|getobject_fetch_pack_share|getobject_get|getobject_ledger_get|getobject_ledger_share|getobject_share|getobject_transaction_get|getobject_transaction_node_get|getobject_transaction_node_share|getobject_transaction_share|getobject_transactions_get|have_transactions|ledger_account_state_node_get|ledger_account_state_node_share|ledger_data_account_state_node_get|ledger_data_account_state_node_share|ledger_data_get|ledger_data_share|ledger_data_transaction_node_get|ledger_data_transaction_node_share|ledger_data_transaction_set_candidate_get|ledger_data_transaction_set_candidate_share|ledger_get|ledger_share|ledger_transaction_node_get|ledger_transaction_node_share|ledger_transaction_set_candidate_get|ledger_transaction_set_candidate_share|overhead|overhead_cluster|overhead_manifest|overhead_overlay|proof_path_request|proof_path_response|proposals|proposals_duplicate|proposals_untrusted|replay_delta_request|replay_delta_response|requested_transactions|set_get|set_share|squelch|squelch_ignored|squelch_suppressed|total|transactions|transactions_duplicate|unknown|validations|validations_duplicate|validations_untrusted|validator_lists)_(bytes|messages)_(in|out)$",
      "family": "Overlay per-category traffic gauges: 57 categories x {bytes,messages} x {in,out} = 228 families, of which the four total_* are asserted literally in overlay_traffic.",
      "reason": "Also mechanically derived: OverlayImpl builds one TrafficGauges per entry of TrafficCount::counts_ (OverlayImpl.cpp:200-203), and TrafficGauges creates Bytes_In / Bytes_Out / Messages_In / Messages_Out for each (OverlayImpl.h:580-593), so the set is exactly the cross product of the category table with those four names. The 57 categories are the 56 entries of TrafficCount::toString's kCategoryMap (TrafficCount.h:239-294) plus 'unknown', which is what Category::Unknown falls through to because it has an entry in counts_ but none in the map. The categories are enumerated here rather than reduced to a shape, because unlike the job types they contain underscores: a structural pattern would have to be ^[a-z_]+_(bytes|messages)_(in|out)$, which would silently swallow any future non-overlay metric ending in _bytes_in. Enumerating means a new traffic category surfaces as unaccounted, which is the right outcome: overlay_traffic's description calls itself an explicit subset and should be revisited when the table grows. The category list was extracted mechanically from TrafficCount.h; regenerate it the same way rather than editing by hand."
    },
    {
      "pattern": "^(target_info|up|scrape_[a-z_]+|promhttp_[a-z_]+)$",
      "family": "Prometheus scrape metadata and exporter housekeeping series.",
      "reason": "Not xrpld telemetry, and not emitted by anything in this repository, so there is nothing for the contract to assert. prometheus.yml defines exactly one scrape job against the collector's Prometheus exporter (otel-collector:8889), and Prometheus synthesises up and the scrape_* family per target on every scrape. target_info is written by the collector's Prometheus exporter to carry the OTLP resource attributes, and several dashboards read it as label_values(target_info, service_instance_id). promhttp_* is the metrics handler's own counter set, present only if the exporter's handler exposes it. Listed so the reverse coverage check reports genuine telemetry gaps instead of scrape plumbing. Deliberately does not cover otelcol_*: the collector's internal telemetry defaults to port 8888 and otel-collector-config.yaml declares no service::telemetry block, so those series are not scraped here \u2014 if they ever appear, that is a collector configuration change worth seeing."
    }
  ],
  "grafana_dashboards": {
    "description": "All 15 Grafana dashboards provisioned on disk under docker/telemetry/grafana/dashboards/ (UID == file stem for every one). validate_dashboards() checks that each UID resolves via GET /api/dashboards/uid/<uid> and reports its panel count \u2014 it verifies provisioning and loadability, not panel data. log-derived-insights is included on that basis even though its panels are Loki-backed and CI runs with --skip-loki: the dashboard itself must still provision cleanly. Its panel data is not asserted anywhere.",
    "uids": [
      "rpc-performance",
      "transaction-overview",
      "consensus-health",
      "ledger-operations",
      "peer-network",
      "peer-quality",
      "fee-market",
      "job-queue",
      "validator-health",
      "node-health",
      "network-traffic",
      "rpc-pathfinding",
      "overlay-traffic-detail",
      "ledger-data-sync",
      "ledger-sync-health",
      "log-derived-insights"
    ]
  }
}