rippled/docker/telemetry/workload/expected_metrics.json

{
  "description": "Expected metric inventory for xrpld telemetry validation. Metric names have no prefix (the xrpld_ prefix was removed). beast::insight metrics are lowercased by formatName. Every name here was verified against its declaration in MetricsRegistry.cpp or include/xrpl/telemetry/GetObjectMetricNames.h and against a panel query under docker/telemetry/grafana/dashboards/. IMPORTANT: validate_telemetry.py has no notion of an optional metric — validate_metrics() iterates every group that has a \"metrics\" key and hard-fails any name with 0 Prometheus series after a 45 s poll. A metric is therefore listed only when the harness workload guarantees it will appear: observable gauges/counters whose callbacks Observe unconditionally (series exist at value 0), or push counters/histograms on a path every run exercises. Workload-gated and defect-gated names are recorded in the \"not_asserted\" group, which intentionally has no \"metrics\" key so the validator skips it. Only series existence is checked, never a value, except for the four bounds checks hardcoded in PARITY_VALUE_SANITY. A group may additionally declare \"required_labels\": each label there becomes one check that at least one of the group's series carries it with a non-empty value, matched as <label>!=\"\" because Prometheus cannot tell an absent label from an empty one. The same guarantee rule applies — list a label only where the workload guarantees it. Two conventions apply to the consumer evidence cited in the group descriptions below. First, only tracked files count: dashboards under docker/telemetry/grafana/dashboards/ and rules under docker/telemetry/grafana/provisioning/alerting/. The docker/telemetry/grafanacloud/ tree is gitignored (sourceCode/.gitignore:27) and has zero tracked files, so a cloud copy cannot be cited as evidence on a clean checkout and is not counted here. Second, every rule in provisioning/alerting/rules.yaml is provisioned with isPaused: true (14 of 14), so a metric feeding a rule is a wiring dependency, not a live pager: losing it leaves the rule wired to a signal that no longer resolves, and it starts silent rather than becoming silent. Finally, the top-level \"accounted_patterns\" list is not a group and asserts nothing. It declares anchored regexes for the bulk families that cannot usefully be enumerated as literal names — the per-job-type gauges and histograms, the overlay per-category traffic cross product — plus the Prometheus scrape plumbing that is not xrpld telemetry at all. Its only consumer is the reverse coverage check in validate_telemetry.py, which walks the metric families Prometheus actually holds and names every one that neither a group's \"metrics\" entry, nor a \"metrics_excluded\" key, nor one of these patterns accounts for. That check warns and never fails: downstream branches legitimately add telemetry this contract has not seen yet, and a hard failure would redden all of them. Its value is visibility, not enforcement.",
  "spanmetrics": {
    "description": "SpanMetrics-derived RED metrics from the OTel Collector spanmetrics connector.",
    "metrics": [
      "span_calls_total",
      "span_duration_milliseconds_bucket",
      "span_duration_milliseconds_count",
      "span_duration_milliseconds_sum"
    ],
    "required_labels": [
      "span_name",
      "status_code",
      "service_name",
      "span_kind"
    ],
    "dimension_labels": [
      "command",
      "rpc_status",
      "consensus_mode",
      "local",
      "proposal_trusted",
      "validation_trusted",
      "tx_type",
      "ter_result",
      "stage",
      "txq_status",
      "close_time_correct",
      "consensus_state",
      "suppressed"
    ],
    "_dimension_labels_note": "Bare label names as configured in otel-collector-config.yaml spanmetrics dimensions. Informational only (not asserted by the validator)."
  },
  "statsd_gauges": {
    "description": "beast::insight gauges exported via OTLP/HTTP to the collector (server=otel). What guarantees these is the export mechanism, not the workload. On the OTel path a beast gauge is an Int64ObservableGauge (OTelCollector.cpp:703); every OTelGaugeImpl registers itself with the collector in its own constructor (:690); onCollectionReady() arms every registered gauge with no per-metric condition (:941-979); and the armed callback Observes currentValue() on each export cycle regardless of the value and regardless of whether set() was ever called (:719-727). Series therefore exist from the first export, at 0 if nothing has happened. The 'gauges only mark dirty on value changes' caveat in _poll_series_count is a property of the StatsD backend and does not apply here. The consequence is that a beast gauge whose object is constructed on an unconditional startup path is as safe to assert as an observable gauge from MetricsRegistry — provided the object is constructed before Application.cpp:1570, which is where onCollectionReady() runs. That precondition is load-bearing and is the one way a gauge on this path can still go missing. Registration and arming are separate steps: addGauge() (:927-931) only appends to gauges_ under the lock, and onCollectionReady() snapshots that vector and calls arm() on each entry, exactly once per process (Application.cpp:1570, on the setup path, with no second call site). A gauge whose owner is built after that line is registered, never armed, and never exported — no callback, no series, and no error. Every gauge asserted in this group clears the precondition: jobQueue_, nodeFamily_, ledgerMaster_ and networkOPs_ are all in ApplicationImp's member-init list (Application.cpp:377, :430, :439, :465), and overlay_ — which owns peerFinder_ in its own member-init list (OverlayImpl.cpp:185) and Stats::peerDisconnects (OverlayImpl.h:603) — is built at Application.cpp:1551, nineteen lines before the arming call. The comment at Application.cpp:1258-1272 states that ordering deliberately: overlay_ is the last service the callbacks read, which is why arming waits for it. This is also why the state_accounting family below is asserted whole rather than one member deep. The ten state_accounting_* names are one family with one guarantee. NetworkOPsImp::Stats creates all ten in the NetworkOPsImp constructor (NetworkOPs.cpp:1069-1080; makeGauge(\"State_Accounting\", \"Full_duration\") joins on '.' at Collector.h:151-155 and formatName lowercases it, giving state_accounting_full_duration), and NetworkOPsImp::collectMetrics() set()s all ten unconditionally in a single block (:5204-5231). No state has to be entered for its pair to exist: a node that never disconnects still publishes state_accounting_disconnected_duration and _transitions at 0. All ten appear on the node-health dashboard, and state_accounting_full_transitions is also the input to the NodeStateFlapping alert rule (provisioning/alerting/rules.yaml:645-649, uid xrpld-node-state-flapping — provisioned isPaused: true, so this is a wiring dependency and not a live pager). Asserting only full_duration left the other nine unchecked, the alert's own input among them, even though a regression could not plausibly drop one sibling without dropping full_duration too. node_family_full_below_cache_hit_rate and _size are the TaggedCache Stats pair (TaggedCache.h:282-283), created in the cache constructor (TaggedCache.ipp:63-66) and published by collectMetrics(), which reports a hit_rate of 0 rather than skipping the gauge when hits+misses is 0 (TaggedCache.ipp:751-765). The instance is NodeFamily's full-below cache, constructed with the real collector in NodeFamily's constructor (NodeFamily.cpp:28-35), which every node builds. Both are on node-health. overlay_peer_disconnects is the first member of OverlayImpl::Stats (OverlayImpl.h:603) and OverlayImpl::collectMetrics() assigns it getPeerDisconnect() unconditionally (:646), in the same hook that publishes the overlay_traffic group's gauges. Its only query consumer is network-traffic.json:148. It also appears once in log-derived-insights.json:1908, but that occurrence is inside a panel description which explicitly disclaims it ('overlay_peer_disconnects exists as a metric but carries no reason breakdown, which is what this panel adds') — that panel is Loki-derived and queries nothing from Prometheus, so it is not a consumer. The third occurrence, validate_dashboards.py:17, is a lint rule listing the name as a cumulative gauge that must be rate()-wrapped; also not a query consumer. It is listed here rather than in overlay_traffic because that group's contract is an explicit subset of the per-category traffic family, and a disconnect count is not one of those categories.",
    "metrics": [
      "ledgermaster_validated_ledger_age",
      "ledgermaster_published_ledger_age",
      "state_accounting_full_duration",
      "peer_finder_active_inbound_peers",
      "peer_finder_active_outbound_peers",
      "jobq_job_count",
      "state_accounting_connected_duration",
      "state_accounting_connected_transitions",
      "state_accounting_disconnected_duration",
      "state_accounting_disconnected_transitions",
      "state_accounting_full_transitions",
      "state_accounting_syncing_duration",
      "state_accounting_syncing_transitions",
      "state_accounting_tracking_duration",
      "state_accounting_tracking_transitions",
      "node_family_full_below_cache_hit_rate",
      "node_family_full_below_cache_size",
      "overlay_peer_disconnects"
    ]
  },
  "statsd_counters": {
    "description": "beast::insight counters exported via OTLP/HTTP. The OTel Prometheus exporter appends _total to monotonic counters.",
    "metrics": ["rpc_requests_total", "ledger_fetches_total"]
  },
  "io_latency": {
    "description": "io_context scheduling-latency histogram — a beast::insight Event exported over OTLP (Application.cpp:515, makeEvent('ios_latency') with the default millisecond unit). It is the one beast Event the node itself guarantees, which is why it is asserted while rpc_time_milliseconds and the jobq_* pairs sit in not_asserted: the sampler starts on the unconditional startup path (Application.cpp:1697, outside the 'if (withTimers)' guard), and its handler always emits the first sample whatever its value (Application.cpp:185-189, 'firstSample_.exchange(false) || lastSample >= 10ms', with a comment stating the point is to register the metric downstream). The OTel histogram is cumulative, so that one notify creates series that persist for the rest of the run. Named with the _bucket/_count/_sum suffixes the Prometheus exporter emits for a histogram — no bare ios_latency_milliseconds series exists, the same convention rpc_method_us and span_duration_milliseconds follow, and every consumer queries ios_latency_milliseconds_bucket: 2 distinct panels (ledger-data-sync 'I/O Scheduler Latency p95' and node-health 'I/O Latency') plus one alert rule (grafana/provisioning/alerting/rules.yaml:586). Only existence is asserted: past the first sample the handler reports only latencies >= 10 ms, so neither the sample count nor the value is predictable.",
    "metrics": [
      "ios_latency_milliseconds_bucket",
      "ios_latency_milliseconds_count",
      "ios_latency_milliseconds_sum"
    ]
  },
  "overlay_traffic": {
    "description": "Overlay traffic metrics (subset — full list has 45+ categories).",
    "metrics": [
      "total_bytes_in",
      "total_bytes_out",
      "total_messages_in",
      "total_messages_out"
    ]
  },
  "nodestore_io": {
    "description": "NodeStore I/O observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label distinguishing sub-metrics.",
    "metrics": ["nodestore_state"]
  },
  "cache_hit_rates": {
    "description": "Cache hit rate observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label.",
    "metrics": ["cache_metrics"]
  },
  "transaction_queue": {
    "description": "Transaction queue observable gauge (MetricsRegistry via OTLP). Single metric with 'metric' label.",
    "metrics": ["txq_metrics"]
  },
  "rpc_method_detail": {
    "description": "Per-RPC-method counters and duration histogram (MetricsRegistry.cpp:351-357). rpc_method_errored_total is deliberately absent — see not_asserted below. rpc_method_us is a Histogram, so the Prometheus exporter emits only the _bucket/_count/_sum triple and there is no bare rpc_method_us series to match — same convention as span_duration_milliseconds in the spanmetrics group above.",
    "metrics": [
      "rpc_method_started_total",
      "rpc_method_finished_total",
      "rpc_method_us_bucket",
      "rpc_method_us_count",
      "rpc_method_us_sum"
    ]
  },
  "job_queue": {
    "description": "Job-queue counters and latency histograms (MetricsRegistry.cpp:360-366). Every xrpld job passes through these, so they populate under any workload. Both histograms are recorded in the same function bodies as job_started_total / job_finished_total, under the same guard and with the same labels, so their presence is equally guaranteed. They are named with the _bucket/_count/_sum suffixes the Prometheus exporter emits: regression-metrics.json and the job-queue dashboard both query job_queued_us_bucket / job_running_us_bucket, and no bare series exists. This group also carries the xrpl_node_id required_labels assertion. 070d29b465 added the xrpl.node.id resource attribute so Grafana Cloud trace ingest would stop folding distinct nodes into one — ingest groups ResourceSpans but ignores service.instance.id, so before the attribute existed the boards reported one node's ledger.build five times over and saw 2 of 9 nodes. Nothing asserted it, so a regression would silently restore the folding. It is asserted here rather than on a statsd_* group because these metrics are MetricsRegistry-backed and guaranteed by any workload, whereas the beast::insight meter provider is constructed before the wallet DB exists and so has no node public key to stamp: those metrics legitimately lack the label and 070d29b465 omits it rather than writing it blank.",
    "required_labels": ["xrpl_node_id"],
    "metrics": [
      "job_queued_total",
      "job_started_total",
      "job_finished_total",
      "job_queued_us_bucket",
      "job_queued_us_count",
      "job_queued_us_sum",
      "job_running_us_bucket",
      "job_running_us_count",
      "job_running_us_sum"
    ]
  },
  "job_queue_per_type_gauges": {
    "description": "Per-job-type queue-depth gauges from the beast::insight \"jobq\" group, for the six job types the ledger-sync diagnostics single out. JobTypeData's constructor creates waiting/running/deferred for every non-special job type (JobTypeData.h:96-103), the JobQueue constructor eagerly constructs a JobTypeData for every type in JobTypes (JobQueue.cpp:45-53), and JobQueue::collect() assigns all three from one snapshot for every type on every hook cycle, with no per-type condition (JobQueue.cpp:104-106). The collector is the \"jobq\" group, so GroupImp::makeName() prefixes \"jobq.\" and formatName lowercases the job type: JtTxnData's \"fetchTxnData\" + \"_running\" exports as jobq_fetchtxndata_running (JobTypeData.h:29-37). Existence rests on the same ObservableGauge mechanism documented on statsd_gauges, so these series exist at 0 on an idle node. All six were present in the CI validation run's own emitted-metric list, which is the strongest evidence behind any entry in this file. All six job types are non-special: makeFetchPack (limit 1), manifest (maxLimit), ledgerRequest (3), ledgerData (3), updatePaths (1) and fetchTxnData (5) all declare a non-zero limit in JobTypes.h, so none is skipped by the special() guard in JobTypeData's constructor. The 11 special types (limit 0) get no gauges at all, which is why no jobq_pathfind_* or jobq_peercommand_* name can be asserted. Why these six and not all 105. The guarantee is identical for every one of the 35 non-special types x 3 states, so the discriminator here is consumer coverage rather than emission: these six are the only per-type names a dashboard or an alert queries literally, so losing one blanks a specific panel or leaves a rule wired to a signal that no longer resolves. jobq_manifest_waiting is the input to the ManifestJobQueueConvoy alert rule (provisioning/alerting/rules.yaml:796-800, uid xrpld-manifest-job-convoy — provisioned isPaused: true like every rule in that file, so the consequence is a broken wire rather than a silenced pager); the other five are per-type saturation panels on node-health.json:5267-5291. The rest of the family is reached only through two regex queries, both on ledger-data-sync: __name__=~\"jobq_.*_deferred\" at :1701 and __name__=~\"jobq_.*_waiting\" at :1708. Those are the only two jobq regexes anywhere in the tree — there is no jobq_.*_running regex — so the 30 unasserted _running gauges have zero consumers of any kind, and the unasserted _waiting and _deferred gauges are reached only by a topk that keeps working as long as the family exists and depends on no single job type. Consumer coverage is the whole argument and it stands alone. An earlier revision also cited poll bandwidth; that reason was wrong and has been removed, because METRIC_POLL_CONCURRENCY is 8 and the semaphore is released across each sleep, so 105 targets cost on the order of a second against the shared 45 s deadline. A bogus cost argument would be a bad reason to refuse a legitimate expansion later: if a panel or rule starts naming a seventh job type, add it.",
    "metrics": [
      "jobq_fetchtxndata_running",
      "jobq_ledgerdata_running",
      "jobq_ledgerrequest_running",
      "jobq_makefetchpack_running",
      "jobq_manifest_waiting",
      "jobq_updatepaths_running"
    ]
  },
  "rpc_in_flight": {
    "description": "In-flight RPC gauge via the XRPL_METRIC_UPDOWN_ADD call-site macro (PerfLogImp.cpp, +1 rpcStart / -1 rpcEnd). UpDownCounter: no _total suffix.",
    "metrics": ["rpc_in_flight_requests"]
  },
  "object_counts": {
    "description": "Counted object instances observable gauge (MetricsRegistry via OTLP).",
    "metrics": ["object_count"]
  },
  "load_factors": {
    "description": "Fee escalation and load factor observable gauge (MetricsRegistry via OTLP).",
    "metrics": ["load_factor_metrics"]
  },
  "parity_validation_agreement": {
    "description": "External dashboard parity: validation agreement percentages (MetricsRegistry).",
    "metrics": [
      "validation_agreement{metric=\"agreement_pct_1h\"}",
      "validation_agreement{metric=\"agreement_pct_24h\"}"
    ]
  },
  "parity_validator_health": {
    "description": "External dashboard parity: validator health indicators (MetricsRegistry).",
    "metrics": [
      "validator_health{metric=\"amendment_blocked\"}",
      "validator_health{metric=\"unl_expiry_days\"}"
    ]
  },
  "parity_peer_quality": {
    "description": "External dashboard parity: peer quality metrics (MetricsRegistry).",
    "metrics": [
      "peer_quality{metric=\"peer_latency_p90_ms\"}",
      "peer_quality{metric=\"peers_insane_count\"}"
    ]
  },
  "parity_ledger_economy": {
    "description": "External dashboard parity: ledger economy metrics (MetricsRegistry.cpp:1401). transaction_rate is observed on every export, in both branches of the ledger-age test (MetricsRegistry.cpp:1444-1451). base_fee_xrp is observed only inside the 'if (ledger)' guard on getValidatedLedger() (MetricsRegistry.cpp:1418-1423), and that returns validLedger_ (LedgerMaster.cpp:1569-1572), which stays null until a ledger validates — the same precondition complete_ledgers has. Both are asserted because run-full-validation.sh waits for a validated ledger before running the workload. base_fee_xrp absent while transaction_rate is present is the signature of a cluster that never validated, not of a missing metric.",
    "metrics": [
      "ledger_economy{metric=\"base_fee_xrp\"}",
      "ledger_economy{metric=\"transaction_rate\"}"
    ]
  },
  "parity_state_tracking": {
    "description": "External dashboard parity: server state tracking (MetricsRegistry).",
    "metrics": ["state_tracking{metric=\"state_value\"}"]
  },
  "parity_counters": {
    "description": "External dashboard parity: monotonic counters (MetricsRegistry). validations_checked_total is incremented unconditionally at the top of NetworkOPsImp::recvValidation (NetworkOPs.cpp:2681), and run-full-validation.sh brings up a 5-node validator cluster, so inbound validations are guaranteed.",
    "metrics": [
      "ledgers_closed_total",
      "validations_sent_total",
      "validations_checked_total",
      "state_changes_total"
    ]
  },
  "parity_storage": {
    "description": "External dashboard parity: storage detail metrics (MetricsRegistry).",
    "metrics": ["storage_detail{metric=\"stored_object_bytes\"}"]
  },
  "node_health_gauges": {
    "description": "Node-health observable gauges (MetricsRegistry.cpp:997, :1081, :1102, :1161). server_info, build_info and db_metrics Observe unconditionally on every periodic export (build_info observes a literal 1; server_info and db_metrics read live services), so their series exist regardless of workload shape. complete_ledgers is the exception and is asserted on a narrower guarantee: its callback returns without observing when the range is empty (MetricsRegistry.cpp:1113-1114) and skips any segment that carries no '-' (:1122-1127), and a one-sequence range renders with no '-' (RangeSet.h:70-71), so it needs a complete range spanning at least two sequences. completeLedgers_ is filled by setFullLedger (LedgerMaster.cpp:862-863), which on a peered node is reached only from the publish path in doAdvance (LedgerMaster.cpp:1972) — closing a ledger is not enough, it has to validate. run-full-validation.sh waits for that before the workload starts, so on a healthy cluster the series always exists — a 5-node run yields 10 series, one start and one end per node. If this check ever fails, read the Step 3 output first: a run that logged 'No validated ledger' cannot produce this series and the cluster, not the exporter, is what broke.",
    "metrics": ["server_info", "build_info", "complete_ledgers", "db_metrics"]
  },
  "overlay_reduce_relay": {
    "description": "Transaction reduce-relay efficiency gauge (MetricsRegistry.cpp:1354, peer-network dashboard). Backed by Overlay::txMetrics(); TxMetrics::json() emits txr_selected_cnt / txr_suppressed_cnt / txr_not_enabled_cnt unconditionally (TxMetrics.cpp:121-127), so the gauge always reports at least the selected_peers series.",
    "metrics": ["reduce_relay_metrics"]
  },
  "overlay_overflow": {
    "description": "Job-queue transaction overflow total (MetricsRegistry.cpp:609, job-queue dashboard). An ObservableCounter that reads Overlay::getJqTransOverflow() and Observes unconditionally, so the series exists at value 0 even when no overflow occurs.",
    "metrics": ["jq_trans_overflow_total"]
  },
  "validation_lifetime_counters": {
    "description": "Lifetime validation agreement/miss ObservableCounters (MetricsRegistry.cpp:1636, :1658, validator-health dashboard). Both callbacks reconcile the tracker and Observe unconditionally, so the series exist even on a node that has not yet agreed or missed (value 0). Only existence is asserted, never the value — validation_missed_total legitimately dominates on a non-validating node.",
    "metrics": ["validation_agreements_total", "validation_missed_total"]
  },
  "not_asserted": {
    "description": "Emitted-and-dashboarded metrics deliberately left unasserted because they are workload-gated or defect-gated: the harness workload cannot guarantee they appear, and a check that fails on a healthy run is worse than no check. This group has no \"metrics\" key, so validate_telemetry.py skips it (validate_metrics iterates category_data.get(\"metrics\", [])). Promote an entry into an asserted group only after the workload is changed to guarantee it.",
    "metrics_excluded": {
      "rpc_method_errored_total": "MetricsRegistry.cpp:332-333, push counter. 'Errored' here means a thrown C++ exception, not an error status in the JSON reply: the only caller is PerfLogImp.cpp:409 under 'if (!finish)', reached only through PerfLogImp::rpcError (PerfLogImp.h:150-153), whose only call site is the catch (std::exception&) handler in RPCHandler.cpp:213. An RPC that returns an error status normally still takes the rpcFinish path at RPCHandler.cpp:190 and increments rpc_method_finished_total. That distinction mattered here while the generator still issued ripple_path_find: those calls were in fact refused — pathfinding is off on every harness node, so doRipplePathFind returns rpcNOT_SUPPORTED (see pathfind_fast_milliseconds below) — and it would have been easy to conclude from that alone that this counter must fire. It did not, because a refusal is a normal return, not a throw. The load was removed on 2026-08-25, so the harness no longer issues that command at all and the question is moot, but the distinction is kept on record because it is the one that decides this entry. Nothing in rpc_load_generator.py's remaining server_info / fee / account / ledger / tx / DEX mix is expected to throw either, so no series may ever be created.",
      "ledger_history_mismatch_total": "MetricsRegistry.cpp:377, incremented only from LedgerHistory.cpp:332 on a built-vs-validated ledger mismatch. On a healthy run it never fires — asserting it would mean asserting a defect.",
      "txq_expired_total": "MetricsRegistry.cpp:379, incremented only at TxQ.cpp:1428 when a queued tx expires past its LastLedgerSequence. CI does run a txq-burst phase (workload-profiles.json:41, 30 s of single-type Payment at 60 TPS), but that does not guarantee sustained fee escalation followed by expiry: a run in which every other check passed still exposed only txq_metrics and no txq_expired_total.",
      "txq_dropped_total": "MetricsRegistry.cpp:381, incremented only at TxQ.cpp:1302 / :1347 on queue-full admission refusal. Same reason as txq_expired_total.",
      "getobject_rejected_total": "GetObjectMetricNames.h:81, emitted from PeerImp.cpp:2725/:2743 only for a TMGetObjectByHash message refused as oversize or malformed_ledgerhash. A cooperating cluster never sends one.",
      "getobject_request_objects": "GetObjectMetricNames.h:86, emitted from PeerImp.cpp:2926 only while serving an inbound TMGetObjectByHash. The XRPL_METRIC_* macros create their instrument lazily on first use (MetricMacros.h:174-285), so no series exists until a peer actually requests objects by hash — which a 5-node cluster started at genesis and already in sync may never do.",
      "getobject_lookup_us": "GetObjectMetricNames.h:95, PeerImp.cpp:2929. Same lazy-creation and same inbound-request gate as getobject_request_objects.",
      "getobject_lookups_total": "GetObjectMetricNames.h:100, PeerImp.cpp:2949/:2956. Same gate.",
      "getobject_charge": "GetObjectMetricNames.h:105, PeerImp.cpp:2931. Same gate.",
      "rpc_size_bytes": "ServerHandler.cpp:191, group('rpc')->makeEvent('size', Unit::Bytes). The OTLP Prometheus exporter derives the metric-name suffix from the declared unit, so a byte unit yields rpc_size_bytes. The Unit::Bytes declaration itself landed earlier, in 76c9051203; what 24094e427b changed was the exporter finally consuming it, replacing a hardcoded CreateDoubleHistogram(name, 'Duration in ms', 'ms') with otelUnitDescription(unit)/otelUnitCode(unit), and that is what renamed the series off rpc_size_milliseconds and the millisecond bucket ladder. Neither name was ever recorded here, so the harness could confirm neither the rename nor a regression back onto that ladder. Notified from ServerHandler::processRequest:1133, the HTTP JSON-RPC path — it computes an HTTP status and appends a trailing newline — and the load generators are WebSocket-only, the same gate regression-metrics.json:4 records for rpc.process, so only the harness's handful of HTTP health polls reach it. Real coverage needs an HTTP JSON-RPC phase in rpc_load_generator.py; that is a workload change rather than a harness correction, and is deliberately out of scope here.",
      "rpc_time_milliseconds": "ServerHandler.cpp:192, group('rpc')->makeEvent('time') with the default millisecond unit. Notified from ServerHandler::processRequest:1129, the same HTTP JSON-RPC call site as rpc_size_bytes and behind the same WebSocket-only gate.",
      "pathfind_fast_milliseconds": "PathRequestManager.h:35, makeEvent('pathfind_fast') with the default millisecond unit, so the exported form is the pathfind_fast_milliseconds_bucket/_count/_sum triple and there is no bare series — the same convention io_latency and rpc_method_us follow, and rpc-pathfinding queries the _bucket. THE OPERATIVE BLOCKER IS THE CONFIG, NOT THE CALL GRAPH: pathfinding is disabled outright on every harness node, so no PathRequest is ever constructed and no pathfind_* histogram can exist. Config.cpp:725-726 sets pathSearchMax to 0 whenever a [validation_seed] or [validator_token] section is present ('By default, validators don't have pathfinding enabled'); run-full-validation.sh writes [validation_seed] into every generated node cfg (:308) and contains no [path_search], [path_search_fast] or [path_search_max] section to put it back (grep count 0 for path_search in that file — the only [path_search*] sections in docker/telemetry/ are in xrpld-telemetry.cfg and xrpld-telemetry-mainnet.cfg, neither of which the harness uses); and doRipplePathFind returns rpcNOT_SUPPORTED at RipplePathFind.cpp:48-49, before context.loadType is set and before any branch on the ledger parameter. That config gate is why the metric could never appear even while the generator was issuing the command (a 3% ripple_path_find weight, removed on 2026-08-25 precisely because every one of those calls was refused at the front door); the harness now issues no path-finding RPC at all, so there are two independent reasons. Read that first: the structural argument below is correct and matters if pathfinding is ever enabled, but it is not why the metric is missing today. STRUCTURAL ARGUMENT (verified, applies once pathSearchMax is non-zero): reportFast's only caller is PathRequest.cpp:852, inside the 'if (fast && quickReply_ == {})' branch of PathRequest::doUpdate. The only doUpdate call that passes fast=true is in PathRequest::doCreate (PathRequest.cpp:259), guarded by '!hasCompletion()'. Both ripple_path_find entry points construct the PathRequest with a completion function, so hasCompletion() (:161-164) is true and the fast pass is skipped: with no ledger specified doRipplePathFind goes to makeLegacyPathRequest, which passes the coroutine-post lambda (RipplePathFind.cpp:140-160); with a ledger specified it goes to doLegacyPathRequest, which passes an empty-body but non-null lambda (PathRequestManager.cpp:317). Only the path_find streaming subscription reaches reportFast, because makePathRequest builds the request from a subscriber with no completion (PathRequestManager.cpp:261). The load generator has never used path_find: it fires one request per send and awaits one reply, which a streaming subscription does not fit. Covering this metric therefore needs both a [path_search_max] override (or a non-validator node) in run-full-validation.sh and a path_find subscription phase in the generator. Both are harness/workload changes and out of scope here. Grafana Cloud shows zero series in 180 days on the devnet nodes, consistent with those nodes receiving no pathfinding RPC.",
      "pathfind_full_milliseconds": "PathRequestManager.h:36, makeEvent('pathfind_full'); same histogram naming as pathfind_fast_milliseconds above. Notified from reportFull (:87-90) via PathRequest.cpp:857, the 'else if (!fast && fullReply_ == {})' branch. Blocked by exactly the same config gate as pathfind_fast_milliseconds, and the probability of emission under the harness workload is zero, not low: pathSearchMax is 0 on every harness node (Config.cpp:725-726 plus the [validation_seed] section at run-full-validation.sh:308, with no [path_search*] override anywhere in that file), so doRipplePathFind returns rpcNOT_SUPPORTED at RipplePathFind.cpp:48-49 and no PathRequest object is ever constructed for reportFull to fire from. An earlier revision of this entry described the path as reachable-but-probabilistic — emitting one ledger close behind the request via PathRequestManager::updateAll's one-shot branch (PathRequestManager.cpp:160-166) — and prescribed sending an explicit ledger_index from the generator so doLegacyPathRequest would call doUpdate(cache, false) synchronously (PathRequestManager.cpp:321). Both halves were wrong. The probability is zero rather than merely unreliable, and the prescribed remedy cannot work at all, because the pathSearchMax guard fires before the ledger parameter is read: adding ledger_index changes nothing while pathfinding is off. Since 2026-08-25 the generator issues no path-finding RPC either, so covering this metric needs a [path_search_max] override (or a non-validator node) in run-full-validation.sh AND the load restored — a harness-topology plus workload change, out of scope here. The workload README section 'Pathfinding is not exercised' holds the recipe. Grafana Cloud shows zero series in 180 days on the devnet nodes, consistent with those nodes receiving no pathfinding RPC rather than with a broken exporter.",
      "warn_total": "include/xrpl/resource/detail/Logic.h:41, makeMeter('warn'). makeMeter maps to CreateUInt64Counter (OTelCollector.cpp:878-881 -> :773-777), so the Prometheus exporter appends _total; the meter is created on the bare collector with no group, hence the unprefixed name. Incremented only at Logic.h:481, inside the 'if (notify)' branch reached when a consumer's balance crosses kWarningThreshold. A cooperating 5-node cluster plus a rate-limited load generator never charges a consumer that far, and Grafana Cloud confirms zero series in 180 days. Recorded explicitly because this was briefly mis-diagnosed as a phantom metric: the rpc-pathfinding panel that queries it is correct, and renders empty only because the condition has not occurred.",
      "drop_total": "include/xrpl/resource/detail/Logic.h:42, makeMeter('drop'); same CreateUInt64Counter mapping and same _total suffix as warn_total. Incremented only at Logic.h:505, when a consumer's balance is at or above kDropThreshold and the connection is dropped. Grafana Cloud shows 2 live series, so unlike warn_total this one does fire in the wild — but only on a genuinely abusive consumer, which the harness deliberately does not create, so it is condition-gated all the same. Its rpc-pathfinding panel is likewise correct rather than phantom.",
      "jobq_*_milliseconds, jobq_*_q_milliseconds": "This key is a pattern rather than a literal metric name — unlike every other entry in this map it stands for a whole family, one pair per job type. Created per job type in JobTypeData.h:97-98 from info.name() and info.name() + kSuffixQueued ('_q'), so the exported names are jobq_<jobtype>_milliseconds and jobq_<jobtype>_q_milliseconds with the job type lowercased by formatName. Which job types appear depends on which jobs a run happens to schedule, so no individual name is guaranteed. They are also rounded up to a whole millisecond at source (Event.h:47-51 applies ceil to a millisecond value type), which is why 6e2b2da772 moved the ledger-data-sync q-wait panels off jobq_<jobtype>_q_milliseconds_bucket onto job_queued_us_bucket — they are poor assertion targets regardless."
    }
  },
  "accounted_patterns": [
    {
      "pattern": "^jobq_[a-z]+_(waiting|running|deferred)$",
      "family": "Per-job-type queue-depth gauges: 35 non-special job types x 3 states = 105 families.",
      "reason": "Cannot be enumerated usefully. JobTypeData's constructor creates waiting/running/deferred for every non-special job type (JobTypeData.h:96-103) and JobQueue eagerly constructs a JobTypeData for every type in JobTypes (JobQueue.cpp:45-53), so the family's membership is derived mechanically from the JobTypes.h table rather than chosen. Six members are asserted by literal name in job_queue_per_type_gauges on consumer-coverage grounds; this pattern accounts for the rest so the reverse coverage check does not report 99 names that the contract has already reasoned about. Narrow by construction: every one of the 35 non-special job type names lowercases to ^[a-z]+$ with no digits and no underscores (verified mechanically against JobTypes.h:52-86), and the state suffix is a closed set of three, so nothing outside this family can match. A new job type therefore stays covered without an edit here, which is correct — adding a job type is not a telemetry design change. A renamed state, a job type containing a digit or an underscore, or any other new jobq gauge shape would surface as unaccounted."
    },
    {
      "pattern": "^jobq_[a-z]+(_q)?_milliseconds$",
      "family": "Per-job-type job-latency and queue-wait histogram families: 35 job types x 2 = 70 families, each exported as a _bucket/_count/_sum triple.",
      "reason": "The same mechanically derived family as the gauges above, in histogram form: JobTypeData.h:97-98 creates one Event from info.name() and one from info.name() + '_q'. Which job types actually record a sample depends on what a run schedules, which is why not_asserted.metrics_excluded carries these under a single pattern key rather than as literal names. The reverse coverage check matches this pattern against the base family name, after stripping the exporter's _bucket/_count/_sum suffix, so all three series of every triple are accounted for by this one entry. Same narrowness argument as above: the job type part is ^[a-z]+$ and the optional _q plus the _milliseconds unit suffix close the shape."
    },
    {
      "pattern": "^(getobject_account_state_node_get|getobject_account_state_node_share|getobject_cas_get|getobject_cas_share|getobject_fetch_pack_get|getobject_fetch_pack_share|getobject_get|getobject_ledger_get|getobject_ledger_share|getobject_share|getobject_transaction_get|getobject_transaction_node_get|getobject_transaction_node_share|getobject_transaction_share|getobject_transactions_get|have_transactions|ledger_account_state_node_get|ledger_account_state_node_share|ledger_data_account_state_node_get|ledger_data_account_state_node_share|ledger_data_get|ledger_data_share|ledger_data_transaction_node_get|ledger_data_transaction_node_share|ledger_data_transaction_set_candidate_get|ledger_data_transaction_set_candidate_share|ledger_get|ledger_share|ledger_transaction_node_get|ledger_transaction_node_share|ledger_transaction_set_candidate_get|ledger_transaction_set_candidate_share|overhead|overhead_cluster|overhead_manifest|overhead_overlay|proof_path_request|proof_path_response|proposals|proposals_duplicate|proposals_untrusted|replay_delta_request|replay_delta_response|requested_transactions|set_get|set_share|squelch|squelch_ignored|squelch_suppressed|total|transactions|transactions_duplicate|unknown|validations|validations_duplicate|validations_untrusted|validator_lists)_(bytes|messages)_(in|out)$",
      "family": "Overlay per-category traffic gauges: 57 categories x {bytes,messages} x {in,out} = 228 families, of which the four total_* are asserted literally in overlay_traffic.",
      "reason": "Also mechanically derived: OverlayImpl builds one TrafficGauges per entry of TrafficCount::counts_ (OverlayImpl.cpp:200-203), and TrafficGauges creates Bytes_In / Bytes_Out / Messages_In / Messages_Out for each (OverlayImpl.h:580-593), so the set is exactly the cross product of the category table with those four names. The 57 categories are the 56 entries of TrafficCount::toString's kCategoryMap (TrafficCount.h:239-294) plus 'unknown', which is what Category::Unknown falls through to because it has an entry in counts_ but none in the map. The categories are enumerated here rather than reduced to a shape, because unlike the job types they contain underscores: a structural pattern would have to be ^[a-z_]+_(bytes|messages)_(in|out)$, which would silently swallow any future non-overlay metric ending in _bytes_in. Enumerating means a new traffic category surfaces as unaccounted, which is the right outcome: overlay_traffic's description calls itself an explicit subset and should be revisited when the table grows. The category list was extracted mechanically from TrafficCount.h; regenerate it the same way rather than editing by hand."
    },
    {
      "pattern": "^(target_info|up|scrape_[a-z_]+|promhttp_[a-z_]+)$",
      "family": "Prometheus scrape metadata and exporter housekeeping series.",
      "reason": "Not xrpld telemetry, and not emitted by anything in this repository, so there is nothing for the contract to assert. prometheus.yml defines exactly one scrape job against the collector's Prometheus exporter (otel-collector:8889), and Prometheus synthesises up and the scrape_* family per target on every scrape. target_info is written by the collector's Prometheus exporter to carry the OTLP resource attributes, and several dashboards read it as label_values(target_info, service_instance_id). promhttp_* is the metrics handler's own counter set, present only if the exporter's handler exposes it. Listed so the reverse coverage check reports genuine telemetry gaps instead of scrape plumbing. Deliberately does not cover otelcol_*: the collector's internal telemetry defaults to port 8888 and otel-collector-config.yaml declares no service::telemetry block, so those series are not scraped here — if they ever appear, that is a collector configuration change worth seeing."
    }
  ],
  "grafana_dashboards": {
    "description": "All 15 Grafana dashboards provisioned on disk under docker/telemetry/grafana/dashboards/ (UID == file stem for every one). validate_dashboards() checks that each UID resolves via GET /api/dashboards/uid/<uid> and reports its panel count — it verifies provisioning and loadability, not panel data. log-derived-insights is included on that basis even though its panels are Loki-backed and CI runs with --skip-loki: the dashboard itself must still provision cleanly. Its panel data is not asserted anywhere.",
    "uids": [
      "rpc-performance",
      "transaction-overview",
      "consensus-health",
      "ledger-operations",
      "peer-network",
      "peer-quality",
      "fee-market",
      "job-queue",
      "validator-health",
      "node-health",
      "network-traffic",
      "rpc-pathfinding",
      "overlay-traffic-detail",
      "ledger-data-sync",
      "log-derived-insights"
    ]
  }
}