Phase-10 independently instrumented the peer object-fetch path while this branch instrumented fresh-node sync, so the two overlapped in three places. Resolved by keeping each side's stronger implementation rather than shipping both. Per-job-type waiting/running/deferred existed twice. Phase-10's version survives: it publishes per-type gauges from JobQueue::collect(), which snapshots under the queue lock and publishes after releasing it, a deliberate lock-order fix against the collector's own lock. This branch's jobq_backlog gauge and the JobQueue::getJobTypeCounts() accessor that fed it are removed, along with their panels, assertions and reference rows. jobq_saturation stays: it reports the whole worker pool, which phase-10 has no equivalent for. The histogram view helper also existed twice with identical bodies under two names; one survives, and the microsecond ladder is now the named array rather than boundaries repeated inline. The job_type label was declared twice, once as a file-local constant invisible to the naming check; both it and handler now come from the constants header. Two things phase-10 adds are complementary, not duplicates, and are kept as they are: the handler label, which separates the two request kinds that both report as the same job type, and getobject_rejected_total, which counts malformed requests where this branch's serve_refused_total counts requests this node declined to serve. Also fixes two naming-check failures that pre-date this merge on phase-10. The check derived label keys only from namespaced constants, so it could not see the per-subsystem headers' flat k-prefixed style and rejected dashboards querying labels the code really emits. It now reads both styles, with the enforcement rules unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OTel naming-consistency check
check_otel_naming.py enforces the OpenTelemetry span-attribute naming
convention documented in
CONTRIBUTING.md
across every layer of the telemetry pipeline. The *SpanNames.h constants are
the single source of truth (L1); every other layer must agree with them.
Running locally
python .github/scripts/otel-naming/check_otel_naming.py
It takes no arguments, can be run from any directory inside the repo, and uses
only the Python standard library (no pip install, matching the levelization
check). A non-zero exit code means a violation was found; the output lists each
violation as RULE | location | token | expected.
What it checks
The valid key set is derived dynamically from the OTel code — there is no hardcoded allowlist:
- L1 keys come from the
namespace attr { ... }blocks of every*SpanNames.h, resolving themakeStr("x")/join(seg::a, seg::b)DSL (cross-file, sojoin(seg::rpc, ...)resolvesseg::rpcfrom the baseSpanNames.h). Each constant is resolved against its own header, so two headers that define a same-named constant (e.g. a baseattr::ledgerHashand a domainattr::ledgerHash) each contribute their real wire key — a later header cannot clobber an earlier one's value in a flat table. - Legitimate dotted keys = ONLY the keys the code actually sets as resource
attributes, i.e. the entries inside
Telemetry.cpp'sResource::Create({...})call: thesemconv::service::*keys (service.*) plus anyattr::<name>constants passed there (xrpl.network.*). A dotted key that is declared in a header but never set as a resource attr is a span attribute in resource clothing — a Rule-A violation, even if it lives in the baseSpanNames.h. - L1-metrics — instrument names, label keys and bounded label values come
from the
namespace metric/namespace label/namespace lvalblocks of every*MetricNames.h, read asinline constexpr char NAME[] = "wire";. These headers deliberately do not use themakeStr/StaticStrDSL the span headers use: the OTel C++ API takesnostd::string_view, which constructs fromchar const*but has no constructor fromstd::string_view, so aStaticStrwill not compile in an instrument-name or label-key position.
Rules (each fails the build, when its inputs are present)
| Rule | Check |
|---|---|
| A | No stray dotted span-attribute key (only the derived resource keys may be dotted). |
| G | Attribute keys are lower_snake_case (^[a-z][a-z0-9_]*$ per dot-segment) — no camelCase, UPPERCASE, or spaces. |
| F | No string literals as attribute keys or span-name arguments in setAttribute/addEvent/span/rootSpan/childSpan (rootSpan shares span's (cat, prefix, name) signature). Attribute values are exempt (runtime data); *SpanNames.h definitions and test files are exempt. |
| B | Every collector spanmetrics.dimensions name exists in the L1 key set. |
| C | Every Tempo span-filter tag exists in the L1 key set. |
| D | Every dashboard label resolves to an L1 span attribute, a native-metric label (L6, emitted by MetricsRegistry), or a Prometheus/Grafana builtin. TraceQL scope prefixes (span./resource./…) are stripped before the L1 lookup. |
| E | No dotted xrpl.<domain>.<field> attribute key in the runbook (only the L1 resource attrs xrpl.network.* may be dotted). Span names, filenames, OTel-standard keys, and metric labels are not flagged. |
| I | No string literals as metric instrument names or label keys — the mirror of Rule F. Applies to the name passed to an XRPL_METRIC_* macro or a meter->Create* factory and to the label keys in its label set. Label values, descriptions, *MetricNames.h, MetricMacros.h and test files are exempt. Scoped by metric family (first underscore segment): declaring a constant opts that family in, so the metric surface can be converted subsystem by subsystem. Unconverted families warn as Rule L. |
| J | Metric instrument names follow the suffix conventions: lower_snake_case, no xrpld_/xrpl_ prefix (the exporter adds it), a counter ends _total, a histogram ends _us/_ms/_seconds, a gauge does not end _total. The instrument kind is read from the emit site, never guessed from words in the name — so a multi-series gauge carrying units in its label values (e.g. nodestore_latency observing write_mean_us) is not a violation. |
| K | Every metric named in docker/telemetry/workload/expected_metrics.json resolves to a declared constant, so a rename in code cannot leave the workload validator asserting a name nothing emits. PromQL selectors (m{label="v"}) and exporter-appended histogram suffixes (_bucket/_count/_sum) are normalized away first; groups fed by another emit path (statsd_gauges, statsd_counters, spanmetrics) are out of scope by design. |
Rule F runs unconditionally (it is a purely syntactic check on the
call-sites and needs no *SpanNames.h), so a code path that calls
SpanGuard::span/setAttribute directly without ever defining a header is
still caught.
Warnings (printed, never fail the build)
| Rule | Check |
|---|---|
| H | A namespace-qualified constant (e.g. foo::bar::myKey) used at a telemetry call-site is not defined in any *SpanNames.h. The constant should live in the proper header; defining it in-place bypasses rules A/G/F. Warns rather than fails — the argument may be a legitimately dynamic value, and the header may live on a later branch. Bare locals and std:: names are not warned. |
| L | A literal metric name in a family that has no *MetricNames.h constants yet. Rule I's ratchet defers these instead of failing the build on the whole pre-existing metric surface at once; the warning keeps the outstanding conversion work visible rather than silently accepted. |
Presence-gated
Every rule runs only when the source files it needs are present in the tree
and is otherwise skipped (printed as SKIP: <rule> — <reason>), never failed.
This keeps the check correct no matter how telemetry work is split across PRs —
a stacked chain, one large PR, or independent per-stage PRs where (for example)
the collector config lands before the dashboards. The collector/Tempo/dashboard/
runbook layers are introduced in later phases; on a branch without them, only
the L1-intrinsic rules (A, G, F) run.