The Mainnet telemetry config carried ssl_verify=0, inherited when the file
was copied from the Devnet config. Mainnet fetches its validator lists from
public HTTPS publishers (vl.ripple.com, unl.xrplf.org) that present valid CA
certificates, so there is no self-signed certificate to work around.
With ssl_verify=0 both halves of TLS peer authentication are skipped on those
fetches: the certificate chain check and the RFC 6125 hostname match. The list
blob's own signature is still verified against the pinned publisher keys, so
this restores defense in depth rather than fixing a validator-injection path.
Restores the secure default and documents why it stays that way.
develop renamed xrpl::Resource to xrpl::resource. MetricsRegistry.cpp
declared `namespace resource = opentelemetry::sdk::resource` at file
scope, but both use sites are inside namespace xrpl::telemetry, where
inner-scope lookup finds the enclosing xrpl::resource first and never
reaches the alias. That namespace has no ResourceAttributes or Resource,
so the build failed on all four platforms:
error: no type named 'ResourceAttributes' in namespace 'xrpl::resource'
error: no member named 'Create' in namespace 'xrpl::resource';
did you mean 'creat'?
Rename the alias to otel_resource. Moving it inside xrpl::telemetry would
also compile, but only by shadowing xrpl::resource -- a reader at the use
site still could not tell which namespace `resource::` meant, and a later
using-directive would reintroduce the ambiguity. A distinct name removes
the collision by construction and matches the metric_sdk / otlp_http
aliases already in this file.
Rule D validated every dashboard label against L1 (*SpanNames.h) and L6
(MetricsRegistry) labels. LogQL labels have a third provenance neither
layer can resolve: they are minted by the collector's regex_parser named
captures (partition, severity) or by an in-query `| regexp` stage
(action, pk, state, mode, phase, jobname, ip, pubkey). Checking them
against L1/L6 reported ten violations for labels correct by
construction.
Make the rule datasource-aware instead of allowlisting a filename. The
dashboard JSON is parsed so each query can be attributed to its
datasource, and queries on a log datasource are skipped. The exemption is
per query, not per file, so a dashboard mixing Prometheus and Loki panels
still has its Prometheus panels validated.
Parsing the JSON also fixed a blind spot: label filters are stored with
backslash-escaped quotes (`label=~\"$v\"`), which the previous raw-text
regex could never match, so only the `sum by (...)` form was ever
checked. With the strings unescaped, 555 queries are now validated where
far fewer were before. That surfaced three legitimate label sources the
rule did not model, each fixed at its source rather than allowlisted:
- deployment_environment / xrpl_network_type: resource attributes the
collector promotes onto metric datapoints. Derived from the config's
resource_metrics_key_attributes, so a new key is picked up
automatically, in both dotted and underscore forms.
- resource.service.instance.id: strips to a dotted service-identity
key, which builtins only held in underscore form.
- name: the TraceQL span-name intrinsic, alongside duration and kind.
A file that does not parse falls back to the raw-text scan, which checks
every query rather than skipping it; JSON validity is already enforced by
the prettier pre-commit hook.
Adds 10 tests: the exemption, per-query scoping in a mixed dashboard,
target-inherits-panel datasource, no sideways inheritance leak, nested
row panels, TraceQL intrinsics, the malformed-JSON fallback, and the
collector-promotion helper.
09-data-collection-reference.md: keep phase-9's structure, which relocated
the §2a call-site-metrics content into §5b (3860c93db2). The incoming §2a
block is superseded; its snake_case namespace fix is already applied to
phase-9's copy of the same text.
The stat block was relaid two-per-row in the previous commit, but the reflow
shifted subsequent panels by a delta computed against the block's old height.
The block shrank, so the delta left a 12-grid-line dead band between the stat
row and "Node Operating State Transitions".
Recompute every panel's y by walking the list in order and pairing half-width
panels, rather than patching offsets incrementally. The layout is now derived
from panel order and heights, so this class of drift cannot recur.
Verified: 192 grid lines, fully covered, no empty band, no overlaps, and no grid
line occupied by more than two panels.
Fixes found while validating the Log-Derived Insights dashboard against live
Loki and Tempo. Every change was verified by executing the query, not by
inspection.
loki.yaml — the ConsensusLedgerHash derived field never matched anything:
- The TraceQL query named `.xrpl.consensus.ledger_id`. The attribute is
`consensus_ledger_id` (ConsensusSpanNames.h); live Tempo lists no dotted or
xrpl-prefixed variant among its span attributes.
- The bare `.` intrinsic scope cannot match a span attribute regardless of
name; TraceQL requires `span.`.
Corrected to `{span.consensus_ledger_id="..."}`, which returns the expected
consensus.round trace for a real ledger hash.
log-derived-insights.json:
- Stat units `suffix: <noun>` rendered raw integers, so large values printed
in full. `si:<noun>` is unsafe because Grafana parses the leading letter as
an SI prefix (`si:fee` rendered "29.0 Mee"). Switched to `short`, the
convention already used by 42 panels; the noun is in each panel title.
- Stat panels showed a single fleet-wide tile labelled "Value #A". Grafana's
Loki backend does not name binary-operation frames from legendFormat, so the
ratio panel could not resolve per-node labels. Replaced it with a plain sum
of duplicate fetches; the ratio remains available as a timeseries. Added an
explicit legendFormat to all eight stat panels.
- The ledger-fetch regex required `good:N dupe:N` together, but
SHAMapAddNode::get() omits any zero counter. That dropped 32% of lines --
disproportionately the efficient fetches -- biasing the ratio upward. Each
field now matches independently.
- state-timeline used `spanNulls: true`, which spans gaps indefinitely and
would render an outage as continuous. Set to the 30-minute threshold.
- Stat panels relaid two-per-row and given a tooltip, per dashboard guidelines.
telemetry-runbook.md:
- LogQL examples used line filters where structured metadata exists.
`|= "ERR"` also matches the literal in a message body: measured 423 ERR
lines plus 4 DBG lines per 6h. Replaced with severity, partition, and
trace_id field filters.
- Documented the new dashboard, its debug-log requirement, and the LogQL traps
these fixes exposed.
- Corrected the dashboard count, which was already stale at eleven against
fifteen on disk, and named the six with no reference section.
The Phase 7 section existed twice in this file; the copies disagreed on this
one label. Consolidating on the single copy kept the older wording, but Jaeger
was removed from the project earlier in this chain, so Tempo is the only trace
backend the diagram should name.
These changes were developed on the phase-10 branch but belong to content this
branch and its upstreams introduced. Carrying them on phase-10 made its PR diff
report churn in files phase-10 does not own, and left each PR claiming a scope
that did not match its contents.
Moved here from phase-10 (identical content, no functional change):
- Dashboards: all 14 existing boards plus the new log-derived-insights board.
- Docs: telemetry-runbook.md (minus the workload/benchmark sections, which
describe phase-10 tooling) and the new telemetry-glossary.md.
- Grafana Cloud + Alloy export path: collector config, compose override, the
two .env examples and alloy/config.alloy.
- Local stack: otel-collector-config.yaml gains sub-millisecond and
second-scale spanmetrics buckets, pins unit=ms, and promotes
close_time_correct; integration-test.sh and TESTING.md follow.
- Node configs: exported_instance -> service_instance_id in comments; the
mainnet sample now logs at warning to bound log volume.
- Metrics code: Telemetry.cpp builds the metrics pipeline in the constructor
via initMetrics() so the global MeterProvider is published before any
subsystem creates a beast::insight instrument, and the histogram view keeps
each instrument's own name instead of collapsing them under one series.
MetricsRegistry gains a last_close_time gauge and skips negative job-queue
durations. OTelCollector drops an unused accessor.
- Naming CI: xrpl_work_item joins EXTERNAL_INFRA_LABELS and Rule E accepts the
dotted perf-iac resource-attribute form. This must travel with the
dashboards and runbook that reference those labels, or the rules fail.
- Doxygen input glob no longer recurses dot-directories.
Sections describing phase-10 tooling stay on phase-10 and keep their
"Future Enhancement" / "Planned, not yet implemented" markers here; phase-10
removes those markers when it lands the tooling.
