Grafana 13 enables the renderAuthJWT feature toggle by default, so the image
renderer silently drops render requests carrying the default token and alert
notifications arrive with no panel image. The runbook's alerting section
covered delivery but not this failure mode.
Adds how the token is wired (one variable feeding both services, so they cannot
drift), how to override it, and the two-command check for when images stop
appearing.
The renderer was the only service in the stack on a floating :latest tag, so
the stack was not reproducible: a renderer release could change under a
checkout that had not been touched. Pins it to v5.12.0, matching how every
other service here is pinned.
Also wires the shared render token. Grafana 13 enables the renderAuthJWT
feature toggle by default, and the renderer rejects requests unless both sides
present the same non-default token, so pinning alone would leave rendering
broken once the Grafana bump merges forward. Both values come from the same
GF_RENDERING_RENDERER_TOKEN variable and cannot drift apart; the local default
keeps `docker compose up` working with no setup.
Verified `docker compose config` resolves and both AUTH_TOKEN and
GF_RENDERING_RENDERER_TOKEN expand to the same value.
The Mainnet telemetry config carried ssl_verify=0, inherited when the file
was copied from the Devnet config. Mainnet fetches its validator lists from
public HTTPS publishers (vl.ripple.com, unl.xrplf.org) that present valid CA
certificates, so there is no self-signed certificate to work around.
With ssl_verify=0 both halves of TLS peer authentication are skipped on those
fetches: the certificate chain check and the RFC 6125 hostname match. The list
blob's own signature is still verified against the pinned publisher keys, so
this restores defense in depth rather than fixing a validator-injection path.
Restores the secure default and documents why it stays that way.
develop renamed xrpl::Resource to xrpl::resource. MetricsRegistry.cpp
declared `namespace resource = opentelemetry::sdk::resource` at file
scope, but both use sites are inside namespace xrpl::telemetry, where
inner-scope lookup finds the enclosing xrpl::resource first and never
reaches the alias. That namespace has no ResourceAttributes or Resource,
so the build failed on all four platforms:
error: no type named 'ResourceAttributes' in namespace 'xrpl::resource'
error: no member named 'Create' in namespace 'xrpl::resource';
did you mean 'creat'?
Rename the alias to otel_resource. Moving it inside xrpl::telemetry would
also compile, but only by shadowing xrpl::resource -- a reader at the use
site still could not tell which namespace `resource::` meant, and a later
using-directive would reintroduce the ambiguity. A distinct name removes
the collision by construction and matches the metric_sdk / otlp_http
aliases already in this file.
Rule D validated every dashboard label against L1 (*SpanNames.h) and L6
(MetricsRegistry) labels. LogQL labels have a third provenance neither
layer can resolve: they are minted by the collector's regex_parser named
captures (partition, severity) or by an in-query `| regexp` stage
(action, pk, state, mode, phase, jobname, ip, pubkey). Checking them
against L1/L6 reported ten violations for labels correct by
construction.
Make the rule datasource-aware instead of allowlisting a filename. The
dashboard JSON is parsed so each query can be attributed to its
datasource, and queries on a log datasource are skipped. The exemption is
per query, not per file, so a dashboard mixing Prometheus and Loki panels
still has its Prometheus panels validated.
Parsing the JSON also fixed a blind spot: label filters are stored with
backslash-escaped quotes (`label=~\"$v\"`), which the previous raw-text
regex could never match, so only the `sum by (...)` form was ever
checked. With the strings unescaped, 555 queries are now validated where
far fewer were before. That surfaced three legitimate label sources the
rule did not model, each fixed at its source rather than allowlisted:
- deployment_environment / xrpl_network_type: resource attributes the
collector promotes onto metric datapoints. Derived from the config's
resource_metrics_key_attributes, so a new key is picked up
automatically, in both dotted and underscore forms.
- resource.service.instance.id: strips to a dotted service-identity
key, which builtins only held in underscore form.
- name: the TraceQL span-name intrinsic, alongside duration and kind.
A file that does not parse falls back to the raw-text scan, which checks
every query rather than skipping it; JSON validity is already enforced by
the prettier pre-commit hook.
Adds 10 tests: the exemption, per-query scoping in a mixed dashboard,
target-inherits-panel datasource, no sideways inheritance leak, nested
row panels, TraceQL intrinsics, the malformed-JSON fallback, and the
collector-promotion helper.
09-data-collection-reference.md: keep phase-9's structure, which relocated
the §2a call-site-metrics content into §5b (3860c93db2). The incoming §2a
block is superseded; its snake_case namespace fix is already applied to
phase-9's copy of the same text.
The stat block was relaid two-per-row in the previous commit, but the reflow
shifted subsequent panels by a delta computed against the block's old height.
The block shrank, so the delta left a 12-grid-line dead band between the stat
row and "Node Operating State Transitions".
Recompute every panel's y by walking the list in order and pairing half-width
panels, rather than patching offsets incrementally. The layout is now derived
from panel order and heights, so this class of drift cannot recur.
Verified: 192 grid lines, fully covered, no empty band, no overlaps, and no grid
line occupied by more than two panels.
Fixes found while validating the Log-Derived Insights dashboard against live
Loki and Tempo. Every change was verified by executing the query, not by
inspection.
loki.yaml — the ConsensusLedgerHash derived field never matched anything:
- The TraceQL query named `.xrpl.consensus.ledger_id`. The attribute is
`consensus_ledger_id` (ConsensusSpanNames.h); live Tempo lists no dotted or
xrpl-prefixed variant among its span attributes.
- The bare `.` intrinsic scope cannot match a span attribute regardless of
name; TraceQL requires `span.`.
Corrected to `{span.consensus_ledger_id="..."}`, which returns the expected
consensus.round trace for a real ledger hash.
log-derived-insights.json:
- Stat units `suffix: <noun>` rendered raw integers, so large values printed
in full. `si:<noun>` is unsafe because Grafana parses the leading letter as
an SI prefix (`si:fee` rendered "29.0 Mee"). Switched to `short`, the
convention already used by 42 panels; the noun is in each panel title.
- Stat panels showed a single fleet-wide tile labelled "Value #A". Grafana's
Loki backend does not name binary-operation frames from legendFormat, so the
ratio panel could not resolve per-node labels. Replaced it with a plain sum
of duplicate fetches; the ratio remains available as a timeseries. Added an
explicit legendFormat to all eight stat panels.
- The ledger-fetch regex required `good:N dupe:N` together, but
SHAMapAddNode::get() omits any zero counter. That dropped 32% of lines --
disproportionately the efficient fetches -- biasing the ratio upward. Each
field now matches independently.
- state-timeline used `spanNulls: true`, which spans gaps indefinitely and
would render an outage as continuous. Set to the 30-minute threshold.
- Stat panels relaid two-per-row and given a tooltip, per dashboard guidelines.
telemetry-runbook.md:
- LogQL examples used line filters where structured metadata exists.
`|= "ERR"` also matches the literal in a message body: measured 423 ERR
lines plus 4 DBG lines per 6h. Replaced with severity, partition, and
trace_id field filters.
- Documented the new dashboard, its debug-log requirement, and the LogQL traps
these fixes exposed.
- Corrected the dashboard count, which was already stale at eleven against
fifteen on disk, and named the six with no reference section.
