Four defects found by a query-correctness audit of all 16 dashboards, all
scoped to panels this branch owns.
job-queue "Current Job Latency (p99 Gauge)": the histogram by-clause dropped
service_instance_id, collapsing every node into one fleet-wide p99 and hiding
a slow node. Measured: gauge read 1325us while the worst node was 1868us. The
panel's displayName already referenced service_instance_id, so it was also
rendering an empty label. Now groups by service_instance_id and xrpl_work_item,
matching sibling panels 5, 6 and 7.
ledger-data-sync "NodeStore Read Latency (Bottleneck Discriminator)": replaced
clamp_min(<denominator>, 1) with (<denominator> > 0). clamp_min clamps the
value, not just the zero case, so any node reading below 1/s got a fabricated
denominator. Demonstrated with a zero-rate denominator: clamp_min invents
40.2/40.3/11.2/9.2 where the > 0 guard correctly returns no data. This panel is
the bottleneck discriminator, read during a stall, which is exactly when the
read rate collapses and the clamp is most wrong. Panels 21 and 23 carry the
same defect but originate on phase-7 and are fixed there.
node-health thresholds: percentunit fields are compared against the raw value,
so a step of 80 needed 8000% and could never fire. Rescaled panels 74, 81 and
85 to 0.8. Panel 81 is a found-ratio where high is healthy, so its bands were
also inverted. Note these three panels use palette-classic with thresholdsStyle
off, so the steps are currently dormant rather than visibly wrong.
node-health panels 81 and 85 descriptions: both described the multi-series
panels they were split from. Panel 81 carried a byte-identical copy of panel
80's text, promising three plotted rate lines where it draws a single bounded
ratio; panel 85's text described read-thread gauges absent from its expression.
Rewritten to match the actual queries.
The Transaction Overview panel "Queue Bypass Ratio (Direct Apply vs
Enqueue)" reported a confident 0.50 on every node while the true bypass
rate was zero. The two spans it divided are not disjoint alternatives:
txq.apply_direct is a child of txq.enqueue. TxQ.cpp creates the
apply_direct span as the first statement of tryDirectApply(), ahead of
the account, sequence and fee-level guards, and tryDirectApply() is
called from inside the live enqueue scope. The span therefore counts
attempts, so the denominator direct + enqueue counts each transaction
twice and pins the ratio to one half algebraically.
Measured on a four-node fleet: 6082443 direct against 6082877 enqueue
over the same population, panel output 0.5000170 on three nodes and
0.5000000 on the fourth. Grouping txq.enqueue by txq_status over seven
days returns only "rejected" -- no transaction has ever taken the
direct-apply path.
Remove the panel rather than repoint it. A correct expression using
txq_status as the disjoint discriminator would render permanently
empty on this fleet, which reads no better than a wrong number.
Widen the band partner "TxQ Enqueue Rate by Transaction Type" from 12
to 24 columns so the y=48 band still fills the grid. Every band in all
ten dashboards sums to 24 columns; leaving a half-width hole would be
the only exception. Panel order and every other panel's position,
width and height are unchanged.
The runbook already listed txq.apply_direct as available but not
paneled, so that row becomes accurate. Rows describing the span itself
are untouched -- the span and its metric are unchanged.
Panels 21 (NuDB Read Latency) and 23 (NuDB Read Found Ratio) on the
Ledger Data & Sync dashboard guarded their divisor with
clamp_min(<denominator>, 1). clamp_min raises the value, not just the
zero case, so any node reading fewer than 1 block per second was
divided by a fabricated 1 instead of by its real read count.
Replace the clamp with the filter (<denominator> > 0). A comparison
without the bool modifier drops the sample rather than rewriting it, so
these panels now show no data instead of a wrong number.
Measured over 7 days: five nodes fall below 1 read/s. On validator-0 the
clamp reported 2.726 us/read against a true 5.493, and on nonUNLmalloc-tc
it reported 0 us/read, which cannot occur. The error is largest exactly
when panel 21 is used as the bottleneck discriminator during a stall,
because that is when the read rate collapses toward zero.
Matches the existing idiom on the same nodestore_state metric family in
the NodeStore Write vs Read Latency panel.
The renderer was the only service in the stack on a floating :latest tag, so
the stack was not reproducible: a renderer release could change under a
checkout that had not been touched. Pins it to v5.12.0, matching how every
other service here is pinned.
Also wires the shared render token. Grafana 13 enables the renderAuthJWT
feature toggle by default, and the renderer rejects requests unless both sides
present the same non-default token, so pinning alone would leave rendering
broken once the Grafana bump merges forward. Both values come from the same
GF_RENDERING_RENDERER_TOKEN variable and cannot drift apart; the local default
keeps `docker compose up` working with no setup.
Verified `docker compose config` resolves and both AUTH_TOKEN and
GF_RENDERING_RENDERER_TOKEN expand to the same value.
Bumps the Loki pin this branch introduced, and updates the two docs that
name the version so they do not drift from the compose file:
- OpenTelemetryPlan/09-data-collection-reference.md (log backend section)
- OpenTelemetryPlan/Phase8_taskList.md (compose snippet)
The service runs the config bundled in the image rather than one from this
repo; `-verify-config` reports "config is valid" under 3.7.6, and native OTLP
ingestion at /otlp is unchanged across 3.4 -> 3.7.
Moves off the v2.53 LTS line, which this branch pinned, onto the current
v3 release, and corrects the accompanying comment that named the old line.
Verified against the new image rather than assumed, since this crosses a
major version:
- prometheus.yml passes `promtool check config` under v3.13.2
- all 391 unique dashboard PromQL queries parse under the v3 parser
- all 13 alert-rule expressions parse under the v3 parser
The config uses none of the surfaces v3 changed: no remote_write, no
holt_winters, no offset modifiers.
Bumps the three images this branch pins:
otel-collector-contrib 0.121.0 -> 0.158.0
tempo 2.7.2 -> 2.9.4
grafana 11.5.2 -> 13.1.2
Verified locally against the new images: the collector config passes
`validate` under 0.158.0 and tempo.yaml passes `-config.verify=true` under
2.9.4, both unchanged. The collector's log path uses the generic otlphttp
exporter (otlphttp/loki), not the dedicated loki exporter removed upstream in
v0.147.0, so the pipeline is unaffected by that removal.
Grafana crosses two majors. Operator-visible consequences are handled on the
branches that own the affected files: Grafana 13 enables the renderAuthJWT
feature toggle by default, so the image renderer now requires a matching
renderer_token on both the server and the renderer container.
The Mainnet telemetry config carried ssl_verify=0, inherited when the file
was copied from the Devnet config. Mainnet fetches its validator lists from
public HTTPS publishers (vl.ripple.com, unl.xrplf.org) that present valid CA
certificates, so there is no self-signed certificate to work around.
With ssl_verify=0 both halves of TLS peer authentication are skipped on those
fetches: the certificate chain check and the RFC 6125 hostname match. The list
blob's own signature is still verified against the pinned publisher keys, so
this restores defense in depth rather than fixing a validator-injection path.
Restores the secure default and documents why it stays that way.
The stat block was relaid two-per-row in the previous commit, but the reflow
shifted subsequent panels by a delta computed against the block's old height.
The block shrank, so the delta left a 12-grid-line dead band between the stat
row and "Node Operating State Transitions".
Recompute every panel's y by walking the list in order and pairing half-width
panels, rather than patching offsets incrementally. The layout is now derived
from panel order and heights, so this class of drift cannot recur.
Verified: 192 grid lines, fully covered, no empty band, no overlaps, and no grid
line occupied by more than two panels.
Fixes found while validating the Log-Derived Insights dashboard against live
Loki and Tempo. Every change was verified by executing the query, not by
inspection.
loki.yaml — the ConsensusLedgerHash derived field never matched anything:
- The TraceQL query named `.xrpl.consensus.ledger_id`. The attribute is
`consensus_ledger_id` (ConsensusSpanNames.h); live Tempo lists no dotted or
xrpl-prefixed variant among its span attributes.
- The bare `.` intrinsic scope cannot match a span attribute regardless of
name; TraceQL requires `span.`.
Corrected to `{span.consensus_ledger_id="..."}`, which returns the expected
consensus.round trace for a real ledger hash.
log-derived-insights.json:
- Stat units `suffix: <noun>` rendered raw integers, so large values printed
in full. `si:<noun>` is unsafe because Grafana parses the leading letter as
an SI prefix (`si:fee` rendered "29.0 Mee"). Switched to `short`, the
convention already used by 42 panels; the noun is in each panel title.
- Stat panels showed a single fleet-wide tile labelled "Value #A". Grafana's
Loki backend does not name binary-operation frames from legendFormat, so the
ratio panel could not resolve per-node labels. Replaced it with a plain sum
of duplicate fetches; the ratio remains available as a timeseries. Added an
explicit legendFormat to all eight stat panels.
- The ledger-fetch regex required `good:N dupe:N` together, but
SHAMapAddNode::get() omits any zero counter. That dropped 32% of lines --
disproportionately the efficient fetches -- biasing the ratio upward. Each
field now matches independently.
- state-timeline used `spanNulls: true`, which spans gaps indefinitely and
would render an outage as continuous. Set to the 30-minute threshold.
- Stat panels relaid two-per-row and given a tooltip, per dashboard guidelines.
telemetry-runbook.md:
- LogQL examples used line filters where structured metadata exists.
`|= "ERR"` also matches the literal in a message body: measured 423 ERR
lines plus 4 DBG lines per 6h. Replaced with severity, partition, and
trace_id field filters.
- Documented the new dashboard, its debug-log requirement, and the LogQL traps
these fixes exposed.
- Corrected the dashboard count, which was already stale at eleven against
fifteen on disk, and named the six with no reference section.
These changes were developed on the phase-10 branch but belong to content this
branch and its upstreams introduced. Carrying them on phase-10 made its PR diff
report churn in files phase-10 does not own, and left each PR claiming a scope
that did not match its contents.
Moved here from phase-10 (identical content, no functional change):
- Dashboards: all 14 existing boards plus the new log-derived-insights board.
- Docs: telemetry-runbook.md (minus the workload/benchmark sections, which
describe phase-10 tooling) and the new telemetry-glossary.md.
- Grafana Cloud + Alloy export path: collector config, compose override, the
two .env examples and alloy/config.alloy.
- Local stack: otel-collector-config.yaml gains sub-millisecond and
second-scale spanmetrics buckets, pins unit=ms, and promotes
close_time_correct; integration-test.sh and TESTING.md follow.
- Node configs: exported_instance -> service_instance_id in comments; the
mainnet sample now logs at warning to bound log volume.
- Metrics code: Telemetry.cpp builds the metrics pipeline in the constructor
via initMetrics() so the global MeterProvider is published before any
subsystem creates a beast::insight instrument, and the histogram view keeps
each instrument's own name instead of collapsing them under one series.
MetricsRegistry gains a last_close_time gauge and skips negative job-queue
durations. OTelCollector drops an unused accessor.
- Naming CI: xrpl_work_item joins EXTERNAL_INFRA_LABELS and Rule E accepts the
dotted perf-iac resource-attribute form. This must travel with the
dashboards and runbook that reference those labels, or the rules fail.
- Doxygen input glob no longer recurses dot-directories.
Sections describing phase-10 tooling stay on phase-10 and keep their
"Future Enhancement" / "Planned, not yet implemented" markers here; phase-10
removes those markers when it lands the tooling.
ValidatedLedgerStale fired on every node, healthy included.
LedgerMaster::getValidatedLedgerAge() returns weeks{2} (1209600s) as a
SENTINEL when no validated ledger exists, not as a measurement. The rule
read that as "14 days stale". Measured over six days it produced sustained
firing on all nine nodes. Excluding the exact sentinel value drops that to
zero while still tracking real staleness.
ManifestFloodInbound at 50 kB/s was routine paging: ~41 sustained 5-minute
samples across six healthy nodes in six days. Healthy p99 is 1.0-1.8 kB/s
and real storms peak at 2.7 MB/s, so 512 kB/s sits ~280x above normal and
~5x below the peaks, cutting sustained samples to 2.
Both thresholds were previously justified from a 24-hour window, which was
too short to expose either problem.
Three dashboards had layout faults that Grafana renders as visible
artifacts.
On Node Health, two row headers carried the same title as the panel
directly beneath them -- "Cache Hit Rates" and "Object Instance Counts"
each appeared twice, once as a row bar and once as a panel. Rename the
rows to "Caches" and "Object Instances"; the panels keep their names.
Worse on the same dashboard, seven bands summed to 48 columns on a
24-column grid, because the Extended Metrics block reused the y-space
already occupied by NodeStore I/O. Grafana cannot lay that out
deterministically, which is what produced the artifacts.
RPC Performance and Transaction Overview each had blank bands -- one
row and eight rows respectively -- left behind by earlier panel moves.
Repack all three grids so every band fits 24 columns, each band starts
where the previous one ends, and rows stay full-width single-height
anchors. Panel order, panel count and every panel's own width and
height are unchanged; only position moves.
The two key-job latency panels on node-health declared milliseconds
while querying `job_running_us` / `job_queued_us`, which record
microseconds. Every reading was a thousand times too large: the p95 for
acceptLedger, 241495us, rendered as "241 sec" instead of 241ms.
job-queue.json already read the same metrics as microseconds, so the
two dashboards disagreed by 1000x on identical data.
Also replace the generic `ops`, `cps` and `cpm` units, which Grafana
renders as the literal "ops/s", "counts/s" and "counts/min", with
custom-suffix units naming what each panel counts -- jobs, ledgers,
validations, lookups. The per-minute panels already scale correctly in
their queries; only the noun was missing.
Normalise the micro sign to U+00B5 throughout. Four axis labels used
the visually identical Greek mu, U+03BC, while every unit field used
the micro sign.
Ledger Close Rate plots ledgers closed beside ledger fetches, which one
unit cannot describe; give the fetches series its own unit on a
right-hand axis. Drop two field overrides on NodeStore Read Latency
that restated the panel unit as a custom suffix -- a suffix is appended
verbatim, so it would have suppressed magnitude scaling and left large
values unreadable, the same defect fixed above.
Queries are unchanged apart from the transaction path piechart, which
moves from rate() to increase(): a per-slice "per second" reading is
not a share of a total.
Alongside, widen the Complete Ledger Ranges table to full width, hoist
the stat panels above the fold, and bring the touched panels up to the
tooltip and null-spanning guidelines.
The four job-latency panels on node-health declared milliseconds while
querying `job_running_us` / `job_queued_us`, which record microseconds
(MetricsRegistry records the raw value, and the instrument description
says microseconds). Every reading was therefore a thousand times too
large: the p95 for acceptLedger, 241495us, rendered as "241 sec"
instead of 241ms. job-queue.json already read these same metrics as
microseconds, so the two dashboards disagreed by 1000x on identical
data. Switch node-health to microseconds to match.
Also replace the generic `ops` and `cps` units, which Grafana renders
as the literal "ops/s" and "counts/s", with custom-suffix units naming
what each panel counts -- messages, fetches, calls, mismatches.
Two panels plot more than one quantity on a single axis, which no
single unit can describe. Give each series its own unit through field
overrides: reads per second beside two queue depths on NuDB Read
Pressure, and ledgers beside fetches on Ledger Close Rate, the latter
on a right-hand axis.
State Duration Rate plots a seconds-per-second time share, which can
exceed 1.0 and so is not a percentage; label it as the ratio it is.
The normalised share already exists as its own panel.
Queries are unchanged; the values were already correct.
Alongside, bring the touched panels up to the dashboard guidelines and
hoist the stat panels above the fold.
Grafana renders `unit: "ops"` as the literal string "ops/s", so every
rate panel read as "operations per second" regardless of what it
actually counted. `Ledger Build Rate` showed "0.258 ops/s" where the
value is one ledger every 3.9s -- the number was right, the unit was
meaningless.
Replace the generic units with Grafana custom-suffix units naming the
quantity, following the existing `suffix:/hr` and `si:drops` precedent
in this repo. Nine of these are `stat` panels with no axis, so the unit
string was the only text a reader ever saw.
Also switch the two trusted/untrusted piecharts and the transaction
path piechart from rate() to increase(): a per-slice "per second"
reading is not a share of a total, counts in the window are.
Queries are unchanged apart from those three; the values were already
correct.
Alongside, bring the touched panels up to the dashboard guidelines:
tooltip mode/sort/max-height, 30-minute null spanning, and axis labels
in title case. Hoist the stat panels above the fold on
ledger-operations and rpc-performance.
Panels that a later branch in this chain removes are deliberately left
alone -- fixing them would only add merge conflicts.
Three template variables (xrpl_work_item, xrpl_branch, xrpl_node_role)
select their datasource via ${DS_PROMETHEUS}, but the variable itself was
absent from templating.list, so those label_values() lookups resolved to
no datasource and the filters stayed empty.
Add the datasource variable as the first templating entry, matching the
other dashboards in this folder.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dashboard's template variables and panel targets select their
datasource via ${DS_PROMETHEUS}, but the variable itself was absent from
templating.list. An unresolvable datasource variable leaves those queries
without a datasource, so the panels render empty.
Add the datasource variable as the first templating entry, matching the
other dashboards in this folder.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three measurement fixes landed with no doc or dashboard change, leaving text
that is now false and one fix unusable from a dashboard.
Deferrals and timeouts are recorded in TimeoutCounter, a base shared by five
subclasses, so the all-lane pair could show the documented livelock
fingerprint while ledger acquisition was healthy. The runbook procedure and
the reference doc now name acquire_ledger_deferrals and
acquire_ledger_timeouts and say why the all-lane pair misleads; a new panel
plots the ledger-scoped pair as rates on one axis, since the divergence is
the signal. The existing panel is retitled All Lanes and points at it.
Writer mean depth is depthSum over depthSamples, not over insertCount, and
the measured 1.60 came from the biased estimator, so it and the 37% queueing
share derived from it are lower bounds rather than values. The reference
table now marks them as such, and the decision rule is shown to survive the
correction rather than depending on the exact figures.
Completions were never counted for acquisitions satisfied from the local
store, so the run that read zero across 510 seconds had in fact reached
full. Every place that treated a zero as a symptom now says it only means
something on a build that has the fix.
Also corrects the sync-diagnosis label-value count from 13 to 15 and a stale
source line range; the instrument count stays 35, because both new values
multiplex onto the existing nodestore_state gauge.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Drops nodestore_read_us and everything added to reach it. read_mean_us already
carries microsecond precision and separated the two sync failure modes cleanly
in live testing -- 8.8 us on a clean store against a 223 us cold-store peak --
so the distribution added no signal that changed a diagnosis.
The cost of getting it was disproportionate. NodeStoreScheduler had no path to
the metrics registry, so its production constructor grew a ServiceRegistry
parameter: a metric addition changing a production signature. That in turn
forced an edit to a pre-existing test, src/test/app/SHAMapStore_test.cpp, whose
only stake in this is that it constructs a scheduler. Worse, the scheduler is
built in Application's member initializer list, long before metricsRegistry_
exists, so the registry could not be captured once and had to be re-resolved on
every fetch -- a lookup on a path that runs millions of times per sync.
The constructor returns to taking JobQueue& alone and SHAMapStore_test.cpp
returns to the single-argument call, leaving that file differing from its
pre-change form only by the NodeStore:: to node_store:: rename it picked up from
develop.
FetchReport::elapsed stays microseconds and onFetch keeps its explicit
duration_cast to milliseconds for addLoadEvents, which takes milliseconds. That
widening was a separate fix and is what makes read latency measurable at all.
kSubMillisecondBoundaries loses its only consumer and regains [[maybe_unused]],
which is the state the commit that introduced it left it in; without the
attribute an unused constant is an error under wextra with werr.
Also removes the ledger-data-sync panel that charted the histogram and the
fetch_type and found template variables, which filtered on labels no metric
emits any more, plus the runbook and reference-doc sections and the two
instrument and view counts that named it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Sync State panel only ever showed Connected and Full. Two causes:
1. state_tracking{metric="state_value"} is emitted with values 0-6, where
5 is FULL+validating and 6 is FULL+proposing, but the panel declared
max: 4 with value mappings for 0-4 only. Values above 4 were pinned to
the axis ceiling and rendered unmapped. Most nodes sit at 6, so the
majority of series were clipped.
2. The gauge samples the instantaneous mode on a 10s export tick, so
states shorter than one tick fall between samples. A real sync showed
SYNCING for a single scrape and skipped TRACKING entirely. This is the
sampling hazard already noted on StateAccounting in NetworkOPs.h.
Extend the panel domain to 0-6 with Validating and Proposing mappings and
matching threshold steps, and add a colour-coded state-timeline panel where
each band's width is the time spent in that state, so brief states appear
as thin slivers rather than disappearing. The timeline reads
server_info{metric="server_state"} (raw OperatingMode 0-4) rather than
state_value, which folds 5 and 6 onto FULL and would split one Full band
into three colours.
Panel layout below the insertion point shifts down by 6 rows.
Note this makes short states legible, not lossless: exporting
StateAccounting's per-state duration accumulators is the sampling-immune
fix and is left as follow-up.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The nudb_bytes label value on the storage_detail gauge named something the
code never measured. It observes Database::getStoreSize(), which returns the
storeSz_ accumulator: the cumulative payload bytes of objects this process
handed to the NodeStore. That is not a NuDB file size. It excludes NuDB's
keys, bucket padding and log, and it resets with the process while the files
on disk do not.
The name caused two concrete errors. It invited sizing the store on disk from
a number that cannot do it, and it invited a write-amplification ratio against
node_written_bytes -- which reads the same accessor at MetricsRegistry.cpp:836,
so that ratio is a constant 1.0 and measures nothing.
The nudb_ prefix was wrong too. storeSz_ is written only by
Database::storeStats(), called from DatabaseNodeImp, DatabaseRotatingImp and
Database itself. No backend code touches it, so the value reads the same on
RocksDB. That distinguishes it from the real nudb_* family
(nudb_writers_in_flight and friends), which come from getWriteStats() and are
absent entirely on a non-NuDB backend.
stored_object_bytes says what the value is and claims nothing about the
filesystem. Docs already described the value correctly; they keep that
explanation and now also record the old name, so a query pinned to it can be
traced. Neither Backend nor Database exposes an on-disk size accessor and none
was added -- no metric reports the store's on-disk size today.
Updates the node-health panel title, description and PromQL, and the four docs
that name the label value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
node_reads_hit is a found count, not a cache hit. fetchHitCount_ increments
whenever a fetch returned an object, whatever served it, and a node with
online_delete uses DatabaseRotatingImp which has no NodeObject cache at all. The
ratio therefore reads near 100% while every fetch goes to disk, which made the
cold-read failure mode look impossible on the board.
- Phase9_taskList: node_reads_hit is a found count, not cache-served reads.
- node-health: "NodeStore Cache Hit Ratio" -> "NodeStore Read Found Ratio",
legends "Cache Hit Ratio" -> "Found Ratio" and "Reads Hit (cache)" ->
"Reads Found"; corrected the ratio panel's axis label, which read
"Operations / Sec" on a percentunit panel.
- node-health: "NuDB Storage Size" -> "NuDB Stored Bytes". nudb_bytes observes
getStoreSize(), the accumulator node_written_bytes also reads, so it is
cumulative object-payload bytes and not a filesystem measurement.
- ledger-data-sync: point the read-latency discriminator at the renamed panel.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
nudb_bytes was documented as a NuDB file size, one place even claiming a
filesystem stat. It observes Database::getStoreSize(), which sums the object
payloads this process has written. It excludes NuDB's keys, bucket padding and
log, and resets with the process. node_written_bytes calls the same accessor, so
the two series are equal by construction and a write-amplification ratio built
from them is a constant 1.0. Neither Backend nor Database exposes a file-size
accessor, so nothing reports on-disk size today.
The Ledger Data & Sync panel plotting node_reads_hit / node_reads_total was
titled "NuDB Cache Hit Ratio" and described as reads served from cache.
fetchHitCount_ increments whenever a fetch returned an object, whatever served
it, so the ratio is a found rate. It reads near 100% while every fetch goes to
disk, which made the cold-read failure mode look impossible. Renamed to
"NuDB Read Found Ratio" and rewrote the guidance to pair it with read latency.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A stalled ledger sync has two distinct causes that look identical from the
job queue: in both, the ledgerData lane sits at its concurrency limit of 3
with work waiting. Lane occupancy therefore discriminates nothing, and the
dashboard had no panel that separated the two.
Adds a Sync Bottleneck Discrimination row to Ledger Data & Sync with seven
panels built on the nodestore_state write-path and acquire counters:
- NodeStore Read Latency, lifetime and windowed, on a log axis with
threshold lines at 10 and 100 us. This is the discriminator: fast reads
mean the cost is on the write side, slow reads with a high hit ratio
mean every access is paying disk latency.
- NuDB Writer Queue Depth, the confirming half. NuDB takes one global
mutex per insert, so depth above 1 is queueing at that mutex.
- NuDB Insert Time, mean against true max, which quantifies how much of
each insert is wait rather than work.
- Acquire Deferrals vs Timeouts on one panel, because the livelock
fingerprint is deferrals rising while timeouts stay flat, and neither
counter shows that alone.
- Acquisition Progress and Discarded Acquire Work, the outcome side: no
completions while the lane is busy means acquisitions resolve neither
way.
- NodeStore Read Latency Distribution split by fetch_type and found, from
the nodestore_read_us histogram, since a mean cannot tell a uniformly
warm store from a mostly-warm one with a cold tail.
Every expression carries the existing service_instance_id and tier filters.
The histogram introduces two new label dimensions, so fetch_type and found
template variables are added to match.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Slowness on the peer object-fetch path could be observed but not
attributed. Job duration metrics carry only `job_type`, and both
`RcvGetLedger` and `RcvGetObjByHash` report as `ledgerRequest`, so a
queue-wait spike could not be traced to a handler. Nothing measured
NodeStore cost, request size, or the differential charge.
Latency now decomposes into three additive parts, each separately
measurable:
end-to-end = queue wait + NodeStore lookup + everything else
- `handler` label on job_queued_total/_started_total/_finished_total and
job_queued_us/job_running_us. The value is sanitised: a name passes
through only if non-empty and all ASCII letters, else "other". Two job
names embed a ledger sequence, so a raw label would mint one series
per ledger; the rule bounds the domain at 43 names plus "other".
- getobject_lookup_us, _request_objects, _lookups_total{result},
_rejected_total{reason} and _charge, recorded at their call sites.
All three histograms get explicit bucket views: the SDK default stops
at 10,000, which every one of them exceeds.
- Per-job-type waiting/running/deferred gauges for the 35 non-special
job types. `deferred` is the leading indicator, since addJob never
rejects -- it defers, so backpressure otherwise shows up only as
latency after the fact.
`JobQueue::collect()` snapshots the counters under the queue lock and
publishes gauges after releasing it. Writing them while holding the lock
would invert a lock order against the collector's own lock, which the
collector's flush thread already holds when it calls this hook.
Tests assert exact values, including that the charge is priced on the
requested count rather than the capped iteration count.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
