ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-22 14:50:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
7c70e142e9249c8b9727807d4964afd8fa8ffa0a
rippled/docker/telemetry/grafana/provisioning/alerting/rules.yaml
Pratik Mankawde 6e2b2da772 fix(telemetry): resolve microsecond latencies below 100us 
The microsecond ladder's first edge was 100us, which sat ABOVE the mass of
every instrument using it. Measured on devnet: 99.3% of job_queued_us
samples, 92.5% of job_running_us and 90.4% of getobject_lookup_us fell in
that first bucket. histogram_quantile then interpolated inside bucket 0 and
returned `quantile / fraction_in_bucket_0 x first_edge` -- p75/p95/p99 of
job_queued_us read 75.52/95.66/99.69us against a prediction of
75.53/95.67/99.70. Three-decimal agreement: those panels were reporting
arithmetic on the bucket edge, not latency.

The fix was already half-written. kSubMillisecondBoundaries had been parked
in MetricsRegistry.cpp as [[maybe_unused]] with a comment noting exactly this
problem for nodestore reads. Its edges are now folded into kMicrosecondBuckets
rather than deleted, so the parked intent is carried forward: 1..1000us
resolution where the mass is, upper edges unchanged so multi-second stalls
stay measurable.

Also moves the GetObject count and charge ladders into HistogramBuckets.h, so
all five ladders have one owner and one set of invariant tests (29 now).

Adds check_bucket_parity.py, wired into the existing OTel naming workflow.
The C++ millisecond ladder and the collector's spanmetrics ladder are
specified to agree over their shared range; they were identical when shipped,
then the collector side alone was extended and nothing noticed for eleven
phases. The check asserts containment rather than equality, because jobs
outlive spans -- jobq_updatepaths averages ~60s, which no span approaches, so
demanding equality would force a ceiling that censors it. Verified it rejects
a missing collector edge, a bogus in-range edge, and a return to the 5s
ceiling.

ledger-data-sync's "Job Queue Wait p95 By Type" moves off the beast
jobq_*_q_milliseconds pair onto job_queued_us filtered by job_type. Those
beast metrics are ms-quantised at the source (Event rounds up to a whole
millisecond), so 94-100% of their samples sat in the first bucket and no
ladder change could fix them. Note the label values are camelCase
(job_type="ledgerData"), not the lowercase metric-name fragments.

Both histogram-fed alert thresholds re-validated and left unchanged, with the
measured basis recorded so neither gets tuned against the old artefact: only
0.0022% of job_queued_us samples exceed the 1s threshold, and every edge
bracketing the 1000ms ios_latency threshold survived the ladder change.

Docs: the rpc_size "known issue -- tracked separately" notes in the runbook
and 09-data-collection-reference are now resolved notes, the stale 10-edge
span_duration bucket list is corrected to the collector's real 20, and the
runbook gains a "Reading A Histogram Percentile" section covering both
saturation traps and the expected discontinuity after a ladder change.
2026-08-21 12:46:56 +01:00

1002 lines
36 KiB
YAML
Raw Blame History

# Grafana alert-rule provisioning for rippled OTel metrics.
#
# Alerting on health-critical internal metrics.
#
# Thirteen rules across five subsystems: consensus/ledger health, validator
# health, the job queue, node operating state, and the overlay (manifests).
#
# Rule shape (Grafana server-side evaluation):
# A Prometheus query — a rate / increase / histogram_quantile, aggregated
# `by (service_instance_id)` so each node evaluates
# independently. Alert rules run headless and cannot
# use the dashboards' `$node` template variable.
# B reduce (last) — collapse A's series to its most recent value.
# C threshold — the firing condition; `condition: C`.
#
# Conventions that are load-bearing — do not "simplify" these away:
#
# * Every expr selects {service_name="xrpld"}. The same Prometheus also
# hosts a legacy statsd fleet under job="integrations/unix" which exports
# some of these names (state_accounting_* in particular) with no xrpld
# resource attributes. Without the selector those series get summed in.
#
# * `isPaused: true` on every rule. The key is camelCase; `is_paused` is
# SILENTLY IGNORED by the provisioning loader (no error, no warning) and
# leaves the rule live. Note the inconsistency: the sibling field
# `notification_settings` IS snake_case.
#
# * Sparse counters use increase(...[15m]) with a short `for`, not
# rate(...[5m]) with for: 5m. A single increment keeps rate[5m] nonzero
# for only ~4 minutes of dwell, so `for: 5m` can never be satisfied and
# the rule silently never fires for one-off events.
#
# * Rules whose intent is "this node stopped doing X" synthesise an explicit
# zero via `or (0 * max_over_time(...))`, because `sum by()` returns rows
# only for still-reporting nodes: a single dead node's row just disappears
# and noDataState never triggers.
#
# Thresholds are documented in docs/telemetry-runbook.md (Alerting section)
# and are derived from measured values across a 7-node dev/devnet population.
# Production nodes (higher peer counts, real traffic) need a re-tune.
apiVersion: 1
groups:
# ------------------------------------------------------------------ #
# Consensus / ledger health #
# ------------------------------------------------------------------ #
- orgId: 1
name: xrpld-consensus
folder: xrpld
interval: 1m
rules:
# A closed ledger that later fails validation against the network —
# any mismatch means this node built history the rest of the network
# rejects. A healthy node never mismatches, so a single event matters:
# hence increase() over a wide window rather than a decaying rate().
- uid: xrpld-ledger-history-mismatch
title: LedgerHistoryMismatch
condition: C
for: 2m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
severity: critical
category: consensus
annotations:
summary: "Ledger history mismatch on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} recorded
{{ $values.B.Value }} ledger history mismatch(es) in the last 15m.
The node's built ledger diverges from the validated network chain.
data:
- refId: A
relativeTimeRange:
from: 1200
to: 0
datasourceUid: prometheus
model:
refId: A
expr: sum by (service_instance_id) (increase(ledger_history_mismatch_total{service_name="xrpld"}[15m]))
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 1200
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 1200
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [0]
datasource:
type: __expr__
uid: __expr__
# Healthy nodes close a ledger every ~3-5s. Zero closes for 3 minutes
# means consensus/ledger advancement is stuck. The `or 0 *
# max_over_time` term synthesises a zero row for a node that was
# reporting within the last hour but has now gone silent, so a single
# dead node trips the threshold instead of vanishing from the result.
- uid: xrpld-ledger-close-stalled
title: LedgerCloseStalled
condition: C
for: 3m
isPaused: true
noDataState: Alerting
execErrState: Error
labels:
severity: critical
category: consensus
annotations:
summary: "Ledger closing stalled on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} has closed no ledgers for
several minutes (5m rate has decayed to zero). Consensus or ledger
advancement is stuck, or the process is gone.
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: prometheus
model:
refId: A
expr: |-
sum by (service_instance_id) (rate(ledgers_closed_total{service_name="xrpld"}[5m]))
or (0 * max by (service_instance_id) (max_over_time(ledgers_closed_total{service_name="xrpld"}[1h])))
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: lt
params: [0.001]
datasource:
type: __expr__
uid: __expr__
# The validated ledger falling behind wall-clock is the single clearest
# "this node is unhealthy" signal on XRPL: it is the symptom every other
# consensus/sync failure eventually produces. Measured p50 2s / p95 4s /
# p99 5s across all nodes over 7d, so 60s carries ~12x headroom over p99.
#
# The `< 1209600` clause is REQUIRED, not defensive. When there is no
# validated ledger at all, LedgerMaster::getValidatedLedgerAge() returns
# weeks{2} == 1209600s as a SENTINEL rather than a measurement
# (LedgerMaster.cpp, "No validated ledger" branch). Without the clause the
# rule reads that sentinel as "14 days stale" and fires on every node
# during startup — measured: it produced sustained firing on all 9 nodes
# including healthy ones over a 6-day window. Excluding the exact sentinel
# keeps the rule measuring real staleness; a node genuinely holding no
# validated ledger is caught by LedgerCloseStalled and NodeNotFull.
- uid: xrpld-validated-ledger-stale
title: ValidatedLedgerStale
condition: C
for: 5m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
severity: critical
category: consensus
annotations:
summary: "Validated ledger stale on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} has a validated ledger age of
{{ $values.B.Value }}s (>60s). The node is not keeping up with the
validated network chain.
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: prometheus
model:
refId: A
expr: max by (service_instance_id) (ledgermaster_validated_ledger_age{service_name="xrpld"} < 1209600)
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [60]
datasource:
type: __expr__
uid: __expr__
# ------------------------------------------------------------------ #
# Validator health #
# ------------------------------------------------------------------ #
- orgId: 1
name: xrpld-validator
folder: xrpld
interval: 1m
rules:
# This validator's own validations are not agreeing with the network.
#
# IMPORTANT — why this is a ratio gated on validations_sent_total, and
# not `rate(validation_missed_total) > 0`:
# ValidationTracker classifies a ledger as a miss whenever
# (weValidated && networkValidated) is not both true. A node that does
# not validate never sets weValidated, so EVERY reconciled ledger counts
# as a miss and the raw rate is permanently nonzero — measured ratio is
# exactly 1.0 on non-validating nodes. No threshold can separate "not a
# validator" from "validator disagreeing"; the `and on(...)` gate
# excludes non-validators entirely, and the ratio then measures real
# disagreement among nodes that do validate.
- uid: xrpld-validations-missed
title: ValidationsMissed
condition: C
for: 15m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
severity: warning
category: validator
annotations:
summary: "Validations missed on {{ $labels.service_instance_id }}"
description: >-
Validator {{ $labels.service_instance_id }} is missing
{{ $values.B.Value }} (fraction) of its validations over 15m. Its
validations are not agreeing with the validated ledger, which risks
removal from UNLs.
data:
- refId: A
relativeTimeRange:
from: 1200
to: 0
datasourceUid: prometheus
model:
refId: A
expr: |-
(
sum by (service_instance_id) (rate(validation_missed_total{service_name="xrpld"}[15m]))
/ clamp_min(
sum by (service_instance_id) (rate(validation_missed_total{service_name="xrpld"}[15m]))
+ sum by (service_instance_id) (rate(validation_agreements_total{service_name="xrpld"}[15m])),
1e-9)
)
and on (service_instance_id)
(sum by (service_instance_id) (rate(validations_sent_total{service_name="xrpld"}[15m])) > 0)
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 1200
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 1200
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [0.1]
datasource:
type: __expr__
uid: __expr__
# The node has stopped checking incoming validations. Zero checked
# validations means it is no longer processing the validation stream
# from peers. Synthesises a zero for a silent-but-recently-seen node
# (see the LedgerCloseStalled comment).
- uid: xrpld-validations-not-checked
title: ValidationsNotChecked
condition: C
for: 5m
isPaused: true
noDataState: Alerting
execErrState: Error
labels:
severity: warning
category: validator
annotations:
summary: "No validations checked on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} has checked no incoming
validations for several minutes (5m rate has decayed to zero). The
validation stream from peers may have stopped.
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: prometheus
model:
refId: A
expr: |-
sum by (service_instance_id) (rate(validations_checked_total{service_name="xrpld"}[5m]))
or (0 * max by (service_instance_id) (max_over_time(validations_checked_total{service_name="xrpld"}[1h])))
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: lt
params: [0.001]
datasource:
type: __expr__
uid: __expr__
# ------------------------------------------------------------------ #
# Job queue / resource health #
# ------------------------------------------------------------------ #
- orgId: 1
name: xrpld-jobqueue
folder: xrpld
interval: 1m
rules:
# Transactions are being dropped because the job queue is full — the
# node is shedding load it cannot process. Overflow arrives in bursts,
# so this uses increase() over a wide window (see the header note on
# sparse counters).
- uid: xrpld-jobqueue-tx-overflow
title: JobQueueTxOverflow
condition: C
for: 2m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
severity: warning
category: jobqueue
annotations:
summary: "Job queue transaction overflow on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} overflowed its transaction
job queue {{ $values.B.Value }} time(s) in the last 15m.
Transactions are being dropped under load.
data:
- refId: A
relativeTimeRange:
from: 1200
to: 0
datasourceUid: prometheus
model:
refId: A
expr: sum by (service_instance_id) (increase(jq_trans_overflow_total{service_name="xrpld"}[15m]))
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 1200
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 1200
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [0]
datasource:
type: __expr__
uid: __expr__
# p99 time a job waits in the queue before running. A sustained p99
# above 1s means the node is saturated and work is backing up. `le` must
# stay inside the inner sum or histogram_quantile cannot interpolate.
#
# Threshold re-validated after the microsecond ladder was re-cut. Do NOT
# tune it down against a casual reading of this p99: before that change
# the ladder's first edge was 100us with 99.3% of samples beneath it, so
# p99 reported 99.7us -- the bucket edge scaled by the quantile, not a
# latency. Measured cumulative distribution: 99.26% of samples land
# within 100us, 99.969% within 5ms, 99.990% within 100ms, and only
# 0.0022% exceed 1s. So 1s sits about four orders of magnitude above the
# healthy p99 and fires only on genuine saturation, which is the intent.
- uid: xrpld-jobqueue-latency-high
title: JobQueueLatencyHigh
condition: C
for: 5m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
severity: warning
category: jobqueue
annotations:
summary: "Job queue latency high on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} has a p99 job-queue wait of
{{ $values.B.Value }}µs (>1s) over 5m. The node is saturated and jobs are
backing up.
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: prometheus
model:
refId: A
expr: histogram_quantile(0.99, sum by (le, service_instance_id) (rate(job_queued_us_bucket{service_name="xrpld"}[5m])))
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [1000000]
datasource:
type: __expr__
uid: __expr__
# Node-store read/write latency. Sustained high IO latency is the usual
# upstream cause of state flapping and sync stalls, so this often fires
# first and explains the others. Measured p99-of-p95 is 37-49ms on
# healthy nodes and 488-566ms on nodes that are actively flapping, so
# 1000ms flags genuine degradation rather than the current baseline.
#
# Still valid after the millisecond ladder was extended: that change only
# ADDED edges above 5s (2s/3s/4s/10s/30s/60s/120s) and removed none, so
# every edge bracketing this threshold -- 25/50/100/250/500/1000ms -- is
# unchanged and the measurements above still hold. ios_latency's own mean
# is 12.9ms, far below the threshold.
- uid: xrpld-nodestore-io-latency-high
title: NodeStoreIOLatencyHigh
condition: C
for: 10m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
severity: warning
category: jobqueue
annotations:
summary: "Node store IO latency high on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} has a p95 node-store IO
latency of {{ $values.B.Value }}ms (>1s) over 10m. Check disk
utilisation and whether the store is on a slow volume.
data:
- refId: A
relativeTimeRange:
from: 900
to: 0
datasourceUid: prometheus
model:
refId: A
expr: histogram_quantile(0.95, sum by (le, service_instance_id) (rate(ios_latency_milliseconds_bucket{service_name="xrpld"}[10m])))
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 900
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 900
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [1000]
datasource:
type: __expr__
uid: __expr__
# ------------------------------------------------------------------ #
# Node operating state #
# ------------------------------------------------------------------ #
- orgId: 1
name: xrpld-node-state
folder: xrpld
interval: 1m
rules:
# Node state flapping: full -> syncing/tracking -> full, repeatedly.
#
# state_accounting_full_transitions counts transitions INTO full
# (NetworkOPs.cpp StateAccounting::mode) and is exported as a cumulative
# gauge, so increase() is correct — and its counter-reset correction
# turns a process restart into a small positive delta rather than a
# false spike.
#
# state_changes_total cannot be used here: it carries no from/to labels,
# so it cannot distinguish a flap from a normal startup walk.
#
# The uptime gate is load-bearing. Every node walks
# disconnected -> connected -> syncing -> tracking -> full once at boot;
# without the gate every restart pages. Measured: flapping nodes re-enter
# full 4-6 times per hour sustained, healthy nodes 0-1, so >3 separates
# the populations with a 3x margin.
- uid: xrpld-node-state-flapping
title: NodeStateFlapping
condition: C
for: 15m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
severity: warning
category: node_state
annotations:
summary: "Node state flapping on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} re-entered the FULL state
{{ $values.B.Value }} times in the last hour (>3). It is oscillating
between full and syncing/connected rather than holding sync. Check
node-store IO latency, peer connectivity, and clock sync.
data:
- refId: A
relativeTimeRange:
from: 3900
to: 0
datasourceUid: prometheus
model:
refId: A
expr: |-
sum by (service_instance_id) (increase(state_accounting_full_transitions{service_name="xrpld"}[1h]))
and on (service_instance_id)
(sum by (service_instance_id) (server_info{service_name="xrpld", metric="uptime"}) > 3600)
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 3900
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 3900
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [3]
datasource:
type: __expr__
uid: __expr__
# A node stuck OUT of full. Distinct from flapping: a node that drops to
# syncing and stays there produces no further full-transitions, so the
# flapping rule by definition cannot catch it.
# server_state enum (NetworkOPs.h): DISCONNECTED=0, CONNECTED=1,
# SYNCING=2, TRACKING=3, FULL=4.
- uid: xrpld-node-not-full
title: NodeNotFull
condition: C
for: 15m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
severity: warning
category: node_state
annotations:
summary: "Node not in FULL state on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} has been below FULL
(state={{ $values.B.Value }}; 0=disconnected 1=connected 2=syncing
3=tracking 4=full) for 15m. It is not fully synced with the network.
data:
- refId: A
relativeTimeRange:
from: 1200
to: 0
datasourceUid: prometheus
model:
refId: A
expr: |-
max by (service_instance_id) (server_info{service_name="xrpld", metric="server_state"})
and on (service_instance_id)
(sum by (service_instance_id) (server_info{service_name="xrpld", metric="uptime"}) > 3600)
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 1200
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 1200
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: lt
params: [4]
datasource:
type: __expr__
uid: __expr__
# ------------------------------------------------------------------ #
# Overlay / manifests #
# ------------------------------------------------------------------ #
- orgId: 1
name: xrpld-overlay
folder: xrpld
interval: 1m
rules:
# Manifest job convoy — the primary manifest-flooding signal.
#
# Peers send TMManifests dumps up to ~57MB (just under
# kMaximumMessageSize, overlay/Message.h). JtManifest is registered with
# maxLimit (core/JobTypes.h), so every peer's dump runs concurrently and
# they convoy on ManifestCache::mutex_; OverlayImpl::onManifests also
# re-verifies the blob a second time on Accept. Measured effect: each
# RcvManifests job took 16-18s and the whole 8-worker pool was occupied.
#
# jobq_manifest_waiting is 0 at the 99.9th percentile on every node over
# 24h, so any sustained backlog is a genuine outlier rather than normal
# variance. Threshold >3 with a 10m dwell keeps the measured startup
# burst (peaks of 5 and 11, lasting well under 10m) from paging.
- uid: xrpld-manifest-job-convoy
title: ManifestJobQueueConvoy
condition: C
for: 10m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
severity: warning
category: overlay
annotations:
summary: "Manifest job convoy on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} has {{ $values.B.Value }}
manifest jobs waiting (>3) for 10m. Peer manifest dumps are
saturating the job pool and convoying on the manifest cache lock.
data:
- refId: A
relativeTimeRange:
from: 900
to: 0
datasourceUid: prometheus
model:
refId: A
expr: sum by (service_instance_id) (jobq_manifest_waiting{service_name="xrpld"})
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 900
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 900
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [3]
datasource:
type: __expr__
uid: __expr__
# Inbound manifest byte-rate flood. Complements the convoy rule: this
# catches the wire-level cause (a peer shipping huge dumps) even when the
# job pool absorbs it without a visible backlog.
#
# Threshold derived from a 7-day sample (uptime-gated), NOT from the
# 24h window that an earlier revision used:
# healthy p95 0.2-0.5 kB/s, p99 1.0-1.8 kB/s
# observed peaks up to 2.7 MB/s during real manifest storms
# 512 KiB/s (524288 B/s) sits ~280x above healthy p99 and ~5x below the
# peaks. An earlier 50 kB/s threshold produced ~41 sustained 5-min samples
# across six healthy nodes over six days (i.e. routine paging);
# 512 KiB/s reduces that to 2 while still catching every genuine storm.
#
# The uptime gate exists because the startup manifest burst is MEASURED
# NORMAL behaviour. It does not hide real floods — the same 7-day sample
# shows firing rates with and without the gate within a factor of two — but a
# flood confined to the first 30 minutes after boot is deliberately not
# alerted. ManifestJobQueueConvoy covers that window via the job pool.
- uid: xrpld-manifest-flood-inbound
title: ManifestFloodInbound
condition: C
for: 10m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
severity: warning
category: overlay
annotations:
summary: "Inbound manifest flood on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} is receiving
{{ $values.B.Value }} B/s of manifest traffic over 10m, above the
512 KiB/s (524288 B/s) threshold.
A peer is flooding oversized TMManifests dumps.
data:
- refId: A
relativeTimeRange:
from: 900
to: 0
datasourceUid: prometheus
model:
refId: A
expr: |-
sum by (service_instance_id) (rate(overhead_manifest_bytes_in{service_name="xrpld"}[10m]))
and on (service_instance_id)
(sum by (service_instance_id) (server_info{service_name="xrpld", metric="uptime"}) > 1800)
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 900
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 900
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [524288]
datasource:
type: __expr__
uid: __expr__
# Resource-driven peer disconnects. The node is dropping peers for
# exceeding resource budgets, which precedes peer starvation and sync
# loss. Sparse and bursty, so increase() over a wide window.
#
# Measured p95 of the 30m increase: 0 on every healthy node, 4.0 and 11.7
# on the two nodes that are independently known to be degraded (the same
# two that flap). So >5 sits above the healthy baseline entirely and only
# trips on a node already in trouble.
- uid: xrpld-peer-resource-disconnects
title: PeerResourceDisconnects
condition: C
for: 5m
isPaused: true
noDataState: NoData
execErrState: Error
labels:
severity: warning
category: overlay
annotations:
summary: "Resource-driven peer disconnects on {{ $labels.service_instance_id }}"
description: >-
Node {{ $labels.service_instance_id }} disconnected
{{ $values.B.Value }} peer(s) for resource-budget violations in the
last 30m. Sustained disconnects can starve the node of peers.
data:
- refId: A
relativeTimeRange:
from: 2100
to: 0
datasourceUid: prometheus
model:
refId: A
expr: sum by (service_instance_id) (increase(server_info{service_name="xrpld", metric="peer_disconnects_resources"}[30m]))
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 2100
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 2100
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [5]
datasource:
type: __expr__
uid: __expr__
Reference in New Issue View Git Blame Copy Permalink