Merge branch 'pratik/otel-phase10-workload-validation' into pratik/otel-phase11-telemetry-off

This commit is contained in:
Pratik Mankawde
2026-07-06 16:26:01 +01:00
8 changed files with 712 additions and 8 deletions

View File

@@ -0,0 +1,133 @@
# rippled OpenTelemetry Alerting Runbook
Phase 9 exports rippled's internal metrics and provisions Grafana alert rules
on the health-critical ones. This document explains each alert, its likely
cause, and how to point alerts at a real receiver.
The rules are provisioned from
`grafana/provisioning/alerting/` and load automatically when the Grafana
container in [docker-compose.yml](docker-compose.yml) starts — no manual setup
in the Grafana UI. Rules appear in Grafana under **Alerting → Alert rules**,
folder **xrpld**.
---
## Alert catalogue
All rules evaluate every minute against the Prometheus datasource, over a
5-minute window, and group by `exported_instance` so each node alerts on its
own. Alerts fire only after the condition holds for the `for` dwell time.
| Alert | Severity | Fires when | For |
| ----------------------- | -------- | ----------------------------------------------- | --- |
| `LedgerHistoryMismatch` | critical | `rate(xrpld_ledger_history_mismatch_total)` > 0 | 5m |
| `LedgerCloseStalled` | critical | `rate(xrpld_ledgers_closed_total)` ≈ 0 | 3m |
| `ValidationsMissed` | warning | `rate(xrpld_validation_missed_total)` > 0 | 5m |
| `ValidationsNotChecked` | warning | `rate(xrpld_validations_checked_total)` ≈ 0 | 5m |
| `JobQueueTxOverflow` | warning | `rate(xrpld_jq_trans_overflow_total)` > 0 | 5m |
| `JobQueueLatencyHigh` | warning | p99 `xrpld_job_queued_duration_us` > 1s | 5m |
### Consensus / ledger health
**LedgerHistoryMismatch** — The node closed a ledger whose history diverges
from the validated network chain. Likely causes: corrupted local state, a bug,
or a node that fell out of sync and rebuilt incorrectly. Investigate the node's
ledger acquisition logs; a healthy node never mismatches.
**LedgerCloseStalled** — No ledgers closed for 3 minutes. A healthy node closes
one every ~3-5s. Likely causes: lost peer connectivity, consensus stall, or the
process is hung. This rule also fires on _NoData_ — if the series disappears the
node is likely down. Check peer count and process health first.
### Validator health
**ValidationsMissed** — This validator's validations are not agreeing with the
validated ledger. Sustained misses risk removal from UNLs. Check clock sync,
peer connectivity, and whether the node is keeping up with ledger close.
**ValidationsNotChecked** — The node has stopped checking incoming validations
from peers. Likely causes: overlay/peer disconnection or a stalled validation
pipeline. Fires on NoData as well.
### Job queue / resource health
**JobQueueTxOverflow** — The transaction job queue is full and transactions are
being dropped. The node is shedding load it cannot process. Check CPU, the
`JobQueueLatencyHigh` alert, and offered load.
**JobQueueLatencyHigh** — p99 queue wait exceeds 1 second, i.e. jobs back up
before running. The node is saturated. Correlate with CPU and the Job Queue
dashboard.
---
## Tuning thresholds
Thresholds live in
[grafana/provisioning/alerting/rules.yaml](grafana/provisioning/alerting/rules.yaml)
as the `params` array of each rule's `C` (threshold) node. Common tunables:
- **`JobQueueLatencyHigh`** — `params: [1000000]` is 1 000 000 µs (1s). Lower
it for latency-sensitive deployments.
- **`LedgerCloseStalled` / `ValidationsNotChecked`** — use `lt` with a tiny
epsilon (`0.001`) rather than `0`, so floating-point rate noise near zero
does not suppress the alert.
Edit the file and restart the Grafana container to reload:
```bash
docker compose -f docker/telemetry/docker-compose.yml restart grafana
```
---
## Sending alerts somewhere real
The provisioned contact point `xrpld-default`
([contactpoints.yaml](grafana/provisioning/alerting/contactpoints.yaml)) ships
as a **local-dev webhook** to a placeholder URL — alerts fire but go nowhere
until you change it.
**Option A — repoint the webhook.** Replace the `url` under the `webhook`
receiver with a real endpoint (PagerDuty Events API, Opsgenie, a custom sink).
**Option B — add a Slack receiver.** Add a second receiver to the same contact
point:
```yaml
- uid: xrpld-slack
type: slack
settings:
url: https://hooks.slack.com/services/XXX/YYY/ZZZ
title: "{{ .CommonLabels.alertname }} on {{ .CommonLabels.exported_instance }}"
```
**Option C — email.** Requires Grafana SMTP configured via `GF_SMTP_*`
environment variables on the Grafana service in `docker-compose.yml`, then an
`email` receiver with `addresses`.
Routing is a single flat policy in
[policies.yaml](grafana/provisioning/alerting/policies.yaml): all alerts →
`xrpld-default`, grouped by `alertname` + `exported_instance`. To route
critical alerts to a different receiver, add child routes matching
`severity = critical`.
---
## Verifying provisioning loaded
After the stack is up:
```bash
# All six rules present?
curl -s http://localhost:3000/api/v1/provisioning/alert-rules | jq '.[].title'
# Contact point present?
curl -s http://localhost:3000/api/v1/provisioning/contact-points | jq '.[].name'
```
Grafana logs a provisioning error and skips the file if the YAML is malformed:
```bash
docker compose -f docker/telemetry/docker-compose.yml logs grafana | grep -i alerting
```

View File

@@ -0,0 +1,28 @@
# Grafana contact-point provisioning for rippled OTel alerts.
#
# Phase 9: Internal metric gap fill — alerting on health-critical metrics.
#
# A contact point is where a firing alert is delivered. This stack ships a
# single webhook receiver as a working default for local development. To route
# alerts to a real destination, replace the webhook `url` below, or add a
# `slack` / `email` receiver to the same contact point (see ALERTING.md).
#
# The default anonymous-admin Grafana in docker-compose.yml picks this file up
# from /etc/grafana/provisioning/alerting/ on startup.
apiVersion: 1
contactPoints:
- orgId: 1
name: xrpld-default
receivers:
# Local-dev webhook sink. Points at a placeholder host; alerts are
# emitted but silently dropped unless a receiver listens there.
# Swap the url for a real endpoint (PagerDuty/Opsgenie/custom) or
# replace the whole receiver block with a slack/email type.
- uid: xrpld-default-webhook
type: webhook
settings:
url: http://localhost:9999/xrpld-alerts
httpMethod: POST
disableResolveMessage: false

View File

@@ -0,0 +1,28 @@
# Grafana notification-policy provisioning for rippled OTel alerts.
#
# Phase 9: Internal metric gap fill — alerting on health-critical metrics.
#
# The notification policy tree decides which contact point receives a firing
# alert and how alerts are batched. This stack uses a single flat policy: every
# xrpld alert routes to the `xrpld-default` contact point.
#
# Grouping by alertname + exported_instance means one notification per
# (alert, node) pair, so a fleet-wide problem does not collapse into a single
# ambiguous message.
apiVersion: 1
policies:
- orgId: 1
receiver: xrpld-default
group_by:
- alertname
- exported_instance
# Wait before sending the first notification for a new group, so a burst of
# related series is batched together.
group_wait: 30s
# Wait before adding new alerts to an already-notified group.
group_interval: 5m
# Minimum time before re-sending a notification for an alert that is still
# firing.
repeat_interval: 4h

View File

@@ -0,0 +1,409 @@
# Grafana alert-rule provisioning for rippled OTel metrics.
#
# Phase 9: Internal metric gap fill — alerting on health-critical metrics.
#
# Six rules across three node-fatal subsystems: consensus/ledger health,
# validator health, and the job queue. Every metric referenced here is
# introduced by phase 9's MetricsRegistry.
#
# Rule shape (Grafana server-side evaluation):
# A Prometheus query — a 5-minute rate / histogram_quantile, aggregated
# `by (exported_instance)` so each node evaluates
# independently. Alert rules run headless and cannot
# use the dashboards' `$node` template variable.
# B reduce (last) — collapse A's series to its most recent value.
# C threshold — the firing condition; `condition: C`.
#
# Thresholds are documented in ALERTING.md and are intended to be tuned.
apiVersion: 1
groups:
# ------------------------------------------------------------------ #
# Consensus / ledger health #
# ------------------------------------------------------------------ #
- orgId: 1
name: xrpld-consensus
folder: xrpld
interval: 1m
rules:
# A closed ledger that later fails validation against the network —
# any sustained nonzero rate means this node built history the rest
# of the network rejects.
- uid: xrpld-ledger-history-mismatch
title: LedgerHistoryMismatch
condition: C
for: 5m
noDataState: NoData
execErrState: Error
labels:
severity: critical
category: consensus
annotations:
summary: "Ledger history mismatch on {{ $labels.exported_instance }}"
description: >-
Node {{ $labels.exported_instance }} is recording ledger history
mismatches ({{ $values.B.Value }}/s over 5m). The node's built ledger
diverges from the validated network chain.
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: prometheus
model:
refId: A
expr: sum by (exported_instance) (rate(xrpld_ledger_history_mismatch_total[5m]))
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [0]
datasource:
type: __expr__
uid: __expr__
# Healthy nodes close a ledger every ~3-5s. Zero closes for 3 minutes
# means consensus/ledger advancement is stuck. NoData (metric absent)
# also fires — a vanished series here means the node is down.
- uid: xrpld-ledger-close-stalled
title: LedgerCloseStalled
condition: C
for: 3m
noDataState: Alerting
execErrState: Error
labels:
severity: critical
category: consensus
annotations:
summary: "Ledger closing stalled on {{ $labels.exported_instance }}"
description: >-
Node {{ $labels.exported_instance }} has closed no ledgers for
several minutes (5m rate has decayed to zero). Consensus or ledger
advancement is stuck.
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: prometheus
model:
refId: A
expr: sum by (exported_instance) (rate(xrpld_ledgers_closed_total[5m]))
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: lt
params: [0.001]
datasource:
type: __expr__
uid: __expr__
# ------------------------------------------------------------------ #
# Validator health #
# ------------------------------------------------------------------ #
- orgId: 1
name: xrpld-validator
folder: xrpld
interval: 1m
rules:
# This validator's own validations are not reaching / agreeing with
# the network. A sustained nonzero miss rate risks the validator being
# dropped from UNLs.
- uid: xrpld-validations-missed
title: ValidationsMissed
condition: C
for: 5m
noDataState: NoData
execErrState: Error
labels:
severity: warning
category: validator
annotations:
summary: "Validations missed on {{ $labels.exported_instance }}"
description: >-
Node {{ $labels.exported_instance }} is missing validations
({{ $values.B.Value }}/s over 5m). Its validations are not agreeing with
the validated ledger.
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: prometheus
model:
refId: A
expr: sum by (exported_instance) (rate(xrpld_validation_missed_total[5m]))
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [0]
datasource:
type: __expr__
uid: __expr__
# The node has stopped checking incoming validations. Zero checked
# validations means it is no longer processing the validation stream
# from peers.
- uid: xrpld-validations-not-checked
title: ValidationsNotChecked
condition: C
for: 5m
noDataState: Alerting
execErrState: Error
labels:
severity: warning
category: validator
annotations:
summary: "No validations checked on {{ $labels.exported_instance }}"
description: >-
Node {{ $labels.exported_instance }} has checked no incoming
validations for several minutes (5m rate has decayed to zero). The
validation stream from peers may have stopped.
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: prometheus
model:
refId: A
expr: sum by (exported_instance) (rate(xrpld_validations_checked_total[5m]))
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: lt
params: [0.001]
datasource:
type: __expr__
uid: __expr__
# ------------------------------------------------------------------ #
# Job queue / resource health #
# ------------------------------------------------------------------ #
- orgId: 1
name: xrpld-jobqueue
folder: xrpld
interval: 1m
rules:
# Transactions are being dropped because the job queue is full — the
# node is shedding load it cannot process.
- uid: xrpld-jobqueue-tx-overflow
title: JobQueueTxOverflow
condition: C
for: 5m
noDataState: NoData
execErrState: Error
labels:
severity: warning
category: jobqueue
annotations:
summary: "Job queue transaction overflow on {{ $labels.exported_instance }}"
description: >-
Node {{ $labels.exported_instance }} is overflowing its transaction
job queue ({{ $values.B.Value }}/s over 5m). Transactions are being
dropped under load.
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: prometheus
model:
refId: A
expr: sum by (exported_instance) (rate(xrpld_jq_trans_overflow_total[5m]))
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [0]
datasource:
type: __expr__
uid: __expr__
# p99 time a job waits in the queue before running. A sustained p99
# above 1s means the node is saturated and work is backing up. Tune the
# threshold (in microseconds) to the deployment.
- uid: xrpld-jobqueue-latency-high
title: JobQueueLatencyHigh
condition: C
for: 5m
noDataState: NoData
execErrState: Error
labels:
severity: warning
category: jobqueue
annotations:
summary: "Job queue latency high on {{ $labels.exported_instance }}"
description: >-
Node {{ $labels.exported_instance }} has a p99 job-queue wait of
{{ $values.B.Value }}µs (>1s) over 5m. The node is saturated and jobs are
backing up.
data:
- refId: A
relativeTimeRange:
from: 600
to: 0
datasourceUid: prometheus
model:
refId: A
expr: histogram_quantile(0.99, sum by (le, exported_instance) (rate(xrpld_job_queued_duration_us_bucket[5m])))
instant: true
range: false
intervalMs: 1000
maxDataPoints: 43200
- refId: B
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
refId: B
type: reduce
reducer: last
expression: A
datasource:
type: __expr__
uid: __expr__
- refId: C
relativeTimeRange:
from: 600
to: 0
datasourceUid: __expr__
model:
refId: C
type: threshold
expression: B
conditions:
- evaluator:
type: gt
params: [1000000]
datasource:
type: __expr__
uid: __expr__

View File

@@ -0,0 +1,99 @@
# Phase-9 Alerting — Design
**Date:** 2026-07-06
**Branch:** `pratik/otel-phase9-metric-gap-fill` (PR #6513, Jira RIPD-5187)
**Status:** Approved
## Purpose
Phase 9 exports ~68 internal rippled metrics and ships Grafana dashboards for
them. This adds the missing operator-facing piece: **provisioned Grafana alert
rules** that fire on the health-critical metrics phase 9 introduces. The
phase-9 task list (line 311) and Jira story RIPD-5187 both already list
"alerting rules" as a phase-9 deliverable, so this closes that gap.
Scope is deliberately narrow — the three subsystems whose failure is
node-fatal: **consensus/ledger health, validator health, job queue**. RPC/API
health is explicitly out of scope.
## Why phase 9 (not phase 11)
Every metric these alerts fire on is _born_ in phase 9
(`xrpld_ledger_history_mismatch_total`, `xrpld_ledgers_closed_total`,
`xrpld_validation_missed_total`, `xrpld_validations_checked_total`,
`xrpld_jq_trans_overflow_total`, `xrpld_job_queued_duration_us_bucket`). Alerts
belong with the metrics they watch, and this is where the dependency lives.
## Delivery
Provisioned YAML, version-controlled — matching the existing datasource /
dashboard provisioning pattern. No docker-compose change: the Grafana service
already mounts `./grafana/provisioning:/etc/grafana/provisioning:ro`, and
Grafana auto-loads `provisioning/alerting/*.yaml`.
New files under `docker/telemetry/grafana/provisioning/alerting/`:
| File | Purpose |
| -------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `contactpoints.yaml` | One contact point `xrpld-default` (webhook to a documented placeholder; comments show how to swap for Slack/email). |
| `policies.yaml` | Default notification policy: route all alerts → `xrpld-default`, grouped by `alertname` + `exported_instance`. |
| `rules.yaml` | 6 alert rules across 3 groups (below). |
Plus `docker/telemetry/ALERTING.md` — operator runbook: what each alert means,
likely causes, and how to point the contact point at a real receiver.
## Alert rules
All rules target Prometheus datasource `uid: prometheus`. Each rule uses the
Grafana rule shape: query (A) → reduce (B, last value) → threshold (C). All
`rate()`/`histogram_quantile()` expressions aggregate with
`sum by (exported_instance)` (or `+ le`) so **each node alerts independently**.
Alert rules run headless, so they cannot use the dashboards' `$node` template
variables — they match all series and group by `exported_instance` instead.
| Group | Alert | Expression (5m window) | Fires | `for` | severity |
| --------- | --------------------- | --------------------------------------------------------------------------------------------------------- | --------------------- | ----- | -------- |
| Consensus | LedgerHistoryMismatch | `sum by (exported_instance)(rate(xrpld_ledger_history_mismatch_total[5m]))` | `> 0` | 5m | critical |
| Consensus | LedgerCloseStalled | `sum by (exported_instance)(rate(xrpld_ledgers_closed_total[5m]))` | `< 0.001` (≈0) | 3m | critical |
| Validator | ValidationsMissed | `sum by (exported_instance)(rate(xrpld_validation_missed_total[5m]))` | `> 0` | 5m | warning |
| Validator | ValidationsNotChecked | `sum by (exported_instance)(rate(xrpld_validations_checked_total[5m]))` | `< 0.001` (≈0) | 5m | warning |
| Job queue | JobQueueTxOverflow | `sum by (exported_instance)(rate(xrpld_jq_trans_overflow_total[5m]))` | `> 0` | 5m | warning |
| Job queue | JobQueueLatencyHigh | `histogram_quantile(0.99, sum by (le, exported_instance)(rate(xrpld_job_queued_duration_us_bucket[5m])))` | `> 1000000` (µs = 1s) | 5m | warning |
Each rule carries labels `severity` and `category` (consensus/validator/jobqueue)
and annotations `summary` + `description` (with `{{ $labels.exported_instance }}`
and `{{ $values.B.Value }}` interpolation).
### Threshold rationale
- **LedgerCloseStalled `< 0.001` for 3m**: healthy nodes close a ledger every
~3-5s; a 5m rate decaying to ~0 means the node is stuck. The epsilon (not
exact `0`) avoids float rate-noise suppressing the alert.
- **JobQueueLatencyHigh 1s p99**: a default starting point, easy to tune — jobs
queued >1s at p99 indicate the node is saturated.
- Others are `> 0` on error/miss counters: any sustained nonzero rate is
actionable.
## Non-goals / YAGNI
- No per-alert silencing schedules, no mute timings.
- No RPC/API, overlay, or fee-market alerts (dashboards cover those visually).
- Single contact point — multi-receiver routing is left to the operator.
## Verification
1. `yamllint` (or `python -c yaml.safe_load`) on all three YAML files.
2. `docker compose -f docker/telemetry/docker-compose.yml config -q` still parses.
3. Optional live check: start stack, `GET /api/v1/provisioning/alert-rules`
returns the 6 rules; Grafana logs show no provisioning errors.
4. Code-review pass (subagent) against phase conventions before commit.
## Branch / PR restructure (independent, same session)
- Rename `pratik/otel-phase11-benchmark``pratik/perf-test-otel-on`
(0 unique commits — equals phase-10 tip; the telemetry-IN baseline).
- Rename `pratik/otel-phase11-telemetry-off``pratik/perf-test-otel-off`
(carries the "compile telemetry OUT" build change + benchmark runbook).
- Close PR #7433, delete stale `origin/pratik/otel-phase11-telemetry-off`.
- Push both; open one perf PR `perf-test-otel-off → perf-test-otel-on`
(diff = the ON-vs-OFF delta; `-on` needs no PR of its own).

View File

@@ -371,6 +371,12 @@ LedgerHistory::handleMismatch(
return;
}
// Tracks whether the mismatch reason has already been recorded. The
// consensus tx-set hash disagreement is the root cause, so it is counted
// once here; the tx-level comparison below still logs its diagnostics but
// must not record a second reason for the same mismatch event.
bool reasonRecorded = false;
if (builtConsensusHash && validatedConsensusHash)
{
if (builtConsensusHash != validatedConsensusHash)
@@ -378,11 +384,8 @@ LedgerHistory::handleMismatch(
JLOG(j_.error()) << "MISMATCH on consensus transaction set "
<< " built: " << to_string(*builtConsensusHash)
<< " validated: " << to_string(*validatedConsensusHash);
// The consensus tx-set hashes disagree — this is the root cause,
// so record it as the single reason and stop. The tx-level
// comparison below would otherwise double-count the same mismatch.
recordReason("consensus_txset");
return;
reasonRecorded = true;
}
else
{
@@ -398,13 +401,15 @@ LedgerHistory::handleMismatch(
if (builtTx == validTx)
{
JLOG(j_.error()) << "MISMATCH with same " << builtTx.size() << " transactions";
recordReason("same_txset_diff_result");
if (!reasonRecorded)
recordReason("same_txset_diff_result");
}
else
{
JLOG(j_.error()) << "MISMATCH with " << builtTx.size() << " built and " << validTx.size()
<< " valid transactions.";
recordReason("different_txset");
if (!reasonRecorded)
recordReason("different_txset");
}
JLOG(j_.error()) << "built\n" << getJson({*builtLedger, {}});

View File

@@ -22,11 +22,11 @@
#include <xrpl/nodestore/Database.h>
#include <xrpl/nodestore/NodeObject.h>
#include <xrpl/protocol/HashPrefix.h>
#include <xrpl/protocol/Indexes.h>
#include <xrpl/protocol/Indexes.h> // IWYU pragma: keep
#include <xrpl/protocol/LedgerHeader.h>
#include <xrpl/protocol/Rules.h>
#include <xrpl/protocol/Serializer.h>
#include <xrpl/protocol/SystemParameters.h>
#include <xrpl/protocol/SystemParameters.h> // IWYU pragma: keep
#include <xrpl/protocol/jss.h>
#include <xrpl/resource/Fees.h>
#include <xrpl/shamap/SHAMapNodeID.h>

View File

@@ -79,10 +79,12 @@
#include <xrpl/protocol/ApiVersion.h>
#include <xrpl/protocol/BuildInfo.h>
#include <xrpl/protocol/Feature.h>
#include <xrpl/protocol/Indexes.h> // IWYU pragma: keep
#include <xrpl/protocol/Protocol.h>
#include <xrpl/protocol/PublicKey.h>
#include <xrpl/protocol/STParsedJSON.h>
#include <xrpl/protocol/Serializer.h>
#include <xrpl/protocol/SystemParameters.h> // IWYU pragma: keep
#include <xrpl/protocol/jss.h>
#include <xrpl/protocol/tokens.h>
#include <xrpl/rdb/DatabaseCon.h>