diff --git a/docker/telemetry/ALERTING.md b/docker/telemetry/ALERTING.md new file mode 100644 index 0000000000..2bb7285dc2 --- /dev/null +++ b/docker/telemetry/ALERTING.md @@ -0,0 +1,133 @@ +# rippled OpenTelemetry Alerting Runbook + +Phase 9 exports rippled's internal metrics and provisions Grafana alert rules +on the health-critical ones. This document explains each alert, its likely +cause, and how to point alerts at a real receiver. + +The rules are provisioned from +`grafana/provisioning/alerting/` and load automatically when the Grafana +container in [docker-compose.yml](docker-compose.yml) starts — no manual setup +in the Grafana UI. Rules appear in Grafana under **Alerting → Alert rules**, +folder **xrpld**. + +--- + +## Alert catalogue + +All rules evaluate every minute against the Prometheus datasource, over a +5-minute window, and group by `exported_instance` so each node alerts on its +own. Alerts fire only after the condition holds for the `for` dwell time. + +| Alert | Severity | Fires when | For | +| ----------------------- | -------- | ----------------------------------------------- | --- | +| `LedgerHistoryMismatch` | critical | `rate(xrpld_ledger_history_mismatch_total)` > 0 | 5m | +| `LedgerCloseStalled` | critical | `rate(xrpld_ledgers_closed_total)` ≈ 0 | 3m | +| `ValidationsMissed` | warning | `rate(xrpld_validation_missed_total)` > 0 | 5m | +| `ValidationsNotChecked` | warning | `rate(xrpld_validations_checked_total)` ≈ 0 | 5m | +| `JobQueueTxOverflow` | warning | `rate(xrpld_jq_trans_overflow_total)` > 0 | 5m | +| `JobQueueLatencyHigh` | warning | p99 `xrpld_job_queued_duration_us` > 1s | 5m | + +### Consensus / ledger health + +**LedgerHistoryMismatch** — The node closed a ledger whose history diverges +from the validated network chain. Likely causes: corrupted local state, a bug, +or a node that fell out of sync and rebuilt incorrectly. Investigate the node's +ledger acquisition logs; a healthy node never mismatches. + +**LedgerCloseStalled** — No ledgers closed for 3 minutes. A healthy node closes +one every ~3-5s. Likely causes: lost peer connectivity, consensus stall, or the +process is hung. This rule also fires on _NoData_ — if the series disappears the +node is likely down. Check peer count and process health first. + +### Validator health + +**ValidationsMissed** — This validator's validations are not agreeing with the +validated ledger. Sustained misses risk removal from UNLs. Check clock sync, +peer connectivity, and whether the node is keeping up with ledger close. + +**ValidationsNotChecked** — The node has stopped checking incoming validations +from peers. Likely causes: overlay/peer disconnection or a stalled validation +pipeline. Fires on NoData as well. + +### Job queue / resource health + +**JobQueueTxOverflow** — The transaction job queue is full and transactions are +being dropped. The node is shedding load it cannot process. Check CPU, the +`JobQueueLatencyHigh` alert, and offered load. + +**JobQueueLatencyHigh** — p99 queue wait exceeds 1 second, i.e. jobs back up +before running. The node is saturated. Correlate with CPU and the Job Queue +dashboard. + +--- + +## Tuning thresholds + +Thresholds live in +[grafana/provisioning/alerting/rules.yaml](grafana/provisioning/alerting/rules.yaml) +as the `params` array of each rule's `C` (threshold) node. Common tunables: + +- **`JobQueueLatencyHigh`** — `params: [1000000]` is 1 000 000 µs (1s). Lower + it for latency-sensitive deployments. +- **`LedgerCloseStalled` / `ValidationsNotChecked`** — use `lt` with a tiny + epsilon (`0.001`) rather than `0`, so floating-point rate noise near zero + does not suppress the alert. + +Edit the file and restart the Grafana container to reload: + +```bash +docker compose -f docker/telemetry/docker-compose.yml restart grafana +``` + +--- + +## Sending alerts somewhere real + +The provisioned contact point `xrpld-default` +([contactpoints.yaml](grafana/provisioning/alerting/contactpoints.yaml)) ships +as a **local-dev webhook** to a placeholder URL — alerts fire but go nowhere +until you change it. + +**Option A — repoint the webhook.** Replace the `url` under the `webhook` +receiver with a real endpoint (PagerDuty Events API, Opsgenie, a custom sink). + +**Option B — add a Slack receiver.** Add a second receiver to the same contact +point: + +```yaml +- uid: xrpld-slack + type: slack + settings: + url: https://hooks.slack.com/services/XXX/YYY/ZZZ + title: "{{ .CommonLabels.alertname }} on {{ .CommonLabels.exported_instance }}" +``` + +**Option C — email.** Requires Grafana SMTP configured via `GF_SMTP_*` +environment variables on the Grafana service in `docker-compose.yml`, then an +`email` receiver with `addresses`. + +Routing is a single flat policy in +[policies.yaml](grafana/provisioning/alerting/policies.yaml): all alerts → +`xrpld-default`, grouped by `alertname` + `exported_instance`. To route +critical alerts to a different receiver, add child routes matching +`severity = critical`. + +--- + +## Verifying provisioning loaded + +After the stack is up: + +```bash +# All six rules present? +curl -s http://localhost:3000/api/v1/provisioning/alert-rules | jq '.[].title' + +# Contact point present? +curl -s http://localhost:3000/api/v1/provisioning/contact-points | jq '.[].name' +``` + +Grafana logs a provisioning error and skips the file if the YAML is malformed: + +```bash +docker compose -f docker/telemetry/docker-compose.yml logs grafana | grep -i alerting +``` diff --git a/docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml b/docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml new file mode 100644 index 0000000000..2cd7f47fd1 --- /dev/null +++ b/docker/telemetry/grafana/provisioning/alerting/contactpoints.yaml @@ -0,0 +1,28 @@ +# Grafana contact-point provisioning for rippled OTel alerts. +# +# Phase 9: Internal metric gap fill — alerting on health-critical metrics. +# +# A contact point is where a firing alert is delivered. This stack ships a +# single webhook receiver as a working default for local development. To route +# alerts to a real destination, replace the webhook `url` below, or add a +# `slack` / `email` receiver to the same contact point (see ALERTING.md). +# +# The default anonymous-admin Grafana in docker-compose.yml picks this file up +# from /etc/grafana/provisioning/alerting/ on startup. + +apiVersion: 1 + +contactPoints: + - orgId: 1 + name: xrpld-default + receivers: + # Local-dev webhook sink. Points at a placeholder host; alerts are + # emitted but silently dropped unless a receiver listens there. + # Swap the url for a real endpoint (PagerDuty/Opsgenie/custom) or + # replace the whole receiver block with a slack/email type. + - uid: xrpld-default-webhook + type: webhook + settings: + url: http://localhost:9999/xrpld-alerts + httpMethod: POST + disableResolveMessage: false diff --git a/docker/telemetry/grafana/provisioning/alerting/policies.yaml b/docker/telemetry/grafana/provisioning/alerting/policies.yaml new file mode 100644 index 0000000000..5ef96a5a9a --- /dev/null +++ b/docker/telemetry/grafana/provisioning/alerting/policies.yaml @@ -0,0 +1,28 @@ +# Grafana notification-policy provisioning for rippled OTel alerts. +# +# Phase 9: Internal metric gap fill — alerting on health-critical metrics. +# +# The notification policy tree decides which contact point receives a firing +# alert and how alerts are batched. This stack uses a single flat policy: every +# xrpld alert routes to the `xrpld-default` contact point. +# +# Grouping by alertname + exported_instance means one notification per +# (alert, node) pair, so a fleet-wide problem does not collapse into a single +# ambiguous message. + +apiVersion: 1 + +policies: + - orgId: 1 + receiver: xrpld-default + group_by: + - alertname + - exported_instance + # Wait before sending the first notification for a new group, so a burst of + # related series is batched together. + group_wait: 30s + # Wait before adding new alerts to an already-notified group. + group_interval: 5m + # Minimum time before re-sending a notification for an alert that is still + # firing. + repeat_interval: 4h diff --git a/docker/telemetry/grafana/provisioning/alerting/rules.yaml b/docker/telemetry/grafana/provisioning/alerting/rules.yaml new file mode 100644 index 0000000000..6b69059720 --- /dev/null +++ b/docker/telemetry/grafana/provisioning/alerting/rules.yaml @@ -0,0 +1,409 @@ +# Grafana alert-rule provisioning for rippled OTel metrics. +# +# Phase 9: Internal metric gap fill — alerting on health-critical metrics. +# +# Six rules across three node-fatal subsystems: consensus/ledger health, +# validator health, and the job queue. Every metric referenced here is +# introduced by phase 9's MetricsRegistry. +# +# Rule shape (Grafana server-side evaluation): +# A Prometheus query — a 5-minute rate / histogram_quantile, aggregated +# `by (exported_instance)` so each node evaluates +# independently. Alert rules run headless and cannot +# use the dashboards' `$node` template variable. +# B reduce (last) — collapse A's series to its most recent value. +# C threshold — the firing condition; `condition: C`. +# +# Thresholds are documented in ALERTING.md and are intended to be tuned. + +apiVersion: 1 + +groups: + # ------------------------------------------------------------------ # + # Consensus / ledger health # + # ------------------------------------------------------------------ # + - orgId: 1 + name: xrpld-consensus + folder: xrpld + interval: 1m + rules: + # A closed ledger that later fails validation against the network — + # any sustained nonzero rate means this node built history the rest + # of the network rejects. + - uid: xrpld-ledger-history-mismatch + title: LedgerHistoryMismatch + condition: C + for: 5m + noDataState: NoData + execErrState: Error + labels: + severity: critical + category: consensus + annotations: + summary: "Ledger history mismatch on {{ $labels.exported_instance }}" + description: >- + Node {{ $labels.exported_instance }} is recording ledger history + mismatches ({{ $values.B.Value }}/s over 5m). The node's built ledger + diverges from the validated network chain. + data: + - refId: A + relativeTimeRange: + from: 600 + to: 0 + datasourceUid: prometheus + model: + refId: A + expr: sum by (exported_instance) (rate(xrpld_ledger_history_mismatch_total[5m])) + instant: true + range: false + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: + from: 600 + to: 0 + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: + type: __expr__ + uid: __expr__ + - refId: C + relativeTimeRange: + from: 600 + to: 0 + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + conditions: + - evaluator: + type: gt + params: [0] + datasource: + type: __expr__ + uid: __expr__ + + # Healthy nodes close a ledger every ~3-5s. Zero closes for 3 minutes + # means consensus/ledger advancement is stuck. NoData (metric absent) + # also fires — a vanished series here means the node is down. + - uid: xrpld-ledger-close-stalled + title: LedgerCloseStalled + condition: C + for: 3m + noDataState: Alerting + execErrState: Error + labels: + severity: critical + category: consensus + annotations: + summary: "Ledger closing stalled on {{ $labels.exported_instance }}" + description: >- + Node {{ $labels.exported_instance }} has closed no ledgers for + several minutes (5m rate has decayed to zero). Consensus or ledger + advancement is stuck. + data: + - refId: A + relativeTimeRange: + from: 600 + to: 0 + datasourceUid: prometheus + model: + refId: A + expr: sum by (exported_instance) (rate(xrpld_ledgers_closed_total[5m])) + instant: true + range: false + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: + from: 600 + to: 0 + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: + type: __expr__ + uid: __expr__ + - refId: C + relativeTimeRange: + from: 600 + to: 0 + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + conditions: + - evaluator: + type: lt + params: [0.001] + datasource: + type: __expr__ + uid: __expr__ + + # ------------------------------------------------------------------ # + # Validator health # + # ------------------------------------------------------------------ # + - orgId: 1 + name: xrpld-validator + folder: xrpld + interval: 1m + rules: + # This validator's own validations are not reaching / agreeing with + # the network. A sustained nonzero miss rate risks the validator being + # dropped from UNLs. + - uid: xrpld-validations-missed + title: ValidationsMissed + condition: C + for: 5m + noDataState: NoData + execErrState: Error + labels: + severity: warning + category: validator + annotations: + summary: "Validations missed on {{ $labels.exported_instance }}" + description: >- + Node {{ $labels.exported_instance }} is missing validations + ({{ $values.B.Value }}/s over 5m). Its validations are not agreeing with + the validated ledger. + data: + - refId: A + relativeTimeRange: + from: 600 + to: 0 + datasourceUid: prometheus + model: + refId: A + expr: sum by (exported_instance) (rate(xrpld_validation_missed_total[5m])) + instant: true + range: false + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: + from: 600 + to: 0 + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: + type: __expr__ + uid: __expr__ + - refId: C + relativeTimeRange: + from: 600 + to: 0 + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + conditions: + - evaluator: + type: gt + params: [0] + datasource: + type: __expr__ + uid: __expr__ + + # The node has stopped checking incoming validations. Zero checked + # validations means it is no longer processing the validation stream + # from peers. + - uid: xrpld-validations-not-checked + title: ValidationsNotChecked + condition: C + for: 5m + noDataState: Alerting + execErrState: Error + labels: + severity: warning + category: validator + annotations: + summary: "No validations checked on {{ $labels.exported_instance }}" + description: >- + Node {{ $labels.exported_instance }} has checked no incoming + validations for several minutes (5m rate has decayed to zero). The + validation stream from peers may have stopped. + data: + - refId: A + relativeTimeRange: + from: 600 + to: 0 + datasourceUid: prometheus + model: + refId: A + expr: sum by (exported_instance) (rate(xrpld_validations_checked_total[5m])) + instant: true + range: false + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: + from: 600 + to: 0 + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: + type: __expr__ + uid: __expr__ + - refId: C + relativeTimeRange: + from: 600 + to: 0 + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + conditions: + - evaluator: + type: lt + params: [0.001] + datasource: + type: __expr__ + uid: __expr__ + + # ------------------------------------------------------------------ # + # Job queue / resource health # + # ------------------------------------------------------------------ # + - orgId: 1 + name: xrpld-jobqueue + folder: xrpld + interval: 1m + rules: + # Transactions are being dropped because the job queue is full — the + # node is shedding load it cannot process. + - uid: xrpld-jobqueue-tx-overflow + title: JobQueueTxOverflow + condition: C + for: 5m + noDataState: NoData + execErrState: Error + labels: + severity: warning + category: jobqueue + annotations: + summary: "Job queue transaction overflow on {{ $labels.exported_instance }}" + description: >- + Node {{ $labels.exported_instance }} is overflowing its transaction + job queue ({{ $values.B.Value }}/s over 5m). Transactions are being + dropped under load. + data: + - refId: A + relativeTimeRange: + from: 600 + to: 0 + datasourceUid: prometheus + model: + refId: A + expr: sum by (exported_instance) (rate(xrpld_jq_trans_overflow_total[5m])) + instant: true + range: false + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: + from: 600 + to: 0 + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: + type: __expr__ + uid: __expr__ + - refId: C + relativeTimeRange: + from: 600 + to: 0 + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + conditions: + - evaluator: + type: gt + params: [0] + datasource: + type: __expr__ + uid: __expr__ + + # p99 time a job waits in the queue before running. A sustained p99 + # above 1s means the node is saturated and work is backing up. Tune the + # threshold (in microseconds) to the deployment. + - uid: xrpld-jobqueue-latency-high + title: JobQueueLatencyHigh + condition: C + for: 5m + noDataState: NoData + execErrState: Error + labels: + severity: warning + category: jobqueue + annotations: + summary: "Job queue latency high on {{ $labels.exported_instance }}" + description: >- + Node {{ $labels.exported_instance }} has a p99 job-queue wait of + {{ $values.B.Value }}µs (>1s) over 5m. The node is saturated and jobs are + backing up. + data: + - refId: A + relativeTimeRange: + from: 600 + to: 0 + datasourceUid: prometheus + model: + refId: A + expr: histogram_quantile(0.99, sum by (le, exported_instance) (rate(xrpld_job_queued_duration_us_bucket[5m]))) + instant: true + range: false + intervalMs: 1000 + maxDataPoints: 43200 + - refId: B + relativeTimeRange: + from: 600 + to: 0 + datasourceUid: __expr__ + model: + refId: B + type: reduce + reducer: last + expression: A + datasource: + type: __expr__ + uid: __expr__ + - refId: C + relativeTimeRange: + from: 600 + to: 0 + datasourceUid: __expr__ + model: + refId: C + type: threshold + expression: B + conditions: + - evaluator: + type: gt + params: [1000000] + datasource: + type: __expr__ + uid: __expr__ diff --git a/docs/superpowers/specs/2026-07-06-phase9-alerting-design.md b/docs/superpowers/specs/2026-07-06-phase9-alerting-design.md new file mode 100644 index 0000000000..0f3cfb39fd --- /dev/null +++ b/docs/superpowers/specs/2026-07-06-phase9-alerting-design.md @@ -0,0 +1,99 @@ +# Phase-9 Alerting — Design + +**Date:** 2026-07-06 +**Branch:** `pratik/otel-phase9-metric-gap-fill` (PR #6513, Jira RIPD-5187) +**Status:** Approved + +## Purpose + +Phase 9 exports ~68 internal rippled metrics and ships Grafana dashboards for +them. This adds the missing operator-facing piece: **provisioned Grafana alert +rules** that fire on the health-critical metrics phase 9 introduces. The +phase-9 task list (line 311) and Jira story RIPD-5187 both already list +"alerting rules" as a phase-9 deliverable, so this closes that gap. + +Scope is deliberately narrow — the three subsystems whose failure is +node-fatal: **consensus/ledger health, validator health, job queue**. RPC/API +health is explicitly out of scope. + +## Why phase 9 (not phase 11) + +Every metric these alerts fire on is _born_ in phase 9 +(`xrpld_ledger_history_mismatch_total`, `xrpld_ledgers_closed_total`, +`xrpld_validation_missed_total`, `xrpld_validations_checked_total`, +`xrpld_jq_trans_overflow_total`, `xrpld_job_queued_duration_us_bucket`). Alerts +belong with the metrics they watch, and this is where the dependency lives. + +## Delivery + +Provisioned YAML, version-controlled — matching the existing datasource / +dashboard provisioning pattern. No docker-compose change: the Grafana service +already mounts `./grafana/provisioning:/etc/grafana/provisioning:ro`, and +Grafana auto-loads `provisioning/alerting/*.yaml`. + +New files under `docker/telemetry/grafana/provisioning/alerting/`: + +| File | Purpose | +| -------------------- | ------------------------------------------------------------------------------------------------------------------- | +| `contactpoints.yaml` | One contact point `xrpld-default` (webhook to a documented placeholder; comments show how to swap for Slack/email). | +| `policies.yaml` | Default notification policy: route all alerts → `xrpld-default`, grouped by `alertname` + `exported_instance`. | +| `rules.yaml` | 6 alert rules across 3 groups (below). | + +Plus `docker/telemetry/ALERTING.md` — operator runbook: what each alert means, +likely causes, and how to point the contact point at a real receiver. + +## Alert rules + +All rules target Prometheus datasource `uid: prometheus`. Each rule uses the +Grafana rule shape: query (A) → reduce (B, last value) → threshold (C). All +`rate()`/`histogram_quantile()` expressions aggregate with +`sum by (exported_instance)` (or `+ le`) so **each node alerts independently**. +Alert rules run headless, so they cannot use the dashboards' `$node` template +variables — they match all series and group by `exported_instance` instead. + +| Group | Alert | Expression (5m window) | Fires | `for` | severity | +| --------- | --------------------- | --------------------------------------------------------------------------------------------------------- | --------------------- | ----- | -------- | +| Consensus | LedgerHistoryMismatch | `sum by (exported_instance)(rate(xrpld_ledger_history_mismatch_total[5m]))` | `> 0` | 5m | critical | +| Consensus | LedgerCloseStalled | `sum by (exported_instance)(rate(xrpld_ledgers_closed_total[5m]))` | `< 0.001` (≈0) | 3m | critical | +| Validator | ValidationsMissed | `sum by (exported_instance)(rate(xrpld_validation_missed_total[5m]))` | `> 0` | 5m | warning | +| Validator | ValidationsNotChecked | `sum by (exported_instance)(rate(xrpld_validations_checked_total[5m]))` | `< 0.001` (≈0) | 5m | warning | +| Job queue | JobQueueTxOverflow | `sum by (exported_instance)(rate(xrpld_jq_trans_overflow_total[5m]))` | `> 0` | 5m | warning | +| Job queue | JobQueueLatencyHigh | `histogram_quantile(0.99, sum by (le, exported_instance)(rate(xrpld_job_queued_duration_us_bucket[5m])))` | `> 1000000` (µs = 1s) | 5m | warning | + +Each rule carries labels `severity` and `category` (consensus/validator/jobqueue) +and annotations `summary` + `description` (with `{{ $labels.exported_instance }}` +and `{{ $values.B.Value }}` interpolation). + +### Threshold rationale + +- **LedgerCloseStalled `< 0.001` for 3m**: healthy nodes close a ledger every + ~3-5s; a 5m rate decaying to ~0 means the node is stuck. The epsilon (not + exact `0`) avoids float rate-noise suppressing the alert. +- **JobQueueLatencyHigh 1s p99**: a default starting point, easy to tune — jobs + queued >1s at p99 indicate the node is saturated. +- Others are `> 0` on error/miss counters: any sustained nonzero rate is + actionable. + +## Non-goals / YAGNI + +- No per-alert silencing schedules, no mute timings. +- No RPC/API, overlay, or fee-market alerts (dashboards cover those visually). +- Single contact point — multi-receiver routing is left to the operator. + +## Verification + +1. `yamllint` (or `python -c yaml.safe_load`) on all three YAML files. +2. `docker compose -f docker/telemetry/docker-compose.yml config -q` still parses. +3. Optional live check: start stack, `GET /api/v1/provisioning/alert-rules` + returns the 6 rules; Grafana logs show no provisioning errors. +4. Code-review pass (subagent) against phase conventions before commit. + +## Branch / PR restructure (independent, same session) + +- Rename `pratik/otel-phase11-benchmark` → `pratik/perf-test-otel-on` + (0 unique commits — equals phase-10 tip; the telemetry-IN baseline). +- Rename `pratik/otel-phase11-telemetry-off` → `pratik/perf-test-otel-off` + (carries the "compile telemetry OUT" build change + benchmark runbook). +- Close PR #7433, delete stale `origin/pratik/otel-phase11-telemetry-off`. +- Push both; open one perf PR `perf-test-otel-off → perf-test-otel-on` + (diff = the ON-vs-OFF delta; `-on` needs no PR of its own). diff --git a/src/xrpld/app/ledger/LedgerHistory.cpp b/src/xrpld/app/ledger/LedgerHistory.cpp index 5655cc2c6e..9b1e36defc 100644 --- a/src/xrpld/app/ledger/LedgerHistory.cpp +++ b/src/xrpld/app/ledger/LedgerHistory.cpp @@ -371,6 +371,12 @@ LedgerHistory::handleMismatch( return; } + // Tracks whether the mismatch reason has already been recorded. The + // consensus tx-set hash disagreement is the root cause, so it is counted + // once here; the tx-level comparison below still logs its diagnostics but + // must not record a second reason for the same mismatch event. + bool reasonRecorded = false; + if (builtConsensusHash && validatedConsensusHash) { if (builtConsensusHash != validatedConsensusHash) @@ -378,11 +384,8 @@ LedgerHistory::handleMismatch( JLOG(j_.error()) << "MISMATCH on consensus transaction set " << " built: " << to_string(*builtConsensusHash) << " validated: " << to_string(*validatedConsensusHash); - // The consensus tx-set hashes disagree — this is the root cause, - // so record it as the single reason and stop. The tx-level - // comparison below would otherwise double-count the same mismatch. recordReason("consensus_txset"); - return; + reasonRecorded = true; } else { @@ -398,13 +401,15 @@ LedgerHistory::handleMismatch( if (builtTx == validTx) { JLOG(j_.error()) << "MISMATCH with same " << builtTx.size() << " transactions"; - recordReason("same_txset_diff_result"); + if (!reasonRecorded) + recordReason("same_txset_diff_result"); } else { JLOG(j_.error()) << "MISMATCH with " << builtTx.size() << " built and " << validTx.size() << " valid transactions."; - recordReason("different_txset"); + if (!reasonRecorded) + recordReason("different_txset"); } JLOG(j_.error()) << "built\n" << getJson({*builtLedger, {}}); diff --git a/src/xrpld/app/ledger/detail/InboundLedger.cpp b/src/xrpld/app/ledger/detail/InboundLedger.cpp index 0f8ac3fa9f..75da7b1078 100644 --- a/src/xrpld/app/ledger/detail/InboundLedger.cpp +++ b/src/xrpld/app/ledger/detail/InboundLedger.cpp @@ -22,11 +22,11 @@ #include #include #include -#include +#include // IWYU pragma: keep #include #include #include -#include +#include // IWYU pragma: keep #include #include #include diff --git a/src/xrpld/app/main/Application.cpp b/src/xrpld/app/main/Application.cpp index e40448da8d..19e9fcd593 100644 --- a/src/xrpld/app/main/Application.cpp +++ b/src/xrpld/app/main/Application.cpp @@ -79,10 +79,12 @@ #include #include #include +#include // IWYU pragma: keep #include #include #include #include +#include // IWYU pragma: keep #include #include #include