mirror of
https://github.com/XRPLF/rippled.git
synced 2026-08-23 15:20:54 +00:00
feat(telemetry): scaffold fresh-node sync diagnostics (WP-A0)
Adds the anchors the sync-diagnostics signals attach to, with no signals emitted yet: - New "Ledger Sync Health" dashboard (uid ledger-sync-health) with the standard template-variable block copied from an existing board, plus empty "Bootstrap (Domain 0)" and "Sync pipeline" rows. - Signal index section in the data-collection reference, an operator-flow stub in the telemetry runbook, and a glossary anchor. - A sync_diagnostics group in expected_metrics.json and a matching assertion helper in validate_telemetry.py so CI fails when a signal regresses to absent. Also registers the new dashboard uid with the harness so the board is covered by validation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -499,6 +499,16 @@ A node stores ledger history as one or more contiguous ranges. One continuous ra
|
||||
|
||||
**Scope:** per node — measured on and specific to this individual server.
|
||||
|
||||
<a id="fresh-node-sync-diagnostics"></a>
|
||||
|
||||
### Fresh-node sync diagnostics
|
||||
|
||||
The set of signals that explain why a freshly-started node is slow to reach, or never reaches, a validated ledger. They split into pre-quorum bootstrap (DNS, peer dial, protocol negotiation, UNL fetch and quorum, clock skew) and the post-peering acquire pipeline (sync state, ledger and tx-set acquire, job queue, quorum and publish lag, back-fill, persistence). Rendered by the Ledger Sync Health dashboard; individual terms are defined below as each signal lands.
|
||||
|
||||
**Scope:** per node — measured on and specific to this individual server.
|
||||
|
||||
**See also:** [Diagnosing slow/stuck fresh sync](./telemetry-runbook.md#diagnosing-slowstuck-fresh-sync) (operator flow) · [Server states on xrpl.org](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states)
|
||||
|
||||
<a id="historical-fetch-rate"></a>
|
||||
|
||||
### Historical fetch rate
|
||||
|
||||
@@ -2043,6 +2043,33 @@ The `getKBUsed*()` methods require SQLite databases to exist. If running with
|
||||
- Verify Loki is running: `curl http://localhost:3100/ready`
|
||||
- Check the filelog receiver glob `/var/log/xrpld/*/debug.log` matches your log layout — the log file must sit one subdirectory below the mount root
|
||||
|
||||
### Diagnosing slow/stuck fresh sync
|
||||
|
||||
A fresh node that is slow to reach `server_state=full`, or never reaches it, is
|
||||
diagnosed from the **Ledger Sync Health** dashboard (uid `ledger-sync-health`).
|
||||
|
||||
Walk the dashboard rows top-down — the first row that looks wrong is the stage
|
||||
that is blocking:
|
||||
|
||||
1. **Bootstrap (Domain 0)** — can the node reach peers and form a quorum at all?
|
||||
Covers DNS resolution, outbound dial, protocol/network-ID negotiation, UNL
|
||||
fetch with trusted-key-vs-quorum, and clock skew. A node that never gets past
|
||||
this row can never validate, no matter how healthy the pipeline looks.
|
||||
2. **Sync pipeline** — is ledger data actually arriving and being applied?
|
||||
Covers sync-state transitions, ledger/tx-set acquire progress, job-queue
|
||||
backlog, quorum and publish lag, back-fill, and persistence latency.
|
||||
|
||||
Signal definitions:
|
||||
[telemetry-glossary.md](./telemetry-glossary.md) "Fresh-node sync diagnostics".
|
||||
Signal index (instrument, emit site, panel):
|
||||
[09-data-collection-reference.md](../OpenTelemetryPlan/09-data-collection-reference.md)
|
||||
"Fresh-node sync diagnostics".
|
||||
|
||||
Note: the sync signals are native metrics, which are never sampled — unlike the
|
||||
span-derived (spanmetrics) series, they are always complete.
|
||||
|
||||
_(Per-signal diagnosis steps are added as each signal lands.)_
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
| Scenario | Recommendation |
|
||||
|
||||
Reference in New Issue
Block a user