feat(telemetry): scaffold fresh-node sync diagnostics (WP-A0)

Adds the anchors the sync-diagnostics signals attach to, with no signals
emitted yet:

- New "Ledger Sync Health" dashboard (uid ledger-sync-health) with the
  standard template-variable block copied from an existing board, plus
  empty "Bootstrap (Domain 0)" and "Sync pipeline" rows.
- Signal index section in the data-collection reference, an operator-flow
  stub in the telemetry runbook, and a glossary anchor.
- A sync_diagnostics group in expected_metrics.json and a matching
  assertion helper in validate_telemetry.py so CI fails when a signal
  regresses to absent.

Also registers the new dashboard uid with the harness so the board is
covered by validation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pratik Mankawde
2026-07-24 21:06:05 +01:00
parent 694062d8fd
commit 96914b9f40
6 changed files with 382 additions and 6 deletions

View File

@@ -499,6 +499,16 @@ A node stores ledger history as one or more contiguous ranges. One continuous ra
**Scope:** per node — measured on and specific to this individual server.
<a id="fresh-node-sync-diagnostics"></a>
### Fresh-node sync diagnostics
The set of signals that explain why a freshly-started node is slow to reach, or never reaches, a validated ledger. They split into pre-quorum bootstrap (DNS, peer dial, protocol negotiation, UNL fetch and quorum, clock skew) and the post-peering acquire pipeline (sync state, ledger and tx-set acquire, job queue, quorum and publish lag, back-fill, persistence). Rendered by the Ledger Sync Health dashboard; individual terms are defined below as each signal lands.
**Scope:** per node — measured on and specific to this individual server.
**See also:** [Diagnosing slow/stuck fresh sync](./telemetry-runbook.md#diagnosing-slowstuck-fresh-sync) (operator flow) · [Server states on xrpl.org](https://xrpl.org/docs/references/http-websocket-apis/api-conventions/xrpld-server-states)
<a id="historical-fetch-rate"></a>
### Historical fetch rate

View File

@@ -2043,6 +2043,33 @@ The `getKBUsed*()` methods require SQLite databases to exist. If running with
- Verify Loki is running: `curl http://localhost:3100/ready`
- Check the filelog receiver glob `/var/log/xrpld/*/debug.log` matches your log layout — the log file must sit one subdirectory below the mount root
### Diagnosing slow/stuck fresh sync
A fresh node that is slow to reach `server_state=full`, or never reaches it, is
diagnosed from the **Ledger Sync Health** dashboard (uid `ledger-sync-health`).
Walk the dashboard rows top-down — the first row that looks wrong is the stage
that is blocking:
1. **Bootstrap (Domain 0)** — can the node reach peers and form a quorum at all?
Covers DNS resolution, outbound dial, protocol/network-ID negotiation, UNL
fetch with trusted-key-vs-quorum, and clock skew. A node that never gets past
this row can never validate, no matter how healthy the pipeline looks.
2. **Sync pipeline** — is ledger data actually arriving and being applied?
Covers sync-state transitions, ledger/tx-set acquire progress, job-queue
backlog, quorum and publish lag, back-fill, and persistence latency.
Signal definitions:
[telemetry-glossary.md](./telemetry-glossary.md) "Fresh-node sync diagnostics".
Signal index (instrument, emit site, panel):
[09-data-collection-reference.md](../OpenTelemetryPlan/09-data-collection-reference.md)
"Fresh-node sync diagnostics".
Note: the sync signals are native metrics, which are never sampled — unlike the
span-derived (spanmetrics) series, they are always complete.
_(Per-signal diagnosis steps are added as each signal lands.)_
## Performance Tuning
| Scenario | Recommendation |