feat(telemetry): add sync-state diagnostics (WP-A2)

Five signals that explain why a node is not advancing toward full, none of
which were observable before:

- state_changes_total now carries {from,to} mode labels, emitted at
  setMode using the existing strOperatingMode helper. A bare count could
  not distinguish a healthy climb from a node flapping between tracking
  and connected. Removes the now-unused incrementStateChanges wrapper.
- sync_state{initial_full_duration_us}: time to first reach full, which
  StateAccounting already computed but exposed only in server_info.
- sync_state{network_ledger_gate}: whether the node is still refusing to
  build ledgers because it has no network ledger.
- sync_state{server_stall_seconds} and server_stall_events_total: how
  long the main thread has been unresponsive. LoadManager computed this
  and only logged it, so a stall was invisible until the fatal threshold.
  The episode rule is a pure function so it can be tested without adding
  a test-only mutator to LoadManager.
- sync_state{ledgers_behind}: how far our validated sequence trails the
  best sequence any peer advertises, read from already-cached peer ranges
  so no extra network traffic is added.

Also fixes the naming checker: it derived only the first label of a
multi-label instrument, so a dashboard querying the second label was
wrongly rejected.

Note: the clang-tidy hook cannot run in this worktree (no build
directory); the remaining pre-commit hooks, the naming check, dashboard
schema and harness syntax all pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Pratik Mankawde
2026-07-25 09:02:03 +01:00
parent 18106e17ae
commit 7c7509d01f
17 changed files with 1503 additions and 64 deletions

View File

@@ -199,6 +199,7 @@ test.unit_test > xrpl.protocol
tests.libxrpl > xrpl.basics
tests.libxrpl > xrpl.config
tests.libxrpl > xrpl.core
tests.libxrpl > xrpld.app
tests.libxrpl > xrpld.telemetry
tests.libxrpl > xrpl.json
tests.libxrpl > xrpl.ledger

View File

@@ -156,7 +156,16 @@ BLOCK_COMMENT = re.compile(r"/\*.*?\*/", re.DOTALL)
TRACEQL_SCOPE = re.compile(r"^(?:span|resource|event|link|instrumentation_scope)\.")
# An OTel metric label key as emitted in C++: `Add(.., {{"label", ...}})` /
# `{{"label", value}}` instrument calls in MetricsRegistry.
#
# Two patterns are needed because a label set is a nested initializer list:
# `{{"a", x}, {"b", y}}`. The FIRST label is preceded by the doubled brace that
# opens both the set and the pair, while every SUBSEQUENT label is preceded by
# `}, {` closing the previous pair and opening the next. Matching only the
# doubled-brace form would derive just the first label of every multi-label
# instrument, silently under-deriving the L6 key set and making Rule D reject a
# dashboard that queries a label the code genuinely emits.
METRIC_LABEL = re.compile(r'\{\{\s*"([a-z_][a-z0-9_]*)"\s*,')
METRIC_LABEL_NEXT = re.compile(r'\}\s*,\s*\{\s*"([a-z_][a-z0-9_]*)"\s*,')
def strip_comments(text: str) -> str:
@@ -799,7 +808,11 @@ def run_rule_c_tempo(root: Path, l1_keys: Set[str], report: Report) -> None:
def metric_label_names(root: Path) -> Set[str]:
"""L6: OTel native-metric label keys emitted by the telemetry code, e.g.
`counter->Add(1, {{"job_type", value}})` in MetricsRegistry.cpp. These are
a valid source of dashboard labels distinct from span attributes (L1)."""
a valid source of dashboard labels distinct from span attributes (L1).
Collects both the first label of a set and every subsequent one, so a
multi-label instrument such as `{{"from", a}, {"to", b}}` contributes ALL
of its keys."""
labels: Set[str] = set()
for base in ("src", "include"):
for p in (root / base).rglob("*.cpp"):
@@ -809,6 +822,7 @@ def metric_label_names(root: Path) -> Set[str]:
if "MetricsRegistry" not in p.name and "metric" not in text.lower():
continue
labels |= set(METRIC_LABEL.findall(text))
labels |= set(METRIC_LABEL_NEXT.findall(text))
return labels