mirror of
https://github.com/XRPLF/rippled.git
synced 2026-08-21 22:30:57 +00:00
feat(telemetry): add sync-state diagnostics (WP-A2)
Five signals that explain why a node is not advancing toward full, none of
which were observable before:
- state_changes_total now carries {from,to} mode labels, emitted at
setMode using the existing strOperatingMode helper. A bare count could
not distinguish a healthy climb from a node flapping between tracking
and connected. Removes the now-unused incrementStateChanges wrapper.
- sync_state{initial_full_duration_us}: time to first reach full, which
StateAccounting already computed but exposed only in server_info.
- sync_state{network_ledger_gate}: whether the node is still refusing to
build ledgers because it has no network ledger.
- sync_state{server_stall_seconds} and server_stall_events_total: how
long the main thread has been unresponsive. LoadManager computed this
and only logged it, so a stall was invisible until the fatal threshold.
The episode rule is a pure function so it can be tested without adding
a test-only mutator to LoadManager.
- sync_state{ledgers_behind}: how far our validated sequence trails the
best sequence any peer advertises, read from already-cached peer ranges
so no extra network traffic is added.
Also fixes the naming checker: it derived only the first label of a
multi-label instrument, so a dashboard querying the second label was
wrongly rejected.
Note: the clang-tidy hook cannot run in this worktree (no build
directory); the remaining pre-commit hooks, the naming check, dashboard
schema and harness syntax all pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -199,6 +199,7 @@ test.unit_test > xrpl.protocol
|
||||
tests.libxrpl > xrpl.basics
|
||||
tests.libxrpl > xrpl.config
|
||||
tests.libxrpl > xrpl.core
|
||||
tests.libxrpl > xrpld.app
|
||||
tests.libxrpl > xrpld.telemetry
|
||||
tests.libxrpl > xrpl.json
|
||||
tests.libxrpl > xrpl.ledger
|
||||
|
||||
16
.github/scripts/otel-naming/check_otel_naming.py
vendored
16
.github/scripts/otel-naming/check_otel_naming.py
vendored
@@ -156,7 +156,16 @@ BLOCK_COMMENT = re.compile(r"/\*.*?\*/", re.DOTALL)
|
||||
TRACEQL_SCOPE = re.compile(r"^(?:span|resource|event|link|instrumentation_scope)\.")
|
||||
# An OTel metric label key as emitted in C++: `Add(.., {{"label", ...}})` /
|
||||
# `{{"label", value}}` instrument calls in MetricsRegistry.
|
||||
#
|
||||
# Two patterns are needed because a label set is a nested initializer list:
|
||||
# `{{"a", x}, {"b", y}}`. The FIRST label is preceded by the doubled brace that
|
||||
# opens both the set and the pair, while every SUBSEQUENT label is preceded by
|
||||
# `}, {` closing the previous pair and opening the next. Matching only the
|
||||
# doubled-brace form would derive just the first label of every multi-label
|
||||
# instrument, silently under-deriving the L6 key set and making Rule D reject a
|
||||
# dashboard that queries a label the code genuinely emits.
|
||||
METRIC_LABEL = re.compile(r'\{\{\s*"([a-z_][a-z0-9_]*)"\s*,')
|
||||
METRIC_LABEL_NEXT = re.compile(r'\}\s*,\s*\{\s*"([a-z_][a-z0-9_]*)"\s*,')
|
||||
|
||||
|
||||
def strip_comments(text: str) -> str:
|
||||
@@ -799,7 +808,11 @@ def run_rule_c_tempo(root: Path, l1_keys: Set[str], report: Report) -> None:
|
||||
def metric_label_names(root: Path) -> Set[str]:
|
||||
"""L6: OTel native-metric label keys emitted by the telemetry code, e.g.
|
||||
`counter->Add(1, {{"job_type", value}})` in MetricsRegistry.cpp. These are
|
||||
a valid source of dashboard labels distinct from span attributes (L1)."""
|
||||
a valid source of dashboard labels distinct from span attributes (L1).
|
||||
|
||||
Collects both the first label of a set and every subsequent one, so a
|
||||
multi-label instrument such as `{{"from", a}, {"to", b}}` contributes ALL
|
||||
of its keys."""
|
||||
labels: Set[str] = set()
|
||||
for base in ("src", "include"):
|
||||
for p in (root / base).rglob("*.cpp"):
|
||||
@@ -809,6 +822,7 @@ def metric_label_names(root: Path) -> Set[str]:
|
||||
if "MetricsRegistry" not in p.name and "metric" not in text.lower():
|
||||
continue
|
||||
labels |= set(METRIC_LABEL.findall(text))
|
||||
labels |= set(METRIC_LABEL_NEXT.findall(text))
|
||||
return labels
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user